kernel/git/stable/linux.git/mm/hugetlb.c, branch linux-rolling-lts

hugetlb: only adjust reservation during unmapping if mapcount is 0

2026-09-07T15:22:35+00:00

commit 5120b1e048d48596ffaec1a8412012a91adba73b upstream. Since df7a6d1f6405, __unmap_hugepage_range can adjust reservations. In the case of folio mapped in both a parent and a child, if the parent unmaps the range first, the reservation adjustment will result in an underflow of the reserved count. Once the child unmaps the range, the count is restored. Change __unmap_hugepage_range() to check the mapcount before adjusting the reservation. Link: https://lore.kernel.org/all/alEJkwn5VlTTH_ZX@bender.morinfr.org/ Link: https://lore.kernel.org/amkC_1Ya6OiUoiLZ@bender.morinfr.org Fixes: df7a6d1f6405 ("mm/hugetlb: restore the reservation if needed") Signed-off-by: Guillaume Morin Reviewed-by: Breno Leitao Reviewed-by: Rik van Riel Cc: Muchun Song Cc: David Hildenbrand Cc: Oscar Salvador Cc: Signed-off-by: Andrew Morton Signed-off-by: Greg Kroah-Hartman

mm/hugetlb: fix swap entry corruption when clearing uffd-wp at fork()

2026-08-09T18:25:26+00:00

[ Upstream commit 83abe2fd5b3aeb3123b5408a5a91709c5538fb23 ] copy_hugetlb_page_range() clears the uffd-wp bit of migration and hwpoison entries with huge_pte_clear_uffd_wp(), which operates on the present-PTE bit position. Swap entries keep the uffd-wp state elsewhere -- the migration branch reads and sets it with pte_swp_uffd_wp() and pte_swp_mkuffd_wp() -- and the present-PTE position falls into the swap payload. On x86-64 it lands in the inverted swap offset, where a naturally-aligned hugetlb PFN always has the affected bit set, so the clear advances the encoded PFN by two pages. No userfaultfd needs to be involved: the clear is guarded only by the child VMA not being uffd-wp registered, so a plain fork() with an in-flight hugetlb migration entry (or a poisoned hugetlb page) corrupts the entry copied into the child. Instrumenting the clear and forking after MADV_HWPOISON on a 2MB anon hugetlb page shows: offset before=120e00 offset after =120e02 The fallout is mostly latent: rmap walks match migration entries by folio range and remove_migration_pte() rebuilds the PTE from the folio, so a within-folio PFN skew heals once migration completes. But any path that re-encodes the corrupted offset -- e.g. hugetlb_change_protection() rewriting a writable migration entry via make_readable_migration_entry(swp_offset(entry)) -- propagates it. Migration entries legitimately carry uffd-wp, so clear it with pte_swp_clear_uffd_wp(), matching copy_nonpresent_pte() and move_huge_pte(). A hwpoison entry, on the other hand, never carries the uffd-wp bit: it is installed fresh by make_hwpoison_entry() (try_to_unmap_one() does not preserve uffd-wp on the hwpoison path) and hugetlb_change_protection() leaves hwpoison entries untouched. There was nothing to clear there, only the corruption, so drop the clear entirely. Link: https://lore.kernel.org/20260708090110.136162-1-kirill@shutemov.name Fixes: bc70fbf269fd ("mm/hugetlb: handle uffd-wp during fork()") Signed-off-by: Kiryl Shutsemau Reported-by: Sashiko AI review Closes: https://lore.kernel.org/all/20260703140011.99E601F000E9@smtp.kernel.org/ Suggested-by: David Hildenbrand Acked-by: David Hildenbrand (Arm) Assisted-by: Claude:claude-fable-5 Cc: Muchun Song Cc: Oscar Salvador Cc: Peter Xu Cc: Signed-off-by: Andrew Morton (cherry picked from commit 83abe2fd5b3aeb3123b5408a5a91709c5538fb23) [ kas: adapt to the pre-softleaf idiom ] Signed-off-by: Kiryl Shutsemau (Meta) Signed-off-by: Sasha Levin

mm/hugetlb: fix list corruption in allocate_file_region_entries()

2026-08-09T18:25:11+00:00

commit dd9623f58ec702a07b2d67179d6fcea79c52231a upstream. allocate_file_region_entries() tops up resv->region_cache with freshly allocated file_region descriptors. The allocation uses GFP_KERNEL, so resv->lock is dropped around it: the new entries are gathered on a stack-local list head, allocated_regions, and spliced into resv->region_cache once the lock is re-acquired. The splice used list_splice(), which moves the entries but does not re-initialize the source head, so allocated_regions is left pointing at an entry that now lives on resv->region_cache. The top-up runs in a while loop that re-checks the cache deficit after re-acquiring the lock. For a shared mapping the resv_map is shared by every mapper of the hugetlbfs inode, so a concurrent region_chg()/region_add()/region_del() on the same resv_map can consume cache entries during the unlocked window and force a second iteration. That iteration calls list_add() on the stale head and corrupts the list; with CONFIG_DEBUG_LIST the __list_add_valid() check trips: list_add corruption. next->prev should be prev (ffffc900011ff7f8), but was ffff88814c281460. (next=ffff88814c545640). kernel BUG at lib/list_debug.c:31! allocate_file_region_entries+0x191/0x420 region_chg+0x267/0x300 hugetlb_reserve_pages+0x387/0xc80 hugetlbfs_file_mmap+0x2ce/0x3f0 mmap_region+0x1348/0x1a80 do_mmap+0x85e/0xb90 vm_mmap_pgoff+0x18c/0x330 ksys_mmap_pgoff+0x2a1/0x3e0 do_syscall_64+0xd7/0x420 Without CONFIG_DEBUG_LIST the bad list_add() silently links a kernel-stack address into resv->region_cache, leading to later use-after-free. This was observed as a real host panic on a dense KVM host where a QEMU guest-RAM hugetlbfs file was mapped MAP_SHARED by both QEMU and a separate SPDK/DPDK vhost-user target, generating concurrent region_* traffic on one shared resv_map. Use list_splice_init() so the source head is re-initialized empty after each splice, making the retry loop safe. Link: https://lore.kernel.org/20260713171456.300518-2-caixiangfeng@bytedance.com Fixes: d3ec7b6e09e5 ("mm/hugetlb: use list_splice to merge two list at once") Signed-off-by: Xiangfeng Cai Reviewed-by: Muchun Song Cc: Baoquan He Cc: David Hildenbrand Cc: Oscar Salvador Cc: Shuah Khan Cc: Wei Yang Cc: Signed-off-by: Andrew Morton Signed-off-by: Greg Kroah-Hartman

mm/hugetlb: fix hugetlb cgroup rsvd charge/uncharge mismatch

2026-07-24T14:17:00+00:00

commit 15807d0ddde37407af72859426b654f3d1972b00 upstream. In alloc_hugetlb_folio(), a single h_cg pointer is used for both the rsvd and non-rsvd hugetlb cgroup charges. When map_chg is set, hugetlb_cgroup_charge_cgroup_rsvd() stores the charged cgroup in h_cg, but the immediately following hugetlb_cgroup_charge_cgroup() overwrites h_cg with the non-rsvd cgroup pointer. As a result, hugetlb_cgroup_commit_charge_rsvd() stores the wrong (non-rsvd) cgroup pointer into the folio's rsvd slot. When the folio is later freed, free_huge_folio() unconditionally calls both hugetlb_cgroup_uncharge_folio() and hugetlb_cgroup_uncharge_folio_rsvd(). The rsvd uncharge reads back the wrong cgroup from the folio and decrements a counter that was never charged for that cgroup, causing a page_counter underflow: page_counter underflow: -512 nr_pages=512 WARNING: mm/page_counter.c:61 at page_counter_cancel Fix this by introducing a separate h_cg_rsvd pointer exclusively for the rsvd charge path, keeping the rsvd and non-rsvd charges fully independent through their charge, commit, and error uncharge paths. Link: https://lore.kernel.org/20260328065534.346053-1-kartikey406@gmail.com Fixes: 08cf9faf7558 ("hugetlb_cgroup: support noreserve mappings") Reported-by: syzbot+226c1f947186f8fef796@syzkaller.appspotmail.com Closes: https://syzkaller.appspot.com/bug?extid=226c1f947186f8fef796 Signed-off-by: Deepanshu Kartikey Reviewed-by: Muchun Song Cc: David Hildenbrand Cc: Oscar Salvador Cc: Mina Almasry Cc: Signed-off-by: Andrew Morton Signed-off-by: Greg Kroah-Hartman

mm/hugetlb: restore reservation on error in hugetlb folio copy paths

2026-06-19T11:44:10+00:00

commit 40c81856e622a9dc59294a90d169ac07ea25b0b0 upstream. Two sites in mm/hugetlb.c allocate a hugetlb folio via alloc_hugetlb_folio() (consuming a VMA reservation) and then call copy_user_large_folio(), which became int-returning in commit 1cb9dc4b475c ("mm: hwpoison: support recovery from HugePage copy-on-write faults") and can now fail (e.g. -EHWPOISON on a hwpoisoned source page). On the failure path, folio_put() restores the global hugetlb pool count through free_huge_folio(), but the per-VMA reservation map entry is left marked consumed: - hugetlb_mfill_atomic_pte() resubmission path (UFFDIO_COPY) - copy_hugetlb_page_range() fork-time CoW path when hugetlb_try_dup_anon_rmap() fails (rare: pinned hugetlb anon folio under fork) User-visible effect: on UFFDIO_COPY into a private hugetlb VMA where the resubmission copy fails, the reservation for that address is leaked from the VMA's reserve map. A subsequent fault at the same address takes the no-reservation path, and under hugetlb pool pressure the task is SIGBUSed at an address it had previously reserved. The fork-time CoW path leaks the same way in the child VMA's reserve map, though it requires the much rarer combination of pinned hugetlb anon page + hwpoisoned source. Add the missing restore_reserve_on_error() call before folio_put() on both error paths. Link: https://lore.kernel.org/20260520044912.6751-1-devnexen@gmail.com Fixes: 1cb9dc4b475c ("mm: hwpoison: support recovery from HugePage copy-on-write faults") Signed-off-by: David Carlier Reviewed-by: Muchun Song Cc: David Hildenbrand Cc: Mina Almasry Cc: Muchun Song Cc: Oscar Salvador Cc: yuehaibing Cc: Signed-off-by: Andrew Morton Signed-off-by: Greg Kroah-Hartman

mm/hugetlb: avoid false positive lockdep assertion

2026-06-19T11:44:10+00:00

commit b4aea43cd37afad714b5684fe9fdfcb0e78dba26 upstream. Commit 081056dc00a2 ("mm/hugetlb: unshare page tables during VMA split, not before") changed the locking model around hugetlbfs PMD unsharing on VMA split, but did not update the function which asserts the locks, hugetlb_vma_assert_locked(). This function asserts that either the hugetlb VMA lock is held (if a shared mapping) or that the reservation map lock is held (if private). If you get an unfortunate race between something which results in one of these locks being released and a hugetlb VMA split and you have CONFIG_LOCKDEP enabled, you can therefore see a false positive assertion arise when there is in fact no issue. Since this change introduced a new take_locks parameter to hugetlb_unshare_pmds(), which, when set to false, indicates that locking is sufficient, simply pass this to the unsharing logic and predicate the lock assertions on this. This is safe, as we already asserted the file rmap lock and the VMA write lock prior to this (implying exclusive mmap write lock), so we cannot be raced by either rmap or page fault page table walkers which the asserted locks are intended to protect against (we don't mind GUP-fast). Separate out huge_pmd_unshare() into __huge_pmd_unshare() to add a check_locks parameter, and update hugetlb_unshare_pmds() to pass this parameter to it. This leaves all other callers of huge_pmd_unshare() still correctly asserting the locks. The below reproducer will trigger the assert in a kernel with CONFIG_LOCKDEP enabled by racing process teardown (which will release the hugetlb lock) against a hugetlb split. void execute_one(void) { void *ptr; pid_t pid; /* * Create a hugetlb mapping spanning a PUD entry. * * We force the hugetlb page allocation with populate and * noreserve. * * |---------------------| * | | * |---------------------| * 0 PUD boundary */ ptr = mmap(0, PUD_SIZE, PROT_READ | PROT_WRITE, MAP_FIXED | MAP_SHARED | MAP_ANON | MAP_NORESERVE | MAP_HUGETLB | MAP_POPULATE, -1, 0); if (ptr == MAP_FAILED) { perror("mmap"); exit(EXIT_FAILURE); } /* * Fork but with a bogus stack pointer so we try to execute code in * a non-VM_EXEC VMA, causing segfault + teardown via exit_mmap(). * * The clone will cause PMD page table sharing between the * processes first via: * copy_process() -> ... -> huge_pte_alloc() -> huge_pmd_share() * * Then tear down and release the hugetlb 'VMA' lock via: * exit_mmap() -> ... -> vma_close() -> hugetlb_vma_lock_free() */ pid = syscall(__NR_clone, 0, 2 * PMD_SIZE, 0, 0, 0); if (pid < 0) { perror("clone"); exit(EXIT_FAILURE); } if (pid == 0) { /* Pop stack... */ return; } /* * We are the parent process. * * Race the child process's teardown with a PMD unshare. * * We do this by triggering: * * __split_vma() -> hugetlb_split() -> hugetlb_unshare_pmds() * * Which, importantly, doesn't hold the hugetlb VMA lock (nor can * it), meaning we assert in hugetlb_vma_assert_locked(). * * . * |----------.----------| * | . | * |----------.----------| * 0 . PUD boundary */ mmap(0, PUD_SIZE / 2, PROT_READ | PROT_WRITE, MAP_FIXED | MAP_ANON | MAP_PRIVATE, -1, 0); } int main(void) { int i; /* Kick off fork children. */ for (i = 0; i < NUM_FORKS; i++) { pid_t pid = fork(); if (pid < 0) { perror("fork"); exit(EXIT_FAILURE); } /* Fork children do their work and exit. */ if (!pid) { int j; for (j = 0; j < NUM_ITERS; j++) execute_one(); return EXIT_SUCCESS; } } /* If we succeeded, wait on children. */ for (i = 0; i < NUM_FORKS; i++) wait(NULL); return EXIT_SUCCESS; } [ljs@kernel.org: account for the !CONFIG_HUGETLB_PMD_PAGE_TABLE_SHARING case] Link: https://lore.kernel.org/agWZsPGYid08uU6O@lucifer Link: https://lore.kernel.org/20260513085658.45264-1-ljs@kernel.org Fixes: 081056dc00a2 ("mm/hugetlb: unshare page tables during VMA split, not before") Signed-off-by: Lorenzo Stoakes Acked-by: David Hildenbrand (Arm) Acked-by: Oscar Salvador Cc: Jann Horn Cc: Muchun Song Cc: Signed-off-by: Andrew Morton Signed-off-by: Greg Kroah-Hartman

mm/memory-failure: fix hugetlb_lock AA deadlock in get_huge_page_for_hwpoison

2026-06-19T11:44:02+00:00

commit 3c2d42b8ee345b17a4ba56b0f6492d1ff4c1178e upstream. Two concurrent madvise(MADV_HWPOISON) calls on the same hugetlb page can trigger a recursive spinlock self-deadlock (AA deadlock) on hugetlb_lock when racing with a concurrent unmap: thread#0 thread#1 -------- -------- madvise(folio, MADV_HWPOISON) -> poisons the folio successfully madvise(folio, MADV_HWPOISON) unmap(folio) try_memory_failure_hugetlb get_huge_page_for_hwpoison spin_lock_irq(&hugetlb_lock) <- held __get_huge_page_for_hwpoison hugetlb_update_hwpoison() -> MF_HUGETLB_FOLIO_PRE_POISONED goto out: folio_put() refcount: 1 -> 0 free_huge_folio() spin_lock_irqsave(&hugetlb_lock) -> AA DEADLOCK! The out: path in __get_huge_page_for_hwpoison() calls folio_put() to drop the GUP reference while the hugetlb_lock is still held by the hugetlb.c wrapper get_huge_page_for_hwpoison(). If concurrent unmap has released the page table mapping reference, folio_put() drops the folio refcount to zero, triggering free_huge_folio() which attempts to re-acquire the non-recursive hugetlb_lock. Fix this by moving hugetlb_lock acquisition from the hugetlb.c wrapper into get_huge_page_for_hwpoison(). Place spin_unlock_irq() before the folio_put() at the out: label so the folio is always released outside the lock. [akpm@linux-foundation.org: fix race, rename label per Miaohe] Link: https://sashiko.dev/#/patchset/20260522010305.4099834-1-mawupeng1@huawei.com Link: https://lore.kernel.org/f39f405e-4b4b-8f79-70fe-a2b5b62114eb@huawei.com Link: https://lore.kernel.org/20260522010305.4099834-1-mawupeng1@huawei.com Fixes: 405ce051236c ("mm/hwpoison: fix race between hugetlb free/demotion and memory_failure_hugetlb()") Signed-off-by: Wupeng Ma Acked-by: Oscar Salvador (SUSE) Acked-by: Muchun Song Reviewed-by: Kefeng Wang Acked-by: Miaohe Lin Cc: David Hildenbrand Cc: Liam Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Naoya Horiguchi Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Signed-off-by: Andrew Morton Signed-off-by: Greg Kroah-Hartman

mm/hugetlb: fix early boot crash on parameters without '=' separator

2026-05-07T04:11:39+00:00

commit c45b354911d01565156e38d7f6bc07edb51fc34c upstream. If hugepages, hugepagesz, or default_hugepagesz are specified on the kernel command line without the '=' separator, early parameter parsing passes NULL to hugetlb_add_param(), which dereferences it in strlen() and can crash the system during early boot. Reject NULL values in hugetlb_add_param() and return -EINVAL instead. Link: https://lore.kernel.org/20260409105437.108686-4-thorsten.blum@linux.dev Fixes: 5b47c02967ab ("mm/hugetlb: convert cmdline parameters from setup to early") Signed-off-by: Thorsten Blum Reviewed-by: Muchun Song Cc: David Hildenbrand Cc: Frank van der Linden Cc: Oscar Salvador Cc: Signed-off-by: Andrew Morton Signed-off-by: Greg Kroah-Hartman

mm/hugetlb: restore failed global reservations to subpool

2026-03-04T12:21:29+00:00

[ Upstream commit 1d3f9bb4c8af70304d19c22e30f5d16a2d589bb5 ] Commit a833a693a490 ("mm: hugetlb: fix incorrect fallback for subpool") fixed an underflow error for hstate->resv_huge_pages caused by incorrectly attributing globally requested pages to the subpool's reservation. Unfortunately, this fix also introduced the opposite problem, which would leave spool->used_hpages elevated if the globally requested pages could not be acquired. This is because while a subpool's reserve pages only accounts for what is requested and allocated from the subpool, its "used" counter keeps track of what is consumed in total, both from the subpool and globally. Thus, we need to adjust spool->used_hpages in the other direction, and make sure that globally requested pages are uncharged from the subpool's used counter. Each failed allocation attempt increments the used_hpages counter by how many pages were requested from the global pool. Ultimately, this renders the subpool unusable, as used_hpages approaches the max limit. The issue can be reproduced as follows: 1. Allocate 4 hugetlb pages 2. Create a hugetlb mount with max=4, min=2 3. Consume 2 pages globally 4. Request 3 pages from the subpool (2 from subpool + 1 from global) 4.1 hugepage_subpool_get_pages(spool, 3) succeeds. used_hpages += 3 4.2 hugetlb_acct_memory(h, 1) fails: no global pages left used_hpages -= 2 5. Subpool now has used_hpages = 1, despite not being able to successfully allocate any hugepages. It believes it can now only allocate 3 more hugepages, not 4. With each failed allocation attempt incrementing the used counter, the subpool eventually reaches a point where its used counter equals its max counter. At that point, any future allocations that try to allocate hugeTLB pages from the subpool will fail, despite the subpool not having any of its hugeTLB pages consumed by any user. Once this happens, there is no way to make the subpool usable again, since there is no way to decrement the used counter as no process is really consuming the hugeTLB pages. The underflow issue that the original commit fixes still remains fixed as well. Without this fix, used_hpages would keep on leaking if hugetlb_acct_memory() fails. Link: https://lkml.kernel.org/r/20260116204037.2270096-1-joshua.hahnjy@gmail.com Fixes: a833a693a490 ("mm: hugetlb: fix incorrect fallback for subpool") Signed-off-by: Joshua Hahn Acked-by: Usama Arif Cc: David Hildenbrand Cc: "Liam R. Howlett" Cc: Lorenzo Stoakes Cc: Ma Wupeng Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Oscar Salvador Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Waiman Long Cc: Signed-off-by: Andrew Morton Signed-off-by: Sasha Levin

mm/hugetlb: fix excessive IPI broadcasts when unsharing PMD tables using mmu_gather

2026-02-19T15:31:34+00:00

commit 8ce720d5bd91e9dc16db3604aa4b1bf76770a9a1 upstream. As reported, ever since commit 1013af4f585f ("mm/hugetlb: fix huge_pmd_unshare() vs GUP-fast race") we can end up in some situations where we perform so many IPI broadcasts when unsharing hugetlb PMD page tables that it severely regresses some workloads. In particular, when we fork()+exit(), or when we munmap() a large area backed by many shared PMD tables, we perform one IPI broadcast per unshared PMD table. There are two optimizations to be had: (1) When we process (unshare) multiple such PMD tables, such as during exit(), it is sufficient to send a single IPI broadcast (as long as we respect locking rules) instead of one per PMD table. Locking prevents that any of these PMD tables could get reused before we drop the lock. (2) When we are not the last sharer (> 2 users including us), there is no need to send the IPI broadcast. The shared PMD tables cannot become exclusive (fully unshared) before an IPI will be broadcasted by the last sharer. Concurrent GUP-fast could walk into a PMD table just before we unshared it. It could then succeed in grabbing a page from the shared page table even after munmap() etc succeeded (and supressed an IPI). But there is not difference compared to GUP-fast just sleeping for a while after grabbing the page and re-enabling IRQs. Most importantly, GUP-fast will never walk into page tables that are no-longer shared, because the last sharer will issue an IPI broadcast. (if ever required, checking whether the PUD changed in GUP-fast after grabbing the page like we do in the PTE case could handle this) So let's rework PMD sharing TLB flushing + IPI sync to use the mmu_gather infrastructure so we can implement these optimizations and demystify the code at least a bit. Extend the mmu_gather infrastructure to be able to deal with our special hugetlb PMD table sharing implementation. To make initialization of the mmu_gather easier when working on a single VMA (in particular, when dealing with hugetlb), provide tlb_gather_mmu_vma(). We'll consolidate the handling for (full) unsharing of PMD tables in tlb_unshare_pmd_ptdesc() and tlb_flush_unshared_tables(), and track in "struct mmu_gather" whether we had (full) unsharing of PMD tables. Because locking is very special (concurrent unsharing+reuse must be prevented), we disallow deferring flushing to tlb_finish_mmu() and instead require an explicit earlier call to tlb_flush_unshared_tables(). From hugetlb code, we call huge_pmd_unshare_flush() where we make sure that the expected lock protecting us from concurrent unsharing+reuse is still held. Check with a VM_WARN_ON_ONCE() in tlb_finish_mmu() that tlb_flush_unshared_tables() was properly called earlier. Document it all properly. Notes about tlb_remove_table_sync_one() interaction with unsharing: There are two fairly tricky things: (1) tlb_remove_table_sync_one() is a NOP on architectures without CONFIG_MMU_GATHER_RCU_TABLE_FREE. Here, the assumption is that the previous TLB flush would send an IPI to all relevant CPUs. Careful: some architectures like x86 only send IPIs to all relevant CPUs when tlb->freed_tables is set. The relevant architectures should be selecting MMU_GATHER_RCU_TABLE_FREE, but x86 might not do that in stable kernels and it might have been problematic before this patch. Also, the arch flushing behavior (independent of IPIs) is different when tlb->freed_tables is set. Do we have to enlighten them to also take care of tlb->unshared_tables? So far we didn't care, so hopefully we are fine. Of course, we could be setting tlb->freed_tables as well, but that might then unnecessarily flush too much, because the semantics of tlb->freed_tables are a bit fuzzy. This patch changes nothing in this regard. (2) tlb_remove_table_sync_one() is not a NOP on architectures with CONFIG_MMU_GATHER_RCU_TABLE_FREE that actually don't need a sync. Take x86 as an example: in the common case (!pv, !X86_FEATURE_INVLPGB) we still issue IPIs during TLB flushes and don't actually need the second tlb_remove_table_sync_one(). This optimized can be implemented on top of this, by checking e.g., in tlb_remove_table_sync_one() whether we really need IPIs. But as described in (1), it really must honor tlb->freed_tables then to send IPIs to all relevant CPUs. Notes on TLB flushing changes: (1) Flushing for non-shared PMD tables We're converting from flush_hugetlb_tlb_range() to tlb_remove_huge_tlb_entry(). Given that we properly initialize the MMU gather in tlb_gather_mmu_vma() to be hugetlb aware, similar to __unmap_hugepage_range(), that should be fine. (2) Flushing for shared PMD tables We're converting from various things (flush_hugetlb_tlb_range(), tlb_flush_pmd_range(), flush_tlb_range()) to tlb_flush_pmd_range(). tlb_flush_pmd_range() achieves the same that tlb_remove_huge_tlb_entry() would achieve in these scenarios. Note that tlb_remove_huge_tlb_entry() also calls __tlb_remove_tlb_entry(), however that is only implemented on powerpc, which does not support PMD table sharing. Similar to (1), tlb_gather_mmu_vma() should make sure that TLB flushing keeps on working as expected. Further, note that the ptdesc_pmd_pts_dec() in huge_pmd_share() is not a concern, as we are holding the i_mmap_lock the whole time, preventing concurrent unsharing. That ptdesc_pmd_pts_dec() usage will be removed separately as a cleanup later. There are plenty more cleanups to be had, but they have to wait until this is fixed. [david@kernel.org: fix kerneldoc] Link: https://lkml.kernel.org/r/f223dd74-331c-412d-93fc-69e360a5006c@kernel.org Link: https://lkml.kernel.org/r/20251223214037.580860-5-david@kernel.org Fixes: 1013af4f585f ("mm/hugetlb: fix huge_pmd_unshare() vs GUP-fast race") Signed-off-by: David Hildenbrand (Red Hat) Reported-by: "Uschakow, Stanislav" Closes: https://lore.kernel.org/all/4d3878531c76479d9f8ca9789dc6485d@amazon.de/ Tested-by: Laurence Oberman Acked-by: Harry Yoo Reviewed-by: Lorenzo Stoakes Cc: Lance Yang Cc: Liu Shixin Cc: Oscar Salvador Cc: Rik van Riel Cc: Signed-off-by: Andrew Morton Signed-off-by: David Hildenbrand (Arm) Signed-off-by: Greg Kroah-Hartman