kernel/git/stable/linux.git/mm/compaction.c, branch linux-6.18.y

mm: page_alloc: fix non-movable reclaim storm in defrag_mode

2026-09-07T15:22:23+00:00

commit 7e8756d7ad22655b935c384f123071aa9de07a27 upstream. As we deployed defrag_mode into Meta production, pressure spikes and excessive swapping were observed on some workloads. Tracing confirmed that this is unmovable/reclaimable requests spinning in the allocator and direct reclaim, causing excessive amounts of swap. The initial plan for defrag_mode was to rely on kswapd/kcompactd to produce blocks, and if those are overwhelmed under high pressure, let the allocator fall back (__rmqueue_steal()) after its retry loops. However, that retrying results in more reclaim on some of these workloads than we'd hoped, sometimes excessively so, spurred on by the !costly order conditions in should_reclaim_retry(). The storms are dependent on the request type. Reclaim will inevitably make room in existing movable blocks, since that's where the LRU pages live. So if movable requests retry on reclaim, they make progress. When non-movable requests spin in reclaim that isn't productive. They cannot use the individually freed pages, and the process is unlikely to accidentally free whole blocks to meet the ALLOC_NOFRAGMENT bar. They spin and overreclaim excessively, which tanks performance and triggers userspace guards like swap exhaustion or pressure based OOM. To fix this, send non-movable requests, regardless of order, into pageblock reclaim/compaction. This way, they help move things along to meet the ALLOC_NOFRAGMENT bar. After this patch, the reclaim storms and excess OOM rates are no longer observed in production. The longer-term plan is still to have all requests, including the movable ones, help make blocks to spread the cost of defragmenting more evenly and fairly; combined with proper watermarking to reduce allocation latencies in the common case. However, doing this naively unearths scaling and concurrency limitations in compaction that need to be addressed first. Promoting just non-movables for now is the minimally viable bug fix for the above issue. [brendan.jackman@linux.dev: fix try_to_compact_pages() kerneldoc] Link: https://lore.kernel.org/DK7NM9RPUJOD.11PNJJ5N2OBED@linux.dev Link: https://lore.kernel.org/20260722150006.3848560-5-hannes@cmpxchg.org Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode") Signed-off-by: Johannes Weiner Signed-off-by: "Brendan Jackman" Reviewed-by: Vlastimil Babka (SUSE) Cc: Brendan Jackman Cc: David Hildenbrand Cc: Gregory Price Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Zi Yan Cc: Signed-off-by: Andrew Morton Signed-off-by: Greg Kroah-Hartman

mm: page_alloc: move capture_control to the page allocator

2026-09-07T15:22:22+00:00

commit aee220f565cce38f0efcff940ae2b44bdc495408 upstream. The compaction capturing code assumes the allocation request order and compaction target order are the same. That won't be true once defrag_mode promotes sub-block allocations to pageblock-order compaction: compaction targets the larger order, while capture should remain at the original allocation order. Move the capture_control to the page allocator and give it its own copies of what the page freeing path matches against - zone, migratetype and the allocation order - rather than reaching into compaction's live compact_control. __alloc_pages_direct_compact() fills in migratetype and order, and installs and hides current->capture_control around the whole compaction call; try_to_compact_pages() aims capc->zone at each zone while it is being compacted. compact_zone_order() no longer deals with capture at all. Pass the capture_control through try_to_compact_pages() / compact_zone_order() in place of the bare struct page **. No functional change. Link: https://lore.kernel.org/20260722150006.3848560-4-hannes@cmpxchg.org Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode") Signed-off-by: Vlastimil Babka (SUSE) Co-developed-by: Johannes Weiner Signed-off-by: Johannes Weiner Reviewed-by: Gregory Price Cc: Brendan Jackman Cc: Brendan Jackman Cc: David Hildenbrand Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Zi Yan Cc: Signed-off-by: Andrew Morton Signed-off-by: Greg Kroah-Hartman

mm: compaction: support non-movable compaction for pageblock requests

2026-09-07T15:22:22+00:00

commit 1b4b697a5743e624cc259d6c31a8d8716b796305 upstream. While trying to fix a reclaim storm in defrag_mode, I noticed that non-movable direct compaction is extremely inefficient. When searching for space to evacuate, compaction only allows blocks of the same type as the incoming request. This is to prevent migratetype pollution, where a small non-movable request frees space in a movable block and provokes the allocator to fall back and pollute it. This protection is reasonable on one hand, but the downside is that it makes non-movable direct compaction nearly useless: if we get the type annotations right, by definition there aren't any movable pages inside the non-movable blocks it is allowed to scan. With defrag_mode, the goal is the production of whole blocks, which are essentially type neutral: __rmqueue_claim() will convert them wholesale on alloc. This makes type mixing and pollution a non-issue. Fix the pollution gates to take the requested order into account, and allow whole-block requests to scan blocks of other types. The only exception is CMA blocks. That type is sticky and these blocks cannot be claimed to other types. Continue to be strict with them, and allow only explicit ALLOC_CMA requests and kcompactd to evacuate them. Link: https://lore.kernel.org/20260722150006.3848560-3-hannes@cmpxchg.org Fixes: e3aa7df331bc ("mm: page_alloc: defrag_mode") Signed-off-by: Johannes Weiner Reviewed-by: Vlastimil Babka (SUSE) Reviewed-by: Gregory Price Cc: Brendan Jackman Cc: Brendan Jackman Cc: David Hildenbrand Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Zi Yan Cc: Signed-off-by: Andrew Morton Signed-off-by: Greg Kroah-Hartman

mm/compaction: handle free_pages_prepare() properly in compaction_free()

2026-07-24T14:17:15+00:00

commit 7da7d599b8a83271c464adfd5ef160202b470570 upstream. free_pages_prepare() can fail but compaction_free() does not handle the failure case. Failed pages should not be added back to cc->freepages for future use, since they can be either PageHWPoison or free_page_is_bad() and might cause data corruption. Link: https://lore.kernel.org/20260622-handle_free_pages_prepare_in_compaction_free-v1-1-fcf3b14abcf7@nvidia.com Fixes: 733aea0b3a7b ("mm/compaction: add support for >0 order folio memory compaction.") Signed-off-by: Zi Yan Reviewed-by: Vlastimil Babka (SUSE) Acked-by: Johannes Weiner Reviewed-by: Baolin Wang Reviewed-by: Lance Yang Cc: Brendan Jackman Cc: Jiaqi Yan Cc: Michal Hocko Cc: Suren Baghdasaryan Cc: Signed-off-by: Andrew Morton Signed-off-by: Greg Kroah-Hartman

mm/compaction: fix low_pfn advance on isolating hugetlb

2025-09-28T18:51:29+00:00

Commit 56ae0bb349b4 ("mm: compaction: convert to use a folio in isolate_migratepages_block()") converts api from page to folio. But the low_pfn advance for hugetlb page seems wrong when low_pfn doesn't point to head page. Originally, if page is a hugetlb tail page, compound_nr() return 1, which means low_pfn only advance one in next iteration. After the change, low_pfn would advance more than the hugetlb range, since folio_nr_pages() always return total number of the large page. This results in skipping some range to isolate and then to migrate. The worst case for alloc_contig is it does all the isolation and migration, but finally find some range is still not isolated. And then undo all the work and try a new range. Advance low_pfn to the end of hugetlb. Link: https://lkml.kernel.org/r/20250910092240.3981-1-richard.weiyang@gmail.com Fixes: 56ae0bb349b4 ("mm: compaction: convert to use a folio in isolate_migratepages_block()") Signed-off-by: Wei Yang Acked-by: Zi Yan Cc: "Vishal Moola (Oracle)" Cc: Kefeng Wang Cc: Oscar Salvador Cc: Brendan Jackman Cc: Johannes Weiner Cc: Michal Hocko Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Wei Yang Signed-off-by: Andrew Morton

mm: rename PG_isolated to PG_movable_ops_isolated

2025-07-13T23:38:30+00:00

Let's rename the flag to make it clearer where it applies (not folios ...). While at it, define the flag only with CONFIG_MIGRATION. Link: https://lkml.kernel.org/r/20250704102524.326966-22-david@redhat.com Signed-off-by: David Hildenbrand Reviewed-by: Zi Yan Reviewed-by: Lorenzo Stoakes Reviewed-by: Harry Yoo Cc: Alistair Popple Cc: Al Viro Cc: Arnd Bergmann Cc: Brendan Jackman Cc: Byungchul Park Cc: Chengming Zhou Cc: Christian Brauner Cc: Christophe Leroy Cc: Eugenio Pé rez Cc: Greg Kroah-Hartman Cc: Gregory Price Cc: "Huang, Ying" Cc: Jan Kara Cc: Jason Gunthorpe Cc: Jason Wang Cc: Jerrin Shaji George Cc: Johannes Weiner Cc: John Hubbard Cc: Jonathan Corbet Cc: Joshua Hahn Cc: Liam Howlett Cc: Madhavan Srinivasan Cc: Mathew Brost Cc: Matthew Wilcox (Oracle) Cc: Miaohe Lin Cc: Michael Ellerman Cc: "Michael S. Tsirkin" Cc: Michal Hocko Cc: Mike Rapoport Cc: Minchan Kim Cc: Naoya Horiguchi Cc: Nicholas Piggin Cc: Oscar Salvador Cc: Peter Xu Cc: Qi Zheng Cc: Rakie Kim Cc: Rik van Riel Cc: Sergey Senozhatsky Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Xuan Zhuo Cc: xu xin Signed-off-by: Andrew Morton

mm: convert "movable" flag in page->mapping to a page flag

2025-07-13T23:38:30+00:00

Instead, let's use a page flag. As the page flag can result in false-positives, glue it to the page types for which we support/implement movable_ops page migration. We are reusing PG_uptodate, that is for example used to track file system state and does not apply to movable_ops pages. So warning in case it is set in page_has_movable_ops() on other page types could result in false-positive warnings. Likely we could set the bit using a non-atomic update: in contrast to page->mapping, we could have others trying to update the flags concurrently when trying to lock the folio. In isolate_movable_ops_page(), we already take care of that by checking if the page has movable_ops before locking it. Let's start with the atomic variant, we could later switch to the non-atomic variant once we are sure other cases are similarly fine. Once we perform the switch, we'll have to introduce __SETPAGEFLAG_NOOP(). [david@redhat.com: add missing `:' in kerneldoc] Link: https://lkml.kernel.org/r/d96e2916-2c43-462c-b6a1-2375ef397d8b@redhat.com Link: https://lkml.kernel.org/r/20250704102524.326966-21-david@redhat.com Signed-off-by: David Hildenbrand Reviewed-by: Zi Yan Reviewed-by: Lorenzo Stoakes Cc: Alistair Popple Cc: Al Viro Cc: Arnd Bergmann Cc: Brendan Jackman Cc: Byungchul Park Cc: Chengming Zhou Cc: Christian Brauner Cc: Christophe Leroy Cc: Eugenio Pé rez Cc: Greg Kroah-Hartman Cc: Gregory Price Cc: Harry Yoo Cc: "Huang, Ying" Cc: Jan Kara Cc: Jason Gunthorpe Cc: Jason Wang Cc: Jerrin Shaji George Cc: Johannes Weiner Cc: John Hubbard Cc: Jonathan Corbet Cc: Joshua Hahn Cc: Liam Howlett Cc: Madhavan Srinivasan Cc: Mathew Brost Cc: Matthew Wilcox (Oracle) Cc: Miaohe Lin Cc: Michael Ellerman Cc: "Michael S. Tsirkin" Cc: Michal Hocko Cc: Mike Rapoport Cc: Minchan Kim Cc: Naoya Horiguchi Cc: Nicholas Piggin Cc: Oscar Salvador Cc: Peter Xu Cc: Qi Zheng Cc: Rakie Kim Cc: Rik van Riel Cc: Sergey Senozhatsky Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Xuan Zhuo Cc: xu xin Signed-off-by: Andrew Morton

mm: stop storing migration_ops in page->mapping

2025-07-13T23:38:29+00:00

... instead, look them up statically based on the page type. Maybe in the future we want a registration interface? At least for now, it can be easily handled using the two page types that actually support page migration. The remaining usage of page->mapping is to flag such pages as actually being movable (having movable_ops), which we will change next. Link: https://lkml.kernel.org/r/20250704102524.326966-20-david@redhat.com Signed-off-by: David Hildenbrand Reviewed-by: Zi Yan Reviewed-by: Lorenzo Stoakes Reviewed-by: Harry Yoo Cc: Alistair Popple Cc: Al Viro Cc: Arnd Bergmann Cc: Brendan Jackman Cc: Byungchul Park Cc: Chengming Zhou Cc: Christian Brauner Cc: Christophe Leroy Cc: Eugenio Pé rez Cc: Greg Kroah-Hartman Cc: Gregory Price Cc: "Huang, Ying" Cc: Jan Kara Cc: Jason Gunthorpe Cc: Jason Wang Cc: Jerrin Shaji George Cc: Johannes Weiner Cc: John Hubbard Cc: Jonathan Corbet Cc: Joshua Hahn Cc: Liam Howlett Cc: Madhavan Srinivasan Cc: Mathew Brost Cc: Matthew Wilcox (Oracle) Cc: Miaohe Lin Cc: Michael Ellerman Cc: "Michael S. Tsirkin" Cc: Michal Hocko Cc: Mike Rapoport Cc: Minchan Kim Cc: Naoya Horiguchi Cc: Nicholas Piggin Cc: Oscar Salvador Cc: Peter Xu Cc: Qi Zheng Cc: Rakie Kim Cc: Rik van Riel Cc: Sergey Senozhatsky Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Xuan Zhuo Cc: xu xin Signed-off-by: Andrew Morton

mm: rename __PageMovable() to page_has_movable_ops()

2025-07-13T23:38:29+00:00

Let's make it clearer that we are talking about movable_ops pages. While at it, convert a VM_BUG_ON to a VM_WARN_ON_ONCE_PAGE. Link: https://lkml.kernel.org/r/20250704102524.326966-17-david@redhat.com Signed-off-by: David Hildenbrand Reviewed-by: Zi Yan Reviewed-by: Lorenzo Stoakes Reviewed-by: Harry Yoo Cc: Alistair Popple Cc: Al Viro Cc: Arnd Bergmann Cc: Brendan Jackman Cc: Byungchul Park Cc: Chengming Zhou Cc: Christian Brauner Cc: Christophe Leroy Cc: Eugenio Pé rez Cc: Greg Kroah-Hartman Cc: Gregory Price Cc: "Huang, Ying" Cc: Jan Kara Cc: Jason Gunthorpe Cc: Jason Wang Cc: Jerrin Shaji George Cc: Johannes Weiner Cc: John Hubbard Cc: Jonathan Corbet Cc: Joshua Hahn Cc: Liam Howlett Cc: Madhavan Srinivasan Cc: Mathew Brost Cc: Matthew Wilcox (Oracle) Cc: Miaohe Lin Cc: Michael Ellerman Cc: "Michael S. Tsirkin" Cc: Michal Hocko Cc: Mike Rapoport Cc: Minchan Kim Cc: Naoya Horiguchi Cc: Nicholas Piggin Cc: Oscar Salvador Cc: Peter Xu Cc: Qi Zheng Cc: Rakie Kim Cc: Rik van Riel Cc: Sergey Senozhatsky Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Xuan Zhuo Cc: xu xin Signed-off-by: Andrew Morton

mm/migration: remove PageMovable()

2025-07-13T23:38:28+00:00

Previously, if __ClearPageMovable() were invoked on a page, this would cause __PageMovable() to return false, but due to the continued existence of page movable ops, PageMovable() would have returned true. With __ClearPageMovable() gone, the two are exactly equivalent. So we can replace PageMovable() checks by __PageMovable(). In fact, __PageMovable() cannot change until a page is freed, so we can turn some PageMovable() into sanity checks for __PageMovable(). Link: https://lkml.kernel.org/r/20250704102524.326966-16-david@redhat.com Signed-off-by: David Hildenbrand Reviewed-by: Zi Yan Reviewed-by: Lorenzo Stoakes Reviewed-by: Harry Yoo Cc: Alistair Popple Cc: Al Viro Cc: Arnd Bergmann Cc: Brendan Jackman Cc: Byungchul Park Cc: Chengming Zhou Cc: Christian Brauner Cc: Christophe Leroy Cc: Eugenio Pé rez Cc: Greg Kroah-Hartman Cc: Gregory Price Cc: "Huang, Ying" Cc: Jan Kara Cc: Jason Gunthorpe Cc: Jason Wang Cc: Jerrin Shaji George Cc: Johannes Weiner Cc: John Hubbard Cc: Jonathan Corbet Cc: Joshua Hahn Cc: Liam Howlett Cc: Madhavan Srinivasan Cc: Mathew Brost Cc: Matthew Wilcox (Oracle) Cc: Miaohe Lin Cc: Michael Ellerman Cc: "Michael S. Tsirkin" Cc: Michal Hocko Cc: Mike Rapoport Cc: Minchan Kim Cc: Naoya Horiguchi Cc: Nicholas Piggin Cc: Oscar Salvador Cc: Peter Xu Cc: Qi Zheng Cc: Rakie Kim Cc: Rik van Riel Cc: Sergey Senozhatsky Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Xuan Zhuo Cc: xu xin Signed-off-by: Andrew Morton