kernel/git/stable/linux.git/mm/swap.c, branch master

mm: rename swap.c to folio.c

2026-08-05T02:18:35+00:00

Rename mm/swap.c to mm/folio.c so the filename better matches the code's main responsibility. This keeps the implementation split from swap-specific code without changing the published LRU helper interfaces. Update MAINTAINERS and the remaining mm/swap.c documentation references after the rename. [wujianyue000@gmail.com: MAINTAINERS: move mm/folio.c to MM CORE] Link: https://lore.kernel.org/20260804111109.393127-1-wujianyue000@gmail.com Link: https://lore.kernel.org/20260708-ch-swap-series-plus-folio-lru-cleanup-v9-2-2bc72b4f8730@gmail.com Signed-off-by: Jianyue Wu Suggested-by: Baoquan He Suggested-by: David Hildenbrand Suggested-by: Matthew Wilcox Reviewed-by: Lorenzo Stoakes Acked-by: Johannes Weiner Acked-by: David Hildenbrand (Arm) Cc: Axel Rasmussen Cc: Baolin Wang Cc: Barry Song Cc: Chris Li Cc: Hugh Dickins Cc: Jonathan Corbet Cc: Kairui Song Cc: Kemeng Shi Cc: Liam R. Howlett Cc: Michal Hocko Cc: Michal Hocko Cc: Mike Rapoport Cc: Nhat Pham Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton

mm/swap: colocate page-cluster sysctl with swap readahead

2026-08-05T02:18:34+00:00

Patch series "mm: clean up folio LRU and swap declarations", v9. This series splits folio LRU and swap cleanup into three steps: - move the page-cluster sysctl next to swap readahead in mm/swap_state.c - rename mm/swap.c to mm/folio.c after the swap-specific code moves out - move MM-internal reclaim declarations out of include/linux/swap.h After this series, swap cache and swap-in readahead stay in mm/swap_state.c. Folio LRU helpers live in mm/folio.c, and MM-internal reclaim/workingset declarations move from include/linux/swap.h to mm/internal.h (public LRU helpers used outside mm/ remain in swap.h). The first patch handles the swap-specific page-cluster state before the file rename, so the rename commit only carries folio LRU code. The last patch keeps the LRU helpers used outside mm/ in include/linux/swap.h and moves the internal reclaim/workingset declarations to mm/internal.h. This patch (of 3): page_cluster and the vm.page-cluster sysctl are only used by swap-in readahead in swap_state.c. Move them out of swap.c together with swap_readahead_setup(), and make page_cluster static to that file. Rename swap_setup() while moving it as well. The helper is internal to MM and now only sets up swap readahead defaults and its sysctl hook, so the more specific name matches its reduced scope. Call swap_readahead_setup() from swap_init() after moving it, keeping the readahead defaults and sysctl registration with swap_state.c initialization. swap_setup() previously lived in mm/swap.c, which is built unconditionally, so the vm.page-cluster sysctl was registered also on CONFIG_SWAP=n kernels. After moving the setup into swap_state.c, which is built only when CONFIG_SWAP is enabled, vm.page-cluster is no longer registered there. The knob only tunes swap-in readahead and had no effect without swap. Link: https://lore.kernel.org/20260708-ch-swap-series-plus-folio-lru-cleanup-v9-0-2bc72b4f8730@gmail.com Link: https://lore.kernel.org/20260708-ch-swap-series-plus-folio-lru-cleanup-v9-1-2bc72b4f8730@gmail.com Signed-off-by: Jianyue Wu Suggested-by: Baoquan He Suggested-by: Barry Song Suggested-by: Johannes Weiner Suggested-by: Matthew Wilcox Reviewed-by: Kairui Song Acked-by: Johannes Weiner Acked-by: David Hildenbrand (Arm) Cc: Axel Rasmussen Cc: Baolin Wang Cc: Chris Li Cc: Hugh Dickins Cc: Jonathan Corbet Cc: Kemeng Shi Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Michal Hocko Cc: Mike Rapoport Cc: Nhat Pham Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton

mm: split out internal page_alloc.h

2026-07-31T02:40:41+00:00

internal.h is a bit bloated, seems like time for a page_alloc.h. Where it wasn't obvious, the heuristic for deciding what goes into this new header was "does it support/correspond to a definition in mm/page_alloc.c?" Only need to include it from ~20 .c files out of ~150 so this does seem like a genuine reduction in scopes, which is nice. And there's no circular internal.h<->page_alloc.h dependency, so it seems worthwhile to split this up before that inevitably emerges! Link: https://lore.kernel.org/20260703-alloc-trylock-v5-4-c87b714e19d3@google.com Signed-off-by: Brendan Jackman Suggested-by: "David Hildenbrand (Arm)" Link: https://lore.kernel.org/all/41e92bab-6882-401a-8de9-154adbdcfb36@kernel.org/ Reviewed-by: Vlastimil Babka (SUSE) Acked-by: Zi Yan Signed-off-by: Andrew Morton

mm: annotate data-race in cpu_needs_drain()

2026-07-29T04:12:02+00:00

KCSAN reports a data-race when cpu_needs_drain() reads another CPU's per-cpu folio_batch->nr without locking, while the owning CPU writes to it via folio_batch_add(). Reading a slightly stale value is harmless -- cpu_needs_drain() only decides whether to schedule a drain, and the next iteration of __lru_add_drain_all() will re-check. Use data_race() to annotate the intentional race. [akpm@linux-foundation.org: reindent cpu_needs_drain, per David & Lorenzo] Link: https://lore.kernel.org/20260626053700.2036899-1-wangxuewen@kylinos.cn Signed-off-by: Xuewen Wang Acked-by: David Hildenbrand (Arm) Reviewed-by: Pedro Falcato Reviewed-by: Lorenzo Stoakes Cc: Axel Rasmussen Cc: Baoquan He Cc: Barry Song Cc: Chris Li Cc: Jann Horn Cc: Kairui Song Cc: Kemeng Shi Cc: Liam R. Howlett Cc: Nhat Pham Cc: Shakeel Butt Cc: Vlastimil Babka Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton

mm/lruvec: trace LRU add drains and drain-all requests

2026-07-29T04:11:45+00:00

LRU add batches can be drained before they reach capacity. This can be a source of LRU lock contention, but it is not currently possible to attribute these drains to callers with existing tracepoints. Add mm_lru_add_drain to report the CPU and lru_add batch count when an lru_add batch is drained. This allows tracing to distinguish full drains from partial drains and attribute them to the calling stack. Add mm_lru_add_drain_all to capture callers of __lru_add_drain_all and whether they set the force flag for all CPUs. The tracepoint resembles the signature of the enclosing function, but is needed because of potential inlining. Note that DECLARE_TRACE() is used for these new trace hooks to avoid creating a new trace event ABI. Link: https://lore.kernel.org/20260622185127.24579-1-jp.kobryn@linux.dev Signed-off-by: JP Kobryn Reviewed-by: Barry Song Acked-by: Shakeel Butt Cc: Axel Rasmussen Cc: Baoquan He Cc: Chris Li Cc: Kairui Song Cc: Kemeng Shi Cc: "Masami Hiramatsu (Google)" Cc: Mathieu Desnoyers Cc: Matthew Wilcox (Oracle) Cc: Michal Hocko Cc: Nhat Pham Cc: Steven Rostedt Cc: Vlastimil Babka Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton

mm/mglru: use folio_mark_accessed to replace folio_set_active

2026-06-04T21:45:02+00:00

MGLRU gives high priority to folios mapped in page tables. As a result, folio_set_active() is invoked for all folios read during page faults. In practice, however, readahead can bring in many folios that are never accessed via page tables. A previous attempt by Lei Liu proposed introducing a separate LRU for readahead[1] to make readahead pages easier to reclaim, but that approach is likely over-engineered. Before commit 4d5d14a01e2c ("mm/mglru: rework workingset protection"), folios with PG_active were always placed in the youngest generation, leading to over-protection and increased refaults. After that commit, PG_active folios are placed in the second youngest generation, which is still too optimistic given the presence of readahead. In contrast, the classic active/inactive scheme is more conservative. This patch switches to using folio_mark_accessed() and begins prefaulted file folios from the second oldest generation instead of active generations. We should also adjust the following accordingly: - WORKINGSET_ACTIVATE: aligned with setting active for refaulted workingset folios; - lru_gen_folio_seq(): place (pre)faulted file folios into the second oldest generation; - promote second-scanned folios to workingset in folio_check_references(): we now have to depend on folio_lru_refs() > 1, since we previously relied on PG_referenced being set during the first scan, but PG_referenced is now set earlier. On x86, running a kernel build inside a memcg with a 1GB memory limit using 20 threads. w/o patch: real 1m50.764s user 25m32.305s sys 4m0.012s pswpin: 1333245 pswpout: 4366443 pgpgin: 6962592 pgpgout: 17780712 swpout_zero: 1019603 swpin_zero: 14764 refault_file: 287794 refault_anon: 1347963 w/ patch: real 1m48.879s user 25m29.224s sys 3m37.421s pswpin: 568480 pswpout: 2322657 pgpgin: 4073416 pgpgout: 9613408 swpout_zero: 593275 swpin_zero: 9118 refault_file: 262505 refault_anon: 577550 active/inactive LRU: real 1m49.928s user 25m28.196s sys 3m40.740s pswpin: 463452 pswpout: 2309119 pgpgin: 4438856 pgpgout: 9568628 swpout_zero: 743704 swpin_zero: 7244 refault_file: 562555 refault_anon: 470694 Lance and Xueyuan made a huge contribution to this patch through testing. Link: https://lore.kernel.org/20260526130938.66253-1-baohua@kernel.org Link: https://lore.kernel.org/linux-mm/20250916072226.220426-1-liulei.rjpt@vivo.com/ [1] Signed-off-by: Barry Song (Xiaomi) Tested-by: Lance Yang Tested-by: Xueyuan Chen Cc: Pedro Falcato Cc: Kairui Song Cc: Qi Zheng Cc: Shakeel Butt Cc: wangzicheng Cc: Suren Baghdasaryan Cc: Lei Liu Cc: Matthew Wilcox (Oracle) Cc: Axel Rasmussen Cc: Yuanchu Xie Cc: Wei Xu Cc: Will Deacon Cc: Kalesh Singh Signed-off-by: Andrew Morton

mm/lruvec: preemptively free dead folios during lru_add drain

2026-05-29T04:04:52+00:00

Of all observable lruvec lock contention in our fleet, we find that ~24% occurs when dead folios are present in lru_add batches at drain time. This is wasteful in the sense that the folio is added to the LRU just to be immediately removed via folios_put_refs(), incurring two unnecessary lock acquisitions. Eliminate this overhead by preemptively cleaning up dead folios before they make it into the LRU. Use folio_ref_freeze() to filter folios whose only remaining refcount is the batch ref. When dead folios are found, move them off the add batch and onto a temporary batch to be freed. PG_active may be set on a batched folio as well as PG_unevictable (via migration path). Since filtered folios bypass the normal lru_add() cleanup, both flags must be cleared before freeing. During A/B testing on one of our prod instagram workloads (high-frequency short-lived requests), the patch intercepted almost all dead folios before they entered the LRU. Data collected using the mm_lru_insertion tracepoint shows the effectiveness of the patch: Per-host LRU add averages at 95% CPU load (60 hosts each side, 3 x 60s intervals) dead folios/min total folios/min dead % unpatched: 1,297,785 19,341,986 6.7097% patched: 14 19,039,996 0.0001% Within this workload, we save ~2.6M lock acquisitions per minute per host as a result. System-wide memory stats improved on the patched side also at 95% CPU load: - direct reclaim scanning reduced 7% - allocation stalls reduced 5.2% - compaction stalls reduced 12.3% - page frees reduced 4.9% No regressions were observed in requests served per second or request tail latency (p99). Both metrics showed directional improvement at higher CPU utilization (comparing 85% to 95%). Note that tests were performed using classic LRU. Link: https://lore.kernel.org/20260425053417.351146-1-jp.kobryn@linux.dev Signed-off-by: JP Kobryn (Meta) Reviewed-by: Matthew Wilcox (Oracle) Acked-by: Shakeel Butt Acked-by: Michal Hocko Cc: Axel Rasmussen Cc: Baoquan He Cc: Barry Song Cc: Chris Li Cc: Johannes Weiner Cc: Kairui Song Cc: Kemeng Shi Cc: Nhat Pham Cc: Rik van Riel Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton

mm: vmscan: prepare for reparenting traditional LRU folios

2026-04-18T07:10:46+00:00

To resolve the dying memcg issue, we need to reparent LRU folios of child memcg to its parent memcg. For traditional LRU list, each lruvec of every memcg comprises four LRU lists. Due to the symmetry of the LRU lists, it is feasible to transfer the LRU lists from a memcg to its parent memcg during the reparenting process. This commit implements the specific function, which will be used during the reparenting process. Link: https://lore.kernel.org/a92d217a9fc82bd0c401210204a095caaf615b1c.1772711148.git.zhengqi.arch@bytedance.com Signed-off-by: Qi Zheng Reviewed-by: Harry Yoo Acked-by: Johannes Weiner Acked-by: Muchun Song Acked-by: Shakeel Butt Cc: Allen Pais Cc: Axel Rasmussen Cc: Baoquan He Cc: Chengming Zhou Cc: Chen Ridong Cc: David Hildenbrand Cc: Hamza Mahfooz Cc: Hugh Dickins Cc: Imran Khan Cc: Kamalesh Babulal Cc: Lance Yang Cc: Liam Howlett Cc: Lorenzo Stoakes (Oracle) Cc: Michal Hocko Cc: Michal Koutný Cc: Mike Rapoport Cc: Muchun Song Cc: Nhat Pham Cc: Roman Gushchin Cc: Suren Baghdasaryan Cc: Usama Arif Cc: Vlastimil Babka Cc: Wei Xu Cc: Yosry Ahmed Cc: Yuanchu Xie Cc: Zi Yan Signed-off-by: Andrew Morton

mm: memcontrol: prepare for reparenting LRU pages for lruvec lock

2026-04-18T07:10:46+00:00

The following diagram illustrates how to ensure the safety of the folio lruvec lock when LRU folios undergo reparenting. In the folio_lruvec_lock(folio) function: rcu_read_lock(); retry: lruvec = folio_lruvec(folio); /* There is a possibility of folio reparenting at this point. */ spin_lock(&lruvec->lru_lock); if (unlikely(lruvec_memcg(lruvec) != folio_memcg(folio))) { /* * The wrong lruvec lock was acquired, and a retry is required. * This is because the folio resides on the parent memcg lruvec * list. */ spin_unlock(&lruvec->lru_lock); goto retry; } /* Reaching here indicates that folio_memcg() is stable. */ In the memcg_reparent_objcgs(memcg) function: spin_lock(&lruvec->lru_lock); spin_lock(&lruvec_parent->lru_lock); /* Transfer folios from the lruvec list to the parent's. */ spin_unlock(&lruvec_parent->lru_lock); spin_unlock(&lruvec->lru_lock); After acquiring the lruvec lock, it is necessary to verify whether the folio has been reparented. If reparenting has occurred, the new lruvec lock must be reacquired. During the LRU folio reparenting process, the lruvec lock will also be acquired (this will be implemented in a subsequent patch). Therefore, folio_memcg() remains unchanged while the lruvec lock is held. Given that lruvec_memcg(lruvec) is always equal to folio_memcg(folio) after the lruvec lock is acquired, the lruvec_memcg_debug() check is redundant. Hence, it is removed. This patch serves as a preparation for the reparenting of LRU folios. Link: https://lore.kernel.org/23f22cbb1419f277a3483018b32158ae2b86c666.1772711148.git.zhengqi.arch@bytedance.com Signed-off-by: Muchun Song Signed-off-by: Qi Zheng Acked-by: Johannes Weiner Acked-by: Shakeel Butt Cc: Allen Pais Cc: Axel Rasmussen Cc: Baoquan He Cc: Chengming Zhou Cc: Chen Ridong Cc: David Hildenbrand Cc: Hamza Mahfooz Cc: Harry Yoo Cc: Hugh Dickins Cc: Imran Khan Cc: Kamalesh Babulal Cc: Lance Yang Cc: Liam Howlett Cc: Lorenzo Stoakes (Oracle) Cc: Michal Hocko Cc: Michal Koutný Cc: Mike Rapoport Cc: Muchun Song Cc: Nhat Pham Cc: Roman Gushchin Cc: Suren Baghdasaryan Cc: Usama Arif Cc: Vlastimil Babka Cc: Wei Xu Cc: Yosry Ahmed Cc: Yuanchu Xie Cc: Zi Yan Signed-off-by: Andrew Morton

mm: swap: prevent lruvec release in lru_gen_clear_refs()

2026-04-18T07:10:46+00:00

In the near future, a folio will no longer pin its corresponding memory cgroup. So an lruvec returned by folio_lruvec() could be released without the rcu read lock or a reference to its memory cgroup. In the current patch, the rcu read lock is employed to safeguard against the release of the lruvec in lru_gen_clear_refs(). This serves as a preparatory measure for the reparenting of the LRU pages. Link: https://lore.kernel.org/986cd26227191a48a7c34a2a15812d361f4ebd53.1772711148.git.zhengqi.arch@bytedance.com Signed-off-by: Muchun Song Signed-off-by: Qi Zheng Reviewed-by: Harry Yoo Acked-by: Johannes Weiner Acked-by: Shakeel Butt Cc: Allen Pais Cc: Axel Rasmussen Cc: Baoquan He Cc: Chengming Zhou Cc: Chen Ridong Cc: David Hildenbrand Cc: Hamza Mahfooz Cc: Hugh Dickins Cc: Imran Khan Cc: Kamalesh Babulal Cc: Lance Yang Cc: Liam Howlett Cc: Lorenzo Stoakes (Oracle) Cc: Michal Hocko Cc: Michal Koutný Cc: Mike Rapoport Cc: Muchun Song Cc: Nhat Pham Cc: Roman Gushchin Cc: Suren Baghdasaryan Cc: Usama Arif Cc: Vlastimil Babka Cc: Wei Xu Cc: Yosry Ahmed Cc: Yuanchu Xie Cc: Zi Yan Signed-off-by: Andrew Morton