kernel/git/stable/linux.git/mm/zswap.c, branch linux-rolling-lts

mm/zswap: fix global shrinker when memory cgroup is disabled

2026-09-07T15:22:22+00:00

commit dc8458f43fe964d8ade74c9b0fce54fe71d156de upstream. Patch series "mm/zswap: Fixes and improves the zswap shrink", v4. This series fixes and improves the zswap global shrinker (shrink_worker()): Patch 1: Fix missing global shrinker when memory cgroup is disabled. Patch 2: Extend shrink_memcg() to support batch writeback and thereby improving the writeback efficiency in the shrink_worker() and zswap_store() paths. This patch (of 2): Zswap writeback when the global pool limit is hit fails when memory cgroup is disabled. The pool remains full until it is organically drained by swapins or memory freeing, leading to zswap store failures and pages bypassing getting written directly to the backing swap device, causing LRU inversion (hotter pages with higher fault latency). This happens because mem_cgroup_iter() always returns NULL when memory cgroups are disabled. As a result, the global shrinker shrink_worker() repeatedly takes empty walks. After MAX_RECLAIM_RETRIES failed attempts, the worker gives up without writing back any pages. Therefore, when memory cgroup is disabled, fall through with the !memcg branch and shrink the root memcg directly. With memcg disabled, shrink_memcg() only returns -ENOENT when the root LRU is empty, which means the total pages are already below thr. In the absence of heavy concurrent zswap stores, the loop then safely bails out via the zswap_total_pages() <= thr check; otherwise, it will resume shrinking the memcg after processing the reschedule check. For any other return value from shrink_memcg(), the loop is guaranteed to terminate, either after MAX_RECLAIM_RETRIES failures or once the threshold is met. This is a potential performance regression for people using zswap without memcg that was introduced by the commit in "Fixes". Link: https://lore.kernel.org/20260806070943.95542-1-jiahao.kernel@gmail.com Link: https://lore.kernel.org/20260806070943.95542-2-jiahao.kernel@gmail.com Fixes: a65b0e7607cc ("zswap: make shrinking memcg-aware") Signed-off-by: Hao Jia Suggested-by: Nhat Pham Acked-by: Nhat Pham Acked-by: Yosry Ahmed Reported-by: Yosry Ahmed Cc: Chengming Zhou Cc: Johannes Weiner Cc: Michal Hocko Cc: Michal Koutný Cc: Muchun Song Cc: Roman Gushchin Cc: Shakeel Butt Cc: Tejun Heo Cc: Signed-off-by: Andrew Morton Signed-off-by: Greg Kroah-Hartman

mm/zswap: fix error pointer free in zswap_cpu_comp_prepare()

2026-01-23T10:21:29+00:00

commit 590b13669b813d55844fecd9142c56abd567914d upstream. crypto_alloc_acomp_node() may return ERR_PTR(), but the fail path checks only for NULL and can pass an error pointer to crypto_free_acomp(). Use IS_ERR_OR_NULL() to only free valid acomp instances. Link: https://lkml.kernel.org/r/20251231074638.2564302-1-pbutsykin@cloudlinux.com Fixes: 779b9955f643 ("mm: zswap: move allocations during CPU init outside the lock") Signed-off-by: Pavel Butsykin Reviewed-by: SeongJae Park Acked-by: Yosry Ahmed Acked-by: Nhat Pham Cc: Johannes Weiner Cc: Chengming Zhou Cc: Signed-off-by: Andrew Morton Signed-off-by: Greg Kroah-Hartman

mm, swap: remove contention workaround for swap cache

2025-09-21T21:22:25+00:00

Swap cluster setup will try to shuffle the clusters on initialization. It was helpful to avoid contention for the swap cache space. The cluster size (2M) was much smaller than each swap cache space (64M), so shuffling the cluster means the allocator will try to allocate swap slots that are in different swap cache spaces for each CPU, reducing the chance of two CPUs using the same swap cache space, and hence reducing the contention. Now, swap cache is managed by swap clusters, this shuffle is pointless. Just remove it, and clean up related macros. This also improves the HDD swap performance as shuffling IO is a bad idea for HDD, and now the shuffling is gone. Test have shown a ~40% performance gain for HDD [1]: Doing sequential swap in of 8G data using 8 processes with usemem, average of 3 test runs: Before: 1270.91 KB/s per process After: 1849.54 KB/s per process Link: https://lore.kernel.org/linux-mm/CAMgjq7AdauQ8=X0zeih2r21QoV=-WWj1hyBxLWRzq74n-C=-Ng@mail.gmail.com/ [1] Link: https://lkml.kernel.org/r/20250916160100.31545-14-ryncsn@gmail.com Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-lkp/202504241621.f27743ec-lkp@intel.com Signed-off-by: Kairui Song Acked-by: Chris Li Reviewed-by: Barry Song Acked-by: David Hildenbrand Suggested-by: Chris Li Cc: Baolin Wang Cc: Baoquan He Cc: "Huang, Ying" Cc: Hugh Dickins Cc: Johannes Weiner Cc: Kemeng Shi Cc: Lorenzo Stoakes Cc: Matthew Wilcox (Oracle) Cc: Nhat Pham Cc: Yosry Ahmed Cc: Zi Yan Cc: SeongJae Park Signed-off-by: Andrew Morton

mm, swap: cleanup swap cache API and add kerneldoc

2025-09-21T21:22:23+00:00

In preparation for replacing the swap cache backend with the swap table, clean up and add proper kernel doc for all swap cache APIs. Now all swap cache APIs are well-defined with consistent names. No feature change, only renaming and documenting. Link: https://lkml.kernel.org/r/20250916160100.31545-9-ryncsn@gmail.com Signed-off-by: Kairui Song Acked-by: Chris Li Reviewed-by: Barry Song Reviewed-by: Baolin Wang Acked-by: David Hildenbrand Suggested-by: Chris Li Cc: Baoquan He Cc: "Huang, Ying" Cc: Hugh Dickins Cc: Johannes Weiner Cc: Kemeng Shi Cc: kernel test robot Cc: Lorenzo Stoakes Cc: Matthew Wilcox (Oracle) Cc: Nhat Pham Cc: Yosry Ahmed Cc: Zi Yan Cc: SeongJae Park Signed-off-by: Andrew Morton

mm: zswap: interact directly with zsmalloc

2025-09-21T21:21:58+00:00

Patch series "mm: remove zpool". zpool is an indirection layer for zswap to switch between multiple allocator backends at runtime. Since 6.15, zsmalloc is the only allocator left in-tree, so there is no point in keeping zpool around. This patch (of 3): zswap goes through the zpool layer to enable runtime-switching of allocator backends for compressed data. However, since zbud and z3fold were removed in 6.15, zsmalloc has been the only option available. As such, the zpool indirection is unnecessary. Make zswap deal with zsmalloc directly. This is comparable to zram, which also directly interacts with zsmalloc and has never supported a different backend. Note that this does not preclude future improvements and experiments with different allocation strategies. Should it become necessary, it's possible to provide an alternate implementation for the zsmalloc API, selectable at compile time. However, zsmalloc is also rather mature and feature rich, with years of widespread production exposure; it's encouraged to make incremental improvements rather than fork it. In any case, the complexity of runtime pluggability seems excessive and unjustified at this time. Switch zswap to zsmalloc to remove the last user of the zpool API. [hannes@cmpxchg.org: fix default compressr test] Link: https://lkml.kernel.org/r/20250915153640.GA828739@cmpxchg.org Link: https://lkml.kernel.org/r/20250829162212.208258-1-hannes@cmpxchg.org Link: https://lkml.kernel.org/r/20250829162212.208258-2-hannes@cmpxchg.org Signed-off-by: Johannes Weiner Nacked-by: Vitaly Wool Acked-by: Nhat Pham Acked-by: Yosry Ahmed Cc: Chengming Zhou Cc: SeongJae Park Signed-off-by: Andrew Morton

mm/zswap: reduce the size of the compression buffer to a single page

2025-09-13T23:55:15+00:00

Reduce the compression buffer size from 2 * PAGE_SIZE to only one page, as the compression output (in the success case) should not exceed the length of the input. In the past, Chengming tried to reduce the compression buffer size, but ran into issues with the LZO algorithm (see [2]). Herbert Xu reported that the issue has been fixed (see [3]). Now we should have the guarantee that compressors' output should not exceed one page in the success case, and the algorithm will just report failure otherwise. With this patch, we save one page per cpu (per compression algorithm). Link: https://lkml.kernel.org/r/20250820181547.3794167-1-nphamcs@gmail.com Link: https://lore.kernel.org/linux-mm/20231213-zswap-dstmem-v4-1-f228b059dd89@bytedance.com/ [1] Link: https://lore.kernel.org/lkml/0000000000000b05cd060d6b5511@google.com/ [2] Link: https://lore.kernel.org/linux-mm/aKUmyl5gUFCdXGn-@gondor.apana.org.au/ [3] Co-developed-by: Chengming Zhou Signed-off-by: Chengming Zhou Signed-off-by: Nhat Pham Acked-by: SeongJae Park Reviewed-by: Chengming Zhou Cc: Herbert Xu Cc: Johannes Weiner Signed-off-by: Andrew Morton

mm/zswap: store
2025-09-13T23:54:59+00:00
When zswap writeback is enabled and it fails compressing a given page, the page is swapped out to the backing swap device. This behavior breaks the zswap's writeback LRU order, and hence users can experience unexpected latency spikes. If the page is compressed without failure, but results in a size of PAGE_SIZE, the LRU order is kept, but the decompression overhead for loading the page back on the later access is unnecessary. Keep the LRU order and optimize unnecessary decompression overheads in those cases, by storing the original content as-is in zpool. The length field of zswap_entry will be set appropriately, as PAGE_SIZE. Hence whether it is saved as-is or not (whether decompression is unnecessary) is identified by 'zswap_entry->length == PAGE_SIZE'. Because the uncompressed data is saved in zpool, same to the compressed ones, this introduces no change in terms of memory management including movability and migratability of involved pages. This change is also not increasing per zswap entry metadata overhead. But as the number of incompressible pages increases, total zswap metadata overhead is proportionally increased. The overhead should not be problematic in usual cases, since the zswap metadata for single zswap entry is much smaller than PAGE_SIZE, and in common zswap use cases there should be a sufficient amount of compressible pages. Also it can be mitigated by the zswap writeback. When the writeback is disabled, the additional overhead could be problematic. For the case, keep the current behavior that just returns the failure and let swap_writeout() put the page back to the active LRU list in the case. Knowing how many incompressible pages are stored at the given moment will be useful for future investigations. Add a new debugfs file called stored_incompressible_pages for the purpose. Tests ----- I tested this patch using a simple self-written microbenchmark that is available at GitHub[1]. You can reproduce the test I did by executing run_tests.sh of the repo on your system. Note that the repo's documentation is not good as of this writing, so you may need to read and use the code. The basic test scenario is simple. Run a test program making artificial accesses to memory having artificial content under memory.high-set memory limit and measure how many accesses were made in a given time. The test program repeatedly and randomly access three anonymous memory regions. The regions are all 500 MiB size, and be accessed in the same probability. Two of those are filled up with a simple content that can easily be compressed, while the remaining one is filled up with a content that s read from /dev/urandom, which is easy to fail at compressing to a size smaller than PAGE_SIZE. The program runs for two minutes and prints out the number of accesses made every five seconds. The test script runs the program under below four configurations. - 0: memory.high is set to 2 GiB, zswap is disabled. - 1-1: memory.high is set to 1350 MiB, zswap is disabled. - 1-2: On 1-1, zswap is enabled without this patch. - 1-3: On 1-2, this patch is applied. For all zswap enabled cases, zswap shrinker is enabled. Configuration '0' is for showing the original memory performance. Configurations 1-1, 1-2 and 1-3 are for showing the performance of swap, zswap, and this patch under a level of memory pressure (~10% of working set). Configurations 0 and 1-1 are not the main focus of this patch, but I'm adding those since their results transparently show how far this microbenchmark test is from the real world. Because the per-5 seconds performance is not very reliable, I measured the average of that for the last one minute period of the test program run. I also measured a few vmstat counters including zswpin, zswpout, zswpwb, pswpin and pswpout during the test runs. The measurement results are as below. To save space, I show performance numbers that are normalized to that of the configuration '0' (no memory pressure). The averaged accesses per 5 seconds of configuration '0' was 36493417.75. config 0 1-1 1-2 1-3 perf_normalized 1.0000 0.0057 0.0235 0.0367 perf_stdev_ratio 0.0582 0.0652 0.0167 0.0346 zswpin 0 0 3548424 1999335 zswpout 0 0 3588817 2361689 zswpwb 0 0 10214 340270 pswpin 0 485806 772038 340967 pswpout 0 649543 144773 340270 'perf_normalized' is the performance metric, normalized to that of configuration '0' (no pressure). 'perf_stdev_ratio' is the standard deviation of the averaged data points, as a ratio to the averaged metric value. For example, configuration '0' performance was showing 5.8% stdev. Configurations 1-1 and 1-3 were having about 6.5% and 6.1% stdev. Also the results were highly variable between multiple runs. So this result is not very stable but just showing ball park figures. Please keep this in your mind when reading these results. Under about 10% of working set memory pressure, the performance was dropped to about 0.57% of no-pressure one, when the normal swap is used (1-1). Note that ~10% working set pressure is already extreme, at least on this test setup. No one would desire system setups that can degrade performance to 0.57% of the best case. By turning zswap on (1-2), the performance was improved about 4x, resulting in about 2.35% of no-pressure one. Because of the incompressible pages in the third memory region, a significant amount of (non-zswap) swap I/O operations were made, though. By applying this patch (1-3), about 56% performance improvement was made, resulting in about 3.67% of no-pressure one. Reduced pswpin of 1-3 compared to 1-2 let us see where this improvement came from. Tests without Zswap Shrinker ---------------------------- Zswap shrinker is not enabled by default, so I ran the above test after disabling zswap shrinker. The results are as below. config 0 1-1 1-2 1-3 perf_normalized 1.0000 0.0056 0.0185 0.0260 perf_stdev_ratio 0.0467 0.0348 0.1832 0.3387 zswpin 0 0 2506765 6049078 zswpout 0 0 2534357 6115426 zswpwb 0 0 0 0 pswpin 0 463694 472978 0 pswpout 0 686227 612149 0 The overall normalized performance of the different configs are very similar to those of zswap shrinker enabled case. By adding the memory pressure, the performance was dropped to 0.56% of the original one. By enabling zswap without zswap shrinker, the performance was increased to 1.85% of the original one. By applying this patch on it, the performance was further increased to 2.6% of the original one. Even though zswap shrinker is disabled, 1-2 shows high numbers of pswpin and pswpout because the incompressible pages are directly swapped out. In the case of 1-3, it shows zero pswpin and pswpout since it saves incompressible pages in the memory, and shows higher performance. Note that the performance of 1-2 and 1-3 varies pretty much. Standard deviation of the performance for 1-2 was about 18.32% of the performance, while that for 1-3 was about 33.87%. Because zswap shrinker is disabled and the memory pressure is induced by memory.high, the workload got penalty_jiffies sleeps, and this resulted in the unstabilized performance. Related Works ------------- This is not an entirely new attempt. Nhat Pham and Takero Funaki tried very similar approaches in October 2023[2] and April 2024[3], respectively. The two approaches didn't get merged mainly due to the metadata overhead concern. I described why I think that shouldn't be a problem for this change, which is automatically disabled when writeback is disabled, at the beginning of this changelog. This patch is not particularly different from those, and actually built upon those. I wrote this from scratch again, though. Hence adding Suggested-by tags for them. Actually Nhat first suggested this to me offlist. Historically, writeback disabling was introduced partially as a way to solve the LRU order issue. Yosry pointed out[4] this is still suboptimal when the incompressible pages are cold, since the incompressible pages will continuously be tried to be zswapped out, and burn CPU cycles for compression attempts that will anyway fail. One imaginable solution for the problem is reusing the swapped-out page and its struct page to store in the zswap pool. But that's out of the scope of this patch. [sj@kernel.org: mark zswap_stored_incompressible_pages as static] Link: https://lkml.kernel.org/r/20250821161750.78192-1-sj@kernel.org [sj@kernel.org: v5] Link: https://lkml.kernel.org/r/20250822190817.49287-1-sj@kernel.org [sj@kernel.org: cleanup incompressible pages handling code] Link: https://lkml.kernel.org/r/20250828163913.57957-1-sj@kernel.org Link: https://lkml.kernel.org/r/20250819193404.46680-1-sj@kernel.org Link: https://github.com/sjp38/eval_zswap/blob/master/run.sh [1] Link: https://lore.kernel.org/20231017003519.1426574-3-nphamcs@gmail.com [2] Link: https://lore.kernel.org/20240706022523.1104080-6-flintglass@gmail.com [3] Link: https://lore.kernel.org/CAJD7tkZXS-UJVAFfvxJ0nNgTzWBiqepPYA4hEozi01_qktkitg@mail.gmail.com [4] Signed-off-by: SeongJae Park Suggested-by: Nhat Pham Suggested-by: Takero Funaki Acked-by: Nhat Pham Acked-by: Chris Li Cc: Chengming Zhou Cc: David Hildenbrand Cc: Johannes Weiner Cc: SeongJae Park Cc: Baoquan He Cc: Barry Song Cc: Kairui Song Cc: Herbert Xu Signed-off-by: Andrew Morton

mm: stop passing a writeback_control structure to __swap_writepage

2025-07-10T05:41:57+00:00

__swap_writepage only needs the swap_iocb cookie from the writeback_control structure, so pass it explicitly and remove the now unused swap_iocb member from struct writeback_control. Link: https://lkml.kernel.org/r/20250610054959.2057526-5-hch@lst.de Signed-off-by: Christoph Hellwig Acked-by: Nhat Pham Cc: Baolin Wang Cc: Chengming Zhou Cc: Hugh Dickins Cc: Johannes Weiner Cc: Matthew Wilcox (Oracle) Signed-off-by: Andrew Morton

zsmalloc: prefer the the original page's node for compressed data

2025-05-12T00:48:06+00:00

Currently, zsmalloc, zswap's and zram's backend memory allocator, does not enforce any policy for the allocation of memory for the compressed data, instead just adopting the memory policy of the task entering reclaim, or the default policy (prefer local node) if no such policy is specified. This can lead to several pathological behaviors in multi-node NUMA systems: 1. Systems with CXL-based memory tiering can encounter the following inversion with zswap/zram: the coldest pages demoted to the CXL tier can return to the high tier when they are reclaimed to compressed swap, creating memory pressure on the high tier. 2. Consider a direct reclaimer scanning nodes in order of allocation preference. If it ventures into remote nodes, the memory it compresses there should stay there. Trying to shift those contents over to the reclaiming thread's preferred node further *increases* its local pressure, and provoking more spills. The remote node is also the most likely to refault this data again. This undesirable behavior was pointed out by Johannes Weiner in [1]. 3. For zswap writeback, the zswap entries are organized in node-specific LRUs, based on the node placement of the original pages, allowing for targeted zswap writeback for specific nodes. However, the compressed data of a zswap entry can be placed on a different node from the LRU it is placed on. This means that reclaim targeted at one node might not free up memory used for zswap entries in that node, but instead reclaiming memory in a different node. All of these issues will be resolved if the compressed data go to the same node as the original page. This patch encourages this behavior by having zswap and zram pass the node of the original page to zsmalloc, and have zsmalloc prefer the specified node if we need to allocate new (zs)pages for the compressed data. Note that we are not strictly binding the allocation to the preferred node. We still allow the allocation to fall back to other nodes when the preferred node is full, or if we have zspages with slots available on a different node. This is OK, and still a strict improvement over the status quo: 1. On a system with demotion enabled, we will generally prefer demotions over compressed swapping, and only swap when pages have already gone to the lowest tier. This patch should achieve the desired effect for the most part. 2. If the preferred node is out of memory, letting the compressed data going to other nodes can be better than the alternative (OOMs, keeping cold memory unreclaimed, disk swapping, etc.). 3. If the allocation go to a separate node because we have a zspage with slots available, at least we're not creating extra immediate memory pressure (since the space is already allocated). 3. While there can be mixings, we generally reclaim pages in same-node batches, which encourage zspage grouping that is more likely to go to the right node. 4. A strict binding would require partitioning zsmalloc by node, which is more complicated, and more prone to regression, since it reduces the storage density of zsmalloc. We need to evaluate the tradeoff and benchmark carefully before adopting such an involved solution. [1]: https://lore.kernel.org/linux-mm/20250331165306.GC2110528@cmpxchg.org/ [senozhatsky@chromium.org: coding-style fixes] Link: https://lkml.kernel.org/r/mnvexa7kseswglcqbhlot4zg3b3la2ypv2rimdl5mh5glbmhvz@wi6bgqn47hge Link: https://lkml.kernel.org/r/20250402204416.3435994-1-nphamcs@gmail.com Signed-off-by: Nhat Pham Suggested-by: Gregory Price Acked-by: Dan Williams Reviewed-by: Chengming Zhou Acked-by: Sergey Senozhatsky [zram, zsmalloc] Acked-by: Johannes Weiner Acked-by: Yosry Ahmed [zswap/zsmalloc] Cc: "Huang, Ying" Cc: Joanthan Cameron Cc: Minchan Kim Cc: SeongJae Park Signed-off-by: Andrew Morton

mm: zswap: fix crypto_free_acomp() deadlock in zswap_cpu_comp_dead()

2025-04-01T22:14:43+00:00

Currently, zswap_cpu_comp_dead() calls crypto_free_acomp() while holding the per-CPU acomp_ctx mutex. crypto_free_acomp() then holds scomp_lock (through crypto_exit_scomp_ops_async()). On the other hand, crypto_alloc_acomp_node() holds the scomp_lock (through crypto_scomp_init_tfm()), and then allocates memory. If the allocation results in reclaim, we may attempt to hold the per-CPU acomp_ctx mutex. The above dependencies can cause an ABBA deadlock. For example in the following scenario: (1) Task A running on CPU #1: crypto_alloc_acomp_node() Holds scomp_lock Enters reclaim Reads per_cpu_ptr(pool->acomp_ctx, 1) (2) Task A is descheduled (3) CPU #1 goes offline zswap_cpu_comp_dead(CPU #1) Holds per_cpu_ptr(pool->acomp_ctx, 1)) Calls crypto_free_acomp() Waits for scomp_lock (4) Task A running on CPU #2: Waits for per_cpu_ptr(pool->acomp_ctx, 1) // Read on CPU #1 DEADLOCK Since there is no requirement to call crypto_free_acomp() with the per-CPU acomp_ctx mutex held in zswap_cpu_comp_dead(), move it after the mutex is unlocked. Also move the acomp_request_free() and kfree() calls for consistency and to avoid any potential sublte locking dependencies in the future. With this, only setting acomp_ctx fields to NULL occurs with the mutex held. This is similar to how zswap_cpu_comp_prepare() only initializes acomp_ctx fields with the mutex held, after performing all allocations before holding the mutex. Opportunistically, move the NULL check on acomp_ctx so that it takes place before the mutex dereference. Link: https://lkml.kernel.org/r/20250226185625.2672936-1-yosry.ahmed@linux.dev Fixes: 12dcb0ef5406 ("mm: zswap: properly synchronize freeing resources during CPU hotunplug") Signed-off-by: Herbert Xu Co-developed-by: Herbert Xu Signed-off-by: Yosry Ahmed Reported-by: syzbot+1a517ccfcbc6a7ab0f82@syzkaller.appspotmail.com Closes: https://lore.kernel.org/all/67bcea51.050a0220.bbfd1.0096.GAE@google.com/ Acked-by: Herbert Xu Reviewed-by: Chengming Zhou Reviewed-by: Nhat Pham Tested-by: Nhat Pham Cc: David S. Miller Cc: Eric Biggers Cc: Johannes Weiner Cc: Chris Murphy Cc: Signed-off-by: Andrew Morton