kernel/git/next/linux-next.git/mm/vmalloc.c, branch master

mm: vmalloc: fix vmap_purge_lock livelock under memory pressure

2026-09-09T06:30:23+00:00

The vmap_purge_lock mutex can be held for an extended period by __purge_vmap_area_lazy() which calls flush_work() to wait for purge_vmap_node workers while holding the lock. Under memory pressure, those workers may themselves be blocked in direct reclaim trying to acquire the same lock via the vmap_node_shrink_scan() shrinker callback, creating a circular dependency that deadlocks the entire system. Two places acquire vmap_purge_lock from paths that can be reached during direct reclaim: 1. vmap_node_shrink_scan(): replace blocking guard(mutex) with mutex_trylock(). This is a shrinker that only decays the vmap pool and returns SHRINK_STOP without freeing memory; skipping a decay cycle when the lock is contended is harmless and prevents tasks from piling up on the mutex in the direct reclaim path. 2. reclaim_and_purge_vmap_areas(): replace mutex_lock() with mutex_trylock(). This is called from the vmalloc allocation overflow path; if trylock fails, another thread is already purging and the allocator's retry will find freed space. The notifier chain provides a fallback if the retry still fails. Both trylock failures break the circular dependency: the lock holder's flush_work() can complete because workers are no longer blocked on vmap_purge_lock in the direct reclaim path. Link: https://lore.kernel.org/20260828091753.299295-1-ye.liu@linux.dev Fixes: 7679ba6b36db ("mm: vmalloc: add a shrinker to drain vmap pools") Signed-off-by: Ye Liu Suggested-by: Uladzislau Rezki Suggested-by: Dev Jain Reviewed-by: Uladzislau Rezki (Sony) Signed-off-by: Andrew Morton

mm/vmalloc: make set_area_direct_map HUGE_VMAP friendly

2026-09-09T06:30:21+00:00

set_area_direct_map() always updates direct map alias permissions in single page increments. For HUGE_VMAP areas it's suboptimal. Not only the loop in set_area_direct_map() needlessly has more iterations (e.g times 512 on x86), but it also causes fragmentation of the direct map that could be avoided for the HUGE_VMAP areas populated with large pages. All pages in an area are always of the same order: either same-order large pages when VM_ALLOW_HUGE_VMAP is set and all huge pages were successfully allocated, or order-0 page when VM_ALLOW_HUGE_VMAP is cleared or when huge pages allocation fails and fallback path is taken. Instead of updating the direct map permissions for every order-0 page in an area, use the area's page_order as the loop increment and update the large pages in one call to set_direct_map_{invalid,default}_noflush(). Link: https://lore.kernel.org/20260903-execmem-set-vm-perms-v0-2-v3-4-949b64a9f755@kernel.org Signed-off-by: Mike Rapoport (Microsoft) Reviewed-by: Dev Jain Cc: Albert Ou Cc: Alexander Gordeev Cc: Alexandre Ghiti Cc: Andy Lutomirski Cc: "Borislav Petkov (AMD)" Cc: Brendan Jackman Cc: Catalin Marinas Cc: Christian Borntraeger Cc: Dave Hansen Cc: David Hildenbrand Cc: Gerald Schaefer Cc: Heiko Carstens Cc: "H. Peter Anvin" Cc: Huacai Chen Cc: Ingo Molnar Cc: Len Brown Cc: Palmer Dabbelt Cc: Peter Zijlstra Cc: "Rafael J. Wysocki" Cc: Ryan Roberts Cc: Sven Schnelle Cc: "Uladzislau Rezki (Sony)" Cc: Vasily Gorbik Cc: WANG Xuerui Cc: Will Deacon Signed-off-by: Andrew Morton

mm/vmalloc: constify vm parameter of get_vm_area_page_order()

2026-09-09T06:30:21+00:00

get_vm_area_page_order() and vm_area_page_order() do not need to modify struct vm_struct passed to them. Constify the parameter. Link: https://lore.kernel.org/20260903-execmem-set-vm-perms-v0-2-v3-3-949b64a9f755@kernel.org Signed-off-by: Mike Rapoport (Microsoft) Reviewed-by: Uladzislau Rezki (Sony) Reviewed-by: Dev Jain Reviewed-by: David Hildenbrand (Arm) Cc: Albert Ou Cc: Alexander Gordeev Cc: Alexandre Ghiti Cc: Andy Lutomirski Cc: "Borislav Petkov (AMD)" Cc: Brendan Jackman Cc: Catalin Marinas Cc: Christian Borntraeger Cc: Dave Hansen Cc: Gerald Schaefer Cc: Heiko Carstens Cc: "H. Peter Anvin" Cc: Huacai Chen Cc: Ingo Molnar Cc: Len Brown Cc: Palmer Dabbelt Cc: Peter Zijlstra Cc: "Rafael J. Wysocki" Cc: Ryan Roberts Cc: Sven Schnelle Cc: Vasily Gorbik Cc: WANG Xuerui Cc: Will Deacon Signed-off-by: Andrew Morton

mm/vmalloc: set area's page_order after allocation succeeds

2026-09-09T06:30:21+00:00

__vmalloc_area_node() calls set_vm_area_page_order() to set area's page_order before actually allocating pages to populate the area. If allocation of large pages in HUGE_VMAP case fails midway, this leaves the area with elevated page_order throughout the cleanup path. There is no actual issue with this because the only place that currently relies on area->page_order on the cleanup path is the loop calculating the direct map alias range in vm_reset_perms() and it anyway skips unpopulated pages. But having set_vm_area_page_order() in the middle of __vmalloc_area_node() makes things very obscure, hard to reason about and error prone against future changes of the cleanup path. Move the call to set_vm_area_page_order() just before the successful return from __vmalloc_area_node() where page order is guaranteed. While on it, initialize local page_order variable with its declaration. Link: https://lore.kernel.org/20260903-execmem-set-vm-perms-v0-2-v3-2-949b64a9f755@kernel.org Signed-off-by: Mike Rapoport (Microsoft) Reviewed-by: Uladzislau Rezki (Sony) Reviewed-by: Dev Jain Cc: Albert Ou Cc: Alexander Gordeev Cc: Alexandre Ghiti Cc: Andy Lutomirski Cc: "Borislav Petkov (AMD)" Cc: Brendan Jackman Cc: Catalin Marinas Cc: Christian Borntraeger Cc: Dave Hansen Cc: David Hildenbrand Cc: Gerald Schaefer Cc: Heiko Carstens Cc: "H. Peter Anvin" Cc: Huacai Chen Cc: Ingo Molnar Cc: Len Brown Cc: Palmer Dabbelt Cc: Peter Zijlstra Cc: "Rafael J. Wysocki" Cc: Ryan Roberts Cc: Sven Schnelle Cc: Vasily Gorbik Cc: WANG Xuerui Cc: Will Deacon Signed-off-by: Andrew Morton

set_memory: add number of pages parameter to set_direct_map APIs

2026-09-09T06:30:20+00:00

Patch series "arch, mm/execmem: resolve confusion about set_direct_map_valid_noflush()", v3. Recent discussion about implementation of execmem's ROX caches on arm64 revealed a confusion about how set_direct_map_valid_noflush() implemented on different architectures. On arm64 it sets or clears the PTE_VALID bit marking a PTE as present or not present. On other architectures it's a range version of set_direct_map_invalid_noflush() and set_direct_map_default_noflush() Unlike arm64::set_direct_map_valid_noflush(), set_direct_map_default_noflush() not only marks PTE as present, but also sets its default protection mode. Other than that, initial design of execmem ROX caches didn't rely on restoration of large mappings that's now available on x86, but completely removed the memory allocated for the ROX cache from the direct map to ensure that large mappings are not split. This precluded usage of VM_FLUSH_RESET_PERMS for the ROX cache allocations and required execmem to implement manipulation of the direct map alias. Current implementation of ROX caches does not remove the direct map alias but simply calls set_memory_rox() that updates the permissions in both vmalloc address space and the direct map and relies on collapse_large_pages() in x86 CPA to keep large mappings. This allow using VM_FLUSH_RESET_PERMS for execmem ROX cache allocations with small adjustments to set_direct_map APIs and vmalloc::reset_perms() behaviour: adding number of pages parameter to set_direct_map APIs and making resetting of the direct map permissions in vmalloc VMAP_HUGE friendly. Implement these adjustments, make execmem always use VM_FLUSH_RESET_PERMS and revert set_direct_map_valid_noflush() changes. This patch (of 6): When set_direct_map APIs were introduced by the commit d253ca0c3865 ("x86/mm/cpa: Add set_direct_map_*() functions") the single page parameter made sense because the initial callers (vmalloc and hibernation) had sets of unsorted struct pages that required changes of their mappings in the direct map. Since there is an increasing demand for direct map manipulation and it is also desirable to be able to update larger physically contiguous mappings, for example an entire large folio, extend set_direct_map APIs to receive number of pages parameter. As there is still only a handful of callers, change the existing functions directly and update all the call sites rather than adding wrappers for single page case. Link: https://lore.kernel.org/20260903-execmem-set-vm-perms-v0-2-v3-0-949b64a9f755@kernel.org Link: https://lore.kernel.org/20260903-execmem-set-vm-perms-v0-2-v3-1-949b64a9f755@kernel.org Link: https://lore.kernel.org/all/20260611130144.1385343-4-abarnas@google.com [1] Signed-off-by: Mike Rapoport (Microsoft) Acked-by: David Hildenbrand (Arm) Cc: Albert Ou Cc: Alexander Gordeev Cc: Alexandre Ghiti Cc: Andy Lutomirski Cc: "Borislav Petkov (AMD)" Cc: Brendan Jackman Cc: Catalin Marinas Cc: Christian Borntraeger Cc: Dave Hansen Cc: Gerald Schaefer Cc: Heiko Carstens Cc: "H. Peter Anvin" Cc: Huacai Chen Cc: Ingo Molnar Cc: Len Brown Cc: Palmer Dabbelt Cc: Peter Zijlstra Cc: "Rafael J. Wysocki" Cc: Ryan Roberts Cc: Sven Schnelle Cc: "Uladzislau Rezki (Sony)" Cc: Vasily Gorbik Cc: WANG Xuerui Cc: Will Deacon Cc: Dev Jain Signed-off-by: Andrew Morton

mm/vmalloc: avoid false sharing with drain_vmap_work

2026-09-09T06:30:09+00:00

free_vmap_area_noflush() queues drain_vmap_work after the number of lazily freed pages exceeds lazy_max_pages(). Until the worker purges those pages, concurrent frees keep calling schedule_work(). Even if the work is already pending, queue_work_on() performs a locked test_and_set_bit() on the pending bit in the work item. On the tested x86-64 build, drain_vmap_work and vmap_nodes occupy the same 64-byte cache line. The work item starts at offset 0 and the vmap_nodes pointer at offset 32. The latter is read by vmap allocation and free paths, so updates to the work item invalidate a cache line read by all CPUs. Put drain_vmap_work in the cacheline-aligned data section. Tests were run on Linux 7.2. On a two-socket Intel Xeon Silver 4208 system using 16 workers, the runtimes of vmalloc.fix_align, vmalloc.fix_size, and vmalloc.no_block_alloc decreased by 12.61%, 5.78%, and 6.87%, respectively. HITM samples for the affected cache line and total HITM samples decreased by 96.55% and 13.36%, respectively. Link: https://lore.kernel.org/20260825104659.100134-1-jonaszhou-oc@zhaoxin.com Signed-off-by: JonasZhou Reviewed-by: Uladzislau Rezki (Sony) Cc: Signed-off-by: Andrew Morton

mm/vmalloc: ase dedicated unbound workqueues for vmap drain

2026-09-09T06:29:34+00:00

drain_vmap_area_work() function can take >10ms to complete when there are many accumulated vmap areas in a system with high CPU count, causing workqueue watchdog warnings when run via schedule_work(): workqueue: drain_vmap_area_work hogged CPU for >10000us Move the top-level drain work to a dedicated WQ_UNBOUND workqueue so the scheduler can run this background work on any available CPU, improving responsiveness. Use the WQ_MEM_RECLAIM to ensure forward progress under memory pressure. Move purge helpers to separate WQ_UNBOUND | WQ_MEM_RECLAIM workqueue. This allows drain_vmap_work to wait for helpers completion without creating dependency on the same rescuer thread and avoid a potential parent/child deadlock. Simplify purge helper scheduling by removing cpumask-based iteration to iterating directly over vmap nodes checking work_queued state. Link: https://lore.kernel.org/20260905152717.11711-1-urezki@gmail.com Fixes: 72210662c5a2 ("mm: vmalloc: offload free_vmap_area_lock lock") Signed-off-by: Uladzislau Rezki (Sony) Reported-by: Li RongQing Closes: https://lore.kernel.org/all/20260319074307.2325-1-lirongqing@baidu.com/ Reviewed-by: Baoquan He Cc: Dev Jain Cc: Ye Liu Cc: Signed-off-by: Andrew Morton

mm/vmalloc: do not warn on -ENOMEM from va_alloc()

2026-08-25T01:43:02+00:00

Since vmalloc() accepts non-blocking GFP flags, allocation requests may fail when callers pass restrictive GFP masks. va_clip() may return -ENOMEM when its GFP_NOWAIT fallback allocation fails during NE_FIT_TYPE splitting. This is an expected failure, so va_alloc() should return the error without triggering a kernel splat. Link: https://lore.kernel.org/20260802104627.63892-1-urezki@gmail.com Signed-off-by: Uladzislau Rezki (Sony) Reported-by: syzbot+61c997e6be1d9bb300ba@syzkaller.appspotmail.com Closes: https://lore.kernel.org/6a6d3cbd.6ce73036.24301b.000e.GAE@google.com Reviewed-by: Anshuman Khandual Reviewed-by: Baoquan He Signed-off-by: Andrew Morton

mm/vmalloc: make vm_struct.nr_pages an unsigned long

2026-08-25T01:43:00+00:00

vm_struct::nr_pages is an unsigned int, and the file keeps deriving byte counts from it as nr_pages << PAGE_SHIFT. A shift is evaluated in the type of its promoted left operand, so those are 32-bit arithmetic and wrap at 4 GiB of bytes, which is 2^20 pages. Every site depends on a cast being remembered; vmap() has one, two recent commits did not. vread_iter() then computes a size of zero for a 4 GiB VM_ALLOC area and /proc/kcore returns it as zeros while reporting a successful read, which drgn, crash or gdb cannot tell from real memory, and the vrealloc() grow-in-place check declines a request that would have fit. Widen the field so the class of bug goes away instead of one site at a time. Everything feeding or consuming it widens too: vm_area_alloc_pages() and its accumulators, nr_small_pages, new_nr_pages and old_nr_pages, the index range of vm_area_free_pages(), and three page indexes that were plain int. Five casts go. Two prints needed fixing as well, %u in vmalloc_dump_obj() and %d for the unsigned field in vmalloc_info_show(). No bug report behind this, I found it reading the code. The 4 GiB wrap needs only a machine with over 4 GiB of memory. Neither larger threshold is a practical concern: 2^32 pages, where the field itself truncates, is 16 TiB and beyond what hardware can populate, and 2^31, where the plain int indexes break, is 8 TiB and larger than anything in the tree asks for. The int *nr cursor in the mapping path is unchanged and is separate work. Users outside mm/vmalloc.c need no change either. Those handing the count to a narrower parameter cannot drive it near 2^31, and kho_preserve_vmalloc() stores it into a 32-bit ABI field that still receives the same low bits; above 2^32 pages the truncation just moves out of vm_struct into that store. sizeof(struct vm_struct) on x86-64 stays 72 bytes with CONFIG_HAVE_ARCH_HUGE_VMALLOC=n and goes from 72 to 80 with it enabled, both inside the kmalloc-96 bucket it already comes from. Link: https://lore.kernel.org/20260801114915.115224-1-iprintercanon@gmail.com Fixes: 0bca23804632 ("mm/vmalloc: use physical page count in vread_iter() for VM_ALLOC areas") Fixes: d57ac904ffdc ("mm/vmalloc: use physical page count for vrealloc() grow-in-place check") Signed-off-by: Artem Lytkin Suggested-by: Andrew Morton Reviewed-by: Uladzislau Rezki (Sony) Assisted-by: Claude:claude-fable-5 Cc: Matthew Wilcox (Oracle) Cc: Cc: Signed-off-by: Andrew Morton

Merge branch 'mm-hotfixes-stable' into mm-stable to pick up

2026-08-25T01:40:27+00:00

already-upstream changes to memcontrol.c, needed by "memcg: move mem_cgroup_swappiness and vm_swappiness to mm/swap.h".