kernel/git/stable/linux.git/lib/vdso/datastore.c, branch master

vdso/datastore: Simplify the mapping logic for VDSO_TIME_PAGE_OFFSET

2026-07-07T21:52:53+00:00

The logic for CONFIG_GENERIC_GETTIMEOFDAY=n and !timens_page is identical now. Use this to simplify the logic a bit. Signed-off-by: Thomas Weißschuh Signed-off-by: Thomas Gleixner Link: https://patch.msgid.link/20260630-vdso-mlockall-v4-6-6c93708ce723@linutronix.de

vdso/datastore: Allow prefaulting by mlockall()

2026-07-07T21:52:52+00:00

While mlockall() is meant to lock page *memory*, effectively it will also create and lock the corresponding page table entries. Latency-sensitive applications expect not to experience any pagefaults after calling mlockall(). However mlockall() ignores VM_IO mappings, which is used by the generic vDSO datastore. While the fault handler itself is very fast, going through the full pagefault exception handling is much slower, on the order of 20us in a test machine. Since the memory behind the datastore mappings is always present and accessible it is not necessary to use VM_IO for them. The data page mapping is now also aligned with the architecture-specific code pages. Some architecture-specific data pages, like the x86 VCLOCK pages, continue to use VM_IO as they are not always mappable. They will require their own special handling later when the general approach has been agreed upon. As a side-effect this will allow GUP on these pages and allow more ways to access the data in them. This is fine, as all data in this mapping is globally visible anyways. Either because it is mapped into all tasks, or in the case of the time namespace pages, can be read from procfs. Regular mlock() would also work, but userspace does not know the boundaries of the vDSO. Reported-by: Nam Cao Signed-off-by: Thomas Weißschuh Signed-off-by: Thomas Gleixner Tested-by: Nam Cao Link: https://patch.msgid.link/20260630-vdso-mlockall-v4-5-6c93708ce723@linutronix.de

vdso/datastore: Explicitly prevent remote access to timens vvar page

2026-07-07T21:52:52+00:00

The fault handler for the timens page does not have access to the target task and therefore can not be invoked remotely. Currently the handler relies on the fact that the vvar mapping is marked as VM_IO and VM_PFNMAP for which the mm core always prevents remote access. However the VM_IO and VM_PFNMAP flags are going to be removed. Add an explicit check to prevent remote access to the mapping. Signed-off-by: Thomas Weißschuh Signed-off-by: Thomas Gleixner Tested-by: Nam Cao Link: https://patch.msgid.link/20260630-vdso-mlockall-v4-4-6c93708ce723@linutronix.de

vdso/datastore: Map zeroed pages for unavailable data

2026-07-07T21:52:52+00:00

mlockall() stops if a page in a VMA is unmappable. As the datastore VMA can contain holes, mlockall() would not process all data pages. Replace the mapping error VM_FAULT_SIGBUS by just mapping the underlying unused and zeroed-out data page. The vDSO will not access these pages in any case and for other userspace these pages have undefined contents. This will allow mlockall() to process all pages within the VMA as soon as VM_IO is removed from the VMA. Signed-off-by: Thomas Weißschuh Signed-off-by: Thomas Gleixner Tested-by: Nam Cao Link: https://patch.msgid.link/20260630-vdso-mlockall-v4-3-6c93708ce723@linutronix.de

vdso/datastore: Map pages in terms of the faults pgoff

2026-07-07T21:52:52+00:00

To support mlockall() on the datapages the VMA can have no holes where inner pages return VM_FAULT_SIGBUS. An upcoming change will avoid these holes by mapping a zeroed pages into these holes. That logic will be simpler when the mapping logic is based on vmf->pgoff instead of the vdso_k_ symbols. Switch to the equivalent vmf->pgoff logic. Signed-off-by: Thomas Weißschuh Signed-off-by: Thomas Gleixner Tested-by: Nam Cao Link: https://patch.msgid.link/20260630-vdso-mlockall-v4-2-6c93708ce723@linutronix.de

vdso/datastore: Rename data pages variable

2026-07-07T21:52:52+00:00

An upcoming change will make this a file-scoped variable, for which it should have a clearer name. Rename the variable to prepare for that. Signed-off-by: Thomas Weißschuh Signed-off-by: Thomas Gleixner Tested-by: Nam Cao Link: https://patch.msgid.link/20260630-vdso-mlockall-v4-1-6c93708ce723@linutronix.de

vdso/datastore: Mark vdso_k_*_data pointers as __ro_after_init

2026-06-02T19:39:29+00:00

These pointers are only modified once in vdso_setup_data_pages(), during the init phase. Make them read-only after that. Drop __refdata as that would conflict with __ro_after_init. Modpost does accept the reference from a __ro_after_init symbol to an __init one. Fixes: 05988dba1179 ("vdso/datastore: Allocate data pages dynamically") Signed-off-by: Thomas Weißschuh Signed-off-by: Thomas Gleixner Link: https://patch.msgid.link/20260513-vdso-ro-after-init-v1-1-4b51f74015a4@linutronix.de

vdso/timens: Move functions to new file

2026-03-26T14:44:22+00:00

As a preparation of the untangling of time namespaces and the vDSO, move the glue functions between those subsystems into a new file. While at it, switch the mutex lock and mmap_read_lock() in the vDSO namespace code to guard(). Signed-off-by: Thomas Weißschuh Signed-off-by: Thomas Gleixner Link: https://patch.msgid.link/20260326-vdso-timens-decoupling-v2-1-c82693a7775f@linutronix.de

vdso/datastore: Allocate data pages dynamically

2026-03-11T09:18:30+00:00

Allocating the data pages as part of the kernel image does not work on SPARC. The MMU will raise a fault when userspace tries to access them. Allocate the data pages through the page allocator instead. Unused pages in the vDSO VMA are still allocated to keep the virtual addresses aligned. Switch the mapping from PFNs to 'struct page' as that is required for dynamically allocated pages. This also aligns the allocation of the datapages with the code pages and is a prerequisite for mlockall() support. VM_MIXEDMAP is necessary for the call to vmf_insert_page() in the timens prefault path to work. The data pages need to be order-0, non-compound pages so that the mapping to userspace and the different orderings work. These pages are also used by the timekeeping, random pool and architecture initialization code. Some of these are running before the page allocator is available. To keep these subsytems working without changes, introduce early, statically data storage which will then replaced by the real one as soon as that is available. Signed-off-by: Thomas Weißschuh Signed-off-by: Thomas Gleixner Reviewed-by: Christophe Leroy (CS GROUP) Link: https://patch.msgid.link/20260304-vdso-sparc64-generic-2-v6-3-d8eb3b0e1410@linutronix.de

vdso/datastore: Drop inclusion of linux/mmap_lock.h

2026-03-11T09:18:30+00:00

This header is unnecessary and together with some upcoming changes would introduce compiler warnings. Signed-off-by: Thomas Weißschuh Signed-off-by: Thomas Gleixner Tested-by: Andreas Larsson Reviewed-by: Andreas Larsson Reviewed-by: Christophe Leroy (CS GROUP) Link: https://lore.kernel.org/lkml/20250916-mm-rcuwait-v1-1-39a3beea6ec3@linutronix.de/ Link: https://patch.msgid.link/20260304-vdso-sparc64-generic-2-v6-2-d8eb3b0e1410@linutronix.de