kernel/git/stable/linux.git/fs/eventpoll.c, branch master

Merge tag 'vfs-7.3-rc1.misc' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs

2026-08-17T19:56:12+00:00

Pull misc vfs updates from Christian Brauner: "Bigger cleanups: - The lockref dead-count handling is tidied up. The open-coded check for a count below zero as the dead marker relies on information the caller should not have. - make put_mnt_ns() leave mounts connected. Destroying a mount namespace disconnected its mounts from their mount points. So a file descriptor still open on the parent of a mount point could be used to peek under it. Locked mounts were already kept connected to prevent exactly that. But a mount is only locked when its tree is copied across a user namespace boundary. So a mount namespace set up by a privileged component had no locked mounts and its mounts were disconnected. Passing UMOUNT_CONNECTED keeps every mount connected and prevents that bug. - vfs_prepare_mode() passes S_IFDIR for directories. I meant to fix that ago but didn't get to it. So now someone finally did it. This kills the exception where the mode could be 0 when a directory was created whereas every other creation operation passed it explicitly already. - move long delayed work for ufs, jffs2, hfsplus, hfs and affs from the per-cpu system_long_wq to the new unbound system_dfl_long_wq. None of that work relies on per-cpu state and the work item is enqueued with queue_delayed_work() whose timer is global anyway. So it may as well benefit from scheduler task placement. Smaller fixes and cleanups: - unlock_buffer() and journal_end_buffer_io_sync() use clear_and_wake_up_bit() - the pipe page pools are unified into a single per-pipe pool and the extra wake_up(rd_wait) is limited to EPOLLET consumers - eventpoll now computes its timer slack lazily in ep_poll() - shrink_dcache_for_umount() keeps making progress on busy roots - excess xarray nodes are freed in clear_inode() - romfs detects hard link cycles - the user path of nested backing files is fixed - pidfd holds exec_update_lock around the namespace ioctl - non-memcg-aware nr_cached_objects is skipped during memcg slab shrink - iomap_write_iter() always returns status - mangle_path() is renamed to seq_mangle_path() - inode timestamp accessors are annotated - new regression test for pipe->poll_usage. - a few documentation, kernel-doc and selftest fixes" * tag 'vfs-7.3-rc1.misc' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: (67 commits) selftests/namespaces: Fix racy pipe handshake in timens and pidns_separate selftests/epoll: add a regression test for pipe->poll_usage pipe: only enable the extra wake_up(rd_wait) for EPOLLET consumers pidfd: hold exec_update_lock around namespace ioctl fs: fix user path of nested backing files fs: remove stale inode_insert5() kernel-doc parameter fs: fix switch/case indentation in sysfs() syscall fs: document semantics of kstat::{uid,gid} fields dcache: keep shrink_dcache_for_umount() making progress on busy roots seq_file: rename mangle_path to seq_mangle_path nstree: add/fix struct ns_id_req kernel-doc member fields dcache: use lockref routines for dead count checks lockref: tidy up dead count handling initramfs: fix typo in reserve_initrd_mem comment fs/pipe: unify the page pools into a single per-pipe pool fs: annotate inode timestamp accessors eventpoll: compute timer slack lazily in ep_poll() selftests/filesystems: add mntns cleanup test put_mnt_ns(): leave mounts connected affs: Move long delayed work on system_dfl_long_wq ...

eventpoll: compute timer slack lazily in ep_poll()

2026-07-31T08:09:11+00:00

ep_poll() computes the timer slack via select_estimate_accuracy() up front, before checking whether events are already available. select_estimate_accuracy() reads the clock (ktime_get_ts64()), and the resulting slack is only consumed by the schedule_hrtimeout_range() call on the blocking path. A busy poller such as an L7 proxy event loop calls epoll_wait() at a very high rate and often finds events already pending, returning via ep_try_send_events() without ever blocking. In that case the up-front slack estimation - including its clock read - is pure overhead. read_tsc() attributable to select_estimate_accuracy() sometimes shows up in perf profiles of such a workload via the epoll_wait() path. Move the slack estimation to the point where the thread is actually about to sleep. The timeout passed to ep_poll() is already an absolute deadline (ep_timeout_to_timespec()), so deferring the estimate does not change the wakeup time; taken closer to the sleep it is, if anything, marginally more accurate. On the common non-blocking path the clock read is skipped entirely. Measured on a host running a Meta production workload with the following bpftrace script: #!/usr/bin/bpftrace fentry:__x64_sys_epoll_wait, fentry:__x64_sys_epoll_pwait { @in[tid] = 1; } fexit:__x64_sys_epoll_wait, fexit:__x64_sys_epoll_pwait { delete(@in, tid); } fentry:select_estimate_accuracy /@in[tid]/ { @sea++; } fentry:schedule_hrtimeout_range /@in[tid]/ { @shr++; } interval:s:30 { printf("sea=%lld shr=%lld wasted=%lld (%d%%)\n", @sea, @shr, @sea - @shr, (@sea - @shr) * 100 / @sea); exit(); } Over a 30s window: sea=3,587,704 shr=3,003,920 wasted=583,784 (16%) So ~16% of ep_poll invocations of select_estimate_accuracy have no consumer. Signed-off-by: Usama Arif Link: https://patch.msgid.link/20260707190238.3478608-1-usama.arif@linux.dev Signed-off-by: Christian Brauner (Amutable)

eventpoll: pin files while checking reverse paths

2026-07-23T10:48:22+00:00

Commit 319c15174757 ("epoll: take epitem list out of struct file") intentionally removed temporary file references from the reverse path check list. At the time, both epitems and their files were freed after an RCU grace period, so unlist_file() could obtain file->f_lock through an epitem while clear_tfile_check_list() held rcu_read_lock(). Commit 0ede61d8589c ("file: convert to SLAB_TYPESAFE_BY_RCU") made struct file SLAB_TYPESAFE_BY_RCU and removed its RCU-delayed freeing. RCU still protects the epitem, but no longer keeps the referenced file from being freed and reused. A concurrent close can therefore make unlist_file() lock or unlock f_lock in a recycled file object. This violates the documented SLAB_TYPESAFE_BY_RCU rule requiring a reference before acquiring an object's lock. The race was reproduced, causing a wild unlock of f_lock in a recycled file and breaking its mutual exclusion. Add ->file to epitems_head to remember the pinned file independently of ->epitems. A concurrent EPOLL_CTL_DEL can empty ->epitems before the head is unlisted, leaving no epi->ffd.file from which to drop the reference. In list_file(), acquire the reference before adding the head to the check list. The caller either owns a reference or holds the ep->mtx for the epitem leading to the file. In the latter case, file_ref_get() can fail after the last reference is dropped, but eventpoll_release_file() must acquire the same mutex before the file can be freed. The dying leaf can be skipped because removing links cannot increase the reverse path count. In unlist_file(), epnested_mutex excludes another list_file() or unlist_file(), while head->next prevents a concurrent EPOLL_CTL_DEL from freeing the head. Save head->file locally, clear it with head->next under f_lock, and drop the reference after the RCU-protected operation. Christian Brauner quotes: > SLAB_TYPESAFE_BY_RCU allows a slab slot to be reused while an RCU reader > still holds its old address. Once that address contains a new live > struct file, KASAN sees valid, unpoisoned memory and cannot distinguish > the stale object identity. CONFIG_DEBUG_SPINLOCK exposes the failure > instead. > > The failing interleaving is: > > CPU0: nested EPOLL_CTL_ADD CPU1: close/open churn > ------------------------------------ --------------------------------- > p = hlist_first_rcu(&head->epitems) > epi = container_of(p, ...) > close(victim) > __fput() > eventpoll_release_file() > file_free(victim) > // the slot is free; f_lock remains > spin_lock(&epi->ffd.file->f_lock) > open() reuses the slot as new_file > spin_lock_init(&new_file->f_lock) > spin_unlock(&epi->ffd.file->f_lock) // wild unlock of new_file's lock > > CONFIG_DEBUG_SPINLOCK reports: > > BUG: spinlock already unlocked on CPU#0, poc_unlist/150 > lock: 0xffff8880067fb200, .magic: dead4ead, .owner: /-1, .owner_cpu: -1 > CPU: 0 UID: 1000 PID: 150 Comm: poc_unlist Not tainted 7.2.0-rc3-dirty #22 PREEMPTLAZY > Hardware name: QEMU Ubuntu 24.04 PC v2 (i440FX + PIIX, arch_caps fix, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 > Call Trace: > > dump_stack_lvl+0x64/0x80 > do_raw_spin_unlock+0x75/0xb0 > _raw_spin_unlock+0xe/0x30 > clear_tfile_check_list+0x88/0xe0 > do_epoll_ctl_file+0x519/0xcf0 > ? __pfx_ep_ptable_queue_proc+0x10/0x10 > do_epoll_ctl+0x8f/0x100 > __x64_sys_epoll_ctl+0x6f/0xa0 > do_syscall_64+0xdc/0x520 > ? srso_alias_return_thunk+0x5/0xfbef5 > entry_SYSCALL_64_after_hwframe+0x76/0x7e > RIP: 0033:0x42034e > Code: 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 e9 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48 > RSP: 002b:00007a657ff3c198 EFLAGS: 00000202 ORIG_RAX: 00000000000000e9 > RAX: ffffffffffffffda RBX: 00007a657ff3ccdc RCX: 000000000042034e > RDX: 0000000000000003 RSI: 0000000000000001 RDI: 0000000000000004 > RBP: 00007a657ff3c2f0 R08: 0000000000000000 R09: 00007a657ff3c6c0 > R10: 00007a657ff3c1a4 R11: 0000000000000202 R12: 00007a657ff3c6c0 > R13: ffffffffffffffb8 R14: 000000000000000d R15: 00007fffb7de0210 > > ------------[ cut here ]------------ > > unlist_file() does not appear as a separate frame because it was inlined > into clear_tfile_check_list(). This report was obtained with mdelay() > instrumentation immediately before spin_lock() and spin_unlock() in > unlist_file() to widen the two race windows. > > More importantly, this is a wild unlock. The stale unlock can target > f_lock of a different live file and invalidate mutual exclusion for > state protected by that lock. Turning this into a reliable exploit > would require precise scheduling and same-slot reuse and is likely > difficult, but the primitive is potentially exploitable. Reported-by: Qi Tang Reported-by: Junxi Qian Fixes: 0ede61d8589c ("file: convert to SLAB_TYPESAFE_BY_RCU") Cc: stable@vger.kernel.org Signed-off-by: Guidong Han <2045gemini@gmail.com> Link: https://patch.msgid.link/20260718104406.27897-1-2045gemini@gmail.com Signed-off-by: Christian Brauner (Amutable)

eventpoll: restore EP_UNACTIVE_PTR sentinel for ctx->tfile_check_list

2026-06-04T11:53:50+00:00

Commit e09c77d94003 ("eventpoll: hoist CTL_ADD scratch state into struct ep_ctl_ctx") moved tfile_check_list from a file-scope global into the stack-allocated struct ep_ctl_ctx, and in doing so replaced the EP_UNACTIVE_PTR sentinel with NULL on the grounds that "NULL is the obvious 'empty' value and zero-init handles it for free", describing the change as "No functional change". It is not. epitems_head->next is overloaded with two roles: 1. the "next" pointer that threads a head onto ctx->tfile_check_list; 2. a membership flag: ep_remove_file() uses !smp_load_acquire(&v->next) to mean "this head is not on any pending ctx->tfile_check_list and is therefore safe to free". Before that change the EP_UNACTIVE_PTR sentinel kept the two roles disjoint: a head on the list always had a non-NULL ->next (another head, or the sentinel at the tail), so ->next == NULL was equivalent to "never listed". With the sentinel gone the list is NULL-terminated, so the tail head's ->next is NULL as well. ep_remove_file()'s gate can no longer distinguish "never listed" from "listed at the tail", and misfires on the tail head. The reader (reverse_path_check_proc) holds epnested_mutex + rcu_read_lock; the freer (ep_remove_file) holds ep->mtx + file->f_lock. The two sides share no mutex -- the sentinel was the invariant the gate relied on to know it could skip the read side. With it gone, ep_remove_file() frees the tail head while reverse_path_check_proc() is still walking it, producing the slab-use-after-free read. The syzbot reproducer hits this within seconds on a multi-CPU VM. Restore the sentinel: initialize ctx.tfile_check_list to EP_UNACTIVE_PTR in do_epoll_ctl_file(), and terminate the walk on "!= EP_UNACTIVE_PTR" in reverse_path_check() and clear_tfile_check_list(). The tail head's ->next becomes the sentinel again rather than NULL, so ep_remove_file()'s gate regains its exclusivity and stops misfiring on the tail. ep_remove_file() itself is unchanged. This restores the invariant the file-scope tfile_check_list relied on before that change while preserving the ctx packaging it introduced. Reported-by: syzbot+e70e1b6cba8714543f7c@syzkaller.appspotmail.com Closes: https://syzkaller.appspot.com/bug?extid=e70e1b6cba8714543f7c Fixes: e09c77d94003 ("eventpoll: hoist CTL_ADD scratch state into struct ep_ctl_ctx") Suggested-by: Christian Brauner Link: https://lore.kernel.org/all/20260528-rotwild-summt-kuhhandel-7276ef4c33b7@brauner.io/ Signed-off-by: Zhan Wei Link: https://patch.msgid.link/20260529142533.23696-1-zhanwei919@gmail.com Signed-off-by: Christian Brauner (Amutable)

eventpoll: Fix epoll_wait() report false negative

2026-06-04T08:25:10+00:00

ep_events_available() checks for available events by looking at ep->rdllist and ep_is_scanning(). However, this is done without a lock and can report false negative if ep_start_scan() or ep_done_scan() are executed by another task concurrently. For example: _________________________________________________________________________ |ep_start_scan() | list_splice_init(&ep->rdllist, ...) ep_events_available() | !list_empty_careful(&ep->rdllist)| || ep_is_scanning(ep) | | ep_enter_scan(ep) ___________________________________|_____________________________________ Another example: _________________________________________________________________________ ep_events_available() | |ep_start_scan() | list_splice_init(&ep->rdllist, ...) | ep_enter_scan(ep) !list_empty_careful(&ep->rdllist)| |ep_done_scan() | ep_exit_scan(ep) | list_splice(..., &ep->rdllist) || ep_is_scanning(ep) | ___________________________________|_____________________________________ In the above examples, ep_events_available() sees no event despite events being available. In case epoll_wait() is called with timeout=0, epoll_wait() will wrongly return "no event" to user. Introduce a sequence lock to resolve this issue. Measuring the time consumption of 10 million loop iterations doing epoll_wait(), the following performance drop is observed: timeout #event before after diff 0ms 0 3727ms 3974ms +6.6% 0ms 1 8099ms 9134ms +13% 1ms 1 13525ms 13586ms +0.45% Considering the use case of epoll_wait() (wait for events, do something with the events, repeat), it should only contribute to a small portion of user's CPU consumption. Therefore this performance drop is not alarming. Fixes: c5a282e9635e ("fs/epoll: reduce the scope of wq lock in epoll_wait()") Suggested-by: Mateusz Guzik Signed-off-by: Nam Cao Link: https://patch.msgid.link/4363cd8e34a21d4f0d257be1b33e84dc25030fdf.1780422138.git.namcao@linutronix.de Signed-off-by: Christian Brauner (Amutable)

eventpoll: add missing kernel-doc for @ctx function parameters

2026-05-22T12:32:06+00:00

Add the missing kernel-doc comments to prevent kernel-doc build warnings while building the documentation. WARNING: fs/eventpoll.c:1684 function parameter 'ctx' not described in 'reverse_path_check' WARNING: fs/eventpoll.c:2349 function parameter 'ctx' not described in 'ep_loop_check_proc' Fixes: e09c77d94003 ("eventpoll: hoist CTL_ADD scratch state into struct ep_ctl_ctx") Signed-off-by: Randy Dunlap Link: https://patch.msgid.link/20260519042314.124041-1-rdunlap@infradead.org Signed-off-by: Christian Brauner (Amutable)

Merge patch series "io_uring related epoll cleanups"

2026-05-15T15:41:05+00:00

Jens Axboe says: One of the nastier things about epoll is how it allows nesting contexts inside each other, leading to the necessity of loop detection and the issues that have come with that. I don't believe there's any reason to support nesting on the io_uring side, in fact IORING_OP_EPOLL_CTL is a historical mistake, imho. But let's at least try and contain the damage and disallow nested contexts from our side. Christian Brauner says: Bring in the eventpoll specific io_uring changes together with the eventpoll cleanup I did this cycle. The io_uring changes can go on top of both through the block tree. * patches from https://patch.msgid.link/20260514140817.623026-1-axboe@kernel.dk: eventpoll: rename struct epoll_filefd to epoll_key eventpoll: add file based control interface eventpoll: export is_file_epoll() eventpoll: pass struct epoll_filefd through ep_find() and ep_insert() Link: https://patch.msgid.link/20260514140817.623026-1-axboe@kernel.dk Signed-off-by: Christian Brauner (Amutable)

eventpoll: rename struct epoll_filefd to epoll_key

2026-05-15T15:34:26+00:00

This more accurately describes what purpose this structure serves, as a lookup key. Suggested-by: Christian Brauner Signed-off-by: Jens Axboe Link: https://patch.msgid.link/20260514140817.623026-5-axboe@kernel.dk Signed-off-by: Christian Brauner

eventpoll: add file based control interface

2026-05-15T15:34:26+00:00

Add do_epoll_ctl_file(), which takes a pre-resolved epoll file and a struct epoll_filefd for the target rather than two integer file descriptors. do_epoll_ctl() remains as a thin wrapper. In preparation for using the file based interface from io_uring. Signed-off-by: Jens Axboe Link: https://patch.msgid.link/20260514140817.623026-4-axboe@kernel.dk Signed-off-by: Christian Brauner

eventpoll: export is_file_epoll()

2026-05-15T15:34:26+00:00

Make is_file_epoll() available outside of epoll. This is in preparation from using it from io_uring. Signed-off-by: Jens Axboe Link: https://patch.msgid.link/20260514140817.623026-3-axboe@kernel.dk Reviewed-by: Christian Brauner Signed-off-by: Christian Brauner