kernel/git/stable/linux.git/io_uring/futex.c, branch master

io_uring: defer eventfd signaling when queued from a wakeup handler

2026-08-15T22:56:49+00:00

io_req_local_work_add() signals the CQ ring eventfd inline when it is the one to push the first entry onto ->work_list. For DEFER_TASKRUN rings that add is frequently done from a waitqueue wakeup handler, where an arbitrary waitqueue lock is held. eventfd_signal_mask() only refuses to recurse when current->in_eventfd is set, but that bit is set by eventfd_signal_mask() itself. If the wake chain starts somewhere else, signal goes out inline and can feed back into epoll. Add IOU_F_TWQ_IN_WAKE, set it on the task_work add done from the three waitqueue callbacks, and use it to force io_eventfd_signal() down the existing call_rcu_hurry() deferral instead of signaling inline. Fixes: 21a091b970cd ("io_uring: signal registered eventfd to process deferred task work") Cc: stable@vger.kernel.org Link: https://lore.kernel.org/all/20260813133843.2933127-1-4ncienth@gmail.com/ Signed-off-by: Jens Axboe

io_uring/futex: only mark private futex waits as inflight

2026-07-30T16:35:28+00:00

Inflight tracking of futex wait requests exists to ensure that do_exit() -> io_uring_files_cancel() cancels them before the mm goes away, as a private futex wait depends on the mm private futex hash staying alive for the duration of the request. Shared futexes have no such dependency. A FLAGS_SHARED request always resolves to either an inode based key or an mm-shared key, both of which fail futex_key_is_private() and hence always hash into the global futex hash, whose lifetime isn't tied to the mm. Only mark vectored futex waits as inflight if the futex is private. Cc: stable@vger.kernel.org Fixes: 079afb081c42 ("io_uring/futex: mark wait requests as inflight") Link: https://lore.kernel.org/io-uring/CANGjgdn=R_qyUdE=j9za+vkmqcxacbP-84OHXF4nZ4ho9qRyVg@mail.gmail.com/ Signed-off-by: Jens Axboe

io_uring/futex: don't mark futex wake requests as inflight

2026-07-30T16:35:26+00:00

Commit 079afb081c42 ("io_uring/futex: mark wait requests as inflight") added inflight tracking to ensure that do_exit() -> io_uring_files_cancel() finds and cancels pending futex waits before the mm goes away, as a private futex wait depends on the mm private futex hash staying alive for the duration of the request. However, as io_futex_prep() is shared between FUTEX_WAIT and FUTEX_WAKE, wake requests got marked as inflight as well. A futex wake executes fully inline at issue time and never depends on the mm staying alive after completion, hence there's no need to track it. Kill it. Cc: stable@vger.kernel.org Fixes: 079afb081c42 ("io_uring/futex: mark wait requests as inflight") Reported-by: Chengfeng Lin Link: https://lore.kernel.org/io-uring/CANGjgdn=R_qyUdE=j9za+vkmqcxacbP-84OHXF4nZ4ho9qRyVg@mail.gmail.com/ Signed-off-by: Jens Axboe

futex: Add support for unlocking robust futexes

2026-06-03T09:38:51+00:00

Unlocking robust non-PI futexes happens in user space with the following sequence: 1) robust_list_set_op_pending(mutex); 2) robust_list_remove(mutex); lval = 0; 3) lval = atomic_xchg(lock, lval); 4) if (lval & WAITERS) 5) sys_futex(WAKE,....); 6) robust_list_clear_op_pending(); That opens a window between #3 and #6 where the mutex could be acquired by some other task which observes that it is the last user and: A) unmaps the mutex memory B) maps a different file, which ends up covering the same address When the original task exits before reaching #6 then the kernel robust list handling observes the pending op entry and tries to fix up user space. In case that the newly mapped data contains the TID of the exiting thread at the address of the mutex/futex the kernel will set the owner died bit in that memory and therefore corrupting unrelated data. PI futexes have a similar problem both for the non-contented user space unlock and the in kernel unlock: 1) robust_list_set_op_pending(mutex); 2) robust_list_remove(mutex); lval = gettid(); 3) if (!atomic_try_cmpxchg(lock, lval, 0)) 4) sys_futex(UNLOCK_PI,....); 5) robust_list_clear_op_pending(); Address the first part of the problem where the futexes have waiters and need to enter the kernel anyway. Add a new FUTEX_ROBUST_UNLOCK flag, which is valid for the sys_futex() FUTEX_UNLOCK_PI, FUTEX_WAKE, FUTEX_WAKE_BITSET operations. This deliberately omits FUTEX_WAKE_OP from this treatment as it's unclear whether this is needed and there is no usage of it in glibc either to investigate. For the futex2 syscall family this needs to be implemented with a new syscall. The sys_futex() case [ab]uses the @uaddr2 argument to hand the pointer to robust_list_head::list_pending_op into the kernel. This argument is only evaluated when the FUTEX_ROBUST_UNLOCK bit is set and is therefore backward compatible. This is an explicit argument to avoid the lookup of the robust list pointer and retrieving the pending op pointer from there. User space has the pointer already available so it can just put it into the @uaddr2 argument. Aside of that this allows the usage of multiple robust lists in the future without any changes to the internal functions as they just operate on the provided pointer. This requires a second flag FUTEX_ROBUST_LIST32 which indicates that the robust list pointer points to an u32 and not to an u64. This is required for two reasons: 1) sys_futex() has no compat variant 2) The gaming emulators use both both 64-bit and compat 32-bit robust lists in the same 64-bit application As a consequence 32-bit applications have to set this flag unconditionally so they can run on a 64-bit kernel in compat mode unmodified. 32-bit kernels return an error code when the flag is not set. 64-bit kernels will happily clear the full 64 bits if user space fails to set it. In case of FUTEX_UNLOCK_PI this clears the robust list pending op when the unlock succeeded. In case of errors, the user space value is still locked by the caller and therefore the above cannot happen. In case of FUTEX_WAKE* this does the unlock of the futex in the kernel and clears the robust list pending op when the unlock was successful. If not, the user space value is still locked and user space has to deal with the returned error. That means that the unlocking of non-PI robust futexes has to use the same try_cmpxchg() unlock scheme as PI futexes. If the clearing of the pending list op fails (fault) then the kernel clears the registered robust list pointer if it matches to prevent that exit() will try to handle invalid data. That's a valid paranoid decision because the robust list head sits usually in the TLS and if the TLS is not longer accessible then the chance for fixing up the resulting mess is very close to zero. The problem of non-contended unlocks still exists and will be addressed separately. Signed-off-by: Thomas Gleixner Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: André Almeida Link: https://patch.msgid.link/20260602090535.670514505@kernel.org

io_uring/futex: ensure partial wakes are appropriately dequeued

2026-04-21T18:19:06+00:00

If a FUTEX_WAITV vectored operation is only partially woken, we should call __futex_wake_mark() on the queue to account for that. If not, then a later wakeup will wake the same entry, rather than the next one in line. Fixes: 8f350194d5cfd ("io_uring: add support for vectored futex waits") Reviewed-by: Gabriel Krisman Bertazi Signed-off-by: Jens Axboe

treewide: Replace kmalloc with kmalloc_obj for non-scalar types

2026-02-21T09:02:28+00:00

This is the result of running the Coccinelle script from scripts/coccinelle/api/kmalloc_objs.cocci. The script is designed to avoid scalar types (which need careful case-by-case checking), and instead replace kmalloc-family calls that allocate struct or union object instances: Single allocations: kmalloc(sizeof(TYPE), ...) are replaced with: kmalloc_obj(TYPE, ...) Array allocations: kmalloc_array(COUNT, sizeof(TYPE), ...) are replaced with: kmalloc_objs(TYPE, COUNT, ...) Flex array allocations: kmalloc(struct_size(PTR, FAM, COUNT), ...) are replaced with: kmalloc_flex(*PTR, FAM, COUNT, ...) (where TYPE may also be *VAR) The resulting allocations no longer return "void *", instead returning "TYPE *". Signed-off-by: Kees Cook

io_uring/futex: use GFP_KERNEL_ACCOUNT for futex data allocation

2026-01-25T17:07:09+00:00

Be a bit nicer and ensure it plays nice with memcg accounting. Signed-off-by: Jens Axboe

io_uring/futex: move futexv owned status to struct io_futexv_data

2025-11-05T19:55:07+00:00

Free up a bit of space in the shared futex opcode private data, by moving the futexv specific futexv_owned out of there and into the struct specific to vectored futexes. Signed-off-by: Jens Axboe

io_uring/futex: move futexv async data handling to struct io_futexv_data

2025-11-05T19:54:31+00:00

Rather than alloc an array of struct futex_vector for the futexv wait handling, wrap it in a struct io_futexv_data struct, similar to what the non-vectored futex wait handling does. No functional changes in this patch. Signed-off-by: Jens Axboe

io_uring: add wrapper type for io_req_tw_func_t arg

2025-11-03T15:31:26+00:00

In preparation for uring_cmd implementations to implement functions with the io_req_tw_func_t signature, introduce a wrapper struct io_tw_req to hide the struct io_kiocb * argument. The intention is for only the io_uring core to access the inner struct io_kiocb *. uring_cmd implementations should instead call a helper from io_uring/cmd.h to convert struct io_tw_req to struct io_uring_cmd *. Signed-off-by: Caleb Sander Mateos Signed-off-by: Jens Axboe