kernel/git/stable/linux.git/net/rds/send.c, branch master

net/rds: acquire the fastpath locks in rds_conn_shutdown()

2026-09-03T02:42:23+00:00

rds_conn_shutdown() quiesces the transmit and receive-refill paths by waiting for RDS_IN_XMIT and RDS_RECV_REFILL to be sampled clear, and then runs the transport shutdown and rds_conn_path_reset(). Sampling the bits clear is not the same as owning them: the moment after the wait_event() returns, rds_send_xmit() can re-acquire RDS_IN_XMIT (or rds_ib_recv_refill() can re-acquire RDS_RECV_REFILL) and run concurrently with the teardown. The sender does recheck the connection state after taking the lock, but that recheck is a classic store-buffering pattern: teardown writes the state and reads the bit while the sender writes the bit and reads the state. acquire_in_xmit() is only an acquire operation, so on weakly ordered architectures both sides can miss each other's write, and the transmit path then runs while the transport zeroes its rings (e.g. rds_ib_ring_init()) and rds_send_path_reset() rewrites the transmit state under it. Oracle UEK fixed the same class of crashes - a 14-year tail of BUG_ON()s in rds_ib_sub_signaled(), unexpected op-codes and NULL dereferences in rds_ib_send_cqe_handler() during failover testing - by making the teardown path *acquire* the fastpath bit locks instead of testing them ("rds: Make sure transmit path and connection tear-down does not run concurrently"). Ownership of a single word is decided by RMW atomicity, so no cross-variable ordering is needed. Do the same here: take both locks before calling the transport shutdown, hold them across rds_conn_path_reset(), and release them explicitly with a wake-up afterwards. Both are released with clear_bit_unlock(), so that the ring re-initialization done by the transport shutdown and the transmit state rewritten by rds_send_path_reset() are ordered before either bit is seen clear by the next acquire_in_xmit() or acquire_refill(). The fastpath users of these bits - rds_send_xmit() and rds_ib_recv_refill() - are trylock style and back off while teardown owns the locks, so no new lock dependency is introduced for them. rds_tcp_reset_callbacks() is different: since the previous patch it acquires RDS_IN_XMIT as well, and it blocks doing so, so its wait now spans the teardown instead of at most one send batch. That waiter runs from rds_tcp_accept_one() on the single-threaded krdsd workqueue and holds rds_tcp_accept_lock and t_conn_path_lock while it waits, so a duelling SYN accepted while its path is being torn down parks accept processing for the duration of the teardown - for TCP bounded by the (up to 5 s) drain loop in rds_tcp_conn_path_shutdown(). An IB path's drain in rds_ib_conn_path_shutdown() has no round cap, but no blocking waiter either: rds_tcp_reset_callbacks() is the only blocking acquirer of these bits and waits only on its own TCP path, and the fastpaths are trylock-and-back-off on both transports, so a long IB drain lengthens only that path's own quiesce. The window is narrow: the accept-side state check has to pass before the teardown moves the path to RDS_CONN_DISCONNECTING. Because krdsd is a single global workqueue, everything else queued there - accept processing for other connections and network namespaces, and the flush_workqueue(rds_wq) in rds_tcp_listen_stop() during namespace teardown - waits behind the parked accept worker for that time. It cannot deadlock, although the waits do point at each other: the teardown blocks until the bit's holder releases it, and the holder may be that krdsd accept worker. The holder finishes without needing anything the teardown owns: the sync cancels rds_tcp_reset_callbacks() issues target cp_send_w and cp_recv_w on the path's ordered cp_wq, whose only execution slot is occupied by the blocked cp_down_w itself, so they are pending at most and cancel without flushing - a reliance on cp_wq being ordered that is now noted next to those cancels (on the allocation-failure fallback where a path shares rds_wq, the work items simply serialize). Nor is the blocking wait itself new: rds_tcp_reset_callbacks() has waited on RDS_IN_XMIT from the krdsd work item since commit 335b48d980f6 ("RDS: TCP: Add/use rds_tcp_reset_callbacks to reset tcp socket safely"); this patch stretches its worst case from a sender's batch to the teardown's drain. The alternative to parking is the accept path racing the teardown, which is what these patches close; making the teardown itself non-blocking is a separate item. One observable side effect: the SENDING flag reported by rds-info has always mirrored RDS_IN_XMIT, so it now also covers the window where teardown owns the bit. The comments that describe the old sample-based handshake or name rds_send_xmit() as the only other holder of these bits - in rds_send_xmit(), above rds_conn_path_reset(), in rds_ib_recv_refill() and in rds_tcp_reset_callbacks() - are updated to match. For anyone backporting this patch standalone: it depends on "net/rds: clear cp_flags bits individually in rds_conn_path_reset()" and "net/rds: acquire RDS_IN_XMIT in rds_tcp_reset_callbacks()" earlier in this series. Without the former, the blanket cp_flags clear in rds_conn_path_reset() would drop both held bits in the middle of the teardown; without the latter, rds_tcp_reset_callbacks() would still sample t_sock without owning RDS_IN_XMIT. "net/rds: use clear_bit_unlock() in release_refill()" is needed for the refill side's release to pair with the acquire added here, and the follow-up "net/rds: don't let rds_conn_shutdown() consume a concurrent drop" completes the teardown-state handling for the waiter this patch parks; a backport should carry all four. Fixes: 0f4b1c7e89e6 ("rds: fix rds_send_xmit() serialization") Signed-off-by: Håkon Bugge [achender: reimplement for net-next shutdown path: acquire the existing RDS_IN_XMIT/RDS_RECV_REFILL bit locks in rds_conn_shutdown() and release after teardown; update comments and commit message] Signed-off-by: Allison Henderson Link: https://patch.msgid.link/20260828223921.202913-7-achender@kernel.org Signed-off-by: Jakub Kicinski

net/rds: use wq_has_sleeper() in release_in_xmit()

2026-09-03T02:42:23+00:00

release_in_xmit() clears RDS_IN_XMIT with clear_bit_unlock() and then checks waitqueue_active() to decide whether anyone needs waking. clear_bit_unlock() is only a release operation: it orders the critical section before the bit clear, but does not order the subsequent plain load of the wait queue head after it. The waiter side does the mirror image - it adds itself to the wait queue and then tests the bit. That is the classic store-buffering pattern: the releasing CPU can read the wait queue as empty while the waiting CPU still reads the bit as set, so the sleeper is never woken. The waiters are rds_conn_shutdown() and rds_tcp_reset_callbacks(), both in uninterruptible wait_event() with no timeout. A lost wake-up strands the shutdown worker on its single-threaded workqueue until some other sender releases the bit again - and on a connection that is being torn down precisely because it failed, there may never be another sender. The barrier used to be there: release_in_xmit() did clear_bit() followed by smp_mb__after_atomic() until commit 1422f28826d2 ("rds: introduce acquire/release ordering in acquire/release_in_xmit()") folded both into clear_bit_unlock(), which strengthened the lock hand-off but silently dropped the full barrier the wake-up check depends on. The refill counterpart, release_refill() in net/rds/ib_recv.c, still carries its smp_mb__after_atomic() for exactly this reason. Use wq_has_sleeper(), which is waitqueue_active() preceded by the required full barrier. Fixes: 1422f28826d2 ("rds: introduce acquire/release ordering in acquire/release_in_xmit()") Signed-off-by: Allison Henderson Link: https://patch.msgid.link/20260828223921.202913-2-achender@kernel.org Signed-off-by: Jakub Kicinski

net/rds: reinitialize to_be_dropped on rds_send_xmit() restart

2026-08-13T10:52:29+00:00

The to_be_dropped list is declared once at the top of rds_send_xmit() but the function can loop via "goto restart" after each batch. The code currently relies on rds_send_remove_from_sock() having emptied the list entry by entry (via list_del_init()) at the end of the previous batch; nothing in rds_send_xmit() itself guarantees the list head is empty when a new batch starts. Re-initialize the list on every restart, and warn once if it is ever found non-empty there: entries left on the list at that point would keep their message reference, their RDS_MSG_ON_SOCK accounting and their pending RDS_RDMA_DROPPED notification, so a silent re-init would orphan them. This is hardening: no user-visible bug is known in the current code. This mirrors Oracle UEK commit "net/rds: rds_send_xmit should INIT_LIST_HEAD(&to_be_dropped) on restart". Signed-off-by: Gerd Rausch Signed-off-by: Sharath Srinivasan [achender: port to net-next (keep the existing LIST_HEAD declaration and add only the restart re-init); warn if the restart invariant is violated; update commit message] Assisted-by: Claude-Code:claude-fable-5 Signed-off-by: Allison Henderson Link: https://patch.msgid.link/20260809005103.82371-2-achender@kernel.org Reviewed-by: Simon Horman Signed-off-by: Paolo Abeni

net/rds: fix rds_message leak in the rds_send_xmit() drop path

2026-08-05T01:00:58+00:00

When rds_send_xmit() picks the next message off cp_send_queue it takes its own reference with rds_message_addref(). If the message then hits the never-retransmit check (RDS_MSG_FLUSH, or an RDMA op that was already retransmitted), it is moved to the local to_be_dropped list and that reference is dropped after the batch. However, if RDS_MSG_ON_CONN has already been cleared, the message is not added to to_be_dropped and the reference taken above is never dropped: cp_xmit_rm has not been set at this point, so the loop simply abandons rm and the rds_message (and everything it pins: pages, MRs, notifiers) leaks after an RDMA error. The only other places that clear RDS_MSG_ON_CONN are rds_send_path_drop_acked() and rds_send_drop_to(), and both can run while rds_send_xmit() has dropped cp_lock between moving the message to cp_retrans and re-taking the lock in the never-retransmit check: rds_send_path_drop_acked() can ack away a message that already sat on cp_retrans - the RDS_MSG_RETRANSMITTED case above - and rds_send_drop_to() runs on socket close. Both unlink the message under cp_lock and put their own reference, leaving the xmit-path reference stranded. Drop the reference directly in that case. This mirrors Oracle UEK commit "net/rds: fix rds_message memleak in rds_send_xmit". Signed-off-by: Gerd Rausch Signed-off-by: Sharath Srinivasan [achender: port to net-next; update commit message, checkpatch nits] Signed-off-by: Allison Henderson Link: https://patch.msgid.link/20260730041629.3512480-4-achender@kernel.org Signed-off-by: Jakub Kicinski

net/rds: use krealloc_array() for iovector growth

2026-07-24T23:20:00+00:00

Use krealloc_array() for growing the RDS iovector array. This makes the array allocation overflow-safe and derives the element size from the array pointer. Reviewed-by: Allison Henderson Signed-off-by: Weimin Xiong Link: https://patch.msgid.link/20260717022537.331863-1-xiongwm2026@163.com Signed-off-by: Jakub Kicinski

net: rds: check cmsg_len before reading rds_rdma_args in size pass

2026-06-19T01:25:28+00:00

rds_rm_size() handles RDS_CMSG_RDMA_ARGS after only CMSG_OK() and then calls rds_rdma_extra_size(), which reads args->local_vec_addr and args->nr_local without first checking that cmsg_len covers struct rds_rdma_args. The other two RDS_CMSG_RDMA_ARGS consumers already guard this: rds_rdma_bytes() in rds_sendmsg() and rds_cmsg_rdma_args() in rds_cmsg_send() both reject cmsg_len < CMSG_LEN(sizeof(struct rds_rdma_args)). Add the same check to rds_rm_size() so all three RDMA args passes are consistent. This is a consistency and hardening change with no behavioral effect for well-formed senders and no reachable bug today: rds_rdma_bytes() runs before rds_rm_size() in rds_sendmsg() and already rejects a short RDS_CMSG_RDMA_ARGS, so the size pass is not reached with an undersized cmsg. But rds_rm_size() reads the args independently of that earlier pass, and nothing in rds_rm_size() itself records or enforces the precondition, so a reader or a future refactor of the size pass cannot tell the cmsg has already been length-checked. Applying the same cmsg_len guard in all three RDS_CMSG_RDMA_ARGS consumers keeps that invariant local to each and robust to reordering. Signed-off-by: Michael Bommarito Reviewed-by: Allison Henderson Link: https://patch.msgid.link/20260617023146.2780077-1-michael.bommarito@gmail.com Signed-off-by: Jakub Kicinski

rds: annotate data-race around rs_seen_congestion

2026-05-25T18:36:37+00:00

rs_seen_congestion is read in rds_poll() and written in rds_sendmsg() and rds_poll() without any lock. Use READ_ONCE()/WRITE_ONCE() to annotate these lockless accesses and silence KCSAN. Reported-by: syzbot+fbf3648ae7f5bdb05c59@syzkaller.appspotmail.com Closes: https://lore.kernel.org/netdev/6a0f8d94.050a0220.6b33c.0000.GAE@google.com/ Signed-off-by: Jiayuan Chen Reviewed-by: Allison Henderson Tested-by: Allison Henderson Link: https://patch.msgid.link/20260522011621.304470-1-jiayuan.chen@linux.dev Signed-off-by: Jakub Kicinski

rds: update outdated comment

2026-02-26T03:03:55+00:00

The function rds_send_reset() was subsumed by rds_send_path_reset() by commit d769ef81d5b5 ("RDS: Update rds_conn_shutdown to work with rds_conn_path"). Update the comment accordingly. Signed-off-by: kexinsun Link: https://patch.msgid.link/20260224020720.1174-1-kexinsun@smail.nju.edu.cn Signed-off-by: Jakub Kicinski

net/rds: rds_sendmsg should not discard payload_len

2026-02-17T11:03:57+00:00

Commit 3db6e0d172c9 ("rds: use RCU to synchronize work-enqueue with connection teardown") modifies rds_sendmsg to avoid enqueueing work while a tear down is in progress. However, it also changed the return value of rds_sendmsg to that of rds_send_xmit instead of the payload_len. This means the user may incorrectly receive errno values when it should have simply received a payload of 0 while the peer attempts a reconnections. So this patch corrects the teardown handling code to only use the out error path in that case, thus restoring the original payload_len return value. Fixes: 3db6e0d172c9 ("rds: use RCU to synchronize work-enqueue with connection teardown") Reviewed-by: Simon Horman Signed-off-by: Allison Henderson Link: https://patch.msgid.link/20260213035409.1963391-1-achender@kernel.org Signed-off-by: Paolo Abeni

net/rds: Trigger rds_send_ping() more than once

2026-02-05T04:46:39+00:00

Even though a peer may have already received a non-zero value for "RDS_EXTHDR_NPATHS" from a node in the past, the current peer may not. Therefore it is important to initiate another rds_send_ping() after a re-connect to any peer: It is unknown at that time if we're still talking to the same instance of RDS kernel modules on the other side. Otherwise, the peer may just operate on a single lane ("c_npaths == 0"), not knowing that more lanes are available. However, if "c_with_sport_idx" is supported, we also need to check that the connection we accepted on lane#0 meets the proper source port modulo requirement, as we fan out: Since the exchange of "RDS_EXTHDR_NPATHS" and "RDS_EXTHDR_SPORT_IDX" is asynchronous, initially we have no choice but to accept an incoming connection (via "accept") in the first slot ("cp_index == 0") for backwards compatibility. But that very connection may have come from a different lane with "cp_index != 0", since the peer thought that we already understood and handled "c_with_sport_idx" properly, as indicated by a previous exchange before a module was reloaded. In short: If a module gets reloaded, we recover from that, but do *not* allow a downgrade to support fewer lanes. Downgrades would require us to merge messages from separate lanes, which is rather tricky with the current RDS design. Each lane has its own sequence number space and all messages would need to be re-sequenced as we merge, all while handling "RDS_FLAG_RETRANSMITTED" and "cp_retrans" properly. Signed-off-by: Gerd Rausch Signed-off-by: Allison Henderson Link: https://patch.msgid.link/20260203055723.1085751-9-achender@kernel.org Signed-off-by: Jakub Kicinski