kernel/git/stable/linux.git/arch/arm64/net, branch master

bpf, arm64: Convert struct_ops arena arguments in the trampoline

2026-08-14T03:55:02+00:00

Implement the struct_ops arena argument conversion on arm64. save_args() receives the arena base from bpf_tramp_arena_base() and consults the btf_func_model argument flags as it copies each native argument into the BPF ctx, routing a marked argument through x10 with the low half of the base materialized once into x11: sub w10, wsrc, w11 /* truncate and clear the upper 32 bits */ str x10, [sp, #slot] A nullable argument tests the full 64-bit kernel pointer first: mov x10, xsrc cbz x10, 1f sub w10, w10, w11 1: str x10, [sp, #slot] The 32-bit subtraction is sufficient since (u32)(kaddr - base) == (u32)kaddr - (u32)base, and it clears the upper half as the JITs require of arena pointer registers. Stack-passed arguments already reload through x10, so only the subtraction (and the NULL test) is inserted there. The register loop now walks arguments rather than registers so that the per-argument flags line up with the slots a multi-slot argument occupies; the sequence of stores is otherwise unchanged. bpf_tramp_arena_base() returns a base only for a single-program struct_ops indirect trampoline, so a tracing trampoline emits exactly what it did before and never touches x11. The size probe reruns the same emission with the same model and nodes, so the image size matches by construction. Conversion must never reach the original function, which takes kernel addresses. That holds because BPF_TRAMP_F_INDIRECT is incompatible with BPF_TRAMP_F_CALL_ORIG, so pass 0 rather than the base to the call-origin save_args() and assert the flag combination the same way x86 does, rather than leaving the invariant to a comment. With both the kfunc and struct_ops directions implemented, flip bpf_jit_supports_arena_args() on for arm64 and drop the x86-64-only qualifier from the kfunc documentation. Signed-off-by: Puranjay Mohan Reviewed-by: Xu Kuohai Link: https://lore.kernel.org/bpf/20260813190356.335181-5-puranjay@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi

bpf, arm64: JIT __arena kfunc argument rebasing

2026-08-14T03:55:01+00:00

Implement arena argument rebasing for kfunc calls on arm64. x28 already holds kern_vm_start whenever the prog has an arena, and the newly added extended-register add zero-extends the 32-bit arena offset in place, so an unconditional argument costs a single instruction emitted right before the call: add xN, x28, wN, uxtw A nullable argument first truncates into wN so that a zero offset leaves xN holding a real NULL, then tests it and jumps over the add: mov wN, wN cbz wN, 1f add xN, x28, wN, uxtw 1: The rebase is native code generated after constant blinding has run on the BPF instruction stream, so blinding never sees it and needs no special handling. The emitted count depends only on the kfunc model, so it is identical across JIT passes. bpf_jit_supports_arena_args() is not flipped yet; that happens when the struct_ops trampoline side is in place as well. Signed-off-by: Puranjay Mohan Reviewed-by: Xu Kuohai Link: https://lore.kernel.org/bpf/20260813190356.335181-4-puranjay@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi

bpf, arm64: Fix stack-passed arguments for indirect trampolines

2026-08-14T03:55:01+00:00

save_args() reads stack-passed arguments relative to FP assuming the trampoline is entered through the fentry call from a traced function, in which case both the parent frame (FP/x9) and the traced function frame (FP/LR) are saved before FP is set, so the arguments start at FP + 32. An indirect trampoline for a struct_ops callback is entered through a function pointer (blr), so only the FP/LR frame is pushed and the arguments start at FP + 16, not FP + 32. Every stack-passed argument of a struct_ops callback with more than eight argument slots is read two slots off. This went unnoticed because no struct_ops member passed arguments on the stack until bpf_testmod_ops3::test_arena_stack, added by commit 2d4de9a493a0 ("selftests/bpf: Test stack-passed struct_ops arena arguments"). That member covers this on arm64 once the JIT gains arena argument support later in this series. Pass is_struct_ops into save_args() and pick the offset accordingly, mirroring the x86 fix. Fixes: 9014cf56f13d ("bpf, arm64: Support up to 12 function arguments") Signed-off-by: Puranjay Mohan Reviewed-by: Xu Kuohai Link: https://lore.kernel.org/bpf/20260813190356.335181-2-puranjay@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi

bpf, arm64: Clear fetch destination on faulting arena atomic

2026-08-12T17:33:53+00:00

Same problem as on x86-64: add_exception_handler() folds "there is no destination register to clear" and "this is a store" into one DONT_CLEAR value ... if (BPF_CLASS(insn->code) != BPF_LDX && !bpf_atomic_is_load_acq(insn)) dst_reg = DONT_CLEAR; ... which ex_handler_bpf() then reads back as the access direction: bool is_write = (dst_reg == DONT_CLEAR); A RMW carrying BPF_FETCH is both. emit_lse_atomic() reads the old value into src_reg for BPF_{ADD,AND,OR,XOR} | BPF_FETCH and BPF_XCHG, and into r0 for BPF_CMPXCHG, so a fault over an unmapped arena page is correctly reported as a WRITE but leaves that register holding a stale value instead of the 0 that every other BPF_PROBE_* access delivers. Same as on x86-64, add a separate ARENA_WRITE bit for the direction. FIXUP_REG is now filled in by the callers of add_exception_handler(), the BPF_PROBE_ATOMIC one deriving it from bpf_atomic_load_reg(), so that the helper only has to determine the direction. This is how the riscv64 JIT already does it, and it stops the two store callers from handing in a dst_reg that was only going to be overwritten with DONT_CLEAR anyway. Fixes: e612b5c1d3ee ("bpf, arm64: Add support for lse atomics in bpf_arena") Signed-off-by: Daniel Borkmann Reviewed-by: Puranjay Mohan Link: https://patch.msgid.link/20260811131600.506721-4-daniel@iogearbox.net Signed-off-by: Eduard Zingerman

bpf, arm64: Fix exception table metadata for arena load-acquire

2026-08-07T12:57:13+00:00

Same problem as on x86-64: add_exception_handler() decides whether an instruction is a load by its class, and a load-acquire is of BPF_STX class even though it reads from src_reg into dst_reg. As a result ... if (BPF_CLASS(insn->code) != BPF_LDX) dst_reg = DONT_CLEAR; ... drops the register to clear, and ... if (BPF_CLASS(insn->code) == BPF_LDX) arena_reg = bpf2a64[insn->src_reg]; else arena_reg = bpf2a64[insn->dst_reg]; ... hands ex_handler_bpf() the value register instead of the address register. A load-acquire from an arena pointer that faults on an unmapped page is therefore reported as a WRITE at a bogus address, and dst_reg keeps its previous value instead of being cleared to 0. Note that emit_atomic_ld_st() already picks src_reg as the address for BPF_LOAD_ACQ, so only the exception table metadata was out of sync with the emitted access. Same as on x86-64, use bpf_atomic_is_load_acq() so a load-acquire takes the load path. Fixes: 9bb12368d539 ("bpf, arm64: Support load-acquire and store-release instructions") Signed-off-by: Daniel Borkmann Reviewed-by: Puranjay Mohan Link: https://lore.kernel.org/bpf/20260806201047.333389-4-daniel@iogearbox.net Signed-off-by: Kumar Kartikeya Dwivedi

bpf: Extract the is_struct_ops_tramp helper

2026-07-21T18:17:45+00:00

Extract the is_struct_ops_tramp helper, and use it in riscv as the current checks are somewhat hacky. Signed-off-by: Pu Lehui Reviewed-by: Björn Töpel Acked-by: Björn Töpel Link: https://lore.kernel.org/bpf/20260708064436.2971933-2-pulehui@huaweicloud.com Signed-off-by: Kumar Kartikeya Dwivedi

bpf, arm64: Optimize cast_user code generation

2026-07-21T16:37:36+00:00

cast_user converts an arena offset into a user address by combining the low 32 bits of the pointer with the upper 32 bits of user_vm_start, while keeping a NULL pointer NULL. The current sequence always emits six instructions: it materializes user_vm_start >> 32 into a register, shifts it into place, and ORs in the offset. The upper half of user_vm_start is a constant, so it can be written directly onto the offset with MOVK. Move the 32-bit offset into dst (which also zeroes the upper 32 bits), then MOVK the non-zero halfwords of the upper address, branching over the MOVKs when the offset is zero so NULL is preserved. This emits at most four instructions, and only one when the upper half of user_vm_start is zero. The generated code is equivalent. Before: ; bpf_addr_space_cast(page1, 1, 0); 7c: mov w10, w8 80: mov w8, #1 84: lsl x8, x8, #32 88: cbz x10, 0xffff800087b80c20 8c: orr x10, x8, x10 90: mov x8, x10 After: ; bpf_addr_space_cast(page1, 1, 0); 7c: mov w8, w8 80: cbz w8, 0xffff800087b80c28 84: movk x8, #1, lsl #32 Signed-off-by: Puranjay Mohan Acked-by: Xu Kuohai Link: https://lore.kernel.org/bpf/20260721105921.1070501-1-puranjay@kernel.org Signed-off-by: Kumar Kartikeya Dwivedi

bpf: Restrict JIT predictor flush to cBPF

2026-07-01T08:33:38+00:00

Currently predictor flush on memory reuse is done for all BPF JIT allocations, but only cBPF programs can be loaded by an unprivileged user. eBPF is privileged by default, and flushing predictors for all CPUs on every eBPF reuse penalizes the common case for no security benefit. eBPF allocations can be frequent on busy systems, only flush predictors for cBPF programs. Trampoline and dispatcher allocations also skip the flush as they are eBPF-only. Signed-off-by: Pawan Gupta Acked-by: Daniel Borkmann Signed-off-by: Daniel Borkmann

bpf: Add struct bpf_tramp_node object

2026-06-07T17:03:01+00:00

Adding struct bpf_tramp_node to decouple the link out of the trampoline attachment info. At the moment the object for attaching bpf program to the trampoline is 'struct bpf_tramp_link': struct bpf_tramp_link { struct bpf_link link; struct hlist_node tramp_hlist; u64 cookie; } The link holds the bpf_prog pointer and forces one link - one program binding logic. In following changes we want to attach program to multiple trampolines but we want to keep just one bpf_link object. Splitting struct bpf_tramp_link into: struct bpf_tramp_link { struct bpf_link link; struct bpf_tramp_node node; }; struct bpf_tramp_node { struct bpf_link *link; struct hlist_node tramp_hlist; u64 cookie; }; The 'struct bpf_tramp_link' defines standard single trampoline link and 'struct bpf_tramp_node' is the attachment trampoline object with pointer to the bpf_link object. This will allow us to define link for multiple trampolines, like: struct bpf_tracing_multi_link { struct bpf_link link; ... int nodes_cnt; struct bpf_tracing_multi_node nodes[] __counted_by(nodes_cnt); }; Cc: Hengqi Chen Signed-off-by: Jiri Olsa Link: https://lore.kernel.org/r/20260606123955.345967-9-jolsa@kernel.org Signed-off-by: Alexei Starovoitov

bpf, arm64: Fix redundant MOV and clarify stack arg comments

2026-06-01T00:49:21+00:00

emit_stack_arg_store_imm() materializes the immediate into tmp and then moves tmp to the target register (x5-x7). Emit the immediate directly into the target register to avoid the redundant MOV. While here, qualify the bare "FP" in the stack-layout ASCII art as "A64_FP" so it is not confused with BPF_FP, and note that incoming stack arguments sit above the FP/LR pair pushed by the callee prologue. Suggested-by: Will Deacon Signed-off-by: Puranjay Mohan Link: https://lore.kernel.org/r/20260528161750.1900674-2-puranjay@kernel.org Signed-off-by: Alexei Starovoitov