From c7a2a3618290594867b4829900b434704ab31dbc Mon Sep 17 00:00:00 2001 From: "Mike Rapoport (Microsoft)" Date: Tue, 18 Aug 2026 16:05:10 +0300 Subject: x86/bpf: Make arch_bpf_trampoline_size allocate from EXECMEM_MODULE_DATA Jiri Olsa reports slowdown of tracing_multi benchmark that allocates huge number of trampolines [1]. The slowdown caused by extra protection changes in execmem_alloc_rw() and execmem_free(). With ROX caches enabled, all execmem allocations except EXECMEM_MODULE_DATA are ROX after the allocation. execmem_alloc_rw() temporarily sets them to W+NX and execmem_free() resets them back to ROX. The only user of bpf_jit_alloc_exec_rw() is x86::arch_bpf_trampoline_size() that only needs a temporary writable buffer in the modules address space. On x86 executable memory and module data are constrained to the same address range, so x86::arch_bpf_trampoline_size() can directly use execmem_alloc(EXECMEM_MODULE_DATA) Replace the call to bpf_jit_alloc_exec_rw() with a call to execmem_alloc(EXECMEM_MODULE_DATA) in x86::arch_bpf_trampoline_size() and drop bpf_jit_alloc_exec_rw() helper. Fixes: 5bf02dbf39fa ("bpf, x86: Make sure allocation in arch_bpf_trampoline_size() is writable") Reported-by: Jiri Olsa Signed-off-by: Mike Rapoport (Microsoft) Signed-off-by: Daniel Borkmann Tested-by: Jiri Olsa Link: https://lore.kernel.org/all/an8r7EODLIL-bZM3@krava Link: https://lore.kernel.org/bpf/20260818130510.3110054-1-rppt@kernel.org --- arch/x86/net/bpf_jit_comp.c | 8 +++++--- include/linux/filter.h | 1 - kernel/bpf/core.c | 5 ----- 3 files changed, 5 insertions(+), 9 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 1a9fb530adc3..2853e87797a7 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -3818,15 +3819,16 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, * * We cannot use kvmalloc here, because we need image to be in * module memory range. - * Since it must be writable use bpf_jit_alloc_exec_rw(). + * Since it must be writable use execmem_alloc(EXECMEM_MODULE_DATA) + * that returns writable memory in the module address space. */ - image = bpf_jit_alloc_exec_rw(PAGE_SIZE); + image = execmem_alloc(EXECMEM_MODULE_DATA, PAGE_SIZE); if (!image) return -ENOMEM; ret = __arch_prepare_bpf_trampoline(&im, image, image + PAGE_SIZE, image, m, flags, tnodes, func_addr); - bpf_jit_free_exec(image); + execmem_free(image); return ret; } diff --git a/include/linux/filter.h b/include/linux/filter.h index 4a9bc6a848f2..39decde7fc73 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -1376,7 +1376,6 @@ bpf_jit_binary_alloc(unsigned int proglen, u8 **image_ptr, void bpf_jit_binary_free(struct bpf_binary_header *hdr); u64 bpf_jit_alloc_exec_limit(void); void *bpf_jit_alloc_exec(unsigned long size); -void *bpf_jit_alloc_exec_rw(unsigned long size); void bpf_jit_free_exec(void *addr); void bpf_jit_free(struct bpf_prog *fp); struct bpf_binary_header * diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index d55e737ed75a..8b294dfc1ad4 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -1128,11 +1128,6 @@ void *bpf_jit_alloc_exec(unsigned long size) return execmem_alloc(EXECMEM_BPF, size); } -void *bpf_jit_alloc_exec_rw(unsigned long size) -{ - return execmem_alloc_rw(EXECMEM_BPF, size); -} - void bpf_jit_free_exec(void *addr) { execmem_free(addr); -- cgit v1.2.3 From 37e5c4f4d2856290b1c56e573ced91dcd88db8ec Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 20 Aug 2026 04:20:18 +0200 Subject: bpf: Reject invalid LDSX instruction in disassembly The signed-load mnemonic table has entries for byte, half-word, and word loads because BPF_MEMSX does not support double-word loads. A BPF_MEMSX | BPF_DW instruction nevertheless selects index 3, past the end of this table. Program Structure diagnostics can disassemble a malformed instruction before check_and_resolve_insns() rejects its opcode. Placing the invalid signed double-word load at the end of a program therefore triggers an out-of-bounds access while reporting subprogram fallthrough. Treat signed double-word loads as invalid in the disassembler and use the existing BUG_ldx fallback instead. Fixes: a8f427835394 ("bpf: Report Program Structure CFG errors") Reported-by: syzbot+3544d9b2a9206be8ba37@syzkaller.appspotmail.com Signed-off-by: Kumar Kartikeya Dwivedi Signed-off-by: Daniel Borkmann Reviewed-by: Jiayuan Chen Link: https://lore.kernel.org/bpf/20260820022020.3450479-2-memxor@gmail.com --- kernel/bpf/disasm.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/disasm.c b/kernel/bpf/disasm.c index 50b3ca5149a0..b1a3fbe3fda5 100644 --- a/kernel/bpf/disasm.c +++ b/kernel/bpf/disasm.c @@ -295,7 +295,8 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, verbose(cbs->private_data, "BUG_st_%02x", insn->code); } } else if (class == BPF_LDX) { - if (BPF_MODE(insn->code) != BPF_MEM && BPF_MODE(insn->code) != BPF_MEMSX) { + if ((BPF_MODE(insn->code) != BPF_MEM && BPF_MODE(insn->code) != BPF_MEMSX) || + (BPF_MODE(insn->code) == BPF_MEMSX && BPF_SIZE(insn->code) == BPF_DW)) { verbose(cbs->private_data, "BUG_ldx_%02x", insn->code); return; } -- cgit v1.2.3 From 175a58668e2d5e96c571177fc7a0d8997bfbb205 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 20 Aug 2026 04:20:19 +0200 Subject: selftests/bpf: Test invalid DW LDSX diagnostics An invalid BPF_MEMSX | BPF_DW instruction can reach Program Structure diagnostics before opcode validation when placed at the end of a subprogram. Exercise this path and require the disassembler fallback so table bounds regressions are caught. Signed-off-by: Kumar Kartikeya Dwivedi Signed-off-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260820022020.3450479-3-memxor@gmail.com --- tools/testing/selftests/bpf/progs/verifier_cfg.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_cfg.c b/tools/testing/selftests/bpf/progs/verifier_cfg.c index c1f55e1d80a4..3c3bb03e8217 100644 --- a/tools/testing/selftests/bpf/progs/verifier_cfg.c +++ b/tools/testing/selftests/bpf/progs/verifier_cfg.c @@ -3,6 +3,7 @@ #include #include +#include "../../../include/linux/filter.h" #include "bpf_misc.h" SEC("socket") @@ -55,6 +56,19 @@ __naked void out_of_range_jump2(void) " ::: __clobber_all); } +SEC("socket") +__description("invalid DW LDSX instruction in diagnostics") +__failure __msg("BUG_ldx_99") +__log_level(2) +__naked void invalid_dw_ldsx(void) +{ + asm volatile (" \ + .8byte %[ldsx_dw]; \ +" : + : __imm_insn(ldsx_dw, BPF_RAW_INSN(BPF_LDX | BPF_MEMSX | BPF_DW, BPF_REG_0, BPF_REG_0, 0, 0)) + : __clobber_all); +} + SEC("socket") __description("loop (back-edge)") __failure __msg("unreachable insn 1") -- cgit v1.2.3 From 150aeba624e8b7cac51c39440d7e8e1fd11de9a0 Mon Sep 17 00:00:00 2001 From: Jiayuan Chen Date: Wed, 19 Aug 2026 20:58:29 +0800 Subject: bpf: Fix REG INVARIANTS VIOLATION on speculative pointer arithmetic Take the following unprivileged program as an example: r0 = bpf_map_lookup_elem(...) /* PTR_TO_MAP_VALUE, offset 0 */ ... 14: r0 += r1 /* r1 is a bounded scalar */ 15: r9 = r0 Loading it triggers a verifier warning from reg_bounds_sanity_check(): verifier bug: REG INVARIANTS VIOLATION (alu): const subreg tnum out of sync with range bounds r64={.base=0x0, .size=0x0} r32={.base=0x0, .size=0xffffffff} var_off=(0x0, 0x0) What happens: 1. Processing insn 14 (r0 += r1) in adjust_ptr_min_max_vals(), the new offset is computed into dst_reg's var_off and 32/64-bit ranges. 2. Because pointer registers do not track 32-bit subregister bounds, __mark_reg32_unbounded() first sets r32 to the full range; r32 is re-derived from the offset at the end of the function by reg_bounds_sync(). 3. On the unprivileged path, sanitize_ptr_alu() is called and, via sanitize_speculative_path() -> push_stack(), snapshots the current register state and schedules the next instruction (insn 15) to be verified directly as a speculative path. 4. That snapshot is taken between step 2 and the final reg_bounds_sync(): at this point dst_reg's var_off still holds the (const) original offset while r32 has just been blanked to the full range, i.e. the two are out of sync. When the speculative path later verifies insn 15 (r9 = r0), the inconsistent state reaches reg_bounds_sanity_check() and trips the warning. var_off and the 32-bit range must always be consistent. There are two ways to keep the snapshot consistent: 1. sync var_off and r32 before the snapshot so they match, or 2. leave r32 at its original (already consistent) value and blank it only after the snapshot. The whole point of sanitize_ptr_alu() is to insert a harmless masking sequence that keeps the access in bounds under speculation, so the state it snapshots should faithfully represent that. Take approach 2: move __mark_reg32_unbounded() to after sanitize_ptr_alu(), so the speculative snapshot keeps the pointer's original, consistent r32. The non-speculative path is unchanged: r32 is still blanked before the offset is applied and re-derived by reg_bounds_sync(). Fixes: 5f99f312bd3b ("bpf: add register bounds sanity checks and sanitization") Reported-by: Hiker Cl Closes: https://lore.kernel.org/bpf/CAGM=xGB1fJ9kT8XTitVo74B0WGqgjkoUHdLwzytwV0AyqeVApw@mail.gmail.com/ Signed-off-by: Jiayuan Chen Acked-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260819125840.286434-1-jiayuan.chen@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index e421ea2b80c3..5e37ca75e5c4 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -14560,9 +14560,6 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, struct bpf_insn return -EINVAL; } - /* pointer types do not carry 32-bit bounds at the moment. */ - __mark_reg32_unbounded(dst_reg); - if (sanitize_needed(opcode)) { ret = sanitize_ptr_alu(env, insn, ptr_reg, off_reg, dst_reg, &info, false); @@ -14570,6 +14567,14 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, struct bpf_insn return sanitize_err(env, insn, ret); } + /* + * Pointer types do not carry 32-bit bounds at the moment. Blank r32 + * only after sanitize_ptr_alu() may have snapshotted dst_reg into a + * speculative path: otherwise reg_bounds_sanity_check() might hit some + * constraints violations. + */ + __mark_reg32_unbounded(dst_reg); + switch (opcode) { case BPF_ADD: /* -- cgit v1.2.3 From 7ee2f20bf20ed59fb269a260c4c4aff1e67f1b7c Mon Sep 17 00:00:00 2001 From: Jiayuan Chen Date: Wed, 19 Aug 2026 20:58:30 +0800 Subject: selftests/bpf: Add reg-invariants test for speculative pointer arithmetic An unprivileged socket filter does variable pointer arithmetic on a PTR_TO_MAP_VALUE whose offset collapses to a constant. The Spectre-v1 speculative path used to snapshot the pointer with a const offset and an unbounded r32, which tripped reg_bounds_sanity_check() on the following register move. Mark the test __success_unpriv (the speculative path only runs unprivileged) and flag it BPF_F_TEST_REG_INVARIANTS so the invariant violation becomes a hard load failure. The unprivileged run fails without the verifier fix and passes with it: verifier_bounds/spec_ptr_alu_const_offset @unpriv:FAIL # without fix verifier_bounds/spec_ptr_alu_const_offset @unpriv:OK # with fix Signed-off-by: Jiayuan Chen Tested-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260819125840.286434-2-jiayuan.chen@linux.dev Signed-off-by: Kumar Kartikeya Dwivedi --- .../testing/selftests/bpf/progs/verifier_bounds.c | 41 ++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_bounds.c b/tools/testing/selftests/bpf/progs/verifier_bounds.c index 1a273e416fed..df8d5309657e 100644 --- a/tools/testing/selftests/bpf/progs/verifier_bounds.c +++ b/tools/testing/selftests/bpf/progs/verifier_bounds.c @@ -2267,6 +2267,47 @@ __naked void deduce64_from_32_wrapping_32bit(void) : __clobber_all); } +/* + * Unprivileged variable pointer arithmetic on a PTR_TO_MAP_VALUE whose + * offset collapses to a constant. The Spectre-v1 speculative path snapshots + * the pointer while its r32 has just been blanked but its offset not yet + * synced; the following register move used to trip reg_bounds_sanity_check() + * ("const subreg tnum out of sync with range bounds"). With + * BPF_F_TEST_REG_INVARIANTS that violation turns into a load failure, so the + * unprivileged program must still load. + */ +SEC("socket") +__success __success_unpriv +__flag(BPF_F_TEST_REG_INVARIANTS) +__naked void spec_ptr_alu_const_offset(void) +{ + asm volatile (" \ + call %[bpf_ktime_get_ns]; \ + *(u64*)(r10 - 16) = r0; \ + r1 = 0; \ + *(u64*)(r10 - 8) = r1; \ + r2 = r10; \ + r2 += -8; \ + r1 = %[map_hash_8b] ll; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto l0_%=; \ + r1 = *(u64*)(r10 - 16); \ + r2 = 0x40000000; \ + if r1 > r2 goto l0_%=; \ + if r1 s> 1 goto l0_%=; /* r1 in [0, 1] */ \ + r0 += r1; /* ptr += bounded scalar */ \ + r9 = r0; /* used to trip the warning */ \ + *(u8*)(r0 + 0) = r1; \ +l0_%=: r0 = 0; \ + exit; \ + " + : + : __imm(bpf_ktime_get_ns), + __imm(bpf_map_lookup_elem), + __imm_addr(map_hash_8b) + : __clobber_all); +} + /* Check that range_within() compares cnum ranges, not min/max projections. */ SEC("socket") __failure __msg("div by zero") -- cgit v1.2.3 From efebf6496685c93150df5bb0794363ae70c5f58a Mon Sep 17 00:00:00 2001 From: Hui Su Date: Fri, 7 Aug 2026 01:56:00 +0800 Subject: bpf: Fix infinite loop in pcpu_freelist push with one possible CPU __pcpu_freelist_push() can loop forever when only one CPU is possible and an NMI re-enters pcpu_freelist_push() while the interrupted context holds that CPU's freelist lock. After the current-CPU fast path fails, the fallback loop walks cpu_possible_mask while skipping the current CPU. With CONFIG_SMP=n, or when an SMP kernel is limited to one possible CPU with nr_cpus=1 or possible_cpus=1, there are no other possible CPUs to examine. The loop therefore makes no lock acquisition attempt and can never make progress. The following stack was observed on a UP system: NMI context: pcpu_freelist_push free_htab_elem htab_map_delete_elem [perf-event BPF program] __perf_event_overflow perf_event_nmi_handler exc_nmi Interrupted context: __pcpu_freelist_push pcpu_freelist_push free_htab_elem htab_map_delete_elem [raw_tp/sys_enter BPF program] __bpf_trace_sys_enter do_syscall_64 raw_res_spin_lock() detects the same-CPU recursive acquisition and returns -EDEADLK, but the subsequent fallback loop has no candidate head on a system with one possible CPU. Restore the extra fallback head that existed before the rqspinlock conversion. Keep the current-CPU fast path, then try the other possible CPUs and finally the extra head. The additional head lets a push, which cannot fail without losing a preallocated element, make progress when the only per-CPU head is held by the interrupted context. Also check the extra head from the pop path so that nodes placed there can be reused. Fixes: f2ac0e5d1c4d ("bpf: Convert percpu_freelist.c to rqspinlock") Signed-off-by: Hui Su Cc: stable@vger.kernel.org Link: https://lore.kernel.org/bpf/20260806175600.1993595-1-sh_def@163.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/percpu_freelist.c | 35 +++++++++++++++++++++++++++-------- kernel/bpf/percpu_freelist.h | 1 + 2 files changed, 28 insertions(+), 8 deletions(-) diff --git a/kernel/bpf/percpu_freelist.c b/kernel/bpf/percpu_freelist.c index 632762b57299..06ce588d13a3 100644 --- a/kernel/bpf/percpu_freelist.c +++ b/kernel/bpf/percpu_freelist.c @@ -17,6 +17,8 @@ int pcpu_freelist_init(struct pcpu_freelist *s) raw_res_spin_lock_init(&head->lock); head->first = NULL; } + raw_res_spin_lock_init(&s->extralist.lock); + s->extralist.first = NULL; return 0; } @@ -46,22 +48,28 @@ void __pcpu_freelist_push(struct pcpu_freelist *s, struct pcpu_freelist_node *node) { struct pcpu_freelist_head *head; - int cpu; + int cpu, this_cpu; if (___pcpu_freelist_push(this_cpu_ptr(s->freelist), node)) return; + this_cpu = raw_smp_processor_id(); while (true) { - for_each_cpu_wrap(cpu, cpu_possible_mask, raw_smp_processor_id()) { - if (cpu == raw_smp_processor_id()) + for_each_cpu_wrap(cpu, cpu_possible_mask, this_cpu) { + if (cpu == this_cpu) continue; + head = per_cpu_ptr(s->freelist, cpu); - if (raw_res_spin_lock(&head->lock)) - continue; - pcpu_freelist_push_node(head, node); - raw_res_spin_unlock(&head->lock); - return; + if (___pcpu_freelist_push(head, node)) + return; } + + /* + * Push cannot fail. Use the extra list when none of the + * per-CPU freelists can accept the node. + */ + if (___pcpu_freelist_push(&s->extralist, node)) + return; } } @@ -117,6 +125,17 @@ static struct pcpu_freelist_node *___pcpu_freelist_pop(struct pcpu_freelist *s) } raw_res_spin_unlock(&head->lock); } + + /* Per-CPU lists are empty or unavailable, try the extra list. */ + head = &s->extralist; + if (!READ_ONCE(head->first)) + return NULL; + if (raw_res_spin_lock(&head->lock)) + return NULL; + node = head->first; + if (node) + WRITE_ONCE(head->first, node->next); + raw_res_spin_unlock(&head->lock); return node; } diff --git a/kernel/bpf/percpu_freelist.h b/kernel/bpf/percpu_freelist.h index 914798b74967..980cf2884fd2 100644 --- a/kernel/bpf/percpu_freelist.h +++ b/kernel/bpf/percpu_freelist.h @@ -14,6 +14,7 @@ struct pcpu_freelist_head { struct pcpu_freelist { struct pcpu_freelist_head __percpu *freelist; + struct pcpu_freelist_head extralist; }; struct pcpu_freelist_node { -- cgit v1.2.3 From ed54bf564ac52699cf4def3d0c2125d493e756f9 Mon Sep 17 00:00:00 2001 From: Hui Su Date: Fri, 14 Aug 2026 00:09:00 +0800 Subject: bpf: Fix BPF_F_CPU validation for sparse CPU IDs BPF_F_CPU stores the target CPU ID in the upper 32 bits of the map operation flags. bpf_map_check_op_flags() currently compares that ID with num_possible_cpus(), which is the number of possible CPUs rather than a bound on CPU IDs. On an arm64 QEMU guest with a CPU device-tree hole, the possible CPU mask was 0,2-3. A userspace program using raw bpf() syscalls creates a BPF_MAP_TYPE_PERCPU_ARRAY and performs update and lookup operations for each CPU by setting BPF_F_CPU and the CPU ID in the flags. With the old check, CPU 1 is incorrectly accepted while valid CPU 3 is rejected with -ERANGE. The CPU 1 update then reaches the per-CPU map access path and triggers: Unable to handle kernel paging request at virtual address ... pc : __pi_memcpy_generic+0x5c/0x22c lr : bpf_percpu_array_update+0x2dc/0x2e8 Call trace: __pi_memcpy_generic bpf_map_update_value map_update_elem __sys_bpf Check the CPU ID against nr_cpu_ids and cpu_possible() instead. This rejects CPU IDs outside the valid range and CPUs absent from the possible mask, while allowing valid sparse CPU IDs. Fixes: 2b421662c788 ("bpf: Introduce BPF_F_CPU and BPF_F_ALL_CPUS flags") Signed-off-by: Hui Su Signed-off-by: Andrii Nakryiko Acked-by: Leon Hwang Link: https://lore.kernel.org/bpf/20260813160858.1042834-3-sh_def@163.com --- include/linux/bpf.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index ffa5626411ac..b7dbf3d9b5c0 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -4209,7 +4209,7 @@ static inline int bpf_map_check_op_flags(struct bpf_map *map, u64 flags, u64 all return -EINVAL; cpu = flags >> 32; - if ((flags & BPF_F_CPU) && cpu >= num_possible_cpus()) + if ((flags & BPF_F_CPU) && (cpu >= nr_cpu_ids || !cpu_possible(cpu))) return -ERANGE; } -- cgit v1.2.3 From 75b0a6db4300e4c2c9e97a0848deaa7acfb42fb7 Mon Sep 17 00:00:00 2001 From: Hui Su Date: Thu, 13 Aug 2026 23:51:33 +0800 Subject: bpf: Fix percpu map update indexing with sparse CPU IDs Per-CPU array, hash, and cgroup storage map updates without BPF_F_CPU or BPF_F_ALL_CPUS use a value buffer whose per-CPU slots are packed in possible-CPU order. The buffer is sized as: round_up(value_size, 8) * num_possible_cpus() The update paths iterate over possible CPUs, but use the logical CPU ID to calculate the source offset: value + size * cpu This only works when possible CPU IDs are contiguous starting at zero. For example, with a possible CPU mask of 0,2-3, the buffer contains three slots corresponding to CPUs 0, 2, and 3. CPU2 is therefore expected to use slot 1 and CPU3 slot 2. Instead, the current code uses slots 2 and 3 respectively, causing incorrect per-CPU values and an out-of-bounds read from the update buffer for CPU3. The corresponding lookup paths already use a dense offset while iterating over possible CPUs. Do the same for the array, hash, and cgroup storage update paths, advancing the source offset once for each possible CPU. BPF_F_ALL_CPUS continues to use the same value for every CPU. Fixes: 8eb76cb03f0f ("bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_array maps") Fixes: c6936161fd55 ("bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_hash and lru_percpu_hash maps") Fixes: 47c79f05aa0d ("bpf: Add BPF_F_CPU and BPF_F_ALL_CPUS flags support for percpu_cgroup_storage maps") Signed-off-by: Hui Su Signed-off-by: Andrii Nakryiko Acked-by: Leon Hwang Link: https://lore.kernel.org/bpf/20260813155131.1022745-3-sh_def@163.com --- kernel/bpf/arraymap.c | 5 +++-- kernel/bpf/hashtab.c | 5 +++-- kernel/bpf/local_storage.c | 5 +++-- 3 files changed, 9 insertions(+), 6 deletions(-) diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index ef315b168b29..0ce26b538075 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -436,7 +436,7 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, void __percpu *pptr; void *ptr, *val; u32 size; - int cpu; + int cpu, off = 0; if (unlikely((map_flags & BPF_F_LOCK) || (u32)map_flags > BPF_F_ALL_CPUS)) /* unknown flags */ @@ -468,9 +468,10 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, } for_each_possible_cpu(cpu) { ptr = per_cpu_ptr(pptr, cpu); - val = (map_flags & BPF_F_ALL_CPUS) ? value : value + size * cpu; + val = (map_flags & BPF_F_ALL_CPUS) ? value : value + off; copy_map_value(map, ptr, val); bpf_obj_cancel_fields(map, ptr); + off += size; } unlock: rcu_read_unlock(); diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index d40cb5dd446c..d8db1cebc193 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -1025,7 +1025,7 @@ static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr, } else { u32 size = round_up(htab->map.value_size, 8); void *val; - int cpu; + int cpu, off = 0; if (map_flags & BPF_F_CPU) { cpu = map_flags >> 32; @@ -1037,9 +1037,10 @@ static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr, for_each_possible_cpu(cpu) { ptr = per_cpu_ptr(pptr, cpu); - val = (map_flags & BPF_F_ALL_CPUS) ? value : value + size * cpu; + val = (map_flags & BPF_F_ALL_CPUS) ? value : value + off; copy_map_value(&htab->map, ptr, val); bpf_obj_cancel_fields(&htab->map, ptr); + off += size; } } } diff --git a/kernel/bpf/local_storage.c b/kernel/bpf/local_storage.c index 23267213a17f..83cd527a2542 100644 --- a/kernel/bpf/local_storage.c +++ b/kernel/bpf/local_storage.c @@ -220,7 +220,7 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *_map, void *key, struct bpf_cgroup_storage *storage; void *val; u32 size; - int cpu; + int cpu, off = 0; if ((u32)map_flags & ~(BPF_ANY | BPF_EXIST | BPF_F_CPU | BPF_F_ALL_CPUS)) return -EINVAL; @@ -245,8 +245,9 @@ int bpf_percpu_cgroup_storage_update(struct bpf_map *_map, void *key, } size = round_up(_map->value_size, 8); for_each_possible_cpu(cpu) { - val = (map_flags & BPF_F_ALL_CPUS) ? value : value + size * cpu; + val = (map_flags & BPF_F_ALL_CPUS) ? value : value + off; copy_map_value(_map, per_cpu_ptr(storage->percpu_buf, cpu), val); + off += size; } unlock: rcu_read_unlock(); -- cgit v1.2.3 From d3ef6c097ba078e1f8c7239d76a0ce8b61e75095 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Wed, 26 Aug 2026 11:18:44 -0700 Subject: bpf: check_cond_jmp_op(): properly infer if register is null Nicholas Carlini reported a bug when verifier can incorrectly infer that a pointer is non-null. The bug occurs when two pointers are compared and one of them has a type w/o PTR_MAYBE_NULL flag, but which allows a value to be NULL at runtime. Here is an example: // `a` is PTR_TO_MEM | MEM_RDONLY | PTR_UNTRUSTED // `a` is 0 at runtime. // `b` is PTR_TO_MAP_VALUE | PTR_MAYBE_NULL void *a = bpf_rdonly_cast(0, 0); int *b = bpf_map_lookup_elem(...); if (a == b) *b = 42; // verifier does not catch null pointer dereference This happens because of a special case in check_cond_jmp_op(), which attempts to strip PTR_MAYBE_NULL flags from pointer types, when processing comparisons like `rA == rB`, if either rA or rB can't be null. The non-null property is derived based on the absence of PTR_MAYBE_NULL flag on rA's or rB's type. But that is not sufficient for types like PTR_TO_MEM, as in the example. This patch replaces type_may_be_null() call with reg_not_null(), which contains an allowlist of types for which absence of PTR_MAYBE_NULL actually means that the value can't be NULL at runtime. At the moment, the list in the reg_not_null() omits two types for which PTR_MAYBE_NULL is applicable: PTR_TO_XDP_SOCK and PTR_TO_BUF. In order to remain backward compatible, and assuming that only comparison between pointers of the same type makes sense, this commit extends reg_not_null(). W/o such an extension e.g. verifier_jeq_infer_not_null/null_ptr_to_map_value fails. reg_not_null() can be extended further, but I deem that out of scope for the fix at hand. Explicit base_type(...) != PTR_TO_BTF_ID checks in the check_cond_jmp_op() can be removed with migration to reg_not_null(), but that is a behavioural change, as the special case would start matching for PTR_TO_BTF_ID that is also is_trusted_reg(). I omit the behavioural change from this commit. Fixes: befae75856ab ("bpf: propagate nullness information for reg to reg comparisons") Suggested-by: Nicholas Carlini Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260826-bug-029-bad-non-null-inference-v2-1-136789ace9e9@localhost Signed-off-by: Alexei Starovoitov --- kernel/bpf/verifier.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 5e37ca75e5c4..e64035683795 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -355,6 +355,8 @@ static bool reg_not_null(struct bpf_verifier_env *env, const struct bpf_reg_stat type = base_type(type); return type == PTR_TO_SOCKET || type == PTR_TO_TCP_SOCK || + type == PTR_TO_XDP_SOCK || + type == PTR_TO_BUF || type == PTR_TO_MAP_VALUE || type == PTR_TO_MAP_KEY || type == PTR_TO_SOCK_COMMON || @@ -16968,7 +16970,6 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env, */ if (!is_jmp32 && BPF_SRC(insn->code) == BPF_X && __is_pointer_value(false, src_reg) && __is_pointer_value(false, dst_reg) && - type_may_be_null(src_reg->type) != type_may_be_null(dst_reg->type) && base_type(src_reg->type) != PTR_TO_BTF_ID && base_type(dst_reg->type) != PTR_TO_BTF_ID) { eq_branch_regs = NULL; @@ -16984,9 +16985,11 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env, break; } if (eq_branch_regs) { - if (type_may_be_null(src_reg->type)) + /* src == dst && dst != NULL => src != NULL */ + if (reg_not_null(env, dst_reg) && type_may_be_null(src_reg->type)) mark_ptr_not_null_reg(&eq_branch_regs[insn->src_reg]); - else + /* src == dst && src != NULL => dst != NULL */ + if (reg_not_null(env, src_reg) && type_may_be_null(dst_reg->type)) mark_ptr_not_null_reg(&eq_branch_regs[insn->dst_reg]); } } -- cgit v1.2.3 From ce6dcd0aed185432d02cafc82b738318af257ccd Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Wed, 26 Aug 2026 11:18:45 -0700 Subject: selftests/bpf: a demo for check_cond_jmp_op() non-null inference bug A comparison between PTR_TO_MEM | MEM_RDONLY | PTR_UNTRUSTED and PTR_TO_MAP_VALUE_OR_NULL should not infer that map pointer is not null. A bug in check_cond_jmp_op() made such inference possible. Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260826-bug-029-bad-non-null-inference-v2-2-136789ace9e9@localhost Signed-off-by: Alexei Starovoitov --- .../bpf/progs/verifier_jeq_infer_not_null.c | 52 ++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_jeq_infer_not_null.c b/tools/testing/selftests/bpf/progs/verifier_jeq_infer_not_null.c index 3d1e8de4390c..b412a542ef76 100644 --- a/tools/testing/selftests/bpf/progs/verifier_jeq_infer_not_null.c +++ b/tools/testing/selftests/bpf/progs/verifier_jeq_infer_not_null.c @@ -3,7 +3,9 @@ #include #include +#include #include "bpf_misc.h" +#include "bpf_kfuncs.h" struct { __uint(type, BPF_MAP_TYPE_XSKMAP); @@ -12,6 +14,13 @@ struct { __type(value, int); } map_xskmap SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_HASH); + __uint(max_entries, 1); + __type(key, int); + __type(value, int); +} map_hash SEC(".maps"); + /* This is equivalent to the following program: * * r6 = skb->sk; @@ -264,4 +273,47 @@ __naked void jne_reg_reg_null_check(void) : __clobber_all); } +/* + * A comparison between PTR_TO_MEM | MEM_RDONLY | PTR_UNTRUSTED and + * PTR_TO_MAP_VALUE_OR_NULL should not infer that map pointer is not null. + * A bug in check_cond_jmp_op() made such inference possible. + */ +SEC("raw_tp") +__failure +__msg("error: invalid dereference of R0 (a nullable map value pointer)") +__msg(">>> 11 | (61) r0 = *(u32 *)(r0 +0)") +__naked void untrusted_mem_does_not_infer_map_value_non_null(void) +{ + asm volatile (" \ + /* r6 = bpf_rdonly_cast(0, 0); */ \ + r1 = 0; \ + r2 = 0; \ + call %[bpf_rdonly_cast]; \ + r6 = r0; \ + /* r0 = bpf_map_lookup_elem(map_hash, &key); */ \ + *(u64 *)(r10 - 8) = 0; \ + r1 = %[map_hash] ll; \ + r2 = r10; \ + r2 += -8; \ + call %[bpf_map_lookup_elem]; \ + /* \ + * buggy verifier assumed that r6 can't be null \ + * and marked r0 non-null as well. \ + */ \ + if r6 != r0 goto 1f; \ + r0 = *(u32 *)(r0 + 0); \ +1: r0 = 0; \ + exit; \ +" : + : __imm(bpf_rdonly_cast), + __imm(bpf_map_lookup_elem), + __imm_addr(map_hash) + : __clobber_all); +} + +void kfunc_root(void) +{ + bpf_rdonly_cast(0, 0); +} + char _license[] SEC("license") = "GPL"; -- cgit v1.2.3 From 2f3536bff8823d3c5fdbbe15e17bfca696cc2b2e Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Thu, 27 Aug 2026 15:48:23 -0700 Subject: bpf: don't downgrade half-dead scalar zero spills to STACK_ZERO states.c:__clean_func_state() can downgrade scalar zero spill to STACK_ZERO in the following case: *(u64 *)(r10 - 8) = 0; ... checkpoint ... r1 = *(u32 *)(r10 - 4); ... no reads from r10-8 ... Here 4 bytes at r10-8 are dead and verifier changes scalar spill to a combination: 0000pppp (p stands for poison). Such a change breaks precision propagation chains. All places that produce STACK_ZERO should call bpf_mark_chain_precision() for the zero source. This patch fixes the bug in a simplest way possible: avoids converting stack spills of zero to STACK_ZERO. Two smarter approaches are possible: - do bpf_mark_chain_precision() from __clean_func_state() - check slot liveness information in check_stack_write_fixed_off() I investigated both and the changes required are a bit tricky, hence go with a simple fix for the time being. Fixes: be23266b4a08 ("bpf: 4-byte precise clean_verifier_state") Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260827-bug-011-cleanfunc-stack-zero-simple-v1-v1-1-c0e996589a52@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/states.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/kernel/bpf/states.c b/kernel/bpf/states.c index 4e6aafad33bd..66fb11b6c6a7 100644 --- a/kernel/bpf/states.c +++ b/kernel/bpf/states.c @@ -445,22 +445,19 @@ static void __clean_func_state(struct bpf_verifier_env *env, struct bpf_reg_state *spill = &st->stack[i].spilled_ptr; if (lo_live && stype == STACK_SPILL) { - u8 val = STACK_MISC; - if (spill->type != SCALAR_VALUE) continue; - /* - * 8 byte spill of scalar 0 where half slot is dead - * should become STACK_ZERO in lo 4 bytes. + * Can't replace with STACK_ZERO, because + * that requires bpf_mark_chain_precision(). */ if (bpf_register_is_null(spill)) - val = STACK_ZERO; + continue; for (j = 0; j < 4; j++) { u8 *t = &st->stack[i].slot_type[j]; if (*t == STACK_SPILL) - *t = val; + *t = STACK_MISC; } } bpf_mark_reg_not_init(env, spill); -- cgit v1.2.3 From c6ff14f1cd9e9b7d5631882ff509fbc29e90cfe0 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Thu, 27 Aug 2026 15:48:24 -0700 Subject: selftests/bpf: half-dead scalar zero stack spill test A test case demonstrating unsafe pruning when spill of a scalar zero spilled on a first pass in replaced by STACK_ZERO in the __clean_func_state(). Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260827-bug-011-cleanfunc-stack-zero-simple-v1-v1-2-c0e996589a52@gmail.com Signed-off-by: Alexei Starovoitov --- .../selftests/bpf/progs/verifier_spill_fill.c | 40 ++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_spill_fill.c b/tools/testing/selftests/bpf/progs/verifier_spill_fill.c index 8b166c42c4e0..39a1766dae3f 100644 --- a/tools/testing/selftests/bpf/progs/verifier_spill_fill.c +++ b/tools/testing/selftests/bpf/progs/verifier_spill_fill.c @@ -1403,6 +1403,46 @@ __naked void partial_fill_from_cleaned_pointer_spill(void) ::: __clobber_all); } +SEC("raw_tp") +__failure +__msg("access may be outside object bounds") +__flag(BPF_F_TEST_STATE_FREQ) +__naked void imprecise_scalar_spill_half_dead(void) +{ + asm volatile ( + /* + * Fork two paths: the one explored first spills an imprecise zero, + * the one explored second, an imprecise non-zero scalar. + */ + "call %[bpf_get_prandom_u32];" + "if r0 > 42 goto 1f;" + "r6 = 0;" + "goto 2f;" +"1:" + /* causes out of bounds access on a second path. */ + "r6 = 100500;" +"2:" + /* Force a checkpoint before the spill. */ + "goto +0;" + "*(u64 *)(r10 - 8) = r6;" + /* + * Force stack cleanup, only the low half of the spill is alive, + * so the dead high half is degraded to raw stack bytes. + * Buggy verifier converted it to STACK_ZERO w/o proper precision propagation. + */ + "goto +0;" + "r7 = *(u32 *)(r10 - 4);" + /* Use r7 as an offset into a one-byte buffer. */ + "r1 = %[single_byte_buf] ll;" + "r1 += r7;" + "r0 = *(u8 *)(r1 + 0);" + "exit;" +: +: __imm(bpf_get_prandom_u32), + __imm_addr(single_byte_buf) +: __clobber_all); +} + /* check valid spill/fill, ptr to tp buffer */ SEC("raw_tracepoint.w") __success -- cgit v1.2.3 From 28d75dd3eb60812b3a87cbdf0d52c42f51b28a78 Mon Sep 17 00:00:00 2001 From: Yonghong Song Date: Fri, 28 Aug 2026 10:05:34 -0700 Subject: selftests/bpf: Bound the offset accumulator in __tld_fetch_key() The LLVM commit c7f4a76da024 [1] "[InstCombine] fold ((x - 1) | (2^k - 1)) + 1 to (x + (2^k - 1)) & -(2^k)" caused test_task_local_data.bpf.o to fail verification: The sequence of 8193 jumps is too complex. processed 188770 insns (limit 1000000) max_states_per_insn 34 total_states 8238 peak_states 12330 mark_read 0 TLD_ROUND_UP(x, 8) expands to ((((x) - 1) | 7) + 1), exactly the pattern that [1] rewrites, so the accumulation in __tld_fetch_key() off += TLD_ROUND_UP(metadata[i].size, 8); is now compiled as (x + 7) & -8 instead of ((x - 1) | 7) + 1. Both are correct, but they leave the verifier in very different states. Note that 'off' is marked as precise. Without [1], "size - 1" wraps at zero (size is a __u16), so the verifier loses all bounds on the increment: 211: (69) r1 = *(u16 *)(r1 +62) ; R1=scalar(...,umax32=0xffff,var_off=(0x0; 0xffff)) 212: (04) w1 += -1 ; R1=scalar(smin=0,smax=umax=0xffffffff,smin32=-1,smax32=0xfffe,var_off=(0x0; 0xffffffff)) 213: (44) w1 |= 7 ; R1=scalar(smin=umin=umin32=7,smax=umax=0xffffffff,var_off=(0x7; 0xfffffff8)) 214: (0c) w6 += w1 ; R6=scalar(smin=umin=umin32=7,smax=umax=0xffffffff,var_off=(0x7; 0xfffffff8)) 215: (04) w6 += 1 ; R6=scalar(smin=0,smax=umax=umax32=0xfffffff8,var_off=(0x0; 0xfffffff8)) Note that 'w6' will be used in the next iteration. In the next iteration after insn 215, the R6 range will be the same as previous iteration. The iterator loop converges at depth 2. With [1] the increment stays precisely bounded at [0, 0x10006]: 211: (69) r9 = *(u16 *)(r1 +62) ; R9=scalar(...,umax32=0xffff,var_off=(0x0; 0xffff)) 212: (04) w9 += 7 ; R9=scalar(...,umax32=0x10006,var_off=(0x0; 0x1ffff)) 213: (54) w9 &= 131064 ; R9=scalar(...,umax32=0x10006,var_off=(0x0; 0x1fff8)) 214: (0c) w9 += w6 ; R9=scalar(...,umax32=0x10006,var_off=(0x0; 0x1fff8)) 215: (bf) r1 = r10 216: (07) r1 += -8 217: (85) call bpf_iter_num_next 218: (bc) w6 = w9 In the next iteration, we will have 211: (69) r9 = *(u16 *)(r1 +62) ; R9=scalar(...,umax32=0xffff,var_off=(0x0; 0xffff)) 212: (04) w9 += 7 ; R9=scalar(...,umax32=0x10006,var_off=(0x0; 0x1ffff)) 213: (54) w9 &= 131064 ; R9=scalar(...,umax32=0x10006,var_off=(0x0; 0x1fff8)) 214: (0c) w9 += w6 ; R9=scalar(...,umax32=0x2000c,var_off=(0x0; 0x3fff8)) ... so 'off' umax grows by 0x10006 on every iteration and the loop-head state never repeats: 218: (bc) w6 = w9 ; R6=scalar(...,umax32=0x10006,var_off=(0x0; 0x1fff8)) 218: (bc) w6 = w9 ; R6=scalar(...,umax32=0x2000c,var_off=(0x0; 0x3fff8)) 218: (bc) w6 = w9 ; R6=scalar(...,umax32=0x30012,var_off=(0x0; 0x3fff8)) ... 218: (bc) w6 = w9 ; R6=scalar(...,umax32=0xff95fd6,var_off=(0x0; 0xffffff8)) That last one is iterator depth 4090. Saturating umax would take ~65531 iterations; the verifier gives up long before that. Note the loop does not diverge from the start. widen_imprecise_scalars() blows 'off' up to an unbounded scalar while it is still imprecise, and that alone converges the first three passes through the loop at depth 4. Once mark_chain_precision() reaches the loop body, maybe_widen_reg() starts skipping the register, and no widening ever happens again. In the failing log widening fires exactly 6 times out of 4098 arrivals at the iter_next() checkpoint, all of them before the umax starts accumulating. With [1] and this fix, here is one full trip through the loop body, entered with 'off' (R6) already clamped by the previous iteration: 208: frame1: R6=scalar(...,umax32=4088,var_off=(0x0; 0xff8)) 208: (67) r7 <<= 6 ; R7=scalar(...,umax32=3968,var_off=(0x0; 0xfc0)) 209: (bf) r1 = r9 ; R1=mem(id=54,sz=4036,imm=4) 210: (0f) r1 += r7 211: (69) r1 = *(u16 *)(r1 +62) ; R1=scalar(...,umax32=0xffff,var_off=(0x0; 0xffff)) 212: (04) w1 += 7 ; R1=scalar(...,umax32=0x10006,var_off=(0x0; 0x1ffff)) 213: (54) w1 &= 131064 ; R1=scalar(...,umax32=0x10006,var_off=(0x0; 0x1fff8)) 214: (0c) w1 += w6 ; R1=scalar(...,umax32=0x10ffe,var_off=(0x0; 0x1fff8)) R6=scalar(...,umax32=4088,var_off=(0x0; 0xff8)) 215: (bc) w6 = w1 ; R6=scalar(...,umax32=0x10ffe,var_off=(0x0; 0x1fff8)) 216: (26) if w1 > 0xff8 goto pc+1 ; R6=scalar(...,umax32=4088,var_off=(0x0; 0xff8)) 217: (05) goto pc-27 This makes the loop body a fixpoint. 'off' (w6) enters at 208 as [0, 4088] with var_off=(0x0; 0xff8); the increment computed at 212/213 is [0, 0x10006], so 214/215 leave it at [0, 0x10ffe]; then 216 truncates it straight back to [0, 4088]/(0x0; 0xff8), and only then is the back edge at 217 taken. Convergence no longer depends on the widening window above. Verification converges at iterator depth 3. [1] https://github.com/llvm/llvm-project/pull/216436 Signed-off-by: Yonghong Song Link: https://lore.kernel.org/r/20260828170534.1011183-1-yonghong.song@linux.dev Signed-off-by: Alexei Starovoitov --- tools/testing/selftests/bpf/progs/task_local_data.bpf.h | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/task_local_data.bpf.h b/tools/testing/selftests/bpf/progs/task_local_data.bpf.h index 0df8a12fd61e..a31a399870be 100644 --- a/tools/testing/selftests/bpf/progs/task_local_data.bpf.h +++ b/tools/testing/selftests/bpf/progs/task_local_data.bpf.h @@ -61,6 +61,7 @@ #define TLD_ROUND_UP(x, y) ((((x) - 1) | TLD_ROUND_MASK(x, y)) + 1) #define TLD_MAX_DATA_CNT (__PAGE_SIZE / sizeof(struct tld_metadata) - 1) +#define TLD_DATA_SIZE (__PAGE_SIZE - sizeof(__u64)) #ifndef TLD_NAME_LEN #define TLD_NAME_LEN 62 @@ -189,6 +190,8 @@ static int __tld_fetch_key(struct tld_object *tld_obj, const char *name, int i_s return start + off; off += TLD_ROUND_UP(metadata[i].size, 8); + if (off > TLD_DATA_SIZE) + break; } return -cnt; -- cgit v1.2.3 From 387b1baefbb776e3f48dc2261e77a49213f470f7 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Wed, 2 Sep 2026 00:28:34 -0700 Subject: bpf: backtrack_insn(): Handle ld_{abs,ind} subprog exit edge Nicholas Carlini reported a bug in precision backtracking mechanism for BPF_LD | BPF_{IND,ABS} instructions. These instructions are modelled as two branches: - fallthrough; - implicit exit from current subprogram. The implicit exit case was not handled by the backtrack_insn() function. When backtracking such a path backtrack_insn() did not call bt_subprog_enter(), which meant that backtracking continued manipulating precision marks in a caller frame, while looking at instructions in a callee frame. This lead to segmentation faults during verification (see the selftest), or unsound state pruning. Fixes: ee861486e377 ("bpf: Fix ld_{abs,ind} failure path analysis in subprogs") Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Eduard Zingerman Signed-off-by: Daniel Borkmann Acked-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260901-bug-016-backtrack-ld-abs-v1-1-59368f1be435@gmail.com --- kernel/bpf/backtrack.c | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c index a2b18a9f1694..eaf7438b9ebf 100644 --- a/kernel/bpf/backtrack.c +++ b/kernel/bpf/backtrack.c @@ -582,16 +582,29 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, */ } } else if (class == BPF_LD) { - if (!bt_is_reg_set(bt, dreg)) - return 0; - bt_clear_reg(bt, dreg); /* It's ld_imm64 or ld_abs or ld_ind. * For ld_imm64 no further tracking of precision * into parent is necessary */ - if (mode == BPF_IND || mode == BPF_ABS) - /* to be analyzed */ - return -ENOTSUPP; + if (mode == BPF_IMM) { + bt_clear_reg(bt, dreg); + return 0; + } + /* + * BPF_{IND,ABS} are modelled as two branches: + * - fallthrough; + * - implicit subprogram exit. + * It is necessary to switch current frame if + * implicit subprogram exit branch is backtracked. + */ + if (mode == BPF_IND || mode == BPF_ABS) { + if (bt_is_reg_set(bt, dreg)) + return -ENOTSUPP; + if (subseq_idx != idx + 1) + if (bt_subprog_enter(bt)) + return -EFAULT; + return 0; + } } /* Propagate precision marks to linked registers, to account for * registers marked as precise in this function. -- cgit v1.2.3 From ce6b9e5dd873de532cd924e2abc928220cdc2738 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Wed, 2 Sep 2026 00:28:35 -0700 Subject: selftests/bpf: Precision tracking across BPF_ABS subprog exit A test case checking that the verifier properly backtracks both fallthrough and implicit subprogram exit paths modelled for BPF_LD | BPF_ABS instruction. Without the previous patch: - the verifier did not call bt_subprog_enter() on the implicit subprogram exit path; - bpf_pseudo_call() branch in backtrack_insn() executed 'bpf_bt_set_frame_reg(bt, bt->frame - 1, i);' with bt->frame == 0; - causing a segmentation fault. Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Eduard Zingerman Signed-off-by: Daniel Borkmann Acked-by: Daniel Borkmann Link: https://lore.kernel.org/bpf/20260901-bug-016-backtrack-ld-abs-v1-2-59368f1be435@gmail.com --- .../bpf/progs/verifier_subprog_precision.c | 51 ++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c b/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c index d21d32f6a676..e174a905c562 100644 --- a/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c +++ b/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c @@ -846,4 +846,55 @@ __naked int subprog_result_tail_call(void) ); } +__naked __noinline __used +static int ld_abs_subprog(void) +{ + asm volatile ( + "r6 = r1;" + "r7 = r1;" + ".8byte %[ld_abs];" + "exit;" + : + : __imm_insn(ld_abs, BPF_LD_ABS(BPF_W, 0)) + : __clobber_all); +} + +/* + * Buggy verifier did not properly backtrack early subprogram exit + * modelled for BPF_LD | BPF_ABS instruction, causing a segfault. + */ +SEC("socket") +__success +__log_level(2) +/* early exit path */ +__msg("3: (0f) r1 += r7") +__msg("mark_precise: frame0: regs=r7 stack= before 2: (bf) r1 = r10") +__msg("mark_precise: frame0: regs=r7 stack= before 9: (20) r0 = *(u32 *)skb[0]") +__msg("mark_precise: frame1: regs= stack= before 8: (bf) r7 = r1") +__msg("mark_precise: frame1: regs= stack= before 7: (bf) r6 = r1") +__msg("mark_precise: frame1: regs= stack= before 1: (85) call pc+5") +__msg("mark_precise: frame0: regs=r7 stack= before 0: (b7) r7 = -8") +/* fallthrough path */ +__msg("3: (0f) r1 += r7") +__msg("mark_precise: frame0: regs=r7 stack= before 2: (bf) r1 = r10") +__msg("mark_precise: frame0: regs=r7 stack= before 10: (95) exit") +__msg("mark_precise: frame1: regs= stack= before 9: (20) r0 = *(u32 *)skb[0]") +__msg("mark_precise: frame1: regs= stack= before 8: (bf) r7 = r1") +__msg("mark_precise: frame1: regs= stack= before 7: (bf) r6 = r1") +__msg("mark_precise: frame1: regs= stack= before 1: (85) call pc+5") +__msg("mark_precise: frame0: regs=r7 stack= before 0: (b7) r7 = -8") +__naked int ld_abs_backtrack_both_paths(void) +{ + asm volatile ( + "r7 = -8;" + "call ld_abs_subprog;" + "r1 = r10;" + "r1 += r7;" /* mark r7 as precise */ + "*(u64 *)(r1 + 0) = 0;" + "r0 = 0;" + "exit;" + ::: __clobber_all + ); +} + char _license[] SEC("license") = "GPL"; -- cgit v1.2.3 From e3e4f66cc4b72333d0886ae2673c360248987889 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Mon, 31 Aug 2026 18:36:09 -0700 Subject: bpf: backtracking shouldn't clear outer frame R1-R5 for callbacks When processing calls to bpf_loop() verifier marks R1 (and R4) as precise. R1 tracks loop iterations number and because of the 'callback_depth < R1' mechanics in check_helper_call() must be marked precise. However, precision propagation for R1 was broken, when bpf_loop() call was verified on a second iteration. Consider the following verification trace: - main: bpf_loop(nr_loops, callback ...) - callback: BPF_EXIT - main: bpf_loop(nr_loops, callback ...) - ... While the first visit of the call to bpf_loop() propagated R1 precision as expected, the second call to mark_chain_precision() in the check_helper_call() set R1, but it was immediately reset when backtrack_insn() processed preceding BPF_EXIT in the loop deleted in this patch. Because of that, the second visit of the call to bpf_loop() injected checkpoint with R1 not marked as precise. Which could trick the verifier into accepting unsafe programs. See the next patch for an example of such program. Commit is structured in a way to minimize conflicts when 'bpf' would be eventually merged with 'bpf-next'. Fixes: ab5cfac139ab ("bpf: verify callbacks as if they are called unknown number of times") Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260831-bug-015-backtrack-cb-args-precise-v1-1-68a8e2a821e0@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/backtrack.c | 37 +++++++++++++++++-------------------- 1 file changed, 17 insertions(+), 20 deletions(-) diff --git a/kernel/bpf/backtrack.c b/kernel/bpf/backtrack.c index eaf7438b9ebf..47282ffeeaf9 100644 --- a/kernel/bpf/backtrack.c +++ b/kernel/bpf/backtrack.c @@ -520,37 +520,34 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, return -EFAULT; } } else if (opcode == BPF_EXIT) { - bool r0_precise; + bool from_subprog_call, r0_precise; + + /* BPF_EXIT in subprog or callback always returns + * right after the call instruction, so by checking + * whether the instruction at subseq_idx-1 is subprog + * call or not we can distinguish actual exit from + * *subprog* from exit from *callback*. In the former + * case, we need to propagate r0 precision, if + * necessary. In the former we never do that. + */ + from_subprog_call = subseq_idx - 1 >= 0 && + bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]); + + r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0); /* Backtracking to a nested function call, 'idx' is a part of * the inner frame 'subseq_idx' is a part of the outer frame. * In case of a regular function call, instructions giving * precision to registers R1-R5 should have been found already. - * In case of a callback, it is ok to have R1-R5 marked for - * backtracking, as these registers are set by the function - * invoking callback. + * In case of a callback from bpf_loop(), R{1,4} in the calling + * frame would be set as precise and that is correct. */ - if (subseq_idx >= 0 && bpf_calls_callback(env, subseq_idx)) - for (i = BPF_REG_1; i <= BPF_REG_5; i++) - bt_clear_reg(bt, i); - if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) { + if (from_subprog_call && (bt_reg_mask(bt) & BPF_REGMASK_ARGS)) { verifier_bug(env, "backtracking exit unexpected regs %x", bt_reg_mask(bt)); return -EFAULT; } - /* BPF_EXIT in subprog or callback always returns - * right after the call instruction, so by checking - * whether the instruction at subseq_idx-1 is subprog - * call or not we can distinguish actual exit from - * *subprog* from exit from *callback*. In the former - * case, we need to propagate r0 precision, if - * necessary. In the former we never do that. - */ - r0_precise = subseq_idx - 1 >= 0 && - bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]) && - bt_is_reg_set(bt, BPF_REG_0); - bt_clear_reg(bt, BPF_REG_0); if (bt_subprog_enter(bt)) return -EFAULT; -- cgit v1.2.3 From 7ac9662189069914a088ec61ad85dc46b5cb1563 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Mon, 31 Aug 2026 18:36:10 -0700 Subject: selftests/bpf: test case for unsafe pruning of bpf_loop checkpoints The following BPF program was erroneously accepted by the verifier: static int cb(int i, __u64 *ctx) { /* unsafe on a second iteration */ small_arr[*ctx] = i; *ctx = 100500; return 0; } int main(void *ctx) { int nr_loops = 1; u64 ctx = 0; if (unlikely(bpf_get_prandom_u32() == 42)) nr_loops = 2; bpf_loop(nr_loops, cb, &ctx, 0); return 0; } The branch with nr_loops == 1 was explored first and injected a checkpoint at the entry to 'cb', such that nr_loops in the main's frame was not marked as precise. This checkpoint pruned the state with nr_loops == 2 and the program was accepted. This test case corresponds to the program above. Entry point is written in assembly to ensure branch processing order. Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260831-bug-015-backtrack-cb-args-precise-v1-2-68a8e2a821e0@gmail.com Signed-off-by: Alexei Starovoitov --- tools/testing/selftests/bpf/progs/iters.c | 39 +++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/iters.c b/tools/testing/selftests/bpf/progs/iters.c index 62d7df9e80be..c6699159dacd 100644 --- a/tools/testing/selftests/bpf/progs/iters.c +++ b/tools/testing/selftests/bpf/progs/iters.c @@ -2149,4 +2149,43 @@ __naked int stack_misc_vs_scalar_in_a_loop(void) ); } +__used +static int loop_cb5(int i, __u64 *ctx) +{ + /* unsafe on a second iteration */ + small_arr[*ctx] = i; + *ctx = 100500; + return 0; +} + +SEC("raw_tp") +__flag(BPF_F_TEST_STATE_FREQ) +__failure __msg("memory access is {{.*}} and is outside of the object of size 64") +__naked void loop_counter_precision_2nd_iter(void) +{ + asm volatile ( + "call %[bpf_get_prandom_u32];" + "*(u64 *)(r10 - 8) = 0;" + "r1 = 2;" + "if r0 == 42 goto +1;" + "r1 = 1;" + "r2 = loop_cb5 ll;" + "r3 = r10;" + "r3 += -8;" + "r4 = 0;" + /* + * Explore with nr_loops=1 on a first path and nr_loops=2 on a second path. + * Buggy verifier did not propagate r1 precision properly, + * and thus checkpoints created for nr_loops=1 case matched nr_loops=2 case. + */ + "call %[bpf_loop];" + "r0 = 0;" + "exit;" + : + : __imm(bpf_loop), + __imm(bpf_get_prandom_u32) + : __clobber_all + ); +} + char _license[] SEC("license") = "GPL"; -- cgit v1.2.3 From 374b2c5561db80fcdd7cdce44af37a49416f61c7 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Wed, 2 Sep 2026 16:36:57 -0700 Subject: bpf: reject BPF_PSEUDO_FUNC reference to the main program fixups.c:jit_subprogs() rewrites BPF_PSEUDO_FUNC loads to contain real function addresses. This function is invoked from bpf_jit_subprogs() only when env->subprog_cnt > 1. Meaning that for any program like below: int main(void *ctx) { void *ptr = main; ... bpf_timer_set_callback(..., ptr); ... } The 'ptr' won't be ever converted to contain an address. In combination with e.g. bpf_timer_set_callback() this would lead to a function call at a bogus address. Instead of complicating the implementation, just assume that no useful program needs main to be a sync or async callback and reject BPF_PSEUDO_FUNC loads for the main subprogram. Fixes: 69c087ba6225 ("bpf: Add bpf_for_each_map_elem() helper") Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260902233658.1186477-1-eddyz87@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/verifier.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index e64035683795..7d8ddb1bee00 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -17089,6 +17089,15 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn) verbose(env, "callback function not static\n"); return -EINVAL; } + /* + * When env->subprog_cnt == 1 this instruction won't be rewritten + * to hold a real function address. Assume that no usable program + * combines e.g. main and timer callback and just reject here. + */ + if (subprogno == 0) { + verbose(env, "callback function cannot be the main program\n"); + return -EINVAL; + } dst_reg->type = PTR_TO_FUNC; dst_reg->subprogno = subprogno; -- cgit v1.2.3 From ac0aaef0aa997fcdcb2458bd584539ba8608d33e Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Wed, 2 Sep 2026 16:36:58 -0700 Subject: selftests/bpf: BPF_PSEUDO_FUNC reference to the main program Add a test case for a BPF_PSEUDO_FUNC load instruction that references the entry function of the program it belongs to. W/o the previous patch the verifier accepts this program thus allowing a runtime call at a bogus address. See previous patch for detailed description. Main function needs to be marked with BTF_FUNC_STATIC for the test to trigger the bug, the patch uses test_verifier harness instead of test_prog because libbpf has no way to convey this. Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260902233658.1186477-2-eddyz87@gmail.com Signed-off-by: Alexei Starovoitov --- tools/testing/selftests/bpf/verifier/pseudo_func.c | 45 ++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 tools/testing/selftests/bpf/verifier/pseudo_func.c diff --git a/tools/testing/selftests/bpf/verifier/pseudo_func.c b/tools/testing/selftests/bpf/verifier/pseudo_func.c new file mode 100644 index 000000000000..63c5c67d51de --- /dev/null +++ b/tools/testing/selftests/bpf/verifier/pseudo_func.c @@ -0,0 +1,45 @@ +/* + * Buggy verifier accepted the program below while not patching BPF_PSEUDO_FUNC + * load instruction to contain a real address. Which resulted in a function call + * to a bogus address. + */ +{ + "BPF_PSEUDO_FUNC reference to the main program", + .insns = { + /* r6 = bpf_map_lookup_elem(&timer_map, &(int){0}); */ + BPF_ST_MEM(BPF_W, BPF_REG_10, -4, 0), + BPF_MOV64_REG(BPF_REG_2, BPF_REG_10), + BPF_ALU64_IMM(BPF_ADD, BPF_REG_2, -4), + BPF_LD_MAP_FD(BPF_REG_1, 0), + BPF_EMIT_CALL(BPF_FUNC_map_lookup_elem), + BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 10), + BPF_MOV64_REG(BPF_REG_6, BPF_REG_0), + /* bpf_timer_init(r6, &timer_map, 0); */ + BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), + BPF_LD_MAP_FD(BPF_REG_2, 0), + BPF_MOV64_IMM(BPF_REG_3, 0), + BPF_EMIT_CALL(BPF_FUNC_timer_init), + /* bpf_timer_set_callback(r6, ); */ + BPF_MOV64_REG(BPF_REG_1, BPF_REG_6), + BPF_RAW_INSN(BPF_LD | BPF_IMM | BPF_DW, BPF_REG_2, BPF_PSEUDO_FUNC, 0, -15), + BPF_RAW_INSN(0, 0, 0, 0, 0), + BPF_EMIT_CALL(BPF_FUNC_timer_set_callback), + BPF_MOV64_IMM(BPF_REG_0, 0), + BPF_EXIT_INSN(), + }, + .prog_type = BPF_PROG_TYPE_TRACEPOINT, + .fixup_map_timer = { 3, 9 }, + .result = REJECT, + .errstr = "callback function cannot be the main program", + .func_info = { { 0, 4 /* main_prog */ } }, + .func_info_cnt = 1, + .btf_strings = "\0int\0ctx\0main_prog", + .btf_types = { + /* 1: int */ BTF_TYPE_INT_ENC(1, BTF_INT_SIGNED, 0, 32, 4), + /* 2: void* */ BTF_PTR_ENC(0), + /* 3: int __(void *) */ BTF_FUNC_PROTO_ENC(1, 1), + BTF_FUNC_PROTO_ARG_ENC(5, 2), + /* 4: main_prog */ BTF_FUNC_ENC(9, 3), + BTF_END_RAW + } +}, -- cgit v1.2.3 From 0895a0c0734703be5532f3883c42db95615fd98b Mon Sep 17 00:00:00 2001 From: Jiayuan Chen Date: Tue, 1 Sep 2026 18:47:35 +0800 Subject: bpf: Reject key-less BTF for hash maps map_check_btf() allows a key-less BTF (btf_key_type_id == 0) only for maps that have a ->map_check_btf callback, and leaves the actual decision to that callback. Hash maps used to have no ->map_check_btf, so a key-less BTF was rejected outright. That changed when htab and rhtab gained a ->map_check_btf to register a dtor - htab in commit 1df97a7453ee ("bpf: Register dtor for freeing special fields") and rhtab in commit 6905f8601298 ("bpf: Allow special fields in resizable hashtab"). Neither looks at the key, so a key-less hash map now passes map_check_btf() and gets created. Reading it back through bpffs feeds the key type_id 0 into btf_type_seq_show(); btf_type_by_id() returns the void type, kind_ops[BTF_KIND_UNKN] is NULL, and btf_type_show() dereferences it: RIP: 0010:btf_type_show+0x223/0x2e0 kernel/bpf/btf.c:8232 RSP: 0018:ffffc9000399f868 EFLAGS: 00010206 RAX: dffffc0000000000 RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000000005 RSI: 0000000000000000 RDI: 0000000000000028 RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000 R10: ffffc9000399f970 R11: 0000000000000001 R12: ffffffff9b96b140 R13: ffffc9000399f8e0 R14: ffff88803d393c00 R15: 0000000000000003 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000200000000000 CR3: 000000003d213000 CR4: 0000000000352ef0 DR0: 0000000039ae8f55 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Call Trace: btf_type_seq_show_flags+0xca/0x120 kernel/bpf/btf.c:8250 htab_map_seq_show_elem+0x12e/0x350 kernel/bpf/hashtab.c:1669 map_seq_show+0x13d/0x1e0 kernel/bpf/inode.c:293 traverse.part.0.constprop.0+0x107/0x650 fs/seq_file.c:112 traverse fs/seq_file.c:99 [inline] seq_read_iter+0x93f/0x1270 fs/seq_file.c:196 seq_read+0x344/0x4d0 fs/seq_file.c:163 vfs_read+0x1e4/0xb40 fs/read_write.c:572 ksys_pread64 fs/read_write.c:764 [inline] __do_sys_pread64 fs/read_write.c:772 [inline] __se_sys_pread64 fs/read_write.c:769 [inline] __x64_sys_pread64+0x1eb/0x250 fs/read_write.c:769 do_syscall_x64 arch/x86/entry/syscall_64.c:61 [inline] do_syscall_64+0x123/0x790 arch/x86/entry/syscall_64.c:84 entry_SYSCALL_64_after_hwframe+0x77/0x7f Reject a key-less BTF in htab_map_check_btf() and rhtab_map_check_btf(), restoring the previous behavior. Fixes: 1df97a7453ee ("bpf: Register dtor for freeing special fields") Fixes: 6905f8601298 ("bpf: Allow special fields in resizable hashtab") Reported-by: syzbot+37b56485bbbf90ad8489@syzkaller.appspotmail.com Closes: https://lore.kernel.org/all/6a8f4e88.27659fcc.2ceef7.0008.GAE@google.com/T/ Signed-off-by: Jiayuan Chen Acked-by: Ihor Solodrai Link: https://lore.kernel.org/r/20260901104924.346187-2-jiayuan.chen@linux.dev Signed-off-by: Alexei Starovoitov --- kernel/bpf/hashtab.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index d8db1cebc193..e89fde188389 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -530,6 +530,9 @@ static int htab_map_check_btf(struct bpf_map *map, const struct btf *btf, { struct bpf_htab *htab = container_of(map, struct bpf_htab, map); + if (btf_type_is_void(key_type)) + return -EINVAL; + if (htab_is_prealloc(htab)) return 0; /* @@ -3111,6 +3114,9 @@ static int rhtab_map_check_btf(struct bpf_map *map, const struct btf *btf, { struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); + if (btf_type_is_void(key_type)) + return -EINVAL; + return bpf_ma_set_dtor(map, &rhtab->ma, rhtab_mem_dtor); } -- cgit v1.2.3 From 4ea508b9ebd78bce7f212166d2e2cba66b875f08 Mon Sep 17 00:00:00 2001 From: Jiayuan Chen Date: Tue, 1 Sep 2026 18:47:36 +0800 Subject: bpf: Fix NULL-ptr-deref when showing a void BTF type btf_modifier_show() resolves the modifier and then calls btf_type_ops(t)->show() unconditionally. For the void type (type_id 0, BTF_KIND_UNKN) kind_ops[] has no entry, so ->show is NULL. A "const void" (a modifier resolving to void) cannot be a map key or value - map_check_btf() rejects it because void has no size - so the map dump path does not reach it. But bpf_snprintf_btf() takes a type_id straight from the BPF program, and passing such a "const void" from the vmlinux BTF NULL-derefs: KASAN: null-ptr-deref in range [0x0000000000000028-0x000000000000002f] RIP: 0010:btf_modifier_show (kernel/bpf/btf.c:2914) Call Trace: btf_type_show (kernel/bpf/btf.c:8251) btf_type_snprintf_show (kernel/bpf/btf.c:8321) bpf_snprintf_btf (kernel/trace/bpf_trace.c:1047) bpf_prog_test_run_raw_tp (net/bpf/test_run.c:829) __sys_bpf (kernel/bpf/syscall.c:4804) do_syscall_64 (arch/x86/entry/syscall_64.c:94) entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:121) Fall back to btf_df_show() when the resolved type has no show op; it emits the "" placeholder already used for kinds like FWD and FUNC. bpf_snprintf_btf() then returns the length as usual. Fixes: c4d0bfb45068 ("bpf: Add bpf_snprintf_btf helper") Signed-off-by: Jiayuan Chen Acked-by: Ihor Solodrai Link: https://lore.kernel.org/r/20260901104924.346187-3-jiayuan.chen@linux.dev Signed-off-by: Alexei Starovoitov --- kernel/bpf/btf.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index da36d4b9d31a..df5f0d059ad1 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -2911,7 +2911,14 @@ static void btf_modifier_show(const struct btf *btf, else t = btf_type_skip_modifiers(btf, type_id, NULL); - btf_type_ops(t)->show(btf, t, type_id, data, bits_offset, show); + /* + * A modifier can resolve to void, which has no show op; print a + * placeholder rather than dereferencing NULL. + */ + if (!btf_type_ops(t)) + btf_df_show(btf, t, type_id, data, bits_offset, show); + else + btf_type_ops(t)->show(btf, t, type_id, data, bits_offset, show); } static void btf_var_show(const struct btf *btf, const struct btf_type *t, -- cgit v1.2.3 From 5403a383f52fc0905703b488f7c3db4b2447dc58 Mon Sep 17 00:00:00 2001 From: Jiayuan Chen Date: Tue, 1 Sep 2026 18:47:37 +0800 Subject: bpf: Fix NULL-ptr-deref in btf_var_show() btf_var_show() calls btf_type_id_resolve() unconditionally, which dereferences btf->resolved_ids. That is NULL for a base BTF - e.g. the vmlinux BTF that bpf_snprintf_btf() renders against - since base BTF is not resolved during parsing. btf_modifier_show() guards this with 'if (btf->resolved_ids)', but btf_var_show() does not. A BPF program that passes the type_id of a BTF_KIND_VAR from the vmlinux BTF to bpf_snprintf_btf() thus NULL-derefs: KASAN: probably user-memory-access in range [0x46638-0x4663f] RIP: 0010:btf_var_show (kernel/bpf/btf.c:2929) Call Trace: btf_type_show (kernel/bpf/btf.c:8259) btf_type_snprintf_show (kernel/bpf/btf.c:8329) bpf_snprintf_btf (kernel/trace/bpf_trace.c:1047) bpf_prog_test_run_raw_tp (net/bpf/test_run.c:829) __sys_bpf (kernel/bpf/syscall.c:4804) do_syscall_64 (arch/x86/entry/syscall_64.c:84) entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:121) Resolve the var's type directly with btf_type_skip_modifiers() when resolved_ids is NULL, mirroring btf_modifier_show(). Fixes: c4d0bfb45068 ("bpf: Add bpf_snprintf_btf helper") Signed-off-by: Jiayuan Chen Acked-by: Ihor Solodrai Link: https://lore.kernel.org/r/20260901104924.346187-4-jiayuan.chen@linux.dev Signed-off-by: Alexei Starovoitov --- kernel/bpf/btf.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index df5f0d059ad1..85ae92c920e4 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -2925,7 +2925,15 @@ static void btf_var_show(const struct btf *btf, const struct btf_type *t, u32 type_id, void *data, u8 bits_offset, struct btf_show *show) { - t = btf_type_id_resolve(btf, &type_id); + /* + * btf_type_id_resolve() dereferences btf->resolved_ids, which is NULL + * for a base BTF (e.g. the vmlinux BTF that bpf_snprintf_btf() uses). + * Resolve the var's type directly in that case. + */ + if (btf->resolved_ids) + t = btf_type_id_resolve(btf, &type_id); + else + t = btf_type_skip_modifiers(btf, t->type, &type_id); btf_type_ops(t)->show(btf, t, type_id, data, bits_offset, show); } -- cgit v1.2.3 From 6265b44f2c3bb2839a306d6088d6e65a58d7e80e Mon Sep 17 00:00:00 2001 From: Jiayuan Chen Date: Tue, 1 Sep 2026 18:47:38 +0800 Subject: selftests/bpf: Add test for key-less BTF hash map Create a hash and an rhash map with btf_key_type_id == 0 and expect bpf_map_create() to fail with -EINVAL; a positive control with a real key type confirms the rejection is about the key-less BTF and not some unrelated failure. Such a map used to be accepted and then NULL-deref in btf_type_show() when dumped through bpffs. Signed-off-by: Jiayuan Chen Link: https://lore.kernel.org/r/20260901104924.346187-5-jiayuan.chen@linux.dev Signed-off-by: Alexei Starovoitov --- .../selftests/bpf/prog_tests/btf_map_keyless.c | 59 ++++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/btf_map_keyless.c diff --git a/tools/testing/selftests/bpf/prog_tests/btf_map_keyless.c b/tools/testing/selftests/bpf/prog_tests/btf_map_keyless.c new file mode 100644 index 000000000000..3248bccc3557 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/btf_map_keyless.c @@ -0,0 +1,59 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include + +/* + * A hash map with a key-less BTF (btf_key_type_id == 0) used to be accepted + * and then NULL-deref in btf_type_show() when dumped through bpffs. A fixed + * kernel rejects it at creation; verify that rejection, with a keyed positive + * control so the -EINVAL is about the missing key type and not some unrelated + * failure. + */ +static void check_keyless(int map_type, __u32 map_flags, int btf_fd, int val_id) +{ + LIBBPF_OPTS(bpf_map_create_opts, opts); + int map_fd; + + opts.map_flags = map_flags; + opts.btf_fd = btf_fd; + opts.btf_value_type_id = val_id; + + /* Positive control: the same map with a real key type is accepted. */ + opts.btf_key_type_id = val_id; + map_fd = bpf_map_create(map_type, "keyed_map", 4, 4, 8, &opts); + if (!ASSERT_GE(map_fd, 0, "keyed create is accepted")) + return; + close(map_fd); + + /* A key-less BTF must be rejected. */ + opts.btf_key_type_id = 0; + map_fd = bpf_map_create(map_type, "keyless_map", 4, 4, 8, &opts); + ASSERT_EQ(map_fd, -EINVAL, "key-less create is rejected"); + if (map_fd >= 0) + close(map_fd); +} + +void test_btf_map_keyless(void) +{ + int btf_fd, val_id; + struct btf *btf; + + btf = btf__new_empty(); + if (!ASSERT_OK_PTR(btf, "btf__new_empty")) + return; + + val_id = btf__add_int(btf, "int", 4, BTF_INT_SIGNED); + if (!ASSERT_GT(val_id, 0, "btf__add_int")) + goto out; + + if (!ASSERT_OK(btf__load_into_kernel(btf), "btf__load_into_kernel")) + goto out; + btf_fd = btf__fd(btf); + + if (test__start_subtest("hash")) + check_keyless(BPF_MAP_TYPE_HASH, 0, btf_fd, val_id); + if (test__start_subtest("rhash")) + check_keyless(BPF_MAP_TYPE_RHASH, BPF_F_NO_PREALLOC, btf_fd, val_id); +out: + btf__free(btf); +} -- cgit v1.2.3 From 1ae6aa61958a0ee6f254cefbee20663ffbadb195 Mon Sep 17 00:00:00 2001 From: Jiayuan Chen Date: Tue, 1 Sep 2026 18:47:39 +0800 Subject: selftests/bpf: Add test for showing a void BTF type Extend the snprintf_btf test with type_ids from the vmlinux BTF that used to NULL-deref in the BTF show path: a "const void", checked to render the "" placeholder, and a BTF_KIND_VAR, checked to resolve and render without error. The program renders from its own buffer and the test picks a VAR whose resolved type fits it, so the render stays in bounds. Signed-off-by: Jiayuan Chen Link: https://lore.kernel.org/r/20260901104924.346187-6-jiayuan.chen@linux.dev Signed-off-by: Alexei Starovoitov --- .../selftests/bpf/prog_tests/snprintf_btf.c | 79 ++++++++++++++++++++++ .../selftests/bpf/progs/snprintf_btf_void.c | 24 +++++++ 2 files changed, 103 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/snprintf_btf_void.c diff --git a/tools/testing/selftests/bpf/prog_tests/snprintf_btf.c b/tools/testing/selftests/bpf/prog_tests/snprintf_btf.c index dd41b826be30..edce9c1b54fb 100644 --- a/tools/testing/selftests/bpf/prog_tests/snprintf_btf.c +++ b/tools/testing/selftests/bpf/prog_tests/snprintf_btf.c @@ -1,7 +1,9 @@ // SPDX-License-Identifier: GPL-2.0 #include #include +#include #include "netif_receive_skb.skel.h" +#include "snprintf_btf_void.skel.h" /* Demonstrate that bpf_snprintf_btf succeeds and that various data types * are formatted correctly. @@ -58,3 +60,80 @@ void serial_test_snprintf_btf(void) cleanup: netif_receive_skb__destroy(skel); } + +/* + * bpf_snprintf_btf() renders a type_id taken straight from the vmlinux BTF. + * Two such type_ids used to NULL-deref in the BTF show path: + * - a "const void" (a modifier resolving to void) in btf_modifier_show() + * - a BTF_KIND_VAR in btf_var_show() (base BTF has no resolved_ids) + * A fixed kernel renders both without crashing. + */ +static long run(struct snprintf_btf_void *skel, __u32 type_id) +{ + LIBBPF_OPTS(bpf_test_run_opts, topts); + char ctx[8] = {}; + + skel->bss->type_id = type_id; + topts.ctx_in = ctx; + topts.ctx_size_in = sizeof(ctx); + if (!ASSERT_OK(bpf_prog_test_run_opts(bpf_program__fd(skel->progs.dump_type), + &topts), "test_run")) + return -1; + return skel->bss->ret; +} + +void test_snprintf_btf_void(void) +{ + const struct btf_type *t; + struct snprintf_btf_void *skel; + int i, n, cv = 0, var = 0; + struct btf *btf; + + btf = btf__parse("/sys/kernel/btf/vmlinux", NULL); + if (!btf) { + test__skip(); + return; + } + + skel = snprintf_btf_void__open_and_load(); + if (!ASSERT_OK_PTR(skel, "skel_open_and_load")) + goto out_btf; + + n = btf__type_cnt(btf); + for (i = 1; i < n && !(cv && var); i++) { + t = btf__type_by_id(btf, i); + if (!cv && btf_kind(t) == BTF_KIND_CONST && t->type == 0) + cv = i; + /* Pick a VAR small enough to render from the program's buffer. */ + if (!var && btf_kind(t) == BTF_KIND_VAR) { + long sz = btf__resolve_size(btf, t->type); + + if (sz > 0 && sz <= (long)sizeof(skel->bss->obj)) + var = i; + } + } + + /* "const void" renders the "" placeholder. */ + if (test__start_subtest("const_void")) { + if (cv) { + ASSERT_EQ(run(skel, cv), + sizeof("") - 1, "ret"); + ASSERT_STREQ(skel->bss->out, "", + "placeholder"); + } else { + test__skip(); + } + } + + /* A BTF_KIND_VAR must resolve and render without error. */ + if (test__start_subtest("var")) { + if (var) + ASSERT_GT(run(skel, var), 0, "ret"); + else + test__skip(); + } + + snprintf_btf_void__destroy(skel); +out_btf: + btf__free(btf); +} diff --git a/tools/testing/selftests/bpf/progs/snprintf_btf_void.c b/tools/testing/selftests/bpf/progs/snprintf_btf_void.c new file mode 100644 index 000000000000..44af80fbbb80 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/snprintf_btf_void.c @@ -0,0 +1,24 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "btf_ptr.h" +#include + +__u32 type_id; +/* A buffer we own to render the selected type from, kept in bounds. */ +char obj[256]; +char out[64]; +long ret; + +SEC("raw_tp/sys_enter") +int dump_type(void *ctx) +{ + struct btf_ptr ptr = { + .ptr = obj, + .type_id = type_id, + .flags = 0, + }; + + ret = bpf_snprintf_btf(out, sizeof(out), &ptr, sizeof(ptr), 0); + return 0; +} + +char _license[] SEC("license") = "GPL"; -- cgit v1.2.3 From 77515ab12e4983e6416f8c35039a3f0c0822ac70 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 3 Sep 2026 16:44:19 +0200 Subject: bpf: Mark signal tracepoint siginfo arguments as scalar The signal_generate and signal_deliver tracepoints declare their info argument as a struct kernel_siginfo pointer. btf_ctx_access() therefore treats it as a trusted pointer for tp_btf programs. Signal delivery also uses SEND_SIG_NOINFO and SEND_SIG_PRIV as special values for this argument. Those values are zero and one respectively, and are not pointers. A tp_btf program can currently dereference either value and fault the kernel. In particular, signal_generate can run from timer interrupt context, turning the fault into a kernel panic. Record both tracepoints in raw_tp_null_args[] and mark argument one as a non-pointer. This preserves scalar access to the cookie while rejecting direct and helper-mediated pointer use. Merely marking it nullable would not suffice because SEND_SIG_PRIV is nonzero. Fixes: 838a10bd2ebf ("bpf: Augment raw_tp arguments with PTR_MAYBE_NULL") Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260903144433.1716731-2-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/btf.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 85ae92c920e4..b5aa802451bd 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -6732,6 +6732,9 @@ static const struct bpf_raw_tp_null_args raw_tp_null_args[] = { { "rxrpc_resend", 0x10 }, { "rxrpc_tq", 0x10 }, { "rxrpc_client", 0x1 }, + /* signal */ + { "signal_generate", 0x20 }, + { "signal_deliver", 0x20 }, /* skb */ {"kfree_skb", 0x1000}, /* sunrpc */ -- cgit v1.2.3 From d7719a1736e6be77d0682f7395acd3701949f1fa Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 3 Sep 2026 16:44:20 +0200 Subject: selftests/bpf: Cover signal tracepoint siginfo sentinels Add load-only verifier coverage for the signal_generate and signal_deliver info arguments. The signal_generate case performs a NULL check before dereferencing info, ensuring that merely making it nullable cannot satisfy the test when the nonzero SEND_SIG_PRIV sentinel is used. Both programs load successfully without the verifier fix, contrary to their expected-failure annotations. With the fix, info is a scalar and the attempted dereferences are rejected. Also add success cases showing that plain raw tracepoint and tp_btf programs can continue to read and compare the context word as a scalar. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260903144433.1716731-3-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- .../testing/selftests/bpf/progs/raw_tp_null_fail.c | 36 ++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/raw_tp_null_fail.c b/tools/testing/selftests/bpf/progs/raw_tp_null_fail.c index 0d58114a4955..7e8842bf9000 100644 --- a/tools/testing/selftests/bpf/progs/raw_tp_null_fail.c +++ b/tools/testing/selftests/bpf/progs/raw_tp_null_fail.c @@ -22,3 +22,39 @@ int test_raw_tp_null_sched_pi_setprio_arg_2(void *ctx) { asm volatile("r1 = *(u64 *)(r1 +8); r1 = *(u64 *)(r1 +0);" ::: __clobber_all); return 0; } + +/* Plain raw tracepoint arguments remain scalar values. */ +SEC("raw_tp/signal_generate") +__success +int test_raw_tp_signal_generate_info_scalar(void *ctx) +{ + asm volatile("r1 = *(u64 *)(r1 +8); if r1 != 1 goto +0;" ::: __clobber_all); + return 0; +} + +/* tp_btf programs may inspect the sentinel as a scalar value. */ +SEC("tp_btf/signal_generate") +__success +int test_tp_btf_signal_generate_info_scalar(void *ctx) +{ + asm volatile("r1 = *(u64 *)(r1 +8); if r1 != 1 goto +0;" ::: __clobber_all); + return 0; +} + +/* SEND_SIG_PRIV is non-NULL, so a NULL check cannot make info safe. */ +SEC("tp_btf/signal_generate") +__failure __msg("R1 invalid mem access 'scalar'") +int test_tp_btf_signal_generate_info_no_deref(void *ctx) +{ + asm volatile("r1 = *(u64 *)(r1 +8); if r1 == 0 goto +1; " + "r1 = *(u32 *)(r1 +0);" ::: __clobber_all); + return 0; +} + +SEC("tp_btf/signal_deliver") +__failure __msg("R1 invalid mem access 'scalar'") +int test_tp_btf_signal_deliver_info_no_deref(void *ctx) +{ + asm volatile("r1 = *(u64 *)(r1 +8); r1 = *(u32 *)(r1 +0);" ::: __clobber_all); + return 0; +} -- cgit v1.2.3 From 266aa4ad0b2e82397cd9045752c9bff03d98eddd Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 3 Sep 2026 16:44:21 +0200 Subject: bpf: Reject tail calls directly from callback frames A tail call from a non-zero frame is modeled as a return from that frame. The verifier makes R0 unknown and calls prepare_func_exit() for the taken branch. When the current frame is a synchronous callback, prepare_func_exit() enforces the callback return-value contract and marks R0 precise. Since the tail-call path synthesized R0 rather than deriving it from an instruction, precision backtracking reaches the callback-calling instruction with R0 still requested and triggers the "callback unexpected regs" verifier bug. A CAP_BPF task can therefore cause a WARN and an -EFAULT BPF_PROG_LOAD. Tail calls reachable from callbacks are already rejected later by check_max_stack_depth(). Reject a tail call made directly by a callback before constructing the inconsistent return state, using the existing diagnostic. Tail calls from ordinary subprograms keep their current behavior. Fixes: e3245f899043 ("bpf: properly verify tail call behavior") Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260903144433.1716731-4-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/verifier.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 7d8ddb1bee00..f540279ff4ab 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -11228,6 +11228,17 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn if (env->cur_state->curframe) { struct bpf_verifier_state *branch; + /* + * A taken tail call is modeled as a return from the current + * frame. A callback frame cannot be left that way because + * prepare_func_exit() would apply its return contract to the + * unknown R0 synthesized below. Stack-depth validation rejects + * this construct anyway. + */ + if (cur_func(env)->in_callback_fn) { + verbose(env, "cannot tail call within callback\n"); + return -EINVAL; + } mark_reg_scratched(env, BPF_REG_0); branch = push_stack(env, env->insn_idx + 1, env->insn_idx, false); if (IS_ERR(branch)) -- cgit v1.2.3 From d9ae3e4c7fb5bfaccc9ca295692d54130862f3b2 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 3 Sep 2026 16:44:22 +0200 Subject: selftests/bpf: Test direct tail calls from callbacks tailcall_callback tests a tail call one static subprogram below a callback. That reaches the later stack-depth rejection, but it does not exercise the tail-call helper while the current frame is itself a callback. Add a callback that calls bpf_tail_call directly and expect the existing "cannot tail call within callback" diagnostic. On an affected kernel, the load instead reaches the "callback unexpected regs" verifier bug, so the expected message is absent and the test fails. The existing ordinary subprogram case remains a success control for legitimate tail calls. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260903144433.1716731-5-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- tools/testing/selftests/bpf/progs/tailcall_callback.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/tailcall_callback.c b/tools/testing/selftests/bpf/progs/tailcall_callback.c index c41632cf423b..14fa7a87028e 100644 --- a/tools/testing/selftests/bpf/progs/tailcall_callback.c +++ b/tools/testing/selftests/bpf/progs/tailcall_callback.c @@ -44,6 +44,13 @@ int callback_loop(int index, void **cb_ctx) return ret ? 1 : 0; } +static __noinline +int callback_tail(int index, void **cb_ctx) +{ + bpf_tail_call_static(*cb_ctx, &jmp_table, 0); + return 0; +} + static __noinline int callback_empty(int index, void *data) { @@ -78,4 +85,13 @@ int tailcall_callback_2(struct __sk_buff *skb) return 0; } +/* callback with a direct tail call is rejected without a verifier bug */ +SEC("tc") +__failure __msg("cannot tail call within callback") +int tailcall_callback_3(struct __sk_buff *skb) +{ + bpf_loop(1, callback_tail, &skb, 0); + return 0; +} + char __license[] SEC("license") = "GPL"; -- cgit v1.2.3 From 7b7b8b5960102566bd625ae829d1f330c5b5d104 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 3 Sep 2026 16:44:23 +0200 Subject: bpf: Reject resilient lock operations in rbtree callbacks __bpf_rbtree_add() keeps parent and link pointers live across calls to the program-supplied comparison callback. The verifier therefore requires the root's lock to remain held throughout the callback. The helper path enforces this rule for bpf_spin_lock() and bpf_spin_unlock(), but the resilient lock kfunc argument path does not. Since resilient locks may protect BPF rbtree roots, a callback can release the root lock and let another CPU remove and free the node referenced by the in-progress tree walk. The walk then resumes using freed pointers. Reject resilient lock kfuncs in an rbtree comparison callback, matching the existing policy for the spin lock helpers. Resilient-lock-protected trees remain valid when their comparison callbacks leave lock state alone. Fixes: 0de2046137f9 ("bpf: Implement verifier support for rqspinlock") Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260903144433.1716731-6-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/verifier.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index f540279ff4ab..32d31fa67036 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -13241,6 +13241,11 @@ check_ok: { int flags = PROCESS_RES_LOCK; + if (in_rbtree_lock_required_cb(env)) { + verbose(env, "can't res_spin_{lock,unlock} in rbtree cb\n"); + return -EACCES; + } + if (reg->type != PTR_TO_MAP_VALUE && reg->type != (PTR_TO_BTF_ID | MEM_ALLOC)) { verbose(env, "%s doesn't point to map value or allocated object\n", reg_arg_name(env, argno)); -- cgit v1.2.3 From 08b4dc83d981bf9136d37aaa4f5cd021ba0d8f2b Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 3 Sep 2026 16:44:24 +0200 Subject: selftests/bpf: Reject resilient unlock in rbtree callback Add a load-only verifier regression for a resilient lock operation in an rbtree comparison callback. The program holds the rbtree's regular spin lock and a separate resilient lock, then releases the resilient lock from the callback. This isolates the missing kfunc policy check without running a concurrent tree mutation. Release the resilient lock before the regular lock on the outer fall-through. The broken verifier therefore accepts the balanced program, while the fixed verifier rejects the resilient unlock specifically while verifying the callback. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260903144433.1716731-7-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- tools/testing/selftests/bpf/progs/rbtree_fail.c | 29 +++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/rbtree_fail.c b/tools/testing/selftests/bpf/progs/rbtree_fail.c index 555379952dcc..803419a47c62 100644 --- a/tools/testing/selftests/bpf/progs/rbtree_fail.c +++ b/tools/testing/selftests/bpf/progs/rbtree_fail.c @@ -16,6 +16,7 @@ struct node_data { private(A) struct bpf_spin_lock glock; private(A) struct bpf_rb_root groot __contains(node_data, node); private(A) struct bpf_rb_root groot2 __contains(node_data, node); +private(B) struct bpf_res_spin_lock res_glock; static bool less(struct bpf_rb_node *a, const struct bpf_rb_node *b) { @@ -265,6 +266,12 @@ static bool less__bad_fn_call_first_unlock_after(struct bpf_rb_node *a, const st return node_a->key < node_b->key; } +static bool less__bad_res_spin_unlock(struct bpf_rb_node *a, const struct bpf_rb_node *b) +{ + bpf_res_spin_unlock(&res_glock); + return false; +} + static __always_inline long add_with_cb(bool (cb)(struct bpf_rb_node *a, const struct bpf_rb_node *b)) { @@ -301,4 +308,26 @@ long rbtree_api_add_bad_cb_bad_fn_call_first_unlock_after(void *ctx) return add_with_cb(less__bad_fn_call_first_unlock_after); } +SEC("?tc") +__failure __msg("can't res_spin_{lock,unlock} in rbtree cb") +long rbtree_api_add_bad_cb_res_spin_unlock(void *ctx) +{ + struct node_data *n; + + n = bpf_obj_new(typeof(*n)); + if (!n) + return 1; + + bpf_spin_lock(&glock); + if (bpf_res_spin_lock(&res_glock)) { + bpf_spin_unlock(&glock); + bpf_obj_drop(n); + return 1; + } + bpf_rbtree_add(&groot, &n->node, less__bad_res_spin_unlock); + bpf_res_spin_unlock(&res_glock); + bpf_spin_unlock(&glock); + return 0; +} + char _license[] SEC("license") = "GPL"; -- cgit v1.2.3 From a453d6e3b8e8e1a321c8744d6189d763af9287d0 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 3 Sep 2026 16:44:25 +0200 Subject: bpf: Mark sched_process_wait argument as nullable do_wait() passes wo->wo_pid to the sched_process_wait tracepoint. kernel_wait4() leaves wo_pid NULL for wait4(-1), and kernel_waitid_prepare() does likewise for waitid(P_ALL). btf_ctx_access() currently types argument 0 as PTR_TO_BTF_ID | PTR_TRUSTED. Without PTR_MAYBE_NULL, the verifier accepts an unchecked dereference. Trusted pointer loads have no fault protection, so a wait for any child can then cause a NULL pointer dereference in JITed BPF code. Add sched_process_wait to raw_tp_null_args[] with argument 0 marked nullable. The verifier rejects an unchecked dereference while preserving access after the program checks the pointer for NULL. Fixes: 838a10bd2ebf ("bpf: Augment raw_tp arguments with PTR_MAYBE_NULL") Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260903144433.1716731-8-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/btf.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index b5aa802451bd..5d93fd82e764 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -6672,6 +6672,10 @@ struct bpf_raw_tp_null_args { static const struct bpf_raw_tp_null_args raw_tp_null_args[] = { /* sched */ { "sched_pi_setprio", 0x10 }, + /* + * do_wait() passes NULL for wait4(-1) and waitid(P_ALL). + */ + { "sched_process_wait", 0x1 }, /* ... from sched_numa_pair_template event class */ { "sched_stick_numa", 0x100 }, { "sched_swap_numa", 0x100 }, -- cgit v1.2.3 From c1992ba73b0339166906eb2224494e04052a8150 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 3 Sep 2026 16:44:26 +0200 Subject: selftests/bpf: Test sched_process_wait nullable argument Add a load-time verifier test that dereferences argument 0 of the sched_process_wait tp_btf program without checking it. The test expects the nullable-pointer diagnostic, so it is accepted unexpectedly before the fix and rejected as expected after it. Add a successful control that checks the argument for NULL before the dereference. This ensures the nullable marking preserves legitimate access to the pid when the tracepoint supplies one. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260903144433.1716731-9-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- tools/testing/selftests/bpf/progs/raw_tp_null_fail.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/raw_tp_null_fail.c b/tools/testing/selftests/bpf/progs/raw_tp_null_fail.c index 7e8842bf9000..725d73c9ffe1 100644 --- a/tools/testing/selftests/bpf/progs/raw_tp_null_fail.c +++ b/tools/testing/selftests/bpf/progs/raw_tp_null_fail.c @@ -58,3 +58,20 @@ int test_tp_btf_signal_deliver_info_no_deref(void *ctx) asm volatile("r1 = *(u64 *)(r1 +8); r1 = *(u32 *)(r1 +0);" ::: __clobber_all); return 0; } + +SEC("tp_btf/sched_process_wait") +__failure __msg("R1 invalid mem access 'trusted_ptr_or_null_'") +int test_raw_tp_null_sched_process_wait_arg_1(void *ctx) +{ + asm volatile("r1 = *(u64 *)(r1 +0); r1 = *(u32 *)(r1 +0);" ::: __clobber_all); + return 0; +} + +SEC("tp_btf/sched_process_wait") +__success +int test_raw_tp_null_sched_process_wait_arg_1_checked(void *ctx) +{ + asm volatile("r1 = *(u64 *)(r1 +0); if r1 == 0 goto +1; " + "r1 = *(u32 *)(r1 +0);" ::: __clobber_all); + return 0; +} -- cgit v1.2.3 From d05524794240b52fdc3b6c1220dd05505715824d Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 3 Sep 2026 16:44:27 +0200 Subject: bpf: Mark syscall helpers as sleepable bpf_sys_bpf() executes the bpf(2) syscall body, which can take mutexes, allocate with GFP_KERNEL, and wait for an RCU grace period. bpf_sys_close() reaches close_fd() and filp_close(), which can sleep as well. Both helpers are limited to BPF_PROG_TYPE_SYSCALL, whose main program is sleepable. That does not make every callback sleepable: a syscall program can register a bpf_timer callback, and the verifier checks that callback in a non-sleepable context while retaining the syscall helper set. Without .might_sleep on the prototypes, such a callback can invoke bpf_sys_bpf() from hrtimer softirq context and trigger a scheduling-while-atomic failure. bpf_sys_close() is exposed through the same missing context check. Set .might_sleep on both prototypes so the existing helper-context check rejects them from timer callbacks and other atomic regions. Calls from the sleepable main body remain valid. Fixes: 79a7f8bdb159 ("bpf: Introduce bpf_sys_bpf() helper and program type.") Fixes: 3abea089246f ("bpf: Add bpf_sys_close() helper.") Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260903144433.1716731-10-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/syscall.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 6874ba1424af..c7bc9ba9b331 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -6568,6 +6568,7 @@ EXPORT_SYMBOL_NS(kern_sys_bpf, "BPF_INTERNAL"); static const struct bpf_func_proto bpf_sys_bpf_proto = { .func = bpf_sys_bpf, .gpl_only = false, + .might_sleep = true, .ret_type = RET_INTEGER, .arg1_type = ARG_ANYTHING, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, @@ -6593,6 +6594,7 @@ BPF_CALL_1(bpf_sys_close, u32, fd) static const struct bpf_func_proto bpf_sys_close_proto = { .func = bpf_sys_close, .gpl_only = false, + .might_sleep = true, .ret_type = RET_INTEGER, .arg1_type = ARG_ANYTHING, }; -- cgit v1.2.3 From 26a3a510cd3433e15f37ea1d5a6f2c17a0170316 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 3 Sep 2026 16:44:28 +0200 Subject: selftests/bpf: Check syscall helpers in timer callbacks A BPF_PROG_TYPE_SYSCALL program is sleepable, but its bpf_timer callbacks run in a non-sleepable hrtimer softirq context. Add verifier cases that call bpf_sys_bpf() and bpf_sys_close() from timer callbacks. Without the syscall helper prototype annotations these programs load, so their failure expectations expose the bug. Also add successful controls that call each helper from the syscall program main body, ensuring that the intended sleepable use remains accepted. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260903144433.1716731-11-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- .../bpf/progs/verifier_async_cb_context.c | 64 ++++++++++++++++++++++ 1 file changed, 64 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_async_cb_context.c b/tools/testing/selftests/bpf/progs/verifier_async_cb_context.c index 6bf95550a024..a7c84d3fa4c7 100644 --- a/tools/testing/selftests/bpf/progs/verifier_async_cb_context.c +++ b/tools/testing/selftests/bpf/progs/verifier_async_cb_context.c @@ -62,6 +62,70 @@ int timer_sleepable_prog(void *ctx) return 0; } +static int timer_sys_bpf_cb(void *map, int *key, struct bpf_timer *timer) +{ + __u64 attr = 0; + + bpf_sys_bpf(BPF_MAP_FREEZE, &attr, sizeof(attr)); + return 0; +} + +SEC("syscall") +__failure __msg("sleepable helper bpf_sys_bpf#{{[0-9]+}} in non-sleepable prog") +int timer_sys_bpf_prog(void *ctx) +{ + struct timer_elem *val; + int key = 0; + + val = bpf_map_lookup_elem(&timer_map, &key); + if (!val) + return 0; + + bpf_timer_init(&val->t, &timer_map, 0); + bpf_timer_set_callback(&val->t, timer_sys_bpf_cb); + return 0; +} + +static int timer_sys_close_cb(void *map, int *key, struct bpf_timer *timer) +{ + bpf_sys_close(0); + return 0; +} + +SEC("syscall") +__failure __msg("sleepable helper bpf_sys_close#{{[0-9]+}} in non-sleepable prog") +int timer_sys_close_prog(void *ctx) +{ + struct timer_elem *val; + int key = 0; + + val = bpf_map_lookup_elem(&timer_map, &key); + if (!val) + return 0; + + bpf_timer_init(&val->t, &timer_map, 0); + bpf_timer_set_callback(&val->t, timer_sys_close_cb); + return 0; +} + +SEC("syscall") +__success +int syscall_sys_bpf_prog(void *ctx) +{ + __u64 attr = 0; + + bpf_sys_bpf(BPF_MAP_FREEZE, &attr, sizeof(attr)); + return 0; +} + +SEC("syscall") +__success +int syscall_sys_close_prog(void *ctx) +{ + bpf_sys_close(0); + return 0; +} + /* Workqueue tests */ struct wq_elem { -- cgit v1.2.3 From 4814ed6406f3493bd554ad046da5f7fc04833571 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Thu, 3 Sep 2026 10:15:39 -0700 Subject: bpf: zero extend the result of an arena 32-bit cmpxchg bpf_convert_ctx_accesses() rewrites an atomic on an arena pointer from BPF_STX | BPF_ATOMIC to BPF_STX | BPF_PROBE_ATOMIC, and it runs before bpf_opt_subreg_zext_lo32_rnd_hi32(). That pass emits an explicit zero extension for a 32-bit cmpxchg even when bpf_jit_needs_zext() is false. This is done because on some architectures 32-bit cmpxchg requires explicit zero extension for the dst register. E.g. on x86-64 'lock cmpxchg' does not change the %eax if comparison is successful, while BPF semantics declare that each operation on a 32-bit register zero extends it's upper half. is_cmpxchg_insn() matches BPF_MODE == BPF_ATOMIC only, so an arena cmpxchg misses said zero extension adjustment. This patch adjusts is_cmpxchg_insn() to match BPF_PROBE_ATOMIC alongside BPF_ATOMIC. Fixes: d503a04f8bc0 ("bpf: Add support for certain atomics in bpf_arena to x86 JIT") Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260903171542.1438050-1-eddyz87@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/fixups.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/fixups.c b/kernel/bpf/fixups.c index 65b441e4a351..52d3cec33672 100644 --- a/kernel/bpf/fixups.c +++ b/kernel/bpf/fixups.c @@ -13,10 +13,15 @@ #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args) +/* + * Matches BPF_PROBE_ATOMIC too: bpf_convert_ctx_accesses() rewrites arena + * atomics before bpf_opt_subreg_zext_lo32_rnd_hi32() runs. + */ static bool is_cmpxchg_insn(const struct bpf_insn *insn) { return BPF_CLASS(insn->code) == BPF_STX && - BPF_MODE(insn->code) == BPF_ATOMIC && + (BPF_MODE(insn->code) == BPF_ATOMIC || + BPF_MODE(insn->code) == BPF_PROBE_ATOMIC) && insn->imm == BPF_CMPXCHG; } -- cgit v1.2.3 From 1f3cd9719c40715a7d6328bdbef817d5731bf61c Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Thu, 3 Sep 2026 10:15:40 -0700 Subject: bpf: update disasm.c to print BPF_PROBE_ATOMIC as atomics bpf_convert_ctx_accesses() rewrites an atomic on an arena pointer from BPF_STX | BPF_ATOMIC to BPF_STX | BPF_PROBE_ATOMIC, this patch adjusts print_bpf_insn() to print such instructions as regular atomics with a 'probe_' prefix (instead of printing them as BUG_XX). Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260903171542.1438050-2-eddyz87@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/disasm.c | 47 +++++++++++++++++++++++++---------------------- 1 file changed, 25 insertions(+), 22 deletions(-) diff --git a/kernel/bpf/disasm.c b/kernel/bpf/disasm.c index b1a3fbe3fda5..3ce8d74b0e40 100644 --- a/kernel/bpf/disasm.c +++ b/kernel/bpf/disasm.c @@ -7,6 +7,9 @@ #include "disasm.h" +/* Only defined by the non-UAPI linux/filter.h, which this file cannot use. */ +#define BPF_PROBE_ATOMIC 0xe0 + #define __BPF_FUNC_STR_FN(x) [BPF_FUNC_ ## x] = __stringify(bpf_ ## x) static const char * const func_id_str[] = { __BPF_FUNC_MAPPER(__BPF_FUNC_STR_FN) @@ -226,57 +229,57 @@ void print_bpf_insn(const struct bpf_insn_cbs *cbs, insn->imm); } } else if (class == BPF_STX) { + const char *probe_pfx = BPF_MODE(insn->code) == BPF_PROBE_ATOMIC ? "probe " : ""; + bool atomic = BPF_MODE(insn->code) == BPF_ATOMIC || + BPF_MODE(insn->code) == BPF_PROBE_ATOMIC; + if (BPF_MODE(insn->code) == BPF_MEM) verbose(cbs->private_data, "(%02x) *(%s *)(r%d %+d) = r%d", insn->code, bpf_ldst_string[BPF_SIZE(insn->code) >> 3], insn->dst_reg, insn->off, insn->src_reg); - else if (BPF_MODE(insn->code) == BPF_ATOMIC && + else if (atomic && (insn->imm == BPF_ADD || insn->imm == BPF_AND || insn->imm == BPF_OR || insn->imm == BPF_XOR)) { - verbose(cbs->private_data, "(%02x) lock *(%s *)(r%d %+d) %s r%d", - insn->code, + verbose(cbs->private_data, "(%02x) %slock *(%s *)(r%d %+d) %s r%d", + insn->code, probe_pfx, bpf_ldst_string[BPF_SIZE(insn->code) >> 3], insn->dst_reg, insn->off, bpf_alu_string[BPF_OP(insn->imm) >> 4], insn->src_reg); - } else if (BPF_MODE(insn->code) == BPF_ATOMIC && + } else if (atomic && (insn->imm == (BPF_ADD | BPF_FETCH) || insn->imm == (BPF_AND | BPF_FETCH) || insn->imm == (BPF_OR | BPF_FETCH) || insn->imm == (BPF_XOR | BPF_FETCH))) { - verbose(cbs->private_data, "(%02x) r%d = atomic%s_fetch_%s((%s *)(r%d %+d), r%d)", - insn->code, insn->src_reg, + verbose(cbs->private_data, "(%02x) %sr%d = atomic%s_fetch_%s((%s *)(r%d %+d), r%d)", + insn->code, probe_pfx, insn->src_reg, BPF_SIZE(insn->code) == BPF_DW ? "64" : "", bpf_atomic_alu_string[BPF_OP(insn->imm) >> 4], bpf_ldst_string[BPF_SIZE(insn->code) >> 3], insn->dst_reg, insn->off, insn->src_reg); - } else if (BPF_MODE(insn->code) == BPF_ATOMIC && - insn->imm == BPF_CMPXCHG) { - verbose(cbs->private_data, "(%02x) r0 = atomic%s_cmpxchg((%s *)(r%d %+d), r0, r%d)", - insn->code, + } else if (atomic && insn->imm == BPF_CMPXCHG) { + verbose(cbs->private_data, "(%02x) %sr0 = atomic%s_cmpxchg((%s *)(r%d %+d), r0, r%d)", + insn->code, probe_pfx, BPF_SIZE(insn->code) == BPF_DW ? "64" : "", bpf_ldst_string[BPF_SIZE(insn->code) >> 3], insn->dst_reg, insn->off, insn->src_reg); - } else if (BPF_MODE(insn->code) == BPF_ATOMIC && - insn->imm == BPF_XCHG) { - verbose(cbs->private_data, "(%02x) r%d = atomic%s_xchg((%s *)(r%d %+d), r%d)", - insn->code, insn->src_reg, + } else if (atomic && insn->imm == BPF_XCHG) { + verbose(cbs->private_data, "(%02x) %sr%d = atomic%s_xchg((%s *)(r%d %+d), r%d)", + insn->code, probe_pfx, insn->src_reg, BPF_SIZE(insn->code) == BPF_DW ? "64" : "", bpf_ldst_string[BPF_SIZE(insn->code) >> 3], insn->dst_reg, insn->off, insn->src_reg); - } else if (BPF_MODE(insn->code) == BPF_ATOMIC && - insn->imm == BPF_LOAD_ACQ) { - verbose(cbs->private_data, "(%02x) r%d = load_acquire((%s *)(r%d %+d))", - insn->code, insn->dst_reg, + } else if (atomic && insn->imm == BPF_LOAD_ACQ) { + verbose(cbs->private_data, "(%02x) %sr%d = load_acquire((%s *)(r%d %+d))", + insn->code, probe_pfx, insn->dst_reg, bpf_ldst_string[BPF_SIZE(insn->code) >> 3], insn->src_reg, insn->off); - } else if (BPF_MODE(insn->code) == BPF_ATOMIC && - insn->imm == BPF_STORE_REL) { - verbose(cbs->private_data, "(%02x) store_release((%s *)(r%d %+d), r%d)", - insn->code, + } else if (atomic && insn->imm == BPF_STORE_REL) { + verbose(cbs->private_data, "(%02x) %sstore_release((%s *)(r%d %+d), r%d)", + insn->code, probe_pfx, bpf_ldst_string[BPF_SIZE(insn->code) >> 3], insn->dst_reg, insn->off, insn->src_reg); } else { -- cgit v1.2.3 From 54ed91950363c116bec9be1b7015ff2bfa989950 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Thu, 3 Sep 2026 10:15:41 -0700 Subject: selftests/bpf: check zero extension of an arena 32-bit cmpxchg Add a test to verify that destination register of a 32-bit cmpxchg operating on an arena pointer is explicitly zero extended. W/o patch #1 this did not happen. Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260903171542.1438050-3-eddyz87@gmail.com Signed-off-by: Alexei Starovoitov --- tools/testing/selftests/bpf/progs/verifier_zext.c | 26 +++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_zext.c b/tools/testing/selftests/bpf/progs/verifier_zext.c index 8f2362da91d6..572017fe28fb 100644 --- a/tools/testing/selftests/bpf/progs/verifier_zext.c +++ b/tools/testing/selftests/bpf/progs/verifier_zext.c @@ -356,6 +356,32 @@ __naked void arena_ptr(void) : __clobber_all); } +/* + * Result of a 32-bit cmpxchg is always explicitly zero extended. + * Check that this holds for arenas (BPF_PROBE_ATOMIC instruction flavor). + */ +SEC("socket") +__success +__xlated("probe r0 = atomic_cmpxchg((u32 *)(r1 +0), r0, r2)") +__xlated("w0 = w0") +__naked void zext_arena_cmpxchg32(void) +{ + asm volatile (" \ + r9 = %[arena] ll; /* associate the arena with the program */ \ + r1 = 0; \ + r1 = addr_space_cast(r1, 0, 1); \ + r0 = 0; \ + r2 = 0; \ + .8byte %[cmpxchg32]; \ + r0 >>= 32; /* make the upper half live */ \ + exit; \ +" : + : __imm_addr(arena), + __imm_insn(cmpxchg32, + BPF_ATOMIC_OP(BPF_W, BPF_CMPXCHG, BPF_REG_1, BPF_REG_2, 0)) + : __clobber_all); +} + #endif /* Check if probe mem loads keep their zero extension. */ -- cgit v1.2.3 From 0b1c83dc3c4401cd7e846548f62e3caf3d06742e Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Thu, 3 Sep 2026 13:58:19 -0700 Subject: bpf: don't rewrite bpf_fastcall patterns entered by a jump mark_fastcall_pattern_for_call() must ensure that matched "spill; call; fill" instruction series is not interrupted by a jump. Otherwise the rewrite applied by bpf_remove_fastcall_spills_fills() is not sound. Record the instructions targeted by jumps in insn_aux_data[*].jump_target when the CFG is built and use this flag to stop growing a pattern at such an instruction. Jumps to the first spill are fine. Note that existing insn_aux_data[*].jmp_point field can't be reused, as it marks subprogram return instructions. Fixes: 5b5f51bff1b6 ("bpf: no_caller_saved_registers attribute for helper calls") Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260903205820.1743087-1-eddyz87@gmail.com Signed-off-by: Alexei Starovoitov --- include/linux/bpf_verifier.h | 12 ++++++++++++ kernel/bpf/cfg.c | 3 +++ kernel/bpf/verifier.c | 8 ++++++++ 3 files changed, 23 insertions(+) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 5fad59fdab0d..1339c2f028db 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -706,6 +706,8 @@ struct bpf_insn_aux_data { */ u32 calls_callback:1; u32 indirect_target:1; /* if it is an indirect jump target */ + /* true if some jump or call instruction targets this instruction */ + u32 jump_target:1; /* * CFG strongly connected component this instruction belongs to, * zero if it is a singleton SCC. @@ -1142,6 +1144,16 @@ static inline void mark_jmp_point(struct bpf_verifier_env *env, int idx) env->insn_aux_data[idx].jmp_point = true; } +static inline void mark_jump_target(struct bpf_verifier_env *env, int idx) +{ + env->insn_aux_data[idx].jump_target = true; +} + +static inline bool bpf_is_jump_target(struct bpf_verifier_env *env, int insn_idx) +{ + return env->insn_aux_data[insn_idx].jump_target; +} + static inline struct bpf_func_state *cur_func(struct bpf_verifier_env *env) { struct bpf_verifier_state *cur = env->cur_state; diff --git a/kernel/bpf/cfg.c b/kernel/bpf/cfg.c index 0f13c13f4133..842c7d1eabcc 100644 --- a/kernel/bpf/cfg.c +++ b/kernel/bpf/cfg.c @@ -125,6 +125,7 @@ static int push_insn(int t, int w, int e, struct bpf_verifier_env *env) /* mark branch target for state pruning */ mark_prune_point(env, w); mark_jmp_point(env, w); + mark_jump_target(env, w); } if (insn_state[w] == 0) { @@ -403,6 +404,7 @@ static int visit_gotox_insn(int t, struct bpf_verifier_env *env) } mark_jmp_point(env, w); + mark_jump_target(env, w); /* EXPLORED || DISCOVERED */ if (insn_state[w]) @@ -564,6 +566,7 @@ static int visit_insn(int t, struct bpf_verifier_env *env) mark_prune_point(env, t + off + 1); mark_jmp_point(env, t + off + 1); + mark_jump_target(env, t + off + 1); return ret; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 32d31fa67036..2ed17edf77f2 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -17656,6 +17656,10 @@ bool bpf_get_call_summary(struct bpf_verifier_env *env, struct bpf_insn *call, * r0 = *(u64 *)(r10 - 8); r0 += r1; * r0 += r1; exit; * exit; + * + * Both uses of the marks assume that a pattern is entered at its first + * spill and thus executes as a unit, hence a pattern is not grown past + * an instruction targeted by a jump. */ static void mark_fastcall_pattern_for_call(struct bpf_verifier_env *env, struct bpf_subprog_info *subprog, @@ -17694,6 +17698,10 @@ static void mark_fastcall_pattern_for_call(struct bpf_verifier_env *env, for (i = 1, off = lowest_off; i <= ARRAY_SIZE(caller_saved); ++i, off += BPF_REG_SIZE) { if (insn_idx - i < 0 || insn_idx + i >= env->prog->len) break; + /* stx/ldx/call must not be a jump targets, a jump to the first stx is fine */ + if (bpf_is_jump_target(env, insn_idx - i + 1) || + bpf_is_jump_target(env, insn_idx + i)) + break; stx = &insns[insn_idx - i]; ldx = &insns[insn_idx + i]; /* must be a stack spill/fill pair */ -- cgit v1.2.3 From 65b1518c995c590ab01f1e87f37d4eb47e8d050f Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Thu, 3 Sep 2026 13:58:20 -0700 Subject: selftests/bpf: bpf_fastcall patterns entered by a jump Check bpf_fastcall pattern detection when the pattern is entered at an instruction other than the first spill: - a jump to the first spill allows the rewrite; - conditional/unconditional a jump to the call or to the fill does not allow the rewrite. Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260903205820.1743087-2-eddyz87@gmail.com Signed-off-by: Alexei Starovoitov --- .../selftests/bpf/progs/verifier_bpf_fastcall.c | 110 +++++++++++++++++++++ 1 file changed, 110 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c b/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c index 328cf630210a..a73b837553fb 100644 --- a/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c +++ b/tools/testing/selftests/bpf/progs/verifier_bpf_fastcall.c @@ -621,6 +621,116 @@ __naked void helper_call_does_not_prevent_bpf_fastcall(void) : __clobber_all); } +/* A jump to the first spill executes the whole pattern, rewrite is safe. */ +SEC("raw_tp") +__arch_x86_64 +__log_level(4) +__msg("subprog 0 (jump_to_first_spill) main {{.*}} stack 0") +__xlated("2: if r0 == 0x2a goto pc+0") +__xlated("3: r0 = ") +__xlated("4: r0 = &(void __percpu *)(r0)") +__success +__naked void jump_to_first_spill(void) +{ + asm volatile ( + "call %[bpf_get_prandom_u32];" + "r1 = 1;" + "if r0 == 42 goto l0_%=;" +"l0_%=:" + "*(u64 *)(r10 - 8) = r1;" + "call %[bpf_get_smp_processor_id];" + "r1 = *(u64 *)(r10 - 8);" + "exit;" + : + : __imm(bpf_get_prandom_u32), + __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +/* A jump to the call skips the spill, the pattern must be kept. */ +SEC("raw_tp") +__arch_x86_64 +__log_level(4) +__msg("subprog 0 (jump_to_call) main {{.*}} stack 8") +__xlated("2: if r0 == 0x2a goto pc+1") +__xlated("3: *(u64 *)(r10 -8) = r1") +__xlated("...") +__xlated("7: r1 = *(u64 *)(r10 -8)") +__success +__naked void jump_to_call(void) +{ + asm volatile ( + "call %[bpf_get_prandom_u32];" + "r1 = 1;" + "if r0 == 42 goto l0_%=;" + "*(u64 *)(r10 - 8) = r1;" +"l0_%=:" + "call %[bpf_get_smp_processor_id];" + "r1 = *(u64 *)(r10 - 8);" + "exit;" + : + : __imm(bpf_get_prandom_u32), + __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +/* A jump to the fill skips the spill, the pattern must be kept. */ +SEC("raw_tp") +__arch_x86_64 +__log_level(4) +__msg("subprog 0 (jump_to_fill) main {{.*}} stack 8") +__xlated("2: if r0 == 0x2a goto pc+4") +__xlated("3: *(u64 *)(r10 -8) = r1") +__xlated("...") +__xlated("7: r1 = *(u64 *)(r10 -8)") +__success +__naked void jump_to_fill(void) +{ + asm volatile ( + "call %[bpf_get_prandom_u32];" + "r1 = 1;" + "if r0 == 42 goto l0_%=;" + "*(u64 *)(r10 - 8) = r1;" + "call %[bpf_get_smp_processor_id];" +"l0_%=:" + "r1 = *(u64 *)(r10 - 8);" + "exit;" + : + : __imm(bpf_get_prandom_u32), + __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + +/* Same as above, but the fill is entered by an unconditional jump. */ +SEC("raw_tp") +__arch_x86_64 +__log_level(4) +__msg("subprog 0 (unconditional_jump_to_fill) main {{.*}} stack 8") +__xlated("3: *(u64 *)(r10 -8) = r1") +__xlated("...") +__xlated("7: r1 = *(u64 *)(r10 -8)") +__xlated("8: exit") +__xlated("9: goto pc-3") +__success +__naked void unconditional_jump_to_fill(void) +{ + asm volatile ( + "call %[bpf_get_prandom_u32];" + "r1 = 1;" + "if r0 == 42 goto l1_%=;" + "*(u64 *)(r10 - 8) = r1;" + "call %[bpf_get_smp_processor_id];" +"l0_%=:" + "r1 = *(u64 *)(r10 - 8);" + "exit;" +"l1_%=:" + "goto l0_%=;" + : + : __imm(bpf_get_prandom_u32), + __imm(bpf_get_smp_processor_id) + : __clobber_all); +} + SEC("raw_tp") __arch_x86_64 __log_level(4) -- cgit v1.2.3 From 369f4ce734570bdfedaa4b5ca50e2a3f6a892728 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 3 Sep 2026 23:47:47 +0200 Subject: bpf: Check ancestor frames for rbtree callbacks bpf_rbtree_add() invokes its comparator while the caller holds the root lock. The native insertion code retains raw parent and link pointers across the callback, so the verifier prohibits unlocking, consuming tree nodes, or changing RCU state from that callback. in_rbtree_lock_required_cb() only checks the innermost verifier frame. Static subprogram calls are permitted while holding a spin lock, and such a call pushes a frame without in_callback_fn set. Consequently, all callback restrictions disappear in the nested frame. The subprogram can unlock the tree, remove and drop the node being compared, then relock. Native insertion resumes with the stale parent pointer and links freed memory into the tree. Walk all active frames for the rbtree callback instead. Benign static subprograms remain permitted, while callback restrictions follow execution into nested frames. Fixes: a44b1334aadd ("bpf: Allow calling static subprogs while holding a bpf_spin_lock") Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260903214758.2727663-2-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/verifier.c | 25 ++++++++++++++----------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 2ed17edf77f2..2b7e5c9b3ffc 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -10235,9 +10235,10 @@ static void account_current_path(struct bpf_verifier_env *env) frame ? state->frame[frame - 1] : NULL); } -/* Are we currently verifying the callback for a rbtree helper that must - * be called with lock held? If so, no need to complain about unreleased - * lock +/* + * Are we currently verifying the callback for an rbtree kfunc that must + * be called with a lock held, or one of that callback's subprogs? If so, + * no need to complain about an unreleased lock. */ static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env) { @@ -10245,17 +10246,19 @@ static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env) struct bpf_insn *insn = env->prog->insnsi; struct bpf_func_state *callee; int kfunc_btf_id; + u32 frame; - if (!state->curframe) - return false; - - callee = state->frame[state->curframe]; + for (frame = state->curframe; frame; frame--) { + callee = state->frame[frame]; + if (!callee->in_callback_fn) + continue; - if (!callee->in_callback_fn) - return false; + kfunc_btf_id = insn[callee->callsite].imm; + if (is_rbtree_lock_required_kfunc(kfunc_btf_id)) + return true; + } - kfunc_btf_id = insn[callee->callsite].imm; - return is_rbtree_lock_required_kfunc(kfunc_btf_id); + return false; } static bool retval_range_within(struct bpf_retval_range range, const struct bpf_reg_state *reg) -- cgit v1.2.3 From 22ab49afe1c901b2c0b9482f38bdb647d46e6a34 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 3 Sep 2026 23:47:48 +0200 Subject: selftests/bpf: Check rbtree callback restrictions in subprogs Add a verifier failure case where an rbtree comparator enters two nested static subprograms and the innermost subprogram unlocks and relocks the tree. Restoring the lock keeps the surrounding callback state balanced, so the test specifically exercises whether the callback restriction follows the nested calls. Also add a load-only positive control whose comparator calls a harmless static subprogram. This preserves the intended support for verified static subprogram calls while holding the tree lock. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260903214758.2727663-3-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- tools/testing/selftests/bpf/progs/rbtree_fail.c | 55 +++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/rbtree_fail.c b/tools/testing/selftests/bpf/progs/rbtree_fail.c index 803419a47c62..4504608196ab 100644 --- a/tools/testing/selftests/bpf/progs/rbtree_fail.c +++ b/tools/testing/selftests/bpf/progs/rbtree_fail.c @@ -272,6 +272,47 @@ static bool less__bad_res_spin_unlock(struct bpf_rb_node *a, const struct bpf_rb return false; } +static __noinline void rbtree_cb_unlock_relock(void) +{ + bpf_spin_unlock(&glock); + bpf_spin_lock(&glock); +} + +static __noinline void rbtree_cb_nested_unlock(void) +{ + rbtree_cb_unlock_relock(); + asm volatile (""); +} + +static bool less__bad_subprog_unlock(struct bpf_rb_node *a, const struct bpf_rb_node *b) +{ + struct node_data *node_a; + struct node_data *node_b; + + node_a = container_of(a, struct node_data, node); + node_b = container_of(b, struct node_data, node); + rbtree_cb_nested_unlock(); + + return node_a->key < node_b->key; +} + +static __noinline void rbtree_cb_noop(void) +{ + asm volatile (""); +} + +static bool less__subprog_allowed(struct bpf_rb_node *a, const struct bpf_rb_node *b) +{ + struct node_data *node_a; + struct node_data *node_b; + + node_a = container_of(a, struct node_data, node); + node_b = container_of(b, struct node_data, node); + rbtree_cb_noop(); + + return node_a->key < node_b->key; +} + static __always_inline long add_with_cb(bool (cb)(struct bpf_rb_node *a, const struct bpf_rb_node *b)) { @@ -330,4 +371,18 @@ long rbtree_api_add_bad_cb_res_spin_unlock(void *ctx) return 0; } +SEC("?tc") +__failure __msg("can't spin_{lock,unlock} in rbtree cb") +long rbtree_api_add_bad_cb_subprog_unlock(void *ctx) +{ + return add_with_cb(less__bad_subprog_unlock); +} + +SEC("?tc") +__success +long rbtree_api_add_cb_subprog_allowed(void *ctx) +{ + return add_with_cb(less__subprog_allowed); +} + char _license[] SEC("license") = "GPL"; -- cgit v1.2.3 From 620614bf7672130c43b3cff375525a2202f61979 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 3 Sep 2026 23:47:49 +0200 Subject: bpf: Mark bpf_btf_find_by_name_kind() as sleepable When bpf_btf_find_by_name_kind() finds a type in module BTF, it returns a new BTF object fd through __btf_new_fd(). This reaches anon_inode_getfd(), which can sleep while allocating or expanding the current task fd table. The helper prototype does not set might_sleep, so the verifier allows the helper in non-sleepable contexts such as BPF timer callbacks. The fd allocation can then sleep in softirq context and install the fd into the interrupted task. Mark the helper as sleepable. This preserves calls from the main body of a sleepable syscall program while rejecting calls from its non-sleepable regions. Fixes: 3d78417b60fb ("bpf: Add bpf_btf_find_by_name_kind() helper.") Reported-by: Sashiko Link: https://lore.kernel.org/bpf/20260903155150.D57251F000E9@smtp.kernel.org Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260903214758.2727663-4-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/btf.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 5d93fd82e764..9f33e95d5741 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -8749,6 +8749,7 @@ BPF_CALL_4(bpf_btf_find_by_name_kind, char *, name, int, name_sz, u32, kind, int const struct bpf_func_proto bpf_btf_find_by_name_kind_proto = { .func = bpf_btf_find_by_name_kind, .gpl_only = false, + .might_sleep = true, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_MEM | MEM_RDONLY, .arg2_type = ARG_MEM_SIZE, -- cgit v1.2.3 From 687b2729ce4c90a3cec76db85d3649e8f5086f85 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 3 Sep 2026 23:47:50 +0200 Subject: selftests/bpf: Test btf lookup helper sleepability Add an expected failure case which calls bpf_btf_find_by_name_kind() from a BPF timer callback. Without the helper prototype being marked sleepable, the verifier accepts the program and the load unexpectedly succeeds. Also add a positive control which calls the helper directly from a syscall program. This verifies that marking the helper sleepable only rejects it in non-sleepable regions. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260903214758.2727663-5-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- .../bpf/progs/verifier_async_cb_context.c | 34 ++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_async_cb_context.c b/tools/testing/selftests/bpf/progs/verifier_async_cb_context.c index a7c84d3fa4c7..e0926767bbd3 100644 --- a/tools/testing/selftests/bpf/progs/verifier_async_cb_context.c +++ b/tools/testing/selftests/bpf/progs/verifier_async_cb_context.c @@ -108,6 +108,30 @@ int timer_sys_close_prog(void *ctx) return 0; } +static int timer_btf_find_cb(void *map, int *key, struct bpf_timer *timer) +{ + char name[] = "task_struct"; + + bpf_btf_find_by_name_kind(name, sizeof(name), BTF_KIND_STRUCT, 0); + return 0; +} + +SEC("syscall") +__failure __msg("sleepable helper bpf_btf_find_by_name_kind#{{[0-9]+}} in non-sleepable prog") +int timer_btf_find_prog(void *ctx) +{ + struct timer_elem *val; + int key = 0; + + val = bpf_map_lookup_elem(&timer_map, &key); + if (!val) + return 0; + + bpf_timer_init(&val->t, &timer_map, 0); + bpf_timer_set_callback(&val->t, timer_btf_find_cb); + return 0; +} + SEC("syscall") __success int syscall_sys_bpf_prog(void *ctx) @@ -126,6 +150,16 @@ int syscall_sys_close_prog(void *ctx) return 0; } +SEC("syscall") +__success +int syscall_btf_find_prog(void *ctx) +{ + char name[] = "task_struct"; + + bpf_btf_find_by_name_kind(name, sizeof(name), BTF_KIND_STRUCT, 0); + return 0; +} + /* Workqueue tests */ struct wq_elem { -- cgit v1.2.3 From 9d02927fdf4e930893c92e35fed01a2704496900 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 3 Sep 2026 23:47:51 +0200 Subject: bpf: Mark faultable stack helpers as sleepable The faultable variants of bpf_get_stack() and bpf_get_task_stack() pass may_fault=true into the common stack collection code. Resolving user-space build IDs may then call build_id_parse_file() and block on filesystem reads. Neither helper prototype sets might_sleep. Since prototype selection uses the sleepability of the whole program, the verifier can still allow these helpers from a non-sleepable region within that program, such as an explicit RCU or preemption-disabled region. The task-stack helper can also be called from a non-sleepable timer callback of a sleepable program. Mark both faultable prototypes as sleepable. The existing helper context check then rejects these calls while continuing to allow them in genuinely sleepable contexts. Fixes: d4dd9775ec24 ("bpf: wire up sleepable bpf_get_stack() and bpf_get_task_stack() helpers") Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260903214758.2727663-6-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/stackmap.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index a839041e0d00..d09d4c3fe547 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -875,6 +875,7 @@ BPF_CALL_4(bpf_get_stack_sleepable, struct pt_regs *, regs, void *, buf, u32, si const struct bpf_func_proto bpf_get_stack_sleepable_proto = { .func = bpf_get_stack_sleepable, .gpl_only = true, + .might_sleep = true, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_UNINIT_MEM, @@ -928,6 +929,7 @@ BPF_CALL_4(bpf_get_task_stack_sleepable, struct task_struct *, task, void *, buf const struct bpf_func_proto bpf_get_task_stack_sleepable_proto = { .func = bpf_get_task_stack_sleepable, .gpl_only = false, + .might_sleep = true, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_BTF_ID, .arg1_btf_id = &btf_tracing_ids[BTF_TRACING_TYPE_TASK], -- cgit v1.2.3 From 1ba0d0d8b6757eaf107f1f0fa0830c9585853db7 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 3 Sep 2026 23:47:52 +0200 Subject: selftests/bpf: Check faultable stack helper contexts Add verifier coverage for the sleepable bpf_get_stack() and bpf_get_task_stack() implementations. Call each helper while preemption is disabled and require the verifier to reject it as sleepable. Both programs load when the prototypes lack might_sleep, so the expected-failure tests fail. Keep success controls outside the non-preemptible region to ensure ordinary calls from sleepable uprobes remain valid. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260903214758.2727663-7-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- tools/testing/selftests/bpf/progs/preempt_lock.c | 52 ++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/preempt_lock.c b/tools/testing/selftests/bpf/progs/preempt_lock.c index 6d5fce7e6ffc..81c459435680 100644 --- a/tools/testing/selftests/bpf/progs/preempt_lock.c +++ b/tools/testing/selftests/bpf/progs/preempt_lock.c @@ -115,6 +115,58 @@ int preempt_sleepable_helper(void *ctx) return 0; } +SEC("?uprobe.s") +__failure __msg("sleepable helper bpf_get_stack#") +int preempt_sleepable_get_stack(struct pt_regs *ctx) +{ + struct bpf_stack_build_id stack; + + bpf_preempt_disable(); + bpf_get_stack(ctx, &stack, sizeof(stack), + BPF_F_USER_STACK | BPF_F_USER_BUILD_ID); + bpf_preempt_enable(); + return 0; +} + +SEC("?uprobe.s") +__failure __msg("sleepable helper bpf_get_task_stack#") +int preempt_sleepable_get_task_stack(void *ctx) +{ + struct bpf_stack_build_id stack; + struct task_struct *task; + + task = bpf_get_current_task_btf(); + bpf_preempt_disable(); + bpf_get_task_stack(task, &stack, sizeof(stack), + BPF_F_USER_STACK | BPF_F_USER_BUILD_ID); + bpf_preempt_enable(); + return 0; +} + +SEC("?uprobe.s") +__success +int sleepable_get_stack(struct pt_regs *ctx) +{ + struct bpf_stack_build_id stack; + + bpf_get_stack(ctx, &stack, sizeof(stack), + BPF_F_USER_STACK | BPF_F_USER_BUILD_ID); + return 0; +} + +SEC("?uprobe.s") +__success +int sleepable_get_task_stack(void *ctx) +{ + struct bpf_stack_build_id stack; + struct task_struct *task; + + task = bpf_get_current_task_btf(); + bpf_get_task_stack(task, &stack, sizeof(stack), + BPF_F_USER_STACK | BPF_F_USER_BUILD_ID); + return 0; +} + SEC("?fentry.s/" SYS_PREFIX "sys_getpgid") __failure __msg("kernel func bpf_copy_from_user_str is sleepable within non-preemptible region") int preempt_sleepable_kfunc(void *ctx) -- cgit v1.2.3 From e7d28823c662128caae63f14e16bd394916c139b Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 3 Sep 2026 23:47:53 +0200 Subject: bpf: Reject legacy packet loads from callbacks check_ld_abs() models a failed BPF_LD_ABS or BPF_LD_IND in a subprogram as an implicit return with R0 set to zero. It calls prepare_func_exit() to explore this synthesized path. When the load is reached directly from a synchronous callback, prepare_func_exit() enforces the callback return contract and marks R0 precise. R0 is not derived from a real instruction on this path, so precision backtracking reaches the callback call with R0 still requested and triggers the "callback unexpected regs" verifier bug. A privileged program loader can therefore cause a verifier warning and an -EFAULT BPF_PROG_LOAD. These legacy packet-load instructions are deprecated. Reject them from callbacks rather than complicating their implicit-return model. Check all active frames before constructing the implicit return so nested static subprograms cannot hide the callback context. Global functions are verified independently with a fresh frame zero, so an active-frame check cannot identify a global function called from a callback. Also check the complete subprogram call graph during stack-depth validation and reject a function containing a legacy load when any caller is a callback. This covers global and static descendants without making has_ld_abs transitive, preserving its per-function BTF return-type check. Ordinary uses outside callbacks remain supported. Fixes: ee861486e377 ("bpf: Fix ld_{abs,ind} failure path analysis in subprogs") Reported-by: Sashiko Link: https://lore.kernel.org/bpf/20260903152147.C0E241F00A3A@smtp.kernel.org Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260903214758.2727663-8-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/verifier.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 2b7e5c9b3ffc..d7dd0befbd10 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -5302,6 +5302,15 @@ static int check_max_stack_depth_subprog(struct bpf_verifier_env *env, int idx, if (!priv_stack_supported) subprog[idx].priv_stack_mode = NO_PRIV_STACK; process_func: + if (subprog[idx].has_ld_abs) { + for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller) { + if (subprog[tmp].is_cb) { + verbose(env, "cannot use BPF_LD_[ABS|IND] within callback\n"); + return -EINVAL; + } + } + } + /* protect against potential stack overflow that might happen when * bpf2bpf calls get combined with tailcalls. Limit the caller's stack * depth for such case down to 256 so that the worst case scenario @@ -17182,6 +17191,7 @@ static bool may_access_skb(enum bpf_prog_type type) */ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn) { + struct bpf_verifier_state *state = env->cur_state; struct bpf_reg_state *regs = cur_regs(env); static const int ctx_reg = BPF_REG_6; u8 mode = BPF_MODE(insn->code); @@ -17192,6 +17202,13 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn) return -EINVAL; } + for (i = state->curframe; i; i--) { + if (state->frame[i]->in_callback_fn) { + verbose(env, "cannot use BPF_LD_[ABS|IND] within callback\n"); + return -EINVAL; + } + } + if (!env->ops->gen_ld_abs) { verifier_bug(env, "gen_ld_abs is null"); return -EFAULT; -- cgit v1.2.3 From 23724e009f65838bd8e1b42bed69e3daa4ecfdab Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Thu, 3 Sep 2026 23:47:54 +0200 Subject: selftests/bpf: Reject legacy packet loads from callbacks Add verifier coverage for the callback restriction on legacy packet loads. Exercise BPF_LD_ABS directly in a bpf_loop callback and BPF_LD_IND from a static subprogram called by the callback, ensuring that callback context follows nested static calls. Also exercise a callback which reaches BPF_LD_IND through a global function and its static descendant. A sibling success case calls the same global chain outside a callback, preserving support for ordinary global packet loads. Existing success cases continue to cover loads from ordinary static subprograms. The failure cases expect the policy-specific rejection instead of reaching the implicit-return path, triggering a verifier warning, or being accepted through a function boundary. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260903214758.2727663-9-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- .../testing/selftests/bpf/progs/verifier_ld_ind.c | 96 ++++++++++++++++++++++ 1 file changed, 96 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_ld_ind.c b/tools/testing/selftests/bpf/progs/verifier_ld_ind.c index 09e81b99eecb..32989f981fb6 100644 --- a/tools/testing/selftests/bpf/progs/verifier_ld_ind.c +++ b/tools/testing/selftests/bpf/progs/verifier_ld_ind.c @@ -194,6 +194,102 @@ __naked void ld_ind_subprog_both_paths_safe(void) ::: __clobber_all); } +__naked __noinline __used +static int ld_abs_callback(void) +{ + asm volatile ( + "r6 = *(u64 *)(r2 + 0);" + ".8byte %[ld_abs];" + "r0 = 0;" + "exit;" + : + : __imm_insn(ld_abs, BPF_LD_ABS(BPF_W, 0)) + : __clobber_all); +} + +SEC("socket") +__description("ld_abs: reject in callback") +__failure __msg("cannot use BPF_LD_[ABS|IND] within callback") +int ld_abs_callback_reject(struct __sk_buff *skb) +{ + bpf_loop(1, ld_abs_callback, &skb, 0); + return 0; +} + +__naked __noinline __used +static int ld_ind_callback_subprog(void) +{ + asm volatile ( + "r6 = r1;" + "r7 = 0;" + ".8byte %[ld_ind];" + "r0 = 0;" + "exit;" + : + : __imm_insn(ld_ind, BPF_LD_IND(BPF_W, BPF_REG_7, 0)) + : __clobber_all); +} + +__naked __noinline __used +static int ld_ind_callback(void) +{ + asm volatile ( + "r1 = *(u64 *)(r2 + 0);" + "call ld_ind_callback_subprog;" + "exit;" + ::: __clobber_all); +} + +SEC("socket") +__description("ld_ind: reject in callback subprog") +__failure __msg("cannot use BPF_LD_[ABS|IND] within callback") +int ld_ind_callback_subprog_reject(struct __sk_buff *skb) +{ + bpf_loop(1, ld_ind_callback, &skb, 0); + return 0; +} + +static __noinline int ld_ind_global_static(struct __sk_buff *skb) +{ + asm volatile ( + "r6 = %[skb];" + "r7 = 0;" + ".8byte %[ld_ind];" + : + : [skb] "r"(skb), + __imm_insn(ld_ind, BPF_LD_IND(BPF_W, BPF_REG_7, 0)) + : __clobber_common, "r6", "r7"); + return skb->mark; +} + +__noinline int ld_ind_global(struct __sk_buff *skb) +{ + return ld_ind_global_static(skb); +} + +static int ld_ind_global_callback(__u32 index, struct __sk_buff **ctx) +{ + ld_ind_global(*ctx); + return 0; +} + +SEC("socket") +__description("ld_ind: reject in callback global subprog") +__failure __msg("cannot use BPF_LD_[ABS|IND] within callback") +int ld_ind_global_callback_reject(struct __sk_buff *skb) +{ + bpf_loop(1, ld_ind_global_callback, &skb, 0); + return 0; +} + +SEC("socket") +__description("ld_ind: allow in non-callback global subprog") +__success +int ld_ind_global_subprog_ok(struct __sk_buff *skb) +{ + return ld_ind_global(skb); +} + /* * ld_{abs,ind} in subprogs require scalar (int) return type in BTF. * A test with void return must be rejected. -- cgit v1.2.3 From 254c881fe0554c5efb16d355c273702a27a32a20 Mon Sep 17 00:00:00 2001 From: Florian Westphal Date: Thu, 3 Sep 2026 08:58:45 +0200 Subject: selftests/bpf: Add tests to assert that netfilter progs cannot write to skb The netfilter framework is allergic to ip header changing after validation done by ip/ipv6 stack. Assert that bpf netfilter programs do not allow skb write access. Following additional tests are expected to be rejected by verifier: 1. alter skb->len. 2. alter skb->data. 3. prog calls bpf_dynptr_slice_rdwr. 4. alter location returned by dynptr API. Add following test case for bpf runtime: - alter skb data via bpf_dynptr_write() Test checks via __retval() that bpf_dynptr_write() returned nonzero value. Signed-off-by: Florian Westphal Reviewed-by: Jiayuan Chen Link: https://lore.kernel.org/r/20260903065845.22762-1-fw@strlen.de Signed-off-by: Alexei Starovoitov --- .../selftests/bpf/progs/verifier_netfilter_ctx.c | 78 ++++++++++++++++++++++ 1 file changed, 78 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_netfilter_ctx.c b/tools/testing/selftests/bpf/progs/verifier_netfilter_ctx.c index e2cbc5bda65e..b5d7f567d0d4 100644 --- a/tools/testing/selftests/bpf/progs/verifier_netfilter_ctx.c +++ b/tools/testing/selftests/bpf/progs/verifier_netfilter_ctx.c @@ -113,4 +113,82 @@ int with_valid_ctx_access_test6(struct bpf_nf_ctx *ctx) return th->dest == bpf_htons(22) ? NF_ACCEPT : NF_DROP; } +SEC("netfilter") +__description("netfilter test prog with skb write access") +__failure __msg("only read is supported") +int skb_len_write(struct bpf_nf_ctx *ctx) +{ + ctx->skb->len = 1; + return 1; +} + +SEC("netfilter") +__description("netfilter test prog with skb data write access") +__failure __msg("cannot write into rdonly_untrusted_mem") +int skb_data_write(struct bpf_nf_ctx *ctx) +{ + ctx->skb->data[0] = 0; + return 1; +} + +SEC("netfilter") +__description("netfilter test prog with bpf_dynptr_write") +__success __failure_unpriv +__retval(0) +int with_dynptr_write(struct bpf_nf_ctx *ctx) +{ + struct __sk_buff *skb = (struct __sk_buff *)ctx->skb; + struct bpf_dynptr ptr; + u8 buffer[1] = {}; + + if (bpf_dynptr_from_skb(skb, 0, &ptr)) + return 1; + + if (bpf_dynptr_write(&ptr, 0, buffer, sizeof(buffer), 0)) + return 0; /* must always fail */ + + return 1; +} + +SEC("netfilter") +__description("netfilter test prog with bpf_dynptr_slice_rdwr") +__failure __msg("the prog does not allow writes to packet data") +int with_dynptr_rdwr(struct bpf_nf_ctx *ctx) +{ + struct __sk_buff *skb = (struct __sk_buff *)ctx->skb; + u8 buffer_iph[20] = {}; + struct bpf_dynptr ptr; + struct iphdr *iph; + + if (bpf_dynptr_from_skb(skb, 0, &ptr)) + return 1; + + iph = bpf_dynptr_slice_rdwr(&ptr, 0, buffer_iph, sizeof(buffer_iph)); + if (!iph) + return 0; + + return 1; +} + +SEC("netfilter") +__description("netfilter test prog with bpf_dynptr_slice + write") +__failure __msg("cannot write into rdonly_mem") +int with_dynptr_store(struct bpf_nf_ctx *ctx) +{ + struct __sk_buff *skb = (struct __sk_buff *)ctx->skb; + u8 buffer_iph[20] = {}; + struct bpf_dynptr ptr; + struct iphdr *iph; + + if (bpf_dynptr_from_skb(skb, 0, &ptr)) + return 1; + + iph = bpf_dynptr_slice(&ptr, 0, buffer_iph, sizeof(buffer_iph)); + if (!iph) + return 0; + iph->protocol = 42; + + return 1; +} + char _license[] SEC("license") = "GPL"; -- cgit v1.2.3 From 67b529f521a6676cdfc78b91b0217d7eaa84216b Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 4 Sep 2026 01:33:18 -0700 Subject: bpf: Don't infer non-NULL from a pointer with an unbounded offset reg_not_null() decides that a register holds a non-NULL value by looking at its type alone. For pointer types that allow arithmetic the type only guarantees a non-NULL base, in case of an unbound offset the runtime offset value might still add up to NULL. Consider the followng program: r6 = bpf_map_lookup_elem(map, &0); /* present */ if (r6 == 0) return 0; r7 = bpf_map_lookup_elem(map, &1); /* absent, NULL at runtime */ r8 = r7; r8 -= r6; /* pointer - pointer: unknown scalar, -r6 */ r8 <<= 1; r8 >>= 1; /* any non-negative offset is accepted by */ /* check_reg_sane_offset_ptr() */ r6 += r8; /* verifier: map value; runtime: zero */ if (r7 != r6) return 0; *(u8 *)(r7 + 0); /* r7 is inferred non-NULL, both are zero */ At runtime both registers are zero, the comparison is true and the load faults with NULL pointer dereference. Require the offset to be within +-BPF_MAX_VAR_OFF in reg_not_null(). Fixes: cac616db39c2 ("bpf: Verifier track null pointer branch_taken with JNE and JEQ") Reported-by: Nicholas Carlini Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260904083325.2083493-1-eddyz87@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index d7dd0befbd10..faf1c8ff243d 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -352,6 +352,13 @@ static bool reg_not_null(struct bpf_verifier_env *env, const struct bpf_reg_stat if (type_may_be_null(type)) return false; + /* + * The types below guarantee a non-NULL base, an unbounded offset can + * still wrap base + offset to zero. + */ + if (reg_smin(reg) <= -BPF_MAX_VAR_OFF || reg_smax(reg) >= BPF_MAX_VAR_OFF) + return false; + type = base_type(type); return type == PTR_TO_SOCKET || type == PTR_TO_TCP_SOCK || -- cgit v1.2.3 From 6752b90ccfb378e311e428932facf37c8625abae Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 4 Sep 2026 01:33:19 -0700 Subject: selftests/bpf: No non-NULL inference from unbounded offset pointers Check that a comparison against a pointer whose offset is not bounded from above does not make the verifier infer that a nullable pointer is not NULL, and that a bounded offset still does. W/o the previous patch the first test is accepted. Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260904083325.2083493-2-eddyz87@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../bpf/progs/verifier_jeq_infer_not_null.c | 80 ++++++++++++++++++++++ 1 file changed, 80 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_jeq_infer_not_null.c b/tools/testing/selftests/bpf/progs/verifier_jeq_infer_not_null.c index b412a542ef76..8657e4a0d601 100644 --- a/tools/testing/selftests/bpf/progs/verifier_jeq_infer_not_null.c +++ b/tools/testing/selftests/bpf/progs/verifier_jeq_infer_not_null.c @@ -311,6 +311,86 @@ __naked void untrusted_mem_does_not_infer_map_value_non_null(void) : __clobber_all); } +/* + * A pointer with an offset that is not bounded from above may be null at + * runtime, hence it is not a witness for the pointer it is compared with. + */ +SEC("socket") +__failure +__msg("error: invalid dereference of R7 (a nullable map value pointer)") +__naked void unbounded_offset_does_not_infer_map_value_non_null(void) +{ + asm volatile (" \ + /* r6 = bpf_map_lookup_elem(map_hash, &0); */ \ + *(u64 *)(r10 - 8) = 0; \ + r1 = %[map_hash] ll; \ + r2 = r10; \ + r2 += -8; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto 1f; \ + r6 = r0; \ + /* r7 = bpf_map_lookup_elem(map_hash, &1); */ \ + *(u64 *)(r10 - 8) = 1; \ + r1 = %[map_hash] ll; \ + r2 = r10; \ + r2 += -8; \ + call %[bpf_map_lookup_elem]; \ + r7 = r0; \ + /* pointer - pointer is an unknown scalar */ \ + r8 = r7; \ + r8 -= r6; \ + /* r8 is in [0, S64_MAX] */ \ + r8 <<= 1; \ + r8 >>= 1; \ + /* r6 may wrap to zero at runtime */ \ + r6 += r8; \ + if r7 != r6 goto 1f; \ + r0 = *(u8 *)(r7 + 0); \ +1: r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash) + : __clobber_all); +} + +/* Same, but the offset is bounded, so the inference is still done. */ +SEC("socket") +__success +__naked void bounded_offset_infers_map_value_non_null(void) +{ + asm volatile (" \ + /* r6 = bpf_map_lookup_elem(map_hash, &0); */ \ + *(u64 *)(r10 - 8) = 0; \ + r1 = %[map_hash] ll; \ + r2 = r10; \ + r2 += -8; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto 1f; \ + r6 = r0; \ + /* r7 = bpf_map_lookup_elem(map_hash, &1); */ \ + *(u64 *)(r10 - 8) = 1; \ + r1 = %[map_hash] ll; \ + r2 = r10; \ + r2 += -8; \ + call %[bpf_map_lookup_elem]; \ + r7 = r0; \ + /* pointer - pointer is an unknown scalar */ \ + r8 = r7; \ + r8 -= r6; \ + /* r8 is in [0, 3] */ \ + r8 &= 3; \ + r6 += r8; \ + if r7 != r6 goto 1f; \ + r0 = *(u8 *)(r7 + 0); \ +1: r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash) + : __clobber_all); +} + void kfunc_root(void) { bpf_rdonly_cast(0, 0); -- cgit v1.2.3 From 73a98f96811e2cb0f4210b1caa8cb322f92f2a2b Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 4 Sep 2026 01:33:20 -0700 Subject: bpf: Don't resurrect a scalar id dropped by collect_linked_regs() check_cond_jmp_op() copies the compared registers into env->{false,true}_reg{1,2} before collect_linked_regs() runs and copies those snapshots back into both branch states afterwards. collect_linked_regs() records at most LINKED_REGS_MAX members of a linked registers group in the jump history and calls clear_scalar_id() for every member that does not fit. The compared register is not exempt from that. As a consequence, sync_linked_regs() might adjust ranges for more registers than bpf_bt_sync_linked_regs() can propagate precision to. Collect the linked registers before the snapshots are taken instead. This might lead to some unnecessary clear_scalar_id's, but from previous testing situations with many linked registers are extremely rare. Fixes: ec1d77cb0ee9 ("bpf: Use bpf_verifier_env buffers for reg_set_min_max") Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260904083325.2083493-3-eddyz87@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index faf1c8ff243d..1fb0c832611c 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -16898,6 +16898,16 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env, return err; } + /* + * Collect the linked registers before env->{true,false}_reg{1,2} setup, + * otherwise ids dropped by collect_linked_regs() would be resurrected + * when env->{true,false}_reg{1,2} are copied back. + */ + if (BPF_SRC(insn->code) == BPF_X && src_reg->type == SCALAR_VALUE && src_reg->id) + collect_linked_regs(env, this_branch, src_reg->id, &linked_regs); + if (dst_reg->type == SCALAR_VALUE && dst_reg->id) + collect_linked_regs(env, this_branch, dst_reg->id, &linked_regs); + is_jmp32 = BPF_CLASS(insn->code) == BPF_JMP32; env->false_reg1 = *dst_reg; env->false_reg2 = *src_reg; @@ -16952,10 +16962,6 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env, * 'this_branch' and 'other_branch' share this history * if parent state is created. */ - if (BPF_SRC(insn->code) == BPF_X && src_reg->type == SCALAR_VALUE && src_reg->id) - collect_linked_regs(env, this_branch, src_reg->id, &linked_regs); - if (dst_reg->type == SCALAR_VALUE && dst_reg->id) - collect_linked_regs(env, this_branch, dst_reg->id, &linked_regs); if (linked_regs.cnt > 1) { err = bpf_push_jmp_history(env, this_branch, 0, 0, 0, linked_regs_pack(&linked_regs)); if (err) -- cgit v1.2.3 From bc412b3fb185540112fcc99ac91a14e57418d28e Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 4 Sep 2026 01:33:21 -0700 Subject: selftests/bpf: Check the linked regs cap for the compared register linked_regs_too_many_regs checks that collect_linked_regs() ties at most LINKED_REGS_MAX registers for a single jump. Compare r5 instead of r0, so that the register the jump compares is itself the member that does not fit, and check that it comes out of the jump unlinked. W/o the previous patch env->{false,true}_reg{1,2} bring r5's id back and insn 7 is logged as "R5=scalar(id=1,...)". Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260904083325.2083493-4-eddyz87@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- tools/testing/selftests/bpf/progs/verifier_scalar_ids.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c b/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c index 663d15fc5fd2..256547048cc4 100644 --- a/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c +++ b/tools/testing/selftests/bpf/progs/verifier_scalar_ids.c @@ -380,13 +380,14 @@ SEC("socket") __success __log_level(2) __flag(BPF_F_TEST_STATE_FREQ) /* - * check that r0 and r5 have different IDs after 'if', - * collect_linked_regs() can't tie more than 5 registers for a single insn. + * check that r5 is unlinked after 'if', collect_linked_regs() can't tie + * more than 5 registers for a single insn and the register compared by + * the jump is not exempt from that. */ -__msg("7: (25) if r0 > 0x7 goto pc+0 ; R0=scalar(id=1") +__msg("7: (25) if r5 > 0x7 goto pc+0 ; R5=scalar(smin=") __msg("12: (bf) r5 = r5 ; R5=scalar(id=2") /* check that r{0-4} are marked precise after 'if' */ -__msg("frame0: regs=r0 stack= before 7: (25) if r0 > 0x7 goto pc+0") +__msg("frame0: regs=r0 stack= before 7: (25) if r5 > 0x7 goto pc+0") __msg("frame0: parent state regs=r0,r1,r2,r3,r4 stack=:") __naked void linked_regs_too_many_regs(void) { @@ -400,8 +401,8 @@ __naked void linked_regs_too_many_regs(void) "r3 = r0;" "r4 = r0;" "r5 = r0;" - /* propagate range for r{0-5} */ - "if r0 > 7 goto +0;" + /* r{0-4} fill the record, r5 does not fit and is unlinked */ + "if r5 > 7 goto +0;" /* keep r{1-4} live */ "r1 = r1;" "r2 = r2;" -- cgit v1.2.3 From e51179a4e09846f8fd0f26a05068520de2b301bf Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 4 Sep 2026 01:33:22 -0700 Subject: bpf: Don't predict JMP32 pointer vs zero comparisons Consider the following program: r1 = map_value; /* low 32 bits are zero at runtime */ r6 = 0xdead000000000000; if w1 != 0 goto l1; l0: r1 += r6; r2 = *(u64 *)(r1 + 0); exit; l1: r6 = 0; goto l0; At the moment is_branch_taken() reports the jump as always taken, because it does not distinguish between BPF_JMP and BPF_JMP32 comparisons when processing 'if w1 != 0 ...'. Fixes: cac616db39c2 ("bpf: Verifier track null pointer branch_taken with JNE and JEQ") Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260904083325.2083493-5-eddyz87@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 1fb0c832611c..303368460ec1 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -16337,6 +16337,13 @@ static int is_branch_taken(struct bpf_verifier_env *env, struct bpf_reg_state *r if (__is_pointer_value(false, reg1) || __is_pointer_value(false, reg2)) { u64 val; + /* + * The low 32 bits of a valid pointer may well be zero, hence + * nothing below applies to a 32-bit comparison. + */ + if (is_jmp32) + return -1; + /* arrange that reg2 is a scalar, and reg1 is a pointer */ if (!is_reg_const(reg2, is_jmp32)) { opcode = flip_opcode(opcode); -- cgit v1.2.3 From 836b2fe544a5e9b5ce116622cb36fba33838c6fd Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 4 Sep 2026 01:33:23 -0700 Subject: selftests/bpf: Check that JMP32 pointer vs zero jumps are not predicted Add jmp32_ptr_vs_zero_jne: the fall-through of the 32-bit compare, which the verifier used to skip, contains an out of bounds map value access, hence w/o the previous patch the program is accepted. See previous patch for detailed description. Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260904083325.2083493-6-eddyz87@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../bpf/progs/verifier_jeq_infer_not_null.c | 27 ++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_jeq_infer_not_null.c b/tools/testing/selftests/bpf/progs/verifier_jeq_infer_not_null.c index 8657e4a0d601..410acbf658c7 100644 --- a/tools/testing/selftests/bpf/progs/verifier_jeq_infer_not_null.c +++ b/tools/testing/selftests/bpf/progs/verifier_jeq_infer_not_null.c @@ -391,6 +391,33 @@ __naked void bounded_offset_infers_map_value_non_null(void) : __clobber_all); } +/* + * The low 32 bits of a map value pointer may be zero, hence a 32-bit + * compare with zero cannot be predicted from the pointer being non-NULL + * and both successors of such a jump have to be verified. + */ +SEC("socket") +__failure __msg("invalid access to map value, value_size=4 off=32 size=4") +__naked void jmp32_ptr_vs_zero_jne(void) +{ + asm volatile (" \ + /* r0 = bpf_map_lookup_elem(map_hash, &key); */ \ + *(u64 *)(r10 - 8) = 0; \ + r1 = %[map_hash] ll; \ + r2 = r10; \ + r2 += -8; \ + call %[bpf_map_lookup_elem]; \ + if r0 == 0 goto 1f; \ + if w0 != 0 goto 1f; \ + r0 = *(u32 *)(r0 + 32); \ +1: r0 = 0; \ + exit; \ +" : + : __imm(bpf_map_lookup_elem), + __imm_addr(map_hash) + : __clobber_all); +} + void kfunc_root(void) { bpf_rdonly_cast(0, 0); -- cgit v1.2.3 From 6aed0134d3cda6382385a734ae0158eb7df6b142 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 4 Sep 2026 01:33:24 -0700 Subject: bpf: Mark the zero register precise for a register-form NULL check check_cond_jmp_op() accepts "if rA rB" as a NULL check for a nullable pointer rA when rB is a scalar known to be zero, lifts PTR_MAYBE_NULL from rA in the corresponding branch and does not mark rB precise. Consider the following program: r0 = bpf_get_prandom_u32(); r6 = 1; /* the r6 == 0 path is explored first */ if (r0 == 0) goto 1f; r6 = 0; 1: r0 = bpf_map_lookup_elem(map, &0); /* absent, NULL at runtime */ if (r0 == r6) goto 2f; /* taken as a NULL check for r0 */ *(u8 *)(r0 + 0); /* verifier: map value; runtime: zero */ 2: return 0; The r6 == 0 path is explored first and the dereference is accepted. The r6 == 1 path is pruned at the checkpoint recorded for (1), so the comparison is never verified with a non-zero r6. At runtime a failed lookup returns NULL, NULL != 1 takes the non-NULL edge and the program dereferences a pointer that is zero. Fixes: 2f4cb53eed44 ("bpf: detect non null pointer with register operand in JEQ/JNE.") Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260904083325.2083493-7-eddyz87@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- kernel/bpf/verifier.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 303368460ec1..fde5d046b6e3 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -17052,6 +17052,15 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env, type_may_be_null(dst_reg->type) && ((BPF_SRC(insn->code) == BPF_K && insn->imm == 0) || (BPF_SRC(insn->code) == BPF_X && bpf_register_is_null(src_reg)))) { + /* + * For BPF_X the zero is a property of this execution path, + * hence src_reg has to be precise. + */ + if (BPF_SRC(insn->code) == BPF_X) { + err = mark_chain_precision(env, insn->src_reg); + if (err) + return err; + } /* Mark all identical registers in each branch as either * safe or unknown depending R == 0 or R != 0 conditional. */ -- cgit v1.2.3 From 6b31560c6bc1a8a7a70792c7b3ca4c1ea322063b Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 4 Sep 2026 01:33:25 -0700 Subject: selftests/bpf: No non-NULL inference from an imprecise zero register Check that a register-form NULL check does not lift PTR_MAYBE_NULL on a path where the compared register is non-zero. W/o the previous patch the program is accepted. Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/bpf/20260904083325.2083493-8-eddyz87@gmail.com Signed-off-by: Kumar Kartikeya Dwivedi --- .../bpf/progs/verifier_jeq_infer_not_null.c | 34 ++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_jeq_infer_not_null.c b/tools/testing/selftests/bpf/progs/verifier_jeq_infer_not_null.c index 410acbf658c7..3c789c565b18 100644 --- a/tools/testing/selftests/bpf/progs/verifier_jeq_infer_not_null.c +++ b/tools/testing/selftests/bpf/progs/verifier_jeq_infer_not_null.c @@ -418,6 +418,40 @@ __naked void jmp32_ptr_vs_zero_jne(void) : __clobber_all); } +/* + * The below program is explored in two paths: r6 == 0 and r6 == 1. + * On the first path comparison "if r0 == r6 goto 2f" should mark r6 as precise, + * otherwise unsafe path with r6 == 1 would be incorrectly pruned. + */ +SEC("socket") +__failure +__flag(BPF_F_TEST_STATE_FREQ) +__msg("error: invalid dereference of R0 (a nullable map value pointer)") +__naked void imprecise_zero_does_not_infer_map_value_non_null(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + /* r6 is 0 on the path explored first, 1 on the other */\ + r6 = 1; \ + if r0 == 0 goto 1f; \ + r6 = 0; \ + /* r0 = bpf_map_lookup_elem(map_hash, &0); */ \ +1: *(u64 *)(r10 - 8) = 0; \ + r1 = %[map_hash] ll; \ + r2 = r10; \ + r2 += -8; \ + call %[bpf_map_lookup_elem]; \ + if r0 == r6 goto 2f; \ + r0 = *(u8 *)(r0 + 0); \ +2: r0 = 0; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32), + __imm(bpf_map_lookup_elem), + __imm_addr(map_hash) + : __clobber_all); +} + void kfunc_root(void) { bpf_rdonly_cast(0, 0); -- cgit v1.2.3 From 048029ba1c793f8cabc4ad5eea765da01903f8f1 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Fri, 4 Sep 2026 10:43:14 +0200 Subject: bpf: Require MEM_PERCPU for percpu kptr stores map_kptr_match_type() treats perm_flags as the set of register type flags that a kptr field permits. Adding MEM_PERCPU to that set for BPF_KPTR_PERCPU does not require the source register to carry it, however. The subset test consequently accepts both a plain bpf_obj_new() allocation and a referenced kernel pointer into a __percpu_kptr map field. Loads from the field are always marked MEM_PERCPU. Consumers then treat the stored value as the cookie returned by bpf_percpu_obj_new(): per-CPU pointer helpers relocate it, and map teardown selects the per-CPU free path. A plain allocation can therefore provide an arbitrary kernel read/write, while a kernel pointer can be relocated into an invalid address or sent through a missing destructor. Require the source MEM_PERCPU flag to match the destination field kind. This preserves valid bpf_percpu_obj_new() stores and rejects both the program-BTF and kernel-BTF variants. Fixes: 36d8bdf75a93 ("bpf: Add alloc/xchg/direct_access support for local percpu kptr") Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260904084325.52250-2-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/verifier.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index fde5d046b6e3..19c932e8c533 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4495,6 +4495,13 @@ static int map_kptr_match_type(struct bpf_verifier_env *env, if (type_flag(reg->type) & ~perm_flags) goto bad_type; + /* + * A BPF_KPTR_PERCPU field is read back as MEM_PERCPU, so the value + * stored in it must carry the same flag. + */ + if ((kptr_field->type == BPF_KPTR_PERCPU) != !!(reg->type & MEM_PERCPU)) + goto bad_type; + /* We need to verify reg->type and reg->btf, before accessing reg->btf */ reg_name = btf_type_name(reg->btf, reg->btf_id); -- cgit v1.2.3 From 17487b31f479c85eda3685e8e44242358bd68f23 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Fri, 4 Sep 2026 10:43:15 +0200 Subject: selftests/bpf: Reject non-percpu values in percpu kptr fields Add verifier coverage for the two ways a non-percpu pointer can be stored in a __percpu_kptr field: a program-BTF local allocation returned by bpf_obj_new(), and a referenced kernel-BTF task_struct pointer. Without the verifier fix, both programs are unexpectedly accepted and the negative tests fail. Requiring MEM_PERCPU makes both programs fail verification with the expected invalid-kptr diagnostic. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260904084325.52250-3-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- .../selftests/bpf/progs/percpu_alloc_fail.c | 59 ++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/percpu_alloc_fail.c b/tools/testing/selftests/bpf/progs/percpu_alloc_fail.c index 08379c3b6a03..3701f4ea58c7 100644 --- a/tools/testing/selftests/bpf/progs/percpu_alloc_fail.c +++ b/tools/testing/selftests/bpf/progs/percpu_alloc_fail.c @@ -33,6 +33,20 @@ struct { __type(value, struct elem); } array SEC(".maps"); +struct kernel_percpu_elem { + struct task_struct __percpu_kptr *task; +}; + +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __uint(max_entries, 1); + __type(key, int); + __type(value, struct kernel_percpu_elem); +} kernel_percpu_array SEC(".maps"); + +struct task_struct *bpf_task_from_pid(s32 pid) __ksym; +void bpf_task_release(struct task_struct *p) __ksym; + long ret; SEC("?fentry/bpf_fentry_test1") @@ -137,6 +151,51 @@ int BPF_PROG(test_array_map_5) return 0; } +SEC("?syscall") +__failure __msg("invalid kptr access, R2 type=trusted_ptr_ expected=ptr_task_struct") +int reject_kernel_ptr_into_percpu_kptr(void *ctx) +{ + struct kernel_percpu_elem *e; + struct task_struct *p, *old; + int index = 0; + + e = bpf_map_lookup_elem(&kernel_percpu_array, &index); + if (!e) + return 0; + + p = bpf_task_from_pid(1); + if (!p) + return 0; + + old = bpf_kptr_xchg(&e->task, p); + if (old) + bpf_task_release(old); + return 0; +} + +SEC("?fentry.s/bpf_fentry_test1") +__failure __msg("invalid kptr access, R2 type=ptr_ expected=ptr_val_t") +int BPF_PROG(reject_plain_alloc_into_percpu_kptr) +{ + struct val_t __percpu_kptr *old; + struct val_t *p; + struct elem *e; + int index = 0; + + e = bpf_map_lookup_elem(&array, &index); + if (!e) + return 0; + + p = bpf_obj_new(struct val_t); + if (!p) + return 0; + + old = bpf_kptr_xchg(&e->pc, p); + if (old) + bpf_percpu_obj_drop(old); + return 0; +} + SEC("?fentry.s/bpf_fentry_test1") __failure __msg("bpf_percpu_obj_new type ID argument must be of a struct of scalars") int BPF_PROG(test_array_map_6) -- cgit v1.2.3 From dc36739e5cc9f60485418a910b42bc95339218d2 Mon Sep 17 00:00:00 2001 From: Ning Ding Date: Fri, 4 Sep 2026 10:43:16 +0200 Subject: bpf: Keep refcount_acquire nullable for borrowed RCU kptrs bpf_refcount_acquire() is fallible for a borrowed reference because the object may have reached a zero refcount. The verifier therefore keeps KF_RET_NULL on the return value unless the argument is an owning reference. An RCU-protected load of a local kptr is marked MEM_ALLOC, but it only receives NON_OWN_REF when the pointee contains a graph node. A refcounted object without a graph node consequently looks like an owning reference even though the loaded register has no acquired reference state. If the program drops the last real reference while remaining in the RCU critical section, refcount_inc_not_zero() returns NULL while the verifier treats the result as non-NULL. Only classify the argument as owning when it is backed by a verifier-tracked reference. This retains the non-NULL return for pointers from bpf_obj_new(), bpf_kptr_xchg(), or an earlier successful acquisition, while requiring a NULL check for borrowed RCU kptrs. Fixes: 1b12171533a9 ("bpf: Mark direct ld of stashed bpf_{rb,list}_node as non-owning ref") Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Ning Ding [ kkd: Rewrote commit log ] Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260904084325.52250-4-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/verifier.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 19c932e8c533..3af8bd838b82 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -13185,7 +13185,7 @@ check_ok: bpf_diag_reg_type_plain(env, reg->type)); return -EINVAL; } - if (!type_is_non_owning_ref(reg->type)) + if (!type_is_non_owning_ref(reg->type) && reg_is_referenced(env, reg)) meta->arg_owning_ref = true; rec = reg_btf_record(reg); -- cgit v1.2.3 From 2edd8339468e4bf0feecb3398aaad25fd7b84286 Mon Sep 17 00:00:00 2001 From: Ning Ding Date: Fri, 4 Sep 2026 10:43:17 +0200 Subject: selftests/bpf: Test borrowed refcount acquisition nullability Add verifier coverage for the distinction between owning and borrowed arguments to bpf_refcount_acquire(). An owning pointer returned by bpf_obj_new() must continue producing a non-NULL result without an extra check. An RCU-loaded local kptr is only borrowed, so a checked result must load successfully while passing an unchecked result to bpf_obj_drop() must be rejected as possibly NULL. Use a sleepable syscall program for the borrowed cases so the explicit RCU critical section is what permits the local kptr load. Without the verifier fix, the unchecked case is incorrectly accepted. With it, the verifier rejects the possibly NULL argument. Signed-off-by: Ning Ding [ kkd: Rewrote commit log ] Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260904084325.52250-5-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- .../testing/selftests/bpf/progs/refcounted_kptr.c | 61 ++++++++++++++++++++++ .../selftests/bpf/progs/refcounted_kptr_fail.c | 48 +++++++++++++++++ 2 files changed, 109 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/refcounted_kptr.c b/tools/testing/selftests/bpf/progs/refcounted_kptr.c index 61906f48025c..cae00f7b0a24 100644 --- a/tools/testing/selftests/bpf/progs/refcounted_kptr.c +++ b/tools/testing/selftests/bpf/progs/refcounted_kptr.c @@ -23,6 +23,15 @@ struct map_value { struct node_data __kptr *node; }; +struct node_refcount_only { + long key; + struct bpf_refcount refcount; +}; + +struct map_value_refcount_only { + struct node_refcount_only __kptr *node; +}; + struct { __uint(type, BPF_MAP_TYPE_ARRAY); __type(key, int); @@ -30,6 +39,13 @@ struct { __uint(max_entries, 2); } stashed_nodes SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __type(key, int); + __type(value, struct map_value_refcount_only); + __uint(max_entries, 1); +} stashed_refcount_only SEC(".maps"); + struct node_acquire { long key; long data; @@ -832,6 +848,51 @@ long rbtree_refcounted_node_ref_escapes_owning_input(void *ctx) return 0; } +SEC("tc") +__success +long refcount_acquire_owning_input_no_null_check(void *ctx) +{ + struct node_refcount_only *n, *m; + + n = bpf_obj_new(typeof(*n)); + if (!n) + return 1; + + m = bpf_refcount_acquire(n); + bpf_obj_drop(m); + bpf_obj_drop(n); + + return 0; +} + +SEC("?syscall") +__success +long refcount_acquire_rcu_map_kptr_null_checked(void *ctx) +{ + struct map_value_refcount_only *mapval; + struct node_refcount_only *n, *m; + int idx = 0; + + mapval = bpf_map_lookup_elem(&stashed_refcount_only, &idx); + if (!mapval) + return 1; + + bpf_rcu_read_lock(); + n = mapval->node; + if (!n) { + bpf_rcu_read_unlock(); + return 2; + } + m = bpf_refcount_acquire(n); + bpf_rcu_read_unlock(); + + if (!m) + return 3; + bpf_obj_drop(m); + + return 0; +} + static long __stash_map_empty_xchg(struct node_data *n, int idx) { struct map_value *mapval = bpf_map_lookup_elem(&stashed_nodes, &idx); diff --git a/tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c b/tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c index eaaed0859f94..7d2f8897e5ad 100644 --- a/tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c +++ b/tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c @@ -19,6 +19,15 @@ struct node_refcounted { struct bpf_refcount refcount; }; +struct node_refcount_only { + long key; + struct bpf_refcount refcount; +}; + +struct map_value_refcount_only { + struct node_refcount_only __kptr *node; +}; + extern void bpf_rcu_read_lock(void) __ksym; extern void bpf_rcu_read_unlock(void) __ksym; @@ -28,6 +37,13 @@ private(A) struct bpf_rb_root groot __contains(node_acquire, node); private(B) struct bpf_spin_lock lock; private(B) struct bpf_list_head head __contains(node_refcounted, list); +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __type(key, int); + __type(value, struct map_value_refcount_only); + __uint(max_entries, 1); +} stashed_refcount_only SEC(".maps"); + static bool less(struct bpf_rb_node *a, const struct bpf_rb_node *b) { struct node_acquire *node_a; @@ -89,6 +105,38 @@ long refcount_acquire_non_object(void *ctx) return bpf_refcount_acquire(ctx) != NULL; } +SEC("?syscall") +__failure __msg("Possibly NULL pointer passed to trusted R1") +long refcount_acquire_rcu_map_kptr_unchecked_drop(void *ctx) +{ + struct map_value_refcount_only *mapval; + struct node_refcount_only *tmp, *n, *m; + int idx = 0; + + /* Force Clang to emit complete BTF for struct node_refcount_only. */ + tmp = bpf_obj_new(typeof(*tmp)); + if (!tmp) + return 3; + bpf_obj_drop(tmp); + + mapval = bpf_map_lookup_elem(&stashed_refcount_only, &idx); + if (!mapval) + return 1; + + bpf_rcu_read_lock(); + n = mapval->node; + if (!n) { + bpf_rcu_read_unlock(); + return 2; + } + m = bpf_refcount_acquire(n); + bpf_rcu_read_unlock(); + + bpf_obj_drop(m); + + return 0; +} + SEC("?tc") __failure __msg("Unreleased reference id=3 alloc_insn={{[0-9]+}}") long rbtree_refcounted_node_ref_escapes_owning_input(void *ctx) -- cgit v1.2.3 From cd6f72d7f38e10aa82fcbc745a6a9e58e0d8e366 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Fri, 4 Sep 2026 10:43:18 +0200 Subject: bpf: Clear NON_OWN_REF after RCU protection ends A local kptr load of an object containing a graph node is marked MEM_RCU and NON_OWN_REF while protected by RCU. When the last RCU read-side critical section ends, invalidate_rcu_protected_refs() removes MEM_RCU and marks the pointer PTR_UNTRUSTED, but leaves NON_OWN_REF set. The stale flag lets graph kfunc argument checks continue treating the pointer as a live borrowed reference. In particular, bpf_rbtree_remove() can accept a pointer after its protection ended and return it as a new owning reference, even though the object may already have been freed. Clear NON_OWN_REF when an RCU-protected pointer is demoted. A spin lock also provides implicit RCU protection, so invalidate non-owning references before demoting RCU-protected pointers when releasing the lock. Otherwise the demotion would clear the flag before invalidate_non_owning_refs() can find and invalidate those aliases. The demoted pointer remains available for fault-protected reads. Exempt such reads from the allocated-object reference-state assertion; writes through a fault-prone pointer are already rejected, and bpf_may_fault_on_deref() makes the surviving loads use BPF_PROBE_MEM. Fixes: 1b12171533a9 ("bpf: Mark direct ld of stashed bpf_{rb,list}_node as non-owning ref") Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260904084325.52250-6-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/verifier.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 3af8bd838b82..9c6ad157a61e 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6045,7 +6045,13 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, return -EACCES; } - if (type_is_alloc(reg->type) && !type_is_non_owning_ref(reg->type) && + /* + * A fault-prone allocated object may still be read through a + * BPF_PROBE_MEM load after its lifetime protection ends. Writes + * through such pointers were rejected above. + */ + if (type_is_alloc(reg->type) && !bpf_may_fault_on_deref(reg->type) && + !type_is_non_owning_ref(reg->type) && !(reg->type & MEM_RCU) && !reg_is_referenced(env, reg)) { verifier_bug(env, "allocated object must have a referenced id"); return -EFAULT; @@ -7423,10 +7429,14 @@ static int process_spin_lock(struct bpf_verifier_env *env, struct bpf_reg_state lock); return -EINVAL; } + /* + * Invalidate non-owning refs before RCU demotion clears their + * NON_OWN_REF flag. + */ + invalidate_non_owning_refs(env); + if (!in_rcu_cs(env)) invalidate_rcu_protected_refs(env); - - invalidate_non_owning_refs(env); } return 0; } @@ -9526,7 +9536,7 @@ static void invalidate_rcu_protected_refs(struct bpf_verifier_env *env) bpf_for_each_reg_in_vstate_mask(env->cur_state, state, reg, stack, clear_mask, ({ if (reg->type & MEM_RCU) { bpf_diag_mod_begin(env, reg, NULL, BPF_DIAG_MOD_WRITE); - reg->type &= ~(MEM_RCU | PTR_MAYBE_NULL); + reg->type &= ~(MEM_RCU | PTR_MAYBE_NULL | NON_OWN_REF); reg->type |= PTR_UNTRUSTED; bpf_diag_mod_end(env); } -- cgit v1.2.3 From 6668ed271eaefaa63e686bdfbedaeb7b8e492722 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Fri, 4 Sep 2026 10:43:19 +0200 Subject: selftests/bpf: Reject graph kptr use after RCU unlock Add a sleepable verifier test that loads a graph-node local kptr in an explicit RCU read-side critical section, then passes its node to bpf_rbtree_remove() after the section ends. Before the verifier fix, the stale NON_OWN_REF flag makes the node look like a live borrowed reference and the program is accepted. After the fix, the pointer is demoted without NON_OWN_REF and the graph kfunc argument is rejected. Also exercise a graph kptr loaded while a spin lock provides implicit RCU protection. The pointer must be invalidated when the lock is released, which guards the required ordering between non-owning-reference invalidation and RCU demotion. Update the existing fault-protected load test state description. The post-unlock pointer no longer carries NON_OWN_REF, but remains readable because the load is rewritten to use BPF_PROBE_MEM. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260904084325.52250-7-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- tools/testing/selftests/bpf/progs/rcu_read_lock.c | 6 +- .../selftests/bpf/progs/refcounted_kptr_fail.c | 75 ++++++++++++++++++++++ 2 files changed, 78 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/bpf/progs/rcu_read_lock.c b/tools/testing/selftests/bpf/progs/rcu_read_lock.c index 31d4081c3a9f..cdb255addbc3 100644 --- a/tools/testing/selftests/bpf/progs/rcu_read_lock.c +++ b/tools/testing/selftests/bpf/progs/rcu_read_lock.c @@ -592,9 +592,9 @@ int non_own_ref_untrusted_ld(void *ctx) } bpf_rcu_read_unlock(); /* - * The unlock leaves node as PTR_TO_BTF_ID | MEM_ALLOC | PTR_UNTRUSTED - * | NON_OWN_REF, and the load below has to get the BPF_PROBE_MEM - * rewrite for it, otherwise a bad address panics the kernel. + * The unlock leaves node as PTR_TO_BTF_ID | MEM_ALLOC | PTR_UNTRUSTED, + * and the load below has to get the BPF_PROBE_MEM rewrite for it, + * otherwise a bad address panics the kernel. */ non_own_ref_key = node->key; return 0; diff --git a/tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c b/tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c index 7d2f8897e5ad..f787ecf189d8 100644 --- a/tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c +++ b/tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c @@ -28,6 +28,17 @@ struct map_value_refcount_only { struct node_refcount_only __kptr *node; }; +struct rcu_graph_node { + struct bpf_rb_node node; + long data; +}; + +struct rcu_graph_node *just_here_because_btf_bug; + +struct map_value_rcu_graph { + struct rcu_graph_node __kptr *node; +}; + extern void bpf_rcu_read_lock(void) __ksym; extern void bpf_rcu_read_unlock(void) __ksym; @@ -36,6 +47,8 @@ private(A) struct bpf_spin_lock glock; private(A) struct bpf_rb_root groot __contains(node_acquire, node); private(B) struct bpf_spin_lock lock; private(B) struct bpf_list_head head __contains(node_refcounted, list); +private(C) struct bpf_spin_lock graph_lock; +private(C) struct bpf_rb_root graph_root __contains(rcu_graph_node, node); struct { __uint(type, BPF_MAP_TYPE_ARRAY); @@ -44,6 +57,13 @@ struct { __uint(max_entries, 1); } stashed_refcount_only SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_ARRAY); + __type(key, int); + __type(value, struct map_value_rcu_graph); + __uint(max_entries, 1); +} stashed_rcu_graph SEC(".maps"); + static bool less(struct bpf_rb_node *a, const struct bpf_rb_node *b) { struct node_acquire *node_a; @@ -137,6 +157,61 @@ long refcount_acquire_rcu_map_kptr_unchecked_drop(void *ctx) return 0; } +SEC("?syscall") +__failure +__msg("bpf_rbtree_remove can only take non-owning or refcounted " + "bpf_rb_node pointer") +long rbtree_remove_after_rcu_unlock(void *ctx) +{ + struct map_value_rcu_graph *mapval; + struct bpf_rb_node *rb_node; + struct rcu_graph_node *node; + int idx = 0; + + mapval = bpf_map_lookup_elem(&stashed_rcu_graph, &idx); + if (!mapval) + return 0; + + bpf_rcu_read_lock(); + node = mapval->node; + if (!node) { + bpf_rcu_read_unlock(); + return 0; + } + bpf_rcu_read_unlock(); + + bpf_spin_lock(&graph_lock); + rb_node = bpf_rbtree_remove(&graph_root, &node->node); + bpf_spin_unlock(&graph_lock); + if (rb_node) + bpf_obj_drop(container_of(rb_node, struct rcu_graph_node, node)); + + return 0; +} + +SEC("?syscall") +__failure __msg("invalid mem access 'scalar'") +long graph_kptr_after_spin_unlock(void *ctx) +{ + struct map_value_rcu_graph *mapval; + struct rcu_graph_node *node; + int idx = 0; + + mapval = bpf_map_lookup_elem(&stashed_rcu_graph, &idx); + if (!mapval) + return 0; + + bpf_spin_lock(&graph_lock); + node = mapval->node; + if (!node) { + bpf_spin_unlock(&graph_lock); + return 0; + } + bpf_spin_unlock(&graph_lock); + + return node->data; +} + SEC("?tc") __failure __msg("Unreleased reference id=3 alloc_insn={{[0-9]+}}") long rbtree_refcounted_node_ref_escapes_owning_input(void *ctx) -- cgit v1.2.3 From 7441ee8276641bddaf1cba7bb75ef9c1458ceb3b Mon Sep 17 00:00:00 2001 From: Ning Ding Date: Fri, 4 Sep 2026 10:43:20 +0200 Subject: bpf: Reject untrusted allocated-object pointers When the final RCU read-side critical section ends, a local kptr is demoted to PTR_UNTRUSTED but retains MEM_ALLOC. The pointer may be NULL or may refer to an object whose lifetime is no longer protected. type_is_ptr_alloc_obj() nevertheless recognizes any PTR_TO_BTF_ID with MEM_ALLOC as a live allocated object. In particular, a refcount-only local kptr never carries NON_OWN_REF, so it still passes the bpf_refcount_acquire() argument check after RCU protection ends. The kfunc can then dereference NULL or stale memory. Make type_is_ptr_alloc_obj() reject PTR_UNTRUSTED pointers. Since type_is_non_owning_ref() is based on the same predicate, graph kfunc arguments obey the same live-object requirement. Fault-protected reads of the demoted pointer remain valid: writes are already rejected, and read fixups use bpf_may_fault_on_deref() rather than this predicate. Fixes: 1b12171533a9 ("bpf: Mark direct ld of stashed bpf_{rb,list}_node as non-owning ref") Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Ning Ding [ kkd: Rewrote commit log ] Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260904084325.52250-8-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- include/linux/bpf_verifier.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 1339c2f028db..36b65797877d 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -1381,7 +1381,9 @@ static inline bool bpf_type_has_unsafe_modifiers(u32 type) static inline bool type_is_ptr_alloc_obj(u32 type) { - return base_type(type) == PTR_TO_BTF_ID && type_flag(type) & MEM_ALLOC; + return base_type(type) == PTR_TO_BTF_ID && + type_flag(type) & MEM_ALLOC && + !(type_flag(type) & PTR_UNTRUSTED); } static inline bool type_is_non_owning_ref(u32 type) -- cgit v1.2.3 From 9492baf8532ca285c58b82a269acd7a57e205ae9 Mon Sep 17 00:00:00 2001 From: Ning Ding Date: Fri, 4 Sep 2026 10:43:21 +0200 Subject: selftests/bpf: Reject refcount acquisition after RCU unlock Add a sleepable verifier test that loads a refcount-only local kptr in an explicit RCU read-side critical section, ends the section, and passes the pointer to bpf_refcount_acquire(). The loaded pointer never carries NON_OWN_REF. After RCU unlock it retains MEM_ALLOC while becoming PTR_UNTRUSTED, which previously made the kfunc argument check accept it as a live allocated object. Expect verification to reject the untrusted argument instead. Signed-off-by: Ning Ding [ kkd: Rewrote commit log ] Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260904084325.52250-9-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- .../selftests/bpf/progs/refcounted_kptr_fail.c | 27 ++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c b/tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c index f787ecf189d8..338e43822ffe 100644 --- a/tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c +++ b/tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c @@ -189,6 +189,33 @@ long rbtree_remove_after_rcu_unlock(void *ctx) return 0; } +SEC("?syscall") +__failure __msg("R1 is neither owning or non-owning ref") +long refcount_acquire_after_rcu_unlock(void *ctx) +{ + struct map_value_refcount_only *mapval; + struct node_refcount_only *node, *ref; + int idx = 0; + + mapval = bpf_map_lookup_elem(&stashed_refcount_only, &idx); + if (!mapval) + return 0; + + bpf_rcu_read_lock(); + node = mapval->node; + if (!node) { + bpf_rcu_read_unlock(); + return 0; + } + bpf_rcu_read_unlock(); + + ref = bpf_refcount_acquire(node); + if (ref) + bpf_obj_drop(ref); + + return 0; +} + SEC("?syscall") __failure __msg("invalid mem access 'scalar'") long graph_kptr_after_spin_unlock(void *ctx) -- cgit v1.2.3 From 5e8c349bc8d790fe031a4332e502f5d4f9878644 Mon Sep 17 00:00:00 2001 From: Jiayuan Chen Date: Fri, 4 Sep 2026 15:37:39 +0800 Subject: selftests/bpf: Fix flaky bpf_nf test when random NAT port is 0 The bpf_nf test allocs a ct, sets snat and dnat with random addr and port via bpf_ct_set_nat_info(), then looks the ct up and checks the reply tuple against what was set. The port comes from bpf_get_prandom_u32() and can be 0. For bpf_ct_set_nat_info(), port 0 means "port not specified", so only the addr is mapped and the kernel keeps the original port. The check then compares that port with 0 and fails, which shows up as a flaky "Test for source natting" failure in CI [1][2]. Keep the random port in 1..65535 so it is always specified. [1] https://github.com/kernel-patches/bpf/actions/runs/33830002889/job/100893868791 [2] https://github.com/kernel-patches/bpf/actions/runs/33829976794/job/100893220999 Fixes: b06b45e82b59 ("selftests/bpf: add tests for bpf_ct_set_nat_info kfunc") Signed-off-by: Jiayuan Chen Link: https://lore.kernel.org/r/20260904073745.363314-1-jiayuan.chen@linux.dev Signed-off-by: Alexei Starovoitov --- tools/testing/selftests/bpf/progs/test_bpf_nf.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/bpf/progs/test_bpf_nf.c b/tools/testing/selftests/bpf/progs/test_bpf_nf.c index df43649ecb78..eda9b7bbab75 100644 --- a/tools/testing/selftests/bpf/progs/test_bpf_nf.c +++ b/tools/testing/selftests/bpf/progs/test_bpf_nf.c @@ -190,8 +190,8 @@ nf_ct_test(struct nf_conn *(*lookup_fn)(void *, struct bpf_sock_tuple *, u32, ct = alloc_fn(ctx, &bpf_tuple, sizeof(bpf_tuple.ipv4), &opts_def, sizeof(opts_def)); if (ct) { - __u16 sport = bpf_get_prandom_u32(); - __u16 dport = bpf_get_prandom_u32(); + __u16 sport = bpf_get_prandom_u32() % 65535 + 1; + __u16 dport = bpf_get_prandom_u32() % 65535 + 1; union nf_inet_addr saddr = {}; union nf_inet_addr daddr = {}; struct nf_conn *ct_ins; @@ -293,8 +293,8 @@ nf_ct_opts_new_test(struct nf_conn *(*lookup_fn)(void *, struct bpf_sock_tuple * ct = alloc_fn(ctx, &bpf_tuple, sizeof(bpf_tuple.ipv4), &opts_def, sizeof(opts_def)); if (ct) { - __u16 sport = bpf_get_prandom_u32(); - __u16 dport = bpf_get_prandom_u32(); + __u16 sport = bpf_get_prandom_u32() % 65535 + 1; + __u16 dport = bpf_get_prandom_u32() % 65535 + 1; union nf_inet_addr saddr = {}; union nf_inet_addr daddr = {}; struct nf_conn *ct_ins; -- cgit v1.2.3 From 5df46ddcb7b36878c1b691e9057a0509042a2567 Mon Sep 17 00:00:00 2001 From: Yuan Chen Date: Fri, 4 Sep 2026 12:41:52 +0200 Subject: bpf: Preserve special fields in recycled rhtab elements rhtab_map_update_elem() initializes special fields after obtaining an element from bpf_mem_cache_alloc(). The allocator can return a fresh, zeroed unit, or recycle one from its RCU-pending lists before the registered destructor has run. A BPF program can retain a map-value pointer after deleting its element and initialize and arm a timer through that pointer. If the deleted unit is recycled, check_and_init_map_value() clears the only pointer to the timer. Neither a later deletion nor rhtab_mem_dtor() can then cancel it, and the callback can run with its key and value pointing into freed memory. Do not reinitialize special fields on insertion. Fresh allocator units are already zeroed. For recycled units, the special fields are ownership state that must remain visible to the eventual destructor. copy_map_value() already skips those fields, matching the non-preallocated hash-map path and the lifecycle established by commit 275c30bcee66 ("bpf: Don't reinit map value in prealloc_lru_pop"). Fixes: 6905f8601298 ("bpf: Allow special fields in resizable hashtab") Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Yuan Chen [ kkd: Split out the fix and rewrote the commit log ] Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260904104203.345917-2-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/hashtab.c | 1 - 1 file changed, 1 deletion(-) diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index e89fde188389..527cc5716ee8 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -3070,7 +3070,6 @@ static long rhtab_map_update_elem(struct bpf_map *map, void *key, void *value, u memcpy(elem->data, key, map->key_size); copy_map_value(map, rhtab_elem_value(elem, map->key_size), value); - check_and_init_map_value(map, rhtab_elem_value(elem, map->key_size)); /* Prevent deadlock for NMI programs attempting to take bucket lock */ bpf_disable_instrumentation(); -- cgit v1.2.3 From dbf6806dc81553edbab72fcec9a6d637dedff2f4 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Fri, 4 Sep 2026 12:41:53 +0200 Subject: selftests/bpf: Test timer field on recycled rhtab element Exercise the rhtab special-field lifecycle with the sequence from the original report. A bpf_for_each_map_elem() callback deletes the sole element, then initializes and arms a timer through the callback value pointer while it remains valid. Use a one-element map and pin userspace and BPF execution to one CPU. Repeated delete-and-replace cycles drain the per-CPU allocator cache, and periodic RCU synchronization makes the deleted units available for recycling. After each replacement, a second BPF program calls bpf_timer_cancel() on its value. A successful cancellation proves both that a timer-bearing unit was recycled and that insertion preserved the timer field. Without the fix, insertion clears that field and cancellation keeps returning -EINVAL. A long expiration keeps the timer callback out of the test, so the regression is detected without accessing freed memory. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260904104203.345917-3-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- .../testing/selftests/bpf/prog_tests/rhash_timer.c | 141 +++++++++++++++++++++ tools/testing/selftests/bpf/progs/rhash_timer.c | 98 ++++++++++++++ 2 files changed, 239 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/rhash_timer.c create mode 100644 tools/testing/selftests/bpf/progs/rhash_timer.c diff --git a/tools/testing/selftests/bpf/prog_tests/rhash_timer.c b/tools/testing/selftests/bpf/prog_tests/rhash_timer.c new file mode 100644 index 000000000000..3aad9fc02e06 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/rhash_timer.c @@ -0,0 +1,141 @@ +// SPDX-License-Identifier: GPL-2.0 + +#define _GNU_SOURCE +#include + +#include +#include "rhash_timer.skel.h" + +#define MAX_ATTEMPTS 256 +#define RCU_SYNC_INTERVAL 64 + +static int pin_to_first_cpu(cpu_set_t *old_mask) +{ + cpu_set_t new_mask; + int cpu; + + if (sched_getaffinity(0, sizeof(*old_mask), old_mask)) + return -errno; + + for (cpu = 0; cpu < CPU_SETSIZE; cpu++) + if (CPU_ISSET(cpu, old_mask)) + break; + if (cpu == CPU_SETSIZE) + return -EINVAL; + + CPU_ZERO(&new_mask); + CPU_SET(cpu, &new_mask); + if (sched_setaffinity(0, sizeof(new_mask), &new_mask)) + return -errno; + return 0; +} + +static int update_timer_map(int map_fd, __u64 key) +{ + __u64 value[3] = {}; + + return bpf_map_update_elem(map_fd, &key, value, BPF_NOEXIST); +} + +static int run_prog(int prog_fd, struct bpf_test_run_opts *opts) +{ + int err; + + err = bpf_prog_test_run_opts(prog_fd, opts); + if (err) + return err; + return opts->retval; +} + +void test_rhash_timer(void) +{ + LIBBPF_OPTS(bpf_test_run_opts, opts); + struct rhash_timer *skel = NULL; + cpu_set_t old_mask; + int map_fd = -1, arm_fd, cancel_fd; + bool affinity_set = false; + __u64 key = 1; + int attempt, err; + + err = pin_to_first_cpu(&old_mask); + if (!ASSERT_OK(err, "pin_to_first_cpu")) + return; + affinity_set = true; + + skel = rhash_timer__open_and_load(); + if (!ASSERT_OK_PTR(skel, "open_and_load")) + goto out; + + map_fd = bpf_map__fd(skel->maps.timer_map); + if (!ASSERT_GE(map_fd, 0, "timer_map fd")) + goto out; + arm_fd = bpf_program__fd(skel->progs.arm_deleted_timer); + if (!ASSERT_GE(arm_fd, 0, "arm_deleted_timer fd")) + goto out; + cancel_fd = bpf_program__fd(skel->progs.cancel_recycled_timer); + if (!ASSERT_GE(cancel_fd, 0, "cancel_recycled_timer fd")) + goto out; + + err = update_timer_map(map_fd, key); + if (!ASSERT_OK(err, "seed_timer_map")) + goto out; + + for (attempt = 0; attempt < MAX_ATTEMPTS; attempt++) { + err = run_prog(arm_fd, &opts); + if (err) { + ASSERT_OK(err, "arm_deleted_timer"); + goto out; + } + if (skel->bss->armed != attempt + 1) { + ASSERT_EQ(skel->bss->armed, attempt + 1, "armed"); + goto out; + } + if (skel->bss->timer_init_err) { + ASSERT_OK(skel->bss->timer_init_err, "timer_init_err"); + goto out; + } + if (skel->bss->timer_set_callback_err) { + ASSERT_OK(skel->bss->timer_set_callback_err, + "timer_set_callback_err"); + goto out; + } + if (skel->bss->timer_start_err) { + ASSERT_OK(skel->bss->timer_start_err, "timer_start_err"); + goto out; + } + + if ((attempt + 1) % RCU_SYNC_INTERVAL == 0) { + err = kern_sync_rcu(); + if (err) { + ASSERT_OK(err, "kern_sync_rcu"); + goto out; + } + } + + err = update_timer_map(map_fd, ++key); + if (err) { + ASSERT_OK(err, "replace_timer_map"); + goto out; + } + + err = run_prog(cancel_fd, &opts); + if (err) { + ASSERT_OK(err, "cancel_recycled_timer"); + goto out; + } + if (skel->bss->timer_cancel_err) { + ASSERT_OK(skel->bss->timer_cancel_err, "timer_cancel_err"); + goto out; + } + if (skel->bss->cancelled) + break; + } + + ASSERT_GT(skel->bss->cancelled, 0, "preserved timer"); +out: + if (map_fd >= 0) + bpf_map_delete_elem(map_fd, &key); + rhash_timer__destroy(skel); + if (affinity_set) + sched_setaffinity(0, sizeof(old_mask), &old_mask); +} diff --git a/tools/testing/selftests/bpf/progs/rhash_timer.c b/tools/testing/selftests/bpf/progs/rhash_timer.c new file mode 100644 index 000000000000..2e06a463c605 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/rhash_timer.c @@ -0,0 +1,98 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include + +#define CLOCK_MONOTONIC 1 +#define TIMER_NSEC (60ULL * 1000 * 1000 * 1000) + +struct timer_value { + struct bpf_timer timer; + u64 data; +}; + +struct { + __uint(type, BPF_MAP_TYPE_RHASH); + __uint(map_flags, BPF_F_NO_PREALLOC); + __uint(max_entries, 1); + __type(key, u64); + __type(value, struct timer_value); +} timer_map SEC(".maps"); + +u64 armed; +u64 cancelled; +long timer_init_err; +long timer_set_callback_err; +long timer_start_err; +long timer_cancel_err; + +static int timer_cb(void *map, u64 *key, struct timer_value *value) +{ + return 0; +} + +static long arm_timer_cb(struct bpf_map *map, u64 *key, + struct timer_value *value, void *ctx) +{ + u64 key_copy = *key; + long err; + + err = bpf_map_delete_elem(map, &key_copy); + if (err) + return 1; + + err = bpf_timer_init(&value->timer, map, CLOCK_MONOTONIC); + if (err) { + timer_init_err = err; + return 1; + } + + err = bpf_timer_set_callback(&value->timer, timer_cb); + if (err) { + timer_set_callback_err = err; + return 1; + } + + err = bpf_timer_start(&value->timer, TIMER_NSEC, BPF_F_TIMER_CPU_PIN); + if (err) { + timer_start_err = err; + return 1; + } + + __sync_fetch_and_add(&armed, 1); + return 1; +} + +static long cancel_timer_cb(struct bpf_map *map, u64 *key, + struct timer_value *value, void *ctx) +{ + long err; + + err = bpf_timer_cancel(&value->timer); + if (err == -EINVAL) + return 1; + if (err < 0) { + timer_cancel_err = err; + return 1; + } + + __sync_fetch_and_add(&cancelled, 1); + return 1; +} + +SEC("syscall") +int arm_deleted_timer(void *ctx) +{ + bpf_for_each_map_elem(&timer_map, arm_timer_cb, NULL, 0); + return 0; +} + +SEC("syscall") +int cancel_recycled_timer(void *ctx) +{ + bpf_for_each_map_elem(&timer_map, cancel_timer_cb, NULL, 0); + return 0; +} + +char _license[] SEC("license") = "GPL"; -- cgit v1.2.3 From 65cc95eba9e8b46312cac38c227473605a4b996a Mon Sep 17 00:00:00 2001 From: Nuoqi Gui Date: Fri, 4 Sep 2026 12:41:54 +0200 Subject: bpf: Cancel special fields when recycling rhtab elements rhtab_map_update_existing() and rhtab_delete_elem() call bpf_obj_free_fields() when replacing or deleting a value. These map operations can run from BPF programs in NMI context, where releasing a referenced kptr or another complex field is not generally safe. Array and hash maps avoid that problem by cancelling only the asynchronous fields which can be stopped safely in the caller context. Other ownership state remains attached to the allocation until its memory allocator destructor performs the final cleanup. Use bpf_obj_cancel_fields() for the corresponding rhtab paths as well. This cancels timers, workqueues, and task work while allowing rhtab_mem_dtor() to release referenced kptrs when the allocation is eventually destroyed. Fixes: 6905f8601298 ("bpf: Allow special fields in resizable hashtab") Signed-off-by: Nuoqi Gui Acked-by: Mykyta Yatsenko [ kkd: Rebased, used direct helper calls, and rewrote the commit log ] Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260904104203.345917-4-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/hashtab.c | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-) diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 527cc5716ee8..cc60e99ffbe9 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -2868,16 +2868,6 @@ static int rhtab_map_alloc_check(union bpf_attr *attr) return htab_map_alloc_check(attr); } -static void rhtab_check_and_free_fields(struct bpf_rhtab *rhtab, - struct rhtab_elem *elem) -{ - if (IS_ERR_OR_NULL(rhtab->map.record)) - return; - - bpf_obj_free_fields(rhtab->map.record, - rhtab_elem_value(elem, rhtab->map.key_size)); -} - static void rhtab_mem_dtor(void *obj, void *ctx) { struct htab_btf_record *hrec = ctx; @@ -2967,8 +2957,8 @@ static int rhtab_delete_elem(struct bpf_rhtab *rhtab, struct rhtab_elem *elem, v rhtab_read_elem_value(&rhtab->map, copy, elem, flags); check_and_init_map_value(&rhtab->map, copy); } - /* Release internal structs: kptr, bpf_timer, task_work, wq */ - rhtab_check_and_free_fields(rhtab, elem); + bpf_obj_cancel_fields(&rhtab->map, + rhtab_elem_value(elem, rhtab->map.key_size)); bpf_mem_cache_free_rcu(&rhtab->ma, elem); return 0; } @@ -3009,7 +2999,6 @@ static int rhtab_map_lookup_and_delete_elem(struct bpf_map *map, void *key, void static long rhtab_map_update_existing(struct bpf_map *map, struct rhtab_elem *elem, void *value, u64 map_flags) { - struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); void *old_val = rhtab_elem_value(elem, map->key_size); if (map_flags & BPF_NOEXIST) @@ -3029,7 +3018,7 @@ static long rhtab_map_update_existing(struct bpf_map *map, struct rhtab_elem *el * kptrs/etc. still sit in the slot. Cancel them after the copy * to match arraymap's update semantics. */ - rhtab_check_and_free_fields(rhtab, elem); + bpf_obj_cancel_fields(map, old_val); return 0; } -- cgit v1.2.3 From 2b97956af60810cd382b86b9ce9aea421b889861 Mon Sep 17 00:00:00 2001 From: Nuoqi Gui Date: Fri, 4 Sep 2026 12:41:55 +0200 Subject: selftests/bpf: Test rhtab kptr cancellation semantics Resizable hash-map updates and deletions must not perform full special-field destruction in their caller context. In particular, a referenced kptr must remain attached to the allocation until the memory allocator destructor can release it safely. Add separate coverage for both affected paths. The update test stores a task kptr, replaces the ordinary value bytes with BPF_EXIST, and verifies that the kptr survived. The delete test removes an element and exchanges its kptr through the still-valid map-value pointer before the allocation is reclaimed. Both cases observe a NULL kptr when rhtab uses bpf_obj_free_fields(). They recover and release the reference after rhtab switches to cancellation semantics. Signed-off-by: Nuoqi Gui [ kkd: Split update and delete coverage and rewrote the commit log ] Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260904104203.345917-5-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- tools/testing/selftests/bpf/prog_tests/rhash.c | 6 ++ tools/testing/selftests/bpf/progs/rhash.c | 112 +++++++++++++++++++++++++ 2 files changed, 118 insertions(+) diff --git a/tools/testing/selftests/bpf/prog_tests/rhash.c b/tools/testing/selftests/bpf/prog_tests/rhash.c index 98bb66907b7f..0641bd5b0a9e 100644 --- a/tools/testing/selftests/bpf/prog_tests/rhash.c +++ b/tools/testing/selftests/bpf/prog_tests/rhash.c @@ -172,6 +172,12 @@ void test_rhash(void) if (test__start_subtest("test_rhash_delete_nonexistent")) rhash_run("test_rhash_delete_nonexistent"); + if (test__start_subtest("test_rhash_kptr_update")) + rhash_run("test_rhash_kptr_update"); + + if (test__start_subtest("test_rhash_kptr_delete")) + rhash_run("test_rhash_kptr_delete"); + if (test__start_subtest("test_rhash_map_extra_presize")) rhash_map_extra_presize(); diff --git a/tools/testing/selftests/bpf/progs/rhash.c b/tools/testing/selftests/bpf/progs/rhash.c index fc2dac3a719e..aea4de8dc781 100644 --- a/tools/testing/selftests/bpf/progs/rhash.c +++ b/tools/testing/selftests/bpf/progs/rhash.c @@ -19,6 +19,11 @@ struct elem { int val; }; +struct special_elem { + struct task_struct __kptr *task; + int val; +}; + struct { __uint(type, BPF_MAP_TYPE_RHASH); __uint(map_flags, BPF_F_NO_PREALLOC); @@ -27,6 +32,17 @@ struct { __type(value, struct elem); } rhmap SEC(".maps"); +struct { + __uint(type, BPF_MAP_TYPE_RHASH); + __uint(map_flags, BPF_F_NO_PREALLOC); + __uint(max_entries, 1); + __type(key, int); + __type(value, struct special_elem); +} special_fields SEC(".maps"); + +extern struct task_struct *bpf_task_acquire(struct task_struct *p) __ksym; +extern void bpf_task_release(struct task_struct *p) __ksym; + SEC("syscall") int test_rhash_lookup_update(void *ctx) { @@ -246,3 +262,99 @@ int test_rhash_delete_nonexistent(void *ctx) err = 0; return 0; } + +SEC("syscall") +int test_rhash_kptr_update(void *ctx) +{ + struct special_elem val1 = { .val = 1 }; + struct special_elem val2 = { .val = 2 }; + struct task_struct *task, *old; + struct special_elem *elem; + int key = 0; + + err = 1; + if (bpf_map_update_elem(&special_fields, &key, &val1, BPF_NOEXIST)) + return 1; + + err = 2; + elem = bpf_map_lookup_elem(&special_fields, &key); + if (!elem) + return 2; + + err = 3; + task = bpf_task_acquire(bpf_get_current_task_btf()); + if (!task) + return 3; + + err = 4; + old = bpf_kptr_xchg(&elem->task, task); + if (old) { + bpf_task_release(old); + return 4; + } + + err = 5; + if (bpf_map_update_elem(&special_fields, &key, &val2, BPF_EXIST)) + return 5; + + err = 6; + elem = bpf_map_lookup_elem(&special_fields, &key); + if (!elem || elem->val != 2) + return 6; + + err = 7; + old = bpf_kptr_xchg(&elem->task, NULL); + if (!old) + return 7; + bpf_task_release(old); + + err = 8; + if (bpf_map_delete_elem(&special_fields, &key)) + return 8; + + err = 0; + return 0; +} + +SEC("syscall") +int test_rhash_kptr_delete(void *ctx) +{ + struct special_elem val = {}; + struct task_struct *task, *old; + struct special_elem *elem; + int key = 0; + + err = 1; + if (bpf_map_update_elem(&special_fields, &key, &val, BPF_NOEXIST)) + return 1; + + err = 2; + elem = bpf_map_lookup_elem(&special_fields, &key); + if (!elem) + return 2; + + err = 3; + task = bpf_task_acquire(bpf_get_current_task_btf()); + if (!task) + return 3; + + err = 4; + old = bpf_kptr_xchg(&elem->task, task); + if (old) { + bpf_task_release(old); + return 4; + } + + err = 5; + if (bpf_map_delete_elem(&special_fields, &key)) + return 5; + + err = 6; + old = bpf_kptr_xchg(&elem->task, NULL); + if (!old) + return 6; + bpf_task_release(old); + + err = 0; + return 0; +} -- cgit v1.2.3 From ecdc5043794c9184aa8e6c814603899479c46b35 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Fri, 4 Sep 2026 12:41:56 +0200 Subject: bpf: Mark NULL kptr stores precise check_map_kptr_access() permits a scalar store into an untrusted kptr field only when the register is known to contain zero. Unlike other verifier checks whose outcome depends on a scalar value, it does not mark that register precise. A state checkpoint reached with an imprecise zero can therefore prune a second path that reaches the store with an arbitrary nonzero scalar. The program can write attacker-controlled bits into the kptr field and load them back as a PTR_TO_BTF_ID. Call mark_chain_precision() before accepting a known-zero register. This forces state equivalence to compare its scalar range and makes the verifier visit and reject a path carrying a nonzero value. Fixes: 61df10c7799e ("bpf: Allow storing unreferenced kptr in map") Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260904104203.345917-6-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/verifier.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 9c6ad157a61e..b71c5274b3dc 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4708,8 +4708,15 @@ static int check_map_kptr_access(struct bpf_verifier_env *env, return ret; } else if (class == BPF_STX) { val_reg = reg_state(env, value_regno); - if (!bpf_register_is_null(val_reg) && - map_kptr_match_type(env, kptr_field, val_reg, value_regno)) + if (bpf_register_is_null(val_reg)) { + /* + * This store is valid only because the scalar is known to be + * zero. Mark it precise so another scalar cannot be pruned + * against this state. + */ + return mark_chain_precision(env, value_regno); + } + if (map_kptr_match_type(env, kptr_field, val_reg, value_regno)) return -EACCES; } else if (class == BPF_ST) { if (insn->imm) { -- cgit v1.2.3 From 9dcddf30ac1a14f18c3221db9292bcaa0735ee2f Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Fri, 4 Sep 2026 12:41:57 +0200 Subject: selftests/bpf: Test imprecise scalar kptr stores Add a verifier regression where an imprecise zero scalar reaches a kptr store first and a nonzero scalar reaches the same instruction on a second path. Without the corresponding verifier fix, the second path is pruned and the program is unexpectedly accepted. With the fix, the scalar range is compared and the invalid store is rejected. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260904104203.345917-7-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- tools/testing/selftests/bpf/progs/map_kptr_fail.c | 37 +++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/map_kptr_fail.c b/tools/testing/selftests/bpf/progs/map_kptr_fail.c index 5e25ca806060..eee35d203b66 100644 --- a/tools/testing/selftests/bpf/progs/map_kptr_fail.c +++ b/tools/testing/selftests/bpf/progs/map_kptr_fail.c @@ -409,4 +409,41 @@ int reject_scalar_store_to_kptr(struct __sk_buff *ctx) return 0; } +SEC("?tc") +__description("reject imprecise scalar store to kptr after state pruning") +__failure __msg("invalid kptr access, R7 type=scalar") +__naked void reject_imprecise_scalar_store_to_kptr(void) +{ + asm volatile ( + "r0 = 0;" + "*(u32 *)(r10 - 4) = r0;" + "r2 = r10;" + "r2 += -4;" + "r1 = %[array_map] ll;" + "call %[bpf_map_lookup_elem];" + "if r0 == 0 goto l2_%=;" + "r6 = r0;" + "r9 = *(u64 *)(r6 + 0);" + "if r9 != 0 goto l0_%=;" + "r7 = 0;" + ".rept 10;" + "r5 = 1;" + ".endr;" + "goto l1_%=;" + "l0_%=:" + "r7 = 0x4141414141414141 ll;" + ".rept 10;" + "r5 = 1;" + ".endr;" + "l1_%=:" + "*(u64 *)(r6 + 8) = r7;" + "l2_%=:" + "r0 = 0;" + "exit;" + : + : __imm(bpf_map_lookup_elem), + __imm_addr(array_map) + : __clobber_all); +} + char _license[] SEC("license") = "GPL"; -- cgit v1.2.3 From b90c5d770dad910fb89e6c1b15052a8a1e8db752 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Fri, 4 Sep 2026 12:41:58 +0200 Subject: bpf: Preserve inner map identity in callback frames Callback frame constructors initialize map-typed argument registers with __mark_reg_known_zero() and then restore map_ptr. This clears map_uid, which is the only field distinguishing inner maps that share an inner_map_meta template. When a timer callback invokes bpf_for_each_map_elem() on a second inner map, both the saved first map and the second map value can reach the nested callback as the same template with map_uid zero. bpf_timer_init() then accepts pairing the timer from the second map with the first map. The runtime records the first map in the timer without taking a reference. Freeing that map does not find the timer stored in the second map, so a later timer callback dereferences the freed map. Copy map_uid from the same caller register as map_ptr when constructing for-each, timer/workqueue, and task-work callback arguments. The existing identity check can then reject mismatched inner maps while allowing a callback value to be paired with its actual map. Fixes: 3e8ce29850f1 ("bpf: Prevent pointer mismatch in bpf_timer_init.") Fixes: 69c087ba6225 ("bpf: Add bpf_for_each_map_elem() helper") Fixes: 5c8fd7e2b5b0 ("bpf: bpf task work plumbing") Reported-by: Nicholas Carlini Suggested-by: Nicholas Carlini Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260904104203.345917-8-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/verifier.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index b71c5274b3dc..c8699a8831df 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -10018,10 +10018,12 @@ int map_set_for_each_callback_args(struct bpf_verifier_env *env, callee->regs[BPF_REG_2].type = PTR_TO_MAP_KEY; __mark_reg_known_zero(&callee->regs[BPF_REG_2]); callee->regs[BPF_REG_2].map_ptr = caller->regs[BPF_REG_1].map_ptr; + callee->regs[BPF_REG_2].map_uid = caller->regs[BPF_REG_1].map_uid; callee->regs[BPF_REG_3].type = PTR_TO_MAP_VALUE; __mark_reg_known_zero(&callee->regs[BPF_REG_3]); callee->regs[BPF_REG_3].map_ptr = caller->regs[BPF_REG_1].map_ptr; + callee->regs[BPF_REG_3].map_uid = caller->regs[BPF_REG_1].map_uid; /* pointer to stack or null */ callee->regs[BPF_REG_4] = caller->regs[BPF_REG_3]; @@ -10099,6 +10101,7 @@ static int set_timer_callback_state(struct bpf_verifier_env *env, int insn_idx) { struct bpf_map *map_ptr = caller->regs[BPF_REG_1].map_ptr; + u32 map_uid = caller->regs[BPF_REG_1].map_uid; /* bpf_timer_set_callback(struct bpf_timer *timer, void *callback_fn); * callback_fn(struct bpf_map *map, void *key, void *value); @@ -10106,14 +10109,17 @@ static int set_timer_callback_state(struct bpf_verifier_env *env, callee->regs[BPF_REG_1].type = CONST_PTR_TO_MAP; __mark_reg_known_zero(&callee->regs[BPF_REG_1]); callee->regs[BPF_REG_1].map_ptr = map_ptr; + callee->regs[BPF_REG_1].map_uid = map_uid; callee->regs[BPF_REG_2].type = PTR_TO_MAP_KEY; __mark_reg_known_zero(&callee->regs[BPF_REG_2]); callee->regs[BPF_REG_2].map_ptr = map_ptr; + callee->regs[BPF_REG_2].map_uid = map_uid; callee->regs[BPF_REG_3].type = PTR_TO_MAP_VALUE; __mark_reg_known_zero(&callee->regs[BPF_REG_3]); callee->regs[BPF_REG_3].map_ptr = map_ptr; + callee->regs[BPF_REG_3].map_uid = map_uid; /* unused */ bpf_mark_reg_not_init(env, &callee->regs[BPF_REG_4]); @@ -10213,6 +10219,7 @@ static int set_task_work_schedule_callback_state(struct bpf_verifier_env *env, int insn_idx) { struct bpf_map *map_ptr = caller->regs[BPF_REG_3].map_ptr; + u32 map_uid = caller->regs[BPF_REG_3].map_uid; /* * callback_fn(struct bpf_map *map, void *key, void *value); @@ -10220,14 +10227,17 @@ static int set_task_work_schedule_callback_state(struct bpf_verifier_env *env, callee->regs[BPF_REG_1].type = CONST_PTR_TO_MAP; __mark_reg_known_zero(&callee->regs[BPF_REG_1]); callee->regs[BPF_REG_1].map_ptr = map_ptr; + callee->regs[BPF_REG_1].map_uid = map_uid; callee->regs[BPF_REG_2].type = PTR_TO_MAP_KEY; __mark_reg_known_zero(&callee->regs[BPF_REG_2]); callee->regs[BPF_REG_2].map_ptr = map_ptr; + callee->regs[BPF_REG_2].map_uid = map_uid; callee->regs[BPF_REG_3].type = PTR_TO_MAP_VALUE; __mark_reg_known_zero(&callee->regs[BPF_REG_3]); callee->regs[BPF_REG_3].map_ptr = map_ptr; + callee->regs[BPF_REG_3].map_uid = map_uid; /* unused */ bpf_mark_reg_not_init(env, &callee->regs[BPF_REG_4]); -- cgit v1.2.3 From e615b9fd4d9df602030d9b57a5eca206abbb0aff Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Fri, 4 Sep 2026 12:41:59 +0200 Subject: selftests/bpf: Test inner map identities in callbacks Add load-only timer_mim coverage for inner map identities propagated through nested timer and bpf_for_each_map_elem() callbacks. The negative case initializes a timer in the second inner map with the map saved from the first inner map timer callback. The positive case pairs the timer value with the map supplied to the same for-each callback. Without the verifier fix, the mismatched-map program is accepted while the same-map control is rejected. Preserving map_uid reverses both verdicts. Signed-off-by: Kumar Kartikeya Dwivedi Link: https://lore.kernel.org/r/20260904104203.345917-9-memxor@gmail.com Signed-off-by: Alexei Starovoitov --- tools/testing/selftests/bpf/prog_tests/timer_mim.c | 29 +++++++- .../testing/selftests/bpf/progs/timer_mim_reject.c | 84 +++++++++++++++++++++- 2 files changed, 109 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/bpf/prog_tests/timer_mim.c b/tools/testing/selftests/bpf/prog_tests/timer_mim.c index c930c7d7105b..fa7bb769ca31 100644 --- a/tools/testing/selftests/bpf/prog_tests/timer_mim.c +++ b/tools/testing/selftests/bpf/prog_tests/timer_mim.c @@ -59,10 +59,32 @@ void serial_test_timer_mim(void) int err; old_print_fn = libbpf_set_print(NULL); - timer_reject_skel = timer_mim_reject__open_and_load(); - libbpf_set_print(old_print_fn); - if (!ASSERT_ERR_PTR(timer_reject_skel, "timer_reject_skel_load")) + timer_reject_skel = timer_mim_reject__open(); + if (!ASSERT_OK_PTR(timer_reject_skel, "timer_reject_skel_open")) + goto cleanup; + bpf_program__set_autoload(timer_reject_skel->progs.test1, true); + err = timer_mim_reject__load(timer_reject_skel); + ASSERT_ERR(err, "timer_reject_skel_load"); + timer_mim_reject__destroy(timer_reject_skel); + + timer_reject_skel = timer_mim_reject__open(); + if (!ASSERT_OK_PTR(timer_reject_skel, "callback_reject_skel_open")) goto cleanup; + bpf_program__set_autoload(timer_reject_skel->progs.callback_map_uid_mismatch, true); + err = timer_mim_reject__load(timer_reject_skel); + ASSERT_ERR(err, "callback_reject_skel_load"); + timer_mim_reject__destroy(timer_reject_skel); + + timer_reject_skel = timer_mim_reject__open(); + if (!ASSERT_OK_PTR(timer_reject_skel, "callback_accept_skel_open")) + goto cleanup; + bpf_program__set_autoload(timer_reject_skel->progs.callback_map_uid_match, true); + err = timer_mim_reject__load(timer_reject_skel); + if (!ASSERT_OK(err, "callback_accept_skel_load")) + goto cleanup; + timer_mim_reject__destroy(timer_reject_skel); + timer_reject_skel = NULL; + libbpf_set_print(old_print_fn); timer_skel = timer_mim__open_and_load(); if (!timer_skel && errno == EOPNOTSUPP) { @@ -75,6 +97,7 @@ void serial_test_timer_mim(void) err = timer_mim(timer_skel); ASSERT_OK(err, "timer_mim"); cleanup: + libbpf_set_print(old_print_fn); timer_mim__destroy(timer_skel); timer_mim_reject__destroy(timer_reject_skel); } diff --git a/tools/testing/selftests/bpf/progs/timer_mim_reject.c b/tools/testing/selftests/bpf/progs/timer_mim_reject.c index dd3f1ed6d6e6..83f31138336b 100644 --- a/tools/testing/selftests/bpf/progs/timer_mim_reject.c +++ b/tools/testing/selftests/bpf/progs/timer_mim_reject.c @@ -43,7 +43,7 @@ static int timer_cb(void *map, int *key, struct hmap_elem *val) return 0; } -SEC("fentry/bpf_fentry_test1") +SEC("?fentry/bpf_fentry_test1") int BPF_PROG(test1, int a) { struct hmap_elem init = {}; @@ -72,3 +72,85 @@ int BPF_PROG(test1, int a) err |= 8; return 0; } + +struct callback_ctx { + void *map; +}; + +static int mismatch_iter_cb(void *map, int *key, struct hmap_elem *val, struct callback_ctx *ctx) +{ + bpf_timer_init(&val->timer, ctx->map, CLOCK_MONOTONIC); + return 0; +} + +static int timer_mismatch_cb(void *map, int *key, struct hmap_elem *val) +{ + struct callback_ctx ctx = { .map = map }; + struct bpf_map *inner_map2; + int array_key2 = ARRAY_KEY2; + + inner_map2 = bpf_map_lookup_elem(&outer_arr, &array_key2); + if (!inner_map2) + return 0; + bpf_for_each_map_elem(inner_map2, mismatch_iter_cb, &ctx, 0); + return 0; +} + +static int match_iter_cb(void *map, int *key, struct hmap_elem *val, struct callback_ctx *ctx) +{ + bpf_timer_init(&val->timer, map, CLOCK_MONOTONIC); + return 0; +} + +static int timer_match_cb(void *map, int *key, struct hmap_elem *val) +{ + struct callback_ctx ctx = {}; + struct bpf_map *inner_map2; + int array_key2 = ARRAY_KEY2; + + inner_map2 = bpf_map_lookup_elem(&outer_arr, &array_key2); + if (!inner_map2) + return 0; + bpf_for_each_map_elem(inner_map2, match_iter_cb, &ctx, 0); + return 0; +} + +SEC("?fentry/bpf_fentry_test1") +int BPF_PROG(callback_map_uid_mismatch, int a) +{ + struct hmap_elem *val; + struct bpf_map *inner_map; + int array_key = ARRAY_KEY; + int hash_key = HASH_KEY; + + inner_map = bpf_map_lookup_elem(&outer_arr, &array_key); + if (!inner_map) + return 0; + val = bpf_map_lookup_elem(inner_map, &hash_key); + if (!val) + return 0; + + bpf_timer_init(&val->timer, inner_map, CLOCK_MONOTONIC); + bpf_timer_set_callback(&val->timer, timer_mismatch_cb); + return 0; +} + +SEC("?fentry/bpf_fentry_test1") +int BPF_PROG(callback_map_uid_match, int a) +{ + struct hmap_elem *val; + struct bpf_map *inner_map; + int array_key = ARRAY_KEY; + int hash_key = HASH_KEY; + + inner_map = bpf_map_lookup_elem(&outer_arr, &array_key); + if (!inner_map) + return 0; + val = bpf_map_lookup_elem(inner_map, &hash_key); + if (!val) + return 0; + + bpf_timer_init(&val->timer, inner_map, CLOCK_MONOTONIC); + bpf_timer_set_callback(&val->timer, timer_match_cb); + return 0; +} -- cgit v1.2.3 From 1a3a10b030c96ea88868ccc060a16827c01eaa5a Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 4 Sep 2026 17:05:52 -0700 Subject: bpf: mark a NULL call argument precise check_func_arg() allows bpf_register_is_null() for nullable arguments w/o marking the underlying scalar register precise. Hence a checkpoint created on such a path would prune against arbitrary scalar value. check_helper_call() enforces second parameter of the bpf_get_local_storage() to be zero, w/o marking the underlying scalar register precise. Hence a checkpoint created on such a path would prune against arbitrary scalar value. Grouping these two into one patch, as they share the same fixes tag. Fixes: b5dc0163d8fd ("bpf: precise scalar_value tracking") Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260904-register-is-null-precise-fixes-v1-1-0f5a360ff15d@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/verifier.c | 9 ++++++++- .../testing/selftests/bpf/progs/verifier_subprog_precision.c | 12 ++++++------ 2 files changed, 14 insertions(+), 7 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index c8699a8831df..b107f551a62d 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -8759,11 +8759,15 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, return err; } - if (bpf_register_is_null(reg) && type_may_be_null(arg_type)) + if (bpf_register_is_null(reg) && type_may_be_null(arg_type)) { /* A NULL register has a SCALAR_VALUE type, so skip * type checking. */ + err = mark_chain_precision(env, regno); + if (err) + return err; goto skip_type_check; + } /* arg_btf_id and arg_size are in a union. */ if (base_type(arg_type) == ARG_PTR_TO_BTF_ID || @@ -10923,6 +10927,9 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn verbose(env, "get_local_storage() doesn't support non-zero flags\n"); return -EINVAL; } + err = mark_chain_precision(env, BPF_REG_2); + if (err) + return err; break; case BPF_FUNC_for_each_map_elem: err = push_callback_call(env, insn, insn_idx, meta.subprogno, diff --git a/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c b/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c index e174a905c562..dc0c7034c04f 100644 --- a/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c +++ b/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c @@ -287,9 +287,9 @@ __msg("17: (b7) r0 = 0") __msg("18: (95) exit") __msg("returning from callee:") __msg("to caller at 9:") -__msg("frame 0: propagating r1,r4") +__msg("frame 0: propagating r1,r3,r4") __msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1") -__msg("mark_precise: frame0: regs=r1,r4 stack= before 18: (95) exit") +__msg("mark_precise: frame0: regs=r1,r3,r4 stack= before 18: (95) exit") __msg("from 18 to 9: safe") __naked int callback_result_precise(void) { @@ -419,9 +419,9 @@ __msg("to caller at 9:") /* r1, r4 are always precise for bpf_loop(), * r6 was marked before backtracking to callback body. */ -__msg("frame 0: propagating r1,r4,r6") +__msg("frame 0: propagating r1,r3,r4,r6") __msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1") -__msg("mark_precise: frame0: regs=r1,r4,r6 stack= before 16: (95) exit") +__msg("mark_precise: frame0: regs=r1,r3,r4,r6 stack= before 16: (95) exit") __msg("mark_precise: frame1: regs= stack= before 15: (b7) r0 = 0") __msg("mark_precise: frame1: regs= stack= before 9: (85) call bpf_loop") __msg("mark_precise: frame0: parent state regs= stack=:") @@ -575,9 +575,9 @@ __msg("to caller at 10:") /* r1, r4 are always precise for bpf_loop(), * fp-8 was marked before backtracking to callback body. */ -__msg("frame 0: propagating r1,r4,fp-8") +__msg("frame 0: propagating r1,r3,r4,fp-8") __msg("mark_precise: frame0: last_idx 10 first_idx 10 subseq_idx -1") -__msg("mark_precise: frame0: regs=r1,r4 stack=-8 before 18: (95) exit") +__msg("mark_precise: frame0: regs=r1,r3,r4 stack=-8 before 18: (95) exit") __msg("mark_precise: frame1: regs= stack= before 17: (b7) r0 = 0") __msg("mark_precise: frame1: regs= stack= before 10: (85) call bpf_loop#181") __msg("mark_precise: frame0: parent state regs= stack=:") -- cgit v1.2.3 From 593c8eb0fb91a24c39244a7f9e7d04412d750544 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 4 Sep 2026 17:05:53 -0700 Subject: selftests/bpf: precision of a NULL helper argument Check that mark_chain_precision() is called for a NULL nullable memory argument and for the zero flags argument of bpf_get_local_storage(). Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260904-register-is-null-precise-fixes-v1-2-0f5a360ff15d@gmail.com Signed-off-by: Alexei Starovoitov --- .../selftests/bpf/progs/verifier_cgroup_storage.c | 29 ++++++++++++++++++ .../selftests/bpf/progs/verifier_precision.c | 34 ++++++++++++++++++++++ 2 files changed, 63 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_cgroup_storage.c b/tools/testing/selftests/bpf/progs/verifier_cgroup_storage.c index 9a13f5c11ac7..884080a5bffc 100644 --- a/tools/testing/selftests/bpf/progs/verifier_cgroup_storage.c +++ b/tools/testing/selftests/bpf/progs/verifier_cgroup_storage.c @@ -305,4 +305,33 @@ __naked void cpu_cgroup_storage_access_6(void) : __clobber_all); } +/* + * Verification takes two paths: with r2 being scalar zero on path (1) + * and with r2 being some other scalar on path (2). + * Check that the verifier does not use checkpoints created + * on path (1) to prune path (2). + */ +SEC("cgroup/skb") +__failure +__flag(BPF_F_TEST_STATE_FREQ) +__msg("get_local_storage() doesn't support non-zero flags") +__naked void non_zero_flags_on_a_pruned_path(void) +{ + asm volatile (" \ + call %[bpf_get_prandom_u32]; \ + /* r2 is 0 on the path explored first, 1 on the other */\ + r2 = 1; \ + if r0 == 0 goto 1f; \ + r2 = 0; \ +1: r1 = %[cgroup_storage] ll; \ + call %[bpf_get_local_storage]; \ + r0 = 0; \ + exit; \ +" : + : __imm(bpf_get_prandom_u32), + __imm(bpf_get_local_storage), + __imm_addr(cgroup_storage) + : __clobber_all); +} + char _license[] SEC("license") = "GPL"; diff --git a/tools/testing/selftests/bpf/progs/verifier_precision.c b/tools/testing/selftests/bpf/progs/verifier_precision.c index 6f325876efdd..3e290b07f672 100644 --- a/tools/testing/selftests/bpf/progs/verifier_precision.c +++ b/tools/testing/selftests/bpf/progs/verifier_precision.c @@ -642,4 +642,38 @@ __naked int bpf_atomic_cmpxchg_32bit_precision(void) : __clobber_all); } +/* + * Verification takes two paths: with r1 being scalar zero on path (1) + * and with r1 being some other scalar on path (2). + * Check that the verifier does not use checkpoints created + * on path (1) to prune path (2). + */ +SEC("?tc") +__flag(BPF_F_TEST_STATE_FREQ) +__failure __msg("R1 type=scalar expected=fp") +__naked int null_mem_arg_zero_size(void) +{ + asm volatile ( + "call %[bpf_get_prandom_u32];" + "r1 = 42;" + "if r0 > 42 goto 1f;" + "r1 = 0;" + "1:" + "r2 = 0;" + "r3 = 0;" + "r4 = 0;" + "r5 = 0;" + /* + * ARG_PTR_TO_MEM | PTR_MAYBE_NULL parameter can be NULL, + * but can't be some other scalar value. + */ + "call %[bpf_csum_diff];" + "r0 = 0;" + "exit;" + : + : __imm(bpf_get_prandom_u32), + __imm(bpf_csum_diff) + : __clobber_all); +} + char _license[] SEC("license") = "GPL"; -- cgit v1.2.3 From f1e418129f2ebb5376df2f1cd19720fa80f8adb4 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 4 Sep 2026 17:05:54 -0700 Subject: bpf: mark a NULL memory argument of a call precise check_mem_reg() allows bpf_register_is_null() for nullable arguments w/o marking the underlying scalar register precise. Hence a checkpoint created on such a path would prune against arbitrary scalar value. The argument may live on the stack rather than in a register when a call has more than MAX_BPF_FUNC_REG_ARGS arguments, hence the new mark_arg_precision() helper. Fixes: e5069b9c23b3 ("bpf: Support pointers in global func args") Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260904-register-is-null-precise-fixes-v1-3-0f5a360ff15d@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/verifier.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index b107f551a62d..7926e131b1cb 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4246,6 +4246,15 @@ static int mark_stack_arg_precision(struct bpf_verifier_env *env, int arg_idx) return mark_chain_precision_batch(env, env->cur_state); } +static int mark_arg_precision(struct bpf_verifier_env *env, argno_t argno) +{ + int regno = reg_from_argno(argno); + + if (regno >= 0) + return mark_chain_precision(env, regno); + return mark_stack_arg_precision(env, arg_idx_from_argno(argno)); +} + static int check_outgoing_stack_args(struct bpf_verifier_env *env, struct bpf_func_state *caller, int nargs, const char *callee_name, const struct btf *btf, const struct btf_param *args) @@ -7175,7 +7184,7 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg int size, err = 0; if (bpf_register_is_null(reg)) - return 0; + return mark_arg_precision(env, argno); if (known_memory) *known_memory = true; -- cgit v1.2.3 From 100f4cc0d59be88d2b4d6eb42e51910ea2548d04 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 4 Sep 2026 17:05:55 -0700 Subject: selftests/bpf: precision of a NULL global subprogram memory argument Check that mark_chain_precision() is called for a NULL pointer passed as a nullable pointer argument of a global subprogram. (Pointer arguments of the global subprograms are nullable by default). Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260904-register-is-null-precise-fixes-v1-4-0f5a360ff15d@gmail.com Signed-off-by: Alexei Starovoitov --- .../selftests/bpf/progs/verifier_precision.c | 32 ++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_precision.c b/tools/testing/selftests/bpf/progs/verifier_precision.c index 3e290b07f672..fb7dfa1246ef 100644 --- a/tools/testing/selftests/bpf/progs/verifier_precision.c +++ b/tools/testing/selftests/bpf/progs/verifier_precision.c @@ -676,4 +676,36 @@ __naked int null_mem_arg_zero_size(void) : __clobber_all); } +__weak int subprog_mem_arg(int *p) +{ + if (p) + return *p; + return 0; +} + +/* + * Verification takes two paths: with r1 being scalar zero on path (1) + * and with r1 being some other scalar on path (2). + * Check that the verifier does not use checkpoints created + * on path (1) to prune path (2). + */ +SEC("?raw_tp") +__flag(BPF_F_TEST_STATE_FREQ) +__failure __msg("R1 type=scalar expected=fp") +__naked int null_mem_arg_global_subprog(void) +{ + asm volatile ( + "call %[bpf_get_prandom_u32];" + "r1 = 42;" + "if r0 > 42 goto 1f;" + "r1 = 0;" + "1:" + "call subprog_mem_arg;" + "r0 = 0;" + "exit;" + : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + char _license[] SEC("license") = "GPL"; -- cgit v1.2.3 From 506ada89629ec7059b96ecfb0dc7d33ece0103ca Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 4 Sep 2026 17:05:56 -0700 Subject: bpf: mark a NULL kfunc argument precise check_kfunc_arg() allows bpf_register_is_null() for nullable arguments w/o marking the underlying scalar register precise. Hence a checkpoint created on such a path would prune against arbitrary scalar value. Fixes: 3bda08b63670 ("bpf: Allow NULL buffers in bpf_dynptr_slice(_rw)") Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260904-register-is-null-precise-fixes-v1-5-0f5a360ff15d@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/verifier.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 7926e131b1cb..2117c39ac332 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -12733,8 +12733,12 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_call_arg_me if (reg_is_referenced(env, reg)) update_ref_obj(&meta->ref_obj, reg); - if (bpf_register_is_null(reg) && type_may_be_null(kf_arg_type)) + if (bpf_register_is_null(reg) && type_may_be_null(kf_arg_type)) { + ret = mark_arg_precision(env, argno); + if (ret) + return ret; continue; + } if (is_kfunc_arg_map(btf, &args[i])) { ref_id = *reg2btf_ids[CONST_PTR_TO_MAP]; -- cgit v1.2.3 From 562d266d3fae571617e72417753df648db261c56 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 4 Sep 2026 17:05:57 -0700 Subject: selftests/bpf: precision of a NULL kfunc argument Check that mark_chain_precision() is called for a NULL pointer passed as a __nullable kfunc memory argument. Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260904-register-is-null-precise-fixes-v1-6-0f5a360ff15d@gmail.com Signed-off-by: Alexei Starovoitov --- .../selftests/bpf/progs/verifier_precision.c | 34 ++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_precision.c b/tools/testing/selftests/bpf/progs/verifier_precision.c index fb7dfa1246ef..f4459561bf39 100644 --- a/tools/testing/selftests/bpf/progs/verifier_precision.c +++ b/tools/testing/selftests/bpf/progs/verifier_precision.c @@ -2,8 +2,10 @@ /* Copyright (C) 2023 SUSE LLC */ #include #include +#include #include "../../../include/linux/filter.h" #include "bpf_misc.h" +#include "bpf_kfuncs.h" struct { __uint(type, BPF_MAP_TYPE_ARRAY); @@ -708,4 +710,36 @@ __naked int null_mem_arg_global_subprog(void) : __clobber_all); } +/* Same as above, check that path with r3 == 0 does not prune the path with r3 != 0 */ +SEC("?tc") +__flag(BPF_F_TEST_STATE_FREQ) +__failure __msg("R3 type=scalar expected=fp") +int null_kfunc_arg_dynptr_slice(struct __sk_buff *skb) +{ + struct bpf_dynptr ptr; + + bpf_dynptr_from_skb(skb, 0, &ptr); + asm volatile ( + "call %[bpf_get_prandom_u32];" + "r3 = 42;" + "if r0 > 42 goto 1f;" + "r3 = 0;" + "1:" + "r1 = %[ptr];" + "r2 = 0;" + "r4 = 8;" + "call %[bpf_dynptr_slice];" + : + : __imm_ptr(ptr), + __imm(bpf_get_prandom_u32), + __imm(bpf_dynptr_slice) + : __clobber_common); + return 0; +} + +void __kfunc_btf_root(void) +{ + bpf_dynptr_slice(0, 0, 0, 0); +} + char _license[] SEC("license") = "GPL"; -- cgit v1.2.3 From e726fc6b9afe6b3a446a31d0f0767b7b9cb5085e Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 4 Sep 2026 17:05:58 -0700 Subject: bpf: mark a NULL BTF_ID argument of a global subprogram precise btf_check_func_arg_match() accepts a NULL register for an ARG_PTR_TO_BTF_ID argument tagged __arg_nullable and skips check_reg_type() and check_func_arg_reg_off() without marking the register precise. Hence a checkpoint created on such a path would prune against arbitrary scalar value. Fixes: e2b3c4ff5d18 ("bpf: add __arg_trusted global func arg tag") Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260904-register-is-null-precise-fixes-v1-7-0f5a360ff15d@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/verifier.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 2117c39ac332..1b9fcbe4621a 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -9774,8 +9774,12 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, struct bpf_call_arg_meta meta; int err; - if (bpf_register_is_null(reg) && type_may_be_null(arg->arg_type)) + if (bpf_register_is_null(reg) && type_may_be_null(arg->arg_type)) { + err = mark_arg_precision(env, argno); + if (err) + return err; continue; + } memset(&meta, 0, sizeof(meta)); /* leave func_id as zero */ err = check_reg_type(env, reg, argno, arg->arg_type, &arg->btf_id, &meta, -- cgit v1.2.3 From 91957791663f49561848c40e982061799b8f86b0 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 4 Sep 2026 17:05:59 -0700 Subject: selftests/bpf: precision of a NULL global subprogram BTF_ID argument Check that mark_chain_precision() is called for a NULL pointer passed as an __arg_trusted __arg_nullable argument of a global subprogram. Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260904-register-is-null-precise-fixes-v1-8-0f5a360ff15d@gmail.com Signed-off-by: Alexei Starovoitov --- .../selftests/bpf/progs/verifier_global_ptr_args.c | 24 ++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_global_ptr_args.c b/tools/testing/selftests/bpf/progs/verifier_global_ptr_args.c index 0bdeb7bc4687..a3d2af8dc839 100644 --- a/tools/testing/selftests/bpf/progs/verifier_global_ptr_args.c +++ b/tools/testing/selftests/bpf/progs/verifier_global_ptr_args.c @@ -56,6 +56,30 @@ int trusted_task_arg_nullable(void *ctx) return res; } +/* + * Check that the verifier does not use checkpoints created + * on path with r1 == 0 to prune path with r1 != 0. + */ +SEC("?tp_btf/task_newtask") +__failure +__flag(BPF_F_TEST_STATE_FREQ) +__msg("R1 type=scalar expected=ptr_, trusted_ptr_, rcu_ptr_") +__naked int null_btf_id_arg_global_subprog(void) +{ + asm volatile ( + "call %[bpf_get_prandom_u32];" + "r1 = 42;" + "if r0 > 42 goto 1f;" + "r1 = 0;" + "1:" + "call subprog_trusted_task_nullable;" + "r0 = 0;" + "exit;" + : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + __weak int subprog_trusted_task_nonnull(struct task_struct *task __arg_trusted) { return task->pid + task->tgid; -- cgit v1.2.3 From 1d7f8f191c06f967a85922c4652dc33c132b585d Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 4 Sep 2026 17:06:00 -0700 Subject: bpf: propagate mark_chain_precision() errors out of loop_flag_is_zero() Stop verification if mark_chain_precision() fails when called from loop_flag_is_zero(). No functional change intended for the paths where backtracking succeeds. Fixes: 1ade23711971 ("bpf: Inline calls to bpf_loop when callback is known") Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260904-register-is-null-precise-fixes-v1-9-0f5a360ff15d@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/verifier.c | 36 +++++++++++++++++++++++++----------- 1 file changed, 25 insertions(+), 11 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 1b9fcbe4621a..e8af1d3dbdeb 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -10702,33 +10702,45 @@ static struct bpf_insn_aux_data *cur_aux(const struct bpf_verifier_env *env) return &env->insn_aux_data[env->insn_idx]; } -static bool loop_flag_is_zero(struct bpf_verifier_env *env) +/* Returns 1 if R4 is a known zero, 0 if it is not, a negative errno on error. */ +static int loop_flag_is_zero(struct bpf_verifier_env *env) { struct bpf_reg_state *reg = reg_state(env, BPF_REG_4); - bool reg_is_null = bpf_register_is_null(reg); + int err; - if (reg_is_null) - mark_chain_precision(env, BPF_REG_4); + if (!bpf_register_is_null(reg)) + return 0; - return reg_is_null; + err = mark_chain_precision(env, BPF_REG_4); + if (err) + return err; + return 1; } -static void update_loop_inline_state(struct bpf_verifier_env *env, u32 subprogno) +static int update_loop_inline_state(struct bpf_verifier_env *env, u32 subprogno) { struct bpf_loop_inline_state *state = &cur_aux(env)->loop_inline_state; + int flag_is_zero; if (!state->initialized) { + flag_is_zero = loop_flag_is_zero(env); + if (flag_is_zero < 0) + return flag_is_zero; state->initialized = 1; - state->fit_for_inline = loop_flag_is_zero(env); + state->fit_for_inline = flag_is_zero; state->callback_subprogno = subprogno; - return; + return 0; } if (!state->fit_for_inline) - return; + return 0; - state->fit_for_inline = (loop_flag_is_zero(env) && + flag_is_zero = loop_flag_is_zero(env); + if (flag_is_zero < 0) + return flag_is_zero; + state->fit_for_inline = (flag_is_zero && state->callback_subprogno == subprogno); + return 0; } /* Returns whether or not the given map can potentially elide @@ -10960,7 +10972,9 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn err = check_bpf_snprintf_call(env, regs); break; case BPF_FUNC_loop: - update_loop_inline_state(env, meta.subprogno); + err = update_loop_inline_state(env, meta.subprogno); + if (err) + return err; /* Verifier relies on R1 value to determine if bpf_loop() iteration * is finished, thus mark it precise. */ -- cgit v1.2.3 From cf2475616b11c0efefdd969d42913f56ca39f918 Mon Sep 17 00:00:00 2001 From: Eduard Zingerman Date: Fri, 4 Sep 2026 17:06:01 -0700 Subject: bpf: use mark_arg_precision() in check_mem_size_reg() Use newly added mark_arg_precision() helper in check_mem_size_reg(). Signed-off-by: Eduard Zingerman Link: https://lore.kernel.org/r/20260904-register-is-null-precise-fixes-v1-10-0f5a360ff15d@gmail.com Signed-off-by: Alexei Starovoitov --- kernel/bpf/verifier.c | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index e8af1d3dbdeb..1c3039f3fc32 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -7160,14 +7160,8 @@ static int check_mem_size_reg(struct bpf_verifier_env *env, if (err && failure) *failure = BPF_MEM_SIZE_FAIL_MEMORY; - if (!err) { - int regno = reg_from_argno(size_argno); - - if (regno >= 0) - err = mark_chain_precision(env, regno); - else - err = mark_stack_arg_precision(env, arg_idx_from_argno(size_argno)); - } + if (!err) + err = mark_arg_precision(env, size_argno); return err; -- cgit v1.2.3 From c3fd8e5fd100f122bad503bdc0e9277219533253 Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sat, 5 Sep 2026 03:47:33 +0200 Subject: bpf: Reject non-scalar bpf_loop iteration counts bpf_loop() declares its nr_loops argument as ARG_ANYTHING. Privileged programs may pass pointer values to such arguments, so check_func_arg() lets a pointer-valued R1 reach the helper-specific checks. Since commit bb124da69c47 ("bpf: keep track of max number of bpf_loop callback iterations"), the verifier marks R1 precise and reads its upper bound to limit callback simulation. Precision backtracking only accepts scalar registers, so passing a pointer instead triggers the "backtracking misuse" verifier warning. Kernels with panic_on_warn enabled subsequently panic. Introduce ARG_SCALAR for helper arguments that only accept scalar values and use it for bpf_loop() nr_loops. Generic helper argument validation then rejects pointers before loop inlining and precision processing. Fixes: bb124da69c47 ("bpf: keep track of max number of bpf_loop callback iterations") Reported-by: syzbot+7b47f87674e9a1569110@syzkaller.appspotmail.com Signed-off-by: Kumar Kartikeya Dwivedi Link: https://patch.msgid.link/20260905014735.1452988-2-memxor@gmail.com Closes: https://lore.kernel.org/bpf/6a9ad24c.b5d4176b.238c3e.0001.GAE@google.com/ Signed-off-by: Eduard Zingerman --- include/linux/bpf.h | 1 + kernel/bpf/bpf_iter.c | 2 +- kernel/bpf/verifier.c | 1 + 3 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index b7dbf3d9b5c0..e57af902560c 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -894,6 +894,7 @@ enum bpf_arg_type { ARG_PTR_TO_CTX, /* pointer to context */ ARG_ANYTHING, /* any (initialized) argument is ok */ + ARG_SCALAR, /* scalar argument */ ARG_PTR_TO_SPIN_LOCK, /* pointer to bpf_spin_lock */ ARG_PTR_TO_SOCK_COMMON, /* pointer to sock_common */ ARG_PTR_TO_SOCKET, /* pointer to bpf_sock (fullsock) */ diff --git a/kernel/bpf/bpf_iter.c b/kernel/bpf/bpf_iter.c index 14a5fdfa0421..b40eb404adab 100644 --- a/kernel/bpf/bpf_iter.c +++ b/kernel/bpf/bpf_iter.c @@ -754,7 +754,7 @@ const struct bpf_func_proto bpf_loop_proto = { .func = bpf_loop, .gpl_only = false, .ret_type = RET_INTEGER, - .arg1_type = ARG_ANYTHING, + .arg1_type = ARG_SCALAR, .arg2_type = ARG_PTR_TO_FUNC, .arg3_type = ARG_PTR_TO_STACK_OR_NULL, .arg4_type = ARG_ANYTHING, diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 1c3039f3fc32..4638a2f85d0f 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -8211,6 +8211,7 @@ static const struct bpf_reg_types *compatible_reg_types[__BPF_ARG_TYPE_MAX] = { [ARG_MEM_SIZE] = &scalar_types, [ARG_MEM_SIZE_OR_ZERO] = &scalar_types, [ARG_CONST_ALLOC_SIZE_OR_ZERO] = &scalar_types, + [ARG_SCALAR] = &scalar_types, [ARG_CONST_MAP_PTR] = &const_map_ptr_types, [ARG_PTR_TO_CTX] = &context_types, [ARG_PTR_TO_SOCK_COMMON] = &sock_types, -- cgit v1.2.3 From bde8901ea14244e7195a2d6b6aa2023b28d4233c Mon Sep 17 00:00:00 2001 From: Kumar Kartikeya Dwivedi Date: Sat, 5 Sep 2026 03:47:34 +0200 Subject: selftests/bpf: Test pointer bpf_loop iteration count rejection Add a verifier test that leaves the raw tracepoint context pointer in R1 when calling bpf_loop(). This is the smallest trigger for the incorrect precision backtracking: it reuses an existing callback and needs no maps or userspace setup. Expect an ordinary scalar-type rejection. Without the verifier fix, the test instead reaches precision backtracking and reports an internal "backtracking misuse" error. Signed-off-by: Kumar Kartikeya Dwivedi Acked-by: Eduard Zingerman Link: https://patch.msgid.link/20260905014735.1452988-3-memxor@gmail.com Signed-off-by: Eduard Zingerman --- .../selftests/bpf/progs/verifier_iterating_callbacks.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c b/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c index 75dd922e4e9f..1fbcc5228306 100644 --- a/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c +++ b/tools/testing/selftests/bpf/progs/verifier_iterating_callbacks.c @@ -168,6 +168,23 @@ static int iter_limit_cb(__u32 idx, struct num_context *ctx) return 0; } +SEC("?raw_tp") +__failure __msg("R1 type=ctx expected=scalar") +__naked void bpf_loop_reject_pointer(void) +{ + asm volatile ( + "r2 = %[iter_limit_cb];" + "r3 = 0;" + "r4 = 0;" + "call %[bpf_loop];" + "exit;" + : + : __imm_ptr(iter_limit_cb), + __imm(bpf_loop) + : __clobber_common + ); +} + SEC("?raw_tp") __success int bpf_loop_iter_limit_ok(void *unused) -- cgit v1.2.3 From 536b523b407397c8d3967c020ce7aad70a0ea030 Mon Sep 17 00:00:00 2001 From: Chen Pei Date: Wed, 2 Sep 2026 14:14:51 +0800 Subject: bpf, riscv: Make arena support depend on ZACAS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The arena range tree allocates its nodes with kmalloc_nolock() since commit f8c67d8550ee ("bpf: Use kmalloc_nolock() in range tree"). kmalloc_nolock() requires slab caches with cmpxchg128 support (__CMPXCHG_DOUBLE); on riscv cmpxchg128 is provided by the ZACAS extension. On systems without ZACAS every arena map creation fails with a misleading -ENOMEM. Report the missing support instead: make bpf_jit_supports_arena() return system_has_cmpxchg128() where it is defined, so arena map creation fails with -EOPNOTSUPP on systems without ZACAS. The macro is only defined when both CONFIG_RISCV_ISA_ZACAS and CONFIG_TOOLCHAIN_HAS_ZACAS are enabled, so guard it with #ifdef the same way mm/slab.h consumes it, and reject arena otherwise. This matches how arena BPF_CMPXCHG instructions are already gated on ZACAS in bpf_jit_supports_insn(). Fixes: f8c67d8550ee ("bpf: Use kmalloc_nolock() in range tree") Signed-off-by: Chen Pei Acked-by: Pu Lehui Acked-by: Björn Töpel Cc: stable@vger.kernel.org Link: https://lore.kernel.org/bpf/20260902061451.1416-1-cp0613@linux.alibaba.com Signed-off-by: Kumar Kartikeya Dwivedi --- arch/riscv/net/bpf_jit_comp64.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/arch/riscv/net/bpf_jit_comp64.c b/arch/riscv/net/bpf_jit_comp64.c index 74efe4b138d2..151031e97a24 100644 --- a/arch/riscv/net/bpf_jit_comp64.c +++ b/arch/riscv/net/bpf_jit_comp64.c @@ -2128,7 +2128,15 @@ bool bpf_jit_supports_ptr_xchg(void) bool bpf_jit_supports_arena(void) { - return true; + /* + * The arena range tree uses kmalloc_nolock(), which needs + * cmpxchg128, provided by ZACAS on riscv. + */ +#ifdef system_has_cmpxchg128 + return system_has_cmpxchg128(); +#else + return false; +#endif } bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena) -- cgit v1.2.3