kernel/git/stable/linux.git/block/blk-cgroup.c, branch master

block: skip blkcg walk in blk_cgroup_congested() when nothing throttled

2026-08-15T23:15:08+00:00

blk_cgroup_congested() walks the current task's blkcg ancestor chain on every readahead decision and, once swap is in use, on every anonymous and shmem folio allocation. The answer is almost always "no", but finding that out costs two loads per level on two cold cache lines, plus an out-of-line kthread_blkcg() and an RCU read-side pair. On a fleet profile of hosts running containers with 5-10 level hierarchies it costs about as much as all of mutex_lock(), 99.4% of it under __folio_throttle_swaprate(). Gate the walk on a global count of blkcgs with a non-zero congestion_count. The counter only moves on the 0 <-> 1 transitions of each blkcg's congestion_count, so the extra atomic stays in the throttle arm/disarm paths and never appears in steady state. When something is throttled the counter is non-zero and the walk runs as before. Signed-off-by: Usama Arif Acked-by: Tejun Heo Link: https://patch.msgid.link/20260814165712.510132-4-usama.arif@linux.dev Signed-off-by: Jens Axboe

blk-cgroup: factor policy pd teardown loop into helper

2026-08-04T02:02:35+00:00

Move the teardown sequence which offlines and frees per-policy blkg_policy_data (pd) into a helper for readability. No functional change intended. Signed-off-by: Zheng Qixing Reviewed-by: Christoph Hellwig Reviewed-by: Tang Yizhou Signed-off-by: Yu Kuai Reviewed-by: Nilay Shroff Link: https://patch.msgid.link/20260802112525.3933753-5-yukuai@kernel.org Signed-off-by: Jens Axboe

blk-cgroup: skip dying blkg in blkcg_activate_policy()

2026-08-04T02:02:35+00:00

When switching IO schedulers on a block device, blkcg_activate_policy() can race with concurrent blkcg deletion, leading to a use-after-free in rcu_accelerate_cbs. T1: T2: blkg_destroy kill(&blkg->refcnt) // blkg->refcnt=1->0 blkg_release // call_rcu(__blkg_release) ... blkg_free_workfn ->pd_free_fn(pd) elv_iosched_store elevator_switch ... iterate blkg list blkg_get(blkg) // blkg->refcnt=0->1 list_del_init(&blkg->q_node) blkg_put(pinned_blkg) // blkg->refcnt=1->0 blkg_release // call_rcu again rcu_accelerate_cbs // uaf Fix this by checking hlist_unhashed(&blkg->blkcg_node) before getting a reference to the blkg. This is the same check used in blkg_destroy() to detect if a blkg has already been destroyed. If the blkg is already unhashed, skip processing it since it's being destroyed. Fixes: f1c006f1c685 ("blk-cgroup: synchronize pd_free_fn() from blkg_free_workfn() and blkcg_deactivate_policy()") Signed-off-by: Zheng Qixing Reviewed-by: Tang Yizhou Signed-off-by: Yu Kuai Reviewed-by: Tao Cui Reviewed-by: Nilay Shroff Link: https://patch.msgid.link/20260802112525.3933753-4-yukuai@kernel.org Signed-off-by: Jens Axboe

blk-cgroup: fix race between policy activation and blkg destruction

2026-08-04T02:02:35+00:00

When switching an IO scheduler on a block device, blkcg_activate_policy() allocates blkg_policy_data (pd) for all blkgs attached to the queue. However, blkcg_activate_policy() may race with concurrent blkcg deletion, leading to use-after-free and memory leak issues. The use-after-free occurs in the following race: T1 (blkcg_activate_policy): - Successfully allocates pd for blkg1 (loop0->queue, blkcgA) - Fails to allocate pd for blkg2 (loop0->queue, blkcgB) - Enters the enomem rollback path to release blkg1 resources T2 (blkcg deletion): - blkcgA is deleted concurrently - blkg1 is freed via blkg_free_workfn() - blkg1->pd is freed T1 (continued): - Rollback path accesses blkg1->pd->online after pd is freed - Triggers use-after-free In addition, blkg_free_workfn() frees pd before removing the blkg from q->blkg_list. This allows blkcg_activate_policy() to allocate a new pd for a blkg that is being destroyed, leaving the newly allocated pd unreachable when the blkg is finally freed. Fix these races by extending blkcg_mutex coverage to serialize blkcg_activate_policy() rollback and blkg destruction, ensuring pd lifecycle is synchronized with blkg list visibility. Fixes: f1c006f1c685 ("blk-cgroup: synchronize pd_free_fn() from blkg_free_workfn() and blkcg_deactivate_policy()") Signed-off-by: Zheng Qixing Reviewed-by: Tang Yizhou Signed-off-by: Yu Kuai Reviewed-by: Tao Cui Reviewed-by: Nilay Shroff Link: https://patch.msgid.link/20260802112525.3933753-3-yukuai@kernel.org Signed-off-by: Jens Axboe

blk-cgroup: protect q->blkg_list iteration in blkg_destroy_all() with blkcg_mutex

2026-08-04T02:02:34+00:00

blkg_destroy_all() iterates q->blkg_list without holding blkcg_mutex, which can race with blkg_free_workfn() that removes blkgs from the list while holding blkcg_mutex. Add blkcg_mutex protection around the q->blkg_list iteration to prevent potential list corruption or use-after-free issues. Reviewed-by: Tang Yizhou Signed-off-by: Yu Kuai Reviewed-by: Tao Cui Reviewed-by: Nilay Shroff Link: https://patch.msgid.link/20260802112525.3933753-2-yukuai@kernel.org Signed-off-by: Jens Axboe

blk-cgroup: fix leaks and online flag on radix_tree_insert failure

2026-07-15T22:31:43+00:00

When radix_tree_insert() fails in blkg_create(), the error path has two issues: 1. blkg->online is set to true unconditionally, even when the blkg was never fully inserted. Move the assignment inside the success block. 2. The error path calls blkg_put() without first calling percpu_ref_kill(). Because the refcount is still in percpu mode, percpu_ref_put() only does this_cpu_sub() without checking for zero, so blkg_release() is never triggered. This permanently leaks the blkg memory, its percpu iostat, policy data, the parent blkg reference, and the cgroup css reference — the latter preventing the cgroup from ever being destroyed. Fix by replacing blkg_put() with percpu_ref_kill(), matching the pattern used in blkg_destroy(). Acked-by: Tejun Heo Signed-off-by: Tao Cui Link: https://patch.msgid.link/20260715132407.1469777-1-cui.tao@linux.dev Signed-off-by: Jens Axboe

blk-cgroup: don't nest queue_lock under blkcg->lock in blkcg_destroy_blkgs()

2026-06-24T12:42:31+00:00

The correct lock order is q->queue_lock before blkcg->lock, and in order to prevent deadlock from blkcg_destroy_blkgs(), trylock is used for q->queue_lock while blkcg->lock is already held, this is hacky. Refactor blkcg_destroy_blkgs() to hold blkcg->lock only long enough to get the first blkg and then release it. Then take q->queue_lock and blkcg->lock in the correct order to destroy the blkg. This is a very cold path, so the extra lock/unlock cycles are acceptable. Also prepare to convert protecting blkcg with blkcg_mutex instead of queue_lock. Signed-off-by: Yu Kuai Link: https://patch.msgid.link/00b03cf74a9937cb4d6dd67a189ddc00a3de0451.1780621988.git.yukuai@fygo.io Signed-off-by: Jens Axboe

blk-cgroup: don't nest queue_lock under rcu in bio_associate_blkg()

2026-06-24T12:42:31+00:00

If a bio is already associated with a blkg, the blkcg is already pinned until the bio is done, so there is no need for RCU protection. Otherwise, protect blkcg_css() with RCU independently. Prepare to protect blkcg with blkcg_mutex instead of queue_lock. Signed-off-by: Yu Kuai Link: https://patch.msgid.link/8496fa234b21d4b31b7f068766906d0bffcac8e6.1780621988.git.yukuai@fygo.io Signed-off-by: Jens Axboe

blk-cgroup: don't nest queue_lock under rcu in blkg_lookup_create()

2026-06-24T12:42:31+00:00

Change this in two steps: 1) hold rcu lock and do blkg_lookup() from fast path; 2) hold queue_lock directly from slow path, and don't nest it under rcu lock; Prepare to convert protecting blkcg with blkcg_mutex instead of queue_lock. Signed-off-by: Yu Kuai Link: https://patch.msgid.link/93f33cc9e5a39dddb78dcd934d0c1d04b564fb00.1780621988.git.yukuai@fygo.io Signed-off-by: Jens Axboe

blk-cgroup: don't nest queue_lock under rcu in blkcg_print_blkgs()

2026-06-24T12:42:19+00:00

With previous modification to delay freeing policy data after an RCU grace period, prfill() can run under RCU instead of taking queue_lock. However, policy teardown can still clear blkg->pd[plid] after blkcg_print_blkgs() observes the policy enabled bit. Load policy data once with READ_ONCE() and skip the blkg if teardown already cleared it. Do the same in recursive stat walks for descendant blkgs. Remove the stale BFQ debug queue_lock assertion because blkcg_print_blkgs() no longer calls prfill() with queue_lock held. This also lets ioc_qos_prfill() and ioc_cost_model_prfill() use IRQ-safe ioc->lock locking without re-enabling IRQs while queue_lock is still held. Signed-off-by: Yu Kuai Link: https://patch.msgid.link/db7633d5e263dd1c2bf9b901762545a84b7d714e.1780621988.git.yukuai@fygo.io Signed-off-by: Jens Axboe