From 1719d035a6fa90b7467b6daf45a573f5180013b2 Mon Sep 17 00:00:00 2001 From: Zhan Xusheng Date: Sat, 22 Aug 2026 18:59:30 +0800 Subject: sched/fair: Use update_curr_eevdf() for the remaining root cfs_rq callers pick_task_fair() and yield_task_fair() call update_curr(&rq->cfs) to bring curr up to date before they look at the eevdf state. With cgroups that does not happen: update_curr() reads ->h_curr, which on the root cfs_rq is the top level group entity, and returns at the !entity_is_task() check before touching vruntime. Both then read ->curr, so the guard and the update disagree about which entity they mean. Counting how often ->h_curr and ->curr differ at pick_task_fair(), on one CPU for 10s with three busy tasks and one 200us-periodic task: all tasks in the root cgroup 43321 calls, 0 no-ops busy tasks in G0, periodic in G1 45211 calls, 45193 no-ops Whether that matters depends on what precedes the pick. Since commit 68e37487810a ("sched/fair: Fix flat hierarchy") the tick and enqueue/dequeue all update curr correctly, so on the normal reschedule path only the microseconds between those and the pick are missing, and I could not measure a latency difference there. Three paths have nothing before them on that rq though: - pick_task() on the sibling rqs of a core under core scheduling (kernel/sched/core.c), which updates that rq's clock first for exactly this reason - fair_server_pick_task() - yield_task_fair(), where the stale value feeds the entity_eligible() test that guards forfeiting the remaining vruntime There curr can be a full tick behind, as it was before that commit. No new behaviour for the entity being updated: without cgroups ->h_curr is already the task, so these two call sites already run the full update_curr() including update_deadline(), dl_server_update() and the resched_curr_lazy() at the end. This makes the cgroup case do the same. Fixes: 85570f10a4c6 ("sched/eevdf: Move to a single runqueue") Signed-off-by: Zhan Xusheng Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Vincent Guittot Link: https://patch.msgid.link/20260822105930.2352761-1-zhanxusheng1024@gmail.com --- kernel/sched/fair.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 8dff37059faf..5d47de512f32 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -10057,7 +10057,7 @@ again: /* Might not have done put_prev_entity() */ if (cfs_rq->curr && cfs_rq->curr->on_rq) - update_curr(cfs_rq); + update_curr_eevdf(cfs_rq); se = pick_next_entity(rq, true); if (!se) @@ -10160,7 +10160,7 @@ static void yield_task_fair(struct rq *rq) /* * Update run-time statistics of the 'current'. */ - update_curr(cfs_rq); + update_curr_eevdf(cfs_rq); /* * Tell update_rq_clock() that we've just updated, * so we don't do microscopic update in schedule() -- cgit v1.2.3 From dae5c0292080dd7b9c7d784268dcf443f1f3d15e Mon Sep 17 00:00:00 2001 From: Seiji Nishikawa Date: Sun, 30 Aug 2026 16:37:46 +0900 Subject: sched/rt,dl: Skip migrate-disabled tasks when picking a push candidate A migrate_disable()'d RT task cannot be moved to another CPU, but the scheduler still keeps such a task on that CPU's pushable list (rq->rt.pushable_tasks) and still marks the runqueue RT-overloaded (rq->rt.overloaded = 1). So the RT balancer keeps treating this CPU as having a task to move away, and keeps trying to move the task, but the push can never succeed. When the head is pinned, push_rt_task() does not give up either. It falls back to pushing rq->curr instead, using the per-CPU stopper, as added by commit a7c81556ec4d ("sched: Fix migrate_disable() vs rt/dl balancing"). The CPU spends tens of milliseconds in this retry loop. The core is isolated for real-time work, but during the loop nearly half of its time is consumed by pushes that cannot succeed. An ftrace capture of the affected CPU, with sched_switch enabled and commit 94894c9c477e ("sched/rt: Skip currently executing CPU in rto_next_cpu()") applied, shows where the CPU time went. Two SCHED_FIFO tasks at equal priority shared the CPU, taskA migrate_disable()'d and queued, taskB as rq->curr. In one 89 ms window, taskB got only 52 ms of CPU. The other 37 ms went to the stopper thread. The scheduler kept trying to push taskA, the pinned head of the pushable list, fell back to pushing taskB instead, and woke the stopper 5204 times. Every one of those pushes failed and no task was moved. taskA stayed runnable and queued the whole time, and never ran. Pushing taskB fails on a re-check. find_lock_lowest_rq() drops the rq lock to take the target rq lock, then checks again with "task != pick_next_pushable_task(rq)". The task being pushed is taskB, but the pick returns taskA, the head of the pushable list. taskB is rq->curr, and set_next_task_rt() removes the running task from that list, so taskB can never be the head. The check expects a candidate taken from the pushable list, but the fallback pushes rq->curr, which is never on that list. So the check fails every time. .--> push-IPI arrives | | | v | pushable head = taskA -> pinned, cannot be pushed | | | v | so push taskB instead -> wake migration/N, a stop-class | | thread, so it preempts taskB | v | re-check compares taskB against the pushable head, | which is still taskA -> give up | | | v | nothing moved, taskA still queued, rq still overloaded | | '----------' repeats every ~17 us, 5204 times, for 89 ms The loop cannot stop itself. Every round leaves the runqueue exactly as it was, so the next push-IPI does the same thing. In the capture it ended only when taskB went to sleep on its own. taskA was then picked locally and left the pushable list. CPU time per task in the window, from sched_switch: taskB 51.95 ms real work migration/N 37.18 ms nothing moved taskA 0.00 ms queued the whole time, never picked idle 0.01 ms Counts over the same window: 7667 push-IPIs handled on this CPU 17481 pick_next_pushable_task() returned taskA, still pinned 5204 find_lock_lowest_rq() gave up on the re-check 1 push that actually completed 0 migrations of taskA The CPU times and the window length come from the standard sched_switch tracepoint. The counts needed tracepoints added inside the RT balancer for this investigation. The self-IPI path is closed by the rto_next_cpu() fix above, and that part works. But the runqueue is still marked overloaded, because the pinned task is still advertised as pushable. Other CPUs now send the push-IPIs during their own RT balancing, and the same loop runs again. Closing the self-IPI path did not stop a pinned task from triggering push balancing. A pinned task should never have been returned as a push candidate in the first place. A migrate_disable()'d task cannot be migrated, so it belongs in the same skip that was added for on_cpu tasks by commit e0ca8991b2de ("sched: Make class_schedulers avoid pushing current, and get rid of proxy_tag_curr()"). Add is_migration_disabled() to the skip condition in pick_next_pushable_task() and pick_next_pushable_dl_task(). With the skip in place, if the pinned task is the only extra runnable task the helpers return NULL, push_rt_task() and push_dl_task() give up early, and no stopper is woken. The pinned task then runs locally once curr yields. If a task that really can be migrated is queued behind the pinned head, it is now picked and pushed for real. This makes the fallback that pushes rq->curr unreachable when the pushable head is migrate-disabled. Nothing is lost, because that path was always stopped by the re-check described above. In the capture it ran 5204 times and moved nothing. Fixes: a7c81556ec4d ("sched: Fix migrate_disable() vs rt/dl balancing") Signed-off-by: Seiji Nishikawa Signed-off-by: Peter Zijlstra (Intel) Link: https://patch.msgid.link/20260830073746.2189355-1-snishika@redhat.com --- kernel/sched/deadline.c | 4 ++-- kernel/sched/rt.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 857dbe3519a8..0663c00c41c0 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -3028,8 +3028,8 @@ static struct task_struct *pick_next_pushable_dl_task(struct rq *rq) next_node = rb_first_cached(&rq->dl.pushable_dl_tasks_root); while (next_node) { i = __node_2_pdl(next_node); - /* make sure task isn't on_cpu (possible with proxy-exec) */ - if (!task_on_cpu(rq, i)) { + /* skip tasks that cannot be migrated */ + if (!task_on_cpu(rq, i) && !is_migration_disabled(i)) { p = i; break; } diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c index e6e5f8a2caaf..85303add726d 100644 --- a/kernel/sched/rt.c +++ b/kernel/sched/rt.c @@ -1872,8 +1872,8 @@ static struct task_struct *pick_next_pushable_task(struct rq *rq) return NULL; plist_for_each_entry(i, head, pushable_tasks) { - /* make sure task isn't on_cpu (possible with proxy-exec) */ - if (!task_on_cpu(rq, i)) { + /* skip tasks that cannot be migrated */ + if (!task_on_cpu(rq, i) && !is_migration_disabled(i)) { p = i; break; } -- cgit v1.2.3 From c6dcd97c8be75f052a1ca52cf79b03e7292962f1 Mon Sep 17 00:00:00 2001 From: "Shubhang Kaushik (Ampere)" Date: Fri, 7 Aug 2026 13:38:52 -0700 Subject: sched/core: Skip rq->avg_idle update without a valid idle_stamp Commit 4b603f1551a73 ("sched: Update rq->avg_idle when a task is moved to an idle CPU") moved rq->avg_idle accounting out of the wakeup path and into put_prev_task_idle(), so that the idle interval is consumed whenever the idle task is switched out. The wakeup-side accounting that it replaced only updated rq->avg_idle when rq->idle_stamp was non-zero. The new helper lost that validity check and unconditionally computes: rq_clock(rq) - rq->idle_stamp If rq->idle_stamp is zero, this uses rq_clock(rq) as the sample. That is not a valid idle duration and can immediately drive rq->avg_idle to its clamp. This can happen when sched_balance_newidle() returns before setting rq->idle_stamp, for example when this_rq->ttwu_pending is set. In that case the rq can switch to the idle task with idle_stamp still zero and leave idle again when the pending wakeup is processed. Other paths can also switch to the idle task without setting rq->idle_stamp via newidle_balance(), for example find_proxy_task() or force-idling. Restore the idle_stamp validity check in update_rq_avg_idle() and skip the rq->avg_idle update when there is no measured idle interval. Fixes: 4b603f1551a73 ("sched: Update rq->avg_idle when a task is moved to an idle CPU") Signed-off-by: Shubhang Kaushik (Ampere) Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: K Prateek Nayak Reviewed-by: Vincent Guittot Acked-by: John Stultz Link: https://patch.msgid.link/20260807-master-v3-1-c328354efed3@gentwo.org --- kernel/sched/core.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index f78275192036..74724501c623 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3742,11 +3742,17 @@ static inline void ttwu_do_wakeup(struct task_struct *p) void update_rq_avg_idle(struct rq *rq) { - u64 delta = rq_clock(rq) - rq->idle_stamp; - u64 max = 2*rq->max_idle_balance_cost; + u64 idle_stamp = rq->idle_stamp; + u64 delta, max; + + if (!idle_stamp) + return; + + delta = rq_clock(rq) - idle_stamp; update_avg(&rq->avg_idle, delta); + max = 2 * rq->max_idle_balance_cost; if (rq->avg_idle > max) rq->avg_idle = max; rq->idle_stamp = 0; -- cgit v1.2.3 From f8610c57f4078c63d1d4e2f3d7134f3dc1768403 Mon Sep 17 00:00:00 2001 From: Wanwu Li Date: Mon, 31 Aug 2026 18:11:40 +0800 Subject: sched/fair: Use cfs_rq->h_curr in throttle_cfs_rq() After commit 85570f10a4c6 ("sched/eevdf: Move to a single runqueue"), cfs_rq->curr is only maintained on the root cfs_rq (set/cleared from set_next_task_fair()/put_prev_task_fair()), while cfs_rq->h_curr is the per-level current entity, set by set_next_entity() at every level of the hierarchy. For an intermediate cfs_rq (a cgroup), cfs_rq->curr is always NULL, but cfs_rq->h_curr is the group entity at that level. throttle_cfs_rq() reads cfs_rq->curr to decide whether there is a running entity at the throttled level, in which case it should request a full sched_cfs_bandwidth_slice() of runtime and arm the deferred throttle task_work via task_throttle_setup_work(). For intermediate cfs_rqs the check is always false, so bandwidth-controlled cgroups always get just 1ns of runtime and never arm the deferred throttle work; the running task then escapes throttling until the next pick arms the work instead, even though there is an on-rq entity at this level. Switch the read to cfs_rq->h_curr so intermediate bandwidth-controlled cgroups behave consistently with the root cfs_rq, matching the existing usage of cfs_rq->h_curr in update_curr() and check_enqueue_throttle(). Fixes: 85570f10a4c6 ("sched/eevdf: Move to a single runqueue") Signed-off-by: Wanwu Li Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Aaron Lu Tested-by: Aaron Lu Link: https://patch.msgid.link/20260831101141.391382-2-liwanwu@kylinos.cn --- kernel/sched/fair.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 5d47de512f32..73797d661486 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6978,14 +6978,14 @@ static int tg_throttle_down(struct task_group *tg, void *data) static bool throttle_cfs_rq(struct cfs_rq *cfs_rq) { struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg); - struct sched_entity *curr = cfs_rq->curr; + struct sched_entity *curr = cfs_rq->h_curr; struct rq *rq = rq_of(cfs_rq); scoped_guard(raw_spinlock, &cfs_b->lock) { u64 target_runtime = 1; /* - * If cfs_rq->curr is still runnable, we are here from an + * If cfs_rq->h_curr is still runnable, we are here from an * update_curr(). Request sysctl_sched_cfs_bandwidth_slice * worth of bandwidth to continue running. * -- cgit v1.2.3 From b038383526d8c7883ea0486dd1911102b6dda414 Mon Sep 17 00:00:00 2001 From: Wanwu Li Date: Mon, 31 Aug 2026 18:11:41 +0800 Subject: sched/fair: Use cfs_rq->h_curr in distribute_cfs_runtime() distribute_cfs_runtime() refreshes the rq clock and accounts elapsed runtime with update_curr() before redistributing bandwidth, but gates this on cfs_rq->curr. Since commit 85570f10a4c6 ("sched/eevdf: Move to a single runqueue") cfs_rq->curr is only maintained on the root cfs_rq, so for the cgroup cfs_rqs it walks, the check never fires and the refresh is dead code. Use cfs_rq->h_curr, the per-level current entity, restoring the intended behaviour: only refresh when something is actually running at the throttled level, i.e. within the deferred throttle window. Without this, runtime consumed by a still-running task of the throttled hierarchy is not docked before redistribution; unthrottle_cfs_rq() catches up unconditionally since commit 28ad5427682b ("sched/fair: Call update_curr() before unthrottling the hierarchy"), so this is not a correctness hole today, but the refresh the check was written for is gone. Fixes: 85570f10a4c6 ("sched/eevdf: Move to a single runqueue") Signed-off-by: Wanwu Li Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Aaron Lu Tested-by: Aaron Lu Link: https://patch.msgid.link/20260831101141.391382-3-liwanwu@kylinos.cn --- kernel/sched/fair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 73797d661486..97021a5033fb 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7192,7 +7192,7 @@ static bool distribute_cfs_runtime(struct cfs_bandwidth *cfs_b) if (!list_empty(&cfs_rq->throttled_csd_list)) continue; - if (cfs_rq->curr) { + if (cfs_rq->h_curr) { update_rq_clock(rq); update_curr(cfs_rq); } -- cgit v1.2.3 From eaece4849991d62fcd6f46637c55dcce00e25d70 Mon Sep 17 00:00:00 2001 From: Mario Limonciello Date: Mon, 31 Aug 2026 00:38:33 -0500 Subject: x86/itmt: Don't make ITMT enablement depend on debugfs sched_set_itmt_support() treats debugfs file creation failures as fatal. When CONFIG_DEBUG_FS is disabled, debugfs stubs return ERR_PTR(-ENODEV), causing ITMT to be silently disabled. debugfs is a debug-only facility; its return values should be ignored. Drop the fatal error handling and enable ITMT unconditionally. Fixes: d04013a4b21b ("x86/itmt: Move the "sched_itmt_enabled" sysctl to debugfs") Reported-by: Klaus Kusche Signed-off-by: Mario Limonciello Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Tim Chen Reviewed-by: K Prateek Nayak Tested-by: K Prateek Nayak Link: https://patch.msgid.link/20260831053836.1881864-1-mario.limonciello@amd.com --- arch/x86/kernel/itmt.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/arch/x86/kernel/itmt.c b/arch/x86/kernel/itmt.c index 243a769fdd97..85ebde361d6a 100644 --- a/arch/x86/kernel/itmt.c +++ b/arch/x86/kernel/itmt.c @@ -110,18 +110,14 @@ int sched_set_itmt_support(void) arch_debugfs_dir, &sysctl_sched_itmt_enabled, &dfs_sched_itmt_fops); - if (IS_ERR_OR_NULL(dfs_sched_itmt)) { + if (IS_ERR(dfs_sched_itmt)) dfs_sched_itmt = NULL; - return -ENOMEM; - } dfs_sched_core_prio = debugfs_create_file("sched_core_priority", 0644, arch_debugfs_dir, NULL, &sched_core_priority_fops); - if (IS_ERR_OR_NULL(dfs_sched_core_prio)) { + if (IS_ERR(dfs_sched_core_prio)) dfs_sched_core_prio = NULL; - return -ENOMEM; - } sched_itmt_capable = true; -- cgit v1.2.3 From f0d243a96f2684ad771d678767d17972cf840bd7 Mon Sep 17 00:00:00 2001 From: Tim Chen Date: Mon, 31 Aug 2026 10:40:53 -0700 Subject: sched/fair: Avoid creating misfits during cache-aware balancing Cache-aware load balancing biases tasks toward their preferred LLC. On asymmetric CPU capacity systems (e.g. big.LITTLE) the destination LLC may contain CPUs that are too small to run the task. Pulling the task there turns it into a misfit, trading a cache-locality gain for a capacity loss that's more detrimental to performance. Guard both cache-aware migration entry points against this: - can_migrate_llc_task(): forbid the LLC migration when the task fits its source CPU but would not fit the destination CPU. - alb_break_llc(): veto the active balance under the same condition so the runnable task is not pushed onto a CPU that cannot accommodate it. Both checks are gated with checks for hybrid processors, so symmetric systems are unaffected. Tasks that already do not fit their source CPU are left to the existing LLC policy, since the move cannot make their fitness worse (this also preserves misfit up-migration to bigger CPUs). Additionally, if there are misfit tasks found in the load balancing classification phase, prioritize misfit task migrations over LLC load aggregation on asymmetric systems. A better fitting CPU will boost performance more than better cache locality. Reviewed-by: Ricardo Neri Tested-by: Ricardo Neri Reviewed-by: Chen Yu Signed-off-by: Tim Chen Signed-off-by: Peter Zijlstra (Intel) Link: https://patch.msgid.link/edbb2503d554c63dc9b72e201fb4a17e1cb119e7.camel@linux.intel.com --- kernel/sched/fair.c | 50 +++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 45 insertions(+), 5 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 97021a5033fb..ade1eceb39b8 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -10691,17 +10691,40 @@ static enum llc_mig can_migrate_llc(int src_cpu, int dst_cpu, return mig_llc; } +static inline bool task_misfits_asym_cpu(struct lb_env *env, struct task_struct *p) +{ + /* + * On asymmetric CPU capacity domains, do not let cache-aware + * balancing pull the task onto a destination CPU that cannot + * accommodate it. Doing so would turn the task into a misfit on + * the destination, trading a cache-locality gain for a capacity + * loss. If the task already does not fit its source CPU, the move + * cannot make things worse, so let the LLC preference decide. + */ + if ((env->sd->flags & SD_ASYM_CPUCAPACITY) && p && + !task_fits_cpu(p, env->dst_cpu) && + task_fits_cpu(p, env->src_cpu)) + return true; + + return false; +} + /* * Check if task p can migrate from source LLC to * destination LLC in terms of cache aware load balance. */ -static enum llc_mig can_migrate_llc_task(int src_cpu, int dst_cpu, +static enum llc_mig can_migrate_llc_task(struct lb_env *env, struct task_struct *p) { struct mm_struct *mm; bool to_pref; - int cpu; + int cpu, src_cpu, dst_cpu; + + if (task_misfits_asym_cpu(env, p)) + return mig_forbid; + src_cpu = env->src_cpu; + dst_cpu = env->dst_cpu; mm = p->mm; if (!mm) return mig_unrestricted; @@ -10758,6 +10781,14 @@ alb_break_llc(struct lb_env *env) unsigned long util = 0; struct task_struct *cur; + /* + * Migrating misfit tasks from current CPU + * to CPU with a better fit. + * Prioritize that over LLC preference. + */ + if (env->migration_type == migrate_misfit) + return false; + if (env->src_rq->nr_running <= 1) return true; @@ -10765,7 +10796,8 @@ alb_break_llc(struct lb_env *env) if (cur && cur->sched_class == &fair_sched_class) util = task_util(cur); - if (can_migrate_llc(env->src_cpu, env->dst_cpu, + if (task_misfits_asym_cpu(env, cur) || + can_migrate_llc(env->src_cpu, env->dst_cpu, util, false) == mig_forbid) return true; } @@ -10805,8 +10837,7 @@ static bool migrate_degrades_llc(struct task_struct *p, struct lb_env *env) READ_ONCE(p->preferred_llc) != llc_id(env->dst_cpu)) return true; - if (can_migrate_llc_task(env->src_cpu, - env->dst_cpu, p) != mig_forbid) + if (can_migrate_llc_task(env, p) != mig_forbid) return false; return true; @@ -11869,6 +11900,15 @@ static inline bool llc_balance(struct lb_env *env, struct sg_lb_stats *sgs, if (env->sd->flags & SD_SHARE_LLC) return false; + /* + * On asymmetric domains, group_misfit_task_load + * should be prioritized to move tasks to CPU that fit them + * over aggregating tasks to their preferred LLC. + */ + if ((env->sd->flags & SD_ASYM_CPUCAPACITY) && + sgs->group_misfit_task_load) + return false; + /* * Skip cache aware tagging if nr_balanced_failed is sufficiently high. * Threshold of cache_nice_tries is set to 1 higher than nr_balance_failed -- cgit v1.2.3