Received: by 2002:a25:ad19:0:0:0:0:0 with SMTP id y25csp11165484ybi; Thu, 25 Jul 2019 11:07:09 -0700 (PDT) X-Google-Smtp-Source: APXvYqxoW26T3jQVwmtGYn224QPhjW1+fLvqnzgnp1ckENxtNxG6x+mNcq38ObiCEQewLOOsd817 X-Received: by 2002:a17:902:361:: with SMTP id 88mr93546382pld.123.1564078029095; Thu, 25 Jul 2019 11:07:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1564078029; cv=none; d=google.com; s=arc-20160816; b=lB+7AW7VCJM7zj51itQJ9GbRiZZ8kPBaeky1c7jnBrx7HAzkcTo59BIgWtQNckm3y2 3x6vrjPIBxPvC0EKPOgZMdwjj1wHTvy/cuRCNWbw3vorIXqzaURjWkiiknVqmk/VIfHs nFOlpGGfV0qkWYhsTO1KaAYRjlHKWMzV9ejAf6akFAJXXDPoKIkwH1+nVhfuVKtg/0GG yweH2H5REvLJgkBPrT/jwQyQ0n0sUcORNIqRDpyW+X40+CtfXPiNojc7eLiSki3iMhMA bJVWCm3w2odjWYLBWSx4K/7Hn09J9CpB3X9Mof0zK6rH3aaUGlXleLccslrEYToC1w4R VqgA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-disposition :content-transfer-encoding:mime-version:robot-unsubscribe:robot-id :git-commit-id:subject:to:references:in-reply-to:reply-to:cc :message-id:from:date:dkim-signature:dkim-filter; bh=WDeeX8BvGqaCTB/0Sgu9Omxb4koUAUlpjeF0FBdEe4U=; b=h+a74pC+X+r/ugQAkw6z/LwHn3+lY/dz16JxfNZpHd0cB4WlhmO3X4VMyhkiuQ3ETR XET2cRaftcY/4BAmfobIRaLxjVAi8s99D1qcwayiHHwYSciDnPqIeXjZZQxDaT9ZFBmH 62b7gepy0pU6tZvvycO0OB6zVe1MH284QFaJCU2Q7ttmGBEoUKibqGm/MfrFF9bdw1j7 M5rj6xbD7/H47BHSsawhhOoEzap5VbI2EkElnmnnGRxXLf2tl6rT4TGH1K19Tm1x3FK0 uCrupwvPRd8DQmT9W9FTXv1aByCZTDPqJykvIpKuNztIBzdTQO7K5qNtaHSqsRCoh3fP Opkw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@zytor.com header.s=2019071901 header.b=t0H8ByBg; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=zytor.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id g13si2876132pgp.405.2019.07.25.11.06.53; Thu, 25 Jul 2019 11:07:09 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@zytor.com header.s=2019071901 header.b=t0H8ByBg; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=zytor.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390217AbfGYQQP (ORCPT + 99 others); Thu, 25 Jul 2019 12:16:15 -0400 Received: from terminus.zytor.com ([198.137.202.136]:54685 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390045AbfGYQQO (ORCPT ); Thu, 25 Jul 2019 12:16:14 -0400 Received: from terminus.zytor.com (localhost [127.0.0.1]) by terminus.zytor.com (8.15.2/8.15.2) with ESMTPS id x6PGG3cD1075138 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 25 Jul 2019 09:16:03 -0700 DKIM-Filter: OpenDKIM Filter v2.11.0 terminus.zytor.com x6PGG3cD1075138 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zytor.com; s=2019071901; t=1564071364; bh=WDeeX8BvGqaCTB/0Sgu9Omxb4koUAUlpjeF0FBdEe4U=; h=Date:From:Cc:Reply-To:In-Reply-To:References:To:Subject:From; b=t0H8ByBg3JW8l0LFYU/T2sWghQwLehNsZ3ylbDwaWUa/sy365KEdpNAsSkQev9nRI Ef/5DzPP9KrznpiF60riA4SXNKn+lk8U9Sr2NX9M0+jUd25iT9eaEsVjL1mYERDCGj CMuHQN/hZF50mq95e4e6ioeb218oNyHoFxP/P+9sJrqufhlsnxt0Qlly33fVmgQUyS 9Nx5f0G8epEYHRWsINP7xlgGpEiTQKwIRZlV+F/uKHMTIfGcutuevUi92hElsTJRl3 ipGFtXUFXZaR7gdkqVvUbE+cU8f0IeX68hfjsTJzPZJfMSfuUgf2qcGbjgS7HdPiOA kneaRV3LddPSw== Received: (from tipbot@localhost) by terminus.zytor.com (8.15.2/8.15.2/Submit) id x6PGG3RB1075134; Thu, 25 Jul 2019 09:16:03 -0700 Date: Thu, 25 Jul 2019 09:16:03 -0700 X-Authentication-Warning: terminus.zytor.com: tipbot set sender to tipbot@zytor.com using -f From: tip-bot for Viresh Kumar Message-ID: Cc: tglx@linutronix.de, daniel.lezcano@linaro.org, linux-kernel@vger.kernel.org, viresh.kumar@linaro.org, hpa@zytor.com, peterz@infradead.org, vincent.guittot@linaro.org, torvalds@linux-foundation.org, mingo@kernel.org Reply-To: tglx@linutronix.de, daniel.lezcano@linaro.org, linux-kernel@vger.kernel.org, viresh.kumar@linaro.org, hpa@zytor.com, vincent.guittot@linaro.org, peterz@infradead.org, mingo@kernel.org, torvalds@linux-foundation.org In-Reply-To: References: To: linux-tip-commits@vger.kernel.org Subject: [tip:sched/core] sched/fair: Fall back to sched-idle CPU if idle CPU isn't found Git-Commit-ID: 3c29e651e16dd3b3179cfb2d055ee9538e37515c X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline X-Spam-Status: No, score=-0.3 required=5.0 tests=ALL_TRUSTED,BAYES_00, DATE_IN_FUTURE_96_Q,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on terminus.zytor.com Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Commit-ID: 3c29e651e16dd3b3179cfb2d055ee9538e37515c Gitweb: https://git.kernel.org/tip/3c29e651e16dd3b3179cfb2d055ee9538e37515c Author: Viresh Kumar AuthorDate: Wed, 26 Jun 2019 10:36:30 +0530 Committer: Ingo Molnar CommitDate: Thu, 25 Jul 2019 15:51:54 +0200 sched/fair: Fall back to sched-idle CPU if idle CPU isn't found We try to find an idle CPU to run the next task, but in case we don't find an idle CPU it is better to pick a CPU which will run the task the soonest, for performance reason. A CPU which isn't idle but has only SCHED_IDLE activity queued on it should be a good target based on this criteria as any normal fair task will most likely preempt the currently running SCHED_IDLE task immediately. In fact, choosing a SCHED_IDLE CPU over a fully idle one shall give better results as it should be able to run the task sooner than an idle CPU (which requires to be woken up from an idle state). This patch updates both fast and slow paths with this optimization. Signed-off-by: Viresh Kumar Signed-off-by: Peter Zijlstra (Intel) Cc: Daniel Lezcano Cc: Linus Torvalds Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Vincent Guittot Cc: chris.redpath@arm.com Cc: quentin.perret@linaro.org Cc: songliubraving@fb.com Cc: steven.sistare@oracle.com Cc: subhra.mazumdar@oracle.com Cc: tkjos@google.com Link: https://lkml.kernel.org/r/eeafa25fdeb6f6edd5b2da716bc8f0ba7708cbcf.1561523542.git.viresh.kumar@linaro.org Signed-off-by: Ingo Molnar --- kernel/sched/fair.c | 43 +++++++++++++++++++++++++++++++++---------- 1 file changed, 33 insertions(+), 10 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 9ed5ab53872f..52564e050062 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -5431,6 +5431,15 @@ static struct { #endif /* CONFIG_NO_HZ_COMMON */ +/* CPU only has SCHED_IDLE tasks enqueued */ +static int sched_idle_cpu(int cpu) +{ + struct rq *rq = cpu_rq(cpu); + + return unlikely(rq->nr_running == rq->cfs.idle_h_nr_running && + rq->nr_running); +} + static unsigned long cpu_runnable_load(struct rq *rq) { return cfs_rq_runnable_load_avg(&rq->cfs); @@ -5753,7 +5762,7 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this unsigned int min_exit_latency = UINT_MAX; u64 latest_idle_timestamp = 0; int least_loaded_cpu = this_cpu; - int shallowest_idle_cpu = -1; + int shallowest_idle_cpu = -1, si_cpu = -1; int i; /* Check if we have any choice: */ @@ -5784,7 +5793,12 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this latest_idle_timestamp = rq->idle_stamp; shallowest_idle_cpu = i; } - } else if (shallowest_idle_cpu == -1) { + } else if (shallowest_idle_cpu == -1 && si_cpu == -1) { + if (sched_idle_cpu(i)) { + si_cpu = i; + continue; + } + load = cpu_runnable_load(cpu_rq(i)); if (load < min_load) { min_load = load; @@ -5793,7 +5807,11 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this } } - return shallowest_idle_cpu != -1 ? shallowest_idle_cpu : least_loaded_cpu; + if (shallowest_idle_cpu != -1) + return shallowest_idle_cpu; + if (si_cpu != -1) + return si_cpu; + return least_loaded_cpu; } static inline int find_idlest_cpu(struct sched_domain *sd, struct task_struct *p, @@ -5946,7 +5964,7 @@ static int select_idle_core(struct task_struct *p, struct sched_domain *sd, int */ static int select_idle_smt(struct task_struct *p, int target) { - int cpu; + int cpu, si_cpu = -1; if (!static_branch_likely(&sched_smt_present)) return -1; @@ -5956,9 +5974,11 @@ static int select_idle_smt(struct task_struct *p, int target) continue; if (available_idle_cpu(cpu)) return cpu; + if (si_cpu == -1 && sched_idle_cpu(cpu)) + si_cpu = cpu; } - return -1; + return si_cpu; } #else /* CONFIG_SCHED_SMT */ @@ -5986,8 +6006,8 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t u64 avg_cost, avg_idle; u64 time, cost; s64 delta; - int cpu, nr = INT_MAX; int this = smp_processor_id(); + int cpu, nr = INT_MAX, si_cpu = -1; this_sd = rcu_dereference(*this_cpu_ptr(&sd_llc)); if (!this_sd) @@ -6015,11 +6035,13 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t for_each_cpu_wrap(cpu, sched_domain_span(sd), target) { if (!--nr) - return -1; + return si_cpu; if (!cpumask_test_cpu(cpu, p->cpus_ptr)) continue; if (available_idle_cpu(cpu)) break; + if (si_cpu == -1 && sched_idle_cpu(cpu)) + si_cpu = cpu; } time = cpu_clock(this) - time; @@ -6038,13 +6060,14 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) struct sched_domain *sd; int i, recent_used_cpu; - if (available_idle_cpu(target)) + if (available_idle_cpu(target) || sched_idle_cpu(target)) return target; /* * If the previous CPU is cache affine and idle, don't be stupid: */ - if (prev != target && cpus_share_cache(prev, target) && available_idle_cpu(prev)) + if (prev != target && cpus_share_cache(prev, target) && + (available_idle_cpu(prev) || sched_idle_cpu(prev))) return prev; /* Check a recently used CPU as a potential idle candidate: */ @@ -6052,7 +6075,7 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target) if (recent_used_cpu != prev && recent_used_cpu != target && cpus_share_cache(recent_used_cpu, target) && - available_idle_cpu(recent_used_cpu) && + (available_idle_cpu(recent_used_cpu) || sched_idle_cpu(recent_used_cpu)) && cpumask_test_cpu(p->recent_used_cpu, p->cpus_ptr)) { /* * Replace recent_used_cpu with prev as it is a potential