Received: by 2002:ac0:a5a6:0:0:0:0:0 with SMTP id m35-v6csp2331973imm; Fri, 7 Sep 2018 14:49:42 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZNcmsHEpouJMGlUwLp/SWG08dCfnQabPZrr2gKtLKpcWhJnTY8UVPIFuXTjxOSmvsS1f7N X-Received: by 2002:a17:902:be07:: with SMTP id r7-v6mr9919216pls.275.1536356982094; Fri, 07 Sep 2018 14:49:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536356982; cv=none; d=google.com; s=arc-20160816; b=anjDEVsdzeLVtlnou3LQzuX+/3XNiMsG6qYDgJTs5b2e+6zgv+o5h/KfNr9SixrOxy REOTpDw7d13/Sxr++LlJxivjDHqPWqSlStTlj2VZ2N9TrWT9QkBDYN4XxMHAP/MaWrgf oRKzjH6ch1/80T3bNk1sNR8NS+Q7VCE2uZNOZVfV0TuPeZDAV+AeIAv72x0gDmKmSL7x ZfBLRKvy26Be98TA8P1HNI/RB6vDXUWW7644fgHyt6DePl6r/NB4cnhUCzoZgQIDdHy7 soeFFwRYgk3Y5u1maHoWU2n4If3l1zIrj1mSbLE3oQYGXqVptAhIC5sPoiluCmQzHcUW 4j+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=FpZ0/MfL21vL2c1L8np9yXoUjyIFjQlrDZOyOtX+O6k=; b=IguOIOw0E50UkNx0/BeQEP4Fh7B9BIpodVF2vFzvn81mfwcxa+Dh1pIcN+GKysXYFP 4mXL6DAWptX+zoZdP1opxdFdNLIamFuiwCGpOM5oTomMpHGtLbvxqfu03UpymJ9aOeuX cNvBTkBdnnjGRYwhZj4z0oAufO9bWcgn4lAjWyp6YbLdx3abE7r50ywRFzUJsfIhxO37 fJZrHjwCpcdLX8kl8e663RWhDm6Q2+5FkKy+vVO/nxqnrhU1TIbJ/GQcQcaNWnF00kpE 2G7ou92TRdZJfgvolgVsI/OUhMMA+HzpKM8Qz+xoD2sa1Zn6fsybkegaIqDDId8rtK3c cjYA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@amazon.de header.s=amazon201209 header.b=LouRQb7u; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.de Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id a61-v6si9449220plc.80.2018.09.07.14.49.27; Fri, 07 Sep 2018 14:49:42 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@amazon.de header.s=amazon201209 header.b=LouRQb7u; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731133AbeIHCac (ORCPT + 99 others); Fri, 7 Sep 2018 22:30:32 -0400 Received: from smtp-fw-33001.amazon.com ([207.171.190.10]:27029 "EHLO smtp-fw-33001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728464AbeIHCab (ORCPT ); Fri, 7 Sep 2018 22:30:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.de; i=@amazon.de; q=dns/txt; s=amazon201209; t=1536356854; x=1567892854; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=FpZ0/MfL21vL2c1L8np9yXoUjyIFjQlrDZOyOtX+O6k=; b=LouRQb7upUF8DoCp7DjmSbmvpiXIKtVXnHip1ZkYbirTec41aKl5WhCX GnI0B/davkdGx56m4MOrKeQCCvvPW6pVYxxJIa2LAonNi8lNU0KbeQfGU 3cWrVkE48P9PjQ8pBGR/am8+D4DebrlEZM7tY6jhrErxWDjrc4/0Ov5ph s=; X-IronPort-AV: E=Sophos;i="5.53,343,1531785600"; d="scan'208";a="752286893" Received: from sea3-co-svc-lb6-vlan2.sea.amazon.com (HELO email-inbound-relay-1a-16acd5e0.us-east-1.amazon.com) ([10.47.22.34]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 07 Sep 2018 21:45:04 +0000 Received: from u7588a65da6b65f.ant.amazon.com (iad7-ws-svc-lb50-vlan2.amazon.com [10.0.93.210]) by email-inbound-relay-1a-16acd5e0.us-east-1.amazon.com (8.14.7/8.14.7) with ESMTP id w87LgDE4005924 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Fri, 7 Sep 2018 21:42:16 GMT Received: from u7588a65da6b65f.ant.amazon.com (localhost [127.0.0.1]) by u7588a65da6b65f.ant.amazon.com (8.15.2/8.15.2/Debian-3) with ESMTPS id w87LgB6S027470 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 7 Sep 2018 23:42:12 +0200 Received: (from jschoenh@localhost) by u7588a65da6b65f.ant.amazon.com (8.15.2/8.15.2/Submit) id w87LgBER027468; Fri, 7 Sep 2018 23:42:11 +0200 From: =?UTF-8?q?Jan=20H=2E=20Sch=C3=B6nherr?= To: Ingo Molnar , Peter Zijlstra Cc: =?UTF-8?q?Jan=20H=2E=20Sch=C3=B6nherr?= , linux-kernel@vger.kernel.org Subject: [RFC 29/60] cosched: Adjust code reflecting on the total number of CFS tasks on a CPU Date: Fri, 7 Sep 2018 23:40:16 +0200 Message-Id: <20180907214047.26914-30-jschoenh@amazon.de> X-Mailer: git-send-email 2.9.3.1.gcba166c.dirty In-Reply-To: <20180907214047.26914-1-jschoenh@amazon.de> References: <20180907214047.26914-1-jschoenh@amazon.de> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org There are a few places that make decisions based on the total number of CFS tasks on a certain CPU. With coscheduling, the inspected value rq->cfs.h_nr_running does not contain all tasks anymore, as some are accounted on higher hierarchy levels instead. This would lead to incorrect conclusions as the system seems more idle than it actually is. Adjust these code paths to use an alternative way of deriving the same value: take the total amount of tasks on the runqueue and subtract all running tasks of other scheduling classes. What remains are all CFS tasks on a certain CPU. Signed-off-by: Jan H. Schönherr --- kernel/sched/core.c | 5 ++--- kernel/sched/fair.c | 11 +++++------ kernel/sched/sched.h | 21 +++++++++++++++++++++ 3 files changed, 28 insertions(+), 9 deletions(-) diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 5350cab7ac4a..337bae6fa836 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -3326,12 +3326,12 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) /* * Optimization: we know that if all tasks are in the fair class we can * call that function directly, but only if the @prev task wasn't of a - * higher scheduling class, because otherwise those loose the + * higher scheduling class, because otherwise those lose the * opportunity to pull in more work from other CPUs. */ if (likely((prev->sched_class == &idle_sched_class || prev->sched_class == &fair_sched_class) && - rq->nr_running == rq->cfs.h_nr_running)) { + rq->nr_running == nr_cfs_tasks(rq))) { p = fair_sched_class.pick_next_task(rq, prev, rf); if (unlikely(p == RETRY_TASK)) @@ -3343,7 +3343,6 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) return p; } - again: for_each_class(class) { p = class->pick_next_task(rq, prev, rf); diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 9cbdd027d449..30e5ff30f442 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -4460,7 +4460,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cfs_rq) add_nr_running(rq, task_delta); /* Determine whether we need to wake up potentially idle CPU: */ - if (rq->curr == rq->idle && rq->cfs.nr_running) + if (rq->curr == rq->idle && nr_cfs_tasks(rq)) resched_curr(rq); } @@ -4937,7 +4937,7 @@ static void hrtick_start_fair(struct rq *rq, struct task_struct *p) SCHED_WARN_ON(task_rq(p) != rq); - if (rq->cfs.h_nr_running > 1) { + if (nr_cfs_tasks(rq) > 1) { u64 slice = sched_slice(cfs_rq, se); u64 ran = se->sum_exec_runtime - se->prev_sum_exec_runtime; s64 delta = slice - ran; @@ -9067,8 +9067,7 @@ static void nohz_balancer_kick(struct rq *rq) sd = rcu_dereference(rq->sd); if (sd) { - if ((rq->cfs.h_nr_running >= 1) && - check_cpu_capacity(rq, sd)) { + if ((nr_cfs_tasks(rq) >= 1) && check_cpu_capacity(rq, sd)) { flags = NOHZ_KICK_MASK; goto unlock; } @@ -9479,7 +9478,7 @@ static int idle_balance(struct rq *this_rq, struct rq_flags *rf) * have been enqueued in the meantime. Since we're not going idle, * pretend we pulled a task. */ - if (this_rq->cfs.h_nr_running && !pulled_task) + if (nr_cfs_tasks(this_rq) && !pulled_task) pulled_task = 1; /* Move the next balance forward */ @@ -9487,7 +9486,7 @@ static int idle_balance(struct rq *this_rq, struct rq_flags *rf) this_rq->next_balance = next_balance; /* Is there a task of a high priority class? */ - if (this_rq->nr_running != this_rq->cfs.h_nr_running) + if (this_rq->nr_running != nr_cfs_tasks(this_rq)) pulled_task = -1; if (pulled_task) diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 5e2d231b1dbf..594eb9489f3d 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1691,6 +1691,27 @@ static inline int task_on_rq_migrating(struct task_struct *p) return p->on_rq == TASK_ON_RQ_MIGRATING; } +#ifdef CONFIG_COSCHEDULING +static inline unsigned int nr_cfs_tasks(struct rq *rq) +{ + unsigned int total = rq->nr_running; + + /* Deadline and real time tasks */ + total -= rq->dl.dl_nr_running + rq->rt.rt_nr_running; + + /* Stop task */ + if (rq->stop && task_on_rq_queued(rq->stop)) + total--; + + return total; +} +#else +static inline unsigned int nr_cfs_tasks(struct rq *rq) +{ + return rq->cfs.h_nr_running; +} +#endif + /* * wake flags */ -- 2.9.3.1.gcba166c.dirty