Received: by 2002:ac0:a5a6:0:0:0:0:0 with SMTP id m35-v6csp2333716imm; Fri, 7 Sep 2018 14:51:57 -0700 (PDT) X-Google-Smtp-Source: ANB0Vdbgi/tmeIRicdRXdKTeYZuSfb2YubETa7QW8D5M1I8W8PUrCLO4pgMES9n64WhcBERBcZp1 X-Received: by 2002:a62:c90a:: with SMTP id k10-v6mr10753895pfg.180.1536357117744; Fri, 07 Sep 2018 14:51:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536357117; cv=none; d=google.com; s=arc-20160816; b=ey00I6O5R3RpjvHFlwhYLKtQzv/3GcYGObwC4aZrUWDAxx7yZRh47q4LhUaEE0bA// mhxy4/5dmFV6OTSfTjisbqrsSLc62gqYcNtmjxUVC5bKUj+VF7mEe7Um42NDS4+4otZp nD//c6J+K5JNUrOQNWItK2yjw/Ch/d1q1mwblS8NHO7xp0eoV763V20FK6FuESklwquC uWHY6HUqfjkxhQg2iqQEWoj5dt/lPSVbgkxZJMO3/zByx9K1c+H7Np6433I58uBJzp5F g7OTrn37AadiR3dOnI3HGVPgbabzJ3B6AX3SVt7AeZhV8O/7JWQ8PVpDG1cgvMme6/de hzEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=km3ah0eo7AG9vo5OgMHmPErlNk+gZ/1QSD4OmP5nt5c=; b=j8UB0CD2Cum5UQP5qBBq+L0kRldtYODTxCtQ/Xecp9rUsN+8yv2YJq+ySiQlk9C7eu A+6oahVPVYSic37zhztAndz72ugxa37mRzpvheHYZbvRlAQeydGMxYr95TlNvW5jlXgg ylZy1e4heMLLirISbkNsKgCJrdjay56fgmd5GV/I82ZrRGNrWgFRcASKb2b8cBfDE0ZK 0Ww/qXcJdg29GXuc8dJ31y/qLzb6jkOkyDMI6TIUSwS5/CYE9vbnvOXwO/rf6jEOMjuz Pzn7QQWtfzWJkJ3HcDF8RhE2JB/KW2AISIruLk1I/Jc/Iq8/mLPERgL1xiM0tPo1PqQd XPXQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@amazon.de header.s=amazon201209 header.b=o6V+5uga; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.de Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id m5-v6si10284431pgt.361.2018.09.07.14.51.42; Fri, 07 Sep 2018 14:51:57 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@amazon.de header.s=amazon201209 header.b=o6V+5uga; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730710AbeIHCZt (ORCPT + 99 others); Fri, 7 Sep 2018 22:25:49 -0400 Received: from smtp-fw-6002.amazon.com ([52.95.49.90]:23044 "EHLO smtp-fw-6002.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730236AbeIHCZs (ORCPT ); Fri, 7 Sep 2018 22:25:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.de; i=@amazon.de; q=dns/txt; s=amazon201209; t=1536356573; x=1567892573; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=km3ah0eo7AG9vo5OgMHmPErlNk+gZ/1QSD4OmP5nt5c=; b=o6V+5ugaEEWhp+Md//PYrV31HQSzCSXnGCKv1rdQFP2N+3MHPbYaN4o7 CgEyofFTMYWxWB/js8gIzKNYvdA+2R2qe7Qr4KyYK8pPPBm+/P+KKbedZ 8Soc/TM8nUv2E1n4HGjh4bDMvgnAwmjW8e/c5/8BiyfxP0+r/xnUuM/zW U=; X-IronPort-AV: E=Sophos;i="5.53,343,1531785600"; d="scan'208";a="361243252" Received: from iad6-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-1a-16acd5e0.us-east-1.amazon.com) ([10.124.125.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 07 Sep 2018 21:42:52 +0000 Received: from u7588a65da6b65f.ant.amazon.com (iad7-ws-svc-lb50-vlan2.amazon.com [10.0.93.210]) by email-inbound-relay-1a-16acd5e0.us-east-1.amazon.com (8.14.7/8.14.7) with ESMTP id w87Lgkvt021471 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Fri, 7 Sep 2018 21:42:48 GMT Received: from u7588a65da6b65f.ant.amazon.com (localhost [127.0.0.1]) by u7588a65da6b65f.ant.amazon.com (8.15.2/8.15.2/Debian-3) with ESMTPS id w87Lgi2o027719 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 7 Sep 2018 23:42:44 +0200 Received: (from jschoenh@localhost) by u7588a65da6b65f.ant.amazon.com (8.15.2/8.15.2/Submit) id w87Lghrn027718; Fri, 7 Sep 2018 23:42:43 +0200 From: =?UTF-8?q?Jan=20H=2E=20Sch=C3=B6nherr?= To: Ingo Molnar , Peter Zijlstra Cc: =?UTF-8?q?Jan=20H=2E=20Sch=C3=B6nherr?= , linux-kernel@vger.kernel.org Subject: [RFC 47/60] cosched: Adjust SE traversal and locking for common leader activities Date: Fri, 7 Sep 2018 23:40:34 +0200 Message-Id: <20180907214047.26914-48-jschoenh@amazon.de> X-Mailer: git-send-email 2.9.3.1.gcba166c.dirty In-Reply-To: <20180907214047.26914-1-jschoenh@amazon.de> References: <20180907214047.26914-1-jschoenh@amazon.de> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Modify some of the core scheduler paths, which function as entry points into the CFS scheduling class and which are activities where the leader operates on behalf of the group. There are (a) handling the tick, (b) picking the next task from the runqueue, (c) setting a task to be current, and (d) putting the current task back. Signed-off-by: Jan H. Schönherr --- kernel/sched/fair.c | 28 +++++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 2aa3a60dfca5..2227e4840355 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6664,12 +6664,14 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ static struct task_struct * pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) { - struct cfs_rq *cfs_rq = &rq->cfs; + struct cfs_rq *cfs_rq, *top_cfs_rq; + struct rq_owner_flags orf; struct sched_entity *se; struct task_struct *p; int new_tasks; again: + top_cfs_rq = cfs_rq = &rq_lock_owned(rq, &orf)->cfs; if (!cfs_rq->nr_running) goto idle; @@ -6707,7 +6709,7 @@ pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf * be correct. */ if (unlikely(check_cfs_rq_runtime(cfs_rq))) { - cfs_rq = &rq->cfs; + cfs_rq = top_cfs_rq; if (!cfs_rq->nr_running) goto idle; @@ -6775,9 +6777,13 @@ done: __maybe_unused; if (hrtick_enabled(rq)) hrtick_start_fair(rq, p); + rq_unlock_owned(rq, &orf); + return p; idle: + rq_unlock_owned(rq, &orf); + new_tasks = idle_balance(rq, rf); /* @@ -6796,12 +6802,15 @@ done: __maybe_unused; void put_prev_entity_fair(struct rq *rq, struct sched_entity *se) { + struct rq_owner_flags orf; struct cfs_rq *cfs_rq; - for_each_sched_entity(se) { + rq_lock_owned(rq, &orf); + for_each_owned_sched_entity(se) { cfs_rq = cfs_rq_of(se); put_prev_entity(cfs_rq, se); } + rq_unlock_owned(rq, &orf); } /* @@ -9712,11 +9721,14 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued) { struct cfs_rq *cfs_rq; struct sched_entity *se = &curr->se; + struct rq_owner_flags orf; - for_each_sched_entity(se) { + rq_lock_owned(rq, &orf); + for_each_owned_sched_entity(se) { cfs_rq = cfs_rq_of(se); entity_tick(cfs_rq, se, queued); } + rq_unlock_owned(rq, &orf); if (static_branch_unlikely(&sched_numa_balancing)) task_tick_numa(rq, curr); @@ -9906,13 +9918,19 @@ static void switched_to_fair(struct rq *rq, struct task_struct *p) void set_curr_entity_fair(struct rq *rq, struct sched_entity *se) { - for_each_sched_entity(se) { + struct rq_owner_flags orf; + + rq_lock_owned(rq, &orf); + + for_each_owned_sched_entity(se) { struct cfs_rq *cfs_rq = cfs_rq_of(se); set_next_entity(cfs_rq, se); /* ensure bandwidth has been allocated on our new cfs_rq */ account_cfs_rq_runtime(cfs_rq, 0); } + + rq_unlock_owned(rq, &orf); } /* -- 2.9.3.1.gcba166c.dirty