Received: by 2002:ac0:a5a6:0:0:0:0:0 with SMTP id m35-v6csp2332292imm; Fri, 7 Sep 2018 14:50:03 -0700 (PDT) X-Google-Smtp-Source: ANB0Vda2Oxv0skEG80DlS0OcFOWyi1Wxlsdoe9cAw4ml6rENVUmwFeqK07kf0hjrGkVRvGOORcod X-Received: by 2002:a63:b705:: with SMTP id t5-v6mr9798833pgf.366.1536357002992; Fri, 07 Sep 2018 14:50:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536357002; cv=none; d=google.com; s=arc-20160816; b=gUbikkmsZplXhCOM4Wldyz2xnsqq4XnpmSo8yAWbNUEjofX/kcEfMR7stEERZ/8745 48ahak6FNsTXEFyU/UFqj/54FLcV1kXJJcgTpO0uzrNSf7dtJiLzcp739DrEQo1KHrzY dx29hO9dWDhM+BO9DOlu/L82iUvvUe7EpdQPddsczkU8Oy8/V4rUKnovtpk6w0Ctigqc qDvLMWLFMpj35dCvnHu1sFgvjEx9UG/qvJGd+YIJ68wwoiFFLiv3tb+OF2H9x+QRHHoX o/hHQ1qKec+iGRZIdgCikiaczOV1yssWiL2WsTQpzyEis1EMUzr4s+I0XwpVs9RgayzP poOg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=qR7M1zUzYR6TOiubja8Qz/e3sN07CZZ/lciZoJ2uZzw=; b=sc9TD7fuDL9vRVjwwWGQM/kmqNETH7/fcmtQesHWaf2l7ib/OZeKFblCAS9ln5Od4Q kOGPBTBbNuC4RE15DP7ZiV9P2Xw/hFeSC3PvL6I38gdMVneLxPqHLeI+OH3BqeEflzNs 3o/3FOxHkZayMI4dDbfq1gXsLxROhP/LYZ6u4wTtf4p0QJla+zSZXWQ3c/crhlo6LLfO Hbo/skFWLxcBX/fxZe7mPKF8NneQMxyuYuz2XZmuem3Zc1k/Pf00xxXDOMfmU+by/Z5u zi908VXg8B/rEYM5o19b+CvdpbT1GEZGLb5O04OAu14EvBYCm2BTzF45HONyhVTo2ZgH QQQw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@amazon.de header.s=amazon201209 header.b=W3GwCJlY; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.de Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id j13-v6si10110406pfj.230.2018.09.07.14.49.47; Fri, 07 Sep 2018 14:50:02 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@amazon.de header.s=amazon201209 header.b=W3GwCJlY; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730848AbeIHCaU (ORCPT + 99 others); Fri, 7 Sep 2018 22:30:20 -0400 Received: from smtp-fw-9101.amazon.com ([207.171.184.25]:14952 "EHLO smtp-fw-9101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726618AbeIHCaT (ORCPT ); Fri, 7 Sep 2018 22:30:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.de; i=@amazon.de; q=dns/txt; s=amazon201209; t=1536356843; x=1567892843; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=qR7M1zUzYR6TOiubja8Qz/e3sN07CZZ/lciZoJ2uZzw=; b=W3GwCJlYsHCJrpiYx6/3uAauEsIR+elk3LpZ2hXbxRQZDBt4plNWobDI 0jiMtNfV7W/5O2Q5Vd0jqccXrWIYOzy8XFiqGxy/hFOELneAI5xX0cv51 DgpDns5e1I3WFofydT3P0/CTWVUWGZfIMjAECA6HPmPuCOCE4V0AnzbY+ 4=; X-IronPort-AV: E=Sophos;i="5.53,343,1531785600"; d="scan'208";a="757370796" Received: from sea3-co-svc-lb6-vlan3.sea.amazon.com (HELO email-inbound-relay-2b-81e76b79.us-west-2.amazon.com) ([10.47.22.38]) by smtp-border-fw-out-9101.sea19.amazon.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 07 Sep 2018 21:44:49 +0000 Received: from u7588a65da6b65f.ant.amazon.com (pdx2-ws-svc-lb17-vlan2.amazon.com [10.247.140.66]) by email-inbound-relay-2b-81e76b79.us-west-2.amazon.com (8.14.7/8.14.7) with ESMTP id w87Lh20e040364 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Fri, 7 Sep 2018 21:43:04 GMT Received: from u7588a65da6b65f.ant.amazon.com (localhost [127.0.0.1]) by u7588a65da6b65f.ant.amazon.com (8.15.2/8.15.2/Debian-3) with ESMTPS id w87Lh0uO027846 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 7 Sep 2018 23:43:01 +0200 Received: (from jschoenh@localhost) by u7588a65da6b65f.ant.amazon.com (8.15.2/8.15.2/Submit) id w87Lh0uh027845; Fri, 7 Sep 2018 23:43:00 +0200 From: =?UTF-8?q?Jan=20H=2E=20Sch=C3=B6nherr?= To: Ingo Molnar , Peter Zijlstra Cc: =?UTF-8?q?Jan=20H=2E=20Sch=C3=B6nherr?= , linux-kernel@vger.kernel.org Subject: [RFC 56/60] cosched: Adjust wakeup preemption rules for coscheduling Date: Fri, 7 Sep 2018 23:40:43 +0200 Message-Id: <20180907214047.26914-57-jschoenh@amazon.de> X-Mailer: git-send-email 2.9.3.1.gcba166c.dirty In-Reply-To: <20180907214047.26914-1-jschoenh@amazon.de> References: <20180907214047.26914-1-jschoenh@amazon.de> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Modify check_preempt_wakeup() to work correctly with coscheduled sets. On the one hand, that means not blindly preempting, when the woken task potentially belongs to a different set and we're not allowed to switch sets. Instead we have to notify the correct leader to follow up. On the other hand, we need to handle additional idle cases, as CPUs are now idle *within* certain coscheduled sets and woken tasks may not preempt the idle task blindly anymore. Signed-off-by: Jan H. Schönherr --- kernel/sched/fair.c | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 83 insertions(+), 2 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 07fd9dd5561d..0c1d9334ea8e 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -6882,6 +6882,9 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ int next_buddy_marked = 0; struct cfs_rq *cfs_rq; int scale; +#ifdef CONFIG_COSCHEDULING + struct rq_flags rf; +#endif /* FIXME: locking may be off after fetching the idle_se */ if (cosched_is_idle(rq, curr)) @@ -6908,6 +6911,13 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ } /* + * FIXME: Check whether this can be re-enabled with coscheduling + * + * We might want to send a reschedule IPI to the leader, which is only + * checked further below. + */ +#ifndef CONFIG_COSCHEDULING + /* * We can come here with TIF_NEED_RESCHED already set from new task * wake up path. * @@ -6919,11 +6929,22 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ */ if (test_tsk_need_resched(curr)) return; +#endif + /* + * FIXME: Check whether this can be re-enabled with coscheduling + * + * curr and p may belong could belong to different coscheduled sets, + * in which case the decision is not straight-forward. Additionally, + * the preempt code needs to know the CPU it has to send an IPI + * to. This is not yet known here. + */ +#ifndef CONFIG_COSCHEDULING /* Idle tasks are by definition preempted by non-idle tasks. */ if (unlikely(curr->policy == SCHED_IDLE) && likely(p->policy != SCHED_IDLE)) goto preempt; +#endif /* * Batch and idle tasks do not preempt non-idle tasks (their preemption @@ -6932,7 +6953,55 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ if (unlikely(p->policy != SCHED_NORMAL) || !sched_feat(WAKEUP_PREEMPTION)) return; + /* + * FIXME: find_matching_se() might end up at SEs where a different CPU + * is leader. While we do get locks *afterwards* the question is + * whether anything bad can happen due to the lock-free traversal. + */ find_matching_se(&se, &pse); + +#ifdef CONFIG_COSCHEDULING + if (se == pse) { + /* + * There is nothing to do on this CPU within the current + * coscheduled set and the newly woken task belongs to this + * coscheduled set. Hence, it is a welcome distraction. + * + * [find_matching_se() walks up the hierarchy for se and pse + * until they are within the same CFS runqueue. As equality + * was eliminated at the beginning, equality now means that + * se was rq->idle_se from the start and pse approached it + * from within a child runqueue.] + */ + SCHED_WARN_ON(!cosched_is_idle(rq, curr)); + SCHED_WARN_ON(cosched_get_idle_se(rq) != se); + goto preempt; + } + + if (hrq_of(cfs_rq_of(se))->sdrq_data.level) { + rq_lock(hrq_of(cfs_rq_of(se)), &rf); + update_rq_clock(hrq_of(cfs_rq_of(se))); + } + + if (!cfs_rq_of(se)->curr) { + /* + * There is nothing to do at a higher level within the current + * coscheduled set and the newly woken task belongs to a + * different coscheduled set. Hence, it is a welcome + * distraction for the leader of that higher level. + * + * [If a leader does not find a SE in its top_cfs_rq, it does + * not record anything as current. Still, it tells its + * children within which coscheduled set they are idle. + * find_matching_se() now ended at such an idle leader. As + * we checked for se==pse earlier, we cannot be this leader.] + */ + SCHED_WARN_ON(leader_of(se) == cpu_of(rq)); + resched_cpu_locked(leader_of(se)); + goto out; + } +#endif + update_curr(cfs_rq_of(se)); BUG_ON(!pse); if (wakeup_preempt_entity(se, pse) == 1) { @@ -6942,18 +7011,30 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ */ if (!next_buddy_marked) set_next_buddy(pse); +#ifdef CONFIG_COSCHEDULING + if (leader_of(se) != cpu_of(rq)) { + resched_cpu_locked(leader_of(se)); + goto out; + } + if (hrq_of(cfs_rq_of(se))->sdrq_data.level) + rq_unlock(hrq_of(cfs_rq_of(se)), &rf); +#endif goto preempt; } +#ifdef CONFIG_COSCHEDULING +out: + if (hrq_of(cfs_rq_of(se))->sdrq_data.level) + rq_unlock(hrq_of(cfs_rq_of(se)), &rf); +#endif return; - preempt: resched_curr(rq); /* * Only set the backward buddy when the current task is still * on the rq. This can happen when a wakeup gets interleaved * with schedule on the ->pre_schedule() or idle_balance() - * point, either of which can * drop the rq lock. + * point, either of which can drop the rq lock. * * Also, during early boot the idle thread is in the fair class, * for obvious reasons its a bad idea to schedule back to it. -- 2.9.3.1.gcba166c.dirty