Received: by 2002:a05:6a10:f347:0:0:0:0 with SMTP id d7csp129370pxu; Wed, 2 Dec 2020 17:10:38 -0800 (PST) X-Google-Smtp-Source: ABdhPJyraXAy2Tzt0JUvzhJK3N1qSBeRu3qExHk/gTLLrRFkHbvDqZa9RPWtbxxuKC8cd8YTZWvQ X-Received: by 2002:a17:906:31c2:: with SMTP id f2mr459550ejf.518.1606957838308; Wed, 02 Dec 2020 17:10:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1606957838; cv=none; d=google.com; s=arc-20160816; b=N3vAzp2y8v8I7GSUvtIzlExF80f2UNwfBAsCtaVDhWuP/1Czdkbm5oT3YIpdzQAvmt AGQcbgLNA/c1vcqiO+xEyHV3vZMQwnVGH+uTnR6g9cruP0UHpukElueAUUrA95Dmy9mm +unv7qCaGbdb2ouO5bORKMLW6We6BTO3YdsEfPkQL+uBKdL9gN4Ir5/+bllFf28wjJYY c89PvwIfGODInBZ10D6etKuyv/WW/gQG7ketPc1TGmU/by3DVI2GvJF21nLLhXgBAvj7 uAZ8LCUBNYQoEnsAUeHHQeIzhkNekO9DZhlzVbUdTpAV/Jof6AnE5Fy0dA2DIE0zrx3x HevQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:content-language :in-reply-to:mime-version:user-agent:date:message-id:references:cc :to:from:subject:ironport-sdr:ironport-sdr; bh=bVRi+fLrVyzKJ5bxJbQXxv+H+iYoiNHH3AfISwVVTxA=; b=dv/ZXOt+QDtJuFJsq/xgnm4uDKSDfHSQHmnsZz38nwNQ6yFQq/PN5QGxqTXaXlvH1P BgYy5SFdhFm6zDRULGqGIPolQB5yQEPpwFkp23mPPbSQQ/FrivPcdTPeHWnFirAnSdXY iwaZt1RSLTnbx5uPC9+wv01X9ljATDP+i60JG7IHY/8jeXEx28ZzSfhkUa8wo/xU7/rS IXh0CKC85aXj4UDNx1OYnyq1E/H34jibhjlJNCffjMlT6qtDoEyc/aOJgTQJBS/QPchn 7yuqZ/IMaACthT2BmM6eWvYfhf3+bnz6gLFBWei3wA8nORqbJhlmvDLNXgw94454yktr sC5Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id q16si175790ejd.199.2020.12.02.17.10.15; Wed, 02 Dec 2020 17:10:38 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727384AbgLCBIO (ORCPT + 99 others); Wed, 2 Dec 2020 20:08:14 -0500 Received: from mga11.intel.com ([192.55.52.93]:15522 "EHLO mga11.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726410AbgLCBIN (ORCPT ); Wed, 2 Dec 2020 20:08:13 -0500 IronPort-SDR: RGAfto8t9ThbIGG2cDar3+LuDiuIzSBYbl2GMeOH8n6R1koH9srt9P4jzttgjji3OxI+Addb34 OXhIQo604ISg== X-IronPort-AV: E=McAfee;i="6000,8403,9823"; a="169609202" X-IronPort-AV: E=Sophos;i="5.78,388,1599548400"; d="scan'208";a="169609202" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Dec 2020 17:06:32 -0800 IronPort-SDR: Ki41DJBOlWwddKCWpTTeByb8x3pQKO4RTdiNwCO1SkF4jU+rjlZiMykFf2EVu7Zb9Mbmz/xqaa kqjdsWSr2IIw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.78,388,1599548400"; d="scan'208";a="373711818" Received: from cli6-desk1.ccr.corp.intel.com (HELO [10.239.161.125]) ([10.239.161.125]) by FMSMGA003.fm.intel.com with ESMTP; 02 Dec 2020 17:06:20 -0800 Subject: Re: [PATCH -tip 14/32] sched: migration changes for core scheduling From: "Li, Aubrey" To: Balbir Singh Cc: Peter Zijlstra , "Joel Fernandes (Google)" , Nishanth Aravamudan , Julien Desfossez , Tim Chen , Vineeth Pillai , Aaron Lu , Aubrey Li , tglx@linutronix.de, linux-kernel@vger.kernel.org, mingo@kernel.org, torvalds@linux-foundation.org, fweisbec@gmail.com, keescook@chromium.org, kerrnel@google.com, Phil Auld , Valentin Schneider , Mel Gorman , Pawan Gupta , Paolo Bonzini , vineeth@bitbyteword.org, Chen Yu , Christian Brauner , Agata Gruza , Antonio Gomez Iglesias , graf@amazon.com, konrad.wilk@oracle.com, dfaggioli@suse.com, pjt@google.com, rostedt@goodmis.org, derkling@google.com, benbjiang@tencent.com, Alexandre Chartre , James.Bottomley@hansenpartnership.com, OWeisse@umich.edu, Dhaval Giani , Junaid Shahid , jsbarnes@google.com, chris.hyser@oracle.com, Ben Segall , Josh Don , Hao Luo , Tom Lendacky , Aubrey Li , "Paul E. McKenney" , Tim Chen References: <20201117232003.3580179-1-joel@joelfernandes.org> <20201117232003.3580179-15-joel@joelfernandes.org> <20201122235456.GF110669@balbir-desktop> <0b2514ef-6cc3-c1a3-280b-5d9062c80a31@linux.intel.com> <20201124154237.GZ3021@hirez.programming.kicks-ass.net> <20201125225731.GB163610@balbir-desktop> <20201126083250.GI163610@balbir-desktop> <20201130093333.GD473773@balbir-desktop> <6cf6c89d-8d33-c11b-1ea9-7d143b89fc2d@linux.intel.com> Message-ID: <1bcc55c5-af1a-a7a0-606f-851a7bc80907@linux.intel.com> Date: Thu, 3 Dec 2020 09:06:19 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2020/12/2 22:09, Li, Aubrey wrote: > Hi Balbir, > > I still placed the patch embedded in this thread, welcome any comments. Sorry, this version needs more work, refined as below, and I realized I should place a version number to the patch, start from v2 now. Thanks, -Aubrey ====================================================================== From aff2919889635aa9311d15bac3e949af0300ddc1 Mon Sep 17 00:00:00 2001 From: Aubrey Li Date: Thu, 3 Dec 2020 00:51:18 +0000 Subject: [PATCH v2] sched: migration changes for core scheduling - Don't migrate if there is a cookie mismatch Load balance tries to move task from busiest CPU to the destination CPU. When core scheduling is enabled, if the task's cookie does not match with the destination CPU's core cookie, this task will be skipped by this CPU. This mitigates the forced idle time on the destination CPU. - Select cookie matched idle CPU In the fast path of task wakeup, select the first cookie matched idle CPU instead of the first idle CPU. - Find cookie matched idlest CPU In the slow path of task wakeup, find the idlest CPU whose core cookie matches with task's cookie - Don't migrate task if cookie not match For the NUMA load balance, don't migrate task to the CPU whose core cookie does not match with task's cookie Cc: Balbir Singh Cc: Vincent Guittot Tested-by: Julien Desfossez Signed-off-by: Aubrey Li Signed-off-by: Tim Chen Signed-off-by: Vineeth Remanan Pillai Signed-off-by: Joel Fernandes (Google) --- kernel/sched/fair.c | 33 +++++++++++++++++--- kernel/sched/sched.h | 72 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 101 insertions(+), 4 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index de82f88ba98c..afdfea70c58c 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -1921,6 +1921,13 @@ static void task_numa_find_cpu(struct task_numa_env *env, if (!cpumask_test_cpu(cpu, env->p->cpus_ptr)) continue; + /* + * Skip this cpu if source task's cookie does not match + * with CPU's core cookie. + */ + if (!sched_core_cookie_match(cpu_rq(cpu), env->p)) + continue; + env->dst_cpu = cpu; if (task_numa_compare(env, taskimp, groupimp, maymove)) break; @@ -5867,11 +5874,15 @@ find_idlest_group_cpu(struct sched_group *group, struct task_struct *p, int this /* Traverse only the allowed CPUs */ for_each_cpu_and(i, sched_group_span(group), p->cpus_ptr) { + struct rq *rq = cpu_rq(i); + + if (!sched_core_cookie_match(rq, p)) + continue; + if (sched_idle_cpu(i)) return i; if (available_idle_cpu(i)) { - struct rq *rq = cpu_rq(i); struct cpuidle_state *idle = idle_get_state(rq); if (idle && idle->exit_latency < min_exit_latency) { /* @@ -6129,7 +6140,9 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t for_each_cpu_wrap(cpu, cpus, target) { if (!--nr) return -1; - if (available_idle_cpu(cpu) || sched_idle_cpu(cpu)) + + if ((available_idle_cpu(cpu) || sched_idle_cpu(cpu)) && + sched_cpu_cookie_match(cpu_rq(cpu), p)) break; } @@ -7530,8 +7543,9 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env) * We do not migrate tasks that are: * 1) throttled_lb_pair, or * 2) cannot be migrated to this CPU due to cpus_ptr, or - * 3) running (obviously), or - * 4) are cache-hot on their current CPU. + * 3) task's cookie does not match with this CPU's core cookie + * 4) running (obviously), or + * 5) are cache-hot on their current CPU. */ if (throttled_lb_pair(task_group(p), env->src_cpu, env->dst_cpu)) return 0; @@ -7566,6 +7580,13 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env) return 0; } + /* + * Don't migrate task if the task's cookie does not match + * with the destination CPU's core cookie. + */ + if (!sched_core_cookie_match(cpu_rq(env->dst_cpu), p)) + return 0; + /* Record that we found atleast one task that could run on dst_cpu */ env->flags &= ~LBF_ALL_PINNED; @@ -8792,6 +8813,10 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu) p->cpus_ptr)) continue; + /* Skip over this group if no cookie matched */ + if (!sched_group_cookie_match(cpu_rq(this_cpu), p, group)) + continue; + local_group = cpumask_test_cpu(this_cpu, sched_group_span(group)); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index e72942a9ee11..82917ce183b4 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1119,6 +1119,7 @@ static inline bool is_migration_disabled(struct task_struct *p) #ifdef CONFIG_SCHED_CORE DECLARE_STATIC_KEY_FALSE(__sched_core_enabled); +static inline struct cpumask *sched_group_span(struct sched_group *sg); static inline bool sched_core_enabled(struct rq *rq) { @@ -1135,6 +1136,61 @@ static inline raw_spinlock_t *rq_lockp(struct rq *rq) bool cfs_prio_less(struct task_struct *a, struct task_struct *b); +/* + * Helpers to check if the CPU's core cookie matches with the task's cookie + * when core scheduling is enabled. + * A special case is that the task's cookie always matches with CPU's core + * cookie if the CPU is in an idle core. + */ +static inline bool sched_cpu_cookie_match(struct rq *rq, struct task_struct *p) +{ + /* Ignore cookie match if core scheduler is not enabled on the CPU. */ + if (!sched_core_enabled(rq)) + return true; + + return rq->core->core_cookie == p->core_cookie; +} + +static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p) +{ + bool idle_core = true; + int cpu; + + /* Ignore cookie match if core scheduler is not enabled on the CPU. */ + if (!sched_core_enabled(rq)) + return true; + + for_each_cpu(cpu, cpu_smt_mask(cpu_of(rq))) { + if (!available_idle_cpu(cpu)) { + idle_core = false; + break; + } + } + + /* + * A CPU in an idle core is always the best choice for tasks with + * cookies. + */ + return idle_core || rq->core->core_cookie == p->core_cookie; +} + +static inline bool sched_group_cookie_match(struct rq *rq, + struct task_struct *p, + struct sched_group *group) +{ + int cpu; + + /* Ignore cookie match if core scheduler is not enabled on the CPU. */ + if (!sched_core_enabled(rq)) + return true; + + for_each_cpu_and(cpu, sched_group_span(group), p->cpus_ptr) { + if (sched_core_cookie_match(rq, p)) + return true; + } + return false; +} + extern void queue_core_balance(struct rq *rq); #else /* !CONFIG_SCHED_CORE */ @@ -1153,6 +1209,22 @@ static inline void queue_core_balance(struct rq *rq) { } +static inline bool sched_cpu_cookie_match(struct rq *rq, struct task_struct *p) +{ + return true; +} + +static inline bool sched_core_cookie_match(struct rq *rq, struct task_struct *p) +{ + return true; +} + +static inline bool sched_group_cookie_match(struct rq *rq, + struct task_struct *p, + struct sched_group *group) +{ + return true; +} #endif /* CONFIG_SCHED_CORE */ #ifdef CONFIG_SCHED_SMT -- 2.17.1