Received-SPF: pass (google.com: domain of linux-kernel+bounces-171676-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Tue, 7 May 2024 23:15:58 +0800
From: Chen Yu <yu.c.chen@intel.com>
To: Peter Zijlstra <peterz@infradead.org>
CC: <mingo@redhat.com>, <juri.lelli@redhat.com>, <vincent.guittot@linaro.org>,
	<dietmar.eggemann@arm.com>, <rostedt@goodmis.org>, <bsegall@google.com>,
	<mgorman@suse.de>, <bristot@redhat.com>, <vschneid@redhat.com>,
	<linux-kernel@vger.kernel.org>, <kprateek.nayak@amd.com>,
	<wuyun.abel@bytedance.com>, <tglx@linutronix.de>, <efault@gmx.de>,
	<tim.c.chen@intel.com>, <yu.c.chen.y@gmail.com>
Subject: Re: [RFC][PATCH 10/10] sched/eevdf: Use sched_attr::sched_runtime to
 set request/slice suggestion
Message-ID: <ZjpFruUiBiNi6VSO@chenyu5-mobl2>
References: <20240405102754.435410987@infradead.org>
 <20240405110010.934104715@infradead.org>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20240405110010.934104715@infradead.org>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Ge6C2brLjq3J1nkCvVh7WBIniVjSC6YPXlt8hLpBCZ+5vGg9Agky0VDcbUMV?=
 =?us-ascii?Q?LSyzI/ivE7brgc2ZuNTDB4nbxqU7gEbhbcqc4JF5kxMNQa7doWZ5wAJc+eF/?=
 =?us-ascii?Q?Mgjn+0e5EjbupRb8piVuNQKOA+ZXgfHzLiOw2NIt6UatRzinyXhmdfNcjwnr?=
 =?us-ascii?Q?4bITKIkiuU5HXJoyyxoLSM8xc2bU4AR2lrbEqwc+o2KlkUHLM+OnctXaCbyX?=
 =?us-ascii?Q?h+KD9mqWooEigN6DzmyrkyMe56FbWIsFjtqMLWawAedWiWfntaZ3KkM7w03H?=
 =?us-ascii?Q?8r1D2yk5CPcpzulSI2k8cKnySrDkRqBHEPmRx4amkxcNqySVyQh577jFf1yK?=
 =?us-ascii?Q?6Lww4NuGd625N+FV8pP6wVrM5JoE4hlrTJtqMRtXC+7VIvdix7aFGSg+QniW?=
 =?us-ascii?Q?hvbgrW5UT5btmlF8r9okvNOmRs2tMtkzF+6aMhXg6X4jCUTlSO7JJXe5G/eJ?=
 =?us-ascii?Q?ti1YI0yxzGhopd9GuNDMhBWqnX2LClI8bZ/cK+AlKtn9louU8rg+T2E7aBJc?=
 =?us-ascii?Q?YpW5A6gARDEKho8JLk2CPYb5uhi6lveuMu3r/cRRZc0Z4etBAcKNRALL3IEe?=
 =?us-ascii?Q?B8IdwZKcOS8uYso7Yh1OFupfz3UuyrTJfht1PajkBqzIKmfUSn/jcrSuino0?=
 =?us-ascii?Q?NE6yRtaYF2FxQIWoaqD60U+FTz82xbb2iMeuiNYyq1+WE6AWQwwnz8oABLgI?=
 =?us-ascii?Q?bvLcAU8UNIxp5I+4MdehKmgU8afK73YxQUGT2iqopzJt50WRFlORkx8JNAqL?=
 =?us-ascii?Q?YmbfWQy5dZgMj27hwHD8nQJ8nnXf4gWn0+tDrv4FR2ycbI5a4yP7GoujwpUG?=
 =?us-ascii?Q?XPAOhYmnH166D0pJqkYUWeP5waxQBjuiRjvPKLhJpfSMkVfZIiSq4IxIU9ZT?=
 =?us-ascii?Q?pibrWeraJRnAPoaupWx9TFmO8CZbn7HX5P1CQW9JvQyV3q6ddjaAX/nxW8ZK?=
 =?us-ascii?Q?9LfQfe2uRY3MqWqFPAXrd0dVxy0pvxAjxdcjHlWBhjbS8eDVLB3nqYFrk2un?=
 =?us-ascii?Q?zT9WdWNj3PWT87lXvkBYyXwKrP8CfuQlR1wgJgl7Q0ZtLyRzD3YBqaYHoMVT?=
 =?us-ascii?Q?f4DASBKLHhMsEsmyYCJSZP5TSwpsdxyGR1Tt0UL9kt4vbf72e/iKJQPNbT8o?=
 =?us-ascii?Q?23uuUCkmW7rIhNsmO9P16WRzR5YG5CwITrDLABu2+7t89Fkn7UeW6R4fSqJR?=
 =?us-ascii?Q?ovqUV8qcFE6qnRC3lJxAs5tfHLn30C/zUPJJ+qg1e9VLVLYOBBb43f/Eejeo?=
 =?us-ascii?Q?wqnL5+RfMApBD6vkeM6UNUwC2OqT/tIbOBAu9oQTon2T/qd01wcJA2FpILu6?=
 =?us-ascii?Q?Uc4Ng3dJ4mtL47iyVuki5gTZlDNCzUm+cDvHelfCRw/MfwOCl5wEg5iNe2Pv?=
 =?us-ascii?Q?zwePhmNd4b72smSD6sGX7M3f9BSDPzUcbIMbo14ES+9xPWCPeMMKdT2idM/X?=
 =?us-ascii?Q?L8NA0Op+3XZDxPbnNaTeiVUirRUA4cx9ChyNuaabwXpNsB482sWSmFBCcLrQ?=
 =?us-ascii?Q?jkROjSJ4j2aB4FeX6zh3Ub0BDCzYzzcagO3Q7f2Y9NbjBmlMddGtCqj3ZvSJ?=
 =?us-ascii?Q?PWz/CgC0E+cNTzml650yRCJn49wUIG9uLLsYWC+l?=
X-MS-Exchange-CrossTenant-Network-Message-Id: b3bed1dc-3322-4dad-a569-08dc6ea8a519
X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB6020.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 May 2024 15:16:18.5324
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 7WZKKcPnRFOJT3cFN6Eh9nGerqBeUfZxYFMRyPB1qYy3VidDRlfV/rhMvsW+86hkZfWQHc3XgR4IFEMO/ZrYcg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR11MB7402
X-OriginatorOrg: intel.com

On 2024-04-05 at 12:28:04 +0200, Peter Zijlstra wrote:
> Allow applications to directly set a suggested request/slice length using
> sched_attr::sched_runtime.
> 
> The implementation clamps the value to: 0.1[ms] <= slice <= 100[ms]
> which is 1/10 the size of HZ=1000 and 10 times the size of HZ=100.
> 
> Applications should strive to use their periodic runtime at a high
> confidence interval (95%+) as the target slice. Using a smaller slice
> will introduce undue preemptions, while using a larger value will
> increase latency.
> 
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
>

Is it possible to leverage this task slice to do better task wakeup placement?
The idea is that, the smaller the slice the wakee has, the less idle CPU it
should scan. This can reduce wake latency and inhibit costly task migration,
especially on large systems.

We did some experiments and got some performance improvements:


From 9cb806476586d7048fcbd0f66d0101f0dbb8fd2b Mon Sep 17 00:00:00 2001
From: Chen Yu <yu.c.chen@intel.com>
Date: Tue, 7 May 2024 22:36:29 +0800
Subject: [RFC PATCH] sched/eevdf: Use customized slice to reduce wakeup latency
 and inhibit task migration

Problem 1:
The overhead of task migration is high on many-core system. The overhead
brings performance penalty due to broken cache locality/higher cache-to-cache
latency.

Problem 2:
During wakeup, the time spent on searching for an idle CPU is costly on
many-core system. Besides, access to other CPU's rq statistics brings
cace contention:

available_idle_cpu(cpu) -> idle_cpu(cpu) -> {rq->curr, rq->nr_running}

Although SIS_UTIL throttles the scan depth based on system utilization,
there is requirement to further limit the scan depth for specific workload,
especially for short duration wakee.

Now we have the interface to customize the request/slice. The smaller the
slice is, the earlier the task can be picked up, and the lower wakeup latency
the task expects. Leverage the wakee's slice to further throttle the
idle CPU scan depth - the shorter slice, the less CPUs to scan.

Test on 240 CPUs, 2 sockets system. With SNC(sub-numa-cluster) enabled,
each LLC domain has 60 CPUs. There is noticeable improvement of netperf.
(With SNC disabled, more improvements should be seen because C2C is higher)

The global slice is 3 msec(sysctl_sched_base_slice) by default on my ubuntu
22.04, and the customized slice is set to 0.1 msec for both netperf and netserver:

for i in $(seq 1 $job); do
	netperf_slice -e 100000 -4 -H 127.0.01 -t TCP_RR -c -C -l 100 &
done

case            	load    	baseline(std%)	compare%( std%)
TCP_RR          	60-threads	 1.00 (  1.60)	 +0.35 (  1.73)
TCP_RR          	120-threads	 1.00 (  1.34)	 -0.96 (  1.24)
TCP_RR          	180-threads	 1.00 (  1.59)	+92.20 (  4.24)
TCP_RR          	240-threads	 1.00 (  9.71)	+43.11 (  2.97)

Suggested-by: Tim Chen <tim.c.chen@intel.com>
Signed-off-by: Chen Yu <yu.c.chen@intel.com>
---
 kernel/sched/fair.c     | 23 ++++++++++++++++++++---
 kernel/sched/features.h |  1 +
 2 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index edc23f6588a3..f269ae7d6e24 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7368,6 +7368,24 @@ static inline int select_idle_smt(struct task_struct *p, struct sched_domain *sd
 
 #endif /* CONFIG_SCHED_SMT */
 
+/*
+ * Scale the scan number of idle CPUs according to customized
+ * wakee's slice. The smaller the slice is, the earlier the task
+ * wants be picked up, thus the lower wakeup latency the task expects.
+ * The baseline is the global sysctl_sched_base_slice. Task slice
+ * smaller than the global one would shrink the scan number.
+ */
+static int adjust_idle_scan(struct task_struct *p, int nr)
+{
+	if (!sched_feat(SIS_FAST))
+		return nr;
+
+	if (!p->se.custom_slice || p->se.slice >= sysctl_sched_base_slice)
+		return nr;
+
+	return div_u64(nr * p->se.slice, sysctl_sched_base_slice);
+}
+
 /*
  * Scan the LLC domain for idle CPUs; this is dynamically regulated by
  * comparing the average scan cost (tracked in sd->avg_scan_cost) against the
@@ -7384,10 +7402,9 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
 	if (sched_feat(SIS_UTIL)) {
 		sd_share = rcu_dereference(per_cpu(sd_llc_shared, target));
 		if (sd_share) {
-			/* because !--nr is the condition to stop scan */
-			nr = READ_ONCE(sd_share->nr_idle_scan) + 1;
+			nr = adjust_idle_scan(p, READ_ONCE(sd_share->nr_idle_scan));
 			/* overloaded LLC is unlikely to have idle cpu/core */
-			if (nr == 1)
+			if (nr <= 0)
 				return -1;
 		}
 	}
diff --git a/kernel/sched/features.h b/kernel/sched/features.h
index 143f55df890b..176324236018 100644
--- a/kernel/sched/features.h
+++ b/kernel/sched/features.h
@@ -50,6 +50,7 @@ SCHED_FEAT(TTWU_QUEUE, true)
  * When doing wakeups, attempt to limit superfluous scans of the LLC domain.
  */
 SCHED_FEAT(SIS_UTIL, true)
+SCHED_FEAT(SIS_FAST, true)
 
 /*
  * Issue a WARN when we do multiple update_rq_clock() calls
-- 
2.25.1