Received-SPF: pass (google.com: domain of linux-kernel+bounces-178929-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Wed, 15 May 2024 00:15:08 +0800
From: Chen Yu <yu.c.chen@intel.com>
To: K Prateek Nayak <kprateek.nayak@amd.com>
CC: Peter Zijlstra <peterz@infradead.org>, <mingo@redhat.com>,
	<juri.lelli@redhat.com>, <vincent.guittot@linaro.org>,
	<dietmar.eggemann@arm.com>, <rostedt@goodmis.org>, <bsegall@google.com>,
	<mgorman@suse.de>, <bristot@redhat.com>, <vschneid@redhat.com>,
	<linux-kernel@vger.kernel.org>, <wuyun.abel@bytedance.com>,
	<tglx@linutronix.de>, <efault@gmx.de>, <tim.c.chen@intel.com>,
	<yu.c.chen.y@gmail.com>
Subject: Re: [RFC][PATCH 10/10] sched/eevdf: Use sched_attr::sched_runtime to
 set request/slice suggestion
Message-ID: <ZkOODAALS9HQ3B9A@chenyu5-mobl2>
References: <20240405102754.435410987@infradead.org>
 <20240405110010.934104715@infradead.org>
 <ZjpFruUiBiNi6VSO@chenyu5-mobl2>
 <422fc38c-6096-8804-17ce-1420661743e8@amd.com>
 <ZkMsf4Fz7/AFoQfC@chenyu5-mobl2>
 <f2eafde3-b9e8-afc9-8934-ca8e597c33e6@amd.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <f2eafde3-b9e8-afc9-8934-ca8e597c33e6@amd.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?f2FoBZoRgYh3OqPjVoZ0Co5tFylWGkrwg04X2ZEfRrcwSm318wS7jaTYesJ5?=
 =?us-ascii?Q?Kt8yM4xwLXgdqWgKBl4dOXNjxkoASswDF3RiKydJ/07Xax1np3EOQLj5vYil?=
 =?us-ascii?Q?HxPEPKP9trpjI3ZwR7mVdyeJUSlxhpfPc/MGKiFrtiQag1GqXawUNZdf0V35?=
 =?us-ascii?Q?oGSCvtWWiA2H2GJ/MFT+HZS/uwvdntfoTtYPA6UD+oeNeFkKMWzN0OdvDznJ?=
 =?us-ascii?Q?Pazy6zRYbvXFlGuLez8j8HpH9FfV/XN37dJ7dxmUCsVLT2CI4wzkB9Vi/i6L?=
 =?us-ascii?Q?bQKSvhKoRnSewy3LYB+bU8ywUVWVegikXSAqUjrSPyWLToozi4XMBREDOdnE?=
 =?us-ascii?Q?d+jtf8Ifj6ctxXR63DJPybQ4zUtfTZoWeA0RiAnhxCF18Q7YzdutEXdau+MB?=
 =?us-ascii?Q?zguWf9Re6jP0w7aIUGJCPk41qdTpCetzMFkU1XScgmxRb/AMptc1iwuXU4//?=
 =?us-ascii?Q?SFrgAXrVmq+x8UqikkNkeNH+zdBGx6viCYL+Zx+14aIUm/JEQtD/I1W2IhTv?=
 =?us-ascii?Q?2XLywcw3kr8TBMqrSLYb8HZR2UfcFJ4S8BhAn+lext++gfnxni4DO5qx8d4G?=
 =?us-ascii?Q?QKLb0LFncA2LYdwxY+1jJxOCUjJy1EBMZIC3LLfqGYuSC3CeD0K8YhmDsMG7?=
 =?us-ascii?Q?gRaIc9tPfNQaS1+UEA6jNap28ovWpYHJQNRhh9glhX3OOc8625CZHNZiHzzD?=
 =?us-ascii?Q?+95waWX8YzCxkPA9cD1wR06JohW4EoF3gF6MLIKgrJ1ertMkRxP5aGvIIV+z?=
 =?us-ascii?Q?LOl+06LneZsYAb7hBQ2YfK3ClaQw/pkBNgD9/BYGJ/JXsrGItVZ1UBhoHW8P?=
 =?us-ascii?Q?IP0tM60NCWMgw2vpuAeSSHfUouRjPvAtMm1DCLidaJCttRlA1mAcuOGbzmmj?=
 =?us-ascii?Q?hcOfnnX7kuKkrtjLI0UHwdNkXIncRPVmkKl1TN5oR/Alau5lDUG5T62UQAWa?=
 =?us-ascii?Q?AFkOzv6zz/3Ubg6fQNbCRde+PO1iYKiqCH3be+mPhX3P0OuVeu5cmlyTR078?=
 =?us-ascii?Q?JWaLjbYrHc7kQOMSh+is2k1500uMkIzkUVG6wNZ30VUMwjtjUEZrnHUzPopx?=
 =?us-ascii?Q?uGks8K1RgACtJUN30S/059h0Puiax1G7WqXfDIQ9fHJrBzhmMeghuAd6dDtT?=
 =?us-ascii?Q?JCIQyqolDcRoJGe2p9jxadyFufcLkV6iYV9BffMMXSwcVq49snVFXmTgXDGN?=
 =?us-ascii?Q?87SiL69hKOjf1Oo+gQb3OqY5SSSSjzXhQ+mp4EFBLz4aFmj94ctR+EpitoJt?=
 =?us-ascii?Q?h2ESKz5Kb8XapFmSrxeTpMrhb/3OvvdXc+NYo/itzZST8zLXCg+9VXRdMLX0?=
 =?us-ascii?Q?WOuYR2Gu7Ppae6GkmwyUa+jaEPeHTtUEjQWYJnqJU9rh+GOD2dyTCxduok1J?=
 =?us-ascii?Q?CmuVEL/OXL9kqSxQf/yyCvcEJ5sJ6j3ndeJJm4P3oUra7/e/W3ldBJzze5Y2?=
 =?us-ascii?Q?0qqFTHzh5LGATs/4/Ui32fr5hYP70KJgDLwStdfN6FRHo/AY/iC1ql3LK/nr?=
 =?us-ascii?Q?ohHBdAe4zXjPziyxXfBbcJxAlkWmfSj8UqFHEcIJc0zSPKdaviSndgGt3NCn?=
 =?us-ascii?Q?EDRs3CtIiShTPpUpPBR+ULwjvplFpxPUatNJD70X?=
X-MS-Exchange-CrossTenant-Network-Message-Id: af0533fe-f587-41ad-620d-08dc74311067
X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB6020.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 May 2024 16:15:25.8440
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: m5Y348Zhd+exRLuiJM2DXA/zrDThgaVjrvAMzRyfRAxvZIbaStH+Y8JGQfcG0O22t5ddSvtEq08hR8BSNpfm8g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR11MB4553
X-OriginatorOrg: intel.com

On 2024-05-14 at 20:53:16 +0530, K Prateek Nayak wrote:
> Hello Chenyu,
> 
> On 5/14/2024 2:48 PM, Chen Yu wrote:
> >>> [..snip..]
> >>>  /*
> >>>   * Scan the LLC domain for idle CPUs; this is dynamically regulated by
> >>>   * comparing the average scan cost (tracked in sd->avg_scan_cost) against the
> >>> @@ -7384,10 +7402,9 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
> >>>  	if (sched_feat(SIS_UTIL)) {
> >>>  		sd_share = rcu_dereference(per_cpu(sd_llc_shared, target));
> >>>  		if (sd_share) {
> >>> -			/* because !--nr is the condition to stop scan */> -			nr = READ_ONCE(sd_share->nr_idle_scan) + 1;
> >>> +			nr = adjust_idle_scan(p, READ_ONCE(sd_share->nr_idle_scan));
> >>>  			/* overloaded LLC is unlikely to have idle cpu/core */
> >>> -			if (nr == 1)
> >>> +			if (nr <= 0)
> >>
> >> I was wondering if this would preserve the current behavior with
> >> SIS_FAST toggled off? Since the implementation below still does a
> >> "--nr <= 0" , wouldn't it effectively visit one CPU less overall now?
> >>
> >> Have you tried something similar to the below hunk?
> >>
> >> 	/* because !--nr is the condition to stop scan */
> >> 	nr = adjust_idle_scan(p, READ_ONCE(sd_share->nr_idle_scan)) + 1;
> >> 	if (nr == 1)
> >> 		return -1;
> >>
> > 
> > Yeah, right, to keep the scan depth consistent, the "+1" should be kept.
> >  
> >> I agree with Mike that looking at slice to limit scan-depth seems odd.
> >> My experience with netperf is that the workload cares more about the
> >> server-client being co-located on the closest cache domain and by
> >> limiting scan-depth using slice, this is indirectly achieved since all
> >> the wakeups carry the WF_SYNc flag.
> >>
> > 
> > Exactly. This is the original motivation.
> >  
> >> P.S. have you tried using the slice in __select_idle_cpu()? Similar to
> >> sched_idle_cpu() check, perhaps an additional sched_preempt_short_cpu()
> >> which compares rq->curr->se.slice with the waking task's slice and
> >> returs that cpu if SIS_SHORT can help run the workload quicker?
> > 
> > This is a good idea, it seems to be benefit PREEMPT_SHORT. If the customized
> > task slice is introduced, we can leverage this hint for latency related
> > optimization. Task wakeup is one thing, I can also think of other aspects,
> > like idle load balance, etc. I'm not sure what is the proper usage of the
> > task slice though, this is why I sent this RFC.
> > 
> >> Note:
> >> This will not work if the SIS scan itself is the largest overhead in the
> >> wakeup cycle and not the task placement itself. Previously during
> >> SIS_UTIL testing, to measure the overheads of scan vs placement, we
> >> would do a full scan but return the result that SIS_UTIL would have
> >> returned to determine the overhead of the search itself.
> >>
> > 
> > Regarding the task placement, do you mean the time between a task is enqueued
> > and picked up? Do you have any recommendation which workload can expose the
> > scan overhead most?
> 
> Sorry for not being clear here. From what I've observed in the past,
> there are two dimensions to slect_idle_sibling():
> 
> i)  Placement: Final CPU select_idle_sibling() returns
> ii) Search: Do we find an idle core/CPU in select_idle_sibling()
>

I see.
 
> In case of netperf, I've observed that i) is more important than ii)
> wherin a placement of client on same core/thread as that of the server
> results in better performance vs finding an idle CPU on a remote LLC.

How about placement of client on same core/thread vs finding an idle CPU
on a local LLC?

> For hackbench/tbench, when runqueues are under high utilization (~75%),
> reduction in search time ii) seems to be more beneficial.
>

I can understand that hackbench is idle-cpu sensitive because it is
MxN wakeup relationship and could result in task stacking easily. While
for tbench, it should be similar to netperf, not sure why it does not
fall into i) > ii) case. Is it because you were testing netperf TCP_RR(full-duplex)
and tbench is half-duplex(the former has stronger cache locality requirement)?

> There was also a wakeup from IPI / without IPI angle that I never quite
> got to the bottom of that Mathieu has highlighted last year. I'll go
> get some more data on that front and give your patch a try. Expect
> results in a couple of days. 

Sure, I'd be glad to try your patch.

thanks,
Chenyu