Received-SPF: pass (google.com: domain of linux-kernel+bounces-178851-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <f2eafde3-b9e8-afc9-8934-ca8e597c33e6@amd.com>
Date: Tue, 14 May 2024 20:53:16 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.13.1
Subject: Re: [RFC][PATCH 10/10] sched/eevdf: Use sched_attr::sched_runtime to
 set request/slice suggestion
Content-Language: en-US
To: Chen Yu <yu.c.chen@intel.com>
Cc: Peter Zijlstra <peterz@infradead.org>, mingo@redhat.com,
 juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com,
 rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de,
 bristot@redhat.com, vschneid@redhat.com, linux-kernel@vger.kernel.org,
 wuyun.abel@bytedance.com, tglx@linutronix.de, efault@gmx.de,
 tim.c.chen@intel.com, yu.c.chen.y@gmail.com
References: <20240405102754.435410987@infradead.org>
 <20240405110010.934104715@infradead.org> <ZjpFruUiBiNi6VSO@chenyu5-mobl2>
 <422fc38c-6096-8804-17ce-1420661743e8@amd.com>
 <ZkMsf4Fz7/AFoQfC@chenyu5-mobl2>
From: K Prateek Nayak <kprateek.nayak@amd.com>
In-Reply-To: <ZkMsf4Fz7/AFoQfC@chenyu5-mobl2>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?MStocHFDd1BWMlcydzdIckFRRXRkUDJ6VThDS2RVV1FvbFJhMkNJMmZKWW1y?=
 =?utf-8?B?blRUcFhqSCtiS3RVbWNDeDdWenUyOGhaamZOK2J2WWtlQStNbkt4V1Q5cEc2?=
 =?utf-8?B?dUtzNUdjVnBTSC9uNEtKNSt1OVovWE90VmpNUytpRE9RL1ljaHRuME1CSzR2?=
 =?utf-8?B?Tys4cURFZ3Mxc3Zpbmt6ZjlVaGVJMElhTXFzSnVHZytkbUgzdTVQQytZSG5M?=
 =?utf-8?B?MERiZGNBZndRY1pqb1hJcTdEOXd4SkdyVVphMHFKSWovakFIYjJjVDl0Vmc4?=
 =?utf-8?B?c1Zzc0pGcytzdlpvOW1KS1gzdGdyQzdHUGRIcjdvS0FoY2RQRjd3TjlKM2dl?=
 =?utf-8?B?TkltZ3dWZWlsR1ZkbWtkNlpaYmpaSFc2QzRLM29KQjU0RzJVdUVQeDdwWVZr?=
 =?utf-8?B?SW5wT05Sa0MzdUJaNko0aTNsRGJiZCsrRmlGNTJCWlpmTXJQcWRjb3BHRmtL?=
 =?utf-8?B?aGlhRlMzdVRYdlFYQytPNHE2RnBLZDNpbE5qL01BQzlkdzhtb0RrWVNtQ0pk?=
 =?utf-8?B?TEtsTElVM0wwWGhyTXk2ZWpMWDZ5c2FXWEFGazF4KzdQQldiZ2o3ZUVFakNS?=
 =?utf-8?B?bmZidEVXTkp1WjJHWTNhSVdseXB0clZkR0JheEtiTm1IaFI1VXFleVBja3JY?=
 =?utf-8?B?R1B4MEsyUjF3NVh0U29TWW0zWEw0b2l3cEI4czNPWGpLbWtHZ085RDV3Ykk3?=
 =?utf-8?B?QitDbjRCcHdtVGliL1ltRndiZXdIcEJ5ZU5yQVBNZGk2N1VYOW5OMkNUeFRP?=
 =?utf-8?B?bjVWTnNSRTBlU1JWa3dyakpRYkd4Y2xJcXJKNUh2MWhpMDcyS0lNSlN0aEJi?=
 =?utf-8?B?OUFHSGRhaklleUFBNGlONEhjK1N3c0xrc0IwWFFUelZ3bFFiLzRidlo3bVZX?=
 =?utf-8?B?K2JVdUxheFdNZFBZbmYzM1FRdFIycHB3UjVsbjJhTEdXcUJBby9CdmpPaFVt?=
 =?utf-8?B?U0VjNmlvS254RnhoMGpkaS9yamRibEYzN0IwTU83SHl0anBnd243TnZ1N25m?=
 =?utf-8?B?V1BGclpZLzdZUFpCbjN5QmFEUlNzeDhUc1RFUkVlMTI1UFN2MTd1OGJkbjEy?=
 =?utf-8?B?YVp5NzN3d3RTbzdqWnU2bzQxVkxwMUVZYnN0RmhRdFAxdW94OGlhK1ZqWXdn?=
 =?utf-8?B?MzByMHhNa2QvOTNGaUFEMWZBOFpUK3FGNjJmRitKbDdYMWxURVFJK2Y2VmY5?=
 =?utf-8?B?YjQwL3pvRWw0Uzh1UGRsMlVQNFMyVWE1alhSckc0MmpMWFFuSW5tNlRqWEUw?=
 =?utf-8?B?R3ppUWpKbXJhVERaMERlbWVGejd1bXpEbXRjM1Q3TWdXLytTWS9SenR4UUNB?=
 =?utf-8?B?Smw4UWVkblcxNG1lSVQrVkJralZiQnR2R0dBWEJDVWovUHhVY0hpTk5HNXIw?=
 =?utf-8?B?WHR2dFcxcDQxaFdaMk01dlNXV2pFYXBubnFGTUoyL1lJZHpUa3l3cVNub25D?=
 =?utf-8?B?R243SHY4emhDdnUvTFpPY2lzZ05EODhucnRpZzNkMHYvdVJtNlZVbm4yeHg2?=
 =?utf-8?B?S1VRcEgyaklkcW45K3pCOW96clVpSk9zREVnY3RrSkJoUW5YajlEQjNldmxy?=
 =?utf-8?B?Vm5GU1lFbSt1cUVTQSs1bkh5WFAzanE0Z2l2ajl4Qkt2SzhaUnFQQ0dxMjRm?=
 =?utf-8?B?RlV5dGlwaTB3WWRaZUhVQkQyampET2dkekhhZ2ZLaFJrcVh6OEpYdHhkNldw?=
 =?utf-8?B?M0l6aHlhTmFIR29KWDJWZ1VuZG9CQVlveGNZQVNObWZVTzZ2S2F0QlJFMDBJ?=
 =?utf-8?B?VExrRHJtQktpUk5SQ2U5cWx4dWlwWWdYQzBkRDBGeTNIaTRwcXl3clJheFA0?=
 =?utf-8?B?UDRGM2VpaVI3WHI1SUdrQlNnNGhidVp1NnliRzNOS29sbU9ZMS95eEZNSjdx?=
 =?utf-8?B?a3BWK3RGUVBYWU9mczRYWUF1aUJhVWEwWUI4UUsxUEhMSlVORVNRVGpFSTd6?=
 =?utf-8?B?d3ZaUVczOFZvdEthR1dmNlJBNXR5cFdNcXBYckY1Z1dyd2NVUHBhS2MvVklG?=
 =?utf-8?B?Q1luaWM4THNiem5OdVRzUEVOK0dhdlRGRkdxREpQS1NjekJKZ09aS2ZsR29J?=
 =?utf-8?B?SHVLWGFOc0hJMGZwU25pM0JvREdHOHc0ejhJZUdTS3p6dFVYUm9uYzh6L0dk?=
 =?utf-8?Q?YByoCMmgv6xXSAWcNhH6naB3d?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e3eb30e2-3306-47dc-6df1-08dc7429cc86
X-MS-Exchange-CrossTenant-AuthSource: MW2PR12MB2379.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 May 2024 15:23:25.8548
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: VqRZ5JQW/plXbQND+CDr72kZIioafM/gcCcXsen+o9/vI1U6hDPxNIo05602/coq2nSejFeNTMueqj9wOAVbWQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB4120

Hello Chenyu,

On 5/14/2024 2:48 PM, Chen Yu wrote:
>>> [..snip..]
>>>  /*
>>>   * Scan the LLC domain for idle CPUs; this is dynamically regulated by
>>>   * comparing the average scan cost (tracked in sd->avg_scan_cost) against the
>>> @@ -7384,10 +7402,9 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool
>>>  	if (sched_feat(SIS_UTIL)) {
>>>  		sd_share = rcu_dereference(per_cpu(sd_llc_shared, target));
>>>  		if (sd_share) {
>>> -			/* because !--nr is the condition to stop scan */> -			nr = READ_ONCE(sd_share->nr_idle_scan) + 1;
>>> +			nr = adjust_idle_scan(p, READ_ONCE(sd_share->nr_idle_scan));
>>>  			/* overloaded LLC is unlikely to have idle cpu/core */
>>> -			if (nr == 1)
>>> +			if (nr <= 0)
>>
>> I was wondering if this would preserve the current behavior with
>> SIS_FAST toggled off? Since the implementation below still does a
>> "--nr <= 0" , wouldn't it effectively visit one CPU less overall now?
>>
>> Have you tried something similar to the below hunk?
>>
>> 	/* because !--nr is the condition to stop scan */
>> 	nr = adjust_idle_scan(p, READ_ONCE(sd_share->nr_idle_scan)) + 1;
>> 	if (nr == 1)
>> 		return -1;
>>
> 
> Yeah, right, to keep the scan depth consistent, the "+1" should be kept.
>  
>> I agree with Mike that looking at slice to limit scan-depth seems odd.
>> My experience with netperf is that the workload cares more about the
>> server-client being co-located on the closest cache domain and by
>> limiting scan-depth using slice, this is indirectly achieved since all
>> the wakeups carry the WF_SYNc flag.
>>
> 
> Exactly. This is the original motivation.
>  
>> P.S. have you tried using the slice in __select_idle_cpu()? Similar to
>> sched_idle_cpu() check, perhaps an additional sched_preempt_short_cpu()
>> which compares rq->curr->se.slice with the waking task's slice and
>> returs that cpu if SIS_SHORT can help run the workload quicker?
> 
> This is a good idea, it seems to be benefit PREEMPT_SHORT. If the customized
> task slice is introduced, we can leverage this hint for latency related
> optimization. Task wakeup is one thing, I can also think of other aspects,
> like idle load balance, etc. I'm not sure what is the proper usage of the
> task slice though, this is why I sent this RFC.
> 
>> Note:
>> This will not work if the SIS scan itself is the largest overhead in the
>> wakeup cycle and not the task placement itself. Previously during
>> SIS_UTIL testing, to measure the overheads of scan vs placement, we
>> would do a full scan but return the result that SIS_UTIL would have
>> returned to determine the overhead of the search itself.
>>
> 
> Regarding the task placement, do you mean the time between a task is enqueued
> and picked up? Do you have any recommendation which workload can expose the
> scan overhead most?

Sorry for not being clear here. From what I've observed in the past,
there are two dimensions to slect_idle_sibling():

i)  Placement: Final CPU select_idle_sibling() returns
ii) Search: Do we find an idle core/CPU in select_idle_sibling()

In case of netperf, I've observed that i) is more important than ii)
wherin a placement of client on same core/thread as that of the server
results in better performance vs finding an idle CPU on a remote LLC.
For hackbench/tbench, when runqueues are under high utilization (~75%),
reduction in search time ii) seems to be more beneficial.

There was also a wakeup from IPI / without IPI angle that I never quite
got to the bottom of that Mathieu has highlighted last year. I'll go
get some more data on that front and give your patch a try. Expect
results in a couple of days. 

Thank you.

> 
> thanks,
> Chenyu
>  
>>>  				return -1;
>>>  		}
>>>  	}
>>> [..snip..]
--
Thanks and Regards,
Prateek