Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34;
Date:   Sun, 26 Nov 2023 15:21:46 +0800
From:   Chen Yu <yu.c.chen@intel.com>
To:     Madadi Vineeth Reddy <vineethr@linux.ibm.com>
CC:     Peter Zijlstra <peterz@infradead.org>,
        Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
        Ingo Molnar <mingo@redhat.com>,
        "Vincent Guittot" <vincent.guittot@linaro.org>,
        Juri Lelli <juri.lelli@redhat.com>,
        <6dccbf0f54cc4ee068a157b9eebfb4b5fa3cc4af.1700548379.git.yu.c.chen@intel.com>,
        Tim Chen <tim.c.chen@intel.com>, Aaron Lu <aaron.lu@intel.com>,
        "Dietmar Eggemann" <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Mel Gorman <mgorman@suse.de>,
        K Prateek Nayak <kprateek.nayak@amd.com>,
        "Gautham R . Shenoy" <gautham.shenoy@amd.com>,
        Chen Yu <yu.chen.surf@gmail.com>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v2 2/3] sched/fair: Calculate the cache-hot time of the
 idle CPU
Message-ID: <ZWLyChTTXq049zRy@chenyu5-mobl2.ccr.corp.intel.com>
References: <cover.1700548379.git.yu.c.chen@intel.com>
 <6dccbf0f54cc4ee068a157b9eebfb4b5fa3cc4af.1700548379.git.yu.c.chen@intel.com>
 <29f5225a-e0fb-2381-4f19-49334a0dbbc8@linux.ibm.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <29f5225a-e0fb-2381-4f19-49334a0dbbc8@linux.ibm.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?gyHfHqQqMBRywSWtr6EfvSq7npUz8OMLXOaill7cyREbNo9JScgH+Fl/M/jo?=
 =?us-ascii?Q?nYJX29DjChGal4AayF9izVfLxHkyB1507eb9nEEKtexU+VTc/J/EPAhPRfUZ?=
 =?us-ascii?Q?S9Y0ITUYFWPInbn0m83GVIMbWdh7bdxALaG3tWVxonM1wSBUEbC34DPm4BYt?=
 =?us-ascii?Q?ANyHch4urTzOPlFHXca7jf+lqyJTpRQu/psZ6flwnzRipRkMxqb9wbwntSXB?=
 =?us-ascii?Q?ZW0Q3zvWvisAjb0+l/s2Ilagx5y/xKa75a7pqsaAIrengLrW/CEYWTxGj+Fc?=
 =?us-ascii?Q?Q1gbjgQ3tv/VJUVjXHW96PcgFcLqB8TZWTfqfDsULE7PWaLajMIdLFanalQF?=
 =?us-ascii?Q?GQXahBj+Xtl/XmUlbxghQcAWkFRm91DmWDX1JlbSwLUpicylD7ipkBk4+OvJ?=
 =?us-ascii?Q?21bU9kZNJymCciQsnPJdRV3m2lsPfPDFoQ0N9PMd4u6VRfZ4wbEYgUosVM0T?=
 =?us-ascii?Q?qw9vvwWvhGfWMMeuoQGCqKxUknwBJ1z2P1B0lzPGkdQWr2uYV6OjrepY+Hbx?=
 =?us-ascii?Q?iIFUR6DQU85YMzNay9dsWitei7UpR7sMccLGsRyrVSJlYfiT9cVpDAft+Ygd?=
 =?us-ascii?Q?JiqgjX5Vqonp4hfTHEn6MqcWIbKqdD5H2AE2qtxX2RmLmOvVCyOyLVxoYI94?=
 =?us-ascii?Q?wOETCPh+1ahc8TuxazzDFrvXNnBnUmIzzVbNGDnhXuYL4dMt08tr0UlyU4iP?=
 =?us-ascii?Q?N3pyLeZH+q4HmZd8+MAeZS/XGzkxxdqDUw4K1Bxr3Gnwz+INEJ7HthBhMC9p?=
 =?us-ascii?Q?Cb/sQ9rY+zzz2XwGpVx0veIjtXzZoEVj1JfZ/V2jU4KNkmCZ//c1EsvRVHH/?=
 =?us-ascii?Q?4KoiuanUKs3KHmCG6Z6caoszKkWbrasoXGioyHdPMpxz9/Fs3t3Cx5KgMQAj?=
 =?us-ascii?Q?rEgzTvylDmmOp77+31EbvHXrjj13LMsHTCAFWJTPadEd9/HHHzTqwceozsdk?=
 =?us-ascii?Q?+3DZAnXoddzoUeVF05NoGt4Ixcytzid65zCi8VxSR6J3lM9+fGo801QupsSf?=
 =?us-ascii?Q?SbUQt2ycQ1Mx/MV9rWbnwpKFzv8J2jlkG7XilmdgW8asGch4FlFHoK0w1YEv?=
 =?us-ascii?Q?+623vxBeouu5aSpN0mtcciHqSsXTLjh2568Mnl++yOP/C/zwRVxOkLyyXTMT?=
 =?us-ascii?Q?IIyuVETrqB3xgtlYXCRrBVBFEhx8LdXILNLBQ0g2QH/63cDwAZlJwSiKLLrW?=
 =?us-ascii?Q?dRtvnO3hPLXhZQl8eAsM+wm7G15OW8OxneVx+MJbKDgea9bkWbnOP3oBr77q?=
 =?us-ascii?Q?GyaidA5vyxlPKv798GtFV4Exa8x/QHfGG3Gsti66UfzYAcPuebqGqa99GQRK?=
 =?us-ascii?Q?3ESeedRAu2H63gh9DdEjCRID5nFwCqiwRdAisnpAa4NEPLiMjb9NBepVClLN?=
 =?us-ascii?Q?BgeLN7GUqblW05HLKWlVuGiokYyoKJidABJpwz9GTMNE1sMyz6lkSyeM68Hb?=
 =?us-ascii?Q?oCd2FFgO4eU4+Zd26NKpg/BEKPjcJaLfdKOIGXm0pwB3Ez1FufDl2V+t15sn?=
 =?us-ascii?Q?x5yIyIPc+S8h5vNpeMEniz7U1ZYer50q7dz/PBMRQwTZ6o+bmez+QgrORsO4?=
 =?us-ascii?Q?fc7s/d7ViXkBBiuO0MFywUmlJcKGDCWwvFqup86A?=
X-MS-Exchange-CrossTenant-Network-Message-Id: d86cf34a-d3e2-4982-1238-08dbee506171
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6206.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Nov 2023 07:22:00.4372
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: HW00vx8lk9GoEf9BBt9Bx/FVOUU8raUgy0ncgpyFOn5kVE5+fMDgt1PQLrbdDVJxDifm4/HT16iBwSJnwz5MkA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB7368
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on howler.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Sat, 25 Nov 2023 23:22:16 -0800 (PST)

Hi Madadi,

On 2023-11-25 at 12:40:18 +0530, Madadi Vineeth Reddy wrote:
> Hi Chen Yu,
> 
> On 21/11/23 13:09, Chen Yu wrote:
> > When a CPU is about to become idle due to task dequeue, uses
> > the dequeued task's average sleep time to set the cache
> > hot timeout of this idle CPU. This information can facilitate
> > SIS to skip the cache-hot idle CPU and scan for the next
> > cache-cold one. When that task is woken up again, it can choose
> > its previous CPU and reuses its hot-cache.
> > 
> > This is a preparation for the next patch to introduce SIS_CACHE
> > based task wakeup.
> > 
> > Signed-off-by: Chen Yu <yu.c.chen@intel.com>
> > ---
> >  kernel/sched/fair.c     | 30 +++++++++++++++++++++++++++++-
> >  kernel/sched/features.h |  1 +
> >  kernel/sched/sched.h    |  1 +
> >  3 files changed, 31 insertions(+), 1 deletion(-)
> > 
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index 672616503e35..c309b3d203c0 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -6853,8 +6853,17 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)
> >  	util_est_update(&rq->cfs, p, task_sleep);
> >  
> >  	if (task_sleep) {
> > -		p->last_dequeue_time = sched_clock_cpu(cpu_of(rq));
> > +		u64 now = sched_clock_cpu(cpu_of(rq));
> > +
> > +		p->last_dequeue_time = now;
> >  		p->last_dequeue_cpu = cpu_of(rq);
> > +
> > +#ifdef CONFIG_SMP
> > +		/* this rq becomes idle, update its cache hot timeout */
> > +		if (sched_feat(SIS_CACHE) && !rq->nr_running &&
> > +		    p->avg_hot_dur)
> > +			rq->cache_hot_timeout = max(rq->cache_hot_timeout, now + p->avg_hot_dur);
> 
> As per the discussion in the rfc patch, you mentioned that SIS_CACHE only honors the average sleep time
> of the latest dequeued task and that we don't know how much of the cache is polluted by the latest task.
> 
> So I was wondering what made you to put max here.
>

Thanks for taking a look. Yes, previously SIS_CACHE only honors the latest dequeue task.
But as Mathieu pointed out[1], the latest dequeue task might not have enough time to scribble
the cache footprint of some older dequeue tasks, and we should honor the sleep time of
those older dequeue tasks. Consider the following scenario:

task p1 is dequeued with an average sleep time of 2 msec. Then p2 is scheduled in
on this cpu, but only runs for 10 us(does not pollute the cache footprint) and
dequeued with average sleep time of 1 msec. Should we tag the CPU runqueue's timeout
as 2 msec or 1 msec later? We choose 2 msec. The idea is to make the timeout moving
forward so the SIS_CACHE could make it easier for the p1 to be woken up on its previous
CPU.

[1] https://lore.kernel.org/lkml/2a47ae82-b8cd-95db-9f48-82b3df0730f3@efficios.com/

thanks,
Chenyu