Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31;
Date:   Wed, 13 Sep 2023 11:02:13 +0800
From:   Chen Yu <yu.c.chen@intel.com>
To:     Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
CC:     Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Juri Lelli <juri.lelli@redhat.com>,
        Tim Chen <tim.c.chen@intel.com>, Aaron Lu <aaron.lu@intel.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        "Steven Rostedt" <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        "Valentin Schneider" <vschneid@redhat.com>,
        K Prateek Nayak <kprateek.nayak@amd.com>,
        "Gautham R . Shenoy" <gautham.shenoy@amd.com>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [RFC PATCH 2/2] sched/fair: skip the cache hot CPU in
 select_idle_cpu()
Message-ID: <ZQEmNWe2iiU1/O+L@chenyu5-mobl2.ccr.corp.intel.com>
References: <cover.1694397335.git.yu.c.chen@intel.com>
 <d49cf5748aa7c6d69580315d2373a9eafa21c21f.1694397335.git.yu.c.chen@intel.com>
 <30a7ff14-3f48-e8cf-333f-cbb7499656e3@efficios.com>
 <db8fa2ae-317b-1c5a-e23f-9d3396165c45@efficios.com>
 <ZQBRIUYT8PE/UbZe@chenyu5-mobl2.ccr.corp.intel.com>
 <244ff1b9-c51e-be9b-0755-757f969b36e1@efficios.com>
 <ZQByY4rDvjejRRs5@chenyu5-mobl2.ccr.corp.intel.com>
 <2a47ae82-b8cd-95db-9f48-82b3df0730f3@efficios.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <2a47ae82-b8cd-95db-9f48-82b3df0730f3@efficios.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?+oKVjPtzlWZioFTx5cfIiD6UmCkMNZp5Sv03VP14RLkgrTM+HHjUEeC6iNYW?=
 =?us-ascii?Q?bVIx00DU0T/zCuBN927RM7oV6UP69Mu4OylSZjHmpykETtxbxP/lPn6Vt3rA?=
 =?us-ascii?Q?huUQdRb1RxCXr++lPyVzy9JTUg/F4PVU86Yse0sdYO8FvWhcF3C6WdVp8Xa5?=
 =?us-ascii?Q?xSTzi+CaobTBVR2DUX+F5zv1GKy51JHkkEN3fLLQyLoqBStLxklOAs78fd6y?=
 =?us-ascii?Q?LImV0QaX+lb55r3kj9DebP+nxNUFoh6HWepadmzt0e5UMW79moGjsMSoNKt7?=
 =?us-ascii?Q?YgtD3zrAQfcpnkaAj26KWfho2XozCYL44IQ2Xo6vRfVf2ux8qTEhn0SQZtfi?=
 =?us-ascii?Q?0xuoTHoghrbjJOPgBlynZpYp8CVfFp2ezGwjW3Ag/5rmfPGiYpBmtREvfcIm?=
 =?us-ascii?Q?KOtyL95jso2zL8krKnjvwY/du1Ik8PKcdEWUaUGeZgfK+rdMrTN1oIcq64mD?=
 =?us-ascii?Q?uXaTnZnl//Yo59UNWDHj+6n5F5SQqLl90f11j/37PTJaVjaDCF8pEN8ANvhz?=
 =?us-ascii?Q?UBtkEcN2hzBg0dnjmo9kgLxRVYQBpkZensZKUm8XXg5M6ekhONiKN18ArjoU?=
 =?us-ascii?Q?VX+hvUjeCnDwyk/8CKkZJoblh2BUSQhTsG590JnST/HAZhL029gSDuhCsmdZ?=
 =?us-ascii?Q?zN9zxbKevAzDAmH7TJbtbcE5qTJAGJBUIlHkL8y6Sb0V2OQcPzr0Crvze04n?=
 =?us-ascii?Q?ZCXvH5agCeoFASYbT7naTf+F+PJx/T6803OWyQX3Mluc5YUKfR8uT2A5Qwfk?=
 =?us-ascii?Q?1dTXObOE7mDoyuacXPdbkVezMnt5/A4JUZQhOZOh3y8TcF4GiuwL/lmMzBvx?=
 =?us-ascii?Q?FfLBKCZOqGNkW8fRui5IhfOJ9Isi4MGAnVhNYRGWa+LDUOlk/33Rvdo2zH6w?=
 =?us-ascii?Q?/xcAOPdmTKUi689daJjkZmvH6ak5jECYeQT5sWKbDsv/z9ya8wh9aQAXstms?=
 =?us-ascii?Q?MDheoNmCIAIKZRiXhtIQ3Q1D6I9u5I1xS6PefLgBrQ0PZXqs/Ws8v17ww1WV?=
 =?us-ascii?Q?QBPbVvfahaBPDjN5KukfootjVRm2k5An549SI/d+u5RxW+VPdMeEdSNhS5yL?=
 =?us-ascii?Q?1oNHf4P+TXDz7chTzZxhcQRGDNxY4hKisD+6fsxHliQPMVyXyeyOUt3oF7db?=
 =?us-ascii?Q?Dzk9OcpR0uzIjGIHOxww4BEMcb2lr7wvjZhThNFPQ+1YSYotdSd1iYuABGEx?=
 =?us-ascii?Q?se5V2SVFZyacszDrrFR8FnzJq5ZbLeWSUz8rwZPBpR3hCu9FLyS3abNJ4sZS?=
 =?us-ascii?Q?CZ6l7W+4c2PkyCda9huR9xecnjg+giqR7jqkCD5xgBnQjkh7LmcWbduAUVtz?=
 =?us-ascii?Q?rWi79sZl7A9CCJOsJLqyFSAk8/Wtk/h34/mHpQsh57XU77hVPLt/TeWXJPCH?=
 =?us-ascii?Q?ce/kV6C8k1bVfBTfgklycfvB+WAYd4++3EPGshaGmv/VBFQ8bU7m8kaynWWO?=
 =?us-ascii?Q?cgezrsBaUfreUvJn6Q/DZ9tTvLfmnN4nqWHR7VDS8F0IXJfYePyCwGEcZFx7?=
 =?us-ascii?Q?Q9OjwDbGBCQLr0vaRzJnrCVz78VC/Bw9tguZQ4yOJwn/iNTu62IljpgzjQ44?=
 =?us-ascii?Q?OiND9InWRN16YVAgdrm2Gcn6laD9UqFfxLuhbNuz?=
X-MS-Exchange-CrossTenant-Network-Message-Id: bf47b5dd-31ac-4f51-a499-08dbb405dd27
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6206.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Sep 2023 03:02:28.2507
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: UuG1gQpxvoKDD/SxREF3aT9Kd62eOh2l3DCOnpR0k2MxTwoEH+h/U1bQ+RKvwlcijH6ZTOiE/Icz26SlqxDX3w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR11MB5450
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Tue, 12 Sep 2023 20:02:40 -0700 (PDT)
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email

On 2023-09-12 at 11:18:33 -0400, Mathieu Desnoyers wrote:
> On 9/12/23 10:14, Chen Yu wrote:
> > On 2023-09-12 at 10:06:27 -0400, Mathieu Desnoyers wrote:
> [...]
> > > 
> > > One more tweak: given that more than one task can update the cache_hot_timeout forward
> > > one after another, and given that some tasks have larger burst_sleep_avg values than
> > > others, I suspect we want to keep the forward movement monotonic with something like:
> > > 
> > > if (sched_feat(SIS_CACHE) && task_sleep && !rq->nr_running && p->se.burst_sleep_avg &&
> > >      rq->cache_hot_timeout < now + p->se.burst_sleep_avg)
> > > 	rq->cache_hot_timeout = now + p->se.burst_sleep_avg;
> > > 
> > 
> > Yeah, Aaron has mentioned this too:
> > https://lore.kernel.org/lkml/ZP7SYu+gxlc%2FYjHu@chenyu5-mobl2/
> > May I know the benefit of keeping forward movement monotonic?
> > I thought that, should we only honor the latest dequeued task's burst_sleep_avg?
> > Because we don't know whether the old deuqued task's cache has been scribbled by the latest
> > dequeued task or not, does it still make sense to wake up the old dequeued task on its
> > previous CPU?
> 
> Here is my reasoning:
> 
> If a second task is scheduled after the first dequeued task (a
> task with large burst_sleep_avg) is dequeued, that second task (with
> small burst_sleep_avg) would need to entirely scribble the other task's
> cache lines within the time given by sysctl_sched_migration_cost, which
> I suspect is typically not very large. So I doubt that the second task
> can entirely kick out the first task cache lines within that time frame,
> and therefore that second task should not move the cache_hot_timeout
> value backwards.
> 
> But perhaps I'm missing something ?
>

You are right, the reservation time itself is not very long, unless someone
enlarges sysctl_sched_migration_cost in user space. Keeping the reservation
time moving forward could help the first dequeued task to be put on its previous
CPU easier(if the second dequeued task has not been woken up yet). I'll modify it
according to your suggestion and to see the results.

thanks,
Chenyu
 
> Thanks,
> 
> Mathieu
> 
> 
> -- 
> Mathieu Desnoyers
> EfficiOS Inc.
> https://www.efficios.com
>