Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6;
Date:   Thu, 28 Sep 2023 15:57:27 +0800
From:   Chen Yu <yu.c.chen@intel.com>
To:     Aaron Lu <aaron.lu@intel.com>
CC:     Peter Zijlstra <peterz@infradead.org>,
        Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
        Ingo Molnar <mingo@redhat.com>,
        "Vincent Guittot" <vincent.guittot@linaro.org>,
        Juri Lelli <juri.lelli@redhat.com>,
        Tim Chen <tim.c.chen@intel.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
        "Daniel Bristot de Oliveira" <bristot@redhat.com>,
        Valentin Schneider <vschneid@redhat.com>,
        "K Prateek Nayak" <kprateek.nayak@amd.com>,
        "Gautham R . Shenoy" <gautham.shenoy@amd.com>,
        <linux-kernel@vger.kernel.org>, Chen Yu <yu.chen.surf@gmail.com>
Subject: Re: [PATCH 1/2] sched/fair: Record the short sleeping time of a task
Message-ID: <ZRUx587pxJkLOTOB@chenyu5-mobl2.ccr.corp.intel.com>
References: <cover.1695704179.git.yu.c.chen@intel.com>
 <b17a3db703494bdae8492843f7303c13b0dbd0fc.1695704179.git.yu.c.chen@intel.com>
 <20230927075333.GA210077@ziqianlu-dell>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230927075333.GA210077@ziqianlu-dell>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?neWfajAEbuV9eJH7udDfL+9vXvWkEuQYBBAl4H6fKA1VWrYoe5XN2fNeLzRL?=
 =?us-ascii?Q?4EkfM/7IsezetCW8baqobMnBKwIpbjdszFn1FiKPaT6tGzi0WWFE2tW+83fQ?=
 =?us-ascii?Q?aMv51nut7Cy9R6LSfQogRDDyfWlmcNR9eG/lXDMkiTZYYUo7aJ2FNIY3hDRe?=
 =?us-ascii?Q?O9JM4EE5IX/nVIMi67EXp7IFHQgl73o2XLweUA3FsBAvKUv3gajSVuWG5QPS?=
 =?us-ascii?Q?74KeVw4u03VC3YudvWzmQ4MoHQRyBdrSRC6mUbyNBaPKaUT4FiMKZEeWgF25?=
 =?us-ascii?Q?IzRILjiyshqBtuKlzUeuLEGM6qV0hnQpdyAqTz7d1qSSRP8+pSU1SvNQjlHi?=
 =?us-ascii?Q?PrXnYmDJqGUaA66tJZThPHz0YnIyISGKq2dNbKwmXSjBjIFkXmgi2AXN3jjR?=
 =?us-ascii?Q?sK5FG6JoBCpOCgrI3OzK72WnsZl+Myebq7fZQEyGPzc0HixoSP15rVUyUEWE?=
 =?us-ascii?Q?G3USn87K4o5J0f6yU9jjoRDjgESh616q2HzSnPGBD4RQc0LtoEaLYtmGk1TN?=
 =?us-ascii?Q?uWVAf5mL40sPVb3vYo0NO30XauPcYxYdQ5UOJlUMwRNi6VGzXhdl4gnZzuOe?=
 =?us-ascii?Q?kr8QlWbGxt0NuwuBOcEYer3T97A60++wsvlgcgBHrzeww276ep1YCgl3qOOZ?=
 =?us-ascii?Q?aB6cX8sMseusNzuWQ0sVxlltJN9KWWmhXUyHhNVUNoNmQ20r6M2F9/npKDUz?=
 =?us-ascii?Q?GhdK/pNtFcFjSqTsdzBE1UZ+38KZwRwAddFETuK02txQoVdBnVIJ7cTQNl3m?=
 =?us-ascii?Q?eNBxMDusQEHtggrufxYuLpHkPwVeyPs7ilawXiO/HV/hBVlIj2SpZldv19Gf?=
 =?us-ascii?Q?hJuPenfxe3V5onvTGwMC3vLbV5ibJt8BgPoqHyi+4D1B/Ir0VhD3pWIZmTah?=
 =?us-ascii?Q?pF+cyozPoFaYaSjNZD5jfTqJFekkiJR77d/qbcejWoIKt7QXtCABbOo6OUpc?=
 =?us-ascii?Q?uN0Ius7+jrd5Q4wovBq7FB2rdGHMQqwW7DsrnGqDv0Ds2MeNvsxU813T1g8t?=
 =?us-ascii?Q?KJJkzwnRGz8Lx3g47C9po8XdzUGhvpOwaNdgNUh83veL/ADx8YXdh1zem3SM?=
 =?us-ascii?Q?Yuzc4ZGnncVTE73Yk/8anCl2LYFInb8mnFGH7uisoZaumQ+F2SuAg3PvkAL0?=
 =?us-ascii?Q?rSe3zEdtX2yjpTeVB6sS5ZbVgONyjhJL8/IvbOA9/KBC7CGLMVgNmi0TpNn/?=
 =?us-ascii?Q?0yCy3elBIfCIvd70wDjho4SzZkSbZG0LXy5Delj/B30HPvcVODcM3EGtHiQ+?=
 =?us-ascii?Q?RHctpufe1ljQKpi5zl179YH/vZnO2Ppfx7kgzHGdl5IvySkpqEyRZ/QRL+Yd?=
 =?us-ascii?Q?gQw3QmNMfcmIo3zO89LquNB91uMoHaoTBtuTAa22ZRamWwvUf2HnDFkixs5M?=
 =?us-ascii?Q?rlm0TS+shKV/G8NRQo0mvJlux09BRISQgLDHxSs24NZqsgG5HAz2k+JmvM5i?=
 =?us-ascii?Q?psVq09ZwfphyojeA0A+m6P9U+ILNyihvkgZAy3ORRG74BL4Jx0ulwLN1YoJc?=
 =?us-ascii?Q?d71U5q3RI25+DsscZPmCIlpQzjG2LIfnZHIGkslUzy4FW8JgkAamYEzY2EOt?=
 =?us-ascii?Q?Hr45xCYJl+m8HSKgSi9PWb0J8H0pRtSBgTR+jSy8?=
X-MS-Exchange-CrossTenant-Network-Message-Id: eb1bce65-ae64-4dcf-f08e-08dbbff898f3
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6206.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Sep 2023 07:57:44.3113
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: pyzz/l7c7v/h6y6tFl0F6y4nxnl7vNcaeqswGIUDQkj/SR1XbL/nst04rEqwL13J0OZuiXuqHK4Lf74qx0VyeA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA2PR11MB4842
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Thu, 28 Sep 2023 00:58:08 -0700 (PDT)

Hi Aaron,

On 2023-09-27 at 15:53:33 +0800, Aaron Lu wrote:
> On Tue, Sep 26, 2023 at 01:11:02PM +0800, Chen Yu wrote:
> > During task wakeup, the wakee firstly checks if its previous
> > running CPU is idle. If yes, choose that CPU as its first
> > choice. However, in most cases, the wakee's previous CPU
> > could be chosen by someone else, which breaks the cache
> > locality.
> > 
> > Proposes a mechanism to reserve the task's previous
> > CPU for a short while. In this reservation period, other
> > tasks are not allowed to pick that CPU until a timeout.
> > The reservation period is defined as the average short
> > sleep time of the task. To be more specific, it is the
> > time delta between this task being dequeued and enqueued.
> > Only the sleep time shorter than sysctl_sched_migration_cost
> > will be recorded. If the sleep time is longer than
> > sysctl_sched_migration_cost, give the reservation period
> > a penalty by shrinking it to half. In this way, the 'burst'
> > sleeping time of the task is honored, meanwhile, if that
> > task becomes a long-sleeper, the reservation time of that
> > task is shrunk to reduce the impact on task wakeup.
> > 
> > Suggested-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
> > Signed-off-by: Chen Yu <yu.c.chen@intel.com>
> > ---
> >  include/linux/sched.h |  3 +++
> >  kernel/sched/fair.c   | 21 +++++++++++++++++++++
> >  2 files changed, 24 insertions(+)
> > 
> > diff --git a/include/linux/sched.h b/include/linux/sched.h
> > index dc37ae787e33..4a0ac0276384 100644
> > --- a/include/linux/sched.h
> > +++ b/include/linux/sched.h
> > @@ -561,6 +561,9 @@ struct sched_entity {
> >  	u64				vruntime;
> >  	s64				vlag;
> >  	u64				slice;
> > +	u64				prev_dequeue_time;
> > +	/* the reservation period of this task during wakeup */
> > +	u64				sis_rsv_avg;
> 
> Nit: these info are only relavant for task, not group so might be better
> to move them to task_struct to save a little memory?
>

Yes, I'll try to do this.
 
> >  
> >  	u64				nr_migrations;
> >  
> > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > index d0877878bcdb..297b9470829c 100644
> > --- a/kernel/sched/fair.c
> > +++ b/kernel/sched/fair.c
> > @@ -6456,6 +6456,24 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
> >  	struct sched_entity *se = &p->se;
> >  	int idle_h_nr_running = task_has_idle_policy(p);
> >  	int task_new = !(flags & ENQUEUE_WAKEUP);
> > +	u64 last_dequeue = p->se.prev_dequeue_time;
> > +	u64 now = sched_clock_cpu(task_cpu(p));
> 
> I think cpu_of(rq) is more clear than task_cpu(p). Using task_cpu(p)
> seems to suggest task_cpu(p) can be different from cpu_of(rq).
>

You are right. My original thought is to use task's previous CPU rather than
the current rq. But at this stage the task's CPU has already been updated
to the same as rq. I'll think more about how to deal with this properly.
 
> > +
> > +	/*
> > +	 * If the task is a short-sleepting task, there is no need
> > +	 * to migrate it to other CPUs. Estimate the average short sleeping
> > +	 * time of the wakee. This sleep time is used as a hint to reserve
> > +	 * the dequeued task's previous CPU for a short while. During this
> > +	 * reservation period, select_idle_cpu() prevents other wakees from
> > +	 * choosing this CPU. This could bring a better cache locality.
> > +	 */
> > +	if ((flags & ENQUEUE_WAKEUP) && last_dequeue && cpu_online(task_cpu(p)) &&
> 
> Hmm...the cpu_online() check looks weird. If the cpu is offlined, no task
> will be enqueued there, right?
>

Right. If rq and the task's CPU are the same, there is no need to check cpu_online.

thanks,
Chenyu