Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3;
Date:   Fri, 13 Oct 2023 23:37:18 +0800
From:   Chen Yu <yu.c.chen@intel.com>
To:     Vincent Guittot <vincent.guittot@linaro.org>
CC:     Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
        Peter Zijlstra <peterz@infradead.org>,
        <linux-kernel@vger.kernel.org>, Ingo Molnar <mingo@redhat.com>,
        Valentin Schneider <vschneid@redhat.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        "Juri Lelli" <juri.lelli@redhat.com>,
        Swapnil Sapkal <Swapnil.Sapkal@amd.com>,
        "Aaron Lu" <aaron.lu@intel.com>, Tim Chen <tim.c.chen@intel.com>,
        K Prateek Nayak <kprateek.nayak@amd.com>,
        "Gautham R . Shenoy" <gautham.shenoy@amd.com>, <x86@kernel.org>
Subject: Re: [RFC PATCH] sched/fair: Bias runqueue selection towards almost
 idle prev CPU
Message-ID: <ZSlkLqa55j4C2rDl@chenyu5-mobl2.ccr.corp.intel.com>
References: <20230929183350.239721-1-mathieu.desnoyers@efficios.com>
 <ZRfKKxBzfu+kf0tM@chenyu5-mobl2.ccr.corp.intel.com>
 <0f3cfff3-0df4-3cb7-95cb-ea378517e13b@efficios.com>
 <ZSOMOhhkPIFmvz97@chenyu5-mobl2.ccr.corp.intel.com>
 <ebe4e40f-37df-40a9-9dfc-7f2a458151bd@efficios.com>
 <ZSZ2ERMysY7iEo+x@chenyu5-mobl2.ccr.corp.intel.com>
 <CAKfTPtCpMx9mHrWko6_hgbw+8HObcJJNkj3A6ZEP4C1PekcxMQ@mail.gmail.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <CAKfTPtCpMx9mHrWko6_hgbw+8HObcJJNkj3A6ZEP4C1PekcxMQ@mail.gmail.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?ZN8GvuspeAUHtAxhlpQTO8w80DQKdaZNIm8K/zoHQorJ4KZvTWBPT7sDA8A5?=
 =?us-ascii?Q?CT5B8eOaDyIiHjZjQp8k9mDdBn+2bjilKnnF4dIysoRMcFuCrozyGhqh7nfR?=
 =?us-ascii?Q?s1a+hybK8Nc1VMrPpQyKWu34N3HsuNGKVgc7a903d7r1uAvGgBWCFmjvoU6a?=
 =?us-ascii?Q?KQFYl+iASyC4/v50Yli/Uyg6zLU019NhAxGa0hZTDa2jtNajK7inpSnHsfR5?=
 =?us-ascii?Q?bWocTTxiIlKgN4RNnHcwIXND93K0gZOPh9IwJhR6H95M0zSvWqcB6fWyMXOj?=
 =?us-ascii?Q?3mFyxL0mqhF/hGfixld/tNyjrggJo9n6goFnFK5ySzZDFItDuv7iirHQDn81?=
 =?us-ascii?Q?Q7ygdl4lFkIC5yqB5khPkHiUn2yQjP6SLCF2KwZrpusK+2AtgK2sHqv60a0t?=
 =?us-ascii?Q?k6IHFhWTUK592nagyrn9vIA8mYndONO1kPFjz6Xp3DLY1ENBb7FIM09brcue?=
 =?us-ascii?Q?vZSfAAhP8XvcSHKUf4W+wIhqmkFLLcL1A9q5Cx2de9tG8J7FsR5XS7Dffopo?=
 =?us-ascii?Q?2IRpCZLPVTIxXNmcG9NBKeKFw2hJJaEGvKePeHB25wJzVS4+rhe3MbjWl1v/?=
 =?us-ascii?Q?LHnzyS+QEHYTwENfOX043ckr7l5cIAvOJF8z6tFCJZZ4iirlfG7Ru1MkEBVY?=
 =?us-ascii?Q?hgC7Vq9Uj1E5zKXQP06fQSoEgcHD8ZUNUGb1NGSduGpBZmvOXyk+NClAEOBu?=
 =?us-ascii?Q?PNnSXPjddHhqvRklHyhrmJE32JKTyrsBTOn2csJNhpqqUeDep8hQkXgzGiLh?=
 =?us-ascii?Q?n+pP+oMYnOechLAzYl9GyrvImwdAvlyD5tkNR3Us3GR1W2U9uqrFR7/R9fmK?=
 =?us-ascii?Q?/Vk9bugDVpiY+jno1Ro2U0agISI1b2NAw9e0Tr/SZzwD94YrxW6q+5B3gg8Q?=
 =?us-ascii?Q?mbjuW5eUNFVFLOPm1CWK0gAki6IFF8ythzD0aYHs7DrlkWuNwyUhHf5DUtXQ?=
 =?us-ascii?Q?OoRDAfDN5lDn0LxS3GgyJHtExoSeYuQILK76hNDtziaohMB5QD7+P6dm0Mp8?=
 =?us-ascii?Q?IjB/6BbrXuxEBoDjp55x9cbSJLge9Ug25MdhGpbFJkerzoUeF6SVOQ3I2JNb?=
 =?us-ascii?Q?BDuU3FtpY0SlLSeH4C7cK54oG9eEjz5IMA9SxEtxmapVsG9M0Kx5TJoffxtq?=
 =?us-ascii?Q?Yx8aDCBs44JEPEEAsh+OFsjGn20VRBDq0d0qfcg1CTkvBVAWt4ClUMVoG/sX?=
 =?us-ascii?Q?rrChFNTGN5+nMP/B+eXVQyfYI+PUJedOISs5G2V9qh2IuLgb1XBz9Fx9XdNa?=
 =?us-ascii?Q?WwmoyrsYW+JqT/10GeQPbFufKS6/S1Gm8xCszNPsVUDLqiQhcRVr743FQutS?=
 =?us-ascii?Q?OuRvN9oEWhLdPXZfuH5+bLbrj/7V38cyCSggBYG4MTcWLdoJ5MKzmzyXrspl?=
 =?us-ascii?Q?yZTU3SV2DcGxm7tHzROGIC+t52mk2Q9GL65tqJGh94Bkk8EmbFFm+c24kYtf?=
 =?us-ascii?Q?kyq866kb72YS+X2ix4HCkhLcr/TE2hJ5B+4Nd0lhcy/a7yMiE9AvR5zmKji1?=
 =?us-ascii?Q?OL40IKIUgl0LD8iKO4Rgb6drglNGdns3IOcGyEhGDUCG3K1sfuEXiR5uDzLe?=
 =?us-ascii?Q?Av651im1cp6u/+ss5Qr03uNBP4v4zM8nu0hD0WGF?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 683ac26a-7e57-484d-1fec-08dbcc0251dd
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6206.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Oct 2023 15:37:34.1300
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Yn8KZjDjSGh5HJcbObfZdUXApQcETq3Diu5b2suc249p2E6f33AFw35CgGZpeuA7s/kqJGvEhMiRWbR51T+haw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR11MB4581
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Fri, 13 Oct 2023 08:37:54 -0700 (PDT)

On 2023-10-12 at 17:26:36 +0200, Vincent Guittot wrote:
> On Wed, 11 Oct 2023 at 12:17, Chen Yu <yu.c.chen@intel.com> wrote:
> >
> > On 2023-10-10 at 09:49:54 -0400, Mathieu Desnoyers wrote:
> > > On 2023-10-09 01:14, Chen Yu wrote:
> > > > On 2023-09-30 at 07:45:38 -0400, Mathieu Desnoyers wrote:
> > > > > On 9/30/23 03:11, Chen Yu wrote:
> > > > > > Hi Mathieu,
> > > > > >
> > > > > > On 2023-09-29 at 14:33:50 -0400, Mathieu Desnoyers wrote:
> > > > > > > Introduce the WAKEUP_BIAS_PREV_IDLE scheduler feature. It biases
> > > > > > > select_task_rq towards the previous CPU if it was almost idle
> > > > > > > (avg_load <= 0.1%).
> > > > > >
> > > > > > Yes, this is a promising direction IMO. One question is that,
> > > > > > can cfs_rq->avg.load_avg be used for percentage comparison?
> > > > > > If I understand correctly, load_avg reflects that more than
> > > > > > 1 tasks could have been running this runqueue, and the
> > > > > > load_avg is the direct proportion to the load_weight of that
> > > > > > cfs_rq. Besides, LOAD_AVG_MAX seems to not be the max value
> > > > > > that load_avg can reach, it is the sum of
> > > > > > 1024 * (y + y^1 + y^2 ... )
> > > > > >
> > > > > > For example,
> > > > > > taskset -c 1 nice -n -20 stress -c 1
> > > > > > cat /sys/kernel/debug/sched/debug | grep 'cfs_rq\[1\]' -A 12 | grep "\.load_avg"
> > > > > >     .load_avg                      : 88763
> > > > > >     .load_avg                      : 1024
> > > > > >
> > > > > > 88763 is higher than LOAD_AVG_MAX=47742
> > > > >
> > > > > I would have expected the load_avg to be limited to LOAD_AVG_MAX somehow,
> > > > > but it appears that it does not happen in practice.
> > > > >
> > > > > That being said, if the cutoff is really at 0.1% or 0.2% of the real max,
> > > > > does it really matter ?
> > > > >
> > > > > > Maybe the util_avg can be used for precentage comparison I suppose?
> > > > > [...]
> > > > > > Or
> > > > > > return cpu_util_without(cpu_rq(cpu), p) * 1000 <= capacity_orig_of(cpu) ?
> > > > >
> > > > > Unfortunately using util_avg does not seem to work based on my testing.
> > > > > Even at utilization thresholds at 0.1%, 1% and 10%.
> > > > >
> > > > > Based on comments in fair.c:
> > > > >
> > > > >   * CPU utilization is the sum of running time of runnable tasks plus the
> > > > >   * recent utilization of currently non-runnable tasks on that CPU.
> > > > >
> > > > > I think we don't want to include currently non-runnable tasks in the
> > > > > statistics we use, because we are trying to figure out if the cpu is a
> > > > > idle-enough target based on the tasks which are currently running, for the
> > > > > purpose of runqueue selection when waking up a task which is considered at
> > > > > that point in time a non-runnable task on that cpu, and which is about to
> > > > > become runnable again.
> > > > >
> > > >
> > > > Although LOAD_AVG_MAX is not the max possible load_avg, we still want to find
> > > > a proper threshold to decide if the CPU is almost idle. The LOAD_AVG_MAX
> > > > based threshold is modified a little bit:
> > > >
> > > > The theory is, if there is only 1 task on the CPU, and that task has a nice
> > > > of 0, the task runs 50 us every 1000 us, then this CPU is regarded as almost
> > > > idle.
> > > >
> > > > The load_sum of the task is:
> > > > 50 * (1 + y + y^2 + ... + y^n)
> > > > The corresponding avg_load of the task is approximately
> > > > NICE_0_WEIGHT * load_sum / LOAD_AVG_MAX = 50.
> > > > So:
> > > >
> > > > /* which is close to LOAD_AVG_MAX/1000 = 47 */
> > > > #define ALMOST_IDLE_CPU_LOAD   50
> > >
> > > Sorry to be slow at understanding this concept, but this whole "load" value
> > > is still somewhat magic to me.
> > >
> > > Should it vary based on CONFIG_HZ_{100,250,300,1000}, or is it independent ?
> > > Where is it documented that the load is a value in "us" out of a window of
> > > 1000 us ?
> > >
> >
> > My understanding is that, the load_sum of a single task is a value in "us" out
> > of a window of 1000 us, while the load_avg of the task will multiply the weight
> 
> I'm not sure we can say this. We use a 1024us sampling rate for
> calculating weighted average but load_sum is in the range [0:47742]
> so what does it mean 47742us out of a window of 1000us ?
> 
> Beside this we have util_avg in the range [0:cpu capacity] which gives
> you the average running time of the cpu
>

Sorry I did not describe it accurately. Yes, it should be 1024us instead of 1000us.
And the load_sum is the decayed accumulated duration. util_avg was once used
previously and Mathieu found that it did not work. But in the latest version
it works again, I'll have a test on that version.

thanks,
Chenyu