Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 19 Jul 2023 16:11:02 +0800
From:   Aaron Lu <aaron.lu@intel.com>
To:     Vincent Guittot <vincent.guittot@linaro.org>
CC:     Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@redhat.com>,
        Juri Lelli <juri.lelli@redhat.com>,
        Daniel Jordan <daniel.m.jordan@oracle.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>,
        "Mel Gorman" <mgorman@suse.de>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        Valentin Schneider <vschneid@redhat.com>,
        Tim Chen <tim.c.chen@intel.com>,
        Nitin Tekchandani <nitin.tekchandani@intel.com>,
        Yu Chen <yu.c.chen@intel.com>,
        Waiman Long <longman@redhat.com>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [RFC PATCH 3/4] sched/fair: delay update_tg_load_avg() for
 cfs_rq's removed load
Message-ID: <20230719081102.GB90441@ziqianlu-dell>
References: <20230718134120.81199-1-aaron.lu@intel.com>
 <20230718134120.81199-4-aaron.lu@intel.com>
 <CAKfTPtAu33AN6=X82T=yOgm40S8OXi+sPcF0QyD-bYRPV=xPEg@mail.gmail.com>
 <20230719051826.GB84059@ziqianlu-dell>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230719051826.GB84059@ziqianlu-dell>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?bAqU93VKfgpeek6c+B/pPj/GePuvYboeQlLicqQnEPGQY76hVDxxYn1TQVpG?=
 =?us-ascii?Q?PMnSh4h3hiZUv+gqTi9Dtg8IgmhmRgrFEl7h0gtBg51dkxLdPhV/dmJYhm/x?=
 =?us-ascii?Q?5J6PGOXyy7S2d2hZ0/gbV8jG2102/6zKu+xJKsZCfurGzWnWqNUQ2MvP9t2i?=
 =?us-ascii?Q?HR62mBsrUrJ+5BUyEFGKPFk5X/1hXbT1CpFFTvjt++u1t4TA4zdSdTQ90UvF?=
 =?us-ascii?Q?q8mi0HtAd+RnpZb9EsSbsMBugmqQpswxGp2pmiZy5L/DCbwa3Fsscz2NeMvR?=
 =?us-ascii?Q?tq4Ihvt+nbopMnmGzQLxAoFNZVR9uvn4LUVNCrCBhuUkgVNzVEaXdeypA13w?=
 =?us-ascii?Q?KEkpiYI6WJb9XrHvZbb2GQGoeqAHn0hfNNiR1ggFsiPRC5LGO1qTO8JG5T/v?=
 =?us-ascii?Q?sN2VCLNcTFZ8X88TAUrQhvHxoZBS5xOWY989p7Vu3iX3GVDKKz7ppxepysIX?=
 =?us-ascii?Q?g7QBy46moRu3/SdcD0Ok3tUsnBxEAKo6N6I27TUFaT2EX6AKkID62yMDXdxC?=
 =?us-ascii?Q?uhBErLBn/IIyCf34LfZ9j5t4SsJh/GCerH/llw9PJ+2UcFJlU2a4WUu39N3A?=
 =?us-ascii?Q?Lu42kTfHjt/2H4J29EORUqYB+9BZ6RAQLHERDsIcQBHh1T+JvFL/JUy5+vt3?=
 =?us-ascii?Q?MIA31bNm82KdolbSYW5S1i1SI3XwpViUP7LVLJ757d8X4EdoLorj6OWi8w6R?=
 =?us-ascii?Q?dOz6Q3qJ011ECJ/48XtfMKtq02nnHBut9jf5nmvktAY3xvjbBM3NE/rtWZm9?=
 =?us-ascii?Q?GqYVz21tn459gVDT1zr+lElrx6wsEuv+KgQe3uKXoM9oKzmfhsyeKutRSaGh?=
 =?us-ascii?Q?aMByiO0yr9133poTAWAmWN4qGsLvtqEnD0IEcByc1IraXrp8WSvD+dDOWgef?=
 =?us-ascii?Q?Z0Mk8n5McMcadE63Maf2OQVNfADbEMTas/qXQSXY+RVwsFj4+Rh2DJ8QVLs+?=
 =?us-ascii?Q?PheOx+7etDJpRdGQE0lC8abd3kLtjytncoEdSKID7GjplKKnpA9NEJwU1zBS?=
 =?us-ascii?Q?0F31zAGKguf4IrB0B+SGTq9c7prX09tjWgzrndVFkcHme8tJCh5OLvIys2nE?=
 =?us-ascii?Q?nEhurJlkamIyz5+pYJMoQseztR6+2FgA6Pe2LSHb2Gvs/IofgItTRz3QpisR?=
 =?us-ascii?Q?UNVpOzNOvozNqRMiH1NyHpoKct6Ot+qfzteZv7S36Qrj1LEKfkE+WpQSI0u6?=
 =?us-ascii?Q?cbN9lwLZAK/pJEYD1Lb6f8vz6HwCTip/Yz7uEZP6962uAgXFqWot/KvdbAjW?=
 =?us-ascii?Q?YYf+COYjYSB20T0Ux+Uot3xJtzxRzoAAYTtmW+n1daGxi3JVBJX+6ycgaqTr?=
 =?us-ascii?Q?LUAiJk9BpzlCWDavuTZvw/5xcOhIZVbwrYSu1E7gFVaNryywnvmippYtHPo+?=
 =?us-ascii?Q?CG+mSQHA8vux7U4TiDynKqMmG6XMgSO26BnLhn1DZwQE/QpvdpIDLQDLONXI?=
 =?us-ascii?Q?q8F0PbYfQ6XbgDpCxqG3oQvPrUrrQg9JEXh2tAXV6f2vEdgY9SVN7AFIAuHt?=
 =?us-ascii?Q?kogvWMExAzWJi4VqY2Lpl0WU97e9BrUJNYdpVFI0pbyB01dx2R06DFtlOxQd?=
 =?us-ascii?Q?FTP9SM33+RWcInFWYOmqDEywJCFp6PWU5pW4tOc+?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 1478b467-4e44-49cc-9abb-08db882fb838
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3062.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2023 08:11:14.0193
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Ua2RoXCeNuXJ2aJA0kn76YcyiqxHBIC/obT++btzbWWcfjd153mNSStN22KOhh5QjNQb5MdiOu0URb8R/7pUMA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR11MB6735
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Jul 19, 2023 at 01:18:26PM +0800, Aaron Lu wrote:
> On Tue, Jul 18, 2023 at 06:01:51PM +0200, Vincent Guittot wrote:
> > Have you tried to remove update_cfs_group() from enqueue/dequeue and
> > only let the tick update the share periodically ?
> 
> patch4 kind of did that :-)
> 

More about this.

If I remove update_cfs_group() in dequeue_task_fair() on top of patch4
like this:

From 43d5c12f0b2180c99149e663a71c610e31023d90 Mon Sep 17 00:00:00 2001
From: Aaron Lu <aaron.lu@intel.com>
Date: Wed, 19 Jul 2023 14:51:07 +0800
Subject: [PATCH 1/2] sched/fair: completely remove update_cfs_group() in
 dequeue path

---
 kernel/sched/fair.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 2adb6a6abbce..a21ab72819ce 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -6434,7 +6434,6 @@ static void dequeue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 
 		update_load_avg(cfs_rq, se, UPDATE_TG);
 		se_update_runnable(se);
-		update_cfs_group(se);
 
 		cfs_rq->h_nr_running--;
 		cfs_rq->idle_h_nr_running -= idle_h_nr_running;
-- 
2.40.1

Than P95 latency of the schbench workload I described in patch4's
changelog will increase to > 1ms(base and patch4's P95 < 100us):

Latency percentiles (usec) runtime 300 (s) (18504 total samples)
        50.0th: 20 (9537 samples)
        75.0th: 25 (4869 samples)
        90.0th: 29 (2264 samples)
        95.0th: 2564 (909 samples)
        *99.0th: 20768 (740 samples)
        99.5th: 23520 (93 samples)
        99.9th: 31520 (74 samples)
        min=6, max=40072

If I further remove update_cfs_group() completely in enqueue path on top
of the last change:

From 4e4cb31590ca2e4080ece9cfa9dfaaf26501c60d Mon Sep 17 00:00:00 2001
From: Aaron Lu <aaron.lu@intel.com>
Date: Wed, 19 Jul 2023 15:36:24 +0800
Subject: [PATCH 2/2] sched/fair: completely remove update_cfs_group() from
 enqueue path

---
 kernel/sched/fair.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index a21ab72819ce..8fc325112282 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4847,8 +4847,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
 	 */
 	update_load_avg(cfs_rq, se, UPDATE_TG | DO_ATTACH);
 	se_update_runnable(se);
-	if (cfs_rq->nr_running > 0)
-		update_cfs_group(se);
 	account_entity_enqueue(cfs_rq, se);
 
 	if (flags & ENQUEUE_WAKEUP)
@@ -6344,7 +6342,6 @@ enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
 
 		update_load_avg(cfs_rq, se, UPDATE_TG);
 		se_update_runnable(se);
-		update_cfs_group(se);
 
 		cfs_rq->h_nr_running++;
 		cfs_rq->idle_h_nr_running += idle_h_nr_running;
-- 
2.40.1

Then P50's latency will bump to ~4ms from ~20us:
Latency percentiles (usec) runtime 300 (s) (17940 total samples)
        50.0th: 3996 (12092 samples)
        75.0th: 4004 (4919 samples)
        90.0th: 4004 (0 samples)
        95.0th: 4012 (353 samples)
        *99.0th: 20000 (487 samples)
        99.5th: 20000 (0 samples)
        99.9th: 31136 (72 samples)
        min=7, max=37402
real    5m36.633s
user    47m33.947s
sys     4m47.097s

So for the read side, maybe just keep what patch4 does?

Thanks,
Aaron