Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 12 Apr 2023 22:11:16 +0800
From:   Aaron Lu <aaron.lu@intel.com>
To:     Peter Zijlstra <peterz@infradead.org>
CC:     Ingo Molnar <mingo@redhat.com>, Juri Lelli <juri.lelli@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
        "Daniel Bristot de Oliveira" <bristot@redhat.com>,
        Valentin Schneider <vschneid@redhat.com>,
        Tim Chen <tim.c.chen@intel.com>,
        Nitin Tekchandani <nitin.tekchandani@intel.com>,
        Waiman Long <longman@redhat.com>,
        Yu Chen <yu.c.chen@intel.com>, <linux-kernel@vger.kernel.org>
Subject: Re: [RFC PATCH] sched/fair: Make tg->load_avg per node
Message-ID: <20230412141116.GB155769@ziqianlu-desk2>
References: <20230327053955.GA570404@ziqianlu-desk2>
 <20230412115936.GC628377@hirez.programming.kicks-ass.net>
 <20230412135828.GB629496@hirez.programming.kicks-ass.net>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230412135828.GB629496@hirez.programming.kicks-ass.net>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?OHGmee+CluOMC34V+tvT6tYKv0YeF5VvrcjyRes5mHYGUsfJPyQOfS/hMcQC?=
 =?us-ascii?Q?AaFmhQrjzDnOrMFibp/bNx0VjuTIVQMUC7htoIKFDe14AzX0fintXfF5rGu0?=
 =?us-ascii?Q?XXD/dAcK+fsW1KNoYE+SnMVjDEG1goA7yHO30+JjFcdodAXyCPbLxB71vGxU?=
 =?us-ascii?Q?Dg+ih1qILpI6SVoorHvcLmem7jMfKegdigVf2nHGPRz50wtN86dLOguJ4jsP?=
 =?us-ascii?Q?xgCquTGlqanUsbasXmzJjUSzHXv+e4IGvwwI0Kv9hCI0LouxKC7pxmgai9+z?=
 =?us-ascii?Q?zVI/FdHAiuo4SSYBre38y7MCMJp7/NGiS8yvA1dYUb907eakJSSY++b9TzAT?=
 =?us-ascii?Q?hbyDMGDsltvEuQYfu6bkj7fbXbAMAfaXs47nUDKd6AspAORLC4ULtoD5jkGm?=
 =?us-ascii?Q?kkQYm45JqWgzExV7wFI/mPb0d723J8be2p4geyaNqJK96ZPCdPTSCkppyJqo?=
 =?us-ascii?Q?pU/5cTUsMB4axmdvIpnR2DzgSBrK4JoBylRed7poZw8ffS1k3KI8BCCGRPtr?=
 =?us-ascii?Q?PF9Ykp3rjXL8oo1+G/ftutuXHwmA73O5zQpcaDCov7dGkAWUtkQ9t6RDkEgb?=
 =?us-ascii?Q?ihi5Np/1/3Q6x5PUbtbdJRenEj2veyOMCNGLzeN0S/RmBmQhFATSw2itnvRP?=
 =?us-ascii?Q?3TtmfWaA1DkVBfTBMcnnWnmEKxDNaEgjjrhqggEQo0WbGUhMGa/7Ha38saQS?=
 =?us-ascii?Q?+M6/T1OKW0JmK37JM6zgLTtZlw/YDOO2FpdxYmrENtzWGvfyAMp13JYlJi2S?=
 =?us-ascii?Q?VQ7QF0AOu7qNsUaP0DeANAz7cooNzmfqspldonXP/Q87IGW8NYKWognE9/7m?=
 =?us-ascii?Q?QotOmLcrRVFZlOw8i0E58NNOYU7aiP2BSVchyh7/YTBzc3cuWWkCTwvijR0F?=
 =?us-ascii?Q?Ygu35AcdeaTJwVjYExDPHo4NRsMRNOaOrIYZsOXo3yVz2G5RrhWFlMKhwNuN?=
 =?us-ascii?Q?NnYphr/q/1DSl4zEMY0WXpudUFWJ21dy80Ll0DcpMkakbrhAf24b6PeEMkIM?=
 =?us-ascii?Q?kB59AFRtnGuYicph23e91eiJz0grG84AvZn754z5Eu6oPWc/a+vgU3u2Wp1T?=
 =?us-ascii?Q?11Qt1j13Wmz7PIoqWV7tULNSqW7dCGh5SKXN/yJv0UnpzOwKzoMJAa1M5bQJ?=
 =?us-ascii?Q?pLj04JSt627KdwWpje3KQdHrvOWjORT3p/5p8IXu9J5h09yHmdMZ82C14iON?=
 =?us-ascii?Q?SjAm5suO7DVhBj5Q13GXGPLjCGG1Ygr0dM2+dRHgifTlfZzWGMSBmDuWcD7L?=
 =?us-ascii?Q?kzFzrULHXn3b4Oe33SRcpndPDnOCXvo3EQQgZnMClw4gH9BNFIwE3rw65Yb0?=
 =?us-ascii?Q?BAZBiGmHuA73DNqISfXKOYPCRjYML98OD5iUXhOwz3choMRZvdj0BwWvHCLl?=
 =?us-ascii?Q?sFDFdPqqojdEr9P68LZM94H7HPgxOEQroy4SlcTuyASTc3zMyRTpDY14MMBe?=
 =?us-ascii?Q?Rl6H+0PcPhk9hoBGYpzaSUamLDy7eiAZkLOSQ3lkAJc1qETjA1IF3NA4drO3?=
 =?us-ascii?Q?TEtIoUfqFJOSeWbjyxnz55ySjWitw+r7lPONJ75qXkah13QPMCD6w01ngyHX?=
 =?us-ascii?Q?VHqvEzUNblCTjT/gYshWmst3XE9gqPclpHx+qWLn?=
X-MS-Exchange-CrossTenant-Network-Message-Id: ee2c9beb-dd7d-482c-a48a-08db3b5fce67
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3062.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Apr 2023 14:11:27.5625
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: zpHs7/P3qmT6Iocv38Cy/MWQGxhyHNPLOUbv4fCHe5X9+J5RQogYOiaDd0YOuFnwiRZ8XLQelFsySduV3Z7sFg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA2PR11MB4972
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Apr 12, 2023 at 03:58:28PM +0200, Peter Zijlstra wrote:
> On Wed, Apr 12, 2023 at 01:59:36PM +0200, Peter Zijlstra wrote:
> > On Mon, Mar 27, 2023 at 01:39:55PM +0800, Aaron Lu wrote:
> > > When using sysbench to benchmark Postgres in a single docker instance
> > > with sysbench's nr_threads set to nr_cpu, it is observed there are times
> > > update_cfs_group() and update_load_avg() shows noticeable overhead on
> > > cpus of one node of a 2sockets/112core/224cpu Intel Sapphire Rapids:
> > > 
> > >     10.01%     9.86%  [kernel.vmlinux]        [k] update_cfs_group
> > >      7.84%     7.43%  [kernel.vmlinux]        [k] update_load_avg
> > > 
> > > While cpus of the other node normally sees a lower cycle percent:
> > > 
> > >      4.46%     4.36%  [kernel.vmlinux]        [k] update_cfs_group
> > >      4.02%     3.40%  [kernel.vmlinux]        [k] update_load_avg
> > > 
> > > Annotate shows the cycles are mostly spent on accessing tg->load_avg
> > > with update_load_avg() being the write side and update_cfs_group() being
> > > the read side.
> > > 
> > > The reason why only cpus of one node has bigger overhead is: task_group
> > > is allocated on demand from a slab and whichever cpu happens to do the
> > > allocation, the allocated tg will be located on that node and accessing
> > > to tg->load_avg will have a lower cost for cpus on the same node and
> > > a higer cost for cpus of the remote node.
> > > 
> > > Tim Chen told me that PeterZ once mentioned a way to solve a similar
> > > problem by making a counter per node so do the same for tg->load_avg.
> > 
> > Yeah, I send him a very similar patch (except horrible) some 5 years ago
> > for testing.
> > 
> > > After this change, the worst number I saw during a 5 minutes run from
> > > both nodes are:
> > > 
> > >      2.77%     2.11%  [kernel.vmlinux]        [k] update_load_avg
> > >      2.72%     2.59%  [kernel.vmlinux]        [k] update_cfs_group
> > 
> > Nice!
> > 
> > > Another observation of this workload is: it has a lot of wakeup time
> > > task migrations and that is the reason why update_load_avg() and
> > > update_cfs_group() shows noticeable cost. Running this workload in N
> > > instances setup where N >= 2 with sysbench's nr_threads set to 1/N nr_cpu,
> > > task migrations on wake up time are greatly reduced and the overhead from
> > > the two above mentioned functions also dropped a lot. It's not clear to
> > > me why running in multiple instances can reduce task migrations on
> > > wakeup path yet.
> > 
> > If there is *any* idle time, we're rather agressive at moving tasks to
> > idle CPUs in an attempt to avoid said idle time. If you're running at
> > about the number of CPUs there will be a fair amount of idle time and
> > hence significant migrations.
> > 
> > When you overload, there will no longer be idle time and hence no more
> > migrations.
> > 
> > > Reported-by: Nitin Tekchandani <nitin.tekchandani@intel.com>
> > > Signed-off-by: Aaron Lu <aaron.lu@intel.com>
> > 
> > If you want to make things more complicated you can check
> > num_possible_nodes()==1 on boot and then avoid the indirection, but
> 
> ... finishing emails is hard :-)
> 
> I think I meant to say we should check if there's measurable overhead on
> single-node systems before we go overboard or somesuch.

Got it, hopefully there is no measurable overhead :-)