Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 4 Apr 2023 16:25:04 +0800
From:   Chen Yu <yu.c.chen@intel.com>
To:     Aaron Lu <aaron.lu@intel.com>
CC:     Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@redhat.com>,
        Juri Lelli <juri.lelli@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>,
        "Mel Gorman" <mgorman@suse.de>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        Valentin Schneider <vschneid@redhat.com>,
        Tim Chen <tim.c.chen@intel.com>,
        Nitin Tekchandani <nitin.tekchandani@intel.com>,
        Waiman Long <longman@redhat.com>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [RFC PATCH] sched/fair: Make tg->load_avg per node
Message-ID: <ZCve4JaH8EhxBcwQ@chenyu5-mobl1>
References: <20230327053955.GA570404@ziqianlu-desk2>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230327053955.GA570404@ziqianlu-desk2>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?FdPyKnQBq9PLbq9AaMcGmUfIKfqKQ0NCPNspnQsUb57Ya1ndQrXdTWGFVzdI?=
 =?us-ascii?Q?iN3AnvoQS6dzn5nvWqb6+gzhXSz4LYS8UbZwPC8/uvyhXKOlFpLbkboC4GjW?=
 =?us-ascii?Q?W0bdNVOcJcUULrv3BQXq/XJkyO8li2dS0x+0wZLHE2kVkjmTMzTohBRvtb4b?=
 =?us-ascii?Q?khpyjRAo8q7CCRKKNcz90yInunn+E/ALo4urIQP1wm5eJ/NljeJGWcdngdYU?=
 =?us-ascii?Q?w8ozc9USFIdogN1p3lt8tyL+FB/AjkgoNTtvxNnJ8OGCfP806vgWnVHOCfRP?=
 =?us-ascii?Q?EWOUmTBEi0iaXzOY8OGlKLCgn83D82acRXZ/chdGH5bbjqAoX4mRQ+wG3cJG?=
 =?us-ascii?Q?ml0NWZoc2J/9lp+87/YbEilN+O3ZPrl2AHXjFjLA3cRu8/YS6OxX9RsjOE4Q?=
 =?us-ascii?Q?qwvr6yHkhJJXb+VND/Of3yi75ILURQPTAmk+3YJK2kxQVl//ITKIAAkKe6MO?=
 =?us-ascii?Q?w/78eYxdzx3Lu36wj4PbB3npgTpY4LAqe74FXpcY50OmDB+v3sD7Up61PtzQ?=
 =?us-ascii?Q?kJWL/NXQN59XIs0SNbIyPUkKYUkinqSE94WScURo0UGPOQn5/jKfO/ZQQMGl?=
 =?us-ascii?Q?ZQ6XqmTQzOtJdOxjJkBv4ID54XF92y0ntXRPSchuTJzHyKV1ca2A2KZML9Oo?=
 =?us-ascii?Q?XxHsaAL6edju8dA8wgjQF2uoKWUJ03OZ1PahsV8LoFljCh+hy+Gd9caCr502?=
 =?us-ascii?Q?g0lyl8EK1chhAa2cj0ySJjRP3/jSnYvAxKpz02QDMY7QdLM6fn36JZj11JE4?=
 =?us-ascii?Q?NnZC6iPaR7rrspcT0AP0rPwvouqEma0/mX5ZGrE6CYb4bnamv/Y8cZum/vGV?=
 =?us-ascii?Q?XYplhLfptNaoEjOYwQ4lLoYnUCJ5t6XLCacisHvuc6Iirn0FGRBTqrIVYZ2D?=
 =?us-ascii?Q?9k7b/a8JGXUXpJAw3H4GPVHwe/TMGQMp1MH94CuyzrnJ8kLiuPGLDi1m/yOu?=
 =?us-ascii?Q?KOOvNhXOiLPCCOJAXlW4nY+WDQ7PNhjhH0sEirbm7NmxCaqeN4TlwYi4bL8i?=
 =?us-ascii?Q?0h1alqaoSYsxH5YcnDNz5l6WaX23mQLej1HTAoFmtE25FLIj2XEsxiwZ9KRm?=
 =?us-ascii?Q?GoOSbCr0HTFpEXaCRTRucE/xaEyXn+lP9NEAdCpbZ7K4qjDihNpIGTohbE0B?=
 =?us-ascii?Q?jF2zgM1Cm+BuUyH9Vmif1nIhwd7LsGc51toaNR3k9jz2E/S3Ege7lzMpmnuN?=
 =?us-ascii?Q?pUf1ifjluLfkoeTTdHRWi5wA0ZwLR0fda+gNdvAg8YYwxnamCoCKlNu//kS4?=
 =?us-ascii?Q?ZI+8W/bQpczxTNvLQeu39nxZPU1f+HTRFRaGxJbMKlcTQaZimvlY7HPyQJuQ?=
 =?us-ascii?Q?RyFJjZK2ia6e8b0bXyeJPu7xXDY45ko/pNQ433L/QRq5GA3FSWqiPC8Loo1i?=
 =?us-ascii?Q?EYfokNgpgAlxFeJXjPUQ6fWpHm8pJNdiIhjXmRbpmWXLmAQNvSEfiJLB/Lh0?=
 =?us-ascii?Q?qpmLe4c1s4QcdCrCUrH7A/QwNynsb4yOSrnJvyXOr9NC5Z8EXZZRr7SRQE09?=
 =?us-ascii?Q?Lv3eAc0e9CO81Mg8NSUcrfwV9pW0K+96+7x5U9QQwaqvhyKywv7Ux5Mem0am?=
 =?us-ascii?Q?UKX+S0LcD35hzS6v4XzT5nfWuCifw6YiGISFRDgq?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 23878ab0-e729-47f1-c4bd-08db34e628d5
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6206.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Apr 2023 08:25:33.7183
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 73kk7ocxWHXC3ORSgJin5/oHMhNWYCMA/KIm4luR8vLbjbb7yJxIR1N4Xebxorn7v1V8yuq2BnXkviUEWr1y5w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM8PR11MB5653
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE,
        SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 2023-03-27 at 13:39:55 +0800, Aaron Lu wrote:
> When using sysbench to benchmark Postgres in a single docker instance
> with sysbench's nr_threads set to nr_cpu, it is observed there are times
> update_cfs_group() and update_load_avg() shows noticeable overhead on
> cpus of one node of a 2sockets/112core/224cpu Intel Sapphire Rapids:
> 
>     10.01%     9.86%  [kernel.vmlinux]        [k] update_cfs_group
>      7.84%     7.43%  [kernel.vmlinux]        [k] update_load_avg
> 
> While cpus of the other node normally sees a lower cycle percent:
> 
>      4.46%     4.36%  [kernel.vmlinux]        [k] update_cfs_group
>      4.02%     3.40%  [kernel.vmlinux]        [k] update_load_avg
> 
> Annotate shows the cycles are mostly spent on accessing tg->load_avg
> with update_load_avg() being the write side and update_cfs_group() being
> the read side.
> 
> The reason why only cpus of one node has bigger overhead is: task_group
> is allocated on demand from a slab and whichever cpu happens to do the
> allocation, the allocated tg will be located on that node and accessing
> to tg->load_avg will have a lower cost for cpus on the same node and
> a higer cost for cpus of the remote node.
> 
> Tim Chen told me that PeterZ once mentioned a way to solve a similar
> problem by making a counter per node so do the same for tg->load_avg.
> After this change, the worst number I saw during a 5 minutes run from
> both nodes are:
> 
>      2.77%     2.11%  [kernel.vmlinux]        [k] update_load_avg
>      2.72%     2.59%  [kernel.vmlinux]        [k] update_cfs_group
>
The same issue was found when running netperf on this platform.
According to the perf profile:

11.90%    11.84%  swapper          [kernel.kallsyms]   [k] update_cfs_group
9.79%     9.43%  swapper           [kernel.kallsyms]   [k] update_load_avg

these two functions took quite some cycles.

1. cpufreq governor set to performance, turbo disabled, C6 disabled
2. launches 224 instances of netperf, and each instance is:
   netperf -4 -H 127.0.0.1 -t UDP_RR/TCP_RR -c -C -l 100 & 
3. perf record -ag sleep 4

Also the test script could be downloaded via
https://github.com/yu-chen-surf/schedtests.git


thanks,
Chenyu