Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 4 Apr 2023 23:15:40 +0800
From:   Aaron Lu <aaron.lu@intel.com>
To:     Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@redhat.com>,
        Juri Lelli <juri.lelli@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Daniel Jordan <daniel.m.jordan@oracle.com>,
        Yu Chen <yu.c.chen@intel.com>
CC:     Steven Rostedt <rostedt@goodmis.org>,
        Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        Valentin Schneider <vschneid@redhat.com>,
        Tim Chen <tim.c.chen@intel.com>,
        Nitin Tekchandani <nitin.tekchandani@intel.com>,
        Waiman Long <longman@redhat.com>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [RFC PATCH] sched/fair: Make tg->load_avg per node
Message-ID: <20230404151540.GA51499@ziqianlu-desk2>
References: <20230327053955.GA570404@ziqianlu-desk2>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230327053955.GA570404@ziqianlu-desk2>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?dmTLUIAOoJtVnGUNJoqKg+ZDclXveIZXxtcXlj6OQV0ZhxxD2H/uKGGTjWdL?=
 =?us-ascii?Q?Fu8TCDcyj4ltVSsHlN3vMx+ME9I0HvI3Po9rRxZzjODEjt/RPZmJ4rUUu15G?=
 =?us-ascii?Q?bCl9apAQyBQsUm0USuzwq0+Yx50he0mD9DzvJyuhO2MowuQnPUVLEsDBG2NV?=
 =?us-ascii?Q?KqejPDvBOqYYN7zLDe1XCRrRVYBoWYBR18P7jB4epIeYqZW64VQYvrl9YooH?=
 =?us-ascii?Q?tuwp2F1XH+tT10WuOjy7d0Qi/ht2PrSEp6hs9s9DER7MXtINo3BkLEDlgGLg?=
 =?us-ascii?Q?XUq5XnwrqVf4fo6UxzJx44SWjrk/nwsXJbg4pE04kShlSrgMDS+oEgdgTCUg?=
 =?us-ascii?Q?tkj8Wbr/tyO8hIxwTl+34J+hnwad9bhraVQu2H/lL7xw4kwf217rjUzcv55U?=
 =?us-ascii?Q?BjYH7+QfAwjZEJKxq6FvgATYpKOuG+sxmm0hESSAH6CPco2Kk6myQu75+5Yv?=
 =?us-ascii?Q?KgrYtYQDw83OYkzS1Z/9NguXguM41xJRPRDYT6dn4XeA91NTqqBlb6K1basU?=
 =?us-ascii?Q?txae1RhBBNJdSgEJkvFQgKSNHdO26wiYk8xsFC86Ql0HuuTAWRpxW6nKFdlP?=
 =?us-ascii?Q?MHfPBC7WnHhJmU558qEwLOGwyWbjuzsdHB/Rch8hxJJZSOWKmXxvWMJzUF0Z?=
 =?us-ascii?Q?XbNXnwZotiUEUCipENtb6z1KyRz5rV9ToIPEwy/KlkXyAoYPK+NC3FpeF7wR?=
 =?us-ascii?Q?0L6Gwh9pqDanSWx1HGHEQnIVqrhMoNAFFoEMqCBXtYmyxXygkmZAqmEtQdBB?=
 =?us-ascii?Q?Z0lFYfe1ftsVVs03vLUj6hwJugP4dyrcThakEx1QIf8DST9QYcdzSsYdT3Ou?=
 =?us-ascii?Q?kCWX8YCD2Km2Da6m52Hqlc89tRIIArsSjjSU6RdO1Spc8G8/rT2YnoWl61eq?=
 =?us-ascii?Q?hCtM0xXnFz40lRvthT1ne/GQod30SWXKzFzc60GT7VRekpcUoa/9G2VsdVdS?=
 =?us-ascii?Q?g56bG2rpccBhI5ee7wprj2vPCfZ9EvxMUfsTeaDFkV31ujHQbISOMQ2pkp1y?=
 =?us-ascii?Q?kwtlEezl8GGBR8LCOR1WH2HMoR3DZxSJ9aSc2D9JIn8JE/iI2wqVLyb4x3+Q?=
 =?us-ascii?Q?bPJ+1E5SATzRPrbPdh2/DXDBKi9zN/zb6NCZZIZVzP8pOQFq6qmKcZvtk6ae?=
 =?us-ascii?Q?j+vK7XBXUddE3MFJpGeSvnR66DrBBzjey01KGrb3nVbCxEuvQC7ri28Co2x9?=
 =?us-ascii?Q?BuxnC8piYvh/WqVMsPkoBgEqJ27e0cY7ZnhDatHMJg6jYDR0RnQ1Q0nPRgrM?=
 =?us-ascii?Q?TKN+m/S0OeJwyirKMt2p1BItdgbUrF1KdowCG8cWu8OQOLHfyhq+cOPznU9A?=
 =?us-ascii?Q?M+jK1CbY4inYAzVeUeabOJnZe0O6y769uDMTKoDIa25TOPnO7y/d8B9GFB5Z?=
 =?us-ascii?Q?vvUEA+tYyIoLzmco5pzAOhmBEGA3oisVHxS5L8JsrPdeeOMVjx1CIxwi9MFk?=
 =?us-ascii?Q?OdqpqpD6S0+uVRzZpWLCEW71r8RSkMAMMASePI4cbkfG+wS3GzNTnZq+VNd9?=
 =?us-ascii?Q?r9Av7RqIm6KC4rt2H1I3k6j4OymUjKX9ixMph6Ars3diGqrJP6Ko+ILvsQw1?=
 =?us-ascii?Q?zGjSieTEZt1ZrskNVVEZ+WYmx/lS6ef8aBGOqNjQ?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 2b03d218-f0c0-4b74-4f9b-08db351f7b51
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3062.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Apr 2023 15:15:53.4218
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: disYp7wicJ0RbWN8oDsyV588zFvvZpXyUkTubpH+9Aw2hVAXh9XE0xNi1Tut72Fv/Utf+VLdFMF+uXZfu2r5gQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR11MB7965
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Mar 27, 2023 at 01:39:55PM +0800, Aaron Lu wrote:
[...]
> Another observation of this workload is: it has a lot of wakeup time
> task migrations and that is the reason why update_load_avg() and
> update_cfs_group() shows noticeable cost. Running this workload in N
> instances setup where N >= 2 with sysbench's nr_threads set to 1/N nr_cpu,
> task migrations on wake up time are greatly reduced and the overhead from
> the two above mentioned functions also dropped a lot. It's not clear to
> me why running in multiple instances can reduce task migrations on
> wakeup path yet.

Regarding this observation, I've some finding. The TLDR is: 1 instance
setup's overall CPU util is lower than N >= 2 instances setup and as a
result, under 1 instance setup, sis() is more likely to find idle cpus
than N >= 2 instances setup and that is the reason why 1 instance setup
has more migrations.

More details:

For 1 instance with nr_thread=nr_cpu=224 setup, during a 5s window,
there are 10 million calls of select_idle_sibling() and 6.1 million
migrations. Of these migrations, 4.6 million comes from select_idle_cpu(),
1.3 million comes from recent_cpu.
mpstat of this time window:
Average:    NODE    %usr   %nice    %sys %iowait    %irq   %soft  %steal %guest  %gnice   %idle
Average:     all   45.15    0.00   18.59    0.00    0.00   17.29    0.00 0.00    0.00   18.98
Average:       0   38.14    0.00   17.29    0.00    0.00   14.77    0.00 0.00    0.00   29.80
Average:       1   52.07    0.00   19.88    0.00    0.00   19.78    0.00 0.00    0.00    8.28


For 4 instance with nr_thread=56 setup, during a 5s window, there are 15
million calls of select_idle_sibling() and only 30k migrations.
select_idle_cpu() is called 15 million times but only 23k of them passed
the sd_share->nr_idle_scan != 0 test.
mpstat of this time window:
Average:    NODE    %usr   %nice    %sys %iowait    %irq   %soft  %steal %guest  %gnice   %idle
Average:     all   68.54    0.00   21.54    0.00    0.00    8.35    0.00 0.00    0.00    1.58
Average:       0   70.05    0.00   20.92    0.00    0.00    8.17    0.00 0.00    0.00    0.87
Average:       1   67.03    0.00   22.16    0.00    0.00    8.53    0.00 0.00    0.00    2.29

For 8 instance with nr_thread=28 setup, during a 5s window, there are
16 million calls of select_idle_sibling() and 9.6k migrations.
select_idle_cpu() is called 16 million times but none of them passed the
sd_share->nr_idle_scan != 0 test.
mpstat of this time window:
Average:    NODE    %usr   %nice    %sys %iowait    %irq   %soft  %steal %guest  %gnice   %idle
Average:     all   70.29    0.00   20.99    0.00    0.00    8.28    0.00 0.00    0.00    0.43
Average:       0   71.58    0.00   19.98    0.00    0.00    8.04    0.00 0.00    0.00    0.40
Average:       1   69.00    0.00   22.01    0.00    0.00    8.52    0.00 0.00    0.00    0.47

On a side note: when sd_share->nr_idle_scan > 0 and has_idle_core is true,
then sd_share->nr_idle_scan is not actually respected. Is this intended?
It seems to say: if there is idle core, then let's try hard and ignore
SIS_UTIL to find that idle core, right?