Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 16 Jun 2023 08:53:38 +0800
From:   Aaron Lu <aaron.lu@intel.com>
To:     David Vernet <void@manifault.com>
CC:     Peter Zijlstra <peterz@infradead.org>,
        <linux-kernel@vger.kernel.org>, <mingo@redhat.com>,
        <juri.lelli@redhat.com>, <vincent.guittot@linaro.org>,
        <rostedt@goodmis.org>, <dietmar.eggemann@arm.com>,
        <bsegall@google.com>, <mgorman@suse.de>, <bristot@redhat.com>,
        <vschneid@redhat.com>, <joshdon@google.com>,
        <roman.gushchin@linux.dev>, <tj@kernel.org>, <kernel-team@meta.com>
Subject: Re: [RFC PATCH 3/3] sched: Implement shared wakequeue in CFS
Message-ID: <20230616005338.GA115001@ziqianlu-dell>
References: <20230613052004.2836135-1-void@manifault.com>
 <20230613052004.2836135-4-void@manifault.com>
 <20230613083203.GR4253@hirez.programming.kicks-ass.net>
 <20230614043529.GA1942@ziqianlu-dell>
 <20230615000103.GC2883716@maniforge>
 <20230615044917.GA109334@ziqianlu-dell>
 <20230615073153.GA110814@ziqianlu-dell>
 <20230615232605.GB2915572@maniforge>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230615232605.GB2915572@maniforge>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?MjaCUN9nn/4/xNvpfsTNXVMPqPAPZp0Xl5yw/QfuNcaUHOFsMEvKBSQpYeEH?=
 =?us-ascii?Q?s9XaT7Ex9MHX7qT9N9rUrFwseEmnifqh3vWituWtHdrc9HdOp3UlNu3iQ6Ss?=
 =?us-ascii?Q?pkIKGDOIPuIlvwmu4cVZ38JDbALsPHXYeZN4JHrEBXSyFtg4iOkdsUbbbiZU?=
 =?us-ascii?Q?7Uc3V2EQg/etUTNrtF65xYi1QzpjYkwtggrPOf7sSVzk3bcX+uuiNTMAFkAi?=
 =?us-ascii?Q?x//TqM9rmkOBC301vjlaEoijHROcHkbTfw+1IkMr20a82uuj2fAgNfl9Uo9K?=
 =?us-ascii?Q?ZYMzohvt5+b3UE9WM4OfzPxDV+jgxpp1hJl+U0/eF0mg9RlFdjuFNea39rKA?=
 =?us-ascii?Q?CnNF8nM72H9hSeNq+88InMF/Z1oZMv/dfJ0tVYMvamdQT85Z8NXG70g1sHVi?=
 =?us-ascii?Q?dUIobM8guUB8R1vucsFKI4vQGCi66q2lrd4Yi/WFwBycM1B+Cgi7yog5ir+Y?=
 =?us-ascii?Q?cvYJRSoo/sw439GC1vodVNVUJaADTBgB+WJybOsBF5i4kejax0zjqG+lc4SH?=
 =?us-ascii?Q?5XJ5Jnmc0faOSGeuVtTvGE5SSGAvoFvNqCJYRl/eRy/attGCul2G0DR31ErP?=
 =?us-ascii?Q?YBCLJB0CxdXgzPCM//NxDZoBLFF7BEtMm1fTDf/eld4T1xXkRRSUmf2a1uPx?=
 =?us-ascii?Q?aJe5MlWPT5WjG5mcvO6Gfm0Sfl9dUOGoEAyst0cAWXXt98qmIVlbS7H/p5Is?=
 =?us-ascii?Q?i3w7nzRd0beY63lX74bDg66kg6zbEdYSMuXBfQld59OSabbtP4HUFwXdAQDM?=
 =?us-ascii?Q?IO0uGFcwulKO9AHGVH6RCb66BDmG7isgWs+uKeepk9ueF+4Mgqdm+9Av/y/Q?=
 =?us-ascii?Q?8bDseVoUrmCP/+t0MgswQ39DSVNfoMohS0Z8mCv5YkE5HOga51ukgLhrKa0N?=
 =?us-ascii?Q?asqM4DQl+DqR1/QDsQwHVl3EZ0gQwd7tBtKNoO42phcDdHrslA+12uVAMgDT?=
 =?us-ascii?Q?JGF1T8hW+OMMUMeFeSXSyf4kXkezi92O8I+80znQIPmn+//27quTsum1jD2O?=
 =?us-ascii?Q?x0a+eL/sZjZT7aTAVMA1zds5Tu7LTL0Ts/KM5121b3uxk/kpWAYFr8cvkh3t?=
 =?us-ascii?Q?jS7FQnjWhNfRuCAzAoT16IXc7H83PDHwCxIDPVKFKutrUoAFbngCt/YL4KsY?=
 =?us-ascii?Q?4aeZuGWpaESEY6Dl7onRHaTfxN2D5l7FJ26lkzWmO2KIPgzCkrihswORIsXk?=
 =?us-ascii?Q?2+KkWs5BF7G9MWCcG/Nqq4kT6/EBpnaTU7vKCj+Ecrv645AQRwT1DGNhofGB?=
 =?us-ascii?Q?bEcrFN6XFzM04wZyGzt0tmS4yz5CURIgecHlHFteAXu0kmYGX9Wo8j5SMQK4?=
 =?us-ascii?Q?aGdjFXR1vheGK69wFXgbA3OdZWk7nn0cu6TXIwcemJWNDClPVBsjqM0f7M3H?=
 =?us-ascii?Q?LuivKc+YhhLQztfa/dOFQO4qYNSuGHJBPzv1OxHE1R4QZ0rJ2Cxl5tZSZxzK?=
 =?us-ascii?Q?X3lCAxP1Fq6xWjwoNp+z/aubMSgFxR92YYDSnYfq/PJdMD9nij4b/riwiefP?=
 =?us-ascii?Q?5L3bCnXFagYl0nTQuRuEmbtkhazO1TnDjQvxs2n5ffuLXSXfMwsmK26RPK6y?=
 =?us-ascii?Q?kzDcnqpPUzMhK25zavAOerEbYfnle3Sds6w0Fw84?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 13c7ceaa-d796-440f-c877-08db6e042681
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3062.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jun 2023 00:53:50.9941
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 0/LpiZbDH9kSeXNY5FYmldiVhUCKny/c8GF9W8VPpJ8Wh5zFHuayo7YIPQr6yGQWYU4J0xZj16EDvnlwS5sM/A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY1PR11MB8056
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Jun 15, 2023 at 06:26:05PM -0500, David Vernet wrote:
 
> Ok, it seems that the issue is that I wasn't creating enough netperf
> clients. I assumed that -n $(nproc) was sufficient. I was able to repro

Yes that switch is confusing.

> the contention on my 26 core / 52 thread skylake client as well:
> 
> 
 
> Thanks for the help in getting the repro on my end.

You are welcome.

> So yes, there is certainly a scalability concern to bear in mind for
> swqueue for LLCs with a lot of cores. If you have a lot of tasks quickly
> e.g. blocking and waking on futexes in a tight loop, I expect a similar
> issue would be observed.
> 
> On the other hand, the issue did not occur on my 7950X. I also wasn't

Using netperf/UDP_RR?

> able to repro the contention on the Skylake if I ran with the default
> netperf workload rather than UDP_RR (even with the additional clients).

I also tried that on the 18cores/36threads/LLC Skylake and the contention
is indeed much smaller than UDP_RR:

     7.30%     7.29%  [kernel.vmlinux]      [k]      native_queued_spin_lock_slowpath

But I wouldn't say it's entirely gone. Also consider Skylake has a lot
fewer cores per LLC than later Intel servers like Icelake and Sapphire
Rapids and I expect things would be worse on those two machines.

> I didn't bother to take the mean of all of the throughput results
> between NO_SWQUEUE and SWQUEUE, but they looked roughly equal.
> 
> So swqueue isn't ideal for every configuration, but I'll echo my
> sentiment from [0] that this shouldn't on its own necessarily preclude
> it from being merged given that it does help a large class of
> configurations and workloads, and it's disabled by default.
> 
> [0]: https://lore.kernel.org/all/20230615000103.GC2883716@maniforge/

I was wondering: does it make sense to do some divide on machines with
big LLCs? Like converting the per-LLC swqueue to per-group swqueue where
the group can be made of ~8 cpus of the same LLC. This will have a
similar effect of reducing the number of CPUs in a single LLC so the
scalability issue can hopefully be fixed while at the same time, it
might still help some workloads. I realized this isn't ideal in that
wakeup happens at LLC scale so the group thing may not fit very well
here.

Just a thought, feel free to ignore it if you don't think this is
feasible :-)

Thanks,
Aaron