Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 21 Jun 2023 10:35:34 +0800
From:   Aaron Lu <aaron.lu@intel.com>
To:     David Vernet <void@manifault.com>
CC:     Peter Zijlstra <peterz@infradead.org>,
        <linux-kernel@vger.kernel.org>, <mingo@redhat.com>,
        <juri.lelli@redhat.com>, <vincent.guittot@linaro.org>,
        <rostedt@goodmis.org>, <dietmar.eggemann@arm.com>,
        <bsegall@google.com>, <mgorman@suse.de>, <bristot@redhat.com>,
        <vschneid@redhat.com>, <joshdon@google.com>,
        <roman.gushchin@linux.dev>, <tj@kernel.org>, <kernel-team@meta.com>
Subject: Re: [RFC PATCH 3/3] sched: Implement shared wakequeue in CFS
Message-ID: <20230621023534.GA236337@ziqianlu-dell>
References: <20230613052004.2836135-1-void@manifault.com>
 <20230613052004.2836135-4-void@manifault.com>
 <20230613083203.GR4253@hirez.programming.kicks-ass.net>
 <20230614043529.GA1942@ziqianlu-dell>
 <20230615000103.GC2883716@maniforge>
 <20230615044917.GA109334@ziqianlu-dell>
 <20230615073153.GA110814@ziqianlu-dell>
 <20230615232605.GB2915572@maniforge>
 <20230616005338.GA115001@ziqianlu-dell>
 <20230620173626.GA3027191@maniforge>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230620173626.GA3027191@maniforge>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?jGtM7R3w5QT7XsG67pKReFEMZS3b4ydTgO1Vn9ZtF1PBbkg5jNiYObIfaKcX?=
 =?us-ascii?Q?0n57BjzwMaYOwmCC/vr3jGI0c472BPOEmkde9IiCcsyO4BCbZ84tRgzWnvP4?=
 =?us-ascii?Q?qm4pLOyLayvpJOpJmUILl2NT3kY2abH/ebEYBBD94mTHFLw+Ix6UqQ97Pwj3?=
 =?us-ascii?Q?AtZEuCbg74wwTXQ0GEgR1hFSwnnKPYVM4iVyp3t8W0gwiVL3jFk549A94Tue?=
 =?us-ascii?Q?GdWc+dOb00+fyi1PYKxDPxwZVMBciyaQAiPpOzeFZrFMi4sGRwD9se//PHKK?=
 =?us-ascii?Q?Wlh1jHDPRy2+NH0n5dLyjcHYQJHeZqm1P5E9w1iR86Qo+syiY0G062J/m9ah?=
 =?us-ascii?Q?WNv1pZLsl6gaURh+eAVCk9jNzIje7RygfCKIzN70TF54se6UecC/mnKgu/Zx?=
 =?us-ascii?Q?q0Rozykxrekrc/ukeLgi6P9jxh9jqRjSTBwrBPUrdF/jiANjydPsaFyWrT+h?=
 =?us-ascii?Q?1t+cjue9TRSst/V9E1h0VhJqUIHLBhffHtjIqtJhMwBvNkHIXDDMKGCeCsag?=
 =?us-ascii?Q?erTOPCxG2s6GR6D8PzEGEkVslPWUCcmHFBp19oZ5/AWLSHLUzeMVG/KJLrcK?=
 =?us-ascii?Q?h6AzSYTdwX446yCsh1GytB5szZRFw8o3BSG/wKNHypi98sUcO4VekACFBPM8?=
 =?us-ascii?Q?DVuiyBj4SgMB8veFhzKVaGyimwSp34BQXNtGsDYF2BsEb5Z4K6hrTBtYhZW6?=
 =?us-ascii?Q?x3o3hldY2KnCyn17ziN00Nb31FMPR5LtL3o2ZuxC1gP4LiPknL+mCH5LVCQW?=
 =?us-ascii?Q?ou+Kov8nDQyAMyZCJLpiVTNNRCU6SN/JeWXQE+q+uTo/m/zQUgzatLmd6jwy?=
 =?us-ascii?Q?fTGe1OzXQU+ICQ7VgGlcKLRIBrXGPyiiM6DSY+8WnF7tNey6KOv6IHfuak/r?=
 =?us-ascii?Q?zvaj+spS6JDY5y9k5h/8jhi1YxCrQpcGKCnCd6ZUFnEBi+pRnvhI08BFTuAJ?=
 =?us-ascii?Q?o0A8e+L/aBqywETCXcGtNMqKzpaxNEhDp4j5EAbcHy7eK7mHoRfNhjoiF85Q?=
 =?us-ascii?Q?iwU8LcPvVZTnCSOpIg9n3xR4+Qsv8c3XiT8xsPldEnXGRkR+PbOw0cKpReBh?=
 =?us-ascii?Q?6lXf75eGbtGUyztAM4uSXUqutuPFSm/9q0Di5yNkTrsZmVvOE0HVYAfFQabB?=
 =?us-ascii?Q?lwOZlRMtFpYazgUWv1gNzlQsloeH7QGcfi+fSK4J6K8Xv9Sx9rMAdJTQUBpG?=
 =?us-ascii?Q?/ptOA5gwkC9DGPkFjKYgIkjvp91DkyoRDtmoYKwtgabvDEEchY1snfpI7P91?=
 =?us-ascii?Q?5IzCClopB+b6eBKcc5wTR24HPB2qU0qxxo8lv2xknD7Rzt2zxm66OpWM6y3G?=
 =?us-ascii?Q?Zl+9Jz9JpigkvUfz6U4CYZnSAxwYMamObsUOjR2X8gyWkxax4KhZE+yyN++8?=
 =?us-ascii?Q?sIbW5mrmwkvuZvqcnbjCTWEKelPBIFTwBtdNaTp0KCKWSZtLlCvPsDedwwcC?=
 =?us-ascii?Q?LOfsGDnyAXpLD2YHtcXvM3WCG3ujEMtyKDs3CfQv0b352ovfOUQpXBaSvqJe?=
 =?us-ascii?Q?Sj49V85iqPc7jJN8cvWVs1qxtF2ThA3Hnk4c3vMnoIfpQsGyMhM9MMU+PeEL?=
 =?us-ascii?Q?jBrwETkE4IL01w8beKXKkUtbLhf6K6s24Ve8+2zz?=
X-MS-Exchange-CrossTenant-Network-Message-Id: de642b92-2898-4b61-6623-08db7200368b
X-MS-Exchange-CrossTenant-AuthSource: BYAPR11MB3062.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jun 2023 02:35:45.0021
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 6PbmMYM9HqRHxpD8qN2YJNOJHGHBrRqT7ictPLex7hsqDeu/NPjX6CrugZ48d+TunpohQ0cR4RipjXATYubeew==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR11MB8401
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Jun 20, 2023 at 12:36:26PM -0500, David Vernet wrote:
> On Fri, Jun 16, 2023 at 08:53:38AM +0800, Aaron Lu wrote:
> > I also tried that on the 18cores/36threads/LLC Skylake and the contention
> > is indeed much smaller than UDP_RR:
> > 
> >      7.30%     7.29%  [kernel.vmlinux]      [k]      native_queued_spin_lock_slowpath
> > 
> > But I wouldn't say it's entirely gone. Also consider Skylake has a lot
> > fewer cores per LLC than later Intel servers like Icelake and Sapphire
> > Rapids and I expect things would be worse on those two machines.
> 
> I cannot reproduce this contention locally, even on a slightly larger

With netperf client number equal to nr_cpu?

> Skylake. Not really sure what to make of the difference here. Perhaps
> it's because you're running with CONFIG_SCHED_CORE=y? What is the

Yes I had that config on but I didn't tag any tasks or groups.

> change in throughput when you run the default workload on your SKL?

The throughput dropped a little with SWQUEUE:

                 avg_throughput    native_queued_spin_lock_slowpath%
NO_SWQUEUE:      9528.061111111108      0.09%
SWQUEUE:         8984.369722222222      8.05%

avg_throughput: average throughput of all netperf client's throughput,
higher is better.

I run this workload like this:
"
netserver

for i in `seq 72`; do
        netperf -l 60 -n 72 -6 &
done

sleep 30
perf record -ag -e cycles:pp -- sleep 5 &

wait
"
(the '-n 72' should be redundant but I just keep it there)

Thanks,
Aaron