Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   David Chen <david.chen@nutanix.com>
To:     "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
CC:     Ingo Molnar <mingo@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>
Subject: Perf regression from scheduler load_balance rework in 5.5?
Thread-Topic: Perf regression from scheduler load_balance rework in 5.5?
Thread-Index: AdiHOIYtzTENFRVgRIe5P/iKHm6Zzg==
Date:   Thu, 23 Jun 2022 19:50:47 +0000
Message-ID: <BYAPR02MB4488F89EC5DB73D1FEAE9B4594B59@BYAPR02MB4488.namprd02.prod.outlook.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?fEi7BQ95Mro+CwAf2/i5AUq1WUvDD7e7cv1tzC9XqUFsTPpUWZGAMutQ4WWi?=
 =?us-ascii?Q?LCLrzKFm0QvQj3qicrkEopHdmllBhJcznguHgGB/PM+ffS15yPxPPsXBkxtS?=
 =?us-ascii?Q?qqORONYb24i82jA39vg7QVxkOtvTRaZLmbbUJYgj+A5fQSH97pWZiQiLL6Za?=
 =?us-ascii?Q?OozW74iz3/R/+p8E4neUOEABL3c1nI+vnCdP3ysiyAswR+PBWY7IdlNphc4k?=
 =?us-ascii?Q?Orgw55ujxrYOpzJTBHzyWLxSKe5bva3KLxsZQc1nWEBBiqVeljcm7MjlA+KZ?=
 =?us-ascii?Q?Cs4roc6H/eA8B68BruKUbMyTIlZgQuQmg6ema2dxAdCiRM11vcwgZ6+yUauI?=
 =?us-ascii?Q?3mkxjQyY86QFQzKKrsw39yfZRjYV7HXPzZqBF6ZPLf9qY6o3Iqk/xRX91hQL?=
 =?us-ascii?Q?j7x3tNrPzJnL/SYoqxtAVf/VU3GBt9d8ANpS6ZLmdWpwCNwz9Y9ST1vx6TZG?=
 =?us-ascii?Q?IbErV54VOeHRi5TgUoOwGNmpDmx3pRESpirL2ZDsQ9hVE2dozpuO313waUpN?=
 =?us-ascii?Q?QA3s2BIQ2Chy7jKnGkFDc5KYB/hoUPm118cKENe/YIjTsLLXJkcuJGzhaD+U?=
 =?us-ascii?Q?RCxRpUEFpN1IXS0aiadAzB2JOTVt2XEiBwoelwg39wb50yNQ8U/uSVuEhLJk?=
 =?us-ascii?Q?W/MktPkAY908tCgZEdgJEf3Fvwe+YVAoHjSVML9ZCrUmQwrTh5skEUDh9qgx?=
 =?us-ascii?Q?vVxvcrHYHdOfU0YNsqzc6rmL6bAzw1KkJu3C/Sam0JG1EUMxER9ITTHcbxnC?=
 =?us-ascii?Q?5zOR/gaxvKDFulSSphpO/O88ckNFbgtDFfC9BKBRfrYQmq4usuC8SGCNBodb?=
 =?us-ascii?Q?ml9xIJkI3tdnclt2H/tGzSsbARB0Ntx1FOAMI40L9iko02djnSmaNn9i8/kH?=
 =?us-ascii?Q?/R/pxmAzWwOIPERe8Pxec7gVCvev288zk0pV39FchEVozjLICICE2upsvAOg?=
 =?us-ascii?Q?iLkfFuQstKasd793yvysjdMPbYiv6nmDI5zMjXpPB9U2ZZKJ8gnuyrQcBiJ0?=
 =?us-ascii?Q?rvzrlSa4loSCL4VAvMFnqhJ6fOA78Hax9g90q56N5YIouZOszpAlgIT1dHX9?=
 =?us-ascii?Q?i7DBUau7fspi2TVRVaPZ3fXtfAvzLuFACKlmul1vdF+ZVrsZXJVhNamJdb48?=
 =?us-ascii?Q?pVx9fgT0XyekIPkmijC/BTUCWcbG86vamBvoXgwwS5nUhT+W797YnfwCJ+Vd?=
 =?us-ascii?Q?u8dq+P80wYiMtf3vYqdRRstYsfjD96sKxeezi7QbpNGrnpZEwagXInVWLFka?=
 =?us-ascii?Q?Vf4K434mqAzJRKeIeA3sKG3/GzoglxRbt/K4I2wFvJbz0oc+9NBtl535pu3Y?=
 =?us-ascii?Q?wNiYB7ScHmbb7FbsFnJcqiO7Q7zPX1fDiCF/AGsGmWhX87+kl0bP/CMG6NeK?=
 =?us-ascii?Q?ZZ6UkCBZCqoWAwKCVMBJQZf6Y+WWIaiilN8n6i/Jik3UanG2K1p8n/ooq6GF?=
 =?us-ascii?Q?R27PyspRt7BlS6zq1woumySbOdpJ1iklXwvK5S87z4aKvZ6mCIR9vpumlhF9?=
 =?us-ascii?Q?l0jtRgF7Qzt77J4NX0DNfB2ZO2Rj6up4eGQ//3/MRTX2G64EzCUzc6uIAObJ?=
 =?us-ascii?Q?0eZDve1iQaHpuo7LlI6gzza1Yl9IZIEmzMqbR0MY?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: nutanix.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BYAPR02MB4488.namprd02.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b50ba5f4-5b0f-4615-bc3d-08da5551ab12
X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Jun 2022 19:50:47.6721
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: bb047546-786f-4de1-bd75-24e5b6f79043
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: XkivkbTZwjWVnDYntBvX7TgJon+bwvYyeEJq490Xo8o2RbXh030K8zFyqm4S/cG4ccbRpVtoW5b6ZL1owCHo+Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR02MB6529
X-Proofpoint-GUID: QSXxenyU7o3OhtPZPk7nRYvrYzje64jO
X-Proofpoint-ORIG-GUID: QSXxenyU7o3OhtPZPk7nRYvrYzje64jO
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1
 definitions=2022-06-23_08,2022-06-23_01,2022-06-22_01
X-Proofpoint-Spam-Reason: safe
X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,
        SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi,

I'm working on upgrading our kernel from 4.14 to 5.10
However, I'm seeing performance regression when doing rand read from window=
s client through smbd
with a well cached file.

One thing I noticed is that on the new kernel, the smbd thread doing socket=
 I/O tends to stay on
the same cpu core as the net_rx softirq, where as in the old kernel it tend=
s to be moved around
more randomly. And when they are on the same cpu, it tends to saturate the =
cpu more and causes
performance to drop.

For example, here's the duration (ns) the thread spend on each cpu I captur=
ed using bpftrace
On 4.14:
@cputime[7]: 20741458382
@cputime[0]: 25219285005
@cputime[6]: 30892418441
@cputime[5]: 31032404613
@cputime[3]: 33511324691
@cputime[1]: 35564174562
@cputime[4]: 39313421965
@cputime[2]: 55779811909 (net_rx cpu)

On 5.10:
@cputime[3]: 2150554823
@cputime[5]: 3294276626
@cputime[7]: 4277890448
@cputime[4]: 5094586003
@cputime[1]: 6058168291
@cputime[0]: 14688093441
@cputime[6]: 17578229533
@cputime[2]: 223473400411 (net_rx cpu)

I also tried setting the cpu affinity of the smbd thread away from the net_=
rx cpu and indeed that
seems to bring the perf on par with old kernel.

I noticed that there's scheduler load_balance rework in 5.5, so I did the t=
est on 5.4 and 5.5 and
it did show the behavior changed between 5.4 and 5.5.

Anyone know how to work around this?

Thanks,
David