Received-SPF: pass (google.com: domain of linux-kernel+bounces-24868-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
From: Michael Kelley <mhklinux@outlook.com>
To: Souradeep Chakrabarti <schakrabarti@linux.microsoft.com>
CC: Yury Norov <yury.norov@gmail.com>, "kys@microsoft.com"
	<kys@microsoft.com>, "haiyangz@microsoft.com" <haiyangz@microsoft.com>,
	"wei.liu@kernel.org" <wei.liu@kernel.org>, "decui@microsoft.com"
	<decui@microsoft.com>, "davem@davemloft.net" <davem@davemloft.net>,
	"edumazet@google.com" <edumazet@google.com>, "kuba@kernel.org"
	<kuba@kernel.org>, "pabeni@redhat.com" <pabeni@redhat.com>,
	"longli@microsoft.com" <longli@microsoft.com>, "leon@kernel.org"
	<leon@kernel.org>, "cai.huoqing@linux.dev" <cai.huoqing@linux.dev>,
	"ssengar@linux.microsoft.com" <ssengar@linux.microsoft.com>,
	"vkuznets@redhat.com" <vkuznets@redhat.com>, "tglx@linutronix.de"
	<tglx@linutronix.de>, "linux-hyperv@vger.kernel.org"
	<linux-hyperv@vger.kernel.org>, "netdev@vger.kernel.org"
	<netdev@vger.kernel.org>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "linux-rdma@vger.kernel.org"
	<linux-rdma@vger.kernel.org>, "schakrabarti@microsoft.com"
	<schakrabarti@microsoft.com>, "paulros@microsoft.com" <paulros@microsoft.com>
Subject: RE: [PATCH 3/4 net-next] net: mana: add a function to spread IRQs per
 CPUs
Thread-Topic: [PATCH 3/4 net-next] net: mana: add a function to spread IRQs
 per CPUs
Thread-Index: AQHaQuoWNi7Bv1N390Ch4p7DlTYrcLDR1sMggABKpoCAAAdtgIAB++CAgAI7qHA=
Date: Fri, 12 Jan 2024 16:36:56 +0000
Message-ID:
 <SN6PR02MB4157234176238D6C1F35B90BD46F2@SN6PR02MB4157.namprd02.prod.outlook.com>
References:
 <1704797478-32377-1-git-send-email-schakrabarti@linux.microsoft.com>
 <1704797478-32377-4-git-send-email-schakrabarti@linux.microsoft.com>
 <SN6PR02MB4157CB3CB55A17255AE61BF6D46A2@SN6PR02MB4157.namprd02.prod.outlook.com>
 <ZZ3Wsxq8rHShTUdA@yury-ThinkPad>
 <SN6PR02MB415704D36B82D5793CC4558FD4692@SN6PR02MB4157.namprd02.prod.outlook.com>
 <20240111061319.GC5436@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net>
In-Reply-To:
 <20240111061319.GC5436@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0:
 =?us-ascii?Q?z9vWkSB48jTRN+uChEGF7/OQ5QGd8WUmKsBE9nMitFAQyQxLIlNsTbk649al?=
 =?us-ascii?Q?Bmhge96MwjT1ViY1jp95eaP5mTiN5kuPvvZUX2C4MTs7PwtLt1APVl+gUPQc?=
 =?us-ascii?Q?iWnm9Z0iefVEdJWI0ZTGaCyRtox1xByGJDFVLOYdr3BTRv40aypI4LAPhXRg?=
 =?us-ascii?Q?saiU+xZMkIRKKmRPvvwXyaVV3UX+hLxuOck2/+0tvlZ8aDWfp4W5uEty8JwC?=
 =?us-ascii?Q?UreXzMlPx54alPJhjMF2oQ1EQBFAKylQy+KCoodxVhrBT0NiJ0gPhoIEkxTi?=
 =?us-ascii?Q?/Ny5CpV9yAub9kp1Hx/sGHsbQ+2bGZHgtnFvosT9t7mrCojuB02b60FyYjpd?=
 =?us-ascii?Q?jcO0U1s+S+ij2IkY0T4D/qiSYwVeeGQbt4wnPMs7fWPt58tfhRFGRTW1cF8A?=
 =?us-ascii?Q?kx5zC159AJQcH+M8Eb5Z/bNfTWrq4s7B4mADMHzqOp6Hebrnpx/Qss+C2lj5?=
 =?us-ascii?Q?5NFHl9GzIYry8oinoRHOUtDv4QuR8OtS401qwoKs6Wcan47YyaX0DF9txfLr?=
 =?us-ascii?Q?fxYpMhGfdlQSKFph2lSSxzG0ZITfGnqvhEL7uRGVms3Agka437N8D0dl0h4A?=
 =?us-ascii?Q?ULiN/D9GEF9LzTVZkk+uFweEH6fKncjq6+J9FPRkBQjADLm5XoVKHJxjfMeO?=
 =?us-ascii?Q?G06SnLjbWaZaTtZBOL4nkuL3Kvio4erwTOvTCDqxcqlHpeIzvRfPgAxuWA1C?=
 =?us-ascii?Q?Ro/ul6NyOpAqVibtwlabr6fxK8G5rSoUCdMubD/Vssjlh5is2prHmuO0fso4?=
 =?us-ascii?Q?XngT1sPYQbyPou8UdCrNs4aZgGhcva/Fk+nLjL39W06gkZX6cLOO75T2J+dO?=
 =?us-ascii?Q?ebkdOyXRUToPRiVwZci3RIufYd6BCuxx6fU0nylkCmfavSSQE9wnf1jzh6Yw?=
 =?us-ascii?Q?plw9IhCnelGMFThzFKpgUHoBeTLeUel6t4lX9+OYcDSxmBZUA3DMWjc33CIM?=
 =?us-ascii?Q?pun0op4z5Gao4hcdsURtqFjiPZMUDSAJExSjtXLY0m4T0abZ8WZZFlnH9OZH?=
 =?us-ascii?Q?KUof4TIo/kq3xMZzsqwC/NDhr+4rFne4PGGT25XKiV4ePkDXzIHgMFBm9M/g?=
 =?us-ascii?Q?3wdx1DcUcDuRKfxlneAPpYB1+VZVNmWPtxGzPkL4vaqKTuMQjflvqCDUX3hu?=
 =?us-ascii?Q?O/Io4c5FNKLMdI0kn/FMBUJkvIQmbcC0uLM6wAelNt9IzuQfikxEWQWJ3BHw?=
 =?us-ascii?Q?G+nxF+JEOl+pP8NwGZdQIePfRP+ucHOCjtMThQ=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-OriginatorOrg: outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SN6PR02MB4157.namprd02.prod.outlook.com
X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000
X-MS-Exchange-CrossTenant-Network-Message-Id: dd336f3a-36cc-4771-4df0-08dc138cb13e
X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000
X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Jan 2024 16:36:56.9753
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR02MB8429

From: Souradeep Chakrabarti <schakrabarti@linux.microsoft.com> Sent: Wednes=
day, January 10, 2024 10:13 PM
>=20
> The test topology was used to check the performance between
> cpu_local_spread() and the new approach is :
> Case 1
> IRQ     Nodes  Cores CPUs
> 0       1      0     0-1
> 1       1      1     2-3
> 2       1      2     4-5
> 3       1      3     6-7
>=20
> and with existing cpu_local_spread()
> Case 2
> IRQ    Nodes  Cores CPUs
> 0      1      0     0
> 1      1      0     1
> 2      1      1     2
> 3      1      1     3
>=20
> Total 4 channels were used, which was set up by ethtool.
> case 1 with ntttcp has given 15 percent better performance, than
> case 2. During the test irqbalance was disabled as well.
>=20
> Also you are right, with 64CPU system this approach will spread
> the irqs like the cpu_local_spread() but in the future we will offer
> MANA nodes, with more than 64 CPUs. There it this new design will
> give better performance.
>=20
> I will add this performance benefit details in commit message of
> next version.

Here are my concerns:

1.  The most commonly used VMs these days have 64 or fewer
vCPUs and won't see any performance benefit.

2.  Larger VMs probably won't see the full 15% benefit because
all vCPUs in the local NUMA node will be assigned IRQs.  For
example, in a VM with 96 vCPUs and 2 NUMA nodes, all 48
vCPUs in NUMA node 0 will all be assigned IRQs.  The remaining
16 IRQs will be spread out on the 48 CPUs in NUMA node 1
in a way that avoids sharing a core.  But overall the means
that 75% of the IRQs will still be sharing a core and
presumably not see any perf benefit.

3.  Your experiment was on a relatively small scale:   4 IRQs
spread across 2 cores vs. across 4 cores.  Have you run any
experiments on VMs with 128 vCPUs (for example) where
most of the IRQs are not sharing a core?  I'm wondering if
the results with 4 IRQs really scale up to 64 IRQs.  A lot can
be different in a VM with 64 cores and 2 NUMA nodes vs.
4 cores in a single node.

4.  The new algorithm prefers assigning to all vCPUs in
each NUMA hop over assigning to separate cores.  Are there
experiments showing that is the right tradeoff?  What
are the results if assigning to separate cores is preferred?

My bottom line:  The new algorithm adds complexity.  We
should be really clear on where it adds performance benefit
and where it doesn't, and be convinced that the benefit
is worth the extra complexity.

Michael