Received-SPF: pass (google.com: domain of linux-kernel+bounces-25333-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
From: Michael Kelley <mhklinux@outlook.com>
To: Souradeep Chakrabarti <schakrabarti@linux.microsoft.com>, Haiyang Zhang
	<haiyangz@microsoft.com>
CC: Yury Norov <yury.norov@gmail.com>, KY Srinivasan <kys@microsoft.com>,
	"wei.liu@kernel.org" <wei.liu@kernel.org>, Dexuan Cui <decui@microsoft.com>,
	"davem@davemloft.net" <davem@davemloft.net>, "edumazet@google.com"
	<edumazet@google.com>, "kuba@kernel.org" <kuba@kernel.org>,
	"pabeni@redhat.com" <pabeni@redhat.com>, Long Li <longli@microsoft.com>,
	"leon@kernel.org" <leon@kernel.org>, "cai.huoqing@linux.dev"
	<cai.huoqing@linux.dev>, "ssengar@linux.microsoft.com"
	<ssengar@linux.microsoft.com>, "vkuznets@redhat.com" <vkuznets@redhat.com>,
	"tglx@linutronix.de" <tglx@linutronix.de>, "linux-hyperv@vger.kernel.org"
	<linux-hyperv@vger.kernel.org>, "netdev@vger.kernel.org"
	<netdev@vger.kernel.org>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "linux-rdma@vger.kernel.org"
	<linux-rdma@vger.kernel.org>, Souradeep Chakrabarti
	<schakrabarti@microsoft.com>, Paul Rosswurm <paulros@microsoft.com>
Subject: RE: [PATCH 3/4 net-next] net: mana: add a function to spread IRQs per
 CPUs
Thread-Topic: [PATCH 3/4 net-next] net: mana: add a function to spread IRQs
 per CPUs
Thread-Index:
 AQHaQuoWNi7Bv1N390Ch4p7DlTYrcLDR1sMggABKpoCAAAdtgIAB++CAgAI7qHCAACS1AIAAySQAgACjPeA=
Date: Sat, 13 Jan 2024 16:20:31 +0000
Message-ID:
 <SN6PR02MB4157372CF70059E8E35D5545D46E2@SN6PR02MB4157.namprd02.prod.outlook.com>
References:
 <1704797478-32377-1-git-send-email-schakrabarti@linux.microsoft.com>
 <1704797478-32377-4-git-send-email-schakrabarti@linux.microsoft.com>
 <SN6PR02MB4157CB3CB55A17255AE61BF6D46A2@SN6PR02MB4157.namprd02.prod.outlook.com>
 <ZZ3Wsxq8rHShTUdA@yury-ThinkPad>
 <SN6PR02MB415704D36B82D5793CC4558FD4692@SN6PR02MB4157.namprd02.prod.outlook.com>
 <20240111061319.GC5436@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net>
 <SN6PR02MB4157234176238D6C1F35B90BD46F2@SN6PR02MB4157.namprd02.prod.outlook.com>
 <DS1PEPF00012A5F513F916690B9F94D3262CA6F2@DS1PEPF00012A5F.namprd21.prod.outlook.com>
 <20240113063038.GD5436@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net>
In-Reply-To:
 <20240113063038.GD5436@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0:
 =?us-ascii?Q?hAmHN/72lnZPuI9t1POPsUzCQGKVmGbW2CLITT0jOrYrCma1c4fV72fOkSc4?=
 =?us-ascii?Q?5wMwaYSlgfDAc4MHi8c6IhLh2lr9u/l4ok9Wmvc702TxSFnVfttCa9a5jFPU?=
 =?us-ascii?Q?7FYfl/Jqh/kE+3rUnPgGTpPDPwBHLnZp2gFJy5DPdQdKcfvTJtTyTamYEGJU?=
 =?us-ascii?Q?UB3k+nDONDdQ67gXqa7kE+hipEovTi+QBE3SIf0l/8/wM7HDjjPafQkM39QM?=
 =?us-ascii?Q?fD1L4eD9LWB8y2xVQiXWvPnxTt28qYrjpkA2eBEUPS7duQqAOLA+RiTRnNy6?=
 =?us-ascii?Q?/MScMOMTWVK+wxsvsdlroRoBi9Iglpg9AD9/nbDk7jIFUnD+Pelar5wDIVZr?=
 =?us-ascii?Q?F1huMQsoloOgKwDiXXmAH994pGEWVcvkF7UnsvGel9LqDGY5F9eTOo+muUv1?=
 =?us-ascii?Q?9D57U0TB4ApRqnYwzcKXongU0QOATFnXPRXqCa6/3B2SRnBam/CBu2novTIY?=
 =?us-ascii?Q?+yqe3q20aQkOtqirQOgfW3qvd03rbel1OgeuNjTS2jF2wN5o+t4BBTTi+sFY?=
 =?us-ascii?Q?9GXPaCd3r9hROLt5lJfINCGJpH4LBX3G/A5YM+3lJ6x1B/vpnPhZW4+lscQj?=
 =?us-ascii?Q?icCAF/n7mCUaahM07End9oFr4yYToTSnG2zJYqFPB9t5siyIWUhinBhe35Ir?=
 =?us-ascii?Q?AUM+CwLlTaHpRneR1Q7GugvP3Ntjwrs1d0M7O/rkDsCo9FS0JoOxLfKG3pSN?=
 =?us-ascii?Q?l1YMRaYTBhJPspmVYLiHPNSQ1byW9IarJGfIf8YPISlZAfHh38xKq8HgZ6RX?=
 =?us-ascii?Q?fqIePgvBaywMsnsL6J7mxH/9envTflC/Cj6TAP0h741g/7ZTYF2P0kL3Uuls?=
 =?us-ascii?Q?0a+eOj9NWgLRF76Uq7CldYbysdyEw+zckLDlm6kjoGObeG8ycnplzFTH2oYP?=
 =?us-ascii?Q?QOv3SR5RCiNmcNDcXsAvxnsoemgelUkTErMhGMLWroPckm6tpMPtiCSizdbK?=
 =?us-ascii?Q?hFDUuwqh/Ovp7943zNYlzTsA3hTpAzuBcbD25Mo2amLr+ZoFBNhkbRyymhz/?=
 =?us-ascii?Q?Ggc0au6XryPTiOPIdgrt/Boiew/UIib7soK1t4Z8ANWN/HI6n5wIUy8X0T3o?=
 =?us-ascii?Q?+/Ql2FM94ybdAweLhRf4+5mEd4LDHVNTqJ31TrSRiiGyxsjJE79QCt0qY2DU?=
 =?us-ascii?Q?Wn8LJvsqkhlHacMRlG6+f1WEEam3+pdSm3Vp3maAGPCMHy+PcXpCmQt1WrHZ?=
 =?us-ascii?Q?W/CymYu/NLzJ6uTtH91NBBoegVQ7ufuNk4AxZQ=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-OriginatorOrg: outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SN6PR02MB4157.namprd02.prod.outlook.com
X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000
X-MS-Exchange-CrossTenant-Network-Message-Id: 431a1d7c-ad08-4755-45c7-08dc1453907e
X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000
X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Jan 2024 16:20:31.8466
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CO6PR02MB8820

From: Souradeep Chakrabarti <schakrabarti@linux.microsoft.com> Sent: Friday=
, January 12, 2024 10:31 PM

> On Fri, Jan 12, 2024 at 06:30:44PM +0000, Haiyang Zhang wrote:
> >
> > > -----Original Message-----
> > From: Michael Kelley <mhklinux@outlook.com> Sent: Friday, January 12, 2=
024 11:37 AM
> > >
> > > From: Souradeep Chakrabarti <schakrabarti@linux.microsoft.com> Sent:
> > > Wednesday, January 10, 2024 10:13 PM
> > > >
> > > > The test topology was used to check the performance between
> > > > cpu_local_spread() and the new approach is :
> > > > Case 1
> > > > IRQ     Nodes  Cores CPUs
> > > > 0       1      0     0-1
> > > > 1       1      1     2-3
> > > > 2       1      2     4-5
> > > > 3       1      3     6-7
> > > >
> > > > and with existing cpu_local_spread()
> > > > Case 2
> > > > IRQ    Nodes  Cores CPUs
> > > > 0      1      0     0
> > > > 1      1      0     1
> > > > 2      1      1     2
> > > > 3      1      1     3
> > > >
> > > > Total 4 channels were used, which was set up by ethtool.
> > > > case 1 with ntttcp has given 15 percent better performance, than
> > > > case 2. During the test irqbalance was disabled as well.
> > > >
> > > > Also you are right, with 64CPU system this approach will spread
> > > > the irqs like the cpu_local_spread() but in the future we will offe=
r
> > > > MANA nodes, with more than 64 CPUs. There it this new design will
> > > > give better performance.
> > > >
> > > > I will add this performance benefit details in commit message of
> > > > next version.
> > >
> > > Here are my concerns:
> > >
> > > 1.  The most commonly used VMs these days have 64 or fewer
> > > vCPUs and won't see any performance benefit.
> > >
> > > 2.  Larger VMs probably won't see the full 15% benefit because
> > > all vCPUs in the local NUMA node will be assigned IRQs.  For
> > > example, in a VM with 96 vCPUs and 2 NUMA nodes, all 48
> > > vCPUs in NUMA node 0 will all be assigned IRQs.  The remaining
> > > 16 IRQs will be spread out on the 48 CPUs in NUMA node 1
> > > in a way that avoids sharing a core.  But overall the means
> > > that 75% of the IRQs will still be sharing a core and
> > > presumably not see any perf benefit.
> > >
> > > 3.  Your experiment was on a relatively small scale:   4 IRQs
> > > spread across 2 cores vs. across 4 cores.  Have you run any
> > > experiments on VMs with 128 vCPUs (for example) where
> > > most of the IRQs are not sharing a core?  I'm wondering if
> > > the results with 4 IRQs really scale up to 64 IRQs.  A lot can
> > > be different in a VM with 64 cores and 2 NUMA nodes vs.
> > > 4 cores in a single node.
> > >
> > > 4.  The new algorithm prefers assigning to all vCPUs in
> > > each NUMA hop over assigning to separate cores.  Are there
> > > experiments showing that is the right tradeoff?  What
> > > are the results if assigning to separate cores is preferred?
> >
> > I remember in a customer case, putting the IRQs on the same
> > NUMA node has better perf. But I agree, this should be re-tested
> > on MANA nic.
>
> 1) and 2) The change will not decrease the existing performance, but for
> system with high number of CPU, will be benefited after this.
>=20
> 3) The result has shown around 6 percent improvement.
>=20
> 4)The test result has shown around 10 percent difference when IRQs are
> spread on multiple numa nodes.

OK, this looks pretty good.  Make clear in the commit messages what
the tradeoffs are, and what the real-world benefits are expected to be.
Some future developer who wants to understand why IRQs are assigned
this way will thank you. :-)

Michael