Received-SPF: pass (google.com: domain of linux-kernel+bounces-103696-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
From: Haiyang Zhang <haiyangz@microsoft.com>
To: Jakub Kicinski <kuba@kernel.org>, Shradha Gupta
	<shradhagupta@linux.microsoft.com>
CC: Shradha Gupta <shradhagupta@microsoft.com>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "linux-hyperv@vger.kernel.org"
	<linux-hyperv@vger.kernel.org>, "linux-rdma@vger.kernel.org"
	<linux-rdma@vger.kernel.org>, "netdev@vger.kernel.org"
	<netdev@vger.kernel.org>, Eric Dumazet <edumazet@google.com>, Paolo Abeni
	<pabeni@redhat.com>, Ajay Sharma <sharmaajay@microsoft.com>, Leon Romanovsky
	<leon@kernel.org>, Thomas Gleixner <tglx@linutronix.de>, Sebastian Andrzej
 Siewior <bigeasy@linutronix.de>, KY Srinivasan <kys@microsoft.com>, Wei Liu
	<wei.liu@kernel.org>, Dexuan Cui <decui@microsoft.com>, Long Li
	<longli@microsoft.com>, Michael Kelley <mikelley@microsoft.com>, Alireza
 Dabagh <alid@microsoft.com>, Paul Rosswurm <paulros@microsoft.com>
Subject: RE: [PATCH] net :mana : Add per-cpu stats for MANA device
Thread-Topic: [PATCH] net :mana : Add per-cpu stats for MANA device
Thread-Index:
 AQHacJ8N8LLwvYYKc0y/h+tCzJvDjrEsZziAgAADR7CAABaHgIABr8IAgAAJ9wCAA7q6AIAEn/IAgAED5wCAAAUH8A==
Date: Thu, 14 Mar 2024 18:54:31 +0000
Message-ID:
 <DM6PR21MB14819A8CDB1431EBF88216ABCA292@DM6PR21MB1481.namprd21.prod.outlook.com>
References:
 <1709823132-22411-1-git-send-email-shradhagupta@linux.microsoft.com>
	<20240307072923.6cc8a2ba@kernel.org>
	<DM6PR21MB14817597567C638DEF020FE3CA202@DM6PR21MB1481.namprd21.prod.outlook.com>
	<20240307090145.2fc7aa2e@kernel.org>
	<CH2PR21MB1480D3ACADFFD2FC3B1BB7ECCA272@CH2PR21MB1480.namprd21.prod.outlook.com>
	<20240308112244.391b3779@kernel.org>
	<20240311041950.GA19647@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net>
	<20240314025720.GA13853@linuxonhyperv3.guj3yctzbm1etfxqx2vob5hsef.xx.internal.cloudapp.net>
 <20240314112734.5f1c9f7e@kernel.org>
In-Reply-To: <20240314112734.5f1c9f7e@kernel.org>
Accept-Language: en-US
Content-Language: en-US
msip_labels:
 MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=bb4edda4-4af1-4b15-849c-2e3e398058d1;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2024-03-14T18:45:33Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47;
x-ms-publictraffictype: Email
x-ms-traffictypediagnostic: DM6PR21MB1481:EE_|LV2PR21MB3372:EE_
x-ms-exchange-senderadcheck: 1
x-ms-exchange-antispam-relay: 0
x-microsoft-antispam: BCL:0;
x-microsoft-antispam-message-info:
 A4t634CTNYeHqrNhRoy3kolYbO+BEPvFcQ8pIRAdwkMAfghg28fuHb8x4XnMlApsEp6vn9gDPG7O0c4mREiSioLlNcys/MTBLD4jpwVMPjHZEP79/zHA1s3j1L33q65sU5U8/tHM/ZHzkww8PH5CUt9rm2uAL5AzCifUYniIdlK2hMUIFOaHJslOO51kNcJ5ByljrPe4rPbjixoVdGl334FO7UxdHLy8TntMmtKkd+YyuWS0zQaded5jZnwr4SSp2A3R4KBUrGL1+VqkKcn7eRqY2/K/9kNcX8yBB46Wxc1REepNfwto41dEDIMf9EkoYtWAbpgyxgDqsv/K0vfZcCB/zvQ0yDEMVgv048jBx/mkrErQSI+Z+uJ9qrgxpe9NHI8XSsVXOsw7kb8MKBnBYCjKyB5l0qWeTyj8Zfh10/WJzYv9iAx9mhoLNcDVfYIbKUshyuNuiBX8Ieeh8n22yYS+DRzqJA/fX6iUD1++vqFbKZ2NbtXpEVe5fLmB6hkgVmY3oUNfpYSumNussvS93hhjQ3fCiAN13cW+nJJsjCxIjHQnsHVLAYnIqEyznU2I+d7OIh2v9zcvslWk56T6ta1CpRbbBSysjah5comcw2SFfyC//SSuak3ZxzGHRGeVh9eBhf01jeendtxursmI8JYyBYgwtzvBALkmSV6CgQQ=
x-forefront-antispam-report:
 CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR21MB1481.namprd21.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(376005)(7416005)(1800799015);DIR:OUT;SFP:1102;
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0:
 =?us-ascii?Q?GCwL7k7Ohjx7EnicFYpWDSUDNQj/oTvssEenpBpbFIhfO+5RYAC6DknWs4w8?=
 =?us-ascii?Q?D0mB8cKCUvxGqbIayDxKjFq272ME9Hgz3MwtOLcx3QLbEg4FJvG8iYBedTue?=
 =?us-ascii?Q?VYPwYmbMISfTkv409aBBSOlX6PeclPAmfu8Ir0BqAlKnNa9Ugj5zsvIKjTcW?=
 =?us-ascii?Q?M5/bXW8ASTafhPDLUNLa0zjYawcu9xdE0vJ7U+dGz2HgJW9ljhirhnAckdD4?=
 =?us-ascii?Q?grVzCkOc+JPAb1cRY6W7JFnQtEyEgGbB0DDf+SE5yUh4Xa00APgiUU3oQ3vG?=
 =?us-ascii?Q?hbqlMUTLylR00KUiArFBdmf4MLuFb8VfF1x/mGmTmxWwmkiJEJhf+3Oruq7T?=
 =?us-ascii?Q?pBUNsWvi6x421Q24DJzDtYErdVJ6DkjJkU3MLNpN888Iwe3NSCEpjLp9NbKA?=
 =?us-ascii?Q?iN1nLlH45dwbxG2b/XmKL4aKVoqZRYIfXypaToBh3IvDE1BnYwu6+icOp40y?=
 =?us-ascii?Q?G7vrgYzZ2beb9TulBUuvxYEmoafelfKG46Eu1Rnxal6SYN6zuymzZ5HYklHs?=
 =?us-ascii?Q?ckCN5X82k5R2H64+kxLrMdyu1u834vdr+m7qeYeYLsmQnjkW8u5IyBzG4Mqf?=
 =?us-ascii?Q?t/dlci1VxIjHqmKETYPGTAHOfXDSQ51HP+gmG5YeT8FeodklZ5AbY+/RBtu5?=
 =?us-ascii?Q?LICXbNdo8C4ElXUpPq9fictTRAuujqw7i+kDNPBlNNjzuK+Q19fiDcZHg7Fg?=
 =?us-ascii?Q?+uYLtxytbmBoQqMRjr3Q3IS2xMbjN4Hd/gFJkjrRyw2W2bMadCksWUbmYEFX?=
 =?us-ascii?Q?TIpKYxT65OMBZOR38X7uPcigOBBPQ1Ls3RXx2GmauirZJ/praJPBMeTzGfLS?=
 =?us-ascii?Q?0RrNcjF9fuGaKSZwUlWyazNnI0jVCjpRSG9oVFWIHMP4Ar6bH01q6dnDoq8P?=
 =?us-ascii?Q?L/k3FeX0EedHlCfby7VCpNefpelWUoHQvkgQviDOvdFr+r/+XB7TveUxd6Xn?=
 =?us-ascii?Q?frdC21TUD/LrZEOVDLyAn3WMDeU3crvJz5Q0gqIL5OQxRGE+itboQ0nRxG2l?=
 =?us-ascii?Q?QeU+y6mC7WH03tiKXGYbatBXaWd45W7yZEatqC1luIwssJKR4AvgpAIkHuyK?=
 =?us-ascii?Q?oo87usiEYJ7F/YAJyUFEMnoBUpq1T6noWJpbfOZ2Yd4uNMb2dinj5gqk401U?=
 =?us-ascii?Q?jeVXQD4vQ08pBZcndkvyHON0MZ8Mh4r4Q9wga9V8TDR1JLAYEha+qNi4eBYZ?=
 =?us-ascii?Q?zs4zmmduAUDro/MowuqZ/MTOpta3JqmUhcCvBczh8bgOFChxVDS8+0SMt+sP?=
 =?us-ascii?Q?YWib6VLMF+3EWwKmN+cp5X88bnvJZ0myeLCgIzpyt403Q3VqCZJGEs9ZSylI?=
 =?us-ascii?Q?CxYOYfB6hNj+7qmRvWL6ghO7FtwciS1b2jTNVrpTczZ6blA12+ZRZHN3in6p?=
 =?us-ascii?Q?95Ss3IvrNGHv4cND3Evk+wtAFD1zsXQVMVyEWaX91rNxD0sLSYxWjeAR4RRp?=
 =?us-ascii?Q?fGKcRQk/ZRf4/hBz/ZXRsy5oGNfDYREcMIrNXQSHW8ZrHHwQoF7u+qT8HHSa?=
 =?us-ascii?Q?pdtwQbBY2xut+pQJwAAx5OfiQgTwWE0iXpLKTEdFdsCmO1Bl5Q51Z4f5WHaN?=
 =?us-ascii?Q?3W7PegiB9eJiJPPpo/TsA9D3S6MMSmONAnYBNkmu?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-OriginatorOrg: microsoft.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: DM6PR21MB1481.namprd21.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4ed0af63-2708-4175-9dc0-08dc44582f2c
X-MS-Exchange-CrossTenant-originalarrivaltime: 14 Mar 2024 18:54:31.8968
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 5gnjieGETaXlCuVK8A1d3ghydRZE2JBtwM7yfyFAestRXw6VcPodL+xGBnFCSimw15RTu8qbQC+fOOE5pHHi9Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR21MB3372


> -----Original Message-----
> From: Jakub Kicinski <kuba@kernel.org>
> Sent: Thursday, March 14, 2024 2:28 PM
> To: Shradha Gupta <shradhagupta@linux.microsoft.com>
> Cc: Haiyang Zhang <haiyangz@microsoft.com>; Shradha Gupta
> <shradhagupta@microsoft.com>; linux-kernel@vger.kernel.org; linux-
> hyperv@vger.kernel.org; linux-rdma@vger.kernel.org;
> netdev@vger.kernel.org; Eric Dumazet <edumazet@google.com>; Paolo Abeni
> <pabeni@redhat.com>; Ajay Sharma <sharmaajay@microsoft.com>; Leon
> Romanovsky <leon@kernel.org>; Thomas Gleixner <tglx@linutronix.de>;
> Sebastian Andrzej Siewior <bigeasy@linutronix.de>; KY Srinivasan
> <kys@microsoft.com>; Wei Liu <wei.liu@kernel.org>; Dexuan Cui
> <decui@microsoft.com>; Long Li <longli@microsoft.com>; Michael Kelley
> <mikelley@microsoft.com>
> Subject: Re: [PATCH] net :mana : Add per-cpu stats for MANA device
>=20
> On Wed, 13 Mar 2024 19:57:20 -0700 Shradha Gupta wrote:
> > Default interrupts affinity for each queue:
> >
> >  25:          1        103          0    2989138  Hyper-V PCIe MSI
> 4138200989697-edge      mana_q0@pci:7870:00:00.0
> >  26:          0          1    4005360          0  Hyper-V PCIe MSI
> 4138200989698-edge      mana_q1@pci:7870:00:00.0
> >  27:          0          0          1    2997584  Hyper-V PCIe MSI
> 4138200989699-edge      mana_q2@pci:7870:00:00.0
> >  28:    3565461          0          0          1  Hyper-V PCIe MSI
> 4138200989700-edge      mana_q3
> > @pci:7870:00:00.0
> >
> > As seen the CPU-queue mapping is not 1:1, Queue 0 and Queue 2 are both
> mapped
> > to cpu3. From this knowledge we can figure out the total RX stats
> processed by
> > each CPU by adding the values of mana_q0 and mana_q2 stats for cpu3.
> But if
> > this data changes dynamically using irqbalance or smp_affinity file
> edits, the
> > above assumption fails.
> >
> > Interrupt affinity for mana_q2 changes and the affinity table looks as
> follows
> >  25:          1        103          0    3038084  Hyper-V PCIe MSI
> 4138200989697-edge      mana_q0@pci:7870:00:00.0
> >  26:          0          1    4012447          0  Hyper-V PCIe MSI
> 4138200989698-edge      mana_q1@pci:7870:00:00.0
> >  27:     157181         10          1    3007990  Hyper-V PCIe MSI
> 4138200989699-edge      mana_q2@pci:7870:00:00.0
> >  28:    3593858          0          0          1  Hyper-V PCIe MSI
> 4138200989700-edge      mana_q3@pci:7870:00:00.0
> >
> > And during this time we might end up calculating the per-CPU stats
> incorrectly,
> > messing up the understanding of CPU usage by MANA driver that is
> consumed by
> > monitoring services.
>=20
> Like Stephen said, forget about irqbalance for networking.
>=20
> Assume that the IRQs are affinitized and XPS set, correctly.
>=20
> Now, presumably you can use your pcpu stats to "trade queues",
> e.g. 4 CPUs / 4 queues, if CPU 0 insists on using queue 1
> instead of queue 0, you can swap the 0 <> 1 assignment.
> That's just an example of an "algorithm", maybe you have other
> use cases. But if the problem is "user runs broken irqbalance"
> the solution is not in the kernel...

We understand irqbalance may be a "bad idea", and recommended some=20
customers to disable it when having problems with it... But it's=20
still enabled by default, and we cannot let all distro vendors=20
and custom image makers to disable the irqbalance. So, our host-
networking team is eager to have per-CPU stats for analyzing CPU=20
usage related to irqbalance or other issues.

Thanks,
- Haiyang