Received-SPF: pass (google.com: domain of linux-kernel+bounces-97523-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
References: <1709823132-22411-1-git-send-email-shradhagupta@linux.microsoft.com>
 <20240307072923.6cc8a2ba@kernel.org>
 <DM6PR21MB14817597567C638DEF020FE3CA202@DM6PR21MB1481.namprd21.prod.outlook.com>
 <20240307090145.2fc7aa2e@kernel.org>
 <CH2PR21MB1480D3ACADFFD2FC3B1BB7ECCA272@CH2PR21MB1480.namprd21.prod.outlook.com>
 <20240308112244.391b3779@kernel.org>
 <CH2PR21MB1480D4AE8D329B5F00B184A7CA272@CH2PR21MB1480.namprd21.prod.outlook.com>
User-agent: mu4e 1.10.8; emacs 28.2
From: Rahul Rameshbabu <rrameshbabu@nvidia.com>
To: Haiyang Zhang <haiyangz@microsoft.com>
Cc: Jakub Kicinski <kuba@kernel.org>, Shradha Gupta
 <shradhagupta@linux.microsoft.com>, Shradha Gupta
 <shradhagupta@microsoft.com>, "linux-kernel@vger.kernel.org"
 <linux-kernel@vger.kernel.org>, "linux-hyperv@vger.kernel.org"
 <linux-hyperv@vger.kernel.org>, "linux-rdma@vger.kernel.org"
 <linux-rdma@vger.kernel.org>, "netdev@vger.kernel.org"
 <netdev@vger.kernel.org>, Eric Dumazet <edumazet@google.com>, Paolo Abeni
 <pabeni@redhat.com>, Ajay Sharma <sharmaajay@microsoft.com>, Leon
 Romanovsky <leon@kernel.org>, Thomas Gleixner <tglx@linutronix.de>,
 Sebastian Andrzej  Siewior <bigeasy@linutronix.de>, KY Srinivasan
 <kys@microsoft.com>, Wei Liu <wei.liu@kernel.org>, Dexuan Cui
 <decui@microsoft.com>, Long Li <longli@microsoft.com>, Paul Rosswurm
 <paulros@microsoft.com>, Alireza Dabagh <alid@microsoft.com>, Sharath
 George John <sgeorgejohn@microsoft.com>
Subject: Re: [PATCH] net :mana : Add per-cpu stats for MANA device
Date: Fri, 08 Mar 2024 11:52:42 -0800
In-reply-to: <CH2PR21MB1480D4AE8D329B5F00B184A7CA272@CH2PR21MB1480.namprd21.prod.outlook.com>
Message-ID: <87bk7oebxk.fsf@nvidia.com>
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?lR3sw8s2Whs4DCEkdzYThy1XpEcT9gKufKAePC7KdNpGyXaOU7WlNzITNDl8?=
 =?us-ascii?Q?VuDsiwwBF+k0mV4iUhJkyeTG/YnWF6kFaVY9yEmFFkkrb4rTS6ItG3hdEakU?=
 =?us-ascii?Q?BdV2f24LflyLAYp+lZ8rZXwdn5ZE9wdLm7X4yyNYC93deyQ7++Yf77pNQ4nL?=
 =?us-ascii?Q?9MQ13giEw7FR2BeZxUHsajiihOhJak0u0ToHUJHepjt+LTu4bjJJSEINJ8gt?=
 =?us-ascii?Q?kk+AqHXO8YWKrP7v9YcmnoD0R4Km3UPnmdzUm8SNbbxdG40kp+IqAvkZ95rR?=
 =?us-ascii?Q?nTXlAIkfHQP+RkMogyi4TansEqdL1LVHspFdIjo4X3JHYwneZkE2biNntWDW?=
 =?us-ascii?Q?+SrZU0nqm5VDNAp8h2n/QBOgQa4LScmW5ZCCFTTygM/0MBeOPoGPY/bOtSP7?=
 =?us-ascii?Q?g8zoSZg/VLIaJx0YrGeUBfGbnK5gqf2spwxaajbSGWWF882JxAzWrcT2kbEa?=
 =?us-ascii?Q?j3oIjB2MZ7hp/iye6bWysmgtJPm8XM5+OAw6QJx67EZbPDCJ2Nu3VuYMNxmf?=
 =?us-ascii?Q?kLUS/8klAf+RkYw8yt/eLUDPFAI4fR5/hdcvFMbtFKicHJ4tNwSBiPI76ILo?=
 =?us-ascii?Q?8sdJ5YOHHt2dOJQDkKhkzJTfx45iwRzoDPCxy/uSr9bVwcPYlAyzB48/sqQt?=
 =?us-ascii?Q?5L4DHpiSv8aUvIfpNXrxcV8E7bC8FACwLm4pbB5D6LjmkzOdF5jXfSfgaGbs?=
 =?us-ascii?Q?UXjek5q/R3j3C/ocTFLJikHjof0bJ3iZYbYUKHONGGEJ6qaX8BMMLQ1nlzGr?=
 =?us-ascii?Q?qeaHIoI9ymY0wIEszgu98cq0O5dd/twedd+RLOf0h4Epl9LMSmfox7syjvdu?=
 =?us-ascii?Q?TveegeOIdo4IALDQFrgc3moRgJprZVwqFpZjJXBgBADoYrDf9ctQBPaO5bSE?=
 =?us-ascii?Q?yqui+/QJENM+rjFqJ629tkFPqKaX+3XOgOmodTs52BsEiVH7M+hvecqj9aV7?=
 =?us-ascii?Q?D2LuxC1OHsbdaXF3zS+SqT9U8oe85YOXWKx5msD/KvByChAbS5QrRdePanOn?=
 =?us-ascii?Q?tbIK9oMiXXikg7WMFF1Ly+r6jXTCkLkSAtnaO/IrioULSt43+BsKs5qClWQt?=
 =?us-ascii?Q?A9aV/PekmPtn5nudqWC8jEofwIQ1lfTuhJEoo+B8NdFfxTl1K9VoLd3jJHb+?=
 =?us-ascii?Q?z1F7iPFLmFOfY57mhzQOvMVzwk5hhhBco+fFns+/XXt1f1L75yrtwIjCHABH?=
 =?us-ascii?Q?PVadLOjqAhl5N2aqgPuZRDPBFPqWgFfQ2CUUqhYg+p2ZNLEarvVdkiPkIbF4?=
 =?us-ascii?Q?i8l+pCHs4MlsYvXPZ5nicaV3WaIeUKTHQPUXMJfR9uYGfLVKN24doVT/tUDm?=
 =?us-ascii?Q?GwUeJ3ufnw8nqG7STw1bfyh0zwys26HpoSsFfsx8NgwOXm/cBhykm1AX0vIj?=
 =?us-ascii?Q?+slJZO6GuaF2fIjnbZchGRGqSeZeC6yqAx77ME4QCJ0bz7tYLsou9nzRMYl7?=
 =?us-ascii?Q?midHsnwwOeo/nD82lGUuZGuQHCTP3PEpls3cxCb1RDl6n4vcCz0k/RERDG+H?=
 =?us-ascii?Q?XNp1LTSncr2gUINwDKNhmjzsqmnRz+PEJ+piybyhUdXSB/xAVaEejD9whm9t?=
 =?us-ascii?Q?G2SzKzq6/PxiqLSHyKViBnWFwaP1xRWPl8MxLRHA+iE0rPWaGY65D86bvT2g?=
 =?us-ascii?Q?OA=3D=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b47b1b9d-1acd-41f9-ea71-08dc3fa9a57a
X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB2743.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Mar 2024 19:55:03.9203
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: /7IrEigsTSzqZjpm2Cp5kRImpPLPlnd6r4LACK3tnvm5o3rLvTEKoaR/o4QBPxxxvezYVCtIL2jfc/IQi5WOOA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB8407


On Fri, 08 Mar, 2024 19:43:57 +0000 Haiyang Zhang <haiyangz@microsoft.com> wrote:
>> -----Original Message-----
>> From: Jakub Kicinski <kuba@kernel.org>
>> Sent: Friday, March 8, 2024 2:23 PM
>> To: Haiyang Zhang <haiyangz@microsoft.com>
>> Cc: Shradha Gupta <shradhagupta@linux.microsoft.com>; Shradha Gupta
>> <shradhagupta@microsoft.com>; linux-kernel@vger.kernel.org; linux-
>> hyperv@vger.kernel.org; linux-rdma@vger.kernel.org;
>> netdev@vger.kernel.org; Eric Dumazet <edumazet@google.com>; Paolo Abeni
>> <pabeni@redhat.com>; Ajay Sharma <sharmaajay@microsoft.com>; Leon
>> Romanovsky <leon@kernel.org>; Thomas Gleixner <tglx@linutronix.de>;
>> Sebastian Andrzej Siewior <bigeasy@linutronix.de>; KY Srinivasan
>> <kys@microsoft.com>; Wei Liu <wei.liu@kernel.org>; Dexuan Cui
>> <decui@microsoft.com>; Long Li <longli@microsoft.com>; Michael Kelley
>> <mikelley@microsoft.com>
>> Subject: Re: [PATCH] net :mana : Add per-cpu stats for MANA device
>> 
>> On Fri, 8 Mar 2024 18:51:58 +0000 Haiyang Zhang wrote:
>> > > Dynamic is a bit of an exaggeration, right? On a well-configured
>> system
>> > > each CPU should use a single queue assigned thru XPS. And for manual
>> > > debug bpftrace should serve the purpose quite well.
>> >
>> > Some programs, like irqbalancer can dynamically change the CPU
>> affinity,
>> > so we want to add the per-CPU counters for better understanding of the
>> CPU
>> > usage.
>> 
>> Do you have experimental data showing this making a difference
>> in production?
> Shradha, could you please add some data before / after enabling irqbalancer 
> which changes cpu affinity?
>
>> 
>> Seems unlikely, but if it does work we should enable it for all
>> devices, no driver by driver.
> There are some existing drivers, like mlx, rmnet, netvsc, etc. using percpu 
> counters. Are you suggesting we add a common API for all drivers?

Wanted to chime in with regards to mlx. You might be conflating per-cpu
with per-queue. When we run ethtool -S, we present counters per netdev
queue rather than per-cpu. The number of queues we instantiate is
related to CPUs but it not always 1-1.

Jakub just recently supported a proper interface for per-queue stats
counters that we are interested in supporting.

  https://lore.kernel.org/netdev/20240222223629.158254-1-kuba@kernel.org/

--
Thanks,

Rahul Rameshbabu