Received-SPF: pass (google.com: domain of linux-kernel+bounces-54211-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
References: <20240206010311.149103-1-jdamato@fastly.com>
 <878r3ymlnk.fsf@nvidia.com> <20240206013246.GA11217@fastly.com>
 <874jemml1j.fsf@nvidia.com> <20240206014151.GA11233@fastly.com>
 <87zfwel5qi.fsf@nvidia.com> <20240206015657.GA11257@fastly.com>
User-agent: mu4e 1.10.8; emacs 28.2
From: Rahul Rameshbabu <rrameshbabu@nvidia.com>
To: Joe Damato <jdamato@fastly.com>
Cc: linux-kernel@vger.kernel.org, netdev@vger.kernel.org, tariqt@nvidia.com,
 Saeed Mahameed <saeedm@nvidia.com>, Leon Romanovsky <leon@kernel.org>,
 "David S. Miller" <davem@davemloft.net>, Eric Dumazet
 <edumazet@google.com>, Jakub Kicinski <kuba@kernel.org>, Paolo Abeni
 <pabeni@redhat.com>, "open list:MELLANOX MLX5 core VPI driver"
 <linux-rdma@vger.kernel.org>
Subject: Re: [PATCH net-next] eth: mlx5: link NAPI instances to queues and IRQs
Date: Mon, 05 Feb 2024 18:38:41 -0800
In-reply-to: <20240206015657.GA11257@fastly.com>
Message-ID: <87sf26l3go.fsf@nvidia.com>
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?sPglla8n7Ur3UEa1KpVVCWVjLtZdOx2JP6bw8jMo9ax1ta5lnMkNvYoZuTnn?=
 =?us-ascii?Q?afo1kA4PIUf7PKdMo1Ng76VF+i2NdosaKs3vy1CQ+8ctrXbQa1dLUIU5e4I6?=
 =?us-ascii?Q?Z7wDULCPfD75Q2hk6ZYFYKf7bW7llwpr6w5h125E98Q/PKHqe/q0x6y4tajs?=
 =?us-ascii?Q?P87i5C8TjyT33ga1Dn6KL5fChPZdHXkZeyVXdEJVozW52Rk1E6Vt7Xc3mBIr?=
 =?us-ascii?Q?X7QMF9Bkb7hinXkwvp8fdiLiQOYaNRqNc+9ItS9K/JlpKAh8Wr8rWHTjLKxQ?=
 =?us-ascii?Q?LtLssBvCfbb0Y6lV3dWK+suiLwkOmV6VOC2VPBD0uGvhQGIINOYNvyiwvUa0?=
 =?us-ascii?Q?WpD6oCiI9xGqRj6huGcWmdD5MKIc2MdV6qM0lmfMVtDi/HTIPLDCTGcH5y0k?=
 =?us-ascii?Q?bogS9WCaydD+o7BRJiwE1oIamU/lf7bN4WC34Wq5dP3uX7whcnXCDJ/CKo1g?=
 =?us-ascii?Q?fp3p6flC5QilX1+pfOU66Zcv5X459SSCoSJCAdPmv9wyChMr/K5gV4RnaD36?=
 =?us-ascii?Q?JQlrleUGW5jKNo/U/VyPXY1CLwCrMALmTgfYKBXeMBKxxedom09ZSPM61Cjs?=
 =?us-ascii?Q?9X7k6moXak5bqdkprtdSHaJlghwTY2QoBIxuJgYoq5fJMwlPtz2J0J44QcCn?=
 =?us-ascii?Q?5SVYFJ+YMUpxyogQ3nNaZ4rUsbJ29nPZzWAOcDD1WHsXmSXKuANaCw83Aj0f?=
 =?us-ascii?Q?DbBChyJpAYBqu4EVrxHEtDRxhj5Fyv2N9vseYjf8vzzZsu5gIePrDzSYf4d7?=
 =?us-ascii?Q?NWxm0jgPTRvq3j50S5F9Y37YMymx7CCzSzGvq9QxX1Rf6UF9p0M/L0ZE/Kx5?=
 =?us-ascii?Q?8iuv205YdfTAEEQ/x14oniaGOrwuSUWT/UoGkE90Y7lStA/XJ587+ejLjiW9?=
 =?us-ascii?Q?HT/4IJOTAxq5eMftDP7EYgjsTnEWy42jJfDSpQVjcbZK57b89Ibzr+0a7/tQ?=
 =?us-ascii?Q?G4hvSY7GXblW9GD86TLAXSatMU13gUezQux2gGy3MQBRmbmmY2K1GOAYBRrx?=
 =?us-ascii?Q?P50y55S0UQJVwz93PzN7kQk74TAw2tkVH5t3mbNWRASXL+8nz7mpS06xec1X?=
 =?us-ascii?Q?GseS9q8TKuepVB1twQ57ltA0hxlO08toHdnEnFI60KtjfRI39x0ynPFr0rdZ?=
 =?us-ascii?Q?R6UGZpqyAv1E31BtCXvi+oFabW/VK2xP9VbZy3GqpO5BbRP09nc5S6J54Qg2?=
 =?us-ascii?Q?X+h/x0kc8p3RjX5xQWPVBj3e6KDLfif0/zXPmp/7G4ify8dLInoDAhn8e7N+?=
 =?us-ascii?Q?84eBVpOE2pWQt8WaMgxPORBSpmtjDsJjK/urZhoM1kwrqO9PoP8SjhxQ7lVf?=
 =?us-ascii?Q?nUK/KUcbJEqEOfqm4gG8JnKCOKr/u8ceNQbq/yrZLurYmHOod/AORUc4LYEK?=
 =?us-ascii?Q?P2AoRufTPzbJ6VGD14f5uW/6NlgVGZudBMPMuZACSYy18en74BhkTu26cYQ3?=
 =?us-ascii?Q?vi4oln7reE96ldnXzyGqqlajfjB9nI0QIMPtVAPDfl61gnemEvRx7/X4Ehff?=
 =?us-ascii?Q?QEAfaQZIuLKgtkQ/Ivbc4Xs/kFBmNnQ3NZHXbXmaJJzn2NGNirAhJnPhyyMm?=
 =?us-ascii?Q?eS3SWb5KO9qsla6Cr0LOelXAxRARXSZ4wlgeTDGx0QArgYSXzyWKxb4dNj/5?=
 =?us-ascii?Q?Yw=3D=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 933c9777-bc41-4e65-33b6-08dc26bd0b88
X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB2743.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2024 02:40:56.5513
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ia1GSg4kEAEtjGMXqr8qOIZ9BqBeqpwT1At5eei5ok0aVQ+aFwVoJfbbS4K9gWHlI9ay8XMmuO5i/SEe7D7LCw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB7549

On Mon, 05 Feb, 2024 17:56:58 -0800 Joe Damato <jdamato@fastly.com> wrote:
> On Mon, Feb 05, 2024 at 05:44:27PM -0800, Rahul Rameshbabu wrote:
>> 
>> On Mon, 05 Feb, 2024 17:41:52 -0800 Joe Damato <jdamato@fastly.com> wrote:
>> > On Mon, Feb 05, 2024 at 05:33:39PM -0800, Rahul Rameshbabu wrote:
>> >> 
>> >> On Mon, 05 Feb, 2024 17:32:47 -0800 Joe Damato <jdamato@fastly.com> wrote:
>> >> > On Mon, Feb 05, 2024 at 05:09:09PM -0800, Rahul Rameshbabu wrote:
>> >> >> On Tue, 06 Feb, 2024 01:03:11 +0000 Joe Damato <jdamato@fastly.com> wrote:
>> >> >> > diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>> >> >> > index c8e8f512803e..e1bfff1fb328 100644
>> >> >> > --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>> >> >> > +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>> >> >> > @@ -2473,6 +2473,9 @@ static void mlx5e_close_queues(struct mlx5e_channel *c)
>> >> >> >  	mlx5e_close_tx_cqs(c);
>> >> >> >  	mlx5e_close_cq(&c->icosq.cq);
>> >> >> >  	mlx5e_close_cq(&c->async_icosq.cq);
>> >> >> > +
>> >> >> > +	netif_queue_set_napi(c->netdev, c->ix, NETDEV_QUEUE_TYPE_TX, NULL);
>> >> >> > +	netif_queue_set_napi(c->netdev, c->ix, NETDEV_QUEUE_TYPE_RX, NULL);
>> >> >> 
>> >> >> This should be set to NULL *before* actually closing the rqs, sqs, and
>> >> >> related cqs right? I would expect these two lines to be the first ones
>> >> >> called in mlx5e_close_queues. Btw, I think this should be done in
>> >> >> mlx5e_deactivate_channel where the NAPI is disabled.
>> >> >> 
>> >> >> >  }
>> >> >> >  
>> >> >> >  static u8 mlx5e_enumerate_lag_port(struct mlx5_core_dev *mdev, int ix)
>> >> >> > @@ -2558,6 +2561,7 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
>> >> >> >  	c->stats    = &priv->channel_stats[ix]->ch;
>> >> >> >  	c->aff_mask = irq_get_effective_affinity_mask(irq);
>> >> >> >  	c->lag_port = mlx5e_enumerate_lag_port(priv->mdev, ix);
>> >> >> > +	c->irq		= irq;
>> >> >> >  
>> >> >> >  	netif_napi_add(netdev, &c->napi, mlx5e_napi_poll);
>> >> >> >  
>> >> >> > @@ -2602,6 +2606,10 @@ static void mlx5e_activate_channel(struct mlx5e_channel *c)
>> >> >> >  		mlx5e_activate_xsk(c);
>> >> >> >  	else
>> >> >> >  		mlx5e_activate_rq(&c->rq);
>> >> >> > +
>> >> >> > +	netif_napi_set_irq(&c->napi, c->irq);
>> >> 
>> >> One small comment that I missed in my previous iteration. I think the
>> >> above should be moved to mlx5e_open_channel right after netif_napi_add.
>> >> This avoids needing to save the irq in struct mlx5e_channel.
>> >
>> > I couldn't move it to mlx5e_open_channel because of how safe_switch_params
>> > and the mechanics around that seem to work (at least as far as I could
>> > tell).
>> >
>> > mlx5 seems to create a new set of channels before closing the previous
>> > channel. So, moving this logic to open_channels and close_channels means
>> > you end up with a flow like this:
>> >
>> >   - Create new channels (NAPI netlink API is used to set NAPIs)
>> >   - Old channels are closed (NAPI netlink API sets NULL and overwrites the
>> >     previous NAPI netlink calls)
>> >
>> > Now, the associations are all NULL.
>> >
>> > I think moving the calls to active / deactivate fixes that problem, but
>> > requires that irq is stored, if I am understanding the driver correctly.
>> 
>> I believe moving the changes to activate / deactivate channels resolves
>> this problem because only one set of channels will be active, so you
>> will no longer have dangling association conflicts for the queue ->
>> napi. This is partially why I suggested the change in that iteration.
>
> As far as I can tell, it does.
>  
>> As for netif_napi_set_irq, that alone can be in mlx5e_open_channel (that
>> was the intention of my most recent comment. Not that all the other
>> associations should be moved as well). I agree that the other
>> association calls should be part of activate / deactivate channels.
>
> OK, sure that makes sense. I make that change, too.
>

Also for your v2, it would be great if you can use the commit message
subject 'net/mlx5e: link NAPI instances to queues and IRQs' rather than
'eth: mlx5: link NAPI instances to queues and IRQs'.

--
Thanks,

Rahul Rameshbabu