Received-SPF: pass (google.com: domain of linux-kernel+bounces-54147-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
References: <20240206010311.149103-1-jdamato@fastly.com>
User-agent: mu4e 1.10.8; emacs 28.2
From: Rahul Rameshbabu <rrameshbabu@nvidia.com>
To: Joe Damato <jdamato@fastly.com>
Cc: linux-kernel@vger.kernel.org, netdev@vger.kernel.org, tariqt@nvidia.com,
 Saeed Mahameed <saeedm@nvidia.com>, Leon Romanovsky <leon@kernel.org>,
 "David S. Miller" <davem@davemloft.net>, Eric Dumazet
 <edumazet@google.com>, Jakub Kicinski <kuba@kernel.org>, Paolo Abeni
 <pabeni@redhat.com>, "open list:MELLANOX MLX5 core VPI driver"
 <linux-rdma@vger.kernel.org>
Subject: Re: [PATCH net-next] eth: mlx5: link NAPI instances to queues and IRQs
Date: Mon, 05 Feb 2024 17:09:09 -0800
In-reply-to: <20240206010311.149103-1-jdamato@fastly.com>
Message-ID: <878r3ymlnk.fsf@nvidia.com>
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?5Fhp2xbuTfPnDg6lZdx+HF/zWsbML2wlxLd1i7NvtL8CBSZBHYkisYeHC4iU?=
 =?us-ascii?Q?3qbYA5q5B3D2wQYTNkco9Jf3mt7+i2eSVPcR/8eyFKxWoGzuWKsoJyvrgG39?=
 =?us-ascii?Q?yhQApMBjxLh4lqHCsMcvDy2+Fm4ZETJz7ECz97x5MN8XAQdlGOuyWyZVRsHn?=
 =?us-ascii?Q?19SjJZVL1Ij7FnLtEQKEf7lM9PfBAZrw+X94zc3Xul9sPdzdzRBxTwimsFiV?=
 =?us-ascii?Q?tmnCitaESeko4tNjzmTrZY/WQ/cndv5uncL7WEhwo+vIFPV8UIoZPEXUSGeI?=
 =?us-ascii?Q?/UncGb0dWfVBeRKZLM+oftqRb9RjjaWjNFpD40HXHfVTQ06qFANd9MmXeXA4?=
 =?us-ascii?Q?MUcjzt/03S2ufFCpJq0cM2HCUawmA4n8Z+QhW677XTPaY2V3tuNM+mLUVrrZ?=
 =?us-ascii?Q?AxrYHn6RMmhd8QWGGPoz67TP2zjOnkBDDOKmdM5V4cdIYM6zmPb55/srT/Ag?=
 =?us-ascii?Q?5pg65nYT9PY//Ro+mNn4749xk9BlXcT/bm8cm3C84Ual8vT111pSxridOvMg?=
 =?us-ascii?Q?Sb4S5W+TWZiN2oRaPzIVP5cBrxwKWKClfwT/s+Klx5DaQZMGT2Xr1oEp8E/q?=
 =?us-ascii?Q?DeRyzNMxb4MiyrXUZXBfRsjr48CBCsZkzA6ChOANM5e64+6kuqE0KMHM1SCY?=
 =?us-ascii?Q?HiqTyNJITpPZjgzBr0MFqNOc+KjhPUCDaXx7BuMLJRpDWQXIQ8SOwzDCjrik?=
 =?us-ascii?Q?c7WFN7mV4SFomytksbEF1rjfeyJRHLYxnyp1oZaBzzjwcMWggcd5sEsUZ/L+?=
 =?us-ascii?Q?4VOZBh4tzT49o8Rlt1QGhM0sZq9XI8nc2wVlefEkrjocBRr7iO7STPgg64zv?=
 =?us-ascii?Q?7ElsY2jckF0fZfZAVpe4K6t5Z3C9/lVkEWQQzFFWGf9NUOlVVwYV6yZRwAXM?=
 =?us-ascii?Q?JV1rpkVd6WxCd8SqeMO0U9S8mgqgNx1uEC9i2yOQxSZHtJzr8CG5rb5Lfvj2?=
 =?us-ascii?Q?q31LofXhP+nJDWAP9DXXDFJ+Rr8qm3qE4Ep5oiJy9Zt0IFMLIPmsx84ITcl6?=
 =?us-ascii?Q?fWb+q8RDw9H/qKS2cKNlmJOaFBUoeCBk48Tgv72n1sWFCXZ2+xIZwECNtjXV?=
 =?us-ascii?Q?XJXXrckXaKjQ26ZQeRweTLav7bERySQknzlYgNz41bQ8ToaENbr/kMFWvTYs?=
 =?us-ascii?Q?E4/ncQjSM5KLePnGoBE+9rf0NXhSQsz36nr4zc+06nmURBsfVi6I26Db+IqP?=
 =?us-ascii?Q?snlNb7+qp2AmzRWirCAHjIV3NQkxg2pcDgmNoa6+c6wnZkt3tSase6j/Ti2J?=
 =?us-ascii?Q?RXSvKvZKfUAml9nyeu3X4HeT/6Z3zmSYTsw7Qhoo17JkR82ooPdCegWKFG0i?=
 =?us-ascii?Q?zWySuLYJE2zV+8e0QiNDIy7HJ5vyE9myTs8Gxpaz968zuzQ33SKZ5WPsgNTM?=
 =?us-ascii?Q?FCty6EUgUIet3WB3T32eU2EX5ajYRrEM2cLMCjxCYlNidDgjLvH9xqZ9RZh8?=
 =?us-ascii?Q?uDEYwQ7LUHv4+KEpGOi8cRocu35fkT7hSbPYNediNSen+AvRCmiFumkJm2P5?=
 =?us-ascii?Q?qZci3zuQTLkX0QIIsScaClea7WAIHm1CL5ZFiiaffldLj7NLechqn2id25gh?=
 =?us-ascii?Q?Gs8cDZAf8XMjX9WGXdi/KntECEQHABCTBGfvHk1T?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0987e9a8-e730-4ff7-eb9e-08dc26b21c56
X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB2743.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2024 01:22:40.2612
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: GDNw44zH2N3wBIbk143MpCN0RUogN9QoKJ+Y194NaVUag8T2FzG5yRZsYBe9QQB0AROopJQXb10LLD8kvJRmog==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7684

On Tue, 06 Feb, 2024 01:03:11 +0000 Joe Damato <jdamato@fastly.com> wrote:
> Make mlx5 compatible with the newly added netlink queue GET APIs.
>
> Signed-off-by: Joe Damato <jdamato@fastly.com>
> ---
>  drivers/net/ethernet/mellanox/mlx5/core/en.h      | 1 +
>  drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 8 ++++++++
>  2 files changed, 9 insertions(+)
>
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
> index 55c6ace0acd5..3f86ee1831a8 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
> @@ -768,6 +768,7 @@ struct mlx5e_channel {
>  	u16                        qos_sqs_size;
>  	u8                         num_tc;
>  	u8                         lag_port;
> +	unsigned int		   irq;
>  
>  	/* XDP_REDIRECT */
>  	struct mlx5e_xdpsq         xdpsq;
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
> index c8e8f512803e..e1bfff1fb328 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
> @@ -2473,6 +2473,9 @@ static void mlx5e_close_queues(struct mlx5e_channel *c)
>  	mlx5e_close_tx_cqs(c);
>  	mlx5e_close_cq(&c->icosq.cq);
>  	mlx5e_close_cq(&c->async_icosq.cq);
> +
> +	netif_queue_set_napi(c->netdev, c->ix, NETDEV_QUEUE_TYPE_TX, NULL);
> +	netif_queue_set_napi(c->netdev, c->ix, NETDEV_QUEUE_TYPE_RX, NULL);

This should be set to NULL *before* actually closing the rqs, sqs, and
related cqs right? I would expect these two lines to be the first ones
called in mlx5e_close_queues. Btw, I think this should be done in
mlx5e_deactivate_channel where the NAPI is disabled.

>  }
>  
>  static u8 mlx5e_enumerate_lag_port(struct mlx5_core_dev *mdev, int ix)
> @@ -2558,6 +2561,7 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
>  	c->stats    = &priv->channel_stats[ix]->ch;
>  	c->aff_mask = irq_get_effective_affinity_mask(irq);
>  	c->lag_port = mlx5e_enumerate_lag_port(priv->mdev, ix);
> +	c->irq		= irq;
>  
>  	netif_napi_add(netdev, &c->napi, mlx5e_napi_poll);
>  
> @@ -2602,6 +2606,10 @@ static void mlx5e_activate_channel(struct mlx5e_channel *c)
>  		mlx5e_activate_xsk(c);
>  	else
>  		mlx5e_activate_rq(&c->rq);
> +
> +	netif_napi_set_irq(&c->napi, c->irq);
> +	netif_queue_set_napi(c->netdev, c->ix, NETDEV_QUEUE_TYPE_TX, &c->napi);
> +	netif_queue_set_napi(c->netdev, c->ix, NETDEV_QUEUE_TYPE_RX, &c->napi);

It's weird that netlink queue API is being configured in
mlx5e_activate_channel and deconfigured in mlx5e_close_queues. This
leads to a problem where the napi will be falsely referred to even when
we deactivate the channels in mlx5e_switch_priv_channels and may not
necessarily get to closing the channels due to an error.

Typically, we use the following clean up patterns.

mlx5e_activate_channel -> mlx5e_deactivate_channel
mlx5e_open_queues -> mlx5e_close_queues


>  }
>  
>  static void mlx5e_deactivate_channel(struct mlx5e_channel *c)

--
Thanks,

Rahul Rameshbabu