Received-SPF: pass (google.com: domain of linux-kernel+bounces-59000-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
References: <20240208030702.27296-1-jdamato@fastly.com>
 <8c083e6d-5fcd-4557-88dd-0f95acdbc747@gmail.com>
User-agent: mu4e 1.10.8; emacs 28.2
From: Rahul Rameshbabu <rrameshbabu@nvidia.com>
To: Tariq Toukan <ttoukan.linux@gmail.com>
Cc: Joe Damato <jdamato@fastly.com>, linux-kernel@vger.kernel.org,
 netdev@vger.kernel.org, tariqt@nvidia.com, Saeed Mahameed
 <saeedm@nvidia.com>, Leon Romanovsky <leon@kernel.org>, "David S. Miller"
 <davem@davemloft.net>, Eric Dumazet <edumazet@google.com>, Jakub Kicinski
 <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>, Richard Cochran
 <richardcochran@gmail.com>, Gal Pressman <gal@nvidia.com>, Vadim Fedorenko
 <vadim.fedorenko@linux.dev>, "open list:MELLANOX MLX5 core VPI driver"
 <linux-rdma@vger.kernel.org>
Subject: Re: [PATCH net-next v3] net/mlx5e: link NAPI instances to queues
 and IRQs
Date: Thu, 08 Feb 2024 20:43:57 -0800
In-reply-to: <8c083e6d-5fcd-4557-88dd-0f95acdbc747@gmail.com>
Message-ID: <871q9mz1a0.fsf@nvidia.com>
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?8yQzByVPiVsIITYBtLD6lwVEEst5TPocAE5LNm6OJ3uFrZAsiPbT4RpNMw0g?=
 =?us-ascii?Q?GUpI/zSAfFv0P83jeVRkdEoGtiaLNT6VRAIzh9N7RRfighjuXnUkmQu20Zi1?=
 =?us-ascii?Q?jQnM31Dc/hnF/Hu13LgIqolHQrf2kaXkPl0tev5wyRo42RuQwOiSC2ArJwbX?=
 =?us-ascii?Q?oxDPrIaQ3nzoSpGUGmBXbT1QkhEpOv1NLpkd7mS5D7bbY7GPwT14NBToSaUp?=
 =?us-ascii?Q?RoThRpR1RNbtSr+DpWsiaB4IOPcc7KCVVaASCeVcDk7AWKp9c4MZAMjsEqs+?=
 =?us-ascii?Q?kODItgwDs0b/2bO/8N8jOQWbkoy0Y/eXd1nYHxNMSdAIRWeJaz7l97mc6H5c?=
 =?us-ascii?Q?t2iWngXPRDjCZ5K+j2tt1K3r/SXM+yvvxivBRz1L7XIq9QhrlxzLAjkI1rmA?=
 =?us-ascii?Q?SqA7LUeD5MFfUyyld9sOe4y3NGCkBoETGL+vAHWhxyqF1/G62Mo0Edv3lwrm?=
 =?us-ascii?Q?q3xCxSQti/482SvzQnFOJMgEWDsDrh7ArUHl4Dh7S6QQuQTl2HKhQ7YqBG15?=
 =?us-ascii?Q?qgqyZJV/2SeD+ZAIsWno4pSJeLm0B3/npyN7xR1O1ITiFMBo7VIsxaTXNFD8?=
 =?us-ascii?Q?gPSnAJqn0uzANMMq9xLh6JNtEbKWRyU5cRQjjckCLWkE0rlteDsB+pw4yv5f?=
 =?us-ascii?Q?r7w1SBvf62nz/1QKcefcjIPcsmXBErDMzcm29d7n/I0Oy53nWPetj36BMlTG?=
 =?us-ascii?Q?v/ED7s31MCjJHKxsj9l2GHcEQOmBDE5sCZ2pQDviVTKpnQewLul/0wpi2lT7?=
 =?us-ascii?Q?Ds2hTLmKdx0ROuuIgBpZysG+a0xtm9ZgYFiAAOR0AyMgGJdCPsRAU/b/lCMW?=
 =?us-ascii?Q?vVLwiWzPWGHZZCceHZMXYI96L5a7n0DjTN0NHg3Pg+dg87zxM/w7H/Fz8nc8?=
 =?us-ascii?Q?l+HzS5SeO1jB4ZGnrHHljudFnB8Ghj2MLrAz1+9FAV+t9E46ri5Kf56zDjrA?=
 =?us-ascii?Q?Qx0DpytogwoatQtDBg1+aZ5kQQQ5QcPYP+WYZgQ9TZE8yv9psGwdnpAVOTRK?=
 =?us-ascii?Q?bISgam+5x0eY/UOvHutb3zBq9pxRljxLaJw6peHYDoNalxJ1X5nVImYnrGAp?=
 =?us-ascii?Q?KIGYD8vIItl91OEFJ9KuDEyC04TH5e4LIoEE/8mcANNXL5Rl1wxp6gyFm/Ay?=
 =?us-ascii?Q?V4D2Rzxz+XqSCjTdfXyr0rqn6hP+WOKY2OGsnlVw/c1GAG4RL9v2vNKYeciG?=
 =?us-ascii?Q?v4X0KQgtC3BhbSdST0MrqqYDOopuUVYgcitqeu0e8gask7RyEl3NxnFWEWdh?=
 =?us-ascii?Q?1vR/f4rox4mv6T/JwOxkc9+Ar1bpGBDU6rIYSZ2LL6bTBG9MJprFX0XTyYZK?=
 =?us-ascii?Q?BgcklyaiYNbCh/Gi2rj0SG+S1WEe0NnmTG0GhTDq1gtvwoyXEKLtKDS2Jwb5?=
 =?us-ascii?Q?Z2EBqxNSrVM8K5L+b1LQ1vvYX5sidFvtzvG2Ooi3cSv/6JNJFXXJhfFhgqPS?=
 =?us-ascii?Q?lshq0zbjgitH1h+46QJqVK9inoJRV0qaNZ/UXbSz4+ktD/hT0ascdsErzDDe?=
 =?us-ascii?Q?evB82oR8PwDMSbHhMnyMfL5H/9NiPb9o0d5HzTSGDaru6f4kpAQO5h5OdUaT?=
 =?us-ascii?Q?dcwAp2oQsZyiY93hbut3Vc3FyjKMW2GrGUElp377MoUv6/6/bQilLhyc4Uvc?=
 =?us-ascii?Q?dg=3D=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 345e77c9-9911-4e45-1710-08dc292b0e77
X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB2743.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2024 04:53:28.4329
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: CWAZTkFI2WGPtWXpSNxo7WAleHkhiZOlV2RYWZUSXtCYwYRYw+lhxT6K6C5s4HL3QNUcSQ/Kq7NpJTCYjzdtjA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB7001

On Thu, 08 Feb, 2024 21:13:25 +0200 Tariq Toukan <ttoukan.linux@gmail.com> wrote:
> On 08/02/2024 5:07, Joe Damato wrote:
>> Make mlx5 compatible with the newly added netlink queue GET APIs.
>> Signed-off-by: Joe Damato <jdamato@fastly.com>
>> ---

Just came back from testing this code. Let me make one cosmetic point
here. I noticed this patch has a line that is past 90 characters. Would
be nice to get it wrapped in the next version. We use 90 instead of 80
characters for the line wrap in the mlx5 driver because of firmware
command interface related code would lead to very hard to read lines if
wrapped at 80.

  https://patchwork.kernel.org/project/netdevbpf/patch/20240208030702.27296-1-jdamato@fastly.com/

>> v2 -> v3:
>>    - Fix commit message subject
>>    - call netif_queue_set_napi in mlx5e_ptp_activate_channel and
>>      mlx5e_ptp_deactivate_channel to enable/disable NETDEV_QUEUE_TYPE_RX for
>>      the PTP channel.
>>    - Modify mlx5e_activate_txqsq and mlx5e_deactivate_txqsq to set
>>      NETDEV_QUEUE_TYPE_TX which should take care of all TX queues including
>>      QoS/HTB and PTP.
>>    - Rearrange mlx5e_activate_channel and mlx5e_deactivate_channel for
>>      better ordering when setting and unsetting NETDEV_QUEUE_TYPE_RX NAPI
>>      structs
>> v1 -> v2:
>>    - Move netlink NULL code to mlx5e_deactivate_channel
>>    - Move netif_napi_set_irq to mlx5e_open_channel and avoid storing the
>>      irq, after netif_napi_add which itself sets the IRQ to -1
>>   drivers/net/ethernet/mellanox/mlx5/core/en/ptp.c  | 3 +++
>>   drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 7 +++++++
>>   2 files changed, 10 insertions(+)
>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/ptp.c
>> b/drivers/net/ethernet/mellanox/mlx5/core/en/ptp.c
>> index 078f56a3cbb2..fbbc287d924d 100644
>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en/ptp.c
>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/ptp.c
>> @@ -927,6 +927,8 @@ void mlx5e_ptp_activate_channel(struct mlx5e_ptp *c)
>>   	int tc;
>>     	napi_enable(&c->napi);
>> +	netif_queue_set_napi(c->netdev, c->rq.ix, NETDEV_QUEUE_TYPE_RX,
>> +			     &c->napi);

This should only be set if MLX5E_PTP_STATE_RX is set. Otherwise, the rq
is not initialized. The following callgraph should help illustrate this.

   mlx5e_ptp_open
    |_ mlx5e_ptp_open_queues
       |_ mlx5e_ptp_open_rq
          |_ mlx5e_open_rq

>>     	if (test_bit(MLX5E_PTP_STATE_TX, c->state)) {
>>   		for (tc = 0; tc < c->num_tc; tc++)
>> @@ -951,6 +953,7 @@ void mlx5e_ptp_deactivate_channel(struct mlx5e_ptp *c)
>>   			mlx5e_deactivate_txqsq(&c->ptpsq[tc].txqsq);
>>   	}
>>   +	netif_queue_set_napi(c->netdev, c->rq.ix, NETDEV_QUEUE_TYPE_RX, NULL);

I believe it would be best to tie this to whether MLX5E_PTP_STATE_RX is
set or not.

>>   	napi_disable(&c->napi);
>>   }
>>   diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>> b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>> index c8e8f512803e..2f1792854dd5 100644
>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>> @@ -1806,6 +1806,7 @@ void mlx5e_activate_txqsq(struct mlx5e_txqsq *sq)
>>   	set_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
>>   	netdev_tx_reset_queue(sq->txq);
>>   	netif_tx_start_queue(sq->txq);
>> +	netif_queue_set_napi(sq->channel->netdev, sq->txq_ix, NETDEV_QUEUE_TYPE_TX, &sq->channel->napi);
>
> Might be called with channel==NULL.
> For example for PTP.
>
> Prefer sq->netdev and sq->cq.napi.
>
>>   }
>>     void mlx5e_tx_disable_queue(struct netdev_queue *txq)
>> @@ -1819,6 +1820,7 @@ void mlx5e_deactivate_txqsq(struct mlx5e_txqsq *sq)
>>   {
>>   	struct mlx5_wq_cyc *wq = &sq->wq;
>>   +	netif_queue_set_napi(sq->channel->netdev, sq->txq_ix,
>> NETDEV_QUEUE_TYPE_TX, NULL);
>
> Same here.
>
>>   	clear_bit(MLX5E_SQ_STATE_ENABLED, &sq->state);
>>   	synchronize_net(); /* Sync with NAPI to prevent netif_tx_wake_queue. */
>>   @@ -2560,6 +2562,7 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv,
>> int ix,
>>   	c->lag_port = mlx5e_enumerate_lag_port(priv->mdev, ix);
>>     	netif_napi_add(netdev, &c->napi, mlx5e_napi_poll);
>> +	netif_napi_set_irq(&c->napi, irq);
>>     	err = mlx5e_open_queues(c, params, cparam);
>>   	if (unlikely(err))
>> @@ -2602,12 +2605,16 @@ static void mlx5e_activate_channel(struct mlx5e_channel *c)
>>   		mlx5e_activate_xsk(c);
>>   	else
>>   		mlx5e_activate_rq(&c->rq);
>> +
>> +	netif_queue_set_napi(c->netdev, c->ix, NETDEV_QUEUE_TYPE_RX, &c->napi);
>>   }
>>     static void mlx5e_deactivate_channel(struct mlx5e_channel *c)
>>   {
>>   	int tc;
>>   +	netif_queue_set_napi(c->netdev, c->ix, NETDEV_QUEUE_TYPE_RX, NULL);
>> +
>>   	if (test_bit(MLX5E_CHANNEL_STATE_XSK, c->state))
>>   		mlx5e_deactivate_xsk(c);
>>   	else

--
Thanks,

Rahul Rameshbabu