Received-SPF: pass (google.com: domain of linux-kernel+bounces-55544-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <44d321bf-88a0-4d6f-8572-dfbda088dd8f@nvidia.com>
Date: Tue, 6 Feb 2024 21:10:27 +0200
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH net-next] eth: mlx5: link NAPI instances to queues and
 IRQs
Content-Language: en-US
To: Joe Damato <jdamato@fastly.com>, Tariq Toukan <ttoukan.linux@gmail.com>
Cc: linux-kernel@vger.kernel.org, netdev@vger.kernel.org,
 Gal Pressman <gal@nvidia.com>, rrameshbabu@nvidia.com,
 Saeed Mahameed <saeedm@nvidia.com>, Leon Romanovsky <leon@kernel.org>,
 "David S. Miller" <davem@davemloft.net>, Eric Dumazet <edumazet@google.com>,
 Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>,
 "open list:MELLANOX MLX5 core VPI driver" <linux-rdma@vger.kernel.org>
References: <20240206010311.149103-1-jdamato@fastly.com>
 <7e338c2a-6091-4093-8ca2-bb3b2af3e79d@gmail.com>
 <20240206171159.GA11565@fastly.com>
From: Tariq Toukan <tariqt@nvidia.com>
In-Reply-To: <20240206171159.GA11565@fastly.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?dHJ2OEdYUERnVzFnc3BEcnFjZC9saVRrd3hqQTRBT1lZVHhhVkk0ZWc3MG44?=
 =?utf-8?B?eTVzSlRyaUNMOXFPYXpCbkJua0RGYm40aWo0WjlsbFMzek5QVVQ3UHF5STh6?=
 =?utf-8?B?SU5mQU9VZUxqSDdLZllYMVRBd3Rtd3NsUVlrKzl3bFNmcnJjUWFRS3RGdkk4?=
 =?utf-8?B?cys0a1RYb3ovZUZGVnJSOVM0RlpRRy9BMllWaVNDWE1sY3ZlWlhHZEFvMUJC?=
 =?utf-8?B?Zk5DaTBHUnRtemp6dmtOSFYxYU9DNjhuTkh0WEMySkg4MFdWZ05CZUlDNUJo?=
 =?utf-8?B?ZTNkNmc1SFRHN0xneXZ4bTA2R09KRllhWC9OeXRxRjNkb2FlWERGY1FucWZU?=
 =?utf-8?B?S1VFanUwQ2U0K1diNmg5WGw4NnV3a2NvU3dNSTdPMXg4MG1kMHdSbDBDRTlY?=
 =?utf-8?B?UXg2aTlPbzZqbUxVMTRmOFg1Szl3SnVQUEdYaFhtc1RDU3BCMWhPdWR1ajZ1?=
 =?utf-8?B?R1Y5OEtSRndvdXRSaGJpbGlZK1RoNUdtSnN5bi9ONjR1RW5mYks0d0dYWHZV?=
 =?utf-8?B?ZERjL1ZPcXJjNVpNbzNMc1BWa1diZzVFVmhQclZ0ZGIrZWJVTDBtYWhGdHZV?=
 =?utf-8?B?QXJMeWtNMmtUb1pSdVBHd3IyNEluaXk5WnVqOVJJOGRDeUdlUllwWWdNazkx?=
 =?utf-8?B?M214VkwwSXhPVWwzZ2k2Y3lKeGI2OVBMRUc1bTNWMVlVU28vd3Z2cm5Fd21H?=
 =?utf-8?B?cUdOVGsyN28xTE45anNvTDBzZU9vUjBMeXhhQTdPTDY1QmUyUUNGdTVBY2ZH?=
 =?utf-8?B?Yk80aHpzSnNMNGZmdkdYUDFrSWFPaXlmcGREaEhSaEpaTWQzZGYrL013djdT?=
 =?utf-8?B?Q1lXNzUxVG1tdlY5cDlZbFNqNGE0T3BiNDEyZW00WXhNSWMrdkZzeUZVVFlr?=
 =?utf-8?B?ajVDaDRXZVIzaE45SDNUTjcvSVF0QmlRSFNzOFhmVEVWc3hvUXVLZDdxeU1h?=
 =?utf-8?B?cmp3YUsyYXJDMkdjWE1xZ1NraUQ4VU5LZWRPMDRteGlpLzBlOC92QTQvcFBm?=
 =?utf-8?B?UFc5aHhCMUJvZ3NMNHh1T29mWm1lNEtTdlNoV2RJcXJnajRTb09TWW04UkUv?=
 =?utf-8?B?R3pUQ0ZKdHRNanhTOEs0M2ppbWJUbnZObXpCTUUrZGhNRkIrVUpNZGRpcXRL?=
 =?utf-8?B?dEtlV0xDZEhLNS9nOTBtRTBLd2x0cG1uQ2UrSG5NaHhUYldoMUp1TnZrK3dy?=
 =?utf-8?B?ZU81WEhFU0JUaHhqNHJheUdlQXVjeTVRUHR6Ulh4WGIzUTh0c2JaY3RlWVVW?=
 =?utf-8?B?RCt6a0pjK0ZOWUpWTUxTR1hkZHBMVGxFb3lndzB0d3h0eDhwU3Raa1o5RHNm?=
 =?utf-8?B?M2dLSWM5Y2t6WUl2SVFIN3VkbUI1Y2hMNzdSQ1JJL0M3RlRTa0FZSTlGOHlp?=
 =?utf-8?B?eWNnbGlYQXpGbVhHZW5aTmd1amZ6K0NPYnhDajBPUG8xbEs3d3VrRkN3UUFG?=
 =?utf-8?B?RTg2dDdabGs3Vm50LytGcDRQcUtWQVdsMmJmKzVEWjhSUkF3ZzBrTE9kUEZI?=
 =?utf-8?B?M2FCM2pRYThGZHhWS1RCMlNmMENZZXhWa0VQVnRhanc2eVhRWnRQcno5MnZo?=
 =?utf-8?B?Q1JQb2JaZWpsR1REd0piMlBqSDFGMVgvK1BQaGJ5Z3hoYWlVamZzMkU0ZG55?=
 =?utf-8?B?RlgwK0RrSTVzOFE3VXpDNThQaFp0cFZjWUFqcTZHU0J2a09SUjFPN2kveHBq?=
 =?utf-8?B?VVVDbXV1SHZqcm9VakM1YXpoNDJXdW1UMGVFZ2F0Ujh6Y1htS1d5bS9xTWxH?=
 =?utf-8?B?UW9TalMzMHR3Uy9zN21vdWRXaEJJNFcrcG9NdW5mS3ZrWVYzck43bCtUUjBh?=
 =?utf-8?B?d3NhS2w0YjRWY3M2M05Na2tpcUNObnpMSG84bHVJL1I4UmRTUzZyVnhISmRi?=
 =?utf-8?B?eWZKSDJWOUlzenJ5SlBGc2JTQlNqUmtXb3NhYnN0cVZDdlZJa0FzbFRnbkRY?=
 =?utf-8?B?R1FsR3ljQ09jK1liQmRaenBDclBWaUcxV1NhenJUNUVBUmJaVmlkbndmajla?=
 =?utf-8?B?ZkROaFBqQ09SMTg2eis4M3hoNCtVV3c1dmVpUGRubGVNK1U5WGNxZitlMGRL?=
 =?utf-8?B?T2p6OGZjYTdRT3BrR2FqcXBrcmtEWlZCUUVIZzRyTXJFSnQzejk5OU9OQXFa?=
 =?utf-8?Q?Tqr7DIq4AYDHQvHJEi1LkOdPG?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: acb36ed6-315d-47ce-a64c-08dc27474b26
X-MS-Exchange-CrossTenant-AuthSource: IA1PR12MB6018.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2024 19:10:33.8647
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: k2Ia47vKvaBE+JSRHy50lAneShnWO06GB3vF/r2E8UP8VEiXPrFBLaSYdl1eyctWhbOAyIJ7s7TFcqSk1OpT+Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7196


On 06/02/2024 19:12, Joe Damato wrote:
> On Tue, Feb 06, 2024 at 10:11:28AM +0200, Tariq Toukan wrote:
>>
>>
>> On 06/02/2024 3:03, Joe Damato wrote:
>>> Make mlx5 compatible with the newly added netlink queue GET APIs.
>>>
>>> Signed-off-by: Joe Damato <jdamato@fastly.com>
>>
>> + Gal
>>
>> Hi Joe,
>> Thanks for your patch.
>>
>> We have already prepared a similar patch, and it's part of our internal
>> submission queue, and planned to be submitted soon.
>>
>> Please see my comments below, let us know if you're welling to respin a V2
>> or wait for our patch.
> 
> Do you have a rough estimate on when it'll be submitted?
> 
> If it's several months out I'll try again, but if it's expected to be
> submit in the next few weeks I'll wait for your official change.

It'll be in the next few weeks.

> 
> BTW, are the per queue coalesce changes in that same queue? It was
> mentioned previously [1] that this feature is coming after we submit a
> simple attempt as an RFC. If that feature isn't planned or won't be submit
> anytime soon, can you let us know and we can try to attempt an RFC v3 for
> it?
> 

The per queue coalesce series is going through internal code review, and 
is expected to also be ready in a matter of a few weeks.

> [1]: https://lore.kernel.org/lkml/874jj34e67.fsf@nvidia.com/
> 
>>> ---
>>>   drivers/net/ethernet/mellanox/mlx5/core/en.h      | 1 +
>>>   drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 8 ++++++++
>>>   2 files changed, 9 insertions(+)
>>>
>>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
>>> index 55c6ace0acd5..3f86ee1831a8 100644
>>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
>>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
>>> @@ -768,6 +768,7 @@ struct mlx5e_channel {
>>>   	u16                        qos_sqs_size;
>>>   	u8                         num_tc;
>>>   	u8                         lag_port;
>>> +	unsigned int		   irq;
>>>   	/* XDP_REDIRECT */
>>>   	struct mlx5e_xdpsq         xdpsq;
>>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>>> index c8e8f512803e..e1bfff1fb328 100644
>>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>>> @@ -2473,6 +2473,9 @@ static void mlx5e_close_queues(struct mlx5e_channel *c)
>>>   	mlx5e_close_tx_cqs(c);
>>>   	mlx5e_close_cq(&c->icosq.cq);
>>>   	mlx5e_close_cq(&c->async_icosq.cq);
>>> +
>>> +	netif_queue_set_napi(c->netdev, c->ix, NETDEV_QUEUE_TYPE_TX, NULL);
>>> +	netif_queue_set_napi(c->netdev, c->ix, NETDEV_QUEUE_TYPE_RX, NULL);
>>>   }
>>>   static u8 mlx5e_enumerate_lag_port(struct mlx5_core_dev *mdev, int ix)
>>> @@ -2558,6 +2561,7 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
>>>   	c->stats    = &priv->channel_stats[ix]->ch;
>>>   	c->aff_mask = irq_get_effective_affinity_mask(irq);
>>>   	c->lag_port = mlx5e_enumerate_lag_port(priv->mdev, ix);
>>> +	c->irq		= irq;
>>>   	netif_napi_add(netdev, &c->napi, mlx5e_napi_poll);
>>> @@ -2602,6 +2606,10 @@ static void mlx5e_activate_channel(struct mlx5e_channel *c)
>>>   		mlx5e_activate_xsk(c);
>>>   	else
>>>   		mlx5e_activate_rq(&c->rq);
>>> +
>>> +	netif_napi_set_irq(&c->napi, c->irq);
>>
>> Can be safely moved to mlx5e_open_channel without interfering with other
>> existing mapping. This would save the new irq field in mlx5e_channel.
> 
> Sure, yea, I have that change queued already from last night.
> 

I see now.. I replied before noticing it.

>>> +	netif_queue_set_napi(c->netdev, c->ix, NETDEV_QUEUE_TYPE_TX, &c->napi);
>>
>> In some configurations we have multiple txqs per channel, so the txq indices
>> are not 1-to-1 with their channel index.
>>
>> This should be called per each txq, with the proper txq index.
>>
>> It should be done also for feture-dedicated SQs (like QOS/HTB).
> 
> OK. I think the above makes sense and I'll look into it if I have some time
> this week.
>   
>>> +	netif_queue_set_napi(c->netdev, c->ix, NETDEV_QUEUE_TYPE_RX, &c->napi);
>>
>> For consistency, I'd move this one as well, to match the TX handling.
> 
> Sure.
> 
>>>   }
>>>   static void mlx5e_deactivate_channel(struct mlx5e_channel *c)