Received-SPF: pass (google.com: domain of linux-kernel+bounces-138224-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Message-ID: <bb0fbd29-c098-4a62-9217-c9fd1a450250@intel.com>
Date: Wed, 10 Apr 2024 02:10:52 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [intel-next 1/2] net/i40e: link NAPI instances to queues and IRQs
To: Joe Damato <jdamato@fastly.com>, <linux-kernel@vger.kernel.org>,
	<netdev@vger.kernel.org>, <intel-wired-lan@lists.osuosl.org>
CC: <sridhar.samudrala@intel.com>, <nalramli@fastly.com>, Jesse Brandeburg
	<jesse.brandeburg@intel.com>, Tony Nguyen <anthony.l.nguyen@intel.com>,
	"David S. Miller" <davem@davemloft.net>, Eric Dumazet <edumazet@google.com>,
	Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>
References: <20240410043936.206169-1-jdamato@fastly.com>
 <20240410043936.206169-2-jdamato@fastly.com>
Content-Language: en-US
From: "Nambiar, Amritha" <amritha.nambiar@intel.com>
In-Reply-To: <20240410043936.206169-2-jdamato@fastly.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?THkrWlRuTnNJRmxkYkttVjgrMUhhZXdwRVBXa21HTm5ZYzVnd2Q2RUduL3lU?=
 =?utf-8?B?R3hmekk2RllOQmZtK25XWDdDL2tvS3BKOEZwNXpXQkk2NkhVaHhEQVd6T3Qv?=
 =?utf-8?B?NFNZU1JtMytyWWI1STAxT2lkYzA5VEtxSTNNRGdsTDdUUUJoL1NQNUFpd0lj?=
 =?utf-8?B?ZzVGNnNpWitxWEd0cmduRDJScXZQdENsMlBiL3Izd2FGRGhvcmFxc2NKL3Ro?=
 =?utf-8?B?cjcyTFZMZkFiSy9KS1I3Tm8wTUJ0aytDclptTkZCVDYvQ3loaXZRNmdQVWZY?=
 =?utf-8?B?cDdyUkx6eTBXWHlwdWs3aFhpdDN5cFlOaVpLcEtROVM1WjNFU3MwSk80V1Vw?=
 =?utf-8?B?VWNsTFFZQkF3WGZOb0ZibmxFQlBMQnVKbS9ySUJ0QXpEaEhLZUFUNGRCVEU3?=
 =?utf-8?B?QXYvYzBwckx4OGZRMXNZN1BSK3JjN0ZyNjJBbWw4L1h1WnZiR0pwN2wxVG94?=
 =?utf-8?B?M2p4dUZ4bkJqa1FXZ1VReHBhd0FBSTNldDd6Sk8wb0dRZTBLNEVObURuWEdn?=
 =?utf-8?B?WkdRcFpkdnlUY3paQWppOHF3M3lSaXlWQ3dVWjBUalJjMDdEdjFFMUhiZkpO?=
 =?utf-8?B?NFI5UUNwbUY1ZFBDSXh1UEErZzZaRnVOM2x3T0NmK3hkdE5SbXJmOGZGQkxM?=
 =?utf-8?B?SzYraUhQTjI2SFVzbmwwZldwVkVGV1Y0UEdCdXZBYnpDa0pFRWNuaHlwVnNs?=
 =?utf-8?B?S3FnRStzVUFLQmNvTE43UXJvRFhhTTUwaS80c1dVd3RoOVpZamJkQXpqVGdM?=
 =?utf-8?B?SWpsZGhPQUsybTJXV3l3TFkreFVQUG1naXl1V0VjQ3ZjSVF5V0VKVGR5Qkds?=
 =?utf-8?B?ME9uL24rN2tGV3Q5SjRYdFQxSGtZYU9HMXJGVTlWeWpoMTcxdXRLaFMrSGdu?=
 =?utf-8?B?Z2hMcFRTN2pDM1ZiZTQzRVg4VU9YOUI3RXlpemZWeGhvSTY1bDdha1dMeEwy?=
 =?utf-8?B?S2FvcVRRYlAwRS90MzEvVUNuVGk4UVA3aUdPNWt5aG0rdWhobTBPQkljb3hD?=
 =?utf-8?B?ZkkrRExNWlhwRkdSbEZrQnMzNzJFZ3lSR1NNb1d2Z2JZSVBCNmY0RVNJNTl2?=
 =?utf-8?B?MlhyRXZiOG1jOXg2TXRBSG9qY2ZubVJ0VDgrYmVLNUozeUo2elZPNHR0bHZj?=
 =?utf-8?B?SExxcDFocTdrQ1JRcktkSERlVzA1WTlWR3dTNVNRNWFxemVENENxaWJlbzgw?=
 =?utf-8?B?VERhTW9PNUxja2ZidDJEWlVFczBSOXZDY3ZUZVhIRVg1WjFUOE1CNk9RUW42?=
 =?utf-8?B?MmRZQ2NZNVB6QXFULzZacC8zVjFkODUyL1cxUnJmMjdCSzlpZHVsUEJIMmp2?=
 =?utf-8?B?NjNyeDgyYmhMeXJtYTg3LytOY2liV0dhaVJyZlRHdktJKzBwSSs3QjR3eUww?=
 =?utf-8?B?WU83eGFGN0JXQ3ZKZHFMaEdzOC9ySzI2ZWVtcFEydkNSQVE5eDFlekQzL0cy?=
 =?utf-8?B?MmRNQkppWm43RjRWbDN0NnlvaWp1NVJ6WXl1ZUFLY3o2b05oUUh2S0FjUDJm?=
 =?utf-8?B?QlBvNy9jRUl0WmdzUWc4M0IrVVNuM1J1RjZuYjJvRHRtQlhFNHgrUUJ4RG9n?=
 =?utf-8?B?KzhDeHJIaXZWdGVJWnRub0ZhSGM2UmdTRDVBOUd2OXlubHhDMXJEQXdPbWdn?=
 =?utf-8?B?YXcwcG1palQzZzcrN2J2T2NpWWJMTE9lMTJQVi93VG0wdlZBSk81UlpWTCtt?=
 =?utf-8?B?SzI5UEREOFdlelY3bGQ0SERzZmdXYWdUajZxVWR0Sno4NDJZdUNmbG1ON1RD?=
 =?utf-8?B?R1pjTkVCd1REZWxWUFJzYWVQMVlYUi90M3U5ZHE4VWJaSlphTS93eDB2UWY1?=
 =?utf-8?B?U01uWVIrRWZnb2lEUGZPRWQvN0dYMUdQNWNFUGJZTWlTdlVXa0VsRlBhMHh5?=
 =?utf-8?B?ekhHS1NhanJUS2gzSG5SMFoxOHJ0VHEyYVVOMzNQRVFWMVVTNTlCYlE4VHFw?=
 =?utf-8?B?bHZibmt6S255N1p0WmZCVVJXSzF4b2QzeklKNHBJYWVKQkhZdXF4WnJWd0xT?=
 =?utf-8?B?eVNXZTRoMm1mUHltVXFBdVllYUYrZ1hwdDcrVVV6ek1kY0VmTW1IbmdhTnMr?=
 =?utf-8?B?MGFZUmFVMmhWUVVha21YVTdMTUErUFU1YkhmNlY0SjE3NTRZcG56M3lPZENR?=
 =?utf-8?B?V3h6NEdnSFZwbHV1NXc0Z1lsV1RHVDl2Z0JYRkdwUXJZcUd4bDE4T2lrOHRH?=
 =?utf-8?B?UEE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: d0b54bf2-5912-443e-782d-08dc593e2124
X-MS-Exchange-CrossTenant-AuthSource: IA1PR11MB7869.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Apr 2024 09:10:56.0889
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: sRgE1uuO1ArTl0tpfzEjHDI9G01Obj4Fsl9XSsFzx36X6m8sm0OBLxZCobC88cf4DXUzQDdJ6lFu1O9dqgvb1N7n7/UDnzNWLdAGtrP2WpM=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR11MB5971
X-OriginatorOrg: intel.com

On 4/9/2024 9:39 PM, Joe Damato wrote:
> Make i40e compatible with the newly added netlink queue GET APIs.
> 
> $ ./cli.py --spec ../../../Documentation/netlink/specs/netdev.yaml \
>    --do queue-get --json '{"ifindex": 3, "id": 1, "type": "rx"}'
> 
> {'id': 1, 'ifindex': 3, 'napi-id': 162, 'type': 'rx'}
> 
> $ ./cli.py --spec ../../../Documentation/netlink/specs/netdev.yaml \
>    --do napi-get --json '{"id": 162}'
> 
> {'id': 162, 'ifindex': 3, 'irq': 136}
> 
> The above output suggests that irq 136 was allocated for queue 1, which has
> a NAPI ID of 162.
> 
> To double check this is correct, the IRQ to queue mapping can be verified
> by checking /proc/interrupts:
> 
> $ cat /proc/interrupts  | grep 136\: | \
>    awk '{print "irq: " $1 " name " $76}'
> 
> irq: 136: name i40e-vlan300-TxRx-1
> 
> Suggests that queue 1 has IRQ 136, as expected.
> 
> Signed-off-by: Joe Damato <jdamato@fastly.com>
> ---
>   drivers/net/ethernet/intel/i40e/i40e.h      |  2 +
>   drivers/net/ethernet/intel/i40e/i40e_main.c | 58 +++++++++++++++++++++
>   drivers/net/ethernet/intel/i40e/i40e_txrx.c |  4 ++
>   3 files changed, 64 insertions(+)
> 
> diff --git a/drivers/net/ethernet/intel/i40e/i40e.h b/drivers/net/ethernet/intel/i40e/i40e.h
> index 2fbabcdb5bb5..5900ed5c7170 100644
> --- a/drivers/net/ethernet/intel/i40e/i40e.h
> +++ b/drivers/net/ethernet/intel/i40e/i40e.h
> @@ -1267,6 +1267,8 @@ int i40e_ioctl(struct net_device *netdev, struct ifreq *ifr, int cmd);
>   int i40e_open(struct net_device *netdev);
>   int i40e_close(struct net_device *netdev);
>   int i40e_vsi_open(struct i40e_vsi *vsi);
> +void i40e_queue_set_napi(struct i40e_vsi *vsi, unsigned int queue_index,
> +			 enum netdev_queue_type type, struct napi_struct *napi);
>   void i40e_vlan_stripping_disable(struct i40e_vsi *vsi);
>   int i40e_add_vlan_all_mac(struct i40e_vsi *vsi, s16 vid);
>   int i40e_vsi_add_vlan(struct i40e_vsi *vsi, u16 vid);
> diff --git a/drivers/net/ethernet/intel/i40e/i40e_main.c b/drivers/net/ethernet/intel/i40e/i40e_main.c
> index 0bdcdea0be3e..6384a0c73a05 100644
> --- a/drivers/net/ethernet/intel/i40e/i40e_main.c
> +++ b/drivers/net/ethernet/intel/i40e/i40e_main.c
> @@ -3448,6 +3448,58 @@ static struct xsk_buff_pool *i40e_xsk_pool(struct i40e_ring *ring)
>   	return xsk_get_pool_from_qid(ring->vsi->netdev, qid);
>   }
>   
> +/**
> + * __i40e_queue_set_napi - Set the napi instance for the queue
> + * @dev: device to which NAPI and queue belong
> + * @queue_index: Index of queue
> + * @type: queue type as RX or TX
> + * @napi: NAPI context
> + * @locked: is the rtnl_lock already held
> + *
> + * Set the napi instance for the queue. Caller indicates the lock status.
> + */
> +static void
> +__i40e_queue_set_napi(struct net_device *dev, unsigned int queue_index,
> +		      enum netdev_queue_type type, struct napi_struct *napi,
> +		      bool locked)
> +{
> +	if (!locked)
> +		rtnl_lock();
> +	netif_queue_set_napi(dev, queue_index, type, napi);
> +	if (!locked)
> +		rtnl_unlock();
> +}
> +
> +/**
> + * i40e_queue_set_napi - Set the napi instance for the queue
> + * @vsi: VSI being configured
> + * @queue_index: Index of queue
> + * @type: queue type as RX or TX
> + * @napi: NAPI context
> + *
> + * Set the napi instance for the queue. The rtnl lock state is derived from the
> + * execution path.
> + */
> +void
> +i40e_queue_set_napi(struct i40e_vsi *vsi, unsigned int queue_index,
> +		    enum netdev_queue_type type, struct napi_struct *napi)
> +{
> +	struct i40e_pf *pf = vsi->back;
> +
> +	if (!vsi->netdev)
> +		return;
> +
> +	if (current_work() == &pf->service_task ||
> +	    test_bit(__I40E_PF_RESET_REQUESTED, pf->state) ||

I think we might need something like ICE_PREPARED_FOR_RESET which 
detects all kinds of resets(PFR/CORE/GLOBR). __I40E_PF_RESET_REQUESTED 
handles PFR only. So, this might assert for RTNL lock on CORER/GLOBR.

> +	    test_bit(__I40E_DOWN, pf->state) ||
> +	    test_bit(__I40E_SUSPENDED, pf->state))
> +		__i40e_queue_set_napi(vsi->netdev, queue_index, type, napi,
> +				      false);
> +	else
> +		__i40e_queue_set_napi(vsi->netdev, queue_index, type, napi,
> +				      true);
> +}
> +
>   /**
>    * i40e_configure_tx_ring - Configure a transmit ring context and rest
>    * @ring: The Tx ring to configure
> @@ -3558,6 +3610,8 @@ static int i40e_configure_tx_ring(struct i40e_ring *ring)
>   	/* cache tail off for easier writes later */
>   	ring->tail = hw->hw_addr + I40E_QTX_TAIL(pf_q);
>   
> +	i40e_queue_set_napi(vsi, ring->queue_index, NETDEV_QUEUE_TYPE_TX,
> +			    &ring->q_vector->napi);

I am not sure very sure of this, have you tested this for the 
reset/rebuild path as well (example: ethtool -L and change queues). Just 
wondering if this path is taken for first time VSI init or additionally 
for any VSI rebuilds as well.

>   	return 0;
>   }
>   
> @@ -3716,6 +3770,8 @@ static int i40e_configure_rx_ring(struct i40e_ring *ring)
>   			 ring->queue_index, pf_q);
>   	}
>   
> +	i40e_queue_set_napi(vsi, ring->queue_index, NETDEV_QUEUE_TYPE_RX,
> +			    &ring->q_vector->napi);
>
Same as above.

   	return 0;
>   }
>   
> @@ -4178,6 +4234,8 @@ static int i40e_vsi_request_irq_msix(struct i40e_vsi *vsi, char *basename)
>   		q_vector->affinity_notify.notify = i40e_irq_affinity_notify;
>   		q_vector->affinity_notify.release = i40e_irq_affinity_release;
>   		irq_set_affinity_notifier(irq_num, &q_vector->affinity_notify);
> +		netif_napi_set_irq(&q_vector->napi, q_vector->irq_num);
> +
>   		/* Spread affinity hints out across online CPUs.
>   		 *
>   		 * get_cpu_mask returns a static constant mask with
> diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.c b/drivers/net/ethernet/intel/i40e/i40e_txrx.c
> index 64d198ed166b..d380885ff26d 100644
> --- a/drivers/net/ethernet/intel/i40e/i40e_txrx.c
> +++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.c
> @@ -821,6 +821,8 @@ void i40e_clean_tx_ring(struct i40e_ring *tx_ring)
>   void i40e_free_tx_resources(struct i40e_ring *tx_ring)
>   {
>   	i40e_clean_tx_ring(tx_ring);
> +	i40e_queue_set_napi(tx_ring->vsi, tx_ring->queue_index,
> +			    NETDEV_QUEUE_TYPE_TX, NULL);
>   	kfree(tx_ring->tx_bi);
>   	tx_ring->tx_bi = NULL;
>   
> @@ -1526,6 +1528,8 @@ void i40e_clean_rx_ring(struct i40e_ring *rx_ring)
>   void i40e_free_rx_resources(struct i40e_ring *rx_ring)
>   {
>   	i40e_clean_rx_ring(rx_ring);
> +	i40e_queue_set_napi(rx_ring->vsi, rx_ring->queue_index,
> +			    NETDEV_QUEUE_TYPE_RX, NULL);
>   	if (rx_ring->vsi->type == I40E_VSI_MAIN)
>   		xdp_rxq_info_unreg(&rx_ring->xdp_rxq);
>   	rx_ring->xdp_prog = NULL;