Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 11 May 2023 15:41:10 +0200
From:   Michal Kubiak <michal.kubiak@intel.com>
To:     Feng Liu <feliu@nvidia.com>
CC:     <virtualization@lists.linux-foundation.org>,
        <netdev@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        <bpf@vger.kernel.org>, Jason Wang <jasowang@redhat.com>,
        "Michael S . Tsirkin" <mst@redhat.com>,
        Xuan Zhuo <xuanzhuo@linux.alibaba.com>,
        Simon Horman <simon.horman@corigine.com>,
        Bodong Wang <bodong@nvidia.com>, Jiri Pirko <jiri@nvidia.com>
Subject: Re: [PATCH net v4] virtio_net: Fix error unwinding of XDP
 initialization
Message-ID: <ZFzwdkJv7eHGeuaM@localhost.localdomain>
References: <20230508222708.68281-1-feliu@nvidia.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230508222708.68281-1-feliu@nvidia.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?uHVfp17Qv302IOfbLbkADZ3Nqeg8ymKkZWzyb/ukm7hCqAaqKBl+F5GiYc7p?=
 =?us-ascii?Q?C0v3Fye/5BGNnl0iMmJL131xFQNA/tweb6/HooZ3LafQJsUZK4hxCwcIcreL?=
 =?us-ascii?Q?szYDYMyoJGNFhH2tfsAT27ZVvyte+RtfqW4RrFa3BYkUgiy3gSVhiFmxnY75?=
 =?us-ascii?Q?ise39DeESX9qSbOE6ApwLy9dFQud4ymtfjfVeogs+aHYhqbkstFUO5k8qwwC?=
 =?us-ascii?Q?tWSJf8Y2eipfjDK4ekcR2IwDadiFiZwYmZkVTwEcw7BheYNHU1lnhAd/rpLr?=
 =?us-ascii?Q?ais8jw/CMsSDuDYbywPPvdZo9FRkipg9SScOCjxBfSQ+KmQEEMzSMeqmMBG9?=
 =?us-ascii?Q?ulk45xCAkpyEM/zFOPSh1OkPJZ4LNkN89TGMI47yxBMebLbXgCJ7m1RKLbUu?=
 =?us-ascii?Q?at2xv5J4hNrC9ayJBCEDnGLdaiLcKx2DoH18AQuokIAchc1Ua42PpvJ6QUks?=
 =?us-ascii?Q?tizkzEghbVb73OZ4B0ZSZ/J9yu94Kyc+YITXVNb6oAY6gaGV8sD0oRGkhGwe?=
 =?us-ascii?Q?VIVYQma8cv+TCgBs7aELle/GwlZDVx5qK76vU1DdLxWijHMhsMi05N8OU5qg?=
 =?us-ascii?Q?0WnWytTkTZpyq3D5dJp+ZvhHZ77qzEvv1AL0pWltfgYj9380Sv1RXSgMVQkE?=
 =?us-ascii?Q?6p67HfQVHv220t4vcaqwtsfPmHzGioPEYYqEfhUD7SFwW6x4VgtQEJssRb5O?=
 =?us-ascii?Q?bLdnFJAIn6C+BhlgWEjasGh/HdMNGf6XndQU83VkLMH07XDjHllVVq6KMVsJ?=
 =?us-ascii?Q?/P1n20Apqc7h+/y0g0JZ5j7yYskxfzvUiPJKbnJaHBXzexmGJ7hwGAxNh3+e?=
 =?us-ascii?Q?1u0pe9WMHzDDkm6D21vS+Wmj/tQJgHfBZyvC1GxAgRlivB+fX1Fu7PKUGxkv?=
 =?us-ascii?Q?nKmDOxd5hn4pAFRh6cTNyUtoduavGfOQcBxb/eqjPmJD9WGnE/IkBjQqEMe1?=
 =?us-ascii?Q?B1Jl2hZHKsFB9GHrdRivqMLaxFeHsZGHJYyraVD/0ym3w5PdWzsfrawn4xnd?=
 =?us-ascii?Q?0+hWizlqMegG/DkvKRQ/8q+Bbi3yESjqGJd2Gn8w078J3teWpLiDQaflFpcw?=
 =?us-ascii?Q?P/hFsP0w6iCh7WXeY0ccpWE95c1U/IVWR8TW80bsrbDYi4tgQkRsfSwS/irs?=
 =?us-ascii?Q?k5jTHpaeNFe4wtMRuxkRL6JYPE6BjAbk32OahbnSscfO64UTAqret+8ZImWU?=
 =?us-ascii?Q?x3NErzNtlClfloVKgkFVQS4KACZPsIfa4LOhhlpfOS9bWWP/HpC1gVxBsCUK?=
 =?us-ascii?Q?6u6Uf+AlVCxHSevte1C4bGJoYWixTor9464qTKPgdYCGU2fBQXVZRWotideD?=
 =?us-ascii?Q?TJpo0M3Kw/1TeHp8N4I8iTkPYQcf/XcVk/Z/9/pGcWJdl+qDuJruSNppI1r2?=
 =?us-ascii?Q?Gz3N17i2Lv7j22EXPEhcyq2lZHo9AmwwXs5FIDXpVb+63mc9QuCWoscr4ieP?=
 =?us-ascii?Q?9D43V19apfGCAUc/2TevNi+iKA5BiJlz7LRJRemAIW3RczmaqxhaapMW2gjG?=
 =?us-ascii?Q?BiDPXfoc3Geq59/Fcc51BF7JBaPgaBprCtxGQpcbRNbA/tvMpeJnKwoBJOo9?=
 =?us-ascii?Q?E8HjGhjVsyUEkZ22IROmbWu4DjkocYjNABsspk5JJfzLjwa94ASHuuqiPwDv?=
 =?us-ascii?Q?qw=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: c609de2d-e00c-48f2-66ac-08db522568ea
X-MS-Exchange-CrossTenant-AuthSource: DM6PR11MB2937.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 May 2023 13:41:23.1992
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 9ow97CNPnXieA7yKLc0wifdX99sneDgVY48YymHtpgw+5hxYBxcxdGD1BP0+JTwDMeaDRJjTa9E0q2GjKOGxCQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB6704
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, May 08, 2023 at 06:27:08PM -0400, Feng Liu wrote:
> When initializing XDP in virtnet_open(), some rq xdp initialization
> may hit an error causing net device open failed. However, previous
> rqs have already initialized XDP and enabled NAPI, which is not the
> expected behavior. Need to roll back the previous rq initialization
> to avoid leaks in error unwinding of init code.
> 
> Also extract helper functions of disable and enable queue pairs.
> Use newly introduced disable helper function in error unwinding and
> virtnet_close. Use enable helper function in virtnet_open.
> 
> Fixes: 754b8a21a96d ("virtio_net: setup xdp_rxq_info")
> Signed-off-by: Feng Liu <feliu@nvidia.com>
> Reviewed-by: Jiri Pirko <jiri@nvidia.com>
> ---
> 
> v3 -> v4
> feedbacks from Jiri Pirko
> - Add symmetric helper function virtnet_enable_qp to enable queues.
> - Error handle:  cleanup current queue pair in virtnet_enable_qp,
>   and complete the reset queue pairs cleanup in virtnet_open.
> - Fix coding style.
> feedbacks from Parav Pandit
> - Remove redundant debug message and white space.
> 
> v2 -> v3
> feedbacks from Michael S. Tsirkin
> - Remove redundant comment.
> 
> v1 -> v2
> feedbacks from Michael S. Tsirkin
> - squash two patches together.
> 
> ---
>  drivers/net/virtio_net.c | 58 ++++++++++++++++++++++++++++------------
>  1 file changed, 41 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
> index 8d8038538fc4..df7c08048fa7 100644
> --- a/drivers/net/virtio_net.c
> +++ b/drivers/net/virtio_net.c
> @@ -1868,6 +1868,38 @@ static int virtnet_poll(struct napi_struct *napi, int budget)
>  	return received;
>  }
>  
> +static void virtnet_disable_qp(struct virtnet_info *vi, int qp_index)
> +{
> +	virtnet_napi_tx_disable(&vi->sq[qp_index].napi);
> +	napi_disable(&vi->rq[qp_index].napi);
> +	xdp_rxq_info_unreg(&vi->rq[qp_index].xdp_rxq);
> +}
> +
> +static int virtnet_enable_qp(struct virtnet_info *vi, int qp_index)
> +{
> +	struct net_device *dev = vi->dev;
> +	int err;
> +
> +	err = xdp_rxq_info_reg(&vi->rq[qp_index].xdp_rxq, dev, qp_index,
> +			       vi->rq[qp_index].napi.napi_id);
> +	if (err < 0)
> +		return err;
> +
> +	err = xdp_rxq_info_reg_mem_model(&vi->rq[qp_index].xdp_rxq,
> +					 MEM_TYPE_PAGE_SHARED, NULL);
> +	if (err < 0)
> +		goto err_xdp_reg_mem_model;
> +
> +	virtnet_napi_enable(vi->rq[qp_index].vq, &vi->rq[qp_index].napi);
> +	virtnet_napi_tx_enable(vi, vi->sq[qp_index].vq, &vi->sq[qp_index].napi);
> +
> +	return 0;
> +
> +err_xdp_reg_mem_model:
> +	xdp_rxq_info_unreg(&vi->rq[qp_index].xdp_rxq);

Is it really necessary to call 'xdp_rxq_info_unreg()' from here?
It seems there is a risk of calling that function twice if 'xdp_rxq_info_reg_mem_model()" fails.

For example in the following scenario:
	1. We call 'virtnet_enable_qp()' from 'virtnet_open()'
	2. 'xdp_rxq_info_reg()' succeeds.
	3. 'xdp_rxq_info_reg_mem_model()' fails, so we go to the label "err_xdp_info_reg_mem_model".
	4. 'xdp_rxq_info_unreg()' is called.
	5. Register state of 'xdp_rxq' changes to 'REG_STATE_UNREGISTERED'.
	6. 'virtnet_enable_qp()' returns an error.
	7. In 'virtnet_open()' we go to the "err_enable_qp" label.
	8. 'virtnet_disable_qp()' is called.
	9. 'xdp_rxq_info_unreg()' is called for the second time on the xdp_rxq which is already
	   in state 'REG_STATE_UNREGISTERED'.
	10. The following warning from 'xdp_rxq_info_unreg_mem_model' can be displayed:

		if (xdp_rxq->reg_state != REG_STATE_REGISTERED)
		{
			WARN(1, "Missing register, driver bug");
			return;
		}

I think the 'xdp_rxq_info_unreg()' can be called only once for the same 'xdp_rxq'.
I believe we should either:
	- remove that call from 'virtnet_enable_qp()',
	- or use the following API in 'virtnet_disable_qp()':
		'xdp_rxq_info_is_reg()'
	     to check if the xdp_rxq is actually registered.

Thanks,
Michal

> +	return err;
> +}
> +
>  static int virtnet_open(struct net_device *dev)
>  {
>  	struct virtnet_info *vi = netdev_priv(dev);
> @@ -1881,22 +1913,17 @@ static int virtnet_open(struct net_device *dev)
>  			if (!try_fill_recv(vi, &vi->rq[i], GFP_KERNEL))
>  				schedule_delayed_work(&vi->refill, 0);
>  
> -		err = xdp_rxq_info_reg(&vi->rq[i].xdp_rxq, dev, i, vi->rq[i].napi.napi_id);
> +		err = virtnet_enable_qp(vi, i);
>  		if (err < 0)
> -			return err;
> -
> -		err = xdp_rxq_info_reg_mem_model(&vi->rq[i].xdp_rxq,
> -						 MEM_TYPE_PAGE_SHARED, NULL);
> -		if (err < 0) {
> -			xdp_rxq_info_unreg(&vi->rq[i].xdp_rxq);
> -			return err;
> -		}
> -
> -		virtnet_napi_enable(vi->rq[i].vq, &vi->rq[i].napi);
> -		virtnet_napi_tx_enable(vi, vi->sq[i].vq, &vi->sq[i].napi);
> +			goto err_enable_qp;
>  	}
>  
>  	return 0;
> +
> +err_enable_qp:
> +	for (i--; i >= 0; i--)
> +		virtnet_disable_qp(vi, i);
> +	return err;
>  }
>  
>  static int virtnet_poll_tx(struct napi_struct *napi, int budget)
> @@ -2305,11 +2332,8 @@ static int virtnet_close(struct net_device *dev)
>  	/* Make sure refill_work doesn't re-enable napi! */
>  	cancel_delayed_work_sync(&vi->refill);
>  
> -	for (i = 0; i < vi->max_queue_pairs; i++) {
> -		virtnet_napi_tx_disable(&vi->sq[i].napi);
> -		napi_disable(&vi->rq[i].napi);
> -		xdp_rxq_info_unreg(&vi->rq[i].xdp_rxq);
> -	}
> +	for (i = 0; i < vi->max_queue_pairs; i++)
> +		virtnet_disable_qp(vi, i);
>  
>  	return 0;
>  }
> -- 
> 2.37.1 (Apple Git-137.1)
> 
>