Received: by 2002:a05:6a10:8c0a:0:0:0:0 with SMTP id go10csp82793pxb; Mon, 1 Feb 2021 23:09:34 -0800 (PST) X-Google-Smtp-Source: ABdhPJx00GixpOSqrhsQknyRookVeBmDworYaqDhCvb7Vy+ni5VGRMhJo3ZRINKBX66LFK8LHTfu X-Received: by 2002:a17:907:767c:: with SMTP id kk28mr10366526ejc.98.1612249774247; Mon, 01 Feb 2021 23:09:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1612249774; cv=none; d=google.com; s=arc-20160816; b=xgZZHfneNaSWhu4VDb5UhFIqW0CjckSNGxMEnWKscRDwWGmhD3gYcgxoBeiJBeqq0o psxJejG/+NUzuETQ1iwDeUDb+AywCEnH/8C7NY5l/LnUVpLsbybf4a4llnzLxnBOLc8D P8FJ3w31etme40AInULHBCG1PERD/j9fp1P1qz88G965KXlRBEN7eS6ZLp0oiKRjSnNm ofsVCS8iRbwcsjkeSesVz2m4iubOGftGrIbUcqwX5yHi9xnfWASQWBOHxn061CoU39so jsQNQcUBXnL9UFyowDSH5dgI2IATMm5OAqZUO11JIZPWPONEZDjRkIayrEG60Fk2F8KG I9Ew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:dkim-signature:user-agent:in-reply-to :content-transfer-encoding:content-disposition:mime-version :references:message-id:subject:cc:to:from:date; bh=oQ6Ri294/3mOiXs/oCOWdeD09SEKwcXEGPnVFpELei4=; b=0oZXapQgabKPXh7e12+8Ghyekv+pg5+T1FQL9XHcey4Go5yWJRAU6iVeCAQ8ocJ7L8 hcvLiAz3g42gbGVhK93h2OZb7TRbyxqvYmKUfmq5ZQ+EvcIeDqYKFZL5PDo0rAqngmfV eDACjxP61RvD5Lt8dfzDynFRuHjug7kxoh/h+QShbkUDQ7zzCryILaeq9xEUtelTlvAA VdqEIuK7NrBgcE/jSjlIas6l3QaL/qcnY4VjK+5TT95vWoLKCXlCD0ac4Y9MvrTR4HKO tzGkh8RdhaalGV3X6YwEsFh/jcByG01wCIpEwiwh6rOlE2rN6cPXiJMfY1H3t+BNw7vG lyMg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@nvidia.com header.s=n1 header.b=ds0u2m10; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id h22si1302776edj.344.2021.02.01.23.09.09; Mon, 01 Feb 2021 23:09:34 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@nvidia.com header.s=n1 header.b=ds0u2m10; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232490AbhBBHHb (ORCPT + 99 others); Tue, 2 Feb 2021 02:07:31 -0500 Received: from hqnvemgate26.nvidia.com ([216.228.121.65]:13859 "EHLO hqnvemgate26.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232456AbhBBHHS (ORCPT ); Tue, 2 Feb 2021 02:07:18 -0500 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate26.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Mon, 01 Feb 2021 23:06:37 -0800 Received: from mtl-vdi-166.wap.labs.mlnx (172.20.145.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 2 Feb 2021 07:06:34 +0000 Date: Tue, 2 Feb 2021 09:06:31 +0200 From: Eli Cohen To: Jason Wang CC: Si-Wei Liu , , , , , , Si-Wei Liu Subject: Re: [PATCH 1/2] vdpa/mlx5: Avoid unnecessary query virtqueue Message-ID: <20210202070631.GA233234@mtl-vdi-166.wap.labs.mlnx> References: <20210128134130.3051-1-elic@nvidia.com> <20210128134130.3051-2-elic@nvidia.com> <9d6058d6-5ce1-0442-8fd9-5a6fe6a0bc6b@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: User-Agent: Mutt/1.9.5 (bf161cf53efb) (2018-04-13) X-Originating-IP: [172.20.145.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To HQMAIL107.nvidia.com (172.20.187.13) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1612249597; bh=oQ6Ri294/3mOiXs/oCOWdeD09SEKwcXEGPnVFpELei4=; h=Date:From:To:CC:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:Content-Transfer-Encoding: In-Reply-To:User-Agent:X-Originating-IP:X-ClientProxiedBy; b=ds0u2m10ILFU0jMgmReLqHd6imLwiwGYwUQ39xzlAtIlkaS6iCO+AB23PfotnNN4S IFigvALYs9KoBXrqk8f2cD+nmJqCu1co/ZsfgCnE9qNI/rGa03VFnvLQ6XRGSjzuNr XqPMGxLi/ZaM1TykXi+ExlmyrXTaQwRfkQyPiHKuN22dnE2lbodrFyMRmchAuw0yrg pXHhBYdsPLDJQ+wFrCAfOeUh9PY8mFIFq5/Ox8+ddCbtLzKcqLD/hI50fsH277dlDr b/SffbKeo8EVstYEW4f3P61kfk5aIYcjvo2MlrqZe64VpJfoY4vVSjDZuctdvu9AoN xSBQKeZtjzQbQ== Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Feb 02, 2021 at 02:02:25PM +0800, Jason Wang wrote: >=20 > On 2021/2/2 =E4=B8=8B=E5=8D=8812:15, Si-Wei Liu wrote: > > On Mon, Feb 1, 2021 at 7:13 PM Jason Wang wrote: > > >=20 > > > On 2021/2/2 =E4=B8=8A=E5=8D=883:17, Si-Wei Liu wrote: > > > > On Mon, Feb 1, 2021 at 10:51 AM Si-Wei Liu wrote: > > > > > On Thu, Jan 28, 2021 at 5:46 AM Eli Cohen wrote= : > > > > > > suspend_vq should only suspend the VQ on not save the current a= vailable > > > > > > index. This is done when a change of map occurs when the driver= calls > > > > > > save_channel_info(). > > > > > Hmmm, suspend_vq() is also called by teardown_vq(), the latter of > > > > > which doesn't save the available index as save_channel_info() doe= sn't > > > > > get called in that path at all. How does it handle the case that > > > > > aget_vq_state() is called from userspace (e.g. QEMU) while the > > > > > hardware VQ object was torn down, but userspace still wants to ac= cess > > > > > the queue index? > > > > >=20 > > > > > Refer to https://lore.kernel.org/netdev/1601583511-15138-1-git-se= nd-email-si-wei.liu@oracle.com/ > > > > >=20 > > > > > vhost VQ 0 ring restore failed: -1: Resource temporarily unavaila= ble (11) > > > > > vhost VQ 1 ring restore failed: -1: Resource temporarily unavaila= ble (11) > > > > >=20 > > > > > QEMU will complain with the above warning while VM is being reboo= ted > > > > > or shut down. > > > > >=20 > > > > > Looks to me either the kernel driver should cover this requiremen= t, or > > > > > the userspace has to bear the burden in saving the index and not = call > > > > > into kernel if VQ is destroyed. > > > > Actually, the userspace doesn't have the insights whether virt queu= e > > > > will be destroyed if just changing the device status via set_status= (). > > > > Looking at other vdpa driver in tree i.e. ifcvf it doesn't behave l= ike > > > > so. Hence this still looks to me to be Mellanox specifics and > > > > mlx5_vdpa implementation detail that shouldn't expose to userspace. > > >=20 > > > So I think we can simply drop this patch? > > Yep, I think so. To be honest I don't know why it has anything to do > > with the memory hotplug issue. >=20 >=20 > Eli may know more, my understanding is that, during memory hotplut, qemu > need to propagate new memory mappings via set_map(). For mellanox, it mea= ns > it needs to rebuild memory keys, so the virtqueue needs to be suspended. >=20 I think Siwei was asking why the first patch was related to the hotplug issue. But you're correct. When memory is added, I get a new memory map. This requires me to build a new memory key object which covers the new memory map. Since the virtqueue objects are referencing this memory key, I need to destroy them and build new ones referncing the new memory key. > Thanks >=20 >=20 > >=20 > > -Siwei > >=20 > > > Thanks > > >=20 > > >=20 > > > > > -Siwei > > > > >=20 > > > > >=20 > > > > > > Signed-off-by: Eli Cohen > > > > > > --- > > > > > > drivers/vdpa/mlx5/net/mlx5_vnet.c | 8 -------- > > > > > > 1 file changed, 8 deletions(-) > > > > > >=20 > > > > > > diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/m= lx5/net/mlx5_vnet.c > > > > > > index 88dde3455bfd..549ded074ff3 100644 > > > > > > --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c > > > > > > +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c > > > > > > @@ -1148,8 +1148,6 @@ static int setup_vq(struct mlx5_vdpa_net = *ndev, struct mlx5_vdpa_virtqueue *mvq) > > > > > >=20 > > > > > > static void suspend_vq(struct mlx5_vdpa_net *ndev, struct ml= x5_vdpa_virtqueue *mvq) > > > > > > { > > > > > > - struct mlx5_virtq_attr attr; > > > > > > - > > > > > > if (!mvq->initialized) > > > > > > return; > > > > > >=20 > > > > > > @@ -1158,12 +1156,6 @@ static void suspend_vq(struct mlx5_vdpa_= net *ndev, struct mlx5_vdpa_virtqueue *m > > > > > >=20 > > > > > > if (modify_virtqueue(ndev, mvq, MLX5_VIRTIO_NET_Q_OBJ= ECT_STATE_SUSPEND)) > > > > > > mlx5_vdpa_warn(&ndev->mvdev, "modify to suspe= nd failed\n"); > > > > > > - > > > > > > - if (query_virtqueue(ndev, mvq, &attr)) { > > > > > > - mlx5_vdpa_warn(&ndev->mvdev, "failed to query v= irtqueue\n"); > > > > > > - return; > > > > > > - } > > > > > > - mvq->avail_idx =3D attr.available_index; > > > > > > } > > > > > >=20 > > > > > > static void suspend_vqs(struct mlx5_vdpa_net *ndev) > > > > > > -- > > > > > > 2.29.2 > > > > > >=20 >=20