Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.112.34 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.112.34; helo=mail.nvidia.com;
Date:   Tue, 1 Jun 2021 13:21:29 +0300
From:   Eli Cohen <elic@nvidia.com>
To:     Jason Wang <jasowang@redhat.com>
CC:     <mst@redhat.com>, <virtualization@lists.linux-foundation.org>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 2/3] vdpa/mlx5: Only return vq ready if vq is initialized
Message-ID: <20210601102129.GB215954@mtl-vdi-166.wap.labs.mlnx>
References: <20210531160448.31537-1-elic@nvidia.com>
 <0fbd8484-9e8b-d7cc-4996-74306e6e2867@redhat.com>
 <20210601041351.GC203469@mtl-vdi-166.wap.labs.mlnx>
 <67b26a20-b418-4893-a42f-f03852c2123e@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <67b26a20-b418-4893-a42f-f03852c2123e@redhat.com>
User-Agent: Mutt/1.9.5 (bf161cf53efb) (2018-04-13)
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Jun 2021 10:21:38.2305
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: edae354b-55d5-406c-36c3-08d924e70aa2
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.34];Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT014.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB3993
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Jun 01, 2021 at 01:03:36PM +0800, Jason Wang wrote:
> 
> 在 2021/6/1 下午12:13, Eli Cohen 写道:
> > On Tue, Jun 01, 2021 at 10:18:04AM +0800, Jason Wang wrote:
> > > 在 2021/6/1 上午12:04, Eli Cohen 写道:
> > > > Only return the value of the ready field if the VQ is initialized in
> > > > which case the value of the field is valid.
> > > > 
> > > > Failing to do so can result in virtio_vdpa failing to load if the device
> > > > was previously used by vhost_vdpa and the old values are ready.
> > > > virtio_vdpa expects to find VQs in "not ready" state.
> > > > 
> > > > Fixes: 1a86b377aa21 ("vdpa/mlx5: Add VDPA driver for supported mlx5 devices")
> > > > Signed-off-by: Eli Cohen <elic@nvidia.com>
> > > > ---
> > > >    drivers/vdpa/mlx5/net/mlx5_vnet.c | 2 +-
> > > >    1 file changed, 1 insertion(+), 1 deletion(-)
> > > > 
> > > > diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > > index 02a05492204c..f6b680d2ab1c 100644
> > > > --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > > +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > > @@ -1407,7 +1407,7 @@ static bool mlx5_vdpa_get_vq_ready(struct vdpa_device *vdev, u16 idx)
> > > >    	struct mlx5_vdpa_net *ndev = to_mlx5_vdpa_ndev(mvdev);
> > > >    	struct mlx5_vdpa_virtqueue *mvq = &ndev->vqs[idx];
> > > > -	return mvq->ready;
> > > > +	return mvq->initialized && mvq->ready;
> > > 
> > > I think the more suitable fix is to reset mvq->ready during reset. The
> > > vq_ready should follow the queue_enable semantic in virtio-pci:
> > > 
> > > "
> > > The device MUST present a 0 in queue_enable on reset.
> > > "
> > Thinking again, I think we should call set_vq_ready() from
> > qemu/virtio_vdpa etc. after reset to explicitly set ready to false.
> 
> 
> This is not what I read from the spec and how the current driver behave.
> 
> And I don't see why we need to stick to 1 after the reset.

Like you said below, this must be clarified by the spec. Because my
understanding of the text is that the ready flag is a means for the
driver to tell the device whether it may or may not execute requests. So
following this reasoning, the driver should tell the device when ready
should be zero. As you suggest, after reset.

Meanwhile I am going to change the code to reset ready after device
reset.

> 
> 
> > 
> > The ready indication is not necessairily a reflection of the hardware
> > queue:
> > 
> > "Virtual queue ready bit
> > Writing one (0x1) to this register notifies the device that it can
> > execute requests from this virtual queue. Reading from this register
> > returns the last value written to it. Both read and write accesses apply
> > to the queue selected by writing to QueueSel."
> 
> 
> My understanding that this applies if not reset in the middle. We can
> clarify this in the spec if needed.
> 
> Thanks
> 
> 
> > 
> > 
> > > Thanks
> > > 
> > > 
> > > >    }
> > > >    static int mlx5_vdpa_set_vq_state(struct vdpa_device *vdev, u16 idx,
>