Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20230703142514.363256-1-eperezma@redhat.com> <20230703110241-mutt-send-email-mst@kernel.org>
 <ba5099b8-f72c-f267-41a7-d0ee18680796@oracle.com> <20230704061356-mutt-send-email-mst@kernel.org>
 <CACGkMEtAfiODhHEMFDKkaVFVs1yjfUFsjfy_=p8Jtd6BXAE1xQ@mail.gmail.com>
 <20230705012917-mutt-send-email-mst@kernel.org> <CACGkMEtBjTmChSQX_E_n6yvTi9rxH5UrXgfAU5qZsFAF-RZQWw@mail.gmail.com>
 <20230705021447-mutt-send-email-mst@kernel.org>
In-Reply-To: <20230705021447-mutt-send-email-mst@kernel.org>
From:   Jason Wang <jasowang@redhat.com>
Date:   Wed, 5 Jul 2023 15:32:04 +0800
Message-ID: <CACGkMEswyCWbXnLnm-i5ydp27kmQDvxF3gEfHhU_t0HJ7g+4Wg@mail.gmail.com>
Subject: Re: [PATCH] mlx5_vdpa: offer VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK
To:     "Michael S. Tsirkin" <mst@redhat.com>
Cc:     Si-Wei Liu <si-wei.liu@oracle.com>,
        =?UTF-8?Q?Eugenio_P=C3=A9rez?= <eperezma@redhat.com>,
        linux-kernel@vger.kernel.org, Dragos Tatulea <dtatulea@nvidia.com>,
        virtualization@lists.linux-foundation.org, leiyang@redhat.com,
        Xuan Zhuo <xuanzhuo@linux.alibaba.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Wed, Jul 5, 2023 at 2:16=E2=80=AFPM Michael S. Tsirkin <mst@redhat.com> =
wrote:
>
> On Wed, Jul 05, 2023 at 01:47:44PM +0800, Jason Wang wrote:
> > On Wed, Jul 5, 2023 at 1:31=E2=80=AFPM Michael S. Tsirkin <mst@redhat.c=
om> wrote:
> > >
> > > On Wed, Jul 05, 2023 at 01:11:37PM +0800, Jason Wang wrote:
> > > > On Tue, Jul 4, 2023 at 6:16=E2=80=AFPM Michael S. Tsirkin <mst@redh=
at.com> wrote:
> > > > >
> > > > > On Mon, Jul 03, 2023 at 05:26:02PM -0700, Si-Wei Liu wrote:
> > > > > >
> > > > > >
> > > > > > On 7/3/2023 8:46 AM, Michael S. Tsirkin wrote:
> > > > > > > On Mon, Jul 03, 2023 at 04:25:14PM +0200, Eugenio P=C3=A9rez =
wrote:
> > > > > > > > Offer this backend feature as mlx5 is compatible with it. I=
t allows it
> > > > > > > > to do live migration with CVQ, dynamically switching betwee=
n passthrough
> > > > > > > > and shadow virtqueue.
> > > > > > > >
> > > > > > > > Signed-off-by: Eugenio P=C3=A9rez <eperezma@redhat.com>
> > > > > > > Same comment.
> > > > > > to which?
> > > > > >
> > > > > > -Siwei
> > > > >
> > > > > VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK is too narrow a use-case t=
o commit to it
> > > > > as a kernel/userspace ABI: what if one wants to start rings in so=
me
> > > > > other specific order?
> > > >
> > > > Just enable a queue by writing e.g 1 to queue_enable in a specific =
order?
> > >
> > >
> > > But then at driver ok time we don't know how many queues are there.
> >
> > There should be a device specific interface for this, for example,
> > num_queue_paris. So the device knows at most how many queues there
> > are. Or anything I miss?
>
> That's a device limitations. Does not tell the device how much is used.

I think I miss something, how kick differs from queue_enable in this way?

>
> > >
> > > > > As was discussed on list, a better promise is not to access ring
> > > > > until the 1st kick. vdpa can then do a kick when it wants
> > > > > the device to start accessing rings.
> > > >
> > > > Rethink about the ACCESS_AFTER_KICK, it sounds functional equivalen=
t
> > > > to allow queue_enable after DRIVER_OK, but it seems to have
> > > > distanvages:
> > > >
> > > > A busy polling software device may disable notifications and never
> > > > respond to register to any kick notifiers. ACCESS_AFTER_KICK will
> > > > introduce overhead to those implementations.
> > > >
> > > > Thanks
> > >
> > > It's just the 1st kick, then you can disable. No?
> >
> > Yes, but:
> >
> > 1) adding hooks for queue_enable
> > 2) adding new codes to register event notifier and toggle the notifier
> >
> > 1) seems much easier? Or for most devices, it already behaves like this=
.
> >
> > Thanks
>
> Well libvhostuser checks enabled queues at DRIVER_OK, does it not?

Probably, but I meant:

1) This behaviour has been supported by some device (e.g MLX)
2) This is the current behaviour of Qemu for vhost-net devices:

static void virtio_net_queue_enable(VirtIODevice *vdev, uint32_t queue_inde=
x)
{
    VirtIONet *n =3D VIRTIO_NET(vdev);
    NetClientState *nc;
    int r;

    ....

    if (get_vhost_net(nc->peer) &&
        nc->peer->info->type =3D=3D NET_CLIENT_DRIVER_TAP) {
        r =3D vhost_net_virtqueue_restart(vdev, nc, queue_index);
        if (r < 0) {
            error_report("unable to restart vhost net virtqueue: %d, "
                            "when resetting the queue", queue_index);
        }
    }
}

Thanks

>
> > >
> > > > >
> > > > > > >
> > > > > > > > ---
> > > > > > > >   drivers/vdpa/mlx5/net/mlx5_vnet.c | 7 +++++++
> > > > > > > >   1 file changed, 7 insertions(+)
> > > > > > > >
> > > > > > > > diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vd=
pa/mlx5/net/mlx5_vnet.c
> > > > > > > > index 9138ef2fb2c8..5f309a16b9dc 100644
> > > > > > > > --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > > > > > > +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
> > > > > > > > @@ -7,6 +7,7 @@
> > > > > > > >   #include <uapi/linux/virtio_net.h>
> > > > > > > >   #include <uapi/linux/virtio_ids.h>
> > > > > > > >   #include <uapi/linux/vdpa.h>
> > > > > > > > +#include <uapi/linux/vhost_types.h>
> > > > > > > >   #include <linux/virtio_config.h>
> > > > > > > >   #include <linux/auxiliary_bus.h>
> > > > > > > >   #include <linux/mlx5/cq.h>
> > > > > > > > @@ -2499,6 +2500,11 @@ static void unregister_link_notifier=
(struct mlx5_vdpa_net *ndev)
> > > > > > > >                   flush_workqueue(ndev->mvdev.wq);
> > > > > > > >   }
> > > > > > > > +static u64 mlx5_vdpa_get_backend_features(const struct vdp=
a_device *vdpa)
> > > > > > > > +{
> > > > > > > > + return BIT_ULL(VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK);
> > > > > > > > +}
> > > > > > > > +
> > > > > > > >   static int mlx5_vdpa_set_driver_features(struct vdpa_devi=
ce *vdev, u64 features)
> > > > > > > >   {
> > > > > > > >           struct mlx5_vdpa_dev *mvdev =3D to_mvdev(vdev);
> > > > > > > > @@ -3140,6 +3146,7 @@ static const struct vdpa_config_ops m=
lx5_vdpa_ops =3D {
> > > > > > > >           .get_vq_align =3D mlx5_vdpa_get_vq_align,
> > > > > > > >           .get_vq_group =3D mlx5_vdpa_get_vq_group,
> > > > > > > >           .get_device_features =3D mlx5_vdpa_get_device_fea=
tures,
> > > > > > > > + .get_backend_features =3D mlx5_vdpa_get_backend_features,
> > > > > > > >           .set_driver_features =3D mlx5_vdpa_set_driver_fea=
tures,
> > > > > > > >           .get_driver_features =3D mlx5_vdpa_get_driver_fea=
tures,
> > > > > > > >           .set_config_cb =3D mlx5_vdpa_set_config_cb,
> > > > > > > > --
> > > > > > > > 2.39.3
> > > > >
> > >
>