Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20220623160738.632852-1-eperezma@redhat.com> <20220623160738.632852-4-eperezma@redhat.com>
 <CAGxU2F43+5zsQOR4ReTtQtEF47s6y-XKcevosMOzUdEqpLhAsg@mail.gmail.com>
In-Reply-To: <CAGxU2F43+5zsQOR4ReTtQtEF47s6y-XKcevosMOzUdEqpLhAsg@mail.gmail.com>
From:   Eugenio Perez Martin <eperezma@redhat.com>
Date:   Fri, 8 Jul 2022 13:47:43 +0200
Message-ID: <CAJaqyWdn=mShLEZzfJB_+PwM+pkLhLMJaMOjdFtuW8tYbbU3FQ@mail.gmail.com>
Subject: Re: [PATCH v6 3/4] vhost-vdpa: uAPI to suspend the device
To:     Stefano Garzarella <sgarzare@redhat.com>
Cc:     netdev <netdev@vger.kernel.org>, Jason Wang <jasowang@redhat.com>,
        kernel list <linux-kernel@vger.kernel.org>,
        Linux Virtualization <virtualization@lists.linux-foundation.org>,
        kvm <kvm@vger.kernel.org>, "Michael S. Tsirkin" <mst@redhat.com>,
        Parav Pandit <parav@nvidia.com>,
        Christophe JAILLET <christophe.jaillet@wanadoo.fr>,
        Cindy Lu <lulu@redhat.com>,
        "Kamde, Tanuj" <tanuj.kamde@amd.com>,
        Si-Wei Liu <si-wei.liu@oracle.com>,
        "Uminski, Piotr" <Piotr.Uminski@intel.com>,
        habetsm.xilinx@gmail.com, "Dawar, Gautam" <gautam.dawar@amd.com>,
        Pablo Cascon Katchadourian <pabloc@xilinx.com>,
        Zhu Lingshan <lingshan.zhu@intel.com>,
        Laurent Vivier <lvivier@redhat.com>,
        Longpeng <longpeng2@huawei.com>,
        Dinan Gunawardena <dinang@xilinx.com>,
        Martin Petrus Hubertus Habets <martinh@xilinx.com>,
        Martin Porter <martinpo@xilinx.com>,
        Eli Cohen <elic@nvidia.com>, ecree.xilinx@gmail.com,
        Wu Zongyong <wuzongyong@linux.alibaba.com>,
        Dan Carpenter <dan.carpenter@oracle.com>,
        Harpreet Singh Anand <hanand@xilinx.com>,
        Xie Yongji <xieyongji@bytedance.com>,
        Zhang Min <zhang.min9@zte.com.cn>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Tue, Jun 28, 2022 at 3:45 PM Stefano Garzarella <sgarzare@redhat.com> wr=
ote:
>
> On Thu, Jun 23, 2022 at 06:07:37PM +0200, Eugenio P=C3=A9rez wrote:
> >The ioctl adds support for suspending the device from userspace.
> >
> >This is a must before getting virtqueue indexes (base) for live migratio=
n,
> >since the device could modify them after userland gets them. There are
> >individual ways to perform that action for some devices
> >(VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no
> >way to perform it for any vhost device (and, in particular, vhost-vdpa).
> >
> >After a successful return of the ioctl call the device must not process
> >more virtqueue descriptors. The device can answer to read or writes of
> >config fields as if it were not suspended. In particular, writing to
> >"queue_enable" with a value of 1 will not make the device start
> >processing buffers of the virtqueue.
> >
> >Signed-off-by: Eugenio P=C3=A9rez <eperezma@redhat.com>
> >---
> > drivers/vhost/vdpa.c       | 19 +++++++++++++++++++
> > include/uapi/linux/vhost.h | 14 ++++++++++++++
> > 2 files changed, 33 insertions(+)
> >
> >diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
> >index 3d636e192061..7fa671ac4bdf 100644
> >--- a/drivers/vhost/vdpa.c
> >+++ b/drivers/vhost/vdpa.c
> >@@ -478,6 +478,22 @@ static long vhost_vdpa_get_vqs_count(struct vhost_v=
dpa *v, u32 __user *argp)
> >       return 0;
> > }
> >
> >+/* After a successful return of ioctl the device must not process more
> >+ * virtqueue descriptors. The device can answer to read or writes of co=
nfig
> >+ * fields as if it were not suspended. In particular, writing to "queue=
_enable"
> >+ * with a value of 1 will not make the device start processing buffers.
> >+ */
> >+static long vhost_vdpa_suspend(struct vhost_vdpa *v)
> >+{
> >+      struct vdpa_device *vdpa =3D v->vdpa;
> >+      const struct vdpa_config_ops *ops =3D vdpa->config;
> >+
> >+      if (!ops->suspend)
> >+              return -EOPNOTSUPP;
> >+
> >+      return ops->suspend(vdpa);
> >+}
> >+
> > static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int c=
md,
> >                                  void __user *argp)
> > {
> >@@ -654,6 +670,9 @@ static long vhost_vdpa_unlocked_ioctl(struct file *f=
ilep,
> >       case VHOST_VDPA_GET_VQS_COUNT:
> >               r =3D vhost_vdpa_get_vqs_count(v, argp);
> >               break;
> >+      case VHOST_VDPA_SUSPEND:
> >+              r =3D vhost_vdpa_suspend(v);
> >+              break;
> >       default:
> >               r =3D vhost_dev_ioctl(&v->vdev, cmd, argp);
> >               if (r =3D=3D -ENOIOCTLCMD)
> >diff --git a/include/uapi/linux/vhost.h b/include/uapi/linux/vhost.h
> >index cab645d4a645..6d9f45163155 100644
> >--- a/include/uapi/linux/vhost.h
> >+++ b/include/uapi/linux/vhost.h
> >@@ -171,4 +171,18 @@
> > #define VHOST_VDPA_SET_GROUP_ASID     _IOW(VHOST_VIRTIO, 0x7C, \
> >                                            struct vhost_vring_state)
> >
> >+/* Suspend or resume a device so it does not process virtqueue requests=
 anymore
> >+ *
> >+ * After the return of ioctl with suspend !=3D 0, the device must finis=
h any
> >+ * pending operations like in flight requests. It must also preserve al=
l the
> >+ * necessary state (the virtqueue vring base plus the possible device s=
pecific
> >+ * states) that is required for restoring in the future. The device mus=
t not
> >+ * change its configuration after that point.
> >+ *
> >+ * After the return of ioctl with suspend =3D=3D 0, the device can cont=
inue
> >+ * processing buffers as long as typical conditions are met (vq is enab=
led,
> >+ * DRIVER_OK status bit is enabled, etc).
> >+ */
> >+#define VHOST_VDPA_SUSPEND            _IOW(VHOST_VIRTIO, 0x7D, int)
>                                          ^
> IIUC we are not using the argument anymore, so this should be changed in
> _IO(VHOST_VIRTIO, 0x7D).
>
> And we should update a bit the documentation.
>

Totally right, replacing it for the next version.

Thanks!

> Thanks,
> Stefano
>