LinuxLists.cc - [PATCH v2 0/4] Implement vdpasim stop operation

2022-05-24 23:43:53

Subject: [PATCH v2 0/4] Implement vdpasim stop operation

2022-05-25 03:04:26

Subject: [PATCH v2 4/4] vdpa_sim: Implement stop vdpa op

Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer
that backend feature and userspace can effectively stop the device.

This is a must before get virtqueue indexes (base) for live migration,
since the device could modify them after userland gets them. There are
individual ways to perform that action for some devices
(VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no
way to perform it for any vhost device (and, in particular, vhost-vdpa).

After the return of ioctl with stop != 0, the device MUST finish any
pending operations like in flight requests. It must also preserve all
the necessary state (the virtqueue vring base plus the possible device
specific states) that is required for restoring in the future. The
device must not change its configuration after that point.

After the return of ioctl with stop == 0, the device can continue
processing buffers as long as typical conditions are met (vq is enabled,
DRIVER_OK status bit is enabled, etc).

In the future, we will provide features similar to
VHOST_USER_GET_INFLIGHT_FD so the device can save pending operations.

Signed-off-by: Eugenio Pérez <[email protected]>
---
drivers/vdpa/vdpa_sim/vdpa_sim.c | 21 +++++++++++++++++++++
drivers/vdpa/vdpa_sim/vdpa_sim.h | 1 +
drivers/vdpa/vdpa_sim/vdpa_sim_blk.c | 3 +++
drivers/vdpa/vdpa_sim/vdpa_sim_net.c | 3 +++
4 files changed, 28 insertions(+)

diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c b/drivers/vdpa/vdpa_sim/vdpa_sim.c
index 50d721072beb..0515cf314bed 100644
--- a/drivers/vdpa/vdpa_sim/vdpa_sim.c
+++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c
@@ -107,6 +107,7 @@ static void vdpasim_do_reset(struct vdpasim *vdpasim)
for (i = 0; i < vdpasim->dev_attr.nas; i++)
vhost_iotlb_reset(&vdpasim->iommu[i]);

+ vdpasim->running = true;
spin_unlock(&vdpasim->iommu_lock);

vdpasim->features = 0;
@@ -505,6 +506,24 @@ static int vdpasim_reset(struct vdpa_device *vdpa)
return 0;
}

+static int vdpasim_stop(struct vdpa_device *vdpa, bool stop)
+{
+ struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
+ int i;
+
+ spin_lock(&vdpasim->lock);
+ vdpasim->running = !stop;
+ if (vdpasim->running) {
+ /* Check for missed buffers */
+ for (i = 0; i < vdpasim->dev_attr.nvqs; ++i)
+ vdpasim_kick_vq(vdpa, i);
+
+ }
+ spin_unlock(&vdpasim->lock);
+
+ return 0;
+}
+
static size_t vdpasim_get_config_size(struct vdpa_device *vdpa)
{
struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
@@ -694,6 +713,7 @@ static const struct vdpa_config_ops vdpasim_config_ops = {
.get_status = vdpasim_get_status,
.set_status = vdpasim_set_status,
.reset = vdpasim_reset,
+ .stop = vdpasim_stop,
.get_config_size = vdpasim_get_config_size,
.get_config = vdpasim_get_config,
.set_config = vdpasim_set_config,
@@ -726,6 +746,7 @@ static const struct vdpa_config_ops vdpasim_batch_config_ops = {
.get_status = vdpasim_get_status,
.set_status = vdpasim_set_status,
.reset = vdpasim_reset,
+ .stop = vdpasim_stop,
.get_config_size = vdpasim_get_config_size,
.get_config = vdpasim_get_config,
.set_config = vdpasim_set_config,
diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.h b/drivers/vdpa/vdpa_sim/vdpa_sim.h
index 622782e92239..061986f30911 100644
--- a/drivers/vdpa/vdpa_sim/vdpa_sim.h
+++ b/drivers/vdpa/vdpa_sim/vdpa_sim.h
@@ -66,6 +66,7 @@ struct vdpasim {
u32 generation;
u64 features;
u32 groups;
+ bool running;
/* spinlock to synchronize iommu table */
spinlock_t iommu_lock;
};
diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c b/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c
index 42d401d43911..bcdb1982c378 100644
--- a/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c
+++ b/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c
@@ -204,6 +204,9 @@ static void vdpasim_blk_work(struct work_struct *work)
if (!(vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK))
goto out;

+ if (!vdpasim->running)
+ goto out;
+
for (i = 0; i < VDPASIM_BLK_VQ_NUM; i++) {
struct vdpasim_virtqueue *vq = &vdpasim->vqs[i];

diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim_net.c b/drivers/vdpa/vdpa_sim/vdpa_sim_net.c
index 5125976a4df8..886449e88502 100644
--- a/drivers/vdpa/vdpa_sim/vdpa_sim_net.c
+++ b/drivers/vdpa/vdpa_sim/vdpa_sim_net.c
@@ -154,6 +154,9 @@ static void vdpasim_net_work(struct work_struct *work)

spin_lock(&vdpasim->lock);

+ if (!vdpasim->running)
+ goto out;
+
if (!(vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK))
goto out;

--
2.27.0

2022-05-25 03:09:45

by Jason Wang

[permalink] [raw]

Subject: Re: [PATCH v2 0/4] Implement vdpasim stop operation

2022-05-25 08:30:10

by Jason Wang

[permalink] [raw]

Subject: Re: [PATCH v2 4/4] vdpa_sim: Implement stop vdpa op

On Wed, May 25, 2022 at 1:06 AM Eugenio Pérez <[email protected]> wrote:
>
> Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer
> that backend feature and userspace can effectively stop the device.
>
> This is a must before get virtqueue indexes (base) for live migration,
> since the device could modify them after userland gets them. There are
> individual ways to perform that action for some devices
> (VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no
> way to perform it for any vhost device (and, in particular, vhost-vdpa).
>
> After the return of ioctl with stop != 0, the device MUST finish any
> pending operations like in flight requests. It must also preserve all
> the necessary state (the virtqueue vring base plus the possible device
> specific states) that is required for restoring in the future. The
> device must not change its configuration after that point.
>
> After the return of ioctl with stop == 0, the device can continue
> processing buffers as long as typical conditions are met (vq is enabled,
> DRIVER_OK status bit is enabled, etc).
>
> In the future, we will provide features similar to
> VHOST_USER_GET_INFLIGHT_FD so the device can save pending operations.
>
> Signed-off-by: Eugenio Pérez <[email protected]>
> ---
> drivers/vdpa/vdpa_sim/vdpa_sim.c | 21 +++++++++++++++++++++
> drivers/vdpa/vdpa_sim/vdpa_sim.h | 1 +
> drivers/vdpa/vdpa_sim/vdpa_sim_blk.c | 3 +++
> drivers/vdpa/vdpa_sim/vdpa_sim_net.c | 3 +++
> 4 files changed, 28 insertions(+)
>
> diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c b/drivers/vdpa/vdpa_sim/vdpa_sim.c
> index 50d721072beb..0515cf314bed 100644
> --- a/drivers/vdpa/vdpa_sim/vdpa_sim.c
> +++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c
> @@ -107,6 +107,7 @@ static void vdpasim_do_reset(struct vdpasim *vdpasim)
> for (i = 0; i < vdpasim->dev_attr.nas; i++)
> vhost_iotlb_reset(&vdpasim->iommu[i]);
>
> + vdpasim->running = true;
> spin_unlock(&vdpasim->iommu_lock);
>
> vdpasim->features = 0;
> @@ -505,6 +506,24 @@ static int vdpasim_reset(struct vdpa_device *vdpa)
> return 0;
> }
>
> +static int vdpasim_stop(struct vdpa_device *vdpa, bool stop)
> +{
> + struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
> + int i;
> +
> + spin_lock(&vdpasim->lock);
> + vdpasim->running = !stop;
> + if (vdpasim->running) {
> + /* Check for missed buffers */
> + for (i = 0; i < vdpasim->dev_attr.nvqs; ++i)
> + vdpasim_kick_vq(vdpa, i);
> +
> + }
> + spin_unlock(&vdpasim->lock);
> +
> + return 0;
> +}
> +
> static size_t vdpasim_get_config_size(struct vdpa_device *vdpa)
> {
> struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
> @@ -694,6 +713,7 @@ static const struct vdpa_config_ops vdpasim_config_ops = {
> .get_status = vdpasim_get_status,
> .set_status = vdpasim_set_status,
> .reset = vdpasim_reset,
> + .stop = vdpasim_stop,
> .get_config_size = vdpasim_get_config_size,
> .get_config = vdpasim_get_config,
> .set_config = vdpasim_set_config,
> @@ -726,6 +746,7 @@ static const struct vdpa_config_ops vdpasim_batch_config_ops = {
> .get_status = vdpasim_get_status,
> .set_status = vdpasim_set_status,
> .reset = vdpasim_reset,
> + .stop = vdpasim_stop,
> .get_config_size = vdpasim_get_config_size,
> .get_config = vdpasim_get_config,
> .set_config = vdpasim_set_config,
> diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.h b/drivers/vdpa/vdpa_sim/vdpa_sim.h
> index 622782e92239..061986f30911 100644
> --- a/drivers/vdpa/vdpa_sim/vdpa_sim.h
> +++ b/drivers/vdpa/vdpa_sim/vdpa_sim.h
> @@ -66,6 +66,7 @@ struct vdpasim {
> u32 generation;
> u64 features;
> u32 groups;
> + bool running;
> /* spinlock to synchronize iommu table */
> spinlock_t iommu_lock;
> };
> diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c b/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c
> index 42d401d43911..bcdb1982c378 100644
> --- a/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c
> +++ b/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c
> @@ -204,6 +204,9 @@ static void vdpasim_blk_work(struct work_struct *work)
> if (!(vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK))
> goto out;
>
> + if (!vdpasim->running)
> + goto out;
> +
> for (i = 0; i < VDPASIM_BLK_VQ_NUM; i++) {
> struct vdpasim_virtqueue *vq = &vdpasim->vqs[i];
>
> diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim_net.c b/drivers/vdpa/vdpa_sim/vdpa_sim_net.c
> index 5125976a4df8..886449e88502 100644
> --- a/drivers/vdpa/vdpa_sim/vdpa_sim_net.c
> +++ b/drivers/vdpa/vdpa_sim/vdpa_sim_net.c
> @@ -154,6 +154,9 @@ static void vdpasim_net_work(struct work_struct *work)
>
> spin_lock(&vdpasim->lock);
>
> + if (!vdpasim->running)
> + goto out;
> +

Do we need to check vdpasim->running in vdpasim_kick_vq()?

Thanks

> if (!(vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK))
> goto out;
>
> --
> 2.27.0
>

2022-05-25 09:08:36

by Eugenio Perez Martin

[permalink] [raw]

Subject: Re: [PATCH v2 4/4] vdpa_sim: Implement stop vdpa op

On Wed, May 25, 2022 at 4:54 AM Jason Wang <[email protected]> wrote:
>
> On Wed, May 25, 2022 at 1:06 AM Eugenio Pérez <[email protected]> wrote:
> >
> > Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer
> > that backend feature and userspace can effectively stop the device.
> >
> > This is a must before get virtqueue indexes (base) for live migration,
> > since the device could modify them after userland gets them. There are
> > individual ways to perform that action for some devices
> > (VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no
> > way to perform it for any vhost device (and, in particular, vhost-vdpa).
> >
> > After the return of ioctl with stop != 0, the device MUST finish any
> > pending operations like in flight requests. It must also preserve all
> > the necessary state (the virtqueue vring base plus the possible device
> > specific states) that is required for restoring in the future. The
> > device must not change its configuration after that point.
> >
> > After the return of ioctl with stop == 0, the device can continue
> > processing buffers as long as typical conditions are met (vq is enabled,
> > DRIVER_OK status bit is enabled, etc).
> >
> > In the future, we will provide features similar to
> > VHOST_USER_GET_INFLIGHT_FD so the device can save pending operations.
> >
> > Signed-off-by: Eugenio Pérez <[email protected]>
> > ---
> > drivers/vdpa/vdpa_sim/vdpa_sim.c | 21 +++++++++++++++++++++
> > drivers/vdpa/vdpa_sim/vdpa_sim.h | 1 +
> > drivers/vdpa/vdpa_sim/vdpa_sim_blk.c | 3 +++
> > drivers/vdpa/vdpa_sim/vdpa_sim_net.c | 3 +++
> > 4 files changed, 28 insertions(+)
> >
> > diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c b/drivers/vdpa/vdpa_sim/vdpa_sim.c
> > index 50d721072beb..0515cf314bed 100644
> > --- a/drivers/vdpa/vdpa_sim/vdpa_sim.c
> > +++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c
> > @@ -107,6 +107,7 @@ static void vdpasim_do_reset(struct vdpasim *vdpasim)
> > for (i = 0; i < vdpasim->dev_attr.nas; i++)
> > vhost_iotlb_reset(&vdpasim->iommu[i]);
> >
> > + vdpasim->running = true;
> > spin_unlock(&vdpasim->iommu_lock);
> >
> > vdpasim->features = 0;
> > @@ -505,6 +506,24 @@ static int vdpasim_reset(struct vdpa_device *vdpa)
> > return 0;
> > }
> >
> > +static int vdpasim_stop(struct vdpa_device *vdpa, bool stop)
> > +{
> > + struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
> > + int i;
> > +
> > + spin_lock(&vdpasim->lock);
> > + vdpasim->running = !stop;
> > + if (vdpasim->running) {
> > + /* Check for missed buffers */
> > + for (i = 0; i < vdpasim->dev_attr.nvqs; ++i)
> > + vdpasim_kick_vq(vdpa, i);
> > +
> > + }
> > + spin_unlock(&vdpasim->lock);
> > +
> > + return 0;
> > +}
> > +
> > static size_t vdpasim_get_config_size(struct vdpa_device *vdpa)
> > {
> > struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
> > @@ -694,6 +713,7 @@ static const struct vdpa_config_ops vdpasim_config_ops = {
> > .get_status = vdpasim_get_status,
> > .set_status = vdpasim_set_status,
> > .reset = vdpasim_reset,
> > + .stop = vdpasim_stop,
> > .get_config_size = vdpasim_get_config_size,
> > .get_config = vdpasim_get_config,
> > .set_config = vdpasim_set_config,
> > @@ -726,6 +746,7 @@ static const struct vdpa_config_ops vdpasim_batch_config_ops = {
> > .get_status = vdpasim_get_status,
> > .set_status = vdpasim_set_status,
> > .reset = vdpasim_reset,
> > + .stop = vdpasim_stop,
> > .get_config_size = vdpasim_get_config_size,
> > .get_config = vdpasim_get_config,
> > .set_config = vdpasim_set_config,
> > diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.h b/drivers/vdpa/vdpa_sim/vdpa_sim.h
> > index 622782e92239..061986f30911 100644
> > --- a/drivers/vdpa/vdpa_sim/vdpa_sim.h
> > +++ b/drivers/vdpa/vdpa_sim/vdpa_sim.h
> > @@ -66,6 +66,7 @@ struct vdpasim {
> > u32 generation;
> > u64 features;
> > u32 groups;
> > + bool running;
> > /* spinlock to synchronize iommu table */
> > spinlock_t iommu_lock;
> > };
> > diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c b/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c
> > index 42d401d43911..bcdb1982c378 100644
> > --- a/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c
> > +++ b/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c
> > @@ -204,6 +204,9 @@ static void vdpasim_blk_work(struct work_struct *work)
> > if (!(vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK))
> > goto out;
> >
> > + if (!vdpasim->running)
> > + goto out;
> > +
> > for (i = 0; i < VDPASIM_BLK_VQ_NUM; i++) {
> > struct vdpasim_virtqueue *vq = &vdpasim->vqs[i];
> >
> > diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim_net.c b/drivers/vdpa/vdpa_sim/vdpa_sim_net.c
> > index 5125976a4df8..886449e88502 100644
> > --- a/drivers/vdpa/vdpa_sim/vdpa_sim_net.c
> > +++ b/drivers/vdpa/vdpa_sim/vdpa_sim_net.c
> > @@ -154,6 +154,9 @@ static void vdpasim_net_work(struct work_struct *work)
> >
> > spin_lock(&vdpasim->lock);
> >
> > + if (!vdpasim->running)
> > + goto out;
> > +
>
> Do we need to check vdpasim->running in vdpasim_kick_vq()?
>

I'd say that not really: The important part is that we don't process
more buffers, and that is more accurate here. To check it here will
always avoid it although we queue work.

Maybe we can see it as an optimization: either to check before queuing
the work as you propose or simply stop polling kick file descriptors?

Thanks!

> Thanks
>
> > if (!(vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK))
> > goto out;
> >
> > --
> > 2.27.0
> >
>

2022-05-25 19:07:20

by Eugenio Perez Martin

[permalink] [raw]

Subject: [PATCH v2 1/4] vdpa: Add stop operation

This operation is optional: It it's not implemented, backend feature bit
will not be exposed.

Signed-off-by: Eugenio Pérez <[email protected]>
---
include/linux/vdpa.h | 6 ++++++
1 file changed, 6 insertions(+)

diff --git a/include/linux/vdpa.h b/include/linux/vdpa.h
index 15af802d41c4..ddfebc4e1e01 100644
--- a/include/linux/vdpa.h
+++ b/include/linux/vdpa.h
@@ -215,6 +215,11 @@ struct vdpa_map_file {
* @reset: Reset device
* @vdev: vdpa device
* Returns integer: success (0) or error (< 0)
+ * @stop: Stop or resume the device (optional, but it must
+ * be implemented if require device stop)
+ * @vdev: vdpa device
+ * @stop: stop (true), not stop (false)
+ * Returns integer: success (0) or error (< 0)
* @get_config_size: Get the size of the configuration space includes
* fields that are conditional on feature bits.
* @vdev: vdpa device
@@ -316,6 +321,7 @@ struct vdpa_config_ops {
u8 (*get_status)(struct vdpa_device *vdev);
void (*set_status)(struct vdpa_device *vdev, u8 status);
int (*reset)(struct vdpa_device *vdev);
+ int (*stop)(struct vdpa_device *vdev, bool stop);
size_t (*get_config_size)(struct vdpa_device *vdev);
void (*get_config)(struct vdpa_device *vdev, unsigned int offset,
void *buf, unsigned int len);
--
2.27.0

2022-05-25 21:34:31

by Stefano Garzarella

[permalink] [raw]

Subject: Re: [PATCH v2 4/4] vdpa_sim: Implement stop vdpa op

On Tue, May 24, 2022 at 07:06:10PM +0200, Eugenio P?rez wrote:
>Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer
>that backend feature and userspace can effectively stop the device.
>
>This is a must before get virtqueue indexes (base) for live migration,
>since the device could modify them after userland gets them. There are
>individual ways to perform that action for some devices
>(VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no
>way to perform it for any vhost device (and, in particular, vhost-vdpa).
>
>After the return of ioctl with stop != 0, the device MUST finish any
>pending operations like in flight requests. It must also preserve all
>the necessary state (the virtqueue vring base plus the possible device
>specific states) that is required for restoring in the future. The
>device must not change its configuration after that point.
>
>After the return of ioctl with stop == 0, the device can continue
>processing buffers as long as typical conditions are met (vq is enabled,
>DRIVER_OK status bit is enabled, etc).
>
>In the future, we will provide features similar to
>VHOST_USER_GET_INFLIGHT_FD so the device can save pending operations.
>
>Signed-off-by: Eugenio P?rez <[email protected]>
>---
> drivers/vdpa/vdpa_sim/vdpa_sim.c | 21 +++++++++++++++++++++
> drivers/vdpa/vdpa_sim/vdpa_sim.h | 1 +
> drivers/vdpa/vdpa_sim/vdpa_sim_blk.c | 3 +++
> drivers/vdpa/vdpa_sim/vdpa_sim_net.c | 3 +++
> 4 files changed, 28 insertions(+)
>
>diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c b/drivers/vdpa/vdpa_sim/vdpa_sim.c
>index 50d721072beb..0515cf314bed 100644
>--- a/drivers/vdpa/vdpa_sim/vdpa_sim.c
>+++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c
>@@ -107,6 +107,7 @@ static void vdpasim_do_reset(struct vdpasim *vdpasim)
> for (i = 0; i < vdpasim->dev_attr.nas; i++)
> vhost_iotlb_reset(&vdpasim->iommu[i]);
>
>+ vdpasim->running = true;
> spin_unlock(&vdpasim->iommu_lock);
>
> vdpasim->features = 0;
>@@ -505,6 +506,24 @@ static int vdpasim_reset(struct vdpa_device *vdpa)
> return 0;
> }
>
>+static int vdpasim_stop(struct vdpa_device *vdpa, bool stop)
>+{
>+ struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
>+ int i;
>+
>+ spin_lock(&vdpasim->lock);
>+ vdpasim->running = !stop;
>+ if (vdpasim->running) {
>+ /* Check for missed buffers */
>+ for (i = 0; i < vdpasim->dev_attr.nvqs; ++i)
>+ vdpasim_kick_vq(vdpa, i);
>+
>+ }
>+ spin_unlock(&vdpasim->lock);
>+
>+ return 0;
>+}
>+
> static size_t vdpasim_get_config_size(struct vdpa_device *vdpa)
> {
> struct vdpasim *vdpasim = vdpa_to_sim(vdpa);
>@@ -694,6 +713,7 @@ static const struct vdpa_config_ops vdpasim_config_ops = {
> .get_status = vdpasim_get_status,
> .set_status = vdpasim_set_status,
> .reset = vdpasim_reset,
>+ .stop = vdpasim_stop,
> .get_config_size = vdpasim_get_config_size,
> .get_config = vdpasim_get_config,
> .set_config = vdpasim_set_config,
>@@ -726,6 +746,7 @@ static const struct vdpa_config_ops vdpasim_batch_config_ops = {
> .get_status = vdpasim_get_status,
> .set_status = vdpasim_set_status,
> .reset = vdpasim_reset,
>+ .stop = vdpasim_stop,
> .get_config_size = vdpasim_get_config_size,
> .get_config = vdpasim_get_config,
> .set_config = vdpasim_set_config,
>diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.h b/drivers/vdpa/vdpa_sim/vdpa_sim.h
>index 622782e92239..061986f30911 100644
>--- a/drivers/vdpa/vdpa_sim/vdpa_sim.h
>+++ b/drivers/vdpa/vdpa_sim/vdpa_sim.h
>@@ -66,6 +66,7 @@ struct vdpasim {
> u32 generation;
> u64 features;
> u32 groups;
>+ bool running;
> /* spinlock to synchronize iommu table */
> spinlock_t iommu_lock;
> };
>diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c b/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c
>index 42d401d43911..bcdb1982c378 100644
>--- a/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c
>+++ b/drivers/vdpa/vdpa_sim/vdpa_sim_blk.c
>@@ -204,6 +204,9 @@ static void vdpasim_blk_work(struct work_struct *work)
> if (!(vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK))
> goto out;
>
>+ if (!vdpasim->running)
>+ goto out;
>+

Not related to this series, but I think in vdpa_sim_blk.c we should
implement something similar to what we already do in vdpa_sim_net.c and
re-schedule the work after X requests handled, otherwise we risk never
stopping if there are always requests to handle.

Also for supporting multiple queues, that could be a problem, but for
now we only support one, so there should be no problem.

I have other patches to send for vdpa_sim_blk.c, so if you want I can do
that in my series.

Thanks,
Stefano

> for (i = 0; i < VDPASIM_BLK_VQ_NUM; i++) {
> struct vdpasim_virtqueue *vq = &vdpasim->vqs[i];
>
>diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim_net.c b/drivers/vdpa/vdpa_sim/vdpa_sim_net.c
>index 5125976a4df8..886449e88502 100644
>--- a/drivers/vdpa/vdpa_sim/vdpa_sim_net.c
>+++ b/drivers/vdpa/vdpa_sim/vdpa_sim_net.c
>@@ -154,6 +154,9 @@ static void vdpasim_net_work(struct work_struct *work)
>
> spin_lock(&vdpasim->lock);
>
>+ if (!vdpasim->running)
>+ goto out;
>+
> if (!(vdpasim->status & VIRTIO_CONFIG_S_DRIVER_OK))
> goto out;
>
>--
>2.27.0
>

2022-05-26 04:28:58

by Eugenio Perez Martin

[permalink] [raw]

Subject: Re: [PATCH v2 0/4] Implement vdpasim stop operation

On Wed, May 25, 2022 at 4:49 AM Jason Wang <[email protected]> wrote:
>
> On Wed, May 25, 2022 at 1:06 AM Eugenio Pérez <[email protected]> wrote:
> >
> > Implement stop operation for vdpa_sim devices, so vhost-vdpa will offer
> > that backend feature and userspace can effectively stop the device.
> >
> > This is a must before get virtqueue indexes (base) for live migration,
> > since the device could modify them after userland gets them. There are
> > individual ways to perform that action for some devices
> > (VHOST_NET_SET_BACKEND, VHOST_VSOCK_SET_RUNNING, ...) but there was no
> > way to perform it for any vhost device (and, in particular, vhost-vdpa).
> >
> > After the return of ioctl with stop != 0, the device MUST finish any
> > pending operations like in flight requests. It must also preserve all
> > the necessary state (the virtqueue vring base plus the possible device
> > specific states) that is required for restoring in the future. The
> > device must not change its configuration after that point.
>
> I'd suggest documenting this in the code maybe around ops->stop()?
>

I agree it'd be better to put in the source code, but both
vdpa_config_ops and ops->stop don't have a lot of space for docs.

Would it work to document at drivers/vdpa/vdpa.c:vhost_vdpa_stop() and
redirect config ops like "for more info, see vhost_vdpa_stop"?

Thanks!

> Thanks
>
> >
> > After the return of ioctl with stop == 0, the device can continue
> > processing buffers as long as typical conditions are met (vq is enabled,
> > DRIVER_OK status bit is enabled, etc).
> >
> > In the future, we will provide features similar to VHOST_USER_GET_INFLIGHT_FD
> > so the device can save pending operations.
> >
> > Comments are welcome.
> >
> > v2:
> > * Replace raw _F_STOP with BIT_ULL(_F_STOP).
> > * Fix obtaining of stop ioctl arg (it was not obtained but written).
> > * Add stop to vdpa_sim_blk.
> >
> > Eugenio Pérez (4):
> > vdpa: Add stop operation
> > vhost-vdpa: introduce STOP backend feature bit
> > vhost-vdpa: uAPI to stop the device
> > vdpa_sim: Implement stop vdpa op
> >
> > drivers/vdpa/vdpa_sim/vdpa_sim.c | 21 +++++++++++++++++
> > drivers/vdpa/vdpa_sim/vdpa_sim.h | 1 +
> > drivers/vdpa/vdpa_sim/vdpa_sim_blk.c | 3 +++
> > drivers/vdpa/vdpa_sim/vdpa_sim_net.c | 3 +++
> > drivers/vhost/vdpa.c | 34 +++++++++++++++++++++++++++-
> > include/linux/vdpa.h | 6 +++++
> > include/uapi/linux/vhost.h | 3 +++
> > include/uapi/linux/vhost_types.h | 2 ++
> > 8 files changed, 72 insertions(+), 1 deletion(-)
> >
> > --
> > 2.27.0
> >
> >
>