Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <cover.1631101392.git.wuzongyong@linux.alibaba.com>
 <cover.1631621507.git.wuzongyong@linux.alibaba.com> <834528d24c839080215b2e077f100e9ed5073edc.1631621507.git.wuzongyong@linux.alibaba.com>
 <20210914085711-mutt-send-email-mst@kernel.org> <CACGkMEu3RUGpe74Vh-FAZD3MwOC3gqU0OEf8A1ULvq7GSMm6Jg@mail.gmail.com>
 <20210915033756-mutt-send-email-mst@kernel.org> <CACGkMEtN0Z=rgMhaWNO=6h-KXGdosBuOdqoWoND-=Tf+afyUYw@mail.gmail.com>
 <20210915070805-mutt-send-email-mst@kernel.org> <CACGkMEvqZqwRxjnBLpX+=MYbho0q8Hs7OZbsx4ATzhfwOkB73w@mail.gmail.com>
 <20210917023451.GA19669@L-PF27918B-1352.localdomain>
In-Reply-To: <20210917023451.GA19669@L-PF27918B-1352.localdomain>
From:   Jason Wang <jasowang@redhat.com>
Date:   Fri, 17 Sep 2021 15:51:52 +0800
Message-ID: <CACGkMEvzqYSNo4JGpqC5O-fkjOEPXNU0huRyufAOU5HxAXeoKg@mail.gmail.com>
Subject: Re: [PATCH v2 4/5] vdpa: add new vdpa attribute VDPA_ATTR_DEV_F_VERSION_1
To:     Wu Zongyong <wuzongyong@linux.alibaba.com>
Cc:     "Michael S. Tsirkin" <mst@redhat.com>,
        virtualization <virtualization@lists.linux-foundation.org>,
        linux-kernel <linux-kernel@vger.kernel.org>,
        wei.yang1@linux.alibaba.com
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Fri, Sep 17, 2021 at 10:35 AM Wu Zongyong
<wuzongyong@linux.alibaba.com> wrote:
>
> On Thu, Sep 16, 2021 at 09:05:58AM +0800, Jason Wang wrote:
> > On Wed, Sep 15, 2021 at 7:09 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Wed, Sep 15, 2021 at 04:06:57PM +0800, Jason Wang wrote:
> > > > On Wed, Sep 15, 2021 at 3:38 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > >
> > > > > On Wed, Sep 15, 2021 at 11:18:06AM +0800, Jason Wang wrote:
> > > > > > On Tue, Sep 14, 2021 at 8:58 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > >
> > > > > > > On Tue, Sep 14, 2021 at 08:24:51PM +0800, Wu Zongyong wrote:
> > > > > > > > This new attribute advertises whether the vdpa device is legacy or not.
> > > > > > > > Users can pick right virtqueue size if the vdpa device is legacy which
> > > > > > > > doesn't support to change virtqueue size.
> > > > > > > >
> > > > > > > > Signed-off-by: Wu Zongyong <wuzongyong@linux.alibaba.com>
> > > > > > >
> > > > > > > So if we are bothering with legacy,
> > > > > >
> > > > > > I think we'd better not. I guess the following may work:
> > > > > >
> > > > > > 1) disable the driver on BE host
> > > > > > 2) present VERSION_1 with ACCESS_PLATFORM in get_features()
> > > > > > 3) extend the management to advertise max_queue_size and
> > > > > > min_queue_size, for ENI they are the same so management layer knows it
> > > > > > needs to set the queue_size correctly during launching qemu
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > Thanks
> > > > >
> > > > > There are other subtle differences such as header size without
> > > > > mergeable buffers for net.
> > > >
> > > > This can be solved by mandating the feature of a mergeable buffer?
> > > >
> > > > Thanks
> > >
> > > PXE and some dpdk versions are only some of the guests that
> > > disable mergeable buffers feature.
> >
> > True, but consider
> >
> > 1) the legacy stuffs requires changes in several software layers
> > 2) it is how virtio 1.0 works e.g device can fail the feature negotiation
> > 3) it is not supported since day 0
> > 4) management API can be extended to advertise the mandated features
>
> So let me confirm what I should do in next revision:
> 1) disable the driver on BE host like that:
>
>    #ifdef __LITTE_ENDIAN
>    int eni_vdpa_probe()
>    {
>        ...
>    }
>    #else
>    int eni_vdpa_probe()
>    {
>        return -ENODEV;
>    }
>    #endif

This might work but I wonder if we can disable it via Kconfig.

>
> 2) report F_VERSION_1 and F_ACCESS_PLATFORM in get_features()
> 3) introduce a new cb get_vq_num_min in vdpa_config_ops
>
> Does I miss something?

And we need this as well.

Fail the feature negotiation if mrg rxbuf is not negotiated. Otherwise
we can meet the 1.0 requirement of header length. Or the hardware can
still preset the mergeable header if the mrg rx buffer is not
negotiated?

Thanks

>
> > It looks affordable.
> >
> > Thanks
> >
> > >
> > > > >
> > > > >
> > > > > > > I think there are
> > > > > > > several things to do when building the interface
> > > > > > > - support transitional devices, that is allow userspace
> > > > > > >   to tell device it's in legacy mode
> > > > > > > - support reporting/setting supporting endian-ness
> > > > > > >
> > > > > > > > ---
> > > > > > > >  drivers/vdpa/vdpa.c          | 6 ++++++
> > > > > > > >  drivers/virtio/virtio_vdpa.c | 7 ++++++-
> > > > > > > >  include/uapi/linux/vdpa.h    | 1 +
> > > > > > > >  3 files changed, 13 insertions(+), 1 deletion(-)
> > > > > > > >
> > > > > > > > diff --git a/drivers/vdpa/vdpa.c b/drivers/vdpa/vdpa.c
> > > > > > > > index 1dc121a07a93..533d7f589eee 100644
> > > > > > > > --- a/drivers/vdpa/vdpa.c
> > > > > > > > +++ b/drivers/vdpa/vdpa.c
> > > > > > > > @@ -12,6 +12,7 @@
> > > > > > > >  #include <linux/slab.h>
> > > > > > > >  #include <linux/vdpa.h>
> > > > > > > >  #include <uapi/linux/vdpa.h>
> > > > > > > > +#include <uapi/linux/virtio_config.h>
> > > > > > > >  #include <net/genetlink.h>
> > > > > > > >  #include <linux/mod_devicetable.h>
> > > > > > > >
> > > > > > > > @@ -494,6 +495,7 @@ vdpa_dev_fill(struct vdpa_device *vdev, struct sk_buff *msg, u32 portid, u32 seq
> > > > > > > >       u16 max_vq_size;
> > > > > > > >       u32 device_id;
> > > > > > > >       u32 vendor_id;
> > > > > > > > +     u64 features;
> > > > > > > >       void *hdr;
> > > > > > > >       int err;
> > > > > > > >
> > > > > > > > @@ -508,6 +510,7 @@ vdpa_dev_fill(struct vdpa_device *vdev, struct sk_buff *msg, u32 portid, u32 seq
> > > > > > > >       device_id = vdev->config->get_device_id(vdev);
> > > > > > > >       vendor_id = vdev->config->get_vendor_id(vdev);
> > > > > > > >       max_vq_size = vdev->config->get_vq_num_max(vdev);
> > > > > > > > +     features = vdev->config->get_features(vdev);
> > > > > > > >
> > > > > > > >       err = -EMSGSIZE;
> > > > > > > >       if (nla_put_string(msg, VDPA_ATTR_DEV_NAME, dev_name(&vdev->dev)))
> > > > > > > > @@ -520,6 +523,9 @@ vdpa_dev_fill(struct vdpa_device *vdev, struct sk_buff *msg, u32 portid, u32 seq
> > > > > > > >               goto msg_err;
> > > > > > > >       if (nla_put_u16(msg, VDPA_ATTR_DEV_MAX_VQ_SIZE, max_vq_size))
> > > > > > > >               goto msg_err;
> > > > > > > > +     if (features & BIT_ULL(VIRTIO_F_VERSION_1) &&
> > > > > > > > +         nla_put_flag(msg, VDPA_ATTR_DEV_VERSION_1))
> > > > > > > > +             goto msg_err;
> > > > > > > >
> > > > > > > >       genlmsg_end(msg, hdr);
> > > > > > > >       return 0;
> > > > > > > > diff --git a/drivers/virtio/virtio_vdpa.c b/drivers/virtio/virtio_vdpa.c
> > > > > > > > index 72eaef2caeb1..1cba957c4cdc 100644
> > > > > > > > --- a/drivers/virtio/virtio_vdpa.c
> > > > > > > > +++ b/drivers/virtio/virtio_vdpa.c
> > > > > > > > @@ -7,6 +7,7 @@
> > > > > > > >   *
> > > > > > > >   */
> > > > > > > >
> > > > > > > > +#include "linux/virtio_config.h"
> > > > > > > >  #include <linux/init.h>
> > > > > > > >  #include <linux/module.h>
> > > > > > > >  #include <linux/device.h>
> > > > > > > > @@ -145,6 +146,7 @@ virtio_vdpa_setup_vq(struct virtio_device *vdev, unsigned int index,
> > > > > > > >       /* Assume split virtqueue, switch to packed if necessary */
> > > > > > > >       struct vdpa_vq_state state = {0};
> > > > > > > >       unsigned long flags;
> > > > > > > > +     bool may_reduce_num = false;
> > > > > > > >       u32 align, num;
> > > > > > > >       int err;
> > > > > > > >
> > > > > > > > @@ -169,10 +171,13 @@ virtio_vdpa_setup_vq(struct virtio_device *vdev, unsigned int index,
> > > > > > > >               goto error_new_virtqueue;
> > > > > > > >       }
> > > > > > > >
> > > > > > > > +     if (ops->get_features(vdpa) & BIT_ULL(VIRTIO_F_VERSION_1))
> > > > > > > > +             may_reduce_num = true;
> > > > > > > > +
> > > > > > > >       /* Create the vring */
> > > > > > > >       align = ops->get_vq_align(vdpa);
> > > > > > > >       vq = vring_create_virtqueue(index, num, align, vdev,
> > > > > > > > -                                 true, true, ctx,
> > > > > > > > +                                 true, may_reduce_num, ctx,
> > > > > > > >                                   virtio_vdpa_notify, callback, name);
> > > > > > > >       if (!vq) {
> > > > > > > >               err = -ENOMEM;
> > > > > > > > diff --git a/include/uapi/linux/vdpa.h b/include/uapi/linux/vdpa.h
> > > > > > > > index 66a41e4ec163..ce0b74276a5b 100644
> > > > > > > > --- a/include/uapi/linux/vdpa.h
> > > > > > > > +++ b/include/uapi/linux/vdpa.h
> > > > > > > > @@ -32,6 +32,7 @@ enum vdpa_attr {
> > > > > > > >       VDPA_ATTR_DEV_VENDOR_ID,                /* u32 */
> > > > > > > >       VDPA_ATTR_DEV_MAX_VQS,                  /* u32 */
> > > > > > > >       VDPA_ATTR_DEV_MAX_VQ_SIZE,              /* u16 */
> > > > > > > > +     VDPA_ATTR_DEV_VERSION_1,                /* flag */
> > > > > > > >
> > > > > > > >       /* new attributes must be added above here */
> > > > > > > >       VDPA_ATTR_MAX,
> > > > > > > > --
> > > > > > > > 2.31.1
> > > > > > >
> > > > >
> > >
>