MIME-Version: 1.0
References: <20210830141737.181-1-xieyongji@bytedance.com> <20220110075546-mutt-send-email-mst@kernel.org>
 <CACycT3v1aEViw7vV4x5qeGVPrSrO-BTDvQshEX35rx_X0Au2vw@mail.gmail.com>
 <20220110100911-mutt-send-email-mst@kernel.org> <CACycT3v6jo3-8ATWUzf659vV94a2oRrm-zQtGNDZd6OQr-MENA@mail.gmail.com>
 <20220110103938-mutt-send-email-mst@kernel.org>
In-Reply-To: <20220110103938-mutt-send-email-mst@kernel.org>
From:   Yongji Xie <xieyongji@bytedance.com>
Date:   Tue, 11 Jan 2022 11:31:37 +0800
Message-ID: <CACycT3sbJC1Jn7NeWk_ccQ_2_YgKybjugfxmKpfgCP3Ayoju4w@mail.gmail.com>
Subject: Re: [PATCH v12 00/13] Introduce VDUSE - vDPA Device in Userspace
To:     "Michael S. Tsirkin" <mst@redhat.com>
Cc:     Jason Wang <jasowang@redhat.com>,
        Stefan Hajnoczi <stefanha@redhat.com>,
        Stefano Garzarella <sgarzare@redhat.com>,
        Parav Pandit <parav@nvidia.com>,
        Christoph Hellwig <hch@infradead.org>,
        Christian Brauner <christian.brauner@canonical.com>,
        Randy Dunlap <rdunlap@infradead.org>,
        Matthew Wilcox <willy@infradead.org>,
        Al Viro <viro@zeniv.linux.org.uk>,
        Jens Axboe <axboe@kernel.dk>, bcrl@kvack.org,
        Jonathan Corbet <corbet@lwn.net>,
        =?UTF-8?Q?Mika_Penttil=C3=A4?= <mika.penttila@nextfour.com>,
        Dan Carpenter <dan.carpenter@oracle.com>, joro@8bytes.org,
        Greg KH <gregkh@linuxfoundation.org>,
        He Zhe <zhe.he@windriver.com>,
        Liu Xiaodong <xiaodong.liu@intel.com>,
        Joe Perches <joe@perches.com>,
        Robin Murphy <robin.murphy@arm.com>,
        Will Deacon <will@kernel.org>,
        John Garry <john.garry@huawei.com>, songmuchun@bytedance.com,
        virtualization <virtualization@lists.linux-foundation.org>,
        Netdev <netdev@vger.kernel.org>, kvm <kvm@vger.kernel.org>,
        linux-fsdevel@vger.kernel.org, iommu@lists.linux-foundation.org,
        linux-kernel <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Mon, Jan 10, 2022 at 11:44 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Mon, Jan 10, 2022 at 11:24:40PM +0800, Yongji Xie wrote:
> > On Mon, Jan 10, 2022 at 11:10 PM Michael S. Tsirkin <mst@redhat.com> wr=
ote:
> > >
> > > On Mon, Jan 10, 2022 at 09:54:08PM +0800, Yongji Xie wrote:
> > > > On Mon, Jan 10, 2022 at 8:57 PM Michael S. Tsirkin <mst@redhat.com>=
 wrote:
> > > > >
> > > > > On Mon, Aug 30, 2021 at 10:17:24PM +0800, Xie Yongji wrote:
> > > > > > This series introduces a framework that makes it possible to im=
plement
> > > > > > software-emulated vDPA devices in userspace. And to make the de=
vice
> > > > > > emulation more secure, the emulated vDPA device's control path =
is handled
> > > > > > in the kernel and only the data path is implemented in the user=
space.
> > > > > >
> > > > > > Since the emuldated vDPA device's control path is handled in th=
e kernel,
> > > > > > a message mechnism is introduced to make userspace be aware of =
the data
> > > > > > path related changes. Userspace can use read()/write() to recei=
ve/reply
> > > > > > the control messages.
> > > > > >
> > > > > > In the data path, the core is mapping dma buffer into VDUSE dae=
mon's
> > > > > > address space, which can be implemented in different ways depen=
ding on
> > > > > > the vdpa bus to which the vDPA device is attached.
> > > > > >
> > > > > > In virtio-vdpa case, we implements a MMU-based software IOTLB w=
ith
> > > > > > bounce-buffering mechanism to achieve that. And in vhost-vdpa c=
ase, the dma
> > > > > > buffer is reside in a userspace memory region which can be shar=
ed to the
> > > > > > VDUSE userspace processs via transferring the shmfd.
> > > > > >
> > > > > > The details and our user case is shown below:
> > > > > >
> > > > > > ------------------------    -------------------------   -------=
---------------------------------------
> > > > > > |            Container |    |              QEMU(VM) |   |      =
                         VDUSE daemon |
> > > > > > |       ---------      |    |  -------------------  |   | -----=
-------------------- ---------------- |
> > > > > > |       |dev/vdx|      |    |  |/dev/vhost-vdpa-x|  |   | | vDP=
A device emulation | | block driver | |
> > > > > > ------------+-----------     -----------+------------   -------=
------+----------------------+---------
> > > > > >             |                           |                      =
      |                      |
> > > > > >             |                           |                      =
      |                      |
> > > > > > ------------+---------------------------+----------------------=
------+----------------------+---------
> > > > > > |    | block device |           |  vhost device |            | =
vduse driver |          | TCP/IP |    |
> > > > > > |    -------+--------           --------+--------            --=
-----+--------          -----+----    |
> > > > > > |           |                           |                      =
     |                       |        |
> > > > > > | ----------+----------       ----------+-----------         --=
-----+-------                |        |
> > > > > > | | virtio-blk driver |       |  vhost-vdpa driver |         | =
vdpa device |                |        |
> > > > > > | ----------+----------       ----------+-----------         --=
-----+-------                |        |
> > > > > > |           |      virtio bus           |                      =
     |                       |        |
> > > > > > |   --------+----+-----------           |                      =
     |                       |        |
> > > > > > |                |                      |                      =
     |                       |        |
> > > > > > |      ----------+----------            |                      =
     |                       |        |
> > > > > > |      | virtio-blk device |            |                      =
     |                       |        |
> > > > > > |      ----------+----------            |                      =
     |                       |        |
> > > > > > |                |                      |                      =
     |                       |        |
> > > > > > |     -----------+-----------           |                      =
     |                       |        |
> > > > > > |     |  virtio-vdpa driver |           |                      =
     |                       |        |
> > > > > > |     -----------+-----------           |                      =
     |                       |        |
> > > > > > |                |                      |                      =
     |    vdpa bus           |        |
> > > > > > |     -----------+----------------------+----------------------=
-----+------------           |        |
> > > > > > |                                                              =
                          ---+---     |
> > > > > > ---------------------------------------------------------------=
--------------------------| NIC |------
> > > > > >                                                                =
                          ---+---
> > > > > >                                                                =
                             |
> > > > > >                                                                =
                    ---------+---------
> > > > > >                                                                =
                    | Remote Storages |
> > > > > >                                                                =
                    -------------------
> > > > > >
> > > > > > We make use of it to implement a block device connecting to
> > > > > > our distributed storage, which can be used both in containers a=
nd
> > > > > > VMs. Thus, we can have an unified technology stack in this two =
cases.
> > > > > >
> > > > > > To test it with null-blk:
> > > > > >
> > > > > >   $ qemu-storage-daemon \
> > > > > >       --chardev socket,id=3Dcharmonitor,path=3D/tmp/qmp.sock,se=
rver,nowait \
> > > > > >       --monitor chardev=3Dcharmonitor \
> > > > > >       --blockdev driver=3Dhost_device,cache.direct=3Don,aio=3Dn=
ative,filename=3D/dev/nullb0,node-name=3Ddisk0 \
> > > > > >       --export type=3Dvduse-blk,id=3Dtest,node-name=3Ddisk0,wri=
table=3Don,name=3Dvduse-null,num-queues=3D16,queue-size=3D128
> > > > > >
> > > > > > The qemu-storage-daemon can be found at https://github.com/byte=
dance/qemu/tree/vduse
> > > > >
> > > > > It's been half a year - any plans to upstream this?
> > > >
> > > > Yeah, this is on my to-do list this month.
> > > >
> > > > Sorry for taking so long... I've been working on another project
> > > > enabling userspace RDMA with VDUSE for the past few months. So I
> > > > didn't have much time for this. Anyway, I will submit the first
> > > > version as soon as possible.
> > > >
> > > > Thanks,
> > > > Yongji
> > >
> > > Oh fun. You mean like virtio-rdma? Or RDMA as a backend for regular
> > > virtio?
> > >
> >
> > Yes, like virtio-rdma. Then we can develop something like userspace
> > rxe=E3=80=81siw or custom protocol with VDUSE.
> >
> > Thanks,
> > Yongji
>
> Would be interesting to see the spec for that.

Will send it ASAP.

> The issues with RDMA revolved around the fact that current
> apps tend to either use non-standard propocols for connection
> establishment or use UD where there's IIRC no standard
> at all. So QP numbers are hard to virtualize.
> Similarly many use LIDs directly with the same effect.
> GUIDs might be virtualizeable but no one went to the effort.
>

Actually we aimed at emulating a soft RDMA with normal NIC (not use
RDMA capability) rather than virtualizing a physical RDMA NIC into
several vRDMA devices. If so, I think we won't have those issues,
right?

> To say nothing about the interaction with memory overcommit.
>

I don't get you here. Could you give me more details?

Thanks,
Yongji