Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20230111062809.25020-1-jasowang@redhat.com> <DM8PR12MB540054712BACEF34E065C2BBABFC9@DM8PR12MB5400.namprd12.prod.outlook.com>
In-Reply-To: <DM8PR12MB540054712BACEF34E065C2BBABFC9@DM8PR12MB5400.namprd12.prod.outlook.com>
From:   Jason Wang <jasowang@redhat.com>
Date:   Thu, 12 Jan 2023 12:03:57 +0800
Message-ID: <CACGkMEtNn5oiQEdY49O4TyCOV68AF7ypGzMQKzRaq0Tc6rvcYA@mail.gmail.com>
Subject: Re: [PATCH 0/5] virtio_ring: per virtqueue DMA device
To:     Eli Cohen <elic@nvidia.com>
Cc:     "mst@redhat.com" <mst@redhat.com>,
        "gdawar@amd.com" <gdawar@amd.com>,
        "virtualization@lists.linux-foundation.org" 
        <virtualization@lists.linux-foundation.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "tanuj.kamde@amd.com" <tanuj.kamde@amd.com>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Wed, Jan 11, 2023 at 9:33 PM Eli Cohen <elic@nvidia.com> wrote:
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Wednesday, 11 January 2023 8:28
> > To: mst@redhat.com; jasowang@redhat.com
> > Cc: Eli Cohen <elic@nvidia.com>; gdawar@amd.com;
> > virtualization@lists.linux-foundation.org; linux-kernel@vger.kernel.org;
> > tanuj.kamde@amd.com
> > Subject: [PATCH 0/5] virtio_ring: per virtqueue DMA device
> >
> > Hi All:
> >
> > In some cases, the virtqueue could be backed by different devices. One
> > example is that in the case of vDPA some parent may emualte virtqueue
> > via vringh. In this case, it would be wrong if we stick with the
> > physical DMA device for software emulated device, since there's no
> > easy way for vringh to know about the hardware IOMMU mappings.
> >
> > So this series tries to introduce per virtqueue DMA device, then
> > software virtqueues can utilize the transport specific method to
> > assign appropirate DMA device.
> >
> > This fixes the crash of mlx5_vdpa + virtio_vdpa when platform IOMMU is
> > enabled but not in the passthrough mode. The reason for the crash is
> > that the virito_ring tries to map the control virtqueue into platform
> > IOMMU but the vringh assumes a direct mapping (PA as IOVA). This is
> > fixed by advetise the vDPA device that doesnt do DMA (without a DMA
> > ops). So DMA API can go with the direct mapping then the vringh will
> > be happy since mlx5_vdpa assuems a direct/identical mapping by
> > default.
> >
>
> Could you provide instructions how to reproduce the crash you were seeing?

It should be something like:

1) boot host kernel with iommu enabled but not in passthrough mode: I
use intel_iommu=on
2) create vdpa device on top of mlx5_vdpa VF
3) bind the vdpa device to virtio_vdpa

Then I can see the crash.

Thanks

>
> > Please review.
> >
> > Thanks
> >
> > Jason Wang (5):
> >   virtio_ring: per virtqueue dma device
> >   vdpa: introduce get_vq_dma_device()
> >   virtio-vdpa: support per vq dma device
> >   vdpa: set dma mask for vDPA device
> >   vdpa: mlx5: support per virtqueue dma device
> >
> >  drivers/vdpa/mlx5/net/mlx5_vnet.c |  11 +++
> >  drivers/vdpa/vdpa.c               |   5 ++
> >  drivers/virtio/virtio_ring.c      | 133 +++++++++++++++++++++---------
> >  drivers/virtio/virtio_vdpa.c      |  13 ++-
> >  include/linux/vdpa.h              |   6 ++
> >  include/linux/virtio_ring.h       |  16 ++++
> >  6 files changed, 141 insertions(+), 43 deletions(-)
> >
> > --
> > 2.25.1
>