Received-SPF: pass (google.com: domain of linux-kernel+bounces-87390-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Feedback-ID: i9e814621:Fastmail
From: Peter Delevoryas <peter@pjd.dev>
Content-Type: text/plain;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.400.31\))
Subject: Re: [q&a] Status of IOMMU virtualization for nested virtualization
 (userspace PCI drivers in VMs)
Date: Thu, 29 Feb 2024 11:53:39 -0800
In-Reply-To: <20240228123810.70663da2.alex.williamson@redhat.com>
Cc: qemu-devel <qemu-devel@nongnu.org>,
 suravee.suthikulpanit@amd.com,
 iommu@lists.linux.dev,
 kvm@vger.kernel.org,
 linux-kernel@vger.kernel.org,
 alex.williamson@redhat.com,
 Peter Delevoryas <peter@pjd.dev>
References: <3D96D76D-85D2-47B5-B4C1-D6F95061D7D6@pjd.dev>
 <20240228123810.70663da2.alex.williamson@redhat.com>
Message-Id: <D4DDA526-5E5D-40DB-86EF-B4B6D7692663@pjd.dev>


> On Feb 28, 2024, at 11:38=E2=80=AFAM, Alex Williamson =
<alex.williamson@redhat.com> wrote:
>=20
> On Wed, 28 Feb 2024 10:29:32 -0800
> Peter Delevoryas <peter@pjd.dev> wrote:
>=20
>> Hey guys,
>>=20
>> I=E2=80=99m having a little trouble reading between the lines on =
various
>> docs, mailing list threads, KVM presentations, github forks, etc, so
>> I figured I=E2=80=99d just ask:
>>=20
>> What is the status of IOMMU virtualization, like in the case where I
>> want a VM guest to have a virtual IOMMU?
>=20
> It works fine for simply nested assignment scenarios, ie. guest
> userspace drivers or nested VMs.
>=20
>> I found this great presentation from KVM Forum 2021: [1]
>>=20
>> 1. I=E2=80=99m using -device intel-iommu right now. This has =
performance
>> implications and large DMA transfers hit the vfio_iommu_type1
>> dma_entry_limit on the host because of how the mappings are made.
>=20
> Hugepages for the guest and mappings within the guest should help both
> the mapping performance and DMA entry limit.  In general the type1 =
vfio
> IOMMU backend is not optimized for dynamic mapping, so =
performance-wise
> your best bet is still to design the userspace driver for static DMA
> buffers.

Yep, huge pages definitely help, will probably switch to allocating them =
at boot for better guarantees.

>=20
>> 2. -device virtio-iommu is an improvement, but it doesn=E2=80=99t =
seem
>> compatible with -device vfio-pci? I was only able to test this with
>> cloud-hypervisor, and it has a better vfio mapping pattern (avoids
>> hitting dma_entry_limit).
>=20
> AFAIK it's just growing pains, it should work but it's working through
> bugs.

Oh really?? Ok: I might even be configuring things incorrectly, or
Maybe I need to upgrade from QEMU 7.1 to 8. I was relying on whatever
libvirt does by default, which seems to just be:

    -device virtio-iommu -device vfio-pci,host=3D<bdf>

But maybe I need some other options?

>=20
>> 3. -object iommufd [2] I haven=E2=80=99t tried this quite yet, =
planning to:
>> if it=E2=80=99s using iommufd, and I have all the right kernel =
features in
>> the guest and host, I assume it=E2=80=99s implementing the =
passthrough mode
>> that AMD has described in their talk? Because I imagine that would be
>> the best solution for me, I=E2=80=99m just having trouble =
understanding if
>> it=E2=80=99s actually related or orthogonal.
>=20
> For now iommufd provides a similar DMA mapping interface to type1, but
> it does remove the DMA entry limit and improves locked page =
accounting.
>=20
> To really see a performance improvement relative to dynamic mappings,
> you'll need nesting support in the IOMMU, which is under active
> development.  =46rom this aspect you will want iommufd since similar
> features will not be provided by type1.  Thanks,

I see, thanks! That=E2=80=99s great to hear.

>=20
> Alex
>=20