v2:
- Fixed missing error unwind in patch 3 (Alex). After more thought,
the ENODEV case is fine, so it stayed the same.
- Rebased on linux-vfio.git/next (no conflicts).
---
The VFIO type1 driver is calling pin_user_pages_remote() once per 4k page, so
let's do it once per 512 4k pages to bring VFIO in line with other drivers such
as IB and vDPA.
qemu guests with at least 2G memory start about 8% faster on a Xeon server,
with more detailed results in the last changelog.
Thanks to Matthew, who first suggested the idea to me.
Daniel
Test Cases
----------
1) qemu passthrough with IOMMU-capable PCI device
2) standalone program to hit
vfio_pin_map_dma() -> vfio_pin_pages_remote()
3) standalone program to hit
vfio_iommu_replay() -> vfio_pin_pages_remote()
Each was run...
- with varying sizes
- with/without disable_hugepages=1
- with/without LOCKED_VM exceeded
I didn't test vfio_pin_page_external() because there was no readily available
hardware, but the changes there are pretty minimal.
Daniel Jordan (3):
vfio/type1: Change success value of vaddr_get_pfn()
vfio/type1: Prepare for batched pinning with struct vfio_batch
vfio/type1: Batch page pinning
drivers/vfio/vfio_iommu_type1.c | 215 +++++++++++++++++++++++---------
1 file changed, 155 insertions(+), 60 deletions(-)
base-commit: 76adb20f924f8d27ed50d02cd29cadedb59fd88f
--
2.30.1
vaddr_get_pfn() simply returns 0 on success. Have it report the number
of pfns successfully gotten instead, whether from page pinning or
follow_fault_pfn(), which will be used later when batching pinning.
Change the last check in vfio_pin_pages_remote() for consistency with
the other two.
Signed-off-by: Daniel Jordan <[email protected]>
---
drivers/vfio/vfio_iommu_type1.c | 21 ++++++++++++++-------
1 file changed, 14 insertions(+), 7 deletions(-)
diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index ec9fd95a138b..7abaaad518a6 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -485,6 +485,10 @@ static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm,
return ret;
}
+/*
+ * Returns the positive number of pfns successfully obtained or a negative
+ * error code.
+ */
static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr,
int prot, unsigned long *pfn)
{
@@ -501,7 +505,6 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr,
page, NULL, NULL);
if (ret == 1) {
*pfn = page_to_pfn(page[0]);
- ret = 0;
goto done;
}
@@ -515,8 +518,12 @@ static int vaddr_get_pfn(struct mm_struct *mm, unsigned long vaddr,
if (ret == -EAGAIN)
goto retry;
- if (!ret && !is_invalid_reserved_pfn(*pfn))
- ret = -EFAULT;
+ if (!ret) {
+ if (is_invalid_reserved_pfn(*pfn))
+ ret = 1;
+ else
+ ret = -EFAULT;
+ }
}
done:
mmap_read_unlock(mm);
@@ -597,7 +604,7 @@ static long vfio_pin_pages_remote(struct vfio_dma *dma, unsigned long vaddr,
return -ENODEV;
ret = vaddr_get_pfn(current->mm, vaddr, dma->prot, pfn_base);
- if (ret)
+ if (ret < 0)
return ret;
pinned++;
@@ -624,7 +631,7 @@ static long vfio_pin_pages_remote(struct vfio_dma *dma, unsigned long vaddr,
for (vaddr += PAGE_SIZE, iova += PAGE_SIZE; pinned < npage;
pinned++, vaddr += PAGE_SIZE, iova += PAGE_SIZE) {
ret = vaddr_get_pfn(current->mm, vaddr, dma->prot, &pfn);
- if (ret)
+ if (ret < 0)
break;
if (pfn != *pfn_base + pinned ||
@@ -650,7 +657,7 @@ static long vfio_pin_pages_remote(struct vfio_dma *dma, unsigned long vaddr,
ret = vfio_lock_acct(dma, lock_acct, false);
unpin_out:
- if (ret) {
+ if (ret < 0) {
if (!rsvd) {
for (pfn = *pfn_base ; pinned ; pfn++, pinned--)
put_pfn(pfn, dma->prot);
@@ -694,7 +701,7 @@ static int vfio_pin_page_external(struct vfio_dma *dma, unsigned long vaddr,
return -ENODEV;
ret = vaddr_get_pfn(mm, vaddr, dma->prot, pfn_base);
- if (!ret && do_accounting && !is_invalid_reserved_pfn(*pfn_base)) {
+ if (ret == 1 && do_accounting && !is_invalid_reserved_pfn(*pfn_base)) {
ret = vfio_lock_acct(dma, 1, true);
if (ret) {
put_pfn(*pfn_base, dma->prot);
--
2.30.1
On Fri, 19 Feb 2021 11:13:02 -0500
Daniel Jordan <[email protected]> wrote:
> v2:
> - Fixed missing error unwind in patch 3 (Alex). After more thought,
> the ENODEV case is fine, so it stayed the same.
>
> - Rebased on linux-vfio.git/next (no conflicts).
>
> ---
>
> The VFIO type1 driver is calling pin_user_pages_remote() once per 4k page, so
> let's do it once per 512 4k pages to bring VFIO in line with other drivers such
> as IB and vDPA.
>
> qemu guests with at least 2G memory start about 8% faster on a Xeon server,
> with more detailed results in the last changelog.
>
> Thanks to Matthew, who first suggested the idea to me.
>
> Daniel
>
>
> Test Cases
> ----------
>
> 1) qemu passthrough with IOMMU-capable PCI device
>
> 2) standalone program to hit
> vfio_pin_map_dma() -> vfio_pin_pages_remote()
>
> 3) standalone program to hit
> vfio_iommu_replay() -> vfio_pin_pages_remote()
>
> Each was run...
>
> - with varying sizes
> - with/without disable_hugepages=1
> - with/without LOCKED_VM exceeded
>
> I didn't test vfio_pin_page_external() because there was no readily available
> hardware, but the changes there are pretty minimal.
>
> Daniel Jordan (3):
> vfio/type1: Change success value of vaddr_get_pfn()
> vfio/type1: Prepare for batched pinning with struct vfio_batch
> vfio/type1: Batch page pinning
>
> drivers/vfio/vfio_iommu_type1.c | 215 +++++++++++++++++++++++---------
> 1 file changed, 155 insertions(+), 60 deletions(-)
>
> base-commit: 76adb20f924f8d27ed50d02cd29cadedb59fd88f
Applied to vfio next branch for v5.12. Thanks,
Alex