Commit 492855939bdb ("vfio/type1: Limit DMA mappings per container") added
a limit to the number of concurrent DMA requests for a vfio container.
However, lazy unmapping in s390 can in fact cause quite a large number of
outstanding DMA requests to build up prior to being purged, potentially
the entire guest DMA space. This results in unexpected 'VFIO_MAP_DMA
failed: No space left on device' conditions seen in QEMU.
This patch proposes to provide the remaining number of allowable DMA
requests via the VFIO_IOMMU_GET_INFO ioctl as a new capability. A
subsequent patchset to QEMU would collect this information and use it in
s390 PCI support to tap the guest on the shoulder before overrunning the
vfio limit.
Changes from v1:
- Report dma_avail instead of the limit, which might not have been accurate
anyway
- Text/naming changes throughout due to the above
Matthew Rosato (1):
vfio iommu: Add dma available capability
drivers/vfio/vfio_iommu_type1.c | 17 +++++++++++++++++
include/uapi/linux/vfio.h | 16 ++++++++++++++++
2 files changed, 33 insertions(+)
--
1.8.3.1
Commit 492855939bdb ("vfio/type1: Limit DMA mappings per container")
added the ability to limit the number of memory backed DMA mappings.
However on s390x, when lazy mapping is in use, we use a very large
number of concurrent mappings. Let's provide the current allowable
number of DMA mappings to userspace via the IOMMU info chain so that
userspace can take appropriate mitigation.
Signed-off-by: Matthew Rosato <[email protected]>
---
drivers/vfio/vfio_iommu_type1.c | 17 +++++++++++++++++
include/uapi/linux/vfio.h | 16 ++++++++++++++++
2 files changed, 33 insertions(+)
diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index 5fbf0c1..15e21db 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -2609,6 +2609,20 @@ static int vfio_iommu_migration_build_caps(struct vfio_iommu *iommu,
return vfio_info_add_capability(caps, &cap_mig.header, sizeof(cap_mig));
}
+static int vfio_iommu_dma_avail_build_caps(struct vfio_iommu *iommu,
+ struct vfio_info_cap *caps)
+{
+ struct vfio_iommu_type1_info_dma_avail cap_dma_avail;
+
+ cap_dma_avail.header.id = VFIO_IOMMU_TYPE1_INFO_DMA_AVAIL;
+ cap_dma_avail.header.version = 1;
+
+ cap_dma_avail.avail = iommu->dma_avail;
+
+ return vfio_info_add_capability(caps, &cap_dma_avail.header,
+ sizeof(cap_dma_avail));
+}
+
static int vfio_iommu_type1_get_info(struct vfio_iommu *iommu,
unsigned long arg)
{
@@ -2642,6 +2656,9 @@ static int vfio_iommu_type1_get_info(struct vfio_iommu *iommu,
ret = vfio_iommu_migration_build_caps(iommu, &caps);
if (!ret)
+ ret = vfio_iommu_dma_avail_build_caps(iommu, &caps);
+
+ if (!ret)
ret = vfio_iommu_iova_build_caps(iommu, &caps);
mutex_unlock(&iommu->lock);
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index 9204705..a8cc4a5 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -1039,6 +1039,22 @@ struct vfio_iommu_type1_info_cap_migration {
__u64 max_dirty_bitmap_size; /* in bytes */
};
+/*
+ * The DMA available capability allows to report the current number of
+ * simultaneously outstanding DMA mappings that are allowed.
+ *
+ * The structures below define version 1 of this capability.
+ *
+ * max: specifies the maximum number of outstanding DMA mappings allowed.
+ */
+#define VFIO_IOMMU_TYPE1_INFO_DMA_AVAIL 3
+
+struct vfio_iommu_type1_info_dma_avail {
+ struct vfio_info_cap_header header;
+ __u32 avail;
+};
+
+
#define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
/**
--
1.8.3.1
On 9/14/20 6:25 PM, Matthew Rosato wrote:
> Commit 492855939bdb ("vfio/type1: Limit DMA mappings per container") added
> a limit to the number of concurrent DMA requests for a vfio container.
> However, lazy unmapping in s390 can in fact cause quite a large number of
> outstanding DMA requests to build up prior to being purged, potentially
> the entire guest DMA space. This results in unexpected 'VFIO_MAP_DMA
> failed: No space left on device' conditions seen in QEMU.
>
> This patch proposes to provide the remaining number of allowable DMA
> requests via the VFIO_IOMMU_GET_INFO ioctl as a new capability. A
> subsequent patchset to QEMU would collect this information and use it in
> s390 PCI support to tap the guest on the shoulder before overrunning the
> vfio limit.
Link to associated v2 QEMU patchset:
https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg05067.html
>
> Changes from v1:
> - Report dma_avail instead of the limit, which might not have been accurate
> anyway
> - Text/naming changes throughout due to the above
>
> Matthew Rosato (1):
> vfio iommu: Add dma available capability
>
> drivers/vfio/vfio_iommu_type1.c | 17 +++++++++++++++++
> include/uapi/linux/vfio.h | 16 ++++++++++++++++
> 2 files changed, 33 insertions(+)
>
On Mon, 14 Sep 2020 18:25:31 -0400
Matthew Rosato <[email protected]> wrote:
> Commit 492855939bdb ("vfio/type1: Limit DMA mappings per container")
> added the ability to limit the number of memory backed DMA mappings.
> However on s390x, when lazy mapping is in use, we use a very large
> number of concurrent mappings. Let's provide the current allowable
> number of DMA mappings to userspace via the IOMMU info chain so that
> userspace can take appropriate mitigation.
>
> Signed-off-by: Matthew Rosato <[email protected]>
> ---
> drivers/vfio/vfio_iommu_type1.c | 17 +++++++++++++++++
> include/uapi/linux/vfio.h | 16 ++++++++++++++++
> 2 files changed, 33 insertions(+)
(...)
> diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
> index 9204705..a8cc4a5 100644
> --- a/include/uapi/linux/vfio.h
> +++ b/include/uapi/linux/vfio.h
> @@ -1039,6 +1039,22 @@ struct vfio_iommu_type1_info_cap_migration {
> __u64 max_dirty_bitmap_size; /* in bytes */
> };
>
> +/*
> + * The DMA available capability allows to report the current number of
> + * simultaneously outstanding DMA mappings that are allowed.
> + *
> + * The structures below define version 1 of this capability.
"The structure below defines..." ?
> + *
> + * max: specifies the maximum number of outstanding DMA mappings allowed.
I think you forgot to tweak that one:
"avail: specifies the current number of outstanding DMA mappings allowed."
?
> + */
> +#define VFIO_IOMMU_TYPE1_INFO_DMA_AVAIL 3
> +
> +struct vfio_iommu_type1_info_dma_avail {
> + struct vfio_info_cap_header header;
> + __u32 avail;
> +};
> +
> +
> #define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
>
> /**
On 9/15/20 5:44 AM, Cornelia Huck wrote:
> On Mon, 14 Sep 2020 18:25:31 -0400
> Matthew Rosato <[email protected]> wrote:
>
>> Commit 492855939bdb ("vfio/type1: Limit DMA mappings per container")
>> added the ability to limit the number of memory backed DMA mappings.
>> However on s390x, when lazy mapping is in use, we use a very large
>> number of concurrent mappings. Let's provide the current allowable
>> number of DMA mappings to userspace via the IOMMU info chain so that
>> userspace can take appropriate mitigation.
>>
>> Signed-off-by: Matthew Rosato <[email protected]>
>> ---
>> drivers/vfio/vfio_iommu_type1.c | 17 +++++++++++++++++
>> include/uapi/linux/vfio.h | 16 ++++++++++++++++
>> 2 files changed, 33 insertions(+)
>
> (...)
>
>> diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
>> index 9204705..a8cc4a5 100644
>> --- a/include/uapi/linux/vfio.h
>> +++ b/include/uapi/linux/vfio.h
>> @@ -1039,6 +1039,22 @@ struct vfio_iommu_type1_info_cap_migration {
>> __u64 max_dirty_bitmap_size; /* in bytes */
>> };
>>
>> +/*
>> + * The DMA available capability allows to report the current number of
>> + * simultaneously outstanding DMA mappings that are allowed.
>> + *
>> + * The structures below define version 1 of this capability.
>
> "The structure below defines..." ?
>
OK
>> + *
>> + * max: specifies the maximum number of outstanding DMA mappings allowed.
>
> I think you forgot to tweak that one:
>
> "avail: specifies the current number of outstanding DMA mappings allowed."
>
> ?
Yep, that's a leftover from v1 :(
>
>> + */
>> +#define VFIO_IOMMU_TYPE1_INFO_DMA_AVAIL 3
>> +
>> +struct vfio_iommu_type1_info_dma_avail {
>> + struct vfio_info_cap_header header;
>> + __u32 avail;
>> +};
>> +
>> +
>> #define VFIO_IOMMU_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 12)
>>
>> /**
>