2020-10-15 04:24:47

by Suthikulpanit, Suravee

Subject: [PATCH] iommu/amd: Increase interrupt remapping table limit to 512 entries

Certain device drivers allocate IO queues on a per-cpu basis.
On AMD EPYC platform, which can support up-to 256 cpu threads,
this can exceed the current MAX_IRQ_PER_TABLE limit of 256,
and result in the error message:

AMD-Vi: Failed to allocate IRTE

This has been observed with certain NVME devices.

AMD IOMMU hardware can actually support upto 512 interrupt
remapping table entries. Therefore, update the driver to
match the hardware limit.

Please note that this also increases the size of interrupt remapping
table to 8KB per device when using the 128-bit IRTE format.

Signed-off-by: Suravee Suthikulpanit <[email protected]>
---
drivers/iommu/amd/amd_iommu_types.h | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
index 30a5d412255a..427484c45589 100644
--- a/drivers/iommu/amd/amd_iommu_types.h
+++ b/drivers/iommu/amd/amd_iommu_types.h
@@ -406,7 +406,11 @@ extern bool amd_iommu_np_cache;
/* Only true if all IOMMUs support device IOTLBs */
extern bool amd_iommu_iotlb_sup;

-#define MAX_IRQS_PER_TABLE 256
+/*
+ * AMD IOMMU hardware only support 512 IRTEs despite
+ * the architectural limitation of 2048 entries.
+ */
+#define MAX_IRQS_PER_TABLE 512
#define IRQ_TABLE_ALIGNMENT 128

struct irq_remap_table {
--
2.17.1

2020-10-26 04:25:32

by Suthikulpanit, Suravee

Subject: Re: [PATCH] iommu/amd: Increase interrupt remapping table limit to 512 entries

Hi Joerg,

Do you have any concerns regarding this patch?

Thanks,
Suravee

On 10/15/20 9:50 AM, Suravee Suthikulpanit wrote:
> Certain device drivers allocate IO queues on a per-cpu basis.
> On AMD EPYC platform, which can support up-to 256 cpu threads,
> this can exceed the current MAX_IRQ_PER_TABLE limit of 256,
> and result in the error message:
>
> AMD-Vi: Failed to allocate IRTE
>
> This has been observed with certain NVME devices.
>
> AMD IOMMU hardware can actually support upto 512 interrupt
> remapping table entries. Therefore, update the driver to
> match the hardware limit.
>
> Please note that this also increases the size of interrupt remapping
> table to 8KB per device when using the 128-bit IRTE format.
>
> Signed-off-by: Suravee Suthikulpanit <[email protected]>
> ---
> drivers/iommu/amd/amd_iommu_types.h | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
> index 30a5d412255a..427484c45589 100644
> --- a/drivers/iommu/amd/amd_iommu_types.h
> +++ b/drivers/iommu/amd/amd_iommu_types.h
> @@ -406,7 +406,11 @@ extern bool amd_iommu_np_cache;
> /* Only true if all IOMMUs support device IOTLBs */
> extern bool amd_iommu_iotlb_sup;
>
> -#define MAX_IRQS_PER_TABLE 256
> +/*
> + * AMD IOMMU hardware only support 512 IRTEs despite
> + * the architectural limitation of 2048 entries.
> + */
> +#define MAX_IRQS_PER_TABLE 512
> #define IRQ_TABLE_ALIGNMENT 128
>
> struct irq_remap_table {
>

2020-11-03 13:25:50

by Joerg Roedel

Subject: Re: [PATCH] iommu/amd: Increase interrupt remapping table limit to 512 entries

On Thu, Oct 15, 2020 at 02:50:02AM +0000, Suravee Suthikulpanit wrote:
> Certain device drivers allocate IO queues on a per-cpu basis.
> On AMD EPYC platform, which can support up-to 256 cpu threads,
> this can exceed the current MAX_IRQ_PER_TABLE limit of 256,
> and result in the error message:
>
> AMD-Vi: Failed to allocate IRTE
>
> This has been observed with certain NVME devices.
>
> AMD IOMMU hardware can actually support upto 512 interrupt
> remapping table entries. Therefore, update the driver to
> match the hardware limit.
>
> Please note that this also increases the size of interrupt remapping
> table to 8KB per device when using the 128-bit IRTE format.
>
> Signed-off-by: Suravee Suthikulpanit <[email protected]>
> ---
> drivers/iommu/amd/amd_iommu_types.h | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)

Applied for 5.10, thanks.

2020-12-01 23:57:32

by Jerry Snitselaar

Subject: Re: [PATCH] iommu/amd: Increase interrupt remapping table limit to 512 entries

Suravee Suthikulpanit @ 2020-10-14 19:50 MST:

> Certain device drivers allocate IO queues on a per-cpu basis.
> On AMD EPYC platform, which can support up-to 256 cpu threads,
> this can exceed the current MAX_IRQ_PER_TABLE limit of 256,
> and result in the error message:
>
> AMD-Vi: Failed to allocate IRTE
>
> This has been observed with certain NVME devices.
>
> AMD IOMMU hardware can actually support upto 512 interrupt
> remapping table entries. Therefore, update the driver to
> match the hardware limit.
>
> Please note that this also increases the size of interrupt remapping
> table to 8KB per device when using the 128-bit IRTE format.
>
> Signed-off-by: Suravee Suthikulpanit <[email protected]>
> ---
> drivers/iommu/amd/amd_iommu_types.h | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
> index 30a5d412255a..427484c45589 100644
> --- a/drivers/iommu/amd/amd_iommu_types.h
> +++ b/drivers/iommu/amd/amd_iommu_types.h
> @@ -406,7 +406,11 @@ extern bool amd_iommu_np_cache;
> /* Only true if all IOMMUs support device IOTLBs */
> extern bool amd_iommu_iotlb_sup;
>
> -#define MAX_IRQS_PER_TABLE 256
> +/*
> + * AMD IOMMU hardware only support 512 IRTEs despite
> + * the architectural limitation of 2048 entries.
> + */
> +#define MAX_IRQS_PER_TABLE 512
> #define IRQ_TABLE_ALIGNMENT 128
>
> struct irq_remap_table {

With this change should DTE_IRQ_TABLE_LEN be changed to 9? IIUC the spec
correctly leaving it at 8 is saying the table is 256 entries long.

Regards,
Jerry

2020-12-07 03:13:24

by Suthikulpanit, Suravee

Subject: Re: [PATCH] iommu/amd: Increase interrupt remapping table limit to 512 entries

Jerry,

On 12/2/20 6:53 AM, Jerry Snitselaar wrote:
>
> Suravee Suthikulpanit @ 2020-10-14 19:50 MST:
>
>> Certain device drivers allocate IO queues on a per-cpu basis.
>> On AMD EPYC platform, which can support up-to 256 cpu threads,
>> this can exceed the current MAX_IRQ_PER_TABLE limit of 256,
>> and result in the error message:
>>
>> AMD-Vi: Failed to allocate IRTE
>>
>> This has been observed with certain NVME devices.
>>
>> AMD IOMMU hardware can actually support upto 512 interrupt
>> remapping table entries. Therefore, update the driver to
>> match the hardware limit.
>>
>> Please note that this also increases the size of interrupt remapping
>> table to 8KB per device when using the 128-bit IRTE format.
>>
>> Signed-off-by: Suravee Suthikulpanit <[email protected]>
>> ---
>> drivers/iommu/amd/amd_iommu_types.h | 6 +++++-
>> 1 file changed, 5 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/iommu/amd/amd_iommu_types.h b/drivers/iommu/amd/amd_iommu_types.h
>> index 30a5d412255a..427484c45589 100644
>> --- a/drivers/iommu/amd/amd_iommu_types.h
>> +++ b/drivers/iommu/amd/amd_iommu_types.h
>> @@ -406,7 +406,11 @@ extern bool amd_iommu_np_cache;
>> /* Only true if all IOMMUs support device IOTLBs */
>> extern bool amd_iommu_iotlb_sup;
>>
>> -#define MAX_IRQS_PER_TABLE 256
>> +/*
>> + * AMD IOMMU hardware only support 512 IRTEs despite
>> + * the architectural limitation of 2048 entries.
>> + */
>> +#define MAX_IRQS_PER_TABLE 512
>> #define IRQ_TABLE_ALIGNMENT 128
>>
>> struct irq_remap_table {
>
> With this change should DTE_IRQ_TABLE_LEN be changed to 9? IIUC the spec
> correctly leaving it at 8 is saying the table is 256 entries long.

You are correct. Sorry I missed this part. I'll send the fix-up patch ASAP.

Thank you,
Suravee