2021-10-27 21:27:51

by Eric Auger

[permalink] [raw]
Subject: [RFC v16 0/9] SMMUv3 Nested Stage Setup (IOMMU part)

This series brings the IOMMU part of HW nested paging support
in the SMMUv3.

The SMMUv3 driver is adapted to support 2 nested stages.

The IOMMU API is extended to convey the guest stage 1
configuration and the hook is implemented in the SMMUv3 driver.

This allows the guest to own the stage 1 tables and context
descriptors (so-called PASID table) while the host owns the
stage 2 tables and main configuration structures (STE).

This work mainly is provided for test purpose as the upper
layer integration is under rework and bound to be based on
/dev/iommu instead of VFIO tunneling. In this version we also get
rid of the MSI BINDING ioctl, assuming the guest enforces
flat mapping of host IOVAs used to bind physical MSI doorbells.
In the current QEMU integration this is achieved by exposing
RMRs to the guest, using Shameer's series [1]. This approach
is RFC as the IORT spec is not really meant to do that
(single mapping flag limitation).

Best Regards

Eric

This series (Host) can be found at:
https://github.com/eauger/linux/tree/v5.15-rc7-nested-v16
This includes a rebased VFIO integration (although not meant
to be upstreamed)

Guest kernel branch can be found at:
https://github.com/eauger/linux/tree/shameer_rmrr_v7
featuring [1]

QEMU integration (still based on VFIO and exposing RMRs)
can be found at:
https://github.com/eauger/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10
(use iommu=nested-smmuv3 ARM virt option)

Guest dependency:
[1] [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node

History:

v15 -> v16:
- guest RIL must support RIL
- additional checks in the cache invalidation hook
- removal of the MSI BINDING ioctl (tentative replacement
by RMRs)


Eric Auger (9):
iommu: Introduce attach/detach_pasid_table API
iommu: Introduce iommu_get_nesting
iommu/smmuv3: Allow s1 and s2 configs to coexist
iommu/smmuv3: Get prepared for nested stage support
iommu/smmuv3: Implement attach/detach_pasid_table
iommu/smmuv3: Allow stage 1 invalidation with unmanaged ASIDs
iommu/smmuv3: Implement cache_invalidate
iommu/smmuv3: report additional recoverable faults
iommu/smmuv3: Disallow nested mode in presence of HW MSI regions

drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 383 ++++++++++++++++++--
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 14 +-
drivers/iommu/arm/arm-smmu/arm-smmu.c | 8 +
drivers/iommu/intel/iommu.c | 13 +
drivers/iommu/iommu.c | 79 ++++
include/linux/iommu.h | 35 ++
include/uapi/linux/iommu.h | 54 +++
7 files changed, 548 insertions(+), 38 deletions(-)

--
2.26.3


2021-10-27 22:07:53

by Eric Auger

[permalink] [raw]
Subject: [RFC v16 9/9] iommu/smmuv3: Disallow nested mode in presence of HW MSI regions

Nested mode currently is not compatible with HW MSI reserved regions.
Indeed MSI transactions targeting those MSI doorbells bypass the SMMU.
This would require the guest to also bypass those ranges but the guest
has no information about them.

Let's check nested mode is not attempted in such configuration.

Signed-off-by: Eric Auger <[email protected]>
---
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 23 +++++++++++++++++++++
1 file changed, 23 insertions(+)

diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index ddfc069c10ae..12e7d7920f27 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2488,6 +2488,23 @@ static void arm_smmu_detach_dev(struct arm_smmu_master *master)
arm_smmu_install_ste_for_dev(master);
}

+static bool arm_smmu_has_hw_msi_resv_region(struct device *dev)
+{
+ struct iommu_resv_region *region;
+ bool has_msi_resv_region = false;
+ LIST_HEAD(resv_regions);
+
+ iommu_get_resv_regions(dev, &resv_regions);
+ list_for_each_entry(region, &resv_regions, list) {
+ if (region->type == IOMMU_RESV_MSI) {
+ has_msi_resv_region = true;
+ break;
+ }
+ }
+ iommu_put_resv_regions(dev, &resv_regions);
+ return has_msi_resv_region;
+}
+
static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
{
int ret = 0;
@@ -2545,6 +2562,12 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev)
ret = -EINVAL;
goto out_unlock;
}
+ /* Nested mode is not compatible with MSI HW reserved regions */
+ if (smmu_domain->stage == ARM_SMMU_DOMAIN_NESTED &&
+ arm_smmu_has_hw_msi_resv_region(dev)) {
+ ret = -EINVAL;
+ goto out_unlock;
+ }

master->domain = smmu_domain;

--
2.26.3

2021-12-03 12:27:51

by zhangfei

[permalink] [raw]
Subject: Re: [RFC v16 0/9] SMMUv3 Nested Stage Setup (IOMMU part)


Hi, Eric

On 2021/10/27 下午6:44, Eric Auger wrote:
> This series brings the IOMMU part of HW nested paging support
> in the SMMUv3.
>
> The SMMUv3 driver is adapted to support 2 nested stages.
>
> The IOMMU API is extended to convey the guest stage 1
> configuration and the hook is implemented in the SMMUv3 driver.
>
> This allows the guest to own the stage 1 tables and context
> descriptors (so-called PASID table) while the host owns the
> stage 2 tables and main configuration structures (STE).
>
> This work mainly is provided for test purpose as the upper
> layer integration is under rework and bound to be based on
> /dev/iommu instead of VFIO tunneling. In this version we also get
> rid of the MSI BINDING ioctl, assuming the guest enforces
> flat mapping of host IOVAs used to bind physical MSI doorbells.
> In the current QEMU integration this is achieved by exposing
> RMRs to the guest, using Shameer's series [1]. This approach
> is RFC as the IORT spec is not really meant to do that
> (single mapping flag limitation).
>
> Best Regards
>
> Eric
>
> This series (Host) can be found at:
> https://github.com/eauger/linux/tree/v5.15-rc7-nested-v16
> This includes a rebased VFIO integration (although not meant
> to be upstreamed)
>
> Guest kernel branch can be found at:
> https://github.com/eauger/linux/tree/shameer_rmrr_v7
> featuring [1]
>
> QEMU integration (still based on VFIO and exposing RMRs)
> can be found at:
> https://github.com/eauger/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10
> (use iommu=nested-smmuv3 ARM virt option)
>
> Guest dependency:
> [1] [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node

Thanks a lot for upgrading these patches.

I have basically verified these patches on HiSilicon Kunpeng920.
And integrated them to these branches.
https://github.com/Linaro/linux-kernel-uadk/tree/uacce-devel-5.16
https://github.com/Linaro/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10

Though they are provided for test purpose,

Tested-by: Zhangfei Gao <[email protected]>

Thanks

2021-12-03 13:14:20

by Sumit Gupta

[permalink] [raw]
Subject: Re: [RFC v16 0/9] SMMUv3 Nested Stage Setup (IOMMU part)

Hi Eric,

> This series brings the IOMMU part of HW nested paging support
> in the SMMUv3.
>
> The SMMUv3 driver is adapted to support 2 nested stages.
>
> The IOMMU API is extended to convey the guest stage 1
> configuration and the hook is implemented in the SMMUv3 driver.
>
> This allows the guest to own the stage 1 tables and context
> descriptors (so-called PASID table) while the host owns the
> stage 2 tables and main configuration structures (STE).
>
> This work mainly is provided for test purpose as the upper
> layer integration is under rework and bound to be based on
> /dev/iommu instead of VFIO tunneling. In this version we also get
> rid of the MSI BINDING ioctl, assuming the guest enforces
> flat mapping of host IOVAs used to bind physical MSI doorbells.
> In the current QEMU integration this is achieved by exposing
> RMRs to the guest, using Shameer's series [1]. This approach
> is RFC as the IORT spec is not really meant to do that
> (single mapping flag limitation).
>
> Best Regards
>
> Eric
>
> This series (Host) can be found at:
> https://github.com/eauger/linux/tree/v5.15-rc7-nested-v16
> This includes a rebased VFIO integration (although not meant
> to be upstreamed)
>
> Guest kernel branch can be found at:
> https://github.com/eauger/linux/tree/shameer_rmrr_v7
> featuring [1]
>
> QEMU integration (still based on VFIO and exposing RMRs)
> can be found at:
> https://github.com/eauger/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10
> (use iommu=nested-smmuv3 ARM virt option)
>
> Guest dependency:
> [1] [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node
>
> History:
>
> v15 -> v16:
> - guest RIL must support RIL
> - additional checks in the cache invalidation hook
> - removal of the MSI BINDING ioctl (tentative replacement
> by RMRs)
>
>
> Eric Auger (9):
> iommu: Introduce attach/detach_pasid_table API
> iommu: Introduce iommu_get_nesting
> iommu/smmuv3: Allow s1 and s2 configs to coexist
> iommu/smmuv3: Get prepared for nested stage support
> iommu/smmuv3: Implement attach/detach_pasid_table
> iommu/smmuv3: Allow stage 1 invalidation with unmanaged ASIDs
> iommu/smmuv3: Implement cache_invalidate
> iommu/smmuv3: report additional recoverable faults
> iommu/smmuv3: Disallow nested mode in presence of HW MSI regions
Hi Eric,

I validated the reworked test patches in v16 from the given
branches with Kernel v5.15 & Qemu v6.2. Verified them with
NVMe PCI device assigned to Guest VM.
Sorry, forgot to update earlier.

Tested-by: Sumit Gupta <[email protected]>

Thanks,
Sumit Gupta


2021-12-07 10:27:50

by Eric Auger

[permalink] [raw]
Subject: Re: [RFC v16 0/9] SMMUv3 Nested Stage Setup (IOMMU part)

Hi Zhangfei,

On 12/3/21 1:27 PM, Zhangfei Gao wrote:
>
> Hi, Eric
>
> On 2021/10/27 下午6:44, Eric Auger wrote:
>> This series brings the IOMMU part of HW nested paging support
>> in the SMMUv3.
>>
>> The SMMUv3 driver is adapted to support 2 nested stages.
>>
>> The IOMMU API is extended to convey the guest stage 1
>> configuration and the hook is implemented in the SMMUv3 driver.
>>
>> This allows the guest to own the stage 1 tables and context
>> descriptors (so-called PASID table) while the host owns the
>> stage 2 tables and main configuration structures (STE).
>>
>> This work mainly is provided for test purpose as the upper
>> layer integration is under rework and bound to be based on
>> /dev/iommu instead of VFIO tunneling. In this version we also get
>> rid of the MSI BINDING ioctl, assuming the guest enforces
>> flat mapping of host IOVAs used to bind physical MSI doorbells.
>> In the current QEMU integration this is achieved by exposing
>> RMRs to the guest, using Shameer's series [1]. This approach
>> is RFC as the IORT spec is not really meant to do that
>> (single mapping flag limitation).
>>
>> Best Regards
>>
>> Eric
>>
>> This series (Host) can be found at:
>> https://github.com/eauger/linux/tree/v5.15-rc7-nested-v16
>> This includes a rebased VFIO integration (although not meant
>> to be upstreamed)
>>
>> Guest kernel branch can be found at:
>> https://github.com/eauger/linux/tree/shameer_rmrr_v7
>> featuring [1]
>>
>> QEMU integration (still based on VFIO and exposing RMRs)
>> can be found at:
>> https://github.com/eauger/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10
>> (use iommu=nested-smmuv3 ARM virt option)
>>
>> Guest dependency:
>> [1] [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node
>
> Thanks a lot for upgrading these patches.
>
> I have basically verified these patches on HiSilicon Kunpeng920.
> And integrated them to these branches.
> https://github.com/Linaro/linux-kernel-uadk/tree/uacce-devel-5.16
> https://github.com/Linaro/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10
>
> Though they are provided for test purpose,
>
> Tested-by: Zhangfei Gao <[email protected]>

Thank you very much. As you mentioned, until we do not have the
/dev/iommu integration this is maintained for testing purpose. The SMMU
changes shouldn't be much impacted though.
The added value of this respin was to propose an MSI binding solution
based on RMRRs which simplify things at kernel level.

Thanks!

Eric
>
> Thanks
>


2021-12-07 10:28:40

by Eric Auger

[permalink] [raw]
Subject: Re: [RFC v16 0/9] SMMUv3 Nested Stage Setup (IOMMU part)

Hi Sumit,

On 12/3/21 2:13 PM, Sumit Gupta wrote:
> Hi Eric,
>
>> This series brings the IOMMU part of HW nested paging support
>> in the SMMUv3.
>>
>> The SMMUv3 driver is adapted to support 2 nested stages.
>>
>> The IOMMU API is extended to convey the guest stage 1
>> configuration and the hook is implemented in the SMMUv3 driver.
>>
>> This allows the guest to own the stage 1 tables and context
>> descriptors (so-called PASID table) while the host owns the
>> stage 2 tables and main configuration structures (STE).
>>
>> This work mainly is provided for test purpose as the upper
>> layer integration is under rework and bound to be based on
>> /dev/iommu instead of VFIO tunneling. In this version we also get
>> rid of the MSI BINDING ioctl, assuming the guest enforces
>> flat mapping of host IOVAs used to bind physical MSI doorbells.
>> In the current QEMU integration this is achieved by exposing
>> RMRs to the guest, using Shameer's series [1]. This approach
>> is RFC as the IORT spec is not really meant to do that
>> (single mapping flag limitation).
>>
>> Best Regards
>>
>> Eric
>>
>> This series (Host) can be found at:
>> https://github.com/eauger/linux/tree/v5.15-rc7-nested-v16
>> This includes a rebased VFIO integration (although not meant
>> to be upstreamed)
>>
>> Guest kernel branch can be found at:
>> https://github.com/eauger/linux/tree/shameer_rmrr_v7
>> featuring [1]
>>
>> QEMU integration (still based on VFIO and exposing RMRs)
>> can be found at:
>> https://github.com/eauger/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10
>> (use iommu=nested-smmuv3 ARM virt option)
>>
>> Guest dependency:
>> [1] [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node
>>
>> History:
>>
>> v15 -> v16:
>> - guest RIL must support RIL
>> - additional checks in the cache invalidation hook
>> - removal of the MSI BINDING ioctl (tentative replacement
>>    by RMRs)
>>
>>
>> Eric Auger (9):
>>    iommu: Introduce attach/detach_pasid_table API
>>    iommu: Introduce iommu_get_nesting
>>    iommu/smmuv3: Allow s1 and s2 configs to coexist
>>    iommu/smmuv3: Get prepared for nested stage support
>>    iommu/smmuv3: Implement attach/detach_pasid_table
>>    iommu/smmuv3: Allow stage 1 invalidation with unmanaged ASIDs
>>    iommu/smmuv3: Implement cache_invalidate
>>    iommu/smmuv3: report additional recoverable faults
>>    iommu/smmuv3: Disallow nested mode in presence of HW MSI regions
> Hi Eric,
>
> I validated the reworked test patches in v16 from the given
> branches with Kernel v5.15 & Qemu v6.2. Verified them with
> NVMe PCI device assigned to Guest VM.
> Sorry, forgot to update earlier.
>
> Tested-by: Sumit Gupta <[email protected]>

Thank you very much!

Best Regards

Eric
>
> Thanks,
> Sumit Gupta
>


2021-12-07 10:35:40

by zhangfei

[permalink] [raw]
Subject: Re: [RFC v16 0/9] SMMUv3 Nested Stage Setup (IOMMU part)



On 2021/12/7 下午6:27, Eric Auger wrote:
> Hi Zhangfei,
>
> On 12/3/21 1:27 PM, Zhangfei Gao wrote:
>> Hi, Eric
>>
>> On 2021/10/27 下午6:44, Eric Auger wrote:
>>> This series brings the IOMMU part of HW nested paging support
>>> in the SMMUv3.
>>>
>>> The SMMUv3 driver is adapted to support 2 nested stages.
>>>
>>> The IOMMU API is extended to convey the guest stage 1
>>> configuration and the hook is implemented in the SMMUv3 driver.
>>>
>>> This allows the guest to own the stage 1 tables and context
>>> descriptors (so-called PASID table) while the host owns the
>>> stage 2 tables and main configuration structures (STE).
>>>
>>> This work mainly is provided for test purpose as the upper
>>> layer integration is under rework and bound to be based on
>>> /dev/iommu instead of VFIO tunneling. In this version we also get
>>> rid of the MSI BINDING ioctl, assuming the guest enforces
>>> flat mapping of host IOVAs used to bind physical MSI doorbells.
>>> In the current QEMU integration this is achieved by exposing
>>> RMRs to the guest, using Shameer's series [1]. This approach
>>> is RFC as the IORT spec is not really meant to do that
>>> (single mapping flag limitation).
>>>
>>> Best Regards
>>>
>>> Eric
>>>
>>> This series (Host) can be found at:
>>> https://github.com/eauger/linux/tree/v5.15-rc7-nested-v16
>>> This includes a rebased VFIO integration (although not meant
>>> to be upstreamed)
>>>
>>> Guest kernel branch can be found at:
>>> https://github.com/eauger/linux/tree/shameer_rmrr_v7
>>> featuring [1]
>>>
>>> QEMU integration (still based on VFIO and exposing RMRs)
>>> can be found at:
>>> https://github.com/eauger/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10
>>> (use iommu=nested-smmuv3 ARM virt option)
>>>
>>> Guest dependency:
>>> [1] [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node
>> Thanks a lot for upgrading these patches.
>>
>> I have basically verified these patches on HiSilicon Kunpeng920.
>> And integrated them to these branches.
>> https://github.com/Linaro/linux-kernel-uadk/tree/uacce-devel-5.16
>> https://github.com/Linaro/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10
>>
>> Though they are provided for test purpose,
>>
>> Tested-by: Zhangfei Gao <[email protected]>
> Thank you very much. As you mentioned, until we do not have the
> /dev/iommu integration this is maintained for testing purpose. The SMMU
> changes shouldn't be much impacted though.
> The added value of this respin was to propose an MSI binding solution
> based on RMRRs which simplify things at kernel level.

Current RMRR solution requires uefi enabled,
and QEMU_EFI.fd  has to be provided to start qemu.

Any plan to support dtb as well, which will be simpler since no need
QEMU_EFI.fd anymore.

Thanks



2021-12-07 11:06:23

by Eric Auger

[permalink] [raw]
Subject: Re: [RFC v16 0/9] SMMUv3 Nested Stage Setup (IOMMU part)

Hi Zhangfei,

On 12/7/21 11:35 AM, Zhangfei Gao wrote:
>
>
> On 2021/12/7 下午6:27, Eric Auger wrote:
>> Hi Zhangfei,
>>
>> On 12/3/21 1:27 PM, Zhangfei Gao wrote:
>>> Hi, Eric
>>>
>>> On 2021/10/27 下午6:44, Eric Auger wrote:
>>>> This series brings the IOMMU part of HW nested paging support
>>>> in the SMMUv3.
>>>>
>>>> The SMMUv3 driver is adapted to support 2 nested stages.
>>>>
>>>> The IOMMU API is extended to convey the guest stage 1
>>>> configuration and the hook is implemented in the SMMUv3 driver.
>>>>
>>>> This allows the guest to own the stage 1 tables and context
>>>> descriptors (so-called PASID table) while the host owns the
>>>> stage 2 tables and main configuration structures (STE).
>>>>
>>>> This work mainly is provided for test purpose as the upper
>>>> layer integration is under rework and bound to be based on
>>>> /dev/iommu instead of VFIO tunneling. In this version we also get
>>>> rid of the MSI BINDING ioctl, assuming the guest enforces
>>>> flat mapping of host IOVAs used to bind physical MSI doorbells.
>>>> In the current QEMU integration this is achieved by exposing
>>>> RMRs to the guest, using Shameer's series [1]. This approach
>>>> is RFC as the IORT spec is not really meant to do that
>>>> (single mapping flag limitation).
>>>>
>>>> Best Regards
>>>>
>>>> Eric
>>>>
>>>> This series (Host) can be found at:
>>>> https://github.com/eauger/linux/tree/v5.15-rc7-nested-v16
>>>> This includes a rebased VFIO integration (although not meant
>>>> to be upstreamed)
>>>>
>>>> Guest kernel branch can be found at:
>>>> https://github.com/eauger/linux/tree/shameer_rmrr_v7
>>>> featuring [1]
>>>>
>>>> QEMU integration (still based on VFIO and exposing RMRs)
>>>> can be found at:
>>>> https://github.com/eauger/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10
>>>> (use iommu=nested-smmuv3 ARM virt option)
>>>>
>>>> Guest dependency:
>>>> [1] [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node
>>> Thanks a lot for upgrading these patches.
>>>
>>> I have basically verified these patches on HiSilicon Kunpeng920.
>>> And integrated them to these branches.
>>> https://github.com/Linaro/linux-kernel-uadk/tree/uacce-devel-5.16
>>> https://github.com/Linaro/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10
>>>
>>> Though they are provided for test purpose,
>>>
>>> Tested-by: Zhangfei Gao <[email protected]>
>> Thank you very much. As you mentioned, until we do not have the
>> /dev/iommu integration this is maintained for testing purpose. The SMMU
>> changes shouldn't be much impacted though.
>> The added value of this respin was to propose an MSI binding solution
>> based on RMRRs which simplify things at kernel level.
>
> Current RMRR solution requires uefi enabled,
> and QEMU_EFI.fd  has to be provided to start qemu.
>
> Any plan to support dtb as well, which will be simpler since no need
> QEMU_EFI.fd anymore.
Yes the solution is based on ACPI IORT nodes. No clue if some DT
integration is under work. Shameer?

Thanks

Eric
>
> Thanks
>
>


Subject: RE: [RFC v16 0/9] SMMUv3 Nested Stage Setup (IOMMU part)



> -----Original Message-----
> From: Eric Auger [mailto:[email protected]]
> Sent: 07 December 2021 11:06
> To: Zhangfei Gao <[email protected]>; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> zhukeqian <[email protected]>
> Cc: [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> Shameerali Kolothum Thodi <[email protected]>;
> wangxingang <[email protected]>; jiangkunkun
> <[email protected]>; yuzenghui <[email protected]>;
> [email protected]; chenxiang (M) <[email protected]>;
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected]; [email protected]
> Subject: Re: [RFC v16 0/9] SMMUv3 Nested Stage Setup (IOMMU part)
>
> Hi Zhangfei,
>
> On 12/7/21 11:35 AM, Zhangfei Gao wrote:
> >
> >
> > On 2021/12/7 下午6:27, Eric Auger wrote:
> >> Hi Zhangfei,
> >>
> >> On 12/3/21 1:27 PM, Zhangfei Gao wrote:
> >>> Hi, Eric
> >>>
> >>> On 2021/10/27 下午6:44, Eric Auger wrote:
> >>>> This series brings the IOMMU part of HW nested paging support
> >>>> in the SMMUv3.
> >>>>
> >>>> The SMMUv3 driver is adapted to support 2 nested stages.
> >>>>
> >>>> The IOMMU API is extended to convey the guest stage 1
> >>>> configuration and the hook is implemented in the SMMUv3 driver.
> >>>>
> >>>> This allows the guest to own the stage 1 tables and context
> >>>> descriptors (so-called PASID table) while the host owns the
> >>>> stage 2 tables and main configuration structures (STE).
> >>>>
> >>>> This work mainly is provided for test purpose as the upper
> >>>> layer integration is under rework and bound to be based on
> >>>> /dev/iommu instead of VFIO tunneling. In this version we also get
> >>>> rid of the MSI BINDING ioctl, assuming the guest enforces
> >>>> flat mapping of host IOVAs used to bind physical MSI doorbells.
> >>>> In the current QEMU integration this is achieved by exposing
> >>>> RMRs to the guest, using Shameer's series [1]. This approach
> >>>> is RFC as the IORT spec is not really meant to do that
> >>>> (single mapping flag limitation).
> >>>>
> >>>> Best Regards
> >>>>
> >>>> Eric
> >>>>
> >>>> This series (Host) can be found at:
> >>>> https://github.com/eauger/linux/tree/v5.15-rc7-nested-v16
> >>>> This includes a rebased VFIO integration (although not meant
> >>>> to be upstreamed)
> >>>>
> >>>> Guest kernel branch can be found at:
> >>>> https://github.com/eauger/linux/tree/shameer_rmrr_v7
> >>>> featuring [1]
> >>>>
> >>>> QEMU integration (still based on VFIO and exposing RMRs)
> >>>> can be found at:
> >>>>
> https://github.com/eauger/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10
> >>>> (use iommu=nested-smmuv3 ARM virt option)
> >>>>
> >>>> Guest dependency:
> >>>> [1] [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node
> >>> Thanks a lot for upgrading these patches.
> >>>
> >>> I have basically verified these patches on HiSilicon Kunpeng920.
> >>> And integrated them to these branches.
> >>> https://github.com/Linaro/linux-kernel-uadk/tree/uacce-devel-5.16
> >>> https://github.com/Linaro/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10
> >>>
> >>> Though they are provided for test purpose,
> >>>
> >>> Tested-by: Zhangfei Gao <[email protected]>
> >> Thank you very much. As you mentioned, until we do not have the
> >> /dev/iommu integration this is maintained for testing purpose. The SMMU
> >> changes shouldn't be much impacted though.
> >> The added value of this respin was to propose an MSI binding solution
> >> based on RMRRs which simplify things at kernel level.
> >
> > Current RMRR solution requires uefi enabled,
> > and QEMU_EFI.fd  has to be provided to start qemu.
> >
> > Any plan to support dtb as well, which will be simpler since no need
> > QEMU_EFI.fd anymore.
> Yes the solution is based on ACPI IORT nodes. No clue if some DT
> integration is under work. Shameer?

There was some attempt in the past to create identity mappings using DT.
This is the latest I can find on it,
https://lore.kernel.org/linux-iommu/YTelDHx2REIIvV%[email protected]/T/

Thanks,
Shameer