Reply-To: eric.auger@redhat.com
Subject: Re: [RFC v16 0/9] SMMUv3 Nested Stage Setup (IOMMU part)
To:     Sumit Gupta <sumitg@nvidia.com>, eric.auger.pro@gmail.com,
        iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org,
        kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, joro@8bytes.org,
        will@kernel.org, robin.murphy@arm.com, jean-philippe@linaro.org,
        zhukeqian1@huawei.com
Cc:     alex.williamson@redhat.com, jacob.jun.pan@linux.intel.com,
        yi.l.liu@intel.com, kevin.tian@intel.com, ashok.raj@intel.com,
        maz@kernel.org, peter.maydell@linaro.org, vivek.gautam@arm.com,
        shameerali.kolothum.thodi@huawei.com, wangxingang5@huawei.com,
        jiangkunkun@huawei.com, yuzenghui@huawei.com,
        nicoleotsuka@gmail.com, chenxiang66@hisilicon.com,
        nicolinc@nvidia.com, vdumpa@nvidia.com, zhangfei.gao@linaro.org,
        zhangfei.gao@gmail.com, lushenming@huawei.com, vsethi@nvidia.com,
        Sachin Nikam <Snikam@nvidia.com>,
        Pritesh Raithatha <praithatha@nvidia.com>
References: <20211027104428.1059740-1-eric.auger@redhat.com>
 <4921cd06-065d-951d-d396-ee9843882c40@nvidia.com>
From:   Eric Auger <eric.auger@redhat.com>
Message-ID: <780faf42-e035-4bfd-252d-8d009355bd6c@redhat.com>
Date:   Tue, 7 Dec 2021 11:28:32 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.10.1
MIME-Version: 1.0
In-Reply-To: <4921cd06-065d-951d-d396-ee9843882c40@nvidia.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Content-Language: en-US
Precedence: bulk

Hi Sumit,

On 12/3/21 2:13 PM, Sumit Gupta wrote:
> Hi Eric,
>
>> This series brings the IOMMU part of HW nested paging support
>> in the SMMUv3.
>>
>> The SMMUv3 driver is adapted to support 2 nested stages.
>>
>> The IOMMU API is extended to convey the guest stage 1
>> configuration and the hook is implemented in the SMMUv3 driver.
>>
>> This allows the guest to own the stage 1 tables and context
>> descriptors (so-called PASID table) while the host owns the
>> stage 2 tables and main configuration structures (STE).
>>
>> This work mainly is provided for test purpose as the upper
>> layer integration is under rework and bound to be based on
>> /dev/iommu instead of VFIO tunneling. In this version we also get
>> rid of the MSI BINDING ioctl, assuming the guest enforces
>> flat mapping of host IOVAs used to bind physical MSI doorbells.
>> In the current QEMU integration this is achieved by exposing
>> RMRs to the guest, using Shameer's series [1]. This approach
>> is RFC as the IORT spec is not really meant to do that
>> (single mapping flag limitation).
>>
>> Best Regards
>>
>> Eric
>>
>> This series (Host) can be found at:
>> https://github.com/eauger/linux/tree/v5.15-rc7-nested-v16
>> This includes a rebased VFIO integration (although not meant
>> to be upstreamed)
>>
>> Guest kernel branch can be found at:
>> https://github.com/eauger/linux/tree/shameer_rmrr_v7
>> featuring [1]
>>
>> QEMU integration (still based on VFIO and exposing RMRs)
>> can be found at:
>> https://github.com/eauger/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10
>> (use iommu=nested-smmuv3 ARM virt option)
>>
>> Guest dependency:
>> [1] [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node
>>
>> History:
>>
>> v15 -> v16:
>> - guest RIL must support RIL
>> - additional checks in the cache invalidation hook
>> - removal of the MSI BINDING ioctl (tentative replacement
>>    by RMRs)
>>
>>
>> Eric Auger (9):
>>    iommu: Introduce attach/detach_pasid_table API
>>    iommu: Introduce iommu_get_nesting
>>    iommu/smmuv3: Allow s1 and s2 configs to coexist
>>    iommu/smmuv3: Get prepared for nested stage support
>>    iommu/smmuv3: Implement attach/detach_pasid_table
>>    iommu/smmuv3: Allow stage 1 invalidation with unmanaged ASIDs
>>    iommu/smmuv3: Implement cache_invalidate
>>    iommu/smmuv3: report additional recoverable faults
>>    iommu/smmuv3: Disallow nested mode in presence of HW MSI regions
> Hi Eric,
>
> I validated the reworked test patches in v16 from the given
> branches with Kernel v5.15 & Qemu v6.2. Verified them with
> NVMe PCI device assigned to Guest VM.
> Sorry, forgot to update earlier.
>
> Tested-by: Sumit Gupta <sumitg@nvidia.com>

Thank you very much!

Best Regards

Eric
>
> Thanks,
> Sumit Gupta
>