Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.112.35 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.112.35; helo=mail.nvidia.com;
Subject: Re: [RFC v16 0/9] SMMUv3 Nested Stage Setup (IOMMU part)
To:     Eric Auger <eric.auger@redhat.com>, <eric.auger.pro@gmail.com>,
        <iommu@lists.linux-foundation.org>, <linux-kernel@vger.kernel.org>,
        <kvm@vger.kernel.org>, <kvmarm@lists.cs.columbia.edu>,
        <joro@8bytes.org>, <will@kernel.org>, <robin.murphy@arm.com>,
        <jean-philippe@linaro.org>, <zhukeqian1@huawei.com>
CC:     <alex.williamson@redhat.com>, <jacob.jun.pan@linux.intel.com>,
        <yi.l.liu@intel.com>, <kevin.tian@intel.com>,
        <ashok.raj@intel.com>, <maz@kernel.org>,
        <peter.maydell@linaro.org>, <vivek.gautam@arm.com>,
        <shameerali.kolothum.thodi@huawei.com>, <wangxingang5@huawei.com>,
        <jiangkunkun@huawei.com>, <yuzenghui@huawei.com>,
        <nicoleotsuka@gmail.com>, <chenxiang66@hisilicon.com>,
        <nicolinc@nvidia.com>, <vdumpa@nvidia.com>,
        <zhangfei.gao@linaro.org>, <zhangfei.gao@gmail.com>,
        <lushenming@huawei.com>, <vsethi@nvidia.com>,
        Sachin Nikam <Snikam@nvidia.com>,
        Sumit Gupta <sumitg@nvidia.com>,
        Pritesh Raithatha <praithatha@nvidia.com>
References: <20211027104428.1059740-1-eric.auger@redhat.com>
From:   Sumit Gupta <sumitg@nvidia.com>
Message-ID: <4921cd06-065d-951d-d396-ee9843882c40@nvidia.com>
Date:   Fri, 3 Dec 2021 18:43:40 +0530
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.14.0
MIME-Version: 1.0
In-Reply-To: <20211027104428.1059740-1-eric.auger@redhat.com>
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 7bit
Content-Language: en-US
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Dec 2021 13:14:08.9820
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 251edf06-843c-40b0-634a-08d9b65eca94
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.112.35];Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT012.eop-nam11.prod.protection.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB5423
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Eric,

> This series brings the IOMMU part of HW nested paging support
> in the SMMUv3.
>
> The SMMUv3 driver is adapted to support 2 nested stages.
>
> The IOMMU API is extended to convey the guest stage 1
> configuration and the hook is implemented in the SMMUv3 driver.
>
> This allows the guest to own the stage 1 tables and context
> descriptors (so-called PASID table) while the host owns the
> stage 2 tables and main configuration structures (STE).
>
> This work mainly is provided for test purpose as the upper
> layer integration is under rework and bound to be based on
> /dev/iommu instead of VFIO tunneling. In this version we also get
> rid of the MSI BINDING ioctl, assuming the guest enforces
> flat mapping of host IOVAs used to bind physical MSI doorbells.
> In the current QEMU integration this is achieved by exposing
> RMRs to the guest, using Shameer's series [1]. This approach
> is RFC as the IORT spec is not really meant to do that
> (single mapping flag limitation).
>
> Best Regards
>
> Eric
>
> This series (Host) can be found at:
> https://github.com/eauger/linux/tree/v5.15-rc7-nested-v16
> This includes a rebased VFIO integration (although not meant
> to be upstreamed)
>
> Guest kernel branch can be found at:
> https://github.com/eauger/linux/tree/shameer_rmrr_v7
> featuring [1]
>
> QEMU integration (still based on VFIO and exposing RMRs)
> can be found at:
> https://github.com/eauger/qemu/tree/v6.1.0-rmr-v2-nested_smmuv3_v10
> (use iommu=nested-smmuv3 ARM virt option)
>
> Guest dependency:
> [1] [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node
>
> History:
>
> v15 -> v16:
> - guest RIL must support RIL
> - additional checks in the cache invalidation hook
> - removal of the MSI BINDING ioctl (tentative replacement
>    by RMRs)
>
>
> Eric Auger (9):
>    iommu: Introduce attach/detach_pasid_table API
>    iommu: Introduce iommu_get_nesting
>    iommu/smmuv3: Allow s1 and s2 configs to coexist
>    iommu/smmuv3: Get prepared for nested stage support
>    iommu/smmuv3: Implement attach/detach_pasid_table
>    iommu/smmuv3: Allow stage 1 invalidation with unmanaged ASIDs
>    iommu/smmuv3: Implement cache_invalidate
>    iommu/smmuv3: report additional recoverable faults
>    iommu/smmuv3: Disallow nested mode in presence of HW MSI regions
Hi Eric,

I validated the reworked test patches in v16 from the given
branches with Kernel v5.15 & Qemu v6.2. Verified them with
NVMe PCI device assigned to Guest VM.
Sorry, forgot to update earlier.

Tested-by: Sumit Gupta <sumitg@nvidia.com>

Thanks,
Sumit Gupta