Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37;
Message-ID: <391ab316-79b1-4535-a45b-4c01bfb80de6@amd.com>
Date:   Tue, 12 Dec 2023 00:35:26 +0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v6 0/6] iommufd: Add nesting infrastructure (part 2/2)
Content-Language: en-US
To:     Jason Gunthorpe <jgg@nvidia.com>, Yi Liu <yi.l.liu@intel.com>,
        "Giani, Dhaval" <Dhaval.Giani@amd.com>,
        Vasant Hegde <vasant.hegde@amd.com>
Cc:     joro@8bytes.org, alex.williamson@redhat.com, kevin.tian@intel.com,
        robin.murphy@arm.com, baolu.lu@linux.intel.com, cohuck@redhat.com,
        eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org,
        mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com,
        yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com,
        shameerali.kolothum.thodi@huawei.com, lulu@redhat.com,
        iommu@lists.linux.dev, linux-kernel@vger.kernel.org,
        linux-kselftest@vger.kernel.org, zhenzhong.duan@intel.com,
        joao.m.martins@oracle.com, xin.zeng@intel.com, yan.y.zhao@intel.com
References: <20231117130717.19875-1-yi.l.liu@intel.com>
 <20231209014726.GA2945299@nvidia.com>
From:   "Suthikulpanit, Suravee" <suravee.suthikulpanit@amd.com>
In-Reply-To: <20231209014726.GA2945299@nvidia.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ckZzMXptVFoxRTdKUUE0cHZubENDU25pZzhBOVpSRUh1eW42cU9FZTlrWU1q?=
 =?utf-8?B?cU9pcEl3c2dKWXpkZGMvRG84SGJ0Y0crQTZHOGZ0czJodE5OS1FpTy9ucVla?=
 =?utf-8?B?YlF3N2NwWk1DOGlKNmk4YU5hb3lESGNiKzh4TDkyVHBqejhKelUrT00xYTR1?=
 =?utf-8?B?bytrZ01WbGQycWMzeVp4bVZWZXI1QnZCZG93aDBDNGgzUFcxY0hnbnJyMnRu?=
 =?utf-8?B?M05ZN1RnUzQ1TElYUkVpVmtjenRZSVQ3SjRSS0VZVGVmU01wZlJLcFZwWjE5?=
 =?utf-8?B?Y2YwQnN3YndmaTRDc3pXNEhOVzFNSFozVDRxSUxBMlQ1bjZ1endxY2tFcnNp?=
 =?utf-8?B?T05PODg4WWptS3BjUTZub1R5ZnhCSmYzOWg3cnk3VWVVelhZd3pKdDFjdi9x?=
 =?utf-8?B?YWVvTXZUeEoyeEQzUEZ3Z3RLWmI1MVZhVUpiRXZIckgxOHFZdjhRUEd5UVlz?=
 =?utf-8?B?M0NHY1VZWHR2RDY2ZS9lTGYzUXBrZUh0MzZHQmU5aGxhcWMxR0M1MWlGazBF?=
 =?utf-8?B?U3Q5RWpMQzJ5UzdzR3NmRFBQT1FKa1lRQ29MTGwvMXhTQmprNXhsOTg0T2hC?=
 =?utf-8?B?anNoSHFmOTlpdUYzc0VPdVRDN2F5VHlWUmJGZHFFZEkvdU14OWpMSlFKREVM?=
 =?utf-8?B?ZEoveUJnMDVuUy90b0s4bW1EdWpQTnJPc0JoUUwzbUc3ZnlCdzM4UjlSS21E?=
 =?utf-8?B?aTFqOU0yTlRmYmhPTVF5N3BNQUhvU3RiRW1FWXpWWnYrUEhsTzZMUndrSHEy?=
 =?utf-8?B?MlY1cW9vQTBCcVZnd3JSVmFTRDhGNG5hYW5aNHFRamdxczlPbTh6dXdwZEZi?=
 =?utf-8?B?dytmaFZWNFg2UjFKMUNVNmhjRUwxR1VUc1BIS1dCenB2eXZGU1FJL1ZGTVR1?=
 =?utf-8?B?NUNpTzF5VTYyUWJwSlZhNzFpbkFoSGpYVU9IK2x1ZjdoeFdLUWlPVjBwYnhX?=
 =?utf-8?B?UG5TNXRMNitzVG1JREJNVjQyUDBtVEhpcDlKY2pSKytScFVQdFFqYVBpOXcr?=
 =?utf-8?B?Q2YzSFlyZGVGS2JGZXd1OEl6aWZXRzY2Q3E4dFhhZHVtUUducUt1dWpPZGtP?=
 =?utf-8?B?N3JMVGZweWdUQWdXMy9UTkNmd0F0VzdraittLzAvTlBOSEdvVFpoQ0NpNHZ3?=
 =?utf-8?B?UEFpTy9zUzcvNHRtNHVKNkVFNDQvZTJ6bEM1djZGU284eC9OcW1jN3Z4SHpJ?=
 =?utf-8?B?ZzJYVmFtdGp4V2x4elZkdCsvbjRtNmFwSjB6VWZPMEJPYW5Sam1BWm81bnNB?=
 =?utf-8?B?V05tSE9Ka2I5azhVYW9TZ3I2VFR3RVZ0emIzeVZjVmU2MEVRWC9DUDlIYW9w?=
 =?utf-8?B?Um8xZFZkYVhIQW1FdktwNVkxb2lLU0VDcG15K3VrTmdiN0Nkc280eno3TGp4?=
 =?utf-8?B?Ym5tUDgyZ0F4TndabFJuRXFiVVBnVm1GMHFGNFhNcmRJRjBtWmxXWGpjdlA3?=
 =?utf-8?B?Z1IvbjlPZjM3VXZWVmppMWo0UU5ickEyVk9XcE9jamFFSEF2Zkt6djdHU2Yx?=
 =?utf-8?B?REJXWTRuZDdHRkZqKzhoMUJQRWwrdmcwdEJJRzRSVUxwUnBrNm0vN3B3dzk2?=
 =?utf-8?B?bFZUaUdycXpqWTYxTUkxc0krZGNpWjY3VUVIOTA2UWxuZzNaVFlTRWdVdG5X?=
 =?utf-8?B?b1JBT2ZJN21wQlU2ZEN3K3NtUGtYWW12MEw4cjR0M210Y3NzSElXMlZLcDkx?=
 =?utf-8?B?TVNiNW81N3hZeVNaQnRpNDl2d1ZKbjRiVFJsZGt3NkFaL0ZhR2kwckFMZE1q?=
 =?utf-8?B?dm1lRmVZajlOMjZiS2puWHBnaGl1SThDS0lBajVMMlNiSVh1RitrY013Rm9p?=
 =?utf-8?B?RFJPV2Zuakg5ck4rNVZhS3VHWUxBejd1Q0dPQ3E1SEpvL3E3bVRTVXVNMTRX?=
 =?utf-8?B?Rk1SSHhVUjF6UmVPV1A3emlDMHgvNWYwZUhpcC9ON21UaSt3RHNtWjhjeEdu?=
 =?utf-8?B?bmUxbUVZR3lQa0hXM3RWNFlZOXNlVGNQcVJ6U2J4bXZaU29WWWtxc1VEekd0?=
 =?utf-8?B?RzB6Z2xpTi96dUVvZklQNlJ4bDBEN2VIVnE5VktZeE1OekFHZ2VkN2Q2VHhn?=
 =?utf-8?B?U25XUlNDV2o5ZFROQ21wUnFaZ08rSzZPNDJiUHNQOHdPOG9GYkJVUUg0a2d6?=
 =?utf-8?Q?t9mW4YMODT3vIKvJASdL7YsNP?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 39ebca23-25df-4f8c-f24e-08dbfa6f96e8
X-MS-Exchange-CrossTenant-AuthSource: DM8PR12MB5445.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Dec 2023 17:35:38.4536
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 39tfLCAY6lEHk00XgM6Rw1gkKi9jL+NzzaGvTu6rURTIDhcCNygBfoYoQyXFbINk2pu3V9o0AM2dbijAXgaZ5Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB6438
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 11 Dec 2023 09:36:01 -0800 (PST)


On 12/9/2023 8:47 AM, Jason Gunthorpe wrote:
> On Fri, Nov 17, 2023 at 05:07:11AM -0800, Yi Liu wrote:
> 
>> Take Intel VT-d as an example, the stage-1 translation table is I/O page
>> table. As the below diagram shows, guest I/O page table pointer in GPA
>> (guest physical address) is passed to host and be used to perform the stage-1
>> address translation. Along with it, modifications to present mappings in the
>> guest I/O page table should be followed with an IOTLB invalidation.
> 
> I've been looking at what the three HW's need for invalidation, it is
> a bit messy.. Here is my thinking. Please let me know if I got it right
> 
> What is the starting point of the guest memory walks:
>   Intel: Single Scalable Mode PASID table entry indexed by a RID & PASID
>   AMD: GCR3 table (a table of PASIDs) indexed by RID

GCR3 table is indexed by PASID.
Device Table (DTE) is indexted by DeviceID (RID)

> ...
> Will ATC be forwarded or synthesized:
>   Intel: The (vDomain-ID,PASID) is a unique nesting domain so
>          the hypervisor knows exactly which RIDs this nesting domain is
> 	linked to and can generate an ATC invalidation. Plan is to
> 	supress/discard the ATC invalidations from the VM and generate
> 	them in the hypervisor.
>   AMD: (vDomain-ID,PASID) is ambiguous, it can refer to multiple GCR3
>        tables. We know which maximal set of RIDs it represents, but not
>        the actual set. I expect AMD will forward the ATC invalidation
>        to avoid over invalidation.

Not sure I understand your description here.

For the AMD IOMMU INVALIDE_IOMMU_PAGES (i.e. invalidate the IOMMU TLB), 
the hypervisor needs to map gDomainId->hDomainId and issue the command 
on behalf of the VM along with the PASID and GVA (or GVA range) provided 
by the guest.

For the AMD IOMMU INVALIDE_IOTLB_PAGES (i.e. invalidate the ATC on the 
device), the hypervisor needs to map gDeviceId->hDeviceId and issue the 
command on behalf of the VM along with the PASID and GVA (or GVA range) 
provided by the guest.

>   ARM: ASID is ambiguous. We have no idea which Nesting Domain/CD table
>        the ASID is contained in. ARM must forward the ATC invalidation
>        from the guest.
> 
> What iommufd object should receive the IOTLB invalidation command list:
>   Intel: The Nesting domain. The command list has to be broken up per
>          (vDomain-ID,PASID) and that batch delivered to the single
> 	nesting domain. Kernel ignores vDomain-ID/PASID and just
> 	invalidates whatever the nesting domain is actually attached to
>   AMD: Any Nesting Domain in the vDomain-ID group. The command list has
>        to be broken up per (vDomain-ID). Kernel replaces
>        vDomain-ID with pDomain-ID from the nesting domain and executes
>        the invalidation.
>   ARM: The Nesting Parent domain. Kernel forces the VMID from the
>        Nesting Parent and executes the invalidation.
> 
> In all cases the VM issues an ATC invalidation with (vRID, PASID) as
> the tag. The VMM must translate vRID -> dev_id -> pRID
> 
> For a pure SW flow the vRID can be mapped to the dev_id and the ATC
> invalidation delivered to the device object (eg IOMMUFD_DEV_INVALIDATE)
> 
> Finally, we have the HW driven invalidation DMA queues that can be
> directly assigned to the guest. AMD and SMMUv3+vCMDQ support this. In
> this case the HW is directly processing invalidation commands without
> a hypervisor trap.
> 
> To make this work the iommu needs to be programmed with:
>   AMD: A vDomain-ID -> pDomain-ID table
>        A vRID -> pRID table
>        This is all bound to some "virtual function"

By "virtual function", I assume you are referring to the AMD vIOMMU 
instance in the guest?

>   ARM: A vRID -> pRID table
>        The vCMDQ is bound to a VM_ID, so to the Nesting Parent
> 
> For AMD, as above, I suggest the vDomain-ID be passed when creating
> the nesting domain
Sure, we can do this part.

> The AMD "virtual function".. It is probably best to create a new iommufd
> object for this and it can be passed in to a few places

Something like IOMMUFD_OBJ_VIOMMU? Then operation would include 
something like:
   * Init
   * Destroy
   * ...

> The vRID->pRID table should be some mostly common
> IOMMUFD_DEV_ASSIGN_VIRTUAL_ID. AMD will need to pass in the virtual
> function ID and ARM will need to pass in the Nesting Parent ID.

Ok.

> ...
> Thus next steps:
>   - Respin this and lets focus on Intel only (this will be tough for
>     the holidays, but if it is available I will try)
>   - Get an ARM patch that just does IOTLB invalidation and add it to my
>     part 3
>   - Start working on IOMMUFD_DEV_INVALIDATE along with an ARM
>     implementation of it
>   - Reorganize the AMD RFC broadly along these lines and lets see it
>     freshened up in the next months as well. I would like to see the
>     AMD support structured to implement the SW paths in first steps and
>     later add in the "virtual function" acceleration stuff. The latter
>     is going to be complex.

Working on refining the part 1 to add HW info reporting and nested 
translation (minus the invalidation stuff). Should be sending out soon.

Suravee