Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6;
Message-ID: <509489ce-0169-4021-ad56-a31544752aa4@amd.com>
Date:   Mon, 11 Dec 2023 22:34:09 +0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v6 0/6] iommufd: Add nesting infrastructure (part 2/2)
Content-Language: en-US
To:     Jason Gunthorpe <jgg@nvidia.com>, Yi Liu <yi.l.liu@intel.com>
Cc:     "Tian, Kevin" <kevin.tian@intel.com>,
        "Giani, Dhaval" <Dhaval.Giani@amd.com>,
        Vasant Hegde <vasant.hegde@amd.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "alex.williamson@redhat.com" <alex.williamson@redhat.com>,
        "robin.murphy@arm.com" <robin.murphy@arm.com>,
        "baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
        "cohuck@redhat.com" <cohuck@redhat.com>,
        "eric.auger@redhat.com" <eric.auger@redhat.com>,
        "nicolinc@nvidia.com" <nicolinc@nvidia.com>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
        "chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
        "yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>,
        "peterx@redhat.com" <peterx@redhat.com>,
        "jasowang@redhat.com" <jasowang@redhat.com>,
        "shameerali.kolothum.thodi@huawei.com" 
        <shameerali.kolothum.thodi@huawei.com>,
        "lulu@redhat.com" <lulu@redhat.com>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
        "Duan, Zhenzhong" <zhenzhong.duan@intel.com>,
        "joao.m.martins@oracle.com" <joao.m.martins@oracle.com>,
        "Zeng, Xin" <xin.zeng@intel.com>,
        "Zhao, Yan Y" <yan.y.zhao@intel.com>
References: <20231117130717.19875-1-yi.l.liu@intel.com>
 <20231209014726.GA2945299@nvidia.com>
 <BN9PR11MB527647A4DA1620DE354983898C8FA@BN9PR11MB5276.namprd11.prod.outlook.com>
 <3e77a4a3-3be8-4e04-9435-1f66df93078d@intel.com>
 <20231211130555.GB2944114@nvidia.com>
From:   "Suthikulpanit, Suravee" <suravee.suthikulpanit@amd.com>
In-Reply-To: <20231211130555.GB2944114@nvidia.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UTBLaWxCVS9taW12WGhmZHdkL0ExZHJkN1o4d3dGMzZ2cnRmOUt2dFpHYmJp?=
 =?utf-8?B?NUZkSmRUTlBGMlBVbG1yNytwUmI4Y2p4L09LV0Z1OHhJZWdnUzg5Zm5HbTRO?=
 =?utf-8?B?VWcxcDlSU3FnU1o5WXBTOFBkcTcrY1FvdlliL29LWEdIMm5OK3ptZE5xOEdr?=
 =?utf-8?B?N3FhU0ZCcTNiRjNRZ2VXOVNhSnc2VC9ENFRRYldoVXVjWk9MaEEzb3A0eWdB?=
 =?utf-8?B?TmJKazlWU2ZyNmdpQ2pDOXBQVmVERExnMjRqbThlTDF5OWFobU14NDY4cmZ1?=
 =?utf-8?B?LzhRWXBTeXUvTGlLSEdxOXR2d1VjakU1OFRmcDBhL3JvWFFmRC9YZVhxQzZv?=
 =?utf-8?B?TXdwZy9SWXBpajVIbU1LZEk5NXZGZnp1NGRMQjlPcU9PZy9oTmpBUjFFUWFh?=
 =?utf-8?B?dzUranZXUjRzTk82Z0xPTXZlSUpqeE11UGtYSU82ZzM2eXh2SzFreFFSRTNt?=
 =?utf-8?B?My83MjhoRlZhclRFRC9PWmhiZFl4OFhUUmxacUlSRE1hSi9XdmdaU3hLMUI1?=
 =?utf-8?B?VkJQaFBRdk4yUlp6VEFsRUFvcDE5VUpYR3BuUWwwRmRSWFlmZVNhQlNROVZ2?=
 =?utf-8?B?b2loNmpQRTVUL2NNNkRIQUE2L0xyV2FWR3FqOGZhNUIrbmloOHM2OHBUYnZO?=
 =?utf-8?B?NTRMTU5OWGVqWHk2b09TK3JWSnlDSXUxUEg0dHhBbFNUU2p2RU5kK3BRcHg0?=
 =?utf-8?B?bXd6Y2NOYUg2VEQ1UDJMVVpCN3pNcGJZMUI4OXR6RDVOQkVNL0c4V3RiK3NZ?=
 =?utf-8?B?WHhyL1JSQnJSMEI1amcyWWsybmcxTlVMdEVXTm8rZ2c3TE96KytwdVBaT0Vu?=
 =?utf-8?B?QnhXazZaSEk5VUNPMlFMajI2L3RWNjFaNk43Y2JkeG1DR3kzRmVhZlFWdEl5?=
 =?utf-8?B?dGNnajZpQ3J2eGVmVmpnTWlpa1BmM3RCcHpJNUxJa2RNdFlMajZMaWdDbmlE?=
 =?utf-8?B?RmJ1MkdWaE40SHV0b2FnalZtUW1WWFdMeUNDQjZkRXZBTERBUmFNSlkrVStn?=
 =?utf-8?B?L1BaWVFSdEt0QVN2cVNpR015UWRUMnBWTEZWQm0vRkV1TDlFeGhycDlOVi8v?=
 =?utf-8?B?VWhoVGN5YjVZdWNjNEg0emh5QjRUTEpqU3RxejhpcGltSHNWK0ZVaHNFU0ZN?=
 =?utf-8?B?bEhIeDlKWGl1SVlGbzhyelR1NG5ucVFFcTREeFY3YXA5allWU25mS0VFMzE3?=
 =?utf-8?B?NVhMWDZoVG9BZ2hsaHB1MStZWEx6T1NoVG1kOVBwYkVBOHFCRFNCNXFubnE5?=
 =?utf-8?B?M1lUK0wySmhSdnZzSVVFMXdJSkQwUnIvVTEvdTA0UXg1dnNDc2lONmlrcWVi?=
 =?utf-8?B?N29qZ1JXNkpkR2J6N0R4LzZoQXlMK21kaStneUNqaVAxZWc4aDZIRHc5REk0?=
 =?utf-8?B?aC94Slo3ZEJoUHpKMkN4cGFuS3ExUUkwVGZzR214Ylh3NlRmaWVlbUhCNVZp?=
 =?utf-8?B?cTVJdXF4YVhmTW9XNEJtaXV4OWdaVllMTVJPL3EzRGdhQU5henRGblBJeDNR?=
 =?utf-8?B?SnZTRHZmM2JUcytYQVZBcHFVcllOQUZxMC9nMXhVdkJQQ0czajNQNVlTK1Ey?=
 =?utf-8?B?OTl4NVhhTW5UdDhPMzBGNnhJT2dIRTVENHNCOC9CRjlSRUttalhwNi9BWENm?=
 =?utf-8?B?YjkvcDNSMHdaYjNScFVFa3A0TWNIcTEyUUl4dEw4VkRmVy9PUzZSUnQzTkUy?=
 =?utf-8?B?QjRubkEzNnZPb3ZKZy8wakF3M0toM0E2VDZGdVN3NDcwRzdGd1RleTAxTWc1?=
 =?utf-8?B?bVdHOElLU0FaSkpha1NCNVFpdU10dXJzbkRIZ1c2cWdraUxqUklSOGRpcWln?=
 =?utf-8?B?TWR4WFQ5WGs1RVliNC9xbjRqaGp5TlhiOXFXY2c1YzQrK1F5djk4T1dSY2t1?=
 =?utf-8?B?SzcwQ3g5VWZuVlU2RVN3NE03VHlWMVRtWllVaTdpTTBxRnNsYitvS21HSHVV?=
 =?utf-8?B?VDJuVzVhRFNOZU1ITmZtUkRqd2k1MVFRU21YMURKY0RKRVBER1g3OVhNREYx?=
 =?utf-8?B?blhSdnZBRUt3dEY1ejlYaUU4MS9XaHhMZy9FYVgzMHdtZW9jNFlWbEtZcllR?=
 =?utf-8?B?Qm5KSU5mWmVtMnFqSXFTdEtBT3BzdVBpUmVrUlZOVkxOa0xOYjFqZFNjRDlB?=
 =?utf-8?Q?U4HW5KlvbSrlqn350mYupWAEw?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b45c2f34-cfa2-4c37-e5ac-08dbfa5ea565
X-MS-Exchange-CrossTenant-AuthSource: DM8PR12MB5445.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Dec 2023 15:34:21.5406
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: SGP4n4hHRDt4C33QtzZ6OxrAmwPEbVsNo8Mb2IrS4fu10PPdM9mTMvo04coQo5oieFMoAK/N8CwZv8348Cnacw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB7079
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Mon, 11 Dec 2023 07:34:36 -0800 (PST)


On 12/11/2023 8:05 PM, Jason Gunthorpe wrote:
> On Mon, Dec 11, 2023 at 08:36:46PM +0800, Yi Liu wrote:
>> On 2023/12/11 10:29, Tian, Kevin wrote:
>>>> From: Jason Gunthorpe <jgg@nvidia.com>
>>>> Sent: Saturday, December 9, 2023 9:47 AM
>>>>
>>>> What is in a Nested domain:
>>>>    Intel: A single IO page table refereed to by a PASID entry
>>>>           Each vDomain-ID,PASID allocates a unique nesting domain
>>>>    AMD: A GCR3 table pointer
>>>>         Nesting domains are created for every unique GCR3 pointer.
>>>>         vDomain-ID can possibly refer to multiple Nesting domains :(
>>>>    ARM: A CD table pointer
>>>>         Nesting domains are created for every unique CD table top pointer.
>>>
>>> this AMD/ARM difference is not very clear to me.
>>>
>>> How could a vDomain-ID refer to multiple GCR3 pointers? Wouldn't it
>>> lead to cache tag conflict when a same PASID entry in multiple GCR3 tables
>>> points to different I/O page tables?
>>
>> Perhaps due to only one DomainID in the DTE table indexed by BDF? Actually,
>> the vDomainID will not be used to tag cache, the host DomainId would be
>> used instead. @Jason?
> 
> The DomainID comes from the DTE table which is indexed by the RID, and
> the DTE entry points to the GCR3 table. So the VM certainly can setup
> a DTE table with multiple entires having the same vDomainID but
> pointing to different GCR3's. So the VMM has to do *something* with
> this.
> 
> Most likely this is not a useful thing to do. However what should the
> VMM do when it sees this? Block a random DTE or push the duplication
> down to real HW would be my options. I'd probably try to do the latter
> just on the basis of better emulation.
> 
> Jason

For AMD, the hardware uses host DomainID (hDomainId) and PASID to tag 
the IOMMU TLB.

The VM can setup vDomainID independently from device (RID) and 
hDomainID. The vDomainId->hDomainId mapping would be managed by the host 
IOMMU driver (since this is also needed by the HW when enabling the 
HW-vIOMMU support a.k.a virtual function).

Currently, the AMD IOMMU driver allocates a DomainId per IOMMU group.
One issue with this is when we have nested translation where we could 
end up with multiple devices (RIDs) sharing same PASID and the same 
hDomainID.

For example:

   - Host view
     Device1 (RID 1) w/ hDomainId 1
     Device2 (RID 2) w/ hDomainId 1
   - Guest view
     Pass-through Device1 (vRID 3) w/ vDomainID A + PASID 0
     Pass-through Device2 (vRID 4) w/ vDomainID B + PASID 0

We should be able to workaround this by changing the way we assign 
hDomainId to be per-device for VFIO pass-through devices although 
sharing the same v1 (stage-2) page table. This would look like.

   - Host view
     Device1 (RID 1) w/ hDomainId 1
     Device2 (RID 2) w/ hDomainId 2
   - Guest view
     Pass-through Device1 (vRID 3) w/ vDomainID A + PASID 0
     Pass-through Device2 (vRID 4) w/ vDomainID B + PASID 0

This should avoid the IOMMU TLB conflict. However, the invalidation 
would need to be done for both DomainId 1 and 2 when updating the v1 
(stage-2) page table.

Thanks,
Suravee