Received-SPF: pass (google.com: domain of linux-kernel+bounces-187528-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Thu, 23 May 2024 09:58:56 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: "Tian, Kevin" <kevin.tian@intel.com>
Cc: Nicolin Chen <nicolinc@nvidia.com>, "will@kernel.org" <will@kernel.org>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>,
	"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
	"joro@8bytes.org" <joro@8bytes.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"linux-arm-kernel@lists.infradead.org" <linux-arm-kernel@lists.infradead.org>,
	"linux-tegra@vger.kernel.org" <linux-tegra@vger.kernel.org>,
	"Liu, Yi L" <yi.l.liu@intel.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>,
	"vasant.hegde@amd.com" <vasant.hegde@amd.com>,
	"jon.grimm@amd.com" <jon.grimm@amd.com>,
	"santosh.shukla@amd.com" <santosh.shukla@amd.com>,
	"Dhaval.Giani@amd.com" <Dhaval.Giani@amd.com>,
	"shameerali.kolothum.thodi@huawei.com" <shameerali.kolothum.thodi@huawei.com>
Subject: Re: [PATCH RFCv1 04/14] iommufd: Add struct iommufd_viommu and
 iommufd_viommu_ops
Message-ID: <20240523125856.GL20229@nvidia.com>
References: <cover.1712978212.git.nicolinc@nvidia.com>
 <8610498e3fc00000e78bb9cef6fac9f6a54978a4.1712978212.git.nicolinc@nvidia.com>
 <ZkDMSfd9BXmsFfb+@nvidia.com>
 <ZkGKKoHnykAdGJq6@nvidia.com>
 <ZkOJgvMNFaZZ06OO@nvidia.com>
 <BN9PR11MB5276423A0BFBDA8346E1ED3C8CEB2@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240522133905.GX20229@nvidia.com>
 <BN9PR11MB5276267BB1CC4AA7008AFBFC8CF42@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <BN9PR11MB5276267BB1CC4AA7008AFBFC8CF42@BN9PR11MB5276.namprd11.prod.outlook.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?dllKaWttUy8rKzE2cHQ0TUY1bGUvQnFwdnpzWmdmSjdjQ01UNnpGSWk5Wm1E?=
 =?utf-8?B?MkhHWStKbFR1clNyN1JRV3B2K2Zzbkg2SmFCMkV3QXZzRlBVakdYbVpGM010?=
 =?utf-8?B?ZUVoYUdhYWFvV3FRMys5TW93dk1FRUUxZkwyN0c1Q1V3NHhqMFdqYTZJTWx3?=
 =?utf-8?B?blNLRHFpdldEQlViSGd6eFFSdHV6U3k3UHJxSmYzaW91T3hhRTY4K290SFRI?=
 =?utf-8?B?QU52d3pxVVJRRjBpem9ad3A4UUk2Ujc1RmIvb0RMQk5zbm5uRWl5RUZ3cmdw?=
 =?utf-8?B?UCt6MnZZclNHdGxwUEVJYmFoOE5malUzUE8yZ1Vrb3hIalpISHNmZllVS05X?=
 =?utf-8?B?RFJIcjNCVHBUeU1XWDgyanBKSEUxWkIwSzR1UTI0cHZNV1g5SW1ra1BiK1lw?=
 =?utf-8?B?M0J2dUtqR3JZblYrWFQ2M3dZMEVYejR1WjBtSStRbjhiQWVCeGZjazEwaXdE?=
 =?utf-8?B?UnhNUUZyRlJQd0hJUWFKTFFSamRLTGZOT29YVzYrVGgzWGRucTB4TEwvUG83?=
 =?utf-8?B?ajd0KzRLelJmZ3BkOGpnb2s1b2U0TExkaFRZMXpWcDgyMmhNL0oxcXM4Mk1v?=
 =?utf-8?B?L3VKSVlLWGtkMjFVRjE5bm04dzNWNWF2MHZLT0ovZ3M4OFBxT3hITm5tNTZN?=
 =?utf-8?B?SG40Q1pHdTBTeUZhNDF0THZ2Nkw0TGZYVUxTc05acUtiY0JoNEFTWFd4UkpM?=
 =?utf-8?B?ek1DWmJvNzRkTHUrRWFIc0JxZVlmZG1NeXV1dWVFQnZhdG9IYm1wdDc2c1Q1?=
 =?utf-8?B?dVdYdERxK2krdWVkSmlMMkh6dTZGUE1mQklZblRHQjdsaGVkaWowVHBIVm12?=
 =?utf-8?B?S0p4cnZ4Z1BUeit0YjViVVZWRk81dTdUYUVXM25KVHY2TVVQQlFmRm1zMkZT?=
 =?utf-8?B?REtGb1MyM2V4VWhib1JIOGNiTUdPL2ZCaXcyK0JPbFpDQjNFVWorTVE2R2Zq?=
 =?utf-8?B?cThIRnZXMEtJWnBXQzFLMFJLd080UlNFemNteEd5ZHFTZzk0N25DSE1ucEo3?=
 =?utf-8?B?RDNXaDB2eko2ZjNucDhEMmlVUXkyd2F0dGFMRXArd05CdXgyVXZ2bkV0WWJZ?=
 =?utf-8?B?SDNsTW45bjY4SHJjTDNuM3NoKytENG9OWEVXM3FsbGRsek5LcGNjQjc4czJJ?=
 =?utf-8?B?dEJNbEZkUHJwOUloRmE0S2c4YnlVOVFOWElsVU9WUXlEdW5KcUpGY2Q5TytD?=
 =?utf-8?B?TlJYbU9CbytMWDF1cTBscjFOMnhYbXVtQ3BsY1Q5QzE1d2VkaW9tSE5RSkJo?=
 =?utf-8?B?RDRyUzRtWVlvWXUrOWpOdnRWRTk1dWhPdkxSa2hFZUJuSnJ5RERCRk11UW1Z?=
 =?utf-8?B?eGNCajN1bEoxNXpQSjBobTQ1VGpISG5PUllXOVlhckJLckpYQlpITU5UbWNr?=
 =?utf-8?B?MEx6WkYyL0VtYk9wWW5uM3NIM2Jhc1ZaQTNLUkRqU2kxa2JHR3BaQWZRR2FI?=
 =?utf-8?B?aVRTaGJBMEJ1SGxYWGlPdVBoM2g2MmRGdFhwVENoL29LdHRrdllyZHk3ejl0?=
 =?utf-8?B?Z3M0Y2JCN3RrUjdEZnJiSkdBU2lWdEtENTRHSW15cHdJRlBiYUttbW5ibDZ2?=
 =?utf-8?B?dzlZY3dabElramF3bmx4WU5tNFJLWGxGNXhzYUt6b2pPeTY3Z0ZxNzYyVkhH?=
 =?utf-8?B?Uk83ZEVhMG14Umt3MEJQOURuRmlqMkRCS3BBMVdKc0FkaGpvV2FNMEVkbXFw?=
 =?utf-8?B?NTl0Z1Y5RlpLaDYzeklRcTFEYXdJZTJqbW5BTlY4SGhWM2RhQjUwTklzWGQr?=
 =?utf-8?B?SlRieWREWHdjNTRJUitRaFE4NDIyaXQ4Q1Z2cEcvRE1VNERxUlE3RzVpY1gw?=
 =?utf-8?B?bmpYSVJFa0s1ZTQvUlhiV0dWWjRKdncvZmVqbVB0d1FtdlRBYTE0Y1Zyd3Ey?=
 =?utf-8?B?bURXdnppOSt2SU5kM2tqU1ppYTRHSW1lOTBxeDc3RkMxL0FaQzlWUVZlRDNh?=
 =?utf-8?B?cTE3RUd2MDczckcwMXpoLzNZVnZ3bHMyTE16S29za05wVHkrc1lNZktUNTlz?=
 =?utf-8?B?elZtS2EvcFFUSkxqNlQ5OEVjR3FkbmdMaE9KYjV0KzZNaCthZEZ1bmVVWWRY?=
 =?utf-8?B?WG9CYzdUVjJIOFJ2clJYdEtoSmZoM2hpa2ovZXVKTEFwTkJ5eTFaODlOZ3Nv?=
 =?utf-8?Q?UPuE=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b1f481ec-1017-4c6c-2052-08dc7b281c8f
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 May 2024 12:58:59.5644
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: z4D2+8hfBDVwt3EInoStsUptzb55/sIMp79nIQoS95ZCAtnFTs137HTTwO25O43h
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB5942

On Thu, May 23, 2024 at 01:43:45AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Wednesday, May 22, 2024 9:39 PM
> > 
> > On Wed, May 22, 2024 at 08:58:34AM +0000, Tian, Kevin wrote:
> > > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > > Sent: Tuesday, May 14, 2024 11:56 PM
> > > >
> > > > > > So we need the S2 to exist before the VIOMMU is created, but the
> > > > > > drivers are going to need some more fixing before that will fully
> > > > > > work.
> > >
> > > Can you elaborate on this point? VIOMMU is a dummy container when
> > > it's created and the association to S2 comes relevant only until when
> > > VQUEUE is created inside and linked to a device?
> > 
> > VIOMMU contains:
> >  - A nesting parent
> >  - A KVM
> >  - Any global per-VM data the driver needs
> >    * In ARM case this is VMID, sometimes shared with KVM
> 
> In which case is it not shared with KVM? I had the impression that
> VMID always comes from KVM in this VCMDQ usage. ????

Not quite, only vBTM needs it to be shared with KVM because the CPU
will forward the KVM VMID to the SMMU during invalidations.

Everything else in the nesting space (including vCMDQ) just needs the
VMID to be unique to the VM since it scopes the ASID that is stored in
guest table.s

For non-nesting cases (ie no viommu) the VMID can be unique to the S2.

> > On ARM the S2 is not divorced from the VIOMMU, ARM requires a single
> > VMID, shared with KVM, and localized to a single VM for some of the
> > bypass features (vBTM, vCMDQ). So to attach a S2 you actually have to
> > attach the VIOMMU to pick up the correct VMID.
> > 
> > I imagine something like this:
> >    hwpt_alloc(deva, nesting_parent=true) = shared_s2
> >    viommu_alloc(deva, shared_s2) = viommu1
> >    viommu_alloc(devb, shared_s2) = viommu2
> >    hwpt_alloc(deva, viommu1, vste) = deva_vste
> >    hwpt_alloc(devb, viommu2, vste) = devb_vste
> >    attach(deva, deva_vste)
> >    attach(devb, devb_vste)
> >    attach(devc, shared_s2)
> 
> I wonder whether we want to make viommu as the 1st-class citizen
> for any nested hwpt if it is desirable to enable it even for VT-d which
> lacks of a hw viommu concept at the moment.

I think we may as well code it like that, yes. It is easy to get the
S2 out of the viommu and feed that into the intel driver.

> > The driver will then know it should program three different VMIDs for
> > the same S2 page table, which matches the ARM expectation for
> > VMID. That is to say we'd pass in the viommu as the pt_id for the
> > iommu_hwpt_alloc. The viommu would imply both the S2 page table and
> > any meta information like VMID the driver needs.
> 
> Can you elaborate the aspect about "three different VMIDs"? 

In SMMUv3 the cache is tagged by (VMID,ASID) where ASID is completely
controlled by the guest.

Every time the guest observes a SMMUv3 instance it is allowed to
creates its own private ASID number space for that instance. The guest
could re-use ASID #1 on two instances.

So every SMMUv3 instance plugged into the guest needs to have its own
unique VMID so that the overlapping ASID's are disambiguate. The above
would create a VM where:

 deva -> vSMMUv3 #1
 devb -> vSMMUv3 #2
 devc -> No IOMMU

> tegra241_cmdqv_viommu_alloc()
> 	vintf->vmid = smmu_domain->vmid;

And same here the 'vintf' is processing ASIDs directly from the guest,
it needs to understand the VMID they are scoped under. Every vSMMUv3
instance in the guest must get a VMID, vintf, and vCMDQ all to its
own. The above needs revising in a viommu world.

> > Both AMD and the vCMDQ thing need to translate some PFNs through the
> > S2 and program them elsewhere, this is manually done by SW, and there
> > are three choices I guess:
> >  - Have the VMM do it and provide  void __user * to the driver
> 
> this sounds redundant to what S2 already provides

Yes, but we don't have to track invalidations here..

> >  - Have the driver do it through the S2 directly and track
> >    S2 invalidations
> 
> this makes more sense to me. Just like the driver already needs to track
> S2 invalidations to flush any nested cache related to the affected S2 range.

I'm a bit worried about 'track invalidations' though..
 
> >  - Have the driver open an access on the IOAS and use the access unmap
> 
> it requires adding more iommufd awareness into the iommu driver. I'm
> inclined to do it only at minimal necessity.

Yes, it is certainly annoying because of the modular/builtin problem.
 
> > Hmm, given we currently have no known hardware entanglement between
> > PRI and VIOMMU it does seem OK for PRI to just exist seperate for
> 
> Isn't AMD vPPRLog for directly sending PRI request into the guest?

I think it is, but that would be a vQUEUE on the VIOMMU not adding a
VIOMMU to Lu's patches, which is what I ment.

> > now. If someone needs them linked someday we can add a viommu_id to
> > the create pri queue command.
> 
> I'm more worried about the potential conflict between the vqueue
> object here and the fault queue object in Baolu's series, if we want
> to introduce vIOMMU concept to platforms which lack of the hw
> support.

I assume the vPPRLog will steal all the PRI before it reaches the
kernel, so once this is turned on Lu's path won't see anything.

I don't know if AMD can turn vPPRLog on individually or if it is a
whole package once they switch on VIOMMU..

Jason