Received-SPF: pass (google.com: domain of linux-kernel+bounces-188393-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
From: "Tian, Kevin" <kevin.tian@intel.com>
To: Nicolin Chen <nicolinc@nvidia.com>
CC: Jason Gunthorpe <jgg@nvidia.com>, "will@kernel.org" <will@kernel.org>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>,
	"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
	"joro@8bytes.org" <joro@8bytes.org>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "iommu@lists.linux.dev"
	<iommu@lists.linux.dev>, "linux-arm-kernel@lists.infradead.org"
	<linux-arm-kernel@lists.infradead.org>, "linux-tegra@vger.kernel.org"
	<linux-tegra@vger.kernel.org>, "Liu, Yi L" <yi.l.liu@intel.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>, "vasant.hegde@amd.com"
	<vasant.hegde@amd.com>, "jon.grimm@amd.com" <jon.grimm@amd.com>,
	"santosh.shukla@amd.com" <santosh.shukla@amd.com>, "Dhaval.Giani@amd.com"
	<Dhaval.Giani@amd.com>, "shameerali.kolothum.thodi@huawei.com"
	<shameerali.kolothum.thodi@huawei.com>
Subject: RE: [PATCH RFCv1 07/14] iommufd: Add viommu set/unset_dev_id ops
Thread-Topic: [PATCH RFCv1 07/14] iommufd: Add viommu set/unset_dev_id ops
Thread-Index: AQHajVVq/X0HjqCDvU+a/LkdwhCQ/LGT28sAgADojoCAAk6vgIAAqT2AgAqBYwCAAlS+oIAAlxIAgAC8r8CAABODgIAAIBrQgAAKLoCAABVEkA==
Date: Fri, 24 May 2024 07:21:59 +0000
Message-ID: <BN9PR11MB52765A6F6044EEF52F74230C8CF52@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <ZkDWXnPW7CaX5TtA@nvidia.com> <ZkGZc5dvLigXcWib@nvidia.com>
 <ZkOI8ztR1mUMJ8oe@nvidia.com> <ZkQW6/OAQ8MzN6Go@nvidia.com>
 <20240521182448.GN20229@nvidia.com>
 <BN9PR11MB5276EEC2A1DDA9C92F96737C8CF42@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240523150120.GO20229@nvidia.com>
 <BN9PR11MB5276A896D215E1FA61DECDC38CF52@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZlAI5d6ktg7yncLE@nvidia.com>
 <BN9PR11MB5276422CE4B77D262CD292208CF52@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZlAsXatxsopbz738@nvidia.com>
In-Reply-To: <ZlAsXatxsopbz738@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?yo56DjWewzU/bKAj9vieKzp7UVsNETRXXg4yHjzP9r3IrwNaoZEXYDiYLUkd?=
 =?us-ascii?Q?H3+JH02pmKtyliaCGOOeBnS3o7S8YB6SeFxZHc6dK+J2rhyE+hs7g8NoEJ1e?=
 =?us-ascii?Q?cFqcwV7DZ8znFPDvreXzgtCUgU7L9IqRaRRtDhvN++YpMx4p2ElU358XJO4U?=
 =?us-ascii?Q?PTnzDkn+FzSKKBdHiY6hGRBGhngRJLEEs6Kbr7N/cNEef6dMEH/bmQSVhsWx?=
 =?us-ascii?Q?nYNYWP6KMr7eYUKxWFC3Cz2/sRKMcuACuk8QRlUesRFiiYbN/MmSUJXRU9YP?=
 =?us-ascii?Q?L4GvhgMBThojTRC4bxL08YhV/iHhKpE6mKnNDu8MTC41VlMFSIGnAWZVM/QX?=
 =?us-ascii?Q?ZYMYjkQ8A9n8B1kR2XS+aH/Cs4y+HJko+YKyRlwfO2oong/e+TUCVCuJQoGr?=
 =?us-ascii?Q?g5yrZEN3NfJhdf9Q+915NeGpS4OTJmE9fywQuH3O7cxWNoGnL0hd953cYVkN?=
 =?us-ascii?Q?eHO1cMN4fHI/1JXAHde8CeYzSTXtBNDjzoDoa8KJBvxqHpse6V5VA8OBER2U?=
 =?us-ascii?Q?GOqczEuDIyg9v7ShRxjZ+zLUb6NAoRsKD1eIe9S7wMem3E/yYeFXOSsHaVuF?=
 =?us-ascii?Q?dlaWUxYeGjjchfD5sqyZZ04DZAgDP9Y/Za/+vo8BV3wEU2RJg8AhRZXf5QuY?=
 =?us-ascii?Q?Kx4BxkwCoByzMKeNpXZQNMJ3Hj2H/h6HSMoa4hXBoUixJtFNIS9Ros9ijgGW?=
 =?us-ascii?Q?ZdbrwQkn9ZJhTItgj6SIpMkiV7u68iPkFhizMXABEKTqcNqGCWmaUvqWZpMn?=
 =?us-ascii?Q?oTc+uxu1nF5iJxBmshkOUCISVLWhHKW91ojEYsvYUwPAH0YcyrZW2cRgz7/G?=
 =?us-ascii?Q?VAsD+t0G/k2XXLfPMbK1N3KKpDJVNMrLXSJl3WUZxsJZiS2simrhN9Ubfit9?=
 =?us-ascii?Q?0I8vheqNwA8vmdc5bsQNgpaNvVrmBxexEdxgs8i7YHDla0i+Vf8CZ49pRbci?=
 =?us-ascii?Q?3jqTR1rqdxApqvD/VAfrkwaiuu4zXht4ocMvRe7YL2j7wLfYoNzYofLeUlwx?=
 =?us-ascii?Q?MHFM8Mat1qHfY5+fet8iCcmsHLvxC/UhZ1e8NsKnQwnpao3Cc+m3tujPPzkz?=
 =?us-ascii?Q?mIZqCluWBJFFaKqsVs9WojI2xkwe442ot6/Kf31C3At+Pk2h/l+ZUlzNz2MS?=
 =?us-ascii?Q?wxYbeUqwgM0npFZizF5s0J/MZP//dIjHRfVkDnhgfGKwy07mxF7IJCGZ0ML2?=
 =?us-ascii?Q?LQ2dnD9xP0/cXQqiTEBYYwYXO8IWh9KKW7N5igceStNWk08ufKd4Z+qaY6Ce?=
 =?us-ascii?Q?9Ldds3LDbqmTXEDHZ+R9K2NLBvwoZEUCBjNT3AMm1s/kD+qVX2ZPYYUJTB0U?=
 =?us-ascii?Q?Rvyb7YK5eqxVAfHfIgdRWYopbeVU+N0WCpXK28K22I8AdTOzB2FLQj5VUm40?=
 =?us-ascii?Q?B4WSZHFWZGIhRAsN9a+a+WU2OF+x2ezOKuaM7oBKry/RctFW+MeFiUgWybqD?=
 =?us-ascii?Q?SqA799M+wosDenGnc9ao1AydUwewttBN2CTcFir3xjXNpR5KLDGyT2/RVDEy?=
 =?us-ascii?Q?9DSt9cFrwyWSRwtMjC8ZPXv8po6JBdJX1TjWU3wNypuuB5wBpQQH2s3E0oWo?=
 =?us-ascii?Q?oP/Tq74qEjoUXc2pEfaMKT290E46lhqHCfa/2x/+?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6aef91ee-3add-4f99-946e-08dc7bc23357
X-MS-Exchange-CrossTenant-originalarrivaltime: 24 May 2024 07:21:59.4404
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: 5S602G2wqxHSuRgwY3mGcZ1F5vFgOiP49ZnNE8C1O3fqHRxXD++4c6Q3sVP1kR21rViYPfS8bkPzArpUJPPECg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN6PR11MB8244
X-OriginatorOrg: intel.com

> From: Nicolin Chen <nicolinc@nvidia.com>
> Sent: Friday, May 24, 2024 1:58 PM
>=20
> On Fri, May 24, 2024 at 05:24:11AM +0000, Tian, Kevin wrote:
> > > > > > > On Tue, May 14, 2024 at 06:59:07PM -0700, Nicolin Chen wrote:
> > > > > > > > So, you want a proxy S1 domain for a device to attach, in c=
ase
> > > > > > > > of a stage-2 only setup, because an S2 domain will no longe=
r has
> > > > > > > > a VMID, since it's shared among viommus. In the SMMU driver
> case,
> > > > > > > > an arm_smmu_domain won't have an smmu pointer, so a device
> > > can't
> > > > > > > > attach to an S2 domain but always an nested S1 domain, righ=
t?
> > > > > > >
> > > > > > > That seems like a simple solution to the VMID lifetime, but i=
t
> means
> > > > > > > the kernel has to decode more types of vSTE.
>=20
> > > And the narrative at the top was trying to describe the links:
> > >   [ device ] =3D> [ proxy identity S1 ] =3D> [ viommu [ shared S2 ] ]
> > > v.s.
> > >   [ device ] =3D> [ non-shareable S2 ]
> > >
> > > So the first case can take advantage of VIOMMU_INVALIDATE v.s.
> > > the second case requires a DEV_INVALIDATE.
> >
> > and one side-effect in the first case is to save one VMID for
> > non-shareable S2 hence improves iotlb efficiency.
>=20
> Hmm, how is that?
>=20
> VMID is currently stored in an S2 domain, actually. The viommu
> is a VMID holder to potentially decouple VMID from S2 domain,
> because VMID is per SMMU instance while S2 domain is shareable:
>    [ dev0 ] =3D> [ S1 dom0 ] =3D> [ viommu0 (VMID0) [ shared S2 ] ]
>    [ dev1 ] =3D> [ S1 dom1 ] =3D> [ viommu1 (VMID1) [ shared S2 ] ]

My point was based on Jason's example about 3 VMIDs:

   hwpt_alloc(deva, nesting_parent=3Dtrue) =3D shared_s2
   viommu_alloc(deva, shared_s2) =3D viommu1
   viommu_alloc(devb, shared_s2) =3D viommu2
   hwpt_alloc(deva, viommu1, vste) =3D deva_vste
   hwpt_alloc(devb, viommu2, vste) =3D devb_vste
   attach(deva, deva_vste)
   attach(devb, devb_vste)
   attach(devc, shared_s2)

for devc it could be:
   hwpt_alloc(deva, viommu1, vproxy_s1) =3D devc_proxys1
   attach(devc, devc_proxys1)

then devc will reuse VMID of viommu1 and we save one VMID.

Does that not work so we need create another viommu to hold the
proxy identity s1 then still need a 3rd VMID?