Received-SPF: pass (google.com: domain of linux-kernel+bounces-19075-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
From: "Tian, Kevin" <kevin.tian@intel.com>
To: Jason Gunthorpe <jgg@nvidia.com>
CC: "Liu, Yi L" <yi.l.liu@intel.com>, "joro@8bytes.org" <joro@8bytes.org>,
	"alex.williamson@redhat.com" <alex.williamson@redhat.com>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>, "baolu.lu@linux.intel.com"
	<baolu.lu@linux.intel.com>, "cohuck@redhat.com" <cohuck@redhat.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>, "nicolinc@nvidia.com"
	<nicolinc@nvidia.com>, "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
	"chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
	"yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>, "peterx@redhat.com"
	<peterx@redhat.com>, "jasowang@redhat.com" <jasowang@redhat.com>,
	"shameerali.kolothum.thodi@huawei.com"
	<shameerali.kolothum.thodi@huawei.com>, "lulu@redhat.com" <lulu@redhat.com>,
	"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>, "Duan,
 Zhenzhong" <zhenzhong.duan@intel.com>, "joao.m.martins@oracle.com"
	<joao.m.martins@oracle.com>, "Zeng, Xin" <xin.zeng@intel.com>, "Zhao, Yan Y"
	<yan.y.zhao@intel.com>
Subject: RE: [PATCH v7 1/3] iommufd: Add data structure for Intel VT-d stage-1
 cache invalidation
Thread-Topic: [PATCH v7 1/3] iommufd: Add data structure for Intel VT-d
 stage-1 cache invalidation
Thread-Index: AQHaGViOWXGe59nutU2rEx/0zRDvrrCozh6AgCE2IQCAALd70IAADhuggADPEQCAA/GQIA==
Date: Mon, 8 Jan 2024 04:07:12 +0000
Message-ID: <BN9PR11MB52765C91893A28A7D21D324E8C6B2@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20231117131816.24359-1-yi.l.liu@intel.com>
 <20231117131816.24359-2-yi.l.liu@intel.com>
 <c967e716-9112-4d1a-b6f7-9a005e28202d@intel.com>
 <20240104143658.GX50406@nvidia.com>
 <BN9PR11MB52769EEDAE2783426144E2588C662@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240105144516.GC50406@nvidia.com>
In-Reply-To: <20240105144516.GC50406@nvidia.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?PhhxOj7L4ttO1n1gYVkTdxRndAb2YTOKLueHxZDblDmw+hSaxZ8apV5EnMj6?=
 =?us-ascii?Q?R9iX3OhZ832eOtgond1wNPlwW/vjzspOrFII7sgx5/EE3GWbCPEnT3/3TFH0?=
 =?us-ascii?Q?n30HuZAZnal/d0ZzUvpW6OOsQ5mlKX/7FbTscjZ1AFUkogT5+9HnC5DY3UHq?=
 =?us-ascii?Q?ug8AMQxk1UQ29ZyXDefd4YIcy7g/mhctYYX+wyFSRAERdm4oKvYrckqCS/+m?=
 =?us-ascii?Q?d27F3khMuOkQBOW+ce7jSgDAGaX+7IQtIAfL8qxidaUmp5C+YzFaGlBK0RPJ?=
 =?us-ascii?Q?jBScKWPppkJt/AGytPC2sx+e4RLhkCajO9m/gjKd3LrBtrxB6Yn25Q6REyz5?=
 =?us-ascii?Q?XUzNwAN+Krml8KOWFiTeH+HOHfM6MgrIDdDWgt+MbIbMFj5vVQKm3s/ubJRZ?=
 =?us-ascii?Q?VVT4PJfGEymjj5qP+25f6AR69ngMEe9mvr53S+8PjgnxqCCctHwz73ASnhge?=
 =?us-ascii?Q?hjxxv5VZaJOWJDmBPKWC881DxOKAWj2iWW75ojnX46ucAfLXtaXRwNZdjMFx?=
 =?us-ascii?Q?49dD5al5KpMdJikMTpSSbFsyK0vuzHa1KcHiGLuREioRsNy4yRJ6qMWK8P/x?=
 =?us-ascii?Q?U8vvCplCMoYu8PrwQbKzisI6xePhXQvNnXTPEZoNKA6lzwi+wxnQSmKYyQm+?=
 =?us-ascii?Q?PnPKiSYQg9JXlfzSahKK2XRbLIbQR//oZ+igK5OOHgTwFv3IHozQ/9wA9NWT?=
 =?us-ascii?Q?2WXvYzN9irWvbByf/AaNoyFjGGwhH2CBJeOBBrZmsrDOXszjEchdDYiCdb9Z?=
 =?us-ascii?Q?Z1NmW96RmMQ3cGMW5+ouaLLtAfHB3tQduwKrbQgASJkZcQ6UyU8zaudwjAOb?=
 =?us-ascii?Q?C4Vm+A5rRvhTZSPqZ++v2c5z51029BaObvYzWkDzxyGhD85fj0Wpj/za6vxp?=
 =?us-ascii?Q?r3ONc/DZ2lTENYj//37F44qqigP+FmjDrYyUp4nFpLEGzLETrdx5PbrtNcF1?=
 =?us-ascii?Q?fqh4bihtiu6CjOKSVw2WOqjvTu0ikNDpT8XlAPvAUzGj+GMBfgSzOinK/mEW?=
 =?us-ascii?Q?bSO5Ns82iUAuteKsWSFebso4y4qqv3GVSJO+4jSr2MAt4pFi3EKrV4I34u73?=
 =?us-ascii?Q?iQP1PYo+kkUy+fPqhg3dEb3C9CMWTCjdAnP4vBx4Z819LLPKIl+9kp0R93GU?=
 =?us-ascii?Q?v78fGK96ecIEPpsIpUvNC16K9/x7efP6ITsqLn668q+clW+4jkxpaEKwLTmE?=
 =?us-ascii?Q?EW9yP7pO/Wxu8IR0R+sV8h4CaLvDj6aswISgI73DqJlD3AbMNNfYkj31YtKo?=
 =?us-ascii?Q?PrlFQlAIPreBlQXQX7iIN3SCvSCfU+/YPV3o1v+BbR94t9sjXvygosY7vyUB?=
 =?us-ascii?Q?L53eeSq6OWVMqwPtwe/Amtsjc9e1HZDdmWBRn98KiaYPpqBlfPK6x4f39+Ns?=
 =?us-ascii?Q?JbDwGPkBs20bMlpIUPVtJ1/dEVUMBN92R3s2NkkMNTjQBFrO1s7Iwnd/i7vp?=
 =?us-ascii?Q?ojMb1JMGyPCudtG5KQpyQhSHPawteii++0ko8189SrtnZoxY84MQW2iibiEU?=
 =?us-ascii?Q?36zzwWaSNfIRtcCrv6aGpBKCoqXJOh5c0uqSxCwl3OtUJOn52+TZLSjLKVhQ?=
 =?us-ascii?Q?hfIO7Bs2SiW6/WrFePfkvET1MiSZW5mTZqD7UXQ4?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 1ff0e0d1-b160-4428-b434-08dc0fff4ab0
X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Jan 2024 04:07:12.3587
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: GaWzIUXkvHB2qk098pPUkEYw3H7g6RCj5uLe7I5TqdJkXBGgZqlPq+CbRsRl3FQMYFMZ/0GiTYqfqcwgxuqbMg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR11MB6991
X-OriginatorOrg: intel.com

> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Friday, January 5, 2024 10:45 PM
>=20
> On Fri, Jan 05, 2024 at 02:52:50AM +0000, Tian, Kevin wrote:
> > > but in reality the relation could be identified in an easy way due to=
 a SIOV
> > > restriction which we discussed before - shared PASID space of PF
> disallows
> > > assigning sibling vdev's to a same VM (otherwise no way to identify w=
hich
> > > sibling vdev triggering an iopf when a pasid is used on both vdev's).=
 That
> > > restriction implies that within an iommufd context every iommufd_devi=
ce
> > > object should contain a unique struct device pointer. So PASID can be
> > > instead ignored in the lookup then just always do iommufd_get_dev_id(=
)
> > > using struct device.
> >
> > A bit more background.
> >
> > Previously we thought this restriction only applies to SIOV+vSVA, as
> > a guest process may bind to both sibling vdev's, leading to the same
> > pasid situation.
> >
> > In concept w/o vSVA it's still possible to assign sibling vdev's to
> > a same VM as each vdev is allocated with a unique pasid to mark vRID
> > so can be differentiated from each other in the fault/error path.
>=20
> I thought the SIOV plan was that each "vdev" ie vpci function would
> get a slice of the pRID's PASID space statically selected at creation?
>=20
> So SVA/etc doesn't matter, you reliably get a disjoint set of pRID &
> pPASID into each VM.
>=20
> From that view you can't identify the iommufd dev_id without knowing
> both the pRID and pPASID which will disambiguate the different SIOV
> iommufd dev_id instances sharing a rid.

true when assigning those instances to different VMs.

Here I was talking about assigning them to a same VM being a problem.
with rid sharing plus same ENQCMD pPASID potentially used on both
instances there'd be ambiguity in vSVA e.g. iopf to identify dev_id.

we agreed before on preventing sibling vdev's in one VM for above
reason, but only as far as vSVA is concerned.

then given the new finding in err reporting I wondered whether this
restriction should be applied to all SIOV scenarios (but irrelevant now
with below explanation after another thinking)

>=20
> > But when looking at this err code issue with Yi closely, we found
> > there is another gap in the VT-d spec. Upon devtlb invalidation
> > timeout the hw doesn't report pasid in the error info register. this
> > makes it impossible to identify the source vdev if a hwpt invalidation
> > request involves sibling vdev's from a same PF.
>=20
> Don't you know which command timed out?

unfortunately no.

for errors related to descriptor fetch the driver can tell the command
by looking at the head pointer of the invalidation queue.

command completion is indirectly detected by inserting a wait descriptor
as fence. completion timeout error is reported in an error register. but
this register doesn't record pasid, nor does the command location. if there
are multiple pending devtlb invalidation commands upon timeout=20
error the spec suggests the driver to treat all of them timeout as the
register can only record one rid.

this is kind of moot. If the driver submits only one command (plus wait)
at a time it doesn't need hw's help to identify the timeout command.=20
If the driver batches invalidation commands it must treat all timeout if
an timeout error is reported.

from this angle whether to record pasid doesn't really matter.

intel-iommu driver doesn't batch commands. so it's possible for
the driver to figure out the timeout device itself and identify rid plus
pasid to find dev_id from iommufd.

Thanks
Kevin