Received-SPF: pass (google.com: domain of linux-kernel+bounces-17440-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
From: "Tian, Kevin" <kevin.tian@intel.com>
To: Jason Gunthorpe <jgg@nvidia.com>, "Liu, Yi L" <yi.l.liu@intel.com>
CC: "joro@8bytes.org" <joro@8bytes.org>, "alex.williamson@redhat.com"
	<alex.williamson@redhat.com>, "robin.murphy@arm.com" <robin.murphy@arm.com>,
	"baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>, "cohuck@redhat.com"
	<cohuck@redhat.com>, "eric.auger@redhat.com" <eric.auger@redhat.com>,
	"nicolinc@nvidia.com" <nicolinc@nvidia.com>, "kvm@vger.kernel.org"
	<kvm@vger.kernel.org>, "mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
	"chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
	"yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>, "peterx@redhat.com"
	<peterx@redhat.com>, "jasowang@redhat.com" <jasowang@redhat.com>,
	"shameerali.kolothum.thodi@huawei.com"
	<shameerali.kolothum.thodi@huawei.com>, "lulu@redhat.com" <lulu@redhat.com>,
	"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>, "Duan,
 Zhenzhong" <zhenzhong.duan@intel.com>, "joao.m.martins@oracle.com"
	<joao.m.martins@oracle.com>, "Zeng, Xin" <xin.zeng@intel.com>, "Zhao, Yan Y"
	<yan.y.zhao@intel.com>
Subject: RE: [PATCH v7 1/3] iommufd: Add data structure for Intel VT-d stage-1
 cache invalidation
Thread-Topic: [PATCH v7 1/3] iommufd: Add data structure for Intel VT-d
 stage-1 cache invalidation
Thread-Index: AQHaGViOWXGe59nutU2rEx/0zRDvrrCozh6AgCE2IQCAALd70IAADhug
Date: Fri, 5 Jan 2024 02:52:50 +0000
Message-ID: <BN9PR11MB52769EEDAE2783426144E2588C662@BN9PR11MB5276.namprd11.prod.outlook.com>
References: <20231117131816.24359-1-yi.l.liu@intel.com>
 <20231117131816.24359-2-yi.l.liu@intel.com>
 <c967e716-9112-4d1a-b6f7-9a005e28202d@intel.com>
 <20240104143658.GX50406@nvidia.com> 
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?6CHcRSO+UOAmhbBb2GhYhTZkKl6XL9L8xr2yUkFiuzpkhwwDFGEBzQtho7bU?=
 =?us-ascii?Q?MU8Z+TONY9Y/0xZBh9hWkinZSGS6NRXGANxbeSyE7VI44hJz2RTKy6rjHmWR?=
 =?us-ascii?Q?55q1KSYqJUiiul5AM+zKVZc8Ebk7WeWoU/KqrGvUnp68l6/XB6vB3XyNdNUK?=
 =?us-ascii?Q?rUDF36UG7up2fyPDOVKng6pThDpQJcrIjy+GHzhQxnq8KE6Nod0yO6m+cdtS?=
 =?us-ascii?Q?XGA0Oa7OUO+wVkA0aO+BoyXCoF5qZzwsrOOykeJW1PR8QH1w6crm0dvY4ViP?=
 =?us-ascii?Q?wz1WSNCLvpPhE7mY3EYPbJoDtVwI/TOITFKfLoUhY1drxAPEZ5GaTrm++G7j?=
 =?us-ascii?Q?ZcjlD0YKCDXXxezB3KzuorHHpzIjPconbK9fjKogt1id4MYkmK1bayJFWKDt?=
 =?us-ascii?Q?EsZku85JK5WMXqzObj0MlWvE+wlV7/9Nta8pSGCx4Gl3QVzbaP6TM0NANVYf?=
 =?us-ascii?Q?jB8x5uaoj20tc3CXMYvT0M/Wq1+Vy4OAH9o9mXzdQNSC/8qUi1E26U8GuDS4?=
 =?us-ascii?Q?QIC3T0n5geDyOlRKGlgSZgjeh6Xa69p9YYNai0XSkDeBaHHvwbCxu7MAZ9k/?=
 =?us-ascii?Q?uI8IgrKnCrdahWD3ZCewLMsNEkRFePYD4f6MCSsovKB3f8mfUDzwZqNL1XGG?=
 =?us-ascii?Q?H6dfjt5rd42fiXxGZlKIJwe7gcgdpexhUU50ibUR1nqjy021kL1Lf5WPMLh3?=
 =?us-ascii?Q?oM1hGU6G2Tg2AQfxEXSAE/Tjbk8LMnh+gZmgSpBJWyF2Tc4HyCK7vkzUfVVt?=
 =?us-ascii?Q?PMz5oNUv3OO/p2ZtAjWgU1+YzblfPC9n46ev5rJJCB+W3iBYEL3fqZbRyad7?=
 =?us-ascii?Q?2i9MKSiqRz+jEFRsH1Yo1s9aT18URXzqYQPD9l3IKZ5uJHjl7r0J6lPlUmhS?=
 =?us-ascii?Q?KhKaN7VklELGAjFqAijHb3w2sR37Ao1zY2HJrpPasBUQyqgAKgXxYBf3Vwn9?=
 =?us-ascii?Q?ZD8bdNGnasBvBSPdinX2owZzRuoswUQev8GIlewmFjVgY/KhsrtexnkCERoL?=
 =?us-ascii?Q?qH0kKOtDEtH4lZFP4ECH6rhHeQz/g1SnGW73IS4RgqqLUf0NRbqI5lR2fvd1?=
 =?us-ascii?Q?1kRWd756aw9rdJ1vMSkZowITx8ZH8fr+fHhksQJ4629CYGocs/xUjCf+gXXw?=
 =?us-ascii?Q?c8p9AtM70hLKQV8jRd9OPz55pDuaazk6uRBwMpaPAmy5P5kMVcES52qxHSwQ?=
 =?us-ascii?Q?ZFPFU8J1oKCzWcCayQYNJtEjaRGVU0HnueteXdc49avPr/5caCYMy1YwW55k?=
 =?us-ascii?Q?4ivgi7YcxvqxQqmpLdt/LvTk4U9xesO79cpLDmv5LdU/HhDq/1J1dvyZLii1?=
 =?us-ascii?Q?5FM1Fc4XuoLGbzBMg78z960rkjqTr3KnxUmXqlbgXHrIfsMqX1qe3sfatOPn?=
 =?us-ascii?Q?SVyhBQp3X0cOUxondzA3SEbSKIxOSPGlDmPwt0FpJ162Vre9u+4pTYxVE/+B?=
 =?us-ascii?Q?3/5sD3Be7jL6bXiiYKC2twjRZgBoSJ+C7g5XmcdnF4ENYdYCyhmN92l7D/TM?=
 =?us-ascii?Q?MdS+IfT9z4IalqoUjeBADl9LrbLtz9XzMBMByNSmjFp8OmCMNODSN8nMZISf?=
 =?us-ascii?Q?BKw7T/6t8s7WLWohcfzYythR6hoodwiyA87sLoBy?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: linux-kernel@vger.kernel.org
List-Id: <linux-kernel.vger.kernel.org>
List-Subscribe: <mailto:linux-kernel+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:linux-kernel+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN9PR11MB5276.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4ef16fa8-cda4-4712-73cb-08dc0d9967ce
X-MS-Exchange-CrossTenant-originalarrivaltime: 05 Jan 2024 02:52:50.2281
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: RsrsOyKCCksMbcPTcGebkkKohkrUJtXUd1rgZTia6licLSxm6xEqoR9+uhwv/KxIqgRfOonuFXSRzZUfzNgwDA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB5791
X-OriginatorOrg: intel.com

> From: Tian, Kevin
> Sent: Friday, January 5, 2024 10:16 AM
>=20
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Thursday, January 4, 2024 10:37 PM
> >
> > On Thu, Dec 14, 2023 at 07:26:39PM +0800, Yi Liu wrote:
> > > Per the prior discussion[1], we agreed to move the error reporting in=
to
> the
> > > driver specific part. On Intel side, we want to report two devTLB
> > > invalidation errors: ICE (invalid completion error) and ITE (invalida=
tion
> > > timeout error). Such errors have an additional SID information to tel=
l
> > > which device failed the devTLB invalidation. I've got the below struc=
ture.
> >
> > IMHO all of this complexity is a consequence of the decision to hide
> > the devtlb invalidation from the VM..
> >
> > On the other hand I guess you want to do this because of the SIOV
> > troubles where the vPCI function in the VM is entirely virtual and
> > can't be trivially mapped to a real PCI function for ATC invalidation
> > like ARM and AMD can do (but they also can't support SIOV because of
> > this). :(
> >
> > However it also makes it very confusing about how the VM would
> > perceive an error - eg if it invalidates an SIOV device single PASID
> > and that devtlb fails then the error should be connected back to the
> > vPCI function for the SIOV's specific PASID and not back to the
> > physical PCI function for the SIOV owner.
> >
> > As the iommu driver itself has no idea about the vPCI functions this
> > seems like it is going to get really confusing. The API I suggested in
> > the other email is not entirely going to work as the vPCI function for
> > SIOV cases will have to be identified by the (struct device, PASID) -
> > while it would be easy enough for the iommu driver to provide the
> > PASID, I'm not sure how the iommufd core will relate the PASID back to
> > the iommu_device to understand SIOV without actually being aware of
> > SIOV to some degree :\
>=20
> we plan to add such awareness with a new binding helper:
>=20
> https://lore.kernel.org/lkml/20231009085123.463179-4-yi.l.liu@intel.com/
>=20
> and with metadata to track association between PASID's and iommufd vdev.
>=20
> but in reality the relation could be identified in an easy way due to a S=
IOV
> restriction which we discussed before - shared PASID space of PF disallow=
s
> assigning sibling vdev's to a same VM (otherwise no way to identify which
> sibling vdev triggering an iopf when a pasid is used on both vdev's). Tha=
t
> restriction implies that within an iommufd context every iommufd_device
> object should contain a unique struct device pointer. So PASID can be
> instead ignored in the lookup then just always do iommufd_get_dev_id()
> using struct device.

A bit more background.

Previously we thought this restriction only applies to SIOV+vSVA, as
a guest process may bind to both sibling vdev's, leading to the same
pasid situation.

In concept w/o vSVA it's still possible to assign sibling vdev's to
a same VM as each vdev is allocated with a unique pasid to mark vRID
so can be differentiated from each other in the fault/error path.

But when looking at this err code issue with Yi closely, we found
there is another gap in the VT-d spec. Upon devtlb invalidation
timeout the hw doesn't report pasid in the error info register. this
makes it impossible to identify the source vdev if a hwpt invalidation
request involves sibling vdev's from a same PF.

with that I'm inclined to always imposing this restriction for SIOV.=20
One may argue that SIOV w/o vSVA w/o devtlb is conceptually immune
but I'm with you that given SIOVr1 is one-off I prefer to limiting its
usability other than complexing the kernel.