Received-SPF: pass (google.com: domain of linux-kernel+bounces-17961-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Fri, 5 Jan 2024 10:45:16 -0400
From: Jason Gunthorpe <jgg@nvidia.com>
To: "Tian, Kevin" <kevin.tian@intel.com>
Cc: "Liu, Yi L" <yi.l.liu@intel.com>, "joro@8bytes.org" <joro@8bytes.org>,
	"alex.williamson@redhat.com" <alex.williamson@redhat.com>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>,
	"baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
	"cohuck@redhat.com" <cohuck@redhat.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>,
	"nicolinc@nvidia.com" <nicolinc@nvidia.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
	"chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
	"yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>,
	"peterx@redhat.com" <peterx@redhat.com>,
	"jasowang@redhat.com" <jasowang@redhat.com>,
	"shameerali.kolothum.thodi@huawei.com" <shameerali.kolothum.thodi@huawei.com>,
	"lulu@redhat.com" <lulu@redhat.com>,
	"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
	"Duan, Zhenzhong" <zhenzhong.duan@intel.com>,
	"joao.m.martins@oracle.com" <joao.m.martins@oracle.com>,
	"Zeng, Xin" <xin.zeng@intel.com>,
	"Zhao, Yan Y" <yan.y.zhao@intel.com>
Subject: Re: [PATCH v7 1/3] iommufd: Add data structure for Intel VT-d
 stage-1 cache invalidation
Message-ID: <20240105144516.GC50406@nvidia.com>
References: <20231117131816.24359-1-yi.l.liu@intel.com>
 <20231117131816.24359-2-yi.l.liu@intel.com>
 <c967e716-9112-4d1a-b6f7-9a005e28202d@intel.com>
 <20240104143658.GX50406@nvidia.com>
 <BN9PR11MB52769EEDAE2783426144E2588C662@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <BN9PR11MB52769EEDAE2783426144E2588C662@BN9PR11MB5276.namprd11.prod.outlook.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?91VQZZ5LrMUSq80YEKI56YK6LLSB6o4ouDplgG6Xf69IS1hvSXK/GvaK11Ge?=
 =?us-ascii?Q?4A3Y02zN6FOHAd+N4yJgFFbfnc2JyG6v6hxBBd67e6H68/THWGiADy5+aAL3?=
 =?us-ascii?Q?FoS3eexKScxOzwuYJKOdDSRvunIUlvJNVdRyWb2MZIZoyEv+dYS/VJaeyHHL?=
 =?us-ascii?Q?ch8lIqdk8EJwXA/TaSgEPajWgYluSgws6pQCXaOsiFQ5WP1ZKtrOIZh4FAub?=
 =?us-ascii?Q?YHvmvFCZ/iYfnjCn5LMXUWas5NtDiQXsj8KhTnPHTRJ9mfU06ZpFFfDml53Q?=
 =?us-ascii?Q?46qku8rsjF7+TP6RTJdbksl29CJLtxBPv7OQhCrDO/bfBVYak7gKpKoqXq1j?=
 =?us-ascii?Q?YGeEIZDbRwRrS9yUema4keccrGdwaEgQA00i2NaNcwP+d6YOx8ItX8jcp6Se?=
 =?us-ascii?Q?/22g5P2rFKbXa8rbdb97HyQmnlbjjnUDU4ivhbLb/pZLdHdaGwUC/v3aau+A?=
 =?us-ascii?Q?TZ5aqDIGezI/IHYXc16RWER+4E/U3dglB/DgY+ciRq2WwZpJOosTrN/2/PHE?=
 =?us-ascii?Q?wP++iHWWkAxezCdKQAXSFFmW+nRBN6r9r25tfUGb3jnDFAwMQHHPYt1HCD1g?=
 =?us-ascii?Q?RhO7cSeUswJNQRg3NN8jkkOPm5exXpNwoOfhIn/qtsdvtUOFB0MtLxNPDbTV?=
 =?us-ascii?Q?uU34ZWc0MDkx+vZ9Ym66czVYQoWLO1OU03mZVHHitmHUwfGPexzOiZV3Qvdo?=
 =?us-ascii?Q?5EkIJF8Boftxg99Bklvl8qlxnVN+s5NgrrHlUmxabzd7CzkqlksIUj6eVi9n?=
 =?us-ascii?Q?AmIPYiFOBw9Um9DrbonaKFs3kiDCZumrD7tPEOKtD+JwvZTNocncoJOa6Iwu?=
 =?us-ascii?Q?lOo+9FRJVwaZnS9Wo2DaB/6v+Wzs1oS6ghjd/qP0Xb2PF4E7e/a9BYot4L1L?=
 =?us-ascii?Q?po2EwfqHdOrlu44fv5QIQwuqk2F/P4mIbn22rhq+bLat9au4kS/j6s1oUsGh?=
 =?us-ascii?Q?SAc1pdtXOhRNvSyZ6KjBXWVFSZyPH0EI3XE3dqlGldiIe/3EPcZPsXty/wub?=
 =?us-ascii?Q?pY++BAFzDy7ES7ISq4gJd+6DXm80M5T1njv/wjhyImzpOlAjUK0qlsUUkr+Y?=
 =?us-ascii?Q?Piiqk+rNIJm6G6dmuadLWhinKLurtdrIDZ1x1EXBSYoQwjlTF/1ZDY2d8btP?=
 =?us-ascii?Q?4Fu8ppgcO0L2h3TT2tb9+01FuzRmSnEpnNaZ2sxcTbR7770eyWFdE0YIzdMq?=
 =?us-ascii?Q?561Y8nw9MILEpt2UySfi7Mgu+k3tXrekCpAjPvnPR7kBr8S1RHpaYK19ZAPA?=
 =?us-ascii?Q?6287Vb2MNM90KenVzQGFUm2IjGqA0KnL5PFZSg/eK7g2IlZmzAfQMvG5S1eS?=
 =?us-ascii?Q?A1vS6UEL8jjTlZRa/djQAlZOoWiXxv0nrO5pM6zwZZY9CrO/nNpbWHzYUdnH?=
 =?us-ascii?Q?oi+RaSLIMZ626NDX5gEnmnKpJtl7pIFpNHPB1oKzkCDHdJ2NUfde+ap8bewR?=
 =?us-ascii?Q?BaOrdtI0AQfBw8GpXkASnnEkPhYXvLsKthP6EYffbeUbCgErY2KTj0waXUD8?=
 =?us-ascii?Q?UcifEAgQzUJt/FLTjFLQZwDo183iPzS8IDpniPQ++sqUr6X07ugc2r/rL588?=
 =?us-ascii?Q?LmcxDDOsuMoXY0qJxkI=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2c4e4b74-5f3b-4969-83c7-08dc0dfceed0
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jan 2024 14:45:17.1032
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: A1U9vpxigpf9kqmL1GbNHyFXMNfZKeUBGeHIMgdhnIRUgyYfkOSVLdV+e3aYn0lm
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5111

On Fri, Jan 05, 2024 at 02:52:50AM +0000, Tian, Kevin wrote:
> > but in reality the relation could be identified in an easy way due to a SIOV
> > restriction which we discussed before - shared PASID space of PF disallows
> > assigning sibling vdev's to a same VM (otherwise no way to identify which
> > sibling vdev triggering an iopf when a pasid is used on both vdev's). That
> > restriction implies that within an iommufd context every iommufd_device
> > object should contain a unique struct device pointer. So PASID can be
> > instead ignored in the lookup then just always do iommufd_get_dev_id()
> > using struct device.
> 
> A bit more background.
> 
> Previously we thought this restriction only applies to SIOV+vSVA, as
> a guest process may bind to both sibling vdev's, leading to the same
> pasid situation.
> 
> In concept w/o vSVA it's still possible to assign sibling vdev's to
> a same VM as each vdev is allocated with a unique pasid to mark vRID
> so can be differentiated from each other in the fault/error path.

I thought the SIOV plan was that each "vdev" ie vpci function would
get a slice of the pRID's PASID space statically selected at creation?

So SVA/etc doesn't matter, you reliably get a disjoint set of pRID &
pPASID into each VM.

From that view you can't identify the iommufd dev_id without knowing
both the pRID and pPASID which will disambiguate the different SIOV
iommufd dev_id instances sharing a rid.

> But when looking at this err code issue with Yi closely, we found
> there is another gap in the VT-d spec. Upon devtlb invalidation
> timeout the hw doesn't report pasid in the error info register. this
> makes it impossible to identify the source vdev if a hwpt invalidation
> request involves sibling vdev's from a same PF.

Don't you know which command timed out?
 
> with that I'm inclined to always imposing this restriction for SIOV. 
> One may argue that SIOV w/o vSVA w/o devtlb is conceptually immune
> but I'm with you that given SIOVr1 is one-off I prefer to limiting its
> usability other than complexing the kernel.

By this you mean give up on SIOV entirely and always assign the full
pRID to an entire VM? I'm confused what restriction you mean if you
can't rely on the PASID?

Jason