Received-SPF: pass (google.com: domain of linux-kernel+bounces-16794-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Date: Thu, 4 Jan 2024 10:36:58 -0400
From: Jason Gunthorpe <jgg@nvidia.com>
To: Yi Liu <yi.l.liu@intel.com>
Cc: joro@8bytes.org, alex.williamson@redhat.com, kevin.tian@intel.com,
	robin.murphy@arm.com, baolu.lu@linux.intel.com, cohuck@redhat.com,
	eric.auger@redhat.com, nicolinc@nvidia.com, kvm@vger.kernel.org,
	mjrosato@linux.ibm.com, chao.p.peng@linux.intel.com,
	yi.y.sun@linux.intel.com, peterx@redhat.com, jasowang@redhat.com,
	shameerali.kolothum.thodi@huawei.com, lulu@redhat.com,
	suravee.suthikulpanit@amd.com, iommu@lists.linux.dev,
	linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org,
	zhenzhong.duan@intel.com, joao.m.martins@oracle.com,
	xin.zeng@intel.com, yan.y.zhao@intel.com
Subject: Re: [PATCH v7 1/3] iommufd: Add data structure for Intel VT-d
 stage-1 cache invalidation
Message-ID: <20240104143658.GX50406@nvidia.com>
References: <20231117131816.24359-1-yi.l.liu@intel.com>
 <20231117131816.24359-2-yi.l.liu@intel.com>
 <c967e716-9112-4d1a-b6f7-9a005e28202d@intel.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <c967e716-9112-4d1a-b6f7-9a005e28202d@intel.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?8l8o+adqCgJoKsiMLTvfKHMvNfFylr+5bnhR7b1kkqjhzbWGvaWEnkX6tpb6?=
 =?us-ascii?Q?9HWf0TGlMvUIdFRlbwEvrxAK/qkt5V9d+/iGxsw/z1E0/qjK339DD3JlPXl2?=
 =?us-ascii?Q?NkehyLkgiFSL4lHOJu0E/zhJnBjOwUYLxfwEzjHulGT9z0MXb4nzYBg119/K?=
 =?us-ascii?Q?SHYekZjpuzbl8q+4zy4fhUEpaNr802WyM+N8gPExLbSatfI1G7p2LDsm9Q7E?=
 =?us-ascii?Q?esn9i3aE9SdGW9/3nGeckORjSSy76uCdm7DcHW87b5f0mgHo6hZ0hZrerZE0?=
 =?us-ascii?Q?pQDioYbjVY9BA9NgdNcyoPPFZnvITaRKfAkfTIu/XEfrVKGyWFgCxpQCeRRi?=
 =?us-ascii?Q?9HcjCJhWVTuU5HW4RHVo/fkVfay1336VgEkeey9qjLscXpMW8kw/auzmu+GT?=
 =?us-ascii?Q?kiiM3/7J+1hzcROJ8GTP7teELowDFZ3un/VZ2zyZR3Q5vsqYgM0YF2dEiu/U?=
 =?us-ascii?Q?GSjlZvcIxkRdnNErdcgFj2BpXpK5aTmek6UpqKUkUzo7zS28bUt0lUWfJprD?=
 =?us-ascii?Q?vR4fwCOoM60+UN04Sdcgtf7SpNqiiR6FFxJ4ACkpI3fi+RaygfoAgJykvzsk?=
 =?us-ascii?Q?yNNwRja6oD4KnKC8qmETH9rMRbe5szb9SmvsyESX1JSxN1WnXx5mYmXMYDZA?=
 =?us-ascii?Q?RHZQpYu4jh1Snc76AXidPe9tSpUHUpg5z7Hur+CclNCghclnwRncbxN4ssfT?=
 =?us-ascii?Q?NbVjfz5EpNGCuOP3y+BJ2Gb9ubVFBijYv8Jno2fcy7HfxqJP6zzASaKM4rHV?=
 =?us-ascii?Q?H2Z9QrjXKpH0UNApebE74IlkbJbAdreSUAvQihcFw1IrcJFuErrt3GM0gOzs?=
 =?us-ascii?Q?omtn7YO1F2Y35gAKHGts1IREOnG7JpjSPwHA8Gc/D5pvYBRksRYoA9H2UV4h?=
 =?us-ascii?Q?6zMKAOfOwxkxGPbOo5a1M1OXlNFXN4ydWnYwu0TVxGHtCTh19vMPqeNchBF2?=
 =?us-ascii?Q?yu8pBhLA5yY22SAoYJf2HyD06QQyyKRxqWpHImrcZpSMO/p5s3LBqeARv92L?=
 =?us-ascii?Q?DJKqcMb1H4YMfObhoX4G8+Gf2hCrDaZC+cGz6uJoE21h+YRj/fgu75j08CQa?=
 =?us-ascii?Q?GE7FLdgdGCc4vxz9dMae0i2TNM5Gh51v+Gqz0lqNTkEkJpLiO/x8czX8i9n/?=
 =?us-ascii?Q?LuRXh1mzQ3XGeiiUjkOeoNpyDb35en6UF6HICDBc25B28XPEJSr39UFpDE1I?=
 =?us-ascii?Q?O21O5xctM/q1Zn+7/StnacNgt1r06cpZiTxlNgxX9gza+WQQymEt1xxB+D1a?=
 =?us-ascii?Q?fCyrh40HkIlYH8edBBwS6qWpcF8GyrH/a8NDn4XG8UjtNmHuMJiddm9SHIiN?=
 =?us-ascii?Q?NJBCw/sHzYknkNtrkWv0L6RZa4nhNGDHFgRMzgouTPGS+7Q5dY3BoLVcv7ga?=
 =?us-ascii?Q?6ZP2jYipCRWJKsvUYY3LgFcUkZ93RZiIdDAP9o2kJkugUHL6C403q796LMoS?=
 =?us-ascii?Q?5sfaFHbYVaaYHW5TbY2nI69cPf5L58nxOcIPSIUJkyOj54JQsy32JWEXFE38?=
 =?us-ascii?Q?M+lPHXjkRJ2fJIlpj23fjcs4mbck/a+zouXaiITyNec/6AsgaTQVstbl9VW1?=
 =?us-ascii?Q?PJFEDROLk+7qMf04XVkqOHnE4xSQ1hMvTOjKUakZ?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: dc1e945c-94f6-47c7-9c2a-08dc0d329bd7
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jan 2024 14:36:59.5578
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: uJxy1/sHJnXVg89OefCFULf6RHk04nfbWkRHb3teoDwQg57gCtVbLua2YNg8rmbB
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5072

On Thu, Dec 14, 2023 at 07:26:39PM +0800, Yi Liu wrote:
> Per the prior discussion[1], we agreed to move the error reporting into the
> driver specific part. On Intel side, we want to report two devTLB
> invalidation errors: ICE (invalid completion error) and ITE (invalidation
> timeout error). Such errors have an additional SID information to tell
> which device failed the devTLB invalidation. I've got the below structure.

IMHO all of this complexity is a consequence of the decision to hide
the devtlb invalidation from the VM..

On the other hand I guess you want to do this because of the SIOV
troubles where the vPCI function in the VM is entirely virtual and
can't be trivially mapped to a real PCI function for ATC invalidation
like ARM and AMD can do (but they also can't support SIOV because of
this). :(

However it also makes it very confusing about how the VM would
perceive an error - eg if it invalidates an SIOV device single PASID
and that devtlb fails then the error should be connected back to the
vPCI function for the SIOV's specific PASID and not back to the
physical PCI function for the SIOV owner.

As the iommu driver itself has no idea about the vPCI functions this
seems like it is going to get really confusing. The API I suggested in
the other email is not entirely going to work as the vPCI function for
SIOV cases will have to be identified by the (struct device, PASID) -
while it would be easy enough for the iommu driver to provide the
PASID, I'm not sure how the iommufd core will relate the PASID back to
the iommu_device to understand SIOV without actually being aware of
SIOV to some degree :\

(Given SIOVr1 seems on track to be replaced by SIOVr2 so this is all a
one-off I was hoping to minimize such awareness)

Jason