Received-SPF: pass (google.com: domain of linux-kernel+bounces-14673-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Tue, 2 Jan 2024 13:46:40 -0400
From: Jason Gunthorpe <jgg@nvidia.com>
To: "Tian, Kevin" <kevin.tian@intel.com>
Cc: "Liu, Yi L" <yi.l.liu@intel.com>, "joro@8bytes.org" <joro@8bytes.org>,
	"alex.williamson@redhat.com" <alex.williamson@redhat.com>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>,
	"baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
	"cohuck@redhat.com" <cohuck@redhat.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>,
	"nicolinc@nvidia.com" <nicolinc@nvidia.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
	"chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
	"yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>,
	"peterx@redhat.com" <peterx@redhat.com>,
	"jasowang@redhat.com" <jasowang@redhat.com>,
	"shameerali.kolothum.thodi@huawei.com" <shameerali.kolothum.thodi@huawei.com>,
	"lulu@redhat.com" <lulu@redhat.com>,
	"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
	"Duan, Zhenzhong" <zhenzhong.duan@intel.com>,
	"joao.m.martins@oracle.com" <joao.m.martins@oracle.com>,
	"Zeng, Xin" <xin.zeng@intel.com>,
	"Zhao, Yan Y" <yan.y.zhao@intel.com>
Subject: Re: [PATCH v7 1/3] iommufd: Add data structure for Intel VT-d
 stage-1 cache invalidation
Message-ID: <20240102174640.GA171005@nvidia.com>
References: <20231117131816.24359-1-yi.l.liu@intel.com>
 <20231117131816.24359-2-yi.l.liu@intel.com>
 <c967e716-9112-4d1a-b6f7-9a005e28202d@intel.com>
 <BN9PR11MB5276D14D2A7FF60B41A6A7B48C93A@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <BN9PR11MB5276D14D2A7FF60B41A6A7B48C93A@BN9PR11MB5276.namprd11.prod.outlook.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?IzjYOORsvPkkzDpdoRD+Kgi8Q9pD1fyEEsSDGT6YBzIzMfvndtiHLHy/heKN?=
 =?us-ascii?Q?gj7beS8jRUic5AOny8+FbzerLXjoTJHmnxp8kbwLUFqoNXcl626bNCZV1T1v?=
 =?us-ascii?Q?XBEsnVBSK7E2IEWhQk1wbPub1IKkRL7nVZ/dm9bzyzqUIVP0ihl7uLkmyKdG?=
 =?us-ascii?Q?1qEbJTPFFC9B7kmFdPPUmXZpa+D/rgjLSakXNlX66mK+DWQ8e8Sosi7yM3qi?=
 =?us-ascii?Q?pCF47oGHknTgEn0j3kxjYZpA62xxIAyrq7e7TT31wfeYncxQGjK4WOdkU5US?=
 =?us-ascii?Q?BcvdWlVRn4JewIiPo8aEGwA/JTccckWOjwuoS/7UDi8Z+n6VE1FDjLZ33vNp?=
 =?us-ascii?Q?JkZKs6hyc+puDqAMQZKDFUBmPzdwxPwbdO6TbViGbaaHVU2ufyQLQenFVY6T?=
 =?us-ascii?Q?cyonCVU7nzzGIcspjAzTg7Z+0pcx7/dEaBK6Im0U2GFGKCLBjtg5dmgsceYK?=
 =?us-ascii?Q?n63LBrpB0d864l3wafgcpU52oBo5c9yfPMW1oElrlncnUPOf00XiMlbw0K2o?=
 =?us-ascii?Q?pp3YpWFwsuAMegMNhZP/4u7y6Ixax6IBfAT77RdvM9p51ZxoDLN9nlqPFbIu?=
 =?us-ascii?Q?yP3CbslvLhsn1eU3h8ANkHHDis0WtCWFMseADnaSRWuwbDI84xZ8BAMw8xuY?=
 =?us-ascii?Q?YXfht/91EbsyZE3mKzayvNm4T1SCw5q0Q6i4M19UMAMygInvjjuQyayPeUcK?=
 =?us-ascii?Q?AT8cSKvh6mMvZNGxmDpW246uJ1d7LVArpIrGNxpadZ1BJRLrGnWe+S0QzDqX?=
 =?us-ascii?Q?9j06i06HOMaTkJpOy7LMR+EYtFp2hBsBYQuIPmek9Wk6xihy2izuBLGrKXai?=
 =?us-ascii?Q?V1xITc9cF6dwqSTmcVn4SNxuIGiT4tvBT3dUYWx6yOYJHGoOkbLZyf0rYtxL?=
 =?us-ascii?Q?cUCyFF40m/97fVkfTCuJYz8z7Xzhi/qJrrg/5ExIx9k6Z15AEYAR0xuFRxFs?=
 =?us-ascii?Q?YvP1jzVQAfmmA18Q2KArgPz//s1PSLf36Q4VXQOjuFjPbwoAKzFw1sVdDQwa?=
 =?us-ascii?Q?wn6e4ySqzdsban3eAUZtAkLE2zmDZoD+XtTJv6S62C++QMfuRvmc+JNwCXOo?=
 =?us-ascii?Q?FG3OhRjXVD3d6S6sXFo22SbGVEItK6HSfDC6eGiPPOvAWoMa/e11iak/5lPc?=
 =?us-ascii?Q?AavctH2B8h8r98I5hR94vFSXf+x85WMCu2jPu221MiAfw0Q61aZlGncEz1nN?=
 =?us-ascii?Q?izvo7xyuG2bLVgy8OGwdu4VxLJCancUO+rW7z2MjtIQtK6FwAlRFl4/6aFYt?=
 =?us-ascii?Q?uApUOXUETDxFVNGcd2PLkaIbvLGLXhqYEkAgCK/nK3dV7pSNqHhg88BQzwZP?=
 =?us-ascii?Q?kXG5sz9Es2lV0Ba2qCZ6HLjE55kKAoxLcel8pHdhB2ICXX4jHBounzjJbBHv?=
 =?us-ascii?Q?7aYpSXDqVk5SbIVBBfimYRTUvaazbVZ5WxxU4W4zF45BYWO5uDPMlCPTROtj?=
 =?us-ascii?Q?n6Wo0YUBaFzI9C6mcgxLXNnjOM5ArWog2uLcNmIhCwCDVbISlp3hrfikANb/?=
 =?us-ascii?Q?ZBTvtvIYWt3d7TGNRdvpMkACzslAoUbwHGy/uRaZQF0V0d+JWs5Enns3Ss6P?=
 =?us-ascii?Q?6hgsNlepFQ1lz05fPeE=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5c802ba9-64e4-4ed9-200f-08dc0bbac74f
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jan 2024 17:46:41.7249
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: htsGWAgAth3edLmxdFo3T680mJ/nP+lfCBoa0juL/3I3f2iitjSfdjuVg9vOpz6m
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB5405

On Fri, Dec 15, 2023 at 01:50:07AM +0000, Tian, Kevin wrote:

> > - Reuse Nicolin's vRID->pRID mapping. If thevRID->pRID mapping is
> > maintained, then intel iommu can report a vRID back to user. But intel
> > iommu driver does not have viommu context, no place to hold the vRID-
> > >pRID
> > mapping. TBH. It may require other reasons to introduce it other than the
> > error reporting need. Anyhow, this requires more thinking and also has
> > dependency even if it is doable in intel side.
> 
> this sounds like a cleaner way to inject knowledge which iommu driver
> requires to find out the user tag. but yes it's a bit weird to introduce
> viommu awareness in intel iommu driver when there is no such thing
> in real hardware.
> 
> and for this error reporting case what we actually require is the
> reverse map i.e. pRID->vRID. Not sure whether we can leverage the
> same RID mapping uAPI as for ARM/AMD but ignore viommu_id
> and then store vRID under device_domain_info. a bit tricky on
> life cycle management and also incompatible with SIOV...
> 
> let's see whether Jason has a better idea here.

I think v10 is OK

struct iommu_hwpt_invalidate {
	__u32 size;
	__u32 hwpt_id;
	__aligned_u64 data_uptr;
	__u32 data_type;
	__u32 entry_len;
	__u32 entry_num;
	__u32 __reserved;
};

Sends the invalidation to the HWPT which matches what Intel wanted
where the entire HWPT and all its associated devices are
invalidated. No seperate per-device invalidation.

For error and event reporting they should be returned to userspace
with the IOMMU dev_id indicating the originating PCI function.

The VMM would have to convert dev_id into vRID according to the vIOMMU
instance that the device is hooked up.

In iommu land we should never have a "RID" but always some kind of
device-specific "device ID" which is the index into the particular HW
table, and that ID is naturally scoped to within the IOMMU instance
that owns the table - so it is very much not a global ID that can be
used alone in any of the uAPI.

The uAPI should use the iommufd device ID to refer to specific
devices.

Jason