Received-SPF: pass (google.com: domain of linux-kernel+bounces-15821-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Date: Wed, 3 Jan 2024 13:52:02 -0400
From: Jason Gunthorpe <jgg@nvidia.com>
To: Nicolin Chen <nicolinc@nvidia.com>
Cc: Yi Liu <yi.l.liu@intel.com>, "Tian, Kevin" <kevin.tian@intel.com>,
	"joro@8bytes.org" <joro@8bytes.org>,
	"alex.williamson@redhat.com" <alex.williamson@redhat.com>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>,
	"baolu.lu@linux.intel.com" <baolu.lu@linux.intel.com>,
	"cohuck@redhat.com" <cohuck@redhat.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
	"chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
	"yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>,
	"peterx@redhat.com" <peterx@redhat.com>,
	"jasowang@redhat.com" <jasowang@redhat.com>,
	"shameerali.kolothum.thodi@huawei.com" <shameerali.kolothum.thodi@huawei.com>,
	"lulu@redhat.com" <lulu@redhat.com>,
	"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-kselftest@vger.kernel.org" <linux-kselftest@vger.kernel.org>,
	"Duan, Zhenzhong" <zhenzhong.duan@intel.com>,
	"joao.m.martins@oracle.com" <joao.m.martins@oracle.com>,
	"Zeng, Xin" <xin.zeng@intel.com>,
	"Zhao, Yan Y" <yan.y.zhao@intel.com>
Subject: Re: [PATCH v7 1/3] iommufd: Add data structure for Intel VT-d
 stage-1 cache invalidation
Message-ID: <20240103175202.GT50406@nvidia.com>
References: <ZXu5whlIGfiq16wF@Asurada-Nvidia>
 <BN9PR11MB52766D7F774510E0181CC89B8C93A@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZXvI2IiXwwuHRE8V@Asurada-Nvidia>
 <7c398efc-8a2f-479d-bcff-ded8cc1ef3d0@intel.com>
 <20240102233849.GK50406@nvidia.com>
 <c59a780d-4030-4815-a34b-fb2e2f902ab3@intel.com>
 <20240103160108.GP50406@nvidia.com>
 <ZZWP7iBqUtbTRb3s@Asurada-Nvidia>
 <20240103165848.GR50406@nvidia.com>
 <ZZWUD+lCw3mRc/15@Asurada-Nvidia>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZZWUD+lCw3mRc/15@Asurada-Nvidia>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?OZPLm9c/cLv964gmLrzEONBk2+JhCey2sBCNGeqxny7IE3PovwEUgOQwCJMk?=
 =?us-ascii?Q?zTnCVxCtuj3aIlTcMoxMTD6LruweyQJAQoxn37B8FuCWgmj+A6DdPHn18Ug5?=
 =?us-ascii?Q?QUaPbWEN5d2UmR4lgGW+ZJAimWdLqkA9Ikw1H+TdbePR3lkkuj6/5UKkBPP6?=
 =?us-ascii?Q?qaV3wT3XcMmNi2iWzOyRB6Rq0vKd0S4aUEha7CwoH1iCrNrfWj1dpNCMMdwU?=
 =?us-ascii?Q?osZgl+2a1vaxNq6cUJsvySmZWw9jzwUBsaNd4H0xoIDFUhKpGb6p7mHDqogl?=
 =?us-ascii?Q?bd6AlOL8yqoVMwD/iqJTbuXw1IUbMFzpfUU1zH48N7Xbyd6RbW80Qs1zwjXb?=
 =?us-ascii?Q?yVE5YMzACS2Ax7HhnWWtSdP3vyv6k7oXvynOMGTYvnqzAylOVkKub66tAwGl?=
 =?us-ascii?Q?2PFgJXtZkpAhGp0ckH43lEdYUegRwNp5yEFdX8TzqQJJWUF1zLavFaGZuyAF?=
 =?us-ascii?Q?lBmQeFygD9UY5G6B/6eVgrj244st4C3lj0flg4SLjSHgCaV9EYCNuQzVsDNV?=
 =?us-ascii?Q?W5JBiYLSsUKh/jQo0RtEQMFiOZwUGhiiVUUQu4pUYT08xwCngoiukVignUDr?=
 =?us-ascii?Q?2ksAGVUhamjWDkewdO2nuO3vOUIrNNd+3AMHAqPMEPIb7F8L/YBwUUw4wIj0?=
 =?us-ascii?Q?kf+iNbN9xHNZRef1MPjhvFl4nLyVl7yovwHPdUoR3s0epRg5YT+f5GtA84fW?=
 =?us-ascii?Q?kJ1igOwMmyNxO7j0pMZ/Le6uI1VoX1m389E8InlsjBj5vi5PlNqXseopXWYb?=
 =?us-ascii?Q?I43P+LkKrPcK+UxF1K1vE+AJ16ijbvCjMKrwgjCuXkL7ILdmgwIt7LTnfLfT?=
 =?us-ascii?Q?EYKAtf/vCMbsLR79oVe5iveXNR2Rlh6IPta4m+izFMy4k/pD/4ykI5SvwhQU?=
 =?us-ascii?Q?CgRUnHojcIgYzBV04hv6E4OtbmBh6epkA/3IT2PzBtX+Wr5FEjTGaafun5VR?=
 =?us-ascii?Q?QunUw2cUcrpdX/cZxkp/hcPhFdIQZQDiE5bjOCVOwZ4fXRLOfaFtAHPKVGwb?=
 =?us-ascii?Q?Pe9aKnpD60vs/02MG+04WAEdrKmVAcwzI8FKnWQq6oNETla4uCmR7Ia71mYH?=
 =?us-ascii?Q?Rf/Dtp8e9X5V2TjJ8ht5zrCfzTSLyMDxiKCeh7TsyzTCb/ePwGhQyp/S+SuA?=
 =?us-ascii?Q?oINth03uSSlp3DOdBwqRCoh38aTVE/3zNiBUk7bV7bnvPyRQlvlaRjegq6o/?=
 =?us-ascii?Q?AWtiM3194jl2jt4Gxoruzx7lRh0bJjSzE0Ev3YRIb/RoXI/zf7Djm/lI9mux?=
 =?us-ascii?Q?+5VWG1Rlba+KLcB0AmhlIgzr7Qo/lSxRUryWQwLIoi0pN193v0zQ8EOrAI8h?=
 =?us-ascii?Q?hUrico2uj7T59jkn9yyOPUTMDZwVjL9Nl1u1No597r2/ZcPIWcyX/KHr6Bvi?=
 =?us-ascii?Q?dfVp9OBmBofevxPzEtelarMXxUQoNvxAPyNavUIHptzEZSr9UGeIZCZtebxG?=
 =?us-ascii?Q?7vQKhcO6fgeY2r2sUbQUhAE43Tq1+wiXn3ym/TcaDYHXWvZvbGEdoCcF15T4?=
 =?us-ascii?Q?8hvUE7PAS/Bc5Srqu8ht7u8gt1YM6VCOFSAeDhVTgin7E1vjFYuSQGnj/ywz?=
 =?us-ascii?Q?NuUslN+Pz2iBtQNPi3Rc6XQy+3Z2mmoUmeuuJ1rl?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ececa460-de5d-4585-38db-08dc0c84b111
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jan 2024 17:52:02.7874
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: jFoPtjlpt7n9Pc2gefnnXgoUHHIWZBMlp7aHpVXJMKoAg4YVzZoYzItdbXU0hmBH
X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR12MB9355

On Wed, Jan 03, 2024 at 09:06:23AM -0800, Nicolin Chen wrote:
> On Wed, Jan 03, 2024 at 12:58:48PM -0400, Jason Gunthorpe wrote:
> > On Wed, Jan 03, 2024 at 08:48:46AM -0800, Nicolin Chen wrote:
> > > > You can pass the ctx to the invalidate op, it is already implied
> > > > because the passed iommu_domain is linked to a single iommufd ctx.
> > > 
> > > The device virtual id lookup API needs something similar, yet it
> > > likely needs a viommu pointer (or its id) instead? As the table
> > > is attached to a viommu while an ictx can have multiple viommus,
> > > right?
> > 
> > Yes, when we get to an API for that it will have to be some op
> > 'invalidate_viommu(..)' and it can get the necessary pointers.
> 
> OK! I will try that first.
> 
> > The viommu object will have to be some driver object like the
> > iommu_domain.
> 
> I drafted something like this, linking it to struct iommu_device:
> 
> +struct iommufd_viommu {
> +       struct iommufd_object obj;
> +       struct iommufd_ctx *ictx;
> +       struct iommu_device *iommu_dev;
> +       struct iommufd_hwpt_paging *hwpt;
> +       /* array of struct iommufd_device, indexed by device virtual id */
> +       struct xarray device_ids;
> +};

The driver would have to create it and there would be some driver
specific enclosing struct to go with it

Perhaps device_ids goes in the driver specific struct, I don't know.

Not sure it should have hwpt at all, probably vmid should come from
the driver specific struct in some driver specific way

Jason