Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33;
Date:   Tue, 12 Dec 2023 10:44:21 -0400
From:   Jason Gunthorpe <jgg@nvidia.com>
To:     Nicolin Chen <nicolinc@nvidia.com>
Cc:     Yi Liu <yi.l.liu@intel.com>,
        "Giani, Dhaval" <Dhaval.Giani@amd.com>,
        Vasant Hegde <vasant.hegde@amd.com>,
        Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>,
        joro@8bytes.org, alex.williamson@redhat.com, kevin.tian@intel.com,
        robin.murphy@arm.com, baolu.lu@linux.intel.com, cohuck@redhat.com,
        eric.auger@redhat.com, kvm@vger.kernel.org, mjrosato@linux.ibm.com,
        chao.p.peng@linux.intel.com, yi.y.sun@linux.intel.com,
        peterx@redhat.com, jasowang@redhat.com,
        shameerali.kolothum.thodi@huawei.com, lulu@redhat.com,
        iommu@lists.linux.dev, linux-kernel@vger.kernel.org,
        linux-kselftest@vger.kernel.org, zhenzhong.duan@intel.com,
        joao.m.martins@oracle.com, xin.zeng@intel.com, yan.y.zhao@intel.com
Subject: Re: [PATCH v6 0/6] iommufd: Add nesting infrastructure (part 2/2)
Message-ID: <20231212144421.GH3014157@nvidia.com>
References: <20231117130717.19875-1-yi.l.liu@intel.com>
 <20231209014726.GA2945299@nvidia.com>
 <ZXd+1UVrcAQePjnD@Asurada-Nvidia>
 <20231211215738.GB3014157@nvidia.com>
 <ZXgL+GCGPgH+hlXo@Asurada-Nvidia>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZXgL+GCGPgH+hlXo@Asurada-Nvidia>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?PfYkKXHCJwfSJEXmj9f9/wvpAEtmhGkI6f4a/6Fh2XlmekyPCwPWNxyW2uwA?=
 =?us-ascii?Q?9CqSMfcFq0JeOnHCZQ5Tjl2L5258fm8egS3tJ0cPlvQIE7Tgy9ROr/j3A4PC?=
 =?us-ascii?Q?j4Uj05s2VES+anyh0eIEvgL//Yp9zPDo0n7CXemGbRzIyo7QT/g5YYideNuW?=
 =?us-ascii?Q?HuHTR9qyNKoWusZvZ7PG+I65Gbhd2HjHdS8H0jDzHtcPvaIHSWMx3IpPnZDd?=
 =?us-ascii?Q?/4GsugbkpSudPaWDjqz1mkiy0tNCLOE6KC/AgkSPXIjpjn1XsvkhCC/0w/LG?=
 =?us-ascii?Q?Sq3cNbGTzmWUt8gQ+qaox+hY+YPQ53uxSQ+kNeyOWtbQNcz4YIV9vhUWHTwV?=
 =?us-ascii?Q?VjMnmPUiMj+uJFnZXmbup+hu+EWZWvZcmC1wptyl326Pd0NE7dgsf2MwxRYv?=
 =?us-ascii?Q?dfvhpcL+l4lePL29MP7x/wOK5eQy7cus4pyDRwOSfTI1Sh9dBetjZfwcCLLe?=
 =?us-ascii?Q?WzgY0eFq8CWBABUAJNBuERxHExUk0QlGzEXLuuAPwg7I303nWa80SKw4L7fu?=
 =?us-ascii?Q?fWfCE1jswKJgSi6BoQR7w62m/avxvobEHjH8CSSfta1Qim1dmeJ6ZCOR9dXH?=
 =?us-ascii?Q?Upw2j3ayO3LTh3rcq2Uhptiq48TkPNsPLvt5+x/2EYdWVtusLe2ZIVgaUuk9?=
 =?us-ascii?Q?73ffM0m3KWrbZehBWH6O4XyOVRtm1JH3im2/Pkd/r+HdbESJPt6sYMqpk9bQ?=
 =?us-ascii?Q?L+U97ZuFcJLxNEEZHumzBivwf0X2ETJ73QrjFJ/gS+F0vh8ery8uvVCqJT6i?=
 =?us-ascii?Q?uUQ9gbahuB/HcqMOoeV3FGR6oAitnJ3Y5WEtiAHqPa5sBfnmk/QmUjm38DKo?=
 =?us-ascii?Q?S60KuLdZfEnqpcKLys3RS7hlJ7PTOkfaRoOuiatLnVTz2mXd0hsMGHAglALu?=
 =?us-ascii?Q?Gf5l1QxcP3Rtim8ZNejWxMxlAb3y3DiVhqGenUQrOkvSUAZwKbXQxn1Mhu3I?=
 =?us-ascii?Q?za+RIRqzFgWE+aHYHJfw04EEZD2wiRExay1V+pJMEhxfmdmXjlNiey7V1bJJ?=
 =?us-ascii?Q?76iSOd8yBLoSGuz77VDeeBcCgc6ZtnQV8zR69O1T6oXgtj3goO4kl+ucLqy2?=
 =?us-ascii?Q?kuo5dmxfcuGD00dEN+sY7ueLT9NvqOxeG1MoCtiLxFCddbFZHygbAbYiix2C?=
 =?us-ascii?Q?42FFv6E/1o5p4O7j5Xu9pwB4Uj50FbliQaOpthnLBa92KVhrea+iD6xb0Vxf?=
 =?us-ascii?Q?gTMHaPuqocC9Y8jpjtvCQe+71+R8QEZB2XTdcggIlFJl2n/DkKfFxOM3Xjip?=
 =?us-ascii?Q?NeHGypd59zynNOjYI+t4AGxQ1rROI4aUlsQk+9EYqz7SqWa1RxMaVWqzEhcx?=
 =?us-ascii?Q?u0QDC/K74RLxZ6RZclE15Src1b06okL6FCb0u7qJA0MOAcZo8h5tTYGFM5FI?=
 =?us-ascii?Q?YoMGmwD43I5B2WqFuApy0ObtbTfSnO9Tp36zMg7iUnt0RDdRokivIv08vCH1?=
 =?us-ascii?Q?RGbTdmt0Xmom/KIA2ASuJeDa0q0ZtBuGjQagOF2vRWiwjDPPyEwa6yn1Qm3E?=
 =?us-ascii?Q?aG+lXofnAiWKeGCLQqBVZ8waPgrOq9ZB0PEdK9DlxYlsFHXroSGOAeUesI1E?=
 =?us-ascii?Q?4vdpYEDBoi3wz7LPQ9EzBIIBxSJu3DfS7UNWyJJA?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 3c772e91-8a67-4618-6820-08dbfb20d435
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Dec 2023 14:44:22.2140
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: YU6yEARZupR7scGTLFZJ44ctQMidS4rYJPnFTrkBD+pyFihyiL/9rXOqqQLQMu/q
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB6804
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Tue, 12 Dec 2023 06:45:07 -0800 (PST)

On Mon, Dec 11, 2023 at 11:30:00PM -0800, Nicolin Chen wrote:

> > > Could the structure just look like this?
> > > struct iommu_dev_assign_virtual_id {
> > >        __u32 size;
> > >        __u32 dev_id;
> > >        __u32 id_type;
> > >        __u32 id;
> > > };
> > 
> > It needs to take in the viommu_id also, and I'd make the id 64 bits
> > just for good luck.
> 
> What is viommu_id required for in this context? I thought we
> already know which SMMU instance to issue commands via dev_id?

The viommu_id would be the container that holds the xarray that maps
the vRID to pRID

Logically we could have multiple mappings per iommufd as we could have
multiple iommu instances working here.

> > > > IOMMUFD_DEV_INVALIDATE should be introduced with the same design as
> > > > HWPT invalidate. This would be used for AMD/ARM's ATC invalidation
> > > > (and just force the stream ID, userspace must direct the vRID to the
> > > > correct dev_id).
> > > 
> > > SMMU's CD invalidations could fall into this category too.
> 
> Do we need a different iommu API for this ioctl? We currently
> have the cache_invalidate_user as a domain op. The new device
> op will be an iommu op?

Yes

> And should we rename the "cache_invalidate_user"? Would VT-d
> still uses it for device cache?

I think vt-d will not implement it
 
> > > I previously drafted something to test it out with iommufd.
> > > Basically it needs the pairing of vRID/pRID in attach_dev()
> > > and another ioctl to mmap/config user queue(s):
> > > +struct iommu_hwpt_cache_config_tegra241_vcmdq {
> > > +       __u32 vcmdq_id;			// queue id
> > > +       __u32 vcmdq_log2size;		// queue size
> > > +       __aligned_u64 vcmdq_base;	// queue guest PA
> > > +};
> > 
> > vRID/pRID pairing should come from IOMMUFD_DEV_ASSIGN_VIRTUAL_ID. When
> > a HWPT is allocated it would be connected to the viommu_id and then it
> > would all be bundled together in the HW somehow
> 
> Since we were talking about sharing stage-2 domain, the HWPT
> to the stage-2 domain will be shared among the vIOMMU/pIOMMU
> instances too?

The HWPT for the stage 2 should be shared

> I think I am not quite getting the viommu_id
> part yet. From the other side of this thread, viommu object
> is created per vIOMMU instance and each one actually tied to
> a pIOMMU by nature?

Yes

Jason