Received-SPF: pass (google.com: domain of linux-kernel+bounces-188726-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Date: Fri, 24 May 2024 10:19:12 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: "Tian, Kevin" <kevin.tian@intel.com>
Cc: Nicolin Chen <nicolinc@nvidia.com>, "will@kernel.org" <will@kernel.org>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>,
	"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
	"joro@8bytes.org" <joro@8bytes.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"linux-arm-kernel@lists.infradead.org" <linux-arm-kernel@lists.infradead.org>,
	"linux-tegra@vger.kernel.org" <linux-tegra@vger.kernel.org>,
	"Liu, Yi L" <yi.l.liu@intel.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>,
	"vasant.hegde@amd.com" <vasant.hegde@amd.com>,
	"jon.grimm@amd.com" <jon.grimm@amd.com>,
	"santosh.shukla@amd.com" <santosh.shukla@amd.com>,
	"Dhaval.Giani@amd.com" <Dhaval.Giani@amd.com>,
	"shameerali.kolothum.thodi@huawei.com" <shameerali.kolothum.thodi@huawei.com>
Subject: Re: [PATCH RFCv1 08/14] iommufd: Add IOMMU_VIOMMU_SET_DEV_ID ioctl
Message-ID: <20240524131912.GT20229@nvidia.com>
References: <cover.1712978212.git.nicolinc@nvidia.com>
 <c97a98a72ee3498c587e5898d6b899553ccd9b27.1712978212.git.nicolinc@nvidia.com>
 <BN9PR11MB5276A897A5386DFAC9A35F9D8CF42@BN9PR11MB5276.namprd11.prod.outlook.com>
 <ZlAoN3s96HL7ROth@nvidia.com>
 <BN9PR11MB52762A0BC67B64D4171480C78CF52@BN9PR11MB5276.namprd11.prod.outlook.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <BN9PR11MB52762A0BC67B64D4171480C78CF52@BN9PR11MB5276.namprd11.prod.outlook.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?qfZj8X6ezOyST0vMqxl0iIuNGbKNAuUglF6tdMOOxy+UPdPjCC2L0HhPAW8h?=
 =?us-ascii?Q?/zZ509gIEqiKInt0t3lMzhasp2qDgd9ErR8tBgKV5WwWHNYAUoIvRpXOnntI?=
 =?us-ascii?Q?ppmRSqAV9mZL6kjL7UrawyQf5pvPXZtrhqBjPfuwI1eqXfV6R7G9yq0D23gv?=
 =?us-ascii?Q?bn8fqLfvCMepZzWiUFO7ngIxdszoR04cwDk8dydjbyIDX47VbH8fnXgKhD/z?=
 =?us-ascii?Q?yflXIlifIo5VSdr9OZaP4g1lm27KlqaWnfySwRrYUfqfLHGgm2sUgXKkY+mi?=
 =?us-ascii?Q?gjBMjQRyRykUrDS4WTXBlA1chx2gmRgZJZ2SIHy/omzqOREfI8lCNfK77K/6?=
 =?us-ascii?Q?d1DKGEJhGY+jPcomsOGhD0Rp+yUvPGLI/oX7oSa6uOQaXLZz0CPoNBOxfyOE?=
 =?us-ascii?Q?U9+IXZRnkJw1qGM0al60Cj3bSkKHbHGyY56aWnlI5pMtvx9BQsNCIcAweWie?=
 =?us-ascii?Q?bKBkz5QmpNDmvzWW2ZBSFLtn8a83quavt4iEJ7Ur7mlZ5qXGXDVwyP/iMS/a?=
 =?us-ascii?Q?K0YSgmwUUGml1v4sUApjAr3eS4RalkF29Y/6fPtDrbUlVgUeI3JzR3YeMLrw?=
 =?us-ascii?Q?nOelwD+zGKYS+freN8NqYjQPWvXQ1IdTllTLKVBl+lvUmoeN9OpfjljK//0V?=
 =?us-ascii?Q?KAtZ7I9y0imj3i1sA0bD5oR0LslYtS1aZD/Z3fEO3dcpsnRQfw5fgQ/sWK1l?=
 =?us-ascii?Q?HKgdaqEPqxdXM23vMY7C2RtjFfqSq0nKSnGEBpNldua2daE9jeOpJ0AuxKFd?=
 =?us-ascii?Q?CfGg4TVGia18/x7unyx9iWCArvj/R/eIzZTAigd8PfSvaEBwH/YJJK77sP0I?=
 =?us-ascii?Q?kXv4q1B35kZpeZV/q9V2KWYQmy8zhZn2EZR9fC6Hxkptfsz1W+xMQ19f+6gg?=
 =?us-ascii?Q?LATpHvNbWqhdOPGrlTfrGKYvIzbjg+GJ6J39bvRcWTHbyEk0rh2NWJaUnYnW?=
 =?us-ascii?Q?abaeSm9vc1mxdTWQTW+AGJxr+FQc4gvZMLheSVkMUiD8m+GWeQMZtAu8q1y3?=
 =?us-ascii?Q?d186twhXw0LR3rViLxPKQSrCqKllTXFgFWjvVI+4qDgkAMBDaBBO9O2G4MAp?=
 =?us-ascii?Q?xmpa02l16N3eYHPaPcW9tPWtwrtdWOTS6/WM6wSFi9KEXVwRhl2lf7DfvrHv?=
 =?us-ascii?Q?4L42mWMaHJ3tSu4nDoKOIj98bK++ho1GmqTcpYvy8Ik9hPcz0jC/UglZ6VGx?=
 =?us-ascii?Q?LtufgCbqZxxFjx7YPuqXlxPAEs46CjJATLkt9FScfb8L6uSpkSIS9UkiyHiD?=
 =?us-ascii?Q?E1tZ5XSppYIMxLx7BCvKiRa/0xsP/6YY612DQ/V7BaE6QlrnJGuK1BazOPBH?=
 =?us-ascii?Q?zT5Lh1AbfJUTe18PGmQFsO66+w3/xNC/nWEtIig1puGjirfcf85aOmqotYTS?=
 =?us-ascii?Q?qjescCbO/W61tIpG+Dp3uhDEZ58Z+wgiSALhF/kV6DA0GRgvKz2U+xw9cE2U?=
 =?us-ascii?Q?Bv8vhhgUfSZ6Ie+sSlWMSj5LL37t8L3nPHafuev6ZZCxvNuw4GBW695X+GeX?=
 =?us-ascii?Q?6uaXtkH6SFotlZkLGgsHFqPNzP7DoqSOytCBuC/vr4EEAq6R0w8VDxsjtgWS?=
 =?us-ascii?Q?mT/M66LczCwPEoLyYO1q0WK5JoSdf9KUYk0ltsny?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d64e5c74-0119-4e69-62c6-08dc7bf41b13
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 May 2024 13:19:13.7360
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Z/F8KnS1dh8sKO68eWIIIYYg3NiT2rxdcsdxrbxnv5mqFuwtiiKwWqXfqXTCHPZb
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB6955

On Fri, May 24, 2024 at 07:13:23AM +0000, Tian, Kevin wrote:
> > From: Nicolin Chen <nicolinc@nvidia.com>
> > Sent: Friday, May 24, 2024 1:40 PM
> > 
> > On Thu, May 23, 2024 at 06:42:56AM +0000, Tian, Kevin wrote:
> > > btw there is a check in the following code:
> > >
> > > +       if (viommu->iommu_dev != idev->dev->iommu->iommu_dev) {
> > > +               rc = -EINVAL;
> > > +               goto out_put_viommu;
> > > +       }
> > >
> > > I vaguely remember an earlier discussion about multiple vSMMU instances
> > > following the physical SMMU topology, but don't quite recall the exact
> > > reason.
> > >
> > > What is the actual technical obstacle prohibiting one to put multiple
> > > VCMDQ's from different SMMU's into one vIOMMU instance?
> > 
> > Because a VCMDQ passthrough involves a direct mmap of a HW MMIO
> > page to the guest-level MMIO region. The MMIO page provides the
> > read/write of queue's head and tail indexes.
> > 
> > With a single pSMMU and a single vSMMU, it's simply 1:1 mapping.
> > 
> > With a multi-pSMMU and a single vSMMU, the single vSMMU will see
> > one guest-level MMIO region backed by multiple physical pages.
> > Since we are talking about MMIO, technically it cannot select the
> > corresponding MMIO page to the device, not to mention we don't
> > really want VMM to involve, i.e. no VM exist, when using VCMDQ.
> 
> can a vSMMU report to support multiple CMDQ's then there are
> several MMIO regions each mapped to a different backend VCMDQ?

This is something I want to support in the API, regardless vCMDQ uses
it or not..

> but I guess even if it's supported there is still a problem describing
> the association between assigned devices and the CMDQ's of the
> single vIOMMU instance. 

CMDQ should be linked to the VIOMMU instance only and not per-device,
I think that is a very important property of the system.

This means if there are multiple pSMMU instances then we need a
matching set of vSMMU instances so that the devices are grouped on the
proper vSMMU instance so they are compatible with the vCMDQ.

> I'm curious to learn the real reason of that design. Is it because you
> want to do certain load-balance between viommu's or due to other
> reasons in the kernel smmuv3 driver which e.g. cannot support a
> viommu spanning multiple pSMMU?

Yeah, there is no concept of support for a SMMUv3 instance where it's
command Q's can only work on a subset of devices.

My expectation was that VIOMMU would be 1:1 with physical iommu
instances, I think AMD needs this too??

Jason