Received-SPF: pass (google.com: domain of linux-kernel+bounces-208975-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Mon, 10 Jun 2024 19:01:10 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: Nicolin Chen <nicolinc@nvidia.com>
Cc: "Tian, Kevin" <kevin.tian@intel.com>,
	"will@kernel.org" <will@kernel.org>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>,
	"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
	"joro@8bytes.org" <joro@8bytes.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"linux-arm-kernel@lists.infradead.org" <linux-arm-kernel@lists.infradead.org>,
	"linux-tegra@vger.kernel.org" <linux-tegra@vger.kernel.org>,
	"Liu, Yi L" <yi.l.liu@intel.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>,
	"vasant.hegde@amd.com" <vasant.hegde@amd.com>,
	"jon.grimm@amd.com" <jon.grimm@amd.com>,
	"santosh.shukla@amd.com" <santosh.shukla@amd.com>,
	"Dhaval.Giani@amd.com" <Dhaval.Giani@amd.com>,
	"shameerali.kolothum.thodi@huawei.com" <shameerali.kolothum.thodi@huawei.com>
Subject: Re: [PATCH RFCv1 08/14] iommufd: Add IOMMU_VIOMMU_SET_DEV_ID ioctl
Message-ID: <20240610220110.GQ19897@nvidia.com>
References: <ZluWXYQjroz0fqqn@nvidia.com>
 <Zl03rsgZl/msa3I+@nvidia.com>
 <20240606182423.GF19897@nvidia.com>
 <ZmIDqgfINXfB0i3L@Asurada-Nvidia>
 <20240607002707.GJ19897@nvidia.com>
 <BN9PR11MB527604637EC37B3B03FBBE468CFB2@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240607144917.GK19897@nvidia.com>
 <ZmN5WZkvDeIS7RRH@Asurada-Nvidia>
 <20240610120446.GP19897@nvidia.com>
 <ZmdbnGnp73yR36N5@Asurada-Nvidia>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZmdbnGnp73yR36N5@Asurada-Nvidia>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?MGXkUGmGN1ZhPjSx0bEFdr8N2Tlb6kG6oe2gRKUSBWKc3U0rn8D5ZwXGWZWF?=
 =?us-ascii?Q?bK2C7HufJ4lPQ0ElmObfgw+pYjf6UPDMhK251nZzv/ftgvEMtIvyyxJP3xIG?=
 =?us-ascii?Q?30z6/vnqV6wkArTOQ5aNZ/6AjtVw6CmS8Vk9L5AuhszmbKMIKPc9uxzJUCxn?=
 =?us-ascii?Q?vxs5//y+mPoVz9YR5EmQqY+dUpw66NjeJaG99I4WEHw5aJX38B/dvBSPlhNA?=
 =?us-ascii?Q?qsXlgV62B1jEZFsbEev3fX3SKYI9mNcJFGl5bzc4YZy85gF3nxqb8AxsWlMh?=
 =?us-ascii?Q?Vyqfp3oujLQfIUmG9eJaNa4FZxYqSwBjxk2/oalIGdQQpytRS44IeVaTcpk4?=
 =?us-ascii?Q?/CRBLWMa3n3lyFwCNGQNvZS/cMtaDDFbsEZM6mInSkAyIdBI3M9qxIktghA1?=
 =?us-ascii?Q?ynMPD4IQykYsIvKlGMfojdCf8+1KCo2VVzDwebHwEevNPbsWuXE4hQfdLiTc?=
 =?us-ascii?Q?DXMh8Ck7iViiY3RP2HheFRVkadUh7JXdFmZrN0w5yw2/Ps73v9Z4lVHuLrC1?=
 =?us-ascii?Q?2WOHWkuAySu1xpoqx0p2k7URbeVqggQciC3uBH+fcHUE2KVvz6a8jbms1PlU?=
 =?us-ascii?Q?7/HLxOntML4Bip7KSyuu8NoNsCZmcwxS28IfbNHmWRPrY75ZTqmWm8tbFhYU?=
 =?us-ascii?Q?dN6W8Ua2RoIA6ERD7iA25Wz3cOcoNQxkzAXQSRE57egJp4ORz5TNEj2ev/bl?=
 =?us-ascii?Q?oiCgxsuiLnlyu1kVpE7f+6XlUAYRkt/SMp5D8+50i9nSyF2J9En51BB2cqUR?=
 =?us-ascii?Q?VbMNH4AYu/zW9kTBFhr+0l83/MIl6BPAaz2tNw1LsR7jp3CKqtrTsRdQAHDM?=
 =?us-ascii?Q?Nnult1sqjvVGFSECrjPAuBJVjmlDkjni9u0b7aODzsYuAFXi02zB39GUsIo0?=
 =?us-ascii?Q?M+pW/m2/pD7JhFoyHW8tzhZZfxfIwi4UG2CGYUxjHKxBoe3mooE7+ygjaYHY?=
 =?us-ascii?Q?CGBHGYJD9i6uYW4gpbiEyZ98uBgXa4qoKkrZ3SWFAS2u8EOKumiLaFjCGwpA?=
 =?us-ascii?Q?0TaAG1Fjh8KrXXitiPA+KDk8wjAr06yjL/XfB/p2r/+0NqNnmJmBDB2/13lS?=
 =?us-ascii?Q?svy6/jUjCvryap3joFfowQyVAejpxW5899NNNFDP2dZh7YO9jApTdnvf1W/O?=
 =?us-ascii?Q?bMiJCcy1w/w+5A9FAEJG/QzTn/UiowoHY3TECaEQUGX2ZFxIa6fbVd+Agbkh?=
 =?us-ascii?Q?jx5s33Khm/gLlajSFK3S6jhFs4oJgD4EUaogOhaf0eVIMSktEKWkTnwqg4z0?=
 =?us-ascii?Q?LNXulkuROTx79PlqN7CtAezv2XdyM+XDmnUFzaSBFBFIe4tzFEi2/7Dn+o4C?=
 =?us-ascii?Q?ducjUF/Wyfid8pMKnkwSXMlIyZkXGNwfw1Pfw4aPyx8buI6FbN0y0wtBxwB6?=
 =?us-ascii?Q?bNabeOgK4ikf2NDJClGA160mETDMxP2Q9epMuRkx8GMgP+g5A/MeLgHxWhLp?=
 =?us-ascii?Q?aaEJpdkz9FXF8676+nUlX3IGAI7J+o7LyTGFYRUrmIDfUTI3XCo68TBKGIWX?=
 =?us-ascii?Q?n6Is9Nk8LKz/vy8020f4P8UlWcEMT75p16+Aj/1I2cz6MDogLjA+exvfi4+m?=
 =?us-ascii?Q?wVArFJTGVgUfKFdppdJMVgBbHHiU8Q0iOPwaj0Pg?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: caa1658c-6d95-453d-2db7-08dc8998d763
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jun 2024 22:01:12.2565
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: BzJwMHo5JDWZQMMu5BoRXR0mqH9hrdkdeefUqpsx/62lP+uuEjAzUBss5OhINfzJ
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB7549

On Mon, Jun 10, 2024 at 01:01:32PM -0700, Nicolin Chen wrote:
> On Mon, Jun 10, 2024 at 09:04:46AM -0300, Jason Gunthorpe wrote:
> > On Fri, Jun 07, 2024 at 02:19:21PM -0700, Nicolin Chen wrote:
> > 
> > > > IOTLB efficiency will suffer though when splitting 1p -> 2v while
> > > > invalidation performance will suffer when joining 2p -> 1v.
> > > 
> > > I think the invalidation efficiency is actually solvable. So,
> > > basically viommu_invalidate would receive a whole batch of cmds
> > > and dispatch them to different pSMMUs (nested_domains/devices).
> > > We already have a vdev_id table for devices, yet we just need a
> > > new vasid table for nested_domains. Right?
> > 
> > You can't know the ASID usage of the hypervisor from the VM, unless
> > you also inspect the CD table memory in the guest. That seems like
> > something we should try hard to avoid.
> 
> Actually, even now as we put a dispatcher in VMM, VMM still does
> decode the CD table to link ASID to s1_hwpt. Otherwise, it could
> only broadcast a TLBI cmd to all pSMMUs.

No, there should be no CD table decoding and no linking ASID to
anything by the VMM.

The ARM architecture is clean, the ASID can remain private to the VM,
there is no reason for the VMM to understand it.

The s1_hwpt is derived only from the vSTE and nothing more. It would
be fine for all the devices to have their own s1_hwpts with their own
vSTE's inside it.

> > > With that being said, it would make the kernel design a bit more
> > > complicated. And the VMM still has to separate the commands for
> > > passthrough devices (HW iotlb) from commands for emulated devices
> > > (emulated iotlb), unless we further split the topology at the VM
> > > level to have a dedicated vSMMU for all passthrough devices --
> > > then VMM could just forward its entire cmdq to the kernel without
> > > deciphering every command (likely?).
> > 
> > I would not include the emulated devices in a shared SMMU.. For the
> > same reason, we should try hard to avoid inspecting the page table
> > memory.
> 
> I wouldn't like the idea of attaching emulated devices to a shared
> vSMMU. Yet, mind elaborating why this would inspect the page table
> memory? Or do you mean we should avoid VMM inspecting user tables?

Emulated devices can't use the HW page table walker in the SMMU since
they won't get a clean CD linkage they can use. They have to manually
walk the page tables and convert them into an IOAS. It is a big PITA,
best to be avoided.

> > If a viommu is needed for emulated then virtio-iommu may be more
> > appropriate..
> > 
> > That said I'm sure someone will want to do this, so as long as it is
> > possible in the VMM, as slow as it may be, then it is fine.
> 
> Eric hasn't replied my previous query regarding how to design this,
> yet I guess the same. And looks like Intel is doing so for emulated
> devices, since there is only one intel_iommu instance in a VM.

Yes, Intel has long has this code to build VFIO containers from page
table walks.

Jason