Received-SPF: pass (google.com: domain of linux-kernel+bounces-210490-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.118.233 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C
Date: Tue, 11 Jun 2024 12:11:44 -0700
From: Nicolin Chen <nicolinc@nvidia.com>
To: Jason Gunthorpe <jgg@nvidia.com>
CC: "Tian, Kevin" <kevin.tian@intel.com>, "will@kernel.org" <will@kernel.org>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>,
	"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
	"joro@8bytes.org" <joro@8bytes.org>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "iommu@lists.linux.dev"
	<iommu@lists.linux.dev>, "linux-arm-kernel@lists.infradead.org"
	<linux-arm-kernel@lists.infradead.org>, "linux-tegra@vger.kernel.org"
	<linux-tegra@vger.kernel.org>, "Liu, Yi L" <yi.l.liu@intel.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>, "vasant.hegde@amd.com"
	<vasant.hegde@amd.com>, "jon.grimm@amd.com" <jon.grimm@amd.com>,
	"santosh.shukla@amd.com" <santosh.shukla@amd.com>, "Dhaval.Giani@amd.com"
	<Dhaval.Giani@amd.com>, "shameerali.kolothum.thodi@huawei.com"
	<shameerali.kolothum.thodi@huawei.com>
Subject: Re: [PATCH RFCv1 08/14] iommufd: Add IOMMU_VIOMMU_SET_DEV_ID ioctl
Message-ID: <ZmihcEA+7rU6Yt8x@Asurada-Nvidia>
References: <BN9PR11MB527604637EC37B3B03FBBE468CFB2@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240607144917.GK19897@nvidia.com>
 <ZmN5WZkvDeIS7RRH@Asurada-Nvidia>
 <20240610120446.GP19897@nvidia.com>
 <ZmdbnGnp73yR36N5@Asurada-Nvidia>
 <20240610220110.GQ19897@nvidia.com>
 <ZmeGfqp26c3sGBmF@Asurada-Nvidia>
 <20240611002839.GS19897@nvidia.com>
 <Zmed4AtFjuF7+lWd@Asurada-Nvidia>
 <20240611121756.GT19897@nvidia.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20240611121756.GT19897@nvidia.com>
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Jun 2024 19:12:01.3510
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 06349bdf-2ee8-4f00-7baa-08dc8a4a5f88
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource:
	DS3PEPF000099DE.namprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB5883

On Tue, Jun 11, 2024 at 09:17:56AM -0300, Jason Gunthorpe wrote:
> On Mon, Jun 10, 2024 at 05:44:16PM -0700, Nicolin Chen wrote:
> > On Mon, Jun 10, 2024 at 09:28:39PM -0300, Jason Gunthorpe wrote:
> > > On Mon, Jun 10, 2024 at 04:04:30PM -0700, Nicolin Chen wrote:
> > > 
> > > > > > Actually, even now as we put a dispatcher in VMM, VMM still does
> > > > > > decode the CD table to link ASID to s1_hwpt. Otherwise, it could
> > > > > > only broadcast a TLBI cmd to all pSMMUs.
> > > > > 
> > > > > No, there should be no CD table decoding and no linking ASID to
> > > > > anything by the VMM.
> > > > > 
> > > > > The ARM architecture is clean, the ASID can remain private to the VM,
> > > > > there is no reason for the VMM to understand it.
> > > > 
> > > > But a guest-level TLBI command usually has only ASID available to
> > > > know which pSMMU to dispatch the command. Without an ASID lookup
> > > > table, how could VMM then dispatch a command to the corresponding
> > > > pSMMU?
> > > 
> > > It can broadcast. The ARM architecture does not expect a N:1 mapping
> > > of SMMUs. This is why I think it is not such a good idea..
> > 
> > Hmm, I thought we had an agreed idea that we shouldn't broadcast
> > a TLBI (except global NH_ALL/VAA) for invalidation performance?
> 
> I wouldn't say agree, there are just lots of different trade offs to
> be made here. You can reduce broadcast by parsing the CD table from
> the VMM. You can reduce broadcast with multiple vSMMUs.
> 
> VMM needs to pick a design. I favour multiple vSMMUs.

Yea, having multiple vSMMUs for nesting too seems to be a cleaner
design. The thing is that we have to put a certain complexity in
the VMM, and it should be more efficient by having it at the boot
stage (creating multi-vSMMUs/PCIs and IORT nodes) v.s. runtime
(trappings and distributing at every command).

> > CD table walkthrough would be always done only by VMM, while the
> > lookup table could be created/maintained by the kernel. I feel a
> > vasid table could make sense since we maintain the vdev_id table
> > in the kernel space too.
> 
> I'm not convinced we should put such a micro optimization in the
> kernel. If the VMM cares about performance then split the vSMMU,
> otherwise lets just put all the mess in the VMM and give it the tools
> to manage the invalidation distribution.
> 
> In the long run things like vCMDQ are going to force the choice to
> multi-smmu, which is why I don't see too much value with investing in
> optimizing the single vSMMU case. The optimization can be done later
> if someone does have a use case.

I see. That makes sense to me.

Nicolin