Received-SPF: pass (google.com: domain of linux-kernel+bounces-209843-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Tue, 11 Jun 2024 09:17:56 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: Nicolin Chen <nicolinc@nvidia.com>
Cc: "Tian, Kevin" <kevin.tian@intel.com>,
	"will@kernel.org" <will@kernel.org>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>,
	"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
	"joro@8bytes.org" <joro@8bytes.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>,
	"linux-arm-kernel@lists.infradead.org" <linux-arm-kernel@lists.infradead.org>,
	"linux-tegra@vger.kernel.org" <linux-tegra@vger.kernel.org>,
	"Liu, Yi L" <yi.l.liu@intel.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>,
	"vasant.hegde@amd.com" <vasant.hegde@amd.com>,
	"jon.grimm@amd.com" <jon.grimm@amd.com>,
	"santosh.shukla@amd.com" <santosh.shukla@amd.com>,
	"Dhaval.Giani@amd.com" <Dhaval.Giani@amd.com>,
	"shameerali.kolothum.thodi@huawei.com" <shameerali.kolothum.thodi@huawei.com>
Subject: Re: [PATCH RFCv1 08/14] iommufd: Add IOMMU_VIOMMU_SET_DEV_ID ioctl
Message-ID: <20240611121756.GT19897@nvidia.com>
References: <20240607002707.GJ19897@nvidia.com>
 <BN9PR11MB527604637EC37B3B03FBBE468CFB2@BN9PR11MB5276.namprd11.prod.outlook.com>
 <20240607144917.GK19897@nvidia.com>
 <ZmN5WZkvDeIS7RRH@Asurada-Nvidia>
 <20240610120446.GP19897@nvidia.com>
 <ZmdbnGnp73yR36N5@Asurada-Nvidia>
 <20240610220110.GQ19897@nvidia.com>
 <ZmeGfqp26c3sGBmF@Asurada-Nvidia>
 <20240611002839.GS19897@nvidia.com>
 <Zmed4AtFjuF7+lWd@Asurada-Nvidia>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Zmed4AtFjuF7+lWd@Asurada-Nvidia>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?A5u/+IQ9LcwYjfzSFifxumpscmkGRcZUo23YuPawIP6pGlQET9qI2CGm+oe6?=
 =?us-ascii?Q?/BT6KkWhyDhNmjCsrFwor++MLvzVyNFxr6s7k+Wzk6SBoXKklnbR3O3Yl2nv?=
 =?us-ascii?Q?Ypi8Ar0QwrOy7G0I9EjJbg1yMNAkpxCee702/+fjNzI0j6aMzHDowfIYaIdh?=
 =?us-ascii?Q?2VsTo/SP/xu+blwaV6Q5bYUs5H7orKxa5hT174mgjKkxR+qk6jvz9myWlv9i?=
 =?us-ascii?Q?4CoykPo8pOa0pEmCORXFXj0QptkE7HbCTtZZCR253al+YGpV9pWq1Y79H1Hp?=
 =?us-ascii?Q?hth7+I0033MzfK6rn34Xh8TG40N/96kTOxSqk8c1YHnIPwBXDTmLMVOEZ7k3?=
 =?us-ascii?Q?X7OL8LW156aKHEvWS6cJlG8qsVpZaI7FLtrz/jnW8Kg8sOhY66TAxTNCdG2q?=
 =?us-ascii?Q?QvNeALODRwWt9rD2AFGYxePREIzapJBw6CujqHPEPO9AQpsqBFwVsT8vJCjY?=
 =?us-ascii?Q?fCsMR/Q/OZ26E0KYXwni5S51bpUNJJRjiThkR4DXimlBObP4tN8XxWzJikDZ?=
 =?us-ascii?Q?o40+2eI/GILxIwJPYBYOGjEPIM1yk8sTR2DmdaeSs7cRT76+fYGRiRz29wkZ?=
 =?us-ascii?Q?z9Har9lihk4LNaWdQ3IqpemXSYZehGv/14dBktFyxmsC2kHpijELL7rsk8RD?=
 =?us-ascii?Q?/7PfaC8YDAVCYabcPREZATIin3SytXMMy+TAhsMmoG5UPKTfWLT2QTpHDPlE?=
 =?us-ascii?Q?TLiOotBI/RNy71X8SrVKvwBg8IeFQ3STRN6yuQaWnGFhh9a+3stTaWT7Pqnl?=
 =?us-ascii?Q?IdHG1RB5cus5Vk/Z/QtyX/g6hiPlb5LcY86d8QI0uuUmp0KM8X4b9O6ujamu?=
 =?us-ascii?Q?SYuaJNCsN49EU3u8OieOOxXWsl0nFK+cofy4RqOfOYadx1ROJy+gAKYy2RUt?=
 =?us-ascii?Q?yqliOZCoukVa47W51pBVkfAkSjBgW7jpj5cRc2WK+T5fhkuZex9KwrDSM/nA?=
 =?us-ascii?Q?8VNFQNi2vXr/O+CULpVdOV+2PeTj9ChopolpugFvlsqq4Z/0n9QCM7ColECb?=
 =?us-ascii?Q?8bxIPbe3fx+1G9p0timDjqlhcdtuJb9y1vNe0r+5pbsSclCgn5AgtwjXckAa?=
 =?us-ascii?Q?tQfio4cJ3MtzI42BgHa0QKSgZSrxFLxdTpdAN2WW0Shb4xU71tLShOZb11Lm?=
 =?us-ascii?Q?PK/o54uZUsevApVISlUZ+Jbw0OUj7Mh3Ydyv4DPNv6JXY63wyBUWqaJc23zm?=
 =?us-ascii?Q?15ueCnTZ1Fajp1NxixEExzoerdplSWyVM4yRhvjEwvMsHb6Lj5ay/8k0lG+9?=
 =?us-ascii?Q?DJuiVjfxGcT4muK2AQC3qI9+W8jEqx7sjoRPA/0FL1+9luSWFY64vToKYm2z?=
 =?us-ascii?Q?3ftSELNewRso5gCybXYVYP16NRHWNVm/rhQ6s9b5IeHAshphQgdsvoDOhbKN?=
 =?us-ascii?Q?t2hSt0QrHhsnCkXMSHnyJ4PvaRSjbvkOnFu6QIj0CYGJZUNo8Q/96rK2QboG?=
 =?us-ascii?Q?iSKr8OOWf1ye1/qEravMv8zFRX0ZElxXoFrIe7jCQ53MADOSLUjKgcz4l6c5?=
 =?us-ascii?Q?7x5CAKNc6rXioO7gb72QCSQy0DuJlr5nUD1X6o3NNS3/SNyrnOBIaQtNtbIw?=
 =?us-ascii?Q?Nyfz/h9Qtk2CKTTJfJh0YzEf6fF1TyV/3x9BlWRI?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 895f63c2-d6c0-4876-1958-08dc8a1087d4
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Jun 2024 12:17:58.4193
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: pfUqYGA1Ly6++3H/P6Axji1Xzy8z6lw1SlKsQUWBU4rDS38YaHS+llXbJOcDoqGm
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB7156

On Mon, Jun 10, 2024 at 05:44:16PM -0700, Nicolin Chen wrote:
> On Mon, Jun 10, 2024 at 09:28:39PM -0300, Jason Gunthorpe wrote:
> > On Mon, Jun 10, 2024 at 04:04:30PM -0700, Nicolin Chen wrote:
> > 
> > > > > Actually, even now as we put a dispatcher in VMM, VMM still does
> > > > > decode the CD table to link ASID to s1_hwpt. Otherwise, it could
> > > > > only broadcast a TLBI cmd to all pSMMUs.
> > > > 
> > > > No, there should be no CD table decoding and no linking ASID to
> > > > anything by the VMM.
> > > > 
> > > > The ARM architecture is clean, the ASID can remain private to the VM,
> > > > there is no reason for the VMM to understand it.
> > > 
> > > But a guest-level TLBI command usually has only ASID available to
> > > know which pSMMU to dispatch the command. Without an ASID lookup
> > > table, how could VMM then dispatch a command to the corresponding
> > > pSMMU?
> > 
> > It can broadcast. The ARM architecture does not expect a N:1 mapping
> > of SMMUs. This is why I think it is not such a good idea..
> 
> Hmm, I thought we had an agreed idea that we shouldn't broadcast
> a TLBI (except global NH_ALL/VAA) for invalidation performance?

I wouldn't say agree, there are just lots of different trade offs to
be made here. You can reduce broadcast by parsing the CD table from
the VMM. You can reduce broadcast with multiple vSMMUs.

VMM needs to pick a design. I favour multiple vSMMUs.

> CD table walkthrough would be always done only by VMM, while the
> lookup table could be created/maintained by the kernel. I feel a
> vasid table could make sense since we maintain the vdev_id table
> in the kernel space too.

I'm not convinced we should put such a micro optimization in the
kernel. If the VMM cares about performance then split the vSMMU,
otherwise lets just put all the mess in the VMM and give it the tools
to manage the invalidation distribution.

In the long run things like vCMDQ are going to force the choice to
multi-smmu, which is why I don't see too much value with investing in
optimizing the single vSMMU case. The optimization can be done later
if someone does have a use case.

Jason