Received-SPF: pass (google.com: domain of linux-kernel+bounces-33879-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Mon, 22 Jan 2024 13:57:00 -0400
From: Jason Gunthorpe <jgg@nvidia.com>
To: Nicolin Chen <nicolinc@nvidia.com>
Cc: "will@kernel.org" <will@kernel.org>,
	Robin Murphy <robin.murphy@arm.com>,
	"joro@8bytes.org" <joro@8bytes.org>,
	"jean-philippe@linaro.org" <jean-philippe@linaro.org>,
	Alistair Popple <apopple@nvidia.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-arm-kernel@lists.infradead.org" <linux-arm-kernel@lists.infradead.org>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>
Subject: Re: [PATCH 1/3] iommu/io-pgtable-arm: Add nents_per_pgtable in
 struct io_pgtable_cfg
Message-ID: <20240122175700.GB1455070@nvidia.com>
References: <cover.1692693557.git.nicolinc@nvidia.com>
 <0fe68babdb3a07adf024ed471fead4e3eb7e703f.1692693557.git.nicolinc@nvidia.com>
 <f468b461-b203-5179-eb6d-9432b9911329@arm.com>
 <ZOTlcFs2NG6nJEPN@Asurada-Nvidia>
 <61f9b371-7c45-26b1-ec0f-600765280c89@arm.com>
 <ZO5R5i4n2WI2GnKQ@Asurada-Nvidia>
 <d234fa8d-d945-3f7f-1110-fe55bea88587@arm.com>
 <ZawmMd6BVqQpfbB5@Asurada-Nvidia>
 <20240122130152.GP734935@nvidia.com>
 <Za6kuE4KUPuwk7+j@Asurada-Nvidia>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Za6kuE4KUPuwk7+j@Asurada-Nvidia>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?LdVAc6CCL2dDI3fKOVSLYvm5k9a7FdahbZS3AgMeZY2upPn6tC39tIGYAS+j?=
 =?us-ascii?Q?qAMWmmpE8c1gJc/FJvDDj1MgDUUNi+sAR4st8dY2tynkRrGiDt/PJ8lND3uK?=
 =?us-ascii?Q?ciphfCgPJ0OKfYnz7gaP9leoPkntZWzga04OmxnYvw1cHuc1vpfj5C8uOXf5?=
 =?us-ascii?Q?qxbfgRcpZN84DCLydFmCCRIJLExQUF7KGTAzkKoWwjZIXO3+XQm6ovcB3RKv?=
 =?us-ascii?Q?x+bgPGUYFztZ9YOwqOU2TnWeEKkzZ1p64gEwQo6UyxbYljRZ2Bx14ETQOJHP?=
 =?us-ascii?Q?nKWgNB2/wjXc5Fv0bKQ1ub5qbDb8nKLem51uArmCBP/QsBA8rSJiAYBvVofc?=
 =?us-ascii?Q?8Zh9cCn7xDPpN42E1aJK1tQf7jDETcifDEHFdW0L9D5wpmA6EjUB8pZB53Zk?=
 =?us-ascii?Q?bt5TBGnMOw35T2HCgHozQS7pGhWemu8j3uS+jdiGECSxBkBn04hm9dUTlObD?=
 =?us-ascii?Q?DlKhyPL7sRmWU3sMDfZmhH0TlttlcbrT/UO9FPXvIJq1fuxbv8ql0MGzQTUg?=
 =?us-ascii?Q?NdhM5xJRmMd9c+OJIqRMApVM1PHvE+vh0f3IQeAbXldTBAHs9aVh72Ar+Qmo?=
 =?us-ascii?Q?aWY3hsYiy5hec6FfSpWEcqWlMfDbborzCCm5HC15beE1v1f30Y8J9nNf3r46?=
 =?us-ascii?Q?GrCjcBeOosO8Z6EPEim9fGIlk1/QoSmPQyLvVySfQqMk7tc5/NwTweCWwpK7?=
 =?us-ascii?Q?hy1B8d89KltJ4R3V+7tiZPn6jKwlhC0CnE4DQX/3rI7J+ZrY/DlA1wKT4BCw?=
 =?us-ascii?Q?pQqfmU3dI+heTgIEvHXrnmSVxlUVyqxqZLPLEgtp7ESbr5cxxXK3LiZAT04j?=
 =?us-ascii?Q?ri/zkjSfFoS0OMI8IDm28mJOJnU9Vi+jm2/NHKuaAKxvhYk1FurqNz4wPI4Y?=
 =?us-ascii?Q?w3ABn7Z7vEpa/YA6HNOjZN+Writlqvr3Z6ejLqYo62pT7WTfnKxKatdUlKLc?=
 =?us-ascii?Q?mByoXM//XR/Y3z8H3xYE8b40YlU9ijvr3EbC9rO3q2Od41BtlXMquPsqxFFD?=
 =?us-ascii?Q?OJOLSJTUBBmSVgku4fuPfEDJgoYF2zpimxriw2/PYzSBcnlJJuEirGl3NS3E?=
 =?us-ascii?Q?d/A0Ci1CV1LScQAmVnwT9xtOjZRmH0DDdbGCRYaMCKEE4S/ANb32SbQNhMay?=
 =?us-ascii?Q?DLYJo3v6GNJuUsG7UHC3hetq2WvDwWdw/EzJjkeEnKL90Cca6Ch6TJ+AgygU?=
 =?us-ascii?Q?rUv8rETu1riWJztauPFaxy4AJNeXLI4rft9kB6OmFTjSuLBQDbLkNeh36XAl?=
 =?us-ascii?Q?PBmtZMEfpA3CSio108cZngpyPQC68G0t0wmDJvlE7sBLT0CrfrES0IiVIW7p?=
 =?us-ascii?Q?m49kmqNcnDAxBNfarjRtWKBTq7QO9ROxYKp+hsjuScm22/5nh53oEAWKcWBg?=
 =?us-ascii?Q?vQDKOVeWdRl5iAhEIT/OD1NomT2ulyFjfxBg/SGzMiuLaYshqGPmgdN4UvK/?=
 =?us-ascii?Q?evjFUB4KSlsEKzQXuM94NUv/mm+RhGd06v8qX8mVAxG9aLl/NFsk1KM0aRoJ?=
 =?us-ascii?Q?x74Hu/EcRJa6/QjDsSu/4FPIJZSjmQQUhuq74N6tItQnvgwByvMlm4w62ntZ?=
 =?us-ascii?Q?TO8dq32sTD541mSm+mnV3Vcoic1rkReOjvJbGGLx?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2a767ad6-7c3f-492c-b9f6-08dc1b7389b5
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jan 2024 17:57:02.6502
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: s/UNgAaYmS/jHeT+NyjlA7/0i2cl8dz2lgSDz4yJP4b4gPtEWd4awTLPwHpdsFhk
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB8025

On Mon, Jan 22, 2024 at 09:24:08AM -0800, Nicolin Chen wrote:
> > Or do we need to measure at boot time invalidation performance and set
> > a threshold that way?
> 
> I see. We can run an invalidation at default max_tlbi_ops to
> get its delay in msec or usec, and then set as the threshold
> "xx ms" in the idea one.
> 
> > Also, it seems to me that SVA use cases and, say, DMA API cases are
> > somewhat different where we may be willing to wait longer for DMA API.
> 
> Hmm, the lockup that my patch fixed was for an SVA case that
> doesn't seem to involve DMA API:
> https://lore.kernel.org/linux-iommu/20230901203904.4073-1-nicolinc@nvidia.com/
> 
> And the other lockup fix for a non-SVA case from Zhang doesn't
> seem to involve DMA API either:
> https://lore.kernel.org/linux-iommu/e74ea905-d107-4202-97ca-c2c509e7aa1e@huawei.com/
> 
> Maybe we can treat DMA API a bit different. But I am not sure
> about the justification of leaving it to wait longer. Mind
> elaborating?

Well, there are two issues.. The first is the soft lockup, that should
just be reliably prevented. The timer, for instance, is a reasonable
stab at making that universally safe.

Then there is the issue of just raw invalidation performance, where
SVA particularly is linked to the mm and the longer invalidation takes
the slower the apps will be. We don't have any idea where future DMA
might hit the cache, so it is hard to know if all invalidation is not
the right thing..

DMA api is often lazy and the active DMA is a bit more predictable, so
perhaps there is more merit in spending more time to narrow the
invalidation.

The other case was vfio unmap for VM tear down, which ideally would
use whole ASID invalidation.

If your issue is softlockup, not performance, then that should be
prevented strongly. Broadly speaking if SVA is pushing too high an
invalidation workload then we need to agressively trim it, and do so
dynamically. Certainly we should not have a tunable that has to be set
right to avoid soft lockup.

A tunable to improve performance, perhaps, but not to achieve basic
correctness.

Maybe it is really just a simple thing - compute how many invalidation
commands are needed, if they don't all fit in the current queue space,
then do an invalidate all instead?

Jason