Received-SPF: pass (google.com: domain of linux-kernel+bounces-38690-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Thu, 25 Jan 2024 09:55:37 -0400
From: Jason Gunthorpe <jgg@nvidia.com>
To: Nicolin Chen <nicolinc@nvidia.com>
Cc: "will@kernel.org" <will@kernel.org>,
	Robin Murphy <robin.murphy@arm.com>,
	"joro@8bytes.org" <joro@8bytes.org>,
	"jean-philippe@linaro.org" <jean-philippe@linaro.org>,
	Alistair Popple <apopple@nvidia.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-arm-kernel@lists.infradead.org" <linux-arm-kernel@lists.infradead.org>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>
Subject: Re: [PATCH 1/3] iommu/io-pgtable-arm: Add nents_per_pgtable in
 struct io_pgtable_cfg
Message-ID: <20240125135537.GP1455070@nvidia.com>
References: <f468b461-b203-5179-eb6d-9432b9911329@arm.com>
 <ZOTlcFs2NG6nJEPN@Asurada-Nvidia>
 <61f9b371-7c45-26b1-ec0f-600765280c89@arm.com>
 <ZO5R5i4n2WI2GnKQ@Asurada-Nvidia>
 <d234fa8d-d945-3f7f-1110-fe55bea88587@arm.com>
 <ZawmMd6BVqQpfbB5@Asurada-Nvidia>
 <20240122130152.GP734935@nvidia.com>
 <Za6kuE4KUPuwk7+j@Asurada-Nvidia>
 <20240122175700.GB1455070@nvidia.com>
 <Za63HOMZE2fuJKQ4@Asurada-Nvidia>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <Za63HOMZE2fuJKQ4@Asurada-Nvidia>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?KrjPJnvyAFvGoBxEyXIemvQJ5hGVecH2JaJWFflX5KMkCapxFm0bR61eRwpj?=
 =?us-ascii?Q?gsoBxfTPmIadu0LysZYg6XyKYGVK7J2wMlu8Mi6XVcKg/6w7wj7TYI9gSv68?=
 =?us-ascii?Q?ynbulqEMsEuL9k64wWZe+RQdkvBUcTpwMj07M150d2TMfqFRk99ANl4UnA7I?=
 =?us-ascii?Q?UNdgRF5L6pS0aDaXLx+Zk4/wtO15sLqxlJJ8sHfWtEL4g0z6UbawRqChyH3m?=
 =?us-ascii?Q?qd6ycUOsfWZZJm5FytiuwhXlNGjmZ5lkbWTticqGrVu1mYUyJevDmV9LwC0P?=
 =?us-ascii?Q?/sNUs7o2oIEg+O883aUWi1j/Lk3IFsB0BnEV8DJ9oAZyjfronCYsWaCZnqsL?=
 =?us-ascii?Q?BNiO0VDFAaK8oo3bdI33huzcXLmYEA+n6JLOwDdpKBu/SQzf26kMefgom+UC?=
 =?us-ascii?Q?oH+4j5yYHv1oNoinlNyMdjxUiS+0FVEEqQfsUWes43rQKxtfOkDeo2VE1ZTE?=
 =?us-ascii?Q?dP3tYU5iZVUwQkaNlbm9pVROjvqpXBvyh0DzkY5YjQwssbEgT2CaFeLieN7L?=
 =?us-ascii?Q?JulKYBuZvEfuhVFzTXYEGayxov6+DwwtGydqUg5VcawibicyQhG+K/aFTFwY?=
 =?us-ascii?Q?GbcVasVVMi/WPAp74NacB3b6sOVSmVFxCmw+8dvBCcnSaJIXOgjKXYx/P/JE?=
 =?us-ascii?Q?KmopHU5BR3KVKWjTneFcsAU8Y/kzxD9ioPbOU5PdezhKiSj4eo+38XDcNeEB?=
 =?us-ascii?Q?nH2G04tcU45nSsarRkgqbA0O4EWubKKMp+vUV78dUna00YD4ZN0wSQ5OGhJ2?=
 =?us-ascii?Q?Gkq9bJW5qqDoOXBy9WmYap6VbAbXH4xRRZF9V1mb1UqsetO46R0SannWx06k?=
 =?us-ascii?Q?HLOVbG7cRM4IwGIxPc4bF7uoC5/LpHmQrvBWxR+1Bl0NrNcdi7OHUMqU7jk3?=
 =?us-ascii?Q?cEWtx8hal8QOBS12Nr+A8U0NIWCW6dkzIgA+nCEv4mUir7Q36S7fy17ybzRA?=
 =?us-ascii?Q?l2+lAOzG2JGR3anHawHH8CbCadNNicsdwJhRjlX9pA2RFDLdgQIak49YbMya?=
 =?us-ascii?Q?QcfIrg9KPE41z2Tacv49lzRcg7WIxFu9jLIO7fuRciwcU2xCwM4S6qshzdPZ?=
 =?us-ascii?Q?oZoMnlxOuIT/gq73QrKkmL8G3YZryBuDsHbMiAdbY1IC2o4qA2ylqO8kFX58?=
 =?us-ascii?Q?vcIjNrDAiaePoQqOAJpIA7SBHclUxue705Ey8iccn+Thq9xG26+njaLLDL+R?=
 =?us-ascii?Q?FDXU1YQd0Zv6u56ZOBGmqiO2H/+7k4UJXalgAulju3prt9IbYOovp+5uEamI?=
 =?us-ascii?Q?lws6xMk+5N23MrH8lvDHcZI58gegWcVZNgOpq6IC/0ldWpa3gEeEBQcGukHa?=
 =?us-ascii?Q?3y1AVz8+GXTyApvmfAt7ifPVDIOaIhvglERX+jhk/APgd7TotpK8aIsOrOG2?=
 =?us-ascii?Q?bgFgTwvA4wk1Z2jz1b5auy4pgVr/jxkkpQADeS6a3sng8WOiE/BQo2to8Qxx?=
 =?us-ascii?Q?wgC391Kb5TAw6jURsR+FbvIyzT1BNpYe77RM7yRUE9AMZlyN42YtMSiOF7Y1?=
 =?us-ascii?Q?ro2sgCYe8/UhEbQtgL/EI6idZv/rtwA10+PbKF/FYXhjpSmWcNP4bGzVLLbo?=
 =?us-ascii?Q?WA++TJMwTBv3DdEasyUKAtbniqCBnzzGGhNAFM+V?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a7ea80bd-19d2-45bb-6f04-08dc1dad4f87
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jan 2024 13:55:38.1760
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: QWaymZCoRuMY9cqpXzW3AAzpL2RF/EF/R9MioZCw1XPeXW8+dwtD1cUQ/6jD2M36
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB8085

On Tue, Jan 23, 2024 at 04:11:09PM -0800, Nicolin Chen wrote:
> > prevented strongly. Broadly speaking if SVA is pushing too high an
> > invalidation workload then we need to agressively trim it, and do so
> > dynamically. Certainly we should not have a tunable that has to be set
> > right to avoid soft lockup.
> > 
> > A tunable to improve performance, perhaps, but not to achieve basic
> > correctness.
> 
> So, should we make an optional tunable only for those who care
> about performance? Though I think having a tunable would just
> fix both issues.

When the soft lockup issue is solved you can consider if a tunable is
still interesting..
 
> > Maybe it is really just a simple thing - compute how many invalidation
> > commands are needed, if they don't all fit in the current queue space,
> > then do an invalidate all instead?
> 
> The queue could actually have a large space. But one large-size
> invalidation would be divided into batches that have to execute
> back-to-back. And the batch size is 64 commands in 64-bit case,
> which might be too small as a cap.

Yes, some notable code reorganizing would be needed to implement
something like this

Broadly I'd sketch sort of:

 - Figure out how fast the HW can execute a lot of commands
 - The above should drive some XX maximum number of commands, maybe we
   need to measure at boot, IDK
 - Strongly time bound SVA invalidation:
   * No more than XX commands, if more needed then push invalidate
     all
   * All commands must fit in the available queue space, if more
     needed then push invalidate all
 - The total queue depth must not be larger than YY based on the
   retire rate so that even a full queue will complete invalidation
   below the target time.

A tunable indicating what the SVA time bound target should be might be
appropriate..

Jason