Received-SPF: pass (google.com: domain of linux-kernel+bounces-33062-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Mon, 22 Jan 2024 09:01:52 -0400
From: Jason Gunthorpe <jgg@nvidia.com>
To: Nicolin Chen <nicolinc@nvidia.com>
Cc: "will@kernel.org" <will@kernel.org>,
	Robin Murphy <robin.murphy@arm.com>,
	"joro@8bytes.org" <joro@8bytes.org>,
	"jean-philippe@linaro.org" <jean-philippe@linaro.org>,
	Alistair Popple <apopple@nvidia.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-arm-kernel@lists.infradead.org" <linux-arm-kernel@lists.infradead.org>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>
Subject: Re: [PATCH 1/3] iommu/io-pgtable-arm: Add nents_per_pgtable in
 struct io_pgtable_cfg
Message-ID: <20240122130152.GP734935@nvidia.com>
References: <cover.1692693557.git.nicolinc@nvidia.com>
 <0fe68babdb3a07adf024ed471fead4e3eb7e703f.1692693557.git.nicolinc@nvidia.com>
 <f468b461-b203-5179-eb6d-9432b9911329@arm.com>
 <ZOTlcFs2NG6nJEPN@Asurada-Nvidia>
 <61f9b371-7c45-26b1-ec0f-600765280c89@arm.com>
 <ZO5R5i4n2WI2GnKQ@Asurada-Nvidia>
 <d234fa8d-d945-3f7f-1110-fe55bea88587@arm.com>
 <ZawmMd6BVqQpfbB5@Asurada-Nvidia>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZawmMd6BVqQpfbB5@Asurada-Nvidia>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?HdaB3srEul6WXc7T7hzTJKPxJ5iTjXfZ1wDz0gt8JzdAZpmwzhg1maQvDqm2?=
 =?us-ascii?Q?bzgfyuIpsGBfoZBxTM0TKRmgV5JmzkweojQ4bPyb/nQe3JDMTn1WByulShHo?=
 =?us-ascii?Q?0GMSFDwk8CP4CHkNOBW6yDQ/MVoML9ZtwA11vIv7zuvJLWhMH9GCo51ap7et?=
 =?us-ascii?Q?OszQGECGoGg+Bg9ptN0pXwo6f5jK0GeicmX5KK5RMAKH1oGng0iGR0ma4z7O?=
 =?us-ascii?Q?+TVYyh267wpuGB7bgO+pdrQpLD94u7P2jJbCpnbjcssJ8xChovbRANBRVUfD?=
 =?us-ascii?Q?Rx2i6VfUhYB2mUWYRs/A4vouDtMf1bXVP3LbZBjnSRJ+Hk7K1G8J8hMWgip0?=
 =?us-ascii?Q?Q4uuhCsNfpkCLnyVOp71HY3X94sBnbeKd8ULbqEgTcAwKFMK2OO9a0D15403?=
 =?us-ascii?Q?A5Zn1rnlQbPrgK6HZk9mmXoAuIHnBuNdFe3/V10PbdoMUocisOI0uRObkWIv?=
 =?us-ascii?Q?fGI+gG+j162ri1jc6YI5NWDsqp1pQjc6PsdtK2uuvStOv10uAK17baMgJo7/?=
 =?us-ascii?Q?7isfwGHFpr7LeAPm3WaEDjYGMDBG4heOb65wejLgkS+yJKKzKeT5gSh/V/x6?=
 =?us-ascii?Q?KADwfFuaX9zHwR6YnO3s1D4IZj0iG/q/ETEDCx9LoFtuCDv9fjgLs/34307/?=
 =?us-ascii?Q?eYqv424QGikqch9iAonZiA7A4ALJuIYQ64gFGr+99GO+4EXwwaG8yTTWc04u?=
 =?us-ascii?Q?HfbqYP4RzoZXTLtlXchNNZQ1koJN1iaV4XZwSpyzuUITF2FOYqLAwnds/2lE?=
 =?us-ascii?Q?5ImqTVrIgLJT8pD/hjZ1JpJMDrzck11msaZDAw7h+1guKIC8vu2y2e70//B4?=
 =?us-ascii?Q?88RLJuiR8kR1MAa17l1O1sPR9oC/f5UoTx+DDUVwvKU1uWnBv0zyhNwxl76t?=
 =?us-ascii?Q?r8nnLFQQj8hjVE4en/FrsmOf5vrOdhgKWqTP99jWtsWeAz5HR7d5MMyJjY2f?=
 =?us-ascii?Q?qGBKx6MguAbcDROmq+yLo8Vc5EUk89wlEawmxO3aGZThy4isaqUgvA/MEtam?=
 =?us-ascii?Q?0oWL6VN+96BeArKTgm08nBVBN+xXKFKumcfmAkafKPmgQSIw+fUCCKpfJ8oa?=
 =?us-ascii?Q?A4TcosjSdlV23kX2XMrvY0VbFQ1Di//buHekhshF85BKsLun7lTA2uuWlZ+T?=
 =?us-ascii?Q?PiknruP/2mXVW2Q+W1GbDL3Kt5SiFRqkkXUNjY+xDVz+W8akanVXKsM10XiL?=
 =?us-ascii?Q?mokk15sNlqqV8BRN+BT8TjTkmQjzSpkJy+ZRp8sr9IlYQlVaUeOYnH5CaAPY?=
 =?us-ascii?Q?TV8coYU18SPDfH1k4HCz6snWvv955BeDxS4kFOZ1Q/Lqh4cBrv0Id5x89wG5?=
 =?us-ascii?Q?6dOLDTuupThkHtE5R/0k+vK1OSTIX1A9Mjag0209S7Ck89joCiE+TbL3uzf0?=
 =?us-ascii?Q?SnDvHA/0+KXinhRZT4A+qy91zdMwDQIjNAbKYsh+Zr1cZtvjugGZmKRDTK5R?=
 =?us-ascii?Q?eoOW+35/Qqx3yXQ+mU3RTxQ4Bgk04bvBKN6Kgh4FL7f4B6et/oc4qytPWVKl?=
 =?us-ascii?Q?SoKeJ3VR/ePYsN0ahAL0leF2nh7288CnONVLGKunNQDcnxkjgKy87Wr1Ij1f?=
 =?us-ascii?Q?y3PvPn4K4f0fOgTwbs4=3D?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d160f608-e69d-401f-2aa7-08dc1b4a4ed2
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jan 2024 13:01:54.4910
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: OtO0bIfG56TzHwbJSLJ12HL76j6DTRpZ5ijpF9YGqYqMBcJVfJyRzX0pB2xJq40W
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR12MB4508

On Sat, Jan 20, 2024 at 11:59:45AM -0800, Nicolin Chen wrote:
> Hi Robin/Will,
> 
> On Tue, Aug 29, 2023 at 02:25:10PM -0700, Robin Murphy wrote:
> > > Also, what we need is actually an arbitrary number for max_tlbi_ops.
> > > And I think it could be irrelevant to the page size, i.e. either a
> > > 4K pgsize or a 64K pgsize could use the same max_tlbi_ops number,
> > > because what eventually impacts the latency is the number of loops
> > > of building/issuing commands.
> > 
> > Although there is perhaps a counter-argument for selective invalidation,
> > that if you're using 64K pages to improve TLB hit rates vs. 4K, then you
> > have more to lose from overinvalidation, so maybe a single threshold
> > isn't so appropriate for both.
> > 
> > Yes, ultimately it all comes down to picking an arbitrary number, but
> > the challenge is that we want to pick a *good* one, and ideally have
> > some reasoning behind it. As Will suggested, copying what the mm layer
> > does gives us an easy line of reasoning, even if it's just in the form
> > of passing the buck. And that's actually quite attractive, since
> > otherwise we'd then have to get into the question of what really is the
> > latency of building and issuing commands, since that clearly depends on
> > how fast the CPU is, and how fast the SMMU is, and how busy the SMMU is,
> > and how large the command queue is, and how many other CPUs are also
> > contending for the command queue... and very quickly it becomes hard to
> > believe that any simple constant can be good for all possible systems.
> 
> So, here we have another request to optimize this number further,
> though the merged arbitrary number copied from MMU side could fix
> the soft lockup. The iommu_unmap delay with a 500MB buffer is not
> quite satisfying on our testing chip, since the threshold now for
> max_tlbi_ops is at 512MB for 64K pgsize (8192 * 64KB).
> 
> As Robin remarked, this could be really a case-by-case situation.
> So, I wonder if we can rethink of adding a configurable threshold
> that has a default value at its current setup matching MMU side.
> 
> If this is acceptable, what can be the preferable way of having a
> configuration: a per-SMMU or a per-IOMMU-group sysfs node? I am
> open for any other option too.

Maybe it should be more dynamic and you get xx ms to push
invalidations otherwise it gives up and does invalidate all?

The busier the system the broader the invalidation?

Or do we need to measure at boot time invalidation performance and set
a threshold that way?

Also, it seems to me that SVA use cases and, say, DMA API cases are
somewhat different where we may be willing to wait longer for DMA API.

Jason