Received-SPF: pass (google.com: domain of linux-kernel+bounces-39053-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Date: Thu, 25 Jan 2024 13:47:28 -0400
From: Jason Gunthorpe <jgg@nvidia.com>
To: Nicolin Chen <nicolinc@nvidia.com>
Cc: "will@kernel.org" <will@kernel.org>,
	Robin Murphy <robin.murphy@arm.com>,
	"joro@8bytes.org" <joro@8bytes.org>,
	"jean-philippe@linaro.org" <jean-philippe@linaro.org>,
	Alistair Popple <apopple@nvidia.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-arm-kernel@lists.infradead.org" <linux-arm-kernel@lists.infradead.org>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>
Subject: Re: [PATCH 1/3] iommu/io-pgtable-arm: Add nents_per_pgtable in
 struct io_pgtable_cfg
Message-ID: <20240125174728.GR1455070@nvidia.com>
References: <61f9b371-7c45-26b1-ec0f-600765280c89@arm.com>
 <ZO5R5i4n2WI2GnKQ@Asurada-Nvidia>
 <d234fa8d-d945-3f7f-1110-fe55bea88587@arm.com>
 <ZawmMd6BVqQpfbB5@Asurada-Nvidia>
 <20240122130152.GP734935@nvidia.com>
 <Za6kuE4KUPuwk7+j@Asurada-Nvidia>
 <20240122175700.GB1455070@nvidia.com>
 <Za63HOMZE2fuJKQ4@Asurada-Nvidia>
 <20240125135537.GP1455070@nvidia.com>
 <ZbKY9BmXZiHPiEOk@Asurada-Nvidia>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZbKY9BmXZiHPiEOk@Asurada-Nvidia>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?7MMPUIiA7mjh2rPTRFBqx+5qjVHHF/bn3ei8n+ZTaAum9y221s1NnCSK+ueQ?=
 =?us-ascii?Q?u7RBxBjGnJiA8FHZoCQAZ5BRQ99s9lZWInezn1x5mxGycps6slJtA/yH4aFo?=
 =?us-ascii?Q?2Q1Q3TKnN2uIafZbN16bTUNqWhFnriY857v/NNZ0GGhjEDb84kO253twSuky?=
 =?us-ascii?Q?/W+0v5Bo16gkUhdswAy9xAp433KWCMbVZ1itaU4MyRn8A5HshameXskwmdAe?=
 =?us-ascii?Q?m1QKTNj7lsW0w99xSGvwY+vTWzS+av7PgrhenMZphfztxiI8zo/tvBuVcX0g?=
 =?us-ascii?Q?870xN/2/snP1sjo6kW4q2XS4+0+YXkk1cHLEyiIIQDKa5BySQydkeHj3obem?=
 =?us-ascii?Q?alJCByljrtim5Yd+Y+lB+RvzqlBGi8erNyaWYEs7udrrl+TuiffanLayaVWG?=
 =?us-ascii?Q?HrqEmXD7nHI4MuTJkeVQysRkxukpuvQcT7hWSz63CVKFbFgEG3PUsJWTWlpW?=
 =?us-ascii?Q?5s/xZnmqLhD5NiX+ox+9kf8omiqiXi/qgUdpWMvnB9veCX5PTlDwtRwfsdSG?=
 =?us-ascii?Q?N5C+ttg4SMR/YPbe3+e+lJdAy4YJb+83+q+ZLbrv1mspQLN631nPEz19M2lo?=
 =?us-ascii?Q?Nus6bh/O/WIgqtMIU9DCSE3DfJS8Ge6XJ5QaR7zsEFvWqgwydSS26EP9czwV?=
 =?us-ascii?Q?W3DauLL1f1C19JSCgLivvNbwga2oWfMYAjQvfXqitiHDuKzkAuyTwE46Pmw/?=
 =?us-ascii?Q?P+qnbvlX6l60LCksG72VPagBWP1sg5+fYjnpRYm0O2K+tuwH/P5q0eVh+YzJ?=
 =?us-ascii?Q?1qePUJdZjYFSRW8Pd1Q6GJVWeDPfpMwGRt+ESxnuRlMsSi8qVAtkhxNfDHVi?=
 =?us-ascii?Q?AIK4cMvEkKL54XkTBSXDsgtndtPLtKjgxU1t9ffzTSHJA26ubT8TNzUH8Fur?=
 =?us-ascii?Q?TAjYa7PLl48l93IVrG4ywyc/zQQfW8RVcHD+C05i0pPHFe3Rk5Lk+QbPZyX1?=
 =?us-ascii?Q?TeMBG3z/f0014emFCoLb/BX1HsRmEC0I4v9gtB5BigVSH+aLwsPzqtuBegVn?=
 =?us-ascii?Q?vdrRBLlfKDjUXfZ3NBabX4YlQpL8SZz6KCwPBza0ZEzHPF//ZEWqqDafABkn?=
 =?us-ascii?Q?k17QydQytug78QqyA2cgKG+ppEK2WB2Od7NPVmaOBjK5lrjSSSP36t6rHPnS?=
 =?us-ascii?Q?ZmAtB10W+W2Ps1/IfpUA9yRsaL3oMPeRjB6gNTuKRKGM/On/Jk0QmIAnEe5m?=
 =?us-ascii?Q?1oXzzFSMWXuaG/1Q698969X+EtW2rU9rBb7n98X5y7agodjs7HYuVObZy4Hr?=
 =?us-ascii?Q?Ot7rxXHDuOMRjYtST/diciUAQGffqzDnj1f27O8TBNYtYUBSgJRAOWVfxCV3?=
 =?us-ascii?Q?ptmtIdtVKdBK4CFEunxc74hArM1lE2FvpM8ySeOMxsoZ/UPGbFosXsjDSIlC?=
 =?us-ascii?Q?Sc+sTS0gzmG2Iaxr5dnYIz9gfo6uVvpEqgVoDetyzS7Q8HMKdG7lD34hxDCe?=
 =?us-ascii?Q?On59xOvb0DnIQz/Gcqlqho3PnhF1OgD+3vyI/XSSy44ZW7DJ3bUkl4Ehh6qU?=
 =?us-ascii?Q?OswmtsMmdJjHYIt/QKoQ06nXXhTbJnY7LLLjzpvzTa3s9U9wnigAcWWbftSv?=
 =?us-ascii?Q?fau3sj07xNEH0MfJm6vD9nCY9HNSNCdQN0SDk+Zd?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 735c78b4-4914-4631-d6a2-08dc1dcdb402
X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jan 2024 17:47:30.6601
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: a4MEB4jMfa04Hs+4+BFaWFG7/gPyFxsHuHJNGQUqucgZgGJKNkF6WzHzQGHUZmGa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB8132

On Thu, Jan 25, 2024 at 09:23:00AM -0800, Nicolin Chen wrote:
> > When the soft lockup issue is solved you can consider if a tunable is
> > still interesting..
> 
> Yea, it would be on top of the soft lockup fix. I assume we are
> still going with your change: arm_smmu_inv_range_too_big, though
> I wonder if we should apply before your rework series to make it
> a bug fix..

It depends what change you settle on..

> > > > Maybe it is really just a simple thing - compute how many invalidation
> > > > commands are needed, if they don't all fit in the current queue space,
> > > > then do an invalidate all instead?
> > > 
> > > The queue could actually have a large space. But one large-size
> > > invalidation would be divided into batches that have to execute
> > > back-to-back. And the batch size is 64 commands in 64-bit case,
> > > which might be too small as a cap.
> > 
> > Yes, some notable code reorganizing would be needed to implement
> > something like this
> > 
> > Broadly I'd sketch sort of:
> > 
> >  - Figure out how fast the HW can execute a lot of commands
> >  - The above should drive some XX maximum number of commands, maybe we
> >    need to measure at boot, IDK
> >  - Strongly time bound SVA invalidation:
> >    * No more than XX commands, if more needed then push invalidate
> >      all
> >    * All commands must fit in the available queue space, if more
> >      needed then push invalidate all
> >  - The total queue depth must not be larger than YY based on the
> >    retire rate so that even a full queue will complete invalidation
> >    below the target time.
> > 
> > A tunable indicating what the SVA time bound target should be might be
> > appropriate..
> 
> Thanks for listing it out. I will draft something with that, and
> should we just confine it to SVA or non DMA callers in general?

Also, how much of this SVA issue is multithreaded? Will multiple
command queues improve anything?

Jason