Received-SPF: pass (google.com: domain of linux-kernel+bounces-166719-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Thu, 2 May 2024 13:25:30 -0300
From: Jason Gunthorpe <jgg@nvidia.com>
To: Tanmay Jagdale <tanmay@marvell.com>
Cc: will@kernel.org, robin.murphy@arm.com, joro@8bytes.org,
	nicolinc@nvidia.com, mshavit@google.com, baolu.lu@linux.intel.com,
	thunder.leizhen@huawei.com, set_pte_at@outlook.com,
	smostafa@google.com, sgoutham@marvell.com, gcherian@marvell.com,
	jcm@jonmasters.org, linux-arm-kernel@lists.infradead.org,
	iommu@lists.linux.dev, linux-kernel@vger.kernel.org
Subject: Re: [PATCH V3 0/2] iommu/arm-smmu-v3: Add support for ECMDQ register
 mode
Message-ID: <20240502162530.GA578685@nvidia.com>
References: <20240425144152.52352-1-tanmay@marvell.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240425144152.52352-1-tanmay@marvell.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?AyQrBS7SMQIM72zGjcitWFNVFLpBcMSWTbcESVIcFUxi17yNdTr1NbgbCbIE?=
 =?us-ascii?Q?kjXb0DFFcp2TozpU4nQWiUOAx6GwG7Bgm8FmYp6i8SzHmXeKj5aVJF9xF7vf?=
 =?us-ascii?Q?jmI78W59EeXHbDlFLxrok2D3/S8guMfj3EZE5Ff7pzI2RsK8zj3+uWJT+5+u?=
 =?us-ascii?Q?hKJBm7sEYlamdsbtaKlQZ4uIlwwVEd9y14rcSq8n5uz8JmveBWvP40ql3V9u?=
 =?us-ascii?Q?Jhm1isHbc7e0TXymIydu/XcJB5sCc1h7Bkg2BCxjTFRBS439Tt0MjqR7ewGo?=
 =?us-ascii?Q?5se1FvEVtuUiauXd9NmEaWcwwrjgQKVKm/C8LidW2MALZQgta4JHcZtkTTDw?=
 =?us-ascii?Q?paD9D2YtwoHIFcIbx/Ef5RdxzB0yKS/WNiWxldUFTjeLTmkDbECDChuToyvU?=
 =?us-ascii?Q?15+T65uvxApvVO3b8vq7gNIkMLxhnYzcOYLNdHImFMFR4eamVJWHL4WUa8o1?=
 =?us-ascii?Q?IlwBR1hAFXaoIgZSCN227CXNW+kfjGh4j8QaThaUjtRQQwhMtQTJlLMsY9hm?=
 =?us-ascii?Q?MSGmvqgp2DWT4QS/rODsqi1+/wY8yMTx9OQyVXWwIzPnyDF/NpGBNxCG7MdL?=
 =?us-ascii?Q?QbtbJ6duKF5L/ZcvLJnpXvmdmps0Afkjh26xoO4wMu8N5qQHYwQD+JzVK+xu?=
 =?us-ascii?Q?R93S0GnkWR2ATYPqcCDdYQJbrYibXpF+ZIv5pn2gnCGNqMPbeKI6lhfumeH0?=
 =?us-ascii?Q?9LonUv06RLDY91NTgGMmV+tY15R6Zq0l3YN9oVB/eel7tnHpqdbWu/HhNh6d?=
 =?us-ascii?Q?gp5BlXHdOjPDrIGuwny/0IdhMEG+lely1jWuBYmip/OfOvXKnqk/d4Arphat?=
 =?us-ascii?Q?hC+LDZE99TagMbDStLPZ97KdC2Ck3z7hNSeHvzwOK6mGQKifcFDTZWVSPlUh?=
 =?us-ascii?Q?hNzpDvum7oJoQp9wzUUHYezzh1xpjwOLcXj4Dpt27udjrtE+dfh+mYHeDYTR?=
 =?us-ascii?Q?mJgn12gKWHQ8koNH11zHWlJZqDD+FxnXlzs6ps+nsGnpC0CMXqOqRWzxOAP7?=
 =?us-ascii?Q?KqVm2wZMgmO1NvXN/OQMPMB9GZoDcM++OoPl4M193tYHhk6ltZRrBeJNr6/I?=
 =?us-ascii?Q?du1O223ULI2W9m7taTwy0FgUUQG46JEqN7LNeZVOvnWD8uGJWVgcwTv3rfvx?=
 =?us-ascii?Q?92pLSjuIPxCXjeCO601ULDZ+4c3ETyrupmOc9DCCKTkVeWzbv1mr8Wj28z6T?=
 =?us-ascii?Q?npbZfM3RnQhPm0SomjT/skAd0zRwF1eknLZSawdRqS9JsXsGdzDU18FPWvNb?=
 =?us-ascii?Q?I7Q6Iem+tVXabzlJdc6XL8Vr53sxttlGj5MohrzIFhimDLC7TiEclBAyR5PY?=
 =?us-ascii?Q?Qf/VkCNoSEmHbi7fGsJ9yObU8WzKAWP3iriKwrBlphr54emDyiNhss18l38x?=
 =?us-ascii?Q?d74NFac30lniycbKVVwnWqSLcBJUe6e+0K+kCIc/uIuhRz8j8NwdM365NwJw?=
 =?us-ascii?Q?VQGUTXL9tFzeOpd/tekhehDhDpihH2xOUjb9mfh5SuvTy+/kLkRp2vcm1RfJ?=
 =?us-ascii?Q?e7kvKzep0u1IERhHq+SzP26luZ8GFiiLLKwFd0DWGIkCAoH67IpN+zGd5vDU?=
 =?us-ascii?Q?71xtx5Y7LRRwsliq4gCSJ4FLOWd5YFvVsh6Qc+RW?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 8711b39a-718c-43e0-1ff0-08dc6ac47d78
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 May 2024 16:25:33.2716
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ppycXuNTRLElPdLkUFfioFXyEvcPLpUP/BxgtlWlg0Tvmqldzz6zo6D2brzPNaLz
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB9128

On Thu, Apr 25, 2024 at 07:41:50AM -0700, Tanmay Jagdale wrote:
> Resending the patches by Zhen Lei <thunder.leizhen@huawei.com> that add
> support for SMMU ECMDQ feature.
> 
> Tested this feature on a Marvell SoC by implementing a smmu-test driver.
> This test driver spawns a thread per CPU and each thread keeps sending
> map, table-walk and unmap requests for a fixed duration.

So this is not just measuring invalidation performance but basically
the DMA API performance to do map/unmap operations? What is "batch
size" ?

Does this HW support the range invalidation? How many invalidation
commands does earch test cycle generate?

>                    Total Requests  Average Requests  Difference
>                                       Per CPU         wrt ECMDQ
> -----------------------------------------------------------------
> ECMDQ                 239286381       2991079
> CMDQ Batch Size 1     228232187       2852902         -4.62%
> CMDQ Batch Size 32    233465784       2918322         -2.43%
> CMDQ Batch Size 64    231679588       2895994         -3.18%
> CMDQ Batch Size 128   233189030       2914862         -2.55%
> CMDQ Batch Size 256   230965773       2887072         -3.48%

If this is really 5% for a typical DMA API map/unmap cycle then that
seems interesting to me.

If it is 5% for just the invalidation command then it is harder to
say.

I'd suggest to present your results in terms of latency to do a dma
API map/unmap cycle, and to show how the results scale as you add more
threads. Does even 2 threads start to show a 4-5% gain?

Also, I'm wondering how ATS would interact, I think we have a
head-of-line blocking issue with ATS.. Allowing ATS to progress
concurrently with unrelated parallel invalidations may also be
interesting, esepcially for multi-SVA workloads.

Jason