Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.118.232 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.118.232; helo=mail.nvidia.com; pr=C
Date:   Thu, 10 Aug 2023 22:00:00 -0700
From:   Nicolin Chen <nicolinc@nvidia.com>
To:     "Leizhen (ThunderTown)" <thunder.leizhen@huaweicloud.com>
CC:     Will Deacon <will@kernel.org>, Robin Murphy <robin.murphy@arm.com>,
        "Joerg Roedel" <joro@8bytes.org>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Zhen Lei <thunder.leizhen@huawei.com>,
        Tanmay Jagdale <tanmay@marvell.com>,
        "Jonathan Cameron" <Jonathan.Cameron@Huawei.com>,
        "jgg@ziepe.ca" <jgg@ziepe.ca>
Subject: Re: [PATCH v2 0/2] iommu/arm-smmu-v3: Add support for ECMDQ register
 mode
Message-ID: <ZNXAUAVZ/nNXl7qK@Asurada-Nvidia>
References: <20230809131303.1355-1-thunder.leizhen@huaweicloud.com>
 <20230809135603.GE4226@willie-the-truck>
 <eaa7758e-9ff4-a039-7f94-734f63c72ba6@huaweicloud.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <eaa7758e-9ff4-a039-7f94-734f63c72ba6@huaweicloud.com>
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Aug 2023 05:00:14.6814
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 4ec7eb8d-8afb-4950-0ac9-08db9a27d993
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.232];Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EE3C.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR12MB6707
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Aug 09, 2023 at 07:18:36PM -0700, Leizhen (ThunderTown) wrote:
> On 2023/8/9 21:56, Will Deacon wrote:
> > On Wed, Aug 09, 2023 at 09:13:01PM +0800, thunder.leizhen@huaweicloud.com wrote:
> >> From: Zhen Lei <thunder.leizhen@huawei.com>
> >>
> >> v1 --> v2:
> >
> > Jason previously asked about performance numbers for ECMDQ:
> >
> > https://lore.kernel.org/r/ZL6n3f01yV7tc4yH@ziepe.ca
> >
> > Do you have any?
> 
> I asked my colleagues in the chip department, and they said that the chip
> was not commercially available and the specific data could not be disclosed.
> However, to be sure, the performance has improved, but not by much, the
> public benchmark is only about 5%. Your optimization patch was so perfect
> that it ruined our jobs.
> 
> However, since Marvell also implements ECMDQ, there are at least two users.
> Do we think about making it available first?

I have seen something similar (~5%) with VCMDQ on NVIDIA Grace,
when running, in host OS, TLB flush benchmark tests concurrently
on different CPUs.

Although VCMDQ could be slightly different from ECMDQ, both have
a multi-queue feature. And the amount of improvement in my case
came from a reduction of congestion at issueing commands to the
multi queues vs. a single queue. And I guess ECMDQ might benefit
its 5% from that too.

If we decide to move ECMDQ forward, perhaps we can converge some
of the functions to support both :)

Thanks
Nicolin