Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates
 216.228.117.161 as permitted sender) receiver=protection.outlook.com;
 client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C
Date:   Thu, 10 Aug 2023 12:58:56 -0700
From:   Nicolin Chen <nicolinc@nvidia.com>
To:     "thunder.leizhen@huaweicloud.com" <thunder.leizhen@huaweicloud.com>
CC:     Will Deacon <will@kernel.org>, Robin Murphy <robin.murphy@arm.com>,
        "Joerg Roedel" <joro@8bytes.org>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Zhen Lei <thunder.leizhen@huawei.com>,
        Tanmay Jagdale <tanmay@marvell.com>,
        "Jonathan Cameron" <Jonathan.Cameron@Huawei.com>
Subject: Re: [PATCH v2 1/2] iommu/arm-smmu-v3: Add support for ECMDQ register
 mode
Message-ID: <ZNVBgCb5w0daExIL@Asurada-Nvidia>
References: <20230809131303.1355-1-thunder.leizhen@huaweicloud.com>
 <20230809131303.1355-2-thunder.leizhen@huaweicloud.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20230809131303.1355-2-thunder.leizhen@huaweicloud.com>
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Aug 2023 19:59:08.8567
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 0aa97e5a-3d5c-4214-8127-08db99dc4276
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com]
X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EE33.namprd05.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB9070
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE
        autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Zhen,

On Wed, Aug 09, 2023 at 06:13:02AM -0700, thunder.leizhen@huaweicloud.com wrote:
> +static int arm_smmu_ecmdq_layout(struct arm_smmu_device *smmu)
> +{
> +       int cpu;
> +       struct arm_smmu_ecmdq __percpu *ecmdq;
> +
> +       if (num_possible_cpus() <= smmu->nr_ecmdq) {

Does the nr_ecmdq always physically match with the number of CPUs?
I saw the spec saying "up to 256 pages", but not sure if they are
always physically present, even if CPU number is smaller i.e. some
of them would be unassigned/wasted.

> +               ecmdq = devm_alloc_percpu(smmu->dev, *ecmdq);
> +               if (!ecmdq)
> +                       return -ENOMEM;
> +
> +               for_each_possible_cpu(cpu)
> +                       *per_cpu_ptr(smmu->ecmdqs, cpu) = per_cpu_ptr(ecmdq, cpu);
> +
> +               /* A core requires at most one ECMDQ */
> +               smmu->nr_ecmdq = num_possible_cpus();
> +
> +               return 0;
> +       }
> +
> +       return -ENOSPC;

This ENOSPC is a software limitation, isn't it? How about using
"smp_processor_id() % smmu->nr_ecmdq" to select a queue?

> +       shift_increment = order_base_2(num_possible_cpus() / smmu->nr_ecmdq);
> +
> +       offset = 0;
> +       for_each_possible_cpu(cpu) {
> +               struct arm_smmu_ecmdq *ecmdq;
> +               struct arm_smmu_queue *q;
> +
> +               ecmdq = *per_cpu_ptr(smmu->ecmdqs, cpu);
> +               ecmdq->base = cp_base + offset;
> +
> +               q = &ecmdq->cmdq.q;
> +
> +               q->llq.max_n_shift = ECMDQ_MAX_SZ_SHIFT + shift_increment;
> +               ret = arm_smmu_init_one_queue(smmu, q, ecmdq->base, ARM_SMMU_ECMDQ_PROD,
> +                               ARM_SMMU_ECMDQ_CONS, CMDQ_ENT_DWORDS, "ecmdq");

Not getting why max_n_shift increases by shift_increment. Mind
elaborating?

Thanks
Nicolin