Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Subject: Re: [PATCH v2 1/2] iommu/arm-smmu-v3: Add support for ECMDQ register
 mode
To:     Nicolin Chen <nicolinc@nvidia.com>
Cc:     Will Deacon <will@kernel.org>, Robin Murphy <robin.murphy@arm.com>,
        Joerg Roedel <joro@8bytes.org>,
        "iommu@lists.linux.dev" <iommu@lists.linux.dev>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Zhen Lei <thunder.leizhen@huawei.com>,
        Tanmay Jagdale <tanmay@marvell.com>,
        Jonathan Cameron <Jonathan.Cameron@Huawei.com>
References: <20230809131303.1355-1-thunder.leizhen@huaweicloud.com>
 <20230809131303.1355-2-thunder.leizhen@huaweicloud.com>
 <ZNVBgCb5w0daExIL@Asurada-Nvidia>
From:   "Leizhen (ThunderTown)" <thunder.leizhen@huaweicloud.com>
Message-ID: <8be55ce1-ae59-e7f0-c774-f0371272fd2a@huaweicloud.com>
Date:   Fri, 11 Aug 2023 14:23:41 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101
 Thunderbird/60.7.0
MIME-Version: 1.0
In-Reply-To: <ZNVBgCb5w0daExIL@Asurada-Nvidia>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk


On 2023/8/11 3:58, Nicolin Chen wrote:
> Hi Zhen,
> 
> On Wed, Aug 09, 2023 at 06:13:02AM -0700, thunder.leizhen@huaweicloud.com wrote:
>> +static int arm_smmu_ecmdq_layout(struct arm_smmu_device *smmu)
>> +{
>> +       int cpu;
>> +       struct arm_smmu_ecmdq __percpu *ecmdq;
>> +
>> +       if (num_possible_cpus() <= smmu->nr_ecmdq) {
> 
> Does the nr_ecmdq always physically match with the number of CPUs?
> I saw the spec saying "up to 256 pages", but not sure if they are
> always physically present, even if CPU number is smaller i.e. some
> of them would be unassigned/wasted.

This is considered in v1, but now priority is to support the most
basic features. Another advantage is that if someone makes suggestions
for improvement, the workload can be smaller.

> 
>> +               ecmdq = devm_alloc_percpu(smmu->dev, *ecmdq);
>> +               if (!ecmdq)
>> +                       return -ENOMEM;
>> +
>> +               for_each_possible_cpu(cpu)
>> +                       *per_cpu_ptr(smmu->ecmdqs, cpu) = per_cpu_ptr(ecmdq, cpu);
>> +
>> +               /* A core requires at most one ECMDQ */
>> +               smmu->nr_ecmdq = num_possible_cpus();
>> +
>> +               return 0;
>> +       }
>> +
>> +       return -ENOSPC;
> 
> This ENOSPC is a software limitation, isn't it? How about using
> "smp_processor_id() % smmu->nr_ecmdq" to select a queue?
> 
>> +       shift_increment = order_base_2(num_possible_cpus() / smmu->nr_ecmdq);
>> +
>> +       offset = 0;
>> +       for_each_possible_cpu(cpu) {
>> +               struct arm_smmu_ecmdq *ecmdq;
>> +               struct arm_smmu_queue *q;
>> +
>> +               ecmdq = *per_cpu_ptr(smmu->ecmdqs, cpu);
>> +               ecmdq->base = cp_base + offset;
>> +
>> +               q = &ecmdq->cmdq.q;
>> +
>> +               q->llq.max_n_shift = ECMDQ_MAX_SZ_SHIFT + shift_increment;
>> +               ret = arm_smmu_init_one_queue(smmu, q, ecmdq->base, ARM_SMMU_ECMDQ_PROD,
>> +                               ARM_SMMU_ECMDQ_CONS, CMDQ_ENT_DWORDS, "ecmdq");
> 
> Not getting why max_n_shift increases by shift_increment. Mind
> elaborating?

For example:
If a CPU exclusively occupies an ECMDQ, the number of queue elements is 256.
If two cores share an ECMDQ, the number of queue elements increases to 512.

> 
> Thanks
> Nicolin
> .
> 

-- 
Regards,
  Zhen Lei