Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: [RFC PATCH 2/5] RISC-V: Use Linux logical cpu number instead of
 hartid
To:     Anup Patel <anup@brainfault.org>
Cc:     "palmer@sifive.com" <palmer@sifive.com>,
        "linux-riscv@lists.infradead.org" <linux-riscv@lists.infradead.org>,
        Mark Rutland <mark.rutland@arm.com>,
        Christoph Hellwig <hch@infradead.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        "linux-kernel@vger.kernel.org List" <linux-kernel@vger.kernel.org>,
        Damien Le Moal <Damien.LeMoal@wdc.com>
References: <1534377377-70108-1-git-send-email-atish.patra@wdc.com>
 <1534377377-70108-3-git-send-email-atish.patra@wdc.com>
 <CAAhSdy0O7qid67J2pvyc3sFHsOPA-PFpiSKrHBB1tRDb678bAQ@mail.gmail.com>
 <adf9579e-62e1-e778-10f0-5e55c5b48406@wdc.com>
 <CAAhSdy2WXSeg7eWM0Jj7bN6KN6KC+WHqetZTSx4ATrfEcRk6Bg@mail.gmail.com>
 <f45cd8bd-0b79-9e01-595d-5d0d19ab543e@wdc.com>
 <CAAhSdy0GLqwWohBjGD_Q+s95wXJKge-6MCf4a1TkMoDS3ezEAg@mail.gmail.com>
From:   Atish Patra <atish.patra@wdc.com>
Message-ID: <004f66f0-4e65-d787-145a-bbf0c6f19763@wdc.com>
Date:   Thu, 16 Aug 2018 10:26:56 -0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0)
 Gecko/20100101 Thunderbird/52.9.1
MIME-Version: 1.0
In-Reply-To: <CAAhSdy0GLqwWohBjGD_Q+s95wXJKge-6MCf4a1TkMoDS3ezEAg@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On 8/15/18 11:03 PM, Anup Patel wrote:
> On Thu, Aug 16, 2018 at 11:22 AM, Atish Patra <atish.patra@wdc.com> wrote:
>> On 8/15/18 10:45 PM, Anup Patel wrote:
>>>
>>> On Thu, Aug 16, 2018 at 10:53 AM, Atish Patra <atish.patra@wdc.com> wrote:
>>>>
>>>> On 8/15/18 9:24 PM, Anup Patel wrote:
>>>>>
>>>>>
>>>>> On Thu, Aug 16, 2018 at 5:26 AM, Atish Patra <atish.patra@wdc.com>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> Setup the cpu_logical_map during boot. Moreover, every SBI call
>>>>>> and PLIC context are based on the physical hartid. Use the logical
>>>>>> cpu to hartid mapping to pass correct hartid to respective functions.
>>>>>>
>>>>>> Signed-off-by: Atish Patra <atish.patra@wdc.com>
>>>>>> ---
>>>>>>     arch/riscv/include/asm/tlbflush.h | 17 +++++++++++++----
>>>>>>     arch/riscv/kernel/cpu.c           |  4 +++-
>>>>>>     arch/riscv/kernel/setup.c         | 10 ++++++++++
>>>>>>     arch/riscv/kernel/smp.c           | 24 +++++++++++++++---------
>>>>>>     arch/riscv/kernel/smpboot.c       | 30
>>>>>> ++++++++++++++++++------------
>>>>>>     drivers/clocksource/riscv_timer.c | 12 ++++++++----
>>>>>>     drivers/irqchip/irq-sifive-plic.c | 11 +++++++----
>>>>>>     7 files changed, 74 insertions(+), 34 deletions(-)
>>>>>>
>>>>>> diff --git a/arch/riscv/include/asm/tlbflush.h
>>>>>> b/arch/riscv/include/asm/tlbflush.h
>>>>>> index 85c2d8ba..ecfd9b0e 100644
>>>>>> --- a/arch/riscv/include/asm/tlbflush.h
>>>>>> +++ b/arch/riscv/include/asm/tlbflush.h
>>>>>> @@ -16,6 +16,7 @@
>>>>>>     #define _ASM_RISCV_TLBFLUSH_H
>>>>>>
>>>>>>     #include <linux/mm_types.h>
>>>>>> +#include <asm/smp.h>
>>>>>>
>>>>>>     /*
>>>>>>      * Flush entire local TLB.  'sfence.vma' implicitly fences with the
>>>>>> instruction
>>>>>> @@ -49,13 +50,21 @@ static inline void flush_tlb_range(struct
>>>>>> vm_area_struct *vma,
>>>>>>
>>>>>>     #include <asm/sbi.h>
>>>>>>
>>>>>> -#define flush_tlb_all() sbi_remote_sfence_vma(NULL, 0, -1)
>>>>>> +static inline void remote_sfence_vma(struct cpumask *cmask, unsigned
>>>>>> long start,
>>>>>> +                                    unsigned long size)
>>>>>> +{
>>>>>> +       struct cpumask hmask;
>>>>>> +
>>>>>> +       cpuid_to_hartid_mask(cmask, &hmask);
>>>>>> +       sbi_remote_sfence_vma(hmask.bits, start, size);
>>>>>> +}
>>>>>> +
>>>>>> +#define flush_tlb_all() remote_sfence_vma(NULL, 0, -1)
>>>>>>     #define flush_tlb_page(vma, addr) flush_tlb_range(vma, addr, 0)
>>>>>>     #define flush_tlb_range(vma, start, end) \
>>>>>> -       sbi_remote_sfence_vma(mm_cpumask((vma)->vm_mm)->bits, \
>>>>>> -                             start, (end) - (start))
>>>>>> +       remote_sfence_vma(mm_cpumask((vma)->vm_mm), start, (end) -
>>>>>> (start))
>>>>>>     #define flush_tlb_mm(mm) \
>>>>>> -       sbi_remote_sfence_vma(mm_cpumask(mm)->bits, 0, -1)
>>>>>> +       remote_sfence_vma(mm_cpumask(mm), 0, -1)
>>>>>>
>>>>>>     #endif /* CONFIG_SMP */
>>>>>>
>>>>>> diff --git a/arch/riscv/kernel/cpu.c b/arch/riscv/kernel/cpu.c
>>>>>> index ca6c81e5..f8a18ace 100644
>>>>>> --- a/arch/riscv/kernel/cpu.c
>>>>>> +++ b/arch/riscv/kernel/cpu.c
>>>>>> @@ -14,6 +14,7 @@
>>>>>>     #include <linux/init.h>
>>>>>>     #include <linux/seq_file.h>
>>>>>>     #include <linux/of.h>
>>>>>> +#include <asm/smp.h>
>>>>>>
>>>>>>     /* Return -1 if not a valid hart */
>>>>>>     int riscv_of_processor_hart(struct device_node *node)
>>>>>> @@ -79,7 +80,8 @@ static void c_stop(struct seq_file *m, void *v)
>>>>>>     static int c_show(struct seq_file *m, void *v)
>>>>>>     {
>>>>>>            unsigned long hart_id = (unsigned long)v - 1;
>>>>>> -       struct device_node *node = of_get_cpu_node(hart_id, NULL);
>>>>>> +       struct device_node *node =
>>>>>> of_get_cpu_node(cpu_logical_map(hart_id),
>>>>>> +                                                  NULL);
>>>>>
>>>>>
>>>>>
>>>>> The hart_id is misleading name here. It should be cpu_id. Please replace
>>>>> all instances of hart_id with cpu_id and where hard ID is to be
>>>>> displayed
>>>>> use cpu_logical_map(cpu_id).
>>>>>
>>>> Correct. Thanks for catching it. I will fix it in v2.
>>>>
>>>>
>>>>>>            const char *compat, *isa, *mmu;
>>>>>>
>>>>>>            seq_printf(m, "hart\t: %lu\n", hart_id);
>>>>>> diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c
>>>>>> index e21ed481..97b586f8 100644
>>>>>> --- a/arch/riscv/kernel/setup.c
>>>>>> +++ b/arch/riscv/kernel/setup.c
>>>>>> @@ -84,6 +84,16 @@ atomic_t hart_lottery;
>>>>>>
>>>>>>     u64 __cpu_logical_map[NR_CPUS];
>>>>>>
>>>>>> +void __init smp_setup_processor_id(void)
>>>>>> +{
>>>>>> +       int cpu = smp_processor_id();
>>>>>> +
>>>>>> +       cpu_logical_map(0) = cpu;
>>>>>
>>>>>
>>>>>
>>>>> I think this should be:
>>>>> cpu_logical_map(cpu) = hart_id;
>>>>>
>>>>> Here hart_id for boot CPU will be value of a0 register passed at
>>>>> boot-time.
>>>>>
>>>> I will change the variable name to hart_id. The assembly code in head.S
>>>> have
>>>> already stored hart id in thread info structure. So smp_processor_id()
>>>> and
>>>> hart id would be the same.
>>>>
>>>>
>>>
>>> I guess this means that for boot CPU, cpuid == hartid
>>>
>>
>> No. I set the cpuid 0 for boot cpu by doing this.
>>
>> +       /* Change the boot cpu ID in thread_info */
>> +       current->thread_info.cpu = 0;
>>
>>> This is very confusing because other places I see CPU0 is the boot CPU.
>>>
>>
>> CPU0 is the boot cpu.
>>>
>>> I think assembly code in head.S should store 0 in thread info for boot
>>> CPU.
>>>
>> If we do that, we loose track of boot cpu hartid. We have to store it
>> somewhere else to update the cpu_logical_map(0). Isn't it ?
> 
> We can have variable of machine word size declared in assembly
> named "boot_cpu_hart_id" which will hold the hart ID of boot CPU.
> 
> No need to update thread_info.cpu on boot CPU. In fact, set
> thread_info.cpu to 0 in head.S itself.
> 

Hmm. That will work too. Thanks.

Atish
> Regards,
> Anup
>