Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20220420112408.155561-1-apatel@ventanamicro.com>
 <3fa5215b-84c3-5e4c-cb53-6c05d05e4350@canonical.com> <CAOnJCUJrN4frY_OdQzO-yr5CrDLvj=ge9KY2d=XnGvAF-uQNvQ@mail.gmail.com>
In-Reply-To: <CAOnJCUJrN4frY_OdQzO-yr5CrDLvj=ge9KY2d=XnGvAF-uQNvQ@mail.gmail.com>
From:   Atish Patra <atishp@atishpatra.org>
Date:   Fri, 22 Apr 2022 18:49:01 -0700
Message-ID: <CAOnJCUK5q0k1qKD0fM4NtbP6KAGcaBBknkCKLBXxcC=EPGVu1Q@mail.gmail.com>
Subject: Re: [PATCH v3] RISC-V: Increase range and default value of NR_CPUS
To:     Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Cc:     Anup Patel <apatel@ventanamicro.com>,
        Palmer Dabbelt <palmer@dabbelt.com>,
        Paul Walmsley <paul.walmsley@sifive.com>,
        Arnd Bergmann <arnd@arndb.de>,
        Alistair Francis <Alistair.Francis@wdc.com>,
        Anup Patel <anup@brainfault.org>,
        linux-riscv <linux-riscv@lists.infradead.org>,
        "linux-kernel@vger.kernel.org List" <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Fri, Apr 22, 2022 at 5:16 PM Atish Patra <atishp@atishpatra.org> wrote:
>
> On Thu, Apr 21, 2022 at 12:05 AM Heinrich Schuchardt
> <heinrich.schuchardt@canonical.com> wrote:
> >
> >
> >
> > On 4/20/22 13:24, Anup Patel wrote:
> > > Currently, the range and default value of NR_CPUS is too restrictive
> > > for high-end RISC-V systems with large number of HARTs. The latest
> > > QEMU virt machine supports upto 512 CPUs so the current NR_CPUS is
> > > restrictive for QEMU as well. Other major architectures (such as
> > > ARM64, x86_64, MIPS, etc) have a much higher range and default
> > > value of NR_CPUS.
> > >
> > > This patch increases NR_CPUS range to 2-512 and default value to
> > > XLEN (i.e. 32 for RV32 and 64 for RV64).
> > >
> > > Signed-off-by: Anup Patel <apatel@ventanamicro.com>
> >
>
> This works upto 207 harts in upstream Qemu + OpenSBI. Beyond that,
> firmware size becomes > 2MB and PMP protection
> kicks in for the kernel boot address (0x80200000)
>

Just wanted to clarify that this is as per the current configuration
in OpenSBI. The boot address
is configured to be 0x80000000 for the generic platform. Thus, it can
support a maximum of 207 harts.

If anybody wishes to boot more than that, the boot address should be
changed to a valid DRAM address such that
it doesn't step on kernel/DT or other resident boot loaders (e.g. U-Boot).

It should be changed per platform (if supporting more than 128 harts)
rather than changing
to a common address for all platforms.

> Here is the OpenSBI boot prints for 256 harts with some hacks around
> sanitize_domain to disable sbi_domain_check_addr
>
> Platform Name             : riscv-virtio,qemu
> Platform Features         : medeleg
> Platform HART Count       : 208
> Platform IPI Device       : aclint-mswi
> Platform Timer Device     : aclint-mtimer @ 10000000Hz
> Platform Console Device   : uart8250
> Platform HSM Device       : ---
> Platform Reboot Device    : sifive_test
> Platform Shutdown Device  : sifive_test
> Firmware Base             : 0x80000000
> Firmware Size             : 2052 KB ---------->>> Firmware size is
> greater than 2MB where
> Runtime SBI Version       : 0.3
>
> Domain0 Name              : root
> Domain0 Boot HART         : 122
> Domain0 HARTs             :
> 0*,1*,2*,3*,4*,5*,6*,7*,8*,9*,10*,11*,12*,13*,14*,15*,16*,17*,18*,19*,20*=
,21*,22*,23*,24*,25*,26*,27*,28*,29*,30*,31*,32*,33*,34*,35*,36*,37*,38*,39=
*,40*,41*,42*,43*,44*,45*,46*,47*,48*,49*,50*,51*,52*,53*,54*,55*,56*,57*,5=
8*,59*,60*,61*,62*,63*,64*,65*,66*,67*,68*,69*,70*,71*,72*,73*,74*,75*,76*,=
77*,78*,79*,80*,81*,82*,83*,84*,85*,86*,87*,88*,89*,90*,91*,92*,93*,94*,95*=
,96*,97*,98*,99*,100*,101*,102*,103*,104*,105*,106*,107*,108*,109*,110*,111=
*,112*,113*,114*,115*,116*,117*,118*,119*,120*,121*,122*,123*,124*,125*,126=
*,127*,128*,129*,130*,131*,132*,133*,134*,135*,136*,137*,138*,139*,140*,141=
*,142*,143*,144*,145*,146*,147*,148*,149*,150*,151*,152*,153*,154*,155*,156=
*,157*,158*,159*,160*,161*,162*,163*,164*,165*,166*,167*,168*,169*,170*,171=
*,172*,173*,174*,175*,176*,177*,178*,179*,180*,181*,182*,183*,184*,185*,186=
*,187*,188*,189*,190*,191*,192*,193*,194*,195*,196*,197*,198*,199*,200*,201=
*,202*,203*,204*,205*,206*,207*
> Domain0 Region00          : 0x0000000002000000-0x000000000200ffff (I)
> Domain0 Region01          : 0x0000000080000000-0x00000000803fffff ()
> Domain0 Region02          : 0x0000000000000000-0xffffffffffffffff (R,W,X)
> Domain0 Next Address      : 0x0000000080200000
> Domain0 Next Arg1         : 0x00000000bf000000
> Domain0 Next Mode         : S-mode
> Domain0 SysReset          : yes
>
> Boot HART ID              : 122
> Boot HART Domain          : root
> Boot HART ISA             : rv64imafdcsuh
> Boot HART Features        : scounteren,mcounteren,time
> Boot HART PMP Count       : 16
> Boot HART PMP Granularity : 4
> Boot HART PMP Address Bits: 54
> Boot HART MHPM Count      : 0
> Boot HART MIDELEG         : 0x0000000000001666
> Boot HART MEDELEG         : 0x0000000000f0b509
>
>
> > Reviewed-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
> >
> > > ---
> > > Changes since v2:
> > >   - Rebased on Linux-5.18-rc3
> > >   - Use a different range when SBI v0.1 is enabled
> > > Changes since v1:
> > >   - Updated NR_CPUS range to 2-512 which reflects maximum number of
> > >     CPUs supported by QEMU virt machine.
> > > ---
> > >   arch/riscv/Kconfig | 9 ++++++---
> > >   1 file changed, 6 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
> > > index 00fd9c548f26..1823f281069f 100644
> > > --- a/arch/riscv/Kconfig
> > > +++ b/arch/riscv/Kconfig
> > > @@ -275,10 +275,13 @@ config SMP
> > >         If you don't know what to do here, say N.
> > >
> > >   config NR_CPUS
> > > -     int "Maximum number of CPUs (2-32)"
> > > -     range 2 32
> > > +     int "Maximum number of CPUs (2-512)"
> > >       depends on SMP
> > > -     default "8"
> > > +     range 2 512 if !SBI_V01
> > > +     range 2 32 if SBI_V01 && 32BIT
> > > +     range 2 64 if SBI_V01 && 64BIT
> > > +     default "32" if 32BIT
> > > +     default "64" if 64BIT
> > >
> > >   config HOTPLUG_CPU
> > >       bool "Support for hot-pluggable CPUs"
>
>
>
> --
> Regards,
> Atish


--=20
Regards,
Atish