Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
MIME-Version: 1.0
References: <20220420112408.155561-1-apatel@ventanamicro.com> <3fa5215b-84c3-5e4c-cb53-6c05d05e4350@canonical.com>
In-Reply-To: <3fa5215b-84c3-5e4c-cb53-6c05d05e4350@canonical.com>
From:   Atish Patra <atishp@atishpatra.org>
Date:   Fri, 22 Apr 2022 17:16:51 -0700
Message-ID: <CAOnJCUJrN4frY_OdQzO-yr5CrDLvj=ge9KY2d=XnGvAF-uQNvQ@mail.gmail.com>
Subject: Re: [PATCH v3] RISC-V: Increase range and default value of NR_CPUS
To:     Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Cc:     Anup Patel <apatel@ventanamicro.com>,
        Palmer Dabbelt <palmer@dabbelt.com>,
        Paul Walmsley <paul.walmsley@sifive.com>,
        Arnd Bergmann <arnd@arndb.de>,
        Alistair Francis <Alistair.Francis@wdc.com>,
        Anup Patel <anup@brainfault.org>,
        linux-riscv <linux-riscv@lists.infradead.org>,
        "linux-kernel@vger.kernel.org List" <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Thu, Apr 21, 2022 at 12:05 AM Heinrich Schuchardt
<heinrich.schuchardt@canonical.com> wrote:
>
>
>
> On 4/20/22 13:24, Anup Patel wrote:
> > Currently, the range and default value of NR_CPUS is too restrictive
> > for high-end RISC-V systems with large number of HARTs. The latest
> > QEMU virt machine supports upto 512 CPUs so the current NR_CPUS is
> > restrictive for QEMU as well. Other major architectures (such as
> > ARM64, x86_64, MIPS, etc) have a much higher range and default
> > value of NR_CPUS.
> >
> > This patch increases NR_CPUS range to 2-512 and default value to
> > XLEN (i.e. 32 for RV32 and 64 for RV64).
> >
> > Signed-off-by: Anup Patel <apatel@ventanamicro.com>
>

This works upto 207 harts in upstream Qemu + OpenSBI. Beyond that,
firmware size becomes > 2MB and PMP protection
kicks in for the kernel boot address (0x80200000)

Here is the OpenSBI boot prints for 256 harts with some hacks around
sanitize_domain to disable sbi_domain_check_addr

Platform Name             : riscv-virtio,qemu
Platform Features         : medeleg
Platform HART Count       : 208
Platform IPI Device       : aclint-mswi
Platform Timer Device     : aclint-mtimer @ 10000000Hz
Platform Console Device   : uart8250
Platform HSM Device       : ---
Platform Reboot Device    : sifive_test
Platform Shutdown Device  : sifive_test
Firmware Base             : 0x80000000
Firmware Size             : 2052 KB ---------->>> Firmware size is
greater than 2MB where
Runtime SBI Version       : 0.3

Domain0 Name              : root
Domain0 Boot HART         : 122
Domain0 HARTs             :
0*,1*,2*,3*,4*,5*,6*,7*,8*,9*,10*,11*,12*,13*,14*,15*,16*,17*,18*,19*,20*,2=
1*,22*,23*,24*,25*,26*,27*,28*,29*,30*,31*,32*,33*,34*,35*,36*,37*,38*,39*,=
40*,41*,42*,43*,44*,45*,46*,47*,48*,49*,50*,51*,52*,53*,54*,55*,56*,57*,58*=
,59*,60*,61*,62*,63*,64*,65*,66*,67*,68*,69*,70*,71*,72*,73*,74*,75*,76*,77=
*,78*,79*,80*,81*,82*,83*,84*,85*,86*,87*,88*,89*,90*,91*,92*,93*,94*,95*,9=
6*,97*,98*,99*,100*,101*,102*,103*,104*,105*,106*,107*,108*,109*,110*,111*,=
112*,113*,114*,115*,116*,117*,118*,119*,120*,121*,122*,123*,124*,125*,126*,=
127*,128*,129*,130*,131*,132*,133*,134*,135*,136*,137*,138*,139*,140*,141*,=
142*,143*,144*,145*,146*,147*,148*,149*,150*,151*,152*,153*,154*,155*,156*,=
157*,158*,159*,160*,161*,162*,163*,164*,165*,166*,167*,168*,169*,170*,171*,=
172*,173*,174*,175*,176*,177*,178*,179*,180*,181*,182*,183*,184*,185*,186*,=
187*,188*,189*,190*,191*,192*,193*,194*,195*,196*,197*,198*,199*,200*,201*,=
202*,203*,204*,205*,206*,207*
Domain0 Region00          : 0x0000000002000000-0x000000000200ffff (I)
Domain0 Region01          : 0x0000000080000000-0x00000000803fffff ()
Domain0 Region02          : 0x0000000000000000-0xffffffffffffffff (R,W,X)
Domain0 Next Address      : 0x0000000080200000
Domain0 Next Arg1         : 0x00000000bf000000
Domain0 Next Mode         : S-mode
Domain0 SysReset          : yes

Boot HART ID              : 122
Boot HART Domain          : root
Boot HART ISA             : rv64imafdcsuh
Boot HART Features        : scounteren,mcounteren,time
Boot HART PMP Count       : 16
Boot HART PMP Granularity : 4
Boot HART PMP Address Bits: 54
Boot HART MHPM Count      : 0
Boot HART MIDELEG         : 0x0000000000001666
Boot HART MEDELEG         : 0x0000000000f0b509


> Reviewed-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
>
> > ---
> > Changes since v2:
> >   - Rebased on Linux-5.18-rc3
> >   - Use a different range when SBI v0.1 is enabled
> > Changes since v1:
> >   - Updated NR_CPUS range to 2-512 which reflects maximum number of
> >     CPUs supported by QEMU virt machine.
> > ---
> >   arch/riscv/Kconfig | 9 ++++++---
> >   1 file changed, 6 insertions(+), 3 deletions(-)
> >
> > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
> > index 00fd9c548f26..1823f281069f 100644
> > --- a/arch/riscv/Kconfig
> > +++ b/arch/riscv/Kconfig
> > @@ -275,10 +275,13 @@ config SMP
> >         If you don't know what to do here, say N.
> >
> >   config NR_CPUS
> > -     int "Maximum number of CPUs (2-32)"
> > -     range 2 32
> > +     int "Maximum number of CPUs (2-512)"
> >       depends on SMP
> > -     default "8"
> > +     range 2 512 if !SBI_V01
> > +     range 2 32 if SBI_V01 && 32BIT
> > +     range 2 64 if SBI_V01 && 64BIT
> > +     default "32" if 32BIT
> > +     default "64" if 64BIT
> >
> >   config HOTPLUG_CPU
> >       bool "Support for hot-pluggable CPUs"


--=20
Regards,
Atish