Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20220317035542.272547-1-apatel@ventanamicro.com> <f1079093-54db-bd20-3342-b06d069c5d1f@codethink.co.uk>
In-Reply-To: <f1079093-54db-bd20-3342-b06d069c5d1f@codethink.co.uk>
From:   Arnd Bergmann <arnd@arndb.de>
Date:   Fri, 18 Mar 2022 16:14:24 +0100
Message-ID: <CAK8P3a0Gt7TOT2zpNw7YNR=MNLBWVAADL5dBgY15ZL8sDW20HQ@mail.gmail.com>
Subject: Re: [PATCH] RISC-V: Increase range and default value of NR_CPUS
To:     Ben Dooks <ben.dooks@codethink.co.uk>
Cc:     Anup Patel <apatel@ventanamicro.com>,
        Palmer Dabbelt <palmer@dabbelt.com>,
        Paul Walmsley <paul.walmsley@sifive.com>,
        Atish Patra <atishp@atishpatra.org>,
        Alistair Francis <Alistair.Francis@wdc.com>,
        Anup Patel <anup@brainfault.org>,
        linux-riscv <linux-riscv@lists.infradead.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Fri, Mar 18, 2022 at 3:46 PM Ben Dooks <ben.dooks@codethink.co.uk> wrote:
>
> On 17/03/2022 03:55, Anup Patel wrote:
> > Currently, the range and default value of NR_CPUS is too restrictive
> > for high-end RISC-V systems with large number of HARTs. The latest
> > QEMU virt machine supports upto 512 CPUs so the current NR_CPUS is
> > restrictive for QEMU as well.

If qemu allows 512, what is the reason for limiting the kernel to 256?

> > Other major architectures (such as
> > ARM64, x86_64, MIPS, etc) have a much higher range and default
> > value of NR_CPUS.
>
> What's the memory overhead for increasing this?

It's supposed to be very small, I would expect three main sources of overhead:

- cpumask_t variables, those grow once you go beyond the size of
   an unsigned long (32 or 64 bits), so with the default just on the limit, this
   makes no difference.
   Note that you can run out of stack space with NR_CPUS values if
   CONFIG_CPUMASK_OFFSTACK is disabled. Should not be
   a problem for 512 or below.

- percpu variables: these are dynamically allocated based on the number of
  CPUs at boot time, so they should not have any real impact.

- NR_CPUS sized arrays, these are sometimes used in place of
  percpu data. This is only a problem if the array members individually
  are more than a few bytes. There are not too many of these in the kernel,
  as using those is discouraged.

      Arnd