Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20220206174359.2986-1-jszhang@kernel.org>
In-Reply-To: <20220206174359.2986-1-jszhang@kernel.org>
From:   Arnd Bergmann <arnd@arndb.de>
Date:   Mon, 7 Feb 2022 08:35:54 +0100
Message-ID: <CAK8P3a3Nnw5gfPAYx5UGdW8Om16GDcBi1-SS9Bs+CyLzssgVbA@mail.gmail.com>
Subject: Re: [RFC PATCH] riscv: reduce THREAD_SIZE from 16KB to 8KB for RV64
To:     Jisheng Zhang <jszhang@kernel.org>
Cc:     Paul Walmsley <paul.walmsley@sifive.com>,
        Palmer Dabbelt <palmer@dabbelt.com>,
        Albert Ou <aou@eecs.berkeley.edu>,
        linux-riscv <linux-riscv@lists.infradead.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Sun, Feb 6, 2022 at 6:43 PM Jisheng Zhang <jszhang@kernel.org> wrote:
>
> After irq stack is supported, it's possible to use small THREAD_SIZE.
> In fact, I tested this patch on a Lichee RV board, looks good so far.
>
> Signed-off-by: Jisheng Zhang <jszhang@kernel.org>

I can definitely see the value in this, in particular when you get hardware with
small RAM configurations that would run a 32-bit kernel on other architectures.

However, it's worth pointing out that all other 64-bit architectures use 16KB or
more, so it is rather dangerous to be the first architecture to try this out in
a long time. Some on-stack structures have a lot of pointers and 'unsigned long'
members, so they need twice the space, while other structures are the
same size either way.

IRQ stacks obviously help here, but I don't think the 8KB size can be made
the default without a lot of testing of some of the more rarely used code paths.

Here are a few things that would be worth doing on the way to a smaller
kernel stack:

- do more compile-time testing with a lower CONFIG_FRAME_WARN value.
  Historically, this defaults to 2048 bytes on 64-bit architectures. This is
  much higher than we really want, but it takes work to find and eliminate
  the outliers. I previously had a series to reduce the limit to 1280 bytes on
  arm64 and still build all 'randconfig' configurations.

- Use a much lower limit during regression testing. There is already a config
   option to randomize the start of the thread stack, but you can also try
   adding a configurable offset to see how far you can push it for a given
   workload before you run into the guard page.

- With vmap stacks, using 12KB may be an option as well, giving you
   three pages for the stack plus one guard page, instead of 4 pages
   stack plus four guard pages.

- If you can make a convincing case for using a lower THREAD_SIZE,
  make this a compile-time option across all 64-bit architectures that
  support both IRQ stacks and VMAP stacks. The actual frame size
  does depend on the ISA a bit, and we know that parisc and ia64 are
  particularly, possibly s390 as well, but I would expect risc-v to be
  not much different from arm64 and x86 here.

       Arnd