Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <YmX7z+BirkA3VAfW@zx2c4.com>
In-Reply-To: <YmX7z+BirkA3VAfW@zx2c4.com>
From:   Arnd Bergmann <arnd@arndb.de>
Date:   Mon, 25 Apr 2022 10:46:45 +0200
Message-ID: <CAK8P3a3Af5FBx-OnedHPrf28ikX4DZK1d0ERLsV+oKyBHyCXiw@mail.gmail.com>
Subject: Re: odd endianness toolchains for crosstool
To:     "Jason A. Donenfeld" <Jason@zx2c4.com>
Cc:     Arnd Bergmann <arnd@arndb.de>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Nick Desaulniers <ndesaulniers@google.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8BIT
Precedence: bulk

On Mon, Apr 25, 2022 at 3:39 AM Jason A. Donenfeld <Jason@zx2c4.com> wrote:
>
> Hey Arnd,
>
> I'm again experimenting with switching to your crosstool toolchains for
> WireGuard's CI. I've hit a few snags in the process:
>
> - For powerpc, gcc needs to be built with `--enable-secureplt
>   --with-long-double-64` in order for musl to run.
> - Need powerpc64le compiler (-mabi=elfv2).
> - Need mipsel compiler.
> - Need aarch64_be compiler.
> - Need armeb compiler.
> - Need mips64el compiler.
>
> While the existing compilers can all produce code for the wrong endian,
> they hit trouble when trying to link against libgcc. So generally a
> separate full toolchain is supplied for the less common endians.

Hi Jason,

I'm definitely interested in improving the user space testing,  and I agree
we need the powerpc64le and mipsel targets at the minimum for that.

The situation on my end is that I'm planning to migrate my main workstation
(which I'm building the compilers on) to an arm64 machine soon, and
will then need to set it all up again. I don't really want to change much before
then to avoid changing things twice.

I've added Nick to Cc, as he's experimenting with a clang based toolchain
that we can put on kernel.org along with the gcc toolchains, and that
would probably include a musl based sysroot roughly the same set of
architectures that you are testing on already. Possibly we could reuse the
same user space between clang and gcc.

> I have had success with arm, arm64, mips, x86_64, i386, m68k. If you're
> up for adding the above compilers to the collection, I'd be able to
> complete the transition, and then look into adding a few more
> architectures.

I've also looked at other projects that do qemu based testing, everyone
seems to be missing one or two architectures out of a common set,
https://tinyurl.com/linux-architectures is where I keep my data.

The most common subset of architectures that get tested as far
as I can tell is x86, arm64, powerpc, arm, riscv, mips, s390: those
are the ones that support all the important pieces (gcc, clang, musl,
glibc, qemu, debian and buildroot) and that have the most users.
The exceptions to this are I think:

* your wireguard tests are missing riscv and s390 tests, those should
  be easy to add. You do support m68k, which the others are missing
* kunit is missing mips tests, but has alpha and sparc
* tuxrun[1] adds sparc64, sh4 and armv5 (softfp), could add rv32
* Günter's linux-build-test[2] has all the above, plus microblaze,
  nios2, parisc, shbe, and xtensa

What I'd really like to see is to have the necessary information for
building and running the most common subset of these in one place
in the kernel tree where at least wireguard, kunit and tuxrun can
share the setup for qemu. One idea I had was to encode the
platform specific qemu command line options using Kconfig
dependencies in a way that "make O=obj ARCH=foo defconfig zImage;
obj/run-qemu" results in a booting kernel on a lot of the typical
defconfigs for supported architectures.

       Arnd

[1] https://gitlab.com/Linaro/tuxrun/-/blob/master/tuxrun/devices/qemu.py
[2] https://github.com/groeck/linux-build-test/tree/master/rootfs