Following build errors noticed on Linux next-20231204 tag with clang-nightly
for arm and arm64.
## Test Regressions (compared to next-20231201)
* arm64, build
- clang-nightly-defconfig
- clang-nightly-defconfig-40bc7ee5
- clang-nightly-lkftconfig
- clang-nightly-lkftconfig-kselftest
* arm, build
- clang-nightly-allnoconfig
- clang-nightly-axm55xx_defconfig
- clang-nightly-bcm2835_defconfig
- clang-nightly-clps711x_defconfig
- clang-nightly-defconfig
- clang-nightly-exynos_defconfig
- clang-nightly-imx_v6_v7_defconfig
- clang-nightly-keystone_defconfig
- clang-nightly-lkftconfig
- clang-nightly-lkftconfig-kselftest
- clang-nightly-omap2plus_defconfig
- clang-nightly-pxa910_defconfig
- clang-nightly-s3c6400_defconfig
- clang-nightly-s5pv210_defconfig
- clang-nightly-sama5_defconfig
- clang-nightly-shmobile_defconfig
- clang-nightly-tinyconfig
- clang-nightly-u8500_defconfig
- clang-nightly-vexpress_defconfig
Reported-by: Linux Kernel Functional Testing <[email protected]>
Build log on arm64:
---------
In file included from lib/vdso/gettimeofday.c:5:
In file included from include/vdso/datapage.h:135:
arch/arm64/include/asm/vdso/compat_gettimeofday.h:152:15: error:
instruction variant requires ARMv6 or later
152 | asm volatile("mov %0, %1" : "=r"(ret) : "r"(_vdso_data));
| ^
<inline asm>:1:2: note: instantiated into assembly here
1 | mov r4, r1
| ^
In file included from <built-in>:3:
lib/vdso/gettimeofday.c:139:3: error: invalid instruction
139 | smp_rmb();
| ^
Build log on arm:
---------
In file included from arch/arm/vfp/vfpmodule.c:23:
arch/arm/include/asm/cp15.h:101:2: error: instruction requires: data-barriers
101 | isb();
| ^
Links:
arm64
- https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20231204/testrun/21471527/suite/build/test/clang-nightly-defconfig/details/
arm
- https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20231204/testrun/21471941/suite/build/test/clang-nightly-exynos_defconfig/log
- https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20231204/testrun/21471941/suite/build/test/clang-nightly-exynos_defconfig/details/
--
Linaro LKFT
https://lkft.linaro.org
Hi Naresh,
On Mon, Dec 04, 2023 at 05:33:26PM +0530, Naresh Kamboju wrote:
> Following build errors noticed on Linux next-20231204 tag with clang-nightly
> for arm and arm64.
>
> ## Test Regressions (compared to next-20231201)
> * arm64, build
> - clang-nightly-defconfig
> - clang-nightly-defconfig-40bc7ee5
> - clang-nightly-lkftconfig
> - clang-nightly-lkftconfig-kselftest
>
> * arm, build
> - clang-nightly-allnoconfig
> - clang-nightly-axm55xx_defconfig
> - clang-nightly-bcm2835_defconfig
> - clang-nightly-clps711x_defconfig
> - clang-nightly-defconfig
> - clang-nightly-exynos_defconfig
> - clang-nightly-imx_v6_v7_defconfig
> - clang-nightly-keystone_defconfig
> - clang-nightly-lkftconfig
> - clang-nightly-lkftconfig-kselftest
> - clang-nightly-omap2plus_defconfig
> - clang-nightly-pxa910_defconfig
> - clang-nightly-s3c6400_defconfig
> - clang-nightly-s5pv210_defconfig
> - clang-nightly-sama5_defconfig
> - clang-nightly-shmobile_defconfig
> - clang-nightly-tinyconfig
> - clang-nightly-u8500_defconfig
> - clang-nightly-vexpress_defconfig
>
>
> Reported-by: Linux Kernel Functional Testing <[email protected]>
>
>
> Build log on arm64:
> ---------
> In file included from lib/vdso/gettimeofday.c:5:
> In file included from include/vdso/datapage.h:135:
> arch/arm64/include/asm/vdso/compat_gettimeofday.h:152:15: error:
> instruction variant requires ARMv6 or later
> 152 | asm volatile("mov %0, %1" : "=r"(ret) : "r"(_vdso_data));
> | ^
> <inline asm>:1:2: note: instantiated into assembly here
> 1 | mov r4, r1
> | ^
> In file included from <built-in>:3:
> lib/vdso/gettimeofday.c:139:3: error: invalid instruction
> 139 | smp_rmb();
> | ^
>
> Build log on arm:
> ---------
> In file included from arch/arm/vfp/vfpmodule.c:23:
> arch/arm/include/asm/cp15.h:101:2: error: instruction requires: data-barriers
> 101 | isb();
> | ^
This is caused by a change to Debian's LLVM that changes the internal
defaults of the arm-linux-gnueabi and arm-linux-gnueabihf tuples:
https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/commit/907baf024b9a5a1626893d9e731b6c79ccf45c87
We use arm-linux-gnueabi for the kernel (see scripts/Makefile.clang) so
now we have a hardcoded armv5te CPU, even if we are building for armv7
or such.
I am still investigating into what (if anything) can be done to resolve
this on the kernel side. We could potentially revert commit
ddc72c9659b5 ("kbuild: clang: do not use CROSS_COMPILE for target
triple") but I am not sure that will save us from that change, as
tuxmake's CROSS_COMPILE=arm-linux-gnueabihf will cause us to have an
armv7 CPU even though we may not be building for armv7.
Cheers,
Nathan
On Mon, Dec 04, 2023 at 11:13:04AM -0700, Nathan Chancellor wrote:
> Hi Naresh,
>
> On Mon, Dec 04, 2023 at 05:33:26PM +0530, Naresh Kamboju wrote:
> > Following build errors noticed on Linux next-20231204 tag with clang-nightly
> > for arm and arm64.
> >
> > ## Test Regressions (compared to next-20231201)
> > * arm64, build
> > - clang-nightly-defconfig
> > - clang-nightly-defconfig-40bc7ee5
> > - clang-nightly-lkftconfig
> > - clang-nightly-lkftconfig-kselftest
> >
> > * arm, build
> > - clang-nightly-allnoconfig
> > - clang-nightly-axm55xx_defconfig
> > - clang-nightly-bcm2835_defconfig
> > - clang-nightly-clps711x_defconfig
> > - clang-nightly-defconfig
> > - clang-nightly-exynos_defconfig
> > - clang-nightly-imx_v6_v7_defconfig
> > - clang-nightly-keystone_defconfig
> > - clang-nightly-lkftconfig
> > - clang-nightly-lkftconfig-kselftest
> > - clang-nightly-omap2plus_defconfig
> > - clang-nightly-pxa910_defconfig
> > - clang-nightly-s3c6400_defconfig
> > - clang-nightly-s5pv210_defconfig
> > - clang-nightly-sama5_defconfig
> > - clang-nightly-shmobile_defconfig
> > - clang-nightly-tinyconfig
> > - clang-nightly-u8500_defconfig
> > - clang-nightly-vexpress_defconfig
> >
> >
> > Reported-by: Linux Kernel Functional Testing <[email protected]>
> >
> >
> > Build log on arm64:
> > ---------
> > In file included from lib/vdso/gettimeofday.c:5:
> > In file included from include/vdso/datapage.h:135:
> > arch/arm64/include/asm/vdso/compat_gettimeofday.h:152:15: error:
> > instruction variant requires ARMv6 or later
> > 152 | asm volatile("mov %0, %1" : "=r"(ret) : "r"(_vdso_data));
> > | ^
> > <inline asm>:1:2: note: instantiated into assembly here
> > 1 | mov r4, r1
> > | ^
I have to wonder why Clang is complaining about "mov r4, r1" because
that certainly should not require "ARMv6 or later". On the face of it,
this to me looks like a bug in Clang.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!
On Mon, Dec 04, 2023 at 06:26:15PM +0000, Russell King (Oracle) wrote:
> On Mon, Dec 04, 2023 at 11:13:04AM -0700, Nathan Chancellor wrote:
> > Hi Naresh,
> >
> > On Mon, Dec 04, 2023 at 05:33:26PM +0530, Naresh Kamboju wrote:
> > > Following build errors noticed on Linux next-20231204 tag with clang-nightly
> > > for arm and arm64.
> > >
> > > ## Test Regressions (compared to next-20231201)
> > > * arm64, build
> > > - clang-nightly-defconfig
> > > - clang-nightly-defconfig-40bc7ee5
> > > - clang-nightly-lkftconfig
> > > - clang-nightly-lkftconfig-kselftest
> > >
> > > * arm, build
> > > - clang-nightly-allnoconfig
> > > - clang-nightly-axm55xx_defconfig
> > > - clang-nightly-bcm2835_defconfig
> > > - clang-nightly-clps711x_defconfig
> > > - clang-nightly-defconfig
> > > - clang-nightly-exynos_defconfig
> > > - clang-nightly-imx_v6_v7_defconfig
> > > - clang-nightly-keystone_defconfig
> > > - clang-nightly-lkftconfig
> > > - clang-nightly-lkftconfig-kselftest
> > > - clang-nightly-omap2plus_defconfig
> > > - clang-nightly-pxa910_defconfig
> > > - clang-nightly-s3c6400_defconfig
> > > - clang-nightly-s5pv210_defconfig
> > > - clang-nightly-sama5_defconfig
> > > - clang-nightly-shmobile_defconfig
> > > - clang-nightly-tinyconfig
> > > - clang-nightly-u8500_defconfig
> > > - clang-nightly-vexpress_defconfig
> > >
> > >
> > > Reported-by: Linux Kernel Functional Testing <[email protected]>
> > >
> > >
> > > Build log on arm64:
> > > ---------
> > > In file included from lib/vdso/gettimeofday.c:5:
> > > In file included from include/vdso/datapage.h:135:
> > > arch/arm64/include/asm/vdso/compat_gettimeofday.h:152:15: error:
> > > instruction variant requires ARMv6 or later
> > > 152 | asm volatile("mov %0, %1" : "=r"(ret) : "r"(_vdso_data));
> > > | ^
> > > <inline asm>:1:2: note: instantiated into assembly here
> > > 1 | mov r4, r1
> > > | ^
>
> I have to wonder why Clang is complaining about "mov r4, r1" because
> that certainly should not require "ARMv6 or later". On the face of it,
> this to me looks like a bug in Clang.
This is because the compat vDSO is compiled with '-mthumb' by default
and Nick helpfully pointed out on IRC that prior to ARMv6, one of the
operands to the mov had to be a HI register, which I do see in the ARM:
Encoding T1 ARMv6*, ARMv7 if <Rd> and <Rm> both from R0-R7
ARMv4T, ARMv5T*, ARMv6*, ARMv7 otherwise
MOV<c> <Rd>, <Rm>
Cheers,
Nathan
On Mon, Dec 04, 2023 at 11:13:04AM -0700, Nathan Chancellor wrote:
> Hi Naresh,
>
> On Mon, Dec 04, 2023 at 05:33:26PM +0530, Naresh Kamboju wrote:
> > Following build errors noticed on Linux next-20231204 tag with clang-nightly
> > for arm and arm64.
> >
> > ## Test Regressions (compared to next-20231201)
> > * arm64, build
> > - clang-nightly-defconfig
> > - clang-nightly-defconfig-40bc7ee5
> > - clang-nightly-lkftconfig
> > - clang-nightly-lkftconfig-kselftest
> >
> > * arm, build
> > - clang-nightly-allnoconfig
> > - clang-nightly-axm55xx_defconfig
> > - clang-nightly-bcm2835_defconfig
> > - clang-nightly-clps711x_defconfig
> > - clang-nightly-defconfig
> > - clang-nightly-exynos_defconfig
> > - clang-nightly-imx_v6_v7_defconfig
> > - clang-nightly-keystone_defconfig
> > - clang-nightly-lkftconfig
> > - clang-nightly-lkftconfig-kselftest
> > - clang-nightly-omap2plus_defconfig
> > - clang-nightly-pxa910_defconfig
> > - clang-nightly-s3c6400_defconfig
> > - clang-nightly-s5pv210_defconfig
> > - clang-nightly-sama5_defconfig
> > - clang-nightly-shmobile_defconfig
> > - clang-nightly-tinyconfig
> > - clang-nightly-u8500_defconfig
> > - clang-nightly-vexpress_defconfig
> >
> >
> > Reported-by: Linux Kernel Functional Testing <[email protected]>
> >
> >
> > Build log on arm64:
> > ---------
> > In file included from lib/vdso/gettimeofday.c:5:
> > In file included from include/vdso/datapage.h:135:
> > arch/arm64/include/asm/vdso/compat_gettimeofday.h:152:15: error:
> > instruction variant requires ARMv6 or later
> > 152 | asm volatile("mov %0, %1" : "=r"(ret) : "r"(_vdso_data));
> > | ^
> > <inline asm>:1:2: note: instantiated into assembly here
> > 1 | mov r4, r1
> > | ^
> > In file included from <built-in>:3:
> > lib/vdso/gettimeofday.c:139:3: error: invalid instruction
> > 139 | smp_rmb();
> > | ^
> >
> > Build log on arm:
> > ---------
> > In file included from arch/arm/vfp/vfpmodule.c:23:
> > arch/arm/include/asm/cp15.h:101:2: error: instruction requires: data-barriers
> > 101 | isb();
> > | ^
>
> This is caused by a change to Debian's LLVM that changes the internal
> defaults of the arm-linux-gnueabi and arm-linux-gnueabihf tuples:
>
> https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/commit/907baf024b9a5a1626893d9e731b6c79ccf45c87
>
> We use arm-linux-gnueabi for the kernel (see scripts/Makefile.clang) so
> now we have a hardcoded armv5te CPU, even if we are building for armv7
> or such.
>
> I am still investigating into what (if anything) can be done to resolve
> this on the kernel side. We could potentially revert commit
> ddc72c9659b5 ("kbuild: clang: do not use CROSS_COMPILE for target
> triple") but I am not sure that will save us from that change, as
> tuxmake's CROSS_COMPILE=arm-linux-gnueabihf will cause us to have an
> armv7 CPU even though we may not be building for armv7.
Okay, this is a pretty awful situation the more I look into it :(
The arm64 compat vDSO build is easy enough to fix because we require use
of the integrated assembler, which means we can add '-mcpu=generic' (the
default in LLVM for those files based on my debugging) to those files
and be done with it:
diff --git a/arch/arm64/kernel/vdso32/Makefile b/arch/arm64/kernel/vdso32/Makefile
index 1f911a76c5af..5f5cb722cfc2 100644
--- a/arch/arm64/kernel/vdso32/Makefile
+++ b/arch/arm64/kernel/vdso32/Makefile
@@ -9,6 +9,10 @@ include $(srctree)/lib/vdso/Makefile
ifeq ($(CONFIG_CC_IS_CLANG), y)
CC_COMPAT ?= $(CC)
CC_COMPAT += --target=arm-linux-gnueabi
+# Some distributions (such as Debian) change the default CPU for the
+# arm-linux-gnueabi target triple, which can break the build. Explicitly set
+# the CPU to generic, which is the default for Linux in LLVM.
+CC_COMPAT += -mcpu=generic
else
CC_COMPAT ?= $(CROSS_COMPILE_COMPAT)gcc
endif
The failures for all the ARCH=arm configurations appear to be much more
difficult to fix because the default CPU value changes based on the
'-march' value, which basically means that we would have to hardcode
LLVM's default CPU logic into the kernel's Makefile, which is just not
maintainable in my opinion. Just doing a multi_v7_defconfig build of
arch/arm/ shows the value returned from ARM::getARMCPUForArch() in
llvm/lib/TargetParser/ARMTargetParser.cpp can vary between "arm7tdmi" or
"generic". Supplying '-mcpu=generic' explicitly won't work with
LLVM_IAS=0 because GNU as does not support it and clang just happily
passes it along, even though it does not do that in the implicit default
case.
Sylvestre, I strongly believe you should consider reverting that change
or give us some compiler flag that allows us to fallback to upstream
LLVM's default CPU selection logic. I think that hardcoding Debian's
architecture defintions based on the target triple into the compiler
could cause issues for other projects as well. For example,
'--target=arm-linux-gnueabi -march=armv7-a' won't actually target ARMv7:
$ echo 'int main(void) { asm("dsb"); return 0; }' | \
clang --target=arm-linux-gnueabi -march=armv7-a \
-x c -c -o /dev/null -v -
...
"/usr/bin/clang-17" -cc1 -triple armv7-unknown-linux-gnueabi ...
...
vs.
$ echo 'int main(void) { asm("dsb"); return 0; }' | \
clang --target=arm-linux-gnueabi -march=armv7-a \
-x c -c -o /dev/null -v -
...
"<prefix>/bin/clang-18" -cc1 -triple armv5e-unknown-linux-gnueabi ...
...
Cheers,
Nathan
Hello,
Le 04/12/2023 à 23:33, Nathan Chancellor a écrit :
> On Mon, Dec 04, 2023 at 11:13:04AM -0700, Nathan Chancellor wrote:
>> Hi Naresh,
>>
>> On Mon, Dec 04, 2023 at 05:33:26PM +0530, Naresh Kamboju wrote:
>>> Following build errors noticed on Linux next-20231204 tag with clang-nightly
>>> for arm and arm64.
>>>
>>> ## Test Regressions (compared to next-20231201)
>>> * arm64, build
>>> - clang-nightly-defconfig
>>> - clang-nightly-defconfig-40bc7ee5
>>> - clang-nightly-lkftconfig
>>> - clang-nightly-lkftconfig-kselftest
>>>
>>> * arm, build
>>> - clang-nightly-allnoconfig
>>> - clang-nightly-axm55xx_defconfig
>>> - clang-nightly-bcm2835_defconfig
>>> - clang-nightly-clps711x_defconfig
>>> - clang-nightly-defconfig
>>> - clang-nightly-exynos_defconfig
>>> - clang-nightly-imx_v6_v7_defconfig
>>> - clang-nightly-keystone_defconfig
>>> - clang-nightly-lkftconfig
>>> - clang-nightly-lkftconfig-kselftest
>>> - clang-nightly-omap2plus_defconfig
>>> - clang-nightly-pxa910_defconfig
>>> - clang-nightly-s3c6400_defconfig
>>> - clang-nightly-s5pv210_defconfig
>>> - clang-nightly-sama5_defconfig
>>> - clang-nightly-shmobile_defconfig
>>> - clang-nightly-tinyconfig
>>> - clang-nightly-u8500_defconfig
>>> - clang-nightly-vexpress_defconfig
>>>
>>>
>>> Reported-by: Linux Kernel Functional Testing <[email protected]>
>>>
>>>
>>> Build log on arm64:
>>> ---------
>>> In file included from lib/vdso/gettimeofday.c:5:
>>> In file included from include/vdso/datapage.h:135:
>>> arch/arm64/include/asm/vdso/compat_gettimeofday.h:152:15: error:
>>> instruction variant requires ARMv6 or later
>>> 152 | asm volatile("mov %0, %1" : "=r"(ret) : "r"(_vdso_data));
>>> | ^
>>> <inline asm>:1:2: note: instantiated into assembly here
>>> 1 | mov r4, r1
>>> | ^
>>> In file included from <built-in>:3:
>>> lib/vdso/gettimeofday.c:139:3: error: invalid instruction
>>> 139 | smp_rmb();
>>> | ^
>>>
>>> Build log on arm:
>>> ---------
>>> In file included from arch/arm/vfp/vfpmodule.c:23:
>>> arch/arm/include/asm/cp15.h:101:2: error: instruction requires: data-barriers
>>> 101 | isb();
>>> | ^
>> This is caused by a change to Debian's LLVM that changes the internal
>> defaults of the arm-linux-gnueabi and arm-linux-gnueabihf tuples:
>>
>> https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/commit/907baf024b9a5a1626893d9e731b6c79ccf45c87
>>
>> We use arm-linux-gnueabi for the kernel (see scripts/Makefile.clang) so
>> now we have a hardcoded armv5te CPU, even if we are building for armv7
>> or such.
>>
>> I am still investigating into what (if anything) can be done to resolve
>> this on the kernel side. We could potentially revert commit
>> ddc72c9659b5 ("kbuild: clang: do not use CROSS_COMPILE for target
>> triple") but I am not sure that will save us from that change, as
>> tuxmake's CROSS_COMPILE=arm-linux-gnueabihf will cause us to have an
>> armv7 CPU even though we may not be building for armv7.
> Okay, this is a pretty awful situation the more I look into it :(
>
> The arm64 compat vDSO build is easy enough to fix because we require use
> of the integrated assembler, which means we can add '-mcpu=generic' (the
> default in LLVM for those files based on my debugging) to those files
> and be done with it:
>
> diff --git a/arch/arm64/kernel/vdso32/Makefile b/arch/arm64/kernel/vdso32/Makefile
> index 1f911a76c5af..5f5cb722cfc2 100644
> --- a/arch/arm64/kernel/vdso32/Makefile
> +++ b/arch/arm64/kernel/vdso32/Makefile
> @@ -9,6 +9,10 @@ include $(srctree)/lib/vdso/Makefile
> ifeq ($(CONFIG_CC_IS_CLANG), y)
> CC_COMPAT ?= $(CC)
> CC_COMPAT += --target=arm-linux-gnueabi
> +# Some distributions (such as Debian) change the default CPU for the
> +# arm-linux-gnueabi target triple, which can break the build. Explicitly set
> +# the CPU to generic, which is the default for Linux in LLVM.
> +CC_COMPAT += -mcpu=generic
> else
> CC_COMPAT ?= $(CROSS_COMPILE_COMPAT)gcc
> endif
>
> The failures for all the ARCH=arm configurations appear to be much more
> difficult to fix because the default CPU value changes based on the
> '-march' value, which basically means that we would have to hardcode
> LLVM's default CPU logic into the kernel's Makefile, which is just not
> maintainable in my opinion. Just doing a multi_v7_defconfig build of
> arch/arm/ shows the value returned from ARM::getARMCPUForArch() in
> llvm/lib/TargetParser/ARMTargetParser.cpp can vary between "arm7tdmi" or
> "generic". Supplying '-mcpu=generic' explicitly won't work with
> LLVM_IAS=0 because GNU as does not support it and clang just happily
> passes it along, even though it does not do that in the implicit default
> case.
>
> Sylvestre, I strongly believe you should consider reverting that change
> or give us some compiler flag that allows us to fallback to upstream
> LLVM's default CPU selection logic. I think that hardcoding Debian's
> architecture defintions based on the target triple into the compiler
> could cause issues for other projects as well. For example,
> '--target=arm-linux-gnueabi -march=armv7-a' won't actually target ARMv7:
>
> $ echo 'int main(void) { asm("dsb"); return 0; }' | \
> clang --target=arm-linux-gnueabi -march=armv7-a \
> -x c -c -o /dev/null -v -
> ...
> "/usr/bin/clang-17" -cc1 -triple armv7-unknown-linux-gnueabi ...
> ...
>
> vs.
>
> $ echo 'int main(void) { asm("dsb"); return 0; }' | \
> clang --target=arm-linux-gnueabi -march=armv7-a \
> -x c -c -o /dev/null -v -
> ...
> "<prefix>/bin/clang-18" -cc1 -triple armv5e-unknown-linux-gnueabi ...
> ...
>
I guess it is this patch, right?
https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/commit/97633b6d51ebc8579c5dbecd12a02fb933620620
if so, do you want me to revert it?
Thanks
S
On Mon, Dec 04, 2023 at 11:42:02PM +0100, Sylvestre Ledru wrote:
> Hello,
>
>
> Le 04/12/2023 ? 23:33, Nathan Chancellor a ?crit?:
> > On Mon, Dec 04, 2023 at 11:13:04AM -0700, Nathan Chancellor wrote:
> > > Hi Naresh,
> > >
> > > On Mon, Dec 04, 2023 at 05:33:26PM +0530, Naresh Kamboju wrote:
> > > > Following build errors noticed on Linux next-20231204 tag with clang-nightly
> > > > for arm and arm64.
> > > >
> > > > ## Test Regressions (compared to next-20231201)
> > > > * arm64, build
> > > > - clang-nightly-defconfig
> > > > - clang-nightly-defconfig-40bc7ee5
> > > > - clang-nightly-lkftconfig
> > > > - clang-nightly-lkftconfig-kselftest
> > > >
> > > > * arm, build
> > > > - clang-nightly-allnoconfig
> > > > - clang-nightly-axm55xx_defconfig
> > > > - clang-nightly-bcm2835_defconfig
> > > > - clang-nightly-clps711x_defconfig
> > > > - clang-nightly-defconfig
> > > > - clang-nightly-exynos_defconfig
> > > > - clang-nightly-imx_v6_v7_defconfig
> > > > - clang-nightly-keystone_defconfig
> > > > - clang-nightly-lkftconfig
> > > > - clang-nightly-lkftconfig-kselftest
> > > > - clang-nightly-omap2plus_defconfig
> > > > - clang-nightly-pxa910_defconfig
> > > > - clang-nightly-s3c6400_defconfig
> > > > - clang-nightly-s5pv210_defconfig
> > > > - clang-nightly-sama5_defconfig
> > > > - clang-nightly-shmobile_defconfig
> > > > - clang-nightly-tinyconfig
> > > > - clang-nightly-u8500_defconfig
> > > > - clang-nightly-vexpress_defconfig
> > > >
> > > >
> > > > Reported-by: Linux Kernel Functional Testing <[email protected]>
> > > >
> > > >
> > > > Build log on arm64:
> > > > ---------
> > > > In file included from lib/vdso/gettimeofday.c:5:
> > > > In file included from include/vdso/datapage.h:135:
> > > > arch/arm64/include/asm/vdso/compat_gettimeofday.h:152:15: error:
> > > > instruction variant requires ARMv6 or later
> > > > 152 | asm volatile("mov %0, %1" : "=r"(ret) : "r"(_vdso_data));
> > > > | ^
> > > > <inline asm>:1:2: note: instantiated into assembly here
> > > > 1 | mov r4, r1
> > > > | ^
> > > > In file included from <built-in>:3:
> > > > lib/vdso/gettimeofday.c:139:3: error: invalid instruction
> > > > 139 | smp_rmb();
> > > > | ^
> > > >
> > > > Build log on arm:
> > > > ---------
> > > > In file included from arch/arm/vfp/vfpmodule.c:23:
> > > > arch/arm/include/asm/cp15.h:101:2: error: instruction requires: data-barriers
> > > > 101 | isb();
> > > > | ^
> > > This is caused by a change to Debian's LLVM that changes the internal
> > > defaults of the arm-linux-gnueabi and arm-linux-gnueabihf tuples:
> > >
> > > https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/commit/907baf024b9a5a1626893d9e731b6c79ccf45c87
> > >
> > > We use arm-linux-gnueabi for the kernel (see scripts/Makefile.clang) so
> > > now we have a hardcoded armv5te CPU, even if we are building for armv7
> > > or such.
> > >
> > > I am still investigating into what (if anything) can be done to resolve
> > > this on the kernel side. We could potentially revert commit
> > > ddc72c9659b5 ("kbuild: clang: do not use CROSS_COMPILE for target
> > > triple") but I am not sure that will save us from that change, as
> > > tuxmake's CROSS_COMPILE=arm-linux-gnueabihf will cause us to have an
> > > armv7 CPU even though we may not be building for armv7.
> > Okay, this is a pretty awful situation the more I look into it :(
> >
> > The arm64 compat vDSO build is easy enough to fix because we require use
> > of the integrated assembler, which means we can add '-mcpu=generic' (the
> > default in LLVM for those files based on my debugging) to those files
> > and be done with it:
> >
> > diff --git a/arch/arm64/kernel/vdso32/Makefile b/arch/arm64/kernel/vdso32/Makefile
> > index 1f911a76c5af..5f5cb722cfc2 100644
> > --- a/arch/arm64/kernel/vdso32/Makefile
> > +++ b/arch/arm64/kernel/vdso32/Makefile
> > @@ -9,6 +9,10 @@ include $(srctree)/lib/vdso/Makefile
> > ifeq ($(CONFIG_CC_IS_CLANG), y)
> > CC_COMPAT ?= $(CC)
> > CC_COMPAT += --target=arm-linux-gnueabi
> > +# Some distributions (such as Debian) change the default CPU for the
> > +# arm-linux-gnueabi target triple, which can break the build. Explicitly set
> > +# the CPU to generic, which is the default for Linux in LLVM.
> > +CC_COMPAT += -mcpu=generic
> > else
> > CC_COMPAT ?= $(CROSS_COMPILE_COMPAT)gcc
> > endif
> >
> > The failures for all the ARCH=arm configurations appear to be much more
> > difficult to fix because the default CPU value changes based on the
> > '-march' value, which basically means that we would have to hardcode
> > LLVM's default CPU logic into the kernel's Makefile, which is just not
> > maintainable in my opinion. Just doing a multi_v7_defconfig build of
> > arch/arm/ shows the value returned from ARM::getARMCPUForArch() in
> > llvm/lib/TargetParser/ARMTargetParser.cpp can vary between "arm7tdmi" or
> > "generic". Supplying '-mcpu=generic' explicitly won't work with
> > LLVM_IAS=0 because GNU as does not support it and clang just happily
> > passes it along, even though it does not do that in the implicit default
> > case.
> >
> > Sylvestre, I strongly believe you should consider reverting that change
> > or give us some compiler flag that allows us to fallback to upstream
> > LLVM's default CPU selection logic. I think that hardcoding Debian's
> > architecture defintions based on the target triple into the compiler
> > could cause issues for other projects as well. For example,
> > '--target=arm-linux-gnueabi -march=armv7-a' won't actually target ARMv7:
> >
> > $ echo 'int main(void) { asm("dsb"); return 0; }' | \
> > clang --target=arm-linux-gnueabi -march=armv7-a \
> > -x c -c -o /dev/null -v -
> > ...
> > "/usr/bin/clang-17" -cc1 -triple armv7-unknown-linux-gnueabi ...
> > ...
> >
> > vs.
> >
> > $ echo 'int main(void) { asm("dsb"); return 0; }' | \
> > clang --target=arm-linux-gnueabi -march=armv7-a \
> > -x c -c -o /dev/null -v -
> > ...
> > "<prefix>/bin/clang-18" -cc1 -triple armv5e-unknown-linux-gnueabi ...
> > ...
> >
> I guess it is this patch, right?
>
> https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/commit/97633b6d51ebc8579c5dbecd12a02fb933620620
Right, I should have made that clearer when bringing you in (I linked to
the snapshot version of that change further up in the thread).
> if so, do you want me to revert it?
Yes, I think it should be reverted.
Cheers,
Nathan
On Mon, Dec 4, 2023, at 23:33, Nathan Chancellor wrote:
> On Mon, Dec 04, 2023 at 11:13:04AM -0700, Nathan Chancellor wrote:
>
>> I am still investigating into what (if anything) can be done to resolve
>> this on the kernel side. We could potentially revert commit
>> ddc72c9659b5 ("kbuild: clang: do not use CROSS_COMPILE for target
>> triple") but I am not sure that will save us from that change, as
>> tuxmake's CROSS_COMPILE=arm-linux-gnueabihf will cause us to have an
>> armv7 CPU even though we may not be building for armv7.
>
> Okay, this is a pretty awful situation the more I look into it :(
>
> The arm64 compat vDSO build is easy enough to fix because we require use
> of the integrated assembler, which means we can add '-mcpu=generic' (the
> default in LLVM for those files based on my debugging) to those files
> and be done with it:
>
> diff --git a/arch/arm64/kernel/vdso32/Makefile
> b/arch/arm64/kernel/vdso32/Makefile
> index 1f911a76c5af..5f5cb722cfc2 100644
> --- a/arch/arm64/kernel/vdso32/Makefile
> +++ b/arch/arm64/kernel/vdso32/Makefile
> @@ -9,6 +9,10 @@ include $(srctree)/lib/vdso/Makefile
> ifeq ($(CONFIG_CC_IS_CLANG), y)
> CC_COMPAT ?= $(CC)
> CC_COMPAT += --target=arm-linux-gnueabi
> +# Some distributions (such as Debian) change the default CPU for the
> +# arm-linux-gnueabi target triple, which can break the build.
> Explicitly set
> +# the CPU to generic, which is the default for Linux in LLVM.
> +CC_COMPAT += -mcpu=generic
> else
> CC_COMPAT ?= $(CROSS_COMPILE_COMPAT)gcc
> endif
I'm still trying to follow what is actually going on. I
see that we pass
VDSO_CAFLAGS += -march=armv8-a
which is meant to tell the compiler that we want it to
use ARMv8 compatible instructions. Is the problem that
clang ignores this flag, or do we not pass it correctly?
I would have expected -march=armv8-a to be better than
-mcpu=generic here, as it allows the compiler to use
a wider set of instructions that is still guaranteed to
be available on everything it will run on.
> Sylvestre, I strongly believe you should consider reverting that change
> or give us some compiler flag that allows us to fallback to upstream
> LLVM's default CPU selection logic. I think that hardcoding Debian's
> architecture defintions based on the target triple into the compiler
> could cause issues for other projects as well. For example,
> '--target=arm-linux-gnueabi -march=armv7-a' won't actually target ARMv7:
>
> $ echo 'int main(void) { asm("dsb"); return 0; }' | \
> clang --target=arm-linux-gnueabi -march=armv7-a \
> -x c -c -o /dev/null -v -
> ...
> "/usr/bin/clang-17" -cc1 -triple armv7-unknown-linux-gnueabi ...
> ...
>
> vs.
>
> $ echo 'int main(void) { asm("dsb"); return 0; }' | \
> clang --target=arm-linux-gnueabi -march=armv7-a \
> -x c -c -o /dev/null -v -
> ...
> "<prefix>/bin/clang-18" -cc1 -triple armv5e-unknown-linux-gnueabi ...
> ...
Right, the kernel definitely relies on -march= taking
precedence over the default CPU, the same way that we
tell the compiler to pick a non-default endianess or ABI.
Arnd
On Tue, Dec 05, 2023 at 07:34:40AM +0100, Arnd Bergmann wrote:
> On Mon, Dec 4, 2023, at 23:33, Nathan Chancellor wrote:
> > On Mon, Dec 04, 2023 at 11:13:04AM -0700, Nathan Chancellor wrote:
> >
> >> I am still investigating into what (if anything) can be done to resolve
> >> this on the kernel side. We could potentially revert commit
> >> ddc72c9659b5 ("kbuild: clang: do not use CROSS_COMPILE for target
> >> triple") but I am not sure that will save us from that change, as
> >> tuxmake's CROSS_COMPILE=arm-linux-gnueabihf will cause us to have an
> >> armv7 CPU even though we may not be building for armv7.
> >
> > Okay, this is a pretty awful situation the more I look into it :(
> >
> > The arm64 compat vDSO build is easy enough to fix because we require use
> > of the integrated assembler, which means we can add '-mcpu=generic' (the
> > default in LLVM for those files based on my debugging) to those files
> > and be done with it:
> >
> > diff --git a/arch/arm64/kernel/vdso32/Makefile
> > b/arch/arm64/kernel/vdso32/Makefile
> > index 1f911a76c5af..5f5cb722cfc2 100644
> > --- a/arch/arm64/kernel/vdso32/Makefile
> > +++ b/arch/arm64/kernel/vdso32/Makefile
> > @@ -9,6 +9,10 @@ include $(srctree)/lib/vdso/Makefile
> > ifeq ($(CONFIG_CC_IS_CLANG), y)
> > CC_COMPAT ?= $(CC)
> > CC_COMPAT += --target=arm-linux-gnueabi
> > +# Some distributions (such as Debian) change the default CPU for the
> > +# arm-linux-gnueabi target triple, which can break the build.
> > Explicitly set
> > +# the CPU to generic, which is the default for Linux in LLVM.
> > +CC_COMPAT += -mcpu=generic
> > else
> > CC_COMPAT ?= $(CROSS_COMPILE_COMPAT)gcc
> > endif
>
> I'm still trying to follow what is actually going on. I
> see that we pass
>
> VDSO_CAFLAGS += -march=armv8-a
>
> which is meant to tell the compiler that we want it to
> use ARMv8 compatible instructions. Is the problem that
> clang ignores this flag, or do we not pass it correctly?
>
> I would have expected -march=armv8-a to be better than
> -mcpu=generic here, as it allows the compiler to use
> a wider set of instructions that is still guaranteed to
> be available on everything it will run on.
I should have made it clearer in that message that adding
'-mcpu=generic' was only to avoid the logic added by that Debian LLVM
change, not because I believe the kernel is doing something incorrectly
now. From what I could tell following through LLVM's code, '-march='
determines the default CPU, which is then used to further inform the
full target triple and by overriding the CPU where that patch did, it
was just blowing away the user's request. By providing an '-mcpu='
option explicitly, it would avoid the default selection logic and we
would get what we asked for.
> > Sylvestre, I strongly believe you should consider reverting that change
> > or give us some compiler flag that allows us to fallback to upstream
> > LLVM's default CPU selection logic. I think that hardcoding Debian's
> > architecture defintions based on the target triple into the compiler
> > could cause issues for other projects as well. For example,
> > '--target=arm-linux-gnueabi -march=armv7-a' won't actually target ARMv7:
> >
> > $ echo 'int main(void) { asm("dsb"); return 0; }' | \
> > clang --target=arm-linux-gnueabi -march=armv7-a \
> > -x c -c -o /dev/null -v -
> > ...
> > "/usr/bin/clang-17" -cc1 -triple armv7-unknown-linux-gnueabi ...
> > ...
> >
> > vs.
> >
> > $ echo 'int main(void) { asm("dsb"); return 0; }' | \
> > clang --target=arm-linux-gnueabi -march=armv7-a \
> > -x c -c -o /dev/null -v -
> > ...
> > "<prefix>/bin/clang-18" -cc1 -triple armv5e-unknown-linux-gnueabi ...
> > ...
>
> Right, the kernel definitely relies on -march= taking
> precedence over the default CPU, the same way that we
> tell the compiler to pick a non-default endianess or ABI.
Agreed, I have yet to test the new version of the patch but I see you
and Ard have given input on it, so hopefully it does not have any
problems like this.
Cheers,
Nathan
Le 05/12/2023 à 16:04, Nathan Chancellor a écrit :
> On Tue, Dec 05, 2023 at 07:34:40AM +0100, Arnd Bergmann wrote:
>> On Mon, Dec 4, 2023, at 23:33, Nathan Chancellor wrote:
>>> On Mon, Dec 04, 2023 at 11:13:04AM -0700, Nathan Chancellor wrote:
>>>
>>>> I am still investigating into what (if anything) can be done to resolve
>>>> this on the kernel side. We could potentially revert commit
>>>> ddc72c9659b5 ("kbuild: clang: do not use CROSS_COMPILE for target
>>>> triple") but I am not sure that will save us from that change, as
>>>> tuxmake's CROSS_COMPILE=arm-linux-gnueabihf will cause us to have an
>>>> armv7 CPU even though we may not be building for armv7.
>>>
>>> Okay, this is a pretty awful situation the more I look into it :(
>>>
>>> The arm64 compat vDSO build is easy enough to fix because we require use
>>> of the integrated assembler, which means we can add '-mcpu=generic' (the
>>> default in LLVM for those files based on my debugging) to those files
>>> and be done with it:
>>>
>>> diff --git a/arch/arm64/kernel/vdso32/Makefile
>>> b/arch/arm64/kernel/vdso32/Makefile
>>> index 1f911a76c5af..5f5cb722cfc2 100644
>>> --- a/arch/arm64/kernel/vdso32/Makefile
>>> +++ b/arch/arm64/kernel/vdso32/Makefile
>>> @@ -9,6 +9,10 @@ include $(srctree)/lib/vdso/Makefile
>>> ifeq ($(CONFIG_CC_IS_CLANG), y)
>>> CC_COMPAT ?= $(CC)
>>> CC_COMPAT += --target=arm-linux-gnueabi
>>> +# Some distributions (such as Debian) change the default CPU for the
>>> +# arm-linux-gnueabi target triple, which can break the build.
>>> Explicitly set
>>> +# the CPU to generic, which is the default for Linux in LLVM.
>>> +CC_COMPAT += -mcpu=generic
>>> else
>>> CC_COMPAT ?= $(CROSS_COMPILE_COMPAT)gcc
>>> endif
>>
>> I'm still trying to follow what is actually going on. I
>> see that we pass
>>
>> VDSO_CAFLAGS += -march=armv8-a
>>
>> which is meant to tell the compiler that we want it to
>> use ARMv8 compatible instructions. Is the problem that
>> clang ignores this flag, or do we not pass it correctly?
>>
>> I would have expected -march=armv8-a to be better than
>> -mcpu=generic here, as it allows the compiler to use
>> a wider set of instructions that is still guaranteed to
>> be available on everything it will run on.
>
> I should have made it clearer in that message that adding
> '-mcpu=generic' was only to avoid the logic added by that Debian LLVM
> change, not because I believe the kernel is doing something incorrectly
> now. From what I could tell following through LLVM's code, '-march='
> determines the default CPU, which is then used to further inform the
> full target triple and by overriding the CPU where that patch did, it
> was just blowing away the user's request. By providing an '-mcpu='
> option explicitly, it would avoid the default selection logic and we
> would get what we asked for.
>
>>> Sylvestre, I strongly believe you should consider reverting that change
>>> or give us some compiler flag that allows us to fallback to upstream
>>> LLVM's default CPU selection logic. I think that hardcoding Debian's
>>> architecture defintions based on the target triple into the compiler
>>> could cause issues for other projects as well. For example,
>>> '--target=arm-linux-gnueabi -march=armv7-a' won't actually target ARMv7:
>>>
>>> $ echo 'int main(void) { asm("dsb"); return 0; }' | \
>>> clang --target=arm-linux-gnueabi -march=armv7-a \
>>> -x c -c -o /dev/null -v -
>>> ...
>>> "/usr/bin/clang-17" -cc1 -triple armv7-unknown-linux-gnueabi ...
>>> ...
>>>
>>> vs.
>>>
>>> $ echo 'int main(void) { asm("dsb"); return 0; }' | \
>>> clang --target=arm-linux-gnueabi -march=armv7-a \
>>> -x c -c -o /dev/null -v -
>>> ...
>>> "<prefix>/bin/clang-18" -cc1 -triple armv5e-unknown-linux-gnueabi ...
>>> ...
>>
>> Right, the kernel definitely relies on -march= taking
>> precedence over the default CPU, the same way that we
>> tell the compiler to pick a non-default endianess or ABI.
>
> Agreed, I have yet to test the new version of the patch but I see you
> and Ard have given input on it, so hopefully it does not have any
> problems like this.
Matthias, as cc, pushed a potential fix for debian/ubuntu packages!
https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/commit/01a06b481e5a2610c7387149b58978c3ec281f2c
Cheers,
Sylvestre
On Tue, Dec 05, 2023 at 04:13:29PM +0100, Sylvestre Ledru wrote:
> Le 05/12/2023 ? 16:04, Nathan Chancellor a ?crit?:
> > On Tue, Dec 05, 2023 at 07:34:40AM +0100, Arnd Bergmann wrote:
> > > On Mon, Dec 4, 2023, at 23:33, Nathan Chancellor wrote:
> > > > On Mon, Dec 04, 2023 at 11:13:04AM -0700, Nathan Chancellor wrote:
> > > >
> > > > > I am still investigating into what (if anything) can be done to resolve
> > > > > this on the kernel side. We could potentially revert commit
> > > > > ddc72c9659b5 ("kbuild: clang: do not use CROSS_COMPILE for target
> > > > > triple") but I am not sure that will save us from that change, as
> > > > > tuxmake's CROSS_COMPILE=arm-linux-gnueabihf will cause us to have an
> > > > > armv7 CPU even though we may not be building for armv7.
> > > >
> > > > Okay, this is a pretty awful situation the more I look into it :(
> > > >
> > > > The arm64 compat vDSO build is easy enough to fix because we require use
> > > > of the integrated assembler, which means we can add '-mcpu=generic' (the
> > > > default in LLVM for those files based on my debugging) to those files
> > > > and be done with it:
> > > >
> > > > diff --git a/arch/arm64/kernel/vdso32/Makefile
> > > > b/arch/arm64/kernel/vdso32/Makefile
> > > > index 1f911a76c5af..5f5cb722cfc2 100644
> > > > --- a/arch/arm64/kernel/vdso32/Makefile
> > > > +++ b/arch/arm64/kernel/vdso32/Makefile
> > > > @@ -9,6 +9,10 @@ include $(srctree)/lib/vdso/Makefile
> > > > ifeq ($(CONFIG_CC_IS_CLANG), y)
> > > > CC_COMPAT ?= $(CC)
> > > > CC_COMPAT += --target=arm-linux-gnueabi
> > > > +# Some distributions (such as Debian) change the default CPU for the
> > > > +# arm-linux-gnueabi target triple, which can break the build.
> > > > Explicitly set
> > > > +# the CPU to generic, which is the default for Linux in LLVM.
> > > > +CC_COMPAT += -mcpu=generic
> > > > else
> > > > CC_COMPAT ?= $(CROSS_COMPILE_COMPAT)gcc
> > > > endif
> > >
> > > I'm still trying to follow what is actually going on. I
> > > see that we pass
> > >
> > > VDSO_CAFLAGS += -march=armv8-a
> > >
> > > which is meant to tell the compiler that we want it to
> > > use ARMv8 compatible instructions. Is the problem that
> > > clang ignores this flag, or do we not pass it correctly?
> > >
> > > I would have expected -march=armv8-a to be better than
> > > -mcpu=generic here, as it allows the compiler to use
> > > a wider set of instructions that is still guaranteed to
> > > be available on everything it will run on.
> >
> > I should have made it clearer in that message that adding
> > '-mcpu=generic' was only to avoid the logic added by that Debian LLVM
> > change, not because I believe the kernel is doing something incorrectly
> > now. From what I could tell following through LLVM's code, '-march='
> > determines the default CPU, which is then used to further inform the
> > full target triple and by overriding the CPU where that patch did, it
> > was just blowing away the user's request. By providing an '-mcpu='
> > option explicitly, it would avoid the default selection logic and we
> > would get what we asked for.
> >
> > > > Sylvestre, I strongly believe you should consider reverting that change
> > > > or give us some compiler flag that allows us to fallback to upstream
> > > > LLVM's default CPU selection logic. I think that hardcoding Debian's
> > > > architecture defintions based on the target triple into the compiler
> > > > could cause issues for other projects as well. For example,
> > > > '--target=arm-linux-gnueabi -march=armv7-a' won't actually target ARMv7:
> > > >
> > > > $ echo 'int main(void) { asm("dsb"); return 0; }' | \
> > > > clang --target=arm-linux-gnueabi -march=armv7-a \
> > > > -x c -c -o /dev/null -v -
> > > > ...
> > > > "/usr/bin/clang-17" -cc1 -triple armv7-unknown-linux-gnueabi ...
> > > > ...
> > > >
> > > > vs.
> > > >
> > > > $ echo 'int main(void) { asm("dsb"); return 0; }' | \
> > > > clang --target=arm-linux-gnueabi -march=armv7-a \
> > > > -x c -c -o /dev/null -v -
> > > > ...
> > > > "<prefix>/bin/clang-18" -cc1 -triple armv5e-unknown-linux-gnueabi ...
> > > > ...
> > >
> > > Right, the kernel definitely relies on -march= taking
> > > precedence over the default CPU, the same way that we
> > > tell the compiler to pick a non-default endianess or ABI.
> >
> > Agreed, I have yet to test the new version of the patch but I see you
> > and Ard have given input on it, so hopefully it does not have any
> > problems like this.
>
> Matthias, as cc, pushed a potential fix for debian/ubuntu packages!
> https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/commit/01a06b481e5a2610c7387149b58978c3ec281f2c
Thanks, that version survives my basic testing of both ARCH=arm and
ARCH=arm64 defconfig. I'll holler if our full matrix explodes later (we
have another regression in LLVM right now so we are not testing the
snapshots daily at the moment).
Cheers,
Nathan