Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <d75ef570-c0ad-cea4-687a-d02b560aa676@iscas.ac.cn>
Date:   Sun, 30 Jul 2023 01:36:49 +0800
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Subject: Re: [PATCH v2] riscv: Handle zicsr/zifencei issue between gcc and
 binutils
Content-Language: en-US
To:     Conor Dooley <conor@kernel.org>
Cc:     Paul Walmsley <paul.walmsley@sifive.com>,
        Palmer Dabbelt <palmer@dabbelt.com>,
        Albert Ou <aou@eecs.berkeley.edu>,
        Nathan Chancellor <nathan@kernel.org>,
        Nick Desaulniers <ndesaulniers@google.com>,
        Tom Rix <trix@redhat.com>, Bin Meng <bmeng@tinylab.org>,
        linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org,
        llvm@lists.linux.dev, stable@vger.kernel.org,
        Guo Ren <guoren@kernel.org>
References: <20230726174524.340952-1-xingmingzheng@iscas.ac.cn>
 <20230726-outclass-parade-2ccea9f6688a@spud>
 <10231b81-ea42-26d0-4c11-92851229e658@iscas.ac.cn>
 <20230726-armchair-evasive-427dd245a9fe@spud>
 <20230727-briskness-sappy-e2d9e4c1ef36@spud>
From:   Mingzheng Xing <xingmingzheng@iscas.ac.cn>
Organization: ISCAS
In-Reply-To: <20230727-briskness-sappy-e2d9e4c1ef36@spud>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk

On 7/27/23 15:53, Conor Dooley wrote:
> On Wed, Jul 26, 2023 at 08:41:55PM +0100, Conor Dooley wrote:
>> On Thu, Jul 27, 2023 at 03:34:16AM +0800, Mingzheng Xing wrote:
>>> On 7/27/23 02:02, Conor Dooley wrote:
>>>> This is still broken for:
>>>> CONFIG_CLANG_VERSION=0
>>>> CONFIG_AS_IS_GNU=y
>>>> CONFIG_AS_VERSION=23500
>>>> CONFIG_LD_IS_BFD=y
>>>> CONFIG_LD_VERSION=23500
>>> Do you mean that these CONFIG_* will cause kernel
>>> compilation errors when paired with certain versions of GCC?
>>> Or perhaps I misunderstood your meaning.
>> No, this section is generated by kconfig, although I messed up my
>> trimming of the list & accidentally removed the gcc version, rather
>> than the clang version. Here's the full thing:
>>
>> CONFIG_CC_VERSION_TEXT="riscv64-unknown-linux-gnu-gcc (g2ee5e430018) 12.2.0"
>> CONFIG_CC_IS_GCC=y
>> CONFIG_GCC_VERSION=120200
>> CONFIG_CLANG_VERSION=0
>> CONFIG_AS_IS_GNU=y
>> CONFIG_AS_VERSION=23500
>> CONFIG_LD_IS_BFD=y
>> CONFIG_LD_VERSION=23500
>> CONFIG_LLD_VERSION=0
>> CONFIG_CC_CAN_LINK=y
>> CONFIG_CC_CAN_LINK_STATIC=y
>> CONFIG_CC_HAS_ASM_GOTO_OUTPUT=y
>> CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=y
>> CONFIG_CC_HAS_ASM_INLINE=y
>> CONFIG_CC_HAS_NO_PROFILE_FN_ATTR=y
>> CONFIG_PAHOLE_VERSION=0
>> CONFIG_CONSTRUCTORS=y
>> CONFIG_IRQ_WORK=y
>> CONFIG_BUILDTIME_TABLE_SORT=y
> I think this should sort things out for the even-older binutils case. I
> took the opportunity to fix some grammatical issues that seem to have
> snuck into the help text in your patch & to drop the \, since the
> depends on fits in one line.

hi, Conor.

I reproduced the error with gcc-12.2.0 and binutils-2.35. I tried a
different solution, which I think makes the logic easier. Showing
the new patch code:

diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index 4c07b9189c86..a6fa1eed895c 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -569,25 +569,24 @@ config TOOLCHAIN_HAS_ZIHINTPAUSE

  config TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI
         def_bool y
-       # https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=aed44286efa8ae8717a77d94b51ac3614e2ca6dc
-       depends on AS_IS_GNU && AS_VERSION >= 23800
+       depends on AS_IS_GNU && AS_VERSION >= 23600
         help
-         Newer binutils versions default to ISA spec version 20191213 which
-         moves some instructions from the I extension to the Zicsr and Zifencei
-         extensions.
+         Binutils has supported zicsr and zifencei extensions since version 2.36,
+         try to adapt to the changes by using explicit zicsr and zifencei via
+         -march. For two special cases, where clang<17 or gcc<11.1.0, we will
+         deal with them in CONFIG_TOOLCHAIN_NEEDS_OLD_ISA_SPEC.

  config TOOLCHAIN_NEEDS_OLD_ISA_SPEC
         def_bool y
         depends on TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI
         # https://github.com/llvm/llvm-project/commit/22e199e6afb1263c943c0c0d4498694e15bf8a16
-       depends on CC_IS_CLANG && CLANG_VERSION < 170000
-       help
-         Certain versions of clang do not support zicsr and zifencei via -march
-         but newer versions of binutils require it for the reasons noted in the
-         help text of CONFIG_TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI. This
-         option causes an older ISA spec compatible with these older versions
-         of clang to be passed to GAS, which has the same result as passing zicsr
-         and zifencei to -march.
+       # https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=b03be74bad08c382da47e048007a78fa3fb4ef49
+       depends on (CC_IS_CLANG && CLANG_VERSION < 170000) || (CC_IS_GCC && GCC_VERSION < 110100)
+       help
+         Certain versions of clang and GCC do not support zicsr and zifencei via
+         -march. This option causes an older ISA spec compatible with these older
+         versions of clang and GCC to be passed to GAS, which has the same result
+         as passing zicsr and zifencei to -march.

  config FPU
         bool "FPU support"
diff --git a/arch/riscv/kernel/compat_vdso/Makefile b/arch/riscv/kernel/compat_vdso/Makefile
index 189345773e7e..b86e5e2c3aea 100644
--- a/arch/riscv/kernel/compat_vdso/Makefile
+++ b/arch/riscv/kernel/compat_vdso/Makefile
@@ -11,7 +11,13 @@ compat_vdso-syms += flush_icache
  COMPAT_CC := $(CC)
  COMPAT_LD := $(LD)

-COMPAT_CC_FLAGS := -march=rv32g -mabi=ilp32
+# binutils 2.35 does not support the zifencei extension, but in the ISA
+# spec 20191213, G stands for IMAFD_ZICSR_ZIFENCEI.
+ifdef CONFIG_TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI
+       COMPAT_CC_FLAGS := -march=rv32g -mabi=ilp32
+else
+       COMPAT_CC_FLAGS := -march=rv32imafd -mabi=ilp32
+endif
  COMPAT_LD_FLAGS := -melf32lriscv

  # Disable attributes, as they're useless and break the build.
-- 
2.34.1


Here are the results of my tests:

gcc          binutils       patched         no patch(test on master)
11.4.0     2.35            ok                  ok
11.4.0     2.36            ok                  ok
11.4.0     2.38            ok                  ok

12.2.0     2.35            ok                  error[1]
12.2.0     2.36            ok                  error[2]
12.2.0     2.38            ok                  ok

10.5.0     2.35            ok                  ok
10.5.0     2.36            ok                  ok
10.5.0     2.38            ok                  error[3]

11.1.0     2.35            ok                  ok
11.1.0     2.36            ok                  ok
11.1.0     2.38            ok                  ok

11.2.0     2.35            ok                  ok
11.2.0     2.36            ok                  ok
11.2.0     2.38            ok                  ok

[1]
Assembler messages:
Fatal error: -march=rv32imafd_zicsr_zifencei: Invalid or unknown z ISA extension: 'zifencei'
make[2]: *** [arch/riscv/kernel/compat_vdso/Makefile:47: arch/riscv/kernel/compat_vdso/rt_sigreturn.o] Error 1

[2]
./arch/riscv/include/asm/vdso/gettimeofday.h: Assembler messages:
./arch/riscv/include/asm/vdso/gettimeofday.h:79: Error: unrecognized opcode `csrr a5,0xc01'
./arch/riscv/include/asm/vdso/gettimeofday.h:79: Error: unrecognized opcode `csrr a5,0xc01'
./arch/riscv/include/asm/vdso/gettimeofday.h:79: Error: unrecognized opcode `csrr a5,0xc01'
./arch/riscv/include/asm/vdso/gettimeofday.h:79: Error: unrecognized opcode `csrr a5,0xc01'
make[2]: *** [scripts/Makefile.build:243: arch/riscv/kernel/vdso/vgettimeofday.o] Error 1

[3]
cc1: error: '-march=rv64imac_zicsr_zifencei': unsupported ISA subset 'z'
cc1: error: ABI requires '-march=rv64'
make[2]: *** [scripts/Makefile.build:243: scripts/mod/empty.o] Error 1
make[2]: *** Waiting for unfinished jobs....
cc1: error: '-march=rv64imac_zicsr_zifencei': unsupported ISA subset 'z'
cc1: error: ABI requires '-march=rv64'

> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
> index e1b66ee88323..2d0d89213c97 100644
> --- a/arch/riscv/Kconfig
> +++ b/arch/riscv/Kconfig
> @@ -571,25 +571,27 @@ config TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI
>   	def_bool y
>   	# https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=aed44286efa8ae8717a77d94b51ac3614e2ca6dc
>   	# https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=98416dbb0a62579d4a7a4a76bab51b5b52fec2cd
> -	depends on GCC_VERSION >= 120100 || (AS_IS_GNU && AS_VERSION >= 23800)
> +	depends on AS_IS_GNU
> +	depends on (GCC_VERSION >= 120100 && AS_VERSION >= 23600) || AS_VERSION >= 23800

Tests verified that explicit _ZICSR_ZIFENCEI via -march is required
for gcc>=12.1.0, but this only happens for binutils>=2.36,
binutils 2.35 + gcc>=12.1.0 does not need that. Considering
binutils 2.35 together complicates things. So what do you think
of the above new version patch?

Some more info:
- The commit[4] for patch changes.
- binutils 2.36 supports the zifencei extension[5] and splits
zifencei and zicsr from I[6].

[4] commit 0715372a06ce ("riscv: compat: vdso: Add COMPAT_VDSO base code implementation")
[5] https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=5a1b31e1e1cee6e9f1c92abff59cdcfff0dddf30
[6] https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=729a53530e86972d1143553a415db34e6e01d5d2

Thanks,
Mingzheng.

>   	help
> -	  Binutils-2.38 and GCC-12.1.0 bump default ISA spec to newer version
> +	  Binutils-2.38 and GCC-12.1.0 bump the default ISA spec to version
>   	  20191213 which moves some instructions from the I extension to the
> -	  Zicsr and Zifencei extensions.
> +	  Zicsr and Zifencei extensions. On the other hand, Binutils prior to
> +	  2.35 does not understand these arguments and will error if they are
> +	  passed.
>   
>   config TOOLCHAIN_NEEDS_OLD_ISA_SPEC
>   	def_bool y
>   	depends on TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI
>   	# https://github.com/llvm/llvm-project/commit/22e199e6afb1263c943c0c0d4498694e15bf8a16
>   	# https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=b03be74bad08c382da47e048007a78fa3fb4ef49
> -	depends on (CC_IS_CLANG && CLANG_VERSION < 170000) || \
> -		   (CC_IS_GCC && GCC_VERSION < 110100)
> +	depends on (CC_IS_CLANG && CLANG_VERSION < 170000) || (CC_IS_GCC && GCC_VERSION < 110100)
>   	help
> -	  Certain versions of clang (or GCC) do not support zicsr and zifencei via
> +	  Certain versions of clang and GCC do not support zicsr and zifencei via
>   	  -march but newer versions of binutils require it for the reasons noted
>   	  in the help text of CONFIG_TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI. This
>   	  option causes an older ISA spec compatible with these older versions
> -	  of clang (or GCC) to be passed to GAS, which has the same result as
> +	  of clang and GCC to be passed to GAS, which has the same result as
>   	  passing zicsr and zifencei to -march.
>   
>   config FPU
>
>
> _______________________________________________
> linux-riscv mailing list
> linux-riscv@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-riscv