2023-07-25 17:32:35

by Mingzheng Xing

[permalink] [raw]
Subject: [PATCH] riscv: Handle zicsr/zifencei issue between gcc and binutils

When compiling the kernel with the toolchain composed of GCC >= 12.1.0 and
binutils < 2.38, default ISA spec used when building binutils and GCC, the
following build failure will appear because the
CONFIG_TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI is not turned on.
(i.e, gcc-12.1.0 and binutils-2.36, or gcc-12.3.0 and binutils-2.37, use
default ISA spec.)

CC arch/riscv/kernel/vdso/vgettimeofday.o
<<BUILDDIR>>/arch/riscv/include/asm/vdso/gettimeofday.h: Assembler messages:
<<BUILDDIR>>/arch/riscv/include/asm/vdso/gettimeofday.h:79: Error: unrecognized opcode `csrr a5,0xc01'

Binutils has updated the default ISA spec version, and the community has
responded well to this[1][2][3], but it appears that this is not over yet.

We also need to consider the situation of binutils < 2.38 but
GCC >= 12.1.0, since the combination between different versions of GCC and
binutils is not unique, which is to some extent flexible. GCC release
12.1.0 updated the default ISA spec version in GCC commit[4].

For more information, please refer to:

commit 6df2a016c0c8 ("riscv: fix build with binutils 2.38")
commit e89c2e815e76 ("riscv: Handle zicsr/zifencei issues between clang and binutils")

[1]: https://groups.google.com/a/groups.riscv.org/g/sw-dev/c/aE1ZeHHCYf4
[2]: https://lore.kernel.org/all/[email protected]
[3]: https://lore.kernel.org/all/[email protected]
[4]: https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=98416dbb0a62579d4a7a4a76bab51b5b52fec2cd

Signed-off-by: Mingzheng Xing <[email protected]>
---
arch/riscv/Kconfig | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index 4c07b9189c86..b49cea30f6cc 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -570,11 +570,15 @@ config TOOLCHAIN_HAS_ZIHINTPAUSE
config TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI
def_bool y
# https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=aed44286efa8ae8717a77d94b51ac3614e2ca6dc
- depends on AS_IS_GNU && AS_VERSION >= 23800
+ # https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=98416dbb0a62579d4a7a4a76bab51b5b52fec2cd
+ depends on CC_IS_GCC && GCC_VERSION >= 120100 || \
+ AS_IS_GNU && AS_VERSION >= 23800
help
Newer binutils versions default to ISA spec version 20191213 which
moves some instructions from the I extension to the Zicsr and Zifencei
extensions.
+ Similarly, GCC release 12.1.0 has changed the default ISA spec version to
+ 20191213, so the above situation requires this option to be enabled.

config TOOLCHAIN_NEEDS_OLD_ISA_SPEC
def_bool y
--
2.34.1



2023-07-25 17:35:43

by Nathan Chancellor

[permalink] [raw]
Subject: Re: [PATCH] riscv: Handle zicsr/zifencei issue between gcc and binutils

Hi Mingzheng,

Thanks for the patch!

On Wed, Jul 26, 2023 at 01:04:05AM +0800, Mingzheng Xing wrote:
> When compiling the kernel with the toolchain composed of GCC >= 12.1.0 and
> binutils < 2.38, default ISA spec used when building binutils and GCC, the
> following build failure will appear because the
> CONFIG_TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI is not turned on.
> (i.e, gcc-12.1.0 and binutils-2.36, or gcc-12.3.0 and binutils-2.37, use
> default ISA spec.)
>
> CC arch/riscv/kernel/vdso/vgettimeofday.o
> <<BUILDDIR>>/arch/riscv/include/asm/vdso/gettimeofday.h: Assembler messages:
> <<BUILDDIR>>/arch/riscv/include/asm/vdso/gettimeofday.h:79: Error: unrecognized opcode `csrr a5,0xc01'

The gift that keeps on giving :/

> Binutils has updated the default ISA spec version, and the community has
> responded well to this[1][2][3], but it appears that this is not over yet.
>
> We also need to consider the situation of binutils < 2.38 but
> GCC >= 12.1.0, since the combination between different versions of GCC and
> binutils is not unique, which is to some extent flexible. GCC release
> 12.1.0 updated the default ISA spec version in GCC commit[4].

I suspect this combination is not too common because binutils 2.38 came
out before GCC 12.1.0 but as you note, it is obviously possible. What
toolchain has this combination in the wild, which would be helpful for
documentation purposes?

> For more information, please refer to:
>
> commit 6df2a016c0c8 ("riscv: fix build with binutils 2.38")
> commit e89c2e815e76 ("riscv: Handle zicsr/zifencei issues between clang and binutils")
>
> [1]: https://groups.google.com/a/groups.riscv.org/g/sw-dev/c/aE1ZeHHCYf4
> [2]: https://lore.kernel.org/all/[email protected]
> [3]: https://lore.kernel.org/all/[email protected]
> [4]: https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=98416dbb0a62579d4a7a4a76bab51b5b52fec2cd
>
> Signed-off-by: Mingzheng Xing <[email protected]>
> ---
> arch/riscv/Kconfig | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
> index 4c07b9189c86..b49cea30f6cc 100644
> --- a/arch/riscv/Kconfig
> +++ b/arch/riscv/Kconfig
> @@ -570,11 +570,15 @@ config TOOLCHAIN_HAS_ZIHINTPAUSE
> config TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI
> def_bool y
> # https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=aed44286efa8ae8717a77d94b51ac3614e2ca6dc
> - depends on AS_IS_GNU && AS_VERSION >= 23800
> + # https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=98416dbb0a62579d4a7a4a76bab51b5b52fec2cd
> + depends on CC_IS_GCC && GCC_VERSION >= 120100 || \
> + AS_IS_GNU && AS_VERSION >= 23800

GCC_VERSION will be 0 for clang, so you don't need the CC_IS_GCC check.
With that change, this should be able to stay on one line:

depends on GCC_VERSION >= 120100 || (AS_IS_GNU && AS_VERSION >= 23800)

> help
> Newer binutils versions default to ISA spec version 20191213 which
> moves some instructions from the I extension to the Zicsr and Zifencei
> extensions.
> + Similarly, GCC release 12.1.0 has changed the default ISA spec version to
> + 20191213, so the above situation requires this option to be enabled.
>
> config TOOLCHAIN_NEEDS_OLD_ISA_SPEC
> def_bool y
> --
> 2.34.1
>

Cheers,
Nathan

2023-07-25 19:25:02

by Conor Dooley

[permalink] [raw]
Subject: Re: [PATCH] riscv: Handle zicsr/zifencei issue between gcc and binutils

On Tue, Jul 25, 2023 at 10:23:44AM -0700, Nathan Chancellor wrote:
> Hi Mingzheng,
>
> Thanks for the patch!
>
> On Wed, Jul 26, 2023 at 01:04:05AM +0800, Mingzheng Xing wrote:
> > When compiling the kernel with the toolchain composed of GCC >= 12.1.0 and
> > binutils < 2.38, default ISA spec used when building binutils and GCC, the
> > following build failure will appear because the
> > CONFIG_TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI is not turned on.
> > (i.e, gcc-12.1.0 and binutils-2.36, or gcc-12.3.0 and binutils-2.37, use
> > default ISA spec.)
> >
> > CC arch/riscv/kernel/vdso/vgettimeofday.o
> > <<BUILDDIR>>/arch/riscv/include/asm/vdso/gettimeofday.h: Assembler messages:
> > <<BUILDDIR>>/arch/riscv/include/asm/vdso/gettimeofday.h:79: Error: unrecognized opcode `csrr a5,0xc01'
>
> The gift that keeps on giving :/
>
> > Binutils has updated the default ISA spec version, and the community has
> > responded well to this[1][2][3], but it appears that this is not over yet.
> >
> > We also need to consider the situation of binutils < 2.38 but
> > GCC >= 12.1.0, since the combination between different versions of GCC and
> > binutils is not unique, which is to some extent flexible. GCC release
> > 12.1.0 updated the default ISA spec version in GCC commit[4].
>
> I suspect this combination is not too common because binutils 2.38 came
> out before GCC 12.1.0 but as you note, it is obviously possible. What
> toolchain has this combination in the wild, which would be helpful for
> documentation purposes?

Yeah, that'd be great to know, at least the other niche stuff that we
are working around had a clear use-case (testing LLVM in debian containers)
whereas there's no clear user for this.
That's doubly interesting, as this patch seems to break things for binutils
< 2.35, and if we have to make a trade-off between those too, then it'd
be good to be able to weigh up the options.
Do we perhaps need the misa-spec workaround instead for this case?
Haven't tested that though, trying to dig myself out of email backlog.

> > For more information, please refer to:
> >
> > commit 6df2a016c0c8 ("riscv: fix build with binutils 2.38")
> > commit e89c2e815e76 ("riscv: Handle zicsr/zifencei issues between clang and binutils")
> >
> > [1]: https://groups.google.com/a/groups.riscv.org/g/sw-dev/c/aE1ZeHHCYf4
> > [2]: https://lore.kernel.org/all/[email protected]
> > [3]: https://lore.kernel.org/all/[email protected]
> > [4]: https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=98416dbb0a62579d4a7a4a76bab51b5b52fec2cd

btw, please make these regular Link: tags (with a [N] at EOL) and drop
the space between them and the sign off. Also, this probably needs to be
CC: [email protected] too.

Cheers,
Conor.

> >
> > Signed-off-by: Mingzheng Xing <[email protected]>
> > ---
> > arch/riscv/Kconfig | 6 +++++-
> > 1 file changed, 5 insertions(+), 1 deletion(-)
> >
> > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
> > index 4c07b9189c86..b49cea30f6cc 100644
> > --- a/arch/riscv/Kconfig
> > +++ b/arch/riscv/Kconfig
> > @@ -570,11 +570,15 @@ config TOOLCHAIN_HAS_ZIHINTPAUSE
> > config TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI
> > def_bool y
> > # https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=aed44286efa8ae8717a77d94b51ac3614e2ca6dc
> > - depends on AS_IS_GNU && AS_VERSION >= 23800
> > + # https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=98416dbb0a62579d4a7a4a76bab51b5b52fec2cd
> > + depends on CC_IS_GCC && GCC_VERSION >= 120100 || \
> > + AS_IS_GNU && AS_VERSION >= 23800
>
> GCC_VERSION will be 0 for clang, so you don't need the CC_IS_GCC check.
> With that change, this should be able to stay on one line:
>
> depends on GCC_VERSION >= 120100 || (AS_IS_GNU && AS_VERSION >= 23800)
>
> > help
> > Newer binutils versions default to ISA spec version 20191213 which
> > moves some instructions from the I extension to the Zicsr and Zifencei
> > extensions.
> > + Similarly, GCC release 12.1.0 has changed the default ISA spec version to
> > + 20191213, so the above situation requires this option to be enabled.
> >
> > config TOOLCHAIN_NEEDS_OLD_ISA_SPEC
> > def_bool y
> > --
> > 2.34.1
> >


Attachments:
(No filename) (4.25 kB)
signature.asc (235.00 B)
Download all attachments

2023-07-25 22:38:52

by Conor Dooley

[permalink] [raw]
Subject: Re: [PATCH] riscv: Handle zicsr/zifencei issue between gcc and binutils

On Tue, Jul 25, 2023 at 07:57:54PM +0100, Conor Dooley wrote:
> On Tue, Jul 25, 2023 at 10:23:44AM -0700, Nathan Chancellor wrote:
> > On Wed, Jul 26, 2023 at 01:04:05AM +0800, Mingzheng Xing wrote:
> > > When compiling the kernel with the toolchain composed of GCC >= 12.1.0 and
> > > binutils < 2.38, default ISA spec used when building binutils and GCC, the
> > > following build failure will appear because the
> > > CONFIG_TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI is not turned on.
> > > (i.e, gcc-12.1.0 and binutils-2.36, or gcc-12.3.0 and binutils-2.37, use
> > > default ISA spec.)
> > >
> > > CC arch/riscv/kernel/vdso/vgettimeofday.o
> > > <<BUILDDIR>>/arch/riscv/include/asm/vdso/gettimeofday.h: Assembler messages:
> > > <<BUILDDIR>>/arch/riscv/include/asm/vdso/gettimeofday.h:79: Error: unrecognized opcode `csrr a5,0xc01'
> >
> > The gift that keeps on giving :/
> >
> > > Binutils has updated the default ISA spec version, and the community has
> > > responded well to this[1][2][3], but it appears that this is not over yet.

Also, I just noticed this comment. I disagree with the wording "well",
and more like "with weeping and gnashing of teeth" ;) This stuff is a
huge pain in the ass, and mixing toolchains between LLVM & GNU stuff (or
using an older binutils with a newer GCC) really makes it a lot worse.
Thanks for submitting a fix for this so that Nathan or I didn't have to!

> > >
> > > We also need to consider the situation of binutils < 2.38 but
> > > GCC >= 12.1.0, since the combination between different versions of GCC and
> > > binutils is not unique, which is to some extent flexible. GCC release
> > > 12.1.0 updated the default ISA spec version in GCC commit[4].
> >
> > I suspect this combination is not too common because binutils 2.38 came
> > out before GCC 12.1.0 but as you note, it is obviously possible. What
> > toolchain has this combination in the wild, which would be helpful for
> > documentation purposes?
>
> Yeah, that'd be great to know, at least the other niche stuff that we
> are working around had a clear use-case (testing LLVM in debian containers)
> whereas there's no clear user for this.
> That's doubly interesting, as this patch seems to break things for binutils
> < 2.35, and if we have to make a trade-off between those too, then it'd
> be good to be able to weigh up the options.
> Do we perhaps need the misa-spec workaround instead for this case?
> Haven't tested that though, trying to dig myself out of email backlog.

I don't think the misa-spec stuff is what we need actually. Instead, the
workaround/fix that this patch implements just needs to be constrained to
versions of GAS greater than 2.35.

Thanks,
Conor.

>
> > > For more information, please refer to:
> > >
> > > commit 6df2a016c0c8 ("riscv: fix build with binutils 2.38")
> > > commit e89c2e815e76 ("riscv: Handle zicsr/zifencei issues between clang and binutils")
> > >
> > > [1]: https://groups.google.com/a/groups.riscv.org/g/sw-dev/c/aE1ZeHHCYf4
> > > [2]: https://lore.kernel.org/all/[email protected]
> > > [3]: https://lore.kernel.org/all/[email protected]
> > > [4]: https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=98416dbb0a62579d4a7a4a76bab51b5b52fec2cd
>
> btw, please make these regular Link: tags (with a [N] at EOL) and drop
> the space between them and the sign off. Also, this probably needs to be
> CC: [email protected] too.

> > >
> > > Signed-off-by: Mingzheng Xing <[email protected]>
> > > ---
> > > arch/riscv/Kconfig | 6 +++++-
> > > 1 file changed, 5 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
> > > index 4c07b9189c86..b49cea30f6cc 100644
> > > --- a/arch/riscv/Kconfig
> > > +++ b/arch/riscv/Kconfig
> > > @@ -570,11 +570,15 @@ config TOOLCHAIN_HAS_ZIHINTPAUSE
> > > config TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI
> > > def_bool y
> > > # https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=aed44286efa8ae8717a77d94b51ac3614e2ca6dc
> > > - depends on AS_IS_GNU && AS_VERSION >= 23800
> > > + # https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=98416dbb0a62579d4a7a4a76bab51b5b52fec2cd
> > > + depends on CC_IS_GCC && GCC_VERSION >= 120100 || \
> > > + AS_IS_GNU && AS_VERSION >= 23800
> >
> > GCC_VERSION will be 0 for clang, so you don't need the CC_IS_GCC check.
> > With that change, this should be able to stay on one line:
> >
> > depends on GCC_VERSION >= 120100 || (AS_IS_GNU && AS_VERSION >= 23800)
> >
> > > help
> > > Newer binutils versions default to ISA spec version 20191213 which
> > > moves some instructions from the I extension to the Zicsr and Zifencei
> > > extensions.
> > > + Similarly, GCC release 12.1.0 has changed the default ISA spec version to
> > > + 20191213, so the above situation requires this option to be enabled.
> > >
> > > config TOOLCHAIN_NEEDS_OLD_ISA_SPEC
> > > def_bool y
> > > --
> > > 2.34.1
> > >
>



Attachments:
(No filename) (4.97 kB)
signature.asc (235.00 B)
Download all attachments

2023-07-26 17:35:50

by Mingzheng Xing

[permalink] [raw]
Subject: Re: [PATCH] riscv: Handle zicsr/zifencei issue between gcc and binutils

On 7/26/23 06:17, Conor Dooley wrote:
> On Tue, Jul 25, 2023 at 07:57:54PM +0100, Conor Dooley wrote:
>> On Tue, Jul 25, 2023 at 10:23:44AM -0700, Nathan Chancellor wrote:
>>> On Wed, Jul 26, 2023 at 01:04:05AM +0800, Mingzheng Xing wrote:
>>>> When compiling the kernel with the toolchain composed of GCC >= 12.1.0 and
>>>> binutils < 2.38, default ISA spec used when building binutils and GCC, the
>>>> following build failure will appear because the
>>>> CONFIG_TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI is not turned on.
>>>> (i.e, gcc-12.1.0 and binutils-2.36, or gcc-12.3.0 and binutils-2.37, use
>>>> default ISA spec.)
>>>>
>>>> CC arch/riscv/kernel/vdso/vgettimeofday.o
>>>> <<BUILDDIR>>/arch/riscv/include/asm/vdso/gettimeofday.h: Assembler messages:
>>>> <<BUILDDIR>>/arch/riscv/include/asm/vdso/gettimeofday.h:79: Error: unrecognized opcode `csrr a5,0xc01'
>>> The gift that keeps on giving :/
>>>
>>>> Binutils has updated the default ISA spec version, and the community has
>>>> responded well to this[1][2][3], but it appears that this is not over yet.
> Also, I just noticed this comment. I disagree with the wording "well",
> and more like "with weeping and gnashing of teeth" ;) This stuff is a
> huge pain in the ass, and mixing toolchains between LLVM & GNU stuff (or
> using an older binutils with a newer GCC) really makes it a lot worse.
At least in the usual way , except in my unusual case...
> Thanks for submitting a fix for this so that Nathan or I didn't have to!
Hopefully this patch will make things better.
>>>> We also need to consider the situation of binutils < 2.38 but
>>>> GCC >= 12.1.0, since the combination between different versions of GCC and
>>>> binutils is not unique, which is to some extent flexible. GCC release
>>>> 12.1.0 updated the default ISA spec version in GCC commit[4].
>>> I suspect this combination is not too common because binutils 2.38 came
>>> out before GCC 12.1.0 but as you note, it is obviously possible. What
>>> toolchain has this combination in the wild, which would be helpful for
>>> documentation purposes?
>> Yeah, that'd be great to know, at least the other niche stuff that we
>> are working around had a clear use-case (testing LLVM in debian containers)
>> whereas there's no clear user for this.
>> That's doubly interesting, as this patch seems to break things for binutils
>> < 2.35, and if we have to make a trade-off between those too, then it'd
>> be good to be able to weigh up the options.
>> Do we perhaps need the misa-spec workaround instead for this case?
>> Haven't tested that though, trying to dig myself out of email backlog.
> I don't think the misa-spec stuff is what we need actually. Instead, the
> workaround/fix that this patch implements just needs to be constrained to
> versions of GAS greater than 2.35.
I'm sorry, I didn't quite understand this, could you provide some more
information, thank you very much!
I'll change the handling in v2. After testing it myself, it seems to
work fine.

Thanks,
Mingzheng.
> Thanks,
> Conor.
>
>>>> For more information, please refer to:
>>>>
>>>> commit 6df2a016c0c8 ("riscv: fix build with binutils 2.38")
>>>> commit e89c2e815e76 ("riscv: Handle zicsr/zifencei issues between clang and binutils")
>>>>
>>>> [1]:https://groups.google.com/a/groups.riscv.org/g/sw-dev/c/aE1ZeHHCYf4
>>>> [2]:https://lore.kernel.org/all/[email protected]
>>>> [3]:https://lore.kernel.org/all/[email protected]
>>>> [4]:https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=98416dbb0a62579d4a7a4a76bab51b5b52fec2cd
>> btw, please make these regular Link: tags (with a [N] at EOL) and drop
>> the space between them and the sign off. Also, this probably needs to be
>> CC:[email protected] too.
>>>> Signed-off-by: Mingzheng Xing<[email protected]>
>>>> ---
>>>> arch/riscv/Kconfig | 6 +++++-
>>>> 1 file changed, 5 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
>>>> index 4c07b9189c86..b49cea30f6cc 100644
>>>> --- a/arch/riscv/Kconfig
>>>> +++ b/arch/riscv/Kconfig
>>>> @@ -570,11 +570,15 @@ config TOOLCHAIN_HAS_ZIHINTPAUSE
>>>> config TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI
>>>> def_bool y
>>>> #https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=aed44286efa8ae8717a77d94b51ac3614e2ca6dc
>>>> - depends on AS_IS_GNU && AS_VERSION >= 23800
>>>> + #https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=98416dbb0a62579d4a7a4a76bab51b5b52fec2cd
>>>> + depends on CC_IS_GCC && GCC_VERSION >= 120100 || \
>>>> + AS_IS_GNU && AS_VERSION >= 23800
>>> GCC_VERSION will be 0 for clang, so you don't need the CC_IS_GCC check.
>>> With that change, this should be able to stay on one line:
>>>
>>> depends on GCC_VERSION >= 120100 || (AS_IS_GNU && AS_VERSION >= 23800)
>>>
>>>> help
>>>> Newer binutils versions default to ISA spec version 20191213 which
>>>> moves some instructions from the I extension to the Zicsr and Zifencei
>>>> extensions.
>>>> + Similarly, GCC release 12.1.0 has changed the default ISA spec version to
>>>> + 20191213, so the above situation requires this option to be enabled.
>>>>
>>>> config TOOLCHAIN_NEEDS_OLD_ISA_SPEC
>>>> def_bool y
>>>> --
>>>> 2.34.1
>>>>


2023-07-26 17:38:54

by Mingzheng Xing

[permalink] [raw]
Subject: Re: [PATCH] riscv: Handle zicsr/zifencei issue between gcc and binutils

On 7/26/23 02:57, Conor Dooley wrote:
> On Tue, Jul 25, 2023 at 10:23:44AM -0700, Nathan Chancellor wrote:
>> Hi Mingzheng,
>>
>> Thanks for the patch!
>>
>> On Wed, Jul 26, 2023 at 01:04:05AM +0800, Mingzheng Xing wrote:
>>> When compiling the kernel with the toolchain composed of GCC >= 12.1.0 and
>>> binutils < 2.38, default ISA spec used when building binutils and GCC, the
>>> following build failure will appear because the
>>> CONFIG_TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI is not turned on.
>>> (i.e, gcc-12.1.0 and binutils-2.36, or gcc-12.3.0 and binutils-2.37, use
>>> default ISA spec.)
>>>
>>> CC arch/riscv/kernel/vdso/vgettimeofday.o
>>> <<BUILDDIR>>/arch/riscv/include/asm/vdso/gettimeofday.h: Assembler messages:
>>> <<BUILDDIR>>/arch/riscv/include/asm/vdso/gettimeofday.h:79: Error: unrecognized opcode `csrr a5,0xc01'
>> The gift that keeps on giving :/
>>
>>> Binutils has updated the default ISA spec version, and the community has
>>> responded well to this[1][2][3], but it appears that this is not over yet.
>>>
>>> We also need to consider the situation of binutils < 2.38 but
>>> GCC >= 12.1.0, since the combination between different versions of GCC and
>>> binutils is not unique, which is to some extent flexible. GCC release
>>> 12.1.0 updated the default ISA spec version in GCC commit[4].
>> I suspect this combination is not too common because binutils 2.38 came
>> out before GCC 12.1.0 but as you note, it is obviously possible. What
>> toolchain has this combination in the wild, which would be helpful for
>> documentation purposes?
> Yeah, that'd be great to know, at least the other niche stuff that we
> are working around had a clear use-case (testing LLVM in debian containers)
> whereas there's no clear user for this.
> That's doubly interesting, as this patch seems to break things for binutils
> < 2.35, and if we have to make a trade-off between those too, then it'd
> be good to be able to weigh up the options.
> Do we perhaps need the misa-spec workaround instead for this case?
> Haven't tested that though, trying to dig myself out of email backlog.
Well, what I encountered use-case was a temporary thing caused by the
inconsistent pace of distro package upgrades, but it really happened and
took quite a bit of time to explore why. There are sites like [1] that count
the "Successful Builds" between different GCC and binutils releases,
(Though they don't seem to be updated much...), but it seems to indicate
that all kinds of available pairings are possible.

Before replying to the e-mail, I tested some "extreme" cases and things
seemed to be a little clearer.

1. binutils-2.38 and GCC-12.1.0 each changed default ISA spec version,
    updating from 2.2 to 20191213.

2. binutils>=2.38 or GCC>=12.1.0
    when meet any of these it is recommended to turn on
    TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI, the good outweighs
    the bad. (My personal understanding.)

3. (Extreme case) binutils>=2.38 AND GCC<11.1.0
    GCC-11.1.0 starts to support zicsr and zifencei extension for
-march[2].
    In this case just turn on TOOLCHAIN_NEEDS_OLD_ISA_SPEC.
    (Otherwise compiling the kernel will report an error whether
    TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI is turned on or not.)

4. (Extreme case) GCC>=12.1.0, binutils<2.36
    I tested GCC-12.1.0 + binutils-2.36 and it compiles the kernel
fine(after
    hitting this patch). Also tested GCC-12.1.0 + binutils2.35 and
    GCC-12.1.0 + binutils-2.34.0, but both pairings gave errors and failed
    to produce a usable toolchain (Default ISA versions used). It seems
safe
    to assume that GCC-12.1.0 + binutils<2.35 is almost non-existent.

I'm no expert on toolchains, so thanks for correcting me if I'm wrong
somewhere...

[1] https://wiki.osdev.org/Cross-Compiler_Successful_Builds
[2]
https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=b03be74bad08c382da47e048007a78fa3fb4ef49
>>> For more information, please refer to:
>>>
>>> commit 6df2a016c0c8 ("riscv: fix build with binutils 2.38")
>>> commit e89c2e815e76 ("riscv: Handle zicsr/zifencei issues between clang and binutils")
>>>
>>> [1]:https://groups.google.com/a/groups.riscv.org/g/sw-dev/c/aE1ZeHHCYf4
>>> [2]:https://lore.kernel.org/all/[email protected]
>>> [3]:https://lore.kernel.org/all/[email protected]
>>> [4]:https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=98416dbb0a62579d4a7a4a76bab51b5b52fec2cd
> btw, please make these regular Link: tags (with a [N] at EOL) and drop
> the space between them and the sign off. Also, this probably needs to be
> CC:[email protected] too.
OK, I'll fix it.

Thanks,
Mingzheng.
> Cheers,
> Conor.
>
>>> Signed-off-by: Mingzheng Xing<[email protected]>
>>> ---
>>> arch/riscv/Kconfig | 6 +++++-
>>> 1 file changed, 5 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
>>> index 4c07b9189c86..b49cea30f6cc 100644
>>> --- a/arch/riscv/Kconfig
>>> +++ b/arch/riscv/Kconfig
>>> @@ -570,11 +570,15 @@ config TOOLCHAIN_HAS_ZIHINTPAUSE
>>> config TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI
>>> def_bool y
>>> #https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=aed44286efa8ae8717a77d94b51ac3614e2ca6dc
>>> - depends on AS_IS_GNU && AS_VERSION >= 23800
>>> + #https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=98416dbb0a62579d4a7a4a76bab51b5b52fec2cd
>>> + depends on CC_IS_GCC && GCC_VERSION >= 120100 || \
>>> + AS_IS_GNU && AS_VERSION >= 23800
>> GCC_VERSION will be 0 for clang, so you don't need the CC_IS_GCC check.
>> With that change, this should be able to stay on one line:
>>
>> depends on GCC_VERSION >= 120100 || (AS_IS_GNU && AS_VERSION >= 23800)
>>
>>> help
>>> Newer binutils versions default to ISA spec version 20191213 which
>>> moves some instructions from the I extension to the Zicsr and Zifencei
>>> extensions.
>>> + Similarly, GCC release 12.1.0 has changed the default ISA spec version to
>>> + 20191213, so the above situation requires this option to be enabled.
>>>
>>> config TOOLCHAIN_NEEDS_OLD_ISA_SPEC
>>> def_bool y
>>> --
>>> 2.34.1
>>>


2023-07-26 18:00:36

by Mingzheng Xing

[permalink] [raw]
Subject: Re: [PATCH] riscv: Handle zicsr/zifencei issue between gcc and binutils

On 7/26/23 01:23, Nathan Chancellor wrote:
> Hi Mingzheng,
>
> Thanks for the patch!
>
> On Wed, Jul 26, 2023 at 01:04:05AM +0800, Mingzheng Xing wrote:
>> When compiling the kernel with the toolchain composed of GCC >= 12.1.0 and
>> binutils < 2.38, default ISA spec used when building binutils and GCC, the
>> following build failure will appear because the
>> CONFIG_TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI is not turned on.
>> (i.e, gcc-12.1.0 and binutils-2.36, or gcc-12.3.0 and binutils-2.37, use
>> default ISA spec.)
>>
>> CC arch/riscv/kernel/vdso/vgettimeofday.o
>> <<BUILDDIR>>/arch/riscv/include/asm/vdso/gettimeofday.h: Assembler messages:
>> <<BUILDDIR>>/arch/riscv/include/asm/vdso/gettimeofday.h:79: Error: unrecognized opcode `csrr a5,0xc01'
> The gift that keeps on giving :/
Yeah, but things will get better :)
>> Binutils has updated the default ISA spec version, and the community has
>> responded well to this[1][2][3], but it appears that this is not over yet.
>>
>> We also need to consider the situation of binutils < 2.38 but
>> GCC >= 12.1.0, since the combination between different versions of GCC and
>> binutils is not unique, which is to some extent flexible. GCC release
>> 12.1.0 updated the default ISA spec version in GCC commit[4].
> I suspect this combination is not too common because binutils 2.38 came
> out before GCC 12.1.0 but as you note, it is obviously possible. What
> toolchain has this combination in the wild, which would be helpful for
> documentation purposes?
Actually, this issue was discovered during the upgrade of the distribution
openEuler for RISC-V. It is a temporary phenomenon caused by
inconsistent upgrade speeds of packages such as GCC and binutils.
From my limited understanding, GCC and binutils are not strictly
version-bound in some other distributions, so I can't rule out this issue
happening in other scenarios. But once it happens it can cause problems
with compiling the kernel.
>> For more information, please refer to:
>>
>> commit 6df2a016c0c8 ("riscv: fix build with binutils 2.38")
>> commit e89c2e815e76 ("riscv: Handle zicsr/zifencei issues between clang and binutils")
>>
>> [1]:https://groups.google.com/a/groups.riscv.org/g/sw-dev/c/aE1ZeHHCYf4
>> [2]:https://lore.kernel.org/all/[email protected]
>> [3]:https://lore.kernel.org/all/[email protected]
>> [4]:https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=98416dbb0a62579d4a7a4a76bab51b5b52fec2cd
>>
>> Signed-off-by: Mingzheng Xing<[email protected]>
>> ---
>> arch/riscv/Kconfig | 6 +++++-
>> 1 file changed, 5 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
>> index 4c07b9189c86..b49cea30f6cc 100644
>> --- a/arch/riscv/Kconfig
>> +++ b/arch/riscv/Kconfig
>> @@ -570,11 +570,15 @@ config TOOLCHAIN_HAS_ZIHINTPAUSE
>> config TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI
>> def_bool y
>> #https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=aed44286efa8ae8717a77d94b51ac3614e2ca6dc
>> - depends on AS_IS_GNU && AS_VERSION >= 23800
>> + #https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=98416dbb0a62579d4a7a4a76bab51b5b52fec2cd
>> + depends on CC_IS_GCC && GCC_VERSION >= 120100 || \
>> + AS_IS_GNU && AS_VERSION >= 23800
> GCC_VERSION will be 0 for clang, so you don't need the CC_IS_GCC check.
> With that change, this should be able to stay on one line:
>
> depends on GCC_VERSION >= 120100 || (AS_IS_GNU && AS_VERSION >= 23800)
OK, I'll change it in v2.

Thanks,
Mingzheng.
>> help
>> Newer binutils versions default to ISA spec version 20191213 which
>> moves some instructions from the I extension to the Zicsr and Zifencei
>> extensions.
>> + Similarly, GCC release 12.1.0 has changed the default ISA spec version to
>> + 20191213, so the above situation requires this option to be enabled.
>>
>> config TOOLCHAIN_NEEDS_OLD_ISA_SPEC
>> def_bool y
>> --
>> 2.34.1
>>
> Cheers,
> Nathan
>
> _______________________________________________
> linux-riscv mailing list
> [email protected]
> http://lists.infradead.org/mailman/listinfo/linux-riscv