LinuxLists.cc - [PATCH] kbuild: support byacc as alternative YACC to bison

2019-10-28 21:13:16

Subject: [PATCH] kbuild: support byacc as alternative YACC to bison

2019-10-28 23:46:58

Subject: Re: [PATCH] kbuild: support byacc as alternative YACC to bison

On Tue, Oct 29, 2019 at 1:02 AM Ethan Sommer <[email protected]> wrote:
>
> Switches to a more portable set of flags for generating the same file
> names instead of the bison-specific --defines, uses the more portable -V
> instead of --version, and explicitly defines YYSTYPE in lex.l, which
> bison implicitly defines if not present but byacc does not.
>
> Signed-off-by: Ethan Sommer <[email protected]>
> ---

I applied this patch, and tried the latest byacc,
but I still cannot build kconfig probably because
%destructor is not supported.
byacc does not seem to work anyway.

$ /home/masahiro/bin/yacc -V
/home/masahiro/bin/yacc - 1.9 20190617
masahiro@grover:~/ref/linux$ make YACC=/home/masahiro/bin/yacc defconfig
YACC scripts/kconfig/parser.tab.[ch]
/home/masahiro/bin/yacc: e - line 104 of "scripts/kconfig/parser.y",
syntax error
%destructor {
^
make[1]: *** [scripts/Makefile.host;17: scripts/kconfig/parser.tab.h] Error 1
make[1]: *** Deleting file 'scripts/kconfig/parser.tab.h'
make[1]: *** [scripts/kconfig/parser.tab.h] Deleting file
'scripts/kconfig/parser.tab.c'
make: *** [Makefile;567: defconfig] Error 2

> scripts/Makefile.host | 2 +-
> scripts/genksyms/Makefile | 2 +-
> scripts/genksyms/lex.l | 2 ++
> 3 files changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/Makefile.host b/scripts/Makefile.host
> index 4c51c95d40f4..64e98e1d4825 100644
> --- a/scripts/Makefile.host
> +++ b/scripts/Makefile.host
> @@ -11,7 +11,7 @@ $(obj)/%.lex.c: $(src)/%.l FORCE
> # YACC
> # ---------------------------------------------------------------------------
> quiet_cmd_bison = YACC $(basename $@).[ch]
> - cmd_bison = $(YACC) -o $(basename $@).c --defines=$(basename $@).h -t -l $<
> + cmd_bison = $(YACC) -b $(basename $(basename $@)) -d -t -l $<
>
> $(obj)/%.tab.c $(obj)/%.tab.h: $(src)/%.y FORCE
> $(call if_changed,bison)
> diff --git a/scripts/genksyms/Makefile b/scripts/genksyms/Makefile
> index 78629f515e78..397c2dc8182b 100644
> --- a/scripts/genksyms/Makefile
> +++ b/scripts/genksyms/Makefile
> @@ -15,7 +15,7 @@ genksyms-objs := genksyms.o parse.tab.o lex.lex.o
> ifeq ($(findstring 1,$(KBUILD_EXTRA_WARN)),)
>
> quiet_cmd_bison_no_warn = $(quiet_cmd_bison)
> - cmd_bison_no_warn = $(YACC) --version >/dev/null; \
> + cmd_bison_no_warn = $(YACC) -V >/dev/null; \
> $(cmd_bison) 2>/dev/null
>
> $(obj)/pars%.tab.c $(obj)/pars%.tab.h: $(src)/pars%.y FORCE
> diff --git a/scripts/genksyms/lex.l b/scripts/genksyms/lex.l
> index e265c5d96861..0580c088527f 100644
> --- a/scripts/genksyms/lex.l
> +++ b/scripts/genksyms/lex.l
> @@ -19,6 +19,8 @@
> #include "genksyms.h"
> #include "parse.tab.h"
>
> +extern YYSTYPE yylval;
> +
> /* We've got a two-level lexer here. We let flex do basic tokenization
> and then we categorize those basic tokens in the second stage. */
> #define YY_DECL static int yylex1(void)
> --
> 2.23.0
>

--
Best Regards
Masahiro Yamada

2019-10-29 07:22:10

by Ethan Sommer

[permalink] [raw]

Subject: Re: [PATCH] kbuild: support byacc as alternative YACC to bison

> I applied this patch, and tried the latest byacc,
> but I still cannot build kconfig probably because
> %destructor is not supported.
> byacc does not seem to work anyway.

Hi Masahiro, because I hadn't built it in a while I completely forgot
that to support %destructor byacc needs to be built with
--enable-btyacc, should I update the commit message to include that
detail?

2019-10-29 07:25:07

by Masahiro Yamada

[permalink] [raw]

Subject: Re: [PATCH] kbuild: support byacc as alternative YACC to bison

Hi Ethan,

On Tue, Oct 29, 2019 at 10:41 AM Ethan Sommer <[email protected]> wrote:
>
> > I applied this patch, and tried the latest byacc,
> > but I still cannot build kconfig probably because
> > %destructor is not supported.
> > byacc does not seem to work anyway.
>
> Hi Masahiro, because I hadn't built it in a while I completely forgot
> that to support %destructor byacc needs to be built with
> --enable-btyacc, should I update the commit message to include that
> detail?

OK, I was able to build kconfig with --enable-btyacc.

byacc still cannot build scripts/dtc/dtc-parser.y
Thought?

dtc is not compiled for x86.
Please try ARCH=arm.

masahiro@pug:~/ref/linux$ make ARCH=arm
CROSS_COMPILE=arm-linux-gnueabihf- YACC=/home/masahiro/bin/yacc
scripts/kconfig/conf --syncconfig Kconfig
SYSHDR arch/arm/include/generated/uapi/asm/unistd-common.h
SYSHDR arch/arm/include/generated/uapi/asm/unistd-oabi.h
SYSHDR arch/arm/include/generated/uapi/asm/unistd-eabi.h
HOSTCC scripts/dtc/dtc.o
HOSTCC scripts/dtc/flattree.o
HOSTCC scripts/dtc/fstree.o
HOSTCC scripts/dtc/data.o
HOSTCC scripts/dtc/livetree.o
HOSTCC scripts/dtc/treesource.o
HOSTCC scripts/dtc/srcpos.o
HOSTCC scripts/dtc/checks.o
HOSTCC scripts/dtc/util.o
LEX scripts/dtc/dtc-lexer.lex.c
YACC scripts/dtc/dtc-parser.tab.[ch]
/home/masahiro/bin/yacc: e - line 120 of "scripts/dtc/dtc-parser.y",
syntax error
ERROR(&@2, "Header flags don't match earlier ones");
^
make[1]: *** [scripts/Makefile.host;17: scripts/dtc/dtc-parser.tab.h] Error 1
make[1]: *** Deleting file 'scripts/dtc/dtc-parser.tab.h'
make[1]: *** [scripts/dtc/dtc-parser.tab.h] Deleting file
'scripts/dtc/dtc-parser.tab.c'
make: *** [Makefile;1260: scripts_dtc] Error 2

--
Best Regards
Masahiro Yamada

2019-10-29 19:43:35

by Ethan Sommer

[permalink] [raw]

Subject: [PATCH v2] kbuild: support byacc as alternative YACC to bison

Switches to a more portable set of flags for generating the same file
names instead of the bison-specific --defines, uses the more portable -V
instead of --version, and explicitly defines YYSTYPE in lex.l, which
bison implicitly defines if not present but byacc does not.

Add %locations to dtc-parser.y to explicitly enable location tracking
for byacc, and define YYERROR_CALL explicitly to prevent the locations
directive from causing it to be defined to a 2-parameter call to
yyerror, which dtc-parser.y defines to accept one parameter.

Requires byacc to be built with --enable-btyacc.

Signed-off-by: Ethan Sommer <[email protected]>
---
scripts/Makefile.host | 2 +-
scripts/dtc/dtc-parser.y | 4 ++++
scripts/genksyms/Makefile | 2 +-
scripts/genksyms/lex.l | 2 ++
4 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/scripts/Makefile.host b/scripts/Makefile.host
index 4c51c95d40f4..64e98e1d4825 100644
--- a/scripts/Makefile.host
+++ b/scripts/Makefile.host
@@ -11,7 +11,7 @@ $(obj)/%.lex.c: $(src)/%.l FORCE
# YACC
# ---------------------------------------------------------------------------
quiet_cmd_bison = YACC $(basename $@).[ch]
- cmd_bison = $(YACC) -o $(basename $@).c --defines=$(basename $@).h -t -l $<
+ cmd_bison = $(YACC) -b $(basename $(basename $@)) -d -t -l $<

$(obj)/%.tab.c $(obj)/%.tab.h: $(src)/%.y FORCE
$(call if_changed,bison)
diff --git a/scripts/dtc/dtc-parser.y b/scripts/dtc/dtc-parser.y
index 2ed4dc1f07fd..40dcf4f149da 100644
--- a/scripts/dtc/dtc-parser.y
+++ b/scripts/dtc/dtc-parser.y
@@ -2,6 +2,8 @@
/*
* (C) Copyright David Gibson <[email protected]>, IBM Corporation. 2005.
*/
+%locations
+
%{
#include <stdio.h>
#include <inttypes.h>
@@ -17,6 +19,8 @@ extern void yyerror(char const *s);
treesource_error = true; \
} while (0)

+#define YYERROR_CALL(msg) yyerror(msg)
+
extern struct dt_info *parser_output;
extern bool treesource_error;
%}
diff --git a/scripts/genksyms/Makefile b/scripts/genksyms/Makefile
index 78629f515e78..397c2dc8182b 100644
--- a/scripts/genksyms/Makefile
+++ b/scripts/genksyms/Makefile
@@ -15,7 +15,7 @@ genksyms-objs := genksyms.o parse.tab.o lex.lex.o
ifeq ($(findstring 1,$(KBUILD_EXTRA_WARN)),)

quiet_cmd_bison_no_warn = $(quiet_cmd_bison)
- cmd_bison_no_warn = $(YACC) --version >/dev/null; \
+ cmd_bison_no_warn = $(YACC) -V >/dev/null; \
$(cmd_bison) 2>/dev/null

$(obj)/pars%.tab.c $(obj)/pars%.tab.h: $(src)/pars%.y FORCE
diff --git a/scripts/genksyms/lex.l b/scripts/genksyms/lex.l
index e265c5d96861..0580c088527f 100644
--- a/scripts/genksyms/lex.l
+++ b/scripts/genksyms/lex.l
@@ -19,6 +19,8 @@
#include "genksyms.h"
#include "parse.tab.h"

+extern YYSTYPE yylval;
+
/* We've got a two-level lexer here. We let flex do basic tokenization
and then we categorize those basic tokens in the second stage. */
#define YY_DECL static int yylex1(void)
--
2.23.0

2019-10-30 03:02:25

by Masahiro Yamada

[permalink] [raw]

Subject: Re: [PATCH v2] kbuild: support byacc as alternative YACC to bison

On Wed, Oct 30, 2019 at 12:02 AM Ethan Sommer <[email protected]> wrote:
>
> Switches to a more portable set of flags for generating the same file
> names instead of the bison-specific --defines, uses the more portable -V
> instead of --version, and explicitly defines YYSTYPE in lex.l, which
> bison implicitly defines if not present but byacc does not.
>
> Add %locations to dtc-parser.y to explicitly enable location tracking
> for byacc, and define YYERROR_CALL explicitly to prevent the locations
> directive from causing it to be defined to a 2-parameter call to
> yyerror, which dtc-parser.y defines to accept one parameter.
>
> Requires byacc to be built with --enable-btyacc.
>
> Signed-off-by: Ethan Sommer <[email protected]>
> ---
> scripts/Makefile.host | 2 +-
> scripts/dtc/dtc-parser.y | 4 ++++
> scripts/genksyms/Makefile | 2 +-
> scripts/genksyms/lex.l | 2 ++
> 4 files changed, 8 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/Makefile.host b/scripts/Makefile.host
> index 4c51c95d40f4..64e98e1d4825 100644
> --- a/scripts/Makefile.host
> +++ b/scripts/Makefile.host
> @@ -11,7 +11,7 @@ $(obj)/%.lex.c: $(src)/%.l FORCE
> # YACC
> # ---------------------------------------------------------------------------
> quiet_cmd_bison = YACC $(basename $@).[ch]
> - cmd_bison = $(YACC) -o $(basename $@).c --defines=$(basename $@).h -t -l $<
> + cmd_bison = $(YACC) -b $(basename $(basename $@)) -d -t -l $<

Hmm, this is unfortunate since there is no common way to
specify the header path directly.

I am not sure how much effort we should invent
to support non-GNU implementation
since we already rely on various GNU tools.

If we decide to support byacc,
we must carry the restriction
that bans GNU-extension.

> $(obj)/%.tab.c $(obj)/%.tab.h: $(src)/%.y FORCE
> $(call if_changed,bison)
> diff --git a/scripts/dtc/dtc-parser.y b/scripts/dtc/dtc-parser.y
> index 2ed4dc1f07fd..40dcf4f149da 100644
> --- a/scripts/dtc/dtc-parser.y
> +++ b/scripts/dtc/dtc-parser.y
> @@ -2,6 +2,8 @@
> /*
> * (C) Copyright David Gibson <[email protected]>, IBM Corporation. 2005.
> */
> +%locations
> +
> %{
> #include <stdio.h>
> #include <inttypes.h>
> @@ -17,6 +19,8 @@ extern void yyerror(char const *s);
> treesource_error = true; \
> } while (0)
>
> +#define YYERROR_CALL(msg) yyerror(msg)
> +
> extern struct dt_info *parser_output;
> extern bool treesource_error;
> %}
> diff --git a/scripts/genksyms/Makefile b/scripts/genksyms/Makefile
> index 78629f515e78..397c2dc8182b 100644
> --- a/scripts/genksyms/Makefile
> +++ b/scripts/genksyms/Makefile
> @@ -15,7 +15,7 @@ genksyms-objs := genksyms.o parse.tab.o lex.lex.o
> ifeq ($(findstring 1,$(KBUILD_EXTRA_WARN)),)
>
> quiet_cmd_bison_no_warn = $(quiet_cmd_bison)
> - cmd_bison_no_warn = $(YACC) --version >/dev/null; \
> + cmd_bison_no_warn = $(YACC) -V >/dev/null; \
> $(cmd_bison) 2>/dev/null
>
> $(obj)/pars%.tab.c $(obj)/pars%.tab.h: $(src)/pars%.y FORCE
> diff --git a/scripts/genksyms/lex.l b/scripts/genksyms/lex.l
> index e265c5d96861..0580c088527f 100644
> --- a/scripts/genksyms/lex.l
> +++ b/scripts/genksyms/lex.l
> @@ -19,6 +19,8 @@
> #include "genksyms.h"
> #include "parse.tab.h"
>
> +extern YYSTYPE yylval;
> +
> /* We've got a two-level lexer here. We let flex do basic tokenization
> and then we categorize those basic tokens in the second stage. */
> #define YY_DECL static int yylex1(void)
> --
> 2.23.0
>

--
Best Regards
Masahiro Yamada

2019-11-03 20:31:29

by Ethan Sommer

[permalink] [raw]

Subject: Re: [PATCH v2] kbuild: support byacc as alternative YACC to bison

> Hmm, this is unfortunate since there is no common way to
> specify the header path directly.
>
> I am not sure how much effort we should invent
> to support non-GNU implementation
> since we already rely on various GNU tools.
>
> If we decide to support byacc,
> we must carry the restriction
> that bans GNU-extension.

I just realized that I accidentally only responded to Masahiro with my
previous email from a few days ago so I'll just quote it here:
"I feel like changing 10 lines to support a different yacc
implementation that supports most of the GNU-specific features here
isn't the same thing as banning all GNU extensions, and in regards to
the file prefix, the method in my patch creates the same file names as
the bison-specific one for the 3 cases it is used for, and the flags
used for it are POSIX yacc compatible. In my opinion increasing
compatibility shouldn't have to be all or nothing, and it makes sense to
make changes that increase compatibility without outright banning GNU
extensions entirely."

Aside from that, patch to dtc has just been applied, so pulling the latest
upstream changes as well as the v2 of this patch should be what would be
needed to support byacc.

2019-11-11 18:47:25

by Dmitry Golovin

[permalink] [raw]

Subject: Re: [PATCH v2] kbuild: support byacc as alternative YACC to bison

> Hmm, this is unfortunate since there is no common way to
> specify the header path directly.
>
> I am not sure how much effort we should invent
> to support non-GNU implementation
> since we already rely on various GNU tools.
>
> If we decide to support byacc,
> we must carry the restriction
> that bans GNU-extension.

In fact Linux now can be built without using GNU
binutils and using LLVM tools instead. It's just
one architecture and a specific config now, but
eventually the others will be built too. You can
follow the progress here:
https://github.com/ClangBuiltLinux/continuous-integration/issues/73

I believe that compatibility with different
alternative tools is a good thing as long as it
doesn't add unwanted complexity. And as this patch
is just changing command-line flags to their
portable variants and explicitly adds a couple of
definitions that are presumed by GNU bison, I
can't see a problem with it.

Regards,
Dmitry