Commit 86cdd2fdc4e39c388d39c7ba2396d1a9dfd66226 ("kheaders: make headers
archive reproducible") introduced a number of options specific to GNU
tar to the `tar` invocation in `gen_kheaders.sh` script. This causes
the script to fail to work on systems where `tar` is not GNU tar. This
can occur e.g. on recent Gentoo Linux installations that support using
bsdtar from libarchive instead.
To achieve better portability, try using `gtar` over `tar` if the former
is available. This is the name frequently used on systems featuring
support for installing GNU tar alongside another tar implementation.
If `gtar` is not present, `tar` is used for compatibility with regular
systems.
Link: https://bugs.gentoo.org/884061
Reported-by: Sam James <[email protected]>
Tested-by: Sam James <[email protected]>
Signed-off-by: Michał Górny <[email protected]>
---
kernel/gen_kheaders.sh | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
index 473036b43..d2445af7f 100755
--- a/kernel/gen_kheaders.sh
+++ b/kernel/gen_kheaders.sh
@@ -80,11 +80,20 @@ done | cpio --quiet -pdu $cpio_dir >/dev/null 2>&1
find $cpio_dir -type f -print0 |
xargs -0 -P8 -n1 perl -pi -e 'BEGIN {undef $/;}; s/\/\*((?!SPDX).)*?\*\///smg;'
+# The following tar invocations use options specific to GNU tar. On some
+# systems (e.g. Gentoo), `tar` can be a different tool (e.g. bsdtar), and GNU
+# tar can be found as `gtar`.
+if [ -x "$(command -v gtar)" ]; then
+ tar=gtar
+else
+ tar=tar
+fi
+
# Create archive and try to normalize metadata for reproducibility.
# For compatibility with older versions of tar, files are fed to tar
# pre-sorted, as --sort=name might not be available.
find $cpio_dir -printf "./%P\n" | LC_ALL=C sort | \
- tar "${KBUILD_BUILD_TIMESTAMP:+--mtime=$KBUILD_BUILD_TIMESTAMP}" \
+ $tar "${KBUILD_BUILD_TIMESTAMP:+--mtime=$KBUILD_BUILD_TIMESTAMP}" \
--owner=0 --group=0 --numeric-owner --no-recursion \
-I $XZ -cf $tarfile -C $cpio_dir/ -T - > /dev/null
--
2.39.0
On Sat, Dec 17, 2022 at 5:42 PM Michał Górny <[email protected]> wrote:
>
> Commit 86cdd2fdc4e39c388d39c7ba2396d1a9dfd66226 ("kheaders: make headers
> archive reproducible") introduced a number of options specific to GNU
> tar to the `tar` invocation in `gen_kheaders.sh` script. This causes
> the script to fail to work on systems where `tar` is not GNU tar. This
> can occur e.g. on recent Gentoo Linux installations that support using
> bsdtar from libarchive instead.
>
> To achieve better portability, try using `gtar` over `tar` if the former
> is available. This is the name frequently used on systems featuring
> support for installing GNU tar alongside another tar implementation.
> If `gtar` is not present, `tar` is used for compatibility with regular
> systems.
>
> Link: https://bugs.gentoo.org/884061
> Reported-by: Sam James <[email protected]>
> Tested-by: Sam James <[email protected]>
> Signed-off-by: Michał Górny <[email protected]>
> ---
> kernel/gen_kheaders.sh | 11 ++++++++++-
> 1 file changed, 10 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
> index 473036b43..d2445af7f 100755
> --- a/kernel/gen_kheaders.sh
> +++ b/kernel/gen_kheaders.sh
> @@ -80,11 +80,20 @@ done | cpio --quiet -pdu $cpio_dir >/dev/null 2>&1
> find $cpio_dir -type f -print0 |
> xargs -0 -P8 -n1 perl -pi -e 'BEGIN {undef $/;}; s/\/\*((?!SPDX).)*?\*\///smg;'
>
> +# The following tar invocations use options specific to GNU tar. On some
> +# systems (e.g. Gentoo), `tar` can be a different tool (e.g. bsdtar), and GNU
> +# tar can be found as `gtar`.
> +if [ -x "$(command -v gtar)" ]; then
> + tar=gtar
> +else
> + tar=tar
> +fi
> +
> # Create archive and try to normalize metadata for reproducibility.
> # For compatibility with older versions of tar, files are fed to tar
> # pre-sorted, as --sort=name might not be available.
> find $cpio_dir -printf "./%P\n" | LC_ALL=C sort | \
> - tar "${KBUILD_BUILD_TIMESTAMP:+--mtime=$KBUILD_BUILD_TIMESTAMP}" \
> + $tar "${KBUILD_BUILD_TIMESTAMP:+--mtime=$KBUILD_BUILD_TIMESTAMP}" \
> --owner=0 --group=0 --numeric-owner --no-recursion \
> -I $XZ -cf $tarfile -C $cpio_dir/ -T - > /dev/null
I checked the options in bsd tar [1].
Presumably, there is no way to make it work for both
due to incompatible options (--owner vs --uid, --group vs --gid).
Instead of inserting a workaround like this,
another way is to allow users to override a variable
from the command line.
See the top Makefile, for example,
AWK = awk
Then, users can do "make AWK=gawk"
[1] https://www.freebsd.org/cgi/man.cgi?tar(1)
> --
> 2.39.0
>
On Sun, 2022-12-25 at 01:45 +0900, Masahiro Yamada wrote:
> Instead of inserting a workaround like this,
> another way is to allow users to override a variable
> from the command line.
>
>
> See the top Makefile, for example,
>
>
> AWK = awk
>
>
> Then, users can do "make AWK=gawk"
I'm sorry but are you requesting that I remove the check and use $TAR
instead, or allow overriding with TAR, and fall back to gtar or tar
respectively? If the former, should the script unconditionally assume
that TAR will be always set in the environment, or include fallback to
tar for when the script is run directly?
>
--
Best regards,
Michał Górny
> On 25 Dec 2022, at 16:33, Michał Górny <[email protected]> wrote:
>
> On Sun, 2022-12-25 at 01:45 +0900, Masahiro Yamada wrote:
>> Instead of inserting a workaround like this,
>> another way is to allow users to override a variable
>> from the command line.
>>
>>
>> See the top Makefile, for example,
>>
>>
>> AWK = awk
>>
>>
>> Then, users can do "make AWK=gawk"
>
> I'm sorry but are you requesting that I remove the check and use $TAR
> instead, or allow overriding with TAR, and fall back to gtar or tar
> respectively? If the former, should the script unconditionally assume
> that TAR will be always set in the environment, or include fallback to
> tar for when the script is run directly?
Masahiro, what do you reckon? Thanks.
Best,
sam
> On 17 Jan 2023, at 19:01, Sam James <[email protected]> wrote:
>
>
>
>> On 25 Dec 2022, at 16:33, Michał Górny <[email protected]> wrote:
>>
>> On Sun, 2022-12-25 at 01:45 +0900, Masahiro Yamada wrote:
>>> Instead of inserting a workaround like this,
>>> another way is to allow users to override a variable
>>> from the command line.
>>>
>>>
>>> See the top Makefile, for example,
>>>
>>>
>>> AWK = awk
>>>
>>>
>>> Then, users can do "make AWK=gawk"
>>
>> I'm sorry but are you requesting that I remove the check and use $TAR
>> instead, or allow overriding with TAR, and fall back to gtar or tar
>> respectively? If the former, should the script unconditionally assume
>> that TAR will be always set in the environment, or include fallback to
>> tar for when the script is run directly?
>
>
> Masahiro, what do you reckon? Thanks.
>
Ping.
> Best,
> sam
Sam James <[email protected]> writes:
> [[PGP Signed Part:Undecided]]
>
>
>> On 17 Jan 2023, at 19:01, Sam James <[email protected]> wrote:
>>
>>
>>
>>> On 25 Dec 2022, at 16:33, Michał Górny <[email protected]> wrote:
>>>
>>> On Sun, 2022-12-25 at 01:45 +0900, Masahiro Yamada wrote:
>>>> Instead of inserting a workaround like this,
>>>> another way is to allow users to override a variable
>>>> from the command line.
>>>>
>>>>
>>>> See the top Makefile, for example,
>>>>
>>>>
>>>> AWK = awk
>>>>
>>>>
>>>> Then, users can do "make AWK=gawk"
>>>
>>> I'm sorry but are you requesting that I remove the check and use $TAR
>>> instead, or allow overriding with TAR, and fall back to gtar or tar
>>> respectively? If the former, should the script unconditionally assume
>>> that TAR will be always set in the environment, or include fallback to
>>> tar for when the script is run directly?
>>
>>
>> Masahiro, what do you reckon? Thanks.
>>
>
> Ping.
ping^2. We'd really love to get this in to fix a bug for our users in
Gentoo.
>
>> Best,
>> sam
>
>
> [[End of PGP Signed Part]]
I thought I had relied this before, but
something awkward was happening because
the original email was addressed to
[email protected], which is
no longer valid.
Resending from [email protected]...
In the top Makefile, please add
TAR = tar
and export TAR.
In kernel/gen_kheaders.sh,
please replace 'tar' with ${TAR:-tar}
On Gentoo systems, you can do
$ make TAR=gtar
or if you need to run gen_kheaders.sh directly
$ TAR=gtar scripts/gen_kheaders.sh
Thanks.
On Fri, Mar 17, 2023 at 9:05 AM Sam James <[email protected]> wrote:
>
>
> Sam James <[email protected]> writes:
>
> > [[PGP Signed Part:Undecided]]
> >
> >
> >> On 17 Jan 2023, at 19:01, Sam James <[email protected]> wrote:
> >>
> >>
> >>
> >>> On 25 Dec 2022, at 16:33, Michał Górny <[email protected]> wrote:
> >>>
> >>> On Sun, 2022-12-25 at 01:45 +0900, Masahiro Yamada wrote:
> >>>> Instead of inserting a workaround like this,
> >>>> another way is to allow users to override a variable
> >>>> from the command line.
> >>>>
> >>>>
> >>>> See the top Makefile, for example,
> >>>>
> >>>>
> >>>> AWK = awk
> >>>>
> >>>>
> >>>> Then, users can do "make AWK=gawk"
> >>>
> >>> I'm sorry but are you requesting that I remove the check and use $TAR
> >>> instead, or allow overriding with TAR, and fall back to gtar or tar
> >>> respectively? If the former, should the script unconditionally assume
> >>> that TAR will be always set in the environment, or include fallback to
> >>> tar for when the script is run directly?
> >>
> >>
> >> Masahiro, what do you reckon? Thanks.
> >>
> >
> > Ping.
>
> ping^2. We'd really love to get this in to fix a bug for our users in
> Gentoo.
>
> >
> >> Best,
> >> sam
> >
> >
> > [[End of PGP Signed Part]]
>
--
Best Regards
Masahiro Yamada
Commit 86cdd2fdc4e39c388d39c7ba2396d1a9dfd66226 ("kheaders: make headers
archive reproducible") introduced a number of options specific to GNU
tar to the `tar` invocation in `gen_kheaders.sh` script. This causes
the script to fail to work on systems where `tar` is not GNU tar. This
can occur e.g. on recent Gentoo Linux installations that support using
bsdtar from libarchive instead.
Add a `TAR` make variable to make it possible to override the tar
executable used, e.g. by specifying:
make TAR=gtar
Link: https://bugs.gentoo.org/884061
Reported-by: Sam James <[email protected]>
Tested-by: Sam James <[email protected]>
Co-developed-by: Masahiro Yamada <[email protected]>
Signed-off-by: Michał Górny <[email protected]>
---
Makefile | 3 ++-
kernel/gen_kheaders.sh | 2 +-
2 files changed, 3 insertions(+), 2 deletions(-)
diff --git a/Makefile b/Makefile
index 5aeea3d98..50045059c 100644
--- a/Makefile
+++ b/Makefile
@@ -520,6 +520,7 @@ LZMA = lzma
LZ4 = lz4c
XZ = xz
ZSTD = zstd
+TAR = tar
PAHOLE_FLAGS = $(shell PAHOLE=$(PAHOLE) $(srctree)/scripts/pahole-flags.sh)
@@ -599,7 +600,7 @@ export RUSTC RUSTDOC RUSTFMT RUSTC_OR_CLIPPY_QUIET RUSTC_OR_CLIPPY BINDGEN CARGO
export HOSTRUSTC KBUILD_HOSTRUSTFLAGS
export CPP AR NM STRIP OBJCOPY OBJDUMP READELF PAHOLE RESOLVE_BTFIDS LEX YACC AWK INSTALLKERNEL
export PERL PYTHON3 CHECK CHECKFLAGS MAKE UTS_MACHINE HOSTCXX
-export KGZIP KBZIP2 KLZOP LZMA LZ4 XZ ZSTD
+export KGZIP KBZIP2 KLZOP LZMA LZ4 XZ ZSTD TAR
export KBUILD_HOSTCXXFLAGS KBUILD_HOSTLDFLAGS KBUILD_HOSTLDLIBS LDFLAGS_MODULE
export KBUILD_USERCFLAGS KBUILD_USERLDFLAGS
diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
index 1ef9a8751..82d539648 100755
--- a/kernel/gen_kheaders.sh
+++ b/kernel/gen_kheaders.sh
@@ -86,7 +86,7 @@ find $cpio_dir -type f -print0 |
# For compatibility with older versions of tar, files are fed to tar
# pre-sorted, as --sort=name might not be available.
find $cpio_dir -printf "./%P\n" | LC_ALL=C sort | \
- tar "${KBUILD_BUILD_TIMESTAMP:+--mtime=$KBUILD_BUILD_TIMESTAMP}" \
+ ${TAR:-tar} "${KBUILD_BUILD_TIMESTAMP:+--mtime=$KBUILD_BUILD_TIMESTAMP}" \
--owner=0 --group=0 --numeric-owner --no-recursion \
-I $XZ -cf $tarfile -C $cpio_dir/ -T - > /dev/null
--
2.40.0