kmod 28 supports modules compressed in zstd format so let's add this possibility to kernel.
V2 -> V3
* Fix a typo
V1 -> V2
* Rebuild against linux-kbuild tree
Signed-off-by: Piotr Gorski <[email protected]>
---
init/Kconfig | 8 +++++++-
scripts/Makefile.modinst | 6 ++++++
2 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/init/Kconfig b/init/Kconfig
index 510f6fcd9b7f..b5744d32c4df 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -2242,7 +2242,7 @@ choice
Please note that the tool used to load modules needs to support the
corresponding algorithm. module-init-tools MAY support gzip, and kmod
- MAY support gzip and xz.
+ MAY support gzip, xz and zstd.
Your build system needs to provide the appropriate compression tool
to compress the modules.
@@ -2267,6 +2267,12 @@ config MODULE_COMPRESS_XZ
Compress modules with XZ. The installed modules are suffixed
with .ko.xz.
+config MODULE_COMPRESS_ZSTD
+ bool "ZSTD"
+ help
+ Compress modules with ZSTD. The installed modules are suffixed
+ with .ko.zst.
+
endchoice
config MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS
diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
index 191408f7a91a..f9fa2a3808b2 100644
--- a/scripts/Makefile.modinst
+++ b/scripts/Makefile.modinst
@@ -21,6 +21,7 @@ endif
suffix-y :=
suffix-$(CONFIG_MODULE_COMPRESS_GZIP) := .gz
suffix-$(CONFIG_MODULE_COMPRESS_XZ) := .xz
+suffix-$(CONFIG_MODULE_COMPRESS_ZSTD) := .zst
modules := $(patsubst $(extmod_prefix)%, $(dst)/%$(suffix-y), $(modules))
@@ -95,6 +96,8 @@ quiet_cmd_gzip = GZIP $@
cmd_gzip = $(KGZIP) -n -f $<
quiet_cmd_xz = XZ $@
cmd_xz = $(XZ) --lzma2=dict=2MiB -f $<
+quiet_cmd_zstd = ZSTD $@
+ cmd_zstd = $(ZSTD) -T0 --rm -f -q $<
$(dst)/%.ko.gz: $(dst)/%.ko FORCE
$(call cmd,gzip)
@@ -102,6 +105,9 @@ $(dst)/%.ko.gz: $(dst)/%.ko FORCE
$(dst)/%.ko.xz: $(dst)/%.ko FORCE
$(call cmd,xz)
+$(dst)/%.ko.zst: $(dst)/%.ko FORCE
+ $(call cmd,zstd)
+
PHONY += FORCE
FORCE:
--
2.31.0.97.g1424303384
On Thu, Apr 8, 2021 at 1:09 AM Piotr Gorski <[email protected]> wrote:
>
> kmod 28 supports modules compressed in zstd format so let's add this possibility to kernel.
>
> V2 -> V3
>
> * Fix a typo
>
> V1 -> V2
>
> * Rebuild against linux-kbuild tree
>
> Signed-off-by: Piotr Gorski <[email protected]>
Applied to linux-kbuild. Thanks!
I slightly changed to fix the log alignment.
diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
index f9fa2a3808b2..ff9b09e4cfca 100644
--- a/scripts/Makefile.modinst
+++ b/scripts/Makefile.modinst
@@ -96,7 +96,7 @@ quiet_cmd_gzip = GZIP $@
cmd_gzip = $(KGZIP) -n -f $<
quiet_cmd_xz = XZ $@
cmd_xz = $(XZ) --lzma2=dict=2MiB -f $<
-quiet_cmd_zstd = ZSTD $@
+quiet_cmd_zstd = ZSTD $@
cmd_zstd = $(ZSTD) -T0 --rm -f -q $<
$(dst)/%.ko.gz: $(dst)/%.ko FORCE
> ---
> init/Kconfig | 8 +++++++-
> scripts/Makefile.modinst | 6 ++++++
> 2 files changed, 13 insertions(+), 1 deletion(-)
>
> diff --git a/init/Kconfig b/init/Kconfig
> index 510f6fcd9b7f..b5744d32c4df 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -2242,7 +2242,7 @@ choice
>
> Please note that the tool used to load modules needs to support the
> corresponding algorithm. module-init-tools MAY support gzip, and kmod
> - MAY support gzip and xz.
> + MAY support gzip, xz and zstd.
>
> Your build system needs to provide the appropriate compression tool
> to compress the modules.
> @@ -2267,6 +2267,12 @@ config MODULE_COMPRESS_XZ
> Compress modules with XZ. The installed modules are suffixed
> with .ko.xz.
>
> +config MODULE_COMPRESS_ZSTD
> + bool "ZSTD"
> + help
> + Compress modules with ZSTD. The installed modules are suffixed
> + with .ko.zst.
> +
> endchoice
>
> config MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS
> diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
> index 191408f7a91a..f9fa2a3808b2 100644
> --- a/scripts/Makefile.modinst
> +++ b/scripts/Makefile.modinst
> @@ -21,6 +21,7 @@ endif
> suffix-y :=
> suffix-$(CONFIG_MODULE_COMPRESS_GZIP) := .gz
> suffix-$(CONFIG_MODULE_COMPRESS_XZ) := .xz
> +suffix-$(CONFIG_MODULE_COMPRESS_ZSTD) := .zst
>
> modules := $(patsubst $(extmod_prefix)%, $(dst)/%$(suffix-y), $(modules))
>
> @@ -95,6 +96,8 @@ quiet_cmd_gzip = GZIP $@
> cmd_gzip = $(KGZIP) -n -f $<
> quiet_cmd_xz = XZ $@
> cmd_xz = $(XZ) --lzma2=dict=2MiB -f $<
> +quiet_cmd_zstd = ZSTD $@
> + cmd_zstd = $(ZSTD) -T0 --rm -f -q $<
>
> $(dst)/%.ko.gz: $(dst)/%.ko FORCE
> $(call cmd,gzip)
> @@ -102,6 +105,9 @@ $(dst)/%.ko.gz: $(dst)/%.ko FORCE
> $(dst)/%.ko.xz: $(dst)/%.ko FORCE
> $(call cmd,xz)
>
> +$(dst)/%.ko.zst: $(dst)/%.ko FORCE
> + $(call cmd,zstd)
> +
> PHONY += FORCE
> FORCE:
>
> --
> 2.31.0.97.g1424303384
>
--
Best Regards
Masahiro Yamada
No, the --rm option is essential. xz and gzip have the --rm option built in as opposed to zstd, which is why I used it. I've been using zstd module compression since last december (although I set a different compression level on mine) and everything works fine. Oleksandr also tested it at his place and didn't report any objections.
On Thu, Apr 8, 2021 at 11:05 PM Piotr Gorski <[email protected]> wrote:
>
> No, the --rm option is essential. xz and gzip have the --rm option built in as opposed to zstd, which is why I used it. I've been using zstd module compression since last december (although I set a different compression level on mine) and everything works fine. Oleksandr also tested it at his place and didn't report any objections.
[ CC me I am not subscribed to linux-kbuild or linux-kernel ]
[ CC Nick ]
Unfortunately, I do not find my initial posting which has all information.
I add the link to the thread on linux-kbuild ML.
So, I gave you as much information as I have (linux-config, make-line
etc.) and you write "everything works fine"?
What do you mean by "everything" - different compressors and none?
Is that working "fine"?
What build environment do you use?
Here: Debian/testing AMD64.
Did you try with...
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_DWARF5=y
CONFIG_MODULE_COMPRESS_ZSTD=y
...Kconfigs enabled?
As said I use builddeb from scripts directory to generate my Debian packages.
Any chance you can test with builddeb?
I have enabled Clang-LTO Kconfig.
Tried with Clang-LTO Kconfig?
This worked *before* and *after*...
kbuild: add support for zstd compressed modules
kbuild: remove CONFIG_MODULE_COMPRESS (CC Nick as he is listed as a
reviewer here)
... not within my build-environment.
For me this is a *regression*.
- Sedat -
[1] https://marc.info/?t=161790914600002&r=1&w=2
I originally posted the patch in a different form [1] even before Masahiro's changes.
I've been testing this solution since December last year and posted it in March this year,
after I made sure everything was working fine. This patch was tested by Oleksandr and he also didn't report any objections. [2]
Masahiro notified me about the planned changes [3] and asked me to resend this patch, adjusted to those changes, which I did.
My current logs:
lucjan@archlinux ~ $ zgrep CONFIG_DEBUG_INFO /proc/config.gz
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_COMPRESSED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
CONFIG_DEBUG_INFO_DWARF4=y
CONFIG_DEBUG_INFO_BTF=y
CONFIG_DEBUG_INFO_BTF_MODULES=y
lucjan@archlinux ~ $ zgrep CONFIG_MODULE_COMPRESS_ZSTD /proc/config.gz
CONFIG_MODULE_COMPRESS_ZSTD=y
CONFIG_MODULE_COMPRESS_ZSTD_LEVEL=19
Pay no attention to CONFIG_MODULE_COMPRESS_ZSTD_LEVEL as this is not in the upstream, it's an additional patch I use.
The only difference - I don't use clang. Maybe those who use will comment on this.
I relied on the opinions of Oleksander and a dozen other users who reported no errors in using zstd module compression.
[1] https://marc.info/?l=linux-kbuild&m=161710402402989&w=2
[2] https://marc.info/?l=linux-kbuild&m=161710503403517&w=2
[3] https://marc.info/?l=linux-kbuild&m=161780602730829&w=2
On Fri, Apr 9, 2021 at 1:10 PM Piotr Gorski <[email protected]> wrote:
>
> I originally posted the patch in a different form [1] even before Masahiro's changes.
> I've been testing this solution since December last year and posted it in March this year,
> after I made sure everything was working fine. This patch was tested by Oleksandr and he also didn't report any objections. [2]
>
> Masahiro notified me about the planned changes [3] and asked me to resend this patch, adjusted to those changes, which I did.
>
> My current logs:
>
> lucjan@archlinux ~ $ zgrep CONFIG_DEBUG_INFO /proc/config.gz
> CONFIG_DEBUG_INFO=y
> # CONFIG_DEBUG_INFO_REDUCED is not set
> # CONFIG_DEBUG_INFO_COMPRESSED is not set
> # CONFIG_DEBUG_INFO_SPLIT is not set
> CONFIG_DEBUG_INFO_DWARF4=y
> CONFIG_DEBUG_INFO_BTF=y
> CONFIG_DEBUG_INFO_BTF_MODULES=y
> lucjan@archlinux ~ $ zgrep CONFIG_MODULE_COMPRESS_ZSTD /proc/config.gz
> CONFIG_MODULE_COMPRESS_ZSTD=y
> CONFIG_MODULE_COMPRESS_ZSTD_LEVEL=19
>
> Pay no attention to CONFIG_MODULE_COMPRESS_ZSTD_LEVEL as this is not in the upstream, it's an additional patch I use.
>
> The only difference - I don't use clang. Maybe those who use will comment on this.
> I relied on the opinions of Oleksander and a dozen other users who reported no errors in using zstd module compression.
>
> [1] https://marc.info/?l=linux-kbuild&m=161710402402989&w=2
>
> [2] https://marc.info/?l=linux-kbuild&m=161710503403517&w=2
>
> [3] https://marc.info/?l=linux-kbuild&m=161780602730829&w=2
I am a big fan of ZSTD and have it as default in all available Linux
Kconfigs and Debian's initramfs-tools.
So, I am highly interested in getting this fixed.
Unfortunately, I have thrown away my yesterday's Clang-LTO build and
switched to Clang-CFI with builddeb - should do handle the same way.
I see three iwlwifi.ko (as an example):
$ LC_ALL=C ll drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
-rw-r--r-- 1 dileks dileks 8.2M Apr 9 11:07
drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
$ file drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
drivers/net/wireless/intel/iwlwifi/iwlwifi.ko: ELF 64-bit LSB
relocatable, x86-64, version 1 (SYSV),
BuildID[sha1]=78d593f4fd2b8efe81caeb8f1ea729107a33e244, with
debug_info, not stripped
That iwlwifi.ko with debug-info is optimized when moving to
debian/linux-image-dbg directory:
$ LC_ALL=C ll debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
-rw-r--r-- 1 dileks dileks 7.9M Apr 9 11:18
debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
$ file debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko:
ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV),
BuildID[sha1]=78d593f4fd2b8efe81caeb8f1ea729107a33e244, with
debug_info, not stripped
And think it's shrunk down and included debian/linux-image directory:
$ LC_ALL=C ll debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
-rw-r--r-- 1 dileks dileks 694K Apr 9 11:18
debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
$ file debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko:
ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV),
BuildID[sha1]=78d593f4fd2b8efe81caeb8f1ea729107a33e244, not stripped
I speculate both iwlwifi.ko below debian directory should be ZSTD-compressed.
Fact is the one with debug-info is done correctly.
Might be builddeb script needs a special treatment.
- Sedat -
On Fri, Apr 9, 2021 at 1:31 PM Sedat Dilek <[email protected]> wrote:
>
> On Fri, Apr 9, 2021 at 1:10 PM Piotr Gorski <[email protected]> wrote:
> >
> > I originally posted the patch in a different form [1] even before Masahiro's changes.
> > I've been testing this solution since December last year and posted it in March this year,
> > after I made sure everything was working fine. This patch was tested by Oleksandr and he also didn't report any objections. [2]
> >
> > Masahiro notified me about the planned changes [3] and asked me to resend this patch, adjusted to those changes, which I did.
> >
> > My current logs:
> >
> > lucjan@archlinux ~ $ zgrep CONFIG_DEBUG_INFO /proc/config.gz
> > CONFIG_DEBUG_INFO=y
> > # CONFIG_DEBUG_INFO_REDUCED is not set
> > # CONFIG_DEBUG_INFO_COMPRESSED is not set
> > # CONFIG_DEBUG_INFO_SPLIT is not set
> > CONFIG_DEBUG_INFO_DWARF4=y
> > CONFIG_DEBUG_INFO_BTF=y
> > CONFIG_DEBUG_INFO_BTF_MODULES=y
> > lucjan@archlinux ~ $ zgrep CONFIG_MODULE_COMPRESS_ZSTD /proc/config.gz
> > CONFIG_MODULE_COMPRESS_ZSTD=y
> > CONFIG_MODULE_COMPRESS_ZSTD_LEVEL=19
> >
> > Pay no attention to CONFIG_MODULE_COMPRESS_ZSTD_LEVEL as this is not in the upstream, it's an additional patch I use.
> >
> > The only difference - I don't use clang. Maybe those who use will comment on this.
> > I relied on the opinions of Oleksander and a dozen other users who reported no errors in using zstd module compression.
> >
> > [1] https://marc.info/?l=linux-kbuild&m=161710402402989&w=2
> >
> > [2] https://marc.info/?l=linux-kbuild&m=161710503403517&w=2
> >
> > [3] https://marc.info/?l=linux-kbuild&m=161780602730829&w=2
>
> I am a big fan of ZSTD and have it as default in all available Linux
> Kconfigs and Debian's initramfs-tools.
> So, I am highly interested in getting this fixed.
>
> Unfortunately, I have thrown away my yesterday's Clang-LTO build and
> switched to Clang-CFI with builddeb - should do handle the same way.
>
> I see three iwlwifi.ko (as an example):
>
> $ LC_ALL=C ll drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
> -rw-r--r-- 1 dileks dileks 8.2M Apr 9 11:07
> drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
>
> $ file drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
> drivers/net/wireless/intel/iwlwifi/iwlwifi.ko: ELF 64-bit LSB
> relocatable, x86-64, version 1 (SYSV),
> BuildID[sha1]=78d593f4fd2b8efe81caeb8f1ea729107a33e244, with
> debug_info, not stripped
>
> That iwlwifi.ko with debug-info is optimized when moving to
> debian/linux-image-dbg directory:
>
> $ LC_ALL=C ll debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
> -rw-r--r-- 1 dileks dileks 7.9M Apr 9 11:18
> debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
>
> $ file debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
> debian/linux-image-dbg/usr/lib/debug/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko:
> ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV),
> BuildID[sha1]=78d593f4fd2b8efe81caeb8f1ea729107a33e244, with
> debug_info, not stripped
>
> And think it's shrunk down and included debian/linux-image directory:
>
> $ LC_ALL=C ll debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
> -rw-r--r-- 1 dileks dileks 694K Apr 9 11:18
> debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
>
> $ file debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko
> debian/linux-image/lib/modules/5.12.0-rc6-5-amd64-clang12-cfi/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko:
> ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV),
> BuildID[sha1]=78d593f4fd2b8efe81caeb8f1ea729107a33e244, not stripped
>
> I speculate both iwlwifi.ko below debian directory should be ZSTD-compressed.
> Fact is the one with debug-info is done correctly.
> Might be builddeb script needs a special treatment.
>
OK, I see (sorry Gmail truncates my paste of snippet).
We need to add in this block a CONFIG_MODULE_COMPRESS_XXX handling:
[ scripts/package/builddeb ]
#159: if is_enabled CONFIG_MODULES; then
...
#184: fi
Maybe other scripts in scripts/package/ directory, too.
What do you say Masahiro?
I have to admit I never used any compression for kernel-modules before
and after recent changes in <kbuild.git#kbuild>.
- Sedat -
[1] https://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git/tree/scripts/package/builddeb?h=kbuild#n159