Andreas Mohr <[email protected]> writes:
> Hi,
>
> I just had a not so nice experience
> when finally upgrading to a new 4.1-rc5
> with CONFIG_MODULE_COMPRESS newly enabled -
> userspace binary parts (kmod 18 or 20 in my case)
> did not have compression enabled
> (at least on Debian 8pre, vs. encountering it enabled on FC21)
> since it does not seem to be
> the default build configuration of kmod (yet?).
Sure. Let's get the maintainers to insert the actual version required
in the help text though.
Thanks,
Rusty.
> Doing a manual
> for gz in $(find . -name "*.gz"); do echo gunzip $gz; done
> depmod -a (somewhere temporarily in bootup scripts)
> manages to fix the grave problem
> introduced by erroneously having enabled CONFIG_MODULE_COMPRESS.
>
> Thus it seems that the Kconfig text was much more optimistic
> than a hapless user would want to encounter,
> thus I decided to have it updated.
>
>
> BTW: kmod and/or module-init-tools packages
> might want to provide a generic way to openly advertise
> certain build-configured "capabilities" in the binary -
> neither
> kmod --help
> nor
> kmod -V
> indicated whether or not they provided capabilities
> such as e.g. providing or not providing support of compression types
> and which ones.
> That would have been a very helpful way
> to reliably determine
> that support in fact is missing in the binary,
> rather than having to resort to clumsy hacks
> such as ldd or even strings.
>
>
> While this is a minor but useful addition
> rather than a severe fix,
> having a CC to stable@ added subsequently might be useful.
>
> Thanks!
>
> Signed-off-by: Andreas Mohr <[email protected]>
>
> ---
> init/Kconfig | 18 +++++++++++++-----
> 1 file changed, 13 insertions(+), 5 deletions(-)
>
> diff --git a/init/Kconfig b/init/Kconfig
> index dc24dec..8e451f30 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -1951,11 +1951,19 @@ config MODULE_COMPRESS
> This option compresses the kernel modules when 'make
> modules_install' is run.
>
> - The modules will be compressed either using gzip or xz depend on the
> - choice made in "Compression algorithm".
> -
> - module-init-tools has support for gzip format while kmod handle gzip
> - and xz compressed modules.
> + The module files will be compressed either using gzip or xz
> + depending on the choice made in "Compression algorithm".
> +
> + Obviously one will then need appropriate userspace parts
> + which are actually able to deal with compressed files, too:
> + module-init-tools has support for gzip format
> + while kmod handles gzip and xz compressed modules.
> + However, we observed that in several environments
> + module loader binaries do not have that enabled (yet?)
> + and thus bootup will fail fatally -
> + manually doing ldd on these binaries
> + to detect compression libraries
> + is a tell-tale sign of having support enabled.
>
> When a kernel module is installed from outside of the main kernel
> source and uses the Kbuild system for installing modules then that
> --
> 2.1.4
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
On 2015-06-01 08:26, Rusty Russell wrote:
> Andreas Mohr <[email protected]> writes:
>> Hi,
>>
>> I just had a not so nice experience
>> when finally upgrading to a new 4.1-rc5
>> with CONFIG_MODULE_COMPRESS newly enabled -
>> userspace binary parts (kmod 18 or 20 in my case)
>> did not have compression enabled
>> (at least on Debian 8pre, vs. encountering it enabled on FC21)
>> since it does not seem to be
>> the default build configuration of kmod (yet?).
>
> Sure. Let's get the maintainers to insert the actual version required
> in the help text though.
I'm not a maintainer of kmod, but the text should probably end with "If
unsure, say N."
Michal
On Mon, Jun 1, 2015 at 3:26 AM, Rusty Russell <[email protected]> wrote:
> Andreas Mohr <[email protected]> writes:
>> Hi,
>>
>> I just had a not so nice experience
>> when finally upgrading to a new 4.1-rc5
>> with CONFIG_MODULE_COMPRESS newly enabled -
>> userspace binary parts (kmod 18 or 20 in my case)
>> did not have compression enabled
>> (at least on Debian 8pre, vs. encountering it enabled on FC21)
>> since it does not seem to be
>> the default build configuration of kmod (yet?).
>
> Sure. Let's get the maintainers to insert the actual version required
> in the help text though.
kmod supports gz since the first version and xz since version 3. So both
of them can be safely fall into "it's supported since the beginning of
kmod IMO".
Regarding the "default configuration", there's no such thing. Each distribution
uses a different one.
--
Lucas De Marchi
On Wed, Jun 3, 2015 at 7:30 PM, Lucas De Marchi
<[email protected]> wrote:
> On Mon, Jun 1, 2015 at 3:26 AM, Rusty Russell <[email protected]> wrote:
>> Andreas Mohr <[email protected]> writes:
>>> Hi,
>>>
>>> I just had a not so nice experience
>>> when finally upgrading to a new 4.1-rc5
>>> with CONFIG_MODULE_COMPRESS newly enabled -
>>> userspace binary parts (kmod 18 or 20 in my case)
>>> did not have compression enabled
>>> (at least on Debian 8pre, vs. encountering it enabled on FC21)
>>> since it does not seem to be
>>> the default build configuration of kmod (yet?).
>>
>> Sure. Let's get the maintainers to insert the actual version required
>> in the help text though.
>
> kmod supports gz since the first version and xz since version 3. So both
> of them can be safely fall into "it's supported since the beginning of
> kmod IMO".
>
> Regarding the "default configuration", there's no such thing. Each distribution
> uses a different one.
You could add something similar to this:
$ /usr/lib/systemd/systemd --version
systemd 220
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP
+LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS
+KMOD +IDN
Kay
On Wed, Jun 3, 2015 at 2:36 PM, Kay Sievers <[email protected]> wrote:
> On Wed, Jun 3, 2015 at 7:30 PM, Lucas De Marchi
> <[email protected]> wrote:
>> On Mon, Jun 1, 2015 at 3:26 AM, Rusty Russell <[email protected]> wrote:
>>> Andreas Mohr <[email protected]> writes:
>>>> Hi,
>>>>
>>>> I just had a not so nice experience
>>>> when finally upgrading to a new 4.1-rc5
>>>> with CONFIG_MODULE_COMPRESS newly enabled -
>>>> userspace binary parts (kmod 18 or 20 in my case)
>>>> did not have compression enabled
>>>> (at least on Debian 8pre, vs. encountering it enabled on FC21)
>>>> since it does not seem to be
>>>> the default build configuration of kmod (yet?).
>>>
>>> Sure. Let's get the maintainers to insert the actual version required
>>> in the help text though.
>>
>> kmod supports gz since the first version and xz since version 3. So both
>> of them can be safely fall into "it's supported since the beginning of
>> kmod IMO".
>>
>> Regarding the "default configuration", there's no such thing. Each distribution
>> uses a different one.
>
> You could add something similar to this:
>
> $ /usr/lib/systemd/systemd --version
> systemd 220
> +PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP
> +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS
> +KMOD +IDN
Indeed. I'm planning to do a release very soon and add something like
this to it. If anyone is
interested in contributing such patch it's very welcome, too :-)
--
Lucas De Marchi
Lucas De Marchi <[email protected]> writes:
> On Mon, Jun 1, 2015 at 3:26 AM, Rusty Russell <[email protected]> wrote:
>> Andreas Mohr <[email protected]> writes:
>>> Hi,
>>>
>>> I just had a not so nice experience
>>> when finally upgrading to a new 4.1-rc5
>>> with CONFIG_MODULE_COMPRESS newly enabled -
>>> userspace binary parts (kmod 18 or 20 in my case)
>>> did not have compression enabled
>>> (at least on Debian 8pre, vs. encountering it enabled on FC21)
>>> since it does not seem to be
>>> the default build configuration of kmod (yet?).
>>
>> Sure. Let's get the maintainers to insert the actual version required
>> in the help text though.
>
> kmod supports gz since the first version and xz since version 3. So both
> of them can be safely fall into "it's supported since the beginning of
> kmod IMO".
Thanks, that's what I needed.
But disappointing that Debian doesn't configure with it, and there's no
easy way to check it. Looks like Ubuntu vivid is the same.
Might be time to change the default in kmod?
Anyway, here's my patch, comments welcome.
Subject: modules: clarify CONFIG_MODULE_COMPRESS help, suggest 'N'.
Andreas turned this option on, only to find out Debian (and Ubuntu!)
don't enable support in their kmod builds.
Shorten the text, and suggest N at the bottom (at least for now).
Reported-by: Andreas Mohr <[email protected]>
Signed-off-by: Rusty Russell <[email protected]>
diff --git a/init/Kconfig b/init/Kconfig
index 968a001790af..5422c44be5f0 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1948,26 +1948,22 @@ config MODULE_COMPRESS
bool "Compress modules on installation"
depends on MODULES
help
- This option compresses the kernel modules when 'make
- modules_install' is run.
- The modules will be compressed either using gzip or xz depend on the
- choice made in "Compression algorithm".
+ Compresses kernel modules when 'make modules_install' is run; gzip or
+ xz depending on "Compression algorithm" below.
- module-init-tools has support for gzip format while kmod handle gzip
- and xz compressed modules.
+ module-init-tools supports gzip, and kmod can be configured to handle
+ gzip and xz (but doesn't by default, at least as of version 18!).
- When a kernel module is installed from outside of the main kernel
- source and uses the Kbuild system for installing modules then that
- kernel module will also be compressed when it is installed.
+ Out-of-tree kernel modules installed using Kbuild will also be
+ compressed upon installation.
- This option provides little benefit when the modules are to be used inside
- an initrd or initramfs, it generally is more efficient to compress the whole
- initrd or initramfs instead.
+ Note: for modules inside an initrd or initramfs, it's more efficient
+ to compress the whole initrd or initramfs instead.
- This is fully compatible with signed modules while the signed module is
- compressed. module-init-tools or kmod handles decompression and provide to
- other layer the uncompressed but signed payload.
+ Note: This is fully compatible with signed modules.
+
+ If in doubt, say N.
choice
prompt "Compression algorithm"
On Wed, Jun 3, 2015 at 10:30 PM, Rusty Russell <[email protected]> wrote:
> Lucas De Marchi <[email protected]> writes:
>> On Mon, Jun 1, 2015 at 3:26 AM, Rusty Russell <[email protected]> wrote:
>>> Andreas Mohr <[email protected]> writes:
>>>> Hi,
>>>>
>>>> I just had a not so nice experience
>>>> when finally upgrading to a new 4.1-rc5
>>>> with CONFIG_MODULE_COMPRESS newly enabled -
>>>> userspace binary parts (kmod 18 or 20 in my case)
>>>> did not have compression enabled
>>>> (at least on Debian 8pre, vs. encountering it enabled on FC21)
>>>> since it does not seem to be
>>>> the default build configuration of kmod (yet?).
>>>
>>> Sure. Let's get the maintainers to insert the actual version required
>>> in the help text though.
>>
>> kmod supports gz since the first version and xz since version 3. So both
>> of them can be safely fall into "it's supported since the beginning of
>> kmod IMO".
>
> Thanks, that's what I needed.
>
> But disappointing that Debian doesn't configure with it, and there's no
> easy way to check it. Looks like Ubuntu vivid is the same.
>
> Might be time to change the default in kmod?
>
> Anyway, here's my patch, comments welcome.
>
> Subject: modules: clarify CONFIG_MODULE_COMPRESS help, suggest 'N'.
>
> Andreas turned this option on, only to find out Debian (and Ubuntu!)
> don't enable support in their kmod builds.
>
> Shorten the text, and suggest N at the bottom (at least for now).
>
> Reported-by: Andreas Mohr <[email protected]>
> Signed-off-by: Rusty Russell <[email protected]>
>
> diff --git a/init/Kconfig b/init/Kconfig
> index 968a001790af..5422c44be5f0 100644
> --- a/init/Kconfig
> +++ b/init/Kconfig
> @@ -1948,26 +1948,22 @@ config MODULE_COMPRESS
> bool "Compress modules on installation"
> depends on MODULES
> help
> - This option compresses the kernel modules when 'make
> - modules_install' is run.
>
> - The modules will be compressed either using gzip or xz depend on the
> - choice made in "Compression algorithm".
> + Compresses kernel modules when 'make modules_install' is run; gzip or
> + xz depending on "Compression algorithm" below.
>
> - module-init-tools has support for gzip format while kmod handle gzip
> - and xz compressed modules.
> + module-init-tools supports gzip, and kmod can be configured to handle
> + gzip and xz (but doesn't by default, at least as of version 18!).
This is pretty much misleading. There's no such a default option. It's
like saying kernel doesn't support feature X, Y or Z by default
because make defconfig doesn't turn them on.
The option is there since the beginning for the distros to adapt to their needs.
--
Lucas De Marchi
On Jun 04, Rusty Russell <[email protected]> wrote:
> But disappointing that Debian doesn't configure with it, and there's no
> easy way to check it. Looks like Ubuntu vivid is the same.
>
> Might be time to change the default in kmod?
Changing the default will not make me enable compression support,
but providing interesting use cases maybe will.
Also, I see no point in enabling support for both gz and xz in kmod.
--
ciao,
Marco
Hi Rusty,
On Thu, 04 Jun 2015 11:00:41 +0930 Rusty Russell <[email protected]> wrote:
>
> Lucas De Marchi <[email protected]> writes:
> > On Mon, Jun 1, 2015 at 3:26 AM, Rusty Russell <[email protected]> wrote:
> >> Andreas Mohr <[email protected]> writes:
> >>>
> >>> I just had a not so nice experience
> >>> when finally upgrading to a new 4.1-rc5
> >>> with CONFIG_MODULE_COMPRESS newly enabled -
> >>> userspace binary parts (kmod 18 or 20 in my case)
> >>> did not have compression enabled
> >>> (at least on Debian 8pre, vs. encountering it enabled on FC21)
> >>> since it does not seem to be
> >>> the default build configuration of kmod (yet?).
> >>
> >> Sure. Let's get the maintainers to insert the actual version required
> >> in the help text though.
> >
> > kmod supports gz since the first version and xz since version 3. So both
> > of them can be safely fall into "it's supported since the beginning of
> > kmod IMO".
>
> Thanks, that's what I needed.
>
> But disappointing that Debian doesn't configure with it, and there's no
> easy way to check it. Looks like Ubuntu vivid is the same.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=772628
--
Cheers,
Stephen Rothwell [email protected]
On Wed, Jun 03, 2015 at 11:31:20PM -0300, Lucas De Marchi wrote:
> On Wed, Jun 3, 2015 at 10:30 PM, Rusty Russell <[email protected]> wrote:
> > diff --git a/init/Kconfig b/init/Kconfig
> > index 968a001790af..5422c44be5f0 100644
> > --- a/init/Kconfig
> > +++ b/init/Kconfig
> > @@ -1948,26 +1948,22 @@ config MODULE_COMPRESS
> > bool "Compress modules on installation"
> > depends on MODULES
> > help
> > - This option compresses the kernel modules when 'make
> > - modules_install' is run.
> >
> > - The modules will be compressed either using gzip or xz depend on the
> > - choice made in "Compression algorithm".
> > + Compresses kernel modules when 'make modules_install' is run; gzip or
> > + xz depending on "Compression algorithm" below.
> >
> > - module-init-tools has support for gzip format while kmod handle gzip
> > - and xz compressed modules.
> > + module-init-tools supports gzip, and kmod can be configured to handle
> > + gzip and xz (but doesn't by default, at least as of version 18!).
>
> This is pretty much misleading. There's no such a default option. It's
> like saying kernel doesn't support feature X, Y or Z by default
> because make defconfig doesn't turn them on.
>
> The option is there since the beginning for the distros to adapt to their needs.
Hmm? Perhaps I'm misunderstanding something in this discussion,
but Debian kmod-20 (kmod_20.orig.tar.xz - BTW side note:
noted the .xz file rather than .gz? ;))
configure.ac contains:
AC_ARG_WITH([xz],
AS_HELP_STRING([--with-xz], [handle Xz-compressed modules
@<:@default=disabled@:>@]),
[], [with_xz=no])
AS_IF([test "x$with_xz" != "xno"], [
PKG_CHECK_MODULES([liblzma], [liblzma >= 4.99])
AC_DEFINE([ENABLE_XZ], [1], [Enable Xz for modules.])
], [
AC_MSG_NOTICE([Xz support not requested])
])
AC_ARG_WITH([zlib],
AS_HELP_STRING([--with-zlib], [handle gzipped modules
@<:@default=disabled@:>@]),
[], [with_zlib=no])
So it clearly seems there *is* a specific
(hard-coded, as opposed to system introspection) default
(i.e., the usual way of getting defaults, namely to simply not specify
--with-something configure options),
and that is (drumroll...) "disabled".
Andreas
Hi,
On Thu, Jun 04, 2015 at 01:19:46PM +1000, Stephen Rothwell wrote:
> Hi Rusty,
> > But disappointing that Debian doesn't configure with it, and there's no
> > easy way to check it. Looks like Ubuntu vivid is the same.
>
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=772628
Crap, that URL now is an implicit strong suggestion
that I didn't do my homework ;)
(it already described large parts
prior to me regurgitating this issue
due to being unaware of this history here).
Thanks to R.Russell for a very nice shortening
and extension of the help text!
Since the issue of .gz vs. .xz redundancy came up
(with people "threatening" to support only one alternative),
I want to mention that
when having to choose one
I'd tend to activating the .xz library dependency:
- while it has higher compression demands,
hardware is getting beefier all the time,
thus it should not matter
(especially vs. the dominantly many decompression runs)
- it's simply the "more modern" and future-proof option,
thus it should be favoured slightly
since the system as a whole
would want to make reasonably quick
development/evolution "forward progress"
rather than sticking to less favourable mechanisms
- .xz has been available for some time already
i.e. the time window of "distro support maturity" is a given
[a counter-point might be that module-init-tools supports .gz only,
but then modern binary setups which chose .xz
would already have been shipped with kmod only]
OK, having said that,
I'm unsure what to think of the Debian package's decision
to not support compression so far
(and that even in times
where kmod does not provide a runtime option yet
to query the actual set of support flags).
>From a library dependency POV it may be attractive to skip compression,
and since Debian usually has a fixed setup
where non-compressed files are a given,
this seems like a valid choice.
It's just that for e.g. these situations:
- people with many custom kernels installed
- space-constrained systems
this is quite a nuisance that is a wee bit too unexpected (/show-stopper) -
took me roughly > 30 minutes to get it researched / resolved.
Thanks,
Andreas
Lucas De Marchi <[email protected]> writes:
>> + gzip and xz (but doesn't by default, at least as of version 18!).
>
> This is pretty much misleading. There's no such a default option. It's
> like saying kernel doesn't support feature X, Y or Z by default
> because make defconfig doesn't turn them on.
Nobody except Linus expects a defconfig to boot.
I expect ./configure to give me a working tool; these *are* defaults,
and setting them *is* your responsibility as author. When it became an
in-kernel option rather than some weird distro thing, the old default
was wrong.
... Though I hadn't realized that Marco had deliberately decided to
leave support off. Surprising, since liblzma and libz seem required on
Debian anyway.
So I'm sorry Andreas: if your maintainer doesn't care about your case,
there's not much I can do :(
Subject: modules: clarify CONFIG_MODULE_COMPRESS help, suggest 'N'.
Andreas turned this option on, only to find out Debian (and Ubuntu!)
don't enable support in their kmod builds.
Shorten the text, and suggest N at the bottom (at least for now).
Reported-by: Andreas Mohr <[email protected]>
Signed-off-by: Rusty Russell <[email protected]>
diff --git a/init/Kconfig b/init/Kconfig
index 968a001790af..bcb7e1e13f7d 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1948,26 +1948,21 @@ config MODULE_COMPRESS
bool "Compress modules on installation"
depends on MODULES
help
- This option compresses the kernel modules when 'make
- modules_install' is run.
- The modules will be compressed either using gzip or xz depend on the
- choice made in "Compression algorithm".
+ Compresses kernel modules when 'make modules_install' is run; gzip or
+ xz depending on "Compression algorithm" below.
- module-init-tools has support for gzip format while kmod handle gzip
- and xz compressed modules.
+ module-init-tools MAY support gzip, and kmod MAY support gzip and xz.
- When a kernel module is installed from outside of the main kernel
- source and uses the Kbuild system for installing modules then that
- kernel module will also be compressed when it is installed.
+ Out-of-tree kernel modules installed using Kbuild will also be
+ compressed upon installation.
- This option provides little benefit when the modules are to be used inside
- an initrd or initramfs, it generally is more efficient to compress the whole
- initrd or initramfs instead.
+ Note: for modules inside an initrd or initramfs, it's more efficient
+ to compress the whole initrd or initramfs instead.
- This is fully compatible with signed modules while the signed module is
- compressed. module-init-tools or kmod handles decompression and provide to
- other layer the uncompressed but signed payload.
+ Note: This is fully compatible with signed modules.
+
+ If in doubt, say N.
choice
prompt "Compression algorithm"
On Thu, Jun 4, 2015 at 5:22 PM, Rusty Russell <[email protected]> wrote:
> Lucas De Marchi <[email protected]> writes:
>>> + gzip and xz (but doesn't by default, at least as of version 18!).
>>
>> This is pretty much misleading. There's no such a default option. It's
>> like saying kernel doesn't support feature X, Y or Z by default
>> because make defconfig doesn't turn them on.
>
> Nobody except Linus expects a defconfig to boot.
>
> I expect ./configure to give me a working tool; these *are* defaults,
> and setting them *is* your responsibility as author. When it became an
> in-kernel option rather than some weird distro thing, the old default
> was wrong.
>
> ... Though I hadn't realized that Marco had deliberately decided to
> leave support off. Surprising, since liblzma and libz seem required on
> Debian anyway.
As you can see, no default to something else will make distros adopt that.
I long ago decided not to provide a default for compression because it's
clearly not agreed upon. Instead it's everything disabled and the
distro chooses what it wants to support.
So if you think the configuration for your distro should be something else,
convince the maintainer to change it showing why it's important.
FYI I'm adding in kmod the ability to query the flags used during build. So
now we have:
$ ./tools/kmod --version
kmod version 20
+XZ +ZLIB -EXPERIMENTAL
This will be present in kmod 21
--
Lucas De Marchi