what about some "make modules_install_compressed" instead ?
as i have run out of diskspace quite often when installing test kernels, i
think we really need a feature like this.
i`d also favour the makefile approach.
why another kconfig option?
jan`s patch looks clean and simple, but i think it`s a little bit
intrusive...
regards
roland
ps:
i`d use gzip without "-9" as this gives very little space savings. it mostly
burns cpu and slows things down too much.
( see
http://www.linuxjournal.com/files/linuxjournal.com/linuxjournal/articles/080/8051/8051f1.png )
On Friday 2008-12-26 20:48, Sam Ravnborg wrote:
> > > >
> > > > This patch allows kernel modules to be compressed when 'make
> > > > modules_install' is run after being copied to
> > > > the /lib/module/<version>/<...> directory which is useful if you
> > > > have
> > > > module-init-tools installed with --enable-zlib. This patch adds an
> > > > option (MODULE_COMPRESS) to the kernel configuration file
> > > > (specifically
> > > > init/Kconfig) so that the kernel modules will compressed if
> > > > MODULE_COMPRESS is set.
> >
> > I recently started compressing my kernel modules and that saved me
> > at least 70 MB of disk space on mostlyallmodconfig.
> > (And no, the argument of disks being cheap is not so true with
> > CF or SSD.)
> > Distro is lazy and wants to wait for upstream to have it,
> > so is there any chance of getting this proposal in?
>
> Steve said he wanted to try to make the solution more
> scalable so I am awaiting a new patch.
Hm, all I needed was this patch. It might fire up some people,
but it's got all the scalability I could think of..
commit b4a3e1c610c99d4e8b543b97fd722076c6f7c5dd
Author: Jan Engelhardt <[email protected]>
Date: Wed Dec 10 20:39:21 2008 +0100
build: install modules compressed
---
scripts/Makefile.modinst | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
index efa5d94..c3421a1 100644
--- a/scripts/Makefile.modinst
+++ b/scripts/Makefile.modinst
@@ -17,7 +17,7 @@ __modinst: $(modules)
@:
quiet_cmd_modules_install = INSTALL $@
- cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) \
$(2)/$(notdir $@) + cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; \
$(mod_strip_cmd) $(2)/$(notdir $@); gzip -9f $(2)/$(notdir $@)
# Modules built outside the kernel source tree go into extra by default
INSTALL_MOD_DIR ?= extra
--
On Fri, 26 Dec 2008, Roland wrote:
> what about some "make modules_install_compressed" instead ?
>
> as i have run out of diskspace quite often when installing test kernels, i
> think we really need a feature like this.
>
> i`d also favour the makefile approach.
> why another kconfig option?
>
> jan`s patch looks clean and simple, but i think it`s a little bit
> intrusive...
by default the main kernel is compressed, why should the modules not be
compressed by default as well?
> regards
> roland
>
> ps:
> i`d use gzip without "-9" as this gives very little space savings. it mostly
> burns cpu and slows things down too much.
> ( see
> http://www.linuxjournal.com/files/linuxjournal.com/linuxjournal/articles/080/8051/8051f1.png
> )
this varies a lot on the particulars of the data being compressed. I don't
know what difference it makes on the kernel modules, but I have seen it be
worth it in some other cases.
however, I do need to ask what the sizes of the resulting files end up
being (on my system the last kernel that used modules was 2.4.31, so I
can't just look this up myself). in most cases disks are formatted with 4K
blocks, so unless it shrinks across such a boundry it's not actually going
to help much.
one other thing to consider would be some ability to load the module out
of a tar or cpio bundle, that way you end up saving all the partial blocks
from each module as well.
David Lang
>
> On Friday 2008-12-26 20:48, Sam Ravnborg wrote:
>> > > >
>> > > > This patch allows kernel modules to be compressed when 'make
>> > > > modules_install' is run after being copied to
>> > > > the /lib/module/<version>/<...> directory which is useful if you > >
>> > have
>> > > > module-init-tools installed with --enable-zlib. This patch adds an
>> > > > option (MODULE_COMPRESS) to the kernel configuration file > > >
>> (specifically
>> > > > init/Kconfig) so that the kernel modules will compressed if
>> > > > MODULE_COMPRESS is set.
>> >
>> > I recently started compressing my kernel modules and that saved me
>> > at least 70 MB of disk space on mostlyallmodconfig.
>> > (And no, the argument of disks being cheap is not so true with
>> > CF or SSD.)
>> > Distro is lazy and wants to wait for upstream to have it,
>> > so is there any chance of getting this proposal in?
>>
>> Steve said he wanted to try to make the solution more
>> scalable so I am awaiting a new patch.
>
> Hm, all I needed was this patch. It might fire up some people,
> but it's got all the scalability I could think of..
>
>
> commit b4a3e1c610c99d4e8b543b97fd722076c6f7c5dd
> Author: Jan Engelhardt <[email protected]>
> Date: Wed Dec 10 20:39:21 2008 +0100
>
> build: install modules compressed
> ---
> scripts/Makefile.modinst | 2 +-
> 1 files changed, 1 insertions(+), 1 deletions(-)
>
> diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
> index efa5d94..c3421a1 100644
> --- a/scripts/Makefile.modinst
> +++ b/scripts/Makefile.modinst
> @@ -17,7 +17,7 @@ __modinst: $(modules)
> @:
>
> quiet_cmd_modules_install = INSTALL $@
> - cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) \
> $(2)/$(notdir $@) + cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; \
> $(mod_strip_cmd) $(2)/$(notdir $@); gzip -9f $(2)/$(notdir $@)
> # Modules built outside the kernel source tree go into extra by default
> INSTALL_MOD_DIR ?= extra
> --
>
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
On Friday 2008-12-26 21:23, Roland wrote:
> what about some "make modules_install_compressed" instead ?
>
> as i have run out of diskspace quite often when installing test kernels, i
> think we really need a feature like this.
>
> i`d also favour the makefile approach.
> why another kconfig option?
>
> jan`s patch looks clean and simple, but i think it`s a little bit intrusive...
Why so? module-ini
>
> regards
> roland
>
> ps:
> i`d use gzip without "-9" as this gives very little space savings. it mostly
> burns cpu and slows things down too much.
> ( see
> http://www.linuxjournal.com/files/linuxjournal.com/linuxjournal/articles/080/8051/8051f1.png
Here is one that is tailored for kernel modules:
http://tinyurl.com/8f2xta
Looks pretty much like the one from LJ but of course, .ko files
compress better than LJ :)
Oh well.
On Friday 2008-12-26 23:02, [email protected] wrote:
>> ( see
>> http://www.linuxjournal.com/files/linuxjournal.com/linuxjournal/articles/080/8051/8051f1.png
>> )
>
> this varies a lot on the particulars of the data being compressed.
see my other reply.
> however, I do need to ask what the sizes of the resulting files end up being
> (on my system the last kernel that used modules was 2.4.31, so I can't just
> look this up myself). in most cases disks are formatted with 4K blocks, so
> unless it shrinks across such a boundry it's not actually going to help much.
Hey, don't forget exotic filesystems that pack that together anyway.
And if I fired up du right, then...
$ find . -iname "*.gz" | xargs du -cs -B 4096 | grep total
7086 total
$ echo $[7086*4096]
29024256
$ echo $[29024256-24782942]
4241314
So that's like 4M going off for the block stuff. Given that
the compression actually saved about 50 MB, I think we
can live with the 4M - especially since they have been
there already one way or another.
> one other thing to consider would be some ability to load the
> module out of a tar or cpio bundle, that way you end up saving all
> the partial blocks from each module as well.
tarfs, it's somewhere around the corner on the fuse pages probably.
On Fri, 26 Dec 2008, Jan Engelhardt wrote:
> On Friday 2008-12-26 23:02, [email protected] wrote:
>>> ( see
>>> http://www.linuxjournal.com/files/linuxjournal.com/linuxjournal/articles/080/8051/8051f1.png
>>> )
>>
>> this varies a lot on the particulars of the data being compressed.
>
> see my other reply.
IIRC the default is 5, from the graph you posted going to 6 or 7 would be
a fairly significant win (final size to final size) without eathing _that_
much more cpu.
instead of doing the compression when doing the modules_install do it when
they are compiled (just like we do for the kernel itself). if you just
spent the cpu to build the thing, the cpu to compress it isn't much.
>> however, I do need to ask what the sizes of the resulting files end up being
>> (on my system the last kernel that used modules was 2.4.31, so I can't just
>> look this up myself). in most cases disks are formatted with 4K blocks, so
>> unless it shrinks across such a boundry it's not actually going to help much.
>
> Hey, don't forget exotic filesystems that pack that together anyway.
> And if I fired up du right, then...
>
> $ find . -iname "*.gz" | xargs du -cs -B 4096 | grep total
> 7086 total
> $ echo $[7086*4096]
> 29024256
> $ echo $[29024256-24782942]
> 4241314
>
> So that's like 4M going off for the block stuff. Given that
> the compression actually saved about 50 MB, I think we
> can live with the 4M - especially since they have been
> there already one way or another.
if I'm reading the numbers correctly that's about 16% of the space taken
by the final result
>> one other thing to consider would be some ability to load the
>> module out of a tar or cpio bundle, that way you end up saving all
>> the partial blocks from each module as well.
>
> tarfs, it's somewhere around the corner on the fuse pages probably.
watch out for a deadlock here. the more external tools that you need to
get at the kernel modules, the more likely it is that one of those tools
will need something that you can't get to without the modules loaded.
David Lang
On Friday 2008-12-26 23:28, [email protected] wrote:
>>>> ( see
>>>> http://www.linuxjournal.com/files/linuxjournal.com/linuxjournal/articles/080/8051/8051f1.png
>>>> )
>>>
>>> this varies a lot on the particulars of the data being compressed.
>>
>> see my other reply.
>
> IIRC the default is 5
default is 6 (man gzip).
> instead of doing the compression when doing the modules_install do it when they
> are compiled (just like we do for the kernel itself). if you just spent the cpu
> to build the thing, the cpu to compress it isn't much.
I'd rather do the compression later on - or do you want
to spend gzipping all day when doing `make foo.ko`
as a developer? ;-)
>> $ find . -iname "*.gz" | xargs du -cs -B 4096 | grep total
>> 7086 total
>> $ echo $[7086*4096]
>> 29024256
>> $ echo $[29024256-24782942]
>> 4241314
>>
>> So that's like 4M going off for the block stuff. Given that
>> the compression actually saved about 50 MB, I think we
>> can live with the 4M - especially since they have been
>> there already one way or another.
>
> if I'm reading the numbers correctly that's about 16% of the space taken by the
> final result
Assuming that the x in (filesize % 4096 == x) is randomly distributed
over 0..4095 for gzip modules, I proclaim that it is similarly
randomly distributed for noncompressed modules. As such, the overhead
is always there and gzipping does not make it worse or better.
So you always have the overhead.
uncompressed:
$ du as above
113360896
$ byte counted:
109270606
differnece: 4.09 MB
You always pay that price for having single files.
So I dismiss it from the discussion ;-)
On Sat, 27 Dec 2008, Jan Engelhardt wrote:
> On Friday 2008-12-26 23:28, [email protected] wrote:
>>>>> ( see
>>>>> http://www.linuxjournal.com/files/linuxjournal.com/linuxjournal/articles/080/8051/8051f1.png
>>>>> )
>>>>
>>>> this varies a lot on the particulars of the data being compressed.
>>>
>>> see my other reply.
>>
>> IIRC the default is 5
>
> default is 6 (man gzip).
>
>> instead of doing the compression when doing the modules_install do it when they
>> are compiled (just like we do for the kernel itself). if you just spent the cpu
>> to build the thing, the cpu to compress it isn't much.
>
> I'd rather do the compression later on - or do you want
> to spend gzipping all day when doing `make foo.ko`
> as a developer? ;-)
>
>>> $ find . -iname "*.gz" | xargs du -cs -B 4096 | grep total
>>> 7086 total
>>> $ echo $[7086*4096]
>>> 29024256
>>> $ echo $[29024256-24782942]
>>> 4241314
>>>
>>> So that's like 4M going off for the block stuff. Given that
>>> the compression actually saved about 50 MB, I think we
>>> can live with the 4M - especially since they have been
>>> there already one way or another.
>>
>> if I'm reading the numbers correctly that's about 16% of the space taken by the
>> final result
>
> Assuming that the x in (filesize % 4096 == x) is randomly distributed
> over 0..4095 for gzip modules, I proclaim that it is similarly
> randomly distributed for noncompressed modules. As such, the overhead
> is always there and gzipping does not make it worse or better.
> So you always have the overhead.
>
> uncompressed:
> $ du as above
> 113360896
> $ byte counted:
> 109270606
>
> differnece: 4.09 MB
>
> You always pay that price for having single files.
> So I dismiss it from the discussion ;-)
the point I was making is that if we are changing things to add new ways
to get the module contents does it make sense to add another wrinkle that
would allow you to eliminate the single files
one pain point/upgrade of tools to take advantage of it instead of doing
the first step and then coming back and do the second one.
David Lang
On Fri, Dec 26, 2008 at 02:02:37PM -0800, [email protected] wrote:
> On Fri, 26 Dec 2008, Roland wrote:
>
> >what about some "make modules_install_compressed" instead ?
> >
> >as i have run out of diskspace quite often when installing test kernels, i
> >think we really need a feature like this.
> >
> >i`d also favour the makefile approach.
> >why another kconfig option?
> >
> >jan`s patch looks clean and simple, but i think it`s a little bit
> >intrusive...
>
> by default the main kernel is compressed, why should the modules not be
> compressed by default as well?
I've always compressed my modules for years, and I can say there is *one*
situation where I uncompress them, it's before making an initrd or initramfs,
because compressing an FS with many compressed files leads to a fairly bigger
image than leaving them uncompressed on the compressed FS.
Willy
