> On Mar 21, 2018, at 6:29 PM, Nick Terrell <[email protected]> wrote:
> This patch set adds support for a ZSTD-compressed kernel and ramdisk
> images in the kernel boot process. It only integrates the support with
> x86, though the first patch is generic to all architectures.
Hi all,
Is there anything blocking this from getting merged?
Please let me know if there is anything I can do to help move this patch forward.
Thanks,
Nick
Hi Nick,
On 09 Jul 2018, at 20:04, Nick Terrell <[email protected]> wrote:
>> On Mar 21, 2018, at 6:29 PM, Nick Terrell <[email protected]> wrote:
>> This patch set adds support for a ZSTD-compressed kernel and ramdisk
>> images in the kernel boot process. It only integrates the support with
>> x86, though the first patch is generic to all architectures.
>
> Hi all,
>
> Is there anything blocking this from getting merged?
> Please let me know if there is anything I can do to help move this patch forward.
works fine on my side, initrd test booted on i486, x86-64, ppc, ppc64 and
sparc64. For testing on mips64 I would first need to re-base the Octane^1
patch-set ;-)
Greetings,
Ren?
^1) https://www.youtube.com/watch?v=AU_RV8uoTIo
--
ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin
http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de
For now, that's arch/x86/boot/compressed/vmlinux.bin.zst but probably more
will come, thus let's be consistent with all other compressors.
Signed-off-by: Adam Borowski <[email protected]>
---
.gitignore | 1 +
1 file changed, 1 insertion(+)
diff --git a/.gitignore b/.gitignore
index 97ba6b79834c..0d09cf1c053c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -42,6 +42,7 @@
*.tab.[ch]
*.tar
*.xz
+*.zst
Module.symvers
modules.builtin
--
2.18.0
Hey,
is there any mainline future for this zstd support? Currently my most favourite compressor for this, and for what it?s worth zstd/initrd now even tested on #t2sde / hp/pa-risc, ? https://www.youtube.com/watch?v=MHplOJxnIHk
Ren?
On 10 Jul 2018, at 00:13, Ren? Rebe <[email protected]> wrote:
> Hi Nick,
>
> On 09 Jul 2018, at 20:04, Nick Terrell <[email protected]> wrote:
>
>>> On Mar 21, 2018, at 6:29 PM, Nick Terrell <[email protected]> wrote:
>>> This patch set adds support for a ZSTD-compressed kernel and ramdisk
>>> images in the kernel boot process. It only integrates the support with
>>> x86, though the first patch is generic to all architectures.
>>
>> Hi all,
>>
>> Is there anything blocking this from getting merged?
>> Please let me know if there is anything I can do to help move this patch forward.
>
> works fine on my side, initrd test booted on i486, x86-64, ppc, ppc64 and
> sparc64. For testing on mips64 I would first need to re-base the Octane^1
> patch-set ;-)
>
> Greetings,
> Ren?
>
> ^1) https://www.youtube.com/watch?v=AU_RV8uoTIo
>
> --
> ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin
> http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de
>
--
ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin
DE Legal: Amtsgericht Berlin (Charlottenburg) HRB 105123B, Tax-ID#: DE251602478
Managing Director: Ren? Rebe
http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de
On Fri, Aug 17, 2018 at 06:15:45PM +0200, René Rebe wrote:
> Hey,
>
> is there any mainline future for this zstd support? Currently my most favourite compressor for this, and for what it’s worth zstd/initrd now even tested on #t2sde / hp/pa-risc, … https://www.youtube.com/watch?v=MHplOJxnIHk
Can you remove some other compressor for the kernel image if you merge this?
The "favourite compressor" seems to roughly change every year, so if
we keep adding new ones things will get more and more convoluted.
Surely zstd is universally better than some existing compressor
the kernel already uses for itself for a particular sweet spot?
If it isn't it's likely not needed.
-Andi
Hi,
On 17 Aug 2018, at 18:54, Andi Kleen <[email protected]> wrote:
> On Fri, Aug 17, 2018 at 06:15:45PM +0200, Ren? Rebe wrote:
>> Hey,
>>
>> is there any mainline future for this zstd support? Currently my most favourite compressor for this, and for what it?s worth zstd/initrd now even tested on #t2sde / hp/pa-risc, ? https://www.youtube.com/watch?v=MHplOJxnIHk
>
> Can you remove some other compressor for the kernel image if you merge this?
>
> The "favourite compressor" seems to roughly change every year, so if
> we keep adding new ones things will get more and more convoluted.
Well, we (t2 formerly known as rocklinux) only changed our pkg archive compression from bzip2 to zstd once in 20 years ,-)
> Surely zstd is universally better than some existing compressor
> the kernel already uses for itself for a particular sweet spot?
> If it isn't it's likely not needed.
Well, IMHO this is not that much code, but if you want to remove something I guess bzip2 or LZMA may be candidates.
http://zstd.net
--
ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin
http://exactcode.com | http://exactscan.com | http://ocrkit.com | http://t2-project.org | http://rene.rebe.de
On Fri, Aug 17, 2018 at 09:54:03AM -0700, Andi Kleen wrote:
> On Fri, Aug 17, 2018 at 06:15:45PM +0200, René Rebe wrote:
> > Hey,
> >
> > is there any mainline future for this zstd support?
> > Currently my most favourite compressor for this, and for what it’s worth
> > zstd/initrd now even tested on #t2sde / hp/pa-risc
>
> Can you remove some other compressor for the kernel image if you merge this?
Funny you say this. I want to post this after the merge window is over:
https://github.com/kilobyte/linux/commits/nobz2-next-20180731
Sorry for a tree atop next, but there are conflicts that needed to be
adressed. The patchset is also undertested (works for me, but not even
compile-tested on archs I don't have a toolchain for at hand).
> The "favourite compressor" seems to roughly change every year, so if
> we keep adding new ones things will get more and more convoluted.
The above patchset drops just bzip2. It is the only one that's strictly
beaten in every way (ratio, time, memory usage), there are also no other
uses of bzip2 anywhere in the kernel so we'd get to drop its code
completely: 900 lines of Linus' happiness.
Other candidates are lzo and bare lzma (you want lz4, zstd or xz instead),
but those are used elsewhere thus there's hardly any gain. If you want them
gone, please say so -- I'll include their droppage.
> Surely zstd is universally better than some existing compressor
> the kernel already uses for itself for a particular sweet spot?
> If it isn't it's likely not needed.
zstd is better in the balanced niche: unless you want extreme speed (lz4) or
best compression (xz), zstd handily beats algorithms in the middle.
Meow!
--
⢀⣴⠾⠻⢶⣦⠀ What Would Jesus Do, MUD/MMORPG edition:
⣾⠁⢰⠒⠀⣿⡁ • multiplay with an admin char to benefit your mortal [Mt3:16-17]
⢿⡄⠘⠷⠚⠋⠀ • abuse item cloning bugs [Mt14:17-20, Mt15:34-37]
⠈⠳⣄⠀⠀⠀⠀ • use glitches to walk on water [Mt14:25-26]
On Fri, Aug 17, 2018 at 07:57:46PM +0200, Adam Borowski wrote:
> > The "favourite compressor" seems to roughly change every year, so if
> > we keep adding new ones things will get more and more convoluted.
>
> The above patchset drops just bzip2. It is the only one that's strictly
> beaten in every way (ratio, time, memory usage), there are also no other
Does time include build time? I've been reverting back to gzip recently
because I care very much about that.
> uses of bzip2 anywhere in the kernel so we'd get to drop its code
> completely: 900 lines of Linus' happiness.
Great!
>
> Other candidates are lzo and bare lzma (you want lz4, zstd or xz instead),
> but those are used elsewhere thus there's hardly any gain. If you want them
> gone, please say so -- I'll include their droppage.
Yes would be good to remove the kernel image support for those too
just to simplify the config process, even if it doesn't save much code.
-Andi
On Fri, Aug 17, 2018 at 12:22:44PM -0700, Andi Kleen wrote:
> On Fri, Aug 17, 2018 at 07:57:46PM +0200, Adam Borowski wrote:
> > > The "favourite compressor" seems to roughly change every year, so if
> > > we keep adding new ones things will get more and more convoluted.
> >
> > The above patchset drops just bzip2. It is the only one that's strictly
> > beaten in every way (ratio, time, memory usage), there are also no other
>
> Does time include build time? I've been reverting back to gzip recently
> because I care very much about that.
Too lazy to benchmark a kernel image (IIRC Nick Terrell posted that a while
ago), here's copypasta of a random 16824672 byte executable, in userspace,
with default level setting:
comp decomp size
xz 8.038s 0.356s 4320292
bz2 2.265s 0.730s 5234516
zst 0.274s 0.102s 5657626
gz 0.880s 0.152s 6515505
Z 0.499s 0.133s 8932459
lzo 0.100s 0.095s 9198874
As you can see, zstd's compression time is drastically better than gzip,
while ratio is better. The default level is very low (-3 on -1..-22 scale)
but you can crank it up for stronger compression.
The defaults fit your use case.
> > uses of bzip2 anywhere in the kernel so we'd get to drop its code
> > completely: 900 lines of Linus' happiness.
>
> Great!
>
> > Other candidates are lzo and bare lzma (you want lz4, zstd or xz instead),
> > but those are used elsewhere thus there's hardly any gain. If you want them
> > gone, please say so -- I'll include their droppage.
>
> Yes would be good to remove the kernel image support for those too
> just to simplify the config process, even if it doesn't save much code.
There's one caveat: fast choices are quite new:
* lz4 userspace tools are not even in current Debian stable (just unstable)
* uncompressed kernel got in only this merge window
* zstd has userspace tools in Debian stable but is not merged into the
kernel yet
(other dists are probably similar)
Thus, it might be a good idea to keep lzo for a while longer.
Bare lzma can probably go -- xz filters are nice for binaries of archs it
knows (disabled otherwise), and lack of header requires hacks to find out
the payload's size.
So it's up to you guys: do you want me to drop lzo and/or lzma?
We can also drop them just for vmlinuz but not initrd.
Meow!
--
⢀⣴⠾⠻⢶⣦⠀ What Would Jesus Do, MUD/MMORPG edition:
⣾⠁⢰⠒⠀⣿⡁ • multiplay with an admin char to benefit your mortal [Mt3:16-17]
⢿⡄⠘⠷⠚⠋⠀ • abuse item cloning bugs [Mt14:17-20, Mt15:34-37]
⠈⠳⣄⠀⠀⠀⠀ • use glitches to walk on water [Mt14:25-26]
On Aug 17, 2018, at 1:07 PM, Adam Borowski <[email protected]> wrote:
> On Fri, Aug 17, 2018 at 12:22:44PM -0700, Andi Kleen wrote:
>> On Fri, Aug 17, 2018 at 07:57:46PM +0200, Adam Borowski wrote:
>>>> The "favourite compressor" seems to roughly change every year, so if
>>>> we keep adding new ones things will get more and more convoluted.
>>>
>>> The above patchset drops just bzip2. It is the only one that's strictly
>>> beaten in every way (ratio, time, memory usage), there are also no other
>>
>> Does time include build time? I've been reverting back to gzip recently
>> because I care very much about that.
>
> Too lazy to benchmark a kernel image (IIRC Nick Terrell posted that a while
> ago), here's copypasta of a random 16824672 byte executable, in userspace,
> with default level setting:
>
> comp decomp size
> xz 8.038s 0.356s 4320292
> bz2 2.265s 0.730s 5234516
> zst 0.274s 0.102s 5657626
> gz 0.880s 0.152s 6515505
> Z 0.499s 0.133s 8932459
> lzo 0.100s 0.095s 9198874
>
> As you can see, zstd's compression time is drastically better than gzip,
> while ratio is better. The default level is very low (-3 on -1..-22 scale)
> but you can crank it up for stronger compression.
>
> The defaults fit your use case.
>
>>> uses of bzip2 anywhere in the kernel so we'd get to drop its code
>>> completely: 900 lines of Linus' happiness.
>>
>> Great!
>>
>>> Other candidates are lzo and bare lzma (you want lz4, zstd or xz instead),
>>> but those are used elsewhere thus there's hardly any gain. If you want them
>>> gone, please say so -- I'll include their droppage.
>>
>> Yes would be good to remove the kernel image support for those too
>> just to simplify the config process, even if it doesn't save much code.
>
> There's one caveat: fast choices are quite new:
> * lz4 userspace tools are not even in current Debian stable (just unstable)
> * uncompressed kernel got in only this merge window
> * zstd has userspace tools in Debian stable but is not merged into the
> kernel yet
> (other dists are probably similar)
>
> Thus, it might be a good idea to keep lzo for a while longer.
>
> Bare lzma can probably go -- xz filters are nice for binaries of archs it
> knows (disabled otherwise), and lack of header requires hacks to find out
> the payload's size.
Personally, I'd be very happy to see LZMA go. It is a custom implementation
that doesn't use the lib/xz/ library. When I was implementing decompress_zstd.c
I fuzzed all of the kernel decompressors, and unlzma() will crash on invalid input.
There is no reason, other than not breaking compatibility, to use LZMA over XZ.
> So it's up to you guys: do you want me to drop lzo and/or lzma?
> We can also drop them just for vmlinuz but not initrd.
>
>
> Meow!
> --
> ⢀⣴⠾⠻⢶⣦⠀ What Would Jesus Do, MUD/MMORPG edition:
> ⣾⠁⢰⠒⠀⣿⡁ • multiplay with an admin char to benefit your mortal [Mt3:16-17]
> ⢿⡄⠘⠷⠚⠋⠀ • abuse item cloning bugs [Mt14:17-20, Mt15:34-37]
> ⠈⠳⣄⠀⠀⠀⠀ • use glitches to walk on water [Mt14:25-26]