Hello,
While building the 3.11 kernel recently, I bumped into this issue.
Menuconfig allows one to choose compression format of an initial ramdisk
image. But this selection does not result in creation of a duly compressed
initramfs image. Because - $ make install - does not pass on the selected
compression option to dracut(8) tool, which generates the initramfs file. This
leads to a situation wherein the initramfs file is gzip(1) compressed, but the
kernel knows only to decompress the user's chosen compression format.
The attached patch herein, attempts to fix this issue by exporting the
compression configuration option to an environment variable of a sub-shell
spawned by Make(1) with the invocation of installkernel(8) during the
installation process.
The dracut(8) & new-kernel-pkg(8) tools need to be patched to recognise and
read such an environment variable and use the selected compression algorithm
while creating the initial ramdisk image files.
Could someone please review this one too?
Thank you.
--
Prasad J Pandit / Red Hat Security Response Team
DB7A 84C5 D3F9 7CD1 B5EB C939 D048 7860 3655 602B
On Sun, 15 Sep 2013 14:49:05 +0530 (IST) P J P <[email protected]> wrote:
> Make menuconfig allows one to choose compression format of an
> initial ramdisk image. But this choice does not result in duly
> compressed ramdisk image. Because - $ make install - does not
> pass on the selected compression choice to the dracut(8) tool,
> which creates the initramfs file. dracut(8) generates the image
> with the default compression, ie. gzip(1).
>
> This patch exports the selected compression option to a sub-shell
> environment, so that it could be used by dracut(8) tool to generate
> appropriately compressed initramfs images.
>
> There isn't a straight forward way to pass on options to dracut(8)
> via positional parameters. Because it is indirectly invoked at the
> end of a $ make install sequence.
>
> # make install
> -> arch/$arch/boot/Makefile
> -> arch/$arch/boot/install.sh
> -> /sbing/installkernel ...
> -> /sbin/new-kernel-pkg ...
> -> /sbin/dracut ...
>
It's a bit confusing whether all this appiles to initrd, to initramfs
or to both. Can you please clarify all this and be sure that it's all
consistent?
> --- a/arch/x86/Makefile
> +++ b/arch/x86/Makefile
> @@ -11,6 +11,23 @@ else
> KBUILD_DEFCONFIG := $(ARCH)_defconfig
> endif
>
> +# Select initial ramdisk compression format, default is gzip(1).
> +# This shall be used by the dracut(8) tool while creating an initramfs image.
> +#
> +INITRD_COMPRESS=gzip
> +ifeq ($(CONFIG_RD_BZIP2), y)
> + INITRD_COMPRESS=bzip2
> +else ifeq ($(CONFIG_RD_LZMA), y)
> + INITRD_COMPRESS=lzma
> +else ifeq ($(CONFIG_RD_XZ), y)
> + INITRD_COMPRESS=xz
> +else ifeq ($(CONFIG_RD_LZO), y)
> + INITRD_COMPRESS=lzo
> +else ifeq ($(CONFIG_RD_LZ4), y)
> + INITRD_COMPRESS=lz4
> +endif
> +export INITRD_COMPRESS
A few things...
- Why is it specific to x86? Other architcetures use initramfs?
- People add new compression schemes fairly regularly. We should add
a code comment somewhere in a place where such people are sure to
find it. That comment should explain what's going on, remind them to
update this environment varlaible and should explain to them the
process by which they get dracut(8) updated if needed.
- Can we avoid having to update dracut each time a new compression
scheme is added? I assume your dracut changes will just exec
"$INITRD_COMPRESS -d", so as long as any new decompression
application uses the expected -d argument in the expected way, it
should work seamlessly?
Hello Andrew,
Thank you so much for reviewing these patches.
+-- On Mon, 23 Sep 2013, Andrew Morton wrote --+
| It's a bit confusing whether all this appiles to initrd, to initramfs
| or to both. Can you please clarify all this and be sure that it's all
| consistent?
IIUC, we no longer use initrd block device images. Instead we use initramfs
which is a cpio(1) archive image. Both mkinitrd(8) & dracut(8) are invoked
from /sbin/new-kernel-pkg tool. And mkinitrd(8) is nothing but a wrapper
around dracut(8) these days.
===
$ man mkinitrd
mkinitrd - is a compat wrapper, which calls dracut to generate an
initramfs
$ less /sbin/new-kernel-pkg # my local patched version
...
doMkinitrd() {
if [ -n "$dracut" ]; then
+ if [ -n "$INITRD_COMPRESS" ]; then
+ dracutcompress=--$INITRD_COMPRESS
+ fi
+ tool="dracut $dracuthostonly -f $dracutcompress $initrdfile $version"
else
tool="mkinitrd --allow-missing -f $initrdfile $version"
fi
[ -n "$verbose" ] && echo "creating initrd: $tool"
$tool
rc=$?
if [ $rc != 0 ]; then
echo "mkinitrd failed" >&2
exit 1
fi
}
===
Because mkinitrd(8) does not support passing of compression argument to
dracut(8), it must use the default compression, ie. gzip(1).
| A few things...
| - Why is it specific to x86? Other architcetures use initramfs?
No, it is not specific to x86, most likely all architecturess' Makefile
would need similar patch. But I haven't looked into those yet.
| - People add new compression schemes fairly regularly. We should add
| a code comment somewhere in a place where such people are sure to
| find it. That comment should explain what's going on, remind them to
| update this environment varlaible and should explain to them the
| process by which they get dracut(8) updated if needed.
True. Maybe user/Kconfig could have a note about updating INITRD_COMPRESS
environment variable through a relevant $arch/Makefile ?
Looking at the dracut(8) tool, it seems it can easily support any number of
compression tools, for it already has options --gzip, --bzip2, --xz & --lzma.
Adding --lzo, --lz4 etc won't be much difficult.
| - Can we avoid having to update dracut each time a new compression
| scheme is added? I assume your dracut changes will just exec
| "$INITRD_COMPRESS -d", so as long as any new decompression
| application uses the expected -d argument in the expected way, it
| should work seamlessly?
Yes, dracut(8) has | --compress=$INITRD_COMPRESS | option which could be
used to pass compression program to dracut(8). Not sure if dracut(8) expects
an absolute path or just the program name. But to use this option, we'll need
to patch /sbin/new-kernel-pkg tool.
Alternatively, dracut(8) could be patched to recognise and read
$INITRD_COMPRESS variable and internally it can define precedence order
between environment variable, command-line options and default fall-back
option.
Thank you.
--
Prasad J Pandit / Red Hat Security Response Team
DB7A 84C5 D3F9 7CD1 B5EB C939 D048 7860 3655 602B
On 09/24/2013 02:41:54 PM, P J P wrote:
> Hello Andrew,
>
> Thank you so much for reviewing these patches.
>
> +-- On Mon, 23 Sep 2013, Andrew Morton wrote --+
> | It's a bit confusing whether all this appiles to initrd, to
> initramfs
> | or to both. Can you please clarify all this and be sure that it's
> all
> | consistent?
>
> IIUC, we no longer use initrd block device images. Instead we use
> initramfs
> which is a cpio(1) archive image. Both mkinitrd(8) & dracut(8) are
> invoked
> from /sbin/new-kernel-pkg tool. And mkinitrd(8) is nothing but a
> wrapper
> around dracut(8) these days.
$ find . -name "Makefile*" | xargs grep new-kernel-pkg
$ grep -r new-kernel-pkg scripts
$
Ah, so it's an out of tree bespoke Red Hat tool. No wonder I couldn't
find it.
> Looking at the dracut(8) tool, it seems it can easily support any
> number of
> compression tools, for it already has options --gzip, --bzip2, --xz &
> --lzma.
> Adding --lzo, --lz4 etc won't be much difficult.
You're reimplemented the posix "pax" command?
> | - Can we avoid having to update dracut each time a new compression
> | scheme is added? I assume your dracut changes will just exec
> | "$INITRD_COMPRESS -d", so as long as any new decompression
> | application uses the expected -d argument in the expected way, it
> | should work seamlessly?
>
> Yes, dracut(8) has | --compress=$INITRD_COMPRESS | option which
> could be
> used to pass compression program to dracut(8).
$ grep -r INITRD_COMPRESS linux
$
Is this what you're currently doing, or the change you're requesting in
the kernel?
> Not sure if dracut(8) expects
> an absolute path or just the program name. But to use this option,
> we'll need
> to patch /sbin/new-kernel-pkg tool.
>
> Alternatively, dracut(8) could be patched to recognise and read
> $INITRD_COMPRESS variable and internally it can define precedence
> order
> between environment variable, command-line options and default
> fall-back
> option.
How does your code get called?
Rob-
+-- On Wed, 25 Sep 2013, Rob Landley wrote --+
| Ah, so it's an out of tree bespoke Red Hat tool. No wonder I couldn't find it.
It is not Red Hat tool.
| You're reimplemented the posix "pax" command?
Ummn, not sure. Didn't see anything about 'pax'.
| Is this what you're currently doing, or the change you're requesting in the
| kernel?
Yes, I'm proposing to export $INITRD_COMPRESS variable so that it can be
used by dracut(8).
| How does your code get called?
The patch's commit log explains it.
Thank you.
--
Prasad J Pandit / Red Hat Security Response Team
DB7A 84C5 D3F9 7CD1 B5EB C939 D048 7860 3655 602B
Hello Andrew,
I was wondering if you had a chance to review this patch further? Should I
send similar patches for other architectures too? As in you aren't waiting on
me for that, are you?
Thank you.
--
Prasad J Pandit / Red Hat Security Response Team
DB7A 84C5 D3F9 7CD1 B5EB C939 D048 7860 3655 602B
Hello Andrew,
Just to check, did you have a chance to review it further? Are you waiting on
me for something?
--
Prasad J Pandit / Red Hat Security Response Team
DB7A 84C5 D3F9 7CD1 B5EB C939 D048 7860 3655 602B
On Wed, 25 Sep 2013 01:11:54 +0530 (IST) P J P <[email protected]> wrote:
> | A few things...
> | - Why is it specific to x86? Other architcetures use initramfs?
>
> No, it is not specific to x86, most likely all architecturess' Makefile
> would need similar patch. But I haven't looked into those yet.
It would be better to make the change in one place, rather than for
each architecture. That would appear to involve moving a hunk from
arch/x86/Makefile into init/Makefile, or perhaps ./Makefile.
That way it can still be tested on x86, and it is very reasonable to
assume that it will work as intended on other architectures.
Hello Andrew,
+-- On Wed, 9 Oct 2013, Andrew Morton wrote --+
| It would be better to make the change in one place, rather than for each
| architecture. That would appear to involve moving a hunk from
| arch/x86/Makefile into init/Makefile, or perhaps ./Makefile.
Right, I was trying to figure the same. I'll do the changes and send an
updated patch.
Thank you so much for reviewing it and the update.
Thank you.
--
Prasad J Pandit / Red Hat Security Response Team
Hi,
+-- On Wed, 9 Oct 2013, Andrew Morton wrote --+
| It would be better to make the change in one place, rather than for
| each architecture. That would appear to involve moving a hunk from
| arch/x86/Makefile into init/Makefile, or perhaps ./Makefile.
Please find attached herein, an updated patch which adds the export snippet
to the top level ./Makefile. I tried init/Makefile too, but it does not seem
to work.
Thank you!
--
Prasad J Pandit / Red Hat Security Response Team
Hello Andrew,
+-- On Wed, 9 Oct 2013, Andrew Morton wrote --+
| It would be better to make the change in one place, rather than for
| each architecture. That would appear to involve moving a hunk from
| arch/x86/Makefile into init/Makefile, or perhaps ./Makefile.
Please see this revised version, I missed to include init/do_mounts_rd.c
changes in the previous one.
Thank you.
--
Prasad J Pandit / Red Hat Security Response Team