The main need for this is to support container runtimes on stateless Linux
system (pivot_root system call from initramfs).
Normally, the task of initramfs is to mount and switch to a "real" root
filesystem. However, on stateless systems (booting over the network) it is just
convenient to have your "real" filesystem as initramfs from the start.
This, however, breaks different container runtimes, because they usually use
pivot_root system call after creating their mount namespace. But pivot_root does
not work from initramfs, because initramfs runs form rootfs, which is the root
of the mount tree and can't be unmounted.
One can solve this problem from userspace, but it is much more cumbersome. We
either have to create a multilayered archive for initramfs, where the outer
layer creates a tmpfs filesystem and unpacks the inner layer, switches root and
does not forget to properly cleanup the old rootfs. Or we need to use keepinitrd
kernel cmdline option, unpack initramfs to rootfs, run a script to create our
target tmpfs root, unpack the same initramfs there, switch root to it and again
properly cleanup the old root, thus unpacking the same archive twice and also
wasting memory, because kernel stores compressed initramfs image indefinitely.
With this change we can ask the kernel (by specifying nonroot_initramfs kernel
cmdline option) to create a "leaf" tmpfs mount for us and switch root to it
before the initramfs handling code, so initramfs gets unpacked directly into
the "leaf" tmpfs with rootfs being empty and no need to clean up anything.
Signed-off-by: Ignat Korchagin <[email protected]>
---
fs/namespace.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 47 insertions(+)
diff --git a/fs/namespace.c b/fs/namespace.c
index 85b5f7bea82e..a1ec862e8146 100644
--- a/fs/namespace.c
+++ b/fs/namespace.c
@@ -3701,6 +3701,49 @@ static void __init init_mount_tree(void)
set_fs_root(current->fs, &root);
}
+#if IS_ENABLED(CONFIG_TMPFS)
+static int __initdata nonroot_initramfs;
+
+static int __init nonroot_initramfs_param(char *str)
+{
+ if (*str)
+ return 0;
+ nonroot_initramfs = 1;
+ return 1;
+}
+__setup("nonroot_initramfs", nonroot_initramfs_param);
+
+static void __init init_nonroot_initramfs(void)
+{
+ int err;
+
+ if (!nonroot_initramfs)
+ return;
+
+ err = ksys_mkdir("/root", 0700);
+ if (err < 0)
+ goto out;
+
+ err = do_mount("tmpfs", "/root", "tmpfs", 0, NULL);
+ if (err)
+ goto out;
+
+ err = ksys_chdir("/root");
+ if (err)
+ goto out;
+
+ err = do_mount(".", "/", NULL, MS_MOVE, NULL);
+ if (err)
+ goto out;
+
+ err = ksys_chroot(".");
+ if (!err)
+ return;
+out:
+ printk(KERN_WARNING "Failed to create a non-root filesystem for initramfs\n");
+}
+#endif /* IS_ENABLED(CONFIG_TMPFS) */
+
void __init mnt_init(void)
{
int err;
@@ -3734,6 +3777,10 @@ void __init mnt_init(void)
shmem_init();
init_rootfs();
init_mount_tree();
+
+#if IS_ENABLED(CONFIG_TMPFS)
+ init_nonroot_initramfs();
+#endif
}
void put_mnt_ns(struct mnt_namespace *ns)
--
2.20.1
On Thu, Mar 05, 2020 at 07:35:11PM +0000, Ignat Korchagin wrote:
> The main need for this is to support container runtimes on stateless Linux
> system (pivot_root system call from initramfs).
>
> Normally, the task of initramfs is to mount and switch to a "real" root
> filesystem. However, on stateless systems (booting over the network) it is just
> convenient to have your "real" filesystem as initramfs from the start.
>
> This, however, breaks different container runtimes, because they usually use
> pivot_root system call after creating their mount namespace. But pivot_root does
> not work from initramfs, because initramfs runs form rootfs, which is the root
> of the mount tree and can't be unmounted.
>
> One can solve this problem from userspace, but it is much more cumbersome. We
> either have to create a multilayered archive for initramfs, where the outer
> layer creates a tmpfs filesystem and unpacks the inner layer, switches root and
> does not forget to properly cleanup the old rootfs. Or we need to use keepinitrd
> kernel cmdline option, unpack initramfs to rootfs, run a script to create our
> target tmpfs root, unpack the same initramfs there, switch root to it and again
> properly cleanup the old root, thus unpacking the same archive twice and also
> wasting memory, because kernel stores compressed initramfs image indefinitely.
>
> With this change we can ask the kernel (by specifying nonroot_initramfs kernel
> cmdline option) to create a "leaf" tmpfs mount for us and switch root to it
> before the initramfs handling code, so initramfs gets unpacked directly into
> the "leaf" tmpfs with rootfs being empty and no need to clean up anything.
IDGI. Why not simply this as the first thing from your userland:
mount("/", "/", NULL, MS_BIND | MS_REC, NULL);
chdir("/..");
chroot(".");
3 syscalls and you should be all set...
On Thu, 2020-03-05 at 19:35 +0000, Ignat Korchagin wrote:
> The main need for this is to support container runtimes on stateless
> Linux system (pivot_root system call from initramfs).
>
> Normally, the task of initramfs is to mount and switch to a "real"
> root filesystem. However, on stateless systems (booting over the
> network) it is just convenient to have your "real" filesystem as
> initramfs from the start.
>
> This, however, breaks different container runtimes, because they
> usually use pivot_root system call after creating their mount
> namespace. But pivot_root does not work from initramfs, because
> initramfs runs form rootfs, which is the root of the mount tree and
> can't be unmounted.
Can you say more about why this is a problem? We use pivot_root to
pivot from the initramfs rootfs to the newly discovered and mounted
real root ... the same mechanism should work for a container (mount
namespace) running from initramfs ... why doesn't it?
The sequence usually looks like: create and enter a mount namespace,
build a tmpfs for the container in some $root directory then do
cd $root
mkdir old-root
pivot_root . old-root
mount --
make-rprivate /old-root
umount -l /old-root
rmdir /old-root
Once that's done you're disconnected from the initramfs root. The
sequence is really no accident because it's what the initramfs would
have done to pivot to the new root anyway (that's where container
people got it from).
James
On Thu, Mar 05, 2020 at 01:09:10PM -0800, James Bottomley wrote:
> On Thu, 2020-03-05 at 19:35 +0000, Ignat Korchagin wrote:
> > The main need for this is to support container runtimes on stateless
> > Linux system (pivot_root system call from initramfs).
> >
> > Normally, the task of initramfs is to mount and switch to a "real"
> > root filesystem. However, on stateless systems (booting over the
> > network) it is just convenient to have your "real" filesystem as
> > initramfs from the start.
> >
> > This, however, breaks different container runtimes, because they
> > usually use pivot_root system call after creating their mount
> > namespace. But pivot_root does not work from initramfs, because
> > initramfs runs form rootfs, which is the root of the mount tree and
> > can't be unmounted.
>
> Can you say more about why this is a problem? We use pivot_root to
> pivot from the initramfs rootfs to the newly discovered and mounted
> real root ... the same mechanism should work for a container (mount
> namespace) running from initramfs ... why doesn't it?
Not sure how it interacts with mount namespaces, but we don't use
pivot_root to go from rootfs to the real root. We use switch_root, which
moves the new root onto the old / using mount with MS_MOVE and then
chroot to it.
https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt
>
> The sequence usually looks like: create and enter a mount namespace,
> build a tmpfs for the container in some $root directory then do
>
>
> cd $root
> mkdir old-root
> pivot_root . old-root
> mount --
> make-rprivate /old-root
> umount -l /old-root
> rmdir /old-root
>
> Once that's done you're disconnected from the initramfs root. The
> sequence is really no accident because it's what the initramfs would
> have done to pivot to the new root anyway (that's where container
> people got it from).
>
>
> James
>
On Thu, Mar 5, 2020 at 8:21 PM Al Viro <[email protected]> wrote:
>
> On Thu, Mar 05, 2020 at 07:35:11PM +0000, Ignat Korchagin wrote:
> > The main need for this is to support container runtimes on stateless Linux
> > system (pivot_root system call from initramfs).
> >
> > Normally, the task of initramfs is to mount and switch to a "real" root
> > filesystem. However, on stateless systems (booting over the network) it is just
> > convenient to have your "real" filesystem as initramfs from the start.
> >
> > This, however, breaks different container runtimes, because they usually use
> > pivot_root system call after creating their mount namespace. But pivot_root does
> > not work from initramfs, because initramfs runs form rootfs, which is the root
> > of the mount tree and can't be unmounted.
> >
> > One can solve this problem from userspace, but it is much more cumbersome. We
> > either have to create a multilayered archive for initramfs, where the outer
> > layer creates a tmpfs filesystem and unpacks the inner layer, switches root and
> > does not forget to properly cleanup the old rootfs. Or we need to use keepinitrd
> > kernel cmdline option, unpack initramfs to rootfs, run a script to create our
> > target tmpfs root, unpack the same initramfs there, switch root to it and again
> > properly cleanup the old root, thus unpacking the same archive twice and also
> > wasting memory, because kernel stores compressed initramfs image indefinitely.
> >
> > With this change we can ask the kernel (by specifying nonroot_initramfs kernel
> > cmdline option) to create a "leaf" tmpfs mount for us and switch root to it
> > before the initramfs handling code, so initramfs gets unpacked directly into
> > the "leaf" tmpfs with rootfs being empty and no need to clean up anything.
>
> IDGI. Why not simply this as the first thing from your userland:
> mount("/", "/", NULL, MS_BIND | MS_REC, NULL);
> chdir("/..");
> chroot(".");
> 3 syscalls and you should be all set...
(sorry for duplicate - didn't press "reply all" the first time)
Container people really prefer pivot_root over chroot due to some
security concerns around chroot.
As far as my (probably limited) understanding goes, while the above
approach will make it work,
it will have the same security implications as just using chroot: we
trick the system to perform
pivot_root, however we don't get rid of the actual host root
filesystem in the cloned namespace.
On Thu, Mar 5, 2020 at 10:21 PM Arvind Sankar <[email protected]> wrote:
>
> On Thu, Mar 05, 2020 at 01:09:10PM -0800, James Bottomley wrote:
> > On Thu, 2020-03-05 at 19:35 +0000, Ignat Korchagin wrote:
> > > The main need for this is to support container runtimes on stateless
> > > Linux system (pivot_root system call from initramfs).
> > >
> > > Normally, the task of initramfs is to mount and switch to a "real"
> > > root filesystem. However, on stateless systems (booting over the
> > > network) it is just convenient to have your "real" filesystem as
> > > initramfs from the start.
> > >
> > > This, however, breaks different container runtimes, because they
> > > usually use pivot_root system call after creating their mount
> > > namespace. But pivot_root does not work from initramfs, because
> > > initramfs runs form rootfs, which is the root of the mount tree and
> > > can't be unmounted.
> >
> > Can you say more about why this is a problem? We use pivot_root to
> > pivot from the initramfs rootfs to the newly discovered and mounted
> > real root ... the same mechanism should work for a container (mount
> > namespace) running from initramfs ... why doesn't it?
>
> Not sure how it interacts with mount namespaces, but we don't use
> pivot_root to go from rootfs to the real root. We use switch_root, which
> moves the new root onto the old / using mount with MS_MOVE and then
> chroot to it.
>
> https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt
>
> >
> > The sequence usually looks like: create and enter a mount namespace,
> > build a tmpfs for the container in some $root directory then do
> >
> >
> > cd $root
> > mkdir old-root
> > pivot_root . old-root
> > mount --
> > make-rprivate /old-root
> > umount -l /old-root
> > rmdir /old-root
> >
> > Once that's done you're disconnected from the initramfs root. The
> > sequence is really no accident because it's what the initramfs would
> > have done to pivot to the new root anyway (that's where container
> > people got it from).
> >
> >
> > James
> >
Yes, to add to Arvind's point the above sequence will only work for
"old style" initrd (block ramdisk with some filesystem image on top),
but will not work for the "new style" initramfs (just a disguised
tmpfs). The sequence will fail on "pivot_root" with EINVAL (see
pivot_root(2)). In fact this patch conceptually tries to have the same
behaviour as with "old style" initrd. As currently, if you use initrd:
1. The kernel will create an empty "dummy" initramfs
2. Create a ramdisk
3. Unpack the FS image into the ramdisk
4. Mount the the disk
5. Do switch_root/move etc
So we have initial mount tree as: rootfs->some_initrd_fs
(and pivot_root works here and you get empty rootfs by default)
With this option we have similar in the end: rootfs->tmpfs
and rootfs is empty, because the kernel never unpacked anything there.
Following up on this: is there a way to move this forward somehow or
we're stuck with complex userspace code to achieve the same, like
https://git.kernel.org/pub/scm/libs/klibc/klibc.git/tree/usr/kinit/run-init/runinitlib.c,
but for tmpfs?
Just FYI, here is an example of a fix for a security issue, which is
caused by using chroot vs pivot_root in containers:
https://github.com/opencontainers/runc/commit/28a697cce3e4f905dca700eda81d681a30eef9cd
Alternatively, if the use-case is not generic enough, we could keep
the patch to ourselves - just would appreciate some advice/potential
concerns with this approach which we might have overlooked.
Thanks,
Ignat
On Thu, Mar 5, 2020 at 7:35 PM Ignat Korchagin <[email protected]> wrote:
>
> The main need for this is to support container runtimes on stateless Linux
> system (pivot_root system call from initramfs).
>
> Normally, the task of initramfs is to mount and switch to a "real" root
> filesystem. However, on stateless systems (booting over the network) it is just
> convenient to have your "real" filesystem as initramfs from the start.
>
> This, however, breaks different container runtimes, because they usually use
> pivot_root system call after creating their mount namespace. But pivot_root does
> not work from initramfs, because initramfs runs form rootfs, which is the root
> of the mount tree and can't be unmounted.
>
> One can solve this problem from userspace, but it is much more cumbersome. We
> either have to create a multilayered archive for initramfs, where the outer
> layer creates a tmpfs filesystem and unpacks the inner layer, switches root and
> does not forget to properly cleanup the old rootfs. Or we need to use keepinitrd
> kernel cmdline option, unpack initramfs to rootfs, run a script to create our
> target tmpfs root, unpack the same initramfs there, switch root to it and again
> properly cleanup the old root, thus unpacking the same archive twice and also
> wasting memory, because kernel stores compressed initramfs image indefinitely.
>
> With this change we can ask the kernel (by specifying nonroot_initramfs kernel
> cmdline option) to create a "leaf" tmpfs mount for us and switch root to it
> before the initramfs handling code, so initramfs gets unpacked directly into
> the "leaf" tmpfs with rootfs being empty and no need to clean up anything.
>
> Signed-off-by: Ignat Korchagin <[email protected]>
> ---
> fs/namespace.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 47 insertions(+)
>
> diff --git a/fs/namespace.c b/fs/namespace.c
> index 85b5f7bea82e..a1ec862e8146 100644
> --- a/fs/namespace.c
> +++ b/fs/namespace.c
> @@ -3701,6 +3701,49 @@ static void __init init_mount_tree(void)
> set_fs_root(current->fs, &root);
> }
>
> +#if IS_ENABLED(CONFIG_TMPFS)
> +static int __initdata nonroot_initramfs;
> +
> +static int __init nonroot_initramfs_param(char *str)
> +{
> + if (*str)
> + return 0;
> + nonroot_initramfs = 1;
> + return 1;
> +}
> +__setup("nonroot_initramfs", nonroot_initramfs_param);
> +
> +static void __init init_nonroot_initramfs(void)
> +{
> + int err;
> +
> + if (!nonroot_initramfs)
> + return;
> +
> + err = ksys_mkdir("/root", 0700);
> + if (err < 0)
> + goto out;
> +
> + err = do_mount("tmpfs", "/root", "tmpfs", 0, NULL);
> + if (err)
> + goto out;
> +
> + err = ksys_chdir("/root");
> + if (err)
> + goto out;
> +
> + err = do_mount(".", "/", NULL, MS_MOVE, NULL);
> + if (err)
> + goto out;
> +
> + err = ksys_chroot(".");
> + if (!err)
> + return;
> +out:
> + printk(KERN_WARNING "Failed to create a non-root filesystem for initramfs\n");
> +}
> +#endif /* IS_ENABLED(CONFIG_TMPFS) */
> +
> void __init mnt_init(void)
> {
> int err;
> @@ -3734,6 +3777,10 @@ void __init mnt_init(void)
> shmem_init();
> init_rootfs();
> init_mount_tree();
> +
> +#if IS_ENABLED(CONFIG_TMPFS)
> + init_nonroot_initramfs();
> +#endif
> }
>
> void put_mnt_ns(struct mnt_namespace *ns)
> --
> 2.20.1
>