From: Menglong Dong <[email protected]>
If using container platforms such as Docker, upon initialization it
wants to use pivot_root() so that currently mounted devices do not
propagate to containers. An example of value in this is that
a USB device connected prior to the creation of a containers on the
host gets disconnected after a container is created; if the
USB device was mounted on containers, but already removed and
umounted on the host, the mount point will not go away until all
containers unmount the USB device.
Another reason for container platforms such as Docker to use pivot_root
is that upon initialization the net-namspace is mounted under
/var/run/docker/netns/ on the host by dockerd. Without pivot_root
Docker must either wait to create the network namespace prior to
the creation of containers or simply deal with leaking this to each
container.
pivot_root is supported if the rootfs is a initrd or block device, but
it's not supported if the rootfs uses an initramfs (tmpfs). This means
container platforms today must resort to using block devices if
they want to pivot_root from the rootfs. A workaround to use chroot()
is not a clean viable option given every container will have a
duplicate of every mount point on the host.
In order to support using container platforms such as Docker on
all the supported rootfs types we must extend Linux to support
pivot_root on initramfs as well. This patch does the work to do
just that.
pivot_root will unmount the mount of the rootfs from its parent mount
and mount the new root to it. However, when it comes to initramfs, it
donesn't work, because the root filesystem has not parent mount, which
makes initramfs not supported by pivot_root.
In order to support pivot_root on initramfs we introduce a second
"user_root" mount which is created before we do the cpio unpacking.
The filesystem of the "user_root" mount is the same the rootfs.
While mounting the 'user_root', 'rootflags' is passed to it, and it means
that we can set options for the mount of rootfs in boot cmd now.
For example, the size of tmpfs can be set with 'rootflags=size=1024M'.
Signed-off-by: Menglong Dong <[email protected]>
---
init/do_mounts.c | 101 +++++++++++++++++++++++++++++++++++++++++++++++
init/do_mounts.h | 18 ++++++++-
init/initramfs.c | 10 +++++
usr/Kconfig | 10 +++++
4 files changed, 138 insertions(+), 1 deletion(-)
diff --git a/init/do_mounts.c b/init/do_mounts.c
index a78e44ee6adb..2fd168cca480 100644
--- a/init/do_mounts.c
+++ b/init/do_mounts.c
@@ -617,6 +617,107 @@ void __init prepare_namespace(void)
init_chroot(".");
}
+#ifdef CONFIG_INITRAMFS_USER_ROOT
+#ifdef CONFIG_TMPFS
+static __init bool is_tmpfs_enabled(void)
+{
+ return (!root_fs_names || strstr(root_fs_names, "tmpfs")) &&
+ !saved_root_name[0];
+}
+#endif
+
+static __init bool is_ramfs_enabled(void)
+{
+ return true;
+}
+
+struct fs_user_root {
+ bool (*enabled)(void);
+ char *dev_name;
+ char *fs_name;
+};
+
+static struct fs_user_root user_roots[] __initdata = {
+#ifdef CONFIG_TMPFS
+ {
+ .enabled = is_tmpfs_enabled,
+ .dev_name = "tmpfs",
+ .fs_name = "tmpfs",
+ },
+#endif
+ {
+ .enabled = is_ramfs_enabled,
+ .dev_name = "ramfs",
+ .fs_name = "ramfs"
+ }
+};
+static struct fs_user_root * __initdata user_root;
+
+/*
+ * The syscall 'pivot_root' is used to change root and it is able to
+ * clean the old mounts, which make it preferred by container platforms
+ * such as Docker. However, initramfs is not supported by pivot_root,
+ * and 'chroot()' has to be used, which is unable to clean the mounts
+ * that propagate from HOST. These useless mounts make the release of
+ * removable device or network namespace a big problem.
+ *
+ * To make initramfs supported by pivot_root, the mount of the root
+ * filesystem should have a parent, which will make it unmountable. In
+ * this function, the second mount, which is called 'user root', is
+ * created and mounted on '/root', and it will be made the root filesystem
+ * in end_mount_user_root() by init_chroot().
+ *
+ * The 'user root' has a parent mount, which makes it unmountable and
+ * pivot_root work.
+ *
+ * What's more, root_mountflags and root_mount_data are used here, which
+ * makes the 'rootflags' in boot cmd work for 'user root'.
+ */
+int __init mount_user_root(void)
+{
+ return do_mount_root(user_root->dev_name,
+ user_root->fs_name,
+ root_mountflags & ~MS_RDONLY,
+ root_mount_data);
+}
+
+/*
+ * This function is used to chroot to new initramfs root that
+ * we unpacked on success. It will chdir to '/' and umount
+ * the secound mount on failure.
+ */
+void __init end_mount_user_root(bool succeed)
+{
+ init_chdir("/");
+ if (!succeed) {
+ init_umount("/root", 0);
+ return;
+ }
+
+ init_mount("/root", "/", NULL, MS_MOVE, NULL);
+ if (!ramdisk_exec_exist()) {
+ init_umount("/..", 0);
+ return;
+ }
+
+ init_chroot("/..");
+}
+
+void __init init_user_rootfs(void)
+{
+ struct fs_user_root *root;
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(user_roots); i++) {
+ root = &user_roots[i];
+ if (root->enabled()) {
+ user_root = root;
+ break;
+ }
+ }
+}
+#endif
+
static bool is_tmpfs;
static int rootfs_init_fs_context(struct fs_context *fc)
{
diff --git a/init/do_mounts.h b/init/do_mounts.h
index 7a29ac3e427b..3802c7a3ba91 100644
--- a/init/do_mounts.h
+++ b/init/do_mounts.h
@@ -10,9 +10,25 @@
#include <linux/root_dev.h>
#include <linux/init_syscalls.h>
+extern int root_mountflags;
+
void mount_block_root(char *name, int flags);
void mount_root(void);
-extern int root_mountflags;
+bool ramdisk_exec_exist(void);
+
+#ifdef CONFIG_INITRAMFS_USER_ROOT
+
+int mount_user_root(void);
+void end_mount_user_root(bool succeed);
+void init_user_rootfs(void);
+
+#else
+
+static inline int mount_user_root(void) { return 0; }
+static inline void end_mount_user_root(bool succeed) { }
+static inline void init_user_rootfs(void) { }
+
+#endif
static inline __init int create_dev(char *name, dev_t dev)
{
diff --git a/init/initramfs.c b/init/initramfs.c
index af27abc59643..ffa78932ae65 100644
--- a/init/initramfs.c
+++ b/init/initramfs.c
@@ -16,6 +16,8 @@
#include <linux/namei.h>
#include <linux/init_syscalls.h>
+#include "do_mounts.h"
+
static ssize_t __init xwrite(struct file *file, const char *p, size_t count,
loff_t *pos)
{
@@ -682,15 +684,23 @@ static void __init do_populate_rootfs(void *unused, async_cookie_t cookie)
else
printk(KERN_INFO "Unpacking initramfs...\n");
+ init_user_rootfs();
+
+ if (mount_user_root())
+ panic("Failed to create user root");
+
err = unpack_to_rootfs((char *)initrd_start, initrd_end - initrd_start);
if (err) {
+ end_mount_user_root(false);
#ifdef CONFIG_BLK_DEV_RAM
populate_initrd_image(err);
#else
printk(KERN_EMERG "Initramfs unpacking failed: %s\n", err);
#endif
+ goto done;
}
+ end_mount_user_root(true);
done:
/*
* If the initrd region is overlapped with crashkernel reserved region,
diff --git a/usr/Kconfig b/usr/Kconfig
index 8bbcf699fe3b..f9c96de539c3 100644
--- a/usr/Kconfig
+++ b/usr/Kconfig
@@ -52,6 +52,16 @@ config INITRAMFS_ROOT_GID
If you are not sure, leave it set to "0".
+config INITRAMFS_USER_ROOT
+ bool "Create 'user root' to make pivot_root supported"
+ default y
+ help
+ Before unpacking cpio, create a second mount and make it become
+ the root filesystem. Therefore, initramfs will be supported by
+ pivot_root().
+
+ If container platforms is used with initramfs, say Y.
+
config RD_GZIP
bool "Support initial ramdisk/ramfs compressed using gzip"
default y
--
2.32.0.rc0
[email protected] writes:
> From: Menglong Dong <[email protected]>
>
> If using container platforms such as Docker, upon initialization it
> wants to use pivot_root() so that currently mounted devices do not
> propagate to containers. An example of value in this is that
> a USB device connected prior to the creation of a containers on the
> host gets disconnected after a container is created; if the
> USB device was mounted on containers, but already removed and
> umounted on the host, the mount point will not go away until all
> containers unmount the USB device.
>
> Another reason for container platforms such as Docker to use pivot_root
> is that upon initialization the net-namspace is mounted under
> /var/run/docker/netns/ on the host by dockerd. Without pivot_root
> Docker must either wait to create the network namespace prior to
> the creation of containers or simply deal with leaking this to each
> container.
>
> pivot_root is supported if the rootfs is a initrd or block device, but
> it's not supported if the rootfs uses an initramfs (tmpfs). This means
> container platforms today must resort to using block devices if
> they want to pivot_root from the rootfs. A workaround to use chroot()
> is not a clean viable option given every container will have a
> duplicate of every mount point on the host.
>
> In order to support using container platforms such as Docker on
> all the supported rootfs types we must extend Linux to support
> pivot_root on initramfs as well. This patch does the work to do
> just that.
>
> pivot_root will unmount the mount of the rootfs from its parent mount
> and mount the new root to it. However, when it comes to initramfs, it
> donesn't work, because the root filesystem has not parent mount, which
> makes initramfs not supported by pivot_root.
>
> In order to support pivot_root on initramfs we introduce a second
> "user_root" mount which is created before we do the cpio unpacking.
> The filesystem of the "user_root" mount is the same the rootfs.
>
> While mounting the 'user_root', 'rootflags' is passed to it, and it means
> that we can set options for the mount of rootfs in boot cmd now.
> For example, the size of tmpfs can be set with 'rootflags=size=1024M'.
What is the flow where docker uses an initramfs?
Just thinking about this I am not being able to connect the dots.
The way I imagine the world is that an initramfs will be used either
when a linux system boots for the first time, or an initramfs would
come from the distribution you are running inside a container. In
neither case do I see docker being in a position to add functionality
to the initramfs as docker is not responsible for it.
Is docker doing something creating like running a container in a VM,
and running some directly out of the initramfs, and wanting that code
to exactly match the non-VM case?
If that is the case I think the easy solution would be to actually use
an actual ramdisk where pivot_root works.
I really don't see why it makes sense for docker to be a special
snowflake and require kernel features that no other distribution does.
It might make sense to create a completely empty filesystem underneath
an initramfs, and use that new rootfs as the unchanging root of the
mount tree, if it can be done with a trivial amount of code, and
generally make everything cleaner.
As this change sits it looks like a lot of code to handle a problem
in the implementation of docker. Which quite frankly will be a pain
to have to maintain if this is not a clean general feature that
other people can also use.
Eric
On Wed, May 26, 2021 at 2:50 AM Eric W. Biederman <[email protected]> wrote:
>
......
>
> What is the flow where docker uses an initramfs?
>
> Just thinking about this I am not being able to connect the dots.
>
> The way I imagine the world is that an initramfs will be used either
> when a linux system boots for the first time, or an initramfs would
> come from the distribution you are running inside a container. In
> neither case do I see docker being in a position to add functionality
> to the initramfs as docker is not responsible for it.
>
> Is docker doing something creating like running a container in a VM,
> and running some directly out of the initramfs, and wanting that code
> to exactly match the non-VM case?
>
> If that is the case I think the easy solution would be to actually use
> an actual ramdisk where pivot_root works.
In fact, nowadays, initramfs is widely used by embedded devices in the
production environment, which makes the whole system run in ram.
That make sense. First, running in ram will speed up the system. The size
of the system won't be too large for embedded devices, which makes this
idea work. Second, this will reduce the I/O of disk devices, which can
extend the life of the disk. Third, RAM is getting cheaper.
So in this scene, Docker runs directly in initramfs.
>
> I really don't see why it makes sense for docker to be a special
> snowflake and require kernel features that no other distribution does.
>
> It might make sense to create a completely empty filesystem underneath
> an initramfs, and use that new rootfs as the unchanging root of the
> mount tree, if it can be done with a trivial amount of code, and
> generally make everything cleaner.
>
> As this change sits it looks like a lot of code to handle a problem
> in the implementation of docker. Which quite frankly will be a pain
> to have to maintain if this is not a clean general feature that
> other people can also use.
>
I don't think that it's all for docker, pivot_root may be used by other
users in the above scene. It may work to create an empty filesystem, as you
mentioned above. But I don't think it's a good idea to make all users,
who want to use pivot_root, do that. After all, it's not friendly to
users.
As for the code, it may look a lot, but it's not complex. Maybe a clean
up for the code I add can make it better?
Thanks!
Menglong Dong
Menglong Dong <[email protected]> writes:
> On Wed, May 26, 2021 at 2:50 AM Eric W. Biederman <[email protected]> wrote:
>>
> ......
>>
>> What is the flow where docker uses an initramfs?
>>
>> Just thinking about this I am not being able to connect the dots.
>>
>> The way I imagine the world is that an initramfs will be used either
>> when a linux system boots for the first time, or an initramfs would
>> come from the distribution you are running inside a container. In
>> neither case do I see docker being in a position to add functionality
>> to the initramfs as docker is not responsible for it.
>>
>> Is docker doing something creating like running a container in a VM,
>> and running some directly out of the initramfs, and wanting that code
>> to exactly match the non-VM case?
>>
>> If that is the case I think the easy solution would be to actually use
>> an actual ramdisk where pivot_root works.
>
> In fact, nowadays, initramfs is widely used by embedded devices in the
> production environment, which makes the whole system run in ram.
>
> That make sense. First, running in ram will speed up the system. The size
> of the system won't be too large for embedded devices, which makes this
> idea work. Second, this will reduce the I/O of disk devices, which can
> extend the life of the disk. Third, RAM is getting cheaper.
>
> So in this scene, Docker runs directly in initramfs.
That is the piece of the puzzle I was missing. An small system
with it's root in an initramfs.
>> I really don't see why it makes sense for docker to be a special
>> snowflake and require kernel features that no other distribution does.
>>
>> It might make sense to create a completely empty filesystem underneath
>> an initramfs, and use that new rootfs as the unchanging root of the
>> mount tree, if it can be done with a trivial amount of code, and
>> generally make everything cleaner.
>>
>> As this change sits it looks like a lot of code to handle a problem
>> in the implementation of docker. Which quite frankly will be a pain
>> to have to maintain if this is not a clean general feature that
>> other people can also use.
>>
>
> I don't think that it's all for docker, pivot_root may be used by other
> users in the above scene. It may work to create an empty filesystem, as you
> mentioned above. But I don't think it's a good idea to make all users,
> who want to use pivot_root, do that. After all, it's not friendly to
> users.
>
> As for the code, it may look a lot, but it's not complex. Maybe a clean
> up for the code I add can make it better?
If we are going to do this something that is so small and clean it can
be done unconditionally always.
I will see if I can dig in and look at little more. I think there is
a reason Al Viro and H. Peter Anvin implemeted initramfs this way.
Perhaps it was just a desire to make pivot_root unnecessary.
Container filesystem setup does throw a bit of a wrench in the works as
unlike a initramfs where you can just delete everything there is not
a clean way to get rid of a root filesystem you don't need without
pivot_root.
The net request as I understand it: Make the filesystem the initramfs
lives in be an ordinary filesystem so it can just be used as the systems
primary filesystem.
There might be technical reasons why that is a bad idea and userspace
would be requested to move everything into another ramfs manually (which
would have the same effect). But it is take a good look to see if it
can be accomplished cleanly.
Eric
On Tue, May 25, 2021 at 10:23:09PM -0500, Eric W. Biederman wrote:
> If we are going to do this something that is so small and clean it can
> be done unconditionally always.
[...]
> The net request as I understand it: Make the filesystem the initramfs
> lives in be an ordinary filesystem so it can just be used as the systems
> primary filesystem.
Including the ability to pivot_root it away, which seems like the main
sticking point.
If this can be done without any overhead, that seems fine, but if this
involves mounting an extra filesystem, that may add an appreciable
amount of boot time for systems trying to boot in milliseconds. (Such
systems would not use an initramfs if they're going to go on and boot a
separate root filesystem, but they can use an initramfs as their *only*
filesystem.)
On Wed, May 26, 2021 at 11:23 AM Eric W. Biederman
<[email protected]> wrote:
>
> Menglong Dong <[email protected]> writes:
>
[...]
>
> If we are going to do this something that is so small and clean it can
> be done unconditionally always.
>
> I will see if I can dig in and look at little more. I think there is
> a reason Al Viro and H. Peter Anvin implemeted initramfs this way.
> Perhaps it was just a desire to make pivot_root unnecessary.
I don't think they are meant to make it this way. Unpack cpio to the
rootfs directly seems to be a normal operation. Maybe initramfs is just
ignored by pivot_root(), as it seems not a common scene to run the
whole system in RAM that time, I guess~
Thanks!
Menglong Dong
On Wed, May 26, 2021 at 12:33 PM Josh Triplett <[email protected]> wrote:
>
> On Tue, May 25, 2021 at 10:23:09PM -0500, Eric W. Biederman wrote:
> > If we are going to do this something that is so small and clean it can
> > be done unconditionally always.
> [...]
> > The net request as I understand it: Make the filesystem the initramfs
> > lives in be an ordinary filesystem so it can just be used as the systems
> > primary filesystem.
>
> Including the ability to pivot_root it away, which seems like the main
> sticking point.
>
> If this can be done without any overhead, that seems fine, but if this
> involves mounting an extra filesystem, that may add an appreciable
> amount of boot time for systems trying to boot in milliseconds. (Such
> systems would not use an initramfs if they're going to go on and boot a
> separate root filesystem, but they can use an initramfs as their *only*
> filesystem.)
Compared to the time the unpacking spent, a mounting seems nothing. In the
scene above, this change can be disabled by kconfig, if pivot_root
is not needed in initramfs.
Thanks!
Menglong Dong
On Wed, May 26, 2021 at 04:33:00PM +0800, Menglong Dong wrote:
> On Wed, May 26, 2021 at 12:33 PM Josh Triplett <[email protected]> wrote:
> >
> > On Tue, May 25, 2021 at 10:23:09PM -0500, Eric W. Biederman wrote:
> > > If we are going to do this something that is so small and clean it can
> > > be done unconditionally always.
> > [...]
> > > The net request as I understand it: Make the filesystem the initramfs
> > > lives in be an ordinary filesystem so it can just be used as the systems
> > > primary filesystem.
> >
> > Including the ability to pivot_root it away, which seems like the main
> > sticking point.
> >
> > If this can be done without any overhead, that seems fine, but if this
> > involves mounting an extra filesystem, that may add an appreciable
> > amount of boot time for systems trying to boot in milliseconds. (Such
> > systems would not use an initramfs if they're going to go on and boot a
> > separate root filesystem, but they can use an initramfs as their *only*
> > filesystem.)
>
> Compared to the time the unpacking spent, a mounting seems nothing. In the
> scene above, this change can be disabled by kconfig, if pivot_root
> is not needed in initramfs.
I asked for the kconfig entry. And it would be good to document then
also the worst case expected on boot for what this could do to you. I
mean, we are opening a different evil universe. So that's why the
kconfig exists. How bad and evil can this be?
I don't think anyone has clarified that yet.
Luis
On Wed, May 26, 2021 at 04:33:00PM +0800, Menglong Dong wrote:
> On Wed, May 26, 2021 at 12:33 PM Josh Triplett <[email protected]> wrote:
> >
> > On Tue, May 25, 2021 at 10:23:09PM -0500, Eric W. Biederman wrote:
> > > If we are going to do this something that is so small and clean it can
> > > be done unconditionally always.
> > [...]
> > > The net request as I understand it: Make the filesystem the initramfs
> > > lives in be an ordinary filesystem so it can just be used as the systems
> > > primary filesystem.
> >
> > Including the ability to pivot_root it away, which seems like the main
> > sticking point.
> >
> > If this can be done without any overhead, that seems fine, but if this
> > involves mounting an extra filesystem, that may add an appreciable
> > amount of boot time for systems trying to boot in milliseconds. (Such
> > systems would not use an initramfs if they're going to go on and boot a
> > separate root filesystem, but they can use an initramfs as their *only*
> > filesystem.)
>
> Compared to the time the unpacking spent, a mounting seems nothing. In the
> scene above, this change can be disabled by kconfig, if pivot_root
> is not needed in initramfs.
The initramfs could be as small as one file.
On Wed, May 26, 2021 at 5:03 PM Luis Chamberlain <[email protected]> wrote:
>
[...]
>
> I asked for the kconfig entry. And it would be good to document then
Wow, I didn't recognize it's you, haha~
> also the worst case expected on boot for what this could do to you. I
> mean, we are opening a different evil universe. So that's why the
> kconfig exists. How bad and evil can this be?
I just dig into it a little deeper today. Except the boot time you
mentioned, I haven't dig out other bad case yet.
I don't think this will affect the path lookup you mentioned before.
As I know, all threads in kernel are using the same root, and after
change root to 'user root', path lookup will be inside this root.
One thing, the name of the root mount will change from 'rootfs' to 'tmpfs'
or 'ramfs'. Before this change, what you can see with 'mount' is:
$ mount
rootfs on / type rootfs (rw,size=903476k,nr_inodes=225869)
After this patch, it will become:
$ mount
tmpfs on / type tmpfs (rw,size=903476k,nr_inodes=225869)
I'm not sure if this is a risk. And I tried to change pivot_root to make
it support the root that have not a parent mount, but not success yet.
That seems much more complex.
Thanks!
Menglong Dong