This is an update to my multi-hierarchy multi-subsystem generic
process containers patch. Changes since V9 (April 27th) include:
- The patchset has been rebased over 2.6.22-rc2-mm1
- A lattice of lists linking tasks to their css_groups and css_groups
to their containers has been added to support more efficient iteration
across the member tasks of a container.
- Support for the cpusets "release agent" functionality has been added
back in; this is based on a workqueue concept similar to the changes
that Cliff Wickman has been pushing for supporting CPU hot-unplug.
- Several uses of tasklist_lock replaced by reliance on RCU
- Misc cleanups
- Tested with a tweaked version of PaulJ's cpuset_test script
Still TODO:
- decide whether "Containers" is an acceptable name for the system
given its usage by some other development groups, or whether something
else (ProcessSets? ResourceGroups? TaskGroups?) would be better. I'm
inclined to leave this political decision to Andrew/Linus once they're
happy with the technical aspects of the patches.
- add a hash-table based lookup for css_group objects.
- use seq_file properly in container tasks files to avoid having to
allocate a big array for all the container's task pointers.
- lots more testing
- define standards for container file names
--
Generic Process Containers
--------------------------
There have recently been various proposals floating around for
resource management/accounting and other task grouping subsystems in
the kernel, including ResGroups, User BeanCounters, NSProxy
containers, and others. These all need the basic abstraction of being
able to group together multiple processes in an aggregate, in order to
track/limit the resources permitted to those processes, or control
other behaviour of the processes, and all implement this grouping in
different ways.
Already existing in the kernel is the cpuset subsystem; this has a
process grouping mechanism that is mature, tested, and well documented
(particularly with regards to synchronization rules).
This patchset extracts the process grouping code from cpusets into a
generic container system, and makes the cpusets code a client of the
container system, along with a couple of simple example subsystems.
The patch set is structured as follows:
1) Basic container framework - filesystem and tracking structures
2) Simple CPU Accounting example subsystem
3) Support for the "tasks" control file
4) Hooks for fork() and exit()
5) Support for the container_clone() operation
6) Add /proc reporting interface
7) Make cpusets a container subsystem
8) Share container subsystem pointer arrays between tasks with the
same assignments
9) Simple container debugging subsystem
10) Support for a userspace "release agent", similar to the cpusets
release agent functionality
The intention is that the various resource management and
virtualization efforts can also become container clients, with the
result that:
- the userspace APIs are (somewhat) normalised
- it's easier to test out e.g. the ResGroups CPU controller in
conjunction with the BeanCounters memory controller, or use either of
them as the resource-control portion of a virtual server system.
- the additional kernel footprint of any of the competing resource
management systems is substantially reduced, since it doesn't need
to provide process grouping/containment, hence improving their
chances of getting into the kernel
Signed-off-by: Paul Menage <[email protected]>
On Tue, 29 May 2007 06:01:04 -0700 [email protected] wrote:
> This is an update to my multi-hierarchy multi-subsystem generic
> process containers patch.
>
> ...
>
> Still TODO:
>
> ...
>
> - lots more testing
>
So how do we do this?
Is there any sneaky way in which we can modify the kernel so that this new
code gets exercised more? Obviously, tossing init into some default
system-wide container would be a start. But I wonder if we can be
sneakier - for example, create a new container on each setuid(), toss the
task into that. Or something along those lines?
On Wed, May 30, 2007 at 12:14:55AM -0700, Andrew Morton wrote:
> So how do we do this?
> Is there any sneaky way in which we can modify the kernel so that this new
> code gets exercised more? Obviously, tossing init into some default
> system-wide container would be a start. But I wonder if we can be
> sneakier - for example, create a new container on each setuid(), toss the
> task into that. Or something along those lines?
How about a container for each thread group, pgrp, session, and user?
-- wli
Andrew Morton wrote:
> On Tue, 29 May 2007 06:01:04 -0700 [email protected] wrote:
>
>> This is an update to my multi-hierarchy multi-subsystem generic
>> process containers patch.
>>
>> ...
>>
>> Still TODO:
>>
>> ...
>>
>> - lots more testing
>>
>
> So how do we do this?
>
> Is there any sneaky way in which we can modify the kernel so that this new
> code gets exercised more? Obviously, tossing init into some default
> system-wide container would be a start. But I wonder if we can be
> sneakier - for example, create a new container on each setuid(), toss the
> task into that. Or something along those lines?
Please, lets get the RSS controller in. It's ready, been tested
and commented on widely.
I'll also start porting my containerstats patches on top of -v10.
--
Thanks,
Balbir Singh
Linux Technology Center
IBM, ISTL
Balbir Singh wrote:
> Andrew Morton wrote:
>> On Tue, 29 May 2007 06:01:04 -0700 [email protected] wrote:
>>
>>> This is an update to my multi-hierarchy multi-subsystem generic
>>> process containers patch.
>>>
>>> ...
>>>
>>> Still TODO:
>>>
>>> ...
>>>
>>> - lots more testing
>>>
>> So how do we do this?
>>
>> Is there any sneaky way in which we can modify the kernel so that this new
>> code gets exercised more? Obviously, tossing init into some default
>> system-wide container would be a start. But I wonder if we can be
>> sneakier - for example, create a new container on each setuid(), toss the
>> task into that. Or something along those lines?
>
> Please, lets get the RSS controller in. It's ready, been tested
It is not 100% ready yet actually :) I am working on it right now
and hope to get ready till tomorrow.
> and commented on widely.
Yup :) Balbir, thanks for testing, your patches are already in.
> I'll also start porting my containerstats patches on top of -v10.
>
Pavel Emelianov wrote:
>>> Is there any sneaky way in which we can modify the kernel so that this new
>>> code gets exercised more? Obviously, tossing init into some default
>>> system-wide container would be a start. But I wonder if we can be
>>> sneakier - for example, create a new container on each setuid(), toss the
>>> task into that. Or something along those lines?
>> Please, lets get the RSS controller in. It's ready, been tested
>
> It is not 100% ready yet actually :) I am working on it right now
> and hope to get ready till tomorrow.
>
By ready, I meant ready for inclusion as a concept/approach.
>> and commented on widely.
>
> Yup :) Balbir, thanks for testing, your patches are already in.
>
Thanks for including them.
--
Warm Regards,
Balbir Singh
Linux Technology Center
IBM, ISTL
Hi Paul.
I have faced a warning during testing your patches.
The testcase is simple:
# ssh to the node
mount -t container none /cnt/rss/ -o rss
mkdir /cnt/rss/0
/bin/echo $$ > /cnt/rss/0/tasks
# exit with ^d and ssh again
rmdir /cnt/rss/0
dmesg
BUG: at mm/slab.c:777 __find_general_cachep()
[<c04656c8>] __kmalloc+0x3f/0xa5
[<c0440e3a>] container_tasks_open+0x56/0x11f
[<c0440bcc>] container_file_open+0x0/0x36
[<c0440bfb>] container_file_open+0x2f/0x36
[<c0467a12>] __dentry_open+0xc1/0x178
[<c0467b43>] nameidata_to_filp+0x24/0x33
[<c0467b84>] do_filp_open+0x32/0x39
[<c04678eb>] get_unused_fd+0x50/0xb6
[<c0467bcd>] do_sys_open+0x42/0xbe
[<c0467c82>] sys_open+0x1c/0x1e
[<c0404c12>] sysenter_past_esp+0x5f/0x85
[<c05b0000>] __xfrm_policy_check+0x11a/0x4f6
The bug seems to be here:
static int container_tasks_open(struct inode *unused, struct file *file)
{
...
npids = container_task_count(cont);
pidarray = kmalloc(npids * sizeof(pid_t), GFP_KERNEL);
if (!pidarray)
goto err1;
...
}
The npids happened to be 0 and kmalloc warns that size is zero.
Thanks,
Pavel.
Hi Paul,
I've got two problems working with this patchset:
1. A task can't join a cpuset unless 'cpus' and 'mems' are set. These
don't seem to automatically inherit the parent's values. So when I do
mount -t container -o ns,cpuset nsproxy /containers
(unshare a namespace)
the unshare fails because container_clone() created a new cpuset
container but the task couldn't automatically enter that new cpuset.
2. I can't delete containers because of the files they contain, and
am not allowed to delete those files by hand.
thanks,
-serge
What you describe, Serge, sounds like semantics carried over from cpusets.
Serge wrote:
> A task can't join a cpuset unless 'cpus' and 'mems' are set.
Yup - don't want to run a task in a cpuset that lacks cpu, or lacks
memory. Hard to run without those.
> These don't seem to automatically inherit the parent's values
Yup - early in the life of cpusets, a created cpuset inherited the cpus
and mems of its parent. But that broke the exclusive property big
time. You will recall that a cpu_exclusive or mem_exclusive cpuset
cannot overlap the cpus or memory, respectively, of any of its sibling
cpusets.
So we changed it to creating new cpusets empty of cpus or memory.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401
On 6/4/07, Paul Jackson <[email protected]> wrote:
>
> Yup - early in the life of cpusets, a created cpuset inherited the cpus
> and mems of its parent. But that broke the exclusive property big
> time. You will recall that a cpu_exclusive or mem_exclusive cpuset
> cannot overlap the cpus or memory, respectively, of any of its sibling
> cpusets.
>
Maybe we could make it a per-cpuset option whether children should
inherit mems/cpus or not?
Paul
On 6/4/07, Serge E. Hallyn <[email protected]> wrote:
>
> 2. I can't delete containers because of the files they contain, and
> am not allowed to delete those files by hand.
>
You should be able to delete a container with rmdir as long as it's
not in use - its control files will get cleaned up automatically.
If you're getting an EBUSY error that means that either there are
still tasks running in the container (look in the "tasks" file) or
else there's a reference counting bug somewhere.
Can you post an example to reproduce the problem?
Paul
Paul M wrote:
> Maybe we could make it a per-cpuset option whether children should
> inherit mems/cpus or not?
I suppose, if those needing inherited mems/cpus need it bad enough.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401
Quoting Paul Menage ([email protected]):
> On 6/4/07, Paul Jackson <[email protected]> wrote:
> >
> >Yup - early in the life of cpusets, a created cpuset inherited the cpus
> >and mems of its parent. But that broke the exclusive property big
> >time. You will recall that a cpu_exclusive or mem_exclusive cpuset
> >cannot overlap the cpus or memory, respectively, of any of its sibling
> >cpusets.
> >
>
> Maybe we could make it a per-cpuset option whether children should
> inherit mems/cpus or not?
The values can be changed after the cpuset is populated, right? So
really these are just defaults? Would it then make sense to just
default to (parent_set - sibling_exclusive_set) for a new sibling's
value?
An option is fine with me, but without such an option at all, cpusets
could not be applied to namespaces...
thanks,
-serge
Quoting Paul Menage ([email protected]):
> On 6/4/07, Serge E. Hallyn <[email protected]> wrote:
> >
> >2. I can't delete containers because of the files they contain, and
> >am not allowed to delete those files by hand.
> >
>
> You should be able to delete a container with rmdir as long as it's
> not in use - its control files will get cleaned up automatically.
>
> If you're getting an EBUSY error that means that either there are
> still tasks running in the container (look in the "tasks" file) or
> else there's a reference counting bug somewhere.
>
> Can you post an example to reproduce the problem?
here is an excerpt:
[root@linuz11 root]# mount -t container -ocpuset cpuset /containers/
[root@linuz11 root]# ls /containers/
cpu_exclusive memory_pressure mems tasks
cpus memory_pressure_enabled notify_on_release
mem_exclusive memory_spread_page releasable
memory_migrate memory_spread_slab release_agent
[root@linuz11 root]# mkdir /containers/1
[root@linuz11 root]# echo 0 > /containers/1/mems
[root@linuz11 root]# echo 0 > /containers/1/cpus
[root@linuz11 root]# sh
sh-2.05b# echo $$ > /containers/1/tasks
sh-2.05b# cat /containers/1/tasks
4325
4326
sh-2.05b# exit
[root@linuz11 root]# ls /containers/1/tasks
/containers/1/tasks
[root@linuz11 root]# rm -rf /containers/1
rm: cannot remove `/containers/1/memory_spread_slab': Operation not
permitted
rm: cannot remove `/containers/1/memory_spread_page': Operation not
permitted
rm: cannot remove `/containers/1/memory_pressure': Operation not
permitted
rm: cannot remove `/containers/1/memory_migrate': Operation not
permitted
rm: cannot remove `/containers/1/mem_exclusive': Operation not permitted
rm: cannot remove `/containers/1/cpu_exclusive': Operation not permitted
rm: cannot remove `/containers/1/mems': Operation not permitted
rm: cannot remove `/containers/1/cpus': Operation not permitted
rm: cannot remove `/containers/1/releasable': Operation not permitted
rm: cannot remove `/containers/1/notify_on_release': Operation not
permitted
rm: cannot remove `/containers/1/tasks': Operation not permitted
Ah, I see the second time I typed 'ls /containers/1/tasks' instead of
cat. When I then used cat, the file was empty, and I got an oops just
like Pavel reported. I bet if I solve the problem he reported, then I
solve my problem :)
thanks,
-serge
On 6/4/07, Serge E. Hallyn <[email protected]> wrote:
> root@linuz11 root]# rm -rf /containers/1
Just use "rmdir /containers/1" here.
>
> Ah, I see the second time I typed 'ls /containers/1/tasks' instead of
> cat. When I then used cat, the file was empty, and I got an oops just
> like Pavel reported. I bet if I solve the problem he reported, then I
> solve my problem :)
>
As far as I could see, Pavel's problem wasn't actually an Oops, it was
a WARN_ON() when allocating a zero length chunk of memory. There's
ongoing discussion as to whether this counts as a problem with the
allocators or the kmalloc() code, since it used to be OK to allocate a
zero-length chunk.
Paul
> Would it then make sense to just
> default to (parent_set - sibling_exclusive_set) for a new sibling's
> value?
Which could well be empty, which in turn puts one back in the position
of dealing with a newborn cpuset that is empty (of cpus or of memory),
or else it introduces a new and odd constraint on when cpusets can be
created (only when there are non-exclusive cpus and mems available.)
> An option is fine with me, but without such an option at all, cpusets
> could not be applied to namespaces...
I wasn't paying close enough attention to understand why you couldn't
do it in two steps - make the container, and then populate it with
resources.
But if indeed that's not possible, then I guess we need some sort of
option specifying whether to create kids empty, or inheriting.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401
> [root@linuz11 root]# rm -rf /containers/1
No - not 'rm -fr'.
'rmdir'
Remove the cpuset directory, not start bottom up
trying to remove the files first.
The poor 'rm -fr' command doesn't understand the
rather odd nature of cpuset file systems, which
have all files coming and going automagically in
the middle of the night, and only allow mkdir and
rmdir operations, not creat or unlink.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401
Quoting Paul Menage ([email protected]):
> On 6/4/07, Serge E. Hallyn <[email protected]> wrote:
> >root@linuz11 root]# rm -rf /containers/1
>
> Just use "rmdir /containers/1" here.
Hmm. Ok, that works... Odd, I thought rm -rf used to work in the past,
but i'm likely wrong.
thanks,
-serge
> >Ah, I see the second time I typed 'ls /containers/1/tasks' instead of
> >cat. When I then used cat, the file was empty, and I got an oops just
> >like Pavel reported. I bet if I solve the problem he reported, then I
> >solve my problem :)
> >
>
> As far as I could see, Pavel's problem wasn't actually an Oops, it was
> a WARN_ON() when allocating a zero length chunk of memory. There's
> ongoing discussion as to whether this counts as a problem with the
> allocators or the kmalloc() code, since it used to be OK to allocate a
> zero-length chunk.
>
> Paul
Serge wrote:
> Odd, I thought rm -rf used to work in the past,
> but i'm likely wrong.
I'm pretty sure it never worked.
And I've probably tested it myself, every few months,
since the birth of cpusets, when I forget and type it
again, and then stare dumbly at the screen wondering
what all the complaining is about.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401
Quoting Paul Jackson ([email protected]):
> > Would it then make sense to just
> > default to (parent_set - sibling_exclusive_set) for a new sibling's
> > value?
>
> Which could well be empty, which in turn puts one back in the position
> of dealing with a newborn cpuset that is empty (of cpus or of memory),
> or else it introduces a new and odd constraint on when cpusets can be
> created (only when there are non-exclusive cpus and mems available.)
>
> > An option is fine with me, but without such an option at all, cpusets
> > could not be applied to namespaces...
>
> I wasn't paying close enough attention to understand why you couldn't
> do it in two steps - make the container, and then populate it with
> resources.
Sorry, please clarify - are you saying that now you do understand, or
that I should explain?
> But if indeed that's not possible, then I guess we need some sort of
> option specifying whether to create kids empty, or inheriting.
Paul (uh, Menage :) should I do a patch for this or have you got it
already?
thanks,
-serge
> > I wasn't paying close enough attention to understand why you couldn't
> > do it in two steps - make the container, and then populate it with
> > resources.
>
> Sorry, please clarify - are you saying that now you do understand, or
> that I should explain?
Could you explain -- I still don't understand why you need this option.
I still don't understand why you can't do it in two steps - make the
container, then add cpu/mem separately.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401
Quoting Paul Jackson ([email protected]):
> > > I wasn't paying close enough attention to understand why you couldn't
> > > do it in two steps - make the container, and then populate it with
> > > resources.
> >
> > Sorry, please clarify - are you saying that now you do understand, or
> > that I should explain?
>
> Could you explain -- I still don't understand why you need this option.
> I still don't understand why you can't do it in two steps - make the
> container, then add cpu/mem separately.
Sure - the key is that the ns subsystem uses container_clone() to
automatically create a new container (on sys_unshare() or clone(2)
with certain flags) and move the current task into it. Let's say
we have done
mount -t container -o ns,cpuset nsproxy /containers
and we, as task 875, happen to be in the topmost container:
/containers/
Now we fork task 999 which does an unshare(CLONE_NEWNS), or we just
clone(CLONE_NEWNS). This will create
/containers/node_999
and move task 999 into that container. Except that when it tries
attach_task() it is refused by cpuset. So the container_clone() fails,
and in turn the sys_unshare() or clone() fails. A login making use
of the pam_namespace.so library would fail this way with the
ns and cpuset subsystems composed.
We could special case this by having
kernel/container.c:container_clone() check whether one of the subsystems
is cpusets and, if so, setting the defaults for mems and cpus, but
that is kind of ugly. I suppose as a cleaner alternative we could
add a container_subsys->inherit_defaults() handler, to be called at
container_clone(), and for cpusets this would set cpus and mems to
the parent values - sibling exclusive values. If that comes to nothing,
then the attach_task() is still refused, and the unshare() or clone()
fails, but this time with good reason.
thanks,
-serge
> I suppose as a cleaner alternative we could
> add a container_subsys->inherit_defaults() handler, to be called at
> container_clone(), and for cpusets this would set cpus and mems to
> the parent values - sibling exclusive values. If that comes to nothing,
> then the attach_task() is still refused, and the unshare() or clone()
> fails, but this time with good reason.
Unfortunately, I haven't spent the time I should thinking about
container cloning, namespaces and such.
I don't know, for the workloads that matter to me, when, how or
if this container cloning will be used.
I'm tempted to suggest the following.
First, I am assuming that the classic method of creating cpuset
children will still work, such as the following (which can fail
for certain combinations of exclusive cpus or mems):
cd /dev/cpuset/foobar
mkdir foochild
cp cpus foochild
cp mems foochild
echo $$ > foochild/tasks
Second, given that, how about you fail the unshare() or clone()
anytime that the instance to be cloned has any sibling cpusets
with any exclusive flags set.
The exclusive property is not really on friendly terms with cloning.
Now if the above classic code must be encoded using cloning under
the covers, then we've got problems, probably more problems than
just this.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401
Quoting Paul Jackson ([email protected]):
> > I suppose as a cleaner alternative we could
> > add a container_subsys->inherit_defaults() handler, to be called at
> > container_clone(), and for cpusets this would set cpus and mems to
> > the parent values - sibling exclusive values. If that comes to nothing,
> > then the attach_task() is still refused, and the unshare() or clone()
> > fails, but this time with good reason.
>
> Unfortunately, I haven't spent the time I should thinking about
> container cloning, namespaces and such.
>
> I don't know, for the workloads that matter to me, when, how or
> if this container cloning will be used.
>
> I'm tempted to suggest the following.
>
> First, I am assuming that the classic method of creating cpuset
> children will still work, such as the following (which can fail
> for certain combinations of exclusive cpus or mems):
> cd /dev/cpuset/foobar
> mkdir foochild
> cp cpus foochild
> cp mems foochild
> echo $$ > foochild/tasks
>
> Second, given that, how about you fail the unshare() or clone()
> anytime that the instance to be cloned has any sibling cpusets
> with any exclusive flags set.
The below patch (on top of my previous patch) does basically that. But
I wasn't able to test it, bc i wasn't able to set cpus_exclusive...
For /cpusets/set0/set1 to have cpu 1 exclusively, does /cpusets/set0
also have to have it exclusively?
If so, then clearly this approach won't work, since if any container has
exclusive cpus, then every container will have siblings with exclusive
cpus, and unshare still isn't possible on the system.
> The exclusive property is not really on friendly terms with cloning.
>
> Now if the above classic code must be encoded using cloning under
> the covers, then we've got problems, probably more problems than
> just this.
>
> --
> I won't rest till it's the best ...
> Programmer, Linux Scalability
> Paul Jackson <[email protected]> 1.925.600.0401
thanks,
-serge
>From 821de58b6ba446e50225606e907baac00130586c Mon Sep 17 00:00:00 2001
From: Serge E. Hallyn <[email protected]>
Date: Thu, 7 Jun 2007 13:53:43 -0400
Subject: [PATCH 1/1] containers: implement subsys->auto_setup
container_clone() in one step creates a new container and moves
the current task into it. Since cpusets do not automatically
fill in the allowed cpus and mems, and do not allow a task to
be attached without these filled in, composing the ns subsystem,
which uses container_clone(), and the cpuset subsystem, results
in sys_unshare() (and clone(CLONE_NEWNS)) always being denied.
To allow the two subsystems to be meaningfully composed, implement
subsystem->auto_setup, called from container_clone() after
creating the new container.
Only the cpuset_auto_setup() is currently implemented. If any
sibling containers have exclusive cpus or mems, then the cpus
and mems are not filled in for the new container, meaning that
unshare/clone(CLONE_NEWNS) will be denied. However so long as
no siblings have exclusive cpus or mems, the new container's
cpus and mems are inherited from the parent container.
Signed-off-by: Serge E. Hallyn <[email protected]>
---
Documentation/containers.txt | 7 +++++++
include/linux/container.h | 1 +
kernel/container.c | 7 +++++++
kernel/cpuset.c | 21 +++++++++++++++++++++
4 files changed, 36 insertions(+), 0 deletions(-)
diff --git a/Documentation/containers.txt b/Documentation/containers.txt
index ae159b9..28c9e10 100644
--- a/Documentation/containers.txt
+++ b/Documentation/containers.txt
@@ -514,6 +514,13 @@ include/linux/container.h for details). Note that although this
method can return an error code, the error code is currently not
always handled well.
+void auto_setup(struct container_subsys *ss, struct container *cont)
+
+Called at container_clone() to do any paramater initialization
+which might be required before a task could attach. For example
+in cpusets, no task may attach before 'cpus' and 'mems' are
+set up.
+
void bind(struct container_subsys *ss, struct container *root)
LL=callback_mutex
diff --git a/include/linux/container.h b/include/linux/container.h
index 37c0bdf..d809b41 100644
--- a/include/linux/container.h
+++ b/include/linux/container.h
@@ -213,6 +213,7 @@ struct container_subsys {
void (*exit)(struct container_subsys *ss, struct task_struct *task);
int (*populate)(struct container_subsys *ss,
struct container *cont);
+ void (*auto_setup)(struct container_subsys *ss, struct container *cont);
void (*bind)(struct container_subsys *ss, struct container *root);
int subsys_id;
int active;
diff --git a/kernel/container.c b/kernel/container.c
index 988cd8b..e0793f4 100644
--- a/kernel/container.c
+++ b/kernel/container.c
@@ -2316,6 +2316,7 @@ int container_clone(struct task_struct *tsk, struct container_subsys *subsys)
struct inode *inode;
struct css_group *cg;
struct containerfs_root *root;
+ struct container_subsys *ss;
/* We shouldn't be called by an unregistered subsystem */
BUG_ON(!subsys->active);
@@ -2397,6 +2398,12 @@ int container_clone(struct task_struct *tsk, struct container_subsys *subsys)
goto again;
}
+ /* do any required auto-setup */
+ for_each_subsys(root, ss) {
+ if (ss->auto_setup)
+ ss->auto_setup(ss, child);
+ }
+
/* All seems fine. Finish by moving the task into the new container */
ret = attach_task(child, tsk);
mutex_unlock(&container_mutex);
diff --git a/kernel/cpuset.c b/kernel/cpuset.c
index 0f9ce7d..ff01aaa 100644
--- a/kernel/cpuset.c
+++ b/kernel/cpuset.c
@@ -1189,6 +1189,26 @@ int cpuset_populate(struct container_subsys *ss, struct container *cont)
return 0;
}
+void cpuset_auto_setup(struct container_subsys *ss,
+ struct container *container)
+{
+ struct container *parent, *child;
+ struct cpuset *cs, *parent_cs;
+
+ parent = container->parent;
+ list_for_each_entry(child, &parent->children, sibling) {
+ cs = container_cs(child);
+ if (is_mem_exclusive(cs) || is_cpu_exclusive(cs))
+ return;
+ }
+ cs = container_cs(container);
+ parent_cs = container_cs(parent);
+
+ cs->mems_allowed = parent_cs->mems_allowed;
+ cs->cpus_allowed = parent_cs->cpus_allowed;
+ return;
+}
+
/*
* cpuset_create - create a cpuset
* parent: cpuset that will be parent of the new cpuset.
@@ -1249,6 +1269,7 @@ struct container_subsys cpuset_subsys = {
.can_attach = cpuset_can_attach,
.attach = cpuset_attach,
.populate = cpuset_populate,
+ .auto_setup = cpuset_auto_setup,
.subsys_id = cpuset_subsys_id,
.early_init = 1,
};
--
1.5.1.1.GIT
> For /cpusets/set0/set1 to have cpu 1 exclusively, does /cpusets/set0
> also have to have it exclusively?
Yes.
> If so, then clearly this approach won't work, since if any container has
> exclusive cpus, then every container will have siblings with exclusive
> cpus, and unshare still isn't possible on the system.
Well, if I'm following you, not exactly.
If we have some exclusive flags set, then every top level container
will have exclusive siblings, but further down the hierarchy, some
subtree might be entirely free of any exclusive settings. Then nodes
below the top of that subtree would not have exclusive set, and would
not have any exclusive siblings.
But, overall, yeah, exclusive is no friend of container cloning.
I just wish I had been thinking harder about how container cloning
will impact my life, and the lives of the customers in my cpuset
intensive corner of the world.
There are certainly a whole bunch of people who will never have any
need for exclusive cpusets.
Perhaps (speculating wildly from great ignorance) there are a whole
bunch of people who will never have need for container cloning.
And perhaps, hoping to get lucky here, the set of people who need both
at the same time on the same system is sufficiently close to empty
that we can just tell them tough toenails - you cannot do both at once.
How wide spread will be the use of container cloning, if it proceeds
as envisioned?
The set of people using exclusive cpusets is roughly some subset of
those running multiple, cpuset isolated, non-cooperating jobs on big
iron, usually with the aid of a batch scheduler. Well, that's what
I am aware of anyway. If there are any other friends of exclusive
cpusets lurking here, you might want to speak up, before I sell your
interests down the river.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401
Quoting Paul Jackson ([email protected]):
> > For /cpusets/set0/set1 to have cpu 1 exclusively, does /cpusets/set0
> > also have to have it exclusively?
>
> Yes.
>
> > If so, then clearly this approach won't work, since if any container has
> > exclusive cpus, then every container will have siblings with exclusive
> > cpus, and unshare still isn't possible on the system.
>
> Well, if I'm following you, not exactly.
>
> If we have some exclusive flags set, then every top level container
> will have exclusive siblings, but further down the hierarchy, some
> subtree might be entirely free of any exclusive settings. Then nodes
> below the top of that subtree would not have exclusive set, and would
> not have any exclusive siblings.
>
> But, overall, yeah, exclusive is no friend of container cloning.
>
> I just wish I had been thinking harder about how container cloning
> will impact my life, and the lives of the customers in my cpuset
> intensive corner of the world.
>
> There are certainly a whole bunch of people who will never have any
> need for exclusive cpusets.
>
> Perhaps (speculating wildly from great ignorance) there are a whole
> bunch of people who will never have need for container cloning.
>
> And perhaps, hoping to get lucky here, the set of people who need both
> at the same time on the same system is sufficiently close to empty
> that we can just tell them tough toenails - you cannot do both at once.
>
> How wide spread will be the use of container cloning, if it proceeds
> as envisioned?
It's not just container cloning, but all namespace unsharing. So uses
include (1) providing 'polyinstantiated directory' functionality, i.e.
private per-user /tmp's or per-security-level /tmp and /home's. (2) any
virtual server usage (3) hpc checkpoint/restart users.
> The set of people using exclusive cpusets is roughly some subset of
> those running multiple, cpuset isolated, non-cooperating jobs on big
> iron, usually with the aid of a batch scheduler.
Unfortunately I would imagine these users to be very intereseted in
providing checkpoint/restart/migrate functionality.
> Well, that's what
> I am aware of anyway. If there are any other friends of exclusive
> cpusets lurking here, you might want to speak up, before I sell your
> interests down the river.
>
> --
> I won't rest till it's the best ...
> Programmer, Linux Scalability
> Paul Jackson <[email protected]> 1.925.600.0401
Can you explain to me, though, why it should be that if /cpusets/set0
has access to cpus 0-8, and /cpusets/set0/set1 has exclusive access to
cpus 0-2, and /cpusets/set0/set2 has exclusive access to cpus 3-4,
why i a process in /cpusets/set0 creates /cpusets/set0/set3 through
container_clone, it would be unsafe to have it automatically get cpus 5-8?
Surely if the admin wants to give cpus 5-6 exclusively to /cpusets/set0/set4
later, those cpus can just be taken away from set3?
thanks,
-serge
> > The set of people using exclusive cpusets is roughly some subset of
> > those running multiple, cpuset isolated, non-cooperating jobs on big
> > iron, usually with the aid of a batch scheduler.
>
> Unfortunately I would imagine these users to be very intereseted in
> providing checkpoint/restart/migrate functionality.
Yup - such customers are very interested in checkpoint, restart, and
migrate functionality.
> Surely if the admin wants to give cpus 5-6 exclusively to /cpusets/set0/set4
> later, those cpus can just be taken away from set3?
Yeah - that works, so far as I know (which isn't all that far ..')
But both:
1) that (using whatever cpus are still available) and
2) my other idea, of not allowing any cloning of cpusets with
exclusive siblings at all,
looked a little ugly to me.
For example, such customers as above would not appreciate having their
checkpoint/restart/migrate fail in any case where there weren't spare
non-exclusive cpus, which for users of the exclusive flag, is often the
more common case.
My rule of thumb when doing ugly stuff is to constrain it as best
I can -- minimize what it allows. This led me to prefer (2) above
over (1) above.
Perhaps there's a better way to think of this ... When we clone
like this for checkpoint/restart/migrate functionality, perhaps
we are not really starting up a new, separate, competing job that
should have its resources isolated and separated from the original.
Perhaps instead we are firing up a convenient alter-ego of the
original job, which will be co-operatively using the same resources
by default. If that's the normal case, then it seems wrong to
force the clone onto disjoint CPUs, or fail for lack thereof.
So perhaps we should refine the meaning of 'exclusive', from:
- no overlapping siblings
to:
- no overlapping siblings other than clones of ones self.
Then default to cloning right on the same CPU resources as the
original, possibly with both original and clone marked exclusive.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401
Quoting Paul Jackson ([email protected]):
> > > The set of people using exclusive cpusets is roughly some subset of
> > > those running multiple, cpuset isolated, non-cooperating jobs on big
> > > iron, usually with the aid of a batch scheduler.
> >
> > Unfortunately I would imagine these users to be very intereseted in
> > providing checkpoint/restart/migrate functionality.
>
> Yup - such customers are very interested in checkpoint, restart, and
> migrate functionality.
>
> > Surely if the admin wants to give cpus 5-6 exclusively to /cpusets/set0/set4
> > later, those cpus can just be taken away from set3?
>
> Yeah - that works, so far as I know (which isn't all that far ..')
>
> But both:
> 1) that (using whatever cpus are still available) and
> 2) my other idea, of not allowing any cloning of cpusets with
> exclusive siblings at all,
>
> looked a little ugly to me.
>
> For example, such customers as above would not appreciate having their
> checkpoint/restart/migrate fail in any case where there weren't spare
> non-exclusive cpus, which for users of the exclusive flag, is often the
> more common case.
>
> My rule of thumb when doing ugly stuff is to constrain it as best
> I can -- minimize what it allows. This led me to prefer (2) above
> over (1) above.
>
> Perhaps there's a better way to think of this ... When we clone
> like this for checkpoint/restart/migrate functionality, perhaps
> we are not really starting up a new, separate, competing job that
> should have its resources isolated and separated from the original.
Depends on whether the cpus are allocated to a customer or to a job.
For the most part I would expect any job to be restart either on a
different machine, or at a different time, but of course it doesn't have
to be that way.
> Perhaps instead we are firing up a convenient alter-ego of the
> original job, which will be co-operatively using the same resources
> by default. If that's the normal case, then it seems wrong to
> force the clone onto disjoint CPUs, or fail for lack thereof.
>
> So perhaps we should refine the meaning of 'exclusive', from:
> - no overlapping siblings
> to:
> - no overlapping siblings other than clones of ones self.
I'm not sure that clones of self will happen often enough to make a
special case for them :)
Anyway the patch I sent is simple enough, and if users end up demanding
the ability to better deal with exclusive cpusets, the patch will be
simple enough to extend by changing cpuset_auto_setup(), so let's
stick with that patch since it's your preference (IIUC).
> Then default to cloning right on the same CPU resources as the
> original, possibly with both original and clone marked exclusive.
Thanks,
-serge
On 6/8/07, Serge E. Hallyn <[email protected]> wrote:
>
> Anyway the patch I sent is simple enough, and if users end up demanding
> the ability to better deal with exclusive cpusets, the patch will be
> simple enough to extend by changing cpuset_auto_setup(), so let's
> stick with that patch since it's your preference (IIUC).
>
Sounds good to me, although I think my preference would be to extend
the "create()" subsystem callback with a "struct task_struct
*clone_task" parameter that indicates that clone_task is cloning its
own container; a subsystem like cpusets that needs to do additional
setup at that point could inherit settings either from the parent or
from clone_task's container (or something else) as desired. (It could
also do permission checking based on properties of clone_task, etc).
Paul
Quoting Paul Menage ([email protected]):
> On 6/8/07, Serge E. Hallyn <[email protected]> wrote:
> >
> >Anyway the patch I sent is simple enough, and if users end up demanding
> >the ability to better deal with exclusive cpusets, the patch will be
> >simple enough to extend by changing cpuset_auto_setup(), so let's
> >stick with that patch since it's your preference (IIUC).
> >
>
> Sounds good to me, although I think my preference would be to extend
> the "create()" subsystem callback with a "struct task_struct
> *clone_task" parameter that indicates that clone_task is cloning its
> own container; a subsystem like cpusets that needs to do additional
> setup at that point could inherit settings either from the parent or
> from clone_task's container (or something else) as desired. (It could
> also do permission checking based on properties of clone_task, etc).
The problem is container_clone() doesn't call ->create explicitly, it
does vfs_mkdir. So we have no real way of passing in clone_task.
-serge
On 6/8/07, Serge E. Hallyn <[email protected]> wrote:
>
> The problem is container_clone() doesn't call ->create explicitly, it
> does vfs_mkdir. So we have no real way of passing in clone_task.
>
Good point.
Looking at vfs_mkdir(), it's pretty simple, and really the only bits
that apply to container_clone() are the call to ->mkdir() and possibly
the call to fsnotify_mkdir(). (I think that's maybe how you did it
originally?)
Maybe it would make sense to just call container_create() at that
point directly, which would allow us more parameters.
Paul
> Anyway the patch I sent is simple enough, and if users end up demanding
> the ability to better deal with exclusive cpusets, the patch will be
> simple enough to extend by changing cpuset_auto_setup(), so let's
> stick with that patch since it's your preference (IIUC).
Yeah - probably so.
When someone gets serious about things like checkpoint, restart, and
migrate functionality, based on this container cloning, working with
cpusets, they will probably have to revisit this interaction with
exclusive cpusets.
Perhaps a comment could be put in the code, saying something like the
above, so whomever does this will realize they are traveling in
unchartered territory.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401
Quoting Paul Menage ([email protected]):
> On 6/8/07, Serge E. Hallyn <[email protected]> wrote:
> >
> >The problem is container_clone() doesn't call ->create explicitly, it
> >does vfs_mkdir. So we have no real way of passing in clone_task.
> >
>
> Good point.
>
> Looking at vfs_mkdir(), it's pretty simple, and really the only bits
> that apply to container_clone() are the call to ->mkdir() and possibly
> the call to fsnotify_mkdir(). (I think that's maybe how you did it
> originally?)
Yes it was.
> Maybe it would make sense to just call container_create() at that
> point directly, which would allow us more parameters.
I do fear that that could become a maintenance nightmare. For instance
right now there's the call to fsnotify_mkdir(). Other such hooks might
be placed at vfs_mkdir, which we'd then likely want to have placed in
our container_mkdir() and container_clone() fns. And of course
may_create() is static inline in fs/namei.c. It's trivial, but still if
it changes we'd want to change the version in kernel/container.c as
well.
What would be the main advantage of doing it this way? Do you consider
the extra subys->auto_setup() hook to be avoidable bloat?
thanks,
-serge
On 6/8/07, Serge E. Hallyn <[email protected]> wrote:
>
> I do fear that that could become a maintenance nightmare. For instance
> right now there's the call to fsnotify_mkdir(). Other such hooks might
> be placed at vfs_mkdir, which we'd then likely want to have placed in
> our container_mkdir() and container_clone() fns. And of course
> may_create() is static inline in fs/namei.c. It's trivial, but still if
> it changes we'd want to change the version in kernel/container.c as
> well.
Do we need to actually need to respect may_create() in
container_clone()? I guess it would provide a way for root to control
which processes could unshare namespaces.
>
> What would be the main advantage of doing it this way? Do you consider
> the extra subys->auto_setup() hook to be avoidable bloat?
>
I was thinking that it would be nice to be able to atomically set up
the resources in the new container at the point when it's created
rather than later. But I guess this way can work too. Can we call it
something like "clone()" rather than "auto_setup()"?
Paul
Quoting Paul Menage ([email protected]):
> On 6/8/07, Serge E. Hallyn <[email protected]> wrote:
> >
> >I do fear that that could become a maintenance nightmare. For instance
> >right now there's the call to fsnotify_mkdir(). Other such hooks might
> >be placed at vfs_mkdir, which we'd then likely want to have placed in
> >our container_mkdir() and container_clone() fns. And of course
> >may_create() is static inline in fs/namei.c. It's trivial, but still if
> >it changes we'd want to change the version in kernel/container.c as
> >well.
>
> Do we need to actually need to respect may_create() in
> container_clone()? I guess it would provide a way for root to control
> which processes could unshare namespaces.
>
> >
> >What would be the main advantage of doing it this way? Do you consider
> >the extra subys->auto_setup() hook to be avoidable bloat?
> >
>
> I was thinking that it would be nice to be able to atomically set up
> the resources in the new container at the point when it's created
> rather than later. But I guess this way can work too. Can we call it
> something like "clone()" rather than "auto_setup()"?
>
> Paul
clone() implies it does the actual cloning, so how about post_clone()
as in the patch below?
I'm still not saying I'm entirely opposed to moving the vfs_mkdir logic
straight into container_clone() - it's more that I would expect other
people to object when they saw that. So if you decide you don't like
the end result with this patch, let me know and I'll give that a shot.
Paul (Jackson), is this comment added in cpusets close enough to what
you were asking for?
thanks,
-serge
>From c2f1a39b231f06cb524c6e95d74de6ddee286f25 Mon Sep 17 00:00:00 2001
From: Serge E. Hallyn <[email protected]>
Date: Fri, 8 Jun 2007 15:36:59 -0400
Subject: [PATCH 4/4] containers: minor clone cleanup
rename auto_setup() to post_clone(), and comment the cpusets version.
Signed-off-by: Serge E. Hallyn <[email protected]>
---
Documentation/containers.txt | 10 +++++-----
include/linux/container.h | 2 +-
kernel/container.c | 4 ++--
kernel/cpuset.c | 20 ++++++++++++++++++--
4 files changed, 26 insertions(+), 10 deletions(-)
diff --git a/Documentation/containers.txt b/Documentation/containers.txt
index 28c9e10..9fdb808 100644
--- a/Documentation/containers.txt
+++ b/Documentation/containers.txt
@@ -514,12 +514,12 @@ include/linux/container.h for details). Note that although this
method can return an error code, the error code is currently not
always handled well.
-void auto_setup(struct container_subsys *ss, struct container *cont)
+void post_clone(struct container_subsys *ss, struct container *cont)
-Called at container_clone() to do any paramater initialization
-which might be required before a task could attach. For example
-in cpusets, no task may attach before 'cpus' and 'mems' are
-set up.
+Called at the end of container_clone() to do any paramater
+initialization which might be required before a task could attach. For
+example in cpusets, no task may attach before 'cpus' and 'mems' are set
+up.
void bind(struct container_subsys *ss, struct container *root)
LL=callback_mutex
diff --git a/include/linux/container.h b/include/linux/container.h
index d809b41..1a83913 100644
--- a/include/linux/container.h
+++ b/include/linux/container.h
@@ -213,7 +213,7 @@ struct container_subsys {
void (*exit)(struct container_subsys *ss, struct task_struct *task);
int (*populate)(struct container_subsys *ss,
struct container *cont);
- void (*auto_setup)(struct container_subsys *ss, struct container *cont);
+ void (*post_clone)(struct container_subsys *ss, struct container *cont);
void (*bind)(struct container_subsys *ss, struct container *root);
int subsys_id;
int active;
diff --git a/kernel/container.c b/kernel/container.c
index e0793f4..11e326a 100644
--- a/kernel/container.c
+++ b/kernel/container.c
@@ -2400,8 +2400,8 @@ int container_clone(struct task_struct *tsk, struct container_subsys *subsys)
/* do any required auto-setup */
for_each_subsys(root, ss) {
- if (ss->auto_setup)
- ss->auto_setup(ss, child);
+ if (ss->post_clone)
+ ss->post_clone(ss, child);
}
/* All seems fine. Finish by moving the task into the new container */
diff --git a/kernel/cpuset.c b/kernel/cpuset.c
index ff01aaa..ecefb1d 100644
--- a/kernel/cpuset.c
+++ b/kernel/cpuset.c
@@ -1189,7 +1189,23 @@ int cpuset_populate(struct container_subsys *ss, struct container *cont)
return 0;
}
-void cpuset_auto_setup(struct container_subsys *ss,
+/*
+ * post_clone() is called at the end of container_clone().
+ * 'container' was just created automatically as a result of
+ * a container_clone(), and the current task is about to
+ * be moved into 'container'.
+ *
+ * Currently we refuse to set up the container - thereby
+ * refusing the task to be entered, and as a result refusing
+ * the sys_unshare() or clone() which initiated it - if any
+ * sibling cpusets have exclusive cpus or mem.
+ *
+ * If this becomes a problem for some users who wish to
+ * allow that scenario, then cpuset_post_clone() could be
+ * changed to grant parent->cpus_allowed-sibling_cpus_exclusive
+ * (and likewise for mems) to the new container.
+ */
+void cpuset_post_clone(struct container_subsys *ss,
struct container *container)
{
struct container *parent, *child;
@@ -1269,7 +1285,7 @@ struct container_subsys cpuset_subsys = {
.can_attach = cpuset_can_attach,
.attach = cpuset_attach,
.populate = cpuset_populate,
- .auto_setup = cpuset_auto_setup,
+ .post_clone = cpuset_post_clone,
.subsys_id = cpuset_subsys_id,
.early_init = 1,
};
--
1.5.1.1.GIT
Serge wrote:
> Paul (Jackson), is this comment added in cpusets close enough to what
> you were asking for?
This comment?
+ * Currently we refuse to set up the container - thereby
+ * refusing the task to be entered, and as a result refusing
+ * the sys_unshare() or clone() which initiated it - if any
+ * sibling cpusets have exclusive cpus or mem.
+ *
+ * If this becomes a problem for some users who wish to
+ * allow that scenario, then cpuset_post_clone() could be
+ * changed to grant parent->cpus_allowed-sibling_cpus_exclusive
+ * (and likewise for mems) to the new container.
Nice - thanks.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.925.600.0401
On 5/30/07, William Lee Irwin III <[email protected]> wrote:
> On Wed, May 30, 2007 at 12:14:55AM -0700, Andrew Morton wrote:
> > So how do we do this?
> > Is there any sneaky way in which we can modify the kernel so that this new
> > code gets exercised more? Obviously, tossing init into some default
> > system-wide container would be a start. But I wonder if we can be
> > sneakier - for example, create a new container on each setuid(), toss the
> > task into that. Or something along those lines?
>
> How about a container for each thread group, pgrp, session, and user?
>
I've been thinking about this, and figured that it could be quite
useful to be able to mount a container tree that groups tasks by
userid or thread group - for doing per-user resource controls, for
example, without having to write a controller that specifically
handles the per-user case.
One option would be to add a mount option, something like
mount -t container -ogroupkey=<X>
where X could be one of: uid, gid, pgrp, sid, tgid
And put hooks in the various places where these ids could change, in
order to move tasks between contaners as appropriate. But after some
thought it seems to me that this is putting complexity in the kernel
that probably doesn't belong there, and additionally is probably not
sufficently flexible for some real-life situations. (E.g. the user
wants all users in the "student" group to be lumped into the same
container, but each user in the "professor" group gets their own
container).
So maybe this would be better handled in userspace? Have a daemon
listing on a process connector socket, and move processes between
containers based on notifications from the connector and user-defined
rules.
We'd probably also want to add some new connector events, such as
PROC_EVENT_PGRP, and PROC_EVENT_SID
A simple daemon that handles the case where we're classifying based on
a single key with no complex rules shouldn't be too hard to write.
It also sounds rather like the classification engine that
ResourceGroups had originally in the kernel and moved to userspace, so
I'll take a look at that and see if it's adaptable for this.
Paul
On Thu, Jun 28, 2007 at 05:27:25PM -0400, Paul Menage wrote:
> So maybe this would be better handled in userspace? Have a daemon
> listing on a process connector socket, and move processes between
> containers based on notifications from the connector and user-defined
> rules.
>
> We'd probably also want to add some new connector events, such as
> PROC_EVENT_PGRP, and PROC_EVENT_SID
Yep, this is what I did to test fair-user scheduling on top of my
patches.
Dhaval has a working program, which listens for UID change events and
moves the task to approp. container. I will review that and have it
posted soon.
--
Regards,
vatsa