Hello Tejun,
I was trying to do some simple testing ot the CPU controller
that is merged into 4.15, and ran immediately into some confusion.
In the root cgroup on a freshly booted 4.150-rc1, I try the following:
# pwd
/sys/fs/cgroup/unified
# echo '+cpu' > cgroup.subtree_control
sh: echo: write error: Invalid argument
What am I missing> I presume I'm missing something obvious, although
nothing jumped out at me as I read the cgroups-v2.txt file.
Cheers,
Michael
--
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
Hello, Michael.
On Mon, Dec 04, 2017 at 10:35:13PM +0100, Michael Kerrisk (man-pages) wrote:
> I was trying to do some simple testing ot the CPU controller
> that is merged into 4.15, and ran immediately into some confusion.
> In the root cgroup on a freshly booted 4.150-rc1, I try the following:
>
> # pwd
> /sys/fs/cgroup/unified
> # echo '+cpu' > cgroup.subtree_control
> sh: echo: write error: Invalid argument
>
> What am I missing> I presume I'm missing something obvious, although
> nothing jumped out at me as I read the cgroups-v2.txt file.
Checking whether I messed up something really basic... hmmm doesn't
seem that way. What do /sys/fs/cgroup/unified/cgroup.controllers and
/proc/cgroups say?
Thanks.
--
tejun
[dropping Lennart into CC]
Hello Tejun,
On 12/04/2017 10:47 PM, Tejun Heo wrote:
> Hello, Michael.
>
> On Mon, Dec 04, 2017 at 10:35:13PM +0100, Michael Kerrisk (man-pages) wrote:
>> I was trying to do some simple testing ot the CPU controller
>> that is merged into 4.15, and ran immediately into some confusion.
>> In the root cgroup on a freshly booted 4.150-rc1, I try the following:
>>
>> # pwd
>> /sys/fs/cgroup/unified
>> # echo '+cpu' > cgroup.subtree_control
>> sh: echo: write error: Invalid argument
>>
>> What am I missing> I presume I'm missing something obvious, although
>> nothing jumped out at me as I read the cgroups-v2.txt file.
>
> Checking whether I messed up something really basic... hmmm doesn't
> seem that way. What do /sys/fs/cgroup/unified/cgroup.controllers and
> /proc/cgroups say?
Oh -- they're all sensible:
In the root cgroup:
# cat cgroup.controllers
cpu io memory pids
$ cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 0 142 1
cpu 0 142 1
cpuacct 0 142 1
blkio 0 142 1
memory 0 142 1
devices 0 142 1
freezer 0 142 1
net_cls 0 142 1
perf_event 0 142 1
net_prio 0 142 1
hugetlb 0 142 1
pids 0 142 1
But, I through some trial and error and printk() I worked out
a) If I first move all tasks to the root cgroup, then I can
write '+cpu' to the cgroup.subtree_control file in the root
cgroup.
b) The reason for my initial problems was this test in
the kernel in cpu_cgroup_can_attach():
#ifdef CONFIG_RT_GROUP_SCHED
if (!sched_rt_can_attach(css_tg(css), task))
return -EINVAL;
#else
/* We don't support RT-tasks being in separate groups */
if (task->sched_class != &fair_sched_class)
return -EINVAL;
#endif
I don't have CONFIG_RT_GROUP_SCHED, and the second 'if' was yielding
false because of some SCHED_RR processes that are in some of the nonroot
cgroups created by systemd, namely:
# ps ax -L -o 'pid tid cls rtprio comm'|grep RR
685 723 RR 99 rtkit-daemon
972 979 RR 5 alsa-sink-ALC26
972 982 RR 5 alsa-source-ALC
1594 1597 RR 5 alsa-sink-ALC26
1594 1600 RR 5 alsa-source-ALC
So, one solution is to move those processes to the root cgroup,
and then it's possible to write '+pids' to cgroup.subtree_control.
Is enabling CONFIG_RT_GROUP_SCHED also a solution? (I have
not had a chance to test that yet.)
Anyway, it seems like this should be documented somewhere in the
kernel Documentation files, since it may be that others will run
into this as well. I'm not quite sure what should be added to the
documentation. Do you have some idea?
Thanks,
Michael
--
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
Hello, Michael.
On Tue, Dec 05, 2017 at 08:45:19AM +0100, Michael Kerrisk (man-pages) wrote:
> b) The reason for my initial problems was this test in
> the kernel in cpu_cgroup_can_attach():
>
> #ifdef CONFIG_RT_GROUP_SCHED
> if (!sched_rt_can_attach(css_tg(css), task))
> return -EINVAL;
> #else
> /* We don't support RT-tasks being in separate groups */
> if (task->sched_class != &fair_sched_class)
> return -EINVAL;
> #endif
>
> I don't have CONFIG_RT_GROUP_SCHED, and the second 'if' was yielding
> false because of some SCHED_RR processes that are in some of the nonroot
> cgroups created by systemd, namely:
I see. Thanks for tracking it down. Yeah, the RT side of things
isn't too well thought out yet.
> # ps ax -L -o 'pid tid cls rtprio comm'|grep RR
> 685 723 RR 99 rtkit-daemon
> 972 979 RR 5 alsa-sink-ALC26
> 972 982 RR 5 alsa-source-ALC
> 1594 1597 RR 5 alsa-sink-ALC26
> 1594 1600 RR 5 alsa-source-ALC
>
> So, one solution is to move those processes to the root cgroup,
> and then it's possible to write '+pids' to cgroup.subtree_control.
You mean '+cpu", right?
> Is enabling CONFIG_RT_GROUP_SCHED also a solution? (I have
> not had a chance to test that yet.)
We aren't yet sure about how we should handle RT and haven't enabled
RT part on cgroup2 yet. You can test the same scenario in cgroup1 tho
and would have to configure RT shares all along the hierarchy to the
leaf cgroup.
> Anyway, it seems like this should be documented somewhere in the
> kernel Documentation files, since it may be that others will run
> into this as well. I'm not quite sure what should be added to the
> documentation. Do you have some idea?
I think the only thing we can say right now is that RT processes
should be in the root cgroup. :(
Thanks a lot.
--
tejun
Hello Tejun,
On 5 December 2017 at 17:00, Tejun Heo <[email protected]> wrote:
> Hello, Michael.
>
> On Tue, Dec 05, 2017 at 08:45:19AM +0100, Michael Kerrisk (man-pages) wrote:
>> b) The reason for my initial problems was this test in
>> the kernel in cpu_cgroup_can_attach():
>>
>> #ifdef CONFIG_RT_GROUP_SCHED
>> if (!sched_rt_can_attach(css_tg(css), task))
>> return -EINVAL;
>> #else
>> /* We don't support RT-tasks being in separate groups */
>> if (task->sched_class != &fair_sched_class)
>> return -EINVAL;
>> #endif
>>
>> I don't have CONFIG_RT_GROUP_SCHED, and the second 'if' was yielding
>> false because of some SCHED_RR processes that are in some of the nonroot
>> cgroups created by systemd, namely:
>
> I see. Thanks for tracking it down. Yeah, the RT side of things
> isn't too well thought out yet.
>
>> # ps ax -L -o 'pid tid cls rtprio comm'|grep RR
>> 685 723 RR 99 rtkit-daemon
>> 972 979 RR 5 alsa-sink-ALC26
>> 972 982 RR 5 alsa-source-ALC
>> 1594 1597 RR 5 alsa-sink-ALC26
>> 1594 1600 RR 5 alsa-source-ALC
>>
>> So, one solution is to move those processes to the root cgroup,
>> and then it's possible to write '+pids' to cgroup.subtree_control.
>
> You mean '+cpu", right?
D'oh! Yes. I was also doing some tests with the 'pids' controller, and
mistyped. (I see I even manage to mistitle this mail thread :-(.)
>> Is enabling CONFIG_RT_GROUP_SCHED also a solution? (I have
>> not had a chance to test that yet.)
>
> We aren't yet sure about how we should handle RT and haven't enabled
> RT part on cgroup2 yet. You can test the same scenario in cgroup1 tho
> and would have to configure RT shares all along the hierarchy to the
> leaf cgroup.
Okay. Thanks for the confirmation that this is at least surprising behavior.
>> Anyway, it seems like this should be documented somewhere in the
>> kernel Documentation files, since it may be that others will run
>> into this as well. I'm not quite sure what should be added to the
>> documentation. Do you have some idea?
>
> I think the only thing we can say right now is that RT processes
> should be in the root cgroup. :(
Okay. (Will you add that to the v2 doc file?)
Cheers,
Michael
--
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
From: Tejun Heo <[email protected]>
Date: Tue, 5 Dec 2017 09:10:17 -0800
We haven't yet figured out what to do with RT threads on cgroup2.
Document the limitation.
Signed-off-by: Tejun Heo <[email protected]>
Reported-by: "Michael Kerrisk (man-pages)" <[email protected]>
---
Documentation/cgroup-v2.txt | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/Documentation/cgroup-v2.txt b/Documentation/cgroup-v2.txt
index 779211f..55a28f4 100644
--- a/Documentation/cgroup-v2.txt
+++ b/Documentation/cgroup-v2.txt
@@ -898,6 +898,10 @@ controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.
+WARNING: cgroup2 doesn't yet support control of realtime processes and
+the cpu controller can only be enabled when all RT processes are in
+the root cgroup.
+
CPU Interface Files
~~~~~~~~~~~~~~~~~~~
--
2.9.5
Hello Tejun,
On 5 December 2017 at 18:13, Tejun Heo <[email protected]> wrote:
> From: Tejun Heo <[email protected]>
> Date: Tue, 5 Dec 2017 09:10:17 -0800
>
> We haven't yet figured out what to do with RT threads on cgroup2.
> Document the limitation.
>
> Signed-off-by: Tejun Heo <[email protected]>
> Reported-by: "Michael Kerrisk (man-pages)" <[email protected]>
> ---
> Documentation/cgroup-v2.txt | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/Documentation/cgroup-v2.txt b/Documentation/cgroup-v2.txt
> index 779211f..55a28f4 100644
> --- a/Documentation/cgroup-v2.txt
> +++ b/Documentation/cgroup-v2.txt
> @@ -898,6 +898,10 @@ controller implements weight and absolute bandwidth limit models for
> normal scheduling policy and absolute bandwidth allocation model for
> realtime scheduling policy.
>
> +WARNING: cgroup2 doesn't yet support control of realtime processes and
> +the cpu controller can only be enabled when all RT processes are in
> +the root cgroup.
> +
The documentation file already carries this useful text:
[[
During transition to v2, system management software might still
automount the v1 cgroup filesystem and so hijack all controllers
during boot, before manual intervention is possible. To make testing
and experimenting easier, the kernel parameter cgroup_no_v1= allows
disabling controllers in v1 and make them always available in v2.
]]
In that spirit, it may be worth alerting the reader that "system
management software" might already have put some RT processes in
nonroot cgroups. Maybe something like this:
[[
WARNING: cgroup2 doesn't yet support control of realtime processes and
the cpu controller can only be enabled when all RT processes are in
the root cgroup. Be aware that that system management software may
already have placed RT processes into nonroot cgroups during the system
boot process, and these processes may need to be moved to the root
cgroup before the cpu controller can be enabled.
]]
Thanks,
Michael
--
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
>From f0a821260f24ea9378ed0e0c149e3bbc1a1bd008 Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Tue, 5 Dec 2017 09:10:17 -0800
We haven't yet figured out what to do with RT threads on cgroup2.
Document the limitation.
v2: Included the warning about system management software behavior as
suggested by Michael.
Signed-off-by: Tejun Heo <[email protected]>
Reported-by: "Michael Kerrisk (man-pages)" <[email protected]>
---
Documentation/cgroup-v2.txt | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/Documentation/cgroup-v2.txt b/Documentation/cgroup-v2.txt
index 779211f..c5e3c21 100644
--- a/Documentation/cgroup-v2.txt
+++ b/Documentation/cgroup-v2.txt
@@ -898,6 +898,13 @@ controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.
+WARNING: cgroup2 doesn't yet support control of realtime processes and
+the cpu controller can only be enabled when all RT processes are in
+the root cgroup. Be aware that that system management software may
+already have placed RT processes into nonroot cgroups during the
+system boot process, and these processes may need to be moved to the
+root cgroup before the cpu controller can be enabled.
+
CPU Interface Files
~~~~~~~~~~~~~~~~~~~
--
2.9.5
On 5 December 2017 at 19:29, Tejun Heo <[email protected]> wrote:
> From f0a821260f24ea9378ed0e0c149e3bbc1a1bd008 Mon Sep 17 00:00:00 2001
> From: Tejun Heo <[email protected]>
> Date: Tue, 5 Dec 2017 09:10:17 -0800
>
> We haven't yet figured out what to do with RT threads on cgroup2.
> Document the limitation.
>
> v2: Included the warning about system management software behavior as
> suggested by Michael.
>
> Signed-off-by: Tejun Heo <[email protected]>
> Reported-by: "Michael Kerrisk (man-pages)" <[email protected]>
> ---
> Documentation/cgroup-v2.txt | 7 +++++++
> 1 file changed, 7 insertions(+)
>
> diff --git a/Documentation/cgroup-v2.txt b/Documentation/cgroup-v2.txt
> index 779211f..c5e3c21 100644
> --- a/Documentation/cgroup-v2.txt
> +++ b/Documentation/cgroup-v2.txt
> @@ -898,6 +898,13 @@ controller implements weight and absolute bandwidth limit models for
> normal scheduling policy and absolute bandwidth allocation model for
> realtime scheduling policy.
>
> +WARNING: cgroup2 doesn't yet support control of realtime processes and
> +the cpu controller can only be enabled when all RT processes are in
> +the root cgroup. Be aware that that system management software may
Sorry, there was a fault in my text: s/that that/that/
Thanks,
Michael
--
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
On Tue, Dec 05, 2017 at 07:53:51PM +0100, Michael Kerrisk (man-pages) wrote:
> > +WARNING: cgroup2 doesn't yet support control of realtime processes and
> > +the cpu controller can only be enabled when all RT processes are in
> > +the root cgroup. Be aware that that system management software may
>
> Sorry, there was a fault in my text: s/that that/that/
Heh, fixed.
Thanks.
--
tejun