Commit d23b5c577715 ("cgroup: Make operations on the cgroup root_list RCU
safe") adds a new rcu_head to the cgroup_root structure and kvfree_rcu()
for freeing the cgroup_root.
The current implementation of kvfree_rcu(), however, has the limitation
that the offset of the rcu_head structure within the larger data
structure must be less than 4096 or the compilation will fail. See the
macro definition of __is_kvfree_rcu_offset() in include/linux/rcupdate.h
for more information.
By putting rcu_head below the large cgroup structure, any change to the
cgroup structure that makes it larger run the risk of causing build
failure under certain configurations. Commit 77070eeb8821 ("cgroup:
Avoid false cacheline sharing of read mostly rstat_cpu") happens to be
the last straw that breaks it. Fix this problem by moving the rcu_head
structure up before the cgroup structure.
Fixes: d23b5c577715 ("cgroup: Make operations on the cgroup root_list RCU safe")
Reported-by: Stephen Rothwell <[email protected]>
Closes: https://lore.kernel.org/lkml/[email protected]/
Signed-off-by: Waiman Long <[email protected]>
Acked-by: Yafang Shao <[email protected]>
---
include/linux/cgroup-defs.h | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h
index 5a97ea95b564..ea48c861cd36 100644
--- a/include/linux/cgroup-defs.h
+++ b/include/linux/cgroup-defs.h
@@ -562,6 +562,10 @@ struct cgroup_root {
/* Unique id for this hierarchy. */
int hierarchy_id;
+ /* A list running through the active hierarchies */
+ struct list_head root_list;
+ struct rcu_head rcu; /* Must be near the top */
+
/*
* The root cgroup. The containing cgroup_root will be destroyed on its
* release. cgrp->ancestors[0] will be used overflowing into the
@@ -575,10 +579,6 @@ struct cgroup_root {
/* Number of cgroups in the hierarchy, used only for /proc/cgroups */
atomic_t nr_cgrps;
- /* A list running through the active hierarchies */
- struct list_head root_list;
- struct rcu_head rcu;
-
/* Hierarchy-specific flags */
unsigned int flags;
--
2.39.3
On Thu, Dec 07, 2023 at 08:46:14AM -0500, Waiman Long <[email protected]> wrote:
> Commit 77070eeb8821 ("cgroup: Avoid false cacheline sharing of read
> mostly rstat_cpu") happens to be the last straw that breaks it.
FTR, when I build kernel from that commit, I can see
> struct cgroup_root {
> struct kernfs_root * kf_root; /* 0 8 */
> unsigned int subsys_mask; /* 8 4 */
> int hierarchy_id; /* 12 4 */
>
> /* XXX 48 bytes hole, try to pack */
>
> /* --- cacheline 1 boundary (64 bytes) --- */
> struct cgroup cgrp __attribute__((__aligned__(64))); /* 64 2368 */
>
> /* XXX last struct has 8 bytes of padding */
>
> /* --- cacheline 38 boundary (2432 bytes) --- */
> struct cgroup * cgrp_ancestor_storage; /* 2432 8 */
> atomic_t nr_cgrps; /* 2440 4 */
>
> /* XXX 4 bytes hole, try to pack */
>
> struct list_head root_list; /* 2448 16 */
> struct callback_head rcu __attribute__((__aligned__(8))); /* 2464 16 */
> unsigned int flags; /* 2480 4 */
> char release_agent_path[4096]; /* 2484 4096 */
> /* --- cacheline 102 boundary (6528 bytes) was 52 bytes ago --- */
> char name[64]; /* 6580 64 */
>
> /* size: 6656, cachelines: 104, members: 11 */
> /* sum members: 6592, holes: 2, sum holes: 52 */
> /* padding: 12 */
> /* paddings: 1, sum paddings: 8 */
> /* forced alignments: 2, forced holes: 1, sum forced holes: 48 */
> } __attribute__((__aligned__(64)));
2480 has still quite a reserve below 4096. (I can't see an CONFIG_*
affecting this.)
Perhaps, I missed something from the linux-next merging thread?
Michal
On 12/7/23 11:46, Michal Koutný wrote:
> On Thu, Dec 07, 2023 at 08:46:14AM -0500, Waiman Long <[email protected]> wrote:
>> Commit 77070eeb8821 ("cgroup: Avoid false cacheline sharing of read
>> mostly rstat_cpu") happens to be the last straw that breaks it.
> FTR, when I build kernel from that commit, I can see
>
>> struct cgroup_root {
>> struct kernfs_root * kf_root; /* 0 8 */
>> unsigned int subsys_mask; /* 8 4 */
>> int hierarchy_id; /* 12 4 */
>>
>> /* XXX 48 bytes hole, try to pack */
>>
>> /* --- cacheline 1 boundary (64 bytes) --- */
>> struct cgroup cgrp __attribute__((__aligned__(64))); /* 64 2368 */
>>
>> /* XXX last struct has 8 bytes of padding */
>>
>> /* --- cacheline 38 boundary (2432 bytes) --- */
>> struct cgroup * cgrp_ancestor_storage; /* 2432 8 */
>> atomic_t nr_cgrps; /* 2440 4 */
>>
>> /* XXX 4 bytes hole, try to pack */
>>
>> struct list_head root_list; /* 2448 16 */
>> struct callback_head rcu __attribute__((__aligned__(8))); /* 2464 16 */
>> unsigned int flags; /* 2480 4 */
>> char release_agent_path[4096]; /* 2484 4096 */
>> /* --- cacheline 102 boundary (6528 bytes) was 52 bytes ago --- */
>> char name[64]; /* 6580 64 */
>>
>> /* size: 6656, cachelines: 104, members: 11 */
>> /* sum members: 6592, holes: 2, sum holes: 52 */
>> /* padding: 12 */
>> /* paddings: 1, sum paddings: 8 */
>> /* forced alignments: 2, forced holes: 1, sum forced holes: 48 */
>> } __attribute__((__aligned__(64)));
> 2480 has still quite a reserve below 4096. (I can't see an CONFIG_*
> affecting this.)
>
> Perhaps, I missed something from the linux-next merging thread?
CONFIG_LOCKDEP and some other debug configs are enabled with
allmodconfig. This can greatly increase the size of some of the
structures. I am not able to use pahole due to missing BTF info so I
don't the exact size. However, I can reproduce the build failure and the
patch is able to fix it.
Cheers,
Longman
On Thu, Dec 7, 2023 at 5:46 AM Waiman Long <[email protected]> wrote:
>
> Commit d23b5c577715 ("cgroup: Make operations on the cgroup root_list RCU
> safe") adds a new rcu_head to the cgroup_root structure and kvfree_rcu()
> for freeing the cgroup_root.
>
> The current implementation of kvfree_rcu(), however, has the limitation
> that the offset of the rcu_head structure within the larger data
> structure must be less than 4096 or the compilation will fail. See the
> macro definition of __is_kvfree_rcu_offset() in include/linux/rcupdate.h
> for more information.
>
> By putting rcu_head below the large cgroup structure, any change to the
> cgroup structure that makes it larger run the risk of causing build
> failure under certain configurations. Commit 77070eeb8821 ("cgroup:
> Avoid false cacheline sharing of read mostly rstat_cpu") happens to be
> the last straw that breaks it. Fix this problem by moving the rcu_head
> structure up before the cgroup structure.
>
> Fixes: d23b5c577715 ("cgroup: Make operations on the cgroup root_list RCU safe")
> Reported-by: Stephen Rothwell <[email protected]>
> Closes: https://lore.kernel.org/lkml/[email protected]/
> Signed-off-by: Waiman Long <[email protected]>
> Acked-by: Yafang Shao <[email protected]>
Reviewed-by: Yosry Ahmed <[email protected]>
On Thu, Dec 07, 2023 at 12:40:33PM -0500, Waiman Long <[email protected]> wrote:
> CONFIG_LOCKDEP and some other debug configs are enabled with allmodconfig.
Interesting, I had CONFIG_LOCKDEP=y.
I shared the numbers to rule out you're after something unrelated
> However, I can reproduce the build failure and the patch is? able to
> fix it.
That is a good proof :-)
The patch makes sense for future robustness,
Reviewed-by: Michal Koutn? <[email protected]>
On Thu, Dec 07, 2023 at 08:46:14AM -0500, Waiman Long wrote:
> Commit d23b5c577715 ("cgroup: Make operations on the cgroup root_list RCU
> safe") adds a new rcu_head to the cgroup_root structure and kvfree_rcu()
> for freeing the cgroup_root.
>
> The current implementation of kvfree_rcu(), however, has the limitation
> that the offset of the rcu_head structure within the larger data
> structure must be less than 4096 or the compilation will fail. See the
> macro definition of __is_kvfree_rcu_offset() in include/linux/rcupdate.h
> for more information.
>
> By putting rcu_head below the large cgroup structure, any change to the
> cgroup structure that makes it larger run the risk of causing build
> failure under certain configurations. Commit 77070eeb8821 ("cgroup:
> Avoid false cacheline sharing of read mostly rstat_cpu") happens to be
> the last straw that breaks it. Fix this problem by moving the rcu_head
> structure up before the cgroup structure.
>
> Fixes: d23b5c577715 ("cgroup: Make operations on the cgroup root_list RCU safe")
> Reported-by: Stephen Rothwell <[email protected]>
> Closes: https://lore.kernel.org/lkml/[email protected]/
> Signed-off-by: Waiman Long <[email protected]>
> Acked-by: Yafang Shao <[email protected]>
Applied to cgroup/for-6.8.
Thanks.
--
tejun