It was reported that a kernel crash happened in mem_cgroup_iter(),
which can be triggered if the legacy cgroup-v1 non-hierarchical
mode is used.
Unable to handle kernel paging request at virtual address 6b6b6b6b6b6b8f
......
Call trace:
mem_cgroup_iter+0x2e0/0x6d4
shrink_zone+0x8c/0x324
balance_pgdat+0x450/0x640
kswapd+0x130/0x4b8
kthread+0xe8/0xfc
ret_from_fork+0x10/0x20
mem_cgroup_iter():
......
if (css_tryget(css)) <-- crash here
break;
......
The crashing reason is that mem_cgroup_iter() uses the memcg object
whose pointer is stored in iter->position, which has been freed before
and filled with POISON_FREE(0x6b).
And the root cause of the use-after-free issue is that
invalidate_reclaim_iterators() fails to reset the value of
iter->position to NULL when the css of the memcg is released in non-
hierarchical mode.
Signed-off-by: Jing Xia <[email protected]>
---
mm/memcontrol.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index e6f0d5e..8c0280b 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -850,7 +850,7 @@ static void invalidate_reclaim_iterators(struct mem_cgroup *dead_memcg)
int nid;
int i;
- while ((memcg = parent_mem_cgroup(memcg))) {
+ for (; memcg; memcg = parent_mem_cgroup(memcg)) {
for_each_node(nid) {
mz = mem_cgroup_nodeinfo(memcg, nid);
for (i = 0; i <= DEF_PRIORITY; i++) {
--
1.9.1
[CC Andrew]
On Thu 19-07-18 18:06:47, Jing Xia wrote:
> It was reported that a kernel crash happened in mem_cgroup_iter(),
> which can be triggered if the legacy cgroup-v1 non-hierarchical
> mode is used.
>
> Unable to handle kernel paging request at virtual address 6b6b6b6b6b6b8f
> ......
> Call trace:
> mem_cgroup_iter+0x2e0/0x6d4
> shrink_zone+0x8c/0x324
> balance_pgdat+0x450/0x640
> kswapd+0x130/0x4b8
> kthread+0xe8/0xfc
> ret_from_fork+0x10/0x20
>
> mem_cgroup_iter():
> ......
> if (css_tryget(css)) <-- crash here
> break;
> ......
>
> The crashing reason is that mem_cgroup_iter() uses the memcg object
> whose pointer is stored in iter->position, which has been freed before
> and filled with POISON_FREE(0x6b).
>
> And the root cause of the use-after-free issue is that
> invalidate_reclaim_iterators() fails to reset the value of
> iter->position to NULL when the css of the memcg is released in non-
> hierarchical mode.
Well, spotted!
I suspect
Fixes: 6df38689e0e9 ("mm: memcontrol: fix possible memcg leak due to interrupted reclaim")
but maybe it goes further into past. I also suggest
Cc: stable
even though the non-hierarchical mode is strongly discouraged. A lack of
reports for 3 years is encouraging that not many people really use this
mode.
> Signed-off-by: Jing Xia <[email protected]>
Acked-by: Michal Hocko <[email protected]>
Thanks!
> ---
> mm/memcontrol.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index e6f0d5e..8c0280b 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -850,7 +850,7 @@ static void invalidate_reclaim_iterators(struct mem_cgroup *dead_memcg)
> int nid;
> int i;
>
> - while ((memcg = parent_mem_cgroup(memcg))) {
> + for (; memcg; memcg = parent_mem_cgroup(memcg)) {
> for_each_node(nid) {
> mz = mem_cgroup_nodeinfo(memcg, nid);
> for (i = 0; i <= DEF_PRIORITY; i++) {
> --
> 1.9.1
--
Michal Hocko
SUSE Labs
On Thu, Jul 19, 2018 at 3:43 AM Michal Hocko <[email protected]> wrote:
>
> [CC Andrew]
>
> On Thu 19-07-18 18:06:47, Jing Xia wrote:
> > It was reported that a kernel crash happened in mem_cgroup_iter(),
> > which can be triggered if the legacy cgroup-v1 non-hierarchical
> > mode is used.
> >
> > Unable to handle kernel paging request at virtual address 6b6b6b6b6b6b8f
> > ......
> > Call trace:
> > mem_cgroup_iter+0x2e0/0x6d4
> > shrink_zone+0x8c/0x324
> > balance_pgdat+0x450/0x640
> > kswapd+0x130/0x4b8
> > kthread+0xe8/0xfc
> > ret_from_fork+0x10/0x20
> >
> > mem_cgroup_iter():
> > ......
> > if (css_tryget(css)) <-- crash here
> > break;
> > ......
> >
> > The crashing reason is that mem_cgroup_iter() uses the memcg object
> > whose pointer is stored in iter->position, which has been freed before
> > and filled with POISON_FREE(0x6b).
> >
> > And the root cause of the use-after-free issue is that
> > invalidate_reclaim_iterators() fails to reset the value of
> > iter->position to NULL when the css of the memcg is released in non-
> > hierarchical mode.
>
> Well, spotted!
>
> I suspect
> Fixes: 6df38689e0e9 ("mm: memcontrol: fix possible memcg leak due to interrupted reclaim")
>
> but maybe it goes further into past. I also suggest
> Cc: stable
>
> even though the non-hierarchical mode is strongly discouraged.
Why not set root_mem_cgroup's use_hierarchy to true by default on
init? If someone wants non-hierarchical mode, they can explicitly set
it to false.
> A lack of
> reports for 3 years is encouraging that not many people really use this
> mode.
>
> > Signed-off-by: Jing Xia <[email protected]>
>
> Acked-by: Michal Hocko <[email protected]>
>
> Thanks!
> > ---
> > mm/memcontrol.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > index e6f0d5e..8c0280b 100644
> > --- a/mm/memcontrol.c
> > +++ b/mm/memcontrol.c
> > @@ -850,7 +850,7 @@ static void invalidate_reclaim_iterators(struct mem_cgroup *dead_memcg)
> > int nid;
> > int i;
> >
> > - while ((memcg = parent_mem_cgroup(memcg))) {
> > + for (; memcg; memcg = parent_mem_cgroup(memcg)) {
> > for_each_node(nid) {
> > mz = mem_cgroup_nodeinfo(memcg, nid);
> > for (i = 0; i <= DEF_PRIORITY; i++) {
> > --
> > 1.9.1
>
> --
> Michal Hocko
> SUSE Labs
> --
> To unsubscribe from this list: send the line "unsubscribe cgroups" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu 19-07-18 09:23:10, Shakeel Butt wrote:
> On Thu, Jul 19, 2018 at 3:43 AM Michal Hocko <[email protected]> wrote:
> >
> > [CC Andrew]
> >
> > On Thu 19-07-18 18:06:47, Jing Xia wrote:
> > > It was reported that a kernel crash happened in mem_cgroup_iter(),
> > > which can be triggered if the legacy cgroup-v1 non-hierarchical
> > > mode is used.
> > >
> > > Unable to handle kernel paging request at virtual address 6b6b6b6b6b6b8f
> > > ......
> > > Call trace:
> > > mem_cgroup_iter+0x2e0/0x6d4
> > > shrink_zone+0x8c/0x324
> > > balance_pgdat+0x450/0x640
> > > kswapd+0x130/0x4b8
> > > kthread+0xe8/0xfc
> > > ret_from_fork+0x10/0x20
> > >
> > > mem_cgroup_iter():
> > > ......
> > > if (css_tryget(css)) <-- crash here
> > > break;
> > > ......
> > >
> > > The crashing reason is that mem_cgroup_iter() uses the memcg object
> > > whose pointer is stored in iter->position, which has been freed before
> > > and filled with POISON_FREE(0x6b).
> > >
> > > And the root cause of the use-after-free issue is that
> > > invalidate_reclaim_iterators() fails to reset the value of
> > > iter->position to NULL when the css of the memcg is released in non-
> > > hierarchical mode.
> >
> > Well, spotted!
> >
> > I suspect
> > Fixes: 6df38689e0e9 ("mm: memcontrol: fix possible memcg leak due to interrupted reclaim")
> >
> > but maybe it goes further into past. I also suggest
> > Cc: stable
> >
> > even though the non-hierarchical mode is strongly discouraged.
>
> Why not set root_mem_cgroup's use_hierarchy to true by default on
> init? If someone wants non-hierarchical mode, they can explicitly set
> it to false.
We do not change defaults under users feet usually.
--
Michal Hocko
SUSE Labs
On Sun, Jul 22, 2018 at 11:44 PM Michal Hocko <[email protected]> wrote:
>
> On Thu 19-07-18 09:23:10, Shakeel Butt wrote:
> > On Thu, Jul 19, 2018 at 3:43 AM Michal Hocko <[email protected]> wrote:
> > >
> > > [CC Andrew]
> > >
> > > On Thu 19-07-18 18:06:47, Jing Xia wrote:
> > > > It was reported that a kernel crash happened in mem_cgroup_iter(),
> > > > which can be triggered if the legacy cgroup-v1 non-hierarchical
> > > > mode is used.
> > > >
> > > > Unable to handle kernel paging request at virtual address 6b6b6b6b6b6b8f
> > > > ......
> > > > Call trace:
> > > > mem_cgroup_iter+0x2e0/0x6d4
> > > > shrink_zone+0x8c/0x324
> > > > balance_pgdat+0x450/0x640
> > > > kswapd+0x130/0x4b8
> > > > kthread+0xe8/0xfc
> > > > ret_from_fork+0x10/0x20
> > > >
> > > > mem_cgroup_iter():
> > > > ......
> > > > if (css_tryget(css)) <-- crash here
> > > > break;
> > > > ......
> > > >
> > > > The crashing reason is that mem_cgroup_iter() uses the memcg object
> > > > whose pointer is stored in iter->position, which has been freed before
> > > > and filled with POISON_FREE(0x6b).
> > > >
> > > > And the root cause of the use-after-free issue is that
> > > > invalidate_reclaim_iterators() fails to reset the value of
> > > > iter->position to NULL when the css of the memcg is released in non-
> > > > hierarchical mode.
> > >
> > > Well, spotted!
> > >
> > > I suspect
> > > Fixes: 6df38689e0e9 ("mm: memcontrol: fix possible memcg leak due to interrupted reclaim")
> > >
> > > but maybe it goes further into past. I also suggest
> > > Cc: stable
> > >
> > > even though the non-hierarchical mode is strongly discouraged.
> >
> > Why not set root_mem_cgroup's use_hierarchy to true by default on
> > init? If someone wants non-hierarchical mode, they can explicitly set
> > it to false.
>
> We do not change defaults under users feet usually.
Then how non-hierarchical mode is being discouraged currently? I don't
see any comments in the docs.
Shakeel
On Mon 23-07-18 09:17:28, Shakeel Butt wrote:
> On Sun, Jul 22, 2018 at 11:44 PM Michal Hocko <[email protected]> wrote:
> >
> > On Thu 19-07-18 09:23:10, Shakeel Butt wrote:
> > > On Thu, Jul 19, 2018 at 3:43 AM Michal Hocko <[email protected]> wrote:
> > > >
> > > > [CC Andrew]
> > > >
> > > > On Thu 19-07-18 18:06:47, Jing Xia wrote:
> > > > > It was reported that a kernel crash happened in mem_cgroup_iter(),
> > > > > which can be triggered if the legacy cgroup-v1 non-hierarchical
> > > > > mode is used.
> > > > >
> > > > > Unable to handle kernel paging request at virtual address 6b6b6b6b6b6b8f
> > > > > ......
> > > > > Call trace:
> > > > > mem_cgroup_iter+0x2e0/0x6d4
> > > > > shrink_zone+0x8c/0x324
> > > > > balance_pgdat+0x450/0x640
> > > > > kswapd+0x130/0x4b8
> > > > > kthread+0xe8/0xfc
> > > > > ret_from_fork+0x10/0x20
> > > > >
> > > > > mem_cgroup_iter():
> > > > > ......
> > > > > if (css_tryget(css)) <-- crash here
> > > > > break;
> > > > > ......
> > > > >
> > > > > The crashing reason is that mem_cgroup_iter() uses the memcg object
> > > > > whose pointer is stored in iter->position, which has been freed before
> > > > > and filled with POISON_FREE(0x6b).
> > > > >
> > > > > And the root cause of the use-after-free issue is that
> > > > > invalidate_reclaim_iterators() fails to reset the value of
> > > > > iter->position to NULL when the css of the memcg is released in non-
> > > > > hierarchical mode.
> > > >
> > > > Well, spotted!
> > > >
> > > > I suspect
> > > > Fixes: 6df38689e0e9 ("mm: memcontrol: fix possible memcg leak due to interrupted reclaim")
> > > >
> > > > but maybe it goes further into past. I also suggest
> > > > Cc: stable
> > > >
> > > > even though the non-hierarchical mode is strongly discouraged.
> > >
> > > Why not set root_mem_cgroup's use_hierarchy to true by default on
> > > init? If someone wants non-hierarchical mode, they can explicitly set
> > > it to false.
> >
> > We do not change defaults under users feet usually.
>
> Then how non-hierarchical mode is being discouraged currently? I don't
> see any comments in the docs.
css_create warns about non-hierarchical hierarchies. We've been running
with a similar warning in (even older) SLES kernels for years now and
quite some tools have been updated because they simply didn't know they
are doing something wrong.
--
Michal Hocko
SUSE Labs