v5.13-rc1 sometimes causes NULL pointer dereference during kdump, where
memcg is disabled with "cgroup_disable=memory" boot option.
I haven't seen this problem with v5.12, so it looks like regression.
[ 73.199590] BUG: kernel NULL pointer dereference, address: 0000000000000000
[ 73.206593] #PF: supervisor write access in kernel mode
[ 73.211845] #PF: error_code(0x0002) - not-present page
[ 73.217010] PGD 0 P4D 0
[ 73.219556] Oops: 0002 [#1] SMP NOPTI
[ 73.223236] CPU: 0 PID: 95 Comm: kswapd0 Tainted: G I 5.13.0-rc1 #1
[ 73.239418] RIP: 0010:do_shrink_slab+0x85/0x2d0
[ 73.243977] Code: 49 63 44 24 04 be 00 00 00 00 49 8b 4c 24 18 f6 c2 02 48 0f 44 c6 48 85 c9 74 09 83 e2 04 0f 85 19 02 00 00 49 8b 4f 38 31 d2 <48> 87 14 c1 48 89 55 b8 41 8b 77 18 4c 89 f0 85 f6 0f 84 82 01 00
[ 73.262856] RSP: 0018:ffffc900001abc18 EFLAGS: 00010246
[ 73.268108] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 73.275281] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000064
[ 73.282454] RBP: ffffc900001abc70 R08: 28f5c28f5c28f5c3 R09: 0000000000000000
[ 73.289628] R10: 0000000000000000 R11: 0000000000000004 R12: ffffc900001abca0
[ 73.296800] R13: 0000000000000400 R14: 0000000000000002 R15: ffff88805344bc10
[ 73.303972] FS: 0000000000000000(0000) GS:ffff888072c00000(0000) knlGS:0000000000000000
[ 73.312108] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 73.317883] CR2: 0000000000000000 CR3: 000000005cf68004 CR4: 00000000007706b0
[ 73.325055] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 73.332227] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 73.339400] PKRU: 55555554
[ 73.342117] Call Trace:
[ 73.344576] shrink_slab+0xa9/0x2b0
[ 73.348083] ? __update_load_avg_se+0x298/0x320
[ 73.352640] shrink_node+0x248/0x6f0
[ 73.356234] balance_pgdat+0x303/0x5f0
[ 73.360002] kswapd+0x20b/0x390
[ 73.363157] ? finish_wait+0x80/0x80
[ 73.366752] ? balance_pgdat+0x5f0/0x5f0
[ 73.370693] kthread+0x124/0x140
[ 73.373937] ? kthread_park+0x90/0x90
[ 73.377617] ret_from_fork+0x1f/0x30
[ 73.381215] Modules linked in: xfs libcrc32c sd_mod t10_pi sr_mod cdrom sg crc32c_intel ahci libahci libata smartpqi scsi_transport_sas overlay squashfs loop
[ 73.395386] CR2: 0000000000000000
[ 73.398716] ---[ end trace 9752d71309d33c00 ]---
The code around do_shrink_slab+0x85 is:
0xffffffff9d094925 <do_shrink_slab+0x65>: mov 0x18(%r12),%rcx
0xffffffff9d09492a <do_shrink_slab+0x6a>: test $0x2,%dl
0xffffffff9d09492d <do_shrink_slab+0x6d>: cmove %rsi,%rax
0xffffffff9d094931 <do_shrink_slab+0x71>: test %rcx,%rcx
0xffffffff9d094934 <do_shrink_slab+0x74>: je 0xffffffff9d09493f <do_shrink_slab+0x7f>
0xffffffff9d094936 <do_shrink_slab+0x76>: and $0x4,%edx
0xffffffff9d094939 <do_shrink_slab+0x79>: jne 0xffffffff9d094b58 <do_shrink_slab+0x298>
0xffffffff9d09493f <do_shrink_slab+0x7f>: mov 0x38(%r15),%rcx
0xffffffff9d094943 <do_shrink_slab+0x83>: xor %edx,%edx
0xffffffff9d094945 <do_shrink_slab+0x85>: xchg %rdx,(%rcx,%rax,8)
The NULL dereference occurred at here in in-lined xchg_nr_deferred():
return atomic_long_xchg(&shrinker->nr_deferred[nid], 0);
that means "shrinker->nr_deferred" was NULL.
Though I haven't fully bisected between v5.12 and v5.13-rc1, I can reproduce
the problem with this commit:
476b30a0949a mm: vmscan: don't need allocate shrinker->nr_deferred for memcg aware shrinkers
but not with this previous commit:
867508304685 mm: vmscan: use per memcg nr_deferred of shrinker
With the commit 476b30a0949a, if a memcg-aware shrinker is registered before
cgroup_init(), shrinker->nr_deferred is NULL. However xchg_nr_deferred()
tries to use it as memcg is turned off via "cgroup_disable=memory".
Any thoughts?
--
Jun'ichi Nomura, NEC Corporation / NEC Solution Innovators, Ltd.
+Tejun Heo
On Wed, May 12, 2021 at 3:48 AM NOMURA JUNICHI(野村 淳一)
<[email protected]> wrote:
>
> v5.13-rc1 sometimes causes NULL pointer dereference during kdump, where
> memcg is disabled with "cgroup_disable=memory" boot option.
> I haven't seen this problem with v5.12, so it looks like regression.
>
> [ 73.199590] BUG: kernel NULL pointer dereference, address: 0000000000000000
> [ 73.206593] #PF: supervisor write access in kernel mode
> [ 73.211845] #PF: error_code(0x0002) - not-present page
> [ 73.217010] PGD 0 P4D 0
> [ 73.219556] Oops: 0002 [#1] SMP NOPTI
> [ 73.223236] CPU: 0 PID: 95 Comm: kswapd0 Tainted: G I 5.13.0-rc1 #1
> [ 73.239418] RIP: 0010:do_shrink_slab+0x85/0x2d0
> [ 73.243977] Code: 49 63 44 24 04 be 00 00 00 00 49 8b 4c 24 18 f6 c2 02 48 0f 44 c6 48 85 c9 74 09 83 e2 04 0f 85 19 02 00 00 49 8b 4f 38 31 d2 <48> 87 14 c1 48 89 55 b8 41 8b 77 18 4c 89 f0 85 f6 0f 84 82 01 00
> [ 73.262856] RSP: 0018:ffffc900001abc18 EFLAGS: 00010246
> [ 73.268108] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
> [ 73.275281] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000064
> [ 73.282454] RBP: ffffc900001abc70 R08: 28f5c28f5c28f5c3 R09: 0000000000000000
> [ 73.289628] R10: 0000000000000000 R11: 0000000000000004 R12: ffffc900001abca0
> [ 73.296800] R13: 0000000000000400 R14: 0000000000000002 R15: ffff88805344bc10
> [ 73.303972] FS: 0000000000000000(0000) GS:ffff888072c00000(0000) knlGS:0000000000000000
> [ 73.312108] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 73.317883] CR2: 0000000000000000 CR3: 000000005cf68004 CR4: 00000000007706b0
> [ 73.325055] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 73.332227] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [ 73.339400] PKRU: 55555554
> [ 73.342117] Call Trace:
> [ 73.344576] shrink_slab+0xa9/0x2b0
> [ 73.348083] ? __update_load_avg_se+0x298/0x320
> [ 73.352640] shrink_node+0x248/0x6f0
> [ 73.356234] balance_pgdat+0x303/0x5f0
> [ 73.360002] kswapd+0x20b/0x390
> [ 73.363157] ? finish_wait+0x80/0x80
> [ 73.366752] ? balance_pgdat+0x5f0/0x5f0
> [ 73.370693] kthread+0x124/0x140
> [ 73.373937] ? kthread_park+0x90/0x90
> [ 73.377617] ret_from_fork+0x1f/0x30
> [ 73.381215] Modules linked in: xfs libcrc32c sd_mod t10_pi sr_mod cdrom sg crc32c_intel ahci libahci libata smartpqi scsi_transport_sas overlay squashfs loop
> [ 73.395386] CR2: 0000000000000000
> [ 73.398716] ---[ end trace 9752d71309d33c00 ]---
>
> The code around do_shrink_slab+0x85 is:
> 0xffffffff9d094925 <do_shrink_slab+0x65>: mov 0x18(%r12),%rcx
> 0xffffffff9d09492a <do_shrink_slab+0x6a>: test $0x2,%dl
> 0xffffffff9d09492d <do_shrink_slab+0x6d>: cmove %rsi,%rax
> 0xffffffff9d094931 <do_shrink_slab+0x71>: test %rcx,%rcx
> 0xffffffff9d094934 <do_shrink_slab+0x74>: je 0xffffffff9d09493f <do_shrink_slab+0x7f>
> 0xffffffff9d094936 <do_shrink_slab+0x76>: and $0x4,%edx
> 0xffffffff9d094939 <do_shrink_slab+0x79>: jne 0xffffffff9d094b58 <do_shrink_slab+0x298>
> 0xffffffff9d09493f <do_shrink_slab+0x7f>: mov 0x38(%r15),%rcx
> 0xffffffff9d094943 <do_shrink_slab+0x83>: xor %edx,%edx
> 0xffffffff9d094945 <do_shrink_slab+0x85>: xchg %rdx,(%rcx,%rax,8)
>
> The NULL dereference occurred at here in in-lined xchg_nr_deferred():
>
> return atomic_long_xchg(&shrinker->nr_deferred[nid], 0);
>
> that means "shrinker->nr_deferred" was NULL.
>
> Though I haven't fully bisected between v5.12 and v5.13-rc1, I can reproduce
> the problem with this commit:
>
> 476b30a0949a mm: vmscan: don't need allocate shrinker->nr_deferred for memcg aware shrinkers
>
> but not with this previous commit:
>
> 867508304685 mm: vmscan: use per memcg nr_deferred of shrinker
>
> With the commit 476b30a0949a, if a memcg-aware shrinker is registered before
> cgroup_init(), shrinker->nr_deferred is NULL. However xchg_nr_deferred()
> tries to use it as memcg is turned off via "cgroup_disable=memory".
>
> Any thoughts?
Is there a way to find the call chain of "memcg-aware shrinker is
registered before cgroup_init()"?
Irrespective I think we can revert a3e72739b7a7e ("cgroup: fix too
early usage of static_branch_disable()") as 6041186a3258 ("init:
initialize jump labels before command line option parsing") has moved
the initialization of jump labels before command line parsing.
On Wed, May 12, 2021 at 5:36 AM Shakeel Butt <[email protected]> wrote:
>
> +Tejun Heo
>
> On Wed, May 12, 2021 at 3:48 AM NOMURA JUNICHI(野村 淳一)
> <[email protected]> wrote:
> >
> > v5.13-rc1 sometimes causes NULL pointer dereference during kdump, where
> > memcg is disabled with "cgroup_disable=memory" boot option.
> > I haven't seen this problem with v5.12, so it looks like regression.
> >
> > [ 73.199590] BUG: kernel NULL pointer dereference, address: 0000000000000000
> > [ 73.206593] #PF: supervisor write access in kernel mode
> > [ 73.211845] #PF: error_code(0x0002) - not-present page
> > [ 73.217010] PGD 0 P4D 0
> > [ 73.219556] Oops: 0002 [#1] SMP NOPTI
> > [ 73.223236] CPU: 0 PID: 95 Comm: kswapd0 Tainted: G I 5.13.0-rc1 #1
> > [ 73.239418] RIP: 0010:do_shrink_slab+0x85/0x2d0
> > [ 73.243977] Code: 49 63 44 24 04 be 00 00 00 00 49 8b 4c 24 18 f6 c2 02 48 0f 44 c6 48 85 c9 74 09 83 e2 04 0f 85 19 02 00 00 49 8b 4f 38 31 d2 <48> 87 14 c1 48 89 55 b8 41 8b 77 18 4c 89 f0 85 f6 0f 84 82 01 00
> > [ 73.262856] RSP: 0018:ffffc900001abc18 EFLAGS: 00010246
> > [ 73.268108] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
> > [ 73.275281] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000064
> > [ 73.282454] RBP: ffffc900001abc70 R08: 28f5c28f5c28f5c3 R09: 0000000000000000
> > [ 73.289628] R10: 0000000000000000 R11: 0000000000000004 R12: ffffc900001abca0
> > [ 73.296800] R13: 0000000000000400 R14: 0000000000000002 R15: ffff88805344bc10
> > [ 73.303972] FS: 0000000000000000(0000) GS:ffff888072c00000(0000) knlGS:0000000000000000
> > [ 73.312108] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [ 73.317883] CR2: 0000000000000000 CR3: 000000005cf68004 CR4: 00000000007706b0
> > [ 73.325055] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > [ 73.332227] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > [ 73.339400] PKRU: 55555554
> > [ 73.342117] Call Trace:
> > [ 73.344576] shrink_slab+0xa9/0x2b0
> > [ 73.348083] ? __update_load_avg_se+0x298/0x320
> > [ 73.352640] shrink_node+0x248/0x6f0
> > [ 73.356234] balance_pgdat+0x303/0x5f0
> > [ 73.360002] kswapd+0x20b/0x390
> > [ 73.363157] ? finish_wait+0x80/0x80
> > [ 73.366752] ? balance_pgdat+0x5f0/0x5f0
> > [ 73.370693] kthread+0x124/0x140
> > [ 73.373937] ? kthread_park+0x90/0x90
> > [ 73.377617] ret_from_fork+0x1f/0x30
> > [ 73.381215] Modules linked in: xfs libcrc32c sd_mod t10_pi sr_mod cdrom sg crc32c_intel ahci libahci libata smartpqi scsi_transport_sas overlay squashfs loop
> > [ 73.395386] CR2: 0000000000000000
> > [ 73.398716] ---[ end trace 9752d71309d33c00 ]---
> >
> > The code around do_shrink_slab+0x85 is:
> > 0xffffffff9d094925 <do_shrink_slab+0x65>: mov 0x18(%r12),%rcx
> > 0xffffffff9d09492a <do_shrink_slab+0x6a>: test $0x2,%dl
> > 0xffffffff9d09492d <do_shrink_slab+0x6d>: cmove %rsi,%rax
> > 0xffffffff9d094931 <do_shrink_slab+0x71>: test %rcx,%rcx
> > 0xffffffff9d094934 <do_shrink_slab+0x74>: je 0xffffffff9d09493f <do_shrink_slab+0x7f>
> > 0xffffffff9d094936 <do_shrink_slab+0x76>: and $0x4,%edx
> > 0xffffffff9d094939 <do_shrink_slab+0x79>: jne 0xffffffff9d094b58 <do_shrink_slab+0x298>
> > 0xffffffff9d09493f <do_shrink_slab+0x7f>: mov 0x38(%r15),%rcx
> > 0xffffffff9d094943 <do_shrink_slab+0x83>: xor %edx,%edx
> > 0xffffffff9d094945 <do_shrink_slab+0x85>: xchg %rdx,(%rcx,%rax,8)
> >
> > The NULL dereference occurred at here in in-lined xchg_nr_deferred():
> >
> > return atomic_long_xchg(&shrinker->nr_deferred[nid], 0);
> >
> > that means "shrinker->nr_deferred" was NULL.
> >
> > Though I haven't fully bisected between v5.12 and v5.13-rc1, I can reproduce
> > the problem with this commit:
> >
> > 476b30a0949a mm: vmscan: don't need allocate shrinker->nr_deferred for memcg aware shrinkers
> >
> > but not with this previous commit:
> >
> > 867508304685 mm: vmscan: use per memcg nr_deferred of shrinker
> >
> > With the commit 476b30a0949a, if a memcg-aware shrinker is registered before
> > cgroup_init(), shrinker->nr_deferred is NULL. However xchg_nr_deferred()
> > tries to use it as memcg is turned off via "cgroup_disable=memory".
> >
> > Any thoughts?
Thanks for the report.
>
> Is there a way to find the call chain of "memcg-aware shrinker is
> registered before cgroup_init()"?
Other than adding some printk in prealloc_memcg_shrinker() then
checking out the output of dmesg I didn't think of a better way. Not
sure if we have something like early trace.
>
> Irrespective I think we can revert a3e72739b7a7e ("cgroup: fix too
> early usage of static_branch_disable()") as 6041186a3258 ("init:
> initialize jump labels before command line option parsing") has moved
> the initialization of jump labels before command line parsing.
Seems make sense to me. If some memcg aware shrinker is registered
before cgroup_init(), the mem_cgroup_disabled() check in
prealloc_memcg_shrinker() would return false negative. And I don't
think any shrinker could be registered before parsing boot
commandline.
On 2021/05/13 1:31, Yang Shi wrote:
> On Wed, May 12, 2021 at 5:36 AM Shakeel Butt <[email protected]> wrote:
>>
>> +Tejun Heo
>>
>> On Wed, May 12, 2021 at 3:48 AM NOMURA JUNICHI($BLnB<!!=_0l(B)
>> <[email protected]> wrote:
>>> With the commit 476b30a0949a, if a memcg-aware shrinker is registered before
>>> cgroup_init(), shrinker->nr_deferred is NULL. However xchg_nr_deferred()
>>> tries to use it as memcg is turned off via "cgroup_disable=memory".
>>>
>>> Any thoughts?
>
> Thanks for the report.
>
>>
>> Is there a way to find the call chain of "memcg-aware shrinker is
>> registered before cgroup_init()"?
>
> Other than adding some printk in prealloc_memcg_shrinker() then
> checking out the output of dmesg I didn't think of a better way. Not
> sure if we have something like early trace.
This is the first registration of memcg-aware shrinker:
[ 1.933693] Call Trace:
[ 1.934694] sget_fc+0x20d/0x2f0
[ 1.935693] ? compare_single+0x10/0x10
[ 1.936693] ? shmem_create+0x30/0x30
[ 1.937693] vfs_get_super+0x3e/0x100
[ 1.938693] get_tree_nodev+0x16/0x20
[ 1.939693] shmem_get_tree+0x15/0x20
[ 1.940693] vfs_get_tree+0x2a/0xc0
[ 1.941693] fc_mount+0x12/0x40
[ 1.942693] vfs_kern_mount.part.43+0x61/0xa0
[ 1.943693] kern_mount+0x24/0x40
[ 1.944693] shmem_init+0x5c/0xc8
[ 1.945693] mnt_init+0x12f/0x24a
[ 1.946693] ? __percpu_counter_init+0x8f/0xb0
[ 1.947693] vfs_caches_init+0xce/0xda
[ 1.948693] start_kernel+0x479/0x4e3
[ 1.949693] x86_64_start_reservations+0x24/0x26
[ 1.950693] x86_64_start_kernel+0x8a/0x8d
[ 1.951693] secondary_startup_64_no_verify+0xc2/0xcb
That is done after command line parsing but before cgroup_init.
>> Irrespective I think we can revert a3e72739b7a7e ("cgroup: fix too
>> early usage of static_branch_disable()") as 6041186a3258 ("init:
>> initialize jump labels before command line option parsing") has moved
>> the initialization of jump labels before command line parsing.
>
> Seems make sense to me. If some memcg aware shrinker is registered
> before cgroup_init(), the mem_cgroup_disabled() check in
> prealloc_memcg_shrinker() would return false negative. And I don't
> think any shrinker could be registered before parsing boot
> commandline.
Thank you. Shakeel 's patch works for me:
[PATCH] cgroup: disable controllers at parse time
https://lore.kernel.org/linux-mm/[email protected]/
--
Jun'ichi Nomura, NEC Corporation / NEC Solution Innovators, Ltd.
On Wed, May 12, 2021 at 5:10 PM NOMURA JUNICHI(野村 淳一)
<[email protected]> wrote:
>
> On 2021/05/13 1:31, Yang Shi wrote:
> > On Wed, May 12, 2021 at 5:36 AM Shakeel Butt <[email protected]> wrote:
> >>
> >> +Tejun Heo
> >>
> >> On Wed, May 12, 2021 at 3:48 AM NOMURA JUNICHI(野村 淳一)
> >> <[email protected]> wrote:
> >>> With the commit 476b30a0949a, if a memcg-aware shrinker is registered before
> >>> cgroup_init(), shrinker->nr_deferred is NULL. However xchg_nr_deferred()
> >>> tries to use it as memcg is turned off via "cgroup_disable=memory".
> >>>
> >>> Any thoughts?
> >
> > Thanks for the report.
> >
> >>
> >> Is there a way to find the call chain of "memcg-aware shrinker is
> >> registered before cgroup_init()"?
> >
> > Other than adding some printk in prealloc_memcg_shrinker() then
> > checking out the output of dmesg I didn't think of a better way. Not
> > sure if we have something like early trace.
>
> This is the first registration of memcg-aware shrinker:
>
> [ 1.933693] Call Trace:
> [ 1.934694] sget_fc+0x20d/0x2f0
> [ 1.935693] ? compare_single+0x10/0x10
> [ 1.936693] ? shmem_create+0x30/0x30
> [ 1.937693] vfs_get_super+0x3e/0x100
> [ 1.938693] get_tree_nodev+0x16/0x20
> [ 1.939693] shmem_get_tree+0x15/0x20
> [ 1.940693] vfs_get_tree+0x2a/0xc0
> [ 1.941693] fc_mount+0x12/0x40
> [ 1.942693] vfs_kern_mount.part.43+0x61/0xa0
> [ 1.943693] kern_mount+0x24/0x40
> [ 1.944693] shmem_init+0x5c/0xc8
> [ 1.945693] mnt_init+0x12f/0x24a
> [ 1.946693] ? __percpu_counter_init+0x8f/0xb0
> [ 1.947693] vfs_caches_init+0xce/0xda
> [ 1.948693] start_kernel+0x479/0x4e3
> [ 1.949693] x86_64_start_reservations+0x24/0x26
> [ 1.950693] x86_64_start_kernel+0x8a/0x8d
> [ 1.951693] secondary_startup_64_no_verify+0xc2/0xcb
>
> That is done after command line parsing but before cgroup_init.
Thanks for sharing the log. I was not aware that shmem is initialized
and mounted so early.
>
> >> Irrespective I think we can revert a3e72739b7a7e ("cgroup: fix too
> >> early usage of static_branch_disable()") as 6041186a3258 ("init:
> >> initialize jump labels before command line option parsing") has moved
> >> the initialization of jump labels before command line parsing.
> >
> > Seems make sense to me. If some memcg aware shrinker is registered
> > before cgroup_init(), the mem_cgroup_disabled() check in
> > prealloc_memcg_shrinker() would return false negative. And I don't
> > think any shrinker could be registered before parsing boot
> > commandline.
>
> Thank you. Shakeel 's patch works for me:
>
> [PATCH] cgroup: disable controllers at parse time
> https://lore.kernel.org/linux-mm/[email protected]/
Thanks for running the test.
>
> --
> Jun'ichi Nomura, NEC Corporation / NEC Solution Innovators, Ltd.