2020-04-16 07:38:07

by David Hildenbrand

[permalink] [raw]
Subject: [PATCH v2] mm/page_alloc: fix watchdog soft lockups during set_zone_contiguous()

Without CONFIG_PREEMPT, it can happen that we get soft lockups detected,
e.g., while booting up.

[ 105.608900] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:1]
[ 105.608933] Modules linked in:
[ 105.608933] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.6.0-next-20200331+ #4
[ 105.608933] Hardware name: Red Hat KVM, BIOS 1.11.1-4.module+el8.1.0+4066+0f1aadab 04/01/2014
[ 105.608933] RIP: 0010:__pageblock_pfn_to_page+0x134/0x1c0
[ 105.608933] Code: 85 c0 74 71 4a 8b 04 d0 48 85 c0 74 68 48 01 c1 74 63 f6 01 04 74 5e 48 c1 e7 06 4c 8b 05 cc 991
[ 105.608933] RSP: 0000:ffffb6d94000fe60 EFLAGS: 00010286 ORIG_RAX: ffffffffffffff13
[ 105.608933] RAX: fffff81953250000 RBX: 000000000a4c9600 RCX: ffff8fe9ff7c1990
[ 105.608933] RDX: ffff8fe9ff7dab80 RSI: 000000000a4c95ff RDI: 0000000293250000
[ 105.608933] RBP: ffff8fe9ff7dab80 R08: fffff816c0000000 R09: 0000000000000008
[ 105.608933] R10: 0000000000000014 R11: 0000000000000014 R12: 0000000000000000
[ 105.608933] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 105.608933] FS: 0000000000000000(0000) GS:ffff8fe1ff400000(0000) knlGS:0000000000000000
[ 105.608933] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 105.608933] CR2: 000000000f613000 CR3: 00000088cf20a000 CR4: 00000000000006f0
[ 105.608933] Call Trace:
[ 105.608933] set_zone_contiguous+0x56/0x70
[ 105.608933] page_alloc_init_late+0x166/0x176
[ 105.608933] kernel_init_freeable+0xfa/0x255
[ 105.608933] ? rest_init+0xaa/0xaa
[ 105.608933] kernel_init+0xa/0x106
[ 105.608933] ret_from_fork+0x35/0x40

The issue becomes visible when having a lot of memory (e.g., 4TB)
assigned to a single NUMA node - a system that can easily be created
using QEMU. Inside VMs on a hypervisor with quite some memory
overcommit, this is fairly easy to trigger.

Reviewed-by: Pavel Tatashin <[email protected]>
Reviewed-by: Pankaj Gupta <[email protected]>
Reviewed-by: Baoquan He <[email protected]>
Reviewed-by: Shile Zhang <[email protected]>
Acked-by: Michal Hocko <[email protected]>
Cc: Andrew Morton <[email protected]>
Cc: Kirill Tkhai <[email protected]>
Cc: Shile Zhang <[email protected]>
Cc: Pavel Tatashin <[email protected]>
Cc: Daniel Jordan <[email protected]>
Cc: Michal Hocko <[email protected]>
Cc: Alexander Duyck <[email protected]>
Cc: Baoquan He <[email protected]>
Cc: Oscar Salvador <[email protected]>
Signed-off-by: David Hildenbrand <[email protected]>
---

This is just a resend of patch #2 of
https://lkml.kernel.org/r/[email protected]

Not that patch #1 was replaced by
https://lkml.kernel.org/r/[email protected]

---
mm/page_alloc.c | 1 +
1 file changed, 1 insertion(+)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 1385d786a01a..ae1545246b69 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1607,6 +1607,7 @@ void set_zone_contiguous(struct zone *zone)
if (!__pageblock_pfn_to_page(block_start_pfn,
block_end_pfn, zone))
return;
+ cond_resched();
}

/* We confirm that there is no hole */
--
2.25.1


2020-04-17 22:14:59

by Andrew Morton

[permalink] [raw]
Subject: Re: [PATCH v2] mm/page_alloc: fix watchdog soft lockups during set_zone_contiguous()

On Thu, 16 Apr 2020 09:34:17 +0200 David Hildenbrand <[email protected]> wrote:

> Without CONFIG_PREEMPT, it can happen that we get soft lockups detected,
> e.g., while booting up.
>
> ...
>
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -1607,6 +1607,7 @@ void set_zone_contiguous(struct zone *zone)
> if (!__pageblock_pfn_to_page(block_start_pfn,
> block_end_pfn, zone))
> return;
> + cond_resched();
> }
>
> /* We confirm that there is no hole */

I added cc:stable to this one. Please let me know if that wasn't a
good idea.

2020-04-20 07:37:10

by Michal Hocko

[permalink] [raw]
Subject: Re: [PATCH v2] mm/page_alloc: fix watchdog soft lockups during set_zone_contiguous()

On Fri 17-04-20 15:12:47, Andrew Morton wrote:
> On Thu, 16 Apr 2020 09:34:17 +0200 David Hildenbrand <[email protected]> wrote:
>
> > Without CONFIG_PREEMPT, it can happen that we get soft lockups detected,
> > e.g., while booting up.
> >
> > ...
> >
> > --- a/mm/page_alloc.c
> > +++ b/mm/page_alloc.c
> > @@ -1607,6 +1607,7 @@ void set_zone_contiguous(struct zone *zone)
> > if (!__pageblock_pfn_to_page(block_start_pfn,
> > block_end_pfn, zone))
> > return;
> > + cond_resched();
> > }
> >
> > /* We confirm that there is no hole */
>
> I added cc:stable to this one. Please let me know if that wasn't a
> good idea.

Really large memory setups tend to run on distribution kernels so
backporting to old kernels doesn't really harm.
--
Michal Hocko
SUSE Labs

2020-04-20 07:43:50

by David Hildenbrand

[permalink] [raw]
Subject: Re: [PATCH v2] mm/page_alloc: fix watchdog soft lockups during set_zone_contiguous()

On 20.04.20 09:34, Michal Hocko wrote:
> On Fri 17-04-20 15:12:47, Andrew Morton wrote:
>> On Thu, 16 Apr 2020 09:34:17 +0200 David Hildenbrand <[email protected]> wrote:
>>
>>> Without CONFIG_PREEMPT, it can happen that we get soft lockups detected,
>>> e.g., while booting up.
>>>
>>> ...
>>>
>>> --- a/mm/page_alloc.c
>>> +++ b/mm/page_alloc.c
>>> @@ -1607,6 +1607,7 @@ void set_zone_contiguous(struct zone *zone)
>>> if (!__pageblock_pfn_to_page(block_start_pfn,
>>> block_end_pfn, zone))
>>> return;
>>> + cond_resched();
>>> }
>>>
>>> /* We confirm that there is no hole */
>>
>> I added cc:stable to this one. Please let me know if that wasn't a
>> good idea.
>
> Really large memory setups tend to run on distribution kernels so
> backporting to old kernels doesn't really harm.
>

Yeah, shouldn't hurt.

--
Thanks,

David / dhildenb