Hi,
I believe I have encountered a bug in kernel 3.6-rc3. It starts with the
assertion in mm/slab.c:2629 failing, and then the system hangs. I can
reproduce this bug by running a large compilation (compiling the kernel
for instance).
Here's what I see in netconsole:
> ------------[ cut here ]------------
> kernel BUG at mm/slab.c:2629!
> invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
I'm attaching netconsole logs I got with kernel 3.6-rc1, which contain a
little more details after the crash, but for some reason netconsole
didn't capture the full stack trace of the assertion. I caught a glimpse
at the console and I saw RIP was at cache_alloc_refill.
I'd be happy to provide further information or perform testing to help
solve this issue.
Regards,
Haggai Eran
On Tue, 28 Aug 2012, Haggai Eran wrote:
> Hi,
>
> I believe I have encountered a bug in kernel 3.6-rc3. It starts with the
> assertion in mm/slab.c:2629 failing, and then the system hangs. I can
> reproduce this bug by running a large compilation (compiling the kernel
> for instance).
>
> Here's what I see in netconsole:
> > ------------[ cut here ]------------
> > kernel BUG at mm/slab.c:2629!
> > invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
>
> I'm attaching netconsole logs I got with kernel 3.6-rc1, which contain a
> little more details after the crash, but for some reason netconsole
> didn't capture the full stack trace of the assertion. I caught a glimpse
> at the console and I saw RIP was at cache_alloc_refill.
>
It only gets called from cache_alloc_refill().
Looks like a problem in 072bb0aa5e0 ("mm: sl[au]b: add knowledge of
PFMEMALLOC reserve pages"). cache_grow() can reenable irqs which allows
this to be scheduled on a different cpu, possibly with a different node.
So it turns out that we lock the wrong node's list_lock because we don't
check the new node id when irqs are disabled again.
I doubt you can reliably reproduce this, but the following should fix the
issue.
mm, slab: lock the correct nodelist after reenabling irqs
cache_grow() can reenable irqs so the cpu (and node) can change, so ensure
that we take list_lock on the correct nodelist.
Fixes an issue with 072bb0aa5e0 ("mm: sl[au]b: add knowledge of PFMEMALLOC
reserve pages") where list_lock for the wrong node was taken after growing
the cache.
Reported-by: Haggai Eran <[email protected]>
Signed-off-by: David Rientjes <[email protected]>
---
mm/slab.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/mm/slab.c b/mm/slab.c
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -3260,6 +3260,7 @@ force_grow:
/* cache_grow can reenable interrupts, then ac could change. */
ac = cpu_cache_get(cachep);
+ node = numa_mem_id();
/* no objects in sight? abort */
if (!x && (ac->avail == 0 || force_refill))
On 29/08/2012 05:57, David Rientjes wrote:
> On Tue, 28 Aug 2012, Haggai Eran wrote:
>
>> Hi,
>>
>> I believe I have encountered a bug in kernel 3.6-rc3. It starts with the
>> assertion in mm/slab.c:2629 failing, and then the system hangs. I can
>> reproduce this bug by running a large compilation (compiling the kernel
>> for instance).
>>
>> Here's what I see in netconsole:
>>> ------------[ cut here ]------------
>>> kernel BUG at mm/slab.c:2629!
>>> invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
>> I'm attaching netconsole logs I got with kernel 3.6-rc1, which contain a
>> little more details after the crash, but for some reason netconsole
>> didn't capture the full stack trace of the assertion. I caught a glimpse
>> at the console and I saw RIP was at cache_alloc_refill.
>>
> It only gets called from cache_alloc_refill().
>
> Looks like a problem in 072bb0aa5e0 ("mm: sl[au]b: add knowledge of
> PFMEMALLOC reserve pages"). cache_grow() can reenable irqs which allows
> this to be scheduled on a different cpu, possibly with a different node.
> So it turns out that we lock the wrong node's list_lock because we don't
> check the new node id when irqs are disabled again.
>
> I doubt you can reliably reproduce this, but the following should fix the
> issue.
Your patch did solve the issue. Thanks!
On Wed, Aug 29, 2012 at 2:41 PM, Haggai Eran <[email protected]> wrote:
>> Looks like a problem in 072bb0aa5e0 ("mm: sl[au]b: add knowledge of
>> PFMEMALLOC reserve pages"). cache_grow() can reenable irqs which allows
>> this to be scheduled on a different cpu, possibly with a different node.
>> So it turns out that we lock the wrong node's list_lock because we don't
>> check the new node id when irqs are disabled again.
>>
>> I doubt you can reliably reproduce this, but the following should fix the
>> issue.
> Your patch did solve the issue. Thanks!
Applied, thanks!