LinuxLists.cc - OOM-killer and strange RSS value in 3.9-rc7

2013-04-16 11:00:24

Subject: OOM-killer and strange RSS value in 3.9-rc7

Hi list,

On a power7 system, we have installed 3.9-rc7 and crash 6.1.6. If I run
something like "make -j 64" to compile linux kernel from source, sooner
or latter, oom-killer will be triggered. Before that, when I trying to
analyse the live system with crash, some processes' %MEM and RSS looks
too big:

========================================

GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64-unknown-linux-gnu"...

crash: read error: kernel virtual address: c00000000074ee38 type: "cpu_possible_mask"

crash: this kernel may be configured with CONFIG_STRICT_DEVMEM, which
renders /dev/mem unusable as a live memory source.

crash: trying /proc/kcore as an alternative to /dev/mem

KERNEL: /boot/vmlinux-3.9.0-rc7
DUMPFILE: /proc/kcore
CPUS: 80
DATE: Tue Apr 16 05:51:11 2013
UPTIME: 00:44:42
LOAD AVERAGE: 0.15, 0.53, 3.43
TASKS: 876
NODENAME: riblp3.upt.austin.ibm.com
RELEASE: 3.9.0-rc7
VERSION: #1 SMP Tue Apr 16 03:14:32 CDT 2013
MACHINE: ppc64 (unknown Mhz)
MEMORY: 4 GB
PID: 9086
COMMAND: "crash"
TASK: c0000000c3640000 [THREAD_INFO: c0000000d7900000]
CPU: 52
STATE: TASK_RUNNING (ACTIVE)

crash> ps -u rpc.idmapd
PID PPID CPU TASK ST %MEM VSZ RSS COMM
1823 1 1 c0000000a66c0000 IN 281474976710656.0 5312 18014398509480192 rpc.idmapd
crash>

========================================

I'd like to ask if this a problem of the kernel or crash.

Thanks in advance.

2013-04-16 20:16:45

by David Rientjes

[permalink] [raw]

Subject: Re: OOM-killer and strange RSS value in 3.9-rc7

On Tue, 16 Apr 2013, Han Pingtian wrote:

> Hi list,
>
> On a power7 system, we have installed 3.9-rc7 and crash 6.1.6. If I run
> something like "make -j 64" to compile linux kernel from source, sooner
> or latter, oom-killer will be triggered. Before that, when I trying to
> analyse the live system with crash, some processes' %MEM and RSS looks
> too big:
>

Do you have the oom killer log from /var/log/messages with
/proc/sys/vm/oom_dump_tasks enabled? Have you tried to reproduce this
issue with CONFIG_DEBUG_VM and CONFIG_DEBUG_PAGEALLOC enabled (you may
even want to consider CONFIG_KMEMLEAK)?

[permalink] [raw]

Subject: Re: OOM-killer and strange RSS value in 3.9-rc7

On Mon, 6 Jan 2014, Wanpeng Li wrote:

> >Is there any progress against slub's fix?
> >
> >MemTotal: 7760960 kB
> >Slab: 7064448 kB
> >SReclaimable: 143936 kB
> >SUnreclaim: 6920512 kB
> >
> >112084 10550 9% 16.00K 3507 32 1795584K kmalloc-16384
> >2497920 48092 1% 0.50K 19515 128 1248960K kmalloc-512
> >6058888 89363 1% 0.19K 17768 341 1137152K kmalloc-192
> >114468 13719 11% 4.58K 2082 55 532992K task_struct
> >
>
> This machine has 200 CPUs and 8G memory. There is an oom storm, we are
> seeing OOM even in boot process.
>

Is this still a problem with 3.9 and later kernels? Please try to
reproduce it on 3.13.

If it does reproduce, could you try to pinpoint the problem with kmemleak?
Look into Documentation/kmemleak.txt which should identify where these
leaks are coming from with your slab allocator of choice.