I'm running a vanilla 2.6.24.2 kernel on an x86 system with 2GB physical
RAM. We had a problem yesterday where Samba and Apache both "died" and
AFAIK, we weren't able to ssh into the system. One of our staff was able
to reboot from the console though.
So, I found the following information in the system log just before it
went down:
Mar 19 17:01:21 hermes kernel: smbd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
Mar 19 17:01:21 hermes kernel: Pid: 2757, comm: smbd Not tainted 2.6.24.2 #1
Mar 19 17:01:21 hermes kernel: [<c0139841>] oom_kill_process+0x54/0xf8
Mar 19 17:01:21 hermes kernel: [<c0139c3b>] out_of_memory+0x15f/0x194
Mar 19 17:01:21 hermes kernel: [<c013b85e>] __alloc_pages+0x239/0x2c7
Mar 19 17:01:21 hermes kernel: [<c0136de7>] sync_page+0x0/0x40
Mar 19 17:01:21 hermes kernel: [<c03a0430>] preempt_schedule+0x3a/0x55
Mar 19 17:01:21 hermes kernel: [<c013d283>] __do_page_cache_readahead+0xd5/0x1be
Mar 19 17:01:21 hermes kernel: [<c013d6b2>] do_page_cache_readahead+0x49/0x53
Mar 19 17:01:21 hermes kernel: [<c0138f95>] filemap_fault+0x19a/0x3b4
Mar 19 17:01:21 hermes kernel: [<c013ec4e>] wakeup_kswapd+0x2d/0x70
Mar 19 17:01:22 hermes kernel: [<c0141d80>] __do_fault+0x51/0x327
Mar 19 17:01:22 hermes kernel: [<c0142485>] do_wp_page+0x42f/0x43b
Mar 19 17:01:22 hermes kernel: [<c0143c3e>] handle_mm_fault+0x2aa/0x5b6
Mar 19 17:01:22 hermes kernel: [<c011861d>] wake_up_new_task+0x77/0x7b
Mar 19 17:01:22 hermes kernel: [<c0110f15>] do_page_fault+0x18d/0x530
Mar 19 17:01:22 hermes kernel: [<c0323142>] sys_socketcall+0xeb/0x242
Mar 19 17:01:22 hermes kernel: [<c015b799>] do_fcntl+0x1f8/0x27e
Mar 19 17:01:22 hermes kernel: [<c0100a20>] sys_clone+0x36/0x3b
Mar 19 17:01:22 hermes kernel: [<c0110d88>] do_page_fault+0x0/0x530
Mar 19 17:01:22 hermes kernel: [<c03a1a72>] error_code+0x72/0x78
Mar 19 17:01:22 hermes kernel: =======================
Mar 19 17:01:23 hermes kernel: Mem-info:
Mar 19 17:01:23 hermes kernel: DMA per-cpu:
Mar 19 17:01:23 hermes kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Mar 19 17:01:23 hermes kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Mar 19 17:01:23 hermes kernel: Normal per-cpu:
Mar 19 17:01:23 hermes kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 93 Cold: hi: 62, btch: 15 usd: 55
Mar 19 17:01:23 hermes kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 120 Cold: hi: 62, btch: 15 usd: 55
Mar 19 17:01:23 hermes kernel: HighMem per-cpu:
Mar 19 17:01:23 hermes kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 46 Cold: hi: 62, btch: 15 usd: 60
Mar 19 17:01:23 hermes kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 12 Cold: hi: 62, btch: 15 usd: 61
Mar 19 17:01:23 hermes kernel: Active:150615 inactive:339150 dirty:0 writeback:0 unstable:0
Mar 19 17:01:23 hermes kernel: free:12180 slab:6561 mapped:78 pagetables:2862 bounce:0
Mar 19 17:01:23 hermes kernel: DMA free:8132kB min:68kB low:84kB high:100kB active:1872kB inactive:1256kB present:16256kB pages_scanned:5836 all_unreclaimable? yes
Mar 19 17:01:23 hermes kernel: lowmem_reserve[]: 0 873 2016 2016
Mar 19 17:01:23 hermes kernel: Normal free:40136kB min:3744kB low:4680kB high:5616kB active:400240kB inactive:398884kB present:894080kB pages_scanned:2546811 all_unreclaimable? yes
Mar 19 17:01:23 hermes kernel: lowmem_reserve[]: 0 0 9143 9143
Mar 19 17:01:23 hermes kernel: HighMem free:452kB min:512kB low:1736kB high:2964kB active:200720kB inactive:956076kB present:1170372kB pages_scanned:3284202 all_unreclaimable? yes
Mar 19 17:01:23 hermes kernel: lowmem_reserve[]: 0 0 0 0
Mar 19 17:01:23 hermes kernel: DMA: 49*4kB 44*8kB 40*16kB 37*32kB 22*64kB 6*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 8132kB
Mar 19 17:01:23 hermes kernel: Normal: 9168*4kB 1*8kB 0*16kB 2*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 40136kB
Mar 19 17:01:23 hermes kernel: HighMem: 11*4kB 5*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 452kB
Mar 19 17:01:23 hermes kernel: Swap cache: add 550090, delete 550089, find 1023187/1039769, race 0+22
Mar 19 17:01:23 hermes kernel: Free swap = 0kB
Mar 19 17:01:23 hermes kernel: Total swap = 899064kB
Mar 19 17:01:23 hermes kernel: Free swap: 0kB
Mar 19 17:01:23 hermes kernel: 524272 pages of RAM
Mar 19 17:01:23 hermes kernel: 294896 pages of HIGHMEM
Mar 19 17:01:23 hermes kernel: 5542 reserved pages
Mar 19 17:01:23 hermes kernel: 55226 pages shared
Mar 19 17:01:23 hermes kernel: 1 pages swap cached
Mar 19 17:01:23 hermes kernel: 0 pages dirty
Mar 19 17:01:23 hermes kernel: 0 pages writeback
Mar 19 17:01:23 hermes kernel: 78 pages mapped
Mar 19 17:01:23 hermes kernel: 6561 pages slab
Mar 19 17:01:23 hermes kernel: 2862 pages pagetables
Mar 19 17:01:23 hermes kernel: Out of memory: kill process 5928 (apache2) score 40674 or a child
Mar 19 17:01:23 hermes kernel: Killed process 5928 (apache2)
Just wondering if anyone could help me with interpreting what is going
on here. Does it look there's anything weird going on in kernel space
based on this, or did something in userspace really just exhaust all
memory?
The best I can understand from the log, the problem the kernel detected
was that HighMem fell below the low watermark. Also very strange to see
that it says we have no free swap space - normally we only see a handful
of pages from the whole ~1GB of swap being used, even after weeks of
uptime.
TIA,
Kevin.
On 03/20/2008 01:48 AM, Kevin Shanahan wrote:
> I'm running a vanilla 2.6.24.2 kernel on an x86 system with 2GB physical
> RAM. We had a problem yesterday where Samba and Apache both "died" and
> AFAIK, we weren't able to ssh into the system. One of our staff was able
> to reboot from the console though.
>
> So, I found the following information in the system log just before it
> went down:
>
> Mar 19 17:01:21 hermes kernel: smbd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
> Mar 19 17:01:21 hermes kernel: Pid: 2757, comm: smbd Not tainted 2.6.24.2 #1
> Mar 19 17:01:21 hermes kernel: [<c0139841>] oom_kill_process+0x54/0xf8
> Mar 19 17:01:21 hermes kernel: [<c0139c3b>] out_of_memory+0x15f/0x194
> Mar 19 17:01:21 hermes kernel: [<c013b85e>] __alloc_pages+0x239/0x2c7
> Mar 19 17:01:21 hermes kernel: [<c0136de7>] sync_page+0x0/0x40
> Mar 19 17:01:21 hermes kernel: [<c03a0430>] preempt_schedule+0x3a/0x55
> Mar 19 17:01:21 hermes kernel: [<c013d283>] __do_page_cache_readahead+0xd5/0x1be
> Mar 19 17:01:21 hermes kernel: [<c013d6b2>] do_page_cache_readahead+0x49/0x53
> Mar 19 17:01:21 hermes kernel: [<c0138f95>] filemap_fault+0x19a/0x3b4
> Mar 19 17:01:21 hermes kernel: [<c013ec4e>] wakeup_kswapd+0x2d/0x70
> Mar 19 17:01:22 hermes kernel: [<c0141d80>] __do_fault+0x51/0x327
> Mar 19 17:01:22 hermes kernel: [<c0142485>] do_wp_page+0x42f/0x43b
> Mar 19 17:01:22 hermes kernel: [<c0143c3e>] handle_mm_fault+0x2aa/0x5b6
> Mar 19 17:01:22 hermes kernel: [<c011861d>] wake_up_new_task+0x77/0x7b
> Mar 19 17:01:22 hermes kernel: [<c0110f15>] do_page_fault+0x18d/0x530
> Mar 19 17:01:22 hermes kernel: [<c0323142>] sys_socketcall+0xeb/0x242
> Mar 19 17:01:22 hermes kernel: [<c015b799>] do_fcntl+0x1f8/0x27e
> Mar 19 17:01:22 hermes kernel: [<c0100a20>] sys_clone+0x36/0x3b
> Mar 19 17:01:22 hermes kernel: [<c0110d88>] do_page_fault+0x0/0x530
> Mar 19 17:01:22 hermes kernel: [<c03a1a72>] error_code+0x72/0x78
> Mar 19 17:01:22 hermes kernel: =======================
> Mar 19 17:01:23 hermes kernel: Mem-info:
> Mar 19 17:01:23 hermes kernel: DMA per-cpu:
> Mar 19 17:01:23 hermes kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
> Mar 19 17:01:23 hermes kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
> Mar 19 17:01:23 hermes kernel: Normal per-cpu:
> Mar 19 17:01:23 hermes kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 93 Cold: hi: 62, btch: 15 usd: 55
> Mar 19 17:01:23 hermes kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 120 Cold: hi: 62, btch: 15 usd: 55
> Mar 19 17:01:23 hermes kernel: HighMem per-cpu:
> Mar 19 17:01:23 hermes kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 46 Cold: hi: 62, btch: 15 usd: 60
> Mar 19 17:01:23 hermes kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 12 Cold: hi: 62, btch: 15 usd: 61
> Mar 19 17:01:23 hermes kernel: Active:150615 inactive:339150 dirty:0 writeback:0 unstable:0
> Mar 19 17:01:23 hermes kernel: free:12180 slab:6561 mapped:78 pagetables:2862 bounce:0
> Mar 19 17:01:23 hermes kernel: DMA free:8132kB min:68kB low:84kB high:100kB active:1872kB inactive:1256kB present:16256kB pages_scanned:5836 all_unreclaimable? yes
> Mar 19 17:01:23 hermes kernel: lowmem_reserve[]: 0 873 2016 2016
> Mar 19 17:01:23 hermes kernel: Normal free:40136kB min:3744kB low:4680kB high:5616kB active:400240kB inactive:398884kB present:894080kB pages_scanned:2546811 all_unreclaimable? yes
> Mar 19 17:01:23 hermes kernel: lowmem_reserve[]: 0 0 9143 9143
> Mar 19 17:01:23 hermes kernel: HighMem free:452kB min:512kB low:1736kB high:2964kB active:200720kB inactive:956076kB present:1170372kB pages_scanned:3284202 all_unreclaimable? yes
> Mar 19 17:01:23 hermes kernel: lowmem_reserve[]: 0 0 0 0
> Mar 19 17:01:23 hermes kernel: DMA: 49*4kB 44*8kB 40*16kB 37*32kB 22*64kB 6*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 8132kB
> Mar 19 17:01:23 hermes kernel: Normal: 9168*4kB 1*8kB 0*16kB 2*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 40136kB
> Mar 19 17:01:23 hermes kernel: HighMem: 11*4kB 5*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 452kB
> Mar 19 17:01:23 hermes kernel: Swap cache: add 550090, delete 550089, find 1023187/1039769, race 0+22
> Mar 19 17:01:23 hermes kernel: Free swap = 0kB
> Mar 19 17:01:23 hermes kernel: Total swap = 899064kB
> Mar 19 17:01:23 hermes kernel: Free swap: 0kB
> Mar 19 17:01:23 hermes kernel: 524272 pages of RAM
> Mar 19 17:01:23 hermes kernel: 294896 pages of HIGHMEM
> Mar 19 17:01:23 hermes kernel: 5542 reserved pages
> Mar 19 17:01:23 hermes kernel: 55226 pages shared
> Mar 19 17:01:23 hermes kernel: 1 pages swap cached
> Mar 19 17:01:23 hermes kernel: 0 pages dirty
> Mar 19 17:01:23 hermes kernel: 0 pages writeback
> Mar 19 17:01:23 hermes kernel: 78 pages mapped
> Mar 19 17:01:23 hermes kernel: 6561 pages slab
> Mar 19 17:01:23 hermes kernel: 2862 pages pagetables
> Mar 19 17:01:23 hermes kernel: Out of memory: kill process 5928 (apache2) score 40674 or a child
> Mar 19 17:01:23 hermes kernel: Killed process 5928 (apache2)
>
> Just wondering if anyone could help me with interpreting what is going
> on here. Does it look there's anything weird going on in kernel space
> based on this, or did something in userspace really just exhaust all
> memory?
lsmod, lspci, lsusb, .config, /proc/mounts would be good, please, I think I get
this too time to time on latest -mm (akpm).
Could somebody from -mm (memory) interpret the output, I must admit, I don't
understand too much by who all the memory is held. The inactive 339150 is memory
which was not touched for longer time got from __get_free_page* and might be
swapped out, right?
On Thu, 2008-03-20 at 11:18 +1030, Kevin Shanahan wrote:
> I'm running a vanilla 2.6.24.2 kernel on an x86 system with 2GB physical
> RAM. We had a problem yesterday where Samba and Apache both "died" and
> AFAIK, we weren't able to ssh into the system. One of our staff was able
> to reboot from the console though.
>
> So, I found the following information in the system log just before it
> went down:
>
> Mar 19 17:01:21 hermes kernel: smbd invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
> Mar 19 17:01:21 hermes kernel: Pid: 2757, comm: smbd Not tainted 2.6.24.2 #1
> Mar 19 17:01:21 hermes kernel: [<c0139841>] oom_kill_process+0x54/0xf8
> Mar 19 17:01:21 hermes kernel: [<c0139c3b>] out_of_memory+0x15f/0x194
> Mar 19 17:01:21 hermes kernel: [<c013b85e>] __alloc_pages+0x239/0x2c7
> Mar 19 17:01:21 hermes kernel: [<c0136de7>] sync_page+0x0/0x40
> Mar 19 17:01:21 hermes kernel: [<c03a0430>] preempt_schedule+0x3a/0x55
> Mar 19 17:01:21 hermes kernel: [<c013d283>] __do_page_cache_readahead+0xd5/0x1be
> Mar 19 17:01:21 hermes kernel: [<c013d6b2>] do_page_cache_readahead+0x49/0x53
> Mar 19 17:01:21 hermes kernel: [<c0138f95>] filemap_fault+0x19a/0x3b4
> Mar 19 17:01:21 hermes kernel: [<c013ec4e>] wakeup_kswapd+0x2d/0x70
> Mar 19 17:01:22 hermes kernel: [<c0141d80>] __do_fault+0x51/0x327
> Mar 19 17:01:22 hermes kernel: [<c0142485>] do_wp_page+0x42f/0x43b
> Mar 19 17:01:22 hermes kernel: [<c0143c3e>] handle_mm_fault+0x2aa/0x5b6
> Mar 19 17:01:22 hermes kernel: [<c011861d>] wake_up_new_task+0x77/0x7b
> Mar 19 17:01:22 hermes kernel: [<c0110f15>] do_page_fault+0x18d/0x530
> Mar 19 17:01:22 hermes kernel: [<c0323142>] sys_socketcall+0xeb/0x242
> Mar 19 17:01:22 hermes kernel: [<c015b799>] do_fcntl+0x1f8/0x27e
> Mar 19 17:01:22 hermes kernel: [<c0100a20>] sys_clone+0x36/0x3b
> Mar 19 17:01:22 hermes kernel: [<c0110d88>] do_page_fault+0x0/0x530
> Mar 19 17:01:22 hermes kernel: [<c03a1a72>] error_code+0x72/0x78
> Mar 19 17:01:22 hermes kernel: =======================
> Mar 19 17:01:23 hermes kernel: Mem-info:
> Mar 19 17:01:23 hermes kernel: DMA per-cpu:
> Mar 19 17:01:23 hermes kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
> Mar 19 17:01:23 hermes kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
> Mar 19 17:01:23 hermes kernel: Normal per-cpu:
> Mar 19 17:01:23 hermes kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 93 Cold: hi: 62, btch: 15 usd: 55
> Mar 19 17:01:23 hermes kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 120 Cold: hi: 62, btch: 15 usd: 55
> Mar 19 17:01:23 hermes kernel: HighMem per-cpu:
> Mar 19 17:01:23 hermes kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 46 Cold: hi: 62, btch: 15 usd: 60
> Mar 19 17:01:23 hermes kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 12 Cold: hi: 62, btch: 15 usd: 61
> Mar 19 17:01:23 hermes kernel: Active:150615 inactive:339150 dirty:0 writeback:0 unstable:0
> Mar 19 17:01:23 hermes kernel: free:12180 slab:6561 mapped:78 pagetables:2862 bounce:0
> Mar 19 17:01:23 hermes kernel: DMA free:8132kB min:68kB low:84kB high:100kB active:1872kB inactive:1256kB present:16256kB pages_scanned:5836 all_unreclaimable? yes
> Mar 19 17:01:23 hermes kernel: lowmem_reserve[]: 0 873 2016 2016
> Mar 19 17:01:23 hermes kernel: Normal free:40136kB min:3744kB low:4680kB high:5616kB active:400240kB inactive:398884kB present:894080kB pages_scanned:2546811 all_unreclaimable? yes
> Mar 19 17:01:23 hermes kernel: lowmem_reserve[]: 0 0 9143 9143
> Mar 19 17:01:23 hermes kernel: HighMem free:452kB min:512kB low:1736kB high:2964kB active:200720kB inactive:956076kB present:1170372kB pages_scanned:3284202 all_unreclaimable? yes
> Mar 19 17:01:23 hermes kernel: lowmem_reserve[]: 0 0 0 0
> Mar 19 17:01:23 hermes kernel: DMA: 49*4kB 44*8kB 40*16kB 37*32kB 22*64kB 6*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 8132kB
> Mar 19 17:01:23 hermes kernel: Normal: 9168*4kB 1*8kB 0*16kB 2*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 40136kB
> Mar 19 17:01:23 hermes kernel: HighMem: 11*4kB 5*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 452kB
> Mar 19 17:01:23 hermes kernel: Swap cache: add 550090, delete 550089, find 1023187/1039769, race 0+22
> Mar 19 17:01:23 hermes kernel: Free swap = 0kB
> Mar 19 17:01:23 hermes kernel: Total swap = 899064kB
> Mar 19 17:01:23 hermes kernel: Free swap: 0kB
> Mar 19 17:01:23 hermes kernel: 524272 pages of RAM
> Mar 19 17:01:23 hermes kernel: 294896 pages of HIGHMEM
> Mar 19 17:01:23 hermes kernel: 5542 reserved pages
> Mar 19 17:01:23 hermes kernel: 55226 pages shared
> Mar 19 17:01:23 hermes kernel: 1 pages swap cached
> Mar 19 17:01:23 hermes kernel: 0 pages dirty
> Mar 19 17:01:23 hermes kernel: 0 pages writeback
> Mar 19 17:01:23 hermes kernel: 78 pages mapped
> Mar 19 17:01:23 hermes kernel: 6561 pages slab
> Mar 19 17:01:23 hermes kernel: 2862 pages pagetables
> Mar 19 17:01:23 hermes kernel: Out of memory: kill process 5928 (apache2) score 40674 or a child
> Mar 19 17:01:23 hermes kernel: Killed process 5928 (apache2)
>
> Just wondering if anyone could help me with interpreting what is going
> on here. Does it look there's anything weird going on in kernel space
> based on this, or did something in userspace really just exhaust all
> memory?
>
> The best I can understand from the log, the problem the kernel detected
> was that HighMem fell below the low watermark. Also very strange to see
> that it says we have no free swap space - normally we only see a handful
> of pages from the whole ~1GB of swap being used, even after weeks of
> uptime.
Looks like you just ran out of memory, swap is filled up, memory is
pretty much used up, except for !HIGHMEM which even though it has enough
free memory does report all_unreclaimable=1. This suggests this is a
second OOM not the first, are you sure this is the first OOM on that
box?
On Thu, 2008-03-20 at 09:54 +0100, Peter Zijlstra wrote:
> Looks like you just ran out of memory, swap is filled up, memory is
> pretty much used up, except for !HIGHMEM which even though it has enough
> free memory does report all_unreclaimable=1. This suggests this is a
> second OOM not the first, are you sure this is the first OOM on that
> box?
Well, it's the first time since we rebooted on Feb 28th.
Ugh, looks like the reason we rebooted then was because of something
similar as well:
Feb 28 06:32:19 hermes kernel: postmaster invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
Feb 28 06:32:19 hermes kernel: Pid: 2505, comm: postmaster Not tainted 2.6.24.2 #1
Feb 28 06:32:19 hermes kernel: [<c0139841>] oom_kill_process+0x54/0xf8
Feb 28 06:32:19 hermes kernel: [<c0139c3b>] out_of_memory+0x15f/0x194
Feb 28 06:32:19 hermes kernel: [<c013b85e>] __alloc_pages+0x239/0x2c7
Feb 28 06:32:19 hermes kernel: [<c013d283>] __do_page_cache_readahead+0xd5/0x1be
Feb 28 06:32:19 hermes kernel: [<c013d6b2>] do_page_cache_readahead+0x49/0x53
Feb 28 06:32:19 hermes kernel: [<c0138f95>] filemap_fault+0x19a/0x3b4
Feb 28 06:32:19 hermes kernel: [<c0141d80>] __do_fault+0x51/0x327
Feb 28 06:32:19 hermes kernel: [<c0143c3e>] handle_mm_fault+0x2aa/0x5b6
Feb 28 06:32:19 hermes kernel: [<c0110f15>] do_page_fault+0x18d/0x530
Feb 28 06:32:19 hermes kernel: [<c0110d88>] do_page_fault+0x0/0x530
Feb 28 06:32:20 hermes kernel: [<c03a1a72>] error_code+0x72/0x78
Feb 28 06:32:20 hermes kernel: =======================
Feb 28 06:32:20 hermes kernel: Mem-info:
Feb 28 06:32:20 hermes kernel: DMA per-cpu:
Feb 28 06:32:20 hermes kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Feb 28 06:32:20 hermes kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Feb 28 06:32:20 hermes kernel: Normal per-cpu:
Feb 28 06:32:20 hermes kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 163 Cold: hi: 62, btch: 15 usd: 59
Feb 28 06:32:20 hermes kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 152 Cold: hi: 62, btch: 15 usd: 61
Feb 28 06:32:20 hermes kernel: HighMem per-cpu:
Feb 28 06:32:20 hermes kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 23 Cold: hi: 62, btch: 15 usd: 47
Feb 28 06:32:20 hermes kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 15 Cold: hi: 62, btch: 15 usd: 56
Feb 28 06:32:20 hermes kernel: Active:247172 inactive:246700 dirty:0 writeback:0 unstable:0
Feb 28 06:32:20 hermes kernel: free:12237 slab:5313 mapped:92 pagetables:2632 bounce:0
Feb 28 06:32:20 hermes kernel: DMA free:8128kB min:68kB low:84kB high:100kB active:1576kB inactive:1336kB present:16256kB pages_scanned:12856 all_unreclaimable? yes
Feb 28 06:32:20 hermes kernel: lowmem_reserve[]: 0 873 2016 2016
Feb 28 06:32:20 hermes kernel: Normal free:40316kB min:3744kB low:4680kB high:5616kB active:399640kB inactive:407172kB present:894080kB pages_scanned:4921459 all_unreclaimable? yes
Feb 28 06:32:20 hermes kernel: lowmem_reserve[]: 0 0 9143 9143
Feb 28 06:32:20 hermes kernel: HighMem free:504kB min:512kB low:1736kB high:2964kB active:587472kB inactive:578292kB present:1170372kB pages_scanned:4714529 all_unreclaimable? yes
Feb 28 06:32:20 hermes kernel: lowmem_reserve[]: 0 0 0 0
Feb 28 06:32:20 hermes kernel: DMA: 96*4kB 138*8kB 131*16kB 22*32kB 4*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 8128kB
Feb 28 06:32:20 hermes kernel: Normal: 9119*4kB 6*8kB 9*16kB 6*32kB 2*64kB 2*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 40316kB
Feb 28 06:32:20 hermes kernel: HighMem: 22*4kB 18*8kB 1*16kB 0*32kB 0*64kB 2*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 504kB
Feb 28 06:32:20 hermes kernel: Swap cache: add 304821, delete 304821, find 81588/89482, race 0+6
Feb 28 06:32:20 hermes kernel: Free swap = 0kB
Feb 28 06:32:20 hermes kernel: Total swap = 899064kB
Feb 28 06:32:20 hermes kernel: Free swap: 0kB
Feb 28 06:32:20 hermes kernel: 524272 pages of RAM
Feb 28 06:32:20 hermes kernel: 294896 pages of HIGHMEM
Feb 28 06:32:20 hermes kernel: 5542 reserved pages
Feb 28 06:32:20 hermes kernel: 49663 pages shared
Feb 28 06:32:20 hermes kernel: 0 pages swap cached
Feb 28 06:32:20 hermes kernel: 0 pages dirty
Feb 28 06:32:20 hermes kernel: 0 pages writeback
Feb 28 06:32:20 hermes kernel: 92 pages mapped
Feb 28 06:32:20 hermes kernel: 5313 pages slab
Feb 28 06:32:20 hermes kernel: 2632 pages pagetables
Feb 28 06:32:20 hermes kernel: Out of memory: kill process 24013 (apache2) score 22257 or a child
Feb 28 06:32:20 hermes kernel: Killed process 1904 (ntlm_auth)
Feb 28 06:32:20 hermes kernel: postmaster invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
Feb 28 06:32:20 hermes kernel: Pid: 2505, comm: postmaster Not tainted 2.6.24.2 #1
Feb 28 06:32:20 hermes kernel: [<c0139841>] oom_kill_process+0x54/0xf8
Feb 28 06:32:20 hermes kernel: [<c0139c3b>] out_of_memory+0x15f/0x194
Feb 28 06:32:20 hermes kernel: [<c013b85e>] __alloc_pages+0x239/0x2c7
Feb 28 06:32:20 hermes kernel: [<c013d283>] __do_page_cache_readahead+0xd5/0x1be
Feb 28 06:32:20 hermes kernel: [<c013d6b2>] do_page_cache_readahead+0x49/0x53
Feb 28 06:32:20 hermes kernel: [<c0138f95>] filemap_fault+0x19a/0x3b4
Feb 28 06:32:20 hermes kernel: [<c0141d80>] __do_fault+0x51/0x327
Feb 28 06:32:20 hermes kernel: [<c0143c3e>] handle_mm_fault+0x2aa/0x5b6
Feb 28 06:32:20 hermes kernel: [<c0110f15>] do_page_fault+0x18d/0x530
Feb 28 06:32:20 hermes kernel: [<c0110d88>] do_page_fault+0x0/0x530
Feb 28 06:32:20 hermes kernel: [<c03a1a72>] error_code+0x72/0x78
Feb 28 06:32:20 hermes kernel: =======================
Feb 28 06:32:20 hermes kernel: Mem-info:
Feb 28 06:32:20 hermes kernel: DMA per-cpu:
Feb 28 06:32:20 hermes kernel: CPU 0: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Feb 28 06:32:20 hermes kernel: CPU 1: Hot: hi: 0, btch: 1 usd: 0 Cold: hi: 0, btch: 1 usd: 0
Feb 28 06:32:20 hermes kernel: Normal per-cpu:
Feb 28 06:32:20 hermes kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 162 Cold: hi: 62, btch: 15 usd: 59
Feb 28 06:32:20 hermes kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 149 Cold: hi: 62, btch: 15 usd: 61
Feb 28 06:32:20 hermes kernel: HighMem per-cpu:
Feb 28 06:32:20 hermes kernel: CPU 0: Hot: hi: 186, btch: 31 usd: 23 Cold: hi: 62, btch: 15 usd: 47
Feb 28 06:32:20 hermes kernel: CPU 1: Hot: hi: 186, btch: 31 usd: 43 Cold: hi: 62, btch: 15 usd: 56
Feb 28 06:32:20 hermes kernel: Active:247230 inactive:246512 dirty:0 writeback:62 unstable:0
Feb 28 06:32:20 hermes kernel: free:12237 slab:5313 mapped:92 pagetables:2632 bounce:0
Feb 28 06:32:20 hermes kernel: DMA free:8128kB min:68kB low:84kB high:100kB active:1576kB inactive:1336kB present:16256kB pages_scanned:12856 all_unreclaimable? yes
Feb 28 06:32:20 hermes kernel: lowmem_reserve[]: 0 873 2016 2016
Feb 28 06:32:20 hermes kernel: Normal free:40316kB min:3744kB low:4680kB high:5616kB active:399512kB inactive:407172kB present:894080kB pages_scanned:4921491 all_unreclaimable? yes
Feb 28 06:32:20 hermes kernel: lowmem_reserve[]: 0 0 9143 9143
Feb 28 06:32:20 hermes kernel: HighMem free:504kB min:512kB low:1736kB high:2964kB active:587956kB inactive:577412kB present:1170372kB pages_scanned:4717047 all_unreclaimable? yes
Feb 28 06:32:20 hermes kernel: lowmem_reserve[]: 0 0 0 0
Feb 28 06:32:20 hermes kernel: DMA: 96*4kB 138*8kB 131*16kB 22*32kB 4*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 8128kB
Feb 28 06:32:20 hermes kernel: Normal: 9119*4kB 6*8kB 9*16kB 6*32kB 2*64kB 2*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 40316kB
Feb 28 06:32:20 hermes kernel: HighMem: 22*4kB 18*8kB 1*16kB 0*32kB 0*64kB 2*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 504kB
Feb 28 06:32:20 hermes kernel: Swap cache: add 304901, delete 304821, find 81588/89483, race 0+6
Feb 28 06:32:20 hermes kernel: Free swap = 0kB
Feb 28 06:32:20 hermes kernel: Total swap = 899064kB
Feb 28 06:32:20 hermes kernel: Free swap: 0kB
Feb 28 06:32:20 hermes kernel: 524272 pages of RAM
Feb 28 06:32:20 hermes kernel: 294896 pages of HIGHMEM
Feb 28 06:32:20 hermes kernel: 5542 reserved pages
Feb 28 06:32:20 hermes kernel: 49707 pages shared
Feb 28 06:32:20 hermes kernel: 80 pages swap cached
Feb 28 06:32:20 hermes kernel: 0 pages dirty
Feb 28 06:32:20 hermes kernel: 62 pages writeback
Feb 28 06:32:20 hermes kernel: 92 pages mapped
Feb 28 06:32:20 hermes kernel: 5313 pages slab
Feb 28 06:32:20 hermes kernel: 2632 pages pagetables
Feb 28 06:32:20 hermes kernel: Out of memory: kill process 24013 (apache2) score 22070 or a child
Feb 28 06:32:20 hermes kernel: Killed process 24013 (apache2)
Looks like the HighMem low watermark was hit again (though I've no idea
if that's actually significant).
Thanks,
Kevin.
On Thu, 2008-03-20 at 09:44 +0100, Jiri Slaby wrote:
> lsmod, lspci, lsusb, .config, /proc/mounts would be good, please, I think I get
> this too time to time on latest -mm (akpm).
>
> Could somebody from -mm (memory) interpret the output, I must admit, I don't
> understand too much by who all the memory is held. The inactive 339150 is memory
> which was not touched for longer time got from __get_free_page* and might be
> swapped out, right?
Thanks Jiri. Info suggested is below.
Cheers,
Kevin.
(# CONFIG_MODULES is not set)
hermes:~# lspci
00:00.0 Memory controller: nVidia Corporation CK804 Memory Controller (rev a3)
00:01.0 ISA bridge: nVidia Corporation CK804 ISA Bridge (rev a3)
00:01.1 SMBus: nVidia Corporation CK804 SMBus (rev a2)
00:02.0 USB Controller: nVidia Corporation CK804 USB Controller (rev a2)
00:02.1 USB Controller: nVidia Corporation CK804 USB Controller (rev a3)
00:04.0 Multimedia audio controller: nVidia Corporation CK804 AC'97 Audio Controller (rev a2)
00:06.0 IDE interface: nVidia Corporation CK804 IDE (rev f2)
00:07.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
00:08.0 IDE interface: nVidia Corporation CK804 Serial ATA Controller (rev f3)
00:09.0 PCI bridge: nVidia Corporation CK804 PCI Bridge (rev a2)
00:0a.0 Bridge: nVidia Corporation CK804 Ethernet Controller (rev a3)
00:0b.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:0c.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:0d.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:0e.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
00:18.0 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
00:18.1 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
00:18.2 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
03:00.0 PCI bridge: PLX Technology, Inc. PEX 8114 PCI Express-to-PCI/PCI-X Bridge (rev bb)
04:08.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev c1)
06:06.0 VGA compatible controller: Cirrus Logic GD 5430/40 [Alpine] (rev 47)
06:08.0 Mass storage controller: Promise Technology, Inc. PDC20318 (SATA150 TX4) (rev 02)
06:0b.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22/A IEEE-1394a-2000 Controller (PHY/Link)
hermes:~# lsusb
Bus 001 Device 001: ID 0000:0000
hermes:~# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 rw,noatime,nodiratime,errors=remount-ro,data=ordered 0 0
tmpfs /lib/init/rw tmpfs rw,nosuid 0 0
proc /proc proc rw,nosuid,nodev,noexec 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec 0 0
usbfs /proc/bus/usb usbfs rw,nosuid,nodev,noexec 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec 0 0
/dev/md0 /boot ext3 rw,noatime,nodiratime,errors=remount-ro,data=ordered 0 0
/dev/vg_raid_store/lv_data /srv/samba/local ext3 rw,noatime,nodiratime,errors=remount-ro,user_xattr,acl,data=ordered 0 0
/dev/vg_raid_store/lv_backup /srv/samba/backup ext3 rw,noatime,nodiratime,errors=remount-ro,user_xattr,acl,data=ordered 0 0
/dev/vg_raid_store/lv_home /home ext3 rw,noatime,nodiratime,errors=remount-ro,user_xattr,acl,data=ordered,usrquota 0 0
hermes:~# grep "=[y|m]" hermes.config
CONFIG_X86_32=y
CONFIG_X86=y
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_SEMAPHORE_SLEEPERS=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_QUICKLIST=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_DMI=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_ARCH_SUPPORTS_OPROFILE=y
CONFIG_ARCH_POPULATES_NODE_MAP=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_X86_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y
CONFIG_KTIME_SCALAR=y
CONFIG_EXPERIMENTAL=y
CONFIG_LOCK_KERNEL=y
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_FAIR_USER_SCHED=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_ANON_INODES=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_SLAB=y
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BLOCK=y
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_DEFAULT_CFQ=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_SMP=y
CONFIG_X86_PC=y
CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y
CONFIG_MK8=y
CONFIG_X86_GENERIC=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_SCHED_MC=y
CONFIG_PREEMPT=y
CONFIG_PREEMPT_BKL=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=y
CONFIG_VM86=y
CONFIG_MICROCODE=y
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
CONFIG_HIGHMEM4G=y
CONFIG_HIGHMEM=y
CONFIG_ARCH_FLATMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
CONFIG_SPARSEMEM_STATIC=y
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_HIGHPTE=y
CONFIG_MTRR=y
CONFIG_IRQBALANCE=y
CONFIG_SECCOMP=y
CONFIG_HZ_300=y
CONFIG_COMPAT_VDSO=y
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_PM=y
CONFIG_PM_LEGACY=y
CONFIG_SUSPEND_SMP_POSSIBLE=y
CONFIG_HIBERNATION_SMP_POSSIBLE=y
CONFIG_ACPI=y
CONFIG_ACPI_PROCFS=y
CONFIG_ACPI_PROCFS_POWER=y
CONFIG_ACPI_SYSFS_POWER=y
CONFIG_ACPI_PROC_EVENT=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_FAN=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_SYSTEM=y
CONFIG_X86_PM_TIMER=y
CONFIG_PCI=y
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
CONFIG_PCIEPORTBUS=y
CONFIG_PCIEAER=y
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
CONFIG_PCI_LEGACY=y
CONFIG_HT_IRQ=y
CONFIG_ISA_DMA_API=y
CONFIG_ISA=y
CONFIG_EISA=y
CONFIG_EISA_PCI_EISA=y
CONFIG_EISA_VIRTUAL_ROOT=y
CONFIG_EISA_NAMES=y
CONFIG_BINFMT_ELF=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
CONFIG_XFRM=y
CONFIG_NET_KEY=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_ASK_IP_FIB_HASH=y
CONFIG_IP_FIB_HASH=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_NET_IPIP=y
CONFIG_SYN_COOKIES=y
CONFIG_INET_TUNNEL=y
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
CONFIG_TCP_CONG_CUBIC=y
CONFIG_NETFILTER=y
CONFIG_BRIDGE_NETFILTER=y
CONFIG_NETFILTER_NETLINK=y
CONFIG_NETFILTER_XTABLES=y
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=y
CONFIG_NETFILTER_XT_TARGET_MARK=y
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=y
CONFIG_NETFILTER_XT_TARGET_TCPMSS=y
CONFIG_NETFILTER_XT_MATCH_COMMENT=y
CONFIG_NETFILTER_XT_MATCH_DCCP=y
CONFIG_NETFILTER_XT_MATCH_LENGTH=y
CONFIG_NETFILTER_XT_MATCH_LIMIT=y
CONFIG_NETFILTER_XT_MATCH_MAC=y
CONFIG_NETFILTER_XT_MATCH_MARK=y
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=y
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=y
CONFIG_NETFILTER_XT_MATCH_REALM=y
CONFIG_NETFILTER_XT_MATCH_SCTP=y
CONFIG_NETFILTER_XT_MATCH_STRING=y
CONFIG_NETFILTER_XT_MATCH_TCPMSS=y
CONFIG_IP_NF_QUEUE=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_MATCH_IPRANGE=y
CONFIG_IP_NF_MATCH_TOS=y
CONFIG_IP_NF_MATCH_RECENT=y
CONFIG_IP_NF_MATCH_ECN=y
CONFIG_IP_NF_MATCH_TTL=y
CONFIG_IP_NF_MATCH_OWNER=y
CONFIG_IP_NF_MATCH_ADDRTYPE=y
CONFIG_IP_NF_FILTER=y
CONFIG_IP_NF_TARGET_REJECT=y
CONFIG_IP_NF_TARGET_LOG=y
CONFIG_IP_NF_TARGET_ULOG=y
CONFIG_IP_NF_MANGLE=y
CONFIG_IP_NF_TARGET_TOS=y
CONFIG_IP_NF_TARGET_ECN=y
CONFIG_IP_NF_TARGET_TTL=y
CONFIG_BRIDGE_NF_EBTABLES=y
CONFIG_BRIDGE_EBT_BROUTE=y
CONFIG_BRIDGE_EBT_T_FILTER=y
CONFIG_BRIDGE_EBT_T_NAT=y
CONFIG_BRIDGE_EBT_802_3=y
CONFIG_BRIDGE_EBT_AMONG=y
CONFIG_BRIDGE_EBT_ARP=y
CONFIG_BRIDGE_EBT_IP=y
CONFIG_BRIDGE_EBT_LIMIT=y
CONFIG_BRIDGE_EBT_MARK=y
CONFIG_BRIDGE_EBT_PKTTYPE=y
CONFIG_BRIDGE_EBT_STP=y
CONFIG_BRIDGE_EBT_VLAN=y
CONFIG_BRIDGE_EBT_ARPREPLY=y
CONFIG_BRIDGE_EBT_DNAT=y
CONFIG_BRIDGE_EBT_MARK_T=y
CONFIG_BRIDGE_EBT_REDIRECT=y
CONFIG_BRIDGE_EBT_SNAT=y
CONFIG_BRIDGE_EBT_LOG=y
CONFIG_BRIDGE=y
CONFIG_VLAN_8021Q=y
CONFIG_LLC=y
CONFIG_NET_SCHED=y
CONFIG_NET_SCH_HTB=y
CONFIG_NET_SCH_PRIO=y
CONFIG_NET_SCH_RED=y
CONFIG_NET_SCH_SFQ=y
CONFIG_NET_SCH_TEQL=y
CONFIG_NET_SCH_TBF=y
CONFIG_NET_SCH_GRED=y
CONFIG_NET_SCH_DSMARK=y
CONFIG_NET_SCH_INGRESS=y
CONFIG_NET_CLS=y
CONFIG_NET_CLS_TCINDEX=y
CONFIG_NET_CLS_ROUTE4=y
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=y
CONFIG_NET_CLS_U32=y
CONFIG_NET_CLS_RSVP=y
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=y
CONFIG_NET_ACT_GACT=y
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=y
CONFIG_NET_ACT_IPT=y
CONFIG_NET_ACT_PEDIT=y
CONFIG_NET_CLS_POLICE=y
CONFIG_NET_SCH_FIFO=y
CONFIG_FIB_RULES=y
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_PARPORT=y
CONFIG_PARPORT_PC=y
CONFIG_PARPORT_1284=y
CONFIG_PNP=y
CONFIG_ISAPNP=y
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_FD=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_RAM=y
CONFIG_MISC_DEVICES=y
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDE_PROC_FS=y
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
CONFIG_IDEPCI_PCIBUS_ORDER=y
CONFIG_BLK_DEV_IDEDMA_PCI=y
CONFIG_BLK_DEV_AMD74XX=y
CONFIG_BLK_DEV_PDC202XX_OLD=y
CONFIG_PDC202XX_BURST=y
CONFIG_BLK_DEV_IDEDMA=y
CONFIG_IDE_ARCH_OBSOLETE_INIT=y
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_PROC_FS=y
CONFIG_BLK_DEV_SD=y
CONFIG_CHR_DEV_ST=y
CONFIG_CHR_DEV_SG=y
CONFIG_CHR_DEV_SCH=y
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_SPI_ATTRS=y
CONFIG_SCSI_SAS_ATTRS=y
CONFIG_SCSI_SAS_LIBSAS=y
CONFIG_SCSI_LOWLEVEL=y
CONFIG_SCSI_AACRAID=y
CONFIG_ATA=y
CONFIG_ATA_ACPI=y
CONFIG_SATA_NV=y
CONFIG_SATA_PROMISE=y
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_RAID1=y
CONFIG_MD_RAID456=y
CONFIG_MD_RAID5_RESHAPE=y
CONFIG_BLK_DEV_DM=y
CONFIG_DM_CRYPT=y
CONFIG_FUSION=y
CONFIG_FUSION_SPI=y
CONFIG_FUSION_CTL=y
CONFIG_NETDEVICES=y
CONFIG_TUN=y
CONFIG_PHYLIB=y
CONFIG_MARVELL_PHY=y
CONFIG_DAVICOM_PHY=y
CONFIG_QSEMI_PHY=y
CONFIG_LXT_PHY=y
CONFIG_CICADA_PHY=y
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
CONFIG_NET_PCI=y
CONFIG_FORCEDETH=y
CONFIG_NETDEV_1000=y
CONFIG_SKGE=y
CONFIG_SK98LIN=y
CONFIG_PLIP=y
CONFIG_PPP=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=y
CONFIG_PPP_DEFLATE=y
CONFIG_PPP_BSDCOMP=y
CONFIG_SLHC=y
CONFIG_INPUT=y
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_EVDEV=y
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_LIBPS2=y
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_SERIAL_8250=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_CORE=y
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_PRINTER=y
CONFIG_PPDEV=y
CONFIG_RTC=y
CONFIG_DEVPORT=y
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_ALGOBIT=y
CONFIG_POWER_SUPPLY=y
CONFIG_HWMON=y
CONFIG_SENSORS_K8TEMP=y
CONFIG_SSB_POSSIBLE=y
CONFIG_AGP=y
CONFIG_AGP_VIA=y
CONFIG_FB=y
CONFIG_FIRMWARE_EDID=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
CONFIG_FB_DEFERRED_IO=y
CONFIG_FB_VESA=y
CONFIG_VGA_CONSOLE=y
CONFIG_VIDEO_SELECT=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_LOGO=y
CONFIG_LOGO_LINUX_CLUT224=y
CONFIG_HID_SUPPORT=y
CONFIG_HID=y
CONFIG_USB_HID=y
CONFIG_USB_HIDDEV=y
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=y
CONFIG_USB_DEVICEFS=y
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_UHCI_HCD=y
CONFIG_USB_STORAGE=y
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
CONFIG_RTC_DRV_DS1672=y
CONFIG_RTC_DRV_RS5C372=y
CONFIG_RTC_DRV_X1205=y
CONFIG_RTC_DRV_PCF8563=y
CONFIG_RTC_DRV_CMOS=y
CONFIG_RTC_DRV_M48T86=y
CONFIG_DMADEVICES=y
CONFIG_DMIID=y
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_JBD=y
CONFIG_FS_MBCACHE=y
CONFIG_FS_POSIX_ACL=y
CONFIG_ROMFS_FS=y
CONFIG_INOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_QUOTA=y
CONFIG_QFMT_V2=y
CONFIG_QUOTACTL=y
CONFIG_DNOTIFY=y
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=y
CONFIG_UDF_NLS=y
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_MSDOS_PARTITION=y
CONFIG_NLS=y
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_CODEPAGE_737=y
CONFIG_NLS_CODEPAGE_775=y
CONFIG_NLS_CODEPAGE_850=y
CONFIG_NLS_CODEPAGE_852=y
CONFIG_NLS_CODEPAGE_855=y
CONFIG_NLS_CODEPAGE_857=y
CONFIG_NLS_CODEPAGE_860=y
CONFIG_NLS_CODEPAGE_861=y
CONFIG_NLS_CODEPAGE_862=y
CONFIG_NLS_CODEPAGE_863=y
CONFIG_NLS_CODEPAGE_864=y
CONFIG_NLS_CODEPAGE_865=y
CONFIG_NLS_CODEPAGE_866=y
CONFIG_NLS_CODEPAGE_869=y
CONFIG_NLS_CODEPAGE_936=y
CONFIG_NLS_CODEPAGE_950=y
CONFIG_NLS_CODEPAGE_932=y
CONFIG_NLS_CODEPAGE_949=y
CONFIG_NLS_CODEPAGE_874=y
CONFIG_NLS_ISO8859_8=y
CONFIG_NLS_CODEPAGE_1250=y
CONFIG_NLS_CODEPAGE_1251=y
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=y
CONFIG_NLS_ISO8859_2=y
CONFIG_NLS_ISO8859_3=y
CONFIG_NLS_ISO8859_4=y
CONFIG_NLS_ISO8859_5=y
CONFIG_NLS_ISO8859_6=y
CONFIG_NLS_ISO8859_7=y
CONFIG_NLS_ISO8859_9=y
CONFIG_NLS_ISO8859_13=y
CONFIG_NLS_ISO8859_14=y
CONFIG_NLS_ISO8859_15=y
CONFIG_NLS_KOI8_R=y
CONFIG_NLS_KOI8_U=y
CONFIG_NLS_UTF8=y
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_EARLY_PRINTK=y
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
CONFIG_DOUBLEFAULT=y
CONFIG_XOR_BLOCKS=y
CONFIG_ASYNC_CORE=y
CONFIG_ASYNC_MEMCPY=y
CONFIG_ASYNC_XOR=y
CONFIG_CRYPTO=y
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_CBC=y
CONFIG_CRYPTO_PCBC=y
CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_AES_586=y
CONFIG_CRYPTO_HW=y
CONFIG_BITREVERSE=y
CONFIG_CRC_CCITT=y
CONFIG_CRC32=y
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=y
CONFIG_TEXTSEARCH_BM=y
CONFIG_TEXTSEARCH_FSM=y
CONFIG_PLIST=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
On Thursday 20 March 2008 11:04, Kevin Shanahan wrote:
> Feb 28 06:32:20 hermes kernel: 5313 pages slab
> Feb 28 06:32:20 hermes kernel: 2632 pages pagetables
> Feb 28 06:32:20 hermes kernel: Out of memory: kill process 24013 (apache2) score 22070 or a child
> Feb 28 06:32:20 hermes kernel: Killed process 24013 (apache2)
>
> Looks like the HighMem low watermark was hit again (though I've no idea
> if that's actually significant).
I kno that people routinely forget to do that, but
at least for purely paranoid reasons you are better
to run all daemons with setrlimit'ed memory size.
At least all daemons which you cannot reasonably
trust to be 100% leak-free and to have bounded
peak memory usage.
This way, only leaking saemon will die (and you will easily see
which one) instead of entire system.
--
vda