Hello, list,
I seriously gets this, and dont know why.
This server have 2GB ram, and ~1.1G always free!
Anybody have an idea?
Thanks,
Janos
May 21 09:05:35 st-0003 kernel: swapper: page allocation failure. order:0,
mode:0x20
May 21 09:05:35 st-0003 kernel: <c013c6ac> __alloc_pages+0x274/0x286
<c014af1d> cache_alloc_refill+0x2a6/0x45c
May 21 09:05:35 st-0003 kernel: <c014b12b> __kmalloc+0x58/0x61 <c03d5bfc>
__alloc_skb+0x49/0xf5
May 21 09:05:35 st-0003 kernel: <f88336b1>
e1000_alloc_rx_buffers+0x5c/0x2e5 [e1000] <f88315de>
e1000_clean_rx_irq+0x4fd/0x50a [e1000]
May 21 09:05:35 st-0003 kernel: <f88342d2> e1000_intr+0x96/0xf4 [e1000]
<c0136927> handle_IRQ_event+0x20/0x4c
May 21 09:05:35 st-0003 kernel: <c01369cf> __do_IRQ+0x7c/0xcd <c0104a65>
do_IRQ+0x44/0x53
May 21 09:05:35 st-0003 kernel: <c01030ba> common_interrupt+0x1a/0x20
<c01016f5> mwait_idle+0x20/0x34
May 21 09:05:35 st-0003 kernel: <c01016c0> cpu_idle+0x5a/0x6f <c05966c9>
start_kernel+0x317/0x31d
May 21 09:05:35 st-0003 kernel: Mem-info:
May 21 09:05:35 st-0003 kernel: DMA per-cpu:
May 21 09:05:35 st-0003 kernel: cpu 0 hot: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: cpu 0 cold: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: cpu 1 hot: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: cpu 1 cold: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: DMA32 per-cpu: empty
May 21 09:05:35 st-0003 kernel: Normal per-cpu:
May 21 09:05:35 st-0003 kernel: cpu 0 hot: high 186, batch 31 used:30
May 21 09:05:35 st-0003 kernel: cpu 0 cold: high 62, batch 15 used:47
May 21 09:05:35 st-0003 kernel: cpu 1 hot: high 186, batch 31 used:51
May 21 09:05:35 st-0003 kernel: cpu 1 cold: high 62, batch 15 used:55
May 21 09:05:35 st-0003 kernel: HighMem per-cpu:
May 21 09:05:35 st-0003 kernel: cpu 0 hot: high 186, batch 31 used:98
May 21 09:05:35 st-0003 kernel: cpu 0 cold: high 62, batch 15 used:10
May 21 09:05:35 st-0003 kernel: cpu 1 hot: high 186, batch 31 used:183
May 21 09:05:35 st-0003 kernel: cpu 1 cold: high 62, batch 15 used:6
May 21 09:05:35 st-0003 kernel: Free pages: 1165384kB (1160424kB
HighMem)
May 21 09:05:35 st-0003 kernel: Active:3002 inactive:212594 dirty:35905
writeback:1 unstable:0 free:291346 slab:6782 mapped:929 pagetables:57
May 21 09:05:35 st-0003 kernel: DMA free:3548kB min:68kB low:84kB high:100kB
active:0kB inactive:5808kB present:16384kB pages_scanned:0
all_unreclaimable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 880 2031
May 21 09:05:35 st-0003 kernel: DMA32 free:0kB min:0kB low:0kB high:0kB
active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 880 2031
May 21 09:05:35 st-0003 kernel: Normal free:1412kB min:3756kB low:4692kB
high:5632kB active:8kB inactive:838708kB present:901120kB pages_scanned:0
all_unrecla
imable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 0 9215
May 21 09:05:35 st-0003 kernel: HighMem free:1160424kB min:512kB low:1740kB
high:2972kB active:12000kB inactive:5860kB present:1179584kB pages_scanned:0
all_u
nreclaimable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 0 0
May 21 09:05:35 st-0003 kernel: DMA: 1*4kB 1*8kB 1*16kB 0*32kB 1*64kB
1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3548kB
May 21 09:05:35 st-0003 kernel: DMA32: empty
May 21 09:05:35 st-0003 kernel: Normal: 1*4kB 0*8kB 0*16kB 0*32kB 0*64kB
1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1412kB
May 21 09:05:35 st-0003 kernel: HighMem: 142*4kB 34*8kB 18*16kB 8*32kB
6*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 282*4096kB = 1160424kB
May 21 09:05:35 st-0003 kernel: Free swap: 0kB
May 21 09:05:35 st-0003 kernel: 524272 pages of RAM
May 21 09:05:35 st-0003 kernel: 294896 pages of HIGHMEM
May 21 09:05:35 st-0003 kernel: 6010 reserved pages
May 21 09:05:35 st-0003 kernel: 213818 pages shared
May 21 09:05:35 st-0003 kernel: 0 pages swap cached
May 21 09:05:35 st-0003 kernel: 35905 pages dirty
May 21 09:05:35 st-0003 kernel: 1 pages writeback
May 21 09:05:35 st-0003 kernel: 929 pages mapped
May 21 09:05:35 st-0003 kernel: 6782 pages slab
May 21 09:05:35 st-0003 kernel: 57 pages pagetables
May 21 09:05:35 st-0003 kernel: swapper: page allocation failure. order:0,
mode:0x20
May 21 09:05:35 st-0003 kernel: <c013c6ac> __alloc_pages+0x274/0x286
<c014af1d> cache_alloc_refill+0x2a6/0x45c
May 21 09:05:35 st-0003 kernel: <c014b12b> __kmalloc+0x58/0x61 <c03d5bfc>
__alloc_skb+0x49/0xf5
May 21 09:05:35 st-0003 kernel: <f88336b1>
e1000_alloc_rx_buffers+0x5c/0x2e5 [e1000] <f8831570>
e1000_clean_rx_irq+0x48f/0x50a [e1000]
May 21 09:05:35 st-0003 kernel: <f88342d2> e1000_intr+0x96/0xf4 [e1000]
<c0136927> handle_IRQ_event+0x20/0x4c
May 21 09:05:35 st-0003 kernel: <c01369cf> __do_IRQ+0x7c/0xcd <c0104a65>
do_IRQ+0x44/0x53
May 21 09:05:35 st-0003 kernel: <c01030ba> common_interrupt+0x1a/0x20
<c01016f5> mwait_idle+0x20/0x34
May 21 09:05:35 st-0003 kernel: <c01016c0> cpu_idle+0x5a/0x6f <c05966c9>
start_kernel+0x317/0x31d
May 21 09:05:35 st-0003 kernel: Mem-info:
May 21 09:05:35 st-0003 kernel: DMA per-cpu:
May 21 09:05:35 st-0003 kernel: cpu 0 hot: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: cpu 0 cold: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: cpu 1 hot: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: cpu 1 cold: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: DMA32 per-cpu: empty
May 21 09:05:35 st-0003 kernel: Normal per-cpu:
May 21 09:05:35 st-0003 kernel: cpu 0 hot: high 186, batch 31 used:30
May 21 09:05:35 st-0003 kernel: cpu 0 cold: high 62, batch 15 used:47
May 21 09:05:35 st-0003 kernel: cpu 1 hot: high 186, batch 31 used:51
May 21 09:05:35 st-0003 kernel: cpu 1 cold: high 62, batch 15 used:55
May 21 09:05:35 st-0003 kernel: HighMem per-cpu:
May 21 09:05:35 st-0003 kernel: cpu 0 hot: high 186, batch 31 used:98
May 21 09:05:35 st-0003 kernel: cpu 0 cold: high 62, batch 15 used:10
May 21 09:05:35 st-0003 kernel: cpu 1 hot: high 186, batch 31 used:183
May 21 09:05:35 st-0003 kernel: cpu 1 cold: high 62, batch 15 used:6
May 21 09:05:35 st-0003 kernel: Free pages: 1165384kB (1160424kB
HighMem)
May 21 09:05:35 st-0003 kernel: Active:3002 inactive:212594 dirty:35905
writeback:1 unstable:0 free:291346 slab:6782 mapped:929 pagetables:57
May 21 09:05:35 st-0003 kernel: DMA free:3548kB min:68kB low:84kB high:100kB
active:0kB inactive:5808kB present:16384kB pages_scanned:0
all_unreclaimable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 880 2031
May 21 09:05:35 st-0003 kernel: DMA32 free:0kB min:0kB low:0kB high:0kB
active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 880 2031
May 21 09:05:35 st-0003 kernel: Normal free:1412kB min:3756kB low:4692kB
high:5632kB active:8kB inactive:838708kB present:901120kB pages_scanned:0
all_unrecla
imable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 0 9215
May 21 09:05:35 st-0003 kernel: HighMem free:1160424kB min:512kB low:1740kB
high:2972kB active:12000kB inactive:5860kB present:1179584kB pages_scanned:0
all_u
nreclaimable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 0 0
May 21 09:05:35 st-0003 kernel: DMA: 1*4kB 1*8kB 1*16kB 0*32kB 1*64kB
1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3548kB
May 21 09:05:35 st-0003 kernel: DMA32: empty
May 21 09:05:35 st-0003 kernel: Normal: 1*4kB 0*8kB 0*16kB 0*32kB 0*64kB
1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1412kB
May 21 09:05:35 st-0003 kernel: HighMem: 142*4kB 34*8kB 18*16kB 8*32kB
6*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 282*4096kB = 1160424kB
May 21 09:05:35 st-0003 kernel: Free swap: 0kB
May 21 09:05:35 st-0003 kernel: 524272 pages of RAM
May 21 09:05:35 st-0003 kernel: 294896 pages of HIGHMEM
May 21 09:05:35 st-0003 kernel: 6010 reserved pages
May 21 09:05:35 st-0003 kernel: 213818 pages shared
May 21 09:05:35 st-0003 kernel: 0 pages swap cached
May 21 09:05:35 st-0003 kernel: 35905 pages dirty
May 21 09:05:35 st-0003 kernel: 1 pages writeback
May 21 09:05:35 st-0003 kernel: 929 pages mapped
May 21 09:05:35 st-0003 kernel: 6782 pages slab
May 21 09:05:35 st-0003 kernel: 57 pages pagetables
May 21 09:05:35 st-0003 kernel: swapper: page allocation failure. order:0,
mode:0x20
May 21 09:05:35 st-0003 kernel: <c013c6ac> __alloc_pages+0x274/0x286
<c014af1d> cache_alloc_refill+0x2a6/0x45c
May 21 09:05:35 st-0003 kernel: <c014b12b> __kmalloc+0x58/0x61 <c03d5bfc>
__alloc_skb+0x49/0xf5
May 21 09:05:35 st-0003 kernel: <f88336b1>
e1000_alloc_rx_buffers+0x5c/0x2e5 [e1000] <f8831570>
e1000_clean_rx_irq+0x48f/0x50a [e1000]
May 21 09:05:35 st-0003 kernel: <f88342d2> e1000_intr+0x96/0xf4 [e1000]
<c0136927> handle_IRQ_event+0x20/0x4c
May 21 09:05:35 st-0003 kernel: <c01369cf> __do_IRQ+0x7c/0xcd <c0104a65>
do_IRQ+0x44/0x53
May 21 09:05:35 st-0003 kernel: <c01030ba> common_interrupt+0x1a/0x20
<c01016f5> mwait_idle+0x20/0x34
May 21 09:05:35 st-0003 kernel: <c01016c0> cpu_idle+0x5a/0x6f <c05966c9>
start_kernel+0x317/0x31d
May 21 09:05:35 st-0003 kernel: Mem-info:
May 21 09:05:35 st-0003 kernel: DMA per-cpu:
May 21 09:05:35 st-0003 kernel: cpu 0 hot: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: cpu 0 cold: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: cpu 1 hot: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: cpu 1 cold: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: DMA32 per-cpu: empty
May 21 09:05:35 st-0003 kernel: Normal per-cpu:
May 21 09:05:35 st-0003 kernel: cpu 0 hot: high 186, batch 31 used:30
May 21 09:05:35 st-0003 kernel: cpu 0 cold: high 62, batch 15 used:47
May 21 09:05:35 st-0003 kernel: cpu 1 hot: high 186, batch 31 used:51
May 21 09:05:35 st-0003 kernel: cpu 1 cold: high 62, batch 15 used:55
May 21 09:05:35 st-0003 kernel: HighMem per-cpu:
May 21 09:05:35 st-0003 kernel: cpu 0 hot: high 186, batch 31 used:98
May 21 09:05:35 st-0003 kernel: cpu 0 cold: high 62, batch 15 used:10
May 21 09:05:35 st-0003 kernel: cpu 1 hot: high 186, batch 31 used:183
May 21 09:05:35 st-0003 kernel: cpu 1 cold: high 62, batch 15 used:6
May 21 09:05:35 st-0003 kernel: Free pages: 1165384kB (1160424kB
HighMem)
May 21 09:05:35 st-0003 kernel: Active:3002 inactive:212594 dirty:35905
writeback:1 unstable:0 free:291346 slab:6782 mapped:929 pagetables:57
May 21 09:05:35 st-0003 kernel: DMA free:3548kB min:68kB low:84kB high:100kB
active:0kB inactive:5808kB present:16384kB pages_scanned:0
all_unreclaimable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 880 2031
May 21 09:05:35 st-0003 kernel: DMA32 free:0kB min:0kB low:0kB high:0kB
active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 880 2031
May 21 09:05:35 st-0003 kernel: Normal free:1412kB min:3756kB low:4692kB
high:5632kB active:8kB inactive:838708kB present:901120kB pages_scanned:0
all_unrecla
imable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 0 9215
May 21 09:05:35 st-0003 kernel: HighMem free:1160424kB min:512kB low:1740kB
high:2972kB active:12000kB inactive:5860kB present:1179584kB pages_scanned:0
all_u
nreclaimable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 0 0
May 21 09:05:35 st-0003 kernel: DMA: 1*4kB 1*8kB 1*16kB 0*32kB 1*64kB
1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3548kB
May 21 09:05:35 st-0003 kernel: DMA32: empty
May 21 09:05:35 st-0003 kernel: Normal: 1*4kB 0*8kB 0*16kB 0*32kB 0*64kB
1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1412kB
May 21 09:05:35 st-0003 kernel: HighMem: 142*4kB 34*8kB 18*16kB 8*32kB
6*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 282*4096kB = 1160424kB
May 21 09:05:35 st-0003 kernel: Free swap: 0kB
May 21 09:05:35 st-0003 kernel: 524272 pages of RAM
May 21 09:05:35 st-0003 kernel: 294896 pages of HIGHMEM
May 21 09:05:35 st-0003 kernel: 6010 reserved pages
May 21 09:05:35 st-0003 kernel: 213818 pages shared
May 21 09:05:35 st-0003 kernel: 0 pages swap cached
May 21 09:05:35 st-0003 kernel: 35905 pages dirty
May 21 09:05:35 st-0003 kernel: 1 pages writeback
May 21 09:05:35 st-0003 kernel: 929 pages mapped
May 21 09:05:35 st-0003 kernel: 6782 pages slab
May 21 09:05:35 st-0003 kernel: 57 pages pagetables
May 21 09:05:35 st-0003 kernel: swapper: page allocation failure. order:0,
mode:0x20
May 21 09:05:35 st-0003 kernel: <c013c6ac> __alloc_pages+0x274/0x286
<c014af1d> cache_alloc_refill+0x2a6/0x45c
May 21 09:05:35 st-0003 kernel: <c014b12b> __kmalloc+0x58/0x61 <c03d5bfc>
__alloc_skb+0x49/0xf5
May 21 09:05:35 st-0003 kernel: <f88336b1>
e1000_alloc_rx_buffers+0x5c/0x2e5 [e1000] <f8831570>
e1000_clean_rx_irq+0x48f/0x50a [e1000]
May 21 09:05:35 st-0003 kernel: <f88342d2> e1000_intr+0x96/0xf4 [e1000]
<c0136927> handle_IRQ_event+0x20/0x4c
May 21 09:05:35 st-0003 kernel: <c01369cf> __do_IRQ+0x7c/0xcd <c0104a65>
do_IRQ+0x44/0x53
May 21 09:05:35 st-0003 kernel: <c01030ba> common_interrupt+0x1a/0x20
<c01016f5> mwait_idle+0x20/0x34
May 21 09:05:35 st-0003 kernel: <c01016c0> cpu_idle+0x5a/0x6f <c05966c9>
start_kernel+0x317/0x31d
May 21 09:05:35 st-0003 kernel: Mem-info:
May 21 09:05:35 st-0003 kernel: DMA per-cpu:
May 21 09:05:35 st-0003 kernel: cpu 0 hot: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: cpu 0 cold: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: cpu 1 hot: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: cpu 1 cold: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: DMA32 per-cpu: empty
May 21 09:05:35 st-0003 kernel: Normal per-cpu:
May 21 09:05:35 st-0003 kernel: cpu 0 hot: high 186, batch 31 used:30
May 21 09:05:35 st-0003 kernel: cpu 0 cold: high 62, batch 15 used:47
May 21 09:05:35 st-0003 kernel: cpu 1 hot: high 186, batch 31 used:51
May 21 09:05:35 st-0003 kernel: cpu 1 cold: high 62, batch 15 used:55
May 21 09:05:35 st-0003 kernel: HighMem per-cpu:
May 21 09:05:35 st-0003 kernel: cpu 0 hot: high 186, batch 31 used:98
May 21 09:05:35 st-0003 kernel: cpu 0 cold: high 62, batch 15 used:10
May 21 09:05:35 st-0003 kernel: cpu 1 hot: high 186, batch 31 used:183
May 21 09:05:35 st-0003 kernel: cpu 1 cold: high 62, batch 15 used:6
May 21 09:05:35 st-0003 kernel: Free pages: 1165384kB (1160424kB
HighMem)
May 21 09:05:35 st-0003 kernel: Active:3002 inactive:212594 dirty:35905
writeback:1 unstable:0 free:291346 slab:6782 mapped:929 pagetables:57
May 21 09:05:35 st-0003 kernel: DMA free:3548kB min:68kB low:84kB high:100kB
active:0kB inactive:5808kB present:16384kB pages_scanned:0
all_unreclaimable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 880 2031
May 21 09:05:35 st-0003 kernel: DMA32 free:0kB min:0kB low:0kB high:0kB
active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 880 2031
May 21 09:05:35 st-0003 kernel: Normal free:1412kB min:3756kB low:4692kB
high:5632kB active:8kB inactive:838708kB present:901120kB pages_scanned:0
all_unrecla
imable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 0 9215
May 21 09:05:35 st-0003 kernel: HighMem free:1160424kB min:512kB low:1740kB
high:2972kB active:12000kB inactive:5860kB present:1179584kB pages_scanned:0
all_u
nreclaimable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 0 0
May 21 09:05:35 st-0003 kernel: DMA: 1*4kB 1*8kB 1*16kB 0*32kB 1*64kB
1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3548kB
May 21 09:05:35 st-0003 kernel: DMA32: empty
May 21 09:05:35 st-0003 kernel: Normal: 1*4kB 0*8kB 0*16kB 0*32kB 0*64kB
1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1412kB
May 21 09:05:35 st-0003 kernel: HighMem: 142*4kB 34*8kB 18*16kB 8*32kB
6*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 282*4096kB = 1160424kB
May 21 09:05:35 st-0003 kernel: Free swap: 0kB
May 21 09:05:35 st-0003 kernel: 524272 pages of RAM
May 21 09:05:35 st-0003 kernel: 294896 pages of HIGHMEM
May 21 09:05:35 st-0003 kernel: 6010 reserved pages
May 21 09:05:35 st-0003 kernel: 213818 pages shared
May 21 09:05:35 st-0003 kernel: 0 pages swap cached
May 21 09:05:35 st-0003 kernel: 35905 pages dirty
May 21 09:05:35 st-0003 kernel: 1 pages writeback
May 21 09:05:35 st-0003 kernel: 929 pages mapped
May 21 09:05:35 st-0003 kernel: 6782 pages slab
May 21 09:05:35 st-0003 kernel: 57 pages pagetables
May 21 09:05:35 st-0003 kernel: swapper: page allocation failure. order:0,
mode:0x20
May 21 09:05:35 st-0003 kernel: <c013c6ac> __alloc_pages+0x274/0x286
<c014af1d> cache_alloc_refill+0x2a6/0x45c
May 21 09:05:35 st-0003 kernel: <c014b12b> __kmalloc+0x58/0x61 <c03d5bfc>
__alloc_skb+0x49/0xf5
May 21 09:05:35 st-0003 kernel: <f88336b1>
e1000_alloc_rx_buffers+0x5c/0x2e5 [e1000] <f8831570>
e1000_clean_rx_irq+0x48f/0x50a [e1000]
May 21 09:05:35 st-0003 kernel: <f88342d2> e1000_intr+0x96/0xf4 [e1000]
<c0136927> handle_IRQ_event+0x20/0x4c
May 21 09:05:35 st-0003 kernel: <c01369cf> __do_IRQ+0x7c/0xcd <c0104a65>
do_IRQ+0x44/0x53
May 21 09:05:35 st-0003 kernel: <c01030ba> common_interrupt+0x1a/0x20
<c01016f5> mwait_idle+0x20/0x34
May 21 09:05:35 st-0003 kernel: <c01016c0> cpu_idle+0x5a/0x6f <c05966c9>
start_kernel+0x317/0x31d
May 21 09:05:35 st-0003 kernel: Mem-info:
May 21 09:05:35 st-0003 kernel: DMA per-cpu:
May 21 09:05:35 st-0003 kernel: cpu 0 hot: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: cpu 0 cold: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: cpu 1 hot: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: cpu 1 cold: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: DMA32 per-cpu: empty
May 21 09:05:35 st-0003 kernel: Normal per-cpu:
May 21 09:05:35 st-0003 kernel: cpu 0 hot: high 186, batch 31 used:30
May 21 09:05:35 st-0003 kernel: cpu 0 cold: high 62, batch 15 used:47
May 21 09:05:35 st-0003 kernel: cpu 1 hot: high 186, batch 31 used:51
May 21 09:05:35 st-0003 kernel: cpu 1 cold: high 62, batch 15 used:55
May 21 09:05:35 st-0003 kernel: HighMem per-cpu:
May 21 09:05:35 st-0003 kernel: cpu 0 hot: high 186, batch 31 used:98
May 21 09:05:35 st-0003 kernel: cpu 0 cold: high 62, batch 15 used:10
May 21 09:05:35 st-0003 kernel: cpu 1 hot: high 186, batch 31 used:183
May 21 09:05:35 st-0003 kernel: cpu 1 cold: high 62, batch 15 used:6
May 21 09:05:35 st-0003 kernel: Free pages: 1165384kB (1160424kB
HighMem)
May 21 09:05:35 st-0003 kernel: Active:3002 inactive:212594 dirty:35905
writeback:1 unstable:0 free:291346 slab:6782 mapped:929 pagetables:57
May 21 09:05:35 st-0003 kernel: DMA free:3548kB min:68kB low:84kB high:100kB
active:0kB inactive:5808kB present:16384kB pages_scanned:0
all_unreclaimable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 880 2031
May 21 09:05:35 st-0003 kernel: DMA32 free:0kB min:0kB low:0kB high:0kB
active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 880 2031
May 21 09:05:35 st-0003 kernel: Normal free:1412kB min:3756kB low:4692kB
high:5632kB active:8kB inactive:838708kB present:901120kB pages_scanned:0
all_unrecla
imable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 0 9215
May 21 09:05:35 st-0003 kernel: HighMem free:1160424kB min:512kB low:1740kB
high:2972kB active:12000kB inactive:5860kB present:1179584kB pages_scanned:0
all_u
nreclaimable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 0 0
May 21 09:05:35 st-0003 kernel: DMA: 1*4kB 1*8kB 1*16kB 0*32kB 1*64kB
1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3548kB
May 21 09:05:35 st-0003 kernel: DMA32: empty
May 21 09:05:35 st-0003 kernel: Normal: 1*4kB 0*8kB 0*16kB 0*32kB 0*64kB
1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1412kB
May 21 09:05:35 st-0003 kernel: HighMem: 142*4kB 34*8kB 18*16kB 8*32kB
6*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 282*4096kB = 1160424kB
May 21 09:05:35 st-0003 kernel: Free swap: 0kB
May 21 09:05:35 st-0003 kernel: 524272 pages of RAM
May 21 09:05:35 st-0003 kernel: 294896 pages of HIGHMEM
May 21 09:05:35 st-0003 kernel: 6010 reserved pages
May 21 09:05:35 st-0003 kernel: 213818 pages shared
May 21 09:05:35 st-0003 kernel: 0 pages swap cached
May 21 09:05:35 st-0003 kernel: 35905 pages dirty
May 21 09:05:35 st-0003 kernel: 1 pages writeback
May 21 09:05:35 st-0003 kernel: 929 pages mapped
May 21 09:05:35 st-0003 kernel: 6782 pages slab
May 21 09:05:35 st-0003 kernel: 57 pages pagetables
May 21 09:05:35 st-0003 kernel: swapper: page allocation failure. order:0,
mode:0x20
May 21 09:05:35 st-0003 kernel: <c013c6ac> __alloc_pages+0x274/0x286
<c014af1d> cache_alloc_refill+0x2a6/0x45c
May 21 09:05:35 st-0003 kernel: <c014b12b> __kmalloc+0x58/0x61 <c03d5bfc>
__alloc_skb+0x49/0xf5
May 21 09:05:35 st-0003 kernel: <f88336b1>
e1000_alloc_rx_buffers+0x5c/0x2e5 [e1000] <f8831570>
e1000_clean_rx_irq+0x48f/0x50a [e1000]
May 21 09:05:35 st-0003 kernel: <f88342d2> e1000_intr+0x96/0xf4 [e1000]
<c0136927> handle_IRQ_event+0x20/0x4c
May 21 09:05:35 st-0003 kernel: <c01369cf> __do_IRQ+0x7c/0xcd <c0104a65>
do_IRQ+0x44/0x53
May 21 09:05:35 st-0003 kernel: <c01030ba> common_interrupt+0x1a/0x20
<c01016f5> mwait_idle+0x20/0x34
May 21 09:05:35 st-0003 kernel: <c01016c0> cpu_idle+0x5a/0x6f <c05966c9>
start_kernel+0x317/0x31d
May 21 09:05:35 st-0003 kernel: Mem-info:
May 21 09:05:35 st-0003 kernel: DMA per-cpu:
May 21 09:05:35 st-0003 kernel: cpu 0 hot: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: cpu 0 cold: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: cpu 1 hot: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: cpu 1 cold: high 0, batch 1 used:0
May 21 09:05:35 st-0003 kernel: DMA32 per-cpu: empty
May 21 09:05:35 st-0003 kernel: Normal per-cpu:
May 21 09:05:35 st-0003 kernel: cpu 0 hot: high 186, batch 31 used:30
May 21 09:05:35 st-0003 kernel: cpu 0 cold: high 62, batch 15 used:47
May 21 09:05:35 st-0003 kernel: cpu 1 hot: high 186, batch 31 used:51
May 21 09:05:35 st-0003 kernel: cpu 1 cold: high 62, batch 15 used:55
May 21 09:05:35 st-0003 kernel: HighMem per-cpu:
May 21 09:05:35 st-0003 kernel: cpu 0 hot: high 186, batch 31 used:98
May 21 09:05:35 st-0003 kernel: cpu 0 cold: high 62, batch 15 used:10
May 21 09:05:35 st-0003 kernel: cpu 1 hot: high 186, batch 31 used:183
May 21 09:05:35 st-0003 kernel: cpu 1 cold: high 62, batch 15 used:6
May 21 09:05:35 st-0003 kernel: Free pages: 1165384kB (1160424kB
HighMem)
May 21 09:05:35 st-0003 kernel: Active:3002 inactive:212594 dirty:35905
writeback:1 unstable:0 free:291346 slab:6782 mapped:929 pagetables:57
May 21 09:05:35 st-0003 kernel: DMA free:3548kB min:68kB low:84kB high:100kB
active:0kB inactive:5808kB present:16384kB pages_scanned:0
all_unreclaimable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 880 2031
May 21 09:05:35 st-0003 kernel: DMA32 free:0kB min:0kB low:0kB high:0kB
active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 880 2031
May 21 09:05:35 st-0003 kernel: Normal free:1412kB min:3756kB low:4692kB
high:5632kB active:8kB inactive:838708kB present:901120kB pages_scanned:0
all_unrecla
imable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 0 9215
May 21 09:05:35 st-0003 kernel: HighMem free:1160424kB min:512kB low:1740kB
high:2972kB active:12000kB inactive:5860kB present:1179584kB pages_scanned:0
all_u
nreclaimable? no
May 21 09:05:35 st-0003 kernel: lowmem_reserve[]: 0 0 0 0
May 21 09:05:35 st-0003 kernel: DMA: 1*4kB 1*8kB 1*16kB 0*32kB 1*64kB
1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3548kB
May 21 09:05:35 st-0003 kernel: DMA32: empty
May 21 09:05:35 st-0003 kernel: Normal: 1*4kB 0*8kB 0*16kB 0*32kB 0*64kB
1*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 1412kB
May 21 09:05:35 st-0003 kernel: HighMem: 142*4kB 34*8kB 18*16kB 8*32kB
6*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 282*4096kB = 1160424kB
May 21 09:05:35 st-0003 kernel: Free swap: 0kB
May 21 09:05:35 st-0003 kernel: 524272 pages of RAM
May 21 09:05:35 st-0003 kernel: 294896 pages of HIGHMEM
May 21 09:05:35 st-0003 kernel: 6010 reserved pages
May 21 09:05:35 st-0003 kernel: 213818 pages shared
May 21 09:05:35 st-0003 kernel: 0 pages swap cached
May 21 09:05:35 st-0003 kernel: 35905 pages dirty
May 21 09:05:35 st-0003 kernel: 1 pages writeback
May 21 09:05:35 st-0003 kernel: 929 pages mapped
May 21 09:05:35 st-0003 kernel: 6782 pages slab
May 21 09:05:35 st-0003 kernel: 57 pages pagetables
On Sun, May 21, 2006 at 10:10:13AM +0200, Haar J?nos wrote:
> This server have 2GB ram, and ~1.1G always free!
> Anybody have an idea?
you ran out of lowmem?
what kernel is this and how do you trigger it?
----- Original Message -----
From: "Chris Wedgwood" <[email protected]>
To: "Haar J?nos" <[email protected]>
Cc: <[email protected]>
Sent: Sunday, May 21, 2006 10:16 AM
Subject: Re: swapper: page allocation failure.
> On Sun, May 21, 2006 at 10:10:13AM +0200, Haar J?nos wrote:
>
> > This server have 2GB ram, and ~1.1G always free!
> > Anybody have an idea?
>
> you ran out of lowmem?
>
> what kernel is this and how do you trigger it?
I have 4 disk nodes that acts as nbd-server.
The only processes:
PID TTY STAT TIME COMMAND
1 ? S 0:12 init [3]
2 ? SW 0:00 [migration/0]
3 ? SWN 0:19 [ksoftirqd/0]
4 ? SW 0:00 [migration/1]
5 ? SWN 0:03 [ksoftirqd/1]
6 ? SW< 0:00 [events/0]
7 ? SW< 0:00 [events/1]
8 ? SW< 0:00 [khelper]
9 ? SW< 0:00 [kthread]
12 ? SW< 0:07 \_ [kblockd/0]
13 ? SW< 0:02 \_ [kblockd/1]
14 ? SW< 0:00 \_ [kacpid]
101 ? SW< 0:00 \_ [khubd]
103 ? SW< 0:00 \_ [kseriod]
247 ? SW 12:09 \_ [pdflush]
248 ? SW 10:04 \_ [pdflush]
250 ? SW< 0:00 \_ [aio/0]
251 ? SW< 0:00 \_ [aio/1]
252 ? SW< 0:00 \_ [xfslogd/0]
253 ? SW< 0:00 \_ [xfslogd/1]
254 ? SW< 0:00 \_ [xfsdatad/0]
255 ? SW< 0:00 \_ [xfsdatad/1]
921 ? SW< 0:00 \_ [ata/0]
922 ? SW< 0:00 \_ [ata/1]
926 ? SW< 0:00 \_ [scsi_eh_0]
927 ? SW< 0:00 \_ [scsi_eh_1]
935 ? SW< 0:00 \_ [scsi_eh_2]
936 ? SW< 0:00 \_ [scsi_eh_3]
937 ? SW< 0:00 \_ [scsi_eh_4]
938 ? SW< 0:00 \_ [scsi_eh_5]
951 ? SW< 0:00 \_ [scsi_eh_6]
952 ? SW< 0:00 \_ [scsi_eh_7]
1045 ? SW< 0:00 \_ [exec-osm/0]
1047 ? SW< 0:00 \_ [exec-osm/1]
1088 ? SW< 0:00 \_ [kcryptd/0]
1089 ? SW< 0:00 \_ [kcryptd/1]
1090 ? SW< 0:00 \_ [kmpathd/0]
1091 ? SW< 0:00 \_ [kmpathd/1]
1092 ? SW< 0:00 \_ [kmirrord]
1093 ? SW< 0:14 \_ [kedac]
1111 ? SW< 219:01 \_ [md0_raid5]
1112 ? SW< 0:23 \_ [rpciod/0]
1113 ? SW< 0:00 \_ [rpciod/1]
249 ? SW 51:30 [kswapd0]
2532 ? S 0:05 syslogd -m 0
2536 ? S 0:00 klogd -x
2615 ? S 0:00 /usr/sbin/sshd
21484 ? S 0:00 \_ /usr/sbin/sshd
21486 pts/2 S 0:00 \_ -bash
22444 pts/2 R 0:00 \_ ps fax
2629 ? S 0:03 crond
2638 ? S 0:55 /bin/bwbar --text-file
/mnt/DY_SYSTEM/sysinfo/st4-eth0_o.php --png-file
/mnt/DY_SYSTEM/sysinfo/st4-eth0_o.png eth0 100 -x 300
2639 ? S 0:54 /bin/bwbar --text-file
/mnt/DY_SYSTEM/sysinfo/st4-eth0_i.php -i --png-file
/mnt/DY_SYSTEM/sysinfo/st4-eth0_i.png eth0 100 -x 300
2674 tty2 S 0:00 /sbin/mingetty tty2
2675 tty3 S 0:00 /sbin/mingetty tty3
2676 tty4 S 0:00 /sbin/mingetty tty4
2677 tty5 S 0:00 /sbin/mingetty tty5
2678 tty6 S 0:00 /sbin/mingetty tty6
2925 tty1 S 0:00 /sbin/mingetty tty1 --noclear
5167 ? SW 0:00 [lockd]
21290 ? S< 0:00 /usr/local/bin/nbd-server 1230 /dev/md0
21780 ? S< 0:38 \_ /usr/local/bin/nbd-server 1230 /dev/md0
The kernel : 2.6.17-rc3-git1, but if i have right, the 2.6.16.1 is the same
if i try to swich back. :-)
2 of 4 nodes frequently reboots without any error message.
I have turned on some debug option, and catch this message.
At this point i watch the free -l -s 1, and i can see, the nbd-server (or
kernel) didn't use the highmem for buffer cache, but why?
total used free shared buffers cached
Mem: 2073048 893360 1179688 0 829092 19820
Low: 893464 868352 25112 0 0 0
High: 1179584 25008 1154576 0 0 0
-/+ buffers/cache: 44448 2028600
Swap: 0 0 0
(min_free_kbytes = 16000 now.)
Cheers,
Janos
On Sun, May 21, 2006 at 10:37:00AM +0200, Haar J?nos wrote:
> total used free shared buffers cached
> Mem: 2073048 893360 1179688 0 829092 19820
> Low: 893464 868352 25112 0 0 0
> High: 1179584 25008 1154576 0 0 0
> -/+ buffers/cache: 44448 2028600
> Swap: 0 0 0
looks bad for lowmem
what does /proc/meminfo say?
what about the output from "slabtop -sc" ?
----- Original Message -----
From: "Chris Wedgwood" <[email protected]>
To: "Haar J?nos" <[email protected]>
Cc: <[email protected]>
Sent: Sunday, May 21, 2006 10:47 AM
Subject: Re: swapper: page allocation failure.
> On Sun, May 21, 2006 at 10:37:00AM +0200, Haar J?nos wrote:
>
> > total used free shared buffers
cached
> > Mem: 2073048 893360 1179688 0 829092
19820
> > Low: 893464 868352 25112 0 0
0
> > High: 1179584 25008 1154576 0 0
0
> > -/+ buffers/cache: 44448 2028600
> > Swap: 0 0 0
>
> looks bad for lowmem
>
> what does /proc/meminfo say?
[root@st-0001 vm]# cat /proc/meminfo
MemTotal: 2073048 kB
MemFree: 1179376 kB
Buffers: 829764 kB
Cached: 19896 kB
SwapCached: 0 kB
Active: 15604 kB
Inactive: 837636 kB
HighTotal: 1179584 kB
HighFree: 1154736 kB
LowTotal: 893464 kB
LowFree: 24640 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 21352 kB
Writeback: 0 kB
Mapped: 7000 kB
Slab: 22612 kB
CommitLimit: 1036524 kB
Committed_AS: 10968 kB
PageTables: 284 kB
VmallocTotal: 114680 kB
VmallocUsed: 308 kB
VmallocChunk: 114036 kB
>
> what about the output from "slabtop -sc" ?
Not installed.
Wich package or where can i find the source? (i use redhat 9.0)
Cheers,
Janos
On Sun, May 21, 2006 at 11:03:33AM +0200, Haar J?nos wrote:
> MemTotal: 2073048 kB
> MemFree: 1179376 kB
fine
> Buffers: 829764 kB
ok
> Cached: 19896 kB
> SwapCached: 0 kB
> Active: 15604 kB
> Inactive: 837636 kB
hrm
> HighTotal: 1179584 kB
> HighFree: 1154736 kB
krm
> LowTotal: 893464 kB
> LowFree: 24640 kB
bad
> SwapTotal: 0 kB
> SwapFree: 0 kB
ok
> Dirty: 21352 kB
ok
> Writeback: 0 kB
> Mapped: 7000 kB
> Slab: 22612 kB
ok
you have very little low
> Not installed.
urgh
> Wich package or where can i find the source? (i use redhat 9.0)
google i guess, i have very little idea how to drive RH to be honest
anyhow, it's not the slab
something is eating/using/leaking all your lowmemory
what kernel version is this?
how long has the machine been up?
do you see it get worse over time?
----- Original Message -----
From: "Chris Wedgwood" <[email protected]>
To: "Haar J?nos" <[email protected]>
Cc: <[email protected]>
Sent: Sunday, May 21, 2006 10:47 AM
Subject: Re: swapper: page allocation failure.
> On Sun, May 21, 2006 at 10:37:00AM +0200, Haar J?nos wrote:
>
> > total used free shared buffers
cached
> > Mem: 2073048 893360 1179688 0 829092
19820
> > Low: 893464 868352 25112 0 0
0
> > High: 1179584 25008 1154576 0 0
0
> > -/+ buffers/cache: 44448 2028600
> > Swap: 0 0 0
>
> looks bad for lowmem
>
> what does /proc/meminfo say?
>
> what about the output from "slabtop -sc" ?
Here comes:
Active / Total Objects (% used) : 243896 / 258956 (94.2%)
Active / Total Slabs (% used) : 6243 / 6261 (99.7%)
Active / Total Caches (% used) : 76 / 151 (50.3%)
Active / Total Size (% used) : 20769.86K / 23348.68K (89.0%)
Minimum / Average / Maximum Object : 0.01K / 0.09K / 128.00K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
208872 206768 98% 0.05K 2901 72 11604K buffer_head
4960 4051 81% 0.68K 992 5 3968K nfs_inode_cache
9996 6478 64% 0.27K 714 14 2856K radix_tree_node
566 530 93% 2.00K 283 2 1132K size-2048
4680 4014 85% 0.12K 156 30 624K dentry_cache
1430 1409 98% 0.39K 143 10 572K inode_cache
65 65 100% 8.00K 65 1 520K size-8192
258 256 99% 1.32K 86 3 344K raid5/md0
5460 5435 99% 0.04K 65 84 260K sysfs_dir_cache
945 882 93% 0.25K 63 15 252K size-256
456 438 96% 0.50K 57 8 228K size-512
2891 2514 86% 0.06K 49 59 196K size-64
675 474 70% 0.25K 45 15 180K skbuff_head_cache
176 165 93% 1.00K 44 4 176K size-1024
1320 744 56% 0.12K 44 30 176K bio
42 37 88% 4.00K 42 1 168K size-4096
1170 1073 91% 0.12K 39 30 156K size-128
351 316 90% 0.40K 39 9 156K proc_inode_cache
4068 3298 81% 0.03K 36 113 144K size-32
660 160 24% 0.19K 33 20 132K filp
84 64 76% 1.30K 28 3 112K task_struct
1232 441 35% 0.09K 28 44 112K vm_area_struct
69 63 91% 1.31K 23 3 92K sighand_cache
20 20 100% 4.00K 20 1 80K pgd
396 244 61% 0.17K 18 22 72K blkdev_requests
1 1 100% 64.00K 1 1 64K size-65536
160 153 95% 0.38K 16 10 64K skbuff_fclone_cache
32 32 100% 2.00K 16 2 64K sgpool-128
15 3 20% 4.00K 15 1 60K names_cache
1196 1138 95% 0.04K 13 92 52K acpi_operand
2639 640 24% 0.02K 13 203 52K biovec-1
12 11 91% 3.00K 6 2 48K biovec-(256)
180 162 90% 0.25K 12 15 48K sgpool-16
96 82 85% 0.50K 12 8 48K sgpool-32
110 64 58% 0.38K 11 10 44K signal_cache
40 37 92% 0.97K 10 4 40K blkdev_queue
32 32 100% 1.00K 8 4 32K sgpool-64
80 80 100% 0.38K 8 10 32K scsi_cmd_cache
413 263 63% 0.06K 7 59 28K as_arq
180 150 83% 0.12K 6 30 24K kmem_cache
54 20 37% 0.44K 6 9 24K mm_struct
15 15 100% 1.50K 3 5 24K biovec-128
180 180 100% 0.12K 6 30 24K sgpool-8
42 36 85% 0.50K 6 7 24K nfs_write_data
505 64 12% 0.04K 5 101 20K pid
50 21 42% 0.38K 5 10 20K files_cache
145 112 77% 0.13K 5 29 20K idr_layer_cache
35 26 74% 0.56K 5 7 20K bdev_cache
35 23 65% 0.50K 5 7 20K sock_inode_cache
25 23 92% 0.75K 5 5 20K biovec-64
35 32 91% 0.50K 5 7 20K nfs_read_data
Cheers,
Janos
----- Original Message -----
From: "Chris Wedgwood" <[email protected]>
To: "Haar J?nos" <[email protected]>
Cc: <[email protected]>
Sent: Sunday, May 21, 2006 11:10 AM
Subject: Re: swapper: page allocation failure.
> On Sun, May 21, 2006 at 11:03:33AM +0200, Haar J?nos wrote:
>
> > MemTotal: 2073048 kB
> > MemFree: 1179376 kB
>
> fine
>
> > Buffers: 829764 kB
>
> ok
>
> > Cached: 19896 kB
> > SwapCached: 0 kB
> > Active: 15604 kB
>
> > Inactive: 837636 kB
>
> hrm
>
> > HighTotal: 1179584 kB
> > HighFree: 1154736 kB
>
> krm
>
> > LowTotal: 893464 kB
> > LowFree: 24640 kB
>
> bad
>
> > SwapTotal: 0 kB
> > SwapFree: 0 kB
>
> ok
>
> > Dirty: 21352 kB
>
> ok
>
> > Writeback: 0 kB
> > Mapped: 7000 kB
> > Slab: 22612 kB
>
> ok
>
>
>
>
> you have very little low
>
>
> > Not installed.
>
> urgh
>
> > Wich package or where can i find the source? (i use redhat 9.0)
>
> google i guess, i have very little idea how to drive RH to be honest
>
> anyhow, it's not the slab
I found it already, thanks.
>
>
> something is eating/using/leaking all your lowmemory
>
>
> what kernel version is this?
[root@st-0001 /]# uname -a
Linux st-0001 2.6.17-rc3-git1 #2 SMP Sun May 21 01:12:22 CEST 2006 i686 i686
i386 GNU/Linux
> how long has the machine been up?
[root@st-0001 /]# uptime
11:22:10 up 2:52, 1 user, load average: 0.35, 0.42, 0.43
> do you see it get worse over time?
No.
This is a simple disk node.
It serves the md0 array, and uses mem for buffering-caching.
If it reboots, fill the memory on the first couple of minutes, and stay on
full, but this is relatively good.
But 2 question is remaining:
1. why don't use highmem for caching?
2. why can not allocate enough lowmem from shared-buffer for the e1000
driver if it needs some memory?
Cheers,
Janos
>
On Sun, May 21, 2006 at 11:31:12AM +0200, Haar J?nos wrote:
> 1. why don't use highmem for caching?
> 2. why can not allocate enough lowmem from shared-buffer for the e1000
> driver if it needs some memory?
highmem can't be used as freely as lowmem, it has additional
complexity and a slight over head that makes it hard or impossible to
use in many places
On Sun, May 21, 2006 at 11:31:12AM +0200, Haar J?nos wrote:
> [root@st-0001 /]# uname -a
> Linux st-0001 2.6.17-rc3-git1 #2 SMP Sun May 21 01:12:22 CEST 2006 i686 i686 i386 GNU/Linux
did earlier kernels work OK?
> This is a simple disk node.
> It serves the md0 array, and uses mem for buffering-caching.
odd, i looks like you've leaked alot of lowmem but i can't think why
i've got major (induced) brain-fog right now so i'll have to think
about it tomorrow sorry
----- Original Message -----
From: "Chris Wedgwood" <[email protected]>
To: "Haar J?nos" <[email protected]>
Cc: <[email protected]>
Sent: Sunday, May 21, 2006 12:24 PM
Subject: Re: swapper: page allocation failure.
> On Sun, May 21, 2006 at 11:31:12AM +0200, Haar J?nos wrote:
>
> > 1. why don't use highmem for caching?
> > 2. why can not allocate enough lowmem from shared-buffer for the e1000
> > driver if it needs some memory?
>
> highmem can't be used as freely as lowmem, it has additional
> complexity and a slight over head that makes it hard or impossible to
> use in many places
OK, i understand this, but buffer-cache is an optional thing, and looks like
really simple (for me), and i cannot understand why cannot use exactly for
buffering the highmem...
On my concentrator the nbd-client uses ~ 3.5GB of ram for PAGE-cache.
The buffer cache why can not use the highmem?
Whats the difference?
Cheers,
Janos
----- Original Message -----
From: "Chris Wedgwood" <[email protected]>
To: "Haar J?nos" <[email protected]>
Cc: <[email protected]>
Sent: Sunday, May 21, 2006 12:26 PM
Subject: Re: swapper: page allocation failure.
> On Sun, May 21, 2006 at 11:31:12AM +0200, Haar J?nos wrote:
>
> > [root@st-0001 /]# uname -a
> > Linux st-0001 2.6.17-rc3-git1 #2 SMP Sun May 21 01:12:22 CEST 2006 i686
i686 i386 GNU/Linux
>
> did earlier kernels work OK?
No, the my reboot problem starts about january, but my tracking progress is
slow because this is the first error message.
(On january i have upgrade the nodes, and few things are changed.)
>
> > This is a simple disk node.
> > It serves the md0 array, and uses mem for buffering-caching.
>
> odd, i looks like you've leaked alot of lowmem but i can't think why
>
> i've got major (induced) brain-fog right now so i'll have to think
> about it tomorrow sorry
OK, thanks.
Cheers,
Janos
Haar J?nos <[email protected]> wrote:
>
> I seriously gets this, and dont know why.
> This server have 2GB ram, and ~1.1G always free!
> Anybody have an idea?
>
> Thanks,
> Janos
>
> May 21 09:05:35 st-0003 kernel: swapper: page allocation failure. order:0,
> mode:0x20
> May 21 09:05:35 st-0003 kernel: <c013c6ac> __alloc_pages+0x274/0x286
> <c014af1d> cache_alloc_refill+0x2a6/0x45c
> May 21 09:05:35 st-0003 kernel: <c014b12b> __kmalloc+0x58/0x61 <c03d5bfc>
> __alloc_skb+0x49/0xf5
> May 21 09:05:35 st-0003 kernel: <f88336b1>
> e1000_alloc_rx_buffers+0x5c/0x2e5 [e1000] <f88315de>
e1000 gobbled up all your lowmem memory from interrupt context.
Increasing /proc/sys/vm/min_free_kbytes will help.
Chris Wedgwood wrote:
> On Sun, May 21, 2006 at 11:31:12AM +0200, Haar J?nos wrote:
>
>
>>[root@st-0001 /]# uname -a
>>Linux st-0001 2.6.17-rc3-git1 #2 SMP Sun May 21 01:12:22 CEST 2006 i686 i686 i386 GNU/Linux
>
>
> did earlier kernels work OK?
>
>
>>This is a simple disk node.
>>It serves the md0 array, and uses mem for buffering-caching.
>
>
> odd, i looks like you've leaked alot of lowmem but i can't think why
>
> i've got major (induced) brain-fog right now so i'll have to think
> about it tomorrow sorry
The buffers are buffercache rather than the usual pagecache; due to
nbd I guess. Buffercache cannot be satisfied by highmem.
This would be a relatively uncommon setup, which explains why it
isn't working 100%. I don't know of any reason why reclaim speed
should be worse for buffercache, however one notable thing will be
that zone_normal's lowmem reserve that is untouchable by pagecache
will be eaten by buffercache...
Anyway, increasing /proc/sys/vm/min_free_kbytes should help. Janos,
perhaps you could try doubling it and see how you go?
--
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com
----- Original Message -----
From: "Nick Piggin" <[email protected]>
To: "Chris Wedgwood" <[email protected]>
Cc: "Haar J?nos" <[email protected]>; <[email protected]>
Sent: Sunday, May 21, 2006 2:01 PM
Subject: Re: swapper: page allocation failure.
> Chris Wedgwood wrote:
> > On Sun, May 21, 2006 at 11:31:12AM +0200, Haar J?nos wrote:
> >
> >
> >>[root@st-0001 /]# uname -a
> >>Linux st-0001 2.6.17-rc3-git1 #2 SMP Sun May 21 01:12:22 CEST 2006 i686
i686 i386 GNU/Linux
> >
> >
> > did earlier kernels work OK?
> >
> >
> >>This is a simple disk node.
> >>It serves the md0 array, and uses mem for buffering-caching.
> >
> >
> > odd, i looks like you've leaked alot of lowmem but i can't think why
> >
> > i've got major (induced) brain-fog right now so i'll have to think
> > about it tomorrow sorry
>
> The buffers are buffercache rather than the usual pagecache; due to
> nbd I guess. Buffercache cannot be satisfied by highmem.
>
> This would be a relatively uncommon setup, which explains why it
> isn't working 100%. I don't know of any reason why reclaim speed
> should be worse for buffercache, however one notable thing will be
> that zone_normal's lowmem reserve that is untouchable by pagecache
> will be eaten by buffercache...
>
> Anyway, increasing /proc/sys/vm/min_free_kbytes should help. Janos,
> perhaps you could try doubling it and see how you go?
I did it allready, and it looks like solves the problem.
Yesterday i have more than 6 random reboots, and after i set from 3800 to
16000 the min free limit, i have none at this point. :-)
15:51:45 up 7:21, 1 user, load average: 0.85, 0.79, 0.67
Anyway, i interested about cache/buffer mechanism, because i have some
performance problems too, and i can see, these systems wastes the half of
memory instead of speeds up the operation.
Thanks,
Janos
>
> --
> SUSE Labs, Novell Inc.
> Send instant messages to your online friends http://au.messenger.yahoo.com
----- Original Message -----
From: "Andrew Morton" <[email protected]>
To: "Haar J?nos" <[email protected]>
Cc: <[email protected]>
Sent: Sunday, May 21, 2006 2:03 PM
Subject: Re: swapper: page allocation failure.
> Haar J?nos <[email protected]> wrote:
> >
> > I seriously gets this, and dont know why.
> > This server have 2GB ram, and ~1.1G always free!
> > Anybody have an idea?
> >
> > Thanks,
> > Janos
> >
> > May 21 09:05:35 st-0003 kernel: swapper: page allocation failure.
order:0,
> > mode:0x20
> > May 21 09:05:35 st-0003 kernel: <c013c6ac> __alloc_pages+0x274/0x286
> > <c014af1d> cache_alloc_refill+0x2a6/0x45c
> > May 21 09:05:35 st-0003 kernel: <c014b12b> __kmalloc+0x58/0x61
<c03d5bfc>
> > __alloc_skb+0x49/0xf5
> > May 21 09:05:35 st-0003 kernel: <f88336b1>
> > e1000_alloc_rx_buffers+0x5c/0x2e5 [e1000] <f88315de>
>
> e1000 gobbled up all your lowmem memory from interrupt context.
>
> Increasing /proc/sys/vm/min_free_kbytes will help.
Yes, i know. ;-)
Thanks,
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
Haar J?nos wrote:
> I did it allready, and it looks like solves the problem.
> Yesterday i have more than 6 random reboots, and after i set from 3800 to
> 16000 the min free limit, i have none at this point. :-)
>
> 15:51:45 up 7:21, 1 user, load average: 0.85, 0.79, 0.67
Oh that's good. It's sad that you had random reboots though :(
>
> Anyway, i interested about cache/buffer mechanism, because i have some
> performance problems too, and i can see, these systems wastes the half of
> memory instead of speeds up the operation.
Yeah, as I said, block device's pagecache (aka buffercache) can't
use highmem. If nbd can export regular files as block devices, or
you use loop devices from regular files, that might help (or slow
things down :P).
--
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com
----- Original Message -----
From: "Nick Piggin" <[email protected]>
To: "Haar J?nos" <[email protected]>
Cc: "Chris Wedgwood" <[email protected]>; <[email protected]>
Sent: Monday, May 22, 2006 2:09 AM
Subject: Re: swapper: page allocation failure.
> Haar J?nos wrote:
>
> > I did it allready, and it looks like solves the problem.
> > Yesterday i have more than 6 random reboots, and after i set from 3800
to
> > 16000 the min free limit, i have none at this point. :-)
> >
> > 15:51:45 up 7:21, 1 user, load average: 0.85, 0.79, 0.67
>
> Oh that's good. It's sad that you had random reboots though :(
09:12:44 up 1 day, 43 min, 1 user, load average: 0.48, 0.44, 0.42
It really fixes this issue. :-)
Thanks to you all!
>
> >
> > Anyway, i interested about cache/buffer mechanism, because i have some
> > performance problems too, and i can see, these systems wastes the half
of
> > memory instead of speeds up the operation.
>
> Yeah, as I said, block device's pagecache (aka buffercache) can't
> use highmem. If nbd can export regular files as block devices, or
> you use loop devices from regular files, that might help (or slow
> things down :P).
Hmm.
That sounds bad.
I think, if highmem is unreachable some times that makes lowmem more
valuable!
The kernel needs to keep (reserve) it free as much as possible.
The buffer-cache is an unimportant thing next to keeping lowmem free, but it
is blocks the performance and wastes the systems resources!
It is possible any workaround?
The NBD is finally fixed, and production ready.
The big systems should use it, because it is easy.
Additionally the big system usually needs the maximum of performance.....
(+ the good cache/buffer extends the hardware lifetime.)
I think it is important to be changed.
(e.g. i use 4 nodes, 4x 3.3TB to use it as one big blockdev.)
Cheers,
Janos
>
> --
> SUSE Labs, Novell Inc.
> Send instant messages to your online friends http://au.messenger.yahoo.com
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
On Monday 22 May 2006 17:41, Haar J?nos wrote:
> ----- Original Message -----
> From: "Nick Piggin" <[email protected]>
> > Yeah, as I said, block device's pagecache (aka buffercache) can't
> > use highmem. If nbd can export regular files as block devices, or
> > you use loop devices from regular files, that might help (or slow
> > things down :P).
>
> Hmm.
> That sounds bad.
> I think, if highmem is unreachable some times that makes lowmem more
> valuable!
> The kernel needs to keep (reserve) it free as much as possible.
> The buffer-cache is an unimportant thing next to keeping lowmem free, but
> it is blocks the performance and wastes the systems resources!
>
> It is possible any workaround?
Try with one of the alternative vmsplit options that gives you more lowmem?
That might break certain applications though.
--
-ck
----- Original Message -----
From: "Con Kolivas" <[email protected]>
To: <[email protected]>
Cc: "Haar J?nos" <[email protected]>; "Nick Piggin"
<[email protected]>; <[email protected]>
Sent: Monday, May 22, 2006 1:17 PM
Subject: Re: swapper: page allocation failure.
> On Monday 22 May 2006 17:41, Haar J?nos wrote:
> > ----- Original Message -----
> > From: "Nick Piggin" <[email protected]>
> > > Yeah, as I said, block device's pagecache (aka buffercache) can't
> > > use highmem. If nbd can export regular files as block devices, or
> > > you use loop devices from regular files, that might help (or slow
> > > things down :P).
> >
> > Hmm.
> > That sounds bad.
> > I think, if highmem is unreachable some times that makes lowmem more
> > valuable!
> > The kernel needs to keep (reserve) it free as much as possible.
> > The buffer-cache is an unimportant thing next to keeping lowmem free,
but
> > it is blocks the performance and wastes the systems resources!
> >
> > It is possible any workaround?
>
> Try with one of the alternative vmsplit options that gives you more
lowmem?
> That might break certain applications though.
total used free shared buffers cached
Mem: 4049724 4021196 28528 0 16384 3217288
Low: 4049724 4021196 28528
High: 0 0 0
-/+ buffers/cache: 787524 3262200
Swap: 0 0 0
This is an 64 bit machine, the "concentrator".
It looks like use all, the 4G ram as "lowmem".
If i replace the cpu on my nodes to 64bit capable ones, i can use all the
memory as buffer-cache? :-)
Cheers,
Janos
>
> --
> -ck
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
On Tuesday 23 May 2006 01:08, Haar J?nos wrote:
> ----- Original Message -----
> From: "Con Kolivas" <[email protected]>
> > Try with one of the alternative vmsplit options that gives you more
>
> lowmem?
>
> > That might break certain applications though.
>
> total used free shared buffers cached
> Mem: 4049724 4021196 28528 0 16384 3217288
> Low: 4049724 4021196 28528
> High: 0 0 0
> -/+ buffers/cache: 787524 3262200
> Swap: 0 0 0
>
> This is an 64 bit machine, the "concentrator".
>
> It looks like use all, the 4G ram as "lowmem".
> If i replace the cpu on my nodes to 64bit capable ones, i can use all the
> memory as buffer-cache? :-)
Heh yes indeed. It's only if you're stuck on 32bit for whatever reason that
you'd need a different vmsplit. There is no need for highmem when 64bit
allows bazillions of bytes of lowmem :)
--
-ck