2005-01-28 17:53:28

by Mark Haverkamp

[permalink] [raw]
Subject: out of memory question


I have a situation where the out of memory killer kicked in and killed
off a process. From the information displayed, it looks like there was
a lot of free memory available. I need some help interpreting the
output. I have included the console output from the oom killer.

It is running 2.6.11-rc2 and has a patch from Nick Piggin:
http://marc.theaimsgroup.com/?l=linux-kernel&m=110665524811826&w=2
The machine is running as an iscsi target with 4K luns configured.

Thanks,
Mark.



oom-killer: gfp_mask=0xd1 (__GFP_FS|__GFP_IO|__GFP_WAIT|__GFP_DMA)
DMA per-cpu:
cpu 0 hot: low 2, high 6, batch 1
cpu 0 cold: low 0, high 2, batch 1
cpu 1 hot: low 2, high 6, batch 1
cpu 1 cold: low 0, high 2, batch 1
cpu 2 hot: low 2, high 6, batch 1
cpu 2 cold: low 0, high 2, batch 1
cpu 3 hot: low 2, high 6, batch 1
cpu 3 cold: low 0, high 2, batch 1
Normal per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16
HighMem per-cpu:
cpu 0 hot: low 32, high 96, batch 16
cpu 0 cold: low 0, high 32, batch 16
cpu 1 hot: low 32, high 96, batch 16
cpu 1 cold: low 0, high 32, batch 16
cpu 2 hot: low 32, high 96, batch 16
cpu 2 cold: low 0, high 32, batch 16
cpu 3 hot: low 32, high 96, batch 16
cpu 3 cold: low 0, high 32, batch 16

Free pages: 2855356kB (2846272kB HighMem)
Active:99217 inactive:107383 dirty:345 writeback:0 unstable:0 free:713839 slab:13402 mapped:4141 pagetables:126
DMA free:348kB min:68kB low:84kB high:100kB active:4kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? no
protections[]: 0 0 0
Normal free:8736kB min:3756kB low:4692kB high:5632kB active:378224kB inactive:420796kB present:901120kB pages_scanned:0 all_unreclaimable? no
protections[]: 0 0 0
HighMem free:2846272kB min:512kB low:640kB high:768kB active:18632kB inactive:8744kB present:3276800kB pages_scanned:0 all_unreclaimable? no
protections[]: 0 0 0
DMA: 55*4kB 4*8kB 2*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 348kB
Normal: 752*4kB 156*8kB 84*16kB 10*32kB 0*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 8736kB
HighMem: 82*4kB 27*8kB 154*16kB 70*32kB 33*64kB 15*128kB 6*256kB 6*512kB 4*1024kB 3*2048kB 689*4096kB = 2846272kB
Swap cache: add 0, delete 0, find 0/0, race 0+0
Out of Memory: Killed process 4513 (lpd).



--
Mark Haverkamp <[email protected]>


Attachments:
config (25.48 kB)

2005-01-28 19:28:42

by Mark Haverkamp

[permalink] [raw]
Subject: Re: out of memory question

On Fri, 2005-01-28 at 13:14 -0600, Matt Domsch wrote:
> On Fri, Jan 28, 2005 at 09:43:06AM -0800, Mark Haverkamp wrote:
> >
> > I have a situation where the out of memory killer kicked in and killed
> > off a process. From the information displayed, it looks like there was
> > a lot of free memory available. I need some help interpreting the
> > output. I have included the console output from the oom killer.
> >
> > It is running 2.6.11-rc2 and has a patch from Nick Piggin:
> > http://marc.theaimsgroup.com/?l=linux-kernel&m=110665524811826&w=2
> > The machine is running as an iscsi target with 4K luns configured.
>
> What is eating all of your ZONE_DMA? Of the 16MB available
>
> > DMA: 55*4kB 4*8kB 2*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 348kB
> > DMA free:348kB min:68kB low:84kB high:100kB active:4kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? no
> > protections[]: 0 0 0
>
> only 348kB is available, and something is requesting more...
>
> > oom-killer: gfp_mask=0xd1 (__GFP_FS|__GFP_IO|__GFP_WAIT|__GFP_DMA)
>
>
> This isn't a 64-bit architecture (you've got some ZONE_HIGHMEM), so
> it's not like the x86_64/ia64 iommu. Perhaps a
> <32-bit DMA address mask PCI device?
>

Thanks for the help. I have somewhere to start looking now.

Mark.

>
--
Mark Haverkamp <[email protected]>

2005-01-28 19:24:33

by Matt Domsch

[permalink] [raw]
Subject: Re: out of memory question

On Fri, Jan 28, 2005 at 09:43:06AM -0800, Mark Haverkamp wrote:
>
> I have a situation where the out of memory killer kicked in and killed
> off a process. From the information displayed, it looks like there was
> a lot of free memory available. I need some help interpreting the
> output. I have included the console output from the oom killer.
>
> It is running 2.6.11-rc2 and has a patch from Nick Piggin:
> http://marc.theaimsgroup.com/?l=linux-kernel&m=110665524811826&w=2
> The machine is running as an iscsi target with 4K luns configured.

What is eating all of your ZONE_DMA? Of the 16MB available

> DMA: 55*4kB 4*8kB 2*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 348kB
> DMA free:348kB min:68kB low:84kB high:100kB active:4kB inactive:0kB present:16384kB pages_scanned:0 all_unreclaimable? no
> protections[]: 0 0 0

only 348kB is available, and something is requesting more...

> oom-killer: gfp_mask=0xd1 (__GFP_FS|__GFP_IO|__GFP_WAIT|__GFP_DMA)


This isn't a 64-bit architecture (you've got some ZONE_HIGHMEM), so
it's not like the x86_64/ia64 iommu. Perhaps a
<32-bit DMA address mask PCI device?

Thanks,
Matt

--
Matt Domsch
Software Architect
Dell Linux Solutions linux.dell.com & http://www.dell.com/linux
Linux on Dell mailing lists @ http://lists.us.dell.com