Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
I am running 2.4.2 on a linux/mips box, with 32MB system RAM (no swap). When
I run a stress test, I will hit memory allocation failure:
__alloc_pages: 1-order allocation failed.
IP: queue_glue: no memory for gluing queue 8108cce0
However, I traced the system and found several questions.
First, free reports enough free memory and LOTS of cache memory. See below.
Should'nt Linux free cache memory to satisfy this request?
total used free shared buffers cached
Mem: 30052 29296 756 0 0 16928
-/+ buffers/cache: 12368 17684
Swap: 0 0 0
I used kgdb and dig into rmqueue() in mm/page_alloc.c file. There are two
zones in this system. They do have many free pages but none of free blocks
seem to be big enough for the 2-page request. Does this make sense? Why does
kernel end up with so many fragmented 1-page free areas? See the kgdb output
below.
(gdb) p order
$14 = 1
(gdb) p *zone
$15 = {lock = {gcc_is_buggy = 0}, offset = 4096, free_pages = 65,
inactive_clean_pages = 262, inactive_dirty_pages = 10, pages_min = 32,
pages_low = 64, pages_high = 96, inactive_clean_list = {next = 0x8107cdf0,
prev = 0x81053c40}, free_area = {{free_list = {next = 0x8105f548,
prev = 0x810562a8}, map = 0x810884e0}, {free_list = {
next = 0x801fe5f0, prev = 0x801fe5f0}, map = 0x810886e0}, {
free_list = {next = 0x801fe5fc, prev = 0x801fe5fc}, map = 0x810887e0}, {
free_list = {next = 0x801fe608, prev = 0x801fe608}, map = 0x81088860}, {
free_list = {next = 0x801fe614, prev = 0x801fe614}, map = 0x810888a0}, {
free_list = {next = 0x801fe620, prev = 0x801fe620}, map = 0x810888c0}, {
free_list = {next = 0x801fe62c, prev = 0x801fe62c}, map = 0x810888e0}, {
free_list = {next = 0x801fe638, prev = 0x801fe638}, map = 0x81088900}, {
free_list = {next = 0x801fe644, prev = 0x801fe644}, map = 0x81088920}, {
free_list = {next = 0x801fe650, prev = 0x801fe650}, map = 0x81088940}},
name = 0x801af0e4 "Normal", size = 4096, zone_pgdat = 0x801fe504,
zone_start_paddr = 16777216, zone_start_mapnr = 4096,
zone_mem_map = 0x81044010}
(gdb) p z
$23 = (zone_t *) 0x801fe504
(gdb) p *z
$24 = {lock = {gcc_is_buggy = 0}, offset = 0, free_pages = 329,
inactive_clean_pages = 861, inactive_dirty_pages = 16, pages_min = 128,
pages_low = 256, pages_high = 384, inactive_clean_list = {next = 0x81020888,
prev = 0x81036508}, free_area = {{free_list = {next = 0x8102d1c4,
prev = 0x81035430}, map = 0x81088060}, {free_list = {
next = 0x801fe538, prev = 0x801fe538}, map = 0x81088260}, {
free_list = {next = 0x801fe544, prev = 0x801fe544}, map = 0x81088360}, {
free_list = {next = 0x801fe550, prev = 0x801fe550}, map = 0x810883e0}, {
free_list = {next = 0x801fe55c, prev = 0x801fe55c}, map = 0x81088420}, {
free_list = {next = 0x801fe568, prev = 0x801fe568}, map = 0x81088440}, {
free_list = {next = 0x801fe574, prev = 0x801fe574}, map = 0x81088460}, {
free_list = {next = 0x801fe580, prev = 0x801fe580}, map = 0x81088480}, {
free_list = {next = 0x801fe58c, prev = 0x801fe58c}, map = 0x810884a0}, {
free_list = {next = 0x801fe598, prev = 0x801fe598}, map = 0x810884c0}},
name = 0x801af0ec "DMA", size = 4096, zone_pgdat = 0x801fe504,
zone_start_paddr = 0, zone_start_mapnr = 0, zone_mem_map = 0x81000010}
The current process does not have PF_MEMALLOC flag set.
The callstack trace is as follows: (kgdb is lying - the gfp_mask should be 2
at frame #1. The size in kmalloc is probably a lie from kgdb too.)
#0 __alloc_pages (zonelist=0x801fe754, order=1) at page_alloc.c:498
#1 0x800b3c08 in __get_free_pages (gfp_mask=0, order=1)
at
/local/jsun/2ND-DISK/work/ddb5477/build/hhl-2.4.2-010709/hhl-2.4.2/linux/
include/linux/mm.h:355
#2 0x800aedbc in kmem_cache_grow (cachep=0x8108b700, flags=2) at slab.c:485
#3 0x800af258 in kmalloc (size=2149573892, flags=2) at slab.c:1331
#4 0x801585bc in alloc_skb (size=4160, gfp_mask=2) at skbuff.c:189
#5 0x80166cbc in ip_frag_reasm (qp=0x8108cce0, dev=0x810a1800)
at
/local/jsun/2ND-DISK/work/ddb5477/build/hhl-2.4.2-010709/hhl-2.4.2/linux/
include/linux/skbuff.h:919
#6 0x80167040 in ip_defrag (skb=0x0) at ip_fragment.c:626
#7 0x80165c28 in ip_local_deliver (skb=0x810dcd20) at ip_input.c:291
#8 0x80166184 in ip_rcv (skb=0x810dcd20, dev=0x1, pt=0x0) at ip_input.c:363
#9 0x8015b5ec in net_rx_action (h=0x801fe504) at dev.c:1403
#10 0x800993d8 in do_softirq () at softirq.c:78
#11 0x801a2eb4 in do_IRQ (irq=16, regs=0x81c79d18) at irq.c:644
#12 0x801a50ec in vrc5477_irq_dispatch (regs=0x81c79d18) at irq.c:175
Would appreciate any pointers.
Jun
On Wednesday 11 July 2001 02:48, Jun Sun wrote:
> Content-Type: text/plain; charset=us-ascii
> Content-Transfer-Encoding: 7bit
>
> I am running 2.4.2 on a linux/mips box, with 32MB system RAM (no
> swap). When I run a stress test, I will hit memory allocation
> failure:
>
> __alloc_pages: 1-order allocation failed.
> IP: queue_glue: no memory for gluing queue 8108cce0
Next step: install the latest stable kernel and see if the problem
persists. (In this case I doubt it will)
--
Daniel
Just found more about this problem....
Jun Sun wrote:
>
> I am running 2.4.2 on a linux/mips box, with 32MB system RAM (no swap). When
> I run a stress test, I will hit memory allocation failure:
>
> __alloc_pages: 1-order allocation failed.
> IP: queue_glue: no memory for gluing queue 8108cce0
>
> However, I traced the system and found several questions.
>
> First, free reports enough free memory and LOTS of cache memory. See below.
> Should'nt Linux free cache memory to satisfy this request?
>
> total used free shared buffers cached
> Mem: 30052 29296 756 0 0 16928
> -/+ buffers/cache: 12368 17684
> Swap: 0 0 0
>
The memory allocation is invoked with GFP_ATOMIC, and thus does not free up
the cached pages. (Why? because it is invoked from interrupt context?)
> I used kgdb and dig into rmqueue() in mm/page_alloc.c file. There are two
> zones in this system. They do have many free pages but none of free blocks
> seem to be big enough for the 2-page request. Does this make sense? Why does
> kernel end up with so many fragmented 1-page free areas? See the kgdb output
> below.
>
Shouldn't some kernel daemon try to maintain a "reasonable" free_area,
including a 2-page block? It seems like in my case the daemon failed to do
the job. (Which daemon is it? kflushd?) What is the condition to kick start
this daemon? I suppose in my case the total free page count is still high,
which may fail to start the daemon.
In the end, allocating memory from interrupt context for re-assembling IP
packet does not sound pretty either ...
Jun