Hi,
A friend of mine just booted 2.6.28 and when someone else tried to send
something to him over ssh (scp), the transfer failed and the following
appeared in dmesg:
Pid: 0, comm: swapper Tainted: P 2.6.28-gentoo #1
Call Trace:
<IRQ> [<ffffffff802a4e0d>] __alloc_pages_internal+0x3dd/0x4e0
[<ffffffff802cb273>] cache_alloc_refill+0x313/0x5b0
[<ffffffff802cb5fb>] __kmalloc+0xeb/0xf0
[<ffffffff806bb8ed>] __alloc_skb+0x6d/0x150
[<ffffffff8027ad90>] ktime_get_ts+0x30/0x60
[<ffffffff806bc457>] __netdev_alloc_skb+0x17/0x40
[<ffffffff804d6b5d>] b44_alloc_rx_skb+0x17d/0x740
[<ffffffff804d912d>] b44_poll+0x30d/0x620
[<ffffffff806bffc3>] net_rx_action+0x83/0x150
[<ffffffff80265e73>] __do_softirq+0x93/0x160
[<ffffffff8022c60c>] call_softirq+0x1c/0x30
[<ffffffff8022dfa5>] do_softirq+0x35/0x70
[<ffffffff80265b1d>] irq_exit+0x9d/0xb0
[<ffffffff8022e259>] do_IRQ+0xc9/0x110
[<ffffffff8022b896>] ret_from_intr+0x0/0xa
<EOI> [<ffffffff8062fbe0>] menu_reflect+0x0/0x90
[<ffffffff804a4029>] acpi_idle_enter_simple+0x1c7/0x237
[<ffffffff804a401f>] acpi_idle_enter_simple+0x1bd/0x237
[<ffffffff8029934d>] rcu_needs_cpu+0x3d/0x50
[<ffffffff8062ee6a>] cpuidle_idle_call+0xba/0x120
[<ffffffff8022a46e>] cpu_idle+0x5e/0xc0
Mem-Info:
DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 69
CPU 1: hi: 186, btch: 31 usd: 106
Active_anon:53932 active_file:23835 inactive_anon:1
inactive_file:49118 unevictable:0 dirty:7381 writeback:0 unstable:0
free:623763 slab:11485 mapped:24549 pagetables:2854 bounce:0
DMA free:4kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB
active_file:0kB inactive_file:0kB unevictable:0kB present:4596kB
pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3012 3012 3012
DMA32 free:2495048kB min:7016kB low:8768kB high:10524kB active_anon:215728kB
inactive_anon:4kB active_file:95340kB inactive_file:196472kB unevictable:0kB
present:3084968kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB
0*2048kB 0*4096kB = 4kB
DMA32: 1*4kB 34*8kB 55*16kB 17*32kB 6*64kB 4*128kB 2*256kB 3*512kB 2*1024kB
1*2048kB 607*4096kB = 2495012kB
73068 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap = 2104472kB
Total swap = 2104472kB
786029 pages RAM
15043 pages reserved
96273 pages shared
105199 pages non-shared
--
Mihai Donțu
Mihai Donțu <[email protected]> wrote:
> Hi,
>
> A friend of mine just booted 2.6.28 and when someone else tried to send
> something to him over ssh (scp), the transfer failed and the following
> appeared in dmesg:
You missed the all important order line before the stack trace.
If it's order 0, then you're just out of memory, if it's greater
than order 0, then either you're using jumbo frames or b44 is
broken.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
On Thursday 15 January 2009, Herbert Xu wrote:
> Mihai Donțu <[email protected]> wrote:
> > Hi,
> >
> > A friend of mine just booted 2.6.28 and when someone else tried to send
> > something to him over ssh (scp), the transfer failed and the following
> > appeared in dmesg:
>
> You missed the all important order line before the stack trace.
Right, sorry for that. Here it is:
swapper: page allocation failure. order:0, mode:0x21
> If it's order 0, then you're just out of memory, if it's greater
> than order 0, then either you're using jumbo frames or b44 is
> broken.
It is order zero, but he's not out of memory. He has just booted 2.6.28 (no X,
no thing), has 3GB of RAM and 2GB of swap.
The attached config was taken from a working 2.6.27.
--
Mihai Donțu
On Thu, Jan 15, 2009 at 03:27:43PM +0200, Mihai Donțu wrote:
> > > A friend of mine just booted 2.6.28 and when someone else tried to send
> > > something to him over ssh (scp), the transfer failed and the following
> > > appeared in dmesg:
> > You missed the all important order line before the stack trace.
> Right, sorry for that. Here it is:
> swapper: page allocation failure. order:0, mode:0x21
> > If it's order 0, then you're just out of memory, if it's greater
> > than order 0, then either you're using jumbo frames or b44 is
> > broken.
> It is order zero, but he's not out of memory. He has just booted 2.6.28 (no X,
> no thing), has 3GB of RAM and 2GB of swap.
b44 uses GFP_DMA bounce buffers in some situations (x86_64 with > 1GB of
memory would do the trick I think, as does x86 with 4:4 memory split),
and that's a very limited resource (Memory < 16MB ).
Lovely hardware feature requiring nasty workarounds
(the chip can't do DMA for addresses > 1GB).
Maybe something else sucks up GFP_DMA memory with 2.6.28?
On Thursday 15 January 2009 13:15:37 you wrote:
> You missed the all important order line before the stack trace.
> If it's order 0, then you're just out of memory, if it's greater
> than order 0, then either you're using jumbo frames or b44 is
> broken.
This is the test I've done: boot the system with 2.6.28 (config is copied from
2.6.27 which works ok); from another system attempt to scp a 45MB file to
this system; the scp stalls (and finally fails) at around 60%. This is 100%
reproducible in this way. Attached is the full dmesg output and the config.
The output of 'free' during all this:
total used free shared buffers cached
Mem: 3083944 114304 2969640 0 4160 61096
-/+ buffers/cache: 49048 3034896
Swap: 2104472 0 2104472
Pekka Pietikainen a écrit :
> On Thu, Jan 15, 2009 at 03:27:43PM +0200, Mihai Donțu wrote:
>>>> A friend of mine just booted 2.6.28 and when someone else tried to send
>>>> something to him over ssh (scp), the transfer failed and the following
>>>> appeared in dmesg:
>>> You missed the all important order line before the stack trace.
>> Right, sorry for that. Here it is:
>> swapper: page allocation failure. order:0, mode:0x21
>>> If it's order 0, then you're just out of memory, if it's greater
>>> than order 0, then either you're using jumbo frames or b44 is
>>> broken.
>> It is order zero, but he's not out of memory. He has just booted 2.6.28 (no X,
>> no thing), has 3GB of RAM and 2GB of swap.
> b44 uses GFP_DMA bounce buffers in some situations (x86_64 with > 1GB of
> memory would do the trick I think, as does x86 with 4:4 memory split),
> and that's a very limited resource (Memory < 16MB ).
>
> Lovely hardware feature requiring nasty workarounds
> (the chip can't do DMA for addresses > 1GB).
>
> Maybe something else sucks up GFP_DMA memory with 2.6.28?
Normal tcp queues (in this case, since a single scp is enough
to trigger the bug)
If a rx buffer was allocated using GFP_DMA, then b44_rx()
should copy it to a new skb, (like done by copybreak algo),
regardless of frame size, to not consume DMA memory in protocol
queues.
Better be slow (because of extra copy) then risking exhausting DMA zone
easy way to test would be to redefine RX_COPY_THRESHOLD to 16000
Eric Dumazet a écrit :
> Pekka Pietikainen a écrit :
>> On Thu, Jan 15, 2009 at 03:27:43PM +0200, Mihai Donțu wrote:
>>>>> A friend of mine just booted 2.6.28 and when someone else tried to send
>>>>> something to him over ssh (scp), the transfer failed and the following
>>>>> appeared in dmesg:
>>>> You missed the all important order line before the stack trace.
>>> Right, sorry for that. Here it is:
>>> swapper: page allocation failure. order:0, mode:0x21
>>>> If it's order 0, then you're just out of memory, if it's greater
>>>> than order 0, then either you're using jumbo frames or b44 is
>>>> broken.
>>> It is order zero, but he's not out of memory. He has just booted 2.6.28 (no X,
>>> no thing), has 3GB of RAM and 2GB of swap.
>> b44 uses GFP_DMA bounce buffers in some situations (x86_64 with > 1GB of
>> memory would do the trick I think, as does x86 with 4:4 memory split),
>> and that's a very limited resource (Memory < 16MB ).
>>
>> Lovely hardware feature requiring nasty workarounds
>> (the chip can't do DMA for addresses > 1GB).
>>
>> Maybe something else sucks up GFP_DMA memory with 2.6.28?
>
> Normal tcp queues (in this case, since a single scp is enough
> to trigger the bug)
>
> If a rx buffer was allocated using GFP_DMA, then b44_rx()
> should copy it to a new skb, (like done by copybreak algo),
> regardless of frame size, to not consume DMA memory in protocol
> queues.
>
> Better be slow (because of extra copy) then risking exhausting DMA zone
>
> easy way to test would be to redefine RX_COPY_THRESHOLD to 16000
Or try following patch (I dont have this hardware)
[PATCH] b44: GFP_DMA skb should not escape from driver
b44 chip has some hardware limitations, that need GFP_DMA bounce
buffers in some situations.
In order to not deplete DMA zone, we should keep allocated GFP_DMA skb
only for driver use. At rx time, we copy such skb to newly allocated
skb, reusing existing copybreak infrastructure.
On machines with low amount of memory, all skb meet the hardware limitation,
so no copy is needed. We detect this situation using a new device flag, set
to one if one GFP_DMA skb was ever allocated by b44_alloc_rx_skb().
Previously allocated skb, even outside from DMA zone will then be recycled,
to have minimal impact on DMA zone use.
Signed-off-by: Eric Dumazet <[email protected]>
---
diff --git a/drivers/net/b44.c b/drivers/net/b44.c
index 5ae131c..c38512e 100644
--- a/drivers/net/b44.c
+++ b/drivers/net/b44.c
@@ -679,6 +679,7 @@ static int b44_alloc_rx_skb(struct b44 *bp, int src_idx, u32 dest_idx_unmasked)
dev_kfree_skb_any(skb);
return -ENOMEM;
}
+ bp->force_copybreak = 1;
}
rh = (struct rx_header *) skb->data;
@@ -800,7 +801,7 @@ static int b44_rx(struct b44 *bp, int budget)
/* Omit CRC. */
len -= 4;
- if (len > RX_COPY_THRESHOLD) {
+ if (!bp->force_copybreak && len > RX_COPY_THRESHOLD) {
int skb_size;
skb_size = b44_alloc_rx_skb(bp, cons, bp->rx_prod);
if (skb_size < 0)
@@ -2152,6 +2153,7 @@ static int __devinit b44_init_one(struct ssb_device *sdev,
bp = netdev_priv(dev);
bp->sdev = sdev;
bp->dev = dev;
+ bp->force_copybreak = 0;
bp->msg_enable = netif_msg_init(b44_debug, B44_DEF_MSG_ENABLE);
diff --git a/drivers/net/b44.h b/drivers/net/b44.h
index 7db0c84..e678498 100644
--- a/drivers/net/b44.h
+++ b/drivers/net/b44.h
@@ -395,7 +395,7 @@ struct b44 {
u32 rx_pending;
u32 tx_pending;
u8 phy_addr;
-
+ u8 force_copybreak;
struct mii_if_info mii_if;
};
Applied, seems to fix the problem.
Ionut Leonte a écrit :
> Applied, seems to fix the problem.
Thanks for your report and testing !
From: Eric Dumazet <[email protected]>
Date: Thu, 15 Jan 2009 18:28:49 +0100
> Ionut Leonte a ?crit :
> > Applied, seems to fix the problem.
>
> Thanks for your report and testing !
Patch applied, thanks everyone.