I have a 2 CPU P4 box w/ HT enabled and 2GB RAM that is a heavy load NFS
server in a HPC environment (it's an IBM x345 system). It's connected to
4TB of SAN space using dual HBAs. Today, I had two instaces of page
allocation failures (see info below). The box has been up for some time
with no issues (never seen this before).
Is there any tuning I can do to fix this? I've changed min_free_pages to
32MB (from defauly of ~3MB) and I read somewhere that giving kswapd a little
higher priority can help (I reniced it to -5) though there's a lot of stuff
on the Internet that's wrong ;).
Chris
> uname -a
Linux nicfs3 2.6.11.5 #1 SMP Thu Mar 24 15:31:56 EST 2005 i686 i686 i386
GNU/Linux
> cat /proc/net/rpc/nfsd
rc 1213 1141688715 607511169
fh 26082 0 0 0 0
io 738654092 2667724944
th 128 1816606 50023.114 10443.992 4106.448 1796.765 1662.135 936.520
578.055 519.936 800.976 2746.497
ra 256 94366366 538753 87285 44256 4504 2646 2402 1743 1797 1192 406827
net 1749208674 47457 1749218587 16041
rpc 1748565371 1295 1295 0 0
proc2 18 4967 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
proc3 22 42524 410845991 13072619 17044551 34409120 22976 95453790
1122463558 2912160 9926 5840 1 2567900 13767 131245 95 120245 457922 137035
7715 65401 48803973
Intel(R) PRO/1000 Network Driver - version 5.6.10.1-k2-NAPI
Copyright (c) 1999-2004 Intel Corporation.
ACPI: PCI interrupt 0000:06:08.0[A] -> GSI 29 (level, low) -> IRQ 29
usb 1-1: new full speed USB device using ohci_hcd and address 2
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
ACPI: PCI interrupt 0000:06:08.1[B] -> GSI 30 (level, low) -> IRQ 30
e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:0D:60:1A:DC:B2
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:426339601 errors:359167 dropped:359167 overruns:57615
frame:0
TX packets:3062378244 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:9693358 (9.2 Mb) TX bytes:2162860348 (2062.6 Mb)
Base address:0x2700 Memory:ebfe0000-ec000000
# ethtool -S eth0
NIC statistics:
rx_packets: 424323852
tx_packets: 3061266959
rx_bytes: 1290078523
tx_bytes: 2068709781
rx_errors: 359167
tx_errors: 0
rx_dropped: 301552
tx_dropped: 0
multicast: 50788
collisions: 0
rx_length_errors: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 57615
rx_missed_errors: 57615
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_window_errors: 0
tx_abort_late_coll: 0
tx_deferred_ok: 0
tx_single_coll_ok: 0
tx_multi_coll_ok: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_align_errors: 0
tx_tcp_seg_good: 92070751
tx_tcp_seg_failed: 0
rx_flow_control_xon: 0
rx_flow_control_xoff: 0
tx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_long_byte_count: 37221476665659
rx_csum_offload_good: 26192288465
rx_csum_offload_errors: 1
# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: on
# ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
Current hardware settings:
RX: 256
RX Mini: 0
RX Jumbo: 0
TX: 256
Relevant parts of dmesg:
swapper: page allocation failure. order:0, mode:0x20
[<c0144793>] __alloc_pages+0x333/0x3b0
[<c0147342>] kmem_getpages+0x32/0xa0
[<c014809e>] cache_grow+0xbe/0x180
[<c01482cd>] cache_alloc_refill+0x16d/0x240
[<c014869c>] __kmalloc+0x6c/0x80
[<c0419780>] init_chipset_sis5513+0x240/0x470
[<c02b4820>] alloc_skb+0x40/0xf0
[<f8b0f900>] e1000_alloc_rx_buffers+0x40/0xf0 [e1000]
[<f8b0f690>] e1000_clean_rx_irq+0x280/0x4b0 [e1000]
[<c02b4b8d>] __kfree_skb+0xad/0x160
[<c02ea8b6>] tcp_transmit_skb+0x426/0x750
[<f8b0f11e>] e1000_clean+0x4e/0x110 [e1000]
[<c02bb0a1>] net_rx_action+0xb1/0x110
[<c01253e2>] __do_softirq+0x62/0xe0
[<c012548d>] do_softirq+0x2d/0x40
[<c01062db>] do_IRQ+0x3b/0x70
[<c01049f2>] common_interrupt+0x1a/0x20
[<c0102053>] default_idle+0x23/0x30
[<c01020ee>] cpu_idle+0x4e/0x60
[<c03f696a>] start_kernel+0x19a/0x220
[<c03f6360>] unknown_bootoption+0x0/0x1e0
klogd: page allocation failure. order:0, mode:0x20
[<c0144793>] __alloc_pages+0x333/0x3b0
[<c0147342>] kmem_getpages+0x32/0xa0
[<c014809e>] cache_grow+0xbe/0x180
[<c01482cd>] cache_alloc_refill+0x16d/0x240
[<c014869c>] __kmalloc+0x6c/0x80
[<c0419580>] init_chipset_sis5513+0x40/0x470
[<c02b4820>] alloc_skb+0x40/0xf0
[<f8b0f900>] e1000_alloc_rx_buffers+0x40/0xf0 [e1000]
[<f8b0f690>] e1000_clean_rx_irq+0x280/0x4b0 [e1000]
[<c02ea8b6>] tcp_transmit_skb+0x426/0x750
[<c0419680>] init_chipset_sis5513+0x140/0x470
[<f8b0f11e>] e1000_clean+0x4e/0x110 [e1000]
[<c02bb0a1>] net_rx_action+0xb1/0x110
[<c01253e2>] __do_softirq+0x62/0xe0
[<c012548d>] do_softirq+0x2d/0x40
[<c0104a80>] apic_timer_interrupt+0x1c/0x24
klogd: page allocation failure. order:0, mode:0x20
[<c0144793>] __alloc_pages+0x333/0x3b0
[<c0147342>] kmem_getpages+0x32/0xa0
[<c014809e>] cache_grow+0xbe/0x180
[<c01482cd>] cache_alloc_refill+0x16d/0x240
[<c014869c>] __kmalloc+0x6c/0x80
[<c0419680>] init_chipset_sis5513+0x140/0x470
[<c02b4820>] alloc_skb+0x40/0xf0
[<f8b0f900>] e1000_alloc_rx_buffers+0x40/0xf0 [e1000]
[<f8b0f690>] e1000_clean_rx_irq+0x280/0x4b0 [e1000]
[<c02b4b8d>] __kfree_skb+0xad/0x160
[<c0419680>] init_chipset_sis5513+0x140/0x470
[<f8b0f11e>] e1000_clean+0x4e/0x110 [e1000]
[<c02bb0a1>] net_rx_action+0xb1/0x110
[<c01253e2>] __do_softirq+0x62/0xe0
[<c012548d>] do_softirq+0x2d/0x40
[<c01062db>] do_IRQ+0x3b/0x70
[<c011c077>] __wake_up_common+0x37/0x70
[<c01049f2>] common_interrupt+0x1a/0x20
[<c0336eb5>] _spin_unlock_irqrestore+0x5/0x10
[<c02b405c>] sock_def_readable+0x6c/0xa0
[<c0419580>] init_chipset_sis5513+0x40/0x470
[<c0316c49>] unix_dgram_sendmsg+0x4b9/0x510
[<c0147342>] kmem_getpages+0x32/0xa0
[<c02b0df5>] sock_aio_write+0x135/0x150
[<c015ee22>] do_sync_write+0xb2/0x100
[<f8b0f900>] e1000_alloc_rx_buffers+0x40/0xf0 [e1000]
[<c02ea8b6>] tcp_transmit_skb+0x426/0x750
[<c0419680>] init_chipset_sis5513+0x140/0x470
[<c0134b60>] autoremove_wake_function+0x0/0x50
[<c015ef9d>] vfs_write+0x12d/0x130
[<c015f051>] sys_write+0x41/0x70
[<c010402d>] sysenter_past_esp+0x52/0x75
swapper: page allocation failure. order:0, mode:0x20
[<c0144793>] __alloc_pages+0x333/0x3b0
[<c0147342>] kmem_getpages+0x32/0xa0
[<c014809e>] cache_grow+0xbe/0x180
[<c01482cd>] cache_alloc_refill+0x16d/0x240
[<c014869c>] __kmalloc+0x6c/0x80
[<c02b4820>] alloc_skb+0x40/0xf0
[<f8b0f900>] e1000_alloc_rx_buffers+0x40/0xf0 [e1000]
[<f8b0f690>] e1000_clean_rx_irq+0x280/0x4b0 [e1000]
[<c02b4b8d>] __kfree_skb+0xad/0x160
[<c02ea8b6>] tcp_transmit_skb+0x426/0x750
[<f8b0f11e>] e1000_clean+0x4e/0x110 [e1000]
[<c02bb0a1>] net_rx_action+0xb1/0x110
[<c01253e2>] __do_softirq+0x62/0xe0
[<c012548d>] do_softirq+0x2d/0x40
[<c01062db>] do_IRQ+0x3b/0x70
[<c01049f2>] common_interrupt+0x1a/0x20
[<c0102053>] default_idle+0x23/0x30
[<c01020ee>] cpu_idle+0x4e/0x60
[<c03f696a>] start_kernel+0x19a/0x220
[<c03f6360>] unknown_bootoption+0x0/0x1e0
swapper: page allocation failure. order:0, mode:0x20
[<c0144793>] __alloc_pages+0x333/0x3b0
[<c0147342>] kmem_getpages+0x32/0xa0
[<c014809e>] cache_grow+0xbe/0x180
[<c01482cd>] cache_alloc_refill+0x16d/0x240
[<c014869c>] __kmalloc+0x6c/0x80
[<c02b4820>] alloc_skb+0x40/0xf0
[<f8b0f900>] e1000_alloc_rx_buffers+0x40/0xf0 [e1000]
[<f8b0f690>] e1000_clean_rx_irq+0x280/0x4b0 [e1000]
[<c02b4b8d>] __kfree_skb+0xad/0x160
[<c02ea8b6>] tcp_transmit_skb+0x426/0x750
[<f8b0f11e>] e1000_clean+0x4e/0x110 [e1000]
[<c02bb0a1>] net_rx_action+0xb1/0x110
[<c01253e2>] __do_softirq+0x62/0xe0
[<c012548d>] do_softirq+0x2d/0x40
[<c01062db>] do_IRQ+0x3b/0x70
[<c01049f2>] common_interrupt+0x1a/0x20
[<c0102053>] default_idle+0x23/0x30
[<c01020ee>] cpu_idle+0x4e/0x60
[<c03f696a>] start_kernel+0x19a/0x220
[<c03f6360>] unknown_bootoption+0x0/0x1e0
swapper: page allocation failure. order:0, mode:0x20
[<c0144793>] __alloc_pages+0x333/0x3b0
[<c0147342>] kmem_getpages+0x32/0xa0
[<c014809e>] cache_grow+0xbe/0x180
[<c01482cd>] cache_alloc_refill+0x16d/0x240
[<c014869c>] __kmalloc+0x6c/0x80
[<c02b4820>] alloc_skb+0x40/0xf0
[<f8b0f900>] e1000_alloc_rx_buffers+0x40/0xf0 [e1000]
[<f8b0f690>] e1000_clean_rx_irq+0x280/0x4b0 [e1000]
[<c02b4b8d>] __kfree_skb+0xad/0x160
[<c02ea8b6>] tcp_transmit_skb+0x426/0x750
[<f8b0f11e>] e1000_clean+0x4e/0x110 [e1000]
[<c02bb0a1>] net_rx_action+0xb1/0x110
[<c01253e2>] __do_softirq+0x62/0xe0
[<c012548d>] do_softirq+0x2d/0x40
[<c01062db>] do_IRQ+0x3b/0x70
[<c01049f2>] common_interrupt+0x1a/0x20
[<c0102053>] default_idle+0x23/0x30
[<c01020ee>] cpu_idle+0x4e/0x60
[<c03f696a>] start_kernel+0x19a/0x220
[<c03f6360>] unknown_bootoption+0x0/0x1e0
swapper: page allocation failure. order:0, mode:0x20
[<c0144793>] __alloc_pages+0x333/0x3b0
[<c0147342>] kmem_getpages+0x32/0xa0
[<c014809e>] cache_grow+0xbe/0x180
[<c01482cd>] cache_alloc_refill+0x16d/0x240
[<c014869c>] __kmalloc+0x6c/0x80
[<c02b4820>] alloc_skb+0x40/0xf0
[<f8b0f900>] e1000_alloc_rx_buffers+0x40/0xf0 [e1000]
[<f8b0f690>] e1000_clean_rx_irq+0x280/0x4b0 [e1000]
[<c02b4b8d>] __kfree_skb+0xad/0x160
[<c02ea8b6>] tcp_transmit_skb+0x426/0x750
[<f8b0f11e>] e1000_clean+0x4e/0x110 [e1000]
[<c02bb0a1>] net_rx_action+0xb1/0x110
[<c01253e2>] __do_softirq+0x62/0xe0
[<c012548d>] do_softirq+0x2d/0x40
[<c01062db>] do_IRQ+0x3b/0x70
[<c01049f2>] common_interrupt+0x1a/0x20
[<c0102053>] default_idle+0x23/0x30
[<c01020ee>] cpu_idle+0x4e/0x60
[<c03f696a>] start_kernel+0x19a/0x220
[<c03f6360>] unknown_bootoption+0x0/0x1e0
swapper: page allocation failure. order:0, mode:0x20
[<c0144793>] __alloc_pages+0x333/0x3b0
[<c0147342>] kmem_getpages+0x32/0xa0
[<c014809e>] cache_grow+0xbe/0x180
[<c01482cd>] cache_alloc_refill+0x16d/0x240
[<c014869c>] __kmalloc+0x6c/0x80
[<c02b4820>] alloc_skb+0x40/0xf0
[<f8b0f900>] e1000_alloc_rx_buffers+0x40/0xf0 [e1000]
[<f8b0f690>] e1000_clean_rx_irq+0x280/0x4b0 [e1000]
[<c02b4b8d>] __kfree_skb+0xad/0x160
[<c02ea8b6>] tcp_transmit_skb+0x426/0x750
[<f8b0f11e>] e1000_clean+0x4e/0x110 [e1000]
[<c02bb0a1>] net_rx_action+0xb1/0x110
[<c01253e2>] __do_softirq+0x62/0xe0
[<c012548d>] do_softirq+0x2d/0x40
[<c01062db>] do_IRQ+0x3b/0x70
[<c01049f2>] common_interrupt+0x1a/0x20
[<c0102053>] default_idle+0x23/0x30
[<c01020ee>] cpu_idle+0x4e/0x60
[<c03f696a>] start_kernel+0x19a/0x220
[<c03f6360>] unknown_bootoption+0x0/0x1e0
swapper: page allocation failure. order:0, mode:0x20
[<c0144793>] __alloc_pages+0x333/0x3b0
[<c0147342>] kmem_getpages+0x32/0xa0
[<c014809e>] cache_grow+0xbe/0x180
[<c01482cd>] cache_alloc_refill+0x16d/0x240
[<c014869c>] __kmalloc+0x6c/0x80
[<c02b4820>] alloc_skb+0x40/0xf0
[<f8b0f900>] e1000_alloc_rx_buffers+0x40/0xf0 [e1000]
[<f8b0f690>] e1000_clean_rx_irq+0x280/0x4b0 [e1000]
[<c02b4b8d>] __kfree_skb+0xad/0x160
[<c02ea8b6>] tcp_transmit_skb+0x426/0x750
[<f8b0f11e>] e1000_clean+0x4e/0x110 [e1000]
[<c02bb0a1>] net_rx_action+0xb1/0x110
[<c01253e2>] __do_softirq+0x62/0xe0
[<c012548d>] do_softirq+0x2d/0x40
[<c01062db>] do_IRQ+0x3b/0x70
[<c01049f2>] common_interrupt+0x1a/0x20
[<c0102053>] default_idle+0x23/0x30
[<c01020ee>] cpu_idle+0x4e/0x60
[<c03f696a>] start_kernel+0x19a/0x220
[<c03f6360>] unknown_bootoption+0x0/0x1e0
swapper: page allocation failure. order:0, mode:0x20
[<c0144793>] __alloc_pages+0x333/0x3b0
[<c0147342>] kmem_getpages+0x32/0xa0
[<c014809e>] cache_grow+0xbe/0x180
[<c01482cd>] cache_alloc_refill+0x16d/0x240
[<c014869c>] __kmalloc+0x6c/0x80
[<c02b4820>] alloc_skb+0x40/0xf0
[<f8b0f900>] e1000_alloc_rx_buffers+0x40/0xf0 [e1000]
[<f8b0f690>] e1000_clean_rx_irq+0x280/0x4b0 [e1000]
[<c02b4b8d>] __kfree_skb+0xad/0x160
[<c02ea8b6>] tcp_transmit_skb+0x426/0x750
[<f8b0f11e>] e1000_clean+0x4e/0x110 [e1000]
[<c02bb0a1>] net_rx_action+0xb1/0x110
[<c01253e2>] __do_softirq+0x62/0xe0
[<c012548d>] do_softirq+0x2d/0x40
[<c01062db>] do_IRQ+0x3b/0x70
[<c01049f2>] common_interrupt+0x1a/0x20
[<c0102053>] default_idle+0x23/0x30
[<c01020ee>] cpu_idle+0x4e/0x60
[<c03f696a>] start_kernel+0x19a/0x220
[<c03f6360>] unknown_bootoption+0x0/0x1e0
printk: 130 messages suppressed.
nfsd: page allocation failure. order:1, mode:0x20
[<c0144793>] __alloc_pages+0x333/0x3b0
[<c0147342>] kmem_getpages+0x32/0xa0
[<c014809e>] cache_grow+0xbe/0x180
[<c01482cd>] cache_alloc_refill+0x16d/0x240
[<c014869c>] __kmalloc+0x6c/0x80
[<c02b4820>] alloc_skb+0x40/0xf0
[<c02e80d2>] tcp_collapse+0x152/0x370
[<c02e83f7>] tcp_prune_queue+0x87/0x1e0
[<c02e79f3>] tcp_data_queue+0x573/0xb00
[<c02e6312>] tcp_ack+0xd2/0x5d0
[<c02e8df6>] tcp_rcv_established+0x2e6/0x7f0
[<c02f1960>] tcp_v4_do_rcv+0xf0/0x110
[<c02f21c1>] tcp_v4_rcv+0x841/0x920
[<c02d9d5a>] ip_queue_xmit+0x30a/0x4f0
[<c02d62d6>] ip_local_deliver+0x96/0x260
[<c02d68c8>] ip_rcv+0x428/0x500
[<c02baeac>] netif_receive_skb+0x27c/0x2a0
[<f8b0f4be>] e1000_clean_rx_irq+0xae/0x4b0 [e1000]
[<c02b4b8d>] __kfree_skb+0xad/0x160
[<f8b0f11e>] e1000_clean+0x4e/0x110 [e1000]
[<c02bb0a1>] net_rx_action+0xb1/0x110
[<c01253e2>] __do_softirq+0x62/0xe0
[<c012548d>] do_softirq+0x2d/0x40
[<c01062db>] do_IRQ+0x3b/0x70
[<c01049f2>] common_interrupt+0x1a/0x20
[<f8a670f0>] xtSearch+0x240/0x650 [jfs]
[<f8a68b58>] xtExtend+0x58/0x660 [jfs]
[<c0162f40>] nobh_commit_write+0x30/0x80
[<c013fb8d>] unlock_page+0x1d/0x30
[<f8a7c4fa>] __write_metapage+0x6a/0xb0 [jfs]
[<c01430b1>] mempool_free+0x81/0xa0
[<f8a71470>] dbAlloc+0x3c0/0x4b0 [jfs]
[<f8a7bd2f>] extBalloc+0x20f/0x230 [jfs]
[<c0181ac8>] __mark_inode_dirty+0xd8/0x1e0
[<c0168864>] inode_add_bytes+0x24/0xa0
[<f8a7b954>] extAlloc+0x434/0x460 [jfs]
[<f8a6316c>] jfs_get_blocks+0x27c/0x340 [jfs]
[<c02d96aa>] ip_finish_output+0xda/0x270
[<f8a63250>] jfs_get_block+0x20/0x30 [jfs]
[<c0162c1c>] nobh_prepare_write+0xec/0x3e0
[<c0144271>] buffered_rmqueue+0xf1/0x210
[<c0177446>] d_alloc_anon+0x26/0x110
[<c0144525>] __alloc_pages+0xc5/0x3b0
[<c013f9f6>] add_to_page_cache+0xb6/0xe0
[<c01418b7>] generic_file_buffered_write+0x1b7/0x5e0
[<f8a63230>] jfs_get_block+0x0/0x30 [jfs]
[<c02ea8b6>] tcp_transmit_skb+0x426/0x750
[<c01799ee>] inode_update_time+0x9e/0xc0
[<c0142013>] __generic_file_aio_write_nolock+0x333/0x590
[<c02b0af1>] sock_recvmsg+0x131/0x170
[<c011a95d>] finish_task_switch+0x3d/0x90
[<c0335608>] schedule+0x378/0xbb0
[<c011a5df>] try_to_wake_up+0x24f/0x280
[<c01423a5>] __generic_file_write_nolock+0x85/0xb0
[<c03350fe>] __down+0xce/0xe0
[<c011c030>] default_wake_function+0x0/0x10
[<c0134b60>] autoremove_wake_function+0x0/0x50
[<c0142560>] generic_file_write+0x0/0xd0
[<c0335233>] __down_failed+0x7/0xc
[<c0142560>] generic_file_write+0x0/0xd0
[<c0142716>] generic_file_writev+0x36/0xc0
[<c020282a>] __copy_from_user_ll+0x5a/0x80
[<c01426e0>] generic_file_writev+0x0/0xc0
[<c0142560>] generic_file_write+0x0/0xd0
[<c015f40b>] do_readv_writev+0x23b/0x250
[<f8a39950>] export_decode_fh+0x0/0xa5 [exportfs]
[<c0142560>] generic_file_write+0x0/0xd0
[<c01253e2>] __do_softirq+0x62/0xe0
[<f8a62a67>] jfs_open+0x17/0xa0 [jfs]
[<c015e008>] dentry_open+0xe8/0x200
[<c015f4b9>] vfs_writev+0x49/0x50
[<f8af545c>] nfsd_write+0xfc/0x340 [nfsd]
[<c0323045>] ip_map_lookup+0x305/0x360
[<c03232ca>] svcauth_unix_set_client+0xba/0xd0
[<f8afc424>] nfsd3_proc_write+0xc4/0x140 [nfsd]
[<f8afe150>] nfs3svc_decode_writeargs+0x0/0x180 [nfsd]
[<f8af1696>] nfsd_dispatch+0x136/0x200 [nfsd]
[<c0322587>] svc_authenticate+0x87/0xe0
[<c031fae9>] svc_process+0x409/0x620
[<c01049f2>] common_interrupt+0x1a/0x20
[<f8af13d6>] nfsd+0x196/0x320 [nfsd]
[<f8af1240>] nfsd+0x0/0x320 [nfsd]
[<c0102345>] kernel_thread_helper+0x5/0x10
printk: 18 messages suppressed.
nfsd: page allocation failure. order:0, mode:0x20
[<c0144793>] __alloc_pages+0x333/0x3b0
[<c0147342>] kmem_getpages+0x32/0xa0
[<c014809e>] cache_grow+0xbe/0x180
[<c01482cd>] cache_alloc_refill+0x16d/0x240
[<c01485c4>] kmem_cache_alloc+0x44/0x50
[<c02b4803>] alloc_skb+0x23/0xf0
[<f8b0f900>] e1000_alloc_rx_buffers+0x40/0xf0 [e1000]
[<f8b0f690>] e1000_clean_rx_irq+0x280/0x4b0 [e1000]
[<c0119fc3>] recalc_task_prio+0xd3/0x140
[<f8b0f11e>] e1000_clean+0x4e/0x110 [e1000]
[<c02bb0a1>] net_rx_action+0xb1/0x110
[<c01253e2>] __do_softirq+0x62/0xe0
[<c012548d>] do_softirq+0x2d/0x40
[<c01062db>] do_IRQ+0x3b/0x70
[<c01049f2>] common_interrupt+0x1a/0x20
[<c018eba6>] dnotify_parent+0x46/0xa0
[<c01426e0>] generic_file_writev+0x0/0xc0
[<c015f3b8>] do_readv_writev+0x1e8/0x250
[<f8a39950>] export_decode_fh+0x0/0xa5 [exportfs]
[<c0142560>] generic_file_write+0x0/0xd0
[<f8a62a67>] jfs_open+0x17/0xa0 [jfs]
[<c015e008>] dentry_open+0xe8/0x200
[<c015f4b9>] vfs_writev+0x49/0x50
[<f8af545c>] nfsd_write+0xfc/0x340 [nfsd]
[<c0323045>] ip_map_lookup+0x305/0x360
[<c03232ca>] svcauth_unix_set_client+0xba/0xd0
[<f8afc424>] nfsd3_proc_write+0xc4/0x140 [nfsd]
[<f8afe150>] nfs3svc_decode_writeargs+0x0/0x180 [nfsd]
[<f8af1696>] nfsd_dispatch+0x136/0x200 [nfsd]
[<c0322587>] svc_authenticate+0x87/0xe0
[<c031fae9>] svc_process+0x409/0x620
[<c01049f2>] common_interrupt+0x1a/0x20
[<f8af13d6>] nfsd+0x196/0x320 [nfsd]
[<f8af1240>] nfsd+0x0/0x320 [nfsd]
[<c0102345>] kernel_thread_helper+0x5/0x10