Hi,
I have a 16 node cluster with dual Intel 82557 network controllers (eepro100) driver. All nodes have
RedHat 7.1 installed and using the latest kernel update from RH (2.4.9.-12). Here's a bit from dmesg
:
eepro100.c:v1.09j-t 9/29/99 Donald Becker http://cesdis.gsfc.nasa.gov/linux/drivers/eepro100.html
eepro100.c: $Revision: 1.36 $ 2000/11/17 Modified by Andrey V. Savochkin <[email protected]> and
others
eth0: OEM i82557/i82558 10/100 Ethernet, 00:30:48:11:3E:53, IRQ 31.
Board assembly 000000-000, Physical connectors present: RJ45
Primary interface chip i82555 PHY #1.
General self-test: passed.
Serial sub-system self-test: passed.
Internal registers self-test: passed.
ROM checksum self-test: passed (0x04f4518b).
eth1: OEM i82557/i82558 10/100 Ethernet, 00:30:48:11:3C:44, IRQ 28.
Board assembly 000000-000, Physical connectors present: RJ45
Primary interface chip i82555 PHY #1.
General self-test: passed.
Serial sub-system self-test: passed.
Internal registers self-test: passed.
ROM checksum self-test: passed (0x04f4518b).
Sometimes under heavy load one of the nodes (random node each time) crash with the following Oops :
Unable to handle kernel paging request at virtual address 0000d5ca
printing eip:
c0200002
*pde = 00000000
Oops: 0002
CPU: 1
EIP: 0010:[<c0200002>] Not tainted
EFLAGS: 00010282
eax: 00000000 ebx: 00000001 ecx: dfa36000 edx: c0311840
esi: dfa36000 edi: 0000d5c2 ebp: df7123c0 esp: c9bc5ee0
ds: 0018 es: 0018 ss: 0018
Process mpi_bomb (pid: 16020, stackpage=c9bc5000)
Stack: 00000001 df7123c0 00000001 df7123c0 c01d109c 00000108 00000002 00000020
0000003c c01ccd6c df7123c0 df7123c0 00000001 00000003 dfa36000 e0902188
dfa36000 00000001 dfa36144 0000000c 00000001 00000001 df7123c0 df7123c0
Call Trace: [<c01d109c>] netif_rx [kernel] 0x8c
[<c01ccd6c>] alloc_skb [kernel] 0xfc
[<e0902188>] __insmod_eepro100_S.text_L11712 [eepro100] 0x2128
[<c01d154b>] net_rx_action [kernel] 0x1eb
[<c011f74b>] do_softirq [kernel] 0x7b
[<c0108c4d>] do_IRQ [kernel] 0xdd
[<c022a550>] call_do_IRQ [kernel] 0x5
Code: 81 47 08 89 44 24 28 b8 b8 1a 31 c0 0f b7 5e 5c f0 83 28 01
<0>Kernel panic: Aiee, killing interrupt handler!
In interrupt handler - not syncing
Is this a known problem with the eepro100 driver in 2.4.9 and fixed in later kernels, or is it a
generic network driver problem (maybe VM related since the machine might be stressed on memory). I
also wonder why the process running at the time (mpi_bomb) shows up in the Oops.
Feedback highly appreciated.
Thanks,
--
Steffen Persvold | Scalable Linux Systems | Try out the world's best
mailto:[email protected] | http://www.scali.com | performing MPI implementation:
Tel: (+47) 2262 8950 | Olaf Helsets vei 6 | - ScaMPI 1.12.2 -
Fax: (+47) 2262 8951 | N0621 Oslo, NORWAY | >300MBytes/s and <4uS latency
What I can say is that your oops look quite strange.
What is located at address c0200002 in your vmlinux file?
The code 81 47 08 89 44 24 28... looks suspicious:
0: 81 47 08 89 44 24 28 addl $0x28244489,0x8(%edi)
For example, if it were 89 47 08 89 44 24 28..., it would be
0: 89 47 08 movl %eax,0x8(%edi)
3: 89 44 24 28 movl %eax,0x28(%esp,1)
You might have just a severe memory corruption.
Andrey
On Sat, Dec 22, 2001 at 11:55:13AM +0100, Steffen Persvold wrote:
> Sometimes under heavy load one of the nodes (random node each time) crash with the following Oops :
>
> Unable to handle kernel paging request at virtual address 0000d5ca
> printing eip:
> c0200002
> *pde = 00000000
> Oops: 0002
> CPU: 1
> EIP: 0010:[<c0200002>] Not tainted
> EFLAGS: 00010282
> eax: 00000000 ebx: 00000001 ecx: dfa36000 edx: c0311840
> esi: dfa36000 edi: 0000d5c2 ebp: df7123c0 esp: c9bc5ee0
> ds: 0018 es: 0018 ss: 0018
> Process mpi_bomb (pid: 16020, stackpage=c9bc5000)
> Stack: 00000001 df7123c0 00000001 df7123c0 c01d109c 00000108 00000002 00000020
> 0000003c c01ccd6c df7123c0 df7123c0 00000001 00000003 dfa36000 e0902188
> dfa36000 00000001 dfa36144 0000000c 00000001 00000001 df7123c0 df7123c0
> Call Trace: [<c01d109c>] netif_rx [kernel] 0x8c
> [<c01ccd6c>] alloc_skb [kernel] 0xfc
> [<e0902188>] __insmod_eepro100_S.text_L11712 [eepro100] 0x2128
> [<c01d154b>] net_rx_action [kernel] 0x1eb
> [<c011f74b>] do_softirq [kernel] 0x7b
> [<c0108c4d>] do_IRQ [kernel] 0xdd
> [<c022a550>] call_do_IRQ [kernel] 0x5
>
>
> Code: 81 47 08 89 44 24 28 b8 b8 1a 31 c0 0f b7 5e 5c f0 83 28 01