2002-04-18 11:56:33

by Mike Black

[permalink] [raw]
Subject: 2.4.17-pre7 oops

I've been having problems resyncing my 2TB RAID5 Ultra160 array for every
kernel version I've tried. Here's the latest.
The drive being resynced is mounted and exported via NFS while it's
resyncing.
I'll try it one more time while it's not exported.

md6 : active raid5 sdm1[12] sdb1[1] sdl1[11] sdk1[10] sdj1[9] sdi1[8]
sdh1[7] sdg1[6] sdf1[5] sde1[4] sdd1[3] sdc1[2]
1950225024 blocks level 5, 128k chunk, algorithm 2 [12/11]
[_UUUUUUUUUUU]
[>....................] recovery = 2.7% (4919104/177293184)
finish=2569.6min speed=1117K/sec

Once the oops occurs on the raid resync thread I end up having to do a hard
reboot

ksymoops 2.4.5 on i686 2.4.19-pre7. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.19-pre7/ (default)
-m /System.map (specified)

Apr 18 07:37:33 yeti kernel: kernel BUG at ll_rw_blk.c:862!
Apr 18 07:37:33 yeti kernel: invalid operand: 0000
Apr 18 07:37:33 yeti kernel: CPU: 1
Apr 18 07:37:33 yeti kernel: EIP: 0010:[__make_request+138/1588]
Tainted: P
Apr 18 07:37:33 yeti kernel: EFLAGS: 00010246
Apr 18 07:37:33 yeti kernel: eax: 00000000 ebx: f5effba0 ecx: 00002000
edx: 00000000
Apr 18 07:37:33 yeti kernel: esi: 00000000 edi: 00000000 ebp: 0a9147fc
esp: f6255e2c
Apr 18 07:37:33 yeti kernel: ds: 0018 es: 0018 ss: 0018
Apr 18 07:37:33 yeti kernel: Process raid5d (pid: 147, stackpage=f6255000)
Apr 18 07:37:33 yeti kernel: Stack: 00000871 f5effba0 00000000 0a9147fc
f7b18e38 00002000 f7b2ce40 f7b18e38
Apr 18 07:37:33 yeti kernel: 00000400 00000000 00000000 00000000
f7b151a0 c018a90c f7b2ce18 00000000
Apr 18 07:37:33 yeti kernel: f5effba0 0000001c 000000c4 f5f18400
00000002 f88265f7 00000000 f5effba0
Apr 18 07:37:33 yeti kernel: Call Trace: [generic_make_request+284/300]
[nfs:__insmod_nfs_O/lib/modules/2.4.19-pre7/kernel/fs/nfs/nfs.o_+-277001/96]
[bh_action+76/136]
[nfs:__insmod_nfs_O/lib/modules/2.4.19-pre7/kernel/fs/nfs/nfs.o_+-275842/96]
[md_thread+341/440]
Apr 18 07:37:33 yeti kernel: Code: 0f 0b 5e 03 62 66 25 c0 53 56 e8 6f be fa
ff 89 c3 0f b6 43
Using defaults from ksymoops -t elf32-i386 -a i386


>>ebx; f5effba0 <_end+35bc22ec/384df74c>
>>ecx; 00002000 Before first symbol
>>ebp; 0a9147fc Before first symbol
>>esp; f6255e2c <_end+35f18578/384df74c>

Code; 00000000 Before first symbol
00000000 <_EIP>:
Code; 00000000 Before first symbol
0: 0f 0b ud2a
Code; 00000002 Before first symbol
2: 5e pop %esi
Code; 00000003 Before first symbol
3: 03 62 66 add 0x66(%edx),%esp
Code; 00000006 Before first symbol
6: 25 c0 53 56 e8 and $0xe85653c0,%eax
Code; 0000000b Before first symbol
b: 6f outsl %ds:(%esi),(%dx)
Code; 0000000c Before first symbol
c: be fa ff 89 c3 mov $0xc389fffa,%esi
Code; 00000011 Before first symbol
11: 0f b6 43 00 movzbl 0x0(%ebx),%eax



________________________________________
Michael D. Black Principal Engineer
[email protected] 321-676-2923,x203
http://www.csihq.com Computer Science Innovations
http://www.csihq.com/~mike My home page
FAX 321-676-2355


2002-04-18 16:41:16

by Mike Black

[permalink] [raw]
Subject: Re: 2.4.17-pre7 oops

Without device even mounted or exported this time:
ksymoops 2.4.5 on i686 2.4.19-pre7. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.19-pre7/ (default)
-m /System.map (specified)

Apr 18 12:08:32 yeti kernel: Unable to handle kernel NULL pointer
dereference at virtual address 00000006
Apr 18 12:08:32 yeti kernel: f8826437
Apr 18 12:08:32 yeti kernel: *pde = 00000000
Apr 18 12:08:32 yeti kernel: Oops: 0000
Apr 18 12:08:32 yeti kernel: CPU: 1
Apr 18 12:08:32 yeti kernel: EIP:
0010:[nfs:__insmod_nfs_O/lib/modules/2.4.19-pre7/kernel/fs/nfs/nfs.o_+-27744
9/96] Tainted: P
Apr 18 12:08:32 yeti kernel: EFLAGS: 00010246
Apr 18 12:08:32 yeti kernel: eax: 00000000 ebx: f5870dc0 ecx: 00000002
edx: f5daed60
Apr 18 12:08:32 yeti kernel: esi: f5ab7c00 edi: f5daec00 ebp: f5daec00
esp: f59fbe88
Apr 18 12:08:32 yeti kernel: ds: 0018 es: 0018 ss: 0018
Apr 18 12:08:32 yeti kernel: Process raid5d (pid: 17544, stackpage=f59fb000)
Apr 18 12:08:32 yeti kernel: Stack: 00000000 f5daec00 f5ab7c00 f5fa1de0
00000001 00000001 000000f8 c011cf73
Apr 18 12:08:32 yeti kernel: 00000000 c009ec00 f5ab7d58 f59fbf24
f5ab7cec f5ab7c80 f5daec14 00000000
Apr 18 12:08:32 yeti kernel: f5ab7c14 00000000 00000001 00000001
00000001 0000000c 00000001 ffffffff
Apr 18 12:08:32 yeti kernel: Call Trace: [tasklet_hi_action+103/160]
[nfs:__insmod_nfs_O/lib/modules/2.4.19-pre7/kernel/fs/nfs/nfs.o_+-275842/96]
[md_thread+341/440] [kernel_thread+40/56]
Apr 18 12:08:32 yeti kernel: Code: 8b 41 04 50 e8 d8 5a 9a c7 00 00 00 00 74
51 83 7c 24 7c 00
Using defaults from ksymoops -t elf32-i386 -a i386


>>ebx; f5870dc0 <_end+3553350c/384df74c>
>>edx; f5daed60 <_end+35a714ac/384df74c>
>>esi; f5ab7c00 <_end+3577a34c/384df74c>
>>edi; f5daec00 <_end+35a7134c/384df74c>
>>ebp; f5daec00 <_end+35a7134c/384df74c>
>>esp; f59fbe88 <_end+356be5d4/384df74c>

Code; 00000000 Before first symbol
00000000 <_EIP>:
Code; 00000000 Before first symbol
0: 8b 41 04 mov 0x4(%ecx),%eax
Code; 00000003 Before first symbol
3: 50 push %eax
Code; 00000004 Before first symbol
4: e8 d8 5a 9a c7 call c79a5ae1 <_EIP+0xc79a5ae1> c79a5ae1
<_end+766822d/384df74c>
Code; 00000009 Before first symbol
9: 00 00 add %al,(%eax)
Code; 0000000b Before first symbol
b: 00 00 add %al,(%eax)
Code; 0000000d Before first symbol
d: 74 51 je 60 <_EIP+0x60> 00000060 Before
first symbol
Code; 0000000f Before first symbol
f: 83 7c 24 7c 00 cmpl $0x0,0x7c(%esp,1)

________________________________________
Michael D. Black Principal Engineer
[email protected] 321-676-2923,x203
http://www.csihq.com Computer Science Innovations
http://www.csihq.com/~mike My home page
FAX 321-676-2355
----- Original Message -----
From: "Mike Black" <[email protected]>
To: "linux-kernel" <[email protected]>
Sent: Thursday, April 18, 2002 7:56 AM
Subject: 2.4.17-pre7 oops


I've been having problems resyncing my 2TB RAID5 Ultra160 array for every
kernel version I've tried. Here's the latest.
The drive being resynced is mounted and exported via NFS while it's
resyncing.
I'll try it one more time while it's not exported.



2002-04-18 23:08:32

by Keith Owens

[permalink] [raw]
Subject: Re: 2.4.17-pre7 oops

On Thu, 18 Apr 2002 12:40:59 -0400,
"Mike Black" <[email protected]> wrote:
>Apr 18 12:08:32 yeti kernel: Unable to handle kernel NULL pointer
>dereference at virtual address 00000006
>Apr 18 12:08:32 yeti kernel: f8826437
>Apr 18 12:08:32 yeti kernel: *pde = 00000000
>Apr 18 12:08:32 yeti kernel: Oops: 0000
>Apr 18 12:08:32 yeti kernel: CPU: 1
>Apr 18 12:08:32 yeti kernel: EIP:
>0010:[nfs:__insmod_nfs_O/lib/modules/2.4.19-pre7/kernel/fs/nfs/nfs.o_+-27744
>9/96] Tainted: P
>Apr 18 12:08:32 yeti kernel: EFLAGS: 00010246
>Apr 18 12:08:32 yeti kernel: eax: 00000000 ebx: f5870dc0 ecx: 00000002
>edx: f5daed60
>Apr 18 12:08:32 yeti kernel: esi: f5ab7c00 edi: f5daec00 ebp: f5daec00
>esp: f59fbe88
>Apr 18 12:08:32 yeti kernel: ds: 0018 es: 0018 ss: 0018
>Apr 18 12:08:32 yeti kernel: Process raid5d (pid: 17544, stackpage=f59fb000)
>Apr 18 12:08:32 yeti kernel: Stack: 00000000 f5daec00 f5ab7c00 f5fa1de0
>00000001 00000001 000000f8 c011cf73
>Apr 18 12:08:32 yeti kernel: 00000000 c009ec00 f5ab7d58 f59fbf24
>f5ab7cec f5ab7c80 f5daec14 00000000
>Apr 18 12:08:32 yeti kernel: f5ab7c14 00000000 00000001 00000001
>00000001 0000000c 00000001 ffffffff
>Apr 18 12:08:32 yeti kernel: Call Trace: [tasklet_hi_action+103/160]
>[nfs:__insmod_nfs_O/lib/modules/2.4.19-pre7/kernel/fs/nfs/nfs.o_+-275842/96]
>[md_thread+341/440] [kernel_thread+40/56]
>Apr 18 12:08:32 yeti kernel: Code: 8b 41 04 50 e8 d8 5a 9a c7 00 00 00 00 74
>51 83 7c 24 7c 00
>Using defaults from ksymoops -t elf32-i386 -a i386
>
>
>>>ebx; f5870dc0 <_end+3553350c/384df74c>
>>>edx; f5daed60 <_end+35a714ac/384df74c>
>>>esi; f5ab7c00 <_end+3577a34c/384df74c>
>>>edi; f5daec00 <_end+35a7134c/384df74c>
>>>ebp; f5daec00 <_end+35a7134c/384df74c>
>>>esp; f59fbe88 <_end+356be5d4/384df74c>

I do not trust that ksymoops output at all. To start with, klogd has
stamped on the real numbers and replaced them with useless lookups.
Always disable klogd oops decoding, run as 'klogd -x'. Secondly the
registers make no sense, it appears that you ran ksymoops against a
kernel without the modules loaded so the address lookup is meaningless.
You need to tell ksymoops which modules were loaded at the time of the
oops and where they were. man insmod and look for /var/log/ksymoops.