2006-08-03 15:42:41

by Richard Mollel

[permalink] [raw]

Subject: Kernel Hangs, EIP is at scsi_decide_disposition

---

Our mandriva boxes are randomly crushing after
migration to 2006.0 download version. The same
hardware has had mandriva 10.0 withuptimes of over 230
days....but nowadays it is hardly a week before a
crash for over 6 nodes with latest kernels.

I captured one of our lockups and am attaching the
output of ksymoops for further interpretation by mroe
experienced eyes when it comes to kernel internals.

Here is the actual error, and attached, the ksymoops
interpretation of it.

Unable to handle kernel NULL pointer dereference at
virtual address 000002a9
printing eip:
f88642bd
*pde = 3702b001
Oops: 0000 [#1]
SMP
Modules linked in: sg nfsd exportfs raw md5 ipv6 nfs
lockd nfs_acl sunrpc parport_pc lp parport e1000
af_packet floppy video thermal tc1100_wmi processor
fan container button battery ac ipt_LOG ipt_REJECT
ipt_state ip_conntrack iptable_filter ip_tables ide_cd
loop hw_random 3w_9xxx tsdev sr_mod ehci_hcd uhci_hcd
usbcore evdev reiserfs sd_mod ahci ata_piix libata
scsi_mod
CPU: 0
EIP: 0060:[<f88642bd>] Tainted: G M VLI
EFLAGS: 00010282 (2.6.12-22mdksmp)
EIP is at scsi_decide_disposition+0xd/0x160 [scsi_mod]
eax: c0437f2c ebx: f7c3902c ecx: 00000001 edx:
f7db6e18
esi: f7c3902c edi: c0437f2c ebp: c0437f1c esp:
c0437f04
ds: 007b es: 007b ss: 0068
Process swapper (pid: 0, threadinfo=c0436000
task=c03d6bc0)
Stack: 00000001 00000000 00000000 00002002 f7da6680
00002002 c0437f40 f885fc71
f7c3902c 00002002 f7db6e18 f7db6818 00000001
c04331e0 c047ef00 c0437f6c
c0181ab2 c04331e0 0000000a 00000000 c047c380
c0436000 c047c380 00000046
Call Trace:
[<c010426b>] show_stack+0xab/0xf0
[<c010444f>] show_registers+0x17f/0x220
[<c0104694>] die+0xf4/0x180
[<c011bf71>] do_page_fault+0x261/0x793
[<c0103e6f>] error_code+0x4f/0x54
[<f885fc71>] scsi_softirq+0x71/0xf0 [scsi_mod]
[<c0181ab2>] __do_softirq+0x82/0xf0
[<c0181b57>] do_softirq+0x37/0x40
[<c0105b11>] do_IRQ+0x21/0x30
[<c0103d16>] common_interrupt+0x1a/0x20
[<c010112f>] cpu_idle+0x6f/0xc0
[<c0438a7a>] start_kernel+0x17a/0x1e0
[<c010020f>] 0xc010020f
Code: 24 89 44 24 04 e8 f4 f2 ff ff 39 fe 8b 16 89 f0
75 a2 83 c4 14 5b 5e 5f 5d c3 8d 74 26 00 55 89 e5 53
83 ec 14 8b 5d 08 8b 4b 04 <83> b9 a8 02 00 00 06 74
2a 8b 93 4c 01 00 00 89 d0 c1 f8 10 25
<0>Kernel panic - not syncing: Fatal exception in
interrupt


# lspci
00:00.0 Host bridge: Intel Corporation E7520 Memory
Controller Hub (rev 0c)
00:00.1 Class ff00: Intel Corporation E7525/E7520
Error Reporting Registers (rev 0c)
00:02.0 PCI bridge: Intel Corporation
E7525/E7520/E7320 PCI Express Port A (rev 0c)
00:03.0 PCI bridge: Intel Corporation
E7525/E7520/E7320 PCI Express Port A1 (rev 0c)
00:04.0 PCI bridge: Intel Corporation E7525/E7520 PCI
Express Port B (rev 0c)
00:06.0 PCI bridge: Intel Corporation E7520 PCI
Express Port C (rev 0c)
00:1d.0 USB Controller: Intel Corporation 82801EB/ER
(ICH5/ICH5R) USB UHCI Controller #1 (rev 02)
00:1d.1 USB Controller: Intel Corporation 82801EB/ER
(ICH5/ICH5R) USB UHCI Controller #2 (rev 02)
00:1d.2 USB Controller: Intel Corporation 82801EB/ER
(ICH5/ICH5R) USB UHCI Controller #3 (rev 02)
00:1d.3 USB Controller: Intel Corporation 82801EB/ER
(ICH5/ICH5R) USB UHCI Controller #4 (rev 02)
00:1d.7 USB Controller: Intel Corporation 82801EB/ER
(ICH5/ICH5R) USB2 EHCI Controller (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge
(rev c2)
00:1f.0 ISA bridge: Intel Corporation 82801EB/ER
(ICH5/ICH5R) LPC Interface Bridge (rev 02)
00:1f.2 IDE interface: Intel Corporation 82801EB
(ICH5) SATA Controller (rev 02)
00:1f.3 SMBus: Intel Corporation 82801EB/ER
(ICH5/ICH5R) SMBus Controller (rev 02)
02:00.0 PCI bridge: Intel Corporation 6700PXH PCI
Express-to-PCI Bridge A (rev 09)
02:00.1 PIC: Intel Corporation 6700/6702PXH I/OxAPIC
Interrupt Controller A (rev 09)
02:00.2 PCI bridge: Intel Corporation 6700PXH PCI
Express-to-PCI Bridge B (rev 09)
02:00.3 PIC: Intel Corporation 6700PXH I/OxAPIC
Interrupt Controller B (rev 09)
03:03.0 RAID bus controller: 3ware Inc: Unknown device
1003
04:01.0 SCSI storage controller: LSI Logic / Symbios
Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev
08)
04:02.0 Ethernet controller: Intel Corporation 82546GB
Gigabit Ethernet Controller (rev 03)
04:02.1 Ethernet controller: Intel Corporation 82546GB
Gigabit Ethernet Controller (rev 03)
07:01.0 VGA compatible controller: ATI Technologies
Inc Rage XL (rev 27)


#cat /etc/redhat-release
Mandriva Linux release 2006.0 (Official) for i586

Kernel::
2.6.12-22mdksmp #1 SMP Tue May 23 15:13:55 MDT 2006
i686 Intel(R) Xeon(TM) CPU 3.40GHz




DEBUG (parse): 'v'
'/usr/src/linux-2.6.12-22mdk/vmlinux'
DEBUG (parse): 'm'
'/usr/src/linux-2.6.12-22mdk/System.map'
DEBUG (convert_uname): /lib/modules/*r/ in
DEBUG (convert_uname): /lib/modules/2.6.12-22mdksmp/
out
ksymoops 2.4.9 on i686 2.6.12-22mdksmp. Options used
-v /usr/src/linux-2.6.12-22mdk/vmlinux
(specified)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.6.12-22mdksmp/ (default)
-m /usr/src/linux-2.6.12-22mdk/System.map
(specified)

DEBUG (main): level 3
DEBUG (read_env): override
KSYMOOPS_NM=/usr/bin/ksymoops-gznm
DEBUG (read_env): default KSYMOOPS_FIND=/usr/bin/find
DEBUG (read_env): default
KSYMOOPS_OBJDUMP=/usr/bin/objdump
DEBUG (re_compile): '^([0-9a-fA-F]{4,}) +([^ ]) +([^
]+)( +\[([^ ]+)\])?$' 5 sub expression(s)
DEBUG (re_compile): '^ *\[*<([0-9a-fA-F]{4,})>\]* *' 1
sub expression(s)
DEBUG (re_compile): '^ *<\[([0-9a-fA-F]{4,})\]> *' 1
sub expression(s)
DEBUG (re_compile): '^ *([0-9a-fA-F]{4,}) *' 1 sub
expression(s)
DEBUG (read_nm_symbols): command
'/usr/bin/ksymoops-gznm
/usr/src/linux-2.6.12-22mdk/vmlinux'
DEBUG (re_compile): '^(.*)_R.*[0-9a-fA-F]{8,}$' 1 sub
expression(s)
DEBUG (read_nm_symbols): vmlinux used 20829 out of
23952 entries
DEBUG (ss_sort_na): vmlinux
DEBUG (ss_compress): table vmlinux, before 20829
DEBUG (ss_compress): table vmlinux, after 20829
DEBUG (ss_compress): table vmlinux, before 20829
DEBUG (ss_compress): table vmlinux, after 20829
DEBUG (add_Version): vmlinux 132620 2.6.12
DEBUG (read_ksyms): /proc/ksyms
Error (regular_file): read_ksyms stat /proc/ksyms
failed
No modules in ksyms, skipping objects
No ksyms, skipping lsmod
DEBUG (read_system_map):
/usr/src/linux-2.6.12-22mdk/System.map
DEBUG (ss_sort_na): System.map
DEBUG (ss_compress): table System.map, before 20829
DEBUG (ss_compress): table System.map, after 20829
DEBUG (ss_compress): table System.map, before 20829
DEBUG (ss_compress): table System.map, after 20829
DEBUG (add_Version): System.map 132620 2.6.12
DEBUG (read_system_map): System.map used 20829 out of
23952 entries
DEBUG (merge_maps):
DEBUG (ss_sort_na): Version_
DEBUG (ss_compress): table Version_, before 2
DEBUG (ss_compress): table Version_, after 2
DEBUG (ss_compress): table Version_, before 2
DEBUG (ss_compress): table Version_, after 2
DEBUG (compare_Version): Version 2.6.12
DEBUG (compare_maps): System.map vs vmlinux, vmlinux
takes precedence
DEBUG (compare_maps): vmlinux vs System.map, vmlinux
takes precedence
DEBUG (append_map): vmlinux to merged
DEBUG (append_map): System.map to merged
DEBUG (ss_sort_atn): merged
DEBUG (ss_compress): table merged, before 41658
DEBUG (ss_compress): table merged, after 41658
DEBUG (ss_sort_atn): merged
DEBUG (ss_compress): table merged, before 41658
DEBUG (ss_compress): table merged, after 20829
DEBUG (find_fullpath): /usr/bin/ksymoops.real
DEBUG (Oops_read): Unable to handle kernel NULL
pointer dereference at virtual address 000002a9
DEBUG (re_compile): '^( +|[^ ]{3} [ 0-9][0-9]
[0-9]{2}:[0-9]{2}:[0-9]{2} [^ ]+ kernel:
+|<[0-9]+>|[0-9]+\|[^|]+\|[^|]+\|[^|]+\||[0-9][AC]
[0-9]{3} [0-9]{3}[cir][0-9]{2}:\+)' 1 sub
expression(s)
DEBUG (re_compile): '^((Stack: )|(Stack from
)|([0-9a-fA-F]{4,})|(Call Trace:
*)|(\[*<([0-9a-fA-F]{4,})>\]*
*)|(Version_[0-9]+)|(Trace:)|(Call
backtrace:)|(bh:)|<\[([0-9a-fA-F]{4,})\]> *|(\([^ ]+\)
*\[*<([0-9a-fA-F]{4,})>\]* *)|([0-9]+ +base=)|(Kernel
BackChain)|EBP *EIP|0x([0-9a-fA-F]{4,})
*0x([0-9a-fA-F]{4,}) *|Process .*stackpage=|Code *:
|Kernel panic|In swapper
task|kmem_free|swapper|Pid:|r[0-9]{1,2}
*[:=]|Corrupted stack page|invalid operand: |Oops:
|Cpu:* +[0-9]|current->tss|\*pde +=|EIP: |EFLAGS:
|eax: |esi: |ds: |CR0: |wait_on_|irq: |Stack
dumps:|RAX: |RSP: |RIP: |RDX: |RBP: |FS: |CS: |CR2:
|PML4|pc[:=]|68060 access|Exception at |d[04]: |Frame
format=|wb [0-9] stat|push data: |baddr=|Arithmetic
fault|Instruction fault|Bad unaligned
kernel|Forwarding unaligned exception|: unhandled
unaligned exception|pc *=|[rvtsa][0-9]+ *=|gp
*=|spinlock stuck|tsk->|PSR: |[goli]0: |Instruction
DUMP: |spin_lock|TSTATE: |[goli]4:
|Caller\[|CPU\[|MSR: |TASK = |last math |GPR[0-9]+:
|.* in .*\.[ch]:.*, line [0-9]+:|\$[0-9 ]+:|Hi *:|Lo
*:|epc *:|badvaddr *:|Status *:|Cause
*:|Backtrace:|Function entered at|\*pgd =|Internal
error|pc :|sp
:|Flags:|Control:|WARNING:|this_stack:|i:|PSW|cr[0-9]+:|machine
check|Exception in |Program Check |System restart
|IUCV |unexpected external |Kernel stack |User process
fault:|failing address|User PSW|User GPRS|User
ACRS|Kernel PSW|Kernel GPRS|Kernel ACRS|illegal
operation|task:|Entering kdb|eax *=|esi *=|ebp *=|ds
*=|psr |unat |rnat |ldrs |xip |iip |ipsr |ifa |pr
|itc |ifs |bsp |[bfr][0-9]+ |irr[0-9] |General
Exception|MCA|SAL|Processor State|Bank [0-9]+|Register
Stack|Processor Error Info|proc err map|proc state
param|proc lid|processor structure|check info|target
identifier|IRP *: )' 18 sub expression(s)
DEBUG (re_compile): 'Unable to handle
kernel|Aiee|die_if_kernel|NMI |BUG
|\(L-TLB\)|\(NOTLB\)|\([0-9]\): Oops |: memory
violation|: Exception at|: Arithmetic fault|:
Instruction fault|: arithmetic trap|: unaligned
trap|\([0-9]+\):
(Whee|Oops|Kernel|.*Penguin|BOGUS)|kernel pc |trap at
PC: |bad area pc |NIP: | ra *==' 1 sub expression(s)
DEBUG (re_compile): '^(i[04]: |Instruction DUMP:
|Caller\[)' 1 sub expression(s)
Unable to handle kernel NULL pointer dereference at
virtual address 000002a9
DEBUG (re_compile): '^PSR: [0-9a-fA-F]+ PC:
([0-9a-fA-F]{4,}) *' 1 sub expression(s)
DEBUG (re_compile): '^TSTATE: [0-9a-fA-F]{16} TPC:
([0-9a-fA-F]{4,}) *' 1 sub expression(s)
DEBUG (re_compile): '(kernel pc |trap at PC: |bad area
pc |NIP: )([0-9a-fA-F]{4,}) *' 2 sub expression(s)
DEBUG (re_compile): '^epc *:+ *([0-9a-fA-F]{4,}) *' 1
sub expression(s)
DEBUG (re_compile): ' ip *:+
*\[*<([0-9a-fA-F]{4,})>\]* *' 1 sub expression(s)
DEBUG (re_compile): '[xi]ip *\([^)]*\) *:
*0x([0-9a-fA-F]{4,}) *' 1 sub expression(s)
DEBUG (re_compile): '(^0x([0-9a-fA-F]{4,})
*0x([0-9a-fA-F]{4,}) *.*+0x)|(^Entering kdb on
processor.*0x([0-9a-fA-F]{4,}) *)' 5 sub expression(s)
DEBUG (re_compile): '^ *IRP *: *([0-9a-fA-F]{4,}) *' 1
sub expression(s)
DEBUG (re_compile): '^(EIP: +.*|RIP: +.*|PC *= *|pc *:
*)\[*<([0-9a-fA-F]{4,})>\]* *' 2 sub expression(s)
DEBUG (re_compile): '^spinlock stuck at
([0-9a-fA-F]{4,}) *.*owner.*at ([0-9a-fA-F]{4,}) *' 2
sub expression(s)
DEBUG (re_compile): '^spin_lock[^
]*\(([0-9a-fA-F]{4,}) *.*stuck at *([0-9a-fA-F]{4,})
*.*PC\(([0-9a-fA-F]{4,}) *' 3 sub expression(s)
DEBUG (re_compile): 'ra *=+ *([0-9a-fA-F]{4,}) *' 1
sub expression(s)
DEBUG (re_compile): '^(\$[0-9]{1,2}) *:
*([0-9a-fA-F]{4,}) *' 2 sub expression(s)
DEBUG (re_compile): '^(GPR[0-9]{1,2}) *:
*([0-9a-fA-F]{4,}) *' 2 sub expression(s)
DEBUG (re_compile): '^PSW *flags: *([0-9a-fA-F]{4,}) *
*PSW addr: *([0-9a-fA-F]{4,}) *' 2 sub expression(s)
DEBUG (re_compile): '^Kernel PSW: *([0-9a-fA-F]{4,})
*([0-9a-fA-F]{4,}) *' 2 sub expression(s)
DEBUG (re_compile): '^((Call Trace:
*)|(Trace:)|(\[*<([0-9a-fA-F]{4,})>\]* *)|(Call
backtrace:)|([0-9a-fA-F]{4,}) *|Function entered at
(\[*<([0-9a-fA-F]{4,})>\]*
*)|Caller\[([0-9a-fA-F]{4,})
*\]|(<\[([0-9a-fA-F]{4,})\]> *)|(\([0-9]+\)
*(\[*<([0-9a-fA-F]{4,})>\]* *))|([0-9]+
+base=0x([0-9a-fA-F]{4,}) *)|(Kernel BackChain.*))' 18
sub expression(s)
DEBUG (re_compile):
'^((GP|o)?r[0-9]{1,2}|[goli][0-9]{1,2}|[eR][ABCD]X|[eR][DS]I|RBP|e[bs]p|[fsi]p|IRP|SRP|D?CCR|USP|MOF|ret_pc)
*[:=] *([0-9a-fA-F]{4,}) * *' 3 sub expression(s)
DEBUG (re_compile): '^(Kernel GPRS.*)' 1 sub
expression(s)
DEBUG (re_compile): '^(IRP|SRP|D?CCR|USP|MOF):
*([0-9a-fA-F]{4,}) *' 2 sub expression(s)
DEBUG (re_compile): '^b[0-7] *(\([^)]*\) *)?:
*(0x)?([0-9a-fA-F]{4,}) *' 3 sub expression(s)
DEBUG (re_compile): '^([^ ]+) +[^ ] *([^
]+).*\((L-TLB|NOTLB)\)' 3 sub expression(s)
DEBUG (re_compile): '^((Instruction DUMP)|(Code:? *)):
+((general protection.*)|(Bad E?IP
value.*)|(<[0-9]+>)|(([<(]?[0-9a-fA-F]+[>)]?
+)+[<(]?[0-9a-fA-F]+[>)]?))(.*)$' 10 sub expression(s)
DEBUG (Oops_read): printing eip:
DEBUG (Oops_read): f88642bd
f88642bd
DEBUG (Oops_read): *pde = 3702b001
*pde = 3702b001
DEBUG (Oops_read): Oops: 0000 [#1]
Oops: 0000 [#1]
DEBUG (Oops_read): SMP
DEBUG (Oops_read): Modules linked in: sg nfsd exportfs
raw md5 ipv6 nfs lockd nfs_acl sunrpc parport_pc lp
parport e1000 af_packet floppy video thermal
tc1100_wmi processor fan container button battery ac
ipt_LOG ipt_REJECT ipt_state ip_conntrack
iptable_filter ip_tables ide_cd loop hw_random arcmsr
tsdev sr_mod ehci_hcd uhci_hcd usbcore evdev reiserfs
sd_mod ahci ata_piix libata scsi_mod
DEBUG (Oops_read): CPU: 0
CPU: 0
DEBUG (Oops_read): EIP: 0060:[<f88642bd>]
Tainted: G M VLI
EIP: 0060:[<f88642bd>] Tainted: G M VLI
DEBUG (re_compile): 'pc *: *\[*<([0-9a-fA-F]{4,})>\]*
* *lr *: *\[*<([0-9a-fA-F]{4,})>\]* *' 2 sub
expression(s)
DEBUG (re_compile): 'pc *= *\[*<([0-9a-fA-F]{4,})>\]*
* *ra *= *\[*<([0-9a-fA-F]{4,})>\]* *' 2 sub
expression(s)
Using defaults from ksymoops -t elf32-i386 -a i386
DEBUG (Oops_read): EFLAGS: 00010282
(2.6.12-22mdksmp)
EFLAGS: 00010282 (2.6.12-22mdksmp)
DEBUG (Oops_read): EIP is at
scsi_decide_disposition+0xd/0x160 [scsi_mod]
DEBUG (Oops_read): eax: c0437f2c ebx: f7c3902c
ecx: 00000001 edx: f7db6e18
eax: c0437f2c ebx: f7c3902c ecx: 00000001 edx:
f7db6e18
DEBUG (Oops_read): esi: f7c3902c edi: c0437f2c
ebp: c0437f1c esp: c0437f04
esi: f7c3902c edi: c0437f2c ebp: c0437f1c esp:
c0437f04
DEBUG (Oops_read): ds: 007b es: 007b ss: 0068
ds: 007b es: 007b ss: 0068
DEBUG (Oops_read): Process swapper (pid: 0,
threadinfo=c0436000 task=c03d6bc0)
DEBUG (Oops_read): Stack: 00000001 00000000 00000000
00002002 f7da6680 00002002 c0437f40 f885fc71
Stack: 00000001 00000000 00000000 00002002 f7da6680
00002002 c0437f40 f885fc71
DEBUG (Oops_read): f7c3902c 00002002 f7db6e18
f7db6818 00000001 c04331e0 c047ef00 c0437f6c
f7c3902c 00002002 f7db6e18 f7db6818 00000001
c04331e0 c047ef00 c0437f6c
DEBUG (Oops_read): c0181ab2 c04331e0 0000000a
00000000 c047c380 c0436000 c047c380 00000046
c0181ab2 c04331e0 0000000a 00000000 c047c380
c0436000 c047c380 00000046
DEBUG (Oops_read): Call Trace:
Call Trace:
DEBUG (re_compile): '^(\([0-9]+\) *)' 1 sub
expression(s)
DEBUG (Oops_read): [<c010426b>] show_stack+0xab/0xf0
[<c010426b>] show_stack+0xab/0xf0
DEBUG (Oops_read): [<c010444f>]
show_registers+0x17f/0x220
[<c010444f>] show_registers+0x17f/0x220
DEBUG (Oops_read): [<c0104694>] die+0xf4/0x180
[<c0104694>] die+0xf4/0x180
DEBUG (Oops_read): [<c011bf71>]
do_page_fault+0x261/0x793
[<c011bf71>] do_page_fault+0x261/0x793
DEBUG (Oops_read): [<c0103e6f>] error_code+0x4f/0x54
[<c0103e6f>] error_code+0x4f/0x54
DEBUG (Oops_read): [<f885fc71>]
scsi_softirq+0x71/0xf0 [scsi_mod]
[<f885fc71>] scsi_softirq+0x71/0xf0 [scsi_mod]
DEBUG (Oops_read): [<c0181ab2>]
__do_softirq+0x82/0xf0
[<c0181ab2>] __do_softirq+0x82/0xf0
DEBUG (Oops_read): [<c0181b57>] do_softirq+0x37/0x40
[<c0181b57>] do_softirq+0x37/0x40
DEBUG (Oops_read): [<c0105b11>] do_IRQ+0x21/0x30
[<c0105b11>] do_IRQ+0x21/0x30
DEBUG (Oops_read): [<c0103d16>]
common_interrupt+0x1a/0x20
[<c0103d16>] common_interrupt+0x1a/0x20
DEBUG (Oops_read): [<c010112f>] cpu_idle+0x6f/0xc0
[<c010112f>] cpu_idle+0x6f/0xc0
DEBUG (Oops_read): [<c0438a7a>]
start_kernel+0x17a/0x1e0
[<c0438a7a>] start_kernel+0x17a/0x1e0
DEBUG (Oops_read): [<c010020f>] 0xc010020f
[<c010020f>] 0xc010020f
DEBUG (Oops_read): Code: 24 89 44 24 04 e8 f4 f2 ff ff
39 fe 8b 16 89 f0 75 a2 83 c4 14 5b 5e 5f 5d c3 8d 74
26 00 55 89 e5 53 83 ec 14 8b 5d 08 8b 4b 04 <83> b9
a8 02 00 00 06 74 2a 8b 93 4c 01 00 00 89 d0 c1 f8 10
25
Code: 24 89 44 24 04 e8 f4 f2 ff ff 39 fe 8b 16 89 f0
75 a2 83 c4 14 5b 5e 5f 5d c3 8d 74 26 00 55 89 e5 53
83 ec 14 8b 5d 08 8b 4b 04 <83> b9 a8 02 00 00 06 74
2a 8b 93 4c 01 00 00 89 d0 c1 f8 10 25
DEBUG (Oops_decode):
DEBUG (re_compile): '^(([<(]?)([0-9a-fA-F]+)[)>]?
*)|(Bad .*)' 4 sub expression(s)
DEBUG (Oops_decode_part):
DEBUG (Oops_objdump): command '/usr/bin/objdump -dhf
/root/tmp/ksymoops.eowtTC'
DEBUG (Oops_decode_part):
DEBUG (re_compile): '^ *([0-9a-fA-F]+)(
<_XXX[^>]*>)?:(.* +<_XXX\+0?x?([0-9a-fA-F]+)> *$)?.*'
4 sub expression(s)
DEBUG (Oops_decode_part): /root/tmp/ksymoops.eowtTC:
file format elf32-i386
DEBUG (Oops_decode_part): architecture: i386, flags
0x00000010:
DEBUG (Oops_decode_part): HAS_SYMS
DEBUG (Oops_decode_part): start address 0x00000000
DEBUG (Oops_decode_part):
DEBUG (Oops_decode_part): Sections:
DEBUG (Oops_decode_part): Idx Name Size
VMA LMA File off Algn
DEBUG (Oops_decode_part): 0 .text 0000002b
00000000 00000000 00000040 2**4
DEBUG (Oops_decode_part): CONTENTS,
ALLOC, LOAD, READONLY, CODE
DEBUG (Oops_decode_part): Disassembly of section
.text:
DEBUG (Oops_decode_part):
DEBUG (Oops_decode_part): 00000000 <_XXX>:
DEBUG (Oops_decode_part): 0: 24 89
and $0x89,%al
DEBUG (Oops_decode_part): 2: 44
inc %esp
DEBUG (Oops_decode_part): 3: 24 04
and $0x4,%al
DEBUG (Oops_decode_part): 5: e8 f4 f2 ff ff
call fffff2fe <_XXX+0xfffff2fe>
DEBUG (Oops_decode_part): a: 39 fe
cmp %edi,%esi
DEBUG (Oops_decode_part): c: 8b 16
mov (%esi),%edx
DEBUG (Oops_decode_part): e: 89 f0
mov %esi,%eax
DEBUG (Oops_decode_part): 10: 75 a2
jne ffffffb4 <_XXX+0xffffffb4>
DEBUG (Oops_decode_part): 12: 83 c4 14
add $0x14,%esp
DEBUG (Oops_decode_part): 15: 5b
pop %ebx
DEBUG (Oops_decode_part): 16: 5e
pop %esi
DEBUG (Oops_decode_part): 17: 5f
pop %edi
DEBUG (Oops_decode_part): 18: 5d
pop %ebp
DEBUG (Oops_decode_part): 19: c3
ret
DEBUG (Oops_decode_part): 1a: 8d 74 26 00
lea 0x0(%esi),%esi
DEBUG (Oops_decode_part): 1e: 55
push %ebp
DEBUG (Oops_decode_part): 1f: 89 e5
mov %esp,%ebp
DEBUG (Oops_decode_part): 21: 53
push %ebx
DEBUG (Oops_decode_part): 22: 83 ec 14
sub $0x14,%esp
DEBUG (Oops_decode_part): 25: 8b 5d 08
mov 0x8(%ebp),%ebx
DEBUG (Oops_decode_part): 28: 8b 4b 04
mov 0x4(%ebx),%ecx
DEBUG (Oops_decode_part):
DEBUG (Oops_objdump): command '/usr/bin/objdump -dhf
/root/tmp/ksymoops.xIweUA'
DEBUG (Oops_decode_part):
DEBUG (Oops_decode_part): /root/tmp/ksymoops.xIweUA:
file format elf32-i386
DEBUG (Oops_decode_part): architecture: i386, flags
0x00000010:
DEBUG (Oops_decode_part): HAS_SYMS
DEBUG (Oops_decode_part): start address 0x00000000
DEBUG (Oops_decode_part):
DEBUG (Oops_decode_part): Sections:
DEBUG (Oops_decode_part): Idx Name Size
VMA LMA File off Algn
DEBUG (Oops_decode_part): 0 .text 00000015
00000000 00000000 00000040 2**4
DEBUG (Oops_decode_part): CONTENTS,
ALLOC, LOAD, READONLY, CODE
DEBUG (Oops_decode_part): Disassembly of section
.text:
DEBUG (Oops_decode_part):
DEBUG (Oops_decode_part): 00000000 <_XXX>:
DEBUG (Oops_decode_part): 0: 83 b9 a8 02 00 00 06
cmpl $0x6,0x2a8(%ecx)
DEBUG (Oops_decode_part): 7: 74 2a
je 33 <_XXX+0x33>
DEBUG (Oops_decode_part): 9: 8b 93 4c 01 00 00
mov 0x14c(%ebx),%edx
DEBUG (Oops_decode_part): f: 89 d0
mov %edx,%eax
DEBUG (Oops_decode_part): 11: c1 f8 10
sar $0x10,%eax
DEBUG (Oops_decode_part): 14: 25
.byte 0x25
DEBUG (Oops_format):

DEBUG (map_address): merged f88642bd

>> >>EIP; f88642bd <pg0+3839f2bd/3fb39400> <=====
DEBUG (map_address): merged c0437f2c

>> >>eax; c0437f2c <setup_IO_APIC+dc/5f0>
DEBUG (map_address): merged f7c3902c
>> >>ebx; f7c3902c <pg0+3777402c/3fb39400>
DEBUG (map_address): merged 00000001
DEBUG (map_address): merged f7db6e18
>> >>edx; f7db6e18 <pg0+378f1e18/3fb39400>
DEBUG (map_address): merged f7c3902c
>> >>esi; f7c3902c <pg0+3777402c/3fb39400>
DEBUG (map_address): merged c0437f2c
>> >>edi; c0437f2c <setup_IO_APIC+dc/5f0>
DEBUG (map_address): merged c0437f1c
>> >>ebp; c0437f1c <setup_IO_APIC+cc/5f0>
DEBUG (map_address): merged c0437f04
>> >>esp; c0437f04 <setup_IO_APIC+b4/5f0>
DEBUG (map_address): merged c010426b

Trace; c010426b <die+7b/150>
DEBUG (map_address): merged c010444f
Trace; c010444f <do_divide_error+1f/c0>
DEBUG (map_address): merged c0104694
Trace; c0104694 <do_invalid_op+84/c0>
DEBUG (map_address): merged c011bf71
Trace; c011bf71 <sys_rsbac_mac_get_p_trulist+c1/f0>
DEBUG (map_address): merged c0103e6f
Trace; c0103e6f <show_stack+3f/f0>
DEBUG (map_address): merged f885fc71
Trace; f885fc71 <pg0+3839ac71/3fb39400>
DEBUG (map_address): merged c0181ab2
Trace; c0181ab2 <sys_setresuid+52/350>
DEBUG (map_address): merged c0181b57
Trace; c0181b57 <sys_setresuid+f7/350>
DEBUG (map_address): merged c0105b11
Trace; c0105b11 <sys_vm86old+c1/d0>
DEBUG (map_address): merged c0103d16
Trace; c0103d16 <page_fault+a/c>
DEBUG (map_address): merged c010112f
Trace; c010112f <cpu_idle_wait+1f/80>
DEBUG (map_address): merged c0438a7a
Trace; c0438a7a <hpet_enable+fa/1a0>
DEBUG (map_address): merged c010020f
Trace; c010020f <ignore_int+23/34>

This architecture has variable length instructions,
decoding before eip
is unreliable, take these instructions with a pinch of
salt.

DEBUG (map_address): merged f8864292
Code; f8864292 <pg0+3839f292/3fb39400>
00000000 <_EIP>:
DEBUG (map_address): merged f8864292
Code; f8864292 <pg0+3839f292/3fb39400>
0: 24 89 and $0x89,%al
DEBUG (map_address): merged f8864294
Code; f8864294 <pg0+3839f294/3fb39400>
2: 44 inc %esp
DEBUG (map_address): merged f8864295
Code; f8864295 <pg0+3839f295/3fb39400>
3: 24 04 and $0x4,%al
DEBUG (map_address): merged f8864297
Code; f8864297 <pg0+3839f297/3fb39400>
5: e8 f4 f2 ff ff call fffff2fe
<_EIP+0xfffff2fe>
DEBUG (map_address): merged f886429c
Code; f886429c <pg0+3839f29c/3fb39400>
a: 39 fe cmp %edi,%esi
DEBUG (map_address): merged f886429e
Code; f886429e <pg0+3839f29e/3fb39400>
c: 8b 16 mov (%esi),%edx
DEBUG (map_address): merged f88642a0
Code; f88642a0 <pg0+3839f2a0/3fb39400>
e: 89 f0 mov %esi,%eax
DEBUG (map_address): merged f88642a2
Code; f88642a2 <pg0+3839f2a2/3fb39400>
10: 75 a2 jne ffffffb4
<_EIP+0xffffffb4>
DEBUG (map_address): merged f88642a4
Code; f88642a4 <pg0+3839f2a4/3fb39400>
12: 83 c4 14 add $0x14,%esp
DEBUG (map_address): merged f88642a7
Code; f88642a7 <pg0+3839f2a7/3fb39400>
15: 5b pop %ebx
DEBUG (map_address): merged f88642a8
Code; f88642a8 <pg0+3839f2a8/3fb39400>
16: 5e pop %esi
DEBUG (map_address): merged f88642a9
Code; f88642a9 <pg0+3839f2a9/3fb39400>
17: 5f pop %edi
DEBUG (map_address): merged f88642aa
Code; f88642aa <pg0+3839f2aa/3fb39400>
18: 5d pop %ebp
DEBUG (map_address): merged f88642ab
Code; f88642ab <pg0+3839f2ab/3fb39400>
19: c3 ret
DEBUG (map_address): merged f88642ac
Code; f88642ac <pg0+3839f2ac/3fb39400>
1a: 8d 74 26 00 lea
0x0(%esi),%esi
DEBUG (map_address): merged f88642b0
Code; f88642b0 <pg0+3839f2b0/3fb39400>
1e: 55 push %ebp
DEBUG (map_address): merged f88642b1
Code; f88642b1 <pg0+3839f2b1/3fb39400>
1f: 89 e5 mov %esp,%ebp
DEBUG (map_address): merged f88642b3
Code; f88642b3 <pg0+3839f2b3/3fb39400>
21: 53 push %ebx
DEBUG (map_address): merged f88642b4
Code; f88642b4 <pg0+3839f2b4/3fb39400>
22: 83 ec 14 sub $0x14,%esp
DEBUG (map_address): merged f88642b7
Code; f88642b7 <pg0+3839f2b7/3fb39400>
25: 8b 5d 08 mov
0x8(%ebp),%ebx
DEBUG (map_address): merged f88642ba
Code; f88642ba <pg0+3839f2ba/3fb39400>
28: 8b 4b 04 mov
0x4(%ebx),%ecx

This decode from eip onwards should be reliable

DEBUG (map_address): merged f88642bd
Code; f88642bd <pg0+3839f2bd/3fb39400>
00000000 <_EIP>:
DEBUG (map_address): merged f88642bd
Code; f88642bd <pg0+3839f2bd/3fb39400> <=====
0: 83 b9 a8 02 00 00 06 cmpl
$0x6,0x2a8(%ecx) <=====
DEBUG (map_address): merged f88642c4
Code; f88642c4 <pg0+3839f2c4/3fb39400>
7: 74 2a je 33
<_EIP+0x33>
DEBUG (map_address): merged f88642c6
Code; f88642c6 <pg0+3839f2c6/3fb39400>
9: 8b 93 4c 01 00 00 mov
0x14c(%ebx),%edx
DEBUG (map_address): merged f88642cc
Code; f88642cc <pg0+3839f2cc/3fb39400>
f: 89 d0 mov %edx,%eax
DEBUG (map_address): merged f88642ce
Code; f88642ce <pg0+3839f2ce/3fb39400>
11: c1 f8 10 sar $0x10,%eax
DEBUG (map_address): merged f88642d1
Code; f88642d1 <pg0+3839f2d1/3fb39400>
14: 25 .byte 0x25

DEBUG (Oops_read): <0>Kernel panic - not syncing:
Fatal exception in interrupt
<0>Kernel panic - not syncing: Fatal exception in
interrupt
DEBUG (Oops_read):

1 error issued. Results may not be reliable.


_Thanks....
Richard

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

---

Attachments:

ksymoops.out3 (21.00 kB)
1297169995-ksymoops.out3

2006-08-03 22:14:57

by Thomas Backlund

[permalink] [raw]

Subject: Re: Kernel Hangs, EIP is at scsi_decide_disposition

---

Richard Mollel wrote:
> Our mandriva boxes are randomly crushing after
> migration to 2006.0 download version. The same
> hardware has had mandriva 10.0 withuptimes of over 230
> days....but nowadays it is hardly a week before a
> crash for over 6 nodes with latest kernels.
>

The correct place to report this is:

http://qa.mandriva.com/


> I captured one of our lockups and am attaching the
> output of ksymoops for further interpretation by mroe
> experienced eyes when it comes to kernel internals.
>
> Here is the actual error, and attached, the ksymoops
> interpretation of it.
>
> Unable to handle kernel NULL pointer dereference at
> virtual address 000002a9
> printing eip:
> f88642bd
> *pde = 3702b001
> Oops: 0000 [#1]
> SMP
> Modules linked in: sg nfsd exportfs raw md5 ipv6 nfs
> lockd nfs_acl sunrpc parport_pc lp parport e1000
> af_packet floppy video thermal tc1100_wmi processor
> fan container button battery ac ipt_LOG ipt_REJECT
> ipt_state ip_conntrack iptable_filter ip_tables ide_cd
> loop hw_random 3w_9xxx tsdev sr_mod ehci_hcd uhci_hcd
> usbcore evdev reiserfs sd_mod ahci ata_piix libata
> scsi_mod
> CPU: 0
> EIP: 0060:[<f88642bd>] Tainted: G M VLI
> EFLAGS: 00010282 (2.6.12-22mdksmp)

Can you reproduce with an untainted kernel ??

This is what you also need to report to the above mentioned address...

--
Thomas

2006-08-04 20:41:04

by Luiz Fernando N. Capitulino

[permalink] [raw]

Subject: Re: Kernel Hangs, EIP is at scsi_decide_disposition

---

Em Fri, 04 Aug 2006 01:15:07 +0300
Thomas Backlund <[email protected]> escreveu:

| Richard Mollel wrote:
| > Our mandriva boxes are randomly crushing after
| > migration to 2006.0 download version. The same
| > hardware has had mandriva 10.0 withuptimes of over 230
| > days....but nowadays it is hardly a week before a
| > crash for over 6 nodes with latest kernels.
| >
|
| The correct place to report this is:
|
| http://qa.mandriva.com/
|
|
| > I captured one of our lockups and am attaching the
| > output of ksymoops for further interpretation by mroe
| > experienced eyes when it comes to kernel internals.
| >
| > Here is the actual error, and attached, the ksymoops
| > interpretation of it.
| >
| > Unable to handle kernel NULL pointer dereference at
| > virtual address 000002a9
| > printing eip:
| > f88642bd
| > *pde = 3702b001
| > Oops: 0000 [#1]
| > SMP
| > Modules linked in: sg nfsd exportfs raw md5 ipv6 nfs
| > lockd nfs_acl sunrpc parport_pc lp parport e1000
| > af_packet floppy video thermal tc1100_wmi processor
| > fan container button battery ac ipt_LOG ipt_REJECT
| > ipt_state ip_conntrack iptable_filter ip_tables ide_cd
| > loop hw_random 3w_9xxx tsdev sr_mod ehci_hcd uhci_hcd
| > usbcore evdev reiserfs sd_mod ahci ata_piix libata
| > scsi_mod
| > CPU: 0
| > EIP: 0060:[<f88642bd>] Tainted: G M VLI
| > EFLAGS: 00010282 (2.6.12-22mdksmp)
|
| Can you reproduce with an untainted kernel ??
|
| This is what you also need to report to the above mentioned address...

Known issue:

http://qa.mandriva.com/show_bug.cgi?id=19593

Richard, it's a very hard to trace problem. Please add a comment
there if you can reproduce it w/o having to run your test-case for a
week.

--
Luiz Fernando N. Capitulino