Hi,
I have experienced a condition I believe to be a bug. Below is a bug report
in the format suggested by REPORTING-BUGS:
Summary:
System slowdown under unexplained excessive disk I/O
Full description:
While running X, KDE, having a few windows open, I ran make -j4 on MySQL
source tree. I do this all the time and it usually works just fine - the
system is a little bit unresponsive. However, occasionally the system becomes
completely unresponsive - the disk goes crazy, the machine pings but neither
ssh or telnet work - connection to the port is established, but nothing
further than that. It does respond to magic SysRQ. I was able to get a memory
info dump + stack traces into syslog, included below. The filesystem is
ReiserFS.
Keywords:
vm, ReiserFS, heavy disk I/O,
Version:
Linux version 2.4.13 (sasha@mysql) (gcc version 2.95.3 20010315 (SuSE)) #3
SMP Tue Oct 30 15:07:27 MST 2001
CPU:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 7
model name : Pentium III (Katmai)
stepping : 3
cpu MHz : 501.149
cache size : 512 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov
pat pse36 mmx fxsr sse
bogomips : 999.42
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 7
model name : Pentium III (Katmai)
stepping : 3
cpu MHz : 501.149
cache size : 512 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov
pat pse36 mmx fxsr sse
bogomips : 999.42
Memory:
total: used: free: shared: buffers: cached:
Mem: 261107712 257200128 3907584 0 37466112 80744448
Swap: 0 0 0
MemTotal: 254988 kB
MemFree: 3816 kB
MemShared: 0 kB
Buffers: 36588 kB
Cached: 78852 kB
SwapCached: 0 kB
Active: 54824 kB
Inactive: 60616 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 254988 kB
LowFree: 3816 kB
SwapTotal: 0 kB
SwapFree: 0 kB
ver_linux output:
Linux mysql 2.4.13 #3 SMP Tue Oct 30 15:07:27 MST 2001 i686 unknown
Gnu C 2.95.3
Gnu make 3.79.1
binutils 2.10.91.0.4
util-linux 2.11b
mount 2.11b
modutils 2.4.2
e2fsprogs 1.18
pcmcia-cs 3.1.17
PPP 2.3.11
isdn4k-utils 3.1pre1a
Linux C Library x 1 root root 1341862 Jun 17 17:24
/lib/libc.so.6
Dynamic linker (ldd) 2.2.2
Procps 2.0.6
Net-tools 1.56
Kbd 0.99
Sh-utils 2.0
Modules Loaded ide-scsi scanner opl3 sb uart401 midi soundbase
sndshield
Debug info from syslog:
Nov 7 17:14:15 mysql kernel: SysRq : Show Memory
Nov 7 17:14:21 mysql kernel: Mem-info:
Nov 7 17:14:21 mysql kernel: Free pages: 4092kB ( 0kB HighMem)
Nov 7 17:14:21 mysql kernel: Zone:DMA freepages: 1080kB min: 512KB low:
1024kB high: 1536kB
Nov 7 17:14:21 mysql kernel: Zone:Normal freepages: 3012kB min: 1020KB
low: 2040kB high: 3060kB
Nov 7 17:14:21 mysql kernel: Zone:HighMem freepages: 0kB min: 0KB
low: 0kB high: 0kB
Nov 7 17:14:21 mysql kernel: Free pages: 4092kB ( 0kB HighMem)
Nov 7 17:14:21 mysql kernel: ( Active: 1476, inactive: 377, free: 1023 )
Nov 7 17:14:21 mysql kernel: 62*4kB 16*8kB 4*16kB 2*32kB 1*64kB 0*128kB
0*256kB 1*512kB 0*1024kB 0*2048kB = 1080kB)
Nov 7 17:14:21 mysql kernel: 297*4kB 92*8kB 4*16kB 0*32kB 0*64kB 0*128kB
0*256kB 0*512kB 1*1024kB 0*2048kB = 3012kB)
Nov 7 17:14:21 mysql kernel: = 0kB)
Nov 7 17:14:21 mysql kernel: Swap cache: add 0, delete 0, find 0/0, race 0+0
Nov 7 17:14:21 mysql kernel: Free swap: 0kB
Nov 7 17:14:22 mysql kernel: 65533 pages of RAM
Nov 7 17:14:23 mysql kernel: 0 pages of HIGHMEM
Nov 7 17:14:23 mysql kernel: 1818 reserved pages
Nov 7 17:14:24 mysql kernel: 5031 pages shared
Nov 7 17:14:24 mysql kernel: 0 pages swap cached
Nov 7 17:14:24 mysql kernel: 21 pages in page table cache
Nov 7 17:14:24 mysql kernel: Buffer memory: 1316kB
Nov 7 17:14:24 mysql kernel: CLEAN: 4 buffers, 13 kbyte, 4 used
(last=4), 0 locked, 0 dirty
Nov 7 17:14:24 mysql kernel: 400 22960 22959 22963 (NOTLB)
Nov 7 17:14:26 mysql kernel: Call Trace: [<c01bf512>] [<c0132caf>]
[<c0133e41>] [<c0197a25>] [<c0193d29>]
Nov 7 17:14:27 mysql kernel: [<c0194972>] [<c0185e65>] [<c0186049>]
[<c0186063>] [<c011c689>] [<c011c977>]
Nov 7 17:14:27 mysql kernel: [<c018687d>] [<c01bee66>] [<c01bee9b>]
[<c01beeb6>] [<c01296d7>] [<c0134798>]
Nov 7 17:14:27 mysql kernel: [<c012c132>] [<c0124390>] [<c0189363>]
[<c0186740>] [<c0124445>] [<c012448e>]
Nov 7 17:14:27 mysql kernel: [<c012556b>] [<c0122012>] [<c01220ed>]
[<c0110604>] [<c011048c>] [<c0106397>]
Nov 7 17:14:27 mysql kernel: [<c010648a>] [<c01070bc>]
Nov 7 17:14:27 mysql kernel: kde S 00000000 0 22963 22959
23114 22960 (NOTLB)
Nov 7 17:14:27 mysql kernel: Call Trace: [<c01175a5>] [<c0106fcb>]
Nov 7 17:14:27 mysql kernel: kdeinit S 7FFFFFFF 4268 23070 1
23073 10009 (NOTLB)
Nov 7 17:14:27 mysql kernel: Call Trace: [<c013f9ac>] [<c0110f67>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:27 mysql kernel: kdeinit S 7FFFFFFF 0 23073 1
23076 23070 (NOTLB)
Nov 7 17:14:27 mysql kernel: Call Trace: [<c013f9ac>] [<c0110f67>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:27 mysql kernel: kdeinit D 00000002 0 23076 1
23082 23073 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01bf512>] [<c0132caf>]
[<c0133e41>] [<c0197a25>] [<c0193d29>]
Nov 7 17:14:28 mysql kernel: [<c0194972>] [<c0185e65>] [<c0186049>]
[<c0186063>] [<c018687d>] [<c0132cc5>]
Nov 7 17:14:28 mysql kernel: [<c0193ca3>] [<c019448f>] [<c01bee66>]
[<c01bf07f>] [<c0134798>] [<c012c132>]
Nov 7 17:14:28 mysql kernel: [<c0124390>] [<c0189363>] [<c0186740>]
[<c0124445>] [<c012448e>] [<c012556b>]
Nov 7 17:14:28 mysql kernel: [<c0122012>] [<c01220ed>] [<c0110604>]
[<c011048c>] [<c01432a9>] [<c013aefd>]
Nov 7 17:14:28 mysql kernel: [<c0138ac8>] [<c01070bc>]
Nov 7 17:14:28 mysql kernel: kdeinit S 7FFFFFFF 0 23082 1
23102 23076 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c013f9ac>] [<c0110f67>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: artsd D 00000002 12 23101 1
23113 23102 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01bf512>] [<c0132caf>]
[<c0133e41>] [<c0197a25>] [<c0193d29>]
Nov 7 17:14:28 mysql kernel: [<c0194972>] [<c0185e65>] [<c0186049>]
[<c0186063>] [<c018687d>] [<c0132cc5>]
Nov 7 17:14:28 mysql kernel: [<c0193ca3>] [<c019448f>] [<c01bee66>]
[<c0134798>] [<c012c132>] [<c0124390>]
Nov 7 17:14:28 mysql kernel: [<c0189363>] [<c0186740>] [<c0124445>]
[<c012448e>] [<c012556b>] [<c0122012>]
Nov 7 17:14:28 mysql kernel: [<c01220ed>] [<c0110604>] [<c011048c>]
[<c01432a9>] [<c013aefd>] [<c0138b38>]
Nov 7 17:14:28 mysql kernel: [<c01070bc>]
Nov 7 17:14:28 mysql kernel: kdeinit S 7FFFFFFF 0 23102 1
8954 23101 23082 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c013f9ac>] [<c0110f67>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: knotify S 7FFFFFFF 2400 23113 1
23122 23101 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c013f9ac>] [<c0110f67>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: ksmserver S 7FFFFFFF 0 23114 22963
(NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c013f9ac>] [<c0110f67>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: kdeinit D 00000002 0 23115 23102
23120 23136 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01bf512>] [<c0132caf>]
[<c0133e41>] [<c0197a25>] [<c0193d29>]
Nov 7 17:14:28 mysql kernel: [<c0194972>] [<c0185e65>] [<c0186049>]
[<c0186063>] [<c01f5ac1>] [<c018687d>]
Nov 7 17:14:28 mysql kernel: [<c01bec1c>] [<c01bec2e>] [<c01bee66>]
[<c01bee9b>] [<c01beeb6>] [<c01296d7>]
Nov 7 17:14:28 mysql kernel: [<c0134798>] [<c012c132>] [<c0124390>]
[<c0189363>] [<c0186740>] [<c0124445>]
Nov 7 17:14:28 mysql kernel: [<c012448e>] [<c012556b>] [<c0122012>]
[<c01220ed>] [<c0110604>] [<c011048c>]
Nov 7 17:14:28 mysql kernel: [<c022fb6e>] [<c0117c99>] [<c01070bc>]
Nov 7 17:14:28 mysql kernel: xemacs S C6637F2C 2400 23116 23115
23118 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0110fca>] [<c0110ef4>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: xemacs S CFE05F2C 0 23118 23115
23119 23116 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0110fca>] [<c0110ef4>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: xemacs S C9695F2C 0 23119 23115
23120 23118 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0110fca>] [<c0110ef4>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: xemacs S CE7B1F2C 0 23120 23115
23119 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0110fca>] [<c0110ef4>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: kdeinit S 7FFFFFFF 2400 23122 1
23125 23124 23113 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c013f9ac>] [<c01ab604>]
[<c01aa03f>] [<c0110f67>] [<c013fc1e>]
Nov 7 17:14:28 mysql kernel: [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: kdeinit S 7FFFFFFF 0 23124 1
23131 23127 23122 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c013f9ac>] [<c0110f67>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: cat S 7FFFFFFF 0 23125 23122
(NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0110f67>] [<c01a996e>]
[<c01a56ca>] [<c0131db7>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: kdeinit D 00000002 24 23127 1
23134 23124 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01bf512>] [<c0132caf>]
[<c0133e41>] [<c0197a25>] [<c0193d29>]
Nov 7 17:14:28 mysql kernel: [<c0194972>] [<c0185e65>] [<c0186049>]
[<c0186063>] [<c01f1ae5>] [<c01f1300>]
Nov 7 17:14:28 mysql kernel: [<c01f1948>] [<c018687d>] [<c01edcdd>]
[<c01ee1f2>] [<c01f1300>] [<c01bee66>]
Nov 7 17:14:28 mysql kernel: [<c0134798>] [<c012c132>] [<c0124390>]
[<c0189363>] [<c0186740>] [<c0124445>]
Nov 7 17:14:28 mysql kernel: [<c012448e>] [<c012556b>] [<c0122012>]
[<c01220ed>] [<c0110604>] [<c011048c>]
Nov 7 17:14:28 mysql kernel: [<c01ee1f2>] [<c011b794>] [<c010ef9d>]
[<c01070bc>]
Nov 7 17:14:28 mysql kernel: wishx S 7FFFFFFF 0 23131 23124
(NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c013f9ac>] [<c0110f67>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: kdeinit S 7FFFFFFF 0 23134 1
23140 23127 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c013f9ac>] [<c0110f67>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: kdeinit D 00000002 0 23136 23102
23210 23138 23115 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01bf512>] [<c0132caf>]
[<c0133e41>] [<c0197a25>] [<c0193d29>]
Nov 7 17:14:28 mysql kernel: [<c0194972>] [<c0185e65>] [<c0186049>]
[<c0186063>] [<c01f1ae5>] [<c01f1300>]
Nov 7 17:14:28 mysql kernel: [<c01f1948>] [<c018687d>] [<c01ee1f2>]
[<c01bec1c>] [<c01bec2e>] [<c01bee66>]
Nov 7 17:14:28 mysql kernel: [<c01bf07f>] [<c01296d7>] [<c0134798>]
[<c012c132>] [<c0124390>] [<c0189363>]
Nov 7 17:14:28 mysql kernel: [<c0186740>] [<c0124445>] [<c012448e>]
[<c012556b>] [<c0122012>] [<c01220ed>]
Nov 7 17:14:28 mysql kernel: [<c0110604>] [<c011048c>] [<c022fb6e>]
[<c0131de4>] [<c01070bc>]
Nov 7 17:14:28 mysql kernel: kdeinit S 7FFFFFFF 2404 23138 23102
23149 8332 23136 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c013f9ac>] [<c01ab604>]
[<c0110f67>] [<c013fc1e>] [<c013ffc0>]
Nov 7 17:14:28 mysql kernel: [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: alarmd S 7FFFFFFF 0 23140 1
23231 23134 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c013f9ac>] [<c0110f67>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: bash S 7FFFFFFF 2404 23141 23138
23145 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0110f67>] [<c01a996e>]
[<c01a56ca>] [<c0131db7>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: bash S 00000000 0 23142 23136
6206 23148 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01a6d57>] [<c01175a5>]
[<c0106fcb>]
Nov 7 17:14:28 mysql kernel: bash S 7FFFFFFF 5104 23145 23138
23149 23141 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0110f67>] [<c01a996e>]
[<c01a56ca>] [<c0131db7>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: bash S 7FFFFFFF 0 23148 23136
23150 23142 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0110f67>] [<c01a996e>]
[<c01a56ca>] [<c0131db7>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: bash S 7FFFFFFF 0 23149 23138
23145 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0110f67>] [<c01a996e>]
[<c01a56ca>] [<c0131db7>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: bash S 00000000 0 23150 23136
23359 23156 23148 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01a6d57>] [<c01175a5>]
[<c0106fcb>]
Nov 7 17:14:28 mysql kernel: bash S 7FFFFFFF 0 23156 23136
31898 23162 23150 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0110f67>] [<c01a996e>]
[<c01a56ca>] [<c0131db7>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: bash S 00000000 0 23162 23136
652 23184 23156 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01a6d57>] [<c01175a5>]
[<c0106fcb>]
Nov 7 17:14:28 mysql kernel: bash S 7FFFFFFF 0 23184 23136
23210 23162 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0110f67>] [<c01a996e>]
[<c01a56ca>] [<c0131db7>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: bash S 7FFFFFFF 2404 23210 23136
23184 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0110f67>] [<c01a996e>]
[<c01a56ca>] [<c0131db7>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: kmail R 00000000 1312 23231 1
23248 23140 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0124390>] [<c01245b5>]
[<c01245f3>] [<c012559c>] [<c0122012>]
Nov 7 17:14:28 mysql kernel: [<c01220ed>] [<c0110604>] [<c011048c>]
[<c022fb6e>] [<c0131de4>] [<c01070bc>]
Nov 7 17:14:28 mysql kernel: kdeinit S 7FFFFFFF 0 23248 1
23259 23231 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c013f9ac>] [<c0110f67>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: kdeinit S 7FFFFFFF 0 23259 1
23277 23248 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c013f9ac>] [<c0110f67>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: kdesud D 00000000 0 23277 1
702 23259 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0124390>] [<c01245b5>]
[<c01245f3>] [<c012559c>] [<c0122012>]
Nov 7 17:14:28 mysql kernel: [<c01220ed>] [<c0110604>] [<c011048c>]
[<c012c312>] [<c013fc7a>] [<c01400ef>]
Nov 7 17:14:28 mysql kernel: [<c01070bc>]
Nov 7 17:14:28 mysql kernel: xemacs S C47B9F2C 0 23559 23162
652 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0110fca>] [<c0110ef4>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: xemacs S C6DF9F2C 0 31898 23156
(NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0110fca>] [<c0110ef4>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: ssh S 7FFFFFFF 0 6206 23142
(NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01ab604>] [<c0110f67>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: mysql-test-ru S 00000000 0 652 23162
739 23559 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c011d7e7>] [<c01175a5>]
[<c0106fcb>]
Nov 7 17:14:28 mysql kernel: mysqlmanager S 7FFFFFFF 0 702 1
704 8409 23277 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c025386a>] [<c0110f67>]
[<c024c4d2>] [<c024c659>] [<c0263d24>]
Nov 7 17:14:28 mysql kernel: [<c0230626>] [<c0106397>] [<c023107c>]
[<c0106fcb>]
Nov 7 17:14:28 mysql kernel: mysqlmanager S CF78A000 0 703 702
726 704 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c013a07d>] [<c013a154>]
[<c0131db7>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: mysqlmanager D 00000000 0 704 702
740 703 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0124390>] [<c01245b5>]
[<c01245f3>] [<c012559c>] [<c0122012>]
Nov 7 17:14:28 mysql kernel: [<c01220ed>] [<c0110604>] [<c011048c>]
[<c012c2f4>] [<c012c312>] [<c013f91a>]
Nov 7 17:14:28 mysql kernel: [<c0140534>] [<c01070bc>]
Nov 7 17:14:28 mysql kernel: mysqlmanager S CD79FFB0 28 705 704
740 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c013a275>] [<c0106143>]
[<c0106fcb>]
Nov 7 17:14:28 mysql kernel: mysqld S C78ABFB0 1624 711 703
712 726 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0106143>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: mysqld S CE209F28 0 712 711
720 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c013f9ac>] [<c0110fca>]
[<c0110ef4>] [<c014023f>] [<c014043d>]
Nov 7 17:14:28 mysql kernel: [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: mysqld S C9FF9FB0 0 713 712
714 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0106143>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: mysqld S C3E69FB0 0 714 712
715 713 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0106143>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: mysqld S C5B8FFB0 0 715 712
716 714 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0106143>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: mysqld S CE94BFB0 0 716 712
717 715 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0106143>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: mysqld S CEB1DFB0 6128 717 712
718 716 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01400ef>] [<c0106143>]
[<c0106fcb>]
Nov 7 17:14:28 mysql kernel: mysqld S CA615F2C 5016 718 712
719 717 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0110fca>] [<c0110ef4>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: mysqld S C723DFB0 5544 719 712
720 718 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0106143>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: mysqld S C6579FB0 4532 720 712
719 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0106143>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: mysqld S 7FFFFFFF 4888 726 703
727 711 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c013f9ac>] [<c0110f67>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: mysqld S C902FF28 4728 727 726
729 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c013f9ac>] [<c0110fca>]
[<c0110ef4>] [<c014023f>] [<c014043d>]
Nov 7 17:14:28 mysql kernel: [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: mysqld S CD601FB0 4532 728 727
729 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0106143>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: mysqld S 7FFFFFFF 4332 729 727
728 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0110f67>] [<c019bcd7>]
[<c0277167>] [<c0277358>] [<c022fa61>]
Nov 7 17:14:28 mysql kernel: [<c022fb6e>] [<c0131db7>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: mysqlmanagerc S 7FFFFFFF 4532 739 652
(NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0232cbc>] [<c0110f67>]
[<c024abe2>] [<c024b22f>] [<c0263f65>]
Nov 7 17:14:28 mysql kernel: [<c022fa61>] [<c022fb6e>] [<c0131db7>]
[<c0106fcb>]
Nov 7 17:14:28 mysql kernel: mysqlmanager S 7FFFFFFF 5056 740 704
705 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0110f67>] [<c0277167>]
[<c0277358>] [<c022fa61>] [<c022fb6e>]
Nov 7 17:14:28 mysql kernel: [<c0131db7>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: compile-penti S 00000000 0 23359 23150
1149 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01175a5>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: gmake S 00000000 4 1149 23359
1789 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01175a5>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: gmake S 00000000 0 1789 1149
1790 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01175a5>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: sh S 00000000 324 1790 1789
10738 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c011d7e7>] [<c01175a5>]
[<c0106fcb>]
Nov 7 17:14:28 mysql kernel: kdeinit S C68C9F2C 4 8332 23102
8944 23138 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0110fca>] [<c0110ef4>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: kdeinit S 7FFFFFFF 4 8409 1
8644 702 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c013f9ac>] [<c0110f67>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: bash S 7FFFFFFF 4 8644 8409
(NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0110f67>] [<c01a996e>]
[<c01a56ca>] [<c0131db7>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: kdeinit S C8F01F2C 0 8944 23102
8952 8332 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0110fca>] [<c0110ef4>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: kdeinit S CB2EBF2C 0 8952 23102
8954 8944 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0110fca>] [<c0110ef4>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: kdeinit S CF7BFF2C 0 8954 23102
8952 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0110fca>] [<c0110ef4>]
[<c013fc1e>] [<c013ffc0>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: sh S 00000000 0 10738 1790
10739 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c011d7e7>] [<c01175a5>]
[<c0106fcb>]
Nov 7 17:14:28 mysql kernel: gmake S 00000000 2400 10739 10738
10742 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01175a5>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: sh S 00000000 0 10742 10739
10748 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c011d7e7>] [<c01175a5>]
[<c0106fcb>]
Nov 7 17:14:28 mysql kernel: gmake S C55F6000 0 10748 10742
11201 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c013a07d>] [<c013a154>]
[<c0131db7>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: gcc S 00000000 0 11163 10748
11175 11190 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01175a5>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: cc1plus D 00000000 2064 11175 11163
(NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0124390>] [<c01245b5>]
[<c01245f3>] [<c012559c>] [<c0122012>]
Nov 7 17:14:28 mysql kernel: [<c01220ed>] [<c0110604>] [<c011048c>]
[<c011b677>] [<c011b886>] [<c011147d>]
Nov 7 17:14:28 mysql kernel: [<c011048c>] [<c01070bc>]
Nov 7 17:14:28 mysql kernel: sh S 00000000 8 11190 10748
11191 11192 11163 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01175a5>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: gcc S 00000000 0 11191 11190
11206 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01175a5>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: sh S 00000000 0 11192 10748
11193 11201 11190 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01175a5>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: gcc S 00000000 4 11193 11192
11204 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01175a5>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: sh S 00000000 2400 11201 10748
11202 11192 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01175a5>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: gcc S 00000000 2064 11202 11201
11205 (NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01175a5>] [<c0106fcb>]
Nov 7 17:14:28 mysql kernel: cc1plus D 00000000 2400 11204 11193
(NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01245b5>] [<c01245f3>]
[<c012559c>] [<c0122012>] [<c01220ed>]
Nov 7 17:14:28 mysql kernel: [<c0110604>] [<c011048c>] [<c01ee1f2>]
[<c011b794>] [<c010ef9d>] [<c01070bc>]
Nov 7 17:14:28 mysql kernel: cc1plus D 00000000 0 11205 11202
(NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c0124390>] [<c01245b5>]
[<c01245f3>] [<c012559c>] [<c0122012>]
Nov 7 17:14:28 mysql kernel: [<c01220ed>] [<c0110604>] [<c011048c>]
[<c01bbf48>] [<c011147d>] [<c011048c>]
Nov 7 17:14:28 mysql kernel: [<c0108866>] [<c01070bc>]
Nov 7 17:14:28 mysql kernel: cc1plus D 00000000 0 11206 11191
(NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01245b5>] [<c01245f3>]
[<c012559c>] [<c0122012>] [<c01220ed>]
Nov 7 17:14:28 mysql kernel: [<c0110604>] [<c011048c>] [<c01177f0>]
[<c011bab5>] [<c01233f4>] [<c011b794>]
Nov 7 17:14:28 mysql kernel: [<c010ef9d>] [<c01070bc>]
Nov 7 17:14:28 mysql kernel: sendmail D 00000002 0 11207 307
(NOTLB)
Nov 7 17:14:28 mysql kernel: Call Trace: [<c01bf512>] [<c0132caf>]
[<c0133e41>] [<c0197a25>] [<c0193d29>]
Nov 7 17:14:28 mysql kernel: [<c0194972>] [<c0185e65>] [<c0186049>]
[<c0186063>] [<c018687d>] [<c01bee66>]
Nov 7 17:14:28 mysql kernel: [<c01296d7>] [<c0134798>] [<c012c132>]
[<c0124390>] [<c0189363>] [<c0186740>]
Nov 7 17:14:28 mysql kernel: [<c0124445>] [<c012448e>] [<c012556b>]
[<c0122012>] [<c01220ed>] [<c0110604>]
Nov 7 17:14:28 mysql kernel: [<c011048c>] [<c011b677>] [<c011b886>]
[<c011b38e>] [<c01187e0>] [<c01186b9>]
Nov 7 17:14:28 mysql kernel: [<c011843f>] [<c010887b>] [<c01070bc>]
Resolved with ksymoops:
>>EIP; 00002960 Before first symbol <=====
Trace; c01bf512 <submit_bh+56/70>
Trace; c0132cae <__wait_on_buffer+6e/94>
Trace; c0133e40 <bread+44/64>
Trace; c0197a24 <reiserfs_bread+14/1c>
Trace; c0193d28 <search_by_key+70/c10>
Trace; c0194972 <search_for_position_by_key+aa/3a4>
Trace; c0185e64 <make_cpu_key+38/40>
Trace; c0186048 <_get_block_create_0+9c/5a0>
Trace; c0186062 <_get_block_create_0+b6/5a0>
Trace; c011c688 <send_sig_info+58/98>
Trace; c011c976 <send_sig+1a/20>
Trace; c018687c <reiserfs_get_block+13c/d88>
Trace; c01bee66 <__make_request+156/66c>
Trace; c01bee9a <__make_request+18a/66c>
Trace; c01beeb6 <__make_request+1a6/66c>
Trace; c01296d6 <kmem_cache_alloc+8a/12c>
Trace; c0134798 <block_read_full_page+e4/21c>
Trace; c012c132 <__alloc_pages+46/1ac>
Trace; c0124390 <add_to_page_cache_unique+74/84>
Trace; c0189362 <reiserfs_readpage+e/14>
Trace; c0186740 <reiserfs_get_block+0/d88>
Trace; c0124444 <page_cache_read+a4/c8>
Trace; c012448e <read_cluster_nonblocking+26/40>
Trace; c012556a <filemap_nopage+156/344>
Trace; c0122012 <do_no_page+5a/d4>
Trace; c01220ec <handle_mm_fault+60/c0>
Trace; c0110604 <do_page_fault+178/4b4>
Trace; c011048c <do_page_fault+0/4b4>
Trace; c0106396 <restore_sigcontext+116/13c>
Trace; c010648a <sys_sigreturn+ce/f8>
Trace; c01070bc <error_code+34/3c>
Proc; kde
>>EIP; 00000000 Before first symbol
Trace; c01175a4 <sys_wait4+398/3d0>
Trace; c0106fca <system_call+32/38>
Proc; kdeinit
>>EIP; 7ffffffe Before first symbol <=====
Trace; c013f9ac <__pollwait+88/90>
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; kdeinit
>>EIP; 7ffffffe Before first symbol <=====
Trace; c013f9ac <__pollwait+88/90>
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; kdeinit
>>EIP; 00000002 Before first symbol <=====
Trace; c01bf512 <submit_bh+56/70>
Trace; c0132cae <__wait_on_buffer+6e/94>
Trace; c0133e40 <bread+44/64>
Trace; c0197a24 <reiserfs_bread+14/1c>
Trace; c0193d28 <search_by_key+70/c10>
Trace; c0194972 <search_for_position_by_key+aa/3a4>
Trace; c0185e64 <make_cpu_key+38/40>
Trace; c0186048 <_get_block_create_0+9c/5a0>
Trace; c0186062 <_get_block_create_0+b6/5a0>
Trace; c018687c <reiserfs_get_block+13c/d88>
Trace; c0132cc4 <__wait_on_buffer+84/94>
Trace; c0193ca2 <is_tree_node+36/4c>
Trace; c019448e <search_by_key+7d6/c10>
Trace; c01bee66 <__make_request+156/66c>
Trace; c01bf07e <__make_request+36e/66c>
Trace; c0134798 <block_read_full_page+e4/21c>
Trace; c012c132 <__alloc_pages+46/1ac>
Trace; c0124390 <add_to_page_cache_unique+74/84>
Trace; c0189362 <reiserfs_readpage+e/14>
Trace; c0186740 <reiserfs_get_block+0/d88>
Trace; c0124444 <page_cache_read+a4/c8>
Trace; c012448e <read_cluster_nonblocking+26/40>
Trace; c012556a <filemap_nopage+156/344>
Trace; c0122012 <do_no_page+5a/d4>
Trace; c01220ec <handle_mm_fault+60/c0>
Trace; c0110604 <do_page_fault+178/4b4>
Trace; c011048c <do_page_fault+0/4b4>
Trace; c01432a8 <dput+18/154>
Trace; c013aefc <path_release+c/30>
Trace; c0138ac8 <sys_stat64+64/70>
Trace; c01070bc <error_code+34/3c>
Proc; kdeinit
>>EIP; 7ffffffe Before first symbol <=====
Trace; c013f9ac <__pollwait+88/90>
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; artsd
>>EIP; 00000002 Before first symbol <=====
Trace; c01bf512 <submit_bh+56/70>
Trace; c0132cae <__wait_on_buffer+6e/94>
Trace; c0133e40 <bread+44/64>
Trace; c0197a24 <reiserfs_bread+14/1c>
Trace; c0193d28 <search_by_key+70/c10>
Trace; c0194972 <search_for_position_by_key+aa/3a4>
Trace; c0185e64 <make_cpu_key+38/40>
Trace; c0186048 <_get_block_create_0+9c/5a0>
Trace; c0186062 <_get_block_create_0+b6/5a0>
Trace; c018687c <reiserfs_get_block+13c/d88>
Trace; c0132cc4 <__wait_on_buffer+84/94>
Trace; c0193ca2 <is_tree_node+36/4c>
Trace; c019448e <search_by_key+7d6/c10>
Trace; c01bee66 <__make_request+156/66c>
Trace; c0134798 <block_read_full_page+e4/21c>
Trace; c012c132 <__alloc_pages+46/1ac>
Trace; c0124390 <add_to_page_cache_unique+74/84>
Trace; c0189362 <reiserfs_readpage+e/14>
Trace; c0186740 <reiserfs_get_block+0/d88>
Trace; c0124444 <page_cache_read+a4/c8>
Trace; c012448e <read_cluster_nonblocking+26/40>
Trace; c012556a <filemap_nopage+156/344>
Trace; c0122012 <do_no_page+5a/d4>
Trace; c01220ec <handle_mm_fault+60/c0>
Trace; c0110604 <do_page_fault+178/4b4>
Trace; c011048c <do_page_fault+0/4b4>
Trace; c01432a8 <dput+18/154>
Trace; c013aefc <path_release+c/30>
Trace; c0138b38 <sys_lstat64+64/70>
Trace; c01070bc <error_code+34/3c>
Proc; kdeinit
>>EIP; 7ffffffe Before first symbol <=====
Trace; c013f9ac <__pollwait+88/90>
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; knotify
>>EIP; 7ffffffe Before first symbol <=====
Trace; c013f9ac <__pollwait+88/90>
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; ksmserver
>>EIP; 7ffffffe Before first symbol <=====
Trace; c013f9ac <__pollwait+88/90>
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; kdeinit
>>EIP; 00000002 Before first symbol <=====
Trace; c01bf512 <submit_bh+56/70>
Trace; c0132cae <__wait_on_buffer+6e/94>
Trace; c0133e40 <bread+44/64>
Trace; c0197a24 <reiserfs_bread+14/1c>
Trace; c0193d28 <search_by_key+70/c10>
Trace; c0194972 <search_for_position_by_key+aa/3a4>
Trace; c0185e64 <make_cpu_key+38/40>
Trace; c0186048 <_get_block_create_0+9c/5a0>
Trace; c0186062 <_get_block_create_0+b6/5a0>
Trace; c01f5ac0 <do_rw_disk+124/328>
Trace; c018687c <reiserfs_get_block+13c/d88>
Trace; c01bec1c <attempt_merge+60/154>
Trace; c01bec2e <attempt_merge+72/154>
Trace; c01bee66 <__make_request+156/66c>
Trace; c01bee9a <__make_request+18a/66c>
Trace; c01beeb6 <__make_request+1a6/66c>
Trace; c01296d6 <kmem_cache_alloc+8a/12c>
Trace; c0134798 <block_read_full_page+e4/21c>
Trace; c012c132 <__alloc_pages+46/1ac>
Trace; c0124390 <add_to_page_cache_unique+74/84>
Trace; c0189362 <reiserfs_readpage+e/14>
Trace; c0186740 <reiserfs_get_block+0/d88>
Trace; c0124444 <page_cache_read+a4/c8>
Trace; c012448e <read_cluster_nonblocking+26/40>
Trace; c012556a <filemap_nopage+156/344>
Trace; c0122012 <do_no_page+5a/d4>
Trace; c01220ec <handle_mm_fault+60/c0>
Trace; c0110604 <do_page_fault+178/4b4>
Trace; c011048c <do_page_fault+0/4b4>
Trace; c022fb6e <sock_read+92/a0>
Trace; c0117c98 <sys_gettimeofday+20/134>
Trace; c01070bc <error_code+34/3c>
Proc; xemacs
>>EIP; c6637f2c <_end+626d4b0/105bd584> <=====
Trace; c0110fca <schedule_timeout+7a/9c>
Trace; c0110ef4 <process_timeout+0/5c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; xemacs
>>EIP; cfe05f2c <_end+fa3b4b0/105bd584> <=====
Trace; c0110fca <schedule_timeout+7a/9c>
Trace; c0110ef4 <process_timeout+0/5c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; xemacs
>>EIP; c9695f2c <_end+92cb4b0/105bd584> <=====
Trace; c0110fca <schedule_timeout+7a/9c>
Trace; c0110ef4 <process_timeout+0/5c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; xemacs
>>EIP; ce7b1f2c <_end+e3e74b0/105bd584> <=====
Trace; c0110fca <schedule_timeout+7a/9c>
Trace; c0110ef4 <process_timeout+0/5c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; kdeinit
>>EIP; 7ffffffe Before first symbol <=====
Trace; c013f9ac <__pollwait+88/90>
Trace; c01ab604 <pty_chars_in_buffer+20/38>
Trace; c01aa03e <normal_poll+102/124>
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; kdeinit
>>EIP; 7ffffffe Before first symbol <=====
Trace; c013f9ac <__pollwait+88/90>
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; cat
>>EIP; 7ffffffe Before first symbol <=====
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c01a996e <read_chan+3ae/754>
Trace; c01a56ca <tty_read+d6/128>
Trace; c0131db6 <sys_read+8e/c4>
Trace; c0106fca <system_call+32/38>
Proc; kdeinit
>>EIP; 00000002 Before first symbol <=====
Trace; c01bf512 <submit_bh+56/70>
Trace; c0132cae <__wait_on_buffer+6e/94>
Trace; c0133e40 <bread+44/64>
Trace; c0197a24 <reiserfs_bread+14/1c>
Trace; c0193d28 <search_by_key+70/c10>
Trace; c0194972 <search_for_position_by_key+aa/3a4>
Trace; c0185e64 <make_cpu_key+38/40>
Trace; c0186048 <_get_block_create_0+9c/5a0>
Trace; c0186062 <_get_block_create_0+b6/5a0>
Trace; c01f1ae4 <ide_dmaproc+13c/210>
Trace; c01f1300 <ide_dma_intr+0/a8>
Trace; c01f1948 <dma_timer_expiry+0/60>
Trace; c018687c <reiserfs_get_block+13c/d88>
Trace; c01edcdc <ide_do_request+28c/2e4>
Trace; c01ee1f2 <ide_intr+166/18c>
Trace; c01f1300 <ide_dma_intr+0/a8>
Trace; c01bee66 <__make_request+156/66c>
Trace; c0134798 <block_read_full_page+e4/21c>
Trace; c012c132 <__alloc_pages+46/1ac>
Trace; c0124390 <add_to_page_cache_unique+74/84>
Trace; c0189362 <reiserfs_readpage+e/14>
Trace; c0186740 <reiserfs_get_block+0/d88>
Trace; c0124444 <page_cache_read+a4/c8>
Trace; c012448e <read_cluster_nonblocking+26/40>
Trace; c012556a <filemap_nopage+156/344>
Trace; c0122012 <do_no_page+5a/d4>
Trace; c01220ec <handle_mm_fault+60/c0>
Trace; c0110604 <do_page_fault+178/4b4>
Trace; c011048c <do_page_fault+0/4b4>
Trace; c01ee1f2 <ide_intr+166/18c>
Trace; c011b794 <update_process_times+20/94>
Trace; c010ef9c <smp_apic_timer_interrupt+ec/110>
Trace; c01070bc <error_code+34/3c>
Proc; wishx
>>EIP; 7ffffffe Before first symbol <=====
Trace; c013f9ac <__pollwait+88/90>
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; kdeinit
>>EIP; 7ffffffe Before first symbol <=====
Trace; c013f9ac <__pollwait+88/90>
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; kdeinit
>>EIP; 00000002 Before first symbol <=====
Trace; c01bf512 <submit_bh+56/70>
Trace; c0132cae <__wait_on_buffer+6e/94>
Trace; c0133e40 <bread+44/64>
Trace; c0197a24 <reiserfs_bread+14/1c>
Trace; c0193d28 <search_by_key+70/c10>
Trace; c0194972 <search_for_position_by_key+aa/3a4>
Trace; c0185e64 <make_cpu_key+38/40>
Trace; c0186048 <_get_block_create_0+9c/5a0>
Trace; c0186062 <_get_block_create_0+b6/5a0>
Trace; c01f1ae4 <ide_dmaproc+13c/210>
Trace; c01f1300 <ide_dma_intr+0/a8>
Trace; c01f1948 <dma_timer_expiry+0/60>
Trace; c018687c <reiserfs_get_block+13c/d88>
Trace; c01ee1f2 <ide_intr+166/18c>
Trace; c01bec1c <attempt_merge+60/154>
Trace; c01bec2e <attempt_merge+72/154>
Trace; c01bee66 <__make_request+156/66c>
Trace; c01bf07e <__make_request+36e/66c>
Trace; c01296d6 <kmem_cache_alloc+8a/12c>
Trace; c0134798 <block_read_full_page+e4/21c>
Trace; c012c132 <__alloc_pages+46/1ac>
Trace; c0124390 <add_to_page_cache_unique+74/84>
Trace; c0189362 <reiserfs_readpage+e/14>
Trace; c0186740 <reiserfs_get_block+0/d88>
Trace; c0124444 <page_cache_read+a4/c8>
Trace; c012448e <read_cluster_nonblocking+26/40>
Trace; c012556a <filemap_nopage+156/344>
Trace; c0122012 <do_no_page+5a/d4>
Trace; c01220ec <handle_mm_fault+60/c0>
Trace; c0110604 <do_page_fault+178/4b4>
Trace; c011048c <do_page_fault+0/4b4>
Trace; c022fb6e <sock_read+92/a0>
Trace; c0131de4 <sys_read+bc/c4>
Trace; c01070bc <error_code+34/3c>
Proc; kdeinit
>>EIP; 7ffffffe Before first symbol <=====
Trace; c013f9ac <__pollwait+88/90>
Trace; c01ab604 <pty_chars_in_buffer+20/38>
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; alarmd
>>EIP; 7ffffffe Before first symbol <=====
Trace; c013f9ac <__pollwait+88/90>
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; bash
>>EIP; 7ffffffe Before first symbol <=====
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c01a996e <read_chan+3ae/754>
Trace; c01a56ca <tty_read+d6/128>
Trace; c0131db6 <sys_read+8e/c4>
Trace; c0106fca <system_call+32/38>
Proc; bash
>>EIP; 00000000 Before first symbol
Trace; c01a6d56 <tiocspgrp+6e/94>
Trace; c01175a4 <sys_wait4+398/3d0>
Trace; c0106fca <system_call+32/38>
Proc; bash
>>EIP; 7ffffffe Before first symbol <=====
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c01a996e <read_chan+3ae/754>
Trace; c01a56ca <tty_read+d6/128>
Trace; c0131db6 <sys_read+8e/c4>
Trace; c0106fca <system_call+32/38>
Proc; bash
>>EIP; 7ffffffe Before first symbol <=====
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c01a996e <read_chan+3ae/754>
Trace; c01a56ca <tty_read+d6/128>
Trace; c0131db6 <sys_read+8e/c4>
Trace; c0106fca <system_call+32/38>
Proc; bash
>>EIP; 7ffffffe Before first symbol <=====
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c01a996e <read_chan+3ae/754>
Trace; c01a56ca <tty_read+d6/128>
Trace; c0131db6 <sys_read+8e/c4>
Trace; c0106fca <system_call+32/38>
Proc; bash
>>EIP; 00000000 Before first symbol
Trace; c01a6d56 <tiocspgrp+6e/94>
Trace; c01175a4 <sys_wait4+398/3d0>
Trace; c0106fca <system_call+32/38>
Proc; bash
>>EIP; 7ffffffe Before first symbol <=====
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c01a996e <read_chan+3ae/754>
Trace; c01a56ca <tty_read+d6/128>
Trace; c0131db6 <sys_read+8e/c4>
Trace; c0106fca <system_call+32/38>
Proc; bash
>>EIP; 00000000 Before first symbol
Trace; c01a6d56 <tiocspgrp+6e/94>
Trace; c01175a4 <sys_wait4+398/3d0>
Trace; c0106fca <system_call+32/38>
Proc; bash
>>EIP; 7ffffffe Before first symbol <=====
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c01a996e <read_chan+3ae/754>
Trace; c01a56ca <tty_read+d6/128>
Trace; c0131db6 <sys_read+8e/c4>
Trace; c0106fca <system_call+32/38>
Proc; bash
>>EIP; 7ffffffe Before first symbol <=====
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c01a996e <read_chan+3ae/754>
Trace; c01a56ca <tty_read+d6/128>
Trace; c0131db6 <sys_read+8e/c4>
Trace; c0106fca <system_call+32/38>
Proc; kmail
>>EIP; 00000000 Before first symbol
Trace; c0124390 <add_to_page_cache_unique+74/84>
Trace; c01245b4 <__lock_page+78/a0>
Trace; c01245f2 <lock_page+16/1c>
Trace; c012559c <filemap_nopage+188/344>
Trace; c0122012 <do_no_page+5a/d4>
Trace; c01220ec <handle_mm_fault+60/c0>
Trace; c0110604 <do_page_fault+178/4b4>
Trace; c011048c <do_page_fault+0/4b4>
Trace; c022fb6e <sock_read+92/a0>
Trace; c0131de4 <sys_read+bc/c4>
Trace; c01070bc <error_code+34/3c>
Proc; kdeinit
>>EIP; 7ffffffe Before first symbol <=====
Trace; c013f9ac <__pollwait+88/90>
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; kdeinit
>>EIP; 7ffffffe Before first symbol <=====
Trace; c013f9ac <__pollwait+88/90>
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; kdesud
>>EIP; 00000000 Before first symbol
Trace; c0124390 <add_to_page_cache_unique+74/84>
Trace; c01245b4 <__lock_page+78/a0>
Trace; c01245f2 <lock_page+16/1c>
Trace; c012559c <filemap_nopage+188/344>
Trace; c0122012 <do_no_page+5a/d4>
Trace; c01220ec <handle_mm_fault+60/c0>
Trace; c0110604 <do_page_fault+178/4b4>
Trace; c011048c <do_page_fault+0/4b4>
Trace; c012c312 <free_pages+1a/1c>
Trace; c013fc7a <select_bits_free+a/10>
Trace; c01400ee <sys_select+46e/47c>
Trace; c01070bc <error_code+34/3c>
Proc; xemacs
>>EIP; c47b9f2c <_end+43ef4b0/105bd584> <=====
Trace; c0110fca <schedule_timeout+7a/9c>
Trace; c0110ef4 <process_timeout+0/5c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; xemacs
>>EIP; c6df9f2c <_end+6a2f4b0/105bd584> <=====
Trace; c0110fca <schedule_timeout+7a/9c>
Trace; c0110ef4 <process_timeout+0/5c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; ssh
>>EIP; 7ffffffe Before first symbol <=====
Trace; c01ab604 <pty_chars_in_buffer+20/38>
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; mysql-test-ru
>>EIP; 00000000 Before first symbol
Trace; c011d7e6 <sys_rt_sigaction+9e/144>
Trace; c01175a4 <sys_wait4+398/3d0>
Trace; c0106fca <system_call+32/38>
Proc; mysqlmanager
>>EIP; 7ffffffe Before first symbol <=====
Trace; c025386a <tcp_transmit_skb+3c6/51c>
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c024c4d2 <wait_for_connect+112/1d8>
Trace; c024c658 <tcp_accept+c0/23c>
Trace; c0263d24 <inet_accept+2c/1b8>
Trace; c0230626 <sys_accept+66/fc>
Trace; c0106396 <restore_sigcontext+116/13c>
Trace; c023107c <sys_socketcall+b4/200>
Trace; c0106fca <system_call+32/38>
Proc; mysqlmanager
>>EIP; cf78a000 <_end+f3bf584/105bd584> <=====
Trace; c013a07c <pipe_wait+7c/a4>
Trace; c013a154 <pipe_read+b0/1f8>
Trace; c0131db6 <sys_read+8e/c4>
Trace; c0106fca <system_call+32/38>
Proc; mysqlmanager
>>EIP; 00000000 Before first symbol
Trace; c0124390 <add_to_page_cache_unique+74/84>
Trace; c01245b4 <__lock_page+78/a0>
Trace; c01245f2 <lock_page+16/1c>
Trace; c012559c <filemap_nopage+188/344>
Trace; c0122012 <do_no_page+5a/d4>
Trace; c01220ec <handle_mm_fault+60/c0>
Trace; c0110604 <do_page_fault+178/4b4>
Trace; c011048c <do_page_fault+0/4b4>
Trace; c012c2f4 <__free_pages+1c/20>
Trace; c012c312 <free_pages+1a/1c>
Trace; c013f91a <poll_freewait+3a/44>
Trace; c0140534 <sys_poll+2d4/2f0>
Trace; c01070bc <error_code+34/3c>
Proc; mysqlmanager
>>EIP; cd79ffb0 <_end+d3d5534/105bd584> <=====
Trace; c013a274 <pipe_read+1d0/1f8>
Trace; c0106142 <sys_rt_sigsuspend+fa/118>
Trace; c0106fca <system_call+32/38>
Proc; mysqld
>>EIP; c78abfb0 <_end+74e1534/105bd584> <=====
Trace; c0106142 <sys_rt_sigsuspend+fa/118>
Trace; c0106fca <system_call+32/38>
Proc; mysqld
>>EIP; ce209f28 <_end+de3f4ac/105bd584> <=====
Trace; c013f9ac <__pollwait+88/90>
Trace; c0110fca <schedule_timeout+7a/9c>
Trace; c0110ef4 <process_timeout+0/5c>
Trace; c014023e <do_poll+ba/dc>
Trace; c014043c <sys_poll+1dc/2f0>
Trace; c0106fca <system_call+32/38>
Proc; mysqld
>>EIP; c9ff9fb0 <_end+9c2f534/105bd584> <=====
Trace; c0106142 <sys_rt_sigsuspend+fa/118>
Trace; c0106fca <system_call+32/38>
Proc; mysqld
>>EIP; c3e69fb0 <_end+3a9f534/105bd584> <=====
Trace; c0106142 <sys_rt_sigsuspend+fa/118>
Trace; c0106fca <system_call+32/38>
Proc; mysqld
>>EIP; c5b8ffb0 <_end+57c5534/105bd584> <=====
Trace; c0106142 <sys_rt_sigsuspend+fa/118>
Trace; c0106fca <system_call+32/38>
Proc; mysqld
>>EIP; ce94bfb0 <_end+e581534/105bd584> <=====
Trace; c0106142 <sys_rt_sigsuspend+fa/118>
Trace; c0106fca <system_call+32/38>
Proc; mysqld
>>EIP; ceb1dfb0 <_end+e753534/105bd584> <=====
Trace; c01400ee <sys_select+46e/47c>
Trace; c0106142 <sys_rt_sigsuspend+fa/118>
Trace; c0106fca <system_call+32/38>
Proc; mysqld
>>EIP; ca615f2c <_end+a24b4b0/105bd584> <=====
Trace; c0110fca <schedule_timeout+7a/9c>
Trace; c0110ef4 <process_timeout+0/5c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; mysqld
>>EIP; c723dfb0 <_end+6e73534/105bd584> <=====
Trace; c0106142 <sys_rt_sigsuspend+fa/118>
Trace; c0106fca <system_call+32/38>
Proc; mysqld
>>EIP; c6579fb0 <_end+61af534/105bd584> <=====
Trace; c0106142 <sys_rt_sigsuspend+fa/118>
Trace; c0106fca <system_call+32/38>
Proc; mysqld
>>EIP; 7ffffffe Before first symbol <=====
Trace; c013f9ac <__pollwait+88/90>
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; mysqld
>>EIP; c902ff28 <_end+8c654ac/105bd584> <=====
Trace; c013f9ac <__pollwait+88/90>
Trace; c0110fca <schedule_timeout+7a/9c>
Trace; c0110ef4 <process_timeout+0/5c>
Trace; c014023e <do_poll+ba/dc>
Trace; c014043c <sys_poll+1dc/2f0>
Trace; c0106fca <system_call+32/38>
Proc; mysqld
>>EIP; cd601fb0 <_end+d237534/105bd584> <=====
Trace; c0106142 <sys_rt_sigsuspend+fa/118>
Trace; c0106fca <system_call+32/38>
Proc; mysqld
>>EIP; 7ffffffe Before first symbol <=====
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c019bcd6 <do_journal_end+b6/a7c>
Trace; c0277166 <unix_stream_data_wait+c6/114>
Trace; c0277358 <unix_stream_recvmsg+1a4/3ac>
Trace; c022fa60 <sock_recvmsg+3c/ac>
Trace; c022fb6e <sock_read+92/a0>
Trace; c0131db6 <sys_read+8e/c4>
Trace; c0106fca <system_call+32/38>
Proc; mysqlmanagerc
>>EIP; 7ffffffe Before first symbol <=====
Trace; c0232cbc <__kfree_skb+144/14c>
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c024abe2 <tcp_data_wait+106/190>
Trace; c024b22e <tcp_recvmsg+502/91c>
Trace; c0263f64 <inet_recvmsg+3c/54>
Trace; c022fa60 <sock_recvmsg+3c/ac>
Trace; c022fb6e <sock_read+92/a0>
Trace; c0131db6 <sys_read+8e/c4>
Trace; c0106fca <system_call+32/38>
Proc; mysqlmanager
>>EIP; 7ffffffe Before first symbol <=====
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c0277166 <unix_stream_data_wait+c6/114>
Trace; c0277358 <unix_stream_recvmsg+1a4/3ac>
Trace; c022fa60 <sock_recvmsg+3c/ac>
Trace; c022fb6e <sock_read+92/a0>
Trace; c0131db6 <sys_read+8e/c4>
Trace; c0106fca <system_call+32/38>
Proc; compile-penti
>>EIP; 00000000 Before first symbol
Trace; c01175a4 <sys_wait4+398/3d0>
Trace; c0106fca <system_call+32/38>
Proc; gmake
>>EIP; 00000000 Before first symbol
Trace; c01175a4 <sys_wait4+398/3d0>
Trace; c0106fca <system_call+32/38>
Proc; gmake
>>EIP; 00000000 Before first symbol
Trace; c01175a4 <sys_wait4+398/3d0>
Trace; c0106fca <system_call+32/38>
Proc; sh
>>EIP; 00000000 Before first symbol
Trace; c011d7e6 <sys_rt_sigaction+9e/144>
Trace; c01175a4 <sys_wait4+398/3d0>
Trace; c0106fca <system_call+32/38>
Proc; kdeinit
>>EIP; c68c9f2c <_end+64ff4b0/105bd584> <=====
Trace; c0110fca <schedule_timeout+7a/9c>
Trace; c0110ef4 <process_timeout+0/5c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; kdeinit
>>EIP; 7ffffffe Before first symbol <=====
Trace; c013f9ac <__pollwait+88/90>
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; bash
>>EIP; 7ffffffe Before first symbol <=====
Trace; c0110f66 <schedule_timeout+16/9c>
Trace; c01a996e <read_chan+3ae/754>
Trace; c01a56ca <tty_read+d6/128>
Trace; c0131db6 <sys_read+8e/c4>
Trace; c0106fca <system_call+32/38>
Proc; kdeinit
>>EIP; c8f01f2c <_end+8b374b0/105bd584> <=====
Trace; c0110fca <schedule_timeout+7a/9c>
Trace; c0110ef4 <process_timeout+0/5c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; kdeinit
>>EIP; cb2ebf2c <_end+af214b0/105bd584> <=====
Trace; c0110fca <schedule_timeout+7a/9c>
Trace; c0110ef4 <process_timeout+0/5c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; kdeinit
>>EIP; cf7bff2c <_end+f3f54b0/105bd584> <=====
Trace; c0110fca <schedule_timeout+7a/9c>
Trace; c0110ef4 <process_timeout+0/5c>
Trace; c013fc1e <do_select+1ca/204>
Trace; c013ffc0 <sys_select+340/47c>
Trace; c0106fca <system_call+32/38>
Proc; sh
>>EIP; 00000000 Before first symbol
Trace; c011d7e6 <sys_rt_sigaction+9e/144>
Trace; c01175a4 <sys_wait4+398/3d0>
Trace; c0106fca <system_call+32/38>
Proc; gmake
>>EIP; 00000000 Before first symbol
Trace; c01175a4 <sys_wait4+398/3d0>
Trace; c0106fca <system_call+32/38>
Proc; sh
>>EIP; 00000000 Before first symbol
Trace; c011d7e6 <sys_rt_sigaction+9e/144>
Trace; c01175a4 <sys_wait4+398/3d0>
Trace; c0106fca <system_call+32/38>
Proc; gmake
>>EIP; c55f6000 <_end+522b584/105bd584> <=====
Trace; c013a07c <pipe_wait+7c/a4>
Trace; c013a154 <pipe_read+b0/1f8>
Trace; c0131db6 <sys_read+8e/c4>
Trace; c0106fca <system_call+32/38>
Proc; gcc
>>EIP; 00000000 Before first symbol
Trace; c01175a4 <sys_wait4+398/3d0>
Trace; c0106fca <system_call+32/38>
Proc; cc1plus
>>EIP; 00000000 Before first symbol
Trace; c0124390 <add_to_page_cache_unique+74/84>
Trace; c01245b4 <__lock_page+78/a0>
Trace; c01245f2 <lock_page+16/1c>
Trace; c012559c <filemap_nopage+188/344>
Trace; c0122012 <do_no_page+5a/d4>
Trace; c01220ec <handle_mm_fault+60/c0>
Trace; c0110604 <do_page_fault+178/4b4>
Trace; c011048c <do_page_fault+0/4b4>
Trace; c011b676 <update_wall_time+a/34>
Trace; c011b886 <timer_bh+36/2bc>
Trace; c011147c <schedule+404/614>
Trace; c011048c <do_page_fault+0/4b4>
Trace; c01070bc <error_code+34/3c>
Proc; sh
>>EIP; 00000000 Before first symbol
Trace; c01175a4 <sys_wait4+398/3d0>
Trace; c0106fca <system_call+32/38>
Proc; gcc
>>EIP; 00000000 Before first symbol
Trace; c01175a4 <sys_wait4+398/3d0>
Trace; c0106fca <system_call+32/38>
Proc; sh
>>EIP; 00000000 Before first symbol
Trace; c01175a4 <sys_wait4+398/3d0>
Trace; c0106fca <system_call+32/38>
Proc; gcc
>>EIP; 00000000 Before first symbol
Trace; c01175a4 <sys_wait4+398/3d0>
Trace; c0106fca <system_call+32/38>
Proc; sh
>>EIP; 00000000 Before first symbol
Trace; c01175a4 <sys_wait4+398/3d0>
Trace; c0106fca <system_call+32/38>
Proc; gcc
>>EIP; 00000000 Before first symbol
Trace; c01175a4 <sys_wait4+398/3d0>
Trace; c0106fca <system_call+32/38>
Proc; cc1plus
>>EIP; 00000000 Before first symbol
Trace; c01245b4 <__lock_page+78/a0>
Trace; c01245f2 <lock_page+16/1c>
Trace; c012559c <filemap_nopage+188/344>
Trace; c0122012 <do_no_page+5a/d4>
Trace; c01220ec <handle_mm_fault+60/c0>
Trace; c0110604 <do_page_fault+178/4b4>
Trace; c011048c <do_page_fault+0/4b4>
Trace; c01ee1f2 <ide_intr+166/18c>
Trace; c011b794 <update_process_times+20/94>
Trace; c010ef9c <smp_apic_timer_interrupt+ec/110>
Trace; c01070bc <error_code+34/3c>
Proc; cc1plus
>>EIP; 00000000 Before first symbol
Trace; c0124390 <add_to_page_cache_unique+74/84>
Trace; c01245b4 <__lock_page+78/a0>
Trace; c01245f2 <lock_page+16/1c>
Trace; c012559c <filemap_nopage+188/344>
Trace; c0122012 <do_no_page+5a/d4>
Trace; c01220ec <handle_mm_fault+60/c0>
Trace; c0110604 <do_page_fault+178/4b4>
Trace; c011048c <do_page_fault+0/4b4>
Trace; c01bbf48 <handle_kbd_event+104/180>
Trace; c011147c <schedule+404/614>
Trace; c011048c <do_page_fault+0/4b4>
Trace; c0108866 <do_IRQ+c6/ec>
Trace; c01070bc <error_code+34/3c>
Proc; cc1plus
>>EIP; 00000000 Before first symbol
Trace; c01245b4 <__lock_page+78/a0>
Trace; c01245f2 <lock_page+16/1c>
Trace; c012559c <filemap_nopage+188/344>
Trace; c0122012 <do_no_page+5a/d4>
Trace; c01220ec <handle_mm_fault+60/c0>
Trace; c0110604 <do_page_fault+178/4b4>
Trace; c011048c <do_page_fault+0/4b4>
Trace; c01177f0 <it_real_fn+40/48>
Trace; c011bab4 <timer_bh+264/2bc>
Trace; c01233f4 <do_brk+114/20c>
Trace; c011b794 <update_process_times+20/94>
Trace; c010ef9c <smp_apic_timer_interrupt+ec/110>
Trace; c01070bc <error_code+34/3c>
Proc; sendmail
>>EIP; 00000002 Before first symbol <=====
Trace; c01bf512 <submit_bh+56/70>
Trace; c0132cae <__wait_on_buffer+6e/94>
Trace; c0133e40 <bread+44/64>
Trace; c0197a24 <reiserfs_bread+14/1c>
Trace; c0193d28 <search_by_key+70/c10>
Trace; c0194972 <search_for_position_by_key+aa/3a4>
Trace; c0185e64 <make_cpu_key+38/40>
Trace; c0186048 <_get_block_create_0+9c/5a0>
Trace; c0186062 <_get_block_create_0+b6/5a0>
Trace; c018687c <reiserfs_get_block+13c/d88>
Trace; c01bee66 <__make_request+156/66c>
Trace; c01296d6 <kmem_cache_alloc+8a/12c>
Trace; c0134798 <block_read_full_page+e4/21c>
Trace; c012c132 <__alloc_pages+46/1ac>
Trace; c0124390 <add_to_page_cache_unique+74/84>
Trace; c0189362 <reiserfs_readpage+e/14>
Trace; c0186740 <reiserfs_get_block+0/d88>
Trace; c0124444 <page_cache_read+a4/c8>
Trace; c012448e <read_cluster_nonblocking+26/40>
Trace; c012556a <filemap_nopage+156/344>
Trace; c0122012 <do_no_page+5a/d4>
Trace; c01220ec <handle_mm_fault+60/c0>
Trace; c0110604 <do_page_fault+178/4b4>
Trace; c011048c <do_page_fault+0/4b4>
Trace; c011b676 <update_wall_time+a/34>
Trace; c011b886 <timer_bh+36/2bc>
Trace; c011b38e <tqueue_bh+16/1c>
Trace; c01187e0 <bh_action+4c/88>
Trace; c01186b8 <tasklet_hi_action+6c/a0>
Trace; c011843e <do_softirq+6e/cc>
Trace; c010887a <do_IRQ+da/ec>
Trace; c01070bc <error_code+34/3c>
Some possibly helpful details:
The compile was happening on the ReiserFS partition that was 93% full. As you
can see from /proc/meminfo, I have 0 swap - this is intentional.
--
MySQL Development Team
For technical support contracts, visit https://order.mysql.com/
__ ___ ___ ____ __
/ |/ /_ __/ __/ __ \/ / Sasha Pachev <[email protected]>
/ /|_/ / // /\ \/ /_/ / /__ MySQL AB, http://www.mysql.com/
/_/ /_/\_, /___/\___\_\___/ Provo, Utah, USA
<___/
On Wed, 7 Nov 2001, Sasha Pachev wrote:
> Summary:
>
> System slowdown under unexplained excessive disk I/O
>
> Full description:
>
> While running X, KDE, having a few windows open, I ran make -j4 on MySQL
> source tree. I do this all the time and it usually works just fine - the
> system is a little bit unresponsive. However, occasionally the system becomes
> completely unresponsive - the disk goes crazy, the machine pings but neither
> ssh or telnet work - connection to the port is established, but nothing
> further than that. It does respond to magic SysRQ. I was able to get a memory
> info dump + stack traces into syslog, included below. The filesystem is
> ReiserFS.
I had a similar Problem while running sql-bench on Mysql ( must be mysql :-) ).
First I thought it happened since the Mysql-Tables were on the root-fs and
the Disk did not manage to retrieve normal libs and files for login usw.
But the System (2.4.14-pre8) took about 25 seconds to give a screen after
wakening from apm (triggered by xscreensaver under gnome).
Console-Switching took even longer. Mouse-Movement stopped etc.
Disk-Activity was very high.
I had seen smaller Problems with huge loads on the root-fs earlier. But
never had a Problem with switching consoles or mouse.
Oktay
I just did a quick retest with sql-bench on my machine and found that
reiserfs seems to be a source for that problem.
I did not reproduce the whole problem, since i stopped
test-create before the system was to unresponsive.
But I started test-create on a partition with
reiserfs. After much files were created the system to very
long for bash command-completion or doing simple thing
free, top etc. this was even after test-create was stopped.
It took about 30 seconds till the system responded normally.
Same test with ext2 on the same partition showed no problems.
Reran the test with reiserfs and the system got unresponsive
again.
Did you use reiserfs ?
Oktay
On Thu, 8 Nov 2001, Oktay Akbal wrote:
> On Wed, 7 Nov 2001, Sasha Pachev wrote:
>
> > Summary:
> >
> > System slowdown under unexplained excessive disk I/O
> >
> > Full description:
> >
> > While running X, KDE, having a few windows open, I ran make -j4 on MySQL
> > source tree. I do this all the time and it usually works just fine - the
> > system is a little bit unresponsive. However, occasionally the system becomes
> > completely unresponsive - the disk goes crazy, the machine pings but neither
> > ssh or telnet work - connection to the port is established, but nothing
> > further than that. It does respond to magic SysRQ. I was able to get a memory
> > info dump + stack traces into syslog, included below. The filesystem is
> > ReiserFS.
On Wed, Nov 07, 2001 at 09:25:38PM -0700, Sasha Pachev wrote:
>
> Summary:
>
> System slowdown under unexplained excessive disk I/O
>
> Full description:
>
> While running X, KDE, having a few windows open, I ran make -j4 on MySQL
> source tree. I do this all the time and it usually works just fine - the
> system is a little bit unresponsive. However, occasionally the system becomes
> completely unresponsive - the disk goes crazy, the machine pings but neither
> ssh or telnet work - connection to the port is established, but nothing
> further than that. It does respond to magic SysRQ. I was able to get a memory
> info dump + stack traces into syslog, included below. The filesystem is
> ReiserFS.
>
> Keywords:
>
> vm, ReiserFS, heavy disk I/O,
Let me guess, IDE disks? Anyway, this is a FAQ. Go http://www.namesys.com, click
on the FAQ, and look at #15.
This issue really can be a problem as the entire machine will freeze up for
a good chunk of time while the disk churns away. Strangely enough, I've
only noticed it on some machines, not all. But it's enough of a problem to
cause me to switch to ext3.
-Dave
On Wed, 7 Nov 2001, David Rees wrote:
> >
> > Keywords:
> >
> > vm, ReiserFS, heavy disk I/O,
>
> Let me guess, IDE disks? Anyway, this is a FAQ. Go http://www.namesys.com, click
> on the FAQ, and look at #15.
To bad I did not see the Reiserfs-Keyword before asking...
In my case this happened on scsi-disks on a adaptec 2940.
I read the faq-entry, but I do not understand why this should only apply
to IDE. Maybe my old-scsi disks suffer the same problem ?
Oktay
On Thu, Nov 08, 2001 at 08:45:12AM +0100, Oktay Akbal wrote:
> On Wed, 7 Nov 2001, David Rees wrote:
>
> > >
> > > Keywords:
> > >
> > > vm, ReiserFS, heavy disk I/O,
> >
> > Let me guess, IDE disks? Anyway, this is a FAQ. Go http://www.namesys.com, click
> > on the FAQ, and look at #15.
>
> To bad I did not see the Reiserfs-Keyword before asking...
> In my case this happened on scsi-disks on a adaptec 2940.
> I read the faq-entry, but I do not understand why this should only apply
> to IDE. Maybe my old-scsi disks suffer the same problem ?
It could affect certain SCSI disks/controllers just as well. You might want
to ask this question on the reiserfs list to see if they have any
solutions...
-Dave
Oktay Akbal writes:
> I just did a quick retest with sql-bench on my machine and found that
> reiserfs seems to be a source for that problem.
> I did not reproduce the whole problem, since i stopped
> test-create before the system was to unresponsive.
>
> But I started test-create on a partition with
> reiserfs. After much files were created the system to very
> long for bash command-completion or doing simple thing
> free, top etc. this was even after test-create was stopped.
> It took about 30 seconds till the system responded normally.
>
> Same test with ext2 on the same partition showed no problems.
>
> Reran the test with reiserfs and the system got unresponsive
> again.
It doesn't seem that you have the same problem as Sasha. Reiserfs has
known problem with dealing with large holes, so may be this explains
situation. We'll try to reproduce your results.
>
> Did you use reiserfs ?
>
> Oktay
>
Nikita.
>
> On Thu, 8 Nov 2001, Oktay Akbal wrote:
>
> > On Wed, 7 Nov 2001, Sasha Pachev wrote:
> >
> > > Summary:
> > >
> > > System slowdown under unexplained excessive disk I/O
> > >
> > > Full description:
> > >
> > > While running X, KDE, having a few windows open, I ran make -j4 on MySQL
> > > source tree. I do this all the time and it usually works just fine - the
> > > system is a little bit unresponsive. However, occasionally the system becomes
> > > completely unresponsive - the disk goes crazy, the machine pings but neither
> > > ssh or telnet work - connection to the port is established, but nothing
> > > further than that. It does respond to magic SysRQ. I was able to get a memory
> > > info dump + stack traces into syslog, included below. The filesystem is
> > > ReiserFS.
>
On Wednesday 07 November 2001 10:33 pm, Oktay Akbal wrote:
> Same test with ext2 on the same partition showed no problems.
>
> Reran the test with reiserfs and the system got unresponsive
> again.
>
> Did you use reiserfs ?
Yes, I did. I suspected from the trace resolve this could very well be
ReiserFS issue.
--
MySQL Development Team
For technical support contracts, visit https://order.mysql.com/
__ ___ ___ ____ __
/ |/ /_ __/ __/ __ \/ / Sasha Pachev <[email protected]>
/ /|_/ / // /\ \/ /_/ / /__ MySQL AB, http://www.mysql.com/
/_/ /_/\_, /___/\___\_\___/ Provo, Utah, USA
<___/