[1.] One line summary of the problem
System hangs on high I/O and doesn't boot properly in smp mode.
[2.] Full description of the problem/report:
Copying large amounts of data from an IDE disk to a SCSI raid array (3Ware
Escalade 7800) hangs the system completely.
System is a Tyan S2469 with two Athlon MP cpus.
Only one is used, because when booting with an SMP kernel, there are
endless time-outs and resets on the 3ware card.
The exact same behaviour was observed on a Tyan S2468 system with
the same 3Ware card. The 3Ware card has been RMAd (it wasn't faulty,
but I did it anyway to be sure), all the disks on the array have
been checked.
[3.] Keywords (i.e., modules, networking, kernel):
kernel, vm, ide, scsi
[4.] Kernel version (from /proc/version):
Linux version 2.4.20-3-k7 (herbert@gondolin) (gcc version 3.3 (Debian)) #1 Sun Jun 8 01:35:14 EST 2003
[5.] Output of Oops.. message (if applicable) with symbolic information
resolved (see Documentation/oops-tracing.txt)
No oops
[6.] A small shell script or example program which triggers the
problem (if possible)
cp large amount of data from ide disk to scsi array
[7.] Environment
[7.1.] Software (add the output of the ver_linux script here)
Gnu C 3.2.3
Gnu make 3.80
util-linux 2.11z
mount 2.11z
modutils 2.4.21
e2fsprogs 1.34-WIP
PPP 2.4.1
Linux C Library 2.3.1
Dynamic linker (ldd) 2.3.1
Procps 3.1.9
Net-tools 1.60
Console-tools 0.2.3
Sh-utils 5.0
Modules Loaded mousedev input agpgart lp parport autofs4 af_packet ext2 i2c-dev i2c-core isa-pnp lvm-mod loop 3w-xxxx sd_mod scsi_mod e1000 eepro100 mii rtc ext3 jbd ide-disk ide-probe-mod ide-mod unix
[7.2.] Processor information (from /proc/cpuinfo):
processor : 0
vendor_id : AuthenticAMD
cpu family : 6
model : 6
model name : AMD Athlon(tm) MP 2000+
stepping : 2
cpu MHz : 1666.741
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr sse syscall mmxext 3dnowext 3dnow
bogomips : 3322.67
Only one cpu active, though it is a dual cpu system.
The system doesn't even boot with two cpus :(, on bootup there are continuously
time-outs on the 3ware card and it gets reset ad infinitum (and ad nauseam...).
[7.3.] Module information (from /proc/modules):
mousedev 4148 0 (autoclean)
input 3520 0 (autoclean) [mousedev]
agpgart 37344 0 (autoclean) (unused)
lp 6816 0 (autoclean) (unused)
parport 25992 0 (autoclean) [lp]
autofs4 9780 2 (autoclean)
af_packet 13448 1 (autoclean)
ext2 34688 1 (autoclean)
i2c-dev 4676 0 (unused)
i2c-core 13476 0 [i2c-dev]
isa-pnp 31888 0 (unused)
lvm-mod 58624 12
loop 9560 0
3w-xxxx 32992 1
sd_mod 11660 2
scsi_mod 93984 2 [3w-xxxx sd_mod]
e1000 51340 1
eepro100 19732 1
mii 2432 0 [eepro100]
rtc 6792 0 (autoclean)
ext3 63808 6 (autoclean)
jbd 41764 6 (autoclean) [ext3]
ide-disk 12448 4 (autoclean)
ide-probe-mod 9744 0 (autoclean)
ide-mod 167736 4 (autoclean) [ide-disk ide-probe-mod]
unix 15148 76 (autoclean)
[7.4.] Loaded driver and hardware information (/proc/ioports, /proc/iomem)
/proc/ioports
0000-001f : dma1
0020-003f : pic1
0040-005f : timer
0060-006f : keyboard
0070-007f : rtc
0080-008f : dma page reg
00a0-00bf : pic2
00c0-00df : dma2
00f0-00ff : fpu
0170-0177 : ide1
01f0-01f7 : ide0
0376-0376 : ide1
03c0-03df : vga+
03f6-03f6 : ide0
0cf8-0cff : PCI conf1
1000-103f : Intel Corp. 82545EM Gigabit Ethernet Controller
1000-103f : e1000
1050-105f : 3ware Inc 3ware 7000-series ATA-RAID
1050-105f : 3ware Storage Controller
1060-1063 : Advanced Micro Devices [AMD] AMD-760 MP [IGD4-2P] System Controller
2000-2fff : PCI Bus #02
2000-20ff : ATI Technologies Inc Rage XL
2400-243f : Intel Corp. 82557/8/9 [Ethernet Pro 100]
2400-243f : eepro100
f000-f00f : Advanced Micro Devices [AMD] AMD-768 [Opus] IDE
f000-f007 : ide0
f008-f00f : ide1
/proc/iomem
00000000-0009efff : System RAM
0009f000-0009ffff : reserved
000a0000-000bffff : Video RAM area
000c0000-000c7fff : Video ROM
000c8000-000c8fff : Extension ROM
000e0000-000effff : Extension ROM
000f0000-000fffff : System ROM
00100000-3feeffff : System RAM
00100000-001f6826 : Kernel code
001f6827-00258503 : Kernel data
3fef0000-3fef5fff : ACPI Tables
3fef6000-3fefffff : ACPI Non-volatile Storage
3ff00000-3ff7ffff : System RAM
3ff80000-3fffffff : reserved
f4000000-f47fffff : 3ware Inc 3ware 7000-series ATA-RAID
f4800000-f481ffff : Intel Corp. 82545EM Gigabit Ethernet Controller
f4800000-f481ffff : e1000
f4820000-f482000f : 3ware Inc 3ware 7000-series ATA-RAID
f4900000-f5ffffff : PCI Bus #02
f4900000-f4900fff : Advanced Micro Devices [AMD] AMD-768 [Opus] USB
f4901000-f4901fff : ATI Technologies Inc Rage XL
f4902000-f4902fff : Intel Corp. 82557/8/9 [Ethernet Pro 100]
f4902000-f4902fff : eepro100
f4920000-f493ffff : Intel Corp. 82557/8/9 [Ethernet Pro 100]
f5000000-f5ffffff : ATI Technologies Inc Rage XL
f6200000-f6200fff : Advanced Micro Devices [AMD] AMD-760 MP [IGD4-2P] System Controller
f8000000-fbffffff : Advanced Micro Devices [AMD] AMD-760 MP [IGD4-2P] System Controller
fec00000-fec03fff : reserved
fee00000-fee00fff : reserved
fff80000-ffffffff : reserved
[7.5.] PCI information ('lspci -vvv' as root)
kalimero:~# lspci -vvv
00:00.0 Host bridge: Advanced Micro Devices [AMD] AMD-760 MP [IGD4-2P] System Controller (rev 11)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ >SERR- <PERR-
Latency: 32
Region 0: Memory at f8000000 (32-bit, prefetchable) [size=64M]
Region 1: Memory at f6200000 (32-bit, prefetchable) [size=4K]
Region 2: I/O ports at 1060 [disabled] [size=4]
Capabilities: [a0] AGP version 2.0
Status: RQ=16 Iso- ArqSz=0 Cal=0 SBA+ ITACoh- GART64- HTrans- 64bit- FW- AGP3- Rate=x1,x2
Command: RQ=1 ArqSz=0 Cal=0 SBA+ AGP+ GART64- 64bit- FW- Rate=<none>
00:01.0 PCI bridge: Advanced Micro Devices [AMD] AMD-760 MP [IGD4-2P] AGP Bridge (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 64
Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: fff00000-000fffff
Prefetchable memory behind bridge: fff00000-000fffff
BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-768 [Opus] ISA (rev 05)
Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-768 [Opus] IDE (rev 04) (prog-if 8a [Master SecP PriP])
Subsystem: Advanced Micro Devices [AMD] AMD-768 [Opus] IDE
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Region 4: I/O ports at f000 [size=16]
00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-768 [Opus] ACPI (rev 03)
Subsystem: Advanced Micro Devices [AMD] AMD-768 [Opus] ACPI
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
00:08.0 RAID bus controller: 3ware Inc 3ware 7000-series ATA-RAID (rev 01)
Subsystem: 3ware Inc 3ware 7000-series ATA-RAID
Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 72 (2250ns min), cache line size 10
Interrupt: pin A routed to IRQ 16
Region 0: I/O ports at 1050 [size=16]
Region 1: Memory at f4820000 (32-bit, non-prefetchable) [size=16]
Region 2: Memory at f4000000 (32-bit, non-prefetchable) [size=8M]
Expansion ROM at <unassigned> [disabled] [size=64K]
Capabilities: [40] Power Management version 1
Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
00:0b.0 Ethernet controller: Intel Corp. 82545EM Gigabit Ethernet Controller (Copper) (rev 01)
Subsystem: Intel Corp. PRO/1000 MT Server Adapter
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 64 (63750ns min), cache line size 10
Interrupt: pin A routed to IRQ 19
Region 0: Memory at f4800000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at 1000 [size=64]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [e4] PCI-X non-bridge device.
Command: DPERE- ERO+ RBC=0 OST=0
Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, DC=simple, DMMRBC=0, DMOST=0, DMCRS=0, RSCEM- Capabilities: [f0] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable-
Address: 0000000000000000 Data: 0000
00:10.0 PCI bridge: Advanced Micro Devices [AMD] AMD-768 [Opus] PCI (rev 05) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ >SERR- <PERR-
Latency: 99
Bus: primary=00, secondary=02, subordinate=02, sec-latency=68
I/O behind bridge: 00002000-00002fff
Memory behind bridge: f4900000-f5ffffff
Prefetchable memory behind bridge: fff00000-000fffff
BridgeCtl: Parity- SERR- NoISA+ VGA+ MAbort- >Reset- FastB2B-
02:00.0 USB Controller: Advanced Micro Devices [AMD] AMD-768 [Opus] USB (rev 07) (prog-if 10 [OHCI])
Subsystem: Advanced Micro Devices [AMD] AMD-768 [Opus] USB
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 64 (20000ns max)
Interrupt: pin D routed to IRQ 19
Region 0: Memory at f4900000 (32-bit, non-prefetchable) [size=4K]
02:07.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27) (prog-if 00 [VGA])
Subsystem: ATI Technologies Inc Rage XL
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping+ SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 66 (2000ns min), cache line size 10
Region 0: Memory at f5000000 (32-bit, non-prefetchable) [size=16M]
Region 1: I/O ports at 2000 [size=256]
Region 2: Memory at f4901000 (32-bit, non-prefetchable) [size=4K]
Expansion ROM at <unassigned> [disabled] [size=128K]
Capabilities: [5c] Power Management version 2
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
02:08.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] (rev 10)
Subsystem: Intel Corp. EtherExpress PRO/100 S Server Adapter
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 66 (2000ns min, 14000ns max), cache line size 10
Interrupt: pin A routed to IRQ 18
Region 0: Memory at f4902000 (32-bit, non-prefetchable) [size=4K]
Region 1: I/O ports at 2400 [size=64]
Region 2: Memory at f4920000 (32-bit, non-prefetchable) [size=128K]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=2 PME-
[7.6.] SCSI information (from /proc/scsi/scsi)
kalimero:~# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: 3ware Model: 3w-xxxx Rev: 1.0
Type: Direct-Access ANSI SCSI revision: ffffffff
[7.7.] Other information that might be relevant to the problem
(please look in /proc and include all information that you
think to be relevant):
The output of vmstat while the copying took place (after the last line, the
system just hangs):
vincent@kalimero:~$ vmstat -n 100
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 821768 12628 57364 0 0 5176 44 296 421 2 4 94 0
1 1 0 451012 20380 392808 0 0 1804 1500 246 804 11 4 86 0
0 2 1412 9580 14752 914764 0 10 16251 16529 574 1399 4 19 77 0
Booting in smp mode now works using acpi=off and acpismp=force on the
kernel command line.
Could anyone care to comment what these options do ?
Would I be far off by saying that it disables the use of ACPI in the
kernel, except for setting up the APICs and the CPUs, for which it will
explicitly use the data provided by the ACPI BIOS ?
Is there something I'm missing by not using ACPI ? :)
best regards,
Vincent
PS: will test if the system still locks up soon, I hope not...
On Mon, Jul 07, 2003 at 02:30:07AM +0200, Vincent Touquet wrote:
>PS: will test if the system still locks up soon, I hope not...
So it does lock up again :(((
But now I was able to quickly switch to console and grab the contents of
/var/log/messages before it totally hanged. I can usually tell when the
hang is going to happen: activity on the array stops, then I have a few
more seconds till it hangs completely ....
The message was:
Jul 7 02:45:36 kalimero kernel: 3w-xxxx: scsi0: Unit #0:
command (f7618800) timed out, resetting card.
Then of course, the system totally hangs.
The same problem occurred to me on a Tyan S2468, with a different 3Ware
card. This has to be a kernel problem ... (or BIOS problem, if the S2468
and 2469 BIOSs are sufficiently similar ?).
thanks for any help,
Vincent
Vincent Touquet <[email protected]> wrote:
>
> Booting in smp mode now works using acpi=off and acpismp=force on the
> kernel command line.
Do 2.4 kernels boot both CPUs OK? Do you use ACPI in 2.4?
The ACPI changes in 2.5 were recently merged into 2.4, so current
2.4 may be broken for you too.
On Sun, Jul 06, 2003 at 05:52:32PM -0700, Andrew Morton wrote:
>Do 2.4 kernels boot both CPUs OK? Do you use ACPI in 2.4?
The dmesg output shows both CPUs being put online.
/proc/cpuinfo lists both CPUs too
I don't use ACPI, even though I enabled it in the BIOS so I could use
acpismp=force (else the kernel finds no ACPI table of course).
I will see if disabling it in the BIOS helps, I will check it tomorrow
(I have to let the array be rebuilt anyway).
>The ACPI changes in 2.5 were recently merged into 2.4, so current
>2.4 may be broken for you too.
Hm, I don't want anything to do with ACPI actually :)
Anything that could give me a lock free smp machine with a working 3ware
card, would make me happy.
I hope I don't have to wait for the 2.6 series to have a stable machine.
best regards,
Vincent
Vincent Touquet <[email protected]> wrote:
>
> On Mon, Jul 07, 2003 at 02:30:07AM +0200, Vincent Touquet wrote:
> >PS: will test if the system still locks up soon, I hope not...
>
> So it does lock up again :(((
>
> But now I was able to quickly switch to console and grab the contents of
> /var/log/messages before it totally hanged. I can usually tell when the
> hang is going to happen: activity on the array stops, then I have a few
> more seconds till it hangs completely ....
>
> The message was:
>
> Jul 7 02:45:36 kalimero kernel: 3w-xxxx: scsi0: Unit #0:
> command (f7618800) timed out, resetting card.
>
> Then of course, the system totally hangs.
The next step would be to try some older versions. There was a big 3ware
update between 2.5.64 and 2.5.65. Can you try both of those?
hmm, I see a "fixme" and an interruptible_sleep_on_timeout() around that
error message. Do the hangs happen on uniprocessor, non-preemptible
kernels?
On Sun, Jul 06, 2003 at 07:19:41PM -0700, Andrew Morton wrote:
>The next step would be to try some older versions. There was a big 3ware
>update between 2.5.64 and 2.5.65. Can you try both of those?
I'm struggling with the 2.4 series for now :)
Haven't tried 2.5.x yet, as it is a production machine.
I'm compiling 2.4.21 with magic sysreq. support now.
>hmm, I see a "fixme" and an interruptible_sleep_on_timeout() around that
>error message. Do the hangs happen on uniprocessor, non-preemptible
>kernels?
Yes, the hangs happen on uniprocessor, non-preemptible kernels.
The smp kernels which don't boot bring up that message, which is solved
by adding acpi=off to the kernel command line, acpismp=force does not
seem vital here.
I will see if I can reproduce the hang and get a trace.
regards,
Vincent
I was pulling my hair out (whats left of it) last week trying to get a Tyan
2466 dual AMD 2800 MP (512 MB REGISTERED DDR, 3ware 7000-2 w/2 WD2000 drives
and a WD800 IDE system drive, 2.4.21 Debian, ReiserFS) to run reliably under
heavy disk and i/o (16 frame grabbers running a surveillance application).
Eventually I would get "hda: missed interrupt .." and soon after ReiserFS
file corruption. I suspected memory and tried unbuffered DDR, and 4
manufacturers of buffered DDR, all with the same results. So I took pulled
out the system drive, 3ware controllers and data drives, and frame grabbers
and put them on a Intel P4/Intel motherboard, and everything booted and
worked like a charm (though with more CPU load). So I am wondering now if
this file system corruption under heavy i/o load has something to do with SMP
code?
On Sunday 06 July 2003 10:19 pm, Andrew Morton wrote:
> Vincent Touquet <[email protected]> wrote:
> > On Mon, Jul 07, 2003 at 02:30:07AM +0200, Vincent Touquet wrote:
> > >PS: will test if the system still locks up soon, I hope not...
> >
> > So it does lock up again :(((
> >
> > But now I was able to quickly switch to console and grab the contents of
> > /var/log/messages before it totally hanged. I can usually tell when the
> > hang is going to happen: activity on the array stops, then I have a few
> > more seconds till it hangs completely ....
> >
> > The message was:
> >
> > Jul 7 02:45:36 kalimero kernel: 3w-xxxx: scsi0: Unit #0:
> > command (f7618800) timed out, resetting card.
> >
> > Then of course, the system totally hangs.
>
> The next step would be to try some older versions. There was a big 3ware
> update between 2.5.64 and 2.5.65. Can you try both of those?
>
> hmm, I see a "fixme" and an interruptible_sleep_on_timeout() around that
> error message. Do the hangs happen on uniprocessor, non-preemptible
> kernels?
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
--
Joe Briggs
Briggs Media Systems
105 Burnsen Ave.
Manchester NH 01304 USA
TEL 603-232-3115 FAX 603-625-5809 MOBILE 603-493-2386
http://www.briggsmedia.com
On Mon, Jul 07, 2003 at 07:43:27AM -0400, joe briggs wrote:
>I was pulling my hair out (whats left of it) last week trying to get a Tyan
>2466 dual AMD 2800 MP (512 MB REGISTERED DDR, 3ware 7000-2 w/2 WD2000 drives
>and a WD800 IDE system drive, 2.4.21 Debian, ReiserFS) to run reliably under
>heavy disk and i/o (16 frame grabbers running a surveillance application).
>Eventually I would get "hda: missed interrupt .." and soon after ReiserFS
>file corruption. I suspected memory and tried unbuffered DDR, and 4
>manufacturers of buffered DDR, all with the same results. So I took pulled
>out the system drive, 3ware controllers and data drives, and frame grabbers
>and put them on a Intel P4/Intel motherboard, and everything booted and
>worked like a charm (though with more CPU load). So I am wondering now if
>this file system corruption under heavy i/o load has something to do with SMP
>code?
Hi Joe :)
I'm still pulling my hair out here too ;)
I don't think smp is to blame, as I have lockups with UP too ...
I think there is something badly wrong at a low level when running any
2.4.x kernel on a Tyan mainboard, which shows up at high IO.
Where you using any IDE related stuff on your Tyan ?
Did you enable highmem ?
I just finished compiling 2.4.21 here with magic sysreq support, I hope
to get some useful data after the lockup.
After the array has been rebuilt though (sigh).
So far I see two different issues, possibly related:
- smp kernel does not boot unless given acpi=off
Without this command line option there is an _endless_ resetting of the
3Ware card at boot time
- lockup when copying large amount of data from disk (ide) to array
(scsi), on the console it says that there is a time-out on a 3Ware
command and the card needs to be reset
The same problem was shown with a copy over the network onto the array
(so not touching ide, except probably for swap).
There is definitely an issue with the mainboard and the kernel here.
I swapped everything (psu, 3Ware card, disks, mainboard !), to no avail.
Any help would be much appreciated.
There should be people running Linux on these boards ?
regards,
Vincent
PS: a possibility is too that the board needs ACPI built into the kernel
in order to work ?
PPS: I own a Tyan S2468 too in my webserver, though the IO there is much
more modest (no 3Ware card either), so far no problems there (knock on
wood)
I think I collected some data we can look at :)
Following is by reproducing the problem on a 2.4.21 Linus kernel,
compiled with magic sysreq. support and all possible debug options (even
frame pointers). Kernel boot options are devfs=mount acpi=off and
nmi_watchdog=1
All of the following data have been acquired as close to the hangup as
possible:
[output of ps -aux]
Several processes stuck in a D state ...
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 1208 464 ? S 12:36 0:05 init
root 2 0.0 0.0 0 0 ? SW 12:36 0:00 [keventd]
root 3 0.0 0.0 0 0 ? SWN 12:36 0:00 [ksoftirqd_CPU0]
root 4 0.0 0.0 0 0 ? SWN 12:36 0:00 [ksoftirqd_CPU1]
root 5 0.0 0.0 0 0 ? SW 12:36 0:00 [kswapd]
root 6 0.0 0.0 0 0 ? SW 12:36 0:00 [bdflush]
root 7 0.0 0.0 0 0 ? DW 12:36 0:00 [kupdated]
root 8 0.0 0.0 0 0 ? RW 12:36 0:00 [scsi_eh_0]
root 9 0.0 0.0 0 0 ? SW 12:36 0:04 [kjournald]
root 24 0.0 0.1 2604 1912 ? S 12:37 0:04 /sbin/devfsd /dev
root 122 0.0 0.0 0 0 ? DW 12:37 0:00 [kjournald]
root 123 0.0 0.0 0 0 ? SW 12:37 0:00 [kjournald]
root 124 0.0 0.0 0 0 ? SW 12:37 0:00 [kjournald]
root 125 0.0 0.0 0 0 ? SW 12:37 0:00 [kjournald]
root 126 0.0 0.0 0 0 ? SW 12:37 0:00 [kjournald]
root 181 0.0 0.0 1840 960 ? S 12:37 0:00 dhclient -pf /var/run/dhclient.eth1.pid eth1
daemon 185 0.0 0.0 1316 404 ? S 12:37 0:00 /sbin/portmap
root 287 0.0 0.0 1288 544 ? S 12:37 0:00 /sbin/syslogd
root 290 0.0 0.1 1984 1256 ? S 12:37 0:00 /sbin/klogd
root 321 0.0 0.1 8380 1072 ? S 12:37 0:00 /usr/sbin/3dmd
root 322 0.0 0.1 8380 1072 ? S 12:37 0:00 /usr/sbin/3dmd
root 323 0.0 0.1 8380 1072 ? S 12:37 0:00 /usr/sbin/3dmd
root 325 0.0 0.1 8380 1072 ? D 12:37 0:02 /usr/sbin/3dmd
root 327 0.0 0.1 8380 1072 ? S 12:37 0:00 /usr/sbin/3dmd
root 394 0.0 0.0 1308 588 ? S 12:37 0:00 /usr/sbin/automount --pid-file=/var/run/autofs/_:var_:autofs_:misc.pid --timeout=300 /var/autofs/misc file /etc/auto.misc
root 410 0.0 0.0 1312 592 ? S 12:37 0:00 /usr/sbin/automount --pid-file=/var/run/autofs/_:var_:autofs_:net.pid --timeout=300 /var/autofs/net program /etc/auto.net
root 601 0.0 0.0 1236 484 ? S 12:37 0:00 /usr/sbin/inetd
root 605 0.0 0.0 1296 544 ? S 12:37 0:00 /usr/sbin/lpd
root 702 0.0 0.0 2396 984 ? S 12:37 0:00 /usr/lib/postfix/master
postfix 706 0.0 0.0 2396 964 ? S 12:37 0:00 nqmgr -l -n qmgr -t fifo -u -c
root 708 0.0 0.1 3052 1444 ? S 12:37 0:00 /usr/sbin/nmbd -D
root 709 0.0 0.1 3096 1328 ? S 12:37 0:00 /usr/sbin/nmbd -D
root 711 0.0 0.1 3868 1392 ? S 12:37 0:00 /usr/sbin/smbd -D
root 717 0.0 0.1 2684 1348 ? S 12:37 0:00 /usr/sbin/sshd
root 731 0.0 0.0 1452 748 ? S 12:37 0:00 /sbin/rpc.statd
root 734 0.0 0.1 1912 1904 ? SL 12:37 0:00 /usr/sbin/ntpd
root 737 0.0 0.0 1708 752 ? S 12:37 0:00 /usr/sbin/rpc.nfsd
root 739 0.0 0.0 1708 772 ? S 12:37 0:00 /usr/sbin/rpc.mountd
daemon 746 0.0 0.0 1316 552 ? S 12:37 0:00 /usr/sbin/atd
root 749 0.0 0.0 1396 664 ? S 12:37 0:00 /usr/sbin/cron
root 754 0.0 0.0 2152 656 ? S 12:37 0:00 /usr/bin/kdm
root 761 0.0 0.0 1200 444 vc/1 S 12:37 0:00 /sbin/getty 38400 tty1
root 762 0.0 0.0 1200 444 vc/2 S 12:37 0:00 /sbin/getty 38400 tty2
root 763 0.0 0.0 1200 444 vc/3 S 12:37 0:00 /sbin/getty 38400 tty3
root 764 0.0 0.0 1200 444 vc/4 S 12:37 0:00 /sbin/getty 38400 tty4
root 765 0.0 0.0 1200 444 vc/5 S 12:37 0:00 /sbin/getty 38400 tty5
root 766 0.0 0.0 1200 444 vc/6 S 12:37 0:00 /sbin/getty 38400 tty6
root 767 2.0 1.3 23720 13924 ? S< 12:37 2:16 /usr/X11R6/bin/X -dpi 100 vt7 -auth /var/lib/kdm/authfiles/A:0-3HOxUJ
root 769 0.0 0.1 2640 1100 ? S 12:37 0:00 -:0
root 771 0.0 0.5 8172 5860 ? S 12:37 0:00 /usr/bin/perl /usr/share/usermin/miniserv.pl /etc/usermin/miniserv.conf
root 772 0.0 0.5 8172 5860 ? S 12:37 0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
root 965 0.0 0.1 5764 1808 ? S 12:39 0:00 sshd: vincent [priv]
vincent 967 0.0 0.1 5852 1892 ? S 12:39 0:03 sshd: vincent@pts/0
vincent 968 0.0 0.1 2276 1336 pts/0 S 12:39 0:00 -bash
root 1485 0.0 0.1 2288 1388 pts/0 S 12:44 0:05 -su
vincent 2149 0.0 0.0 2040 976 ? S 12:48 0:00 /bin/sh /usr/bin/x-session-manager
vincent 2174 0.0 0.0 2368 812 ? S 12:48 0:00 /usr/bin/ssh-agent x-session-manager
vincent 2181 0.0 0.4 15344 4884 ? S 12:48 0:00 kdeinit: Running...
vincent 2187 0.0 0.5 15904 6028 ? S 12:48 0:00 kdeinit: klauncher
vincent 2190 0.1 0.6 16368 7228 ? S 12:48 0:07 kdeinit: kded
vincent 2198 0.0 0.2 5348 2724 ? S 12:48 0:01 /usr/bin/artsd -F 10 -S 4096 -s 60 -m artsmessage -l 3 -f
vincent 2209 0.0 0.7 18176 7944 ? S 12:48 0:00 kdeinit: knotify
vincent 2210 0.0 0.4 10800 4804 ? S 12:48 0:00 ksmserver --restore
vincent 2211 0.1 0.8 17456 8892 ? S 12:48 0:06 kdeinit: kwin -session 11c0a8000a000105049672100000008610000
vincent 2215 0.0 0.9 17044 9428 ? S 12:48 0:01 kdeinit: kdesktop
vincent 2217 0.0 1.0 18400 10336 ? S 12:48 0:04 kdeinit: kicker
vincent 2247 0.0 0.7 16476 7672 ? S 12:48 0:01 kdeinit: klipper -icon klipper -miniicon klipper
vincent 2252 0.0 0.6 13220 6236 ? S 12:48 0:00 ksensors -caption KSensors -icon ksensors.png -miniicon ksensors.png
vincent 2253 0.0 0.6 16116 6760 ? S 12:49 0:00 kdeinit: kwrited
vincent 2258 0.0 0.0 1192 312 pts/1 S 12:49 0:00 /bin/cat
vincent 2497 0.0 0.8 16188 8844 ? S 12:51 0:05 ksysguard
vincent 2498 0.0 0.0 1316 576 ? S 12:51 0:04 ksysguardd
vincent 2525 0.0 1.6 24188 16564 ? S 12:51 0:03 kdeinit: konqueror --silent
vincent 2535 0.0 0.6 16008 6652 ? S 12:51 0:00 kdeinit: kcookiejar
vincent 2538 0.0 0.3 10496 3176 ? S 12:51 0:00 kdesud
root 2617 0.0 0.1 4252 1976 ? S 12:51 0:00 /usr/sbin/smbd -D
root 4216 0.0 0.2 4512 2320 ? S 13:04 0:03 /usr/sbin/smbd -D
root 6999 0.0 0.1 5764 1808 ? S 13:24 0:00 sshd: vincent [priv]
vincent 7016 0.0 0.1 5784 1880 ? S 13:24 0:00 sshd: vincent@pts/2
vincent 7017 0.0 0.1 2276 1336 pts/2 S 13:24 0:00 -bash
root 7053 0.0 0.1 2276 1336 pts/2 S 13:25 0:00 -su
root 7188 0.0 0.1 5764 1808 ? S 13:25 0:00 sshd: vincent [priv]
vincent 7210 0.0 0.1 5892 1948 ? S 13:25 0:00 sshd: vincent@pts/3
vincent 7211 0.0 0.1 2284 1376 pts/3 S 13:25 0:00 -bash
root 7273 0.0 0.1 2288 1388 pts/3 S 13:25 0:00 -su
root 17039 0.0 0.1 5764 1808 ? S 13:48 0:00 sshd: vincent [priv]
vincent 17061 0.0 0.1 5856 1908 ? S 13:48 0:00 sshd: vincent@pts/4
vincent 17062 0.0 0.1 2284 1380 pts/4 S 13:48 0:00 -bash
postfix 29791 0.0 0.0 2352 924 ? S 14:17 0:00 pickup -l -t fifo -u -c
root 29818 0.0 0.1 2284 1376 pts/4 S 14:17 0:00 -su
root 29822 0.0 0.0 1736 492 pts/4 S 14:17 0:00 tail -f /var/log/messages
vincent 30131 0.0 0.5 15588 5732 ? S 14:20 0:00 kdeinit: kio_http http /tmp/ksocket-vincent/klauncher89Rdtc.slave-socket /tmp/ksocket-vincent/konquerorsIO5nb.slave-socket
vincent 30677 0.0 0.5 15588 5708 ? S 14:24 0:00 kdeinit: kio_http http /tmp/ksocket-vincent/klauncher89Rdtc.slave-socket /tmp/ksocket-vincent/konqueror5UQgBb.slave-socket
vincent 31205 0.1 0.7 16844 8240 ? S 14:28 0:00 kdeinit: konsole -icon konsole -miniicon konsole
vincent 31206 0.0 0.1 2280 1372 pts/5 S 14:28 0:00 /bin/bash
root 31236 0.0 0.1 2276 1336 pts/5 S 14:28 0:00 -su
root 31622 0.0 0.1 5764 1808 ? S 14:29 0:00 sshd: vincent [priv]
vincent 31672 0.0 0.1 5784 1880 ? S 14:29 0:00 sshd: vincent@pts/6
vincent 31673 0.0 0.1 2276 1336 pts/6 S 14:29 0:00 -bash
root 31705 0.0 0.1 2276 1336 pts/6 S 14:29 0:00 -su
callaer 31787 0.0 0.1 2244 1340 pts/6 S 14:29 0:00 -su
root 32418 0.0 0.0 1260 496 pts/2 S 14:30 0:00 vmstat -n 1
callaer 32441 4.7 0.0 1228 372 pts/6 D 14:30 0:02 cp -r /mnt/hercules1/backup_dirk/ ./25_02_2003/
root 462 0.0 0.0 1708 464 pts/3 S 14:31 0:00 sleep .5
root 466 0.0 0.1 2288 1388 pts/0 R 14:31 0:00 -su
Stuck in a D state:
root 122 0.0 0.0 0 0 ? DW 12:37 0:00 [kjournald]
root 325 0.0 0.1 8380 1072 ? D 12:37 0:02 /usr/sbin/3dmd
callaer 32441 4.7 0.0 1228 372 pts/6 D 14:30 0:02 cp -r /mnt/hercules1/backup_dirk/ ./25_02_2003
The first is the ext3 journalling daemon ?
The second is the 3ware raid management daemon.
The third is the copying process from ide (/mnt/hercules1) to the array (./)
[The output of vmstat -n 1]
Started right before the copying kicks in:
kalimero:~# vmstat -n 1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 269524 46788 620124 0 0 31 65 81 164 2 1 97 0
0 0 0 269516 46788 620124 0 0 0 392 123 414 3 2 95 0
0 0 0 269516 46788 620124 0 0 0 0 132 454 0 1 98 0
FROM HERE ON THE CP KICKS IN
1 0 0 275520 46960 613908 0 0 4 0 127 504 4 43 53 0
0 1 0 260100 47084 627812 0 0 3856 0 173 679 0 46 54 0
0 1 0 204908 47220 681028 0 0 26700 0 562 1212 5 16 79 0
0 2 0 200952 47240 684824 0 0 1800 59528 350 538 0 5 94 0
0 2 0 200944 47248 684824 0 0 8 24336 293 447 3 2 95 0
1 2 0 200944 47248 684824 0 0 0 20372 281 403 0 2 97 0
0 3 0 200936 47252 684824 0 0 4 21484 445 1051 6 3 91 0
0 3 0 200904 47264 684824 0 0 4 23664 288 421 1 1 98 0
1 3 0 200892 47264 684824 0 0 0 12416 218 416 3 1 96 0
0 3 0 200880 47264 684824 0 0 0 0 114 398 0 1 98 0
0 3 0 200868 47264 684824 0 0 0 0 111 395 3 1 96 0
0 3 0 200856 47264 684824 0 0 0 0 114 376 0 1 98 0
0 3 0 200944 47264 684824 0 0 0 560 134 483 2 2 95 0
2 3 0 200916 47264 684824 0 0 0 0 124 454 0 1 98 0
0 3 0 200916 47264 684824 0 0 0 0 109 403 4 2 94 0
0 3 0 200896 47264 684824 0 0 0 0 119 466 1 1 98 0
0 3 0 200868 47264 684824 0 0 0 0 125 490 3 1 95 0
0 3 0 200844 47264 684824 0 0 0 388 127 383 1 1 98 0
1 3 0 200832 47264 684824 0 0 0 0 125 506 3 1 95 0
0 3 0 200832 47264 684824 0 0 0 0 108 391 0 1 99 0
0 3 0 200944 47264 684824 0 0 0 0 124 515 3 2 95 0
1 3 0 200940 47264 684824 0 0 0 0 132 459 1 1 98 0
0 3 0 200940 47264 684824 0 0 0 388 122 508 4 1 95 0
0 3 0 200916 47264 684824 0 0 0 0 122 476 2 0 98 0
0 3 0 200900 47264 684824 0 0 0 0 120 427 3 2 95 0
0 3 0 200916 47264 684824 0 0 0 0 110 381 0 2 98 0
3 3 0 200892 47264 684824 0 0 0 0 108 382 3 0 96 0
0 3 0 200892 47264 684824 0 0 0 384 124 384 0 2 98 0
1 3 0 200892 47264 684824 0 0 0 0 118 408 3 2 94 0
0 3 0 200880 47264 684824 0 0 0 0 112 377 0 2 98 0
0 3 0 200884 47264 684824 0 0 0 0 292 1184 3 1 95 0
1 3 0 200880 47264 684824 0 0 0 0 119 459 1 1 98 0
0 4 0 200856 47264 684824 0 0 0 1864 133 502 3 2 95 0
1 4 0 200856 47264 684824 0 0 0 0 110 424 1 1 98 0
0 4 0 200860 47264 684824 0 0 0 0 133 554 3 1 96 0
0 4 0 200856 47264 684824 0 0 0 0 133 566 2 0 98 0
0 4 0 200856 47264 684824 0 0 0 0 238 1050 3 2 95 0
0 4 0 200856 47264 684824 0 0 0 388 241 1065 0 2 98 0
3 4 0 200852 47264 684824 0 0 0 0 413 1495 3 39 58 0
2 4 0 200852 47264 684824 0 0 0 0 406 1368 1 15 83 0
1 4 0 200852 47264 684824 0 0 0 0 180 670 3 3 94 0
1 4 0 200852 47264 684824 0 0 0 0 123 555 1 1 98 0
Notice that no blocks get read in no more and a very intermittent write out.
[Stack trace]
Finally the stack trace (which I hope is close enough to the hanging,
else I might have to redo it :/)
Jul 7 14:28:30 kalimero kernel: SysRq : Show State
Jul 7 14:28:30 kalimero kernel:
Jul 7 14:28:30 kalimero kernel: free sibling
Jul 7 14:28:30 kalimero kernel: task PC stack pid father child younger older
Jul 7 14:28:30 kalimero kernel: init S C02DF7B8 4124 1 0 2538 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [schedule_timeout+88/176] [__pollwait+62/176] [process_timeout+0/32] [do_select+276/544] [sys_select+794/1200]
Jul 7 14:28:30 kalimero kernel: [path_release+22/64] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: keventd R F6AF136C 5844 2 1 3 (L-TLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [console_callback+159/192] [__run_task_queue+173/192] [context_thread+417/672] [context_thread+0/672] [arch_kernel_thread+46/64]
Jul 7 14:28:30 kalimero kernel: [context_thread+0/672]
Jul 7 14:28:30 kalimero kernel: ksoftirqd_CPU S C1C2FF8C 5860 3 1 4 2 (L-TLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [bh_action+132/224] [tasklet_hi_action+99/160] [ksoftirqd+175/256] [arch_kernel_thread+46/64] [ksoftirqd+0/256] Jul 7 14:28:30 kalimero kernel: ksoftirqd_CPU S F72C0000 5876 4 1 5 3 (L-TLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [bh_action+132/224] [tasklet_hi_action+99/160] [ksoftirqd+175/256] [arch_kernel_thread+46/64] [ksoftirqd+0/256] Jul 7 14:28:30 kalimero kernel: kswapd S 00000000 6332 5 1 6 4 (L-TLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [kswapd+134/192] [kswapd+0/192] [arch_kernel_thread+46/64] [kswapd+0/192]
Jul 7 14:28:30 kalimero kernel: bdflush S C02DBE40 6284 6 1 7 5 (L-TLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [interruptible_sleep_on+143/272] [bdflush+335/352] [arch_kernel_thread+46/64] [bdflush+0/352]
Jul 7 14:28:30 kalimero kernel: kupdated S F7547614 5512 7 1 8 6 (L-TLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [log_start_commit+216/256] [ext3_write_super+62/64] [schedule_timeout+88/176] [process_timeout+0/32] [kupdate+241/480]
Jul 7 14:28:30 kalimero kernel: [kupdate+0/480] [arch_kernel_thread+46/64] [kupdate+0/480]
Jul 7 14:28:30 kalimero kernel: scsi_eh_0 S 00000000 6080 8 1 9 7 (L-TLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__down_interruptible+221/416] [__down_failed_interruptible+10/16] [.text.lock.scsi_error+229/290] [arch_kernel_thread+46/64] [scsi_error_handler+0/608]
Jul 7 14:28:30 kalimero kernel: kjournald S F7ED6000 4492 9 1 24 8 (L-TLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [interruptible_sleep_on+143/272] [kjournald+444/720] [ret_from_fork+6/32] [commit_timeout+0/16] [arch_kernel_thread+46/64]
Jul 7 14:28:30 kalimero kernel: [kjournald+0/720]
Jul 7 14:28:30 kalimero kernel: devfsd S 380C5067 5000 24 1 122 9 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [devfsd_read+224/1040] [free_uid+32/128] [release_task+324/352] [sys_read+150/448] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: kjournald S E7792000 5092 122 1 123 24 (L-TLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [interruptible_sleep_on+143/272] [printk+389/512] [kjournald+444/720] [ret_from_fork+6/32] [commit_timeout+0/16]Jul 7 14:28:30 kalimero kernel: [arch_kernel_thread+46/64] [kjournald+0/720]
Jul 7 14:28:30 kalimero kernel: kjournald S F7C12000 4 123 1 124 122 (L-TLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [interruptible_sleep_on+143/272] [kjournald+444/720] [ret_from_fork+6/32] [commit_timeout+0/16] [arch_kernel_thread+46/64]
Jul 7 14:28:30 kalimero kernel: [kjournald+0/720]
Jul 7 14:28:30 kalimero kernel: kjournald S F7C12000 4 124 1 125 123 (L-TLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [release_console_sem+285/288] [interruptible_sleep_on+143/272] [printk+389/512] [kjournald+444/720] [ret_from_fork+6/32]
Jul 7 14:28:30 kalimero kernel: [commit_timeout+0/16] [arch_kernel_thread+46/64] [kjournald+0/720]
Jul 7 14:28:30 kalimero kernel: kjournald S 00000282 4 125 1 126 124 (L-TLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [release_console_sem+285/288] [interruptible_sleep_on+143/272] [printk+389/512] [kjournald+444/720] [ret_from_fork+6/32]
Jul 7 14:28:30 kalimero kernel: [kjournald+0/720] [commit_timeout+0/16] [arch_kernel_thread+46/64] [kjournald+0/720]
Jul 7 14:28:30 kalimero kernel: kjournald S 00000282 4 126 1 181 125 (L-TLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [release_console_sem+285/288] [interruptible_sleep_on+143/272] [printk+389/512] [kjournald+444/720] [ret_from_fork+6/32]
Jul 7 14:28:30 kalimero kernel: [kjournald+0/720] [commit_timeout+0/16] [arch_kernel_thread+46/64] [kjournald+0/720]
Jul 7 14:28:30 kalimero kernel: dhclient S F7423F2C 0 181 1 185 126 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [__get_free_pages+32/48] [schedule_timeout+88/176] [datagram_poll+46/219] [process_timeout+0/32]
Jul 7 14:28:30 kalimero kernel: [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: portmap S 00000246 0 185 1 287 181 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [datagram_poll+46/219] [schedule_timeout+165/176] [sock_poll+44/64] [do_pollfd+92/160] [do_poll+158/240]
Jul 7 14:28:30 kalimero kernel: [sys_poll+355/720] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: syslogd S F7373F2C 0 287 1 290 185 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [__get_free_pages+32/48] [schedule_timeout+165/176] [datagram_poll+46/219] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1952480/128]
Jul 7 14:28:30 kalimero kernel: [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200] [do_setitimer+215/256] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: klogd R 00000000 0 290 1 321 287 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [sock_sendmsg+115/176] [do_syslog+520/1664] [sock_write+153/176] [sys_read+150/448] [sys_time+29/96]
Jul 7 14:28:30 kalimero kernel: [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: 3dmd S BFFFF9CC 2384 321 1 322 394 290 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [schedule_timeout+165/176] [sys_rt_sigaction+150/176] [sys_rt_sigprocmask+81/576] [sys_nanosleep+203/336] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: 3dmd S C02F4000 2384 322 321 327 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__pollwait+62/176] [schedule_timeout+88/176] [process_timeout+0/32] [do_poll+158/240] [sys_poll+355/720]
Jul 7 14:28:30 kalimero kernel: [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: 3dmd S F72C3E30 2388 323 322 325 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [ip_queue_xmit2+197/592] [schedule_timeout+165/176] [tcp_v4_send_check+70/208] [wait_for_connect+576/624] [tcp_transmit_skb+676/1072]
Jul 7 14:28:30 kalimero kernel: [tcp_accept+490/848] [alloc_inode+194/336] [inet_accept+48/640] [sock_alloc+25/208] [sys_accept+111/288] [destroy_inode+76/80]
Jul 7 14:28:30 kalimero kernel: [iput+265/944] [dput+35/496] [fput+300/384] [sys_socketcall+198/576] [sys_ioctl+225/657] [sys_close+120/144]
Jul 7 14:28:30 kalimero kernel: [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: 3dmd S 40000000 2388 325 322 327 323 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [unmap_fixup+533/560] [schedule_timeout+88/176] [process_timeout+0/32] [sys_nanosleep+203/336] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: 3dmd S 00000246 2388 327 322 325 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [datagram_poll+46/219] [schedule_timeout+165/176] [sock_poll+44/64] [do_pollfd+92/160] [do_poll+158/240]
Jul 7 14:28:30 kalimero kernel: [sys_poll+355/720] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: automount S F7C05A9C 5168 394 1 410 321 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__get_free_pages+32/48] [__pollwait+62/176] [schedule_timeout+165/176] [do_pollfd+92/160] [do_poll+158/240]
Jul 7 14:28:30 kalimero kernel: [sys_poll+355/720] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: automount S F7960B34 2384 410 1 426 394 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__get_free_pages+32/48] [__pollwait+62/176] [schedule_timeout+165/176] [do_pollfd+92/160] [do_poll+158/240]
Jul 7 14:28:30 kalimero kernel: [sys_poll+355/720] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: cupsd S C02DF7B8 0 426 1 601 410 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [schedule_timeout+88/176] [process_timeout+0/32] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: inetd S C02DF7B8 0 601 1 605 426 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__pollwait+62/176] [schedule_timeout+165/176] [datagram_poll+46/219] [sock_poll+44/64] [do_select+276/544]
Jul 7 14:28:30 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: lpd S F6F21F2C 0 605 1 702 601 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [schedule_timeout+165/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1957506/128] [sock_poll+44/64] [do_select+276/544]
Jul 7 14:28:30 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: master S C02DF7B8 4612 702 1 29791 708 605 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [schedule_timeout+88/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1957506/128] [process_timeout+0/32] [sock_poll+44/64] [do_select+276/544]
Jul 7 14:28:30 kalimero kernel: [sys_select+794/1200] [do_setitimer+215/256] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: nqmgr S F6CA3F2C 4488 706 702 29791 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [__get_free_pages+32/48] [schedule_timeout+88/176] [process_timeout+0/32] [do_select+276/544]
Jul 7 14:28:30 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: nmbd S C02F4000 0 708 1 709 711 702 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [schedule_timeout+88/176] [process_timeout+0/32] [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200]
Jul 7 14:28:30 kalimero kernel: [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: nmbd S F6C84000 4456 709 708 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [pipe_wait+122/176] [pipe_read+178/496] [sys_read+150/448] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: smbd S C02DF7B8 0 711 1 4216 717 708 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__pollwait+62/176] [schedule_timeout+165/176] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: sshd S C02DF7B8 0 717 1 17039 731 711 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__pollwait+62/176] [schedule_timeout+165/176] [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200]
Jul 7 14:28:30 kalimero kernel: [sys_close+120/144] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: rpc.statd S F6C1FF2C 0 731 1 734 717 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [schedule_timeout+165/176] [datagram_poll+46/219] [sock_poll+44/64] [do_select+276/544]
Jul 7 14:28:30 kalimero kernel: [sys_select+794/1200] [sys_close+120/144] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: ntpd S F6C19F2C 0 734 1 737 731 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [__get_free_pages+32/48] [schedule_timeout+165/176] [datagram_poll+46/219] [sock_poll+44/64]
Jul 7 14:28:30 kalimero kernel: [do_select+276/544] [sys_select+794/1200] [restore_sigcontext+296/320] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: rpc.nfsd S 00000246 0 737 1 739 734 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [datagram_poll+46/219] [schedule_timeout+165/176] [sock_poll+44/64] [do_pollfd+92/160] [do_poll+158/240]
Jul 7 14:28:30 kalimero kernel: [sys_poll+355/720] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: rpc.mountd S 00000246 0 739 1 746 737 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [datagram_poll+46/219] [schedule_timeout+165/176] [sock_poll+44/64] [do_pollfd+92/160] [do_poll+158/240]
Jul 7 14:28:30 kalimero kernel: [sys_poll+355/720] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: atd S F6BE3F1C 0 746 1 749 739 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [schedule_timeout+88/176] [process_timeout+0/32] [sys_rt_sigprocmask+81/576] [sys_nanosleep+203/336] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: cron S F6BCFF1C 0 749 1 754 746 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [schedule_timeout+88/176] [process_timeout+0/32] [sys_rt_sigprocmask+81/576] [sys_nanosleep+203/336] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: kdm S F6B95F2C 4712 754 1 769 761 749 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [schedule_timeout+165/176] [datagram_poll+46/219] [sock_poll+44/64] [do_select+276/544]
Jul 7 14:28:30 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: getty S C01F369B 0 761 1 762 754 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [clear_selection+27/96] [set_cursor+115/144] [con_flush_chars+132/224] [schedule_timeout+165/176] [write_chan+335/512]
Jul 7 14:28:30 kalimero kernel: [read_chan+579/1616] [tty_read+352/416] [sys_read+150/448] [sys_newuname+57/96] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: getty S F751E21C 4720 762 1 763 761 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [con_flush_chars+132/224] [schedule_timeout+165/176] [write_chan+335/512] [read_chan+579/1616] [tty_read+352/416]
Jul 7 14:28:30 kalimero kernel: [sys_read+150/448] [sys_newuname+57/96] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: getty S F750D21C 4964 763 1 764 762 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [con_flush_chars+132/224] [schedule_timeout+165/176] [write_chan+335/512] [read_chan+579/1616] [tty_read+352/416]
Jul 7 14:28:30 kalimero kernel: [sys_read+150/448] [sys_newuname+57/96] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: getty S F739521C 4964 764 1 765 763 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [con_flush_chars+132/224] [schedule_timeout+165/176] [write_chan+335/512] [read_chan+579/1616] [tty_read+352/416]
Jul 7 14:28:30 kalimero kernel: [sys_read+150/448] [sys_newuname+57/96] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: getty S F751021C 4612 765 1 766 764 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [con_flush_chars+132/224] [schedule_timeout+165/176] [write_chan+335/512] [read_chan+579/1616] [tty_read+352/416]
Jul 7 14:28:30 kalimero kernel: [sys_read+150/448] [sys_newuname+57/96] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: getty S F6B7121C 4964 766 1 771 765 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [con_flush_chars+132/224] [schedule_timeout+165/176] [write_chan+335/512] [read_chan+579/1616] [tty_read+352/416]
Jul 7 14:28:30 kalimero kernel: [sys_read+150/448] [sys_newuname+57/96] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: XFree86 S C02DF7B8 0 767 754 769 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [schedule_timeout+88/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1957506/128] [process_timeout+0/32] [sock_poll+44/64] [do_select+276/544]
Jul 7 14:28:30 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: kdm S 00000282 0 769 754 2149 767 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [do_getitimer+92/176] [sys_wait4+303/1040] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: miniserv.pl S C02DF7B8 0 771 1 772 766 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [schedule_timeout+88/176] [datagram_poll+46/219] [process_timeout+0/32] [sock_poll+44/64] [do_select+276/544]
Jul 7 14:28:30 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: miniserv.pl S C02DF7B8 5112 772 1 2184 771 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [schedule_timeout+88/176] [datagram_poll+46/219] [process_timeout+0/32] [sock_poll+44/64] [do_select+276/544]
Jul 7 14:28:30 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: sshd S 00000001 2384 965 717 967 6999 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [journal_alloc_journal_head+27/128] [schedule_timeout+165/176] [journal_dirty_data+522/640] [balance_dirty_state+15/80] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1959618/128]
Jul 7 14:28:30 kalimero kernel: [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1958327/128] [sock_recvmsg+79/240] [get_empty_filp+133/544] [sock_read+147/160] [sys_read+150/448] [sys_close+120/144]
Jul 7 14:28:30 kalimero kernel: [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: sshd S C02DF7B8 0 967 965 968 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [schedule_timeout+165/176] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: bash S F6614000 4964 968 967 1485 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [tty_check_change+66/160] [sys_wait4+303/1040] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: bash S 0000000C 4612 1485 968 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [do_wp_page+1025/1152] [__global_cli+100/112] [schedule_timeout+165/176] [write_chan+335/512] [read_chan+579/1616]
Jul 7 14:28:30 kalimero kernel: [tty_read+352/416] [sys_read+150/448] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: x-session-man S 000008A2 0 2149 769 2210 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [sys_rt_sigaction+150/176] [sys_wait4+303/1040] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: ssh-agent S F4A93F2C 0 2174 2149 2210 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [__get_free_pages+32/48] [schedule_timeout+165/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1957506/128] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1952576/128]
Jul 7 14:28:30 kalimero kernel: [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200] [restore_sigcontext+296/320] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: kdeinit S C02DF7B8 0 2181 1 31205 2215 2209 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [schedule_timeout+165/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1957506/128] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1952576/128] [sock_poll+44/64] [do_select+276/544]
Jul 7 14:28:30 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: kdeinit S F48C4000 0 2184 1 2187 772 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [schedule_timeout+165/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1957506/128] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1952576/128] [sock_poll+44/64] [do_select+276/544]
Jul 7 14:28:30 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: kdeinit S C02DF7B8 0 2187 1 2190 2184 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [schedule_timeout+165/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1957506/128] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1952576/128] [sock_poll+44/64] [do_select+276/544]
Jul 7 14:28:30 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: kdeinit S F49FBF2C 0 2190 1 2209 2187 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [__get_free_pages+32/48] [schedule_timeout+88/176] [process_timeout+0/32] [do_select+276/544]
Jul 7 14:28:30 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: artsd S C02DF7B8 0 2198 2181 2211 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [schedule_timeout+88/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1957506/128] [process_timeout+0/32] [sock_poll+44/64] [do_select+276/544]
Jul 7 14:28:30 kalimero kernel: [sys_select+794/1200] [restore_sigcontext+296/320] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: kdeinit S F48C5F2C 0 2209 1 2181 2190 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [__get_free_pages+32/48] [schedule_timeout+165/176] [do_select+276/544] [sys_select+794/1200]
Jul 7 14:28:30 kalimero kernel: [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: ksmserver S F4A83F2C 4272 2210 2149 2174 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [__get_free_pages+32/48] [schedule_timeout+165/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1957506/128] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1952576/128]
Jul 7 14:28:30 kalimero kernel: [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: kdeinit S C02F4000 0 2211 2181 2497 2198 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [__get_free_pages+32/48] [schedule_timeout+165/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1952576/128] [sock_poll+44/64]
Jul 7 14:28:30 kalimero kernel: [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: kdeinit S F4869F2C 0 2215 1 2217 2181 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [__get_free_pages+32/48] [schedule_timeout+88/176] [process_timeout+0/32] [do_select+276/544]
Jul 7 14:28:30 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: kdeinit S C1C12000 4996 2217 1 2247 2215 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [__get_free_pages+32/48] [schedule_timeout+88/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1957506/128] [process_timeout+0/32]
Jul 7 14:28:30 kalimero kernel: [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: kdeinit S 00000282 0 2247 1 2252 2217 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [schedule_timeout+88/176] [process_timeout+0/32] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: ksensors S F3CD9F2C 0 2252 1 2253 2247 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [__get_free_pages+32/48] [schedule_timeout+88/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1957506/128] [process_timeout+0/32]
Jul 7 14:28:30 kalimero kernel: [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: kdeinit S F3C25F2C 0 2253 1 2258 2535 2252 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [schedule_timeout+165/176] [do_select+276/544] [sys_select+794/1200] [system_call+51/56] Jul 7 14:28:30 kalimero kernel: cat S C013278C 4964 2258 2253 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [do_no_page+140/864] [schedule_timeout+165/176] [read_chan+579/1616] [cp_new_stat64+227/272] [tty_read+352/416] Jul 7 14:28:30 kalimero kernel: [sys_read+150/448] [sys_fstat64+72/128] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: ksysguard S F3A11F2C 1360 2497 2181 2498 2525 2211 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [__get_free_pages+32/48] [schedule_timeout+88/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1957506/128] [process_timeout+0/32]
Jul 7 14:28:30 kalimero kernel: [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: ksysguardd S F3B29F2C 4964 2498 2497 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [__get_free_pages+32/48] [schedule_timeout+88/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1957506/128] [process_timeout+0/32]
Jul 7 14:28:30 kalimero kernel: [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: kdeinit S F477DF2C 0 2525 2181 30131 2497 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [__get_free_pages+32/48] [schedule_timeout+88/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1957506/128] [process_timeout+0/32]
Jul 7 14:28:30 kalimero kernel: [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: kdeinit S F2E05F2C 272 2535 1 2538 2253 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [__get_free_pages+32/48] [schedule_timeout+165/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1957506/128] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1952576/128]
Jul 7 14:28:30 kalimero kernel: [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: kdesud S F2AEDF2C 5360 2538 1 2535 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [__get_free_pages+32/48] [schedule_timeout+88/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1957506/128] [process_timeout+0/32]
Jul 7 14:28:30 kalimero kernel: [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: smbd S F29CFF2C 0 2617 711 4216 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [schedule_timeout+88/176] [process_timeout+0/32] [do_select+276/544] [sys_select+794/1200]
Jul 7 14:28:30 kalimero kernel: [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: smbd S C02DF7B8 1360 4216 711 2617 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [schedule_timeout+88/176] [process_timeout+0/32] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: sshd S 00000001 672 6999 717 7016 7188 965 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [journal_alloc_journal_head+27/128] [schedule_timeout+165/176] [journal_dirty_data+522/640] [balance_dirty_state+15/80] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1959618/128]
Jul 7 14:28:30 kalimero kernel: [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1958327/128] [sock_recvmsg+79/240] [get_empty_filp+133/544] [sock_read+147/160] [sys_read+150/448] [sys_close+120/144]
Jul 7 14:28:30 kalimero kernel: [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: sshd S C02DF7B8 4992 7016 6999 7017 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [schedule_timeout+165/176] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: bash S E82E4000 36 7017 7016 7053 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [tty_check_change+66/160] [sys_wait4+303/1040] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: bash S 0000000C 2432 7053 7017 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [do_wp_page+1025/1152] [__global_cli+100/112] [schedule_timeout+165/176] [write_chan+335/512] [read_chan+579/1616]
Jul 7 14:28:30 kalimero kernel: [tty_read+352/416] [sys_read+150/448] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: sshd S 00000001 2384 7188 717 7210 17039 6999 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [journal_alloc_journal_head+27/128] [schedule_timeout+165/176] [journal_dirty_data+522/640] [balance_dirty_state+15/80] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1959618/128]
Jul 7 14:28:30 kalimero kernel: [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1958327/128] [sock_recvmsg+79/240] [get_empty_filp+133/544] [sock_read+147/160] [sys_read+150/448] [sys_close+120/144]
Jul 7 14:28:30 kalimero kernel: [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: sshd S C02DF7B8 2376 7210 7188 7211 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [schedule_timeout+165/176] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: bash S E8228000 2468 7211 7210 7273 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [tty_check_change+66/160] [sys_wait4+303/1040] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: bash S 0000000C 16 7273 7211 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [do_wp_page+1025/1152] [__global_cli+100/112] [schedule_timeout+165/176] [write_chan+335/512] [read_chan+579/1616]
Jul 7 14:28:30 kalimero kernel: [tty_read+352/416] [sys_read+150/448] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: sshd S 00000001 0 17039 717 17061 7188 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [journal_alloc_journal_head+27/128] [schedule_timeout+165/176] [journal_dirty_data+522/640] [balance_dirty_state+15/80] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1959618/128]
Jul 7 14:28:30 kalimero kernel: [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1958327/128] [sock_recvmsg+79/240] [get_empty_filp+133/544] [sock_read+147/160] [sys_read+150/448] [sys_close+120/144]
Jul 7 14:28:30 kalimero kernel: [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: sshd S C02DF7B8 60 17061 17039 17062 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [schedule_timeout+165/176] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: bash S E83D6000 0 17062 17061 29818 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [tty_check_change+66/160] [sys_wait4+303/1040] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: pickup S F6EB2000 0 29791 702 706 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [__get_free_pages+32/48] [schedule_timeout+88/176] [process_timeout+0/32] [sock_poll+44/64]
Jul 7 14:28:30 kalimero kernel: [do_select+276/544] [sys_select+794/1200] [do_setitimer+215/256] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: bash S E7A9C000 0 29818 17062 29822 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [tty_check_change+66/160] [sys_wait4+303/1040] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: tail S C1C12000 0 29822 29818 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [schedule_timeout+88/176] [process_timeout+0/32] [sys_nanosleep+203/336] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: kdeinit S E7EEDF2C 2416 30131 2181 30677 2525 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [__get_free_pages+32/48] [schedule_timeout+88/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1957506/128] [process_timeout+0/32]
Jul 7 14:28:30 kalimero kernel: [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: kdeinit S E7B7BF2C 8 30677 2181 31205 30131 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [__get_free_pages+32/48] [schedule_timeout+88/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.21/kernel/net/packet/a+-1957506/128] [process_timeout+0/32]
Jul 7 14:28:30 kalimero kernel: [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: kdeinit S E7793F2C 0 31205 2181 31206 30677 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [__alloc_pages+75/400] [schedule_timeout+88/176] [process_timeout+0/32] [do_select+276/544] [sys_select+794/1200]
Jul 7 14:28:30 kalimero kernel: [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: bash S E7B20000 0 31206 31205 31236 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [tty_check_change+66/160] [sys_wait4+303/1040] [system_call+51/56]
Jul 7 14:28:30 kalimero kernel: bash S C013278C 0 31236 31206 (NOTLB)
Jul 7 14:28:30 kalimero kernel: Call Trace: [do_no_page+140/864] [__global_cli+100/112] [schedule_timeout+165/176] [write_chan+335/512] [read_chan+579/1616]Jul 7 14:28:30 kalimero kernel: [tty_read+352/416] [sys_read+150/448] [system_call+51/56]
Jul 7 14:31:00 kalimero kernel: 3w-xxxx: scsi0: Unit #0: Command (f7c48400) timed out, resetting card.
Hm, still 90 seconds between the trace and the hang :( No processes seem to be stuck yet.
I'll redo it, to provide a more useful report.
I wonder if anyone can distillate something from these data already ?
regards,
Vincent
Ok, I tried another time, and this time I have a trace really up close
and personal with the hangup.
Kernel is 2.4.19 (cause Joe has the same hardware running fine with
2.4.19), plain vanilla Linus kernel, using the same config as the 2.4.21
kernel that crashed.
These are the three stack traces leading to the hang:
All the stack traces I got before the crash is some 270K in text,
I added it gzipped in dmesg.gz, maybe it could provide more info.
Jul 7 17:52:52 kalimero kernel: SysRq : Show State
Jul 7 17:52:52 kalimero kernel:
Jul 7 17:52:52 kalimero kernel: free sibling
Jul 7 17:52:52 kalimero kernel: task PC stack pid father child younger older
Jul 7 17:52:52 kalimero kernel: init S C02D0078 4120 1 0 1274 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [schedule_timeout+88/176] [__pollwait+62/176] [process_timeout+0/192] [do_select+276/544] [sys_select+794/1200] Jul 7 17:52:52 kalimero kernel: [path_release+22/64] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: keventd S 0006FA72 5216 2 1 3 (L-TLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [rest_init+0/144] [context_thread+380/672] [context_thread+0/672] [rest_init+0/144] [kernel_thread+46/64]
Jul 7 17:52:52 kalimero kernel: [context_thread+0/672]
Jul 7 17:52:52 kalimero kernel: ksoftirqd_CPU S F7EEFF8C 5736 3 1 4 2 (L-TLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [bh_action+132/224] [tasklet_hi_action+99/160] [ksoftirqd+175/256] [kernel_thread+46/64] [ksoftirqd+0/256]
Jul 7 17:52:52 kalimero kernel: ksoftirqd_CPU S F6F7A000 5736 4 1 5 3 (L-TLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [bh_action+132/224] [tasklet_action+99/160] [ksoftirqd+175/256] [kernel_thread+46/64] [ksoftirqd+0/256]
Jul 7 17:52:52 kalimero kernel: kswapd S C013F199 5204 5 1 6 4 (L-TLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [try_to_free_pages+57/96] [kswapd+134/192] [kswapd+0/192] [rest_init+0/144] [kernel_thread+46/64]
Jul 7 17:52:52 kalimero kernel: [kswapd+0/192]
Jul 7 17:52:52 kalimero kernel: bdflush S 0098993E 6284 6 1 7 5 (L-TLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [interruptible_sleep_on+143/256] [bdflush+335/352] [rest_init+0/144] [rest_init+0/144] [kernel_thread+46/64]
Jul 7 17:52:52 kalimero kernel: [bdflush+0/352]
Jul 7 17:52:52 kalimero kernel: kupdated D 00000001 5204 7 1 8 6 (L-TLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [__down+192/352] [log_start_commit+216/256] [__down_failed+11/20] [.text.lock.super+279/550] [sync_old_buffers+94/336]
Jul 7 17:52:52 kalimero kernel: [kupdate+418/480] [kupdate+0/480] [rest_init+0/144] [rest_init+0/144] [kernel_thread+46/64] [kupdate+0/480]
Jul 7 17:52:52 kalimero kernel: scsi_eh_0 S 00000000 6080 8 1 9 7 (L-TLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [vsnprintf+500/1056] [__down_interruptible+221/416] [__down_failed_interruptible+10/16] [.text.lock.scsi_error+229/290] [kernel_thread+46/64]
Jul 7 17:52:52 kalimero kernel: [scsi_error_handler+0/608]
Jul 7 17:52:52 kalimero kernel: kjournald S C94C1C28 4664 9 1 24 8 (L-TLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [interruptible_sleep_on+143/256] [kjournald+444/704] [ret_from_fork+6/32] [commit_timeout+0/16] [kernel_thread+46/64]
Jul 7 17:52:52 kalimero kernel: [kjournald+0/704]
Jul 7 17:52:52 kalimero kernel: devfsd S 00000082 5452 24 1 122 9 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [devfsd_read+223/1040] [free_uid+32/128] [release_task+324/352] [sys_read+150/448] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: kjournald D C02E4000 4624 122 1 123 24 (L-TLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [generic_make_request+222/320] [submit_bh+94/128] [__wait_on_buffer+94/144] [journal_commit_transaction+1145/5360] [bh_action+132/224]
Jul 7 17:52:52 kalimero kernel: [tasklet_hi_action+99/160] [kjournald+490/704] [ret_from_fork+6/32] [commit_timeout+0/16] [kernel_thread+46/64] [kjournald+0/704]
Jul 7 17:52:52 kalimero kernel: kjournald S 00002F99 4 123 1 124 122 (L-TLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [release_console_sem+285/288] [interruptible_sleep_on+143/256] [kjournald+444/704] [ret_from_fork+6/32] [kjournald+0/704]
Jul 7 17:52:52 kalimero kernel: [commit_timeout+0/16] [kernel_thread+46/64] [kjournald+0/704]
Jul 7 17:52:52 kalimero kernel: kjournald S 00003044 484 124 1 125 123 (L-TLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [release_console_sem+285/288] [interruptible_sleep_on+143/256] [kjournald+444/704] [ret_from_fork+6/32] [kjournald+0/704]
Jul 7 17:52:52 kalimero kernel: [commit_timeout+0/16] [kernel_thread+46/64] [kjournald+0/704]
Jul 7 17:52:52 kalimero kernel: kjournald S 000030EF 4 125 1 126 124 (L-TLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [release_console_sem+285/288] [interruptible_sleep_on+143/256] [kjournald+444/704] [ret_from_fork+6/32] [kjournald+0/704]
Jul 7 17:52:52 kalimero kernel: [commit_timeout+0/16] [kernel_thread+46/64] [kjournald+0/704]
Jul 7 17:52:52 kalimero kernel: kjournald S F7EB8000 4 126 1 176 125 (L-TLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [release_console_sem+285/288] [interruptible_sleep_on+143/256] [kjournald+444/704] [ret_from_fork+6/32] [commit_timeout+0/16]
Jul 7 17:52:52 kalimero kernel: [kernel_thread+46/64] [kjournald+0/704]
Jul 7 17:52:52 kalimero kernel: portmap S 00000246 2384 176 1 282 126 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [datagram_poll+46/219] [schedule_timeout+163/176] [sock_poll+44/64] [do_pollfd+92/160] [do_poll+158/240]
Jul 7 17:52:52 kalimero kernel: [sys_poll+336/704] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: syslogd R F7660914 0 282 1 285 176 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [journal_dirty_sync_data+0/128] [__find_lock_page+75/128] [generic_file_write+1005/1904] [do_readv_writev+691/816] [sys_select+794/1200]
Jul 7 17:52:52 kalimero kernel: [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: klogd R F7666000 0 285 1 316 282 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [sock_sendmsg+115/176] [do_syslog+520/1664] [sock_write+153/176] [sys_read+150/448] [sys_time+29/96]
Jul 7 17:52:52 kalimero kernel: [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: 3dmd S BFFFF9CC 2384 316 1 317 374 285 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [schedule_timeout+163/176] [sys_rt_sigaction+150/176] [sys_rt_sigprocmask+81/576] [sys_nanosleep+203/336] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: 3dmd S C1C12000 2384 317 316 322 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [__pollwait+62/176] [schedule_timeout+88/176] [process_timeout+0/192] [do_poll+158/240] [sys_poll+336/704]
Jul 7 17:52:52 kalimero kernel: [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: 3dmd S C01893EE 2384 318 317 320 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [journal_dirty_metadata+462/608] [schedule_timeout+163/176] [journal_get_write_access+90/128] [wait_for_connect+576/624] [journal_get_write_access+90/128]
Jul 7 17:52:52 kalimero kernel: [tcp_accept+490/848] [clean_inode+146/224] [inet_accept+48/640] [sock_alloc+11/192] [sys_accept+111/288] [check_pgt_cache+28/32]
Jul 7 17:52:52 kalimero kernel: [clear_page_tables+153/176] [do_munmap+708/736] [sys_socketcall+198/576] [sys_munmap+66/96] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: 3dmd D F7E3AE1C 2388 320 317 322 318 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [wait_for_completion+200/320] [scsi_wait_req+132/176] [scsi_wait_done+0/48] [scsi_allocate_request+71/128] [scsi_ioctl_send_command+556/816]
Jul 7 17:52:52 kalimero kernel: [scsi_ioctl+317/944] [get_empty_filp+120/528] [sd_ioctl+216/1136] [filp_open+96/112] [blkdev_ioctl+59/64] [sys_ioctl+310/657]Jul 7 17:52:52 kalimero kernel: [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: 3dmd S 00000246 2388 322 317 320 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [datagram_poll+46/219] [schedule_timeout+163/176] [sock_poll+44/64] [do_pollfd+92/160] [do_poll+158/240]
Jul 7 17:52:52 kalimero kernel: [sys_poll+336/704] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: automount S C0132755 4968 374 1 402 316 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [handle_mm_fault+325/384] [__get_free_pages+32/48] [__pollwait+62/176] [schedule_timeout+163/176] [do_pollfd+92/160]
Jul 7 17:52:52 kalimero kernel: [do_poll+158/240] [sys_poll+336/704] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: automount S C0132755 2384 402 1 418 374 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [handle_mm_fault+325/384] [__get_free_pages+32/48] [__pollwait+62/176] [schedule_timeout+163/176] [do_pollfd+92/160]
Jul 7 17:52:52 kalimero kernel: [do_poll+158/240] [sys_poll+336/704] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: cupsd S C02D0078 5124 418 1 596 402 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [schedule_timeout+88/176] [process_timeout+0/192] [do_select+276/544] [sys_select+794/1200] [system_call+51/56] Jul 7 17:52:52 kalimero kernel: inetd S C02D0078 4 596 1 600 418 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [__pollwait+62/176] [schedule_timeout+163/176] [datagram_poll+46/219] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:52 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: lpd S F7209F2C 0 600 1 697 596 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [__alloc_pages+75/416] [schedule_timeout+163/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.19/kernel/net/packet/a+-1740386/128] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:52 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: master S C02D0078 4496 697 1 701 703 600 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [schedule_timeout+88/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.19/kernel/net/packet/a+-1740386/128] [process_timeout+0/192] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:52 kalimero kernel: [sys_select+794/1200] [do_setitimer+215/256] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: pickup S C1C12000 4620 700 697 701 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [__alloc_pages+75/416] [__get_free_pages+32/48] [schedule_timeout+88/176] [process_timeout+0/192] [sock_poll+44/64]
Jul 7 17:52:52 kalimero kernel: [do_select+276/544] [sys_select+794/1200] [do_setitimer+215/256] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: nqmgr S F71C3F2C 4620 701 697 700 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [__alloc_pages+75/416] [__get_free_pages+32/48] [schedule_timeout+88/176] [process_timeout+0/192] [do_select+276/544]
Jul 7 17:52:52 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: nmbd S C1C12000 0 703 1 704 706 697 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [schedule_timeout+88/176] [process_timeout+0/192] [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200]
Jul 7 17:52:52 kalimero kernel: [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: nmbd S F7D86430 4464 704 703 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [pipe_wait+122/176] [pipe_read+178/496] [sys_read+150/448] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: smbd S C02D0078 0 706 1 712 703 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [__pollwait+62/176] [schedule_timeout+163/176] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: sshd S C02D0078 0 712 1 726 706 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [__pollwait+62/176] [schedule_timeout+163/176] [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200]
Jul 7 17:52:52 kalimero kernel: [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: rpc.statd S F6F3DF2C 4 726 1 729 712 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [__alloc_pages+75/416] [schedule_timeout+163/176] [datagram_poll+46/219] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:52 kalimero kernel: [sys_select+794/1200] [sys_close+120/144] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: ntpd S F6F2BF2C 0 729 1 734 732 726 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [__alloc_pages+75/416] [__get_free_pages+32/48] [schedule_timeout+163/176] [datagram_poll+46/219] [sock_poll+44/64]
Jul 7 17:52:52 kalimero kernel: [do_select+276/544] [sys_select+794/1200] [restore_sigcontext+296/320] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: rpc.nfsd S 00000246 0 732 1 735 729 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [datagram_poll+46/219] [schedule_timeout+163/176] [sock_poll+44/64] [do_pollfd+92/160] [do_poll+158/240]
Jul 7 17:52:52 kalimero kernel: [sys_poll+336/704] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: ntpd S C02E4000 5336 734 729 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [restore_i387+142/208] [restore_sigcontext+296/320] [sys_rt_sigsuspend+287/320] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: rpc.mountd S 00000246 0 735 1 741 732 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [datagram_poll+46/219] [schedule_timeout+163/176] [sock_poll+44/64] [do_pollfd+92/160] [do_poll+158/240]
Jul 7 17:52:52 kalimero kernel: [sys_poll+336/704] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: atd S 00000000 0 741 1 744 735 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [dput+35/480] [schedule_timeout+88/176] [process_timeout+0/192] [sys_rt_sigprocmask+81/576] [sys_nanosleep+203/336]
Jul 7 17:52:52 kalimero kernel: [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: cron S F6EC7F1C 0 744 1 749 741 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [schedule_timeout+88/176] [process_timeout+0/192] [sys_rt_sigprocmask+81/576] [sys_nanosleep+203/336] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: kdm S F6E9BF2C 0 749 1 764 757 744 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [__alloc_pages+75/416] [schedule_timeout+163/176] [datagram_poll+46/219] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:52 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: bash S F6F0A000 0 757 1 1094 758 749 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [tty_check_change+66/160] [sys_wait4+303/1056] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: bash S 00001947 4968 758 1 6471 759 757 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [sys_wait4+303/1056] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: bash S 00000000 4968 759 1 760 758 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [__global_cli+100/112] [schedule_timeout+163/176] [con_flush_chars+132/224] [write_chan+335/512] [read_chan+579/1600]
Jul 7 17:52:52 kalimero kernel: [tty_read+352/416] [sys_read+150/448] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: bash S F6E66000 4836 760 1 6971 761 759 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [tty_check_change+66/160] [sys_wait4+303/1056] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: bash S 00001A02 4968 761 1 6658 766 760 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [sys_wait4+303/1056] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: XFree86 S C02D0078 4464 762 749 764 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [schedule_timeout+88/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.19/kernel/net/packet/a+-1740386/128] [process_timeout+0/192] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:52 kalimero kernel: [sys_select+794/1200] [tty_ioctl+617/1200] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: kdm S F718F430 0 764 749 7032 762 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [pipe_wait+122/176] [pipe_read+178/496] [sys_read+150/448] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: miniserv.pl S C02D0078 0 766 1 767 761 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [schedule_timeout+88/176] [datagram_poll+46/219] [process_timeout+0/192] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:52 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: miniserv.pl S C02D0078 0 767 1 1274 766 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [schedule_timeout+88/176] [datagram_poll+46/219] [process_timeout+0/192] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:52 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: kdm_greet S FFFFFFFF 0 772 764 7032 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [do_readv_writev+329/816] [pipe_wait+122/176] [pipe_read+178/496] [sys_read+150/448] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: bash S F664A000 4620 1094 757 5358 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [tty_check_change+66/160] [sys_wait4+303/1056] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: bash S F7D8C000 0 1274 1 7197 767 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [tty_check_change+66/160] [sys_wait4+303/1056] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: cp D F7E3AE1C 0 5358 1094 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [generic_make_request+222/320] [submit_bh+94/128] [__wait_on_buffer+94/144] [bread+128/144] [read_block_bitmap+88/176]
Jul 7 17:52:52 kalimero kernel: [__load_block_bitmap+261/400] [do_get_write_access+762/2032] [ext3_new_block+538/2864] [journal_dirty_metadata+462/608] [ext3_do_update_inode+375/1008] [journal_get_write_access+90/128]
Jul 7 17:52:52 kalimero kernel: [ext3_reserve_inode_write+104/224] [ext3_alloc_block+54/64] [ext3_alloc_branch+74/720] [ext3_get_branch+114/224] [ext3_get_block_handle+369/816] [journal_dirty_metadata+462/608]
Jul 7 17:52:52 kalimero kernel: [ext3_get_block+71/144] [__block_prepare_write+380/784] [__jbd_kmalloc+37/128] [block_prepare_write+54/144] [ext3_get_block+0/144] [ext3_prepare_write+241/768]
Jul 7 17:52:52 kalimero kernel: [ext3_get_block+0/144] [add_to_page_cache_unique+178/240] [generic_file_write+1005/1904] [sys_write+150/448] [sys_fstat64+72/128] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: vmstat S C01361AB 92 6471 758 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [__find_lock_page+75/128] [schedule_timeout+88/176] [process_timeout+0/192] [sys_rt_sigprocmask+81/576] [sys_nanosleep+203/336]
Jul 7 17:52:52 kalimero kernel: [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: vmstat S CB1F4000 0 6658 761 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [schedule_timeout+88/176] [process_timeout+0/192] [sys_rt_sigprocmask+81/576] [sys_nanosleep+203/336] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: du D F7E3AE1C 0 6971 760 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [generic_make_request+222/320] [submit_bh+94/128] [__wait_on_buffer+94/144] [ext3_bread+148/160] [ext3_readdir+140/992]
Jul 7 17:52:52 kalimero kernel: [dput+35/480] [link_path_walk+1667/2080] [in_group_p+37/48] [permission+224/240] [cp_new_stat64+227/272] [vfs_readdir+198/336]
Jul 7 17:52:52 kalimero kernel: [filldir64+0/272] [in_group_p+37/48] [sys_getdents64+83/176] [filldir64+0/272] [dput+35/480] [sys_fchdir+74/272]
Jul 7 17:52:52 kalimero kernel: [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: kdm D F7E3AE1C 0 7032 764 772 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [generic_make_request+222/320] [submit_bh+94/128] [__wait_on_buffer+94/144] [ext3_find_entry+745/816] [ext3_lookup+52/160]
Jul 7 17:52:52 kalimero kernel: [real_lookup+262/384] [cached_lookup+27/96] [link_path_walk+1774/2080] [open_namei+1041/2080] [link_path_walk+1667/2080] [filp_open+65/112]
Jul 7 17:52:52 kalimero kernel: [getname+138/192] [sys_open+77/192] [system_call+51/56]
Jul 7 17:52:52 kalimero kernel: sleep S 00000000 0 7197 1274 (NOTLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [schedule_timeout+88/176] [process_timeout+0/192] [sys_nanosleep+203/336] [system_call+51/56]
Jul 7 17:52:53 kalimero kernel: SysRq : HELP : loglevel0-8 reBoot tErm kIll saK showMem showPc unRaw Sync showTasks Unmount
Jul 7 17:52:54 kalimero last message repeated 17 times
Jul 7 17:52:56 kalimero kernel: SysRq : Show State
Jul 7 17:52:56 kalimero kernel:
Jul 7 17:52:56 kalimero kernel: free sibling
Jul 7 17:52:56 kalimero kernel: task PC stack pid father child younger older
Jul 7 17:52:56 kalimero kernel: init S C02D0078 4120 1 0 1274 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [schedule_timeout+88/176] [__pollwait+62/176] [process_timeout+0/192] [do_select+276/544] [sys_select+794/1200] Jul 7 17:52:56 kalimero kernel: [path_release+22/64] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: keventd S 0006FBDA 5216 2 1 3 (L-TLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [rest_init+0/144] [context_thread+380/672] [context_thread+0/672] [rest_init+0/144] [kernel_thread+46/64]
Jul 7 17:52:56 kalimero kernel: [context_thread+0/672]
Jul 7 17:52:56 kalimero kernel: ksoftirqd_CPU S F7EEFF8C 5736 3 1 4 2 (L-TLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [bh_action+132/224] [tasklet_hi_action+99/160] [ksoftirqd+175/256] [kernel_thread+46/64] [ksoftirqd+0/256]
Jul 7 17:52:56 kalimero kernel: ksoftirqd_CPU S F6F7A000 5736 4 1 5 3 (L-TLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [bh_action+132/224] [tasklet_action+99/160] [ksoftirqd+175/256] [kernel_thread+46/64] [ksoftirqd+0/256]
Jul 7 17:52:56 kalimero kernel: kswapd S C013F199 5204 5 1 6 4 (L-TLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [try_to_free_pages+57/96] [kswapd+134/192] [kswapd+0/192] [rest_init+0/144] [kernel_thread+46/64]
Jul 7 17:52:56 kalimero kernel: [kswapd+0/192]
Jul 7 17:52:56 kalimero kernel: bdflush S 0098993E 6284 6 1 7 5 (L-TLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [interruptible_sleep_on+143/256] [bdflush+335/352] [rest_init+0/144] [rest_init+0/144] [kernel_thread+46/64]
Jul 7 17:52:56 kalimero kernel: [bdflush+0/352]
Jul 7 17:52:56 kalimero kernel: kupdated D 00000001 5204 7 1 8 6 (L-TLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [__down+192/352] [log_start_commit+216/256] [__down_failed+11/20] [.text.lock.super+279/550] [sync_old_buffers+94/336]
Jul 7 17:52:56 kalimero kernel: [kupdate+418/480] [kupdate+0/480] [rest_init+0/144] [rest_init+0/144] [kernel_thread+46/64] [kupdate+0/480]
Jul 7 17:52:56 kalimero kernel: scsi_eh_0 S 00000000 6080 8 1 9 7 (L-TLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [vsnprintf+500/1056] [__down_interruptible+221/416] [__down_failed_interruptible+10/16] [.text.lock.scsi_error+229/290] [kernel_thread+46/64]
Jul 7 17:52:56 kalimero kernel: [scsi_error_handler+0/608]
Jul 7 17:52:56 kalimero kernel: kjournald S C69A5490 4664 9 1 24 8 (L-TLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [interruptible_sleep_on+143/256] [kjournald+444/704] [ret_from_fork+6/32] [commit_timeout+0/16] [kernel_thread+46/64]
Jul 7 17:52:56 kalimero kernel: [kjournald+0/704]
Jul 7 17:52:56 kalimero kernel: devfsd S 00000082 5452 24 1 122 9 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [devfsd_read+223/1040] [free_uid+32/128] [release_task+324/352] [sys_read+150/448] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: kjournald D C02E4000 4624 122 1 123 24 (L-TLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [generic_make_request+222/320] [submit_bh+94/128] [__wait_on_buffer+94/144] [journal_commit_transaction+1145/5360] [bh_action+132/224]
Jul 7 17:52:56 kalimero kernel: [tasklet_hi_action+99/160] [kjournald+490/704] [ret_from_fork+6/32] [commit_timeout+0/16] [kernel_thread+46/64] [kjournald+0/704]
Jul 7 17:52:56 kalimero kernel: kjournald S 00002F99 4 123 1 124 122 (L-TLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [release_console_sem+285/288] [interruptible_sleep_on+143/256] [kjournald+444/704] [ret_from_fork+6/32] [kjournald+0/704]
Jul 7 17:52:56 kalimero kernel: [commit_timeout+0/16] [kernel_thread+46/64] [kjournald+0/704]
Jul 7 17:52:56 kalimero kernel: kjournald S 00003044 484 124 1 125 123 (L-TLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [release_console_sem+285/288] [interruptible_sleep_on+143/256] [kjournald+444/704] [ret_from_fork+6/32] [kjournald+0/704]
Jul 7 17:52:56 kalimero kernel: [commit_timeout+0/16] [kernel_thread+46/64] [kjournald+0/704]
Jul 7 17:52:56 kalimero kernel: kjournald S 000030EF 4 125 1 126 124 (L-TLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [release_console_sem+285/288] [interruptible_sleep_on+143/256] [kjournald+444/704] [ret_from_fork+6/32] [kjournald+0/704]
Jul 7 17:52:56 kalimero kernel: [commit_timeout+0/16] [kernel_thread+46/64] [kjournald+0/704]
Jul 7 17:52:56 kalimero kernel: kjournald S F7EB8000 4 126 1 176 125 (L-TLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [release_console_sem+285/288] [interruptible_sleep_on+143/256] [kjournald+444/704] [ret_from_fork+6/32] [commit_timeout+0/16]
Jul 7 17:52:56 kalimero kernel: [kernel_thread+46/64] [kjournald+0/704]
Jul 7 17:52:56 kalimero kernel: portmap S 00000246 2384 176 1 282 126 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [datagram_poll+46/219] [schedule_timeout+163/176] [sock_poll+44/64] [do_pollfd+92/160] [do_poll+158/240]
Jul 7 17:52:56 kalimero kernel: [sys_poll+336/704] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: syslogd S F7667F2C 0 282 1 285 176 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [__alloc_pages+75/416] [__get_free_pages+32/48] [schedule_timeout+163/176] [datagram_poll+46/219] [af_packet:__insmod_af_packet_O/lib/modules/2.4.19/kernel/net/packet/a+-1735712/128]
Jul 7 17:52:56 kalimero kernel: [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: klogd R F7EF0000 0 285 1 316 282 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [sock_sendmsg+115/176] [do_syslog+520/1664] [sock_write+153/176] [sys_read+150/448] [sys_time+29/96]
Jul 7 17:52:56 kalimero kernel: [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: 3dmd S BFFFF9CC 2384 316 1 317 374 285 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [schedule_timeout+163/176] [sys_rt_sigaction+150/176] [sys_rt_sigprocmask+81/576] [sys_nanosleep+203/336] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: 3dmd S C1C12000 2384 317 316 322 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [__pollwait+62/176] [schedule_timeout+88/176] [process_timeout+0/192] [do_poll+158/240] [sys_poll+336/704]
Jul 7 17:52:56 kalimero kernel: [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: 3dmd S C01893EE 2384 318 317 320 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [journal_dirty_metadata+462/608] [schedule_timeout+163/176] [journal_get_write_access+90/128] [wait_for_connect+576/624] [journal_get_write_access+90/128]
Jul 7 17:52:56 kalimero kernel: [tcp_accept+490/848] [clean_inode+146/224] [inet_accept+48/640] [sock_alloc+11/192] [sys_accept+111/288] [check_pgt_cache+28/32]
Jul 7 17:52:56 kalimero kernel: [clear_page_tables+153/176] [do_munmap+708/736] [sys_socketcall+198/576] [sys_munmap+66/96] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: 3dmd D F7E3AE1C 2388 320 317 322 318 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [wait_for_completion+200/320] [scsi_wait_req+132/176] [scsi_wait_done+0/48] [scsi_allocate_request+71/128] [scsi_ioctl_send_command+556/816]
Jul 7 17:52:56 kalimero kernel: [scsi_ioctl+317/944] [get_empty_filp+120/528] [sd_ioctl+216/1136] [filp_open+96/112] [blkdev_ioctl+59/64] [sys_ioctl+310/657]Jul 7 17:52:56 kalimero kernel: [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: 3dmd S 00000246 2388 322 317 320 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [datagram_poll+46/219] [schedule_timeout+163/176] [sock_poll+44/64] [do_pollfd+92/160] [do_poll+158/240]
Jul 7 17:52:56 kalimero kernel: [sys_poll+336/704] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: automount S C0132755 4968 374 1 402 316 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [handle_mm_fault+325/384] [__get_free_pages+32/48] [__pollwait+62/176] [schedule_timeout+163/176] [do_pollfd+92/160]
Jul 7 17:52:56 kalimero kernel: [do_poll+158/240] [sys_poll+336/704] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: automount S C0132755 2384 402 1 418 374 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [handle_mm_fault+325/384] [__get_free_pages+32/48] [__pollwait+62/176] [schedule_timeout+163/176] [do_pollfd+92/160]
Jul 7 17:52:56 kalimero kernel: [do_poll+158/240] [sys_poll+336/704] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: cupsd S C02D0078 5124 418 1 596 402 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [schedule_timeout+88/176] [process_timeout+0/192] [do_select+276/544] [sys_select+794/1200] [system_call+51/56] Jul 7 17:52:56 kalimero kernel: inetd S C02D0078 4 596 1 600 418 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [__pollwait+62/176] [schedule_timeout+163/176] [datagram_poll+46/219] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:56 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: lpd S F7209F2C 0 600 1 697 596 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [__alloc_pages+75/416] [schedule_timeout+163/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.19/kernel/net/packet/a+-1740386/128] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:56 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: master S C02D0078 4496 697 1 701 703 600 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [schedule_timeout+88/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.19/kernel/net/packet/a+-1740386/128] [process_timeout+0/192] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:56 kalimero kernel: [sys_select+794/1200] [do_setitimer+215/256] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: pickup S C1C12000 4620 700 697 701 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [__alloc_pages+75/416] [__get_free_pages+32/48] [schedule_timeout+88/176] [process_timeout+0/192] [sock_poll+44/64]
Jul 7 17:52:56 kalimero kernel: [do_select+276/544] [sys_select+794/1200] [do_setitimer+215/256] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: nqmgr S F71C3F2C 4620 701 697 700 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [__alloc_pages+75/416] [__get_free_pages+32/48] [schedule_timeout+88/176] [process_timeout+0/192] [do_select+276/544]
Jul 7 17:52:56 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: nmbd S C1C12000 0 703 1 704 706 697 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [schedule_timeout+88/176] [process_timeout+0/192] [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200]
Jul 7 17:52:56 kalimero kernel: [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: nmbd S F7D86430 4464 704 703 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [pipe_wait+122/176] [pipe_read+178/496] [sys_read+150/448] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: smbd S C02D0078 0 706 1 712 703 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [__pollwait+62/176] [schedule_timeout+163/176] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: sshd S C02D0078 0 712 1 726 706 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [__pollwait+62/176] [schedule_timeout+163/176] [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200]
Jul 7 17:52:56 kalimero kernel: [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: rpc.statd S F6F3DF2C 4 726 1 729 712 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [__alloc_pages+75/416] [schedule_timeout+163/176] [datagram_poll+46/219] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:56 kalimero kernel: [sys_select+794/1200] [sys_close+120/144] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: ntpd S F6F2BF2C 0 729 1 734 732 726 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [__alloc_pages+75/416] [__get_free_pages+32/48] [schedule_timeout+163/176] [datagram_poll+46/219] [sock_poll+44/64]
Jul 7 17:52:56 kalimero kernel: [do_select+276/544] [sys_select+794/1200] [restore_sigcontext+296/320] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: rpc.nfsd S 00000246 0 732 1 735 729 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [datagram_poll+46/219] [schedule_timeout+163/176] [sock_poll+44/64] [do_pollfd+92/160] [do_poll+158/240]
Jul 7 17:52:56 kalimero kernel: [sys_poll+336/704] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: ntpd S C02E4000 5336 734 729 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [restore_i387+142/208] [restore_sigcontext+296/320] [sys_rt_sigsuspend+287/320] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: rpc.mountd S 00000246 0 735 1 741 732 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [datagram_poll+46/219] [schedule_timeout+163/176] [sock_poll+44/64] [do_pollfd+92/160] [do_poll+158/240]
Jul 7 17:52:56 kalimero kernel: [sys_poll+336/704] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: atd S 00000000 0 741 1 744 735 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [dput+35/480] [schedule_timeout+88/176] [process_timeout+0/192] [sys_rt_sigprocmask+81/576] [sys_nanosleep+203/336]
Jul 7 17:52:56 kalimero kernel: [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: cron S F6EC7F1C 0 744 1 749 741 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [schedule_timeout+88/176] [process_timeout+0/192] [sys_rt_sigprocmask+81/576] [sys_nanosleep+203/336] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: kdm S F6E9BF2C 0 749 1 764 757 744 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [__alloc_pages+75/416] [schedule_timeout+163/176] [datagram_poll+46/219] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:56 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: bash S F6F0A000 0 757 1 1094 758 749 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [tty_check_change+66/160] [sys_wait4+303/1056] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: bash S 00001947 4968 758 1 6471 759 757 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [sys_wait4+303/1056] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: bash S 00000000 4968 759 1 760 758 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [__global_cli+100/112] [schedule_timeout+163/176] [con_flush_chars+132/224] [write_chan+335/512] [read_chan+579/1600]
Jul 7 17:52:56 kalimero kernel: [tty_read+352/416] [sys_read+150/448] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: bash S F6E66000 4836 760 1 6971 761 759 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [tty_check_change+66/160] [sys_wait4+303/1056] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: bash S 00001A02 4968 761 1 6658 766 760 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [sys_wait4+303/1056] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: XFree86 S C02D0078 4464 762 749 764 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [schedule_timeout+88/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.19/kernel/net/packet/a+-1740386/128] [process_timeout+0/192] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:56 kalimero kernel: [sys_select+794/1200] [tty_ioctl+617/1200] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: kdm S F718F430 0 764 749 7032 762 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [pipe_wait+122/176] [pipe_read+178/496] [sys_read+150/448] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: miniserv.pl S C02D0078 0 766 1 767 761 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [schedule_timeout+88/176] [datagram_poll+46/219] [process_timeout+0/192] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:56 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: miniserv.pl S C02D0078 0 767 1 1274 766 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [schedule_timeout+88/176] [datagram_poll+46/219] [process_timeout+0/192] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:56 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: kdm_greet S FFFFFFFF 0 772 764 7032 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [do_readv_writev+329/816] [pipe_wait+122/176] [pipe_read+178/496] [sys_read+150/448] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: bash S F664A000 4620 1094 757 5358 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [tty_check_change+66/160] [sys_wait4+303/1056] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: bash S F7D8C000 0 1274 1 7218 767 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [tty_check_change+66/160] [sys_wait4+303/1056] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: cp D F7E3AE1C 0 5358 1094 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [generic_make_request+222/320] [submit_bh+94/128] [__wait_on_buffer+94/144] [bread+128/144] [read_block_bitmap+88/176]
Jul 7 17:52:56 kalimero kernel: [__load_block_bitmap+261/400] [do_get_write_access+762/2032] [ext3_new_block+538/2864] [journal_dirty_metadata+462/608] [ext3_do_update_inode+375/1008] [journal_get_write_access+90/128]
Jul 7 17:52:56 kalimero kernel: [ext3_reserve_inode_write+104/224] [ext3_alloc_block+54/64] [ext3_alloc_branch+74/720] [ext3_get_branch+114/224] [ext3_get_block_handle+369/816] [journal_dirty_metadata+462/608]
Jul 7 17:52:56 kalimero kernel: [ext3_get_block+71/144] [__block_prepare_write+380/784] [__jbd_kmalloc+37/128] [block_prepare_write+54/144] [ext3_get_block+0/144] [ext3_prepare_write+241/768]
Jul 7 17:52:56 kalimero kernel: [ext3_get_block+0/144] [add_to_page_cache_unique+178/240] [generic_file_write+1005/1904] [sys_write+150/448] [sys_fstat64+72/128] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: vmstat S C01361AB 92 6471 758 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [__find_lock_page+75/128] [schedule_timeout+88/176] [process_timeout+0/192] [sys_rt_sigprocmask+81/576] [sys_nanosleep+203/336]
Jul 7 17:52:56 kalimero kernel: [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: vmstat S CB1F4000 0 6658 761 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [schedule_timeout+88/176] [process_timeout+0/192] [sys_rt_sigprocmask+81/576] [sys_nanosleep+203/336] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: du D F7E3AE1C 0 6971 760 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [generic_make_request+222/320] [submit_bh+94/128] [__wait_on_buffer+94/144] [ext3_bread+148/160] [ext3_readdir+140/992]
Jul 7 17:52:56 kalimero kernel: [dput+35/480] [link_path_walk+1667/2080] [in_group_p+37/48] [permission+224/240] [cp_new_stat64+227/272] [vfs_readdir+198/336]
Jul 7 17:52:56 kalimero kernel: [filldir64+0/272] [in_group_p+37/48] [sys_getdents64+83/176] [filldir64+0/272] [dput+35/480] [sys_fchdir+74/272]
Jul 7 17:52:56 kalimero kernel: [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: kdm D F7E3AE1C 0 7032 764 772 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [generic_make_request+222/320] [submit_bh+94/128] [__wait_on_buffer+94/144] [ext3_find_entry+745/816] [ext3_lookup+52/160]
Jul 7 17:52:56 kalimero kernel: [real_lookup+262/384] [cached_lookup+27/96] [link_path_walk+1774/2080] [open_namei+1041/2080] [link_path_walk+1667/2080] [filp_open+65/112]
Jul 7 17:52:56 kalimero kernel: [getname+138/192] [sys_open+77/192] [system_call+51/56]
Jul 7 17:52:56 kalimero kernel: sleep S 00000000 0 7218 1274 (NOTLB)
Jul 7 17:52:56 kalimero kernel: Call Trace: [schedule_timeout+88/176] [process_timeout+0/192] [sys_nanosleep+203/336] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: SysRq : Show State
Jul 7 17:52:58 kalimero kernel:
Jul 7 17:52:58 kalimero kernel: free sibling
Jul 7 17:52:58 kalimero kernel: task PC stack pid father child younger older
Jul 7 17:52:58 kalimero kernel: init S C02D0078 4120 1 0 1274 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [schedule_timeout+88/176] [__pollwait+62/176] [process_timeout+0/192] [do_select+276/544] [sys_select+794/1200] Jul 7 17:52:58 kalimero kernel: [path_release+22/64] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: keventd S 0006FC2D 5216 2 1 3 (L-TLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [rest_init+0/144] [context_thread+380/672] [context_thread+0/672] [rest_init+0/144] [kernel_thread+46/64]
Jul 7 17:52:58 kalimero kernel: [context_thread+0/672]
Jul 7 17:52:58 kalimero kernel: ksoftirqd_CPU S F7EEFF8C 5736 3 1 4 2 (L-TLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [bh_action+132/224] [tasklet_hi_action+99/160] [ksoftirqd+175/256] [kernel_thread+46/64] [ksoftirqd+0/256]
Jul 7 17:52:58 kalimero kernel: ksoftirqd_CPU S F6F7A000 5736 4 1 5 3 (L-TLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [bh_action+132/224] [tasklet_action+99/160] [ksoftirqd+175/256] [kernel_thread+46/64] [ksoftirqd+0/256]
Jul 7 17:52:58 kalimero kernel: kswapd S C013F199 5204 5 1 6 4 (L-TLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [try_to_free_pages+57/96] [kswapd+134/192] [kswapd+0/192] [rest_init+0/144] [kernel_thread+46/64]
Jul 7 17:52:58 kalimero kernel: [kswapd+0/192]
Jul 7 17:52:58 kalimero kernel: bdflush S 0098993E 6284 6 1 7 5 (L-TLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [interruptible_sleep_on+143/256] [bdflush+335/352] [rest_init+0/144] [rest_init+0/144] [kernel_thread+46/64]
Jul 7 17:52:58 kalimero kernel: [bdflush+0/352]
Jul 7 17:52:58 kalimero kernel: kupdated D 00000001 5204 7 1 8 6 (L-TLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [__down+192/352] [log_start_commit+216/256] [__down_failed+11/20] [.text.lock.super+279/550] [sync_old_buffers+94/336]
Jul 7 17:52:58 kalimero kernel: [kupdate+418/480] [kupdate+0/480] [rest_init+0/144] [rest_init+0/144] [kernel_thread+46/64] [kupdate+0/480]
Jul 7 17:52:58 kalimero kernel: scsi_eh_0 S 00000000 6080 8 1 9 7 (L-TLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [vsnprintf+500/1056] [__down_interruptible+221/416] [__down_failed_interruptible+10/16] [.text.lock.scsi_error+229/290] [kernel_thread+46/64]
Jul 7 17:52:58 kalimero kernel: [scsi_error_handler+0/608]
Jul 7 17:52:58 kalimero kernel: kjournald S C69A5490 4664 9 1 24 8 (L-TLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [interruptible_sleep_on+143/256] [kjournald+444/704] [ret_from_fork+6/32] [commit_timeout+0/16] [kernel_thread+46/64]
Jul 7 17:52:58 kalimero kernel: [kjournald+0/704]
Jul 7 17:52:58 kalimero kernel: devfsd S 00000082 5452 24 1 122 9 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [devfsd_read+223/1040] [free_uid+32/128] [release_task+324/352] [sys_read+150/448] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: kjournald D C02E4000 4624 122 1 123 24 (L-TLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [generic_make_request+222/320] [submit_bh+94/128] [__wait_on_buffer+94/144] [journal_commit_transaction+1145/5360] [bh_action+132/224]
Jul 7 17:52:58 kalimero kernel: [tasklet_hi_action+99/160] [kjournald+490/704] [ret_from_fork+6/32] [commit_timeout+0/16] [kernel_thread+46/64] [kjournald+0/704]
Jul 7 17:52:58 kalimero kernel: kjournald S 00002F99 4 123 1 124 122 (L-TLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [release_console_sem+285/288] [interruptible_sleep_on+143/256] [kjournald+444/704] [ret_from_fork+6/32] [kjournald+0/704]
Jul 7 17:52:58 kalimero kernel: [commit_timeout+0/16] [kernel_thread+46/64] [kjournald+0/704]
Jul 7 17:52:58 kalimero kernel: kjournald S 00003044 484 124 1 125 123 (L-TLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [release_console_sem+285/288] [interruptible_sleep_on+143/256] [kjournald+444/704] [ret_from_fork+6/32] [kjournald+0/704]
Jul 7 17:52:58 kalimero kernel: [commit_timeout+0/16] [kernel_thread+46/64] [kjournald+0/704]
Jul 7 17:52:58 kalimero kernel: kjournald S 000030EF 4 125 1 126 124 (L-TLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [release_console_sem+285/288] [interruptible_sleep_on+143/256] [kjournald+444/704] [ret_from_fork+6/32] [kjournald+0/704]
Jul 7 17:52:58 kalimero kernel: [commit_timeout+0/16] [kernel_thread+46/64] [kjournald+0/704]
Jul 7 17:52:58 kalimero kernel: kjournald S F7EB8000 4 126 1 176 125 (L-TLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [release_console_sem+285/288] [interruptible_sleep_on+143/256] [kjournald+444/704] [ret_from_fork+6/32] [commit_timeout+0/16]
Jul 7 17:52:58 kalimero kernel: [kernel_thread+46/64] [kjournald+0/704]
Jul 7 17:52:58 kalimero kernel: portmap S 00000246 2384 176 1 282 126 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [datagram_poll+46/219] [schedule_timeout+163/176] [sock_poll+44/64] [do_pollfd+92/160] [do_poll+158/240]
Jul 7 17:52:58 kalimero kernel: [sys_poll+336/704] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: syslogd S F7667F2C 0 282 1 285 176 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [__alloc_pages+75/416] [__get_free_pages+32/48] [schedule_timeout+163/176] [datagram_poll+46/219] [af_packet:__insmod_af_packet_O/lib/modules/2.4.19/kernel/net/packet/a+-1735712/128]
Jul 7 17:52:58 kalimero kernel: [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200] [do_setitimer+215/256] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: klogd R 00000000 0 285 1 316 282 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [sock_sendmsg+115/176] [do_syslog+520/1664] [sock_write+153/176] [sys_read+150/448] [sys_time+29/96]
Jul 7 17:52:58 kalimero kernel: [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: 3dmd S BFFFF9CC 2384 316 1 317 374 285 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [schedule_timeout+163/176] [sys_rt_sigaction+150/176] [sys_rt_sigprocmask+81/576] [sys_nanosleep+203/336] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: 3dmd S C1C12000 2384 317 316 322 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [__pollwait+62/176] [schedule_timeout+88/176] [process_timeout+0/192] [do_poll+158/240] [sys_poll+336/704]
Jul 7 17:52:58 kalimero kernel: [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: 3dmd S C01893EE 2384 318 317 320 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [journal_dirty_metadata+462/608] [schedule_timeout+163/176] [journal_get_write_access+90/128] [wait_for_connect+576/624] [journal_get_write_access+90/128]
Jul 7 17:52:58 kalimero kernel: [tcp_accept+490/848] [clean_inode+146/224] [inet_accept+48/640] [sock_alloc+11/192] [sys_accept+111/288] [check_pgt_cache+28/32]
Jul 7 17:52:58 kalimero kernel: [clear_page_tables+153/176] [do_munmap+708/736] [sys_socketcall+198/576] [sys_munmap+66/96] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: 3dmd D F7E3AE1C 2388 320 317 322 318 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [wait_for_completion+200/320] [scsi_wait_req+132/176] [scsi_wait_done+0/48] [scsi_allocate_request+71/128] [scsi_ioctl_send_command+556/816]
Jul 7 17:52:58 kalimero kernel: [scsi_ioctl+317/944] [get_empty_filp+120/528] [sd_ioctl+216/1136] [filp_open+96/112] [blkdev_ioctl+59/64] [sys_ioctl+310/657]Jul 7 17:52:58 kalimero kernel: [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: 3dmd S 00000246 2388 322 317 320 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [datagram_poll+46/219] [schedule_timeout+163/176] [sock_poll+44/64] [do_pollfd+92/160] [do_poll+158/240]
Jul 7 17:52:58 kalimero kernel: [sys_poll+336/704] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: automount S C0132755 4968 374 1 402 316 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [handle_mm_fault+325/384] [__get_free_pages+32/48] [__pollwait+62/176] [schedule_timeout+163/176] [do_pollfd+92/160]
Jul 7 17:52:58 kalimero kernel: [do_poll+158/240] [sys_poll+336/704] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: automount S C0132755 2384 402 1 418 374 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [handle_mm_fault+325/384] [__get_free_pages+32/48] [__pollwait+62/176] [schedule_timeout+163/176] [do_pollfd+92/160]
Jul 7 17:52:58 kalimero kernel: [do_poll+158/240] [sys_poll+336/704] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: cupsd S C02D0078 5124 418 1 596 402 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [schedule_timeout+88/176] [process_timeout+0/192] [do_select+276/544] [sys_select+794/1200] [system_call+51/56] Jul 7 17:52:58 kalimero kernel: inetd S C02D0078 4 596 1 600 418 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [__pollwait+62/176] [schedule_timeout+163/176] [datagram_poll+46/219] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:58 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: lpd S F7209F2C 0 600 1 697 596 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [__alloc_pages+75/416] [schedule_timeout+163/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.19/kernel/net/packet/a+-1740386/128] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:58 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: master S C02D0078 4496 697 1 701 703 600 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [schedule_timeout+88/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.19/kernel/net/packet/a+-1740386/128] [process_timeout+0/192] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:58 kalimero kernel: [sys_select+794/1200] [do_setitimer+215/256] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: pickup S C1C12000 4620 700 697 701 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [__alloc_pages+75/416] [__get_free_pages+32/48] [schedule_timeout+88/176] [process_timeout+0/192] [sock_poll+44/64]
Jul 7 17:52:58 kalimero kernel: [do_select+276/544] [sys_select+794/1200] [do_setitimer+215/256] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: nqmgr S F71C3F2C 4620 701 697 700 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [__alloc_pages+75/416] [__get_free_pages+32/48] [schedule_timeout+88/176] [process_timeout+0/192] [do_select+276/544]
Jul 7 17:52:58 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: nmbd S C1C12000 0 703 1 704 706 697 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [schedule_timeout+88/176] [process_timeout+0/192] [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200]
Jul 7 17:52:58 kalimero kernel: [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: nmbd S F7D86430 4464 704 703 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [pipe_wait+122/176] [pipe_read+178/496] [sys_read+150/448] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: smbd S C02D0078 0 706 1 712 703 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [__pollwait+62/176] [schedule_timeout+163/176] [do_select+276/544] [sys_select+794/1200] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: sshd S C02D0078 0 712 1 726 706 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [__pollwait+62/176] [schedule_timeout+163/176] [sock_poll+44/64] [do_select+276/544] [sys_select+794/1200]
Jul 7 17:52:58 kalimero kernel: [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: rpc.statd S F6F3DF2C 4 726 1 729 712 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [__alloc_pages+75/416] [schedule_timeout+163/176] [datagram_poll+46/219] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:58 kalimero kernel: [sys_select+794/1200] [sys_close+120/144] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: ntpd S F6F2BF2C 0 729 1 734 732 726 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [__alloc_pages+75/416] [__get_free_pages+32/48] [schedule_timeout+163/176] [datagram_poll+46/219] [sock_poll+44/64]
Jul 7 17:52:58 kalimero kernel: [do_select+276/544] [sys_select+794/1200] [restore_sigcontext+296/320] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: rpc.nfsd S 00000246 0 732 1 735 729 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [datagram_poll+46/219] [schedule_timeout+163/176] [sock_poll+44/64] [do_pollfd+92/160] [do_poll+158/240]
Jul 7 17:52:58 kalimero kernel: [sys_poll+336/704] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: ntpd S C02E4000 5336 734 729 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [restore_i387+142/208] [restore_sigcontext+296/320] [sys_rt_sigsuspend+287/320] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: rpc.mountd S 00000246 0 735 1 741 732 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [datagram_poll+46/219] [schedule_timeout+163/176] [sock_poll+44/64] [do_pollfd+92/160] [do_poll+158/240]
Jul 7 17:52:58 kalimero kernel: [sys_poll+336/704] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: atd S 00000000 0 741 1 744 735 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [dput+35/480] [schedule_timeout+88/176] [process_timeout+0/192] [sys_rt_sigprocmask+81/576] [sys_nanosleep+203/336]
Jul 7 17:52:58 kalimero kernel: [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: cron S F6EC7F1C 0 744 1 749 741 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [schedule_timeout+88/176] [process_timeout+0/192] [sys_rt_sigprocmask+81/576] [sys_nanosleep+203/336] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: kdm S F6E9BF2C 0 749 1 764 757 744 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [__alloc_pages+75/416] [schedule_timeout+163/176] [datagram_poll+46/219] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:58 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: bash S F6F0A000 0 757 1 1094 758 749 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [tty_check_change+66/160] [sys_wait4+303/1056] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: bash S 00001947 4968 758 1 6471 759 757 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [sys_wait4+303/1056] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: bash S 00000000 4968 759 1 760 758 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [__global_cli+100/112] [schedule_timeout+163/176] [con_flush_chars+132/224] [write_chan+335/512] [read_chan+579/1600]
Jul 7 17:52:58 kalimero kernel: [tty_read+352/416] [sys_read+150/448] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: bash S F6E66000 4836 760 1 6971 761 759 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [tty_check_change+66/160] [sys_wait4+303/1056] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: bash S 00001A02 4968 761 1 6658 766 760 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [sys_wait4+303/1056] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: XFree86 S C02D0078 4464 762 749 764 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [schedule_timeout+88/176] [af_packet:__insmod_af_packet_O/lib/modules/2.4.19/kernel/net/packet/a+-1740386/128] [process_timeout+0/192] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:58 kalimero kernel: [sys_select+794/1200] [tty_ioctl+617/1200] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: kdm S F718F430 0 764 749 7032 762 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [pipe_wait+122/176] [pipe_read+178/496] [sys_read+150/448] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: miniserv.pl S C02D0078 0 766 1 767 761 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [schedule_timeout+88/176] [datagram_poll+46/219] [process_timeout+0/192] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:58 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: miniserv.pl S C02D0078 0 767 1 1274 766 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [schedule_timeout+88/176] [datagram_poll+46/219] [process_timeout+0/192] [sock_poll+44/64] [do_select+276/544]
Jul 7 17:52:58 kalimero kernel: [sys_select+794/1200] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: kdm_greet S FFFFFFFF 0 772 764 7032 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [do_readv_writev+329/816] [pipe_wait+122/176] [pipe_read+178/496] [sys_read+150/448] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: bash S F664A000 4620 1094 757 5358 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [tty_check_change+66/160] [sys_wait4+303/1056] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: bash S F7D8C000 0 1274 1 7233 767 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [tty_check_change+66/160] [sys_wait4+303/1056] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: cp D F7E3AE1C 0 5358 1094 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [generic_make_request+222/320] [submit_bh+94/128] [__wait_on_buffer+94/144] [bread+128/144] [read_block_bitmap+88/176]
Jul 7 17:52:58 kalimero kernel: [__load_block_bitmap+261/400] [do_get_write_access+762/2032] [ext3_new_block+538/2864] [journal_dirty_metadata+462/608] [ext3_do_update_inode+375/1008] [journal_get_write_access+90/128]
Jul 7 17:52:58 kalimero kernel: [ext3_reserve_inode_write+104/224] [ext3_alloc_block+54/64] [ext3_alloc_branch+74/720] [ext3_get_branch+114/224] [ext3_get_block_handle+369/816] [journal_dirty_metadata+462/608]
Jul 7 17:52:58 kalimero kernel: [ext3_get_block+71/144] [__block_prepare_write+380/784] [__jbd_kmalloc+37/128] [block_prepare_write+54/144] [ext3_get_block+0/144] [ext3_prepare_write+241/768]
Jul 7 17:52:58 kalimero kernel: [ext3_get_block+0/144] [add_to_page_cache_unique+178/240] [generic_file_write+1005/1904] [sys_write+150/448] [sys_fstat64+72/128] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: vmstat S C01361AB 92 6471 758 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [__find_lock_page+75/128] [schedule_timeout+88/176] [process_timeout+0/192] [sys_rt_sigprocmask+81/576] [sys_nanosleep+203/336]
Jul 7 17:52:58 kalimero kernel: [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: vmstat S CB1F4000 0 6658 761 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [schedule_timeout+88/176] [process_timeout+0/192] [sys_rt_sigprocmask+81/576] [sys_nanosleep+203/336] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: du D F7E3AE1C 0 6971 760 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [generic_make_request+222/320] [submit_bh+94/128] [__wait_on_buffer+94/144] [ext3_bread+148/160] [ext3_readdir+140/992]
Jul 7 17:52:58 kalimero kernel: [dput+35/480] [link_path_walk+1667/2080] [in_group_p+37/48] [permission+224/240] [cp_new_stat64+227/272] [vfs_readdir+198/336]
Jul 7 17:52:58 kalimero kernel: [filldir64+0/272] [in_group_p+37/48] [sys_getdents64+83/176] [filldir64+0/272] [dput+35/480] [sys_fchdir+74/272]
Jul 7 17:52:58 kalimero kernel: [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: kdm D F7E3AE1C 0 7032 764 772 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [generic_make_request+222/320] [submit_bh+94/128] [__wait_on_buffer+94/144] [ext3_find_entry+745/816] [ext3_lookup+52/160]
Jul 7 17:52:58 kalimero kernel: [real_lookup+262/384] [cached_lookup+27/96] [link_path_walk+1774/2080] [open_namei+1041/2080] [link_path_walk+1667/2080] [filp_open+65/112]
Jul 7 17:52:58 kalimero kernel: [getname+138/192] [sys_open+77/192] [system_call+51/56]
Jul 7 17:52:58 kalimero kernel: sleep S 00000000 0 7233 1274 (NOTLB)
Jul 7 17:52:58 kalimero kernel: Call Trace: [schedule_timeout+88/176] [process_timeout+0/192] [sys_nanosleep+203/336] [system_call+51/56]
Jul 7 17:52:59 kalimero kernel: 3w-xxxx: scsi0: Unit #0: Command (0xf7dec200) timed out, resetting card.
Here the system hangs (note a lot of Ds ...)
thanks for any help provided.
regards,
Vincent Touquet
The attachment I forgot (all the traces)...
v
Ok, so I forgot the vmstat output too :/
Looks pretty similar to the vmstat output with the hang of 2.4.21.
Using 2.4.19 it takes longer to take the system down, there is a long
time where data actually gets to the array, before the behaviour where
only a few blocks get written out anymore, untill the system hangs.
kalimero:~# cat vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 1 424 9440 11656 943172 0 0 511 521 73 72 0 1 98 0
0 2 424 9568 11676 942972 0 0 3852 24576 365 323 0 4 96 0
1 1 424 9564 11732 943548 0 0 12180 6636 502 816 0 9 91 0
2 1 424 10488 11772 942796 0 0 32032 0 749 2013 0 22 77 0
0 2 552 10376 11816 942984 0 0 26140 53220 671 1646 0 23 76 0
0 2 552 10512 11856 942732 0 0 16788 24448 547 1094 1 20 79 0
0 2 552 10492 11864 942732 0 0 0 20232 262 70 0 2 98 0
1 2 552 10120 11944 942664 0 0 24632 21012 695 1580 1 24 75 0
0 3 552 9164 11948 939296 0 0 30240 16708 789 2101 1 23 76 0
0 2 552 9996 12000 939788 0 0 30628 0 758 2119 0 18 82 0
4 0 552 3324 12076 926332 0 0 23672 40244 605 1451 1 25 74 0
2 3 552 9456 12140 926544 0 0 26532 22560 708 1659 1 22 77 0
2 2 552 8840 12352 938000 0 0 16912 26104 661 1173 3 16 80 0
0 5 552 9308 12560 940076 0 0 19988 21176 693 1503 3 17 80 0
0 3 552 9512 12244 942024 0 0 16912 21820 582 1169 1 15 84 0
0 3 552 9576 12272 941988 0 0 8336 16588 451 593 1 6 92 0
0 3 552 9432 12288 942928 0 0 6920 0 395 463 1 6 93 0
1 1 552 9556 12376 943136 0 0 28400 48644 778 1974 0 22 78 0
0 2 552 9488 12476 943072 0 0 13448 17392 559 972 1 12 87 0
0 2 552 9484 12480 943072 0 0 4 20608 277 61 0 1 99 0
0 2 552 9484 12480 943072 0 0 0 28420 303 72 0 3 97 0
0 2 552 9472 12484 943072 0 0 4 2068 269 82 0 3 97 0
0 2 552 9468 12496 943072 0 0 8 0 261 68 1 0 99 0
2 1 552 9448 12620 944316 0 0 20492 25280 630 1372 0 15 84 0
0 2 552 9524 12684 944080 0 0 26784 41224 663 1731 0 21 79 0
1 1 552 9476 12740 943896 0 0 33472 2900 832 2140 1 22 77 0
0 2 552 9448 12780 943792 0 0 9984 0 393 686 0 7 93 0
1 1 552 9548 12860 943640 0 0 18972 104 443 1245 1 13 86 0
1 1 552 9532 12916 943340 0 0 27676 49432 642 1772 2 21 77 0
0 2 552 9972 12968 942372 0 0 30620 20480 743 1959 0 24 75 0
0 2 680 10308 12968 942060 0 256 7304 10708 371 504 0 8 91 0
1 4 680 10428 13148 941512 0 0 22040 928 681 1585 3 20 77 0
0 1 680 10424 13464 941384 0 0 15508 996 628 1530 3 21 75 0
0 2 680 10540 13504 940960 0 0 9996 41460 376 701 1 14 85 0
0 2 680 8736 13544 936476 0 0 29980 22400 759 1940 0 38 62 0
2 1 680 6504 13576 923180 0 0 17296 22656 552 1227 0 18 82 0
1 2 680 7704 13656 924124 0 0 29644 7108 754 1844 0 22 77 0
0 2 680 19544 13688 927160 0 0 5980 0 356 380 0 4 96 0
1 2 680 9688 13768 940848 0 0 14364 54420 510 780 0 13 87 0
0 2 680 9984 13772 940848 0 0 4 24008 298 77 0 1 99 0
0 2 680 10108 13772 940848 0 0 0 12776 268 66 0 3 97 0
0 2 680 10156 13776 940848 0 0 4 0 262 57 0 1 99 0
0 1 680 9500 13844 942340 0 0 11180 180 377 743 0 8 92 0
0 2 680 9468 13868 942668 0 0 28852 32152 724 1944 1 17 82 0
1 1 680 9480 13920 942588 0 0 8680 0 479 738 0 9 91 0
0 1 680 9468 14000 942572 0 0 20216 148 637 1550 0 13 87 0
4 1 680 9444 14244 942052 0 0 20724 432 564 1598 0 11 89 0
1 0 680 9500 14856 941152 0 0 24588 1036 849 2420 4 19 76 0
0 2 680 9568 14968 940724 0 0 15028 50876 505 1058 0 11 88 0
0 2 680 9556 14972 940728 0 0 4 15260 232 67 0 1 99 0
0 2 680 9520 15048 940228 0 0 30376 16236 764 2029 1 22 77 0
0 2 680 9480 15164 940060 0 0 3176 15012 345 374 1 2 97 0
0 3 680 9388 15512 939804 0 0 344 1788 317 234 0 2 97 0
0 2 680 9480 15660 939796 0 0 5356 944 381 458 0 4 96 0
0 2 680 9440 15704 940148 0 0 31776 47596 729 2005 0 24 76 0
0 2 680 9496 15764 940076 0 0 34084 17712 812 2144 0 26 74 0
0 2 680 9552 15816 939780 0 0 28188 0 708 1807 0 19 81 0
1 1 680 9468 15820 939676 0 0 13072 0 430 864 1 7 92 0
0 2 680 9528 15876 939576 0 0 17424 41548 492 1150 1 14 85 0
1 1 808 10528 15916 938360 0 128 30116 16952 761 1984 1 25 74 0
0 2 808 10368 15600 938420 0 0 32384 24704 813 2081 1 25 74 0
0 4 808 9716 16056 938388 0 0 23464 17940 829 2069 2 28 70 0
0 4 808 9712 16056 938388 0 0 0 0 271 55 0 1 99 0
1 1 808 9452 16112 939096 0 0 23836 452 702 1684 0 16 83 0
0 2 808 7416 16168 924056 0 0 25112 59440 672 1751 0 29 71 0
1 2 808 7524 16216 918320 0 0 24772 17372 665 1631 0 23 77 0
1 3 808 8284 16236 929356 0 0 31260 20736 761 1794 1 19 80 0
0 2 808 12964 16256 930940 0 0 5388 22700 385 396 0 11 89 0
1 2 808 9160 16288 938016 0 0 5160 476 379 298 0 4 96 0
1 2 808 9232 16324 939480 0 0 31272 0 729 1932 1 22 77 0
0 3 808 9284 16312 939376 0 0 29868 53036 744 1916 1 23 76 0
0 3 808 9308 16060 939184 0 0 27720 24540 746 1798 0 21 79 0
0 3 808 9252 16064 939184 0 0 4 24320 289 82 0 4 96 0
1 2 808 9240 16128 938856 0 0 19868 20820 620 1309 0 16 84 0
0 3 808 9280 15784 939652 0 0 24676 6044 664 1594 0 20 80 0
1 3 808 9228 15840 939680 0 0 27300 376 707 1765 10 21 68 0
1 2 808 9292 15888 939256 0 0 24628 0 491 1546 2 18 80 0
0 8 808 9236 15944 938632 0 0 24224 352 574 1796 3 18 79 0
0 5 808 9448 16392 937860 0 0 18328 1144 565 1711 5 18 77 0
IMPORTANT POINT
0 5 808 9444 16392 937860 0 0 0 0 113 98 0 2 98 0
0 5 808 9432 16396 937860 0 0 0 0 108 67 1 2 97 0
0 5 808 9424 16400 937860 0 0 0 0 129 86 0 2 97 0
0 5 808 9404 16400 937860 0 0 0 0 337 506 0 2 97 0
0 6 808 9384 16416 937860 0 0 0 328 164 141 1 1 98 0
0 6 808 9384 16416 937860 0 0 0 0 101 36 1 1 98 0
0 6 808 9368 16416 937860 0 0 0 0 277 384 1 1 98 0
0 6 808 9368 16416 937860 0 0 0 0 105 42 0 1 98 0
0 6 808 9360 16416 937860 0 0 0 0 233 305 1 2 97 0
0 6 808 9328 16428 937860 0 0 0 328 119 41 0 3 97 0
0 6 808 9328 16428 937860 0 0 0 0 101 33 0 4 96 0
0 6 808 9316 16428 937860 0 0 0 0 285 400 0 0 100 0
0 6 808 9312 16428 937860 0 0 0 0 139 121 1 1 98 0
0 6 808 9312 16428 937860 0 0 0 0 105 42 0 0 100 0
0 6 808 9284 16440 937808 0 0 8 328 128 297 1 2 97 0
0 6 808 9284 16440 937808 0 0 0 0 108 50 0 0 100 0
0 6 808 9284 16440 937808 0 0 0 0 114 55 0 0 100 0
0 6 808 9216 16440 937872 0 0 0 0 114 485 0 2 98 0
0 6 808 9344 16440 937744 0 0 0 0 112 61 0 0 100 0
1 6 808 9276 16452 937804 0 0 0 548 132 346 0 2 98 0
0 6 808 9248 16452 937804 0 0 0 0 109 48 0 0 100 0
0 6 808 9248 16452 937804 0 0 0 0 108 46 0 0 100 0
0 6 808 9308 16452 937736 0 0 0 0 105 341 0 1 98 0
0 6 808 9292 16460 937744 0 0 0 0 105 49 0 0 100 0
0 6 808 9276 16476 937744 0 0 0 424 139 323 0 1 99 0
0 6 808 9272 16480 937744 0 0 0 0 104 41 0 1 99 0
0 6 808 9332 16488 937676 0 0 0 0 103 345 0 2 98 0
0 6 808 9324 16496 937676 0 0 0 0 101 32 0 0 100 0
0 6 808 9212 16500 937748 0 0 0 0 103 339 0 2 98 0
The date on the file the vmstat was written to is Jul 7 17:52,
every second it was written (last possible one at 17:52:59, when the
system hung)... That makes the 'important point in time' about 17:52:59 -
29, which is 17:52:30. The closest trace I have is from 17:52:44
The cp process is then already in state D (uninterruptible sleep), it
wasn't in state D in the trace at time [17:50:39], when it was on the
run queue.
Can anyone make sense out of this ? :))
best regards,
Vincent
After my search for what caused my hangs, I decided to wonder if DMA could
be the culprit. I put ide=nodma in the commandline and the system is still
not hanging under the copy task (the system hangs when copying from an ide disk
to a raid array).
Looking at the output of vmstat with dma on:
http://marc.theaimsgroup.com/?l=linux-kernel&m=105759652518028&w=2
You can see that sometimes there are stalls on the blocks in (bi) side
[reading data from the IDE disk]. Performance is rather 'stellar' with on
average 20.000 blocks per second input from the ide disk, but sometimes this
drops to zero. This could be problems with reading from the disk (or is the
write-out not happening fast enough ?)
Vmstat without dma on the ide disk is much more moderate (reading less blocks per second from the ide disk):
Extract from the now ongoing copy process.
2 0 3968 9568 20624 915104 0 0 3972 0 605 748 1 51 48 0
1 0 3968 9472 20652 915132 0 0 3588 14980 700 795 1 52 47 0
0 1 3968 9488 20644 915212 0 0 3972 0 613 751 0 48 51 0
1 0 3968 9540 20652 915092 0 0 3972 0 603 756 3 45 51 0
0 1 3968 9564 20668 915036 0 0 4108 0 616 752 0 56 44 0
1 0 3968 9456 20688 915072 0 0 3976 0 605 749 2 43 55 0
1 1 3968 9532 20700 914960 0 0 3532 19344 716 873 1 43 56 0
3 0 3968 9460 20712 915100 0 0 3832 4 609 727 3 50 48 0
1 1 3968 9508 20724 915032 0 0 4108 0 601 761 0 48 51 0
0 1 3968 9480 20752 915040 0 0 4112 0 613 814 1 52 47 0
0 4 3968 9532 20780 914888 0 0 3720 19316 610 704 3 48 50 0
1 0 3968 9500 20836 914764 0 0 2100 88 535 782 3 33 64 0
There seem to be no stalls on the reader side.
Now the big question is: is dma really at fault here, or are there problems on
the write-out side ? [if dma is the problem, maybe we should reopen the discussion
of enabling dma by default ;)]
I think the answer is in the traces near the point were the machine hangs:
http://marc.theaimsgroup.com/?l=linux-kernel&m=105759465915936&w=2
This snippet then again, makes me think there is something wrong on the scsi side...
Or is the problem with the IDE somehow also disturbing the scsi system
(PCI bus hang ?).
Jul 7 17:52:52 kalimero kernel: kupdated D 00000001 5204 7 1 8 6 (L-TLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [__down+192/352] [log_start_commit+216/256] [__down_failed+11/20] [.text.lock.super+279/550] [sync_old_buffers+94/336]
Jul 7 17:52:52 kalimero kernel: [kupdate+418/480] [kupdate+0/480] [rest_init+0/144] [rest_init+0/144] [kernel_thread+46/64] [kupdate+0/480]
Jul 7 17:52:52 kalimero kernel: scsi_eh_0 S 00000000 6080 8 1 9 7 (L-TLB)
Jul 7 17:52:52 kalimero kernel: Call Trace: [vsnprintf+500/1056] [__down_interruptible+221/416] [__down_failed_interruptible+10/16] [.text.lock.scsi_error+229/290] [kernel_thread+46/64]
Jul 7 17:52:52 kalimero kernel: [scsi_error_handler+0/608]
I wonder how I could decide the case of dma vs. scsi (as the root cause of the
problem).
best regards,
Vincent
Vincent -
I wonder if what is really happening is a problem in the in the arbitration
between the PCI bus and the local bus that the onboard IDE devices are. In
your case the problem (onboard IDE device data corruption) manifests when you
are performing sustained transfers (large files) between the onboard IDE
device and a PCI block device (the 3ware RAID). In my case, the same problem
manifests when I have sustained data activity from multiple frame grabbers to
memory, then from memory to RAID. When the system drive is used (code load,
swap, etc.) it gets corrupted. My point is, the onboard data device only
seems to get corrupted when there is lots of i/o activity with PCI
bus-masters that are DMA'ing data to/from memory. What do you think?
On Tuesday 08 July 2003 06:19 am, Vincent Touquet wrote:
> After my search for what caused my hangs, I decided to wonder if DMA could
> be the culprit. I put ide=nodma in the commandline and the system is still
> not hanging under the copy task (the system hangs when copying from an ide
> disk to a raid array).
>
> Looking at the output of vmstat with dma on:
> http://marc.theaimsgroup.com/?l=linux-kernel&m=105759652518028&w=2
>
> You can see that sometimes there are stalls on the blocks in (bi) side
> [reading data from the IDE disk]. Performance is rather 'stellar' with on
> average 20.000 blocks per second input from the ide disk, but sometimes
> this drops to zero. This could be problems with reading from the disk (or
> is the write-out not happening fast enough ?)
>
> Vmstat without dma on the ide disk is much more moderate (reading less
> blocks per second from the ide disk):
>
> Extract from the now ongoing copy process.
> 2 0 3968 9568 20624 915104 0 0 3972 0 605 748 1 51 48
> 0 1 0 3968 9472 20652 915132 0 0 3588 14980 700 795 1 52
> 47 0 0 1 3968 9488 20644 915212 0 0 3972 0 613 751 0
> 48 51 0 1 0 3968 9540 20652 915092 0 0 3972 0 603 756
> 3 45 51 0 0 1 3968 9564 20668 915036 0 0 4108 0 616
> 752 0 56 44 0 1 0 3968 9456 20688 915072 0 0 3976 0 605
> 749 2 43 55 0 1 1 3968 9532 20700 914960 0 0 3532 19344
> 716 873 1 43 56 0 3 0 3968 9460 20712 915100 0 0 3832
> 4 609 727 3 50 48 0 1 1 3968 9508 20724 915032 0 0 4108
> 0 601 761 0 48 51 0 0 1 3968 9480 20752 915040 0 0 4112
> 0 613 814 1 52 47 0 0 4 3968 9532 20780 914888 0 0
> 3720 19316 610 704 3 48 50 0 1 0 3968 9500 20836 914764 0
> 0 2100 88 535 782 3 33 64 0
>
> There seem to be no stalls on the reader side.
>
> Now the big question is: is dma really at fault here, or are there problems
> on the write-out side ? [if dma is the problem, maybe we should reopen the
> discussion of enabling dma by default ;)]
>
> I think the answer is in the traces near the point were the machine hangs:
> http://marc.theaimsgroup.com/?l=linux-kernel&m=105759465915936&w=2
>
> This snippet then again, makes me think there is something wrong on the
> scsi side... Or is the problem with the IDE somehow also disturbing the
> scsi system (PCI bus hang ?).
>
> Jul 7 17:52:52 kalimero kernel: kupdated D 00000001 5204 7
> 1 8 6 (L-TLB) Jul 7 17:52:52 kalimero kernel: Call Trace:
> [__down+192/352] [log_start_commit+216/256] [__down_failed+11/20]
> [.text.lock.super+279/550] [sync_old_buffers+94/336] Jul 7 17:52:52
> kalimero kernel: [kupdate+418/480] [kupdate+0/480] [rest_init+0/144]
> [rest_init+0/144] [kernel_thread+46/64] [kupdate+0/480] Jul 7 17:52:52
> kalimero kernel: scsi_eh_0 S 00000000 6080 8 1 9
> 7 (L-TLB) Jul 7 17:52:52 kalimero kernel: Call Trace:
> [vsnprintf+500/1056] [__down_interruptible+221/416]
> [__down_failed_interruptible+10/16] [.text.lock.scsi_error+229/290]
> [kernel_thread+46/64] Jul 7 17:52:52 kalimero kernel:
> [scsi_error_handler+0/608]
>
> I wonder how I could decide the case of dma vs. scsi (as the root cause of
> the problem).
>
> best regards,
>
> Vincent
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
--
Joe Briggs
Briggs Media Systems
105 Burnsen Ave.
Manchester NH 01304 USA
TEL 603-232-3115 FAX 603-625-5809 MOBILE 603-493-2386
http://www.briggsmedia.com
On Tue, Jul 08, 2003 at 09:59:54AM -0400, joe briggs wrote:
>Vincent -
>I wonder if what is really happening is a problem in the in the arbitration
>between the PCI bus and the local bus that the onboard IDE devices are. In
>your case the problem (onboard IDE device data corruption) manifests when you
>are performing sustained transfers (large files) between the onboard IDE
>device and a PCI block device (the 3ware RAID).
Yes, that seems very feasible.
>In my case, the same problem
>manifests when I have sustained data activity from multiple frame grabbers to
>memory, then from memory to RAID. When the system drive is used (code load,
>swap, etc.) it gets corrupted. My point is, the onboard data device only
>seems to get corrupted when there is lots of i/o activity with PCI
>bus-masters that are DMA'ing data to/from memory. What do you think?
I had the same lockups too when pumping a lot of data over the network
onto the array on a similar mainboard (Tyan S2468). So maybe there is
the added problem that there is funny things going on on the PCI bus.
Maybe the problem only occurs when you stress the bus and the dma is not
the real culprit, it just enables high transfers and hence corruption on
the PCI bus.
I would very much like to nail this one down, as its nasty.
regards,
v
By the way, I'm having the same problem as described in here too:
http://www.cs.helsinki.fi/linux/linux-kernel/2003-21/0619.html
Perhaps time we took another look at the AMD 74xx ide code ?
Or is this particular bug considered harmless ?
regards,
v
On Tue, Jul 08, 2003 at 06:14:06PM +0200, Vincent Touquet wrote:
> By the way, I'm having the same problem as described in here too:
> http://www.cs.helsinki.fi/linux/linux-kernel/2003-21/0619.html
>
> Perhaps time we took another look at the AMD 74xx ide code ?
> Or is this particular bug considered harmless ?
Most likely caused by the slave devices confusing the BIOS cable
detection. The amd74xx driver can only use what the BIOS tells it. You
can use 'ide0=ata66' to override the cable detection.
--
Vojtech Pavlik
SuSE Labs, SuSE CR
On Tue, Jul 08, 2003 at 06:41:32PM +0200, Vojtech Pavlik wrote:
>Most likely caused by the slave devices confusing the BIOS cable
>detection. The amd74xx driver can only use what the BIOS tells it. You
>can use 'ide0=ata66' to override the cable detection.
Thanks, that should solve it.
Any idea on what could cause the lockups of my system ?
Some output of vmstat near the hangs, and also stack traces can be found
in this thread: http://marc.theaimsgroup.com/?t=105752570500001&r=1&w=2
best regards,
Vincent
Another way to lockup the system:
dd if=/dev/zero of=/array/file bs=1024k count=10000
So now I didn't even use any code that came near ide
(unless you take into account swapping ?)
The process ends again in a hangup:
Jul 8 22:54:55 kalimero kernel: 3w-xxxx: scsi0: AEN drain failed,
retrying.
Jul 8 22:54:55 kalimero kernel: 3w-xxxx: scsi0: Controller errors, card
not responding, check all cabling.
Jul 8 22:54:55 kalimero kernel: 3w-xxxx: scsi0: Reset sequence failed.
Jul 8 22:54:55 kalimero kernel: 3w-xxxx: scsi0: Unit #0: Command
(f7c1cc00) timed out, resetting card.
Some interesting bits in the traces show the scsi being in a limbo:
Jul 8 22:54:55 kalimero kernel: kupdated D 00000046 5052 7
1 8 6 (L-TLB)
Jul 8 22:54:55 kalimero kernel: Call Trace:
[call_reschedule_interrupt+5/11] [__down+192/352] [__down_failed+11/20]
[.text.lock.super+279/518] [sync_old_buffers+102/336]
Jul 8 22:54:55 kalimero kernel: [kupdate+418/480] [kupdate+0/480]
[arch_kernel_thread+46/64] [kupdate+0/480]
Jul 8 22:54:55 kalimero kernel: scsi_eh_0 R F7C64080 5760 8
1 9 7 (L-TLB)
Jul 8 22:54:55 kalimero kernel: Call Trace:
[tw_scsi_eh_abort+504/768] [scsi_try_to_abort_command+136/208]
[__down_interruptible+373/416]
[scsi_unjam_host+2045/2672] [scsi_error_handler+376/608]
Jul 8 22:54:55 kalimero kernel: [arch_kernel_thread+46/64]
[scsi_error_handler+0/608]
And of course the dd process is in state 'D'...
I should start browsing the sources for these scsi_* functions.
I would really like to know if I'm looking at a software or a hardware
issue here.
best regards,
Vincent