Linux incandescent 2.4.18-pre7-ac3 #1 Tue Feb 5 01:00:50 EST 2002 i686 unknown
I came home today to a bunch of processes apparently hung in jbd's
do_get_write_access(); apparently, something deadlocked, where no
processes could write to one of my partition's journal. That's my
theory, anyways. Several processes had stacks similar to:
Call Trace: [<c011271d>] [<d08ecac7>] [<d08ecd08>] [<d08fb892>]
[<d08ec262>]
[<d08fb92a>] [<d08fb9d7>] [<c014367e>] [<c01260c0>] [<c01a452e>]
[<d08f7916>] [<c0131286>] [<c01016c4b>]
Warning (Oops_read): Code line not seen, dumping what data is available
Trace; c011271c <sleep_on+3c/50>
Trace; d08ecac6 <[jbd]do_get_write_access+2e6/4f0>
Trace; d08ecd08 <[jbd]journal_get_write_access+38/60>
Trace; d08fb892 <[ext3]ext3_reserve_inode_write+32/b0>
Trace; d08ec262 <[jbd]__kstrtab_journal_update_format+2/20>
Trace; d08fb92a <[ext3]ext3_mark_inode_dirty+1a/40>
Trace; d08fb9d6 <[ext3]ext3_dirty_inode+86/d0>
Trace; c014367e <igrab+2e/60>
Trace; c01260c0 <generic_file_write+330/750>
Trace; c01a452e <sock_read+8e/a0>
Trace; d08f7916 <[ext3]ext3_file_write+46/50>
Trace; c0131286 <sys_write+96/d0>
Trace; 0000000c01016c4a <END_OF_CODE+b3068a72c/????>
And, kjournald had the following:
Call Trace: [<c0131f0a>] [<d08ee6e6>] [<c0112380>] [<d08f08eb>]
[<d08f07c0>]
[<c010f4e8>]
Warning (Oops_read): Code line not seen, dumping what data is available
Trace; c0131f0a <__wait_on_buffer+6a/90>
Trace; d08ee6e6 <[jbd]journal_commit_transaction+776/dd6>
Trace; c0112380 <schedule+2c0/2f0>
Trace; d08f08ea <[jbd]kjournald+10a/1a0>
Trace; d08f07c0 <[jbd]commit_timeout+0/10>
Trace; c010f4e8 <mtrr_read+68/80>
Not sure what caused the actual deadlock; the partition uses lvm and
ext3, and the underlying controller uses hedrick's pdc driver.
Machine info:
00:00.0 Host bridge: Advanced Micro Devices [AMD] AMD-751 [Irongate] System Controller (rev 25)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ >SERR- <PERR-
Latency: 64
Region 0: Memory at e0000000 (32-bit, prefetchable) [size=64M]
Region 1: Memory at e7dff000 (32-bit, prefetchable) [size=4K]
Region 2: I/O ports at c800 [disabled] [size=4]
Capabilities: <available only to root>
00:01.0 PCI bridge: Advanced Micro Devices [AMD] AMD-751 [Irongate] AGP Bridge (rev 01) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap- 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 64
Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
I/O behind bridge: 00008000-00008fff
Memory behind bridge: e7e00000-e7efffff
Prefetchable memory behind bridge: dfc00000-dfcfffff
BridgeCtl: Parity- SERR+ NoISA+ VGA- MAbort- >Reset- FastB2B-
00:07.0 ISA bridge: Advanced Micro Devices [AMD] AMD-756 [Viper] ISA (rev 01)
Subsystem: Advanced Micro Devices [AMD] AMD-756 [Viper] ISA
Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
00:07.1 IDE interface: Advanced Micro Devices [AMD] AMD-756 [Viper] IDE (rev 03) (prog-if 8a [Master SecP PriP])
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32
Region 4: I/O ports at f000 [size=16]
00:07.3 Bridge: Advanced Micro Devices [AMD] AMD-756 [Viper] ACPI (rev 03)
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
00:07.4 USB Controller: Advanced Micro Devices [AMD] AMD-756 [Viper] USB (rev 06) (prog-if 10 [OHCI])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 16 (20000ns max), cache line size 08
Interrupt: pin D routed to IRQ 5
Region 0: Memory at ebfbf000 (32-bit, non-prefetchable) [size=4K]
00:08.0 Multimedia audio controller: Ensoniq ES1370 [AudioPCI]
Subsystem: Unknown device 4942:4c4c
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=slow >TAbort- <TAbort+ <MAbort+ >SERR- <PERR-
Latency: 64 (3000ns min, 32000ns max)
Interrupt: pin A routed to IRQ 5
Region 0: I/O ports at c400 [size=64]
00:09.0 VGA compatible controller: S3 Inc. ViRGE/DX or /GX (rev 01) (prog-if 00 [VGA])
Subsystem: S3 Inc. ViRGE/DX
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 64 (1000ns min, 63750ns max)
Interrupt: pin A routed to IRQ 11
Region 0: Memory at ec000000 (32-bit, non-prefetchable) [size=64M]
Expansion ROM at ebff0000 [disabled] [size=64K]
00:0b.0 Unknown mass storage controller: Promise Technology, Inc. 20262 (rev 01)
Subsystem: Promise Technology, Inc.: Unknown device 4d33
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 64
Interrupt: pin A routed to IRQ 10
Region 0: I/O ports at dc00 [size=8]
Region 1: I/O ports at d800 [size=4]
Region 2: I/O ports at d400 [size=8]
Region 3: I/O ports at d000 [size=4]
Region 4: I/O ports at cc00 [size=64]
Region 5: Memory at ebfc0000 (32-bit, non-prefetchable) [size=128K]
Expansion ROM at ebfe0000 [disabled] [size=64K]
00:0c.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8139 (rev 10)
Subsystem: D-Link System Inc DFE-538TX
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 64 (8000ns min, 16000ns max)
Interrupt: pin A routed to IRQ 5
Region 0: I/O ports at c000 [size=256]
Region 1: Memory at ebfbef00 (32-bit, non-prefetchable) [size=256]
Capabilities: <available only to root>
Module Size Used by Tainted: P
ipt_MASQUERADE 1312 1 (autoclean)
iptable_nat 14420 1 (autoclean) [ipt_MASQUERADE]
ip_conntrack 14092 1 (autoclean) [ipt_MASQUERADE iptable_nat]
ip_tables 10752 4 [ipt_MASQUERADE iptable_nat]
lvm-mod 47712 3
8139too 13536 1 (autoclean)
mii 1120 0 (autoclean) [8139too]
3c515 13468 1 (autoclean)
ext3 56992 1 (autoclean)
jbd 35848 1 (autoclean) [ext3]
ide-scsi 7680 0
sr_mod 11832 0 (unused)
cdrom 29248 0 [sr_mod]
scsi_mod 50524 2 [ide-scsi sr_mod]
es1370 25936 1
soundcore 3684 4 [es1370]
partition in question:
/dev/site/lvol1
ext3 297364216 165087832 132276384 56% /mnt/site
Linux version 2.4.18-pre7-ac3 (dilinger@incandescent) (gcc version 2.95.4 (Debian prerelease)) #1 Tue Feb 5 01:00:50 EST 2002
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000dc000 - 00000000000e0000 (reserved)
BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000000fff0000 (usable)
BIOS-e820: 000000000fff0000 - 000000000fff8000 (ACPI data)
BIOS-e820: 000000000fff8000 - 0000000010000000 (ACPI NVS)
BIOS-e820: 00000000ffff0000 - 0000000100000000 (reserved)
On node 0 totalpages: 65520
zone(0): 4096 pages.
zone(1): 61424 pages.
zone(2): 0 pages.
Local APIC disabled by BIOS -- reenabling.
Found and enabled local APIC!
Kernel command line: root=/dev/hda2 devfs=mount mem=262080K
Initializing CPU#0
Detected 704.939 MHz processor.
Console: colour VGA+ 80x25
Calibrating delay loop... 1405.74 BogoMIPS
Memory: 256776k/262080k available (911k kernel code, 4916k reserved, 229k data, 208k init, 0k highmem)
Dentry-cache hash table entries: 32768 (order: 6, 262144 bytes)
Inode-cache hash table entries: 16384 (order: 5, 131072 bytes)
Mount-cache hash table entries: 4096 (order: 3, 32768 bytes)
Buffer-cache hash table entries: 16384 (order: 4, 65536 bytes)
Page-cache hash table entries: 65536 (order: 6, 262144 bytes)
CPU: Before vendor init, caps: 0183fbff c1c3fbff 00000000, vendor = 2
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 512K (64 bytes/line)
CPU: After vendor init, caps: 0183fbff c1c3fbff 00000000 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU: After generic, caps: 0183fbff c1c3fbff 00000000 00000000
CPU: Common caps: 0183fbff c1c3fbff 00000000 00000000
CPU: AMD Athlon(tm) Processor stepping 01
Enabling fast FPU save and restore... done.
Checking 'hlt' instruction... OK.
POSIX conformance testing by UNIFIX
enabled ExtINT on CPU#0
ESR value before enabling vector: 00000000
ESR value after enabling vector: 00000000
Using local APIC timer interrupts.
calibrating APIC timer ...
..... CPU clock speed is 704.9229 MHz.
..... host bus clock speed is 201.4065 MHz.
cpu: 0, clocks: 2014065, slice: 1007032
CPU0<T0:2014064,T1:1007024,D:8,S:1007032,C:2014065>
mtrr: v1.40 (20010327) Richard Gooch ([email protected])
mtrr: detected mtrr type: Intel
PCI: PCI BIOS revision 2.10 entry at 0xfdaf1, last bus=1
PCI: Using configuration type 1
PCI: Probing PCI hardware
PCI: Using IRQ router AMD756 VIPER [1022/740b] at 00:07.3
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Initializing RT netlink socket
Starting kswapd
devfs: v1.10 (20020120) Richard Gooch ([email protected])
devfs: boot_options: 0x1
pty: 256 Unix98 ptys configured
Real Time Clock Driver v1.10e
block: 128 slots per queue, batch=32
Uniform Multi-Platform E-IDE driver Revision: 6.31
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
AMD7409: IDE controller on PCI bus 00 dev 39
AMD7409: chipset revision 3
AMD7409: not 100% native mode: will probe irqs later
AMD7409: disabling single-word DMA support (revision < C4)
ide0: BM-DMA at 0xf000-0xf007, BIOS settings: hda:DMA, hdb:DMA
ide1: BM-DMA at 0xf008-0xf00f, BIOS settings: hdc:DMA, hdd:DMA
PDC20262: IDE controller on PCI bus 00 dev 58
AMD756: dev 105a:4d38, router pirq : 3 get irq : 10
PCI: Found IRQ 10 for device 00:0b.0
PDC20262: chipset revision 1
PDC20262: not 100% native mode: will probe irqs later
PDC20262: ROM enabled at 0xebfe0000
PDC20262: (U)DMA Burst Bit ENABLED Primary PCI Mode Secondary PCI Mode.
ide2: BM-DMA at 0xcc00-0xcc07, BIOS settings: hde:DMA, hdf:DMA
ide3: BM-DMA at 0xcc08-0xcc0f, BIOS settings: hdg:DMA, hdh:pio
hda: IBM-DJNA-352030, ATA DISK drive
hdb: Maxtor 92041U4, ATA DISK drive
hdc: Maxtor 54098H8, ATA DISK drive
hdd: WDC WD450AA-00BAA0, ATA DISK drive
hde: IC35L060AVER07-0, ATA DISK drive
hdf: IC35L060AVER07-0, ATA DISK drive
hdg: WDC WD800AB-00BTA0, ATA DISK drive
hdh: SONY CD-RW CRX140E, ATAPI CD/DVD-ROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
ide2 at 0xdc00-0xdc07,0xd802 on irq 10
ide3 at 0xd400-0xd407,0xd002 on irq 10
hda: 39876480 sectors (20417 MB) w/1966KiB Cache, CHS=2482/255/63, UDMA(33)
hdb: 40020624 sectors (20491 MB) w/512KiB Cache, CHS=2491/255/63, UDMA(33)
hdc: 80041248 sectors (40981 MB) w/2048KiB Cache, CHS=79406/16/63, UDMA(33)
hdd: 87930864 sectors (45021 MB) w/2048KiB Cache, CHS=87233/16/63, UDMA(33)
hde: 120103200 sectors (61493 MB) w/1916KiB Cache, CHS=119150/16/63, UDMA(33)
hdf: 120103200 sectors (61493 MB) w/1916KiB Cache, CHS=119150/16/63, UDMA(33)
hdg: 156301488 sectors (80026 MB) w/2048KiB Cache, CHS=155061/16/63, UDMA(66)
Partition check:
/dev/ide/host0/bus0/target0/lun0: p1 p2
/dev/ide/host0/bus0/target1/lun0: p1
/dev/ide/host0/bus1/target0/lun0: p1
/dev/ide/host0/bus1/target1/lun0: p1
/dev/ide/host2/bus0/target0/lun0: [PTBL] [7476/255/63] p1
/dev/ide/host2/bus0/target1/lun0: p1
/dev/ide/host2/bus1/target0/lun0: p1
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP
IP: routing cache hash table of 2048 buckets, 16Kbytes
TCP: Hash tables configured (established 16384 bind 16384)
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
reiserfs: checking transaction log (device 03:02) ...
Warning, log replay starting on readonly filesystem
reiserfs: replayed 63 transactions in 10 seconds
Using r5 hash to sort names
ReiserFS version 3.6.25
VFS: Mounted root (reiserfs filesystem) readonly.
Mounted devfs on /dev
Freeing unused kernel memory: 208k freed
Adding Swap: 136512k swap-space (priority -1)
es1370: version v0.37 time 01:07:48 Feb 5 2002
AMD756: dev 1274:5000, router pirq : 4 get irq : 5
PCI: Found IRQ 5 for device 00:08.0
PCI: Sharing IRQ 5 with 00:07.4
PCI: Sharing IRQ 5 with 00:0c.0
es1370: found adapter at io 0xc400 irq 5
es1370: features: joystick off, line in, mic impedance 0
SCSI subsystem driver Revision: 1.00
scsi0 : SCSI host adapter emulation for IDE ATAPI devices
Vendor: SONY Model: CD-RW CRX140E Rev: 1.0n
Type: CD-ROM ANSI SCSI revision: 02
Attached scsi CD-ROM sr0 at scsi0, channel 0, id 0, lun 0
sr0: scsi3-mmc drive: 20x/32x writer cd/rw xa/form2 cdda tray
Uniform CD-ROM driver Revision: 3.12
Journalled Block Device driver loaded
3c515.c:v0.99t 17-Nov-2001 [email protected] and others
3c515 Resource configuration register 0x00a9, DCR 1485.
eth0: 3Com 3c515 at 0x2a0, 00:10:4b:d4:40:27, DMA 5, IRQ 9
64K word-wide RAM 1:1 Rx:Tx split, autoselect/10baseT interface.
1 3c515 cards found.
8139too Fast Ethernet driver 0.9.23
AMD756: dev 10ec:8139, router pirq : 4 get irq : 5
PCI: Found IRQ 5 for device 00:0c.0
PCI: Sharing IRQ 5 with 00:07.4
PCI: Sharing IRQ 5 with 00:08.0
eth1: RealTek RTL8139 Fast Ethernet at 0xd0911f00, 00:50:ba:d8:1a:16, IRQ 5
eth1: Identified 8139 chip type 'RTL-8139B'
eth1: Setting half-duplex based on auto-negotiated partner ability 0000.
spurious 8259A interrupt: IRQ7.
LVM version 1.0.1-rc4(ish)(03/10/2001) module loaded
kjournald starting. Commit interval 5 seconds
EXT3 FS 2.4-0.9.17, 10 Jan 2002 on lvm(58,0), internal journal
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
ip_tables: (C) 2000-2002 Netfilter core team
ip_conntrack (2047 buckets, 16376 max)
I'm sure it'll happen again, so if anyone has any ideas about what may
be causing this, or what to look for the next time this happens..
--
"I think a lot of the basis of the open source movement comes from
procrastinating students..."
-- Andrew Tridgell <http://www.linux-mag.com/2001-07/tridgell_04.html>
Andres Salomon wrote:
>
> Linux incandescent 2.4.18-pre7-ac3 #1 Tue Feb 5 01:00:50 EST 2002 i686 unknown
>
> I came home today to a bunch of processes apparently hung in jbd's
> do_get_write_access(); apparently, something deadlocked, where no
> processes could write to one of my partition's journal. That's my
> theory, anyways. Several processes had stacks similar to:
>
> ...
>
> And, kjournald had the following:
>
> Call Trace: [<c0131f0a>] [<d08ee6e6>] [<c0112380>] [<d08f08eb>]
> [<d08f07c0>]
> [<c010f4e8>]
> Warning (Oops_read): Code line not seen, dumping what data is available
>
> Trace; c0131f0a <__wait_on_buffer+6a/90>
> Trace; d08ee6e6 <[jbd]journal_commit_transaction+776/dd6>
> Trace; c0112380 <schedule+2c0/2f0>
> Trace; d08f08ea <[jbd]kjournald+10a/1a0>
> Trace; d08f07c0 <[jbd]commit_timeout+0/10>
This is the important part - we sent a write request down the
stack and it seems that I/O completion was never signalled.
You should check for any odd messages from the device driver
or LVM layers, if the logs made it to disk.
-
Hi,
On Thu, Feb 07, 2002 at 04:02:37AM -0500, Andres Salomon wrote:
> Linux incandescent 2.4.18-pre7-ac3 #1 Tue Feb 5 01:00:50 EST 2002 i686 unknown
>
> I came home today to a bunch of processes apparently hung in jbd's
> do_get_write_access(); apparently, something deadlocked, where no
> processes could write to one of my partition's journal. That's my
> theory, anyways. Several processes had stacks similar to:
> Trace; c011271c <sleep_on+3c/50>
> Trace; d08ecac6 <[jbd]do_get_write_access+2e6/4f0>
They are blocked waiting on the journal thread, and:
> And, kjournald had the following:
>
> Call Trace: [<c0131f0a>] [<d08ee6e6>] [<c0112380>] [<d08f08eb>]
> [<d08f07c0>]
> [<c010f4e8>]
> Warning (Oops_read): Code line not seen, dumping what data is available
>
> Trace; c0131f0a <__wait_on_buffer+6a/90>
... the journal thread is waiting for some IO to complete.
> Not sure what caused the actual deadlock; the partition uses lvm and
> ext3, and the underlying controller uses hedrick's pdc driver.
Were you doing anything particular on the LVM at the time? Is there
anything in the log which might indicate a failed IO? The trace above
just indicates that IO was initialised and never completed --- it
doesn't really give us any clue WHY that IO was lost.
Cheers,
Stephen
No messages at all, actually. I noticed my sysrq-t made it into my logs
(although it doesn't look all too correct..), so I've included it
below. The D-state processes themselves seem to be the only indication of
something gone wrong.
Feb 7 02:30:01 incandescent kernel: SysRq : Show State
Feb 7 02:30:01 incandescent kernel:
Feb 7 02:30:01 incandescent kernel: free sibling
Feb 7 02:30:01 incandescent kernel: task PC stack pid father child younger older
Feb 7 02:30:01 incandescent kernel: init S CFFE7F2C 4628 1 0 26034 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: keventd S 00010000 5684 2 1 3 (L-TLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [context_thread+251/416] [kernel_thread+40/64]
Feb 7 02:30:01 incandescent kernel: ksoftirqd_CPU S C1376000 5368 3 1 4 2 (L-TLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [ksoftirqd+114/176] [kernel_thread+40/64]
Feb 7 02:30:01 incandescent kernel: kswapd S C1375F88 5092 4 1 5 3 (L-TLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [interruptible_sleep_on_timeout+66/96] [wakeup_memwaiters+121/256] [kswapd+707/720]
Feb 7 02:30:01 incandescent kernel: [kernel_thread+40/64]
Feb 7 02:30:01 incandescent kernel: bdflush S 00000286 5436 5 1 6 4 (L-TLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [interruptible_sleep_on+61/80] [bdflush+149/160] [kernel_thread+40/64]
Feb 7 02:30:01 incandescent kernel: kupdated S C1371FC8 4772 6 1 7 5 (L-TLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [kupdate+130/272] [kernel_thread+40/64]
Feb 7 02:30:01 incandescent kernel: kreiserfsd S CFDDBFB4 5544 7 1 20 6 (L-TLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [interruptible_sleep_on_timeout+66/96] [reiserfs_journal_commit_thread+149/208] [kernel_thread+40/64]
Feb 7 02:30:01 incandescent kernel: devfsd S CF9D8000 5928 20 1 109 7 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [devfsd_read+248/944] [sys_read+150/208] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: eth1 S CF5B3F98 0 109 1 151 20 (L-TLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [interruptible_sleep_on_timeout+66/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-520802/96] [kernel_thread+40/64]
Feb 7 02:30:01 incandescent kernel: kjournald D CF47BE74 8 151 1 159 109 (L-TLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [__wait_on_buffer+106/144] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-645402/96] [schedule+704/752] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-636693/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-636992/96]
Feb 7 02:30:01 incandescent kernel: [kernel_thread+40/64]
Feb 7 02:30:01 incandescent kernel: httpd S CEFA1F2C 4560 159 1 1968 172 151 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [do_select+417/480] [kfree+146/160] [sys_select+810/1136]
Feb 7 02:30:01 incandescent kernel: [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: syslogd D CEF1DE6C 3896 172 1 175 159 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [__wait_on_buffer+106/144] [_update_journal_header_block+190/240] [update_journal_header_block+21/48] [flush_journal_list+805/912] [do_journal_end+2440/2704]
Feb 7 02:30:01 incandescent kernel: [journal_end_sync+68/80] [reiserfs_commit_for_inode+82/128] [reiserfs_sync_file+54/80] [sys_fsync+99/176] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: klogd R CEED8000 4828 175 1 181 172 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [do_syslog+193/736] [kmsg_read+17/32] [sys_read+150/208] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: exim S 7FFFFFFF 2672 181 1 205 175 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: inetd S CEF29FB0 0 205 1 208 181 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [sys_rt_sigsuspend+230/256] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: ippl S CEBE9F88 428 208 1 210 226 205 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [sys_nanosleep+278/496] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: ippl S CEC21F28 0 210 208 211 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [do_poll+188/224] [sys_poll+477/752] [end_8259A_irq+24/32]
Feb 7 02:30:01 incandescent kernel: [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: ippl S 7FFFFFFF 2676 211 210 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+23/160] [wait_for_packet+210/320] [skb_recv_datagram+174/208] [raw_recvmsg+117/336] [inet_recvmsg+61/96]
Feb 7 02:30:01 incandescent kernel: [sock_recvmsg+61/192] [sock_read+142/160] [sys_read+150/208] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: oidentd S 7FFFFFFF 0 226 1 230 208 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+23/160] [wait_for_connect+224/384] [tcp_accept+129/416] [inet_accept+48/304] [sys_accept+102/256]
Feb 7 02:30:01 incandescent kernel: [do_page_fault+0/1172] [sys_rt_sigaction+152/320] [sys_socketcall+179/512] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: nmbd S CEAADF2C 4828 230 1 232 226 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: smbd S 7FFFFFFF 5236 232 1 26861 238 230 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: sshd S 7FFFFFFF 2672 238 1 4730 243 232 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: cron S CE82DF88 2672 243 1 246 238 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [sys_nanosleep+278/496] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: getty S 7FFFFFFF 2256 246 1 2055 243 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+23/160] [read_chan+941/1792] [tty_read+184/224] [sys_read+150/208] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: sshd S 7FFFFFFF 0 308 238 313 12543 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [normal_poll+259/287] [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: bash S 00000000 0 313 308 1889 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [sys_wait4+862/912] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: smbd S C8C15F2C 16 26858 232 26859 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: smbd S C9145F2C 3536 26859 232 26861 26858 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: smbd S CA687F2C 0 26861 232 26859 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: screen S 7FFFFFFF 2672 2055 1 9884 27389 246 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: bash S 00000000 660 2056 2055 7457 2772 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [sys_wait4+862/912] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: screen S 7FFFFFFF 584 27389 1 3561 26034 2055 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [normal_poll+259/287] [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: bash S 00000000 2672 27391 27389 14574 14718 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [sys_wait4+862/912] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: bash S 00000000 5444 2772 2055 3065 3239 2056 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [sys_wait4+862/912] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: ssh S 7FFFFFFF 0 3065 2772 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: bash S 7FFFFFFF 0 3239 2055 3964 2772 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+23/160] [read_chan+941/1792] [tty_read+184/224] [sys_read+150/208] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: bash S 00000000 660 3964 2055 5543 9884 3239 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [sys_wait4+862/912] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: lftp S C0BF1F28 2144 5543 3964 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [do_poll+188/224] [sys_poll+477/752] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: mutt S CDC5DF28 2676 7457 2056 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [do_poll+188/224] [sys_poll+477/752] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: bash S 00000000 2676 9884 2055 9517 3964 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [sys_wait4+862/912] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: sshd S 7FFFFFFF 0 12543 238 12556 31886 308 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: bash S 00000000 0 12556 12543 12590 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [sys_wait4+862/912] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: screen S C8A08000 660 12590 12556 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [sys_pause+18/24] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: ncftp D 00000282 0 14574 27391 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [sleep_on+61/80] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-652601/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-652024/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-591726/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-654750/96]
Feb 7 02:30:01 incandescent kernel: [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-591574/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-591401/96] [__mark_inode_dirty+46/128] [generic_file_write+816/1872] [sock_read+142/160] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-607978/96]
Feb 7 02:30:01 incandescent kernel: [sys_write+150/208] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: bash D C0079EB0 0 14718 27389 3494 27391 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [__wait_on_buffer+106/144] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-599850/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-608890/96] [do_page_fault+355/1172] [do_page_fault+0/1172]
Feb 7 02:30:01 incandescent kernel: [vfs_readdir+97/144] [filldir64+0/368] [sys_getdents64+79/259] [filldir64+0/368] [sys_fcntl64+128/144] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: httpd S 7FFFFFFF 16 25786 159 25787 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: httpd S C801A000 0 25787 159 25788 25786 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [sys_semop+907/1184] [net_tx_action+141/160] [do_softirq+90/176] [ip_queue_xmit2+0/523] [_text_lock_netfilter+182/219]
Feb 7 02:30:01 incandescent kernel: [skb_release_data+103/112] [kfree_skbmem+11/96] [tcp_recvmsg+1739/2032] [tcp_v4_destroy_sock+20/352] [sk_free+57/64] [tcp_close+1409/1424]
Feb 7 02:30:01 incandescent kernel: [__free_pages+27/32] [free_pages+26/32] [release_task+295/336] [sys_wait4+899/912] [sys_ipc+64/640] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: httpd S C2112000 0 25788 159 25789 25787 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [sys_semop+907/1184] [net_tx_action+141/160] [do_softirq+90/176] [ip_queue_xmit2+0/523] [_text_lock_netfilter+182/219]
Feb 7 02:30:01 incandescent kernel: [skb_release_data+103/112] [kfree_skbmem+11/96] [tcp_recvmsg+1739/2032] [tcp_v4_destroy_sock+20/352] [sk_free+57/64] [tcp_close+1409/1424]
Feb 7 02:30:01 incandescent kernel: [__free_pages+27/32] [free_pages+26/32] [release_task+295/336] [sys_wait4+899/912] [sys_ipc+64/640] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: httpd S CF5A0000 32 25789 159 25790 25788 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [sys_semop+907/1184] [net_tx_action+141/160] [do_softirq+90/176] [ip_queue_xmit2+0/523] [_text_lock_netfilter+182/219]
Feb 7 02:30:01 incandescent kernel: [skb_release_data+103/112] [kfree_skbmem+11/96] [tcp_recvmsg+1739/2032] [tcp_v4_destroy_sock+20/352] [sk_free+57/64] [tcp_close+1409/1424]
Feb 7 02:30:01 incandescent kernel: [__free_pages+27/32] [free_pages+26/32] [release_task+295/336] [sys_wait4+899/912] [sys_ipc+64/640] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: httpd S C7B96000 0 25790 159 26038 25789 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [sys_semop+907/1184] [net_tx_action+141/160] [do_softirq+90/176] [ip_queue_xmit2+0/523] [_text_lock_netfilter+182/219]
Feb 7 02:30:01 incandescent kernel: [skb_release_data+103/112] [kfree_skbmem+11/96] [tcp_recvmsg+1739/2032] [tcp_v4_destroy_sock+20/352] [sk_free+57/64] [tcp_close+1409/1424]
Feb 7 02:30:01 incandescent kernel: [__free_pages+27/32] [free_pages+26/32] [release_task+295/336] [sys_wait4+899/912] [sys_ipc+64/640] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: httpd S CE5E2000 0 26038 159 29152 25790 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [sys_semop+907/1184] [net_tx_action+141/160] [do_softirq+90/176] [ip_queue_xmit2+0/523] [_text_lock_netfilter+182/219]
Feb 7 02:30:01 incandescent kernel: [skb_release_data+103/112] [kfree_skbmem+11/96] [tcp_recvmsg+1739/2032] [tcp_v4_destroy_sock+20/352] [sk_free+57/64] [tcp_close+1409/1424]
Feb 7 02:30:01 incandescent kernel: [__free_pages+27/32] [free_pages+26/32] [release_task+295/336] [sys_wait4+899/912] [sys_ipc+64/640] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: httpd S CDFDA000 0 29152 159 29168 26038 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [sys_semop+907/1184] [net_tx_action+141/160] [do_softirq+90/176] [ip_queue_xmit2+0/523] [_text_lock_netfilter+182/219]
Feb 7 02:30:01 incandescent kernel: [skb_release_data+103/112] [kfree_skbmem+11/96] [tcp_recvmsg+1739/2032] [tcp_v4_destroy_sock+20/352] [sk_free+57/64] [tcp_close+1409/1424]
Feb 7 02:30:01 incandescent kernel: [__free_pages+27/32] [free_pages+26/32] [release_task+295/336] [sys_wait4+899/912] [sys_ipc+64/640] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: httpd S C298E000 4 29168 159 29169 29152 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [sys_semop+907/1184] [net_tx_action+141/160] [do_softirq+90/176] [ip_queue_xmit2+0/523] [_text_lock_netfilter+182/219]
Feb 7 02:30:01 incandescent kernel: [skb_release_data+103/112] [kfree_skbmem+11/96] [tcp_recvmsg+1739/2032] [tcp_v4_destroy_sock+20/352] [sk_free+57/64] [tcp_close+1409/1424]
Feb 7 02:30:01 incandescent kernel: [__free_pages+27/32] [free_pages+26/32] [release_task+295/336] [sys_wait4+899/912] [sys_ipc+64/640] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: httpd S C298C000 0 29169 159 1968 29168 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [sys_semop+907/1184] [net_tx_action+141/160] [do_softirq+90/176] [ip_queue_xmit2+0/523] [_text_lock_netfilter+182/219]
Feb 7 02:30:01 incandescent kernel: [skb_release_data+103/112] [kfree_skbmem+11/96] [tcp_recvmsg+1739/2032] [tcp_v4_destroy_sock+20/352] [sk_free+57/64] [tcp_close+1409/1424]
Feb 7 02:30:01 incandescent kernel: [__free_pages+27/32] [free_pages+26/32] [release_task+295/336] [sys_wait4+899/912] [sys_ipc+64/640] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: httpd S CE53A000 2672 1968 159 29169 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [sys_semop+907/1184] [net_tx_action+141/160] [do_softirq+90/176] [ip_queue_xmit2+0/523] [_text_lock_netfilter+182/219]
Feb 7 02:30:01 incandescent kernel: [skb_release_data+103/112] [kfree_skbmem+11/96] [tcp_recvmsg+1739/2032] [tcp_v4_destroy_sock+20/352] [sk_free+57/64] [tcp_close+1409/1424]
Feb 7 02:30:01 incandescent kernel: [__free_pages+27/32] [free_pages+26/32] [release_task+295/336] [sys_wait4+899/912] [sys_ipc+64/640] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: mutt S CE789F28 0 9517 9884 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [do_poll+188/224] [sys_poll+477/752] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: sshd S 7FFFFFFF 996 31886 238 31893 4730 12543 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: rsync S CA775F2C 1268 31893 31886 31894 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: rsync S C81FFF2C 16 31894 31893 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: scp D 00000282 0 26034 1 27389 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [sleep_on+61/80] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-652601/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-652024/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-591726/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-654750/96]
Feb 7 02:30:01 incandescent kernel: [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-591574/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-591401/96] [__mark_inode_dirty+46/128] [update_atime+75/80] [do_generic_file_read+1117/1136] [generic_file_read+133/320]
Feb 7 02:30:01 incandescent kernel: [file_read_actor+0/144] [sys_read+150/208] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: ssh S 7FFFFFFF 0 1889 313 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: bash D C72569EC 0 3494 27389 3561 14718 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [__down+84/160] [__down_failed+8/12] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-559040/96] [_text_lock_readdir+5/45] [sys_getdents64+79/259]
Feb 7 02:30:01 incandescent kernel: [filldir64+0/368] [sys_fcntl64+128/144] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: bash S 7FFFFFFF 4804 3561 27389 3494 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+23/160] [read_chan+941/1792] [tty_read+184/224] [sys_read+150/208] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: sshd S 7FFFFFFF 336 4730 238 4734 31886 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: bash S 00000000 4804 4734 4730 5796 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [sys_wait4+862/912] [system_call+51/56]
Feb 7 02:30:01 incandescent kernel: vim S 7FFFFFFF 2672 5796 4734 (NOTLB)
Feb 7 02:30:01 incandescent kernel: Call Trace: [normal_poll+259/287] [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:46 incandescent kernel: SysRq : HELP : loglevel0-8 reBoot tErm kIll saK killalL showMem showPc unRaw Sync showTasks Unmount
Feb 7 02:30:58 incandescent kernel: SysRq : HELP : loglevel0-8 reBoot tErm kIll saK killalL showMem showPc unRaw Sync showTasks Unmount
Feb 7 02:30:59 incandescent kernel: SysRq : Show State
Feb 7 02:30:59 incandescent kernel:
Feb 7 02:30:59 incandescent kernel: free sibling
Feb 7 02:30:59 incandescent kernel: task PC stack pid father child younger older
Feb 7 02:30:59 incandescent kernel: init S CFFE7F2C 4628 1 0 26034 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: keventd S 00010000 5684 2 1 3 (L-TLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [context_thread+251/416] [kernel_thread+40/64]
Feb 7 02:30:59 incandescent kernel: ksoftirqd_CPU S C1376000 5368 3 1 4 2 (L-TLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [ksoftirqd+114/176] [kernel_thread+40/64]
Feb 7 02:30:59 incandescent kernel: kswapd S C1375F88 5092 4 1 5 3 (L-TLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [interruptible_sleep_on_timeout+66/96] [wakeup_memwaiters+121/256] [kswapd+707/720]
Feb 7 02:30:59 incandescent kernel: [kernel_thread+40/64]
Feb 7 02:30:59 incandescent kernel: bdflush S 00000286 5436 5 1 6 4 (L-TLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [interruptible_sleep_on+61/80] [bdflush+149/160] [kernel_thread+40/64]
Feb 7 02:30:59 incandescent kernel: kupdated S C1371FC8 4772 6 1 7 5 (L-TLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [kupdate+130/272] [kernel_thread+40/64]
Feb 7 02:30:59 incandescent kernel: kreiserfsd S CFDDBFB4 5544 7 1 20 6 (L-TLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [interruptible_sleep_on_timeout+66/96] [reiserfs_journal_commit_thread+149/208] [kernel_thread+40/64]
Feb 7 02:30:59 incandescent kernel: devfsd S CF9D8000 5928 20 1 109 7 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [devfsd_read+248/944] [sys_read+150/208] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: eth1 S CF5B3F98 0 109 1 151 20 (L-TLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [interruptible_sleep_on_timeout+66/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-520802/96] [kernel_thread+40/64]
Feb 7 02:30:59 incandescent kernel: kjournald D CF47BE74 8 151 1 159 109 (L-TLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [__wait_on_buffer+106/144] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-645402/96] [schedule+704/752] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-636693/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-636992/96]
Feb 7 02:30:59 incandescent kernel: [kernel_thread+40/64]
Feb 7 02:30:59 incandescent kernel: httpd S CEFA1F2C 4560 159 1 1968 172 151 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [do_select+417/480] [kfree+146/160] [sys_select+810/1136]
Feb 7 02:30:59 incandescent kernel: [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: syslogd S 7FFFFFFF 3896 172 1 175 159 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: klogd R CEED8000 4828 175 1 181 172 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [do_syslog+193/736] [kmsg_read+17/32] [sys_read+150/208] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: exim S 7FFFFFFF 2672 181 1 205 175 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: inetd S CEF29FB0 0 205 1 208 181 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [sys_rt_sigsuspend+230/256] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: ippl S CEBE9F88 428 208 1 210 226 205 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [sys_nanosleep+278/496] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: ippl S CEC21F28 0 210 208 211 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [do_poll+188/224] [sys_poll+477/752] [end_8259A_irq+24/32]
Feb 7 02:30:59 incandescent kernel: [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: ippl S 7FFFFFFF 2676 211 210 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+23/160] [wait_for_packet+210/320] [skb_recv_datagram+174/208] [raw_recvmsg+117/336] [inet_recvmsg+61/96]
Feb 7 02:30:59 incandescent kernel: [sock_recvmsg+61/192] [sock_read+142/160] [sys_read+150/208] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: oidentd S 7FFFFFFF 0 226 1 230 208 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+23/160] [wait_for_connect+224/384] [tcp_accept+129/416] [inet_accept+48/304] [sys_accept+102/256]
Feb 7 02:30:59 incandescent kernel: [do_page_fault+0/1172] [sys_rt_sigaction+152/320] [sys_socketcall+179/512] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: nmbd S CEAADF2C 4828 230 1 232 226 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: smbd S 7FFFFFFF 5236 232 1 26861 238 230 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: sshd S 7FFFFFFF 2672 238 1 4730 243 232 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: cron S CE82DF88 2672 243 1 246 238 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [sys_nanosleep+278/496] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: bash S 7FFFFFFF 2256 246 1 2055 243 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+23/160] [read_chan+941/1792] [tty_read+184/224] [sys_read+150/208] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: sshd S 7FFFFFFF 0 308 238 313 12543 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [normal_poll+259/287] [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: bash S 00000000 0 313 308 1889 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [sys_wait4+862/912] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: smbd S C8C15F2C 16 26858 232 26859 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: smbd S C9145F2C 3536 26859 232 26861 26858 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: smbd S CA687F2C 0 26861 232 26859 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: screen S 7FFFFFFF 2672 2055 1 9884 27389 246 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: bash S 00000000 660 2056 2055 7457 2772 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [sys_wait4+862/912] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: screen S 7FFFFFFF 584 27389 1 3561 26034 2055 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [normal_poll+259/287] [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: bash S 00000000 2672 27391 27389 14574 14718 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [sys_wait4+862/912] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: bash S 00000000 5444 2772 2055 3065 3239 2056 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [sys_wait4+862/912] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: ssh S 7FFFFFFF 0 3065 2772 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: bash S 7FFFFFFF 0 3239 2055 3964 2772 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+23/160] [read_chan+941/1792] [tty_read+184/224] [sys_read+150/208] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: bash S 00000000 660 3964 2055 5543 9884 3239 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [sys_wait4+862/912] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: lftp S C0BF1F28 2144 5543 3964 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [do_poll+188/224] [sys_poll+477/752] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: mutt S CDC5DF28 2676 7457 2056 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [do_poll+188/224] [sys_poll+477/752] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: bash S 00000000 2676 9884 2055 9517 3964 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [sys_wait4+862/912] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: sshd S 7FFFFFFF 0 12543 238 12556 4730 308 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: bash S 00000000 0 12556 12543 12590 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [sys_wait4+862/912] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: screen S C8A08000 660 12590 12556 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [sys_pause+18/24] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: ncftp D 00000282 0 14574 27391 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [sleep_on+61/80] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-652601/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-652024/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-591726/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-654750/96]
Feb 7 02:30:59 incandescent kernel: [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-591574/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-591401/96] [__mark_inode_dirty+46/128] [generic_file_write+816/1872] [sock_read+142/160] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-607978/96]
Feb 7 02:30:59 incandescent kernel: [sys_write+150/208] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: bash D C0079EB0 0 14718 27389 3494 27391 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [__wait_on_buffer+106/144] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-599850/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-608890/96] [do_page_fault+355/1172] [do_page_fault+0/1172]
Feb 7 02:30:59 incandescent kernel: [vfs_readdir+97/144] [filldir64+0/368] [sys_getdents64+79/259] [filldir64+0/368] [sys_fcntl64+128/144] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: httpd S 7FFFFFFF 16 25786 159 25787 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: httpd S C801A000 0 25787 159 25788 25786 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [sys_semop+907/1184] [net_tx_action+141/160] [do_softirq+90/176] [ip_queue_xmit2+0/523] [_text_lock_netfilter+182/219]
Feb 7 02:30:59 incandescent kernel: [skb_release_data+103/112] [kfree_skbmem+11/96] [tcp_recvmsg+1739/2032] [tcp_v4_destroy_sock+20/352] [sk_free+57/64] [tcp_close+1409/1424]
Feb 7 02:30:59 incandescent kernel: [__free_pages+27/32] [free_pages+26/32] [release_task+295/336] [sys_wait4+899/912] [sys_ipc+64/640] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: httpd S C2112000 0 25788 159 25789 25787 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [sys_semop+907/1184] [net_tx_action+141/160] [do_softirq+90/176] [ip_queue_xmit2+0/523] [_text_lock_netfilter+182/219]
Feb 7 02:30:59 incandescent kernel: [skb_release_data+103/112] [kfree_skbmem+11/96] [tcp_recvmsg+1739/2032] [tcp_v4_destroy_sock+20/352] [sk_free+57/64] [tcp_close+1409/1424]
Feb 7 02:30:59 incandescent kernel: [__free_pages+27/32] [free_pages+26/32] [release_task+295/336] [sys_wait4+899/912] [sys_ipc+64/640] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: httpd S CF5A0000 32 25789 159 25790 25788 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [sys_semop+907/1184] [net_tx_action+141/160] [do_softirq+90/176] [ip_queue_xmit2+0/523] [_text_lock_netfilter+182/219]
Feb 7 02:30:59 incandescent kernel: [skb_release_data+103/112] [kfree_skbmem+11/96] [tcp_recvmsg+1739/2032] [tcp_v4_destroy_sock+20/352] [sk_free+57/64] [tcp_close+1409/1424]
Feb 7 02:30:59 incandescent kernel: [__free_pages+27/32] [free_pages+26/32] [release_task+295/336] [sys_wait4+899/912] [sys_ipc+64/640] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: httpd S C7B96000 0 25790 159 26038 25789 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [sys_semop+907/1184] [net_tx_action+141/160] [do_softirq+90/176] [ip_queue_xmit2+0/523] [_text_lock_netfilter+182/219]
Feb 7 02:30:59 incandescent kernel: [skb_release_data+103/112] [kfree_skbmem+11/96] [tcp_recvmsg+1739/2032] [tcp_v4_destroy_sock+20/352] [sk_free+57/64] [tcp_close+1409/1424]
Feb 7 02:30:59 incandescent kernel: [__free_pages+27/32] [free_pages+26/32] [release_task+295/336] [sys_wait4+899/912] [sys_ipc+64/640] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: httpd S CE5E2000 0 26038 159 29152 25790 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [sys_semop+907/1184] [net_tx_action+141/160] [do_softirq+90/176] [ip_queue_xmit2+0/523] [_text_lock_netfilter+182/219]
Feb 7 02:30:59 incandescent kernel: [skb_release_data+103/112] [kfree_skbmem+11/96] [tcp_recvmsg+1739/2032] [tcp_v4_destroy_sock+20/352] [sk_free+57/64] [tcp_close+1409/1424]
Feb 7 02:30:59 incandescent kernel: [__free_pages+27/32] [free_pages+26/32] [release_task+295/336] [sys_wait4+899/912] [sys_ipc+64/640] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: httpd S CDFDA000 0 29152 159 29168 26038 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [sys_semop+907/1184] [net_tx_action+141/160] [do_softirq+90/176] [ip_queue_xmit2+0/523] [_text_lock_netfilter+182/219]
Feb 7 02:30:59 incandescent kernel: [skb_release_data+103/112] [kfree_skbmem+11/96] [tcp_recvmsg+1739/2032] [tcp_v4_destroy_sock+20/352] [sk_free+57/64] [tcp_close+1409/1424]
Feb 7 02:30:59 incandescent kernel: [__free_pages+27/32] [free_pages+26/32] [release_task+295/336] [sys_wait4+899/912] [sys_ipc+64/640] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: httpd S C298E000 4 29168 159 29169 29152 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [sys_semop+907/1184] [net_tx_action+141/160] [do_softirq+90/176] [ip_queue_xmit2+0/523] [_text_lock_netfilter+182/219]
Feb 7 02:30:59 incandescent kernel: [skb_release_data+103/112] [kfree_skbmem+11/96] [tcp_recvmsg+1739/2032] [tcp_v4_destroy_sock+20/352] [sk_free+57/64] [tcp_close+1409/1424]
Feb 7 02:30:59 incandescent kernel: [__free_pages+27/32] [free_pages+26/32] [release_task+295/336] [sys_wait4+899/912] [sys_ipc+64/640] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: httpd S C298C000 0 29169 159 1968 29168 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [sys_semop+907/1184] [net_tx_action+141/160] [do_softirq+90/176] [ip_queue_xmit2+0/523] [_text_lock_netfilter+182/219]
Feb 7 02:30:59 incandescent kernel: [skb_release_data+103/112] [kfree_skbmem+11/96] [tcp_recvmsg+1739/2032] [tcp_v4_destroy_sock+20/352] [sk_free+57/64] [tcp_close+1409/1424]
Feb 7 02:30:59 incandescent kernel: [__free_pages+27/32] [free_pages+26/32] [release_task+295/336] [sys_wait4+899/912] [sys_ipc+64/640] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: httpd S CE53A000 2672 1968 159 29169 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [sys_semop+907/1184] [net_tx_action+141/160] [do_softirq+90/176] [ip_queue_xmit2+0/523] [_text_lock_netfilter+182/219]
Feb 7 02:30:59 incandescent kernel: [skb_release_data+103/112] [kfree_skbmem+11/96] [tcp_recvmsg+1739/2032] [tcp_v4_destroy_sock+20/352] [sk_free+57/64] [tcp_close+1409/1424]
Feb 7 02:30:59 incandescent kernel: [__free_pages+27/32] [free_pages+26/32] [release_task+295/336] [sys_wait4+899/912] [sys_ipc+64/640] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: mutt S CE789F28 0 9517 9884 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+122/160] [process_timeout+0/80] [do_poll+188/224] [sys_poll+477/752] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: scp D 00000282 0 26034 1 27389 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [sleep_on+61/80] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-652601/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-652024/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-591726/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-654750/96]
Feb 7 02:30:59 incandescent kernel: [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-591574/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-591401/96] [__mark_inode_dirty+46/128] [update_atime+75/80] [do_generic_file_read+1117/1136] [generic_file_read+133/320]
Feb 7 02:30:59 incandescent kernel: [file_read_actor+0/144] [sys_read+150/208] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: ssh S 7FFFFFFF 0 1889 313 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: bash D C72569EC 0 3494 27389 3561 14718 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [__down+84/160] [__down_failed+8/12] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-559040/96] [_text_lock_readdir+5/45] [sys_getdents64+79/259]
Feb 7 02:30:59 incandescent kernel: [filldir64+0/368] [sys_fcntl64+128/144] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: bash S 7FFFFFFF 4804 3561 27389 3494 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+23/160] [read_chan+941/1792] [tty_read+184/224] [sys_read+150/208] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: sshd S 7FFFFFFF 336 4730 238 4734 12543 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: bash S 00000000 4804 4734 4730 5796 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [sys_wait4+862/912] [system_call+51/56]
Feb 7 02:30:59 incandescent kernel: vim S 7FFFFFFF 2672 5796 4734 (NOTLB)
Feb 7 02:30:59 incandescent kernel: Call Trace: [normal_poll+259/287] [schedule_timeout+23/160] [do_select+417/480] [sys_select+810/1136] [system_call+51/56]
On Thu, Feb 07, 2002 at 01:23:14AM -0800, Andrew Morton wrote:
[...]
>
> You should check for any odd messages from the device driver
> or LVM layers, if the logs made it to disk.
>
> -
>
>
--
"I think a lot of the basis of the open source movement comes from
procrastinating students..."
-- Andrew Tridgell <http://www.linux-mag.com/2001-07/tridgell_04.html>
Hi,
On Thu, Feb 07, 2002 at 09:53:56AM -0500, Andres Salomon wrote:
> No messages at all, actually. I noticed my sysrq-t made it into my logs
> (although it doesn't look all too correct..), so I've included it
> below. The D-state processes themselves seem to be the only indication of
> something gone wrong.
>From this:
> Feb 7 02:30:01 incandescent kernel: bash D C0079EB0 0 14718 27389 3494 27391 (NOTLB)
> Feb 7 02:30:01 incandescent kernel: Call Trace: [__wait_on_buffer+106/144] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-599850/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-608890/96] [do_page_fault+355/1172] [do_page_fault+0/1172]
> Feb 7 02:30:01 incandescent kernel: [vfs_readdir+97/144] [filldir64+0/368] [sys_getdents64+79/259] [filldir64+0/368] [sys_fcntl64+128/144] [system_call+51/56]
it looks as if there are other processes stalled on IO too.
Unfortunately the modules decoding of the call traces looks broken, so
it's hard from this to really work out what might be the source of
that particular stall.
Is this at all reproducible? Had you been running any LVM utils
recently?
--Stephen
It seems reproducible, but I'm not sure yet. I first saw it (it being
D-state processes similar to what I see now; I didn't have sysrq
compiled into the kernel then) happen when I upgraded to -ac2; at the
time, I was using reiserfs on that partition instead of ext3. I thought
it might be a reiser bug, so I converted it to ext3, and upgraded to -ac3.
The conversion required a lot of LVM util running; I don't remember how
much time lapsed between finishing the conversion, and processes
hanging.
This leads me to believe it's not ext3, but either lvm or pdc202xx. Or
something common to ext3(jbd)/reiserfs..
On Thu, Feb 07, 2002 at 06:29:25PM +0000, Stephen C. Tweedie wrote:
>
> Hi,
>
> On Thu, Feb 07, 2002 at 09:53:56AM -0500, Andres Salomon wrote:
> > No messages at all, actually. I noticed my sysrq-t made it into my logs
> > (although it doesn't look all too correct..), so I've included it
> > below. The D-state processes themselves seem to be the only indication of
> > something gone wrong.
>
> >From this:
>
> > Feb 7 02:30:01 incandescent kernel: bash D C0079EB0 0 14718 27389 3494 27391 (NOTLB)
> > Feb 7 02:30:01 incandescent kernel: Call Trace: [__wait_on_buffer+106/144] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-599850/96] [ipt_MASQUERADE:__insmod_ipt_MASQUERADE_O/lib/modules/2.4.18-pre7-ac3/kerne+-608890/96] [do_page_fault+355/1172] [do_page_fault+0/1172]
> > Feb 7 02:30:01 incandescent kernel: [vfs_readdir+97/144] [filldir64+0/368] [sys_getdents64+79/259] [filldir64+0/368] [sys_fcntl64+128/144] [system_call+51/56]
>
> it looks as if there are other processes stalled on IO too.
> Unfortunately the modules decoding of the call traces looks broken, so
> it's hard from this to really work out what might be the source of
> that particular stall.
>
> Is this at all reproducible? Had you been running any LVM utils
> recently?
>
> --Stephen
>
>
--
"I think a lot of the basis of the open source movement comes from
procrastinating students..."
-- Andrew Tridgell <http://www.linux-mag.com/2001-07/tridgell_04.html>