I think I found a bug introduced in v2.6.18-g23541d2 (git-bisect).
The computer has a dual core amd64 with 1GB ram and 3GB swap space,
64 bit kernel and 32 bit userland.
What I did:
- run the mprime 'torture test' with the memory option set to 1500MB
(ftp://mersenne.org/gimps/mprime235.tar.gz, I ran
echo -e '15\n\n1500\n1500\n\n\n\n17\n\n6\n' | ./mprime)
- mount a tmpfs with 2GB and write to it (stops at 1.0-1.1GB)
What happened:
With mprime normal keyboard input was ignored totally. The machine
still responds to ping/sysrq but killing all processes with sysrq does
not help. Sometimes with tmpfs I was able to type, but ^C did not
work and I was not able to do anything (e.g. get a new shell, start a
program etc.).
The swap space has to be on a dm-crypt and smp has to be enabled,
without this the bug did not occur. Just ask if you need any other
information about the system, output from sysrq debug commands or
anything else.
I found that enabling this options from 'Kernel hacking' helps, even
though it once froze when I tried to quit mprime.
> CONFIG_DEBUG_KERNEL=y
> CONFIG_DEBUG_PREEMPT=y
> CONFIG_DEBUG_SPINLOCK=y
> CONFIG_DEBUG_MUTEXES=y
> CONFIG_DEBUG_RWSEMS=y
> CONFIG_DEBUG_LOCK_ALLOC=y
> CONFIG_PROVE_LOCKING=y
> CONFIG_LOCKDEP=y
> CONFIG_DEBUG_LOCKDEP=y
> CONFIG_TRACE_IRQFLAGS=y
> CONFIG_STACKTRACE=y
> CONFIG_DEBUG_VM=y
While running mprime with this options I got this warning:
> warning: many lost ticks.
> Your time source seems to be instable or some driver is hogging interupts
> rip xor_128+0x2/0x20
kernel boot output:
Linux version 2.6.18-g23541d2d (matled@foo) (gcc version 4.1.2 20061028 (prerelease) (Debian 4.1.1-19)) #1 SMP PREEMPT Sun Nov 26 21:20:27 CET 2006
Command line: root=/dev/md10 ro panic=20 console=tty0 console=ttyS0,115200
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000003eff0000 (usable)
BIOS-e820: 000000003eff0000 - 000000003eff3000 (ACPI NVS)
BIOS-e820: 000000003eff3000 - 000000003f000000 (ACPI data)
BIOS-e820: 000000003f000000 - 0000000040000000 (reserved)
BIOS-e820: 00000000f0000000 - 00000000f2000000 (reserved)
BIOS-e820: 00000000fec00000 - 0000000100000000 (reserved)
end_pfn_map = 1048576
DMI 2.3 present.
Zone PFN ranges:
DMA 0 -> 4096
DMA32 4096 -> 1048576
Normal 1048576 -> 1048576
early_node_map[2] active PFN ranges
0: 0 -> 159
0: 256 -> 258032
Intel MultiProcessor Specification v1.4
MPTABLE: OEM ID: OEM00000 MPTABLE: Product ID: PROD00000000 MPTABLE: APIC at: 0xFEE00000
Processor #0 (Bootup-CPU)
Processor #1
I/O APIC #2 at 0xFEC00000.
Setting APIC routing to flat
Processors: 2
Nosave address range: 000000000009f000 - 00000000000a0000
Nosave address range: 00000000000a0000 - 00000000000f0000
Nosave address range: 00000000000f0000 - 0000000000100000
Allocating PCI resources starting at 50000000 (gap: 40000000:b0000000)
PERCPU: Allocating 21952 bytes of per cpu data
Built 1 zonelists. Total pages: 253687
Kernel command line: root=/dev/md10 ro panic=20 console=tty0 console=ttyS0,115200
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 32768 bytes)
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
Checking aperture...
CPU 0: aperture @ 6e20000000 size 32 MB
Aperture too small (32 MB)
No AGP bridge found
Memory: 1012796k/1032128k available (1826k kernel code, 18644k reserved, 549k data, 200k init)
Calibrating delay using timer specific routine.. 4024.83 BogoMIPS (lpj=8049661)
Mount-cache hash table entries: 256
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 512K (64 bytes/line)
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
Freeing SMP alternatives: 20k freed
ExtINT not setup in hardware but reported by MP table
Using local APIC timer interrupts.
result 12557378
Detected 12.557 MHz APIC timer.
Booting processor 1/2 APIC 0x1
Initializing CPU#1
Calibrating delay using timer specific routine.. 4018.56 BogoMIPS (lpj=8037126)
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 512K (64 bytes/line)
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 1
AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ stepping 02
CPU 1: Syncing TSC to CPU 0.
CPU 1: synchronized TSC with CPU 0 (last diff 0 cycles, maxerr 482 cycles)
Brought up 2 CPUs
testing NMI watchdog ... OK.
time.c: Using 1.193182 MHz WALL PIT GTOD PIT/TSC timer.
time.c: Detected 2009.181 MHz processor.
migration_cost=276
NET: Registered protocol family 16
PCI: Using configuration type 1
SCSI subsystem initialized
PCI: Probing PCI hardware
PCI: Transparent bridge - 0000:00:10.0
PCI: Discovered primary peer bus ff [IRQ]
PCI: Using IRQ router default [10de/02f0] at 0000:00:00.0
PCI->APIC IRQ transform: 0000:00:05.0[A] -> IRQ 11
PCI->APIC IRQ transform: 0000:00:0a.1[A] -> IRQ 5
PCI->APIC IRQ transform: 0000:00:0b.0[A] -> IRQ 11
PCI->APIC IRQ transform: 0000:00:0b.1[B] -> IRQ 10
PCI->APIC IRQ transform: 0000:00:0e.0[A] -> IRQ 11
PCI->APIC IRQ transform: 0000:00:0f.0[A] -> IRQ 10
PCI->APIC IRQ transform: 0000:00:10.1[B] -> IRQ 10
PCI->APIC IRQ transform: 0000:00:14.0[A] -> IRQ 5
PCI->APIC IRQ transform: 0000:01:0e.0[A] -> IRQ 5
PCI-DMA: Disabling IOMMU.
PCI: Bridge: 0000:00:10.0
IO window: disabled.
MEM window: f5000000-f50fffff
PREFETCH window: disabled.
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 6, 262144 bytes)
TCP established hash table entries: 131072 (order: 9, 2097152 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
Linux agpgart interface v0.101 (c) Dave Jones
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
forcedeth.c: Reverse Engineered nForce ethernet driver. Version 0.57.
forcedeth: using HIGHDMA
eth0: forcedeth.c: subsystem: 01458:e000 bound to 0000:00:14.0
ata1: SATA max UDMA/133 cmd 0x9F0 ctl 0xBF2 bmdma 0xD800 irq 11
ata2: SATA max UDMA/133 cmd 0x970 ctl 0xB72 bmdma 0xD808 irq 11
scsi0 : sata_nv
ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata1.00: ATA-7, max UDMA/133, 625140335 sectors: LBA48 NCQ (depth 0/32)
ata1.00: ata1: dev 0 multi count 16
ata1.00: configured for UDMA/133
scsi1 : sata_nv
ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata2.00: ATA-7, max UDMA/133, 625140335 sectors: LBA48 NCQ (depth 0/32)
ata2.00: ata2: dev 0 multi count 16
ata2.00: configured for UDMA/133
scsi 0:0:0:0: Direct-Access ATA ST3320620AS 3.AA PQ: 0 ANSI: 5
SCSI device sda: 625140335 512-byte hdwr sectors (320072 MB)
sda: Write Protect is off
SCSI device sda: drive cache: write back
SCSI device sda: 625140335 512-byte hdwr sectors (320072 MB)
sda: Write Protect is off
SCSI device sda: drive cache: write back
sda: sda1 sda2 sda3 sda4 < sda5 >
sd 0:0:0:0: Attached scsi disk sda
scsi 1:0:0:0: Direct-Access ATA ST3320620AS 3.AA PQ: 0 ANSI: 5
SCSI device sdb: 625140335 512-byte hdwr sectors (320072 MB)
sdb: Write Protect is off
SCSI device sdb: drive cache: write back
SCSI device sdb: 625140335 512-byte hdwr sectors (320072 MB)
sdb: Write Protect is off
SCSI device sdb: drive cache: write back
sdb: sdb1 sdb2 sdb3 sdb4 < sdb5 >
sd 1:0:0:0: Attached scsi disk sdb
ata3: SATA max UDMA/133 cmd 0x9E0 ctl 0xBE2 bmdma 0xC400 irq 10
ata4: SATA max UDMA/133 cmd 0x960 ctl 0xB62 bmdma 0xC408 irq 10
scsi2 : sata_nv
ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata3.00: ATA-7, max UDMA/133, 625142448 sectors: LBA48 NCQ (depth 0/32)
ata3.00: ata3: dev 0 multi count 16
ata3.00: configured for UDMA/133
scsi3 : sata_nv
ata4: SATA link down (SStatus 0 SControl 300)
ATA: abnormal status 0x7F on port 0x967
scsi 2:0:0:0: Direct-Access ATA ST3320620AS 3.AA PQ: 0 ANSI: 5
SCSI device sdc: 625142448 512-byte hdwr sectors (320073 MB)
sdc: Write Protect is off
SCSI device sdc: drive cache: write back
SCSI device sdc: 625142448 512-byte hdwr sectors (320073 MB)
sdc: Write Protect is off
SCSI device sdc: drive cache: write back
sdc: sdc1 sdc2 sdc3 sdc4 < sdc5 >
sd 2:0:0:0: Attached scsi disk sdc
serio: i8042 KBD port at 0x60,0x64 irq 1
md: raid1 personality registered for level 1
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
device-mapper: ioctl: 4.9.0-ioctl (2006-09-14) initialised: [email protected]
TCP cubic registered
NET: Registered protocol family 1
input: AT Translated Set 2 keyboard as /class/input/input0
md: Autodetecting RAID arrays.
md: autorun ...
md: considering sdc3 ...
md: adding sdc3 ...
md: sdc2 has different UUID to sdc3
md: adding sdb3 ...
md: sdb2 has different UUID to sdc3
md: adding sda3 ...
md: sda2 has different UUID to sdc3
md: created md11
md: bind<sda3>
md: bind<sdb3>
md: bind<sdc3>
md: running: <sdc3><sdb3><sda3>
raid1: raid set md11 active with 3 out of 3 mirrors
md: considering sdc2 ...
md: adding sdc2 ...
md: adding sdb2 ...
md: adding sda2 ...
md: created md10
md: bind<sda2>
md: bind<sdb2>
md: bind<sdc2>
md: running: <sdc2><sdb2><sda2>
raid1: raid set md10 active with 3 out of 3 mirrors
md: ... autorun DONE.
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 200k freed
INIT: version 2.86 booting
Matthias Lederhofer wrote:
> I think I found a bug introduced in v2.6.18-g23541d2 (git-bisect).
> The computer has a dual core amd64 with 1GB ram and 3GB swap space,
> 64 bit kernel and 32 bit userland.
[23541d2] dm crypt: move io to workqueue
So this patch is the source of problems ?
> What I did:
> - run the mprime 'torture test' with the memory option set to 1500MB
> (ftp://mersenne.org/gimps/mprime235.tar.gz, I ran
> echo -e '15\n\n1500\n1500\n\n\n\n17\n\n6\n' | ./mprime)
> - mount a tmpfs with 2GB and write to it (stops at 1.0-1.1GB)
> What happened:
> With mprime normal keyboard input was ignored totally. The machine
> still responds to ping/sysrq but killing all processes with sysrq does
> not help. Sometimes with tmpfs I was able to type, but ^C did not
> work and I was not able to do anything (e.g. get a new shell, start a
> program etc.).
>
> The swap space has to be on a dm-crypt and smp has to be enabled,
> without this the bug did not occur. Just ask if you need any other
> information about the system, output from sysrq debug commands or
> anything else.
Please, if you can send task list when system stops responding (sysrq + t)
and output of
dmsetup table
dmsetup info -c
Are there any messages in syslog related to this hang ?
>
> I found that enabling this options from 'Kernel hacking' helps, even
> though it once froze when I tried to quit mprime.
>> CONFIG_DEBUG_KERNEL=y
>> CONFIG_DEBUG_PREEMPT=y
>> CONFIG_DEBUG_SPINLOCK=y
>> CONFIG_DEBUG_MUTEXES=y
>> CONFIG_DEBUG_RWSEMS=y
>> CONFIG_DEBUG_LOCK_ALLOC=y
>> CONFIG_PROVE_LOCKING=y
>> CONFIG_LOCKDEP=y
>> CONFIG_DEBUG_LOCKDEP=y
>> CONFIG_TRACE_IRQFLAGS=y
>> CONFIG_STACKTRACE=y
>> CONFIG_DEBUG_VM=y
> While running mprime with this options I got this warning:
>> warning: many lost ticks.
>> Your time source seems to be instable or some driver is hogging interupts
>> rip xor_128+0x2/0x20
Not sure if this can be related... did you try compile it without config_preempt ?
(and dynamic overclocking in BIOS is disabled ?)
Milan
>
> kernel boot output:
> Linux version 2.6.18-g23541d2d (matled@foo) (gcc version 4.1.2 20061028 (prerelease) (Debian 4.1.1-19)) #1 SMP PREEMPT Sun Nov 26 21:20:27 CET 2006
> Command line: root=/dev/md10 ro panic=20 console=tty0 console=ttyS0,115200
> BIOS-provided physical RAM map:
...
Milan Broz <[email protected]> wrote:
> [23541d2] dm crypt: move io to workqueue
>
> So this patch is the source of problems ?
That's it.
> Please, if you can send task list when system stops responding (sysrq + t)
> and output of
> dmsetup table
> dmsetup info -c
sysrq+t output is at the end of the mail. Is there any way I'm
supposed to post such long stuff? E.g. put it on a webserver, paste
site, in an attachment?
# dmsetup table
swap2: 0 1959867 crypt cbc(aes)-cbc-plain \
c4992e55743a12068f1767e9b855a6388b9bf0c7191ff22823a2cf74ea0dcde8 0 8:37 0
swap1: 0 1959867 crypt cbc(aes)-cbc-plain \
cb922dc385d0d1df09139eabfb32220a968a715428c91708411affbbfecaba08 0 8:21 0
swap0: 0 1959867 crypt cbc(aes)-cbc-plain \
93c36d63556fb9a42308349d18f62a78553445add2424577e39ce1fbf175dcf7 0 8:5 0
# dmsetup info -c
Name Maj Min Stat Open Targ Event UUID
swap2 253 2 L--w 1 1 0
swap1 253 1 L--w 1 1 0
swap0 253 0 L--w 1 1 0
> Are there any messages in syslog related to this hang ?
I can't find anything, there is also nothing on the console with
loglevel set to 9.
> > While running mprime with this options I got this warning:
> >> warning: many lost ticks.
> >> Your time source seems to be instable or some driver is hogging interupts
> >> rip xor_128+0x2/0x20
>
> Not sure if this can be related... did you try compile it without config_preempt ?
> (and dynamic overclocking in BIOS is disabled ?)
It still happens without config_preempt and afaik I don't have dynamic
overclocking enabled (I did not find anything in the bios setup, it's
a GA-M55PLUS-S3G).
SysRq : Show State
sibling
task PC pid father child younger older
init D 000000000000000a 0 1 0 2 (NOTLB)
ffff81003ef83b78 0000000000000082 0000000000000296 ffffffff801f2d31
0000000000000000 ffff81003ef814e0 ffff81003ef8e080 00000000000164f8
ffff81003ef816b8 0000000139e071c0 0000000000000000 0000000000000046
Call Trace:
[<ffffffff801f2d31>] __up_read+0x21/0xb0
[<ffffffff801e4a33>] __generic_unplug_device+0x13/0x30
[<ffffffff8014d880>] sync_page+0x0/0x50
[<ffffffff802bd398>] io_schedule+0x28/0x40
[<ffffffff8014d8c5>] sync_page+0x45/0x50
[<ffffffff802bd690>] __wait_on_bit_lock+0x40/0x80
[<ffffffff8014d85f>] __lock_page+0x5f/0x70
[<ffffffff80140f20>] wake_bit_function+0x0/0x30
[<ffffffff80166d50>] grab_swap_token+0xa0/0x160
[<ffffffff8015aa15>] __handle_mm_fault+0x885/0xb40
[<ffffffff801a7d10>] proc_delete_inode+0x0/0x40
[<ffffffff801921ce>] invalidate_inode_buffers+0xe/0x60
[<ffffffff80119637>] do_page_fault+0x4b7/0x8d0
[<ffffffff80126960>] default_wake_function+0x0/0x10
[<ffffffff8019bcf3>] compat_core_sys_select+0x203/0x240
[<ffffffff80171bcb>] vfs_getattr+0x4b/0xa0
[<ffffffff802bed7d>] error_exit+0x0/0x84
[<ffffffff8019b83a>] compat_set_fd_set+0x4a/0x50
[<ffffffff8019bcc3>] compat_core_sys_select+0x1d3/0x240
[<ffffffff8019d9ca>] compat_sys_select+0xba/0x1a0
[<ffffffff8011c382>] ia32_sysret+0x0/0xa
migration/0 S 0000000000000001 0 2 1 3 (L-TLB)
ffff81003ef87e90 0000000000000046 0000000000000082 ffff81003ca77e98
0000000000000001 ffff81003ef80e00 ffffffff802f73c0 0000000000000d17
ffff81003ef80fd8 0000000080125124 0000000300000000 ffff81003ca77e88
Call Trace:
[<ffffffff80129083>] migration_thread+0x193/0x2c0
[<ffffffff80128ef0>] migration_thread+0x0/0x2c0
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80201be0>] vgacon_cursor+0x0/0x1d0
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
ksoftirqd/0 S 0000000000000001 0 3 1 4 2 (L-TLB)
ffff81003ef8bef0 0000000000000046 ffff81003ef8bf00 0000000000000046
0000000000000086 ffff81003ef80720 ffff81003e6f77a0 00000000000005e4
ffff81003ef808f8 000000008012584c 0000000000000013 ffff81003e6f77a0
Call Trace:
[<ffffffff80131e90>] ksoftirqd+0x0/0xc0
[<ffffffff80131ee1>] ksoftirqd+0x51/0xc0
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
watchdog/0 S 0000000000000001 0 4 1 5 3 (L-TLB)
ffff81003ef8dee0 0000000000000046 0000000000000001 ffffffff802bcb83
ffff81003ef8df00 ffff81003ef80040 ffffffff802f73c0 0000000000000157
ffff81003ef80218 0000000000000001 ffff810001dd4b18 ffffffff802f73c0
Call Trace:
[<ffffffff802bcb83>] thread_return+0x0/0xfd
[<ffffffff80125124>] __wake_up_common+0x44/0x80
[<ffffffff8014acb0>] watchdog+0x0/0x70
[<ffffffff8014ad09>] watchdog+0x59/0x70
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
migration/1 S 0000000000000001 0 5 1 6 4 (L-TLB)
ffff81003ef91e90 0000000000000046 0000000000000082 ffff81003d5a1e98
0000000000000001 ffff81003ef8f520 ffff81003ef8e080 0000000000001022
ffff81003ef8f6f8 0000000180125124 0000000300000000 ffff81003d5a1e88
Call Trace:
[<ffffffff80129083>] migration_thread+0x193/0x2c0
[<ffffffff80128ef0>] migration_thread+0x0/0x2c0
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
ksoftirqd/1 S 000000000000000a 0 6 1 7 5 (L-TLB)
ffff81003ef99ef0 0000000000000046 ffff810001f818e0 000000000001325b
ffff81003ef8f020 ffff81003ef8ee40 ffff81003ef8e080 0000000000000386
ffff81003ef8f018 000000018012584c ffff810001dd97f8 ffff81003ef1ee80
Call Trace:
[<ffffffff80131e90>] ksoftirqd+0x0/0xc0
[<ffffffff80131ee1>] ksoftirqd+0x51/0xc0
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
watchdog/1 S 0000000000000009 0 7 1 8 6 (L-TLB)
ffff81003ef9bee0 0000000000000046 000000000000000a ffffffff802bcb83
ffff81003ef9bf00 ffff81003ef8e760 ffff81003ef8e080 00000000000003ff
ffff81003ef8e938 0000000100000001 ffff810001dda0d8 ffff81003ef8e080
Call Trace:
[<ffffffff802bcb83>] thread_return+0x0/0xfd
[<ffffffff80125124>] __wake_up_common+0x44/0x80
[<ffffffff8014acb0>] watchdog+0x0/0x70
[<ffffffff8014ad09>] watchdog+0x59/0x70
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
events/0 S 000000000000000a 0 8 1 9 7 (L-TLB)
ffff81003efb1e70 0000000000000046 0000000000000086 0000000000000003
ffff81003efb1e20 ffff81003ef1f560 ffffffff802f73c0 0000000000005a8a
ffff81003ef1f738 0000000000000000 0000000000000286 ffffffff8016c250
Call Trace:
[<ffffffff8016c250>] cache_reap+0x0/0x130
[<ffffffff8013d468>] run_workqueue+0xd8/0x110
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff8013dc36>] worker_thread+0xf6/0x160
[<ffffffff80126960>] default_wake_function+0x0/0x10
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
events/1 S 000000000000000a 0 9 1 10 8 (L-TLB)
ffff81003efb3e70 0000000000000046 0000000000000086 0000000000000003
ffff81003efb3e20 ffff81003ef1ee80 ffff81003ef8e080 00000000000058f9
ffff81003ef1f058 0000000100000000 0000000000000286 ffffffff8016c250
Call Trace:
[<ffffffff8016c250>] cache_reap+0x0/0x130
[<ffffffff8013d468>] run_workqueue+0xd8/0x110
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff8013dc36>] worker_thread+0xf6/0x160
[<ffffffff80126960>] default_wake_function+0x0/0x10
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
khelper S 0000000000000006 0 10 1 11 9 (L-TLB)
ffff81003efb5e70 0000000000000046 0000000000000086 0000000000000003
ffff81003efb5e20 ffff81003ef1e7a0 ffffffff802f73c0 0000000000000577
ffff81003ef1e978 000000003ef83cb8 0000000000000286 ffffffff8013ce00
Call Trace:
[<ffffffff8013ce00>] __call_usermodehelper+0x0/0x80
[<ffffffff8013d468>] run_workqueue+0xd8/0x110
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff8013dc36>] worker_thread+0xf6/0x160
[<ffffffff80126960>] default_wake_function+0x0/0x10
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80116410>] flat_send_IPI_mask+0x0/0x50
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
kthread S 000000000000000a 0 11 1 35 321 10 (L-TLB)
ffff81003efb9e70 0000000000000046 0000000000000086 0000000000000003
ffff81003efb9e20 ffff81003ef1e0c0 ffffffff802f73c0 00000000000007d8
ffff81003ef1e298 000000003c51f9b8 0000000000000286 ffffffff80140a60
Call Trace:
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff8013d468>] run_workqueue+0xd8/0x110
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff8013dc36>] worker_thread+0xf6/0x160
[<ffffffff80126960>] default_wake_function+0x0/0x10
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
kblockd/0 S ffff81003ef02540 0 35 11 36 (L-TLB)
ffff81003e517e70 0000000000000046 0000000000000086 0000000000000003
ffff81003e517e20 ffff81003e62c240 ffff810001f81200 0000000000001637
ffff81003e62c418 000000003ee1b2d8 0000000000000282 ffffffff801ec410
Call Trace:
[<ffffffff801ec410>] cfq_kick_queue+0x0/0x50
[<ffffffff8013d468>] run_workqueue+0xd8/0x110
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff8013dc36>] worker_thread+0xf6/0x160
[<ffffffff80126960>] default_wake_function+0x0/0x10
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
kblockd/1 S 000000000000000a 0 36 11 37 35 (L-TLB)
ffff81003e51be70 0000000000000046 0000000000000086 0000000000000003
ffff81003e51be20 ffff81003e519720 ffff81003ef8e080 000000000000d34a
ffff81003e5198f8 000000013ee1b2d8 0000000000000282 ffffffff801ec410
Call Trace:
[<ffffffff801ec410>] cfq_kick_queue+0x0/0x50
[<ffffffff8013d468>] run_workqueue+0xd8/0x110
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff8013dc36>] worker_thread+0xf6/0x160
[<ffffffff80126960>] default_wake_function+0x0/0x10
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
kseriod S ffffffff803187c0 0 37 11 92 36 (L-TLB)
ffff81003ef5deb0 0000000000000046 ffffffff80314f80 ffff81003ef1c788
00000000ffffffef ffff81003e519040 ffff81003ef1ee80 0000000000507007
ffff81003e519218 0000000180314f80 ffff810001dd97f8 ffffffff80319670
Call Trace:
[<ffffffff8022418d>] driver_create_file+0x3d/0x60
[<ffffffff80249350>] serio_thread+0x0/0x330
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80249643>] serio_thread+0x2f3/0x330
[<ffffffff80140ef0>] autoremove_wake_function+0x0/0x30
[<ffffffff80249350>] serio_thread+0x0/0x330
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
pdflush S 000000000000000a 0 92 11 93 37 (L-TLB)
ffff81003ef29ec0 0000000000000046 0000000000000400 0000000000000000
ffffffff80153940 ffff810001f80b20 ffffffff802f73c0 0000000000000195
ffff810001f80cf8 0000000000000400 0000000000000000 0000000000000000
Call Trace:
[<ffffffff80153940>] pdflush+0x0/0x1d0
[<ffffffff80153940>] pdflush+0x0/0x1d0
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80153a10>] pdflush+0xd0/0x1d0
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
pdflush S 000000000000000a 0 93 11 94 92 (L-TLB)
ffff81003ef2bec0 0000000000000046 0000000000000282 0000000000000286
00000000000004e2 ffff810001f81200 ffffffff802f73c0 000000000000063f
ffff810001f813d8 00000000801534d5 0000000000000000 0000000000000000
Call Trace:
[<ffffffff80153940>] pdflush+0x0/0x1d0
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80153a10>] pdflush+0xd0/0x1d0
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
kswapd0 D 000000000000000a 0 94 11 95 93 (L-TLB)
ffff81003ef27a40 0000000000000046 ffff81003ef279e0 ffffffff80300a10
ffff810001f818e0 ffff810001f818e0 ffff81003ef8e080 00000000000008bd
ffff810001f81ab8 000000018015218b 0000000000000286 ffffffff80135f89
Call Trace:
[<ffffffff80135f89>] lock_timer_base+0x29/0x60
[<ffffffff802bd5da>] schedule_timeout+0x9a/0xd0
[<ffffffff80135bd0>] process_timeout+0x0/0x10
[<ffffffff802bce18>] io_schedule_timeout+0x28/0x40
[<ffffffff801503a3>] mempool_alloc+0xd3/0x110
[<ffffffff80140ef0>] autoremove_wake_function+0x0/0x30
[<ffffffff801939a1>] bio_alloc_bioset+0xd1/0x150
[<ffffffff80163c40>] end_swap_bio_write+0x0/0x90
[<ffffffff80193a80>] bio_alloc+0x10/0x30
[<ffffffff80163a9f>] get_swap_bio+0x2f/0xd0
[<ffffffff80163d1e>] swap_writepage+0x4e/0xd0
[<ffffffff80156178>] shrink_zone+0x9a8/0xe20
[<ffffffff80156b6f>] kswapd+0x30f/0x460
[<ffffffff80140ef0>] autoremove_wake_function+0x0/0x30
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80156860>] kswapd+0x0/0x460
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
aio/0 S ffff810001f65440 0 95 11 96 94 (L-TLB)
ffff81003e543e70 0000000000000046 0000000000000064 0000000000000000
ffff81003e542000 ffff81003e676400 ffff81003ef1e0c0 000000000000098a
ffff81003e6765d8 000000003e676400 ffff810001dd4238 ffffffff80136d0b
Call Trace:
[<ffffffff80136d0b>] do_sigaction+0x1ab/0x1d0
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff8013dc36>] worker_thread+0xf6/0x160
[<ffffffff80126960>] default_wake_function+0x0/0x10
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
aio/1 S ffff810001f653c0 0 96 11 193 95 (L-TLB)
ffff81003efbde70 0000000000000046 ffff81003efbde40 ffff81003efbddf0
ffffffff80124fad ffff81003e676ae0 ffff81003ef1e7a0 00000000000012d4
ffff81003e676cb8 000000013ef1e0c0 ffff810001dd97f8 0000000000000000
Call Trace:
[<ffffffff80124fad>] deactivate_task+0x1d/0x30
[<ffffffff80136cd2>] do_sigaction+0x172/0x1d0
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff8013dc36>] worker_thread+0xf6/0x160
[<ffffffff80126960>] default_wake_function+0x0/0x10
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
ata/0 S ffff81003e5a2640 0 193 11 194 96 (L-TLB)
ffff81003ee17e70 0000000000000046 0000000000000064 0000000000000000
ffff81003ee16000 ffff81003ee14aa0 ffff81003ef1e0c0 0000000000000b4a
ffff81003ee14c78 000000003ee14aa0 ffff810001dd4238 ffffffff80136d0b
Call Trace:
[<ffffffff80136d0b>] do_sigaction+0x1ab/0x1d0
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff8013dc36>] worker_thread+0xf6/0x160
[<ffffffff80126960>] default_wake_function+0x0/0x10
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
ata/1 S 0000000000000008 0 194 11 195 193 (L-TLB)
ffff81003ee0fe70 0000000000000046 ffff81003ee0fe40 ffff81003ee0fdf0
ffffffff80124fad ffff81003ee15180 ffff81003ef8e080 00000000000009e4
ffff81003ee15358 000000013ef1e0c0 0000000000000086 0000000000000000
Call Trace:
[<ffffffff80124fad>] deactivate_task+0x1d/0x30
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff8013dc36>] worker_thread+0xf6/0x160
[<ffffffff80126960>] default_wake_function+0x0/0x10
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
ata_aux S 0000000000000008 0 195 11 197 194 (L-TLB)
ffff81003ee0be70 0000000000000046 ffff81003ee0be40 ffff81003ee0bdf0
ffffffff80124fad ffff81003ee15860 ffffffff802f73c0 00000000000003f6
ffff81003ee15a38 000000003ef1e0c0 0000000000000086 0000000000000000
Call Trace:
[<ffffffff80124fad>] deactivate_task+0x1d/0x30
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff8013dc36>] worker_thread+0xf6/0x160
[<ffffffff80126960>] default_wake_function+0x0/0x10
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
scsi_eh_0 S 0000000000000246 0 197 11 198 195 (L-TLB)
ffff81003e62be70 0000000000000046 ffff81003e62be10 ffffffff80125783
0000000000000296 ffff81003ee08a60 ffff81003ef814e0 00000000002df9e2
ffff81003ee08c38 000000013e684098 0000000000000000 0000000000000001
Call Trace:
[<ffffffff80125783>] __wake_up+0x43/0x70
[<ffffffff8022ecfa>] __scsi_iterate_devices+0x5a/0x80
[<ffffffff8023218f>] scsi_error_handler+0x6f/0x730
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80232120>] scsi_error_handler+0x0/0x730
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
scsi_eh_1 S 0000000000000246 0 198 11 219 197 (L-TLB)
ffff81003ef2fe70 0000000000000046 ffff81003ef2fe10 ffffffff80125783
0000000000000296 ffff81003ee09140 ffff81003ef814e0 00000000002dce40
ffff81003ee09318 000000013ee1c098 0000000000000000 0000000000000001
Call Trace:
[<ffffffff80125783>] __wake_up+0x43/0x70
[<ffffffff8022ecfa>] __scsi_iterate_devices+0x5a/0x80
[<ffffffff8023218f>] scsi_error_handler+0x6f/0x730
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80232120>] scsi_error_handler+0x0/0x730
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
scsi_eh_2 S 0000000000000246 0 219 11 220 198 (L-TLB)
ffff81003e627e70 0000000000000046 ffff81003e627e10 ffffffff80125783
0000000000000296 ffff81003e6b6180 ffff81003ef814e0 00000000002df821
ffff81003e6b6358 000000013ee10098 0000000000000000 0000000000000001
Call Trace:
[<ffffffff80125783>] __wake_up+0x43/0x70
[<ffffffff8022ecfa>] __scsi_iterate_devices+0x5a/0x80
[<ffffffff8023218f>] scsi_error_handler+0x6f/0x730
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80232120>] scsi_error_handler+0x0/0x730
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
scsi_eh_3 S 000000000000000a 0 220 11 237 219 (L-TLB)
ffff81003e645e70 0000000000000046 ffff81003e645e10 ffffffff80125783
0000000000000296 ffff81003ee08380 ffffffff802f73c0 000000000116007a
ffff81003ee08558 000000003e66c098 0000000000000000 0000000000000001
Call Trace:
[<ffffffff80125783>] __wake_up+0x43/0x70
[<ffffffff8022ecfa>] __scsi_iterate_devices+0x5a/0x80
[<ffffffff8023218f>] scsi_error_handler+0x6f/0x730
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80232120>] scsi_error_handler+0x0/0x730
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
kcryptd/0 D 000000000000000a 0 237 11 238 220 (L-TLB)
ffff81003e6a7c50 0000000000000046 ffff810022780ff0 0000000000000001
ffff81003ef36340 ffff81003ef36340 ffffffff802f73c0 0000000000000956
ffff81003ef36518 000000008015227d 0000000000000286 ffffffff80135f89
Call Trace:
[<ffffffff80135f89>] lock_timer_base+0x29/0x60
[<ffffffff802bd5da>] schedule_timeout+0x9a/0xd0
[<ffffffff80135bd0>] process_timeout+0x0/0x10
[<ffffffff802bce18>] io_schedule_timeout+0x28/0x40
[<ffffffff801503a3>] mempool_alloc+0xd3/0x110
[<ffffffff80140ef0>] autoremove_wake_function+0x0/0x30
[<ffffffff801939a1>] bio_alloc_bioset+0xd1/0x150
[<ffffffff80193a80>] bio_alloc+0x10/0x30
[<ffffffff80265d3a>] kcryptd_do_work+0x1ea/0x370
[<ffffffff80265b50>] kcryptd_do_work+0x0/0x370
[<ffffffff8013d442>] run_workqueue+0xb2/0x110
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff8013dc61>] worker_thread+0x121/0x160
[<ffffffff80126960>] default_wake_function+0x0/0x10
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
kcryptd/1 D 000000000000000a 0 238 11 242 237 (L-TLB)
ffff81003e69dc50 0000000000000046 ffff810033338ff0 0000000000000001
ffff81003ee09820 ffff81003ee09820 ffff81003ef8e080 0000000000000877
ffff81003ee099f8 000000018015227d 0000000000000286 ffffffff80135f89
Call Trace:
[<ffffffff80135f89>] lock_timer_base+0x29/0x60
[<ffffffff802bd5da>] schedule_timeout+0x9a/0xd0
[<ffffffff80135bd0>] process_timeout+0x0/0x10
[<ffffffff802bce18>] io_schedule_timeout+0x28/0x40
[<ffffffff801503a3>] mempool_alloc+0xd3/0x110
[<ffffffff80140ef0>] autoremove_wake_function+0x0/0x30
[<ffffffff801939a1>] bio_alloc_bioset+0xd1/0x150
[<ffffffff80193a80>] bio_alloc+0x10/0x30
[<ffffffff80265d3a>] kcryptd_do_work+0x1ea/0x370
[<ffffffff80265b50>] kcryptd_do_work+0x0/0x370
[<ffffffff8013d442>] run_workqueue+0xb2/0x110
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff8013dc61>] worker_thread+0x121/0x160
[<ffffffff80126960>] default_wake_function+0x0/0x10
[<ffffffff8013db40>] worker_thread+0x0/0x160
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
md11_raid1 S 7fffffffffffffff 0 242 11 244 238 (L-TLB)
ffff81003e6dde50 0000000000000046 0000000000000002 ffff81003e6ce800
ffff81003e6dddf0 ffff81003ef36a20 ffff81003ef2cf40 0000000000000235
ffff81003ef36bf8 000000008012584c ffff81003e6dde10 ffff81003ef2cf40
Call Trace:
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff802bd565>] schedule_timeout+0x25/0xd0
[<ffffffff80141083>] prepare_to_wait+0x23/0x80
[<ffffffff8025c13e>] md_thread+0xee/0x130
[<ffffffff80140ef0>] autoremove_wake_function+0x0/0x30
[<ffffffff8025c050>] md_thread+0x0/0x130
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
md10_raid1 S 000000000000000a 0 244 11 245 242 (L-TLB)
ffff81003e6dfe50 0000000000000046 0000000000000002 ffff81003e6ce400
ffff81003e6dfdf0 ffff81003ef37100 ffff81003ef8e080 0000000000001338
ffff81003ef372d8 000000018012584c ffff810001dd97f8 ffff81003ef8e080
Call Trace:
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff802bd565>] schedule_timeout+0x25/0xd0
[<ffffffff80141083>] prepare_to_wait+0x23/0x80
[<ffffffff8025c13e>] md_thread+0xee/0x130
[<ffffffff80140ef0>] autoremove_wake_function+0x0/0x30
[<ffffffff8025c050>] md_thread+0x0/0x130
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
kjournald S 000000000000000a 0 245 11 696 244 (L-TLB)
ffff81003d61fe80 0000000000000046 ffff81003e6be000 0000000000000000
ffff81003a329024 ffff81003e661820 ffffffff802f73c0 0000000000002b21
ffff81003e6619f8 000000003e6c5a90 0000000000000000 0000000000000001
Call Trace:
[<ffffffff801cd0f8>] kjournald+0x1a8/0x230
[<ffffffff80140ef0>] autoremove_wake_function+0x0/0x30
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff801ccf50>] kjournald+0x0/0x230
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
udevd S 0000000000000001 0 321 1 977 11 (NOTLB)
ffff81003d321b08 0000000000000082 00000000fff57000 ffff81003d770740
0000000000000001 ffff81003e660380 ffffffff802f73c0 000000000000c060
ffff81003e660558 00000000802db5d9 0000000000000003 ffff81003e6d7000
Call Trace:
[<ffffffff802bd565>] schedule_timeout+0x25/0xd0
[<ffffffff801411bc>] add_wait_queue+0x1c/0x60
[<ffffffff80199c4d>] inotify_poll+0x5d/0x80
[<ffffffff8017c217>] do_select+0x477/0x530
[<ffffffff8017c8a0>] __pollwait+0x0/0x120
[<ffffffff80126960>] default_wake_function+0x0/0x10
[<ffffffff80126960>] default_wake_function+0x0/0x10
[<ffffffff80126960>] default_wake_function+0x0/0x10
[<ffffffff80126960>] default_wake_function+0x0/0x10
[<ffffffff80184bb4>] mntput_no_expire+0x24/0xa0
[<ffffffff801780b5>] __link_path_walk+0x6e5/0xd90
[<ffffffff80180174>] d_lookup+0x24/0x50
[<ffffffff80184bb4>] mntput_no_expire+0x24/0xa0
[<ffffffff80178860>] link_path_walk+0x100/0x140
[<ffffffff801410f9>] remove_wait_queue+0x19/0x60
[<ffffffff8012fa3d>] do_wait+0xabd/0xbb0
[<ffffffff8016f08b>] vfs_read+0x11b/0x160
[<ffffffff80126960>] default_wake_function+0x0/0x10
[<ffffffff8019bc8d>] compat_core_sys_select+0x19d/0x240
[<ffffffff8019d9ee>] compat_sys_select+0xde/0x1a0
[<ffffffff8011c382>] ia32_sysret+0x0/0xa
md126_raid1 S 000000000000000a 0 696 11 245 (L-TLB)
ffff81003ce01e50 0000000000000046 0000000000000002 ffff81003d2eb400
0000000000000000 ffff81003e6c01c0 ffff81003ef8e080 000000000000073e
ffff81003e6c0398 000000018012584c ffff81003ce01e10 ffff81003ef8e080
Call Trace:
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff802bd565>] schedule_timeout+0x25/0xd0
[<ffffffff80141083>] prepare_to_wait+0x23/0x80
[<ffffffff8025c13e>] md_thread+0xee/0x130
[<ffffffff80140ef0>] autoremove_wake_function+0x0/0x30
[<ffffffff8025c050>] md_thread+0x0/0x130
[<ffffffff80140bb9>] kthread+0xd9/0x120
[<ffffffff8010a958>] child_rip+0xa/0x12
[<ffffffff80140a60>] keventd_create_kthread+0x0/0x80
[<ffffffff80140ae0>] kthread+0x0/0x120
[<ffffffff8010a94e>] child_rip+0x0/0x12
syslogd D 0000000000000009 0 977 1 983 321 (NOTLB)
ffff81003aa45628 0000000000000082 0000000000000000 ffffffff80300a10
ffff81003e6f69e0 ffff81003e6f69e0 ffff81003ef8e080 0000000000000995
ffff81003e6f6bb8 000000018015218b 0000000000000286 ffffffff80135f89
Call Trace:
[<ffffffff80135f89>] lock_timer_base+0x29/0x60
[<ffffffff802bd5da>] schedule_timeout+0x9a/0xd0
[<ffffffff80135bd0>] process_timeout+0x0/0x10
[<ffffffff802bce18>] io_schedule_timeout+0x28/0x40
[<ffffffff801503a3>] mempool_alloc+0xd3/0x110
[<ffffffff80140ef0>] autoremove_wake_function+0x0/0x30
[<ffffffff8012693c>] try_to_wake_up+0x38c/0x3b0
[<ffffffff801939a1>] bio_alloc_bioset+0xd1/0x150
[<ffffffff80163c40>] end_swap_bio_write+0x0/0x90
[<ffffffff80193a80>] bio_alloc+0x10/0x30
[<ffffffff80163a9f>] get_swap_bio+0x2f/0xd0
[<ffffffff80163d1e>] swap_writepage+0x4e/0xd0
[<ffffffff80156178>] shrink_zone+0x9a8/0xe20
[<ffffffff801907e9>] __find_get_block+0x169/0x180
[<ffffffff801c6d2c>] do_get_write_access+0x53c/0x590
[<ffffffff80156e27>] try_to_free_pages+0x167/0x260
[<ffffffff8015224d>] __alloc_pages+0x1bd/0x2d0
[<ffffffff80163ef3>] read_swap_cache_async+0x43/0xe0
[<ffffffff80158ac6>] swapin_readahead+0x56/0x90
[<ffffffff8015a969>] __handle_mm_fault+0x7d9/0xb40
[<ffffffff80119637>] do_page_fault+0x4b7/0x8d0
[<ffffffff80131a6b>] current_fs_time+0x3b/0x40
[<ffffffff80125783>] __wake_up+0x43/0x70
[<ffffffff80183350>] file_update_time+0x30/0xb0
[<ffffffff802bed7d>] error_exit+0x0/0x84
[<ffffffff80271760>] datagram_poll+0x0/0xf0
[<ffffffff8011e6e7>] ia32_setup_frame+0x67/0x1a0
[<ffffffff8011e6b7>] ia32_setup_frame+0x37/0x1a0
[<ffffffff801091ac>] do_notify_resume+0x1cc/0x7a0
[<ffffffff801316b0>] getnstimeofday+0x10/0x30
[<ffffffff80109dc6>] int_signal+0x12/0x17
klogd R running task 0 983 1 1048 977 (NOTLB)
login S 0000000000000008 0 1048 1 1167 1091 983 (NOTLB)
ffff81003b34fde8 0000000000000082 0000000000000001 ffffffff80158e0a
0000000100000008 ffff81003ee143c0 ffff81003ef8e080 0000000000005a82
ffff81003ee14598 000000013d103000 800000003b300065 ffff810001d5ba40
Call Trace:
[<ffffffff80158e0a>] do_wp_page+0x1aa/0x4d0
[<ffffffff8012fa05>] do_wait+0xa85/0xbb0
[<ffffffff8012b83f>] copy_process+0x10ef/0x1280
[<ffffffff80126960>] default_wake_function+0x0/0x10
[<ffffffff801082eb>] __switch_to+0x12b/0x2c0
[<ffffffff801498d1>] compat_sys_wait4+0x41/0xf0
[<ffffffff802bcb83>] thread_return+0x0/0xfd
[<ffffffff802bed7d>] error_exit+0x0/0x84
[<ffffffff8011c382>] ia32_sysret+0x0/0xa
login S ffff81003e6cb7d8 0 1091 1 1092 1048 (NOTLB)
ffff81003c9d3de8 0000000000000082 0000000000000001 ffffffff80158e0a
0000000100000008 ffff81003ef22f00 ffff81003e6f69e0 00000000000056b1
ffff81003ef230d8 000000013d798000 ffff810001dda0d8 ffff810001cd6180
Call Trace:
[<ffffffff80158e0a>] do_wp_page+0x1aa/0x4d0
[<ffffffff8015ac10>] __handle_mm_fault+0xa80/0xb40
[<ffffffff8012fa05>] do_wait+0xa85/0xbb0
[<ffffffff8012b83f>] copy_process+0x10ef/0x1280
[<ffffffff80126960>] default_wake_function+0x0/0x10
[<ffffffff801082eb>] __switch_to+0x12b/0x2c0
[<ffffffff801498d1>] compat_sys_wait4+0x41/0xf0
[<ffffffff802bcb83>] thread_return+0x0/0xfd
[<ffffffff802bed7d>] error_exit+0x0/0x84
[<ffffffff8011c382>] ia32_sysret+0x0/0xa
bash D 0000000000000009 0 1092 1091 (NOTLB)
ffff81003e629638 0000000000000082 00000000001763fc ffffffff80300a10
ffff81003e6cb6a0 ffff81003e6cb6a0 ffff81003ef8e080 0000000000000917
ffff81003e6cb878 000000018015218b 0000000000000286 ffffffff80135f89
Call Trace:
[<ffffffff80135f89>] lock_timer_base+0x29/0x60
[<ffffffff802bd5da>] schedule_timeout+0x9a/0xd0
[<ffffffff80135bd0>] process_timeout+0x0/0x10
[<ffffffff802bce18>] io_schedule_timeout+0x28/0x40
[<ffffffff801503a3>] mempool_alloc+0xd3/0x110
[<ffffffff80140ef0>] autoremove_wake_function+0x0/0x30
[<ffffffff801939a1>] bio_alloc_bioset+0xd1/0x150
[<ffffffff80163c40>] end_swap_bio_write+0x0/0x90
[<ffffffff80193a80>] bio_alloc+0x10/0x30
[<ffffffff80163a9f>] get_swap_bio+0x2f/0xd0
[<ffffffff80163d1e>] swap_writepage+0x4e/0xd0
[<ffffffff80156178>] shrink_zone+0x9a8/0xe20
[<ffffffff8014d7c9>] find_get_page+0x29/0x60
[<ffffffff801907e9>] __find_get_block+0x169/0x180
[<ffffffff801907e9>] __find_get_block+0x169/0x180
[<ffffffff80156e27>] try_to_free_pages+0x167/0x260
[<ffffffff8015224d>] __alloc_pages+0x1bd/0x2d0
[<ffffffff80163ef3>] read_swap_cache_async+0x43/0xe0
[<ffffffff80158ac6>] swapin_readahead+0x56/0x90
[<ffffffff8015a969>] __handle_mm_fault+0x7d9/0xb40
[<ffffffff802bd5e2>] schedule_timeout+0xa2/0xd0
[<ffffffff8021dfa0>] serial8250_tx_empty+0x40/0x60
[<ffffffff80119637>] do_page_fault+0x4b7/0x8d0
[<ffffffff80125e99>] find_busiest_group+0x219/0x4c0
[<ffffffff801082eb>] __switch_to+0x12b/0x2c0
[<ffffffff802bed7d>] error_exit+0x0/0x84
[<ffffffff801f4ec2>] copy_user_generic_string+0x12/0x40
[<ffffffff80208def>] copy_from_read_buf+0x8f/0xf0
[<ffffffff8020a7a5>] read_chan+0x535/0x6c0
[<ffffffff80126960>] default_wake_function+0x0/0x10
[<ffffffff80207086>] tty_read+0x86/0xf0
[<ffffffff8016f02a>] vfs_read+0xba/0x160
[<ffffffff8016f523>] sys_read+0x53/0x90
[<ffffffff8011c382>] ia32_sysret+0x0/0xa
mprime D 000000000000000a 0 1167 1048 (NOTLB)
ffff81003d5a18d8 0000000000200086 ffff81003d5a1878 ffffffff80300a10
ffff81003e6f77a0 ffff81003e6f77a0 ffff81003ef8e080 0000000000000a2b
ffff81003e6f7978 000000018015218b 0000000000200286 ffffffff80135f89
Call Trace:
[<ffffffff80135f89>] lock_timer_base+0x29/0x60
[<ffffffff802bd5da>] schedule_timeout+0x9a/0xd0
[<ffffffff80135bd0>] process_timeout+0x0/0x10
[<ffffffff802bce18>] io_schedule_timeout+0x28/0x40
[<ffffffff801503a3>] mempool_alloc+0xd3/0x110
[<ffffffff80140ef0>] autoremove_wake_function+0x0/0x30
[<ffffffff801939a1>] bio_alloc_bioset+0xd1/0x150
[<ffffffff80163c40>] end_swap_bio_write+0x0/0x90
[<ffffffff80193a80>] bio_alloc+0x10/0x30
[<ffffffff80163a9f>] get_swap_bio+0x2f/0xd0
[<ffffffff80163d1e>] swap_writepage+0x4e/0xd0
[<ffffffff80156178>] shrink_zone+0x9a8/0xe20
[<ffffffff802bcb83>] thread_return+0x0/0xfd
[<ffffffff80140ef0>] autoremove_wake_function+0x0/0x30
[<ffffffff80156e27>] try_to_free_pages+0x167/0x260
[<ffffffff80140ef0>] autoremove_wake_function+0x0/0x30
[<ffffffff8015224d>] __alloc_pages+0x1bd/0x2d0
[<ffffffff8015a6a3>] __handle_mm_fault+0x513/0xb40
[<ffffffff80119637>] do_page_fault+0x4b7/0x8d0
[<ffffffff8015f906>] do_mmap_pgoff+0x656/0x7e0
[<ffffffff802bed7d>] error_exit+0x0/0x84