Dave Kleikamp wrote:
>
> I don't have an answer to an ext3 deadlock, but this looks like a jfs
> problem that was recently fixed in linux-2.6.22-rc1. I had intended to
> send it to the stable kernel after it was picked up in mainline, but
> hadn't gotten to it yet.
>
> The patch is here:
> http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=05ec9e26be1f668ccba4ca54d9a4966c6208c611
>
Dave,
That appears to have fixed the JFS hangup.
MTBF before was about 1 hour, under the same test I am over 20 hours
and things appear to still be holding together.
Roger
On Thu, 2007-05-17 at 09:37 -0500, Roger Heflin wrote:
> Dave Kleikamp wrote:
>
> >
> > I don't have an answer to an ext3 deadlock, but this looks like a jfs
> > problem that was recently fixed in linux-2.6.22-rc1. I had intended to
> > send it to the stable kernel after it was picked up in mainline, but
> > hadn't gotten to it yet.
> >
> > The patch is here:
> > http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=05ec9e26be1f668ccba4ca54d9a4966c6208c611
> >
>
> Dave,
>
> That appears to have fixed the JFS hangup.
>
> MTBF before was about 1 hour, under the same test I am over 20 hours
> and things appear to still be holding together.
Great. The patch is queued for the 2.6.21 stable tree now.
Thanks,
Shaggy
--
David Kleikamp
IBM Linux Technology Center
May 29 11:58:20 vault1 kernel: SysRq : Show State
May 29 11:58:20 vault1 kernel:
May 29 11:58:20 vault1 kernel: free sibling
May 29 11:58:20 vault1 kernel: task PC stack pid father child younger older
May 29 11:58:20 vault1 kernel: init S ffff81011f96fa98 0 1 0 (NOTLB)
May 29 11:58:20 vault1 kernel: ffff81011fc89a18 0000000000000046 0000000000000000 ffff810037e10040
May 29 11:58:20 vault1 kernel: ffff810037e10040 ffffffff802638fc 000000000000000a ffff810037e10040
May 29 11:58:20 vault1 kernel: ffff81011fc78040 0000000000006e1f ffff810037e10248 0000000137e28000
May 29 11:58:20 vault1 kernel: Call Trace:
May 29 11:58:20 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:20 vault1 kernel: [<ffffffff80261790>] schedule_timeout+0x8a/0xad
May 29 11:58:20 vault1 kernel: [<ffffffff802913d9>] process_timeout+0x0/0x5
May 29 11:58:20 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 29 11:58:20 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:20 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:20 vault1 kernel: [<ffffffff80209596>] __d_lookup+0x77/0x110
May 29 11:58:20 vault1 kernel: [<ffffffff802095f2>] __d_lookup+0xd3/0x110
May 29 11:58:20 vault1 kernel: [<ffffffff802635ce>] _spin_unlock+0x17/0x20
May 29 11:58:20 vault1 kernel: [<ffffffff802095f2>] __d_lookup+0xd3/0x110
May 29 11:58:20 vault1 kernel: [<ffffffff8020cff9>] do_lookup+0x63/0x1ae
May 29 11:58:20 vault1 kernel: [<ffffffff8020d542>] dput+0x3c/0x12b
May 29 11:58:20 vault1 kernel: [<ffffffff8020a2b6>] __link_path_walk+0xc87/0xde9
May 29 11:58:20 vault1 kernel: [<ffffffff8028dc04>] _local_bh_enable+0xbb/0xd0
May 29 11:58:20 vault1 kernel: [<ffffffff8022d806>] mntput_no_expire+0x1c/0x96
May 29 11:58:20 vault1 kernel: [<ffffffff8020ed97>] link_path_walk+0xce/0xe0
May 29 11:58:20 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 29 11:58:20 vault1 kernel: [<ffffffff8022d806>] mntput_no_expire+0x1c/0x96
May 29 11:58:20 vault1 kernel: [<ffffffff8021b2bb>] cp_new_stat+0xe5/0xfd
May 29 11:58:20 vault1 kernel: [<ffffffff80216b98>] sys_select+0x15a/0x183
May 29 11:58:20 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:20 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:20 vault1 kernel:
May 29 11:58:20 vault1 kernel: migration/0 S ffff81011f2b9218 0 2 1 (L-TLB)
May 29 11:58:20 vault1 kernel: ffff810037e1bec0 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:20 vault1 kernel: 0000000100000000 0000000000000002 0000000000000001 ffff81011fc6f040
May 29 11:58:20 vault1 kernel: ffffffff8058a6e0 0000000000000f63 ffff81011fc6f248 0000000000000000
May 29 11:58:20 vault1 kernel: Call Trace:
May 29 11:58:21 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:21 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:21 vault1 kernel: [<ffffffff80243daa>] migration_thread+0x1a2/0x23f
May 29 11:58:21 vault1 kernel: [<ffffffff80243c08>] migration_thread+0x0/0x23f
May 29 11:58:21 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:21 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:21 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:21 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:21 vault1 kernel: [<ffffffff80356044>] vgacon_cursor+0x0/0x1a5
May 29 11:58:21 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:21 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:21 vault1 kernel:
May 29 11:58:21 vault1 kernel: ksoftirqd/0 S ffff81007fe18098 0 3 1 (L-TLB)
May 29 11:58:21 vault1 kernel: ffff810037e1ff00 0000000000000046 0000000000000000 ffff81011fc72080
May 29 11:58:21 vault1 kernel: ffff81011fc72080 00000000000721d2 0000000000000009 ffff81011fc72080
May 29 11:58:21 vault1 kernel: ffffffff8058a6e0 000000000043167f ffff81011fc72288 00000000805ea150
May 29 11:58:21 vault1 kernel: Call Trace:
May 29 11:58:21 vault1 kernel: [<ffffffff8026b5d6>] do_softirq+0x90/0x94
May 29 11:58:21 vault1 kernel: [<ffffffff8028deb7>] ksoftirqd+0x0/0xbf
May 29 11:58:21 vault1 kernel: [<ffffffff8028def2>] ksoftirqd+0x3b/0xbf
May 29 11:58:21 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:21 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:21 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:21 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:21 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:21 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:21 vault1 kernel:
May 29 11:58:21 vault1 kernel: watchdog/0 S ffff81007fe18098 0 4 1 (L-TLB)
May 29 11:58:21 vault1 kernel: ffff810037e21ef0 0000000000000046 0000000000000000 ffffffff8028dc04
May 29 11:58:21 vault1 kernel: 0000000000000007 0000000000000001 0000000000000001 ffff81011fc730c0
May 29 11:58:21 vault1 kernel: ffffffff8058a6e0 000000000000034a ffff81011fc732c8 0000000000000000
May 29 11:58:21 vault1 kernel: Call Trace:
May 29 11:58:21 vault1 kernel: [<ffffffff8028dc04>] _local_bh_enable+0xbb/0xd0
May 29 11:58:21 vault1 kernel: [<ffffffff802b73e8>] watchdog+0x0/0x5f
May 29 11:58:21 vault1 kernel: [<ffffffff802b7437>] watchdog+0x4f/0x5f
May 29 11:58:21 vault1 kernel: [<ffffffff802b73e8>] watchdog+0x0/0x5f
May 29 11:58:21 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:21 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:21 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:21 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:21 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:21 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:21 vault1 kernel:
May 29 11:58:21 vault1 kernel: migration/1 S ffff81011f2b9218 0 5 1 (L-TLB)
May 29 11:58:21 vault1 kernel: ffff810037e23ec0 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:21 vault1 kernel: 0000000100000000 0000000000000002 0000000000000001 ffff81011fc74040
May 29 11:58:21 vault1 kernel: ffff81011fc78040 00000000000013c6 ffff81011fc74248 0000000100000000
May 29 11:58:21 vault1 kernel: Call Trace:
May 29 11:58:21 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:21 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:21 vault1 kernel: [<ffffffff80243daa>] migration_thread+0x1a2/0x23f
May 29 11:58:21 vault1 kernel: [<ffffffff80243c08>] migration_thread+0x0/0x23f
May 29 11:58:21 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:21 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:21 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:21 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:21 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:21 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:21 vault1 kernel:
May 29 11:58:21 vault1 kernel: ksoftirqd/1 S ffff81007f1149d8 0 6 1 (L-TLB)
May 29 11:58:21 vault1 kernel: ffff81000428bf00 0000000000000046 0000000000000000 ffff81011fc75080
May 29 11:58:21 vault1 kernel: ffff81011fc75080 00000000000ca563 000000000000000a ffff81011fc75080
May 29 11:58:21 vault1 kernel: ffff81011fc78040 0000000000000c5d ffff81011fc75288 00000001806bc060
May 29 11:58:21 vault1 kernel: Call Trace:
May 29 11:58:21 vault1 kernel: [<ffffffff8026b5d6>] do_softirq+0x90/0x94
May 29 11:58:21 vault1 kernel: [<ffffffff8028deb7>] ksoftirqd+0x0/0xbf
May 29 11:58:21 vault1 kernel: [<ffffffff8028def2>] ksoftirqd+0x3b/0xbf
May 29 11:58:21 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:21 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:21 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:21 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:21 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:21 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:21 vault1 kernel:
May 29 11:58:21 vault1 kernel: watchdog/1 S ffff81007f1149d8 0 7 1 (L-TLB)
May 29 11:58:21 vault1 kernel: ffff81000428def0 0000000000000046 0000000000000000 0000000000000001
May 29 11:58:21 vault1 kernel: ffff81000428c000 ffff81008354ee00 0000000000000008 ffff81011fc760c0
May 29 11:58:21 vault1 kernel: ffff81011fc78040 0000000000000336 ffff81011fc762c8 0000000100000001
May 29 11:58:21 vault1 kernel: Call Trace:
May 29 11:58:21 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:21 vault1 kernel: [<ffffffff802b73e8>] watchdog+0x0/0x5f
May 29 11:58:21 vault1 kernel: [<ffffffff802b7437>] watchdog+0x4f/0x5f
May 29 11:58:21 vault1 kernel: [<ffffffff802b73e8>] watchdog+0x0/0x5f
May 29 11:58:21 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:21 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:21 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:21 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:22 vault1 kernel: [<ffffffff802c038d>] next_zone+0x23/0x24
May 29 11:58:22 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:22 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:22 vault1 kernel:
May 29 11:58:22 vault1 kernel: migration/2 S ffff81011f96e058 0 8 1 (L-TLB)
May 29 11:58:22 vault1 kernel: ffff8100042b1ec0 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:22 vault1 kernel: 0000000100000000 0000000000000002 0000000000000001 ffff81011fc7a080
May 29 11:58:22 vault1 kernel: ffff81007ff1d080 000000000000190d ffff81011fc7a288 0000000200000000
May 29 11:58:22 vault1 kernel: Call Trace:
May 29 11:58:22 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:22 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:22 vault1 kernel: [<ffffffff80243daa>] migration_thread+0x1a2/0x23f
May 29 11:58:22 vault1 kernel: [<ffffffff80243c08>] migration_thread+0x0/0x23f
May 29 11:58:22 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:22 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:22 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:22 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:22 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:22 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:22 vault1 kernel:
May 29 11:58:22 vault1 kernel: ksoftirqd/2 S ffff81007e8de9d8 0 9 1 (L-TLB)
May 29 11:58:22 vault1 kernel: ffff8100042b3f00 0000000000000046 0000000000000000 ffff81011fccd0c0
May 29 11:58:22 vault1 kernel: ffff81011fccd0c0 00000000000d838b 000000000000000a ffff81011fccd0c0
May 29 11:58:22 vault1 kernel: ffff81011fccf080 0000000000000a83 ffff81011fccd2c8 00000002042b3ef0
May 29 11:58:22 vault1 kernel: Call Trace:
May 29 11:58:22 vault1 kernel: [<ffffffff8026b5d6>] do_softirq+0x90/0x94
May 29 11:58:22 vault1 kernel: [<ffffffff8028deb7>] ksoftirqd+0x0/0xbf
May 29 11:58:22 vault1 kernel: [<ffffffff8028def2>] ksoftirqd+0x3b/0xbf
May 29 11:58:22 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:22 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:22 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:22 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:22 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:22 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:22 vault1 kernel:
May 29 11:58:22 vault1 kernel: watchdog/2 S ffff81007fe19218 0 10 1 (L-TLB)
May 29 11:58:22 vault1 kernel: ffff8100042b5ef0 0000000000000046 0000000000000000 0000000000000001
May 29 11:58:22 vault1 kernel: ffff8100042b4000 ffff81010255df00 0000000000000007 ffff81011fcce040
May 29 11:58:22 vault1 kernel: ffff81011fccf080 000000000000024f ffff81011fcce248 0000000200000001
May 29 11:58:22 vault1 kernel: Call Trace:
May 29 11:58:22 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:22 vault1 kernel: [<ffffffff802b73e8>] watchdog+0x0/0x5f
May 29 11:58:22 vault1 kernel: [<ffffffff802b7437>] watchdog+0x4f/0x5f
May 29 11:58:22 vault1 kernel: [<ffffffff802b73e8>] watchdog+0x0/0x5f
May 29 11:58:22 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:22 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:22 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:22 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:22 vault1 kernel: [<ffffffff802c038d>] next_zone+0x23/0x24
May 29 11:58:22 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:22 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:22 vault1 kernel:
May 29 11:58:22 vault1 kernel: migration/3 S ffff81007ebd24b8 0 11 1 (L-TLB)
May 29 11:58:22 vault1 kernel: ffff8100042dbec0 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:22 vault1 kernel: 0000000100000000 0000000000000002 0000000000000001 ffff81011fcd10c0
May 29 11:58:22 vault1 kernel: ffff81011fd360c0 0000000000001383 ffff81011fcd12c8 0000000300000000
May 29 11:58:22 vault1 kernel: Call Trace:
May 29 11:58:22 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:22 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:22 vault1 kernel: [<ffffffff80243daa>] migration_thread+0x1a2/0x23f
May 29 11:58:22 vault1 kernel: [<ffffffff80243c08>] migration_thread+0x0/0x23f
May 29 11:58:22 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:22 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:22 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:22 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:22 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:22 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:22 vault1 kernel:
May 29 11:58:22 vault1 kernel: ksoftirqd/3 S ffff81007e8de9d8 0 12 1 (L-TLB)
May 29 11:58:22 vault1 kernel: ffff8100042ddf00 0000000000000046 0000000000000000 ffff81011fd34040
May 29 11:58:22 vault1 kernel: ffff81007fc350c0 000000000022a171 000000000000000a ffff81011fd34040
May 29 11:58:22 vault1 kernel: ffff81011fd360c0 00000000000007f8 ffff81011fd34248 00000003042ddfd8
May 29 11:58:22 vault1 kernel: Call Trace:
May 29 11:58:22 vault1 kernel: [<ffffffff8026b5d6>] do_softirq+0x90/0x94
May 29 11:58:22 vault1 kernel: [<ffffffff8028deb7>] ksoftirqd+0x0/0xbf
May 29 11:58:22 vault1 kernel: [<ffffffff8028def2>] ksoftirqd+0x3b/0xbf
May 29 11:58:22 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:22 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:22 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:22 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:22 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:22 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:22 vault1 kernel:
May 29 11:58:23 vault1 kernel: watchdog/3 S ffff810037d4ba98 0 13 1 (L-TLB)
May 29 11:58:23 vault1 kernel: ffff8100042e1ef0 0000000000000046 0000000000000000 0000000000000001
May 29 11:58:23 vault1 kernel: ffff8100042e0000 ffff810102568d00 0000000000000001 ffff81011fd35080
May 29 11:58:23 vault1 kernel: ffff81011fd360c0 00000000000002b1 ffff81011fd35288 0000000300000001
May 29 11:58:23 vault1 kernel: Call Trace:
May 29 11:58:23 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:23 vault1 kernel: [<ffffffff802b73e8>] watchdog+0x0/0x5f
May 29 11:58:23 vault1 kernel: [<ffffffff802b7437>] watchdog+0x4f/0x5f
May 29 11:58:23 vault1 kernel: [<ffffffff802b73e8>] watchdog+0x0/0x5f
May 29 11:58:23 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:23 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:23 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:23 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:23 vault1 kernel: [<ffffffff802c038d>] next_zone+0x23/0x24
May 29 11:58:23 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:23 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:23 vault1 kernel:
May 29 11:58:23 vault1 kernel: events/0 S ffff81007e8de118 0 14 1 (L-TLB)
May 29 11:58:23 vault1 kernel: ffff81007fdfbea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:23 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011fa61040
May 29 11:58:23 vault1 kernel: ffffffff8058a6e0 0000000000009929 ffff81011fa61248 000000001fa61040
May 29 11:58:23 vault1 kernel: Call Trace:
May 29 11:58:23 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:23 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:23 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:23 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:23 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:23 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:23 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:23 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:23 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:23 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:23 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:23 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:23 vault1 kernel:
May 29 11:58:23 vault1 kernel: events/1 S ffff81007e837b18 0 15 1 (L-TLB)
May 29 11:58:23 vault1 kernel: ffff810004303ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:23 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011fa63080
May 29 11:58:23 vault1 kernel: ffff81011fc78040 000000000000aa1b ffff81011fa63288 000000011fa63080
May 29 11:58:23 vault1 kernel: Call Trace:
May 29 11:58:23 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:23 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:23 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:23 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:23 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:23 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:23 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:23 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:23 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:23 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:23 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:23 vault1 kernel:
May 29 11:58:23 vault1 kernel: events/2 S ffff81007ebd3638 0 16 1 (L-TLB)
May 29 11:58:23 vault1 kernel: ffff81007fde9ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:23 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011fa640c0
May 29 11:58:23 vault1 kernel: ffff81011fccf080 000000000001434a ffff81011fa642c8 000000021fa640c0
May 29 11:58:23 vault1 kernel: Call Trace:
May 29 11:58:23 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:23 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:23 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:23 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:23 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:23 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:23 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:23 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:23 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:23 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:23 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:23 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:23 vault1 kernel:
May 29 11:58:23 vault1 kernel: events/3 S ffff810037d4ba98 0 17 1 (L-TLB)
May 29 11:58:23 vault1 kernel: ffff8100042bfea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:23 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011fa65040
May 29 11:58:23 vault1 kernel: ffff81011fd360c0 000000000000f2ba ffff81011fa65248 000000031fa65040
May 29 11:58:23 vault1 kernel: Call Trace:
May 29 11:58:23 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:23 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:23 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:23 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:23 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:23 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:23 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:23 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:23 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:23 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:23 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:23 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:23 vault1 kernel:
May 29 11:58:23 vault1 kernel: khelper S ffff81007f114118 0 18 1 (L-TLB)
May 29 11:58:23 vault1 kernel: ffff810037d39ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:23 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011fa66080
May 29 11:58:23 vault1 kernel: ffff81011f2ce080 0000000000007797 ffff81011fa66288 000000001fa66080
May 29 11:58:24 vault1 kernel: Call Trace:
May 29 11:58:24 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:24 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:24 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:24 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:24 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:24 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:24 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:24 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:24 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:24 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:24 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:24 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:24 vault1 kernel:
May 29 11:58:24 vault1 kernel: kthread S ffff81007f1149d8 0 19 1 (L-TLB)
May 29 11:58:24 vault1 kernel: ffff81011fa6fea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:24 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81007fdf3080
May 29 11:58:24 vault1 kernel: ffff81011fc78040 000000000000199a ffff81007fdf3288 000000017fdf3080
May 29 11:58:24 vault1 kernel: Call Trace:
May 29 11:58:24 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:24 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:24 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:24 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:24 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:24 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:24 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:24 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:24 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:24 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:24 vault1 kernel:
May 29 11:58:24 vault1 kernel: kblockd/0 S ffff81011f2b8958 0 59 19 (L-TLB)
May 29 11:58:24 vault1 kernel: ffff81011fe7fea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:24 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011fe24040
May 29 11:58:24 vault1 kernel: ffffffff8058a6e0 000000000000324c ffff81011fe24248 000000001fe24040
May 29 11:58:24 vault1 kernel: Call Trace:
May 29 11:58:24 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:24 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:24 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:24 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:24 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:24 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:24 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:24 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:24 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:24 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:24 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:24 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:24 vault1 kernel:
May 29 11:58:24 vault1 kernel: kblockd/1 S ffff81011f2b8958 0 60 19 (L-TLB)
May 29 11:58:24 vault1 kernel: ffff81011fb81ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:24 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011fe23080
May 29 11:58:24 vault1 kernel: ffff81011fc78040 000000000000334b ffff81011fe23288 000000011fe23080
May 29 11:58:24 vault1 kernel: Call Trace:
May 29 11:58:24 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:24 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:24 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:24 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:24 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:24 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:24 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:24 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:24 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:24 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:24 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:24 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:24 vault1 kernel:
May 29 11:58:24 vault1 kernel: kblockd/2 S ffff81007e8de9d8 0 61 19 (L-TLB)
May 29 11:58:24 vault1 kernel: ffff81011fb83ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:24 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011fe220c0
May 29 11:58:24 vault1 kernel: ffff81007e4c5080 000000000001120f ffff81011fe222c8 000000021fe220c0
May 29 11:58:24 vault1 kernel: Call Trace:
May 29 11:58:24 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:24 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:24 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:24 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:24 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:24 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:24 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:24 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:24 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:24 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:24 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:24 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:24 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:24 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:24 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:24 vault1 kernel:
May 29 11:58:24 vault1 kernel: kblockd/3 S ffff81007e8de9d8 0 62 19 (L-TLB)
May 29 11:58:25 vault1 kernel: ffff81007fde5ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:25 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff810037e3e0c0
May 29 11:58:25 vault1 kernel: ffff81011fd360c0 0000000000007674 ffff810037e3e2c8 0000000337e3e0c0
May 29 11:58:25 vault1 kernel: Call Trace:
May 29 11:58:25 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:25 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:25 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:25 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:25 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:25 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:25 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:25 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:25 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:25 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:25 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:25 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:25 vault1 kernel:
May 29 11:58:25 vault1 kernel: kacpid S ffffffff8058a280 0 63 19 (L-TLB)
May 29 11:58:25 vault1 kernel: ffff8100042d1ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:25 vault1 kernel: 0000000000000000 0000000000000000 0000000000000001 ffff810004281040
May 29 11:58:25 vault1 kernel: ffff81011fccf080 0000000000000ff3 ffff810004281248 0000000204281040
May 29 11:58:25 vault1 kernel: Call Trace:
May 29 11:58:25 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:25 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:25 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:25 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:25 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:25 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:25 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:25 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:25 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:25 vault1 kernel: [<ffffffff8031ff4a>] selinux_d_instantiate+0x0/0x14
May 29 11:58:25 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:25 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:25 vault1 kernel:
May 29 11:58:25 vault1 kernel: cqueue/0 S ffffffff8058a280 0 217 19 (L-TLB)
May 29 11:58:25 vault1 kernel: ffff81007fdd5ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:25 vault1 kernel: 0000000000000000 0000000000000000 0000000000000001 ffff81007fdca080
May 29 11:58:25 vault1 kernel: ffff81007fdf3080 0000000000001336 ffff81007fdca288 000000007fdca080
May 29 11:58:25 vault1 kernel: Call Trace:
May 29 11:58:25 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:25 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:25 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:25 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:25 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:25 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:25 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:25 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:25 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:25 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:25 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:25 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:25 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:25 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:25 vault1 kernel:
May 29 11:58:25 vault1 kernel: cqueue/1 S ffffffff8058a280 0 218 19 (L-TLB)
May 29 11:58:25 vault1 kernel: ffff81007fdd7ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:25 vault1 kernel: 0000000000000000 0000000000000000 0000000000000001 ffff81007fdcb0c0
May 29 11:58:25 vault1 kernel: ffff81011fc78040 00000000000014e8 ffff81007fdcb2c8 000000017fdcb0c0
May 29 11:58:25 vault1 kernel: Call Trace:
May 29 11:58:25 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:25 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:25 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:25 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:25 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:25 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:25 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:25 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:25 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:25 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:25 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:25 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:25 vault1 kernel:
May 29 11:58:25 vault1 kernel: cqueue/2 S ffffffff8058a280 0 219 19 (L-TLB)
May 29 11:58:25 vault1 kernel: ffff81007fdd9ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:25 vault1 kernel: 0000000000000000 0000000000000000 0000000000000001 ffff81007fdcd040
May 29 11:58:25 vault1 kernel: ffff81011fccf080 00000000000012c0 ffff81007fdcd248 000000027fdcd040
May 29 11:58:25 vault1 kernel: Call Trace:
May 29 11:58:25 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:25 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:25 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:25 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:25 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:25 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:26 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:26 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:26 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:26 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:26 vault1 kernel:
May 29 11:58:26 vault1 kernel: cqueue/3 S ffffffff8058a280 0 220 19 (L-TLB)
May 29 11:58:26 vault1 kernel: ffff81007fdddea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:26 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81007fdce080
May 29 11:58:26 vault1 kernel: ffff81011fd360c0 0000000000001847 ffff81007fdce288 000000037fdce080
May 29 11:58:26 vault1 kernel: Call Trace:
May 29 11:58:26 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:26 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:26 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:26 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:26 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:26 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:26 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:26 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:26 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:26 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:26 vault1 kernel:
May 29 11:58:26 vault1 kernel: ksuspend_usbd S ffffffff8058a280 0 221 19 (L-TLB)
May 29 11:58:26 vault1 kernel: ffff81007fddfea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:26 vault1 kernel: 0000000000000000 0000000000000000 0000000000000001 ffff81007fdd00c0
May 29 11:58:26 vault1 kernel: ffff81011fccf080 00000000000011b5 ffff81007fdd02c8 000000027fdd00c0
May 29 11:58:26 vault1 kernel: Call Trace:
May 29 11:58:26 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:26 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:26 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:26 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:26 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:26 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:26 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:26 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:26 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:26 vault1 kernel:
May 29 11:58:26 vault1 kernel: khubd S ffff81011f96fa98 0 224 19 (L-TLB)
May 29 11:58:26 vault1 kernel: ffff810004361e20 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:26 vault1 kernel: 0000000000000000 0000000000000046 0000000000000009 ffff810004342040
May 29 11:58:26 vault1 kernel: ffff81011fc72080 00000000007498c5 ffff810004342248 0000000004342040
May 29 11:58:26 vault1 kernel: Call Trace:
May 29 11:58:26 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:26 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:26 vault1 kernel: [<ffffffff803d6dc8>] hub_thread+0xac0/0xb3c
May 29 11:58:26 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:26 vault1 kernel: [<ffffffff803d6308>] hub_thread+0x0/0xb3c
May 29 11:58:26 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:26 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:26 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:26 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:26 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:26 vault1 kernel:
May 29 11:58:26 vault1 kernel: kseriod S ffff810037d4ba98 0 226 19 (L-TLB)
May 29 11:58:26 vault1 kernel: ffff810004365ec0 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:26 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff810004344080
May 29 11:58:26 vault1 kernel: ffffffff8058a6e0 000000000000270d ffff810004344288 0000000004344080
May 29 11:58:26 vault1 kernel: Call Trace:
May 29 11:58:26 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:26 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:26 vault1 kernel: [<ffffffff803eaaea>] serio_thread+0x2a5/0x2f8
May 29 11:58:26 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:26 vault1 kernel: [<ffffffff803ea845>] serio_thread+0x0/0x2f8
May 29 11:58:26 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:26 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:26 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:26 vault1 kernel: [<ffffffff8031ff4a>] selinux_d_instantiate+0x0/0x14
May 29 11:58:26 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:26 vault1 kernel:
May 29 11:58:26 vault1 kernel: kswapd0 S ffff81007e836538 0 283 19 (L-TLB)
May 29 11:58:26 vault1 kernel: ffff810037e6fe20 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:26 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff810037fe8080
May 29 11:58:26 vault1 kernel: ffff81006b4ed040 00000000000b1784 ffff810037fe8288 0000000037fe8080
May 29 11:58:26 vault1 kernel: Call Trace:
May 29 11:58:26 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:26 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:26 vault1 kernel: [<ffffffff802560c6>] kswapd+0xea/0x469
May 29 11:58:26 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:26 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:26 vault1 kernel: [<ffffffff80255fdc>] kswapd+0x0/0x469
May 29 11:58:26 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:26 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:26 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:26 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:26 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:26 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:26 vault1 kernel:
May 29 11:58:27 vault1 kernel: kswapd1 S ffff81011febc958 0 284 19 (L-TLB)
May 29 11:58:27 vault1 kernel: ffff81007fce9e20 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:27 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff810037fe70c0
May 29 11:58:27 vault1 kernel: ffff81007af01080 00000000000debec ffff810037fe72c8 0000000237fe70c0
May 29 11:58:27 vault1 kernel: Call Trace:
May 29 11:58:27 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:27 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:27 vault1 kernel: [<ffffffff802560c6>] kswapd+0xea/0x469
May 29 11:58:27 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:27 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:27 vault1 kernel: [<ffffffff80255fdc>] kswapd+0x0/0x469
May 29 11:58:27 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:27 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:27 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:27 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:27 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:27 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:27 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:27 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:27 vault1 kernel:
May 29 11:58:27 vault1 kernel: aio/0 S ffffffff8058a280 0 285 19 (L-TLB)
May 29 11:58:27 vault1 kernel: ffff81007fcedea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:27 vault1 kernel: 0000000000000000 0000000000000000 0000000000000003 ffff810037fe6040
May 29 11:58:27 vault1 kernel: ffff81007fdf3080 0000000000001520 ffff810037fe6248 0000000037fe6040
May 29 11:58:27 vault1 kernel: Call Trace:
May 29 11:58:27 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:27 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:27 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:27 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:27 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:27 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:27 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:27 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:27 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:27 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:27 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:27 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:27 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:27 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:27 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:27 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:27 vault1 kernel:
May 29 11:58:27 vault1 kernel: aio/1 S ffffffff8058a280 0 286 19 (L-TLB)
May 29 11:58:27 vault1 kernel: ffff810037f41ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:27 vault1 kernel: 0000000000000000 0000000000000000 0000000000000002 ffff810037fe4080
May 29 11:58:27 vault1 kernel: ffff81011fc78040 0000000000001723 ffff810037fe4288 0000000137fe4080
May 29 11:58:27 vault1 kernel: Call Trace:
May 29 11:58:27 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:27 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:27 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:27 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:27 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:27 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:27 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:27 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:27 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:27 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:27 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:27 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:27 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:27 vault1 kernel:
May 29 11:58:27 vault1 kernel: aio/2 S ffffffff8058a280 0 287 19 (L-TLB)
May 29 11:58:27 vault1 kernel: ffff810037f43ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:27 vault1 kernel: 0000000000000000 0000000000000000 0000000000000002 ffff810037fe30c0
May 29 11:58:27 vault1 kernel: ffff81011fccf080 0000000000001b18 ffff810037fe32c8 0000000237fe30c0
May 29 11:58:27 vault1 kernel: Call Trace:
May 29 11:58:27 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:27 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:27 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:27 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:27 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:27 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:27 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:27 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:27 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:27 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:27 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:27 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:27 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:27 vault1 kernel:
May 29 11:58:27 vault1 kernel: aio/3 S ffffffff8058a280 0 288 19 (L-TLB)
May 29 11:58:27 vault1 kernel: ffff810037f45ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:27 vault1 kernel: 0000000000000000 0000000000000000 0000000000000003 ffff810037fe2040
May 29 11:58:28 vault1 kernel: ffff81011fd360c0 0000000000001406 ffff810037fe2248 0000000337fe2040
May 29 11:58:28 vault1 kernel: Call Trace:
May 29 11:58:28 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:28 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:28 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:28 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:28 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:28 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:28 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:28 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:28 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:28 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:28 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:28 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:28 vault1 kernel:
May 29 11:58:28 vault1 kernel: kpsmoused S ffffffff8058a280 0 446 19 (L-TLB)
May 29 11:58:28 vault1 kernel: ffff810037d59ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:28 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff810037f81080
May 29 11:58:28 vault1 kernel: ffffffff8058a6e0 0000000000000d43 ffff810037f81288 0000000037f81080
May 29 11:58:28 vault1 kernel: Call Trace:
May 29 11:58:28 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:28 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:28 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:28 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:28 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:28 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:28 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:28 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:28 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:28 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:28 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:28 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:28 vault1 kernel:
May 29 11:58:28 vault1 kernel: scsi_eh_0 S ffff810037d4b638 0 499 19 (L-TLB)
May 29 11:58:28 vault1 kernel: ffff81007f41fe70 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:28 vault1 kernel: 0000000000000000 0000000000000046 0000000000000004 ffff810037f69040
May 29 11:58:28 vault1 kernel: ffff81011fccf080 0000000000000c44 ffff810037f69248 0000000280263894
May 29 11:58:28 vault1 kernel: Call Trace:
May 29 11:58:28 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:28 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 29 11:58:28 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 29 11:58:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:28 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:28 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:28 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:28 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:28 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:28 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:28 vault1 kernel:
May 29 11:58:28 vault1 kernel: kjournald S ffff81007f1149d8 0 535 19 (L-TLB)
May 29 11:58:28 vault1 kernel: ffff81007f79bec0 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:28 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff810037d50080
May 29 11:58:28 vault1 kernel: ffff81011fc78040 0000000000000b58 ffff810037d50288 000000018029fc0f
May 29 11:58:28 vault1 kernel: Call Trace:
May 29 11:58:28 vault1 kernel: [<ffffffff880279be>] :jbd:kjournald+0x18e/0x21d
May 29 11:58:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:28 vault1 kernel: [<ffffffff880279c3>] :jbd:kjournald+0x193/0x21d
May 29 11:58:28 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:28 vault1 kernel: [<ffffffff88027830>] :jbd:kjournald+0x0/0x21d
May 29 11:58:28 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:28 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:28 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:28 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:28 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:28 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:28 vault1 kernel:
May 29 11:58:28 vault1 kernel: kauditd S ffff810037d4a4b8 0 557 19 (L-TLB)
May 29 11:58:28 vault1 kernel: ffff81007f3a9ec0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:28 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81007f19b040
May 29 11:58:28 vault1 kernel: ffff81007f32f0c0 000000000000750a ffff81007f19b248 000000007f19b040
May 29 11:58:28 vault1 kernel: Call Trace:
May 29 11:58:28 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:28 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:28 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:28 vault1 kernel: [<ffffffff802b2b39>] kauditd_thread+0x111/0x148
May 29 11:58:28 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:28 vault1 kernel: [<ffffffff802b2a28>] kauditd_thread+0x0/0x148
May 29 11:58:28 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:28 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:28 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:29 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:29 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:29 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:29 vault1 kernel:
May 29 11:58:29 vault1 kernel: udevd S ffff81007f343638 0 586 1 (NOTLB)
May 29 11:58:29 vault1 kernel: ffff81007fe1ba18 0000000000000046 0000000000000000 0000000000000046
May 29 11:58:29 vault1 kernel: ffffffff802ea6a4 ffff81007fe107f8 0000000000000005 ffff81007fe100c0
May 29 11:58:29 vault1 kernel: ffff81011fc78040 00000000000282bc ffff81007fe102c8 0000000180262123
May 29 11:58:29 vault1 kernel: Call Trace:
May 29 11:58:29 vault1 kernel: [<ffffffff802ea6a4>] inotify_poll+0x31/0x56
May 29 11:58:29 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:29 vault1 kernel: [<ffffffff802ea6c2>] inotify_poll+0x4f/0x56
May 29 11:58:29 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 29 11:58:29 vault1 kernel: [<ffffffff80217eea>] cache_grow+0x31f/0x378
May 29 11:58:29 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:29 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:29 vault1 last message repeated 3 times
May 29 11:58:29 vault1 kernel: [<ffffffff8023e153>] lookup_mnt+0x19/0x4c
May 29 11:58:29 vault1 kernel: [<ffffffff8023e17e>] lookup_mnt+0x44/0x4c
May 29 11:58:29 vault1 kernel: [<ffffffff8020d542>] dput+0x3c/0x12b
May 29 11:58:29 vault1 kernel: [<ffffffff80219f0b>] __follow_mount+0x26/0x7b
May 29 11:58:29 vault1 kernel: [<ffffffff8020cff9>] do_lookup+0x63/0x1ae
May 29 11:58:29 vault1 kernel: [<ffffffff802119b8>] do_wp_page+0x5af/0x60b
May 29 11:58:29 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 29 11:58:29 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 29 11:58:29 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:29 vault1 kernel: [<ffffffff8023bee1>] remove_wait_queue+0x10/0x2c
May 29 11:58:29 vault1 kernel: [<ffffffff8023bee1>] remove_wait_queue+0x10/0x2c
May 29 11:58:29 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:29 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:29 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 29 11:58:29 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:29 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:29 vault1 kernel:
May 29 11:58:29 vault1 kernel: khpsbpkt S ffff810037d4a4b8 0 1142 19 (L-TLB)
May 29 11:58:29 vault1 kernel: ffff81007e7a5f00 0000000000000046 ffffffff80246747 0000000000000002
May 29 11:58:29 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81007fd63040
May 29 11:58:29 vault1 kernel: ffff81007e1680c0 0000000000001b49 ffff81007fd63248 000000001f3cbd30
May 29 11:58:29 vault1 kernel: Call Trace:
May 29 11:58:29 vault1 kernel: [<ffffffff80246747>] skb_dequeue+0x13/0x50
May 29 11:58:29 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:29 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:29 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:29 vault1 kernel: [<ffffffff88105239>] :ieee1394:hpsbpkt_thread+0x0/0x8d
May 29 11:58:29 vault1 kernel: [<ffffffff881052a9>] :ieee1394:hpsbpkt_thread+0x70/0x8d
May 29 11:58:29 vault1 kernel: [<ffffffff88105239>] :ieee1394:hpsbpkt_thread+0x0/0x8d
May 29 11:58:29 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:29 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:29 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:29 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:29 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:29 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:29 vault1 kernel:
May 29 11:58:29 vault1 kernel: knodemgrd_0 S ffff81007e4304f8 0 1204 19 (L-TLB)
May 29 11:58:29 vault1 kernel: ffff81007e1f5e70 0000000000000046 ffff81007e883080 ffffffff80261ece
May 29 11:58:29 vault1 kernel: 0000000000000246 ffff81007e7e4378 0000000000000009 ffff81007e883080
May 29 11:58:29 vault1 kernel: ffff810037f5d040 00000000000e6dc7 ffff81007e883288 0000000180261ece
May 29 11:58:29 vault1 kernel: Call Trace:
May 29 11:58:29 vault1 kernel: [<ffffffff80261ece>] __mutex_unlock_slowpath+0x128/0x133
May 29 11:58:29 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:29 vault1 kernel: [<ffffffff88109d00>] :ieee1394:nodemgr_host_thread+0x87/0xa58
May 29 11:58:29 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:29 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:29 vault1 kernel: [<ffffffff88109c79>] :ieee1394:nodemgr_host_thread+0x0/0xa58
May 29 11:58:29 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:29 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:29 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:29 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:29 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:29 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:29 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:29 vault1 kernel:
May 29 11:58:29 vault1 kernel: ata/0 S ffff81007ebd24b8 0 1289 19 (L-TLB)
May 29 11:58:29 vault1 kernel: ffff81011e58fea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:29 vault1 kernel: 0000000000000000 0000000000000000 0000000000000004 ffff81011fe3d080
May 29 11:58:29 vault1 kernel: ffff81007e917040 0000000000002176 ffff81011fe3d288 000000001fe3d080
May 29 11:58:29 vault1 kernel: Call Trace:
May 29 11:58:29 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:29 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:29 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:29 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:29 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:29 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:29 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:29 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:29 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:29 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:29 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:29 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:29 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:29 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:29 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:29 vault1 kernel:
May 29 11:58:30 vault1 kernel: ata/1 S ffff81007f343638 0 1290 19 (L-TLB)
May 29 11:58:30 vault1 kernel: ffff81011e505ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:30 vault1 kernel: 0000000000000000 0000000000000000 0000000000000004 ffff81011fe34080
May 29 11:58:30 vault1 kernel: ffff81007fe100c0 00000000000020c5 ffff81011fe34288 000000011fe34080
May 29 11:58:30 vault1 kernel: Call Trace:
May 29 11:58:30 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:30 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:30 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:30 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:30 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:30 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:30 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:30 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:30 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:30 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:30 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:30 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:30 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:30 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:30 vault1 kernel:
May 29 11:58:30 vault1 kernel: ata/2 S ffff81011fe0b1d8 0 1291 19 (L-TLB)
May 29 11:58:30 vault1 kernel: ffff81011e69fea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:30 vault1 kernel: 0000000000000000 0000000000000000 0000000000000004 ffff81011f35c080
May 29 11:58:30 vault1 kernel: ffff81011edcd0c0 00000000000012dc ffff81011f35c288 000000021f35c080
May 29 11:58:30 vault1 kernel: Call Trace:
May 29 11:58:30 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:30 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:30 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:30 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:30 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:30 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:30 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:30 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:30 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:30 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:30 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:30 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:30 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:30 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:30 vault1 kernel:
May 29 11:58:30 vault1 kernel: ata/3 S ffff81011edb5b58 0 1292 19 (L-TLB)
May 29 11:58:30 vault1 kernel: ffff81011e6abea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:30 vault1 kernel: 0000000000000000 0000000000000000 0000000000000003 ffff81011edcd0c0
May 29 11:58:30 vault1 kernel: ffff81011f0330c0 00000000000026f9 ffff81011edcd2c8 000000031edcd0c0
May 29 11:58:30 vault1 kernel: Call Trace:
May 29 11:58:30 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:30 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:30 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:30 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:30 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:30 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:30 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:30 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:30 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:30 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:30 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:30 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:30 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:30 vault1 kernel:
May 29 11:58:30 vault1 kernel: ata_aux S ffff81011fe0b1d8 0 1293 19 (L-TLB)
May 29 11:58:30 vault1 kernel: ffff81011e6d3ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:30 vault1 kernel: 0000000000000000 0000000000000000 0000000000000004 ffff81011fe3e040
May 29 11:58:30 vault1 kernel: ffff81011eec3080 0000000000001260 ffff81011fe3e248 000000021fe3e040
May 29 11:58:30 vault1 kernel: Call Trace:
May 29 11:58:30 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:30 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:30 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:30 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:30 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:30 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:30 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:30 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:30 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:30 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:30 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:30 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:30 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:30 vault1 kernel:
May 29 11:58:30 vault1 kernel: scsi_eh_5 S ffff81007e8df6f8 0 1315 19 (L-TLB)
May 29 11:58:30 vault1 kernel: ffff81011ee35e70 0000000000000046 ffff81011f2e3040 ffffffff802638fc
May 29 11:58:30 vault1 kernel: ffff81007f3a4000 0000000000000000 0000000000000005 ffff81011f2e3040
May 29 11:58:30 vault1 kernel: ffff81011fe2c0c0 00000000009d77e9 ffff81011f2e3248 0000000000000282
May 29 11:58:31 vault1 kernel: Call Trace:
May 29 11:58:31 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:31 vault1 kernel: [<ffffffff88060287>] :scsi_mod:__scsi_iterate_devices+0x5c/0x75
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:31 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:31 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:31 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:31 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:31 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:31 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:31 vault1 kernel:
May 29 11:58:31 vault1 kernel: scsi_eh_6 S ffff81007f343638 0 1316 19 (L-TLB)
May 29 11:58:31 vault1 kernel: ffff81011fecbe70 0000000000000046 ffff81011ec6d080 ffffffff802638fc
May 29 11:58:31 vault1 kernel: ffff81007e2ec000 0000000000000000 0000000000000008 ffff81011ec6d080
May 29 11:58:31 vault1 kernel: ffff81011fd4a040 00000000009d51e7 ffff81011ec6d288 0000000200000282
May 29 11:58:31 vault1 kernel: Call Trace:
May 29 11:58:31 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:31 vault1 kernel: [<ffffffff88060287>] :scsi_mod:__scsi_iterate_devices+0x5c/0x75
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:31 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:31 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:31 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:31 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:31 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:31 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:31 vault1 kernel:
May 29 11:58:31 vault1 kernel: scsi_eh_7 S ffff81007fe19ad8 0 1741 19 (L-TLB)
May 29 11:58:31 vault1 kernel: ffff81011e8dde70 0000000000000046 ffff81011ec7f0c0 ffffffff802638fc
May 29 11:58:31 vault1 kernel: ffff81007fd5c000 0000000000000000 0000000000000008 ffff81011ec7f0c0
May 29 11:58:31 vault1 kernel: ffff81011fd4a040 00000000009d3031 ffff81011ec7f2c8 0000000200000282
May 29 11:58:31 vault1 kernel: Call Trace:
May 29 11:58:31 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:31 vault1 kernel: [<ffffffff88060287>] :scsi_mod:__scsi_iterate_devices+0x5c/0x75
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:31 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:31 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:31 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:31 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:31 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:31 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:31 vault1 kernel:
May 29 11:58:31 vault1 kernel: scsi_eh_8 S ffff81011f1a5b18 0 1742 19 (L-TLB)
May 29 11:58:31 vault1 kernel: ffff81011e4ffe70 0000000000000046 ffff81011ff03080 ffffffff802638fc
May 29 11:58:31 vault1 kernel: ffff81007e4ac000 0000000000000000 000000000000000a ffff81011ff03080
May 29 11:58:31 vault1 kernel: ffff81011fd4a040 00000000009d292b ffff81011ff03288 0000000200000282
May 29 11:58:31 vault1 kernel: Call Trace:
May 29 11:58:31 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:31 vault1 kernel: [<ffffffff88060287>] :scsi_mod:__scsi_iterate_devices+0x5c/0x75
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:31 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:31 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:31 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:31 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:31 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:31 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:31 vault1 kernel:
May 29 11:58:31 vault1 kernel: scsi_eh_9 S ffff81007f343638 0 2015 19 (L-TLB)
May 29 11:58:31 vault1 kernel: ffff81011f3f9e70 0000000000000046 ffff81011f030080 ffffffff802638fc
May 29 11:58:31 vault1 kernel: ffff81007e120000 0000000000000000 000000000000000a ffff81011f030080
May 29 11:58:31 vault1 kernel: ffff81011fd4a040 00000000009d3f30 ffff81011f030288 0000000200000282
May 29 11:58:31 vault1 kernel: Call Trace:
May 29 11:58:31 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:31 vault1 kernel: [<ffffffff88060287>] :scsi_mod:__scsi_iterate_devices+0x5c/0x75
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:31 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:31 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:31 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:32 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:32 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:32 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:32 vault1 kernel:
May 29 11:58:32 vault1 kernel: scsi_eh_10 S ffff810037d4ad78 0 2016 19 (L-TLB)
May 29 11:58:32 vault1 kernel: ffff81011e7fde70 0000000000000046 ffff81011ffe00c0 ffffffff802638fc
May 29 11:58:32 vault1 kernel: ffff81007e288000 0000000000000000 000000000000000a ffff81011ffe00c0
May 29 11:58:32 vault1 kernel: ffff81007f22b080 00000000009c774d ffff81011ffe02c8 0000000000000282
May 29 11:58:32 vault1 kernel: Call Trace:
May 29 11:58:32 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:32 vault1 kernel: [<ffffffff88060287>] :scsi_mod:__scsi_iterate_devices+0x5c/0x75
May 29 11:58:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:32 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 29 11:58:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:32 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:32 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 29 11:58:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:32 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:32 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:32 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:32 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:32 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:32 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:32 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:32 vault1 kernel:
May 29 11:58:32 vault1 kernel: kmpathd/0 S ffff81011f96e918 0 2243 19 (L-TLB)
May 29 11:58:32 vault1 kernel: ffff81007dc5dea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:32 vault1 kernel: 0000000000000000 0000000000000000 0000000000000001 ffff81007e125040
May 29 11:58:32 vault1 kernel: ffff81007fdf3080 0000000000000d28 ffff81007e125248 000000007e125040
May 29 11:58:32 vault1 kernel: Call Trace:
May 29 11:58:32 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:32 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:32 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:32 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:32 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:32 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:32 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:32 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:32 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:32 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:32 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:32 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:32 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:32 vault1 kernel:
May 29 11:58:32 vault1 kernel: kmpathd/1 S ffff81007e836998 0 2244 19 (L-TLB)
May 29 11:58:32 vault1 kernel: ffff81007dc5fea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:32 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81007e962040
May 29 11:58:32 vault1 kernel: ffff81011fc78040 0000000000001a16 ffff81007e962248 000000017e962040
May 29 11:58:32 vault1 kernel: Call Trace:
May 29 11:58:32 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:32 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:32 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:32 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:32 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:32 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:32 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:32 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:32 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:32 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:32 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:32 vault1 kernel:
May 29 11:58:32 vault1 kernel: kmpathd/2 S ffff81007f343638 0 2245 19 (L-TLB)
May 29 11:58:32 vault1 kernel: ffff81007dea1ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:32 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81007e5ae080
May 29 11:58:32 vault1 kernel: ffff81007fe100c0 0000000000002189 ffff81007e5ae288 000000027e5ae080
May 29 11:58:32 vault1 kernel: Call Trace:
May 29 11:58:32 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:32 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:32 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:32 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:32 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:32 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:32 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:32 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:32 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:32 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:32 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:32 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:32 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:32 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:32 vault1 kernel:
May 29 11:58:32 vault1 kernel: kmpathd/3 S ffff81011f96e918 0 2246 19 (L-TLB)
May 29 11:58:32 vault1 kernel: ffff81007dea3ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:32 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81007f22b080
May 29 11:58:32 vault1 kernel: ffff81011fd360c0 00000000000032a3 ffff81007f22b288 000000037f22b080
May 29 11:58:32 vault1 kernel: Call Trace:
May 29 11:58:32 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:33 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:33 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:33 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:33 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:33 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:33 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:33 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:33 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:33 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:33 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:33 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:33 vault1 kernel:
May 29 11:58:33 vault1 kernel: kmirrord S ffff81007e837b18 0 2261 19 (L-TLB)
May 29 11:58:33 vault1 kernel: ffff81007dfbfea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:33 vault1 kernel: 0000000000000000 0000000000000000 0000000000000001 ffff81007e9c7040
May 29 11:58:33 vault1 kernel: ffff81007e9970c0 0000000000000c37 ffff81007e9c7248 000000007e9c7040
May 29 11:58:33 vault1 kernel: Call Trace:
May 29 11:58:33 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:33 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:33 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:33 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:33 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:33 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:33 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:33 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:33 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:33 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:33 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:33 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:33 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:33 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:33 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:33 vault1 kernel:
May 29 11:58:33 vault1 kernel: jfsIO S ffff81007e8de9d8 0 2299 19 (L-TLB)
May 29 11:58:33 vault1 kernel: ffff81007c859f00 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:33 vault1 kernel: 000000017c858000 0000000000000002 000000000000000a ffff81007e54c040
May 29 11:58:33 vault1 kernel: ffff81011fccf080 00000000000074c3 ffff81007e54c248 0000000200000000
May 29 11:58:33 vault1 kernel: Call Trace:
May 29 11:58:33 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:33 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:33 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:33 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:33 vault1 kernel: [<ffffffff88242385>] :jfs:jfsIOWait+0x0/0xa2
May 29 11:58:33 vault1 kernel: [<ffffffff8824240a>] :jfs:jfsIOWait+0x85/0xa2
May 29 11:58:33 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:33 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:33 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:33 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:33 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:33 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:33 vault1 kernel:
May 29 11:58:33 vault1 kernel: jfsCommit S ffff81007e8de9d8 0 2300 19 (L-TLB)
May 29 11:58:33 vault1 kernel: ffff81007c85beb0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:33 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81007e1680c0
May 29 11:58:33 vault1 kernel: ffff81007e4620c0 000000000000e4f1 ffff81007e1682c8 000000027e1680c0
May 29 11:58:33 vault1 kernel: Call Trace:
May 29 11:58:33 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:33 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:33 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:33 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:33 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:33 vault1 kernel: [<ffffffff88245114>] :jfs:jfs_lazycommit+0x1d9/0x212
May 29 11:58:33 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:33 vault1 kernel: [<ffffffff88244f3b>] :jfs:jfs_lazycommit+0x0/0x212
May 29 11:58:33 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:33 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:33 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:33 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:33 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:33 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:33 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:33 vault1 kernel:
May 29 11:58:33 vault1 kernel: jfsCommit S ffff81007e8de9d8 0 2301 19 (L-TLB)
May 29 11:58:33 vault1 kernel: ffff81007c85deb0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:34 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81007e4620c0
May 29 11:58:34 vault1 kernel: ffff81007ef410c0 000000000000097b ffff81007e4622c8 000000027e4620c0
May 29 11:58:34 vault1 kernel: Call Trace:
May 29 11:58:34 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:34 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:34 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:34 vault1 kernel: [<ffffffff88245114>] :jfs:jfs_lazycommit+0x1d9/0x212
May 29 11:58:34 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:34 vault1 kernel: [<ffffffff88244f3b>] :jfs:jfs_lazycommit+0x0/0x212
May 29 11:58:34 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:34 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:34 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:34 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:34 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:34 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:34 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:34 vault1 kernel:
May 29 11:58:34 vault1 kernel: jfsCommit S ffff81007e8de9d8 0 2302 19 (L-TLB)
May 29 11:58:34 vault1 kernel: ffff81007c85feb0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:34 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81007e5e50c0
May 29 11:58:34 vault1 kernel: ffff81007e0c80c0 0000000000000719 ffff81007e5e52c8 000000027e5e50c0
May 29 11:58:34 vault1 kernel: Call Trace:
May 29 11:58:34 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:34 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:34 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:34 vault1 kernel: [<ffffffff88245114>] :jfs:jfs_lazycommit+0x1d9/0x212
May 29 11:58:34 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:34 vault1 kernel: [<ffffffff88244f3b>] :jfs:jfs_lazycommit+0x0/0x212
May 29 11:58:34 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:34 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:34 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:34 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:34 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:34 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:34 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:34 vault1 kernel:
May 29 11:58:34 vault1 kernel: jfsCommit S ffff81007e8de9d8 0 2303 19 (L-TLB)
May 29 11:58:34 vault1 kernel: ffff81007c861eb0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:34 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81007ef410c0
May 29 11:58:34 vault1 kernel: ffff81007e5e50c0 000000000000071c ffff81007ef412c8 000000027ef410c0
May 29 11:58:34 vault1 kernel: Call Trace:
May 29 11:58:34 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:34 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:34 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:34 vault1 kernel: [<ffffffff88245114>] :jfs:jfs_lazycommit+0x1d9/0x212
May 29 11:58:34 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:34 vault1 kernel: [<ffffffff88244f3b>] :jfs:jfs_lazycommit+0x0/0x212
May 29 11:58:34 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:34 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:34 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:34 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:34 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:34 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:34 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:34 vault1 kernel:
May 29 11:58:34 vault1 kernel: jfsSync S ffff81007f343638 0 2304 19 (L-TLB)
May 29 11:58:34 vault1 kernel: ffff81007c863ee0 0000000000000046 0000000000000000 ffff81000438c040
May 29 11:58:34 vault1 kernel: 000000000000018e 0000000000000001 0000000000000007 ffff81000438c040
May 29 11:58:34 vault1 kernel: ffff81011fccf080 000000000000d778 ffff81000438c248 0000000200000246
May 29 11:58:34 vault1 kernel: Call Trace:
May 29 11:58:34 vault1 kernel: [<ffffffff882448ef>] :jfs:jfs_sync+0x181/0x1ac
May 29 11:58:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:34 vault1 kernel: [<ffffffff8824476e>] :jfs:jfs_sync+0x0/0x1ac
May 29 11:58:34 vault1 kernel: [<ffffffff882448f4>] :jfs:jfs_sync+0x186/0x1ac
May 29 11:58:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:34 vault1 kernel: [<ffffffff8824476e>] :jfs:jfs_sync+0x0/0x1ac
May 29 11:58:34 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:34 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:34 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:34 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:34 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:34 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:34 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:34 vault1 kernel:
May 29 11:58:34 vault1 kernel: kjournald S ffff81007f1149d8 0 2306 19 (L-TLB)
May 29 11:58:34 vault1 kernel: ffff81007d54fec0 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:34 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff810037f5d040
May 29 11:58:34 vault1 kernel: ffff81011fc78040 0000000000004c8c ffff810037f5d248 000000018029fc0f
May 29 11:58:34 vault1 kernel: Call Trace:
May 29 11:58:34 vault1 kernel: [<ffffffff880279be>] :jbd:kjournald+0x18e/0x21d
May 29 11:58:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:34 vault1 kernel: [<ffffffff880279c3>] :jbd:kjournald+0x193/0x21d
May 29 11:58:34 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:34 vault1 kernel: [<ffffffff88027830>] :jbd:kjournald+0x0/0x21d
May 29 11:58:34 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:35 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:35 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:35 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:35 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:35 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:35 vault1 kernel:
May 29 11:58:35 vault1 kernel: kjournald S ffff81011f96fa98 0 2308 19 (L-TLB)
May 29 11:58:35 vault1 kernel: ffff81007d541ec0 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:35 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff810037f60040
May 29 11:58:35 vault1 kernel: ffff81011fc78040 000000000000287b ffff810037f60248 000000018029fc0f
May 29 11:58:35 vault1 kernel: Call Trace:
May 29 11:58:35 vault1 kernel: [<ffffffff880279be>] :jbd:kjournald+0x18e/0x21d
May 29 11:58:35 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:35 vault1 kernel: [<ffffffff880279c3>] :jbd:kjournald+0x193/0x21d
May 29 11:58:35 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:35 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:35 vault1 kernel: [<ffffffff88027830>] :jbd:kjournald+0x0/0x21d
May 29 11:58:35 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:35 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:35 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:35 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:35 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:35 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:35 vault1 kernel:
May 29 11:58:35 vault1 kernel: kjournald D ffff81007e837b18 0 2310 19 (L-TLB)
May 29 11:58:35 vault1 kernel: ffff81007d0efd40 0000000000000046 0000000000000000 ffff81007e9970c0
May 29 11:58:35 vault1 kernel: ffff81011fc78040 00000000000055bc ffff81007e9972c8 000000017e9970c0
May 29 11:58:35 vault1 kernel: ffff81011fc78040 0000000000004b2c ffffffffffffffff ffff810082c32700
May 29 11:58:35 vault1 kernel: Call Trace:
May 29 11:58:35 vault1 kernel: [<ffffffff80258fc6>] generic_unplug_device+0x18/0x25
May 29 11:58:35 vault1 kernel: [<ffffffff80258fc6>] generic_unplug_device+0x18/0x25
May 29 11:58:35 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:35 vault1 kernel: [<ffffffff8021576b>] sync_buffer+0x0/0x43
May 29 11:58:35 vault1 kernel: [<ffffffff8021576b>] sync_buffer+0x0/0x43
May 29 11:58:35 vault1 kernel: [<ffffffff802157a9>] sync_buffer+0x3e/0x43
May 29 11:58:35 vault1 kernel: [<ffffffff802618fb>] __wait_on_bit+0x40/0x6e
May 29 11:58:35 vault1 kernel: [<ffffffff8021576b>] sync_buffer+0x0/0x43
May 29 11:58:35 vault1 kernel: [<ffffffff8021576b>] sync_buffer+0x0/0x43
May 29 11:58:35 vault1 kernel: [<ffffffff80261995>] out_of_line_wait_on_bit+0x6c/0x78
May 29 11:58:35 vault1 kernel: [<ffffffff8029a120>] wake_bit_function+0x0/0x23
May 29 11:58:35 vault1 kernel: [<ffffffff8023aa60>] sync_dirty_buffer+0xae/0xe2
May 29 11:58:35 vault1 kernel: [<ffffffff88023eff>] :jbd:journal_commit_transaction+0xbc7/0x109c
May 29 11:58:35 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:35 vault1 kernel: [<ffffffff880278ea>] :jbd:kjournald+0xba/0x21d
May 29 11:58:35 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:35 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:35 vault1 kernel: [<ffffffff88027830>] :jbd:kjournald+0x0/0x21d
May 29 11:58:35 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:35 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:35 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:35 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:35 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:35 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:35 vault1 kernel:
May 29 11:58:35 vault1 kernel: dhclient S ffff81007f1156f8 0 2686 1 (NOTLB)
May 29 11:58:35 vault1 kernel: ffff81007747da18 0000000000000046 0000000000000000 ffff81007e9fb0c0
May 29 11:58:35 vault1 kernel: ffff81007e9fb0c0 ffffffff802638fc 0000000000000007 ffff81007e9fb0c0
May 29 11:58:35 vault1 kernel: ffff81011fc78040 0000000000002776 ffff81007e9fb2c8 0000000137e28000
May 29 11:58:35 vault1 kernel: Call Trace:
May 29 11:58:35 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:35 vault1 kernel: [<ffffffff80261790>] schedule_timeout+0x8a/0xad
May 29 11:58:35 vault1 kernel: [<ffffffff802913d9>] process_timeout+0x0/0x5
May 29 11:58:35 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 29 11:58:35 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:35 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:35 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:35 vault1 kernel: [<ffffffff802078c3>] kmem_cache_free+0x257/0x267
May 29 11:58:35 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:35 vault1 kernel: [<ffffffff804435a0>] udp_recvmsg+0x1ee/0x270
May 29 11:58:35 vault1 kernel: [<ffffffff80232600>] sock_common_recvmsg+0x30/0x45
May 29 11:58:35 vault1 kernel: [<ffffffff80230bc6>] sock_recvmsg+0xf8/0x117
May 29 11:58:35 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:35 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 29 11:58:35 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 29 11:58:35 vault1 kernel: [<ffffffff8040bc77>] move_addr_to_user+0x5d/0x78
May 29 11:58:35 vault1 kernel: [<ffffffff8022c605>] sys_recvfrom+0x104/0x130
May 29 11:58:35 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:35 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:35 vault1 kernel: [<ffffffff80216b98>] sys_select+0x15a/0x183
May 29 11:58:35 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:35 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:35 vault1 kernel:
May 29 11:58:35 vault1 kernel: syslogd D ffff81007e837b18 0 2963 1 (NOTLB)
May 29 11:58:35 vault1 kernel: ffff81007869fd88 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:35 vault1 kernel: 0000000000000000 0000000000000046 0000000000000009 ffff81007e1b3080
May 29 11:58:36 vault1 kernel: ffff81007e9970c0 0000000000014ad9 ffff81007e1b3288 000000018029fc0f
May 29 11:58:36 vault1 kernel: Call Trace:
May 29 11:58:36 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:36 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:36 vault1 kernel: [<ffffffff88026c13>] :jbd:log_wait_commit+0xa4/0xf7
May 29 11:58:36 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:36 vault1 kernel: [<ffffffff88021730>] :jbd:journal_stop+0x1e5/0x21a
May 29 11:58:36 vault1 kernel: [<ffffffff802304c1>] __writeback_single_inode+0x1fa/0x338
May 29 11:58:36 vault1 kernel: [<ffffffff802e2073>] sync_inode+0x19/0x39
May 29 11:58:36 vault1 kernel: [<ffffffff802e2073>] sync_inode+0x19/0x39
May 29 11:58:36 vault1 kernel: [<ffffffff802e207e>] sync_inode+0x24/0x39
May 29 11:58:36 vault1 kernel: [<ffffffff88037332>] :ext3:ext3_sync_file+0xb2/0xc4
May 29 11:58:36 vault1 kernel: [<ffffffff8024e527>] do_fsync+0x52/0xa4
May 29 11:58:36 vault1 kernel: [<ffffffff802e4478>] __do_fsync+0x23/0x36
May 29 11:58:36 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:36 vault1 kernel:
May 29 11:58:36 vault1 kernel: klogd S ffff81007ebd3638 0 2966 1 (NOTLB)
May 29 11:58:36 vault1 kernel: ffff810077c2fbe8 0000000000000046 0000000000000000 ffff81007e0fe080
May 29 11:58:36 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff81007e0fe080
May 29 11:58:36 vault1 kernel: ffff81011fccf080 000000000000343a ffff81007e0fe288 0000000200000002
May 29 11:58:36 vault1 kernel: Call Trace:
May 29 11:58:36 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:36 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:36 vault1 kernel: [<ffffffff80460f4b>] unix_wait_for_peer+0x86/0xb0
May 29 11:58:36 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:36 vault1 kernel: [<ffffffff804614fb>] unix_dgram_sendmsg+0x3d3/0x4cd
May 29 11:58:36 vault1 kernel: [<ffffffff8024671c>] sock_aio_write+0x110/0x128
May 29 11:58:36 vault1 kernel: [<ffffffff80260e98>] __sched_text_start+0x928/0x960
May 29 11:58:36 vault1 kernel: [<ffffffff80218507>] do_sync_write+0xc9/0x10c
May 29 11:58:36 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:36 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:36 vault1 kernel: [<ffffffff80216cda>] vfs_write+0xe1/0x177
May 29 11:58:36 vault1 kernel: [<ffffffff8020b86d>] fget_light+0x19/0x7b
May 29 11:58:36 vault1 kernel: [<ffffffff802176bf>] sys_write+0x45/0x6e
May 29 11:58:36 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:36 vault1 kernel:
May 29 11:58:36 vault1 kernel: irqbalance S ffff81007fe18098 0 2977 1 (NOTLB)
May 29 11:58:36 vault1 kernel: ffff8100774e9eb8 0000000000000046 0000000000000000 ffffffff802638fc
May 29 11:58:36 vault1 kernel: ffff810003c2cbf8 ffff810003c2cbf0 0000000000000007 ffff810037e90040
May 29 11:58:36 vault1 kernel: ffffffff8058a6e0 00000000001be12b ffff810037e90248 0000000000000292
May 29 11:58:36 vault1 kernel: Call Trace:
May 29 11:58:36 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:36 vault1 kernel: [<ffffffff8023c6d0>] hrtimer_start+0xfd/0x110
May 29 11:58:36 vault1 kernel: [<ffffffff802626ef>] do_nanosleep+0x46/0x77
May 29 11:58:36 vault1 kernel: [<ffffffff80258b18>] hrtimer_nanosleep+0x58/0x11e
May 29 11:58:36 vault1 kernel: [<ffffffff8029c17e>] hrtimer_wakeup+0x0/0x22
May 29 11:58:36 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:36 vault1 kernel: [<ffffffff80252bb1>] sys_nanosleep+0x4c/0x62
May 29 11:58:36 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:36 vault1 kernel:
May 29 11:58:36 vault1 kernel: portmap S ffff81007fe18db8 0 2997 1 (NOTLB)
May 29 11:58:36 vault1 kernel: ffff8100774d5b28 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:36 vault1 kernel: 0000000180248a3b 0000000000000002 000000000000000a ffff810037f6e080
May 29 11:58:36 vault1 kernel: ffff81011fd34040 0000000000b33289 ffff810037f6e288 0000000300000000
May 29 11:58:36 vault1 kernel: Call Trace:
May 29 11:58:36 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:36 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:36 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:36 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:36 vault1 kernel: [<ffffffff80213f07>] tcp_poll+0x1f/0x133
May 29 11:58:36 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 29 11:58:36 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:36 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:36 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:36 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:36 vault1 kernel: [<ffffffff8029ff51>] debug_check_no_locks_freed+0x11d/0x129
May 29 11:58:36 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:36 vault1 kernel: [<ffffffff8020b848>] free_hot_cold_page+0x184/0x190
May 29 11:58:36 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:36 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:36 vault1 kernel: [<ffffffff8040b777>] sys_sendmsg+0x217/0x28a
May 29 11:58:36 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:36 vault1 kernel: [<ffffffff8023089b>] __up_write+0x27/0xf2
May 29 11:58:36 vault1 kernel: [<ffffffff8023089b>] __up_write+0x27/0xf2
May 29 11:58:36 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:36 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 29 11:58:36 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:36 vault1 kernel:
May 29 11:58:36 vault1 kernel: rpc.statd S ffff81007e1bb258 0 3017 1 (NOTLB)
May 29 11:58:37 vault1 kernel: ffff810077419a18 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:37 vault1 kernel: 0000000180248a3b 0000000000000002 0000000000000001 ffff810037f5e080
May 29 11:58:37 vault1 kernel: ffff81007e917040 0000000000002c39 ffff810037f5e288 0000000000000000
May 29 11:58:37 vault1 kernel: Call Trace:
May 29 11:58:37 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:37 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:37 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:37 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:37 vault1 kernel: [<ffffffff80213f07>] tcp_poll+0x1f/0x133
May 29 11:58:37 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 29 11:58:37 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:37 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:37 vault1 last message repeated 2 times
May 29 11:58:37 vault1 kernel: [<ffffffff80263584>] _read_unlock+0x17/0x20
May 29 11:58:37 vault1 kernel: [<ffffffff8046155d>] unix_dgram_sendmsg+0x435/0x4cd
May 29 11:58:37 vault1 kernel: [<ffffffff8020d542>] dput+0x3c/0x12b
May 29 11:58:37 vault1 kernel: [<ffffffff802530fc>] sock_sendmsg+0xea/0x107
May 29 11:58:37 vault1 kernel: [<ffffffff8020ed97>] link_path_walk+0xce/0xe0
May 29 11:58:37 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:37 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 29 11:58:37 vault1 kernel: [<ffffffff8022d806>] mntput_no_expire+0x1c/0x96
May 29 11:58:37 vault1 kernel: [<ffffffff8040ba92>] sys_sendto+0x128/0x151
May 29 11:58:37 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 29 11:58:37 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:37 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:37 vault1 kernel:
May 29 11:58:37 vault1 kernel: dbus-daemon S ffff81007f115b58 0 3035 1 (NOTLB)
May 29 11:58:37 vault1 kernel: ffff81007773bb28 0000000000000046 0000000000000000 ffff81007f1d5080
May 29 11:58:37 vault1 kernel: 0000000000000000 ffffffff802a0de5 000000000000000a ffff81007f1d5080
May 29 11:58:37 vault1 kernel: ffff81011fc78040 0000000000009ca4 ffff81007f1d5288 0000000100000002
May 29 11:58:37 vault1 kernel: Call Trace:
May 29 11:58:37 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:37 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:37 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 29 11:58:37 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:37 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:37 vault1 last message repeated 2 times
May 29 11:58:37 vault1 kernel: [<ffffffff8024671c>] sock_aio_write+0x110/0x128
May 29 11:58:37 vault1 kernel: [<ffffffff8040ae7a>] sock_aio_read+0x10c/0x124
May 29 11:58:37 vault1 kernel: [<ffffffff8020cf53>] do_sync_read+0xc9/0x10c
May 29 11:58:37 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:37 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:37 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:37 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:37 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 29 11:58:37 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:37 vault1 kernel:
May 29 11:58:37 vault1 kernel: hidd S ffff810037d4ad78 0 3067 1 (NOTLB)
May 29 11:58:37 vault1 kernel: ffff81007734bad8 0000000000000046 0000000000000000 ffff81007f0fb0c0
May 29 11:58:37 vault1 kernel: 0000000000000000 ffffffff802a0de5 0000000000000001 ffff81007f0fb0c0
May 29 11:58:37 vault1 kernel: ffff81011fccf080 0000000000008764 ffff81007f0fb2c8 0000000200000002
May 29 11:58:37 vault1 kernel: Call Trace:
May 29 11:58:37 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:37 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:37 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 29 11:58:37 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:37 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:37 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:37 vault1 kernel: [<ffffffff80263584>] _read_unlock+0x17/0x20
May 29 11:58:37 vault1 kernel: [<ffffffff8046155d>] unix_dgram_sendmsg+0x435/0x4cd
May 29 11:58:37 vault1 kernel: [<ffffffff8020d542>] dput+0x3c/0x12b
May 29 11:58:37 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:37 vault1 kernel: [<ffffffff802078e8>] find_get_page+0x15/0x4b
May 29 11:58:37 vault1 kernel: [<ffffffff802638bb>] _read_unlock_irq+0x24/0x27
May 29 11:58:37 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:37 vault1 kernel: [<ffffffff80208c1f>] __handle_mm_fault+0x3cf/0xccf
May 29 11:58:37 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:37 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 29 11:58:37 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:37 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:37 vault1 kernel: [<ffffffff8021e12c>] sigprocmask+0x28/0xc1
May 29 11:58:37 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:37 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:37 vault1 kernel: [<ffffffff802d861f>] sys_ppoll+0xf1/0x1ff
May 29 11:58:37 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:37 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:37 vault1 kernel:
May 29 11:58:37 vault1 kernel: acpid S ffff81007ebd2d78 0 3079 1 (NOTLB)
May 29 11:58:37 vault1 kernel: ffff8100775ffb28 0000000000000046 0000000000000627 ffff81007e740080
May 29 11:58:37 vault1 kernel: 0000000000000000 ffffffff802a0de5 0000000000000007 ffff81007e740080
May 29 11:58:37 vault1 kernel: ffff81007e932080 0000000000017592 ffff81007e740288 0000000000000002
May 29 11:58:37 vault1 kernel: Call Trace:
May 29 11:58:37 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:37 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:37 vault1 kernel: [<ffffffff88021755>] :jbd:journal_stop+0x20a/0x21a
May 29 11:58:37 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:37 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 29 11:58:37 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:37 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:37 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:37 vault1 kernel: [<ffffffff8020e533>] current_fs_time+0x3b/0x40
May 29 11:58:37 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:37 vault1 kernel: [<ffffffff80216881>] __generic_file_aio_write_nolock+0x378/0x3eb
May 29 11:58:38 vault1 kernel: [<ffffffff80262123>] __mutex_lock_slowpath+0x245/0x25b
May 29 11:58:38 vault1 kernel: [<ffffffff80261ece>] __mutex_unlock_slowpath+0x128/0x133
May 29 11:58:38 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:38 vault1 kernel: [<ffffffff802224af>] generic_file_aio_write+0x6c/0xc1
May 29 11:58:38 vault1 kernel: [<ffffffff88037192>] :ext3:ext3_file_write+0x16/0x94
May 29 11:58:38 vault1 kernel: [<ffffffff80218507>] do_sync_write+0xc9/0x10c
May 29 11:58:38 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:38 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:38 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:38 vault1 kernel: [<ffffffff8040bfa2>] sys_accept+0x19e/0x1fb
May 29 11:58:38 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:38 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 29 11:58:38 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:38 vault1 kernel:
May 29 11:58:38 vault1 kernel: sshd S ffff81007f342918 0 3092 1 (NOTLB)
May 29 11:58:38 vault1 kernel: ffff810076dbda18 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:38 vault1 kernel: 000000018023bee1 0000000000000002 000000000000000a ffff81007eaf3080
May 29 11:58:38 vault1 kernel: ffff81011fccf080 0000000000007534 ffff81007eaf3288 0000000200000000
May 29 11:58:38 vault1 kernel: Call Trace:
May 29 11:58:38 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:38 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:38 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:38 vault1 kernel: [<ffffffff80213f07>] tcp_poll+0x1f/0x133
May 29 11:58:38 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 29 11:58:38 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:38 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:38 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:38 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:38 vault1 kernel: [<ffffffff802133c4>] sock_def_readable+0x18/0x6c
May 29 11:58:38 vault1 kernel: [<ffffffff802133c4>] sock_def_readable+0x18/0x6c
May 29 11:58:38 vault1 kernel: [<ffffffff8020a6c3>] get_page_from_freelist+0x2ab/0x534
May 29 11:58:38 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:38 vault1 kernel: [<ffffffff8020a6c3>] get_page_from_freelist+0x2ab/0x534
May 29 11:58:38 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:38 vault1 kernel: [<ffffffff802c070d>] zone_statistics+0x3f/0x60
May 29 11:58:38 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:38 vault1 kernel: [<ffffffff802118a0>] do_wp_page+0x497/0x60b
May 29 11:58:38 vault1 kernel: [<ffffffff80211a03>] do_wp_page+0x5fa/0x60b
May 29 11:58:38 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 29 11:58:38 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:38 vault1 kernel: [<ffffffff8020c6a9>] _atomic_dec_and_lock+0x39/0x58
May 29 11:58:38 vault1 kernel: [<ffffffff80212fc9>] poison_obj+0x26/0x2f
May 29 11:58:38 vault1 kernel: [<ffffffff802078c3>] kmem_cache_free+0x257/0x267
May 29 11:58:38 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:38 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 29 11:58:38 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:38 vault1 kernel: [<ffffffff80239ba9>] generic_drop_inode+0x0/0x15c
May 29 11:58:38 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:38 vault1 kernel:
May 29 11:58:38 vault1 kernel: rpc.idmapd S ffff81007e8de118 0 3193 1 (NOTLB)
May 29 11:58:38 vault1 kernel: ffff810075fabe78 0000000000000046 0000000000000000 ffff81007ee64040
May 29 11:58:38 vault1 kernel: ffff81007ee64040 ffffffff802638fc 000000000000000a ffff81007ee64040
May 29 11:58:38 vault1 kernel: ffffffff8058a6e0 00000000000027d9 ffff81007ee64248 0000000080733000
May 29 11:58:38 vault1 kernel: Call Trace:
May 29 11:58:38 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:38 vault1 kernel: [<ffffffff80261790>] schedule_timeout+0x8a/0xad
May 29 11:58:38 vault1 kernel: [<ffffffff802913d9>] process_timeout+0x0/0x5
May 29 11:58:38 vault1 kernel: [<ffffffff802ead19>] sys_epoll_wait+0x193/0x426
May 29 11:58:38 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:38 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:38 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:38 vault1 kernel:
May 29 11:58:38 vault1 kernel: crond S ffff81007f114118 0 3203 1 (NOTLB)
May 29 11:58:38 vault1 kernel: ffff810075c15eb8 0000000000000046 0000000000000000 ffffffff802638fc
May 29 11:58:38 vault1 kernel: ffff810003c379f8 ffff810003c379f0 000000000000000a ffff81007e1b80c0
May 29 11:58:38 vault1 kernel: ffff81011fc78040 0000000000027cfa ffff81007e1b82c8 0000000100000292
May 29 11:58:38 vault1 kernel: Call Trace:
May 29 11:58:38 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:38 vault1 kernel: [<ffffffff8023c6d0>] hrtimer_start+0xfd/0x110
May 29 11:58:38 vault1 kernel: [<ffffffff802626ef>] do_nanosleep+0x46/0x77
May 29 11:58:38 vault1 kernel: [<ffffffff80258b18>] hrtimer_nanosleep+0x58/0x11e
May 29 11:58:38 vault1 kernel: [<ffffffff8029c17e>] hrtimer_wakeup+0x0/0x22
May 29 11:58:38 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:38 vault1 kernel: [<ffffffff80252bb1>] sys_nanosleep+0x4c/0x62
May 29 11:58:38 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:38 vault1 kernel:
May 29 11:58:38 vault1 kernel: atd S ffff81007fe184f8 0 3213 1 (NOTLB)
May 29 11:58:39 vault1 kernel: ffff810075c55eb8 0000000000000046 0000000000000000 ffffffff802638fc
May 29 11:58:39 vault1 kernel: ffff810003c379f8 ffff810003c379f0 0000000000000007 ffff810037ec00c0
May 29 11:58:39 vault1 kernel: ffff81011fc78040 0000000000014409 ffff810037ec02c8 0000000100000292
May 29 11:58:39 vault1 kernel: Call Trace:
May 29 11:58:39 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:39 vault1 kernel: [<ffffffff8023c6d0>] hrtimer_start+0xfd/0x110
May 29 11:58:39 vault1 kernel: [<ffffffff802626ef>] do_nanosleep+0x46/0x77
May 29 11:58:39 vault1 kernel: [<ffffffff80258b18>] hrtimer_nanosleep+0x58/0x11e
May 29 11:58:39 vault1 kernel: [<ffffffff8029c17e>] hrtimer_wakeup+0x0/0x22
May 29 11:58:39 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:39 vault1 kernel: [<ffffffff80252bb1>] sys_nanosleep+0x4c/0x62
May 29 11:58:39 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:39 vault1 kernel:
May 29 11:58:39 vault1 kernel: hald S ffff81007f3424b8 0 3223 1 (NOTLB)
May 29 11:58:39 vault1 kernel: ffff810075dabb28 0000000000000046 0000000000000000 ffff81007f4a2080
May 29 11:58:39 vault1 kernel: ffff81007f4a2080 ffffffff802638fc 000000000000000a ffff81007f4a2080
May 29 11:58:39 vault1 kernel: ffffffff8058a6e0 000000000005abda ffff81007f4a2288 0000000080733000
May 29 11:58:39 vault1 kernel: Call Trace:
May 29 11:58:39 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:39 vault1 kernel: [<ffffffff80261790>] schedule_timeout+0x8a/0xad
May 29 11:58:39 vault1 kernel: [<ffffffff802913d9>] process_timeout+0x0/0x5
May 29 11:58:39 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 29 11:58:39 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:39 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:39 vault1 last message repeated 9 times
May 29 11:58:39 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:39 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:39 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 29 11:58:39 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:39 vault1 kernel:
May 29 11:58:39 vault1 kernel: hald-runner S ffff81011f3c9678 0 3224 3223 (NOTLB)
May 29 11:58:39 vault1 kernel: ffff81011ecafb28 0000000000000046 0000000000000000 ffff81011f39d080
May 29 11:58:39 vault1 kernel: 0000000000000000 ffffffff802a0de5 0000000000000008 ffff81011f39d080
May 29 11:58:39 vault1 kernel: ffff81011fc78040 00000000000006d2 ffff81011f39d288 0000000100000002
May 29 11:58:39 vault1 kernel: Call Trace:
May 29 11:58:39 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:39 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:39 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 29 11:58:39 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:39 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:39 vault1 kernel: [<ffffffff8024671c>] sock_aio_write+0x110/0x128
May 29 11:58:39 vault1 kernel: [<ffffffff8026212e>] __mutex_lock_slowpath+0x250/0x25b
May 29 11:58:39 vault1 kernel: [<ffffffff8024660c>] sock_aio_write+0x0/0x128
May 29 11:58:39 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 29 11:58:39 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:39 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:39 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:39 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:39 vault1 kernel: [<ffffffff802d3dfa>] do_readv_writev+0x199/0x1ae
May 29 11:58:39 vault1 kernel: [<ffffffff80229266>] do_wait+0x9a7/0xa75
May 29 11:58:39 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:39 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 29 11:58:39 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:39 vault1 kernel:
May 29 11:58:39 vault1 kernel: hald-addon-ac S ffff81007f114e38 0 3230 3224 (NOTLB)
May 29 11:58:39 vault1 kernel: ffff810075593be8 0000000000000046 0000000000000627 ffff81007e932080
May 29 11:58:39 vault1 kernel: 0000000000000001 ffffffff802a0de5 0000000000000003 ffff81007e932080
May 29 11:58:39 vault1 kernel: ffff81007f4a2080 0000000000002a2c ffff81007e932288 0000000000000002
May 29 11:58:39 vault1 kernel: Call Trace:
May 29 11:58:39 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:39 vault1 kernel: [<ffffffff8023ad89>] prepare_to_wait+0x1b/0x5c
May 29 11:58:39 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:39 vault1 kernel: [<ffffffff80249a11>] unix_stream_recvmsg+0x233/0x548
May 29 11:58:39 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:39 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:39 vault1 kernel: [<ffffffff8040ae7a>] sock_aio_read+0x10c/0x124
May 29 11:58:39 vault1 kernel: [<ffffffff8020cf53>] do_sync_read+0xc9/0x10c
May 29 11:58:39 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:39 vault1 kernel: [<ffffffff8020e38c>] do_mmap_pgoff+0x65d/0x7c9
May 29 11:58:39 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:39 vault1 kernel: [<ffffffff8020b5b0>] vfs_read+0xde/0x173
May 29 11:58:39 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 29 11:58:39 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:39 vault1 kernel:
May 29 11:58:39 vault1 kernel: hald-addon-cp S ffff81007e8360d8 0 3231 3224 (NOTLB)
May 29 11:58:39 vault1 kernel: ffff810075563b28 0000000000000046 0000000000000627 ffff81007e9fc080
May 29 11:58:39 vault1 kernel: 0000000000000000 ffffffff802a0de5 0000000000000003 ffff81007e9fc080
May 29 11:58:39 vault1 kernel: ffff81007f4a2080 000000000001e385 ffff81007e9fc288 0000000000000002
May 29 11:58:39 vault1 kernel: Call Trace:
May 29 11:58:39 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:39 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:39 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:39 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 29 11:58:39 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:39 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:39 vault1 kernel: [<ffffffff80249bf3>] unix_stream_recvmsg+0x415/0x548
May 29 11:58:39 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:39 vault1 kernel: [<ffffffff802078e8>] find_get_page+0x15/0x4b
May 29 11:58:39 vault1 kernel: [<ffffffff802638bb>] _read_unlock_irq+0x24/0x27
May 29 11:58:39 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:39 vault1 kernel: [<ffffffff80208c1f>] __handle_mm_fault+0x3cf/0xccf
May 29 11:58:40 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:40 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 29 11:58:40 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 29 11:58:40 vault1 kernel: [<ffffffff80208d64>] __handle_mm_fault+0x514/0xccf
May 29 11:58:40 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:40 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:40 vault1 kernel: [<ffffffff80265bc5>] do_page_fault+0x490/0x7cf
May 29 11:58:40 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:40 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 29 11:58:40 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:40 vault1 kernel:
May 29 11:58:40 vault1 kernel: hald-addon-ke S ffff81007fe19678 0 3235 3224 (NOTLB)
May 29 11:58:40 vault1 kernel: ffff8100756adea8 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:40 vault1 kernel: 0000000000000000 0000000000000046 0000000000000004 ffff81007fe9d040
May 29 11:58:40 vault1 kernel: ffffffff8058a6e0 00000000000b8af2 ffff81007fe9d248 000000007fe9d040
May 29 11:58:40 vault1 kernel: Call Trace:
May 29 11:58:40 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:40 vault1 kernel: [<ffffffff803f1497>] evdev_read+0x112/0x235
May 29 11:58:40 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:40 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:40 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 29 11:58:40 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 29 11:58:40 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:40 vault1 kernel:
May 29 11:58:40 vault1 kernel: hald-addon-ke S ffff81011ed8f638 0 3245 3224 (NOTLB)
May 29 11:58:40 vault1 kernel: ffff810075c2bea8 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:40 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff81007eb3f040
May 29 11:58:40 vault1 kernel: ffff81011fccf080 000000000000243f ffff81007eb3f248 000000027eb3f040
May 29 11:58:40 vault1 kernel: Call Trace:
May 29 11:58:40 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:40 vault1 kernel: [<ffffffff803f1497>] evdev_read+0x112/0x235
May 29 11:58:40 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:40 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 29 11:58:40 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 29 11:58:40 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:40 vault1 kernel:
May 29 11:58:40 vault1 kernel: hald-addon-ke S ffff81007e1ba0d8 0 3252 3224 (NOTLB)
May 29 11:58:40 vault1 kernel: ffff810075f9dea8 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:40 vault1 kernel: 0000000000000000 0000000000000046 0000000000000001 ffff81007f2710c0
May 29 11:58:40 vault1 kernel: ffff81011fc78040 00000000000be1fa ffff81007f2712c8 000000017f2710c0
May 29 11:58:40 vault1 kernel: Call Trace:
May 29 11:58:40 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:40 vault1 kernel: [<ffffffff803f1497>] evdev_read+0x112/0x235
May 29 11:58:40 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:40 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:40 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 29 11:58:40 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 29 11:58:40 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:40 vault1 kernel:
May 29 11:58:40 vault1 kernel: hald-addon-ke S ffff81007fe18958 0 3255 3224 (NOTLB)
May 29 11:58:40 vault1 kernel: ffff810076715ea8 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:40 vault1 kernel: 0000000000000000 0000000000000046 0000000000000001 ffff810037f5f0c0
May 29 11:58:40 vault1 kernel: ffff81011fc78040 00000000000bd9f3 ffff810037f5f2c8 0000000137f5f0c0
May 29 11:58:40 vault1 kernel: Call Trace:
May 29 11:58:40 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:40 vault1 kernel: [<ffffffff803f1497>] evdev_read+0x112/0x235
May 29 11:58:40 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:40 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:40 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 29 11:58:40 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 29 11:58:40 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:40 vault1 kernel:
May 29 11:58:40 vault1 kernel: hald-addon-st S ffff81007e8de9d8 0 3263 3224 (NOTLB)
May 29 11:58:40 vault1 kernel: ffff810075027eb8 0000000000000046 0000000000000000 ffffffff802638fc
May 29 11:58:40 vault1 kernel: ffff810003c379f8 ffff810003c379f0 0000000000000009 ffff81007e864040
May 29 11:58:40 vault1 kernel: ffff81011fc78040 0000000000002aae ffff81007e864248 0000000100000292
May 29 11:58:40 vault1 kernel: Call Trace:
May 29 11:58:40 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:40 vault1 kernel: [<ffffffff8023c6d0>] hrtimer_start+0xfd/0x110
May 29 11:58:40 vault1 kernel: [<ffffffff802626ef>] do_nanosleep+0x46/0x77
May 29 11:58:40 vault1 kernel: [<ffffffff80258b18>] hrtimer_nanosleep+0x58/0x11e
May 29 11:58:40 vault1 kernel: [<ffffffff8029c17e>] hrtimer_wakeup+0x0/0x22
May 29 11:58:40 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:40 vault1 kernel: [<ffffffff80252bb1>] sys_nanosleep+0x4c/0x62
May 29 11:58:40 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:40 vault1 kernel:
May 29 11:58:40 vault1 kernel: smartd S ffff81011febcdb8 0 3307 1 (NOTLB)
May 29 11:58:40 vault1 kernel: ffff81011ed2deb8 0000000000000046 0000000000000000 ffffffff802638fc
May 29 11:58:40 vault1 kernel: ffff810003c379f8 ffff810003c379f0 0000000000000007 ffff81011ee23040
May 29 11:58:40 vault1 kernel: ffff81011fc78040 0000000000002d86 ffff81011ee23248 0000000100000292
May 29 11:58:40 vault1 kernel: Call Trace:
May 29 11:58:40 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:40 vault1 kernel: [<ffffffff8023c6d0>] hrtimer_start+0xfd/0x110
May 29 11:58:40 vault1 kernel: [<ffffffff802626ef>] do_nanosleep+0x46/0x77
May 29 11:58:40 vault1 kernel: [<ffffffff80258b18>] hrtimer_nanosleep+0x58/0x11e
May 29 11:58:40 vault1 kernel: [<ffffffff8029c17e>] hrtimer_wakeup+0x0/0x22
May 29 11:58:40 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:41 vault1 kernel: [<ffffffff80252bb1>] sys_nanosleep+0x4c/0x62
May 29 11:58:41 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:41 vault1 kernel:
May 29 11:58:41 vault1 kernel: login S ffff81007f115298 0 3310 1 (NOTLB)
May 29 11:58:41 vault1 kernel: ffff810075cd3ec8 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:41 vault1 kernel: 0000000000000000 0000000200000002 0000000000000009 ffff81007fd33040
May 29 11:58:41 vault1 kernel: ffff81011fd360c0 000000000007dc88 ffff81007fd33248 0000000300000000
May 29 11:58:41 vault1 kernel: Call Trace:
May 29 11:58:41 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:41 vault1 kernel: [<ffffffff8022920b>] do_wait+0x94c/0xa75
May 29 11:58:41 vault1 kernel: [<ffffffff8022922e>] do_wait+0x96f/0xa75
May 29 11:58:41 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:41 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:41 vault1 kernel:
May 29 11:58:41 vault1 kernel: login S ffff81007e8de578 0 3311 1 (NOTLB)
May 29 11:58:41 vault1 kernel: ffff810076361ec8 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:41 vault1 kernel: 0000000000000000 0000000200000002 0000000000000007 ffff81007fd3e0c0
May 29 11:58:41 vault1 kernel: ffff81011fc78040 0000000000087012 ffff81007fd3e2c8 0000000100000000
May 29 11:58:41 vault1 kernel: Call Trace:
May 29 11:58:41 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:41 vault1 kernel: [<ffffffff8022920b>] do_wait+0x94c/0xa75
May 29 11:58:41 vault1 kernel: [<ffffffff8022922e>] do_wait+0x96f/0xa75
May 29 11:58:41 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:41 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:41 vault1 kernel:
May 29 11:58:41 vault1 kernel: mingetty S ffff81011f3c8958 0 3312 1 (NOTLB)
May 29 11:58:41 vault1 kernel: ffff810075fd5dc8 0000000000000046 0000000000026a68 ffff81011fd1da10
May 29 11:58:41 vault1 kernel: 0000000000000296 ffffffff80263178 0000000000000004 ffff81007fc0e040
May 29 11:58:41 vault1 kernel: ffff81007ea550c0 00000000000441eb ffff81007fc0e248 0000000375fd4000
May 29 11:58:41 vault1 kernel: Call Trace:
May 29 11:58:41 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:41 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:41 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:41 vault1 kernel: [<ffffffff802638fe>] _spin_unlock_irqrestore+0x40/0x44
May 29 11:58:41 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 29 11:58:41 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:41 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 29 11:58:41 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 29 11:58:41 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 29 11:58:41 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:41 vault1 kernel:
May 29 11:58:41 vault1 kernel: mingetty S ffff81011f2b9ad8 0 3313 1 (NOTLB)
May 29 11:58:41 vault1 kernel: ffff810075f07dc8 0000000000000046 0000000000000000 ffff81007f4a37b0
May 29 11:58:41 vault1 kernel: 0000000000000627 ffff81007f4a3040 0000000000000004 ffff81007f4a3040
May 29 11:58:41 vault1 kernel: ffff81007e6b4080 000000000002f310 ffff81007f4a3248 0000000200000046
May 29 11:58:41 vault1 kernel: Call Trace:
May 29 11:58:41 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:41 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:41 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:41 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 29 11:58:41 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:41 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:41 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 29 11:58:41 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 29 11:58:41 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 29 11:58:41 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:41 vault1 kernel:
May 29 11:58:41 vault1 kernel: mingetty S ffff81011f2b8098 0 3314 1 (NOTLB)
May 29 11:58:41 vault1 kernel: ffff810075d57dc8 0000000000000046 0000000000000000 ffff81007ea55830
May 29 11:58:41 vault1 kernel: 0000000000000627 ffff81007ea550c0 0000000000000004 ffff81007ea550c0
May 29 11:58:41 vault1 kernel: ffff81011fd360c0 0000000000002997 ffff81007ea552c8 0000000300000046
May 29 11:58:41 vault1 kernel: Call Trace:
May 29 11:58:41 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:41 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:41 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 29 11:58:41 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:41 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:41 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 29 11:58:41 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 29 11:58:41 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 29 11:58:41 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:41 vault1 kernel:
May 29 11:58:41 vault1 kernel: mingetty S ffff81011f3c8db8 0 3315 1 (NOTLB)
May 29 11:58:41 vault1 kernel: ffff810075c95dc8 0000000000000046 0000000000000000 ffff81007e4167f0
May 29 11:58:41 vault1 kernel: 0000000000000627 ffff81007e416080 0000000000000004 ffff81007e416080
May 29 11:58:41 vault1 kernel: ffff81007e6b4080 0000000000002b22 ffff81007e416288 0000000200000046
May 29 11:58:41 vault1 kernel: Call Trace:
May 29 11:58:41 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:42 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:42 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:42 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 29 11:58:42 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:42 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:42 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 29 11:58:42 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 29 11:58:42 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 29 11:58:42 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:42 vault1 kernel:
May 29 11:58:42 vault1 kernel: mingetty S ffff81011edb4578 0 3316 1 (NOTLB)
May 29 11:58:42 vault1 kernel: ffff81007e0bddc8 0000000000000046 0000000000000000 ffff81007e6b47f0
May 29 11:58:42 vault1 kernel: 0000000000000627 ffff81007e6b4080 0000000000000003 ffff81007e6b4080
May 29 11:58:42 vault1 kernel: ffff81011fccf080 000000000001b72f ffff81007e6b4288 0000000200000046
May 29 11:58:42 vault1 kernel: Call Trace:
May 29 11:58:42 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:42 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:42 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 29 11:58:42 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:42 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 29 11:58:42 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 29 11:58:42 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 29 11:58:42 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:42 vault1 kernel:
May 29 11:58:42 vault1 kernel: bash S ffff81007f114578 0 3377 3310 (NOTLB)
May 29 11:58:42 vault1 kernel: ffff81011f889dc8 0000000000000046 0000000000000000 ffff81011f3fc830
May 29 11:58:42 vault1 kernel: 0000000000000627 ffff81011f3fc0c0 0000000000000009 ffff81011f3fc0c0
May 29 11:58:42 vault1 kernel: ffff81011fc78040 000000000000af71 ffff81011f3fc2c8 0000000100000046
May 29 11:58:42 vault1 kernel: Call Trace:
May 29 11:58:42 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:42 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:42 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 29 11:58:42 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:42 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 29 11:58:42 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 29 11:58:42 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 29 11:58:42 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:42 vault1 kernel:
May 29 11:58:42 vault1 kernel: scsi_eh_11 S ffff81011f2b84f8 0 3447 19 (L-TLB)
May 29 11:58:42 vault1 kernel: ffff81011e1bbe70 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:42 vault1 kernel: 0000000000000000 0000000000000046 0000000000000008 ffff81011eeb50c0
May 29 11:58:42 vault1 kernel: ffff81011fd360c0 0000000000000758 ffff81011eeb52c8 0000000380263894
May 29 11:58:42 vault1 kernel: Call Trace:
May 29 11:58:42 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:42 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:42 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 29 11:58:42 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:42 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:42 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 29 11:58:42 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:42 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:42 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:42 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:42 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:42 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:42 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:42 vault1 kernel:
May 29 11:58:42 vault1 kernel: qla2xxx_11_dp S ffff81011f96fa98 0 3455 19 (L-TLB)
May 29 11:58:42 vault1 kernel: ffff81011e775ee0 0000000000000046 0000000000000000 0000000000200200
May 29 11:58:42 vault1 kernel: 0000000100032ce3 ffffffff802913d9 000000000000000a ffff81011e452080
May 29 11:58:42 vault1 kernel: ffff81011fc78040 0000000000000923 ffff81011e452288 0000000100000102
May 29 11:58:42 vault1 kernel: Call Trace:
May 29 11:58:42 vault1 kernel: [<ffffffff802913d9>] process_timeout+0x0/0x5
May 29 11:58:42 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 29 11:58:42 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:42 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 29 11:58:42 vault1 kernel: [<ffffffff880ba348>] :qla2xxx:qla2x00_do_dpc+0x39/0x439
May 29 11:58:42 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:42 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 29 11:58:42 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:42 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:42 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:42 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:42 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:42 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:42 vault1 kernel:
May 29 11:58:42 vault1 kernel: scsi_wq_11 S ffff81011ed8e058 0 3456 19 (L-TLB)
May 29 11:58:42 vault1 kernel: ffff81011e777ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:42 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81011e5d5080
May 29 11:58:42 vault1 kernel: ffff81011fd360c0 00000000000014e9 ffff81011e5d5288 000000031e5d5080
May 29 11:58:42 vault1 kernel: Call Trace:
May 29 11:58:42 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:42 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:42 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:42 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:42 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:42 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:42 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:42 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:42 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:43 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:43 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:43 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:43 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:43 vault1 kernel:
May 29 11:58:43 vault1 kernel: fc_wq_11 S ffff81011ed8e058 0 3457 19 (L-TLB)
May 29 11:58:43 vault1 kernel: ffff81011e779ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:43 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81011fff7040
May 29 11:58:43 vault1 kernel: ffff81011fd360c0 0000000000000e63 ffff81011fff7248 000000031fff7040
May 29 11:58:43 vault1 kernel: Call Trace:
May 29 11:58:43 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:43 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:43 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:43 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:43 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:43 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:43 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:43 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:43 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:43 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:43 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:43 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:43 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:43 vault1 kernel:
May 29 11:58:43 vault1 kernel: fc_dl_11 S ffff81011f2b84f8 0 3458 19 (L-TLB)
May 29 11:58:43 vault1 kernel: ffff81011e77bea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:43 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011e96a080
May 29 11:58:43 vault1 kernel: ffff810037d460c0 00000000000022c2 ffff81011e96a288 000000011e96a080
May 29 11:58:43 vault1 kernel: Call Trace:
May 29 11:58:43 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:43 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:43 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:43 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:43 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:43 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:43 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:43 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:43 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:43 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:43 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:43 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:43 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:43 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:43 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:43 vault1 kernel:
May 29 11:58:43 vault1 kernel: scsi_eh_12 S ffff81011ed8e058 0 3465 19 (L-TLB)
May 29 11:58:43 vault1 kernel: ffff81011ee2be70 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:43 vault1 kernel: 0000000000000000 0000000000000046 0000000000000009 ffff81011e50e040
May 29 11:58:43 vault1 kernel: ffff81011fd360c0 0000000000000913 ffff81011e50e248 0000000380263894
May 29 11:58:43 vault1 kernel: Call Trace:
May 29 11:58:43 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:43 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:43 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 29 11:58:43 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:43 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:43 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 29 11:58:43 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:43 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:43 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:43 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:43 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:43 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:43 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:43 vault1 kernel:
May 29 11:58:43 vault1 kernel: qla2xxx_12_dp S ffff81007f3424b8 0 3466 19 (L-TLB)
May 29 11:58:43 vault1 kernel: ffff81011f20bee0 0000000000000046 0000000000000000 0000000000200200
May 29 11:58:43 vault1 kernel: 00000001000341a9 ffffffff802913d9 000000000000000a ffff81011e80d0c0
May 29 11:58:43 vault1 kernel: ffffffff8058a6e0 000000000000080f ffff81011e80d2c8 0000000000000102
May 29 11:58:43 vault1 kernel: Call Trace:
May 29 11:58:43 vault1 kernel: [<ffffffff802913d9>] process_timeout+0x0/0x5
May 29 11:58:43 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 29 11:58:43 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:43 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 29 11:58:43 vault1 kernel: [<ffffffff880ba348>] :qla2xxx:qla2x00_do_dpc+0x39/0x439
May 29 11:58:43 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:43 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 29 11:58:43 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:43 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:43 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:43 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:43 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:44 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:44 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:44 vault1 kernel:
May 29 11:58:44 vault1 kernel: scsi_wq_12 S ffff81011ed8e058 0 3467 19 (L-TLB)
May 29 11:58:44 vault1 kernel: ffff81011f257ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:44 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81011f03b040
May 29 11:58:44 vault1 kernel: ffff81011fd360c0 0000000000001d8d ffff81011f03b248 000000031f03b040
May 29 11:58:44 vault1 kernel: Call Trace:
May 29 11:58:44 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:44 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:44 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:44 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:44 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:44 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:44 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:44 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:44 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:44 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:44 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:44 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:44 vault1 kernel:
May 29 11:58:44 vault1 kernel: fc_wq_12 S ffff81011ed8e058 0 3468 19 (L-TLB)
May 29 11:58:44 vault1 kernel: ffff81011e77dea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:44 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81011eeb4080
May 29 11:58:44 vault1 kernel: ffff81011fd360c0 000000000000127d ffff81011eeb4288 000000031eeb4080
May 29 11:58:44 vault1 kernel: Call Trace:
May 29 11:58:44 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:44 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:44 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:44 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:44 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:44 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:44 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:44 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:44 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:44 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:44 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:44 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:44 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:44 vault1 kernel:
May 29 11:58:44 vault1 kernel: fc_dl_12 S ffff81011ed8e058 0 3469 19 (L-TLB)
May 29 11:58:44 vault1 kernel: ffff81011e7bdea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:44 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81011e949040
May 29 11:58:44 vault1 kernel: ffff81011fd360c0 0000000000001255 ffff81011e949248 000000031e949040
May 29 11:58:44 vault1 kernel: Call Trace:
May 29 11:58:44 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:44 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:44 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:44 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:44 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:44 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:44 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:44 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:44 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:44 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:44 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:44 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:44 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:44 vault1 kernel:
May 29 11:58:44 vault1 kernel: scsi_eh_13 S ffff81011ed8e058 0 3476 19 (L-TLB)
May 29 11:58:44 vault1 kernel: ffff81011e71be70 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:44 vault1 kernel: 0000000000000000 0000000000000046 0000000000000009 ffff81011e869080
May 29 11:58:44 vault1 kernel: ffff81011fd360c0 00000000000008a3 ffff81011e869288 0000000380263894
May 29 11:58:44 vault1 kernel: Call Trace:
May 29 11:58:44 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:44 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:44 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 29 11:58:44 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:44 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:44 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 29 11:58:44 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:44 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:44 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:44 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:44 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:44 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:44 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:44 vault1 kernel:
May 29 11:58:44 vault1 kernel: qla2xxx_13_dp S ffff81007e8de9d8 0 3477 19 (L-TLB)
May 29 11:58:44 vault1 kernel: ffff81011e90fee0 0000000000000046 0000000000000000 0000000000000282
May 29 11:58:44 vault1 kernel: ffff81011e90fe40 ffff81011e90fe40 0000000000000009 ffff81011fbcc0c0
May 29 11:58:44 vault1 kernel: ffff81011fc78040 0000000000000b5a ffff81011fbcc2c8 0000000100000000
May 29 11:58:44 vault1 kernel: Call Trace:
May 29 11:58:44 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:44 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 29 11:58:44 vault1 kernel: [<ffffffff880ba348>] :qla2xxx:qla2x00_do_dpc+0x39/0x439
May 29 11:58:44 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:44 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 29 11:58:44 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:44 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:44 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:44 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:44 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:44 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:45 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:45 vault1 kernel:
May 29 11:58:45 vault1 kernel: scsi_wq_13 S ffff81011f2b84f8 0 3478 19 (L-TLB)
May 29 11:58:45 vault1 kernel: ffff81011e77fea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:45 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81011ed7f040
May 29 11:58:45 vault1 kernel: ffff81011fc78040 000000000032574d ffff81011ed7f248 000000011ed7f040
May 29 11:58:45 vault1 kernel: Call Trace:
May 29 11:58:45 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:45 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:45 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:45 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:45 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:45 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:45 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:45 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:45 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:45 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:45 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:45 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:45 vault1 kernel:
May 29 11:58:45 vault1 kernel: fc_wq_13 S ffff81011ed8e058 0 3479 19 (L-TLB)
May 29 11:58:45 vault1 kernel: ffff81011e079ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:45 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81011ec4b040
May 29 11:58:45 vault1 kernel: ffff81011fd360c0 0000000000000d40 ffff81011ec4b248 000000031ec4b040
May 29 11:58:45 vault1 kernel: Call Trace:
May 29 11:58:45 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:45 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:45 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:45 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:45 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:45 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:45 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:45 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:45 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:45 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:45 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:45 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:45 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:45 vault1 kernel:
May 29 11:58:45 vault1 kernel: fc_dl_13 S ffff81011ed8e058 0 3480 19 (L-TLB)
May 29 11:58:45 vault1 kernel: ffff81011feafea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:45 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81011fcb7080
May 29 11:58:45 vault1 kernel: ffff81011fd360c0 0000000000000c43 ffff81011fcb7288 000000031fcb7080
May 29 11:58:45 vault1 kernel: Call Trace:
May 29 11:58:45 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:45 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:45 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:45 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:45 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:45 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:45 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:45 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:45 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:45 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:45 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:45 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:45 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:45 vault1 kernel:
May 29 11:58:45 vault1 kernel: scsi_eh_14 S ffff81011ed8e058 0 3493 19 (L-TLB)
May 29 11:58:45 vault1 kernel: ffff81011e873e70 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:45 vault1 kernel: 0000000000000000 0000000000000046 0000000000000009 ffff81011e6030c0
May 29 11:58:45 vault1 kernel: ffff81011fd360c0 00000000000007e0 ffff81011e6032c8 0000000380263894
May 29 11:58:45 vault1 kernel: Call Trace:
May 29 11:58:45 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:45 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:45 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 29 11:58:45 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:45 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:45 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 29 11:58:45 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:45 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:45 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:45 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:45 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:45 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:45 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:45 vault1 kernel:
May 29 11:58:45 vault1 kernel: qla2xxx_14_dp S ffff81007f343638 0 3501 19 (L-TLB)
May 29 11:58:45 vault1 kernel: ffff81011e111ee0 0000000000000046 ffff810074d4c7d0 0000000000000282
May 29 11:58:45 vault1 kernel: ffff81011e111e40 ffff81011e111e40 000000000000000a ffff81011e122080
May 29 11:58:45 vault1 kernel: ffff81007fe100c0 0000000000000afb ffff81011e122288 0000000000000000
May 29 11:58:45 vault1 kernel: Call Trace:
May 29 11:58:45 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:45 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 29 11:58:45 vault1 kernel: [<ffffffff880ba348>] :qla2xxx:qla2x00_do_dpc+0x39/0x439
May 29 11:58:45 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:45 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 29 11:58:45 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:45 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:45 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:45 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:45 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:45 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:46 vault1 kernel:
May 29 11:58:46 vault1 kernel: scsi_wq_14 S ffff81007ebd3638 0 3502 19 (L-TLB)
May 29 11:58:46 vault1 kernel: ffff81011e049ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:46 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81011e0570c0
May 29 11:58:46 vault1 kernel: ffff81007e0fe080 000000000036e7a4 ffff81011e0572c8 000000001e0570c0
May 29 11:58:46 vault1 kernel: Call Trace:
May 29 11:58:46 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:46 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:46 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:46 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:46 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:46 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:46 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:46 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:46 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:46 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:46 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:46 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:46 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:46 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:46 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:46 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:46 vault1 kernel:
May 29 11:58:46 vault1 kernel: fc_wq_14 S ffff81011ed8e058 0 3503 19 (L-TLB)
May 29 11:58:46 vault1 kernel: ffff81011f0b3ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:46 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81011e117040
May 29 11:58:46 vault1 kernel: ffff81011fd360c0 0000000000000dc9 ffff81011e117248 000000031e117040
May 29 11:58:46 vault1 kernel: Call Trace:
May 29 11:58:46 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:46 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:46 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:46 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:46 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:46 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:46 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:46 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:46 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:46 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:46 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:46 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:46 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:46 vault1 kernel:
May 29 11:58:46 vault1 kernel: fc_dl_14 S ffff81011ed8e058 0 3504 19 (L-TLB)
May 29 11:58:46 vault1 kernel: ffff81011e189ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:46 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81011f9ea080
May 29 11:58:46 vault1 kernel: ffff81011fd360c0 0000000000000dda ffff81011f9ea288 000000031f9ea080
May 29 11:58:46 vault1 kernel: Call Trace:
May 29 11:58:46 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:46 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:46 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:46 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:46 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:46 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:46 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:46 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:46 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:46 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:46 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:46 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:46 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:46 vault1 kernel:
May 29 11:58:46 vault1 kernel: rpc.rquotad S ffff81011febdad8 0 3740 1 (NOTLB)
May 29 11:58:46 vault1 kernel: ffff81011dd5fb28 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:46 vault1 kernel: 0000000180248a3b 0000000000000002 0000000000000001 ffff81011ddca080
May 29 11:58:46 vault1 kernel: ffffffff8058a6e0 000000000002fcdc ffff81011ddca288 0000000000000000
May 29 11:58:46 vault1 kernel: Call Trace:
May 29 11:58:46 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:46 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:46 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:46 vault1 kernel: [<ffffffff80213f07>] tcp_poll+0x1f/0x133
May 29 11:58:46 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 29 11:58:46 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:46 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:46 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:46 vault1 kernel: [<ffffffff8020a6c3>] get_page_from_freelist+0x2ab/0x534
May 29 11:58:46 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:46 vault1 kernel: [<ffffffff8020a6c3>] get_page_from_freelist+0x2ab/0x534
May 29 11:58:46 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:46 vault1 kernel: [<ffffffff802078e8>] find_get_page+0x15/0x4b
May 29 11:58:46 vault1 kernel: [<ffffffff802638bb>] _read_unlock_irq+0x24/0x27
May 29 11:58:46 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:46 vault1 kernel: [<ffffffff80208c1f>] __handle_mm_fault+0x3cf/0xccf
May 29 11:58:46 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:46 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 29 11:58:46 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 29 11:58:46 vault1 kernel: [<ffffffff80208d64>] __handle_mm_fault+0x514/0xccf
May 29 11:58:46 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:46 vault1 kernel: [<ffffffff80265bc5>] do_page_fault+0x490/0x7cf
May 29 11:58:46 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:47 vault1 kernel: [<ffffffff8021e472>] sys_close+0x2b/0xd1
May 29 11:58:47 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:47 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 29 11:58:47 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:47 vault1 kernel:
May 29 11:58:47 vault1 kernel: lockd S ffff81007fe18db8 0 3743 1 (L-TLB)
May 29 11:58:47 vault1 kernel: ffff81007620de10 0000000000000046 0000000000000000 ffff81007ffe3830
May 29 11:58:47 vault1 kernel: 0000000000000627 ffff81007ffe30c0 0000000000000001 ffff81007ffe30c0
May 29 11:58:47 vault1 kernel: ffff81011fccf080 0000000000003c35 ffff81007ffe32c8 0000000200000046
May 29 11:58:47 vault1 kernel: Call Trace:
May 29 11:58:47 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:47 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:47 vault1 kernel: [<ffffffff882e824e>] :sunrpc:svc_recv+0x2b5/0x44f
May 29 11:58:47 vault1 kernel: [<ffffffff882e8257>] :sunrpc:svc_recv+0x2be/0x44f
May 29 11:58:47 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:47 vault1 kernel: [<ffffffff88315556>] :lockd:lockd+0x139/0x263
May 29 11:58:47 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:47 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:47 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:47 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:47 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:47 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:47 vault1 kernel: [<ffffffff8831541d>] :lockd:lockd+0x0/0x263
May 29 11:58:47 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:47 vault1 kernel:
May 29 11:58:47 vault1 kernel: rpciod/0 S ffff81011f96e4b8 0 3744 19 (L-TLB)
May 29 11:58:47 vault1 kernel: ffff81011d85dea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:47 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011e7ae0c0
May 29 11:58:47 vault1 kernel: ffffffff8058a6e0 0000000000001725 ffff81011e7ae2c8 000000001e7ae0c0
May 29 11:58:47 vault1 kernel: Call Trace:
May 29 11:58:47 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:47 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:47 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:47 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:47 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:47 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:47 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:47 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:47 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:47 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:47 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:47 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:47 vault1 kernel:
May 29 11:58:47 vault1 kernel: rpciod/1 S ffff81007e836df8 0 3745 19 (L-TLB)
May 29 11:58:47 vault1 kernel: ffff81011e8ebea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:47 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011dd85040
May 29 11:58:47 vault1 kernel: ffff81011eca90c0 0000000000001573 ffff81011dd85248 000000011dd85040
May 29 11:58:47 vault1 kernel: Call Trace:
May 29 11:58:47 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:47 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:47 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:47 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:47 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:47 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:47 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:47 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:47 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:47 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:47 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:47 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:47 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:47 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:47 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:47 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:47 vault1 kernel:
May 29 11:58:47 vault1 kernel: rpciod/2 S ffff81007e836df8 0 3746 19 (L-TLB)
May 29 11:58:47 vault1 kernel: ffff81011e2ebea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:47 vault1 kernel: 0000000000000000 0000000000000000 0000000000000005 ffff81011dd83080
May 29 11:58:47 vault1 kernel: ffff81007ffe30c0 00000000000013c6 ffff81011dd83288 000000021dd83080
May 29 11:58:47 vault1 kernel: Call Trace:
May 29 11:58:47 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:47 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:47 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:47 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:47 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:47 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:47 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:47 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:47 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:47 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:47 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:47 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:47 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:47 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:48 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:48 vault1 kernel:
May 29 11:58:48 vault1 kernel: rpciod/3 S ffff81011ed8e058 0 3747 19 (L-TLB)
May 29 11:58:48 vault1 kernel: ffff81011d86fea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:48 vault1 kernel: 0000000000000000 0000000000000000 0000000000000005 ffff81011d8d10c0
May 29 11:58:48 vault1 kernel: ffff81011fd360c0 000000000000117b ffff81011d8d12c8 000000031d8d10c0
May 29 11:58:48 vault1 kernel: Call Trace:
May 29 11:58:48 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:48 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:48 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:48 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:48 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:48 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:48 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:48 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:48 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:48 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:48 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:48 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:48 vault1 kernel:
May 29 11:58:48 vault1 kernel: nfsd4 S ffff81007f3424b8 0 3748 19 (L-TLB)
May 29 11:58:48 vault1 kernel: ffff81011e229ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:48 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011d8d0040
May 29 11:58:48 vault1 kernel: ffffffff8058a6e0 00000000000018ad ffff81011d8d0248 000000001d8d0040
May 29 11:58:48 vault1 kernel: Call Trace:
May 29 11:58:48 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:48 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:48 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:48 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:48 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:48 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:48 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:48 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:48 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:48 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:48 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:48 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:48 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:48 vault1 kernel:
May 29 11:58:48 vault1 kernel: nfsd D ffff81007e8de9d8 0 3749 1 (L-TLB)
May 29 11:58:48 vault1 kernel: ffff810075cf3a60 0000000000000046 0000000000000540 ffff810074f460c0
May 29 11:58:48 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff810074f460c0
May 29 11:58:48 vault1 kernel: ffff81007e0c80c0 000000000003c1a3 ffff810074f462c8 0000000274f46830
May 29 11:58:48 vault1 kernel: Call Trace:
May 29 11:58:48 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:48 vault1 kernel: [<ffffffff80262019>] __mutex_lock_slowpath+0x13b/0x25b
May 29 11:58:48 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:48 vault1 kernel: [<ffffffff80262019>] __mutex_lock_slowpath+0x13b/0x25b
May 29 11:58:48 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 29 11:58:48 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 29 11:58:48 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:48 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 29 11:58:48 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 29 11:58:48 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:48 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:48 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 29 11:58:48 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:48 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 29 11:58:48 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 29 11:58:48 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 29 11:58:48 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:48 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:48 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:48 vault1 kernel: [<ffffffff8832fc8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 29 11:58:48 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 29 11:58:48 vault1 kernel: [<ffffffff88330305>] :nfsd:nfsd_open+0x13c/0x16f
May 29 11:58:48 vault1 kernel: [<ffffffff8833053b>] :nfsd:nfsd_write+0xc5/0xe2
May 29 11:58:48 vault1 kernel: [<ffffffff88337611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 29 11:58:48 vault1 kernel: [<ffffffff8832c244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 29 11:58:48 vault1 kernel: [<ffffffff882e5353>] :sunrpc:svc_process+0x3ea/0x6f3
May 29 11:58:48 vault1 kernel: [<ffffffff8832c7fe>] :nfsd:nfsd+0x194/0x2b2
May 29 11:58:48 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:48 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:48 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:48 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:48 vault1 kernel: [<ffffffff8832c66a>] :nfsd:nfsd+0x0/0x2b2
May 29 11:58:48 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:48 vault1 kernel:
May 29 11:58:48 vault1 kernel: nfsd D ffff81007e8de118 0 3750 1 (L-TLB)
May 29 11:58:48 vault1 kernel: ffff8100765fba60 0000000000000046 0000000000000000 ffff81007ea050c0
May 29 11:58:48 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff81007ea050c0
May 29 11:58:48 vault1 kernel: ffffffff8058a6e0 000000000004b18a ffff81007ea052c8 000000007ea05830
May 29 11:58:48 vault1 kernel: Call Trace:
May 29 11:58:48 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:48 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:48 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 29 11:58:48 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 29 11:58:48 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:48 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 29 11:58:48 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 29 11:58:48 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:48 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:48 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 29 11:58:48 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:48 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 29 11:58:48 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 29 11:58:49 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 29 11:58:49 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:49 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:49 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:49 vault1 kernel: [<ffffffff8832fc8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 29 11:58:49 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 29 11:58:49 vault1 kernel: [<ffffffff882ec9af>] :sunrpc:sunrpc_cache_lookup+0x30/0x131
May 29 11:58:49 vault1 kernel: [<ffffffff88330305>] :nfsd:nfsd_open+0x13c/0x16f
May 29 11:58:49 vault1 kernel: [<ffffffff8833053b>] :nfsd:nfsd_write+0xc5/0xe2
May 29 11:58:49 vault1 kernel: [<ffffffff88337611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 29 11:58:49 vault1 kernel: [<ffffffff8832c244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 29 11:58:49 vault1 kernel: [<ffffffff882e5353>] :sunrpc:svc_process+0x3ea/0x6f3
May 29 11:58:49 vault1 kernel: [<ffffffff8832c7fe>] :nfsd:nfsd+0x194/0x2b2
May 29 11:58:49 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:49 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:49 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:49 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:49 vault1 kernel: [<ffffffff8832c66a>] :nfsd:nfsd+0x0/0x2b2
May 29 11:58:49 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:49 vault1 kernel:
May 29 11:58:49 vault1 kernel: nfsd D ffff81007e8de9d8 0 3751 1 (L-TLB)
May 29 11:58:49 vault1 kernel: ffff81007618fa60 0000000000000046 0000000000000000 ffff81007e9fd040
May 29 11:58:49 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff81007e9fd040
May 29 11:58:49 vault1 kernel: ffff81011fccf080 000000000005db1b ffff81007e9fd248 000000027e9fd7b0
May 29 11:58:49 vault1 kernel: Call Trace:
May 29 11:58:49 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:49 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:49 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 29 11:58:49 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 29 11:58:49 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:49 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 29 11:58:49 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 29 11:58:49 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:49 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:49 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 29 11:58:49 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:49 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 29 11:58:49 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 29 11:58:49 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 29 11:58:49 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:49 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:49 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:49 vault1 kernel: [<ffffffff8832fc8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 29 11:58:49 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 29 11:58:49 vault1 kernel: [<ffffffff88330305>] :nfsd:nfsd_open+0x13c/0x16f
May 29 11:58:49 vault1 kernel: [<ffffffff8833053b>] :nfsd:nfsd_write+0xc5/0xe2
May 29 11:58:49 vault1 kernel: [<ffffffff88337611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 29 11:58:49 vault1 kernel: [<ffffffff8832c244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 29 11:58:49 vault1 kernel: [<ffffffff882e5353>] :sunrpc:svc_process+0x3ea/0x6f3
May 29 11:58:49 vault1 kernel: [<ffffffff8832c7fe>] :nfsd:nfsd+0x194/0x2b2
May 29 11:58:49 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:49 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:49 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:49 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:49 vault1 kernel: [<ffffffff8832c66a>] :nfsd:nfsd+0x0/0x2b2
May 29 11:58:49 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:49 vault1 kernel:
May 29 11:58:49 vault1 kernel: nfsd D ffff81007e8de118 0 3752 1 (L-TLB)
May 29 11:58:49 vault1 kernel: ffff81007699fa60 0000000000000046 0000000000000540 ffff81007e4c5080
May 29 11:58:49 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff81007e4c5080
May 29 11:58:49 vault1 kernel: ffff81007e1b9040 000000000001b7e6 ffff81007e4c5288 000000007e4c57f0
May 29 11:58:49 vault1 kernel: Call Trace:
May 29 11:58:49 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:49 vault1 kernel: [<ffffffff80262019>] __mutex_lock_slowpath+0x13b/0x25b
May 29 11:58:49 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:49 vault1 kernel: [<ffffffff80262019>] __mutex_lock_slowpath+0x13b/0x25b
May 29 11:58:49 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 29 11:58:49 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 29 11:58:49 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:49 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 29 11:58:49 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 29 11:58:49 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:49 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:49 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 29 11:58:49 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:49 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 29 11:58:49 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 29 11:58:49 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 29 11:58:49 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:49 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:49 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:49 vault1 kernel: [<ffffffff8832fc8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 29 11:58:49 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 29 11:58:49 vault1 kernel: [<ffffffff882ec9af>] :sunrpc:sunrpc_cache_lookup+0x30/0x131
May 29 11:58:49 vault1 kernel: [<ffffffff88330305>] :nfsd:nfsd_open+0x13c/0x16f
May 29 11:58:49 vault1 kernel: [<ffffffff8833053b>] :nfsd:nfsd_write+0xc5/0xe2
May 29 11:58:49 vault1 kernel: [<ffffffff88337611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 29 11:58:49 vault1 kernel: [<ffffffff8832c244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 29 11:58:49 vault1 kernel: [<ffffffff882e5353>] :sunrpc:svc_process+0x3ea/0x6f3
May 29 11:58:49 vault1 kernel: [<ffffffff8832c7fe>] :nfsd:nfsd+0x194/0x2b2
May 29 11:58:50 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:50 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:50 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:50 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:50 vault1 kernel: [<ffffffff8832c66a>] :nfsd:nfsd+0x0/0x2b2
May 29 11:58:50 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:50 vault1 kernel:
May 29 11:58:50 vault1 kernel: nfsd D ffff81007e8de118 0 3753 1 (L-TLB)
May 29 11:58:50 vault1 kernel: ffff81007663ba60 0000000000000046 0000000000000000 ffff81007e1b9040
May 29 11:58:50 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff81007e1b9040
May 29 11:58:50 vault1 kernel: ffffffff8058a6e0 00000000000396b7 ffff81007e1b9248 000000007e1b97b0
May 29 11:58:50 vault1 kernel: Call Trace:
May 29 11:58:50 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:50 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:50 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 29 11:58:50 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 29 11:58:50 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:50 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 29 11:58:50 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 29 11:58:50 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:50 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:50 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 29 11:58:50 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:50 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 29 11:58:50 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 29 11:58:50 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 29 11:58:50 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:50 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:50 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:50 vault1 kernel: [<ffffffff8832fc8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 29 11:58:50 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 29 11:58:50 vault1 kernel: [<ffffffff882ec9af>] :sunrpc:sunrpc_cache_lookup+0x30/0x131
May 29 11:58:50 vault1 kernel: [<ffffffff88330305>] :nfsd:nfsd_open+0x13c/0x16f
May 29 11:58:50 vault1 kernel: [<ffffffff8833053b>] :nfsd:nfsd_write+0xc5/0xe2
May 29 11:58:50 vault1 kernel: [<ffffffff88337611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 29 11:58:50 vault1 kernel: [<ffffffff8832c244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 29 11:58:50 vault1 kernel: [<ffffffff882e5353>] :sunrpc:svc_process+0x3ea/0x6f3
May 29 11:58:50 vault1 kernel: [<ffffffff8832c7fe>] :nfsd:nfsd+0x194/0x2b2
May 29 11:58:50 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:50 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:50 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:50 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:50 vault1 kernel: [<ffffffff8832c66a>] :nfsd:nfsd+0x0/0x2b2
May 29 11:58:50 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:50 vault1 kernel:
May 29 11:58:50 vault1 kernel: nfsd D ffff81007e8de9d8 0 3754 1 (L-TLB)
May 29 11:58:50 vault1 kernel: ffff810076203a60 0000000000000046 0000000000000540 ffff81007e0c80c0
May 29 11:58:50 vault1 kernel: 0000000000000001 ffffffff802a0de5 0000000000000009 ffff81007e0c80c0
May 29 11:58:50 vault1 kernel: ffff81007e9fd040 0000000000051705 ffff81007e0c82c8 000000027e0c8830
May 29 11:58:50 vault1 kernel: Call Trace:
May 29 11:58:50 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:50 vault1 kernel: [<ffffffff80262019>] __mutex_lock_slowpath+0x13b/0x25b
May 29 11:58:50 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:50 vault1 kernel: [<ffffffff80262019>] __mutex_lock_slowpath+0x13b/0x25b
May 29 11:58:50 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 29 11:58:50 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 29 11:58:50 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:50 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 29 11:58:50 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 29 11:58:50 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:50 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:50 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 29 11:58:50 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:50 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 29 11:58:50 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 29 11:58:50 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 29 11:58:50 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:50 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:50 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:50 vault1 kernel: [<ffffffff8832fc8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 29 11:58:50 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 29 11:58:50 vault1 kernel: [<ffffffff88330305>] :nfsd:nfsd_open+0x13c/0x16f
May 29 11:58:50 vault1 kernel: [<ffffffff8833053b>] :nfsd:nfsd_write+0xc5/0xe2
May 29 11:58:50 vault1 kernel: [<ffffffff88337611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 29 11:58:50 vault1 kernel: [<ffffffff8832c244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 29 11:58:50 vault1 kernel: [<ffffffff882e5353>] :sunrpc:svc_process+0x3ea/0x6f3
May 29 11:58:50 vault1 kernel: [<ffffffff8832c7fe>] :nfsd:nfsd+0x194/0x2b2
May 29 11:58:50 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:50 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:50 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:50 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:50 vault1 kernel: [<ffffffff8832c66a>] :nfsd:nfsd+0x0/0x2b2
May 29 11:58:50 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:50 vault1 kernel:
May 29 11:58:50 vault1 kernel: nfsd D ffff81007e8de9d8 0 3755 1 (L-TLB)
May 29 11:58:50 vault1 kernel: ffff810071643a60 0000000000000046 0000000000000540 ffff81007fc350c0
May 29 11:58:50 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff81007fc350c0
May 29 11:58:51 vault1 kernel: ffff81011fd34040 000000000001ead4 ffff81007fc352c8 000000037fc35830
May 29 11:58:51 vault1 kernel: Call Trace:
May 29 11:58:51 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:51 vault1 kernel: [<ffffffff80262019>] __mutex_lock_slowpath+0x13b/0x25b
May 29 11:58:51 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:51 vault1 kernel: [<ffffffff80262019>] __mutex_lock_slowpath+0x13b/0x25b
May 29 11:58:51 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 29 11:58:51 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 29 11:58:51 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:51 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 29 11:58:51 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 29 11:58:51 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:51 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:51 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 29 11:58:51 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:51 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 29 11:58:51 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 29 11:58:51 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 29 11:58:51 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:51 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:51 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:51 vault1 kernel: [<ffffffff8832fc8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 29 11:58:51 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 29 11:58:51 vault1 kernel: [<ffffffff88330305>] :nfsd:nfsd_open+0x13c/0x16f
May 29 11:58:51 vault1 kernel: [<ffffffff8833053b>] :nfsd:nfsd_write+0xc5/0xe2
May 29 11:58:51 vault1 kernel: [<ffffffff88337611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 29 11:58:51 vault1 kernel: [<ffffffff8832c244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 29 11:58:51 vault1 kernel: [<ffffffff882e5353>] :sunrpc:svc_process+0x3ea/0x6f3
May 29 11:58:51 vault1 kernel: [<ffffffff8832c7fe>] :nfsd:nfsd+0x194/0x2b2
May 29 11:58:51 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:51 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:51 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:51 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:51 vault1 kernel: [<ffffffff8832c66a>] :nfsd:nfsd+0x0/0x2b2
May 29 11:58:51 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:51 vault1 kernel:
May 29 11:58:51 vault1 kernel: nfsd D ffff81007e8de9d8 0 3756 1 (L-TLB)
May 29 11:58:51 vault1 kernel: ffff8100761f3620 0000000000000046 0000000000000002 0000000000000000
May 29 11:58:51 vault1 kernel: 0000000000000000 0000000000000046 0000000000000009 ffff81007ea200c0
May 29 11:58:51 vault1 kernel: ffff81007e0c80c0 000000000000c278 ffff81007ea202c8 000000037ea208a0
May 29 11:58:51 vault1 kernel: Call Trace:
May 29 11:58:51 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:51 vault1 kernel: [<ffffffff802616cf>] io_schedule+0x3f/0x67
May 29 11:58:51 vault1 kernel: [<ffffffff882456d9>] :jfs:txBeginAnon+0x85/0x13c
May 29 11:58:51 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:51 vault1 kernel: [<ffffffff8823f2f2>] :jfs:extAlloc+0x3d/0x423
May 29 11:58:51 vault1 kernel: [<ffffffff8823f786>] :jfs:extHint+0xae/0xfc
May 29 11:58:51 vault1 kernel: [<ffffffff8822c65c>] :jfs:jfs_get_block+0x167/0x237
May 29 11:58:51 vault1 kernel: [<ffffffff802e5c6c>] nobh_prepare_write+0xd3/0x458
May 29 11:58:51 vault1 kernel: [<ffffffff8822c4f5>] :jfs:jfs_get_block+0x0/0x237
May 29 11:58:51 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:51 vault1 kernel: [<ffffffff8020c722>] add_to_page_cache+0x2d/0x8c
May 29 11:58:51 vault1 kernel: [<ffffffff80263540>] _write_unlock_irq+0x24/0x27
May 29 11:58:51 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:51 vault1 kernel: [<ffffffff80263540>] _write_unlock_irq+0x24/0x27
May 29 11:58:51 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:51 vault1 kernel: [<ffffffff80263540>] _write_unlock_irq+0x24/0x27
May 29 11:58:51 vault1 kernel: [<ffffffff80210354>] generic_file_buffered_write+0x27b/0x77b
May 29 11:58:51 vault1 kernel: [<ffffffff802606f4>] __sched_text_start+0x184/0x960
May 29 11:58:51 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:51 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:51 vault1 kernel: [<ffffffff8020e533>] current_fs_time+0x3b/0x40
May 29 11:58:51 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:51 vault1 kernel: [<ffffffff80216881>] __generic_file_aio_write_nolock+0x378/0x3eb
May 29 11:58:51 vault1 kernel: [<ffffffff80262123>] __mutex_lock_slowpath+0x245/0x25b
May 29 11:58:51 vault1 kernel: [<ffffffff8026212e>] __mutex_lock_slowpath+0x250/0x25b
May 29 11:58:51 vault1 kernel: [<ffffffff802224a4>] generic_file_aio_write+0x61/0xc1
May 29 11:58:51 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 29 11:58:51 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 29 11:58:51 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:51 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:51 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 29 11:58:51 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:51 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 29 11:58:51 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 29 11:58:51 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 29 11:58:51 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:51 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:51 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:51 vault1 kernel: [<ffffffff8832fc8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 29 11:58:51 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 29 11:58:51 vault1 kernel: [<ffffffff88330305>] :nfsd:nfsd_open+0x13c/0x16f
May 29 11:58:51 vault1 kernel: [<ffffffff8833053b>] :nfsd:nfsd_write+0xc5/0xe2
May 29 11:58:51 vault1 kernel: [<ffffffff88337611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 29 11:58:51 vault1 kernel: [<ffffffff8832c244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 29 11:58:51 vault1 kernel: [<ffffffff882e5353>] :sunrpc:svc_process+0x3ea/0x6f3
May 29 11:58:51 vault1 kernel: [<ffffffff8832c7fe>] :nfsd:nfsd+0x194/0x2b2
May 29 11:58:51 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:51 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:51 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:51 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:51 vault1 kernel: [<ffffffff8832c66a>] :nfsd:nfsd+0x0/0x2b2
May 29 11:58:51 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:51 vault1 kernel:
May 29 11:58:51 vault1 kernel: rpc.mountd S ffff81007e8dee38 0 3759 1 (NOTLB)
May 29 11:58:51 vault1 kernel: ffff8100766c5a18 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:51 vault1 kernel: 0000000180248a3b 0000000000000002 0000000000000009 ffff81007e618040
May 29 11:58:51 vault1 kernel: ffff81011fc78040 00000000000120f1 ffff81007e618248 0000000100000000
May 29 11:58:51 vault1 kernel: Call Trace:
May 29 11:58:51 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:52 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:52 vault1 kernel: [<ffffffff80213f07>] tcp_poll+0x1f/0x133
May 29 11:58:52 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 29 11:58:52 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:52 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:52 vault1 last message repeated 4 times
May 29 11:58:52 vault1 kernel: [<ffffffff8020b406>] kfree+0x1a8/0x274
May 29 11:58:52 vault1 kernel: [<ffffffff8020b4c4>] kfree+0x266/0x274
May 29 11:58:52 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:52 vault1 kernel: [<ffffffff8020b406>] kfree+0x1a8/0x274
May 29 11:58:52 vault1 kernel: [<ffffffff8020b4c4>] kfree+0x266/0x274
May 29 11:58:52 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:52 vault1 kernel: [<ffffffff8020b4c4>] kfree+0x266/0x274
May 29 11:58:52 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:52 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 29 11:58:52 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:52 vault1 kernel: [<ffffffff882ec363>] :sunrpc:cache_write+0x45/0xc0
May 29 11:58:52 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:52 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:52 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:52 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:52 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 29 11:58:52 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:52 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:52 vault1 kernel:
May 29 11:58:52 vault1 kernel: bash S ffff81011fff8998 0 4891 3311 (NOTLB)
May 29 11:58:52 vault1 kernel: ffff8100339c3ec8 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:52 vault1 kernel: 0000000000000000 0000000200000002 0000000000000009 ffff81004a1760c0
May 29 11:58:52 vault1 kernel: ffff81011fccf080 00000000000a29d2 ffff81004a1762c8 0000000200000000
May 29 11:58:52 vault1 kernel: Call Trace:
May 29 11:58:52 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:52 vault1 kernel: [<ffffffff8022920b>] do_wait+0x94c/0xa75
May 29 11:58:52 vault1 kernel: [<ffffffff8022922e>] do_wait+0x96f/0xa75
May 29 11:58:52 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:52 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:52 vault1 kernel:
May 29 11:58:52 vault1 kernel: ssh S ffff81007f343a98 0 4918 4891 (NOTLB)
May 29 11:58:52 vault1 kernel: ffff810116ffba18 0000000000000046 0000000000000000 0000000000000046
May 29 11:58:52 vault1 kernel: ffffffff8022edfd 0000000000000046 000000000000000a ffff810091f5d0c0
May 29 11:58:52 vault1 kernel: ffff81011fd360c0 0000000000073aab ffff810091f5d2c8 0000000391f5d0c0
May 29 11:58:52 vault1 kernel: Call Trace:
May 29 11:58:52 vault1 kernel: [<ffffffff8022edfd>] __wake_up+0x22/0x4e
May 29 11:58:52 vault1 kernel: [<ffffffff8023a80a>] tty_ldisc_deref+0x62/0x75
May 29 11:58:52 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:52 vault1 kernel: [<ffffffff80226c98>] tty_poll+0x5f/0x6d
May 29 11:58:52 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 29 11:58:52 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:52 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:52 vault1 last message repeated 4 times
May 29 11:58:52 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:52 vault1 kernel: [<ffffffff802174d3>] release_console_sem+0x44/0x1eb
May 29 11:58:52 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:52 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:52 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:52 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:52 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:52 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 29 11:58:52 vault1 kernel: [<ffffffff8022edfd>] __wake_up+0x22/0x4e
May 29 11:58:52 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:52 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:52 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:52 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 29 11:58:52 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:52 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:52 vault1 kernel:
May 29 11:58:52 vault1 kernel: sshd S ffff81007f1149d8 0 6871 3092 (NOTLB)
May 29 11:58:52 vault1 kernel: ffff81011e20da18 0000000000000046 0000000000000000 0000000000000046
May 29 11:58:52 vault1 kernel: ffffffff8022edfd 0000000000000046 000000000000000a ffff81010bea70c0
May 29 11:58:52 vault1 kernel: ffff81011fc78040 0000000000012ee2 ffff81010bea72c8 000000010bea70c0
May 29 11:58:52 vault1 kernel: Call Trace:
May 29 11:58:52 vault1 kernel: [<ffffffff8022edfd>] __wake_up+0x22/0x4e
May 29 11:58:52 vault1 kernel: [<ffffffff8023a80a>] tty_ldisc_deref+0x62/0x75
May 29 11:58:52 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:52 vault1 kernel: [<ffffffff80226c98>] tty_poll+0x5f/0x6d
May 29 11:58:52 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 29 11:58:52 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:52 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:53 vault1 last message repeated 3 times
May 29 11:58:53 vault1 kernel: [<ffffffff802313cd>] release_sock+0x2f/0xd6
May 29 11:58:53 vault1 kernel: [<ffffffff8028de66>] local_bh_enable_ip+0xe4/0xea
May 29 11:58:53 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:53 vault1 kernel: [<ffffffff80227651>] tcp_sendmsg+0x9ab/0xab5
May 29 11:58:53 vault1 kernel: [<ffffffff80227651>] tcp_sendmsg+0x9ab/0xab5
May 29 11:58:53 vault1 kernel: [<ffffffff8024671c>] sock_aio_write+0x110/0x128
May 29 11:58:53 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:53 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 29 11:58:53 vault1 kernel: [<ffffffff80218507>] do_sync_write+0xc9/0x10c
May 29 11:58:53 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:53 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:53 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:53 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 29 11:58:53 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:53 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:53 vault1 kernel:
May 29 11:58:53 vault1 kernel: pdflush D ffff81007e8de9d8 0 8692 19 (L-TLB)
May 29 11:58:53 vault1 kernel: ffff81005aba3a20 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:53 vault1 kernel: 0000000137d30000 0000000000000002 000000000000000a ffff81004687d0c0
May 29 11:58:53 vault1 kernel: ffff81011fd360c0 0000000000003dcf ffff81004687d2c8 0000000300000001
May 29 11:58:53 vault1 kernel: Call Trace:
May 29 11:58:53 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:53 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:53 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:53 vault1 kernel: [<ffffffff80262fa6>] __down_write_nested+0x7e/0x96
May 29 11:58:53 vault1 kernel: [<ffffffff8822c53a>] :jfs:jfs_get_block+0x45/0x237
May 29 11:58:53 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 29 11:58:53 vault1 kernel: [<ffffffff802e9478>] __mpage_writepage+0x1f5/0x55d
May 29 11:58:53 vault1 kernel: [<ffffffff8822c4f5>] :jfs:jfs_get_block+0x0/0x237
May 29 11:58:53 vault1 kernel: [<ffffffff802638bb>] _read_unlock_irq+0x24/0x27
May 29 11:58:53 vault1 kernel: [<ffffffff802c0835>] dec_zone_page_state+0x33/0x3a
May 29 11:58:53 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:53 vault1 kernel: [<ffffffff802c0835>] dec_zone_page_state+0x33/0x3a
May 29 11:58:53 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:53 vault1 kernel: [<ffffffff8021d493>] mpage_writepages+0x228/0x375
May 29 11:58:53 vault1 kernel: [<ffffffff8822c0bb>] :jfs:jfs_writepage+0x0/0xf
May 29 11:58:53 vault1 kernel: [<ffffffff8822c4f5>] :jfs:jfs_get_block+0x0/0x237
May 29 11:58:53 vault1 kernel: [<ffffffff802593b8>] do_writepages+0x23/0x30
May 29 11:58:53 vault1 kernel: [<ffffffff80230487>] __writeback_single_inode+0x1c0/0x338
May 29 11:58:53 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:53 vault1 kernel: [<ffffffff80263584>] _read_unlock+0x17/0x20
May 29 11:58:53 vault1 kernel: [<ffffffff80221c26>] sync_sb_inodes+0x1a6/0x267
May 29 11:58:53 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:53 vault1 kernel: [<ffffffff8024f304>] writeback_inodes+0x95/0xee
May 29 11:58:53 vault1 kernel: [<ffffffff802be735>] wb_kupdate+0x9e/0x112
May 29 11:58:53 vault1 kernel: [<ffffffff80254d52>] pdflush+0x0/0x1e7
May 29 11:58:53 vault1 kernel: [<ffffffff80254d52>] pdflush+0x0/0x1e7
May 29 11:58:53 vault1 kernel: [<ffffffff80254e8e>] pdflush+0x13c/0x1e7
May 29 11:58:53 vault1 kernel: [<ffffffff802be697>] wb_kupdate+0x0/0x112
May 29 11:58:53 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:53 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:53 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:53 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:53 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:53 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:53 vault1 kernel:
May 29 11:58:53 vault1 kernel: pdflush S ffff81007e8de9d8 0 10661 19 (L-TLB)
May 29 11:58:53 vault1 kernel: ffff81007e775ed0 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:53 vault1 kernel: 0000000180598080 0000000000000002 000000000000000a ffff810055df5040
May 29 11:58:53 vault1 kernel: ffff81011fccf080 0000000000003d7f ffff810055df5248 0000000200000000
May 29 11:58:53 vault1 kernel: Call Trace:
May 29 11:58:53 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:53 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:53 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:53 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:53 vault1 kernel: [<ffffffff80254d52>] pdflush+0x0/0x1e7
May 29 11:58:53 vault1 kernel: [<ffffffff80254e21>] pdflush+0xcf/0x1e7
May 29 11:58:53 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:53 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:53 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:53 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:53 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:53 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:53 vault1 kernel:
May 29 11:58:53 vault1 kernel:
May 29 11:58:53 vault1 kernel: Showing all locks held in the system:
May 29 11:58:53 vault1 kernel: 1 lock held by syslogd/2963:
May 29 11:58:53 vault1 kernel: #0: (&inode->i_mutex){--..}, at: [<ffffffff8024e516>] do_fsync+0x41/0xa4
May 29 11:58:53 vault1 kernel: 1 lock held by mingetty/3312:
May 29 11:58:53 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 29 11:58:53 vault1 kernel: 1 lock held by mingetty/3313:
May 29 11:58:53 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 29 11:58:53 vault1 kernel: 1 lock held by mingetty/3314:
May 29 11:58:53 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 29 11:58:53 vault1 kernel: 1 lock held by mingetty/3315:
May 29 11:58:53 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 29 11:58:53 vault1 kernel: 1 lock held by mingetty/3316:
May 29 11:58:53 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 29 11:58:53 vault1 kernel: 1 lock held by bash/3377:
May 29 11:58:53 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 29 11:58:53 vault1 kernel: 2 locks held by nfsd/3749:
May 29 11:58:53 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff8832c7e7>] nfsd+0x17d/0x2b2 [nfsd]
May 29 11:58:54 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:54 vault1 kernel: 2 locks held by nfsd/3750:
May 29 11:58:54 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff8832c7e7>] nfsd+0x17d/0x2b2 [nfsd]
May 29 11:58:54 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:54 vault1 kernel: 2 locks held by nfsd/3751:
May 29 11:58:54 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff8832c7e7>] nfsd+0x17d/0x2b2 [nfsd]
May 29 11:58:54 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:54 vault1 kernel: 2 locks held by nfsd/3752:
May 29 11:58:54 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff8832c7e7>] nfsd+0x17d/0x2b2 [nfsd]
May 29 11:58:54 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:54 vault1 kernel: 2 locks held by nfsd/3753:
May 29 11:58:54 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff8832c7e7>] nfsd+0x17d/0x2b2 [nfsd]
May 29 11:58:54 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:54 vault1 kernel: 2 locks held by nfsd/3754:
May 29 11:58:54 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff8832c7e7>] nfsd+0x17d/0x2b2 [nfsd]
May 29 11:58:54 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:54 vault1 kernel: 2 locks held by nfsd/3755:
May 29 11:58:54 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff8832c7e7>] nfsd+0x17d/0x2b2 [nfsd]
May 29 11:58:54 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:54 vault1 kernel: 3 locks held by nfsd/3756:
May 29 11:58:54 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff8832c7e7>] nfsd+0x17d/0x2b2 [nfsd]
May 29 11:58:54 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:54 vault1 kernel: #2: (&jfs_ip->rdwrlock#2){----}, at: [<ffffffff8822c53a>] jfs_get_block+0x45/0x237 [jfs]
May 29 11:58:54 vault1 kernel: 2 locks held by bash/6873:
May 29 11:58:54 vault1 kernel: #0: (sysrq_key_table_lock){....}, at: [<ffffffff80395b2a>] __handle_sysrq+0x22/0x128
May 29 11:58:54 vault1 kernel: #1: (tasklist_lock){..-?}, at: [<ffffffff8029ebd4>] debug_show_all_locks+0x3e/0x152
May 29 11:58:54 vault1 kernel: 2 locks held by pdflush/8692:
May 29 11:58:54 vault1 kernel: #0: (&type->s_umount_key#20){----}, at: [<ffffffff8024f2e2>] writeback_inodes+0x73/0xee
May 29 11:58:54 vault1 kernel: #1: (&jfs_ip->rdwrlock#2){----}, at: [<ffffffff8822c53a>] jfs_get_block+0x45/0x237 [jfs]
May 29 11:58:54 vault1 kernel:
May 29 11:58:54 vault1 kernel: =============================================
May 29 11:58:54 vault1 kernel:
On Tue, 2007-05-29 at 12:16 -0500, Roger Heflin wrote:
> Dave,
>
> Apparently there appears to be another different similar lockup,
> The MTBF has risen from 1-2 hours without that patch to >100 hours,
> so I am fairly sure the patch did correct the original lockup, or
> at the very least make it a lot less likely.
>
> I hit the machine across NFS for 5 days before it deadlocked, before
> the patch I could only get an hour or two (2-4 different tries).
>
> Given that pdflush is "D" it does not appear to be an NFS issue.
>
> Included is the sysrq-t.
>
> This is with 2.6.21.1 + the JFSIO patch.
Is the system still in this state? Can you cat /proc/fs/jfs/TxAnchor
(if CONFIG_JFS_DEBUG is defined) and /proc/fs/jfs/txstats (if
CONFIG_JFS_STATISTICS is defined)?
Thanks,
Shaggy
--
David Kleikamp
IBM Linux Technology Center
Dave Kleikamp wrote:
> On Tue, 2007-05-29 at 12:16 -0500, Roger Heflin wrote:
>
>> Dave,
>>
>> Apparently there appears to be another different similar lockup,
>> The MTBF has risen from 1-2 hours without that patch to >100 hours,
>> so I am fairly sure the patch did correct the original lockup, or
>> at the very least make it a lot less likely.
>>
>> I hit the machine across NFS for 5 days before it deadlocked, before
>> the patch I could only get an hour or two (2-4 different tries).
>>
>> Given that pdflush is "D" it does not appear to be an NFS issue.
>>
>> Included is the sysrq-t.
>>
>> This is with 2.6.21.1 + the JFSIO patch.
>
> Is the system still in this state? Can you cat /proc/fs/jfs/TxAnchor
> (if CONFIG_JFS_DEBUG is defined) and /proc/fs/jfs/txstats (if
> CONFIG_JFS_STATISTICS is defined)?
>
> Thanks,
> Shaggy
Yes, the machine is still in that state.
Apparently I don't have either of those configured.
Anything else that we can collect before I rebuild the kernel with
those options setup?
Roger
On Tue, 2007-05-29 at 13:14 -0500, Roger Heflin wrote:
> Dave Kleikamp wrote:
> > On Tue, 2007-05-29 at 12:16 -0500, Roger Heflin wrote:
> >
> >> Dave,
> >>
> >> Apparently there appears to be another different similar lockup,
> >> The MTBF has risen from 1-2 hours without that patch to >100 hours,
> >> so I am fairly sure the patch did correct the original lockup, or
> >> at the very least make it a lot less likely.
> >>
> >> I hit the machine across NFS for 5 days before it deadlocked, before
> >> the patch I could only get an hour or two (2-4 different tries).
> >>
> >> Given that pdflush is "D" it does not appear to be an NFS issue.
> >>
> >> Included is the sysrq-t.
> >>
> >> This is with 2.6.21.1 + the JFSIO patch.
> >
> > Is the system still in this state? Can you cat /proc/fs/jfs/TxAnchor
> > (if CONFIG_JFS_DEBUG is defined) and /proc/fs/jfs/txstats (if
> > CONFIG_JFS_STATISTICS is defined)?
> >
> > Thanks,
> > Shaggy
>
> Yes, the machine is still in that state.
>
> Apparently I don't have either of those configured.
>
> Anything else that we can collect before I rebuild the kernel with
> those options setup?
No. I think I may have found something to explain the hang. I need to
look a bit closer. Go ahead and rebuild the kernel with those options
in case I'm mistaken.
Thanks,
Shaggy
>
> Roger
>
--
David Kleikamp
IBM Linux Technology Center