Greetings everyone,
I've been having a bit of a problem since upgrading to the linux 3.x
series, I have a machine that we're using as a NAS that runs various
rsync processes (mostly at night), lately after a day or two, I will
come in in the morning to a load average of 49, but the machine not
really doing anything, when trying to run 'dstat' the command just
hung with no output at all. there were no errors in the logs, or even
anything that would vaguely point at anything I could work with.
So needing to get the machine back to work I attempted to reboot it
"shutdown -r now" on console... it gives a nice message saying it's
going to reboot, but nothing ever happens.. the only way to reboot it
is by using ctrl + alt + sysrq + b. after which the machine reboots
and the raid array comes back clean.
I'm not sure how to troubleshoot this, any pointers would be appreciated.
I'm compiling 3.2.4 at the moment and found a bunch of possibly useful
options in the kernel debugging section:
detect hard/soft lockups and detect hung tasks, maybe it'll give me
something more to go on.
Some details about the machine:
Linux xenbox 3.2.2 #1 SMP Sun Jan 29 10:28:22 CST 2012 x86_64 Intel(R)
Xeon(R) CPU 5140 @ 2.33GHz GenuineIntel GNU/Linux
It has 3 software raid arrays (2 x 5 drives and 1 x 4 drives) LVM'ed
together into a 23TB XFS filesystem.
6GB memory and a pair of Intel Gigabit ethernet controllers bonded together.
Also totally forgot, possibly quite important information, most of the
disks are connected via a pair of LSI controllers:
08:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E
PCI-Express Fusion-MPT SAS (rev 08)
0a:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS
2008 [Falcon] (rev 03)
a couple disks are connected with the onboard Intel SATA controller..
-Gerard Saraber
On Mon, Feb 6, 2012 at 9:40 AM, Gerard Saraber <[email protected]> wrote:
> Greetings everyone,
> I've been having a bit of a problem since upgrading to the linux 3.x
> series, I have a machine that we're using as a NAS that runs various
> rsync processes (mostly at night), lately after a day or two, I will
> come in in the morning to a load average of 49, but the machine not
> really doing anything, when trying to run 'dstat' the command just
> hung with no output at all. there were no errors in the logs, or even
> anything that would vaguely point at anything I could work with.
> So needing to get the machine back to work I attempted to reboot it
> "shutdown -r now" on console... it gives a nice message saying it's
> going to reboot, but nothing ever happens.. the only way to reboot it
> is by using ctrl + alt + sysrq + b. after which the machine reboots
> and the raid array comes back clean.
>
> I'm not sure how to troubleshoot this, any pointers would be appreciated.
>
> I'm compiling 3.2.4 at the moment and found a bunch of possibly useful
> options in the kernel debugging section:
> detect hard/soft lockups and detect hung tasks, maybe it'll give me
> something more to go on.
>
> Some details about the machine:
> Linux xenbox 3.2.2 #1 SMP Sun Jan 29 10:28:22 CST 2012 x86_64 Intel(R)
> Xeon(R) CPU 5140 @ 2.33GHz GenuineIntel GNU/Linux
> It has 3 software raid arrays (2 x 5 drives and 1 x 4 drives) LVM'ed
> together into a 23TB XFS filesystem.
> 6GB memory and a pair of Intel Gigabit ethernet controllers bonded together.
On Mon 06-02-12 09:40:45, Gerard Saraber wrote:
> Greetings everyone,
> I've been having a bit of a problem since upgrading to the linux 3.x
> series, I have a machine that we're using as a NAS that runs various
> rsync processes (mostly at night), lately after a day or two, I will
> come in in the morning to a load average of 49, but the machine not
> really doing anything, when trying to run 'dstat' the command just
> hung with no output at all. there were no errors in the logs, or even
> anything that would vaguely point at anything I could work with.
> So needing to get the machine back to work I attempted to reboot it
> "shutdown -r now" on console... it gives a nice message saying it's
> going to reboot, but nothing ever happens.. the only way to reboot it
> is by using ctrl + alt + sysrq + b. after which the machine reboots
> and the raid array comes back clean.
>
> I'm not sure how to troubleshoot this, any pointers would be appreciated.
>
> I'm compiling 3.2.4 at the moment and found a bunch of possibly useful
> options in the kernel debugging section:
> detect hard/soft lockups and detect hung tasks, maybe it'll give me
> something more to go on.
>
> Some details about the machine:
> Linux xenbox 3.2.2 #1 SMP Sun Jan 29 10:28:22 CST 2012 x86_64 Intel(R)
> Xeon(R) CPU 5140 @ 2.33GHz GenuineIntel GNU/Linux
> It has 3 software raid arrays (2 x 5 drives and 1 x 4 drives) LVM'ed
> together into a 23TB XFS filesystem.
> 6GB memory and a pair of Intel Gigabit ethernet controllers bonded together.
Hmm, might be some deadlock in the filesystem. Adding XFS guys to CC.
Can you run 'echo w >/proc/sysrq-trigger' and post output of dmesg here?
Honza
--
Jan Kara <[email protected]>
SUSE Labs, CR
On Mon, Feb 6, 2012 at 4:51 PM, Jan Kara <[email protected]> wrote:
> On Mon 06-02-12 09:40:45, Gerard Saraber wrote:
>> Greetings everyone,
>> I've been having a bit of a problem since upgrading to the linux 3.x
>> series, I have a machine that we're using as a NAS that runs various
>> rsync processes (mostly at night), lately after a day or two, I will
>> come in in the morning to a load average of 49, but the machine not
>> really doing anything, when trying to run 'dstat' the command just
>> hung with no output at all. there were no errors in the logs, or even
>> anything that would vaguely point at anything I could work with.
>> So needing to get the machine back to work I attempted to reboot it
>> "shutdown -r now" on console... it gives a nice message saying it's
>> going to reboot, but nothing ever happens.. the only way to reboot it
>> is by using ctrl + alt + sysrq + b. after which the machine reboots
>> and the raid array comes back clean.
>>
>> I'm not sure how to troubleshoot this, any pointers would be appreciated.
>>
>> I'm compiling 3.2.4 at the moment and found a bunch of possibly useful
>> options in the kernel debugging section:
>> detect hard/soft lockups and detect hung tasks, maybe it'll give me
>> something more to go on.
>>
>> Some details about the machine:
>> Linux xenbox 3.2.2 #1 SMP Sun Jan 29 10:28:22 CST 2012 x86_64 Intel(R)
>> Xeon(R) CPU 5140 @ 2.33GHz GenuineIntel GNU/Linux
>> It has 3 software raid arrays (2 x 5 drives and 1 x 4 drives) LVM'ed
>> together into a 23TB XFS filesystem.
>> 6GB memory and a pair of Intel Gigabit ethernet controllers bonded together.
> ?Hmm, might be some deadlock in the filesystem. Adding XFS guys to CC.
> Can you run 'echo w >/proc/sysrq-trigger' and post output of dmesg here?
>
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Honza
> --
> Jan Kara <[email protected]>
> SUSE Labs, CR
Thanks for the quick reply,
the machine is running good at the moment so I'm not sure if the
output helps, but here it is:
[I'll also be sure to grab this log the next time it locks]
-Gerard
Feb 07 10:32:04 [kernel] [88365.922121] SysRq : Show Blocked State
Feb 07 10:32:04 [kernel] [88365.922141] task
PC stack pid father
Feb 07 10:32:04 [kernel] [88365.922170] flush-253:0 D
0000000000000000 1320 2941 2 0x00000000
Feb 07 10:32:04 [kernel] [88365.922177] ffff8801ada63570
0000000000000046 ffffffff8189813a ffff8801b554a800
Feb 07 10:32:04 [kernel] [88365.922182] ffff880100000000
00000000001d2880 ffff8801ad8d8000 00000000001d2880
Feb 07 10:32:04 [kernel] [88365.922188] ffff8801ada63fd8
ffff8801ada62000 00000000001d2880 00000000001d2880
Feb 07 10:32:04 [kernel] [88365.922193] Call Trace:
Feb 07 10:32:04 [kernel] [88365.922203] [<ffffffff8189813a>] ?
__schedule+0x70a/0x920
Feb 07 10:32:04 [kernel] [88365.922207] [<ffffffff8189841a>] schedule+0x3a/0x50
Feb 07 10:32:04 [kernel] [88365.922211] [<ffffffff816bc640>]
get_active_stripe+0x2f0/0x660
Feb 07 10:32:04 [kernel] [88365.922217] [<ffffffff810c0000>] ?
sysfs_show_available_clocksources+0x10/0xf0
Feb 07 10:32:04 [kernel] [88365.922222] [<ffffffff81088560>] ?
try_to_wake_up+0x2d0/0x2d0
Feb 07 10:32:04 [kernel] [88365.922226] [<ffffffff816c39b6>]
make_request+0x196/0x430
Feb 07 10:32:04 [kernel] [88365.922231] [<ffffffff810b4e30>] ?
wake_up_bit+0x40/0x40
Feb 07 10:32:04 [kernel] [88365.922236] [<ffffffff816da26b>] ?
dm_request+0x13b/0x230
Feb 07 10:32:04 [kernel] [88365.922239] [<ffffffff816cf1a4>]
md_make_request+0x174/0x2b0
Feb 07 10:32:04 [kernel] [88365.922243] [<ffffffff816cf098>] ?
md_make_request+0x68/0x2b0
Feb 07 10:32:04 [kernel] [88365.922247] [<ffffffff810b9d5e>] ?
up_read+0x1e/0x40
Feb 07 10:32:04 [kernel] [88365.922250] [<ffffffff816da26b>] ?
dm_request+0x13b/0x230
Feb 07 10:32:04 [kernel] [88365.922253] [<ffffffff816da15e>] ?
dm_request+0x2e/0x230
Feb 07 10:32:04 [kernel] [88365.922259] [<ffffffff8145c722>]
generic_make_request+0xc2/0x100
Feb 07 10:32:04 [kernel] [88365.922262] [<ffffffff8145d4d5>]
submit_bio+0x75/0xf0
Feb 07 10:32:04 [kernel] [88365.922267] [<ffffffff811ac88b>] ?
__mark_inode_dirty+0x3b/0x220
Feb 07 10:32:04 [kernel] [88365.922272] [<ffffffff8133f512>]
xfs_submit_ioend_bio.clone.12+0x52/0x80
Feb 07 10:32:04 [kernel] [88365.922275] [<ffffffff8133f62e>]
xfs_submit_ioend+0xee/0x110
Feb 07 10:32:04 [kernel] [88365.922278] [<ffffffff8133f89c>]
xfs_vm_writepage+0x24c/0x520
Feb 07 10:32:04 [kernel] [88365.922284] [<ffffffff81136115>]
__writepage+0x15/0x50
Feb 07 10:32:04 [kernel] [88365.922287] [<ffffffff81136749>]
write_cache_pages+0x219/0x4c0
Feb 07 10:32:04 [kernel] [88365.922291] [<ffffffff81136100>] ?
set_page_dirty+0x70/0x70
Feb 07 10:32:04 [kernel] [88365.922295] [<ffffffff81136a3f>]
generic_writepages+0x4f/0x70
Feb 07 10:32:04 [kernel] [88365.922298] [<ffffffff8133e4e8>]
xfs_vm_writepages+0x58/0x70
Feb 07 10:32:04 [kernel] [88365.922302] [<ffffffff8113822f>]
do_writepages+0x1f/0x40
Feb 07 10:32:04 [kernel] [88365.922306] [<ffffffff811ad5c9>]
writeback_single_inode+0x189/0x400
Feb 07 10:32:04 [kernel] [88365.922309] [<ffffffff811adc28>]
writeback_sb_inodes+0x1a8/0x270
Feb 07 10:32:04 [kernel] [88365.922313] [<ffffffff811add86>]
__writeback_inodes_wb+0x96/0xc0
Feb 07 10:32:04 [kernel] [88365.922316] [<ffffffff811adf93>]
wb_writeback+0x1e3/0x2c0
Feb 07 10:32:04 [kernel] [88365.922319] [<ffffffff811370e5>] ?
determine_dirtyable_memory+0x15/0x30
Feb 07 10:32:04 [kernel] [88365.922323] [<ffffffff81137192>] ?
global_dirty_limits+0x32/0x160
Feb 07 10:32:04 [kernel] [88365.922326] [<ffffffff811aee3e>]
wb_do_writeback+0x11e/0x210
Feb 07 10:32:04 [kernel] [88365.922330] [<ffffffff811aefda>]
bdi_writeback_thread+0xaa/0x290
Feb 07 10:32:04 [kernel] [88365.922333] [<ffffffff811aef30>] ?
wb_do_writeback+0x210/0x210
Feb 07 10:32:04 [kernel] [88365.922337] [<ffffffff810b48c6>] kthread+0xa6/0xb0
Feb 07 10:32:04 [kernel] [88365.922342] [<ffffffff8189e1b4>]
kernel_thread_helper+0x4/0x10
Feb 07 10:32:04 [kernel] [88365.922346] [<ffffffff8189b8dd>] ?
retint_restore_args+0xe/0xe
Feb 07 10:32:04 [kernel] [88365.922350] [<ffffffff810b4820>] ?
__init_kthread_worker+0x70/0x70
Feb 07 10:32:04 [kernel] [88365.922353] [<ffffffff8189e1b0>] ?
gs_change+0xb/0xb
Feb 07 10:32:04 [kernel] [88365.922357] ssh D
0000000000000000 4080 23086 23078 0x00000000
Feb 07 10:32:04 [kernel] [88365.922363] ffff8801492d79a8
0000000000000046 ffffffff8189813a 0000000000000006
Feb 07 10:32:04 [kernel] [88365.922368] 0000000000000000
00000000001d2880 ffff8801b7902680 00000000001d2880
Feb 07 10:32:04 [kernel] [88365.922373] ffff8801492d7fd8
ffff8801492d6000 00000000001d2880 00000000001d2880
Feb 07 10:32:04 [kernel] [88365.922379] Call Trace:
Feb 07 10:32:04 [kernel] [88365.922382] [<ffffffff8189813a>] ?
__schedule+0x70a/0x920
Feb 07 10:32:04 [kernel] [88365.922385] [<ffffffff8189841a>] schedule+0x3a/0x50
Feb 07 10:32:04 [kernel] [88365.922389] [<ffffffff8189891e>]
schedule_timeout+0x18e/0x2e0
Feb 07 10:32:04 [kernel] [88365.922393] [<ffffffff810a0a10>] ?
cascade+0xa0/0xa0
Feb 07 10:32:04 [kernel] [88365.922397] [<ffffffff810be378>] ?
ktime_get_ts+0xa8/0xe0
Feb 07 10:32:04 [kernel] [88365.922401] [<ffffffff8189871a>]
io_schedule_timeout+0x9a/0xf0
Feb 07 10:32:04 [kernel] [88365.922404] [<ffffffff811372ec>] ?
bdi_dirty_limit+0x2c/0xc0
Feb 07 10:32:04 [kernel] [88365.922408] [<ffffffff811379ea>]
balance_dirty_pages_ratelimited_nr+0x2ba/0x770
Feb 07 10:32:04 [kernel] [88365.922417] [<ffffffff81348114>] ?
xfs_iunlock+0x74/0xf0
Feb 07 10:32:04 [kernel] [88365.922422] [<ffffffff8112c349>]
generic_file_buffered_write+0x1b9/0x280
Feb 07 10:32:04 [kernel] [88365.922426] [<ffffffff81345900>]
xfs_file_buffered_aio_write+0xf0/0x1a0
Feb 07 10:32:04 [kernel] [88365.922430] [<ffffffff8114e7ab>] ?
might_fault+0x3b/0x90
Feb 07 10:32:04 [kernel] [88365.922434] [<ffffffff81345b62>]
xfs_file_aio_write+0x1b2/0x2f0
Feb 07 10:32:04 [kernel] [88365.922439] [<ffffffff81198677>] ?
core_sys_select+0x47/0x390
Feb 07 10:32:04 [kernel] [88365.922442] [<ffffffff8114e7ab>] ?
might_fault+0x3b/0x90
Feb 07 10:32:04 [kernel] [88365.922446] [<ffffffff8118432a>]
do_sync_write+0xda/0x120
Feb 07 10:32:04 [kernel] [88365.922452] [<ffffffff8143f167>] ?
security_file_permission+0x27/0xb0
Feb 07 10:32:04 [kernel] [88365.922455] [<ffffffff81184c46>]
vfs_write+0xc6/0x190
Feb 07 10:32:04 [kernel] [88365.922458] [<ffffffff81184f7f>]
sys_write+0x4f/0xa0
Feb 07 10:32:04 [kernel] [88365.922462] [<ffffffff8189bffb>]
system_call_fastpath+0x16/0x1b
Feb 07 10:32:04 [kernel] [88365.922465] ssh D
0000000000000000 4080 23087 23078 0x00000000
Feb 07 10:32:04 [kernel] [88365.922471] ffff88010e2159a8
0000000000000046 ffffffff8189813a 0000000000000006
Feb 07 10:32:04 [kernel] [88365.922476] 0000000000000000
00000000001d2880 ffff8801b32d2680 00000000001d2880
Feb 07 10:32:04 [kernel] [88365.922482] ffff88010e215fd8
ffff88010e214000 00000000001d2880 00000000001d2880
Feb 07 10:32:04 [kernel] [88365.922487] Call Trace:
Feb 07 10:32:04 [kernel] [88365.922490] [<ffffffff8189813a>] ?
__schedule+0x70a/0x920
Feb 07 10:32:04 [kernel] [88365.922493] [<ffffffff8189841a>] schedule+0x3a/0x50
Feb 07 10:32:04 [kernel] [88365.922497] [<ffffffff8189891e>]
schedule_timeout+0x18e/0x2e0
Feb 07 10:32:04 [kernel] [88365.922500] [<ffffffff810a0a10>] ?
cascade+0xa0/0xa0
Feb 07 10:32:04 [kernel] [88365.922504] [<ffffffff810be378>] ?
ktime_get_ts+0xa8/0xe0
Feb 07 10:32:04 [kernel] [88365.922507] [<ffffffff8189871a>]
io_schedule_timeout+0x9a/0xf0
Feb 07 10:32:04 [kernel] [88365.922511] [<ffffffff811372ec>] ?
bdi_dirty_limit+0x2c/0xc0
Feb 07 10:32:04 [kernel] [88365.922515] [<ffffffff811379ea>]
balance_dirty_pages_ratelimited_nr+0x2ba/0x770
Feb 07 10:32:04 [kernel] [88365.922518] [<ffffffff811ac88b>] ?
__mark_inode_dirty+0x3b/0x220
Feb 07 10:32:04 [kernel] [88365.922522] [<ffffffff81348114>] ?
xfs_iunlock+0x74/0xf0
Feb 07 10:32:04 [kernel] [88365.922526] [<ffffffff8112c349>]
generic_file_buffered_write+0x1b9/0x280
Feb 07 10:32:04 [kernel] [88365.922537] [<ffffffff81345b62>]
xfs_file_aio_write+0x1b2/0x2f0
Feb 07 10:32:04 [kernel] [88365.922541] [<ffffffff81198677>] ?
core_sys_select+0x47/0x390
Feb 07 10:32:04 [kernel] [88365.922544] [<ffffffff8114e7ab>] ?
might_fault+0x3b/0x90
Feb 07 10:32:04 [kernel] [88365.922547] [<ffffffff8118432a>]
do_sync_write+0xda/0x120
Feb 07 10:32:04 [kernel] [88365.922551] [<ffffffff8143f167>] ?
security_file_permission+0x27/0xb0
Feb 07 10:32:04 [kernel] [88365.922555] [<ffffffff81184c46>]
vfs_write+0xc6/0x190
Feb 07 10:32:04 [kernel] [88365.922558] [<ffffffff81184f7f>]
sys_write+0x4f/0xa0
Feb 07 10:32:04 [kernel] [88365.922562] [<ffffffff8189bffb>]
system_call_fastpath+0x16/0x1b
Feb 07 10:32:04 [kernel] [88365.922568] Sched Debug Version: v0.10, 3.2.4 #2
Feb 07 10:32:04 [kernel] [88365.922571] ktime
: 88365922.568795
Feb 07 10:32:04 [kernel] [88365.922574] sched_clk
: 88191370.131237
Feb 07 10:32:04 [kernel] [88365.922576] cpu_clk
: 88365922.567625
Feb 07 10:32:04 [kernel] [88365.922578] jiffies
: 4383033218
Feb 07 10:32:04 [kernel] [88365.922580] sched_clock_stable
: 0
Feb 07 10:32:04 [kernel] [88365.922582]
Feb 07 10:32:04 [kernel] [88365.922583] sysctl_sched
Feb 07 10:32:04 [kernel] [88365.922585] .sysctl_sched_latency
: 18.000000
Feb 07 10:32:04 [kernel] [88365.922588]
.sysctl_sched_min_granularity : 2.250000
Feb 07 10:32:04 [kernel] [88365.922590]
.sysctl_sched_wakeup_granularity : 3.000000
Feb 07 10:32:04 [kernel] [88365.922592]
.sysctl_sched_child_runs_first : 0
Feb 07 10:32:04 [kernel] [88365.922594] .sysctl_sched_features
: 24119
Feb 07 10:32:04 [kernel] [88365.922597]
.sysctl_sched_tunable_scaling : 1 (logaritmic)
Feb 07 10:32:04 [kernel] [88365.922600]
Feb 07 10:32:04 [kernel] [88365.922600] cpu#0, 2333.231 MHz
Feb 07 10:32:04 [kernel] [88365.922602] .nr_running : 2
Feb 07 10:32:04 [kernel] [88365.922605] .load : 2048
Feb 07 10:32:04 [kernel] [88365.922607] .nr_switches
: 47384343
Feb 07 10:32:04 [kernel] [88365.922609] .nr_load_updates
: 27680552
Feb 07 10:32:04 [kernel] [88365.922611] .nr_uninterruptible : 5780
Feb 07 10:32:04 [kernel] [88365.922613] .next_balance
: 4383.033233
Feb 07 10:32:04 [kernel] [88365.922615] .curr->pid : 26616
Feb 07 10:32:04 [kernel] [88365.922620] .cpu_load[0] : 2048
Feb 07 10:32:04 [kernel] [88365.922622] .cpu_load[1] : 1024
Feb 07 10:32:04 [kernel] [88365.922623] .cpu_load[2] : 512
Feb 07 10:32:04 [kernel] [88365.922625] .cpu_load[3] : 256
Feb 07 10:32:04 [kernel] [88365.922627] .cpu_load[4] : 128
Feb 07 10:32:04 [kernel] [88365.922629] .yld_count : 0
Feb 07 10:32:04 [kernel] [88365.922631] .sched_switch : 0
Feb 07 10:32:04 [kernel] [88365.922633] .sched_count
: 47523458
Feb 07 10:32:04 [kernel] [88365.922635] .sched_goidle
: 22579080
Feb 07 10:32:04 [kernel] [88365.922637] .avg_idle
: 888834
Feb 07 10:32:04 [kernel] [88365.922639] .ttwu_count
: 25633221
Feb 07 10:32:04 [kernel] [88365.922641] .ttwu_local
: 19078353
Feb 07 10:32:04 [kernel] [88365.922645]
Feb 07 10:32:04 [kernel] [88365.922646] cfs_rq[0]:/autogroup-861
Feb 07 10:32:04 [kernel] [88365.922648] .exec_clock
: 13.544484
Feb 07 10:32:04 [kernel] [88365.922652] .MIN_vruntime
: 0.000001
Feb 07 10:32:04 [kernel] [88365.922654] .min_vruntime
: 6.378969
Feb 07 10:32:04 [kernel] [88365.922656] .max_vruntime
: 0.000001
Feb 07 10:32:04 [kernel] [88365.922658] .spread
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922660] .spread0
: -3347531.515424
Feb 07 10:32:04 [kernel] [88365.922663] .nr_spread_over : 0
Feb 07 10:32:04 [kernel] [88365.922664] .nr_running : 1
Feb 07 10:32:04 [kernel] [88365.922666] .load : 1024
Feb 07 10:32:04 [kernel] [88365.922668] .load_avg
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922671] .load_period
: 9.999999
Feb 07 10:32:04 [kernel] [88365.922673] .load_contrib : 0
Feb 07 10:32:04 [kernel] [88365.922674] .load_tg : 0
Feb 07 10:32:04 [kernel] [88365.922677] .se->exec_start
: 88365922.005056
Feb 07 10:32:04 [kernel] [88365.922679] .se->vruntime
: 3347529.024236
Feb 07 10:32:04 [kernel] [88365.922681] .se->sum_exec_runtime
: 13.544484
Feb 07 10:32:04 [kernel] [88365.922683] .se->statistics.wait_start
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922686] .se->statistics.sleep_start
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922688] .se->statistics.block_start
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922690] .se->statistics.sleep_max
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922692] .se->statistics.block_max
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922694] .se->statistics.exec_max
: 1.397658
Feb 07 10:32:04 [kernel] [88365.922698] .se->statistics.wait_max
: 0.037956
Feb 07 10:32:04 [kernel] [88365.922700] .se->statistics.wait_sum
: 0.405641
Feb 07 10:32:04 [kernel] [88365.922702] .se->statistics.wait_count : 43
Feb 07 10:32:04 [kernel] [88365.922704] .se->load.weight : 1024
Feb 07 10:32:04 [kernel] [88365.922707]
Feb 07 10:32:04 [kernel] [88365.922707] cfs_rq[0]:/
Feb 07 10:32:04 [kernel] [88365.922709] .exec_clock
: 3857099.252074
Feb 07 10:32:04 [kernel] [88365.922712] .MIN_vruntime
: 3347528.894393
Feb 07 10:32:04 [kernel] [88365.922714] .min_vruntime
: 3347537.894393
Feb 07 10:32:04 [kernel] [88365.922716] .max_vruntime
: 3347528.894393
Feb 07 10:32:04 [kernel] [88365.922719] .spread
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922721] .spread0
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922723] .nr_spread_over : 278
Feb 07 10:32:04 [kernel] [88365.922725] .nr_running : 2
Feb 07 10:32:04 [kernel] [88365.922727] .load : 2048
Feb 07 10:32:04 [kernel] [88365.922729] .load_avg
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922731] .load_period
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922733] .load_contrib : 0
Feb 07 10:32:04 [kernel] [88365.922734] .load_tg : 0
Feb 07 10:32:04 [kernel] [88365.922737]
Feb 07 10:32:04 [kernel] [88365.922737] rt_rq[0]:
Feb 07 10:32:04 [kernel] [88365.922739] .rt_nr_running : 0
Feb 07 10:32:04 [kernel] [88365.922741] .rt_throttled : 0
Feb 07 10:32:04 [kernel] [88365.922743] .rt_time
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922745] .rt_runtime
: 950.000000
Feb 07 10:32:04 [kernel] [88365.922747]
Feb 07 10:32:04 [kernel] [88365.922748] runnable tasks:
Feb 07 10:32:04 [kernel] [88365.922749] task PID
tree-key switches prio exec-runtime sum-exec
sum-sleep
Feb 07 10:32:04 [kernel] [88365.922750]
----------------------------------------------------------------------------------------------------------
Feb 07 10:32:04 [kernel] [88365.922759] kworker/0:1 25833
3347528.894393 19293 120 3347528.894393 1314.935421
2056117.209075 /
Feb 07 10:32:04 [kernel] [88365.922767] R bash 26616
5.470536 54 120 5.470536 18.185136
37388.063060 /autogroup-861
Feb 07 10:32:04 [kernel] [88365.922777]
Feb 07 10:32:04 [kernel] [88365.922777] cpu#1, 2333.231 MHz
Feb 07 10:32:04 [kernel] [88365.922779] .nr_running : 0
Feb 07 10:32:04 [kernel] [88365.922781] .load : 0
Feb 07 10:32:04 [kernel] [88365.922783] .nr_switches
: 48208480
Feb 07 10:32:04 [kernel] [88365.922785] .nr_load_updates
: 28422632
Feb 07 10:32:04 [kernel] [88365.922788] .nr_uninterruptible : 6243
Feb 07 10:32:04 [kernel] [88365.922790] .next_balance
: 4383.033438
Feb 07 10:32:04 [kernel] [88365.922792] .curr->pid : 0
Feb 07 10:32:04 [kernel] [88365.922794] .clock
: 88365922.005609
Feb 07 10:32:04 [kernel] [88365.922796] .cpu_load[0] : 0
Feb 07 10:32:04 [kernel] [88365.922798] .cpu_load[1] : 30
Feb 07 10:32:04 [kernel] [88365.922800] .cpu_load[2] : 78
Feb 07 10:32:04 [kernel] [88365.922802] .cpu_load[3] : 74
Feb 07 10:32:04 [kernel] [88365.922803] .cpu_load[4] : 62
Feb 07 10:32:04 [kernel] [88365.922805] .yld_count : 2
Feb 07 10:32:04 [kernel] [88365.922807] .sched_switch : 0
Feb 07 10:32:04 [kernel] [88365.922809] .sched_count
: 48351933
Feb 07 10:32:04 [kernel] [88365.922811] .sched_goidle
: 22985997
Feb 07 10:32:04 [kernel] [88365.922813] .avg_idle
: 988263
Feb 07 10:32:04 [kernel] [88365.922815] .ttwu_count
: 26052282
Feb 07 10:32:04 [kernel] [88365.922817] .ttwu_local
: 19528683
Feb 07 10:32:04 [kernel] [88365.922820]
Feb 07 10:32:04 [kernel] [88365.922821] cfs_rq[1]:/autogroup-776
Feb 07 10:32:04 [kernel] [88365.922823] .exec_clock
: 358297.296203
Feb 07 10:32:04 [kernel] [88365.922826] .MIN_vruntime
: 0.000001
Feb 07 10:32:04 [kernel] [88365.922828] .min_vruntime
: 299834.428916
Feb 07 10:32:04 [kernel] [88365.922831] .max_vruntime
: 0.000001
Feb 07 10:32:04 [kernel] [88365.922833] .spread
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922835] .spread0
: -3047703.465477
Feb 07 10:32:04 [kernel] [88365.922837] .nr_spread_over : 322
Feb 07 10:32:04 [kernel] [88365.922839] .nr_running : 0
Feb 07 10:32:04 [kernel] [88365.922841] .load : 0
Feb 07 10:32:04 [kernel] [88365.922843] .load_avg
: 943.694796
Feb 07 10:32:04 [kernel] [88365.922845] .load_period
: 8.490465
Feb 07 10:32:04 [kernel] [88365.922849] .load_tg : 214
Feb 07 10:32:04 [kernel] [88365.922854] .se->vruntime
: 3442136.706757
Feb 07 10:32:04 [kernel] [88365.922856] .se->sum_exec_runtime
: 358335.275642
Feb 07 10:32:04 [kernel] [88365.922858] .se->statistics.wait_start
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922860] .se->statistics.sleep_start
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922862] .se->statistics.block_start
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922864] .se->statistics.sleep_max
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922866] .se->statistics.block_max
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922869] .se->statistics.exec_max
: 6.979434
Feb 07 10:32:04 [kernel] [88365.922871] .se->statistics.slice_max
: 8.909837
Feb 07 10:32:04 [kernel] [88365.922873] .se->statistics.wait_max
: 335.735845
Feb 07 10:32:04 [kernel] [88365.922875] .se->statistics.wait_sum
: 4606.806398
Feb 07 10:32:04 [kernel] [88365.922877] .se->statistics.wait_count
: 318111
Feb 07 10:32:04 [kernel] [88365.922879] .se->load.weight : 2
Feb 07 10:32:04 [kernel] [88365.922881]
Feb 07 10:32:04 [kernel] [88365.922882] cfs_rq[1]:/
Feb 07 10:32:04 [kernel] [88365.922883] .exec_clock
: 3999784.031074
Feb 07 10:32:04 [kernel] [88365.922886] .MIN_vruntime
: 0.000001
Feb 07 10:32:04 [kernel] [88365.922888] .min_vruntime
: 3442142.654334
Feb 07 10:32:04 [kernel] [88365.922891] .max_vruntime
: 0.000001
Feb 07 10:32:04 [kernel] [88365.922893] .spread
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922895] .spread0
: 94604.759941
Feb 07 10:32:04 [kernel] [88365.922897] .nr_spread_over : 267
Feb 07 10:32:04 [kernel] [88365.922899] .nr_running : 0
Feb 07 10:32:04 [kernel] [88365.922901] .load : 0
Feb 07 10:32:04 [kernel] [88365.922903] .load_avg
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922905] .load_period
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922907] .load_contrib : 0
Feb 07 10:32:04 [kernel] [88365.922909] .load_tg : 0
Feb 07 10:32:04 [kernel] [88365.922911]
Feb 07 10:32:04 [kernel] [88365.922911] rt_rq[1]:
Feb 07 10:32:04 [kernel] [88365.922913] .rt_nr_running : 0
Feb 07 10:32:04 [kernel] [88365.922915] .rt_throttled : 0
Feb 07 10:32:04 [kernel] [88365.922917] .rt_time
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922919] .rt_runtime
: 950.000000
Feb 07 10:32:04 [kernel] [88365.922921]
Feb 07 10:32:04 [kernel] [88365.922922] runnable tasks:
Feb 07 10:32:04 [kernel] [88365.922922] task PID
tree-key switches prio exec-runtime sum-exec
sum-sleep
Feb 07 10:32:04 [kernel] [88365.922924]
----------------------------------------------------------------------------------------------------------
Feb 07 10:32:04 [kernel] [88365.922932]
Feb 07 10:32:04 [kernel] [88365.922932] cpu#2, 2333.231 MHz
Feb 07 10:32:04 [kernel] [88365.922934] .nr_running : 0
Feb 07 10:32:04 [kernel] [88365.922936] .load : 0
Feb 07 10:32:04 [kernel] [88365.922938] .nr_switches
: 46035661
Feb 07 10:32:04 [kernel] [88365.922940] .nr_load_updates
: 27665281
Feb 07 10:32:04 [kernel] [88365.922942] .nr_uninterruptible : -5740
Feb 07 10:32:04 [kernel] [88365.922944] .next_balance
: 4383.033403
Feb 07 10:32:04 [kernel] [88365.922946] .curr->pid : 0
Feb 07 10:32:04 [kernel] [88365.922949] .clock
: 88365922.005806
Feb 07 10:32:04 [kernel] [88365.922951] .cpu_load[0] : 0
Feb 07 10:32:04 [kernel] [88365.922953] .cpu_load[1] : 256
Feb 07 10:32:04 [kernel] [88365.922954] .cpu_load[2] : 197
Feb 07 10:32:04 [kernel] [88365.922956] .cpu_load[3] : 134
Feb 07 10:32:04 [kernel] [88365.922958] .cpu_load[4] : 94
Feb 07 10:32:04 [kernel] [88365.922960] .yld_count : 1
Feb 07 10:32:04 [kernel] [88365.922962] .sched_switch : 0
Feb 07 10:32:04 [kernel] [88365.922964] .sched_count
: 46171251
Feb 07 10:32:04 [kernel] [88365.922966] .sched_goidle
: 22169750
Feb 07 10:32:04 [kernel] [88365.922968] .avg_idle
: 674493
Feb 07 10:32:04 [kernel] [88365.922970] .ttwu_count
: 22246832
Feb 07 10:32:04 [kernel] [88365.922972] .ttwu_local
: 17360455
Feb 07 10:32:04 [kernel] [88365.922975]
Feb 07 10:32:04 [kernel] [88365.922976] cfs_rq[2]:/autogroup-860
Feb 07 10:32:04 [kernel] [88365.922978] .exec_clock
: 11.396092
Feb 07 10:32:04 [kernel] [88365.922981] .MIN_vruntime
: 0.000001
Feb 07 10:32:04 [kernel] [88365.922983] .min_vruntime
: 9.488006
Feb 07 10:32:04 [kernel] [88365.922985] .max_vruntime
: 0.000001
Feb 07 10:32:04 [kernel] [88365.922987] .spread
: 0.000000
Feb 07 10:32:04 [kernel] [88365.922989] .spread0
: -3347528.406387
Feb 07 10:32:04 [kernel] [88365.922991] .nr_spread_over : 0
Feb 07 10:32:04 [kernel] [88365.922993] .nr_running : 0
Feb 07 10:32:04 [kernel] [88365.923002] .load_contrib : 1021
Feb 07 10:32:04 [kernel] [88365.923004] .load_tg : 1021
Feb 07 10:32:04 [kernel] [88365.923004] .se->exec_start
: 88365921.858706
Feb 07 10:32:04 [kernel] [88365.923004] .se->vruntime
: 2777211.488892
Feb 07 10:32:04 [kernel] [88365.923004] .se->sum_exec_runtime
: 11.396092
Feb 07 10:32:04 [kernel] [88365.923004] .se->statistics.wait_start
: 0.000000
Feb 07 10:32:04 [kernel] [88365.923004] .se->statistics.sleep_start
: 0.000000
Feb 07 10:32:04 [kernel] [88365.923004] .se->statistics.block_start
: 0.000000
Feb 07 10:32:04 [kernel] [88365.923004] .se->statistics.sleep_max
: 0.000000
Feb 07 10:32:04 [kernel] [88365.923004] .se->statistics.block_max
: 0.000000
Feb 07 10:32:04 [kernel] [88365.923004] .se->statistics.exec_max
: 0.999893
Feb 07 10:32:04 [kernel] [88365.923004] .se->statistics.slice_max
: 0.000000
Feb 07 10:32:04 [kernel] [88365.923004] .se->statistics.wait_max
: 0.051092
Feb 07 10:32:04 [kernel] [88365.923004] .se->statistics.wait_sum
: 0.064620
Feb 07 10:32:04 [kernel] [88365.923004] .se->statistics.wait_count : 36
Feb 07 10:32:04 [kernel] [88365.923004] .se->load.weight : 2
Feb 07 10:32:04 [kernel] [88365.923004]
Feb 07 10:32:04 [kernel] [88365.923004] cfs_rq[2]:/autogroup-776
Feb 07 10:32:04 [kernel] [88365.923004] .exec_clock
: 342105.615121
Feb 07 10:32:04 [kernel] [88365.923004] .MIN_vruntime
: 0.000001
Feb 07 10:32:04 [kernel] [88365.923004] .min_vruntime
: 289931.479153
Feb 07 10:32:04 [kernel] [88365.923004] .max_vruntime
: 0.000001
Feb 07 10:32:04 [kernel] [88365.923004] .spread
: 0.000000
Feb 07 10:32:04 [kernel] [88365.923004] .spread0
: -3057606.415240
Feb 07 10:32:04 [kernel] [88365.923004] .nr_spread_over : 307
Feb 07 10:32:04 [kernel] [88365.923004] .nr_running : 0
Feb 07 10:32:04 [kernel] [88365.923004] .load : 0
Feb 07 10:32:04 [kernel] [88365.923004] .load_avg
: 525.511404
Feb 07 10:32:04 [kernel] [88365.923004] .load_period
: 5.090393
Feb 07 10:32:04 [kernel] [88365.923004] .load_contrib : 103
Feb 07 10:32:04 [kernel] [88365.923004] .load_tg : 214
Feb 07 10:32:04 [kernel] [88365.923004] .se->exec_start
: 88365920.891042
Feb 07 10:32:04 [kernel] [88365.923004] .se->vruntime
: 2777220.245050
Feb 07 10:32:04 [kernel] [88365.923004] .se->sum_exec_runtime
: 342139.770710
Feb 07 10:32:04 [kernel] [88365.923004] .se->statistics.wait_start
: 0.000000
Feb 07 10:32:04 [kernel] [88365.923004] .se->statistics.sleep_start
: 0.000000
Feb 07 10:32:04 [kernel] [88365.923004] .se->statistics.block_start
: 0.000000
Feb 07 10:32:04 [kernel] [88365.923004] .se->statistics.sleep_max
: 0.000000
Feb 07 10:32:04 [kernel] [88365.923004] .se->statistics.block_max
: 0.000000
Feb 07 10:32:04 [kernel] [88365.923004] .se->statistics.exec_max
: 6.074755
Feb 07 10:32:04 [kernel] [88365.923004] .se->statistics.slice_max
: 15.625920
Feb 07 10:32:04 [kernel] [88365.923004] .se->statistics.wait_max
: 16.153034
Feb 07 10:32:04 [kernel] [88365.923004] .se->statistics.wait_sum
: 4251.654869
Feb 07 10:32:04 [kernel] [88365.923004] .se->statistics.wait_count
: 308775
Feb 07 10:32:04 [kernel] [88365.923004] .se->load.weight : 2
Feb 07 10:32:04 [kernel] [88365.923004]
Feb 07 10:32:04 [kernel] [88365.923004] cfs_rq[2]:/
Feb 07 10:32:04 [kernel] [88365.923004] .exec_clock
: 3362268.845416
Feb 07 10:32:04 [kernel] [88365.923004] .MIN_vruntime
: 0.000001
Feb 07 10:32:04 [kernel] [88365.923004] .min_vruntime
: 2777220.368423
Feb 07 10:32:04 [kernel] [88365.923004] .max_vruntime
: 0.000001
Feb 07 10:32:04 [kernel] [88365.923004] .spread
: 0.000000
Feb 07 10:32:04 [kernel] [88365.923004] .spread0
: -570317.525970
Feb 07 10:32:04 [kernel] [88365.923004] .nr_spread_over : 500
Feb 07 10:32:04 [kernel] [88365.923004] .nr_running : 0
Feb 07 10:32:04 [kernel] [88365.923004] .load : 0
Feb 07 10:32:04 [kernel] [88365.923004] .load_avg
: 0.000000
Feb 07 10:32:04 [kernel] [88365.923004] .load_period
: 0.000000
Feb 07 10:32:04 [kernel] [88365.923004] .load_contrib : 0
Feb 07 10:32:04 [kernel] [88365.923004] .load_tg : 0
Feb 07 10:32:04 [kernel] [88365.923004]
Feb 07 10:32:04 [kernel] [88365.923004] rt_rq[2]:
Feb 07 10:32:04 [kernel] [88365.923004] .rt_nr_running : 0
Feb 07 10:32:04 [kernel] [88365.923004] .rt_throttled : 0
Feb 07 10:32:04 [kernel] [88365.923004] .rt_time
: 0.000000
Feb 07 10:32:04 [kernel] [88365.923004] .rt_runtime
: 950.000000
Feb 07 10:32:04 [kernel] [88365.923004]
Feb 07 10:32:04 [kernel] [88365.923004] runnable tasks:
Feb 07 10:32:04 [kernel] [88365.923004] task PID
tree-key switches prio exec-runtime sum-exec
sum-sleep
Feb 07 10:32:04 [kernel] [88365.923004]
Feb 07 10:32:04 [kernel] [88365.923004] cpu#3, 2333.231 MHz
Feb 07 10:32:04 [kernel] [88365.923004] .nr_running : 0
Feb 07 10:32:04 [kernel] [88365.923004] .load : 0
Feb 07 10:32:04 [kernel] [88365.923004] .nr_switches
: 45074548
Feb 07 10:32:04 [kernel] [88365.923004] .nr_load_updates
: 27257305
Feb 07 10:32:04 [kernel] [88365.923004] .nr_uninterruptible : -6280
Feb 07 10:32:04 [kernel] [88365.923004] .next_balance
: 4383.033407
Feb 07 10:32:04 [kernel] [88365.923004] .curr->pid : 0
Feb 07 10:32:04 [kernel] [88365.923004] .clock
: 88365923.004961
Feb 07 10:32:04 [kernel] [88365.923004] .cpu_load[0] : 0
Feb 07 10:32:04 [kernel] [88365.923004] .cpu_load[1] : 0
Feb 07 10:32:04 [kernel] [88365.923004] .cpu_load[2] : 2
Feb 07 10:32:04 [kernel] [88365.923004] .cpu_load[3] : 31
Feb 07 10:32:04 [kernel] [88365.923004] .cpu_load[4] : 71
Feb 07 10:32:04 [kernel] [88365.923004] .yld_count : 1
Feb 07 10:32:04 [kernel] [88365.923004] .sched_switch : 0
Feb 07 10:32:04 [kernel] [88365.923004] .sched_count
: 45213014
Feb 07 10:32:04 [kernel] [88365.923004] .sched_goidle
: 21712500
Feb 07 10:32:04 [kernel] [88365.923004] .avg_idle
: 1000000
Feb 07 10:32:04 [kernel] [88365.923004] .ttwu_count
: 21742662
Feb 07 10:32:04 [kernel] [88365.923004] .ttwu_local
: 16876989
Feb 07 10:32:04 [kernel] [88365.923004]
Feb 07 10:32:04 [kernel] [88365.923004] cfs_rq[3]:/
Feb 07 10:32:04 [kernel] [88365.923004] .exec_clock
: 3311584.271942
Feb 07 10:32:04 [kernel] [88365.923004] .MIN_vruntime
: 0.000001
Feb 07 10:32:04 [kernel] [88365.923004] .min_vruntime
: 2796366.160205
Feb 07 10:32:04 [kernel] [88365.923004] .max_vruntime
: 0.000001
Feb 07 10:32:04 [kernel] [88365.923004] .spread
: 0.000000
Feb 07 10:32:04 [kernel] [88365.923004] .spread0
: -551171.734188
Feb 07 10:32:04 [kernel] [88365.923004] .nr_spread_over : 1027
Feb 07 10:32:04 [kernel] [88365.923004] .nr_running : 0
Feb 07 10:32:04 [kernel] [88365.923004] .load : 0
Feb 07 10:32:04 [kernel] [88365.923004] .load_avg
: 0.000000
Feb 07 10:32:04 [kernel] [88365.923004] .load_period
: 0.000000
Feb 07 10:32:04 [kernel] [88365.923004] .load_contrib : 0
Feb 07 10:32:04 [kernel] [88365.923004] .load_tg : 0
Feb 07 10:32:04 [kernel] [88365.923004]
Feb 07 10:32:04 [kernel] [88365.923004] rt_rq[3]:
Feb 07 10:32:04 [kernel] [88365.923004] .rt_nr_running : 0
Feb 07 10:32:04 [kernel] [88365.923004] .rt_throttled : 0
Feb 07 10:32:04 [kernel] [88365.923004] .rt_time
: 0.000000
Feb 07 10:32:04 [kernel] [88365.923004] .rt_runtime
: 950.000000
Feb 07 10:32:04 [kernel] [88365.923004]
Feb 07 10:32:04 [kernel] [88365.923004] runnable tasks:
Feb 07 10:32:04 [kernel] [88365.923004] task PID
tree-key switches prio exec-runtime sum-exec
sum-sleep
Feb 07 10:32:04 [kernel] [88365.923004]
----------------------------------------------------------------------------------------------------------
Feb 07 10:32:04 [kernel] [88365.923004]
On Tue 07-02-12 10:35:37, Gerard Saraber wrote:
> On Mon, Feb 6, 2012 at 4:51 PM, Jan Kara <[email protected]> wrote:
> > On Mon 06-02-12 09:40:45, Gerard Saraber wrote:
> >> Greetings everyone,
> >> I've been having a bit of a problem since upgrading to the linux 3.x
> >> series, I have a machine that we're using as a NAS that runs various
> >> rsync processes (mostly at night), lately after a day or two, I will
> >> come in in the morning to a load average of 49, but the machine not
> >> really doing anything, when trying to run 'dstat' the command just
> >> hung with no output at all. there were no errors in the logs, or even
> >> anything that would vaguely point at anything I could work with.
> >> So needing to get the machine back to work I attempted to reboot it
> >> "shutdown -r now" on console... it gives a nice message saying it's
> >> going to reboot, but nothing ever happens.. the only way to reboot it
> >> is by using ctrl + alt + sysrq + b. after which the machine reboots
> >> and the raid array comes back clean.
> >>
> >> I'm not sure how to troubleshoot this, any pointers would be appreciated.
> >>
> >> I'm compiling 3.2.4 at the moment and found a bunch of possibly useful
> >> options in the kernel debugging section:
> >> detect hard/soft lockups and detect hung tasks, maybe it'll give me
> >> something more to go on.
> >>
> >> Some details about the machine:
> >> Linux xenbox 3.2.2 #1 SMP Sun Jan 29 10:28:22 CST 2012 x86_64 Intel(R)
> >> Xeon(R) CPU 5140 @ 2.33GHz GenuineIntel GNU/Linux
> >> It has 3 software raid arrays (2 x 5 drives and 1 x 4 drives) LVM'ed
> >> together into a 23TB XFS filesystem.
> >> 6GB memory and a pair of Intel Gigabit ethernet controllers bonded together.
> > ?Hmm, might be some deadlock in the filesystem. Adding XFS guys to CC.
> > Can you run 'echo w >/proc/sysrq-trigger' and post output of dmesg here?
> >
> > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Honza
> > --
> > Jan Kara <[email protected]>
> > SUSE Labs, CR
>
> Thanks for the quick reply,
> the machine is running good at the moment so I'm not sure if the
> output helps, but here it is:
> [I'll also be sure to grab this log the next time it locks]
Yeah. Sorry, I was not clear but I meant you should grab the traces when
the machine locks up again...
Honza
--
Jan Kara <[email protected]>
SUSE Labs, CR
Hello,
On Tue 07-02-12 22:57:37, Wolfram Gloger wrote:
> I have _exactly_ the same problem as Gerard Saraber on LKML.
>
> This has occurred twice after upgrading to 3.2.x and the second time
> _right now_. I seem to be able to reproduce this by building
> iceweasel_10.0-2 from source.
>
> Appended is the output of 'echo w >/proc/sysrq-trigger'. I'm
> currently trying to reboot it with an older kernel, therefore the
> 'grub-probe's, but it does not reboot (yet). A sync is extremely slow
> but does complete eventually.
Hmm, then this isn't really a deadlock (at least in your case) but only a
rather unresponsive system. Looking at the backtraces you provided I don't
see anything really suspicious. Only maybe fsnotify_mark process - do you
use fsnotify?
> I use no xfs (only ext3) so filesystem issue is improbable.
> I do however use mdraid (0 and 1).
Thanks for report.
Honza
> SysRq : Show Blocked State
> task PC stack pid father
> fsnotify_mark D ffff88021fc90600 0 437 2 0x00000000
> ffff880216533000 0000000000000046 ffff8802143dc010 0000000000004000
> ffff880216533000 0000000000010600 ffff8802143ddfd8 ffff8802143ddfd8
> 0000000000010600 ffff880217404560 0000000000000400 0000000000001000
> Call Trace:
> [<ffffffff812a3427>] ? schedule_timeout+0x1f/0x193
> [<ffffffff812a30a7>] ? __schedule+0x7aa/0x83a
> [<ffffffff812a2724>] ? wait_for_common+0xcc/0x142
> [<ffffffff8102fbe7>] ? try_to_wake_up+0x1ee/0x1ee
> [<ffffffff8106f939>] ? call_rcu_bh+0xc/0xc
> [<ffffffff8106fd88>] ? synchronize_rcu_bh+0x1f/0x1f
> [<ffffffff810468f8>] ? wait_rcu_gp+0x38/0x3f
> [<ffffffff810468ff>] ? wait_rcu_gp+0x3f/0x3f
> [<ffffffff8102fbd7>] ? try_to_wake_up+0x1de/0x1ee
> [<ffffffff8104c26e>] ? __synchronize_srcu+0x32/0x109
> [<ffffffff810e094f>] ? fsnotify_mark_destroy+0x87/0x143
> [<ffffffff81048ff9>] ? wake_up_bit+0x22/0x22
> [<ffffffff810e08c8>] ? fsnotify_duplicate_mark+0x37/0x37
> [<ffffffff81048c31>] ? kthread+0x7e/0x86
> [<ffffffff812a6e74>] ? kernel_thread_helper+0x4/0x10
> [<ffffffff81048bb3>] ? kthread_worker_fn+0x141/0x141
> [<ffffffff812a6e70>] ? gs_change+0xb/0xb
> Xorg D ffff88021fc50600 0 3939 3914 0x00400004
> ffff8802142ea400 0000000000000082 ffff880214034010 0000000000004000
> ffff8802142ea400 0000000000010600 ffff880214035fd8 ffff880214035fd8
> 0000000000010600 0000000000000028 ffff880213a37f00 ffffffff8107e7f7
> Call Trace:
> [<ffffffff8107e7f7>] ? filemap_fault+0x62/0x319
> [<ffffffff81093816>] ? __do_fault+0x3f6/0x435
> [<ffffffff812a3427>] ? schedule_timeout+0x1f/0x193
> [<ffffffff812a2724>] ? wait_for_common+0xcc/0x142
> [<ffffffff8102fbe7>] ? try_to_wake_up+0x1ee/0x1ee
> [<ffffffff8106f939>] ? call_rcu_bh+0xc/0xc
> [<ffffffff810468f8>] ? wait_rcu_gp+0x38/0x3f
> [<ffffffff810468ff>] ? wait_rcu_gp+0x3f/0x3f
> [<ffffffff811f5dc8>] ? mousedev_release+0x40/0x63
> [<ffffffff810b508d>] ? fput+0xf0/0x1b5
> [<ffffffff810b23d9>] ? filp_close+0x61/0x68
> [<ffffffff810b2473>] ? sys_close+0x93/0xcb
> [<ffffffff812a507b>] ? system_call_fastpath+0x16/0x1b
> grub-probe D ffff88021fc50600 0 1065 1 0x00000004
> ffff8800ac1c1e00 0000000000000086 ffff880083142010 0000000000004000
> ffff8800ac1c1e00 0000000000010600 ffff880083143fd8 ffff880083143fd8
> 0000000000010600 0000000000000000 ffff88021fffcc08 0000000000000000
> Call Trace:
> [<ffffffff8107de40>] ? find_get_pages_tag+0x46/0xe1
> [<ffffffff810866ba>] ? pagevec_lookup_tag+0x1a/0x21
> [<ffffffff8102a093>] ? select_task_rq_fair+0x608/0x934
> [<ffffffff81084747>] ? bdi_position_ratio+0x12b/0x12b
> [<ffffffff812a3427>] ? schedule_timeout+0x1f/0x193
> [<ffffffff81028a9b>] ? check_preempt_wakeup+0x10c/0x15c
> [<ffffffff812a2724>] ? wait_for_common+0xcc/0x142
> [<ffffffff8102fbe7>] ? try_to_wake_up+0x1ee/0x1ee
> [<ffffffff81087140>] ? lru_add_drain+0xb8/0xb8
> [<ffffffff81044f5c>] ? flush_work+0x22/0x2c
> [<ffffffff8104347f>] ? do_work_for_cpu+0x1b/0x1b
> [<ffffffff8104566e>] ? schedule_on_each_cpu+0xa6/0xdb
> [<ffffffff810d64ae>] ? invalidate_bdev+0x28/0x39
> [<ffffffff8113fdd9>] ? blkdev_ioctl+0x1c2/0x7c5
> [<ffffffff810c0391>] ? path_openat+0x342/0x362
> [<ffffffff810c0484>] ? do_filp_open+0x2c/0x74
> [<ffffffff810d9e2e>] ? block_ioctl+0x38/0x3c
> [<ffffffff810c160b>] ? do_vfs_ioctl+0x3f1/0x432
> [<ffffffff810b22ef>] ? fd_install+0x2e/0x59
> [<ffffffff810ae1e0>] ? kmem_cache_free+0x14/0x8f
> [<ffffffff810c169d>] ? sys_ioctl+0x51/0x70
> [<ffffffff812a507b>] ? system_call_fastpath+0x16/0x1b
> grub-probe D ffff88021fc90600 0 1137 1 0x00000004
> ffff8800ac1ea400 0000000000000082 ffff8800831f4010 0000000000004000
> ffff8800ac1ea400 0000000000010600 ffff8800831f5fd8 ffff8800831f5fd8
> 0000000000010600 ffff88001703e538 00000000ffffffff 0000000000000000
> Call Trace:
> [<ffffffff8107de40>] ? find_get_pages_tag+0x46/0xe1
> [<ffffffff810866ba>] ? pagevec_lookup_tag+0x1a/0x21
> [<ffffffff8102a093>] ? select_task_rq_fair+0x608/0x934
> [<ffffffff81084747>] ? bdi_position_ratio+0x12b/0x12b
> [<ffffffff812a3427>] ? schedule_timeout+0x1f/0x193
> [<ffffffff81028a9b>] ? check_preempt_wakeup+0x10c/0x15c
> [<ffffffff812a2724>] ? wait_for_common+0xcc/0x142
> [<ffffffff8102fbe7>] ? try_to_wake_up+0x1ee/0x1ee
> [<ffffffff81087140>] ? lru_add_drain+0xb8/0xb8
> [<ffffffff81044f5c>] ? flush_work+0x22/0x2c
> [<ffffffff8104347f>] ? do_work_for_cpu+0x1b/0x1b
> [<ffffffff8104566e>] ? schedule_on_each_cpu+0xa6/0xdb
> [<ffffffff810d64ae>] ? invalidate_bdev+0x28/0x39
> [<ffffffff8113fdd9>] ? blkdev_ioctl+0x1c2/0x7c5
> [<ffffffff810c0391>] ? path_openat+0x342/0x362
> [<ffffffff810c0484>] ? do_filp_open+0x2c/0x74
> [<ffffffff810d9e2e>] ? block_ioctl+0x38/0x3c
> [<ffffffff810c160b>] ? do_vfs_ioctl+0x3f1/0x432
> [<ffffffff810b22ef>] ? fd_install+0x2e/0x59
> [<ffffffff810ae1e0>] ? kmem_cache_free+0x14/0x8f
> [<ffffffff810c169d>] ? sys_ioctl+0x51/0x70
> [<ffffffff812a507b>] ? system_call_fastpath+0x16/0x1b
> grub-probe D ffff88021fc90600 0 1285 1 0x00000004
> ffff8801b7a46c00 0000000000000082 ffff8801b93e4010 0000000000004000
> ffff8801b7a46c00 0000000000010600 ffff8801b93e5fd8 ffff8801b93e5fd8
> 0000000000010600 0000000000000246 ffff8801b93e4000 0000000000000000
> Call Trace:
> [<ffffffff8107de40>] ? find_get_pages_tag+0x46/0xe1
> [<ffffffff810866ba>] ? pagevec_lookup_tag+0x1a/0x21
> [<ffffffff8102a093>] ? select_task_rq_fair+0x608/0x934
> [<ffffffff81084747>] ? bdi_position_ratio+0x12b/0x12b
> [<ffffffff812a3427>] ? schedule_timeout+0x1f/0x193
> [<ffffffff81028a9b>] ? check_preempt_wakeup+0x10c/0x15c
> [<ffffffff812a2724>] ? wait_for_common+0xcc/0x142
> [<ffffffff8102fbe7>] ? try_to_wake_up+0x1ee/0x1ee
> [<ffffffff81087140>] ? lru_add_drain+0xb8/0xb8
> [<ffffffff81044f5c>] ? flush_work+0x22/0x2c
> [<ffffffff8104347f>] ? do_work_for_cpu+0x1b/0x1b
> [<ffffffff8104566e>] ? schedule_on_each_cpu+0xa6/0xdb
> [<ffffffff810d64ae>] ? invalidate_bdev+0x28/0x39
> [<ffffffff8113fdd9>] ? blkdev_ioctl+0x1c2/0x7c5
> [<ffffffff810c0391>] ? path_openat+0x342/0x362
> [<ffffffff810c0484>] ? do_filp_open+0x2c/0x74
> [<ffffffff810d9e2e>] ? block_ioctl+0x38/0x3c
> [<ffffffff810c160b>] ? do_vfs_ioctl+0x3f1/0x432
> [<ffffffff810b22ef>] ? fd_install+0x2e/0x59
> [<ffffffff810ae1e0>] ? kmem_cache_free+0x14/0x8f
> [<ffffffff810c169d>] ? sys_ioctl+0x51/0x70
> [<ffffffff812a507b>] ? system_call_fastpath+0x16/0x1b
> killall5 D ffff88021fc10600 0 1831 1827 0x00000000
> ffff880100278000 0000000000000082 ffff88020edfc010 0000000000004000
> ffff880100278000 0000000000010600 ffff88020edfdfd8 ffff88020edfdfd8
> 0000000000010600 00000000000000cc ffff88020edfdf58 ffffffff00000000
> Call Trace:
> [<ffffffff8102a093>] ? select_task_rq_fair+0x608/0x934
> [<ffffffff812a3427>] ? schedule_timeout+0x1f/0x193
> [<ffffffff81028a9b>] ? check_preempt_wakeup+0x10c/0x15c
> [<ffffffff812a2724>] ? wait_for_common+0xcc/0x142
> [<ffffffff8102fbe7>] ? try_to_wake_up+0x1ee/0x1ee
> [<ffffffff81087140>] ? lru_add_drain+0xb8/0xb8
> [<ffffffff81044f5c>] ? flush_work+0x22/0x2c
> [<ffffffff8104347f>] ? do_work_for_cpu+0x1b/0x1b
> [<ffffffff8104566e>] ? schedule_on_each_cpu+0xa6/0xdb
> [<ffffffff810986a5>] ? sys_mlockall+0x38/0x111
> [<ffffffff812a507b>] ? system_call_fastpath+0x16/0x1b
--
Jan Kara <[email protected]>
SUSE Labs, CR
Thanks for your response.
> From: Jan Kara <[email protected]>
> Hmm, then this isn't really a deadlock (at least in your case) but only a
> rather unresponsive system.
Well, like the subject says this is a "soft lockup"..
When it happened the first time the system actually recovered
after some hours.
But like Gerard said, there is no apparent ground for this
unresponsiveness, there are tons of memory free and no swapping.
> Looking at the backtraces you provided I don't
> see anything really suspicious. Only maybe fsnotify_mark process - do you
> use fsnotify?
I have configured CONFIG_FSNOTIFY=y indeed but don't think I'm using it
from userspace..
Regards,
Wolfram.
On Wed 08-02-12 00:16:15, Wolfram Gloger wrote:
> Thanks for your response.
>
> > From: Jan Kara <[email protected]>
>
> > Hmm, then this isn't really a deadlock (at least in your case) but only a
> > rather unresponsive system.
>
> Well, like the subject says this is a "soft lockup"..
We differ in terminology a bit but that's not really important when I
know what you mean :)
> When it happened the first time the system actually recovered
> after some hours.
> But like Gerard said, there is no apparent ground for this
> unresponsiveness, there are tons of memory free and no swapping.
Yeah, interesting. There are no processes waiting for IO either so I'm
puzzled what causes you problems.
> > Looking at the backtraces you provided I don't
> > see anything really suspicious. Only maybe fsnotify_mark process - do you
> > use fsnotify?
>
> I have configured CONFIG_FSNOTIFY=y indeed but don't think I'm using it
> from userspace..
Well, apparently something uses it because fsnotify_mark thread is just
in the process of destroying some mark. Maybe you could run with
CONFIG_FSNOTIFY=n just to rule out some problems there...
Honza
--
Jan Kara <[email protected]>
SUSE Labs, CR
Hi everyone,
It finally did it again, it took longer then I expected, it also
locked itself up so bad that I couldn't get into it to hit
ctrl+alt+sysrq+w ..
I had turned on the debugging feature that automatically logs the hung
tasks, and I've attached the log below, I hope it's helpful.
I was running 3.2.4 from kernel.org on a 4 core Xeon machine:
model name : Intel(R) Xeon(R) CPU 5140 @ 2.33GHz
6GB Ram
2x Intel 80003ES2LAN Gigabit Ethernet Controllers bonded together
2 LSI SAS controllers:
08:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E
PCI-Express Fusion-MPT SAS (rev 08)
0a:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS
2008 [Falcon] (rev 03)
16 drives in a mix of 2 and 3TB, in 3 raid5 arrays and combined
together with LVM
/dev/mapper/pool-main 23T 12T 11T 52%
for a 23TB volume formatted with XFS.
The root partition is ext4 on an older SATA drive, the reason I bring
this up is that when I hit (on a whim) ctrl+sysrq+J that is supposed
to unfreeze frozen filesystems, the console started dumping lots of
messages about attempting to unfreeze /dev/sda3 [my root partition] so
maybe there's a problem with my sda drive.
But I get no i/o or other errors in my logs at all. I monitor all
drives with smartd to head off any drive failures before they happen
and it seems to think sda is fine.
Hopefully my attached log helps.
I appreciate any input, also please call me an idiot if I'm missing
something obvious.
-Gerard Saraber
On Tue, Feb 7, 2012 at 10:54 AM, Jan Kara <[email protected]> wrote:
> On Tue 07-02-12 10:35:37, Gerard Saraber wrote:
>> On Mon, Feb 6, 2012 at 4:51 PM, Jan Kara <[email protected]> wrote:
>> > On Mon 06-02-12 09:40:45, Gerard Saraber wrote:
>> >> Greetings everyone,
>> >> I've been having a bit of a problem since upgrading to the linux 3.x
>> >> series, I have a machine that we're using as a NAS that runs various
>> >> rsync processes (mostly at night), lately after a day or two, I will
>> >> come in in the morning to a load average of 49, but the machine not
>> >> really doing anything, when trying to run 'dstat' the command just
>> >> hung with no output at all. there were no errors in the logs, or even
>> >> anything that would vaguely point at anything I could work with.
>> >> So needing to get the machine back to work I attempted to reboot it
>> >> "shutdown -r now" on console... it gives a nice message saying it's
>> >> going to reboot, but nothing ever happens.. the only way to reboot it
>> >> is by using ctrl + alt + sysrq + b. after which the machine reboots
>> >> and the raid array comes back clean.
>> >>
>> >> I'm not sure how to troubleshoot this, any pointers would be appreciated.
>> >>
>> >> I'm compiling 3.2.4 at the moment and found a bunch of possibly useful
>> >> options in the kernel debugging section:
>> >> detect hard/soft lockups and detect hung tasks, maybe it'll give me
>> >> something more to go on.
>> >>
>> >> Some details about the machine:
>> >> Linux xenbox 3.2.2 #1 SMP Sun Jan 29 10:28:22 CST 2012 x86_64 Intel(R)
>> >> Xeon(R) CPU 5140 @ 2.33GHz GenuineIntel GNU/Linux
>> >> It has 3 software raid arrays (2 x 5 drives and 1 x 4 drives) LVM'ed
>> >> together into a 23TB XFS filesystem.
>> >> 6GB memory and a pair of Intel Gigabit ethernet controllers bonded together.
>> > ?Hmm, might be some deadlock in the filesystem. Adding XFS guys to CC.
>> > Can you run 'echo w >/proc/sysrq-trigger' and post output of dmesg here?
>> >
>> > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Honza
>> > --
>> > Jan Kara <[email protected]>
>> > SUSE Labs, CR
>>
>> Thanks for the quick reply,
>> the machine is running good at the moment so I'm not sure if the
>> output helps, but here it is:
>> [I'll also be sure to grab this log the next time it locks]
> ?Yeah. Sorry, I was not clear but I meant you should grab the traces when
> the machine locks up again...
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Honza
>
> --
> Jan Kara <[email protected]>
> SUSE Labs, CR
Hi,
On Wed 29-02-12 12:20:10, Gerard Saraber wrote:
> I have another update, after rebooting two days ago,
> it's now running :
> Linux xenbox 3.2.5 #1 SMP Tue Feb 7 13:51:06 CST 2012 x86_64 Intel(R)
> Xeon(R) CPU 5140 @ 2.33GHz GenuineIntel GNU/Linux
>
> I noticed high load on the machine, and another dump in the logs about
> stuck processes, so I ran ctrl+alt+sysrq + w and got this log
> [attached below]
>
> I hope finally getting the right log will be helpful!
Thanks for the traces. I was looking at them and I don't see the
immediate problem. There is a task (kswapd0) reclaiming XFS inodes and
waiting for IO to complete. Quite some other tasks are waiting in
xfs_reclaim_inodes_ag() for reclaim mutex. Since some of them are called
from NFS code (e.g. nfs4_process_open1()), that seems to be a reason why
other tasks are blocked in nfs4_lock_state().
So all in all it seems like some problem in how mm does reclaim or we are
inefficient doing inode reclaim in XFS or for some reason the IO is too
slow. As a result we are low on memory and things go bad. Do you know with
which kernel version the problem started to appear?
Honza
> On Mon, Feb 27, 2012 at 8:38 AM, Gerard Saraber <[email protected]> wrote:
> > Hi everyone,
> > It finally did it again, it took longer then I expected, it also
> > locked itself up so bad that I couldn't get into it to hit
> > ctrl+alt+sysrq+w ..
> > I had turned on the debugging feature that automatically logs the hung
> > tasks, and I've attached the log below, I hope it's helpful.
> >
> > I was running 3.2.4 from kernel.org on a 4 core Xeon machine:
> > model name ? ? ?: Intel(R) Xeon(R) CPU ? ? ? ? ? ?5140 ?@ 2.33GHz
> >
> > 6GB Ram
> >
> > 2x Intel 80003ES2LAN Gigabit Ethernet Controllers bonded together
> > 2 LSI SAS controllers:
> > 08:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E
> > PCI-Express Fusion-MPT SAS (rev 08)
> > 0a:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS
> > 2008 [Falcon] (rev 03)
> >
> > 16 ?drives in a mix of 2 and 3TB, in 3 raid5 arrays and combined
> > together with LVM
> > /dev/mapper/pool-main ? 23T ? 12T ? 11T ?52%
> > for a 23TB volume formatted with XFS.
> >
> > The root partition is ext4 on an older SATA drive, the reason I bring
> > this up is that when I hit (on a whim) ctrl+sysrq+J that is supposed
> > to unfreeze frozen filesystems, the console started dumping lots of
> > messages about attempting to unfreeze /dev/sda3 [my root partition] so
> > maybe there's a problem with my sda drive.
> > But I get no i/o or other errors in my logs at all. I monitor all
> > drives with smartd to head off any drive failures before they happen
> > and it seems to think sda is fine.
> >
> > Hopefully my attached log helps.
> > I appreciate any input, also please call me an idiot if I'm missing
> > something obvious.
> >
> > -Gerard Saraber
> >
> >
> > On Tue, Feb 7, 2012 at 10:54 AM, Jan Kara <[email protected]> wrote:
> >> On Tue 07-02-12 10:35:37, Gerard Saraber wrote:
> >>> On Mon, Feb 6, 2012 at 4:51 PM, Jan Kara <[email protected]> wrote:
> >>> > On Mon 06-02-12 09:40:45, Gerard Saraber wrote:
> >>> >> Greetings everyone,
> >>> >> I've been having a bit of a problem since upgrading to the linux 3.x
> >>> >> series, I have a machine that we're using as a NAS that runs various
> >>> >> rsync processes (mostly at night), lately after a day or two, I will
> >>> >> come in in the morning to a load average of 49, but the machine not
> >>> >> really doing anything, when trying to run 'dstat' the command just
> >>> >> hung with no output at all. there were no errors in the logs, or even
> >>> >> anything that would vaguely point at anything I could work with.
> >>> >> So needing to get the machine back to work I attempted to reboot it
> >>> >> "shutdown -r now" on console... it gives a nice message saying it's
> >>> >> going to reboot, but nothing ever happens.. the only way to reboot it
> >>> >> is by using ctrl + alt + sysrq + b. after which the machine reboots
> >>> >> and the raid array comes back clean.
> >>> >>
> >>> >> I'm not sure how to troubleshoot this, any pointers would be appreciated.
> >>> >>
> >>> >> I'm compiling 3.2.4 at the moment and found a bunch of possibly useful
> >>> >> options in the kernel debugging section:
> >>> >> detect hard/soft lockups and detect hung tasks, maybe it'll give me
> >>> >> something more to go on.
> >>> >>
> >>> >> Some details about the machine:
> >>> >> Linux xenbox 3.2.2 #1 SMP Sun Jan 29 10:28:22 CST 2012 x86_64 Intel(R)
> >>> >> Xeon(R) CPU 5140 @ 2.33GHz GenuineIntel GNU/Linux
> >>> >> It has 3 software raid arrays (2 x 5 drives and 1 x 4 drives) LVM'ed
> >>> >> together into a 23TB XFS filesystem.
> >>> >> 6GB memory and a pair of Intel Gigabit ethernet controllers bonded together.
> >>> > ?Hmm, might be some deadlock in the filesystem. Adding XFS guys to CC.
> >>> > Can you run 'echo w >/proc/sysrq-trigger' and post output of dmesg here?
> >>> >
> >>> > ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Honza
> >>> > --
> >>> > Jan Kara <[email protected]>
> >>> > SUSE Labs, CR
> >>>
> >>> Thanks for the quick reply,
> >>> the machine is running good at the moment so I'm not sure if the
> >>> output helps, but here it is:
> >>> [I'll also be sure to grab this log the next time it locks]
> >> ?Yeah. Sorry, I was not clear but I meant you should grab the traces when
> >> the machine locks up again...
> >> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?Honza
> >>
> >> --
> >> Jan Kara <[email protected]>
> >> SUSE Labs, CR
> Feb 29 12:05:38 [kernel] [186732.828746] SysRq : Show Blocked State
> Feb 29 12:05:38 [kernel] [186732.828829] task PC stack pid father
> Feb 29 12:05:38 [kernel] [186732.828833] kthreadd D 0000000000000000 2560 2 0 0x00000000
> Feb 29 12:05:38 [kernel] [186732.828839] ffff8801b78836c0 0000000000000046 ffffffff8189813a 0000000000000001
> Feb 29 12:05:38 [kernel] [186732.828845] ffff880100000000 00000000001d2880 ffff8801b7879340 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.828850] ffff8801b7883fd8 ffff8801b7882000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.828856] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.828865] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.828869] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.828873] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.828876] [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.828881] [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.828885] [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.828888] [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.828894] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.828900] [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.828904] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.828907] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.828911] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.828916] [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.828920] [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.828924] [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.828928] [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.828933] [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.828937] [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.828941] [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.828945] [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.828949] [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.828959] [<ffffffff810915e9>] do_fork+0x59/0x350
> Feb 29 12:05:38 [kernel] [186732.828963] [<ffffffff810ce685>] ? trace_hardirqs_on_caller+0x105/0x190
> Feb 29 12:05:38 [kernel] [186732.828967] [<ffffffff810ce71d>] ? trace_hardirqs_on+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.828971] [<ffffffff8107d350>] ? finish_task_switch+0x80/0x110
> Feb 29 12:05:38 [kernel] [186732.828974] [<ffffffff810bb325>] ? sched_clock_local+0x25/0x90
> Feb 29 12:05:38 [kernel] [186732.828979] [<ffffffff8104a54c>] kernel_thread+0x6c/0x70
> Feb 29 12:05:38 [kernel] [186732.828984] [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.828988] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.828991] [<ffffffff810b4cc2>] kthreadd+0x122/0x160
> Feb 29 12:05:38 [kernel] [186732.828995] [<ffffffff810ce685>] ? trace_hardirqs_on_caller+0x105/0x190
> Feb 29 12:05:38 [kernel] [186732.828998] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829002] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829006] [<ffffffff810b4ba0>] ? tsk_fork_get_node+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829009] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829015] kswapd0 D 0000000000000000 2248 590 2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829020] ffff8801b70872c0 0000000000000046 ffffffff8189813a 0000000000000006
> Feb 29 12:05:38 [kernel] [186732.829025] 0000000000000000 00000000001d2880 ffff8801b7a1b9c0 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829030] ffff8801b7087fd8 ffff8801b7086000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829035] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829039] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829042] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829048] [<ffffffff816cb13d>] md_flush_request+0x7d/0x140
> Feb 29 12:05:38 [kernel] [186732.829052] [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829056] [<ffffffff816c3c28>] make_request+0x428/0x430
> Feb 29 12:05:38 [kernel] [186732.829059] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829063] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829066] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829070] [<ffffffff816cf14f>] ? md_make_request+0x13f/0x2b0
> Feb 29 12:05:38 [kernel] [186732.829074] [<ffffffff816da24b>] ? dm_request+0x13b/0x230
> Feb 29 12:05:38 [kernel] [186732.829080] [<ffffffff816cf078>] ? md_make_request+0x68/0x2b0
> Feb 29 12:05:38 [kernel] [186732.829084] [<ffffffff810b9d5e>] ? up_read+0x1e/0x40
> Feb 29 12:05:38 [kernel] [186732.829087] [<ffffffff816da24b>] ? dm_request+0x13b/0x230
> Feb 29 12:05:38 [kernel] [186732.829091] [<ffffffff816da13e>] ? dm_request+0x2e/0x230
> Feb 29 12:05:38 [kernel] [186732.829095] [<ffffffff8145c722>] generic_make_request+0xc2/0x100
> Feb 29 12:05:38 [kernel] [186732.829099] [<ffffffff8145d4d5>] submit_bio+0x75/0xf0
> Feb 29 12:05:38 [kernel] [186732.829104] [<ffffffff81341a82>] _xfs_buf_ioapply+0x162/0x1d0
> Feb 29 12:05:38 [kernel] [186732.829108] [<ffffffff81394111>] ? xlog_bdstrat+0x21/0x60
> Feb 29 12:05:38 [kernel] [186732.829111] [<ffffffff81341bca>] xfs_buf_iorequest+0x4a/0xa0
> Feb 29 12:05:38 [kernel] [186732.829115] [<ffffffff81394111>] xlog_bdstrat+0x21/0x60
> Feb 29 12:05:38 [kernel] [186732.829118] [<ffffffff81395d9e>] xlog_sync+0x23e/0x3d0
> Feb 29 12:05:38 [kernel] [186732.829122] [<ffffffff81396793>] xlog_state_release_iclog+0xe3/0xf0
> Feb 29 12:05:38 [kernel] [186732.829125] [<ffffffff81397258>] xlog_write+0x618/0x780
> Feb 29 12:05:38 [kernel] [186732.829129] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829132] [<ffffffff81398830>] ? xlog_cil_push+0x1c0/0x3b0
> Feb 29 12:05:38 [kernel] [186732.829136] [<ffffffff813988a9>] xlog_cil_push+0x239/0x3b0
> Feb 29 12:05:38 [kernel] [186732.829139] [<ffffffff81399184>] xlog_cil_force_lsn+0x114/0x120
> Feb 29 12:05:38 [kernel] [186732.829143] [<ffffffff81397871>] _xfs_log_force_lsn+0x51/0x2f0
> Feb 29 12:05:38 [kernel] [186732.829147] [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829151] [<ffffffff81386675>] ? xfs_iunpin_wait+0x45/0xd0
> Feb 29 12:05:38 [kernel] [186732.829155] [<ffffffff81397b23>] xfs_log_force_lsn+0x13/0x40
> Feb 29 12:05:38 [kernel] [186732.829158] [<ffffffff8189b68b>] ? _raw_spin_unlock_irqrestore+0x3b/0x70
> Feb 29 12:05:38 [kernel] [186732.829162] [<ffffffff813846f5>] xfs_iunpin_nowait+0x35/0x70
> Feb 29 12:05:38 [kernel] [186732.829165] [<ffffffff81386675>] xfs_iunpin_wait+0x45/0xd0
> Feb 29 12:05:38 [kernel] [186732.829170] [<ffffffff8107cab6>] ? try_wait_for_completion+0x46/0x60
> Feb 29 12:05:38 [kernel] [186732.829173] [<ffffffff81352274>] xfs_reclaim_inode+0xb4/0x2c0
> Feb 29 12:05:38 [kernel] [186732.829177] [<ffffffff81352fdf>] xfs_reclaim_inodes_ag+0x23f/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829180] [<ffffffff81352e78>] ? xfs_reclaim_inodes_ag+0xd8/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829188] [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829192] [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829195] [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829199] [<ffffffff81141fda>] kswapd+0x70a/0xa60
> Feb 29 12:05:38 [kernel] [186732.829203] [<ffffffff810b4e30>] ? wake_up_bit+0x40/0x40
> Feb 29 12:05:38 [kernel] [186732.829206] [<ffffffff811418d0>] ? try_to_free_pages+0x110/0x110
> Feb 29 12:05:38 [kernel] [186732.829210] [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829214] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829217] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829221] [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829224] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829232] exim D 0000000000000000 2816 2718 1 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829237] ffff8801afb837c0 0000000000000046 ffffffff8189813a 0000000000000001
> Feb 29 12:05:38 [kernel] [186732.829242] ffff880100000000 00000000001d2880 ffff8801af161340 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829248] ffff8801afb83fd8 ffff8801afb82000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829253] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829256] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829259] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829263] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829266] [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829270] [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829273] [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829277] [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829280] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829284] [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829288] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829291] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829295] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829302] [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829313] [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829316] [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829320] [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829324] [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829328] [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829332] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829336] [<ffffffff81090212>] copy_process+0x122/0x1470
> Feb 29 12:05:38 [kernel] [186732.829339] [<ffffffff8189b68b>] ? _raw_spin_unlock_irqrestore+0x3b/0x70
> Feb 29 12:05:38 [kernel] [186732.829343] [<ffffffff810ce71d>] ? trace_hardirqs_on+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829347] [<ffffffff810915e9>] do_fork+0x59/0x350
> Feb 29 12:05:38 [kernel] [186732.829350] [<ffffffff81095b9c>] ? do_wait+0x12c/0x250
> Feb 29 12:05:38 [kernel] [186732.829354] [<ffffffff8189c02c>] ? sysret_check+0x27/0x62
> Feb 29 12:05:38 [kernel] [186732.829358] [<ffffffff8104b0e3>] sys_clone+0x23/0x30
> Feb 29 12:05:38 [kernel] [186732.829361] [<ffffffff8189c363>] stub_clone+0x13/0x20
> Feb 29 12:05:38 [kernel] [186732.829365] [<ffffffff8189bffb>] ? system_call_fastpath+0x16/0x1b
> Feb 29 12:05:38 [kernel] [186732.829368] gkrellmd D 0000000000000000 1872 2732 1 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829373] ffff8801afafd7c0 0000000000000046 ffffffff8189813a 0000000000000000
> Feb 29 12:05:38 [kernel] [186732.829378] ffff880100000000 00000000001d2880 ffff8801b1996040 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829383] ffff8801afafdfd8 ffff8801afafc000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829388] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829391] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829395] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829398] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829402] [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829405] [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829412] [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829416] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829420] [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829423] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829427] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829430] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829434] [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829438] [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829441] [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829445] [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829448] [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829452] [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829455] [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829459] [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829463] [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829467] [<ffffffff8114e7ab>] ? might_fault+0x3b/0x90
> Feb 29 12:05:38 [kernel] [186732.829471] [<ffffffff81090212>] copy_process+0x122/0x1470
> Feb 29 12:05:38 [kernel] [186732.829476] [<ffffffff8119c185>] ? dput+0x145/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829479] [<ffffffff8114e7f4>] ? might_fault+0x84/0x90
> Feb 29 12:05:38 [kernel] [186732.829482] [<ffffffff8114e7ab>] ? might_fault+0x3b/0x90
> Feb 29 12:05:38 [kernel] [186732.829486] [<ffffffff810915e9>] do_fork+0x59/0x350
> Feb 29 12:05:38 [kernel] [186732.829490] [<ffffffff8189c02c>] ? sysret_check+0x27/0x62
> Feb 29 12:05:38 [kernel] [186732.829493] [<ffffffff8104b0e3>] sys_clone+0x23/0x30
> Feb 29 12:05:38 [kernel] [186732.829497] [<ffffffff8189c363>] stub_clone+0x13/0x20
> Feb 29 12:05:38 [kernel] [186732.829500] [<ffffffff8189bffb>] ? system_call_fastpath+0x16/0x1b
> Feb 29 12:05:38 [kernel] [186732.829503] smbd D 0000000000000000 2792 2878 1 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829508] ffff8801b57037c0 0000000000000046 ffffffff8189813a 0000000000000001
> Feb 29 12:05:38 [kernel] [186732.829513] ffff880100000000 00000000001d2880 ffff8801b7a6cd00 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829524] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829527] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829531] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829534] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829548] [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829552] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829555] [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829559] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829563] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81090212>] copy_process+0x122/0x1470
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b9d5e>] ? up_read+0x1e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810915e9>] do_fork+0x59/0x350
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8114e7ab>] ? might_fault+0x3b/0x90
> - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8ca>] ? retint_swapgs+0xe/0x13
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8104b0e3>] sys_clone+0x23/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189c363>] stub_clone+0x13/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189bffb>] ? system_call_fastpath+0x16/0x1b
> Feb 29 12:05:38 [kernel] [186732.829564] sshd D 0000000000000000 2920 2919 1 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801afbb57c0 0000000000000046 ffffffff8189813a 0000000000000002
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880100000000 00000000001d2880 ffff8801b7946040 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801afbb5fd8 ffff8801afbb4000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
> - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81090212>] copy_process+0x122/0x1470
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810915e9>] do_fork+0x59/0x350
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8114e7ab>] ? might_fault+0x3b/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8104b0e3>] sys_clone+0x23/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189c363>] stub_clone+0x13/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189bffb>] ? system_call_fastpath+0x16/0x1b
> Feb 29 12:05:38 [kernel] [186732.829564] cron D 0000000000000000 2504 2952 1 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801afaf77c0 0000000000000046 ffffffff8189813a 0000000000000001
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880100000000 00000000001d2880 ffff8801b7ab8000 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801afaf7fd8 ffff8801afaf6000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
> - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8114e7ab>] ? might_fault+0x3b/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81090212>] copy_process+0x122/0x1470
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8119c185>] ? dput+0x145/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8114e7f4>] ? might_fault+0x84/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8114e7ab>] ? might_fault+0x3b/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810915e9>] do_fork+0x59/0x350
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189c02c>] ? sysret_check+0x27/0x62
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8104b0e3>] sys_clone+0x23/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189c363>] stub_clone+0x13/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189bffb>] ? system_call_fastpath+0x16/0x1b
> Feb 29 12:05:38 [kernel] [186732.829564] xfsaild/dm-0 D 0000000000000000 3800 3130 2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880185593b40 0000000000000046 ffffffff8189813a ffffffff810c7b3d
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880100000000 00000000001d2880 ffff8801b1b34d00 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880185593fd8 ffff880185592000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8139864c>] xlog_wait+0x6c/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189acdd>] ? _raw_spin_lock+0x5d/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81399184>] xlog_cil_force_lsn+0x114/0x120
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb325>] ? sched_clock_local+0x25/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810a1900>] ? try_to_del_timer_sync+0xe0/0xe0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff813975a6>] _xfs_log_force+0x66/0x2a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8139b710>] ? xfsaild+0x110/0x700
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff813977f3>] xfs_log_force+0x13/0x40
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8139b73d>] xfsaild+0x13d/0x700
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd D 0000000000000000 1656 3221 2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801b0b81c40 0000000000000046 ffffffff8189813a 0000000000000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880100000000 00000000001d2880 ffff8801b7a6b9c0 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801b0b81fd8 ffff8801b0b80000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812eb2e7>] nfsd4_open+0xb7/0x400
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812e97ee>] nfsd4_proc_compound+0x1ae/0x5d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183c625>] svc_process_common+0x345/0x690
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183cca1>] svc_process+0x101/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8d85>] nfsd+0xb5/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd D 0000000000000000 1480 3222 2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801b0afbc40 0000000000000046 ffffffff8189813a 0000000000000003
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880100000000 00000000001d2880 ffff880164900000 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801b0afbfd8 ffff8801b0afa000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812eb2e7>] nfsd4_open+0xb7/0x400
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812e97ee>] nfsd4_proc_compound+0x1ae/0x5d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d9663>] nfsd_dispatch+0xe3/0x220
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183c625>] svc_process_common+0x345/0x690
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183cca1>] svc_process+0x101/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8d85>] nfsd+0xb5/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801648a5c40 0000000000000046 ffffffff8189813a 0000000000000001
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880100000000 00000000001d2880 ffff880164901340 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801648a5fd8 ffff8801648a4000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812eb2e7>] nfsd4_open+0xb7/0x400
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812e97ee>] nfsd4_proc_compound+0x1ae/0x5d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d9663>] nfsd_dispatch+0xe3/0x220
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183c625>] svc_process_common+0x345/0x690
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183cca1>] svc_process+0x101/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8d85>] nfsd+0xb5/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd D 0000000000000000 1544 3224 2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff88018579f4a0 0000000000000046 ffffffff8189813a 0000000000000001
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880100000000 00000000001d2880 ffff880164902680 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] ffff88018579ffd8 ffff88018579e000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
> - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810ce4a1>] ? mark_held_locks+0x81/0x160
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8116b141>] alloc_pages_current+0xa1/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81174335>] new_slab+0x265/0x300
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81176b5a>] __slab_alloc+0x2ca/0x540
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff814773c8>] ? idr_pre_get+0x68/0x90
> - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8117891b>] kmem_cache_alloc+0xdb/0x120
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff814773c8>] idr_pre_get+0x68/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f506a>] nfs4_alloc_stid+0x1a/0x40
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f8e26>] nfsd4_process_open1+0x176/0x3d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812eb2fa>] nfsd4_open+0xca/0x400
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812e97ee>] nfsd4_proc_compound+0x1ae/0x5d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d9663>] nfsd_dispatch+0xe3/0x220
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183c625>] svc_process_common+0x345/0x690
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183cca1>] svc_process+0x101/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8d85>] nfsd+0xb5/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd D 0000000000000000 1400 3225 2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880164699c40 0000000000000046 ffffffff8189813a 0000000000000002
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880100000000 00000000001d2880 ffff8801649039c0 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812eb2e7>] nfsd4_open+0xb7/0x400
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812e97ee>] nfsd4_proc_compound+0x1ae/0x5d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d9663>] nfsd_dispatch+0xe3/0x220
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183c625>] svc_process_common+0x345/0x690
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183cca1>] svc_process+0x101/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8d85>] nfsd+0xb5/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd D 0000000000000000 2120 3226 2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801647b17a0 0000000000000046 ffffffff8189813a 0000000000000002
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880100000000 00000000001d2880 ffff880164904d00 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801647b1fd8 ffff8801647b0000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
> - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8116b141>] alloc_pages_current+0xa1/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8184b05c>] svc_recv+0xec/0x900
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8d6d>] nfsd+0x9d/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd D 0000000000000000 1880 3227 2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880164cb37a0 0000000000000046 ffffffff8189813a 0000000000000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880100000000 00000000001d2880 ffff880164906040 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880164cb3fd8 ffff880164cb2000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
> - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8116b141>] alloc_pages_current+0xa1/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd D 0000000000000000 1768 3228 2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880164dadc40 0000000000000046 ffffffff8189813a 0000000000000002
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880100000000 00000000001d2880 ffff880164db0000 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880164dadfd8 ffff880164dac000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812eb2e7>] nfsd4_open+0xb7/0x400
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812e97ee>] nfsd4_proc_compound+0x1ae/0x5d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d9663>] nfsd_dispatch+0xe3/0x220
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183c625>] svc_process_common+0x345/0x690
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183cca1>] svc_process+0x101/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd D 0000000000000000 1928 3229 2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880164ecf940 0000000000000046 ffffffff8189813a ffffffff810c7b3d
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880100000000 00000000001d2880 ffff880164db1340 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880164ecffd8 ffff880164ece000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8172154d>] __lock_sock+0x6d/0xa0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b4e30>] ? wake_up_bit+0x40/0x40
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189aff2>] ? _raw_spin_lock_bh+0x62/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81721639>] lock_sock_nested+0x39/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8178b640>] tcp_recvmsg+0x30/0xb00
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff817230cf>] ? sock_update_classid+0x5f/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff817b0208>] inet_recvmsg+0xc8/0xf0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81723110>] ? sock_update_classid+0xa0/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8171d518>] sock_recvmsg+0xf8/0x130
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810a18a5>] ? try_to_del_timer_sync+0x85/0xe0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8171d58f>] kernel_recvmsg+0x3f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183e1ae>] svc_tcp_recvfrom+0x36e/0x760
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8184b256>] ? svc_recv+0x2e6/0x900
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8d6d>] nfsd+0x9d/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd D 0000000000000000 1480 3230 2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880164fc9c40 0000000000000046 ffffffff8189813a 0000000000000003
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880100000000 00000000001d2880 ffff880164db2680 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880164fc9fd8 ffff880164fc8000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812eb2e7>] nfsd4_open+0xb7/0x400
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d9663>] nfsd_dispatch+0xe3/0x220
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183c625>] svc_process_common+0x345/0x690
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183cca1>] svc_process+0x101/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8d85>] nfsd+0xb5/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd D 0000000000000000 1704 3231 2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801650cb2d0 0000000000000046 00000000001d3300 0000000000000001
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801650cb260 00000000001d2880 ffff880164db39c0 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801650cbfd8 ffff8801650ca000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
> - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810ce4a1>] ? mark_held_locks+0x81/0x160
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8116b141>] alloc_pages_current+0xa1/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81174335>] new_slab+0x265/0x300
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81176b5a>] __slab_alloc+0x2ca/0x540
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8178976c>] ? sk_stream_alloc_skb+0x3c/0x130
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810ce4a1>] ? mark_held_locks+0x81/0x160
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81179c33>] __kmalloc_node_track_caller+0x83/0x190
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811770bf>] ? kmem_cache_alloc_node+0x2f/0x130
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8178976c>] ? sk_stream_alloc_skb+0x3c/0x130
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81726965>] __alloc_skb+0x75/0x170
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8178976c>] sk_stream_alloc_skb+0x3c/0x130
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8178a9f2>] tcp_sendpage+0x3f2/0x740
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff817230cf>] ? sock_update_classid+0x5f/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff817b03ee>] inet_sendpage+0xce/0x130
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff817b0320>] ? inet_sendmsg+0xf0/0xf0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81723110>] ? sock_update_classid+0xa0/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8171bee3>] kernel_sendpage+0x53/0x80
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183f2e2>] svc_send_common+0x62/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183f42e>] svc_sendto+0x5e/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810ce4a1>] ? mark_held_locks+0x81/0x160
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810ce685>] ? trace_hardirqs_on_caller+0x105/0x190
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81899108>] ? mutex_lock_nested+0x278/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8184ad59>] ? svc_send+0x59/0xf0
> - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183f604>] svc_tcp_sendto+0x34/0xa0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8184adab>] svc_send+0xab/0xf0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183ccc0>] svc_process+0x120/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8d85>] nfsd+0xb5/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd D 0000000000000000 1848 3232 2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801651c57a0 0000000000000046 ffffffff8189813a 0000000000000003
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880100000000 00000000001d2880 ffff880164db4d00 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801651c5fd8 ffff8801651c4000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
> - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183e34d>] ? svc_tcp_recvfrom+0x50d/0x760
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8116b141>] alloc_pages_current+0xa1/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8184b05c>] svc_recv+0xec/0x900
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8d6d>] nfsd+0x9d/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd D 0000000000000000 1304 3233 2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801652e5c40 0000000000000046 ffffffff8189813a 0000000000000003
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801652e5fd8 ffff8801652e4000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812eb2e7>] nfsd4_open+0xb7/0x400
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812e97ee>] nfsd4_proc_compound+0x1ae/0x5d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d9663>] nfsd_dispatch+0xe3/0x220
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183c625>] svc_process_common+0x345/0x690
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183cca1>] svc_process+0x101/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8d85>] nfsd+0xb5/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd D 0000000000000000 1848 3234 2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801653dfc40 0000000000000046 ffffffff8189813a 0000000000000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880100000000 00000000001d2880 ffff8801653e0000 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801653dffd8 ffff8801653de000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812eb2e7>] nfsd4_open+0xb7/0x400
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812e97ee>] nfsd4_proc_compound+0x1ae/0x5d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183c625>] svc_process_common+0x345/0x690
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183cca1>] svc_process+0x101/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8d85>] nfsd+0xb5/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd D 0000000000000000 2024 3235 2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801654e9c40 0000000000000046 ffffffff8189813a 0000000000000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880100000000 00000000001d2880 ffff8801653e1340 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801654e9fd8 ffff8801654e8000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812eb2e7>] nfsd4_open+0xb7/0x400
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812e97ee>] nfsd4_proc_compound+0x1ae/0x5d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d9663>] nfsd_dispatch+0xe3/0x220
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183c625>] svc_process_common+0x345/0x690
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183cca1>] svc_process+0x101/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8d85>] nfsd+0xb5/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] nfsd D 0000000000000000 1304 3236 2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8801656032d0 0000000000000046 ffffffff8189813a 0000000000000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880100000000 00000000001d2880 ffff8801653e2680 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880165603fd8 ffff880165602000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810ce4a1>] ? mark_held_locks+0x81/0x160
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8116b141>] alloc_pages_current+0xa1/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81174335>] new_slab+0x265/0x300
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81176b5a>] __slab_alloc+0x2ca/0x540
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8178976c>] ? sk_stream_alloc_skb+0x3c/0x130
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810ce4a1>] ? mark_held_locks+0x81/0x160
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81179c33>] __kmalloc_node_track_caller+0x83/0x190
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811770bf>] ? kmem_cache_alloc_node+0x2f/0x130
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8178976c>] ? sk_stream_alloc_skb+0x3c/0x130
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81726965>] __alloc_skb+0x75/0x170
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8178976c>] sk_stream_alloc_skb+0x3c/0x130
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8178a9f2>] tcp_sendpage+0x3f2/0x740
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff817b03ee>] inet_sendpage+0xce/0x130
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff817b0320>] ? inet_sendmsg+0xf0/0xf0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81723110>] ? sock_update_classid+0xa0/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8171bee3>] kernel_sendpage+0x53/0x80
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183f2e2>] svc_send_common+0x62/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183f42e>] svc_sendto+0x5e/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810ce4a1>] ? mark_held_locks+0x81/0x160
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189911c>] ? mutex_lock_nested+0x28c/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810ce685>] ? trace_hardirqs_on_caller+0x105/0x190
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81899108>] ? mutex_lock_nested+0x278/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8184ad59>] ? svc_send+0x59/0xf0
> - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183f604>] svc_tcp_sendto+0x34/0xa0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8184adab>] svc_send+0xab/0xf0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8183ccc0>] svc_process+0x120/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8d85>] nfsd+0xb5/0x150
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812d8cd0>] ? nfsd_shutdown+0x30/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] kworker/u:4 D 0000000000000000 4520 23447 2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8800471c3af0 0000000000000046 ffff8800471c3a70 ffffffff810bb325
> Feb 29 12:05:38 [kernel] [186732.829564] 00000000001d3300 00000000001d2880 ffff8801b5af2680 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb325>] ? sched_clock_local+0x25/0x90
> - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81898985>] schedule_timeout+0x1f5/0x2e0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b63b>] ? _raw_spin_unlock_irq+0x2b/0x40
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810ce685>] ? trace_hardirqs_on_caller+0x105/0x190
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff818978e0>] wait_for_common+0xc0/0x160
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81897a28>] wait_for_completion+0x18/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81897846>] ? wait_for_common+0x26/0x160
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810af11b>] create_worker+0xbb/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810af31d>] manage_workers.clone.20+0x11d/0x240
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810af6c8>] worker_thread+0x288/0x340
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810af440>] ? manage_workers.clone.20+0x240/0x240
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] kworker/u:2 D 0000000000000000 4520 25347 2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8800a9a05c50 0000000000000046 ffffffff8189813a 0000000000000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff8800a9a05fd8 ffff8800a9a04000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] ? nfs4_lock_state+0x12/0x20
> - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f61e2>] nfs4_lock_state+0x12/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f88c5>] laundromat_main+0x25/0x410
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810ada09>] process_one_work+0x1a9/0x4f0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810ad99a>] ? process_one_work+0x13a/0x4f0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff812f88a0>] ? nfsd4_exchange_id+0x440/0x440
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810af5a2>] worker_thread+0x162/0x340
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810af440>] ? manage_workers.clone.20+0x240/0x240
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] kworker/3:3 D 0000000000000000 4264 27581 2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff88000ec6d640 0000000000000046 ffffffff8189813a 0000000000000006
> Feb 29 12:05:38 [kernel] [186732.829564] 0000000000000001 00000000001d2880 ffff8801b26ab9c0 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] ffff88000ec6dfd8 ffff88000ec6c000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff816cb13d>] md_flush_request+0x7d/0x140
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81088560>] ? try_to_wake_up+0x2d0/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff816cf14f>] ? md_make_request+0x13f/0x2b0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff816da24b>] ? dm_request+0x13b/0x230
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff816cf184>] md_make_request+0x174/0x2b0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff816cf078>] ? md_make_request+0x68/0x2b0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b9d5e>] ? up_read+0x1e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff816da24b>] ? dm_request+0x13b/0x230
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff816da13e>] ? dm_request+0x2e/0x230
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8145c722>] generic_make_request+0xc2/0x100
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8145d4d5>] submit_bio+0x75/0xf0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81341a82>] _xfs_buf_ioapply+0x162/0x1d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81394111>] ? xlog_bdstrat+0x21/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81341bca>] xfs_buf_iorequest+0x4a/0xa0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81394111>] xlog_bdstrat+0x21/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81395d9e>] xlog_sync+0x23e/0x3d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81397258>] xlog_write+0x618/0x780
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81398830>] ? xlog_cil_push+0x1c0/0x3b0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff813988a9>] xlog_cil_push+0x239/0x3b0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81399184>] xlog_cil_force_lsn+0x114/0x120
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810ce4a1>] ? mark_held_locks+0x81/0x160
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81397871>] _xfs_log_force_lsn+0x51/0x2f0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810ce71d>] ? trace_hardirqs_on+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81393d3b>] xfs_trans_commit+0x28b/0x2a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8134782d>] xfs_fs_log_dummy+0x5d/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81351e14>] xfs_sync_worker+0x84/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810ad99a>] ? process_one_work+0x13a/0x4f0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81351d90>] ? xfs_sync_inode_attr+0xf0/0xf0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810af5a2>] worker_thread+0x162/0x340
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810af440>] ? manage_workers.clone.20+0x240/0x240
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] kworker/0:1 D 0000000000000000 1928 27768 2 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff88005ee6d460 0000000000000046 ffffffff8189813a 0000000000000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880000000000 00000000001d2880 ffff8801653e6040 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] ffff88005ee6dfd8 ffff88005ee6c000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
> - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810d633d>] ? __module_text_address+0xd/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810ce4a1>] ? mark_held_locks+0x81/0x160
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8116b141>] alloc_pages_current+0xa1/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81174335>] new_slab+0x265/0x300
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81176b5a>] __slab_alloc+0x2ca/0x540
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8112f3a0>] ? mempool_alloc_slab+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810ce4a1>] ? mark_held_locks+0x81/0x160
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810ce71d>] ? trace_hardirqs_on+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8112f3a0>] ? mempool_alloc_slab+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8117891b>] kmem_cache_alloc+0xdb/0x120
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811082c2>] ? __delayacct_blkio_end+0x32/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8112f3a0>] mempool_alloc_slab+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b4e30>] ? wake_up_bit+0x40/0x40
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811b9919>] bio_alloc_bioset+0x39/0xf0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff816ceef9>] ? submit_flushes+0xd9/0x1f0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff816cb2d5>] bio_alloc_mddev+0x35/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff816cef2e>] submit_flushes+0x10e/0x1f0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff816ceef9>] ? submit_flushes+0xd9/0x1f0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810ada09>] process_one_work+0x1a9/0x4f0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810ad99a>] ? process_one_work+0x13a/0x4f0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff816cee20>] ? md_do_sync+0xcd0/0xcd0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810af5a2>] worker_thread+0x162/0x340
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b48c6>] kthread+0xa6/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b4>] kernel_thread_helper+0x4/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b8dd>] ? retint_restore_args+0xe/0xe
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810b4820>] ? __init_kthread_worker+0x70/0x70
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189e1b0>] ? gs_change+0xb/0xb
> Feb 29 12:05:38 [kernel] [186732.829564] rsync D 0000000000000000 1976 28091 28084 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff88000bcc7478 0000000000000046 ffffffff8189813a 0000000000000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880000000001 00000000001d2880 ffff880165682680 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] ffff88000bcc7fd8 ffff88000bcc6000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81899095>] ? mutex_lock_nested+0x205/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
> - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8116b141>] alloc_pages_current+0xa1/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8112cc27>] __page_cache_alloc+0x87/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8113871a>] __do_page_cache_readahead+0x13a/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8113869e>] ? __do_page_cache_readahead+0xbe/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81138a2c>] ra_submit+0x1c/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81138b3e>] ondemand_readahead+0x10e/0x2b0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81138b70>] ? ondemand_readahead+0x140/0x2b0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81138daf>] page_cache_sync_readahead+0x2f/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8112e774>] generic_file_aio_read+0x444/0x740
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81344fc9>] xfs_file_aio_read+0x179/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8118444a>] do_sync_read+0xda/0x120
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811c1898>] ? fsnotify+0x1d8/0x320
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811c1745>] ? fsnotify+0x85/0x320
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8143f1d6>] ? security_file_permission+0x96/0xb0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81897e1c>] ? __schedule+0x3ec/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81184edf>] sys_read+0x4f/0xa0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189bffb>] system_call_fastpath+0x16/0x1b
> Feb 29 12:05:38 [kernel] [186732.829564] run-crons D 0000000000000000 2904 28235 28234 0x00000000
> Feb 29 12:05:38 [kernel] [186732.829564] ffff88005199b7c0 0000000000000046 ffffffff8189813a 0000000000000001
> Feb 29 12:05:38 [kernel] [186732.829564] ffff880000000000 00000000001d2880 ffff880165680000 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] ffff88005199bfd8 ffff88005199a000 00000000001d2880 00000000001d2880
> Feb 29 12:05:38 [kernel] [186732.829564] Call Trace:
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189813a>] ? __schedule+0x70a/0x920
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189841a>] schedule+0x3a/0x50
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81898fed>] mutex_lock_nested+0x15d/0x330
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8135308e>] ? xfs_reclaim_inodes_ag+0x2ee/0x3a0
> - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8135308e>] xfs_reclaim_inodes_ag+0x2ee/0x3a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb57f>] ? local_clock+0x4f/0x60
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8139bdf2>] ? xfs_ail_push_all+0x42/0x90
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189b6e6>] ? _raw_spin_unlock+0x26/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff813532ce>] xfs_reclaim_inodes_nr+0x2e/0x40
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8134f840>] xfs_fs_free_cached_objects+0x10/0x20
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811881f1>] prune_super+0x101/0x1b0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8113eb45>] shrink_slab+0x165/0x2d0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff811414e7>] do_try_to_free_pages+0x267/0x460
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81141856>] try_to_free_pages+0x96/0x110
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81134ecd>] __alloc_pages_nodemask+0x4cd/0x820
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810bb4b8>] ? sched_clock_cpu+0xa8/0x120
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff81090212>] copy_process+0x122/0x1470
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c81ed>] ? lock_release_holdtime+0x3d/0x1a0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810cd87f>] ? lock_release_non_nested+0x37f/0x3c0
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810c7b3d>] ? trace_hardirqs_off+0xd/0x10
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff810915e9>] do_fork+0x59/0x350
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8114e7ab>] ? might_fault+0x3b/0x90
> - Last output repeated twice -
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189c02c>] ? sysret_check+0x27/0x62
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8104b0e3>] sys_clone+0x23/0x30
> Feb 29 12:05:38 [kernel] [186732.829564] [<ffffffff8189bffb>] ? system_call_fastpath+0x16/0x1b
> Feb 29 12:05:38 [kernel] [186732.829564] Sched Debug Version: v0.10, 3.2.5 #1
> Feb 29 12:05:38 [kernel] [186732.829564] ktime : 186732833.030196
> Feb 29 12:05:38 [kernel] [186732.829564] sched_clk : 186363810.286830
> Feb 29 12:05:38 [kernel] [186732.829564] cpu_clk : 186732829.564698
> Feb 29 12:05:38 [kernel] [186732.829564] jiffies : 4481400126
> Feb 29 12:05:38 [kernel] [186732.829564] sched_clock_stable : 0
> Feb 29 12:05:38 [kernel] [186732.829564]
> Feb 29 12:05:38 [kernel] [186732.829564] sysctl_sched
> Feb 29 12:05:38 [kernel] [186732.829564] .sysctl_sched_latency : 18.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .sysctl_sched_min_granularity : 2.250000
> Feb 29 12:05:38 [kernel] [186732.829564] .sysctl_sched_wakeup_granularity : 3.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .sysctl_sched_child_runs_first : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .sysctl_sched_features : 24119
> Feb 29 12:05:38 [kernel] [186732.829564] .sysctl_sched_tunable_scaling : 1 (logaritmic)
> Feb 29 12:05:38 [kernel] [186732.829564]
> Feb 29 12:05:38 [kernel] [186732.829564] cpu#0, 2333.084 MHz
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_running : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .load : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_switches : 127626917
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_load_updates : 64695581
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_uninterruptible : -5185
> Feb 29 12:05:38 [kernel] [186732.829564] .clock : 186732809.005246
> Feb 29 12:05:38 [kernel] [186732.829564] .cpu_load[0] : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .cpu_load[1] : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .cpu_load[2] : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .cpu_load[3] : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .cpu_load[4] : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .yld_count : 2
> Feb 29 12:05:38 [kernel] [186732.829564] .sched_switch : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .sched_goidle : 59908314
> Feb 29 12:05:38 [kernel] [186732.829564] .avg_idle : 1000000
> Feb 29 12:05:38 [kernel] [186732.829564] .ttwu_count : 70228802
> Feb 29 12:05:38 [kernel] [186732.829564] .ttwu_local : 49835264
> Feb 29 12:05:38 [kernel] [186732.829564]
> Feb 29 12:05:38 [kernel] [186732.829564] cfs_rq[0]:/
> Feb 29 12:05:38 [kernel] [186732.829564] .exec_clock : 12099620.925352
> Feb 29 12:05:38 [kernel] [186732.829564] .MIN_vruntime : 0.000001
> Feb 29 12:05:38 [kernel] [186732.829564] .min_vruntime : 10865130.739847
> Feb 29 12:05:38 [kernel] [186732.829564] .max_vruntime : 0.000001
> Feb 29 12:05:38 [kernel] [186732.829564] .spread : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .spread0 : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_spread_over : 633
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_running : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .load : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .load_avg : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .load_period : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .load_contrib : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .load_tg : 0
> Feb 29 12:05:38 [kernel] [186732.829564]
> Feb 29 12:05:38 [kernel] [186732.829564] rt_rq[0]:
> Feb 29 12:05:38 [kernel] [186732.829564] .rt_nr_running : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .rt_throttled : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .rt_time : 0.010808
> Feb 29 12:05:38 [kernel] [186732.829564] .rt_runtime : 950.000000
> Feb 29 12:05:38 [kernel] [186732.829564]
> Feb 29 12:05:38 [kernel] [186732.829564] runnable tasks:
> Feb 29 12:05:38 [kernel] [186732.829564] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> Feb 29 12:05:38 [kernel] [186732.829564] ----------------------------------------------------------------------------------------------------------
> Feb 29 12:05:38 [kernel] [186732.829564]
> Feb 29 12:05:38 [kernel] [186732.829564] cpu#1, 2333.084 MHz
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_running : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .load : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_switches : 128426315
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_load_updates : 65499669
> Feb 29 12:05:38 [kernel] [186732.829564] .next_balance : 4481.400004
> Feb 29 12:05:38 [kernel] [186732.829564] .curr->pid : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .clock : 186732704.045991
> Feb 29 12:05:38 [kernel] [186732.829564] .cpu_load[0] : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .cpu_load[1] : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .cpu_load[2] : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .cpu_load[3] : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .cpu_load[4] : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .yld_count : 3
> Feb 29 12:05:38 [kernel] [186732.829564] .sched_switch : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .sched_count : 128816327
> Feb 29 12:05:38 [kernel] [186732.829564] .sched_goidle : 60378764
> Feb 29 12:05:38 [kernel] [186732.829564] .avg_idle : 1000000
> Feb 29 12:05:38 [kernel] [186732.829564] .ttwu_count : 70202501
> Feb 29 12:05:38 [kernel] [186732.829564] .ttwu_local : 50152935
> Feb 29 12:05:38 [kernel] [186732.829564]
> Feb 29 12:05:38 [kernel] [186732.829564] cfs_rq[1]:/
> Feb 29 12:05:38 [kernel] [186732.829564] .exec_clock : 11942938.971418
> Feb 29 12:05:38 [kernel] [186732.829564] .MIN_vruntime : 0.000001
> Feb 29 12:05:38 [kernel] [186732.829564] .min_vruntime : 10965585.421468
> Feb 29 12:05:38 [kernel] [186732.829564] .max_vruntime : 0.000001
> Feb 29 12:05:38 [kernel] [186732.829564] .spread : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .spread0 : 100454.681621
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_spread_over : 637
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_running : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .load : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .load_avg : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .load_contrib : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .load_tg : 0
> Feb 29 12:05:38 [kernel] [186732.829564]
> Feb 29 12:05:38 [kernel] [186732.829564] rt_rq[1]:
> Feb 29 12:05:38 [kernel] [186732.829564] .rt_nr_running : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .rt_throttled : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .rt_time : 0.011839
> Feb 29 12:05:38 [kernel] [186732.829564] .rt_runtime : 950.000000
> Feb 29 12:05:38 [kernel] [186732.829564]
> Feb 29 12:05:38 [kernel] [186732.829564] runnable tasks:
> Feb 29 12:05:38 [kernel] [186732.829564] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> Feb 29 12:05:38 [kernel] [186732.829564] ----------------------------------------------------------------------------------------------------------
> Feb 29 12:05:38 [kernel] [186732.829564]
> Feb 29 12:05:38 [kernel] [186732.829564] cpu#2, 2333.084 MHz
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_running : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .load : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_switches : 124877810
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_load_updates : 63257405
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_uninterruptible : 4945
> Feb 29 12:05:38 [kernel] [186732.829564] .next_balance : 4481.400130
> Feb 29 12:05:38 [kernel] [186732.829564] .curr->pid : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .clock : 186732830.026053
> Feb 29 12:05:38 [kernel] [186732.829564] .cpu_load[0] : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .cpu_load[1] : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .cpu_load[2] : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .cpu_load[3] : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .cpu_load[4] : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .yld_count : 1
> Feb 29 12:05:38 [kernel] [186732.829564] .sched_switch : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .sched_count : 125239550
> Feb 29 12:05:38 [kernel] [186732.829564] .sched_goidle : 59393188
> Feb 29 12:05:38 [kernel] [186732.829564] .avg_idle : 1000000
> Feb 29 12:05:38 [kernel] [186732.829564] .ttwu_count : 60314669
> Feb 29 12:05:38 [kernel] [186732.829564] .ttwu_local : 44799719
> Feb 29 12:05:38 [kernel] [186732.829564]
> Feb 29 12:05:38 [kernel] [186732.829564] cfs_rq[2]:/autogroup-41
> Feb 29 12:05:38 [kernel] [186732.829564] .exec_clock : 1550.967553
> Feb 29 12:05:38 [kernel] [186732.829564] .MIN_vruntime : 0.000001
> Feb 29 12:05:38 [kernel] [186732.829564] .min_vruntime : 1549.918977
> Feb 29 12:05:38 [kernel] [186732.829564] .max_vruntime : 0.000001
> Feb 29 12:05:38 [kernel] [186732.829564] .spread : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .spread0 : -10863580.820870
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_running : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .load : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .load_avg : 5119.999488
> Feb 29 12:05:38 [kernel] [186732.829564] .load_period : 6.468593
> Feb 29 12:05:38 [kernel] [186732.829564] .load_contrib : 791
> Feb 29 12:05:38 [kernel] [186732.829564] .load_tg : 791
> Feb 29 12:05:38 [kernel] [186732.829564] .se->exec_start : 186732828.548911
> Feb 29 12:05:38 [kernel] [186732.829564] .se->vruntime : 8360658.005921
> Feb 29 12:05:38 [kernel] [186732.829564] .se->sum_exec_runtime : 1552.122095
> Feb 29 12:05:38 [kernel] [186732.829564] .se->statistics.wait_start : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .se->statistics.sleep_start : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .se->statistics.block_start : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .se->statistics.sleep_max : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .se->statistics.block_max : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .se->statistics.exec_max : 3.133477
> Feb 29 12:05:38 [kernel] [186732.829564] .se->statistics.slice_max : 1.465148
> Feb 29 12:05:38 [kernel] [186732.829564] .se->statistics.wait_max : 3.850534
> Feb 29 12:05:38 [kernel] [186732.829564] .se->statistics.wait_sum : 133.221461
> Feb 29 12:05:38 [kernel] [186732.829564] .se->statistics.wait_count : 16329
> Feb 29 12:05:38 [kernel] [186732.829564] .se->load.weight : 2
> Feb 29 12:05:38 [kernel] [186732.829564]
> Feb 29 12:05:38 [kernel] [186732.829564] cfs_rq[2]:/
> Feb 29 12:05:38 [kernel] [186732.829564] .exec_clock : 9867308.118613
> Feb 29 12:05:38 [kernel] [186732.829564] .MIN_vruntime : 0.000001
> Feb 29 12:05:38 [kernel] [186732.829564] .min_vruntime : 8360666.433800
> Feb 29 12:05:38 [kernel] [186732.829564] .max_vruntime : 0.000001
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_spread_over : 745
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_running : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .load : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .load_avg : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .load_period : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .load_contrib : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .load_tg : 0
> Feb 29 12:05:38 [kernel] [186732.829564]
> Feb 29 12:05:38 [kernel] [186732.829564] rt_rq[2]:
> Feb 29 12:05:38 [kernel] [186732.829564] .rt_nr_running : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .rt_throttled : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .rt_time : 0.007240
> Feb 29 12:05:38 [kernel] [186732.829564] .rt_runtime : 950.000000
> Feb 29 12:05:38 [kernel] [186732.829564]
> Feb 29 12:05:38 [kernel] [186732.829564] runnable tasks:
> Feb 29 12:05:38 [kernel] [186732.829564] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> Feb 29 12:05:38 [kernel] [186732.829564] ----------------------------------------------------------------------------------------------------------
> Feb 29 12:05:38 [kernel] [186732.829564]
> Feb 29 12:05:38 [kernel] [186732.829564] cpu#3, 2333.084 MHz
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_running : 2
> Feb 29 12:05:38 [kernel] [186732.829564] .load : 2048
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_switches : 124026375
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_load_updates : 62682931
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_uninterruptible : 3772
> Feb 29 12:05:38 [kernel] [186732.829564] .next_balance : 4481.400098
> Feb 29 12:05:38 [kernel] [186732.829564] .curr->pid : 2991
> Feb 29 12:05:38 [kernel] [186732.829564] .clock : 186732828.596971
> Feb 29 12:05:38 [kernel] [186732.829564] .cpu_load[0] : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .cpu_load[1] : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .cpu_load[2] : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .cpu_load[3] : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .cpu_load[4] : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .yld_count : 4
> Feb 29 12:05:38 [kernel] [186732.829564] .sched_switch : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .sched_count : 124393065
> Feb 29 12:05:38 [kernel] [186732.829564] .sched_goidle : 58899951
> Feb 29 12:05:38 [kernel] [186732.829564] .avg_idle : 1000000
> Feb 29 12:05:38 [kernel] [186732.829564] .ttwu_count : 59877024
> Feb 29 12:05:38 [kernel] [186732.829564] .ttwu_local : 44537104
> Feb 29 12:05:38 [kernel] [186732.829564]
> Feb 29 12:05:38 [kernel] [186732.829564] cfs_rq[3]:/autogroup-42
> Feb 29 12:05:38 [kernel] [186732.829564] .MIN_vruntime : 0.000001
> Feb 29 12:05:38 [kernel] [186732.829564] .min_vruntime : 626168.958817
> Feb 29 12:05:38 [kernel] [186732.829564] .max_vruntime : 0.000001
> Feb 29 12:05:38 [kernel] [186732.829564] .spread : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .spread0 : -10238961.781030
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_spread_over : 23016
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_running : 1
> Feb 29 12:05:38 [kernel] [186732.829564] .load : 1024
> Feb 29 12:05:38 [kernel] [186732.829564] .load_avg : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .load_period : 9.999999
> Feb 29 12:05:38 [kernel] [186732.829564] .load_contrib : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .load_tg : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .se->exec_start : 186732828.596971
> Feb 29 12:05:38 [kernel] [186732.829564] .se->vruntime : 8279374.470424
> Feb 29 12:05:38 [kernel] [186732.829564] .se->sum_exec_runtime : 263285.952949
> Feb 29 12:05:38 [kernel] [186732.829564] .se->statistics.wait_start : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .se->statistics.sleep_start : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .se->statistics.block_start : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .se->statistics.sleep_max : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .se->statistics.block_max : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .se->statistics.exec_max : 13.400424
> Feb 29 12:05:38 [kernel] [186732.829564] .se->statistics.slice_max : 46.505344
> Feb 29 12:05:38 [kernel] [186732.829564] .se->statistics.wait_max : 16.575587
> Feb 29 12:05:38 [kernel] [186732.829564] .se->statistics.wait_sum : 2448.726587
> Feb 29 12:05:38 [kernel] [186732.829564] .se->statistics.wait_count : 242426
> Feb 29 12:05:38 [kernel] [186732.829564] .se->load.weight : 1024
> Feb 29 12:05:38 [kernel] [186732.829564]
> Feb 29 12:05:38 [kernel] [186732.829564] .exec_clock : 9783020.902642
> Feb 29 12:05:38 [kernel] [186732.829564] .MIN_vruntime : 8279372.216698
> Feb 29 12:05:38 [kernel] [186732.829564] .min_vruntime : 8279380.963316
> Feb 29 12:05:38 [kernel] [186732.829564] .max_vruntime : 8279372.216698
> Feb 29 12:05:38 [kernel] [186732.829564] .spread : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .spread0 : -2585749.776531
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_spread_over : 1511
> Feb 29 12:05:38 [kernel] [186732.829564] .nr_running : 2
> Feb 29 12:05:38 [kernel] [186732.829564] .load : 2048
> Feb 29 12:05:38 [kernel] [186732.829564] .load_avg : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .load_period : 0.000000
> Feb 29 12:05:38 [kernel] [186732.829564] .load_contrib : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .load_tg : 0
> Feb 29 12:05:38 [kernel] [186732.829564]
> Feb 29 12:05:38 [kernel] [186732.829564] rt_rq[3]:
> Feb 29 12:05:38 [kernel] [186732.829564] .rt_nr_running : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .rt_throttled : 0
> Feb 29 12:05:38 [kernel] [186732.829564] .rt_time : 0.109158
> Feb 29 12:05:38 [kernel] [186732.829564] .rt_runtime : 950.000000
> Feb 29 12:05:38 [kernel] [186732.829564]
> Feb 29 12:05:38 [kernel] [186732.829564] runnable tasks:
> Feb 29 12:05:38 [kernel] [186732.829564] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
> Feb 29 12:05:38 [kernel] [186732.829564] ----------------------------------------------------------------------------------------------------------
> Feb 29 12:05:38 [kernel] [186732.829564] R bash 2991 626164.399889 1092 120 626164.399889 164.832603 186663838.488864 /autogroup-42
> Feb 29 12:05:38 [kernel] [186732.829564] kworker/3:1 27652 8279372.216698 31913 120 8279372.216698 2529.050613 7144046.892045 /
> Feb 29 12:05:38 [kernel] [186732.829564]
>
--
Jan Kara <[email protected]>
SUSE Labs, CR