This is very reproducible. It occurs in two of my test cases, with
different panic messages.
It also shows up in 3.2.0-rc3. Kernel config is attached.
In the test case
snb/JBOD-4HDD-thresh=100M/ext4-100dd-1-3.2.0-rc5
panic dmesg is:
[ 541.548310] BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
[ 541.558118] IP: [<ffffffff811ff00b>] perf_trace_ext4_mb_release_group_pa+0x81/0xd6
[ 541.567553] PGD 0
[ 541.570360] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
[ 541.576136] CPU 19
[ 541.578358] Modules linked in:
[ 541.583036]
[ 541.585153] Pid: 4244, comm: flush-8:64 Tainted: G W 3.2.0-rc5 #990 Intel Corporation SandyBridge Platform/To be filled by O.E.M.
[ 541.600259] RIP: 0010:[<ffffffff811ff00b>] [<ffffffff811ff00b>] perf_trace_ext4_mb_release_group_pa+0x81/0xd6
[ 541.612406] RSP: 0018:ffff88041e901540 EFLAGS: 00010286
[ 541.618812] RAX: ffffe8fbfe407250 RBX: ffff8807ff28a348 RCX: ffff880419de2200
[ 541.627255] RDX: 0000000000000000 RSI: 0000000000000286 RDI: ffffe8fbfe407250
[ 541.635698] RBP: ffff88041e901620 R08: ffff88041e901560 R09: 0000000000000000
[ 541.644126] R10: ffffe8fffce07220 R11: ffffffff812162cf R12: ffffffff81ee4108
[ 541.652554] R13: ffff88041e9016d0 R14: ffff8804153120d0 R15: ffff8804144767e0
[ 541.661012] FS: 0000000000000000(0000) GS:ffff88042d400000(0000) knlGS:0000000000000000
[ 541.670970] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 541.677858] CR2: 0000000000000028 CR3: 0000000001e91000 CR4: 00000000000406e0
[ 541.686284] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 541.694740] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 541.703160] Process flush-8:64 (pid: 4244, threadinfo ffff88041e900000, task ffff880419de2290)
[ 541.713657] Stack:
[ 541.716373] 0000000000000000 0000000000000004 ffff88041e9015d0 ffff88041e901560
[ 541.725845] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 541.735225] ffff88041e901670 0000000000000000 0000000000000000 0000000000000000
[ 541.744730] Call Trace:
[ 541.747939] [<ffffffff8121383c>] ? ext4_mb_release_group_pa+0x40/0xfe
[ 541.755708] [<ffffffff8121383c>] ext4_mb_release_group_pa+0x40/0xfe
[ 541.763270] [<ffffffff8121630f>] ext4_mb_discard_group_preallocations+0x355/0x3eb
[ 541.772598] [<ffffffff8121925e>] ext4_mb_new_blocks+0x2fd/0x422
[ 541.779788] [<ffffffff81210bb8>] ext4_ext_map_blocks+0x14ba/0x19bd
[ 541.787213] [<ffffffff81099c65>] ? local_clock+0x41/0x5a
[ 541.793723] [<ffffffff810a554f>] ? __lock_acquire+0x564/0x932
[ 541.800708] [<ffffffff810a554f>] ? __lock_acquire+0x564/0x932
[ 541.807694] [<ffffffff811ec728>] ? ext4_map_blocks+0x103/0x221
[ 541.814773] [<ffffffff811ec759>] ext4_map_blocks+0x134/0x221
[ 541.821647] [<ffffffff811effad>] mpage_da_map_and_submit+0xef/0x404
[ 541.829212] [<ffffffff811f0a22>] ext4_da_writepages+0x350/0x505
[ 541.836409] [<ffffffff810a4a9c>] ? lock_release_holdtime+0xa3/0xac
[ 541.843904] [<ffffffff8110a08f>] do_writepages+0x24/0x2d
[ 541.850406] [<ffffffff8116eda2>] writeback_single_inode+0x126/0x2b4
[ 541.857943] [<ffffffff8116f600>] writeback_sb_inodes+0x17f/0x229
[ 541.865229] [<ffffffff8116fbe5>] __writeback_inodes_wb+0x78/0xb9
[ 541.872501] [<ffffffff8116fd63>] wb_writeback+0x13d/0x23a
[ 541.879086] [<ffffffff8117018e>] wb_do_writeback+0x19c/0x1b7
[ 541.885979] [<ffffffff81170235>] bdi_writeback_thread+0x8c/0x215
[ 541.893254] [<ffffffff811701a9>] ? wb_do_writeback+0x1b7/0x1b7
[ 541.900339] [<ffffffff8109445c>] kthread+0x8e/0x96
[ 541.906233] [<ffffffff819a7084>] kernel_thread_helper+0x4/0x10
[ 541.913305] [<ffffffff8199e474>] ? retint_restore_args+0x13/0x13
[ 541.920596] [<ffffffff810943ce>] ? __init_kthread_worker+0x5b/0x5b
[ 541.928083] [<ffffffff819a7080>] ? gs_change+0x13/0x13
[ 541.934349] Code: 89 c2 4c 89 85 38 ff ff ff 48 8d 4d ec 41 0f b7 74 24 48 e8 15 4b 7a 00 4c 8b 85 38 ff ff ff 48 85 c0 74 50 48 8b 93 a0 00 00 00
[ 541.953075] 8b 52 28 8b 52 10 89 50 0c 48 8b 93 80 00 00 00 48 89 50 10
[ 541.963605] RIP [<ffffffff811ff00b>] perf_trace_ext4_mb_release_group_pa+0x81/0xd6
[ 541.973102] RSP <ffff88041e901540>
[ 541.977465] CR2: 0000000000000028
[ 541.981703] ---[ end trace 606734373157fadb ]---
In another case fat/thresh=1M/ext4:wb-10dd-1-3.2.0-rc5-ioless-full+:
[ 404.171336] BUG: unable to handle kernel NULL pointer dereference at 0000000000000178
[ 404.171946] IP: [<ffffffff810a5092>] __lock_acquire+0x8b/0x932
[ 404.172340] PGD a3dbb067 PUD a3dba067 PMD 0
[ 404.172747] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
[ 404.173150] CPU 2
[ 404.173249] Modules linked in:
[ 404.173690]
[ 404.173896] Pid: 4435, comm: dd Not tainted 3.2.0-rc5-ioless-full+ #989 /DX58SO
[ 404.174563] RIP: 0010:[<ffffffff810a5092>] [<ffffffff810a5092>] __lock_acquire+0x8b/0x932
[ 404.175143] RSP: 0018:ffff8800b37238d8 EFLAGS: 00010097
[ 404.175481] RAX: 0000000000000000 RBX: 0000000000000170 RCX: 0000000000000000
[ 404.175877] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000170
[ 404.176279] RBP: ffff8800b3723948 R08: 0000000000000000 R09: 0000000000000000
[ 404.176686] R10: 0000000000000170 R11: ffffffff81175d30 R12: 0000000000000000
[ 404.177085] R13: 0000000000000000 R14: ffff8800b18ac540 R15: 0000000000000000
[ 404.177488] FS: 00007f27ad959700(0000) GS:ffff8800b8000000(0000) knlGS:0000000000000000
[ 404.178038] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 404.178390] CR2: 0000000000000178 CR3: 00000000a3db8000 CR4: 00000000000006e0
[ 404.178788] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 404.179187] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 404.179585] Process dd (pid: 4435, threadinfo ffff8800b3722000, task ffff8800b18ac540)
[ 404.180116] Stack:
[ 404.180370] ffff8800b37238e8 ffffffff8103c1fb ffff8800b37238f8 0000000000000002
[ 404.181051] 0000000000000000 0000000000008010 ffff8800be806c00 ffff8800a549db60
[ 404.181707] ffff8800b3723978 0000000000000000 0000000000000170 0000000000000000
[ 404.182370] Call Trace:
[ 404.182604] [<ffffffff8103c1fb>] ? native_sched_clock+0x29/0x70
[ 404.182962] [<ffffffff810a5a16>] lock_acquire+0xdd/0x10a
[ 404.183298] [<ffffffff81175d30>] ? create_empty_buffers+0x4a/0xc1
[ 404.183667] [<ffffffff8199f563>] _raw_spin_lock+0x36/0x69
[ 404.184013] [<ffffffff81175d30>] ? create_empty_buffers+0x4a/0xc1
[ 404.184384] [<ffffffff81175d30>] create_empty_buffers+0x4a/0xc1
[ 404.184752] [<ffffffff811efc7b>] ext4_discard_partial_page_buffers_no_lock+0x9f/0x406
[ 404.185302] [<ffffffff8199ff2b>] ? _raw_spin_unlock+0x2b/0x2f
[ 404.185659] [<ffffffff81170b72>] ? __mark_inode_dirty+0x1ac/0x1cc
[ 404.186027] [<ffffffff8117673f>] ? generic_write_end+0x6d/0x7f
[ 404.186388] [<ffffffff811f1531>] ext4_da_write_end+0x244/0x2ed
[ 404.186749] [<ffffffff810ffeec>] generic_file_buffered_write+0x183/0x22d
[ 404.187142] [<ffffffff8107946a>] ? current_fs_time+0x27/0x2e
[ 404.187495] [<ffffffff8110198c>] __generic_file_aio_write+0x334/0x364
[ 404.187875] [<ffffffff8199e49c>] ? mutex_lock_nested+0x2e2/0x2f1
[ 404.188240] [<ffffffff81101a06>] ? generic_file_aio_write+0x4a/0xc1
[ 404.188623] [<ffffffff81101a22>] generic_file_aio_write+0x66/0xc1
[ 404.188999] [<ffffffff8106787f>] ? finish_task_switch+0x8a/0xf7
[ 404.189374] [<ffffffff811e9f6c>] ext4_file_write+0x1f9/0x251
[ 404.189736] [<ffffffff8118175a>] ? fsnotify+0x216/0x26f
[ 404.190084] [<ffffffff8114d3aa>] do_sync_write+0xce/0x10b
[ 404.190436] [<ffffffff8118175a>] ? fsnotify+0x216/0x26f
[ 404.190777] [<ffffffff811815ba>] ? fsnotify+0x76/0x26f
[ 404.191110] [<ffffffff8114db67>] vfs_write+0xb8/0x157
[ 404.191451] [<ffffffff8114de1e>] sys_write+0x4d/0x77
[ 404.191790] [<ffffffff819a6b42>] system_call_fastpath+0x16/0x1b
[ 404.192162] Code: bd 08 00 00 be d5 0b 00 00 48 c7 c7 76 41 d3 81 83 3d 82 d2 9f 01 00 0f 85 a4 08 00 00 e9 bb 03 00 00 41 83 fc 01 77 13 44 89 e0 <4c> 8b 6c c3 08 4d 85 ed 0f 85 5b 03 00 00 eb 34 41 83 fc 07 76
[ 404.195454] RIP [<ffffffff810a5092>] __lock_acquire+0x8b/0x932
[ 404.195857] RSP <ffff8800b37238d8>
[ 404.196135] CR2: 0000000000000178
[ 404.196407] ---[ end trace 1a4a260a8830abb5 ]---
Thanks,
Fengguang
On Sun, Dec 18, 2011 at 01:53:59PM +0800, Wu Fengguang wrote:
> This is very reproducible. It occurs in two of my test cases, with
> different panic messages.
>
> It also shows up in 3.2.0-rc3. Kernel config is attached.
Oh well, it happens even in 3.1. Here is the dmesg:
[ 210.758668] ------------[ cut here ]------------
[ 210.764665] WARNING: at /c/wfg/linux/kernel/lockdep.c:690 __lock_acquire+0x155/0x938()
[ 210.774369] Hardware name: SandyBridge Platform
[ 210.779898] Modules linked in:
[ 210.783852] Pid: 4409, comm: dd Not tainted 3.1.0+ #299
[ 210.790158] Call Trace:
[ 210.793359] [<ffffffff810734ec>] warn_slowpath_common+0x85/0x9d
[ 210.800538] [<ffffffff8107351e>] warn_slowpath_null+0x1a/0x1c
[ 210.807523] [<ffffffff810a1900>] __lock_acquire+0x155/0x938
[ 210.814312] [<ffffffff810a21c0>] lock_acquire+0xdd/0x10a
[ 210.820815] [<ffffffff81063568>] ? double_rq_lock+0x45/0x57
[ 210.827607] [<ffffffff8199a23b>] _raw_spin_lock+0x36/0x69
[ 210.834145] [<ffffffff81063568>] ? double_rq_lock+0x45/0x57
[ 210.840927] [<ffffffff81063568>] double_rq_lock+0x45/0x57
[ 210.847528] [<ffffffff8106d57c>] load_balance+0x1f1/0x69b
[ 210.854085] [<ffffffff810a1d12>] ? __lock_acquire+0x567/0x938
[ 210.861076] [<ffffffff810a1265>] ? lock_release_holdtime+0xa3/0xac
[ 210.868548] [<ffffffff81997390>] __schedule+0x39a/0x937
[ 210.874952] [<ffffffff81997309>] ? __schedule+0x313/0x937
[ 210.881556] [<ffffffff810a1d12>] ? __lock_acquire+0x567/0x938
[ 210.888548] [<ffffffff8103bebe>] ? native_sched_clock+0x2d/0x5f
[ 210.895733] [<ffffffff81998180>] ? schedule_timeout+0x242/0x274
[ 210.902904] [<ffffffff81096611>] ? local_clock+0x41/0x5a
[ 210.909395] [<ffffffff810a1265>] ? lock_release_holdtime+0xa3/0xac
[ 210.916814] [<ffffffff81998180>] ? schedule_timeout+0x242/0x274
[ 210.923995] [<ffffffff81997beb>] schedule+0x5a/0x5c
[ 210.929943] [<ffffffff81998185>] schedule_timeout+0x247/0x274
[ 210.936933] [<ffffffff810812e0>] ? del_timer+0xc8/0xc8
[ 210.943246] [<ffffffff81997ed4>] io_schedule_timeout+0x98/0xe9
[ 210.950271] [<ffffffff81102e41>] balance_dirty_pages_ratelimited_nr+0x4d2/0x686
[ 210.959399] [<ffffffff810f9354>] generic_file_buffered_write+0x1c9/0x22d
[ 210.967400] [<ffffffff810782c2>] ? current_fs_time+0x27/0x2e
[ 210.974295] [<ffffffff810fadbd>] __generic_file_aio_write+0x32e/0x35e
[ 210.981993] [<ffffffff810fae53>] generic_file_aio_write+0x66/0xc1
[ 210.989370] [<ffffffff811deb83>] ext4_file_write+0x1f6/0x24f
[ 210.996212] [<ffffffff8103bebe>] ? native_sched_clock+0x2d/0x5f
[ 211.003399] [<ffffffff8117729a>] ? fsnotify+0x216/0x26f
[ 211.009805] [<ffffffff81096611>] ? local_clock+0x41/0x5a
[ 211.016310] [<ffffffff81144e86>] do_sync_write+0xcb/0x108
[ 211.022918] [<ffffffff81145550>] vfs_write+0xb8/0x157
[ 211.029125] [<ffffffff81145807>] sys_write+0x4d/0x77
[ 211.035228] [<ffffffff819a1802>] system_call_fastpath+0x16/0x1b
[ 211.042370] ---[ end trace a682372b42c1df33 ]---
[ 443.558240] BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
[ 443.567910] IP: [<ffffffff811f2e62>] perf_trace_ext4_mb_release_group_pa+0x81/0xd6
[ 443.577206] PGD 80e790067 PUD 8133be067 PMD 0
[ 443.582895] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
[ 443.588602] CPU 11
[ 443.590821] Modules linked in:
[ 443.595323]
[ 443.597439] Pid: 4252, comm: flush-8:16 Tainted: G W 3.1.0+ #299 Intel Corporation SandyBridge Platform/To be filled by O.E.M.
[ 443.612154] RIP: 0010:[<ffffffff811f2e62>] [<ffffffff811f2e62>] perf_trace_ext4_mb_release_group_pa+0x81/0xd6
[ 443.624266] RSP: 0018:ffff88041eb695d0 EFLAGS: 00010282
[ 443.630679] RAX: ffffe8fffc407380 RBX: ffff880815c061f8 RCX: ffff88041d9e4500
[ 443.639117] RDX: 0000000000000000 RSI: 0000000000000282 RDI: ffffe8fffc407380
[ 443.647564] RBP: ffff88041eb696b0 R08: ffff88041eb695f0 R09: 0000000000000000
[ 443.656006] R10: ffffe8fffc006ea0 R11: ffffffff8120745f R12: ffffffff81f07698
[ 443.664382] R13: ffff88041eb69760 R14: ffff88081f709750 R15: ffff88080a1a29d8
[ 443.672828] FS: 0000000000000000(0000) GS:ffff88082b400000(0000) knlGS:0000000000000000
[ 443.682722] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 443.689600] CR2: 0000000000000028 CR3: 00000008133bf000 CR4: 00000000000406e0
[ 443.697984] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 443.706430] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 443.714820] Process flush-8:16 (pid: 4252, threadinfo ffff88041eb68000, task ffff88041d9e45e0)
[ 443.725298] Stack:
[ 443.728009] 0000000000000000 0000000000000004 ffff88041eb69660 ffff88041eb695f0
[ 443.737474] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 443.746928] ffff88041eb69700 0000000000000000 0000000000000000 0000000000000000
[ 443.756380] Call Trace:
[ 443.759585] [<ffffffff81204abe>] ? ext4_mb_release_group_pa+0x40/0xfe
[ 443.767347] [<ffffffff81204abe>] ext4_mb_release_group_pa+0x40/0xfe
[ 443.774916] [<ffffffff8120749f>] ext4_mb_discard_group_preallocations+0x355/0x3eb
[ 443.784233] [<ffffffff8120a3a5>] ext4_mb_new_blocks+0x2de/0x3f4
[ 443.791414] [<ffffffff812021c8>] ext4_ext_map_blocks+0xf63/0x1331
[ 443.798796] [<ffffffff811e2810>] ? ext4_map_blocks+0xf5/0x1fa
[ 443.805782] [<ffffffff811e2841>] ext4_map_blocks+0x126/0x1fa
[ 443.812671] [<ffffffff811e5d56>] mpage_da_map_and_submit+0xec/0x3b2
[ 443.820241] [<ffffffff811e6742>] ext4_da_writepages+0x316/0x4b0
[ 443.827429] [<ffffffff810a3359>] ? lock_release_non_nested+0x22c/0x293
[ 443.835299] [<ffffffff81103272>] do_writepages+0x24/0x2d
[ 443.851241] [<ffffffff811675c9>] writeback_single_inode+0x135/0x2c3
[ 443.858809] [<ffffffff81167e23>] writeback_sb_inodes+0x182/0x22c
[ 443.866078] [<ffffffff81168371>] __writeback_inodes_wb+0x78/0xb9
[ 443.873355] [<ffffffff8116851f>] wb_writeback+0x16d/0x276
[ 443.879951] [<ffffffff8116897b>] wb_do_writeback+0x1c8/0x1e3
[ 443.886831] [<ffffffff810812e0>] ? del_timer+0xc8/0xc8
[ 443.893136] [<ffffffff81168a22>] bdi_writeback_thread+0x8c/0x215
[ 443.900413] [<ffffffff81168996>] ? wb_do_writeback+0x1e3/0x1e3
[ 443.907490] [<ffffffff81090930>] kthread+0x8e/0x96
[ 443.913355] [<ffffffff819a3a04>] kernel_thread_helper+0x4/0x10
[ 443.920403] [<ffffffff8199ae74>] ? retint_restore_args+0x13/0x13
[ 443.927626] [<ffffffff810908a2>] ? __init_kthread_worker+0x5b/0x5b
[ 443.935099] [<ffffffff819a3a00>] ? gs_change+0x13/0x13
[ 443.941388] Code: 89 c2 4c 89 85 38 ff ff ff 48 8d 4d ec 41 0f b7 74 24 48 e8 3a d6 7a 00 4c 8b 85 38 ff ff ff 48 85 c0 74 50 48 8b 93 a0 00 00 00
[ 443.960015] 8b 52 28 8b 52 10 89 50 0c 48 8b 93 80 00 00 00 48 89 50 10
[ 443.970522] RIP [<ffffffff811f2e62>] perf_trace_ext4_mb_release_group_pa+0x81/0xd6
[ 443.980029] RSP <ffff88041eb695d0>
[ 443.984374] CR2: 0000000000000028
[ 443.988542] BUG: unable to handle kernel
[ 443.988745] ---[ end trace a682372b42c1df34 ]---
[ 443.999169] NULL pointer dereference at 0000000000000028
[ 444.005637] IP: [<ffffffff811f2e62>] perf_trace_ext4_mb_release_group_pa+0x81/0xd6
[ 444.014987] PGD 810faa067 PUD 8110f6067 PMD 0
[ 444.020575] Oops: 0000 [#2] SMP DEBUG_PAGEALLOC
[ 444.026317] CPU 1
[ 444.028441] Modules linked in:
[ 444.032997]
[ 444.035107] Pid: 5480, comm: dd Tainted: G D W 3.1.0+ #299 Intel Corporation SandyBridge Platform/To be filled by O.E.M.
[ 444.049013] RIP: 0010:[<ffffffff811f2e62>] [<ffffffff811f2e62>] perf_trace_ext4_mb_release_group_pa+0x81/0xd6
[ 444.061108] RSP: 0018:ffff880809f252e8 EFLAGS: 00010282
[ 444.067494] RAX: ffffe8fbfd007380 RBX: ffff880811f6f458 RCX: ffff880809862200
[ 444.075922] RDX: 0000000000000000 RSI: 0000000000000282 RDI: ffffe8fbfd007380
[ 444.084349] RBP: ffff880809f253c8 R08: ffff880809f25308 R09: 0000000000000000
[ 444.092776] R10: ffffe8fffd206ea0 R11: ffffffff8120745f R12: ffffffff81f07698
[ 444.101194] R13: ffff880809f25478 R14: ffff88081f709750 R15: ffff880811f42930
[ 444.109607] FS: 00007f2265a23700(0000) GS:ffff88042c000000(0000) knlGS:0000000000000000
[ 444.119477] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 444.126350] CR2: 0000000000000028 CR3: 00000008110d3000 CR4: 00000000000406e0
[ 444.134777] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 444.143205] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 444.151633] Process dd (pid: 5480, threadinfo ffff880809f24000, task ffff8808098622f0)
[ 444.161311] Stack:
[ 444.164001] ffff880809f25358 0000000000000000 ffff88081d13a650 ffff880809f25308
[ 444.173422] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 444.178355] ffff880809f25418 0000000000000000 0000000000000000 0000000000000000
[ 444.178358] Call Trace:
[ 444.178364] [<ffffffff81204abe>] ? ext4_mb_release_group_pa+0x40/0xfe
[ 444.178367] [<ffffffff81204abe>] ext4_mb_release_group_pa+0x40/0xfe
[ 444.178370] [<ffffffff8120749f>] ext4_mb_discard_group_preallocations+0x355/0x3eb
[ 444.178374] [<ffffffff8120a3a5>] ext4_mb_new_blocks+0x2de/0x3f4
[ 444.178378] [<ffffffff812021c8>] ext4_ext_map_blocks+0xf63/0x1331
[ 444.178382] [<ffffffff811e2810>] ? ext4_map_blocks+0xf5/0x1fa
[ 444.178385] [<ffffffff811e2841>] ext4_map_blocks+0x126/0x1fa
[ 444.178388] [<ffffffff811e5d56>] mpage_da_map_and_submit+0xec/0x3b2
[ 444.178391] [<ffffffff811e6742>] ext4_da_writepages+0x316/0x4b0
[ 444.178396] [<ffffffff81103272>] do_writepages+0x24/0x2d
[ 444.178400] [<ffffffff811675c9>] writeback_single_inode+0x135/0x2c3
[ 444.178403] [<ffffffff81167e23>] writeback_sb_inodes+0x182/0x22c
[ 444.178406] [<ffffffff81168371>] __writeback_inodes_wb+0x78/0xb9
[ 444.178409] [<ffffffff81168799>] writeback_inodes_wb+0x62/0x7c
[ 444.178412] [<ffffffff81102d4b>] balance_dirty_pages_ratelimited_nr+0x3dc/0x686
[ 444.178417] [<ffffffff810f9354>] generic_file_buffered_write+0x1c9/0x22d
[ 444.178421] [<ffffffff810782c2>] ? current_fs_time+0x27/0x2e
[ 444.178424] [<ffffffff810fadbd>] __generic_file_aio_write+0x32e/0x35e
[ 444.178428] [<ffffffff810fae53>] generic_file_aio_write+0x66/0xc1
[ 444.178431] [<ffffffff811deb83>] ext4_file_write+0x1f6/0x24f
[ 444.178435] [<ffffffff8103bebe>] ? native_sched_clock+0x2d/0x5f
[ 444.178439] [<ffffffff8117729a>] ? fsnotify+0x216/0x26f
[ 444.178443] [<ffffffff81096611>] ? local_clock+0x41/0x5a
[ 444.178448] [<ffffffff81144e86>] do_sync_write+0xcb/0x108
[ 444.178451] [<ffffffff81145550>] vfs_write+0xb8/0x157
[ 444.178453] [<ffffffff81145807>] sys_write+0x4d/0x77
[ 444.178456] [<ffffffff819a1802>] system_call_fastpath+0x16/0x1b
[ 444.178458] Code: 89 c2 4c 89 85 38 ff ff ff 48 8d 4d ec 41 0f b7 74 24 48 e8 3a d6 7a 00 4c 8b 85 38 ff ff ff 48 85 c0 74 50 48 8b 93 a0 00 00 00
[ 444.178471] 8b 52 28 8b 52 10 89 50 0c 48 8b 93 80 00 00 00 48 89 50 10
[ 444.178477] RIP [<ffffffff811f2e62>] perf_trace_ext4_mb_release_group_pa+0x81/0xd6
[ 444.178481] RSP <ffff880809f252e8>
[ 444.178482] CR2: 0000000000000028
[ 444.178933] ---[ end trace a682372b42c1df35 ]---
[ 444.178940] ------------[ cut here ]------------
[ 444.178944] WARNING: at /c/wfg/linux/kernel/exit.c:900 do_exit+0x67/0x771()
[ 444.178949] Hardware name: SandyBridge Platform
[ 444.178956] Modules linked in:
[ 444.178962] Pid: 5480, comm: dd Tainted: G D W 3.1.0+ #299
[ 444.178966] Call Trace:
[ 444.178972] [<ffffffff810734ec>] warn_slowpath_common+0x85/0x9d
[ 444.178979] [<ffffffff81074c7a>] ? kmsg_dump+0x8a/0x10c
[ 444.178983] [<ffffffff8107351e>] warn_slowpath_null+0x1a/0x1c
[ 444.178990] [<ffffffff81076bab>] do_exit+0x67/0x771
[ 444.178996] [<ffffffff81074ceb>] ? kmsg_dump+0xfb/0x10c
[ 444.179001] [<ffffffff81074c7a>] ? kmsg_dump+0x8a/0x10c
[ 444.179013] [<ffffffff8199bd4c>] oops_end+0xbe/0xc6
[ 444.179022] [<ffffffff81055e86>] no_context+0x1f9/0x208
[ 444.179029] [<ffffffff8105605e>] __bad_area_nosemaphore+0x1c9/0x1e9
[ 444.179035] [<ffffffff810560dd>] bad_area+0x4a/0x51
[ 444.179041] [<ffffffff8199dece>] do_page_fault+0x27e/0x431
[ 444.179049] [<ffffffff810f810c>] ? perf_output_copy+0x71/0x71
[ 444.179055] [<ffffffff810f5d7f>] ? perf_swevent_overflow+0x90/0xaa
[ 444.179072] [<ffffffff813e985d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 444.179162] [<ffffffff8120745f>] ? ext4_mb_discard_group_preallocations+0x315/0x3eb
[ 444.179174] [<ffffffff8199b085>] page_fault+0x25/0x30
[ 444.179178] [<ffffffff8120745f>] ? ext4_mb_discard_group_preallocations+0x315/0x3eb
[ 444.179222] [<ffffffff811f2e62>] ? perf_trace_ext4_mb_release_group_pa+0x81/0xd6
[ 444.179248] [<ffffffff811f2e4f>] ? perf_trace_ext4_mb_release_group_pa+0x6e/0xd6
[ 444.179261] [<ffffffff81204abe>] ? ext4_mb_release_group_pa+0x40/0xfe
[ 444.179271] [<ffffffff81204abe>] ext4_mb_release_group_pa+0x40/0xfe
[ 444.179276] [<ffffffff8120749f>] ext4_mb_discard_group_preallocations+0x355/0x3eb
[ 444.179282] [<ffffffff8120a3a5>] ext4_mb_new_blocks+0x2de/0x3f4
[ 444.179288] [<ffffffff812021c8>] ext4_ext_map_blocks+0xf63/0x1331
[ 444.179295] [<ffffffff811e2810>] ? ext4_map_blocks+0xf5/0x1fa
[ 444.179300] [<ffffffff811e2841>] ext4_map_blocks+0x126/0x1fa
[ 444.179306] [<ffffffff811e5d56>] mpage_da_map_and_submit+0xec/0x3b2
[ 444.179311] [<ffffffff811e6742>] ext4_da_writepages+0x316/0x4b0
[ 444.179317] [<ffffffff81103272>] do_writepages+0x24/0x2d
[ 444.179477] [<ffffffff811675c9>] writeback_single_inode+0x135/0x2c3
[ 444.179505] [<ffffffff81167e23>] writeback_sb_inodes+0x182/0x22c
[ 444.179515] [<ffffffff81168371>] __writeback_inodes_wb+0x78/0xb9
[ 444.179521] [<ffffffff81168799>] writeback_inodes_wb+0x62/0x7c
[ 444.179526] [<ffffffff81102d4b>] balance_dirty_pages_ratelimited_nr+0x3dc/0x686
[ 444.179532] [<ffffffff810f9354>] generic_file_buffered_write+0x1c9/0x22d
[ 444.179535] [<ffffffff810782c2>] ? current_fs_time+0x27/0x2e
[ 444.179541] [<ffffffff810fadbd>] __generic_file_aio_write+0x32e/0x35e
[ 444.179549] [<ffffffff810fae53>] generic_file_aio_write+0x66/0xc1
[ 444.179555] [<ffffffff811deb83>] ext4_file_write+0x1f6/0x24f
[ 444.179560] [<ffffffff8103bebe>] ? native_sched_clock+0x2d/0x5f
[ 444.179563] [<ffffffff8117729a>] ? fsnotify+0x216/0x26f
[ 444.179569] [<ffffffff81096611>] ? local_clock+0x41/0x5a
[ 444.179577] [<ffffffff81144e86>] do_sync_write+0xcb/0x108
[ 444.179581] [<ffffffff81145550>] vfs_write+0xb8/0x157
[ 444.179586] [<ffffffff81145807>] sys_write+0x4d/0x77
[ 444.179593] [<ffffffff819a1802>] system_call_fastpath+0x16/0x1b
[ 444.179597] ---[ end trace a682372b42c1df36 ]---
[ 444.179888] dd used greatest stack depth: 2088 bytes left
[ 504.787849] INFO: rcu_sched_state detected stall on CPU 14 (t=60000 jiffies)
[ 504.787854] INFO: rcu_sched_state detected stall on CPU 27 (t=60000 jiffies)
[ 539.837210] BUG: spinlock lockup on CPU#14, dd/4796, ffff88081103c180
[ 539.845293] Pid: 4796, comm: dd Tainted: G D W 3.1.0+ #299
[ 539.852582] Call Trace:
[ 539.855792] [<ffffffff813edf90>] do_raw_spin_lock+0x10c/0x137
[ 539.862796] [<ffffffff8199a25b>] _raw_spin_lock+0x56/0x69
[ 539.869412] [<ffffffff81208578>] ? ext4_mb_find_by_goal+0xdb/0x1ce
[ 539.876900] [<ffffffff81208578>] ext4_mb_find_by_goal+0xdb/0x1ce
[ 539.883455] BUG: spinlock lockup on CPU#27, dd/5336, ffff88081103c080
[ 539.883458] Pid: 5336, comm: dd Tainted: G D W 3.1.0+ #299
[ 539.883459] Call Trace:
[ 539.883464] [<ffffffff813edf90>] do_raw_spin_lock+0x10c/0x137
[ 539.883467] [<ffffffff8199a25b>] _raw_spin_lock+0x56/0x69
[ 539.883471] [<ffffffff81208578>] ? ext4_mb_find_by_goal+0xdb/0x1ce
[ 539.883474] [<ffffffff81208578>] ext4_mb_find_by_goal+0xdb/0x1ce
[ 539.883477] [<ffffffff81205856>] ? ext4_mb_normalize_request+0x2f4/0x480
[ 539.883480] [<ffffffff81208c6f>] ext4_mb_regular_allocator+0x59/0x312
[ 539.883484] [<ffffffff8120a25e>] ext4_mb_new_blocks+0x197/0x3f4
[ 539.883487] [<ffffffff811ff836>] ? ext4_ext_find_extent+0x130/0x2fb
[ 539.883490] [<ffffffff8199ac03>] ? _raw_spin_unlock+0x2b/0x2f
[ 539.883493] [<ffffffff812021c8>] ext4_ext_map_blocks+0xf63/0x1331
[ 539.883499] [<ffffffff811e2810>] ? ext4_map_blocks+0xf5/0x1fa
[ 539.883502] [<ffffffff811e2841>] ext4_map_blocks+0x126/0x1fa
[ 539.883504] [<ffffffff811e5d56>] mpage_da_map_and_submit+0xec/0x3b2
[ 539.883507] [<ffffffff811e6742>] ext4_da_writepages+0x316/0x4b0
[ 539.883513] [<ffffffff81103272>] do_writepages+0x24/0x2d
[ 539.883520] [<ffffffff811675c9>] writeback_single_inode+0x135/0x2c3
[ 539.883523] [<ffffffff81167e23>] writeback_sb_inodes+0x182/0x22c
[ 539.883526] [<ffffffff81168371>] __writeback_inodes_wb+0x78/0xb9
[ 539.883528] [<ffffffff81168799>] writeback_inodes_wb+0x62/0x7c
[ 539.883532] [<ffffffff81102d4b>] balance_dirty_pages_ratelimited_nr+0x3dc/0x686
[ 539.883534] [<ffffffff8116809e>] ? __mark_inode_dirty+0x2e/0x1cf
[ 539.883539] [<ffffffff810f9354>] generic_file_buffered_write+0x1c9/0x22d
[ 539.883545] [<ffffffff810782c2>] ? current_fs_time+0x27/0x2e
[ 539.883548] [<ffffffff810fadbd>] __generic_file_aio_write+0x32e/0x35e
[ 539.883552] [<ffffffff810fae53>] generic_file_aio_write+0x66/0xc1
[ 539.883555] [<ffffffff811deb83>] ext4_file_write+0x1f6/0x24f
[ 539.883559] [<ffffffff8117729a>] ? fsnotify+0x216/0x26f
[ 539.883565] [<ffffffff81144e86>] do_sync_write+0xcb/0x108
[ 539.883567] [<ffffffff81145550>] vfs_write+0xb8/0x157
[ 539.883570] [<ffffffff81145807>] sys_write+0x4d/0x77
[ 539.883574] [<ffffffff819a1802>] system_call_fastpath+0x16/0x1b
[ 540.125746] [<ffffffff81205856>] ? ext4_mb_normalize_request+0x2f4/0x480
[ 540.133817] [<ffffffff81208c6f>] ext4_mb_regular_allocator+0x59/0x312
[ 540.133820] [<ffffffff8120a25e>] ext4_mb_new_blocks+0x197/0x3f4
[ 540.133823] [<ffffffff811ff836>] ? ext4_ext_find_extent+0x130/0x2fb
[ 540.133829] [<ffffffff8199ac03>] ? _raw_spin_unlock+0x2b/0x2f
[ 540.133831] [<ffffffff812021c8>] ext4_ext_map_blocks+0xf63/0x1331
[ 540.133838] [<ffffffff811e2810>] ? ext4_map_blocks+0xf5/0x1fa
[ 540.133840] [<ffffffff811e2841>] ext4_map_blocks+0x126/0x1fa
[ 540.133843] [<ffffffff811e5d56>] mpage_da_map_and_submit+0xec/0x3b2
[ 540.133845] [<ffffffff811e6742>] ext4_da_writepages+0x316/0x4b0
[ 540.133854] [<ffffffff81103272>] do_writepages+0x24/0x2d
[ 540.133861] [<ffffffff811675c9>] writeback_single_inode+0x135/0x2c3
[ 540.133863] [<ffffffff81167e23>] writeback_sb_inodes+0x182/0x22c
[ 540.133866] [<ffffffff81168371>] __writeback_inodes_wb+0x78/0xb9
[ 540.133869] [<ffffffff81168799>] writeback_inodes_wb+0x62/0x7c
[ 540.133872] [<ffffffff81102d4b>] balance_dirty_pages_ratelimited_nr+0x3dc/0x686
[ 540.133875] [<ffffffff8116809e>] ? __mark_inode_dirty+0x2e/0x1cf
[ 540.133880] [<ffffffff810f9354>] generic_file_buffered_write+0x1c9/0x22d
[ 540.133886] [<ffffffff810782c2>] ? current_fs_time+0x27/0x2e
[ 540.133889] [<ffffffff810fadbd>] __generic_file_aio_write+0x32e/0x35e
[ 540.133892] [<ffffffff810fae53>] generic_file_aio_write+0x66/0xc1
[ 540.133896] [<ffffffff811deb83>] ext4_file_write+0x1f6/0x24f
[ 540.133900] [<ffffffff8117729a>] ? fsnotify+0x216/0x26f
[ 540.133905] [<ffffffff81144e86>] do_sync_write+0xcb/0x108
[ 540.133908] [<ffffffff81145550>] vfs_write+0xb8/0x157
[ 540.133910] [<ffffffff81145807>] sys_write+0x4d/0x77
[ 540.133914] [<ffffffff819a1802>] system_call_fastpath+0x16/0x1b
[ 684.729866] INFO: rcu_sched_state detected stall on CPU 27 (t=240030 jiffies)
[ 684.730858] INFO: rcu_sched_state detected stall on CPU 14 (t=240030 jiffies)
[ 864.673871] INFO: rcu_sched_state detected stall on CPU 27 (t=420061 jiffies)
[ 864.673877] INFO: rcu_sched_state detected stall on CPU 14 (t=420061 jiffies)
[ 1044.615889] INFO: rcu_sched_state detected stall on CPU 27 (t=600092 jiffies)
[ 1044.616884] INFO: rcu_sched_state detected stall on CPU 14 (t=600092 jiffies)
[ 1224.558902] INFO: rcu_sched_state detected stall on CPU 27 (t=780123 jiffies)
[ 1224.559899] INFO: rcu_sched_state detected stall on CPU 14 (t=780123 jiffies)
[ 1404.501917] INFO: rcu_sched_state detected stall on CPU 14 (t=960154 jiffies)
[ 1404.501922] INFO: rcu_sched_state detected stall on CPU 27 (t=960154 jiffies)
[ 1584.443929] INFO: rcu_sched_state detected stall on CPU 27 (t=1140184 jiffies)
[ 1584.443935] INFO: rcu_sched_state detected stall on CPU 14 (t=1140184 jiffies)
[ 1764.386941] INFO: rcu_sched_state detected stall on CPU 14 (t=1320214 jiffies)
[ 1764.386944] INFO: rcu_sched_state detected stall on CPU 27 (t=1320215 jiffies)
[ 1944.328956] INFO: rcu_sched_state detected stall on CPU 14 (t=1500245 jiffies)
[ 1944.328960] INFO: rcu_sched_state detected stall on CPU 27 (t=1500245 jiffies)
[ 2124.270971] INFO: rcu_sched_state detected stall on CPU 14 (t=1680275 jiffies)
[ 2124.270976] INFO: rcu_sched_state detected stall on CPU 27 (t=1680275 jiffies)
[ 2304.212984] INFO: rcu_sched_state detected stall on CPU 14 (t=1860305 jiffies)
[ 2304.213983] INFO: rcu_sched_state detected stall on CPU 27 (t=1860306 jiffies)
[ 2484.155998] INFO: rcu_sched_state detected stall on CPU 27 (t=2040336 jiffies)
[ 2484.156998] INFO: rcu_sched_state detected stall on CPU 14 (t=2040337 jiffies)
Hi Fengguang,
Could you try the patch [ext4: do not reference pa_inode from group_pa]?
Thanks,
Yongqiang.
On Sun, Dec 18, 2011 at 2:30 PM, Wu Fengguang <[email protected]> wrote:
> On Sun, Dec 18, 2011 at 01:53:59PM +0800, Wu Fengguang wrote:
>> This is very reproducible. It occurs in two of my test cases, with
>> different panic messages.
>>
>> It also shows up in 3.2.0-rc3. Kernel config is attached.
>
> Oh well, it happens even in 3.1. Here is the dmesg:
>
> [ ?210.758668] ------------[ cut here ]------------
> [ ?210.764665] WARNING: at /c/wfg/linux/kernel/lockdep.c:690 __lock_acquire+0x155/0x938()
> [ ?210.774369] Hardware name: SandyBridge Platform
> [ ?210.779898] Modules linked in:
> [ ?210.783852] Pid: 4409, comm: dd Not tainted 3.1.0+ #299
> [ ?210.790158] Call Trace:
> [ ?210.793359] ?[<ffffffff810734ec>] warn_slowpath_common+0x85/0x9d
> [ ?210.800538] ?[<ffffffff8107351e>] warn_slowpath_null+0x1a/0x1c
> [ ?210.807523] ?[<ffffffff810a1900>] __lock_acquire+0x155/0x938
> [ ?210.814312] ?[<ffffffff810a21c0>] lock_acquire+0xdd/0x10a
> [ ?210.820815] ?[<ffffffff81063568>] ? double_rq_lock+0x45/0x57
> [ ?210.827607] ?[<ffffffff8199a23b>] _raw_spin_lock+0x36/0x69
> [ ?210.834145] ?[<ffffffff81063568>] ? double_rq_lock+0x45/0x57
> [ ?210.840927] ?[<ffffffff81063568>] double_rq_lock+0x45/0x57
> [ ?210.847528] ?[<ffffffff8106d57c>] load_balance+0x1f1/0x69b
> [ ?210.854085] ?[<ffffffff810a1d12>] ? __lock_acquire+0x567/0x938
> [ ?210.861076] ?[<ffffffff810a1265>] ? lock_release_holdtime+0xa3/0xac
> [ ?210.868548] ?[<ffffffff81997390>] __schedule+0x39a/0x937
> [ ?210.874952] ?[<ffffffff81997309>] ? __schedule+0x313/0x937
> [ ?210.881556] ?[<ffffffff810a1d12>] ? __lock_acquire+0x567/0x938
> [ ?210.888548] ?[<ffffffff8103bebe>] ? native_sched_clock+0x2d/0x5f
> [ ?210.895733] ?[<ffffffff81998180>] ? schedule_timeout+0x242/0x274
> [ ?210.902904] ?[<ffffffff81096611>] ? local_clock+0x41/0x5a
> [ ?210.909395] ?[<ffffffff810a1265>] ? lock_release_holdtime+0xa3/0xac
> [ ?210.916814] ?[<ffffffff81998180>] ? schedule_timeout+0x242/0x274
> [ ?210.923995] ?[<ffffffff81997beb>] schedule+0x5a/0x5c
> [ ?210.929943] ?[<ffffffff81998185>] schedule_timeout+0x247/0x274
> [ ?210.936933] ?[<ffffffff810812e0>] ? del_timer+0xc8/0xc8
> [ ?210.943246] ?[<ffffffff81997ed4>] io_schedule_timeout+0x98/0xe9
> [ ?210.950271] ?[<ffffffff81102e41>] balance_dirty_pages_ratelimited_nr+0x4d2/0x686
> [ ?210.959399] ?[<ffffffff810f9354>] generic_file_buffered_write+0x1c9/0x22d
> [ ?210.967400] ?[<ffffffff810782c2>] ? current_fs_time+0x27/0x2e
> [ ?210.974295] ?[<ffffffff810fadbd>] __generic_file_aio_write+0x32e/0x35e
> [ ?210.981993] ?[<ffffffff810fae53>] generic_file_aio_write+0x66/0xc1
> [ ?210.989370] ?[<ffffffff811deb83>] ext4_file_write+0x1f6/0x24f
> [ ?210.996212] ?[<ffffffff8103bebe>] ? native_sched_clock+0x2d/0x5f
> [ ?211.003399] ?[<ffffffff8117729a>] ? fsnotify+0x216/0x26f
> [ ?211.009805] ?[<ffffffff81096611>] ? local_clock+0x41/0x5a
> [ ?211.016310] ?[<ffffffff81144e86>] do_sync_write+0xcb/0x108
> [ ?211.022918] ?[<ffffffff81145550>] vfs_write+0xb8/0x157
> [ ?211.029125] ?[<ffffffff81145807>] sys_write+0x4d/0x77
> [ ?211.035228] ?[<ffffffff819a1802>] system_call_fastpath+0x16/0x1b
> [ ?211.042370] ---[ end trace a682372b42c1df33 ]---
> [ ?443.558240] BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
> [ ?443.567910] IP: [<ffffffff811f2e62>] perf_trace_ext4_mb_release_group_pa+0x81/0xd6
> [ ?443.577206] PGD 80e790067 PUD 8133be067 PMD 0
> [ ?443.582895] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
> [ ?443.588602] CPU 11
> [ ?443.590821] Modules linked in:
> [ ?443.595323]
> [ ?443.597439] Pid: 4252, comm: flush-8:16 Tainted: G ? ? ? ?W ? 3.1.0+ #299 Intel Corporation SandyBridge Platform/To be filled by O.E.M.
> [ ?443.612154] RIP: 0010:[<ffffffff811f2e62>] ?[<ffffffff811f2e62>] perf_trace_ext4_mb_release_group_pa+0x81/0xd6
> [ ?443.624266] RSP: 0018:ffff88041eb695d0 ?EFLAGS: 00010282
> [ ?443.630679] RAX: ffffe8fffc407380 RBX: ffff880815c061f8 RCX: ffff88041d9e4500
> [ ?443.639117] RDX: 0000000000000000 RSI: 0000000000000282 RDI: ffffe8fffc407380
> [ ?443.647564] RBP: ffff88041eb696b0 R08: ffff88041eb695f0 R09: 0000000000000000
> [ ?443.656006] R10: ffffe8fffc006ea0 R11: ffffffff8120745f R12: ffffffff81f07698
> [ ?443.664382] R13: ffff88041eb69760 R14: ffff88081f709750 R15: ffff88080a1a29d8
> [ ?443.672828] FS: ?0000000000000000(0000) GS:ffff88082b400000(0000) knlGS:0000000000000000
> [ ?443.682722] CS: ?0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ ?443.689600] CR2: 0000000000000028 CR3: 00000008133bf000 CR4: 00000000000406e0
> [ ?443.697984] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ ?443.706430] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ ?443.714820] Process flush-8:16 (pid: 4252, threadinfo ffff88041eb68000, task ffff88041d9e45e0)
> [ ?443.725298] Stack:
> [ ?443.728009] ?0000000000000000 0000000000000004 ffff88041eb69660 ffff88041eb695f0
> [ ?443.737474] ?0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [ ?443.746928] ?ffff88041eb69700 0000000000000000 0000000000000000 0000000000000000
> [ ?443.756380] Call Trace:
> [ ?443.759585] ?[<ffffffff81204abe>] ? ext4_mb_release_group_pa+0x40/0xfe
> [ ?443.767347] ?[<ffffffff81204abe>] ext4_mb_release_group_pa+0x40/0xfe
> [ ?443.774916] ?[<ffffffff8120749f>] ext4_mb_discard_group_preallocations+0x355/0x3eb
> [ ?443.784233] ?[<ffffffff8120a3a5>] ext4_mb_new_blocks+0x2de/0x3f4
> [ ?443.791414] ?[<ffffffff812021c8>] ext4_ext_map_blocks+0xf63/0x1331
> [ ?443.798796] ?[<ffffffff811e2810>] ? ext4_map_blocks+0xf5/0x1fa
> [ ?443.805782] ?[<ffffffff811e2841>] ext4_map_blocks+0x126/0x1fa
> [ ?443.812671] ?[<ffffffff811e5d56>] mpage_da_map_and_submit+0xec/0x3b2
> [ ?443.820241] ?[<ffffffff811e6742>] ext4_da_writepages+0x316/0x4b0
> [ ?443.827429] ?[<ffffffff810a3359>] ? lock_release_non_nested+0x22c/0x293
> [ ?443.835299] ?[<ffffffff81103272>] do_writepages+0x24/0x2d
> [ ?443.851241] ?[<ffffffff811675c9>] writeback_single_inode+0x135/0x2c3
> [ ?443.858809] ?[<ffffffff81167e23>] writeback_sb_inodes+0x182/0x22c
> [ ?443.866078] ?[<ffffffff81168371>] __writeback_inodes_wb+0x78/0xb9
> [ ?443.873355] ?[<ffffffff8116851f>] wb_writeback+0x16d/0x276
> [ ?443.879951] ?[<ffffffff8116897b>] wb_do_writeback+0x1c8/0x1e3
> [ ?443.886831] ?[<ffffffff810812e0>] ? del_timer+0xc8/0xc8
> [ ?443.893136] ?[<ffffffff81168a22>] bdi_writeback_thread+0x8c/0x215
> [ ?443.900413] ?[<ffffffff81168996>] ? wb_do_writeback+0x1e3/0x1e3
> [ ?443.907490] ?[<ffffffff81090930>] kthread+0x8e/0x96
> [ ?443.913355] ?[<ffffffff819a3a04>] kernel_thread_helper+0x4/0x10
> [ ?443.920403] ?[<ffffffff8199ae74>] ? retint_restore_args+0x13/0x13
> [ ?443.927626] ?[<ffffffff810908a2>] ? __init_kthread_worker+0x5b/0x5b
> [ ?443.935099] ?[<ffffffff819a3a00>] ? gs_change+0x13/0x13
> [ ?443.941388] Code: 89 c2 4c 89 85 38 ff ff ff 48 8d 4d ec 41 0f b7 74 24 48 e8 3a d6 7a 00 4c 8b 85 38 ff ff ff 48 85 c0 74 50 48 8b 93 a0 00 00 00
> [ ?443.960015] ?8b 52 28 8b 52 10 89 50 0c 48 8b 93 80 00 00 00 48 89 50 10
> [ ?443.970522] RIP ?[<ffffffff811f2e62>] perf_trace_ext4_mb_release_group_pa+0x81/0xd6
> [ ?443.980029] ?RSP <ffff88041eb695d0>
> [ ?443.984374] CR2: 0000000000000028
> [ ?443.988542] BUG: unable to handle kernel
> [ ?443.988745] ---[ end trace a682372b42c1df34 ]---
> [ ?443.999169] NULL pointer dereference at 0000000000000028
> [ ?444.005637] IP: [<ffffffff811f2e62>] perf_trace_ext4_mb_release_group_pa+0x81/0xd6
> [ ?444.014987] PGD 810faa067 PUD 8110f6067 PMD 0
> [ ?444.020575] Oops: 0000 [#2] SMP DEBUG_PAGEALLOC
> [ ?444.026317] CPU 1
> [ ?444.028441] Modules linked in:
> [ ?444.032997]
> [ ?444.035107] Pid: 5480, comm: dd Tainted: G ? ? ?D W ? 3.1.0+ #299 Intel Corporation SandyBridge Platform/To be filled by O.E.M.
> [ ?444.049013] RIP: 0010:[<ffffffff811f2e62>] ?[<ffffffff811f2e62>] perf_trace_ext4_mb_release_group_pa+0x81/0xd6
> [ ?444.061108] RSP: 0018:ffff880809f252e8 ?EFLAGS: 00010282
> [ ?444.067494] RAX: ffffe8fbfd007380 RBX: ffff880811f6f458 RCX: ffff880809862200
> [ ?444.075922] RDX: 0000000000000000 RSI: 0000000000000282 RDI: ffffe8fbfd007380
> [ ?444.084349] RBP: ffff880809f253c8 R08: ffff880809f25308 R09: 0000000000000000
> [ ?444.092776] R10: ffffe8fffd206ea0 R11: ffffffff8120745f R12: ffffffff81f07698
> [ ?444.101194] R13: ffff880809f25478 R14: ffff88081f709750 R15: ffff880811f42930
> [ ?444.109607] FS: ?00007f2265a23700(0000) GS:ffff88042c000000(0000) knlGS:0000000000000000
> [ ?444.119477] CS: ?0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ ?444.126350] CR2: 0000000000000028 CR3: 00000008110d3000 CR4: 00000000000406e0
> [ ?444.134777] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ ?444.143205] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ ?444.151633] Process dd (pid: 5480, threadinfo ffff880809f24000, task ffff8808098622f0)
> [ ?444.161311] Stack:
> [ ?444.164001] ?ffff880809f25358 0000000000000000 ffff88081d13a650 ffff880809f25308
> [ ?444.173422] ?0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [ ?444.178355] ?ffff880809f25418 0000000000000000 0000000000000000 0000000000000000
> [ ?444.178358] Call Trace:
> [ ?444.178364] ?[<ffffffff81204abe>] ? ext4_mb_release_group_pa+0x40/0xfe
> [ ?444.178367] ?[<ffffffff81204abe>] ext4_mb_release_group_pa+0x40/0xfe
> [ ?444.178370] ?[<ffffffff8120749f>] ext4_mb_discard_group_preallocations+0x355/0x3eb
> [ ?444.178374] ?[<ffffffff8120a3a5>] ext4_mb_new_blocks+0x2de/0x3f4
> [ ?444.178378] ?[<ffffffff812021c8>] ext4_ext_map_blocks+0xf63/0x1331
> [ ?444.178382] ?[<ffffffff811e2810>] ? ext4_map_blocks+0xf5/0x1fa
> [ ?444.178385] ?[<ffffffff811e2841>] ext4_map_blocks+0x126/0x1fa
> [ ?444.178388] ?[<ffffffff811e5d56>] mpage_da_map_and_submit+0xec/0x3b2
> [ ?444.178391] ?[<ffffffff811e6742>] ext4_da_writepages+0x316/0x4b0
> [ ?444.178396] ?[<ffffffff81103272>] do_writepages+0x24/0x2d
> [ ?444.178400] ?[<ffffffff811675c9>] writeback_single_inode+0x135/0x2c3
> [ ?444.178403] ?[<ffffffff81167e23>] writeback_sb_inodes+0x182/0x22c
> [ ?444.178406] ?[<ffffffff81168371>] __writeback_inodes_wb+0x78/0xb9
> [ ?444.178409] ?[<ffffffff81168799>] writeback_inodes_wb+0x62/0x7c
> [ ?444.178412] ?[<ffffffff81102d4b>] balance_dirty_pages_ratelimited_nr+0x3dc/0x686
> [ ?444.178417] ?[<ffffffff810f9354>] generic_file_buffered_write+0x1c9/0x22d
> [ ?444.178421] ?[<ffffffff810782c2>] ? current_fs_time+0x27/0x2e
> [ ?444.178424] ?[<ffffffff810fadbd>] __generic_file_aio_write+0x32e/0x35e
> [ ?444.178428] ?[<ffffffff810fae53>] generic_file_aio_write+0x66/0xc1
> [ ?444.178431] ?[<ffffffff811deb83>] ext4_file_write+0x1f6/0x24f
> [ ?444.178435] ?[<ffffffff8103bebe>] ? native_sched_clock+0x2d/0x5f
> [ ?444.178439] ?[<ffffffff8117729a>] ? fsnotify+0x216/0x26f
> [ ?444.178443] ?[<ffffffff81096611>] ? local_clock+0x41/0x5a
> [ ?444.178448] ?[<ffffffff81144e86>] do_sync_write+0xcb/0x108
> [ ?444.178451] ?[<ffffffff81145550>] vfs_write+0xb8/0x157
> [ ?444.178453] ?[<ffffffff81145807>] sys_write+0x4d/0x77
> [ ?444.178456] ?[<ffffffff819a1802>] system_call_fastpath+0x16/0x1b
> [ ?444.178458] Code: 89 c2 4c 89 85 38 ff ff ff 48 8d 4d ec 41 0f b7 74 24 48 e8 3a d6 7a 00 4c 8b 85 38 ff ff ff 48 85 c0 74 50 48 8b 93 a0 00 00 00
> [ ?444.178471] ?8b 52 28 8b 52 10 89 50 0c 48 8b 93 80 00 00 00 48 89 50 10
> [ ?444.178477] RIP ?[<ffffffff811f2e62>] perf_trace_ext4_mb_release_group_pa+0x81/0xd6
> [ ?444.178481] ?RSP <ffff880809f252e8>
> [ ?444.178482] CR2: 0000000000000028
> [ ?444.178933] ---[ end trace a682372b42c1df35 ]---
> [ ?444.178940] ------------[ cut here ]------------
> [ ?444.178944] WARNING: at /c/wfg/linux/kernel/exit.c:900 do_exit+0x67/0x771()
> [ ?444.178949] Hardware name: SandyBridge Platform
> [ ?444.178956] Modules linked in:
> [ ?444.178962] Pid: 5480, comm: dd Tainted: G ? ? ?D W ? 3.1.0+ #299
> [ ?444.178966] Call Trace:
> [ ?444.178972] ?[<ffffffff810734ec>] warn_slowpath_common+0x85/0x9d
> [ ?444.178979] ?[<ffffffff81074c7a>] ? kmsg_dump+0x8a/0x10c
> [ ?444.178983] ?[<ffffffff8107351e>] warn_slowpath_null+0x1a/0x1c
> [ ?444.178990] ?[<ffffffff81076bab>] do_exit+0x67/0x771
> [ ?444.178996] ?[<ffffffff81074ceb>] ? kmsg_dump+0xfb/0x10c
> [ ?444.179001] ?[<ffffffff81074c7a>] ? kmsg_dump+0x8a/0x10c
> [ ?444.179013] ?[<ffffffff8199bd4c>] oops_end+0xbe/0xc6
> [ ?444.179022] ?[<ffffffff81055e86>] no_context+0x1f9/0x208
> [ ?444.179029] ?[<ffffffff8105605e>] __bad_area_nosemaphore+0x1c9/0x1e9
> [ ?444.179035] ?[<ffffffff810560dd>] bad_area+0x4a/0x51
> [ ?444.179041] ?[<ffffffff8199dece>] do_page_fault+0x27e/0x431
> [ ?444.179049] ?[<ffffffff810f810c>] ? perf_output_copy+0x71/0x71
> [ ?444.179055] ?[<ffffffff810f5d7f>] ? perf_swevent_overflow+0x90/0xaa
> [ ?444.179072] ?[<ffffffff813e985d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
> [ ?444.179162] ?[<ffffffff8120745f>] ? ext4_mb_discard_group_preallocations+0x315/0x3eb
> [ ?444.179174] ?[<ffffffff8199b085>] page_fault+0x25/0x30
> [ ?444.179178] ?[<ffffffff8120745f>] ? ext4_mb_discard_group_preallocations+0x315/0x3eb
> [ ?444.179222] ?[<ffffffff811f2e62>] ? perf_trace_ext4_mb_release_group_pa+0x81/0xd6
> [ ?444.179248] ?[<ffffffff811f2e4f>] ? perf_trace_ext4_mb_release_group_pa+0x6e/0xd6
> [ ?444.179261] ?[<ffffffff81204abe>] ? ext4_mb_release_group_pa+0x40/0xfe
> [ ?444.179271] ?[<ffffffff81204abe>] ext4_mb_release_group_pa+0x40/0xfe
> [ ?444.179276] ?[<ffffffff8120749f>] ext4_mb_discard_group_preallocations+0x355/0x3eb
> [ ?444.179282] ?[<ffffffff8120a3a5>] ext4_mb_new_blocks+0x2de/0x3f4
> [ ?444.179288] ?[<ffffffff812021c8>] ext4_ext_map_blocks+0xf63/0x1331
> [ ?444.179295] ?[<ffffffff811e2810>] ? ext4_map_blocks+0xf5/0x1fa
> [ ?444.179300] ?[<ffffffff811e2841>] ext4_map_blocks+0x126/0x1fa
> [ ?444.179306] ?[<ffffffff811e5d56>] mpage_da_map_and_submit+0xec/0x3b2
> [ ?444.179311] ?[<ffffffff811e6742>] ext4_da_writepages+0x316/0x4b0
> [ ?444.179317] ?[<ffffffff81103272>] do_writepages+0x24/0x2d
> [ ?444.179477] ?[<ffffffff811675c9>] writeback_single_inode+0x135/0x2c3
> [ ?444.179505] ?[<ffffffff81167e23>] writeback_sb_inodes+0x182/0x22c
> [ ?444.179515] ?[<ffffffff81168371>] __writeback_inodes_wb+0x78/0xb9
> [ ?444.179521] ?[<ffffffff81168799>] writeback_inodes_wb+0x62/0x7c
> [ ?444.179526] ?[<ffffffff81102d4b>] balance_dirty_pages_ratelimited_nr+0x3dc/0x686
> [ ?444.179532] ?[<ffffffff810f9354>] generic_file_buffered_write+0x1c9/0x22d
> [ ?444.179535] ?[<ffffffff810782c2>] ? current_fs_time+0x27/0x2e
> [ ?444.179541] ?[<ffffffff810fadbd>] __generic_file_aio_write+0x32e/0x35e
> [ ?444.179549] ?[<ffffffff810fae53>] generic_file_aio_write+0x66/0xc1
> [ ?444.179555] ?[<ffffffff811deb83>] ext4_file_write+0x1f6/0x24f
> [ ?444.179560] ?[<ffffffff8103bebe>] ? native_sched_clock+0x2d/0x5f
> [ ?444.179563] ?[<ffffffff8117729a>] ? fsnotify+0x216/0x26f
> [ ?444.179569] ?[<ffffffff81096611>] ? local_clock+0x41/0x5a
> [ ?444.179577] ?[<ffffffff81144e86>] do_sync_write+0xcb/0x108
> [ ?444.179581] ?[<ffffffff81145550>] vfs_write+0xb8/0x157
> [ ?444.179586] ?[<ffffffff81145807>] sys_write+0x4d/0x77
> [ ?444.179593] ?[<ffffffff819a1802>] system_call_fastpath+0x16/0x1b
> [ ?444.179597] ---[ end trace a682372b42c1df36 ]---
> [ ?444.179888] dd used greatest stack depth: 2088 bytes left
> [ ?504.787849] INFO: rcu_sched_state detected stall on CPU 14 (t=60000 jiffies)
> [ ?504.787854] INFO: rcu_sched_state detected stall on CPU 27 (t=60000 jiffies)
> [ ?539.837210] BUG: spinlock lockup on CPU#14, dd/4796, ffff88081103c180
> [ ?539.845293] Pid: 4796, comm: dd Tainted: G ? ? ?D W ? 3.1.0+ #299
> [ ?539.852582] Call Trace:
> [ ?539.855792] ?[<ffffffff813edf90>] do_raw_spin_lock+0x10c/0x137
> [ ?539.862796] ?[<ffffffff8199a25b>] _raw_spin_lock+0x56/0x69
> [ ?539.869412] ?[<ffffffff81208578>] ? ext4_mb_find_by_goal+0xdb/0x1ce
> [ ?539.876900] ?[<ffffffff81208578>] ext4_mb_find_by_goal+0xdb/0x1ce
> [ ?539.883455] BUG: spinlock lockup on CPU#27, dd/5336, ffff88081103c080
> [ ?539.883458] Pid: 5336, comm: dd Tainted: G ? ? ?D W ? 3.1.0+ #299
> [ ?539.883459] Call Trace:
> [ ?539.883464] ?[<ffffffff813edf90>] do_raw_spin_lock+0x10c/0x137
> [ ?539.883467] ?[<ffffffff8199a25b>] _raw_spin_lock+0x56/0x69
> [ ?539.883471] ?[<ffffffff81208578>] ? ext4_mb_find_by_goal+0xdb/0x1ce
> [ ?539.883474] ?[<ffffffff81208578>] ext4_mb_find_by_goal+0xdb/0x1ce
> [ ?539.883477] ?[<ffffffff81205856>] ? ext4_mb_normalize_request+0x2f4/0x480
> [ ?539.883480] ?[<ffffffff81208c6f>] ext4_mb_regular_allocator+0x59/0x312
> [ ?539.883484] ?[<ffffffff8120a25e>] ext4_mb_new_blocks+0x197/0x3f4
> [ ?539.883487] ?[<ffffffff811ff836>] ? ext4_ext_find_extent+0x130/0x2fb
> [ ?539.883490] ?[<ffffffff8199ac03>] ? _raw_spin_unlock+0x2b/0x2f
> [ ?539.883493] ?[<ffffffff812021c8>] ext4_ext_map_blocks+0xf63/0x1331
> [ ?539.883499] ?[<ffffffff811e2810>] ? ext4_map_blocks+0xf5/0x1fa
> [ ?539.883502] ?[<ffffffff811e2841>] ext4_map_blocks+0x126/0x1fa
> [ ?539.883504] ?[<ffffffff811e5d56>] mpage_da_map_and_submit+0xec/0x3b2
> [ ?539.883507] ?[<ffffffff811e6742>] ext4_da_writepages+0x316/0x4b0
> [ ?539.883513] ?[<ffffffff81103272>] do_writepages+0x24/0x2d
> [ ?539.883520] ?[<ffffffff811675c9>] writeback_single_inode+0x135/0x2c3
> [ ?539.883523] ?[<ffffffff81167e23>] writeback_sb_inodes+0x182/0x22c
> [ ?539.883526] ?[<ffffffff81168371>] __writeback_inodes_wb+0x78/0xb9
> [ ?539.883528] ?[<ffffffff81168799>] writeback_inodes_wb+0x62/0x7c
> [ ?539.883532] ?[<ffffffff81102d4b>] balance_dirty_pages_ratelimited_nr+0x3dc/0x686
> [ ?539.883534] ?[<ffffffff8116809e>] ? __mark_inode_dirty+0x2e/0x1cf
> [ ?539.883539] ?[<ffffffff810f9354>] generic_file_buffered_write+0x1c9/0x22d
> [ ?539.883545] ?[<ffffffff810782c2>] ? current_fs_time+0x27/0x2e
> [ ?539.883548] ?[<ffffffff810fadbd>] __generic_file_aio_write+0x32e/0x35e
> [ ?539.883552] ?[<ffffffff810fae53>] generic_file_aio_write+0x66/0xc1
> [ ?539.883555] ?[<ffffffff811deb83>] ext4_file_write+0x1f6/0x24f
> [ ?539.883559] ?[<ffffffff8117729a>] ? fsnotify+0x216/0x26f
> [ ?539.883565] ?[<ffffffff81144e86>] do_sync_write+0xcb/0x108
> [ ?539.883567] ?[<ffffffff81145550>] vfs_write+0xb8/0x157
> [ ?539.883570] ?[<ffffffff81145807>] sys_write+0x4d/0x77
> [ ?539.883574] ?[<ffffffff819a1802>] system_call_fastpath+0x16/0x1b
> [ ?540.125746] ?[<ffffffff81205856>] ? ext4_mb_normalize_request+0x2f4/0x480
> [ ?540.133817] ?[<ffffffff81208c6f>] ext4_mb_regular_allocator+0x59/0x312
> [ ?540.133820] ?[<ffffffff8120a25e>] ext4_mb_new_blocks+0x197/0x3f4
> [ ?540.133823] ?[<ffffffff811ff836>] ? ext4_ext_find_extent+0x130/0x2fb
> [ ?540.133829] ?[<ffffffff8199ac03>] ? _raw_spin_unlock+0x2b/0x2f
> [ ?540.133831] ?[<ffffffff812021c8>] ext4_ext_map_blocks+0xf63/0x1331
> [ ?540.133838] ?[<ffffffff811e2810>] ? ext4_map_blocks+0xf5/0x1fa
> [ ?540.133840] ?[<ffffffff811e2841>] ext4_map_blocks+0x126/0x1fa
> [ ?540.133843] ?[<ffffffff811e5d56>] mpage_da_map_and_submit+0xec/0x3b2
> [ ?540.133845] ?[<ffffffff811e6742>] ext4_da_writepages+0x316/0x4b0
> [ ?540.133854] ?[<ffffffff81103272>] do_writepages+0x24/0x2d
> [ ?540.133861] ?[<ffffffff811675c9>] writeback_single_inode+0x135/0x2c3
> [ ?540.133863] ?[<ffffffff81167e23>] writeback_sb_inodes+0x182/0x22c
> [ ?540.133866] ?[<ffffffff81168371>] __writeback_inodes_wb+0x78/0xb9
> [ ?540.133869] ?[<ffffffff81168799>] writeback_inodes_wb+0x62/0x7c
> [ ?540.133872] ?[<ffffffff81102d4b>] balance_dirty_pages_ratelimited_nr+0x3dc/0x686
> [ ?540.133875] ?[<ffffffff8116809e>] ? __mark_inode_dirty+0x2e/0x1cf
> [ ?540.133880] ?[<ffffffff810f9354>] generic_file_buffered_write+0x1c9/0x22d
> [ ?540.133886] ?[<ffffffff810782c2>] ? current_fs_time+0x27/0x2e
> [ ?540.133889] ?[<ffffffff810fadbd>] __generic_file_aio_write+0x32e/0x35e
> [ ?540.133892] ?[<ffffffff810fae53>] generic_file_aio_write+0x66/0xc1
> [ ?540.133896] ?[<ffffffff811deb83>] ext4_file_write+0x1f6/0x24f
> [ ?540.133900] ?[<ffffffff8117729a>] ? fsnotify+0x216/0x26f
> [ ?540.133905] ?[<ffffffff81144e86>] do_sync_write+0xcb/0x108
> [ ?540.133908] ?[<ffffffff81145550>] vfs_write+0xb8/0x157
> [ ?540.133910] ?[<ffffffff81145807>] sys_write+0x4d/0x77
> [ ?540.133914] ?[<ffffffff819a1802>] system_call_fastpath+0x16/0x1b
> [ ?684.729866] INFO: rcu_sched_state detected stall on CPU 27 (t=240030 jiffies)
> [ ?684.730858] INFO: rcu_sched_state detected stall on CPU 14 (t=240030 jiffies)
> [ ?864.673871] INFO: rcu_sched_state detected stall on CPU 27 (t=420061 jiffies)
> [ ?864.673877] INFO: rcu_sched_state detected stall on CPU 14 (t=420061 jiffies)
> [ 1044.615889] INFO: rcu_sched_state detected stall on CPU 27 (t=600092 jiffies)
> [ 1044.616884] INFO: rcu_sched_state detected stall on CPU 14 (t=600092 jiffies)
> [ 1224.558902] INFO: rcu_sched_state detected stall on CPU 27 (t=780123 jiffies)
> [ 1224.559899] INFO: rcu_sched_state detected stall on CPU 14 (t=780123 jiffies)
> [ 1404.501917] INFO: rcu_sched_state detected stall on CPU 14 (t=960154 jiffies)
> [ 1404.501922] INFO: rcu_sched_state detected stall on CPU 27 (t=960154 jiffies)
> [ 1584.443929] INFO: rcu_sched_state detected stall on CPU 27 (t=1140184 jiffies)
> [ 1584.443935] INFO: rcu_sched_state detected stall on CPU 14 (t=1140184 jiffies)
> [ 1764.386941] INFO: rcu_sched_state detected stall on CPU 14 (t=1320214 jiffies)
> [ 1764.386944] INFO: rcu_sched_state detected stall on CPU 27 (t=1320215 jiffies)
> [ 1944.328956] INFO: rcu_sched_state detected stall on CPU 14 (t=1500245 jiffies)
> [ 1944.328960] INFO: rcu_sched_state detected stall on CPU 27 (t=1500245 jiffies)
> [ 2124.270971] INFO: rcu_sched_state detected stall on CPU 14 (t=1680275 jiffies)
> [ 2124.270976] INFO: rcu_sched_state detected stall on CPU 27 (t=1680275 jiffies)
> [ 2304.212984] INFO: rcu_sched_state detected stall on CPU 14 (t=1860305 jiffies)
> [ 2304.213983] INFO: rcu_sched_state detected stall on CPU 27 (t=1860306 jiffies)
> [ 2484.155998] INFO: rcu_sched_state detected stall on CPU 27 (t=2040336 jiffies)
> [ 2484.156998] INFO: rcu_sched_state detected stall on CPU 14 (t=2040337 jiffies)
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to [email protected]
> More majordomo info at ?http://vger.kernel.org/majordomo-info.html
--
Best Wishes
Yongqiang Yang
Yongqiang,
Thanks for the quick fix!
On Sun, Dec 18, 2011 at 03:17:18PM +0800, Yongqiang Yang wrote:
> Hi Fengguang,
>
> Could you try the patch [ext4: do not reference pa_inode from group_pa]?
It works! You can add my tested-by and CC stable.
Thanks,
Fengguang
On Sun, Dec 18, 2011 at 07:32:37PM +0800, Wu Fengguang wrote:
> Yongqiang,
>
> Thanks for the quick fix!
>
> On Sun, Dec 18, 2011 at 03:17:18PM +0800, Yongqiang Yang wrote:
> > Hi Fengguang,
> >
> > Could you try the patch [ext4: do not reference pa_inode from group_pa]?
>
> It works! You can add my tested-by and CC stable.
The patch seems to only fix part of the problem. Today I get this slightly
different dmesg (the kernel has been patched with [ext4: do not reference
pa_inode from group_pa]):
[ 646.026574] BUG: unable to handle kernel NULL pointer dereference at 0000000000000178
[ 646.027004] IP: [<ffffffff810a5092>] __lock_acquire+0x8b/0x932
[ 646.027004] PGD 4f85067 PUD 99cb4067 PMD 0
[ 646.027004] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
[ 646.027004] CPU 6
[ 646.051405] Modules linked in:
[ 646.051405]
[ 646.051405] Pid: 6149, comm: dd Not tainted 3.2.0-rc5-ioless-full+ #1009 Supermicro X7DW3/X7DWN
[ 646.051405] RIP: 0010:[<ffffffff810a5092>] [<ffffffff810a5092>] __lock_acquire+0x8b/0x932
[ 646.051405] RSP: 0018:ffff880004ee18d8 EFLAGS: 00010097
[ 646.051405] RAX: 0000000000000000 RBX: 0000000000000170 RCX: 0000000000000000
[ 646.051405] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000170
[ 646.051405] RBP: ffff880004ee1948 R08: 0000000000000000 R09: 0000000000000000
[ 646.051405] R10: 0000000000000170 R11: ffffffff81175de4 R12: 0000000000000000
[ 646.051405] R13: 0000000000000000 R14: ffff880004fc4540 R15: 0000000000000000
[ 646.051405] FS: 00007f193aa90700(0000) GS:ffff880226a00000(0000) knlGS:0000000000000000
[ 646.051405] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 646.051405] CR2: 0000000000000178 CR3: 00000000b17cb000 CR4: 00000000000006e0
[ 646.051405] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 646.051405] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 646.051405] Process dd (pid: 6149, threadinfo ffff880004ee0000, task ffff880004fc4540)
[ 646.051405] Stack:
[ 646.051405] ffff880004ee18f8 ffffffff81099aa3 0000000000000006 0000000000000002
[ 646.051405] 0000000000000000 0000000000008010 ffff880225806b00 ffff88005fc08d68
[ 646.051405] ffff880004ee1978 0000000000000000 0000000000000170 0000000000000000
[ 646.051405] Call Trace:
[ 646.051405] [<ffffffff81099aa3>] ? sched_clock_local+0x12/0x75
[ 646.051405] [<ffffffff810a5a16>] lock_acquire+0xdd/0x10a
[ 646.051405] [<ffffffff81175de4>] ? create_empty_buffers+0x4a/0xc1
[ 646.051405] [<ffffffff8199f623>] _raw_spin_lock+0x36/0x69
[ 646.051405] [<ffffffff81175de4>] ? create_empty_buffers+0x4a/0xc1
[ 646.051405] [<ffffffff81175de4>] create_empty_buffers+0x4a/0xc1
[ 646.051405] [<ffffffff811efd2f>] ext4_discard_partial_page_buffers_no_lock+0x9f/0x406
[ 646.051405] [<ffffffff8199ffeb>] ? _raw_spin_unlock+0x2b/0x2f
[ 646.051405] [<ffffffff81170c26>] ? __mark_inode_dirty+0x1ac/0x1cc
[ 646.051405] [<ffffffff811767f3>] ? generic_write_end+0x6d/0x7f
[ 646.051405] [<ffffffff811f15e5>] ext4_da_write_end+0x244/0x2ed
[ 646.051405] [<ffffffff810ffeec>] generic_file_buffered_write+0x183/0x22d
[ 646.051405] [<ffffffff8107946a>] ? current_fs_time+0x27/0x2e
[ 646.051405] [<ffffffff8110198c>] __generic_file_aio_write+0x334/0x364
[ 646.051405] [<ffffffff8199e55c>] ? mutex_lock_nested+0x2e2/0x2f1
[ 646.051405] [<ffffffff81101a06>] ? generic_file_aio_write+0x4a/0xc1
[ 646.051405] [<ffffffff81101a22>] generic_file_aio_write+0x66/0xc1
[ 646.051405] [<ffffffff811ea020>] ext4_file_write+0x1f9/0x251
[ 646.051405] [<ffffffff8103c24b>] ? sched_clock+0x9/0xd
[ 646.051405] [<ffffffff8118180e>] ? fsnotify+0x216/0x26f
[ 646.051405] [<ffffffff8114d45e>] do_sync_write+0xce/0x10b
[ 646.051405] [<ffffffff8118180e>] ? fsnotify+0x216/0x26f
[ 646.051405] [<ffffffff8118166e>] ? fsnotify+0x76/0x26f
[ 646.051405] [<ffffffff8114dc1b>] vfs_write+0xb8/0x157
[ 646.051405] [<ffffffff8114ded2>] sys_write+0x4d/0x77
[ 646.051405] [<ffffffff819a6c02>] system_call_fastpath+0x16/0x1b
[ 646.051405] Code: bd 08 00 00 be d5 0b 00 00 48 c7 c7 86 41 d3 81 83 3d 82 f2 9f 01 00 0f 85 a4 08 00 00 e9 bb 03 00 00 41 83 fc 01 77 13 44 89 e0 <4c> 8b 6c c3 08 4d 85 ed 0f 85 5b 03 00 00 eb 34 41 83 fc 07 76
[ 646.051405] RIP [<ffffffff810a5092>] __lock_acquire+0x8b/0x932
[ 646.051405] RSP <ffff880004ee18d8>
[ 646.051405] CR2: 0000000000000178
[ 646.051405] ---[ end trace ebd0c8e3a842a6f1 ]---
The test case is about running 100 dd tasks on each of the 10 JBOD disks:
lkp-st02-x8664/JBOD-10HDD-thresh=100M/ext4-100dd-1-3.2.0-rc5-ioless-full+
Thanks,
Fengguang
Hi Fengguang,
> [ ?646.051405] ?ffff880004ee1978 0000000000000000 0000000000000170 0000000000000000
> [ ?646.051405] Call Trace:
> [ ?646.051405] ?[<ffffffff81099aa3>] ? sched_clock_local+0x12/0x75
> [ ?646.051405] ?[<ffffffff810a5a16>] lock_acquire+0xdd/0x10a
> [ ?646.051405] ?[<ffffffff81175de4>] ? create_empty_buffers+0x4a/0xc1
> [ ?646.051405] ?[<ffffffff8199f623>] _raw_spin_lock+0x36/0x69
> [ ?646.051405] ?[<ffffffff81175de4>] ? create_empty_buffers+0x4a/0xc1
> [ ?646.051405] ?[<ffffffff81175de4>] create_empty_buffers+0x4a/0xc1
> [ ?646.051405] ?[<ffffffff811efd2f>] ext4_discard_partial_page_buffers_no_lock+0x9f/0x406
> [ ?646.051405] ?[<ffffffff8199ffeb>] ? _raw_spin_unlock+0x2b/0x2f
> [ ?646.051405] ?[<ffffffff81170c26>] ? __mark_inode_dirty+0x1ac/0x1cc
> [ ?646.051405] ?[<ffffffff811767f3>] ? generic_write_end+0x6d/0x7f
> [ ?646.051405] ?[<ffffffff811f15e5>] ext4_da_write_end+0x244/0x2ed
> [ ?646.051405] ?[<ffffffff810ffeec>] generic_file_buffered_write+0x183/0x22d
It seems that you are running in to several problems related to
discard_page_buffers_no_lock in write_end. The problems had been
fixed by following patches [ext4: avoid potential hang in
mpage_submit_io() when blocksize < pagesize], [ext4: correctly handle
pages w/o buffers in ext4_discard_partial_buffers()] and [ext4: handle
EOF correctly in ext4_bio_write_page()]. The patches had been merged
to rc6.
Yongqiang.
>
> The test case is about running 100 dd tasks on each of the 10 JBOD disks:
>
> ? ? ? ?lkp-st02-x8664/JBOD-10HDD-thresh=100M/ext4-100dd-1-3.2.0-rc5-ioless-full+
>
> Thanks,
> Fengguang
--
Best Wishes
Yongqiang Yang
On Thu, Jan 05, 2012 at 10:23:13AM +0800, Yongqiang Yang wrote:
> Hi Fengguang,
>
> > [ 646.051405] ffff880004ee1978 0000000000000000 0000000000000170 0000000000000000
> > [ 646.051405] Call Trace:
> > [ 646.051405] [<ffffffff81099aa3>] ? sched_clock_local+0x12/0x75
> > [ 646.051405] [<ffffffff810a5a16>] lock_acquire+0xdd/0x10a
> > [ 646.051405] [<ffffffff81175de4>] ? create_empty_buffers+0x4a/0xc1
> > [ 646.051405] [<ffffffff8199f623>] _raw_spin_lock+0x36/0x69
> > [ 646.051405] [<ffffffff81175de4>] ? create_empty_buffers+0x4a/0xc1
> > [ 646.051405] [<ffffffff81175de4>] create_empty_buffers+0x4a/0xc1
> > [ 646.051405] [<ffffffff811efd2f>] ext4_discard_partial_page_buffers_no_lock+0x9f/0x406
> > [ 646.051405] [<ffffffff8199ffeb>] ? _raw_spin_unlock+0x2b/0x2f
> > [ 646.051405] [<ffffffff81170c26>] ? __mark_inode_dirty+0x1ac/0x1cc
> > [ 646.051405] [<ffffffff811767f3>] ? generic_write_end+0x6d/0x7f
> > [ 646.051405] [<ffffffff811f15e5>] ext4_da_write_end+0x244/0x2ed
> > [ 646.051405] [<ffffffff810ffeec>] generic_file_buffered_write+0x183/0x22d
> It seems that you are running in to several problems related to
> discard_page_buffers_no_lock in write_end. The problems had been
> fixed by following patches [ext4: avoid potential hang in
> mpage_submit_io() when blocksize < pagesize], [ext4: correctly handle
> pages w/o buffers in ext4_discard_partial_buffers()] and [ext4: handle
> EOF correctly in ext4_bio_write_page()]. The patches had been merged
> to rc6.
That's great! I'll test rc7+.
Thanks,
Fengguang
On Thu, Jan 05, 2012 at 10:45:09AM +0800, Yongqiang Yang wrote:
> On Thu, Jan 5, 2012 at 10:43 AM, Wu Fengguang <[email protected]> wrote:
> > On Thu, Jan 05, 2012 at 10:37:15AM +0800, Yongqiang Yang wrote:
> >> On Thu, Jan 5, 2012 at 10:34 AM, Wu Fengguang <[email protected]> wrote:
> >> > Yongqiang,
> >> >
> >> > I noticed that Linus's master does not contain your initial fix
> >> >
> >> > ext4: do not reference pa_inode from group_pa
> >> >
> >> > Is that *replaced* by the patches you mentioned below?
> >> nope. They are different stories. [ext4: do not reference pa_inode
> >> from group_pa] is merged into Ted's tree and has not been pushed to
> >> Linus. You can have a loot at Ted's tree
> >> http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git;a=summary
> >>
> >> [ext4: do not reference pa_inode from group_pa] is merged after the
> >> following patches.
> >
> > Thanks for the explanation. Is it planned to be pushed before the 3.2
> > release?
> I am not sure. I am guessing it will be pushed before 3.2 release:-).
This bug appears again in 3.2. I'd recommend to send the patch to
-stable once it hits 3.3-rcX.
Thanks,
Fengguang
[ 613.505459] BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
[ 613.506004] IP: [<ffffffff81208933>] perf_trace_ext4_mb_release_group_pa+0x81/0xd6
[ 613.506004] PGD 203e2e067 PUD 203e2d067 PMD 0
[ 613.506004] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
[ 613.506004] CPU 1
[ 613.506004] Modules linked in:
[ 613.506004]
[ 613.506004] Pid: 4112, comm: flush-8:80 Not tainted 3.2.0 #313 Supermicro X7DW3/X7DWN
[ 613.506004] RIP: 0010:[<ffffffff81208933>] [<ffffffff81208933>] perf_trace_ext4_mb_release_group_pa+0x81/0xd6
[ 613.506004] RSP: 0018:ffff880211981590 EFLAGS: 00010286
[ 613.506004] RAX: ffffe8ffff0091e8 RBX: ffff8801c5517e70 RCX: ffff880211954500
[ 613.506004] RDX: 0000000000000000 RSI: 0000000000000286 RDI: ffffe8ffff0091e8
[ 613.506004] RBP: ffff880211981670 R08: ffff8802119815b0 R09: 0000000000000000
[ 613.506004] R10: ffffe8fffee087b0 R11: ffffffff8121fba3 R12: ffffffff81f62ff8
[ 613.506004] R13: ffff880211981720 R14: ffff8802080ece50 R15: ffff880211981740
[ 613.506004] FS: 0000000000000000(0000) GS:ffff880226000000(0000) knlGS:0000000000000000
[ 613.506004] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 613.506004] CR2: 0000000000000028 CR3: 00000001efa02000 CR4: 00000000000006e0
[ 613.506004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 613.506004] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 613.506004] Process flush-8:80 (pid: 4112, threadinfo ffff880211980000, task ffff880211954520)
[ 613.506004] Stack:
[ 613.506004] 0000000000000000 0000000000000003 ffff880211981620 ffff8802119815b0
[ 613.506004] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 613.506004] ffff8802119816c0 0000000000000000 0000000000000000 0000000000000000
[ 613.506004] Call Trace:
[ 613.506004] [<ffffffff8121d110>] ? ext4_mb_release_group_pa+0x40/0xfe
[ 613.506004] [<ffffffff8121d110>] ext4_mb_release_group_pa+0x40/0xfe
[ 613.506004] [<ffffffff8121fbe3>] ext4_mb_discard_group_preallocations+0x355/0x3eb
[ 613.506004] [<ffffffff81222b32>] ext4_mb_new_blocks+0x2fd/0x422
[ 613.506004] [<ffffffff8121a495>] ext4_ext_map_blocks+0x146f/0x1969
[ 613.506004] [<ffffffff810a3339>] ? local_clock+0x41/0x5a
[ 613.506004] [<ffffffff810aec1b>] ? __lock_acquire+0x564/0x932
[ 613.506004] [<ffffffff810aec1b>] ? __lock_acquire+0x564/0x932
[ 613.506004] [<ffffffff811f601b>] ? ext4_map_blocks+0x103/0x221
[ 613.506004] [<ffffffff811f604c>] ext4_map_blocks+0x134/0x221
[ 613.506004] [<ffffffff811f9857>] mpage_da_map_and_submit+0xef/0x404
[ 613.506004] [<ffffffff811fa2cc>] ext4_da_writepages+0x350/0x505
[ 613.506004] [<ffffffff8103c24b>] ? sched_clock+0x9/0xd
[ 613.506004] [<ffffffff810ae168>] ? lock_release_holdtime+0xa3/0xac
[ 613.506004] [<ffffffff811137b4>] do_writepages+0x24/0x2d
[ 613.506004] [<ffffffff811786a5>] writeback_single_inode+0x126/0x2b4
[ 613.506004] [<ffffffff81178f03>] writeback_sb_inodes+0x17f/0x229
[ 613.506004] [<ffffffff81179657>] wb_writeback+0x130/0x23a
[ 613.506004] [<ffffffff81179982>] wb_do_writeback+0x8f/0x1b7
[ 613.506004] [<ffffffff81179b5d>] ? bdi_writeback_thread+0xb3/0x215
[ 613.506004] [<ffffffff81179b36>] bdi_writeback_thread+0x8c/0x215
[ 613.506004] [<ffffffff81179aaa>] ? wb_do_writeback+0x1b7/0x1b7
[ 613.506004] [<ffffffff8109db30>] kthread+0x8e/0x96
[ 613.506004] [<ffffffff819ec584>] kernel_thread_helper+0x4/0x10
[ 613.506004] [<ffffffff819e3974>] ? retint_restore_args+0x13/0x13
[ 613.506004] [<ffffffff8109daa2>] ? __init_kthread_worker+0x5b/0x5b
[ 613.506004] [<ffffffff819ec580>] ? gs_change+0x13/0x13
[ 613.506004] Code: 89 c2 4c 89 85 38 ff ff ff 48 8d 4d ec 41 0f b7 74 24 48 e8 ed 06 7e 00 4c 8b 85 38 ff ff ff 48 85 c0 74 50 48 8b 93 a0 00 00 00 <48> 8b 52 28 8b 52 10 89 50 0c 48 8b 93 80 00 00 00 48 89 50 10
[ 613.506004] RIP [<ffffffff81208933>] perf_trace_ext4_mb_release_group_pa+0x81/0xd6
[ 613.876032] RSP <ffff880211981590>
[ 613.876032] CR2: 0000000000000028
[ 613.882620] ---[ end trace af3c59e20d0fb446 ]---
[ 613.882624] ------------[ cut here ]------------
[ 613.882630] WARNING: at /c/wfg/linux/kernel/exit.c:898 do_exit+0x67/0x76e()
[ 613.882632] Hardware name: X7DW3
[ 613.882633] Modules linked in:
[ 613.882636] Pid: 4112, comm: flush-8:80 Tainted: G D 3.2.0 #313
[ 613.882638] Call Trace:
[ 613.882643] [<ffffffff8107dca4>] warn_slowpath_common+0x85/0x9d
[ 613.882646] [<ffffffff8107f42e>] ? kmsg_dump+0x8a/0x10c
[ 613.882649] [<ffffffff8107dcd6>] warn_slowpath_null+0x1a/0x1c
[ 613.882651] [<ffffffff81081417>] do_exit+0x67/0x76e
[ 613.882653] [<ffffffff8107f49f>] ? kmsg_dump+0xfb/0x10c
[ 613.882656] [<ffffffff8107f42e>] ? kmsg_dump+0x8a/0x10c
[ 613.882660] [<ffffffff819e4629>] oops_end+0xbe/0xc6
[ 613.882664] [<ffffffff81056e15>] no_context+0x184/0x193
[ 613.882667] [<ffffffff81056fed>] __bad_area_nosemaphore+0x1c9/0x1e9
[ 613.882670] [<ffffffff8103c24b>] ? sched_clock+0x9/0xd
[ 613.882674] [<ffffffff810a315b>] ? sched_clock_local+0x12/0x75
[ 613.882677] [<ffffffff81057020>] bad_area_nosemaphore+0x13/0x15
[ 613.882679] [<ffffffff819e69e7>] do_page_fault+0x213/0x431
[ 613.882684] [<ffffffff811084fc>] ? perf_output_begin+0x1c2/0x1f5
[ 613.882686] [<ffffffff8103c1fb>] ? native_sched_clock+0x29/0x70
[ 613.882688] [<ffffffff8103c24b>] ? sched_clock+0x9/0xd
[ 613.882693] [<ffffffff8140a92d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 613.882696] [<ffffffff8121fba3>] ? ext4_mb_discard_group_preallocations+0x315/0x3eb
[ 613.882699] [<ffffffff819e3b85>] page_fault+0x25/0x30
[ 613.882702] [<ffffffff8121fba3>] ? ext4_mb_discard_group_preallocations+0x315/0x3eb
[ 613.882705] [<ffffffff81208933>] ? perf_trace_ext4_mb_release_group_pa+0x81/0xd6
[ 613.882708] [<ffffffff8121d110>] ? ext4_mb_release_group_pa+0x40/0xfe
[ 613.882710] [<ffffffff8121d110>] ext4_mb_release_group_pa+0x40/0xfe
[ 613.882712] [<ffffffff8121fbe3>] ext4_mb_discard_group_preallocations+0x355/0x3eb
[ 613.882716] [<ffffffff81222b32>] ext4_mb_new_blocks+0x2fd/0x422
[ 613.882719] [<ffffffff8121a495>] ext4_ext_map_blocks+0x146f/0x1969
[ 613.882721] [<ffffffff810a3339>] ? local_clock+0x41/0x5a
[ 613.882725] [<ffffffff810aec1b>] ? __lock_acquire+0x564/0x932
[ 613.882728] [<ffffffff810aec1b>] ? __lock_acquire+0x564/0x932
[ 613.882731] [<ffffffff811f601b>] ? ext4_map_blocks+0x103/0x221
[ 613.882733] [<ffffffff811f604c>] ext4_map_blocks+0x134/0x221
[ 613.882736] [<ffffffff811f9857>] mpage_da_map_and_submit+0xef/0x404
[ 613.882739] [<ffffffff811fa2cc>] ext4_da_writepages+0x350/0x505
[ 613.882741] [<ffffffff8103c24b>] ? sched_clock+0x9/0xd
[ 613.882744] [<ffffffff810ae168>] ? lock_release_holdtime+0xa3/0xac
[ 613.882747] [<ffffffff811137b4>] do_writepages+0x24/0x2d
[ 613.882751] [<ffffffff811786a5>] writeback_single_inode+0x126/0x2b4
[ 613.882753] [<ffffffff81178f03>] writeback_sb_inodes+0x17f/0x229
[ 613.882756] [<ffffffff81179657>] wb_writeback+0x130/0x23a
[ 613.882759] [<ffffffff81179982>] wb_do_writeback+0x8f/0x1b7
[ 613.882761] [<ffffffff81179b5d>] ? bdi_writeback_thread+0xb3/0x215
[ 613.882764] [<ffffffff81179b36>] bdi_writeback_thread+0x8c/0x215
[ 613.882767] [<ffffffff81179aaa>] ? wb_do_writeback+0x1b7/0x1b7
[ 613.882769] [<ffffffff8109db30>] kthread+0x8e/0x96
[ 613.882773] [<ffffffff819ec584>] kernel_thread_helper+0x4/0x10
[ 613.882776] [<ffffffff819e3974>] ? retint_restore_args+0x13/0x13
[ 613.882779] [<ffffffff8109daa2>] ? __init_kthread_worker+0x5b/0x5b
[ 613.882782] [<ffffffff819ec580>] ? gs_change+0x13/0x13
[ 613.882783] ---[ end trace af3c59e20d0fb447 ]---
[ 613.882796] flush-8:80 used greatest stack depth: 2352 bytes left
[ 614.468204] BUG: unable to handle kernel NULL pointer dereference at 0000000000000028
[ 614.469003] IP: [<ffffffff81208933>] perf_trace_ext4_mb_release_group_pa+0x81/0xd6
[ 614.469003] PGD 211942067 PUD 21be9d067 PMD 0
[ 614.469003] Oops: 0000 [#2] SMP DEBUG_PAGEALLOC
[ 614.469003] CPU 3
[ 614.469003] Modules linked in:
[ 614.469003]
[ 614.469003] Pid: 4117, comm: flush-8:160 Tainted: G D W 3.2.0 #313 Supermicro X7DW3/X7DWN
[ 614.469003] RIP: 0010:[<ffffffff81208933>] [<ffffffff81208933>] perf_trace_ext4_mb_release_group_pa+0x81/0xd6
[ 614.469003] RSP: 0018:ffff880211a17590 EFLAGS: 00010286
[ 614.469003] RAX: ffffe8ffff4091e8 RBX: ffff8801c55179d8 RCX: ffff8802119b2200
[ 614.469003] RDX: 0000000000000000 RSI: 0000000000000286 RDI: ffffe8ffff4091e8
[ 614.469003] RBP: ffff880211a17670 R08: ffff880211a175b0 R09: 0000000000000000
[ 614.469003] R10: ffffe8fffee08ff8 R11: ffffffff8121fba3 R12: ffffffff81f62ff8
[ 614.469003] R13: ffff880211a17720 R14: ffff8802080ece50 R15: ffff8801c5478000
[ 614.560062] FS: 0000000000000000(0000) GS:ffff880226400000(0000) knlGS:0000000000000000
[ 614.560062] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 614.560062] CR2: 0000000000000028 CR3: 0000000211945000 CR4: 00000000000006e0
[ 614.560062] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 614.560062] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 614.560062] Process flush-8:160 (pid: 4117, threadinfo ffff880211a16000, task ffff8802119b2290)
[ 614.610013] Stack:
[ 614.610013] ffff880218a5a020 0000000000000000 ffff8801f88d9858 ffff880211a175b0
[ 614.610013] 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 614.610013] ffff880211a176c0 0000000000000000 0000000000000000 0000000000000000
[ 614.610013] Call Trace:
[ 614.610013] [<ffffffff8121d110>] ? ext4_mb_release_group_pa+0x40/0xfe
[ 614.610013] [<ffffffff8121d110>] ext4_mb_release_group_pa+0x40/0xfe
[ 614.610013] [<ffffffff8121fbe3>] ext4_mb_discard_group_preallocations+0x355/0x3eb
[ 614.610013] [<ffffffff81222b32>] ext4_mb_new_blocks+0x2fd/0x422
[ 614.610013] [<ffffffff8121a495>] ext4_ext_map_blocks+0x146f/0x1969
[ 614.610013] [<ffffffff81404010>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2
[ 614.610013] [<ffffffff811f5f5f>] ? ext4_map_blocks+0x47/0x221
[ 614.610013] [<ffffffff811f601b>] ? ext4_map_blocks+0x103/0x221
[ 614.610013] [<ffffffff811f604c>] ext4_map_blocks+0x134/0x221
[ 614.610013] [<ffffffff811f9857>] mpage_da_map_and_submit+0xef/0x404
[ 614.610013] [<ffffffff811fa2cc>] ext4_da_writepages+0x350/0x505
[ 614.610013] [<ffffffff811137b4>] do_writepages+0x24/0x2d
[ 614.610013] [<ffffffff811786a5>] writeback_single_inode+0x126/0x2b4
[ 614.610013] [<ffffffff81178f03>] writeback_sb_inodes+0x17f/0x229
[ 614.610013] [<ffffffff81179657>] wb_writeback+0x130/0x23a
[ 614.610013] [<ffffffff81179982>] wb_do_writeback+0x8f/0x1b7
[ 614.610013] [<ffffffff81179b5d>] ? bdi_writeback_thread+0xb3/0x215
[ 614.610013] [<ffffffff81179b36>] bdi_writeback_thread+0x8c/0x215
[ 614.610013] [<ffffffff81179aaa>] ? wb_do_writeback+0x1b7/0x1b7
[ 614.610013] [<ffffffff8109db30>] kthread+0x8e/0x96
[ 614.610013] [<ffffffff819ec584>] kernel_thread_helper+0x4/0x10
[ 614.610013] [<ffffffff819e3974>] ? retint_restore_args+0x13/0x13
[ 614.610013] [<ffffffff8109daa2>] ? __init_kthread_worker+0x5b/0x5b
[ 614.610013] [<ffffffff819ec580>] ? gs_change+0x13/0x13
[ 614.610013] Code: 89 c2 4c 89 85 38 ff ff ff 48 8d 4d ec 41 0f b7 74 24 48 e8 ed 06 7e 00 4c 8b 85 38 ff ff ff 48 85 c0 74 50 48 8b 93 a0 00 00 00 <48> 8b 52 28 8b 52 10 89 50 0c 48 8b 93 80 00 00 00 48 89 50 10
[ 614.610013] RIP [<ffffffff81208933>] perf_trace_ext4_mb_release_group_pa+0x81/0xd6
[ 614.610013] RSP <ffff880211a17590>
[ 614.610013] CR2: 0000000000000028
[ 614.615263] ---[ end trace af3c59e20d0fb448 ]---
[ 614.615266] ------------[ cut here ]------------
[ 614.615271] WARNING: at /c/wfg/linux/kernel/exit.c:898 do_exit+0x67/0x76e()
[ 614.615272] Hardware name: X7DW3
[ 614.615273] Modules linked in:
[ 614.615276] Pid: 4117, comm: flush-8:160 Tainted: G D W 3.2.0 #313
[ 614.615278] Call Trace:
[ 614.615282] [<ffffffff8107dca4>] warn_slowpath_common+0x85/0x9d
[ 614.615285] [<ffffffff8107f42e>] ? kmsg_dump+0x8a/0x10c
[ 614.615287] [<ffffffff8107dcd6>] warn_slowpath_null+0x1a/0x1c
[ 614.615289] [<ffffffff81081417>] do_exit+0x67/0x76e
[ 614.615292] [<ffffffff8107f49f>] ? kmsg_dump+0xfb/0x10c
[ 614.615294] [<ffffffff8107f42e>] ? kmsg_dump+0x8a/0x10c
[ 614.615298] [<ffffffff819e4629>] oops_end+0xbe/0xc6
[ 614.615302] [<ffffffff81056e15>] no_context+0x184/0x193
[ 614.615305] [<ffffffff81056fed>] __bad_area_nosemaphore+0x1c9/0x1e9
[ 614.615307] [<ffffffff81057020>] bad_area_nosemaphore+0x13/0x15
[ 614.615310] [<ffffffff819e69e7>] do_page_fault+0x213/0x431
[ 614.615314] [<ffffffff8110833a>] ? perf_output_copy+0x74/0x74
[ 614.615318] [<ffffffff8140a92d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[ 614.615321] [<ffffffff8121fba3>] ? ext4_mb_discard_group_preallocations+0x315/0x3eb
[ 614.615323] [<ffffffff819e3b85>] page_fault+0x25/0x30
[ 614.615326] [<ffffffff8121fba3>] ? ext4_mb_discard_group_preallocations+0x315/0x3eb
[ 614.615328] [<ffffffff81208933>] ? perf_trace_ext4_mb_release_group_pa+0x81/0xd6
[ 614.615331] [<ffffffff8121d110>] ? ext4_mb_release_group_pa+0x40/0xfe
[ 614.615333] [<ffffffff8121d110>] ext4_mb_release_group_pa+0x40/0xfe
[ 614.615336] [<ffffffff8121fbe3>] ext4_mb_discard_group_preallocations+0x355/0x3eb
[ 614.615339] [<ffffffff81222b32>] ext4_mb_new_blocks+0x2fd/0x422
[ 614.615342] [<ffffffff8121a495>] ext4_ext_map_blocks+0x146f/0x1969
[ 614.615346] [<ffffffff81404010>] ? radix_tree_gang_lookup_tag_slot+0x81/0xa2
[ 614.615348] [<ffffffff811f5f5f>] ? ext4_map_blocks+0x47/0x221
[ 614.615350] [<ffffffff811f601b>] ? ext4_map_blocks+0x103/0x221
[ 614.615353] [<ffffffff811f604c>] ext4_map_blocks+0x134/0x221
[ 614.615355] [<ffffffff811f9857>] mpage_da_map_and_submit+0xef/0x404
[ 614.615358] [<ffffffff811fa2cc>] ext4_da_writepages+0x350/0x505
[ 614.615361] [<ffffffff811137b4>] do_writepages+0x24/0x2d
[ 614.615364] [<ffffffff811786a5>] writeback_single_inode+0x126/0x2b4
[ 614.615366] [<ffffffff81178f03>] writeback_sb_inodes+0x17f/0x229
[ 614.615369] [<ffffffff81179657>] wb_writeback+0x130/0x23a
[ 614.615372] [<ffffffff81179982>] wb_do_writeback+0x8f/0x1b7
[ 614.615374] [<ffffffff81179b5d>] ? bdi_writeback_thread+0xb3/0x215
[ 614.615377] [<ffffffff81179b36>] bdi_writeback_thread+0x8c/0x215
[ 614.615379] [<ffffffff81179aaa>] ? wb_do_writeback+0x1b7/0x1b7
[ 614.615382] [<ffffffff8109db30>] kthread+0x8e/0x96
[ 614.615385] [<ffffffff819ec584>] kernel_thread_helper+0x4/0x10
[ 614.615388] [<ffffffff819e3974>] ? retint_restore_args+0x13/0x13
[ 614.615391] [<ffffffff8109daa2>] ? __init_kthread_worker+0x5b/0x5b
[ 614.615393] [<ffffffff819ec580>] ? gs_change+0x13/0x13
[ 614.615395] ---[ end trace af3c59e20d0fb449 ]---