Greetings,
Sorry the report is a bit late, this bug looks still active:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master
commit 044f1daaaaf7c86bc4fcf433848b7baae236946b
Merge: d557d1b58b354 daaadb3e9453a
Author: Linus Torvalds <[email protected]>
AuthorDate: Sat May 6 11:25:08 2017 -0700
Commit: Linus Torvalds <[email protected]>
CommitDate: Sat May 6 11:25:08 2017 -0700
Merge branch 'for-linus' of git://git.kernel.dk/linux-block
Pull block fixes and updates from Jens Axboe:
"Some fixes and followup features/changes that should go in, in this
merge window. This contains:
- Two fixes for lightnvm from Javier, fixing problems in the new code
merge previously in this merge window.
- A fix from Jan for the backing device changes, fixing an issue in
NFS that causes a failure to mount on certain setups.
- A change from Christoph, cleaning up the blk-mq init and exit
request paths.
- Remove elevator_change(), which is now unused. From Bart.
- A fix for queue operation invocation on a dead queue, from Bart.
- A series fixing up mtip32xx for blk-mq scheduling, removing a
bandaid we previously had in place for this. From me.
- A regression fix for this series, fixing a case where we wait on
workqueue flushing from an invalid (non-blocking) context. From me.
- A fix/optimization from Ming, ensuring that we don't both quiesce
and freeze a queue at the same time.
- A fix from Peter on lock ordering for CPU hotplug. Not a real
problem right now, but will be once the CPU hotplug rework goes in.
- A series from Omar, cleaning up out blk-mq debugfs support, and
adding support for exporting info from schedulers in debugfs as
well. This is really useful in debugging stalls or livelocks. From
Omar"
* 'for-linus' of git://git.kernel.dk/linux-block: (28 commits)
mq-deadline: add debugfs attributes
kyber: add debugfs attributes
blk-mq-debugfs: allow schedulers to register debugfs attributes
blk-mq: untangle debugfs and sysfs
blk-mq: move debugfs declarations to a separate header file
blk-mq: Do not invoke queue operations on a dead queue
blk-mq-debugfs: get rid of a bunch of boilerplate
blk-mq-debugfs: rename hw queue directories from <n> to hctx<n>
blk-mq-debugfs: don't open code strstrip()
blk-mq-debugfs: error on long write to queue "state" file
blk-mq-debugfs: clean up flag definitions
blk-mq-debugfs: separate flags with |
nfs: Fix bdi handling for cloned superblocks
block/mq: Cure cpu hotplug lock inversion
lightnvm: fix bad back free on error path
lightnvm: create cmd before allocating request
blk-mq: don't use sync workqueue flushing from drivers
mtip32xx: convert internal commands to regular block infrastructure
mtip32xx: cleanup internal tag assumptions
block: don't call blk_mq_quiesce_queue() after queue is frozen
...
d557d1b58b refcount: change EXPORT_SYMBOL markings
daaadb3e94 mq-deadline: add debugfs attributes
044f1daaaa Merge branch 'for-linus' of git://git.kernel.dk/linux-block
8d829b9bb8 Merge branch 'for-linus' of git://git.kernel.dk/linux-block
09c1728473 Add linux-next specific files for 20170622
+------------------------------------------------+------------+------------+------------+------------+---------------+
| | d557d1b58b | daaadb3e94 | 044f1daaaa | 8d829b9bb8 | next-20170622 |
+------------------------------------------------+------------+------------+------------+------------+---------------+
| boot_successes | 914 | 0 | 306 | 314 | 123 |
| boot_failures | 0 | 1224 | 8 | 4 | 6 |
| BUG:unable_to_handle_kernel | 0 | 1224 | | | |
| Oops:#[##] | 0 | 1224 | | | |
| Kernel_panic-not_syncing:Fatal_exception | 0 | 1224 | 8 | 4 | 1 |
| kernel_BUG_at_block/blk-mq.c | 0 | 0 | 7 | 4 | 1 |
| invalid_opcode:#[##] | 0 | 0 | 8 | 4 | 1 |
| kernel_BUG_at_block/elevator.c | 0 | 0 | 1 | | |
| is_trying_to_release_lock(rcu_preempt_state)at | 0 | 0 | 0 | 0 | 5 |
+------------------------------------------------+------------+------------+------------+------------+---------------+
[ 36.037364] sector 2, nr/cnr 0/2
[ 36.037367] bio ffff88001af84e80, biotail ffff88001af84e80, len 0
[ 36.037367] bio ffff88001af84e80, biotail ffff88001af84e80, len 0
[ 36.037424] ------------[ cut here ]------------
[ 36.037424] ------------[ cut here ]------------
[ 36.037428] kernel BUG at block/blk-mq.c:415!
[ 36.037428] kernel BUG at block/blk-mq.c:415!
[ 36.037433] invalid opcode: 0000 [#1] PREEMPT SMP
[ 36.037433] invalid opcode: 0000 [#1] PREEMPT SMP
[ 36.037434] Modules linked in:
[ 36.037434] Modules linked in:
[ 36.037441] CPU: 0 PID: 178 Comm: kworker/0:1H Not tainted 4.11.0-10534-g044f1da #1
[ 36.037441] CPU: 0 PID: 178 Comm: kworker/0:1H Not tainted 4.11.0-10534-g044f1da #1
[ 36.037448] Workqueue: kblockd blk_mq_run_work_fn
[ 36.037448] Workqueue: kblockd blk_mq_run_work_fn
[ 36.037452] task: ffff88001abb8000 task.stack: ffffc90000388000
[ 36.037452] task: ffff88001abb8000 task.stack: ffffc90000388000
[ 36.037457] RIP: 0010:blk_mq_end_request+0x97/0xb0
[ 36.037457] RIP: 0010:blk_mq_end_request+0x97/0xb0
[ 36.037460] RSP: 0018:ffffc9000038bce8 EFLAGS: 00010297
[ 36.037460] RSP: 0018:ffffc9000038bce8 EFLAGS: 00010297
[ 36.037464] RAX: ffff88001abb8000 RBX: ffff88001df0b980 RCX: 0000000000000400
[ 36.037464] RAX: ffff88001abb8000 RBX: ffff88001df0b980 RCX: 0000000000000400
[ 36.037467] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000400
[ 36.037467] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000400
[ 36.037470] RBP: ffffc9000038bcf8 R08: 0000000000000000 R09: 0000000000000000
[ 36.037470] RBP: ffffc9000038bcf8 R08: 0000000000000000 R09: 0000000000000000
[ 36.037473] R10: 0000000000000001 R11: 0000000072f531ca R12: 00000000fffffffb
[ 36.037473] R10: 0000000000000001 R11: 0000000072f531ca R12: 00000000fffffffb
[ 36.037476] R13: ffffc9000038bd80 R14: ffff88001a847008 R15: ffff88001a8482d8
[ 36.037476] R13: ffffc9000038bd80 R14: ffff88001a847008 R15: ffff88001a8482d8
[ 36.037481] FS: 0000000000000000(0000) GS:ffff88001fa00000(0000) knlGS:0000000000000000
[ 36.037481] FS: 0000000000000000(0000) GS:ffff88001fa00000(0000) knlGS:0000000000000000
[ 36.037484] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 36.037484] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 36.037487] CR2: 00007f414771adb0 CR3: 000000001ae45000 CR4: 00000000000006b0
[ 36.037487] CR2: 00007f414771adb0 CR3: 000000001ae45000 CR4: 00000000000006b0
[ 36.037491] Call Trace:
[ 36.037491] Call Trace:
[ 36.037497] blk_mq_dispatch_rq_list+0x427/0x520
[ 36.037497] blk_mq_dispatch_rq_list+0x427/0x520
[ 36.037505] blk_mq_sched_dispatch_requests+0x160/0x1e0
[ 36.037505] blk_mq_sched_dispatch_requests+0x160/0x1e0
[ 36.037510] __blk_mq_run_hw_queue+0x1a0/0x2a0
[ 36.037510] __blk_mq_run_hw_queue+0x1a0/0x2a0
[ 36.037515] ? __blk_mq_run_hw_queue+0x168/0x2a0
[ 36.037515] ? __blk_mq_run_hw_queue+0x168/0x2a0
[ 36.037519] blk_mq_run_work_fn+0x3f/0x50
[ 36.037519] blk_mq_run_work_fn+0x3f/0x50
[ 36.037525] process_one_work+0x20f/0x4b0
[ 36.037525] process_one_work+0x20f/0x4b0
[ 36.037529] ? process_one_work+0x19c/0x4b0
[ 36.037529] ? process_one_work+0x19c/0x4b0
[ 36.037535] worker_thread+0x53/0x4d0
[ 36.037535] worker_thread+0x53/0x4d0
[ 36.037541] kthread+0x12b/0x170
[ 36.037541] kthread+0x12b/0x170
[ 36.037545] ? process_one_work+0x4b0/0x4b0
[ 36.037545] ? process_one_work+0x4b0/0x4b0
[ 36.037550] ? kthread_create_on_node+0x70/0x70
[ 36.037550] ? kthread_create_on_node+0x70/0x70
[ 36.037557] ret_from_fork+0x31/0x40
[ 36.037557] ret_from_fork+0x31/0x40
[ 36.037560] Code: 38 ca ff 4c 8b a3 40 01 00 00 4d 85 e4 75 1e e8 c0 38 ca ff 48 89 df e8 e8 6d 00 00 e8 b3 38 ca ff 5b 41 5c 5d c3 e8 a9 38 ca ff <0f> 0b e8 a2 38 ca ff 4c 89 e7 e8 ca 6d 00 00 eb d3 0f 1f 84 00
[ 36.037560] Code: 38 ca ff 4c 8b a3 40 01 00 00 4d 85 e4 75 1e e8 c0 38 ca ff 48 89 df e8 e8 6d 00 00 e8 b3 38 ca ff 5b 41 5c 5d c3 e8 a9 38 ca ff <0f> 0b e8 a2 38 ca ff 4c 89 e7 e8 ca 6d 00 00 eb d3 0f 1f 84 00
[ 36.037643] RIP: blk_mq_end_request+0x97/0xb0 RSP: ffffc9000038bce8
[ 36.037643] RIP: blk_mq_end_request+0x97/0xb0 RSP: ffffc9000038bce8
[ 36.037646] ---[ end trace 411e8acef3348c6b ]---
[ 36.037646] ---[ end trace 411e8acef3348c6b ]---
# HH:MM RESULT GOOD BAD GOOD_BUT_DIRTY DIRTY_NOT_BAD
git bisect start 41f1830f5a7af77cf5c86359aba3cbd706687e52 v4.11 --
git bisect bad 1be627dfa74d9eee1eca1bc48653e461366a2bb9 # 11:48 B 62 2 0 0 Merge tag 'staging-4.12-rc6' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
git bisect bad ec059019b7e6ae3926a8e1dec02a62d64dd8c060 # 12:09 B 20 1 0 0 Merge branch 'for-linus-4.12-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rw/uml
git bisect bad 6b402bdfdbafce241cb6789ae38d94f45641078c # 12:37 B 51 1 0 0 Merge tag 'for-v4.12-2' of git://git.kernel.org/pub/scm/linux/kernel/git/sre/linux-power-supply
git bisect good ab182e67ec99ea0c8d7435a32a4a1ed9bb02559a # 13:28 G 208 0 0 0 Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
git bisect bad 28b47809b2171a6cfbab839936b24280639c9f85 # 13:39 B 0 1 13 0 Merge tag 'iommu-updates-v4.12' of git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu
git bisect bad 70ef8f0d37573079e093305214d0cc9eb71100f7 # 14:05 B 53 1 0 0 Merge tag 'for-f2fs-4.12' of git://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs
git bisect bad 677375cef8cb7763ef620e007873117b9bad72a0 # 14:47 B 98 1 0 0 Merge tag 'fscrypt_for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/fscrypt
git bisect bad 13e0988140374123bead1dd27c287354cb95108e # 15:34 B 165 1 0 0 docs: complete bumping minimal GNU Make version to 3.81
git bisect bad fe7a719b30dfdb4d55680461954b99b257ebe671 # 16:03 B 49 1 0 0 Merge branch 'for-next' of git://git.samba.org/sfrench/cifs-2.6
git bisect bad d484467c860dab3e17893d23b2238e1f581460fa # 16:25 B 8 1 0 0 Merge tag 'xfs-4.12-merge-7' of git://git.kernel.org/pub/scm/fs/xfs/xfs-linux
git bisect good d557d1b58b3546bab2c5bc2d624c5709840e6b10 # 17:43 G 310 0 0 0 refcount: change EXPORT_SYMBOL markings
git bisect bad 044f1daaaaf7c86bc4fcf433848b7baae236946b # 17:53 B 0 1 13 0 Merge branch 'for-linus' of git://git.kernel.dk/linux-block
git bisect good 507f7d68fe5c24973dcd6e48f011bdfbd2197f01 # 19:00 G 310 0 310 314 lightnvm: fix bad back free on error path
git bisect good 88aabbd7e7ace77fb3fbf06d22dd3cc0bfc0ca79 # 19:46 G 310 0 310 314 blk-mq-debugfs: rename hw queue directories from <n> to hctx<n>
git bisect good 9c1051aacde828073dbbab5e8e59c0fc802efa9a # 20:29 G 308 0 308 312 blk-mq: untangle debugfs and sysfs
git bisect good 16b738f651c83a01db057e5db02ec4b830af9130 # 21:30 G 310 0 310 314 kyber: add debugfs attributes
git bisect good daaadb3e9453ab89c2e113a2d1df8e19e30944cc # 22:22 G 302 0 302 306 mq-deadline: add debugfs attributes
# first bad commit: [044f1daaaaf7c86bc4fcf433848b7baae236946b] Merge branch 'for-linus' of git://git.kernel.dk/linux-block
git bisect good d557d1b58b3546bab2c5bc2d624c5709840e6b10 # 23:18 G 910 0 0 0 refcount: change EXPORT_SYMBOL markings
git bisect good daaadb3e9453ab89c2e113a2d1df8e19e30944cc # 01:30 G 907 0 907 1221 mq-deadline: add debugfs attributes
# extra tests with CONFIG_DEBUG_INFO_REDUCED
git bisect bad 044f1daaaaf7c86bc4fcf433848b7baae236946b # 01:47 B 16 1 0 0 Merge branch 'for-linus' of git://git.kernel.dk/linux-block
# extra tests on HEAD of linux-devel/devel-spot-201706220724
git bisect bad 7d64799d1536df3eef4ba6cc7cb2258a1943f637 # 01:47 B 43 2 0 2 0day head guard for 'devel-spot-201706220724'
# extra tests on tree/branch linus/master
git bisect bad 8d829b9bb878904f3419af8003176a12910b94e8 # 02:24 B 85 1 0 0 Merge branch 'for-linus' of git://git.kernel.dk/linux-block
# extra tests on tree/branch linux-next/master
git bisect bad 09c17284731e42dbe4c6d334603e9c05ba1219ae # 02:51 B 57 1 0 1 Add linux-next specific files for 20170622
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/lkp Intel Corporation
On 06/26/2017 09:53 PM, Fengguang Wu wrote:
> Greetings,
>
> Sorry the report is a bit late, this bug looks still active:
>
First I've heard about it!
> [ 36.037364] sector 2, nr/cnr 0/2
> [ 36.037367] bio ffff88001af84e80, biotail ffff88001af84e80, len 0
> [ 36.037367] bio ffff88001af84e80, biotail ffff88001af84e80, len 0
> [ 36.037424] ------------[ cut here ]------------
> [ 36.037424] ------------[ cut here ]------------
> [ 36.037428] kernel BUG at block/blk-mq.c:415!
> [ 36.037428] kernel BUG at block/blk-mq.c:415!
> [ 36.037433] invalid opcode: 0000 [#1] PREEMPT SMP
> [ 36.037433] invalid opcode: 0000 [#1] PREEMPT SMP
> [ 36.037434] Modules linked in:
> [ 36.037434] Modules linked in:
Do you have the full dmesg of this? I looked at the one attached to the
bug report, but that's a completely different bug (NULL deref in drm).
Are you sure it's still the above blk-mq BUG() we're hitting?
--
Jens Axboe