Hi,
Boot of mainline kernel is currently broken.
On commit 7d3bf613e99abbd96ac7b90ee3694a246c975021.
Config:
https://gist.githubusercontent.com/dvyukov/9f7f1fd9e477efd85b221b3a21036c20/raw/7c56ede0840494b26045976960866f2b265c6f64/gistfile1.txt
Should have been introduced very recently.
You may need to also patch "umh: fix race condition", because that's
another boot crasher currently present in tree.
==================================================================
BUG: KASAN: use-after-free in blk_flush_complete_seq+0xfde/0x1810
block/blk-flush.c:215
Read of size 4 at addr ffff880063f876d4 by task swapper/2/0
CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.17.0+ #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014
Call Trace:
<IRQ>
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x1a7/0x275 lib/dump_stack.c:113
print_address_description+0x73/0x250 mm/kasan/report.c:256
kasan_report_error mm/kasan/report.c:354 [inline]
kasan_report+0x23c/0x360 mm/kasan/report.c:412
__asan_report_load4_noabort+0x14/0x20 mm/kasan/report.c:432
blk_flush_complete_seq+0xfde/0x1810 block/blk-flush.c:215
flush_end_io+0x523/0xb50 block/blk-flush.c:257
blk_finish_request+0x3fa/0x6a0 block/blk-core.c:3209
scsi_end_request+0x3d3/0x820 drivers/scsi/scsi_lib.c:715
scsi_io_completion+0x1869/0x1910 drivers/scsi/scsi_lib.c:898
scsi_finish_command+0x4de/0x8d0 drivers/scsi/scsi.c:248
scsi_softirq_done+0x3ad/0x4b0 drivers/scsi/scsi_lib.c:1683
blk_done_softirq+0x48c/0x700 block/blk-softirq.c:37
__do_softirq+0x2e0/0xb94 kernel/softirq.c:284
invoke_softirq kernel/softirq.c:364 [inline]
irq_exit+0x1cc/0x200 kernel/softirq.c:404
exiting_irq arch/x86/include/asm/apic.h:525 [inline]
do_IRQ+0xf1/0x190 arch/x86/kernel/irq.c:257
common_interrupt+0xf/0xf arch/x86/entry/entry_64.S:642
</IRQ>
RIP: 0010:native_safe_halt+0x6/0x10 arch/x86/include/asm/irqflags.h:54
Code: 48 89 df e8 5c 88 a3 fa e9 51 ff ff ff 4c 89 ef e8 4f 88 a3 fa
eb 9b 90 90 90 90 90 90 90 90 90 90 90 90 90 55 48 89 e5 fb f4 <5d> c3
0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 90 90 90 90 90
RSP: 0000:ffff88006bdcfc38 EFLAGS: 00000282 ORIG_RAX: ffffffffffffffde
RAX: dffffc0000000000 RBX: 1ffff1000d7b9f8a RCX: 0000000000000000
RDX: 1ffffffff10e2b78 RSI: 0000000000000001 RDI: ffffffff88715bc0
RBP: ffff88006bdcfc38 R08: ffffed000d9246d7 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
R13: ffff88006bdcfcf0 R14: 0000000000000000 R15: ffffffff88715bb0
arch_safe_halt arch/x86/include/asm/paravirt.h:94 [inline]
default_idle+0xbf/0x440 arch/x86/kernel/process.c:500
arch_cpu_idle+0x10/0x20 arch/x86/kernel/process.c:491
default_idle_call+0x36/0x90 kernel/sched/idle.c:93
cpuidle_idle_call kernel/sched/idle.c:153 [inline]
do_idle+0x345/0x560 kernel/sched/idle.c:262
cpu_startup_entry+0x104/0x120 kernel/sched/idle.c:368
start_secondary+0x42b/0x5b0 arch/x86/kernel/smpboot.c:265
secondary_startup_64+0xa5/0xb0 arch/x86/kernel/head_64.S:242
Allocated by task 1:
save_stack+0x43/0xd0 mm/kasan/kasan.c:448
set_track mm/kasan/kasan.c:460 [inline]
kasan_kmalloc+0xad/0xe0 mm/kasan/kasan.c:553
__do_kmalloc_node mm/slab.c:3682 [inline]
__kmalloc_node+0x47/0x70 mm/slab.c:3689
kmalloc_node include/linux/slab.h:555 [inline]
alloc_request_size+0x80/0x120 block/blk-core.c:844
mempool_alloc+0x16c/0x4b0 mm/mempool.c:384
__get_request block/blk-core.c:1424 [inline]
get_request+0xe04/0x2480 block/blk-core.c:1535
blk_queue_bio+0x5e9/0x1980 block/blk-core.c:2054
generic_make_request+0x919/0x1540 block/blk-core.c:2447
submit_bio+0x19a/0x4d0 block/blk-core.c:2555
submit_bio_wait+0x137/0x1e0 block/bio.c:977
blkdev_issue_flush+0x204/0x2f0 block/blk-flush.c:551
jbd2_journal_recover+0x3f3/0x4b0 fs/jbd2/recovery.c:289
jbd2_journal_load+0x256/0xca0 fs/jbd2/journal.c:1686
ext4_load_journal fs/ext4/super.c:4704 [inline]
ext4_fill_super+0x8e60/0xb700 fs/ext4/super.c:4076
mount_bdev+0x2ba/0x370 fs/super.c:1174
ext4_mount+0x34/0x40 fs/ext4/super.c:5771
mount_fs+0x6b/0x2d0 fs/super.c:1277
vfs_kern_mount.part.26+0xc6/0x4a0 fs/namespace.c:1037
vfs_kern_mount fs/namespace.c:2515 [inline]
do_new_mount fs/namespace.c:2518 [inline]
do_mount+0xea4/0x2bd0 fs/namespace.c:2848
ksys_mount+0xab/0x120 fs/namespace.c:3064
do_mount_root init/do_mounts.c:366 [inline]
mount_block_root+0x381/0x685 init/do_mounts.c:395
mount_root+0x287/0x2bb init/do_mounts.c:540
prepare_namespace+0x1cf/0x20c init/do_mounts.c:599
kernel_init_freeable+0x53f/0x55d init/main.c:1154
kernel_init+0x13/0x180 init/main.c:1061
ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412
Freed by task 0:
save_stack+0x43/0xd0 mm/kasan/kasan.c:448
set_track mm/kasan/kasan.c:460 [inline]
__kasan_slab_free+0x11a/0x170 mm/kasan/kasan.c:521
kasan_slab_free+0xe/0x10 mm/kasan/kasan.c:528
__cache_free mm/slab.c:3498 [inline]
kfree+0xd9/0x260 mm/slab.c:3813
free_request_size+0x5b/0x70 block/blk-core.c:859
mempool_free+0xda/0x350 mm/mempool.c:493
blk_free_request block/blk-core.c:1210 [inline]
__blk_put_request+0x513/0xc70 block/blk-core.c:1768
blk_finish_request+0x349/0x6a0 block/blk-core.c:3214
__blk_end_bidi_request+0xeb/0x1e0 block/blk-core.c:3278
__blk_end_request_all+0x129/0x270 block/blk-core.c:3368
blk_flush_complete_seq+0xf92/0x1810 block/blk-flush.c:208
flush_end_io+0x523/0xb50 block/blk-flush.c:257
blk_finish_request+0x3fa/0x6a0 block/blk-core.c:3209
scsi_end_request+0x3d3/0x820 drivers/scsi/scsi_lib.c:715
scsi_io_completion+0x1869/0x1910 drivers/scsi/scsi_lib.c:898
scsi_finish_command+0x4de/0x8d0 drivers/scsi/scsi.c:248
scsi_softirq_done+0x3ad/0x4b0 drivers/scsi/scsi_lib.c:1683
blk_done_softirq+0x48c/0x700 block/blk-softirq.c:37
__do_softirq+0x2e0/0xb94 kernel/softirq.c:284
The buggy address belongs to the object at ffff880063f876c0
which belongs to the cache kmalloc-1024 of size 1024
The buggy address is located 20 bytes inside of
1024-byte region [ffff880063f876c0, ffff880063f87ac0)
The buggy address belongs to the page:
page:ffffea00018fe180 count:1 mapcount:0 mapping:ffff88006c400ac0
index:0x0 compound_mapcount: 0
flags: 0x1fffc0000008100(slab|head)
raw: 01fffc0000008100 ffffea00018fe108 ffff88006c401848 ffff88006c400ac0
raw: 0000000000000000 ffff880063f86040 0000000100000007 0000000000000000
page dumped because: kasan: bad access detected
Memory state around the buggy address:
ffff880063f87580: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff880063f87600: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
>ffff880063f87680: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb
^
ffff880063f87700: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff880063f87780: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================
On 6/9/18 3:34 AM, Dmitry Vyukov wrote:
> Hi,
>
> Boot of mainline kernel is currently broken.
> On commit 7d3bf613e99abbd96ac7b90ee3694a246c975021.
> Config:
> https://gist.githubusercontent.com/dvyukov/9f7f1fd9e477efd85b221b3a21036c20/raw/7c56ede0840494b26045976960866f2b265c6f64/gistfile1.txt
> Should have been introduced very recently.
Can you try the below?
> You may need to also patch "umh: fix race condition", because that's
> another boot crasher currently present in tree.
Not sure that that refers to.
diff --git a/block/blk-flush.c b/block/blk-flush.c
index 058abdb50f31..ce41f666de3e 100644
--- a/block/blk-flush.c
+++ b/block/blk-flush.c
@@ -169,9 +169,11 @@ static bool blk_flush_complete_seq(struct request *rq,
struct request_queue *q = rq->q;
struct list_head *pending = &fq->flush_queue[fq->flush_pending_idx];
bool queued = false, kicked;
+ unsigned int cmd_flags;
BUG_ON(rq->flush.seq & seq);
rq->flush.seq |= seq;
+ cmd_flags = rq->cmd_flags;
if (likely(!error))
seq = blk_flush_cur_seq(rq);
@@ -212,7 +214,7 @@ static bool blk_flush_complete_seq(struct request *rq,
BUG();
}
- kicked = blk_kick_flush(q, fq, rq->cmd_flags);
+ kicked = blk_kick_flush(q, fq, cmd_flags);
return kicked | queued;
}
--
Jens Axboe
On Sat, Jun 09, 2018 at 06:33:06AM -0600, Jens Axboe wrote:
> On 6/9/18 3:34 AM, Dmitry Vyukov wrote:
> > Boot of mainline kernel is currently broken.
> > On commit 7d3bf613e99abbd96ac7b90ee3694a246c975021.
> > Config:
> > https://gist.githubusercontent.com/dvyukov/9f7f1fd9e477efd85b221b3a21036c20/raw/7c56ede0840494b26045976960866f2b265c6f64/gistfile1.txt
> > Should have been introduced very recently.
>
> Can you try the below?
Helps on my setup.
> > You may need to also patch "umh: fix race condition", because that's
> > another boot crasher currently present in tree.
>
> Not sure that that refers to.
I for one didn't trip into that.
> diff --git a/block/blk-flush.c b/block/blk-flush.c
> index 058abdb50f31..ce41f666de3e 100644
> --- a/block/blk-flush.c
> +++ b/block/blk-flush.c
> @@ -169,9 +169,11 @@ static bool blk_flush_complete_seq(struct request *rq,
> struct request_queue *q = rq->q;
> struct list_head *pending = &fq->flush_queue[fq->flush_pending_idx];
> bool queued = false, kicked;
> + unsigned int cmd_flags;
>
> BUG_ON(rq->flush.seq & seq);
> rq->flush.seq |= seq;
> + cmd_flags = rq->cmd_flags;
>
> if (likely(!error))
> seq = blk_flush_cur_seq(rq);
> @@ -212,7 +214,7 @@ static bool blk_flush_complete_seq(struct request *rq,
> BUG();
> }
>
> - kicked = blk_kick_flush(q, fq, rq->cmd_flags);
> + kicked = blk_kick_flush(q, fq, cmd_flags);
> return kicked | queued;
> }
>
Meow!
--
⢀⣴⠾⠻⢶⣦⠀ I've read an article about how lively happy music boosts
⣾⠁⢰⠒⠀⣿⡁ productivity. You can read it, too, you just need the
⢿⡄⠘⠷⠚⠋⠀ right music while doing so. I recommend Skepticism
⠈⠳⣄⠀⠀⠀⠀ (funeral doom metal).
On Sat, Jun 9, 2018 at 2:33 PM, Jens Axboe <[email protected]> wrote:
> On 6/9/18 3:34 AM, Dmitry Vyukov wrote:
>> Hi,
>>
>> Boot of mainline kernel is currently broken.
>> On commit 7d3bf613e99abbd96ac7b90ee3694a246c975021.
>> Config:
>> https://gist.githubusercontent.com/dvyukov/9f7f1fd9e477efd85b221b3a21036c20/raw/7c56ede0840494b26045976960866f2b265c6f64/gistfile1.txt
>> Should have been introduced very recently.
>
> Can you try the below?
Yes, this fixes the use-after-free:
Tested-by: Dmitry Vyukov <[email protected]>
>> You may need to also patch "umh: fix race condition", because that's
>> another boot crasher currently present in tree.
>
> Not sure that that refers to.
Currently there are 2 boot bugs present in upstream tree: this block
bug and an umh bug. I assumed that anybody who will be fixing the
block bug will want to first reproduce it and then test the fix
locally. But if one would try to do it, they will actually hit the umh
bug first. So I provided the fixing commit for the umh bug to simplify
things for whoever would be fixing this block bug.
> diff --git a/block/blk-flush.c b/block/blk-flush.c
> index 058abdb50f31..ce41f666de3e 100644
> --- a/block/blk-flush.c
> +++ b/block/blk-flush.c
> @@ -169,9 +169,11 @@ static bool blk_flush_complete_seq(struct request *rq,
> struct request_queue *q = rq->q;
> struct list_head *pending = &fq->flush_queue[fq->flush_pending_idx];
> bool queued = false, kicked;
> + unsigned int cmd_flags;
>
> BUG_ON(rq->flush.seq & seq);
> rq->flush.seq |= seq;
> + cmd_flags = rq->cmd_flags;
>
> if (likely(!error))
> seq = blk_flush_cur_seq(rq);
> @@ -212,7 +214,7 @@ static bool blk_flush_complete_seq(struct request *rq,
> BUG();
> }
>
> - kicked = blk_kick_flush(q, fq, rq->cmd_flags);
> + kicked = blk_kick_flush(q, fq, cmd_flags);
> return kicked | queued;
> }
>
>
> --
> Jens Axboe
>