Hello,
syzbot found the following issue on:
HEAD commit: a5541c0811a0 Merge branch 'for-next/core' into for-kernelci
git tree: git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
console output: https://syzkaller.appspot.com/x/log.txt?x=1560b830480000
kernel config: https://syzkaller.appspot.com/x/.config?x=cbd4e584773e9397
dashboard link: https://syzkaller.appspot.com/bug?extid=9743a41f74f00e50fc77
compiler: Debian clang version 13.0.1-++20220126092033+75e33f71c2da-1~exp1~20220126212112.63, GNU ld (GNU Binutils for Debian) 2.35.2
userspace arch: arm64
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=15e320b3880000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=147c0577880000
Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/4b7702208fb9/disk-a5541c08.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/9ec0153ec051/vmlinux-a5541c08.xz
kernel image: https://storage.googleapis.com/syzbot-assets/6f8725ad290a/Image-a5541c08.gz.xz
mounted in repro: https://storage.googleapis.com/syzbot-assets/93008694e408/mount_0.gz
IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: [email protected]
------------[ cut here ]------------
VFS: brelse: Trying to free free buffer
WARNING: CPU: 0 PID: 0 at fs/buffer.c:1145 __brelse fs/buffer.c:1145 [inline]
WARNING: CPU: 0 PID: 0 at fs/buffer.c:1145 brelse include/linux/buffer_head.h:326 [inline]
WARNING: CPU: 0 PID: 0 at fs/buffer.c:1145 __invalidate_bh_lrus fs/buffer.c:1380 [inline]
WARNING: CPU: 0 PID: 0 at fs/buffer.c:1145 invalidate_bh_lru+0xa0/0x134 fs/buffer.c:1393
Modules linked in:
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.1.0-rc8-syzkaller-33330-ga5541c0811a0 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/26/2022
pstate: 604000c5 (nZCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : __brelse fs/buffer.c:1145 [inline]
pc : brelse include/linux/buffer_head.h:326 [inline]
pc : __invalidate_bh_lrus fs/buffer.c:1380 [inline]
pc : invalidate_bh_lru+0xa0/0x134 fs/buffer.c:1393
lr : __brelse fs/buffer.c:1145 [inline]
lr : brelse include/linux/buffer_head.h:326 [inline]
lr : __invalidate_bh_lrus fs/buffer.c:1380 [inline]
lr : invalidate_bh_lru+0xa0/0x134 fs/buffer.c:1393
sp : ffff800008003e50
x29: ffff800008003e50 x28: ffff80000d2d42f0 x27: ffff80000d2d42e8
x26: ffff800008648d54 x25: ffff0000c054e2a0 x24: 00000000ffffffff
x23: ffff0001fefcc830 x22: 0000000000000000 x21: 0000000000000001
x20: 0000000000000000 x19: ffff80000cbe89d6 x18: 000000000000035e
x17: ffff8001f1cee000 x16: ffff80000dbe6158 x15: ffff80000d39bc80
x14: 0000000000000000 x13: 00000000ffffffff x12: ffff80000d39bc80
x11: ff808000081c4d64 x10: 0000000000010002 x9 : ad49151c1e37eb00
x8 : ad49151c1e37eb00 x7 : ffff80000c091ebc x6 : 0000000000000000
x5 : 0000000000000080 x4 : 0000000000000001 x3 : 0000000000000000
x2 : 0000000000000000 x1 : 0000000100010002 x0 : 0000000000000027
Call trace:
__brelse fs/buffer.c:1145 [inline]
brelse include/linux/buffer_head.h:326 [inline]
__invalidate_bh_lrus fs/buffer.c:1380 [inline]
invalidate_bh_lru+0xa0/0x134 fs/buffer.c:1393
__flush_smp_call_function_queue+0x26c/0x8d8 kernel/smp.c:630
generic_smp_call_function_single_interrupt+0x28/0xfc kernel/smp.c:546
do_handle_IPI arch/arm64/kernel/smp.c:876 [inline]
ipi_handler+0x108/0x1a8 arch/arm64/kernel/smp.c:922
handle_percpu_devid_irq+0xb0/0x1cc kernel/irq/chip.c:930
generic_handle_irq_desc include/linux/irqdesc.h:158 [inline]
handle_irq_desc kernel/irq/irqdesc.c:648 [inline]
generic_handle_domain_irq+0x4c/0x6c kernel/irq/irqdesc.c:704
__gic_handle_irq drivers/irqchip/irq-gic-v3.c:695 [inline]
__gic_handle_irq_from_irqson drivers/irqchip/irq-gic-v3.c:746 [inline]
gic_handle_irq+0x78/0x1b4 drivers/irqchip/irq-gic-v3.c:790
call_on_irq_stack+0x2c/0x54 arch/arm64/kernel/entry.S:892
do_interrupt_handler+0x7c/0xc0 arch/arm64/kernel/entry-common.c:274
__el1_irq arch/arm64/kernel/entry-common.c:471 [inline]
el1_interrupt+0x34/0x68 arch/arm64/kernel/entry-common.c:486
el1h_64_irq_handler+0x18/0x24 arch/arm64/kernel/entry-common.c:491
el1h_64_irq+0x64/0x68 arch/arm64/kernel/entry.S:580
arch_local_irq_enable+0xc/0x18 arch/arm64/include/asm/irqflags.h:35
default_idle_call+0x48/0xb8 kernel/sched/idle.c:109
cpuidle_idle_call kernel/sched/idle.c:191 [inline]
do_idle+0x110/0x2d4 kernel/sched/idle.c:303
cpu_startup_entry+0x24/0x28 kernel/sched/idle.c:400
kernel_init+0x0/0x290 init/main.c:729
start_kernel+0x0/0x620 init/main.c:890
start_kernel+0x450/0x620 init/main.c:1145
__primary_switched+0xb4/0xbc arch/arm64/kernel/head.S:471
irq event stamp: 114628
hardirqs last enabled at (114627): [<ffff80000c096a3c>] default_idle_call+0x34/0xb8 kernel/sched/idle.c:106
hardirqs last disabled at (114628): [<ffff80000c084174>] __el1_irq arch/arm64/kernel/entry-common.c:468 [inline]
hardirqs last disabled at (114628): [<ffff80000c084174>] el1_interrupt+0x24/0x68 arch/arm64/kernel/entry-common.c:486
softirqs last enabled at (114616): [<ffff8000080102e4>] _stext+0x2e4/0x37c
softirqs last disabled at (114581): [<ffff800008017c88>] ____do_softirq+0x14/0x20 arch/arm64/kernel/irq.c:80
---[ end trace 0000000000000000 ]---
---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at [email protected].
syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
syzbot can test patches for this issue, for details see:
https://goo.gl/tpsmEJ#testing-patches
> On Wed, Dec 21, 2022 at 06:57:38PM -0800, syzbot wrote:
>> Hello,
>>
>> syzbot found the following issue on:
>>
>> HEAD commit: a5541c0811a0 Merge branch 'for-next/core' into for-kernelci
>> git tree: git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
>> console output: https://syzkaller.appspot.com/x/log.txt?x=1560b830480000
>> kernel config: https://syzkaller.appspot.com/x/.config?x=cbd4e584773e9397
>> dashboard link: https://syzkaller.appspot.com/bug?extid=9743a41f74f00e50fc77
>> compiler: Debian clang version 13.0.1-++20220126092033+75e33f71c2da-1~exp1~20220126212112.63, GNU ld (GNU Binutils for Debian) 2.35.2
>> userspace arch: arm64
>> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=15e320b3880000
>> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=147c0577880000
>
> #syz set subsystems: sysv, udf
The specified label value is incorrect.
"sysv" is not among the allowed values.
Please use one of the supported label values.
The following labels are suported:
no-reminders, prio: {low, normal, high}, subsystems: {.. see below ..}
The list of subsystems: https://syzkaller.appspot.com/upstream/subsystems?all=true
>
> There are two reproducers, one that mounts a sysv file system, and the
> other which mounts a udf file system. There is no mention of ext4 in
> the stack trace, and yet syzbot has assigned this to the ext4
> subsystem for some unknown reason.
>
> - Ted
On Wed, Dec 21, 2022 at 06:57:38PM -0800, syzbot wrote:
> Hello,
>
> syzbot found the following issue on:
>
> HEAD commit: a5541c0811a0 Merge branch 'for-next/core' into for-kernelci
> git tree: git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
> console output: https://syzkaller.appspot.com/x/log.txt?x=1560b830480000
> kernel config: https://syzkaller.appspot.com/x/.config?x=cbd4e584773e9397
> dashboard link: https://syzkaller.appspot.com/bug?extid=9743a41f74f00e50fc77
> compiler: Debian clang version 13.0.1-++20220126092033+75e33f71c2da-1~exp1~20220126212112.63, GNU ld (GNU Binutils for Debian) 2.35.2
> userspace arch: arm64
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=15e320b3880000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=147c0577880000
#syz set subsystems: sysv, udf
There are two reproducers, one that mounts a sysv file system, and the
other which mounts a udf file system. There is no mention of ext4 in
the stack trace, and yet syzbot has assigned this to the ext4
subsystem for some unknown reason.
- Ted
#syz set subsystems: udf
There are two reproducers, one that mounts a sysv file system, and the
other which mounts a udf file system. There is no mention of ext4 in
the stack trace, and yet syzbot has assigned this to the ext4
subsystem for some unknown reason.
- Ted
Hi Ted,
On Sun, Apr 30, 2023 at 8:40 AM Theodore Ts'o <[email protected]> wrote:
>
> On Wed, Dec 21, 2022 at 06:57:38PM -0800, syzbot wrote:
> > Hello,
> >
> > syzbot found the following issue on:
> >
> > HEAD commit: a5541c0811a0 Merge branch 'for-next/core' into for-kernelci
> > git tree: git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci
> > console output: https://syzkaller.appspot.com/x/log.txt?x=1560b830480000
> > kernel config: https://syzkaller.appspot.com/x/.config?x=cbd4e584773e9397
> > dashboard link: https://syzkaller.appspot.com/bug?extid=9743a41f74f00e50fc77
> > compiler: Debian clang version 13.0.1-++20220126092033+75e33f71c2da-1~exp1~20220126212112.63, GNU ld (GNU Binutils for Debian) 2.35.2
> > userspace arch: arm64
> > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=15e320b3880000
> > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=147c0577880000
>
> #syz set subsystems: sysv, udf
>
> There are two reproducers, one that mounts a sysv file system, and the
> other which mounts a udf file system. There is no mention of ext4 in
> the stack trace, and yet syzbot has assigned this to the ext4
> subsystem for some unknown reason.
In this particular case, there were two ext4-related crashes as well:
https://syzkaller.appspot.com/text?tag=CrashReport&x=14c7dd1b480000
https://syzkaller.appspot.com/text?tag=CrashReport&x=1153a07f480000
I think syzbot picked a too generic frame as a bug title and it just
got confused by crashes belonging to different filesystems.
Maybe we need to display subsystems for individual crashes as well, so
that it's easier for a human to understand what's going on..
--
Aleksandr
>
> - Ted
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/ZE4NVo6rTOeGQdK%2B%40mit.edu.
On Mon, May 01, 2023 at 11:18:28AM +0200, Aleksandr Nogikh wrote:
> >
> > There are two reproducers, one that mounts a sysv file system, and the
> > other which mounts a udf file system. There is no mention of ext4 in
> > the stack trace, and yet syzbot has assigned this to the ext4
> > subsystem for some unknown reason.
>
> In this particular case, there were two ext4-related crashes as well:
>
> https://syzkaller.appspot.com/text?tag=CrashReport&x=14c7dd1b480000
> https://syzkaller.appspot.com/text?tag=CrashReport&x=1153a07f480000
>
> I think syzbot picked a too generic frame as a bug title and it just
> got confused by crashes belonging to different filesystems.
> Maybe we need to display subsystems for individual crashes as well, so
> that it's easier for a human to understand what's going on..
Hmm.... the the two ext4-related crashes have very different stack
traces. In the first, there was some ext4 functions on the stack
trace, but those are not relevant, since we take an interrupt, and it
was apparently an arm64 IPI, which then results in the call to
invalidate_bh_lrus(), which then triggers the warning. So this
*might* be related to ext4, but it could be any other file system that
could have been mounted at the same time, since syzbot runs multiple
streams of system calls in parallel.
In the second stack trace, we were in the middle of unmounting the
file system, and ext4_but_super() called invalidate_bdev(), which in
turn called invalidate_bh_lrus(), and that's when the buffer head
layer finds a buffer head whose refcount is zero, and that triggers
the:
VFS: brelse: Trying to free free buffer
... which then leads to the WARNING in invalidate_bh_lru().
So yes, the problem is that syzbot chose generic description based on
the warning. An analogy might be a medical examiner in a police
procedural TV show[1] having seen a dozen corpses that were shot in
the torso, leading to them bleeding out and dying, erroneously
concluding that all six homocides *must* be related. In fact, the
size of the bullet might be different, and who might have been holding
the gun might be very different.
[1] https://www.youtube.com/watch?v=lMalvNeJFLk
Worse, even if we did a better job disambiguating based on the stack
trace, without a reliable reproducer, the stack trace is of very
limited utility for trying to track down this kind of bug. Going back
to the crime scene analogy, the stack trace basically tells us where
the body was found. It doesn't tell us much about who might have been
fired the bullet. That is, we need to know who called brelse() or
put_bh() on the buffer_head one too many times, leading to the report
that there was an attempt to free a free buffer.
(Fortunately, such bugs a relatively rare, although as this syzbot
report demonstrates, some still exist --- if I had to guess, the bugs
are on some error handling path which is very rarely hit.)
Perhaps syzbot could special case handling for those objects (like
struct buffer_head) which have a refcount and where the refcount can
go negative, and treat functions that manipulate such objects much
like how syzbot handles KASAN errors? The tricky bit is the immediate
caller of the invalidate_bh_lrus() is not necessarily the best one to
use. For example, invalidate_bdev() calls invalidate_bh_lrus(), and
it's the caller of invalidate_bdev() which is interesting in this case
--- e.g., ext4_put_super().
In the first stack trace, nothing in the stack trace is helpful, so
there may not much we can do debug this unless we can get a reproducer
that reproduces this particular stack trace.
In general, though, syzbot should disregard any functions after the
functions indicating that an interrupt had taken place, since which
code was interrupted when the IRQ came in is just an innocent victim.
For now, given that we *do* have a two reproducers, one involving sysv
and the other udf file systems, it's probably best that we try to let
the maintainers of those file systems try to figure out what be going
on.
- Ted
syzbot has bisected this issue to:
commit f6e2c20ca7604e6a267c93a511d19dda72573be1
Author: Liu Shixin <[email protected]>
Date: Fri Apr 29 21:38:04 2022 +0000
fs: sysv: check sbi->s_firstdatazone in complete_read_super
bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=13eed371280000
start commit: 4ecd704a4c51 tpm, tpm_tis: correct tpm_tis_flags enumerati..
git tree: upstream
final oops: https://syzkaller.appspot.com/x/report.txt?x=101ed371280000
console output: https://syzkaller.appspot.com/x/log.txt?x=17eed371280000
kernel config: https://syzkaller.appspot.com/x/.config?x=162cf2103e4a7453
dashboard link: https://syzkaller.appspot.com/bug?extid=9743a41f74f00e50fc77
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=16ebf8c9280000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=12df6ab6280000
Reported-by: [email protected]
Fixes: f6e2c20ca760 ("fs: sysv: check sbi->s_firstdatazone in complete_read_super")
For information about bisection process see: https://goo.gl/tpsmEJ#bisection
syzbot suspects this issue was fixed by commit:
commit 6f861765464f43a71462d52026fbddfc858239a5
Author: Jan Kara <[email protected]>
Date: Wed Nov 1 17:43:10 2023 +0000
fs: Block writes to mounted block devices
bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=14a093ce180000
start commit: 9a3dad63edbe Merge tag '6.6-rc5-ksmbd-server-fixes' of git..
git tree: upstream
kernel config: https://syzkaller.appspot.com/x/.config?x=11e478e28144788c
dashboard link: https://syzkaller.appspot.com/bug?extid=9743a41f74f00e50fc77
syz repro: https://syzkaller.appspot.com/x/repro.syz?x=16ebc3c5680000
C reproducer: https://syzkaller.appspot.com/x/repro.c?x=122e8275680000
If the result looks correct, please mark the issue as fixed by replying with:
#syz fix: fs: Block writes to mounted block devices
For information about bisection process see: https://goo.gl/tpsmEJ#bisection
On Sat 02-03-24 04:14:04, syzbot wrote:
> syzbot suspects this issue was fixed by commit:
>
> commit 6f861765464f43a71462d52026fbddfc858239a5
> Author: Jan Kara <[email protected]>
> Date: Wed Nov 1 17:43:10 2023 +0000
>
> fs: Block writes to mounted block devices
>
> bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=14a093ce180000
> start commit: 9a3dad63edbe Merge tag '6.6-rc5-ksmbd-server-fixes' of git..
> git tree: upstream
> kernel config: https://syzkaller.appspot.com/x/.config?x=11e478e28144788c
> dashboard link: https://syzkaller.appspot.com/bug?extid=9743a41f74f00e50fc77
> syz repro: https://syzkaller.appspot.com/x/repro.syz?x=16ebc3c5680000
> C reproducer: https://syzkaller.appspot.com/x/repro.c?x=122e8275680000
>
> If the result looks correct, please mark the issue as fixed by replying with:
UDF is fixed but sysv still seems to trigger the warning so:
#syz set subsystems: fs
(we don't seem to have separate category for sysv bugs).
Honza
--
Jan Kara <[email protected]>
SUSE Labs, CR