All qemu's mount rootfs failed on Linux next-20230206 tag due to the following
kernel crash.
Reported-by: Linux Kernel Functional Testing <[email protected]>
Crash log:
---------
<3>[ 3.257960] /dev/root: Can't open blockdev
<4>[ 3.258940] VFS: Cannot open root device "/dev/sda" or
unknown-block(8,0): error -16
<4>[ 3.259704] Please append a correct "root=" boot option; here
are the available partitions:
<4>[ 3.261088] 0800 2500336 sda
<4>[ 3.261186] driver: sd
<4>[ 3.262260] 0b00 1048575 sr0
<4>[ 3.262409] driver: sr
<3>[ 3.263022] List of all bdev filesystems:
<3>[ 3.263553] ext3
<3>[ 3.263708] ext2
<3>[ 3.263994] ext4
<3>[ 3.264160] vfat
<3>[ 3.264419] msdos
<3>[ 3.264589] iso9660
<3>[ 3.264773]
<0>[ 3.265665] Kernel panic - not syncing: VFS: Unable to mount
root fs on unknown-block(8,0)
<4>[ 3.266991] CPU: 0 PID: 1 Comm: swapper/0 Not tainted
6.8.0-rc3-next-20240206 #1
<4>[ 3.267593] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009),
BIOS 1.16.3-debian-1.16.3-2 04/01/2014
<4>[ 3.268937] Call Trace:
<4>[ 3.269316] <TASK>
<4>[ 3.270113] dump_stack_lvl+0x71/0xb0
<4>[ 3.271837] dump_stack+0x14/0x20
<4>[ 3.272128] panic+0x12f/0x2f0
<4>[ 3.272812] ? _printk+0x5d/0x80
<4>[ 3.273097] mount_root_generic+0x26e/0x2b0
<4>[ 3.273941] mount_block_root+0x3f/0x50
<4>[ 3.274212] mount_root+0x60/0x80
<4>[ 3.274610] prepare_namespace+0x7a/0xb0
<4>[ 3.276008] kernel_init_freeable+0x137/0x180
<4>[ 3.276285] ? __pfx_kernel_init+0x10/0x10
<4>[ 3.276563] kernel_init+0x1e/0x1a0
<4>[ 3.276837] ret_from_fork+0x45/0x50
<4>[ 3.277319] ? __pfx_kernel_init+0x10/0x10
<4>[ 3.278176] ret_from_fork_asm+0x1a/0x30
<4>[ 3.278560] </TASK>
<0>[ 3.280750] Kernel Offset: 0x1a800000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
<0>[ 3.281985] ---[ end Kernel panic - not syncing: VFS: Unable to
mount root fs on unknown-block(8,0) ]---
Links:
- https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20240206/testrun/22547673/suite/log-parser-test/test/check-kernel-panic/log
- https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20240206/testrun/22547673/suite/log-parser-test/tests/
--
Linaro LKFT
https://lkft.linaro.org
On Tue 06-02-24 14:41:17, Naresh Kamboju wrote:
> All qemu's mount rootfs failed on Linux next-20230206 tag due to the following
> kernel crash.
>
> Reported-by: Linux Kernel Functional Testing <[email protected]>
>
> Crash log:
> ---------
> <3>[ 3.257960] /dev/root: Can't open blockdev
> <4>[ 3.258940] VFS: Cannot open root device "/dev/sda" or
> unknown-block(8,0): error -16
Uhuh, -16 is EBUSY so it seems Christian's block device opening changes are
suspect? Do you have some sample kconfig available somewhere?
Honza
> <4>[ 3.259704] Please append a correct "root=" boot option; here
> are the available partitions:
> <4>[ 3.261088] 0800 2500336 sda
> <4>[ 3.261186] driver: sd
> <4>[ 3.262260] 0b00 1048575 sr0
> <4>[ 3.262409] driver: sr
> <3>[ 3.263022] List of all bdev filesystems:
> <3>[ 3.263553] ext3
> <3>[ 3.263708] ext2
> <3>[ 3.263994] ext4
> <3>[ 3.264160] vfat
> <3>[ 3.264419] msdos
> <3>[ 3.264589] iso9660
> <3>[ 3.264773]
> <0>[ 3.265665] Kernel panic - not syncing: VFS: Unable to mount
> root fs on unknown-block(8,0)
> <4>[ 3.266991] CPU: 0 PID: 1 Comm: swapper/0 Not tainted
> 6.8.0-rc3-next-20240206 #1
> <4>[ 3.267593] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009),
> BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> <4>[ 3.268937] Call Trace:
> <4>[ 3.269316] <TASK>
> <4>[ 3.270113] dump_stack_lvl+0x71/0xb0
> <4>[ 3.271837] dump_stack+0x14/0x20
> <4>[ 3.272128] panic+0x12f/0x2f0
> <4>[ 3.272812] ? _printk+0x5d/0x80
> <4>[ 3.273097] mount_root_generic+0x26e/0x2b0
> <4>[ 3.273941] mount_block_root+0x3f/0x50
> <4>[ 3.274212] mount_root+0x60/0x80
> <4>[ 3.274610] prepare_namespace+0x7a/0xb0
> <4>[ 3.276008] kernel_init_freeable+0x137/0x180
> <4>[ 3.276285] ? __pfx_kernel_init+0x10/0x10
> <4>[ 3.276563] kernel_init+0x1e/0x1a0
> <4>[ 3.276837] ret_from_fork+0x45/0x50
> <4>[ 3.277319] ? __pfx_kernel_init+0x10/0x10
> <4>[ 3.278176] ret_from_fork_asm+0x1a/0x30
> <4>[ 3.278560] </TASK>
> <0>[ 3.280750] Kernel Offset: 0x1a800000 from 0xffffffff81000000
> (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
> <0>[ 3.281985] ---[ end Kernel panic - not syncing: VFS: Unable to
> mount root fs on unknown-block(8,0) ]---
>
>
> Links:
> - https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20240206/testrun/22547673/suite/log-parser-test/test/check-kernel-panic/log
> - https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20240206/testrun/22547673/suite/log-parser-test/tests/
>
> --
> Linaro LKFT
> https://lkft.linaro.org
>
--
Jan Kara <[email protected]>
SUSE Labs, CR
On Tue, 6 Feb 2024 at 15:45, Jan Kara <[email protected]> wrote:
>
> On Tue 06-02-24 14:41:17, Naresh Kamboju wrote:
> > All qemu's mount rootfs failed on Linux next-20230206 tag due to the following
> > kernel crash.
> >
> > Reported-by: Linux Kernel Functional Testing <[email protected]>
> >
> > Crash log:
> > ---------
> > <3>[ 3.257960] /dev/root: Can't open blockdev
> > <4>[ 3.258940] VFS: Cannot open root device "/dev/sda" or
> > unknown-block(8,0): error -16
>
> Uhuh, -16 is EBUSY so it seems Christian's block device opening changes are
> suspect? Do you have some sample kconfig available somewhere?
All build information is in this url,
https://storage.tuxsuite.com/public/linaro/lkft/builds/2byqguFVp7MYAEjKo6nJGba2FcP/
- Naresh
On Tue 06-02-24 15:53:34, Naresh Kamboju wrote:
> On Tue, 6 Feb 2024 at 15:45, Jan Kara <[email protected]> wrote:
> >
> > On Tue 06-02-24 14:41:17, Naresh Kamboju wrote:
> > > All qemu's mount rootfs failed on Linux next-20230206 tag due to the following
> > > kernel crash.
> > >
> > > Reported-by: Linux Kernel Functional Testing <[email protected]>
> > >
> > > Crash log:
> > > ---------
> > > <3>[ 3.257960] /dev/root: Can't open blockdev
> > > <4>[ 3.258940] VFS: Cannot open root device "/dev/sda" or
> > > unknown-block(8,0): error -16
> >
> > Uhuh, -16 is EBUSY so it seems Christian's block device opening changes are
> > suspect? Do you have some sample kconfig available somewhere?
>
> All build information is in this url,
> https://storage.tuxsuite.com/public/linaro/lkft/builds/2byqguFVp7MYAEjKo6nJGba2FcP/
Thanks! So for record the config has:
CONFIG_BLK_DEV_WRITE_MOUNTED=y
So we are not hitting any weird corner case with blocking writes to mounted
filesystems. It must be something else.
Honza
--
Jan Kara <[email protected]>
SUSE Labs, CR
On Tue, 6 Feb 2024 at 17:58, Jan Kara <[email protected]> wrote:
>
> On Tue 06-02-24 15:53:34, Naresh Kamboju wrote:
> > On Tue, 6 Feb 2024 at 15:45, Jan Kara <[email protected]> wrote:
> > >
> > > On Tue 06-02-24 14:41:17, Naresh Kamboju wrote:
> > > > All qemu's mount rootfs failed on Linux next-20230206 tag due to the following
> > > > kernel crash.
> > > >
> > > > Reported-by: Linux Kernel Functional Testing <[email protected]>
> > > >
> > > > Crash log:
> > > > ---------
> > > > <3>[ 3.257960] /dev/root: Can't open blockdev
> > > > <4>[ 3.258940] VFS: Cannot open root device "/dev/sda" or
> > > > unknown-block(8,0): error -16
> > >
> > > Uhuh, -16 is EBUSY so it seems Christian's block device opening changes are
> > > suspect? Do you have some sample kconfig available somewhere?
> >
> > All build information is in this url,
> > https://storage.tuxsuite.com/public/linaro/lkft/builds/2byqguFVp7MYAEjKo6nJGba2FcP/
>
> Thanks! So for record the config has:
>
> CONFIG_BLK_DEV_WRITE_MOUNTED=y
>
> So we are not hitting any weird corner case with blocking writes to mounted
> filesystems. It must be something else.
As per Anders bisection results the first bad commit pointing to
ba858e55b205 ("bdev: open block device as files")
- Naresh
> As per Anders bisection results the first bad commit pointing to
> ba858e55b205 ("bdev: open block device as files")
On it.
> On it.
Ok, can you try:
git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git vfs.super.debug
please?
Hi Christian,
On 06.02.2024 16:53, Christian Brauner wrote:
>> On it.
> Ok, can you try:
> git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git vfs.super.debug
> please?
I've also encountered this issue during my linux-next daily tests and I
confirm that the above branch works fine.
I've applied the diff between e3bfad989976^2 and the above branch
(bc7cb6c829e2), which looks following:
diff --git a/init/do_mounts.c b/init/do_mounts.c
index 279ad28bf4fb..d8ea839463a5 100644
--- a/init/do_mounts.c
+++ b/init/do_mounts.c
@@ -19,6 +19,7 @@
#include <linux/ramfs.h>
#include <linux/shmem_fs.h>
#include <linux/ktime.h>
+#include <linux/task_work.h>
#include <linux/nfs_fs.h>
#include <linux/nfs_fs_sb.h>
@@ -208,6 +209,10 @@ void __init mount_root_generic(char *name, char
*pretty_name, int flags)
goto out;
case -EACCES:
case -EINVAL:
+#ifdef CONFIG_BLOCK
+ flush_delayed_fput();
+ task_work_run();
+#endif
continue;
}
/*
onto next-20240206 and it fixed all boot problems I've observed on my
test farm. :)
Tested-by: Marek Szyprowski <[email protected]>
Best regards
--
Marek Szyprowski, PhD
Samsung R&D Institute Poland
On 2/7/2024 4:35 AM, Marek Szyprowski wrote:
> Hi Christian,
>
> On 06.02.2024 16:53, Christian Brauner wrote:
>>> On it.
>> Ok, can you try:
>> git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git vfs.super.debug
>> please?
>
>
> I've also encountered this issue during my linux-next daily tests and I
> confirm that the above branch works fine.
>
>
> I've applied the diff between e3bfad989976^2 and the above branch
> (bc7cb6c829e2), which looks following:
>
> diff --git a/init/do_mounts.c b/init/do_mounts.c
> index 279ad28bf4fb..d8ea839463a5 100644
> --- a/init/do_mounts.c
> +++ b/init/do_mounts.c
> @@ -19,6 +19,7 @@
> #include <linux/ramfs.h>
> #include <linux/shmem_fs.h>
> #include <linux/ktime.h>
> +#include <linux/task_work.h>
>
> #include <linux/nfs_fs.h>
> #include <linux/nfs_fs_sb.h>
> @@ -208,6 +209,10 @@ void __init mount_root_generic(char *name, char
> *pretty_name, int flags)
> goto out;
> case -EACCES:
> case -EINVAL:
> +#ifdef CONFIG_BLOCK
> + flush_delayed_fput();
> + task_work_run();
> +#endif
> continue;
> }
> /*
>
>
> onto next-20240206 and it fixed all boot problems I've observed on my
> test farm. :)
>
> Tested-by: Marek Szyprowski <[email protected]>
>
> Best regards
We as well encountered this issue during our CI run. The given branch
fixes the issues seen in our run.
Tested-by: Srikanth Aithal <[email protected]>