Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Subject: Re: Panic on ppc64le using kernel 5.13.0-rc3
To:     Bruno Goncalves <bgoncalv@redhat.com>, linux-kernel@vger.kernel.org
Cc:     CKI Project <cki-project@redhat.com>
References: <CA+QYu4qxf2CYe2gC6EYnOHXPKS-+cEXL=MnUvqRFaN7W1i6ahQ@mail.gmail.com>
From:   Rasmus Villemoes <linux@rasmusvillemoes.dk>
Message-ID: <d13db73d-0806-00cd-ff84-5f5b03ffbef6@rasmusvillemoes.dk>
Date:   Thu, 10 Jun 2021 15:02:22 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.8.1
MIME-Version: 1.0
In-Reply-To: <CA+QYu4qxf2CYe2gC6EYnOHXPKS-+cEXL=MnUvqRFaN7W1i6ahQ@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 10/06/2021 13.47, Bruno Goncalves wrote:
> Hello,
> 
> We've observed in some cases kernel panic when trying to boot on
> ppc64le using a kernel based on 5.13.0-rc3. We are not sure if it
> could be related to patch
> https://lore.kernel.org/lkml/20210313212528.2956377-2-linux@rasmusvillemoes.dk/
> 

Thanks for the report. It's possible, but I'll need some help from you
to get more info.

First, can you send me the .config?

> 
> [    1.516075] wait_for_initramfs() called before rootfs_initcalls

This is likely because you have CONFIG_UEVENT_HELPER_PATH set to some
non-empty path (/sbin/hotplug perhaps). This did get reported once before:

https://lore.kernel.org/lkml/45556f52-cd2f-5512-ba65-81e4acee21ff@rasmusvillemoes.dk/

I think I should go and prepare a patch that moves the
usermodehelper_enable() call to after initramfs unpacking has been
initiated.

But until then, can you check if you do have UEVENT_HELPER_PATH set, and
if so, does changing it to the empty string make a change wrt this crash?


> [    1.559757] PCI: CLS 128 bytes, default 128
> [    1.560150] Trying to unpack rootfs image as initramfs...

OK, so now we got to populate_rootfs() and have kicked off a worker to
do the unpacking. Meanwhile, PID1 goes on to do other initcalls.

...

> [    1.764430] Initramfs unpacking failed: no cpio magic

Whoa, that's not good. Did something scramble over the initramfs memory
while it was being unpacked? It's been .2 seconds since the start of the
unpacking, so it's unlikely the very beginning of the initramfs is corrupt.

Can you try booting with initramfs_async=0 on the command line and see
if the kernel still crashes?

> [    1.766204] Freeing initrd memory: 18176K
...


> [    1.787649] Run /init as init process
> [    1.787793] Failed to execute /init (error -2)
> [    1.787801] Run /sbin/init as init process
> [    1.787842] Run /etc/init as init process
> [    1.787880] Run /bin/init as init process
> [    1.787921] Run /bin/sh as init process
> [    1.787978] Kernel panic - not syncing: No working init found.  Try

Yeah, well, this is expected when unpacking the initramfs failed.

So I think the problem is the "no cpio magic", i.e. the initramfs got
corrupted somehow. But I don't have any idea why that would happen -
freeing the initramfs memory only happens after unpacking is done
(naturally...).

Rasmus