Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 14 Jun 2023 11:21:58 +0200
From:   Peter Zijlstra <peterz@infradead.org>
To:     "Richard W.M. Jones" <rjones@redhat.com>
Cc:     Aaron Thompson <dev@aaront.org>, linux-kernel@vger.kernel.org
Subject: Re: printk.time causes rare kernel boot hangs
Message-ID: <20230614092158.GF1639749@hirez.programming.kicks-ass.net>
References: <20230613134105.GA10301@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20230613134105.GA10301@redhat.com>
Precedence: bulk

On Tue, Jun 13, 2023 at 02:41:05PM +0100, Richard W.M. Jones wrote:
> [Being tracked in this bug which contains much more detail:
> https://gitlab.com/qemu-project/qemu/-/issues/1696 ]

Can I please just get the detail in mail instead of having to go look at
random websites?

> Recent kernels hang rarely when booted on qemu.  Usually you need to
> boot 100s or 1,000s of times to see the hang, compared to 292,612 [sic]
> successful boots which I was able to do before the problematic commit.
> 
> A reproducer (you'll probably need to use Fedora) is:

Debian only shop here... in fact, I still have machines without systemd.

>   $ while guestfish -a /dev/null -v run >& /tmp/log; do echo -n . ; done
> 
> You will need to leave it running for probably several hours, and
> examine the /tmp/log file at the end.
> 
> I tracked this down to the following commit:
> 
>   commit f31dcb152a3d0816e2f1deab4e64572336da197d
>   Author: Aaron Thompson <dev@aaront.org>
>   Date:   Thu Apr 13 17:50:12 2023 +0000
> 
>     sched/clock: Fix local_clock() before sched_clock_init()
>     
>     Have local_clock() return sched_clock() if sched_clock_init() has not
>     yet run. sched_clock_cpu() has this check but it was not included in the
>     new noinstr implementation of local_clock().
> 
>   (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f31dcb152a3d0816e2f1deab4e64572336da197d)
> 
> Reverting this commit fixes the problem.
> 
> I don't know _why_ this commit is wrong, but can we revert it as it
> causes serious problems with libguestfs hanging randomly.
> 
> Or if there's anything you want me to try out then let me know,
> because I can reproduce the problem locally quite easily.

Well, since it's virt and all, can you attach gdb to the gdb-stub and
see where it's at? Any clue is better than no clue.