Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 14 Jun 2023 12:22:59 +0100
From:   "Richard W.M. Jones" <rjones@redhat.com>
To:     Peter Zijlstra <peterz@infradead.org>
Cc:     Aaron Thompson <dev@aaront.org>, linux-kernel@vger.kernel.org
Subject: Re: printk.time causes rare kernel boot hangs
Message-ID: <20230614112259.GC7636@redhat.com>
References: <20230613134105.GA10301@redhat.com>
 <20230614092158.GF1639749@hirez.programming.kicks-ass.net>
 <20230614094522.GA7636@redhat.com>
 <20230614111644.GH1639749@hirez.programming.kicks-ass.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20230614111644.GH1639749@hirez.programming.kicks-ass.net>
User-Agent: Mutt/1.5.21 (2010-09-15)
Precedence: bulk

On Wed, Jun 14, 2023 at 01:16:44PM +0200, Peter Zijlstra wrote:
> On Wed, Jun 14, 2023 at 10:45:22AM +0100, Richard W.M. Jones wrote:
> > Debian should work too actually, just run the following command until
> > it hangs:
> > 
> > > >   $ while guestfish -a /dev/null -v run >& /tmp/log; do echo -n . ; done
> 
> What kernel with guestfish use?

I think it's easier to test using the qemu command I suggested in a
later email.  (The command line is very long but should be simple to
understand.)  That way you can just use a local vmlinux or vmlinuz
file:

https://lkml.org/lkml/2023/6/14/491

Note for gdb you will need to add '-s' to the qemu command line as well.

Unfortunately since the hang is so rare it's not really possible to
attach gdb at the beginning, since you'd have to do hundreds of runs
before seeing the problem.  The stack trace seems like it isn't very
interesting (since the kernel still appears to be running while "hung").

If you have any other suggestions for things I can try please let me
know - this is a pretty important issue for us as it breaks CI tests,
so I can try whatever you like.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org