Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 14 Jun 2023 13:35:36 +0200
From:   Peter Zijlstra <peterz@infradead.org>
To:     "Richard W.M. Jones" <rjones@redhat.com>
Cc:     Aaron Thompson <dev@aaront.org>, linux-kernel@vger.kernel.org
Subject: Re: printk.time causes rare kernel boot hangs
Message-ID: <20230614113536.GJ1639749@hirez.programming.kicks-ass.net>
References: <20230613134105.GA10301@redhat.com>
 <20230614092158.GF1639749@hirez.programming.kicks-ass.net>
 <20230614094522.GA7636@redhat.com>
 <20230614103011.GL7912@redhat.com>
 <20230614103953.GM7912@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20230614103953.GM7912@redhat.com>
Precedence: bulk

On Wed, Jun 14, 2023 at 11:39:53AM +0100, Richard W.M. Jones wrote:
> Got it!
> 
> #0  arch_static_branch (branch=false, key=<optimized out>)
>     at ./arch/x86/include/asm/jump_label.h:27
> #1  static_key_false (key=<optimized out>) at ./include/linux/jump_label.h:207
> #2  native_write_msr (msr=1760, low=1876580734, high=106)
>     at ./arch/x86/include/asm/msr.h:147
> #3  0xffffffff8107997c in paravirt_write_msr (high=<optimized out>, 
>     low=1876580734, msr=1760) at ./arch/x86/include/asm/paravirt.h:196
> #4  wrmsrl (val=<optimized out>, msr=1760)
>     at ./arch/x86/include/asm/paravirt.h:229
> #5  lapic_next_deadline (delta=<optimized out>, evt=<optimized out>)
>     at arch/x86/kernel/apic/apic.c:491
> #6  0xffffffff811f7b1d in clockevents_program_event (dev=0xffff88804e820dc0, 
>     expires=<optimized out>, force=<optimized out>)
>     at kernel/time/clockevents.c:334
> #7  0xffffffff811f81b0 in tick_handle_periodic (dev=0xffff88804e820dc0)
>     at kernel/time/tick-common.c:133
> #8  0xffffffff810796c1 in local_apic_timer_interrupt ()
>     at arch/x86/kernel/apic/apic.c:1095
> #9  __sysvec_apic_timer_interrupt (regs=regs@entry=0xffffc90000003ee8)
>     at arch/x86/kernel/apic/apic.c:1112
> #10 0xffffffff81f9cf09 in sysvec_apic_timer_interrupt (regs=0xffffc90000003ee8)
>     at arch/x86/kernel/apic/apic.c:1106
> #11 0xffffffff820015ca in asm_sysvec_apic_timer_interrupt ()
>     at ./arch/x86/include/asm/idtentry.h:645
> #12 0x0000000000000000 in ?? ()

Uuuhh.. something is really fishy here. The thing in common between the
fingered patch and this stacktrace is the jump_label/static_branch
usage, but they're quite different paths.

There is no printk or local_clock() in sight here.

I've got that plain qemu thing running on:

  defconfig + kvm_guest.config + CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y

and have added "nokaslr" to the -append string. Lets see if it wants to
go wobbly for me.