Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Mon, 24 May 2021 16:46:23 +0100
Message-ID: <87eedww2wg.wl-maz@kernel.org>
From:   Marc Zyngier <maz@kernel.org>
To:     Andreas =?UTF-8?B?RsOkcmJlcg==?= <afaerber@suse.de>
Cc:     linux-rockchip@lists.infradead.org,
        linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org,
        Rob Herring <robh+dt@kernel.org>,
        Heiko Stuebner <heiko@sntech.de>, devicetree@vger.kernel.org
Subject: Re: [PATCH RFC 5/9] arm64: dts: rockchip: rk1808k-toybrick-m0: Suppress vGIC interrupt
In-Reply-To: <f9523a3c-16a7-16d1-4d2e-9994c3fe03d9@suse.de>
References: <20210516230551.12469-1-afaerber@suse.de>
        <20210516230551.12469-6-afaerber@suse.de>
        <87fsylvhck.wl-maz@kernel.org>
        <f9523a3c-16a7-16d1-4d2e-9994c3fe03d9@suse.de>
User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue)
 FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1
 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO)
MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue")
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8BIT
Precedence: bulk

On Mon, 24 May 2021 15:40:22 +0100,
Andreas Färber <afaerber@suse.de> wrote:
> 
> On 17.05.21 11:29, Marc Zyngier wrote:
> > On Mon, 17 May 2021 00:05:47 +0100,
> > Andreas Färber <afaerber@suse.de> wrote:
> >>
> >> Avoid the kernel getting stuck after:
> >>
> >> [    1.175956] kvm [1]: IPA Size Limit: 40 bits
> >> [    1.177164] kvm [1]: vgic-v2@ff320000
> >> [    1.177545] kvm [1]: GIC system register CPU interface enabled
> >>
> >> or when dropping GICV reg entry:
> >>
> >> [    1.176001] kvm [1]: IPA Size Limit: 40 bits
> >> [    1.177191] kvm [1]: GICv3: no GICV resource entry
> >> [    1.177664] kvm [1]: disabling GICv2 emulation
> >> [    1.178115] kvm [1]: GIC system register CPU interface enabled
> >>
> >> Signed-off-by: Andreas Färber <afaerber@suse.de>
> >> ---
> >>  arch/arm64/boot/dts/rockchip/rk1808k-toybrick-m0.dts | 4 ++++
> >>  1 file changed, 4 insertions(+)
> >>
> >> diff --git a/arch/arm64/boot/dts/rockchip/rk1808k-toybrick-m0.dts b/arch/arm64/boot/dts/rockchip/rk1808k-toybrick-m0.dts
> >> index 2f8075d2391c..15293a8576c6 100644
> >> --- a/arch/arm64/boot/dts/rockchip/rk1808k-toybrick-m0.dts
> >> +++ b/arch/arm64/boot/dts/rockchip/rk1808k-toybrick-m0.dts
> >> @@ -48,6 +48,10 @@ &cpu1 {
> >>  	cpu-supply = <&vdd_cpu>;
> >>  };
> >>  
> >> +&gic {
> >> +	/delete-property/ interrupts;
> >> +};
> >> +
> >>  &uart2 {
> >>  	status = "okay";
> >>  	clocks = <&xin24m>;
> > 
> > As I said in my reply to the cover letter, this is not an acceptable
> > outcome. Please add some debug to kvm_vgic_hyp_init() to understand
> > where this is hanging and why.
> 
> Many thanks for that pointer.
> 
> So, as alternative to dropping the DT interrupts property above, I could
> also work around this issue by commenting out
> vgic-init.c:vgic_init_cpu_starting()'s enable_percpu_irq() call.
> 
> Otherwise I am seeing the following call flow:
> 
> cpuhp_setup_state() -> __cpuhp_setup_state_cpuslocked() ->
> cpuhp_issue_call() -> cpuhp_invoke_ap_callback() -> __cpuhp_kick_ap() ->
> wait_for_ap_thread() -> wait_for_completion() --- doesn't return
> 
> With kvm_info() / printk():
> 
> [    1.244079] kvm [1]: IPA Size Limit: 40 bits
> 
> [    1.245205] kvm [1]: vgic-v2@ff320000
> 
> [    1.245584] kvm [1]: GIC system register CPU interface enabled
> 
> [    1.246177] kvm [1]: before cpuhp_setup_state
> 
> [    1.246605] __cpuhp_setup_state_cpuslocked: kvm/arm/vgic:starting
> 
> [    1.247198] __cpuhp_setup_state_cpuslocked: for_each_present_cpu 0:
> state 225
> 
> [    1.247933] __cpuhp_setup_state_cpuslocked: for_each_present_cpu 0:
> before cpuhp_issue_call
> 
> [    1.248745] cpuhp_issue_call: before invoke
> 
> [    1.249154] cpuhp_issue_call: before AP invoke
> 
> [    1.249585] cpuhp_invoke_ap_callback
> 
> [    1.249936] cpuhp_invoke_ap_callback: after cpu_online
> 
> [    1.250435] cpuhp_invoke_ap_callback: before st->thread
> 
> [    1.250944] cpuhp_invoke_ap_callback: after st->thread
> 
> [    1.251445] __cpuhp_kick_ap
> 
> [    1.251731] __cpuhp_kick_ap: not returned
> 
> [    1.252140] vgic_init_cpu_starting: 9
> 
> [    1.252507] vgic_init_cpu_starting: done
> 
> [    1.255538] __cpuhp_kick_ap: wait_for_ap_thread

And you never see any RCU stall after that? It looks like a CPU has
disappeared in the weeds after enabling the per-CPU interrupt.

Please instrument what happens in
drivers/irqchip/irq-gic-v3.c::gic_unmask_irq() when d->hwirq == 9, a
well as vgic_maintenance_handler(), just in case it gets called...

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.