Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Thu, 28 Jan 2021 06:58:13 -0800
From:   "Paul E. McKenney" <paulmck@kernel.org>
To:     Dexuan Cui <decui@microsoft.com>
Cc:     Neeraj Upadhyay <neeraju@codeaurora.org>,
        "boqun.feng@gmail.com" <boqun.feng@gmail.com>,
        Ingo Molnar <mingo@redhat.com>,
        "rcu@vger.kernel.org" <rcu@vger.kernel.org>,
        vkuznets <vkuznets@redhat.com>,
        Michael Kelley <mikelley@microsoft.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: kdump always hangs in rcu_barrier() -> wait_for_completion()
Message-ID: <20210128145813.GO2743@paulmck-ThinkPad-P72>
Reply-To: paulmck@kernel.org
References: <SN6PR2101MB1807BDF049D7155201A8178DBFFA1@SN6PR2101MB1807.namprd21.prod.outlook.com>
 <20201126154630.GR1437@paulmck-ThinkPad-P72>
 <MW2PR2101MB18014505C01027A9486D45EEBFF91@MW2PR2101MB1801.namprd21.prod.outlook.com>
 <20201126214226.GS1437@paulmck-ThinkPad-P72>
 <MW2PR2101MB18011DA2FCF66D03BF8BB0CCBFF91@MW2PR2101MB1801.namprd21.prod.outlook.com>
 <20201126235440.GT1437@paulmck-ThinkPad-P72>
 <MW2PR2101MB1787FF5912C90FC4FFEF3993BFBA9@MW2PR2101MB1787.namprd21.prod.outlook.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <MW2PR2101MB1787FF5912C90FC4FFEF3993BFBA9@MW2PR2101MB1787.namprd21.prod.outlook.com>
User-Agent: Mutt/1.9.4 (2018-02-28)
Precedence: bulk

On Thu, Jan 28, 2021 at 07:28:20AM +0000, Dexuan Cui wrote:
> > From: Paul E. McKenney <paulmck@kernel.org>
> > Sent: Thursday, November 26, 2020 3:55 PM
> > To: Dexuan Cui <decui@microsoft.com>
> > Cc: boqun.feng@gmail.com; Ingo Molnar <mingo@redhat.com>;
> > rcu@vger.kernel.org; vkuznets <vkuznets@redhat.com>; Michael Kelley
> > <mikelley@microsoft.com>; linux-kernel@vger.kernel.org
> > Subject: Re: kdump always hangs in rcu_barrier() -> wait_for_completion()
> > 
> > On Thu, Nov 26, 2020 at 10:59:19PM +0000, Dexuan Cui wrote:
> > > > From: Paul E. McKenney <paulmck@kernel.org>
> > > > Sent: Thursday, November 26, 2020 1:42 PM
> > > >
> > > > > > Another possibility is that rcu_state.gp_kthread is non-NULL, but that
> > > > > > something else is preventing RCU grace periods from completing, but in
> > > > >
> > > > > It looks like somehow the scheduling is not working here: in rcu_barrier()
> > > > > , if I replace the wait_for_completion() with
> > > > > wait_for_completion_timeout(&rcu_state.barrier_completion, 30*HZ),
> > the
> > > > > issue persists.
> > > >
> > > > Have you tried using sysreq-t to see what the various tasks are doing?
> > >
> > > Will try it.
> > >
> > > BTW, this is a "Generation 2" VM on Hyper-V, meaning sysrq only starts to
> > > work after the Hyper-V para-virtualized keyboard driver loads... So, at this
> > > early point, sysrq is not working. :-( I'll have to hack the code and use a
> > > virtual NMI interrupt to force the sysrq handler to be called.
> > 
> > Whatever works!
> > 
> > > > Having interrupts disabled on all CPUs would have the effect of disabling
> > > > the RCU CPU stall warnings.
> > > > 							Thanx, Paul
> > >
> > > I'm sure the interrupts are not disabled. Here the VM only has 1 virtual CPU,
> > > and when the hang issue happens the virtual serial console is still responding
> > > when I press Enter (it prints a new line) or Ctrl+C (it prints ^C).
> > >
> > > Here the VM does not use the "legacy timers" (PIT, Local APIC timer, etc.) at
> > all.
> > > Instead, the VM uses the Hyper-V para-virtualized timers. It looks the
> > Hyper-V
> > > timer never fires in the kdump kernel when the hang issue happens. I'm
> > > looking into this... I suspect this hang issue may only be specific to Hyper-V.
> > 
> > Fair enough, given that timers not working can also suppress RCU CPU
> > stall warnings.  ;-)
> > 
> > 							Thanx, Paul
> 
> FYI: the issue has been fixed by this fix:
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fff7b5e6ee63c5d20406a131b260c619cdd24fd1

Thank you for the update!

							Thanx, Paul