Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 14 Jun 2023 11:14:14 +0200
From:   "gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>
To:     "Bhatnagar, Rishabh" <risbhat@amazon.com>
Cc:     bigeasy@linutronix.de,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "tglx@linutronix.de" <tglx@linutronix.de>,
        "sashal@kernel.org" <sashal@kernel.org>, luizcap@amazon.com,
        abuehaze@amazon.com,
        "stable@vger.kernel.org" <stable@vger.kernel.org>
Subject: Re: Observing RCU stalls in kernel 5.4/5.10/5.15/6.1 stable trees
Message-ID: <2023061457-king-broadcast-f47e@gregkh>
References: <12c6f9a3-d087-b824-0d05-0d18c9bc1bf3@amazon.com>
 <c4724b40-89f6-5aa7-720d-c4a4af57cf45@amazon.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <c4724b40-89f6-5aa7-720d-c4a4af57cf45@amazon.com>
Precedence: bulk

On Tue, Jun 13, 2023 at 11:58:05AM -0700, Bhatnagar, Rishabh wrote:
> 
> On 6/13/23 11:49 AM, Bhatnagar, Rishabh wrote:
> > Hi Sebastian/Greg
> > 
> > We are seeing RCU stall warnings from recent stable tree updates:
> > 5.4.243, 5.10.180, 5.15.113, 6.1.31 onwards.
> > This is seen in the upstream stable trees without any downstream patches.
> > 
> > The issue is seen few minutes after booting without any workload.
> > We launch hundred's of virtual instances and this shows up in 1-2
> > instances, so its hard to reproduce.
> > Attaching a few stack traces below.
> > 
> > The issue can be seen on virtual and baremetal instances.
> > Another interesting point is we only see this on x86 based instances.
> > We also did test this on linux-mainline but were not able to reproduce
> > the issue.
> > So maybe there's a fixup or related commit that has gone in?
> > 
> > We tried bisecting the stable trees and found that after reverting the
> > below commit we couldn't reproduce this in any of the kernels
> > consistently.
> > 
> > tick/common: Align tick period with the HZ tick. [ Upstream commit
> > e9523a0d81899361214d118ad60ef76f0e92f71d ]
> > 
> > 
> > Not exactly sure how this commit is affecting all stable kernels.
> > Can you take a look at this issue and share your insight?

Does this issue also show up in 6.3.y and in 6.4-rc5?

thanks,

greg k-h