Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Atul Kulkarni <Atul.Kulkarni@katerra.com>
To:     "paulmck@kernel.org" <paulmck@kernel.org>
CC:     "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Paul Reeves <Paul.Reeves@katerra.com>,
        Mikhail Shoykher <Mikhail.Shoykher@katerra.com>
Subject: RE: Need help on "Self Detected Stall on CPU"
Thread-Topic: Need help on "Self Detected Stall on CPU"
Thread-Index: AdYfH2sMtOQXo4fbTb+klSbNR9S/QgABI7KAAA4w5dA=
Date:   Fri, 1 May 2020 02:11:01 +0000
Message-ID: <BY5PR11MB411817B78A6FB4AF81FF2821E0AB0@BY5PR11MB4118.namprd11.prod.outlook.com>
References: <BY5PR11MB4118B89CB3FAEB897AFC3819E0AA0@BY5PR11MB4118.namprd11.prod.outlook.com>
 <20200430191723.GX7560@paulmck-ThinkPad-P72>
In-Reply-To: <20200430191723.GX7560@paulmck-ThinkPad-P72>
Accept-Language: en-IN, en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: cde4adca-e33a-4329-060e-08d7ed74e58d
X-MS-Exchange-CrossTenant-originalarrivaltime: 01 May 2020 02:11:01.7799
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 82a61cce-7c23-4d88-931c-13b9a487cb25
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: +16lDxl//4eqzCreK61tNmW9QTbSnsASyXg2tJSgeC7hDqC24vecPo8mA+0UApYLbxmTNAlD16S7HX1sIKGcSO7Js3hkgNpSE4Z9AXPAi1k=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR11MB4070
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Thank you sir for your guidance and quick response.

Let me introduce my colleagues Paul and Mikhail here (copied in CC). They w=
ould be taking actions based on your guidance in this email and may reach y=
ou with further queries.

Appreciate your support and help.

Thanks,
Atul

-----Original Message-----
From: Paul E. McKenney <paulmck@kernel.org>=20
Sent: 01 May 2020 00:47
To: Atul Kulkarni <Atul.Kulkarni@katerra.com>
Cc: linux-kernel@vger.kernel.org
Subject: Re: Need help on "Self Detected Stall on CPU"

On Thu, Apr 30, 2020 at 06:47:20PM +0000, Atul Kulkarni wrote:
> Dear Sir,
>=20
> Hope you are doing well.  I have watched your various conference videos a=
nd have read technical papers.
> We are facing an issue with CPU stall on our systems and I felt like ther=
e is no one better who can guide us on how we can deal with it.
>=20
> I have attached logs for your reference. Towards end I have run couple of=
 sysreq commands and have taken crash dump using sysreq which may help prov=
ide additional information.
> Could you please guide us on how we could fix  this issue or identify wha=
t is going wrong here?

Let's focus on the first few lines of your console message:

[20526.345089] INFO: rcu_preempt self-detected stall on CPU [20526.351110] =
 0-...: (1051 ticks this GP) idle=3D1fe/140000000000002/0 softirq=3D146268/=
146268 fqs=3D0
[20526.360163]   (t=3D2101 jiffies g=3D96468 c=3D96467 q=3D2)
[20526.365535] rcu_preempt kthread starved for 2101 jiffies! g96468 c96467 =
f0x0 RCU_GP_WAIT_FQS(3) ->state=3D0x402 ->cpu=3D0

The last line contains the hint, namely "rcu_preempt kthread starved for
2101 jiffies!"  If you don't let RCU's kernel threads run, then RCU CPU sta=
ll warnings are expected behavior.

The "RCU_GP_WAIT_FQS(3)" means that this kthread's last act was to sleep fo=
r three jiffies.  As you can see from earlier in that same line, that was 2=
101 jiffies ago.  The "->state=3D0x402" means that the scheduler believes t=
hat this kthread is blocked, that is not yet runnable.

The usual way this sort of thing happens is a timer problem, be it a hardwa=
re configuration problem, a timer-driver bug, an interrupt-handling problem=
, and so on.  This sort of problem is especially common when bringing up ne=
w hardware or when modifying timer code or when modifying code on the inter=
rupt/exception paths.

So the question to ask yourself is "Why is the timer wakeup not reaching th=
is kthread?", with special attention to changed code and new hardware.

							Thanx, Paul