Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
From:   Andy Lutomirski <luto@amacapital.net>
Mime-Version: 1.0 (1.0)
Subject: Re: [PATCH tick-sched] Clarify "NOHZ: local_softirq_pending" warning
Date:   Sat, 27 Jun 2020 15:14:14 -0700
Message-Id: <83B12EF8-3792-4943-A548-5DB0C6FC71D1@amacapital.net>
References: <20200627214629.GH9247@paulmck-ThinkPad-P72>
Cc:     Andy Lutomirski <luto@kernel.org>,
        Frederic Weisbecker <fweisbec@gmail.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@kernel.org>,
        LKML <linux-kernel@vger.kernel.org>,
        kernel-team <Kernel-team@fb.com>
In-Reply-To: <20200627214629.GH9247@paulmck-ThinkPad-P72>
To:     paulmck@kernel.org
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


> On Jun 27, 2020, at 2:46 PM, Paul E. McKenney <paulmck@kernel.org> wrote:
>=20
> =EF=BB=BFOn Sat, Jun 27, 2020 at 02:02:15PM -0700, Andy Lutomirski wrote:
>>> On Fri, Jun 26, 2020 at 2:05 PM Paul E. McKenney <paulmck@kernel.org> wr=
ote:
>>>=20
>>> Currently, can_stop_idle_tick() prints "NOHZ: local_softirq_pending HH"
>>> (where "HH" is the hexadecimal softirq vector number) when one or more
>>> non-RCU softirq handlers are still enablded when checking to stop the
>>> scheduler-tick interrupt.  This message is not as enlightening as one
>>> might hope, so this commit changes it to "NOHZ tick-stop error: Non-RCU
>>> local softirq work is pending, handler #HH.
>>=20
>> Thank you!  It would be even better if it would explain *why* the
>> problem happened, but I suppose this code doesn't actually know.
>=20
> Glad to help!
>=20
> To your point, is it possible to bisect the appearance of this message,
> or is it as usual non-reproducible?  (Hey, had to ask!)
>=20
>                           =20

In this particular case, I tracked it down by good old fashioned sleuthing f=
or bugs, but it=E2=80=99s still unclear to me precisely how NOHZ gets involv=
ed. The bug is that we were entering the kernel from usermode, doing nmi_ent=
er(), turning on interrupts, maybe getting a page fault, raising a signal, t=
urning off interrupts, nmi_exit(), and back to usermode, with the signal sti=
ll queued and undelivered.  This is all kinds of bad, but I still don=E2=80=99=
t understand what softirqs or idle have to do with it.

But I have the bug fixed now!=