Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Mon, 29 Jul 2019 12:47:45 +0200
From:   Peter Zijlstra <peterz@infradead.org>
To:     Thomas Gleixner <tglx@linutronix.de>
Cc:     Guenter Roeck <linux@roeck-us.net>, x86@kernel.org,
        Ingo Molnar <mingo@redhat.com>, linux-kernel@vger.kernel.org,
        Borislav Petkov <bp@alien8.de>
Subject: Re: sched: Unexpected reschedule of offline CPU#2!
Message-ID: <20190729104745.GA31398@hirez.programming.kicks-ass.net>
References: <20190727164450.GA11726@roeck-us.net>
 <20190729093545.GV31381@hirez.programming.kicks-ass.net>
 <alpine.DEB.2.21.1907291156170.1791@nanos.tec.linutronix.de>
 <20190729101349.GX31381@hirez.programming.kicks-ass.net>
 <alpine.DEB.2.21.1907291235580.1791@nanos.tec.linutronix.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <alpine.DEB.2.21.1907291235580.1791@nanos.tec.linutronix.de>
User-Agent: Mutt/1.10.1 (2018-07-13)
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Mon, Jul 29, 2019 at 12:38:30PM +0200, Thomas Gleixner wrote:
> On Mon, 29 Jul 2019, Peter Zijlstra wrote:
> > On Mon, Jul 29, 2019 at 11:58:24AM +0200, Thomas Gleixner wrote:
> > > On Mon, 29 Jul 2019, Peter Zijlstra wrote:
> > > > On Sat, Jul 27, 2019 at 09:44:50AM -0700, Guenter Roeck wrote:
> > > > > [   61.348866] Call Trace:
> > > > > [   61.349392]  kick_ilb+0x90/0xa0
> > > > > [   61.349629]  trigger_load_balance+0xf0/0x5c0
> > > > > [   61.349859]  ? check_preempt_wakeup+0x1b0/0x1b0
> > > > > [   61.350057]  scheduler_tick+0xa7/0xd0
> > > > 
> > > > kick_ilb() iterates nohz.idle_cpus_mask to find itself an idle_cpu().
> > > > 
> > > > idle_cpus_mask() is set from nohz_balance_enter_idle() and cleared from
> > > > nohz_balance_exit_idle(). nohz_balance_enter_idle() is called from
> > > > __tick_nohz_idle_stop_tick() when entering nohz idle, this includes the
> > > > cpu_is_offline() clause of the idle loop.
> > > > 
> > > > However, when offline, cpu_active() should also be false, and this
> > > > function should no-op.
> > > 
> > > Ha. That reboot mess is not clearing cpu active as it's not going through
> > > the regular cpu hotplug path. It's using reboot IPI which 'stops' the cpus
> > > dead in their tracks after clearing cpu online....
> > 
> > $string-of-cock-compliant-curses
> > 
> > What a trainwreck...
> > 
> > So if it doesn't play by the normal rules; how does it expect to work?
> > 
> > So what do we do? 'Fix' reboot or extend the rules?
> 
> Reboot has two modes:
> 
>  - Regular reboot initiated from user space
> 
>  - Panic reboot
> 
> For the regular reboot we can make it go through proper hotplug, 

That seems sensible.

> for the panic case not so much.

It's panic, shit has already hit fan, one or two more pieces shouldn't
something anybody cares about.