Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Message-ID: <1524116324.21378.1.camel@gmx.de>
Subject: Re: cpu stopper threads and load balancing leads to deadlock
From:   Mike Galbraith <efault@gmx.de>
To:     Matt Fleming <matt@codeblueprint.co.uk>,
        Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@kernel.org>
Cc:     linux-kernel@vger.kernel.org, Michal Hocko <mhocko@suse.com>
Date:   Thu, 19 Apr 2018 07:38:44 +0200
In-Reply-To: <1524030475.5645.2.camel@gmx.de>
References: <20180417142119.GA4511@codeblueprint.co.uk>
         <1524030475.5645.2.camel@gmx.de>
Content-Type: text/plain; charset="ISO-8859-15"
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Wed, 2018-04-18 at 07:47 +0200, Mike Galbraith wrote:
> On Tue, 2018-04-17 at 15:21 +0100, Matt Fleming wrote:
> > Hi guys,
> > 
> > We've seen a bug in one of our SLE kernels where the cpu stopper
> > thread ("migration/15") is entering idle balance. This then triggers
> > active load balance.
> > 
> > At the same time, a task on another CPU triggers a page fault and NUMA
> > balancing kicks in to try and migrate the task closer to the NUMA node
> > for that page (we're inside stop_two_cpus()). This faulting task is
> > spinning in try_to_wake_up() (inside smp_cond_load_acquire(&p->on_cpu,
> > !VAL)), waiting for "migration/15" to context switch.
> > 
> > Unfortunately, because "migration/15" is doing active load balance
> > it's spinning waiting for the NUMA-page-faulting CPU's stopper lock,
> > which is already held (since it's inside stop_two_cpus()).
> > 
> > Deadlock ensues.
> > 
> > This seems like a situation that should be prohibited, but I cannot
> > find any code to prevent it. Is it OK for stopper threads to load
> > balance? Is there something that should prevent this situation from
> > happening?
> 
> I don't see anything to stop the deadlock either, would exclude stop
> class from playing idle balancer entirely...

Bah, insufficient: __do_softirq() -> rebalance_domains() still bites.