Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Thu, 2 Aug 2018 11:49:08 +0200
From:   Peter Zijlstra <peterz@infradead.org>
To:     Sodagudi Prasad <psodagud@codeaurora.org>
Cc:     mingo@kernel.org, gregkh@linuxfoundation.org,
        bigeasy@linutronix.de, tglx@linutronix.de, isaacm@codeaurora.org,
        linux-kernel@vger.kernel.org
Subject: Re: cpu stopper threads and setaffinity leads to deadlock
Message-ID: <20180802094908.GK2494@hirez.programming.kicks-ass.net>
References: <24eebe1d874cb8e3b9a18087554544fa@codeaurora.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <24eebe1d874cb8e3b9a18087554544fa@codeaurora.org>
User-Agent: Mutt/1.10.0 (2018-05-17)
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Wed, Aug 01, 2018 at 06:34:40PM -0700, Sodagudi Prasad wrote:
> Due to cross migration of tasks between cpu7 and cpu3, migration/7 has
> started executing and waits for the migration/3 task, so that they can
> proceed within the multi cpu stop state machine together.
> Unfortunately stress-ng-affin is affine to cpu7, and since migration 7 has
> started running, and has monopolized cpu7’s execution, stress-ng will never
> run on cpu7, and cpu3’s migration task is never woken up.

> diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
> index e190d1e..f932e1e 100644
> --- a/kernel/stop_machine.c
> +++ b/kernel/stop_machine.c
> @@ -87,9 +87,9 @@ static bool cpu_stop_queue_work(unsigned int cpu, struct
> cpu_stop_work *work)
>                 __cpu_stop_queue_work(stopper, work, &wakeq);
>         else if (work->done)
>                 cpu_stop_signal_done(work->done);
> -       raw_spin_unlock_irqrestore(&stopper->lock, flags);
> 
>         wake_up_q(&wakeq);
> +       raw_spin_unlock_irqrestore(&stopper->lock, flags);
> 

So why didn't you do the 'obvious' parallel to what you did for
cpu_stop_queue_two_works(), namely:

--- a/kernel/stop_machine.c
+++ b/kernel/stop_machine.c
@@ -81,6 +81,7 @@ static bool cpu_stop_queue_work(unsigned
 	unsigned long flags;
 	bool enabled;
 
+	preempt_disable();
 	raw_spin_lock_irqsave(&stopper->lock, flags);
 	enabled = stopper->enabled;
 	if (enabled)
@@ -90,6 +91,7 @@ static bool cpu_stop_queue_work(unsigned
 	raw_spin_unlock_irqrestore(&stopper->lock, flags);
 
 	wake_up_q(&wakeq);
+	preempt_enable();
 
 	return enabled;
 }