Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII;
 format=flowed
Content-Transfer-Encoding: 7bit
Date:   Wed, 01 Aug 2018 01:07:03 -0700
From:   Sodagudi Prasad <psodagud@codeaurora.org>
To:     Peter Zijlstra <peterz@infradead.org>
Cc:     Thomas Gleixner <tglx@linutronix.de>,
        Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
        isaacm@codeaurora.org, matt@codeblueprint.co.uk, mingo@kernel.org,
        linux-kernel@vger.kernel.org, gregkh@linuxfoundation.org,
        pkondeti@codeaurora.org, stable@vger.kernel.org
Subject: Re: [PATCH] stop_machine: Disable preemption after queueing stopper
 threads
In-Reply-To: <20180730210728.GQ2494@hirez.programming.kicks-ass.net>
References: <1531856129-9871-1-git-send-email-isaacm@codeaurora.org>
 <a30d1e147d3181fc7b294262e380a10c@codeaurora.org>
 <20180724062350.nlem2suuy5wlxpts@linutronix.de>
 <alpine.DEB.2.21.1807301220280.2518@nanos.tec.linutronix.de>
 <20180730112140.GH2494@hirez.programming.kicks-ass.net>
 <alpine.DEB.2.21.1807301440340.2518@nanos.tec.linutronix.de>
 <109d0e70606ccd34861a80525d6d11aa@codeaurora.org>
 <20180730210728.GQ2494@hirez.programming.kicks-ass.net>
Message-ID: <52a34ed365cd560457e9abf5877c5b37@codeaurora.org>
User-Agent: Roundcube Webmail/1.2.5
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On 2018-07-30 14:07, Peter Zijlstra wrote:
> On Mon, Jul 30, 2018 at 10:12:43AM -0700, Sodagudi Prasad wrote:
>> How about including below change as well?  Currently, there is no way 
>> to
>> identify thread migrations completed or not.  When we observe this 
>> issue,
>> the symptom was work queue lock up. It is better to have some timeout 
>> here
>> and induce the bug_on.
> 
> You'd trigger the soft-lockup or hung-task detector I think. And if 
> not,
> we ought to look at making it trigger at least one of those.
> 
>> There is no way to identify the migration threads stuck or not.
> 
> Should be pretty obvious from the splat generated by the above, no?
Hi Peter and Thomas,

Thanks for your support.
I have another question on this flow and retry mechanism used in this 
cpu_stop_queue_two_works() function using the global variable 
stop_cpus_in_progress.

This variable is getting used in various paths, such as task migration, 
set task affinity, and CPU hotplug.

For example cpu hotplug path, stop_cpus_in_progress variable getting set 
with true with out checking.
takedown_cpu()
--stop_machine_cpuslocked()
---stop_cpus()
---__stop_cpus()
----queue_stop_cpus_work()
setting stop_cpus_in_progress to true directly.

But in the task migration path only, the stop_cpus_in_progress  variable 
is used for retry.

I am thinking that stop_cpus_in_progress variable lead race conditions, 
where CPU hotplug and task migration happening simultaneously. Please 
correct me If my understanding wrong.

-Thanks, Prasad

> 
>> --- a/kernel/stop_machine.c
>> +++ b/kernel/stop_machine.c
>> @@ -290,6 +290,7 @@ int stop_two_cpus(unsigned int cpu1, unsigned int 
>> cpu2,
>> cpu_stop_fn_t fn, void *
>>         struct cpu_stop_done done;
>>         struct cpu_stop_work work1, work2;
>>         struct multi_stop_data msdata;
>> +       int ret;
>> 
>>         msdata = (struct multi_stop_data){
>>                 .fn = fn,
>> @@ -312,7 +313,10 @@ int stop_two_cpus(unsigned int cpu1, unsigned int 
>> cpu2,
>> cpu_stop_fn_t fn, void *
>>         if (cpu_stop_queue_two_works(cpu1, &work1, cpu2, &work2))
>>                 return -ENOENT;
>> 
>> -       wait_for_completion(&done.completion);
>> +       ret = wait_for_completion_timeout(&done.completion,
>> msecs_to_jiffies(1000));
>> +       if (!ret)
>> +               BUG_ON(1);
>> +
> 
> That's a random timeout, which if you spuriously trigger it, will take
> down your machine. That seems like a cure worse than the disease.

-- 
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora 
Forum,
Linux Foundation Collaborative Project