Date: Thu, 7 Sep 2017 11:26:16 +0200
From: Peter Zijlstra <peterz@infradead.org>
To: Mike Galbraith <efault@gmx.de>
Cc: Andy Lutomirski <luto@amacapital.net>,
        Andy Lutomirski <luto@kernel.org>, Ingo Molnar <mingo@kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Tejun Heo <tj@kernel.org>, "Rafael J. Wysocki" <rjw@rjwysocki.net>,
        Thomas Gleixner <tglx@linutronix.de>
Subject: Re: [PATCH] sched/cpuset/pm: Fix cpuset vs suspend-resume
Message-ID: <20170907092616.thsuyqklit4463wj@hirez.programming.kicks-ass.net>
References: <CALCETrXZ9LZ+T5T-FS2hy0TU_c0MWPYokN9-d0qx3-1jFOOHcw@mail.gmail.com>
 <20170906082520.xgvo3hewje7jvdyo@hirez.programming.kicks-ass.net>
 <877A43A3-AC8F-4D7C-88E4-8E3D36B1DAFA@amacapital.net>
 <20170906090333.7v627vdyvhe3x2cs@hirez.programming.kicks-ass.net>
 <20170906161422.kvqe5y6b3ng6ouyu@hirez.programming.kicks-ass.net>
 <1504764929.6355.5.camel@gmx.de>
 <20170907091338.orwxrqkbfkki3c24@hirez.programming.kicks-ass.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20170907091338.orwxrqkbfkki3c24@hirez.programming.kicks-ass.net>
User-Agent: NeoMutt/20170609 (1.8.3)
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 2071
Lines: 48

On Thu, Sep 07, 2017 at 11:13:38AM +0200, Peter Zijlstra wrote:
> Subject: sched/cpuset/pm: Fix cpuset vs suspend-resume
> 
> Cpusets vs suspend-resume is _completely_ broken. And it got noticed
> because it now resulted in non-cpuset usage breaking too.
> 
> On suspend cpuset_cpu_inactive() doesn't call into
> cpuset_update_active_cpus() because it doesn't want to move tasks about,
> there is no need, all tasks are frozen and won't run again until after
> we've resumed everything.
> 
> But this means that when we finally do call into
> cpuset_update_active_cpus() after resuming the last frozen cpu in
> cpuset_cpu_active(), the top_cpuset will not have any difference with
> the cpu_active_mask and this it will not in fact do _anything_.
> 
> So the cpuset configuration will not be restored. This was largely
> hidden because we would unconditionally create identity domains and
> mobile users would not in fact use cpusets much. And servers what do use
> cpusets tend to not suspend-resume much.
> 
> An addition problem is that we'd not in fact wait for the cpuset work to
> finish before resuming the tasks, allowing spurious migrations outside
> of the specified domains.
> 
> Fix the rebuild by introducing cpuset_force_rebuild() and fix the
> ordering with cpuset_wait_for_hotplug().
> 
> Cc: tj@kernel.org
> Cc: rjw@rjwysocki.net
> Cc: efault@gmx.de
> Reported-by: Andy Lutomirski <luto@kernel.org>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>

TJ, I _think_ it was commit:

  deb7aa308ea2 ("cpuset: reorganize CPU / memory hotplug handling")

That wrecked things, but there's been so much changes in this area it is
really hard to tell. Note how before that commit it would
unconditionally rebuild the domains, and you 'optimized' that ;-)

That commit also introduced the work to do the async rebuild and failed
to do that flush on resume.

In any case, I think we should put a fixes tag on this commit such that
it gets picked up into stable kernels. Not sure anybody will try and
backport it into 4 year old kernels, but who knows.