Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Wed, 16 Jun 2021 09:15:12 +0200
From:   Peter Zijlstra <peterz@infradead.org>
To:     Yafang Shao <laoar.shao@gmail.com>
Cc:     Vincent Guittot <vincent.guittot@linaro.org>,
        Ingo Molnar <mingo@redhat.com>,
        Juri Lelli <juri.lelli@redhat.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Benjamin Segall <bsegall@google.com>,
        Mel Gorman <mgorman@suse.de>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        LKML <linux-kernel@vger.kernel.org>,
        Valentin Schneider <valentin.schneider@arm.com>,
        Thomas Gleixner <tglx@linutronix.de>
Subject: Re: [PATCH] sched, fair: try to prevent migration thread from
 preempting non-cfs task
Message-ID: <YMmlAP/QhE6SWhCF@hirez.programming.kicks-ass.net>
References: <20210615121551.31138-1-laoar.shao@gmail.com>
 <20210615203534.GA4272@worktop.programming.kicks-ass.net>
 <CALOAHbBuZJaK+fEg7toRUHJNP8rJKDoADeAUxorUuNU17kdTOA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CALOAHbBuZJaK+fEg7toRUHJNP8rJKDoADeAUxorUuNU17kdTOA@mail.gmail.com>
Precedence: bulk

On Wed, Jun 16, 2021 at 09:44:46AM +0800, Yafang Shao wrote:
> On Wed, Jun 16, 2021 at 4:35 AM Peter Zijlstra <peterz@infradead.org> wrote:
> >
> > On Tue, Jun 15, 2021 at 08:15:51PM +0800, Yafang Shao wrote:
> > > ---
> > >  kernel/sched/fair.c | 14 ++++++++++++++
> > >  1 file changed, 14 insertions(+)
> > >
> > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > > index 3248e24a90b0..597c7a940746 100644
> > > --- a/kernel/sched/fair.c
> > > +++ b/kernel/sched/fair.c
> > > @@ -9797,6 +9797,20 @@ static int load_balance(int this_cpu, struct rq *this_rq,
> > >                       /* Record that we found at least one task that could run on this_cpu */
> > >                       env.flags &= ~LBF_ALL_PINNED;
> > >
> > > +                     /*
> > > +                      * There may be a race between load balance starting migration
> > > +                      * thread to pull the cfs running thread and the RT thread
> > > +                      * waking up and preempting cfs task before migration threads
> > > +                      * which then preempt the RT thread.
> > > +                      * We'd better do the last minute check before starting
> > > +                      * migration thread to avoid preempting latency-sensitive thread.
> > > +                      */
> > > +                     if (busiest->curr->sched_class != &fair_sched_class) {
> > > +                             raw_spin_unlock_irqrestore(&busiest->lock,
> > > +                                                        flags);
> >
> > This won't apply.
> >
> > Also, there's still a race window, you've just shrunk it, not fixed it.
> > Busiest can reschedule between the mandatory rq unlock and doing the
> > stopper wakeup.
> >
> > An actual fix might be to have the active migration done by a FIFO-1
> > task, instead of stopper. The obvious down-side is that that would mean
> > spawning yet another per-cpu kthread.
> >
> 
> The stopper and the migration thread are different threads in the earlier days.
> commit 969c79215a35 ("sched: replace migration_thread with cpu_stop")
> merged them into one thread.

Yes, I know, I was there. But that's not what I'm saying, we need the
migration thread to be super high perio for other cases. That change
still makes sense.

> Regarding the priority of stopper (with highest priority) and
> migration (higher than CFS, but lower than RT) , keeping them in one
> single thread seems not a good way.

I never suggested as such.

Only the active migration of CFS can be done by a FIFO-1 task (the
lowest prio that is higher than CFS) and possible the numa balancing
thing.

Other migrations will still need to use stopper, and as such you'll keep
having interference from stopper.

The suggestion was adding a cfs_migration thread, specifically for
active balance (and maybe numa). Just not sure the cost of carrying yet
another per-cpu kernel thread is worth the benefit.