Received-SPF: pass (google.com: domain of linux-kernel+bounces-17455-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Precedence: bulk
MIME-Version: 1.0
References: <20231220001856.3710363-1-jstultz@google.com> <20231220001856.3710363-24-jstultz@google.com>
 <8001a9bc-3f87-403c-a091-215c56c9ebf0@arm.com>
In-Reply-To: <8001a9bc-3f87-403c-a091-215c56c9ebf0@arm.com>
From: John Stultz <jstultz@google.com>
Date: Thu, 4 Jan 2024 19:42:34 -0800
Message-ID: <CANDhNCpqjiUi4iKKKXH8PLtUKe2JTJdt8W=7CQ6VJi409+RaEw@mail.gmail.com>
Subject: Re: [PATCH v7 23/23] sched: Fix rt/dl load balancing via chain level balance
To: Metin Kaya <metin.kaya@arm.com>
Cc: LKML <linux-kernel@vger.kernel.org>, Joel Fernandes <joelaf@google.com>, 
	Qais Yousef <qyousef@google.com>, Ingo Molnar <mingo@redhat.com>, 
	Peter Zijlstra <peterz@infradead.org>, Juri Lelli <juri.lelli@redhat.com>, 
	Vincent Guittot <vincent.guittot@linaro.org>, Dietmar Eggemann <dietmar.eggemann@arm.com>, 
	Valentin Schneider <vschneid@redhat.com>, Steven Rostedt <rostedt@goodmis.org>, 
	Ben Segall <bsegall@google.com>, Zimuzo Ezeozue <zezeozue@google.com>, 
	Youssef Esmat <youssefesmat@google.com>, Mel Gorman <mgorman@suse.de>, 
	Daniel Bristot de Oliveira <bristot@redhat.com>, Will Deacon <will@kernel.org>, Waiman Long <longman@redhat.com>, 
	Boqun Feng <boqun.feng@gmail.com>, "Paul E. McKenney" <paulmck@kernel.org>, 
	Xuewen Yan <xuewen.yan94@gmail.com>, K Prateek Nayak <kprateek.nayak@amd.com>, 
	Thomas Gleixner <tglx@linutronix.de>, kernel-team@android.com
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Fri, Dec 22, 2023 at 6:51=E2=80=AFAM Metin Kaya <metin.kaya@arm.com> wro=
te:
> On 20/12/2023 12:18 am, John Stultz wrote:
> > +     /*
> > +      * Chain leads off the rq, we're free to push it anywhere.
> > +      *
> > +      * One wrinkle with relying on find_exec_ctx is that when the cha=
in
> > +      * leads to a task currently migrating to rq, we see the chain as
> > +      * pushable & push everything prior to the migrating task. Even i=
f
> > +      * we checked explicitly for this case, we could still race with =
a
> > +      * migration after the check.
> > +      * This shouldn't permanently produce a bad state though, as prox=
y()
>
>                                                         find_proxy_task()

Fixed.

> > +#ifdef CONFIG_SCHED_PROXY_EXEC
> >   static inline bool dl_revalidate_rq_state(struct task_struct *task, s=
truct rq *rq,
> > -                                       struct rq *later)
> > +                                       struct rq *later, bool *retry)
> > +{
> > +     if (!dl_task(task) || is_migration_disabled(task))
> > +             return false;
> > +
> > +     if (rq !=3D this_rq()) {
> > +             struct task_struct *next_task =3D pick_next_pushable_dl_t=
ask(rq);
> > +
> > +             if (next_task =3D=3D task) {
>
> Nit: We can `return false;` if next_task !=3D task and save one level of
> indentation.

Ah, good point. Fixed.

> > +                     struct task_struct *exec_ctx;
> > +
> > +                     exec_ctx =3D find_exec_ctx(rq, next_task);
> > +                     *retry =3D (exec_ctx && !cpumask_test_cpu(later->=
cpu,
> > +                                                            &exec_ctx-=
>cpus_mask));
> > +             } else {
> > +                     return false;
> > +             }
> > +     } else {
> > +             int pushable =3D task_is_pushable(rq, task, later->cpu);
> > +
> > +             *retry =3D pushable =3D=3D -1;
> > +             if (!pushable)
> > +                     return false;
>
> `return pushable;` can replace above 2 lines.
> The same for rt_revalidate_rq_state().

Hrm. It does save lines, but I fret (in my estimation) it makes the
code a touch more complex to read. I might hold off on this for the
moment unless someone else pushes for it.

> > diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
> > index fabb19891e95..d5ce95dc5c09 100644
> > --- a/kernel/sched/rt.c
> > +++ b/kernel/sched/rt.c
...
> > +#ifdef CONFIG_SCHED_PROXY_EXEC
> > +static inline bool rt_revalidate_rq_state(struct task_struct *task, st=
ruct rq *rq,
> > +                                       struct rq *lowest, bool *retry)
>
> This function can be consolidated with dl_revalidate_rq_state() as you
> noted in the previous patch, although rt_revalidate_rq_state() has few
> comments.

Yeah. I need to stare at it a bit to try to figure out what might be
the best name to use for the common chunk.
I'd also like to figure a better way to do the retry stuff, as it feels mes=
sy.

> > +{
> > +     /*
> > +      * Releasing the rq lock means we need to re-check pushability.
> > +      * Some scenarios:
> > +      * 1) If a migration from another CPU sent a task/chain to rq
> > +      *    that made task newly unpushable by completing a chain
> > +      *    from task to rq->curr, then we need to bail out and push so=
mething
> > +      *    else.
> > +      * 2) If our chain led off this CPU or to a dequeued task, the la=
st waiter
> > +      *    on this CPU might have acquired the lock and woken (or even=
 migrated
> > +      *    & run, handed off the lock it held, etc...). This can inval=
idate the
> > +      *    result of find_lowest_rq() if our chain previously ended in=
 a blocked
> > +      *    task whose affinity we could ignore, but now ends in an unb=
locked
> > +      *    task that can't run on lowest_rq.
> > +      * 3) Race described at https://lore.kernel.org/all/1523536384-26=
781-2-git-send-email-huawei.libin@huawei.com/
> > +      *
> > +      * Notes on these:
> > +      * - Scenario #2 is properly handled by rerunning find_lowest_rq
> > +      * - Scenario #1 requires that we fail
> > +      * - Scenario #3 can AFAICT only occur when rq is not this_rq(). =
And the
> > +      *   suggested fix is not universally correct now that push_cpu_s=
top() can
> > +      *   call this function.
> > +      */
> > +     if (!rt_task(task) || is_migration_disabled(task)) {
> > +             return false;
> > +     } else if (rq !=3D this_rq()) {
>
> Nit: `else` can be dropped as in dl_revalidate_rq_state().

Ack. Done.

Thanks again for all the feedback!
-john