Received-SPF: pass (google.com: domain of linux-kernel+bounces-13134-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Date: Fri, 29 Dec 2023 16:18:22 +0100 (CET)
From: Julia Lawall <julia.lawall@inria.fr>
To: Vincent Guittot <vincent.guittot@linaro.org>
cc: Peter Zijlstra <peterz@infradead.org>, Ingo Molnar <mingo@redhat.com>, 
    Dietmar Eggemann <dietmar.eggemann@arm.com>, Mel Gorman <mgorman@suse.de>, 
    linux-kernel@vger.kernel.org
Subject: Re: EEVDF and NUMA balancing
In-Reply-To: <edbd8ecd-148c-b366-fd46-3531dec39d49@inria.fr>
Message-ID: <cecfd395-f067-99e1-bdd2-fec2ebc3db3@inria.fr>
References: <alpine.DEB.2.22.394.2310032059060.3220@hadrien> <alpine.DEB.2.22.394.2310041958380.3108@hadrien> <20231009102949.GC14330@noisy.programming.kicks-ass.net> <b8ab29de-1775-46e-dd75-cdf98be8b0@inria.fr> <CAKfTPtBhWwk9sf9F1=KwubiAWFDC2A9ZT-SSJ+tgFxme1cFmYA@mail.gmail.com>
 <alpine.DEB.2.22.394.2312182302310.3361@hadrien> <CAKfTPtALEFtrapi3Kk97KLGQN4259eEQEwwftVUK4RG42Vgoyw@mail.gmail.com> <98b3df1-79b7-836f-e334-afbdd594b55@inria.fr> <CAKfTPtCRN_eWgVdK2-h6E_ifJKwwJEtMjeNjB=5DXZFWyBS+tQ@mail.gmail.com>
 <93112fbe-30be-eab8-427c-5d4670a0f94e@inria.fr> <CAKfTPtAeFvrZxApK3RruWwCjMxbQvOkU+_YgZSo4QPT_AD6FxA@mail.gmail.com> <9dc451b5-9dd8-89f2-1c9c-7c358faeaad@inria.fr> <CAKfTPtDCsLnDnVje9maP5s-L7TbtSu4CvF19xHOxbkvSNd7vZg@mail.gmail.com>
 <2359ab5-4556-1a73-9255-3fcf2fc57ec@inria.fr> <6618dcfa-a42f-567c-2a9d-a76786683b29@inria.fr> <CAKfTPtDrULyOB9+RhjoPfCpHKVhx5kRf6dq79DSE6jZgsEairw@mail.gmail.com> <edbd8ecd-148c-b366-fd46-3531dec39d49@inria.fr>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII


On Thu, 28 Dec 2023, Julia Lawall wrote:

> > > > > > > I'm surprised that you have mainly CPU_NEWLY_IDLE. Do you know the reason ?
> > > > > >
> > > > > > No.  They come from do_idle calling the scheduler.  I will look into why
> > > > > > this happens so often.
> > > > >
> > > > > Hmm, the CPU was idle and received a need resched which triggered the
> > > > > scheduler but there was nothing to schedule so it goes back to idle
> > > > > after running a newly_idle _load_balance.
> > > >
> > > > I spent quite some time thinking the same until I saw the following code
> > > > in do_idle:
> > > >
> > > > preempt_set_need_resched();
> > > >
> > > > So I have the impression that do_idle sets need resched itself.
> > >
> > > But of course that code is only executed if need_resched is true.  But I
> >
> > Yes, that is your root cause. something, most probably in interrupt
> > context, wakes up your CPU and expect to wake up a thread
> >
> > > don't know who would be setting need resched on each clock tick.
> >
> > that can be a timer, interrupt, ipi, rcu ...
> > a trace should give you some hints
>
> I have the impression that it is the goal of calling nohz_csd_func on each
> clock tick that causes the calls to need_resched.  If the idle process is
> polling, call_function_single_prep_ipi just sets need_resched to get the
> idle process to stop polling.  But there is no actual task that the idle
> process should schedule.  The need_resched then prevents the idle process
> from stealing, due to the CPU_NEWLY_IDLE flag, contradicting the whole
> purpose of calling nohz_csd_func in the first place.

Looking in more detail, do_idle contains the following after existing the
polling loop:

        flush_smp_call_function_queue();
 	schedule_idle();

flush_smp_call_function_queue() does end up calling nohz_csd_func, but
this has no impact, because it first checks that need_resched() is false,
whereas it is currently true to cause existing the polling loop.  Removing
that test causes:

raise_softirq_irqoff(SCHED_SOFTIRQ);

but that causes the load balancing code to be executed from a ksoftirqd
task, which means that there is now no load imbalance.

So the only chance to detect an imbalance does seem to be to have the load
balance call be executed by the idle task, via schedule_idle(), as is
done currently.  But that leads to the core being considered to be newly
idle.

julia