Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <168553468754.404.2298362895524875073.tip-bot2@tip-bot2>
 <3de5c24f-6437-f21b-ed61-76b86a199e8c@amd.com> <20230601111326.GV4253@hirez.programming.kicks-ass.net>
In-Reply-To: <20230601111326.GV4253@hirez.programming.kicks-ass.net>
From:   Chen Yu <yu.chen.surf@gmail.com>
Date:   Fri, 2 Jun 2023 00:44:25 +0800
Message-ID: <CADjb_WQcP2TUH98=W9YqnbkBSKOxEp0_8jB_af7E3-ghXf1awg@mail.gmail.com>
Subject: Re: [tip: sched/core] sched/fair: Multi-LLC select_idle_sibling()
To:     Peter Zijlstra <peterz@infradead.org>
Cc:     K Prateek Nayak <kprateek.nayak@amd.com>,
        linux-kernel@vger.kernel.org, linux-tip-commits@vger.kernel.org,
        Tejun Heo <tj@kernel.org>, x86@kernel.org,
        Gautham Shenoy <gautham.shenoy@amd.com>,
        Chen Yu <yu.c.chen@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Thu, Jun 1, 2023 at 8:11=E2=80=AFPM Peter Zijlstra <peterz@infradead.org=
> wrote:
>
> On Thu, Jun 01, 2023 at 03:03:39PM +0530, K Prateek Nayak wrote:
[...]
> > I wonder if extending SIS_UTIL for SIS_NODE would help some of these
> > cases but I've not tried tinkering with it yet. I'll continue
> > testing on other NPS modes which would decrease the search scope.
> > I'll also try running the same bunch of workloads on an even larger
> > 4th Generation EPYC server to see if the behavior there is similar.
>
> > >  /*
> > > + * For the multiple-LLC per node case, make sure to try the other LL=
C's if the
> > > + * local LLC comes up empty.
> > > + */
> > > +static int
> > > +select_idle_node(struct task_struct *p, struct sched_domain *sd, int=
 target)
> > > +{
> > > +   struct sched_domain *parent =3D sd->parent;
> > > +   struct sched_group *sg;
> > > +
> > > +   /* Make sure to not cross nodes. */
> > > +   if (!parent || parent->flags & SD_NUMA)
> > > +           return -1;
> > > +
> > > +   sg =3D parent->groups;
> > > +   do {
> > > +           int cpu =3D cpumask_first(sched_group_span(sg));
> > > +           struct sched_domain *sd_child;
> > > +
> > > +           sd_child =3D per_cpu(sd_llc, cpu);
> > > +           if (sd_child !=3D sd) {
> > > +                   int i =3D select_idle_cpu(p, sd_child, test_idle_=
cores(cpu), cpu);
>
> Given how SIS_UTIL is inside select_idle_cpu() it should already be
> effective here, no?
>
I'm thinking of this scenario, when the system is overloaded and with
SIS_NODE disabled,
the SIS_UTIL could scan for example 4 CPUs and terminates, then wakeup
on local LLC.
When SIS_NODE is enabled, it could scan for 4 * number_of_llc_domain
CPUs.  The more
CPU it scans, the more likely it can find an idle CPU.
This seems to be a question of: what type of wakee is prefered to be
put on a non-idle CPU from local LLC,
or  an idle CPU  from remote LLC. It seems to depend on the working
set and task duration.


thanks,
Chenyu