Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <0b20535f4bd6908942c91be86bd17bc3c07514f2.camel@linux.intel.com>
Subject: Re: [Patch v3 2/6] sched/topology: Record number of cores in sched
 group
From:   Tim Chen <tim.c.chen@linux.intel.com>
To:     Valentin Schneider <vschneid@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>
Cc:     Juri Lelli <juri.lelli@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Ricardo Neri <ricardo.neri@intel.com>,
        "Ravi V . Shankar" <ravi.v.shankar@intel.com>,
        Ben Segall <bsegall@google.com>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Len Brown <len.brown@intel.com>, Mel Gorman <mgorman@suse.de>,
        "Rafael J . Wysocki" <rafael.j.wysocki@intel.com>,
        Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Ionela Voinescu <ionela.voinescu@arm.com>, x86@kernel.org,
        linux-kernel@vger.kernel.org,
        Shrikanth Hegde <sshegde@linux.vnet.ibm.com>,
        Srikar Dronamraju <srikar@linux.vnet.ibm.com>,
        naveen.n.rao@linux.vnet.ibm.com,
        Yicong Yang <yangyicong@hisilicon.com>,
        Barry Song <v-songbaohua@oppo.com>,
        Chen Yu <yu.c.chen@intel.com>, Hillf Danton <hdanton@sina.com>
Date:   Mon, 10 Jul 2023 15:13:43 -0700
In-Reply-To: <xhsmhedlfv74k.mognet@vschneid.remote.csb>
References: <cover.1688770494.git.tim.c.chen@linux.intel.com>
         <04641eeb0e95c21224352f5743ecb93dfac44654.1688770494.git.tim.c.chen@linux.intel.com>
         <xhsmhedlfv74k.mognet@vschneid.remote.csb>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.44.4 (3.44.4-2.fc36) 
MIME-Version: 1.0
Precedence: bulk

On Mon, 2023-07-10 at 21:33 +0100, Valentin Schneider wrote:
> On 07/07/23 15:57, Tim Chen wrote:
> > From: Tim C Chen <tim.c.chen@linux.intel.com>
> >=20
> > When balancing sibling domains that have different number of cores,
> > tasks in respective sibling domain should be proportional to the number
> > of cores in each domain. In preparation of implementing such a policy,
> > record the number of tasks in a scheduling group.
> >=20
> > Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com>
> > ---
> >  kernel/sched/sched.h    |  1 +
> >  kernel/sched/topology.c | 10 +++++++++-
> >  2 files changed, 10 insertions(+), 1 deletion(-)
> >=20
> > diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> > index 3d0eb36350d2..5f7f36e45b87 100644
> > --- a/kernel/sched/sched.h
> > +++ b/kernel/sched/sched.h
> > @@ -1860,6 +1860,7 @@ struct sched_group {
> >       atomic_t		ref;
> >=20
> >       unsigned int		group_weight;
> > +	unsigned int		cores;
> >       struct sched_group_capacity *sgc;
> >       int			asym_prefer_cpu;	/* CPU of highest priority in group */
> >       int			flags;
> > diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
> > index 6d5628fcebcf..6b099dbdfb39 100644
> > --- a/kernel/sched/topology.c
> > +++ b/kernel/sched/topology.c
> > @@ -1275,14 +1275,22 @@ build_sched_groups(struct sched_domain *sd, int=
 cpu)
> >  static void init_sched_groups_capacity(int cpu, struct sched_domain *s=
d)
> >  {
> >       struct sched_group *sg =3D sd->groups;
> > +	struct cpumask *mask =3D sched_domains_tmpmask2;
> >=20
> >       WARN_ON(!sg);
> >=20
> >       do {
> > -		int cpu, max_cpu =3D -1;
> > +		int cpu, cores =3D 0, max_cpu =3D -1;
> >=20
> >               sg->group_weight =3D cpumask_weight(sched_group_span(sg))=
;
> >=20
> > +		cpumask_copy(mask, sched_group_span(sg));
> > +		for_each_cpu(cpu, mask) {
> > +			cores++;
> > +			cpumask_andnot(mask, mask, cpu_smt_mask(cpu));
> > +		}
>=20
>=20
> This rekindled my desire for an SMT core cpumask/iterator. I played aroun=
d
> with a global mask but that's a headache: what if we end up with a core
> whose SMT threads are split across two exclusive cpusets?

Peter and I pondered that for a while.  But it seems like partitioning
threads in a core between two different sched domains is not a very
reasonable thing to do.=20

https://lore.kernel.org/all/20230612112945.GK4253@hirez.programming.kicks-a=
ss.net/

Tim

>=20
> I ended up necro'ing a patch from Peter [1], but didn't get anywhere nice
> (the LLC shared storage caused me issues).
>=20
> All that to say, I couldn't think of a nicer way :(
>=20
> [1]: https://lore.kernel.org/all/20180530143106.082002139@infradead.org/#=
t
>=20