Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <321a474bfa562164a56f504144d6b33eb2f7acbd.camel@linux.intel.com>
Subject: Re: [Patch v2 3/6] sched/fair: Implement prefer sibling imbalance
 calculation between asymmetric groups
From:   Tim Chen <tim.c.chen@linux.intel.com>
To:     Peter Zijlstra <peterz@infradead.org>
Cc:     Juri Lelli <juri.lelli@redhat.com>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Ricardo Neri <ricardo.neri@intel.com>,
        "Ravi V . Shankar" <ravi.v.shankar@intel.com>,
        Ben Segall <bsegall@google.com>,
        Daniel Bristot de Oliveira <bristot@redhat.com>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Len Brown <len.brown@intel.com>, Mel Gorman <mgorman@suse.de>,
        "Rafael J . Wysocki" <rafael.j.wysocki@intel.com>,
        Srinivas Pandruvada <srinivas.pandruvada@linux.intel.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Valentin Schneider <vschneid@redhat.com>,
        Ionela Voinescu <ionela.voinescu@arm.com>, x86@kernel.org,
        linux-kernel@vger.kernel.org,
        Shrikanth Hegde <sshegde@linux.vnet.ibm.com>,
        Srikar Dronamraju <srikar@linux.vnet.ibm.com>,
        naveen.n.rao@linux.vnet.ibm.com,
        Yicong Yang <yangyicong@hisilicon.com>,
        Barry Song <v-songbaohua@oppo.com>,
        Chen Yu <yu.c.chen@intel.com>, Hillf Danton <hdanton@sina.com>
Date:   Tue, 13 Jun 2023 10:46:36 -0700
In-Reply-To: <20230612120528.GL4253@hirez.programming.kicks-ass.net>
References: <cover.1686263351.git.tim.c.chen@linux.intel.com>
         <ef2400ea29f0e00aed1e2d75c1a352e63349059e.1686263351.git.tim.c.chen@linux.intel.com>
         <20230612120528.GL4253@hirez.programming.kicks-ass.net>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.44.4 (3.44.4-2.fc36) 
MIME-Version: 1.0
Precedence: bulk

On Mon, 2023-06-12 at 14:05 +0200, Peter Zijlstra wrote:
> On Thu, Jun 08, 2023 at 03:32:29PM -0700, Tim Chen wrote:
>=20
> >=20
> > +	if (env->sd->flags & SD_ASYM_PACKING) {
> > +		int limit;
> > +
> > +		if (!busiest->sum_nr_running)
> > +			goto out;
>=20
> This seems out-of-place, shouldn't we have terminate sooner if busiest
> is empty?

Yes.  Should move this check to the beginning.

>=20
> > +
> > +		if (sched_asym_prefer(env->dst_cpu, sds->busiest->asym_prefer_cpu)) =
{
> > +			/* Don't leave preferred core idle */
> > +			if (imbalance =3D=3D 0 && local->sum_nr_running < ncores_local)
> > +				imbalance =3D 1;
> > +			goto out;
> > +		}
> > +
> > +		/* Limit tasks moved from preferred group, don't leave cores idle */
> > +		limit =3D busiest->sum_nr_running;
> > +		lsub_positive(&limit, ncores_busiest);
> > +		if (imbalance > limit)
> > +			imbalance =3D limit;
>=20
> How does this affect the server parts that have larger than single core
> turbo domains?

Are you thinking about the case where the local group is completely empty
so there's turbo headroom and we should move at least one task, even though
CPU in busiest group has higher priority?

In other words, are you suggesting we should add

		if (imbalance =3D=3D 0 && busiest->sum_nr_running > 0 &&
			local->sum_nr_running =3D=3D 0)
			imbalance =3D 1;
	=09

>=20
> > +
> > +		goto out;
> > +	}
> > +
> > +	/* Take advantage of resource in an empty sched group */
> > +	if (imbalance =3D=3D 0 && local->sum_nr_running =3D=3D 0 &&
> > +	    busiest->sum_nr_running > 1)
> > +		imbalance =3D 1;
> > +out:
> > +	return imbalance << 1;
> > +}
>=20
>=20
> But basically you have:
>=20
>         LcBn - BcLn
>   imb =3D -----------
>            LcBc
>=20
> Which makes sense, except you then return:
>=20
>   imb * 2
>=20
> which then made me wonder about rounding.
>=20
> Do we want to to add (LcBc -1) or (LcBc/2) to resp. ceil() or round()
> the thing before division? Because currently it uses floor().
>=20
> If you evaludate it like:
>=20
>=20
>         2 * (LcBn - BcLn)
>   imb =3D -----------------
>               LcBc
>=20
> The result is different from what you have now.

If I do the rounding after multiplying imb by two (call it imb_new),
the difference with imb I am returning now (call it imb_old)
will be at most 1.  Note that imb_old returned is always a multiple of 2.

I will be using imb in calculate_imbalance() and divide it
by 2 there to get the number tasks to move from busiest group.
So when there is a difference of 1 between imb_old and imb_new,
the difference will be trimmed off after the division of 2.

We will get the same number of tasks to move with either
imb_old or imb_new in calculate_imbalance() so the two
computations will arrive at the same result eventually.

>=20
> What actual behaviour is desired in these low imbalance cases? and can
> you write a comment as to why we do as we do etc..?

I do not keep imb as=20

           2 * (LcBn - BcLn)
   imb =3D -----------------
               LcBc

as it is easier to leave out the factor of 2
in the middle of sibling_imblalance() computation
so I can directly interpret imb as the number
of tasks to move, and add the factor of two
when I actually need to return the imbalance.

Would you like me to add this reasoning in the comments?

Thanks.

Tim =20