Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Wed, 10 Feb 2021 12:21:44 +0100
From:   Peter Zijlstra <peterz@infradead.org>
To:     "Song Bao Hua (Barry Song)" <song.bao.hua@hisilicon.com>
Cc:     "valentin.schneider@arm.com" <valentin.schneider@arm.com>,
        "vincent.guittot@linaro.org" <vincent.guittot@linaro.org>,
        "mgorman@suse.de" <mgorman@suse.de>,
        "mingo@kernel.org" <mingo@kernel.org>,
        "dietmar.eggemann@arm.com" <dietmar.eggemann@arm.com>,
        "morten.rasmussen@arm.com" <morten.rasmussen@arm.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linuxarm@openeuler.org" <linuxarm@openeuler.org>,
        "xuwei (O)" <xuwei5@huawei.com>,
        "Liguozhu (Kenneth)" <liguozhu@hisilicon.com>,
        "tiantao (H)" <tiantao6@hisilicon.com>,
        wanghuiqiang <wanghuiqiang@huawei.com>,
        "Zengtao (B)" <prime.zeng@hisilicon.com>,
        Jonathan Cameron <jonathan.cameron@huawei.com>,
        "guodong.xu@linaro.org" <guodong.xu@linaro.org>,
        Meelis Roos <mroos@linux.ee>
Subject: Re: [PATCH v2] sched/topology: fix the issue groups don't span
 domain->span for NUMA diameter > 2
Message-ID: <YCPByAdQ+rZFzYWp@hirez.programming.kicks-ass.net>
References: <20210203111201.20720-1-song.bao.hua@hisilicon.com>
 <YCKGVBnXzRsE6/Er@hirez.programming.kicks-ass.net>
 <4bdaa3e1a54f445fa8e629ea392e7bce@hisilicon.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <4bdaa3e1a54f445fa8e629ea392e7bce@hisilicon.com>
Precedence: bulk

On Tue, Feb 09, 2021 at 08:58:15PM +0000, Song Bao Hua (Barry Song) wrote:

> > I've finally had a moment to think about this, would it make sense to
> > also break up group: node0+1, such that we then end up with 3 groups of
> > equal size?
> 

> Since the sched_domain[n-1] of a part of node[m]'s siblings are able
> to cover the whole span of sched_domain[n] of node[m], there is no
> necessity to scan over all siblings of node[m], once sched_domain[n]
> of node[m] has been covered, we can stop making more sched_groups. So
> the number of sched_groups is small.
> 
> So historically, the code has never tried to make sched_groups result
> in equal size. And it permits the overlapping of local group and remote
> groups.

Histrorically groups have (typically) always been the same size though.

The reason I did ask is because when you get one large and a bunch of
smaller groups, the load-balancing 'pull' is relatively smaller to the
large groups.

That is, IIRC should_we_balance() ensures only 1 CPU out of the group
continues the load-balancing pass. So if, for example, we have one group
of 4 CPUs and one group of 2 CPUs, then the group of 2 CPUs will pull
1/2 times, while the group of 4 CPUs will pull 1/4 times.

By making sure all groups are of the same level, and thus of equal size,
this doesn't happen.