Date:   Fri, 3 Dec 2021 13:27:19 +0000
From:   Mel Gorman <mgorman@techsingularity.net>
To:     Barry Song <21cnbao@gmail.com>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@kernel.org>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Valentin Schneider <valentin.schneider@arm.com>,
        Aubrey Li <aubrey.li@linux.intel.com>,
        Barry Song <song.bao.hua@hisilicon.com>,
        Mike Galbraith <efault@gmx.de>,
        Srikar Dronamraju <srikar@linux.vnet.ibm.com>,
        LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 2/2] sched/fair: Adjust the allowed NUMA imbalance when
 SD_NUMA spans multiple LLCs
Message-ID: <20211203132719.GD3366@techsingularity.net>
References: <20211201151844.20488-1-mgorman@techsingularity.net>
 <20211201151844.20488-3-mgorman@techsingularity.net>
 <CAGsJ_4xuYH8d18r9f5vUGYjNxtZ31FZrNZepEh8QCZbLW+3a6Q@mail.gmail.com>
 <20211203105055.GB3366@techsingularity.net>
 <CAGsJ_4zigz0rs9ubPjJKn0s9FsHp6A1Gpoh+cyc=cQnnMEMgYg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-15
Content-Disposition: inline
In-Reply-To: <CAGsJ_4zigz0rs9ubPjJKn0s9FsHp6A1Gpoh+cyc=cQnnMEMgYg@mail.gmail.com>
User-Agent: Mutt/1.10.1 (2018-07-13)
Precedence: bulk

On Sat, Dec 04, 2021 at 12:14:33AM +1300, Barry Song wrote:
> > > Hi Mel, you used to have 25% * numa_weight if node has only one LLC.
> > > for a system with 4 numa,  In case sd has 2 nodes, child is 1 numa node,
> > > then  nr_groups=2, num_online_nodes()=4,  imb_numa_nr will be
> > > child->span_weight/2/2/4?
> > >
> > > Does this patch change the behaviour for machines whose numa equals LLC?
> > >
> >
> > Yes, it changes behaviour. Instead of a flat 25%, it takes into account
> > the number of LLCs per node and the number of nodes overall.
> 
> Considering  the number of nodes overall seems to be quite weird to me.
> for example, for the below machines
> 
> 1P * 2DIE = 2NUMA:    node1 - node0
> 2P * 2DIE = 4NUMA:    node1 - node0  ------ node2 - node3
> 4P * 2DIE = 8NUMA:    node1 - node0  ------ node2 - node3
>                                       node5 - node4  ------ node6 - node7
> 
> if one service pins node1 and node0 in all above configurations, it seems in all
> different machines, the app will result in different behavior.
> 

The intent is to balance between LLCs across the whole machine, hence
accounting for the number of online nodes.

> the other example is:
> in a 2P machine, if one app pins the first two NUMAs, the other app pins
> the last two NUMAs, why would the  num_online_nodes() matter to them?
> there is no balance requirement between the two P.
> 

The previous 25% imbalance also did not take pinning into account and
the choice was somewhat arbitrary.

-- 
Mel Gorman
SUSE Labs