Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 9 Mar 2022 10:55:54 +0530
From:   Srikar Dronamraju <srikar@linux.vnet.ibm.com>
To:     K Prateek Nayak <kprateek.nayak@amd.com>
Cc:     peterz@infradead.org, aubrey.li@linux.intel.com, efault@gmx.de,
        gautham.shenoy@amd.com, linux-kernel@vger.kernel.org,
        mgorman@techsingularity.net, mingo@kernel.org,
        song.bao.hua@hisilicon.com, valentin.schneider@arm.com,
        vincent.guittot@linaro.org
Subject: Re: [PATCH v6] sched/fair: Consider cpu affinity when allowing NUMA
 imbalance in find_idlest_group
Message-ID: <20220309052554.GK618915@linux.vnet.ibm.com>
Reply-To: Srikar Dronamraju <srikar@linux.vnet.ibm.com>
References: <20220308063749.6929-1-kprateek.nayak@amd.com>
 <20220308092944.GJ618915@linux.vnet.ibm.com>
 <7e5bdc95-a30a-58bc-fc67-98b03fe1fa22@amd.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
In-Reply-To: <7e5bdc95-a30a-58bc-fc67-98b03fe1fa22@amd.com>
Precedence: bulk

* K Prateek Nayak <kprateek.nayak@amd.com> [2022-03-08 17:18:16]:

> Hello Srikar,
> 
> On 3/8/2022 2:59 PM, Srikar Dronamraju wrote:
> > [..snip..]


> >> @@ -9200,10 +9201,19 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu)
> >>  			 * Otherwise, keep the task close to the wakeup source
> >>  			 * and improve locality if the number of running tasks
> >>  			 * would remain below threshold where an imbalance is
> >> -			 * allowed. If there is a real need of migration,
> >> -			 * periodic load balance will take care of it.
> >> +			 * allowed while accounting for the possibility the
> >> +			 * task is pinned to a subset of CPUs. If there is a
> >> +			 * real need of migration, periodic load balance will
> >> +			 * take care of it.
> >>  			 */
> >> -			if (allow_numa_imbalance(local_sgs.sum_nr_running + 1, sd->imb_numa_nr))
> >> +			imb = sd->imb_numa_nr;
> >> +			if (p->nr_cpus_allowed != num_online_cpus()) {

> > Again, repeating, is the problem only happening in the pinned case?
> Yes. We've tested stream with 8 and 16 stream threads on a Zen3 system
> with 16 LLCs and in both cases, with unbound runs, we've seen each
> Stream thread get a separate LLC and we didn't observe any stacking.

If the problem is only happening with pinned case, then it means that in the
in unpinned case, the load balancer is able to do the load balancing
correctly and quickly but for some reason may not be able to do the same in
pinned case. Without the patch, even in the unpinned case, the initial CPU
range is more less the same number of LLCs as the pinned. However its able
to spread better.

I believe the problem could be in can_migrate_task() checking for
!cpumask_test_cpu(env->dst_cpu, p->cpus_ptr)

i.e dst_cpu is doing a load balance on behalf of the entire LLC, however it
only will pull tasks that can be pulled into it.

> --
> Thanks and Regards,
> Prateek

-- 
Thanks and Regards
Srikar Dronamraju