Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
Date:   Tue, 8 Feb 2022 20:36:31 +0530
From:   "Gautham R. Shenoy" <gautham.shenoy@amd.com>
To:     Mel Gorman <mgorman@techsingularity.net>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@kernel.org>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Valentin Schneider <valentin.schneider@arm.com>,
        Aubrey Li <aubrey.li@linux.intel.com>,
        Barry Song <song.bao.hua@hisilicon.com>,
        Mike Galbraith <efault@gmx.de>,
        Srikar Dronamraju <srikar@linux.vnet.ibm.com>,
        K Prateek Nayak <kprateek.nayak@amd.com>,
        LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 1/2] sched/fair: Improve consistency of allowed NUMA
 balance calculations
Message-ID: <YgKG9/xN6YOTJg7M@BLR-5CG11610CF.amd.com>
References: <20220208094334.16379-1-mgorman@techsingularity.net>
 <20220208094334.16379-2-mgorman@techsingularity.net>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220208094334.16379-2-mgorman@techsingularity.net>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?+rg6shLTtkeWGH7yOc2b+/KoEqXDRoUecTLK8ZglgLTBk74kdfLsL0N4O00c?=
 =?us-ascii?Q?658eYpkOpZ3120at1KgYKmBfDeMiU4BwkeusvSG5OmBcQ5P98XY83TQnevzT?=
 =?us-ascii?Q?H/DIS+t75gmyu3wIljAvR9La3VvslQ0Wc2krTQhqfnKSw5qgaKPzYgFiJyCW?=
 =?us-ascii?Q?hZj8tSRcFN+/68q8sBh7MctJsiXJUZLIi97psbJ15iIz3pnrNuU3gHXMp5Kn?=
 =?us-ascii?Q?Ts3ezsNWBWV4EVJB6MtEZSEhfLqpZj3OjnWrc/Bz+j/FQ3vueUfp1tSG2G+z?=
 =?us-ascii?Q?AU1Fg16R7+efw4ewQdAkIrnglHuwuB6iydNh7+Hduim2ASkzhXYJG9Y/d/Zk?=
 =?us-ascii?Q?D1Fbxp26uQSEvwlJdYFGVhwd5kHkPF8OC58rPT+99V+pzyJ0PdfpcXl5zNfF?=
 =?us-ascii?Q?THMVDm7asicG++OGerpOupIsW3WVJ+Qy6Sm7Uxsq3WFI2cFCQqo9TQIdqmhb?=
 =?us-ascii?Q?cvVhmZf2gYJC71KEuJG0jaqphMVtMSALyZMopwawJIj8lga8r7iGZ0539Uf1?=
 =?us-ascii?Q?75vE0N51klDjCLY6Com6ONxFSa8SAunOPK4uze5dy0T5QGpVzZ8hMjSeD4pZ?=
 =?us-ascii?Q?l9zTraWsT8JuPbn2BFL0BqSKHAdUH7XXJCxYqVu5XVrXFdKsIBjeRM2qoHjA?=
 =?us-ascii?Q?C81krkpfgeRMgukE67GUvu+r+oKiCXutbOUgnhw+MsnoL/Z2nvZ8KaSuzfv/?=
 =?us-ascii?Q?/K1D2ykjTi1mNnn7cT92IR1xD9BFzJmgpc7Pk/u/Pgpvxrgj2h27/XFYkkZ2?=
 =?us-ascii?Q?rZTPHwIHjEh9eo49M8p1pOaQijCa/5jkKtkaW7qJHHEdEdPfaps/lozNd/zl?=
 =?us-ascii?Q?uzlfwhQYvUuhNgWcBjrpUNOVY7GjSUoPPA3oTViY+L/+W7WhRqoA1g0+Mg7d?=
 =?us-ascii?Q?6IlBKsqbRXH0RNhVVJV/uu7pXWdTnXK7gPJePaz7WEMKWiKk6Ih5Pj4nQqVX?=
 =?us-ascii?Q?YpuvpjO6o6OI+Fh8fXDATq/LX+PmBDyNiB/xHyxs5GIVSllGXlbF7WWl0drW?=
 =?us-ascii?Q?GvvKlcwSz+9XvLIMv+M5u/0ukVsslR7ydA4d9D8He4ZXOnFWfWroi8i7QS9h?=
 =?us-ascii?Q?W7Bsmpc7O052fzKv7bNToMmpFvkCpNifKGkrbEiIyi08On1vXW2Ighu2gzvW?=
 =?us-ascii?Q?RWHLk0yBo/84bIjSG2QPYyOuE1ePaEIE2JY9gOT5HY0F2BltaoKZttNlVANc?=
 =?us-ascii?Q?VWcHhkFmpFY6WAmBryAGFevwAiVuDFkVNLoGCBRF7jThjIGvaF0B1lN24V7E?=
 =?us-ascii?Q?Oqv9EZaCJ5Ow5oK3Aq51eqcXtUWF1CXAF7WEWIMWHY/iD+1V6d3/ZdQC29zV?=
 =?us-ascii?Q?Qhe4wCUPL4OKZKv0ABnFSyqz1SavTnpIsPdWCrwsuDDkVJqET4ec431oCr6i?=
 =?us-ascii?Q?i8aX4iBOqCBKPLmBIV2mIYroNmevNQQdEWfnZ8/lSxK8180YWDfc6o3b5zuR?=
 =?us-ascii?Q?AF2jMIXb6exf/LloBlZWa6Oqk1HWwT8ad0osa94m7liWT2QUwLRbZ+7SRnlt?=
 =?us-ascii?Q?5mxiccJWAND0jgA2J7r0/s1dcDyYbW1hGu2JF5U6IIUCUR1FRN1LL4Opp/TF?=
 =?us-ascii?Q?Uhn9hfu7I2b0WRIkgNl2ki7YnbYkP04j8vsRAapM+JPf5LsobokoHfrQbka8?=
 =?us-ascii?Q?Ucp+bmFrwoelWvH8AhznS94=3D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 26aef0e2-5341-4383-be19-08d9eb14a196
X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3286.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Feb 2022 15:06:49.4082
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: kB8WIU4Y2CCCQnCGuZ1VYeghp9X8IC8PWDgLvp7w8XHq1PcJET9TsmtMuy98ccYaP466KKAWs7ocslAkBEs8Ug==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR12MB5465
X-Spam-Status: No, score=-7.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,SPF_HELO_PASS,SPF_PASS,
	T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hello Mel,

On Tue, Feb 08, 2022 at 09:43:33AM +0000, Mel Gorman wrote:
> There are inconsistencies when determining if a NUMA imbalance is allowed
> that should be corrected.
> 
> o allow_numa_imbalance changes types and is not always examining
>   the destination group so both the type should be corrected as
>   well as the naming.
> o find_idlest_group uses the sched_domain's weight instead of the
>   group weight which is different to find_busiest_group
> o find_busiest_group uses the source group instead of the destination
>   which is different to task_numa_find_cpu
> o Both find_idlest_group and find_busiest_group should account
>   for the number of running tasks if a move was allowed to be
>   consistent with task_numa_find_cpu
>

This patch looks good to me. Using the local_sgs.group_weight instead
of sd->span_weight is definitely helping improve the spread of the
stream threads on AMD EPYC processors.

Reviewed-by: Gautham R. Shenoy <gautham.shenoy@amd.com>


> Fixes: 7d2b5dd0bcc4 ("sched/numa: Allow a floating imbalance between NUMA nodes")
> Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
> ---
>  kernel/sched/fair.c | 18 ++++++++++--------
>  1 file changed, 10 insertions(+), 8 deletions(-)
> 
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index 095b0aa378df..4592ccf82c34 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -9003,9 +9003,10 @@ static bool update_pick_idlest(struct sched_group *idlest,
>   * This is an approximation as the number of running tasks may not be
>   * related to the number of busy CPUs due to sched_setaffinity.
>   */
> -static inline bool allow_numa_imbalance(int dst_running, int dst_weight)
> +static inline bool
> +allow_numa_imbalance(unsigned int running, unsigned int weight)
>  {
> -	return (dst_running < (dst_weight >> 2));
> +	return (running < (weight >> 2));
>  }
>  
>  /*
> @@ -9139,12 +9140,13 @@ find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu)
>  				return idlest;
>  #endif
>  			/*
> -			 * Otherwise, keep the task on this node to stay close
> -			 * its wakeup source and improve locality. If there is
> -			 * a real need of migration, periodic load balance will
> -			 * take care of it.
> +			 * Otherwise, keep the task close to the wakeup source
> +			 * and improve locality if the number of running tasks
> +			 * would remain below threshold where an imbalance is
> +			 * allowed. If there is a real need of migration,
> +			 * periodic load balance will take care of it.
>  			 */
> -			if (allow_numa_imbalance(local_sgs.sum_nr_running, sd->span_weight))
> +			if (allow_numa_imbalance(local_sgs.sum_nr_running + 1, local_sgs.group_weight))
>  				return NULL;
>  		}
>  
> @@ -9350,7 +9352,7 @@ static inline void calculate_imbalance(struct lb_env *env, struct sd_lb_stats *s
>  		/* Consider allowing a small imbalance between NUMA groups */
>  		if (env->sd->flags & SD_NUMA) {
>  			env->imbalance = adjust_numa_imbalance(env->imbalance,
> -				busiest->sum_nr_running, busiest->group_weight);
> +				local->sum_nr_running + 1, local->group_weight);
>  		}
>  
>  		return;
> -- 
> 2.31.1
>