Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Date:   Tue, 8 Feb 2022 21:49:05 +0530
From:   "Gautham R. Shenoy" <gautham.shenoy@amd.com>
To:     Mel Gorman <mgorman@techsingularity.net>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@kernel.org>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Valentin Schneider <valentin.schneider@arm.com>,
        Aubrey Li <aubrey.li@linux.intel.com>,
        Barry Song <song.bao.hua@hisilicon.com>,
        Mike Galbraith <efault@gmx.de>,
        Srikar Dronamraju <srikar@linux.vnet.ibm.com>,
        K Prateek Nayak <kprateek.nayak@amd.com>,
        LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 2/2] sched/fair: Adjust the allowed NUMA imbalance when
 SD_NUMA spans multiple LLCs
Message-ID: <YgKX+cmDDA1VO9kX@BLR-5CG11610CF.amd.com>
References: <20220208094334.16379-1-mgorman@techsingularity.net>
 <20220208094334.16379-3-mgorman@techsingularity.net>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20220208094334.16379-3-mgorman@techsingularity.net>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?L+41rlrrd8NfpCSCmG7F1TP2hxBb0eqocdSn02e8eF1pvS6SfayW2CFOIbmG?=
 =?us-ascii?Q?4MGHiT1tY2DlbQOqEBi1mBh4H3WVj0EqedfVRmr0ujF6M8wPinf3QtpZFC38?=
 =?us-ascii?Q?c7IdLKffgGtaoeP4d1lPf5Slzc746I8Jw1otUy6WZkqcdhkQ9oUsM10MLkvF?=
 =?us-ascii?Q?OSO5hJGe/DgXNOCcGtkOchdyeuvAqN9h403ZE03WXYNvp0VSOKc2PINTkENt?=
 =?us-ascii?Q?jferFYfEKbLOmNSXo9F+qSUCb+pcPCF3yrxU76ntdhxRw7JNeXjnKttsTBEM?=
 =?us-ascii?Q?8IAPg6dlyr6v/AmCeZelyf5+jbmp8t4vuF7oDYgaUnT9IjVhwMRUmg+3RgLu?=
 =?us-ascii?Q?PuD7WVQAqLLb2KkAbtfGsh0NtTq55RsR/u+hi50rTKW/TotUxHadmv21BCPE?=
 =?us-ascii?Q?MVnXhpWSGJMY0rFOZeFIQ2s8V6XHbbTOyzRKcIDlt95ifO/ocfNWIdmolYD4?=
 =?us-ascii?Q?9sUQx14IdPWxM2zYmv1oY6RAgY3X3w5ZNjTttxNWRHap9KprXsCDF5RAn2j+?=
 =?us-ascii?Q?CdirN77O9zA6GiSLJRInTpFi48+WjWjVF0N2srQn1YsJL1051672tzmKqkwP?=
 =?us-ascii?Q?t2K7Ai/sKH4iaJ680TR9v++bqyGCmfXpK41PKMEqsye8Ot3vHZ2blJKV1cR9?=
 =?us-ascii?Q?jOQvdtgRjOtz/y9gpiYS49EzpSS00+tCR/R8MP+bBKZmhP9pfE043hVj8ttT?=
 =?us-ascii?Q?4HBR8X4MVjAKwhcIIJudKzrEjkL98MF5mBiXmSWlYaj8FKoK8RNI9Xal0Cuq?=
 =?us-ascii?Q?Pyw+11UMdSd909VPVeGzcahyR58TmSBvJ+JzsLuTbCowsOcSKhQUA7DId4DH?=
 =?us-ascii?Q?MQdqNQstWmUlyUCDwkDkGW4O4PpDbKOVbHMfrwugwQvmERPGBdu6Dn8TPY0t?=
 =?us-ascii?Q?JoDphTQHAyl0t27CSXW3TWYZ/tK/YS1IobLCUvDwdMavNy9lBs6TEgjAS6gt?=
 =?us-ascii?Q?90WrTwu/K0YhtF4oiJeYjHCiO6RtXpfaGJhXlUcAiyU2Gz/l/evdhCqXdfYw?=
 =?us-ascii?Q?0dYys67DpZl8V24hqZvoCh5s0HVvT2bRbXIqgU8zEu1HYBXf+YJCkXuOMXH2?=
 =?us-ascii?Q?p3waOBwlog81dmhLhclLEUEwhxgCkKB/XXV8aYNqd6xqg75Job3wi9W5JMAh?=
 =?us-ascii?Q?j8q0Nbf5o9fr/oj2AS9i6kNIgs4WYGRVszWmlZOzeVsCi5GutECmbve3tPZ8?=
 =?us-ascii?Q?YTzmergEKHDsl9QfhZI527vf/xmrFxM/hCaWDyR8EKrEItWk3d+Z24Y6HR2I?=
 =?us-ascii?Q?WItJm1Ti7QIviJ9g2xTVqz/7dCp8yW/pTStuknMpT4NOD3z9NFJVqR5RAbVP?=
 =?us-ascii?Q?ZC3pezTRkOCLHNUZkQF2bKTGOYFvqQMOLo5AB9IQtbl6uAZFilwEeqn9ggW2?=
 =?us-ascii?Q?vo8+GSrWm3iJPEmWSR7YLjHfpXtQ275wOAoNnF2Zp8h7Ky6Z7wlPlJMxf0p+?=
 =?us-ascii?Q?5gq7R5blxW3bPG/sSu9lroxw1laIDbZEbe6dNkuwO+0RsBnQSrPoQETK590k?=
 =?us-ascii?Q?Dg0Zhl5+sM2D79yfEraeJvG3F2lGNM5luSY9wNqK53SsJJhfk1sWJSwoodB2?=
 =?us-ascii?Q?fcfpnH9uppQ7QBmKYAbTCu4MOH75Z50dOvj70xHQrOKI7K7ObLtWeyBYI4Py?=
 =?us-ascii?Q?FclolTK86B4rjLLbb4dhTJw=3D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4ab1d501-a7ca-4054-0d6b-08d9eb1ec5ff
X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3286.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Feb 2022 16:19:25.7070
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: D/xj7GAPYg9wMxVdmMkhUglUiqZcblV2G6iGwiOgQ/+iJUVMfbt5a9/2+eDtLynH1o+9Ko1qJIgcy3e5xdeFjA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR1201MB0221
X-Spam-Status: No, score=-7.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,SPF_HELO_PASS,SPF_PASS,
	T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Feb 08, 2022 at 09:43:34AM +0000, Mel Gorman wrote:

[..snip..]

> diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c
> index d201a7052a29..e6cd55951304 100644
> --- a/kernel/sched/topology.c
> +++ b/kernel/sched/topology.c
> @@ -2242,6 +2242,59 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att
>  		}
>  	}
>  
> +	/*
> +	 * Calculate an allowed NUMA imbalance such that LLCs do not get
> +	 * imbalanced.
> +	 */
> +	for_each_cpu(i, cpu_map) {
> +		unsigned int imb = 0;
> +		unsigned int imb_span = 1;
> +
> +		for (sd = *per_cpu_ptr(d.sd, i); sd; sd = sd->parent) {
> +			struct sched_domain *child = sd->child;
> +
> +			if (!(sd->flags & SD_SHARE_PKG_RESOURCES) && child &&
> +			    (child->flags & SD_SHARE_PKG_RESOURCES)) {
> +				struct sched_domain *top, *top_p;
> +				unsigned int nr_llcs;
> +
> +				/*
> +				 * For a single LLC per node, allow an
> +				 * imbalance up to 25% of the node. This is an
> +				 * arbitrary cutoff based on SMT-2 to balance
> +				 * between memory bandwidth and avoiding
> +				 * premature sharing of HT resources and SMT-4
> +				 * or SMT-8 *may* benefit from a different
> +				 * cutoff.
> +				 *
> +				 * For multiple LLCs, allow an imbalance
> +				 * until multiple tasks would share an LLC
> +				 * on one node while LLCs on another node
> +				 * remain idle.
> +				 */
> +				nr_llcs = sd->span_weight / child->span_weight;
> +				if (nr_llcs == 1)
> +					imb = sd->span_weight >> 2;
> +				else
> +					imb = nr_llcs;
> +				sd->imb_numa_nr = imb;
> +
> +				/* Set span based on the first NUMA domain. */
> +				top = sd;
> +				top_p = top->parent;
> +				while (top_p && !(top_p->flags & SD_NUMA)) {
> +					top = top->parent;
> +					top_p = top->parent;
> +				}
> +				imb_span = top_p ? top_p->span_weight : sd->span_weight;
> +			} else {
> +				int factor = max(1U, (sd->span_weight / imb_span));
> +
> +				sd->imb_numa_nr = imb * factor;
> +			}
> +		}
> +	}


On a 2 Socket Zen3 servers with 64 cores per socket, the imb_numa_nr
works out to be as follows for different Node Per Socket (NPS) modes

NPS = 1:
======
SMT(span = 2) -- > MC (span = 16) --> DIE (span = 128) --> NUMA (span = 256)
	 Parent of LLC is DIE. nr_llcs = 128/16 = 8. imb = 8.
	 top_p = NUMA. imb_span = 256.

for NUMA doman, factor = max(1U, 256/256) = 1. Thus sd->imb_numa_nr = 8.


NPS = 2
========
SMT(span=2)--> MC(span=16)--> NODE(span=64)--> NUMA1(span=128)--> NUMA2(span=256)

Parent of LLC = NODE. nr_llcs = 64/16 = 4. imb = 4.
top_p = NUMA1. imb_span = 128.

For NUMA1 domain, factor = 1. sd->imb_numa_nr = 4.
For NUMA2 domain, factor = 2. sd->imb_numa_nr = 8


NPS = 4
========
SMT(span=2)--> MC(span=16)--> NODE(span=32)--> NUMA1(span=128)--> NUMA2(span=256)

Parent of LLC = NODE. nr_llcs = 32/16 = 2. imb = 2.
top_p = NUMA1. imb_span = 128.

For NUMA1 domain, factor = 1. sd->imb_numa_nr = 2.
For NUMA2 domain, factor = 2. sd->imb_numa_nr = 4


The imb_numa_nr looks good for all the NPS modes. Furthermore, running
stream with 16 threads (equal to the number of LLCs in the system)
yields good results on all the NPS modes with this imb_numa_nr.

Reviewed-by: Gautham R. Shenoy <gautham.shenoy@amd.com>

--
Thanks and Regards
gautham.