Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31;
References: <20230912082101.342002-1-ying.huang@intel.com>
 <20230912082101.342002-2-ying.huang@intel.com>
User-agent: mu4e 1.8.13; emacs 28.2
From:   Alistair Popple <apopple@nvidia.com>
To:     Huang Ying <ying.huang@intel.com>
Cc:     Andrew Morton <akpm@linux-foundation.org>, linux-mm@kvack.org,
        linux-kernel@vger.kernel.org, Bharata B Rao <bharata@amd.com>,
        "Aneesh Kumar K . V" <aneesh.kumar@linux.ibm.com>,
        Wei Xu <weixugc@google.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Michal Hocko <mhocko@kernel.org>,
        Yang Shi <shy828301@gmail.com>,
        Dave Jiang <dave.jiang@intel.com>,
        Rafael J Wysocki <rafael.j.wysocki@intel.com>
Subject: Re: [PATCH -V3 1/4] memory tiering: add abstract distance
 calculation algorithms management
Date:   Tue, 19 Sep 2023 15:13:18 +1000
In-reply-to: <20230912082101.342002-2-ying.huang@intel.com>
Message-ID: <877com68zm.fsf@nvdebian.thelocal>
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?+qk368C/b4Xt76Y5EClmCmE8da0M9IjqXUtfsPKvY3NqwH0eRYsGKAIzMD++?=
 =?us-ascii?Q?0h++fONk0fcEJpoe39CrRnOTKLYDA7Anq8Wc0+vmAALRtr34RCTX8i3OqMea?=
 =?us-ascii?Q?qDdNrnR4lpM/23F4CqJTb1ZlDg+v2ik8eRRpkLbbFQs/M6ffDKRMpuhanvlv?=
 =?us-ascii?Q?wjXPMKFl+gpgSE4fFsm1F0fTvQiTPenYATpNbD9D4MJA4S/B+DAuZpaw963n?=
 =?us-ascii?Q?6AaAKOvAHnTVc5QKHgBk8i9455/jwItZO3W7IZaHYrXkOa/UxBmIl5AAi9XH?=
 =?us-ascii?Q?AniXNSOLJhi9lkUQUcVVDxeXQM0EEc3y2gvxOM0E4UZF3SzJy5FZDkk/+21v?=
 =?us-ascii?Q?aOtlr691dMxC+hk5RWyQlxcb4kN2OYwvwViAa1lfRqDJq8AdwI1EzHFN/VW9?=
 =?us-ascii?Q?8bIRD95wvJ81ebv8twUJf67xRPnLvvFQy8h1Vh8V3tdr0nLgQ4eKwOOmJpX6?=
 =?us-ascii?Q?Gl14SF4WLXcw6x/qciLanRjZhdahD0zBzbORwUAOjwyanZbsEvixPgF50+UI?=
 =?us-ascii?Q?juhmsc2Czxy7284IJJMr3cSxX8wtyRxxxF+EcoTWSGd8oeTzqpVx7U7A6a+5?=
 =?us-ascii?Q?FqYUlokTgRseKyEJXaaX+Sw27kgcRSTB9Ve3vYQRVabNC3oPRnK2kICpSPU4?=
 =?us-ascii?Q?ku6MklPm9qnPPYlUFWQr9Av2pT4fv/T0QUKAh6PwLGNQA1y2DBsh8Je8DpMK?=
 =?us-ascii?Q?yK9HSN0gOLPaPSUNJnAroTKXZ1NGaCKLUwwvqZXfOFX3eH8CqHTOCpdYWJe9?=
 =?us-ascii?Q?2hVS9oupL1U7x8j7F8jrJYdai49UYuhrej65/T4kdGBitRRqOCVRzgcr4YRS?=
 =?us-ascii?Q?xLKSj1Sr2N1Y9dRVSs2nLHIC4XpHPbHXzwuVqwR0F0m9GD4UhWYyI7KpB2cf?=
 =?us-ascii?Q?uLwJIsR10Lq+jJUXWgN+bIHCHAsSdtzk18ScKVQo55CW0crnpX2Z3QpOVMVg?=
 =?us-ascii?Q?6C9a5tM5YhwMwKS/1mMpNUxdJ6x90KUDwkBw1DcQvFhnYgD855bYZh+bzlPm?=
 =?us-ascii?Q?zZ0MfjvK4jxzxnGoLaX7ahp6w6AV/79Njw1FfjfKUSX3SNkwrauLiHb48LgQ?=
 =?us-ascii?Q?tb8ILEGR/d8y5NzuSpys3yhrylLgh4KxJLcnEujSrYZxRKTDge/yDOWrWHWf?=
 =?us-ascii?Q?Xy+GQUV2GDZZmoKbK1BxKtmiNtBwCqjvu44+YIH59Jb4nS/YfPYBilhS9uZD?=
 =?us-ascii?Q?LI6HMami+noA3yrCyNDeMd20k/aBZiBc7DTOmeJrK6ZbKAOug5QD9xSOrJCZ?=
 =?us-ascii?Q?+e1dvFhInxaGP+JMZCIDlU9fcnIxdIFa1paEQYXO9O/xBSmVJDVe0KdYNs5W?=
 =?us-ascii?Q?GdODc9w1DtR/CqsmI7Ug1e3rpjEYr5JagmGH/lMoSBWoAnvgMdXcX/NIfcav?=
 =?us-ascii?Q?8T2o7eVrSjoACIHaooSYutenp/ZIF2OsOjvkmNbfzNErUN3cbghpkrNEFfi8?=
 =?us-ascii?Q?noSIdoexou6KWGJ0evFdCCfZvBrubFJSWbDt/t9npIr/5e3DcJ6Lnq0BN6IW?=
 =?us-ascii?Q?gyy8VYGefRl9pIJMzh+5gg5KvLHNt56No6lBx4PfJpjrV1iatndxc6XnzUjf?=
 =?us-ascii?Q?T8KKAKC5XyzXetpJWg2DegOCMyty1p6EYpGH+/z5?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5524c969-2405-4760-4475-08dbb8cf2f98
X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Sep 2023 05:13:40.1117
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: vf/Bq51+9TMrD2zO5zPVRQXQr2EIJx38bcLGpQNKO5zkIpr1ykwKzlEiBXEGH8B04YZ8EMlBkfpl2vNWexs71g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6209
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 18 Sep 2023 22:13:54 -0700 (PDT)


Reviewed-by: Alistair Popple <apopple@nvidia.com>

Huang Ying <ying.huang@intel.com> writes:

> The abstract distance may be calculated by various drivers, such as
> ACPI HMAT, CXL CDAT, etc.  While it may be used by various code which
> hot-add memory node, such as dax/kmem etc.  To decouple the algorithm
> users and the providers, the abstract distance calculation algorithms
> management mechanism is implemented in this patch.  It provides
> interface for the providers to register the implementation, and
> interface for the users.
>
> Multiple algorithm implementations can cooperate via calculating
> abstract distance for different memory nodes.  The preference of
> algorithm implementations can be specified via
> priority (notifier_block.priority).
>
> Signed-off-by: "Huang, Ying" <ying.huang@intel.com>
> Tested-by: Bharata B Rao <bharata@amd.com>
> Cc: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
> Cc: Wei Xu <weixugc@google.com>
> Cc: Alistair Popple <apopple@nvidia.com>
> Cc: Dan Williams <dan.j.williams@intel.com>
> Cc: Dave Hansen <dave.hansen@intel.com>
> Cc: Davidlohr Bueso <dave@stgolabs.net>
> Cc: Johannes Weiner <hannes@cmpxchg.org>
> Cc: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> Cc: Michal Hocko <mhocko@kernel.org>
> Cc: Yang Shi <shy828301@gmail.com>
> Cc: Dave Jiang <dave.jiang@intel.com>
> Cc: Rafael J Wysocki <rafael.j.wysocki@intel.com>
> ---
>  include/linux/memory-tiers.h | 19 ++++++++++++
>  mm/memory-tiers.c            | 59 ++++++++++++++++++++++++++++++++++++
>  2 files changed, 78 insertions(+)
>
> diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h
> index 437441cdf78f..c8382220cced 100644
> --- a/include/linux/memory-tiers.h
> +++ b/include/linux/memory-tiers.h
> @@ -6,6 +6,7 @@
>  #include <linux/nodemask.h>
>  #include <linux/kref.h>
>  #include <linux/mmzone.h>
> +#include <linux/notifier.h>
>  /*
>   * Each tier cover a abstrace distance chunk size of 128
>   */
> @@ -36,6 +37,9 @@ struct memory_dev_type *alloc_memory_type(int adistance);
>  void put_memory_type(struct memory_dev_type *memtype);
>  void init_node_memory_type(int node, struct memory_dev_type *default_type);
>  void clear_node_memory_type(int node, struct memory_dev_type *memtype);
> +int register_mt_adistance_algorithm(struct notifier_block *nb);
> +int unregister_mt_adistance_algorithm(struct notifier_block *nb);
> +int mt_calc_adistance(int node, int *adist);
>  #ifdef CONFIG_MIGRATION
>  int next_demotion_node(int node);
>  void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets);
> @@ -97,5 +101,20 @@ static inline bool node_is_toptier(int node)
>  {
>  	return true;
>  }
> +
> +static inline int register_mt_adistance_algorithm(struct notifier_block *nb)
> +{
> +	return 0;
> +}
> +
> +static inline int unregister_mt_adistance_algorithm(struct notifier_block *nb)
> +{
> +	return 0;
> +}
> +
> +static inline int mt_calc_adistance(int node, int *adist)
> +{
> +	return NOTIFY_DONE;
> +}
>  #endif	/* CONFIG_NUMA */
>  #endif  /* _LINUX_MEMORY_TIERS_H */
> diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
> index 37a4f59d9585..76c0ad47a5ad 100644
> --- a/mm/memory-tiers.c
> +++ b/mm/memory-tiers.c
> @@ -5,6 +5,7 @@
>  #include <linux/kobject.h>
>  #include <linux/memory.h>
>  #include <linux/memory-tiers.h>
> +#include <linux/notifier.h>
>  
>  #include "internal.h"
>  
> @@ -105,6 +106,8 @@ static int top_tier_adistance;
>  static struct demotion_nodes *node_demotion __read_mostly;
>  #endif /* CONFIG_MIGRATION */
>  
> +static BLOCKING_NOTIFIER_HEAD(mt_adistance_algorithms);
> +
>  static inline struct memory_tier *to_memory_tier(struct device *device)
>  {
>  	return container_of(device, struct memory_tier, dev);
> @@ -592,6 +595,62 @@ void clear_node_memory_type(int node, struct memory_dev_type *memtype)
>  }
>  EXPORT_SYMBOL_GPL(clear_node_memory_type);
>  
> +/**
> + * register_mt_adistance_algorithm() - Register memory tiering abstract distance algorithm
> + * @nb: The notifier block which describe the algorithm
> + *
> + * Return: 0 on success, errno on error.
> + *
> + * Every memory tiering abstract distance algorithm provider needs to
> + * register the algorithm with register_mt_adistance_algorithm().  To
> + * calculate the abstract distance for a specified memory node, the
> + * notifier function will be called unless some high priority
> + * algorithm has provided result.  The prototype of the notifier
> + * function is as follows,
> + *
> + *   int (*algorithm_notifier)(struct notifier_block *nb,
> + *                             unsigned long nid, void *data);
> + *
> + * Where "nid" specifies the memory node, "data" is the pointer to the
> + * returned abstract distance (that is, "int *adist").  If the
> + * algorithm provides the result, NOTIFY_STOP should be returned.
> + * Otherwise, return_value & %NOTIFY_STOP_MASK == 0 to allow the next
> + * algorithm in the chain to provide the result.
> + */
> +int register_mt_adistance_algorithm(struct notifier_block *nb)
> +{
> +	return blocking_notifier_chain_register(&mt_adistance_algorithms, nb);
> +}
> +EXPORT_SYMBOL_GPL(register_mt_adistance_algorithm);
> +
> +/**
> + * unregister_mt_adistance_algorithm() - Unregister memory tiering abstract distance algorithm
> + * @nb: the notifier block which describe the algorithm
> + *
> + * Return: 0 on success, errno on error.
> + */
> +int unregister_mt_adistance_algorithm(struct notifier_block *nb)
> +{
> +	return blocking_notifier_chain_unregister(&mt_adistance_algorithms, nb);
> +}
> +EXPORT_SYMBOL_GPL(unregister_mt_adistance_algorithm);
> +
> +/**
> + * mt_calc_adistance() - Calculate abstract distance with registered algorithms
> + * @node: the node to calculate abstract distance for
> + * @adist: the returned abstract distance
> + *
> + * Return: if return_value & %NOTIFY_STOP_MASK != 0, then some
> + * abstract distance algorithm provides the result, and return it via
> + * @adist.  Otherwise, no algorithm can provide the result and @adist
> + * will be kept as it is.
> + */
> +int mt_calc_adistance(int node, int *adist)
> +{
> +	return blocking_notifier_call_chain(&mt_adistance_algorithms, node, adist);
> +}
> +EXPORT_SYMBOL_GPL(mt_calc_adistance);
> +
>  static int __meminit memtier_hotplug_callback(struct notifier_block *self,
>  					      unsigned long action, void *_arg)
>  {