Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:6 as permitted sender) client-ip=2620:137:e000::3:6;
Message-ID: <138395ff-1ff0-bcbd-3636-a1b8cfbabd13@intel.com>
Date:   Thu, 14 Sep 2023 10:29:57 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Betterbird/102.13.0
Subject: Re: [PATCH -V3 1/4] memory tiering: add abstract distance calculation
 algorithms management
To:     Huang Ying <ying.huang@intel.com>,
        Andrew Morton <akpm@linux-foundation.org>
CC:     <linux-mm@kvack.org>, <linux-kernel@vger.kernel.org>,
        Bharata B Rao <bharata@amd.com>,
        "Aneesh Kumar K . V" <aneesh.kumar@linux.ibm.com>,
        Wei Xu <weixugc@google.com>,
        Alistair Popple <apopple@nvidia.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Dave Hansen <dave.hansen@intel.com>,
        "Davidlohr Bueso" <dave@stgolabs.net>,
        Johannes Weiner <hannes@cmpxchg.org>,
        "Jonathan Cameron" <Jonathan.Cameron@huawei.com>,
        Michal Hocko <mhocko@kernel.org>,
        "Yang Shi" <shy828301@gmail.com>,
        Rafael J Wysocki <rafael.j.wysocki@intel.com>
References: <20230912082101.342002-1-ying.huang@intel.com>
 <20230912082101.342002-2-ying.huang@intel.com>
Content-Language: en-US
From:   Dave Jiang <dave.jiang@intel.com>
In-Reply-To: <20230912082101.342002-2-ying.huang@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bUhkVHBsU2ZHWTF3VGp6NDBhQXU5L1RkZXcyY3JtdWtmcTlIWW81QUdJekFZ?=
 =?utf-8?B?RHBONW9JdXd4UVI4aDRMQjBNR2E3MklGUlh2TlRMQ1UwUHo1VTdQaS9TTzZJ?=
 =?utf-8?B?UHR6ZkdNSlAzeHU0czgyWWhXSVA3UlpyekZLczVWV1RKb243NFF4ZGZ6Y1k3?=
 =?utf-8?B?K1hER1ZsQlhkNXdzWnZSeFVzbU42MTZaNkJWVUxCQm5mbGF5Zk5iQ0FCVE1R?=
 =?utf-8?B?SlBTUU8xamxnMElCMzJOZWFwOFZTTndKZEIwNlVHdWhqSEtpSm13YVhydVFt?=
 =?utf-8?B?UXB2SStFZ1pZWFo2Qys3dWJST0kyczhiZmd1cWI5YzNxaHlhOGppNTZCV1Mz?=
 =?utf-8?B?Yi9YM1F2dnZEalZDdWYwUXNuMGZ1bjRJNWRHZ2pMZC8wc1J6NDE5Q1NjWHFR?=
 =?utf-8?B?YTNlOGFSWmJyWEl6bFRkalpqQXZmYVRodDdtbjlUbFdQMnUzMGxsZFAyZ2JJ?=
 =?utf-8?B?Yy9HdkswYnNtTmJFZWVOOGQ5TEc4bSs5OW10bllHUjF3WUkrVkNMRzEzTXlI?=
 =?utf-8?B?blAxL1ZqWEl4VTNvdGxmTFp4MHIxdHNZcDRpN3pYUGVHTFY5ajdyMXFMdnBT?=
 =?utf-8?B?eHc5S1BHMW13WDc4WUVTWmQ4eHc0SXRKN2ZEZjlGcVg5ZWpocENKNmRLY3M0?=
 =?utf-8?B?VFhFRGwxcEJPaFVmWmp6M3VQaDRWeXBGWGhwMUpnTG1maGFBNGtwN1FaK3dP?=
 =?utf-8?B?aGxQMUUrVVphb1RNWlFweENNc3lMb0dFaGVCL01Cc2ZPOHZ0RVJEbGdsNE8x?=
 =?utf-8?B?ak16UlRxb0JCOXVUdEdWL3NwMElLMnlJYWU3b2UxY1o2TjR6cnpEdUVFeGlk?=
 =?utf-8?B?WTBFczdaVS9iYTQ0VGp0ZWJjZzN2S1BadGlJenJUQjJzQnl5UzlSbnBvdDFI?=
 =?utf-8?B?L2VCWmRxMFp5T1dURmZJVC9GUHNCWGRraXdQSUVxYXVqWEFxMWl6bDdNdnNl?=
 =?utf-8?B?UGpYb2FRcjUvMC8zalV4WjQxbHMwcXpFTzg5OUNqTWpvSzFmblF1YmJUdlFa?=
 =?utf-8?B?RVNlMVF6M0luc29nZ0FKVmZmT0t0VGJtQTZJeWlBUUQ0ZmJVWXQybGtOMFdX?=
 =?utf-8?B?em9WK3l4aEZuQUpvRHQ4bkJBSnFGckxGK2FLQUkxQTMvMW5FNUJKT2RkRVhu?=
 =?utf-8?B?aHR4cExuQi80UG1TNHM1SWhCaitxWDk1aG05L2xOMkNvdnNHUHZWY2xmMktz?=
 =?utf-8?B?NlFoVjZsbW14YjhDNExMajNXTWx3M3hlMUhkT3FNMEV0YVJxQ25iRi9zVXFn?=
 =?utf-8?B?WDk2ckFqbFNPMXBiaGE0R3Z0SEFDOWUvYjBselRjV0Z0RXhCeGhTRTdKcW0z?=
 =?utf-8?B?Y28rSktHK2tLRDlCT3Yxc0wySTR1NlNsN202SWIzeDZlYzF3MkZpemYrVk9i?=
 =?utf-8?B?c0lVaEJwUUJTVFRJZHFSaE9JeVhBeXpQZ21KUS9XRTlBSWxQTUNEMG9DYUp5?=
 =?utf-8?B?RkJDdEZnSkJaSVZCdk1WQ2x5YmRNUEVNajVUVUlTNURCaFpxMk9FMWFvb2RC?=
 =?utf-8?B?bTY0Z0lueG5UakwxRXVuTUNiRkxaa2VMUENVOWhhOEp2S1ZaK0xKRzNMZXE1?=
 =?utf-8?B?SWVuN1ZmcjN1MUFMalJDdGd0OGlRcU5mWnVpSisrVTdZQ0JZZTNEK1psOUFV?=
 =?utf-8?B?dGhDWWUxVG5NRXZkUm55YXFkaWR1Z3F3TzhiRC9qOFpMSUVLVTBOamdTdlk0?=
 =?utf-8?B?UzFOUDRENkdkbkxYcnU3b1ZqUnBMRGh1KzcxVCthUnNLWlNwTUhBRHNmbjdr?=
 =?utf-8?B?VDlVSDRTWlFYNTh3Q29DNDlnbDJZQktsU2ZXNW56WUNwUVVWbGlWUHN6aHdS?=
 =?utf-8?B?WU1DWGkxUFhSek1vNGJIcGtuQ3lCVklEc0h5L1FUT1lGNmJ0WVF3Wk1EOWYv?=
 =?utf-8?B?MlNzQU1CVTFmUU96dVRubmdnbmNGNnJSRGQ2aTdCY0xDREpreHdIay9nL3Jk?=
 =?utf-8?B?NENDa3BmbGZ1c24vUUh5T2FEWENwcEFaakc1bHF2NlFYUENOc0tzNkpuOVdC?=
 =?utf-8?B?dW1XRHFCank1V2ZJTk8rUHZMaFlQd1pGa29iNGhheDZmU0NpdFRZbGcxWVo5?=
 =?utf-8?B?ckRRWCtSdDdOWHdEcitHMFlRencraDlKckRnQ2puYkd0UE51QTN5MUtZbEN3?=
 =?utf-8?Q?SZRxYi/f1fZUQ7URWpalIkk/l?=
X-MS-Exchange-CrossTenant-Network-Message-Id: f0fa7330-1111-4c6d-ef6e-08dbb5483a25
X-MS-Exchange-CrossTenant-AuthSource: PH7PR11MB5984.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Sep 2023 17:30:02.5307
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 7bSwvHTxQcVhuK0XQwnxTovL1als7jl4D7vYWHDm2fE7eL2nkaTdtptbVAJYujbT400T0o9pqE++ieuS8xSXBg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR11MB7325
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Thu, 14 Sep 2023 10:30:21 -0700 (PDT)
X-Spam-Status: No, score=-2.3 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email


On 9/12/23 01:20, Huang Ying wrote:
> The abstract distance may be calculated by various drivers, such as
> ACPI HMAT, CXL CDAT, etc.  While it may be used by various code which
> hot-add memory node, such as dax/kmem etc.  To decouple the algorithm
> users and the providers, the abstract distance calculation algorithms
> management mechanism is implemented in this patch.  It provides
> interface for the providers to register the implementation, and
> interface for the users.
> 
> Multiple algorithm implementations can cooperate via calculating
> abstract distance for different memory nodes.  The preference of
> algorithm implementations can be specified via
> priority (notifier_block.priority).
> 
> Signed-off-by: "Huang, Ying" <ying.huang@intel.com>
> Tested-by: Bharata B Rao <bharata@amd.com>
> Cc: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
> Cc: Wei Xu <weixugc@google.com>
> Cc: Alistair Popple <apopple@nvidia.com>
> Cc: Dan Williams <dan.j.williams@intel.com>
> Cc: Dave Hansen <dave.hansen@intel.com>
> Cc: Davidlohr Bueso <dave@stgolabs.net>
> Cc: Johannes Weiner <hannes@cmpxchg.org>
> Cc: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> Cc: Michal Hocko <mhocko@kernel.org>
> Cc: Yang Shi <shy828301@gmail.com>
> Cc: Dave Jiang <dave.jiang@intel.com>
> Cc: Rafael J Wysocki <rafael.j.wysocki@intel.com>

Reviewed-by: Dave Jiang <dave.jiang@intel.com>

> ---
>  include/linux/memory-tiers.h | 19 ++++++++++++
>  mm/memory-tiers.c            | 59 ++++++++++++++++++++++++++++++++++++
>  2 files changed, 78 insertions(+)
> 
> diff --git a/include/linux/memory-tiers.h b/include/linux/memory-tiers.h
> index 437441cdf78f..c8382220cced 100644
> --- a/include/linux/memory-tiers.h
> +++ b/include/linux/memory-tiers.h
> @@ -6,6 +6,7 @@
>  #include <linux/nodemask.h>
>  #include <linux/kref.h>
>  #include <linux/mmzone.h>
> +#include <linux/notifier.h>
>  /*
>   * Each tier cover a abstrace distance chunk size of 128
>   */
> @@ -36,6 +37,9 @@ struct memory_dev_type *alloc_memory_type(int adistance);
>  void put_memory_type(struct memory_dev_type *memtype);
>  void init_node_memory_type(int node, struct memory_dev_type *default_type);
>  void clear_node_memory_type(int node, struct memory_dev_type *memtype);
> +int register_mt_adistance_algorithm(struct notifier_block *nb);
> +int unregister_mt_adistance_algorithm(struct notifier_block *nb);
> +int mt_calc_adistance(int node, int *adist);
>  #ifdef CONFIG_MIGRATION
>  int next_demotion_node(int node);
>  void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets);
> @@ -97,5 +101,20 @@ static inline bool node_is_toptier(int node)
>  {
>  	return true;
>  }
> +
> +static inline int register_mt_adistance_algorithm(struct notifier_block *nb)
> +{
> +	return 0;
> +}
> +
> +static inline int unregister_mt_adistance_algorithm(struct notifier_block *nb)
> +{
> +	return 0;
> +}
> +
> +static inline int mt_calc_adistance(int node, int *adist)
> +{
> +	return NOTIFY_DONE;
> +}
>  #endif	/* CONFIG_NUMA */
>  #endif  /* _LINUX_MEMORY_TIERS_H */
> diff --git a/mm/memory-tiers.c b/mm/memory-tiers.c
> index 37a4f59d9585..76c0ad47a5ad 100644
> --- a/mm/memory-tiers.c
> +++ b/mm/memory-tiers.c
> @@ -5,6 +5,7 @@
>  #include <linux/kobject.h>
>  #include <linux/memory.h>
>  #include <linux/memory-tiers.h>
> +#include <linux/notifier.h>
>  
>  #include "internal.h"
>  
> @@ -105,6 +106,8 @@ static int top_tier_adistance;
>  static struct demotion_nodes *node_demotion __read_mostly;
>  #endif /* CONFIG_MIGRATION */
>  
> +static BLOCKING_NOTIFIER_HEAD(mt_adistance_algorithms);
> +
>  static inline struct memory_tier *to_memory_tier(struct device *device)
>  {
>  	return container_of(device, struct memory_tier, dev);
> @@ -592,6 +595,62 @@ void clear_node_memory_type(int node, struct memory_dev_type *memtype)
>  }
>  EXPORT_SYMBOL_GPL(clear_node_memory_type);
>  
> +/**
> + * register_mt_adistance_algorithm() - Register memory tiering abstract distance algorithm
> + * @nb: The notifier block which describe the algorithm
> + *
> + * Return: 0 on success, errno on error.
> + *
> + * Every memory tiering abstract distance algorithm provider needs to
> + * register the algorithm with register_mt_adistance_algorithm().  To
> + * calculate the abstract distance for a specified memory node, the
> + * notifier function will be called unless some high priority
> + * algorithm has provided result.  The prototype of the notifier
> + * function is as follows,
> + *
> + *   int (*algorithm_notifier)(struct notifier_block *nb,
> + *                             unsigned long nid, void *data);
> + *
> + * Where "nid" specifies the memory node, "data" is the pointer to the
> + * returned abstract distance (that is, "int *adist").  If the
> + * algorithm provides the result, NOTIFY_STOP should be returned.
> + * Otherwise, return_value & %NOTIFY_STOP_MASK == 0 to allow the next
> + * algorithm in the chain to provide the result.
> + */
> +int register_mt_adistance_algorithm(struct notifier_block *nb)
> +{
> +	return blocking_notifier_chain_register(&mt_adistance_algorithms, nb);
> +}
> +EXPORT_SYMBOL_GPL(register_mt_adistance_algorithm);
> +
> +/**
> + * unregister_mt_adistance_algorithm() - Unregister memory tiering abstract distance algorithm
> + * @nb: the notifier block which describe the algorithm
> + *
> + * Return: 0 on success, errno on error.
> + */
> +int unregister_mt_adistance_algorithm(struct notifier_block *nb)
> +{
> +	return blocking_notifier_chain_unregister(&mt_adistance_algorithms, nb);
> +}
> +EXPORT_SYMBOL_GPL(unregister_mt_adistance_algorithm);
> +
> +/**
> + * mt_calc_adistance() - Calculate abstract distance with registered algorithms
> + * @node: the node to calculate abstract distance for
> + * @adist: the returned abstract distance
> + *
> + * Return: if return_value & %NOTIFY_STOP_MASK != 0, then some
> + * abstract distance algorithm provides the result, and return it via
> + * @adist.  Otherwise, no algorithm can provide the result and @adist
> + * will be kept as it is.
> + */
> +int mt_calc_adistance(int node, int *adist)
> +{
> +	return blocking_notifier_call_chain(&mt_adistance_algorithms, node, adist);
> +}
> +EXPORT_SYMBOL_GPL(mt_calc_adistance);
> +
>  static int __meminit memtier_hotplug_callback(struct notifier_block *self,
>  					      unsigned long action, void *_arg)
>  {