Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
Message-ID: <5706f5e9-0609-98c9-a0cd-7d96336d73dd@amd.com>
Date:   Wed, 1 Jun 2022 11:59:49 +0530
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101
 Thunderbird/91.8.1
Subject: Re: [RFC PATCH v4 4/7] mm/demotion/dax/kmem: Set node's memory tier
 to MEMORY_TIER_PMEM
Content-Language: en-US
To:     "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>,
        linux-mm@kvack.org, akpm@linux-foundation.org
Cc:     Huang Ying <ying.huang@intel.com>,
        Greg Thelen <gthelen@google.com>,
        Yang Shi <shy828301@gmail.com>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Tim C Chen <tim.c.chen@intel.com>,
        Brice Goglin <brice.goglin@gmail.com>,
        Michal Hocko <mhocko@kernel.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Hesham Almatary <hesham.almatary@huawei.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Alistair Popple <apopple@nvidia.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Feng Tang <feng.tang@intel.com>,
        Jagdish Gediya <jvgediya@linux.ibm.com>,
        Baolin Wang <baolin.wang@linux.alibaba.com>,
        David Rientjes <rientjes@google.com>
References: <CAAPL-u-dFp7PwPH6DfbYdnY8xaGsHz3tRQ0CPGVkiqURvdN8=A@mail.gmail.com>
 <20220527122528.129445-1-aneesh.kumar@linux.ibm.com>
 <20220527122528.129445-5-aneesh.kumar@linux.ibm.com>
From:   Bharata B Rao <bharata@amd.com>
In-Reply-To: <20220527122528.129445-5-aneesh.kumar@linux.ibm.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SW9RNUV3ZUVUSjZRK3BhMnNFbUYzNkVLMTVQOWlkQUhPR2M4ODBpZ2xzUEFJ?=
 =?utf-8?B?Ym96YUNSU1VvRXNrRGRsK3VpdVgrcEt3czhQL3ZwOEw2L2V6TEZFb3pnV29a?=
 =?utf-8?B?SEN4UzJQOHNhRlByVnVBQ1hJNTJIZnJ2VG9nd0pLM3ZJQUIrZ1ViaVFpcnAr?=
 =?utf-8?B?ekV1L01Ia3l6QW5jTGZ5VThaOTVTUWhsVWtlU0dUK3IrTnJVd3g3WWJEQVda?=
 =?utf-8?B?S09ROVZrb1FCVmVUWk9YOER4M29SWkQ3TDFGdkpjSnNFQmpGTzBRaS80TnN3?=
 =?utf-8?B?Z0h5eVphQ2paN2E4cEtyblpMd3ZpMG1oSGFwcW42NFZQaHQ2SFkxMnVURllm?=
 =?utf-8?B?aVRjeCtpREoxL3FBWDJBNVZmWnNrZXphaEI4UDZXSlE2bWNVQUZrN3dFWVR5?=
 =?utf-8?B?aU16RW1Ta0tKdFY4SXhMQUZ1QXYzaXN4NWF3U3lLRHdlK21uYTBkbW5YcERV?=
 =?utf-8?B?aFBWNzFMcVhURkxOVUtnbUhBT05HaUg2dGpvQUdhM2c0VjBGa3RVRWxnV3Rp?=
 =?utf-8?B?VFFQdzZMbVVxYlJaNXpKeFg1OW9VS09SY1Y0ZHUwdDFaSTJHdHJYY0xHVEph?=
 =?utf-8?B?dWNwQUlXT0djODRYVVM2QkhKS0ErNklOdndKSE1ESzJZQ3pqS1hwTTNFanNk?=
 =?utf-8?B?cnNNbkVNZm9CeFpmanBMb0FRVWV5VG5BYlZ6UHYvTzJJaisxV0JzSGhGcVda?=
 =?utf-8?B?OWh4SUtiT2VLdWZYR1VISHZaako1M1dkMm5OY1B3ZXYyOElZalZtQ1dVNGJz?=
 =?utf-8?B?emFCZGd0TGFCazQ1MGl0ZGdVOEZDMmlOT1FDVFV2RERRaXVhdGhrNzZIOGx6?=
 =?utf-8?B?ZWFmM2M5WlhLdENYY2Q4OWRrM3pMdVNZbkRyNUxPa1E5blR4cUcwdXRwZlN6?=
 =?utf-8?B?clJwdkdUbzNVN2J0SXhkRFdxYXVSL201TEZSeU1yd3FHb3pGSDkwWUNMd080?=
 =?utf-8?B?ZkIwV0tNZC9ERGRqOVJoUjdHMU5FUTJpTHZVcHlDR3E1aDRUWnJNdXJqaWdQ?=
 =?utf-8?B?QnZ1Mmh5ckxUYldFK0hKTHpJUkhheVJROVBxR1VEek5pUTUrbGJoTnJpQzBv?=
 =?utf-8?B?aWNjeVE0clp2WHhQWFdvdXVOdENxcHlheUpZL3MwZElYUlc0d2h6eHVSbWtD?=
 =?utf-8?B?V2hjbXZBdHVLMVVOTnM0TGJhc2RzVmdNenE1WW1ZR1JxdlUvNHA2eEhoVVRp?=
 =?utf-8?B?bnRCbklLQS8xK3prNTAwMytPNzRCN0dVYzgwUEd1ZTFFOXFTaW05YnMzeUtk?=
 =?utf-8?B?SGlkWDA5SThRYnM2WjM3ejhaWTMwVi9WR004U3pVK1h3dXpIbjROcnVzTnJq?=
 =?utf-8?B?TEpWczZSRExINUp2WHE2QWtmTGlaR3NmNHpOZjh2UTR3U3NrL2tBQmlwYmlF?=
 =?utf-8?B?Wit3Uzh4U1Jzd0ZZa1pzZVFFMEVJYlcxOTZtRUUxbFZKYjRFT1lCUVB0UXFL?=
 =?utf-8?B?Q3VwTG5lY0JYenF2OGl1NGUwOWhmSS9hV3JZdzBFSlF2bW1QREc0SzhOaHhm?=
 =?utf-8?B?N0R3TDM2cHVxTTM3SlJDcWRBSUFVZkRpY2dRSlZZWWVSbXFqbW5GUmVOUTF3?=
 =?utf-8?B?R29NTnhubllkQm42VkdNS21TVHg4NXMvc2hQY3htWUpVTkxWWXMyNmJqd1pC?=
 =?utf-8?B?ZU5LZUZxdlcya0kyRkl6R0d4NFJ3U29zemxsaGY0ZW1LOXVWWndtVGJRTHBH?=
 =?utf-8?B?OFF5alJPQ1h4Q0JNcDFLeENmaGtxOUxPZGJwUUthNjJzZGFaWFlpUElmYm5O?=
 =?utf-8?B?THB1emI3WGtCOEFHSWNFemY0V1RYMXg4MWJVQ0F1a2hYcTVmZldBZ1VaRzdG?=
 =?utf-8?B?ZmFoTXpWK3ZtaXVXRVRLLzk2dUg1Y3dsdWt0NUMydWNGcnpVall6aE1oN0dX?=
 =?utf-8?B?NFBvR0xoTzRzdFZaM0g2SGhVMGhIVkFITmtlcnlnT2Q1Y0VISU9yRXFXN1Z0?=
 =?utf-8?B?Q2ZOS1BrQVVKRFdBNXZVdmtjRTR2bDhZVXJYVDRHbXgwc1BLdTBybytJeHFN?=
 =?utf-8?B?YzRuSVcyU3U3aStTMWYyUjBpeThZL0MyMEsxT2w4RnRiclB0alBJZVhuQWxX?=
 =?utf-8?B?QytUektzeXFlSTI5aGJGNGgzN2JibUt3SHZwVEEwNUxVQStEV0NNanhTUWJq?=
 =?utf-8?B?SGZ0WkxmcXo0L2l3QTJzMnVjN0d1T0NkSjFGM0ZUWkMxeDZXdHRybTFpVm4w?=
 =?utf-8?B?bmxQeFZjeWRBajFjUG5oMFpBSUc5SUFKc2kxMURNTEJwQlBNWlBOUlRXU1Jk?=
 =?utf-8?B?TDF6aW8wek5MbWdWcU1qa2ZhTlVONzFxUy9uTFJRRm11SjIxbkNmQnhNYTc2?=
 =?utf-8?B?d1p1TDVjcUdrektoZWhOclJOWVM3T2lITVhvSzRRcGt2Mkd0cUxXUT09?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f2ff1d18-c109-45ed-815d-08da4398291d
X-MS-Exchange-CrossTenant-AuthSource: IA1PR12MB6434.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Jun 2022 06:30:03.3374
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 7jkpFpjCUXon/dUrlg3OPmA0OrGz0TIxzPzwU5rCqlhxWkmdQIfjZ8dtH6Y6c6NeDAr1LxkGZxLqZgIuu9vR3w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB5654
X-Spam-Status: No, score=-3.2 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,NICE_REPLY_A,RDNS_NONE,SPF_HELO_NONE,
	T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no
	version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 5/27/2022 5:55 PM, Aneesh Kumar K.V wrote:
> From: Jagdish Gediya <jvgediya@linux.ibm.com>
> 
> By default, all nodes are assigned to DEFAULT_MEMORY_TIER which
> is memory tier 1 which is designated for nodes with DRAM, so it
> is not the right tier for dax devices.
> 
> Set dax kmem device node's tier to MEMORY_TIER_PMEM, In future,
> support should be added to distinguish the dax-devices which should
> not be MEMORY_TIER_PMEM and right memory tier should be set for them.
> 
> Signed-off-by: Jagdish Gediya <jvgediya@linux.ibm.com>
> Signed-off-by: Aneesh Kumar K.V <aneesh.kumar@linux.ibm.com>
> ---
>  drivers/dax/kmem.c | 4 ++++
>  mm/migrate.c       | 2 ++
>  2 files changed, 6 insertions(+)
> 
> diff --git a/drivers/dax/kmem.c b/drivers/dax/kmem.c
> index a37622060fff..991782aa2448 100644
> --- a/drivers/dax/kmem.c
> +++ b/drivers/dax/kmem.c
> @@ -11,6 +11,7 @@
>  #include <linux/fs.h>
>  #include <linux/mm.h>
>  #include <linux/mman.h>
> +#include <linux/migrate.h>
>  #include "dax-private.h"
>  #include "bus.h"
>  
> @@ -147,6 +148,9 @@ static int dev_dax_kmem_probe(struct dev_dax *dev_dax)
>  
>  	dev_set_drvdata(dev, data);
>  
> +#ifdef CONFIG_TIERED_MEMORY
> +	node_set_memory_tier(numa_node, MEMORY_TIER_PMEM);
> +#endif

I was experimenting with this patchset and found this behaviour.
Here's what I did:

Boot a KVM guest with vNVDIMM device which ends up with device_dax
driver by default.

Use it as RAM by binding it to dax kmem driver. It now appears as
RAM with a new NUMA node that is put to memtier1 (the existing tier
where DRAM already exists)

I can move it to memtier2 (MEMORY_RANK_PMEM) manually, but isn't
that expected to happen automatically when a node with dax kmem
device comes up?

Regards,
Bharata.