Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
References: <20220704070612.299585-1-aneesh.kumar@linux.ibm.com>
 <YsMAeU2fwEoysohr@casper.infradead.org>
User-agent: mu4e 1.6.9; emacs 27.1
From:   Alistair Popple <apopple@nvidia.com>
To:     Matthew Wilcox <willy@infradead.org>
Cc:     "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>,
        linux-mm@kvack.org, akpm@linux-foundation.org,
        Wei Xu <weixugc@google.com>, Huang Ying <ying.huang@intel.com>,
        Yang Shi <shy828301@gmail.com>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Tim C Chen <tim.c.chen@intel.com>,
        Michal Hocko <mhocko@kernel.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Hesham Almatary <hesham.almatary@huawei.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Johannes Weiner <hannes@cmpxchg.org>, jvgediya.oss@gmail.com
Subject: Re: [PATCH v8 00/12] mm/demotion: Memory tiers and demotion
Date:   Tue, 05 Jul 2022 13:45:58 +1000
In-reply-to: <YsMAeU2fwEoysohr@casper.infradead.org>
Message-ID: <871qv0tczn.fsf@nvdebian.thelocal>
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?gU9jE+t/u7BmZlNAHist7jyl9/PmYA3JA0XbtHbmJoA064IkRek6k5rNPU05?=
 =?us-ascii?Q?kvLbU7m1oNZZJ8I7fQleFkt3ulUiZAdBGrDN2tMdectHYdfiQAbgHR9g1fnl?=
 =?us-ascii?Q?seeNP0vcvpZI2IcuzpODc0c99YIbYKV6APWX9yhDkW0fSzXL0iRkag0scMmZ?=
 =?us-ascii?Q?30uS5NGGf3VYedqFFyJFWyuzLSoXlkBlond5zTcmUHmtlcooIeuAhPd8PjR9?=
 =?us-ascii?Q?3oi5eFyAho0lv63Hg8CWj1M89JBMblRspmn+PiatdTa9FMtP4xAvijpUUxLg?=
 =?us-ascii?Q?G6JVY5SphnyaZkjezZ/ssreh6v8wkUB9d5JIm90u2E+ACY5GzJ6E8bsK9mGy?=
 =?us-ascii?Q?cZqZ/cdqO4BDsc9/R22TtQXBqdHjMshdUbAojI1Iq5EHzynewp70KP0cgSJj?=
 =?us-ascii?Q?yNq4yYo5bKrqYr1HHM0ivmyjGekVa0C+6AGxHKtPIc2x7YrLf0Baaa7UpobR?=
 =?us-ascii?Q?NlRFam6pKYrED85imD9axcPT+MVlf93egeUt1FBFSh8/BzITfwtydl+yH2PJ?=
 =?us-ascii?Q?590KuZl6STmZb24iuBbXcex76T8cYL1el+KtBQAwlejCwC2ecpRinRGXYAWE?=
 =?us-ascii?Q?DytLokyHluKJ2KJgmzmXWFCa/BWzNLL8I0+946qXw4ATQST0N7cY6JmD3EeZ?=
 =?us-ascii?Q?X0gZhEDAkkEiLB50j+rkN88n68N+BxUpu7vr33vuRRdf/Cp6AAoveIi4Tcnm?=
 =?us-ascii?Q?YiUZl8ssoQ2EfBssTnrUmsILRAC9pTFiFasgT+tUkiwZ1FWfxAOp9gI5SEx6?=
 =?us-ascii?Q?AIraURhVpeUkKXG9rxV/vaDRg3WfTbXljWjzDVEY7NdF3r9LQHX2uciuYaK/?=
 =?us-ascii?Q?Cfay15+hb3mnOO71njEJLn8XqzNW3TesGi1YyVREbR2JxfECUadshV9V3Nel?=
 =?us-ascii?Q?UKZw2dFrSVNeMWvAWi9+zeqG7oV9/5zINg0SfrxaBdSzK6nbI2wBKRuxmfYL?=
 =?us-ascii?Q?edC3lpOpp173UlVhz+/0gdvKZVfqrHobIKo/DN79muwvLm3arZpn9UQt3+4F?=
 =?us-ascii?Q?ovkzCw3zHRkBa/ib1NfODiabkCJ3mY4bwDGnfdBV1SzDn2X2OJc4GVflRWuN?=
 =?us-ascii?Q?7joul9R++kZEuLYC4cCNcCGltcpD2EGjUfSY2XG1tucwavC7HMsxA7ucGcSF?=
 =?us-ascii?Q?vfXtaBdWAL0folnYdOGGsPYtKx4Sq2X6XhAhXmxepwehXyF9mSjoy+g0cXH9?=
 =?us-ascii?Q?17Rc2XobIt9Nm5kWmlFGKYfvEZI/jWR8fQQPkSSQM4aXsgL8vwjuFc3QC34F?=
 =?us-ascii?Q?EQgZEtpp6Z9wLqQTEaAxrPr70YUNJoeTyF9QEZOCI9QYnLU+OaFkjrdQ+/4m?=
 =?us-ascii?Q?mT6Z4RsOSnMqn+JtM6x5j1Ymn1kGpilWe8ECkNPGRJ06ki6msdEqWQcqXjXN?=
 =?us-ascii?Q?ZzxoKykCpYsk31JuHTYeFfNcWIZDb1BNEfoIQwdWnt74O/AC07XjSn2F5K9X?=
 =?us-ascii?Q?vpe6Q++a0wsEhNSu6hHedSxehvnNRDUUIYkfqaxXD9zDgPagOJsXkln/Hib5?=
 =?us-ascii?Q?kmd1KW5tZbMNJxg3xurBhkbqwv26BG6A0r29OPr7zGAG76907BfhSe0czKlr?=
 =?us-ascii?Q?wT7Y66gi6A5Mnl2NulioKTkL5/WYD9dtG60TlVAd?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: d6e362f1-f7bb-42d1-2c6c-08da5e3c7804
X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jul 2022 04:11:43.2653
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: EDsI7WrPP6lWlaXTMNl8fKf9dVBhJdNkg+yQdueLsOuHnTIiEcokB8QEevMw7atS3tlW+MByH5TqsWDLRsPwEw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR1201MB2545
X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


Matthew Wilcox <willy@infradead.org> writes:

> On Mon, Jul 04, 2022 at 12:36:00PM +0530, Aneesh Kumar K.V wrote:
>> * The current tier initialization code always initializes
>>   each memory-only NUMA node into a lower tier.  But a memory-only
>>   NUMA node may have a high performance memory device (e.g. a DRAM
>>   device attached via CXL.mem or a DRAM-backed memory-only node on
>>   a virtual machine) and should be put into a higher tier.
>>
>> * The current tier hierarchy always puts CPU nodes into the top
>>   tier. But on a system with HBM (e.g. GPU memory) devices, these
>>   memory-only HBM NUMA nodes should be in the top tier, and DRAM nodes
>>   with CPUs are better to be placed into the next lower tier.
>
> These things that you identify as problems seem perfectly sensible to me.
> Memory which is attached to this CPU has the lowest latency and should
> be preferred over more remote memory, no matter its bandwidth.

It is a problem because HBM NUMA node memory is generally also used by
some kind of device/accelerator (eg. GPU). Typically users would prefer
to keep HBM memory for use by the accelerator rather than random pages
demoted from the CPU as accelerators have orders of magnitude better
performance when accessing local HBM vs. remote memory.