Received: by 2002:a5d:9c59:0:0:0:0:0 with SMTP id 25csp194420iof; Mon, 6 Jun 2022 01:13:17 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxEual1Z9VjbEaqj12Em5IQHd5Yc/41sAh9lTHOKMWimIVQPFqH7lcrcoElFbgQbn7aboLL X-Received: by 2002:a62:31c3:0:b0:51b:9a81:9196 with SMTP id x186-20020a6231c3000000b0051b9a819196mr30622716pfx.70.1654503197397; Mon, 06 Jun 2022 01:13:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1654503197; cv=none; d=google.com; s=arc-20160816; b=fiavSIotwWv3Zlv/D3efBw6GeetB8Oi5IV6IGJJCfmqEVeYbkuVqUtaujNsR0a5RTE J4n/G97iUqmRCu9CbFZ3NauMmGiAt9sduKU8fGD+x5v+VOCuIVKNFXfaobkVVDVx0B5n PN+cPsiTJTfXTmZujsHh7xaIW8Jx1wSBk8Z1G0uwKmnK025pXe5Ie0sxkDAqf0OK9RDa jOFvaqMf3DiKPxYhcPyAY8JueVXccj8Pi9mysPL8XBfIO+q5WfhTW4BhegvuumgQ7wPw 2SLdpfHA2LPKkTgdHZeLB0xalSkXzVf5vaUGpdUCuFhVs11HsPAFI+x7/xbgrdqTGm9A 15jg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:date:cc:to:from:subject :message-id:dkim-signature; bh=Nhij5NvAExtBd/wOGzYACpbHGA2xNymGzer12fNiATM=; b=X4d+0Sx4nd7dXA55diCz3S97CxxrBOVQC0TIN9pbH5RvCi7e4uQbfW9cTWO0yLv6H2 nGlBJ+pPmr6T8g5ZS9DY7F7XWMWfCSbAq8mHD3QdrAG6w//rbA5JYSVx1unTfarPs7TL gP3fNQqIQ7s4mJoer4tKXLrD15ueJuVnwYrHz6og9UZ4uHMsg/LO1a2/i8KPfkgKQIf8 Y3SSFS0BiTxNYsL0x7qefA0QYyo3r5Xo7fQ5n90AIbhB1K+tdhaJT41enknEn86H988D ckB7LlqF68fH1uflt3aZ5FKBrfMi2cWrnYZB8GeLZIB/iUV/YYq16kI6mNegGHTd8+Xx oLmg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=k9A1AgYj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net. [2620:137:e000::1:18]) by mx.google.com with ESMTPS id d12-20020a170902cecc00b00163f8ddf15asi22884281plg.478.2022.06.06.01.13.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Jun 2022 01:13:17 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=k9A1AgYj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id BCF9C12AA4; Mon, 6 Jun 2022 00:53:44 -0700 (PDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231559AbiFFHxc (ORCPT + 99 others); Mon, 6 Jun 2022 03:53:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54972 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231548AbiFFHxX (ORCPT ); Mon, 6 Jun 2022 03:53:23 -0400 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CCFFC12620 for ; Mon, 6 Jun 2022 00:53:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1654502002; x=1686038002; h=message-id:subject:from:to:cc:date:in-reply-to: references:mime-version:content-transfer-encoding; bh=VA0SN9ClYUnctkxy98LYpq0J8mYdtxgaq9LWNKW7Ydw=; b=k9A1AgYj6iGde6e9XnMC7LLjxOsdrV+BWxdvc8tHIRvgaApP+hhdjOJD lCGW22/6YoPdBnkvo+hG5EUfG8AbRhQCol22BgKlZxoB6tgX8WeiYVVCm b5LSIn5MFzFB6ur8c8AXzewqCud0OKBqVB0KOBOJRVfMzzuzgpPs0hxjv cPJjzPkFGjQ560xoI0Qu80epzsG/L9mZlIRr31sebMazZzRxfrkdKBD+O t9JF2Ie8EM8gADg3VTokAiEWuh88Ncg7gthXrmoymd7Y8Jg7rLaKAwfom vNGtLSXrPwbLmKbPRkrOoNHMYbOVOMC8FVo7HrV14NX4Mly/9QViV0YOZ w==; X-IronPort-AV: E=McAfee;i="6400,9594,10369"; a="256472741" X-IronPort-AV: E=Sophos;i="5.91,280,1647327600"; d="scan'208";a="256472741" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Jun 2022 00:53:22 -0700 X-IronPort-AV: E=Sophos;i="5.91,280,1647327600"; d="scan'208";a="635470930" Received: from xingguom-mobl.ccr.corp.intel.com ([10.254.213.116]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Jun 2022 00:53:14 -0700 Message-ID: Subject: Re: [RFC PATCH v4 1/7] mm/demotion: Add support for explicit memory tiers From: Ying Huang To: "Aneesh Kumar K.V" Cc: Greg Thelen , Yang Shi , Davidlohr Bueso , Tim C Chen , Brice Goglin , Michal Hocko , Linux Kernel Mailing List , Hesham Almatary , Dave Hansen , Jonathan Cameron , Alistair Popple , Dan Williams , Feng Tang , Jagdish Gediya , Baolin Wang , David Rientjes , linux-mm@kvack.org, akpm@linux-foundation.org Date: Mon, 06 Jun 2022 15:53:09 +0800 In-Reply-To: <87ilpe8fxh.fsf@linux.ibm.com> References: <20220527122528.129445-1-aneesh.kumar@linux.ibm.com> <20220527122528.129445-2-aneesh.kumar@linux.ibm.com> <352ae5f408b6d7d4d3d820d68e2f2c6b494e95e1.camel@intel.com> <143e40bcf46097d14514504518fdc1870fd8d4a1.camel@intel.com> <87ilpe8fxh.fsf@linux.ibm.com> Content-Type: text/plain; charset="UTF-8" User-Agent: Evolution 3.38.3-1 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-3.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 2022-06-06 at 11:57 +0530, Aneesh Kumar K.V wrote: > Aneesh Kumar K V writes: > > > On 6/6/22 11:03 AM, Ying Huang wrote: > > > On Mon, 2022-06-06 at 09:26 +0530, Aneesh Kumar K V wrote: > > > > On 6/6/22 8:19 AM, Ying Huang wrote: > > > > > On Thu, 2022-06-02 at 14:07 +0800, Ying Huang wrote: > > > > > > On Fri, 2022-05-27 at 17:55 +0530, Aneesh Kumar K.V wrote: > > > > > > > From: Jagdish Gediya > > > > > > > > > > > > > > In the current kernel, memory tiers are defined implicitly via a > > > > > > > demotion path relationship between NUMA nodes, which is created > > > > > > > during the kernel initialization and updated when a NUMA node is > > > > > > > hot-added or hot-removed. The current implementation puts all > > > > > > > nodes with CPU into the top tier, and builds the tier hierarchy > > > > > > > tier-by-tier by establishing the per-node demotion targets based > > > > > > > on the distances between nodes. > > > > > > > > > > > > > > This current memory tier kernel interface needs to be improved for > > > > > > > several important use cases, > > > > > > > > > > > > > > The current tier initialization code always initializes > > > > > > > each memory-only NUMA node into a lower tier. But a memory-only > > > > > > > NUMA node may have a high performance memory device (e.g. a DRAM > > > > > > > device attached via CXL.mem or a DRAM-backed memory-only node on > > > > > > > a virtual machine) and should be put into a higher tier. > > > > > > > > > > > > > > The current tier hierarchy always puts CPU nodes into the top > > > > > > > tier. But on a system with HBM or GPU devices, the > > > > > > > memory-only NUMA nodes mapping these devices should be in the > > > > > > > top tier, and DRAM nodes with CPUs are better to be placed into the > > > > > > > next lower tier. > > > > > > > > > > > > > > With current kernel higher tier node can only be demoted to selected nodes on the > > > > > > > next lower tier as defined by the demotion path, not any other > > > > > > > node from any lower tier. This strict, hard-coded demotion order > > > > > > > does not work in all use cases (e.g. some use cases may want to > > > > > > > allow cross-socket demotion to another node in the same demotion > > > > > > > tier as a fallback when the preferred demotion node is out of > > > > > > > space), This demotion order is also inconsistent with the page > > > > > > > allocation fallback order when all the nodes in a higher tier are > > > > > > > out of space: The page allocation can fall back to any node from > > > > > > > any lower tier, whereas the demotion order doesn't allow that. > > > > > > > > > > > > > > The current kernel also don't provide any interfaces for the > > > > > > > userspace to learn about the memory tier hierarchy in order to > > > > > > > optimize its memory allocations. > > > > > > > > > > > > > > This patch series address the above by defining memory tiers explicitly. > > > > > > > > > > > > > > This patch adds below sysfs interface which is read-only and > > > > > > > can be used to read nodes available in specific tier. > > > > > > > > > > > > > > /sys/devices/system/memtier/memtierN/nodelist > > > > > > > > > > > > > > Tier 0 is the highest tier, while tier MAX_MEMORY_TIERS - 1 is the > > > > > > > lowest tier. The absolute value of a tier id number has no specific > > > > > > > meaning. what matters is the relative order of the tier id numbers. > > > > > > > > > > > > > > All the tiered memory code is guarded by CONFIG_TIERED_MEMORY. > > > > > > > Default number of memory tiers are MAX_MEMORY_TIERS(3). All the > > > > > > > nodes are by default assigned to DEFAULT_MEMORY_TIER(1). > > > > > > > > > > > > > > Default memory tier can be read from, > > > > > > > /sys/devices/system/memtier/default_tier > > > > > > > > > > > > > > Max memory tier can be read from, > > > > > > > /sys/devices/system/memtier/max_tiers > > > > > > > > > > > > > > This patch implements the RFC spec sent by Wei Xu at [1]. > > > > > > > > > > > > > > [1] https://lore.kernel.org/linux-mm/CAAPL-u-DGLcKRVDnChN9ZhxPkfxQvz9Sb93kVoX_4J2oiJSkUw@mail.gmail.com/ > > > > > > > > > > > > > > Signed-off-by: Jagdish Gediya > > > > > > > Signed-off-by: Aneesh Kumar K.V > > > > > > > > > > > > IMHO, we should change the kernel internal implementation firstly, then > > > > > > implement the kerne/user space interface. That is, make memory tier > > > > > > explicit inside kernel, then expose it to user space. > > > > > > > > > > Why ignore this comment for v5? If you don't agree, please respond me. > > > > > > > > > > > > > I am not sure what benefit such a rearrange would bring in? Right now I > > > > am writing the series from the point of view of introducing all the > > > > plumbing and them switching the existing demotion logic to use the new > > > > infrastructure. Redoing the code to hide all the userspace sysfs till we > > > > switch the demotion logic to use the new infrastructure doesn't really > > > > bring any additional clarity to patch review and would require me to > > > > redo the series with a lot of conflicts across the patches in the patchset. > > > > > > IMHO, we shouldn't introduce regression even in the middle of a > > > patchset. Each step should only rely on previous patches in the series > > > to work correctly. In your current way of organization, after patch > > > [1/7], on a system with 2 memory tiers, the user space interface will > > > output wrong information (only 1 memory tier). So I think the correct > > > way is to make it right inside the kenrel firstly, then expose the right > > > information to user space. > > > > > > > The patchset doesn't add additional tier until "mm/demotion/dax/kmem: > > Set node's memory tier to MEMORY_TIER_PMEM". ie, there is no additional > > tiers done till all the demotion logic is in place. So even if the > > system got dax/kmem, the support for adding dax/kmem as a memory tier > > comes later in the patch series. > > Let me clarify this a bit more. This patchset doesn't change the > existing kernel behavior till "mm/demotion: Build demotion targets > based on explicit memory tiers". So there is no regression till then. > It adds a parallel framework (memory tiers to the existing demotion > logic). > > I can move the patch "mm/demotion/dax/kmem: Set node's memory tier to > MEMORY_TIER_PMEM" before switching the demotion logic so that on systems > with two memory tiers (DRAM and pmem) the demotion continues to work > as expected after patch 3 ("mm/demotion: Build demotion targets based on > explicit memory tiers"). With that, there will not be any regression in > between the patch series. > Thanks! Please do that. And I think you can add sysfs interface after that patch too. That is, in [1/7] +struct memory_tier { + nodemask_t nodelist; +}; And struct device can be added after the kernel has switched the implementation based on explicit memory tiers. +struct memory_tier { + struct device dev; + nodemask_t nodelist; +}; But I don't think it's a good idea to have "struct device" embedded in "struct memory_tier". We don't have "struct device" embedded in "struct pgdata_list"... Best Regards, Huang, Ying