Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.36 as permitted sender) client-ip=23.128.96.36;
Received-SPF: Pass (protection.outlook.com: domain of micron.com designates
 137.201.242.130 as permitted sender) receiver=protection.outlook.com;
 client-ip=137.201.242.130; helo=mail.micron.com; pr=C
From:   <sthanneeru.opensrc@micron.com>
To:     <sthanneeru.opensrc@micron.com>, <linux-cxl@vger.kernel.org>,
        <linux-mm@kvack.org>
CC:     <sthanneeru@micron.com>, <aneesh.kumar@linux.ibm.com>,
        <dan.j.williams@intel.com>, <ying.huang@intel.com>,
        <gregory.price@memverge.com>, <mhocko@suse.com>, <tj@kernel.org>,
        <john@jagalactic.com>, <emirakhur@micron.com>,
        <vtavarespetr@micron.com>, <Ravis.OpenSrc@micron.com>,
        <Jonathan.Cameron@huawei.com>, <linux-kernel@vger.kernel.org>
Subject: [RFC PATCH v2 0/2] Node migration between memory tiers
Date:   Wed, 13 Dec 2023 23:23:27 +0530
Message-ID: <20231213175329.594-1-sthanneeru.opensrc@micron.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Dec 2023 17:53:49.8673
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 84b880a5-a234-4407-0a36-08dbfc047678
X-MS-Exchange-CrossTenant-Id: f38a5ecd-2813-4862-b11b-ac1d563c806f
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f38a5ecd-2813-4862-b11b-ac1d563c806f;Ip=[137.201.242.130];Helo=[mail.micron.com]
X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB52.namprd02.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Anonymous
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR08MB9424
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on pete.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (pete.vger.email [0.0.0.0]); Wed, 13 Dec 2023 09:54:12 -0800 (PST)

From: Srinivasulu Thanneeru <sthanneeru.opensrc@micron.com>

The memory tiers feature allows nodes with similar memory types
or performance characteristics to be grouped together in a
memory tier. However, there is currently no provision for
moving a node from one tier to another on demand.

This patch series aims to support node migration between tiers
on demand by sysadmin/root user using the provided sysfs for
node migration.

To migrate a node to a tier, the corresponding node’s sysfs
memtier_override is written with target tier id.

Example: Move node2 to memory tier2 from its default tier(i.e 4)

1. To check current memtier of node2
$cat  /sys/devices/system/node/node2/memtier_override
memory_tier4

2. To migrate node2 to memory_tier2
$echo 2 > /sys/devices/system/node/node2/memtier_override
$cat  /sys/devices/system/node/node2/memtier_override
memory_tier2

Usecases:

1. Useful to move cxl nodes to the right tiers from userspace, when
   the hardware fails to assign the tiers correctly based on
   memorytypes.

   On some platforms we have observed cxl memory being assigned to
   the same tier as DDR memory. This is arguably a system firmware
   bug, but it is true that tiers represent *ranges* of performance
   and we believe it's important for the system operator to have
   the ability to override bad firmware or OS decisions about tier
   assignment as a fail-safe against potential bad outcomes.

2. Useful if we want interleave weights to be applied on memory tiers
   instead of nodes.
In a previous thread, Huang Ying <ying.huang@intel.com> thought
this feature might be useful to overcome limitations of systems
where nodes with different bandwidth characteristics are grouped
in a single tier.
https://lore.kernel.org/lkml/87a5rw1wu8.fsf@yhuang6-desk2.ccr.corp.intel.com/

=============
Version Notes:

V2 : Changed interface to memtier_override from adistance_offset.
memtier_override was recommended by
1. John Groves <john@jagalactic.com>
2. Ravi Shankar <ravis.opensrc@micron.com>
3. Brice Goglin <Brice.Goglin@inria.fr>

V1 : Introduced adistance_offset sysfs.

=============

Srinivasulu Thanneeru (2):
  base/node: Add sysfs for memtier_override
  memory tier: Support node migration between tiers

 Documentation/ABI/stable/sysfs-devices-node |  7 ++
 drivers/base/node.c                         | 47 ++++++++++++
 include/linux/memory-tiers.h                | 11 +++
 include/linux/node.h                        | 11 +++
 mm/memory-tiers.c                           | 85 ++++++++++++---------
 5 files changed, 125 insertions(+), 36 deletions(-)

-- 
2.25.1