Received-SPF: pass (google.com: domain of linux-kernel+bounces-125247-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Fri, 29 Mar 2024 13:28:13 -0400
From: Gregory Price <gregory.price@memverge.com>
To: Yuanchu Xie <yuanchu@google.com>
Cc: David Hildenbrand <david@redhat.com>,
	"Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>,
	Khalid Aziz <khalid.aziz@oracle.com>,
	Henry Huang <henry.hj@antgroup.com>, Yu Zhao <yuzhao@google.com>,
	Dan Williams <dan.j.williams@intel.com>,
	Huang Ying <ying.huang@intel.com>, Wei Xu <weixugc@google.com>,
	David Rientjes <rientjes@google.com>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	"Rafael J. Wysocki" <rafael@kernel.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	Johannes Weiner <hannes@cmpxchg.org>,
	Michal Hocko <mhocko@kernel.org>,
	Roman Gushchin <roman.gushchin@linux.dev>,
	Muchun Song <muchun.song@linux.dev>, Shuah Khan <shuah@kernel.org>,
	Yosry Ahmed <yosryahmed@google.com>,
	Matthew Wilcox <willy@infradead.org>,
	Sudarshan Rajagopalan <quic_sudaraja@quicinc.com>,
	Kairui Song <kasong@tencent.com>,
	"Michael S. Tsirkin" <mst@redhat.com>,
	Vasily Averin <vasily.averin@linux.dev>,
	Nhat Pham <nphamcs@gmail.com>, Miaohe Lin <linmiaohe@huawei.com>,
	Qi Zheng <zhengqi.arch@bytedance.com>,
	Abel Wu <wuyun.abel@bytedance.com>,
	"Vishal Moola (Oracle)" <vishal.moola@gmail.com>,
	Kefeng Wang <wangkefeng.wang@huawei.com>,
	linux-kernel@vger.kernel.org, linux-mm@kvack.org,
	cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org
Subject: Re: [RFC PATCH v3 0/8] mm: workingset reporting
Message-ID: <Zgb6LQndjoFVu4pv@memverge.com>
References: <20240327213108.2384666-1-yuanchu@google.com>
 <ZgSTNCP5f+T5VtBI@memverge.com>
 <CAJj2-QEg3+Ztg3rK6FpVVCxSG4DaDPWsO_bha5v5GrJazc5DVQ@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <CAJj2-QEg3+Ztg3rK6FpVVCxSG4DaDPWsO_bha5v5GrJazc5DVQ@mail.gmail.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?alY5QzNYOUFSOUNXUTVrVzYrNElGeGZJZVFKVURNdkpHZUdURnd4Q2dzK2N1?=
 =?utf-8?B?QnlvMG41aXJQcnlCTjNIMkdvbVZoNkVQY2QzNmNVQTk1YkRsekFIYTMvSjlY?=
 =?utf-8?B?dmpQVzQ2RWFsOXMxTERYWTdOaGlNNHBSSkZMc0lNSmUwbS9HdENtRGFIMi90?=
 =?utf-8?B?QVN3ekZsaG51QlljeUsxNk9pSG9ucVpYM3R5SFJFNWtsWGloUW9zSnZhWE5m?=
 =?utf-8?B?Y0VZaHAwTENQemJiUGRZWmN5b1oyaDZSU1hWK3I2VjNRWG1aTm5CelpuaG1s?=
 =?utf-8?B?T3lobXpDQ0xJYUtlSEt5a04xVzZ3WWNLZjZSZUp6MUlvVDBtTlVmbVZET1c4?=
 =?utf-8?B?NDdmZ3BXdjJRNTBLakJ3N2R4SERuUDJjR0RwUGkwS1BzL0U0SjhDU1Z2d2ZX?=
 =?utf-8?B?RXN0akdxeWFtS3A4M3FUbHhqQ1IvNXR0aTNQSk9qcU9ud25CZjIyOVlHVVlp?=
 =?utf-8?B?emhXcW5zK3pWMWZqSzFYNXYvUmJvTkFIK2tjUmY2UTJsV2hNb1FvUkdXNEQ4?=
 =?utf-8?B?RUtvZ0VGK1NtL2tBdzNobnk5ak92NTVBL1RZeWVWcFlSUTNKK3FMREFqWnBC?=
 =?utf-8?B?UEdlMFIvZ3RqWWhZd3ZMMk9acDFMTWtBaExvc3AreGRSRTIvMDM3RUI5OHhI?=
 =?utf-8?B?NTEvNyt4VkxhWlV3THRoM0dGVlVzVCt5cW4wNSsrL3lMTUszTmF3M0oyNGpk?=
 =?utf-8?B?eUFhMlNtZGF1bUFPcWZETHZOeG5iS0YvUHlDWGtuS2NQSHltSG8zUGlqdG5B?=
 =?utf-8?B?SXVQM2R1dHhSa2w5RW4rZWdrMDY1S1FjS0h0ZFNBbVN4dGtMbWdNTXJTSDhs?=
 =?utf-8?B?WUVmUkd2Y3FXaDRCRldBT0lPZEhXUVRmemtUT1BJdnpCWnVqQ2JqTitROTRr?=
 =?utf-8?B?SEVXenMxTzdDaVVNdTBNN1N3NWprSVZnZ25DYW1YQWpzVDNSZW5xcFVINDNH?=
 =?utf-8?B?NHVvTVdXbDl1UEt6Zm50U09nSGRXcFVtbVd1L0k0QjEyMFgzd0dXTGdNdkt5?=
 =?utf-8?B?RU45YUo2TkZPOVpocnA2ZFhTUjVNZGRDNGJhUEcvUlhtNmNnTmNJeTRFam1M?=
 =?utf-8?B?a2Zpblh5TmhxSVV0NUxRbDdxcElydDMwMnVpSWFLRjN1b3VqKzJ5RjhvaFF5?=
 =?utf-8?B?Y0h2dGNPUUduMXRpRUxJUit1ZEtIMnRTZXNsWW0zZVRYUEdzd1RGSzl3Rkpv?=
 =?utf-8?B?Vjl4empISWJISDNibmN3Qnlpem5ydWVIRUVlYjhGV2lpMDNyUEJ6eVcrMkNF?=
 =?utf-8?B?alBUSjZZbmR0aDBqVFRsS1hjUytVU3JCQnY2U1NJR1J5RmozTWdoaWdZQXZR?=
 =?utf-8?B?OVFzbUg4dkNFZDVOREErK0ovR29BS0hjK2ZmRXNCZVEvQUY3QzJKM0pXM0VK?=
 =?utf-8?B?eFR4eWRpYWVDZ2VPalVxN1JUUzg1cG1DN1FKTXRXanhNdlVpQW9LcWxoUVkw?=
 =?utf-8?B?b3BsekZXeXRBanlUQkpIWS9zbWFQOVBET0hnZ25IcURKS1h2T25rcHg4bEx4?=
 =?utf-8?B?LzR5UERNNUxSa0FMM1hBWnJUcXJxWWpFcG9EWHNYZEJrc3lnS09uSHNKbkZj?=
 =?utf-8?B?RmJocVp4ODRIN2liSlBoSU9NSVJuZHdncnhrSDdUUjc5aDZnRlJTRXplVFla?=
 =?utf-8?B?eUxDWjBsem5ncEM2THBKcUZXb2VzbEladzZFakJZbFBXbDRmcWFaelFWL2Nw?=
 =?utf-8?B?WERwNjRpaHQybTM1cU9CQmRpSTlkTUcwZkliMlVmamNpK2xMMGZ4RDdLSjNZ?=
 =?utf-8?B?Wk5tbWhrQWpabFF0STVmR05zUTdSN1RsM1h6SGFHYUQ1ZGtSYlRib09HRk5T?=
 =?utf-8?B?SjRRbXhvVTVxTXhuTGxEcnVPVmFGeGFCUnZOUlBKL2VrbkFoNkUrdEhFK2tS?=
 =?utf-8?B?RXFiR2hmWnVUdFFaZFoxWVcwcnVHUzRqcTRjM1djdWxhQ3VJcU9RaWNwbG40?=
 =?utf-8?B?d2dodkRtbEZFdkZtZE83N2dNOTNoRUhGbEVHK21KeFBEY25YazBaK2djV1BI?=
 =?utf-8?B?Snpza2pHdkNodUNQVXpXekFzeHhXbTN5OXlHNWlZQmUzZnpyZk1wR2RsYjVl?=
 =?utf-8?B?T3VaUG9IL005UFFzTzBSTGRhQmJXaWh2b1pVR1NvRlJwTkVLemR6QjRIR3lY?=
 =?utf-8?B?Ui9XcWprZEowZFRXbnQxM3QwOEpxbm0yZ0VVY0tuWldYbnRxc2M5dncyZjFa?=
 =?utf-8?B?aXc9PQ==?=
X-OriginatorOrg: memverge.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f767b94f-d74b-4396-d193-08dc50159f9c
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR17MB5512.namprd17.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Mar 2024 17:28:18.3368
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: P3vWrgl5ZKYm1c/aLqaIIdgVn6LkhTeVw2POOIzyQ+jtfNALeW/4EVp613jBNaTQCUKwNhYCoh1YrMdheUoCg3vxP00bGz+m1Hb9RfWwIUU=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR17MB6213

On Wed, Mar 27, 2024 at 03:53:39PM -0700, Yuanchu Xie wrote:
> On Wed, Mar 27, 2024 at 2:44 PM Gregory Price
> <gregory.price@memverge.com> wrote:
> >
> > Please note that this proposed interface (move_phys_pages) is very
> > unlikely to be received upstream due to side channel concerns. Instead,
> > it's more likely that the tiering component will expose a "promote X
> > pages from tier A to tier B", and the kernel component would then
> > use/consume hotness information to determine which pages to promote.
> 
> I see that mm/memory-tiers.c only has support for demotion. What kind
> of hotness information do devices typically provide? The OCP proposal
> is not very specific about this.
> A list of hot pages with configurable threshold?
> Access frequency for all pages at configured granularity?
> Is there a way to tell which NUMA node is accessing them, for page promotion?

(caveat: i'm not a memory-tiers maintainer, you may want to poke at them
directly for more information, this is simply spitballing an idea)

I don't know of any public proposals of explicit hotness information
provided by hardware yet, just the general proposal.

For the sake of simplicity, I would make the assumption that you have
the least information possible - a simple list of "hot addresses" in
Host Physcal Address format.

I.e. there's some driver function that amounts to:

uint32_t device_get_hot_addresses(uint64_t *addresses, uint32_t buf_max);

Where the return value is number of addresses the device returned, and
the buf_max is the number of addresses that can be read.

Drives providing this functionality would then register this as a
callback when its memory becomes a member of some numa node.


Re: source node -
Devices have no real way of determining upstream source information.

> >
> > (Just as one example, there are many more realistic designs)
> >
> > So if there is a way to expose workingset data to the mm/memory_tiers.c
> > component instead of via sysfs/cgroup - that is preferable.
> 
> Appreciate the feedback. The data in its current form might be useful
> to inform demotion decisions, but for promotion, are you aware of any
> recent developments? I would like to encode hotness as workingset data
> as well.

There were some recent patches to DAMON about promotion/demotion.  You
might look there.

~Gregory