Received-SPF: pass (google.com: domain of linux-kernel+bounces-122045-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Date: Wed, 27 Mar 2024 17:44:20 -0400
From: Gregory Price <gregory.price@memverge.com>
To: Yuanchu Xie <yuanchu@google.com>
Cc: David Hildenbrand <david@redhat.com>,
	"Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>,
	Khalid Aziz <khalid.aziz@oracle.com>,
	Henry Huang <henry.hj@antgroup.com>, Yu Zhao <yuzhao@google.com>,
	Dan Williams <dan.j.williams@intel.com>,
	Huang Ying <ying.huang@intel.com>, Wei Xu <weixugc@google.com>,
	David Rientjes <rientjes@google.com>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	"Rafael J. Wysocki" <rafael@kernel.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	Johannes Weiner <hannes@cmpxchg.org>,
	Michal Hocko <mhocko@kernel.org>,
	Roman Gushchin <roman.gushchin@linux.dev>,
	Muchun Song <muchun.song@linux.dev>, Shuah Khan <shuah@kernel.org>,
	Yosry Ahmed <yosryahmed@google.com>,
	Matthew Wilcox <willy@infradead.org>,
	Sudarshan Rajagopalan <quic_sudaraja@quicinc.com>,
	Kairui Song <kasong@tencent.com>,
	"Michael S. Tsirkin" <mst@redhat.com>,
	Vasily Averin <vasily.averin@linux.dev>,
	Nhat Pham <nphamcs@gmail.com>, Miaohe Lin <linmiaohe@huawei.com>,
	Qi Zheng <zhengqi.arch@bytedance.com>,
	Abel Wu <wuyun.abel@bytedance.com>,
	"Vishal Moola (Oracle)" <vishal.moola@gmail.com>,
	Kefeng Wang <wangkefeng.wang@huawei.com>,
	linux-kernel@vger.kernel.org, linux-mm@kvack.org,
	cgroups@vger.kernel.org, linux-kselftest@vger.kernel.org
Subject: Re: [RFC PATCH v3 0/8] mm: workingset reporting
Message-ID: <ZgSTNCP5f+T5VtBI@memverge.com>
References: <20240327213108.2384666-1-yuanchu@google.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240327213108.2384666-1-yuanchu@google.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?opcnGY6zV3HhA/ubBvv0n2D52orFU1EjwDOa9aUyD7a/vxthwyYWXPqvT9P5?=
 =?us-ascii?Q?FCyluu7uSeRyWATN5Gl9tgqFg37MuRkjoym+pbxfC8r1Vriqgit5drQfS3Zt?=
 =?us-ascii?Q?6r2Z2GMPV9gTw9grJ8W6GvyvBPvcPFTYInPtmaRGgPHPkFjdUZzFPZK4Wdak?=
 =?us-ascii?Q?R8pGbXmmtIUiZCkFmaIDBl8avIT3ynFdpz2QXT/m9FuDMU6uVgH8LhkeQX7M?=
 =?us-ascii?Q?8ApfV+9n7WdrAXkHy6dYO47FTrwJ3YfwJNP+2bHQ08oT6X7ttqMPKo7lCRtJ?=
 =?us-ascii?Q?D2llU6+Zfye2Kgq4Nc+Rp4vdBOLgYKTDbpmsXQtAWQ9QbOt/qXDxZHhkzNOh?=
 =?us-ascii?Q?1PR4QckiS/bBNaDpqtFig65ALC8czwePJucmT6be/oizzUa1DwgIrUZbs/U6?=
 =?us-ascii?Q?q/R66ULZZEqYUPMQr8Qs3CUaMJ/68+DzdDeQrtASP7upGMDOPqHGhpCBvqlI?=
 =?us-ascii?Q?bAOEBHlrxBTL7mJh+b4RTRFwZIUPT1Xub4CcCxfUrE5oPW2zeVETOKafbvFm?=
 =?us-ascii?Q?RHsTFbfhsSfvunjB8aSaE763rHBiw6JMWgdhHFVUE4UgVMQ7q914zcYB23DH?=
 =?us-ascii?Q?+zJ7QCI9p1YKiy1tIz6hEs+vUF5Uai9Syz9Z6iQGQgtajozX1W9EjKQxeNqW?=
 =?us-ascii?Q?VnQR4/nfxH9TKOMuzVz00bqtsXYG9VV+Pkl+gg0y5n4b3ifhu+dMv8p1T/lR?=
 =?us-ascii?Q?Z3kLnUnvM9uywhmrm6CagHdjfyEfCh4wt2sNmSo3tZaKvY0Y10cPEmb3ime5?=
 =?us-ascii?Q?aC3KgdQX7BbE13x0JtfgMwfdg6TUkblFlXFl4HhxVoFR/uiL1iXT6F+M7bsS?=
 =?us-ascii?Q?YiHHQCCTmt1ni+qjYTE0CQBlJeJPdaggyKqR4WpSp4nFqniRUB2GEhhqSbNX?=
 =?us-ascii?Q?m74f7GmMhvumPjvZGxL/tz/A6bdpSItSU1ryYVOZ0ZpRa8gvsqS2JxuR2vyZ?=
 =?us-ascii?Q?OvJNsM0OKZBwtWvy8aXzOYQvOn3lhN+YMvQBSDkPMGgAI604WI/QhJz9ofkb?=
 =?us-ascii?Q?M6xtG0ik602f/PgNIOmZkjxS4tYy9IQvyqq16PVXNoujBE7eHMuGrxI7C1jg?=
 =?us-ascii?Q?fc81D/h5SHR+xP/05ha/7Pv59AHntCmEMCfYjLr4YC65dr6kJ/hIHELynXNh?=
 =?us-ascii?Q?ONH3+WC8TVkX1dDQrrKQxHTH/Z3aKdCxIZ1g689IXWdcp5mUUJuzjch9sbjh?=
 =?us-ascii?Q?mqKaoFoIC6KvZBP7oLd1mc1GX/1MvCI0Q1Q7xwel6K8yDtTzn0kRmBmnGINE?=
 =?us-ascii?Q?RQW+lX6gAghTuUavV8XLSuJlZoRMM8G51KS7yIknGn9Q6gXDoBVhDHvn+BHG?=
 =?us-ascii?Q?x/FtKPiooS9ByVhoa6YiQSCnG2NQk7jAFCFGQ9ObbRORBXYhkkNZfNHbXHyG?=
 =?us-ascii?Q?r/QKtu/3gn86CT0qoIHT9OM2Q8K3gm5oH1KzsCrHun32Q+toXZDUbHVHEnMA?=
 =?us-ascii?Q?2NRyTZVhyFKG7qvEkG8diCwmOt71KinBrrWOREO2ltx7vHTwFgJq5fvtSVb/?=
 =?us-ascii?Q?PJSylsK5+urA9+XI+ouH5ur9LO9wTS32gjX0wYH6m3ZEezXdzw0avWypeUr8?=
 =?us-ascii?Q?GOxQoYgu+NyjAXKlzVE/IByDwt0WpAGzqJvYz6F1VlKCh2dI8jHotp1PtbB7?=
 =?us-ascii?Q?jg=3D=3D?=
X-OriginatorOrg: memverge.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 66b0b045-0b2e-4a29-427a-08dc4ea714f6
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR17MB5512.namprd17.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2024 21:44:29.9421
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: HYpFakEk3+fZcHt45p0ZpBMAoZNcrg0eTijptV41JUG7T+qb2Vo3Wowi9OSA+9rfUiKb8PPCUdmWiL6GbYeDWHdXrh0+o3a/XrUojVPsRPk=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR17MB4415

On Wed, Mar 27, 2024 at 02:30:59PM -0700, Yuanchu Xie wrote:
> 
> Promotion/Demotion
> Similar to proactive reclaim, a workingset report enables demotion to a
> slower tier of memory.
> For promotion, the workingset report interfaces need to be extended to
> report hotness and gather hotness information from the devices[1].
> 
> [1]
> https://www.opencompute.org/documents/ocp-cms-hotness-tracking-requirements-white-paper-pdf-1
> 
> Sysfs and Cgroup Interfaces
> ==========
> The interfaces are detailed in the patches that introduce them. The main
> idea here is we break down the workingset per-node per-memcg into time
> intervals (ms), e.g.
> 
> 1000 anon=137368 file=24530
> 20000 anon=34342 file=0
> 30000 anon=353232 file=333608
> 40000 anon=407198 file=206052
> 9223372036854775807 anon=4925624 file=892892
> 
> I realize this does not generalize well to hotness information, but I
> lack the intuition for an abstraction that presents hotness in a useful
> way. Based on a recent proposal for move_phys_pages[2], it seems like
> userspace tiering software would like to move specific physical pages,
> instead of informing the kernel "move x number of hot pages to y
> device". Please advise.
> 
> [2]
> https://lore.kernel.org/lkml/20240319172609.332900-1-gregory.price@memverge.com/
> 

Please note that this proposed interface (move_phys_pages) is very
unlikely to be received upstream due to side channel concerns. Instead,
it's more likely that the tiering component will expose a "promote X
pages from tier A to tier B", and the kernel component would then
use/consume hotness information to determine which pages to promote.

(Just as one example, there are many more realistic designs)

So if there is a way to expose workingset data to the mm/memory_tiers.c
component instead of via sysfs/cgroup - that is preferable.

The 'move_phys_pages' interface is more of an experimental interface to
test the effectiveness of this approach without having to plumb out the
entire system.  Definitely anything userland interface should not be
designed to generate physical address information for consumption unless
it is hard-locked behind admin caps.

Regards,
Gregory