Received-SPF: pass (google.com: domain of linux-kernel+bounces-108427-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Wed, 20 Mar 2024 00:39:34 -0400
From: Gregory Price <gregory.price@memverge.com>
To: "Huang, Ying" <ying.huang@intel.com>
Cc: Gregory Price <gourry.memverge@gmail.com>, linux-mm@kvack.org,
	linux-api@vger.kernel.org, linux-arch@vger.kernel.org,
	linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org,
	dan.j.williams@intel.com, honggyu.kim@sk.com, corbet@lwn.net,
	arnd@arndb.de, luto@kernel.org, akpm@linux-foundation.org,
	shuah@kernel.org
Subject: Re: [RFC v3 0/3] move_phys_pages syscall - migrate page contents
 given
Message-ID: <Zfpohg3EGxxOEcWg@memverge.com>
References: <20240319172609.332900-1-gregory.price@memverge.com>
 <87v85hsjn7.fsf@yhuang6-desk2.ccr.corp.intel.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87v85hsjn7.fsf@yhuang6-desk2.ccr.corp.intel.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?xyptoRBv+AO6Pa8tmCxyiw60L7kt6GkKNCzS/MbQFsGRvdWQjXKGUHMQYq2B?=
 =?us-ascii?Q?38+edWjd+2W40otCn80lcy2G2A761ab6PwD2BvEAXm6sBakGOPSgmW5VliAo?=
 =?us-ascii?Q?tUnUiqIh8kmjbMO8481Lgsk0abniMXzdkjRsWvYlXKcIGaLdi30HqTtvUJh6?=
 =?us-ascii?Q?gn8cv7Z/bMA7iklfmVCi6D3GYCAOBAMBR1Ssg5xF4+IEv6R0RisXgiiRTb1F?=
 =?us-ascii?Q?lXibAABWMJQNkOYySX80RNNHKYNOGaaz0qxtZXKv6PonYzC84yEdMVFZy1Ys?=
 =?us-ascii?Q?VzOV5Ks1VKP3tU4zKQz60J8IweHVN+OUZvhKSWvUWrVa/2mhF3qinF9WVMu7?=
 =?us-ascii?Q?7Mxotii8jgZNM6Uy/ZPtRi61DivW9nm3suP9BnqnNT03GggeNU5USRpxj26G?=
 =?us-ascii?Q?q3moUcVJckZML+gPwqlvoBCX4Hs9eBeiF16ouMI1yxBqdtiz3CHtv2Xj4ArZ?=
 =?us-ascii?Q?O4ijTbA66dWU4HpXZanfZ7XgjF4JTcG54j9dYnZFaWeBl9NVXIlnuYUBtT3k?=
 =?us-ascii?Q?u4PBLx2xsIFdQIRJPM1SlgCu1hrI+XOypL17id3Hclg7j0WDeWXcw/vN/rnz?=
 =?us-ascii?Q?jYijcJ0EHdCbbPHZLDeOiG9jUydNowkQEqBTrVDTW1uweezrx+/yEkpmaP+j?=
 =?us-ascii?Q?VSQpdjH42TtfSWL1JDGIZjSpCzQjtZW/ylnCOmXQoDuOulXaxIjQsqqQycbo?=
 =?us-ascii?Q?D1zoZuGnC3jYqedy2gNBx41uPe9lJgicxC52q3CnkSVXvdSAwfmuyhr/xEbT?=
 =?us-ascii?Q?dgeoWO4OHmbD1adnNqLBHA7E4l49PiHzjaoZTqiBWiXdDhsHiEn/Cj3PU0NW?=
 =?us-ascii?Q?1eL7aLj//ZK1HLZCgu6zKJ3igq0qqPK7KYLId4e1lDeFT+ccUZTiaYrkzEhg?=
 =?us-ascii?Q?ect/pdNk0J+iN/M+QFbOtX+X+5lhduhkj6ibIOk8rx7TtDjC73SaEtBSuaXi?=
 =?us-ascii?Q?6OtyC041jEViwKdtT5y1h5EEUj0xHl6EBb91jUK4mdYRygOa3autSUs8gBkj?=
 =?us-ascii?Q?q2ThzvPVR5UDuHbEteQClWbQ6yjJwuCb0aIXhlfatDj35Snkqo5LiWqyXSh4?=
 =?us-ascii?Q?/3mG/0rbH9qfZoSKy4TiIeqcl7NxbNk1TNK+Qs0o8dlVbCI79K3+HiUH8X70?=
 =?us-ascii?Q?JzKETs40W0f0RIV5SEN3++QFWhuF8QzwdlvZIn2u2nI2fhw7gUp+j5ZjKbLy?=
 =?us-ascii?Q?fZ7RpkvGZs1idq795QKrtpLGMxv2cU/ki82oaobgyWIYwsxbi35P9LH46rS6?=
 =?us-ascii?Q?G2U/SoXJhBzvDli/S2i4SqlQ5x6G69CjMLlZBW+wQVKujnAAGBOvt0AOVaci?=
 =?us-ascii?Q?NStI4J68oGlG+41luefZepzWn/evF7Uk3mEuKiMbJCxQOPVsOEJ4MLQalJ0l?=
 =?us-ascii?Q?hNNn4JCvhuhLI7obiindiZE4/J9XgLBUXNmHY6w+DAV4DUOKOIzkYyaQfdwf?=
 =?us-ascii?Q?zAL4jH7dF6PP0ujzOIW4cvEtWUGw8fSoh+fDiBQ0XF0VSFwTmFhqnilPpxoc?=
 =?us-ascii?Q?eJ2e3L8lgQvIa0nYDaLyGqZddlxWwN2+mtn9r4MmfVlFqkOdQzYM/++S4adn?=
 =?us-ascii?Q?ZPnVUZQxDuCeFTvavSp4FTO3xdhsulpf90hYTJqHFUJMz7ntZYCPHNVum8kC?=
 =?us-ascii?Q?Ug=3D=3D?=
X-OriginatorOrg: memverge.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c5be5050-a02e-4e1d-070c-08dc4897c306
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR17MB5512.namprd17.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Mar 2024 04:39:43.0732
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: m6D9eP/ToaZQ49E0B4SjCbS9x7nVltHt9Bv/5PW63YUltpQl1OCsg8GYcaMfnWpJc1WeTitYlT/QVoxp8+5XXkeVvfuJsoTbQvLmomtE/Sk=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW3PR17MB4201

On Wed, Mar 20, 2024 at 10:48:44AM +0800, Huang, Ying wrote:
> Gregory Price <gourry.memverge@gmail.com> writes:
> 
> > Doing this reverse-translation outside of the kernel requires considerable
> > space and compute, and it will have to be performed again by the existing
> > system calls.  Much of this work can be avoided if the pages can be
> > migrated directly with physical memory addressing.
> 
> One difficulty of the idea of the physical address is that we lacks some
> user space specified policy information to make decision.  For example,
> users may want to pin some pages in DRAM to improve latency, or pin some
> pages in CXL memory to do some best effort work.  To make the correct
> decision, we need PID and virtual address.
> 

I think of this as a second or third order problem.  The core problem
right now isn't the practicality of how userland would actually use this
interface - the core problem is whether the data generated by offloaded
monitoring is even worth collecting and operating on in the first place.  

So this is a quick hack to do some research about whether it's even
worth developing the whole abstraction described by Willy.

This is why it's labeled RFC.  I upped a v3 because I know of two groups
actively looking at using it for research, and because the folio updates
broke the old version.  It's also easier for me to engage through the
list than via private channels for this particular work.


Do I suggest we merge this interface as-is? No, too many concerns about
side channels.  However, it's a clean reuse of move_pages code to
bootstrap the investigation, and it at least gets the gears turning.

Example notes from a sidebar earlier today:

* An interesting proposal from Dan Williams would be to provide some
  sort of `/sys/.../memory_tiering/tierN/promote_hot` interface, with
  a callback mechanism into the relevant hardware drivers that allows
  for this to be abstracted.  This could be done on some interval and
  some threshhold (# pages, hotness threshhold, etc).


The code to execute promotions ends up looking like what I have now

1) Validate the page is elgibile to be promoted by walking the vmas
2) invoking the existing move_pages code

The above idea can be implemented trivially in userland without
having to plumb through a whole brand new callback system.


Sometimes you have to post stupid ideas to get to the good ones :]

~Gregory