Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4;
Date:   Tue, 19 Sep 2023 14:20:04 -0400
From:   Gregory Price <gregory.price@memverge.com>
To:     Andy Lutomirski <luto@kernel.org>
Cc:     Jonathan Corbet <corbet@lwn.net>,
        Gregory Price <gourry.memverge@gmail.com>,
        linux-mm@vger.kernel.org,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        linux-arch@vger.kernel.org, Linux API <linux-api@vger.kernel.org>,
        linux-cxl@vger.kernel.org, Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        "H. Peter Anvin" <hpa@zytor.com>, Arnd Bergmann <arnd@arndb.de>,
        Andrew Morton <akpm@linux-foundation.org>,
        the arch/x86 maintainers <x86@kernel.org>
Subject: Re: [RFC PATCH 3/3] mm/migrate: Create move_phys_pages syscall
Message-ID: <ZQnmVI0Q/Al5UKgQ@memverge.com>
References: <20230907075453.350554-1-gregory.price@memverge.com>
 <20230907075453.350554-4-gregory.price@memverge.com>
 <878r9dzrxj.fsf@meer.lwn.net>
 <ZP2tYY00/q9ElFQn@memverge.com>
 <42d97bb4-fa0c-4ecc-8a1b-337b40dca930@app.fastmail.com>
 <ZQnMzD26VI3C/ivf@memverge.com>
 <0a7e3ccc-db66-428e-8c09-66e67bfded51@app.fastmail.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <0a7e3ccc-db66-428e-8c09-66e67bfded51@app.fastmail.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?HqRGDCcN1EkY2MjkfgAuXrBFlHMDAI6jSFJkqWbx8JSMXXDlPkmAQYB/F1Gs?=
 =?us-ascii?Q?1CGMZb8/jlP8K7lvikA/8Yypl/7gFQSZq1YTC1C60I/kxhG9IedgeBgU43xP?=
 =?us-ascii?Q?tf80JDL0P6UHpMCqDlrdDajWmea9zaTJDu2O5U0Gekl73XG9zmrjwXPt6H5B?=
 =?us-ascii?Q?GaXoWXr3tKM+YSvO+Sz/4dquwR4Qies0j++9wJLWRT7rBCl5KHw/0P4t15x+?=
 =?us-ascii?Q?2RfEcK3ORaUvsH82/UPdmToCXNkk+GPmg62TT+BJT3Q01t9hFvD6S3bDIfn5?=
 =?us-ascii?Q?VREadmt394yLBm2ySOCUOFBQ0LWjWTGn/NzoktqUozFj2MrkxiX1VVeTXtqb?=
 =?us-ascii?Q?cbvZf9DT4WMaDAuYKYOQEU/q0s7zUXEh05bdP4dMzRrIXaBiuOxP6poD7vb3?=
 =?us-ascii?Q?xZFdWlpWnRcJbwKu7Hp5kXFcSILTu86HAxFcbZhimhaWFOK5c2AhuEjA2OgF?=
 =?us-ascii?Q?M4BuVr64Utskc0wXBVVlW8aue9LvkvKr7xWaa0pbuyPr1u8XXvpp+W71pXhw?=
 =?us-ascii?Q?odWrElv9mFIA6m+WhUDrIFBtra42k+M7QlKnla6mQwTGgBOvUURkJkIrpz0y?=
 =?us-ascii?Q?C/kcbB0B5KXGQs1NopkhugWaqdj9sK4fk3p2CDiQ8iE45zJg3YIFw3+GSFTg?=
 =?us-ascii?Q?bqVz5eJ1Q4pZpWVkMAydda0yEk91q8x4E7OU0QFwl2lGIc2l1Wsk05wI+x//?=
 =?us-ascii?Q?Gl+Tn1xByM/kjOissDGNU+r+DpFolcm2U3+Q91U+zHKt8EN9R5G9o80iHExD?=
 =?us-ascii?Q?6q18HObFuhbcnThjGFO5ScV+io02r+rVHJE5rzFHjhBqxSj4RpVR9211S1FD?=
 =?us-ascii?Q?JlNbzqSusC3IprEQAgLJrsdrXm9AfuGt9rD74ij+cia+enVIN5SOZxT0yKaU?=
 =?us-ascii?Q?3s0Ellk3CSboKqrpj1bY8gbVArRml5fFjBMr7OuBcmWyJZJ/uwtxQt0RTLLV?=
 =?us-ascii?Q?xexj+PPq/0rzNrcnL5o5VslBiF55qQ+EFykioKa8Vr00nS6FB96jndFLB5oA?=
 =?us-ascii?Q?VI5VbwSswdYtfcx5dzEjzMy4pPhHG60vj7TFJW4kj93yFdYxWMYiEJOuWqwT?=
 =?us-ascii?Q?nxmzUY8xQoDJ6duq1ffJtv2h/3fg5z5GFIv6inb/fTF9L9L03X72DTSeUfR0?=
 =?us-ascii?Q?Dgl5khJndTwhF9lXpHQw9zzbY7UkrWqCb2YOMZQpZvYPcAeEOESDWI7xQhu+?=
 =?us-ascii?Q?jo2S+EW99VfW9KdvgeVvRcViXFLMwlxl8UichsVVnT0PTuAa1/RbKBP6mTks?=
 =?us-ascii?Q?Da17B0NcUJdffr35ewzsotcfnCXdiUxbwJInIB/a2KJxwnWoPsfed5uug45O?=
 =?us-ascii?Q?XY2SwVt63FsUPy8dg18cpRUUpIUv3v1je9ubeAmUArhc79Fx6ZWa4pnFNKAq?=
 =?us-ascii?Q?KQD9f7qNJdXC0CyA0BU4Hw2o8vPlwiDozLo+RI4YJ87i2LCHFxe7iPTDyXHQ?=
 =?us-ascii?Q?oO3T07ianeqaZva4XYzrki5LbPQS3LRW/EeJk6iPBMKaxfynhnrrPV/iv0s+?=
 =?us-ascii?Q?IuIBvJQe36SRj07wiaPT+rCF4Jw3jHQPa6KHERItu1kdVZTxIttkt4j99gGa?=
 =?us-ascii?Q?5li+ODa92RD33cIe23nvnxOPsYZYFTrbI6qgQleKieVANL7vS5Q3AcfOiDA8?=
 =?us-ascii?Q?TA=3D=3D?=
X-OriginatorOrg: memverge.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b67734fc-ed2c-412c-87dc-08dbb93d107d
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR17MB5512.namprd17.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Sep 2023 18:20:12.5540
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 8gEbkMJWVW1UQrR3D9XATg7tewc+yjRDc8XM+o5TAPKHW3qPBBnZtSHZoEE4bWMN6HKzQWUKKDhADHPAi11M/sP8G7VJ2ZiwAoNeKtfihx0=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR17MB6403
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Tue, 19 Sep 2023 11:20:23 -0700 (PDT)

On Tue, Sep 19, 2023 at 10:59:33AM -0700, Andy Lutomirski wrote:
> 
> I'm not complaining about the name.  I'm objecting about the semantics.
> 
> Apparently you have a system to collect usage statistics of physical addresses, but you have no idea what those pages map do (without crawling /proc or /sys, anyway).  But that means you have no idea when the logical contents of those pages *changes*.  So you fundamentally have a nasty race: anything else that swaps or migrates those pages will mess up your statistics, and you'll start trying to migrate the wrong thing.

How does this change if I use virtual address based migration?

I could do sampling based on virtual address (page faults, IBS/PEBs,
whatever), and by the time I make a decision, the kernel could have
migrated the data or even my task from Node A to Node B.  The sample I
took is now stale, and I could make a poor migration decision.

If I do move_pages(pid, some_virt_addr, some_node) and it migrates the
page from NodeA to NodeB, then the device-side collection is likewise
no longer valid.  This problem doesn't change because I used virtual
address compared to physical address.

But if i have a 512GB memory device, and i can see a wide swath of that
512GB is hot, while a good chunk of my local DRAM is not - then I
probably don't care *what* gets migrated up to DRAM, i just care that a
vast majority of that hot data does.

The goal here isn't 100% precision, you will never get there. The goal
here is broad-scope performance enhancements of the overall system
while minimizing the cost to compute the migration actions to be taken.

I don't think the contents of the page are always relevant.  The entire
concept here is to enable migration without caring about what programs
are using the memory for - just so long as the memcg's and zoning is
respected.

~Gregory