Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2;
Date:   Sun, 3 Dec 2023 22:33:08 -0500
From:   Gregory Price <gregory.price@memverge.com>
To:     "Huang, Ying" <ying.huang@intel.com>
Cc:     Michal Hocko <mhocko@suse.com>, "tj@kernel.org" <tj@kernel.org>,
        John Groves <john@jagalactic.com>,
        Gregory Price <gourry.memverge@gmail.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-cxl@vger.kernel.org" <linux-cxl@vger.kernel.org>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "cgroups@vger.kernel.org" <cgroups@vger.kernel.org>,
        "linux-doc@vger.kernel.org" <linux-doc@vger.kernel.org>,
        "akpm@linux-foundation.org" <akpm@linux-foundation.org>,
        "lizefan.x@bytedance.com" <lizefan.x@bytedance.com>,
        "hannes@cmpxchg.org" <hannes@cmpxchg.org>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "roman.gushchin@linux.dev" <roman.gushchin@linux.dev>,
        "shakeelb@google.com" <shakeelb@google.com>,
        "muchun.song@linux.dev" <muchun.song@linux.dev>,
        "jgroves@micron.com" <jgroves@micron.com>
Subject: Re: [RFC PATCH v4 0/3] memcg weighted interleave mempolicy control
Message-ID: <ZW1IdPI11nhKcdZl@memverge.com>
References: <0100018bb64636ef-9daaf0c0-813c-4209-94e4-96ba6854f554-000000@email.amazonses.com>
 <ZU6pR46kiuzPricM@slm.duckdns.org>
 <ZU6uxSrj75EiXise@memverge.com>
 <ZU7vjsSkGbRLza-K@slm.duckdns.org>
 <ZU74L9oxWOoTTfpM@memverge.com>
 <ZVNBMW8iJIGDyp0y@tiehlicka>
 <ZVOXWx8XNJJNC23A@memverge.com>
 <ZVOn2T_Qg_NTKlB2@tiehlicka>
 <ZVOzMEtDYB4l8qFy@memverge.com>
 <87o7fveeze.fsf@yhuang6-desk2.ccr.corp.intel.com>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <87o7fveeze.fsf@yhuang6-desk2.ccr.corp.intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?V21aODRuYjZJZ2ZYay9PWUt3M2VOYm5XTHVMQ3R5SlY3clVzUXN2YklwS3hv?=
 =?utf-8?B?MTExNWsxSlh0WWdyUzc2WjhRT01nRHgvbUZ2azBJZE9tejlScE1yeThCK0Fx?=
 =?utf-8?B?S1JxNHlLUVpGT2VEN2VBMDdsU3lCc1VLbE5nMG5MbWNhdkZkS1R5S1JFVEZy?=
 =?utf-8?B?RWxUcGJYZ1JXV3h0Z3NiQXRPbjhvMnZnTGxuOXhFWHg1K1FNaUhtSTFxTG1z?=
 =?utf-8?B?NkZTZWtRZkdESWoxT2xIRmVMT1FVSWxXWms1Z0hpdmt6OWZWSlZJT1g1MWJs?=
 =?utf-8?B?UUhhU01YYmVSZlVCeVpDMjNrcGcxM29ONnMvQ2kySWc5b1RQQlhmRUV0S0Rr?=
 =?utf-8?B?WDdHaDg1N25Ebit4T1V4Z2pBTXhtbFN4cCtiWGdxbWs5Rk4ySmU5TkZodXNP?=
 =?utf-8?B?NVlwMzd1Q0VaU0U3eXFHWHl0OEhnQS9CT2RMbkU0U1dpNGxiVHpJSG9XLzBq?=
 =?utf-8?B?ZXR2dW4zRkZSTXA4UDVKTy9IN2xtdFBBSDJvRmh2ZlVzbHpiUVF4VEU5bVIv?=
 =?utf-8?B?OHUrZWZaajFvUytZVUtMVjlXOVR6RTlYWTBFZUtjbTNCTmxlOFR1YmVhYmVw?=
 =?utf-8?B?bmY0UEJaMUNJeU9GYnNjQmlJdU9wZ21NeTJlTDl3RHlMMlhPckUydng0N2Ri?=
 =?utf-8?B?VzQ5a3orTXRlaDJMK2NPdzV5TVppUUFvS0VRUGxhSnFDUlVGcjByMDZNekti?=
 =?utf-8?B?REdmdy9LbDRUSXA5UzliaS9DUG5lQVl5dy9TQkhyU0UvQUF6VUhyM2FSL1FU?=
 =?utf-8?B?azBVN2pSTTIzaHB5eFRnYmRkQlo2TDNvOTdnSDllQTRGdHdwZk5LNWliUWpP?=
 =?utf-8?B?eEhOaHRHOFovNm82cDdxQk96VG1zTHB1ZUE5WXdrd3prMnpqRU1hbXFubklJ?=
 =?utf-8?B?b2lXRmtGVE5acnA4NXlLejk4N1k5TWUwaHMxano4SEpMdkQ0SkNraEtRZTVJ?=
 =?utf-8?B?S09ackpjcjlCRCtEQWN2TDQrcWYyZVdwcTh1MUYxMGY4SjBNcnZqdmlLVVVr?=
 =?utf-8?B?QjlvcSs1dHBKOTZKUnRlUzJkMU9renNxYzdtRER4a1JiRkNZTnhUSDErcndW?=
 =?utf-8?B?WTVkUStMMVZrU3FHdWU1dkQ3SVNwNzA1NGVyNm1VOThxSWtDQXhUeWVTQ0JN?=
 =?utf-8?B?MHRZWk96dUhyWm0wbEFQZldaeXFjRGFXS2pCZlpLaUI3L3lJdVhFMUlJbUFq?=
 =?utf-8?B?ZlR5ek40Y3F6S1MwNFkwaTZ4OElHRUV0ZnJGNVJZWnQ5bFdRdmRjeG96S0du?=
 =?utf-8?B?RTRBdWFHQWhxS2wzNmhORE1ZeTNzNlhSdkEzWmtzUGVQdjlwNHd6eHNBTTJ2?=
 =?utf-8?B?TURhUEVjRjdIZEkweE5SZEJ1eFpieXFhQnpDNFErZWVrd3gzMWlxU1NISXZ4?=
 =?utf-8?B?elZUUUNpR2lsVVcxUzRiaWlCeTRkQkJFR1R2a2lYUkQ3QUpGdkxBaVFHS2pD?=
 =?utf-8?B?SGFMazZTWmtwejZ4NGpObkwxY1VEcndJelJXdTRHbW9KLzFwOEV6Z3RXTkxZ?=
 =?utf-8?B?dFpsRmxhSkZSOCt4OFgvZ2dwcU0zM2Z3UUFVVjA2eGU4SjRMdVQvb3JvQjlM?=
 =?utf-8?B?a2VtTG1MdE90TlhxWUEycFc5VkQzcHBOT3M5M3pBV1RHTGdrVDRUSlZhN2RT?=
 =?utf-8?B?bFRkQ0tucG5iUTU3WnJ3VHgvWXI4T2xqREQ4MTVBWXJJdlk1emN3Z0dNSTd5?=
 =?utf-8?B?ZHlobTkxSHBHdkM1ZktVQXJ1bGpYei9Wb3dCOEhGa3poa25wWHR3WTFpUWFh?=
 =?utf-8?B?TTZsL2JHQjFRN25NbUcwZXlldWdxaWIvc1FWNHFuUUZ2YVNMNmdKWDhiblE2?=
 =?utf-8?B?TEg3WkJQNHJZQXA3OUV0TERaRU1iNXgyZVNxNlFDZjhqdmNMV1Z3MStVNTFZ?=
 =?utf-8?B?b053N3RaVzFRNmJkK2VJQVF0ZjBiUUhNQ09vZktZRkNpalJrNGVndGZ1RlVX?=
 =?utf-8?B?MDI2YWE3MEo4c25XMStBeTNhWW5haEtHKzQ4THJrVFQ5a0dMSGdocUNMeFlD?=
 =?utf-8?B?V3laWDZiNjdpTC93ZXY1U1EzcFQ0clFhR0ZKOEtwTnJBbW9aclFmbldLQnZs?=
 =?utf-8?B?aDY2OElTdUtabHlVMHJjZ3J1Y2ZVUlZ3aEUrVDE0RnpPQUFrc3VVVkNWTWU0?=
 =?utf-8?B?aG1NOUZwZ0g4am9nU0FoU2pKN2pRNnFnRFU0Y2lxYzJ6azh1NDlDWW53Q1l2?=
 =?utf-8?B?Zmc9PQ==?=
X-OriginatorOrg: memverge.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b0bfb101-1553-4e2d-d043-08dbf479c0b9
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR17MB5512.namprd17.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Dec 2023 03:33:16.9398
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: TR34PvB/NPC+UsYzDcnknlZbnU6jTLyMMvBQ6gF6Bm0mnAhThrJmWKOUYfbZCZzqoPZIuG5vO99NMBvMe2pmbH5KX8uV+7FBWqOKPRFeoBQ=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR17MB6501
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Sun, 03 Dec 2023 19:45:15 -0800 (PST)

On Wed, Nov 15, 2023 at 01:56:53PM +0800, Huang, Ying wrote:
> Gregory Price <gregory.price@memverge.com> writes:
> 
> Because we usually have multiple nodes in one mem-tier, I still think
> mem-tier-based interface is simpler than node-based.  But, it seems more
> complex to introduce mem-tier into mempolicy.  Especially if we have
> per-task weights.  So, I am fine to go with node-based interface.
> 
> > * cgroups: "this doesn't involve dynamic resource accounting /
> >             enforcement at all" and "these aren't resource
> > 	    allocations, it's unclear what the hierarchical
> > 	    relationship mean".
> >
> > * node: too global, explore smaller scope first then expand.
> 
> Why is it too global?  I understand that it doesn't cover all possible
> use cases (although I don't know whether these use cases are practical
> or not).  But it can provide a reasonable default per-node weight based
> on available node performance information (such as, HMAT, CDAT, etc.).
> And, quite some workloads can just use it.  I think this is an useful
> feature.
>

Have been sharing notes with more folks.  Michal thinks a global set of
weights is unintuitive and not useful, and would prefer to see the
per-task weights first.

Though this may have been in response to adding it as an attribute of
nodes directly. 

Another proposal here suggested adding a new sysfs setting
https://github.com/skhynix/linux/commit/61d2fcc7a880185df186fa2544edcd2f8785952a

  $ tree /sys/kernel/mm/interleave_weight/
  /sys/kernel/mm/interleave_weight/
  ├── enabled [1]
  ├── possible [2]
  └── node
      ├── node0
      │   └── interleave_weight [3]
      └── node1
          └── interleave_weight [3]

(this could be changed to /sys/kernel/mm/mempolicy/...)

I think the internal representation of this can be simplified greatly,
over what the patch provides now, but maybe this solves the "it doesn't
belong in these other components" issue.

Answer: Simply leave it as a static global kobject in mempolicy, which
also deals with many of the issues regarding race conditions.

If a user provides weights, use those.  If they do not, use globals.

On a cpuset rebind event (container migration, mems_allowed changes),
manually set weights would have to remain, so in a bad case, the
weights would be very out of line with the real distribution of memory.

Example: if your nodemask is (0,1,2) and a migration changes it to
(3,4,5), then unfortunately your weights will likely revert to [1,1,1]

If set with global weights, they could automatically adjust.  It
would not be perfect, but it would be better than the potential worst
case above.  If that same migration occurs, the next allocation would
simply use whatever the target node weights are in the global config.

So if globally you have weights [3,2,1,1,2,3], and you move from
nodemask (0,1,2) to (3,4,5), your weights change from [3,2,1] to
[1,2,3].  If the structure is built as a matrix of (cpu_node,mem_nodes),
the you can also optimize based on the node the task is running on.

That feels very intuitive, deals with many race condition issues, and
the global setting can actually be implemented without the need for
set_mempolicy2 at all - which is certainly a bonus.

Would love more thoughts here.  Will have a new RFC with set_mempolicy2,
mbind2, and MPOL_WEIGHTED_INTERLEAVE soon that demonstrate the above.

Regards
~Gregory