Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35;
Date:   Thu, 2 Nov 2023 14:21:14 -0400
From:   Gregory Price <gregory.price@memverge.com>
To:     Michal Hocko <mhocko@suse.com>
Cc:     Johannes Weiner <hannes@cmpxchg.org>,
        Gregory Price <gourry.memverge@gmail.com>,
        linux-kernel@vger.kernel.org, linux-cxl@vger.kernel.org,
        linux-mm@kvack.org, ying.huang@intel.com,
        akpm@linux-foundation.org, aneesh.kumar@linux.ibm.com,
        weixugc@google.com, apopple@nvidia.com, tim.c.chen@intel.com,
        dave.hansen@intel.com, shy828301@gmail.com,
        gregkh@linuxfoundation.org, rafael@kernel.org
Subject: Re: [RFC PATCH v3 0/4] Node Weights and Weighted Interleave
Message-ID: <ZUPomvqNwZgDbo51@memverge.com>
References: <20231031003810.4532-1-gregory.price@memverge.com>
 <rm43wgtlvwowjolzcf6gj4un4qac4myngxqnd2jwt5yqxree62@t66scnrruttc>
 <20231031152142.GA3029315@cmpxchg.org>
 <jgh5b5bm73qe7m3qmnsjo3drazgfaix3ycqmom5u6tfp6hcerj@ij4vftrutvrt>
 <ZUCCGJgrqqk87aGN@memverge.com>
 <pmxrljwp4ayl3fcu7rxm6prbumgb5l3lwb75lqfipmxxxwnqfo@nb5qjcxw22gp>
 <ZUKDz5NpMsoyzWtZ@memverge.com>
 <a4f5das6ckw5lwj3qv2eaygx4nypb762b6mdnxivrjjndqlhjk@zpjao2ewfdoc>
 <ZUMVI4YG7mB54u0D@memverge.com>
 <s72oio7nmez565i7h6fb4bdnhqkcablr34rz5gqteyrrf7yeux@lqrztvy35si5>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <s72oio7nmez565i7h6fb4bdnhqkcablr34rz5gqteyrrf7yeux@lqrztvy35si5>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?rdTJEG6NxJPNFCmp5LXuYv8+8vvnagMCxdn1bYv13a6+CyPHCb9HlVqt10+J?=
 =?us-ascii?Q?IuNhHLRlQzHOKm3fujThqtqEs5KMicJro4TbKswQ72EIkqyojaCu2MX+YYet?=
 =?us-ascii?Q?TbmatQryfsrT2171Xmyf7fASDnwASK6f0XlVS5gMxWNTdcHNy6xp4eDgR6wt?=
 =?us-ascii?Q?E/wn832nGXv2aGYK54Y7ro+tuTbinXz61o5TkwWLYSNlF3DVPeNaGp5RtTHn?=
 =?us-ascii?Q?I8bjRaY/Kr3UA/gfc51hDtF6K9goDjdvXAyEJ3Hr66ZRKl1lmzygCMku22Bh?=
 =?us-ascii?Q?26+wHo4js2psvxtMYcEfw1zdGkQhAlUQKlSM25DNqS4PG5kXIaj3LTv2aakO?=
 =?us-ascii?Q?ksMHrFlqx8aHU3L+j/3l5gzJRhKtZ1l9FRKaB1+SW4qyxEypEbBwxY6i37g+?=
 =?us-ascii?Q?rs5yWEafGb1nLtXuPJEtWxq4QuykAphv+tnfXoakR/AahGbN0Kij9cJjP5lY?=
 =?us-ascii?Q?lAUGGDRSl1da3mgfJe/Wx+Q/5loODZ5SpLJL5P0zuiQHT/Mll84G6BwcrwpV?=
 =?us-ascii?Q?RCHtS30/87ImsGnrZl5ipoQISuDxd624Gf5DxhT3k7lWjQ6Qvm3CyvGgsvMs?=
 =?us-ascii?Q?2UV6F6m/YeAIobvvVdojky/VgzP+1ZKwXg/ZDb0sawgHEV6jSNHLnsrak31B?=
 =?us-ascii?Q?WPQUkr7qgRSwv84A7RQzYFUd0eNGcoonfWyb4YtDjd3/lgkf50kyNON73Kx0?=
 =?us-ascii?Q?VQ+jfjMsqpXE8RamVJMhi+qBfmWwPRZpxZWKDoQtYfE31MoAJsFb6rqctoJP?=
 =?us-ascii?Q?UKO+DXwCBOR8AxMhOJ/fgQSXyT1VmeIw0vH88/5d29X9yPDk0GEdsLIuIDIE?=
 =?us-ascii?Q?OI2QICEzoZHzWlctppYx0PLZ0DRoroGdVfAJPybESl31M5t7CPx5jPP7+Qwg?=
 =?us-ascii?Q?+fYATXgJn/7VKxIzVSXuYLtvhR2H9noiO0FndnTQO/ZBLvlZK+AC4M4zHzuA?=
 =?us-ascii?Q?DzAf0zusjKzH/d2gJL1rKr06uTIhaOtaYgGuq3KAQh7PUaCES9hWfL8VOxeB?=
 =?us-ascii?Q?bh9KBTZ+1kV5bO0oo3MF+/3nEMdUhMAzuseZrbllkcVmtHasJ3+2N9FcPmZa?=
 =?us-ascii?Q?2cGSIukJlI9wXTJSKD10m6+NU600tzwJ/hOw5kZ4HxuvK1tj/tO+MasTEU3A?=
 =?us-ascii?Q?wgtFmlbL3I4RARqSqPqcG/T0xaBv9CmotR+/aZwvVrmUpKfmCsSQq5+EFSKe?=
 =?us-ascii?Q?hdYnVfLgc9lWf6aw4+fGjXqdI1yynSNDRwDaWPBXE1j2bkdRQzRK2+9Rte0X?=
 =?us-ascii?Q?3foCvGoYO/BPN6LwMPNQmbm2meB8LXpHkjHE4AjV7B7BQQnuNKOsYwwp4Ly1?=
 =?us-ascii?Q?oDgM76mURsw9ElkbOWPecEZdT2W1V6b3EeFFsUm7mBYesLqdwCrGTQHTiAmj?=
 =?us-ascii?Q?QPx0zBY7/pcFIlKxilZ7DlGEeFIA+3skSoP0POOfpRLbUs9NEmLFAgbLo3iB?=
 =?us-ascii?Q?knhse9CaoXnTn9Ht57MbUV3OFgqIOFP25/bXs/Y7dB8MPSmA9kfXY0aD9imA?=
 =?us-ascii?Q?q2S2RkFsqYx5Ma22i7zkTEBpGXy6850aHo4b57odLDBtZEX8uvVTU29TMg2D?=
 =?us-ascii?Q?KqSuH3e6OxrpWxJWef5N6An6EwouCnpKoZBhxD6TADJAxz3MW1Th2hiR12x/?=
 =?us-ascii?Q?bw=3D=3D?=
X-OriginatorOrg: memverge.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e428d061-19bc-4b71-84be-08dbdc815735
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR17MB5512.namprd17.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Nov 2023 15:27:07.7353
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: vl7z4qLkLB0/ZREtajeq+a7QhFDsg+LVi2k1Duv1KDLR0X2lwliCJuAWq9+BZ8vZcdQjSG1i28kMFS/XpfNe1M5+tWCriw78BsO1uIIQK+k=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR17MB4319
X-Spam-Status: No, score=-0.1 required=5.0 tests=DATE_IN_PAST_12_24,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE
	autolearn=unavailable autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Fri, 03 Nov 2023 08:27:41 -0700 (PDT)

On Fri, Nov 03, 2023 at 10:56:01AM +0100, Michal Hocko wrote:
> On Wed 01-11-23 23:18:59, Gregory Price wrote:
> > On Thu, Nov 02, 2023 at 10:47:33AM +0100, Michal Hocko wrote:
> > > On Wed 01-11-23 12:58:55, Gregory Price wrote:
> > > > Basically consider: `numactl --interleave=all ...`
> > > > 
> > > > If `--weights=...`: when a node hotplug event occurs, there is no
> > > > recourse for adding a weight for the new node (it will default to 1).
> > > 
> > > Correct and this is what I was asking about in an earlier email. How
> > > much do we really need to consider this setup. Is this something nice to
> > > have or does the nature of the technology requires to be fully dynamic
> > > and expect new nodes coming up at any moment?
> > >  
> > 
> > Dynamic Capacity is expected to cause a numa node to change size (in
> > number of memory blocks) rather than cause numa nodes to come and go, so
> > maybe handling the full node hotplug is a bit of an overreach.
> > 
> > Good call, I'll stop considering this problem for now.
> > 
> > > > If the node is removed from the system, I believe (need to validate
> > > > this, but IIRC) the node will be removed from any registered cpusets.
> > > > As a result, that falls down to mempolicy, and the node is removed.
> > > 
> > > I do not think we do anything like that. Userspace might decide to
> > > change the numa mask when a node is offlined but I do not think we do
> > > anything like that automagically.
> > >
> > 
> > mpol_rebind_policy called by update_tasks_nodemask
> > https://elixir.bootlin.com/linux/latest/source/mm/mempolicy.c#L319
> > https://elixir.bootlin.com/linux/latest/source/kernel/cgroup/cpuset.c#L2016
> > 
> > falls down from cpuset_hotplug_workfn:
> > https://elixir.bootlin.com/linux/latest/source/kernel/cgroup/cpuset.c#L3771
> 
> Ohh, have missed that. Thanks for the reference. Quite honestly I am not
> sure this code is really a) necessary and b) ever exercised. For the
> former I would argue that offline node could be treated as completely
> depleted one. From the correctness POV it shouldn't make any difference
> and I am rather skeptical it would have performance improvements. 

Only thing I'm not sure of is what happens if mempolicy is allowed to
select a node that doesn't exist.  I could hack up a contrived test, but
i don't think the state is reachable at the moment.

More importantly, the rebind code is needed for task migration and for
allowing the cpusets to be change-able.  From the perspective of
mempolicy, a node being hotplugged and the nodemask being changed due to
cgroup cpuset changing looks very similar and comes with the same
question:  What do i do about weights when a change to the effective
nodemask is made.

This is why i'm falling toward "cgroups seem about right", because we
can make mempolicy ask cgroups for the weight, and also allow mempolicy
to carry its own explicit weight array - which allows for flexiblity.

I think this may end up generalizing to a cgroup-wide mempolicy interface
ala cgroup/mempolicy/[policy, nodemask, weights, ...]

but one thing as a time :]

> for the latter, full node offlines are really rare from experience. I
> would be interested about actual real life usecases which do that

yeah i'm just going to drop this from my requirement list and go OBO,
for areas where i see it may cause an issue (potential for 0-weights) i
will do something simple (initialize weights to 1), but otherwise I
think it's too much to expect from the kernel.

> > 
> > As a user I would assume it would operate much the same way as other
> > nested cgroups, which is inherit by default (with subsets) or an
> > explicit overwrite that can't exceed the higher level settings.
> 
> This would make it rather impractical because a default (everything set
> to 1) would be cast in stone. As mentioned above this this not an
> enforcement limit. So I _think_ that a simple hierarchical rule like
> 	cgroup_interleaving_mask(cgroup)
> 		interleaving_mask = (cgroup->interleaving_mask) ? : cgroup_interleaving_mask(parent_cgroup(cgroup))
> 
> So child cgroups could overwrite parent as they wish. If there is any
> enforcement (like a cpuset) that would filter useable nodes and the
> allocation policy would simply apply weights on those.
> 

Sorry yes, this is what I intended, I'm just bad at words.

~Gregory