Received-SPF: pass (google.com: domain of linux-kernel+bounces-12083-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Date: Tue, 26 Dec 2023 01:48:05 -0500
From: Gregory Price <gregory.price@memverge.com>
To: "Huang, Ying" <ying.huang@intel.com>
Cc: Gregory Price <gourry.memverge@gmail.com>, linux-mm@kvack.org,
	linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org,
	linux-kernel@vger.kernel.org, linux-api@vger.kernel.org,
	x86@kernel.org, akpm@linux-foundation.org, arnd@arndb.de,
	tglx@linutronix.de, luto@kernel.org, mingo@redhat.com, bp@alien8.de,
	dave.hansen@linux.intel.com, hpa@zytor.com, mhocko@kernel.org,
	tj@kernel.org, corbet@lwn.net, rakie.kim@sk.com,
	hyeongtak.ji@sk.com, honggyu.kim@sk.com, vtavarespetr@micron.com,
	peterz@infradead.org, jgroves@micron.com, ravis.opensrc@micron.com,
	sthanneeru@micron.com, emirakhur@micron.com, Hasan.Maruf@amd.com,
	seungjun.ha@samsung.com
Subject: Re: [PATCH v5 01/11] mm/mempolicy: implement the sysfs-based
 weighted_interleave interface
Message-ID: <ZYp3JbcCPQc4fUrB@memverge.com>
References: <20231223181101.1954-1-gregory.price@memverge.com>
 <20231223181101.1954-2-gregory.price@memverge.com>
 <877cl0f8oo.fsf@yhuang6-desk2.ccr.corp.intel.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <877cl0f8oo.fsf@yhuang6-desk2.ccr.corp.intel.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?OYPdG4aDkf1FTRiD7Nl6AIz3MvE+/I7cc9K+BTOCSvCIODkMhHJ2c5Hkcels?=
 =?us-ascii?Q?SfSK1YHaEl3tM+s+poxzdMlrUrU7LO4tNV+kA5ou8ZB/p3FOqvMVVVWTO7LD?=
 =?us-ascii?Q?9FQTbGPb7x4X9rWFv7GBvBjQLfo12QKPS67UhkILvuCVN2I0W6ZqJqx9gSla?=
 =?us-ascii?Q?zdfvnjejtXiH/6Vl8wrbzSKjmsteaUSsECSNpdKrfetnX8oInfdfrnTGF4Y9?=
 =?us-ascii?Q?++LtF+mQutQXIQAclFodzQUFQRA/b3YHqKgZqxtmqN7Cx7Ll0HWVOqY67f3I?=
 =?us-ascii?Q?3mey55dDCMw1WfMb8fzrYfJD/ozsB+z+Ey/eYMtsGt4Cw4SAdStvZs5tRI50?=
 =?us-ascii?Q?525c4MvqrWf7o/9yrXS3Ibf7DSy+9C6TkSWLP2dghpuawPxXHPwvwfitNXME?=
 =?us-ascii?Q?1H0xvULMl5K1EYLTaikitFnR9nUvLUJLbJw5ynWsjcr5/zh5xDyr9eWsraQz?=
 =?us-ascii?Q?aEXgOUS7x2LVoAhpLMT2T5wkzy7jCykwFfBbLQFBx2RtGygTXlicaxFIUW+H?=
 =?us-ascii?Q?5/tVwSYKooB5HKy9MYQsQeueQ4qyICIlOaw7fnjXsPyNqBEjUS3ddPPFEkX0?=
 =?us-ascii?Q?f5EYPPMScZTg8N9hfuEIiVDcCev3G3Ng4ZARv9ZPq77mmNbQ0Xs6sz92JxVe?=
 =?us-ascii?Q?uHPtnc9NvJFVL8aXpww8BOdPjvCV2bsvp5QFiYt0qGCCaNLHo6sarBAu78X2?=
 =?us-ascii?Q?TcHfrp610TtR2cIhLgtviEpELEdBbtpC01YjcekB3aZPFIRyyoy8C3qHJz5E?=
 =?us-ascii?Q?CiNWNAUIXnZ7+xSNEyqMUMlEqctF9fBmwJw/HdTVf7qZvhdMawPIkcSFDK7T?=
 =?us-ascii?Q?5AVjRcVQzL6rqa/mfZvfxWRUrvm9SwDPaObXBRB/LOVM0lIHYNeHxAsSQfBx?=
 =?us-ascii?Q?Ww7pNPEl0UA3pBK9qhpTBFNPBT1n0dtLOkqF+4gZ0zIif2TUkk9XZmTqUTjc?=
 =?us-ascii?Q?bmJswCaDus2QtXVAkAgzVTKtMcNZOgrHHshmtbBPuUiv/CsDfcsiy0f1AjHr?=
 =?us-ascii?Q?TYK3HCw3wRrnRFUZzF3wUIAdet7EKgX4liICBSo2IsXF/otjraXPB0z85FRQ?=
 =?us-ascii?Q?GhIcObE4rnhJQEcnvSNe+lJ//4oIkjW7HWCCmz8JcDw7f3iO7PeBmoFT4IGI?=
 =?us-ascii?Q?r4CIftQCcesXN2TtufUNInOvlf+Fgv4SjkBKUA75AF1Crz3rkr5BMhSp9p6s?=
 =?us-ascii?Q?gXQM6CKPyoxtauc1L8Xqo3wcy/A0H87OQazO5pxvzsLbDXqElqysnINO/tZW?=
 =?us-ascii?Q?MXmtzSyvLI2EiAZM4D8BlK2HzSJ43+bAd1+7CNvKJ/vKJPf0usO/Z9gT/Tvd?=
 =?us-ascii?Q?S77kpsy5Mo/jU/ZX85N9nzf6m5/fUy6rmadygOyHCd2jrZH/qXZpgG1ABHUL?=
 =?us-ascii?Q?wwuH1IIFmoRHBD7XAhaKodMLI1pgehAhlLpZsxmcqc9NX8/QiZ/sj9EelX/q?=
 =?us-ascii?Q?6ZTrbFrqURBkQiPvJ1wxts0Rr/QCilUNEi+Hfw++xLSXXnfGpoMkNcg4Fau3?=
 =?us-ascii?Q?isvRTEEiW71Fge2GEDyKMhixuv/NPgbVXFYLGvf7llkJz4azvKutkTUhQqfI?=
 =?us-ascii?Q?c2d/yvR1lztgzCmpPfGAqbGO9aZpQeNA/j3pcGM95DSd/1MMceUQTVU8Pfze?=
 =?us-ascii?Q?hQ=3D=3D?=
X-OriginatorOrg: memverge.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 62deed50-ec3c-49fd-7f62-08dc06f09bdb
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR17MB5512.namprd17.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Dec 2023 15:29:25.7841
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 88gQ0xnxDEI7ZLr/uEzGef25HQ3bqlfqEsty13iitzl5SDwqISfXLX0dIRDa1AR2+NeVGTL2ZRszKTrC9hXdrT9sWN+iGFkIvPXRogTrU0Y=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR17MB6548

On Wed, Dec 27, 2023 at 02:42:15PM +0800, Huang, Ying wrote:
> Gregory Price <gourry.memverge@gmail.com> writes:
> 
> > +		These weights only affect new allocations, and changes at runtime
> > +		will not cause migrations on already allocated pages.
> > +
> > +		Writing an empty string resets the weight value to 1.
> 
> I still think that it's a good idea to provide some better default
> weight value with HMAT or CDAT if available.  So, better not to make "1"
> as part of ABI?
> 

That's the eventual goal, but this is just the initial mechanism.

My current thought is that the CXL driver will apply weights as the
system iterates through devices and creates numa nodes.  In the
meantime, you have to give the "possible" nodes a default value to
prevent nodes onlined after boot from showing up with 0-value.

Not allowing 0-value weights is simply easier in many respects.

> > +
> > +		Minimum weight: 1
> 
> Can weight be "0"?  Do we need a way to specify that a node don't want
> to participate weighted interleave?
> 

In this code, weight cannot be 0.  My thoguht is that removing the node
from the nodemask is the way to denote 0.

The problem with 0 is hotplug, migration, and cpusets.mems_allowed.  

Example issue:  Use set local weights to [1,0,1,0] for nodes [0-3],
and has a cpusets.mems_allowed mask of (0, 2).

Lets say the user migrates the task via cgroups from nodes (0,2) to
(1,3).

The task will instantly crash as basically OOM because weights of
[1,0,1,0] will prevent memory from being allocations.

Not allowing nodes weights of 0 is defensive.  Instead, simply removing
the node from the nodemask and/or mems_allowed is both equivalent to and
the preferred way to apply a weight of 0.

> > +		Maximum weight: 255
> > diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> > index 10a590ee1c89..0e77633b07a5 100644
> > --- a/mm/mempolicy.c
> > +++ b/mm/mempolicy.c
> > @@ -131,6 +131,8 @@ static struct mempolicy default_policy = {
> >  
> >  static struct mempolicy preferred_node_policy[MAX_NUMNODES];
> >  
> > +static char iw_table[MAX_NUMNODES];
> > +
> 
> It's kind of obscure whether "char" is "signed" or "unsigned".  Given
> the max weight is 255 above, it's better to use "u8"?
>

bah, stupid mistake.  I will switch this to u8.

> And, we may need a way to specify whether the weight has been overridden
> by the user.
> A special value (such as 255) can be used for that.  If
> so, the maximum weight should be 254 instead of 255.  As a user space
> interface, is it better to use 100 as the maximum value?
> 

There's global weights and local weights.  These are the global weights.

Local weights are stored in task->mempolicy.wil.il_weights.

(policy->mode_flags & MPOL_F_GWEIGHT) denotes the override.
This is set if (mempolicy_args->il_weights) was provided.

This simplifies the interface.

(note: local weights are not introduced until the last patch 11/11)

> > +
> > +static void sysfs_mempolicy_release(struct kobject *mempolicy_kobj)
> > +{
> > +	int i;
> > +
> > +	for (i = 0; i < MAX_NUMNODES; i++)
> > +		sysfs_wi_node_release(node_attrs[i], mempolicy_kobj);
> 
> IIUC, if this is called in error path (such as, in
> add_weighted_interleave_group()), some node_attrs[] element may be
> "NULL"?
> 

The null check is present in sysfs_wi_node_release

if (!node_attr)
	return;

Is it preferable to pull this out? Seemed more defensive to put it
inside the function.

~Gregory