Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31;
Date:   Mon, 4 Dec 2023 08:50:12 -0500
From:   Gregory Price <gregory.price@memverge.com>
To:     "Huang, Ying" <ying.huang@intel.com>
Cc:     Michal Hocko <mhocko@suse.com>, "tj@kernel.org" <tj@kernel.org>,
        John Groves <john@jagalactic.com>,
        Gregory Price <gourry.memverge@gmail.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-cxl@vger.kernel.org" <linux-cxl@vger.kernel.org>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "cgroups@vger.kernel.org" <cgroups@vger.kernel.org>,
        "linux-doc@vger.kernel.org" <linux-doc@vger.kernel.org>,
        "akpm@linux-foundation.org" <akpm@linux-foundation.org>,
        "lizefan.x@bytedance.com" <lizefan.x@bytedance.com>,
        "hannes@cmpxchg.org" <hannes@cmpxchg.org>,
        "corbet@lwn.net" <corbet@lwn.net>,
        "roman.gushchin@linux.dev" <roman.gushchin@linux.dev>,
        "shakeelb@google.com" <shakeelb@google.com>,
        "muchun.song@linux.dev" <muchun.song@linux.dev>,
        "jgroves@micron.com" <jgroves@micron.com>
Subject: Re: [RFC PATCH v4 0/3] memcg weighted interleave mempolicy control
Message-ID: <ZW3ZFDeTs7xotImL@memverge.com>
References: <ZU6uxSrj75EiXise@memverge.com>
 <ZU7vjsSkGbRLza-K@slm.duckdns.org>
 <ZU74L9oxWOoTTfpM@memverge.com>
 <ZVNBMW8iJIGDyp0y@tiehlicka>
 <ZVOXWx8XNJJNC23A@memverge.com>
 <ZVOn2T_Qg_NTKlB2@tiehlicka>
 <ZVOzMEtDYB4l8qFy@memverge.com>
 <87o7fveeze.fsf@yhuang6-desk2.ccr.corp.intel.com>
 <ZW1IdPI11nhKcdZl@memverge.com>
 <87sf4i2xe1.fsf@yhuang6-desk2.ccr.corp.intel.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <87sf4i2xe1.fsf@yhuang6-desk2.ccr.corp.intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?nJoxg69U4mROXt31wFMl2vOUgXax2YSHXBH+hYC1oX6HE7SbK21L3qLSgiQs?=
 =?us-ascii?Q?RJk5Bqdv5FX5BmxGBc8gy17Atn2O5vwUwzFqkRQF4PRYV4835ptOdZpb5Inz?=
 =?us-ascii?Q?etk/ReWN3arhenzqOC8yy6AqYtLggcCM6soXnpqclOHi2JNHbpczfN7ur8jl?=
 =?us-ascii?Q?bKfSgBGNQfGOIB4MF6wbZvmtxZlRasAz7t+dK5cv1drhcQpv6wFGi7v8MMat?=
 =?us-ascii?Q?QGUytGeYwp8FbV3B2SVwekWj5ETdkMWZHzKP5fn/QCQjXYIaClyN1dKbVc2Z?=
 =?us-ascii?Q?zmLlN3yEWSxHTtZBfxG8/RPBGVaEXc5w0TLQTJjW5LI+PwVgBso9CpUu28+O?=
 =?us-ascii?Q?4W/Bx/BTnBSVJPIhYZtig0sUphnoEf6ioFxR9yO+UoCP9n7oTMK5FygQU4bt?=
 =?us-ascii?Q?bStkxrMiyQHQnumYYRr4V/+iKTsn6qk9tVN6D1yP14maoDbCQosTek9Ijt7b?=
 =?us-ascii?Q?6KXMGV7HhVeD1r4y7NZN4XklrmKZ1u2K3ExUrU2GHEoqxC+OmY+uuXOeQ5PE?=
 =?us-ascii?Q?efwaDHvoNVfCQoU8gKyhXYSsyJcPGdVdF2+CF7QbjJlowrQQIrlGAf2V7HEI?=
 =?us-ascii?Q?5uREzhMWViJhR3IkicmHwZfJEswLR4xJBGLt7uonvHJFojzMmV+otvJATSva?=
 =?us-ascii?Q?yRmZR3rYDYonAhDuY+t1mrHzvU/1cT/FhQP5lL2hrOCLrNaFMu53YKOVLBqp?=
 =?us-ascii?Q?MS38MQSVODQT2xowOWjIqMRhNgzs0WClgTSnPsedr4sXaRcxiZjaNZy0nopL?=
 =?us-ascii?Q?Xz99k9mx5fYraq4hMwAWfBjmmijpoJJtKOXMOGSTeNLWdTA8mAVskw1lozvU?=
 =?us-ascii?Q?ZI0ZjGr5Fb9rK123G3HW5mNiIOT/dXaakd2945saJ+qmlLSGnnaflSC5m7RC?=
 =?us-ascii?Q?qIMi+oPOE+7CtV+O/sdYRv9Yk412APzGIKl0ILpbt1qTN1s5a+knHdDW5kWB?=
 =?us-ascii?Q?3bi6Xb0jSKoSn4YZ08sEyYSenkCzY05ecxqeLSZzCEoWli4QMYwzkmUyQPLc?=
 =?us-ascii?Q?SmxdHL8eMehq2C8BavyUZl+kizUA7yeJAyMAV3619PibkomnMHQM8IaH6d23?=
 =?us-ascii?Q?WY1+K5uNvg7/BfJtuPPNbdV/vSS3OSkTuwK4pN9D40EuzcE39UYFSqsWzgNf?=
 =?us-ascii?Q?8Z0nky5+VLsILgNlTsMNkKEDhS6a3oHqCJEggyzyehHhtgxC7taw5D8Qtdp6?=
 =?us-ascii?Q?KpCelIctXKaeVBqkPTi05bxrXC7Wbcfo7O+xQ2zUplYwJQJZJgJMySVqIop7?=
 =?us-ascii?Q?HcVd1hNi5wSYpqj8hBT5FAmDcpe2MhB62F8HsRtE3rHSoHpMz0N76mu4a91o?=
 =?us-ascii?Q?eV5A7eph1MzKOs3iVGA631UB6ssuG7Ffhssfg5BiiFjnLSx4nO7qQ953u0zp?=
 =?us-ascii?Q?o6f3OQi3Dv0F5etX9Oavv7Oo26Uxos4+qfN3zi9XlazfMkyTo2P3aVmqRRVG?=
 =?us-ascii?Q?8FzAL6SxeH+AMxLFgm2j5JQ3i2IrIzf7fG4WfPqOvOc8TuuoKqVQex5bh/tX?=
 =?us-ascii?Q?MEzuuqfHijvEL1KAjvM+76K9vg/kbaXbjX8e+MH3cYdVsBXJX9FnKkiHVjHK?=
 =?us-ascii?Q?uT/ydl0cegVh8fbPAyOOZEpIhqZOyHf0wT0qE2b/OTh2ozzafLLRJ1w/A/q7?=
 =?us-ascii?Q?Zg=3D=3D?=
X-OriginatorOrg: memverge.com
X-MS-Exchange-CrossTenant-Network-Message-Id: dcd3711f-09c8-457e-8c16-08dbf4cff243
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR17MB5512.namprd17.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Dec 2023 13:50:16.4965
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 5c90cb59-37e7-4c81-9c07-00473d5fb682
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: D6iaAjG/DPD12mnxMPuO4Hq0SUoxjRfeP5Fcxw1DIYQJbqUnpsNmi6thyWIVWHzSmZvE0CWuRkYmQ1plq7VaXskYNO6fj87EIMjZNWsnHIQ=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR17MB4482
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
	DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,
	SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 04 Dec 2023 05:50:33 -0800 (PST)

On Mon, Dec 04, 2023 at 04:19:02PM +0800, Huang, Ying wrote:
> Gregory Price <gregory.price@memverge.com> writes:
> 
> > If the structure is built as a matrix of (cpu_node,mem_nodes),
> > the you can also optimize based on the node the task is running on.
> 
> The matrix stuff makes the situation complex.  If people do need
> something like that, they can just use set_memorypolicy2() with user
> specified weights.  I still believe that "make simple stuff simple, and
> complex stuff possible".
> 

I don't think it's particularly complex, since we already have a
distance matrix for numa nodes:

available: 2 nodes (0-1)
... snip ...
node distances:
node   0   1
  0:  10  21
  1:  21  10

This would follow the same thing, just adjustable for bandwidth.

I personally find the (src,dst) matrix very important for flexibility.

But if there is particular pushback against it, having a one dimensional
array is better than not having it, so I will take what I can get.

> > That feels very intuitive, deals with many race condition issues, and
> > the global setting can actually be implemented without the need for
> > set_mempolicy2 at all - which is certainly a bonus.
> >
> > Would love more thoughts here.  Will have a new RFC with set_mempolicy2,
> > mbind2, and MPOL_WEIGHTED_INTERLEAVE soon that demonstrate the above.
> 
> Thanks for doing all these!
> 

Someone's got to :]

~Gregory