Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 14 Oct 2022 09:56:23 -0700
From:   Mike Kravetz <mike.kravetz@oracle.com>
To:     Hugh Dickins <hughd@google.com>
Cc:     Albert Huang <huangjie.albert@bytedance.com>,
        Muchun Song <songmuchun@bytedance.com>,
        Andi Kleen <ak@linux.intel.com>,
        Andrew Morton <akpm@linux-foundation.org>, linux-mm@kvack.org,
        linux-kernel@vger.kernel.org
Subject: Re: [PATCH] mm: hugetlb: support get/set_policy for hugetlb_vm_ops
Message-ID: <Y0mUt84TctGP3BtT@monkey>
References: <20221012081526.73067-1-huangjie.albert@bytedance.com>
 <5f7ef6ee-6241-9912-f434-962be53272c@google.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <5f7ef6ee-6241-9912-f434-962be53272c@google.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?zKFnLpF+7q6HVvTj2JJhPc5n02xk+bRJ2+RNK7ADSA4AfGB3Z0Rxjoz3hxnn?=
 =?us-ascii?Q?m9m85Gd8EHUNs9im50yqbRM5c/wcVkQaFNr6hem/fen/X4kvRzsYzaQSNqiR?=
 =?us-ascii?Q?lXKwzlvLI13uqVXotqpK1kfKAad4I4ET7qc+YEWaCKhumh1Jzj9fZVrhCSZ8?=
 =?us-ascii?Q?6Q48woRW/pFndYdqN7h2TAXDIjNT3QaMoNbR/vK3xTz1/5Ouwclwsm0zuWeQ?=
 =?us-ascii?Q?jnp5yj1Mve1ylnlbBUQ8tm5MceloGfGjMRl7dEfmTaU9wFzWXd2M9Lp0toeI?=
 =?us-ascii?Q?WZOyBxjZN+lLyCr5zUnJO4jTNt+0l4LQ4sgcMjvS7hChgbreyslTSgkBfj9L?=
 =?us-ascii?Q?L5derk2OYY4zl3hiY6eLC6SJojrPpnqjVWs6MCnIEmvsHrHRStD8IrU+Z9dn?=
 =?us-ascii?Q?1NhilHWjFljRbioG+T1DvARCWNqs79NBuijUgYxgZAPI+mcEScgeI/WuYT3f?=
 =?us-ascii?Q?LtxhA+f+Oj+vxjn0y2j+0le0tajvLTslLgD8KdWDgXAd1DtYi7x5K8eA6M1E?=
 =?us-ascii?Q?UKiMRh5L1xrhxL6eWSPfnEAN2/h1nfJth7xqORssgCDSYriZaPd7HBB7Y7OF?=
 =?us-ascii?Q?2gPo3xlOovDimDRwihE0gLt1rRBuFsyYXSLhG9wLSn6xc8XkDpUM/yO4tLQ+?=
 =?us-ascii?Q?Hrz2CYr/Tt10W+kuULNlLxBr5NqoWDY10xVzWJoypxyfa+3oJ8oOeBQ4PMi5?=
 =?us-ascii?Q?lT+88L6AolAaoe3DcefnKt108ef0wV4T805OPfd2Z/c7Rfhv7Vp1JrJkNsMV?=
 =?us-ascii?Q?vJzC0jOvmYZ+RKfFzatT9aqoe8CNCvdr6kyCUgHv9G/Sg1MnaniAkRKEAJEd?=
 =?us-ascii?Q?r9ZTCa4bK2GrsGx2BwtANbeo396+CSErCpNj/T6uHgAtCVvUFf/lmmr+Rlag?=
 =?us-ascii?Q?V8BVmmutZtyOMWJksnm+KQZrnIuvItH7PYLJK1gZUZKzddyAGKzDJ2x30NWX?=
 =?us-ascii?Q?QHiHapBKoMzbMHJNK+QOmstnWIqo+cZuCUJ7zjieNAj2xdIElG0PhfaMCcRy?=
 =?us-ascii?Q?12MnAgU/Lq6gOxSUXe0bNz9UOE5G7jj2TtV5s+qLxjs4PB+P5QI8OQ2laoZj?=
 =?us-ascii?Q?4cMLHA7tVhFCTb06qlVflsMr9n8Hnl8ugySGQQJ4toIuRy8ng4QW4IQvHeWU?=
 =?us-ascii?Q?x0pM7UZkVEQCSh/fqTw6qW6GeCFPjReS3Wp7fOYapwBVIbLaQefhBQ4/4nFm?=
 =?us-ascii?Q?GoCMF7a31pH6tFYOyX/pVZmxRJCPRbVQe+W+T3hUJXAgKrzOkv814+k8IrQ1?=
 =?us-ascii?Q?ydby6Tu/A+sU+QXmpskVJnDfM+yPKo1BNAVNxl991Sa8zRnQMkZrD9bU5DBQ?=
 =?us-ascii?Q?nCYYFZj9hGKB1fePz+g++jSP7pF1aFZR98eQLAYaaiUd0J8r/PhQYzzeVeVS?=
 =?us-ascii?Q?I8k+GU6HXUuByNlV1mG8gIkD2a4PYU9SGFROUUnAVFNdbbBtb+HKUXtq8gip?=
 =?us-ascii?Q?gS0F4NVFMt9ISzNB6oh3piESq5N4Q1IJLM+ZoMWzqN8Lx6NVj7bBrdu4+rBM?=
 =?us-ascii?Q?OVG6bHkMbO4RGBhRiKDdlKW+mLXvfUjXR5zX/rFS0T3SS+Ha60mF1D41Q4lC?=
 =?us-ascii?Q?zoYcfpPWANaDRXnbaQypuRaIb0wPt4sC7tXFXz/gh3HliHDZnZR9Bgk3C4xw?=
 =?us-ascii?Q?5Q=3D=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 460872e3-2cfd-480b-b64b-08daae05083a
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Oct 2022 16:56:26.4512
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: guF7aQfVfSIm7b4mTcnHzCdwiu1U6k/uG3ZUeGjcZYrAslEas4c2uP4RIGdcoyk/W22azFUvqOtwxVzc/yUgyg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN0PR10MB5205
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1
 definitions=2022-10-14_09,2022-10-14_01,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=661 bulkscore=0
 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0 phishscore=0
 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2209130000 definitions=main-2210140094
X-Proofpoint-GUID: xAwdzgLGBWZZTHFYO5cBjUBOPg14pzm3
X-Proofpoint-ORIG-GUID: xAwdzgLGBWZZTHFYO5cBjUBOPg14pzm3
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 10/12/22 12:45, Hugh Dickins wrote:
> On Wed, 12 Oct 2022, Albert Huang wrote:
> 
> > From: "huangjie.albert" <huangjie.albert@bytedance.com>
> > 
> > implement these two functions so that we can set the mempolicy to
> > the inode of the hugetlb file. This ensures that the mempolicy of
> > all processes sharing this huge page file is consistent.
> > 
> > In some scenarios where huge pages are shared:
> > if we need to limit the memory usage of vm within node0, so I set qemu's
> > mempilciy bind to node0, but if there is a process (such as virtiofsd)
> > shared memory with the vm, in this case. If the page fault is triggered
> > by virtiofsd, the allocated memory may go to node1 which  depends on
> > virtiofsd.
> > 
> > Signed-off-by: huangjie.albert <huangjie.albert@bytedance.com>

Thanks for the patch Albert, and thank you Hugh for the comments!

> Aha!  Congratulations for noticing, after all this time.  hugetlbfs
> contains various little pieces of code that pretend to be supporting
> shared NUMA mempolicy, but in fact there was nothing connecting it up.

I actually had to look this up to verify it was not supported.  However, the
documentation is fairly clear.
From admin-guide/mm/numa_memory_policy.rst.

"As of 2.6.22, only shared memory segments, created by shmget() or
 mmap(MAP_ANONYMOUS|MAP_SHARED), support shared policy.  When shared
 policy support was added to Linux, the associated data structures were
 added to hugetlbfs shmem segments.  At the time, hugetlbfs did not
 support allocation at fault time--a.k.a lazy allocation--so hugetlbfs
 shmem segments were never "hooked up" to the shared policy support.
 Although hugetlbfs segments now support lazy allocation, their support
 for shared policy has not been completed."

It is somewhat embarrassing that this has been known for so long and
nothing has changed.

> It will be for Mike to decide, but personally I oppose adding
> shared NUMA mempolicy support to hugetlbfs, after eighteen years.
> 
> The thing is, it will change the behaviour of NUMA on hugetlbfs:
> in ways that would have been sensible way back then, yes; but surely
> those who have invested in NUMA and hugetlbfs have developed other
> ways of administering it successfully, without shared NUMA mempolicy.
> 
> At the least, I would expect some tests to break (I could easily be
> wrong), and there's a chance that some app or tool would break too.
> 
> I have carried the reverse of Albert's patch for a long time, stripping
> out the pretence of shared NUMA mempolicy support from hugetlbfs: I
> wanted that, so that I could work on modifying the tmpfs implementation,
> without having to worry about other users.
> 
> Mike, if you would prefer to see my patch stripping out the pretence,
> let us know: it has never been a priority to send in, but I can update
> it to 6.1-rc1 if you'd like to see it.  (Once upon a time, it removed
> all need for struct hugetlbfs_inode_info, but nowadays that's still
> required for the memfd seals.)
> 
> Whether Albert's patch is complete and correct, I haven't begun to think
> about: I am not saying it isn't, but shared NUMA mempolicy adds another
> dimension of complexity, and need for support, that I think hugetlbfs
> would be better off continuing to survive without.

To be honest, I have not looked into the complexities of shared NUMA
mempolicy and exactly what is required for it's support.  With my limited
knowledge, it appears that this patch adds some type of support for shared
policy, but it may not provide all support mentioned in the documentation.

At the very least, this patch should also update documentation to state
what type of support is provided.

Albert, can you look into what would be required for full support?  I can take
a look as well but have some other higher priority tasks to work first.

TBH, I like Hugh's idea of removing the 'pretence of shared policy support'.
We are currently wasting memory carrying around extra unused fields in
hugetlbfs_inode_info. :(
-- 
Mike Kravetz