Received-SPF: pass (google.com: domain of linux-kernel+bounces-49458-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Date: Fri, 2 Feb 2024 09:17:42 +0100
From: Michal Hocko <mhocko@suse.com>
To: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: akpm@linux-foundation.org, muchun.song@linux.dev, osalvador@suse.de,
	david@redhat.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org
Subject: Re: [RFC PATCH] mm: hugetlb: remove __GFP_THISNODE flag when
 dissolving the old hugetlb
Message-ID: <ZbylJr_bbWCUMjMl@tiehlicka>
References: <6f26ce22d2fcd523418a085f2c588fe0776d46e7.1706794035.git.baolin.wang@linux.alibaba.com>
 <Zbu4cD1XLFLfKan8@tiehlicka>
 <3f31cd89-f349-4f9e-bc29-35f29f489633@linux.alibaba.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <3f31cd89-f349-4f9e-bc29-35f29f489633@linux.alibaba.com>

On Fri 02-02-24 09:35:58, Baolin Wang wrote:
> 
> 
> On 2/1/2024 11:27 PM, Michal Hocko wrote:
> > On Thu 01-02-24 21:31:13, Baolin Wang wrote:
> > > Since commit 369fa227c219 ("mm: make alloc_contig_range handle free
> > > hugetlb pages"), the alloc_contig_range() can handle free hugetlb pages
> > > by allocating a new fresh hugepage, and replacing the old one in the
> > > free hugepage pool.
> > > 
> > > However, our customers can still see the failure of alloc_contig_range()
> > > when seeing a free hugetlb page. The reason is that, there are few memory
> > > on the old hugetlb page's node, and it can not allocate a fresh hugetlb
> > > page on the old hugetlb page's node in isolate_or_dissolve_huge_page() with
> > > setting __GFP_THISNODE flag. This makes sense to some degree.
> > > 
> > > Later, the commit ae37c7ff79f1 (" mm: make alloc_contig_range handle
> > > in-use hugetlb pages") handles the in-use hugetlb pages by isolating it
> > > and doing migration in __alloc_contig_migrate_range(), but it can allow
> > > fallbacking to other numa node when allocating a new hugetlb in
> > > alloc_migration_target().
> > > 
> > > This introduces inconsistency to handling free and in-use hugetlb.
> > > Considering the CMA allocation and memory hotplug relying on the
> > > alloc_contig_range() are important in some scenarios, as well as keeping
> > > the consistent hugetlb handling, we should remove the __GFP_THISNODE flag
> > > in isolate_or_dissolve_huge_page() to allow fallbacking to other numa node,
> > > which can solve the failure of alloc_contig_range() in our case.
> > 
> > I do agree that the inconsistency is not really good but I am not sure
> > dropping __GFP_THISNODE is the right way forward. Breaking pre-allocated
> > per-node pools might result in unexpected failures when node bound
> > workloads doesn't get what is asssumed available. Keep in mind that our
> > user APIs allow to pre-allocate per-node pools separately.
> 
> Yes, I agree, that is also what I concered. But sometimes users don't care
> about the distribution of per-node hugetlb, instead they are more concerned
> about the success of cma allocation or memory hotplug.

Yes, sometimes the exact per-node distribution is not really important.
But the kernel has no way of knowing that right now. And we have to make
a conservative guess here.
 
> > The in-use hugetlb is a very similar case. While having a temporarily
> > misplaced page doesn't really look terrible once that hugetlb page is
> > released back into the pool we are back to the case above. Either we
> > make sure that the node affinity is restored later on or it shouldn't be
> > migrated to a different node at all.
> 
> Agree. So how about below changing?
> (1) disallow fallbacking to other nodes when handing in-use hugetlb, which
> can ensure consistent behavior in handling hugetlb.

I can see two cases here. alloc_contig_range which is an internal kernel
user and then we have memory offlining. The former shouldn't break the
per-node hugetlb pool reservations, the latter might not have any other
choice (the whole node could get offline and that resembles breaking cpu
affininty if the cpu is gone).

Now I can see how a hugetlb page sitting inside a CMA region breaks CMA
users expectations but hugetlb migration already tries hard to allocate
a replacement hugetlb so the system must be under a heavy memory
pressure if that fails, right? Is it possible that the hugetlb
reservation is just overshooted here? Maybe the memory is just terribly
fragmented though?

Could you be more specific about numbers in your failure case?

> (2) introduce a new sysctl (may be named as "hugetlb_allow_fallback_nodes")
> for users to control to allow fallbacking, that can solve the CMA or memory
> hotplug failures that users are more concerned about.

I do not think this is a good idea. The policy might be different on
each node and this would get messy pretty quickly. If anything we could
try to detect a dedicated per node pool allocation instead. It is quite
likely that if admin preallocates pool without any memory policy then
the exact distribution of pages doesn't play a huge role.

-- 
Michal Hocko
SUSE Labs