Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.34 as permitted sender) client-ip=23.128.96.34;
Date:   Mon, 23 Oct 2023 16:52:29 +0900
From:   Hyesoo Yu <hyesoo.yu@samsung.com>
To:     Alexandru Elisei <alexandru.elisei@arm.com>
Cc:     catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev,
        maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com,
        yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org,
        mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com,
        vincent.guittot@linaro.org, dietmar.eggemann@arm.com,
        rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de,
        bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org,
        rppt@kernel.org, hughd@google.com, pcc@google.com,
        steven.price@arm.com, anshuman.khandual@arm.com,
        vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com,
        kcc@google.com, linux-arm-kernel@lists.infradead.org,
        linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev,
        linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org,
        linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org
Subject: Re: [PATCH RFC 04/37] mm: Add MIGRATE_METADATA allocation policy
Message-ID: <20231023075229.GB344850@tiffany>
MIME-Version: 1.0
In-Reply-To: <ZS0vRz6PlUJM8MN9@monolith>
Content-Type: multipart/mixed;
        boundary="----QkVgGvrvWiyli5pqMJpqhgOu5Z7YYftLzn.eTqKOKf81mxr7=_53dc1_"
CMS-TYPE: 102P
DLP-Filter: Pass
References: <20230823131350.114942-1-alexandru.elisei@arm.com>
        <20230823131350.114942-5-alexandru.elisei@arm.com>
        <CGME20231012013834epcas2p28ff3162673294077caef3b0794b69e72@epcas2p2.samsung.com>
        <20231012012824.GA2426387@tiffany> <ZS0vRz6PlUJM8MN9@monolith>
Precedence: bulk

------QkVgGvrvWiyli5pqMJpqhgOu5Z7YYftLzn.eTqKOKf81mxr7=_53dc1_
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline

On Mon, Oct 16, 2023 at 01:40:39PM +0100, Alexandru Elisei wrote:
> Hello,
> 
> On Thu, Oct 12, 2023 at 10:28:24AM +0900, Hyesoo Yu wrote:
> > On Wed, Aug 23, 2023 at 02:13:17PM +0100, Alexandru Elisei wrote:
> > > Some architectures implement hardware memory coloring to catch incorrect
> > > usage of memory allocation. One such architecture is arm64, which calls its
> > > hardware implementation Memory Tagging Extension.
> > > 
> > > So far, the memory which stores the metadata has been configured by
> > > firmware and hidden from Linux. For arm64, it is impossible to to have the
> > > entire system RAM allocated with metadata because executable memory cannot
> > > be tagged. Furthermore, in practice, only a chunk of all the memory that
> > > can have tags is actually used as tagged. which leaves a portion of
> > > metadata memory unused. As such, it would be beneficial to use this memory,
> > > which so far has been unaccessible to Linux, to service allocation
> > > requests. To prepare for exposing this metadata memory a new migratetype is
> > > being added to the page allocator, called MIGRATE_METADATA.
> > > 
> > > One important aspect is that for arm64 the memory that stores metadata
> > > cannot have metadata associated with it, it can only be used to store
> > > metadata for other pages. This means that the page allocator will *not*
> > > allocate from this migratetype if at least one of the following is true:
> > > 
> > > - The allocation also needs metadata to be allocated.
> > > - The allocation isn't movable. A metadata page storing data must be
> > >   able to be migrated at any given time so it can be repurposed to store
> > >   metadata.
> > > 
> > > Both cases are specific to arm64's implementation of memory metadata.
> > > 
> > > For now, metadata storage pages management is disabled, and it will be
> > > enabled once the architecture-specific handling is added.
> > > 
> > > Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
> > > ---
> > > [..]
> > > @@ -2144,6 +2156,15 @@ __rmqueue(struct zone *zone, unsigned int order, int migratetype,
> > >  		if (alloc_flags & ALLOC_CMA)
> > >  			page = __rmqueue_cma_fallback(zone, order);
> > >  
> > > +		/*
> > > +		 * Allocate data pages from MIGRATE_METADATA only if the regular
> > > +		 * allocation path fails to increase the chance that the
> > > +		 * metadata page is available when the associated data page
> > > +		 * needs it.
> > > +		 */
> > > +		if (!page && (alloc_flags & ALLOC_FROM_METADATA))
> > > +			page = __rmqueue_metadata_fallback(zone, order);
> > > +
> > 
> > Hi!
> > 
> > I guess it would cause non-movable page starving issue as CMA.
> 
> I don't understand what you mean by "non-movable page starving issue as
> CMA". Would you care to elaborate?
> 

Before below patch, I frequently encountered situations where there was free CMA memory
available but the allocation of unmovable page failed. That patch has improved this
issue. ("mm,page_alloc,cma: conditionally prefer cma pageblocks for movable allocations)
https://lore.kernel.org/linux-mm/20200306150102.3e77354b@imladris.surriel.com/

I guess it would be beneficial to add a policy for effectively utilizing the metadata
area as well. I think migration is cheaper than app killing or swap in terms
of performance.

But, if the next iteration tries to use only cma, as discussed in recent mailing lists,
I think this concern would be fine.

Thanks,
Regards.

> > The metadata pages cannot be used for non-movable allocations.
> > Metadata pages are utilized poorly, non-movable allocations may end up
> > getting starved if all regular movable pages are allocated and the only
> > pages left are metadata. If the system has a lot of CMA pages, then
> > this problem would become more bad. I think it would be better to make
> > use of it in places where performance is not critical, including some
> > GFP_METADATA ?
> 
> GFP_METADATA pages must be used only for movable allocations. The kernel
> must be able to migrate GFP_METADATA pages (if they have been allocated)
> when they are reserved to serve as tag storage for a newly allocated tagged
> page.
> 
> If you are referring to the fact that GFP_METADATA pages are allocated only
> when there are no more free pages in the zone, then yes, I can understand
> that that might be an issue. However, it's worth keeping in mind that if a
> GFP_METADATA page is in use when it needs to be repurposed to serve as tag
> storage, its contents must be migrated first, and this is obviously slow.
> 
> To put it another way, the more eager the page allocator is to allocate
> from GFP_METADATA, the slower it will be to allocate tagged pages because
> reserving the corresponding tag storage will be slow due to migration.
> 
> Before making a decision, I think it would be very helpful to run
> performance tests with different allocation policies for GFP_METADATA. But I
> would say that it's a bit premature for that, and I think it would be best
> to wait until the series stabilizes.
> 
> And thank you for the feedback!
> 
> Alex
> 
> > 
> > Thanks,
> > Hyesoo Yu.
> 

------QkVgGvrvWiyli5pqMJpqhgOu5Z7YYftLzn.eTqKOKf81mxr7=_53dc1_
Content-Type: text/plain; charset="utf-8"


------QkVgGvrvWiyli5pqMJpqhgOu5Z7YYftLzn.eTqKOKf81mxr7=_53dc1_--