Received-SPF: pass (google.com: domain of linux-kernel+bounces-208486-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Mon, 10 Jun 2024 11:28:33 -0400
From: Johannes Weiner <hannes@cmpxchg.org>
To: Yu Zhao <yuzhao@google.com>
Cc: Andrew Morton <akpm@linux-foundation.org>,
	Vlastimil Babka <vbabka@suse.cz>,
	Mel Gorman <mgorman@techsingularity.net>, Zi Yan <ziy@nvidia.com>,
	"Huang, Ying" <ying.huang@intel.com>,
	David Hildenbrand <david@redhat.com>, linux-mm@kvack.org,
	linux-kernel@vger.kernel.org, Kalesh Singh <kaleshsingh@google.com>,
	Chun-Tse Shao <ctshao@google.com>
Subject: Re: [PATCH V4 00/10] mm: page_alloc: freelist migratetype hygiene
Message-ID: <20240610152833.GA2298848@cmpxchg.org>
References: <20240320180429.678181-1-hannes@cmpxchg.org>
 <CAOUHufbOP-RLmxmG_7Dp1vWAz8TTHvbmyfuR_mtceg0KgZoSnA@mail.gmail.com>
 <20240513160331.GA320190@cmpxchg.org>
 <CAOUHufaCQPW=p_r-Sg4oeDgtxwEGp6E5j1MhU3OCrTLUZTymZA@mail.gmail.com>
 <20240513190400.GA2270702@cmpxchg.org>
 <CAOUHufb8Daf276SdA2L=fq=LeAWN-JE08F5zW3buC1CBAoS=Ww@mail.gmail.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <CAOUHufb8Daf276SdA2L=fq=LeAWN-JE08F5zW3buC1CBAoS=Ww@mail.gmail.com>

On Tue, Jun 04, 2024 at 10:53:55PM -0600, Yu Zhao wrote:
> On Mon, May 13, 2024 at 1:04 PM Johannes Weiner <hannes@cmpxchg.org> wrote:
> > On Mon, May 13, 2024 at 12:10:04PM -0600, Yu Zhao wrote:
> > > On Mon, May 13, 2024 at 10:03 AM Johannes Weiner <hannes@cmpxchg.org> wrote:
> > > > On Fri, May 10, 2024 at 11:14:43PM -0600, Yu Zhao wrote:
> > > > > This series significantly regresses Android and ChromeOS under memory
> > > > > pressure. THPs are virtually nonexistent on client devices, and IIRC,
> > > > > it was mentioned in the early discussions that potential regressions
> > > > > for such a case are somewhat expected?
> > > >
> > > > This is not expected for the 10 patches here. You might be referring
> > > > to the discussion around the huge page allocator series, which had
> > > > fallback restrictions and many changes to reclaim and compaction.
> > >
> > > Right, now I remember.
> > >
> > > > > On Android (ARMv8.2), app launch time regressed by about 7%; On
> > > > > ChromeOS (Intel ADL), tab switch time regressed by about 8%. Also PSI
> > > > > (full and some) on both platforms increased by over 20%. I could post
> > > > > the details of the benchmarks and the metrics they measure, but I
> > > > > doubt they would mean much to you. I did ask our test teams to save
> > > > > extra kernel logs that might be more helpful, and I could forward them
> > > > > to you.
> > > >
> > > > If the issue persists with the latest patches in -mm, a kernel config
> > > > and snapshots of /proc/vmstat, /proc/pagetypeinfo, /proc/zoneinfo
> > > > before/during/after the problematic behavior would be very helpful.
> > >
> > > Assuming all the fixes were included, do you want the logs from 6.8?
> > > We have them available now.
> >
> > Yes, that would be helpful.
> >
> > If you have them, it would also be quite useful to have the vmstat
> > before-after-test delta from a good kernel, for baseline comparison.
> 
> Sorry for taking this long -- I wanted to see if the regression is
> still reproducible on v6.9.
> 
> Apparently we got the similar results on v6.9 with the following
> patches cherry-picked cleanly from v6.10-rc1:
> 
>      1  mm: page_alloc: remove pcppage migratetype caching
>      2  mm: page_alloc: optimize free_unref_folios()
>      3  mm: page_alloc: fix up block types when merging compatible blocks
>      4  mm: page_alloc: move free pages when converting block during isolation
>      5  mm: page_alloc: fix move_freepages_block() range error
>      6  mm: page_alloc: fix freelist movement during block conversion
>      7  mm: page_alloc: close migratetype race between freeing and stealing
>      8  mm: page_alloc: set migratetype inside move_freepages()
>      9  mm: page_isolation: prepare for hygienic freelists
>     10  mm: page_alloc: consolidate free page accounting
>     11  mm: page_alloc: change move_freepages() to __move_freepages_block()
>     12  mm: page_alloc: batch vmstat updates in expand()
> 
> Unfortunately I just realized that that automated benchmark didn't
> collect the kernel stats before it starts (since it always starts on a
> freshly booted device). While this is being fixed, I'm attaching the
> kernel stats collected after the benchmark finished. I grabbed 10 runs
> for each (baseline/patched), and if you need more, please let me know.
> (And we should have the stats before the benchmark soon.)

Thanks for grabbing these, and sorry about the delay, I was traveling
last week.

You mentioned "THPs are virtually non-existant". But the workload
doesn't seem to allocate anon THPs at all. For file THP, the patched
kernel's median for allocation success is 90% of baseline, but the
inter-run min/max deviation from the median in baseline is 85%/108%
and in patched and 85%/112% in patched, so this is quite noisy. Was
that initial comment regarding a different workload?

This other data point has me stumped. Comparing medians, there is a
1.5% reduction in anon refaults and a 4.8% increase in file
refaults. And indeed, there is more files and less anon being scanned.
I think this could explain the PSI delta, since AFAIK you have zram or
zswap, and anon decompression loads are cheaper than filesystem IO.

The above patches don't do anything that directly influences the
anon-file reclaim balance. However, if file THPs fall back to 4k file
pages more, that *might* be able to explain a shift in reclaim
balance, if some hot subpages in those THPs were protecting colder
subpages from being reclaimed and refaulting.

In that case, the root cause would still be a simple THP success rate
regression. To confirm this theory, could you run the baseline and the
patched sets both with THP disabled entirely?

Can you elaborate more on what the workload is doing exactly? What are
the parameters of the test machine (CPUs, memory size)? It'd be
helpful if I could reproduce this locally as well.