Received-SPF: pass (google.com: domain of linux-kernel+bounces-95215-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Date: Thu, 7 Mar 2024 10:23:08 +0100
From: Jan Kara <jack@suse.cz>
To: "Yin, Fengwei" <fengwei.yin@intel.com>
Cc: Yujie Liu <yujie.liu@intel.com>, Jan Kara <jack@suse.cz>,
	Oliver Sang <oliver.sang@intel.com>, oe-lkp@lists.linux.dev,
	lkp@intel.com, linux-kernel@vger.kernel.org,
	Andrew Morton <akpm@linux-foundation.org>,
	Matthew Wilcox <willy@infradead.org>,
	Guo Xuenan <guoxuenan@huawei.com>, linux-fsdevel@vger.kernel.org,
	ying.huang@intel.com, feng.tang@intel.com
Subject: Re: [linus:master] [readahead] ab4443fe3c: vm-scalability.throughput
 -21.4% regression
Message-ID: <20240307092308.u54fjngivmx23ty3@quack3>
References: <202402201642.c8d6bbc3-oliver.sang@intel.com>
 <20240221111425.ozdozcbl3konmkov@quack3>
 <ZdakRFhEouIF5o6D@xsang-OptiPlex-9020>
 <20240222115032.u5h2phfxpn77lu5a@quack3>
 <20240222183756.td7avnk2srg4tydu@quack3>
 <ZeVVN75kh9Ey4M4G@yujie-X299>
 <dee823ca-7100-4289-8670-95047463c09d@intel.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <dee823ca-7100-4289-8670-95047463c09d@intel.com>

On Mon 04-03-24 13:35:10, Yin, Fengwei wrote:
> Hi Jan,
> 
> On 3/4/2024 12:59 PM, Yujie Liu wrote:
> >  From the perf profile, we can see that the contention of folio lru lock
> > becomes more intense. We also did a simple one-file "dd" test. Looks
> > like it is more likely that low-order folios are allocated after commit
> > ab4443fe3c (Fengwei will help provide the data soon). Therefore, the
> > average folio size decreases while the total folio amount increases,
> > which leads to touching lru lock more often.
> 
> I did following testing:
>   With a xfs image in tmpfs + mount it to /mnt and create 12G test file
>   (sparse-file), use one process to read it on a Ice Lake machine with
>   256G system memory. So we could make sure we are doing a sequential
>   file read with no page reclaim triggered.
> 
>   At the same time, profiling the distribution of order parameter of
>   filemap_alloc_folio() call to understand how the large folio order
>   for page cache is generated.
> 
> Here is what we got:
> 
> - Commit f0b7a0d1d46625db:
> $ dd bs=4k if=/mnt/sparse-file of=/dev/null
> 3145728+0 records in
> 3145728+0 records out
> 12884901888 bytes (13 GB, 12 GiB) copied, 2.52208 s, 5.01 GB/s
> 
> filemap_alloc_folio
>      page order    : count     distribution
>         0          : 57       |                                        |
>         1          : 0        |                                        |
>         2          : 20       |                                        |
>         3          : 2        |                                        |
>         4          : 4        |                                        |
>         5          : 98300    |****************************************|
> 
> - Commit ab4443fe3ca6:
> $ dd bs=4k if=/mnt/sparse-file of=/dev/null
> 3145728+0 records in
> 3145728+0 records out
> 12884901888 bytes (13 GB, 12 GiB) copied, 2.51469 s, 5.1 GB/s
> 
> filemap_alloc_folio
>      page order    : count     distribution
>         0          : 21       |                                        |
>         1          : 0        |                                        |
>         2          : 196615   |****************************************|
>         3          : 98303    |*******************                     |
>         4          : 98303    |*******************                     |
> 
> 
> Even the file read throughput is almost same. But the distribution of
> order looks like a regression with ab4443fe3ca6 (more smaller order
> page cache is generated than parent commit). Thanks.

Thanks for testing! This is an interesting result and certainly unexpected
for me. The readahead code allocates naturally aligned pages so based on
the distribution of allocations it seems that before commit ab4443fe3ca6
readahead window was at least 32 pages (128KB) aligned and so we allocated
order 5 pages. After the commit, the readahead window somehow ended up only
aligned to 20 modulo 32. To follow natural alignment and fill 128KB
readahead window we allocated order 2 page (got us to offset 24 modulo 32),
then order 3 page (got us to offset 0 modulo 32), order 4 page (larger
would not fit in 128KB readahead window now), and order 2 page to finish
filling the readahead window.

Now I'm not 100% sure why the readahead window alignment changed with
different rounding when placing readahead mark - probably that's some
artifact when readahead window is tiny in the beginning before we scale it
up (I'll verify by tracing whether everything ends up looking correctly
with the current code). So I don't expect this is a problem in ab4443fe3ca6
as such but it exposes the issue that readahead page insertion code should
perhaps strive to achieve better readahead window alignment with logical
file offset even at the cost of occasionally performing somewhat shorter
readahead. I'll look into this once I dig out of the huge heap of email
after vacation...

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR