Received-SPF: pass (google.com: domain of linux-kernel+bounces-90058-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <dee823ca-7100-4289-8670-95047463c09d@intel.com>
Date: Mon, 4 Mar 2024 13:35:10 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [linus:master] [readahead] ab4443fe3c: vm-scalability.throughput
 -21.4% regression
Content-Language: en-US
To: Yujie Liu <yujie.liu@intel.com>, Jan Kara <jack@suse.cz>
CC: Oliver Sang <oliver.sang@intel.com>, <oe-lkp@lists.linux.dev>,
	<lkp@intel.com>, <linux-kernel@vger.kernel.org>, Andrew Morton
	<akpm@linux-foundation.org>, Matthew Wilcox <willy@infradead.org>, Guo Xuenan
	<guoxuenan@huawei.com>, <linux-fsdevel@vger.kernel.org>,
	<ying.huang@intel.com>, <feng.tang@intel.com>
References: <202402201642.c8d6bbc3-oliver.sang@intel.com>
 <20240221111425.ozdozcbl3konmkov@quack3>
 <ZdakRFhEouIF5o6D@xsang-OptiPlex-9020>
 <20240222115032.u5h2phfxpn77lu5a@quack3>
 <20240222183756.td7avnk2srg4tydu@quack3> <ZeVVN75kh9Ey4M4G@yujie-X299>
From: "Yin, Fengwei" <fengwei.yin@intel.com>
In-Reply-To: <ZeVVN75kh9Ey4M4G@yujie-X299>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RTNvZyt1dEc1MnFBY1YyamJPUTU2dHEyS0VWYUVMWG0wNnc3aWpxZUt2V214?=
 =?utf-8?B?U0hKaEZLdVJJY1lQanFuUlN5YW13SXdaSjFHYmJUcjBxSFNQV2R1QjgwdDhl?=
 =?utf-8?B?UGlNKy9QR2lKWVlxUFdWTmlnRlNDUU5ya3ZiVE5nQWhYWFNWZEtLZmVsbGdv?=
 =?utf-8?B?UG1FbVAwL3E3WEh3NytrLy9DbFMyamwvRDFMYjNMeEI2Nm9ubExqV2Fsd0Zs?=
 =?utf-8?B?RStPcFNoaVVTaXcxaS83OCtKalZjOEpVMTV6QnVDdklLUEJjdmZaSDhXYWhp?=
 =?utf-8?B?Skpaams2TFhLVTNpWThNUzhEZDBOdVI1VWpNWG5sTG1VRlJiZUwwcWs3UUtL?=
 =?utf-8?B?VDNTajlpejlZYUcyVHJ5WmVUZ0FMcVByLzJ0YkI2TjBCUW9KSW1mcVJSOUI0?=
 =?utf-8?B?NFdQZXp4OXFxT1oxYkR6aklMa0F0ZHdOaW5iT2lBQld4VkYyU0E2YTQvSUVX?=
 =?utf-8?B?cnRNNWorOXliU1dzdXBuTUxTVTRKeDc0clVtMjlWd1pNWFk2SWNxQzAreFNv?=
 =?utf-8?B?MmJKOUVJekRsbzVjMW13VG83RGpvQS91dDUvUlJVcGloLzhSak54c1FrOTAr?=
 =?utf-8?B?b2VYYjNEZUY3L3dZQjZxK0RkYkhIYkNFUjkxMDAvUDZnN1I5cU9HZ1MzNjJl?=
 =?utf-8?B?WDBHVFBobTEvY0RnenNISE9jc3pRUkNON2dDQ1ptWHRmQ0l1NWwrZ1pVTDFa?=
 =?utf-8?B?Q2NDL0tNVW5mOGJJNUY5anN3RlcxNTNjTmV1UzZJdm56Umd0cm1TU2cybGR6?=
 =?utf-8?B?elNMN0I3YnFVMG5KaVhVMUtBeFJjeG0wbHBzT3d2QWRINm8xOTYyUXBHbnpJ?=
 =?utf-8?B?M1ozTlAvUTE5UEVWeitRVnVoR1djcks4RmpsM2NTNExzMXdsU2dPL0QyRmZ0?=
 =?utf-8?B?Rk54T3BBbzlHZU05NllQM0U5TEl1UVhJUU82RWxoMzJ0SmNVSGltb2V4Zisz?=
 =?utf-8?B?K3RaS2dRczhyRmJBV1Q2bmt2b3ZveW1KWTBKZjdJcDBBMThVSzZ2ZWpPNEdS?=
 =?utf-8?B?OXhuVmloUFNzQVYrd2NrOS9uQVlIZlVXWHExUERvd251WXh3RWRBeWpYV1Yz?=
 =?utf-8?B?Uldqa1FRUVA4bkJ3bFhzaVk4NHNmbFdTcjVrVGl3SEYwR0FkVFArZ0pYa3BE?=
 =?utf-8?B?ZURyZVFmMHFrdnlJdVI4b09tMnhGMmllUU5NckpESVpSZVlhbDBUOGo1V09L?=
 =?utf-8?B?MDBSWmZLT0llaksxZy9UaUtxQVAyUWppdDFUTDZzSEc0UDRsZ1EzbzNVT05t?=
 =?utf-8?B?VEpRWGQ3RlFnV25aeE5wdjVMYkYrblRFNHl1T3dxaWtQTUZrUkhLQko3eG9K?=
 =?utf-8?B?RmhaQXc2UjZUM3hNK1lTUEgrWGs4QXdUc0pQdjd1QTd5aks0em1RWStveE5I?=
 =?utf-8?B?cWtoVSsvNTd6QllPVk5Mc3FoQXhpVWZqMy8xbVR4MEp2eU5yZWZyWlo5eWpL?=
 =?utf-8?B?SUpLdCs2amgyT0NTV3ljVlhvcUN3WjhFZ3Y5TERkN0Z1b2xzZjFxaU9FdHds?=
 =?utf-8?B?RjlVbjhEQ1F0T2xHakZwQW93UmJmYm00YTlDbEZwOXkyekJwUTJ2NjFZU0la?=
 =?utf-8?B?K29XUG1kYi9vWjV1SW41MTJhVkl0R1BhMExOMk81S0wvS1h1SjExUlFpbnRQ?=
 =?utf-8?B?eGszMXMwNU5IZy9UWGZFSHVrOGtoMHMxdTMrQkQrVzQrbklvWW1xamx3T0J1?=
 =?utf-8?B?b2Y4UFFwTldHUXR5TGJrM0xVenBNZHhLMk1SMHRJNm5Hb0RMSUp6b1RER3JT?=
 =?utf-8?B?YWdraUV6M0xsb2JuMm5mYkgyeGE2VlA3UDBiNFpZaSs0dFB0STZWVnpob2kr?=
 =?utf-8?B?WXVUZVZPR2FSUFZRd0VWZ2hvUFhoR2tSSmJEUU9NeksxT3FHVXVHNXFsMWZV?=
 =?utf-8?B?VjJTeXVRZU85am5rb2ZQQ0huTGcxd0FYclpSWncvNHZnMVNvSERmNW16TGdq?=
 =?utf-8?B?dE1PUVFDMEFIbjEzOFN6MEhHNDROVUMxanA2c1NWSHdQWTVZa1RUa01tTlg1?=
 =?utf-8?B?ZTZvUXZpU2p6QTd5TkhKdkQ3YitNdTJqZStkc3lEc3FuMHRRTEQ3WTlkVm9y?=
 =?utf-8?B?TytaNjRvaE83QzZlSHJVblpMT1NrOFZaNmFqTzJoaitXalhJdk1IL2dzeWJZ?=
 =?utf-8?Q?Jk60Qw/iesSYNp+Q96upl00E8?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 1cb34912-7087-47f3-baa8-08dc3c0ce05e
X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB4820.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Mar 2024 05:35:18.4197
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: M90CKyU6siWdcZWcWOaZCoPAnDwL5YCHvweEvPxH723fwNAbUDyUYj1QQ9xfvT4twk5hfRudR8P84scb4u3X0Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB7524
X-OriginatorOrg: intel.com

Hi Jan,

On 3/4/2024 12:59 PM, Yujie Liu wrote:
>  From the perf profile, we can see that the contention of folio lru lock
> becomes more intense. We also did a simple one-file "dd" test. Looks
> like it is more likely that low-order folios are allocated after commit
> ab4443fe3c (Fengwei will help provide the data soon). Therefore, the
> average folio size decreases while the total folio amount increases,
> which leads to touching lru lock more often.

I did following testing:
   With a xfs image in tmpfs + mount it to /mnt and create 12G test file
   (sparse-file), use one process to read it on a Ice Lake machine with
   256G system memory. So we could make sure we are doing a sequential
   file read with no page reclaim triggered.

   At the same time, profiling the distribution of order parameter of
   filemap_alloc_folio() call to understand how the large folio order
   for page cache is generated.

Here is what we got:

- Commit f0b7a0d1d46625db:
$ dd bs=4k if=/mnt/sparse-file of=/dev/null
3145728+0 records in
3145728+0 records out
12884901888 bytes (13 GB, 12 GiB) copied, 2.52208 s, 5.01 GB/s

filemap_alloc_folio
      page order    : count     distribution
         0          : 57       |                                        |
         1          : 0        |                                        |
         2          : 20       |                                        |
         3          : 2        |                                        |
         4          : 4        |                                        |
         5          : 98300    |****************************************|

- Commit ab4443fe3ca6:
$ dd bs=4k if=/mnt/sparse-file of=/dev/null
3145728+0 records in
3145728+0 records out
12884901888 bytes (13 GB, 12 GiB) copied, 2.51469 s, 5.1 GB/s

filemap_alloc_folio
      page order    : count     distribution
         0          : 21       |                                        |
         1          : 0        |                                        |
         2          : 196615   |****************************************|
         3          : 98303    |*******************                     |
         4          : 98303    |*******************                     |


Even the file read throughput is almost same. But the distribution of
order looks like a regression with ab4443fe3ca6 (more smaller order
page cache is generated than parent commit). Thanks.


Regards
Yin, Fengwei