Received-SPF: pass (google.com: domain of linux-kernel+bounces-98139-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Message-ID: <561465df-1370-4519-abe3-3998bd78233f@intel.com>
Date: Sun, 10 Mar 2024 14:40:00 +0800
User-Agent: Mozilla Thunderbird
Subject: Re: [linus:master] [readahead] ab4443fe3c: vm-scalability.throughput
 -21.4% regression
To: Jan Kara <jack@suse.cz>
CC: Yujie Liu <yujie.liu@intel.com>, Oliver Sang <oliver.sang@intel.com>,
	<oe-lkp@lists.linux.dev>, <lkp@intel.com>, <linux-kernel@vger.kernel.org>,
	Andrew Morton <akpm@linux-foundation.org>, Matthew Wilcox
	<willy@infradead.org>, Guo Xuenan <guoxuenan@huawei.com>,
	<linux-fsdevel@vger.kernel.org>, <ying.huang@intel.com>,
	<feng.tang@intel.com>
References: <202402201642.c8d6bbc3-oliver.sang@intel.com>
 <20240221111425.ozdozcbl3konmkov@quack3>
 <ZdakRFhEouIF5o6D@xsang-OptiPlex-9020>
 <20240222115032.u5h2phfxpn77lu5a@quack3>
 <20240222183756.td7avnk2srg4tydu@quack3> <ZeVVN75kh9Ey4M4G@yujie-X299>
 <dee823ca-7100-4289-8670-95047463c09d@intel.com>
 <20240307092308.u54fjngivmx23ty3@quack3>
Content-Language: en-US
From: "Yin, Fengwei" <fengwei.yin@intel.com>
In-Reply-To: <20240307092308.u54fjngivmx23ty3@quack3>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?eE5uWUM2RUo5OUxWK1BSVllpcXJqSENJdjRXNUNCdHREdnVZSUxsc0ZuZmY4?=
 =?utf-8?B?dU5SNElKQTAxdE5lVys0WnQvQmZBS1FUNGJndlYxdHBTdE1Sa2JtNmwvM2kv?=
 =?utf-8?B?cXZVZjhCNWlVVjJFbEJ6N2RrY3NEeW1VZWZOK2pxdWcyMlVadHQxTE0vQ1o4?=
 =?utf-8?B?bFllRjdKV2Y3ZG5hZ0E2Tk5VNUMzREp0dk14N3hTYnI5RTFLdVMwOWVKckFF?=
 =?utf-8?B?ZkQzWkRhQk1MNjRxd3k4SUpVKzljMjlweXg5TDZnTkhpMjlialFVN0U5Zzg1?=
 =?utf-8?B?TU5mcW43TzQvbFNYcmkwbzh2OVhmWllwcHFqeFJ5VUlKejRaeld4dVY2a29Y?=
 =?utf-8?B?UEFsOFlDQ1gzVlJhc3F4aW0rNXY1Y1BXSjV0M1B6SE5zczBjMVpPZmZrcytV?=
 =?utf-8?B?YUFVMS9UYW96NkgwS1c5VndRKzRxZ2daNTdDSU9uV0FDNHMwK01kSWJYallk?=
 =?utf-8?B?YitBV0xPbFpTbGlVYkZQK0J2R1JabC85T3hzbzg2NlV4dXoyL2ttaVpRT09y?=
 =?utf-8?B?MjZReFAxWkVMT3lMZEx0ZjFSa21yRHRqQnFkekVKbTczakVTRncxa3M5cTZY?=
 =?utf-8?B?MlNOdFlrTlJjZ1FtWmNiTFJxdEJ2TCs3anBTRjBrdEhQUnRvazc4Z2VPRVVX?=
 =?utf-8?B?Y2R3N21hSVNEWDRWMm5pWWtnaEIzajNtTERRUlZoT1NvZFZjdVNxRit4TFJC?=
 =?utf-8?B?RGVML0FJdXNtME1CNDNSOW0wTmxhTEMyUkdnc01JTHZkZjhRZHk5U0lDWG01?=
 =?utf-8?B?KzZHc280aEJKbElCN3padGEyY3cwSGNPWGUxcityUUdmZ3hXclBHbnM2L0hS?=
 =?utf-8?B?aWF3OTZFVHlhKysxWGd0K29UN0lCUE5mUk9EMHNKWDBkZDVsWXl2b2Vvcjlw?=
 =?utf-8?B?SUlzQzhxeko5ZXlzRGhtUWVxdjRLbkVCaXd2bUl5V0g5aFZSM1lHM01jeCtT?=
 =?utf-8?B?UnVzQTdQRDdCZzRGVFlNMGxuU3VaUytuS0k5TlNESWRhTzJEWmdEZUJWMEVC?=
 =?utf-8?B?ZG1tSlE5M3JBNktlc09xQ0lXQ2FyVW5paVNMeldDOFdMZlRCZzVIWkUxU2lY?=
 =?utf-8?B?R2xPNWVxaC9UUHFvbkRBcndRcXhqQ0NXbjZ1QjB1VzdiYU9iQVZQK0pBSjNl?=
 =?utf-8?B?NGF3R0xRMENTYXdBTm5HeS8wcjNVOGxaeUkzN1h5dU00RFdBV2dmY1AwNXNR?=
 =?utf-8?B?UkI2emdLZkVKRHg4dnlyMWszdGs3MlExb2tpWHNmRVp0a0dabnFvcWhJbzJX?=
 =?utf-8?B?RkZvUTFsTC80WjhBcGsxNnJOb3JGenlKdnA4V3M3U1JtTmxzc1RpL2FlRzFC?=
 =?utf-8?B?aVdzc3o5NDE2aXB0QjFmcDhwOG9Jd2ZzbHZXRitIcW9pajZwa25IRk9BTzBK?=
 =?utf-8?B?VGs3ZkNKemdqQ3JjNHgvZXJtUnNLS3A0RmkzRjVsSHEzVWIwVUdZaExiTGJ2?=
 =?utf-8?B?U25vNzhtQ0ZvQ3ZOcTlyWjlFM3hhWmNzS0dhYVlyeVQvbU9WbkEyUEtnTFgw?=
 =?utf-8?B?QlVLVGJoVXZrZFV0b1Z2b20rYnNiZHNpc1Q5QXVBZG43cG82c1F1MURUN2NC?=
 =?utf-8?B?Tkd1WWVNaEk5bkV1TWMzVjV6Tms4ZjV1RXR4Um1wN3VMVE1ZajlCQ1dmdHdV?=
 =?utf-8?B?ZXpWMTRxL0xNNjJkSlRvQm1hYWloM2wrMU9FMExqK0dvNEs2MFF3WEhkUHBt?=
 =?utf-8?B?MGZwSzN1Q3VwSnMxVzFjQ3BONW1vYWpqb29aRXpjbk81TXZUcm1ab0xQM2k5?=
 =?utf-8?B?dnpqZFcxVVlTazNJWU9EN0JDMVZlaVc2bXNQMzd0REJqMytKOVFiVUFXckRU?=
 =?utf-8?B?NGhYV0RkZEJUelVvK2VUM2QxKzRUK1RVWDdmWFlkNE9zd1FFZ01rcE1GMFNa?=
 =?utf-8?B?NXNYUmk3ZXNPd1BOWElaejV1bWpxUUVvMlpsS090LzRyNVl6SXcrTm9WVGFz?=
 =?utf-8?B?RUdMaUJpTVdnVWY2eTArSFh2VVdtdHBCOU5jNWhTUzhDL1hpZFNSVkpVSzNY?=
 =?utf-8?B?Z0tFaVVUMDMyNXNQMWRzSmo5b3pkSjZRNDJ2Nk1LVGNuSU5HNDRCK1IwRTR0?=
 =?utf-8?B?TDIzNHV2dzQ0NnFKbDQySzdINGVQYmJjYjQyaWxwVE5FWTNTbW03bGVKQXVS?=
 =?utf-8?Q?oSdLeDjtnAIRk1bThXgWhWhwG?=
X-MS-Exchange-CrossTenant-Network-Message-Id: d4824035-0052-4c6f-b1b9-08dc40cced93
X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB4820.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Mar 2024 06:40:08.6033
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: JuV5j1wDH8ZE50Zb227Ffo0ATYccTosrNbXig9xKzVCYLyyRxJBqyHjDSfm35JRuy9h6WOW/rpKCOZQwe/sXIA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB6399
X-OriginatorOrg: intel.com

On 3/7/2024 5:23 PM, Jan Kara wrote:
> Thanks for testing! This is an interesting result and certainly unexpected
> for me. The readahead code allocates naturally aligned pages so based on
> the distribution of allocations it seems that before commit ab4443fe3ca6
> readahead window was at least 32 pages (128KB) aligned and so we allocated
> order 5 pages. After the commit, the readahead window somehow ended up only
> aligned to 20 modulo 32. To follow natural alignment and fill 128KB
> readahead window we allocated order 2 page (got us to offset 24 modulo 32),
> then order 3 page (got us to offset 0 modulo 32), order 4 page (larger
> would not fit in 128KB readahead window now), and order 2 page to finish
> filling the readahead window.
> 
> Now I'm not 100% sure why the readahead window alignment changed with
> different rounding when placing readahead mark - probably that's some
> artifact when readahead window is tiny in the beginning before we scale it
> up (I'll verify by tracing whether everything ends up looking correctly
> with the current code). So I don't expect this is a problem in ab4443fe3ca6
> as such but it exposes the issue that readahead page insertion code should
> perhaps strive to achieve better readahead window alignment with logical
> file offset even at the cost of occasionally performing somewhat shorter
> readahead. I'll look into this once I dig out of the huge heap of email
> after vacation...
Hi Jan,
I am also curious to this behavior and add tried add logs to understand
the behavior here. Here is something difference w/o ab4443fe3ca6:
  - with ab4443fe3ca6:
  You are right about the folio order as the readahead window is 0x20.
  The folio order sequence is like order 2, order 4, order3, order2.

  But different thing is always mark the first order 2 folio readahead.
  So the max order is boosted to 4 in page_cache_ra_order(). The code
  path always hit
     if (index == expected || index == (ra->start + ra->size))
  in ondemand_readahead().

  If just change the round_down() to round_up() in ra_alloc_folio(),
  the major folio order will be restored to 5.

  - without ab4443fe3ca6:
  at the beginning, the folio order sequence is same like 2, 4, 3, 2.
  But besides the first order2 folio, order4 folio will be marked as
  readahead also. So it's possible the order boosted to 5.
  Also, not just path
     if (index == expected || index == (ra->start + ra->size))
  is hit. but also
      if (folio) {
  can be hit (I didn't check other path as this testing is sequential
  read).

  There are some back and forth between 5 and 2,4,3,2, the order is
  stabilized on 5.

  I didn't fully understand the whole thing and will dig deeper. The
  above is just what the log showed.


Hi Matthew,
I noticed one thing when readahead folio order is being pushed forward,
there are several times readahead trying to allocate and add folios to
page cache. But failed as there is folio inserted to page cache cover
the requested index already. Once the folio order is correct, there is
no such case anymore. I suppose this is expected.


Regards
Yin, Fengwei