Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <eda0d716-4292-6117-b036-4df64c5df110@intel.com>
Date:   Tue, 27 Jun 2023 12:38:59 +0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.11.0
Subject: Re: [linus:master] [page cache] 9425c591e0: vm-scalability.throughput
 -20.0% regression
Content-Language: en-US
From:   Yin Fengwei <fengwei.yin@intel.com>
To:     Mike Kravetz <mike.kravetz@oracle.com>,
        kernel test robot <oliver.sang@intel.com>
CC:     <oe-lkp@lists.linux.dev>, <lkp@intel.com>,
        <linux-kernel@vger.kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        Ackerley Tng <ackerleytng@google.com>,
        Erdem Aktas <erdemaktas@google.com>,
        Matthew Wilcox <willy@infradead.org>,
        Muchun Song <songmuchun@bytedance.com>,
        "Sidhartha Kumar" <sidhartha.kumar@oracle.com>,
        Vishal Annapurve <vannapurve@google.com>,
        Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        <linux-fsdevel@vger.kernel.org>, <ying.huang@intel.com>,
        <feng.tang@intel.com>
References: <202306211346.1e9ff03e-oliver.sang@intel.com>
 <20230621152854.GA4155@monkey>
 <126cb393-31aa-d27f-ac0e-f86724eae0db@intel.com>
In-Reply-To: <126cb393-31aa-d27f-ac0e-f86724eae0db@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?eHg1USt5NGdOQk5Mclk0Wm1BV3BGOVRndllJSlhQcUM2ME1kZDVyUDcxWW1J?=
 =?utf-8?B?NWY5S0UvN0ZoTUVoNWtGN3MrYVEzcmVJOTdRcksrZ1M4cFAyVXFNZlB5S0tS?=
 =?utf-8?B?S1B0QjZkNFdFOU5HNGtDRTJuaVlYWjdWSVhaaGIwQVZ1Y0lOcG9OMEdmM2hD?=
 =?utf-8?B?eEwzT1N1TFFHYzgycWRXQVdTQUZla1duN2JZd21ZeEZzblIzSytBbWlOOUVv?=
 =?utf-8?B?MHhuMmdXaXk3YnNnbkZCcEFkUndLOWxWdmVPbWltcWpGWnB6Yk5vTjcxK2RU?=
 =?utf-8?B?bjNldlQ0KzhFKzJDOGpYWHM4KytZSTVqWGNXQWh0RC9UOGFHeTdJemxGWnQ2?=
 =?utf-8?B?NXdTeVNpWWhwbUJRbmE3QlQxMlFMV0Rob0Z4YjI1RHR6cmR0Tlk2aFg2dmhW?=
 =?utf-8?B?UWhYN3d2dWI5Sm5LN2JaaHp4RVF0a2dCTWZKWi9WdDdRVm84aGM0Rm9RV3BR?=
 =?utf-8?B?RC9JSHJsU1JPTFVjb1VZZTU4eFBaVFk1SW51akh0UkdEZno5Z2lZRThKL1Q2?=
 =?utf-8?B?M09OTTNReFQvR3VnM0I4THE4Zkp6ZFhudTZsR0hlRDRyeXViT0M2VjdMazNC?=
 =?utf-8?B?YWI5NGpFZi9XUWdRakZTYjNMYXYvcStIaFZrWDRJRzFrM3Ircm80SWlKT2hJ?=
 =?utf-8?B?bjVCQlQvMGpIc2NCNm1YNngwWjRqdk1SS2tzRG1TaVFaUm90bUZtTGFxMm9S?=
 =?utf-8?B?TXNEckxRL1d5VTg2M3FkWk1DYkV6TmREYU1ZeThJa3FjbG5YdTNxdXd4RHE5?=
 =?utf-8?B?V1UwOURVeEFmVm1maE0ya09xZjk3Nml1RTdkc0ZJZnFFejNRblNUaWh1Z2pQ?=
 =?utf-8?B?SFlpSHNpaHNqOXd4RVJUS2RlYnN6ZUxRSGtKNmJLbjJNeDIzakZEZk5sT2pv?=
 =?utf-8?B?Wnl1SkJjQTd3KzdmOUQ4aWRuR0QrM2NmT3lESENBenpDQnJralNXRE93elZa?=
 =?utf-8?B?U0VEZ3dEaWV0VEZkMktnZUJuWHFMTTloaktLVzdzcEZUZHYzTmRIMUxMemg1?=
 =?utf-8?B?WjBhT1dOeEQ2V3BkdTR6d0c0eHhiejFlS3BXVy9NSFN1UlBUd0xBbUNXRFp1?=
 =?utf-8?B?cEhNdGVqZkNkSTBsN0plYkF6WUNmTDZiQ2Z6ZnFxNjNjZExNOTBYempwY0Rn?=
 =?utf-8?B?VXdrUEtuV0NKVlpUN0cxeENoYzc4R3V1L0hLUWY3T2gzSEJFT2tFVjlEejNh?=
 =?utf-8?B?bWplMnBNK1hZdGR0OGFqSkIxc2hjL25aYXFyaXZVZStrdXM1VkIyVTVydVV4?=
 =?utf-8?B?SVdDSmlZNTI1TTZrdXltVFVrQ0taWitKeDBYYlFUb2xVbE8vT2g1UnhlbVpJ?=
 =?utf-8?B?M3FmU3k0UjJ6ckpGU21ISlphWCtqSnF6L1k2WUNZd1kvYWhwQ1N5RTJyZDYw?=
 =?utf-8?B?ZWRCNUxYN21yWDVMa0dLNjRGSXNPRzFHRFRvNWJIMHg4TzYveW9GekgyRTd2?=
 =?utf-8?B?cklxV3lmV3JuVjJvbU55dnFBcFZmV3RCc3JlZlM0UlFJaEhjQTRZbGdyQ2V5?=
 =?utf-8?B?MkRxeWF0clM1KzhKaGlOOWhHeFFRSGp2ek9rS2RYcHJFWDIwV2dQQzVkUnl2?=
 =?utf-8?B?SWc3cXRZbmZRd0RIU2tDOEQzM1hOd3hDOEpSc1lxS1RKMlZOUFBBWmhmTkxE?=
 =?utf-8?B?T05BNjhCVSs0VE9LY3BOQitaYmZDV0tsbWpOT2c3QVJ3bEpqNU1oY1VqUDJE?=
 =?utf-8?B?MWxML0Fkbkw3NFZ2aGRkV3pETEpja3pyYW1SRE5RcitnU2paeEZ6ZUdZSVBt?=
 =?utf-8?B?Znlja2pQbVhNR3VSS1ZYRHpyaWFBQnNBWHQwMHB4cjdBS3BCVHpWRFBtbTRK?=
 =?utf-8?B?amJUS2hmNk9kaWVUeUdqMlRIajgxMC92UnAzSmxPTzJpbitvc0cyMXpKbnJW?=
 =?utf-8?B?dVp4dDdETkxoNlFJVWV5UXR2cTl6czVQRk13UXlheVJkeUxNMDd6b3lESjJ4?=
 =?utf-8?B?cWdLTHQ3TklLblNjd1lJVUxqOHN3aVhubm1vd1FuY245UmMydHAxZXhHZ0dX?=
 =?utf-8?B?RkZ5cWhIZHRLem5nYjd2T0UxY1FLdStkMmpkcVB4UjAwNHdPMlBlaXhnZFU1?=
 =?utf-8?B?Q2xPNERPZ0c1VGhHS2NoL1hSMEIzVm1UV05aTm9lem5JL1pWdDRrOCtBcCsy?=
 =?utf-8?Q?tm1ETI4Ue07hIsAzdaD6Qabb2?=
X-MS-Exchange-CrossTenant-Network-Message-Id: f68a16fa-357b-46ee-9a5d-08db76c876d2
X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB4820.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jun 2023 04:39:16.3245
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: iiz5oN6r15lT/ijB+31H4Knz4Lhro4ibPhoiVaY5JD6b0Knt7jgdjK696xM0xdshILQIR8XhkPLTc6Wyx5PQmQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW5PR11MB5883
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,
        SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 6/26/23 17:05, Yin, Fengwei wrote:
> Hi Mike,
> 
>> On 06/21/23 15:19, kernel test robot wrote:
>   <snip> 
>> I suspected this change could impact page_cache_next/prev_miss users, but had
>> no idea how much.
>>
>> Unless someone sees something wrong in 9425c591e06a, the best approach
>> might be to revert and then add a simple interface to check for 'folio at
>> a given index in the cache' as suggested by Ackerley Tng.
>> https://lore.kernel.org/linux-mm/98624c2f481966492b4eb8272aef747790229b73.1683069252.git.ackerleytng@google.com/
> 
> Some findings in my side.
> 1. You patch impact the folio order for file readahead. I collect the histogram of
>    order parameter to filemap_alloc_folio() call w/o your patch:
> 
>    With your patch:
>      page order    : count     distribution
>         0          : 892073   |                                        |
>         1          : 0        |                                        |
>         2          : 65120457 |****************************************|
>         3          : 32914005 |********************                    |
>         4          : 33020991 |********************                    |
> 
>    Without your patch:
>      page order    : count     distribution
>         0          : 3417288  |****                                    |
>         1          : 0        |                                        |
>         2          : 877012   |*                                       |
>         3          : 288      |                                        |
>         4          : 5607522  |*******                                 |
>         5          : 29974228 |****************************************|
> 
>    We could see the order 5 dominate the filemap folio without your patch. With your
>    patch, order 2,3,4 are most used for filemap folio.
> 
> 2. My understanding is your patch is correct and shouldn't be reverted. I made
>    a small change based on your patch. The performance regression is gone.
> 
> diff --git a/mm/readahead.c b/mm/readahead.c
> index 47afbca1d122..cca333f9b560 100644
> --- a/mm/readahead.c
> +++ b/mm/readahead.c
> @@ -610,7 +610,7 @@ static void ondemand_readahead(struct readahead_control *ractl,
>                 pgoff_t start;
> 
>                 rcu_read_lock();
> -               start = page_cache_next_miss(ractl->mapping, index + 1,
> +               start = page_cache_next_miss(ractl->mapping, index,
>                                 max_pages);
>                 rcu_read_unlock();
> 
>    And the filemap folio order is restored also:
>      page order    : count     distribution
>         0          : 3357622  |****                                    |
>         1          : 0        |                                        |
>         2          : 861726   |*                                       |
>         3          : 285      |                                        |
>         4          : 4511637  |*****                                   |
>         5          : 30505713 |****************************************|
> 
>    I still didn't figure out why this simple change can restore the performance.
>    And why index + 1 was used. Will check more.

The thing is the ra initialization after page_cache_next_miss() in function
ondemand_readahead():
  ra->start = start; (start is index + max_pages + 1 + 1 after your patch)
  ra->size = start - index;

And +1 will be accumulated to ra->start and breaks the filemap folio order.


Regards
Yin, Fengwei

> 
> 
> Regards
> Yin, Fengwei