Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <99bc3831-8f5f-4a79-b0c3-8a492a7e5867@intel.com>
Date:   Fri, 23 Dec 2022 14:27:01 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.6.1
Subject: Re: [linus:master] [mm] 0ba09b1733: will-it-scale.per_thread_ops
 -21.1% regression in mmap1 benchmark
Content-Language: en-US
To:     Liam Howlett <liam.howlett@oracle.com>
CC:     Yang Shi <shy828301@gmail.com>, Yujie Liu <yujie.liu@intel.com>,
        "Linus Torvalds" <torvalds@linux-foundation.org>,
        "oe-lkp@lists.linux.dev" <oe-lkp@lists.linux.dev>,
        "lkp@intel.com" <lkp@intel.com>,
        Nathan Chancellor <nathan@kernel.org>,
        "Huang, Ying" <ying.huang@intel.com>,
        Rik van Riel <riel@surriel.com>,
        Andrew Morton <akpm@linux-foundation.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-mm@kvack.org" <linux-mm@kvack.org>,
        "feng.tang@intel.com" <feng.tang@intel.com>,
        "zhengjun.xing@linux.intel.com" <zhengjun.xing@linux.intel.com>
References: <202212151657.5d11a672-yujie.liu@intel.com>
 <CAHbLzkpVr62M4dWCb9J+eMErDvxx0hiTF6DD6bp2qEivzZYXCg@mail.gmail.com>
 <20221220204845.ul2wf4bj75dl5gbc@revolver> <Y6JwkEraN79acHP0@yujie-X299>
 <CAHbLzkpF4tuuukmhqC1soeVUaS4D5XtDjNDugy9t9D77s_xG8g@mail.gmail.com>
 <ed6c9173-dbbd-6275-a285-04b21de26f13@intel.com>
 <20221223024321.itxwvcdyckepnyiz@revolver>
From:   "Yin, Fengwei" <fengwei.yin@intel.com>
In-Reply-To: <20221223024321.itxwvcdyckepnyiz@revolver>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MHlDeHRoNkx4emNEYjZTdEYwM1dGZTJNTXQ2c2pzYzdmMlA1MDNEb21USHpx?=
 =?utf-8?B?S3BId21meTNFSnBSbWNVUWZxSzQ3MFp5NkNMRmczRWZQZXFyNDBIRHZqaWNG?=
 =?utf-8?B?eWYyaCtKWE1ITmZIZkZuU2dhSDgzY1d2bnduNWQrRjBnYXNnZVQ0NnFiY1l3?=
 =?utf-8?B?NkxQMFJvOTZET0JaSFcrMk9lS2FDNVF4UVp3UDh3TVMwcFNnRmtSb1dHUEo2?=
 =?utf-8?B?T3ZEZnYxRzVFY0lLOGdOVUw2cjMxdGovMWU5aWVKS2hudEFqY1RoNEJLUy8w?=
 =?utf-8?B?ZTFqMjc0Smk2SFU3Zmp1cVIvMjl2REVPdURjYTRyQWxSQXRtV2dtT1FEd1R2?=
 =?utf-8?B?bDgrYmwyelV5dzAvZ2NvMXRqZDQ5UUd1TDc3UldadU5nT0M0ZW1zcDRkdFRl?=
 =?utf-8?B?Y2Q2RXBsUEIzR1ZCMUl4TDFOaXROWld1enhlbmRGSmZ0L2FCQm1KY2JRRjBR?=
 =?utf-8?B?SFpSLzlmOXJXeGE2R1RJYnU2ZktTSzE5MWRaem00d0RkakJUOE5udjdzQkIv?=
 =?utf-8?B?UGV4T05MYU9xU2lrSkxaUFgzQ2d6S3FuMUx3MWlTTzZuU0wxeXlnRTh3cHc3?=
 =?utf-8?B?MnJ6QUxKNTdQZWRSRG12T210SEpzUDFQdi9qMGt2MkR4cGRDRWN6alpoeklG?=
 =?utf-8?B?bjNxKzFSRHdUVDc4WHNWVzNKdGl6TVEybkhNeTlSRWlieW9rd1V6ZldQOS9R?=
 =?utf-8?B?cExGbVNhWlhiU1MxUnRKRm5IWWtndC9pT240Mk0xWkNoN1pTL2M1TXYwRVBy?=
 =?utf-8?B?bXVrMnlUL3VmQmFleHdjbDJCajhNV2FwMURFd3RtbU5YdC80aVRaeVBQSmdV?=
 =?utf-8?B?eWExbzlSKzRiNWdVcnNJa0ZEODZISzlRZ1hoMEM4ZmthS0ZoSVZHaWpqeXIw?=
 =?utf-8?B?RmxWSE5NNDZOL01IelM5YjhSQzlaUFE0cks5enBybG82ZXVZK09LM1Jpc3pZ?=
 =?utf-8?B?ektpTWY3a0NTcm04bmNKSkxGYUREeDA1TjNBMmdIRXdyVzN6Y2tNRHRBWGY0?=
 =?utf-8?B?Y0JJZWFPczN0K2pKYjh1WW5YdDAvTG9UaHhvRkRmM0JBTGFwRFJ0QlpQbDZS?=
 =?utf-8?B?Y0J2NGNTR1Y4NFpwOUNTQnRmbHpJMVh6TkdsYUdNb1ljQVJsYjY0NmlDdFNi?=
 =?utf-8?B?dC9neno3aUdiZVl0TkFOTVkyQlZsQ0M2a0hJbzM3TUhTSktGWENSVWp0QkEr?=
 =?utf-8?B?TDQwYU1lVmpYM3gvQ0IwTkxIajVkTzMwdnlDYThJMU9xa0NBNkplaTlVSWlW?=
 =?utf-8?B?aW5JYXkvckQrMGFkUnN3bFFrdlVGU1d5Mmx6VjJIdm9VekJDaFBCbUxxOWEv?=
 =?utf-8?B?cHVHRkxSa1orMTZlTmtqRTI2WHRtUkFtOEtmVXRxcHdMTCtWaG5TRDBXcC9o?=
 =?utf-8?B?WjQ1TTVPNmJTcVE4REtrb3VFQklvVlVWbEg2ZjFOVEEvQ29acWVOUXh5TE9x?=
 =?utf-8?B?SU4zWDRqbDF4OWZzYkdUQitja1R5RS93UTZ6T0l1Wk5heFlFUmM3MXFOK1Qz?=
 =?utf-8?B?YmRQbWRvVEhza0RGWityanZsWENiajRwMXZ3cUlWWjJqd0JUMUh2ci9VdDJP?=
 =?utf-8?B?ZmZUY3lyNFhNV2hHalpsaE5rTmc4bjlSUlRjZCszTXFOUCswRkYxL2ZTYVV2?=
 =?utf-8?B?OHREREpHSVdsWEMxTVdXRVEwWTAxV01pU0ZKMXBwRlJxMG9pSkd3bHhQdytE?=
 =?utf-8?B?TG9zeHVPd2g0akFCL0djRDB1dUhSWHplMUsxd3Zyd3UwVzJKc25ySHU1aGpG?=
 =?utf-8?B?S3pscVJ5ZU1pTEU2VTFBYUpOS29HMXhuS1gvYTg3NXdndEphbHlOU040WjF0?=
 =?utf-8?B?bmUxeUxHK1A2U1pPcFBVTFJ5ZEtFU2RiNjdRTllBS3ZFMHVjbWtUQnA4MVVW?=
 =?utf-8?B?bWxneWRraERQSlFLT1NEaEdKckRRQUpoUXpDY0xTYjZ5WEExNnJJTXQ3czNP?=
 =?utf-8?B?V0ltNFJaSWhrbXNmYmtLZ3diUHE2c0M1TXFXVEJHdXk5STl0a1pldUptUEFk?=
 =?utf-8?B?djRLVmV3aXkxUnpydm1IbzcrZWpuSC85TG1NOFNmK3E3elBhZkg3R3k1N01m?=
 =?utf-8?B?RFB1T0EwSTNBUFR2TTd2Y3lQdFJnNzVJa0RNWFVobERaVGxGbnN2a2pZVVds?=
 =?utf-8?B?ek84aU16RlUrLy9xNXhQOHZNbnh5bDl4Q1VBWlBqUXlScUhwMWd6ekhmMlFq?=
 =?utf-8?B?Rnc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 52f3ab60-cbb2-46df-7e72-08dae4aeb956
X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB4820.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Dec 2022 06:27:11.2960
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ILRlCi34ZudUFFPnyuvNSGELzvwVCJWDFIVg/IvH5zzrFpozTc8bh6/WeDSqOH8fSktOrMI0Zgnpl/TC084AuA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR11MB5370
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-3.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 12/23/2022 10:45 AM, Liam Howlett wrote:
> * Yin, Fengwei <fengwei.yin@intel.com> [221221 20:19]:
>>
>>
>> On 12/22/2022 12:45 AM, Yang Shi wrote:
>>>> We caught two mmap1 regressions on mailine, please see the data below:
>>>>
>>>> 830b3c68c1fb1 Linux 6.1                                                              2085 2355 2088
>>>> 76dcd734eca23 Linux 6.1-rc8                                                          2093 2082 2094 2073 2304 2088
>>>> 0ba09b1733878 Revert "mm: align larger anonymous mappings on THP boundaries"         2124 2286 2086 2114 2065 2081
>>>> 23393c6461422 char: tpm: Protect tpm_pm_suspend with locks                           2756 2711 2689 2696 2660 2665
>>>> b7b275e60bcd5 Linux 6.1-rc7                                                          2670 2656 2720 2691 2667
>>>> ...
>>>> 9abf2313adc1c Linux 6.1-rc1                                                          2725 2717 2690 2691 2710
>>>> 3b0e81a1cdc9a mmap: change zeroing of maple tree in __vma_adjust()                   2736 2781 2748
>>>> 524e00b36e8c5 mm: remove rb tree.                                                    2747 2744 2747
>>>> 0c563f1480435 proc: remove VMA rbtree use from nommu
>>>> d0cf3dd47f0d5 damon: convert __damon_va_three_regions to use the VMA iterator
>>>> 3499a13168da6 mm/mmap: use maple tree for unmapped_area{_topdown}
>>>> 7fdbd37da5c6f mm/mmap: use the maple tree for find_vma_prev() instead of the rbtree
>>>> f39af05949a42 mm: add VMA iterator
>>>> d4af56c5c7c67 mm: start tracking VMAs with maple tree
>>>> e15e06a839232 lib/test_maple_tree: add testing for maple tree                        4638 4628 4502
>>>> 9832fb87834e2 mm/demotion: expose memory tier details via sysfs                      4625 4509 4548
>>>> 4fe89d07dcc28 Linux 6.0                                                              4385 4205 4348 4228 4504
>>>>
>>>>
>>>> The first regression was between v6.0 and v6.1-rc1. The score dropped
>>>> from 4600 to 2700, and bisected to the patches switching from rb tree to
>>>> maple tree. This was reported at
>>>> https://lore.kernel.org/oe-lkp/202212191714.524e00b3-yujie.liu@intel.com/
>>>> Thanks for the explanation that it is an expected regression as a trade
>>>> off to benefit read performance.
>>>>
>>>> The second regression was between v6.1-rc7 and v6.1-rc8. The score
>>>> dropped from 2700 to 2100, and bisected to this "Revert "mm: align larger
>>>> anonymous mappings on THP boundaries"" commit.
>>> So it means "mm: align larger anonymous mappings on THP boundaries"
>>> actually improved the mmap1 benchmark? But it caused regression for
>>> other usecase, for example, building kernel with clang, which is a
>>> regression for a real life usecase.
>> Yes. The patch "mm: align larger anonymous mappings on THP boundaries"
>> can improve the mmap1 benchmark.
>>
> 
> If the aligned VMAs cannot be merged, then they do not need to be split
> on freeing.  This means we are just allocating a new vma, write it in
> the tree, removing it from the tree, free the vma.  We can do this 4600
> times a second, apparently.
> 
> If the VMAs do get merged, we will go through __vma_adjust() to expand a
> boundary, write it to the tree, allocate a new vma, __vma_adjust() the
> vma boundary back, insert the new VMA that covers the boundary area,
> remove the new vma from the tree, free the vma.  We can only do this
> 2700 times a second.  Note this is writing 3 times to the tree in this
> loop vs 2 in the other option.
Thanks a lot for sharing the quantification of two path.

> 
> So yes, merging/splitting is more work and always has been.  We are
> doing this to avoid having too many VMAs though.  There really isn't a
> good reason an application would do this for any meaningful number of
> iterations.
> 
>> For building kernel regression, looks like it's not related with the
>> patch "mm: align larger anonymous mappings on THP boundaries" directly.
>> It's another existing behavior more visible with the patch.
>> https://lore.kernel.org/all/a4bcddad-e56f-cedc-891a-916e86d9a02c@intel.com/
>>
> 
> Having a snapshot of the VMA layout would help here since the THP
Can you share how to do snapshot of VMA layout?


> boundary alignment may be changing if the VMAs can be merged or not.  I
> suspect it is not able to merge and is fragmenting the VMA space which
> would speed up this benchmark at the expense of having more VMAs.
Same thought in my side for more VMAs here. Thanks.


Regards
Yin, Fengwei

> 
> Thanks,
> Liam