Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <69aada71-0b3f-e928-6413-742fe7926576@intel.com>
Date:   Tue, 4 Jul 2023 14:22:26 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.12.0
Subject: Re: [PATCH v2 0/5] variable-order, large folios for anonymous memory
Content-Language: en-US
To:     Yu Zhao <yuzhao@google.com>, Ryan Roberts <ryan.roberts@arm.com>
CC:     Andrew Morton <akpm@linux-foundation.org>,
        Matthew Wilcox <willy@infradead.org>,
        "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>,
        Yin Fengwei <fengwei.yin@intel.com>,
        David Hildenbrand <david@redhat.com>,
        Catalin Marinas <catalin.marinas@arm.com>,
        Will Deacon <will@kernel.org>,
        Anshuman Khandual <anshuman.khandual@arm.com>,
        Yang Shi <shy828301@gmail.com>,
        <linux-arm-kernel@lists.infradead.org>,
        <linux-kernel@vger.kernel.org>, <linux-mm@kvack.org>
References: <20230703135330.1865927-1-ryan.roberts@arm.com>
 <CAOUHufYB2kB0r9hhSbzfEzdF85MkXVfWoFOhy3LwLfJ5Qo8H6g@mail.gmail.com>
From:   "Yin, Fengwei" <fengwei.yin@intel.com>
In-Reply-To: <CAOUHufYB2kB0r9hhSbzfEzdF85MkXVfWoFOhy3LwLfJ5Qo8H6g@mail.gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SndBOXZGMTRzNXBPSEIzRE15WHpJWkdOYmtrMVlsS0JQK3NWK2NIdklRQkV3?=
 =?utf-8?B?NXovbUlLZ0toZk8rUThIOEQ4dlh6bW9PWlM2OUo4Q3BKVWNZaVlKU1VzajY4?=
 =?utf-8?B?MnNkaXdlb2Urby8zNzdvOW5aTVVFZENzTDAxL0ZSQ3JPeWVxSml4WkpjSDJo?=
 =?utf-8?B?ZkRqamhsbWp6L2FOUmR4YjRDMXhkU1VKbUd1anpFekpaVjJobGFzWExDMmMv?=
 =?utf-8?B?Q2dsZGpPeWUvYy96b3pURFZqejlvZjVWZXYvelVPZTlWQ0hVclJzTXk4azRo?=
 =?utf-8?B?NTV5QWNNZDhUNFlPVzFhWEt0V2tqZitRRE5nVllyK1NhazZlR3NsMmJYd09Y?=
 =?utf-8?B?M01tRjVDekpJamJlOUdYMllNa085aDhuTDBVYUp1bFY3TlhqaTVEVFh0RGI3?=
 =?utf-8?B?RGt5MmZmNFN2Vm5XMGI5SWM1L08zQUFBcWdTUGhtZlFqUnJNaEgxNWJsOWxT?=
 =?utf-8?B?Z0x4ekxGQjlINWdIMHFxeDIrRFRvbEVVZzFueVFNd1ViRnVadVpWQ2laZE9z?=
 =?utf-8?B?OWlLbnZ0bHcvMXRsY09XNzFxeGM4cE1ITi9wMEdpNDZMOEY2N0s2Ymc3RmZB?=
 =?utf-8?B?NWhnUEZGVHN4VXNNenMxd2hiejRQTE5BTXVDUDM0S1k2TXlmY1J1Ym1mOHVi?=
 =?utf-8?B?L0RuYzFSQ2hqdVNWNytHYUpoVmpoWld0WVlsR1ZxQnVtaENKMmZudk4ycStQ?=
 =?utf-8?B?VWlpUU43T3BacldpbXgxRmg3SDU3VEZrYlc1ZE1WMGxJMUNYWVRnVUdBTTE2?=
 =?utf-8?B?eFkvRWdEbEhCZ29nVm90WjBpYnZTODJxVzBlWHNxNDdNeHoyb0tQTUpSYmVa?=
 =?utf-8?B?dFV1MC8wYWpkdGtiM2htTzNOTFZtQUIwVmtxVG01cVhSRDlCcGltY2JKQkxW?=
 =?utf-8?B?TzJIS3NkTm05b3YrR1hDa2ZBNUlrcU5aREFtUzg2Z0xaZU1FOXhEaWZqUnA3?=
 =?utf-8?B?L0Z1WTRDZTZ0ZHZaSjZDalppckN4TGJxa3lQOFJzbFlncU5oVE5GYVF5bjk2?=
 =?utf-8?B?WU4xSThxUjlnYis2Uk5na3daVkd3dzhXRVFzWW1RbnNPdk5KS2wwTlBLWGlO?=
 =?utf-8?B?a1luZEloVy9FZDZpS0pJbjZYWXN4VmRjQ25CQmtsVjU1dUxpL09KVFpiRlEy?=
 =?utf-8?B?NUZNNWp0NTF2WGJFVmpGWjA5dXJ2YjlCeHRQL09qbER4MFR6dmdLT3dYbGFh?=
 =?utf-8?B?Vm9XWDRZeDhwUjhJL0tUVWFEWVAwczlUSndRM1J3OXFnZHdod25WR3dHajBy?=
 =?utf-8?B?SDkra0QvbzBWVWZIanZXTjArK3FmdmhUNFowMWh6aG40akpBK05pYXFQcnFp?=
 =?utf-8?B?aGdPeWlkWndTY1JiSml0eHFRalFkNWlnMWcySmhETDFJdE1KQXg1R0xleGlK?=
 =?utf-8?B?THkvbkxUMTFNMVdBVXhvNjNETVF6cXU1c1ozUjF2K0lMSXY5S1dJbmJFT0h6?=
 =?utf-8?B?WDQ4L3JOVlloQWhORGFwRWNvemxDV2lZMm1YdG9IVTBaWVlITjJDWVpTSEpU?=
 =?utf-8?B?aTBSSFp4b2t1UGdDckhOOVV3TGRESzVDbXl0K3c2cm1ZWUVpbGlPYXV3d3lM?=
 =?utf-8?B?dC80RXdMOURTNEMxRWRqQTQ0Z0pJQXduWktPdmVLNjRuc1pjR1h0UFJtYmho?=
 =?utf-8?B?WlZGc0hQS0lsek5XdERNZFFHMXkvK2JLbyt6a3A4YnRLV0pPMldkRllPbDFJ?=
 =?utf-8?B?UmkxTEdMWFJ3V25hU2ZJQnlncTZrTVBhaVNJdzJKR1Q4VCtpa2s5R3QwcGdF?=
 =?utf-8?B?KzZCRE5OcjBKRXJpd3RuUk12RXI2WmpEdzY5bVBzR2lDT3hDVmxLYnRldWg3?=
 =?utf-8?B?QU5hSlViZ0VCejZRNGJMR0NoeWVodUF3S3c3WEZrUUM0ay9obUE0OEwwa3hn?=
 =?utf-8?B?ZmY0cVVEWFFRano2RmhxbFJFODBBcmxsNDBNOHMzUFdsd2tkb1JOYXgxbCs2?=
 =?utf-8?B?RDNXbUJ6Y1Zkb00vQ0N6YU9BUjdGdFNEQnpQbzdxcFdaRDB2TGdLT2k5dmE0?=
 =?utf-8?B?QURYMzJiWkpQR0FTYkZiUzlwc2ZBTlNuM1BheFhvcHlWR3ZXMWxGR2lHc01C?=
 =?utf-8?B?YXhhL2hzamw1RG5sdFgrYmtaVGFkMzdqOWpSUi9RaWovRHVNUFB1Y29RazN0?=
 =?utf-8?B?STIzeThodVRzUXg1ZWYrdy9OdG4zZURsYlhFQXRYM3ptcEZYd083LzZiWFMv?=
 =?utf-8?B?blE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 0af86952-419a-4382-9218-08db7c570ff9
X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB4820.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jul 2023 06:22:37.7877
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: szrTUSjaBz86C1uN/NqAa48NSQn2R/wDs54HA04raLPTEE3uRkPKmEC6/A3h3NlapxeqLaPRfjj9KczQBln5gw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYYPR11MB8305
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_NONE,T_SCC_BODY_TEXT_LINE,
        URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 7/4/2023 10:18 AM, Yu Zhao wrote:
> On Mon, Jul 3, 2023 at 7:53 AM Ryan Roberts <ryan.roberts@arm.com> wrote:
>>
>> Hi All,
>>
>> This is v2 of a series to implement variable order, large folios for anonymous
>> memory. The objective of this is to improve performance by allocating larger
>> chunks of memory during anonymous page faults. See [1] for background.
> 
> Thanks for the quick response!
> 
>> I've significantly reworked and simplified the patch set based on comments from
>> Yu Zhao (thanks for all your feedback!). I've also renamed the feature to
>> VARIABLE_THP, on Yu's advice.
>>
>> The last patch is for arm64 to explicitly override the default
>> arch_wants_pte_order() and is intended as an example. If this series is accepted
>> I suggest taking the first 4 patches through the mm tree and the arm64 change
>> could be handled through the arm64 tree separately. Neither has any build
>> dependency on the other.
>>
>> The one area where I haven't followed Yu's advice is in the determination of the
>> size of folio to use. It was suggested that I have a single preferred large
>> order, and if it doesn't fit in the VMA (due to exceeding VMA bounds, or there
>> being existing overlapping populated PTEs, etc) then fallback immediately to
>> order-0. It turned out that this approach caused a performance regression in the
>> Speedometer benchmark.
> 
> I suppose it's regression against the v1, not the unpatched kernel.
From the performance data Ryan shared, it's against unpatched kernel:

Speedometer 2.0:

| kernel                         |   runs_per_min |
|:-------------------------------|---------------:|
| baseline-4k                    |           0.0% |
| anonfolio-lkml-v1              |           0.7% |
| anonfolio-lkml-v2-simple-order |          -0.9% |
| anonfolio-lkml-v2              |           0.5% |


What if we use 32K or 16K instead of 64K as default anonymous folio size? I suspect
this app may have 32K or 16K anon folio as sweet spot.


Regards
Yin, Fengwei

> 
>> With my v1 patch, there were significant quantities of
>> memory which could not be placed in the 64K bucket and were instead being
>> allocated for the 32K and 16K buckets. With the proposed simplification, that
>> memory ended up using the 4K bucket, so page faults increased by 2.75x compared
>> to the v1 patch (although due to the 64K bucket, this number is still a bit
>> lower than the baseline). So instead, I continue to calculate a folio order that
>> is somewhere between the preferred order and 0. (See below for more details).
> 
> I suppose the benchmark wasn't running under memory pressure, which is
> uncommon for client devices. It could be easier the other way around:
> using 32/16KB shows regression whereas order-0 shows better
> performance under memory pressure.
> 
> I'm not sure we should use v1 as the baseline. Unpatched kernel sounds
> more reasonable at this point. If 32/16KB is proven to be better in
> most scenarios including under memory pressure, we can reintroduce
> that policy. I highly doubt this is the case: we tried 16KB base page
> size on client devices, and overall, the regressions outweighs the
> benefits.
> 
>> The patches are based on top of v6.4 plus Matthew Wilcox's set_ptes() series
>> [2], which is a hard dependency. I have a branch at [3].
> 
> It's not clear to me why [2] is a hard dependency.
> 
> It seems to me we are getting close and I was hoping we could get into
> mm-unstable soon without depending on other series...
>