Message-ID: <12d7564f-5b33-bdcc-1a06-504ad8487aca@intel.com>
Date:   Fri, 17 Mar 2023 14:33:47 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.8.0
Subject: Re: [PATCH v4 35/36] mm: Convert do_set_pte() to set_pte_range()
Content-Language: en-US
To:     Matthew Wilcox <willy@infradead.org>
CC:     Ryan Roberts <ryan.roberts@arm.com>, <linux-arch@vger.kernel.org>,
        <will@kernel.org>, <linux-mm@kvack.org>,
        <linux-kernel@vger.kernel.org>
References: <20230315051444.3229621-1-willy@infradead.org>
 <20230315051444.3229621-36-willy@infradead.org>
 <6dd5cdf8-400e-8378-22be-994f0ada5cc2@arm.com>
 <b39f4816-2064-e402-4e02-908f40c396d4@intel.com>
 <2fa5a911-8432-2fce-c6e1-de4e592219d8@arm.com>
 <ZBNXcmOrrOS4Rydg@casper.infradead.org>
 <b2c00aab-82ad-ea7a-df9d-c816b216b0f1@intel.com>
 <ZBPiOgYDLYBmVwOc@casper.infradead.org>
From:   "Yin, Fengwei" <fengwei.yin@intel.com>
In-Reply-To: <ZBPiOgYDLYBmVwOc@casper.infradead.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?N3dWYm1iYWdvNjR6dHFsK3BqSEdRZ0xBaXRBQ2RYV21mMjA5VkpxclhJWTM5?=
 =?utf-8?B?QWN5UEMxZHNGbmF4S2VDeGZwUWFOOWFMcVNVaGhrT1kxMVExSTVRV1U0T2ZU?=
 =?utf-8?B?Q2gwR1JadVFQbDh1SDByQjZiZG5UV0dWOWEvK2ZrS0h3emFCSTBDWVc0UGxh?=
 =?utf-8?B?ZEEwSHB5ZUNSOWFEbW9vTHo4VkJWZGFaZmpmYXN6WkFTcTJoeHRjaW96bjQ0?=
 =?utf-8?B?NGxnV1ZLTUJyMkNPYy9UdHZ3MkRBV0ZoemVod251cU9oaE4yQnhhMGJUV1Mx?=
 =?utf-8?B?V0hzUmpsMVorWkxmdytYbWNjY1owU20xSTltOHdkQXV4L1FpSzNKRmdOWHFZ?=
 =?utf-8?B?S2RGSWh2a3UvL284TldMWHJ0RmF6RHY4MHZVSzduTXdFVmJLaTh3L1Uyd1Zt?=
 =?utf-8?B?YzFUQUJEbU93MVFvVHZBNms2RUd1SkFqeGN0WEowRU44NjEvaVYveHhjV1d0?=
 =?utf-8?B?U0NscTVYMFRBZnlIc0txTmlYcUxVdzVlcE9LaFF6d2FnOUdBaVI4MkgrN1dQ?=
 =?utf-8?B?UzE3RFdFTm5NYTJzUDBSQjdQSk80bzRRWWRBQlhkOXhEQjF4V1NpWXRGakJS?=
 =?utf-8?B?U0wrT3MxNk9TMm1YOTdXZ0hmcG5MdkxQWGc5VWdDQ1lBaFFKSkZPeXB3RHlF?=
 =?utf-8?B?ZGppOXl0VGFQemxCQ0ZjRkpTR0VTWnVZRUpnRDF6MnpxZTRUbTcvRWQ0cUlv?=
 =?utf-8?B?SkVaVE9wOUl0eXhXeGlJanlXYzJIUm1NSUE2VkZOb1M5K2wxck1oMVZyZkI4?=
 =?utf-8?B?ZURYWW0rYmtXb1g3d0JkbFRFRFNDRi9DRHBwd0NFZDdxb1Qydk1SMlVZeTNp?=
 =?utf-8?B?L3NZaWtvWUR1NnE2R2lKdEwwWVJlUHlvRWQ5TSt5aEg1SGR1VmZ0Zm5sVVFp?=
 =?utf-8?B?dkRDaVhJeUMwNml2ZmlYNHBEQjhXSS9GK3Vacm1JczFSV3hjTXl0ei9FQlJU?=
 =?utf-8?B?Rnl0Y3FzM29mVXRwcFpsK1c2aFdXMlNmd3lhV0Q3YitxNjI0WXBzd3lrd2w1?=
 =?utf-8?B?UUVkdXlocnEwZUR6clhId0VEdHZSTU1YSU0zU3VrNlJVMmdEUlhYNWZvV0Va?=
 =?utf-8?B?bmpQallvSlRpMW9ES1lpa3FXNDVoWVNoa3FhcXhOVUVqTTQ0WVhlK004YVVF?=
 =?utf-8?B?dzBtYjF1QW56SWkrRkhYUmd1Z1F2UVR6YnhPbTJPOUxGOElCVzF4QkRDRlhG?=
 =?utf-8?B?QVlyWUl4QWowVU85a3JzTC9DTzhkckVrTk5UeXlIc3BHd3RjOVplUUErcmpR?=
 =?utf-8?B?aU9DRytObEJxZnUvQUUrZzNJdnVRUnlxTDRKMXBpU2lLOVZGOHlvOHk2M2la?=
 =?utf-8?B?MGZXbXNSYmsvdTdZcGl6SlExZTdHUXI3bTU5VmZGT3lWMGdzNldTR01kTVFw?=
 =?utf-8?B?OWpTUDV5WjVEYjlDZWYwbjhjUzUwenlBUjNONk5ncUVaUXBZSnhqZTMycXJP?=
 =?utf-8?B?ZVkxMzlyZnZuLzUrVEM1SnpSTVhmbWhaSjQrQkhQcDltUmZIZ2lPd0I1UnBl?=
 =?utf-8?B?MnFsdUZmK0t3VGpWMTdyc0xCcWlnMEwyL0hWSWVPZ2k5TXE4ckVVMGhUM0FO?=
 =?utf-8?B?MWJ3WG9hZEZtNjk0UnUzT3h4ZjZQdGpldlZ4cnk1NEpNU3RwSStjRHhhS1pP?=
 =?utf-8?B?Y0o5Y0RBTnMwRXgxVDdTN2d3SVRXQkhPM2YvSzJzSFVPZFJWK3l1WGFrZHZV?=
 =?utf-8?B?U3VtckNvVi9oUlZkRmJhL2I1T2xpUGw2VU1ZYWFVclMxU3pBZENNelpvVmxn?=
 =?utf-8?B?TUNtRVg1NXdZZVBvejZNWU91WndyeUM2UXY5MmdiSlBIR25qakltaURZSzNz?=
 =?utf-8?B?U0JKSHZLc05TRXFSRkNNbnEyMmhrT0dUL0Fyd01pQXVKWXhCeWJKUWkrOGk1?=
 =?utf-8?B?L1crdFZ6S0F4cmdaKzlDS3F2eVFxd25uckxhNzF1bXIzZXo5UmpkaVB0V282?=
 =?utf-8?B?YmJzUTNHQ0c1VlA5VnRIdlpZMUtrcG5ZUXhNTzVqTm1WdlNtSThkQ2JId2RR?=
 =?utf-8?B?U1dIWWdvcGJ4M1UyVWw1cDVSWEZZUDZDRDl1QUxyc2ozakZQVzVkQkNmbGxI?=
 =?utf-8?B?ZDZvV2g5RlRwT0tmZFl3OFZ4R3BRelc0WDhrUDc0Y1AvaytaK05zVHp2dWY5?=
 =?utf-8?B?U3FocWpZMERONUxaNUJTQjg1TnZlbGIvamV3Wi9MWFh2M3M1TWV3bHpHbW43?=
 =?utf-8?B?Vmc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: f38f76fc-e1bb-45c9-aac7-08db26b1955b
X-MS-Exchange-CrossTenant-AuthSource: CO1PR11MB4820.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Mar 2023 06:33:56.1655
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: gmp5TLJefBBN8nVEAD/NWNJwuUj3D+xg9mR1CmiDelEsMVbSTsrl+o79CRshlulodtDVlFxoig4tbGMprndy8A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR11MB5416
X-OriginatorOrg: intel.com
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


On 3/17/2023 11:44 AM, Matthew Wilcox wrote:
> On Fri, Mar 17, 2023 at 09:58:17AM +0800, Yin, Fengwei wrote:
>>
>>
>> On 3/17/2023 1:52 AM, Matthew Wilcox wrote:
>>> On Thu, Mar 16, 2023 at 04:38:58PM +0000, Ryan Roberts wrote:
>>>> On 16/03/2023 16:23, Yin, Fengwei wrote:
>>>>>> I think you are changing behavior here - is this intentional? Previously this
>>>>>> would be evaluated per page, now its evaluated once for the whole range. The
>>>>>> intention below is that directly faulted pages are mapped young and prefaulted
>>>>>> pages are mapped old. But now a whole range will be mapped the same.
>>>>>
>>>>> Yes. You are right here.
>>>>>
>>>>> Look at the prefault and cpu_has_hw_af for ARM64, it looks like we
>>>>> can avoid to handle vmf->address == addr specially. It's OK to 
>>>>> drop prefault and change the logic here a little bit to:
>>>>>   if (arch_wants_old_prefaulted_pte())
>>>>>       entry = pte_mkold(entry);
>>>>>   else
>>>>>       entry = pte_sw_mkyong(entry);
>>>>>
>>>>> It's not necessary to use pte_sw_mkyong for vmf->address == addr
>>>>> because HW will set the ACCESS bit in page table entry.
>>>>>
>>>>> Add Will Deacon in case I missed something here. Thanks.
>>>>
>>>> I'll defer to Will's response, but not all arm HW supports HW access flag
>>>> management. In that case it's done by SW, so I would imagine that by setting
>>>> this to old initially, we will get a second fault to set the access bit, which
>>>> will slow things down. I wonder if you will need to split this into (up to) 3
>>>> calls to set_ptes()?
>>>
>>> I don't think we should do that.  The limited information I have from
>>> various microarchitectures is that the PTEs must differ only in their
>>> PFN bits in order to use larger TLB entries.  That includes the Accessed
>>> bit (or equivalent).  So we should mkyoung all the PTEs in the same
>>> folio, at least initially.
>>>
>>> That said, we should still do this conditionally.  We'll prefault some
>>> other folios too.  So I think this should be:
>>>
>>>         bool prefault = (addr > vmf->address) || ((addr + nr) < vmf->address);
>>>
>> According to commit 46bdb4277f98e70d0c91f4289897ade533fe9e80, if hardware access
>> flag is supported on ARM64, there is benefit if prefault PTEs is set as "old".
>> If we change prefault like above, the PTEs is set as "yong" which loose benefit
>> on ARM64 with hardware access flag.
>>
>> ITOH, if from "old" to "yong" is cheap, why not leave all PTEs of folio as "old"
>> and let hardware to update it to "yong"?
> 
> Because we're tracking the entire folio as a single entity.  So we're
> better off avoiding the extra pagefaults to update the accessed bit,
> which won't actually give us any information (vmscan needs to know "were
> any of the accessed bits set", not "how many of them were set").
There is no extra pagefaults to update the accessed bit. There are three cases here:
1. hardware support access flag and cheap from "old" to "yong" without extra fault
2. hardware support access flag and expensive from "old" to "yong" without extra fault
3. no hardware support access flag (extra pagefaults from "old" to "yong". Expensive)

For #2 and #3, it's expensive from "old" to "yong", so we always set PTEs "yong" in
page fault.
For #1, It's cheap from "old" to "yong", so it's OK to set PTEs "old" in page fault.
And hardware will set it to "yong" when access memory. Actually, ARM64 with hardware
access bit requires to set PTEs "old".

> 
> Anyway, hopefully Ryan can test this and let us know if it fixes the
> regression he sees.
I highly suspect the regression Ryan saw is not related with this but another my
stupid work. I will send out the testing patch soon. Thanks.


Regards
Yin, Fengwei