Message-ID: <7423472b-a6d4-4b37-0953-24b11aba2fad@oracle.com>
Date:   Wed, 24 Nov 2021 11:05:09 -0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.3.0
Subject: Re: [PATCH 2/3] mm: migrate: Correct the hugetlb migration stats
Content-Language: en-US
To:     Baolin Wang <baolin.wang@linux.alibaba.com>,
        Andrew Morton <akpm@linux-foundation.org>
Cc:     ziy@nvidia.com, shy828301@gmail.com, linux-mm@kvack.org,
        linux-kernel@vger.kernel.org
References: <cover.1636275127.git.baolin.wang@linux.alibaba.com>
 <71a4b6c22f208728fe8c78ad26375436c4ff9704.1636275127.git.baolin.wang@linux.alibaba.com>
 <20211115202146.473fff2404d7fb200dd48bd3@linux-foundation.org>
 <71816b8f-93e5-5a2a-e616-d52a1c4d354c@linux.alibaba.com>
 <3e6dcac6-c947-5f94-cd94-b59a8247dbcf@oracle.com>
 <d51d8f1a-ebda-e45a-9dd5-e5cca707ccdc@linux.alibaba.com>
From:   Mike Kravetz <mike.kravetz@oracle.com>
In-Reply-To: <d51d8f1a-ebda-e45a-9dd5-e5cca707ccdc@linux.alibaba.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MnZMUGNkL01UQ2xybGNSbkJpUitrbzh3T29yNXl6YkNuMW45WXJRWDUwUG40?=
 =?utf-8?B?ZW84OUtPVXQvdFVMZjI4cjFKYnBYUENwVWZTd2M2YUR4cEZPSHUzWmhIdzls?=
 =?utf-8?B?WHZQUUxsMnlxZW5jY1Vrc2VRY0tjMExKZ3psaTljSkxNTTU0VVF3cUU2SDZV?=
 =?utf-8?B?OFZYc2QxMmV1TTRyZ3BwYy9KcG44WkkrcUw3Q2FabXhQaFU0TC9rVnYwRzho?=
 =?utf-8?B?R3g3Qkhic1prcllGczEwbGI2VHZjVWdpU3hDajJhM0V5ZXdoRVAwc3huc2E3?=
 =?utf-8?B?YnZzMlYyOVVNdStXTlExc0JkTHZIdXhZQzJ1U255ZGd3RFFKY045NzA1UUpL?=
 =?utf-8?B?VXJwRFJzYkc0d2Jzdm1TcVZmUEIyWHNvc2ZhcU5yT2NlZHgydXBBWVFsMnlF?=
 =?utf-8?B?RGRmZ0hCM0dYaTVDRGtNeDZPOFh1YUtRbS9BMElkY1J1NjduQ3lwK1NwS2hP?=
 =?utf-8?B?NUljdkNZVWt6UXRvaDN0SmhqalUyU1o5dTFmeUxwSzI4aUUyaHhNSlhBRmF6?=
 =?utf-8?B?OHRJZkIxZVRkWUd5bW14UUI4aUt3eUZiWitRVGhUQTJZaEdxcGF5eUhtbTNk?=
 =?utf-8?B?UjJFbENpMGhnaExoa3Uza29SS1BVTmtoazJZMW1vRUFTdVpxQ2s5WW01MFdP?=
 =?utf-8?B?dGVnTFZZemlqWTkvV1VWZFNuNWpwdTVWWWczY0J4WE5UWW8vZndWNzRjOWg3?=
 =?utf-8?B?TWsxcGFPWjdyOE03eXBwUDkzWE1XSThFazMxaklPUC81bUQwVmdBWmxOLzJY?=
 =?utf-8?B?QWJyV2M3eU90R1hwQmUweXRSVVZmczJ4Y2hZQ1pnWkNwa3BpbnAyN3U3UVlX?=
 =?utf-8?B?QUNKSHZ0TlZVY1VVYmJZYXNtZDJ0TWV0c2ZKWERwckhHbzdJTkE0WWU0enZX?=
 =?utf-8?B?NUZ2RU1yMDZCVTBrY0Ira1hIV21aMGVoZTBtTVFmbWx3a0FlZm5JNEk0UWc4?=
 =?utf-8?B?amlUS3NRZlFiQ3JFTlltNmc3TmhBMGRPcW1wSkpJdXVkVHRNSjR2dmFubzIx?=
 =?utf-8?B?UndkekxMclBUUDQ1NWswclVtVnh3aW5VNGFtTW1ZRFlZUkVqY3FpcjRwUVZZ?=
 =?utf-8?B?dlhqREE5OVhvdmpLWStJb2tpS2RjK2pvNXFHQkhqblhtaTZWQ3NqOTU1STFq?=
 =?utf-8?B?aWZFS29PNkl0dFpNajhNRjNRM2NuOEdlbjk3NVBRekhYenE4WCt1S2pMV2RP?=
 =?utf-8?B?Y1g3djQrNTh4WmdtMml5bFJDRURFM0ZyaFFyZi9tdk9Xcm8vQ0RhYklGbDdS?=
 =?utf-8?B?ZkUyMllQRkduMUl4aWFBUGd1akZrQzkxdGk0anBOdVdpb3EzcWtjNjJRaDU0?=
 =?utf-8?B?cE55WmUxeHJiek5td1dQeWZiQmM1cU1QL3VDUnNMVlVIRW45bi9meW9Kcjd4?=
 =?utf-8?B?MVNWc2hQWmFUMGZYM3REOW5Ybk5XR0YvSkFiblJydkJqYW1FWW15dUhJWERS?=
 =?utf-8?B?N29JMVJhRmNWcFhlN2cyOG5RWU85VVZuZUVuNjJWQlYvTFhBZGs1V25XVlFq?=
 =?utf-8?B?TnN5N3N2YlNVM3RCbjZVNkh5N1hLNUVLUEpqMUVnTC9ORTBRSlNJL1Q4RVR0?=
 =?utf-8?B?OHlhVkxOQzFZejluemdES2NoWUxnR2pUNUNVcWNwaVBQY2E2NHVrQllQOTZx?=
 =?utf-8?B?RDBoYzVHcitEamlyK2Q0MWl4V2xRdE83UDVzNXp6VStlSllVZ3hCTXJTS2Ro?=
 =?utf-8?B?bStxZkt0d3BGSzFhY0dEVU1KSnh6NU9Gc0drM0kzK095Qnl2V0VlSElodXds?=
 =?utf-8?B?QXJvRmdJelFFTEdHcHdkd2h6SnZQcjFjWUtoSUtDV3FpM0loVDl6a2s3UmlL?=
 =?utf-8?B?b3JxTmxXY3NYVmhwOHJscmg5Q3luejZiRTYwcU1ZM2IvMTFTSU8ybXVoY3FS?=
 =?utf-8?B?VEprQkxaWnBxblhyRHhZMWNRY2hQdWp5bHBjb2JWb25vczJCZGI0dng3Q0FG?=
 =?utf-8?B?UUJCb0N5SHRycTNSN0p6SHgwNStES1I5YXp2MlpKUm4yclYremlMQ3RlQWI4?=
 =?utf-8?B?amZaNm5FUk1PRmp1VnRjdlRJcFp3a253N0NmTmYvVDhIdUFDOTRwZXIyRnE1?=
 =?utf-8?B?Y3V5L09vWmNERkFkY2lVT2s0UUZaQTVRd3Z1RGJxZCtnNFJCQkcyelBtU1hh?=
 =?utf-8?B?Q1ltY3NXcWRZQ21MMmprZllvUVVTZzFzQ3U3Y3Frb29BZWhEcG42MEdyNkFD?=
 =?utf-8?Q?be7JA98GXOBposUEjHvH8Sw=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: cc6a0ee3-d9e5-42e5-472f-08d9af7d5712
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Nov 2021 19:05:11.7667
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: jYSyyl/g//PmYTjOc/iTgesf9SMHdwwPBLLAkeTJD/NbmTKpcDWqvy94oXz0LQTcT1jqKiC9kbLYQUzIHXsp5w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB3906
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10178 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxscore=0 spamscore=0
 mlxlogscore=999 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000
 definitions=main-2111240099
X-Proofpoint-ORIG-GUID: feFzlZ3luvJdZM9AMmtSf2lef_6kTjW0
X-Proofpoint-GUID: feFzlZ3luvJdZM9AMmtSf2lef_6kTjW0
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 11/24/21 02:47, Baolin Wang wrote:
> 
> 
> On 2021/11/24 3:25, Mike Kravetz wrote:
>> On 11/15/21 22:03, Baolin Wang wrote:
>>> On 2021/11/16 12:21, Andrew Morton wrote:
>>>> On Sun,  7 Nov 2021 16:57:26 +0800 Baolin Wang <baolin.wang@linux.alibaba.com> wrote:
>>>>
>> I 'think' this is OK since the behavior is not really defined today.  But, we
>> are changing user visible output.
> 
> Actually we did not change the user visible output for a hugetlb migration. Since we still return the number of hugetlb failed to migrate as before (though previous hugetlb behavior is not reasonable), not the number of hguetlb subpages. We just correct the hugetlb migration stats for the hugetlb in kernel, like PGMIGRATE_SUCCESS/FAIL stats.
> 

Yes, the values returned by move_pages() will not change.

The 'stats' in the kernel which are changing are user visible.  Specifically.
the fields pgmigrate_success and pgmigrate_fail in the file /proc/vmstat.
The values reported there for migrated hugetlb pages is changing as a result
of this series.

In addition, if someone monitors the trace point at the end of migrate_pages
they will start seeing different values.

As mentioned, these values are not currently documented for hugetlb pages so
I think it is OK to change.  If someone thinks otherwise, please speak up!

Making them be similar to what is reported for THP pages would be a good
thing.

>>
>> Perhaps we should go ahead and document the hugetlb behavior when making these
>> changes?
> 
> Sure. How about adding below modification for hugetlb?

Yes, please do make the below changes as well.
-- 
Mike Kravetz

> diff --git a/Documentation/vm/page_migration.rst b/Documentation/vm/page_migration.rst
> index 08810f5..8c5cb81 100644
> --- a/Documentation/vm/page_migration.rst
> +++ b/Documentation/vm/page_migration.rst
> @@ -263,15 +263,15 @@ Monitoring Migration
>  The following events (counters) can be used to monitor page migration.
> 
>  1. PGMIGRATE_SUCCESS: Normal page migration success. Each count means that a
> -   page was migrated. If the page was a non-THP page, then this counter is
> -   increased by one. If the page was a THP, then this counter is increased by
> -   the number of THP subpages. For example, migration of a single 2MB THP that
> -   has 4KB-size base pages (subpages) will cause this counter to increase by
> -   512.
> +   page was migrated. If the page was a non-THP and non-hugetlb page, then
> +   this counter is increased by one. If the page was a THP or hugetlb, then
> +   this counter is increased by the number of THP or hugetlb subpages.
> +   For example, migration of a single 2MB THP that has 4KB-size base pages
> +   (subpages) will cause this counter to increase by 512.
> 
>  2. PGMIGRATE_FAIL: Normal page migration failure. Same counting rules as for
>     PGMIGRATE_SUCCESS, above: this will be increased by the number of subpages,
> -   if it was a THP.
> +   if it was a THP or hugetlb.
> 
>  3. THP_MIGRATION_SUCCESS: A THP was migrated without being split.