Message-ID: <53ce6e47-109d-af25-25fc-151ddf512287@oracle.com>
Date:   Mon, 22 Nov 2021 19:30:21 -0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.3.0
Subject: Re: [PATCH v8] hugetlb: Add hugetlb.*.numa_stat file
Content-Language: en-US
To:     Mina Almasry <almasrymina@google.com>, Tejun Heo <tj@kernel.org>,
        Zefan Li <lizefan.x@bytedance.com>,
        Johannes Weiner <hannes@cmpxchg.org>,
        Jonathan Corbet <corbet@lwn.net>,
        Andrew Morton <akpm@linux-foundation.org>
Cc:     Shuah Khan <shuah@kernel.org>, Miaohe Lin <linmiaohe@huawei.com>,
        Oscar Salvador <osalvador@suse.de>,
        Michal Hocko <mhocko@suse.com>,
        Muchun Song <songmuchun@bytedance.com>,
        David Rientjes <rientjes@google.com>,
        Shakeel Butt <shakeelb@google.com>, Jue Wang <juew@google.com>,
        Yang Yao <ygyao@google.com>, Joanna Li <joannali@google.com>,
        Cannon Matthews <cannonmatthews@google.com>,
        linux-mm@kvack.org, linux-kernel@vger.kernel.org,
        cgroups@vger.kernel.org, linux-doc@vger.kernel.org
References: <20211123001020.4083653-1-almasrymina@google.com>
From:   Mike Kravetz <mike.kravetz@oracle.com>
In-Reply-To: <20211123001020.4083653-1-almasrymina@google.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?TnFJSjdRU1FqR2ptUkdoVW10Ly9WbUZIOG02dTd1aUlvWmZ4MnU0aGloRngw?=
 =?utf-8?B?NWMzU2JuMkhGNk1YOS9pVzRWNHZwVkRwT0pGRm9JRllwV2phdGwrY2JveGVV?=
 =?utf-8?B?R1FqSHhJZDNOakZ2WTlGbGxYU2luRi9lOXlCdDZFT2N2OTA2ZTRKU0FWTnZx?=
 =?utf-8?B?MnB3RHArbkpMZUszbUs0eXdTNm9xa2xyU0c0VkZPR3RFV3p2WkFKZTVFOGVX?=
 =?utf-8?B?ZXlEa1FBbWxqdmVZVi91bXZld1NvcU1pZ2F0aFF0dzhwVGFYaG9BQ2I1VkQ0?=
 =?utf-8?B?RkJHdGJPOHVxaVM0NmozWGkyQWE1K0srb3hOZDJtL1ZTREtyS1hrU1pDRzUy?=
 =?utf-8?B?SVduL1RobjdGOFZ3Tm1jZGpMcTlFM0RYMWZTaVBZMFAwRHJrRG5URDl2cWQ3?=
 =?utf-8?B?dS93U0dKcXBRd3hwcnl6N2dacWJ2UjFaYmNpRkNiYW1MeklkQjZPeTZTS2pj?=
 =?utf-8?B?U0hneHNxSTg1c2dwRnpUUjVRbTFVQSthK1Zyd1R2VThqVW9GRUVtQjY0OUFM?=
 =?utf-8?B?TkphMUowZlkxU3gxN0pVaDQxQ29yUVRlMlBOOFJQQWlFd2E5K3dzV1NUZXdq?=
 =?utf-8?B?WFlQTHcyNldaNHRITmxKUlQ0WDBaUjdKai9XVno0WTZSRUtLVElLZjZkMWQ3?=
 =?utf-8?B?Wlg0MG1INE1UbmpIV0hDaUxiQjBlaDBhNWtnTjQ3aXFkc1ZwQmR3VEpxN3M2?=
 =?utf-8?B?NTdhbk9jMGkxSnRreWFVdVNyZGNEWGFNVWJlN0x0QmdtVWxxVG0wYzNaM1dU?=
 =?utf-8?B?U3loQmVFUHlTV1p0QlMzZTI0ZUZ1N21ybmJYM0p6NmNHa0dlWG00WW84Zjcw?=
 =?utf-8?B?US9OUWY0ZHlyTlJPY3M0T014ZTdtbjZvZU9rVGg2MXhRRHNmQ0srbnpTQTl2?=
 =?utf-8?B?TTNpUDN0S1VKQTdSQXYvUUl4c05ySC81NFhqN0o0dEVuQVFFMHJzVWRMNjRz?=
 =?utf-8?B?UDhUZ09OWGdtQXhuNU43bDhqeTh3WG9oY3AxN0lPK2NEMkhNL3RON0FsWHFB?=
 =?utf-8?B?dlZ0dzJFS0JzaTErMHptZTZseHQvcUV2SmpsODAzanp1VWhwd0J0SnRhd252?=
 =?utf-8?B?QzRIdlVHTm9MbDg2ekJmcTlKSFFZNXZsY2ZnR21mMFo1azhEY3BsMVhDTklx?=
 =?utf-8?B?NW12ZUJEVEFWK2tKL01XMFM2bTRGUGR2L21wVnJCelc5eG8xanptK295ZUZB?=
 =?utf-8?B?WnN4Y1VSWGdzQzFEV29SMUxkRGk3enJCdFhSQVljbTVuRXR4SnBSMG9NbFhK?=
 =?utf-8?B?Q2tzRGtKb0Ezb0tXM3VURjN3cTdGVk1pTUdIdjZUdWtjcDlQWng4UGtVYzZ2?=
 =?utf-8?B?bTZ3ekdPWjkvanRoMmxOUFF5Q2dndk5vKzNIRmtmenp5WEFPTXptTHhiRlVy?=
 =?utf-8?B?RjArRi9TdEw0dCtnb1g5SU13akk0WkZQQXR3bWFsaWNuczI5YzJZUWx6Y2tG?=
 =?utf-8?B?ZFhHQWxXQ1ljSzBURG1ac3krNW5va0ZwYVBIN2tPaXMvZUtXcWJ3VWpSY3hp?=
 =?utf-8?B?VGp2S2d6bzAzYTNGYTZlay9JbWVYLzFRTHU3TmJkZi9IcVFSOFFWZ3M0Ky9Q?=
 =?utf-8?B?WEdQelhMSjFPdDYwVnFQN0pvQWQrT3JzVk9vcENHZzFmbFg3SGFEMFRGUW1Q?=
 =?utf-8?B?Zis3YXBBSXcxd0JaMmVFWGtuaUlKQU9WWTl2dTNCVFB6TW56K20reDNSRjV4?=
 =?utf-8?B?QkdaRno5U2kwRWtjdjBtVnlrdWQ4bmpSMUQrQW9zSkR6SjlwWElFNXJaUVdV?=
 =?utf-8?B?YmI1bUV1S2JLWFp0cWdoUjFBOUkxTFp2OFpzd3E2QTZCbkRqQ2Z0S1hrbTJV?=
 =?utf-8?B?elgvTXFZdlZRMmN0YVNNR0w3RmhYMkl2ZHNOVk5MY3JTTlVDai9xYmp6VWMw?=
 =?utf-8?B?ZW1CZmJNZXMrL1M5QnZGU08zSGZoaVhESUNYRWl2RlZWRy9QSTRiVFUzT3Ra?=
 =?utf-8?B?dzEvSW5ad2lwMUxneEFGcnhtblNZdkxmUzU5ZWdLZ0t5ZDFBOHE0cVA4MEZZ?=
 =?utf-8?B?dm9CRVlGN2V6SC9HNEs5S3VHSGZ2SHZkWTIyb1A3V0JaY3lOVzBGMWFTN1Fp?=
 =?utf-8?B?T0FzZnFIanlmOFV6VXE4Zzh0bk5zUGVONHAydHBxbXA5TWNMTnRhWVJQUnF0?=
 =?utf-8?B?M2lBUzNBWUwxWFJsYkx5S2hma0Y0a2l5WnRRSjA4b1lQdythZkNobEtOdkts?=
 =?utf-8?Q?2N4KZRsrMACkuQWwWNxcWHI=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2a4ffa34-0afe-46b7-f4e9-08d9ae319639
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Nov 2021 03:30:24.9703
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: JhjlGtlgXTKSvq+H1lUknAlCjlXRXS1YHlBe+wQtbf8dQe9DHVSKJeRFmvTLcwC2nyYCqJR+1WtedXPyxaKSFw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4671
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10176 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxscore=0 spamscore=0
 mlxlogscore=999 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000
 definitions=main-2111230015
X-Proofpoint-GUID: xD8mbVFgRkLLjCzlZL78hd234Mv_fctq
X-Proofpoint-ORIG-GUID: xD8mbVFgRkLLjCzlZL78hd234Mv_fctq
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 11/22/21 16:10, Mina Almasry wrote:
> For hugetlb backed jobs/VMs it's critical to understand the numa
> information for the memory backing these jobs to deliver optimal
> performance.
> 
> Currently this technically can be queried from /proc/self/numa_maps, but
> there are significant issues with that. Namely:
> 1. Memory can be mapped or unmapped.
> 2. numa_maps are per process and need to be aggregated across all
>    processes in the cgroup. For shared memory this is more involved as
>    the userspace needs to make sure it doesn't double count shared
>    mappings.
> 3. I believe querying numa_maps needs to hold the mmap_lock which adds
>    to the contention on this lock.
> 
> For these reasons I propose simply adding hugetlb.*.numa_stat file,
> which shows the numa information of the cgroup similarly to
> memory.numa_stat.
> 
> On cgroup-v2:
>    cat /sys/fs/cgroup/unified/test/hugetlb.2MB.numa_stat
>    total=2097152 N0=2097152 N1=0
> 
> On cgroup-v1:
>    cat /sys/fs/cgroup/hugetlb/test/hugetlb.2MB.numa_stat
>    total=2097152 N0=2097152 N1=0
>    hierarichal_total=2097152 N0=2097152 N1=0
> 
> This patch was tested manually by allocating hugetlb memory and querying
> the hugetlb.*.numa_stat file of the cgroup and its parents.
> ￼
> Cc: Mike Kravetz <mike.kravetz@oracle.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Shuah Khan <shuah@kernel.org>
> Cc: Miaohe Lin <linmiaohe@huawei.com>
> Cc: Oscar Salvador <osalvador@suse.de>
> Cc: Michal Hocko <mhocko@suse.com>
> Cc: Muchun Song <songmuchun@bytedance.com>
> Cc: David Rientjes <rientjes@google.com>
> Cc: Shakeel Butt <shakeelb@google.com>
> Cc: Jue Wang <juew@google.com>
> Cc: Yang Yao <ygyao@google.com>
> Cc: Joanna Li <joannali@google.com>
> Cc: Cannon Matthews <cannonmatthews@google.com>
> Cc: linux-mm@kvack.org
> Cc: linux-kernel@vger.kernel.org
> 
> Signed-off-by: Mina Almasry <almasrymina@google.com>
> 

Thanks for continuing to update, this version looks good.

Reviewed-by: Mike Kravetz <mike.kravetz@oracle.com>
-- 
Mike Kravetz