Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7;
Date:   Thu, 21 Sep 2023 15:42:01 -0700
From:   Mike Kravetz <mike.kravetz@oracle.com>
To:     riel@surriel.com
Cc:     linux-kernel@vger.kernel.org, kernel-team@meta.com,
        linux-mm@kvack.org, akpm@linux-foundation.org,
        muchun.song@linux.dev, leit@meta.com
Subject: Re: [PATCH 1/2] hugetlbfs: extend hugetlb_vma_lock to private VMAs
Message-ID: <20230921224201.GB21193@monkey>
References: <20230920021811.3095089-1-riel@surriel.com>
 <20230920021811.3095089-2-riel@surriel.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20230920021811.3095089-2-riel@surriel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?nZXChnYaLlh+nL0GyviPTu9GoDNrDZUsYu2ypKnzaGoBKq8hKHhnoinqeywQ?=
 =?us-ascii?Q?KcAkkBYifVGlZMWr/+Tli21Q845ki5TTxHZeVM3TEc1C9FCw3CPXpWU9inAo?=
 =?us-ascii?Q?yNJ5Blg9nxWhzVBQYQw0u/PMjz6BjgjlS8qzkPlPCMycODO9EE6RkYlxdw+d?=
 =?us-ascii?Q?uwrTJAYmIWe/64GzwzK3B8Ei9Tt+LMaX2dA7vHoUY1JlZCuCaHIBHvSQNBQK?=
 =?us-ascii?Q?feFKmuYB98oxyTSu2rdbZkkRWXKFpLV6kfKBtXdqQ7BoPbW/J8jPeFgrJvoN?=
 =?us-ascii?Q?x7twiJldIczae76i2IY7ympQgH48DxRcWy129TII2+hdWcinixVB/B83/S1M?=
 =?us-ascii?Q?pHXZlSd+sCcFlrw483AVt+gPAYUki3aF+6erUNa/YqscI1egkJSfF/sWIzYG?=
 =?us-ascii?Q?cTQqHjikkIQJzwUs50Qu+0oWNSC6MfrO1Ve1trkL3shXZ6IXRrp8AjGngjzZ?=
 =?us-ascii?Q?ufC7d2cFZedcxzIUVk1eAtMYmRBmQRdDL7ekPcbm0ypw0pKE5F6MJpM4ex+/?=
 =?us-ascii?Q?NieqdiK1Btv8gA/yxvc0QuMhRJZBd8RwHFJ5onvaqzdgWgXwpJe7Jzpj+wq9?=
 =?us-ascii?Q?IexWuFEWnhQRKfS/t3l8ZvRxqAEXekLviLSMrCEPtkYoCZM1ogzyjflovs6q?=
 =?us-ascii?Q?hw5dki3fNOOs35sVJJDhD42TNnWvk4n15g9DD8po/Vn483qAp6/XaBmXfctB?=
 =?us-ascii?Q?3y69uJHQunE/EPLBB+PzMpCL0UXaGQkPwiOAKxD1fS9d/M4nTH65KASnM+Kl?=
 =?us-ascii?Q?JRYfNJVAuk5O3dN12AmMBbHvksqiMlKSgzhxNVpGsTOcCA/ZxQ2AHQWHnkVK?=
 =?us-ascii?Q?k9izsJnbasp+DoYyRx5H43pdKFtzALf925d8EtmJey9GYSlehMPe7o2swigB?=
 =?us-ascii?Q?iElwIIm1CL6WOdDYJRxi75hwEAtmHz4OkQbf6eJREvxS+T2oHBJCv36ORbhP?=
 =?us-ascii?Q?nyfY+Iw8cpWHeBdlyFNeggGz8S4oZ3gycmlzwsYvJG0z8z459/hvHhfHbUnd?=
 =?us-ascii?Q?11HWertnlo0hfkad07LOeBMb2s30HtN3Amvl7nMmwVdSCZSqti35fUWyFsLm?=
 =?us-ascii?Q?QnRULedsg7PxvPAJGgBZRERgsNXXOCUFY3gk5esoN8LoGjZHygJDq4Fjp/RW?=
 =?us-ascii?Q?njyw4wXSSUoh49eEu/1f68K3XFlNtuEBH9CPx71VMt/f3AC06yyqjXGT4+w3?=
 =?us-ascii?Q?F19Pzt1VsButVfYJpCzs2ji7CtqZMlqjaz2/r9tF+kPNLPcOa5DywWCMvgLo?=
 =?us-ascii?Q?xJWC5LIgw/R7qvQ6BRIvbeghMSePW5xfdta4a4UKqUAAMiinFtXcXLOkY8CH?=
 =?us-ascii?Q?r62zRaDcU8rBx9I1H6KezpP8ZqN3gAlgbx5FS/yc86HorrX0MBJFejwZ01gY?=
 =?us-ascii?Q?+WOf9I8TIN6iLvh3eUrI/Ie9AQwzhi4jAGdOWkC6H1R0mU3Ff3L1tfQx95bO?=
 =?us-ascii?Q?3P/2RocB/xCumk8M9BQPHteSjTyIUktGjnMzJgWVVqoAsd+p8I/DndPbxDl8?=
 =?us-ascii?Q?4+6W6dyZkAajRqnRWjLP0eyhhi6I2ao3h47jKsV7xtgCi7XhZxJPUsdLSfEf?=
 =?us-ascii?Q?4dASJTg7O8numArGkfdh72D+ClCH7w3JvNHZUuqzYtyi00z4oyGUlk0j5D/w?=
 =?us-ascii?Q?Mw=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: q70Ebb6nNAg+wP78MSE7bWou4DgqRgOoiD1A7n54fTPHiDd7sbEeVpkXv0XJVaK3zj+7krGxavMo8OYkM1wW/Yso05hi4wQyBkOD72eXFfRdISvz2/UR0Ce1In2uiowpnCW8Kk1cIKcRWy+/nFn5QrxgyFJB8v52C5R3eXcgl7ruufCFRz9MCoJXXK4NQmKSSd7pWVUzlgX69F8Nzdf8/u5jF3hIhsJHmoKKFzcnesgKAKmwk4bLyzo4v94/A5j5d2Mg9kgZJnaOGl1UCRWtZUod2LTYHDwM8LzlOgRG0PvD/GU9R1N+bMC4O+2R7MkYu2jBjgWLazCFYpfQo/KdN4eyuVBp/TUoNtb9B+G34rWEvNiyl4N81DwE0tW3wGEEp+W1jxnrRk3zs08KNmjqzICYQUU1ZMUyeF5Z3uxi4Ho2L6l9DxTffkdzPx7kCNcaDdxI2tUAKtyJaVHOidYoDvBV1QHCDG2+VBW/gArcqZNQBUF0e54eac+gAHwK5zE52s8anyOwx5eZ8EySK7bj2LP8YCZx1kg26J6nklbxjDTxwuqVaYtJPV+Tyf5sV6P6ddvhDivAI7hBPnil0/SIOO/jGkror/hzAWMSk225HJpLXsP42KPn6r0sCczPtkbuSn9uzy7d/CXVDsEL2Rm56zhdD02ogavye9VLODT5wfxT/0F0RDMx2fToptQNmBOfTn1jLlz0xf85v5CuHTUswmXo3eJJR4Sc1wiEBzX8fbpZimTLQbjJBX/JfGfh2hLni/XXoG+r40OtMdUTy6ZixJXguKDkT1aV2Cs4gpLGNqFZH1NXp7eegOPw/rExUuXw0cj6rA38zA76T3If4SVhiA/9QmKWz2FsngHKKhcIZsG+rtzeq+aXkX1Nv4k18WgdxzizHZggaEXaqokXJUR7cuEOp+4UQTFzDZGhYsp8s+g=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: af657992-6c80-4a78-1066-08dbbaf3f97c
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Sep 2023 22:42:03.1467
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: WIPnQQ2CDfQS/NI/WVhRQI6DCGK0HH6evOGDWykLzZmERDLigsMEEfJgYg0XxvwWwVB6pTbTHWvW5Qvu7IjFMg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR10MB6801
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-09-21_19,2023-09-21_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 malwarescore=0 spamscore=0
 suspectscore=0 mlxlogscore=547 phishscore=0 bulkscore=0 mlxscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000
 definitions=main-2309210196
X-Proofpoint-ORIG-GUID: ASZXSnr9ihbqI_kWMu3eT0Vb5XXlGYOq
X-Proofpoint-GUID: ASZXSnr9ihbqI_kWMu3eT0Vb5XXlGYOq
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Thu, 21 Sep 2023 15:42:53 -0700 (PDT)

On 09/19/23 22:16, riel@surriel.com wrote:
> From: Rik van Riel <riel@surriel.com>
> 
> Extend the locking scheme used to protect shared hugetlb mappings
> from truncate vs page fault races, in order to protect private
> hugetlb mappings (with resv_map) against MADV_DONTNEED.
> 
> Add a read-write semaphore to the resv_map data structure, and
> use that from the hugetlb_vma_(un)lock_* functions, in preparation
> for closing the race between MADV_DONTNEED and page faults.
> 
> Signed-off-by: Rik van Riel <riel@surriel.com>
> ---
>  include/linux/hugetlb.h |  6 ++++++
>  mm/hugetlb.c            | 36 ++++++++++++++++++++++++++++++++----
>  2 files changed, 38 insertions(+), 4 deletions(-)

This looks straight forward.

However, I ran just this patch through libhugetlbfs test suite and it hung on
misaligned_offset (2M: 32).
https://github.com/libhugetlbfs/libhugetlbfs/blob/master/tests/misaligned_offset.c

Added lock/semaphore debugging to the kernel and got:
[   38.094690] =========================
[   38.095517] WARNING: held lock freed!
[   38.096350] 6.6.0-rc2-next-20230921-dirty #4 Not tainted
[   38.097556] -------------------------
[   38.098439] mlock/1002 is freeing memory ffff8881eff8dc00-ffff8881eff8ddff, with a lock still held there!
[   38.100550] ffff8881eff8dce8 (&resv_map->rw_sema){++++}-{3:3}, at: __unmap_hugepage_range_final+0x29/0x120
[   38.103564] 2 locks held by mlock/1002:
[   38.104552]  #0: ffff8881effa42a0 (&mm->mmap_lock){++++}-{3:3}, at: do_vmi_align_munmap+0x5c6/0x650
[   38.106611]  #1: ffff8881eff8dce8 (&resv_map->rw_sema){++++}-{3:3}, at: __unmap_hugepage_range_final+0x29/0x120
[   38.108827] 
[   38.108827] stack backtrace:
[   38.109929] CPU: 0 PID: 1002 Comm: mlock Not tainted 6.6.0-rc2-next-20230921-dirty #4
[   38.111812] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-1.fc37 04/01/2014
[   38.113784] Call Trace:
[   38.114456]  <TASK>
[   38.115066]  dump_stack_lvl+0x57/0x90
[   38.116001]  debug_check_no_locks_freed+0x137/0x170
[   38.117193]  ? remove_vma+0x28/0x70
[   38.118088]  __kmem_cache_free+0x8f/0x2b0
[   38.119080]  remove_vma+0x28/0x70
[   38.119960]  do_vmi_align_munmap+0x3b1/0x650
[   38.121051]  do_vmi_munmap+0xc9/0x1a0
[   38.122006]  __vm_munmap+0xa4/0x190
[   38.122931]  __ia32_sys_munmap+0x15/0x20
[   38.123926]  __do_fast_syscall_32+0x68/0x100
[   38.125031]  do_fast_syscall_32+0x2f/0x70
[   38.126060]  entry_SYSENTER_compat_after_hwframe+0x7b/0x8d
[   38.127366] RIP: 0023:0xf7f05579
[   38.128198] Code: b8 01 10 06 03 74 b4 01 10 07 03 74 b0 01 10 08 03 74 d8 01 00 00 00 00 00 00 00 00 00 00 00 00 00 51 52 55 89 e5 0f 34 cd 80 <5d> 5a 59 c3 90 90 90 90 8d b4 26 00 00 00 00 8d b4 26 00 00 00 00
[   38.132534] RSP: 002b:00000000fffa877c EFLAGS: 00000286 ORIG_RAX: 000000000000005b
[   38.135703] RAX: ffffffffffffffda RBX: 00000000f7a00000 RCX: 0000000000200000
[   38.137323] RDX: 00000000f7a00000 RSI: 0000000000200000 RDI: 0000000000000003
[   38.138965] RBP: 0000000000000002 R08: 0000000000000000 R09: 0000000000000000
[   38.140574] R10: 0000000000000000 R11: 0000000000000286 R12: 0000000000000000
[   38.142191] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[   38.143865]  </TASK>

Something is not quite right.  If you do not get to it first, I will take a
look as time permits.
-- 
Mike Kravetz