Received-SPF: pass (google.com: domain of linux-kernel+bounces-31657-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
From: Prakash Sangappa <prakash.sangappa@oracle.com>
To: linux-mm@kvack.org, linux-kernel@vger.kernel.org
Cc: muchun.song@linux.dev, mike.kravetz@oracle.com, akpm@linux-foundation.org
Subject: [PATCH] Hugetlb pages should not be reserved by shmat() if SHM_NORESERVE
Date: Fri, 19 Jan 2024 17:17:52 -0800
Message-Id: <1705713472-3537-1-git-send-email-prakash.sangappa@oracle.com>
Precedence: bulk

For shared memory of type SHM_HUGETLB, hugetlb pages are reserved in
shmget() call. If SHM_NORESERVE flags is specified then the hugetlb
pages are not reserved. However when the shared memory is attached
with the shmat() call the hugetlb pages are getting reserved incorrectly
for SHM_HUGETLB shared memory created with SHM_NORESERVE.

Ensure that the hugetlb pages are no reserved for SHM_HUGETLB shared
memory in the shmat() call.

Signed-off-by: Prakash Sangappa <prakash.sangappa@oracle.com>
---
 fs/hugetlbfs/inode.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index f757d4f..93cafd2 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -141,7 +141,13 @@ static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma)
 	file_accessed(file);
 
 	ret = -ENOMEM;
-	if (!hugetlb_reserve_pages(inode,
+
+	/*
+	 * for SHM_HUGETLB, the pages are reserved in the shmget() call so skip
+	 * reserving here. Note only for SHM hugetlbfs file, the inode
+	 * flag S_PRIVATE is set.
+	 */
+	if (!(inode->i_flags & S_PRIVATE) && !hugetlb_reserve_pages(inode,
 				vma->vm_pgoff >> huge_page_order(h),
 				len >> huge_page_shift(h), vma,
 				vma->vm_flags))
-- 
2.7.4