Received: by 2002:ac0:bc90:0:0:0:0:0 with SMTP id a16csp578031img; Fri, 22 Mar 2019 04:25:26 -0700 (PDT) X-Google-Smtp-Source: APXvYqzQ7HuE8BNT4u1g/G/hwYxPaA+xa5q0+zAq4mL1f04hqU8f8Qx+idsvIzoQtZZ1YtNymdyM X-Received: by 2002:a17:902:1003:: with SMTP id b3mr9107701pla.306.1553253926568; Fri, 22 Mar 2019 04:25:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1553253926; cv=none; d=google.com; s=arc-20160816; b=DA6CC6fNRxwKHQ5nc4KUtnXdXpAOI6iEGFOpHd0FqegBwpFcmKIkwDgdAEab6x98JC 9+8aopJ1Juyy+izSh3wVit/8p5wce+p6TizNkn5Ic7iAPGiaMGgh6pDXyRBkwY4DYYiw 95D2R/G6jBURQXOgXImku0iaE7+dAVUZ/v3ri4dkzV8+eoWSf0BcMxVkCj5IIsJbCyVt LphsabQQGOHjleQTSuGo/sUYmGTSXjJnqQq2qksSJ1izQQ8szv9X+IAe1weGU2wqIJN7 wyqUvra1fhn9JeaHn14ii0zBHfH+E9mDQlV392xECFpBsZSPKiRFgN44gVkjnr4u/XKU n6sQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=GWebFWjbDnOznkJGaJikm5peBecn6SJG8qes36IpR2Q=; b=pQxUOa6BZhPykbfgUKb2dRHoRYz3pQeUqat8JLxy3jhVLoFlFZz9Rlqs70eYMywXpb lM1cdysK4KJ4X+P4TsmLBwNbiu9pu9T8EskfCMGIaLvjWk+PMuG2Xnmaz1QHBlJj8VJE llNjHEDQGVM23c8ITZykMcmy67pW4p0xlVSsmog4MGuHiIrVXgl4JQuAc3KxgILEwcqZ B84eST6LkdYfsyHiCpcp5YeNpYFvnTGHiY3ouTLVRt5ED/fJhe5Oqnq9nNH7+FBjpbk/ 0qP4adYCHTzl11bn0AEiVtRiXLxxkpgQk8zMFuMZ/1R+xp9PqXl0w1TUwLi1uPmRgeRi EHWA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b="Uk6/ROtt"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id s26si6293254pfm.143.2019.03.22.04.25.08; Fri, 22 Mar 2019 04:25:26 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b="Uk6/ROtt"; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728745AbfCVLWL (ORCPT + 99 others); Fri, 22 Mar 2019 07:22:11 -0400 Received: from mail.kernel.org ([198.145.29.99]:48788 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728712AbfCVLWJ (ORCPT ); Fri, 22 Mar 2019 07:22:09 -0400 Received: from localhost (83-86-89-107.cable.dynamic.v4.ziggo.nl [83.86.89.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 6DCEA218D4; Fri, 22 Mar 2019 11:22:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1553253727; bh=0UP7fCFN9QuzhLy2SI7yfhU415sL94/Jvou3FGAQFRQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Uk6/ROttYUHqKUwmJGRfH0+ujWnD/LflNfY9ghJmC9pELhhgkEQdjJCD/0Pn/gnGs XobzvMMNadbZpAasKxAFd2iWqV0R0E1r8xKXOw/nvT2R7BqrvXBhe+Z19CO8ZgYkEy jMToS1vzisGWDH4/g0K9E4e6QESeOI/qp8+WSX8A= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Mike Kravetz , Naoya Horiguchi , Michal Hocko , Andrea Arcangeli , "Kirill A . Shutemov" , Mel Gorman , Davidlohr Bueso , Andrew Morton , Linus Torvalds Subject: [PATCH 3.18 053/134] hugetlbfs: fix races and page leaks during migration Date: Fri, 22 Mar 2019 12:14:26 +0100 Message-Id: <20190322111213.794733574@linuxfoundation.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190322111210.465931067@linuxfoundation.org> References: <20190322111210.465931067@linuxfoundation.org> User-Agent: quilt/0.65 X-stable: review X-Patchwork-Hint: ignore MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org 3.18-stable review patch. If anyone has any objections, please let me know. ------------------ From: Mike Kravetz commit cb6acd01e2e43fd8bad11155752b7699c3d0fb76 upstream. hugetlb pages should only be migrated if they are 'active'. The routines set/clear_page_huge_active() modify the active state of hugetlb pages. When a new hugetlb page is allocated at fault time, set_page_huge_active is called before the page is locked. Therefore, another thread could race and migrate the page while it is being added to page table by the fault code. This race is somewhat hard to trigger, but can be seen by strategically adding udelay to simulate worst case scheduling behavior. Depending on 'how' the code races, various BUG()s could be triggered. To address this issue, simply delay the set_page_huge_active call until after the page is successfully added to the page table. Hugetlb pages can also be leaked at migration time if the pages are associated with a file in an explicitly mounted hugetlbfs filesystem. For example, consider a two node system with 4GB worth of huge pages available. A program mmaps a 2G file in a hugetlbfs filesystem. It then migrates the pages associated with the file from one node to another. When the program exits, huge page counts are as follows: node0 1024 free_hugepages 1024 nr_hugepages node1 0 free_hugepages 1024 nr_hugepages Filesystem Size Used Avail Use% Mounted on nodev 4.0G 2.0G 2.0G 50% /var/opt/hugepool That is as expected. 2G of huge pages are taken from the free_hugepages counts, and 2G is the size of the file in the explicitly mounted filesystem. If the file is then removed, the counts become: node0 1024 free_hugepages 1024 nr_hugepages node1 1024 free_hugepages 1024 nr_hugepages Filesystem Size Used Avail Use% Mounted on nodev 4.0G 2.0G 2.0G 50% /var/opt/hugepool Note that the filesystem still shows 2G of pages used, while there actually are no huge pages in use. The only way to 'fix' the filesystem accounting is to unmount the filesystem If a hugetlb page is associated with an explicitly mounted filesystem, this information in contained in the page_private field. At migration time, this information is not preserved. To fix, simply transfer page_private from old to new page at migration time if necessary. There is a related race with removing a huge page from a file and migration. When a huge page is removed from the pagecache, the page_mapping() field is cleared, yet page_private remains set until the page is actually freed by free_huge_page(). A page could be migrated while in this state. However, since page_mapping() is not set the hugetlbfs specific routine to transfer page_private is not called and we leak the page count in the filesystem. To fix that, check for this condition before migrating a huge page. If the condition is detected, return EBUSY for the page. Link: http://lkml.kernel.org/r/74510272-7319-7372-9ea6-ec914734c179@oracle.com Link: http://lkml.kernel.org/r/20190212221400.3512-1-mike.kravetz@oracle.com Fixes: bcc54222309c ("mm: hugetlb: introduce page_huge_active") Signed-off-by: Mike Kravetz Reviewed-by: Naoya Horiguchi Cc: Michal Hocko Cc: Andrea Arcangeli Cc: "Kirill A . Shutemov" Cc: Mel Gorman Cc: Davidlohr Bueso Cc: [mike.kravetz@oracle.com: v2] Link: http://lkml.kernel.org/r/7534d322-d782-8ac6-1c8d-a8dc380eb3ab@oracle.com [mike.kravetz@oracle.com: update comment and changelog] Link: http://lkml.kernel.org/r/420bcfd6-158b-38e4-98da-26d0cd85bd01@oracle.com Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds Signed-off-by: Greg Kroah-Hartman --- fs/hugetlbfs/inode.c | 12 ++++++++++++ mm/hugetlb.c | 14 ++++++++++++-- mm/migrate.c | 11 +++++++++++ 3 files changed, 35 insertions(+), 2 deletions(-) --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -609,6 +609,18 @@ static int hugetlbfs_migrate_page(struct rc = migrate_huge_page_move_mapping(mapping, newpage, page); if (rc != MIGRATEPAGE_SUCCESS) return rc; + + /* + * page_private is subpool pointer in hugetlb pages. Transfer to + * new page. PagePrivate is not associated with page_private for + * hugetlb pages and can not be set here as only page_huge_active + * pages can be migrated. + */ + if (page_private(page)) { + set_page_private(newpage, page_private(page)); + set_page_private(page, 0); + } + migrate_page_copy(newpage, page); return MIGRATEPAGE_SUCCESS; --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2938,7 +2938,6 @@ retry_avoidcopy: copy_user_huge_page(new_page, old_page, address, vma, pages_per_huge_page(h)); __SetPageUptodate(new_page); - set_page_huge_active(new_page); mmun_start = address & huge_page_mask(h); mmun_end = mmun_start + huge_page_size(h); @@ -2959,6 +2958,7 @@ retry_avoidcopy: make_huge_pte(vma, new_page, 1)); page_remove_rmap(old_page); hugepage_add_new_anon_rmap(new_page, vma, address); + set_page_huge_active(new_page); /* Make the old page be freed below */ new_page = old_page; } @@ -3017,6 +3017,7 @@ static int hugetlb_no_page(struct mm_str struct page *page; pte_t new_pte; spinlock_t *ptl; + bool new_page = false; /* * Currently, we are forced to kill the process in the event the @@ -3050,7 +3051,7 @@ retry: } clear_huge_page(page, address, pages_per_huge_page(h)); __SetPageUptodate(page); - set_page_huge_active(page); + new_page = true; if (vma->vm_flags & VM_MAYSHARE) { int err; @@ -3126,6 +3127,15 @@ retry: } spin_unlock(ptl); + + /* + * Only make newly allocated pages active. Existing pages found + * in the pagecache could be !page_huge_active() if they have been + * isolated for migration. + */ + if (new_page) + set_page_huge_active(page); + unlock_page(page); out: return ret; --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1050,6 +1050,16 @@ static int unmap_and_move_huge_page(new_ lock_page(hpage); } + /* + * Check for pages which are in the process of being freed. Without + * page_mapping() set, hugetlbfs specific move page routine will not + * be called and we could leak usage counts for subpools. + */ + if (page_private(hpage) && !page_mapping(hpage)) { + rc = -EBUSY; + goto out_unlock; + } + if (PageAnon(hpage)) anon_vma = page_get_anon_vma(hpage); @@ -1067,6 +1077,7 @@ static int unmap_and_move_huge_page(new_ if (rc == MIGRATEPAGE_SUCCESS) hugetlb_cgroup_migrate(hpage, new_hpage); +out_unlock: unlock_page(hpage); out: if (rc != -EAGAIN)