Received: by 2002:a25:31c3:0:0:0:0:0 with SMTP id x186csp6856670ybx; Mon, 11 Nov 2019 16:13:07 -0800 (PST) X-Google-Smtp-Source: APXvYqx63RS5GV5/dd9Aoc3+pp0fB3t2Pq6J3ItKDbS2mETlv5F7D/O3VMQHRGY84SnYorNm0B0n X-Received: by 2002:a50:d717:: with SMTP id t23mr29253232edi.231.1573517587207; Mon, 11 Nov 2019 16:13:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1573517587; cv=none; d=google.com; s=arc-20160816; b=LY1llVPVZ/COyyojgfs91JdT+bFAJevWhqeAbULT1OUVv8+c6tFmk0TGgPh3PIhocB xxL9cpEw8CJzMrQEf43Yy9hPEMW37DZ7xWDGf/fXCtOqpmVLbHZAP8qw7pdzmoAgilXo J9/JHWHgvTbqjCrhu3PipYaPYOFSy23UjevPtWsL2NEpT/n0yi1IDSrX2XoW5xsc7DAr svUNV5foBR8Hg1eXKDz+t0JuC6lK/Q1VqxI93LWyWSP85obREK46b97H2V351CmbCdae Z38us63KOK5kezncyp8+4rAToLEO1A4a97q5/RLgHLXFvaZbBzqfIMbZ0Aw9ePop80bU MJoQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:dkim-signature:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=eh5PKTqFnv3H2C1oWk46Hm4jlzYBaiSiPR2xbAo8/Xk=; b=K3mt9WPQfbHBTSab6HSAA7Zb9SZLyqxocxxaXwJ09CztdubnBhug3IWxdeHcuq2siQ yycL2rX6hq4vOUlMfnDbm56jUC/lLqdWk5FeyGtkSOqr5xR52KpFmaUX7jQKi8NvLwaY YwLZcbQibP3eHxJMxWHVwv4y9pfAU+OCog3MER6Xi9/15RYBsKTYmdJKVwWwykF1XNdk +1Z+zQ+0RDuo1qssH3Mfnv/g1EbNqsNJ25fvIsIoE3mlvUhSJTgDz1coT6Bxznktc+I/ LiPNpz4Mbme4XJY8DmtzbyD9Mjwp5KQlj79FJ14FPVA4SGUjcHtda4GQ/5Cmbq8WxIcd 9Pmg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@nvidia.com header.s=n1 header.b=adfHMgrN; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id p20si9771311ejj.394.2019.11.11.16.12.43; Mon, 11 Nov 2019 16:13:07 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@nvidia.com header.s=n1 header.b=adfHMgrN; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727852AbfKLALZ (ORCPT + 99 others); Mon, 11 Nov 2019 19:11:25 -0500 Received: from hqemgate15.nvidia.com ([216.228.121.64]:16630 "EHLO hqemgate15.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726908AbfKLAHN (ORCPT ); Mon, 11 Nov 2019 19:07:13 -0500 Received: from hqpgpgate102.nvidia.com (Not Verified[216.228.121.13]) by hqemgate15.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Mon, 11 Nov 2019 16:06:07 -0800 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate102.nvidia.com (PGP Universal service); Mon, 11 Nov 2019 16:07:09 -0800 X-PGP-Universal: processed; by hqpgpgate102.nvidia.com on Mon, 11 Nov 2019 16:07:09 -0800 Received: from HQMAIL111.nvidia.com (172.20.187.18) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Tue, 12 Nov 2019 00:07:08 +0000 Received: from rnnvemgw01.nvidia.com (10.128.109.123) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Tue, 12 Nov 2019 00:07:07 +0000 Received: from blueforge.nvidia.com (Not Verified[10.110.48.28]) by rnnvemgw01.nvidia.com with Trustwave SEG (v7,5,8,10121) id ; Mon, 11 Nov 2019 16:07:07 -0800 From: John Hubbard To: Andrew Morton CC: Al Viro , Alex Williamson , Benjamin Herrenschmidt , =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= , Christoph Hellwig , Dan Williams , Daniel Vetter , Dave Chinner , David Airlie , "David S . Miller" , Ira Weiny , Jan Kara , Jason Gunthorpe , Jens Axboe , Jonathan Corbet , =?UTF-8?q?J=C3=A9r=C3=B4me=20Glisse?= , Magnus Karlsson , Mauro Carvalho Chehab , Michael Ellerman , Michal Hocko , Mike Kravetz , Paul Mackerras , Shuah Khan , Vlastimil Babka , , , , , , , , , , , , , LKML , John Hubbard Subject: [PATCH v3 04/23] mm: devmap: refactor 1-based refcounting for ZONE_DEVICE pages Date: Mon, 11 Nov 2019 16:06:41 -0800 Message-ID: <20191112000700.3455038-5-jhubbard@nvidia.com> X-Mailer: git-send-email 2.24.0 In-Reply-To: <20191112000700.3455038-1-jhubbard@nvidia.com> References: <20191112000700.3455038-1-jhubbard@nvidia.com> MIME-Version: 1.0 X-NVConfidentiality: public Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1573517167; bh=eh5PKTqFnv3H2C1oWk46Hm4jlzYBaiSiPR2xbAo8/Xk=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:MIME-Version:X-NVConfidentiality: Content-Type:Content-Transfer-Encoding; b=adfHMgrN5jjkYVKLpULgMMWgAAyNhWrnIEq2gG0krc84wIe5DqW2HFzzxJxJzHIrG Q4SMQIqJPunErE+KDabNclG7+yPRQM1rbHN9kjnmobU9luoWgogwM08Ig6xBrYzT+8 M1Otb8NEwlqrd2K6zNLVLcuzFS2o3KsS+mlQYJu4TNJt8AxTcIQk5KKm3GEA97K6Qs Gnl5NOdsaNBoy4dPOubgCBkbhuzz6aSRrdwtKi2YFqtByoOat5K4PpX4+mgTYRZFJz kwO8BxgUl0DLC6pXM0I8aGV+8wGdM57g3KJobQK8NnZciawV1jrn2TFZRg9Xau0MTU ONfrW7dMbEHRQ== Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org An upcoming patch changes and complicates the refcounting and especially the "put page" aspects of it. In order to keep everything clean, refactor the devmap page release routines: * Rename put_devmap_managed_page() to page_is_devmap_managed(), and limit the functionality to "read only": return a bool, with no side effects. * Add a new routine, put_devmap_managed_page(), to handle checking what kind of page it is, and what kind of refcount handling it requires. * Rename __put_devmap_managed_page() to free_devmap_managed_page(), and limit the functionality to unconditionally freeing a devmap page. This is originally based on a separate patch by Ira Weiny, which applied to an early version of the put_user_page() experiments. Since then, J=C3=A9r=C3=B4me Glisse suggested the refactoring described abo= ve. Suggested-by: J=C3=A9r=C3=B4me Glisse Signed-off-by: Ira Weiny Signed-off-by: John Hubbard --- include/linux/mm.h | 27 ++++++++++++++++--- mm/memremap.c | 67 ++++++++++++++++++++-------------------------- 2 files changed, 53 insertions(+), 41 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a2adf95b3f9c..96228376139c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -967,9 +967,10 @@ static inline bool is_zone_device_page(const struct pa= ge *page) #endif =20 #ifdef CONFIG_DEV_PAGEMAP_OPS -void __put_devmap_managed_page(struct page *page); +void free_devmap_managed_page(struct page *page); DECLARE_STATIC_KEY_FALSE(devmap_managed_key); -static inline bool put_devmap_managed_page(struct page *page) + +static inline bool page_is_devmap_managed(struct page *page) { if (!static_branch_unlikely(&devmap_managed_key)) return false; @@ -978,7 +979,6 @@ static inline bool put_devmap_managed_page(struct page = *page) switch (page->pgmap->type) { case MEMORY_DEVICE_PRIVATE: case MEMORY_DEVICE_FS_DAX: - __put_devmap_managed_page(page); return true; default: break; @@ -986,6 +986,27 @@ static inline bool put_devmap_managed_page(struct page= *page) return false; } =20 +static inline bool put_devmap_managed_page(struct page *page) +{ + bool is_devmap =3D page_is_devmap_managed(page); + + if (is_devmap) { + int count =3D page_ref_dec_return(page); + + /* + * devmap page refcounts are 1-based, rather than 0-based: if + * refcount is 1, then the page is free and the refcount is + * stable because nobody holds a reference on the page. + */ + if (count =3D=3D 1) + free_devmap_managed_page(page); + else if (!count) + __put_page(page); + } + + return is_devmap; +} + #else /* CONFIG_DEV_PAGEMAP_OPS */ static inline bool put_devmap_managed_page(struct page *page) { diff --git a/mm/memremap.c b/mm/memremap.c index 03ccbdfeb697..bc7e2a27d025 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -410,48 +410,39 @@ struct dev_pagemap *get_dev_pagemap(unsigned long pfn= , EXPORT_SYMBOL_GPL(get_dev_pagemap); =20 #ifdef CONFIG_DEV_PAGEMAP_OPS -void __put_devmap_managed_page(struct page *page) +void free_devmap_managed_page(struct page *page) { - int count =3D page_ref_dec_return(page); + /* Clear Active bit in case of parallel mark_page_accessed */ + __ClearPageActive(page); + __ClearPageWaiters(page); + + mem_cgroup_uncharge(page); =20 /* - * If refcount is 1 then page is freed and refcount is stable as nobody - * holds a reference on the page. + * When a device_private page is freed, the page->mapping field + * may still contain a (stale) mapping value. For example, the + * lower bits of page->mapping may still identify the page as + * an anonymous page. Ultimately, this entire field is just + * stale and wrong, and it will cause errors if not cleared. + * One example is: + * + * migrate_vma_pages() + * migrate_vma_insert_page() + * page_add_new_anon_rmap() + * __page_set_anon_rmap() + * ...checks page->mapping, via PageAnon(page) call, + * and incorrectly concludes that the page is an + * anonymous page. Therefore, it incorrectly, + * silently fails to set up the new anon rmap. + * + * For other types of ZONE_DEVICE pages, migration is either + * handled differently or not done at all, so there is no need + * to clear page->mapping. */ - if (count =3D=3D 1) { - /* Clear Active bit in case of parallel mark_page_accessed */ - __ClearPageActive(page); - __ClearPageWaiters(page); - - mem_cgroup_uncharge(page); - - /* - * When a device_private page is freed, the page->mapping field - * may still contain a (stale) mapping value. For example, the - * lower bits of page->mapping may still identify the page as - * an anonymous page. Ultimately, this entire field is just - * stale and wrong, and it will cause errors if not cleared. - * One example is: - * - * migrate_vma_pages() - * migrate_vma_insert_page() - * page_add_new_anon_rmap() - * __page_set_anon_rmap() - * ...checks page->mapping, via PageAnon(page) call, - * and incorrectly concludes that the page is an - * anonymous page. Therefore, it incorrectly, - * silently fails to set up the new anon rmap. - * - * For other types of ZONE_DEVICE pages, migration is either - * handled differently or not done at all, so there is no need - * to clear page->mapping. - */ - if (is_device_private_page(page)) - page->mapping =3D NULL; + if (is_device_private_page(page)) + page->mapping =3D NULL; =20 - page->pgmap->ops->page_free(page); - } else if (!count) - __put_page(page); + page->pgmap->ops->page_free(page); } -EXPORT_SYMBOL(__put_devmap_managed_page); +EXPORT_SYMBOL(free_devmap_managed_page); #endif /* CONFIG_DEV_PAGEMAP_OPS */ --=20 2.24.0