Received: by 2002:a25:868d:0:0:0:0:0 with SMTP id z13csp1994186ybk; Thu, 21 May 2020 22:21:19 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzxPZKTN3z5CiK9dI6/hYKm+NfZiYUFH6BHij7E/D4J0bOAiVggar5ltbh3dS/5+6CXnyeR X-Received: by 2002:a17:906:70d:: with SMTP id y13mr6228229ejb.99.1590124879671; Thu, 21 May 2020 22:21:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1590124879; cv=none; d=google.com; s=arc-20160816; b=gvAPG9L1Zxh4o5Wj+glLem2ZYUZxNp1HWgCJ6BAbUUDUDJi+13wCK2wNZEKiO4Trz8 EGuGEplhWOILzj1bkSeeODzdfdaEK04dhqFPY+uTbQHfRtxgqgG+wvhI1BC9Mluj5UaN Dlp3gwSl/Cj3OdbkBdJqv+/d9z975Uu4o3nboxt2PZKNPKOV06AMRuSqWxrwUZtydgRh 7vQl2+zgUkFqASuKAqspVsZZ5NGlDsLQUBvhqrm2ZNGeo13t7PCXw074EEdlW0hAX/AV tiVMGY7tgmaEtEOFUVcJTqYM/LE2CfB5Lshm9gxod4Tqohid2ou98RG2LRRqXeiQaDe8 V04Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:dkim-signature:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=nsPsKapZXn+OxTjoLo2YVB0tASsozoFnU20LRvrrKZw=; b=uhR6RelZisANpRI9u/bMLb9qusHtC64hQNZrNH7N2Mbj0hHwzKNaD7czQEaWp0Vro0 tArx0Czgof4BMKFtpPyFWUKCnT02LkVOe7+u19ZGwpB30ADitrsFMCwTqLfUPWnuBoTC 3gdPDf2qwQl+uNp3alJVBTn/v5wXCoGtrB8otJpYxVOn89CgLp3a6iZ5anNbzICqQzHZ UW0J5Y+5PFazz/I78SvfpZd9XiK/oBmSdHXtKwXTkmaPWBZDDlxn3MNxzdHgYIk3F/im 9EyVxosII5GiWtOtYfKnIESBleGig15lqP6TzznBkT/Q05WtUGzvFxEbm5Y/iYwBcfhU 5SEQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@nvidia.com header.s=n1 header.b=T1RyAMgg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id 94si4104331edl.161.2020.05.21.22.20.57; Thu, 21 May 2020 22:21:19 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@nvidia.com header.s=n1 header.b=T1RyAMgg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728203AbgEVFTe (ORCPT + 99 others); Fri, 22 May 2020 01:19:34 -0400 Received: from hqnvemgate25.nvidia.com ([216.228.121.64]:14933 "EHLO hqnvemgate25.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726338AbgEVFTd (ORCPT ); Fri, 22 May 2020 01:19:33 -0400 Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate25.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Thu, 21 May 2020 22:18:13 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Thu, 21 May 2020 22:19:33 -0700 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Thu, 21 May 2020 22:19:33 -0700 Received: from HQMAIL111.nvidia.com (172.20.187.18) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Fri, 22 May 2020 05:19:32 +0000 Received: from hqnvemgw03.nvidia.com (10.124.88.68) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Fri, 22 May 2020 05:19:32 +0000 Received: from sandstorm.nvidia.com (Not Verified[10.2.48.182]) by hqnvemgw03.nvidia.com with Trustwave SEG (v7,5,8,10121) id ; Thu, 21 May 2020 22:19:32 -0700 From: John Hubbard To: Andrew Morton CC: Souptick Joarder , Matthew Wilcox , Jani Nikula , Joonas Lahtinen , Rodrigo Vivi , David Airlie , Daniel Vetter , Chris Wilson , Tvrtko Ursulin , Matthew Auld , , , LKML , , John Hubbard Subject: [PATCH v2 2/4] mm/gup: refactor and de-duplicate gup_fast() code Date: Thu, 21 May 2020 22:19:29 -0700 Message-ID: <20200522051931.54191-3-jhubbard@nvidia.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20200522051931.54191-1-jhubbard@nvidia.com> References: <20200522051931.54191-1-jhubbard@nvidia.com> MIME-Version: 1.0 X-NVConfidentiality: public Content-Transfer-Encoding: quoted-printable Content-Type: text/plain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1590124693; bh=nsPsKapZXn+OxTjoLo2YVB0tASsozoFnU20LRvrrKZw=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:MIME-Version:X-NVConfidentiality: Content-Transfer-Encoding:Content-Type; b=T1RyAMggc45BKtYidiW62E4esSR6VNCnKwFHHSOzOhQxSXgnITiPOPtvw74pAWd3Y DXJCuDTOlB0OoKwqdmmRMFTwP7WlcmN+ACt8a1olwx1e6KOwCGZS4C2NsyPceRqxls Fo2tN1tz6gIwmlmFw/JtjhzfcilkSrolCDvt13eUp4ScRrxh9hUbOtU/VM2QQFC/t0 7d1NNi71+TYPnofSt/ZF2Yi34nDJ93/YhfPRJzjgFRw4zmtWuPlBdXWJhGXy98bxNz Mn1oxPCtQ8E11mzkPsRohafWbmqsp+WWLFqZw82/d8qFjv30+i0a9CXW8sfC37CsZ+ N2lvSYwtCQMUw== Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org There were two nearly identical sets of code for gup_fast() style of walking the page tables with interrupts disabled. This has lead to the usual maintenance problems that arise from having duplicated code. There is already a core internal routine in gup.c for gup_fast(), so just enhance it very slightly: allow skipping the fall-back to "slow" (regular) get_user_pages(), via the new FOLL_FAST_ONLY flag. Then, just call internal_get_user_pages_fast() from __get_user_pages_fast(), and adjust the API to match pre-existing API behavior. There is a change in behavior from this refactoring: the nested form of interrupt disabling is used in all gup_fast() variants now. That's because there is only one place that interrupt disabling for page walking is done, and so the safer form is required. This should, if anything, eliminate possible (rare) bugs, because the non-nested form of enabling interrupts was fragile at best. Signed-off-by: John Hubbard --- include/linux/mm.h | 1 + mm/gup.c | 63 ++++++++++++++++++++++------------------------ 2 files changed, 31 insertions(+), 33 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a5594ac9ebe3..84b601cab699 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2782,6 +2782,7 @@ struct page *follow_page(struct vm_area_struct *vma, = unsigned long address, #define FOLL_LONGTERM 0x10000 /* mapping lifetime is indefinite: see below= */ #define FOLL_SPLIT_PMD 0x20000 /* split huge pmd before returning */ #define FOLL_PIN 0x40000 /* pages must be released via unpin_user_page */ +#define FOLL_FAST_ONLY 0x80000 /* gup_fast: prevent fall-back to slow gup = */ =20 /* * FOLL_PIN and FOLL_LONGTERM may be used in various combinations with eac= h diff --git a/mm/gup.c b/mm/gup.c index 4502846d57f9..4564b0dc7d0b 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2694,10 +2694,12 @@ static int internal_get_user_pages_fast(unsigned lo= ng start, int nr_pages, struct page **pages) { unsigned long addr, len, end; + unsigned long flags; int nr_pinned =3D 0, ret =3D 0; =20 if (WARN_ON_ONCE(gup_flags & ~(FOLL_WRITE | FOLL_LONGTERM | - FOLL_FORCE | FOLL_PIN | FOLL_GET))) + FOLL_FORCE | FOLL_PIN | FOLL_GET | + FOLL_FAST_ONLY))) return -EINVAL; =20 start =3D untagged_addr(start) & PAGE_MASK; @@ -2710,15 +2712,26 @@ static int internal_get_user_pages_fast(unsigned lo= ng start, int nr_pages, if (unlikely(!access_ok((void __user *)start, len))) return -EFAULT; =20 + /* + * Disable interrupts. The nested form is used, in order to allow full, + * general purpose use of this routine. + * + * With interrupts disabled, we block page table pages from being + * freed from under us. See struct mmu_table_batch comments in + * include/asm-generic/tlb.h for more details. + * + * We do not adopt an rcu_read_lock(.) here as we also want to + * block IPIs that come from THPs splitting. + */ if (IS_ENABLED(CONFIG_HAVE_FAST_GUP) && gup_fast_permitted(start, end)) { - local_irq_disable(); + local_irq_save(flags); gup_pgd_range(addr, end, gup_flags, pages, &nr_pinned); - local_irq_enable(); + local_irq_restore(flags); ret =3D nr_pinned; } =20 - if (nr_pinned < nr_pages) { + if (nr_pinned < nr_pages && !(gup_flags & FOLL_FAST_ONLY)) { /* Try to get the remaining pages with get_user_pages */ start +=3D nr_pinned << PAGE_SHIFT; pages +=3D nr_pinned; @@ -2750,45 +2763,29 @@ static int internal_get_user_pages_fast(unsigned lo= ng start, int nr_pages, int __get_user_pages_fast(unsigned long start, int nr_pages, int write, struct page **pages) { - unsigned long len, end; - unsigned long flags; - int nr_pinned =3D 0; + int nr_pinned; /* * Internally (within mm/gup.c), gup fast variants must set FOLL_GET, * because gup fast is always a "pin with a +1 page refcount" request. + * + * FOLL_FAST_ONLY is required in order to match the API description of + * this routine: no fall back to regular ("slow") GUP. */ - unsigned int gup_flags =3D FOLL_GET; + unsigned int gup_flags =3D FOLL_GET | FOLL_FAST_ONLY; =20 if (write) gup_flags |=3D FOLL_WRITE; =20 - start =3D untagged_addr(start) & PAGE_MASK; - len =3D (unsigned long) nr_pages << PAGE_SHIFT; - end =3D start + len; - - if (end <=3D start) - return 0; - if (unlikely(!access_ok((void __user *)start, len))) - return 0; - + nr_pinned =3D internal_get_user_pages_fast(start, nr_pages, gup_flags, + pages); /* - * Disable interrupts. We use the nested form as we can already have - * interrupts disabled by get_futex_key. - * - * With interrupts disabled, we block page table pages from being - * freed from under us. See struct mmu_table_batch comments in - * include/asm-generic/tlb.h for more details. - * - * We do not adopt an rcu_read_lock(.) here as we also want to - * block IPIs that come from THPs splitting. + * As specified in the API description above, this routine is not + * allowed to return negative values. However, the common core + * routine internal_get_user_pages_fast() *can* return -errno. + * Therefore, correct for that here: */ - - if (IS_ENABLED(CONFIG_HAVE_FAST_GUP) && - gup_fast_permitted(start, end)) { - local_irq_save(flags); - gup_pgd_range(start, end, gup_flags, pages, &nr_pinned); - local_irq_restore(flags); - } + if (nr_pinned < 0) + nr_pinned =3D 0; =20 return nr_pinned; } --=20 2.26.2