Received: by 2002:ab2:6991:0:b0:1f7:f6c3:9cb1 with SMTP id v17csp716876lqo; Wed, 8 May 2024 12:31:51 -0700 (PDT) X-Forwarded-Encrypted: i=3; AJvYcCUFxJ/BKqpaB38Vw/7PJq6+KHjFEnehDdc1XaIohDNsfT3qzxBA99FS2TvoExL3d2hZqC5M3kfyHReviqKk/AKIVLXmAPZ0BmvqwCmPiA== X-Google-Smtp-Source: AGHT+IH9Hpo2zsMBv46iAi7wQmpz0lqRcT06BSW+AIgIkDjZhaL3R0sjDB7B7HBIku9F9Xk7hpJB X-Received: by 2002:a17:903:484:b0:1e8:26e4:d084 with SMTP id d9443c01a7336-1eeb088d1f1mr32450935ad.60.1715196711595; Wed, 08 May 2024 12:31:51 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1715196711; cv=pass; d=google.com; s=arc-20160816; b=DuDVZD0qGjHZUkd46PVhbf34tmss23f3rtRPDlB5MA3m8P0y15ooK0xkt6pErTdqYF kCcpW5vCL1C919g4co4ZIfCw28qk28hVjvhk4urAsVRpgpAmoxisnBOnin1VNAPTvVZA DSP969OfqeGNQuqvaID5r+JkHrpXIFqA3uSk6aHP+Pw7i6NlcVtkfW4hORwXiJkQS5La qYd3GFZRmkgTFdGROUdAsMGc/oHWoBIFfYd9G1JKxCUJyjEE+EOYS7Ylo3UVbBs0K0F8 mP9wP6S+gQiaRHhu7j2sELm1IF9M8eMulpAfDdmSOTbPqpSe1LHmqEbEC/Pkp7gM796/ 4JLA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=K1s8quCXzhpwj2tdUrwB8sMaHFPnndxHFKfGkIDXtWo=; fh=vVStF6QKOd5VOEUo0JnLaMOmwP1zNmZVKkuk1m9bO+M=; b=WYFthcKpxI+Jeait6hWhhnXHr57IEPpEaVlA+xKqf5hY0g1qA41kQpt3KEnsQHtfd5 GNZsZPpH/8CdB1tUGYK8OJm6LC/jeAUAyVrJqCDVeI4C+txZPDqLvz2oIgRNVaJc795w 6Q7FUDoN9DziLXBCtg4gneL7EVTT8kDig8Iy/ZkS7d2J9fY/saaMDzJpl30SLeupLEJS xBcQZ/hTHobcmGHf3F82kUHZ1klg4t+KF50ORbYE/DkJqNFJxKe3BcTW5chU/H+stEsI j4c2Tr/kVpj4Ie8bDKploIXIe4lsAsRXboOi2/phCQo67mvdDsYZFvaM7ybrH3Gay0w/ tnow==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=ZVRcCf0A; arc=pass (i=1 spf=pass spfdomain=rivosinc.com dkim=pass dkdomain=rivosinc-com.20230601.gappssmtp.com); spf=pass (google.com: domain of linux-kernel+bounces-173764-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-173764-linux.lists.archive=gmail.com@vger.kernel.org" Return-Path: Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id t11-20020a170902e1cb00b001e85dc64073si12268741pla.436.2024.05.08.12.31.51 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:31:51 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel+bounces-173764-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=ZVRcCf0A; arc=pass (i=1 spf=pass spfdomain=rivosinc.com dkim=pass dkdomain=rivosinc-com.20230601.gappssmtp.com); spf=pass (google.com: domain of linux-kernel+bounces-173764-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-173764-linux.lists.archive=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 78F8128877D for ; Wed, 8 May 2024 19:31:02 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 8E075130AC2; Wed, 8 May 2024 19:30:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="ZVRcCf0A" Received: from mail-lj1-f175.google.com (mail-lj1-f175.google.com [209.85.208.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 66F2612D75D for ; Wed, 8 May 2024 19:30:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.175 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715196653; cv=none; b=EBaV1KevF/HvwHnvwUeZUFuxh2UpQRdx4Fiir9TjJGoXnH7g0NewwCZoe+qQx+ovYihFbWKvfKnioTrzed8/YN+tRg3s+gCUpAEhjC0XTZ+Kk44k8KULQqkB71Nh0Rb8h5Edd5U2O2iwzCbB7QtldqKFgLp/Mgcpyj7LLBgiMok= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715196653; c=relaxed/simple; bh=X5FE53OY3eT3DtMIjRJ8IEAc0okRiwSAW9RSi2xWEnU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=kijqIngbcDHGJRB+3gkvawEcwizXLtfkFbJWEUtOrjhv1oZcGxWO99y8ST7xErmWF2Yp7Aq4Z61lVu7LFq3abYJCeUYcfkHZptV0D5o+QcZFp6VKX8Oafkg/pA/cVxvfm/D6uP8rKm9InHf39A3PPw0bNJe+M1zicWvKMEe4kpA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=ZVRcCf0A; arc=none smtp.client-ip=209.85.208.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-lj1-f175.google.com with SMTP id 38308e7fff4ca-2e3e18c240fso1536191fa.0 for ; Wed, 08 May 2024 12:30:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196649; x=1715801449; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=K1s8quCXzhpwj2tdUrwB8sMaHFPnndxHFKfGkIDXtWo=; b=ZVRcCf0A0JtRmQC5tZ3g4zKQr6HbvObjCzvt5zjAws+Ib9N2O08aoIkIQW6i2JyfFH HDneGWKxcmNtuxRwOhJRT7fovkmOS2ZFV0k28CvrXH0copGQ5e+IejRMW7yrqMvkSKjZ 1NTYTs+qot57B7T/ki/bgnp8HB/WjQIHaGLAJP+7zLC+seKMErnAamn3IVlro5QewyJ0 Sak9LIUjdee8kQAWccLgUs6jMyXSNKK2ePteSDUf0JhJJ4D0NhYJa4nKA+eMamCPMIxq K9s0KXy0gqieEyQraTU6W3M74GhVA5dd8Rzy/3vKk8hnDw2ZSb7FVYZwOAEwE5MusUi3 l+kg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196649; x=1715801449; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=K1s8quCXzhpwj2tdUrwB8sMaHFPnndxHFKfGkIDXtWo=; b=RqbSVtY5HMmjMCoMlLkQEQL8Iv7gjPPMXbv3pFnV3p1p3jtkjWDMCmN1iqernWM/hp X820ZM4RBrAB5lGmznqG0iY0yP3sRX8HqVTSIYE7Cbp2XdDDRQeFZoJfmk7OKHDLtueg 4w9z8HNcsELq5zmazJgcaB/uPP0iSv0YYK0BGCCxnAudEYLRW3v38doxSmKhF7IHao6S TOt5tiLxxp2KvaECtXsPD965XJK/yyC8q1tB7KTEB4JrLYtyGGFnRJbidVRCuhQVY4je kvc+OKvPtQpAqUmtY11kRG2aV7ad36h7lR/I9+qbq6WR/Yv/Zcgb6Q7XldowUdG8U4OK zd/w== X-Forwarded-Encrypted: i=1; AJvYcCXaRt7vMDmIPda1hLdydOmPgmZz06sV1aHrrGKN5nzqX+74LEhxXSnOh65lAufQQfVHjd9L5Cwj4OvZ8T/6Br7+vZo07rfdKiPL/0dr X-Gm-Message-State: AOJu0YzEFUCnIU/LwXUjuej6rjlf9yHpPYTuRe1w1hvJLJ5pPaYu/i2Y EVSzxDGZcODsvrR403atXXbXzF1hF0Ei/tEux0xbaEgDEJZfxiEj7I0sMhUcl50= X-Received: by 2002:a05:651c:1541:b0:2df:e192:47ec with SMTP id 38308e7fff4ca-2e447081ef2mr37890971fa.29.1715196648606; Wed, 08 May 2024 12:30:48 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id b15-20020a05600c4e0f00b0041aa79f27a0sm3273819wmq.38.2024.05.08.12.30.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:30:48 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 11/12] mm, riscv, arm64: Use common ptep_set_wrprotect()/wrprotect_ptes() functions Date: Wed, 8 May 2024 21:19:30 +0200 Message-Id: <20240508191931.46060-12-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Make riscv use the contpte aware ptep_set_wrprotect()/wrprotect_ptes() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 56 ++++++------------------ arch/arm64/mm/contpte.c | 18 -------- arch/riscv/include/asm/pgtable.h | 25 +++++++++-- include/linux/contpte.h | 2 + mm/contpte.c | 75 +++++++++++++++++++++++++++++++- 5 files changed, 110 insertions(+), 66 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 6591aab11c67..162efd9647dd 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1208,7 +1208,11 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline void ___ptep_set_wrprotect(struct mm_struct *mm, +/* + * __ptep_set_wrprotect - mark read-only while trasferring potential hardware + * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. + */ +static inline void __ptep_set_wrprotect(struct mm_struct *mm, unsigned long address, pte_t *ptep, pte_t pte) { @@ -1222,23 +1226,13 @@ static inline void ___ptep_set_wrprotect(struct mm_struct *mm, } while (pte_val(pte) != pte_val(old_pte)); } -/* - * __ptep_set_wrprotect - mark read-only while trasferring potential hardware - * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. - */ -static inline void __ptep_set_wrprotect(struct mm_struct *mm, - unsigned long address, pte_t *ptep) -{ - ___ptep_set_wrprotect(mm, address, ptep, __ptep_get(ptep)); -} - static inline void __wrprotect_ptes(struct mm_struct *mm, unsigned long address, pte_t *ptep, unsigned int nr) { unsigned int i; for (i = 0; i < nr; i++, address += PAGE_SIZE, ptep++) - __ptep_set_wrprotect(mm, address, ptep); + __ptep_set_wrprotect(mm, address, ptep, __ptep_get(ptep)); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -1246,7 +1240,7 @@ static inline void __wrprotect_ptes(struct mm_struct *mm, unsigned long address, static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long address, pmd_t *pmdp) { - __ptep_set_wrprotect(mm, address, (pte_t *)pmdp); + __ptep_set_wrprotect(mm, address, (pte_t *)pmdp, __ptep_get((pte_t *)pmdp)); } #define pmdp_establish pmdp_establish @@ -1389,8 +1383,6 @@ extern void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr, int full); -extern void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned int nr); #define pte_batch_hint pte_batch_hint static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) @@ -1478,35 +1470,12 @@ extern int ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); #define wrprotect_ptes wrprotect_ptes -static __always_inline void wrprotect_ptes(struct mm_struct *mm, - unsigned long addr, pte_t *ptep, unsigned int nr) -{ - if (likely(nr == 1)) { - /* - * Optimization: wrprotect_ptes() can only be called for present - * ptes so we only need to check contig bit as condition for - * unfold, and we can remove the contig bit from the pte we read - * to avoid re-reading. This speeds up fork() which is sensitive - * for order-0 folios. Equivalent to contpte_try_unfold(). - */ - pte_t orig_pte = __ptep_get(ptep); - - if (unlikely(pte_cont(orig_pte))) { - __contpte_try_unfold(mm, addr, ptep, orig_pte); - orig_pte = pte_mknoncont(orig_pte); - } - ___ptep_set_wrprotect(mm, addr, ptep, orig_pte); - } else { - contpte_wrprotect_ptes(mm, addr, ptep, nr); - } -} +extern void wrprotect_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, unsigned int nr); #define __HAVE_ARCH_PTEP_SET_WRPROTECT -static inline void ptep_set_wrprotect(struct mm_struct *mm, - unsigned long addr, pte_t *ptep) -{ - wrprotect_ptes(mm, addr, ptep, 1); -} +extern void ptep_set_wrprotect(struct mm_struct *mm, + unsigned long addr, pte_t *ptep); #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS extern int ptep_set_access_flags(struct vm_area_struct *vma, @@ -1528,7 +1497,8 @@ extern int ptep_set_access_flags(struct vm_area_struct *vma, #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH #define ptep_clear_flush_young __ptep_clear_flush_young #define __HAVE_ARCH_PTEP_SET_WRPROTECT -#define ptep_set_wrprotect __ptep_set_wrprotect +#define ptep_set_wrprotect(mm, addr, ptep) \ + __ptep_set_wrprotect(mm, addr, ptep, __ptep_get(ptep)) #define wrprotect_ptes __wrprotect_ptes #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 5675a61452ac..1cef93b15d6e 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -44,21 +44,3 @@ pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, return __get_and_clear_full_ptes(mm, addr, ptep, nr, full); } EXPORT_SYMBOL_GPL(contpte_get_and_clear_full_ptes); - -void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned int nr) -{ - /* - * If wrprotecting an entire contig range, we can avoid unfolding. Just - * set wrprotect and wait for the later mmu_gather flush to invalidate - * the tlb. Until the flush, the page may or may not be wrprotected. - * After the flush, it is guaranteed wrprotected. If it's a partial - * range though, we must unfold, because we can't have a case where - * CONT_PTE is set but wrprotect applies to a subset of the PTEs; this - * would cause it to continue to be unpredictable after the flush. - */ - - contpte_try_unfold_partial(mm, addr, ptep, nr); - __wrprotect_ptes(mm, addr, ptep, nr); -} -EXPORT_SYMBOL_GPL(contpte_wrprotect_ptes); diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index b151a5aa4de8..728f31da5e6a 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -755,11 +755,21 @@ static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, } static inline void __ptep_set_wrprotect(struct mm_struct *mm, - unsigned long address, pte_t *ptep) + unsigned long address, pte_t *ptep, + pte_t pte) { atomic_long_and(~(unsigned long)_PAGE_WRITE, (atomic_long_t *)ptep); } +static inline void __wrprotect_ptes(struct mm_struct *mm, unsigned long address, + pte_t *ptep, unsigned int nr) +{ + unsigned int i; + + for (i = 0; i < nr; i++, address += PAGE_SIZE, ptep++) + __ptep_set_wrprotect(mm, address, ptep, __ptep_get(ptep)); +} + static inline int __ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { @@ -807,6 +817,12 @@ extern int ptep_clear_flush_young(struct vm_area_struct *vma, extern int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty); +#define __HAVE_ARCH_PTEP_SET_WRPROTECT +extern void ptep_set_wrprotect(struct mm_struct *mm, + unsigned long addr, pte_t *ptep); +extern void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr); +#define wrprotect_ptes wrprotect_ptes #else /* CONFIG_THP_CONTPTE */ @@ -822,12 +838,13 @@ extern int ptep_set_access_flags(struct vm_area_struct *vma, #define ptep_clear_flush_young __ptep_clear_flush_young #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags +#define __HAVE_ARCH_PTEP_SET_WRPROTECT +#define ptep_set_wrprotect(mm, addr, ptep) \ + __ptep_set_wrprotect(mm, addr, ptep, __ptep_get(ptep)) +#define wrprotect_ptes __wrprotect_ptes #endif /* CONFIG_THP_CONTPTE */ -#define __HAVE_ARCH_PTEP_SET_WRPROTECT -#define ptep_set_wrprotect __ptep_set_wrprotect - #define pgprot_nx pgprot_nx static inline pgprot_t pgprot_nx(pgprot_t _prot) { diff --git a/include/linux/contpte.h b/include/linux/contpte.h index 76244b0c678a..d1439db1706c 100644 --- a/include/linux/contpte.h +++ b/include/linux/contpte.h @@ -26,5 +26,7 @@ int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty); +void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr); #endif /* _LINUX_CONTPTE_H */ diff --git a/mm/contpte.c b/mm/contpte.c index 9cbbff1f67ad..fe36b6b1d20a 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -49,6 +49,8 @@ * - ptep_get_and_clear() * - ptep_test_and_clear_young() * - ptep_clear_flush_young() + * - wrprotect_ptes() + * - ptep_set_wrprotect() */ pte_t huge_ptep_get(pte_t *ptep) @@ -266,7 +268,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, pte_t pte; if (!pte_cont(__ptep_get(ptep))) { - __ptep_set_wrprotect(mm, addr, ptep); + __ptep_set_wrprotect(mm, addr, ptep, __ptep_get(ptep)); return; } @@ -832,4 +834,75 @@ __always_inline int ptep_set_access_flags(struct vm_area_struct *vma, return contpte_ptep_set_access_flags(vma, addr, ptep, entry, dirty); } + +static void contpte_try_unfold_partial(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + /* + * Unfold any partially covered contpte block at the beginning and end + * of the range. + */ + size_t pgsize; + int ncontig; + + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, 0, &pgsize); + + if (ptep != arch_contpte_align_down(ptep) || nr < ncontig) + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + + if (ptep + nr != arch_contpte_align_down(ptep + nr)) { + unsigned long last_addr = addr + pgsize * (nr - 1); + pte_t *last_ptep = ptep + nr - 1; + + contpte_try_unfold(mm, last_addr, last_ptep, + __ptep_get(last_ptep)); + } +} + +void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + /* + * If wrprotecting an entire contig range, we can avoid unfolding. Just + * set wrprotect and wait for the later mmu_gather flush to invalidate + * the tlb. Until the flush, the page may or may not be wrprotected. + * After the flush, it is guaranteed wrprotected. If it's a partial + * range though, we must unfold, because we can't have a case where + * CONT_PTE is set but wrprotect applies to a subset of the PTEs; this + * would cause it to continue to be unpredictable after the flush. + */ + + contpte_try_unfold_partial(mm, addr, ptep, nr); + __wrprotect_ptes(mm, addr, ptep, nr); +} +EXPORT_SYMBOL_GPL(contpte_wrprotect_ptes); + +__always_inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + if (likely(nr == 1)) { + /* + * Optimization: wrprotect_ptes() can only be called for present + * ptes so we only need to check contig bit as condition for + * unfold, and we can remove the contig bit from the pte we read + * to avoid re-reading. This speeds up fork() which is sensitive + * for order-0 folios. Equivalent to contpte_try_unfold(). + */ + pte_t orig_pte = __ptep_get(ptep); + + if (unlikely(pte_cont(orig_pte))) { + __contpte_try_unfold(mm, addr, ptep, orig_pte); + orig_pte = pte_mknoncont(orig_pte); + } + __ptep_set_wrprotect(mm, addr, ptep, orig_pte); + } else { + contpte_wrprotect_ptes(mm, addr, ptep, nr); + } +} + +__always_inline void ptep_set_wrprotect(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + wrprotect_ptes(mm, addr, ptep, 1); +} #endif /* CONFIG_THP_CONTPTE */ -- 2.39.2