Received: by 2002:ab2:6991:0:b0:1f7:f6c3:9cb1 with SMTP id v17csp715932lqo; Wed, 8 May 2024 12:29:58 -0700 (PDT) X-Forwarded-Encrypted: i=3; AJvYcCV+F0TqqaLB/xXfWaXJCVQPBODTCVl6okKErbidRYEr4SKx5cz6X8WmVyhgSUy5/74BqSPBYorjcbywo5jbO5Fc8BHhzhGZztbfPpCjRQ== X-Google-Smtp-Source: AGHT+IFR1mtHi3aDOvD0/48xuxh/6D4KHQByOf1OEcXSiYOQBrCy6ev3TjFDvrq39cMC7hYLR66+ X-Received: by 2002:a17:906:ad88:b0:a59:a83b:d43d with SMTP id a640c23a62f3a-a59fb94baa1mr200523566b.16.1715196598390; Wed, 08 May 2024 12:29:58 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1715196598; cv=pass; d=google.com; s=arc-20160816; b=rDaJ5UhvnhJMpzlEIQZDAG6oQxWudewTd/44iyCg2sTZkGxvk0/PfTuJfgSjxsHvpn A+JGAkLSevmziyfBtR72TuixoEFl6r0EUgcK/E34ZTDxQWs8nORnrALCErrhYxekMxjL 2UZFpHbU4b3rCqVkAyroR6ih1EoSXhoCy5lQTRVQuN6R5S62sfI8i3LjLODCgl13pleY IdbM1uaIWO97otVSiH7IloLikHrAsY/3d1t9wEbYgMdU9bElUtvlNAria+vt+nzmuedV PwCjJTsiT41fRDLm2iE9GRGMI/gWr8GAqyI0A9STxR18dCEHjmabwveCqNX/Yf0hY0WT E2PQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=uckq1a6OF7H/9ROfgD12E/DRrLJPxYkFYbl5+hSyETI=; fh=99we/rHWZcdRcg47Jzcs9qG4MmJxFWotJB1wp1Mqwe8=; b=V86H7yeiIYw8sp+wYsSFTDIYiU7PbVl8VxAgi2BxYTya7QYuuBAaoRXB7qcLYpsPIq x44VhfQIqdAd2YXBjDdK2lapCW+dBzRUOpIol+u2vcM7XL5aeYSPZMQmPHGna7a7qdt7 xGxSrZ2Ou6fRUD1TrXvhgA8BGEVxKtj9z9zle/TFz/piJhiBvdFXD3b7soGmg2uKAXYH tbQCozAV+N4gZ+CC83g75gsCmu2DPoksOjJOoKqUfeWvUH2G5SnCRtFVjy4FEmqaweN3 44iNsIcn5C+vlDm7ftS6bmcMkdzFvBsQy52IhWDP22TW2jMjCXN5GTgpBVEroRszTuze SzWQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=aNY+A7nq; arc=pass (i=1 spf=pass spfdomain=rivosinc.com dkim=pass dkdomain=rivosinc-com.20230601.gappssmtp.com); spf=pass (google.com: domain of linux-kernel+bounces-173763-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-173763-linux.lists.archive=gmail.com@vger.kernel.org" Return-Path: Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [147.75.80.249]) by mx.google.com with ESMTPS id gb36-20020a170907962400b00a59d244d532si3615489ejc.642.2024.05.08.12.29.58 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:29:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel+bounces-173763-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249; Authentication-Results: mx.google.com; dkim=pass header.i=@rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=aNY+A7nq; arc=pass (i=1 spf=pass spfdomain=rivosinc.com dkim=pass dkdomain=rivosinc-com.20230601.gappssmtp.com); spf=pass (google.com: domain of linux-kernel+bounces-173763-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) smtp.mailfrom="linux-kernel+bounces-173763-linux.lists.archive=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id EC49B1F21B0E for ; Wed, 8 May 2024 19:29:57 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2CBE0130A4B; Wed, 8 May 2024 19:29:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="aNY+A7nq" Received: from mail-wm1-f53.google.com (mail-wm1-f53.google.com [209.85.128.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 272CAD534 for ; Wed, 8 May 2024 19:29:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.53 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715196591; cv=none; b=DIgjGZSYecxp0TJSMrqqmmXRpLTRGjo+MdCp6seoODdXiB0nrpSvL/F5+h4x0QVgUbU/rpKL3KaheoyF1RIZhnOhOihyPlmsqEDVNseG9AXHbbFoHFRdmvFm6L84LVzWDMmqybun0TFYtFZPlkiKTB+17933AvPI74EM91yWrSE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715196591; c=relaxed/simple; bh=gHarnxk3+TQDW2Of9KtkFTyUbjl3FMDQzz2bLlhHcjo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=fgRp/DJiIW/WUeYxRCUOgS6ofOtUW2OTjhhYLE2v1p9P6rbfb9yFEwlE02T3VPcLwwyWkTOc3uU38TbIId8qTnHi+fFW/CL9o0UvKjwJ42D1plU9UdVnZrHxOxgvyeemCJUYGVXIdoCMsypnglHq/ZnlQXj9Gb7kGtEKPCHMnjg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=aNY+A7nq; arc=none smtp.client-ip=209.85.128.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-wm1-f53.google.com with SMTP id 5b1f17b1804b1-41b794510cdso611845e9.2 for ; Wed, 08 May 2024 12:29:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196587; x=1715801387; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uckq1a6OF7H/9ROfgD12E/DRrLJPxYkFYbl5+hSyETI=; b=aNY+A7nqvoGdE8Zq3WOfOfLYZWlHlLYK88ZlsUU+DYmX5vtL4glONYUnD46AZSoiVH UYMHQ/PgKZzxXQ990RbM2HyNfpIo57AyYpbusjBLUKJE3ozks+TK0b6YEGUTn6fy7gzR 1q2vzmh93+FH3H2RqnQfPgjOm0DAZS3qJrLr+rh/F+/bh/zKPR3WjPFxo9Fcaf1DTvzl AkCLymgFRQ0JTl4D6g0YnMxEq5Tqgu6g83U+3kxuYgey83bz8DLoKHmbgW2J2OM1Hsqy IJEIguqyYO4qU9iu4I4uJnFDlAJvYUVl11FCBPBv67ittsZ9i5zDadcnB6hJfZkSEObo p1HQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196587; x=1715801387; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uckq1a6OF7H/9ROfgD12E/DRrLJPxYkFYbl5+hSyETI=; b=A/g7Ph3uTAh+e0xks5iG9ounrvTwoJ1m877HY3fWJ9YgPE86sGa3f1DLyjgC0+t03Y 8MbpsbQ3SOC8bKFoee64OETh53QKBTAGlQnlLF/HbmZMh0sQhWFP/WCRFmWXyWs31Yqc tgMLh5Xutmi9eN/IL7Kli6UTWGgJohP5tsMTFOhHuZC90qBklXULQPYkuBdJKTmVAcZq ungbxhufdahVSqx1ZujIlthWWo5uvDpN89KogOOIASEuBN+sbhZNJVJjvyCNzsbEZC07 jLujYmum936/lJoh6sDoL6n2n6+tFB03VDuJpa4PGhiQ9NYz5swp74xxXoDFNfBdKuUh a56A== X-Forwarded-Encrypted: i=1; AJvYcCWvTMXK/lzx3lv6edgBaFUtzTtiliEKegi5Mc9I1Ae3SKk+qkDSQUznvUO1/ClIL9Kzy3xYGHhP+cwRDIZP7RuNQPlQEDo9lxXoehPQ X-Gm-Message-State: AOJu0YxxsHUMcEMYuKYqZLci4MUv3rJM/YkNp/ykVRgzv+vWx3ohMbM2 szxMB24wu+VyhW2gCyaIeiCQ14WfTwGs6Zw53tnxLLZ0qC1W87dkbHEwtDONJ6I= X-Received: by 2002:a05:600c:4e93:b0:41b:fc3a:f1ef with SMTP id 5b1f17b1804b1-41f71acca18mr25217385e9.33.1715196587426; Wed, 08 May 2024 12:29:47 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id d16-20020a05600c34d000b00419f572671dsm3314921wmq.20.2024.05.08.12.29.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:29:47 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 10/12] mm, riscv, arm64: Use common ptep_set_access_flags() function Date: Wed, 8 May 2024 21:19:29 +0200 Message-Id: <20240508191931.46060-11-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Make riscv use the contpte aware ptep_set_access_flags() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 19 ++-------- arch/arm64/mm/contpte.c | 46 ----------------------- arch/riscv/include/asm/pgtable.h | 10 +++-- include/linux/contpte.h | 3 ++ mm/contpte.c | 63 ++++++++++++++++++++++++++++++++ 5 files changed, 76 insertions(+), 65 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 92c12fb85cb4..6591aab11c67 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1391,9 +1391,6 @@ extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, unsigned int nr, int full); extern void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr); -extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, - pte_t entry, int dirty); #define pte_batch_hint pte_batch_hint static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) @@ -1512,19 +1509,9 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, } #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS -static inline int ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, - pte_t entry, int dirty) -{ - pte_t orig_pte = __ptep_get(ptep); - - entry = pte_mknoncont(entry); - - if (likely(!pte_valid_cont(orig_pte))) - return __ptep_set_access_flags(vma, addr, ptep, entry, dirty); - - return contpte_ptep_set_access_flags(vma, addr, ptep, entry, dirty); -} +extern int ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty); #else /* CONFIG_THP_CONTPTE */ diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 16940511943c..5675a61452ac 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -62,49 +62,3 @@ void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, __wrprotect_ptes(mm, addr, ptep, nr); } EXPORT_SYMBOL_GPL(contpte_wrprotect_ptes); - -int contpte_ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, - pte_t entry, int dirty) -{ - unsigned long start_addr; - pte_t orig_pte; - int i; - - /* - * Gather the access/dirty bits for the contiguous range. If nothing has - * changed, its a noop. - */ - orig_pte = pte_mknoncont(ptep_get(ptep)); - if (pte_val(orig_pte) == pte_val(entry)) - return 0; - - /* - * We can fix up access/dirty bits without having to unfold the contig - * range. But if the write bit is changing, we must unfold. - */ - if (pte_write(orig_pte) == pte_write(entry)) { - /* - * For HW access management, we technically only need to update - * the flag on a single pte in the range. But for SW access - * management, we need to update all the ptes to prevent extra - * faults. Avoid per-page tlb flush in __ptep_set_access_flags() - * and instead flush the whole range at the end. - */ - ptep = arch_contpte_align_down(ptep); - start_addr = addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); - - for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) - __ptep_set_access_flags(vma, addr, ptep, entry, 0); - - if (dirty) - __flush_tlb_range(vma, start_addr, addr, - PAGE_SIZE, true, 3); - } else { - __contpte_try_unfold(vma->vm_mm, addr, ptep, orig_pte); - __ptep_set_access_flags(vma, addr, ptep, entry, dirty); - } - - return 1; -} -EXPORT_SYMBOL_GPL(contpte_ptep_set_access_flags); diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 42c7884b8d2e..b151a5aa4de8 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -803,6 +803,10 @@ extern int ptep_test_and_clear_young(struct vm_area_struct *vma, #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH extern int ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); +#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS +extern int ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep, + pte_t entry, int dirty); #else /* CONFIG_THP_CONTPTE */ @@ -816,11 +820,11 @@ extern int ptep_clear_flush_young(struct vm_area_struct *vma, #define ptep_test_and_clear_young __ptep_test_and_clear_young #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH #define ptep_clear_flush_young __ptep_clear_flush_young +#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS +#define ptep_set_access_flags __ptep_set_access_flags #endif /* CONFIG_THP_CONTPTE */ -#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS -#define ptep_set_access_flags __ptep_set_access_flags #define __HAVE_ARCH_PTEP_SET_WRPROTECT #define ptep_set_wrprotect __ptep_set_wrprotect @@ -990,7 +994,7 @@ static inline int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp, pmd_t entry, int dirty) { - return ptep_set_access_flags(vma, address, (pte_t *)pmdp, pmd_pte(entry), dirty); + return __ptep_set_access_flags(vma, address, (pte_t *)pmdp, pmd_pte(entry), dirty); } #define __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG diff --git a/include/linux/contpte.h b/include/linux/contpte.h index 76a49ac8b6f5..76244b0c678a 100644 --- a/include/linux/contpte.h +++ b/include/linux/contpte.h @@ -23,5 +23,8 @@ int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); +int contpte_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty); #endif /* _LINUX_CONTPTE_H */ diff --git a/mm/contpte.c b/mm/contpte.c index 600277b1196c..9cbbff1f67ad 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -769,4 +769,67 @@ __always_inline int ptep_clear_flush_young(struct vm_area_struct *vma, return contpte_ptep_clear_flush_young(vma, addr, ptep); } + +int contpte_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty) +{ + unsigned long start_addr; + pte_t orig_pte; + int i; + + /* + * Gather the access/dirty bits for the contiguous range. If nothing has + * changed, its a noop. + */ + orig_pte = pte_mknoncont(ptep_get(ptep)); + if (pte_val(orig_pte) == pte_val(entry)) + return 0; + + /* + * We can fix up access/dirty bits without having to unfold the contig + * range. But if the write bit is changing, we must unfold. + */ + if (pte_write(orig_pte) == pte_write(entry)) { + /* + * For HW access management, we technically only need to update + * the flag on a single pte in the range. But for SW access + * management, we need to update all the ptes to prevent extra + * faults. Avoid per-page tlb flush in __ptep_set_access_flags() + * and instead flush the whole range at the end. + */ + size_t pgsize; + int ncontig; + + ptep = arch_contpte_align_down(ptep); + ncontig = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, &pgsize); + start_addr = addr = ALIGN_DOWN(addr, ncontig * pgsize); + + for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) + __ptep_set_access_flags(vma, addr, ptep, entry, 0); + + if (dirty) + arch_contpte_flush_tlb_range(vma, start_addr, addr, pgsize); + } else { + __contpte_try_unfold(vma->vm_mm, addr, ptep, orig_pte); + __ptep_set_access_flags(vma, addr, ptep, entry, dirty); + } + + return 1; +} +EXPORT_SYMBOL_GPL(contpte_ptep_set_access_flags); + +__always_inline int ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty) +{ + pte_t orig_pte = __ptep_get(ptep); + + entry = pte_mknoncont(entry); + + if (likely(!pte_valid_cont(orig_pte))) + return __ptep_set_access_flags(vma, addr, ptep, entry, dirty); + + return contpte_ptep_set_access_flags(vma, addr, ptep, entry, dirty); +} #endif /* CONFIG_THP_CONTPTE */ -- 2.39.2