Received: by 2002:a5d:925a:0:0:0:0:0 with SMTP id e26csp1472656iol; Fri, 10 Jun 2022 08:10:54 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzLLFwHvnMRbwF/CeF6NxfcOohJcR6sPPfLQ1Nxi4dIEETZnKYx0INK4cARmZ/xo9IWP6tG X-Received: by 2002:a05:6a00:1946:b0:4fe:309f:d612 with SMTP id s6-20020a056a00194600b004fe309fd612mr114364484pfk.10.1654873854486; Fri, 10 Jun 2022 08:10:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1654873854; cv=none; d=google.com; s=arc-20160816; b=tuLZnAQCLuqgHtGrReVqp76gPFD1yOGlFADkJUzoM/yite2f6ugb+KWLHSHunewJRG cQZkxsyxvnN34e8FRh5wJ+uySADPzle8B2M7TD3aJVxfiBk3r9Pc9b7d/AXgXn9FZ6yd Ny1aR4w/FTjCGcRABmtg139yv2AjZOlRBAA0MBGs7VFhj2nlfrlJXIMADkdpioWWx0dD fV0CfzJXMV6jpdtvuakZCfO0y2vzZO6QG/fthlGWQvbOW6xK2VLRaRYeNqgJZfH2FhqT kwnuoKarCaO1PWXm/RSY/3/ZqQ/bBG2w7cPZ2z3c8fc+wcYShIElpyhWnuAsmfvzo38W u2kw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=NIxeD7Kqe/3LmXsLmDe5Pals+bwMQFL4JJKQf0RbNTg=; b=n6a3I0RQ7ybR6jD49z/Ogpyx/dpFWFxlq5frZbCrwOHqbELnVLdTe/xHXF88I1Jw3H S3vYYHo8SJl7bpd7PpjIlSvNqgk8ebtNKEff3BxC+sfYZcTVpMyCA39LpbwqZ1giKOB9 GQDNnZaLklhuxiOHm66WNcHrs38oKwcLqnclq+tssmUG4GmExJMjJfwF8SjWdsHiTEgA biaAkjAiPNen5k+LpN2801WLDehMTgqXFfdWg0zEi1j02n4334ErAOmUvYVUmL2QQEQg QksCEQS5EbC+EKY3tzam/a2/9aKswgVgfA6aUgJ8cwdDxFoSxOWoWNYA8gxXozg/+veM nEpA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=C3UJLDeV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c9-20020a170902b68900b001639f0ff47esi7158799pls.40.2022.06.10.08.10.39; Fri, 10 Jun 2022 08:10:54 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=C3UJLDeV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1349677AbiFJOge (ORCPT + 99 others); Fri, 10 Jun 2022 10:36:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58750 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1349603AbiFJOgH (ORCPT ); Fri, 10 Jun 2022 10:36:07 -0400 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4D5D5393C3 for ; Fri, 10 Jun 2022 07:36:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1654871766; x=1686407766; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=R6QXymG4GlH91ebp/ZT4U7nCHZ8Wto7mGw90OqLzgZc=; b=C3UJLDeV6eomQkJXVsQL8vZydMweENwejzIN8yCUb83rDq2rGUTXkAbg qVZ+LYF434mQ9kRcn2QohzToQ9mrII6j9kua7CSD50ligE/6BsvZFUbMc IpGca6i1uAysZvlNhareFeb3cH7HnqsCu+2fhuKK/ajXUdr+ckthH+zn/ 30SpArSHZKVe77MMAN+bVk8tnS6/1TO8vmwss3UhXSFSclL2h4xeET9gJ YN7vKwWODREDKUwM7PGe4mHIk2C2qcbem8TxGVyavKwYX5ZDcxBHw00Gf SSguZYZjGVSnBLKd/Kk3isxWOA0DYl9qOuz9AFRG1Xs43D7epO1cgiKSB Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10374"; a="278781422" X-IronPort-AV: E=Sophos;i="5.91,290,1647327600"; d="scan'208";a="278781422" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Jun 2022 07:35:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,290,1647327600"; d="scan'208";a="908939814" Received: from black.fi.intel.com ([10.237.72.28]) by fmsmga005.fm.intel.com with ESMTP; 10 Jun 2022 07:35:31 -0700 Received: by black.fi.intel.com (Postfix, from userid 1000) id 4C0E05D3; Fri, 10 Jun 2022 17:35:30 +0300 (EEST) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv3 5/8] x86/uaccess: Provide untagged_addr() and remove tags before address check Date: Fri, 10 Jun 2022 17:35:24 +0300 Message-Id: <20220610143527.22974-6-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220610143527.22974-1-kirill.shutemov@linux.intel.com> References: <20220610143527.22974-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-5.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org untagged_addr() is a helper used by the core-mm to strip tag bits and get the address to the canonical shape. In only handles userspace addresses. The untagging mask is stored in mmu_context and will be set on enabling LAM for the process. The tags must not be included into check whether it's okay to access the userspace address. Strip tags in access_ok(). get_user() and put_user() don't use access_ok(), but check access against TASK_SIZE directly in assembly. Strip tags, before calling into the assembly helper. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/mmu.h | 1 + arch/x86/include/asm/mmu_context.h | 11 ++++++++ arch/x86/include/asm/uaccess.h | 44 ++++++++++++++++++++++++++++-- arch/x86/kernel/process.c | 3 ++ 4 files changed, 56 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index d150e92163b6..59c617fc7c6f 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -41,6 +41,7 @@ typedef struct { #ifdef CONFIG_X86_64 unsigned short flags; u64 lam_cr3_mask; + u64 untag_mask; #endif struct mutex lock; diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index e6eac047c728..05821534aadc 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -100,6 +100,12 @@ static inline u64 mm_cr3_lam_mask(struct mm_struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask; + mm->context.untag_mask = oldmm->context.untag_mask; +} + +static inline void mm_reset_untag_mask(struct mm_struct *mm) +{ + mm->context.untag_mask = -1UL; } #else @@ -112,6 +118,10 @@ static inline u64 mm_cr3_lam_mask(struct mm_struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { } + +static inline void mm_reset_untag_mask(struct mm_struct *mm) +{ +} #endif #define enter_lazy_tlb enter_lazy_tlb @@ -138,6 +148,7 @@ static inline int init_new_context(struct task_struct *tsk, mm->context.execute_only_pkey = -1; } #endif + mm_reset_untag_mask(mm); init_new_context_ldt(mm); return 0; } diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index 35f222aa66bf..ca754521a4db 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -6,6 +6,7 @@ */ #include #include +#include #include #include #include @@ -20,6 +21,32 @@ static inline bool pagefault_disabled(void); # define WARN_ON_IN_IRQ() #endif +#ifdef CONFIG_X86_64 +/* + * Mask out tag bits from the address. + * + * Magic with the 'sign' allows to untag userspace pointer without any branches + * while leaving kernel addresses intact. + */ +#define untagged_addr(mm, addr) ({ \ + u64 __addr = (__force u64)(addr); \ + s64 sign = (s64)__addr >> 63; \ + __addr ^= sign; \ + __addr &= (mm)->context.untag_mask; \ + __addr ^= sign; \ + (__force __typeof__(addr))__addr; \ +}) + +#define untagged_ptr(mm, ptr) ({ \ + u64 __ptrval = (__force u64)(ptr); \ + __ptrval = untagged_addr(mm, __ptrval); \ + (__force __typeof__(*(ptr)) *)__ptrval; \ +}) +#else +#define untagged_addr(mm, addr) (addr) +#define untagged_ptr(mm, ptr) (ptr) +#endif + /** * access_ok - Checks if a user space pointer is valid * @addr: User space pointer to start of block to check @@ -40,7 +67,7 @@ static inline bool pagefault_disabled(void); #define access_ok(addr, size) \ ({ \ WARN_ON_IN_IRQ(); \ - likely(__access_ok(addr, size)); \ + likely(__access_ok(untagged_addr(current->mm, addr), size)); \ }) #include @@ -125,7 +152,13 @@ extern int __get_user_bad(void); * Return: zero on success, or -EFAULT on error. * On error, the variable @x is set to zero. */ -#define get_user(x,ptr) ({ might_fault(); do_get_user_call(get_user,x,ptr); }) +#define get_user(x,ptr) \ +({ \ + __typeof__(*(ptr)) __user *__ptr_clean; \ + __ptr_clean = untagged_ptr(current->mm, ptr); \ + might_fault(); \ + do_get_user_call(get_user,x,__ptr_clean); \ +}) /** * __get_user - Get a simple variable from user space, with less checking. @@ -222,7 +255,12 @@ extern void __put_user_nocheck_8(void); * * Return: zero on success, or -EFAULT on error. */ -#define put_user(x, ptr) ({ might_fault(); do_put_user_call(put_user,x,ptr); }) +#define put_user(x, ptr) ({ \ + __typeof__(*(ptr)) __user *__ptr_clean; \ + __ptr_clean = untagged_ptr(current->mm, ptr); \ + might_fault(); \ + do_put_user_call(put_user,x,__ptr_clean); \ +}) /** * __put_user - Write a simple value into user space, with less checking. diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index 9b2772b7e1f3..18b2bfdf7b9b 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -47,6 +47,7 @@ #include #include #include +#include #include "process.h" @@ -367,6 +368,8 @@ void arch_setup_new_exec(void) task_clear_spec_ssb_noexec(current); speculation_ctrl_update(read_thread_flags()); } + + mm_reset_untag_mask(current->mm); } #ifdef CONFIG_X86_IOPL_IOPERM -- 2.35.1