Received: by 2002:a05:6a10:5bc5:0:0:0:0 with SMTP id os5csp1583632pxb; Tue, 26 Oct 2021 11:41:57 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwl1W2kVkViiZDmmkd9yVcJjgnOn4B4pva1lZJLxWdrKoCTKkskvn8tvyTs5g6MtvkQDKNn X-Received: by 2002:a63:8342:: with SMTP id h63mr17146302pge.460.1635273717405; Tue, 26 Oct 2021 11:41:57 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1635273717; cv=none; d=google.com; s=arc-20160816; b=UViJHAskhqMI/CKF8ZqHVzDzzMV5ZcaRRmf3xaZvohKrGJmYlMyS6vbOaJ4lHtiifh FGOXnLJcrhyRV0lZWbT8Z5TlOISaSnycBvQVz23GjcjDG29Z7KbO4JRiltB0EMOvyTfV NHz6UKhzfC4ro5cM+OKoUrQ3FajQffLyR3Pta7HzFoalL/RELN9o1/gmUeAL/nIe1NbK bW5Tg7MhcyUDxBo2tMqkpDemElXZGUD749ylpAHkDlFrC3qvxMTr3/7TUXU2OXnSgSoN wHRsBfsrD+pD/3sXwuP4Bm9a+saNU/q7hW25VOUx9jFkP1vKqd1e6KUz/U+lvmgZtQsb EyLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=anKSpqSrV77lA2Jw/TrzCwt7GostmDO7PEGwJX+kIAc=; b=e8krmtqy/fy8rCakjFNW5J6ajnHn6Wwo5f3pglT0jusY37TIXfTkqbRxqWi+280FK9 sWmi8hEppUy9u7Ps4i3vjM69b4Wzy1wqgTcC8bTNtnu1kXCitTmEArVIzppz5vYqmNFC qTkZf75782WY3fstudGyQVfGX9I3UROZje07Y1ATbVgYMgcI7FClXmjKo7sVoDPGhvL1 ZYREJw47T7kT4AUYnxfB5DkFQNXcj6stWg4sGvkOA6CyB1YdgL8K1bZHdE5pSGlTXNte TXJuYKy65mbnZNGS+RruYxjeUh65WYqy5HLHcEjXhYhhUq5xWvxjmDKx0iIpQXiW2mfE HltA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=AK9bdSwe; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id j190si5182852pgd.491.2021.10.26.11.41.44; Tue, 26 Oct 2021 11:41:57 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=AK9bdSwe; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236794AbhJZOin (ORCPT + 99 others); Tue, 26 Oct 2021 10:38:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57068 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231546AbhJZOib (ORCPT ); Tue, 26 Oct 2021 10:38:31 -0400 Received: from mail-pf1-x435.google.com (mail-pf1-x435.google.com [IPv6:2607:f8b0:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B060DC061767 for ; Tue, 26 Oct 2021 07:36:07 -0700 (PDT) Received: by mail-pf1-x435.google.com with SMTP id 187so14489960pfc.10 for ; Tue, 26 Oct 2021 07:36:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=anKSpqSrV77lA2Jw/TrzCwt7GostmDO7PEGwJX+kIAc=; b=AK9bdSwe9kMV8lAg11CIX7qfNdVQaXTG+tlqCD5ANb11ZK+Ys3NAd9yMaNh2Tklzmf aYeJMfLWTiMYdcSsxstzm8NPTtvOQat6fyVevZNDTnVIOE+CyYS4fYf+DBuTcTTtMl1p FXXoBZX9jL9QHPzrK/FXOybs/Rn2ZbW+3W5KeBW1vpqMqPTLzP4GF1pVQbSY3OMNfYAB 90LnStIyGSsQY7GDR4kh08hJjJYQc8eB5Qj0cHtuDLRA0dskc3AeT3K8d9zuFHLfwKLV ZlnLQxx9/mzTXjQ/4lWvKtjBL5N8RQdFR8P8B/NBIpxQCdZ7u++JGHJyzgWyoOJUhYcm 0a3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=anKSpqSrV77lA2Jw/TrzCwt7GostmDO7PEGwJX+kIAc=; b=TcIPZrV0BerzmFL+lINu9ubVu89RRdtQJlWY60oyQWmTcnnj508wQNfm+alOw8HKHi lXWAwIzlVgSAXISvOv+mGBpilWSYwVXVbtXSsIx+NfmPuTLx0a+Jb6Qgt0GcUVo52u+f bfVGRB627ddLz/qYOYnZxeQOKbXBEdIVpKjwVX+w1Chzziibla7BbA1A5tdnc/51RAm2 srDOJ1m9aiTKU1ra+mCK/4xoAPGUS2qUbMCyU1nAo08VMb2tZpSSgWty05I4mErOUbpL 8mjUzucDzNfT/zxPyjw4/DXtrkbhwD/0Ikt2nIl6QFZ7R+qTJHxp4i5C9GK5U8e4nHzQ KkAA== X-Gm-Message-State: AOAM533JLs8YtNOG77xaOGpwv+ltzs3f1j3U0HcaNNuFZnF+1+9xYjPS dF01w8P+4A+Z/zQY8bNv5cgm22dvx9s= X-Received: by 2002:a63:7506:: with SMTP id q6mr19019167pgc.319.1635258967103; Tue, 26 Oct 2021 07:36:07 -0700 (PDT) Received: from localhost ([47.88.5.130]) by smtp.gmail.com with ESMTPSA id y4sm3133716pfi.178.2021.10.26.07.36.05 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 26 Oct 2021 07:36:06 -0700 (PDT) From: Lai Jiangshan To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, Lai Jiangshan , Andy Lutomirski , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Juergen Gross , "Peter Zijlstra (Intel)" , Joerg Roedel Subject: [PATCH V4 37/50] x86/entry: Implement the C version ist_paranoid_exit() Date: Tue, 26 Oct 2021 22:34:23 +0800 Message-Id: <20211026143436.19071-12-jiangshanlai@gmail.com> X-Mailer: git-send-email 2.19.1.6.gb485710b In-Reply-To: <20211026141420.17138-1-jiangshanlai@gmail.com> References: <20211026141420.17138-1-jiangshanlai@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Lai Jiangshan It implements the whole ASM version paranoid_exit(). No functional difference intended. Signed-off-by: Lai Jiangshan --- arch/x86/entry/entry64.c | 41 +++++++++++++++++++++++++++++++++ arch/x86/include/asm/idtentry.h | 2 ++ 2 files changed, 43 insertions(+) diff --git a/arch/x86/entry/entry64.c b/arch/x86/entry/entry64.c index 14005122f94b..161c025d5e7a 100644 --- a/arch/x86/entry/entry64.c +++ b/arch/x86/entry/entry64.c @@ -275,6 +275,29 @@ static __always_inline unsigned long ist_switch_to_kernel_gsbase(void) return 0; } +static __always_inline void ist_restore_gsbase(unsigned long gsbase) +{ + /* + * Handle the three GSBASE cases. + * + * @gsbase contains the GSBASE related information depending + * on the availability of the FSGSBASE instructions: + * + * FSGSBASE @gsbase + * N 0 -> SWAPGS on exit + * 1 -> no SWAPGS on exit + * + * Y User space GSBASE, must be restored unconditionally + */ + if (static_cpu_has(X86_FEATURE_FSGSBASE)) { + wrgsbase(gsbase); + return; + } + + if (gsbase) + native_swapgs(); +} + /* * Switch and save CR3 in *@cr3 if PTI enabled. Return GSBASE related * information in *@gsbase depending on the availability of the FSGSBASE @@ -311,3 +334,21 @@ void ist_paranoid_entry(unsigned long *cr3, unsigned long *gsbase) /* Handle GSBASE, store the return value in *@gsbase for exit. */ *gsbase = ist_switch_to_kernel_gsbase(); } + +/* + * "Paranoid" exit path from exception stack. This is invoked + * only on return from IST interrupts that came from kernel space. + * + * We may be returning to very strange contexts (e.g. very early + * in syscall entry), so checking for preemption here would + * be complicated. Fortunately, there's no good reason to try + * to handle preemption here. + */ +__visible __entry_text +void ist_paranoid_exit(unsigned long cr3, unsigned long gsbase) +{ + /* Restore CR3 at first, it can use kernel GSBASE. */ + ist_restore_cr3(cr3); + barrier(); + ist_restore_gsbase(gsbase); +} diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h index f6efa21ec242..cf41901227ed 100644 --- a/arch/x86/include/asm/idtentry.h +++ b/arch/x86/include/asm/idtentry.h @@ -309,6 +309,8 @@ static __always_inline void __##func(struct pt_regs *regs) #ifdef CONFIG_X86_64 __visible __entry_text void ist_paranoid_entry(unsigned long *cr3, unsigned long *gsbase); +__visible __entry_text +void ist_paranoid_exit(unsigned long cr3, unsigned long gsbase); /** * DECLARE_IDTENTRY_IST - Declare functions for IST handling IDT entry points -- 2.19.1.6.gb485710b