Received: by 2002:a05:6358:1087:b0:cb:c9d3:cd90 with SMTP id j7csp875501rwi; Wed, 19 Oct 2022 04:11:24 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5sY7jFTJM82EznYNYzaVlkbgPScm4iJX+dfrhVsowLuFuEgFU+QCOa4HOHrbcd5VfuY0mD X-Received: by 2002:a17:902:b606:b0:184:1183:5f9f with SMTP id b6-20020a170902b60600b0018411835f9fmr7981134pls.84.1666177884234; Wed, 19 Oct 2022 04:11:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666177884; cv=none; d=google.com; s=arc-20160816; b=S6zypVeIs2mVWLHhYosJGd/QlXol/LiTzyU85EAkBBWRAw7CuZFj/c8L4UV6PKnObk u+TzA+fojkzilt+EKcV85/LvL8ImPZP/9Mc9gJfQa6zXLR13GdQIKPgfp/k4UPlGmzyI 90ZW40PV5rzMmfWyuoG4MS/ud/xaCWJtX0P3bhmw9v3uGzk7fkZMIZe5NQafCdRGC4lt fFZLFUBlkYdHzuFZHH12N6RytpRkTjZmCYdFSCsxh1qNF+LceXuFXQXgBajUUFMBdZWG stUGY9ISKXrs2zU0W0SoXjwflOxayQ0bie5eBXNEbHNcA3KKTb9OVFgvF/v3l3aqud7k u0/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=SypgHWzDVrjCOskq+/eec/SSmis91nrWLgYl9xZcaFw=; b=Vb4Odln+dawwniK0ke91oyDf+jDmFGcWbDUuqRJQtcHJz0i43Q5HewFnMoxR0/zqxd TrLYbkdbTsuYW0cm1oaGpmbCdOHxgBgx9xWKjPIVGKcanEzUU/NvGOJnLVnZY9AtbK/h 3vIkCPm5amIet6duQYvleAfkYgegxz3Z6Msleq68NzhDQ58dkMJlw4N6gNu+8W2dIDA+ GxVb0r3XUtuh2AKcpxNHMbqPakzznVqNPmcDsUGzZH6+wXu1tlUBEf/OB4RqCKr4caeZ DKjtDFiJn5r2BmqnpWN05tSCbfn3b5zZk7ias23N0a25GKc78Fj9cAyS2C+PqBhhY9ED 9xTA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linuxfoundation.org header.s=korg header.b=Ix1ily4U; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linuxfoundation.org Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j11-20020a056a00234b00b00550ec562c79si18785517pfj.369.2022.10.19.04.11.11; Wed, 19 Oct 2022 04:11:24 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linuxfoundation.org header.s=korg header.b=Ix1ily4U; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linuxfoundation.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233480AbiJSJ0y (ORCPT + 99 others); Wed, 19 Oct 2022 05:26:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48516 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233421AbiJSJZS (ORCPT ); Wed, 19 Oct 2022 05:25:18 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 285BBE09E1; Wed, 19 Oct 2022 02:11:16 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 787E761851; Wed, 19 Oct 2022 09:08:59 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8AB18C433D6; Wed, 19 Oct 2022 09:08:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1666170538; bh=qsJ5r+UWQAT9I+BtLd+LlxD1anjr0Hmry+7Ipw0+Uew=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ix1ily4UzZR/Dlg7W7QG2/VLATh/87FgSSbLHBMH5kllGG7DdombPMiZYyOiPQd9W 1Q4TtlLZzowMPq1wzWt1HM8f0GL5gMPqwJZbC1CO+faEw21ldDO92d8tF1lQ6vNpzr lka0jA70lM+MgTm0PvE8E++R6Q1n4sBkzbKEORfM= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Daniel Sneddon , Dave Hansen , Neelima Krishnan , Sasha Levin Subject: [PATCH 6.0 693/862] x86/apic: Dont disable x2APIC if locked Date: Wed, 19 Oct 2022 10:33:00 +0200 Message-Id: <20221019083320.592790450@linuxfoundation.org> X-Mailer: git-send-email 2.38.0 In-Reply-To: <20221019083249.951566199@linuxfoundation.org> References: <20221019083249.951566199@linuxfoundation.org> User-Agent: quilt/0.67 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-7.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Daniel Sneddon [ Upstream commit b8d1d163604bd1e600b062fb00de5dc42baa355f ] The APIC supports two modes, legacy APIC (or xAPIC), and Extended APIC (or x2APIC). X2APIC mode is mostly compatible with legacy APIC, but it disables the memory-mapped APIC interface in favor of one that uses MSRs. The APIC mode is controlled by the EXT bit in the APIC MSR. The MMIO/xAPIC interface has some problems, most notably the APIC LEAK [1]. This bug allows an attacker to use the APIC MMIO interface to extract data from the SGX enclave. Introduce support for a new feature that will allow the BIOS to lock the APIC in x2APIC mode. If the APIC is locked in x2APIC mode and the kernel tries to disable the APIC or revert to legacy APIC mode a GP fault will occur. Introduce support for a new MSR (IA32_XAPIC_DISABLE_STATUS) and handle the new locked mode when the LEGACY_XAPIC_DISABLED bit is set by preventing the kernel from trying to disable the x2APIC. On platforms with the IA32_XAPIC_DISABLE_STATUS MSR, if SGX or TDX are enabled the LEGACY_XAPIC_DISABLED will be set by the BIOS. If legacy APIC is required, then it SGX and TDX need to be disabled in the BIOS. [1]: https://aepicleak.com/aepicleak.pdf Signed-off-by: Daniel Sneddon Signed-off-by: Dave Hansen Acked-by: Dave Hansen Tested-by: Neelima Krishnan Link: https://lkml.kernel.org/r/20220816231943.1152579-1-daniel.sneddon@linux.intel.com Signed-off-by: Sasha Levin --- .../admin-guide/kernel-parameters.txt | 4 ++ arch/x86/Kconfig | 7 ++- arch/x86/include/asm/cpu.h | 2 + arch/x86/include/asm/msr-index.h | 13 ++++++ arch/x86/kernel/apic/apic.c | 44 +++++++++++++++++-- 5 files changed, 65 insertions(+), 5 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 426fa892d311..2bc11a61c4d0 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3805,6 +3805,10 @@ nox2apic [X86-64,APIC] Do not enable x2APIC mode. + NOTE: this parameter will be ignored on systems with the + LEGACY_XAPIC_DISABLED bit set in the + IA32_XAPIC_DISABLE_STATUS MSR. + nps_mtm_hs_ctr= [KNL,ARC] This parameter sets the maximum duration, in cycles, each HW thread of the CTOP can run diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index f9920f1341c8..159c025ebb03 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -448,6 +448,11 @@ config X86_X2APIC This allows 32-bit apic IDs (so it can support very large systems), and accesses the local apic via MSRs not via mmio. + Some Intel systems circa 2022 and later are locked into x2APIC mode + and can not fall back to the legacy APIC modes if SGX or TDX are + enabled in the BIOS. They will be unable to boot without enabling + this option. + If you don't know what to do here, say N. config X86_MPPARSE @@ -1919,7 +1924,7 @@ endchoice config X86_SGX bool "Software Guard eXtensions (SGX)" - depends on X86_64 && CPU_SUP_INTEL + depends on X86_64 && CPU_SUP_INTEL && X86_X2APIC depends on CRYPTO=y depends on CRYPTO_SHA256=y select SRCU diff --git a/arch/x86/include/asm/cpu.h b/arch/x86/include/asm/cpu.h index 8cbf623f0ecf..b472ef76826a 100644 --- a/arch/x86/include/asm/cpu.h +++ b/arch/x86/include/asm/cpu.h @@ -94,4 +94,6 @@ static inline bool intel_cpu_signatures_match(unsigned int s1, unsigned int p1, return p1 & p2; } +extern u64 x86_read_arch_cap_msr(void); + #endif /* _ASM_X86_CPU_H */ diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h index 6674bdb096f3..1e086b37a307 100644 --- a/arch/x86/include/asm/msr-index.h +++ b/arch/x86/include/asm/msr-index.h @@ -155,6 +155,11 @@ * Return Stack Buffer Predictions. */ +#define ARCH_CAP_XAPIC_DISABLE BIT(21) /* + * IA32_XAPIC_DISABLE_STATUS MSR + * supported + */ + #define MSR_IA32_FLUSH_CMD 0x0000010b #define L1D_FLUSH BIT(0) /* * Writeback and invalidate the @@ -1054,4 +1059,12 @@ #define MSR_IA32_HW_FEEDBACK_PTR 0x17d0 #define MSR_IA32_HW_FEEDBACK_CONFIG 0x17d1 +/* x2APIC locked status */ +#define MSR_IA32_XAPIC_DISABLE_STATUS 0xBD +#define LEGACY_XAPIC_DISABLED BIT(0) /* + * x2APIC mode is locked and + * disabling x2APIC will cause + * a #GP + */ + #endif /* _ASM_X86_MSR_INDEX_H */ diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c index 6d303d1d276c..c6876d3ea4b1 100644 --- a/arch/x86/kernel/apic/apic.c +++ b/arch/x86/kernel/apic/apic.c @@ -61,6 +61,7 @@ #include #include #include +#include unsigned int num_processors; @@ -1751,11 +1752,26 @@ EXPORT_SYMBOL_GPL(x2apic_mode); enum { X2APIC_OFF, - X2APIC_ON, X2APIC_DISABLED, + /* All states below here have X2APIC enabled */ + X2APIC_ON, + X2APIC_ON_LOCKED }; static int x2apic_state; +static bool x2apic_hw_locked(void) +{ + u64 ia32_cap; + u64 msr; + + ia32_cap = x86_read_arch_cap_msr(); + if (ia32_cap & ARCH_CAP_XAPIC_DISABLE) { + rdmsrl(MSR_IA32_XAPIC_DISABLE_STATUS, msr); + return (msr & LEGACY_XAPIC_DISABLED); + } + return false; +} + static void __x2apic_disable(void) { u64 msr; @@ -1793,6 +1809,10 @@ static int __init setup_nox2apic(char *str) apicid); return 0; } + if (x2apic_hw_locked()) { + pr_warn("APIC locked in x2apic mode, can't disable\n"); + return 0; + } pr_warn("x2apic already enabled.\n"); __x2apic_disable(); } @@ -1807,10 +1827,18 @@ early_param("nox2apic", setup_nox2apic); void x2apic_setup(void) { /* - * If x2apic is not in ON state, disable it if already enabled + * Try to make the AP's APIC state match that of the BSP, but if the + * BSP is unlocked and the AP is locked then there is a state mismatch. + * Warn about the mismatch in case a GP fault occurs due to a locked AP + * trying to be turned off. + */ + if (x2apic_state != X2APIC_ON_LOCKED && x2apic_hw_locked()) + pr_warn("x2apic lock mismatch between BSP and AP.\n"); + /* + * If x2apic is not in ON or LOCKED state, disable it if already enabled * from BIOS. */ - if (x2apic_state != X2APIC_ON) { + if (x2apic_state < X2APIC_ON) { __x2apic_disable(); return; } @@ -1831,6 +1859,11 @@ static __init void x2apic_disable(void) if (x2apic_id >= 255) panic("Cannot disable x2apic, id: %08x\n", x2apic_id); + if (x2apic_hw_locked()) { + pr_warn("Cannot disable locked x2apic, id: %08x\n", x2apic_id); + return; + } + __x2apic_disable(); register_lapic_address(mp_lapic_addr); } @@ -1889,7 +1922,10 @@ void __init check_x2apic(void) if (x2apic_enabled()) { pr_info("x2apic: enabled by BIOS, switching to x2apic ops\n"); x2apic_mode = 1; - x2apic_state = X2APIC_ON; + if (x2apic_hw_locked()) + x2apic_state = X2APIC_ON_LOCKED; + else + x2apic_state = X2APIC_ON; } else if (!boot_cpu_has(X86_FEATURE_X2APIC)) { x2apic_state = X2APIC_DISABLED; } -- 2.35.1