Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
From:   Jon Kohler <jon@nutanix.com>
To:     Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>, x86@kernel.org,
        "H. Peter Anvin" <hpa@zytor.com>, Andi Kleen <ak@linux.intel.com>,
        Pawan Gupta <pawan.kumar.gupta@linux.intel.com>,
        Jon Kohler <jon@nutanix.com>, Tony Luck <tony.luck@intel.com>,
        linux-kernel@vger.kernel.org
Cc:     dave.hansen@intel.com, Borislav Petkov <bp@suse.de>,
        Neelima Krishnan <neelima.krishnan@intel.com>,
        "kvm @ vger . kernel . org" <kvm@vger.kernel.org>
Subject: [PATCH v2] x86/tsx: fix KVM guest live migration for tsx=on
Date:   Mon, 11 Apr 2022 16:07:01 -0400
Message-Id: <20220411200703.48654-1-jon@nutanix.com>
In-Reply-To: <20220411180131.5054-1-jon@nutanix.com>
References: <20220411180131.5054-1-jon@nutanix.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 2
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?KsPulO7LYrQoqoOzB4O+6caxK2SCSfOndXf158C196W9BwF5AWC+EmowQwcc?=
 =?us-ascii?Q?+LXPDChQlvX/VxRl0Cm/5XLvFnDOia0CmuyTvAZNyH4b73f9LR5JNjDkMx18?=
 =?us-ascii?Q?7+CsWxULu4dM7zzMYaPUbaREi3gjQOp2cEW96V9L7M9J3CMHoXzteMNUVx+T?=
 =?us-ascii?Q?S9gEJqDQq2hS5wKK1sgBtxE37clhbDRaiP9hBeGs/nBLCTAc092zJ0Tnit3G?=
 =?us-ascii?Q?OfWdDoEZysfwzOULk8emlMgFa/nM1qfZKYZXZkmwu1JFMIS79hZ1I8xoCL0S?=
 =?us-ascii?Q?S3FfD4HSxIDzQTpRKCLIfe7ifm9roehhvRB4Sfel1R4W+K0J7LpYLHOJW+op?=
 =?us-ascii?Q?8QncVimGI9BJn9ypM2+9FOu9gBySSSQw+lHpFtfAMKlpMx1p6hAyrbNqsocC?=
 =?us-ascii?Q?/QKmJJuh+DSFpkpLyLkELfuXhTrVfAQMq+H7nAI/IkyZERvrG2rLIoOVA+lq?=
 =?us-ascii?Q?9mS4SOGfGG+MTkvBdIsBhvUVaIBuNoZJEoVZELBMsYY1bFIBo2Zp14tFn3JR?=
 =?us-ascii?Q?V4LR4LpgK2Oh90tAyLt2U4tpjiVSQYBIJ/x0nN9kDRSAxCeXJS2LSQM93h29?=
 =?us-ascii?Q?Eo7VxUKv+40YSNzZotb07EhsjAmIABXjzcqAvvg38NXIfvebD7ec5QvyupwS?=
 =?us-ascii?Q?m2RvG5Ut4Or3EI9hZaDJqIQyASQ3mEVbrQ/dWBwXYjM6yDSZ2MZK+XkkaNTI?=
 =?us-ascii?Q?icDDV26zaAdpX2lQ/gMNDknFzaMKcBKWgdSJ8WZI6UPv4SEa17DTOldJ5EzZ?=
 =?us-ascii?Q?cH7WGwoV50Vq1apqUQbcUovLgEjc/oQCzd3w9GSu2v5+30HMxAy7EnSrUvG4?=
 =?us-ascii?Q?lYgizrG268Gg+TjjPEupSCIDrrm7J9S36cj7GNsg3vJVd291vQzM5Yel6bWV?=
 =?us-ascii?Q?PtvnLWj4OcVaNjM1dHM0xk1fCVdfuLS5X8U5Otgmf+fD2N/q4hjFbhJRajFJ?=
 =?us-ascii?Q?beSsGKFyFC95n0f2J7jjek5G8yyW2SVfi25ZMn4r3SfKGmvObPIxv4RFkaGG?=
 =?us-ascii?Q?GFRUkzm4mHABTu3nvC2dNMEONbzV+t7hwWIiavVcOz2hnaVydebZhXVl/1D6?=
 =?us-ascii?Q?hFWwCTV0on3ZWhfRBZB/Fs2khz6KHRQMbiGeyx7lyXe4mI4huSEnBbh6+SqJ?=
 =?us-ascii?Q?X9ibRZyv9PJUQdry+6hHiuXjdYeokvKSLGNaPd4Xx+/aMbHfGZqJyYgv9Fwe?=
 =?us-ascii?Q?WJg+yHYEK4lFxw92peXxQSVGY0Q/SZYb3SqY11NSRCYJErSzBWxR/qmjBGyP?=
 =?us-ascii?Q?li8X8dJOt9bOtZYkufGp6HQXjoIQ2E/CB4h5A9hTKG8AeWAnbVKMFbuvDwij?=
 =?us-ascii?Q?Tmf2CXUNaifBivfU4ZQNKVFmQqtvfG6yjXCl4WTqTSpXfgRaM+Z4HchRLm8V?=
 =?us-ascii?Q?rP8Q/PLNDQaNdulCvnn0rgyiYTmYZ+/MEnQWEkYQSRgKLpeu7LAHy5rYEJFd?=
 =?us-ascii?Q?KxWA+zO1bvKnsEbQ9iURktkBXFJwpkLo8IGnm61myjDuOyd6Fu/abxbABGTh?=
 =?us-ascii?Q?kaFxhXLcdxcsRaYubIgsOmryvoRl5m/ryqWJ9Vchbd20okodean/fr3wCMn4?=
 =?us-ascii?Q?eptidu8PWF8wAtiqcnoX3njXOZj/4Vn6JPphgXVLdMxDdnf3rEAe3mE48bsd?=
 =?us-ascii?Q?4BoCwODix84B9lInGm0rpjgOyiV1OpmwH7QWTLWNQhYEdUnGE9c1txbPMZF2?=
 =?us-ascii?Q?azbmfkTc0tpKGkqP5GQUL7en2f/UTq4uGG2Aui6TF/FyCgaHHBm3fUPCboFQ?=
 =?us-ascii?Q?sFrDIBe4v9rgNKlcslyflAu0xrRE+iQE8JD1dGIUjMqkV4L/BUIXdW+BV2DY?=
X-MS-Exchange-AntiSpam-MessageData-1: 8yxXpsOfHXQtGnt2EGp0UJtF+/Shf8miyUQ=
X-OriginatorOrg: nutanix.com
X-MS-Exchange-CrossTenant-Network-Message-Id: ab42ae10-42c5-497d-8a1b-08da1bf6eae6
X-MS-Exchange-CrossTenant-AuthSource: BL0PR02MB4579.namprd02.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Apr 2022 20:07:34.4634
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: xXK8QQKoF5mDt9z58HCsuhrWQaB55nWiSU2Qd7lpXIBt7/xcUqkHdQGiXO3MP+Wa3mpEjbEdQPy4RtfSa6BSF0FYxQAcN8XAxmGKWP/rGMA=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR02MB2382
X-Proofpoint-ORIG-GUID: e3q1xkVQiJqogFzEk0ppvTOixxso6ocV
X-Proofpoint-GUID: e3q1xkVQiJqogFzEk0ppvTOixxso6ocV
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.858,Hydra:6.0.425,FMLib:17.11.64.514
 definitions=2022-04-11_08,2022-04-11_01,2022-02-23_01
X-Proofpoint-Spam-Reason: safe
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Move automatic disablement for TSX microcode deprecation from tsx_init() to
x86_get_tsx_auto_mode(), such that systems with tsx=on will continue to
see the TSX CPU features (HLE, RTM) even on updated microcode.

KVM live migration could be possibly be broken in 5.14+ commit 293649307ef9
("x86/tsx: Clear CPUID bits when TSX always force aborts"). Consider the
following scenario:

1. KVM hosts clustered in a live migration capable setup.
2. KVM guests have TSX CPU features HLE and/or RTM presented.
3. One of the three maintenance events occur:
3a. An existing host running kernel >= 5.14 in the pool updated with the
    new microcode.
3b. A new host running kernel >= 5.14 is commissioned that already has the
    microcode update preloaded.
3c. All hosts are running kernel < 5.14 with microcode update already
    loaded and one existing host gets updated to kernel >= 5.14.
4. After maintenance event, the impacted host will not have HLE and RTM
   exposed, and live migrations with guests with TSX features might not
   migrate.

Users using tsx=on or CONFIG_X86_INTEL_TSX_MODE_ON should always see
HLE and RTM on capable Intel SKUs, even if microcode has been clubbed to
prevent functionality.

Users using tsx=auto get or CONFIG_X86_INTEL_TSX_MODE_AUTO get to roll the
dice with whatever the kernel believes the appropriate default is, which
includes the feature disappearing after a kernel and/or microcode update.
These users should consider masking HLE and RTM at a higher control plane
level, e.g. qemu or libvirt, such that guests on TSX enabled systems do not
see HLE/RTM and therefore do not enable TAA mitigation.

Fixes: 293649307ef9 ("x86/tsx: Clear CPUID bits when TSX always force aborts")

Signed-off-by: Jon Kohler <jon@nutanix.com>
Cc: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Neelima Krishnan <neelima.krishnan@intel.com>
Cc: kvm@vger.kernel.org <kvm@vger.kernel.org>
---
v1 -> v2:
 - Addressed comments on approach from Dave.

 arch/x86/kernel/cpu/tsx.c | 30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)

diff --git a/arch/x86/kernel/cpu/tsx.c b/arch/x86/kernel/cpu/tsx.c
index 9c7a5f049292..4b701fa64869 100644
--- a/arch/x86/kernel/cpu/tsx.c
+++ b/arch/x86/kernel/cpu/tsx.c
@@ -78,6 +78,10 @@ static bool __init tsx_ctrl_is_supported(void)

 static enum tsx_ctrl_states x86_get_tsx_auto_mode(void)
 {
+	if (boot_cpu_has(X86_FEATURE_RTM_ALWAYS_ABORT) &&
+	    boot_cpu_has(X86_FEATURE_TSX_FORCE_ABORT))
+		return TSX_CTRL_RTM_ALWAYS_ABORT;
+
 	if (boot_cpu_has_bug(X86_BUG_TAA))
 		return TSX_CTRL_DISABLE;

@@ -105,21 +109,6 @@ void __init tsx_init(void)
 	char arg[5] = {};
 	int ret;

-	/*
-	 * Hardware will always abort a TSX transaction if both CPUID bits
-	 * RTM_ALWAYS_ABORT and TSX_FORCE_ABORT are set. In this case, it is
-	 * better not to enumerate CPUID.RTM and CPUID.HLE bits. Clear them
-	 * here.
-	 */
-	if (boot_cpu_has(X86_FEATURE_RTM_ALWAYS_ABORT) &&
-	    boot_cpu_has(X86_FEATURE_TSX_FORCE_ABORT)) {
-		tsx_ctrl_state = TSX_CTRL_RTM_ALWAYS_ABORT;
-		tsx_clear_cpuid();
-		setup_clear_cpu_cap(X86_FEATURE_RTM);
-		setup_clear_cpu_cap(X86_FEATURE_HLE);
-		return;
-	}
-
 	if (!tsx_ctrl_is_supported()) {
 		tsx_ctrl_state = TSX_CTRL_NOT_SUPPORTED;
 		return;
@@ -173,5 +162,16 @@ void __init tsx_init(void)
 		 */
 		setup_force_cpu_cap(X86_FEATURE_RTM);
 		setup_force_cpu_cap(X86_FEATURE_HLE);
+	} else if (tsx_ctrl_state == TSX_CTRL_RTM_ALWAYS_ABORT) {
+
+		/*
+		 * Hardware will always abort a TSX transaction if both CPUID bits
+		 * RTM_ALWAYS_ABORT and TSX_FORCE_ABORT are set. In this case, it is
+		 * better not to enumerate CPUID.RTM and CPUID.HLE bits. Clear them
+		 * here.
+		 */
+		tsx_clear_cpuid();
+		setup_clear_cpu_cap(X86_FEATURE_RTM);
+		setup_clear_cpu_cap(X86_FEATURE_HLE);
 	}
 }
--
2.30.1 (Apple Git-130)