Received: by 2002:a05:7412:b10a:b0:f3:1519:9f41 with SMTP id az10csp3081040rdb; Mon, 4 Dec 2023 16:46:48 -0800 (PST) X-Google-Smtp-Source: AGHT+IEEWrZc2qLs4Jdny/c5kScJtikcTfzpGTbrPwBC+tXNtVtcj4xszIULrHag9E75rOkC1wiM X-Received: by 2002:a17:90a:1d9:b0:286:6cc1:5fd4 with SMTP id 25-20020a17090a01d900b002866cc15fd4mr346134pjd.87.1701737208489; Mon, 04 Dec 2023 16:46:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701737208; cv=none; d=google.com; s=arc-20160816; b=sVYA77V6gzIDOBuXMQK1OPI9DQOFIjmQcKi5zxcTcycomq5MgIlsFUrTgn1AH4BJFk 23apeUIyjPrVorSSemTeccTpiB/4w8iceZ+Wa695SZAn3aZvp5d5pZ0nkkO7L5YfClR2 QnYMXPhxjupPePq8W0j0Q8Sospjyb0L7n7C9hMpc59E848CjDgIEAAH6b/B7UNrfAF+E UeI6DAAbfB47IZAXMNMrmt/PNOAajYhb/fVXYBoxdCrq7VgpWb+M7/aqq86IF4KPaDYo hCt2XuQM+l/oDQ+xgczC7TuX0T4bjgcLx/gY4LIym36BUA0lRbDhcnDQHPX1kl/Px0Sd fP6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=8I2NZ9PRNtWbYqvbfYaAykO7pcWUrs3fbCW9zohCMWg=; fh=OvJRnOqsMTm9XoNmEwebcqh9Ud7yh1CTeKAP84ols98=; b=e3QvNvKyEneVbxx5JUL3r3fCo83WNiJ856BH8OqTdYw2bdFm8r+x7ToPbbN5ZwP2sK lDTFq21Aoufk27gOQ6Nckw/ADhCWGCktBt9Ji8P+4FCSwcGVXcj3ZgFj4es50c4jqYfZ uNQxNCMoW/yY3vGkzQ1lD7fOMgUXObeOU6Jfla/6V9vcHOPSsdyOp0UJr4z49AcWBO2T a2iMpTNXcO7Mu9/gXQpQHJW5TP1ES3dxEb3mSCR8JrVZBhjA+MnzTiV8FfUuIgNapPEI HO9JE2PFC9xm7JuORgJEMqLA63b1MA1OZLCmEYx2V8iGBbwdlXx6VTwJpa/CJXObICxn Z3uQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Fd4Xsdbo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id d21-20020a170902729500b001d0107ee537si8168695pll.428.2023.12.04.16.46.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 04 Dec 2023 16:46:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Fd4Xsdbo; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 0C98E802208A; Mon, 4 Dec 2023 16:46:44 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346419AbjLEAqO (ORCPT + 99 others); Mon, 4 Dec 2023 19:46:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40184 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231627AbjLEApa (ORCPT ); Mon, 4 Dec 2023 19:45:30 -0500 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 496D2101 for ; Mon, 4 Dec 2023 16:45:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1701737136; x=1733273136; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=YHWGA0FuyU0K8kQA0J/DX6ZgopiBD6JnQLF5bZFmbyE=; b=Fd4Xsdbo6W0b+7GZd0VV7ftC6tx/UKJ2eaYnrdSyWU4HL6VEXJdESeoj IwN0v/I+HIA1xOq2plKX6ZKGHgyIr8szvxedPViptnOffTs9ik1QB5W/c Cg3J28Ondttd3QwbAXj/o8Rt6avodz3Er/ZrkEZrx6Zj8gQvemdeaaNii p1Zda2sjucPzcdzfm31/sERFSgNbLqsbXsSkHc57XEMbFYhk9/z0VJU3S YDbhKR+8pctkSIaJ67mRLWgSoqzQwYtQHyagi7U+RUwocwfqBvdwdmzbo 5V3AJ0dmeV5VKBA8qA9f4zTmOICLD3hgaX+sUfMKoHM3/1ocw11YwtlNl A==; X-IronPort-AV: E=McAfee;i="6600,9927,10914"; a="392688705" X-IronPort-AV: E=Sophos;i="6.04,251,1695711600"; d="scan'208";a="392688705" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Dec 2023 16:45:35 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10914"; a="944067946" X-IronPort-AV: E=Sophos;i="6.04,251,1695711600"; d="scan'208";a="944067946" Received: from abijaz-mobl2.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.61.240]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Dec 2023 16:45:30 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 94C0110A44C; Tue, 5 Dec 2023 03:45:20 +0300 (+03) From: "Kirill A. Shutemov" To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org Cc: "Rafael J. Wysocki" , Peter Zijlstra , Adrian Hunter , Kuppuswamy Sathyanarayanan , Elena Reshetova , Jun Nakajima , Rick Edgecombe , Tom Lendacky , "Kalra, Ashish" , Sean Christopherson , "Huang, Kai" , Baoquan He , kexec@lists.infradead.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv4 10/14] x86/tdx: Convert shared memory back to private on kexec Date: Tue, 5 Dec 2023 03:45:06 +0300 Message-ID: <20231205004510.27164-11-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231205004510.27164-1-kirill.shutemov@linux.intel.com> References: <20231205004510.27164-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Mon, 04 Dec 2023 16:46:44 -0800 (PST) TDX guests allocate shared buffers to perform I/O. It is done by allocating pages normally from the buddy allocator and converting them to shared with set_memory_decrypted(). The second kernel has no idea what memory is converted this way. It only sees E820_TYPE_RAM. Accessing shared memory via private mapping is fatal. It leads to unrecoverable TD exit. On kexec walk direct mapping and convert all shared memory back to private. It makes all RAM private again and second kernel may use it normally. Signed-off-by: Kirill A. Shutemov --- arch/x86/coco/tdx/kexec.c | 0 arch/x86/coco/tdx/tdx.c | 120 +++++++++++++++++++++++++++++++- arch/x86/include/asm/x86_init.h | 1 + arch/x86/kernel/crash.c | 4 ++ arch/x86/kernel/reboot.c | 10 +++ 5 files changed, 133 insertions(+), 2 deletions(-) create mode 100644 arch/x86/coco/tdx/kexec.c diff --git a/arch/x86/coco/tdx/kexec.c b/arch/x86/coco/tdx/kexec.c new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/arch/x86/coco/tdx/tdx.c b/arch/x86/coco/tdx/tdx.c index fcc159497554..46355ea9f4cf 100644 --- a/arch/x86/coco/tdx/tdx.c +++ b/arch/x86/coco/tdx/tdx.c @@ -6,14 +6,17 @@ #include #include +#include #include #include +#include #include #include #include #include #include #include +#include /* MMIO direction */ #define EPT_READ 0 @@ -40,6 +43,9 @@ static atomic_long_t nr_shared; +static atomic_t conversions_in_progress; +static bool conversion_allowed = true; + static inline bool pte_decrypted(pte_t pte) { return cc_mkdec(pte_val(pte)) == pte_val(pte); @@ -725,6 +731,14 @@ static bool tdx_tlb_flush_required(bool private) static bool tdx_cache_flush_required(void) { + /* + * Avoid issuing CLFLUSH on set_memory_decrypted() if conversions + * stopped. Otherwise it can race with unshare_all_memory() and trigger + * implicit conversion to shared. + */ + if (!conversion_allowed) + return false; + /* * AMD SME/SEV can avoid cache flushing if HW enforces cache coherence. * TDX doesn't have such capability. @@ -808,12 +822,25 @@ static bool tdx_enc_status_changed(unsigned long vaddr, int numpages, bool enc) static int tdx_enc_status_change_prepare(unsigned long vaddr, int numpages, bool enc) { + atomic_inc(&conversions_in_progress); + + /* + * Check after bumping conversions_in_progress to serialize + * against tdx_shutdown(). + */ + if (!conversion_allowed) { + atomic_dec(&conversions_in_progress); + return -EBUSY; + } + /* * Only handle shared->private conversion here. * See the comment in tdx_early_init(). */ - if (enc && !tdx_enc_status_changed(vaddr, numpages, enc)) + if (enc && !tdx_enc_status_changed(vaddr, numpages, enc)) { + atomic_dec(&conversions_in_progress); return -EIO; + } return 0; } @@ -825,17 +852,104 @@ static int tdx_enc_status_change_finish(unsigned long vaddr, int numpages, * Only handle private->shared conversion here. * See the comment in tdx_early_init(). */ - if (!enc && !tdx_enc_status_changed(vaddr, numpages, enc)) + if (!enc && !tdx_enc_status_changed(vaddr, numpages, enc)) { + atomic_dec(&conversions_in_progress); return -EIO; + } if (enc) atomic_long_sub(numpages, &nr_shared); else atomic_long_add(numpages, &nr_shared); + atomic_dec(&conversions_in_progress); + return 0; } +static void tdx_kexec_unshare_mem(bool crash) +{ + unsigned long addr, end; + long found = 0, shared; + + /* Stop new private<->shared conversions */ + conversion_allowed = false; + + /* + * Crash kernel reaches here with interrupts disabled: can't wait for + * conversions to finish. + * + * If race happened, just report and proceed. + */ + if (!crash) { + unsigned long timeout; + + /* + * Wait for in-flight conversions to complete. + * + * Do not wait more than 30 seconds. + */ + timeout = 30 * USEC_PER_SEC; + while (atomic_read(&conversions_in_progress) && timeout--) + udelay(1); + } + + if (atomic_read(&conversions_in_progress)) + pr_warn("Failed to finish shared<->private conversions\n"); + + /* + * Walk direct mapping and convert all shared memory back to private, + */ + + addr = PAGE_OFFSET; + end = PAGE_OFFSET + get_max_mapped(); + + while (addr < end) { + unsigned long size; + unsigned int level; + pte_t *pte; + + pte = lookup_address(addr, &level); + size = page_level_size(level); + + if (pte && pte_decrypted(*pte)) { + int pages = size / PAGE_SIZE; + + /* + * Touching memory with shared bit set triggers implicit + * conversion to shared. + * + * Make sure nobody touches the shared range from + * now on. + * + * Bypass unmapping for crash scenario. Unmapping + * requires sleepable context, but in crash case kernel + * hits the code path with interrupts disabled. + * It shouldn't be a problem as all secondary CPUs are + * down and kernel runs with interrupts disabled, so + * there is no room for race. + */ + if (!crash) + set_memory_np(addr, pages); + + if (!tdx_enc_status_changed(addr, pages, true)) { + pr_err("Failed to unshare range %#lx-%#lx\n", + addr, addr + size); + } + + found += pages; + } + + addr += size; + } + + shared = atomic_long_read(&nr_shared); + if (shared != found) { + pr_err("shared page accounting is off\n"); + pr_err("nr_shared = %ld, nr_found = %ld\n", shared, found); + } +} + void __init tdx_early_init(void) { struct tdx_module_args args = { @@ -895,6 +1009,8 @@ void __init tdx_early_init(void) x86_platform.guest.enc_cache_flush_required = tdx_cache_flush_required; x86_platform.guest.enc_tlb_flush_required = tdx_tlb_flush_required; + x86_platform.guest.enc_kexec_unshare_mem = tdx_kexec_unshare_mem; + /* * TDX intercepts the RDMSR to read the X2APIC ID in the parallel * bringup low level code. That raises #VE which cannot be handled diff --git a/arch/x86/include/asm/x86_init.h b/arch/x86/include/asm/x86_init.h index c9503fe2d13a..917358821a31 100644 --- a/arch/x86/include/asm/x86_init.h +++ b/arch/x86/include/asm/x86_init.h @@ -154,6 +154,7 @@ struct x86_guest { int (*enc_status_change_finish)(unsigned long vaddr, int npages, bool enc); bool (*enc_tlb_flush_required)(bool enc); bool (*enc_cache_flush_required)(void); + void (*enc_kexec_unshare_mem)(bool crash); }; /** diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c index c92d88680dbf..1618224775f5 100644 --- a/arch/x86/kernel/crash.c +++ b/arch/x86/kernel/crash.c @@ -40,6 +40,7 @@ #include #include #include +#include /* Used while preparing memory map entries for second kernel */ struct crash_memmap_data { @@ -107,6 +108,9 @@ void native_machine_crash_shutdown(struct pt_regs *regs) crash_smp_send_stop(); + if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT)) + x86_platform.guest.enc_kexec_unshare_mem(true); + cpu_emergency_disable_virtualization(); /* diff --git a/arch/x86/kernel/reboot.c b/arch/x86/kernel/reboot.c index 830425e6d38e..c81afffaa954 100644 --- a/arch/x86/kernel/reboot.c +++ b/arch/x86/kernel/reboot.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -31,6 +32,7 @@ #include #include #include +#include /* * Power off function, if any @@ -716,6 +718,14 @@ static void native_machine_emergency_restart(void) void native_machine_shutdown(void) { + /* + * Call enc_kexec_unshare_mem() while all CPUs are still active and + * interrupts are enabled. This will allow all in-flight memory + * conversions to finish cleanly before unsharing all memory. + */ + if (cc_platform_has(CC_ATTR_GUEST_MEM_ENCRYPT) && kexec_in_progress) + x86_platform.guest.enc_kexec_unshare_mem(false); + /* Stop the cpus and apics */ #ifdef CONFIG_X86_IO_APIC /* -- 2.41.0