Received: by 2002:ab2:6d45:0:b0:1fb:d597:ff75 with SMTP id d5csp174185lqr; Wed, 5 Jun 2024 02:34:37 -0700 (PDT) X-Forwarded-Encrypted: i=3; AJvYcCW3PZLlPpX0CxlvjvYY1adVr/h3mrgL87kuHzWHeY3Z9qJ94miXW6JV/qdIWqeAdyq2EkMDLbTJw83hhFm+y8U/aX6eEZBBj29D7PVlWA== X-Google-Smtp-Source: AGHT+IF0GeJFoJbX9U7SzQWWSK7ODA5ZyGfroxD8kRtpskz9Zdriz+0RcvUctSkrBrlxHG+Zj8Cx X-Received: by 2002:a05:6122:2004:b0:4d8:7a5e:392f with SMTP id 71dfb90a1353d-4eb3a4a8ecbmr2807889e0c.12.1717580077062; Wed, 05 Jun 2024 02:34:37 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1717580077; cv=pass; d=google.com; s=arc-20160816; b=WnFRAN65g48Wvb+Lz1aoInaGjLi73RXckc1pMuV25/YGDyfyTOE4/4zLkwgL5dRnPN tktSqRZoYcyK/1tcpr+U58SxKDKfBYFB5v+k8cULtA+hulB8D1jjvLk7ULxnMp8MMxaW keezue0voo28pOAeFDg5J1ttDIpPpHGmlJZg8aOqLT9XLK2JZO2cS9CsZwqyuafIqo9a q1P/NN0Pf0IxU5QIgwAiFy2CGdcJGf/qeUk7fZvhxndctwFVQ40+bfdbhwl2q9M07V1v vRZ5FjBc3WfsjHJyJyCy/l0ydqHRiHhTYveYITAp+rVeR0nb49PXZWwolQ67trMybbwF 7QmA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from; bh=Lmoo49M0SkdiUnawptA5XaD6c+V/s8bzPSv+72JkKqM=; fh=mU6xl5FN63PgqIKyLHx/u5uRervM0V89cYoVjXl22Bs=; b=yQqa6/IewvD2WfHnSQA+O8WUqGuF7xez2+RA9qge9A3u58o+ddZfV3YbknlUPG5J0C feI4keH8gd4C5lkD3Bi/CUQP6eRbS/KOhOmNZ7RuntiRaGMA6Esv1ZoT/SYjN+iA24Qr aTLwXaxhFosaQMvVg2jxacBVuqx3G6JFwExTl1JMYfTz9MsOqlCDccYtSPe3jSsg7F+x w3ebSITK08SqfnmNUxcSGlUANLSMsaYo0NQrZjS9NPkmAwo/k0upmlVhbHOkYsDf03xc mYphyLge0PJy4bj0MSCpBZyPlKPL0LDC5jpSBr5bkmsgnv4BN+eL2G4FbOg+wdAmmylV ZRGA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-202108-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-202108-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Return-Path: Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id af79cd13be357-794f32bd663si47156785a.760.2024.06.05.02.34.36 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 05 Jun 2024 02:34:37 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel+bounces-202108-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; arc=pass (i=1 spf=pass spfdomain=arm.com dmarc=pass fromdomain=arm.com); spf=pass (google.com: domain of linux-kernel+bounces-202108-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-202108-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 987A01C235AC for ; Wed, 5 Jun 2024 09:34:36 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 05980194A63; Wed, 5 Jun 2024 09:30:58 +0000 (UTC) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D1C54190045; Wed, 5 Jun 2024 09:30:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717579857; cv=none; b=W2jB45gsRBUXsU2AS5PC1NW8qLdKTumfaCHMjzCPfc1rKc7TmrbTI7O4ou9oIYh/nvHQf/OLqTKOvyLxn/lPgUsYIKH/0g6cj58sP86rf6QPmPOcgVmVnJbQv/VJXkwMq1tZL63KXwPkYPSNMX6Wci4ivy20b4VC8i8qNqIhkiY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717579857; c=relaxed/simple; bh=gmlsJyB6T3MNrPgQ/q4sZa5vwmD5AVjFI3kLXf/9P6o=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=PtIbVNcJU7ggSfRgp9neXzn/2vqmPvDyjUqZbxzgNBQnTKstUrbqfLwVPXVcy5XnIs9h6Ty5i+0yPdWx7CSt8U/5XAPWc7dnzbrc10Z64Y31KWdmVCVs2c9jQzrSEFy+YNSfS45Kp17iu2ZAAKMthq1g2DIqqBC3+IRxWbq/q18= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BC250FEC; Wed, 5 Jun 2024 02:31:19 -0700 (PDT) Received: from e122027.arm.com (unknown [10.57.39.129]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id AEC353F792; Wed, 5 Jun 2024 02:30:51 -0700 (PDT) From: Steven Price To: kvm@vger.kernel.org, kvmarm@lists.linux.dev Cc: Suzuki K Poulose , Catalin Marinas , Marc Zyngier , Will Deacon , James Morse , Oliver Upton , Zenghui Yu , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Joey Gouly , Alexandru Elisei , Christoffer Dall , Fuad Tabba , linux-coco@lists.linux.dev, Ganapatrao Kulkarni , Steven Price Subject: [PATCH v3 09/14] arm64: Enable memory encrypt for Realms Date: Wed, 5 Jun 2024 10:30:01 +0100 Message-Id: <20240605093006.145492-10-steven.price@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240605093006.145492-1-steven.price@arm.com> References: <20240605093006.145492-1-steven.price@arm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: Suzuki K Poulose Use the memory encryption APIs to trigger a RSI call to request a transition between protected memory and shared memory (or vice versa) and updating the kernel's linear map of modified pages to flip the top bit of the IPA. This requires that block mappings are not used in the direct map for realm guests. Signed-off-by: Suzuki K Poulose Co-developed-by: Steven Price Signed-off-by: Steven Price --- Changes since v2: * Fix location of set_memory_{en,de}crypted() and export them. * Break-before-make when changing the top bit of the IPA for transitioning to/from shared. --- arch/arm64/Kconfig | 3 ++ arch/arm64/include/asm/mem_encrypt.h | 17 ++++++++ arch/arm64/include/asm/set_memory.h | 3 ++ arch/arm64/kernel/rsi.c | 12 +++++ arch/arm64/mm/pageattr.c | 65 ++++++++++++++++++++++++++-- 5 files changed, 97 insertions(+), 3 deletions(-) create mode 100644 arch/arm64/include/asm/mem_encrypt.h diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 5d91259ee7b5..0f1480caeeec 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -20,6 +20,7 @@ config ARM64 select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2 select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE select ARCH_HAS_CACHE_LINE_SIZE + select ARCH_HAS_CC_PLATFORM select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL select ARCH_HAS_DEBUG_VM_PGTABLE @@ -41,6 +42,8 @@ config ARM64 select ARCH_HAS_SETUP_DMA_OPS select ARCH_HAS_SET_DIRECT_MAP select ARCH_HAS_SET_MEMORY + select ARCH_HAS_MEM_ENCRYPT + select ARCH_HAS_FORCE_DMA_UNENCRYPTED select ARCH_STACKWALK select ARCH_HAS_STRICT_KERNEL_RWX select ARCH_HAS_STRICT_MODULE_RWX diff --git a/arch/arm64/include/asm/mem_encrypt.h b/arch/arm64/include/asm/mem_encrypt.h new file mode 100644 index 000000000000..e47265cd180a --- /dev/null +++ b/arch/arm64/include/asm/mem_encrypt.h @@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2023 ARM Ltd. + */ + +#ifndef __ASM_MEM_ENCRYPT_H +#define __ASM_MEM_ENCRYPT_H + +#include + +/* All DMA must be to non-secure memory for now */ +static inline bool force_dma_unencrypted(struct device *dev) +{ + return is_realm_world(); +} + +#endif diff --git a/arch/arm64/include/asm/set_memory.h b/arch/arm64/include/asm/set_memory.h index 0f740b781187..3b6619c04677 100644 --- a/arch/arm64/include/asm/set_memory.h +++ b/arch/arm64/include/asm/set_memory.h @@ -14,4 +14,7 @@ int set_direct_map_invalid_noflush(struct page *page); int set_direct_map_default_noflush(struct page *page); bool kernel_page_present(struct page *page); +int set_memory_encrypted(unsigned long addr, int numpages); +int set_memory_decrypted(unsigned long addr, int numpages); + #endif /* _ASM_ARM64_SET_MEMORY_H */ diff --git a/arch/arm64/kernel/rsi.c b/arch/arm64/kernel/rsi.c index 5cb42609219f..898952d135b0 100644 --- a/arch/arm64/kernel/rsi.c +++ b/arch/arm64/kernel/rsi.c @@ -6,6 +6,7 @@ #include #include #include +#include #include @@ -19,6 +20,17 @@ unsigned int phys_mask_shift = CONFIG_ARM64_PA_BITS; DEFINE_STATIC_KEY_FALSE_RO(rsi_present); EXPORT_SYMBOL(rsi_present); +bool cc_platform_has(enum cc_attr attr) +{ + switch (attr) { + case CC_ATTR_MEM_ENCRYPT: + return is_realm_world(); + default: + return false; + } +} +EXPORT_SYMBOL_GPL(cc_platform_has); + static bool rsi_version_matches(void) { unsigned long ver_lower, ver_higher; diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 0e270a1c51e6..3e7d81696756 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -5,10 +5,12 @@ #include #include #include +#include #include #include #include +#include #include #include #include @@ -23,14 +25,16 @@ bool rodata_full __ro_after_init = IS_ENABLED(CONFIG_RODATA_FULL_DEFAULT_ENABLED bool can_set_direct_map(void) { /* - * rodata_full and DEBUG_PAGEALLOC require linear map to be - * mapped at page granularity, so that it is possible to + * rodata_full, DEBUG_PAGEALLOC and a Realm guest all require linear + * map to be mapped at page granularity, so that it is possible to * protect/unprotect single pages. * * KFENCE pool requires page-granular mapping if initialized late. + * + * Realms need to make pages shared/protected at page granularity. */ return rodata_full || debug_pagealloc_enabled() || - arm64_kfence_can_set_direct_map(); + arm64_kfence_can_set_direct_map() || is_realm_world(); } static int change_page_range(pte_t *ptep, unsigned long addr, void *data) @@ -192,6 +196,61 @@ int set_direct_map_default_noflush(struct page *page) PAGE_SIZE, change_page_range, &data); } +static int __set_memory_encrypted(unsigned long addr, + int numpages, + bool encrypt) +{ + unsigned long set_prot = 0, clear_prot = 0; + phys_addr_t start, end; + int ret; + + if (!is_realm_world()) + return 0; + + if (!__is_lm_address(addr)) + return -EINVAL; + + start = __virt_to_phys(addr); + end = start + numpages * PAGE_SIZE; + + /* + * Break the mapping before we make any changes to avoid stale TLB + * entries or Synchronous External Aborts caused by RIPAS_EMPTY + */ + ret = __change_memory_common(addr, PAGE_SIZE * numpages, + __pgprot(0), + __pgprot(PTE_VALID)); + + if (encrypt) { + clear_prot = PROT_NS_SHARED; + ret = rsi_set_memory_range_protected(start, end); + } else { + set_prot = PROT_NS_SHARED; + ret = rsi_set_memory_range_shared(start, end); + } + + if (ret) + return ret; + + set_prot |= PTE_VALID; + + return __change_memory_common(addr, PAGE_SIZE * numpages, + __pgprot(set_prot), + __pgprot(clear_prot)); +} + +int set_memory_encrypted(unsigned long addr, int numpages) +{ + return __set_memory_encrypted(addr, numpages, true); +} +EXPORT_SYMBOL_GPL(set_memory_encrypted); + +int set_memory_decrypted(unsigned long addr, int numpages) +{ + return __set_memory_encrypted(addr, numpages, false); +} +EXPORT_SYMBOL_GPL(set_memory_decrypted); + #ifdef CONFIG_DEBUG_PAGEALLOC void __kernel_map_pages(struct page *page, int numpages, int enable) { -- 2.34.1