Received: by 2002:a05:6a10:f347:0:0:0:0 with SMTP id d7csp4033437pxu; Wed, 9 Dec 2020 06:51:45 -0800 (PST) X-Google-Smtp-Source: ABdhPJyxmMFIQvrSSdsQPCxQ6AykpzATmVvTg8PUnaiiHog6W+zkJd1T9PGQCYwmB3r1dTXzyk46 X-Received: by 2002:a17:906:e093:: with SMTP id gh19mr2390610ejb.510.1607525504981; Wed, 09 Dec 2020 06:51:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1607525504; cv=none; d=google.com; s=arc-20160816; b=UTBViT3/PiXhMZPwG9G5ULau8a9EshpiSNa6t7Iw29F5QNe1MR6E96cVjmkJdmMG8v 4c4Eqs9UBaimZvqbQ7fp8R2TrxIhHihdV+v981fMs0sXEyUUOxtpYGd/ESsrjBaqMCRD gjNVz0qBwRAjNXH/T9Ev3VYwj2SlbcbZjZXfaSxIoq3vqkh91Q3CIAVBvabIYBHljJN4 pvp4CgYqd5dJk4ZurLiwvd5oSGmDBefGaADmBFjDAHzUz9KupBsCpEdjdoHEuI1e1mje BYeygAXPlLM5PH1Vb+8GATYL6pgHJLa9GNA3FX8vloVVHAmfW25GOrmF9EAayeQ/i+9o FkUw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=AnSc1Sgxbdzkn7VrpHIozks65PIKCh74xA+FbntzNXY=; b=tU9BkeJLEIqp9oJQWAUjkwrWxNP1GlRAYzkt6LVo3TsGJ06Obb+XiVwfW4Fx4nNYiB hd+hsBagN6RJcLi6G6OgMM6fAwYRdFyYJPetJgfG02xqxzf9qZDpZCeCxAqgQ9A1uc5s TmqvT2V3fEAjzFDs5P82XWYY42/X2ii1NREo4phZvcR4lTDIgJlB3Y/Ww4ZEvhvDmSFm T4rXqRDKnrJ6uQYkAdaNqwpQo6vzpY4+RGisBd4w+snqV2v1OO5nH9Ne50yg5bdXvgcg s6b52CTEojELjJTNGJ5emyK/ir/jyqd2iDXxGL19KqgADeu/9C15K/zxD0fR2yyXfBFC 1sCw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=fV0IqsCK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id i8si1023483edv.161.2020.12.09.06.51.18; Wed, 09 Dec 2020 06:51:44 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=fV0IqsCK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732315AbgLINTe (ORCPT + 99 others); Wed, 9 Dec 2020 08:19:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53774 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732239AbgLINTM (ORCPT ); Wed, 9 Dec 2020 08:19:12 -0500 Received: from mail-wr1-x441.google.com (mail-wr1-x441.google.com [IPv6:2a00:1450:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EE259C0617A7 for ; Wed, 9 Dec 2020 05:18:03 -0800 (PST) Received: by mail-wr1-x441.google.com with SMTP id c1so1703992wrq.6 for ; Wed, 09 Dec 2020 05:18:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AnSc1Sgxbdzkn7VrpHIozks65PIKCh74xA+FbntzNXY=; b=fV0IqsCKJt3ivo+aspsTCn4tySosMb+82n5RnA2sAYTUUnT7GT1fshweSpCBAReyj/ 2SoMtQDPP3FnvvJAKFaqDKH5dcWNHZqK/CZL7MGg9dFP7zD8KCf5PZ2a6BBcicj977Ou 0zgxnB/Skz+GAbr34KDFIGm1SWSVJpJ4mJqz8BgANsF6kW6dH0nSYakL6ykDEUf64EjV KnZWuetkZ8TvwYngLljRR9HKZHMH6bIxtjytbs5/EALMbZ/N/nU5gn7BWB3n4tc72taH T4YeQt7WHrQgQmEdBWJY/V/9Ypebzu9I3VWoWfG9dWXZ4KSomVBQkPfeyuxeCJJ1U/Cq 1mEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AnSc1Sgxbdzkn7VrpHIozks65PIKCh74xA+FbntzNXY=; b=dFZa9QaF8HGf/Sua9MzKzhe79l5a3XFnPvIOv0KzIM2VQ5H0yyRi1fMrAewoS5IIuJ WoDebPlIlxak9L5l3qilozX9ofHZr1pCDRfODAj3eJoNtzRDlUihvoT3x4agJTNEU3lU obvddd6vuxXgWyG/lNVWG8j/n5hkdYAwNCJ3dqzGbwaq6oinhOouVPnsMQICu7h4kWr0 NYebLzaOQXpdb77JRdGJ7BVxvMKbZEEoqAgS9gxvaqoySH9inpznlq3nDW5IjXIo8EXE YCT1V512w16mgoRfwImdkx6lO8MEB55gTydEmMyVDDIGD0rMglZyRwBXAjy/eE2BobbB vAEg== X-Gm-Message-State: AOAM533UZHZy1eEMMkzylsMKw/cqqvFcuDqduC0HhZdshlkZwp/ifgU/ htNwsb27roLzaglDcq3rWmU8Hg== X-Received: by 2002:adf:97ce:: with SMTP id t14mr2716349wrb.368.1607519882606; Wed, 09 Dec 2020 05:18:02 -0800 (PST) Received: from localhost ([2a01:4b00:8523:2d03:9d1b:d0eb:db43:6cd2]) by smtp.gmail.com with ESMTPSA id k11sm3202279wmj.42.2020.12.09.05.18.01 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 09 Dec 2020 05:18:01 -0800 (PST) From: David Brazdil To: kvmarm@lists.cs.columbia.edu Cc: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Ard Biesheuvel , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kernel-team@android.com, David Brazdil Subject: [PATCH 6/9] KVM: arm64: Apply hyp relocations at runtime Date: Wed, 9 Dec 2020 13:17:43 +0000 Message-Id: <20201209131746.85622-7-dbrazdil@google.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201209131746.85622-1-dbrazdil@google.com> References: <20201209131746.85622-1-dbrazdil@google.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org KVM nVHE code runs under a different VA mapping than the kernel, hence so far it avoided using absolute addressing because the VA in a constant pool is relocated by the linker to a kernel VA (see hyp_symbol_addr). Now the kernel has access to a list of positions that contain a kimg VA but will be accessed only in hyp execution context. These are generated by the gen-hyprel build-time tool and stored in .hyp.reloc. Add early boot pass over the entries and convert the kimg VAs to hyp VAs. Note that this requires for .hyp* ELF sections to be mapped read-write at that point. Signed-off-by: David Brazdil --- arch/arm64/configs/defconfig | 1 + arch/arm64/include/asm/kvm_mmu.h | 1 + arch/arm64/include/asm/sections.h | 1 + arch/arm64/kernel/smp.c | 4 +++- arch/arm64/kvm/va_layout.c | 28 ++++++++++++++++++++++++++++ 5 files changed, 34 insertions(+), 1 deletion(-) diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig index 5cfe3cf6f2ac..73fc9f2f2661 100644 --- a/arch/arm64/configs/defconfig +++ b/arch/arm64/configs/defconfig @@ -1092,3 +1092,4 @@ CONFIG_DEBUG_KERNEL=y # CONFIG_DEBUG_PREEMPT is not set # CONFIG_FTRACE is not set CONFIG_MEMTEST=y +# CONFIG_ARM64_VHE is not set diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index e52d82aeadca..6bbb44011c84 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -129,6 +129,7 @@ alternative_cb_end void kvm_update_va_mask(struct alt_instr *alt, __le32 *origptr, __le32 *updptr, int nr_inst); void kvm_compute_layout(void); +void kvm_apply_hyp_relocations(void); static __always_inline unsigned long __kern_hyp_va(unsigned long v) { diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h index a6f3557d1ab2..2f36b16a5b5d 100644 --- a/arch/arm64/include/asm/sections.h +++ b/arch/arm64/include/asm/sections.h @@ -12,6 +12,7 @@ extern char __hibernate_exit_text_start[], __hibernate_exit_text_end[]; extern char __hyp_idmap_text_start[], __hyp_idmap_text_end[]; extern char __hyp_text_start[], __hyp_text_end[]; extern char __hyp_rodata_start[], __hyp_rodata_end[]; +extern char __hyp_reloc_begin[], __hyp_reloc_end[]; extern char __idmap_text_start[], __idmap_text_end[]; extern char __initdata_begin[], __initdata_end[]; extern char __inittext_begin[], __inittext_end[]; diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c index 18e9727d3f64..47142395bc91 100644 --- a/arch/arm64/kernel/smp.c +++ b/arch/arm64/kernel/smp.c @@ -434,8 +434,10 @@ static void __init hyp_mode_check(void) "CPU: CPUs started in inconsistent modes"); else pr_info("CPU: All CPU(s) started at EL1\n"); - if (IS_ENABLED(CONFIG_KVM)) + if (IS_ENABLED(CONFIG_KVM)) { kvm_compute_layout(); + kvm_apply_hyp_relocations(); + } } void __init smp_cpus_done(unsigned int max_cpus) diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c index d8cc51bd60bf..fb2ca02b7270 100644 --- a/arch/arm64/kvm/va_layout.c +++ b/arch/arm64/kvm/va_layout.c @@ -82,6 +82,34 @@ __init void kvm_compute_layout(void) init_hyp_physvirt_offset(); } +/* + * The .hyp.reloc ELF section contains a list of kimg positions that + * contains kimg VAs but will be accessed only in hyp execution context. + * Convert them to hyp VAs. See gen-hyprel.c for more details. + */ +__init void kvm_apply_hyp_relocations(void) +{ + int32_t *rel; + int32_t *begin = (int32_t*)__hyp_reloc_begin; + int32_t *end = (int32_t*)__hyp_reloc_end; + + for (rel = begin; rel < end; ++rel) { + uintptr_t *ptr, kimg_va; + + /* + * Each entry contains a 32-bit relative offset from itself + * to a kimg VA position. + */ + ptr = (uintptr_t*)lm_alias((char*)rel + *rel); + + /* Read the kimg VA value at the relocation address. */ + kimg_va = *ptr; + + /* Convert to hyp VA and store back to the relocation address. */ + *ptr = __early_kern_hyp_va((uintptr_t)lm_alias(kimg_va)); + } +} + static u32 compute_instruction(int n, u32 rd, u32 rn) { u32 insn = AARCH64_BREAK_FAULT; -- 2.29.2.576.ga3fc446d84-goog