Received: by 2002:a05:6a10:22f:0:0:0:0 with SMTP id 15csp640896pxk; Wed, 16 Sep 2020 13:03:33 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxOWDqe7PYzXJa3HSGx3Oii35gCpGBnNU3qAqz8MZLFowUAng90t6oxEKMj6SHqEW9j8QBf X-Received: by 2002:a50:cf8a:: with SMTP id h10mr29960665edk.43.1600286613721; Wed, 16 Sep 2020 13:03:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1600286613; cv=none; d=google.com; s=arc-20160816; b=uGXUmNT9mHp9gbLqLkaAvxF3GIAvYC0VNpZpI84qCaDCHnnV3WUwKQaWswBMzbZBKy 9m64HAQoDINfS4Q8SU3KlIpRKnrTogKyF52GNT8M0Tu1S0/Va5nJmsiBOcjouEPQx+26 9AhMqA47KnJuqaFtyVrmTdjotKJASewXQ57QedfUndFmnLxWPilu5bR9m1JVIk8LWaap DUhQpNHU8QKGpQc81KKGV/UkbGUeiPd2YgNlpKmvtYjJJ1Ox7CLjZ03NMpS0i0xGS6Jt yS/VjAynT3LmecMPn2XojnQdVZoVfuFMCsnfwjB7XsJxG2cBMgsxtiyj6Gohy51qK9p0 LrJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=//UIJKuiVTRRrspCnLdOhX4a4b1lbE5JqtpGTGr3jiw=; b=Z9nFUJbCtiPiCSLPgw2DoA9mbYgZBA+qjKT0s45mAxAv2iQKs6htgyXaDYAqPugVms oasb7CxVOFmPqD6ji0JyDP1usbj7ex+FFDvEqerS35QM0jIRVKpC5o9P/CrJ6rqqj874 fCsMBTzW3kQ4V/AZl0Q3QBgBOZPVBqPYvpuOATOZ0QKOMiCBo1xj3IgUwJUkfRK/KuB3 7p0jJqIju3Eau19xr/XliPN8VaLXXjls2ZoqQ1oloRhia2Jm6Tg1gVuAku9JbYzFo0kY ZvJtTOa2USxefTSjj3Xq7kGPfQoYKF1ovwbA65lRiCQJ99VHe8xLB3WB73WiLuZcgSre 3Ztw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=Z52uL0av; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id i5si12717559edg.62.2020.09.16.13.03.09; Wed, 16 Sep 2020 13:03:33 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=Z52uL0av; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727591AbgIPUBn (ORCPT + 99 others); Wed, 16 Sep 2020 16:01:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33576 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726480AbgIPRfW (ORCPT ); Wed, 16 Sep 2020 13:35:22 -0400 Received: from mail-wr1-x444.google.com (mail-wr1-x444.google.com [IPv6:2a00:1450:4864:20::444]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B0FC9C061354 for ; Wed, 16 Sep 2020 10:34:54 -0700 (PDT) Received: by mail-wr1-x444.google.com with SMTP id t10so7812717wrv.1 for ; Wed, 16 Sep 2020 10:34:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=//UIJKuiVTRRrspCnLdOhX4a4b1lbE5JqtpGTGr3jiw=; b=Z52uL0av7yOsLEyKoqbm0tv4yHwcy/58U0liuI/vgliA5YsLjwCwgMMouzHQn3GKIe qTNMvSzr4j2p1mQd4VOmMVPdomyPgWalVLhFKql9fKekptBcPE784SRU6uPRJ0vKXaLy l8OeDlarQfDToPaJsrreJ78USw9qA4HzEKI79auO0kbaJNZEk8pe7ZmreZUmn1FxZCEs fiGI7n8UsZDRwCV4PSCaDspYD6Fl3vWotR4D7IbOpbHGK4oBpW2mGLvv4LusU7FC+DJ3 JhPHI37Nlg6j6Vr4TTH6F7loZd9dfmsqk+jYUEE1SnEbaK+sdZz93gzLlneJREiiC/vr 2ZbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=//UIJKuiVTRRrspCnLdOhX4a4b1lbE5JqtpGTGr3jiw=; b=QK45obiRUMIzaJttVEcMm9tONf/Y6cJLyC8NCCVg6qo+7g9ugb4CCf2TXB0ldJtBxK SNicF6i5wFVIwTcoEII87i32LiWum/P+v6O1MMun1zXBlN940K5+mPRS7jxPLmYncfoC hpth5Qp48FrUQePSCEqOiPtkQus24mVQ4uHQWbpbTnmJLVj9SgSWmHQjj8sv+XcMEEoz u4UKmySUzNMT8cz27RC+dPgWfBhFmbMP2KTc2LhH7SeYiByzkE7eQl+nz5piiaieLMfe hOy5VQgwfHq+WXo0ChVQ1lj5StvUo8z6++NOcSclIfsObiU07TeraS3qWYtFew6m2fKj vTZg== X-Gm-Message-State: AOAM532OA8g8WbfCrc6vdDgncs1FUDK4O+Zehb/XunynDx9oSxjgmAnl pH1mCc+Pe9AM8grKEdoz5w5xpg== X-Received: by 2002:a5d:540e:: with SMTP id g14mr29024834wrv.148.1600277693139; Wed, 16 Sep 2020 10:34:53 -0700 (PDT) Received: from localhost ([2a01:4b00:8523:2d03:e49d:f6be:d31b:ad3c]) by smtp.gmail.com with ESMTPSA id t16sm34508322wrm.57.2020.09.16.10.34.51 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 16 Sep 2020 10:34:52 -0700 (PDT) From: David Brazdil To: kvmarm@lists.cs.columbia.edu Cc: Catalin Marinas , Will Deacon , Marc Zyngier , James Morse , Julien Thierry , Suzuki K Poulose , Dennis Zhou , Tejun Heo , Christoph Lameter , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kernel-team@android.com, David Brazdil , Andrew Scull Subject: [PATCH v3 04/11] kvm: arm64: Remove __hyp_this_cpu_read Date: Wed, 16 Sep 2020 18:34:32 +0100 Message-Id: <20200916173439.32265-5-dbrazdil@google.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200916173439.32265-1-dbrazdil@google.com> References: <20200916173439.32265-1-dbrazdil@google.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org this_cpu_ptr is meant for use in kernel proper because it selects between TPIDR_EL1/2 based on nVHE/VHE. __hyp_this_cpu_ptr was used in hyp to always select TPIDR_EL2. Unify all users behind this_cpu_ptr and friends by selecting _EL2 register under __KVM_VHE/NVHE_HYPERVISOR__. Under CONFIG_DEBUG_PREEMPT, the kernel helpers perform a preemption check which is omitted by the hyp helpers. Preserve the behavior for nVHE by overriding the corresponding macros under __KVM_NVHE_HYPERVISOR__. Extend the checks into VHE hyp code. Acked-by: Andrew Scull Signed-off-by: David Brazdil --- arch/arm64/include/asm/kvm_asm.h | 20 -------------- arch/arm64/include/asm/percpu.h | 33 +++++++++++++++++++++-- arch/arm64/kvm/hyp/include/hyp/debug-sr.h | 4 +-- arch/arm64/kvm/hyp/include/hyp/switch.h | 8 +++--- arch/arm64/kvm/hyp/nvhe/switch.c | 2 +- arch/arm64/kvm/hyp/vhe/switch.c | 2 +- arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 4 +-- 7 files changed, 41 insertions(+), 32 deletions(-) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index c085032e2e3e..c196eec25498 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -143,26 +143,6 @@ extern char __smccc_workaround_1_smc[__SMCCC_WORKAROUND_1_SMC_SZ]; addr; \ }) -/* - * Home-grown __this_cpu_{ptr,read} variants that always work at HYP, - * provided that sym is really a *symbol* and not a pointer obtained from - * a data structure. As for SHIFT_PERCPU_PTR(), the creative casting keeps - * sparse quiet. - */ -#define __hyp_this_cpu_ptr(sym) \ - ({ \ - void *__ptr; \ - __verify_pcpu_ptr(&sym); \ - __ptr = hyp_symbol_addr(sym); \ - __ptr += read_sysreg(tpidr_el2); \ - (typeof(sym) __kernel __force *)__ptr; \ - }) - -#define __hyp_this_cpu_read(sym) \ - ({ \ - *__hyp_this_cpu_ptr(sym); \ - }) - #define __KVM_EXTABLE(from, to) \ " .pushsection __kvm_ex_table, \"a\"\n" \ " .align 3\n" \ diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h index 0b6409b89e5e..36f304401c38 100644 --- a/arch/arm64/include/asm/percpu.h +++ b/arch/arm64/include/asm/percpu.h @@ -19,7 +19,21 @@ static inline void set_my_cpu_offset(unsigned long off) :: "r" (off) : "memory"); } -static inline unsigned long __my_cpu_offset(void) +static inline unsigned long __hyp_my_cpu_offset(void) +{ + unsigned long off; + + /* + * We want to allow caching the value, so avoid using volatile and + * instead use a fake stack read to hazard against barrier(). + */ + asm("mrs %0, tpidr_el2" : "=r" (off) : + "Q" (*(const unsigned long *)current_stack_pointer)); + + return off; +} + +static inline unsigned long __kern_my_cpu_offset(void) { unsigned long off; @@ -35,7 +49,12 @@ static inline unsigned long __my_cpu_offset(void) return off; } -#define __my_cpu_offset __my_cpu_offset() + +#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__) +#define __my_cpu_offset __hyp_my_cpu_offset() +#else +#define __my_cpu_offset __kern_my_cpu_offset() +#endif #define PERCPU_RW_OPS(sz) \ static inline unsigned long __percpu_read_##sz(void *ptr) \ @@ -227,4 +246,14 @@ PERCPU_RET_OP(add, add, ldadd) #include +/* Redefine macros for nVHE hyp under DEBUG_PREEMPT to avoid its dependencies. */ +#if defined(__KVM_NVHE_HYPERVISOR__) && defined(CONFIG_DEBUG_PREEMPT) +#undef this_cpu_ptr +#define this_cpu_ptr raw_cpu_ptr +#undef __this_cpu_read +#define __this_cpu_read raw_cpu_read +#undef __this_cpu_write +#define __this_cpu_write raw_cpu_write +#endif + #endif /* __ASM_PERCPU_H */ diff --git a/arch/arm64/kvm/hyp/include/hyp/debug-sr.h b/arch/arm64/kvm/hyp/include/hyp/debug-sr.h index 5e28ea6aa097..4ebe9f558f3a 100644 --- a/arch/arm64/kvm/hyp/include/hyp/debug-sr.h +++ b/arch/arm64/kvm/hyp/include/hyp/debug-sr.h @@ -135,7 +135,7 @@ static inline void __debug_switch_to_guest_common(struct kvm_vcpu *vcpu) if (!(vcpu->arch.flags & KVM_ARM64_DEBUG_DIRTY)) return; - host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt; + host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt; guest_ctxt = &vcpu->arch.ctxt; host_dbg = &vcpu->arch.host_debug_state.regs; guest_dbg = kern_hyp_va(vcpu->arch.debug_ptr); @@ -154,7 +154,7 @@ static inline void __debug_switch_to_host_common(struct kvm_vcpu *vcpu) if (!(vcpu->arch.flags & KVM_ARM64_DEBUG_DIRTY)) return; - host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt; + host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt; guest_ctxt = &vcpu->arch.ctxt; host_dbg = &vcpu->arch.host_debug_state.regs; guest_dbg = kern_hyp_va(vcpu->arch.debug_ptr); diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 5b6b8fa00f0a..f150407fa798 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -386,7 +386,7 @@ static inline bool __hyp_handle_ptrauth(struct kvm_vcpu *vcpu) !esr_is_ptrauth_trap(kvm_vcpu_get_esr(vcpu))) return false; - ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt; + ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt; __ptrauth_save_key(ctxt, APIA); __ptrauth_save_key(ctxt, APIB); __ptrauth_save_key(ctxt, APDA); @@ -495,7 +495,7 @@ static inline void __set_guest_arch_workaround_state(struct kvm_vcpu *vcpu) * guest wants it disabled, so be it... */ if (__needs_ssbd_off(vcpu) && - __hyp_this_cpu_read(arm64_ssbd_callback_required)) + __this_cpu_read(arm64_ssbd_callback_required)) arm_smccc_1_1_smc(ARM_SMCCC_ARCH_WORKAROUND_2, 0, NULL); #endif } @@ -507,7 +507,7 @@ static inline void __set_host_arch_workaround_state(struct kvm_vcpu *vcpu) * If the guest has disabled the workaround, bring it back on. */ if (__needs_ssbd_off(vcpu) && - __hyp_this_cpu_read(arm64_ssbd_callback_required)) + __this_cpu_read(arm64_ssbd_callback_required)) arm_smccc_1_1_smc(ARM_SMCCC_ARCH_WORKAROUND_2, 1, NULL); #endif } @@ -521,7 +521,7 @@ static inline void __kvm_unexpected_el2_exception(void) entry = hyp_symbol_addr(__start___kvm_ex_table); end = hyp_symbol_addr(__stop___kvm_ex_table); - host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt; + host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt; while (entry < end) { addr = (unsigned long)&entry->insn + entry->insn; diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index 0970442d2dbc..cc4f8e790fb3 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -175,7 +175,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu) vcpu = kern_hyp_va(vcpu); - host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt; + host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt; host_ctxt->__hyp_running_vcpu = vcpu; guest_ctxt = &vcpu->arch.ctxt; diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index c1da4f86ccac..575e8054f116 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -108,7 +108,7 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) struct kvm_cpu_context *guest_ctxt; u64 exit_code; - host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt; + host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt; host_ctxt->__hyp_running_vcpu = vcpu; guest_ctxt = &vcpu->arch.ctxt; diff --git a/arch/arm64/kvm/hyp/vhe/sysreg-sr.c b/arch/arm64/kvm/hyp/vhe/sysreg-sr.c index 996471e4c138..2a0b8c88d74f 100644 --- a/arch/arm64/kvm/hyp/vhe/sysreg-sr.c +++ b/arch/arm64/kvm/hyp/vhe/sysreg-sr.c @@ -66,7 +66,7 @@ void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu) struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt; struct kvm_cpu_context *host_ctxt; - host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt; + host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt; __sysreg_save_user_state(host_ctxt); /* @@ -100,7 +100,7 @@ void kvm_vcpu_put_sysregs_vhe(struct kvm_vcpu *vcpu) struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt; struct kvm_cpu_context *host_ctxt; - host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt; + host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt; deactivate_traps_vhe_put(); __sysreg_save_el1_state(guest_ctxt); -- 2.28.0.618.gf4bc123cb7-goog