Received: by 2002:a05:6a10:8c0a:0:0:0:0 with SMTP id go10csp857103pxb; Wed, 3 Feb 2021 21:38:40 -0800 (PST) X-Google-Smtp-Source: ABdhPJxO70Zo0+vQgHvrMqeCbXmmB1dQFrKPoTQitgkoeGk3r/2qlrJn7ehkNg8Q+iq1fHmWIkzf X-Received: by 2002:a05:6402:702:: with SMTP id w2mr6616227edx.78.1612417120434; Wed, 03 Feb 2021 21:38:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1612417120; cv=none; d=google.com; s=arc-20160816; b=ETDw9DtzWEplBFg9HaBFkOyMVYb5V732j7CXTzU2NHKYK/DqJkOIRacdpbQ8U4jCbN tl4bP0Xgkbi8MqN+jJ6NiRuAFWDLRIUEyUKsQ9Hbgtktc2606mfNxi102Wh4qyjLG2ai AiQ3a8iiwI2NwXmsj4P4GA0BrAai1PLAykCI4/06Ma4SwdTtGFEGXTC79KkfclJL4ouV r7wp8OBVEV751NVRIQCniTQIo9EEgVWoGbP1Fh+wepO8DPfcHnAfquyEmHUEfF8tN+ie DRXN1MlE97h+YgSfQONn+Z/ua4UQz/k2D5BlAKq2NCnCflAZM8osZkg8ye/7kpdAmBfe eTfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :wdcironportexception:ironport-sdr:ironport-sdr:ironport-sdr :dkim-signature; bh=/87GTN2pGljOTwmyjeHEh8t2nfCMTg57me4beMhrFak=; b=QGT1yhDqmJBmC1uWE9/C8WnhxX4ksM24+Oz931LlcTOSoX2LZGdAzhUcTBLqaZ8Bg4 Lh5TClFk7zpUO7g3Gy1AL6NWs/GELQbMzRIkGrkYFgfDF3bv4vbUMNcPapIE/55WtSyt EVJ4EvPKyZsLYpPfctekQHSTMtswSvoM6XVEdw19KBmhPfTK51vAjxZ5CmG9VFX1n6ys s1A5sLnFqaXCqpiuuoE/+smY3vsgiqUaYh6lFXJMXnpbE/CrsuU1mBQUClqdWdVGmCH8 DnvQNb0my85DTgHxixsxEj/5VUiMir9wKPTOsUzAOx2bzzhxCMXa1BN2pbWlBgjFDfje rFtQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@wdc.com header.s=dkim.wdc.com header.b=KEnr6P13; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=wdc.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id g21si2661381edb.12.2021.02.03.21.38.16; Wed, 03 Feb 2021 21:38:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=fail header.i=@wdc.com header.s=dkim.wdc.com header.b=KEnr6P13; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=wdc.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232151AbhBDFfn (ORCPT + 99 others); Thu, 4 Feb 2021 00:35:43 -0500 Received: from esa5.hgst.iphmx.com ([216.71.153.144]:38827 "EHLO esa5.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229790AbhBDFfc (ORCPT ); Thu, 4 Feb 2021 00:35:32 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wdc.com; i=@wdc.com; q=dns/txt; s=dkim.wdc.com; t=1612416931; x=1643952931; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=iJJFwG2gkbnbjw60ClRHydP01kI34Va6crHsBOaLVlg=; b=KEnr6P134Z4Qp/s2SRNFi5mK0tLVHr17m+FgzvBKIlt/v3dmoOPBAMja ZnkDiCcF58KXjBzHw+EfpH3+vKwtaFzWQv8M9W/ryiTXCTnaBx1iHiZfY ZccxYY7tOZZRJJmGQOSYuHLbvQ3tn50LCCpijMzXGXRlOipYF0VMyv8pk V2krB+iLpaLZOBWi6WHy2phjft09ym6tQCfhWnhXkmlK/G3d63ctOMV5V 3FHwkIsLUtuP+5EC1owi1lVH7wbjhzyH2ZCT5IRrPISZtfgnIQjj/MXp5 Zc9TYY4hhWMsQrfURflaesjLczV7lr6fdio1nfM+VLZ9jIUNrjOfBBjfH Q==; IronPort-SDR: fKfwEcZ5E/5Tyje9b8TjhYoILLgqYpkxi05PiI1fz0+8uHx+SwZO4tbcpKoKzCjQ8+IB7CMV1V yjeWXt+Xj7weoo3G4/IpuHN8S8m/myOr3CGXmsNuHpN+kdBiCDGvUynds8ws85h6J+eBJcqeBS cpXJRaMVJ8c6L2axSWKPfO6V6QWfrH/WB9mH4Adw8YS9NnDQIAt0hS2J2BK0JIC7A+6FGsAw6r 72pxieo26AcxZfomSbyAfDMiHrwxZ6PH+eIrWRCwM9c3j8RgrW8vQRw0IFIfGiqKFeC579v0SO jfA= X-IronPort-AV: E=Sophos;i="5.79,400,1602518400"; d="scan'208";a="159086449" Received: from uls-op-cesaip01.wdc.com (HELO uls-op-cesaep01.wdc.com) ([199.255.45.14]) by ob1.hgst.iphmx.com with ESMTP; 04 Feb 2021 13:32:55 +0800 IronPort-SDR: QGKVbJ8XpEX2PGFHNPWSr5IuHvnbFVCZqtlw7FHZDZ7VT/wkFqwSRUIOp0Bu1geSn8YXvipj5k rcDPNMw8uFOmfy5nk2vqUsf8tsGzh6KvC4MYkXb0owWZzi297ejbP5l2L3CZ5rxL5ARdbd02Hl E4We6ESHI9VMZS6qzXT5b3tzAoO7NHOhE4vFSwcIzLSVRwxUVnfmsQzGbJPWfPva7yElpkt547 feZY2NJ9CalTDipmwdqxHbtzqZkJohOfwxRxhFJuRmU65pzwNeK74up+6wnEnzfx/GZi4SA4Kj ys9TWHmICGzawY5yi1/3t5tz Received: from uls-op-cesaip02.wdc.com ([10.248.3.37]) by uls-op-cesaep01.wdc.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Feb 2021 21:16:54 -0800 IronPort-SDR: 4R48TJ4/jE8aBRDzH1Bve/9tnQIAQY1hzu5HSv0ZaOglyETzd51vYvYkiNrKl5znvMCta/BT6S 4PNQQ1106jmLs63vgZ1ARujbRdG4YRHR8ZKY83MfEd8We+BXODBSthzWf/ieH/Fb2Kz18b+PWy gLchyCfdpYZJexMrtvAcs0msiz8d/w9NupwdzdCpN3XAc8ghOTsd9S3q4ttPAq1l9cMAFnyiN9 kVXh7g1qoV+VabhHIQ0FBd1iSwl/MvL/fFhhOe76XWTfU4JOpwA/9/TlQranVwVcrAEHcvDKbv 1sM= WDCIronportException: Internal Received: from cnf008142.ad.shared (HELO jedi-01.hgst.com) ([10.86.63.165]) by uls-op-cesaip02.wdc.com with ESMTP; 03 Feb 2021 21:32:55 -0800 From: Atish Patra To: linux-kernel@vger.kernel.org Cc: Atish Patra , Albert Ou , Anup Patel , Kefeng Wang , kvm-riscv@lists.infradead.org, kvm@vger.kernel.org, linux-riscv@lists.infradead.org, Palmer Dabbelt , Paul Walmsley Subject: [PATCH v2 5/6] RISC-V: Add SBI HSM extension in KVM Date: Wed, 3 Feb 2021 21:32:38 -0800 Message-Id: <20210204053239.1609558-6-atish.patra@wdc.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210204053239.1609558-1-atish.patra@wdc.com> References: <20210204053239.1609558-1-atish.patra@wdc.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org SBI HSM extension allows OS to start/stop harts any time. It also allows ordered booting of harts instead of random booting. Implement SBI HSM exntesion and designate the vcpu 0 as the boot vcpu id. All other non-zero non-booting vcpus should be brought up by the OS implementing HSM extension. If the guest OS doesn't implement HSM extension, only single vcpu will be available to OS. Signed-off-by: Atish Patra --- arch/riscv/include/asm/sbi.h | 1 + arch/riscv/kvm/Makefile | 2 +- arch/riscv/kvm/vcpu.c | 19 ++++++ arch/riscv/kvm/vcpu_sbi.c | 4 ++ arch/riscv/kvm/vcpu_sbi_hsm.c | 109 ++++++++++++++++++++++++++++++++++ 5 files changed, 134 insertions(+), 1 deletion(-) create mode 100644 arch/riscv/kvm/vcpu_sbi_hsm.c diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index 4a405f583d32..881e89078785 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -106,6 +106,7 @@ enum sbi_srst_reset_reason { #define SBI_ERR_INVALID_PARAM -3 #define SBI_ERR_DENIED -4 #define SBI_ERR_INVALID_ADDRESS -5 +#define SBI_ERR_ALREADY_AVAILABLE -6 extern unsigned long sbi_spec_version; struct sbiret { diff --git a/arch/riscv/kvm/Makefile b/arch/riscv/kvm/Makefile index 302501295397..5627c9c7f249 100644 --- a/arch/riscv/kvm/Makefile +++ b/arch/riscv/kvm/Makefile @@ -11,5 +11,5 @@ kvm-objs := $(common-objs-y) kvm-objs += main.o vm.o vmid.o tlb.o mmu.o kvm-objs += vcpu.o vcpu_exit.o vcpu_switch.o vcpu_timer.o -kvm-objs += vcpu_sbi.o vcpu_sbi_base.o vcpu_sbi_legacy.o vcpu_sbi_replace.o +kvm-objs += vcpu_sbi.o vcpu_sbi_base.o vcpu_sbi_legacy.o vcpu_sbi_replace.o vcpu_sbi_hsm.o obj-$(CONFIG_KVM) += kvm.o diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c index bcc4af9d2fa9..99f94ab27cea 100644 --- a/arch/riscv/kvm/vcpu.c +++ b/arch/riscv/kvm/vcpu.c @@ -130,6 +130,13 @@ static void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu) struct kvm_vcpu_csr *reset_csr = &vcpu->arch.guest_reset_csr; struct kvm_cpu_context *cntx = &vcpu->arch.guest_context; struct kvm_cpu_context *reset_cntx = &vcpu->arch.guest_reset_context; + bool loaded; + + /* Disable preemption to avoid race with preempt notifiers */ + preempt_disable(); + loaded = (vcpu->cpu != -1); + if (loaded) + kvm_arch_vcpu_put(vcpu); memcpy(csr, reset_csr, sizeof(*csr)); @@ -141,6 +148,11 @@ static void kvm_riscv_reset_vcpu(struct kvm_vcpu *vcpu) WRITE_ONCE(vcpu->arch.irqs_pending, 0); WRITE_ONCE(vcpu->arch.irqs_pending_mask, 0); + + /* Reset the guest CSRs for hotplug usecase */ + if (loaded) + kvm_arch_vcpu_load(vcpu, smp_processor_id()); + preempt_enable(); } int kvm_arch_vcpu_precreate(struct kvm *kvm, unsigned int id) @@ -182,6 +194,13 @@ int kvm_arch_vcpu_setup(struct kvm_vcpu *vcpu) void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu) { + /** + * vcpu with id 0 is the designated boot cpu. + * Keep all vcpus with non-zero cpu id in power-off state so that they + * can brought to online using SBI HSM extension. + */ + if (vcpu->vcpu_idx != 0) + kvm_riscv_vcpu_power_off(vcpu); } void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu) diff --git a/arch/riscv/kvm/vcpu_sbi.c b/arch/riscv/kvm/vcpu_sbi.c index e21ce1e69e03..20ef59ed83a6 100644 --- a/arch/riscv/kvm/vcpu_sbi.c +++ b/arch/riscv/kvm/vcpu_sbi.c @@ -25,6 +25,8 @@ static int kvm_linux_err_map_sbi(int err) return SBI_ERR_INVALID_ADDRESS; case -EOPNOTSUPP: return SBI_ERR_NOT_SUPPORTED; + case -EALREADY: + return SBI_ERR_ALREADY_AVAILABLE; default: return SBI_ERR_FAILURE; }; @@ -35,6 +37,7 @@ extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_base; extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_time; extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_ipi; extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_rfence; +extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_hsm; static const struct kvm_vcpu_sbi_extension *sbi_ext[] = { &vcpu_sbi_ext_legacy, @@ -42,6 +45,7 @@ static const struct kvm_vcpu_sbi_extension *sbi_ext[] = { &vcpu_sbi_ext_time, &vcpu_sbi_ext_ipi, &vcpu_sbi_ext_rfence, + &vcpu_sbi_ext_hsm, }; void kvm_riscv_vcpu_sbi_forward(struct kvm_vcpu *vcpu, struct kvm_run *run) diff --git a/arch/riscv/kvm/vcpu_sbi_hsm.c b/arch/riscv/kvm/vcpu_sbi_hsm.c new file mode 100644 index 000000000000..ce6cfe125559 --- /dev/null +++ b/arch/riscv/kvm/vcpu_sbi_hsm.c @@ -0,0 +1,109 @@ +// SPDX-License-Identifier: GPL-2.0 +/** + * Copyright (c) 2020 Western Digital Corporation or its affiliates. + * + * Authors: + * Atish Patra + */ + +#include +#include +#include +#include +#include +#include + +static int kvm_sbi_hsm_vcpu_start(struct kvm_vcpu *vcpu) +{ + struct kvm_cpu_context *reset_cntx; + struct kvm_cpu_context *cp = &vcpu->arch.guest_context; + struct kvm_vcpu *target_vcpu; + unsigned long target_vcpuid = cp->a0; + + target_vcpu = kvm_get_vcpu_by_id(vcpu->kvm, target_vcpuid); + if (!target_vcpu) + return -EINVAL; + if (!target_vcpu->arch.power_off) + return -EALREADY; + + reset_cntx = &target_vcpu->arch.guest_reset_context; + /* start address */ + reset_cntx->sepc = cp->a1; + /* target vcpu id to start */ + reset_cntx->a0 = target_vcpuid; + /* private data passed from kernel */ + reset_cntx->a1 = cp->a2; + kvm_make_request(KVM_REQ_VCPU_RESET, target_vcpu); + + /* Make sure that the reset request is enqueued before power on */ + smp_wmb(); + kvm_riscv_vcpu_power_on(target_vcpu); + + return 0; +} + +static int kvm_sbi_hsm_vcpu_stop(struct kvm_vcpu *vcpu) +{ + if ((!vcpu) || (vcpu->arch.power_off)) + return -EINVAL; + + kvm_riscv_vcpu_power_off(vcpu); + + return 0; +} + +static int kvm_sbi_hsm_vcpu_get_status(struct kvm_vcpu *vcpu) +{ + struct kvm_cpu_context *cp = &vcpu->arch.guest_context; + unsigned long target_vcpuid = cp->a0; + struct kvm_vcpu *target_vcpu; + + target_vcpu = kvm_get_vcpu_by_id(vcpu->kvm, target_vcpuid); + if (!target_vcpu) + return -EINVAL; + if (!target_vcpu->arch.power_off) + return SBI_HSM_HART_STATUS_STARTED; + else + return SBI_HSM_HART_STATUS_STOPPED; +} + +static int kvm_sbi_ext_hsm_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, + unsigned long *out_val, + struct kvm_cpu_trap *utrap, + bool *exit) +{ + int ret = 0; + struct kvm_cpu_context *cp = &vcpu->arch.guest_context; + struct kvm *kvm = vcpu->kvm; + unsigned long funcid = cp->a6; + + if (!cp) + return -EINVAL; + switch (funcid) { + case SBI_EXT_HSM_HART_START: + mutex_lock(&kvm->lock); + ret = kvm_sbi_hsm_vcpu_start(vcpu); + mutex_unlock(&kvm->lock); + break; + case SBI_EXT_HSM_HART_STOP: + ret = kvm_sbi_hsm_vcpu_stop(vcpu); + break; + case SBI_EXT_HSM_HART_STATUS: + ret = kvm_sbi_hsm_vcpu_get_status(vcpu); + if (ret >= 0) { + *out_val = ret; + ret = 0; + } + break; + default: + ret = -EOPNOTSUPP; + } + + return ret; +} + +const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_hsm = { + .extid_start = SBI_EXT_HSM, + .extid_end = SBI_EXT_HSM, + .handler = kvm_sbi_ext_hsm_handler, +}; -- 2.25.1