Received: by 10.223.185.116 with SMTP id b49csp6138670wrg; Thu, 8 Mar 2018 02:25:30 -0800 (PST) X-Google-Smtp-Source: AG47ELveBLSUuL79daT9U+EnUYirv4xJDznIPobWDnkdZsEIVtnQVLPljPCgP3/8zHkcRsUfrTmo X-Received: by 10.99.125.29 with SMTP id y29mr20674416pgc.277.1520504730228; Thu, 08 Mar 2018 02:25:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1520504730; cv=none; d=google.com; s=arc-20160816; b=cI+c57H+44loBAV0bxlaQdJ8JuIQ7XQgkrAGYGEI91wxTuH2+eRQ4rccR8KhUKK9ab m3QlN9qL4LfxdiiTcqrtusRZEBB0XTBWAFxeOdpujDHEjIRnPhicXOkkiGb1/zHw9SZR 3s9dv5Ws1MzHcEEepnZQl/r4m3WyqLZyyBP6BC8bHno8wdJGBI9NaEuh1Q7ybFk6En3K +5pPSJqlzDo9g1rOz9RGMvbRv3PiH/I0MPtE0JFV/N1RG2UeUX3AB0VDWMOomh1vYOYR 0ZQ7usS80nfpJyBp+buk1uw9Y00qiS/zCqOcHDrQiNBkAuFc3LjAjhAflYwZOq5QrouN 235Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:message-id:in-reply-to:date:references:subject:cc:to :from:arc-authentication-results; bh=pJdOfV5gK4e8KfxmazOAElhNfeRj4fXYrHYbVs7wR9M=; b=ef1+vDDc6VPrIpGOv1ek7a+ldH4qnHY+giQIR4i3EqescjVk05I/3ildA7sdEjXTpZ 8ZUIs/sVS2iOOa8ZM5Xl9gpG+codA4mgCaW1okyfSCTYPd9BVCpXspLxMUpl66YekCjN YOFA0whFnehMPqZzIeIbcMiDuE1dDkI4KPLhGGWojI5pcBEgF8mHjIKXBr+tRkeMo0/s IrOML75qTQk0UaMrq0At3mM0f2LnU2oWY3tbDMOEm1Llci95KMohsnx5rKuGQJUuc8yP mW8lxuLw+R60hVVKXQmENEimxL16PLLeW89vhbEnj4jKFsxXPY/iO9JiFFrfrmQQBdeS M0EQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q1-v6si14500940plb.764.2018.03.08.02.25.16; Thu, 08 Mar 2018 02:25:30 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755636AbeCHKYI (ORCPT + 99 others); Thu, 8 Mar 2018 05:24:08 -0500 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:37136 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751616AbeCHKYG (ORCPT ); Thu, 8 Mar 2018 05:24:06 -0500 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0A7BE8182D16; Thu, 8 Mar 2018 10:24:06 +0000 (UTC) Received: from vitty.brq.redhat.com.redhat.com (unknown [10.43.2.155]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9F3E110AF9C2; Thu, 8 Mar 2018 10:24:00 +0000 (UTC) From: Vitaly Kuznetsov To: Radim =?utf-8?B?S3LEjW3DocWZ?= Cc: kvm@vger.kernel.org, x86@kernel.org, Paolo Bonzini , "K. Y. Srinivasan" , Haiyang Zhang , Stephen Hemminger , "Michael Kelley \(EOSG\)" , Mohammed Gamal , Cathy Avery , Bandan Das , linux-kernel@vger.kernel.org Subject: Re: [PATCH v2 5/5] x86/kvm: use Enlightened VMCS when running on Hyper-V References: <20180226171121.18974-1-vkuznets@redhat.com> <20180226171121.18974-6-vkuznets@redhat.com> <20180307175611.GF12290@flask> Date: Thu, 08 Mar 2018 11:23:59 +0100 In-Reply-To: <20180307175611.GF12290@flask> ("Radim \=\?utf-8\?B\?S3LEjW3DocWZ\?\= \=\?utf-8\?B\?Iidz\?\= message of "Wed, 7 Mar 2018 18:56:11 +0100") Message-ID: <87h8pqyif4.fsf@vitty.brq.redhat.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Thu, 08 Mar 2018 10:24:06 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.8]); Thu, 08 Mar 2018 10:24:06 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'vkuznets@redhat.com' RCPT:'' Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Radim Krčmář writes: > 2018-02-26 18:11+0100, Vitaly Kuznetsov: >> Enlightened VMCS is just a structure in memory, the main benefit >> besides avoiding somewhat slower VMREAD/VMWRITE is using clean field >> mask: we tell the underlying hypervisor which fields were modified >> since VMEXIT so there's no need to inspect them all. >> >> Tight CPUID loop test shows significant speedup: >> Before: 20766 cycles >> After: 8912 cycles >> >> Static key is being used to avoid performance penalty for non-Hyper-V >> deployments. Tests show we add around 3 (three) CPU cycles on each >> VMEXIT (1077.5 cycles before, 1080.7 cycles after for the same CPUID >> loop on bare metal). We can probably avoid one test/jmp in vmx_vcpu_run() >> but I don't see a clean way to use static key in assembly. > > Patching the correct instruction should be simpler than replicating > static_branch (because we have to support assemblers without ASM_GOTO), > but I'd care about that later, if ever. > >> Signed-off-by: Vitaly Kuznetsov >> --- >> diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c >> @@ -999,6 +1000,442 @@ static const u32 vmx_msr_index[] = { >> +/* >> + * Enlightened VMCSv1 doesn't support these: > > I take it that the code assumes that the hypervisor will never enable > these features if we have enlightened VMCS. I think it would be best to > disable those features when turning on evmcs. Sure. > >> + * POSTED_INTR_NV = 0x00000002, >> + * GUEST_INTR_STATUS = 0x00000810, >> + * APIC_ACCESS_ADDR = 0x00002014, >> + * POSTED_INTR_DESC_ADDR = 0x00002016, >> + * EOI_EXIT_BITMAP0 = 0x0000201c, >> + * EOI_EXIT_BITMAP1 = 0x0000201e, >> + * EOI_EXIT_BITMAP2 = 0x00002020, >> + * EOI_EXIT_BITMAP3 = 0x00002022, > > enable_apicv, flexpriority_enabled > >> + * GUEST_PML_INDEX = 0x00000812, >> + * PML_ADDRESS = 0x0000200e, > > enable_pml > >> + * VM_FUNCTION_CONTROL = 0x00002018, >> + * EPTP_LIST_ADDRESS = 0x00002024, > > (only vm controls) > >> + * VMREAD_BITMAP = 0x00002026, >> + * VMWRITE_BITMAP = 0x00002028, > > enable_shadow_vmcs > >> + * TSC_MULTIPLIER = 0x00002032, > > (only vm controls) > >> + * GUEST_IA32_PERF_GLOBAL_CTRL = 0x00002808, >> + * GUEST_IA32_RTIT_CTL = 0x00002814, >> + * HOST_IA32_PERF_GLOBAL_CTRL = 0x00002c04, > > (only vm controls) > >> + * PLE_GAP = 0x00004020, >> + * PLE_WINDOW = 0x00004022, > > ple_gap > >> + * VMX_PREEMPTION_TIMER_VALUE = 0x0000482E, > > enable_preemption_timer > >> + */ >> +}; >> + >> +static inline u16 get_evmcs_offset(unsigned long field) >> +{ >> + unsigned int index = ROL16(field, 6); >> + >> + if (index >= ARRAY_SIZE(vmcs_field_to_evmcs_1)) >> + return 0; > > Please add a warning when trying to use an EVMCS that doesn't exist, > just like the VMREAD/WRITE does -- it's an internal KVM error. > Will do. >> + >> + return (u16)vmcs_field_to_evmcs_1[index]; >> +} >> + >> @@ -3828,7 +4302,12 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf) >> vmcs_conf->size = vmx_msr_high & 0x1fff; >> vmcs_conf->order = get_order(vmcs_conf->size); >> vmcs_conf->basic_cap = vmx_msr_high & ~0x1fff; >> - vmcs_conf->revision_id = vmx_msr_low; >> + >> + /* KVM supports Enlightened VMCS v1 only */ >> + if (static_branch_unlikely(&enable_evmcs)) >> + vmcs_conf->revision_id = 1; > > I think we have to put the bottom bits from ms_hyperv.nested_features > here. 16.5.1 Enlightened VMCS Versioning: > > Each enlightened VMCS structure contains a version field, which is > reported by the L0 hypervisor (see 2.4.11) see below > >> + else >> + vmcs_conf->revision_id = vmx_msr_low; >> >> vmcs_conf->pin_based_exec_ctrl = _pin_based_exec_control; >> vmcs_conf->cpu_based_exec_ctrl = _cpu_based_exec_control; >> @@ -12414,7 +12908,35 @@ static struct kvm_x86_ops vmx_x86_ops __ro_after_init = { >> >> static int __init vmx_init(void) >> { >> - int r = kvm_init(&vmx_x86_ops, sizeof(struct vcpu_vmx), >> + int r; >> + >> +#if IS_ENABLED(CONFIG_HYPERV) >> + /* >> + * Enlightened VMCS usage should be recommended and the host needs >> + * to support eVMCS v1 or above. We can also disable eVMCS support >> + * with module parameter. >> + */ >> + if (enlightened_vmcs && >> + ms_hyperv.hints & HV_X64_ENLIGHTENED_VMCS_RECOMMENDED && >> + (ms_hyperv.nested_features & 0xff) >= 1) { > > Please add macros like HV_X64_ENLIGHTENED_VMCS_VERSION and > KVM_EVMCS_VERSION for those numbers. This can be arranged > I think we should not proceed with version other than 1 for now -- there > is no mention of backward compatibility in the spec. I think the check is correct. eVMCS version represents the format of the structure in memory. New versions may have nothing in common but at the same time Ver1 support can't be dropped from future Hyper-V versions without regressing L1s which don't support new version. So currently we check that L0 supports at least ver1 (or some newer version which implies all previously released versions are supported too) and we declare that KVM supports ver1 exactly (this is the format of the structure we put to memory and share with L0). We can't declare support for any other version. Currently, it is only Ver1 on WS2016. -- Vitaly