Date: Fri, 8 Dec 2017 20:39:10 +0300
From: Roman Kagan <rkagan@virtuozzo.com>
To: Vitaly Kuznetsov <vkuznets@redhat.com>
Cc: kvm@vger.kernel.org, x86@kernel.org,
        Paolo Bonzini <pbonzini@redhat.com>,
        Radim =?utf-8?B?S3LEjW3DocWZ?= <rkrcmar@redhat.com>,
        Thomas Gleixner <tglx@linutronix.de>, Ingo Molnar <mingo@redhat.com>,
        "H. Peter Anvin" <hpa@zytor.com>,
        "K. Y. Srinivasan" <kys@microsoft.com>,
        Haiyang Zhang <haiyangz@microsoft.com>,
        Stephen Hemminger <sthemmin@microsoft.com>,
        "Michael Kelley (EOSG)" <Michael.H.Kelley@microsoft.com>,
        Andy Lutomirski <luto@kernel.org>, Mohammed Gamal <mmorsy@redhat.com>,
        Cathy Avery <cavery@redhat.com>, linux-kernel@vger.kernel.org,
        devel@linuxdriverproject.org
Subject: Re: [PATCH 6/6] x86/kvm: support Hyper-V reenlightenment
Message-ID: <20171208173909.GA4777@rkaganb.sw.ru>
Mail-Followup-To: Roman Kagan <rkagan@virtuozzo.com>,
        Vitaly Kuznetsov <vkuznets@redhat.com>, kvm@vger.kernel.org,
        x86@kernel.org, Paolo Bonzini <pbonzini@redhat.com>,
        Radim =?utf-8?B?S3LEjW3DocWZ?= <rkrcmar@redhat.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, "H. Peter Anvin" <hpa@zytor.com>,
        "K. Y. Srinivasan" <kys@microsoft.com>,
        Haiyang Zhang <haiyangz@microsoft.com>,
        Stephen Hemminger <sthemmin@microsoft.com>,
        "Michael Kelley (EOSG)" <Michael.H.Kelley@microsoft.com>,
        Andy Lutomirski <luto@kernel.org>,
        Mohammed Gamal <mmorsy@redhat.com>, Cathy Avery <cavery@redhat.com>,
        linux-kernel@vger.kernel.org, devel@linuxdriverproject.org
References: <20171208105000.25116-1-vkuznets@redhat.com>
 <20171208105000.25116-7-vkuznets@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20171208105000.25116-7-vkuznets@redhat.com>
User-Agent: Mutt/1.9.1 (2017-09-22)
SpamDiagnosticOutput: 1:22
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Dec 2017 17:39:14.5948
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 37cf5263-090a-4b8d-842d-08d53e6299d5
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 0bc7f26d-0264-416e-a6fc-8352af79c58f
X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR08MB0844
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 2385
Lines: 75

On Fri, Dec 08, 2017 at 11:50:00AM +0100, Vitaly Kuznetsov wrote:
> When we run nested KVM on Hyper-V guests we need to update masterclocks for
> all guests when L1 migrates to a host with different TSC frequency.
> Implement the procedure in the following way:
> - Pause all guests.
> - Tell our host (Hyper-V) to stop emulating TSC accesses.
> - Update our gtod copy, recompute clocks.
> - Unpause all guests.
> 
> This is somewhat similar to cpufreq but we have two important differences:
> we can only disable TSC emulation globally (on all CPUs) and we don't know
> the new TSC frequency until we turn the emulation off so we can't
> 'prepare' ourselves to the event.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
>  arch/x86/kvm/x86.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 45 insertions(+)
> 
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 96e04a0cb921..04d90712ffd2 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -68,6 +68,7 @@
>  #include <asm/div64.h>
>  #include <asm/irq_remapping.h>
>  #include <asm/mshyperv.h>
> +#include <asm/hypervisor.h>
>  
>  #define CREATE_TRACE_POINTS
>  #include "trace.h"
> @@ -5946,6 +5947,43 @@ static void tsc_khz_changed(void *data)
>  	__this_cpu_write(cpu_tsc_khz, khz);
>  }
>  
> +void kvm_hyperv_tsc_notifier(void)
> +{
> +#ifdef CONFIG_X86_64
> +	struct kvm *kvm;
> +	struct kvm_vcpu *vcpu;
> +	int cpu;
> +
> +	spin_lock(&kvm_lock);
> +	list_for_each_entry(kvm, &vm_list, vm_list)
> +		kvm_make_mclock_inprogress_request(kvm);
> +
> +	hyperv_stop_tsc_emulation();
> +
> +	/* TSC frequency always matches when on Hyper-V */
> +	for_each_present_cpu(cpu)
> +		per_cpu(cpu_tsc_khz, cpu) = tsc_khz;
> +	kvm_max_guest_tsc_khz = tsc_khz;
> +
> +	list_for_each_entry(kvm, &vm_list, vm_list) {
> +		struct kvm_arch *ka = &kvm->arch;
> +
> +		spin_lock(&ka->pvclock_gtod_sync_lock);
> +
> +		pvclock_update_vm_gtod_copy(kvm);
> +
> +		kvm_for_each_vcpu(cpu, vcpu, kvm)
> +			kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu);
> +
> +		kvm_for_each_vcpu(cpu, vcpu, kvm)
> +			kvm_clear_request(KVM_REQ_MCLOCK_INPROGRESS, vcpu);
> +
> +		spin_unlock(&ka->pvclock_gtod_sync_lock);
> +	}
> +	spin_unlock(&kvm_lock);

Can't you skip all this if the tsc frequency hasn't changed (which
should probably be the case when the CPU supports tsc frequency
scaling)?

Roman.