Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Tue, 8 Jun 2021 09:47:51 +0900
From:   Sergey Senozhatsky <senozhatsky@chromium.org>
To:     Maxim Levitsky <mlevitsk@redhat.com>
Cc:     Sergey Senozhatsky <senozhatsky@chromium.org>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Marc Zyngier <maz@kernel.org>,
        Vitaly Kuznetsov <vkuznets@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Suleiman Souhlal <suleiman@google.com>, x86@kernel.org,
        kvm@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCHv3 2/2] kvm: x86: implement KVM PM-notifier
Message-ID: <YL6+NysuqhORZEs1@google.com>
References: <20210606021045.14159-1-senozhatsky@chromium.org>
 <20210606021045.14159-2-senozhatsky@chromium.org>
 <fe13fe734a01bb54f47fea06624c617beb062fdd.camel@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <fe13fe734a01bb54f47fea06624c617beb062fdd.camel@redhat.com>
Precedence: bulk

On (21/06/07 15:54), Maxim Levitsky wrote:
[..]
> Also I would like to add my .02 cents on my observations on what happens when I suspend my system
> with guests running, which I do once in a while.
> I haven't dug deep into it yet as host suspend with VM running wasn't high on my priority list.
>  
> First of all after a host suspend/resume cycle (and that is true on all 3 machines I own),
> the host TSC is reset to 0 on all CPUs, thus while it is still synchronized, it jumps backward.
>  
> Host kernel has no issues coping with this.
>  
> Guests however complain about clocksource watchdog and mark the tsc clocksource as unstable,
> at least when invtsc is used (regardless of this patch, I wasn't able to notice a difference
> with and without it yet).
>  
>  
> [  287.515864] clocksource: timekeeping watchdog on CPU0: Marking clocksource 'tsc' as unstable because the skew is too large:
> [  287.516926] clocksource:                       'kvm-clock' wd_now: 4437767926 wd_last: 429c3c42f5 mask: ffffffffffffffff
> [  287.527100] clocksource:                       'tsc' cs_now: c33f6ce157 cs_last: c1be2ad19f mask: ffffffffffffffff
> [  287.528493] tsc: Marking TSC unstable due to clocksource watchdog
> [  287.556640] clocksource: Switched to clocksource kvm-clock
>  
>  
> This is from Intel system with stable TSC, but I have seen this on my AMD systems as well,
> but these have other issues which might affect this (see below).
>  
> AFAIK, we have code in kvm_arch_hardware_enable for this exact case but it might not work
> correctly or be not enough to deal with this.
>  
> Also I notice that this code sets kvm->arch.backwards_tsc_observed = true which 
> in turn disables the master clock which is not good as well.
>  
> I haven't yet allocated time to investigate this.
>  
>  
> Another bit of information which I didn't start a discussion (but I think I should), 
> which is relevant to AMD systems, is in 'unsynchronized_tsc' function.
> 
> On AMD guest it will mark the TSC as unstable in the guest as long as invtsc is not used.
> I patched that code out for myself, that is why I am mentioning it.

If you are going to fork this discussion the could you please Cc Suleiman
and me? I believe we are having a bunch of problems with the guest clocks
here on our side.