Normally, genuine Hyper-V doesn't expose architectural invariant TSC
(CPUID.80000007H:EDX[8]) to its guests by default. A special PV MSR
(HV_X64_MSR_TSC_INVARIANT_CONTROL, 0x40000118) and corresponding CPUID
feature bit (CPUID.0x40000003.EAX[15]) were introduced. When bit 0 of the
PV MSR is set, invariant TSC bit starts to show up in CPUID. When the
feature is exposed to Hyper-V guests, reenlightenment becomes unneeded.
Note: strictly speaking, KVM doesn't have to have the feature as exposing
raw invariant TSC bit (CPUID.80000007H:EDX[8]) also seems to work for
modern Windows versions. The feature is, however, tiny and straitforward
and gives additional flexibility so why not.
Vitaly Kuznetsov (3):
KVM: x86: Hyper-V invariant TSC control
KVM: selftests: Fix wrmsr_safe()
KVM: selftests: Test Hyper-V invariant TSC control
arch/x86/include/asm/kvm_host.h | 1 +
arch/x86/kvm/cpuid.c | 7 ++
arch/x86/kvm/hyperv.c | 19 +++++
arch/x86/kvm/hyperv.h | 15 ++++
arch/x86/kvm/x86.c | 4 +-
.../selftests/kvm/include/x86_64/processor.h | 2 +-
.../selftests/kvm/x86_64/hyperv_features.c | 73 ++++++++++++++++++-
7 files changed, 115 insertions(+), 6 deletions(-)
--
2.35.3
It seems to be a misconception that "A" places an u64 operand to
EAX:EDX, at least with GCC11.
While writing a new test, I've noticed that wrmsr_safe() tries putting
garbage to the upper bits of the MSR, e.g.:
kvm_exit: reason MSR_WRITE rip 0x402919 info 0 0
kvm_msr: msr_write 40000118 = 0x60000000001 (#GP)
...
when it was supposed to write '1'. Apparently, "A" works the same as
"a" and not as EAX/EDX. Here's the relevant disassembled part:
With "A":
48 8b 43 08 mov 0x8(%rbx),%rax
49 b9 ba da ca ba 0a movabs $0xabacadaba,%r9
00 00 00
4c 8d 15 07 00 00 00 lea 0x7(%rip),%r10 # 402f44 <guest_msr+0x34>
4c 8d 1d 06 00 00 00 lea 0x6(%rip),%r11 # 402f4a <guest_msr+0x3a>
0f 30 wrmsr
With "a"/"d":
48 8b 43 08 mov 0x8(%rbx),%rax
48 89 c2 mov %rax,%rdx
48 c1 ea 20 shr $0x20,%rdx
49 b9 ba da ca ba 0a movabs $0xabacadaba,%r9
00 00 00
4c 8d 15 07 00 00 00 lea 0x7(%rip),%r10 # 402fc3 <guest_msr+0xb3>
4c 8d 1d 06 00 00 00 lea 0x6(%rip),%r11 # 402fc9 <guest_msr+0xb9>
0f 30 wrmsr
I was only able to find one online reference that "A" gives "eax and
edx combined into a 64-bit integer", other places don't mention it at
all.
Fixes: 3b23054cd3f5 ("KVM: selftests: Add x86-64 support for exception fixup")
Signed-off-by: Vitaly Kuznetsov <[email protected]>
---
tools/testing/selftests/kvm/include/x86_64/processor.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/testing/selftests/kvm/include/x86_64/processor.h b/tools/testing/selftests/kvm/include/x86_64/processor.h
index 79dcf6be1b47..3d412c578e78 100644
--- a/tools/testing/selftests/kvm/include/x86_64/processor.h
+++ b/tools/testing/selftests/kvm/include/x86_64/processor.h
@@ -612,7 +612,7 @@ static inline uint8_t rdmsr_safe(uint32_t msr, uint64_t *val)
static inline uint8_t wrmsr_safe(uint32_t msr, uint64_t val)
{
- return kvm_asm_safe("wrmsr", "A"(val), "c"(msr));
+ return kvm_asm_safe("wrmsr", "a"((u32)val), "d"(val >> 32), "c"(msr));
}
uint64_t vm_get_page_table_entry(struct kvm_vm *vm, struct kvm_vcpu *vcpu,
--
2.35.3
On Wed, 2022-07-13 at 17:05 +0200, Vitaly Kuznetsov wrote:
> Normally, genuine Hyper-V doesn't expose architectural invariant TSC
> (CPUID.80000007H:EDX[8]) to its guests by default. A special PV MSR
> (HV_X64_MSR_TSC_INVARIANT_CONTROL, 0x40000118) and corresponding CPUID
> feature bit (CPUID.0x40000003.EAX[15]) were introduced. When bit 0 of the
> PV MSR is set, invariant TSC bit starts to show up in CPUID. When the
> feature is exposed to Hyper-V guests, reenlightenment becomes unneeded.
If I understood the feature correctly from the code, it allows the HyperV, or in this
case KVM acting as HyperV, to avoid unconditionally exposing the invltsc bit
in CPUID, but rather let the guest know that it can opt-in into this,
by giving the guest another CPUID bit to indicate this ability
and a MSR which the guest uses to opt-in.
Are there known use cases of this, are there guests which won't opt-in?
>
> Note: strictly speaking, KVM doesn't have to have the feature as exposing
> raw invariant TSC bit (CPUID.80000007H:EDX[8]) also seems to work for
> modern Windows versions. The feature is, however, tiny and straitforward
> and gives additional flexibility so why not.
This means that KVM can also just unconditionally expose the invtsc bit
to the guest, and the guest still uses it.
Nitpick: It might be worth it to document it a bit better somewhere,
as I tried to do in this mail.
Best regards,
Maxim Levitsky
>
> Vitaly Kuznetsov (3):
> KVM: x86: Hyper-V invariant TSC control
> KVM: selftests: Fix wrmsr_safe()
> KVM: selftests: Test Hyper-V invariant TSC control
>
> arch/x86/include/asm/kvm_host.h | 1 +
> arch/x86/kvm/cpuid.c | 7 ++
> arch/x86/kvm/hyperv.c | 19 +++++
> arch/x86/kvm/hyperv.h | 15 ++++
> arch/x86/kvm/x86.c | 4 +-
> .../selftests/kvm/include/x86_64/processor.h | 2 +-
> .../selftests/kvm/x86_64/hyperv_features.c | 73 ++++++++++++++++++-
> 7 files changed, 115 insertions(+), 6 deletions(-)
>
Maxim Levitsky <[email protected]> writes:
> On Wed, 2022-07-13 at 17:05 +0200, Vitaly Kuznetsov wrote:
>> Normally, genuine Hyper-V doesn't expose architectural invariant TSC
>> (CPUID.80000007H:EDX[8]) to its guests by default. A special PV MSR
>> (HV_X64_MSR_TSC_INVARIANT_CONTROL, 0x40000118) and corresponding CPUID
>> feature bit (CPUID.0x40000003.EAX[15]) were introduced. When bit 0 of the
>> PV MSR is set, invariant TSC bit starts to show up in CPUID. When the
>> feature is exposed to Hyper-V guests, reenlightenment becomes unneeded.
>
> If I understood the feature correctly from the code, it allows the HyperV, or in this
> case KVM acting as HyperV, to avoid unconditionally exposing the invltsc bit
> in CPUID, but rather let the guest know that it can opt-in into this,
> by giving the guest another CPUID bit to indicate this ability
> and a MSR which the guest uses to opt-in.
>
> Are there known use cases of this, are there guests which won't opt-in?
>
Linux prior to dce7cd62754b and some older Windows guests I guess.
>>
>> Note: strictly speaking, KVM doesn't have to have the feature as exposing
>> raw invariant TSC bit (CPUID.80000007H:EDX[8]) also seems to work for
>> modern Windows versions. The feature is, however, tiny and straitforward
>> and gives additional flexibility so why not.
>
> This means that KVM can also just unconditionally expose the invtsc bit
> to the guest, and the guest still uses it.
Yes, this feature doesn't bring much by itself (at least with modern
Windows versions). I've implemented it while debugging what ended up
being
https://lore.kernel.org/kvm/[email protected]/
(so the issue wasn't enlightenments related after all) but as I think it
may come handy some day so why keeping it in my private stash.
>
>
> Nitpick: It might be worth it to document it a bit better somewhere,
> as I tried to do in this mail.
TLFS sounds like the right place for it but ... it's not there... oh well.
>
>
> Best regards,
> Maxim Levitsky
>
>>
>> Vitaly Kuznetsov (3):
>> KVM: x86: Hyper-V invariant TSC control
>> KVM: selftests: Fix wrmsr_safe()
>> KVM: selftests: Test Hyper-V invariant TSC control
>>
>> arch/x86/include/asm/kvm_host.h | 1 +
>> arch/x86/kvm/cpuid.c | 7 ++
>> arch/x86/kvm/hyperv.c | 19 +++++
>> arch/x86/kvm/hyperv.h | 15 ++++
>> arch/x86/kvm/x86.c | 4 +-
>> .../selftests/kvm/include/x86_64/processor.h | 2 +-
>> .../selftests/kvm/x86_64/hyperv_features.c | 73 ++++++++++++++++++-
>> 7 files changed, 115 insertions(+), 6 deletions(-)
>>
>
>
--
Vitaly
From: Vitaly Kuznetsov <[email protected]> Sent: Thursday, July 14, 2022 8:03 AM
>
> Maxim Levitsky <[email protected]> writes:
>
> > On Wed, 2022-07-13 at 17:05 +0200, Vitaly Kuznetsov wrote:
> >> Normally, genuine Hyper-V doesn't expose architectural invariant TSC
> >> (CPUID.80000007H:EDX[8]) to its guests by default. A special PV MSR
> >> (HV_X64_MSR_TSC_INVARIANT_CONTROL, 0x40000118) and corresponding CPUID
> >> feature bit (CPUID.0x40000003.EAX[15]) were introduced. When bit 0 of the
> >> PV MSR is set, invariant TSC bit starts to show up in CPUID. When the
> >> feature is exposed to Hyper-V guests, reenlightenment becomes unneeded.
> >
> > If I understood the feature correctly from the code, it allows the HyperV, or in this
> > case KVM acting as HyperV, to avoid unconditionally exposing the invltsc bit
> > in CPUID, but rather let the guest know that it can opt-in into this,
> > by giving the guest another CPUID bit to indicate this ability
> > and a MSR which the guest uses to opt-in.
> >
> > Are there known use cases of this, are there guests which won't opt-in?
> >
>
> Linux prior to dce7cd62754b and some older Windows guests I guess.
FWIW, the idea is to avoid having this new functionality magically show
up in existing guests when the Hyper-V host is upgraded. A guest OS
version with the code to opt-in has presumably been tested to make sure
it works correctly when the functionality is exposed.
>
> >>
> >> Note: strictly speaking, KVM doesn't have to have the feature as exposing
> >> raw invariant TSC bit (CPUID.80000007H:EDX[8]) also seems to work for
> >> modern Windows versions. The feature is, however, tiny and straitforward
> >> and gives additional flexibility so why not.
> >
> > This means that KVM can also just unconditionally expose the invtsc bit
> > to the guest, and the guest still uses it.
>
> Yes, this feature doesn't bring much by itself (at least with modern
> Windows versions). I've implemented it while debugging what ended up
> being https://lore.kernel.org/kvm/[email protected]/
> (so the issue wasn't enlightenments related after all) but as I think it
> may come handy some day so why keeping it in my private stash.
>
> >
> > Nitpick: It might be worth it to document it a bit better somewhere,
> > as I tried to do in this mail.
>
> TLFS sounds like the right place for it but ... it's not there... oh well.
>
I've sent a nag email to the Hyper-V folks about updating the TLFS.
Michael