Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20230310105346.12302-1-likexu@tencent.com> <20230310105346.12302-6-likexu@tencent.com>
 <ZC99f+AO1tZguu1I@google.com> <509b697f-4e60-94e5-f785-95f7f0a14006@gmail.com>
 <ZDAvDhV/bpPyt3oX@google.com> <34b5dd08-edac-e32f-1884-c8f2b85f7971@gmail.com>
 <59ef9af0-9528-e220-625a-ff16e6971f23@amd.com> <ZG52cgmjgaqY8jvq@google.com>
 <CALMp9eR_xYapRm=zJ3OdAzBVFjpzeQWYv9nTs1ZstAsugEwWRQ@mail.gmail.com>
 <ZG6BrSXDnOdDvUZh@google.com> <CALMp9eQrDX6=gJzybegjzDJ665NCuWmESt-sZrKHcncnuENdpA@mail.gmail.com>
 <ec42501c-2e66-5248-5b97-4827344418f3@gmail.com>
In-Reply-To: <ec42501c-2e66-5248-5b97-4827344418f3@gmail.com>
From:   Jim Mattson <jmattson@google.com>
Date:   Tue, 30 May 2023 13:00:57 -0700
Message-ID: <CALMp9eSQWTTGQoJQ+f=ondF2wiiCaMiO-PMV0eaYJNXXrt4gQA@mail.gmail.com>
Subject: Re: [PATCH 5/5] KVM: x86/pmu: Hide guest counter updates from the
 VMRUN instruction
To:     Like Xu <like.xu.linux@gmail.com>
Cc:     Sean Christopherson <seanjc@google.com>,
        Sandipan Das <sandipan.das@amd.com>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Ravi Bangoria <ravi.bangoria@amd.com>, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org,
        Santosh Shukla <santosh.shukla@amd.com>,
        "Tom Lendacky (AMD)" <thomas.lendacky@amd.com>,
        Ananth Narayan <ananth.narayan@amd.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Mon, May 29, 2023 at 7:51=E2=80=AFAM Like Xu <like.xu.linux@gmail.com> w=
rote:
>
> On 25/5/2023 5:32 am, Jim Mattson wrote:
> > On Wed, May 24, 2023 at 2:29=E2=80=AFPM Sean Christopherson <seanjc@goo=
gle.com> wrote:
> >>
> >> On Wed, May 24, 2023, Jim Mattson wrote:
> >>> On Wed, May 24, 2023 at 1:41=E2=80=AFPM Sean Christopherson <seanjc@g=
oogle.com> wrote:
> >>>>
> >>>> On Wed, Apr 26, 2023, Sandipan Das wrote:
> >>>>> Hi Sean, Like,
> >>>>>
> >>>>> On 4/19/2023 7:11 PM, Like Xu wrote:
> >>>>>>> Heh, it's very much explicable, it's just not desirable, and you =
and I would argue
> >>>>>>> that it's also incorrect.
> >>>>>>
> >>>>>> This is completely inaccurate from the end guest pmu user's perspe=
ctive.
> >>>>>>
> >>>>>> I have a toy that looks like virtio-pmu, through which guest users=
 can get hypervisor performance data.
> >>>>>> But the side effect of letting the guest see the VMRUN instruction=
 by default is unacceptable, isn't it ?
> >>>>>>
> >>>>>>>
> >>>>>>> AMD folks, are there plans to document this as an erratum?=C3=AF=
=C2=BF=C2=BD I agree with Like that
> >>>>>>> counting VMRUN as a taken branch in guest context is a CPU bug, e=
ven if the behavior
> >>>>>>> is known/expected.
> >>>>>>
> >>>>>
> >>>>> This behaviour is architectural and an erratum will not be issued. =
However, for clarity, a future
> >>>>> release of the APM will include additional details like the followi=
ng:
> >>>>>
> >>>>>    1) From the perspective of performance monitoring counters, VMRU=
Ns are considered as far control
> >>>>>       transfers and VMEXITs as exceptions.
> >>>>>
> >>>>>    2) When the performance monitoring counters are set up to count =
events only in certain modes
> >>>>>       through the "OsUserMode" and "HostGuestOnly" bits, instructio=
ns and events that change the
> >>>>>       mode are counted in the target mode. For example, a SYSCALL f=
rom CPL 3 to CPL 0 with a
> >>>>>       counter set to count retired instructions with USR=3D1 and OS=
=3D0 will not cause an increment of
> >>>>>       the counter. However, the SYSRET back from CPL 0 to CPL 3 wil=
l cause an increment of the
> >>>>>       counter and the total count will end up correct. Similarly, w=
hen counting PMCx0C6 (retired
> >>>>>       far control transfers, including exceptions and interrupts) w=
ith Guest=3D1 and Host=3D0, a VMRUN
> >>>>>       instruction will cause an increment of the counter. However, =
the subsequent VMEXIT that occurs,
> >>>>>       since the target is in the host, will not cause an increment =
of the counter and so the total
> >>>>>       count will end up correct.
> >>>>
> >>>> The count from the guest's perspective does not "end up correct".  U=
nlike SYSCALL,
> >>>> where _userspace_ deliberately and synchronously executes a branch i=
nstruction,
> >>>> VMEXIT and VMRUN are supposed to be transparent to the guest and can=
 be completely
> >>>> asynchronous with respect to guest code execution, e.g. if the host =
is spamming
> >>>> IRQs, the guest will see a potentially large number of bogus (from i=
t's perspective)
> >>>> branches retired.
> >>>
> >>> The reverse problem occurs when a PMC is configured to count "CPUID
> >>> instructions retired." Since KVM intercepts CPUID and emulates it, th=
e
> >>> PMC will always read 0, even if the guest executes a tight loop of
> >>> CPUID instructions.
>
> Unlikely. KVM will count any emulated instructions based on kvm_pmu_incr_=
counter().
> Did I miss some conditions ?

That code only increments PMCs configured to count "instructions
retired" and "branch instructions retired." It does not increment PMCs
configured to count "CPUID instructions retired."

> >>>
> >>> The PMU is not virtualizable on AMD CPUs without significant
> >>> hypervisor corrections. I have to wonder if it's really worth the
> >>> effort.
>
> I used to think so, until I saw the AMD64_EVENTSEL_GUESTONLY bit.
> Hardware architects are expected to put more effort into this area.
>
> >>
> >> Per our offlist chat, my understanding is that there are caveats with =
vPMUs that
> >> it's simply not feasible for a hypervisor to handle.  I.e. virtualizin=
g any x86
> >> PMU with 100% accuracy isn't happening anytime soon.
>
> Indeed, and any more detailed complaints ?

Reference cycles unhalted fails to increment outside of guest mode.

SMIs received counts *physical* rather than virtual SMIs

Interrupts taken counts *physical* rather than virtual interrupts taken.

> >>
> >> The way forward is likely to evaluate each caveat on a case-by-case ba=
sis to
> >> determine whether or not the cost of the fixup in KVM is worth the ben=
efit to
> >> the guest.  E.g. emulating "CPUID instructions retired" seems like it =
would be
> >> fairly straightforward.  AFAICT, fixing up the VMRUN stuff is quite di=
fficult though.
> >
> > Yeah. The problem with fixing up "CPUID instructions retired" is
> > tracking what the event encoding is for every F/M/S out there. It's
> > not worth it.
>
> I don't think it's feasible to emulate 100% accuracy on Intel. For guest =
pmu
> users, it is motivated by wanting to know how effective they are running =
on
> the current pCPU, and any vPMU eimulation behavior that helps this
> understanding would be valuable.

But at least Intel has a list of architected events, which are mostly
amenable to virtualization.