Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 24 May 2023 14:29:17 -0700
In-Reply-To: <CALMp9eR_xYapRm=zJ3OdAzBVFjpzeQWYv9nTs1ZstAsugEwWRQ@mail.gmail.com>
Mime-Version: 1.0
References: <20230310105346.12302-1-likexu@tencent.com> <20230310105346.12302-6-likexu@tencent.com>
 <ZC99f+AO1tZguu1I@google.com> <509b697f-4e60-94e5-f785-95f7f0a14006@gmail.com>
 <ZDAvDhV/bpPyt3oX@google.com> <34b5dd08-edac-e32f-1884-c8f2b85f7971@gmail.com>
 <59ef9af0-9528-e220-625a-ff16e6971f23@amd.com> <ZG52cgmjgaqY8jvq@google.com> <CALMp9eR_xYapRm=zJ3OdAzBVFjpzeQWYv9nTs1ZstAsugEwWRQ@mail.gmail.com>
Message-ID: <ZG6BrSXDnOdDvUZh@google.com>
Subject: Re: [PATCH 5/5] KVM: x86/pmu: Hide guest counter updates from the
 VMRUN instruction
From:   Sean Christopherson <seanjc@google.com>
To:     Jim Mattson <jmattson@google.com>
Cc:     Sandipan Das <sandipan.das@amd.com>,
        Like Xu <like.xu.linux@gmail.com>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Ravi Bangoria <ravi.bangoria@amd.com>, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org,
        Santosh Shukla <santosh.shukla@amd.com>,
        "Tom Lendacky (AMD)" <thomas.lendacky@amd.com>,
        Ananth Narayan <ananth.narayan@amd.com>
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Wed, May 24, 2023, Jim Mattson wrote:
> On Wed, May 24, 2023 at 1:41=E2=80=AFPM Sean Christopherson <seanjc@googl=
e.com> wrote:
> >
> > On Wed, Apr 26, 2023, Sandipan Das wrote:
> > > Hi Sean, Like,
> > >
> > > On 4/19/2023 7:11 PM, Like Xu wrote:
> > > >> Heh, it's very much explicable, it's just not desirable, and you a=
nd I would argue
> > > >> that it's also incorrect.
> > > >
> > > > This is completely inaccurate from the end guest pmu user's perspec=
tive.
> > > >
> > > > I have a toy that looks like virtio-pmu, through which guest users =
can get hypervisor performance data.
> > > > But the side effect of letting the guest see the VMRUN instruction =
by default is unacceptable, isn't it ?
> > > >
> > > >>
> > > >> AMD folks, are there plans to document this as an erratum?=C3=AF=
=C2=BF=C2=BD I agree with Like that
> > > >> counting VMRUN as a taken branch in guest context is a CPU bug, ev=
en if the behavior
> > > >> is known/expected.
> > > >
> > >
> > > This behaviour is architectural and an erratum will not be issued. Ho=
wever, for clarity, a future
> > > release of the APM will include additional details like the following=
:
> > >
> > >   1) From the perspective of performance monitoring counters, VMRUNs =
are considered as far control
> > >      transfers and VMEXITs as exceptions.
> > >
> > >   2) When the performance monitoring counters are set up to count eve=
nts only in certain modes
> > >      through the "OsUserMode" and "HostGuestOnly" bits, instructions =
and events that change the
> > >      mode are counted in the target mode. For example, a SYSCALL from=
 CPL 3 to CPL 0 with a
> > >      counter set to count retired instructions with USR=3D1 and OS=3D=
0 will not cause an increment of
> > >      the counter. However, the SYSRET back from CPL 0 to CPL 3 will c=
ause an increment of the
> > >      counter and the total count will end up correct. Similarly, when=
 counting PMCx0C6 (retired
> > >      far control transfers, including exceptions and interrupts) with=
 Guest=3D1 and Host=3D0, a VMRUN
> > >      instruction will cause an increment of the counter. However, the=
 subsequent VMEXIT that occurs,
> > >      since the target is in the host, will not cause an increment of =
the counter and so the total
> > >      count will end up correct.
> >
> > The count from the guest's perspective does not "end up correct".  Unli=
ke SYSCALL,
> > where _userspace_ deliberately and synchronously executes a branch inst=
ruction,
> > VMEXIT and VMRUN are supposed to be transparent to the guest and can be=
 completely
> > asynchronous with respect to guest code execution, e.g. if the host is =
spamming
> > IRQs, the guest will see a potentially large number of bogus (from it's=
 perspective)
> > branches retired.
>=20
> The reverse problem occurs when a PMC is configured to count "CPUID
> instructions retired." Since KVM intercepts CPUID and emulates it, the
> PMC will always read 0, even if the guest executes a tight loop of
> CPUID instructions.
>
> The PMU is not virtualizable on AMD CPUs without significant
> hypervisor corrections. I have to wonder if it's really worth the
> effort.

Per our offlist chat, my understanding is that there are caveats with vPMUs=
 that
it's simply not feasible for a hypervisor to handle.  I.e. virtualizing any=
 x86
PMU with 100% accuracy isn't happening anytime soon.

The way forward is likely to evaluate each caveat on a case-by-case basis t=
o
determine whether or not the cost of the fixup in KVM is worth the benefit =
to
the guest.  E.g. emulating "CPUID instructions retired" seems like it would=
 be
fairly straightforward.  AFAICT, fixing up the VMRUN stuff is quite difficu=
lt though.