Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 11 May 2022 19:42:13 +0000
From:   Sean Christopherson <seanjc@google.com>
To:     Wei Zhang <zhanwei@google.com>
Cc:     Suleiman Souhlal <suleiman@google.com>,
        Sangwhan Moon <sxm@google.com>, Ingo Molnar <mingo@redhat.com>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Vitaly Kuznetsov <vkuznets@redhat.com>,
        Wanpeng Li <wanpengli@tencent.com>,
        Jim Mattson <jmattson@google.com>,
        Joerg Roedel <joro@8bytes.org>, kvm@vger.kernel.org,
        linux-kernel@vger.kernel.org, Jing Zhang <jingzhangos@google.com>,
        David Matlack <dmatlack@google.com>
Subject: Re: [PATCH 0/2] KVM: x86: Fix incorrect VM-exit profiling
Message-ID: <YnwRld0aH8489+XQ@google.com>
References: <20220412195846.3692374-1-zhanwei@google.com>
 <YnmqgFkhqWklrQIw@google.com>
 <CAN86XOYNpzEUN0aL9g=_GQFz5zdXX9Pvcs_TDmBVyJZDTfXREg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAN86XOYNpzEUN0aL9g=_GQFz5zdXX9Pvcs_TDmBVyJZDTfXREg@mail.gmail.com>
Precedence: bulk

+Jing and David

On Wed, May 11, 2022, Wei Zhang wrote:

Please don't top-post.  From https://people.kernel.org/tglx/notes-about-netiquette:

  A: Because it messes up the order in which people normally read text.
  Q: Why is top-posting such a bad thing?

  A: Top-posting.
  Q: What is the most annoying thing in e-mail?

  A: No.
  Q: Should I include quotations after my reply?


> On Tue, May 10, 2022 at 1:57 AM Sean Christopherson <seanjc@google.com> wrote:
> >
> > On Tue, Apr 12, 2022, Wei Zhang wrote:
> > > The profile=kvm boot option has been useful because it provides a
> > > convenient approach to profile VM exits.
> >
> > What exactly are you profiling?  Where the guest executing at any given exit?  Mostly
> > out of curiosity, but also in the hope that we might be able to replace profiling with
> > a dedicated KVM stat(s).
>
> Yes, the profiling is about finding out which instructions in the
> guest trigger VM exits and the corresponding frequencies.

Do you actually what to profile which instructions _trigger_ exits?  Because that's
not what this does.  This profiles every exit, regardless of whether or not the exit
was due to a guest action.  E.g. host IRQs/NMIs, page faults, etc... will all get
included and pollute the profile.  Over time, the signal-to-noise ratio will likely
improve, but there's definitely still going to be noise.

We actually tried to upstream histograms for exit reasons[*] (link is for arm64,
but we want it for x86 too, just can't find a link), but it was deemed too expensive
in terms of memory cost for general use.

An idea that's on our (GCP folks) todo list is to explore adding an eBPF hook into
the exit path that would allow userspace to inspect e.g. struct kvm_run on VM-Exit.
That would allow userspace to collect all kinds of info about VM-Exits without
committing to ABI beyond kvm_run, and without bloating the size of a vCPU for
environments that don't want detailed histograms/profiling.

My preference would be to find a more complete, KVM-specific solution.  The
profiling stuff seems like it's a dead end, i.e. will always be flawed in some
way.  If this cleanup didn't require a new hypercall then I wouldn't care, but
I don't love having to extend KVM's guest/host ABI for something that ideally
will become obsolete sooner than later.

[*] https://lore.kernel.org/all/20210922010851.2312845-3-jingzhangos@google.com