Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7;
Date:   Wed, 4 Oct 2023 13:43:50 -0700
In-Reply-To: <CAL715WLbAnnGUiTdHPO0L7v2FHGa5qmTnWJDi8k9qVkGry5GGQ@mail.gmail.com>
Mime-Version: 1.0
References: <20230927113312.GD21810@noisy.programming.kicks-ass.net>
 <ZRRl6y1GL-7RM63x@google.com> <20230929115344.GE6282@noisy.programming.kicks-ass.net>
 <ZRbxb15Opa2_AusF@google.com> <20231002115718.GB13957@noisy.programming.kicks-ass.net>
 <ZRrF38RGllA04R8o@gmail.com> <ZRroQg6flyGBtZTG@google.com>
 <20231002204017.GB27267@noisy.programming.kicks-ass.net> <ZRtmvLJFGfjcusQW@google.com>
 <CAL715WLbAnnGUiTdHPO0L7v2FHGa5qmTnWJDi8k9qVkGry5GGQ@mail.gmail.com>
Message-ID: <ZR3Ohk50rSofAnSL@google.com>
Subject: Re: [Patch v4 07/13] perf/x86: Add constraint for guest perf metrics event
From:   Sean Christopherson <seanjc@google.com>
To:     Mingwei Zhang <mizhang@google.com>
Cc:     Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@kernel.org>,
        Dapeng Mi <dapeng1.mi@linux.intel.com>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Arnaldo Carvalho de Melo <acme@kernel.org>,
        Kan Liang <kan.liang@linux.intel.com>,
        Like Xu <likexu@tencent.com>,
        Mark Rutland <mark.rutland@arm.com>,
        Alexander Shishkin <alexander.shishkin@linux.intel.com>,
        Jiri Olsa <jolsa@kernel.org>,
        Namhyung Kim <namhyung@kernel.org>,
        Ian Rogers <irogers@google.com>,
        Adrian Hunter <adrian.hunter@intel.com>, kvm@vger.kernel.org,
        linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org,
        Zhenyu Wang <zhenyuw@linux.intel.com>,
        Zhang Xiong <xiong.y.zhang@intel.com>,
        Lv Zhiyuan <zhiyuan.lv@intel.com>,
        Yang Weijiang <weijiang.yang@intel.com>,
        Dapeng Mi <dapeng1.mi@intel.com>,
        Jim Mattson <jmattson@google.com>,
        David Dunn <daviddunn@google.com>,
        Thomas Gleixner <tglx@linutronix.de>
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Tue, Oct 03, 2023, Mingwei Zhang wrote:
> On Mon, Oct 2, 2023 at 5:56=E2=80=AFPM Sean Christopherson <seanjc@google=
.com> wrote:
> > The "when" is what's important.   If KVM took a literal interpretation =
of
> > "exclude guest" for pass-through MSRs, then KVM would context switch al=
l those
> > MSRs twice for every VM-Exit=3D>VM-Enter roundtrip, even when the VM-Ex=
it isn't a
> > reschedule IRQ to schedule in a different task (or vCPU).  The overhead=
 to save
> > all the host/guest MSRs and load all of the guest/host MSRs *twice* for=
 every
> > VM-Exit would be a non-starter.  E.g. simple VM-Exits are completely ha=
ndled in
> > <1500 cycles, and "fastpath" exits are something like half that.  Switc=
hing all
> > the MSRs is likely 1000+ cycles, if not double that.
>=20
> Hi Sean,
>=20
> Sorry, I have no intention to interrupt the conversation, but this is
> slightly confusing to me.
>=20
> I remember when doing AMX, we added gigantic 8KB memory in the FPU
> context switch. That works well in Linux today. Why can't we do the
> same for PMU? Assuming we context switch all counters, selectors and
> global stuff there?

That's what we (Google folks) are proposing.  However, there are significan=
t
side effects if KVM context switches PMU outside of vcpu_run(), whereas the=
 FPU
doesn't suffer the same problems.

Keeping the guest FPU resident for the duration of vcpu_run() is, in terms =
of
functionality, completely transparent to the rest of the kernel.  From the =
kernel's
perspective, the guest FPU is just a variation of a userspace FPU, and the =
kernel
is already designed to save/restore userspace/guest FPU state when the kern=
el wants
to use the FPU for whatever reason.  And crucially, kernel FPU usage is exp=
licit
and contained, e.g. see kernel_fpu_{begin,end}(), and comes with mechanisms=
 for
KVM to detect when the guest FPU needs to be reloaded (see TIF_NEED_FPU_LOA=
D).

The PMU is a completely different story.  PMU usage, a.k.a. perf, by design=
 is
"always running".  KVM can't transparently stop host usage of the PMU, as d=
isabling
host PMU usage stops perf events from counting/profiling whatever it is the=
y're
supposed to profile.

Today, KVM minimizes the "downtime" of host PMU usage by context switching =
PMU
state at VM-Enter and VM-Exit, or at least as close as possible, e.g. for L=
BRs
and Intel PT.

What we are proposing would *significantly* increase the downtime, to the p=
oint
where it would almost be unbounded in some paths, e.g. if KVM faults in a p=
age,
gup() could go swap in memory from disk, install PTEs, and so on and so for=
th.
If the host is trying to profile something related to swap or memory manage=
ment,
they're out of luck.