Received-SPF: pass (google.com: domain of linux-kernel+bounces-75171-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Subject: Re: [RFC] cputime: Introduce option to force full dynticks accounting on
 NOHZ & NOHZ_IDLE CPUs
Precedence: bulk
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="UTF-8"
Date: Wed, 21 Feb 2024 17:11:09 +0000
Message-ID: <CZAX9ZA1VRUZ.353NNERCBGKUU@amazon.com>
To: Sean Christopherson <seanjc@google.com>
CC: <frederic@kernel.org>, <paulmck@kernel.org>, <jalliste@amazon.co.uk>,
	<mhiramat@kernel.org>, <akpm@linux-foundation.org>, <pmladek@suse.com>,
	<rdunlap@infradead.org>, <tsi@tuyoix.net>, <nphamcs@gmail.com>,
	<gregkh@linuxfoundation.org>, <linux-kernel@vger.kernel.org>,
	<kvm@vger.kernel.org>, <pbonzini@redhat.com>
From: Nicolas Saenz Julienne <nsaenz@amazon.com>
References: <20240219175735.33171-1-nsaenz@amazon.com>
 <ZdTQyb23KJEYqbcw@google.com> <CZA43Y64EK8R.1M8J5Q6L39LFB@amazon.com>
 <ZdYjvBItrl20oHXC@google.com>
In-Reply-To: <ZdYjvBItrl20oHXC@google.com>

On Wed Feb 21, 2024 at 4:24 PM UTC, Sean Christopherson wrote:
> On Tue, Feb 20, 2024, Nicolas Saenz Julienne wrote:
> > Hi Sean,
> >
> > On Tue Feb 20, 2024 at 4:18 PM UTC, Sean Christopherson wrote:
> > > On Mon, Feb 19, 2024, Nicolas Saenz Julienne wrote:
> > > > Under certain extreme conditions, the tick-based cputime accounting=
 may
> > > > produce inaccurate data. For instance, guest CPU usage is sensitive=
 to
> > > > interrupts firing right before the tick's expiration.
>
> Ah, this confused me.  The "right before" is a bit misleading.  It's more=
 like
> "shortly before", because if the interrupt that occurs due to the guest's=
 tick
> arrives _right_ before the host tick expires, then commit 160457140187 sh=
ould
> avoid horrific accounting.
>
> > > > This forces the guest into kernel context, and has that time slice
> > > > wrongly accounted as system time. This issue is exacerbated if the
> > > > interrupt source is in sync with the tick,
>
> It's worth calling out why this can happen, to make it clear that getting=
 into
> such syncopation can happen quite naturally.  E.g. something like:
>
>       interrupt source is in sync with the tick, e.g. if the guest's tick
>       is configured to run at the same frequency as the host tick, and th=
e
>       guest tick is every so slightly ahead of the host tick.

I'll incorporate both comments into the description. :)

> > > > significantly skewing usage metrics towards system time.
> > >
> > > ...
> > >
> > > > NOTE: This wasn't tested in depth, and it's mostly intended to high=
light
> > > > the issue we're trying to solve. Also ccing KVM folks, since it's
> > > > relevant to guest CPU usage accounting.
> > >
> > > How bad is the synchronization issue on upstream kernels?  We tried t=
o address
> > > that in commit 160457140187 ("KVM: x86: Defer vtime accounting 'til a=
fter IRQ handling").
> > >
> > > I don't expect it to be foolproof, but it'd be good to know if there'=
s a blatant
> > > flaw and/or easily closed hole.
> >
> > The issue is not really about the interrupts themselves, but their side
> > effects.
> >
> > For instance, let's say the guest sets up an Hyper-V stimer that
> > consistently fires 1 us before the preemption tick. The preemption tick
> > will expire while the vCPU thread is running with !PF_VCPU (maybe insid=
e
> > kvm_hv_process_stimers() for ex.). As long as they both keep in sync,
> > you'll get a 100% system usage. I was able to reproduce this one throug=
h
> > kvm-unit-tests, but the race window is too small to keep the interrupts
> > in sync for long periods of time, yet still capable of producing random
> > system usage bursts (which unacceptable for some use-cases).
> >
> > Other use-cases have bigger race windows and managed to maintain high
> > system CPU usage over long periods of time. For example, with user-spac=
e
> > HPET emulation, or KVM+Xen (don't know the fine details on these, but
> > VIRT_CPU_ACCOUNTING_GEN fixes the mis-accounting). It all comes down to
> > the same situation. Something triggers an exit, and the vCPU thread goe=
s
> > past 'vtime_account_guest_exit()' just in time for the tick interrupt t=
o
> > show up.
>
> I suspect the common "problem" with those flows is that emulating the gue=
st timer
> interrupt is (a) slow, relatively speaking and (b) done with interrupts e=
nabled.
>
> E.g. on VMX, the TSC deadline timer is emulated via VMX preemption timer,=
 and both
> the programming of the guest's TSC deadline timer and the handling of the=
 expiration
> interrupt is done in the VM-Exit fastpath with IRQs disabled.  As a resul=
t, even
> if the host tick interrupt is a hair behind the guest tick, it doesn't af=
fect
> accounting because the host tick interrupt will never be delivered while =
KVM is
> emulating the guest's periodic tick.
>
> I'm guessing that if you tested on SVM (or a guest that doesn't use the A=
PIC timer
> in deadline mode), which doesn't utilize the fastpath since KVM needs to =
bounce
> through hrtimers, then you'd see similar accounting problems even without=
 using
> any of the problematic "slow" timer sources.

That's right, the "problem" will show up when periodically emulating
something with interrupts enabled. The slower the emulation the bigger
the race window. It's just a limitation of tick based accounting, I have
the feeling there isn't much KVM can do.

Nicolas