Received-SPF: pass (google.com: domain of linux-kernel+bounces-11-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Precedence: bulk
MIME-Version: 1.0
References: <20231214024727.3503870-1-vineeth@bitbyteword.org> <ZXsvl7mabUuNkWcY@google.com>
In-Reply-To: <ZXsvl7mabUuNkWcY@google.com>
From: Vineeth Remanan Pillai <vineeth@bitbyteword.org>
Date: Thu, 14 Dec 2023 14:25:00 -0500
Message-ID: <CAO7JXPihjjko6qe8tr6e6UE=L7uSR6AACq1Zwg+7n95s5A-yoQ@mail.gmail.com>
Subject: Re: [RFC PATCH 0/8] Dynamic vcpu priority management in kvm
To: Sean Christopherson <seanjc@google.com>
Cc: Ben Segall <bsegall@google.com>, Borislav Petkov <bp@alien8.de>, 
	Daniel Bristot de Oliveira <bristot@redhat.com>, Dave Hansen <dave.hansen@linux.intel.com>, 
	Dietmar Eggemann <dietmar.eggemann@arm.com>, "H . Peter Anvin" <hpa@zytor.com>, Ingo Molnar <mingo@redhat.com>, 
	Juri Lelli <juri.lelli@redhat.com>, Mel Gorman <mgorman@suse.de>, 
	Paolo Bonzini <pbonzini@redhat.com>, Andy Lutomirski <luto@kernel.org>, 
	Peter Zijlstra <peterz@infradead.org>, Steven Rostedt <rostedt@goodmis.org>, 
	Thomas Gleixner <tglx@linutronix.de>, Valentin Schneider <vschneid@redhat.com>, 
	Vincent Guittot <vincent.guittot@linaro.org>, Vitaly Kuznetsov <vkuznets@redhat.com>, 
	Wanpeng Li <wanpengli@tencent.com>, Suleiman Souhlal <suleiman@google.com>, 
	Masami Hiramatsu <mhiramat@google.com>, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, 
	x86@kernel.org, Tejun Heo <tj@kernel.org>, Josh Don <joshdon@google.com>, 
	Barret Rhoden <brho@google.com>, David Vernet <dvernet@meta.com>, 
	Joel Fernandes <joel@joelfernandes.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Thu, Dec 14, 2023 at 11:38=E2=80=AFAM Sean Christopherson <seanjc@google=
.com> wrote:
>
> +sched_ext folks
>
> On Wed, Dec 13, 2023, Vineeth Pillai (Google) wrote:
> > Double scheduling is a concern with virtualization hosts where the host
> > schedules vcpus without knowing whats run by the vcpu and guest schedul=
es
> > tasks without knowing where the vcpu is physically running. This causes
> > issues related to latencies, power consumption, resource utilization
> > etc. An ideal solution would be to have a cooperative scheduling
> > framework where the guest and host shares scheduling related informatio=
n
> > and makes an educated scheduling decision to optimally handle the
> > workloads. As a first step, we are taking a stab at reducing latencies
> > for latency sensitive workloads in the guest.
> >
> > This series of patches aims to implement a framework for dynamically
> > managing the priority of vcpu threads based on the needs of the workloa=
d
> > running on the vcpu. Latency sensitive workloads (nmi, irq, softirq,
> > critcal sections, RT tasks etc) will get a boost from the host so as to
> > minimize the latency.
> >
> > The host can proactively boost the vcpu threads when it has enough
> > information about what is going to run on the vcpu - fo eg: injecting
> > interrupts. For rest of the case, guest can request boost if the vcpu i=
s
> > not already boosted. The guest can subsequently request unboost after
> > the latency sensitive workloads completes. Guest can also request a
> > boost if needed.
> >
> > A shared memory region is used to communicate the scheduling informatio=
n.
> > Guest shares its needs for priority boosting and host shares the boosti=
ng
> > status of the vcpu. Guest sets a flag when it needs a boost and continu=
es
> > running. Host reads this on next VMEXIT and boosts the vcpu thread. For
> > unboosting, it is done synchronously so that host workloads can fairly
> > compete with guests when guest is not running any latency sensitive
> > workload.
>
> Big thumbs down on my end.  Nothing in this RFC explains why this should =
be done
> in KVM.  In general, I am very opposed to putting policy of any kind into=
 KVM,
> and this puts a _lot_ of unmaintainable policy into KVM by deciding when =
to
> start/stop boosting a vCPU.
>
I am sorry for not clearly explaining the goal. The intent was not to
have scheduling policies implemented in kvm, but to have a mechanism
for guest and host schedulers to communicate so that guest workloads
get a fair treatment from host scheduler while competing with host
workloads. Now when I think about it, the implementation seems to
suggest that we are putting policies in kvm. Ideally, the goal is:
- guest scheduler communicates the priority requirements of the workload
- kvm applies the priority to the vcpu task.
- Now that vcpu is appropriately prioritized, host scheduler can make
the right choice of picking the next best task.

We have an exception of proactive boosting for interrupts/nmis. I
don't expect these proactive boosting cases to grow. And I think this
also to be controlled by the guest where the guest can say what
scenarios would it like to be proactive boosted.

That would make kvm just a medium to communicate the scheduler
requirements from guest to host and not house any policies.  What do
you think?

> Concretely, boosting vCPUs for most events is far too coarse grained.  E.=
g. boosting
> a vCPU that is running a low priority workload just because the vCPU trig=
gered
> an NMI due to PMU counter overflow doesn't make sense.  Ditto for if a gu=
est's
> hrtimer expires on a vCPU running a low priority workload.
>
> And as evidenced by patch 8/8, boosting vCPUs based on when an event is _=
pending_
> is not maintainable.  As hardware virtualizes more and more functionality=
, KVM's
> visibility into the guest effectively decreases, e.g. Intel and AMD both =
support
> with IPI virtualization.
>
> Boosting the target of a PV spinlock kick is similarly flawed.  In that c=
ase, KVM
> only gets involved _after_ there is a problem, i.e. after a lock is conte=
nded so
> heavily that a vCPU stops spinning and instead decided to HLT.  It's not =
hard to
> imagine scenarios where a guest would want to communicate to the host tha=
t it's
> acquiring a spinlock for a latency sensitive path and so shouldn't be sch=
eduled
> out.  And of course that's predicated on the assumption that all vCPUs ar=
e subject
> to CPU overcommit.
>
> Initiating a boost from the host is also flawed in the sense that it reli=
es on
> the guest to be on the same page as to when it should stop boosting.  E.g=
. if
> KVM boosts a vCPU because an IRQ is pending, but the guest doesn't want t=
o boost
> IRQs on that vCPU and thus doesn't stop boosting at the end of the IRQ ha=
ndler,
> then the vCPU could end up being boosted long after its done with the IRQ=
.
>
> Throw nested virtualization into the mix and then all of this becomes nig=
h
> impossible to sort out in KVM.  E.g. if an L1 vCPU is a running an L2 vCP=
U, i.e.
> a nested guest, and L2 is spamming interrupts for whatever reason, KVM wi=
ll end
> repeatedly boosting the L1 vCPU regardless of the priority of the L2 work=
load.
>
> For things that aren't clearly in KVM's domain, I don't think we should i=
mplement
> KVM-specific functionality until every other option has been tried (and f=
ailed).
> I don't see any reason why KVM needs to get involved in scheduling, beyon=
d maybe
> providing *input* regarding event injection, emphasis on *input* because =
KVM
> providing information to userspace or some other entity is wildly differe=
nt than
> KVM making scheduling decisions based on that information.
>
Agreed with all the points above and it doesn't make sense to have
policies in kvm. But if kvm can act as a medium to communicate
scheduling requirements between guest and host and not make any
decisions, would that be more reasonable?

> Pushing the scheduling policies to host userspace would allow for far mor=
e control
> and flexibility.  E.g. a heavily paravirtualized environment where host u=
serspace
> knows *exactly* what workloads are being run could have wildly different =
policies
> than an environment where the guest is a fairly vanilla Linux VM that has=
 received
> a small amount of enlightment.
>
> Lastly, if the concern/argument is that userspace doesn't have the right =
knobs
> to (quickly) boost vCPU tasks, then the proposed sched_ext functionality =
seems
> tailor made for the problems you are trying to solve.
>
> https://lkml.kernel.org/r/20231111024835.2164816-1-tj%40kernel.org
>
You are right, sched_ext is a good choice to have policies
implemented. In our case, we would need a communication mechanism as
well and hence we thought kvm would work best to be a medium between
the guest and the host. The policies could be in the guest and the
guest shall communicate its priority requirements(based on policy) to
the host via kvm and then the host scheduler takes action based on
that.

Please let me know.

Thanks,
Vineeth