Received-SPF: pass (google.com: domain of linux-kernel+bounces-55-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Date: Thu, 14 Dec 2023 12:13:34 -0800
In-Reply-To: <CAO7JXPihjjko6qe8tr6e6UE=L7uSR6AACq1Zwg+7n95s5A-yoQ@mail.gmail.com>
Precedence: bulk
Mime-Version: 1.0
References: <20231214024727.3503870-1-vineeth@bitbyteword.org>
 <ZXsvl7mabUuNkWcY@google.com> <CAO7JXPihjjko6qe8tr6e6UE=L7uSR6AACq1Zwg+7n95s5A-yoQ@mail.gmail.com>
Message-ID: <ZXth7hu7jaHbJZnj@google.com>
Subject: Re: [RFC PATCH 0/8] Dynamic vcpu priority management in kvm
From: Sean Christopherson <seanjc@google.com>
To: Vineeth Remanan Pillai <vineeth@bitbyteword.org>
Cc: Ben Segall <bsegall@google.com>, Borislav Petkov <bp@alien8.de>, 
	Daniel Bristot de Oliveira <bristot@redhat.com>, Dave Hansen <dave.hansen@linux.intel.com>, 
	Dietmar Eggemann <dietmar.eggemann@arm.com>, "H . Peter Anvin" <hpa@zytor.com>, Ingo Molnar <mingo@redhat.com>, 
	Juri Lelli <juri.lelli@redhat.com>, Mel Gorman <mgorman@suse.de>, 
	Paolo Bonzini <pbonzini@redhat.com>, Andy Lutomirski <luto@kernel.org>, 
	Peter Zijlstra <peterz@infradead.org>, Steven Rostedt <rostedt@goodmis.org>, 
	Thomas Gleixner <tglx@linutronix.de>, Valentin Schneider <vschneid@redhat.com>, 
	Vincent Guittot <vincent.guittot@linaro.org>, Vitaly Kuznetsov <vkuznets@redhat.com>, 
	Wanpeng Li <wanpengli@tencent.com>, Suleiman Souhlal <suleiman@google.com>, 
	Masami Hiramatsu <mhiramat@google.com>, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, 
	x86@kernel.org, Tejun Heo <tj@kernel.org>, Josh Don <joshdon@google.com>, 
	Barret Rhoden <brho@google.com>, David Vernet <dvernet@meta.com>, 
	Joel Fernandes <joel@joelfernandes.org>
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

On Thu, Dec 14, 2023, Vineeth Remanan Pillai wrote:
> On Thu, Dec 14, 2023 at 11:38=E2=80=AFAM Sean Christopherson <seanjc@goog=
le.com> wrote:
> Now when I think about it, the implementation seems to
> suggest that we are putting policies in kvm. Ideally, the goal is:
> - guest scheduler communicates the priority requirements of the workload
> - kvm applies the priority to the vcpu task.

Why?  Tasks are tasks, why does KVM need to get involved?  E.g. if the prob=
lem
is that userspace doesn't have the right knobs to adjust the priority of a =
task
quickly and efficiently, then wouldn't it be better to solve that problem i=
n a
generic way?

> - Now that vcpu is appropriately prioritized, host scheduler can make
> the right choice of picking the next best task.
>=20
> We have an exception of proactive boosting for interrupts/nmis. I
> don't expect these proactive boosting cases to grow. And I think this
> also to be controlled by the guest where the guest can say what
> scenarios would it like to be proactive boosted.
>=20
> That would make kvm just a medium to communicate the scheduler
> requirements from guest to host and not house any policies.  What do
> you think?

...
=20
> > Pushing the scheduling policies to host userspace would allow for far m=
ore control
> > and flexibility.  E.g. a heavily paravirtualized environment where host=
 userspace
> > knows *exactly* what workloads are being run could have wildly differen=
t policies
> > than an environment where the guest is a fairly vanilla Linux VM that h=
as received
> > a small amount of enlightment.
> >
> > Lastly, if the concern/argument is that userspace doesn't have the righ=
t knobs
> > to (quickly) boost vCPU tasks, then the proposed sched_ext functionalit=
y seems
> > tailor made for the problems you are trying to solve.
> >
> > https://lkml.kernel.org/r/20231111024835.2164816-1-tj%40kernel.org
> >
> You are right, sched_ext is a good choice to have policies
> implemented. In our case, we would need a communication mechanism as
> well and hence we thought kvm would work best to be a medium between
> the guest and the host.

Making KVM be the medium may be convenient and the quickest way to get a Po=
C
out the door, but effectively making KVM a middle-man is going to be a huge=
 net
negative in the long term.  Userspace can communicate with the guest just a=
s
easily as KVM, and if you make KVM the middle-man, then you effectively *mu=
st*
define a relatively rigid guest/host ABI.

If instead the contract is between host userspace and the guest, the ABI ca=
n be
much more fluid, e.g. if you (or any setup) can control at least some amoun=
t of
code that runs in the guest, then the contract between the guest and host d=
oesn't
even need to be formally defined, it could simply be a matter of bundling h=
ost
and guest code appropriately.

If you want to land support for a given contract in upstream repositories, =
e.g.
to broadly enable paravirt scheduling support across a variety of usersepac=
e VMMs
and/or guests, then yeah, you'll need a formal ABI.  But that's still not a=
 good
reason to have KVM define the ABI.  Doing it in KVM might be a wee bit easi=
er because
it's largely just a matter of writing code, and LKML provides a centralized=
 channel
for getting buyin from all parties.  But defining an ABI that's independent=
 of the
kernel is absolutely doable, e.g. see the many virtio specs.

I'm not saying KVM can't help, e.g. if there is information that is known o=
nly
to KVM, but the vast majority of the contract doesn't need to be defined by=
 KVM.