Received-SPF: pass (google.com: domain of linux-kernel+bounces-169201-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Sender: Tejun Heo <htejun@gmail.com>
Date: Sun, 5 May 2024 13:31:26 -1000
From: Tejun Heo <tj@kernel.org>
To: Peter Zijlstra <peterz@infradead.org>
Cc: torvalds@linux-foundation.org, mingo@redhat.com, juri.lelli@redhat.com,
	vincent.guittot@linaro.org, dietmar.eggemann@arm.com,
	rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de,
	bristot@redhat.com, vschneid@redhat.com, ast@kernel.org,
	daniel@iogearbox.net, andrii@kernel.org, martin.lau@kernel.org,
	joshdon@google.com, brho@google.com, pjt@google.com,
	derkling@google.com, haoluo@google.com, dvernet@meta.com,
	dschatzberg@meta.com, dskarlat@cs.cmu.edu, riel@surriel.com,
	changwoo@igalia.com, himadrics@inria.fr, memxor@gmail.com,
	andrea.righi@canonical.com, joel@joelfernandes.org,
	linux-kernel@vger.kernel.org, bpf@vger.kernel.org,
	kernel-team@meta.com
Subject: Re: [PATCHSET v6] sched: Implement BPF extensible scheduler class
Message-ID: <ZjgWzhruwo8euPC0@slm.duckdns.org>
References: <20240501151312.635565-1-tj@kernel.org>
 <20240502084800.GY30852@noisy.programming.kicks-ass.net>
 <ZjPnb1vdt80FrksA@slm.duckdns.org>
 <20240503085232.GC30852@noisy.programming.kicks-ass.net>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20240503085232.GC30852@noisy.programming.kicks-ass.net>

Hello,

On Fri, May 03, 2024 at 10:52:32AM +0200, Peter Zijlstra wrote:
> On Thu, May 02, 2024 at 09:20:15AM -1000, Tejun Heo wrote:
> > We can resurrect the discussion on that patchset but how is that connected
> > to sched_ext? 
> 
> I'm absolutely not taking any of this until at the very least the cgroup
> situation that's been created is solved. And even then, I fundamentally
> believe the approach to be detrimental to the scheduler eco-system.

Please see below for more on the flattened hierarchy patchset. However, no
matter how that discussion works out, what you seem to be suggesting -
suspending discussion or further push for upstream on sched_ext until a
mostly unrelated work is done - doesn't seem reasonable, especially when
most of the input that you have provided is not constructive.

Even if we agree that, for some reason, the two projects are linked and Meta
and Google owe to push the flattened hierarchy patchset to land sched_ext
upstream, it should be obvious how your proposition puts us in an impossible
spot - A is a prerequisite for B but B isn't going to happen. That's not a
motivating situation for anyone.

If working on the flattened hierarchy patchset is something you want us to
commit to as a gesture of good will, we can surely consider that, but that
shouldn't block further discussions on sched_ext or its upstreaming.

(I reordered your comment about the number of sched_ext schedulers and
developer attention towards the end of the reply to avoid jumping back and
forth between subjects.)

> You guys Google/Facebook got us the cgroup thing, Google did a lot of

We can't divorce ourselves completely from the organizations that we work
for but the above is still a pretty broad stroke. Neither David nor I was
involved in the CPU controller design or implementation and I don't think
it's the same group of people on the Google side either. We sure can discuss
how to proceed on the flattened hierarchy patchset but I don't think the
picture you're painting is a fair depiction of the overall situation.

> the work for cpu-cgroup, and now you Facebook say you can't live with it
> because it's too expensive. Yes Rik did put a lot of effort into it, but
> Google shot it down. What am I to do?

You could have encouraged and guided the project if it felt important
enough. You didn't have to but that was an option.

> You Google/Facebook are touting collaboration, collaborate on fixing it.
> Instead of re-posting this over and over. After all, your main
> motivation for starting this was the cpu-cgroup overhead.

The hierarchical scheduling overhead isn't the main motivation for us. We
can't use the CPU controller for all workloads and while it'd be nice to
improve that, it's pretty easy to work around especially with constantly
increasing number of CPUs. Currently, most sched_ext experiments are without
cgroups and even when cgroups are considered, they're just used as grouping
hints.

In fact, we want to try implementing hierarchical scheduling by dynamically
soft-affinitizing cgroups to CPUs, which would be a bridge too far for the
in-kernel scheduler, at least for now, as it wouldn't be able to handle
custom affinities properly, but it is an idea worth exploring. Enabling
experiments like that is definitely one of our main motivations.

> From where I'm sitting, you created a problem (cpu-cgroup) and now
> you're creating an even bigger problem as a work-around. Very much not
> appreciated.

I have a hard time agreeing. These projects don't overlap all that much.
Their scopes are wildly different. That said, if this is somehow the
blocker, we can talk and try to find a solution but such a solution would
have to be reasonable from our end too. How else would it work?

> Witness the metric ton of toy schedulers written for it, that's all
> effort not put into improving the existing code.

This view works only if you assume that the entire world contains only a
handful of developers who can work on schedulers. The only way that would be
the case is if the barrier of entry is raised unreasonably high. Sometimes a
high barrier of entry can't be avoided or is beneficial. However, if it's
pushed up high enough to leave only a handful of people to work on an area
as large as scheduling, something probably is wrong.

You know better than anyone that there's no such thing as the perfect
scheduler for all, or even most, workloads. There are too many interacting
factors and second-order effects for a single implementation, no matter how
advanced, to be perfect or even great for the multitudes of situations that
scheduling encounters. With hardware and workloads becoming more complex,
the situation isn't getting any better. This partially explains why we can
easily achieve significantly better behaviors for specific workloads even
with a toy scheduler which is just there to demonstrate an idea.

The built-in scheduler has to be good enough for everyone, and, thanks to
the effort of you and the other sched maintainers, it serves that role
admirably. However, that requirement also comes with stringent constraints.
Radical ideas are difficult to play with. Each change has to make some sense
for every use case. Nothing drastic can be introduced unless the future path
can reasonably be forecast. So, development efforts must be highly
orchestrated and stay consistent which justifies a higher barrier of entry
and strict control.

Yet, the many different ways that even simple schedulers can demonstrates
sometimes significant behavior and performance benefits for specific
workloads suggest that there are a lot of low hanging fruits in the area.
Low hanging fruits that we can't easily reach from our current local
optimum. A single implementation which has to satisfy all users all the time
is unlikely to be an effective vehicle for mapping out such landscape.

I believe we agree that we want more people contributing to the scheduling
area. We need that. However, I have a hard time seeing how that would be
achieved in the current structure. Most people can't afford to sink six
months, a year, two years into a project only to eventually be nacked
without any way to deploy and prove their ideas and efforts. Unfortunately,
that is where we end up today in many cases.

There are many smart people with bright ideas just outside the fence who are
eager to develop, tune and even just play with schedulers. I believe they
will flourish when they can work in an environment where scheduling
experimentation is accessible and encouraged. In fact, we are already seeing
that. Out of the four non-trivial sched_ext schedulers, three are either
primarily driven by or have significant contributions from people who had
not and wouldn't have worked on the in-kernel schedulers at all.

So, here's my proposition. Let's please open it up. sched_ext hooks into
sched infra but the contact surface is limited and we'll try our best to
stay out of your way. I can't promise that it won't ever get in your way,
but, if it ever does, just ping me and David. Resolving such situations
would be our highest priority. Let us and others try out crazy ideas and
find out what works.

Thanks.

-- 
tejun