Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <2bef1006-3798-3fbe-87ad-4adfaee08cc0@google.com>
Date:   Mon, 24 Jul 2023 11:11:10 -0400
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Firefox/102.0 Thunderbird/102.12.0
Subject: Re: [PATCHSET v4] sched: Implement BPF extensible scheduler class
Content-Language: en-US
To:     Tejun Heo <tj@kernel.org>
Cc:     torvalds@linux-foundation.org, mingo@redhat.com,
        peterz@infradead.org, juri.lelli@redhat.com,
        vincent.guittot@linaro.org, dietmar.eggemann@arm.com,
        rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de,
        bristot@redhat.com, vschneid@redhat.com, ast@kernel.org,
        daniel@iogearbox.net, andrii@kernel.org, martin.lau@kernel.org,
        joshdon@google.com, pjt@google.com, derkling@google.com,
        haoluo@google.com, dvernet@meta.com, dschatzberg@meta.com,
        dskarlat@cs.cmu.edu, riel@surriel.com,
        linux-kernel@vger.kernel.org, bpf@vger.kernel.org,
        kernel-team@meta.com
References: <20230711011412.100319-1-tj@kernel.org>
 <ZLrQdTvzbmi5XFeq@slm.duckdns.org>
From:   Barret Rhoden <brho@google.com>
In-Reply-To: <ZLrQdTvzbmi5XFeq@slm.duckdns.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk

Hi -

On 7/21/23 14:37, Tejun Heo wrote:
> Hello,
> 
> It's been more than half a year since the initial posting of the patchset
> and we are now at the fourth iteration. There have been some reviews around
> specifics (should be all addressed now except for the ones Andrea raised on
> this iteration) but none at high level. There also were some in-person and
> off-list discussions. Some, I believe, are addressed by the cover letter but
> it'd be nonetheless useful to delve into them on-list.
> 
> On our side, we've been diligently experimenting. 

On the google side, we're still experimenting and developing schedulers 
based on ghost, which we think we can port over to sched_ext.

Specifically, I've been working on a framework to write multicore 
schedulers in BPF called 'Flux'.  The idea in brief is to compose a 
scheduler as a hierarchy of "subschedulers", where cpus allocations go 
up and down the tree.

Flux is open-source, but it needs the ghost kernel and our BPF 
extensions currently (which are also open source, but harder to use for 
people).  I'll send a proposal to talk about it at LPC in case people 
are interested - if not the scheduler framework itself, then as a "this 
is some crazy stuff people can do with BPF".

As far as results go, I wrote a custom scheduler with Flux for our 
Search app and have been testing it on our single-leaf loadtester.  The 
initial results out of the box were pretty great: 17% QPS increase, 43% 
p99 decrease (default settings for the loadtester).  But the loadtester 
varies a bit, so it's hard to get reliable numbers out of it for an A/B 
comparison of schedulers.  Overall, we run equal or better than CFS.  I 
did a sweep across various offered loads, and we got 5% better QPS on 
average, 30% better p99 latency, 6% lower utilization.  The better 
numbers come under higher load, as you'd expect, when there are more 
threads competing for the cpu.

The big caveat to those numbers is the single-leaf loadtester isn't a 
representative test.  It's more of a microbenchmark.  Our next step is 
to run a full cluster load test, which will give us a better signal.

Anyway, this scheduler is highly specific to our app, including shared 
memory regions where the app's threads can tell us stuff like RPC 
deadlines.  It's the sort of thing you could only reasonably do with a 
pluggable scheduler like sched_ext or ghost.


> We are comfortable with the current API. Everything we tried fit pretty
> well. It will continue to evolve but sched_ext now seems mature enough for
> initial inclusion. I suppose lack of response doesn't indicate tacit
> agreement from everyone, so what are you guys all thinking?

Btw, I backported your patchset to our "franken-kernel".  I was able to 
boot it on one of our nodes, and run the search loadtest on CFS. 
Nothing broke, performance was the same, etc.  Not a huge surprise, 
since I didn't turn on sched_ext.  I haven't been able to get a 
sched_ext scheduler to work yet with our kernel - there's more patch 
backporting needed for your schedulers themselves (the bpf_for iterators 
and whatnot).  I'll report back if/when I can get it running.

Thanks,

Barret