Received-SPF: pass (google.com: domain of linux-kernel+bounces-179137-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Message-ID: <6eb9302e-59c9-4242-bfb1-e473d3e5380e@meta.com>
Date: Tue, 14 May 2024 16:22:25 -0400
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCHSET v6] sched: Implement BPF extensible scheduler class
To: Peter Zijlstra <peterz@infradead.org>, Tejun Heo <tj@kernel.org>
Cc: torvalds@linux-foundation.org, mingo@redhat.com, juri.lelli@redhat.com,
        vincent.guittot@linaro.org, dietmar.eggemann@arm.com,
        rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de,
        bristot@redhat.com, vschneid@redhat.com, ast@kernel.org,
        daniel@iogearbox.net, andrii@kernel.org, martin.lau@kernel.org,
        joshdon@google.com, brho@google.com, pjt@google.com,
        derkling@google.com, haoluo@google.com, dvernet@meta.com,
        dschatzberg@meta.com, dskarlat@cs.cmu.edu, riel@surriel.com,
        changwoo@igalia.com, himadrics@inria.fr, memxor@gmail.com,
        andrea.righi@canonical.com, joel@joelfernandes.org,
        linux-kernel@vger.kernel.org, bpf@vger.kernel.org,
        kernel-team@meta.com
References: <20240501151312.635565-1-tj@kernel.org>
 <20240502084800.GY30852@noisy.programming.kicks-ass.net>
 <ZjPnb1vdt80FrksA@slm.duckdns.org>
 <20240503085232.GC30852@noisy.programming.kicks-ass.net>
 <ZjgWzhruwo8euPC0@slm.duckdns.org>
 <20240513080359.GI30852@noisy.programming.kicks-ass.net>
From: Chris Mason <clm@meta.com>
Content-Language: en-US
In-Reply-To: <20240513080359.GI30852@noisy.programming.kicks-ass.net>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?OCttYURUVWlWVDRpQmlIUnhNWmRaQXl3Wi9xdHhVc3VQaHA4UEdhZG5ZdU5u?=
 =?utf-8?B?V2ExbXVMa2hMcmsrM04xNXZoMll4U0E5cDlzWmZJZHJVaGF3VFNBQ25MQ3Ri?=
 =?utf-8?B?d2w1OVdWbE5KWDdKTy8yTmlKbFB2cS8vTnFOU04vM0RhUHRhby9IbTBkY1V6?=
 =?utf-8?B?dGZHQjB0bmFLMFdlWWlBcko2cm91M2J4Z0VJNWwzMjJFYm5SckYvWXJ5YWI0?=
 =?utf-8?B?S2dqQUpvcmlJbjg4WnY2VTBqT2Qxc1kvVXdkam5vNWJOVWIva1p5TzJEMGdm?=
 =?utf-8?B?eldaNGNMcUMvK09yTlhwajNBb0ZkWWdlUlJ0ck1OQ2xCTTlOK3h0SERRZ1JV?=
 =?utf-8?B?ZlcvdGsxVTFEREVLVjd4V3R2V25mVnE4cTJwaGJnSzJCTFMwa3FLRlg5eStU?=
 =?utf-8?B?WUZWVzFKTkFlY3hHRjNyRUJDZkg4blFnV1BaQXN3ZjhVQnZMRjFpa2xPanhh?=
 =?utf-8?B?QStpTGsydDExZnRDaHFrRXBvWDZzQXJKY21QTjYyR2ltdGZURTk4WGZhYkRZ?=
 =?utf-8?B?clpoclVBRUYySHkzbklpMGVxOVY4OXBMbHI1TzVmc3dFejh5ZzBCV1hXMUFs?=
 =?utf-8?B?TzRGU2ZZME9YS01LeEt5Vm1kTjZqcG9hMGNOZjJhSi9ibDRTZnE0UzVyNzBX?=
 =?utf-8?B?MGYzaTNMMHhYMDZXdi9lMDZRMmF1dWRqNy91ZHNNWWpuZjY0YlBzV2ptSFJL?=
 =?utf-8?B?UWZjaldLYWY0OFh4UTNTL08vQlcwbmRLblgrNzNJbmNjL2N2cTJFa0x3M3hm?=
 =?utf-8?B?R0VPUHYxcEZybmVrbzVnd0twZDhWZkEyMWJqT3pnc0tIM1JLdzRhOEFSVnBJ?=
 =?utf-8?B?V1JPSkxjMERnMFhBUE1EWmttVitHRmI1UG02MXNyU0Rnbm5YemR2SWRLblZ1?=
 =?utf-8?B?VjZ3Qk9iNE9YQWRFd1Yzdjc1K3JGd1pzamhDNEZEUGR6Qlo5OVJBNTJFSkR2?=
 =?utf-8?B?NWZYUEhGaTE1OEZjVkxRRVFQdE9YVS9mRU1wYTNXYkR4SWIzNTdBQlBHNjVC?=
 =?utf-8?B?Z1hRbWhRNUo3Z2hXdk5Cby9zN3lRc2grRTV0ZTZlUWJFeUJyZFdGRXdhaG96?=
 =?utf-8?B?OHlMKzdmNzRTcnpHZTdvaDhYS1d3QkdnV0hNSHBXS1R0QkdRTTJtTlB1dmV0?=
 =?utf-8?B?Wjd2dkdlanBOYkZYR1hSL3RtOUFtUnQrR1M3Z29jWkx5OHc2ejVaMFhwbHZQ?=
 =?utf-8?B?MGY2M3JZejZBSmFWcm9XRy83dm5MblJ0bUFILzY5UmlHMWxnSFdub3hIUGlm?=
 =?utf-8?B?aTBmNkdSWEFsUnJxWUZvQnZSM05MSkpTQW5NaE9rcTZJeUkzZWt2M2VyNGI4?=
 =?utf-8?B?SHJuRjdHTGpJR3MwWG5JS0hQNUlXa0FJUGF5Q0FNdFVvTE01VXh5ajlIdVpy?=
 =?utf-8?B?R3h0M3YrTnZZb2pTWktOeXRhL0Yrb3h6bVkxa2gyNVdkdmY4VXp6QzVNR3F5?=
 =?utf-8?B?YkhLTHY4a2JseWY1QXhQNSsyS0RqTlVFUGxlMno0U0RqeGVSSFcyM1pwN2d6?=
 =?utf-8?B?MkptR29MVU5ZMUp0cjVpaGcvSzhFU080K2lMUUtBNlk3WTJzQ0RyZWpXZ3pR?=
 =?utf-8?B?NDNQdkJXSk93ZGRqYU0rZXByM0VsZ2Y2V3FQWm5mbTl2SERUNG9rclh3QlpB?=
 =?utf-8?B?aHdwQzN3Sk5Zd0M2VEVqYVBuNThNL3pxNVlydDJWdjRCS3FsYkg1ekxHUUhr?=
 =?utf-8?B?OVRSZTJna2dVWjJrVUtsRGRWbTJXTXlVa0k0ZlU1OGI0a2t5UWZDYXdUTVVx?=
 =?utf-8?B?NHlMNE14TFA5VHh5akc4UkhBZGJBcW82cTQ4SDlTRWtaNTJ3L1daQ1dVdlNX?=
 =?utf-8?B?dlVDNDBTQldmZ05sNDZrUmNSSnZ6ZFBHZldlNzFKdlZrSkt2Q1Q4YWxkcjUr?=
 =?utf-8?B?eXhuOW9nRTkzZGh0NEhWUWxQOFA0L24yM3hOMkcrcktySlVPTDk0VlNnSzJO?=
 =?utf-8?B?NWhHdEtHR29McE02SjRxTkFianh6dWxyZmViL1JYQzJsQnZaT1pWOFhqNGhz?=
 =?utf-8?B?NEUzQUNFYmFlalJtd0RPZFRUU2YvSVo3c3JNemhITVhBZG9idjAyYVVJblF3?=
 =?utf-8?B?NGp1a3A1UEZPVUo3bXBiVVlaaVYxM2dJM0xSWG8yQTVTaHljc3ZYZVBiTHZ0?=
 =?utf-8?Q?eH9o=3D?=
X-OriginatorOrg: meta.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 807f7bac-cacb-40f4-3aab-08dc7453a512
X-MS-Exchange-CrossTenant-AuthSource: LV3PR15MB6455.namprd15.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 May 2024 20:22:58.1753
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: HYIaKUdsYKsRiz9z1+u7vM8LMGhFx1V+UbJG8mRuJj7PtKYNRs/EpwzCBx14uVUm
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR15MB6420
X-Proofpoint-ORIG-GUID: lAeR__RG9MY9s0VVTbsRTnpbuzhG6cXL
X-Proofpoint-GUID: lAeR__RG9MY9s0VVTbsRTnpbuzhG6cXL
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.11.176.26
 definitions=2024-05-14_12,2024-05-14_01,2023-05-22_02

On 5/13/24 4:03 AM, Peter Zijlstra wrote:
> On Sun, May 05, 2024 at 01:31:26PM -1000, Tejun Heo wrote:
> 
>>> You Google/Facebook are touting collaboration, collaborate on fixing it.
>>> Instead of re-posting this over and over. After all, your main
>>> motivation for starting this was the cpu-cgroup overhead.
>>
>> The hierarchical scheduling overhead isn't the main motivation for us. We
>> can't use the CPU controller for all workloads and while it'd be nice to
>> improve that,
> 
> Hurmph, I had the impression from the earlier threads that this ~5%
> cgroup overhead was most definitely a problem and a motivator for all
> this.
> 
> The overhead was prohibitive, it was claimed, and you needed a solution.
> Did not previous versions use this very argument in order to push for
> all this?
> 
> By improving the cgroup mess -- I very much agree that the cgroup thing
> is not very nice. This whole argument goes away and we all get a better
> cgroup implementation.
> 
>> This view works only if you assume that the entire world contains only a
>> handful of developers who can work on schedulers. The only way that would be
>> the case is if the barrier of entry is raised unreasonably high. Sometimes a
>> high barrier of entry can't be avoided or is beneficial. However, if it's
>> pushed up high enough to leave only a handful of people to work on an area
>> as large as scheduling, something probably is wrong.
> 
> I've never really felt there were too few sched patches to stare at on
> any one day (quite the opposite on many days in fact).
> 
> There have also always been plenty out of tree scheduler patches --
> although I rarely if ever have time to look at them.
> 
> Writing a custom scheduler isn't that hard, simply ripping out
> fair_sched_class and replacing it with something simple really isn't
> *that* hard.
> 
> The only really hard requirement is respecting affinities, you'll crash
> and burn real hard if you get that wrong (think of all the per-cpu
> kthreads that hard rely on the per-cpu-ness of them).
> 
> But you can easily ignore cgroups, uclamp and a ton of other stuff and
> still boot and play around.
> 
>> I believe we agree that we want more people contributing to the scheduling
>> area. 
> 
> I think therein lies the rub -- contribution. If we were to do this
> thing, random loadable BPF schedulers, then how do we ensure people will
> contribute back?
> 
> That is, from where I am sitting I see $vendor mandate their $enterprise
> product needs their $BPF scheduler. At which point $vendor will have no
> incentive to ever contribute back.

Especially in the scheduler space, the incentive to contribute back
today is somewhat inverted. As you mention above, it's relatively easy
to make custom things, and it's also very difficult to get features and
patches included. The cost of maintaining patches out of tree is
relatively low in comparison with the cost of working through inclusion,
and the scheduler stands out in terms of how hard it is to land changes.

I think the scheduler balances the needs of a wide variety of workloads
exceptionally well, but based on the volume of out of tree scheduler
infrastructure, it feels like the community is struggling to meet their
collaboration needs in the upstream tree.

Just like I can’t imagine one filesystem working for everything, I think
we need to open up the field a little on schedulers.  As we develop for
new variations in workloads, power management, and hardware types, I
think sched_ext gives us a way to do more collaboration in the upstream
tree, and while I’m not pretending it’s perfect, it’s definitely ready
for expansion and broader use.

I do think that sched_ext developers will keep participating upstream,
and I agree with a lot of the points that Steve makes in his reply.
People are going to keep sending patches in because the kernel community
is just the best place to build and maintain this functionality.

-chris