Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Date:   Wed, 15 Sep 2021 18:42:32 -0700
From:   Roman Gushchin <guro@fb.com>
To:     Hao Luo <haoluo@google.com>
CC:     Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@redhat.com>,
        Alexei Starovoitov <ast@kernel.org>,
        Daniel Borkmann <daniel@iogearbox.net>,
        Andrii Nakryiko <andrii@kernel.org>,
        Rik van Riel <riel@surriel.com>,
        Mel Gorman <mgorman@techsingularity.net>,
        Vincent Guittot <vincent.guittot@linaro.org>,
        Valentin Schneider <Valentin.Schneider@arm.com>,
        Will Deacon <will@kernel.org>,
        Dietmar Eggemann <dietmar.eggemann@arm.com>,
        Barret Rhoden <brho@google.com>, <bpf@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH rfc 0/6] Scheduler BPF
Message-ID: <YUKhCHarl+Ng1xxR@carbon.dhcp.thefacebook.com>
References: <20210915213550.3696532-1-guro@fb.com>
 <CA+khW7i460ey-UFzpMSJ8AP9QeD8ufa4FzLA4PQckNP00ShQSw@mail.gmail.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <CA+khW7i460ey-UFzpMSJ8AP9QeD8ufa4FzLA4PQckNP00ShQSw@mail.gmail.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?0Aoyw5nLzCPJnGD6evl6fBHv+vksSjYeS1c4tsBp/DWDA/DNCDJTkiytQhjD?=
 =?us-ascii?Q?QSw0nMQe/Rs/4GKzcTGpDO8yokjRoNdA7mI8ThkSAjlmMD/FwceWi5veFOmk?=
 =?us-ascii?Q?rdHgFmid6sJ/4BWVMV9qhm+ifwvpwjDj/QYEZqs7bz1Pf+w9mh6yDx1LSSBQ?=
 =?us-ascii?Q?dXIgNFphJGwPXeJs97VUUaRra6uhBOv1XlrYu2FxF5512cgzP7qsYHl6/3pb?=
 =?us-ascii?Q?y+LA/u80LhOJjEij0P54TzA5b6NlvIDXgpgx2PDiYTI9a1Xn92kwuGTTaqij?=
 =?us-ascii?Q?kFNpfA+o/5+IOepO0fzU21RuJ+WezNl93MxD1OFcfSQgIUeBIR53LK5Hq/I7?=
 =?us-ascii?Q?2lGKaxN2ACdwDJieIL+z0tseYOm+/Iuu2rTjRLxPiN4zvyAz1DcvnFDhSzeE?=
 =?us-ascii?Q?6PjfFAFYOpRJW+BbknHRQwb3O429t5iwgkVyz+/HasZEsJTc0+fC332sL2tc?=
 =?us-ascii?Q?IMPSbCRitm8+q0ioUN2MJlY8xduR73OCNyBkRuauFI4Q2BTy7dCAoJ/DRGg/?=
 =?us-ascii?Q?7eZC4uj/TVtiHfN2p8H4JjrpuOT6rQRKiB6afCw8plgPeyAUHylxQAkKI9YV?=
 =?us-ascii?Q?0KlFhNas2SNKFAlVXmlrm7zsbJiuB5oOKY0nMQXF5gjqU08+9bbmgmcxNTbF?=
 =?us-ascii?Q?iS984i1y/FgfCs003Fc8Ipo+6deOnYuiUM4odrGqBKJbPPSGd49Ph5RsD3oe?=
 =?us-ascii?Q?qjDvJUiUDAa1Hhtmcx3qsdiOuqQ4eBXwGSVUpwZ9nIkEKzUQttGKY0pmywO6?=
 =?us-ascii?Q?v/bCZUbkfZh67NPfEIs2Ttc4NDev1x3jpBLqAlqUnOP5kBmfMtr12bWR6jSW?=
 =?us-ascii?Q?7IHJ+PDOSkkVbsfszE7J/ql1Ha+jSfz2WnC5Sm1W1KaOqEMCWS3WJikVLKzo?=
 =?us-ascii?Q?pa2+ye4BGHMv0dejpfPkkXSDhPZCKEANUC9tNhXKkya1IkvSN1wovnppLk2k?=
 =?us-ascii?Q?x1ZR6/8tG6WuqM/y9l+HEB31en8xOU/9WCwOWeuYq87niKEGSaF8fujjggEg?=
 =?us-ascii?Q?bjh3m/6IxybZmx4OAI6KYwpTer742uKlHyrZJPlJUDTfHhq+OpAIUiNDKwA+?=
 =?us-ascii?Q?Tnh6dVXEc5/gXOU2C8nmDkZpEhowUP8zzgF6/o8nvNHGQqEu90U2sG23Ywd9?=
 =?us-ascii?Q?FZcCnvSqd13jlcnjB6wZH/54dVk8C70ewY6YzY0PC2SkBZCcVf+hU1DnzzBx?=
 =?us-ascii?Q?EplM7wE7XmcpUpk0mx49g27Fm1bCYfUEmykNBubeReRKNwJUJr03gRe7nOGL?=
 =?us-ascii?Q?3CEKdRxlGGVEVxUbvGlh8GFyN+OKWKOkmIrO4nean52e5GyVSDFpNTCvtytt?=
 =?us-ascii?Q?fHlTTYRYMDncospHFui4abrY0gzhbr2ZEEdJjJQy1OYtZw=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: d4d09dc7-e041-4380-52d3-08d978b3422c
X-MS-Exchange-CrossTenant-AuthSource: BYAPR15MB4136.namprd15.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Sep 2021 01:42:35.7014
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: DXZR5GF/+hRx7vCFC31KyLWjcStOJAT62bUfIt3Zt35pwT8lZxNDf/HyMJd0a80e
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR15MB2871
X-OriginatorOrg: fb.com
X-Proofpoint-GUID: DwIPp9AvWvy5Ozw0dj9OnkPisvF-178M
X-Proofpoint-ORIG-GUID: DwIPp9AvWvy5Ozw0dj9OnkPisvF-178M
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.182.1,Aquarius:18.0.790,Hydra:6.0.391,FMLib:17.0.607.475
 definitions=2021-09-15_07,2021-09-15_01,2020-04-07_01
X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 priorityscore=1501
 suspectscore=0 bulkscore=0 adultscore=0 clxscore=1011 mlxlogscore=691
 lowpriorityscore=0 impostorscore=0 spamscore=0 phishscore=0 malwarescore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2109030001 definitions=main-2109160009
X-FB-Internal: deliver
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Sep 15, 2021 at 05:19:03PM -0700, Hao Luo wrote:
> Hi Roman,
> 
> On Wed, Sep 15, 2021 at 3:04 PM Roman Gushchin <guro@fb.com> wrote:
> >

Hi Hao!

> 
> Thanks for initiating the effort of bringing BPF to sched. I've been
> looking at the potential applications of BPF in sched for some time
> and I'm very excited about this work!
> 
> My current focus has been using BPF for profiling performance and
> exporting sched related stats. I think BPF can provide a great help
> there. We have many users in Google that want the kernel to export
> various scheduling metrics to userspace. I think BPF is a good fit for
> such a task. So one of my recent attempts is to use BPF to account for
> the forced idle time caused by core scheduling [1]. This is one of the
> topics I want to discuss in my upcoming LPC BPF talk [2].

I guess for profiling we don't necessarily need a dedicated program type
etc, but it might be convenient, and some helpers can be useful too.

Unfortunately I won't be able to attend your talk, but hopefully I can
see it in a record later. I'm very interested.

> 
> Looking forward, I agree that BPF has a great potential in customizing
> policies in the scheduler. It has the advantage of quick
> experimentation and deployment. One of the use cases I'm thinking of
> is to customize load balancing policies. For example, allow using BPF
> to influence whether a task can migrate (can_migrate_task). This is
> currently only an idea.
> 
> > Our very first experiments with using BPF in CFS look very promising. We're
> > at a very early stage, however already have seen a nice latency and ~1% RPS
> > wins for our (Facebook's) main web workload.
> >
> > As I know, Google is working on a more radical approach [2]: they aim to move
> > the scheduling code into userspace. It seems that their core motivation is
> > somewhat similar: to make the scheduler changes easier to develop, validate
> > and deploy. Even though their approach is different, they also use BPF for
> > speeding up some hot paths. I think the suggested infrastructure can serve
> > their purpose too.
> 
> Yes. Barret can talk more about this, but I think it summarized the
> work of ghOSt [3] and the use of BPF in ghOSt well.

I took a brief look over how you use BPF in ghOSt and I think what I suggest
will work for you as well. I'd appreciate any comments/feedback whether it's
definitely true.

Thank you!

Roman