Received: by 2002:a05:7208:3228:b0:82:47:81bb with SMTP id cb40csp2631538rbb; Tue, 16 Apr 2024 07:18:31 -0700 (PDT) X-Forwarded-Encrypted: i=3; AJvYcCUHzWTxbuec+EMjgNQno/W2mPsa7v3wn2lz+lY4C37rmmsR8jEcn26yLC3Yn5BVxiEu/0w+EsIcnj5GOJLIdgMK/mO2OCOFNmJU1pFo+A== X-Google-Smtp-Source: AGHT+IG/I+z2Z8rWmcMGxd2oLVkBR7/a8Bf6bt0LdFFAfvuhAfJdYXu5gnEyHWKr5JmZQmAUdX3/ X-Received: by 2002:a05:6871:552:b0:22e:c4e7:8aac with SMTP id t18-20020a056871055200b0022ec4e78aacmr15628697oal.36.1713277110765; Tue, 16 Apr 2024 07:18:30 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1713277110; cv=pass; d=google.com; s=arc-20160816; b=q6QmKhj7T0jFCteyNudnzpgKM8AKN+T7BTp+drdCb1AkmPyirjcg2gzbbjbkmeUTq/ ZevJtpGQMJNGaVwt39v+crjU4gsPqDw7DMEHcppYv5gyPQQoWR7ALOS2M/YtYMzjP2fO zRto4XuUZge2odDah6rb0G2WVHWmTa7Z8ylx6bZT3dESieeolnZyt4oHkmjcm6sykHZC DqXx0PF37mLEDGhTKHgDO0xkAQGnZbKbkyncon5M5a8/yjLzP1SSgkDFMnLA1y4bkvrV HlyfB4nkkAB+UPXzCrghA/wjuPhzpM9IqNkSfyfTVkrFmX8GdUuTneQv5T8/6hEDqSwk haTQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:list-unsubscribe:list-subscribe:list-id:precedence :subject:date:from:dkim-signature; bh=SALX+sDfdvSOoP6Ft95gy+Ymn1NVMHLRE3sC5dFsIxg=; fh=Cxwo/A+C89tjOzVq8R/o/DH+l+P6Lpt8VwbqwppZI0Y=; b=A3Qe5E/qcO5AVYUSVqGY3X+sqWx1QeC+wBFncsHpn0y3+abztNO6u5wJfn2/SpZS7v /hqRqRQnEmTOvspjlbu5dazFeX1+IO5VMPwrbgBB3XTAKo9moyhN/mp1tmHsms1fXFaG 7R9+G1WlQC+WW+WBaIC0wpojMT+fk2LL2jOK33wvNBEbp/7WbEVr0BUpjHtg0sEjf42E NBTSTTc2rQAI+4ZGVi2KhZKB0focaE9UXVg4MIbriAxAGeiXuSHH4AxiXej+Yd925K9U Zwt0n6dbgqdOc9jz1TTkB1TY+X/isslYVYb83MEO63tF4zujHPt+M+rFLbnqCPoQkVqd SMdg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=jNT5AKZX; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-146967-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-146967-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id k5-20020a63ba05000000b005dc81a6b2b6si10115621pgf.782.2024.04.16.07.18.30 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Apr 2024 07:18:30 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel+bounces-146967-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=jNT5AKZX; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-146967-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-146967-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id EE8FD287E72 for ; Tue, 16 Apr 2024 14:15:06 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id DE7CC134418; Tue, 16 Apr 2024 14:10:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="jNT5AKZX" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6CCE1339A5; Tue, 16 Apr 2024 14:10:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276651; cv=none; b=HH3hnifyg4FysqcPZor2JDP+56c+ZHhXbo1qJfA8S19kmr00DSA9OyiamF0DL6KN2nlQnLtcVVTDdLvw4WYOi7zfIjPULWJbbntf1stYjwua7pnwHkQKEXRV6QDeJoy1J7jgzsTdFsz5DKas93WZGgtdQOzJ3efviPtQeGum4eA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713276651; c=relaxed/simple; bh=PseaAsRe0UQgLtMjT0SGl7UnaF8PrYknBxggC5vUVrU=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=CoEuf++WZJl0RYHxx4hfdjVdC4auFXmG0A4G+tzVLeM/uBvc0M5Oima8G78b+Qyc56ht8zdCvW8kha9E24bFekeogAlsetAVCjl7pmuiRx0URKxJixQCVD92BSxvyBdcG2N803whx4FWAYJJQA1sizmbOO+KpUtLkfEHnY2uguE= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=jNT5AKZX; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5ADDAC113CE; Tue, 16 Apr 2024 14:10:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1713276651; bh=PseaAsRe0UQgLtMjT0SGl7UnaF8PrYknBxggC5vUVrU=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=jNT5AKZXe9raa0hcklIyPMICum30AslIk4X57nihBFUwLghvPU7vA4DkQrP/whpwV N5tbq8p4D+em6zZ6mObXV1Zha+oWEnRZCrphNXZJMHRL7qkxn12WhI4o1NU0Ejagbx bquZbvg26nqsf7X+tbQAqbX9olZBjxQHdX/ElF4HParMFiNhlOjVW9UpK48gYyd8D2 nOkzwrIc/mwmalCu9gIFPSZSIvpw8lvBgWfprZoG3Pr5Usni7RNPXgjieeVo9cCc6+ 57qgYUZTapOuByoNnDzQBv+mOxgXaE3UFkTHDsWMLSEUaSK91+lLMAUxGTh/Xe1ZR6 1QL3P2vceTzaQ== From: Benjamin Tissoires Date: Tue, 16 Apr 2024 16:08:27 +0200 Subject: [PATCH bpf-next 14/18] bpf/verifier: add is_sleepable argument to push_callback_call Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20240416-bpf_wq-v1-14-c9e66092f842@kernel.org> References: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> In-Reply-To: <20240416-bpf_wq-v1-0-c9e66092f842@kernel.org> To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Mykola Lysenko , Shuah Khan Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Benjamin Tissoires X-Mailer: b4 0.12.4 X-Developer-Signature: v=1; a=ed25519-sha256; t=1713276593; l=5920; i=bentiss@kernel.org; s=20230215; h=from:subject:message-id; bh=PseaAsRe0UQgLtMjT0SGl7UnaF8PrYknBxggC5vUVrU=; b=SoWxMt1mk464DpWnaOlnqSg4DfOq8FtKLR75n5jkaMxB72wkkzSi+uM6qqeLgiXPxyr6yfu2h jkVAzdCBgZTBoY+bCNT3sfrXCgzY6RAqyPGDiDTNozgRwOHtTj6yCzw X-Developer-Key: i=bentiss@kernel.org; a=ed25519; pk=7D1DyAVh6ajCkuUTudt/chMuXWIJHlv2qCsRkIizvFw= To support sleepable async callbacks, we need to tell push_callback_call() whether the cb is sleepable or not. Doing so while checking for the kfunc arguments (when we call push_callback_call()) is simpler than adding a check for a function while inside push_callback_call(). When a callback is tagged as sleepable, the verifier now knows that it is the case and can allow a sleepable callback to happen. Signed-off-by: Benjamin Tissoires --- include/linux/bpf_verifier.h | 1 + kernel/bpf/verifier.c | 26 ++++++++++++++++---------- 2 files changed, 17 insertions(+), 10 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 36d19cd32eb5..9db35530c878 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -426,6 +426,7 @@ struct bpf_verifier_state { * while they are still in use. */ bool used_as_loop_entry; + bool in_sleepable; /* first and last insn idx of this verifier state */ u32 first_insn_idx; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 5e8c1e65fe8c..6a45d88244c6 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -1429,6 +1429,7 @@ static int copy_verifier_state(struct bpf_verifier_state *dst_state, } dst_state->speculative = src->speculative; dst_state->active_rcu_lock = src->active_rcu_lock; + dst_state->in_sleepable = src->in_sleepable; dst_state->curframe = src->curframe; dst_state->active_lock.ptr = src->active_lock.ptr; dst_state->active_lock.id = src->active_lock.id; @@ -2404,7 +2405,7 @@ static void init_func_state(struct bpf_verifier_env *env, /* Similar to push_stack(), but for async callbacks */ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env, int insn_idx, int prev_insn_idx, - int subprog) + int subprog, bool is_sleepable) { struct bpf_verifier_stack_elem *elem; struct bpf_func_state *frame; @@ -2431,6 +2432,7 @@ static struct bpf_verifier_state *push_async_cb(struct bpf_verifier_env *env, * Initialize it similar to do_check_common(). */ elem->st.branches = 1; + elem->st.in_sleepable = is_sleepable; frame = kzalloc(sizeof(*frame), GFP_KERNEL); if (!frame) goto err; @@ -5278,7 +5280,8 @@ static int map_kptr_match_type(struct bpf_verifier_env *env, static bool in_sleepable(struct bpf_verifier_env *env) { - return env->prog->sleepable; + return env->prog->sleepable || + (env->cur_state && env->cur_state->in_sleepable); } /* The non-sleepable programs and sleepable programs with explicit bpf_rcu_read_lock() @@ -9515,7 +9518,7 @@ static int btf_check_subprog_call(struct bpf_verifier_env *env, int subprog, } static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *insn, - int insn_idx, int subprog, + int insn_idx, int subprog, bool is_sleepable, set_callee_state_fn set_callee_state_cb) { struct bpf_verifier_state *state = env->cur_state, *callback_state; @@ -9550,7 +9553,7 @@ static int push_callback_call(struct bpf_verifier_env *env, struct bpf_insn *ins /* there is no real recursion here. timer callbacks are async */ env->subprog_info[subprog].is_async_cb = true; async_cb = push_async_cb(env, env->subprog_info[subprog].start, - insn_idx, subprog); + insn_idx, subprog, is_sleepable); if (!async_cb) return -EFAULT; callee = async_cb->frame[0]; @@ -10389,15 +10392,15 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn } break; case BPF_FUNC_for_each_map_elem: - err = push_callback_call(env, insn, insn_idx, meta.subprogno, + err = push_callback_call(env, insn, insn_idx, meta.subprogno, false, set_map_elem_callback_state); break; case BPF_FUNC_timer_set_callback: - err = push_callback_call(env, insn, insn_idx, meta.subprogno, + err = push_callback_call(env, insn, insn_idx, meta.subprogno, false, set_timer_callback_state); break; case BPF_FUNC_find_vma: - err = push_callback_call(env, insn, insn_idx, meta.subprogno, + err = push_callback_call(env, insn, insn_idx, meta.subprogno, false, set_find_vma_callback_state); break; case BPF_FUNC_snprintf: @@ -10412,7 +10415,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn if (err) return err; if (cur_func(env)->callback_depth < regs[BPF_REG_1].umax_value) { - err = push_callback_call(env, insn, insn_idx, meta.subprogno, + err = push_callback_call(env, insn, insn_idx, meta.subprogno, false, set_loop_callback_state); } else { cur_func(env)->callback_depth = 0; @@ -10515,7 +10518,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn break; } case BPF_FUNC_user_ringbuf_drain: - err = push_callback_call(env, insn, insn_idx, meta.subprogno, + err = push_callback_call(env, insn, insn_idx, meta.subprogno, false, set_user_ringbuf_callback_state); break; } @@ -12232,7 +12235,7 @@ static int check_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, return err; if (meta.func_id == special_kfunc_list[KF_bpf_rbtree_add_impl]) { - err = push_callback_call(env, insn, insn_idx, meta.subprogno, + err = push_callback_call(env, insn, insn_idx, meta.subprogno, false, set_rbtree_add_callback_state); if (err) { verbose(env, "kfunc %s#%d failed callback verification\n", @@ -17004,6 +17007,9 @@ static bool states_equal(struct bpf_verifier_env *env, if (old->active_rcu_lock != cur->active_rcu_lock) return false; + if (old->in_sleepable != cur->in_sleepable) + return false; + /* for states to be equal callsites have to be the same * and all frame states need to be equivalent */ -- 2.44.0