Received-SPF: pass (google.com: domain of linux-kernel+bounces-274-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Precedence: bulk
MIME-Version: 1.0
References: <CACkBjsbj4y4EhqpV-ZVt645UtERJRTxfEab21jXD1ahPyzH4_g@mail.gmail.com>
 <CAEf4BzZ0xidVCqB47XnkXcNhkPWF6_nTV7yt+_Lf0kcFEut2Mg@mail.gmail.com>
 <CACkBjsaEQxCaZ0ERRnBXduBqdw3MXB5r7naJx_anqxi0Wa-M_Q@mail.gmail.com>
 <480a5cfefc23446f7c82c5b87eef6306364132b9.camel@gmail.com>
 <917DAD9F-8697-45B8-8890-D33393F6CDF1@gmail.com> <9dee19c7d39795242c15b2f7aa56fb4a6c3ebffa.camel@gmail.com>
 <73d021e3f77161668aae833e478b210ed5cd2f4d.camel@gmail.com>
In-Reply-To: <73d021e3f77161668aae833e478b210ed5cd2f4d.camel@gmail.com>
From: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Date: Thu, 14 Dec 2023 16:06:07 -0800
Message-ID: <CAEf4BzYuV3odyj8A77ZW8H9jyx_YLhAkSiM+1hkvtH=OYcHL3w@mail.gmail.com>
Subject: Re: [Bug Report] bpf: incorrectly pruning runtime execution path
To: Eduard Zingerman <eddyz87@gmail.com>
Cc: Hao Sun <sunhao.th@gmail.com>, Alexei Starovoitov <ast@kernel.org>, 
	Andrii Nakryiko <andrii@kernel.org>, Daniel Borkmann <daniel@iogearbox.net>, bpf <bpf@vger.kernel.org>, 
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Thu, Dec 14, 2023 at 8:26=E2=80=AFAM Eduard Zingerman <eddyz87@gmail.com=
> wrote:
>
> On Thu, 2023-12-14 at 17:10 +0200, Eduard Zingerman wrote:
> > [...]
> > > The reason why retval checks fails is that the way you disable dead
> > > code removal pass is not complete. Disable opt_remove_dead_code()
> > > just prevent the instruction #30 from being removed, but also note
> > > opt_hard_wire_dead_code_branches(), which convert conditional jump
> > > into unconditional one, so #30 is still skipped.
> > >
> > > > Note that I tried this test with two functions:
> > > > - bpf_get_current_cgroup_id, with this function I get retval 2, not=
 4 :)
> > > > - bpf_get_prandom_u32, with this function I get a random retval eac=
h time.
> > > >
> > > > What is the expectation when 'bpf_get_current_cgroup_id' is used?
> > > > That it is some known (to us) number, but verifier treats it as unk=
nown scalar?
> > > >
> > >
> > > Either one would work, but to make #30 always taken, r0 should be
> > > non-zero.
> >
> > Oh, thank you, I made opt_hard_wire_dead_code_branches() a noop,
> > replaced r0 =3D 0x4 by r0 /=3D 0 and see "divide error: 0000 [#1] PREEM=
PT SMP NOPTI"
> > error in the kernel log on every second or third run of the test
> > (when using prandom).
> >
> > Working to minimize the test case will share results a bit later.
>
> Here is the minimized version of the test:
> https://gist.github.com/eddyz87/fb4d3c7d5aabdc2ae247ed73fefccd32
>
> If executed several times: ./test_progs -vvv -a verifier_and/pruning_test
> it eventually crashes VM with the following error:
>
> [    2.039066] divide error: 0000 [#1] PREEMPT SMP NOPTI
>                ...
> [    2.039987] Call Trace:
> [    2.039987]  <TASK>
> [    2.039987]  ? die+0x36/0x90
> [    2.039987]  ? do_trap+0xdb/0x100
> [    2.039987]  ? bpf_prog_32cfdb2c00b08250_pruning_test+0x4d/0x60
> [    2.039987]  ? do_error_trap+0x7d/0x110
> [    2.039987]  ? bpf_prog_32cfdb2c00b08250_pruning_test+0x4d/0x60
> [    2.039987]  ? exc_divide_error+0x38/0x50
> [    2.039987]  ? bpf_prog_32cfdb2c00b08250_pruning_test+0x4d/0x60
> [    2.039987]  ? asm_exc_divide_error+0x1a/0x20
> [    2.039987]  ? bpf_prog_32cfdb2c00b08250_pruning_test+0x4d/0x60
> [    2.039987]  bpf_test_run+0x1b5/0x350
> [    2.039987]  ? bpf_test_run+0x115/0x350
>                ...
>
> I'll continue debugging this a bit later today.
>

Great, thanks a lot, Eduard. Let's paste the program here for discussion:

$ cat progs/verifier_blah.c
// SPDX-License-Identifier: GPL-2.0
/* Copyright (C) 2023 SUSE LLC */
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#include "bpf_misc.h"

SEC("socket")
__success
__flag(BPF_F_TEST_STATE_FREQ)
__retval(42)
__naked void pruning_test(void)
{
        asm volatile (
        "   call %[bpf_get_prandom_u32];\n"
        "   r7 =3D r0;\n"
        "   call %[bpf_get_prandom_u32];\n"
        "   r8 =3D 2;\n"
        "   if r0 > 1 goto 1f;\n"
        "   r8 =3D r7;\n"
        "1: r5 =3D r8;\n"
        "   if r8 >=3D r0 goto 2f;\n"
        "   r8 +=3D r8;\n"
        "   if r5 =3D=3D 0 goto 2f;\n"
        "   r0 /=3D 0;\n"
        "2: r0 =3D 42;\n"
        "   exit;\n"
        :
        : __imm(bpf_get_prandom_u32)
        : __clobber_all);
}

char _license[] SEC("license") =3D "GPL";


If we look at relevant portion of verifier log for `if r5 =3D=3D 0` we see =
this:

9: (15) if r5 =3D=3D 0x0 goto pc+1
mark_precise: frame0: last_idx 9 first_idx 7 subseq_idx -1
mark_precise: frame0: regs=3Dr5,r7 stack=3D before 8: (0f) r8 +=3D r8
mark_precise: frame0: regs=3Dr5,r7 stack=3D before 7: (3d) if r8 >=3D r0 go=
to pc+3

^^ Note here that we only have r5 and r7, not r8.

mark_precise: frame0: parent state regs=3Dr5,r7 stack=3D:
R0_rw=3Dscalar(smin=3Dsmin32=3D0,smax=3Dumax=3Dsmax32=3Dumax32=3D1,var_off=
=3D(0x0;
0x1)) R5_rw=3DPscalar(id=3D1) R7_w=3DPscalar(id=3D1) R8_rw=3Dscalar(id=3D1)
R10=3Dfp0
mark_precise: frame0: last_idx 6 first_idx 0 subseq_idx 7
mark_precise: frame0: regs=3Dr5,r7,r8 stack=3D before 6: (bf) r5 =3D r8
mark_precise: frame0: regs=3Dr7,r8 stack=3D before 5: (bf) r8 =3D r7
mark_precise: frame0: regs=3Dr7 stack=3D before 4: (25) if r0 > 0x1 goto pc=
+1
mark_precise: frame0: regs=3Dr7 stack=3D before 3: (b7) r8 =3D 2
mark_precise: frame0: regs=3Dr7 stack=3D before 2: (85) call bpf_get_prando=
m_u32#7
mark_precise: frame0: regs=3Dr7 stack=3D before 1: (bf) r7 =3D r0
mark_precise: frame0: regs=3Dr0 stack=3D before 0: (85) call bpf_get_prando=
m_u32#7

Note above that r0 in `if r8 >=3D r0` is not marked as precise because
at that point we don't know that r8 should be precise (due to us
"forgetting" linked ID information).

Now, let's comment out the "r8 +=3D r8" instruction so that we preserve
linkage between r5 and r8 (and also r7, but that's less relevant
here).

8: (15) if r5 =3D=3D 0x0 goto pc+1
mark_precise: frame0: last_idx 8 first_idx 7 subseq_idx -1
mark_precise: frame0: regs=3Dr5,r7,r8 stack=3D before 7: (3d) if r8 >=3D r0=
 goto pc+2

^^ Here note how we seek for r5,r7, *and* r8 to be precise...

mark_precise: frame0: parent state regs=3Dr0,r5,r7,r8 stack=3D:

... which leads to us adding r0 to the set due to that `if r8 >=3D r0`
instruction.

(btw, I was wrong yesterday, we do have logic to mark *both* registers
of conditional jump if at least one of them is precise, so seems like
we handle that well)


 R0_rw=3DPscalar(smin=3Dsmin32=3D0,smax=3Dumax=3Dsmax32=3Dumax32=3D1,var_of=
f=3D(0x0;
0x1)) R5_rw=3DPscalar(id=3D1) R7_w=3DPscalar(id=3D1) R8_rw=3DPscalar(id=3D1=
)
R10=3Dfp0
mark_precise: frame0: last_idx 6 first_idx 0 subseq_idx 7
mark_precise: frame0: regs=3Dr0,r5,r7,r8 stack=3D before 6: (bf) r5 =3D r8
mark_precise: frame0: regs=3Dr0,r7,r8 stack=3D before 5: (bf) r8 =3D r7
mark_precise: frame0: regs=3Dr0,r7 stack=3D before 4: (25) if r0 > 0x1 goto=
 pc+1
mark_precise: frame0: regs=3Dr0,r7 stack=3D before 3: (b7) r8 =3D 2
mark_precise: frame0: regs=3Dr0,r7 stack=3D before 2: (85) call
bpf_get_prandom_u32#7
mark_precise: frame0: regs=3Dr7 stack=3D before 1: (bf) r7 =3D r0
mark_precise: frame0: regs=3Dr0 stack=3D before 0: (85) call bpf_get_prando=
m_u32#7


So all in all, I still think that the root cause is what I said
yesterday. We don't preserve information about linked registers at the
per-instruction level, but we should.


If you agree with the analysis, we can start discussing what's the
best way to fix this.