Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <20230607125911.145345-1-imagedong@tencent.com>
 <20230607125911.145345-2-imagedong@tencent.com> <4ca27e23-b027-0e39-495b-2ba3376342cc@meta.com>
In-Reply-To: <4ca27e23-b027-0e39-495b-2ba3376342cc@meta.com>
From:   Menglong Dong <menglong8.dong@gmail.com>
Date:   Fri, 9 Jun 2023 10:12:54 +0800
Message-ID: <CADxym3a=_FF3NUG3-210GQN0JSvbcsGdYRiVwBEQzGTtqN3kVQ@mail.gmail.com>
Subject: Re: [PATCH bpf-next v3 1/3] bpf, x86: allow function arguments up to
 12 for TRACING
To:     Yonghong Song <yhs@meta.com>
Cc:     alexei.starovoitov@gmail.com, davem@davemloft.net,
        dsahern@kernel.org, ast@kernel.org, daniel@iogearbox.net,
        andrii@kernel.org, martin.lau@linux.dev, song@kernel.org,
        yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org,
        sdf@google.com, x86@kernel.org, imagedong@tencent.com,
        benbjiang@tencent.com, netdev@vger.kernel.org, bpf@vger.kernel.org,
        linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Fri, Jun 9, 2023 at 5:07=E2=80=AFAM Yonghong Song <yhs@meta.com> wrote:
>
>
>
> On 6/7/23 5:59 AM, menglong8.dong@gmail.com wrote:
> > From: Menglong Dong <imagedong@tencent.com>
> >
> > For now, the BPF program of type BPF_PROG_TYPE_TRACING can only be used
> > on the kernel functions whose arguments count less than 6. This is not
> > friendly at all, as too many functions have arguments count more than 6=
.
>
> Since you already have some statistics, maybe listed in the commit messag=
e.
>
> >
> > Therefore, let's enhance it by increasing the function arguments count
> > allowed in arch_prepare_bpf_trampoline(), for now, only x86_64.
> >
> > For the case that we don't need to call origin function, which means
> > without BPF_TRAMP_F_CALL_ORIG, we need only copy the function arguments
> > that stored in the frame of the caller to current frame. The arguments
> > of arg6-argN are stored in "$rbp + 0x18", we need copy them to
> > "$rbp - regs_off + (6 * 8)".
>
> Maybe I missed something, could you explain why it is '$rbp + 0x18'?
>
> In the current upstream code, we have
>
>          /* Generated trampoline stack layout:
>           *
>           * RBP + 8         [ return address  ]
>           * RBP + 0         [ RBP             ]
>           *
>           * RBP - 8         [ return value    ]  BPF_TRAMP_F_CALL_ORIG or
>           *
> BPF_TRAMP_F_RET_FENTRY_RET flags
>           *
>           *                 [ reg_argN        ]  always
>           *                 [ ...             ]
>           * RBP - regs_off  [ reg_arg1        ]  program's ctx pointer
>           *
>           * RBP - nregs_off [ regs count      ]  always
>           *
>           * RBP - ip_off    [ traced function ]  BPF_TRAMP_F_IP_ARG flag
>           *
>           * RBP - run_ctx_off [ bpf_tramp_run_ctx ]
>           */
>
> Next on-stack argument will be RBP + 16, right?
>

Sorry for the confusing, it seems there should be
some comments here.

It's not the next on-stack argument, but the next next on-stack
argument. The call chain is:

caller -> origin call -> trampoline

So, we have to skip the "RIP" in the stack frame of "origin call",
which means RBP + 16 + 8. To be clear, there are only 8-byte
in the stack frame of "origin call".

Thanks!
Menglong Dong


> >
> > For the case with BPF_TRAMP_F_CALL_ORIG, we need prepare the arguments
> > in stack before call origin function, which means we need alloc extra
> > "8 * (arg_count - 6)" memory in the top of the stack. Note, there shoul=
d
> > not be any data be pushed to the stack before call the origin function.
> > Then, we have to store rbx with 'mov' instead of 'push'.
> >
> > We use EMIT3_off32() or EMIT4() for "lea" and "sub". The range of the
> > imm in "lea" and "sub" is [-128, 127] if EMIT4() is used. Therefore,
> > we use EMIT3_off32() instead if the imm out of the range.
> >
> > It works well for the FENTRY and FEXIT, I'm not sure if there are other
> > complicated cases.
>
> MODIFY_RETURN is also impacted by this patch.
>
> >
> > Reviewed-by: Jiang Biao <benbjiang@tencent.com>
> > Signed-off-by: Menglong Dong <imagedong@tencent.com>
> [...]