Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <CA+wXwBQ-VhK+hpBtYtyZP-NiX4g8fqRRWithFOHQW-0coQ3vLg@mail.gmail.com>
 <20220223115457.7bb575c1@gandalf.local.home> <CABWYdi1aKXWDjT+-aAQvn7eLcySbmeEucVS+8RAbUPE+bprK6A@mail.gmail.com>
 <20220805145333.1360dbbb@gandalf.local.home> <CABWYdi05VjumiXUT1sNWrLKxSVm0FeaGzjkVqbesKK3C_8nu=A@mail.gmail.com>
 <20220805182552.742c4162@gandalf.local.home>
In-Reply-To: <20220805182552.742c4162@gandalf.local.home>
From:   Ivan Babrou <ivan@cloudflare.com>
Date:   Fri, 12 Aug 2022 12:03:28 -0700
Message-ID: <CABWYdi0BumUX=UzzMDp1H2G40TSh9mL5chyAc6f7Mfgv1yW2pA@mail.gmail.com>
Subject: Re: 5.15 ftrace bug when enabling function_graph
To:     Steven Rostedt <rostedt@goodmis.org>
Cc:     Daniel Dao <dqminh@cloudflare.com>,
        kernel-team <kernel-team@cloudflare.com>,
        Ingo Molnar <mingo@redhat.com>,
        linux-kernel <linux-kernel@vger.kernel.org>,
        Masami Hiramatsu <mhiramat@kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Fri, Aug 5, 2022 at 3:25 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Fri, 5 Aug 2022 13:14:47 -0700
> Ivan Babrou <ivan@cloudflare.com> wrote:
>
> > On Fri, Aug 5, 2022 at 11:53 AM Steven Rostedt <rostedt@goodmis.org> wrote:
> > >
> > > On Fri, 5 Aug 2022 11:37:56 -0700
> > > Ivan Babrou <ivan@cloudflare.com> wrote:
> > >
> > > > root@foo:~# echo 'bpf_dispatcher_*_func' >
> > > > /sys/kernel/tracing/set_ftrace_notrace
> > > >
> > > > root@foo:~# trace-cmd record -p function_graph -l handle_mm_fault -P 3367417
> > > > plugin 'function_graph'
> > > > Hit Ctrl^C to stop recording
> > >
> > >
> > > > [1459709.204253] ------------[ ftrace bug ]------------
> > > > [1459709.204257] ftrace failed to modify
> > > > [1459709.204262] [<ffffffffac3ea960>] bpf_dispatcher_xdp_func+0x0/0x10
> > > > [1459709.204275]  actual:   ffffffe9:ffffff9b:76:ffffffb0:14
> > > > [1459709.204302] Setting ftrace call site to call ftrace function
> > > > [1459709.204305] ftrace record flags: 10000001
> > > > [1459709.204309]  (1)
> > > >                   expected tramp: ffffffffabc4bcb0
> > > > [1459709.204325] ------------[ cut here ]------------
> > >
> > > Did you reboot since the error happened? Because the command you showed
> > > should not have touched bpf_dispatcher_xdp_func.
> >
> > I did reboot, but I can reproduce if you have things for me to try.
>
> Let's see if this works manually.
>
> Reboot again.
>
> Make sure the above "ftrace bug" is not there.
>
> If not already mounted, as root:
>
>  # mount -t tracefs tracefs /sys/krenel/tracing
>  # cd /sys/kernel/tracing
>  # cat enabled_functions
>
> Let me know what is there.
>
>  # echo handle_mm_fault > set_ftrace_filter
>  # cat set_ftrace_filter
>
> to make sure that handle_mm_fault is listed.
>
>  # echo function_graph > current_tracer
>
> If it fails again, let me know. It should not, but if it does, it needs to
> be investigated.

It does not fail this way. It also doesn't fail when I run perf:

perf ftrace -N 'bpf_dispatcher_*_func' -G handle_mm_fault -a -- sleep 1

It does fail without "-N 'bpf_dispatcher_*_func'".

I suppose there's something about trace-cmd that makes it fail for me.