Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 5 Aug 2022 18:25:52 -0400
From:   Steven Rostedt <rostedt@goodmis.org>
To:     Ivan Babrou <ivan@cloudflare.com>
Cc:     Daniel Dao <dqminh@cloudflare.com>,
        kernel-team <kernel-team@cloudflare.com>,
        Ingo Molnar <mingo@redhat.com>,
        linux-kernel <linux-kernel@vger.kernel.org>,
        Masami Hiramatsu <mhiramat@kernel.org>
Subject: Re: 5.15 ftrace bug when enabling function_graph
Message-ID: <20220805182552.742c4162@gandalf.local.home>
In-Reply-To: <CABWYdi05VjumiXUT1sNWrLKxSVm0FeaGzjkVqbesKK3C_8nu=A@mail.gmail.com>
References: <CA+wXwBQ-VhK+hpBtYtyZP-NiX4g8fqRRWithFOHQW-0coQ3vLg@mail.gmail.com>
        <20220223115457.7bb575c1@gandalf.local.home>
        <CABWYdi1aKXWDjT+-aAQvn7eLcySbmeEucVS+8RAbUPE+bprK6A@mail.gmail.com>
        <20220805145333.1360dbbb@gandalf.local.home>
        <CABWYdi05VjumiXUT1sNWrLKxSVm0FeaGzjkVqbesKK3C_8nu=A@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Precedence: bulk

On Fri, 5 Aug 2022 13:14:47 -0700
Ivan Babrou <ivan@cloudflare.com> wrote:

> On Fri, Aug 5, 2022 at 11:53 AM Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > On Fri, 5 Aug 2022 11:37:56 -0700
> > Ivan Babrou <ivan@cloudflare.com> wrote:
> >  
> > > root@foo:~# echo 'bpf_dispatcher_*_func' >
> > > /sys/kernel/tracing/set_ftrace_notrace
> > >
> > > root@foo:~# trace-cmd record -p function_graph -l handle_mm_fault -P 3367417
> > > plugin 'function_graph'
> > > Hit Ctrl^C to stop recording  
> >
> >  
> > > [1459709.204253] ------------[ ftrace bug ]------------
> > > [1459709.204257] ftrace failed to modify
> > > [1459709.204262] [<ffffffffac3ea960>] bpf_dispatcher_xdp_func+0x0/0x10
> > > [1459709.204275]  actual:   ffffffe9:ffffff9b:76:ffffffb0:14
> > > [1459709.204302] Setting ftrace call site to call ftrace function
> > > [1459709.204305] ftrace record flags: 10000001
> > > [1459709.204309]  (1)
> > >                   expected tramp: ffffffffabc4bcb0
> > > [1459709.204325] ------------[ cut here ]------------  
> >
> > Did you reboot since the error happened? Because the command you showed
> > should not have touched bpf_dispatcher_xdp_func.  
> 
> I did reboot, but I can reproduce if you have things for me to try.

Let's see if this works manually.

Reboot again.

Make sure the above "ftrace bug" is not there.

If not already mounted, as root:

 # mount -t tracefs tracefs /sys/krenel/tracing
 # cd /sys/kernel/tracing
 # cat enabled_functions

Let me know what is there.

 # echo handle_mm_fault > set_ftrace_filter
 # cat set_ftrace_filter

to make sure that handle_mm_fault is listed.

 # echo function_graph > current_tracer

If it fails again, let me know. It should not, but if it does, it needs to
be investigated.

Oh, and what exactly is this kernel version. The backtrace you posted shows:

  5.15.19-cloudflare-2022.2.1

If you can not reproduce this on a vanilla 5.15.19 kernel, then it may be a
bad backport of some patch.

-- Steve