Received-SPF: pass (google.com: domain of linux-kernel+bounces-49891-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <13f53836-b1f8-4e2a-a3df-80b60102d396@pengutronix.de>
Date: Fri, 2 Feb 2024 14:19:14 +0100
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Boot-time dumping of ftrace fuctiongraph buffer
Content-Language: en-US
To: Steven Rostedt <rostedt@goodmis.org>
Cc: linux-trace-kernel@vger.kernel.org, Masami Hiramatsu
 <mhiramat@kernel.org>,
 "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
 Pengutronix Kernel Team <kernel@pengutronix.de>
References: <d33e5271-219d-4b8e-be5a-8903219d7fd6@pengutronix.de>
 <20240201204637.2afab2db@gandalf.local.home>
From: Ahmad Fatoum <a.fatoum@pengutronix.de>
In-Reply-To: <20240201204637.2afab2db@gandalf.local.home>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

Hello Steve,

On 02.02.24 02:46, Steven Rostedt wrote:
> On Thu, 1 Feb 2024 13:21:37 +0100
> Ahmad Fatoum <a.fatoum@pengutronix.de> wrote:
>> For this to be maximally useful, I need to configure this not only at boot-time,
>> but also dump the ftrace buffer at boot time. Probe deferral can hinder the kernel from
>> calling init and providing a shell, where I could read /sys/kernel/tracing/trace.
> 
> OK so the driver is built in.

Yes. The modules are easy, because I will have an access to shell by then
on my systems.

>> I found following two mechanisms that looked relevant, but seem not to
>> do exactly what I want:
>>
>>   - tp_printk: seems to be related to trace points only and not usable
>>     for the function graph output
>>
>>   - dump_on_oops: I don't get an Oops if probe deferral times out, but maybe
>>     one could patch the kernel to check a oops_on_probe_deferral or dump_on_probe_deferral
>>     kernel command line parameter in deferred_probe_timeout_work_func()?
>>
>>
>> Is there existing support that I am missing? Any input on whether this
>> would be a welcome feature to have?
> 
> Well you can start function_graph on the kernel command line and event
> filter on a give function
> 
>  ftrace=function_graph function_graph_filter=probe_func

Ye, that's what I am doing, but I was stuck on actually outputting the log
at boot time.

> You can add your own ftrace_dump() on some kind of detected error and put
> that in the kernel command line. For example RCU has:
> 
>   rcupdate.rcu_cpu_stall_ftrace_dump=
> 
> Which will do a ftrace dump when a RCU stall is triggered.

Ah, thanks for the pointer. I take this as meaning that there is no builtin
way to dump on arbitrary function return. I will see if it's possible with
bootconfig before looking into adding a probe deferral specific kernel
command-line parameter.

Thanks,
Ahmad

> 
> -- Steve
> 
> 

-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |