Date: Wed, 9 Aug 2017 05:58:04 -0700
From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
To: Pratyush Anand <panand@redhat.com>
Cc: =?utf-8?B?6rmA64+Z7ZiE?= <austinkernel.kim@gmail.com>,
        Daniel Lezcano <daniel.lezcano@linaro.org>, john.stultz@linaro.org,
        Steven Rostedt <rostedt@goodmis.org>, linux-kernel@vger.kernel.org
Subject: Re: RCU stall when using function_graph
Reply-To: paulmck@linux.vnet.ibm.com
References: <20170802124239.GD1919@mai>
 <20170802090744.6922e9e9@gandalf.local.home>
 <11d179df-d8a9-5d3e-3bc4-080df464e85d@linaro.org>
 <20170803124421.GP3730@linux.vnet.ibm.com>
 <20170803143801.GE1919@mai>
 <CAOoBcBXo-=VYy2+TYEp=8+WSkOpDBr1x6uY=-r_GnTFKctXndQ@mail.gmail.com>
 <CAOoBcBVKpQkAVXji5qQu8r8GErqxpy9Ae9N97NhGpOQPgXudZg@mail.gmail.com>
 <CAOoBcBU00VRXmrNNEOjJHgXf9BimxKYOorJC0d3766mNdda=Bg@mail.gmail.com>
 <20170806170220.GQ3730@linux.vnet.ibm.com>
 <db4dc3c5-8a3d-9752-802e-ab509201e251@redhat.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <db4dc3c5-8a3d-9752-802e-ab509201e251@redhat.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
Message-Id: <20170809125804.GT3730@linux.vnet.ibm.com>
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 2020
Lines: 56

On Wed, Aug 09, 2017 at 02:43:49PM +0530, Pratyush Anand wrote:
> 
> 
> On Sunday 06 August 2017 10:32 PM, Paul E. McKenney wrote:
> >On Sat, Aug 05, 2017 at 02:24:21PM +0900, 김동현 wrote:
> >>Dear All
> >>
> >>As for me, after configuring function_graph as below, crash disappears.
> >>"echo 0 > d/tracing/tracing_on"
> >>"sleep 1"
> >>
> >>"echo function_graph > d/tracing/current_tracer"
> >>"sleep 1"
> >>
> >>"echo smp_call_function_single > d/tracing/set_ftrace_filter"
> 
> It will limit trace output to only for the filtered function
> (smp_call_function_single).
> 
> >>adb shell "sleep 1"
> >>
> >>"echo 1 > d/tracing/tracing_on"
> >>adb shell "sleep 1"
> >>
> >>Right after function_graph is enabled, too many logs are traced upon IRQ
> >>transaction which many times eventually causes stall.
> >
> >That would do it!
> >
> >Hmmm...
> >
> >Steven, would it be helpful if RCU were to inform tracing (say) halfway
> >through the RCU CPU stall interval, allowing the tracer to do something
> >like cond_resched_rcu_qs()?  I can imagine all sorts of reasons why this
> >wouldn't work, for example, if all the tracing was with irqs disabled
> >or some such, but figured I should ask.
> >
> >Does Guillermo's approach work for others?
> 
> Limited output with a couple of filtered function will definitely
> not cause RCU schedule stall. But the question is whether we should
> expect a full function graph trace working on every platform or not
> (specially the one which generates high interrupts)?

It might well be that the user must disable RCU CPU stall warnings via
the rcu_cpu_stall_suppress sysfs entry (or increase their timeout via the
rcu_cpu_stall_timeout sysfs entry) before doing something that greatly
increases overhead.  Like enabling large quantities of tracing.  ;-)

It -might- be possible to do this automatically, but reliable
automation would require that tracing understand how often each
function was called, which sounds to me to be a bit of a stretch.

Thoughts?

							Thanx, Paul