Lately I've been testing with an allyesconfig. When I ran the function graph
tracer, it immediately crashed the kernel. Thanks to the new frame pointer
test in function graph, it reported directly what the issue was and then
panicked the kernel to prevent any unexpected damage from happening.
It pointed the error to be with jtcp_rcv_established. Which is a jprobe
function added to tcp_rcv_established at bootup when CONFIG_NET_TCPPROBE
is enabled.
Jprobes and the function graph tracer use the same mechanism to trace
the exit of a function. Unfortunately, only one can be done at a time.
The function graph tracer replaces the return address with its own handler,
but so does jprobes. The two are not compatible.
The solution I am proposing with this patch set is to add a call in
ftrace that lets other code in the kernel permanently disable functions from
being traced by the function and function graph tracer. As a probe function
is registered with jprobes, it calls this new function and that entry
will be removed from being traced.
I tested this with this patch series and it does solve the problem.
Some issues though:
1) this only works when DYNAMIC_FTRACE is enabled. We can prevent
function graph tracing with jprobes when DYNAMIC_FTRACE is not
enabled through Kconfig dependencies. Or have the registering of
a jprobe permanently disable function graph tracing.
2) This also prevents the function tracer from being able to trace a
function probe, even though the function tracer is not at issue
with this bug.
Feedback welcomed.
-- Steve
The following patches are in:
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
branch: rfc/trace
Steven Rostedt (3):
tracing: Clean up ftrace.h header and add ftrace_set_notrace() declaration
tracing: Add calls to permanently disable functions from tracing
tracing/kprobes: Disable tracing registered jprobe callback functions
----
include/linux/ftrace.h | 26 +++++++--
kernel/kprobes.c | 4 +
kernel/trace/ftrace.c | 147 +++++++++++++++++++++++++++++++++++++++++-------
3 files changed, 150 insertions(+), 27 deletions(-)
Steven Rostedt wrote:
> Lately I've been testing with an allyesconfig. When I ran the function graph
> tracer, it immediately crashed the kernel. Thanks to the new frame pointer
> test in function graph, it reported directly what the issue was and then
> panicked the kernel to prevent any unexpected damage from happening.
>
> It pointed the error to be with jtcp_rcv_established. Which is a jprobe
> function added to tcp_rcv_established at bootup when CONFIG_NET_TCPPROBE
> is enabled.
>
> Jprobes and the function graph tracer use the same mechanism to trace
> the exit of a function. Unfortunately, only one can be done at a time.
> The function graph tracer replaces the return address with its own handler,
> but so does jprobes. The two are not compatible.
AFAIK, Jprobe doesn't trace the exit of a function. I assume that
jprobe's user handler causes the problem, since the handler never
returns normal way.
Instead of that, it just calls jprobe_return() which causes
int3 to be trapped by kprobe's break handler. And the break handler
fixup regs->ip to back to traced function.
Actually, this will cause a problem with function graph tracer.
The f-g-tracer push the return address into the special stack and replaces
it with fixup function (This is similar (not same) mechanism of kretprobe.)
And then the traced function returns, it returns to the fixup function and
it pops the return address up and back to the real caller.
So, if the f-g-tracer traces jprobe user handler, the pop operation
will be skipped because the the handler never returns.
> The solution I am proposing with this patch set is to add a call in
> ftrace that lets other code in the kernel permanently disable functions from
> being traced by the function and function graph tracer. As a probe function
> is registered with jprobes, it calls this new function and that entry
> will be removed from being traced.
>
> I tested this with this patch series and it does solve the problem.
>
> Some issues though:
>
> 1) this only works when DYNAMIC_FTRACE is enabled. We can prevent
> function graph tracing with jprobes when DYNAMIC_FTRACE is not
> enabled through Kconfig dependencies. Or have the registering of
> a jprobe permanently disable function graph tracing.
IMHO, those *probe handler should be tagged as __kprobes and notrace.
> 2) This also prevents the function tracer from being able to trace a
> function probe, even though the function tracer is not at issue
> with this bug.
I think we can skip those user handlers, because those are irregular
functions and user can control (enable/disable) it.
BTW, in this specific case, I assume that it can use tracepoint
instead of jprobe and move tcp_probe to a part of ftrace :-), isn't it?
(Or, if it is just for a debugging, Systemtap can help it.)
Thank you,
--
Masami Hiramatsu
Software Engineer
Hitachi Computer Products (America), Inc.
Software Solutions Division
e-mail: [email protected]
On Thu, 2009-10-29 at 18:02 -0400, Masami Hiramatsu wrote:
> >
> > Jprobes and the function graph tracer use the same mechanism to trace
> > the exit of a function. Unfortunately, only one can be done at a time.
> > The function graph tracer replaces the return address with its own handler,
> > but so does jprobes. The two are not compatible.
>
> AFAIK, Jprobe doesn't trace the exit of a function. I assume that
> jprobe's user handler causes the problem, since the handler never
> returns normal way.
> Instead of that, it just calls jprobe_return() which causes
> int3 to be trapped by kprobe's break handler. And the break handler
> fixup regs->ip to back to traced function.
Ah, yes, my documenting this is wrong. It's the skipped jprobe that
messed it up.
>
> Actually, this will cause a problem with function graph tracer.
> The f-g-tracer push the return address into the special stack and replaces
> it with fixup function (This is similar (not same) mechanism of kretprobe.)
> And then the traced function returns, it returns to the fixup function and
> it pops the return address up and back to the real caller.
>
> So, if the f-g-tracer traces jprobe user handler, the pop operation
> will be skipped because the the handler never returns.
Exactly!
>
> > The solution I am proposing with this patch set is to add a call in
> > ftrace that lets other code in the kernel permanently disable functions from
> > being traced by the function and function graph tracer. As a probe function
> > is registered with jprobes, it calls this new function and that entry
> > will be removed from being traced.
> >
> > I tested this with this patch series and it does solve the problem.
> >
> > Some issues though:
> >
> > 1) this only works when DYNAMIC_FTRACE is enabled. We can prevent
> > function graph tracing with jprobes when DYNAMIC_FTRACE is not
> > enabled through Kconfig dependencies. Or have the registering of
> > a jprobe permanently disable function graph tracing.
>
> IMHO, those *probe handler should be tagged as __kprobes and notrace.
Yeah, I agree. But how do you guarantee that it does. If one forgets,
than we still have the issue. We can perhaps test to make sure the
function is in the kprobes section. But that does not mean they will not
be notraced. The __kprobes and notrace are no longer in the same set.
>
> > 2) This also prevents the function tracer from being able to trace a
> > function probe, even though the function tracer is not at issue
> > with this bug.
>
> I think we can skip those user handlers, because those are irregular
> functions and user can control (enable/disable) it.
True, but it may be nice to still trace them.
>
> BTW, in this specific case, I assume that it can use tracepoint
> instead of jprobe and move tcp_probe to a part of ftrace :-), isn't it?
> (Or, if it is just for a debugging, Systemtap can help it.)
That's a question for the networking guys.
-- Steve
On Thu, 29 Oct 2009 18:17:33 -0400
Steven Rostedt <[email protected]> wrote:
> On Thu, 2009-10-29 at 18:02 -0400, Masami Hiramatsu wrote:
> > >
> > > Jprobes and the function graph tracer use the same mechanism to trace
> > > the exit of a function. Unfortunately, only one can be done at a time.
> > > The function graph tracer replaces the return address with its own handler,
> > > but so does jprobes. The two are not compatible.
> >
> > AFAIK, Jprobe doesn't trace the exit of a function. I assume that
> > jprobe's user handler causes the problem, since the handler never
> > returns normal way.
> > Instead of that, it just calls jprobe_return() which causes
> > int3 to be trapped by kprobe's break handler. And the break handler
> > fixup regs->ip to back to traced function.
>
> Ah, yes, my documenting this is wrong. It's the skipped jprobe that
> messed it up.
>
> >
> > Actually, this will cause a problem with function graph tracer.
> > The f-g-tracer push the return address into the special stack and replaces
> > it with fixup function (This is similar (not same) mechanism of kretprobe.)
> > And then the traced function returns, it returns to the fixup function and
> > it pops the return address up and back to the real caller.
> >
> > So, if the f-g-tracer traces jprobe user handler, the pop operation
> > will be skipped because the the handler never returns.
>
> Exactly!
>
> >
> > > The solution I am proposing with this patch set is to add a call in
> > > ftrace that lets other code in the kernel permanently disable functions from
> > > being traced by the function and function graph tracer. As a probe function
> > > is registered with jprobes, it calls this new function and that entry
> > > will be removed from being traced.
> > >
> > > I tested this with this patch series and it does solve the problem.
> > >
> > > Some issues though:
> > >
> > > 1) this only works when DYNAMIC_FTRACE is enabled. We can prevent
> > > function graph tracing with jprobes when DYNAMIC_FTRACE is not
> > > enabled through Kconfig dependencies. Or have the registering of
> > > a jprobe permanently disable function graph tracing.
> >
> > IMHO, those *probe handler should be tagged as __kprobes and notrace.
>
> Yeah, I agree. But how do you guarantee that it does. If one forgets,
> than we still have the issue. We can perhaps test to make sure the
> function is in the kprobes section. But that does not mean they will not
> be notraced. The __kprobes and notrace are no longer in the same set.
>
> >
> > > 2) This also prevents the function tracer from being able to trace a
> > > function probe, even though the function tracer is not at issue
> > > with this bug.
> >
> > I think we can skip those user handlers, because those are irregular
> > functions and user can control (enable/disable) it.
>
> True, but it may be nice to still trace them.
>
> >
> > BTW, in this specific case, I assume that it can use tracepoint
> > instead of jprobe and move tcp_probe to a part of ftrace :-), isn't it?
> > (Or, if it is just for a debugging, Systemtap can help it.)
>
> That's a question for the networking guys.
>
tcp_probe is simple tool used for research graphs, there are scripts
and stuff wrapped around it. If you keep the ABI, go ahead and convert
it to ftrace.
--
Steven Rostedt wrote:
> On Thu, 2009-10-29 at 18:02 -0400, Masami Hiramatsu wrote:\
>>> The solution I am proposing with this patch set is to add a call in
>>> ftrace that lets other code in the kernel permanently disable functions from
>>> being traced by the function and function graph tracer. As a probe function
>>> is registered with jprobes, it calls this new function and that entry
>>> will be removed from being traced.
>>>
>>> I tested this with this patch series and it does solve the problem.
>>>
>>> Some issues though:
>>>
>>> 1) this only works when DYNAMIC_FTRACE is enabled. We can prevent
>>> function graph tracing with jprobes when DYNAMIC_FTRACE is not
>>> enabled through Kconfig dependencies. Or have the registering of
>>> a jprobe permanently disable function graph tracing.
>>
>> IMHO, those *probe handler should be tagged as __kprobes and notrace.
>
> Yeah, I agree. But how do you guarantee that it does. If one forgets,
> than we still have the issue. We can perhaps test to make sure the
> function is in the kprobes section. But that does not mean they will not
> be notraced. The __kprobes and notrace are no longer in the same set.
>
>>
>>> 2) This also prevents the function tracer from being able to trace a
>>> function probe, even though the function tracer is not at issue
>>> with this bug.
>>
>> I think we can skip those user handlers, because those are irregular
>> functions and user can control (enable/disable) it.
>
> True, but it may be nice to still trace them.
Hm, in that case, I think we can change jprobe_return() to call
f-g-tracer's return handler if needed as below;
---
static inline jprobe_return(void)
{
implicit_function_return(); /* This executes f-g-tracer prologue */
__jprobe_return(); /* This should be notraced */
}
---
Thank you,
--
Masami Hiramatsu
Software Engineer
Hitachi Computer Products (America), Inc.
Software Solutions Division
e-mail: [email protected]
On Thu, 2009-10-29 at 19:22 -0400, Masami Hiramatsu wrote:
> Steven Rostedt wrote:
> >> I think we can skip those user handlers, because those are irregular
> >> functions and user can control (enable/disable) it.
> >
> > True, but it may be nice to still trace them.
>
> Hm, in that case, I think we can change jprobe_return() to call
> f-g-tracer's return handler if needed as below;
> ---
> static inline jprobe_return(void)
> {
> implicit_function_return(); /* This executes f-g-tracer prologue */
> __jprobe_return(); /* This should be notraced */
> }
Hmm, That looks like it can be quite a hack. We don't know at that
moment if the handler has been traced or not. We can't always do the
function graph logic.
-- Steve
Steven Rostedt wrote:
> On Thu, 2009-10-29 at 19:22 -0400, Masami Hiramatsu wrote:
>> Steven Rostedt wrote:
>
>>>> I think we can skip those user handlers, because those are irregular
>>>> functions and user can control (enable/disable) it.
>>>
>>> True, but it may be nice to still trace them.
>>
>> Hm, in that case, I think we can change jprobe_return() to call
>> f-g-tracer's return handler if needed as below;
>> ---
>> static inline jprobe_return(void)
>> {
>> implicit_function_return(); /* This executes f-g-tracer prologue */
>> __jprobe_return(); /* This should be notraced */
>> }
>
> Hmm, That looks like it can be quite a hack. We don't know at that
> moment if the handler has been traced or not. We can't always do the
> function graph logic.
Ah, right. Perhaps, jprobe might be able to save the address of top
of the stack and check if it has been modified to return_to_handler
at jprobe_return().
But anyway, I think your patch can fix that too. :-)
Thank you,
--
Masami Hiramatsu
Software Engineer
Hitachi Computer Products (America), Inc.
Software Solutions Division
e-mail: [email protected]
Steven Rostedt <[email protected]> writes:
> [...] Jprobes and the function graph tracer use the same mechanism
> to trace the exit of a function. Unfortunately, only one can be done
> at a time. The function graph tracer replaces the return address
> with its own handler, but so does jprobes. The two are not
> compatible. [...]
What about kretprobes? It too uses the same mechanism.
- FChE
Frank Ch. Eigler wrote:
> Steven Rostedt<[email protected]> writes:
>
>> [...] Jprobes and the function graph tracer use the same mechanism
>> to trace the exit of a function. Unfortunately, only one can be done
>> at a time. The function graph tracer replaces the return address
>> with its own handler, but so does jprobes. The two are not
>> compatible. [...]
>
> What about kretprobes? It too uses the same mechanism.
Exactly, kretprobe uses similar mechanism with func-graph tracer.
Fortunately, it doesn't cause any problem, because kretprobe doesn't
skip any function-return, and func-graph tracer(mcount) always intrudes
a function after kretprobe.
---
call func
<-- kretprobe prehandler changes return address
func()
<-- func-graph prehandler changes return address
...
return
--> func-graph trampoline sets ip to "kretprobe trampoline"
--> kretprobe trampoline sets ip to "real return address"
(return address)
---
So that kretprobe handler doesn't conflict with function graph tracer.
Thank you,
--
Masami Hiramatsu
Software Engineer
Hitachi Computer Products (America), Inc.
Software Solutions Division
e-mail: [email protected]
On Thu, Oct 29, 2009 at 06:02:20PM -0400, Masami Hiramatsu wrote:
> Steven Rostedt wrote:
> > Lately I've been testing with an allyesconfig. When I ran the function graph
> > tracer, it immediately crashed the kernel. Thanks to the new frame pointer
> > test in function graph, it reported directly what the issue was and then
> > panicked the kernel to prevent any unexpected damage from happening.
> >
> > It pointed the error to be with jtcp_rcv_established. Which is a jprobe
> > function added to tcp_rcv_established at bootup when CONFIG_NET_TCPPROBE
> > is enabled.
> >
> > Jprobes and the function graph tracer use the same mechanism to trace
> > the exit of a function. Unfortunately, only one can be done at a time.
> > The function graph tracer replaces the return address with its own handler,
> > but so does jprobes. The two are not compatible.
>
> AFAIK, Jprobe doesn't trace the exit of a function. I assume that
> jprobe's user handler causes the problem, since the handler never
> returns normal way.
> Instead of that, it just calls jprobe_return() which causes
> int3 to be trapped by kprobe's break handler. And the break handler
> fixup regs->ip to back to traced function.
>
> Actually, this will cause a problem with function graph tracer.
> The f-g-tracer push the return address into the special stack and replaces
> it with fixup function (This is similar (not same) mechanism of kretprobe.)
> And then the traced function returns, it returns to the fixup function and
> it pops the return address up and back to the real caller.
>
> So, if the f-g-tracer traces jprobe user handler, the pop operation
> will be skipped because the the handler never returns.
I'm not sure I've well understood how is performed the call to the jprobe
handler.
But if I understand well we have:
func() {
int3() {
jprobe_handler() {
(-)
set ip after iret to user_handler()
}
}
user_handler() {
jprobe_return() {
(+)
int3() {
set ip after iret to func+...()
}
|
|
|
<--------------
(execute the rest of func())
}
If we replace (-) with pause_graph_tracing() and (+) with
unpause_graph_tracing(), this should do the trick...I hope.
Frederic Weisbecker wrote:
> On Thu, Oct 29, 2009 at 06:02:20PM -0400, Masami Hiramatsu wrote:
>> Steven Rostedt wrote:
>>> Lately I've been testing with an allyesconfig. When I ran the function graph
>>> tracer, it immediately crashed the kernel. Thanks to the new frame pointer
>>> test in function graph, it reported directly what the issue was and then
>>> panicked the kernel to prevent any unexpected damage from happening.
>>>
>>> It pointed the error to be with jtcp_rcv_established. Which is a jprobe
>>> function added to tcp_rcv_established at bootup when CONFIG_NET_TCPPROBE
>>> is enabled.
>>>
>>> Jprobes and the function graph tracer use the same mechanism to trace
>>> the exit of a function. Unfortunately, only one can be done at a time.
>>> The function graph tracer replaces the return address with its own handler,
>>> but so does jprobes. The two are not compatible.
>>
>> AFAIK, Jprobe doesn't trace the exit of a function. I assume that
>> jprobe's user handler causes the problem, since the handler never
>> returns normal way.
>> Instead of that, it just calls jprobe_return() which causes
>> int3 to be trapped by kprobe's break handler. And the break handler
>> fixup regs->ip to back to traced function.
>>
>> Actually, this will cause a problem with function graph tracer.
>> The f-g-tracer push the return address into the special stack and replaces
>> it with fixup function (This is similar (not same) mechanism of kretprobe.)
>> And then the traced function returns, it returns to the fixup function and
>> it pops the return address up and back to the real caller.
>>
>> So, if the f-g-tracer traces jprobe user handler, the pop operation
>> will be skipped because the the handler never returns.
>
>
> I'm not sure I've well understood how is performed the call to the jprobe
> handler.
> But if I understand well we have:
>
> func() {
> int3() {
> jprobe_handler() {
> (-)
> set ip after iret to user_handler()
> }
> }
> user_handler() {
> jprobe_return() {
> (+)
> int3() {
> set ip after iret to func+...()
> }
> |
> |
> |
> <--------------
> (execute the rest of func())
> }
>
> If we replace (-) with pause_graph_tracing() and (+) with
> unpause_graph_tracing(), this should do the trick...I hope.
I'm not so sure about pause_graph_tracing(), however, it seems that
int3() and jprobe_handler() already pushed on the stack of the
func graph tracer at (-). If it's true, where are those entries
popped up?
Thank you,
--
Masami Hiramatsu
Software Engineer
Hitachi Computer Products (America), Inc.
Software Solutions Division
e-mail: [email protected]
On Mon, Nov 02, 2009 at 10:02:23AM -0500, Masami Hiramatsu wrote:
> Frederic Weisbecker wrote:
>> I'm not sure I've well understood how is performed the call to the jprobe
>> handler.
>> But if I understand well we have:
>>
>> func() {
>> int3() {
>> jprobe_handler() {
>> (-)
>> set ip after iret to user_handler()
>> }
>> }
>> user_handler() {
>> jprobe_return() {
>> (+)
>> int3() {
>> set ip after iret to func+...()
>> }
>> |
>> |
>> |
>> <--------------
>> (execute the rest of func())
>> }
>>
>> If we replace (-) with pause_graph_tracing() and (+) with
>> unpause_graph_tracing(), this should do the trick...I hope.
>
> I'm not so sure about pause_graph_tracing(), however, it seems that
> int3() and jprobe_handler() already pushed on the stack of the
> func graph tracer at (-). If it's true, where are those entries
> popped up?
>
pause_graph_tracing() will disable the tracing for the current task
but it won't disable the address pop from stack.
If the above jprobe scheme is right, the scenario will be:
func() {
/* push func ret */
int3() {
/* push do_trap ret */
jprobe_handler() {
/* push jprobe_handler ret */
pause_graph_tracing();
set ip after iret to user_handler()
} /* pop jprobe_handler ret */
} /* pop do_trap ret */
user_handler() {
jprobe_return() {
unpause_graph_tracing()
int3() {
/* push do_trap ret */
set ip after iret to func+...()
} /* pop do_trap ret */
|
|
|
<--------------
(execute the rest of func())
} /* pop func ret */
Hmm?
Frederic Weisbecker wrote:
> On Mon, Nov 02, 2009 at 10:02:23AM -0500, Masami Hiramatsu wrote:
>> Frederic Weisbecker wrote:
>>> I'm not sure I've well understood how is performed the call to the jprobe
>>> handler.
>>> But if I understand well we have:
>>>
>>> func() {
>>> int3() {
>>> jprobe_handler() {
>>> (-)
>>> set ip after iret to user_handler()
>>> }
>>> }
>>> user_handler() {
>>> jprobe_return() {
>>> (+)
>>> int3() {
>>> set ip after iret to func+...()
>>> }
>>> |
>>> |
>>> |
>>> <--------------
>>> (execute the rest of func())
>>> }
>>>
>>> If we replace (-) with pause_graph_tracing() and (+) with
>>> unpause_graph_tracing(), this should do the trick...I hope.
>>
>> I'm not so sure about pause_graph_tracing(), however, it seems that
>> int3() and jprobe_handler() already pushed on the stack of the
>> func graph tracer at (-). If it's true, where are those entries
>> popped up?
>>
>
>
> pause_graph_tracing() will disable the tracing for the current task
> but it won't disable the address pop from stack.
>
> If the above jprobe scheme is right, the scenario will be:
>
> func() {
> /* push func ret */
> int3() {
> /* push do_trap ret */
> jprobe_handler() {
> /* push jprobe_handler ret */
> pause_graph_tracing();
> set ip after iret to user_handler()
> } /* pop jprobe_handler ret */
> } /* pop do_trap ret */
> user_handler() {
> jprobe_return() {
> unpause_graph_tracing()
> int3() {
> /* push do_trap ret */
> set ip after iret to func+...()
> } /* pop do_trap ret */
> |
> |
> |
> <--------------
> (execute the rest of func())
> } /* pop func ret */
>
>
> Hmm?
Oh, I see. That should work. :-)
Thank you,
--
Masami Hiramatsu
Software Engineer
Hitachi Computer Products (America), Inc.
Software Solutions Division
e-mail: [email protected]