2010-07-27 22:47:19

by Marcin Slusarz

[permalink] [raw]
Subject: [PATCH] tracing: wake up tasks reading trace_pipe on write to trace_marker

Currently we rely on other code periodically waking up trace reader.
If there aren't any other data than markers, reader will never be woken up.
Fix it.

Signed-off-by: Marcin Slusarz <[email protected]>
Cc: Steven Rostedt <[email protected]>
Cc: Frederic Weisbecker <[email protected]>
Cc: Ingo Molnar <[email protected]>
---
kernel/trace/trace.c | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 086d363..02e04c8 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -1520,6 +1520,7 @@ int trace_array_vprintk(struct trace_array *tr,
if (!filter_check_discard(call, entry, buffer, event)) {
ring_buffer_unlock_commit(buffer, event);
ftrace_trace_stack(buffer, irq_flags, 6, pc);
+ trace_wake_up();
}

out_unlock:
--
1.7.1.1


2010-08-06 20:39:02

by Steven Rostedt

[permalink] [raw]
Subject: Re: [PATCH] tracing: wake up tasks reading trace_pipe on write to trace_marker

On Wed, 2010-07-28 at 00:44 +0200, Marcin Slusarz wrote:
> Currently we rely on other code periodically waking up trace reader.
> If there aren't any other data than markers, reader will never be woken up.
> Fix it.
>
> Signed-off-by: Marcin Slusarz <[email protected]>
> Cc: Steven Rostedt <[email protected]>
> Cc: Frederic Weisbecker <[email protected]>
> Cc: Ingo Molnar <[email protected]>
> ---
> kernel/trace/trace.c | 1 +
> 1 files changed, 1 insertions(+), 0 deletions(-)
>
> diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
> index 086d363..02e04c8 100644
> --- a/kernel/trace/trace.c
> +++ b/kernel/trace/trace.c
> @@ -1520,6 +1520,7 @@ int trace_array_vprintk(struct trace_array *tr,
> if (!filter_check_discard(call, entry, buffer, event)) {
> ring_buffer_unlock_commit(buffer, event);
> ftrace_trace_stack(buffer, irq_flags, 6, pc);
> + trace_wake_up();
> }
>

This can't work. trace_printk() and friends must be able to be used
anywhere. This can cause race conditions with the rq locks in the
scheduler.

But you do bring up a good idea. That is, perhaps we should have a way
to attach to known safe tracepoints that we can hook to to check if a
wake up should happen or not.

-- Steve

2010-08-06 20:56:03

by Frederic Weisbecker

[permalink] [raw]
Subject: Re: [PATCH] tracing: wake up tasks reading trace_pipe on write to trace_marker

On Fri, Aug 06, 2010 at 04:38:56PM -0400, Steven Rostedt wrote:
> On Wed, 2010-07-28 at 00:44 +0200, Marcin Slusarz wrote:
> > Currently we rely on other code periodically waking up trace reader.
> > If there aren't any other data than markers, reader will never be woken up.
> > Fix it.
> >
> > Signed-off-by: Marcin Slusarz <[email protected]>
> > Cc: Steven Rostedt <[email protected]>
> > Cc: Frederic Weisbecker <[email protected]>
> > Cc: Ingo Molnar <[email protected]>
> > ---
> > kernel/trace/trace.c | 1 +
> > 1 files changed, 1 insertions(+), 0 deletions(-)
> >
> > diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
> > index 086d363..02e04c8 100644
> > --- a/kernel/trace/trace.c
> > +++ b/kernel/trace/trace.c
> > @@ -1520,6 +1520,7 @@ int trace_array_vprintk(struct trace_array *tr,
> > if (!filter_check_discard(call, entry, buffer, event)) {
> > ring_buffer_unlock_commit(buffer, event);
> > ftrace_trace_stack(buffer, irq_flags, 6, pc);
> > + trace_wake_up();
> > }
> >
>
> This can't work. trace_printk() and friends must be able to be used
> anywhere. This can cause race conditions with the rq locks in the
> scheduler.
>
> But you do bring up a good idea. That is, perhaps we should have a way
> to attach to known safe tracepoints that we can hook to to check if a
> wake up should happen or not.


This could be a simple macro that takes the name of the trace event:


DEFINE_EVENT(event_tpl, event_name, ...);


TRACE_EVENT_NO_WAKE(event_name);

I think trace events should be wakeable by default as it looks safe for
most of them. But probably we don't want that per event class.

In the unsafe list, I only have some sched and lock events in
mind, but I bet there are some others.

2010-08-06 21:29:46

by Steven Rostedt

[permalink] [raw]
Subject: Re: [PATCH] tracing: wake up tasks reading trace_pipe on write to trace_marker

On Fri, 2010-08-06 at 22:55 +0200, Frederic Weisbecker wrote:

> > This can't work. trace_printk() and friends must be able to be used
> > anywhere. This can cause race conditions with the rq locks in the
> > scheduler.
> >
> > But you do bring up a good idea. That is, perhaps we should have a way
> > to attach to known safe tracepoints that we can hook to to check if a
> > wake up should happen or not.
>
>
> This could be a simple macro that takes the name of the trace event:
>
>
> DEFINE_EVENT(event_tpl, event_name, ...);
>
>
> TRACE_EVENT_NO_WAKE(event_name);
>

Yeah, that may be worth doing for 2.6.37. Might as well also add a
trace_printk_nowake() too, when you know you are in dangerous locations
like the scheduler or NMI.


> I think trace events should be wakeable by default as it looks safe for
> most of them. But probably we don't want that per event class.
>
> In the unsafe list, I only have some sched and lock events in
> mind, but I bet there are some others.

Yep, will put that on my todo list.

Thanks,

-- Steve

2010-08-06 21:50:40

by Frederic Weisbecker

[permalink] [raw]
Subject: Re: [PATCH] tracing: wake up tasks reading trace_pipe on write to trace_marker

On Fri, Aug 06, 2010 at 05:29:35PM -0400, Steven Rostedt wrote:
> On Fri, 2010-08-06 at 22:55 +0200, Frederic Weisbecker wrote:
>
> > > This can't work. trace_printk() and friends must be able to be used
> > > anywhere. This can cause race conditions with the rq locks in the
> > > scheduler.
> > >
> > > But you do bring up a good idea. That is, perhaps we should have a way
> > > to attach to known safe tracepoints that we can hook to to check if a
> > > wake up should happen or not.
> >
> >
> > This could be a simple macro that takes the name of the trace event:
> >
> >
> > DEFINE_EVENT(event_tpl, event_name, ...);
> >
> >
> > TRACE_EVENT_NO_WAKE(event_name);
> >
>
> Yeah, that may be worth doing for 2.6.37. Might as well also add a
> trace_printk_nowake() too, when you know you are in dangerous locations
> like the scheduler or NMI.



Yeah.



>
> > I think trace events should be wakeable by default as it looks safe for
> > most of them. But probably we don't want that per event class.
> >
> > In the unsafe list, I only have some sched and lock events in
> > mind, but I bet there are some others.
>
> Yep, will put that on my todo list.
>
> Thanks,


Cool. This is going to be useful in perf as well. The "nmi" argument in
perf_swevent_add tells wether we can wake up or not. If not we do a
kind of delayed wake up using a self IPI.

Currently we always consider we can't wake up when a trace event triggers.
If we know we can wake up, this is going to be less costly.