Hi,
A little while back I posted a RFC patch which added gfs2 glock tracing
to blktrace. There was a suggestion that I should look instead at the
generic tracing code and add a new tracer, but I've come up with a
couple of issues along the way.
I think from what I can see that its only possible to run a single
tracer at once, so running blktrace (for example) would preclude me from
also tracing gfs2's glocks at the same time. Also, I can see no
mechanism by which I could ensure the sequencing between the blktrace
and glock traces other than exporting the blktrace sequence number, even
if running multiple tracers at the same time was possible.
Is there a solution to this which I've not spotted, or would something
along the lines of my previous patch[*] be a reasonable solution at this
stage?
Steve.
[*] http://markmail.org/message/ujkd6bre2td27pfb
* Steven Whitehouse <[email protected]> wrote:
> Hi,
>
> A little while back I posted a RFC patch which added gfs2 glock
> tracing to blktrace. There was a suggestion that I should look
> instead at the generic tracing code and add a new tracer, but I've
> come up with a couple of issues along the way.
>
> I think from what I can see that its only possible to run a single
> tracer at once, so running blktrace (for example) would preclude
> me from also tracing gfs2's glocks at the same time. Also, I can
> see no mechanism by which I could ensure the sequencing between
> the blktrace and glock traces other than exporting the blktrace
> sequence number, even if running multiple tracers at the same time
> was possible.
There's a recent addition: the EVENT_TRACE() facility. Would that
suit your purposes?
Ingo
Hi,
On Wed, 2009-04-08 at 15:53 +0200, Ingo Molnar wrote:
> * Steven Whitehouse <[email protected]> wrote:
>
> > Hi,
> >
> > A little while back I posted a RFC patch which added gfs2 glock
> > tracing to blktrace. There was a suggestion that I should look
> > instead at the generic tracing code and add a new tracer, but I've
> > come up with a couple of issues along the way.
> >
> > I think from what I can see that its only possible to run a single
> > tracer at once, so running blktrace (for example) would preclude
> > me from also tracing gfs2's glocks at the same time. Also, I can
> > see no mechanism by which I could ensure the sequencing between
> > the blktrace and glock traces other than exporting the blktrace
> > sequence number, even if running multiple tracers at the same time
> > was possible.
>
> There's a recent addition: the EVENT_TRACE() facility. Would that
> suit your purposes?
>
> Ingo
If I can get both blktrace and glock trace info from it, then yes. I
thought that the blktrace output went either via the original relayfs
path, or via its own tracer so that it wasn't possible to use it and the
event trace facility at the same time, unless I use blktrace via
relayfs. From what I've read I thought that probably the relayfs
interface for blktrace might eventually be removed in favour of the
generic tracing interface, but I'm not 100% sure of that, so perhaps
someone can confirm the plans in that area?
Even if I did use EVENT_TRACE(), wouldn't I still need to put the
blktrace sequence numbers in my glock messages in order to maintain the
correct ordering?
Perhaps I ought to explain a bit more about what I'd like to do... my
plan was to do some automated testing of GFS2's cache control by
watching the glocks and the block I/O and ensuring that the only blocks
which are read/written are those for which the currently held glocks
allow for. Obviously the exact sequencing between the block I/O and
glocks is critical.
I've used the previously posted patches for some testing here, and
already they have shown their worth with one bug already found & fixed
in the current tree that way.
I have some other ideas too, based on measuring latency between glocks
and I/O events which might be used for future performance
analysis/tuning which would also require something similar,
Steve.
On Wed, 8 Apr 2009, Ingo Molnar wrote:
>
> * Steven Whitehouse <[email protected]> wrote:
>
> > Hi,
> >
> > A little while back I posted a RFC patch which added gfs2 glock
> > tracing to blktrace. There was a suggestion that I should look
> > instead at the generic tracing code and add a new tracer, but I've
> > come up with a couple of issues along the way.
> >
> > I think from what I can see that its only possible to run a single
> > tracer at once, so running blktrace (for example) would preclude
> > me from also tracing gfs2's glocks at the same time. Also, I can
> > see no mechanism by which I could ensure the sequencing between
> > the blktrace and glock traces other than exporting the blktrace
> > sequence number, even if running multiple tracers at the same time
> > was possible.
>
> There's a recent addition: the EVENT_TRACE() facility. Would that
> suit your purposes?
>
A while ago I've been playing with the idea of alloing multple tracers.
I've even kept the plugins to keep track of their own buffers, even though
the real design only used one buffer.
After the creation of the EVENT_TRACE facility, I no longer see a true
need for that feature. Note, anything event enabled will be recorded by
all tracers.
-- Steve
P.S. I'm currently traveling, and my Hotel has horrible internet service.
The kind you have to wait 5 seconds to see what you typed :-(
So do not expect quick responses from me.
* Steven Whitehouse <[email protected]> wrote:
> Hi,
>
> On Wed, 2009-04-08 at 15:53 +0200, Ingo Molnar wrote:
> > * Steven Whitehouse <[email protected]> wrote:
> >
> > > Hi,
> > >
> > > A little while back I posted a RFC patch which added gfs2 glock
> > > tracing to blktrace. There was a suggestion that I should look
> > > instead at the generic tracing code and add a new tracer, but I've
> > > come up with a couple of issues along the way.
> > >
> > > I think from what I can see that its only possible to run a single
> > > tracer at once, so running blktrace (for example) would preclude
> > > me from also tracing gfs2's glocks at the same time. Also, I can
> > > see no mechanism by which I could ensure the sequencing between
> > > the blktrace and glock traces other than exporting the blktrace
> > > sequence number, even if running multiple tracers at the same time
> > > was possible.
> >
> > There's a recent addition: the EVENT_TRACE() facility. Would that
> > suit your purposes?
> >
> > Ingo
>
> If I can get both blktrace and glock trace info from it, then yes.
> I thought that the blktrace output went either via the original
> relayfs path, or via its own tracer so that it wasn't possible to
> use it and the event trace facility at the same time, unless I use
> blktrace via relayfs. From what I've read I thought that probably
> the relayfs interface for blktrace might eventually be removed in
> favour of the generic tracing interface, but I'm not 100% sure of
> that, so perhaps someone can confirm the plans in that area?
Tom Zanussi (Cc:-ed) recently converted the blktrace tracepoints to
a tracing framework - and i think much of that could be reused to
add EVENT_TRACE() tracepoints to blktrace too.
Tom, what's your take on this?
In general, this is the direction we want to take: EVENT_TRACE()
tracepoints that can be used in a generic way, in any tracer.
Ingo
CC: Arnaldo Carvalho de Melo <[email protected]>
Ingo Molnar wrote:
> * Steven Whitehouse <[email protected]> wrote:
>
>> Hi,
>>
>> On Wed, 2009-04-08 at 15:53 +0200, Ingo Molnar wrote:
>>> * Steven Whitehouse <[email protected]> wrote:
>>>
>>>> Hi,
>>>>
>>>> A little while back I posted a RFC patch which added gfs2 glock
>>>> tracing to blktrace. There was a suggestion that I should look
>>>> instead at the generic tracing code and add a new tracer, but I've
>>>> come up with a couple of issues along the way.
>>>>
>>>> I think from what I can see that its only possible to run a single
>>>> tracer at once, so running blktrace (for example) would preclude
>>>> me from also tracing gfs2's glocks at the same time. Also, I can
>>>> see no mechanism by which I could ensure the sequencing between
>>>> the blktrace and glock traces other than exporting the blktrace
>>>> sequence number, even if running multiple tracers at the same time
>>>> was possible.
>>> There's a recent addition: the EVENT_TRACE() facility. Would that
>>> suit your purposes?
>>>
>>> Ingo
>> If I can get both blktrace and glock trace info from it, then yes.
>> I thought that the blktrace output went either via the original
>> relayfs path, or via its own tracer so that it wasn't possible to
>> use it and the event trace facility at the same time, unless I use
>> blktrace via relayfs. From what I've read I thought that probably
>> the relayfs interface for blktrace might eventually be removed in
>> favour of the generic tracing interface, but I'm not 100% sure of
>> that, so perhaps someone can confirm the plans in that area?
>
> Tom Zanussi (Cc:-ed) recently converted the blktrace tracepoints to
I guess you mean Arnaldo <[email protected]> ;)
> a tracing framework - and i think much of that could be reused to
> add EVENT_TRACE() tracepoints to blktrace too.
>
> Tom, what's your take on this?
>
> In general, this is the direction we want to take: EVENT_TRACE()
> tracepoints that can be used in a generic way, in any tracer.
>
Actually we've planed to do this, and for other tracers.
--
Zefan
Em Thu, Apr 09, 2009 at 10:11:48AM +0800, Li Zefan escreveu:
> CC: Arnaldo Carvalho de Melo <[email protected]>
>
> Ingo Molnar wrote:
> > * Steven Whitehouse <[email protected]> wrote:
> >
> >> Hi,
> >>
> >> On Wed, 2009-04-08 at 15:53 +0200, Ingo Molnar wrote:
> >>> * Steven Whitehouse <[email protected]> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> A little while back I posted a RFC patch which added gfs2 glock
> >>>> tracing to blktrace. There was a suggestion that I should look
> >>>> instead at the generic tracing code and add a new tracer, but I've
> >>>> come up with a couple of issues along the way.
> >>>>
> >>>> I think from what I can see that its only possible to run a single
> >>>> tracer at once, so running blktrace (for example) would preclude
> >>>> me from also tracing gfs2's glocks at the same time. Also, I can
> >>>> see no mechanism by which I could ensure the sequencing between
> >>>> the blktrace and glock traces other than exporting the blktrace
> >>>> sequence number, even if running multiple tracers at the same time
> >>>> was possible.
> >>> There's a recent addition: the EVENT_TRACE() facility. Would that
> >>> suit your purposes?
> >>>
> >>> Ingo
> >> If I can get both blktrace and glock trace info from it, then yes.
> >> I thought that the blktrace output went either via the original
> >> relayfs path, or via its own tracer so that it wasn't possible to
> >> use it and the event trace facility at the same time, unless I use
> >> blktrace via relayfs. From what I've read I thought that probably
> >> the relayfs interface for blktrace might eventually be removed in
> >> favour of the generic tracing interface, but I'm not 100% sure of
> >> that, so perhaps someone can confirm the plans in that area?
> >
> > Tom Zanussi (Cc:-ed) recently converted the blktrace tracepoints to
>
> I guess you mean Arnaldo <[email protected]> ;)
Nope, Tom:
Message-Id: <1235725261.8512.89.camel@charm-linux>
> > a tracing framework - and i think much of that could be reused to
> > add EVENT_TRACE() tracepoints to blktrace too.
> >
> > Tom, what's your take on this?
> >
> > In general, this is the direction we want to take: EVENT_TRACE()
> > tracepoints that can be used in a generic way, in any tracer.
> >
>
> Actually we've planed to do this, and for other tracers.
Yeah, being able to mix all kinds of tracers is the way to go.
- Arnaldo
>>> Tom Zanussi (Cc:-ed) recently converted the blktrace tracepoints to
>> I guess you mean Arnaldo <[email protected]> ;)
>
> Nope, Tom:
>
> Message-Id: <1235725261.8512.89.camel@charm-linux>
>
Ah, sorry for the mistake, I didn't notice that patch, and I thought
Ingo meant the ftrace-plugin patch.
>>> a tracing framework - and i think much of that could be reused to
>>> add EVENT_TRACE() tracepoints to blktrace too.
>>>
>>> Tom, what's your take on this?
>>>
>>> In general, this is the direction we want to take: EVENT_TRACE()
>>> tracepoints that can be used in a generic way, in any tracer.
>>>
>> Actually we've planed to do this, and for other tracers.
>
> Yeah, being able to mix all kinds of tracers is the way to go.
>
* Li Zefan <[email protected]> wrote:
> >>> Tom Zanussi (Cc:-ed) recently converted the blktrace tracepoints to
> >> I guess you mean Arnaldo <[email protected]> ;)
> >
> > Nope, Tom:
> >
> > Message-Id: <1235725261.8512.89.camel@charm-linux>
> >
>
> Ah, sorry for the mistake, I didn't notice that patch, and I
> thought Ingo meant the ftrace-plugin patch.
yeah - i meant the Zed-trace patches Tom posted some time ago.
I think it would be nice to use the blktrace bits of it too, to
extend built-in user-defined per tracepoint filtering expressions to
blktrace and to make blktrace events more generally available. It
does not impact existing blktrace functionality, it extends on it.
Ingo
Hi,
On Wed, 2009-04-08 at 16:24 +0200, Ingo Molnar wrote:
> * Steven Whitehouse <[email protected]> wrote:
>
> > Hi,
> >
> > On Wed, 2009-04-08 at 15:53 +0200, Ingo Molnar wrote:
> > > * Steven Whitehouse <[email protected]> wrote:
> > >
> > > > Hi,
> > > >
> > > > A little while back I posted a RFC patch which added gfs2 glock
> > > > tracing to blktrace. There was a suggestion that I should look
> > > > instead at the generic tracing code and add a new tracer, but I've
> > > > come up with a couple of issues along the way.
> > > >
> > > > I think from what I can see that its only possible to run a single
> > > > tracer at once, so running blktrace (for example) would preclude
> > > > me from also tracing gfs2's glocks at the same time. Also, I can
> > > > see no mechanism by which I could ensure the sequencing between
> > > > the blktrace and glock traces other than exporting the blktrace
> > > > sequence number, even if running multiple tracers at the same time
> > > > was possible.
> > >
> > > There's a recent addition: the EVENT_TRACE() facility. Would that
> > > suit your purposes?
> > >
> > > Ingo
> >
> > If I can get both blktrace and glock trace info from it, then yes.
> > I thought that the blktrace output went either via the original
> > relayfs path, or via its own tracer so that it wasn't possible to
> > use it and the event trace facility at the same time, unless I use
> > blktrace via relayfs. From what I've read I thought that probably
> > the relayfs interface for blktrace might eventually be removed in
> > favour of the generic tracing interface, but I'm not 100% sure of
> > that, so perhaps someone can confirm the plans in that area?
>
> Tom Zanussi (Cc:-ed) recently converted the blktrace tracepoints to
> a tracing framework - and i think much of that could be reused to
> add EVENT_TRACE() tracepoints to blktrace too.
>
> Tom, what's your take on this?
>
> In general, this is the direction we want to take: EVENT_TRACE()
> tracepoints that can be used in a generic way, in any tracer.
>
Yeah, the EVENT_TRACE() tracepoints can be mixed any way you want by
just enabling the ones you want to see:
# echo nop > /sys/kernel/debug/tracing/current_tracer
# echo 1 > /sys/kernel/debug/tracing/events/sched/sched_process_fork/enable
# echo 1 > /sys/kernel/debug/tracing/events/sched/sched_process_fork/enable
# cat /sys/kernel/debug/tracing/trace
<idle>-0 [001] 9100.344901: sched_signal_send: sig: 29 task Xorg:5899
<idle>-0 [001] 9100.344906: sched_signal_send: sig: 29 task Xorg:5899
bash-6938 [000] 9100.346222: sched_process_fork: parent bash:6938 child bash:7120
<...>-7120 [000] 9100.409853: sched_signal_send: sig: 17 task bash:6938
<idle>-0 [001] 9100.433021: sched_signal_send: sig: 29 task Xorg:5899
But you can also mix those EVENT_TRACE() events with another tracer that
uses the global trace buffer (see the EVENT_TRACE() events mixed in with
the sched_switch tracer output):
# echo sched_switch > /sys/kernel/debug/tracing/current_tracer
<idle>-0 [001] 9405.386800: 0:140:R + [001] 5899:120:S Xorg
<idle>-0 [001] 9405.386803: 0:140:R + [001] 5798:120:S hald-addon-inpu
<idle>-0 [001] 9405.386807: sched_signal_send: sig: 29 task Xorg:5899
<idle>-0 [001] 9405.386811: sched_signal_send: sig: 29 task Xorg:5899
<idle>-0 [001] 9405.386819: 0:140:R ==> [001] 5798:120:R hald-addon-inpu
hald-addon-inpu-5798 [001] 9405.386824: 5798:120:S ==> [001] 5899:120:R Xorg
Xorg-5899 [001] 9405.386957: 5899:120:R + [001] 6301:120:S gnome-screensav
Xorg-5899 [001] 9405.386961: 5899:120:R ==> [001] 6301:120:R gnome-screensav
gnome-screensav-6301 [001] 9405.387049: 6301:120:S ==> [001] 5899:120:R Xorg
Xorg-5899 [001] 9405.387062: 5899:120:R + [001] 6386:120:S wterm
Xorg-5899 [001] 9405.387065: 5899:120:R ==> [001] 6386:120:R wterm
wterm-6386 [001] 9405.387112: 6386:120:S + [000] 6938:120:S bash
wterm-6386 [001] 9405.387142: 6386:120:S ==> [001] 5899:120:R Xorg
Xorg-5899 [001] 9405.387156: 5899:120:R ==> [001] 6386:120:R wterm
wterm-6386 [001] 9405.387197: 6386:120:S ==> [001] 5899:120:R Xorg
Xorg-5899 [001] 9405.387250: 5899:120:S ==> [001] 0:140:R <idle>
<idle>-0 [000] 9405.387633: 0:140:R ==> [000] 6938:120:R bash
bash-6938 [000] 9405.387640: 6938:120:R + [001] 6386:120:S wterm
bash-6938 [000] 9405.387657: 6938:120:S + [001] 6386:120:S wterm
<idle>-0 [001] 9405.387703: 0:140:R ==> [001] 7144:120:R cat
cat-7144 [001] 9405.387909: 7144:120:S ==> [001] 0:140:R <idle>
bash-6938 [000] 9405.388195: sched_process_fork: parent bash:6938 child bash:7144
bash-6938 [000] 9405.388202: 6938:120:R + [001] 7144:120:R cat
There aren't currently any blktrace TRACE_EVENTS(), but if there were,
and you also added a set of glock TRACE_EVENTS(), you'd be able to
enable any or all of those events, along with e.g the sched
TRACE_EVENTS() or even some other tracer, and see all of those outputs
together in the output. I did something similar for blktrace in the
zedtrace tracer - those could probably be converted to TRACE_EVENTS(),
but I haven't really looked into it since then.
I haven't done anything myself with it yet, but if you define
TRACE_EVENTS() for those things, you should also be able to get the
event data as binary too using the per_cpu files. At that point, you'd
need some generic userspace to sort things out using the exported event
formats, which I also did something similar for in zedtrace or you could
modify blktrace/blkparse themselves to understand the new glock events.
As far as sequencing, that's already part of the tracing infrastructure,
so you shouldn't have to worry about it.
If you wanted to write your own special-purpose tracer such as the
current sched_switch tracer that would add something or change the
output of the TRACE_EVENTS() you define, I think it should be possible,
but I don't know that part of the tracing system well enough to say how
to do it; maybe someone else can be more helpful there. I think there
may be a little more infrastructure work needed to do that, since e.g. I
notice that currently the sched_switch tracer uses different but almost
identical code to the sched and wakeup TRACE_EVENTS(), which isn't
surprising since the event tracing code is a lot newer than the
sched_switch code. Converting the sched_switch tracer over to re-using
the TRACE_EVENTS() might be a good exercise and show the way for other
tracers...
Hope that helps,
Tom
> Ingo
Hi,
On Thu, 2009-04-09 at 00:37 -0500, Tom Zanussi wrote:
> Hi,
>
> On Wed, 2009-04-08 at 16:24 +0200, Ingo Molnar wrote:
> > * Steven Whitehouse <[email protected]> wrote:
> >
> > > Hi,
> > >
> > > On Wed, 2009-04-08 at 15:53 +0200, Ingo Molnar wrote:
> > > > * Steven Whitehouse <[email protected]> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > A little while back I posted a RFC patch which added gfs2 glock
> > > > > tracing to blktrace. There was a suggestion that I should look
> > > > > instead at the generic tracing code and add a new tracer, but I've
> > > > > come up with a couple of issues along the way.
> > > > >
> > > > > I think from what I can see that its only possible to run a single
> > > > > tracer at once, so running blktrace (for example) would preclude
> > > > > me from also tracing gfs2's glocks at the same time. Also, I can
> > > > > see no mechanism by which I could ensure the sequencing between
> > > > > the blktrace and glock traces other than exporting the blktrace
> > > > > sequence number, even if running multiple tracers at the same time
> > > > > was possible.
> > > >
> > > > There's a recent addition: the EVENT_TRACE() facility. Would that
> > > > suit your purposes?
> > > >
> > > > Ingo
> > >
> > > If I can get both blktrace and glock trace info from it, then yes.
> > > I thought that the blktrace output went either via the original
> > > relayfs path, or via its own tracer so that it wasn't possible to
> > > use it and the event trace facility at the same time, unless I use
> > > blktrace via relayfs. From what I've read I thought that probably
> > > the relayfs interface for blktrace might eventually be removed in
> > > favour of the generic tracing interface, but I'm not 100% sure of
> > > that, so perhaps someone can confirm the plans in that area?
> >
> > Tom Zanussi (Cc:-ed) recently converted the blktrace tracepoints to
> > a tracing framework - and i think much of that could be reused to
> > add EVENT_TRACE() tracepoints to blktrace too.
> >
> > Tom, what's your take on this?
> >
> > In general, this is the direction we want to take: EVENT_TRACE()
> > tracepoints that can be used in a generic way, in any tracer.
> >
>
> Yeah, the EVENT_TRACE() tracepoints can be mixed any way you want by
> just enabling the ones you want to see:
>
[lots of useful explanation cut here]
Thanks for the low down on this. Thats exactly what I need to know to
answer my question. I will continue to work on trace events support for
GFS2 and when I'm done with that, I may look at other things such as
blktrace if nobody has beaten me to it in the mean time,
Steve.