Em Thu, Jan 22, 2009 at 09:36:53AM +1100, Greg Banks escreveu:
> Chuck Lever wrote:
> >
> >
> > I think we need to visit this issue on a case-by-case basis.
> > Sometimes dprintk is appropriate. Sometimes printk(KERN_ERR).
> > Sometimes a performance metric.
> Well said.
>
> > Trond has always maintained that dprintk() is best for developers, but
> > probably inappropriate for field debugging,
> It's not a perfect tool but it beats nothing at all.
> > and I think that may also
> > apply to trace points.
> It depends on whether distros can be convinced to enable it by default,
> and install by default any necessary userspace infrastructure. The
> most important thing for field debugging is Just Knowing that you have
> all the bits necessary to perform useful debugging without having to
> find some RPM that matches the kernel that the machine is actually
> running now, and not the one that was present when the machine was
> installed.
Exactly, that is why an ftrace plugin, that only when selected using
echo "nfs" > /debug/tracing/current_tracer will activate the tracepoints
and provide output via /debug/tracing/trace or /deb/tracing/trace_pipe,
possibly combined with other ftrace plugins such as the stacktrace,
blktrace, etc.
I.e. no need at all for any matching userspace tool, near zero impact
when not activated, useful, if done right, for both developers and for
admins.
Again, an example can be found in the blktrace ftrace plugin[1], that
instead of adding a requirement will eventually drop an existing, well
established one (blktrace(8)).
- Arnaldo
[1] http://lkml.org/lkml/2009/1/20/190
On Wed, 2009-01-21 at 20:47 -0200, Arnaldo Carvalho de Melo wrote:
> Em Thu, Jan 22, 2009 at 09:36:53AM +1100, Greg Banks escreveu:
> > Chuck Lever wrote:
> > >
> > >
> > > I think we need to visit this issue on a case-by-case basis.
> > > Sometimes dprintk is appropriate. Sometimes printk(KERN_ERR).
> > > Sometimes a performance metric.
> > Well said.
> >
> > > Trond has always maintained that dprintk() is best for developers, but
> > > probably inappropriate for field debugging,
> > It's not a perfect tool but it beats nothing at all.
> > > and I think that may also
> > > apply to trace points.
> > It depends on whether distros can be convinced to enable it by default,
> > and install by default any necessary userspace infrastructure. The
> > most important thing for field debugging is Just Knowing that you have
> > all the bits necessary to perform useful debugging without having to
> > find some RPM that matches the kernel that the machine is actually
> > running now, and not the one that was present when the machine was
> > installed.
>
> Exactly, that is why an ftrace plugin, that only when selected using
> echo "nfs" > /debug/tracing/current_tracer will activate the tracepoints
> and provide output via /debug/tracing/trace or /deb/tracing/trace_pipe,
> possibly combined with other ftrace plugins such as the stacktrace,
> blktrace, etc.
>
> I.e. no need at all for any matching userspace tool, near zero impact
> when not activated, useful, if done right, for both developers and for
> admins.
>
> Again, an example can be found in the blktrace ftrace plugin[1], that
> instead of adding a requirement will eventually drop an existing, well
> established one (blktrace(8)).
I must be missing something. Exactly what functionality does this then
give us that we don't have already with the existing RPC/NFS dprintk()
scheme?
Trond
Em Wed, Jan 21, 2009 at 05:57:46PM -0500, Trond Myklebust escreveu:
> On Wed, 2009-01-21 at 20:47 -0200, Arnaldo Carvalho de Melo wrote:
> > Em Thu, Jan 22, 2009 at 09:36:53AM +1100, Greg Banks escreveu:
> > > Chuck Lever wrote:
> > > >
> > > >
> > > > I think we need to visit this issue on a case-by-case basis.
> > > > Sometimes dprintk is appropriate. Sometimes printk(KERN_ERR).
> > > > Sometimes a performance metric.
> > > Well said.
> > >
> > > > Trond has always maintained that dprintk() is best for developers, but
> > > > probably inappropriate for field debugging,
> > > It's not a perfect tool but it beats nothing at all.
> > > > and I think that may also
> > > > apply to trace points.
> > > It depends on whether distros can be convinced to enable it by default,
> > > and install by default any necessary userspace infrastructure. The
> > > most important thing for field debugging is Just Knowing that you have
> > > all the bits necessary to perform useful debugging without having to
> > > find some RPM that matches the kernel that the machine is actually
> > > running now, and not the one that was present when the machine was
> > > installed.
> >
> > Exactly, that is why an ftrace plugin, that only when selected using
> > echo "nfs" > /debug/tracing/current_tracer will activate the tracepoints
> > and provide output via /debug/tracing/trace or /deb/tracing/trace_pipe,
> > possibly combined with other ftrace plugins such as the stacktrace,
> > blktrace, etc.
> >
> > I.e. no need at all for any matching userspace tool, near zero impact
> > when not activated, useful, if done right, for both developers and for
> > admins.
> >
> > Again, an example can be found in the blktrace ftrace plugin[1], that
> > instead of adding a requirement will eventually drop an existing, well
> > established one (blktrace(8)).
>
> I must be missing something. Exactly what functionality does this then
> give us that we don't have already with the existing RPC/NFS dprintk()
> scheme?
Filtering by CPU, the possibity of printing stack traces when the
tracepoints are hit, combination with other tracers to try to mix events
and correlate problems, a request may be taking too long because some
problems are happening on a lower layer subsystem that has, in turn,
tracepoints exposed thru another ftrace plugin.
But then I haven't looked too closely to the places that are being
proposed for conversion to tracepoints in NFS land, will do.
- Arnaldo