Return-path: Received: from mx3.mail.elte.hu ([157.181.1.138]:49480 "EHLO mx3.mail.elte.hu" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756616AbYISIca (ORCPT ); Fri, 19 Sep 2008 04:32:30 -0400 Date: Fri, 19 Sep 2008 10:32:08 +0200 From: Ingo Molnar To: Steven Noonan Cc: "Luis R. Rodriguez" , Luis Rodriguez , "ath9k-devel@lists.ath9k.org" , linux-wireless , LKML , Steven Rostedt Subject: Re: [ath9k-devel] ath9k: massive unexplained latency in 2.6.27 (rc5, rc6, probably others) Message-ID: <20080919083208.GA15730@elte.hu> (sfid-20080919_103257_169029_342A1E14) References: <20080918220102.GE7408@tesla> <43e72e890809181508w5232a14ewbf2bf18fe90a92d5@mail.gmail.com> <43e72e890809181610h3a7729d8s4c8484d97b21932e@mail.gmail.com> <20080919030125.GG7408@tesla> <20080919081733.GC19491@elte.hu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii In-Reply-To: Sender: linux-wireless-owner@vger.kernel.org List-ID: * Steven Noonan wrote: > On Fri, Sep 19, 2008 at 1:17 AM, Ingo Molnar wrote: > > in tip/master there's an ftrace_printk() facility. You can just replace > > the verbose printk()s with ftrace_printk() and see the result in > > /debug/tracing/trace. > > > > This sort of tracing has very low overhead and can be used in an easy > > ad-hoc manner with no extra infrastructure. Here's a few quick-start > > links about how to enable the scheduler tracer: > > > > http://redhat.com/~mingo/sched-devel.git/readme-tracer.txt > > http://redhat.com/~mingo/sched-devel.git/howto-trace-latencies.txt > > > > any ftrace_printk() you add in the kernel will show up in that trace. > > > > (If the scheduling events are uninteresting and clutter the output then > > you might want to remove the scheduler tracing entries from kernel/*.c > > by removing the trace_sched_*() calls or use a less noisy tracer.) > > > > to get enough of a trace history you might want to increase the number > > of trace entries in /debug/tracing/trace_entries from 16K to 128K or so. > > > > Thanks for the suggestion, I'll do that. :) one more suggestion: you should first check how large the 'scope' of your trace is - i.e. how long of a time span it can cover, in the workload that you are going to trace. if it's too short - say just a few hundred milliseconds then you need a larger trace buffer or fewer trace entries. another thing: you can auto-stop tracing on the failure condition. That way you dont have to worry about whether you notice the badness. Unfortunately there's no easy callback to just anonymously stop all tracing that goes on. There's ftrace_kill_atomic() but i'm not sure it properly stops the scheduler tracer. What is needed is to set kernel/trace/trace.c's tracing_disabled variable to 0. I've Cc:-ed Steve Rostedt - Steve, why is there no easy ad-hoc way to trigger a stop-tracing event without elaborate changes? Ingo