>Can anyone answer Ingo's questions about JACK & the best way to profile
>it? This is a bit over my head.
>
>Lee
>
>
>-------- Forwarded Message --------
>Subject: Re: [patch] Real-Time Preemption, -RT-2.6.9-mm1-V0.4
>Date: Thu, 28 Oct 2004 21:16:05 +0200
>* Rui Nuno Capela <[email protected]> wrote:
>
>> OK. Here are my early consolidated results. Feel free to comment.
>>
>> 2.6.9 RT-U3 RT-V0.4.3
>> --------- --------- ---------
>> XRUN Rate . . . . . . . . . . . 424 8 4 /hour
>> Delay Rate (>spare time) . . . 496 0 0 /hour
>> Delay Rate (>1000 usecs) . . . 940 8 4 /hour
>> Maximum Delay . . . . . . . . . 6904 921 721 usecs
>> Maximum Process Cycle . . . . . 1449 1469 1590 usecs
>> Average DSP CPU Load . . . . . 38 39 40 %
>> Average Context-Switch Rate . . 7480 8929 9726 /sec
>
>looks pretty good, doesnt it?
yes and no. its troubling that we're using an extra 100usecs of time
for the max process cycle, if the statistics make that number
meaningful. and why a 30% increase in the context switch rate? is that
an artifact or a real behavioural change? the xrun rate is not bad,
although i'd need to know the period size. 4 clicks per hour would
actually be unacceptable to most professionals, but this may have been
with significant loading outside of JACK - i don't know.
>how is the 'maximum delay' calculated? Could you put in a tracing hook
>into jackd whenever such a ~720 usecs maximum is hit? I'd _love_ to see
>how such a latency path looks like, it seems a bit long.
"maximum delay" is computed like this:
a) we know that jackd should be woken at regular, known
intervals based on the interrupt of the audio interface.
b) jackd is (often) blocked in poll(2), waiting for the
audio interface to become readable+writable.
c) on return from poll, we check the time we are returning
and compare it to the last wakeup. the difference should
be almost exactly the interrupt interval.
d) any amount larger than this is a delay caused by
kernel scheduling. we track the largest such delay.
>also, i looked at the sound/pci/ali5451/ali5451.c driver code and it has
>one weird piece of code on line 988:
>
> udelay(100);
yeah, some ALSA drivers have broken stuff like this in them. its a
definite problem. this is why we know that some audio interfaces are
not able to provide acceptable performance for JACK - that is OK, the
same is true under Win/Mac where some h/w just can't be used for low
latency work.
>plus i've also got questions about how Jackd interfaces with ALSA: does
>it use SIGIO, or some direct driver ioctl? If SIGIO is used then how is
>it done precisely - is an 'RT' queued signal used or ordinary SIGIO?
>Also, how is the 'channel' information established upon getting a SIGIO,
>is it in the siginfo structure?
we absolutely do not use SIGIO - i am one of the most vocal critics of
this interface even be present. POSIX says that signal handling
context is essentially crippled, so there is very little you can do
from there. in addition, measurements of signal delivery time on Linux
suggest that it is really quite slow. i considered using signals to
wake jack clients, but it was dramatically slower than using a FIFO
plus poll(2).
we block in poll(2), waiting for the interrupt from the audio
interface to mark the relevant jackd thread as runnable.
i don't know what you mean by "channel" information in this context.
--p
On Thu, 2004-10-28 at 20:57 -0400, Paul Davis wrote:
> >> XRUN Rate . . . . . . . . . . . 424 8 4 /hour
> >> Delay Rate (>spare time) . . . 496 0 0 /hour
> >> Delay Rate (>1000 usecs) . . . 940 8 4 /hour
> >> Maximum Delay . . . . . . . . . 6904 921 721 usecs
> >> Maximum Process Cycle . . . . . 1449 1469 1590 usecs
> >> Average DSP CPU Load . . . . . 38 39 40 %
> >> Average Context-Switch Rate . . 7480 8929 9726 /sec
> >
> >looks pretty good, doesnt it?
>
> yes and no. its troubling that we're using an extra 100usecs of time
> for the max process cycle, if the statistics make that number
> meaningful. and why a 30% increase in the context switch rate? is that
> an artifact or a real behavioural change? the xrun rate is not bad,
> although i'd need to know the period size. 4 clicks per hour would
> actually be unacceptable to most professionals, but this may have been
> with significant loading outside of JACK - i don't know.
I would not take these results too seriously yet, they are comparing one
highly experimental kernel to another. Neither of these setups claims
to be ready for professional use yet - we are definitely going for zero
xruns, period, this seems to be achievable with most hardware. I just
left this in to give you some context.
Lee
* Paul Davis <[email protected]> wrote:
> >how is the 'maximum delay' calculated? Could you put in a tracing hook
> >into jackd whenever such a ~720 usecs maximum is hit? I'd _love_ to see
> >how such a latency path looks like, it seems a bit long.
>
> "maximum delay" is computed like this:
>
> a) we know that jackd should be woken at regular, known
> intervals based on the interrupt of the audio interface.
>
> b) jackd is (often) blocked in poll(2), waiting for the
> audio interface to become readable+writable.
>
> c) on return from poll, we check the time we are returning
> and compare it to the last wakeup. the difference should
> be almost exactly the interrupt interval.
>
> d) any amount larger than this is a delay caused by
> kernel scheduling. we track the largest such delay.
there are multiple possibilities of how this ~700 usecs delay occured:
- the kernel still has a wakeup bug. But this should be detected by the
tracer which measures the time between when the task hits the
runqueue and the task gets to execute on the CPU. Also, if there is a
critical section in the kernel that is 700 usecs long it would be
detected by _another_, independent timing/tracing mechanism that
measures critical sections. The likelyhood of both the scheduler
_and_ two independent kernel-tracers being buggy in the same way is
quite significantly low. (not to mention the user-space amlat tool
which seems to agree with the kernel instrumentation.)
note that currently the maximum delay reported by the tracer is around
20-30 usecs. It is nowhere near the 700 usecs measured here. Here are
some of the other latency possibilities:
- ALSA latency bug: Jackd didnt get woken up in time. Or driver bug, it
somehow didnt do the wakeup. I'd say this has pretty low probability
as well since it would break for alot of other scenarios as well.
- poll() latency bug: pretty low probability too. It's simple and does
the wakeup immediately, and would be noticed by many people if this
were buggy.
- Jackd related latency: is the thread that waits in poll() _truly_ the
highest-prio task in the system? IIRC there is some other Jackd
thread around too that has SCHED_FIFO priority 50, couldnt that
thread come in occasionally and delay the poll()-ing thread? Or, is
it 100% sure that the only processing the poll()-ing thread does is
waiting for the next interrupt and immediately and atomically
handling it? If there is any delay (either due to processing
overhead or due to unintended scheduling) then Jackd wont hit poll()
in time and could miss the next interrupt - and there's nothing the
kernel can do about this. I know that Jackd does alot of precautions
to avoid unintentional scheduling (mlockall, the use of SCHED_FIFO),
but are you absolutely sure it doesnt happen? This scenario could be
excluded by measuring the time Jackd calls poll(), and comparing it
to the expected value. [Or is this value already included in the
stats Rui collected? Maybe the "Maximum Process Cycle" value?]
but at this point i'd say that the likelyhood that the scheduler caused
this delay is pretty low - all of that codepath is instrumented and i
think we should instrument Jackd a bit more before putting the blame
back on the scheduler. So while ~3 months ago, when i started this
project, the blame was clearly on the kernel and on the scheduler, i'd
now say for the first time that this particular phenomenon is not
directly caused by it :-)
Could the kernel help some more in debugging this? E.g. if the Jackd
SCHED_FIFO thread (after it has started up) can never legitimately
reschedule except via poll(), i could try to put in a syscall hack in
where in the Jackd code you could call say gettimeofday(0,3) to turn on
'do not reschedule' mode, and call gettimeofday(0,4) to turn it off. If
Jackd reschedules while in 'do not reschedule' mode then the scheduler
code will detect this and will print out the offending user-space EIP
and a stacktrace. [and will turn off 'do-not schedule' mode.]
(i could even make it a clean scheduler feature and send a SIGBUS (or
any other alert signal) if this occurs, and implement a new syscall to
control it. I guess this could be useful to most RT applications. This
signal would only be sent if the RT task got scheduled voluntarily (i.e.
not delayed due to some even higher-prio task).)
> we block in poll(2), waiting for the interrupt from the audio
> interface to mark the relevant jackd thread as runnable.
ok, good. Not having signals involved certainly makes this simpler ;)
> i don't know what you mean by "channel" information in this context.
it was just a stupid question based on the ALSA source code, i assumed
that somehow you could have multiple audio streams [the driver called it
'voices'?] per driver, and the same interrupt comes for each stream. If
multiple streams are handled then there must be some sort of index to
find out where stream (or streams) got that particular interrupt, so
that Jackd can refill the proper output stream. I assumed (it seems
incorrectly) that when Rui ran 9 'fluidsynth' instances then Jackd would
open 9 'channels', one for each fludsynth instance?
but i suspect your comment means that there is a single 'output stream'
assembled by Jackd out of the 9 streams coming from the 9 fluidsynth
instances, with no concurrency issues on the driver level and the
SCHED_FIFO Jackd thread is the sole owner of that periodic interrupt,
and that there's no additional per-device information it needs to
associate it with each interrupt to ge to the current audio stream?
(i.e. there is only one output audio stream per device?)
and here i'm specifically concentrating on the scenario that Rui's
testing triggers: 9 fluidsynth instances running, and Jackd handling a
single ali5451 output device.
Ingo
On Fri, 2004-10-29 at 10:02 +0200, Ingo Molnar wrote:
> there are multiple possibilities of how this ~700 usecs delay occured:
>
> - the kernel still has a wakeup bug. But this should be detected by the
> tracer which measures the time between when the task hits the
> runqueue and the task gets to execute on the CPU. Also, if there is a
> critical section in the kernel that is 700 usecs long it would be
> detected by _another_, independent timing/tracing mechanism that
> measures critical sections. The likelyhood of both the scheduler
> _and_ two independent kernel-tracers being buggy in the same way is
> quite significantly low. (not to mention the user-space amlat tool
> which seems to agree with the kernel instrumentation.)
>
The sound subsystem uses a lot of sleep_on() variants. We know that they
are racy. Might this be related ?
tglx
* Ingo Molnar <[email protected]> wrote:
> Could the kernel help some more in debugging this? E.g. if the Jackd
> SCHED_FIFO thread (after it has started up) can never legitimately
> reschedule except via poll(), i could try to put in a syscall hack in
> where in the Jackd code you could call say gettimeofday(0,3) to turn
> on 'do not reschedule' mode, and call gettimeofday(0,4) to turn it
> off. If Jackd reschedules while in 'do not reschedule' mode then the
> scheduler code will detect this and will print out the offending
> user-space EIP and a stacktrace. [and will turn off 'do-not schedule'
> mode.]
i've implemented this feature and have put it into -RT-V0.5.5 which can
be downloaded from the usual place:
http://redhat.com/~mingo/realtime-preempt/
i've attached a simple testcase showing how to use it. When running the
testcode on a -V0.5.5 kernel it gives:
saturn:~> ./rt-atomic
testing atomic mode functionality.
ok, kernel supports atomic mode.
atomic mode is now off - doing sleep(), should succeed:
turning atomic mode on.
doing getppid() syscall - should succeed.
doing sleep() syscall - should abort!
User defined signal 1
the kernel sends SIGUSR1 when it detects illegal scheduling. (change the
SIGUSR1 in the patch if you want another signal.)
could someone with more Jackd experience than me add this to the Jackd
code and check whether and where it triggers? I'd suggest to do
something like around the poll() call:
atomic_off();
poll();
atomic_on();
and add an atomic_off() to the Jackd shutdown handler (or any codepath
that is legitimately allowed to schedule).
Ingo
* Thomas Gleixner <[email protected]> wrote:
> The sound subsystem uses a lot of sleep_on() variants. We know that
> they are racy. Might this be related ?
i doubt it is related - sleep_on() is mostly racy on SMP and Rui is
doing UP tests. Also, for a sleep_on() race to occur you'd have to race
with the wakeup - but in the Jackd case that is highly unlikely because
it sleeps well before the next interrupt is due. So while a sleep_on()
race could in theory make _existing_ latencies worse, it cannot by
itself introduce the first latency.
but now it should be possible to prove that Jackd itself doesnt
introduce latencies, via the userspace-atomicity mode feature in -V0.5.5
:-) If that proves that the latencies dont occur in Jackd (and if it's
not another, even higher prio thread that causes the delay) then we can
pass the ball back to the kernel.
Ingo
* Ingo Molnar <[email protected]> wrote:
> i've implemented this feature and have put it into -RT-V0.5.5 which can
> be downloaded from the usual place:
>
> http://redhat.com/~mingo/realtime-preempt/
there was a typo in it, i've uploaded -RT-V0.5.6 with that fixed.
Ingo
>but now it should be possible to prove that Jackd itself doesnt
>introduce latencies, via the userspace-atomicity mode feature in -V0.5.5
>:-) If that proves that the latencies dont occur in Jackd (and if it's
>not another, even higher prio thread that causes the delay) then we can
>pass the ball back to the kernel.
this is something i haven't mentioned. when running in RT mode, jackd
also runs a higher prio RT thread called the watchdog. it wakes every
5 seconds to check that the "main" thread is not stalled (i.e. a
client that is stuck in a loop) and kills everything if it
is. however, this thread's loop is incredibly simple (it checks and
resets a single int variable and goes back to sleep) that causing a
700usec delay would itself seem to me to be indicative of a
problem. do you agree?
--p
* Paul Davis <[email protected]> wrote:
> this is something i haven't mentioned. when running in RT mode, jackd
> also runs a higher prio RT thread called the watchdog. it wakes every
> 5 seconds to check that the "main" thread is not stalled (i.e. a
> client that is stuck in a loop) and kills everything if it is.
> however, this thread's loop is incredibly simple (it checks and resets
> a single int variable and goes back to sleep) that causing a 700usec
> delay would itself seem to me to be indicative of a problem. do you
> agree?
agreed, the watchdog thread should have no measurable impact - it should
have all of its work done well under 10 usecs on a fast box, and not
above 20 usecs even on a slow box. As long as it doesnt log to a file it
should have no impact. It's also easy to exclude this from the list of
things, Rui could perhaps chrt this particular jackd thread to below the
main jack thread's priority. (it shouldnt break the watchdog
functionality too much, unless the main jackd thread itself goes into a
loop which clearly isnt the problem currently.)
my main suspicion is that either the main jackd thread itself calls the
kernel where the kernel (unexpectedly for jackd) schedules away for
whatever reason, or that the chain of wakeup in the audio path somehow
gets violated (i.e. a kernel problem). There's one quick thing we could
try: could you send me an 'strace -p' log of a couple of interrupt
cycles of jackd handling an ordinary audio stream? There's so many ways
the kernel can schedule away, perhaps i can spot it by looking at the
syscalls called.
in fact, does the jackd high-prio thread (excluding the watchdog) call
any syscalls normally, besides the poll() and then the read()/write() on
the audio device fd? (perhaps it does an ioctl too?)
ahh ... the ioctls. All ioctls in Linux take the 'big kernel lock'. So
if jackd calls an ioctl() in the fastpath then that could easily get
delayed. Also, there are a couple of other syscalls too that touch the
BKL, so an strace of the 'jackd latency path' would be quite useful.
Ingo
>my main suspicion is that either the main jackd thread itself calls the
>kernel where the kernel (unexpectedly for jackd) schedules away for
>whatever reason, or that the chain of wakeup in the audio path somehow
>gets violated (i.e. a kernel problem). There's one quick thing we could
the "max delay" measurement isn't a reflection of the runtime activity
of jackd. its simply a measurement of the delay between when jackd
expected to be next woken from poll and when it actually was.
as you noted, jackd generally goes back to sleep in poll typically a
long time before the next interrupt is expected. hence any delay in
the wakeup is between the interrupt handler and the scheduler getting
jackd's main thread back on the processor. i think.
* Paul Davis <[email protected]> wrote:
> >my main suspicion is that either the main jackd thread itself calls the
> >kernel where the kernel (unexpectedly for jackd) schedules away for
> >whatever reason, or that the chain of wakeup in the audio path somehow
> >gets violated (i.e. a kernel problem). There's one quick thing we could
>
> the "max delay" measurement isn't a reflection of the runtime activity
> of jackd. its simply a measurement of the delay between when jackd
> expected to be next woken from poll and when it actually was.
>
> as you noted, jackd generally goes back to sleep in poll typically a
> long time before the next interrupt is expected. hence any delay in
> the wakeup is between the interrupt handler and the scheduler getting
> jackd's main thread back on the processor. i think.
this brings up the next question: does the jackd measurement also
timestamp the time when it calls poll() - hence detecting in-jackd
processing delays? If yes, which value is this from Rui's stats? If not
then it might make sense to add it.
Ingo
(sorry for the slow turnaround. i am also going offline for the next 6
hours, so this is my last reply till later ...)
>> the "max delay" measurement isn't a reflection of the runtime activity
>> of jackd. its simply a measurement of the delay between when jackd
>> expected to be next woken from poll and when it actually was.
>>
>> as you noted, jackd generally goes back to sleep in poll typically a
>> long time before the next interrupt is expected. hence any delay in
>> the wakeup is between the interrupt handler and the scheduler getting
>> jackd's main thread back on the processor. i think.
>
>this brings up the next question: does the jackd measurement also
>timestamp the time when it calls poll() - hence detecting in-jackd
>processing delays? If yes, which value is this from Rui's stats? If not
>then it might make sense to add it.
i'm not quite sure what you mean. the timestamps used to measure
max_delay look like this:
... set up poll fd struct ....
poll_enter = jack_get_microseconds ();
if (poll (driver->pfd, nfds, driver->poll_timeout) < 0) {
if (errno == EINTR) {
printf ("poll interrupt\n");
// this happens mostly when run
// under gdb, or when exiting due to a signal
if (under_gdb) {
goto again;
}
*status = -2;
return 0;
}
jack_error ("ALSA: poll call failed (%s)",
strerror (errno));
*status = -3;
return 0;
}
poll_ret = jack_get_microseconds ();
if (extra_fd < 0) {
if (driver->poll_next && poll_ret > driver->poll_next) {
*delayed_usecs = poll_ret - driver->poll_next;
}
driver->poll_last = poll_ret;
driver->poll_next = poll_ret + driver->period_usecs;
driver->engine->transport_cycle_start (driver->engine,
poll_ret);
}
you can view this here in context:
http://jackit.sourceforge.net/cgi-bin/lxr/http/source/drivers/alsa/alsa_driver.c
(its at line 1195)
the max_delay value is taken from "delayed_usecs" above. each backend
(the one shown is the ALSA driver) sets this every time its "wait"
function is called; jackd collects the figures and these days manages
them as a set of statistics.
as you can see, poll_enter is available, but we don't use it for much
at all (mostly just some information in the very unusual event that
poll times out).
jack_get_microseconds() uses the cycle timer to compute elapsed time,
there is no system call involved.
--p
> i don't know what you mean by "channel" information in this context.
after reading some more code i believe the concept is called 'stream' in
ALSA speak. How many streams did Rui's test utilize? (nine?)
if multiple streams were used, then can the handling of one stream delay
another stream? How are streams prioritized, and are they queued in a
FIFO or in a LIFO manner? (Also, are such 'streams' directly mapped to
some hardware concept on the audio card ('channel' or 'voice'?) or am i
confusing things?)
Ingo
* Paul Davis <[email protected]> wrote:
> if (poll (driver->pfd, nfds, driver->poll_timeout) < 0) {
in Rui's test (9 fluidsynth instances running), what would be the value
of nfds - 9 or 1? I.e. is the '9 streams' abstraction provided by jackd,
and you map it to a single alsa driver fd over which you poll, or is
each stream a separate fd?
Ingo
>> if (poll (driver->pfd, nfds, driver->poll_timeout) < 0) {
>
>in Rui's test (9 fluidsynth instances running), what would be the value
>of nfds - 9 or 1? I.e. is the '9 streams' abstraction provided by jackd,
>and you map it to a single alsa driver fd over which you poll, or is
>each stream a separate fd?
for almost all audio interfaces, there is one fd per hardware
device. so whether its a 26 channel device or a simple stereo device,
there is one fd. jackd will be polling a single fd in that call under
almost all circumstances, and certainly those used in rui's testing.
--p
* Paul Davis <[email protected]> wrote:
> >> if (poll (driver->pfd, nfds, driver->poll_timeout) < 0) {
> >
> >in Rui's test (9 fluidsynth instances running), what would be the value
> >of nfds - 9 or 1? I.e. is the '9 streams' abstraction provided by jackd,
> >and you map it to a single alsa driver fd over which you poll, or is
> >each stream a separate fd?
>
> for almost all audio interfaces, there is one fd per hardware device.
> so whether its a 26 channel device or a simple stereo device, there is
> one fd. jackd will be polling a single fd in that call under almost
> all circumstances, and certainly those used in rui's testing.
great - this certainly simplifies the scenario.
Ingo
>after reading some more code i believe the concept is called 'stream' in
>ALSA speak. How many streams did Rui's test utilize? (nine?)
it doesn't matter, actually ...
>if multiple streams were used, then can the handling of one stream delay
>another stream? How are streams prioritized, and are they queued in a
>FIFO or in a LIFO manner? (Also, are such 'streams' directly mapped to
>some hardware concept on the audio card ('channel' or 'voice'?) or am i
>confusing things?)
for our purposes, the number of channels is irrelevant. the audio
interface interrupts when it has and/or needs data. whether that data
covers 2, 9, 26 or 128 channels isn't particularly important. each
channel is handled at the same time. for most stereo devices, the data
for the channels is actually interleaved; for some high end
multichannel devices, each channel uses a separate memory
buffer. either way, its really not central.
\begin{digression}
what may confuse you is the kernel-level ALSA concept of a "stream"
and a "substream". this is not relevant to the issue at hand, but i
may as well explain anyway :)
these are not actually channels at all, but abstractions that ALSA
uses to handle many different audio h/w architectures with a
reasonable level of uniformity. many consumer audio devices these days
have multiple independent audio engines on them. for example, some
cards have analog stereo output, analog surround (so-called "5.1")
output and/or digital S/PDIF output. if they are truly independent
(i.e. they carry different data and can be started and stopped
independently), then ALSA generally represents each device
independently. in the driver architecture, each device is represented
by a "stream" when it is open. audio interfaces with just a single
device has only one stream that can be opened.
in addition, there are a number of audio interfaces that can do h/w
mixing of multiple data flowing into the same device. i have a
consumer card, for example, that can handle 32 different stereo
streams and it mixes them down to a single stereo output. these are
not independent in the sense that they cannot be started/stopped
independently, and so ALSA represents each of these as a
"substream". An audio interface that doesn't do h/w mixing has only a
single substream associated with the stream.
When an application opens an audio interface, it is opening a
substream of a stream. By default, it will get the first (typically
only) substream on the first stream (typically only) of the first
audio interface.
The term "voice" is a driver-specific one that has been inherited from
the way the makes of certain audio interfaces talk about their hardware.
It corresponds very loosely to ALSA's notion of a substream, but not
precisely.
\end{digression}
to get back to max_delay .... the important thing here is that
max_delay is measuring the delay between when jackd believes it should
*next* be woken and when it actually is. on the assumption (valid for
rui's test AFAIK) that jackd has gone back to sleep a long time before
the next audio interface interrupt wakes it up, then this delay is
troublesome, because it happens entirely in the kernel.
some things we need to rule out:
(a) jackd is not asleep when the next interrupt arrives
(though this should not cause a delay of 700usecs)
(b) the audio interface delivered the interrupt on time
(c) the audio interface doesn't interrupt independently for
capture and playback. Rui will need to get back
to me with details on what type of audio interface
he is using for me to comment on this. If its
a consumer device with poor support for full
duplex operation, then it can happen that
capture and playback streams are running out
of sync and this can cause some odd timing.
--p
* Paul Davis <[email protected]> wrote:
> for our purposes, the number of channels is irrelevant. the audio
> interface interrupts when it has and/or needs data. whether that data
> covers 2, 9, 26 or 128 channels isn't particularly important. each
> channel is handled at the same time. for most stereo devices, the data
> for the channels is actually interleaved; for some high end
> multichannel devices, each channel uses a separate memory buffer.
> either way, its really not central.
ok, thanks for the explanations.
Ingo
>here's Rui's setup:
>
>| Note: all tests were carried out running jackd -v -dalsa -dhw:0
>| -r44100 -p128 -n2 -S -P, loaded with 9 (nine) fluidsynth instances,
>| on a [email protected] laptop, against the onboard sound device
>| (snd-ali5451).
>
>i suspect this means there was playback only, no capture, and thus the
>capture/playback interrupt interaction cannot be the case, right?
ok, he used the -P flag which means playback only. so yes, the ALSA
backend would only have opened the playback stream.
oh, and i must correct what i said before. the ALSA backend will
typically be polling on *two* fd's. ALSA represents playback and
capture as different devices, so a default jackd will open both and
poll both of them. the backend will wait until they are both ready
(ie. the capture device has the correct amount of data, the playback
device has the correct amount of space).
--p
* Paul Davis <[email protected]> wrote:
> (c) the audio interface doesn't interrupt independently for
> capture and playback. Rui will need to get back
> to me with details on what type of audio interface
> he is using for me to comment on this. If its
> a consumer device with poor support for full
> duplex operation, then it can happen that
> capture and playback streams are running out
> of sync and this can cause some odd timing.
here's Rui's setup:
| Note: all tests were carried out running jackd -v -dalsa -dhw:0
| -r44100 -p128 -n2 -S -P, loaded with 9 (nine) fluidsynth instances,
| on a [email protected] laptop, against the onboard sound device
| (snd-ali5451).
i suspect this means there was playback only, no capture, and thus the
capture/playback interrupt interaction cannot be the case, right?
Ingo
* Ingo Molnar <[email protected]> wrote:
> ahh ... the ioctls. All ioctls in Linux take the 'big kernel lock'. So
> if jackd calls an ioctl() in the fastpath then that could easily get
> delayed. Also, there are a couple of other syscalls too that touch the
> BKL, so an strace of the 'jackd latency path' would be quite useful.
i straced jackd and it seems to do ioctls:
5971 ioctl(7, 0x4143, 0x446b7d3c) = 0
5971 ioctl(7, 0x4140, 0x446b7d3c) = 0
5971 ioctl(7, 0x4142, 0x446b7d3c) = 0
5971 poll([{fd=7, events=POLLOUT|POLLERR|POLLNVAL, revents=POLLOUT}], 1, 4353) = 1
5971 poll([{fd=7, events=POLLOUT|POLLERR|POLLNVAL, revents=POLLOUT}], 1, 4353)= 1
5971 poll([{fd=7, events=POLLOUT|POLLERR|POLLNVAL, revents=POLLOUT|POLLERR}], 1, 4353) = 1
5971 ioctl(7, 0x806c4120, 0xb77bb960) = 0
5971 gettimeofday({1099064915, 776665}, NULL) = 0
the BKL can generate arbitrary latencies. Anything up to 100-200
milliseconds. Rui, Florian, could you try the quick hack below?
Ingo
--- linux/fs/ioctl.c.orig2
+++ linux/fs/ioctl.c
@@ -68,19 +68,26 @@ asmlinkage long sys_ioctl(unsigned int f
goto out;
}
- lock_kernel();
switch (cmd) {
case FIOCLEX:
+ lock_kernel();
set_close_on_exec(fd, 1);
+ unlock_kernel();
break;
case FIONCLEX:
+ lock_kernel();
set_close_on_exec(fd, 0);
+ unlock_kernel();
break;
case FIONBIO:
- if ((error = get_user(on, (int __user *)arg)) != 0)
+ lock_kernel();
+ unlock_kernel();
+ if ((error = get_user(on, (int __user *)arg)) != 0) {
+ unlock_kernel();
break;
+ }
flag = O_NONBLOCK;
#ifdef __sparc__
/* SunOS compatibility item. */
@@ -91,11 +98,15 @@ asmlinkage long sys_ioctl(unsigned int f
filp->f_flags |= flag;
else
filp->f_flags &= ~flag;
+ unlock_kernel();
break;
case FIOASYNC:
- if ((error = get_user(on, (int __user *)arg)) != 0)
+ lock_kernel();
+ if ((error = get_user(on, (int __user *)arg)) != 0) {
+ unlock_kernel();
break;
+ }
flag = on ? FASYNC : 0;
/* Did FASYNC state change ? */
@@ -104,16 +115,20 @@ asmlinkage long sys_ioctl(unsigned int f
error = filp->f_op->fasync(fd, filp, on);
else error = -ENOTTY;
}
- if (error != 0)
+ if (error != 0) {
+ unlock_kernel();
break;
+ }
if (on)
filp->f_flags |= FASYNC;
else
filp->f_flags &= ~FASYNC;
+ unlock_kernel();
break;
case FIOQSIZE:
+ lock_kernel();
if (S_ISDIR(filp->f_dentry->d_inode->i_mode) ||
S_ISREG(filp->f_dentry->d_inode->i_mode) ||
S_ISLNK(filp->f_dentry->d_inode->i_mode)) {
@@ -122,6 +137,7 @@ asmlinkage long sys_ioctl(unsigned int f
}
else
error = -ENOTTY;
+ unlock_kernel();
break;
default:
error = -ENOTTY;
@@ -130,7 +146,6 @@ asmlinkage long sys_ioctl(unsigned int f
else if (filp->f_op && filp->f_op->ioctl)
error = filp->f_op->ioctl(filp->f_dentry->d_inode, filp, cmd, arg);
}
- unlock_kernel();
fput(filp);
out:
On Fri, 29 Oct 2004 18:14:33 +0200
Ingo Molnar <[email protected]> wrote:
> the BKL can generate arbitrary latencies. Anything up to 100-200
> milliseconds. Rui, Florian, could you try the quick hack below?
sure, with a fully REALTIME_PREEMPTION kernel?
flo
* Ingo Molnar <[email protected]> wrote:
> the BKL can generate arbitrary latencies. Anything up to 100-200
> milliseconds. Rui, Florian, could you try the quick hack below?
hm, this alone probably wont work, because the audio layer does
unlock_kernel() in many places.
could you try another hack in addition to the ioctl.c hack, save the
attached script, and do this in your Linux source tree:
cd sound
~/changeall-tree "unlock_kernel()" ""
~/changeall-tree "lock_kernel()" ""
(in strictly this order.) This gets rid of the BKL from the sound
subsystem in quite drastic ways... Totally untested, so be careful ...
Ingo
* Florian Schmidt <[email protected]> wrote:
> > the BKL can generate arbitrary latencies. Anything up to 100-200
> > milliseconds. Rui, Florian, could you try the quick hack below?
>
> sure, with a fully REALTIME_PREEMPTION kernel?
correct, and with the changeall-tree hack in addition. And keep your
finger near the reset button, just in case ...
Ingo
* Ingo Molnar <[email protected]> wrote:
> > correct, and with the changeall-tree hack in addition. And keep your
> > finger near the reset button, just in case ...
>
> it wont even boot ...
>
> let me try some more hacks to make this a little bit safer.
ok, next attempt. Please apply this patch and do the changeall-tree
thing in the sound/ directory. This should in theory switch the ALSA
ioctls (and only them) to a BKL-less variant.
Ingo
--- linux/sound/core/pcm_native.c.orig
+++ linux/sound/core/pcm_native.c
@@ -3317,7 +3317,7 @@ static struct file_operations snd_pcm_f_
.open = snd_pcm_open,
.release = snd_pcm_release,
.poll = snd_pcm_playback_poll,
- .ioctl = snd_pcm_playback_ioctl,
+ .ioctl2 = snd_pcm_playback_ioctl,
.mmap = snd_pcm_mmap,
.fasync = snd_pcm_fasync,
};
@@ -3329,7 +3329,7 @@ static struct file_operations snd_pcm_f_
.open = snd_pcm_open,
.release = snd_pcm_release,
.poll = snd_pcm_capture_poll,
- .ioctl = snd_pcm_capture_ioctl,
+ .ioctl2 = snd_pcm_capture_ioctl,
.mmap = snd_pcm_mmap,
.fasync = snd_pcm_fasync,
};
--- linux/fs/ioctl.c.orig
+++ linux/fs/ioctl.c
@@ -68,19 +68,26 @@ asmlinkage long sys_ioctl(unsigned int f
goto out;
}
- lock_kernel();
switch (cmd) {
case FIOCLEX:
+ lock_kernel();
set_close_on_exec(fd, 1);
+ unlock_kernel();
break;
case FIONCLEX:
+ lock_kernel();
set_close_on_exec(fd, 0);
+ unlock_kernel();
break;
case FIONBIO:
- if ((error = get_user(on, (int __user *)arg)) != 0)
+ lock_kernel();
+ unlock_kernel();
+ if ((error = get_user(on, (int __user *)arg)) != 0) {
+ unlock_kernel();
break;
+ }
flag = O_NONBLOCK;
#ifdef __sparc__
/* SunOS compatibility item. */
@@ -91,11 +98,15 @@ asmlinkage long sys_ioctl(unsigned int f
filp->f_flags |= flag;
else
filp->f_flags &= ~flag;
+ unlock_kernel();
break;
case FIOASYNC:
- if ((error = get_user(on, (int __user *)arg)) != 0)
+ lock_kernel();
+ if ((error = get_user(on, (int __user *)arg)) != 0) {
+ unlock_kernel();
break;
+ }
flag = on ? FASYNC : 0;
/* Did FASYNC state change ? */
@@ -104,16 +115,20 @@ asmlinkage long sys_ioctl(unsigned int f
error = filp->f_op->fasync(fd, filp, on);
else error = -ENOTTY;
}
- if (error != 0)
+ if (error != 0) {
+ unlock_kernel();
break;
+ }
if (on)
filp->f_flags |= FASYNC;
else
filp->f_flags &= ~FASYNC;
+ unlock_kernel();
break;
case FIOQSIZE:
+ lock_kernel();
if (S_ISDIR(filp->f_dentry->d_inode->i_mode) ||
S_ISREG(filp->f_dentry->d_inode->i_mode) ||
S_ISLNK(filp->f_dentry->d_inode->i_mode)) {
@@ -122,15 +137,21 @@ asmlinkage long sys_ioctl(unsigned int f
}
else
error = -ENOTTY;
+ unlock_kernel();
break;
default:
error = -ENOTTY;
- if (S_ISREG(filp->f_dentry->d_inode->i_mode))
+ if (S_ISREG(filp->f_dentry->d_inode->i_mode)) {
+ lock_kernel();
error = file_ioctl(filp, cmd, arg);
- else if (filp->f_op && filp->f_op->ioctl)
+ unlock_kernel();
+ } else if (filp->f_op && filp->f_op->ioctl) {
+ lock_kernel();
error = filp->f_op->ioctl(filp->f_dentry->d_inode, filp, cmd, arg);
+ unlock_kernel();
+ } else if (filp->f_op && filp->f_op->ioctl2)
+ error = filp->f_op->ioctl2(filp->f_dentry->d_inode, filp, cmd, arg);
}
- unlock_kernel();
fput(filp);
out:
--- linux/include/linux/fs.h.orig
+++ linux/include/linux/fs.h
@@ -993,6 +993,7 @@ struct file_operations {
int (*check_flags)(int);
int (*dir_notify)(struct file *filp, unsigned long arg);
int (*flock) (struct file *, int, struct file_lock *);
+ int (*ioctl2) (struct inode *, struct file *, unsigned int, unsigned long);
};
struct inode_operations {
* Ingo Molnar <[email protected]> wrote:
> > > the BKL can generate arbitrary latencies. Anything up to 100-200
> > > milliseconds. Rui, Florian, could you try the quick hack below?
> >
> > sure, with a fully REALTIME_PREEMPTION kernel?
>
> correct, and with the changeall-tree hack in addition. And keep your
> finger near the reset button, just in case ...
it wont even boot ...
let me try some more hacks to make this a little bit safer.
Ingo
* Florian Schmidt <[email protected]> wrote:
> > let me try some more hacks to make this a little bit safer.
>
> hehe, it even booted for me [kinda]. will build the one where you got
> xmms to run. but i will sure as hell hit "get new emails" during the
> build more than once ;)
indeed - i'm preparing 5.0.12 with a better way to do this. (the trick
is to allow the BKL to 'underflow' - this way ALSA can be kept largely
unmodified.)
Ingo
* Ingo Molnar <[email protected]> wrote:
> ok, next attempt. Please apply this patch and do the changeall-tree
> thing in the sound/ directory. This should in theory switch the ALSA
> ioctls (and only them) to a BKL-less variant.
ok, new version - the previous one was slighly racy. The one below works
better - it boots on my box and simple audio playback (xmms) works.
Ingo
--- linux/sound/core/pcm_native.c.orig2
+++ linux/sound/core/pcm_native.c
@@ -3317,7 +3317,7 @@ static struct file_operations snd_pcm_f_
.open = snd_pcm_open,
.release = snd_pcm_release,
.poll = snd_pcm_playback_poll,
- .ioctl = snd_pcm_playback_ioctl,
+ .ioctl2 = snd_pcm_playback_ioctl,
.mmap = snd_pcm_mmap,
.fasync = snd_pcm_fasync,
};
@@ -3329,7 +3329,7 @@ static struct file_operations snd_pcm_f_
.open = snd_pcm_open,
.release = snd_pcm_release,
.poll = snd_pcm_capture_poll,
- .ioctl = snd_pcm_capture_ioctl,
+ .ioctl2 = snd_pcm_capture_ioctl,
.mmap = snd_pcm_mmap,
.fasync = snd_pcm_fasync,
};
--- linux/sound/pci/ali5451/ali5451.c.orig2
+++ linux/sound/pci/ali5451/ali5451.c
@@ -33,6 +33,7 @@
#include <linux/pci.h>
#include <linux/slab.h>
#include <linux/moduleparam.h>
+#include <linux/syscalls.h>
#include <sound/core.h>
#include <sound/pcm.h>
#include <sound/info.h>
@@ -985,11 +986,11 @@ static void snd_ali_update_ptr(ali_t *co
pvoice = &codec->synth.voices[channel];
runtime = pvoice->substream->runtime;
- udelay(100);
spin_lock(&codec->reg_lock);
if (pvoice->pcm && pvoice->substream) {
/* pcm interrupt */
+ sys_gettimeofday((void *)0, (void *)1); // start the tracer
#ifdef ALI_DEBUG
outb((u8)(pvoice->number), ALI_REG(codec, ALI_GC_CIR));
temp = inw(ALI_REG(codec, ALI_CSO_ALPHA_FMS + 2));
--- linux/fs/ioctl.c.orig2
+++ linux/fs/ioctl.c
@@ -68,6 +68,13 @@ asmlinkage long sys_ioctl(unsigned int f
goto out;
}
+ error = -ENOTTY;
+ if (filp->f_op && filp->f_op->ioctl2) {
+ error = filp->f_op->ioctl2(filp->f_dentry->d_inode, filp, cmd, arg);
+ fput(filp);
+ return error;
+ }
+
lock_kernel();
switch (cmd) {
case FIOCLEX:
@@ -132,7 +139,6 @@ asmlinkage long sys_ioctl(unsigned int f
}
unlock_kernel();
fput(filp);
-
out:
return error;
}
--- linux/include/linux/fs.h.orig2
+++ linux/include/linux/fs.h
@@ -993,6 +993,7 @@ struct file_operations {
int (*check_flags)(int);
int (*dir_notify)(struct file *filp, unsigned long arg);
int (*flock) (struct file *, int, struct file_lock *);
+ int (*ioctl2) (struct inode *, struct file *, unsigned int, unsigned long);
};
struct inode_operations {
On Fri, 29 Oct 2004 18:31:55 +0200
Ingo Molnar <[email protected]> wrote:
>
> * Ingo Molnar <[email protected]> wrote:
>
> > > > the BKL can generate arbitrary latencies. Anything up to 100-200
> > > > milliseconds. Rui, Florian, could you try the quick hack below?
> > >
> > > sure, with a fully REALTIME_PREEMPTION kernel?
> >
> > correct, and with the changeall-tree hack in addition. And keep your
> > finger near the reset button, just in case ...
>
> it wont even boot ...
>
> let me try some more hacks to make this a little bit safer.
hehe, it even booted for me [kinda]. will build the one where you got xmms
to run. but i will sure as hell hit "get new emails" during the build more
than once ;)
flo
On Fri, 29 Oct 2004 19:09:48 +0200
Ingo Molnar <[email protected]> wrote:
> ok, i've uploaded -V0.5.12 which has this BKL trick. I hope i got the
> ALSA ioctls right: the ones that go outside the BKL for now are the
> pcm_native playback/capture ones, and rawmidi's ioctl. All the others
> are still BKL users. To recap, it's these ioctls that matter:
doesn't build here:
CC fs/exec.o
CC fs/pipe.o
CC fs/namei.o
CC fs/fcntl.o
CC fs/ioctl.o
fs/ioctl.c: In function `sys_ioctl':
fs/ioctl.c:75: error: structure has no member named `ioctl_nobkl'
fs/ioctl.c:76: error: structure has no member named `ioctl_nobkl'
make[1]: *** [fs/ioctl.o] Error 1
make: *** [fs] Error 2
flo
* Ingo Molnar <[email protected]> wrote:
> > > let me try some more hacks to make this a little bit safer.
> >
> > hehe, it even booted for me [kinda]. will build the one where you got
> > xmms to run. but i will sure as hell hit "get new emails" during the
> > build more than once ;)
>
> indeed - i'm preparing 5.0.12 with a better way to do this. (the trick
> is to allow the BKL to 'underflow' - this way ALSA can be kept largely
> unmodified.)
ok, i've uploaded -V0.5.12 which has this BKL trick. I hope i got the
ALSA ioctls right: the ones that go outside the BKL for now are the
pcm_native playback/capture ones, and rawmidi's ioctl. All the others
are still BKL users. To recap, it's these ioctls that matter:
5971 ioctl(7, 0x4143, 0x446b7d3c) = 0
5971 ioctl(7, 0x4140, 0x446b7d3c) = 0
5971 ioctl(7, 0x4142, 0x446b7d3c) = 0
which ones are these? Look at the patch for how to change a .ioctl one
to .ioctl_nobkl (lame solution ...). So if your setup uses any other
ioctl (sndctl perhaps?) then you should change that one to nobkl too.
Ingo
* Ingo Molnar <[email protected]> wrote:
>
> * Florian Schmidt <[email protected]> wrote:
>
> > CC fs/ioctl.o
> > fs/ioctl.c: In function `sys_ioctl':
> > fs/ioctl.c:75: error: structure has no member named `ioctl_nobkl'
> > fs/ioctl.c:76: error: structure has no member named `ioctl_nobkl'
>
> fs.h chunk went missing ... uploading -V0.5.14 in a minute.
done.
Ingo
* Ingo Molnar <[email protected]> wrote:
> 5971 ioctl(7, 0x4143, 0x446b7d3c) = 0
> 5971 ioctl(7, 0x4140, 0x446b7d3c) = 0
> 5971 ioctl(7, 0x4142, 0x446b7d3c) = 0
>
> which ones are these? Look at the patch for how to change a .ioctl one
> to .ioctl_nobkl (lame solution ...). So if your setup uses any other
> ioctl (sndctl perhaps?) then you should change that one to nobkl too.
note that even if just one of these ioctls is still a BKL user then
-V0.5.12 will show no significant difference. So i've uploaded -V0.5.13
with some more converted. I think this should cover the above ones, but
i'm not 100% sure.
Ingo
* Florian Schmidt <[email protected]> wrote:
> CC fs/ioctl.o
> fs/ioctl.c: In function `sys_ioctl':
> fs/ioctl.c:75: error: structure has no member named `ioctl_nobkl'
> fs/ioctl.c:76: error: structure has no member named `ioctl_nobkl'
fs.h chunk went missing ... uploading -V0.5.14 in a minute.
Ingo
On Fri, 29 Oct 2004 19:22:43 +0200
Ingo Molnar <[email protected]> wrote:
>
> * Ingo Molnar <[email protected]> wrote:
>
> >
> > * Florian Schmidt <[email protected]> wrote:
> >
> > > CC fs/ioctl.o
> > > fs/ioctl.c: In function `sys_ioctl':
> > > fs/ioctl.c:75: error: structure has no member named `ioctl_nobkl'
> > > fs/ioctl.c:76: error: structure has no member named `ioctl_nobkl'
> >
> > fs.h chunk went missing ... uploading -V0.5.14 in a minute.
>
> done.
compiles and boots fine. no observable change in xrun behaviour though.
flo
On Fri, 2004-10-29 at 10:02 +0200, Ingo Molnar wrote:
> I know that Jackd does alot of precautions
> to avoid unintentional scheduling (mlockall, the use of SCHED_FIFO),
> but are you absolutely sure it doesnt happen? This scenario could be
> excluded by measuring the time Jackd calls poll(), and comparing it
> to the expected value. [Or is this value already included in the
> stats Rui collected? Maybe the "Maximum Process Cycle" value?]
Yes, this is already accounted for in the 'Maximum Process Cycle' value.
This measures the time between returning from poll() and entering it
again. I will try to add some instrumentation to jackd and test this
weekend. I do agree that it could be a jackd bug; this would not be the
first time the VP patches exposed bugs in other apps.
Lee
On Fri, 2004-10-29 at 15:33 -0400, Lee Revell wrote:
> On Fri, 2004-10-29 at 10:02 +0200, Ingo Molnar wrote:
> > I know that Jackd does alot of precautions
> > to avoid unintentional scheduling (mlockall, the use of SCHED_FIFO),
> > but are you absolutely sure it doesnt happen? This scenario could be
> > excluded by measuring the time Jackd calls poll(), and comparing it
> > to the expected value. [Or is this value already included in the
> > stats Rui collected? Maybe the "Maximum Process Cycle" value?]
>
> Yes, this is already accounted for in the 'Maximum Process Cycle' value.
> This measures the time between returning from poll() and entering it
> again. I will try to add some instrumentation to jackd and test this
> weekend. I do agree that it could be a jackd bug; this would not be the
> first time the VP patches exposed bugs in other apps.
>
Can you check out, whether the memory is getting low when you are doing
this tests ? The VM code has a serious problem, which might be related
to those latencies.
tglx
* Florian Schmidt <[email protected]> wrote:
> > > > fs/ioctl.c: In function `sys_ioctl':
> > > > fs/ioctl.c:75: error: structure has no member named `ioctl_nobkl'
> > > > fs/ioctl.c:76: error: structure has no member named `ioctl_nobkl'
> > >
> > > fs.h chunk went missing ... uploading -V0.5.14 in a minute.
> >
> > done.
>
> compiles and boots fine. no observable change in xrun behaviour though.
ok, so there's something else going on as well - or i missed an ioctl.
Even if it's not the BKL, the ioctl/BKL problem would come back later
and haunt us during stresstests anyway so it's not bad that we have a
solution for it.
Ingo
* Florian Schmidt <[email protected]> wrote:
> > > fs.h chunk went missing ... uploading -V0.5.14 in a minute.
> >
> > done.
>
> compiles and boots fine. no observable change in xrun behaviour
> though.
do you compile jackd from sources? If yes then could you try the patch
below? With this added, the kernel will produce a stackdump whenever
jackd does an 'illegal' sleep.
Also, could you do a small modification to kernel/sched.c and remove
this line:
send_sig(SIGUSR1, current, 1);
just to make it easier to get Jack up and running. (by default an
atomicity violation triggers a signal to make it easier to debug it in
userspace, but i suspect there will be alot of such violations so jackd
would stop all the time.)
Ingo
--- jack-audio-connection-kit-0.99.0/drivers/alsa/alsa_driver.c.orig
+++ jack-audio-connection-kit-0.99.0/drivers/alsa/alsa_driver.c
@@ -1161,6 +1161,7 @@ alsa_driver_wait (alsa_driver_t *driver,
unsigned int p_timed_out, c_timed_out;
unsigned int ci = 0;
unsigned int nfds;
+ int ret;
nfds = 0;
@@ -1194,7 +1195,11 @@ alsa_driver_wait (alsa_driver_t *driver,
poll_enter = jack_get_microseconds ();
- if (poll (driver->pfd, nfds, driver->poll_timeout) < 0) {
+ gettimeofday((void *)1,(void *)0); // atomic off
+ ret = poll (driver->pfd, nfds, driver->poll_timeout);
+ gettimeofday((void *)1,(void *)1); // atomic on
+
+ if (ret < 0) {
if (errno == EINTR) {
printf ("poll interrupt\n");
* Paul Davis <[email protected]> wrote:
> [ I trimmed the CC: line because several people there are on
> jackit-devel. ]
>
> >> compiles and boots fine. no observable change in xrun behaviour though.
> >
> >ok, so there's something else going on as well - or i missed an ioctl.
>
> i really don't think the ioctl's are relevant.
>
> consider what will happen if jackd does make a system call that causes
> a major delay (say, because of the BKL). we will get an xrun, yes, but
> this will cause jackd to stop the audio interface and restart.
> max_delay is not affected by this behaviour.
indeed. I'd exclude the ioctls at this point. But:
> as far as i can tell, the number reported by max_delay entirely (or
> almost entirely) represents problems in kernel scheduling, specifically
> with a combination of:
>
> a) handling the audio interface interrupt in time.
> b) marking the relevant jackd thread runnable
> c) context switching back to the relevant jackd thread
>
> things that jackd does once its running do not, it appear to me, have
> any impact on max_delay, which is based on the simple observation:
>
> "i was just woken, i expect to be awakened again in N usecs or
> less.
i dont yet see how this conclusion follows. Here's the poll() code
(simplified):
poll_enter = jack_get_microseconds ();
ret = poll (driver->pfd, nfds, driver->poll_timeout);
[...]
if (extra_fd < 0) {
if (driver->poll_next && poll_ret > driver->poll_next) {
*delayed_usecs = poll_ret - driver->poll_next;
}
driver->poll_last = poll_ret;
driver->poll_next = poll_ret + driver->period_usecs;
driver->engine->transport_cycle_start (driver->engine,
poll_ret);
}
is there a mechanism that ensures that the next poll() will be called
_before_ ->poll_next? Do you get a real hard ALSA xrun in that case or
something similar?
if it's possible to 'silently' overrun the next due interrupt (somewhat,
but not large enough overrun to cause a hard ALSA xrun) then the
processing delay will i believe be accounted as a 'wakeup delay'. In
that case to make the delayed_usecs value truly accurate, i'd at least
add this:
poll_enter = jack_get_microseconds ();
if (poll_enter > driver->poll_next) {
/*
* This processing cycle got delayed over
* the next due interrupt! Do not account this
* as a wakeup delay:
*/
driver->poll_next = 0;
}
but i'd also suggest to put in a counter into that branch so that this
condition doesnt get lost. In fact the Maximum Process Cycle stat from
Rui:
>> Maximum Delay . . . . . . . . . 6904 921 721 usecs
>> Maximum Process Cycle . . . . . 1449 1469 1590 usecs
seems to suggest that there can be significant processing delays? (if
Maximum Process Cycle is indeed the time spent from poll_ret to the next
poll_enter.)
Ingo
On Fri, 29 Oct 2004 22:42:20 +0200
Ingo Molnar <[email protected]> wrote:
> > compiles and boots fine. no observable change in xrun behaviour
> > though.
>
> do you compile jackd from sources? If yes then could you try the patch
> below? With this added, the kernel will produce a stackdump whenever
> jackd does an 'illegal' sleep.
>
> Also, could you do a small modification to kernel/sched.c and remove
> this line:
>
> send_sig(SIGUSR1, current, 1);
>
> just to make it easier to get Jack up and running. (by default an
> atomicity violation triggers a signal to make it easier to debug it in
> userspace, but i suspect there will be alot of such violations so jackd
> would stop all the time.)
[snip]
will do so. btw: i think i'm a bit confused right now. What debugging
features should i have enabled for this test?
flo
* Florian Schmidt <[email protected]> wrote:
> > do you compile jackd from sources? If yes then could you try the patch
> > below? With this added, the kernel will produce a stackdump whenever
> > jackd does an 'illegal' sleep.
> >
> > Also, could you do a small modification to kernel/sched.c and remove
> > this line:
> >
> > send_sig(SIGUSR1, current, 1);
> >
> > just to make it easier to get Jack up and running. (by default an
> > atomicity violation triggers a signal to make it easier to debug it in
> > userspace, but i suspect there will be alot of such violations so jackd
> > would stop all the time.)
>
> [snip]
>
> will do so. btw: i think i'm a bit confused right now. What debugging
> features should i have enabled for this test?
this particular one (atomicity-checking) is always-enabled if you have
the -RT patch applied (it's a really cheap check).
for the 'application-triggered tracing' facility we talked about earlier
is only active if LATENCY_TRACING is enabled. In that case to turn the
tracer on, call:
gettimeofday(0,1);
and to turn the tracer off and save the current trace into
/proc/latency_trace, call:
gettimeofday(0,0);
or apply the patch below - i've added the tracer bits too. I've added a
simple limit: all delays above 2 msec will be saved - you might want to
do a maximum search there or something. And dont forget to:
echo 2 > /proc/sys/kernel/trace_enabled
to activate the jackd-triggered kernel tracer.
Ingo
--- jack-audio-connection-kit-0.99.0/drivers/alsa/alsa_driver.c.orig
+++ jack-audio-connection-kit-0.99.0/drivers/alsa/alsa_driver.c
@@ -1161,6 +1161,7 @@ alsa_driver_wait (alsa_driver_t *driver,
unsigned int p_timed_out, c_timed_out;
unsigned int ci = 0;
unsigned int nfds;
+ int ret;
nfds = 0;
@@ -1194,7 +1195,20 @@ alsa_driver_wait (alsa_driver_t *driver,
poll_enter = jack_get_microseconds ();
- if (poll (driver->pfd, nfds, driver->poll_timeout) < 0) {
+ gettimeofday((void *)1,(void *)0); // atomic off
+
+ gettimeofday((void *)0,(void *)1); // start tracing
+
+ ret = poll (driver->pfd, nfds, driver->poll_timeout);
+
+ poll_ret = jack_get_microseconds ();
+
+ if (poll_ret - poll_enter > 2000)
+ gettimeofday((void *)0,(void *)0); // save trace
+
+ gettimeofday((void *)1,(void *)1); // atomic on
+
+ if (ret < 0) {
if (errno == EINTR) {
printf ("poll interrupt\n");
@@ -1214,8 +1228,6 @@ alsa_driver_wait (alsa_driver_t *driver,
}
- poll_ret = jack_get_microseconds ();
-
if (extra_fd < 0) {
if (driver->poll_next && poll_ret > driver->poll_next) {
*delayed_usecs = poll_ret - driver->poll_next;
On Fri, 2004-10-29 at 23:25 +0200, Ingo Molnar wrote:
> > will do so. btw: i think i'm a bit confused right now. What debugging
> > features should i have enabled for this test?
>
> this particular one (atomicity-checking) is always-enabled if you have
> the -RT patch applied (it's a really cheap check).
One more question, what do you recommend the priorities of the IRQ
threads be set to? AIUI for xrun-free operation with JACK, all that is
needed is to set the RT priorities of the soundcard IRQ thread highest,
followed by the JACK threads, then the other IRQ threads. Is this
correct?
Lee
* Lee Revell <[email protected]> wrote:
> On Fri, 2004-10-29 at 23:25 +0200, Ingo Molnar wrote:
> > > will do so. btw: i think i'm a bit confused right now. What debugging
> > > features should i have enabled for this test?
> >
> > this particular one (atomicity-checking) is always-enabled if you have
> > the -RT patch applied (it's a really cheap check).
>
> One more question, what do you recommend the priorities of the IRQ
> threads be set to? AIUI for xrun-free operation with JACK, all that
> is needed is to set the RT priorities of the soundcard IRQ thread
> highest, followed by the JACK threads, then the other IRQ threads. Is
> this correct?
correct. softirqs are not used by the sound subsystem so there's no
ksoftirqd dependency.
Ingo
On Fri, 2004-10-29 at 23:46 +0200, Ingo Molnar wrote:
> correct. softirqs are not used by the sound subsystem so there's no
> ksoftirqd dependency.
>
I only recommended this because it did seem to make a difference in a
previous version, probably due to debug overhead or something. It does
not seem like it should.
Lee
On Fri, 29 Oct 2004 23:25:45 +0200
Ingo Molnar <[email protected]> wrote:
> > will do so. btw: i think i'm a bit confused right now. What debugging
> > features should i have enabled for this test?
>
> this particular one (atomicity-checking) is always-enabled if you have
> the -RT patch applied (it's a really cheap check).
>
> for the 'application-triggered tracing' facility we talked about earlier
> is only active if LATENCY_TRACING is enabled. In that case to turn the
> tracer on, call:
[snip]
Ok,
.config attached,
running the patched jackd -R -P 60 -d alsa -p64 for a bit and provoking
xruns via window wiggling :)
mango:~# cat /proc/sys/kernel/trace_enabled
2
mango:~# cat /proc/sys/kernel/preempt_max_latency
75
mango:~# cat /proc/sys/kernel/preempt_thresh
0
mango:~# cat /proc/latency_trace
preemption latency trace v1.0.7 on 2.6.9-mm1-RT-V0.5.14-rt
-------------------------------------------------------
latency: 75 us, entries: 54 (54) | [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
-----------------
| task: jackd/1083, uid:1000 nice:0 policy:1 rt_prio:60
-----------------
=> started at: try_to_wake_up+0x5a/0x110 <c01151ca>
=> ended at: finish_task_switch+0x31/0xc0 <c0115691>
=======>
834 80000000 0.000ms (+0.000ms): (39) ((115))
834 80000000 0.000ms (+0.000ms): (1083) ((834))
834 80000000 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up)
834 80000000 0.000ms (+0.000ms): preempt_schedule (__up_write)
834 00000000 0.000ms (+0.000ms): preempt_schedule (up_write_mutex)
834 80000000 0.000ms (+0.000ms): __schedule (preempt_schedule)
834 80000000 0.000ms (+0.000ms): profile_hit (__schedule)
834 80000000 0.000ms (+0.000ms): sched_clock (__schedule)
1083 80000000 0.001ms (+0.000ms): __switch_to (__schedule)
1083 80000000 0.001ms (+0.000ms): (834) ((1083))
1083 80000000 0.001ms (+0.000ms): (115) ((39))
1083 80000000 0.002ms (+0.000ms): finish_task_switch (__schedule)
1083 80000000 0.002ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
1083 80000000 0.002ms (+0.000ms): (1083) ((39))
1083 00000000 0.003ms (+0.000ms): _mutex_unlock (kfree)
1083 00000000 0.003ms (+0.000ms): up_mutex (kfree)
1083 00000000 0.003ms (+0.000ms): up_write_mutex (kfree)
1083 00000000 0.003ms (+0.000ms): __up_write (up_write_mutex)
1083 00000000 0.003ms (+0.000ms): poll_freewait (sys_poll)
1083 00000000 0.003ms (+0.000ms): remove_wait_queue (poll_freewait)
1083 00000000 0.003ms (+0.000ms): _mutex_lock_irqsave (remove_wait_queue)
1083 00000000 0.003ms (+0.000ms): __mutex_lock (_mutex_lock_irqsave)
1083 00000000 0.003ms (+0.000ms): __might_sleep (__mutex_lock)
1083 00000000 0.004ms (+0.000ms): down_mutex (__mutex_lock)
1083 00000000 0.004ms (+0.000ms): down_write_mutex (__mutex_lock)
1083 00000000 0.004ms (+0.000ms): __might_sleep (down_write_mutex)
1083 00000000 0.004ms (+0.000ms): _mutex_unlock_irqrestore (remove_wait_queue)
1083 00000000 0.004ms (+0.000ms): up_mutex (remove_wait_queue)
1083 00000000 0.004ms (+0.000ms): up_write_mutex (remove_wait_queue)
1083 00000000 0.004ms (+0.000ms): __up_write (up_write_mutex)
1083 00000000 0.004ms (+0.000ms): fput (poll_freewait)
1083 00000000 0.005ms (+0.000ms): free_pages (poll_freewait)
1083 00000000 0.005ms (+0.000ms): __free_pages (poll_freewait)
1083 00000000 0.005ms (+0.000ms): free_hot_page (poll_freewait)
1083 00000000 0.006ms (+0.000ms): __free_pages_ok (free_hot_page)
1083 00000000 0.006ms (+0.000ms): free_pages_bulk (__free_pages_ok)
1083 00000000 0.006ms (+0.000ms): _mutex_lock_irqsave (free_pages_bulk)
1083 00000000 0.006ms (+0.000ms): __mutex_lock (_mutex_lock_irqsave)
1083 00000000 0.007ms (+0.000ms): __might_sleep (__mutex_lock)
1083 00000000 0.007ms (+0.000ms): down_mutex (__mutex_lock)
1083 00000000 0.007ms (+0.000ms): down_write_mutex (__mutex_lock)
1083 00000000 0.007ms (+0.000ms): __might_sleep (down_write_mutex)
1083 00000000 0.007ms (+0.000ms): bad_range (free_pages_bulk)
1083 00000000 0.007ms (+0.000ms): bad_range (free_pages_bulk)
1083 00000000 0.008ms (+0.000ms): _mutex_unlock_irqrestore (free_pages_bulk)
1083 00000000 0.008ms (+0.000ms): up_mutex (free_pages_bulk)
1083 00000000 0.008ms (+0.000ms): up_write_mutex (free_pages_bulk)
1083 00000000 0.008ms (+0.004ms): __up_write (up_write_mutex)
1083 00000000 0.013ms (+0.000ms): sys_gettimeofday (syscall_call)
1083 00000000 0.013ms (+0.000ms): user_trace_stop (sys_gettimeofday)
1083 80000000 0.013ms (+0.000ms): user_trace_stop (sys_gettimeofday)
1083 80000000 0.013ms (+0.000ms): update_max_trace (user_trace_stop)
1083 80000000 0.014ms (+0.000ms): _mmx_memcpy (update_max_trace)
1083 80000000 0.014ms (+0.000ms): kernel_fpu_begin (_mmx_memcpy)
mango:~# chrt -p `pidof "IRQ 3"`
pid 118's current scheduling policy: SCHED_FIFO
pid 118's current scheduling priority: 99
mango:~# ps aux|grep jackd
tapas 1080 4.2 3.6 28244 28220 pts/1 SLl+ 00:22 0:18 jackd -R -P 60 -d alsa -p 64
mango:~# chrt -p 1080
pid 1080's current scheduling policy: SCHED_OTHER
pid 1080's current scheduling priority: 0
mango:~# chrt -p 1081
pid 1081's current scheduling policy: SCHED_OTHER
pid 1081's current scheduling priority: 0
mango:~# chrt -p 1082
pid 1082's current scheduling policy: SCHED_FIFO
pid 1082's current scheduling priority: 70
mango:~# chrt -p 1083
pid 1083's current scheduling policy: SCHED_FIFO
pid 1083's current scheduling priority: 60
and here's some syslog stuff:
Oct 30 00:20:32 mango kernel: Realtime LSM initialized (group 1002, mlock=1)
Oct 30 00:20:35 mango kernel: IRQ#3 thread RT prio: 43.
Oct 30 00:20:38 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:20:38 mango kernel: [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:20:38 mango kernel: [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:20:38 mango kernel: [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:20:38 mango kernel: ---------------------------
Oct 30 00:20:38 mango kernel: | preempt count: 00000001 ]
Oct 30 00:20:38 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:20:38 mango kernel: ----------------------------------------
Oct 30 00:20:38 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:20:38 mango kernel: .....[show_registers+99/464] .. ( <= dump_stack+0x23/0x30)
Oct 30 00:20:38 mango kernel:
Oct 30 00:20:47 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:20:47 mango kernel: [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:20:47 mango kernel: [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:20:47 mango kernel: [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:20:47 mango kernel: ---------------------------
Oct 30 00:20:47 mango kernel: | preempt count: 00000001 ]
Oct 30 00:20:47 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:20:47 mango kernel: ----------------------------------------
Oct 30 00:20:47 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:20:47 mango kernel: .....[show_registers+99/464] .. ( <= dump_stack+0x23/0x30)
Oct 30 00:20:47 mango kernel:
Oct 30 00:20:51 mango kernel: (IRQ 3/118/CPU#0): new 4 us maximum-latency wakeup.
Oct 30 00:20:51 mango kernel: (bash/867/CPU#0): new 6 us maximum-latency wakeup.
Oct 30 00:20:52 mango kernel: (bash/1035/CPU#0): new 7 us maximum-latency wakeup.
Oct 30 00:20:52 mango kernel: (ksoftirqd/0/2/CPU#0): new 8 us maximum-latency wakeup.
Oct 30 00:20:52 mango kernel: (ksoftirqd/0/2/CPU#0): new 13 us maximum-latency wakeup.
Oct 30 00:21:11 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:11 mango kernel: (IRQ 12/15/CPU#0): new 19 us maximum-latency wakeup.
Oct 30 00:21:11 mango kernel: [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:11 mango kernel: [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:11 mango kernel: [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:11 mango kernel: ---------------------------
Oct 30 00:21:11 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:11 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:11 mango kernel: ----------------------------------------
Oct 30 00:21:11 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:11 mango kernel: .....[show_registers+99/464] .. ( <= dump_stack+0x23/0x30)
Oct 30 00:21:11 mango kernel:
Oct 30 00:21:11 mango kernel: (IRQ 12/15/CPU#0): new 75 us maximum-latency wakeup.
Oct 30 00:21:11 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:11 mango kernel: [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:11 mango kernel: [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:11 mango kernel: [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:11 mango kernel: ---------------------------
Oct 30 00:21:11 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:11 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:11 mango kernel: ----------------------------------------
Oct 30 00:21:11 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:11 mango kernel: .....[show_registers+99/464] .. ( <= dump_stack+0x23/0x30)
Oct 30 00:21:11 mango kernel:
Oct 30 00:21:11 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:11 mango kernel: [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:11 mango kernel: [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:11 mango kernel: [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:11 mango kernel: ---------------------------
Oct 30 00:21:11 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:11 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:11 mango kernel: ----------------------------------------
Oct 30 00:21:11 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:11 mango kernel: .....[show_registers+99/464] .. ( <= dump_stack+0x23/0x30)
Oct 30 00:21:11 mango kernel:
Oct 30 00:21:11 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:11 mango kernel: [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:11 mango kernel: [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:11 mango kernel: [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:11 mango kernel: ---------------------------
Oct 30 00:21:11 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:11 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:11 mango kernel: ----------------------------------------
Oct 30 00:21:11 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:11 mango kernel: .....[show_registers+99/464] .. ( <= dump_stack+0x23/0x30)
Oct 30 00:21:11 mango kernel:
Oct 30 00:21:11 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:11 mango kernel: [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:11 mango kernel: [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:11 mango kernel: [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:11 mango kernel: ---------------------------
Oct 30 00:21:11 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:11 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:11 mango kernel: ----------------------------------------
Oct 30 00:21:11 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:11 mango kernel: .....[show_registers+99/464] .. ( <= dump_stack+0x23/0x30)
Oct 30 00:21:11 mango kernel:
Oct 30 00:21:11 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:11 mango kernel: [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:11 mango kernel: [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:11 mango kernel: [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:11 mango kernel: ---------------------------
Oct 30 00:21:11 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:11 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:11 mango kernel: ----------------------------------------
Oct 30 00:21:11 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:11 mango kernel: .....[show_registers+99/464] .. ( <= dump_stack+0x23/0x30)
Oct 30 00:21:11 mango kernel:
Oct 30 00:21:12 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:12 mango kernel: [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:12 mango kernel: [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:12 mango kernel: [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:12 mango kernel: ---------------------------
Oct 30 00:21:12 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:12 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:12 mango kernel: ----------------------------------------
Oct 30 00:21:12 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:12 mango kernel: .....[show_registers+99/464] .. ( <= dump_stack+0x23/0x30)
Oct 30 00:21:12 mango kernel:
Oct 30 00:21:13 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:13 mango kernel: [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:13 mango kernel: [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:13 mango kernel: [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:13 mango kernel: ---------------------------
Oct 30 00:21:13 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:13 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:13 mango kernel: ----------------------------------------
Oct 30 00:21:13 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:13 mango kernel: .....[show_registers+99/464] .. ( <= dump_stack+0x23/0x30)
Oct 30 00:21:13 mango kernel:
Oct 30 00:21:31 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:31 mango kernel: [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:31 mango kernel: [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:31 mango kernel: [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:31 mango kernel: ---------------------------
Oct 30 00:21:31 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:31 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:31 mango kernel: ----------------------------------------
Oct 30 00:21:31 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:31 mango kernel: .....[show_registers+99/464] .. ( <= dump_stack+0x23/0x30)
Oct 30 00:21:31 mango kernel:
Oct 30 00:21:32 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:32 mango kernel: [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:32 mango kernel: [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:32 mango kernel: [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:32 mango kernel: ---------------------------
Oct 30 00:21:32 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:32 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:32 mango kernel: ----------------------------------------
Oct 30 00:21:32 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:32 mango kernel: .....[show_registers+99/464] .. ( <= dump_stack+0x23/0x30)
Oct 30 00:21:32 mango kernel:
Oct 30 00:21:32 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:32 mango kernel: [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:32 mango kernel: [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:32 mango kernel: [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:32 mango kernel: ---------------------------
Oct 30 00:21:32 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:32 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:32 mango kernel: ----------------------------------------
Oct 30 00:21:32 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:32 mango kernel: .....[show_registers+99/464] .. ( <= dump_stack+0x23/0x30)
Oct 30 00:21:32 mango kernel:
Oct 30 00:21:32 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:32 mango kernel: [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:32 mango kernel: [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:32 mango kernel: [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:32 mango kernel: ---------------------------
Oct 30 00:21:32 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:32 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:32 mango kernel: ----------------------------------------
Oct 30 00:21:32 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:32 mango kernel: .....[show_registers+99/464] .. ( <= dump_stack+0x23/0x30)
Oct 30 00:21:32 mango kernel:
Oct 30 00:21:34 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:34 mango kernel: [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:34 mango kernel: [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:34 mango kernel: [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:34 mango kernel: ---------------------------
Oct 30 00:21:34 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:34 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:34 mango kernel: ----------------------------------------
Oct 30 00:21:34 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:34 mango kernel: .....[show_registers+99/464] .. ( <= dump_stack+0x23/0x30)
Oct 30 00:21:34 mango kernel:
Oct 30 00:21:35 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:35 mango kernel: [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:35 mango kernel: [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:35 mango kernel: [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:35 mango kernel: ---------------------------
Oct 30 00:21:35 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:35 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:35 mango kernel: ----------------------------------------
Oct 30 00:21:35 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:35 mango kernel: .....[show_registers+99/464] .. ( <= dump_stack+0x23/0x30)
Oct 30 00:21:35 mango kernel:
Oct 30 00:21:37 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:21:37 mango kernel: [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:21:37 mango kernel: [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:21:37 mango kernel: [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:21:37 mango kernel: ---------------------------
Oct 30 00:21:37 mango kernel: | preempt count: 00000001 ]
Oct 30 00:21:37 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:21:37 mango kernel: ----------------------------------------
Oct 30 00:21:37 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:21:37 mango kernel: .....[show_registers+99/464] .. ( <= dump_stack+0x23/0x30)
Oct 30 00:21:37 mango kernel:
Oct 30 00:22:09 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:22:09 mango kernel: [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:22:09 mango kernel: [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:22:09 mango kernel: [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:22:09 mango kernel: ---------------------------
Oct 30 00:22:09 mango kernel: | preempt count: 00000001 ]
Oct 30 00:22:09 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:22:09 mango kernel: ----------------------------------------
Oct 30 00:22:09 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:22:09 mango kernel: .....[show_registers+99/464] .. ( <= dump_stack+0x23/0x30)
Oct 30 00:22:09 mango kernel:
Oct 30 00:22:10 mango kernel: jackd:1019 userspace BUG: scheduling in user-atomic context!
Oct 30 00:22:10 mango kernel: [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:22:10 mango kernel: [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:22:10 mango kernel: [kernel_config_data+5091/6569] io_schedule+0x13/0x20 (8)
Oct 30 00:22:10 mango kernel: [do_hardirq+104/304] sync_page+0x38/0x50 (12)
Oct 30 00:22:10 mango kernel: [kernel_config_data+5981/6569] __wait_on_bit_lock+0x5d/0x70 (32)
Oct 30 00:22:10 mango kernel: [init_irq_proc+137/176] __lock_page+0x89/0xa0 (92)
Oct 30 00:22:10 mango kernel: [__generic_file_aio_read+402/544] filemap_nopage+0x2b2/0x3e0 (60)
Oct 30 00:22:10 mango kernel: [do_no_page+74/848] do_no_page+0xba/0x390 (68)
Oct 30 00:22:10 mango kernel: [handle_mm_fault+207/416] handle_mm_fault+0xef/0x180 (48)
Oct 30 00:22:10 mango kernel: [get_user_pages+169/912] get_user_pages+0x139/0x380 (60)
Oct 30 00:22:10 mango kernel: [make_pages_present+108/192] make_pages_present+0x9c/0x110 (56)
Oct 30 00:22:10 mango kernel: [do_mmap_pgoff+699/1808] do_mmap_pgoff+0x47b/0x700 (88)
Oct 30 00:22:10 mango kernel: [old_mmap+149/304] old_mmap+0xc5/0xf0 (96)
Oct 30 00:22:10 mango kernel: [irq_entries_start+3/128] syscall_call+0x7/0xb (-8124)
Oct 30 00:22:10 mango kernel: ---------------------------
Oct 30 00:22:10 mango kernel: | preempt count: 00000001 ]
Oct 30 00:22:10 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:22:10 mango kernel: ----------------------------------------
Oct 30 00:22:10 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:22:10 mango kernel: .....[show_registers+99/464] .. ( <= dump_stack+0x23/0x30)
Oct 30 00:22:10 mango kernel:
Oct 30 00:24:40 mango kernel: jackd:1083 userspace BUG: scheduling in user-atomic context!
Oct 30 00:24:40 mango kernel: [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:24:40 mango kernel: [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:24:40 mango kernel: [irq_entries_start+43/128] work_resched+0x6/0x17 (-8124)
Oct 30 00:24:40 mango kernel: ---------------------------
Oct 30 00:24:40 mango kernel: | preempt count: 00000001 ]
Oct 30 00:24:40 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:24:40 mango kernel: ----------------------------------------
Oct 30 00:24:40 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:24:40 mango kernel: .....[show_registers+99/464] .. ( <= dump_stack+0x23/0x30)
Oct 30 00:24:40 mango kernel:
Oct 30 00:28:29 mango kernel: (IRQ 3/118/CPU#0): new 205 us maximum-latency wakeup.
[ I trimmed the CC: line because several people there are on
jackit-devel. ]
>> compiles and boots fine. no observable change in xrun behaviour though.
>
>ok, so there's something else going on as well - or i missed an ioctl.
i really don't think the ioctl's are relevant.
consider what will happen if jackd does make a system call that causes
a major delay (say, because of the BKL). we will get an xrun, yes, but
this will cause jackd to stop the audio interface and
restart. max_delay is not affected by this behaviour.
as far as i can tell, the number reported by max_delay entirely (or
almost entirely) represents problems in kernel scheduling, specifically
with a combination of:
a) handling the audio interface interrupt in time.
b) marking the relevant jackd thread runnable
c) context switching back to the relevant jackd thread
things that jackd does once its running do not, it appear to me, have
any impact on max_delay, which is based on the simple observation:
"i was just woken, i expect to be awakened again in N usecs or
less.
--p
On Sat, 30 Oct 2004 00:31:22 +0200
Florian Schmidt <[email protected]> wrote:
> Ok,
>
> .config attached,
now it is :)
> Oct 30 00:28:29 mango kernel: (IRQ 3/118/CPU#0): new 205 us maximum-latency wakeup.
i could as well include this one
preemption latency trace v1.0.7 on 2.6.9-mm1-RT-V0.5.14-rt
-------------------------------------------------------
latency: 205 us, entries: 36 (36) | [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
-----------------
| task: jackd/1083, uid:1000 nice:0 policy:1 rt_prio:60
-----------------
=> started at: try_to_wake_up+0x5a/0x110 <c01151ca>
=> ended at: finish_task_switch+0x31/0xc0 <c0115691>
=======>
768 80000000 0.000ms (+0.000ms): (39) ((115))
768 80000000 0.000ms (+0.000ms): (1083) ((768))
768 80000000 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up)
768 80000000 0.000ms (+0.000ms): preempt_schedule (__up_write)
768 00000000 0.000ms (+0.000ms): preempt_schedule (up_write_mutex)
768 80000000 0.000ms (+0.000ms): __schedule (preempt_schedule)
768 80000000 0.000ms (+0.000ms): profile_hit (__schedule)
768 80000000 0.000ms (+0.000ms): sched_clock (__schedule)
1083 80000000 0.001ms (+0.000ms): __switch_to (__schedule)
1083 80000000 0.002ms (+0.000ms): (768) ((1083))
1083 80000000 0.002ms (+0.000ms): (115) ((39))
1083 80000000 0.002ms (+0.000ms): finish_task_switch (__schedule)
1083 80000000 0.002ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
1083 80000000 0.002ms (+0.000ms): (1083) ((39))
1083 00000000 0.003ms (+0.000ms): bad_range (free_pages_bulk)
1083 00000000 0.003ms (+0.000ms): bad_range (free_pages_bulk)
1083 00000000 0.003ms (+0.000ms): _mutex_unlock_irqrestore (free_pages_bulk)
1083 00000000 0.003ms (+0.000ms): up_mutex (free_pages_bulk)
1083 00000000 0.003ms (+0.000ms): up_write_mutex (free_pages_bulk)
1083 00000000 0.003ms (+0.000ms): __up_write (up_write_mutex)
1083 80000000 0.004ms (+0.000ms): wake_up_process_mutex (__up_write)
1083 80000000 0.004ms (+0.000ms): try_to_wake_up (wake_up_process_mutex)
1083 80000000 0.004ms (+0.000ms): task_rq_lock (try_to_wake_up)
1083 80000000 0.004ms (+0.000ms): activate_task (try_to_wake_up)
1083 80000000 0.004ms (+0.000ms): sched_clock (activate_task)
1083 80000000 0.004ms (+0.000ms): recalc_task_prio (activate_task)
1083 80000000 0.004ms (+0.000ms): effective_prio (recalc_task_prio)
1083 80000000 0.005ms (+0.000ms): enqueue_task (activate_task)
1083 80000000 0.005ms (+0.000ms): (115) ((39))
1083 80000000 0.005ms (+0.004ms): (834) ((1083))
1083 00000000 0.010ms (+0.000ms): sys_gettimeofday (syscall_call)
1083 00000000 0.010ms (+0.000ms): user_trace_stop (sys_gettimeofday)
1083 80000000 0.010ms (+0.000ms): user_trace_stop (sys_gettimeofday)
1083 80000000 0.010ms (+0.000ms): update_max_trace (user_trace_stop)
1083 80000000 0.011ms (+0.000ms): _mmx_memcpy (update_max_trace)
1083 80000000 0.011ms (+0.000ms): kernel_fpu_begin (_mmx_memcpy)
and another one:
preemption latency trace v1.0.7 on 2.6.9-mm1-RT-V0.5.14-rt
-------------------------------------------------------
latency: 225 us, entries: 36 (36) | [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
-----------------
| task: jackd/1083, uid:1000 nice:0 policy:1 rt_prio:60
-----------------
=> started at: try_to_wake_up+0x5a/0x110 <c01151ca>
=> ended at: finish_task_switch+0x31/0xc0 <c0115691>
=======>
637 80000000 0.000ms (+0.000ms): (39) ((116))
637 80000000 0.000ms (+0.000ms): (1083) ((637))
637 80000000 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up)
637 80000000 0.000ms (+0.000ms): preempt_schedule (__up_write)
637 00000000 0.000ms (+0.000ms): preempt_schedule (up_write_mutex)
637 80000000 0.000ms (+0.000ms): __schedule (preempt_schedule)
637 80000000 0.000ms (+0.000ms): profile_hit (__schedule)
637 80000000 0.000ms (+0.001ms): sched_clock (__schedule)
1083 80000000 0.002ms (+0.000ms): __switch_to (__schedule)
1083 80000000 0.002ms (+0.000ms): (637) ((1083))
1083 80000000 0.002ms (+0.000ms): (116) ((39))
1083 80000000 0.002ms (+0.000ms): finish_task_switch (__schedule)
1083 80000000 0.002ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
1083 80000000 0.002ms (+0.000ms): (1083) ((39))
1083 00000000 0.003ms (+0.000ms): bad_range (free_pages_bulk)
1083 00000000 0.003ms (+0.000ms): bad_range (free_pages_bulk)
1083 00000000 0.003ms (+0.000ms): _mutex_unlock_irqrestore (free_pages_bulk)
1083 00000000 0.003ms (+0.000ms): up_mutex (free_pages_bulk)
1083 00000000 0.004ms (+0.000ms): up_write_mutex (free_pages_bulk)
1083 00000000 0.004ms (+0.000ms): __up_write (up_write_mutex)
1083 80000000 0.004ms (+0.000ms): wake_up_process_mutex (__up_write)
1083 80000000 0.004ms (+0.000ms): try_to_wake_up (wake_up_process_mutex)
1083 80000000 0.004ms (+0.000ms): task_rq_lock (try_to_wake_up)
1083 80000000 0.004ms (+0.000ms): activate_task (try_to_wake_up)
1083 80000000 0.004ms (+0.000ms): sched_clock (activate_task)
1083 80000000 0.004ms (+0.000ms): recalc_task_prio (activate_task)
1083 80000000 0.004ms (+0.000ms): effective_prio (recalc_task_prio)
1083 80000000 0.004ms (+0.000ms): enqueue_task (activate_task)
1083 80000000 0.005ms (+0.000ms): (115) ((39))
1083 80000000 0.005ms (+0.006ms): (782) ((1083))
1083 00000000 0.011ms (+0.000ms): sys_gettimeofday (syscall_call)
1083 00000000 0.011ms (+0.000ms): user_trace_stop (sys_gettimeofday)
1083 80000000 0.012ms (+0.000ms): user_trace_stop (sys_gettimeofday)
1083 80000000 0.012ms (+0.000ms): update_max_trace (user_trace_stop)
1083 80000000 0.012ms (+0.000ms): _mmx_memcpy (update_max_trace)
1083 80000000 0.013ms (+0.000ms): kernel_fpu_begin (_mmx_memcpy)
Oct 30 00:48:45 mango kernel: (IRQ 3/118/CPU#0): new 225 us maximum-latency wakeup.
Oct 30 00:49:15 mango kernel: jackd:1083 userspace BUG: scheduling in user-atomic context!
Oct 30 00:49:15 mango kernel: [show_registers+99/464] dump_stack+0x23/0x30 (20)
Oct 30 00:49:15 mango kernel: [kernel_config_data+2090/6569] schedule+0x7a/0x120 (36)
Oct 30 00:49:15 mango kernel: [Bad_offset+13/23] down_write_mutex+0xc2/0x1a0 (48)
Oct 30 00:49:15 mango kernel: [do_posix_clock_monotonic_gettime_parts+83/96] __mutex_lock+0x43/0x50 (24)
Oct 30 00:49:15 mango kernel: [do_posix_clock_nonanosleep+13/16] _mutex_lock_irqsave+0x1d/0x30 (16)
Oct 30 00:49:15 mango kernel: [stop_tty+2/144] tty_ldisc_try+0x22/0x60 (24)
Oct 30 00:49:15 mango kernel: [stop_tty+88/144] tty_ldisc_ref_wait+0x18/0xb0 (84)
Oct 30 00:49:15 mango kernel: [release_dev+865/1984] tty_write+0x91/0x230 (80)
Oct 30 00:49:15 mango kernel: [sys_llseek+90/176] vfs_write+0xca/0x140 (44)
Oct 30 00:49:15 mango kernel: [vfs_read+16/320] sys_write+0x50/0x80 (44)
Oct 30 00:49:15 mango kernel: [irq_entries_start+3/128] syscall_call+0x7/0xb (-8124)
Oct 30 00:49:15 mango kernel: ---------------------------
Oct 30 00:49:15 mango kernel: | preempt count: 00000001 ]
Oct 30 00:49:15 mango kernel: | 1-level deep critical section nesting:
Oct 30 00:49:15 mango kernel: ----------------------------------------
Oct 30 00:49:15 mango kernel: .. [l_show+173/272] .... print_traces+0x1d/0x90
Oct 30 00:49:15 mango kernel: .....[show_registers+99/464] .. ( <= dump_stack+0x23/0x30)
Oct 30 00:49:15 mango kernel:
On Fri, 2004-10-29 at 23:46 +0200, Ingo Molnar wrote:
> * Lee Revell <[email protected]> wrote:
>
> > On Fri, 2004-10-29 at 23:25 +0200, Ingo Molnar wrote:
> > > > will do so. btw: i think i'm a bit confused right now. What debugging
> > > > features should i have enabled for this test?
> > >
> > > this particular one (atomicity-checking) is always-enabled if you have
> > > the -RT patch applied (it's a really cheap check).
> >
> > One more question, what do you recommend the priorities of the IRQ
> > threads be set to? AIUI for xrun-free operation with JACK, all that
> > is needed is to set the RT priorities of the soundcard IRQ thread
> > highest, followed by the JACK threads, then the other IRQ threads. Is
> > this correct?
>
> correct. softirqs are not used by the sound subsystem so there's no
> ksoftirqd dependency.
>
OK well I set all IRQ threads to SCHED_OTHER except the soundcard, which
is SCHED_FIFO, and I still get a LOT of xruns, compared to zero xruns
over tens of millions of cycles with T3.
rlrevell@mindpipe:~$ for p in `ps auxww | grep IRQ | grep -v grep | awk
'{print $2}'`; do chrt -p $p ;done
pid 647's current scheduling policy: SCHED_OTHER
pid 647's current scheduling priority: 0
pid 655's current scheduling policy: SCHED_OTHER
pid 655's current scheduling priority: 0
pid 678's current scheduling policy: SCHED_OTHER
pid 678's current scheduling priority: 0
pid 693's current scheduling policy: SCHED_OTHER
pid 693's current scheduling priority: 0
pid 831's current scheduling policy: SCHED_OTHER
pid 831's current scheduling priority: 0
pid 835's current scheduling policy: SCHED_FIFO <-- soundcard irq
pid 835's current scheduling priority: 90
Here is the dmesg output. It looks like the problem could be related to
jackd's printing from the realtime thread. But, this has to be the
kernel's fault on some level, because with an earlier version I get no
xruns.
jackd:1726 userspace BUG: scheduling in user-atomic context!
[<c01069fc>] dump_stack+0x1c/0x20 (20)
[<c0283e60>] schedule+0x70/0x100 (24)
[<c010639b>] work_resched+0x6/0x17 (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e833>] .... print_traces+0x13/0x50
.....[<c01069fc>] .. ( <= dump_stack+0x1c/0x20)
jackd:1726 userspace BUG: scheduling in user-atomic context!
[<c01069fc>] dump_stack+0x1c/0x20 (20)
[<c0283e60>] schedule+0x70/0x100 (24)
[<c0284e2c>] _down_write+0xcc/0x170 (32)
[<c0113413>] lock_kernel+0x23/0x30 (16)
[<c01f2570>] tty_write+0x170/0x230 (64)
[<c01548ec>] vfs_write+0xbc/0x110 (36)
[<c01549f1>] sys_write+0x41/0x70 (44)
[<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e833>] .... print_traces+0x13/0x50
.....[<c01069fc>] .. ( <= dump_stack+0x1c/0x20)
jackd:1726 userspace BUG: scheduling in user-atomic context!
[<c01069fc>] dump_stack+0x1c/0x20 (20)
[<c0283e60>] schedule+0x70/0x100 (24)
[<c0119efa>] do_exit+0x29a/0x500 (24)
[<c011a196>] sys_exit+0x16/0x20 (12)
[<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e833>] .... print_traces+0x13/0x50
.....[<c01069fc>] .. ( <= dump_stack+0x1c/0x20)
jackd:1731 userspace BUG: scheduling in user-atomic context!
[<c01069fc>] dump_stack+0x1c/0x20 (20)
[<c0283e60>] schedule+0x70/0x100 (24)
[<c0119efa>] do_exit+0x29a/0x500 (24)
[<c011a196>] sys_exit+0x16/0x20 (12)
[<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e833>] .... print_traces+0x13/0x50
.....[<c01069fc>] .. ( <= dump_stack+0x1c/0x20)
jackd:1736 userspace BUG: scheduling in user-atomic context!
[<c01069fc>] dump_stack+0x1c/0x20 (20)
[<c0283e60>] schedule+0x70/0x100 (24)
[<c0119efa>] do_exit+0x29a/0x500 (24)
[<c011a196>] sys_exit+0x16/0x20 (12)
[<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e833>] .... print_traces+0x13/0x50
.....[<c01069fc>] .. ( <= dump_stack+0x1c/0x20)
jackd:1775 userspace BUG: scheduling in user-atomic context!
[<c01069fc>] dump_stack+0x1c/0x20 (20)
[<c0283e60>] schedule+0x70/0x100 (24)
[<c0119efa>] do_exit+0x29a/0x500 (24)
[<c011a196>] (ksoftirqd/0/2/CPU#0): new 12 us maximum-latency wakeup.
(ksoftirqd/0/2/CPU#0): new 15 us maximum-latency wakeup.
(ksoftirqd/0/2/CPU#0): new 22 us maximum-latency wakeup.
(ksoftirqd/0/2/CPU#0): new 31 us maximum-latency wakeup.
(ksoftirqd/0/2/CPU#0): new 32 us maximum-latency wakeup.
(IRQ 1/693/CPU#0): new 39 us maximum-latency wakeup.
jackd:1787 userspace BUG: scheduling in user-atomic context!
[<c01069fc>] dump_stack+0x1c/0x20 (20)
[<c0283e60>] schedule+0x70/0x100 (24)
[<c0119efa>] do_exit+0x29a/0x500 (24)
[<c011a196>] sys_exit+0x16/0x20 (12)
[<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e833>] .... print_traces+0x13/0x50
.....[<c01069fc>] .. ( <= dump_stack+0x1c/0x20)
(ksoftirqd/0/2/CPU#0): new 42 us maximum-latency wakeup.
(desched/0/3/CPU#0): new 43 us maximum-latency wakeup.
(IRQ 15/678/CPU#0): new 44 us maximum-latency wakeup.
(IRQ 11/831/CPU#0): new 45 us maximum-latency wakeup.
(IRQ 11/831/CPU#0): new 52 us maximum-latency wakeup.
(IRQ 11/831/CPU#0): new 55 us maximum-latency wakeup.
jackd:1846 userspace BUG: scheduling in user-atomic context!
[<c01069fc>] dump_stack+0x1c/0x20 (20)
[<c0283e60>] schedule+0x70/0x100 (24)
[<c0284acb>] schedule_timeout+0xbb/0xc0 (80)
[<c012f11f>] futex_wait+0x10f/0x190 (168)
[<c012f406>] do_futex+0x36/0x80 (32)
[<c012f51a>] sys_futex+0xca/0xe0 (68)
[<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e833>] .... print_traces+0x13/0x50
.....[<c01069fc>] .. ( <= dump_stack+0x1c/0x20)
jackd:1846 userspace BUG: scheduling in user-atomic context!
[<c01069fc>] dump_stack+0x1c/0x20 (20)
[<c0283e60>] schedule+0x70/0x100 (24)
[<c0119efa>] do_exit+0x29a/0x500 (24)
[<c011a196>] sys_exit+0x16/0x20 (12)
[<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e833>] .... print_traces+0x13/0x50
.....[<c01069fc>] .. ( <= dump_stack+0x1c/0x20)
jackd:1854 userspace BUG: scheduling in user-atomic context!
[<c01069fc>] dump_stack+0x1c/0x20 (20)
[<c0283e60>] schedule+0x70/0x100 (24)
[<c0119efa>] do_exit+0x29a/0x500 (24)
[<c011a196>] sys_exit+0x16/0x20 (12)
[<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e833>] .... print_traces+0x13/0x50
.....[<c01069fc>] .. ( <= dump_stack+0x1c/0x20)
Lee
On Fri, 2004-10-29 at 23:46 +0200, Ingo Molnar wrote:
> > One more question, what do you recommend the priorities of the IRQ
> > threads be set to? AIUI for xrun-free operation with JACK, all that
> > is needed is to set the RT priorities of the soundcard IRQ thread
> > highest, followed by the JACK threads, then the other IRQ threads. Is
> > this correct?
>
> correct. softirqs are not used by the sound subsystem so there's no
> ksoftirqd dependency.
>
I don't see much point in latency testing this one yet; I have had to go
for the reset button twice in the past hour. Web surfing and a kernel
compile kill it pretty quickly.
Lockups aside, it's not necessary to use JACK to demonstrate the
problem, playing AVIs with mplayer will skip.
As a final test I am recompiling with all debug options disabled.
Lee
On Fri, 2004-10-29 at 23:25 +0200, Ingo Molnar wrote:
> this particular one (atomicity-checking) is always-enabled if you have
> the -RT patch applied (it's a really cheap check).
I tried this patch with all DEBUG options disabled, and this reduces the
xruns, but I still get a few. This is what I get in the logs when
running jackd with your patch:
jackd:1590 userspace BUG: scheduling in user-atomic context!
[<c02733eb>] schedule+0x6b/0xf0 (8)
[<c0105d3b>] work_resched+0x6/0x17 (24)
jackd:1590 userspace BUG: scheduling in user-atomic context!
[<c02733eb>] schedule+0x6b/0xf0 (8)
[<c0105d3b>] work_resched+0x6/0x17 (24)
jackd:1590 userspace BUG: scheduling in user-atomic context!
[<c02733eb>] schedule+0x6b/0xf0 (8)
[<c0105d3b>] work_resched+0x6/0x17 (24)
jackd:1590 userspace BUG: scheduling in user-atomic context!
[<c02733eb>] schedule+0x6b/0xf0 (8)
[<c0105d3b>] work_resched+0x6/0x17 (24)
jackd:1590 userspace BUG: scheduling in user-atomic context!
[<c02733eb>] schedule+0x6b/0xf0 (8)
[<c0105d3b>] work_resched+0x6/0x17 (24)
jackd:1590 userspace BUG: scheduling in user-atomic context!
[<c02733eb>] schedule+0x6b/0xf0 (8)
[<c0105d3b>] work_resched+0x6/0x17 (24)
jackd:1590 userspace BUG: scheduling in user-atomic context!
[<c02733eb>] schedule+0x6b/0xf0 (8)
[<c0105d3b>] work_resched+0x6/0x17 (24)
jackd:1590 userspace BUG: scheduling in user-atomic context!
[<c02733eb>] schedule+0x6b/0xf0 (8)
[<c0105d3b>] work_resched+0x6/0x17 (24)
Lee
>but i'd also suggest to put in a counter into that branch so that this
>condition doesnt get lost. In fact the Maximum Process Cycle stat from
>Rui:
>
>>> Maximum Delay . . . . . . . . . 6904 921 721 usecs
>>> Maximum Process Cycle . . . . . 1449 1469 1590 usecs
>
>seems to suggest that there can be significant processing delays? (if
>Maximum Process Cycle is indeed the time spent from poll_ret to the next
>poll_enter.)
IIRC, Rui was running with -p128, which at 48000kHz is 2600usecs (and
longer at 44100kHz). If the maximum process cycle was on the order of
1500usecs, that leaves nearly 1ms until the next interrupt is
due. Unless jackd was held up between computing the process cycle time
and entering poll, it doesn't seem that the process cycle ever gets
close to the interrupt interval duration.
So I don't think that delays caused *during* jackd's processing cycle
are involved here. However, I agree that your suggestion to check for
this before computing max_delay is probably wise in general.
--p
On Fri, 2004-10-29 at 23:25 +0200, Ingo Molnar wrote:
> this particular one (atomicity-checking) is always-enabled if you have
> the -RT patch applied (it's a really cheap check).
I get these as well. Basically I think these are the exact same results
Florian had.
jackd:1646 userspace BUG: scheduling in user-atomic context!
[<c02733eb>] schedule+0x6b/0xf0 (8)
[<c027431d>] _down_write+0xcd/0x130 (24)
[<c0111e1e>] lock_kernel+0x1e/0x30 (32)
[<c01e7ad9>] tty_write+0x189/0x240 (16)
[<c01ecf20>] write_chan+0x0/0x210 (28)
[<c014ea4a>] vfs_write+0xba/0x100 (32)
[<c014eb3d>] sys_write+0x3d/0x70 (36)
[<c0105d07>] syscall_call+0x7/0xb (40)
jackd:1646 userspace BUG: scheduling in user-atomic context!
[<c02733eb>] schedule+0x6b/0xf0 (8)
[<c027431d>] _down_write+0xcd/0x130 (24)
[<c0111e1e>] lock_kernel+0x1e/0x30 (32)
[<c01e7ad9>] tty_write+0x189/0x240 (16)
[<c01ecf20>] write_chan+0x0/0x210 (28)
[<c014ea4a>] vfs_write+0xba/0x100 (32)
[<c014eb3d>] sys_write+0x3d/0x70 (36)
[<c0105d07>] syscall_call+0x7/0xb (40)
A known issue with JACK is that it prints from the realtime thread. I
think someone might have posted a fix for this recently.
Lee
On Sat, 2004-10-30 at 00:31 +0200, Florian Schmidt wrote:
> On Fri, 29 Oct 2004 23:25:45 +0200
> Ingo Molnar <[email protected]> wrote:
>
> > > will do so. btw: i think i'm a bit confused right now. What debugging
> > > features should i have enabled for this test?
> >
> > this particular one (atomicity-checking) is always-enabled if you have
> > the -RT patch applied (it's a really cheap check).
> >
> > for the 'application-triggered tracing' facility we talked about earlier
> > is only active if LATENCY_TRACING is enabled. In that case to turn the
> > tracer on, call:
> [snip]
>
> Ok,
>
> .config attached,
>
> running the patched jackd -R -P 60 -d alsa -p64 for a bit and provoking
> xruns via window wiggling :)
>
Ingo, here are some of my traces with the same settings. These do seem
to correspond to the xruns. Many of them look tty related - could the
recent changes to the tty layer be responsible? Possibly this has
nothing to do with RT preempt, but is an unrelated bug in -mm? The
xruns do seem to correspond to display activity such as switching tabs
in gnome-terminal.
jackd:1507 userspace BUG: scheduling in user-atomic context!
[<c01069fc>] dump_stack+0x1c/0x20 (20)
[<c02834d0>] schedule+0x70/0x100 (24)
[<c010639b>] work_resched+0x6/0x17 (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e5d3>] .... print_traces+0x13/0x50
.....[<c01069fc>] .. ( <= dump_stack+0x1c/0x20)
jackd:1507 userspace BUG: scheduling in user-atomic context!
[<c01069fc>] dump_stack+0x1c/0x20 (20)
[<c02834d0>] schedule+0x70/0x100 (24)
[<c028465d>] down_write_mutex+0xbd/0x180 (36)
[<c012cbf6>] __mutex_lock+0x36/0x40 (16)
[<c012cc75>] _mutex_lock_irqsave+0x15/0x20 (16)
[<c01f1237>] tty_ldisc_try+0x17/0x50 (20)
[<c01f1287>] tty_ldisc_ref_wait+0x17/0xc0 (88)
[<c01f21fd>] tty_write+0x7d/0x230 (68)
[<c01546ac>] vfs_write+0xbc/0x110 (36)
[<c01547b1>] sys_write+0x41/0x70 (44)
[<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e5d3>] .... print_traces+0x13/0x50
.....[<c01069fc>] .. ( <= dump_stack+0x1c/0x20)
jackd:1507 userspace BUG: scheduling in user-atomic context!
[<c01069fc>] dump_stack+0x1c/0x20 (20)
[<c02834d0>] schedule+0x70/0x100 (24)
[<c010639b>] work_resched+0x6/0x17 (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e5d3>] .... print_traces+0x13/0x50
.....[<c01069fc>] .. ( <= dump_stack+0x1c/0x20)
jackd:1507 userspace BUG: scheduling in user-atomic context!
[<c01069fc>] dump_stack+0x1c/0x20 (20)
[<c02834d0>] schedule+0x70/0x100 (24)
[<c028465d>] down_write_mutex+0xbd/0x180 (36)
[<c012cbf6>] __mutex_lock+0x36/0x40 (16)
[<c012cc75>] _mutex_lock_irqsave+0x15/0x20 (16)
[<c01f1237>] tty_ldisc_try+0x17/0x50 (20)
[<c01f1287>] tty_ldisc_ref_wait+0x17/0xc0 (88)
[<c01f21fd>] tty_write+0x7d/0x230 (68)
[<c01546ac>] vfs_write+0xbc/0x110 (36)
[<c01547b1>] sys_write+0x41/0x70 (44)
[<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e5d3>] .... print_traces+0x13/0x50
.....[<c01069fc>] .. ( <= dump_stack+0x1c/0x20)
jackd:1507 userspace BUG: scheduling in user-atomic context!
[<c01069fc>] dump_stack+0x1c/0x20 (20)
[<c02834d0>] schedule+0x70/0x100 (24)
[<c028449c>] _down_write+0xcc/0x170 (32)
[<c01131d3>] lock_kernel+0x23/0x30 (16)
[<c01f22f0>] tty_write+0x170/0x230 (64)
[<c01546ac>] vfs_write+0xbc/0x110 (36)
[<c01547b1>] sys_write+0x41/0x70 (44)
[<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e5d3>] .... print_traces+0x13/0x50
.....[<c01069fc>] .. ( <= dump_stack+0x1c/0x20)
jackd:1507 userspace BUG: scheduling in user-atomic context!
[<c01069fc>] dump_stack+0x1c/0x20 (20)
[<c02834d0>] schedule+0x70/0x100 (24)
[<c0119cba>] do_exit+0x29a/0x500 (24)
[<c0119f56>] sys_exit+0x16/0x20 (12)
[<c0106367>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c012e5d3>] .... print_traces+0x13/0x50
.....[<c01069fc>] .. ( <= dump_stack+0x1c/0x20)
However the longest latency recorded during all these xruns is 40-150 usecs:
preemption latency trace v1.0.7 on 2.6.9-mm1-RT-V0.5.14
-------------------------------------------------------
latency: 48 us, entries: 26 (26) | [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
-----------------
| task: jackd/1629, uid:1000 nice:0 policy:1 rt_prio:10
-----------------
=> started at: try_to_wake_up+0x4c/0x100 <c01126ec>
=> ended at: finish_task_switch+0x27/0xb0 <c0112b77>
=======>
1101 80000000 0.000ms (+0.000ms): (89) ((116))
1101 80000000 0.000ms (+0.000ms): (1629) ((1101))
1101 80000000 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up)
1101 80000000 0.001ms (+0.000ms): preempt_schedule (__up_write)
1101 00000000 0.001ms (+0.000ms): preempt_schedule (up_write_mutex)
1101 80000000 0.002ms (+0.000ms): __schedule (preempt_schedule)
1101 80000000 0.002ms (+0.000ms): profile_hit (__schedule)
1101 80000000 0.003ms (+0.002ms): sched_clock (__schedule)
1629 80000000 0.005ms (+0.000ms): __switch_to (__schedule)
1629 80000000 0.006ms (+0.000ms): (1101) ((1629))
1629 80000000 0.006ms (+0.000ms): (116) ((89))
1629 80000000 0.006ms (+0.000ms): finish_task_switch (__schedule)
1629 80000000 0.007ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
1629 80000000 0.007ms (+0.002ms): (1629) ((89))
1629 00000000 0.010ms (+0.000ms): bad_range (free_pages_bulk)
1629 00000000 0.010ms (+0.000ms): bad_range (free_pages_bulk)
1629 00000000 0.011ms (+0.000ms): _mutex_unlock_irqrestore (free_pages_bulk)
1629 00000000 0.011ms (+0.000ms): up_mutex (free_pages_bulk)
1629 00000000 0.011ms (+0.000ms): up_write_mutex (free_pages_bulk)
1629 00000000 0.012ms (+0.004ms): __up_write (up_write_mutex)
1629 00000000 0.016ms (+0.000ms): sys_gettimeofday (syscall_call)
1629 00000000 0.016ms (+0.000ms): user_trace_stop (sys_gettimeofday)
1629 80000000 0.017ms (+0.000ms): user_trace_stop (sys_gettimeofday)
1629 80000000 0.017ms (+0.000ms): update_max_trace (user_trace_stop)
1629 80000000 0.018ms (+0.000ms): _mmx_memcpy (update_max_trace)
1629 80000000 0.018ms (+0.000ms): kernel_fpu_begin (_mmx_memcpy)
We know that jackd prints from the realtime thread, and that in theory
this could be a problem, in practice it works OK. Maybe some recent
changes to the tty layer made this problematic.
I think there was a patch posted to the JACK mailing list to print from
a separate thread, I will look into this.
Lee
On Fri, 2004-10-29 at 23:36 -0400, Lee Revell wrote:
> Ingo, here are some of my traces with the same settings. These do seem
> to correspond to the xruns. Many of them look tty related - could the
> recent changes to the tty layer be responsible? Possibly this has
> nothing to do with RT preempt, but is an unrelated bug in -mm? The
> xruns do seem to correspond to display activity such as switching tabs
> in gnome-terminal.
>
OK here is a trace that might explain the xruns:
preemption latency trace v1.0.7 on 2.6.9-mm1-RT-V0.5.14
-------------------------------------------------------
latency: 476 us, entries: 260 (260) | [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
-----------------
| task: jackd/1726, uid:1000 nice:0 policy:1 rt_prio:10
-----------------
=> started at: try_to_wake_up+0x4c/0x100 <c01126ec>
=> ended at: finish_task_switch+0x27/0xb0 <c0112b77>
=======>
4 80000000 0.000ms (+0.000ms): (49) ((98))
4 80000000 0.000ms (+0.000ms): (835) ((4))
4 80000000 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up)
4 80000000 0.001ms (+0.000ms): preempt_schedule (__do_IRQ)
4 80000000 0.001ms (+0.000ms): irq_exit (do_IRQ)
4 80000000 0.002ms (+0.000ms): do_softirq (irq_exit)
4 80000000 0.002ms (+0.075ms): __do_softirq (do_softirq)
4 80000000 0.078ms (+0.000ms): do_IRQ (_mmx_memcpy)
4 80000000 0.078ms (+0.000ms): do_IRQ ((0))
4 80000000 0.079ms (+0.002ms): mask_and_ack_8259A (__do_IRQ)
4 80000000 0.081ms (+0.000ms): preempt_schedule (__do_IRQ)
4 80000000 0.082ms (+0.000ms): redirect_hardirq (__do_IRQ)
4 80000000 0.082ms (+0.000ms): preempt_schedule (__do_IRQ)
4 80000000 0.083ms (+0.000ms): handle_IRQ_event (__do_IRQ)
4 80000000 0.083ms (+0.000ms): timer_interrupt (handle_IRQ_event)
4 80000000 0.084ms (+0.003ms): mark_offset_tsc (timer_interrupt)
4 80000000 0.088ms (+0.001ms): preempt_schedule (mark_offset_tsc)
4 80000000 0.089ms (+0.000ms): preempt_schedule (mark_offset_tsc)
4 80000000 0.089ms (+0.000ms): do_timer (timer_interrupt)
4 80000000 0.090ms (+0.000ms): update_process_times (timer_interrupt)
4 80000000 0.090ms (+0.000ms): update_one_process (update_process_times)
4 80000000 0.091ms (+0.000ms): run_local_timers (update_process_times)
4 80000000 0.091ms (+0.000ms): raise_softirq (update_process_times)
4 80000000 0.091ms (+0.000ms): scheduler_tick (update_process_times)
4 80000000 0.092ms (+0.001ms): sched_clock (scheduler_tick)
4 80000000 0.093ms (+0.000ms): preempt_schedule (scheduler_tick)
4 80000000 0.093ms (+0.000ms): profile_hit (timer_interrupt)
4 80000000 0.094ms (+0.000ms): preempt_schedule (timer_interrupt)
4 80000000 0.094ms (+0.000ms): note_interrupt (__do_IRQ)
4 80000000 0.095ms (+0.000ms): end_8259A_irq (__do_IRQ)
4 80000000 0.095ms (+0.001ms): enable_8259A_irq (__do_IRQ)
4 80000000 0.097ms (+0.000ms): preempt_schedule (__do_IRQ)
4 80000000 0.097ms (+0.000ms): preempt_schedule (__do_IRQ)
4 80000000 0.098ms (+0.000ms): irq_exit (do_IRQ)
4 80000000 0.098ms (+0.000ms): do_softirq (irq_exit)
4 80000000 0.098ms (+0.371ms): __do_softirq (do_softirq)
4 00000000 0.470ms (+0.000ms): preempt_schedule (_mmx_memcpy)
4 80000000 0.470ms (+0.000ms): __schedule (preempt_schedule)
4 80000000 0.470ms (+0.000ms): profile_hit (__schedule)
4 80000000 0.471ms (+0.001ms): sched_clock (__schedule)
835 80000000 0.472ms (+0.000ms): __switch_to (__schedule)
835 80000000 0.473ms (+0.000ms): (4) ((835))
835 80000000 0.473ms (+0.000ms): (98) ((49))
835 80000000 0.473ms (+0.000ms): finish_task_switch (__schedule)
835 80000000 0.474ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
835 80000000 0.474ms (+0.849ms): (835) ((49))
835 80000000 1.324ms (+0.000ms): do_IRQ (finish_task_switch)
835 80000000 1.324ms (+0.001ms): do_IRQ ((0))
835 80000000 1.325ms (+0.002ms): mask_and_ack_8259A (__do_IRQ)
835 80000000 1.328ms (+0.000ms): redirect_hardirq (__do_IRQ)
835 80000000 1.329ms (+0.000ms): handle_IRQ_event (__do_IRQ)
835 80000000 1.329ms (+0.001ms): timer_interrupt (handle_IRQ_event)
835 80000000 1.330ms (+0.004ms): mark_offset_tsc (timer_interrupt)
835 80000000 1.335ms (+0.000ms): do_timer (timer_interrupt)
835 80000000 1.335ms (+0.000ms): update_process_times (timer_interrupt)
835 80000000 1.336ms (+0.000ms): update_one_process (update_process_times)
835 80000000 1.336ms (+0.000ms): run_local_timers (update_process_times)
835 80000000 1.337ms (+0.000ms): raise_softirq (update_process_times)
Lee
* Lee Revell <[email protected]> wrote:
> 835 80000000 0.473ms (+0.000ms): finish_task_switch (__schedule)
> 835 80000000 0.474ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
> 835 80000000 0.474ms (+0.849ms): (835) ((49))
> 835 80000000 1.324ms (+0.000ms): do_IRQ (finish_task_switch)
> 835 80000000 1.324ms (+0.001ms): do_IRQ ((0))
> 835 80000000 1.325ms (+0.002ms): mask_and_ack_8259A (__do_IRQ)
this seems to be very similar to the DMA problems Mark H Johnson had. It
is almost certainly not caused by the kernel. It could in theory be some
SMM overhead, but the bigger likelyhood is disk DMA.
Ingo
* Lee Revell <[email protected]> wrote:
> Ingo, here are some of my traces with the same settings. These do
> seem to correspond to the xruns. Many of them look tty related -
> could the recent changes to the tty layer be responsible? Possibly
> this has nothing to do with RT preempt, but is an unrelated bug in
> -mm? The xruns do seem to correspond to display activity such as
> switching tabs in gnome-terminal.
>
> jackd:1507 userspace BUG: scheduling in user-atomic context!
> [<c01069fc>] dump_stack+0x1c/0x20 (20)
> [<c02834d0>] schedule+0x70/0x100 (24)
> [<c010639b>] work_resched+0x6/0x17 (-8124)
ok, this shows jackd getting preempted by a higher-prio task. This could
be the watchdog thread - could you chrt the watchdog thread to make sure
jackd itself never gets (legitimately) preempted by any other thread?
also, in my tree i fixed this codepath to not trigger an atomicity
warning, since this is an involuntary rescheduling event.
> jackd:1507 userspace BUG: scheduling in user-atomic context!
> [<c01069fc>] dump_stack+0x1c/0x20 (20)
> [<c02834d0>] schedule+0x70/0x100 (24)
> [<c028465d>] down_write_mutex+0xbd/0x180 (36)
> [<c012cbf6>] __mutex_lock+0x36/0x40 (16)
> [<c012cc75>] _mutex_lock_irqsave+0x15/0x20 (16)
> [<c01f1237>] tty_ldisc_try+0x17/0x50 (20)
> [<c01f1287>] tty_ldisc_ref_wait+0x17/0xc0 (88)
> [<c01f21fd>] tty_write+0x7d/0x230 (68)
> [<c01546ac>] vfs_write+0xbc/0x110 (36)
> [<c01547b1>] sys_write+0x41/0x70 (44)
> [<c0106367>] syscall_call+0x7/0xb (-8124)
this too i'd consider a false positive, because this particular printout
happened due to a mutex, and a mutex blockage we can consider
involuntary blockage. But ... tty writes can cause BKL locking.
> We know that jackd prints from the realtime thread, and that in theory
> this could be a problem, in practice it works OK. Maybe some recent
> changes to the tty layer made this problematic.
i'd suggest to do an atomic_off() whenever jackd prints an xrun warning
from the highprio thread.
> I think there was a patch posted to the JACK mailing list to print
> from a separate thread, I will look into this.
that would be the better longterm solution.
Ingo
* Ingo Molnar <[email protected]> wrote:
> > jackd:1507 userspace BUG: scheduling in user-atomic context!
> > [<c01069fc>] dump_stack+0x1c/0x20 (20)
> > [<c02834d0>] schedule+0x70/0x100 (24)
> > [<c010639b>] work_resched+0x6/0x17 (-8124)
> > jackd:1507 userspace BUG: scheduling in user-atomic context!
> > [<c01069fc>] dump_stack+0x1c/0x20 (20)
> > [<c02834d0>] schedule+0x70/0x100 (24)
> > [<c028465d>] down_write_mutex+0xbd/0x180 (36)
> > [<c012cbf6>] __mutex_lock+0x36/0x40 (16)
> > [<c012cc75>] _mutex_lock_irqsave+0x15/0x20 (16)
> > [<c01f1237>] tty_ldisc_try+0x17/0x50 (20)
> > [<c01f1287>] tty_ldisc_ref_wait+0x17/0xc0 (88)
> this too i'd consider a false positive, because this particular
> printout happened due to a mutex, and a mutex blockage we can consider
> involuntary blockage. [...]
i've uploaded -V0.5.15 with these two false positives fixed, could you
retry your Jackd workload and see whether any other type of scheduling
gets detected? Please also chrt the watchdog thread to a lower priority
than the main thread's priority.
Ingo
* Paul Davis <[email protected]> wrote:
> IIRC, Rui was running with -p128, which at 48000kHz is 2600usecs (and
> longer at 44100kHz). If the maximum process cycle was on the order of
> 1500usecs, that leaves nearly 1ms until the next interrupt is due.
> Unless jackd was held up between computing the process cycle time and
> entering poll, it doesn't seem that the process cycle ever gets close
> to the interrupt interval duration.
yeah, i'd agree with this, there must be something else going on.
> So I don't think that delays caused *during* jackd's processing cycle
> are involved here. However, I agree that your suggestion to check for
> this before computing max_delay is probably wise in general.
yep.
Ingo
* Lee Revell <[email protected]> wrote:
> Here is the dmesg output. It looks like the problem could be related
> to jackd's printing from the realtime thread. But, this has to be the
> kernel's fault on some level, because with an earlier version I get no
> xruns.
with the earlier version these spinlocks were simply disabling
preemption, while now they will schedule away on contention. If that tty
lock is held for a long time by a lowprio task then that could delay the
highprio thread. We are starting to see priority inversion problems.
But, the core issue is doing tty printouts - does jackd do that
periodically, or only as a reaction to an already existing latency?
> jackd:1846 userspace BUG: scheduling in user-atomic context!
> [<c01069fc>] dump_stack+0x1c/0x20 (20)
> [<c0283e60>] schedule+0x70/0x100 (24)
> [<c0119efa>] do_exit+0x29a/0x500 (24)
> [<c011a196>] sys_exit+0x16/0x20 (12)
> [<c0106367>] syscall_call+0x7/0xb (-8124)
this one is interesting - does the jackd highprio thread start new
threads and lets them exit? The above schedule() is the final one of an
exit()-ing thread.
> jackd:1854 userspace BUG: scheduling in user-atomic context!
> [<c01069fc>] dump_stack+0x1c/0x20 (20)
> [<c0283e60>] schedule+0x70/0x100 (24)
> [<c0119efa>] do_exit+0x29a/0x500 (24)
same exit() scenario. That would be pretty much a no-no, a new child
thread inherits the parent's SCHED_FIFO priority and due to
child-runs-first it could delay the parent possibly indefinitely.
Ingo
On Sat, 30 Oct 2004 13:28:05 +0200
Ingo Molnar <[email protected]> wrote:
> > I think there was a patch posted to the JACK mailing list to print
> > from a separate thread, I will look into this.
>
> that would be the better longterm solution.
afaik jack should ony print from the RT thread when something bad happened
(like an xrun), so in normal operation this shouldn't make problems. but i'm
not sure. maybe paul knows more.
flo
* Florian Schmidt <[email protected]> wrote:
> and another one:
>
> preemption latency trace v1.0.7 on 2.6.9-mm1-RT-V0.5.14-rt
> -------------------------------------------------------
> latency: 225 us, entries: 36 (36) | [VP:0 KP:1 SP:1 HP:1 #CPUS:1]
> -----------------
> | task: jackd/1083, uid:1000 nice:0 policy:1 rt_prio:60
> -----------------
> => started at: try_to_wake_up+0x5a/0x110 <c01151ca>
> => ended at: finish_task_switch+0x31/0xc0 <c0115691>
> =======>
> 637 80000000 0.000ms (+0.000ms): (39) ((116))
> 637 80000000 0.000ms (+0.000ms): (1083) ((637))
> 637 80000000 0.000ms (+0.000ms): preempt_schedule (try_to_wake_up)
> 637 80000000 0.000ms (+0.000ms): preempt_schedule (__up_write)
> 637 00000000 0.000ms (+0.000ms): preempt_schedule (up_write_mutex)
> 637 80000000 0.000ms (+0.000ms): __schedule (preempt_schedule)
> 637 80000000 0.000ms (+0.000ms): profile_hit (__schedule)
> 637 80000000 0.000ms (+0.001ms): sched_clock (__schedule)
> 1083 80000000 0.002ms (+0.000ms): __switch_to (__schedule)
> 1083 80000000 0.002ms (+0.000ms): (637) ((1083))
> 1083 80000000 0.002ms (+0.000ms): (116) ((39))
> 1083 80000000 0.002ms (+0.000ms): finish_task_switch (__schedule)
> 1083 80000000 0.002ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
> 1083 80000000 0.002ms (+0.000ms): (1083) ((39))
> 1083 00000000 0.003ms (+0.000ms): bad_range (free_pages_bulk)
> 1083 00000000 0.003ms (+0.000ms): bad_range (free_pages_bulk)
> 1083 00000000 0.003ms (+0.000ms): _mutex_unlock_irqrestore (free_pages_bulk)
> 1083 00000000 0.003ms (+0.000ms): up_mutex (free_pages_bulk)
> 1083 00000000 0.004ms (+0.000ms): up_write_mutex (free_pages_bulk)
> 1083 00000000 0.004ms (+0.000ms): __up_write (up_write_mutex)
> 1083 80000000 0.004ms (+0.000ms): wake_up_process_mutex (__up_write)
> 1083 80000000 0.004ms (+0.000ms): try_to_wake_up (wake_up_process_mutex)
> 1083 80000000 0.004ms (+0.000ms): task_rq_lock (try_to_wake_up)
> 1083 80000000 0.004ms (+0.000ms): activate_task (try_to_wake_up)
> 1083 80000000 0.004ms (+0.000ms): sched_clock (activate_task)
> 1083 80000000 0.004ms (+0.000ms): recalc_task_prio (activate_task)
> 1083 80000000 0.004ms (+0.000ms): effective_prio (recalc_task_prio)
> 1083 80000000 0.004ms (+0.000ms): enqueue_task (activate_task)
> 1083 80000000 0.005ms (+0.000ms): (115) ((39))
> 1083 80000000 0.005ms (+0.006ms): (782) ((1083))
> 1083 00000000 0.011ms (+0.000ms): sys_gettimeofday (syscall_call)
> 1083 00000000 0.011ms (+0.000ms): user_trace_stop (sys_gettimeofday)
> 1083 80000000 0.012ms (+0.000ms): user_trace_stop (sys_gettimeofday)
> 1083 80000000 0.012ms (+0.000ms): update_max_trace (user_trace_stop)
> 1083 80000000 0.012ms (+0.000ms): _mmx_memcpy (update_max_trace)
> 1083 80000000 0.013ms (+0.000ms): kernel_fpu_begin (_mmx_memcpy)
should have mentioned that in the user-triggered modus you have to do
the latency measurement in userspace. It is only the trace that is
correct, for the time being. This trace shows what i'd expect a jackd
wakeup to look like normally: 13 usecs.
Ingo
On Sat, 30 Oct 2004 15:15:07 +0200
Ingo Molnar <[email protected]> wrote:
> should have mentioned that in the user-triggered modus you have to do
> the latency measurement in userspace. It is only the trace that is
> correct, for the time being. This trace shows what i'd expect a jackd
> wakeup to look like normally: 13 usecs.
>
> Ingo
>
Hi,
i thought, why not try to get jack out of the equation first and use a
different irq source. So i wrote a small test app for /dev/rtc to see if i
see the same erratic behaviour [.cc source file atached. find a small
tarball here: http://affenbande.org/~tapas/wakeup.tgz (it has a simple
makefile, plus a script which does the chrt thing to make it SCHED_FIFO)].
This program takes two parameters: the desired freq of the rtc and the
number of irq's to be measured.
Then it polls on the fd of /dev/rtc and takes a cpu cycle count timestamp as
soon as poll returns. At the end of the program i tried to gather some
[useful ???] statistics of the data:
- the mean difference in cycles between two wakeups
- the max difference in cycles between two wakeups (and how much this
differs from the mean difference)
- the min difference in cycles between two wakeups (and how much this
differs from the mean difference)
- the mean difference from the mean difference :)
And alas, wiggling windows screws up the timing on V0.5.6 for this, too.
dunno, if this is any useful to you, but i felt the urge to try it :)
flo
P.S.: don't forget to make your rtc irq SCHED_FIFO with a high priority, too
here's a sample output with window wiggling in X (rt_wakeup runs chrt -f 90
wakeup, so make rtc at least prio 91):
~/source/my_projects/wakeup$ ./rt_wakeup 1024 6000
freq: 1024 #: 6000
setting up /dev/rtc...
locking memory...
turning irq on, beginning measurement (might take a while)...
...measurement done
mean cycle difference betweem two wakeups: 1.17845e+06 cycles
min. cycle difference betweem two wakeups: 185992 cycles (#: 1563)
diff from mean diff: 992461
max. cycle difference betweem two wakeups: 9.73166e+06 cycles (#: 1546)
diff from mean diff: 8.5532e+06
^this is bad i suppose :)
mean difference from mean difference: 25279.2 cycles
here's one on a rather idle system:
~/source/my_projects/wakeup$ ./rt_wakeup 1024 6000
freq: 1024 #: 6000
setting up /dev/rtc...
locking memory...
turning irq on, beginning measurement (might take a while)...
...measurement done
mean cycle difference betweem two wakeups: 1.16697e+06 cycles
min. cycle difference betweem two wakeups: 1.1486e+06 cycles (#: 5492)
diff from mean diff: 18379.2
max. cycle difference betweem two wakeups: 1.18439e+06 cycles (#: 5491)
diff from mean diff: 17417.8
mean difference from mean difference: 1144.82 cycles
~/source/my_projects/wakeup$
On Sat, 2004-10-30 at 13:16 +0200, Ingo Molnar wrote:
> * Lee Revell <[email protected]> wrote:
>
> > 835 80000000 0.473ms (+0.000ms): finish_task_switch (__schedule)
> > 835 80000000 0.474ms (+0.000ms): trace_stop_sched_switched (finish_task_switch)
> > 835 80000000 0.474ms (+0.849ms): (835) ((49))
> > 835 80000000 1.324ms (+0.000ms): do_IRQ (finish_task_switch)
> > 835 80000000 1.324ms (+0.001ms): do_IRQ ((0))
> > 835 80000000 1.325ms (+0.002ms): mask_and_ack_8259A (__do_IRQ)
>
> this seems to be very similar to the DMA problems Mark H Johnson had. It
> is almost certainly not caused by the kernel. It could in theory be some
> SMM overhead, but the bigger likelyhood is disk DMA.
>
I find this hard to believe - I _never_ saw that problem on this system
before. And it works _perfectly_ with T3 - I can run hundreds of
millions of process cycles without a single xrun.
Lee
On Sat, 2004-10-30 at 13:58 +0200, Ingo Molnar wrote:
> * Lee Revell <[email protected]> wrote:
>
> > Here is the dmesg output. It looks like the problem could be related
> > to jackd's printing from the realtime thread. But, this has to be the
> > kernel's fault on some level, because with an earlier version I get no
> > xruns.
>
> with the earlier version these spinlocks were simply disabling
> preemption, while now they will schedule away on contention. If that tty
> lock is held for a long time by a lowprio task then that could delay the
> highprio thread. We are starting to see priority inversion problems.
> But, the core issue is doing tty printouts - does jackd do that
> periodically, or only as a reaction to an already existing latency?
>
No, this cannot be the whole story, because unless verbose mode is
specified, jackd will only prints anything if there is an xrun. So
there is something else going on.
This _really_ feels like a kernel bug. Are you saying that this could
still be a jackd problem, even though T3 works perfectly with the exact
same jackd binary?
Lee
* Lee Revell <[email protected]> wrote:
> No, this cannot be the whole story, because unless verbose mode is
> specified, jackd will only prints anything if there is an xrun. So
> there is something else going on.
>
> This _really_ feels like a kernel bug. Are you saying that this could
> still be a jackd problem, even though T3 works perfectly with the
> exact same jackd binary?
i think you are right - there are too many independent indicators
pointing towards some sort of kernel problem. I'll try Florian's testapp
and see whether i can see the 'window wiggle' problem.
Ingo
On Sat, 30 Oct 2004 21:17:25 +0200
Ingo Molnar <[email protected]> wrote:
> > No, this cannot be the whole story, because unless verbose mode is
> > specified, jackd will only prints anything if there is an xrun. So
> > there is something else going on.
> >
> > This _really_ feels like a kernel bug. Are you saying that this could
> > still be a jackd problem, even though T3 works perfectly with the
> > exact same jackd binary?
>
> i think you are right - there are too many independent indicators
> pointing towards some sort of kernel problem. I'll try Florian's testapp
> and see whether i can see the 'window wiggle' problem.
Hi, in the meantime i also booted into P9 again and the results differ
dramatically. Much better in P9. Anyways, i reuploaded the tarball. The
program tries to detect missed irq's now and counts the total number of
irq's delivered by /dev/rtc. Since the program does not recover from missed
irq's the "statistical" data for these runs is useless [except for the
knowledge of the fact that one or more irq was missed :)]
flo
* Florian Schmidt <[email protected]> wrote:
> Hi, in the meantime i also booted into P9 again and the results differ
> dramatically. Much better in P9. Anyways, i reuploaded the tarball.
> The program tries to detect missed irq's now and counts the total
> number of irq's delivered by /dev/rtc. Since the program does not
> recover from missed irq's the "statistical" data for these runs is
> useless [except for the knowledge of the fact that one or more irq was
> missed :)]
just to make sure - you are running this on an UP system, correct?
Ingo
On Sat, 30 Oct 2004 21:37:06 +0200
Ingo Molnar <[email protected]> wrote:
> > Hi, in the meantime i also booted into P9 again and the results differ
> > dramatically. Much better in P9. Anyways, i reuploaded the tarball.
> > The program tries to detect missed irq's now and counts the total
> > number of irq's delivered by /dev/rtc. Since the program does not
> > recover from missed irq's the "statistical" data for these runs is
> > useless [except for the knowledge of the fact that one or more irq was
> > missed :)]
>
> just to make sure - you are running this on an UP system, correct?
right. all debugging off [when possible], RP on.
mango:~# cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 6
model : 4
model name : AMD Athlon(tm) Processor
stepping : 2
cpu MHz : 1195.144
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat pse36 mmx fxsr syscall mmxext 3dnowext 3dnow
bogomips : 2375.68
flo
On Sat, 2004-10-30 at 21:17 +0200, Ingo Molnar wrote:
> i think you are right - there are too many independent indicators
> pointing towards some sort of kernel problem. I'll try Florian's testapp
> and see whether i can see the 'window wiggle' problem.
This is kind of weird:
Checking 'hlt' instruction... OK.
spawn_desched_task(00000000)
desched cpu_callback 3/00000000
ksoftirqd started up.
softirq RT prio: 24.
desched cpu_callback 2/00000000
desched thread 0 started up.
But:
rlrevell@mindpipe:~/cvs/wakeup$ chrt -p 2
pid 2's current scheduling policy: SCHED_OTHER
pid 2's current scheduling priority: 0
I do not think I changed ksoftirqd's priority.
Lee
On Sat, 2004-10-30 at 21:47 +0200, Florian Schmidt wrote:
> Hi, in the meantime i also booted into P9 again and the results differ
> dramatically. Much better in P9. Anyways, i reuploaded the tarball. The
> program tries to detect missed irq's now and counts the total number of
> irq's delivered by /dev/rtc. Since the program does not recover from missed
> irq's the "statistical" data for these runs is useless [except for the
> knowledge of the fact that one or more irq was missed :)]
Yup there is definitely something not right:
rlrevell@mindpipe:~/cvs/wakeup$ ./rt_wakeup 1024 100000
freq: 1024 #: 100000
setting up /dev/rtc.
locking memory...
turning irq on, beginning measurement (might take a while).
done.
total # of irqs: 100065. missed irq's: 4
mean cycle difference betweem two wakeups: 586332 cycles
min. cycle difference betweem two wakeups: 25184 cycles (#: 28766)
diff from mean diff: 561148
max. cycle difference betweem two wakeups: 2.78432e+07 cycles (#: 98195)
diff from mean diff: 2.72569e+07
mean difference from mean difference: 3759.45 cycles
You should modify the program to print something when it sees a big
miss. This would make it easier to figure out what kind of system
activity triggers the problem.
Lee
On Sat, 2004-10-30 at 21:47 +0200, Florian Schmidt wrote:
> Hi, in the meantime i also booted into P9 again and the results differ
> dramatically. Much better in P9. Anyways, i reuploaded the tarball. The
> program tries to detect missed irq's now and counts the total number of
> irq's delivered by /dev/rtc. Since the program does not recover from missed
> irq's the "statistical" data for these runs is useless [except for the
> knowledge of the fact that one or more irq was missed :)]
Here are my results with T3. As expected, much better.
--
rlrevell@mindpipe:~/cvs/wakeup$ ./rt_wakeup 1024 100000 freq: 1024 #:
100000
setting up /dev/rtc.
locking memory...
turning irq on, beginning measurement (might take a while).
done.
total # of irqs: 100000. missed irq's: 0
mean cycle difference betweem two wakeups: 585875 cycles
min. cycle difference betweem two wakeups: 517708 cycles (#: 27478)
diff from mean diff: 68166.8
max. cycle difference betweem two wakeups: 656018 cycles (#: 27477)
diff from mean diff: 70143.2
mean difference from mean difference: 3052.86 cycles
--
So maybe the bug is not related to scheduling of SCHED_FIFO threads, but
that we are missing IRQs. I think this would explain the choppy
playback with mplayer (it uses the RTC and does not run SCHED_FIFO).
Lee
On Sat, 30 Oct 2004 15:52:04 -0400
Lee Revell <[email protected]> wrote:
> Yup there is definitely something not right:
[snip]
> You should modify the program to print something when it sees a big
> miss. This would make it easier to figure out what kind of system
> activity triggers the problem.
right, i just wanted to avoid doing that from the process that polls itself,
because a std::cout << "ugh!" << std::endl; might already be enough to skew
the following irq's, right?
anyways, this new version [just upped] prints a line when a missed irq was
detected. Also this version understands a third parameter which acts as an
upper threshold. A line is printed when the difference of the cycle count of
two consecutive wakeups is greater than the threshold.
run it once w/o threshold on an idle system to see what a useful thresh
would be.
i use it like this for example:
./rt_wakeup 1024 50000 1200000
What's the best way to find out the cycles/s of the cpu? This way the
input/output could become a little nicer [because then i can calculate
programatically how long a "perfect" period should be in cycles].
flo
On Sat, 2004-10-30 at 22:15 +0200, Florian Schmidt wrote:
> What's the best way to find out the cycles/s of the cpu? This way the
> input/output could become a little nicer [because then i can calculate
> programatically how long a "perfect" period should be in cycles].
Take a look at the patch I posted to jackit-devel the other day to
calculate the CPU speed (previously we grabbed it from /proc/cpuinfo).
I just copied the code from realfeel2.
Lee
On Sat, 30 Oct 2004 16:05:14 -0400
Lee Revell <[email protected]> wrote:
> So maybe the bug is not related to scheduling of SCHED_FIFO threads, but
> that we are missing IRQs. I think this would explain the choppy
> playback with mplayer (it uses the RTC and does not run SCHED_FIFO).
I wonder about what X11 is doing. Is it maybe doing some locking which is
"out of line" for RP or something? I mean the RP patches touch everything
that accesses hw directly with the exception of X11, right? [/me never
grokked how X fits into linux' driver model anyways]
flo
On Sat, 30 Oct 2004 16:26:35 -0400
Lee Revell <[email protected]> wrote:
> Take a look at the patch I posted to jackit-devel the other day to
> calculate the CPU speed (previously we grabbed it from /proc/cpuinfo).
> I just copied the code from realfeel2.
>
ah, ok.. tarball updated. The third argument is now a percentage. If the
cycle count difference between two different wakeups differs more than the
specified percentage from the "perfect" period, then a line is printed to
the terminal showing by how much percent it differs.
flo
On Sat, 30 Oct 2004 17:26:06 -0400
Lee Revell <[email protected]> wrote:
> On Sat, 2004-10-30 at 23:13 +0200, Florian Schmidt wrote:
> > ah, ok.. tarball updated. The third argument is now a percentage. If the
> > cycle count difference between two different wakeups differs more than the
> > specified percentage from the "perfect" period, then a line is printed to
> > the terminal showing by how much percent it differs.
>
> OK this is pretty sweet. With T3 the jitter never exceeds 7% on an idle
> system. As soon as I start moving the mouse this goes to 7 or 8%. I
> cannot get it to go higher than 10%. Moving windows around has no
> effect, the highest jitter happens when I type or move the mouse really
> fast IOW it corresponds to the interrupt rate.
>
> This is a pretty good baseline for what an xrun-free system would look
> like. Now to test the latest version...
Well,
on V0.5.16 i see something like the below output (which is much worse). It
seems that missed irq's with rtc show up at the same time as the xruns in
jackd do [i ran both jackd and wakeup in parallel].
While there's no way to deterministically force missed irq's by window
wiggling [we should make it olympic discipline :)], UI action seems to raise
the probability somewhat.
~/source/my_projects/wakeup$ ./rt_wakeup 1024 50000 10
freq: 1024 #: 50000
getting cpu speed
1194.908 MHz
# of cycles for "perfect" period: 1166902
setting up /dev/rtc.
locking memory...
turning irq on, beginning measurement (might take a while).
threshold violated: 1485237 cycles since last wakeup (27.2804%).
threshold violated: 1047469 cycles since last wakeup (10.235%).
threshold violated: 964069 cycles since last wakeup (17.3822%).
ouch! we missed one ore more irq[s]
threshold violated: 4037774 cycles since last wakeup (246.025%).
threshold violated: 620764 cycles since last wakeup (46.8024%).
threshold violated: 10327889 cycles since last wakeup (785.069%).
ouch! we missed one ore more irq[s]
threshold violated: 700832 cycles since last wakeup (39.9408%).
threshold violated: 3148900 cycles since last wakeup (169.851%).
ouch! we missed one ore more irq[s]
threshold violated: 231791 cycles since last wakeup (80.1362%).
threshold violated: 768584 cycles since last wakeup (34.1347%).
ouch! we missed one ore more irq[s]
threshold violated: 3500123 cycles since last wakeup (199.95%).
threshold violated: 2581587 cycles since last wakeup (121.234%).
threshold violated: 920020 cycles since last wakeup (21.157%).
threshold violated: 2430290 cycles since last wakeup (108.269%).
threshold violated: 240850 cycles since last wakeup (79.3599%).
threshold violated: 3614768 cycles since last wakeup (209.775%).
ouch! we missed one ore more irq[s]
threshold violated: 883969 cycles since last wakeup (24.2465%).
threshold violated: 479302 cycles since last wakeup (58.9253%).
threshold violated: 830208 cycles since last wakeup (28.8537%).
threshold violated: 2313154 cycles since last wakeup (98.2304%).
threshold violated: 516698 cycles since last wakeup (55.7205%).
threshold violated: 2545998 cycles since last wakeup (118.184%).
ouch! we missed one ore more irq[s]
threshold violated: 4466286 cycles since last wakeup (282.747%).
ouch! we missed one ore more irq[s]
threshold violated: 565233 cycles since last wakeup (51.5612%).
threshold violated: 184668 cycles since last wakeup (84.1745%).
threshold violated: 2854676 cycles since last wakeup (144.637%).
threshold violated: 3384620 cycles since last wakeup (190.052%).
ouch! we missed one ore more irq[s]
threshold violated: 282832 cycles since last wakeup (75.7621%).
threshold violated: 2741798 cycles since last wakeup (134.964%).
ouch! we missed one ore more irq[s]
threshold violated: 1405657 cycles since last wakeup (20.4606%).
threshold violated: 227030 cycles since last wakeup (80.5442%).
threshold violated: 3476092 cycles since last wakeup (197.891%).
ouch! we missed one ore more irq[s]
threshold violated: 448723 cycles since last wakeup (61.5458%).
threshold violated: 2390327 cycles since last wakeup (104.844%).
ouch! we missed one ore more irq[s]
threshold violated: 1489252 cycles since last wakeup (27.6244%).
threshold violated: 517884 cycles since last wakeup (55.6189%).
threshold violated: 3774860 cycles since last wakeup (223.494%).
ouch! we missed one ore more irq[s]
threshold violated: 1762452 cycles since last wakeup (51.0368%).
threshold violated: 1545052 cycles since last wakeup (32.4063%).
ouch! we missed one ore more irq[s]
threshold violated: 442749 cycles since last wakeup (62.0577%).
threshold violated: 2230841 cycles since last wakeup (91.1764%).
threshold violated: 5825655 cycles since last wakeup (399.241%).
ouch! we missed one ore more irq[s]
threshold violated: 2874721 cycles since last wakeup (146.355%).
ouch! we missed one ore more irq[s]
threshold violated: 647431 cycles since last wakeup (44.5171%).
threshold violated: 126309 cycles since last wakeup (89.1757%).
threshold violated: 5817415 cycles since last wakeup (398.535%).
ouch! we missed one ore more irq[s]
threshold violated: 661937 cycles since last wakeup (43.274%).
threshold violated: 4351747 cycles since last wakeup (272.932%).
ouch! we missed one ore more irq[s]
threshold violated: 442189 cycles since last wakeup (62.1057%).
threshold violated: 115427 cycles since last wakeup (90.1083%).
done.
total # of irqs: 50032. missed irq deadlines: 17
mean cycle difference betweem two wakeups: 1.16779e+06 cycles
min. cycle difference betweem two wakeups: 115427 cycles (#: 44584)
diff from mean diff: 1.05237e+06
max. cycle difference betweem two wakeups: 1.03279e+07 cycles (#: 8869)
diff from mean diff: 9.1601e+06
mean difference from mean difference: 6458.8 cycles
~/source/my_projects/wakeup$
On Sat, 2004-10-30 at 23:13 +0200, Florian Schmidt wrote:
> ah, ok.. tarball updated. The third argument is now a percentage. If the
> cycle count difference between two different wakeups differs more than the
> specified percentage from the "perfect" period, then a line is printed to
> the terminal showing by how much percent it differs.
OK this is pretty sweet. With T3 the jitter never exceeds 7% on an idle
system. As soon as I start moving the mouse this goes to 7 or 8%. I
cannot get it to go higher than 10%. Moving windows around has no
effect, the highest jitter happens when I type or move the mouse really
fast IOW it corresponds to the interrupt rate.
This is a pretty good baseline for what an xrun-free system would look
like. Now to test the latest version...
Lee
On Sat, 2004-10-30 at 23:38 +0200, Florian Schmidt wrote:
> While there's no way to deterministically force missed irq's by window
> wiggling [we should make it olympic discipline :)], UI action seems to raise
> the probability somewhat.
>
OK, I am seeing the exact same results. They are so close to yours that
I am not going to bother posting them. UI activity does seem to trigger
missed IRQs - if I switch to my mail window or Mozilla and back to the
test I see that 4 or 5 IRQs in a row got skipped. So maybe context
switching triggers the bug.
Lee
On Sat, 2004-10-30 at 23:13 +0200, Florian Schmidt wrote:
> On Sat, 30 Oct 2004 16:26:35 -0400
> Lee Revell <[email protected]> wrote:
>
> > Take a look at the patch I posted to jackit-devel the other day to
> > calculate the CPU speed (previously we grabbed it from /proc/cpuinfo).
> > I just copied the code from realfeel2.
> >
>
> ah, ok.. tarball updated. The third argument is now a percentage. If the
> cycle count difference between two different wakeups differs more than the
> specified percentage from the "perfect" period, then a line is printed to
> the terminal showing by how much percent it differs.
>
> flo
>
Here is a patch that causes the number of consecutive missed interrupts
to be printed, as well as setting RT priority itself instead of relying
on the rt_wakeup wrapper.
I have found that moving the mouse quickly will reliably cause many
interrupts to be missed.
Lee
--- wakeup/wakeup.cc 2004-10-30 17:11:08.000000000 -0400
+++ wakeup2/wakeup.cc 2004-10-30 21:03:21.000000000 -0400
@@ -90,6 +90,22 @@
}
// realfeel
+int set_realtime_priority(void)
+{
+ struct sched_param schp;
+ /*
+ * set the process to realtime privs
+ */
+ memset(&schp, 0, sizeof(schp));
+ schp.sched_priority = sched_get_priority_max(SCHED_FIFO);
+
+ if (sched_setscheduler(0, SCHED_FIFO, &schp) != 0) {
+ perror("sched_setscheduler");
+ exit(1);
+ }
+
+ return 0;
+}
@@ -181,6 +197,8 @@
perror("ioctl");
exit(errno);
}
+
+ set_realtime_priority();
unsigned int irqs = 0;
unsigned int missed_irqs = 0;
@@ -220,7 +238,7 @@
irqs += data;
// std::cout << data << std::endl;
if (data > 1) {
- std::cout << "ouch! we missed one ore more irq[s]" << std::endl;
+ std::cout << "ouch! we missed " << data << " or more irq[s]" << std::endl;
missed_irqs++;
}
}
Lee
On Sat, 30 Oct 2004 22:20:24 -0400
Lee Revell <[email protected]> wrote:
> Here is a patch that causes the number of consecutive missed interrupts
> to be printed, as well as setting RT priority itself instead of relying
> on the rt_wakeup wrapper.
>
> I have found that moving the mouse quickly will reliably cause many
> interrupts to be missed.
thanks for the patch (it has a little problem, since it uses prio 99 which
is always equal or greater than the rtc thread prio. i changed it in my
local version to accept a parameter). will do some more cleanup of the
program [saner option handling, msec/usec output of the relevant cycle
counts, history file output, ctrl-c handling etc].
U\L in an hour or two.
flo
* Florian Schmidt <[email protected]> wrote:
> On Sat, 30 Oct 2004 17:26:06 -0400
> Lee Revell <[email protected]> wrote:
>
> > OK this is pretty sweet. With T3 the jitter never exceeds 7% on an idle
> > system. As soon as I start moving the mouse this goes to 7 or 8%. I
> > cannot get it to go higher than 10%. Moving windows around has no
> > effect, the highest jitter happens when I type or move the mouse really
> > fast IOW it corresponds to the interrupt rate.
> >
> > This is a pretty good baseline for what an xrun-free system would look
> > like. Now to test the latest version...
>
> Well,
>
> on V0.5.16 i see something like the below output (which is much worse). It
> seems that missed irq's with rtc show up at the same time as the xruns in
> jackd do [i ran both jackd and wakeup in parallel].
ok, could you try the -RT-V0.6.0 patch i've just uploaded? It could i
believe improve these latencies.
Ingo
On Sun, 2004-10-31 at 11:00 +0100, Florian Schmidt wrote:
> thanks for the patch (it has a little problem, since it uses prio 99 which
> is always equal or greater than the rtc thread prio. i changed it in my
> local version to accept a parameter).
OK good idea. As in the JACK case, the relative priorities of the RTC
irq thread and the test program should not matter as these two should
never contend - something is seriously wrong if both are ever runnable
at the same time.
Actually this raises an interesting point. Maybe all IRQ threads should
get the same RT priority by default, so we get FIFO scheduling among IRQ
threads. It seems like this would make it harder for IRQ threads to
starve each other. Then we only have to elevate the priority of the IRQ
thread(s) we are interested in.
Another idea is to allow SCHED_FIFO processes of equal priority to
preempt one another on a LIFO basis. Wouldn't this be very close to the
traditional Linux interrupt model, where interrupts can interrupt each
other and we handle the most recent interrupt first?
Lee
* Lee Revell <[email protected]> wrote:
> Actually this raises an interesting point. Maybe all IRQ threads
> should get the same RT priority by default, so we get FIFO scheduling
> among IRQ threads. It seems like this would make it harder for IRQ
> threads to starve each other. Then we only have to elevate the
> priority of the IRQ thread(s) we are interested in.
we could do this too. The reason why i picked the current "start at
SCHED_FIFO prio 49 and decrease it by 1 until it reaches 25, then stay
constant" logic is that typically the irqs registered first are 'more
important' - e.g. the timer interrupt.
> Another idea is to allow SCHED_FIFO processes of equal priority to
> preempt one another on a LIFO basis. Wouldn't this be very close to
> the traditional Linux interrupt model, where interrupts can interrupt
> each other and we handle the most recent interrupt first?
well, it's called SCHED_*FIFO* for a reason :-) What might make sense is
a SCHED_LIFO policy. But, i'm not so sure it's the right thing to do:
the best work-queueing model is almost always FIFO, as it gives the best
possible fairness between equals.
Fairness also often translates into better performance, because a system
that 'fluctuates' due to LIFO often underperforms a FIFO one because
when it fluctuates towards lower load it under-utilizes the resources,
and when it fluctuates up it overloads. LIFO makes sense for anonymous
resources like pages/slabs where LIFO allocation leads to better cache
utilization. But i'd say for non-anonymous workloads it's almost always
a loss.
Ingo
On Sun, 2004-10-31 at 13:19 +0100, Ingo Molnar wrote:
> we could do this too. The reason why i picked the current "start at
> SCHED_FIFO prio 49 and decrease it by 1 until it reaches 25, then stay
> constant" logic is that typically the irqs registered first are 'more
> important' - e.g. the timer interrupt.
Hmm, maybe the timer interrupt should be 99 and the rest say 50.
Wouldn't it be bad if you had a fully loaded jackd (DSP load in JACK is
the proportion of the process cycle to the period time; in a fully
loaded jack setup the clients are using all the available time) and
jackd + the soundcard IRQ's RT priorities are higher than the timer
interrupt? Seems like you could starve the timer interrupt
indefinitely.
In fact, the only IRQ thread that currently _needs_ to be lower prio
than the others is IDE - the others all execute quickly enough to only
cause a problem at _extreme_ latencies that you would never use in the
real world, at least for audio/JACK. Last time I checked no other
hardirq ran for more than about 50 usecs. With Jens' patch to move IDE
bh processing into a softirq, I suspect the relative priorities of the
IRQ threads would not matter at all.
Lee
* Lee Revell <[email protected]> wrote:
> Hmm, maybe the timer interrupt should be 99 and the rest say 50.
> Wouldn't it be bad if you had a fully loaded jackd (DSP load in JACK
> is the proportion of the process cycle to the period time; in a fully
> loaded jack setup the clients are using all the available time) and
> jackd + the soundcard IRQ's RT priorities are higher than the timer
> interrupt? Seems like you could starve the timer interrupt
> indefinitely.
i misspoke when i said timer interrupt - that one is the lone
non-threaded IRQ so its priority is 100 in essence.
Ingo
* Ingo Molnar <[email protected]> wrote:
> ok, could you try the -RT-V0.6.0 patch i've just uploaded? It could i
> believe improve these latencies.
hm, CONFIG_PARPORT_1284 seems broken, and USB too, it locks up during
bootup. Investigating ...
Ingo
On Sun, 2004-10-31 at 13:48 +0100, Ingo Molnar wrote:
> * Ingo Molnar <[email protected]> wrote:
>
> > ok, could you try the -RT-V0.6.0 patch i've just uploaded? It could i
> > believe improve these latencies.
>
> hm, CONFIG_PARPORT_1284 seems broken, and USB too, it locks up during
> bootup. Investigating ...
>
FWIW with V0.5.16 I had several hard lockups when running Florian's test
app at 2048 Hz.
Lee
* Ingo Molnar <[email protected]> wrote:
> > ok, could you try the -RT-V0.6.0 patch i've just uploaded? It could i
> > believe improve these latencies.
>
> hm, CONFIG_PARPORT_1284 seems broken, and USB too, it locks up during
> bootup. Investigating ...
ok, fixed it and uploaded -V0.6.1.
Ingo
* Lee Revell <[email protected]> wrote:
> > > ok, could you try the -RT-V0.6.0 patch i've just uploaded? It could i
> > > believe improve these latencies.
> >
> > hm, CONFIG_PARPORT_1284 seems broken, and USB too, it locks up during
> > bootup. Investigating ...
> >
>
> FWIW with V0.5.16 I had several hard lockups when running Florian's
> test app at 2048 Hz.
please check out V0.6.1, i made the semaphore code more robust and more
compatible. The V0.6 series could also fix the XFS and reiserfs problems
reported.
Ingo
On Sun, 31 Oct 2004 13:48:28 +0100
Ingo Molnar <[email protected]> wrote:
>
> * Ingo Molnar <[email protected]> wrote:
>
> > ok, could you try the -RT-V0.6.0 patch i've just uploaded? It could i
> > believe improve these latencies.
>
> hm, CONFIG_PARPORT_1284 seems broken, and USB too, it locks up during
> bootup. Investigating ...
i will try anyways since i don't use neither USB nor my parallel port :)
also, i have uploaded an overhauled version of the wakeup timer program. It
now deferres all output to another SCHED_FIFO thread [with prio 1 lower than
the main process]. The data is passed via a lockless fifo [ripped from jack
sourcecode]. Also it handles commandline options better and has sensible [?]
defaults:
~/source/my_projects/wakeup$ ./rtc_wakeup -h
usage: wakeup [options]
options:
-f freqency(hz) (default 1024)
-p realtime prio (default 90(91))
-n max number of interrupts (default 0: run until stopped)
-t jitter threshold (%) (default 5)
-o history_file (default /dev/null)
-h show help
The "history" file contains three rows: irq #, cycle count at wakeup,
jitter(%). The first few entries are probably off [due to startup stuff, all
reporting during runtime takes only irq's after the third into account, to
avoid the startup mess]..
grab it at http://affenbande.org/~tapas/rtc_wakeup.tgz
Here's a typical run (still on V0.5.16, will try V0.6 now):
~/source/my_projects/rtc_wakeup$ ./rtc_wakeup
rtc_wakeup - press ctrl-c to stop
freq: 1024
max # of irqs: 0 (run until stopped)
jitter threshold: 5%
output filename: /dev/null
rt priority: 90(91)
getting cpu speed
1194909286.409 Hz (1194.909 MHz)
# of cycles for "perfect" period: 1166903
setting up ringbuffer
setting up consumer thread
setting up /dev/rtc.
locking memory...
turning irq on, beginning measurement (might take a while).
new max. jitter: 0.268231%
threshold violated: 2523891286957 (26.3406%)
new max. jitter: 26.3406%
threshold violated: 2523892156958 (25.4436%)
threshold violated: 2523896074240 (36.5712%)
new max. jitter: 36.5712%
threshold violated: 2523896831195 (35.1313%)
threshold violated: 2523899757527 (50.2766%)
new max. jitter: 50.2766%
threshold violated: 2523900328433 (51.0751%)
new max. jitter: 51.0751%
threshold violated: 2523903338753 (57.0041%)
new max. jitter: 57.0041%
threshold violated: 2523903827419 (58.1228%)
new max. jitter: 58.1228%
threshold violated: 2523905321471 (28.0357%)
threshold violated: 2523906161154 (28.0417%)
threshold violated: 2523909924647 (22.6959%)
threshold violated: 2523911166386 (6.41322%)
threshold violated: 2523911990002 (29.4186%)
threshold violated: 2523913267054 (9.43943%)
threshold violated: 2523914321799 (9.6116%)
threshold violated: 2523915653813 (14.1495%)
threshold violated: 2523916760657 (5.14687%)
threshold violated: 2523917830826 (8.28981%)
threshold violated: 2523920520433 (29.74%)
threshold violated: 2523921329137 (30.6966%)
done.
total # of irqs: 2754
missed irqs: 0
threshold violations: 20
max jitter: 58.1228%
flo
* Florian Schmidt <[email protected]> wrote:
> also, i have uploaded an overhauled version of the wakeup timer
> program. It now deferres all output to another SCHED_FIFO thread [with
> prio 1 lower than the main process]. The data is passed via a lockless
> fifo [ripped from jack sourcecode]. Also it handles commandline
> options better and has sensible [?] defaults:
>
> ~/source/my_projects/wakeup$ ./rtc_wakeup -h
> usage: wakeup [options]
> options:
> -f freqency(hz) (default 1024)
> -p realtime prio (default 90(91))
> -n max number of interrupts (default 0: run until stopped)
> -t jitter threshold (%) (default 5)
> -o history_file (default /dev/null)
> -h show help
cool! May i have a feature request :-) It would be quite useful to have
an option to see the jitter output in percentage and in microseconds as
well. I.e. an option to have such a format:
late wakeup: +151.3 usecs (14.6%) jitter.
cycles are too large to be human readable, and absolute values are
harder to read when HZ is not 1024. So to get a 'quick feel' of the
delays in a system the above line would be the most informative (for me
that is). Also, the percentage can go back to %.1f i think, instead of
%.5f or so - it's good to have a decimal point in the percentage but one
number after it is more than enough.) The more compact the output, the
fewer useless info, the quicker the human brain can read it. (Obviously
to generate a nice graph out of it needs a different format.)
> Here's a typical run (still on V0.5.16, will try V0.6 now):
please try -V0.6.1, -V0.6 had a pretty stupid bug that could trigger
frequently.
> threshold violations: 20
> max jitter: 58.1228%
here it would also be useful to have the 'max jitter' in usecs. I.e. in
the above case it was somewhere around ~570 usecs.
Ingo
On Sun, 31 Oct 2004 14:22:37 +0100
Ingo Molnar <[email protected]> wrote:
> cool! May i have a feature request :-) It would be quite useful to have
> an option to see the jitter output in percentage and in microseconds as
> well. I.e. an option to have such a format:
>
> late wakeup: +151.3 usecs (14.6%) jitter.
yep, will do so ater trying the 0.6.1 kernel.
> cycles are too large to be human readable, and absolute values are
> harder to read when HZ is not 1024. So to get a 'quick feel' of the
> delays in a system the above line would be the most informative (for me
> that is). Also, the percentage can go back to %.1f i think, instead of
> %.5f or so - it's good to have a decimal point in the percentage but one
> number after it is more than enough.) The more compact the output, the
> fewer useless info, the quicker the human brain can read it. (Obviously
> to generate a nice graph out of it needs a different format.)
well, to generate graphs out of the data one can choose to write the raw
cycle counts into a "history file". Then one can process this raw data to
hearts content. i will update the output routines to truncate the percentage
to %.1f [or the c++ equivalent :)]
>
> > Here's a typical run (still on V0.5.16, will try V0.6 now):
>
> please try -V0.6.1, -V0.6 had a pretty stupid bug that could trigger
> frequently.
>
> > threshold violations: 20
> > max jitter: 58.1228%
>
> here it would also be useful to have the 'max jitter' in usecs. I.e. in
> the above case it was somewhere around ~570 usecs.
ack.
flo
* Florian Schmidt <[email protected]> wrote:
> also, i have uploaded an overhauled version of the wakeup timer
> program. It now deferres all output to another SCHED_FIFO thread [with
> prio 1 lower than the main process]. The data is passed via a lockless
> fifo [ripped from jack sourcecode]. [...]
it works really great - previously i had problems with 'secondary'
latencies when running the app with HZ=8192, now i get only the primary
message(s).
i also like it that if you run the app it does a very useful type of
measurement right out of box. The simpler you make a validation app for
kernel developers the more they will use it ;-)
[ btw., FC3's gcc produces this (harmless) warning during the build:
cc1plus: warning: command line option "-Wstrict-prototypes" is valid
for Ada/C/ObjC but not for C++ ]
Ingo
* Ingo Molnar <[email protected]> wrote:
> > FWIW with V0.5.16 I had several hard lockups when running Florian's
> > test app at 2048 Hz.
>
> please check out V0.6.1, i made the semaphore code more robust and
> more compatible. The V0.6 series could also fix the XFS and reiserfs
> problems reported.
i've just uploaded V0.6.2 that fixes a console-unblanking-timer thinko.
This bug was present for quite some time, but this is the first time it
triggered on my testbox - might be more common on others.
Ingo
On Sun, 31 Oct 2004 14:40:16 +0100
Ingo Molnar <[email protected]> wrote:
>
> * Ingo Molnar <[email protected]> wrote:
>
> > > FWIW with V0.5.16 I had several hard lockups when running Florian's
> > > test app at 2048 Hz.
> >
> > please check out V0.6.1, i made the semaphore code more robust and
> > more compatible. The V0.6 series could also fix the XFS and reiserfs
> > problems reported.
>
> i've just uploaded V0.6.2 that fixes a console-unblanking-timer thinko.
> This bug was present for quite some time, but this is the first time it
> triggered on my testbox - might be more common on others.
V0.6.2 works pretty good. max jitter until now 21% [205us]. still fiddling
with the output formatting for rtc_wakeup.
flo
On Sun, 31 Oct 2004 16:20:59 +0100
Florian Schmidt <[email protected]> wrote:
>
> V0.6.2 works pretty good. max jitter until now 21% [205us]. still fiddling
> with the output formatting for rtc_wakeup.
>
i got a deadlock though. it was a weird one. mouse and keyboard [including
sysrq] froze. but the find / i started kept on running in an xterm. had to
press reset due to lack of second machine..
flo
p.s. new rtc_wakeup version uploaded, which shows the percentage converted
to usecs (always positive, you need the sign?). btw: with V0.6.2 i sometimes
see jitter > 100% but still no lost irq (/dev/rtc still reports only 1
delivered irq on next wakeup). I cannot provke lost irqs with -f up to 2048.
With -f 8192 i do get lost irq's [not amazing though].
On Sun, 31 Oct 2004 16:59:13 +0100
Florian Schmidt <[email protected]> wrote:
> p.s. new rtc_wakeup version uploaded, which shows the percentage converted
> to usecs (always positive, you need the sign?). btw: with V0.6.2 i sometimes
> see jitter > 100% but still no lost irq (/dev/rtc still reports only 1
> delivered irq on next wakeup). I cannot provke lost irqs with -f up to 2048.
> With -f 8192 i do get lost irq's [not amazing though].
another update upped. small bugfix (still depended on jack, should work now
w/o jack installed) plus minor cosmetic corrections. Here's example output
for 2.6.8.1-P9 (under heavy load (multiple finds plus kernel compile plus UI
artistics):
~/source/my_projects/rtc_wakeup$ ./rtc_wakeup
rtc_wakeup - press ctrl-c to stop
freq: 1024
max # of irqs: 0 (run until stopped)
jitter threshold: 5% (48 usec)
output filename: /dev/null
rt priority: 90(91)
getting cpu speed
1194913254.155 Hz (1194.913 MHz)
# of cycles for "perfect" period: 1166907 (976 usec)
setting up ringbuffer
setting up consumer thread
setting up /dev/rtc
locking memory
turning irq on, beginning measurement
new max. jitter: 1.6% (15 usec)
new max. jitter: 1.9% (18 usec)
new max. jitter: 2.6% (25 usec)
new max. jitter: 2.9% (28 usec)
new max. jitter: 3.2% (31 usec)
new max. jitter: 3.3% (31 usec)
new max. jitter: 3.3% (32 usec)
new max. jitter: 3.5% (34 usec)
new max. jitter: 3.5% (34 usec)
new max. jitter: 3.7% (35 usec)
new max. jitter: 4.3% (41 usec)
new max. jitter: 4.9% (47 usec)
threshold violated: 5.6% (54usec)
new max. jitter: 5.6% (54 usec)
threshold violated: 5.3% (51usec)
done.
total # of irqs: 168072
missed irqs: 0
threshold violations: 2
max jitter: 5.6% (54 usec)
So basically like lee said, 7% seems to be a normal upper limit for VP
boxes (there might be spikes though).
V0.6.2 looked good, too, until it locked up again (under heavy load with
rtc_wakeup running) ;) Will build one with debugging enabled tomorrow.
flo
Ingo Molnar wrote:
>
> ok, could you try the -RT-V0.6.0 patch i've just uploaded? It could i
> believe improve these latencies.
>
Just made a try with RT-V0.6.2 and it locks pretty easily on my P4/UP
laptop while running my jackd-R + 9*fluidsynth workload. These tests are
setup to run for 5 min. Not once it got through, and I've tried several
times now.
Although it's been 100% reproducible, the system lockup varies in tine
from the start of the test. Unfortunately it says nothing through
netconsole, so there's no additional info/dump/trace to give you for
analysis.
Most of the kernel hacking/debugging options are unset (N), except
Magic-SysRq. Hitting on SysRq+T, right after when system hangs, comes out
with absolutely nothing, only this line (via netconsole):
SysRq : Show State
Guess something's got wrong, yet again.
--
rncbc aka Rui Nuno Capela
[email protected]
Ingo Molnar wrote:
> * Florian Schmidt <[email protected]> wrote:
>
>
>>On Sat, 30 Oct 2004 17:26:06 -0400
>>Lee Revell <[email protected]> wrote:
>>
>>
>>>OK this is pretty sweet. With T3 the jitter never exceeds 7% on an idle
>>>system. As soon as I start moving the mouse this goes to 7 or 8%. I
>>>cannot get it to go higher than 10%. Moving windows around has no
>>>effect, the highest jitter happens when I type or move the mouse really
>>>fast IOW it corresponds to the interrupt rate.
>>>
>>>This is a pretty good baseline for what an xrun-free system would look
>>>like. Now to test the latest version...
>>
>>Well,
>>
>>on V0.5.16 i see something like the below output (which is much worse). It
>>seems that missed irq's with rtc show up at the same time as the xruns in
>>jackd do [i ran both jackd and wakeup in parallel].
>
>
> ok, could you try the -RT-V0.6.0 patch i've just uploaded? It could i
> believe improve these latencies.
>
> Ingo
> -
I am not able to get V0.6.2 to boot on my SMP system here. There is no
indication as to why. It just locks. I don't see anything in the log
that gives an indication as to why so I am only attaching the end of the
log. If you would like all of it, just let me know. I am going to try
building it for my SMP system now.
kr
Oct 31 09:28:05 porky vsftpd: vsftpd vsftpd succeeded
Oct 31 09:28:06 porky xinetd[2824]: xinetd Version 2.3.13 started with
libwrap loadavg options compiled in.
Oct 31 09:28:06 porky xinetd[2824]: Started working: 2 available services
Oct 31 09:28:06 porky sendmail: sendmail startup succeeded
Oct 31 09:28:06 porky sendmail: sm-client startup succeeded
Oct 31 09:28:07 porky gpm[2897]: *** info [startup.c(95)]:
Oct 31 09:28:07 porky gpm[2897]: Started gpm successfully. Entered
daemon mode.
Oct 31 09:28:07 porky gpm[2897]: *** info [mice.c(1766)]:
Oct 31 09:28:07 porky gpm[2897]: imps2: Auto-detected intellimouse PS/2
Oct 31 09:28:07 porky gpm: gpm startup succeeded
Oct 31 09:28:07 porky crond: crond startup succeeded
Oct 31 09:28:08 porky xfs: xfs startup succeeded
Oct 31 09:28:08 porky anacron: anacron startup succeeded
Oct 31 09:28:09 porky atd: atd startup succeeded
Oct 31 09:28:09 porky readahead: Starting background readahead:
Oct 31 09:28:09 porky rc: Starting readahead: succeeded
Oct 31 09:28:10 porky messagebus: messagebus startup succeeded
Ingo Molnar wrote:
> i've just uploaded V0.6.2 that fixes a console-unblanking-timer thinko.
> This bug was present for quite some time, but this is the first time it
> triggered on my testbox - might be more common on others.
>
> Ingo
Hi Ingo,
I finally got to test your patch again. Now I've been running -V0.6.2
for almost two hours and have not yet encountered a single deadlock.
So it seems that the netfilter deadlock, that I could easily reproduce
in -V0.4.1, is solved.
There is one strange thing, though:
michich@k4-912b:~$ uptime
00:09:17 up 1:49, 7 users, load average: 707.72, 706.40, 682.16
In fact my computer is mostly idle.
ps shows no zombies nor any D-state processes. The system runs fine.
Michal
Hi Ingo,
On Sunday 31 October 2004 14.40, Ingo Molnar wrote:
> * Ingo Molnar <[email protected]> wrote:
>
> i've just uploaded V0.6.2 that fixes a console-unblanking-timer
> thinko. This bug was present for quite some time, but this is the
> first time it triggered on my testbox - might be more common on
> others.
I'm trying to boot V0.6.2 on my laptop, which doesn't have a serial
port, and I got a hardlock on boot. So I setup netconsole, which also
hardlocks the machine, so I'm back to U9.3 for now..
It's a different hardlock when booting without netconsole. SysRq keys
doesn't work.
Config:
CONFIG_PREEMPT=y
CONFIG_PREEMPT_SOFTIRQS=y
CONFIG_PREEMPT_HARDIRQS=y
CONFIG_PREEMPT_REALTIME=y
CONFIG_PREEMPT_BKL=y
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_PREEMPT=y
CONFIG_PREEMPT_TIMING=y
CONFIG_PREEMPT_TRACE=y
CONFIG_DEBUG_STACKOVERFLOW=y
CONFIG_DEBUG_STACK_USAGE=y
No USB or PARPORT support in kernel.
Here's the boot log until it hardlock:
Linux version 2.6.9-mm1-RT-V0.6.2 (root@barbara) (gcc version 3.3.3
(SuSE Linux)) #12 SMP Mon Nov 1 01:57:57 CET 2004
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000ce000 - 00000000000d0000 (reserved)
BIOS-e820: 00000000000d8000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000001f6e0000 (usable)
BIOS-e820: 000000001f6e0000 - 000000001f6eb000 (ACPI data)
BIOS-e820: 000000001f6eb000 - 000000001f700000 (ACPI NVS)
BIOS-e820: 000000001f700000 - 0000000020000000 (reserved)
BIOS-e820: 00000000fec10000 - 00000000fec20000 (reserved)
BIOS-e820: 00000000ff800000 - 00000000ffc00000 (reserved)
BIOS-e820: 00000000fffffc00 - 0000000100000000 (reserved)
502MB LOWMEM available.
DMI present.
ACPI: PM-Timer IO Port: 0x1008
Built 1 zonelists
No local APIC present or hardware disabled
Initializing CPU#0
Kernel command line: vga=0x314 resume=/dev/hda9 nmi_watchdog=1
[email protected]/eth0,[email protected]/00:50:BA:C2:11:34
netconsole: local port 1234
netconsole: local IP 10.0.3.6
netconsole: interface eth0
netconsole: remote port 4567
netconsole: remote IP 10.0.2.1
netconsole: remote ethernet address 00:50:ba:c2:11:34
PID hash table entries: 2048 (order: 11, 32768 bytes)
Detected 1500.066 MHz processor.
Using pmtmr for high-res timesource
Console: colour dummy device 80x25
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
Memory: 502708k/514944k available (2931k kernel code, 11848k reserved,
807k data, 228k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor
mode... Ok.
Security Framework v1.0.0 initialized
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
CPU: L1 I cache: 32K, L1 D cache: 32K
CPU: L2 cache: 1024K
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... OK.
ACPI: IRQ9 SCI: Edge set to Level Trigger.
CPU0: Intel(R) Pentium(R) M processor 1500MHz stepping 05
per-CPU timeslice cutoff: 2924.02 usecs.
task migration cache decay timeout: 3 msecs.
SMP motherboard not detected.
Local APIC not detected. Using dummy APIC emulation.
spawn_desched_task(00000000)
desched cpu_callback 3/00000000
ksoftirqd started up.
softirq RT prio: 24.
desched cpu_callback 2/00000000
Brought up 1 CPUs
desched thread 0 started up.
NET: Registered protocol family 16
PCI: PCI BIOS revision 2.10 entry at 0xfd6c4, last bus=3
PCI: Using configuration type 1
mtrr: v2.0 (20020519)
ACPI: Subsystem revision 20040816
ACPI: Interpreter enabled
ACPI: Using PIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (00:00)
PCI: Probing PCI hardware (bus 00)
PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.1
PCI: Transparent bridge - 0000:00:1e.0
ACPI: PCI Interrupt Link [LNKA] (IRQs 10 *11)
ACPI: PCI Interrupt Link [LNKB] (IRQs *10 11)
ACPI: PCI Interrupt Link [LNKC] (IRQs 10 *11)
ACPI: PCI Interrupt Link [LNKD] (IRQs *11)
ACPI: PCI Interrupt Link [LNKE] (IRQs *10 11)
ACPI: PCI Interrupt Link [LNKF] (IRQs *10 11)
ACPI: PCI Interrupt Link [LNKG] (IRQs *10 11)
ACPI: PCI Interrupt Link [LNKH] (IRQs 10 *11)
ACPI: Embedded Controller [EC] (gpe 28)
Linux Plug and Play Support v0.97 (c) Adam Belay
PCI: Using ACPI for IRQ routing
** PCI interrupts are no longer routed automatically. If this
** causes a device to stop working, it is probably because the
** driver failed to call pci_enable_device(). As a temporary
** workaround, the "pci=routeirq" argument restores the old
** behavior. If this argument makes the device work again,
** please email the output of "lspci" to [email protected]
** so I can fix the driver.
Simple Boot Flag at 0x36 set to 0x1
apm: BIOS version 1.2 Flags 0x03 (Driver version 1.16ac)
apm: overridden by ACPI.
audit: initializing netlink socket (disabled)
audit(1099274447.119:0): initialized
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
Initializing Cryptographic API
vesafb: framebuffer at 0xe8000000, mapped to 0xe0080000, using 1875k,
total 8000k
vesafb: mode is 800x600x16, linelength=1600, pages=7
vesafb: protected mode interface info at 00ff:44f0
vesafb: scrolling: redraw
vesafb: Truecolor: size=0:5:6:5, shift=0:11:5:0
Console: switching to colour frame buffer device 100x37
fb0: VESA VGA frame buffer device
isapnp: Scanning for PnP cards...
isapnp: No Plug & Play device found
Real Time Clock Driver v1.12
ACPI: PS/2 Keyboard Controller [PS2K] at I/O 0x60, 0x64, irq 1
ACPI: PS/2 Mouse Controller [PS2M] at irq 12
i8042.c: Detected active multiplexing controller, rev 1.1.
serio: i8042 AUX0 port at 0x60,0x64 irq 12
serio: i8042 AUX1 port at 0x60,0x64 irq 12
serio: i8042 AUX2 port at 0x60,0x64 irq 12
serio: i8042 AUX3 port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
Serial: 8250/16550 driver $Revision: 1.90 $ 48 ports, IRQ sharing
enabled
ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10
PCI: setting IRQ 10 as level-triggered
ACPI: PCI interrupt 0000:00:1f.6[B] -> GSI 10 (level, low) -> IRQ 10
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
RAMDISK driver initialized: 16 RAM disks of 64000K size 1024 blocksize
loop: loaded (max 8 devices)
b44.c:v0.95 (Aug 3, 2004)
ACPI: PCI Interrupt Link [LNKE] enabled at IRQ 10
ACPI: PCI interrupt 0000:02:05.0[A] -> GSI 10 (level, low) -> IRQ 10
eth0: Broadcom 4400 10/100BaseT Ethernet 00:0a:e4:27:0f:c2
netconsole: device eth0 not up yet, forcing it
netconsole: carrier detect appears flaky, waiting 10 seconds
b44: eth0: Link is down.
b44: eth0: Link is up at 10 Mbps, half duplex.
b44: eth0: Flow control is off for TX and off for RX.
IRQ#10 thread RT prio: 49.
/ Magnus M??tt?
* Florian Schmidt <[email protected]> wrote:
> i got a deadlock though. it was a weird one. mouse and keyboard
> [including sysrq] froze. but the find / i started kept on running in
> an xterm. had to press reset due to lack of second machine..
i've uploaded -V0.6.3 to the usual place:
http://redhat.com/~mingo/realtime-preempt/
which fixes two issues:
- priorities of SCHED_OTHER tasks not getting properly managed, hence
they could end up 'starving' other SCHED_OTHER tasks pretty
indefinitely. This could possibly solve the 'temporary lockup'
problem reported by Mark H Johnson.
- fixed the 'high load average' bug
i dont know whether this will solve the 'hard lockups' reported though.
It could solve your problem because the 'find /' kept running so it
wasnt a hard lockup. I'll keep testing and i'll fix any problem i can
reproduce myself.
Ingo
* Ingo Molnar <[email protected]> wrote:
> i've uploaded -V0.6.3 to the usual place:
>
> http://redhat.com/~mingo/realtime-preempt/
hm, make it -V0.6.4. -V0.6.3 had a bug in where the system clock
'stopped' a couple of minutes after bootup, creating weird effects. (it
stopped when jiffies overflowed from negative territory they default to
at bootup, into positive numbers.)
Ingo
On Mon, 2004-11-01 at 13:37 +0100, Ingo Molnar wrote:
> * Ingo Molnar <[email protected]> wrote:
>
> > i've uploaded -V0.6.3 to the usual place:
> >
> > http://redhat.com/~mingo/realtime-preempt/
>
> hm, make it -V0.6.4. -V0.6.3 had a bug in where the system clock
> 'stopped' a couple of minutes after bootup, creating weird effects. (it
> stopped when jiffies overflowed from negative territory they default to
> at bootup, into positive numbers.)
>
> Ingo
I'm doing some tests from my RT environment on 0.6.2. I'm quite sure,
that interrupts are sporadically disabled for > 200?s. Did you change
anything relevant to this between 0.6.2 and 0.6.4 ?
I also had a freeze of the machine. Sysrq-T was the only thing that
worked.
tglx
sibling
task PC pid father child younger older
init [c113d6f0]D C03D9540 0 1 0 2
(NOTLB)
c113fe5c 00000086 c113d6f0 c03d9540 00000246 c113d6f0 00000001 c113fea4
c012007c c0326060 c113fea4 c01c9988 c113fea4 00001f7c 26c698f9
000002d0
c113d8a4 c113e000 c113fe98 0000000b c113fe80 c02d49fb 00000296
c0327ff4
Call Trace:
[<c012007c>] __mod_timer+0x11c/0x1e0 (36)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c02d49fb>] schedule+0x3b/0x130 (40)
[<c02d5678>] schedule_timeout+0x88/0xe0 (36)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (60)
[<c0120d30>] process_timeout+0x0/0x10 (20)
[<c0169ec7>] do_select+0x197/0x2f0 (28)
[<c0169b60>] __pollwait+0x0/0xd0 (84)
[<c016a33b>] sys_select+0x2db/0x4f0 (32)
[<c010602b>] syscall_call+0x7/0xb (92)
ksoftirqd/0 [c113d140]R C03D9540 0 2 1 3
(L-TLB)
c7fe3f88 00000046 c113d140 c03d9540 c7fe3f6c c0326370 c7fe3f6c c7fe2000
c0120a86 00000000 c0364580 00000000 c7fe2000 000000d5 02053ba5
000002d1
c113d2f4 c7fe2000 c7fe2000 00000000 c7fe3fac c02d49fb c011cad0
00000000
Call Trace:
[<c0120a86>] run_timer_softirq+0x2a6/0x460 (36)
[<c02d49fb>] schedule+0x3b/0x130 (52)
[<c011cad0>] ksoftirqd+0x0/0xe0 (4)
[<c011cad0>] ksoftirqd+0x0/0xe0 (28)
[<c011cb96>] ksoftirqd+0xc6/0xe0 (4)
[<c012ca7a>] kthread+0xaa/0xb0 (20)
[<c012c9d0>] kthread+0x0/0xb0 (28)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
desched/0 [c113cb90]D C03D9540 0 3 1 4
2 (L-TLB)
c7fe5f88 00000046 c113cb90 c03d9540 c013f0dd c7ff9a40 c1137ee4 c012d597
c7ff9a40 c1137ee4 c013f0dd c1137ee4 c135a0d0 00000107 95a45f2d
0000029e
c113cd44 c7fe4000 c7fe4000 00000000 c7fe5fac c02d49fb c2d1e7a4
c0116cf3
Call Trace:
[<c013f0dd>] kmem_cache_free+0x4d/0xe0 (20)
[<c012d597>] _mutex_lock+0x17/0x20 (12)
[<c013f0dd>] kmem_cache_free+0x4d/0xe0 (12)
[<c02d49fb>] schedule+0x3b/0x130 (44)
[<c0116cf3>] mmdrop_complete+0x73/0x110 (8)
[<c0116de0>] desched_thread+0x0/0x90 (24)
[<c0116e4e>] desched_thread+0x6e/0x90 (4)
[<c012ca7a>] kthread+0xaa/0xb0 (20)
[<c012c9d0>] kthread+0x0/0xb0 (28)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
events/0 [c113c5e0]R C03D9570 0 4 1 5
3 (L-TLB)
c7fedf20 00000046 c113d140 c03d9570 000002d1 00000000 00000246 c113c5e0
00000246 c113c5e0 015f136e 000002d1 c113d140 000002f7 015f3f88
000002d1
c113c794 c7fec000 c7fca740 c7fedf94 c7fedf44 c02d49fb c7fedf94
c7fca77c
Call Trace:
[<c02d49fb>] schedule+0x3b/0x130 (88)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (12)
[<c0127fe8>] worker_thread+0x258/0x280 (24)
[<c01f8f60>] console_callback+0x0/0xe0 (24)
[<c0113460>] default_wake_function+0x0/0x30 (28)
[<c0113460>] default_wake_function+0x0/0x30 (32)
[<c0127d90>] worker_thread+0x0/0x280 (36)
[<c012ca7a>] kthread+0xaa/0xb0 (4)
[<c012c9d0>] kthread+0x0/0xb0 (28)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
khelper [c113c030]D C03D9540 0 5 1 10
4 (L-TLB)
c7fcdf20 00000046 c113c030 c03d9540 c113c030 00000000 00000246 c113c030
00000246 c113c030 00000001 c7fca67c c113d140 00004693 32e90bc6
00000023
c113c1e4 c7fcc000 c7fca640 c7fcdf94 c7fcdf44 c02d49fb c7fcdf94
c7fca67c
Call Trace:
[<c02d49fb>] schedule+0x3b/0x130 (88)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (12)
[<c0127fe8>] worker_thread+0x258/0x280 (24)
[<c0127910>] __call_usermodehelper+0x0/0x70 (24)
[<c0113460>] default_wake_function+0x0/0x30 (28)
[<c0113460>] default_wake_function+0x0/0x30 (32)
[<c0127d90>] worker_thread+0x0/0x280 (36)
[<c012ca7a>] kthread+0xaa/0xb0 (4)
[<c012c9d0>] kthread+0x0/0xb0 (28)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
kthread [c7fd1710]D C03D9540 0 10 1 24 77
5 (L-TLB)
c7fd3f20 00000046 c7fd1710 c03d9540 c7fd1710 00000000 00000246 c7fd1710
00000246 c7fd1710 00000001 c7fca57c c7d05770 00001363 36d5ab71
00000023
c7fd18c4 c7fd2000 c7fca540 c7fd3f94 c7fd3f44 c02d49fb c7fd3f94
c7fca57c
Call Trace:
[<c02d49fb>] schedule+0x3b/0x130 (88)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (12)
[<c0127fe8>] worker_thread+0x258/0x280 (24)
[<c012ca80>] keventd_create_kthread+0x0/0x60 (24)
[<c0113460>] default_wake_function+0x0/0x30 (28)
[<c0113460>] default_wake_function+0x0/0x30 (32)
[<c0127d90>] worker_thread+0x0/0x280 (36)
[<c012ca7a>] kthread+0xaa/0xb0 (4)
[<c012c9d0>] kthread+0x0/0xb0 (28)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
kblockd/0 [c7fd1160]D C03D9540 0 24 10 75
(L-TLB)
c7fdff20 00000046 c7fd1160 c03d9540 c7fd1160 00000000 00000246 c7fd1160
00000246 c7fd1160 00000001 c7fca37c c135ac30 0000228e 9bb8ebd4
00000190
c7fd1314 c7fde000 c7fca340 c7fdff94 c7fdff44 c02d49fb c7fdff94
c7fca37c
Call Trace:
[<c02d49fb>] schedule+0x3b/0x130 (88)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (12)
[<c0127fe8>] worker_thread+0x258/0x280 (24)
[<c0215410>] as_work_handler+0x0/0x60 (24)
[<c0113460>] default_wake_function+0x0/0x30 (28)
[<c0113460>] default_wake_function+0x0/0x30 (32)
[<c0127d90>] worker_thread+0x0/0x280 (36)
[<c012ca7a>] kthread+0xaa/0xb0 (4)
[<c012c9d0>] kthread+0x0/0xb0 (28)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
pdflush [c7fd0bb0]D C03D9540 0 75 10 76
24 (L-TLB)
c7f43f5c 00000046 c7fd0bb0 c03d9540 c013b0c8 c013bd30 00000246 c7fd0bb0
00000ccc 00002146 00000000 00000000 c0d28c30 00001fb6 bba27e42
00000190
c7fd0d64 c7f42000 c7f43fb0 c7f43fa4 c7f43f80 c02d49fb c0328500
00000001
Call Trace:
[<c013b0c8>] background_writeout+0x98/0xc0 (20)
[<c013bd30>] pdflush+0x0/0x30 (4)
[<c02d49fb>] schedule+0x3b/0x130 (64)
[<c013bbff>] __pdflush+0x7f/0x1b0 (12)
[<c012d5d7>] _mutex_lock_irq+0x17/0x20 (12)
[<c013bd30>] pdflush+0x0/0x30 (8)
[<c013bc04>] __pdflush+0x84/0x1b0 (4)
[<c013bd56>] pdflush+0x26/0x30 (20)
[<c013bd30>] pdflush+0x0/0x30 (40)
[<c012ca7a>] kthread+0xaa/0xb0 (4)
[<c012c9d0>] kthread+0x0/0xb0 (28)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
pdflush [c7fd0600]D C03D9540 0 76 10 78
75 (L-TLB)
c7f57f5c 00000046 c7fd0600 c03d9540 002a8188 c013bd30 00000246 c7fd0600
c0328420 002a8188 0029f8d0 00000000 c113d140 000003b9 2698f358
000002d0
c7fd07b4 c7f56000 c7f57fb0 c7f57fa4 c7f57f80 c02d49fb c0328500
00000001
Call Trace:
[<c013bd30>] pdflush+0x0/0x30 (24)
[<c02d49fb>] schedule+0x3b/0x130 (64)
[<c013bbff>] __pdflush+0x7f/0x1b0 (12)
[<c012d5d7>] _mutex_lock_irq+0x17/0x20 (12)
[<c013bd30>] pdflush+0x0/0x30 (8)
[<c013bc04>] __pdflush+0x84/0x1b0 (4)
[<c013bd56>] pdflush+0x26/0x30 (20)
[<c013bd30>] pdflush+0x0/0x30 (40)
[<c012ca7a>] kthread+0xaa/0xb0 (4)
[<c012c9d0>] kthread+0x0/0xb0 (28)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
aio/0 [c7f5b730]D C03D9540 0 78 10 664
76 (L-TLB)
c7f5df20 00000046 c7f5b730 c03d9540 00000000 00000000 00000246 c7f5b730
00000246 c7f5b730 00000001 c7fca07c c113d6f0 00003100 deb9886e
00000004
c7f5b8e4 c7f5c000 c7fca040 c7f5df94 c7f5df44 c02d49fb c7f5df94
c7fca07c
Call Trace:
[<c02d49fb>] schedule+0x3b/0x130 (88)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (12)
[<c0127fe8>] worker_thread+0x258/0x280 (24)
[<c0113460>] default_wake_function+0x0/0x30 (52)
[<c0113460>] default_wake_function+0x0/0x30 (32)
[<c0127d90>] worker_thread+0x0/0x280 (36)
[<c012ca7a>] kthread+0xaa/0xb0 (4)
[<c012c9d0>] kthread+0x0/0xb0 (28)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
kswapd0 [c7fd0050]D C03D9540 0 77 1 666
10 (L-TLB)
c7f59f60 00000046 c7fd0050 c03d9540 00000000 00000000 c0327ea0 00000246
c7fd0050 00000001 c03282ec c012cf9e c2314700 00000508 d7982974
00000190
c7fd0204 c7f58000 c7f59f98 c7f59fc0 c7f59f84 c02d49fb c03282ec
c012cf9e
Call Trace:
[<c012cf9e>] prepare_to_wait+0x5e/0x80 (48)
[<c02d49fb>] schedule+0x3b/0x130 (40)
[<c012cf9e>] prepare_to_wait+0x5e/0x80 (8)
[<c0142cb3>] kswapd+0xd3/0xe0 (28)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (32)
[<c0105f02>] ret_from_fork+0x6/0x14 (20)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (12)
[<c0142be0>] kswapd+0x0/0xe0 (24)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
IRQ 8 [c7f5b180]D C03D9570 0 664 10 672
78 (L-TLB)
c7f71f80 00000046 c7d04c10 c03d9570 00000009 00000246 00000000 c01fb371
c03e8090 0000001d e9bf49d0 00000009 c7d04c10 000043a0 e9bf57a8
00000009
c7f5b334 c7f70000 c0398d80 c0398d80 c7f71fa4 c02d49fb 00000008
c0398d80
Call Trace:
[<c01fb371>] rtc_interrupt+0xe1/0x130 (32)
[<c02d49fb>] schedule+0x3b/0x130 (56)
[<c0133e60>] do_irqd+0x0/0xd0 (32)
[<c0133f08>] do_irqd+0xa8/0xd0 (4)
[<c012ca7a>] kthread+0xaa/0xb0 (28)
[<c012c9d0>] kthread+0x0/0xb0 (28)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
IRQ 12 [c7f5a620]D C03D99E8 0 672 10 686
664 (L-TLB)
c7e0ff80 00000046 c113d6f0 c03d99e8 00000006 c1187400 000000fa 00000000
00000000 35fa4d60 0907682f 00000006 c113d6f0 000004f6 090772a7
00000006
c7f5a7d4 c7e0e000 c0398f00 c0398f00 c7e0ffa4 c02d49fb 0000000c
c0398f00
Call Trace:
[<c02d49fb>] schedule+0x3b/0x130 (88)
[<c0133e60>] do_irqd+0x0/0xd0 (32)
[<c0133f08>] do_irqd+0xa8/0xd0 (4)
[<c012ca7a>] kthread+0xaa/0xb0 (28)
[<c012c9d0>] kthread+0x0/0xb0 (28)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
IRQ 6 [c7f5a070]D C03D9540 0 686 10 709
672 (L-TLB)
c7e11f80 00000046 c7f5a070 c03d9540 0836a1c0 00000000 c021c610 c021c52b
c0303866 00000292 c7e15460 00000000 c113c5e0 0005a201 026a5625
00000005
c7f5a224 c7e10000 c0398cc0 c0398cc0 c7e11fa4 c02d49fb 00000006
c0398cc0
Call Trace:
[<c021c610>] reset_interrupt+0x0/0x90 (28)
[<c021c52b>] floppy_interrupt+0x14b/0x1b0 (4)
[<c02d49fb>] schedule+0x3b/0x130 (56)
[<c0133e60>] do_irqd+0x0/0xd0 (32)
[<c0133f08>] do_irqd+0xa8/0xd0 (4)
[<c012ca7a>] kthread+0xaa/0xb0 (28)
[<c012c9d0>] kthread+0x0/0xb0 (28)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
kseriod [c7f5abd0]D C03D9540 0 666 1 778
77 (L-TLB)
c7e0df6c 00000046 c7f5abd0 c03d9540 00000000 00000246 c7f5abd0 00000246
c7f5abd0 00000001 c0366b40 c012cf9e 00000286 00008752 01c979f5
00000005
c7f5ad84 c7e0c000 c7e0c000 c7e0c000 c7e0df90 c02d49fb c0366b40
c012cf9e
Call Trace:
[<c012cf9e>] prepare_to_wait+0x5e/0x80 (48)
[<c02d49fb>] schedule+0x3b/0x130 (40)
[<c012cf9e>] prepare_to_wait+0x5e/0x80 (8)
[<c01fcd3e>] serio_thread+0xce/0x110 (28)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (20)
[<c0105f02>] ret_from_fork+0x6/0x14 (20)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (12)
[<c01fcc70>] serio_thread+0x0/0x110 (24)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
IRQ 14 [c7e1d750]T C03D9540 0 709 10 741
686 (L-TLB)
c7e1fd98 00000046 c7e1d750 c03d9540 c03d9570 00000190 c03d9540 c6604330
c7e1fd84 c7e1fd94 00000086 c6604330 c6604330 00001fbe 9bc32661
00000190
c7e1d904 c7e1e000 00000000 c7e1d750 c7e1fdbc c02d49fb c03d9570
00000000
Call Trace:
[<c02d49fb>] schedule+0x3b/0x130 (88)
[<c02d5c32>] down_write_mutex+0x102/0x1b0 (36)
[<c013b98d>] test_clear_page_writeback+0x3d/0xe0 (20)
[<c012d7bc>] _rw_mutex_write_lock_irqsave+0xc/0x10 (56)
[<c013b98d>] test_clear_page_writeback+0x3d/0xe0 (4)
[<c01572c0>] end_buffer_async_write+0x70/0x140 (8)
[<c0134e59>] end_page_writeback+0x29/0x70 (28)
[<c01572c8>] end_buffer_async_write+0x78/0x140 (20)
[<c015ab9d>] bio_destructor+0x3d/0x90 (16)
[<c015ae0d>] bio_put+0x3d/0x80 (24)
[<c015a409>] end_bio_bh_io_sync+0x39/0x50 (20)
[<c015bad3>] bio_endio+0x53/0x80 (16)
[<c020f825>] __end_that_request_first+0x205/0x270 (28)
[<c01c9988>] up_write_mutex+0x38/0xc0 (16)
[<c0239368>] __ide_end_request+0x78/0x1c0 (36)
[<c0239515>] ide_end_request+0x65/0xc0 (36)
[<c024226b>] ide_dma_intr+0x8b/0xb0 (36)
[<c02421e0>] ide_dma_intr+0x0/0xb0 (20)
[<c023b0b1>] ide_intr+0xe1/0x170 (4)
[<c0133624>] handle_IRQ_event+0x44/0x80 (32)
[<c0133dc3>] do_hardirq+0x63/0x100 (32)
[<c0133e60>] do_irqd+0x0/0xd0 (32)
[<c0133ef9>] do_irqd+0x99/0xd0 (4)
[<c012ca7a>] kthread+0xaa/0xb0 (28)
[<c012c9d0>] kthread+0x0/0xb0 (28)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
IRQ 1 [c7e1d1a0]D 00000000 0 741 10 779
709 (L-TLB)
00000015 00000000 00000000 00000000 c01fdff0 c1187200 00000046 00000000
00000000 15468ae0 00000001 c7e39de0 00000000 c0133624 00000001
c03e81c0
00000000 c0398ae0 00000001 c7e30000 00000000 c0133dc3 00000001
00000000
Call Trace:
[<c01fdff0>] i8042_interrupt+0xc0/0x1a0 (20)
[<c0133624>] handle_IRQ_event+0x44/0x80 (36)
[<c0133dc3>] do_hardirq+0x63/0x100 (32)
[<c0133e60>] do_irqd+0x0/0xd0 (32)
[<c0133ef9>] do_irqd+0x99/0xd0 (4)
[<c012ca7a>] kthread+0xaa/0xb0 (28)
[<c012c9d0>] kthread+0x0/0xb0 (28)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
kjournald [c7e1cbf0]T C03D9540 0 778 1 929
666 (L-TLB)
c7e43d68 00000046 c7e1cbf0 c03d9540 00000000 c212b8c0 00000246 c7e1cbf0
c7e1cbf0 00000001 c7f75580 c012cf9e c53879f0 0000068b ae093765
00000191
c7e1cda4 c7e42000 c7f75580 c7e43df4 c7e43d8c c02d49fb c7f75414
00000001
Call Trace:
[<c012cf9e>] prepare_to_wait+0x5e/0x80 (48)
[<c02d49fb>] schedule+0x3b/0x130 (40)
[<c01adf3b>] journal_commit_transaction+0x156b/0x16c0 (12)
[<c01adf40>] journal_commit_transaction+0x1570/0x16c0 (24)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (88)
[<c0133624>] handle_IRQ_event+0x44/0x80 (12)
[<c01cbde2>] __delay+0x12/0x20 (8)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (12)
[<c02d4671>] __schedule+0x2d1/0x620 (136)
[<c02d4b45>] preempt_schedule+0x55/0x70 (8)
[<c02d4b45>] preempt_schedule+0x55/0x70 (72)
[<c012d08f>] finish_wait+0x4f/0x70 (8)
[<c01b0bf6>] kjournald+0xf6/0x280 (24)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c01b0bf6>] kjournald+0xf6/0x280 (12)
[<c01b0c0c>] kjournald+0x10c/0x280 (24)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (40)
[<c0119910>] exit_notify+0x350/0x980 (12)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (20)
[<c0105f02>] ret_from_fork+0x6/0x14 (68)
[<c01b0b00>] kjournald+0x0/0x280 (8)
[<c01b0ae0>] commit_timeout+0x0/0x10 (8)
[<c01b0b00>] kjournald+0x0/0x280 (20)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
IRQ 4 [c7e1c640]D C03D9540 0 779 10 994
741 (L-TLB)
c7e8df80 00000046 c7e1c640 c03d9540 c03e8300 00000002 00000000 00000001
00000060 c7e4b5e0 00000000 00000000 c682ccf0 00000505 16bd0447
0000000b
c7e1c7f4 c7e8c000 c0398c00 c0398c00 c7e8dfa4 c02d49fb 00000004
c0398c00
Call Trace:
[<c02d49fb>] schedule+0x3b/0x130 (88)
[<c0133e60>] do_irqd+0x0/0xd0 (32)
[<c0133f08>] do_irqd+0xa8/0xd0 (4)
[<c012ca7a>] kthread+0xaa/0xb0 (28)
[<c012c9d0>] kthread+0x0/0xb0 (28)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
kjournald [c135a680]T C03D9540 0 929 1 1135
778 (L-TLB)
c787dcdc 00000046 c135a680 c03d9540 00000246 c1100570 00000246 c135a680
00000000 c135a834 c787c000 ffffffff c135a680 000002c4 66b1a4ec
0000019d
c135a834 c787c000 00000000 c787dd68 c787dd00 c02d49fb c036f180
00000001
Call Trace:
[<c02d49fb>] schedule+0x3b/0x130 (88)
[<c020d46f>] blk_backing_dev_unplug+0x1f/0x30 (12)
[<c020d431>] generic_unplug_device+0x21/0x40 (12)
[<c02d559e>] io_schedule+0xe/0x20 (12)
[<c0156701>] sync_buffer+0x31/0x50 (8)
[<c02d57e7>] __wait_on_bit+0x67/0x70 (12)
[<c01566d0>] sync_buffer+0x0/0x50 (8)
[<c01566d0>] sync_buffer+0x0/0x50 (16)
[<c02d5883>] out_of_line_wait_on_bit+0x93/0xa0 (4)
[<c012d110>] wake_bit_function+0x0/0x60 (28)
[<c012d110>] wake_bit_function+0x0/0x60 (32)
[<c01ad908>] journal_commit_transaction+0xf38/0x16c0 (32)
[<c01cbde2>] __delay+0x12/0x20 (84)
[<c888d031>] lpptest_irq+0x31/0x50 [lpptest] (8)
[<c0133624>] handle_IRQ_event+0x44/0x80 (8)
[<c0133775>] __do_IRQ+0x115/0x160 (32)
[<c0112f7f>] scheduler_tick+0x1f/0x470 (60)
[<c01127d8>] recalc_task_prio+0xa8/0x1a0 (12)
[<c02d4671>] __schedule+0x2d1/0x620 (52)
[<c02d4b45>] preempt_schedule+0x55/0x70 (8)
[<c02d4b45>] preempt_schedule+0x55/0x70 (72)
[<c012d08f>] finish_wait+0x4f/0x70 (8)
[<c01b0bf6>] kjournald+0xf6/0x280 (24)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c01b0bf6>] kjournald+0xf6/0x280 (12)
[<c01b0c0c>] kjournald+0x10c/0x280 (24)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (40)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (32)
[<c0105f02>] ret_from_fork+0x6/0x14 (68)
[<c01b0b00>] kjournald+0x0/0x280 (8)
[<c01b0ae0>] commit_timeout+0x0/0x10 (8)
[<c01b0b00>] kjournald+0x0/0x280 (20)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
IRQ 3 [c135b1e0]D C03D9540 0 994 10 1282
779 (L-TLB)
c1379f80 00000046 c135b1e0 c03d9540 c03e82d0 00000002 00000001 00000001
00000060 c7e4bd20 00000000 00000000 c7d04c10 00002227 b5d387fe
00000009
c135b394 c1378000 c0398ba0 c0398ba0 c1379fa4 c02d49fb 00000003
c0398ba0
Call Trace:
[<c02d49fb>] schedule+0x3b/0x130 (88)
[<c0133e60>] do_irqd+0x0/0xd0 (32)
[<c0133f08>] do_irqd+0xa8/0xd0 (4)
[<c012ca7a>] kthread+0xaa/0xb0 (28)
[<c012c9d0>] kthread+0x0/0xb0 (28)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
portmap [c7e1c090]D C03D9540 0 1135 1 1199
929 (NOTLB)
c7cd5eb8 00000086 c7e1c090 c03d9540 c7e1c090 00000282 c0328040 00000282
00000000 00000000 00000246 00000246 c7e1c090 000202dc efd174bd
0000000a
c7e1c244 c7cd4000 7fffffff c7cd5f68 c7cd5edc c02d49fb c77fb028
c12caa78
Call Trace:
[<c02d49fb>] schedule+0x3b/0x130 (88)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (24)
[<c02d56c4>] schedule_timeout+0xd4/0xe0 (12)
[<c0292b97>] tcp_poll+0x37/0x190 (12)
[<c015647b>] fget+0x4b/0x70 (20)
[<c02674a9>] sock_poll+0x29/0x40 (28)
[<c016a5ab>] do_pollfd+0x5b/0xa0 (20)
[<c016a698>] do_poll+0xa8/0xd0 (28)
[<c016a821>] sys_poll+0x161/0x240 (48)
[<c0169b60>] __pollwait+0x0/0xd0 (36)
[<c010602b>] syscall_call+0x7/0xb (32)
syslogd [c78600f0]T C03D9540 0 1199 1 1206
1135 (NOTLB)
c78afc7c 00000086 c78600f0 c03d9540 00000000 c7f7547c 00000246 c78600f0
c78600f0 00000001 c7f7547c c012cf9e 00000282 00008fb6 2ad9eccb
00000239
c78602a4 c78ae000 00000000 c78afd18 c78afca0 c02d49fb c7f75414
00000001
Call Trace:
[<c012cf9e>] prepare_to_wait+0x5e/0x80 (48)
[<c02d49fb>] schedule+0x3b/0x130 (40)
[<c01a9891>] start_this_handle+0x2a1/0x410 (12)
[<c01a9896>] start_this_handle+0x2a6/0x410 (24)
[<c013ee9c>] kmem_cache_alloc+0x4c/0xe0 (56)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (16)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (32)
[<c01a9b1b>] journal_start+0xcb/0xf0 (40)
[<c0107deb>] do_IRQ+0x2b/0x40 (20)
[<c019d3e9>] ext3_dirty_inode+0x39/0x90 (16)
[<c01786a6>] __mark_inode_dirty+0x1e6/0x1f0 (24)
[<c0136c83>] remove_suid+0x3/0x90 (16)
[<c0133775>] __do_IRQ+0x115/0x160 (20)
[<c0171f50>] inode_update_time+0xd0/0xe0 (24)
[<c01377f0>] generic_file_aio_write_nolock+0x230/0x4c0 (48)
[<c0137b23>] generic_file_write_nolock+0xa3/0xc0 (100)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (112)
[<c0105e5d>] do_signal+0xcd/0x130 (16)
[<c0137e14>] generic_file_writev+0x44/0xb0 (24)
[<c0155a0c>] do_readv_writev+0x22c/0x280 (44)
[<c0155360>] do_sync_write+0x0/0xf0 (40)
[<c01cbde2>] __delay+0x12/0x20 (52)
[<c888d031>] lpptest_irq+0x31/0x50 [lpptest] (8)
[<c0155b28>] vfs_writev+0x58/0x70 (24)
[<c0155c11>] sys_writev+0x51/0x80 (24)
[<c010602b>] syscall_call+0x7/0xb (40)
klogd [c7d04c10]D C03D9540 0 1206 1 1214
1199 (NOTLB)
c7d15d08 00000082 c7d04c10 c03d9540 00000060 00000246 c7d04c10 c032829c
c0327ff4 000000d0 00000000 00000001 c113d6f0 00089689 bda57400
000002a9
c7d04dc4 c7d14000 7fffffff 00000001 c7d15d2c c02d49fb c7d14000
c7d14000
Call Trace:
[<c02d49fb>] schedule+0x3b/0x130 (88)
[<c02d56c4>] schedule_timeout+0xd4/0xe0 (36)
[<c012d01d>] prepare_to_wait_exclusive+0x5d/0x80 (32)
[<c02c7a7e>] unix_wait_for_peer+0xae/0xe0 (28)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c02c7a7e>] unix_wait_for_peer+0xae/0xe0 (12)
[<c02c7aa7>] unix_wait_for_peer+0xd7/0xe0 (24)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (16)
[<c01cc432>] copy_from_user+0x42/0x70 (12)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (20)
[<c02c85a2>] unix_dgram_sendmsg+0x282/0x5f0 (40)
[<c0134fde>] find_get_page+0x3e/0x50 (48)
[<c0145f51>] do_no_page+0x1b1/0x3b0 (32)
[<c0266fa6>] sock_aio_write+0xf6/0x120 (20)
[<c0145f51>] do_no_page+0x1b1/0x3b0 (28)
[<c0155417>] do_sync_write+0xb7/0xf0 (84)
[<c01118e9>] do_page_fault+0x409/0x610 (24)
[<c01118e9>] do_page_fault+0x409/0x610 (4)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (88)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c015554f>] vfs_write+0xff/0x130 (36)
[<c0155651>] sys_write+0x51/0x80 (44)
[<c010602b>] syscall_call+0x7/0xb (40)
inetd [c135b790]D C03D9540 0 1214 1 1218
1206 (NOTLB)
c135de5c 00000082 c135b790 c03d9540 00000292 00000292 c0328040 00000292
c0139999 0000000d 00000246 c135b790 c135a0d0 000d1e8d 3dd989aa
0000000a
c135b944 c135c000 7fffffff 0000000e c135de80 c02d49fb c012ce1f
c7661ae4
Call Trace:
[<c0139999>] buffered_rmqueue+0x89/0xf0 (36)
[<c02d49fb>] schedule+0x3b/0x130 (52)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (4)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c02d56c4>] schedule_timeout+0xd4/0xe0 (20)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (16)
[<c0292b97>] tcp_poll+0x37/0x190 (24)
[<c015647b>] fget+0x4b/0x70 (20)
[<c02674a9>] sock_poll+0x29/0x40 (28)
[<c0169ec7>] do_select+0x197/0x2f0 (20)
[<c0169b60>] __pollwait+0x0/0xd0 (84)
[<c016a33b>] sys_select+0x2db/0x4f0 (32)
[<c010602b>] syscall_call+0x7/0xb (92)
lpd [c78606a0]D C03D9540 0 1218 1 1272
1214 (NOTLB)
c78cde5c 00000086 c78606a0 c03d9540 c10ed420 c0327ff4 c01c9988 c0328040
00000001 00000006 00000246 c78606a0 00000286 00085c44 51ea0297
0000000a
c7860854 c78cc000 7fffffff 00000007 c78cde80 c02d49fb c012ce1f
c7661364
Call Trace:
[<c01c9988>] up_write_mutex+0x38/0xc0 (28)
[<c02d49fb>] schedule+0x3b/0x130 (60)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (4)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c02d56c4>] schedule_timeout+0xd4/0xe0 (20)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (16)
[<c0292b97>] tcp_poll+0x37/0x190 (24)
[<c015647b>] fget+0x4b/0x70 (20)
[<c02674a9>] sock_poll+0x29/0x40 (28)
[<c0169ec7>] do_select+0x197/0x2f0 (20)
[<c0169b60>] __pollwait+0x0/0xd0 (84)
[<c016a33b>] sys_select+0x2db/0x4f0 (32)
[<c0148d72>] sys_munmap+0x42/0x50 (68)
[<c010602b>] syscall_call+0x7/0xb (24)
nfsd [c76d37f0]D C03D9540 0 1272 1 1273
1218 (L-TLB)
c76e1eb0 00000046 c76d37f0 c03d9540 00000246 c76d37f0 00000001 c76e1ef8
c012007c c0326060 c76e1ef8 c01c9988 c76d3240 000046bc a7bad305
0000000a
c76d39a4 c76e0000 c76e1eec c76fd278 c76e1ed4 c02d49fb 00000292
c79c1b14
Call Trace:
[<c012007c>] __mod_timer+0x11c/0x1e0 (36)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c02d49fb>] schedule+0x3b/0x130 (40)
[<c02d5678>] schedule_timeout+0x88/0xe0 (36)
[<c883f907>] svc_recv+0x3a7/0x540 [sunrpc] (60)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c0120d30>] process_timeout+0x0/0x10 (8)
[<c883f910>] svc_recv+0x3b0/0x540 [sunrpc] (28)
[<c02d4b45>] preempt_schedule+0x55/0x70 (16)
[<c0113460>] default_wake_function+0x0/0x30 (20)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c0113460>] default_wake_function+0x0/0x30 (20)
[<c0123e7d>] sigprocmask+0x6d/0xe0 (16)
[<c8865359>] nfsd+0x109/0x3c0 [nfsd] (24)
[<c8865250>] nfsd+0x0/0x3c0 [nfsd] (48)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
nfsd [c76d3240]D C03D9540 0 1273 1 1274
1272 (L-TLB)
c76e3eb0 00000046 c76d3240 c03d9540 00000246 c76d3240 00000001 c76e3ef8
c012007c c0326060 c76e3ef8 c01c9988 c76d2130 00002394 a7baf699
0000000a
c76d33f4 c76e2000 c76e3eec c76fd278 c76e3ed4 c02d49fb 00000292
00000000
Call Trace:
[<c012007c>] __mod_timer+0x11c/0x1e0 (36)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c02d49fb>] schedule+0x3b/0x130 (40)
[<c02d5678>] schedule_timeout+0x88/0xe0 (36)
[<c883f907>] svc_recv+0x3a7/0x540 [sunrpc] (60)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c0120d30>] process_timeout+0x0/0x10 (8)
[<c883f910>] svc_recv+0x3b0/0x540 [sunrpc] (28)
[<c02d4b45>] preempt_schedule+0x55/0x70 (16)
[<c0113460>] default_wake_function+0x0/0x30 (20)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c0113460>] default_wake_function+0x0/0x30 (20)
[<c0123e7d>] sigprocmask+0x6d/0xe0 (16)
[<c8865359>] nfsd+0x109/0x3c0 [nfsd] (24)
[<c8865250>] nfsd+0x0/0x3c0 [nfsd] (48)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
nfsd [c76d2130]D C03D9540 0 1274 1 1275
1273 (L-TLB)
c6c33eb0 00000046 c76d2130 c03d9540 00000246 c76d2130 00000001 c6c33ef8
c012007c c0326060 c6c33ef8 c01c9988 c76d26e0 000020ea a7bb1783
0000000a
c76d22e4 c6c32000 c6c33eec c76fd278 c6c33ed4 c02d49fb 00000292
00000000
Call Trace:
[<c012007c>] __mod_timer+0x11c/0x1e0 (36)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c02d49fb>] schedule+0x3b/0x130 (40)
[<c02d5678>] schedule_timeout+0x88/0xe0 (36)
[<c883f907>] svc_recv+0x3a7/0x540 [sunrpc] (60)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c0120d30>] process_timeout+0x0/0x10 (8)
[<c883f910>] svc_recv+0x3b0/0x540 [sunrpc] (28)
[<c02d4b45>] preempt_schedule+0x55/0x70 (16)
[<c0113460>] default_wake_function+0x0/0x30 (20)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c0113460>] default_wake_function+0x0/0x30 (20)
[<c0123e7d>] sigprocmask+0x6d/0xe0 (16)
[<c8865359>] nfsd+0x109/0x3c0 [nfsd] (24)
[<c8865250>] nfsd+0x0/0x3c0 [nfsd] (48)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
nfsd [c76d26e0]D C03D9540 0 1275 1 1276
1274 (L-TLB)
c73fbeb0 00000046 c76d26e0 c03d9540 00000246 c76d26e0 00000001 c73fbef8
c012007c c0326060 c73fbef8 c01c9988 c76d2c90 000020ea a7bb386d
0000000a
c76d2894 c73fa000 c73fbeec c76fd278 c73fbed4 c02d49fb 00000292
00000000
Call Trace:
[<c012007c>] __mod_timer+0x11c/0x1e0 (36)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c02d49fb>] schedule+0x3b/0x130 (40)
[<c02d5678>] schedule_timeout+0x88/0xe0 (36)
[<c883f907>] svc_recv+0x3a7/0x540 [sunrpc] (60)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c0120d30>] process_timeout+0x0/0x10 (8)
[<c883f910>] svc_recv+0x3b0/0x540 [sunrpc] (28)
[<c02d4b45>] preempt_schedule+0x55/0x70 (16)
[<c0113460>] default_wake_function+0x0/0x30 (20)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c0113460>] default_wake_function+0x0/0x30 (20)
[<c0123e7d>] sigprocmask+0x6d/0xe0 (16)
[<c8865359>] nfsd+0x109/0x3c0 [nfsd] (24)
[<c8865250>] nfsd+0x0/0x3c0 [nfsd] (48)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
nfsd [c76d2c90]D C03D9540 0 1276 1 1277
1275 (L-TLB)
c73c3eb0 00000046 c76d2c90 c03d9540 00000246 c76d2c90 00000001 c73c3ef8
c012007c c0326060 c73c3ef8 c01c9988 c6c4b810 0000221b a7bb5a88
0000000a
c76d2e44 c73c2000 c73c3eec c76fd278 c73c3ed4 c02d49fb 00000292
00000000
Call Trace:
[<c012007c>] __mod_timer+0x11c/0x1e0 (36)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c02d49fb>] schedule+0x3b/0x130 (40)
[<c02d5678>] schedule_timeout+0x88/0xe0 (36)
[<c883f907>] svc_recv+0x3a7/0x540 [sunrpc] (60)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c0120d30>] process_timeout+0x0/0x10 (8)
[<c883f910>] svc_recv+0x3b0/0x540 [sunrpc] (28)
[<c02d4b45>] preempt_schedule+0x55/0x70 (16)
[<c0113460>] default_wake_function+0x0/0x30 (20)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c0113460>] default_wake_function+0x0/0x30 (20)
[<c0123e7d>] sigprocmask+0x6d/0xe0 (16)
[<c8865359>] nfsd+0x109/0x3c0 [nfsd] (24)
[<c8865250>] nfsd+0x0/0x3c0 [nfsd] (48)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
nfsd [c6c4b810]D C03D9540 0 1277 1 1278
1276 (L-TLB)
c6c4deb0 00000046 c6c4b810 c03d9540 00000246 c6c4b810 00000001 c6c4def8
c012007c c0326060 c6c4def8 c01c9988 c6c4b260 0000189c a7bbaf55
0000000a
c6c4b9c4 c6c4c000 c6c4deec c76fd278 c6c4ded4 c02d49fb 00000292
00000000
Call Trace:
[<c012007c>] __mod_timer+0x11c/0x1e0 (36)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c02d49fb>] schedule+0x3b/0x130 (40)
[<c02d5678>] schedule_timeout+0x88/0xe0 (36)
[<c883f907>] svc_recv+0x3a7/0x540 [sunrpc] (60)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c0120d30>] process_timeout+0x0/0x10 (8)
[<c883f910>] svc_recv+0x3b0/0x540 [sunrpc] (28)
[<c0113460>] default_wake_function+0x0/0x30 (36)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c0113460>] default_wake_function+0x0/0x30 (20)
[<c0123e7d>] sigprocmask+0x6d/0xe0 (16)
[<c8865359>] nfsd+0x109/0x3c0 [nfsd] (24)
[<c8865250>] nfsd+0x0/0x3c0 [nfsd] (48)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
nfsd [c6c4b260]D C03D9540 0 1278 1 1279
1277 (L-TLB)
c7113eb0 00000046 c6c4b260 c03d9540 00000246 c6c4b260 00000001 c7113ef8
c012007c c0326060 c7113ef8 c01c9988 c7113ef8 000032c2 ab0b7866
0000000a
c6c4b414 c7112000 c7113eec c76fd278 c7113ed4 c02d49fb 00000292
c79c1b14
Call Trace:
[<c012007c>] __mod_timer+0x11c/0x1e0 (36)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c02d49fb>] schedule+0x3b/0x130 (40)
[<c02d5678>] schedule_timeout+0x88/0xe0 (36)
[<c883f907>] svc_recv+0x3a7/0x540 [sunrpc] (60)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c0120d30>] process_timeout+0x0/0x10 (8)
[<c883f910>] svc_recv+0x3b0/0x540 [sunrpc] (28)
[<c0113460>] default_wake_function+0x0/0x30 (36)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c0113460>] default_wake_function+0x0/0x30 (20)
[<c0123e7d>] sigprocmask+0x6d/0xe0 (16)
[<c8865359>] nfsd+0x109/0x3c0 [nfsd] (24)
[<c8865250>] nfsd+0x0/0x3c0 [nfsd] (48)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
nfsd [c6c4acb0]D C03D9540 0 1279 1 1281
1278 (L-TLB)
c6cbbeb0 00000046 c6c4acb0 c03d9540 00000246 c6c4acb0 00000001 c6cbbef8
c012007c c0326060 c6cbbef8 c01c9988 c6c4b260 00018d87 ab0b45a4
0000000a
c6c4ae64 c6cba000 c6cbbeec c76fd278 c6cbbed4 c02d49fb 00000292
c01c9988
Call Trace:
[<c012007c>] __mod_timer+0x11c/0x1e0 (36)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c02d49fb>] schedule+0x3b/0x130 (40)
[<c01c9988>] up_write_mutex+0x38/0xc0 (8)
[<c02d5678>] schedule_timeout+0x88/0xe0 (28)
[<c883f907>] svc_recv+0x3a7/0x540 [sunrpc] (60)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c0120d30>] process_timeout+0x0/0x10 (8)
[<c883f910>] svc_recv+0x3b0/0x540 [sunrpc] (28)
[<c0113460>] default_wake_function+0x0/0x30 (36)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c0113460>] default_wake_function+0x0/0x30 (20)
[<c0123e7d>] sigprocmask+0x6d/0xe0 (16)
[<c8865359>] nfsd+0x109/0x3c0 [nfsd] (24)
[<c8865250>] nfsd+0x0/0x3c0 [nfsd] (48)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
lockd [c6c4a700]D C03D9540 0 1281 1 1285
1279 (L-TLB)
c6c8becc 00000046 c6c4a700 c03d9540 c6c8be8c c6c8be8c c79c0108 00000246
c6c4a700 00000000 00000000 00000000 c76d37f0 00005dec a7b980d4
0000000a
c6c4a8b4 c6c8a000 7fffffff c76fd178 c6c8bef0 c02d49fb c01c9988
c79c0114
Call Trace:
[<c02d49fb>] schedule+0x3b/0x130 (88)
[<c01c9988>] up_write_mutex+0x38/0xc0 (4)
[<c02d56c4>] schedule_timeout+0xd4/0xe0 (32)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (36)
[<c883f907>] svc_recv+0x3a7/0x540 [sunrpc] (24)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c883f907>] svc_recv+0x3a7/0x540 [sunrpc] (12)
[<c883f910>] svc_recv+0x3b0/0x540 [sunrpc] (24)
[<c0113460>] default_wake_function+0x0/0x30 (36)
[<c01ca189>] up+0x69/0xc0 (20)
[<c0113460>] default_wake_function+0x0/0x30 (12)
[<c8a0e574>] lockd+0x134/0x2b0 [lockd] (40)
[<c8a0e440>] lockd+0x0/0x2b0 [lockd] (20)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
rpciod/0 [c6c4a150]D C03D9540 0 1282 10 1471
994 (L-TLB)
c6c65f20 00000046 c6c4a150 c03d9540 0000000a 00000000 00000246 c6c4a150
00000246 c6c4a150 00000001 c75295dc c6c4a700 00001be9 a7b922e8
0000000a
c6c4a304 c6c64000 c75295a0 c6c65f94 c6c65f44 c02d49fb c6c65f94
c75295dc
Call Trace:
[<c02d49fb>] schedule+0x3b/0x130 (88)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (12)
[<c0127fe8>] worker_thread+0x258/0x280 (24)
[<c0113460>] default_wake_function+0x0/0x30 (52)
[<c0113460>] default_wake_function+0x0/0x30 (32)
[<c0127d90>] worker_thread+0x0/0x280 (36)
[<c012ca7a>] kthread+0xaa/0xb0 (4)
[<c012c9d0>] kthread+0x0/0xb0 (28)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
rpc.mountd [c6f49830]D C03D9540 0 1285 1 1309
1281 (NOTLB)
c6f4be5c 00000082 c6f49830 c03d9540 c10de0e0 c0327ff4 c01c9988 c0328040
00000001 00000004 00000246 c6f49830 c135a0d0 00082460 af6c86af
0000000a
c6f499e4 c6f4a000 7fffffff 00000005 c6f4be80 c02d49fb c012ce1f
c6d47884
Call Trace:
[<c01c9988>] up_write_mutex+0x38/0xc0 (28)
[<c02d49fb>] schedule+0x3b/0x130 (60)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (4)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c02d56c4>] schedule_timeout+0xd4/0xe0 (20)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (16)
[<c0292b97>] tcp_poll+0x37/0x190 (24)
[<c015647b>] fget+0x4b/0x70 (20)
[<c02674a9>] sock_poll+0x29/0x40 (28)
[<c0169ec7>] do_select+0x197/0x2f0 (20)
[<c0169b60>] __pollwait+0x0/0xd0 (84)
[<c016a33b>] sys_select+0x2db/0x4f0 (32)
[<c010602b>] syscall_call+0x7/0xb (92)
sshd [c6f48cd0]D C03D9540 0 1309 1 1893 1315
1285 (NOTLB)
c6f71e5c 00000086 c6f48cd0 c03d9540 00000292 00000292 c0328040 00000292
c0139999 00000003 00000246 c6f48cd0 c6988800 0000944a fb0b23a8
0000005a
c6f48e84 c6f70000 7fffffff 00000004 c6f71e80 c02d49fb c012ce1f
c6d47604
Call Trace:
[<c0139999>] buffered_rmqueue+0x89/0xf0 (36)
[<c02d49fb>] schedule+0x3b/0x130 (52)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (4)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c02d56c4>] schedule_timeout+0xd4/0xe0 (20)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (16)
[<c0292b97>] tcp_poll+0x37/0x190 (24)
[<c015647b>] fget+0x4b/0x70 (20)
[<c02674a9>] sock_poll+0x29/0x40 (28)
[<c0169ec7>] do_select+0x197/0x2f0 (20)
[<c0169b60>] __pollwait+0x0/0xd0 (84)
[<c016a33b>] sys_select+0x2db/0x4f0 (32)
[<c02d4ba8>] preempt_schedule_irq+0x48/0x70 (72)
[<c010602b>] syscall_call+0x7/0xb (20)
rpc.statd [c6f49280]D C03D9540 0 1315 1 1318
1309 (NOTLB)
c70c7e5c 00000082 c6f49280 c03d9540 c10e0f80 c0327ff4 c01c9988 c0328040
00000001 00000006 00000246 c6f49280 00000286 0001de0e efe0198d
0000000a
c6f49434 c70c6000 7fffffff 00000007 c70c7e80 c02d49fb c012ce1f
c6fc6b24
Call Trace:
[<c01c9988>] up_write_mutex+0x38/0xc0 (28)
[<c02d49fb>] schedule+0x3b/0x130 (60)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (4)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c02d56c4>] schedule_timeout+0xd4/0xe0 (20)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (16)
[<c0292b97>] tcp_poll+0x37/0x190 (24)
[<c015647b>] fget+0x4b/0x70 (20)
[<c02674a9>] sock_poll+0x29/0x40 (28)
[<c0169ec7>] do_select+0x197/0x2f0 (20)
[<c0169b60>] __pollwait+0x0/0xd0 (84)
[<c016a33b>] sys_select+0x2db/0x4f0 (32)
[<c010602b>] syscall_call+0x7/0xb (92)
atd [c6f48720]D C03D9540 0 1318 1 1321
1315 (NOTLB)
c6f97ef8 00000086 c6f48720 c03d9540 00000246 c6f48720 00000001 c6f97f40
c012007c c0326060 c6f97f40 c01c9988 c7d040b0 000313e4 f3b1a1b5
0000000a
c6f488d4 c6f96000 c6f97f34 000f41a7 c6f97f1c c02d49fb 00000282
00000246
Call Trace:
[<c012007c>] __mod_timer+0x11c/0x1e0 (36)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c02d49fb>] schedule+0x3b/0x130 (40)
[<c02d5678>] schedule_timeout+0x88/0xe0 (36)
[<c0123e7d>] sigprocmask+0x6d/0xe0 (64)
[<c0120d30>] process_timeout+0x0/0x10 (16)
[<c0120e2e>] sys_nanosleep+0xde/0x170 (28)
[<c010602b>] syscall_call+0x7/0xb (52)
cron [c6f48170]T C03D9540 0 1321 1 1327
1318 (NOTLB)
c680fce0 00000082 c6f48170 c03d9540 00000246 c1100508 00000246 c6f48170
00000000 00000000 c7ffb680 00000000 c553ec50 00036951 3041657d
00000192
c6f48324 c680e000 00000000 c680fd6c c680fd04 c02d49fb c036f180
00000001
Call Trace:
[<c02d49fb>] schedule+0x3b/0x130 (88)
[<c020d46f>] blk_backing_dev_unplug+0x1f/0x30 (12)
[<c020d431>] generic_unplug_device+0x21/0x40 (12)
[<c02d559e>] io_schedule+0xe/0x20 (12)
[<c0156701>] sync_buffer+0x31/0x50 (8)
[<c02d57e7>] __wait_on_bit+0x67/0x70 (12)
[<c01566d0>] sync_buffer+0x0/0x50 (8)
[<c01566d0>] sync_buffer+0x0/0x50 (16)
[<c02d5883>] out_of_line_wait_on_bit+0x93/0xa0 (4)
[<c012d110>] wake_bit_function+0x0/0x60 (28)
[<c012d110>] wake_bit_function+0x0/0x60 (32)
[<c019e846>] ext3_find_entry+0x196/0x440 (32)
[<c016f267>] d_alloc+0x1b7/0x240 (40)
[<c016f6f7>] __d_lookup+0x97/0x130 (60)
[<c019ed71>] ext3_lookup+0x41/0xd0 (40)
[<c0163955>] real_lookup+0xd5/0x100 (28)
[<c0163cd6>] do_lookup+0x96/0xb0 (32)
[<c01644dc>] link_path_walk+0x7ec/0xfc0 (32)
[<c01c9988>] up_write_mutex+0x38/0xc0 (48)
[<c0164fd9>] path_lookup+0x99/0x1b0 (36)
[<c01652b3>] __user_walk+0x33/0x60 (28)
[<c015f78f>] vfs_stat+0x1f/0x60 (28)
[<c02d5682>] schedule_timeout+0x92/0xe0 (24)
[<c015fedb>] sys_stat64+0x1b/0x40 (64)
[<c0120d30>] process_timeout+0x0/0x10 (16)
[<c0120e2e>] sys_nanosleep+0xde/0x170 (28)
[<c010ae4a>] do_gettimeofday+0x1a/0xd0 (8)
[<c011bd2b>] sys_time+0x1b/0x60 (20)
[<c010602b>] syscall_call+0x7/0xb (24)
bash [c7d04660]D C03D9540 0 1327 1 12577 1328
1321 (NOTLB)
c1233ef8 00000086 c7d04660 c03d9540 c13b07ec c7d04704 00000246 c7d04660
00000246 c7d04660 00000001 00000086 c012ce1f 0000ebf9 3ccd43c6
0000019c
c7d04814 c1232000 c7d04660 fffffe00 c1233f1c c02d49fb c0324d60
00000001
Call Trace:
[<c012ce1f>] add_wait_queue+0x3f/0x50 (52)
[<c02d49fb>] schedule+0x3b/0x130 (36)
[<c011b42b>] do_wait+0x1bb/0x4f0 (12)
[<c011b459>] do_wait+0x1e9/0x4f0 (24)
[<c0113460>] default_wake_function+0x0/0x30 (44)
[<c0123e7d>] sigprocmask+0x6d/0xe0 (20)
[<c0113460>] default_wake_function+0x0/0x30 (12)
[<c01cc3de>] copy_to_user+0x3e/0x50 (12)
[<c011b82d>] sys_wait4+0x3d/0x50 (28)
[<c011b865>] sys_waitpid+0x25/0x30 (24)
[<c010602b>] syscall_call+0x7/0xb (20)
bash [c7d040b0]D C03D9540 0 1328 1 2302 1329
1327 (NOTLB)
c1243ef8 00000086 c7d040b0 c03d9540 c78f282c c7d04154 00000246 c7d040b0
00000246 c7d040b0 00000001 00000086 c4d9f300 000802fc e5d1231c
00000064
c7d04264 c1242000 c7d040b0 fffffe00 c1243f1c c02d49fb c0324d60
00000001
Call Trace:
[<c02d49fb>] schedule+0x3b/0x130 (88)
[<c011b42b>] do_wait+0x1bb/0x4f0 (12)
[<c011b459>] do_wait+0x1e9/0x4f0 (24)
[<c0113460>] default_wake_function+0x0/0x30 (44)
[<c0123e7d>] sigprocmask+0x6d/0xe0 (20)
[<c0113460>] default_wake_function+0x0/0x30 (12)
[<c01cc3de>] copy_to_user+0x3e/0x50 (12)
[<c011b82d>] sys_wait4+0x3d/0x50 (28)
[<c011b865>] sys_waitpid+0x25/0x30 (24)
[<c010602b>] syscall_call+0x7/0xb (20)
bash [c135a0d0]T C03D9540 0 1329 1 1330
1328 (NOTLB)
c7847ce0 00000086 c135a0d0 c03d9540 00000246 c1100f98 00000246 c135a0d0
00000000 00000000 c7ffb680 00000000 00000246 00016576 95b0edcf
0000029e
c135a284 c7846000 00000000 c7847d6c c7847d04 c02d49fb c036f180
00000001
Call Trace:
[<c02d49fb>] schedule+0x3b/0x130 (88)
[<c020d46f>] blk_backing_dev_unplug+0x1f/0x30 (12)
[<c020d431>] generic_unplug_device+0x21/0x40 (12)
[<c02d559e>] io_schedule+0xe/0x20 (12)
[<c0156701>] sync_buffer+0x31/0x50 (8)
[<c02d57e7>] __wait_on_bit+0x67/0x70 (12)
[<c01566d0>] sync_buffer+0x0/0x50 (8)
[<c01566d0>] sync_buffer+0x0/0x50 (16)
[<c02d5883>] out_of_line_wait_on_bit+0x93/0xa0 (4)
[<c012d110>] wake_bit_function+0x0/0x60 (28)
[<c012d110>] wake_bit_function+0x0/0x60 (32)
[<c019e846>] ext3_find_entry+0x196/0x440 (32)
[<c016f267>] d_alloc+0x1b7/0x240 (40)
[<c016f6f7>] __d_lookup+0x97/0x130 (60)
[<c019ed71>] ext3_lookup+0x41/0xd0 (40)
[<c0163955>] real_lookup+0xd5/0x100 (28)
[<c0163cd6>] do_lookup+0x96/0xb0 (32)
[<c0163e15>] link_path_walk+0x125/0xfc0 (32)
[<c0164fd9>] path_lookup+0x99/0x1b0 (84)
[<c01652b3>] __user_walk+0x33/0x60 (28)
[<c015f78f>] vfs_stat+0x1f/0x60 (28)
[<c01c9988>] up_write_mutex+0x38/0xc0 (24)
[<c012d5d7>] _mutex_lock_irq+0x17/0x20 (24)
[<c01cc3de>] copy_to_user+0x3e/0x50 (20)
[<c015fedb>] sys_stat64+0x1b/0x40 (20)
[<c0123f76>] sys_rt_sigprocmask+0x86/0x120 (48)
[<c010ae4a>] do_gettimeofday+0x1a/0xd0 (4)
[<c01114e0>] do_page_fault+0x0/0x610 (28)
[<c01061d5>] error_code+0x2d/0x38 (8)
[<c010602b>] syscall_call+0x7/0xb (8)
getty [c682d850]T C03D9540 0 1330 1 1331
1329 (NOTLB)
c682fd10 00000082 c682d850 c03d9540 00000246 c1101410 00000246 c682d850
00000000 00000000 c7ffb680 00000000 00000246 0003b5b2 e0ecd1d1
000002a0
c682da04 c682e000 00000000 c682fd9c c682fd34 c02d49fb c036f180
00000001
Call Trace:
[<c02d49fb>] schedule+0x3b/0x130 (88)
[<c020d46f>] blk_backing_dev_unplug+0x1f/0x30 (12)
[<c020d431>] generic_unplug_device+0x21/0x40 (12)
[<c02d559e>] io_schedule+0xe/0x20 (12)
[<c0156701>] sync_buffer+0x31/0x50 (8)
[<c02d57e7>] __wait_on_bit+0x67/0x70 (12)
[<c01566d0>] sync_buffer+0x0/0x50 (8)
[<c01566d0>] sync_buffer+0x0/0x50 (16)
[<c02d5883>] out_of_line_wait_on_bit+0x93/0xa0 (4)
[<c012d110>] wake_bit_function+0x0/0x60 (28)
[<c012d110>] wake_bit_function+0x0/0x60 (32)
[<c019e846>] ext3_find_entry+0x196/0x440 (32)
[<c016f267>] d_alloc+0x1b7/0x240 (40)
[<c016f6f7>] __d_lookup+0x97/0x130 (60)
[<c019ed71>] ext3_lookup+0x41/0xd0 (40)
[<c0163955>] real_lookup+0xd5/0x100 (28)
[<c0163cd6>] do_lookup+0x96/0xb0 (32)
[<c01644dc>] link_path_walk+0x7ec/0xfc0 (32)
[<c0164fd9>] path_lookup+0x99/0x1b0 (84)
[<c0160990>] open_exec+0x30/0x100 (28)
[<c01e7c76>] tty_write+0xb6/0x260 (4)
[<c013ee9c>] kmem_cache_alloc+0x4c/0xe0 (24)
[<c013ee9c>] kmem_cache_alloc+0x4c/0xe0 (24)
[<c013ee9c>] kmem_cache_alloc+0x4c/0xe0 (24)
[<c0161b13>] do_execve+0x13/0x260 (20)
[<c0104b36>] sys_execve+0x46/0xb0 (36)
[<c010602b>] syscall_call+0x7/0xb (32)
getty [c682d2a0]D C03D9540 0 1331 1 1332
1330 (NOTLB)
c1281e10 00000082 c682d2a0 c03d9540 00000000 00000000 00000286 00000046
c682d2a0 c682d2a0 00000001 c0328040 c01391e3 0007b645 53bd158d
0000000b
c682d454 c1280000 7fffffff c1281f04 c1281e34 c02d49fb c01ca189
00000286
Call Trace:
[<c01391e3>] free_pages_bulk+0x1e3/0x300 (52)
[<c02d49fb>] schedule+0x3b/0x130 (36)
[<c01ca189>] up+0x69/0xc0 (4)
[<c02d56c4>] schedule_timeout+0xd4/0xe0 (32)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (36)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (12)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (24)
[<c01ed789>] read_chan+0x699/0x7a0 (24)
[<c01c9988>] up_write_mutex+0x38/0xc0 (20)
[<c01e6c25>] tty_ldisc_try+0x45/0x60 (12)
[<c0113460>] default_wake_function+0x0/0x30 (52)
[<c01c9d6e>] down+0x2e/0x140 (12)
[<c0113460>] default_wake_function+0x0/0x30 (20)
[<c01e7b96>] tty_read+0xf6/0x120 (40)
[<c0155522>] vfs_write+0xd2/0x130 (20)
[<c01552e8>] vfs_read+0xb8/0x130 (24)
[<c01555d1>] sys_read+0x51/0x80 (44)
[<c010602b>] syscall_call+0x7/0xb (40)
getty [c682ccf0]D C03D9540 0 1332 1 12175
1331 (NOTLB)
c1273e10 00000082 c682ccf0 c03d9540 00000000 00000000 00000286 00000046
c682ccf0 c682ccf0 00000001 c0328040 c7d04660 0007cdc6 539abfcc
0000000b
c682cea4 c1272000 7fffffff c1273f04 c1273e34 c02d49fb c01ca189
00000286
Call Trace:
[<c02d49fb>] schedule+0x3b/0x130 (88)
[<c01ca189>] up+0x69/0xc0 (4)
[<c02d56c4>] schedule_timeout+0xd4/0xe0 (32)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (36)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (12)
[<c012ce1f>] add_wait_queue+0x3f/0x50 (24)
[<c01ed789>] read_chan+0x699/0x7a0 (24)
[<c01c9988>] up_write_mutex+0x38/0xc0 (20)
[<c01e6c25>] tty_ldisc_try+0x45/0x60 (12)
[<c0113460>] default_wake_function+0x0/0x30 (52)
[<c01c9d6e>] down+0x2e/0x140 (12)
[<c0113460>] default_wake_function+0x0/0x30 (20)
[<c01e7b96>] tty_read+0xf6/0x120 (40)
[<c0155522>] vfs_write+0xd2/0x130 (20)
[<c01552e8>] vfs_read+0xb8/0x130 (24)
[<c01555d1>] sys_read+0x51/0x80 (44)
[<c010602b>] syscall_call+0x7/0xb (40)
IRQ 5 [c7860c50]R C03D9570 0 1471 10 1486
1282 (L-TLB)
c7a73f80 00000046 c113d140 c03d9570 000002ce c7a73f64 c0112932 c113d140
c03d9570 000002ce 8a5ae53c 000002ce c113d140 00000420 8a5ae838
000002ce
c7860e04 c7a72000 c0398c60 c0398c60 c7a73fa4 c02d49fb c0398c60
c7a73fa4
Call Trace:
[<c0112932>] activate_task+0x62/0x80 (28)
[<c02d49fb>] schedule+0x3b/0x130 (60)
[<c0112a83>] wake_up_process+0x23/0x30 (12)
[<c0133e60>] do_irqd+0x0/0xd0 (20)
[<c0133f08>] do_irqd+0xa8/0xd0 (4)
[<c012ca7a>] kthread+0xaa/0xb0 (28)
[<c012c9d0>] kthread+0x0/0xb0 (28)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
IRQ 7 [c7d051c0]D C03D9540 0 1486 10
1471 (L-TLB)
c6865f9c 00000046 c7d051c0 c03d9540 00000246 c7d051c0 00000001 c6aefea8
c0113652 00000296 c6865fc0 c01c9988 c7fd1710 000011a2 36d58ac6
00000023
c7d05374 c6864000 c6aefe9c c0398d20 c6865fc0 c02d49fb 00000296
00000001
Call Trace:
[<c0113652>] complete+0x52/0x60 (36)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c02d49fb>] schedule+0x3b/0x130 (40)
[<c0133e60>] do_irqd+0x0/0xd0 (32)
[<c012ca40>] kthread+0x70/0xb0 (4)
[<c012c9d0>] kthread+0x0/0xb0 (28)
[<c01040ed>] kernel_thread_helper+0x5/0x18 (16)
sshd [c7d05770]D C03D9540 0 1893 1309 1896
(NOTLB)
c6aefcf0 00000082 c7d05770 c03d9540 00000001 c013f0dd c7f50ec0 c7f00ce4
c012d597 c7f50ec0 c7f00ce4 c013f0dd c62d67a0 00046315 fff1f1f7
0000005a
c7d05924 c6aee000 7fffffff c12cc9a0 c6aefd14 c02d49fb c0159141
00000000
Call Trace:
[<c013f0dd>] kmem_cache_free+0x4d/0xe0 (24)
[<c012d597>] _mutex_lock+0x17/0x20 (12)
[<c013f0dd>] kmem_cache_free+0x4d/0xe0 (12)
[<c02d49fb>] schedule+0x3b/0x130 (40)
[<c0159141>] __block_commit_write+0x91/0xc0 (4)
[<c019a5b3>] ext3_journal_dirty_data+0x23/0x70 (8)
[<c02d56c4>] schedule_timeout+0xd4/0xe0 (24)
[<c012cf9e>] prepare_to_wait+0x5e/0x80 (32)
[<c02c9112>] unix_stream_data_wait+0xd2/0x120 (28)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c02c9112>] unix_stream_data_wait+0xd2/0x120 (12)
[<c02c9119>] unix_stream_data_wait+0xd9/0x120 (24)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (20)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (32)
[<c02c95a2>] unix_stream_recvmsg+0x442/0x4a0 (16)
[<c02c95b6>] unix_stream_recvmsg+0x456/0x4a0 (24)
[<c0266e96>] sock_aio_read+0xf6/0x110 (116)
[<c01551f7>] do_sync_read+0xb7/0xf0 (116)
[<c012d6fd>] atomic_dec_and_mutex_lock+0x4d/0x70 (68)
[<c012d6fd>] atomic_dec_and_mutex_lock+0x4d/0x70 (24)
[<c012d56a>] __mutex_lock+0x2a/0x40 (12)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (12)
[<c016e423>] dput+0x33/0x2b0 (20)
[<c015532f>] vfs_read+0xff/0x130 (28)
[<c01555d1>] sys_read+0x51/0x80 (44)
[<c010602b>] syscall_call+0x7/0xb (40)
sshd [c135ac30]D C03D9540 0 1896 1893 1897
(NOTLB)
c5337e5c 00000086 c135ac30 c03d9540 00000000 00000292 c5337eb0 00000246
c135ac30 c56fdcc0 00000246 c135ac30 00000000 00012732 7bd4c399
000002a4
c135ade4 c5336000 7fffffff 0000000a c5337e80 c02d49fb c01c9988
c0363020
Call Trace:
[<c02d49fb>] schedule+0x3b/0x130 (88)
[<c01c9988>] up_write_mutex+0x38/0xc0 (4)
[<c011354c>] __wake_up+0x4c/0x60 (12)
[<c02d56c4>] schedule_timeout+0xd4/0xe0 (20)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c01e6d73>] tty_ldisc_deref+0x43/0xb0 (12)
[<c01e6d73>] tty_ldisc_deref+0x43/0xb0 (24)
[<c01e940b>] tty_poll+0x8b/0xc0 (24)
[<c0169ec7>] do_select+0x197/0x2f0 (36)
[<c0169b60>] __pollwait+0x0/0xd0 (84)
[<c016a33b>] sys_select+0x2db/0x4f0 (32)
[<c010602b>] syscall_call+0x7/0xb (92)
bash [c4d8ed30]D C03D9540 0 1897 1896 2299
(NOTLB)
c4db5ef8 00000086 c4d8ed30 c03d9540 c64fbc2c c4d8edd4 00000246 c4d8ed30
00000246 c4d8ed30 00000001 00000086 c68bc8a0 00075636 3fecedf3
00000061
c4d8eee4 c4db4000 c4d8ed30 fffffe00 c4db5f1c c02d49fb c0324d60
00000001
Call Trace:
[<c02d49fb>] schedule+0x3b/0x130 (88)
[<c011b42b>] do_wait+0x1bb/0x4f0 (12)
[<c011b459>] do_wait+0x1e9/0x4f0 (24)
[<c0113460>] default_wake_function+0x0/0x30 (44)
[<c0123e7d>] sigprocmask+0x6d/0xe0 (20)
[<c0113460>] default_wake_function+0x0/0x30 (12)
[<c01cc3de>] copy_to_user+0x3e/0x50 (12)
[<c011b82d>] sys_wait4+0x3d/0x50 (28)
[<c011b865>] sys_waitpid+0x25/0x30 (24)
[<c010602b>] syscall_call+0x7/0xb (20)
bash [c5040130]D C03D9540 0 2299 1897 12578
(NOTLB)
c454def8 00000082 c5040130 c03d9540 c5735c4c c50401d4 00000246 c5040130
00000246 c5040130 00000001 00000086 c012ce1f 000066d2 abd7f311
0000019e
c50402e4 c454c000 c5040130 fffffe00 c454df1c c02d49fb c0324d60
00000001
Call Trace:
[<c012ce1f>] add_wait_queue+0x3f/0x50 (52)
[<c02d49fb>] schedule+0x3b/0x130 (36)
[<c011b42b>] do_wait+0x1bb/0x4f0 (12)
[<c011b459>] do_wait+0x1e9/0x4f0 (24)
[<c01127d8>] recalc_task_prio+0xa8/0x1a0 (20)
[<c0113460>] default_wake_function+0x0/0x30 (24)
[<c0123e7d>] sigprocmask+0x6d/0xe0 (20)
[<c0113460>] default_wake_function+0x0/0x30 (12)
[<c01cc3de>] copy_to_user+0x3e/0x50 (12)
[<c011b82d>] sys_wait4+0x3d/0x50 (28)
[<c011b865>] sys_waitpid+0x25/0x30 (24)
[<c010602b>] syscall_call+0x7/0xb (20)
top [c53fe150]T C03D9540 0 2302 1328
(NOTLB)
c5edbba8 00000082 c53fe150 c03d9540 00000246 00000000 00000246 c53fe150
00000000 00000000 c01127d8 42e4e86d c53fecb0 00006272 d78f69cb
00000190
c53fe304 c5eda000 c5edbc34 c1100508 c5edbbcc c02d49fb c036f180
00000001
Call Trace:
[<c01127d8>] recalc_task_prio+0xa8/0x1a0 (44)
[<c02d49fb>] schedule+0x3b/0x130 (44)
[<c020d46f>] blk_backing_dev_unplug+0x1f/0x30 (12)
[<c020d431>] generic_unplug_device+0x21/0x40 (12)
[<c02d559e>] io_schedule+0xe/0x20 (12)
[<c0156701>] sync_buffer+0x31/0x50 (8)
[<c02d58ee>] __wait_on_bit_lock+0x5e/0x70 (12)
[<c01566d0>] sync_buffer+0x0/0x50 (8)
[<c01566d0>] sync_buffer+0x0/0x50 (16)
[<c02d5993>] out_of_line_wait_on_bit_lock+0x93/0xa0 (4)
[<c012d110>] wake_bit_function+0x0/0x60 (28)
[<c012d110>] wake_bit_function+0x0/0x60 (32)
[<c0156758>] __lock_buffer+0x38/0x40 (32)
[<c01aa804>] do_get_write_access+0x834/0x850 (12)
[<c01583cb>] __getblk+0x2b/0x60 (72)
[<c019c4ba>] ext3_get_inode_loc+0x5a/0x260 (28)
[<c01aa857>] journal_get_write_access+0x37/0x50 (24)
[<c019d2d5>] ext3_reserve_inode_write+0x55/0xd0 (28)
[<c019d37b>] ext3_mark_inode_dirty+0x2b/0x60 (40)
[<c013d162>] do_page_cache_readahead+0xc2/0x1c0 (28)
[<c019d43d>] ext3_dirty_inode+0x8d/0x90 (16)
[<c01786a6>] __mark_inode_dirty+0x1e6/0x1f0 (24)
[<c013d162>] do_page_cache_readahead+0xc2/0x1c0 (8)
[<c0171e79>] update_atime+0xd9/0xe0 (52)
[<c01356ee>] do_generic_mapping_read+0x2ae/0x5a0 (48)
[<c0135cdb>] __generic_file_aio_read+0x20b/0x240 (116)
[<c01359e0>] file_read_actor+0x0/0xf0 (24)
[<c013f0dd>] kmem_cache_free+0x4d/0xe0 (8)
[<c0135d6b>] generic_file_aio_read+0x5b/0xa0 (52)
[<c01551f7>] do_sync_read+0xb7/0xf0 (40)
[<c016d37b>] fcntl_setlk+0x11b/0x2f0 (64)
[<c012d597>] _mutex_lock+0x17/0x20 (44)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (8)
[<c015647b>] fget+0x4b/0x70 (24)
[<c01552e8>] vfs_read+0xb8/0x130 (24)
[<c0168a77>] do_fcntl+0xf7/0x1b0 (24)
[<c01555d1>] sys_read+0x51/0x80 (20)
[<c010602b>] syscall_call+0x7/0xb (40)
grep [c6604330]T C03D9540 0 12175 1
1332 (NOTLB)
c3289d2c 00000082 c6604330 c03d9540 00000190 c113d140 00032f46 9c11a953
00000190 c66044e4 c3288000 ffffffff c62dc880 00000d0c 9c11f78e
00000190
c66044e4 c3288000 c3289db0 c1101bc8 c3289d50 c02d49fb c66048c8
c3289db0
Call Trace:
[<c02d49fb>] schedule+0x3b/0x130 (88)
[<c020007b>] uart_set_info+0x29b/0x4d0 (24)
[<c02d559e>] io_schedule+0xe/0x20 (12)
[<c01347a5>] sync_page+0x35/0x50 (8)
[<c02d58ee>] __wait_on_bit_lock+0x5e/0x70 (8)
[<c0134770>] sync_page+0x0/0x50 (8)
[<c012d110>] wake_bit_function+0x0/0x60 (4)
[<c0134f31>] __lock_page+0x91/0xa0 (16)
[<c012d110>] wake_bit_function+0x0/0x60 (24)
[<c012d110>] wake_bit_function+0x0/0x60 (32)
[<c0134fde>] find_get_page+0x3e/0x50 (12)
[<c013591d>] do_generic_mapping_read+0x4dd/0x5a0 (20)
[<c0135cdb>] __generic_file_aio_read+0x20b/0x240 (116)
[<c01359e0>] file_read_actor+0x0/0xf0 (24)
[<c0135d6b>] generic_file_aio_read+0x5b/0xa0 (60)
[<c01551f7>] do_sync_read+0xb7/0xf0 (40)
[<c015fea8>] cp_new_stat64+0xf8/0x110 (24)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (92)
[<c01552e8>] vfs_read+0xb8/0x130 (48)
[<c01555d1>] sys_read+0x51/0x80 (44)
[<c010602b>] syscall_call+0x7/0xb (40)
bash [c62dc880]T C03D9540 0 12577 1327
(NOTLB)
c5573be0 00000082 c62dc880 c03d9540 00000000 c01c9988 c1100508 00000246
c62dc880 00000001 c1101854 c012cf9e c7d04660 000c5ac8 3cc49ca5
0000019c
c62dca34 c5572000 c79cc6bc c5573c5c c5573c04 c02d49fb c1101854
c012cf9e
Call Trace:
[<c01c9988>] up_write_mutex+0x38/0xc0 (24)
[<c012cf9e>] prepare_to_wait+0x5e/0x80 (24)
[<c02d49fb>] schedule+0x3b/0x130 (40)
[<c012cf9e>] prepare_to_wait+0x5e/0x80 (8)
[<c01aa61c>] do_get_write_access+0x64c/0x850 (28)
[<c012d110>] wake_bit_function+0x0/0x60 (60)
[<c01583cb>] __getblk+0x2b/0x60 (12)
[<c012d110>] wake_bit_function+0x0/0x60 (20)
[<c01aa857>] journal_get_write_access+0x37/0x50 (32)
[<c019d2d5>] ext3_reserve_inode_write+0x55/0xd0 (28)
[<c019d37b>] ext3_mark_inode_dirty+0x2b/0x60 (40)
[<c013d162>] do_page_cache_readahead+0xc2/0x1c0 (28)
[<c019d43d>] ext3_dirty_inode+0x8d/0x90 (16)
[<c01786a6>] __mark_inode_dirty+0x1e6/0x1f0 (24)
[<c013d1ca>] do_page_cache_readahead+0x12a/0x1c0 (8)
[<c0171e79>] update_atime+0xd9/0xe0 (52)
[<c01356ee>] do_generic_mapping_read+0x2ae/0x5a0 (48)
[<c0135cdb>] __generic_file_aio_read+0x20b/0x240 (116)
[<c01359e0>] file_read_actor+0x0/0xf0 (24)
[<c0135d6b>] generic_file_aio_read+0x5b/0xa0 (60)
[<c01551f7>] do_sync_read+0xb7/0xf0 (40)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (116)
[<c01552e8>] vfs_read+0xb8/0x130 (48)
[<c0160aae>] kernel_read+0x4e/0x60 (44)
[<c016172a>] prepare_binprm+0xca/0xf0 (36)
[<c0161c15>] do_execve+0x115/0x260 (32)
[<c0104b36>] sys_execve+0x46/0xb0 (36)
[<c010602b>] syscall_call+0x7/0xb (32)
bash [c47fb8f0]T C03D9540 0 12578 2299
(NOTLB)
c4ae5be0 00000086 c47fb8f0 c03d9540 00000000 c01c9988 c1100508 00000246
c47fb8f0 00000001 c1101854 c012cf9e c5040130 00075001 abcf7095
0000019e
c47fbaa4 c4ae4000 c79cc6bc c4ae5c5c c4ae5c04 c02d49fb c1101854
c012cf9e
Call Trace:
[<c01c9988>] up_write_mutex+0x38/0xc0 (24)
[<c012cf9e>] prepare_to_wait+0x5e/0x80 (24)
[<c02d49fb>] schedule+0x3b/0x130 (40)
[<c012cf9e>] prepare_to_wait+0x5e/0x80 (8)
[<c01aa61c>] do_get_write_access+0x64c/0x850 (28)
[<c01a975a>] start_this_handle+0x16a/0x410 (8)
[<c01c9988>] up_write_mutex+0x38/0xc0 (12)
[<c012d110>] wake_bit_function+0x0/0x60 (40)
[<c01583cb>] __getblk+0x2b/0x60 (12)
[<c012d110>] wake_bit_function+0x0/0x60 (20)
[<c01aa857>] journal_get_write_access+0x37/0x50 (32)
[<c019d2d5>] ext3_reserve_inode_write+0x55/0xd0 (28)
[<c019d37b>] ext3_mark_inode_dirty+0x2b/0x60 (40)
[<c013d162>] do_page_cache_readahead+0xc2/0x1c0 (28)
[<c019d43d>] ext3_dirty_inode+0x8d/0x90 (16)
[<c01786a6>] __mark_inode_dirty+0x1e6/0x1f0 (24)
[<c013d162>] do_page_cache_readahead+0xc2/0x1c0 (8)
[<c0171e79>] update_atime+0xd9/0xe0 (52)
[<c01356ee>] do_generic_mapping_read+0x2ae/0x5a0 (48)
[<c0135cdb>] __generic_file_aio_read+0x20b/0x240 (116)
[<c01359e0>] file_read_actor+0x0/0xf0 (24)
[<c0135d6b>] generic_file_aio_read+0x5b/0xa0 (60)
[<c01551f7>] do_sync_read+0xb7/0xf0 (40)
[<c012d0b0>] autoremove_wake_function+0x0/0x60 (116)
[<c01552e8>] vfs_read+0xb8/0x130 (48)
[<c0160aae>] kernel_read+0x4e/0x60 (44)
[<c016172a>] prepare_binprm+0xca/0xf0 (36)
[<c0161c15>] do_execve+0x115/0x260 (32)
[<c0104b36>] sys_execve+0x46/0xb0 (36)
[<c010602b>] syscall_call+0x7/0xb (32)
* Thomas Gleixner <[email protected]> wrote:
> I'm doing some tests from my RT environment on 0.6.2. I'm quite sure,
> that interrupts are sporadically disabled for > 200?s. Did you change
> anything relevant to this between 0.6.2 and 0.6.4 ?
hm, i changed the task-exit schedule() to be called with irqs-off and
__schedule() - but that should be fine. I've attached the delta diff -
there's nothing suspicious at first sight. Maybe -V0.6.4 just made some
already existing bug more likely to trigger? I am too seeing rtc_wakeup
weirdnesses that were not present in earlier -V0.6 kernels.
Ingo
--- linux.old/Makefile
+++ linux.new/Makefile
@@ -1,7 +1,7 @@
VERSION = 2
PATCHLEVEL = 6
SUBLEVEL = 9
-EXTRAVERSION = -mm1-RT-V0.6.2
+EXTRAVERSION = -mm1-RT-V0.6.4
NAME=Zonked Quokka
# *DOCUMENTATION*
--- linux.old/kernel/exit.c
+++ linux.new/kernel/exit.c
@@ -840,9 +840,9 @@ asmlinkage NORET_TYPE void do_exit(long
#endif
check_no_held_locks(tsk);
/* PF_DEAD causes final put_task_struct after we schedule. */
- wmb();
+ local_irq_disable();
tsk->flags |= PF_DEAD;
- schedule();
+ __schedule();
BUG();
/* Avoid "noreturn function does return". */
for (;;) ;
--- linux.old/kernel/sched.c
+++ linux.new/kernel/sched.c
@@ -599,13 +599,11 @@ static inline void enqueue_task_head(str
*
* Both properties are important to certain workloads.
*/
-static int effective_prio(task_t *p)
+
+static inline int __effective_prio(task_t *p)
{
int bonus, prio;
- if (rt_task(p))
- return p->prio;
-
bonus = CURRENT_BONUS(p) - MAX_BONUS / 2;
prio = p->static_prio - bonus;
@@ -616,6 +614,13 @@ static int effective_prio(task_t *p)
return prio;
}
+static int effective_prio(task_t *p)
+{
+ if (rt_task(p))
+ return p->prio;
+ return __effective_prio(p);
+}
+
/*
* __activate_task - move a task to the runqueue.
*/
@@ -3540,8 +3545,7 @@ int mutex_getprio(task_t *p)
if (p->policy != SCHED_NORMAL)
return MAX_USER_RT_PRIO-1 - p->rt_priority;
else
-// return effective_prio(p);
- return p->static_prio;
+ return __effective_prio(p);
}
/*
@@ -3559,15 +3563,14 @@ void mutex_setprio(task_t *p, int prio)
rq = task_rq_lock(p, &flags);
+ oldprio = p->prio;
array = p->array;
if (array)
- deactivate_task(p, rq);
- oldprio = p->prio;
-
+ dequeue_task(p, array);
p->prio = prio;
if (array) {
- __activate_task(p, rq);
+ enqueue_task(p, array);
/*
* Reschedule if we are currently running on this runqueue and
* our priority decreased, or if we are not currently running on
--- linux.old/kernel/timer.c
+++ linux.new/kernel/timer.c
@@ -956,23 +956,25 @@ EXPORT_SYMBOL(xtime_lock);
*/
static inline void update_times(void)
{
+ unsigned long ticks = 0;
/*
* First test outside the lock for performance reasons:
*/
- if (jiffies - wall_jiffies) {
+ if (jiffies != wall_jiffies) {
unsigned long flags;
write_seqlock_irqsave(&xtime_lock, flags);
- while (jiffies - wall_jiffies) {
+ while (jiffies != wall_jiffies) {
wall_jiffies++;
+ ticks++;
update_wall_time(1);
- calc_load(1);
if (seqlock_need_resched(&xtime_lock)) {
write_sequnlock_irqrestore(&xtime_lock, flags);
cond_resched();
write_seqlock_irqsave(&xtime_lock, flags);
}
}
+ calc_load(ticks);
write_sequnlock_irqrestore(&xtime_lock, flags);
}
}
* Ingo Molnar <[email protected]> wrote:
> [...] I am too seeing rtc_wakeup weirdnesses that were not present in
> earlier -V0.6 kernels.
this turned out to be a user error - used the wrong script to renice
IRQ8. Perhaps rtc_wakeup could check for the presence of IRQ 8 and chrt
it to 99 if it's present? :-|
Ingo
On Mon, 1 Nov 2004 14:15:11 +0100
Ingo Molnar <[email protected]> wrote:
>
> * Ingo Molnar <[email protected]> wrote:
>
> > [...] I am too seeing rtc_wakeup weirdnesses that were not present in
> > earlier -V0.6 kernels.
>
> this turned out to be a user error - used the wrong script to renice
> IRQ8. Perhaps rtc_wakeup could check for the presence of IRQ 8 and chrt
> it to 99 if it's present? :-|
Hi,
well, how would i best check for the presence of the process/thread "IRQ 8"?
A quick glance through the output of apropos pid didn't show anything. i
suppose in the worst case i'd have to iterate over the list of all processes
and find the one that matches the name. Dunno how to do that either yet. If
it's fairly straightforward, i'll add it.
Although: is rtc always garanteed to be "IRQ 8" or is this only the case on
ia32 with XT-PIC?
?
flo
* Florian Schmidt <[email protected]> wrote:
> new max. jitter: 4.3% (41 usec)
> new max. jitter: 4.9% (47 usec)
a couple of conceptual questions: why does rtc_wakeup poll() on
/dev/rtc? Shouldnt a read() be enough?
i'm seeing some weird traces, which show rtc_wakeup doing this cycle:
[~900 usecs pass]
hardirq 8 comes in, wakes IRQ 8 thread
IRQ 8 thread wakes up rtc_wakeup
rtc_wakeup fast-thread returns from sys_read()
rtc_wakeup fast-thread enters sys_poll() and returns immediately
rtc_wakeup fast-thread enters sys_read() and blocks
rtc_wakeup slow-thread runs and does the calculations.
[repeat]
this i think shows that the logic is wrong somewhere and that read()
will achieve the blocking. This also means that the sys_read()-return +
sys_poll() overhead is added to the 'IRQ wakeup' overhead!
removing the poll() lines doesnt seem to impact the quality of the data,
but i still see roughly 50 usecs added to the 'real' latency that i see
in traces.
Ingo
* Florian Schmidt <[email protected]> wrote:
> well, how would i best check for the presence of the process/thread "IRQ 8"?
pidof 'IRQ 8' seems to work pretty well.
> Although: is rtc always garanteed to be "IRQ 8" or is this only the
> case on ia32 with XT-PIC?
no. But the following command should work even if rtc is not on
IRQ8:
chrt -f 99 -p `pidof "IRQ \`ls -d /proc/irq/*/rtc | cut -d/ -f4\`"`
Ingo
* Ingo Molnar <[email protected]> wrote:
> removing the poll() lines doesnt seem to impact the quality of the
> data, but i still see roughly 50 usecs added to the 'real' latency
> that i see in traces.
this i think is related to what Thomas observed, that there's a new
irqs-off critical section somewhere. (it's in the new priority handling
code i think.)
Ingo
>
>* Florian Schmidt <[email protected]> wrote:
>
>> new max. jitter: 4.3% (41 usec)
>> new max. jitter: 4.9% (47 usec)
>
>a couple of conceptual questions: why does rtc_wakeup poll() on
>/dev/rtc? Shouldnt a read() be enough?
i suggested to florian that it should model jackd's behaviour as
closely as possible. because jackd requires duplex operation, using
just read/write doesn't work.
On Mon, 1 Nov 2004 14:42:35 +0100
Ingo Molnar <[email protected]> wrote:
>
> * Florian Schmidt <[email protected]> wrote:
>
> > new max. jitter: 4.3% (41 usec)
> > new max. jitter: 4.9% (47 usec)
>
> a couple of conceptual questions: why does rtc_wakeup poll() on
> /dev/rtc? Shouldnt a read() be enough?
well, it works like this:
while(!stopit) {
// returns when data is ready
poll(on /dev/rtc);
// when ready generate the timestamp
cycles = get_cycles();
// now read the data
read(on /dev/rtc);
// and now stuff the data (including timestamp) into the ringbuffer
// rinse and repeat
}
i get the timestamp before reading because i figured i want to take the
timestamp as close as possible to data being available. The read() and
passing the data to the other thread done after the timestamp generation (in
that ca. 1 - 0.1 ms (1024 - 8192 hz) time window which we have until the
next irq occurs)
>
> i'm seeing some weird traces, which show rtc_wakeup doing this cycle:
>
> [~900 usecs pass]
>
> hardirq 8 comes in, wakes IRQ 8 thread
> IRQ 8 thread wakes up rtc_wakeup
>
> rtc_wakeup fast-thread returns from sys_read()
> rtc_wakeup fast-thread enters sys_poll() and returns immediately
> rtc_wakeup fast-thread enters sys_read() and blocks
weird. why could poll return immeaditly? Only when data should be available
right? Ahh, maybe there's less data available than which is needed by
read(). I suppose i need to check if enough data is available. If not,
repoll(), then generate the timestamp. Then read(). I had the impression
that this small amount of data which rtc delivers (4 bytes i think) would
not be split into smaller parts.
It never occured to me that poll() could return with incomplete rtc data
available..
As i don't know of any way of finding out how much data is available i
suppose we can just make the poll() a read(). I suppose overhead is
neglectable right?
>
> rtc_wakeup slow-thread runs and does the calculations.
>
> [repeat]
>
> this i think shows that the logic is wrong somewhere and that read()
> will achieve the blocking. This also means that the sys_read()-return +
> sys_poll() overhead is added to the 'IRQ wakeup' overhead!
>
> removing the poll() lines doesnt seem to impact the quality of the data,
> but i still see roughly 50 usecs added to the 'real' latency that i see
> in traces.
* Ingo Molnar <[email protected]> wrote:
> > removing the poll() lines doesnt seem to impact the quality of the
> > data, but i still see roughly 50 usecs added to the 'real' latency
> > that i see in traces.
>
> this i think is related to what Thomas observed, that there's a new
> irqs-off critical section somewhere. (it's in the new priority
> handling code i think.)
ah, found it. Only RT tasks were supposed to get special priority
handling, while in fact all tasks got it - so when Thomas ran hackbench
(Thomas, you did, right?) it created an O(nr_hackbench) overhead within
the mutex code ... I've uploaded -V0.6.5 to the usual place:
http://redhat.com/~mingo/realtime-preempt/
Thomas, can you confirm that this kernel fixes the irqs-off latencies?
(the priority loop indeed was done with irqs turned off.)
i'm not sure this fix is related to the deadlocks reported though.
Ingo
* Paul Davis <[email protected]> wrote:
> >
> >* Florian Schmidt <[email protected]> wrote:
> >
> >> new max. jitter: 4.3% (41 usec)
> >> new max. jitter: 4.9% (47 usec)
> >
> >a couple of conceptual questions: why does rtc_wakeup poll() on
> >/dev/rtc? Shouldnt a read() be enough?
>
> i suggested to florian that it should model jackd's behaviour as
> closely as possible. because jackd requires duplex operation, using
> just read/write doesn't work.
ok - but i think there should at least be an option to turn the
poll()-ing off. To showcase the best-possible wakeup latency offered by
the kernel :-)
poll() is quite complex and with a good number of locks in the path the
maximum latency increases accordingly.
btw., couldnt jackd use a separate input and output thread (of identical
priority), to be purely read()/write() based? This method should also
solve the priority problems of poll(): the thread woken up later will do
the work later. (hence the _earlier_ interrupt source will be handled
first.) With poll() how do you tell which fd needs attention first, if
both are set?
Ingo
* Florian Schmidt <[email protected]> wrote:
> > i'm seeing some weird traces, which show rtc_wakeup doing this cycle:
> >
> > [~900 usecs pass]
> >
> > hardirq 8 comes in, wakes IRQ 8 thread
> > IRQ 8 thread wakes up rtc_wakeup
> >
> > rtc_wakeup fast-thread returns from sys_read()
> > rtc_wakeup fast-thread enters sys_poll() and returns immediately
> > rtc_wakeup fast-thread enters sys_read() and blocks
>
> weird. why could poll return immeaditly? Only when data should be
> available right? Ahh, maybe there's less data available than which is
> needed by read(). I suppose i need to check if enough data is
> available. If not, repoll(), then generate the timestamp. Then read().
> I had the impression that this small amount of data which rtc delivers
> (4 bytes i think) would not be split into smaller parts.
>
> It never occured to me that poll() could return with incomplete rtc
> data available..
this is how it works:
static unsigned int rtc_poll(struct file *file, poll_table *wait)
{
unsigned long l;
if (rtc_has_irq == 0)
return 0;
poll_wait(file, &rtc_wait, wait);
spin_lock_irq (&rtc_lock);
l = rtc_irq_data;
spin_unlock_irq (&rtc_lock);
if (l != 0)
return POLLIN | POLLRDNORM;
return 0;
}
it seems that it cannot return with incomplete data. rtc_has_irq is a
boot-time-only thing - either the RTC can generate interrupts or it
cannot. rtc_irq_data is updated atomically - either it's full with 4
bytes or it's completely empty.
safest would be to call the read() with O_NONBLOCK and figure out how
often -EAGAIN happens? (O_NONBLOCK is honored by /dev/rtc.)
but i could imagine that you could get into a 'wrong phase' if for
whatever reason an RTC interrupt slips in before the poll()?
Ingo
On Mon, 2004-11-01 at 15:06 +0100, Ingo Molnar wrote:
> ah, found it. Only RT tasks were supposed to get special priority
> handling, while in fact all tasks got it - so when Thomas ran hackbench
> (Thomas, you did, right?) it created an O(nr_hackbench) overhead within
> the mutex code ... I've uploaded -V0.6.5 to the usual place:
Yes, I was running hackbench as usual
> http://redhat.com/~mingo/realtime-preempt/
>
> Thomas, can you confirm that this kernel fixes the irqs-off latencies?
> (the priority loop indeed was done with irqs turned off.)
The latencies are still there. I have the feeling it's worse than 0.6.2.
It's definitely irq-off. I have a card with a controller, which produces
an 2ms interrupt. The controller busy loops until the second level ack
is done. The time is measured from raising the irq to the 2nd level ack.
The irqhandler is using NODELAY and keeps irqs disabled. So the measured
time in the controller is the irqs disabled time + the irq latency
(which is ~7?s). The testrun on 0.6.5 showed latencies up to 600?s
within 10 minutes.
After 15 Mintes the keyboard was dead.
I'm porting some of the Libertos debugging stuff into 0.6.5, so I can
instrument the problem.
More later.
tglx
Ingo Molnar wrote:
>
>
> i've uploaded -V0.6.3 to the usual place:
>
> http://redhat.com/~mingo/realtime-preempt/
>
> which fixes two issues:
>
> - priorities of SCHED_OTHER tasks not getting properly managed, hence
> they could end up 'starving' other SCHED_OTHER tasks pretty
> indefinitely. This could possibly solve the 'temporary lockup'
> problem reported by Mark H Johnson.
>
> - fixed the 'high load average' bug
>
> i dont know whether this will solve the 'hard lockups' reported though.
> It could solve your problem because the 'find /' kept running so it
> wasnt a hard lockup. I'll keep testing and i'll fix any problem i can
> reproduce myself.
>
OK. Just tested in a hurry with RT-V0.6.4, and it passed three times in a
row my jackd-R + 9*fluidsynth tests, without freezing, so I think the
problem was the 'starvation' one.
OTOH, I can say that the results I'm early reading are really an
improvement. Fewer xruns and smaller delays. Nice.
Cheers.
--
rncbc aka Rui Nuno Capela
[email protected]
Florian Schmidt wrote:
> On Mon, 1 Nov 2004 14:42:35 +0100
> Ingo Molnar <[email protected]> wrote:
>
>
>>* Florian Schmidt <[email protected]> wrote:
>>
>>
>>>new max. jitter: 4.3% (41 usec)
>>>new max. jitter: 4.9% (47 usec)
>>
>>a couple of conceptual questions: why does rtc_wakeup poll() on
>>/dev/rtc? Shouldnt a read() be enough?
>
>
> well, it works like this:
>
>
> while(!stopit) {
> // returns when data is ready
> poll(on /dev/rtc);
>
> // when ready generate the timestamp
> cycles = get_cycles();
>
> // now read the data
> read(on /dev/rtc);
>
> // and now stuff the data (including timestamp) into the ringbuffer
>
> // rinse and repeat
> }
>
> i get the timestamp before reading because i figured i want to take the
> timestamp as close as possible to data being available. The read() and
> passing the data to the other thread done after the timestamp generation (in
> that ca. 1 - 0.1 ms (1024 - 8192 hz) time window which we have until the
> next irq occurs)
>
>
>>i'm seeing some weird traces, which show rtc_wakeup doing this cycle:
>>
>> [~900 usecs pass]
>>
>> hardirq 8 comes in, wakes IRQ 8 thread
>> IRQ 8 thread wakes up rtc_wakeup
>>
>> rtc_wakeup fast-thread returns from sys_read()
>> rtc_wakeup fast-thread enters sys_poll() and returns immediately
>> rtc_wakeup fast-thread enters sys_read() and blocks
>
>
> weird. why could poll return immeaditly? Only when data should be available
> right? Ahh, maybe there's less data available than which is needed by
> read(). I suppose i need to check if enough data is available. If not,
> repoll(), then generate the timestamp. Then read(). I had the impression
> that this small amount of data which rtc delivers (4 bytes i think) would
> not be split into smaller parts.
>
> It never occured to me that poll() could return with incomplete rtc data
> available..
>
> As i don't know of any way of finding out how much data is available i
> suppose we can just make the poll() a read(). I suppose overhead is
> neglectable right?
>
Someone please correct me if I'm wrong, but is there any MORE overhead
associated with a blocking read than with a poll or select? Won't the
process just go to sleep until there is data available?
>
>> rtc_wakeup slow-thread runs and does the calculations.
>>
>> [repeat]
>>
>>this i think shows that the logic is wrong somewhere and that read()
>>will achieve the blocking. This also means that the sys_read()-return +
>>sys_poll() overhead is added to the 'IRQ wakeup' overhead!
>>
>>removing the poll() lines doesnt seem to impact the quality of the data,
>>but i still see roughly 50 usecs added to the 'real' latency that i see
>>in traces.
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
Ingo Molnar wrote:
>
> I've uploaded -V0.6.5 to the usual place:
>
> http://redhat.com/~mingo/realtime-preempt/
>
Build error:
kernel/built-in.o(.text+0x308a): In function `show_state':
: undefined reference to `show_all_locks'
kernel/built-in.o(.text+0x30bc): In function `show_state':
: undefined reference to `show_all_locks'
make: *** [.tmp_vmlinux1] Error 1
Bye.
--
rncbc aka Rui Nuno Capela
[email protected]
On Mon, 2004-11-01 at 17:04 +0000, Rui Nuno Capela wrote:
> Ingo Molnar wrote:
> >
> > I've uploaded -V0.6.5 to the usual place:
> >
> > http://redhat.com/~mingo/realtime-preempt/
> >
>
> Build error:
>
> kernel/built-in.o(.text+0x308a): In function `show_state':
> : undefined reference to `show_all_locks'
> kernel/built-in.o(.text+0x30bc): In function `show_state':
> : undefined reference to `show_all_locks'
> make: *** [.tmp_vmlinux1] Error 1
You have lock debugging disabled.
tglx
--- linux/lib/rwsem-generic.o.c 2004-11-01 18:07:02.000000000 +0100
+++ linux/lib/rwsem-generic.c 2004-11-01 15:49:27.000000000 +0100
@@ -413,6 +413,9 @@
{
}
+void show_all_locks(void)
+{
+}
#endif
/*
Ingo Molnar wrote:
> * Ingo Molnar <[email protected]> wrote:
>
>
>>>removing the poll() lines doesnt seem to impact the quality of the
>>>data, but i still see roughly 50 usecs added to the 'real' latency
>>>that i see in traces.
>>
>>this i think is related to what Thomas observed, that there's a new
>>irqs-off critical section somewhere. (it's in the new priority
>>handling code i think.)
>
>
> ah, found it. Only RT tasks were supposed to get special priority
> handling, while in fact all tasks got it - so when Thomas ran hackbench
> (Thomas, you did, right?) it created an O(nr_hackbench) overhead within
> the mutex code ... I've uploaded -V0.6.5 to the usual place:
>
> http://redhat.com/~mingo/realtime-preempt/
>
V0.6.5 built and booted fine on my SMP workstation. However, just a few
minutes after booting it exhibited behavior like a system does when you
can't fork any new processses.
The system was responsive.
I could switch between windows.
I could switch from X to a virtual terminal.
I could type in commands but they never return.
At the virtual terminal login I could type the login but never get a
passwd prompt.
Nothing in the logs during this timeframe.
Reset button was the only way to recover.
kr
On Mon, 2004-11-01 at 16:47 +0100, Thomas Gleixner wrote:
> The latencies are still there. I have the feeling it's worse than 0.6.2.
>
> It's definitely irq-off. I have a card with a controller, which produces
> an 2ms interrupt. The controller busy loops until the second level ack
> is done. The time is measured from raising the irq to the 2nd level ack.
This was my conclusion as well. I have a patch sitting around to add
this to the emu10k1 ALSA driver, it's quite useful. It would be nice if
there were a facility in the kernel to easily identify missed interrupts
like this or (even better) unbalanced irq disable/enable - AFAICT
userspace alone cannot reliably distinguish lost interrupts from
scheduling problems (though you can get a lot of hints). Paul mentioned
trying to debug the unbalanced irq disable in his talk at ZKM 2003, and
said it's hard because the hardware will enable/disable interrupts on
its own and he could not identify all those places. Ingo, is there an
easy way to trace this like we do for unbalanced preempt count?
I think there might even be a _very_ rare errant irq disable in T3 even.
On my 2-day test runs with 100s of millions of samples, I got 2 or 3
outliers in each graph. Jow Gwinn from LKML (thanks Joe) ran a
statistical analysis that showed these to be independent from the 4 or 5
underlying exponential distributions (each corresponding to a known or
suspected nonpreemptible section). Our theory was that these were very
rare code paths or race conditions that left IRQs off. Unfortunately
this seems impossible to debug unless you have a way to make the machine
crash dump immediately when it detects the situation.
Anyway, the clearest way to demonstrate the problem with the -V series
here is to run the version of Florian's tool that tells you how many
interrupts were missed. If I spin my (USB, not sharing irq with
soundcard) trackball as fast as I can I can get it up to 54 or 55 in a
row. If I move it just the right way I can see the humps appear - 2,
then 15, then 50 then 15 then 2 missed interrupts in a row. This is at
1024Hz - at 2048 I can get it to miss several hundred IRQs, but this
inevitably locks the machine.
I suspect the lockups and the latencies are same bug.
Lee
On Mon, 2004-11-01 at 12:55 -0500, Lee Revell wrote:
> I think there might even be a _very_ rare errant irq disable in T3 even.
> On my 2-day test runs with 100s of millions of samples, I got 2 or 3
> outliers in each graph. Jow Gwinn from LKML (thanks Joe) ran a
> statistical analysis that showed these to be independent from the 4 or 5
> underlying exponential distributions (each corresponding to a known or
> suspected nonpreemptible section). Our theory was that these were very
> rare code paths or race conditions that left IRQs off. Unfortunately
> this seems impossible to debug unless you have a way to make the machine
> crash dump immediately when it detects the situation.
>
> Anyway, the clearest way to demonstrate the problem with the -V series
> here is to run the version of Florian's tool that tells you how many
> interrupts were missed. If I spin my (USB, not sharing irq with
> soundcard) trackball as fast as I can I can get it up to 54 or 55 in a
> row. If I move it just the right way I can see the humps appear - 2,
> then 15, then 50 then 15 then 2 missed interrupts in a row. This is at
> 1024Hz - at 2048 I can get it to miss several hundred IRQs, but this
> inevitably locks the machine.
>
> I suspect the lockups and the latencies are same bug.
Until now there was no output on lockup. No I got one.
The command was: cat /proc/interrupts
Hope that helps
tglx
cat/1488: BUG in down_mutex
at /work/thomas/Linux/2.6.9-mm1-RT/lib/rwsem-generic.c:1059
[<c01ca9e2>] down_mutex+0x72/0x80 (8)
[<c012db9a>] __mutex_lock+0x2a/0x40 (36)
[<c012dbc7>] _mutex_lock+0x17/0x20 (12)
[<c013fd62>] kfree+0x52/0xf0 (8)
[<c013fd62>] kfree+0x52/0xf0 (4)
[<c01774a2>] single_release+0x32/0x40 (32)
[<c01571ba>] __fput+0x13a/0x150 (20)
[<c0155899>] filp_close+0x59/0x90 (32)
[<c015593f>] sys_close+0x6f/0xa0 (24)
[<c010603b>] syscall_call+0x7/0xb (28)
* Thomas Gleixner <[email protected]> wrote:
> > http://redhat.com/~mingo/realtime-preempt/
> >
> > Thomas, can you confirm that this kernel fixes the irqs-off latencies?
> > (the priority loop indeed was done with irqs turned off.)
>
> The latencies are still there. I have the feeling it's worse than 0.6.2.
what is the worst latency you can trigger with Florian's latest
rtc_wakeup code? (please re-download it, there has been a recent update)
also, there are no "arbitrary load" latency guarantees with
DEADLOCK_DETECTION turned on, since we search the list of all held locks
during task-exit time - this can generate pretty bad latencies e.g.
during hackbench.
Ingo
* Lee Revell <[email protected]> wrote:
> This was my conclusion as well. I have a patch sitting around to add
> this to the emu10k1 ALSA driver, it's quite useful. It would be nice
> if there were a facility in the kernel to easily identify missed
> interrupts like this or (even better) unbalanced irq disable/enable -
> AFAICT userspace alone cannot reliably distinguish lost interrupts
> from scheduling problems (though you can get a lot of hints). Paul
> mentioned trying to debug the unbalanced irq disable in his talk at
> ZKM 2003, and said it's hard because the hardware will enable/disable
> interrupts on its own and he could not identify all those places.
> Ingo, is there an easy way to trace this like we do for unbalanced
> preempt count?
i wrote a cli/sti latency tracer a couple of years ago so it's possible.
Note that an irqs-off condition is near impossible to 'leak' into
userspace code though, since the x86 iret path restores flags to the
previous value. Worst-case the irqs-off condition may leak into
kernelspace, and that can still cause bad effects. X startup/shutdown
can disable interrupts for a long time, was that excluded from your
testing?
Ingo
>poll() is quite complex and with a good number of locks in the path the
>maximum latency increases accordingly.
how can poll(2) be more complex than read/write? if it is, it
shouldn't be ;)
>btw., couldnt jackd use a separate input and output thread (of identical
>priority), to be purely read()/write() based? This method should also
>solve the priority problems of poll(): the thread woken up later will do
>the work later. (hence the _earlier_ interrupt source will be handled
>first.) With poll() how do you tell which fd needs attention first, if
>both are set?
we don't really care which one needs attention "first". the jack process
cycle ends up consuming and producing data -
before the ALSA backend tells jackd to run the process cycle,
it will read the data from the capture fd (though its done
using mmap, so there is no read(2) call)
just before jackd goes back to sleep waiting for the next
interrupt, the ALSA backend will write data to the playback fd
(again, using mmap, so there is no actual write(2) call)
it is important to use mmap - it avoids multiple kernel entries and
data copying. for consumer cards this doesn't matter much, but for
high end multichannel cards, the data copy would be inexcusable.
--p
On Mon, 1 Nov 2004 14:40:47 -0800
Bill Huey (hui) <[email protected]> wrote:
> Unlikely, it's got to lock the entire kernel to make sure that things aren't
> changing under it. Getting measurements is useful at this stage, but don't expect
> it to be a finished product any time soon and please keep that in mind. Stability
> should be, if it's not already, the single most important things in this project
> at this phase. Getting numbers now for a specific single application is going to
> be of limit use until the system is stable enough for general characterization.
>
> Keep doing it, but keep in mind that anything you're going to get at this time
> is going to be very coarse. Don't put too much weight on it.
>
> That's my take.
ack. understood. i was just asking since i don't have a second machine and
thus am not really able to help with the deadlock debugging. so i figured i
could at least do some timing. btw: even with deadlock detection, the
results for 0.6.5 looked pretty good [in 10 minutes uptime ca.3-4% max
jitter [30something usecs]. until the deadlock that is [i head three finds
plus kernel compile at nice -10 running]..
flo
Hi,
This is after "modprobe -r uhci-hcd" on -V0.6.5:
BUG: sleeping function called from invalid context rmmod(5093) at
kernel/mutex.c:30
in_atomic():1 [00000001], irqs_disabled():1
[<c01060d3>] dump_stack+0x23/0x30 (20)
[<c011b2e2>] __might_sleep+0xc2/0xe0 (36)
[<c0134d78>] __mutex_lock+0x38/0x50 (24)
[<c0134dad>] _mutex_lock+0x1d/0x20 (16)
[<c01482c4>] do_drain+0x24/0x60 (32)
[<c0148269>] smp_call_function_all_cpus+0x29/0x60 (28)
[<c0148327>] drain_cpu_caches+0x27/0x60 (28)
[<c014837c>] __cache_shrink+0x1c/0x100 (36)
[<c014855d>] kmem_cache_destroy+0x9d/0x190 (28)
[<f8a40e24>] uhci_hcd_cleanup+0x24/0x67 [uhci_hcd] (16)
[<c0138c40>] sys_delete_module+0x120/0x150 (96)
[<c0105293>] syscall_call+0x7/0xb (-8124)
---------------------------
| preempt count: 00000002 ]
| 2-level deep critical section nesting:
----------------------------------------
.. [<c0148263>] .... smp_call_function_all_cpus+0x23/0x60
.....[<c0148327>] .. ( <= drain_cpu_caches+0x27/0x60)
.. [<c0136b0d>] .... print_traces+0x1d/0x90
.....[<c01060d3>] .. ( <= dump_stack+0x23/0x30)
Michal
took jackit-devel off cc, since it bounced anyways [too many recipients]
On Mon, 1 Nov 2004 19:46:15 +0100
Ingo Molnar <[email protected]> wrote:
>
> * Thomas Gleixner <[email protected]> wrote:
>
> > > http://redhat.com/~mingo/realtime-preempt/
> > >
> > > Thomas, can you confirm that this kernel fixes the irqs-off latencies?
> > > (the priority loop indeed was done with irqs turned off.)
> >
> > The latencies are still there. I have the feeling it's worse than 0.6.2.
>
> what is the worst latency you can trigger with Florian's latest
> rtc_wakeup code? (please re-download it, there has been a recent update)
I have fixed one or two more small issues:
1] the cpu cycles/s measurement was done while not yet running SCHED_FIFO.
changed it so the rt priv is aquired beforehand
2] when one or more irq's were missed, the cycle timestamps for the last wakeup
do not get used anymore for neither threshold nor max jitter reporting.
It's kind of pointless to calculate jitter for a period with the
fundamental requirement that no irq's be missed being violated.
3] the output file format (-o) is now
num_of_irqs_since_last_wakeup cycles_count
and basically looks like this:
1 116817809121123
1 116817810280681
1 116817811456573
1 116817812617197
1 116817813788473
1 116817814948983
1 116817816121533
...
i suppose this data might be easily fed into a histogram producing script or
something..
New download place:
http://www.affenbande.org/~tapas/wiki/index.php?rtc_wakeup
direct link (has changed. i'll remove the old tarball)
http://affenbande.org/~tapas/rtc_wakeup/rtc_wakeup.tgz
Whoever wants to be notified of new versions should mail me a reply
(private), so i can drop em a line in case (lkml is hi traffic enough
already)..
>
> also, there are no "arbitrary load" latency guarantees with
> DEADLOCK_DETECTION turned on, since we search the list of all held locks
> during task-exit time - this can generate pretty bad latencies e.g.
> during hackbench.
btw: i see the same build failure as Rui with lock debugging disabled. Since
the lock debugging can screw timings, will this be fixed in [one of] the next
version[s]?
flo
On Mon, Nov 01, 2004 at 11:51:25PM +0100, Florian Schmidt wrote:
[...lock debugging...]
> ack. understood. i was just asking since i don't have a second machine and
> thus am not really able to help with the deadlock debugging. so i figured i
> could at least do some timing. btw: even with deadlock detection, the
> results for 0.6.5 looked pretty good [in 10 minutes uptime ca.3-4% max
> jitter [30something usecs]. until the deadlock that is [i head three finds
> plus kernel compile at nice -10 running]..
The lock chains aren't that deep in Linux so the algorithmic complexity
is not going to hit some crazy polynomial time unless there's some seriously
nasty contention at a certain point in the kernel (billions of readers for
example against a write aquire). But when we start to see things like that
under pressure is when we need to start shortening the need for that/those
lock(s) for that/those critical section(s) in question.
bill
On Mon, Nov 01, 2004 at 11:30:37PM +0100, Florian Schmidt wrote:
> took jackit-devel off cc, since it bounced anyways [too many recipients]
> On Mon, 1 Nov 2004 19:46:15 +0100
> Ingo Molnar <[email protected]> wrote:
...
> >
> > also, there are no "arbitrary load" latency guarantees with
> > DEADLOCK_DETECTION turned on, since we search the list of all held locks
> > during task-exit time - this can generate pretty bad latencies e.g.
> > during hackbench.
> btw: i see the same build failure as Rui with lock debugging disabled. Since
> the lock debugging can screw timings, will this be fixed in [one of] the next
> version[s]?
Unlikely, it's got to lock the entire kernel to make sure that things aren't
changing under it. Getting measurements is useful at this stage, but don't expect
it to be a finished product any time soon and please keep that in mind. Stability
should be, if it's not already, the single most important things in this project
at this phase. Getting numbers now for a specific single application is going to
be of limit use until the system is stable enough for general characterization.
Keep doing it, but keep in mind that anything you're going to get at this time
is going to be very coarse. Don't put too much weight on it.
That's my take.
bill
Thomas Gleixner wrote:
> Rui Nuno Capela wrote:
>> Ingo Molnar wrote:
>> >
>> > I've uploaded -V0.6.5 to the usual place:
>> >
>> > http://redhat.com/~mingo/realtime-preempt/
>> >
>>
>> Build error:
>>
>> kernel/built-in.o(.text+0x308a): In function `show_state':
>> : undefined reference to `show_all_locks'
>> kernel/built-in.o(.text+0x30bc): In function `show_state':
>> : undefined reference to `show_all_locks'
>> make: *** [.tmp_vmlinux1] Error 1
>
> You have lock debugging disabled.
>
OK. Applied Thomas' corrective patch and now a debug-stripped RT-V0.6.5
has been built fine on both of my machines.
But I'm afraid to have bad news.
My (now infamous:) jackd-R + 9*fluidsynth test is being abruptly killed by
jack_watchdog. Annoyingly always. On both of my configurations, laptop
(P4/UP) and desktop (P4/SMT), and in any other circunstance, jackd -R
can't survive for long. It just stalls after a while.
To be sincere, this happened once while testing with RT-V0.6.4, but only
after too many successful test runs.
Take care.
--
rncbc aka Rui Nuno Capela
[email protected]
* Bill Huey <[email protected]> wrote:
> The lock chains aren't that deep in Linux so the algorithmic
> complexity is not going to hit some crazy polynomial time unless
> there's some seriously nasty contention at a certain point in the
> kernel (billions of readers for example against a write aquire). But
> when we start to see things like that under pressure is when we need
> to start shortening the need for that/those lock(s) for that/those
> critical section(s) in question.
also note that in the -U series i removed the true 'read' logic from
semaphores. What we have now are single writers only, plus readers
emulated as a writer plus the ability to self-recurse. ('writers' are
not allowed to self-recurse.) This is quite close to the semantic needs
of Linux rwlocks and rwsems and it simplified both locking, deadlock
detection and PI quite significantly.
Ingo
* Florian Schmidt <[email protected]> wrote:
> > also, there are no "arbitrary load" latency guarantees with
> > DEADLOCK_DETECTION turned on, since we search the list of all held locks
> > during task-exit time - this can generate pretty bad latencies e.g.
> > during hackbench.
>
> btw: i see the same build failure as Rui with lock debugging disabled.
(just remove the offending call to show_all_locks())
> Since the lock debugging can screw timings, will this be fixed in [one
> of] the next version[s]?
yeah, i think so. Right now i've increased the complexity of the checks
to root out bugs as clearly there's a stability problem. Once things
have calmed down i think we can remove the 'check all locks at exit
time' portion that is the problematic one.
note that you really need some insane loads with thousands of tasks in
the runqueue (hackbench) to really trigger that kind of overhead. Normal
loads are not supposed to trigger any of this, even with full debugging
turned on.
Ingo
* Paul Davis <[email protected]> wrote:
> >poll() is quite complex and with a good number of locks in the path the
> >maximum latency increases accordingly.
>
> how can poll(2) be more complex than read/write? if it is, it
> shouldn't be ;)
poll() is fundamentally more complex: it has to watch multiple channels,
while read()/write() has to watch only a single event channel. The fact
that read()/write() also has to do some actual IO makes little
difference to complexity, as poll() already has to do most of the
locking read()/write() has to do, to figure out that it _could_ do the
read()/write().
> >btw., couldnt jackd use a separate input and output thread (of identical
> >priority), to be purely read()/write() based? This method should also
> >solve the priority problems of poll(): the thread woken up later will do
> >the work later. (hence the _earlier_ interrupt source will be handled
> >first.) With poll() how do you tell which fd needs attention first, if
> >both are set?
>
> we don't really care which one needs attention "first". [...]
well, order of processing can make a difference under a high event load.
Couldnt capture and playback interrupts be separate and differently
timed? I understand your previous points that the audio 'channels' are
highly coupled and cannot be considered separate 'event sources', but is
the same true for all the fds that jackd passes into poll()? Is it true
if multiple cards are used?
the scenario that could trigger problems is that if an event (or group
of events) triggers some processing in the highprio thread, and two more
events arrive, one at the beginning of the previous processing, one at
the end of it. Once the highprio thread calls poll() again, the timing
of the two events has been lost - and jackd could end up processing the
_later_ event.
while this should normally make no difference with low audio loads, i
can very much see this causing problem as the number of cards/events
increases. What is typically the longest amount of time the highprio
thread can spend 'processing' without being actively poll()-ing? I know
it is typically short, but what is roughly the longest amount of time?
(is it the 1.4 msecs displayed in one of Rui's earlier testresults?)
Ingo
* Thomas Gleixner <[email protected]> wrote:
> > Thomas, can you confirm that this kernel fixes the irqs-off latencies?
> > (the priority loop indeed was done with irqs turned off.)
>
> The latencies are still there. I have the feeling it's worse than 0.6.2.
update to others: Thomas debugged this problem today and found the place
that kept irqs disabled for a long time: it was update_process_times().
(which i recently touched to break latencies there - but forgot that the
lock is an irqs-off lock!)
i've uploaded a fixed kernel (-V0.6.8) to:
http://redhat.com/~mingo/realtime-preempt/
(this kernel also has the module-put-unlock-kernel fix that should solve
the other warning reported by Thomas and Bill.)
Ingo
>i've uploaded a fixed kernel (-V0.6.8) to:
>
> http://redhat.com/~mingo/realtime-preempt/
OK. Will step up shortly but have some test results from -V0.6.7
that I just collected. The system crashed with LOCKUP messages during
both runs but I did get some real time tests run. No build problems.
First run
[1] boot to single user was uneventful.
[2] telinit 5 was also OK, no atomic underflow warnings.
[3] Logged in, set up my RT settings (udma2, IRQ priorities) and
did a quick sample of wakeup timing. Still appear to have the false
positives, even on an otherwise idle system. For example:
(get_ltrace.sh/3502/CPU#0): new 936 us maximum-latency wakeup.
(kdeinit/3198/CPU#0): new 971 us maximum-latency wakeup.
(cat/3504/CPU#0): new 1776 us maximum-latency wakeup.
(cat/3504/CPU#0): new 5286 us maximum-latency wakeup.
(cat/3504/CPU#0): new 71828 us maximum-latency wakeup.
(cupsd/2061/CPU#1): new 341 us maximum-latency wakeup.
(ksoftirqd/0/3/CPU#0): new 64940 us maximum-latency wakeup.
(get_ltrace.sh/3554/CPU#1): new 1299 us maximum-latency wakeup.
(cat/3556/CPU#1): new 4361 us maximum-latency wakeup.
(cat/3556/CPU#1): new 68847 us maximum-latency wakeup.
(get_ltrace.sh/3582/CPU#1): new 963 us maximum-latency wakeup.
(cat/3584/CPU#1): new 2273 us maximum-latency wakeup.
(cat/3584/CPU#1): new 66211 us maximum-latency wakeup.
At this point, the system locked up and a little bit later dumped
a LOCKUP on the serial console (at end of message).
Second run.
[1] Booting to single user and telinit 5 was uneventful.
[2] Logged in and used
echo 0 > /proc/sys/kernel/preempt_wakeup_timing
to go back to tracking the long latency locks. Set up RT environment
as well.
[3] Started to collect latency timing data; no false positives during
a few minutes of "idle" time.
[4] Started the real time stress test.
- X - data looks generally OK with huge (max 13 msec) spikes near
the start and end.
- top - data is noisy; had a number of periods (up to 1 second) where
the CPU loop was delayed up to a millisecond or so (about 100% overhead)
and some spikes even longer than above (max 22 msec).
- network output - machine crashed again, a similar lockup problem
to the first run. (dump at end of message)
Something I noticed during these runs. If I moved the mouse up / down, the
audio seemed to change (slightly lower tone). Mouse movements left / right
did not seem to cause such a noticeable change. This may be related to the
spikes in the measured top stress test.
--Mark
First LOCKUP
NMI Watchdog detected LOCKUP
Pid: 3933, comm: cpu_burn
EIP: 0073:[<08048340>] CPU: 0
EIP is at 0x8048340
ESP: 007b:bffffa40 EFLAGS: 00200282 Not tainted (2.6.9-mm1-RT-V0.6.7)
EAX: 00000000 EBX: 00711ffc ECX: bffffadc EDX: bffffad4
ESI: 00000001 EDI: 007140fc EBP: bffffa48 DS: 007b ES: 007b
CR0: 8005003b CR2: 00681400 CR3: 015735e0 CR4: 000006f0
[<c0105bec>] show_regs+0x14c/0x174 (36)
[<c0115f4f>] nmi_watchdog_tick+0x12f/0x140 (28)
[<c0109c0c>] default_do_nmi+0x6c/0x110 (96)
[<c0109d2d>] do_nmi+0x6d/0x70 (24)
[<c0108735>] nmi_stack_correct+0x1e/0x2e (-196314476)
---------------------------
| preempt count: 00010002 ]
| 2-level deep critical section nesting:
----------------------------------------
.. [<c0326a8f>] .... _spin_lock+0x1f/0x70
.....[<c0115f47>] .. ( <= nmi_watchdog_tick+0x127/0x140)
.. [<c013d7dd>] .... print_traces+0x1d/0x60
.....[<c0105bec>] .. ( <= show_regs+0x14c/0x174)
on CPU1, eip c013be90, registers:
Modules linked in: snd_pcm_oss snd_mixer_oss snd_ens1371 snd_rawmidi
snd_seq_d
evice snd_ac97_codec snd_pcm snd_timer snd_page_alloc gameport snd
soundcore ipv
6 parport_pc lp parport autofs4 sunrpc 8139too mii floppy sg scsi_mod
microcode
dm_mod uhci_hcd ext3 jbd
CPU: 1
EIP: 0060:[<c013be90>] Not tainted VLI
EFLAGS: 00200002 (2.6.9-mm1-RT-V0.6.7)
EIP is at ___trace+0x40/0x170
eax: 00017780 ebx: c0445180 ecx: c01e24ea edx: c0326a81
esi: 00000001 edi: 00200002 ebp: cad35d48 esp: cad35d18
ds: 007b es: 007b ss: 0068 preempt: 00010004
Process sleep (pid: 4185, threadinfo=cad34000 task=cb3c2a90)
Stack: c0109cc0 00200002 cad35d54 c0372424 00200002 cad35d44 c0109d2d
cad34000
c0372420 c0372424 cb3c2a90 00200046 cad35d58 c013bfed c0326a81
c01e2308
cad35d6c c0114fd8 00000003 c01e24ea c0372428 cad35d80 c0326a81
00000001
Call Trace:
[<c0108a1f>] show_stack+0x8f/0xb0 (28)
[<c0108bdf>] show_registers+0x16f/0x1e0 (56)
[<c0109b5f>] die_nmi+0x5f/0xa0 (24)
[<c0115f0f>] nmi_watchdog_tick+0xef/0x140 (28)
[<c0109c0c>] default_do_nmi+0x6c/0x110 (96)
[<c0109d2d>] do_nmi+0x6d/0x70 (24)
[<c0108735>] nmi_stack_correct+0x1e/0x2e (116)
[<c013bfed>] __mcount+0x1d/0x30 (16)
[<c0114fd8>] mcount+0x14/0x18 (20)
[<c0326a81>] _spin_lock+0x11/0x70 (20)
[<c01e2308>] _down_write_trylock+0x58/0x290 (52)
[<c01e3325>] down_trylock+0x45/0x180 (52)
[<c0123f95>] vprintk+0xf5/0x170 (36)
[<c0123e8d>] printk+0x1d/0x30 (16)
[<c0108945>] show_trace+0x95/0xe0 (32)
[<c0108a63>] dump_stack+0x23/0x30 (20)
[<c013c98e>] check_preempt_timing+0x16e/0x300 (76)
[<c013ce7f>] sub_preempt_count+0x7f/0xf0 (32)
[<c01147ba>] flush_tlb_mm+0x5a/0x110 (36)
[<c015965e>] unmap_vmas+0x10e/0x1c0 (56)
[<c015e60c>] exit_mmap+0x5c/0x130 (56)
[<c0120c39>] mmput+0x49/0x160 (28)
[<c0126bf3>] do_exit+0x183/0x5c0 (40)
[<c0127170>] do_group_exit+0x40/0xe0 (40)
[<c0107b89>] sysenter_past_esp+0x52/0x71 (-8124)
---------------------------
| preempt count: 00010005 ]
| 5-level deep critical section nesting:
----------------------------------------
.. [<c0114783>] .... flush_tlb_mm+0x23/0x110
.....[<c015965e>] .. ( <= unmap_vmas+0x10e/0x1c0)
.. [<c0326aff>] .... _spin_lock_irqsave+0x1f/0x80
.....[<c0123ecb>] .. ( <= vprintk+0x2b/0x170)
.. [<c0326a8f>] .... _spin_lock+0x1f/0x70
.....[<c01e24ea>] .. ( <= _down_write_trylock+0x23a/0x290)
.. [<c0326a8f>] .... _spin_lock+0x1f/0x70
.....[<c0109b22>] .. ( <= die_nmi+0x22/0xa0)
.. [<c013d7dd>] .... print_traces+0x1d/0x60
.....[<c0108a1f>] .. ( <= show_stack+0x8f/0xb0)
Code: b8 00 e0 ff ff 89 fe c1 ee 09 21 e0 89 45 ec 83 f6 01 8b 40 10 83 e6
01
69 c0 80 77 01 00 8d 98 00 da 42 c0 f0 ff 80 00 da 42 c0 <8b> 80 00 da 42
c0 48
74 0d f0 ff 0b 57 9d 83 c4 24 5b 5e 5f 5d
console shuts up ...
Second LOCKUP
NMI Watchdog detected LOCKUP
Pid: 3933, comm: cpu_burn
EIP: 0073:[<08048340>] CPU: 0
EIP is at 0x8048340
ESP: 007b:bffffa40 EFLAGS: 00200282 Not tainted (2.6.9-mm1-RT-V0.6.7)
EAX: 00000000 EBX: 00711ffc ECX: bffffadc EDX: bffffad4
ESI: 00000001 EDI: 007140fc EBP: bffffa48 DS: 007b ES: 007b
CR0: 8005003b CR2: 00681400 CR3: 015735e0 CR4: 000006f0
[<c0105bec>] show_regs+0x14c/0x174 (36)
[<c0115f4f>] nmi_watchdog_tick+0x12f/0x140 (28)
[<c0109c0c>] default_do_nmi+0x6c/0x110 (96)
[<c0109d2d>] do_nmi+0x6d/0x70 (24)
[<c0108735>] nmi_stack_correct+0x1e/0x2e (-196314476)
---------------------------
| preempt count: 00010002 ]
| 2-level deep critical section nesting:
----------------------------------------
.. [<c0326a8f>] .... _spin_lock+0x1f/0x70
.....[<c0115f47>] .. ( <= nmi_watchdog_tick+0x127/0x140)
.. [<c013d7dd>] .... print_traces+0x1d/0x60
.....[<c0105bec>] .. ( <= show_regs+0x14c/0x174)
on CPU1, eip c013be90, registers:
Modules linked in: snd_pcm_oss snd_mixer_oss snd_ens1371 snd_rawmidi
snd_seq_d
evice snd_ac97_codec snd_pcm snd_timer snd_page_alloc gameport snd
soundcore ipv
6 parport_pc lp parport autofs4 sunrpc 8139too mii floppy sg scsi_mod
microcode
dm_mod uhci_hcd ext3 jbd
CPU: 1
EIP: 0060:[<c013be90>] Not tainted VLI
EFLAGS: 00200002 (2.6.9-mm1-RT-V0.6.7)
EIP is at ___trace+0x40/0x170
eax: 00017780 ebx: c0445180 ecx: c01e24ea edx: c0326a81
esi: 00000001 edi: 00200002 ebp: cad35d48 esp: cad35d18
ds: 007b es: 007b ss: 0068 preempt: 00010004
Process sleep (pid: 4185, threadinfo=cad34000 task=cb3c2a90)
Stack: c0109cc0 00200002 cad35d54 c0372424 00200002 cad35d44 c0109d2d
cad34000
c0372420 c0372424 cb3c2a90 00200046 cad35d58 c013bfed c0326a81
c01e2308
cad35d6c c0114fd8 00000003 c01e24ea c0372428 cad35d80 c0326a81
00000001
Call Trace:
[<c0108a1f>] show_stack+0x8f/0xb0 (28)
[<c0108bdf>] show_registers+0x16f/0x1e0 (56)
[<c0109b5f>] die_nmi+0x5f/0xa0 (24)
[<c0115f0f>] nmi_watchdog_tick+0xef/0x140 (28)
[<c0109c0c>] default_do_nmi+0x6c/0x110 (96)
[<c0109d2d>] do_nmi+0x6d/0x70 (24)
[<c0108735>] nmi_stack_correct+0x1e/0x2e (116)
[<c013bfed>] __mcount+0x1d/0x30 (16)
[<c0114fd8>] mcount+0x14/0x18 (20)
[<c0326a81>] _spin_lock+0x11/0x70 (20)
[<c01e2308>] _down_write_trylock+0x58/0x290 (52)
[<c01e3325>] down_trylock+0x45/0x180 (52)
[<c0123f95>] vprintk+0xf5/0x170 (36)
[<c0123e8d>] printk+0x1d/0x30 (16)
[<c0108945>] show_trace+0x95/0xe0 (32)
[<c0108a63>] dump_stack+0x23/0x30 (20)
[<c013c98e>] check_preempt_timing+0x16e/0x300 (76)
[<c013ce7f>] sub_preempt_count+0x7f/0xf0 (32)
[<c01147ba>] flush_tlb_mm+0x5a/0x110 (36)
[<c015965e>] unmap_vmas+0x10e/0x1c0 (56)
[<c015e60c>] exit_mmap+0x5c/0x130 (56)
[<c0120c39>] mmput+0x49/0x160 (28)
[<c0126bf3>] do_exit+0x183/0x5c0 (40)
[<c0127170>] do_group_exit+0x40/0xe0 (40)
[<c0107b89>] sysenter_past_esp+0x52/0x71 (-8124)
---------------------------
| preempt count: 00010005 ]
| 5-level deep critical section nesting:
----------------------------------------
.. [<c0114783>] .... flush_tlb_mm+0x23/0x110
.....[<c015965e>] .. ( <= unmap_vmas+0x10e/0x1c0)
.. [<c0326aff>] .... _spin_lock_irqsave+0x1f/0x80
.....[<c0123ecb>] .. ( <= vprintk+0x2b/0x170)
.. [<c0326a8f>] .... _spin_lock+0x1f/0x70
.....[<c01e24ea>] .. ( <= _down_write_trylock+0x23a/0x290)
.. [<c0326a8f>] .... _spin_lock+0x1f/0x70
.....[<c0109b22>] .. ( <= die_nmi+0x22/0xa0)
.. [<c013d7dd>] .... print_traces+0x1d/0x60
.....[<c0108a1f>] .. ( <= show_stack+0x8f/0xb0)
Code: b8 00 e0 ff ff 89 fe c1 ee 09 21 e0 89 45 ec 83 f6 01 8b 40 10 83 e6
01
69 c0 80 77 01 00 8d 98 00 da 42 c0 f0 ff 80 00 da 42 c0 <8b> 80 00 da 42
c0 48
74 0d f0 ff 0b 57 9d 83 c4 24 5b 5e 5f 5d
console shuts up ...
Ingo Molnar wrote:
> * Thomas Gleixner <[email protected]> wrote:
>
>
>>>Thomas, can you confirm that this kernel fixes the irqs-off latencies?
>>>(the priority loop indeed was done with irqs turned off.)
>>
>>The latencies are still there. I have the feeling it's worse than 0.6.2.
>
>
> update to others: Thomas debugged this problem today and found the place
> that kept irqs disabled for a long time: it was update_process_times().
> (which i recently touched to break latencies there - but forgot that the
> lock is an irqs-off lock!)
>
> i've uploaded a fixed kernel (-V0.6.8) to:
>
> http://redhat.com/~mingo/realtime-preempt/
>
> (this kernel also has the module-put-unlock-kernel fix that should solve
> the other warning reported by Thomas and Bill.)
>
> Ingo
>
This one initially booted fine on my SMP workstation at the office. Ran
for about 1 hr. 10 mins. then locked with no indications as to why. Then
failed reboot after hitting the reset switch. The last thing that I saw
on the console seems to match the following that I found in the log:
Nov 2 11:21:49 swdev14 rc: Starting readahead: succeeded
Nov 2 11:21:50 swdev14 messagebus: messagebus startup succeeded
Nov 2 11:21:50 swdev14 rhnsd[3245]: Red Hat Network Services Daemon
starting up
.
Nov 2 11:21:50 swdev14 rhnsd: rhnsd startup succeeded
Nov 2 11:21:51 swdev14 kernel: (rc/2112/CPU#2): new 912 us
maximum-latency wake
up.
Nov 2 11:21:51 swdev14 kernel: (ksoftirqd/1/6/CPU#1): new 3147 us
maximum-laten
cy wakeup.
Nov 2 11:21:51 swdev14 kernel: (mingetty/3251/CPU#1): new 3916 us
maximum-laten
cy wakeup.
Nov 2 11:21:51 swdev14 kernel: (init/3253/CPU#2): new 4321 us
maximum-latency w
akeup.
Nov 2 11:21:51 swdev14 kernel: (init/1/CPU#0): new 5332 us
maximum-latency wake
up.
Nov 2 11:21:51 swdev14 kernel: (init/1/CPU#0): new 5819 us
maximum-latency wake
up.
Nov 2 11:21:51 swdev14 kernel: (hotplug/3259/CPU#2): new 6847 us
maximum-latenc
y wakeup.
Nov 2 11:21:51 swdev14 kernel: (hotplug/3274/CPU#2): new 7378 us
maximum-latenc
y wakeup.
Nov 2 11:31:12 swdev14 syslogd 1.4.1: restart.
After the failed reboot a subsequent reboot went fine. This behavior
seems to be pretty much the same for a while now.
My SMP system at home has been running for for 1 hr. 23 mins. so far. No
signs of problems.
kr
>i've uploaded a fixed kernel (-V0.6.8) to:
>
> http://redhat.com/~mingo/realtime-preempt/
This build appears to run OK and then in the middle of the real time
tests stops doing useful work (during network test).
- can move mouse & switch displays
- script checking latency hangs (no errors, apparently stuck in sleep)
- ps is OK, but top does not work (no display, Ctrl-C gives prompt again)
- cannot ping the test system from another system (don't get the "no route
message, just no response)
- Alt-SysRq keys still work
- eventually the mouse would not switch displays
Rebooted with Alt-SysRq-B.
Sending serial console results separately.
--Mark
* [email protected] <[email protected]> wrote:
> This build appears to run OK and then in the middle of the real time
> tests stops doing useful work (during network test).
weird, the deadlock detector did not trigger, although it is a clear
circular deadlock:
R ksoftirqd/0/ 3 [dffe8020, 105] blocked on: [dfcafc20] {r:0,a:-1,&((sk)->sk_lock.slock)}
.. held by: rcp/ 4791 [d84bd910, 118]
... acquired at: ip_send_reply+0x12b/0x250
D rcp/ 4791 [d84bd910, 118] blocked on: [c03cb900] {r:0,a:-1,ptype_lock}
.. held by: ksoftirqd/0/ 3 [dffe8020, 105]
... acquired at: netif_receive_skb+0xae/0x330
i.e. ksoftirqd blocked on rcp and rcp blocked on ksoftirqd, cleanly
deadlocking each other.
anyway, does the patch below, ontop of -V0.6.8, fix this type of hang?
Ingo
--- linux/net/core/dev.c.orig
+++ linux/net/core/dev.c
@@ -1916,7 +1916,9 @@ static void net_rx_action(struct softirq
dev = list_entry(queue->poll_list.next,
struct net_device, poll_list);
+ rcu_read_lock_read(&ptype_lock);
if (dev->quota <= 0 || dev->poll(dev, &budget)) {
+ rcu_read_unlock_read(&ptype_lock);
local_irq_disable();
list_del(&dev->poll_list);
list_add_tail(&dev->poll_list, &queue->poll_list);
@@ -1926,6 +1928,7 @@ static void net_rx_action(struct softirq
dev->quota = dev->weight;
} else {
dev_put(dev);
+ rcu_read_unlock_read(&ptype_lock);
local_irq_disable();
}
Hello Ingo,
Ingo Molnar wrote:
> i've uploaded a fixed kernel (-V0.6.8) to:
>
> http://redhat.com/~mingo/realtime-preempt/
Doesn't compile:
CC [M] fs/lockd/svc.o
fs/lockd/svc.c:49: warning: type defaults to `int' in declaration of
`DECLARE_MUTEX_NOCHECK'
fs/lockd/svc.c:49: warning: parameter names (without types) in function
declaration
fs/lockd/svc.c: In function `lockd':
fs/lockd/svc.c:112: error: `lockd_start' undeclared (first use in this
function)
fs/lockd/svc.c:112: error: (Each undeclared identifier is reported only once
fs/lockd/svc.c:112: error: for each function it appears in.)
fs/lockd/svc.c: In function `lockd_up':
fs/lockd/svc.c:261: error: `lockd_start' undeclared (first use in this
function)
fs/lockd/svc.c: In function `init_nlm':
fs/lockd/svc.c:426: error: `lockd_start' undeclared (first use in this
function)
fs/lockd/svc.c: At top level:
fs/lockd/svc.c:49: warning: 'DECLARE_MUTEX_NOCHECK' declared `static' but
never defined
make[2]: *** [fs/lockd/svc.o] Error 1
make[1]: *** [fs/lockd] Error 2
make: *** [fs] Error 2
Regards,
Norberto
* Ingo Molnar <[email protected]> wrote:
> > This build appears to run OK and then in the middle of the real time
> > tests stops doing useful work (during network test).
>
> weird, the deadlock detector did not trigger, although it is a clear
> circular deadlock:
ah ... found it - a fair portion of spinlocks and rwlocks had deadlock
detection turned off in -V0.6.6 - amongst them ptype_lock. I've uploaded
-V0.6.9 that fixes this.
Ingo
* Norberto Bensa <[email protected]> wrote:
> Ingo Molnar wrote:
> > i've uploaded a fixed kernel (-V0.6.8) to:
> >
> > http://redhat.com/~mingo/realtime-preempt/
>
> Doesn't compile:
>
> CC [M] fs/lockd/svc.o
> fs/lockd/svc.c:49: warning: type defaults to `int' in declaration of
> `DECLARE_MUTEX_NOCHECK'
ok - the reason was this:
> # CONFIG_PREEMPT_REALTIME is not set
i fixed these build problems in the -V0.6.9 patch i just uploaded.
Ingo
* K.R. Foley <[email protected]> wrote:
> This one initially booted fine on my SMP workstation at the office.
> Ran for about 1 hr. 10 mins. then locked with no indications as to
> why. [...]
soft hang or hard hang? In any case, the freshly uploaded -V0.6.9 kernel
both fixes one more deadlock and extends deadlock-detection to virtually
every locking object in Linux so it would be nice to check whether you
are getting a silent hang again, or perhaps something more verbose.
Ingo
* [email protected] <[email protected]> wrote:
> (cat/3504/CPU#0): new 5286 us maximum-latency wakeup.
> (cat/3504/CPU#0): new 71828 us maximum-latency wakeup.
ok - wakeup-latency timing is completely unreliable on SMP and will
produce bogus results.
Ingo
Ingo Molnar wrote:
> * K.R. Foley <[email protected]> wrote:
>
>
>>This one initially booted fine on my SMP workstation at the office.
>>Ran for about 1 hr. 10 mins. then locked with no indications as to
>>why. [...]
>
>
> soft hang or hard hang? In any case, the freshly uploaded -V0.6.9 kernel
> both fixes one more deadlock and extends deadlock-detection to virtually
> every locking object in Linux so it would be nice to check whether you
> are getting a silent hang again, or perhaps something more verbose.
>
> Ingo
>
Doh! Sorry. Hard hang. Building -V0.6.9 already.
kr
* [email protected] <[email protected]> wrote:
> NMI Watchdog detected LOCKUP
> Pid: 3933, comm: cpu_burn
> EIP: 0073:[<08048340>] CPU: 0
> EIP is at 0x8048340
> ESP: 007b:bffffa40 EFLAGS: 00200282 Not tainted
> (2.6.9-mm1-RT-V0.6.7)
> EAX: 00000000 EBX: 00711ffc ECX: bffffadc EDX: bffffad4
> ESI: 00000001 EDI: 007140fc EBP: bffffa48 DS: 007b ES: 007b
> CR0: 8005003b CR2: 00681400 CR3: 015735e0 CR4: 000006f0
> [<c0105bec>] show_regs+0x14c/0x174 (36)
> [<c0115f4f>] nmi_watchdog_tick+0x12f/0x140 (28)
> [<c0109c0c>] default_do_nmi+0x6c/0x110 (96)
> [<c0109d2d>] do_nmi+0x6d/0x70 (24)
> [<c0108735>] nmi_stack_correct+0x1e/0x2e (-196314476)
hm, this one is an extremely weird deadlock - the NMI watchdog detected
a _user-space_ deadlock - i.e. the "cpu_burn" user-space code disabled
interrupts for more than ~5 seconds? Sounds quite unlikely and the
EFLAGS register also directly contradicts it, it has 0x200 set so
interrupts are enabled! The only other way for the NMI watchdog to
trigger is if for whatever reason the local APIC timer interrupts are
not getting through and the NMI ticks (which come via a different
interrupt pin) get through.
this is what's happening on the other CPU:
> [<c0109b5f>] die_nmi+0x5f/0xa0 (24)
> [<c0115f0f>] nmi_watchdog_tick+0xef/0x140 (28)
> [<c0109c0c>] default_do_nmi+0x6c/0x110 (96)
> [<c0109d2d>] do_nmi+0x6d/0x70 (24)
> [<c0108735>] nmi_stack_correct+0x1e/0x2e (116)
> [<c013bfed>] __mcount+0x1d/0x30 (16)
> [<c0114fd8>] mcount+0x14/0x18 (20)
> [<c0326a81>] _spin_lock+0x11/0x70 (20)
> [<c01e2308>] _down_write_trylock+0x58/0x290 (52)
> [<c01e3325>] down_trylock+0x45/0x180 (52)
> [<c0123f95>] vprintk+0xf5/0x170 (36)
> [<c0123e8d>] printk+0x1d/0x30 (16)
> [<c0108945>] show_trace+0x95/0xe0 (32)
> [<c0108a63>] dump_stack+0x23/0x30 (20)
> [<c013c98e>] check_preempt_timing+0x16e/0x300 (76)
> [<c013ce7f>] sub_preempt_count+0x7f/0xf0 (32)
> [<c01147ba>] flush_tlb_mm+0x5a/0x110 (36)
flush_tlb_mm sends an IPI to the other CPU - maybe there's a connection.
Ingo
>hm, this one is an extremely weird deadlock - the NMI watchdog detected
>a _user-space_ deadlock - i.e. the "cpu_burn" user-space code disabled
>interrupts for more than ~5 seconds? Sounds quite unlikely and the
>EFLAGS register also directly contradicts it, it has 0x200 set so
>interrupts are enabled!
Very unlikely - cpu_burn.c (all of it...)
int main() {
while (1) { }
return 1;
}
No - it did not disable interrupts.
>The only other way for the NMI watchdog to
>trigger is if for whatever reason the local APIC timer interrupts are
>not getting through and the NMI ticks (which come via a different
>interrupt pin) get through.
The other symptoms I was seeing appeared to be timer related
- never returned from "sleep 1s" in the shell script
- "ps -fe" worked fine, but "top" did not
You may be on to something here.
--Mark
* [email protected] <[email protected]> wrote:
> >The only other way for the NMI watchdog to
> >trigger is if for whatever reason the local APIC timer interrupts are
> >not getting through and the NMI ticks (which come via a different
> >interrupt pin) get through.
>
> The other symptoms I was seeing appeared to be timer related
> - never returned from "sleep 1s" in the shell script
> - "ps -fe" worked fine, but "top" did not
> You may be on to something here.
was this already in the default bootup, or only after the crash
happened? Does this also happen if you chrt ksoftirqd to FIFO prio 1?
Does the 'LOC' count increase for both cpus in /proc/interrupts?
Ingo
Ingo Molnar wrote:
>i've uploaded a fixed kernel (-V0.6.8) to :
>
>http://redhat.com/~mingo/realtime-preempt/ <http://redhat.com/%7Emingo/realtime-preempt/>
>
>
>
I have thousands of the following BUG with V0.6.9
Nov 2 23:10:14 tigre02 kernel: printk: 6553 messages suppressed.
Nov 2 23:10:14 tigre02 kernel: BUG: using smp_processor_id() in
preemptible [00000001] code: udev/1159
Nov 2 23:10:14 tigre02 kernel: caller is store_stackinfo+0x5d/0xb0
Nov 2 23:10:14 tigre02 kernel: [<c0106503>] dump_stack+0x23/0x30 (20)
Nov 2 23:10:14 tigre02 kernel: [<c011f177>] smp_processor_id+0xb7/0xc0 (32)
Nov 2 23:10:14 tigre02 kernel: [<c0159a5d>] store_stackinfo+0x5d/0xb0 (28)
Nov 2 23:10:14 tigre02 kernel: [<c015b6ba>]
cache_free_debugcheck+0x1aa/0x390 (52)
Nov 2 23:10:14 tigre02 kernel: [<c015c656>] kmem_cache_free+0x46/0xf0 (32)
Nov 2 23:10:14 tigre02 kernel: [<c01731f1>] sys_open+0x81/0xb0 (32)
Nov 2 23:10:14 tigre02 kernel: [<c0105561>] sysenter_past_esp+0x52/0x71
(-8124)
Nov 2 23:10:14 tigre02 kernel: ---------------------------
Nov 2 23:10:14 tigre02 kernel: | preempt count: 00000002 ]
Nov 2 23:10:14 tigre02 kernel: | 2-level deep critical section nesting:
Nov 2 23:10:14 tigre02 kernel: ----------------------------------------
Nov 2 23:10:14 tigre02 kernel: .. [<c011f11f>] ....
smp_processor_id+0x5f/0xc0
Nov 2 23:10:14 tigre02 kernel: .....[<c0159a5d>] .. ( <=
store_stackinfo+0x5d/0xb0)
Nov 2 23:10:14 tigre02 kernel: .. [<c014083d>] .... print_traces+0x1d/0x60
Nov 2 23:10:14 tigre02 kernel: .....[<c0106503>] .. ( <=
dump_stack+0x23/0x30)
Nov 2 23:10:14 tigre02 kernel:
Nov 2 23:10:14 tigre02 kernel: printk: 6548 messages suppressed.
Nov 2 23:10:14 tigre02 kernel: BUG: using smp_processor_id() in
preemptible [00000001] code: udev/1299
Nov 2 23:10:14 tigre02 kernel: caller is store_stackinfo+0x5d/0xb0
Nov 2 23:10:14 tigre02 kernel: [<c0106503>] dump_stack+0x23/0x30 (20)
Nov 2 23:10:14 tigre02 kernel: [<c011f177>] smp_processor_id+0xb7/0xc0 (32)
Nov 2 23:10:14 tigre02 kernel: [<c0159a5d>] store_stackinfo+0x5d/0xb0 (28)
Nov 2 23:10:14 tigre02 kernel: [<c015b6ba>]
cache_free_debugcheck+0x1aa/0x390 (52)
Nov 2 23:10:14 tigre02 kernel: [<c015c656>] kmem_cache_free+0x46/0xf0 (32)
Nov 2 23:10:14 tigre02 kernel: [<c01ab892>] proc_destroy_inode+0x22/0x30
(16)
Nov 2 23:10:14 tigre02 kernel: [<c01908c1>] destroy_inode+0x41/0x90 (24)
Nov 2 23:10:14 tigre02 kernel: [<c01920cb>] iput+0x6b/0xb0 (28)
Nov 2 23:10:14 tigre02 kernel: [<c018e638>] dput+0x138/0x2d0 (32)
Nov 2 23:10:14 tigre02 kernel: [<c0183ed2>] link_path_walk+0x8c2/0xff0 (108)
Nov 2 23:10:14 tigre02 kernel: [<c018495c>] path_lookup+0xac/0x220 (36)
Nov 2 23:10:14 tigre02 kernel: [<c01852d2>] open_namei+0x92/0x6f0 (60)
Nov 2 23:10:14 tigre02 kernel: [<c0172cc6>] filp_open+0x46/0x70 (92)
Nov 2 23:10:14 tigre02 kernel: [<c01731bb>] sys_open+0x4b/0xb0 (32)
Nov 2 23:10:14 tigre02 kernel: [<c0105561>] sysenter_past_esp+0x52/0x71
(-8124)
Nov 2 23:10:14 tigre02 kernel: ---------------------------
Nov 2 23:10:14 tigre02 kernel: | preempt count: 00000002 ]
Nov 2 23:10:14 tigre02 kernel: | 2-level deep critical section nesting:
Nov 2 23:10:14 tigre02 kernel: ----------------------------------------
Nov 2 23:10:14 tigre02 kernel: .. [<c011f11f>] ....
smp_processor_id+0x5f/0xc0
Nov 2 23:10:14 tigre02 kernel: .....[<c0159a5d>] .. ( <=
store_stackinfo+0x5d/0xb0)
Nov 2 23:10:14 tigre02 kernel: .. [<c014083d>] .... print_traces+0x1d/0x60
Nov 2 23:10:14 tigre02 kernel: .....[<c0106503>] .. ( <=
dump_stack+0x23/0x30)
Nov 2 23:10:14 tigre02 kernel:
Nov 2 23:10:14 tigre02 kernel: Real Time Clock Driver v1.12
Nov 2 23:10:14 tigre02 kernel: thinkpad_acpi: ACPI IBM Thinkpad Fn+Fx
key driver version 1.1
Nov 2 23:10:14 tigre02 kernel: USB Universal Host Controller Interface
driver v2.2
Nov 2 23:10:14 tigre02 kernel: printk: 5080 messages suppressed.
Nov 2 23:10:14 tigre02 kernel: BUG: using smp_processor_id() in
preemptible [00000001] code: modprobe/1386
Nov 2 23:10:14 tigre02 kernel: caller is store_stackinfo+0x5d/0xb0
Nov 2 23:10:14 tigre02 kernel: [<c0106503>] dump_stack+0x23/0x30 (20)
Nov 2 23:10:14 tigre02 kernel: [<c011f177>] smp_processor_id+0xb7/0xc0 (32)
Nov 2 23:10:14 tigre02 kernel: [<c0159a5d>] store_stackinfo+0x5d/0xb0 (28)
Nov 2 23:10:14 tigre02 kernel: [<c015b6ba>]
cache_free_debugcheck+0x1aa/0x390 (52)
Nov 2 23:10:14 tigre02 kernel: [<c015c7e1>] kfree+0x71/0x120 (40)
Nov 2 23:10:14 tigre02 kernel: [<c02cf7c8>]
acpi_ut_free_and_track+0x71/0xda (40)
Nov 2 23:10:14 tigre02 kernel: [<c02c3222>]
acpi_ns_delete_children+0xe9/0x146 (56)
Nov 2 23:10:14 tigre02 kernel: [<c02c331a>]
acpi_ns_delete_namespace_subtree+0x9b/0xaf (44)
Nov 2 23:10:14 tigre02 kernel: [<c02af59e>]
acpi_ds_terminate_control_method+0x10d/0x1bb (40)
Nov 2 23:10:14 tigre02 kernel: [<c02c7a37>]
acpi_ps_parse_aml+0x1f5/0x2be (48)
Nov 2 23:10:14 tigre02 kernel: [<c02c87a7>] acpi_psx_execute+0x26f/0x2b4
(44)
Nov 2 23:10:14 tigre02 kernel: [<c02c38f7>]
acpi_ns_execute_control_method+0xb9/0xf6 (40)
Nov 2 23:10:14 tigre02 kernel: [<c02c380f>]
acpi_ns_evaluate_by_handle+0xe5/0x114 (40)
Hope this help
Remi
>Does this also happen if you chrt ksoftirqd to FIFO prio 1?
>Does the 'LOC' count increase for both cpus in /proc/interrupts?
With -V0.6.9, I got some slightly different symptoms. I did the change
chrt change (though I looked up the PID by hand, my attempt to use
pidof with 'ksoftirqd/0' did not work).
The LOC count did increase on both CPU's during my test (up until
I got the deadlock listed below). I also noticed that almost all
interrupts were processed by CPU0; should have checked before starting
the real time test to see if that was consistent when the system
was idle. I was about to send you a message from the test system with
that data when the system locked up.
The crash sequence was...
- boot to single user (uneventful)
- telnet 5 (uneventful)
- X and top tests (uneventful)
- network test started (and did not finish)
- 2343 usec latency dumped
- 55962 usec latency dumped
- 74229 usec latency dumped
- 83374 usec latency dumped
- deadlock
- long string of other BUG messages
I tried to Sync & reboot with Alt-SysRq keys at this point and was
unable to do so. Hard reset to reboot. Serial console output to
follow separately.
--Mark
Am Dienstag 02 November 2004 16:06 schrieb Ingo Molnar:
> i've uploaded a fixed kernel (-V0.6.8) to:
>
> http://redhat.com/~mingo/realtime-preempt/
>
> (this kernel also has the module-put-unlock-kernel fix that should solve
> the other warning reported by Thomas and Bill.)
>
> Ingo
Fixed a deadlock in snd-es1968 with attached patch.
Just for information, the error messages on V0.6.9 were:
===============================================
[ BUG: semaphore recursion deadlock detected! |
-----------------------------------------------
already locked: [c91ea080] {r:0,a:-1,&chip->reg_lock}
.. held by: insmod/ 3185 [cf6c2750, 118]
... acquired at: es1968_measure_clock+0x3b6/0x640 [snd_es1968]
------------------------------
| showing all locks held by: | (insmod/3185 [cf6c2750, 118]):
------------------------------
#001: [c036a4ec] {r:0,a:-1,&s->rwsem}
... acquired at: bus_add_driver+0x8a/0xc0
#002: [c91ea080] {r:0,a:-1,&chip->reg_lock}
... acquired at: es1968_measure_clock+0x3b6/0x640 [snd_es1968]
-{current task's backtrace}----------------->
[<c0107923>] dump_stack+0x23/0x30 (20)
[<c01bae36>] check_deadlock+0x276/0x2a0 (44)
[<c01bb455>] task_blocks_on_sem+0x195/0x1d0 (56)
[<c02de4d3>] down_write_mutex+0x283/0x3c0 (88)
[<c0134bc4>] __mutex_lock+0x44/0x60 (24)
[<c0134c8d>] _mutex_lock_irqsave+0x1d/0x30 (16)
[<d099266e>] snd_es1968_bob_start+0xbe/0x160 [snd_es1968] (44)
[<d0994a1a>] es1968_measure_clock+0x3ca/0x640 [snd_es1968] (72)
[<d0996318>] snd_es1968_probe+0x1f8/0x340 [snd_es1968] (60)
[<c01ca5b2>] pci_device_probe_static+0x52/0x70 (28)
[<c01ca60d>] __pci_device_probe+0x3d/0x50 (24)
[<c01ca651>] pci_device_probe+0x31/0x50 (24)
[<c021c1b5>] bus_match+0x45/0x80 (28)
[<c021c31c>] driver_attach+0x6c/0xb0 (36)
[<c021c802>] bus_add_driver+0x92/0xc0 (36)
[<c01ca969>] pci_register_driver+0x99/0xc0 (36)
[<d0861017>] alsa_card_es1968_init+0x17/0x1b [snd_es1968] (12)
[<c013ab29>] sys_init_module+0x169/0x220 (28)
[<c01072ed>] sysenter_past_esp+0x52/0x71 (-8124)
---------------------------
| preempt count: 00000002 ]
| 2-level deep critical section nesting:
----------------------------------------
.. [<c02de604>] .... down_write_mutex+0x3b4/0x3c0
.....[<c0134bc4>] .. ( <= __mutex_lock+0x44/0x60)
.. [<c01368dd>] .... print_traces+0x1d/0x60
.....[<c0107923>] .. ( <= dump_stack+0x23/0x30)
showing all tasks:
s init/ 1 [c1271370, 116] (not blocked)
s ksoftirqd/0/ 2 [c1270d00, 105] (not blocked)
s desched/0/ 3 [c1270690, 105] (not blocked)
s events/0/ 4 [c1270020, 98] (not blocked)
s khelper/ 5 [cff27390, 105] (not blocked)
s kthread/ 10 [cff26d20, 105] (not blocked)
s kacpid/ 18 [cff266b0, 105] (not blocked)
s IRQ 9/ 19 [cff26040, 50] (not blocked)
s kblockd/0/ 86 [cfec93b0, 105] (not blocked)
s khubd/ 94 [cfec8d40, 115] (not blocked)
s pdflush/ 141 [cfec86d0, 120] (not blocked)
s pdflush/ 142 [cfec8060, 116] (not blocked)
s aio/0/ 144 [c1304d60, 110] (not blocked)
s kswapd0/ 143 [c13053d0, 125] (not blocked)
s IRQ 8/ 732 [c13046f0, 56] (not blocked)
s IRQ 12/ 740 [c13cf3f0, 54] (not blocked)
s IRQ 6/ 753 [c13ced80, 51] (not blocked)
s kseriod/ 734 [c1304080, 124] (not blocked)
s IRQ 14/ 776 [c13ce710, 53] (not blocked)
s IRQ 15/ 778 [c13ce0a0, 52] (not blocked)
s IRQ 1/ 796 [c13fd410, 55] (not blocked)
s kjournald/ 838 [c13fc730, 116] (not blocked)
s portmap/ 2013 [c13fcda0, 116] (not blocked)
s sshd/ 2074 [ceac2970, 124] (not blocked)
s xfs/ 2343 [ceed03e0, 119] (not blocked)
s IRQ 5/ 2411 [cefa6220, 106] (not blocked)
s mingetty/ 2485 [ce88a8d0, 118] (not blocked)
s mingetty/ 2498 [cebfa5a0, 118] (not blocked)
s mingetty/ 2499 [ce6298b0, 118] (not blocked)
s mingetty/ 2500 [cebfb280, 118] (not blocked)
s mingetty/ 2501 [cea8afc0, 122] (not blocked)
s mingetty/ 2610 [cebfb8f0, 122] (not blocked)
s kdm/ 2611 [ce88b5b0, 117] (not blocked)
s X/ 2698 [ce628bd0, 115] (not blocked)
s kdm/ 2699 [cf6c2dc0, 121] (not blocked)
s IRQ 11/ 2792 [ceac2fe0, 57] (not blocked)
s startkde/ 2796 [ceac2300, 117] (not blocked)
s ssh-agent/ 2855 [cea8a950, 116] (not blocked)
s kdeinit/ 2889 [cddd8c30, 117] (not blocked)
s kdeinit/ 2892 [cefa7570, 116] (not blocked)
s kdeinit/ 2894 [cea8a2e0, 116] (not blocked)
s kdeinit/ 2897 [ce88af40, 115] (not blocked)
s kdeinit/ 2908 [ce88a260, 116] (not blocked)
s kdeinit/ 2982 [ceed10c0, 117] (not blocked)
s kwrapper/ 2985 [ceed1730, 116] (not blocked)
s kdeinit/ 2987 [ceed0a50, 115] (not blocked)
s kdeinit/ 2989 [cddd85c0, 116] (not blocked)
s kdeinit/ 2993 [ce629240, 116] (not blocked)
s kdeinit/ 2995 [cebfac10, 116] (not blocked)
s kdeinit/ 2997 [cefa6890, 116] (not blocked)
s kdeinit/ 3004 [c13fc0c0, 116] (not blocked)
s kdeinit/ 3007 [ce628560, 116] (not blocked)
s kdeinit/ 3009 [cddd9910, 116] (not blocked)
s kdeinit/ 3011 [cf6c20e0, 116] (not blocked)
s bash/ 3080 [cddd92a0, 117] (not blocked)
s su/ 3136 [cefa6f00, 119] (not blocked)
s bash/ 3139 [cea8b630, 118] (not blocked)
R insmod/ 3185 [cf6c2750, 118] (not blocked)
---------------------------
| showing all locks held: |
---------------------------
#001: [c04445c0] {r:0,a:-1,&hwif->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x9d/0x190
#002: [c04441b8] {r:0,a:-1,&drive->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x172/0x190
#003: [c04443a0] {r:0,a:-1,&drive->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x172/0x190
#004: [c0444c64] {r:0,a:-1,&hwif->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x9d/0x190
#005: [c044485c] {r:0,a:-1,&drive->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x172/0x190
#006: [c0444a44] {r:0,a:-1,&drive->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x172/0x190
#007: [c0445308] {r:0,a:-1,&hwif->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x9d/0x190
#008: [c0444f00] {r:0,a:-1,&drive->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x172/0x190
#009: [c04450e8] {r:0,a:-1,&drive->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x172/0x190
#010: [c04459ac] {r:0,a:-1,&hwif->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x9d/0x190
#011: [c04455a4] {r:0,a:-1,&drive->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x172/0x190
#012: [c044578c] {r:0,a:-1,&drive->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x172/0x190
#013: [c0446050] {r:0,a:-1,&hwif->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x9d/0x190
#014: [c0445c48] {r:0,a:-1,&drive->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x172/0x190
#015: [c0445e30] {r:0,a:-1,&drive->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x172/0x190
#016: [c04466f4] {r:0,a:-1,&hwif->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x9d/0x190
#017: [c04462ec] {r:0,a:-1,&drive->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x172/0x190
#018: [c04464d4] {r:0,a:-1,&drive->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x172/0x190
#019: [c0446d98] {r:0,a:-1,&hwif->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x9d/0x190
#020: [c0446990] {r:0,a:-1,&drive->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x172/0x190
#021: [c0446b78] {r:0,a:-1,&drive->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x172/0x190
#022: [c044743c] {r:0,a:-1,&hwif->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x9d/0x190
#023: [c0447034] {r:0,a:-1,&drive->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x172/0x190
#024: [c044721c] {r:0,a:-1,&drive->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x172/0x190
#025: [c0447ae0] {r:0,a:-1,&hwif->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x9d/0x190
#026: [c04476d8] {r:0,a:-1,&drive->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x172/0x190
#027: [c04478c0] {r:0,a:-1,&drive->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x172/0x190
#028: [c0448184] {r:0,a:-1,&hwif->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x9d/0x190
#029: [c0447d7c] {r:0,a:-1,&drive->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x172/0x190
#030: [c0447f64] {r:0,a:-1,&drive->gendev_rel_sem}
.. held by: init/ 1 [c1271370, 116]
... acquired at: init_hwif_data+0x172/0x190
#031: [cfc24d20] {r:0,a:-1,&tty->atomic_read}
.. held by: mingetty/ 2485 [ce88a8d0, 118]
... acquired at: read_chan+0x716/0x770
#032: [cf624d20] {r:0,a:-1,&tty->atomic_read}
.. held by: mingetty/ 2499 [ce6298b0, 118]
... acquired at: read_chan+0x716/0x770
#033: [cefe2d20] {r:0,a:-1,&tty->atomic_read}
.. held by: mingetty/ 2498 [cebfa5a0, 118]
... acquired at: read_chan+0x716/0x770
#034: [ce0bcd20] {r:0,a:-1,&tty->atomic_read}
.. held by: mingetty/ 2500 [cebfb280, 118]
... acquired at: read_chan+0x716/0x770
#035: [cfb26d20] {r:0,a:-1,&tty->atomic_read}
.. held by: mingetty/ 2610 [cebfb8f0, 122]
... acquired at: read_chan+0x716/0x770
#036: [cfb2bd20] {r:0,a:-1,&tty->atomic_read}
.. held by: mingetty/ 2501 [cea8afc0, 122]
... acquired at: read_chan+0x716/0x770
#037: [c036a4ec] {r:0,a:-1,&s->rwsem}
.. held by: insmod/ 3185 [cf6c2750, 118]
... acquired at: bus_add_driver+0x8a/0xc0
#038: [c91ea080] {r:0,a:-1,&chip->reg_lock}
.. held by: insmod/ 3185 [cf6c2750, 118]
... acquired at: es1968_measure_clock+0x3b6/0x640 [snd_es1968]
#039: [c032d360] {r:0,a:-1,tasklist_lock}
.. held by: insmod/ 3185 [cf6c2750, 118]
... acquired at: show_all_locks+0x30/0x130
=============================================
[ turning off deadlock detection. Please report this trace. ]
Karsten
Ingo Molnar wrote:
>
> ah ... found it - a fair portion of spinlocks and rwlocks had deadlock
> detection turned off in -V0.6.6 - amongst them ptype_lock. I've uploaded
> -V0.6.9 that fixes this.
>
OK. Already tested RT-V0.6.9 and things are really good, so far. Got it
under the jackd-R + 9*fluidsynth workload test, and the comparison to
2.6.9 (vanilla) is getting kind of humiliating :)
2.6.9 RT-V0.6.9
--------- ---------
XRUN Rate . . . . . . . . . : 415 0 /hour
Delay Rate (>spare time) . . : 493 0 /hour
Delay Rate (>1000 usecs) . . : 913 1 /hour
Delay Maximum . . . . . . . . : 6877 864 usecs
Cycle Maximum . . . . . . . . : 1440 1552 usecs
Average DSP Load. . . . . . . : 38.9 40.5 %
Average Interrupt Rate . . . : 1337 1338 /sec
Average Context-Switch Rate . : 7488 9048 /sec
As before, these stats were taken by running jackd -v -dalsa -dhw:0
-r44100 -p128 -n2 -S -P, loaded with 9 (nine) fluidsynth instances, on a
[email protected] laptop, against the onboard sound device (snd-ali5451).
The results were averaged for 12 consecutive runs of 5 minutes each.
On the RT kernel, the IRQ 5 handler thread, that serves the ali5451 sound
device, has been chrt'ed to SCHED_FIFO and to priority=60 (chrt -p -f 60
`pidof "IRQ 5"`). This time thought, I haven't touched the ksoftirqd/0
scheduling policy nor priority.
I am also rehearsing these same tests on my P4/SMT desktop. I'll post
those a bit later today.
As a personal comment, I've never, never seen so good figures in any other
kernel I've came across in the last couple of years. I hope this can only
go to the better ;)
See ya.
--
rncbc aka Rui Nuno Capela
[email protected]
On Tue, Nov 02, 2004 at 11:21:43PM -0000, Rui Nuno Capela wrote:
> As a personal comment, I've never, never seen so good figures in any other
> kernel I've came across in the last couple of years. I hope this can only
> go to the better ;)
Hard RT guarantees is the goal of this project afterall. It's absolutely
going to happen. You can bet on it.
bill
Remi Colinet wrote:
> Ingo Molnar wrote:
>
>> i've uploaded a fixed kernel (-V0.6.8) to :
>>
>> http://redhat.com/~mingo/realtime-preempt/
>> <http://redhat.com/%7Emingo/realtime-preempt/>
>>
Without DEBUG_PAGE_ALLOC, the kernel (V0.6.9) is fairly stable.
I can even feel the difference of latency under KDE, between the stock
kernel and V0.6.9. :-)
I like this patch.
Remi
> I have thousands of the following BUG with V0.6.9
>
> Nov 2 23:10:14 tigre02 kernel: printk: 6553 messages suppressed.
> Nov 2 23:10:14 tigre02 kernel: BUG: using smp_processor_id() in
> preemptible [00000001] code: udev/1159
> Nov 2 23:10:14 tigre02 kernel: caller is store_stackinfo+0x5d/0xb0
> Nov 2 23:10:14 tigre02 kernel: [<c0106503>] dump_stack+0x23/0x30 (20)
> Nov 2 23:10:14 tigre02 kernel: [<c011f177>] smp_processor_id+0xb7/0xc0
> (32)
> Nov 2 23:10:14 tigre02 kernel: [<c0159a5d>] store_stackinfo+0x5d/0xb0
> (28)
> Nov 2 23:10:14 tigre02 kernel: [<c015b6ba>]
> cache_free_debugcheck+0x1aa/0x390 (52)
> Nov 2 23:10:14 tigre02 kernel: [<c015c656>] kmem_cache_free+0x46/0xf0
> (32)
> Nov 2 23:10:14 tigre02 kernel: [<c01731f1>] sys_open+0x81/0xb0 (32)
> Nov 2 23:10:14 tigre02 kernel: [<c0105561>]
> sysenter_past_esp+0x52/0x71 (-8124)
> Nov 2 23:10:14 tigre02 kernel: ---------------------------
> Nov 2 23:10:14 tigre02 kernel: | preempt count: 00000002 ]
> Nov 2 23:10:14 tigre02 kernel: | 2-level deep critical section nesting:
> Nov 2 23:10:14 tigre02 kernel: ----------------------------------------
> Nov 2 23:10:14 tigre02 kernel: .. [<c011f11f>] ....
> smp_processor_id+0x5f/0xc0
> Nov 2 23:10:14 tigre02 kernel: .....[<c0159a5d>] .. ( <=
> store_stackinfo+0x5d/0xb0)
> Nov 2 23:10:14 tigre02 kernel: .. [<c014083d>] ....
> print_traces+0x1d/0x60
> Nov 2 23:10:14 tigre02 kernel: .....[<c0106503>] .. ( <=
> dump_stack+0x23/0x30)
Rui Nuno Capela wrote:
>
> I am also rehearsing these same tests on my P4/SMT desktop.
> I'll post those a bit later today.
>
So here they are. This machine is a P4 [email protected], HT enabled on a
Asus P4P800 mobo, Intel 82801EB onboard sound device (snd-intel8x0).
2.6.9smp RT-V0.6.9smp
------------ ------------
XRUN Rate . . . . . . . . . : 0 0 /hour
Delay Rate (>spare time) . . : 0 0 /hour
Delay Rate (>1000 usecs) . . : 0 0 /hour
Delay Maximum . . . . . . . . : 346 166 usecs
Cycle Maximum . . . . . . . . : 986 1028 usecs
Average DSP Load. . . . . . . : 25.0 25.7 %
Average Interrupt Rate . . . : 1717 1718 /sec
Average Context-Switch Rate . : 10082 15793 /sec
As you can see, the results here aren't so disparate as on my UP laptop.
I guess that I can stress this out and lower the period size on jackd -R,
to -p64. And even increase the workload with more client instances (from 9
to 18 ?) to let the RT show it's potential, if that makes sense at all.
Bye now.
--
rncbc aka Rui Nuno Capela
[email protected]
Ingo Molnar wrote:
> * Ingo Molnar <[email protected]> wrote:
>
>
>>>This build appears to run OK and then in the middle of the real time
>>>tests stops doing useful work (during network test).
>>
>>weird, the deadlock detector did not trigger, although it is a clear
>>circular deadlock:
>
>
> ah ... found it - a fair portion of spinlocks and rwlocks had deadlock
> detection turned off in -V0.6.6 - amongst them ptype_lock. I've uploaded
> -V0.6.9 that fixes this.
>
> Ingo
>
I probably shouldn't say this, but I have yet to have -V0.6.9 die on me.
And at least right now, the peppiness is back in the X interface, the
menus, etc.
kr
>was this already in the default bootup, or only after the crash
>happened?
Not quite a "crash", but definitely after I noticed the system no
longer doing any useful work. As I said before, it was as if anything
to do with timers was not working (e.g., sleep 1s).
>Does this also happen if you chrt ksoftirqd to FIFO prio 1?
>Does the 'LOC' count increase for both cpus in /proc/interrupts?
It will be an hour or so before I can try either of these,
I have a build in progress. I can check both of these with the new
build (or the old one if you want that too).
--Mark
* Karsten Wiese <[email protected]> wrote:
> Am Dienstag 02 November 2004 16:06 schrieb Ingo Molnar:
> > i've uploaded a fixed kernel (-V0.6.8) to:
> >
> > http://redhat.com/~mingo/realtime-preempt/
> >
> > (this kernel also has the module-put-unlock-kernel fix that should solve
> > the other warning reported by Thomas and Bill.)
> >
> > Ingo
>
> Fixed a deadlock in snd-es1968 with attached patch.
thanks. This is a (SMP-only) bug in the vanilla driver too, correct?
Ingo
* Rui Nuno Capela <[email protected]> wrote:
> Rui Nuno Capela wrote:
> >
> > I am also rehearsing these same tests on my P4/SMT desktop.
> > I'll post those a bit later today.
> >
>
> So here they are. This machine is a P4 [email protected], HT enabled on a
> Asus P4P800 mobo, Intel 82801EB onboard sound device (snd-intel8x0).
>
> 2.6.9smp RT-V0.6.9smp
> ------------ ------------
> XRUN Rate . . . . . . . . . : 0 0 /hour
> Delay Rate (>spare time) . . : 0 0 /hour
> Delay Rate (>1000 usecs) . . : 0 0 /hour
> Delay Maximum . . . . . . . . : 346 166 usecs
> Cycle Maximum . . . . . . . . : 986 1028 usecs
> Average DSP Load. . . . . . . : 25.0 25.7 %
> Average Interrupt Rate . . . : 1717 1718 /sec
> Average Context-Switch Rate . : 10082 15793 /sec
>
> As you can see, the results here aren't so disparate as on my UP
> laptop.
i suspect it has to do with the different load - the UP laptop had 40%
utilization, this box is 25% utilized.
> I guess that I can stress this out and lower the period size on jackd
> -R, to -p64. And even increase the workload with more client instances
> (from 9 to 18 ?) to let the RT show it's potential, if that makes
> sense at all.
yeah, it would really be nice to see how much it can take and to see the
point where the -RT kernel starts to suffer from xruns too.
Ingo
* Rui Nuno Capela <[email protected]> wrote:
> OK. Already tested RT-V0.6.9 and things are really good, so far. Got
> it under the jackd-R + 9*fluidsynth workload test, and the comparison
> to 2.6.9 (vanilla) is getting kind of humiliating :)
>
> 2.6.9 RT-V0.6.9
> --------- ---------
> XRUN Rate . . . . . . . . . : 415 0 /hour
> Delay Rate (>spare time) . . : 493 0 /hour
> Delay Rate (>1000 usecs) . . : 913 1 /hour
> Delay Maximum . . . . . . . . : 6877 864 usecs
> Cycle Maximum . . . . . . . . : 1440 1552 usecs
> Average DSP Load. . . . . . . : 38.9 40.5 %
> Average Interrupt Rate . . . : 1337 1338 /sec
> Average Context-Switch Rate . : 7488 9048 /sec
cool, good numbers. May i have a stats suggestion for future runs? Could
you monitor the system (kernel) CPU overhead and userspace overhead as
well?
> As before, these stats were taken by running jackd -v -dalsa -dhw:0
> -r44100 -p128 -n2 -S -P, loaded with 9 (nine) fluidsynth instances, on
> a [email protected] laptop, against the onboard sound device (snd-ali5451).
> The results were averaged for 12 consecutive runs of 5 minutes each.
>
> On the RT kernel, the IRQ 5 handler thread, that serves the ali5451
> sound device, has been chrt'ed to SCHED_FIFO and to priority=60 (chrt
> -p -f 60 `pidof "IRQ 5"`). This time thought, I haven't touched the
> ksoftirqd/0 scheduling policy nor priority.
>
> I am also rehearsing these same tests on my P4/SMT desktop. I'll post
> those a bit later today.
>
> As a personal comment, I've never, never seen so good figures in any
> other kernel I've came across in the last couple of years. I hope this
> can only go to the better ;)
i hope so too :) There's no reason why the performance of the current
-RT kernel couldnt be maintained, there are no corners cut.
Ingo
On Tue, 2004-11-02 at 11:27, Ingo Molnar wrote:
> * Ingo Molnar <[email protected]> wrote:
> > > This build appears to run OK and then in the middle of the real time
> > > tests stops doing useful work (during network test).
> >
> > weird, the deadlock detector did not trigger, although it is a clear
> > circular deadlock:
>
> ah ... found it - a fair portion of spinlocks and rwlocks had deadlock
> detection turned off in -V0.6.6 - amongst them ptype_lock. I've uploaded
> -V0.6.9 that fixes this.
This happens with V0.6.9 running on an Athlon64:
r8169 Gigabit Ethernet driver 1.6LK loaded
ACPI: PCI interrupt 0000:00:0b.0[A] -> GSI 16 (level, low) -> IRQ 16
divert: allocating divert_blk for eth0
eth0: Identified chip type is 'RTL8169s/8110s'.
eth0: RTL8169 at 0xf88a0f00, 00:0c:76:b3:c2:43, IRQ 16
BUG: atomic counter underflow at:
[<c0108153>] dump_stack+0x23/0x30 (20)
[<c02f5e91>] qdisc_destroy+0xe1/0xf0 (28)
[<c02f60ad>] dev_shutdown+0x3d/0xa0 (28)
[<c02e6ffb>] unregister_netdevice+0x12b/0x2a0 (36)
[<c0273ffe>] unregister_netdev+0x1e/0x30 (16)
[<f8b0b8b5>] rtl8169_remove_one+0x25/0x50 [r8169] (32)
[<c01f55a6>] pci_device_remove+0x76/0x80 (24)
[<c025f15d>] device_release_driver+0x6d/0x70 (24)
[<c025f18b>] driver_detach+0x2b/0x40 (20)
[<c025f5ff>] bus_remove_driver+0x3f/0x70 (20)
[<c025fb0c>] driver_unregister+0x1c/0x30 (16)
[<c01f586c>] pci_unregister_driver+0x1c/0x30 (16)
[<f8b0d357>] rtl8169_cleanup_module+0x17/0x1b [r8169] (12)
[<c013fdb1>] sys_delete_module+0x121/0x150 (96)
[<c010729d>] sysenter_past_esp+0x52/0x71 (-8124)
---------------------------
| preempt count: 00000001 ]
| 1-level deep critical section nesting:
----------------------------------------
.. [<c013dc3d>] .... print_traces+0x1d/0x60
.....[<c0108153>] .. ( <= dump_stack+0x23/0x30)
divert: freeing divert_blk for eth0
ip_tables: (C) 2000-2002 Netfilter core team
-- Fernando
Fernando Pablo Lopez-Lezcano wrote:
> On Tue, 2004-11-02 at 11:27, Ingo Molnar wrote:
>
>>* Ingo Molnar <[email protected]> wrote:
>>
>>>>This build appears to run OK and then in the middle of the real time
>>>>tests stops doing useful work (during network test).
>>>
>>>weird, the deadlock detector did not trigger, although it is a clear
>>>circular deadlock:
>>
>>ah ... found it - a fair portion of spinlocks and rwlocks had deadlock
>>detection turned off in -V0.6.6 - amongst them ptype_lock. I've uploaded
>>-V0.6.9 that fixes this.
>
>
> This happens with V0.6.9 running on an Athlon64:
>
> r8169 Gigabit Ethernet driver 1.6LK loaded
> ACPI: PCI interrupt 0000:00:0b.0[A] -> GSI 16 (level, low) -> IRQ 16
> divert: allocating divert_blk for eth0
> eth0: Identified chip type is 'RTL8169s/8110s'.
> eth0: RTL8169 at 0xf88a0f00, 00:0c:76:b3:c2:43, IRQ 16
> BUG: atomic counter underflow at:
> [<c0108153>] dump_stack+0x23/0x30 (20)
> [<c02f5e91>] qdisc_destroy+0xe1/0xf0 (28)
> [<c02f60ad>] dev_shutdown+0x3d/0xa0 (28)
> [<c02e6ffb>] unregister_netdevice+0x12b/0x2a0 (36)
> [<c0273ffe>] unregister_netdev+0x1e/0x30 (16)
> [<f8b0b8b5>] rtl8169_remove_one+0x25/0x50 [r8169] (32)
> [<c01f55a6>] pci_device_remove+0x76/0x80 (24)
> [<c025f15d>] device_release_driver+0x6d/0x70 (24)
> [<c025f18b>] driver_detach+0x2b/0x40 (20)
> [<c025f5ff>] bus_remove_driver+0x3f/0x70 (20)
> [<c025fb0c>] driver_unregister+0x1c/0x30 (16)
> [<c01f586c>] pci_unregister_driver+0x1c/0x30 (16)
> [<f8b0d357>] rtl8169_cleanup_module+0x17/0x1b [r8169] (12)
> [<c013fdb1>] sys_delete_module+0x121/0x150 (96)
> [<c010729d>] sysenter_past_esp+0x52/0x71 (-8124)
> ---------------------------
> | preempt count: 00000001 ]
> | 1-level deep critical section nesting:
> ----------------------------------------
> .. [<c013dc3d>] .... print_traces+0x1d/0x60
> .....[<c0108153>] .. ( <= dump_stack+0x23/0x30)
>
> divert: freeing divert_blk for eth0
> ip_tables: (C) 2000-2002 Netfilter core team
>
> -- Fernando
>
>
>
Several of us get this. I believe this bug exists in upstream kernels
also, it just doesn't get caught without the detection Ingo has added.
kr
* [email protected] <[email protected]> wrote:
> The crash sequence was...
>
> - boot to single user (uneventful)
> - telnet 5 (uneventful)
> - X and top tests (uneventful)
> - network test started (and did not finish)
> - 2343 usec latency dumped
> - 55962 usec latency dumped
> - 74229 usec latency dumped
> - 83374 usec latency dumped
> - deadlock
yeah, this is yet another networking deadlock, nicely detected and
logged. Since the deadlock locks up ksoftirqd, timer handling (also
driven by ksoftirqd) wont work - i think this explains the followup
symptoms you got.
Ingo
* Ingo Molnar <[email protected]> wrote:
> yeah, this is yet another networking deadlock, nicely detected and
> logged. Since the deadlock locks up ksoftirqd, timer handling (also
> driven by ksoftirqd) wont work - i think this explains the followup
> symptoms you got.
the patch below should fix this deadlock but there might be others
around ...
Ingo
--- linux/include/net/sock.h.orig2
+++ linux/include/net/sock.h
@@ -706,8 +706,8 @@ extern void FASTCALL(lock_sock(struct so
extern void FASTCALL(release_sock(struct sock *sk));
/* BH context may only use the following locking interface. */
-#define bh_lock_sock(__sk) spin_lock(&((__sk)->sk_lock.slock))
-#define bh_unlock_sock(__sk) spin_unlock(&((__sk)->sk_lock.slock))
+#define bh_lock_sock(__sk) do { rcu_read_lock_read(&ptype_lock); spin_lock(&((__sk)->sk_lock.slock)); } while (0)
+#define bh_unlock_sock(__sk) do { spin_unlock(&((__sk)->sk_lock.slock)); rcu_read_unlock_read(&ptype_lock); } while (0)
extern struct sock * sk_alloc(int family, int priority, int zero_it,
kmem_cache_t *slab);
* Ingo Molnar <[email protected]> wrote:
> > yeah, this is yet another networking deadlock, nicely detected and
> > logged. Since the deadlock locks up ksoftirqd, timer handling (also
> > driven by ksoftirqd) wont work - i think this explains the followup
> > symptoms you got.
>
> the patch below should fix this deadlock but there might be others
> around ...
the patch doesnt work. Working on a better solution.
Ingo
Am Mittwoch 03 November 2004 02:12 schrieb Ingo Molnar:
>
> * Karsten Wiese <[email protected]> wrote:
>
> > Am Dienstag 02 November 2004 16:06 schrieb Ingo Molnar:
> > > i've uploaded a fixed kernel (-V0.6.8) to:
> > >
> > > http://redhat.com/~mingo/realtime-preempt/
> > >
> > > (this kernel also has the module-put-unlock-kernel fix that should solve
> > > the other warning reported by Thomas and Bill.)
> > >
> > > Ingo
> >
> > Fixed a deadlock in snd-es1968 with attached patch.
>
> thanks. This is a (SMP-only) bug in the vanilla driver too, correct?
Yes. I sent the patch to alsa-devel too.
Karsten
> > the patch below should fix this deadlock but there might be others
> > around ...
>
> the patch doesnt work. Working on a better solution.
this hopefully better solution is included in -V0.7.1.
Ingo
On Wed, 3 Nov 2004 13:39:35 +0100
Ingo Molnar <[email protected]> wrote:
>
> > > the patch below should fix this deadlock but there might be others
> > > around ...
> >
> > the patch doesnt work. Working on a better solution.
>
> this hopefully better solution is included in -V0.7.1.
Hi,
i tried V0.7.7 and since all other RP kernels locked for me in X, i decided
to go to the console and try to see if i get some debugging output. Well i
put the usual stress on the system but besides some higher rtc_wakeup
jitters [up to 60%] which didn't show during the earlier runs that locked
when in X, the system ran fine. i wasn't able to lock the system from within
the console (X was started but idling at gdm).
Ok, so i suspected, maybe the locks i see got to do with X, so i changed to
the X console, logged in and quicklky switched back to the console. and no
surprise 10seconds later or so it locked (i suppose gnome was still loading
at that point in time). Without any console output. Does it matter which
virtual text console one is on? i thought not.
anyways. i might just get myself a serial console in a few days (old pc
froma buddy). but i thought a serial console would be mostly useful to
capture more of the debug messages. if none shows at all it won't help
either right?
flo
On Wed, 3 Nov 2004, Florian Schmidt wrote:
> i tried V0.7.7 ...
I can't use 0.7.7. I have the known problem with ide+lvm. Backing out
dio-handle-eof.patch fixes it for me, at least on 2.6.9-mm1. Haven't tried
against 2.6.10-rc1-mm2(which is still broken).
Ingo Molnar <[email protected]> writes:
> if it's possible to 'silently' overrun the next due interrupt (somewhat,
> but not large enough overrun to cause a hard ALSA xrun) then the
> processing delay will i believe be accounted as a 'wakeup delay'. In
> that case to make the delayed_usecs value truly accurate, i'd at least
> add this:
>
> poll_enter = jack_get_microseconds ();
>
> if (poll_enter > driver->poll_next) {
> /*
> * This processing cycle got delayed over
> * the next due interrupt! Do not account this
> * as a wakeup delay:
> */
> driver->poll_next = 0;
> }
>
> but i'd also suggest to put in a counter into that branch so that this
> condition doesnt get lost.
Added the test Ingo suggests plus a new counter (poll_late) to CVS,
JACK version 0.99.13.
--
joq