I recently noted that my system was spending a lot of time in i/o wait
when doing some tasks which I thought didn't involve i/o, as noted by
the lack of disk light activity most of the time. I thought of network,
certainly the NIC had no activity for this job. So I set up a little
loop to capture all disk i/o and network activity (including loopback).
That was no obvious help, and the program doesn't use pipes.
At this point I'm really curious, does someone have a good clue?
Note: I don't think this is a bug or performance issue, unless the
kernel is doing something and charging time to iowait instead of system
I don't see anything to fix, but I would like to understand.
--
bill davidsen <[email protected]>
CTO TMR Associates, Inc
Doing interesting things with small computers since 1979
On 28/05/07, Bill Davidsen <[email protected]> wrote:
> I recently noted that my system was spending a lot of time in i/o wait
> when doing some tasks which I thought didn't involve i/o, as noted by
> the lack of disk light activity most of the time. I thought of network,
> certainly the NIC had no activity for this job. So I set up a little
> loop to capture all disk i/o and network activity (including loopback).
> That was no obvious help, and the program doesn't use pipes.
>
> At this point I'm really curious, does someone have a good clue?
>
> Note: I don't think this is a bug or performance issue, unless the
> kernel is doing something and charging time to iowait instead of system
> I don't see anything to fix, but I would like to understand.
>
Nothing /should/ cause time to be attributed to i/o wait other than
waiting for i/o. But the accounting isn't completely accurate. When
the kernel records what the has happen (governed by when the timer
interrupt triggers as far as I remember), many different processes may
have run and done may different things, but only what the last one did
is recorded. So if you are unlucky you may have the system doing all
sorts of activities, but only recording i/o wait if the last process
to run before the kernel looked was doing i/o wait... The numbers are
not exactely accurate...
--
Jesper Juhl <[email protected]>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html
Hi Bill,
On 5/29/07, Bill Davidsen <[email protected]> wrote:
> I recently noted that my system was spending a lot of time in i/o wait
> when doing some tasks which I thought didn't involve i/o, as noted by
> the lack of disk light activity most of the time. I thought of network,
> certainly the NIC had no activity for this job. So I set up a little
> loop to capture all disk i/o and network activity (including loopback).
> That was no obvious help, and the program doesn't use pipes.
>
> At this point I'm really curious, does someone have a good clue?
>
> Note: I don't think this is a bug or performance issue, unless the
> kernel is doing something and charging time to iowait instead of system
> I don't see anything to fix, but I would like to understand.
What tool / kernel instrumentation / mechanism are you using to
determine that some task(s) are indeed blocked waiting for i/o? Perhaps
some userspace process accounting tools could be "broken" in the sense
that they generalize all uninterruptible sleep as waiting for i/o ...
Satyam
Satyam Sharma wrote:
> Hi Bill,
>
> On 5/29/07, Bill Davidsen <[email protected]> wrote:
>> I recently noted that my system was spending a lot of time in i/o wait
>> when doing some tasks which I thought didn't involve i/o, as noted by
>> the lack of disk light activity most of the time. I thought of network,
>> certainly the NIC had no activity for this job. So I set up a little
>> loop to capture all disk i/o and network activity (including loopback).
>> That was no obvious help, and the program doesn't use pipes.
>>
>> At this point I'm really curious, does someone have a good clue?
>>
>> Note: I don't think this is a bug or performance issue, unless the
>> kernel is doing something and charging time to iowait instead of system
>> I don't see anything to fix, but I would like to understand.
>
> What tool / kernel instrumentation / mechanism are you using to
> determine that some task(s) are indeed blocked waiting for i/o? Perhaps
> some userspace process accounting tools could be "broken" in the sense
> that they generalize all uninterruptible sleep as waiting for i/o ...
I wouldn't expect /proc/stat and similar to be broken in that way, but
If no one has a better idea I guess I will assume there's a check needed
of where time is added to iowait. I was hoping to avoid a full kernel
search. Never thought of /proc data as a user space tool, but I guess.
--
Bill Davidsen <[email protected]>
"We have more to fear from the bungling of the incompetent than from
the machinations of the wicked." - from Slashdot
Bill Davidsen wrote:
> I recently noted that my system was spending a lot of time in i/o wait
> when doing some tasks which I thought didn't involve i/o, as noted by
> the lack of disk light activity most of the time. I thought of network,
> certainly the NIC had no activity for this job. So I set up a little
> loop to capture all disk i/o and network activity (including loopback).
> That was no obvious help, and the program doesn't use pipes.
>
> At this point I'm really curious, does someone have a good clue?
>
> Note: I don't think this is a bug or performance issue, unless the
> kernel is doing something and charging time to iowait instead of system
> I don't see anything to fix, but I would like to understand.
All filesystem IO and direct disk IO can cause iowait.
This includes NFS activity.
--
Politics is the struggle between those who want to make their country
the best in the world, and those who believe it already is. Each group
calls the other unpatriotic.
Rik van Riel wrote:
> Bill Davidsen wrote:
>> I recently noted that my system was spending a lot of time in i/o
>> wait when doing some tasks which I thought didn't involve i/o, as
>> noted by the lack of disk light activity most of the time. I thought
>> of network, certainly the NIC had no activity for this job. So I set
>> up a little loop to capture all disk i/o and network activity
>> (including loopback). That was no obvious help, and the program
>> doesn't use pipes.
>>
>> At this point I'm really curious, does someone have a good clue?
>>
>> Note: I don't think this is a bug or performance issue, unless the
>> kernel is doing something and charging time to iowait instead of
>> system I don't see anything to fix, but I would like to understand.
>
> All filesystem IO and direct disk IO can cause iowait.
>
> This includes NFS activity.
>
If I didn't note it before, I'm read the the data from /proc, cpustats,
net/dev, and diskstats. I assume that all i/o would show up in one of
those places. NFS isn't involved, although this machine is a fileserver
as a side job the modules weren't even loaded during testing.
A puzzlement for future consideration. If I get a chance later this week
I'll make a pretty graphic of all the stuff going on when the iowait
spiked, ctx rate, inq rate, hell the last time I even grabbed the CPU
temp to see if it told me anything (didn't, thermal throttling NOT).
Thanks for the feedback, I think that lets out the obvious stuff.
--
Bill Davidsen <[email protected]>
"We have more to fear from the bungling of the incompetent than from
the machinations of the wicked." - from Slashdot