Hi everyone. I'm seeing something very Odd with 2.6.11 / 2.6.12 and 2.6.13, it has me stumped so I want to describe my
problem here and hopefully some kind soul can verify that what I am seeing is a Feature and NotABug :-)
It it's a Feature, then I'll find some way to work around it, but it seems to me that it's a Bug :-)
I have a digital video stream coming in at 25Mb/s over a firewire connector, and this data is being written out to a
usb2 hard drive as it's captured. The hard drive can only sustain about 17Mb/s write speed, so I am in deficit by 7Mb
per second.
I'm using a Dell Inspiron laptop with a 2GHz p4 and 1.5Gb of physical RAM, running a FC3 based distro but with a number
of custom kernels, including 2.6.11, 2.6.12 and 2.6.13. I see the same issue on all of them.
Apart from X and my capture application (Coriander) there is nothing else happening on the machine, and so I see about
1.3Gb of physical ram is available for buffering this data.
Okay, so what I *expect* to see is this: The video will stream in and out of Coriander at 25Mb/s until all the free RAM
is full of dirty data waiting to be written to disk. At a deficit of 7Mb/s this should take about 3 minutes to fill
1.3Gb. When this point is reached I would see my write speed suddenly drop from 25Mb/s to 17Mb/s because there is
effectively no more RAM to allocate for buffers.
*But* what I actually see is that I hit this point after only 50 seconds. This is very puzzling. After 50 seconds of
operation I see my video write speed suddenly drop to 17Mb/s.
Coincidentally (or not...) this is about how long it takes the kernel to allocate all of the 1.3Gb of RAM to disk
buffers. If I watch "free" while I am running I can see the amount of allocated buffers growing by 25Mb/s until it is
all used, and then my write speed immediately drops.
It's as if the kernel isn't bothering to write out any of my data until it runs out of RAM.
Now, I'm pretty sure that the data is streaming out to the disk, cause I can see the disk activity light solidly lit on
the usb2 drive (very scientific). There is a short delay after I start streaming before it comes on, maybe 4 or 5
seconds. I've tinkered with the VM settings in /proc/sys/vm and I can make the kernel start writing data faster by
lowering dirty_ratio, but this does not change the symptoms that I am seeing.
When all 1.3Gb has been allocated to buffers after 50 seconds, 900M of that should have become free to be recycled as it
has been written out to the usb2 drive at 17Mb/s - Somehow this doesn't seem to be happening.
The result that I am seeing could be explained if the kernel was not releasing dirty pages after they have been written
to disk, but I know this doesn't make sense.
Can anyone make sense of this and let me know whether this is expected behaviour or not?
I'd prefer to read any responses on the list - if you email me directly then you'll have to get past my spam filter.
Thanks in advance,
Anthony
--
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd
[email protected]
Phone: (02) 62595404 or 0419409836
On Fri, 9 Sep 2005, Anthony Wesley wrote:
> Hi everyone. I'm seeing something very Odd with 2.6.11 / 2.6.12 and 2.6.13,
> it has me stumped so I want to describe my problem here and hopefully some
> kind soul can verify that what I am seeing is a Feature and NotABug :-)
>
> It it's a Feature, then I'll find some way to work around it, but it seems to
> me that it's a Bug :-)
>
> I have a digital video stream coming in at 25Mb/s over a firewire connector,
> and this data is being written out to a usb2 hard drive as it's captured. The
> hard drive can only sustain about 17Mb/s write speed, so I am in deficit by
> 7Mb per second.
>
> I'm using a Dell Inspiron laptop with a 2GHz p4 and 1.5Gb of physical RAM,
> running a FC3 based distro but with a number of custom kernels, including
> 2.6.11, 2.6.12 and 2.6.13. I see the same issue on all of them.
>
> Apart from X and my capture application (Coriander) there is nothing else
> happening on the machine, and so I see about 1.3Gb of physical ram is
> available for buffering this data.
>
> Okay, so what I *expect* to see is this: The video will stream in and out of
> Coriander at 25Mb/s until all the free RAM is full of dirty data waiting to
> be written to disk. At a deficit of 7Mb/s this should take about 3 minutes to
> fill 1.3Gb. When this point is reached I would see my write speed suddenly
> drop from 25Mb/s to 17Mb/s because there is effectively no more RAM to
> allocate for buffers.
>
> *But* what I actually see is that I hit this point after only 50 seconds.
> This is very puzzling. After 50 seconds of operation I see my video write
> speed suddenly drop to 17Mb/s.
>
> Coincidentally (or not...) this is about how long it takes the kernel to
> allocate all of the 1.3Gb of RAM to disk buffers. If I watch "free" while I
> am running I can see the amount of allocated buffers growing by 25Mb/s until
> it is all used, and then my write speed immediately drops.
>
> It's as if the kernel isn't bothering to write out any of my data until it
> runs out of RAM.
>
> Now, I'm pretty sure that the data is streaming out to the disk, cause I can
> see the disk activity light solidly lit on the usb2 drive (very scientific).
> There is a short delay after I start streaming before it comes on, maybe 4 or
> 5 seconds. I've tinkered with the VM settings in /proc/sys/vm and I can make
> the kernel start writing data faster by lowering dirty_ratio, but this does
> not change the symptoms that I am seeing.
>
> When all 1.3Gb has been allocated to buffers after 50 seconds, 900M of that
> should have become free to be recycled as it has been written out to the usb2
> drive at 17Mb/s - Somehow this doesn't seem to be happening.
>
> The result that I am seeing could be explained if the kernel was not
> releasing dirty pages after they have been written to disk, but I know this
> doesn't make sense.
>
> Can anyone make sense of this and let me know whether this is expected
> behaviour or not?
>
the kernel does wait before writing things out look at the values in
/proc/sys/vm/dirty* to tune this and have it start writing sooner.
David Lang
--
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
-- C.A.R. Hoare
Thanks David, but if you read my original post in full you'll see that I've tried that, and while I can start the write out sooner by lowering /proc/sys/vm/dirty_ratio , it makes no difference to the results that I am getting. I still seem to run out of steam after only 50 seconds where it should take about 3 minutes.
regards, Anthony
--
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd
[email protected]
Phone: (02) 62595404 or 0419409836
I saw it mentioned before that the kernel only allows a certain
percentage of total memory to be dirty, I thought the number was
around 40%, and I have seen machines with large amounts of ram,
hit the 40% and then put the writing application into disk wait
until certain amounts of things are written out, and then take
it out of disk wait, and repeat when it again hits 40%, given your
rate different it would be close to 40% in 50seconds.
And I think that you mean MB(yte) not Mb(it).
Roger
> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]] On Behalf Of
> Anthony Wesley
> Sent: Friday, September 09, 2005 4:11 AM
> To: [email protected]
> Subject: Re: kernel 2.6.13 buffer strangeness
>
> Thanks David, but if you read my original post in full you'll
> see that I've tried that, and while I can start the write out
> sooner by lowering /proc/sys/vm/dirty_ratio , it makes no
> difference to the results that I am getting. I still seem to
> run out of steam after only 50 seconds where it should take
> about 3 minutes.
>
> regards, Anthony
>
> --
> Anthony Wesley
> Director and IT/Network Consultant
> Smart Networks Pty Ltd
> Acquerra Pty Ltd
>
> [email protected]
> Phone: (02) 62595404 or 0419409836
>
> -
> To unsubscribe from this list: send the line "unsubscribe
> linux-kernel" in the body of a message to
> [email protected] More majordomo info at
> http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
On 9/9/05, Roger Heflin <[email protected]> wrote:
>
> I saw it mentioned before that the kernel only allows a certain
> percentage of total memory to be dirty, I thought the number was
> around 40%, and I have seen machines with large amounts of ram,
> hit the 40% and then put the writing application into disk wait
> until certain amounts of things are written out, and then take
> it out of disk wait, and repeat when it again hits 40%, given your
> rate different it would be close to 40% in 50seconds.
>
yes, on 2.6 there are two tunables which are important here.
dirty_background_ratio is the threshold where the kernel will begin
flushing dirty buffers, so it should change how soon the disk becomes
active. dirty_ratio changes when the write-throttling code kicks in,
which is what Anthony is seeing. The purpose of the write throttling
code is to limit the dirtying process to disk bandwidth, so that is a
Feature. Anthony, try *increasing* dirty_ratio, you can go up to 100,
but you could trigger an OOM if you let it get too high, so maybe try
setting it at 85 or so. This should effectively disable the write
throttling and give you the bandwidth you want.
NATE
> And I think that you mean MB(yte) not Mb(it).
>
> Roger
>
> > -----Original Message-----
> > From: [email protected]
> > [mailto:[email protected]] On Behalf Of
> > Anthony Wesley
> > Sent: Friday, September 09, 2005 4:11 AM
> > To: [email protected]
> > Subject: Re: kernel 2.6.13 buffer strangeness
> >
> > Thanks David, but if you read my original post in full you'll
> > see that I've tried that, and while I can start the write out
> > sooner by lowering /proc/sys/vm/dirty_ratio , it makes no
> > difference to the results that I am getting. I still seem to
> > run out of steam after only 50 seconds where it should take
> > about 3 minutes.
> >
> > regards, Anthony
> >
> > --
> > Anthony Wesley
> > Director and IT/Network Consultant
> > Smart Networks Pty Ltd
> > Acquerra Pty Ltd
> >
> > [email protected]
> > Phone: (02) 62595404 or 0419409836
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe
> > linux-kernel" in the body of a message to
> > [email protected] More majordomo info at
> > http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at http://www.tux.org/lkml/
> >
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
Hi Nate and Roger,
First off, thanks fo taking some time to answer my call for help :-)
I've already spent some time fiddling with dirty_ratio before I posted my question - yes, I can make different things happen by increasing or decreasing it, but overall I still have the same problem - my video streaming runs out of steam after at most 50 seconds instead of the 3 minutes that I think it should take.
Setting dirty_ratio and dirty_background_ratio to 15/10 respectively makes things worse, I hit the wall after only about 25 seconds.
Setting dirty_ratio and dirty_background_ratio to 85/80 makes things worse - disk writes start after about 15 seconds and I hit the wall after about 35 seconds.
Setting dirty_ratio and dirty_background_ratio to 90/10 puts me back at around 50 seconds, i.e. where I started.
So as far as I can see there is *no way* to get 3 minutes worth of buffering by adjusting these parameters.
Just to remind everyone - I have video data coming in at 25MBytes/sec and I am writing it out to a usb2 hard disk that can sustain 17MBytes/sec. I want my video capture to run at full speed as long as possible by having the 7MBytes/sec deficit slowly eat up the available RAM in the machine. I have 1.5Gb of RAM, 1.3Gb available for buffers, so this should take 3 minutes to consume at 7MBytes/sec.
So, I've tried all the combinations on dirty_ratio and dirty_background_ratio and they *do not help*.
Can anyone suggest something else that I might try? The goal is to have 25MBytes/sec streaming video for about 3 minutes.
Or is this simply not possible with the current kernel I/O setup? Do I have to do something elaborate myself, like build a big RAM buffer, mount the disk synchronous, do the buffering myself in userland...??
regards, Anthony
Nate Diller wrote:
> On 9/9/05, Roger Heflin <[email protected]> wrote:
>
>>I saw it mentioned before that the kernel only allows a certain
>>percentage of total memory to be dirty, I thought the number was
>>around 40%, and I have seen machines with large amounts of ram,
>>hit the 40% and then put the writing application into disk wait
>>until certain amounts of things are written out, and then take
>>it out of disk wait, and repeat when it again hits 40%, given your
>>rate different it would be close to 40% in 50seconds.
>>
>
> yes, on 2.6 there are two tunables which are important here.
> dirty_background_ratio is the threshold where the kernel will begin
> flushing dirty buffers, so it should change how soon the disk becomes
> active. dirty_ratio changes when the write-throttling code kicks in,
> which is what Anthony is seeing. The purpose of the write throttling
> code is to limit the dirtying process to disk bandwidth, so that is a
> Feature. Anthony, try *increasing* dirty_ratio, you can go up to 100,
> but you could trigger an OOM if you let it get too high, so maybe try
> setting it at 85 or so. This should effectively disable the write
> throttling and give you the bandwidth you want.
>
> NATE
>
>
>>And I think that you mean MB(yte) not Mb(it).
>>
>> Roger
>>
>>
>>>-----Original Message-----
>>>From: [email protected]
>>>[mailto:[email protected]] On Behalf Of
>>>Anthony Wesley
>>>Sent: Friday, September 09, 2005 4:11 AM
>>>To: [email protected]
>>>Subject: Re: kernel 2.6.13 buffer strangeness
>>>
>>>Thanks David, but if you read my original post in full you'll
>>>see that I've tried that, and while I can start the write out
>>>sooner by lowering /proc/sys/vm/dirty_ratio , it makes no
>>>difference to the results that I am getting. I still seem to
>>>run out of steam after only 50 seconds where it should take
>>>about 3 minutes.
>>>
>>>regards, Anthony
>>>
>>>--
>>>Anthony Wesley
>>>Director and IT/Network Consultant
>>>Smart Networks Pty Ltd
>>>Acquerra Pty Ltd
>>>
>>>[email protected]
>>>Phone: (02) 62595404 or 0419409836
>>>
>>>-
>>>To unsubscribe from this list: send the line "unsubscribe
>>>linux-kernel" in the body of a message to
>>>[email protected] More majordomo info at
>>>http://vger.kernel.org/majordomo-info.html
>>>Please read the FAQ at http://www.tux.org/lkml/
>>>
>>
>>-
>>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>>the body of a message to [email protected]
>>More majordomo info at http://vger.kernel.org/majordomo-info.html
>>Please read the FAQ at http://www.tux.org/lkml/
>>
>
>
--
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd
[email protected]
Phone: (02) 62595404 or 0419409836
On 9/9/05, Anthony Wesley <[email protected]> wrote:
> Hi Nate and Roger,
>
> First off, thanks fo taking some time to answer my call for help :-)
>
> I've already spent some time fiddling with dirty_ratio before I posted my
> question - yes, I can make different things happen by increasing or
> decreasing it, but overall I still have the same problem - my video
> streaming runs out of steam after at most 50 seconds instead of the 3
> minutes that I think it should take.
>
> Setting dirty_ratio and dirty_background_ratio to 15/10 respectively makes
> things worse, I hit the wall after only about 25 seconds.
>
not surprising
> Setting dirty_ratio and dirty_background_ratio to 85/80 makes things worse -
> disk writes start after about 15 seconds and I hit the wall after about 35
> seconds.
>
so dirty_background_ratio is working, that's why it took a long time
to start writing anything out
> Setting dirty_ratio and dirty_background_ratio to 90/10 puts me back at
> around 50 seconds, i.e. where I started.
>
this setting should do the trick, so there's something going on here
that isn't expected.
> So as far as I can see there is *no way* to get 3 minutes worth of buffering
> by adjusting these parameters.
>
> Just to remind everyone - I have video data coming in at 25MBytes/sec and I
> am writing it out to a usb2 hard disk that can sustain 17MBytes/sec. I want
> my video capture to run at full speed as long as possible by having the
> 7MBytes/sec deficit slowly eat up the available RAM in the machine. I have
> 1.5Gb of RAM, 1.3Gb available for buffers, so this should take 3 minutes to
> consume at 7MBytes/sec.
>
> So, I've tried all the combinations on dirty_ratio and
> dirty_background_ratio and they *do not help*.
>
dirty_ratio is the tubable you want, if it's not working correctly,
either there's a problem with your setup, or a bug
> Can anyone suggest something else that I might try? The goal is to have
> 25MBytes/sec streaming video for about 3 minutes.
>
how sure are you that you're getting 17MB/s during this test? can you
run "vmstat 1" while this is running to verify? which FS and
scheduler?
just for interest, what's the raw disk bandwidth (use hdparm, or run a
dd, or something)? it would obviously be much better to sustain
25MB/s to disk
> Or is this simply not possible with the current kernel I/O setup? Do I have
> to do something elaborate myself, like build a big RAM buffer, mount the
> disk synchronous, do the buffering myself in userland...??
>
this should be possible, although it could be considered a bit risky WRT OOM.
NATE
> regards, Anthony
>
> Nate Diller wrote:
>
> > On 9/9/05, Roger Heflin <[email protected]> wrote:
> >
> >>I saw it mentioned before that the kernel only allows a certain
> >>percentage of total memory to be dirty, I thought the number was
> >>around 40%, and I have seen machines with large amounts of ram,
> >>hit the 40% and then put the writing application into disk wait
> >>until certain amounts of things are written out, and then take
> >>it out of disk wait, and repeat when it again hits 40%, given your
> >>rate different it would be close to 40% in 50seconds.
> >>
> >
> > yes, on 2.6 there are two tunables which are important here.
> > dirty_background_ratio is the threshold where the kernel will begin
> > flushing dirty buffers, so it should change how soon the disk becomes
> > active. dirty_ratio changes when the write-throttling code kicks in,
> > which is what Anthony is seeing. The purpose of the write throttling
> > code is to limit the dirtying process to disk bandwidth, so that is a
> > Feature. Anthony, try *increasing* dirty_ratio, you can go up to 100,
> > but you could trigger an OOM if you let it get too high, so maybe try
> > setting it at 85 or so. This should effectively disable the write
> > throttling and give you the bandwidth you want.
> >
> > NATE
> >
> >
> >>And I think that you mean MB(yte) not Mb(it).
> >>
> >> Roger
> >>
> >>
> >>>-----Original Message-----
> >>>From: [email protected]
> >>>[mailto:[email protected]] On Behalf Of
> >>>Anthony Wesley
> >>>Sent: Friday, September 09, 2005 4:11 AM
> >>>To: [email protected]
> >>>Subject: Re: kernel 2.6.13 buffer strangeness
> >>>
> >>>Thanks David, but if you read my original post in full you'll
> >>>see that I've tried that, and while I can start the write out
> >>>sooner by lowering /proc/sys/vm/dirty_ratio , it makes no
> >>>difference to the results that I am getting. I still seem to
> >>>run out of steam after only 50 seconds where it should take
> >>>about 3 minutes.
> >>>
> >>>regards, Anthony
> >>>
> >>>--
> >>>Anthony Wesley
> >>>Director and IT/Network Consultant
> >>>Smart Networks Pty Ltd
> >>>Acquerra Pty Ltd
> >>>
> >>>[email protected]
> >>>Phone: (02) 62595404 or 0419409836
> >>>
> >>>-
> >>>To unsubscribe from this list: send the line "unsubscribe
> >>>linux-kernel" in the body of a message to
> >>>[email protected] More majordomo info at
> >>>http://vger.kernel.org/majordomo-info.html
> >>>Please read the FAQ at http://www.tux.org/lkml/
> >>>
> >>
> >>-
> >>To unsubscribe from this list: send the line "unsubscribe linux-kernel"
> in
> >>the body of a message to [email protected]
> >>More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>Please read the FAQ at http://www.tux.org/lkml/
> >>
> >
> >
>
> --
> Anthony Wesley
> Director and IT/Network Consultant
> Smart Networks Pty Ltd
> Acquerra Pty Ltd
>
> [email protected]
> Phone: (02) 62595404 or 0419409836
>
Based on a better understanding of dirty_ratio and dirty_background ratio (thanks Nate) I just tried
the following test:
dirty_ratio set to 95
dirty_background_ratio set to 1
>From Nate's description of these parameters, this should mean that the disk writes
start almost immediately, and the kernel will allow 95% of RAM to become dirty before
applying the throttle.
Ok, so with 25Mbytes/s coming in, and 17Mbytes/sec going out to disk, the dirty pages should be growing
at 7Mbytes/sec. With these parameters set as above I should see about 3 minutes of full speed
video before the throttle is applied since I have about 1.3Gb of RAM free for buffering..
*But* when I try this experiment I hit the throttle after only 65 seconds - an improvement to
be sure, but still a long way short of the 180 seconds that it ought to take.
Part of the test works as expected - the disk writes begin almost immediately due to the low
value for dirty_background_ratio, but the rest is a mystery.
It really looks as if the pages aren't being marked as clean fast enough after they're written.
How else can it take only 70 seconds to reach 95% dirty when I have 1.3Gb of available RAM and data coming in at 25MBytes/sec and out at 17MBytes/sec? It doesn't make any sense...
regards, Anthony
Nate Diller wrote:
> yes, on 2.6 there are two tunables which are important here.
> dirty_background_ratio is the threshold where the kernel will begin
> flushing dirty buffers, so it should change how soon the disk becomes
> active. dirty_ratio changes when the write-throttling code kicks in,
> which is what Anthony is seeing. The purpose of the write throttling
> code is to limit the dirtying process to disk bandwidth, so that is a
> Feature. Anthony, try *increasing* dirty_ratio, you can go up to 100,
> but you could trigger an OOM if you let it get too high, so maybe try
> setting it at 85 or so. This should effectively disable the write
> throttling and give you the bandwidth you want.
>
> NATE
--
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd
[email protected]
Phone: (02) 62595404 or 0419409836
Okay, just tested a couple of things, here's what I see...
I tested the write speed to the usb2 hard disk and got 21MBytes/sec. It's a laptop
hard drive, only 5400rpm so this is not surprising.
I did this test with my video capture app running, but just displaying data and not writing,
I have another laptop usb2 hard drive here which I just tested and got 17MBytes/sec - it's
a 4200rpm drive so again not surprising numbers.
The video data is written as individual frames, so the efficiency is a bit below the raw throughput,
but my tests were transferring 1.5Gb of data as raw frames - exactly the same way that Coriander would
write its data.
I'd bet a large sum of money that these hard disk figures are correct to within a few percent.
Also, when I am actually capturing I have timed by stopwatch how long the disk activity light
is on, and reached about the same conclusion.
Working the problem from the other direction, the only way to explain the early throttling that
I see would be if *almost no data* is being written to the disk, and this is plainly not the case. Even if
the disk were running at a greatly reduced rate of (say) 10MBytes/sec I would still see 86 seconds
of buffering before the throttle kicks in, and so far I have managed only to get to about 65 or 70.
regards, Anthony
Nate Diller wrote:
>>Setting dirty_ratio and dirty_background_ratio to 90/10 puts me back at
>>around 50 seconds, i.e. where I started.
>>
>
> this setting should do the trick, so there's something going on here
> that isn't expected.
>
>
>>So as far as I can see there is *no way* to get 3 minutes worth of buffering
>>by adjusting these parameters.
>>
>>Just to remind everyone - I have video data coming in at 25MBytes/sec and I
>>am writing it out to a usb2 hard disk that can sustain 17MBytes/sec. I want
>>my video capture to run at full speed as long as possible by having the
>>7MBytes/sec deficit slowly eat up the available RAM in the machine. I have
>>1.5Gb of RAM, 1.3Gb available for buffers, so this should take 3 minutes to
>>consume at 7MBytes/sec.
>>
>>So, I've tried all the combinations on dirty_ratio and
>>dirty_background_ratio and they *do not help*.
>>
>
> dirty_ratio is the tubable you want, if it's not working correctly,
> either there's a problem with your setup, or a bug
>
>
>>Can anyone suggest something else that I might try? The goal is to have
>>25MBytes/sec streaming video for about 3 minutes.
>>
>
> how sure are you that you're getting 17MB/s during this test? can you
> run "vmstat 1" while this is running to verify? which FS and
> scheduler?
>
> just for interest, what's the raw disk bandwidth (use hdparm, or run a
> dd, or something)? it would obviously be much better to sustain
> 25MB/s to disk
>
>
>>Or is this simply not possible with the current kernel I/O setup? Do I have
>>to do something elaborate myself, like build a big RAM buffer, mount the
>>disk synchronous, do the buffering myself in userland...??
>>
>
> this should be possible, although it could be considered a bit risky WRT OOM.
>
> NATE
--
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd
[email protected]
Phone: (02) 62595404 or 0419409836
On 9/9/05, Anthony Wesley <[email protected]> wrote:
> Okay, just tested a couple of things, here's what I see...
>
> I tested the write speed to the usb2 hard disk and got 21MBytes/sec. It's a
> laptop
> hard drive, only 5400rpm so this is not surprising.
>
> I did this test with my video capture app running, but just displaying data
> and not writing,
>
> I have another laptop usb2 hard drive here which I just tested and got
> 17MBytes/sec - it's
> a 4200rpm drive so again not surprising numbers.
>
> The video data is written as individual frames, so the efficiency is a bit
> below the raw throughput,
> but my tests were transferring 1.5Gb of data as raw frames - exactly the
> same way that Coriander would
> write its data.
>
> I'd bet a large sum of money that these hard disk figures are correct to
> within a few percent.
>
> Also, when I am actually capturing I have timed by stopwatch how long the
> disk activity light
> is on, and reached about the same conclusion.
>
> Working the problem from the other direction, the only way to explain the
> early throttling that
> I see would be if *almost no data* is being written to the disk, and this is
> plainly not the case. Even if
> the disk were running at a greatly reduced rate of (say) 10MBytes/sec I
> would still see 86 seconds
> of buffering before the throttle kicks in, and so far I have managed only to
> get to about 65 or 70.
>
> regards, Anthony
>
i need a vmstat trace ('vmstat 1 > logfile', then run the test, then
^c) before i can comment further. also, your filesystem and
scheduler would be interesting to know. you can find out the
scheduler with 'cat /sys/block/[drive]/queue/scheduler'.
NATE
> Nate Diller wrote:
>
>
> >>Setting dirty_ratio and dirty_background_ratio to 90/10 puts me back at
> >>around 50 seconds, i.e. where I started.
> >>
> >
> > this setting should do the trick, so there's something going on here
> > that isn't expected.
> >
> >
> >>So as far as I can see there is *no way* to get 3 minutes worth of
> buffering
> >>by adjusting these parameters.
> >>
> >>Just to remind everyone - I have video data coming in at 25MBytes/sec and
> I
> >>am writing it out to a usb2 hard disk that can sustain 17MBytes/sec. I
> want
> >>my video capture to run at full speed as long as possible by having the
> >>7MBytes/sec deficit slowly eat up the available RAM in the machine. I
> have
> >>1.5Gb of RAM, 1.3Gb available for buffers, so this should take 3 minutes
> to
> >>consume at 7MBytes/sec.
> >>
> >>So, I've tried all the combinations on dirty_ratio and
> >>dirty_background_ratio and they *do not help*.
> >>
> >
> > dirty_ratio is the tubable you want, if it's not working correctly,
> > either there's a problem with your setup, or a bug
> >
> >
> >>Can anyone suggest something else that I might try? The goal is to have
> >>25MBytes/sec streaming video for about 3 minutes.
> >>
> >
> > how sure are you that you're getting 17MB/s during this test? can you
> > run "vmstat 1" while this is running to verify? which FS and
> > scheduler?
> >
> > just for interest, what's the raw disk bandwidth (use hdparm, or run a
> > dd, or something)? it would obviously be much better to sustain
> > 25MB/s to disk
> >
> >
> >>Or is this simply not possible with the current kernel I/O setup? Do I
> have
> >>to do something elaborate myself, like build a big RAM buffer, mount the
> >>disk synchronous, do the buffering myself in userland...??
> >>
> >
> > this should be possible, although it could be considered a bit risky WRT
> OOM.
> >
> > NATE
>
> --
> Anthony Wesley
> Director and IT/Network Consultant
> Smart Networks Pty Ltd
> Acquerra Pty Ltd
>
> [email protected]
> Phone: (02) 62595404 or 0419409836
>
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
4 0 144 1314396 29440 67444 0 0 16 407 290 17 30 30 38 2
0 0 144 1314396 29440 67436 0 0 0 0 405 1062 26 4 70 0
0 0 144 1314388 29440 67436 0 0 0 0 721 1397 27 4 69 0
0 0 144 1314388 29440 67436 0 0 0 0 732 1554 30 3 67 0
0 0 144 1300624 29548 80220 0 0 24 0 426 1207 26 13 40 21
0 0 144 1275444 29744 105148 0 0 8 20752 933 2262 31 21 49 0
3 2 144 1250396 29912 130076 0 0 4 15286 1004 2357 31 22 0 47
0 3 144 1225224 30076 155004 0 0 0 14008 1030 2439 28 22 0 49
1 3 144 1200788 30244 179324 0 0 8 13772 1049 2526 29 21 0 50
0 3 144 1175616 30412 204256 0 0 4 15964 1102 2696 20 18 0 62
0 5 144 1150552 30584 229184 0 0 0 15980 1100 2724 21 17 0 62
2 1 144 1125836 30756 253504 0 0 12 14672 1106 2715 20 19 0 62
0 7 144 1100788 30920 278432 0 0 0 15380 1079 2665 19 18 0 63
2 6 144 1075616 31092 303360 0 0 8 15440 1112 2731 21 18 0 61
0 8 144 1049948 31264 328900 0 0 4 14832 1063 2685 18 22 0 60
2 8 144 1024776 31436 353828 0 0 0 15620 1117 2815 19 17 0 64
0 2 144 1001588 31600 376932 0 0 12 15828 1093 2677 18 20 0 62
2 6 144 976432 31764 401860 0 0 0 14724 1085 2604 21 19 0 60
0 5 144 950764 31932 427396 0 0 0 15844 1093 2734 21 18 0 61
1 4 144 924964 32112 452936 0 0 12 15028 1044 2583 22 18 0 61
0 4 144 899792 32292 477864 0 0 8 14788 1071 2670 18 18 0 64
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 2 144 880200 32432 497320 0 0 12 15228 1103 2676 18 18 0 64
0 5 144 855152 32596 522248 0 0 0 14604 1053 2598 19 20 0 61
0 6 144 829980 32760 547176 0 0 0 16004 1124 2741 21 17 0 63
0 6 144 804188 32932 572716 0 0 4 15896 1081 2704 19 18 0 63
1 4 144 779116 33112 597644 0 0 8 15020 1094 2722 19 19 0 62
0 11 144 755672 33272 620840 0 0 4 15876 1076 2750 17 20 0 63
1 11 144 733104 33416 643244 0 0 0 16220 1110 2795 19 16 0 65
0 2 144 708056 33580 668172 0 0 0 15836 1074 2668 20 18 0 63
0 8 144 682884 33744 693104 0 0 0 15952 1115 2758 20 16 0 64
1 4 144 657216 33920 718640 0 0 0 15908 1115 2765 18 18 0 64
0 6 144 632664 34088 742960 0 0 8 14928 1080 2663 21 17 0 62
0 3 144 609972 34252 765456 0 0 16 14888 1055 2606 20 19 0 61
1 2 144 584320 34432 790992 0 0 12 15044 1094 2709 18 20 0 62
0 1 144 559768 34600 815316 0 0 8 14948 1069 2634 20 19 0 61
0 6 144 534596 34772 840244 0 0 0 16188 1092 2645 20 20 0 60
1 5 144 508928 34944 865780 0 0 4 14832 1063 2625 21 20 0 60
0 2 144 489972 35080 884628 0 0 12 13700 1057 2515 21 18 0 61
1 3 144 464800 35256 909556 0 0 12 17008 1101 2728 23 15 0 62
1 1 144 439008 35432 935096 0 0 8 13692 1109 2708 22 18 0 60
0 3 144 413960 35604 960024 0 0 0 17072 1096 2676 19 20 0 61
1 5 144 388804 35776 984952 0 0 8 14484 1082 2640 22 20 0 59
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 2 144 366112 35936 1007448 0 0 12 15324 1078 2626 20 21 0 59
2 2 144 341064 36108 1032376 0 0 8 14544 1060 2551 22 18 0 60
1 3 144 315272 36276 1057912 0 0 0 15836 1057 2639 22 20 0 59
0 5 144 290100 36448 1082844 0 0 0 15420 1130 2764 21 19 0 60
1 5 144 264060 36616 1108384 0 0 0 14836 1078 2648 23 18 0 59
0 5 144 238640 36780 1133312 0 0 0 16124 1109 2725 20 20 0 60
2 4 144 215544 36932 1155808 0 0 4 14768 1058 2613 21 18 0 61
0 9 144 189504 37104 1181344 0 0 4 14064 1125 2823 21 21 0 58
1 6 144 164084 37276 1206276 0 0 0 16040 1071 2728 19 20 0 61
1 9 144 138664 37440 1231204 0 0 0 16020 1136 2822 20 20 0 60
1 1 144 113120 37608 1256132 0 0 4 14604 1070 2734 18 20 0 62
2 6 144 89560 37764 1279236 0 0 4 14896 1081 2612 17 22 0 61
0 9 144 64140 37928 1304164 0 0 0 17184 1092 2818 18 21 0 61
0 9 144 38572 38100 1329096 0 0 0 15784 1126 2833 22 19 0 59
2 3 144 32628 27604 1345272 0 0 0 14780 1101 2675 21 23 0 56
1 9 144 32628 16548 1356284 0 0 0 16024 1100 2740 21 22 0 57
3 9 144 32628 16712 1356028 0 0 4 14852 1069 2729 19 19 0 62
0 4 144 32628 16880 1355844 0 0 8 15044 1082 2692 19 21 0 60
1 9 144 32132 17052 1356200 0 0 0 16100 1090 2776 21 20 0 59
2 6 144 32504 17216 1355784 0 0 0 14712 1097 2719 20 22 0 58
0 9 144 32628 17384 1355368 0 0 4 14908 1037 2690 22 20 0 58
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 5 144 32380 17536 1355528 0 0 12 13812 1060 2593 24 17 0 59
0 9 144 32752 17704 1354952 0 0 0 14932 1063 2654 22 23 0 55
0 9 144 32256 17880 1355244 0 0 0 7228 1113 2881 22 21 0 57
0 2 144 32008 18044 1355388 0 0 8 14892 1090 2761 23 22 0 56
1 3 144 32008 18220 1355052 0 0 8 13996 1050 2585 20 22 0 58
1 9 144 32504 18360 1354524 0 0 8 13252 1021 2530 19 19 0 62
1 8 144 32876 18516 1353884 0 0 0 15956 1120 2771 22 18 0 60
1 6 144 32804 18700 1353760 0 0 8 14120 1079 2738 23 22 0 56
0 8 144 32628 18868 1353748 0 0 4 12808 1221 3006 21 22 0 57
1 8 144 33052 19024 1353080 0 0 0 6320 989 2631 21 21 0 58
0 8 144 32756 19192 1353224 0 0 0 6308 1099 2833 21 22 0 57
0 2 144 32628 19300 1353296 0 0 12 10540 1019 2564 20 18 0 62
2 8 144 31992 19428 1353644 0 0 4 20276 973 2448 20 19 0 61
0 8 144 32256 19520 1353376 0 0 4 9816 1103 2694 19 20 0 62
1 4 144 32028 19600 1353384 0 0 4 18116 1114 2643 19 16 0 65
1 5 144 33000 19740 1352244 0 0 4 12424 1055 2593 19 21 0 60
0 8 144 33076 19848 1352068 0 0 4 22344 1111 2682 18 17 0 65
2 3 144 32480 19964 1352520 0 0 12 9428 1078 2709 18 17 0 65
1 8 144 32224 20080 1352584 0 0 4 17816 1071 2622 17 21 0 62
0 11 144 32224 20184 1352604 0 0 4 14052 1054 2643 19 17 0 64
1 13 144 32052 20292 1352540 0 0 4 15940 1110 2752 21 17 0 63
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 13 144 32596 20384 1351948 0 0 4 14792 1041 2631 21 14 0 65
2 14 144 32720 20456 1351652 0 0 4 18476 1077 2676 19 14 0 67
0 5 144 32720 20604 1351568 0 0 0 13072 1078 2692 21 18 0 61
0 10 144 32720 20724 1351336 0 0 4 19720 1106 2690 20 18 0 62
1 13 144 31976 20816 1352016 0 0 4 11964 1065 2639 19 15 0 66
0 14 144 32596 20876 1351476 0 0 8 19012 1092 2702 19 17 0 65
1 14 144 31976 21012 1351784 0 0 8 13204 1084 2805 20 19 0 61
0 2 144 31976 21100 1351776 0 0 0 14896 1078 2706 20 16 0 64
0 9 144 32476 21240 1351048 0 0 4 18760 1069 2603 20 23 0 58
1 13 144 32100 21344 1351392 0 0 0 14720 1126 2774 20 17 0 63
0 14 144 32596 21460 1350860 0 0 4 18992 1095 2735 22 18 0 60
0 14 144 33096 21576 1350072 0 0 0 14660 1118 2860 22 17 0 61
1 13 144 31976 21656 1351056 0 0 4 13764 1074 2674 20 18 0 63
0 4 144 32724 21788 1350176 0 0 0 19236 1089 2710 20 20 0 60
1 9 144 32720 21888 1350128 0 0 4 12440 1079 2604 20 20 0 61
0 14 144 32348 21996 1350320 0 0 8 24964 1111 2753 21 15 0 64
0 13 144 32704 22112 1349948 0 0 4 7960 1088 2791 17 22 0 61
2 14 144 32208 22196 1350360 0 0 0 22596 1136 2790 20 21 0 60
0 2 144 32456 22328 1349764 0 0 4 9816 1049 2681 21 19 0 60
0 8 144 32084 22448 1350124 0 0 0 17024 1105 2668 19 21 0 60
2 13 144 32084 22544 1350000 0 0 0 19064 1102 2740 18 20 0 62
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 4 144 32828 22668 1349180 0 0 8 18948 1106 2710 18 17 0 65
1 7 144 31960 22768 1349892 0 0 8 10664 1068 2580 18 21 0 62
1 12 144 32084 22892 1349488 0 0 0 18428 1118 2753 21 18 0 61
0 13 144 31960 22996 1349656 0 0 4 14540 1082 2713 19 17 0 64
0 14 144 32084 23088 1349364 0 0 4 23028 1089 2751 17 17 0 66
1 13 144 31960 23228 1349220 0 0 0 8720 1103 2785 19 19 0 62
0 13 144 32580 23320 1348772 0 0 4 14816 1053 2626 20 17 0 64
0 13 144 32208 23404 1348964 0 0 4 15972 1068 2628 18 16 0 66
1 14 144 32580 23380 1348432 0 0 0 25216 1106 2808 20 19 0 61
0 13 144 32704 23144 1348680 0 0 4 7140 1099 2742 17 17 0 66
0 13 144 32580 22776 1349172 0 0 4 22132 1128 2822 24 15 0 62
2 14 144 32332 22192 1349820 0 0 0 14152 1062 2691 18 21 0 61
0 13 144 32208 21628 1350684 0 0 0 15448 1121 2750 18 18 0 64
1 5 144 31960 20996 1351536 0 0 4 18192 1038 2541 21 21 0 58
0 9 144 32208 21024 1351088 0 0 0 11980 1119 2689 19 19 0 62
1 13 144 32828 21100 1350576 0 0 0 19696 1095 2783 21 14 0 65
0 13 144 32084 21160 1351156 0 0 4 18324 1097 2769 22 16 0 62
1 13 144 32832 21276 1350352 0 0 4 10688 1076 2709 20 18 0 62
0 13 144 32952 21388 1350604 0 0 4 20564 1124 2843 22 18 0 60
2 1 144 32952 21392 1350604 0 0 4 14460 1146 2729 17 15 0 68
0 1 144 32952 21404 1350604 0 0 4 16548 1402 3150 19 12 0 69
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 1 144 32952 21408 1350604 0 0 4 18056 1167 2657 16 14 0 70
0 1 144 32952 21408 1350604 0 0 0 18548 1189 2675 17 17 0 67
0 1 144 32952 21412 1350604 0 0 4 14948 1067 2490 17 12 0 71
Nate Diller wrote:
> just found the culprit. guess i should have read the code the first
> time. get_dirty_limits() in drivers/block/page_writeback.c has a
> hard-coded upper limit to dirty_ratio. it's capped to half of the
> unmapped pages, so maybe 30-40% of your system's memory. so if you are
> brave, just remove the "/ 2" parts from the 'if (dirty_ratio >
> unmapped_ratio / 2) dirty_ratio = unmapped_ratio / 2;' check, and you
> can have all the OOM goodness you want.
Excellent. OOM here I come.
> i really recommend you focus on getting better disk bandwidth, you stand
> to gain a lot more from that approach. i presume you're on ext3;
> perhaps you should try reiser4 or xfs, they are more likely to meet your
> disk bandwidth requirements.
Yep, pursuing this as well, also looking to add more RAM to the machine. At this stage
I was just trying to understand the numbers that I was seeing, so I could work out the
best way to proceed.
I'm using ext2 at the moment, on the assumption that the journal would cost
me a bit of performance so I left it out :-)
I'll certainly try the other filesystems as you suggest.
Thanks again, Anthony
--
Anthony Wesley
Director and IT/Network Consultant
Smart Networks Pty Ltd
Acquerra Pty Ltd
[email protected]
Phone: (02) 62595404 or 0419409836
Anthony Wesley <[email protected]> wrote:
>
> How else can it take only 70 seconds to reach 95% dirty when I have 1.3Gb of available RAM and data coming in at 25MBytes/sec and out at 17MBytes/sec? It doesn't make any sense...
What architecture? x86?
If so then bear in mind that your memory is split into 500MB highmem and
800MB lowmem. The kernel might be starting I/O due to the highmem zone
being full of dirty pages. That'd be wrong of it if so - it's supposed to
just fall back to lowmem for the page allocations, but that code has
changed quite a bit in the two years since I got all that working...
You need to run `watch -n1 cat /proc/meminfo' while doing these tests...