LinuxLists.cc - Warning - running *really* short on DMA buffers while doing file transfers

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

On Wed, Sep 25 2002, Mathieu Chouquet-Stringer wrote:
> Hello!
>
> I've upgraded a while to 2.4.19 and my box has been happy for the last 52
> days (it's a dual PIII). Tonight while going through my logs, I've found
> these:
>
> Sep 25 22:18:41 bigip kernel: Warning - running *really* short on DMA buffers
> Sep 25 22:18:47 bigip last message repeated 55 times
> Sep 25 22:19:41 bigip last message repeated 71 times

This is fixed in 2.4.20-pre

> Oh BTW, just one thing, I wanted to give the throughput of the ide drived
> but it failed:
> Sep 25 23:18:32 bigip kernel: hdb: dma_intr: status=0x51 { DriveReady SeekComplete Error }
> Sep 25 23:18:32 bigip kernel: hdb: dma_intr: error=0x40 { UncorrectableError }, LBAsect=102882, sector=102784

Yep, looks like the end of the road for that drive.

--
Jens Axboe

2002-09-26 06:59:43

by Pedro M. Rodrigues

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

Jens Axboe wrote:
> On Wed, Sep 25 2002, Mathieu Chouquet-Stringer wrote:
>
>> Hello!
>>
>>I've upgraded a while to 2.4.19 and my box has been happy for the last 52
>>days (it's a dual PIII). Tonight while going through my logs, I've found
>>these:
>>
>>Sep 25 22:18:41 bigip kernel: Warning - running *really* short on DMA buffers
>>Sep 25 22:18:47 bigip last message repeated 55 times
>>Sep 25 22:19:41 bigip last message repeated 71 times
>
>
> This is fixed in 2.4.20-pre
>
>

I reported this same problem some weeks ago -
http://marc.theaimsgroup.com/?l=linux-kernel&m=103069116227685&w=2 .
2.4.20pre kernels solved the error messages flooding the console, and
improved things a bit, but system load still got very high and disk read
and write performance was lousy. Adding more memory and using a
completely different machine didn't help. What did? Changing the Adaptec
scsi driver to aic7xxx_old . The performance was up 50% for writes and
90% for reads, and the system load was acceptable. And i didn't even had
to change the RedHat kernel (2.4.18-10) for a custom one. The storage
was two external Arena raid boxes, btw.

Regards,
Pedro

2002-09-26 15:26:38

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

> I reported this same problem some weeks ago -
> http://marc.theaimsgroup.com/?l=linux-kernel&m=103069116227685&w=2 .
> 2.4.20pre kernels solved the error messages flooding the console, and
> improved things a bit, but system load still got very high and disk read
> and write performance was lousy. Adding more memory and using a
> completely different machine didn't help. What did? Changing the Adaptec
> scsi driver to aic7xxx_old . The performance was up 50% for writes and
> 90% for reads, and the system load was acceptable. And i didn't even had
> to change the RedHat kernel (2.4.18-10) for a custom one. The storage was
> two external Arena raid boxes, btw.

I would be interested in knowing if reducing the maximum tag depth for
the driver improves things for you. There is a large difference in the
defaults between the two drivers. It has only reacently come to my
attention that the SCSI layer per-transaction overhead is so high that
you can completely starve the kernel of resources if this setting is too
high. For example, a 4GB system installing RedHat 7.3 could not even
complete an install on a 20 drive system with the default of 253 commands.
The latest version of the aic7xxx driver already sent to Marcelo drops the
default to 32.

--
Justin

2002-09-27 06:08:26

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

On Thu, Sep 26 2002, Justin T. Gibbs wrote:
> > I reported this same problem some weeks ago -
> > http://marc.theaimsgroup.com/?l=linux-kernel&m=103069116227685&w=2 .
> > 2.4.20pre kernels solved the error messages flooding the console, and
> > improved things a bit, but system load still got very high and disk read
> > and write performance was lousy. Adding more memory and using a
> > completely different machine didn't help. What did? Changing the Adaptec
> > scsi driver to aic7xxx_old . The performance was up 50% for writes and
> > 90% for reads, and the system load was acceptable. And i didn't even had
> > to change the RedHat kernel (2.4.18-10) for a custom one. The storage was
> > two external Arena raid boxes, btw.
>
> I would be interested in knowing if reducing the maximum tag depth for
> the driver improves things for you. There is a large difference in the
> defaults between the two drivers. It has only reacently come to my
> attention that the SCSI layer per-transaction overhead is so high that
> you can completely starve the kernel of resources if this setting is too
> high. For example, a 4GB system installing RedHat 7.3 could not even
> complete an install on a 20 drive system with the default of 253 commands.
> The latest version of the aic7xxx driver already sent to Marcelo drops the
> default to 32.

2.4 layer is most horrible there, 2.5 at least gets rid of the old
scsi_dma crap. That said, 253 default depth is a bit over the top, no?

--
Jens Axboe, who always uses 4

2002-09-27 06:31:22

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

On Thu, Sep 26 2002, Matthew Jacob wrote:
>
> > 2.4 layer is most horrible there, 2.5 at least gets rid of the old
> > scsi_dma crap. That said, 253 default depth is a bit over the top, no?
>
> Why? Something like a large Hitachi 9*** storage system can take ~1000
> tags w/o wincing.

Yeah, I bet that most of the devices attached to aic7xxx controllers are
exactly such beasts.

I didn't say that 253 is a silly default for _everything_, I think it's
a silly default for most users.

--
Jens Axboe

2002-09-27 06:28:57

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

> 2.4 layer is most horrible there, 2.5 at least gets rid of the old
> scsi_dma crap. That said, 253 default depth is a bit over the top, no?

Why? Something like a large Hitachi 9*** storage system can take ~1000
tags w/o wincing.

2002-09-27 06:44:58

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

> > > scsi_dma crap. That said, 253 default depth is a bit over the top, no?
> >
> > Why? Something like a large Hitachi 9*** storage system can take ~1000
> > tags w/o wincing.
>
> Yeah, I bet that most of the devices attached to aic7xxx controllers are
> exactly such beasts.
>
> I didn't say that 253 is a silly default for _everything_, I think it's
> a silly default for most users.
>

Well, no, I'm not sure I agree. In the expected life time of this
particular set of software that gets shipped out, the next generation of
100GB or better disk drives will be attached, and they'll likely eat all
of that many tags too, and usefully, considering the speed and bit
density of drives. For example, the current U160 Fujitsu drives will
take ~130 tags before sending back a QFULL.

On the other hand, we can also find a large class of existing devices
and situations where anything over 4 tags is overload too.

With some perspective on this, I'd have to say that in the last 25 years
I've seen more errors on the side of 'too conservative' for limits
rather than the opposite.

That said, the only problem with allowing such generous limits is the
impact on the system, which allows you to saturate as it does. Getting
that fixed is more important than saying a driver writer's choice for
limits is 'over the top'.

2002-09-27 06:51:08

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

On Thu, Sep 26 2002, Matthew Jacob wrote:
>
> > > > scsi_dma crap. That said, 253 default depth is a bit over the top, no?
> > >
> > > Why? Something like a large Hitachi 9*** storage system can take ~1000
> > > tags w/o wincing.
> >
> > Yeah, I bet that most of the devices attached to aic7xxx controllers are
> > exactly such beasts.
> >
> > I didn't say that 253 is a silly default for _everything_, I think it's
> > a silly default for most users.
> >
>
> Well, no, I'm not sure I agree. In the expected life time of this
> particular set of software that gets shipped out, the next generation of
> 100GB or better disk drives will be attached, and they'll likely eat all
> of that many tags too, and usefully, considering the speed and bit
> density of drives. For example, the current U160 Fujitsu drives will
> take ~130 tags before sending back a QFULL.

Just because a device can eat XXX number of tags does definitely _not_
make it a good idea. At least not if you care the slightest bit about
latency.

> On the other hand, we can also find a large class of existing devices
> and situations where anything over 4 tags is overload too.
>
> With some perspective on this, I'd have to say that in the last 25 years
> I've seen more errors on the side of 'too conservative' for limits
> rather than the opposite.

At least for this tagging discussion, I'm of the exact opposite
oppinion. What's the worst that can happen with a tag setting that is
too low? Theoretical loss of disk bandwidth. I say theoretical, because
it's not even given that tags are that much faster then the Linux io
scheduler. More tags might even give you _worse_ throughput, because you
end up leaving the io scheduler with much less to work on (if you have a
253 depth to you device, you have 3 requests left for the queue...).

So I think the 'more tags the better!' belief is very much bogus, at
least for the common case.

--
Jens Axboe

2002-09-27 07:13:42

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

>
> So I think the 'more tags the better!' belief is very much bogus, at
> least for the common case.

Well, that's one theory.

2002-09-27 07:19:38

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

On Fri, Sep 27 2002, Matthew Jacob wrote:
> >
> > So I think the 'more tags the better!' belief is very much bogus, at
> > least for the common case.
>
> Well, that's one theory.

Numbers talk, theory spinning walks

Both Andrew and I did latency numbers for even small depths of tagging,
and the result was not pretty. Sure this is just your regular plaino
SCSI drives, however that's also what I care most about. People with
big-ass hardware tend to find a way to tweak them as well, I'd like the
typical systems to run fine out of the box though.

--
Jens Axboe

2002-09-27 07:29:04

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

The issue here is not whether it's appropriate to oversaturate the
'standard' SCSI drive- it isn't- I never suggested it was.

I'd just suggest that it's asinine to criticise an HBA for running up to
reasonable limits when it's the non-toy OS that will do sensible I/O
scheduling. So point your gums elsewhere.

On Fri, 27 Sep 2002, Matthew Jacob wrote:

>
>
> On Fri, 27 Sep 2002, Jens Axboe wrote:
>
> > On Fri, Sep 27 2002, Matthew Jacob wrote:
> > > >
> > > > So I think the 'more tags the better!' belief is very much bogus, at
> > > > least for the common case.
> > >
> > > Well, that's one theory.
> >
> > Numbers talk, theory spinning walks
> >
> > Both Andrew and I did latency numbers for even small depths of tagging,
> > and the result was not pretty. Sure this is just your regular plaino
> > SCSI drives, however that's also what I care most about. People with
> > big-ass hardware tend to find a way to tweak them as well, I'd like the
> > typical systems to run fine out of the box though.
> >
>
> Fair enough.
>
>
>
>
>

2002-09-27 07:24:42

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

On Fri, 27 Sep 2002, Jens Axboe wrote:

> On Fri, Sep 27 2002, Matthew Jacob wrote:
> > >
> > > So I think the 'more tags the better!' belief is very much bogus, at
> > > least for the common case.
> >
> > Well, that's one theory.
>
> Numbers talk, theory spinning walks
>
> Both Andrew and I did latency numbers for even small depths of tagging,
> and the result was not pretty. Sure this is just your regular plaino
> SCSI drives, however that's also what I care most about. People with
> big-ass hardware tend to find a way to tweak them as well, I'd like the
> typical systems to run fine out of the box though.
>

Fair enough.

2002-09-27 07:39:59

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

On Fri, Sep 27 2002, Matthew Jacob wrote:
>
> The issue here is not whether it's appropriate to oversaturate the
> 'standard' SCSI drive- it isn't- I never suggested it was.

Ok so we agree. I think our oversaturate thresholds are different,
though.

> I'd just suggest that it's asinine to criticise an HBA for running up to
> reasonable limits when it's the non-toy OS that will do sensible I/O
> scheduling. So point your gums elsewhere.

Well I don't think 253 is a reasonable limit, that was the whole point.
How can sane io scheduling ever prevent starvation in that case? I can't
point my gums elsewhere, this is where I'm seeing starvation.

--
Jens Axboe

2002-09-27 08:32:09

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

> On Fri, Sep 27 2002, Matthew Jacob wrote:
> >
> > The issue here is not whether it's appropriate to oversaturate the
> > 'standard' SCSI drive- it isn't- I never suggested it was.
>
> Ok so we agree. I think our oversaturate thresholds are different,
> though.

I think we simply disagree as to where to put them. See below.

>
> > I'd just suggest that it's asinine to criticise an HBA for running up to
> > reasonable limits when it's the non-toy OS that will do sensible I/O
> > scheduling. So point your gums elsewhere.
>
> Well I don't think 253 is a reasonable limit, that was the whole point.
> How can sane io scheduling ever prevent starvation in that case? I can't
> point my gums elsewhere, this is where I'm seeing starvation.

You're in starvation because the I/O midlayer and buffer cache are
allowing you to build enough transactions on one bus to impact system
response times. This is an old problem with Linux that comes and goes
(as it has come and gone for most systems). There are a number of
possible solutions to this problem- but because this is in 2.4 perhaps
the most sensible one is to limit how much you *ask* from an HBA,
perhaps based upon even as vague a set of parameters as CPU speed and
available memory divided by the number of <n-scsibus/total spindles).

It's the job of the HBA driver to manage resources on the the HBA and on
the bus the HBA interfaces to. If the HBA and its driver can efficiently
manage 1000 concurrent commands per lun and 16384 luns per target and
500 'targets' in a fabric, let it.

Let oversaturation of a *spindle* be informed by device quirks and the
rate of QFULLs received, or even, if you will, by finding the knee in
the per-command latency curve (if you can and you think that it's
meaningful). Let oversaturation of the system be done elsewhere- let the
buffer cache manager and system policy knobs decide whether the fact
that the AIC driver is so busy moving I/O that the user can't get window
focus onto the window in N-P complete time to kill the runaway tar.

Sorry- an overlong response. It *is* easier to just say "well, 'fix' the
HBA driver so it doesn't allow the system to get too busy or
overloaded". But it seems to me that this is even best solved in the
midlayer which should, in fact, know best (better than a single HBA).

-matt

2002-09-27 10:20:03

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

On Fri, Sep 27 2002, Matthew Jacob wrote:
> > > I'd just suggest that it's asinine to criticise an HBA for running up to
> > > reasonable limits when it's the non-toy OS that will do sensible I/O
> > > scheduling. So point your gums elsewhere.
> >
> > Well I don't think 253 is a reasonable limit, that was the whole point.
> > How can sane io scheduling ever prevent starvation in that case? I can't
> > point my gums elsewhere, this is where I'm seeing starvation.
>
> You're in starvation because the I/O midlayer and buffer cache are
> allowing you to build enough transactions on one bus to impact system
> response times. This is an old problem with Linux that comes and goes

That's one type of starvation, but that's one that can be easily
prevented by the os. Not a problem.

The starvation I'm talking about is the drive starving requests. Just
keep it moderately busy (10-30 outstanding tags), and a read can take a
long time to complete. The hba can try to prevent this from happening by
issuing ever Xth request as on ordered tag, however the fact that we
even need to consider doing this suggests to me that something is
broken. That a drive can starve a single request for that long is _bad_.
Issuing every 1024th request as ordered helps absolutely zip for good
interactive behaviour. It might help on a single request basis, but good
latency feel typically requires more than that.

We _can_ try and prevent this type of starvation. If we encounter a
read, don't queue any more writes to the drive before it completes.
That's some simple logic that will probably help a lot. This is the type
of thing the deadline io scheduler tries to do. This is working around
broken drive firmware in my oppinion, the drive shouldn't be starving
requests like that.

However, it's stupid to try and work around a problem if we can simply
prevent the problem in the first place. What is the problem? It's lots
of tags causing the drive to starve requests. Do we need lots of tags?
In my experience 4 tags is more than plenty for a typical drive, there's
simply _no_ benefit from going beyond that. It doesn't buy you extra
throughput, it doesn't buy you better io scheduling (au contraire). All
it gets you is lots of extra latency. So why would I want lots of tags
on a typical scsi drive? I don't.

> (as it has come and gone for most systems). There are a number of
> possible solutions to this problem- but because this is in 2.4 perhaps
> the most sensible one is to limit how much you *ask* from an HBA,
> perhaps based upon even as vague a set of parameters as CPU speed and
> available memory divided by the number of <n-scsibus/total spindles).

This doesn't make much sense to me. Why would the CPU speed and
available memory impact this at all? We don't want to deplete system
resources (the case Justin mentioned), of course, but beyond that I
don't think it makes much sense.

> It's the job of the HBA driver to manage resources on the the HBA and on
> the bus the HBA interfaces to. If the HBA and its driver can efficiently
> manage 1000 concurrent commands per lun and 16384 luns per target and
> 500 'targets' in a fabric, let it.

Yes, if the hba and its drive _and_ the target can _efficiently_ handle
it, I'm all for it. Again you seem to be comparing the typical scsi hard
drive to more esoteric devices. I'll note again that I'm not talking
about such devices.

> Let oversaturation of a *spindle* be informed by device quirks and the
> rate of QFULLs received, or even, if you will, by finding the knee in

If device quirks are that 90% (pulling this number out of my ass) of
scsi drives use pure internal sptf scheduling and thus heavily starve
requests, then why bother? Queue full contains no latency information.

> the per-command latency curve (if you can and you think that it's

I can try to get a decent default. 253 clearly isn't it, far from it.

> meaningful). Let oversaturation of the system be done elsewhere- let the
> buffer cache manager and system policy knobs decide whether the fact
> that the AIC driver is so busy moving I/O that the user can't get window
> focus onto the window in N-P complete time to kill the runaway tar.

This is not a problem with the vm flooding a spindle. We want it to be
flooded, the more we can shove into the io scheduler to work with, the
better chance it has of doing a good job.

> Sorry- an overlong response. It *is* easier to just say "well, 'fix' the
> HBA driver so it doesn't allow the system to get too busy or
> overloaded". But it seems to me that this is even best solved in the
> midlayer which should, in fact, know best (better than a single HBA).

Agrh. Who's saying 'fix' the hba driver? Either I'm not expressing
myself very clearly, or you are simply not reading what I write.

--
Jens Axboe

2002-09-27 12:13:17

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

[ .. all sorts of nice discussion, but not on our argument point ]
>
> Agrh. Who's saying 'fix' the hba driver? Either I'm not expressing
> myself very clearly, or you are simply not reading what I write.

I (foolishly) leapt in when you said "253 is 'over the top'". You seemed
to imply that the aic7xxx driver was at fault and should be limiting the
amount it is sending out. My (mostly) only beef with what you've written
is with that implication- mainly as "don't send so many damned commands
if you think they're too many". If the finger pointing at aic7xx is not
what you're implying, then this has been a waste of email bandwidth-
sorry.

-matt

2002-09-27 12:23:31

by Pedro M. Rodrigues

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

Justin T. Gibbs wrote:
>> I reported this same problem some weeks ago -
>>http://marc.theaimsgroup.com/?l=linux-kernel&m=103069116227685&w=2 .
>>2.4.20pre kernels solved the error messages flooding the console, and
>>improved things a bit, but system load still got very high and disk read
>>and write performance was lousy. Adding more memory and using a
>>completely different machine didn't help. What did? Changing the Adaptec
>>scsi driver to aic7xxx_old . The performance was up 50% for writes and
>>90% for reads, and the system load was acceptable. And i didn't even had
>>to change the RedHat kernel (2.4.18-10) for a custom one. The storage was
>>two external Arena raid boxes, btw.
>
>
> I would be interested in knowing if reducing the maximum tag depth for
> the driver improves things for you. There is a large difference in the
> defaults between the two drivers. It has only reacently come to my
> attention that the SCSI layer per-transaction overhead is so high that
> you can completely starve the kernel of resources if this setting is too
> high. For example, a 4GB system installing RedHat 7.3 could not even
> complete an install on a 20 drive system with the default of 253 commands.
> The latest version of the aic7xxx driver already sent to Marcelo drops the
> default to 32.
>
> --
> Justin
>
>
>

I have a server available to test it, but the storage in question is
already deployed. Yet, by luck (irony apart) i have a maintenance window
this weekend for tuning and other matters, i can decrease the maximum
number of TCQ commands per device in the proper aic7xxx driver to 32 and
report on the results. While trying to solve this problem i browsed
RedHat's bugzilla, and there were several people burned with this
problem. Hope this sorts it out for them.

/Pedro

2002-09-27 12:50:07

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

On Fri, Sep 27 2002, Matthew Jacob wrote:
>
> [ .. all sorts of nice discussion, but not on our argument point ]
> >
> > Agrh. Who's saying 'fix' the hba driver? Either I'm not expressing
> > myself very clearly, or you are simply not reading what I write.
>
> I (foolishly) leapt in when you said "253 is 'over the top'". You seemed
> to imply that the aic7xxx driver was at fault and should be limiting the
> amount it is sending out. My (mostly) only beef with what you've written
> is with that implication- mainly as "don't send so many damned commands
> if you think they're too many". If the finger pointing at aic7xx is not
> what you're implying, then this has been a waste of email bandwidth-
> sorry.

It's not aimed at any specific hba driver, it could be any. 253 would be
over the top for any of them, it just so happens that aic7xxx has this
as the default :-)

So while it is definitely not the aic7xxx driver doing the starvation
(it's the device), the aic7xxx driver is (_in my oppinion) somewhat at
fault for setting it so high _as a default_.

Hopefully that's the end of this thread :)

--
Jens Axboe

2002-09-27 13:45:32

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

> The starvation I'm talking about is the drive starving requests. Just
> keep it moderately busy (10-30 outstanding tags), and a read can take a
> long time to complete.

As I tried to explain to Andrew just the other day, this is neither a
drive nor HBA problem. You've essentially constructed a benchmark where
a single process can get so far ahead of the I/O subsystem in terms of
buffered writes that there is no choice but for there to be a long delay
for the device to handle your read. Consider that because you are queuing,
the drive will completely fill its cache with write data that is pending
to hit the platters. The number of transactions in the cache is marginally
dependant on the number of tags in use since that will affect the ability
of the controller to saturate the drive cache with write data. Depending
on your drive, mode page settings, etc, the drive may allow your read to
pass the write, but many do not. So you have to wait for the cache to
at least have space to handle your read and perhaps have even additional
write data flush before your read can even be started. If you don't like
this behavior, which actually maximizes the throughput of the device, have
the I/O scheduler hold back a single processes from creating such a large
backlog. You can also read the SCSI spec and tune your disk to behave
differently.

Now consider the read case. I maintain that any reasonable drive will
*always* outperform the OS's transaction reordering/elevator algorithms
for seek reduction. This is the whole point of having high tag depths.
In all I/O studies that have been performed todate, reads far outnumber
writes *unless* you are creating an ISO image on your disk. In my opinion
it is much more important to optimize for the more common, concurrent
read case, than it is for the sequential write case with intermittent
reads. Of course, you can fix the latter case too without any change to
the driver's queue depth as outlined above. Why not have your cake and
eat it too?

--
Justin

2002-09-27 14:25:37

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

On Fri, Sep 27 2002, Justin T. Gibbs wrote:
> > The starvation I'm talking about is the drive starving requests. Just
> > keep it moderately busy (10-30 outstanding tags), and a read can take a
> > long time to complete.
>
> As I tried to explain to Andrew just the other day, this is neither a
> drive nor HBA problem. You've essentially constructed a benchmark where
> a single process can get so far ahead of the I/O subsystem in terms of
> buffered writes that there is no choice but for there to be a long delay
> for the device to handle your read. Consider that because you are queuing,
> the drive will completely fill its cache with write data that is pending
> to hit the platters. The number of transactions in the cache is marginally
> dependant on the number of tags in use since that will affect the ability
> of the controller to saturate the drive cache with write data. Depending
> on your drive, mode page settings, etc, the drive may allow your read to
> pass the write, but many do not. So you have to wait for the cache to
> at least have space to handle your read and perhaps have even additional
> write data flush before your read can even be started. If you don't like
> this behavior, which actually maximizes the throughput of the device, have
> the I/O scheduler hold back a single processes from creating such a large
> backlog. You can also read the SCSI spec and tune your disk to behave
> differently.

If the above is what has been observed in the real world, then there
would be no problem. Lets say I have 32 tags pending, all writes. Now I
issue a read. Then I go ahead and through my writes at the drive,
basically keeping it at 32 tags all the time. When will this read
complete? The answer is, well it might not within any reasonable time,
because the drive happily starves the read to get the best write
throughput.

The size of the dirty cache back log, or whatever you want to call it,
does not matter _at all_. I don't know why both you and Matt keep
bringing that point up. The 'back log' is just that, it will be
processed in due time. If a read comes in, the io scheduler will decide
it's the most important thing on earth. So I may have 1 gig of dirty
cache waiting to be flushed to disk, that _does not_ mean that the read
that now comes in has to wait for the 1 gig to be flushed first.

> Now consider the read case. I maintain that any reasonable drive will
> *always* outperform the OS's transaction reordering/elevator algorithms
> for seek reduction. This is the whole point of having high tag depths.

Well given that the drive has intimate knowledge of itself, then yes of
course it is the only one that can order any number of pending requests
most optimally. So the drive might provide the best layout of requests
when it comes to total number of seek time spent, and throughput. But
often at the cost of increased (some times much, see the trivial
examples given) latency.

However, I maintain that going beyond any reasonable number of tags for
a standard drive is *stupid*. The Linux io scheduler gets very good
performance without any queueing at all. Going from 4 to 64 tags gets
you very very little increase in performance, if any at all.

> In all I/O studies that have been performed todate, reads far outnumber
> writes *unless* you are creating an ISO image on your disk. In my opinion

Well it's my experience that it's pretty balanced, at least for my own
workload. atime updates and compiles etc put a nice load on writes.

> it is much more important to optimize for the more common, concurrent
> read case, than it is for the sequential write case with intermittent
> reads. Of course, you can fix the latter case too without any change to
> the driver's queue depth as outlined above. Why not have your cake and
> eat it too?

If you care to show me this cake, I'd be happy to devour it. I see
nothing even resembling a solution to this problem in your email, except
from you above saying I should ignore it and optimize for 'the common'
concurrent read case.

It's pointless to argue that tagging is oh so great and always
outperforms the os io scheduler, and that we should just use 253 tags
because the drive knwos best, when several examples have shown that this
is _not the case_.

--
Jens Axboe

2002-09-27 14:52:01

by Rik van Riel

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

On Fri, 27 Sep 2002, Justin T. Gibbs wrote:

> writes *unless* you are creating an ISO image on your disk. In my opinion
> it is much more important to optimize for the more common, concurrent
> read case, than it is for the sequential write case with intermittent
> reads.

You're missing the point. The only reason the reads are
intermittent is that the application can't proceed until
the read is done and the read is being starved by writes.

If the read was serviced immediately, the next read could
get scheduled quickly and they wouldn't be intermittant.

Rik
--
Bravely reimplemented by the knights who say "NIH".

http://www.surriel.com/ http://distro.conectiva.com/

Spamtraps of the month: [email protected] [email protected]

2002-09-27 15:29:28

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

> If you don't like this behavior, which actually maximizes the
> throughput of the device, have the I/O scheduler hold back a single
> processes from creating such a large backlog.

Justin and I are (for once) in 100% agreement.

2002-09-27 15:32:46

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

On Fri, Sep 27 2002, Matthew Jacob wrote:
>
> > If you don't like this behavior, which actually maximizes the
> > throughput of the device, have the I/O scheduler hold back a single
> > processes from creating such a large backlog.
>
>
> Justin and I are (for once) in 100% agreement.

Well Justin and you are both, it seems, missing the point.

I'm now saying for the 3rd time, that there's zero problem in having a
huge dirty cache backlog. This is not the problem, please disregard any
reference to that. Count only the time spent for servicing a read
request, _from when it enters the drive_ and until it completes. IO
scheduler is _not_ involved.

--
Jens Axboe

2002-09-27 17:14:41

[permalink] [raw]

Subject: Re: Warning - running really short on DMA buffers while doing file transfers

> If the above is what has been observed in the real world, then there
> would be no problem. Lets say I have 32 tags pending, all writes. Now I
> issue a read. Then I go ahead and through my writes at the drive,
> basically keeping it at 32 tags all the time. When will this read
> complete? The answer is, well it might not within any reasonable time,
> because the drive happily starves the read to get the best write
> throughput.

Just because you use 32 or 4 or 8 or whatever tags you cannot know the
number of commands still in the drive's cache. Have you disabled
turned off the WCE bit on your drive and retested your latency numbers?

> The size of the dirty cache back log, or whatever you want to call it,
> does not matter _at all_. I don't know why both you and Matt keep
> bringing that point up. The 'back log' is just that, it will be
> processed in due time. If a read comes in, the io scheduler will decide
> it's the most important thing on earth. So I may have 1 gig of dirty
> cache waiting to be flushed to disk, that _does not_ mean that the read
> that now comes in has to wait for the 1 gig to be flushed first.

But it does matter. If single process can fill the drive's or array's
cache with silly write data as well as have all outstanding tags busy
on its writes, you will incur a significant delay. No single process
should be allowed to do that. It doesn't matter that the read becomes
the most important thing on earth to the OS, you can't take back what
you've already issued to the device. Sorry. It doesn't work that
way.

> However, I maintain that going beyond any reasonable number of tags for
> a standard drive is *stupid*. The Linux io scheduler gets very good
> performance without any queueing at all. Going from 4 to 64 tags gets
> you very very little increase in performance, if any at all.

Under what benchmarks? http load? Squid, News, or mail simulations?
All I've seen are examples crafted to prove your point that I don't
think mirror real world workloads.

>> In all I/O studies that have been performed todate, reads far outnumber
>> writes *unless* you are creating an ISO image on your disk. In my
>> opinion
>
> Well it's my experience that it's pretty balanced, at least for my own
> workload. atime updates and compiles etc put a nice load on writes.

These are very differnet than the "benchmark" I've seen used in this
dicussion:

dd if=/dev/zero of=somefile bs=1M &
cat somefile.

Have you actually timed some of your common activities (say a full
build of the Linux kernel w/modules) at different tag depths, with
or without write caching enabled, etc?

> If you care to show me this cake, I'd be happy to devour it. I see
> nothing even resembling a solution to this problem in your email, except
> from you above saying I should ignore it and optimize for 'the common'
> concurrent read case.

Take a look inside True64 (I believe there are a few papers about this)
to see how to use command response times to modulate device workload.
FreeBSD has several algorithms in its VM to prevent a single process
from holding onto too many dirty buffers. FreeBSD, Solaris, True64,
even WindowsNT have effective algorithms for sanely retiring dirty
buffers without saturating the system. All of these algorithms have
been discussed at length in conference papers. You just need to go
do a google search. None of these issues are new and the solutions
are not novel.

> It's pointless to argue that tagging is oh so great and always
> outperforms the os io scheduler, and that we should just use 253 tags
> because the drive knwos best, when several examples have shown that this
> is _not the case_.

You are trying to solve these problems at the wrong level.

--
Justin

2002-09-27 17:15:49