----- Forwarded message from Iozone <[email protected]> -----
Subject: Adaptive read-ahead V12
From: Iozone <[email protected]>
To: Wu Fengguang <[email protected]>
X-Mailer: Microsoft Outlook Express 6.00.2900.2670
Date: Thu, 25 May 2006 11:44:37 -0500
Wu Fengguang,
I see that Andrew M. is giving you some pushback....
His argument is that the application could do a better job
of scheduling its own read-ahead. ( I've heard this one
before)
My thoughts on this argument would be along the
lines of:
Indeed the application might be able to do a better
job, however expecting, or demanding, the rewrite
of all applications to behave better might be an unreasonable
expectation. Rewriting all the weather codes, all the
structrual analysis codes, all the data-bases, video streaming
and so on, would solve the problem but the timeline for
such an re-codification may approach infinity, while
systems that have the ability to silently increase
performance maintain an advantage in the market.
Note: Many vendors already have sophisticated A.I. read-ahead
algorithms. Examples: Microsoft, Veritas HP, and Isolon.
Linux already does read-ahead that does not require
the application to be involved. It just does it in
a simpler way that could be improved for handling
a wider range of applications. This new technology is
a simple evolutionary step in that direction. Linux read-ahead
and is currently 8 years behind existing technologies in this area.
Isn't it time to catch up a bit ?
Note: You probably will need to reword this before replying
to Andrew, as I tend to be somewhat blunt. :-)
Hint: I took me a while to convince the kernel folks in my
company to accept such a technology. But when the
prototype accelerated one of their key partner's code by
30 % wall clock, and another showed 50X speedup in
file I/O across a wide stripe, that did the trick.
Enjoy,
Don Capps
[...]
----- End forwarded message -----
Wu Fengguang wrote:
> ----- Forwarded message from Iozone <[email protected]> -----
>
> Subject: Adaptive read-ahead V12
> From: Iozone <[email protected]>
> To: Wu Fengguang <[email protected]>
> X-Mailer: Microsoft Outlook Express 6.00.2900.2670
> Date: Thu, 25 May 2006 11:44:37 -0500
>
> Wu Fengguang,
>
> I see that Andrew M. is giving you some pushback....
> His argument is that the application could do a better job
> of scheduling its own read-ahead. ( I've heard this one
> before)
>
> My thoughts on this argument would be along the
> lines of:
>
> Indeed the application might be able to do a better
> job, however expecting, or demanding, the rewrite
> of all applications to behave better might be an unreasonable
> expectation.
A reasonable expectation would be that the application would have a way
to tell the kernel to ignore readahead for a given file, other than
changing the behavior of the kernel as a whole for all processes on the
machine. This smart application could then use aio or some other similar
method to do preread itself.
My personal opinion is that the kernel only does a good job reading
ahead for sequential access, and since that's the common case only a
means of preventing that effort need be provided.
I can think of several ways the kernel might be smarter about the way it
reads (or not) random access, but since I have no way to test them on
applications other than my own I won't belabor them here.
--
Bill Davidsen <[email protected]>
Obscure bug of 2004: BASH BUFFER OVERFLOW - if bash is being run by a
normal user and is setuid root, with the "vi" line edit mode selected,
and the character set is "big5," an off-by-one errors occurs during
wildcard (glob) expansion.
On Wed, May 31, 2006 at 04:17:58PM -0400, Bill Davidsen wrote:
> Wu Fengguang wrote:
> >----- Forwarded message from Iozone <[email protected]> -----
> >
> >Subject: Adaptive read-ahead V12
> >From: Iozone <[email protected]>
> >To: Wu Fengguang <[email protected]>
> >X-Mailer: Microsoft Outlook Express 6.00.2900.2670
> >Date: Thu, 25 May 2006 11:44:37 -0500
> >
> >Wu Fengguang,
> >
> > I see that Andrew M. is giving you some pushback....
> > His argument is that the application could do a better job
> > of scheduling its own read-ahead. ( I've heard this one
> > before)
> >
> > My thoughts on this argument would be along the
> > lines of:
> >
> > Indeed the application might be able to do a better
> > job, however expecting, or demanding, the rewrite
> > of all applications to behave better might be an unreasonable
> > expectation.
>
> A reasonable expectation would be that the application would have a way
> to tell the kernel to ignore readahead for a given file, other than
> changing the behavior of the kernel as a whole for all processes on the
> machine. This smart application could then use aio or some other similar
> method to do preread itself.
Sure. The adaptive readahead works fine with user level readahead via
the readahead() or posix_fadvise() syscall. I.e. if a program do
necessary readahead() calls that can cover all the file pages
requested by the following read() calls, it avoids triggering
unnecessary kernel readaheads.
> My personal opinion is that the kernel only does a good job reading
> ahead for sequential access, and since that's the common case only a
> means of preventing that effort need be provided.
posix_fadvise(fd, ..., POSIX_FADV_RANDOM) can do the trick for a file.
blockdev --setra 0 /dev/hda does so for a device.
Wu