Changelog
=========
V8 2005-11-25
- balance zone aging only in page relaim paths and do it right
- do the aging of slabs in the same way as zones
- add debug code to dump the detailed page reclaim steps
- undo exposing of struct radix_tree_node and uninline related functions
- work better with nfsd
- generalize accelerated context based read-ahead
- account smooth read-ahead aging based on page referenced/activate bits
- avoid divide error in compute_thrashing_threshold()
- more low lantency efforts
- update some comments
- rebase debug actions on debugfs entries instead of magic readahead_ratio values
V7 2005-11-09
- new tunable parameters: readahead_hit_rate/readahead_live_chunk
- support sparse sequential accesses
- delay look-ahead if drive is spinned down in laptop mode
- disable look-ahead for loopback file
- make mandatory thrashing protection more simple and robust
- attempt to improve responsiveness on large read-ahead size
V6 2005-11-01
- cancel look-ahead in laptop mode
- increase read-ahead limit to 0xFFFF pages
V5 2005-10-28
- rewrite context based method to make it clean and robust
- improved accuracy of stateful thrashing threshold estimation
- make page aging equal to the number of code pages scanned
- sort out the thrashing protection logic
- enhanced debug/accounting facilities
V4 2005-10-15
- detect and save live chunks on page reclaim
- support database workload
- support reading backward
- radix tree lookup look-aside cache
V3 2005-10-06
- major code reorganization and documention
- stateful estimation of thrashing-threshold
- context method with accelerated grow up phase
- adaptive look-ahead
- early detection and rescue of pages in danger
- statitics data collection
- synchronized page aging between zones
V2 2005-09-15
- delayed page activation
- look-ahead: towards pipelined read-ahead
V1 2005-09-13
Initial release which features:
o stateless (for now)
o adapts to available memory / read speed
o free of thrashing (in theory)
And handles:
o large number of slow streams (FTP server)
o open/read/close access patterns (NFS server)
o multiple interleaved, sequential streams in one file
(multithread / multimedia / database)
Overview
========
The current read-ahead logic uses an inflexible algorithm with 128KB
VM_MAX_READAHEAD. Less memory leads to thrashing, more memory helps no
throughput. The new logic is simply safer and faster. It makes sure
every single read-ahead request is safe for the current load. Memory
tight systems are expected to benefit a lot: no thrashing any more.
It can also help boost I/O throughput for large memory systems, for
VM_MAX_READAHEAD now defaults to 1MB. The value is no longer tightly
coupled with the thrashing problem, and therefore constrainted by it.
Thanks,
Wu Fengguang
--
Dept. Automation University of Science and Technology of China
El Fri, 25 Nov 2005 23:12:10 +0800,
Wu Fengguang <[email protected]> escribi?:
> Overview
> ========
>
> The current read-ahead logic uses an inflexible algorithm with 128KB
> VM_MAX_READAHEAD. Less memory leads to thrashing, more memory helps no
> throughput. The new logic is simply safer and faster. It makes sure
> every single read-ahead request is safe for the current load. Memory
> tight systems are expected to benefit a lot: no thrashing any more.
> It can also help boost I/O throughput for large memory systems, for
> VM_MAX_READAHEAD now defaults to 1MB. The value is no longer tightly
> coupled with the thrashing problem, and therefore constrainted by it.
Recently, a openoffice hacker wrote in his blog that the kernel was
culprit of applications not starting as fast as in other systems.
Most of the reasons he gave were wrong, but there was a interesting
one: When you start your system, you've lots of free memory. Since
you have lots of memory, he said it was reasonable to expect that
kernel would readahead *heavily* everything it can to fill that
memory as soon as possible (hoping that what you readahead'ed was
part of the kde/gnome/openoffice libraries etc) and go back to the
normal behaviour when your free memory is used by caches etc.
"Teorically" it looks like a nice heuristic for desktops. Does
adaptative readahead does something like this?
On Fri, 2005-11-25 at 16:43 +0100, Diego Calleja wrote:
> Recently, a openoffice hacker wrote in his blog that the kernel was
> culprit of applications not starting as fast as in other systems.
Useless without a link ;-)
Lee
On Fri, Nov 25, 2005 at 04:43:17PM +0100, Diego Calleja wrote:
> Recently, a openoffice hacker wrote in his blog that the kernel was
> culprit of applications not starting as fast as in other systems.
> Most of the reasons he gave were wrong, but there was a interesting
> one: When you start your system, you've lots of free memory. Since
> you have lots of memory, he said it was reasonable to expect that
> kernel would readahead *heavily* everything it can to fill that
> memory as soon as possible (hoping that what you readahead'ed was
> part of the kde/gnome/openoffice libraries etc) and go back to the
> normal behaviour when your free memory is used by caches etc.
> "Teorically" it looks like a nice heuristic for desktops. Does
> adaptative readahead does something like this?
It's interesting ;)
In fact some distributions do have a read-ahead script to preload files on
startup. The readahead system call should be enough for this purpose:
NAME
readahead - perform file readahead into page cache
SYNOPSIS
#include <fcntl.h>
ssize_t readahead(int fd, off64_t *offset, size_t count);
Thanks,
Wu
On Sat, Nov 26, 2005 at 11:17:55AM +0800, Wu Fengguang wrote:
> On Fri, Nov 25, 2005 at 04:43:17PM +0100, Diego Calleja wrote:
> > Recently, a openoffice hacker wrote in his blog that the kernel was
> > culprit of applications not starting as fast as in other systems.
> > Most of the reasons he gave were wrong, but there was a interesting
> > one: When you start your system, you've lots of free memory. Since
> > you have lots of memory, he said it was reasonable to expect that
> > kernel would readahead *heavily* everything it can to fill that
> > memory as soon as possible (hoping that what you readahead'ed was
> > part of the kde/gnome/openoffice libraries etc) and go back to the
> > normal behaviour when your free memory is used by caches etc.
> > "Teorically" it looks like a nice heuristic for desktops. Does
> > adaptative readahead does something like this?
>
> It's interesting ;)
> In fact some distributions do have a read-ahead script to preload files on
> startup. The readahead system call should be enough for this purpose:
>
> NAME
> readahead - perform file readahead into page cache
posix_fadvise() with POSIX_FADV_WILLNEED hint?
The specified data will be accessed in the near future.
--
Tomasz Torcz Morality must always be based on practicality.
[email protected] -- Baron Vladimir Harkonnen
On Sat, Nov 26, 2005 at 02:25:24PM +0100, Tomasz Torcz wrote:
> > It's interesting ;)
> > In fact some distributions do have a read-ahead script to preload files on
> > startup. The readahead system call should be enough for this purpose:
> >
> > NAME
> > readahead - perform file readahead into page cache
>
> posix_fadvise() with POSIX_FADV_WILLNEED hint?
> The specified data will be accessed in the near future.
Nod, this one is better.
They do roughly the same thing, while the latter interface is more portable.
Another note: if you do not know precisely which files to readahead, turn on
aggressive readahead feature temporary might help. There are mainly three
parameters that controls the aggressiveness, the following example values should
be large enough:
echo 16 > /proc/sys/vm/readahead_hit_rate # overlook hit rate?
echo 1000 > /proc/sys/vm/readahead_ratio # grow up ra-size quickly?
blockdev --setra 8192 /dev/had # large ra-size limit?
Thanks,
Wu
On 11/25/05, Lee Revell <[email protected]> wrote:
> On Fri, 2005-11-25 at 16:43 +0100, Diego Calleja wrote:
> > Recently, a openoffice hacker wrote in his blog that the kernel was
> > culprit of applications not starting as fast as in other systems.
>
> Useless without a link ;-)
>
I think Diego refers to Michael Meeks blog:
http://www.gnome.org/~michael/activity.html#2005-11-04
Michael Meek also speaks about kernel issues in an interview in Linux
Format 72 (nov 2005).
Mark
On Sun, 2005-11-27 at 00:03 +0100, Mark van der Made wrote:
> On 11/25/05, Lee Revell <[email protected]> wrote:
> > On Fri, 2005-11-25 at 16:43 +0100, Diego Calleja wrote:
> > > Recently, a openoffice hacker wrote in his blog that the kernel was
> > > culprit of applications not starting as fast as in other systems.
> >
> > Useless without a link ;-)
> >
> I think Diego refers to Michael Meeks blog:
> http://www.gnome.org/~michael/activity.html#2005-11-04
> Michael Meek also speaks about kernel issues in an interview in Linux
> Format 72 (nov 2005).
>
This link tells a much more interesting story:
http://www.gnome.org/~lcolitti/gnome-startup/analysis/
Seems like most of the problems are in userspace at the application
level.
Maybe the excessive seeking when loading libraries could be solved by
physically rearranging the libraries on disk so that the "hot paths"
don't induce as many seeks.
Lee