Here is a working swap prefetching patch for 2.6.13. I have resuscitated and
rewritten some early prefetch code Thomas Schlichter did in late 2.5 to
create a configurable kernel thread that reads in swap from ram in reverse
order it was written out. It does this once kswapd has been idle for a minute
(implying no current vm stress). This patch attached below is a rollup of two
patches the current versions of which are here:
http://ck.kolivas.org/patches/swap-prefetch/
These add an exclusive_timer function, and the patch that does the swap
prefetching. I'm posting this rollup to lkml to see what the interest is in
this feature, and for people to test it if they desire. I'm planning on
including it in the next -ck but wanted to gauge general user opinion for
mainline. Note that swapped in pages are kept on backing store (swap),
meaning no further I/O is required if the page needs to swap back out.
Cheers,
Con Kolivas
On Thu, 2005-09-01 at 23:46 +1000, Con Kolivas wrote:
> Here is a working swap prefetching patch for 2.6.13. I have resuscitated and
> rewritten some early prefetch code Thomas Schlichter did in late 2.5 to
> create a configurable kernel thread that reads in swap from ram in reverse
> order it was written out. It does this once kswapd has been idle for a minute
> (implying no current vm stress). This patch attached below is a rollup of two
> patches the current versions of which are here:
>
> http://ck.kolivas.org/patches/swap-prefetch/
>
> These add an exclusive_timer function, and the patch that does the swap
> prefetching. I'm posting this rollup to lkml to see what the interest is in
> this feature, and for people to test it if they desire. I'm planning on
> including it in the next -ck but wanted to gauge general user opinion for
> mainline. Note that swapped in pages are kept on backing store (swap),
> meaning no further I/O is required if the page needs to swap back out.
I would definitely use this if available.
That said, I have often thought it might be good to have something like
pre-writing swap, ie reverse what your patch does.
In other words it'd keep as much of swappable data on disk as possible,
but without removing it from memory. So when it comes time to free up
some memory, the data is already on disk so no performance penalty from
writing it out.
Hopefully something worth thinking about.
-HK
On Fri, 2 Sep 2005 00:18, Hans Kristian Rosbach wrote:
> On Thu, 2005-09-01 at 23:46 +1000, Con Kolivas wrote:
> > Here is a working swap prefetching patch for 2.6.13. I have resuscitated
> > and rewritten some early prefetch code Thomas Schlichter did in late 2.5
> > to create a configurable kernel thread that reads in swap from ram in
> > reverse order it was written out. It does this once kswapd has been idle
> > for a minute (implying no current vm stress). This patch attached below
> > is a rollup of two patches the current versions of which are here:
> >
> > http://ck.kolivas.org/patches/swap-prefetch/
> >
> > These add an exclusive_timer function, and the patch that does the swap
> > prefetching. I'm posting this rollup to lkml to see what the interest is
> > in this feature, and for people to test it if they desire. I'm planning
> > on including it in the next -ck but wanted to gauge general user opinion
> > for mainline. Note that swapped in pages are kept on backing store
> > (swap), meaning no further I/O is required if the page needs to swap back
> > out.
>
> I would definitely use this if available.
Great.
> That said, I have often thought it might be good to have something like
> pre-writing swap, ie reverse what your patch does.
>
> In other words it'd keep as much of swappable data on disk as possible,
> but without removing it from memory. So when it comes time to free up
> some memory, the data is already on disk so no performance penalty from
> writing it out.
>
> Hopefully something worth thinking about.
Actually to some degree this patch does that, albeit only on things that are
swapped out "naturally". Anything that is swapped out and is unnaturally
swapped back in using prefetching is kept on swap, and you often find much
more swap sitting around ready for freeing up ram whenever there is memory
pressure again.
Cheers,
Con
Hi Con!
Am Donnerstag, 1. September 2005 15:46 schrieb Con Kolivas:
> Here is a working swap prefetching patch for 2.6.13. I have resuscitated
> and rewritten some early prefetch code Thomas Schlichter did in late 2.5 to
> create a configurable kernel thread that reads in swap from ram in reverse
> order it was written out. It does this once kswapd has been idle for a
> minute (implying no current vm stress). This patch attached below is a
> rollup of two patches the current versions of which are here:
>
> http://ck.kolivas.org/patches/swap-prefetch/
>
> These add an exclusive_timer function, and the patch that does the swap
> prefetching. I'm posting this rollup to lkml to see what the interest is in
> this feature, and for people to test it if they desire. I'm planning on
> including it in the next -ck but wanted to gauge general user opinion for
> mainline. Note that swapped in pages are kept on backing store (swap),
> meaning no further I/O is required if the page needs to swap back out.
I am (and some of my friends are) still interested in this functionality, so
I'm definitly going to test your improved patch, of course. By the way, I'm
quite happy that you came up with this new version of swap-prefetching,
because I didn't and still don't have the time to develop or maintain it
more...
So thanks for your good work, and keep on helping Linux-Desktop-Users! :-)
Thomas
On Thursday 01 September 2005 14:18, Con Kolivas wrote:
> On Fri, 2 Sep 2005 00:18, Hans Kristian Rosbach wrote:
> > On Thu, 2005-09-01 at 23:46 +1000, Con Kolivas wrote:
> > > Here is a working swap prefetching patch for 2.6.13. I have
> > > resuscitated and rewritten some early prefetch code Thomas Schlichter
> > > did in late 2.5 to create a configurable kernel thread that reads in
> > > swap from ram in reverse order it was written out. It does this once
> > > kswapd has been idle for a minute (implying no current vm stress). This
> > > patch attached below is a rollup of two patches the current versions of
> > > which are here:
> > That said, I have often thought it might be good to have something like
> > pre-writing swap, ie reverse what your patch does.
> >
> > In other words it'd keep as much of swappable data on disk as possible,
> > but without removing it from memory. So when it comes time to free up
> > some memory, the data is already on disk so no performance penalty from
> > writing it out.
both ideas make all the sense to me. I'll give it a try, but in what way can
we test this kind of enhancement? maybe write a small program that starts
fills a good part of swap space and then, after 1min idle, 'watch free -m'
should show free memory decreasing (not counting cache/buffers) with idle
activity. decent?
about the Hans's proposal - it would increase power consumption, because of
increased disk activity. about con's swap prefetch, I'm not so sure...
regards,
pedro venda.
--
Pedro Jo?o Lopes Venda
email: pjvenda < at > arrakis.dhis.org
http://arrakis.dhis.org
On Sat, 3 Sep 2005 01:01, Pedro Venda wrote:
> On Thursday 01 September 2005 14:18, Con Kolivas wrote:
> > On Fri, 2 Sep 2005 00:18, Hans Kristian Rosbach wrote:
> > > On Thu, 2005-09-01 at 23:46 +1000, Con Kolivas wrote:
> > > > Here is a working swap prefetching patch for 2.6.13. I have
> > > > resuscitated and rewritten some early prefetch code Thomas Schlichter
> > > > did in late 2.5 to create a configurable kernel thread that reads in
> > > > swap from ram in reverse order it was written out. It does this once
> > > > kswapd has been idle for a minute (implying no current vm stress).
> > > > This patch attached below is a rollup of two patches the current
> > > > versions of which are here:
> > >
> > > That said, I have often thought it might be good to have something like
> > > pre-writing swap, ie reverse what your patch does.
> > >
> > > In other words it'd keep as much of swappable data on disk as possible,
> > > but without removing it from memory. So when it comes time to free up
> > > some memory, the data is already on disk so no performance penalty from
> > > writing it out.
>
> both ideas make all the sense to me. I'll give it a try, but in what way
> can we test this kind of enhancement? maybe write a small program that
> starts fills a good part of swap space and then, after 1min idle, 'watch
> free -m' should show free memory decreasing (not counting cache/buffers)
> with idle activity. decent?
Yes that about wraps up what it does. It would be even better used in a real
world situation on a machine that has trouble "getting started" after an
overnight updatedb run and has been idle for a while.
> about the Hans's proposal - it would increase power consumption, because of
> increased disk activity. about con's swap prefetch, I'm not so sure...
Depends entirely on workload. Overall I think this increases power consumption
because the disk does tiny little reads and keeps spinning until it has
finished prefetching as much as it can. I was thinking it could be made
configurable and to detect when laptop mode was enabled and so on... if it is
thought that this is desirable of course.
Cheers,
Con
On Friday 02 September 2005 14:10, Con Kolivas wrote:
> Yes that about wraps up what it does. It would be even better used in a
> real world situation on a machine that has trouble "getting started" after
> an overnight updatedb run and has been idle for a while.
thanks. your last comment about updatedb reminds me of my "server"'s kind of
load :-p
> > about the Hans's proposal - it would increase power consumption, because
> > of increased disk activity. about con's swap prefetch, I'm not so sure...
>
> Depends entirely on workload. Overall I think this increases power
> consumption because the disk does tiny little reads and keeps spinning
> until it has finished prefetching as much as it can. I was thinking it
> could be made configurable and to detect when laptop mode was enabled and
> so on... if it is thought that this is desirable of course.
Not counting hard drive spin-down or even standby, I find no reason for
increased power consumption on swap prefetch mechanism, except for page
syncing (swap&mem) and perhaps upon process killing...
Question: Imagine some big app like firefox completely swapped. killing it
implies swapping out the process memory entirely to free allocated pages?
- If it does, then your swap prefetch may not increase power consumption by a
measurable margin, because some of that swapped information was prefetched.
- If it does not, than prefetched swapped pages could be useless if the
applications dies.
I don't know enough about the kernel VM to answer this on my own.
regards,
pedro venda.
--
Pedro Jo?o Lopes Venda
email: pjvenda < at > arrakis.dhis.org
http://arrakis.dhis.org