LinuxLists.cc - [patch][rfc][rft] vm throughput 2.4.2-ac4

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

On Tue, 27 Feb 2001, Mike Galbraith wrote:

> Attempting to avoid doing I/O has been harmful to throughput here
> ever since the queueing/elevator woes were fixed. Ever since then,
> tossing attempts at avoidance has improved throughput markedly.
>
> IMHO, any patch which claims to improve throughput via code deletion
> should be worth a little eyeball time.. and maybe even a test run ;-)
>
> Comments welcome.

Before even thinking about testing this thing, I'd like to
see some (detailed?) explanation from you why exactly you
think the changes in this patch are good and how + why they
work.

IMHO it would be good to not apply ANY code to the stable
kernel tree unless we understand what it does and what the
author meant the code to do...

regards,

Rik
--
Linux MM bugzilla: http://linux-mm.org/bugzilla.shtml

Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/
http://www.conectiva.com/ http://distro.conectiva.com/

2001-02-27 21:29:08

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

On Tue, 27 Feb 2001, Rik van Riel wrote:

> On Tue, 27 Feb 2001, Mike Galbraith wrote:
>
> > Attempting to avoid doing I/O has been harmful to throughput here
> > ever since the queueing/elevator woes were fixed. Ever since then,
> > tossing attempts at avoidance has improved throughput markedly.
> >
> > IMHO, any patch which claims to improve throughput via code deletion
> > should be worth a little eyeball time.. and maybe even a test run ;-)
> >
> > Comments welcome.
>
> Before even thinking about testing this thing, I'd like to
> see some (detailed?) explanation from you why exactly you
> think the changes in this patch are good and how + why they
> work.

Ok.. quite reasonable ;-)

First and foremost: What does refill_inactive_scan do? It places
work to do on a list.. and nothing more. It frees no memory in and
of itself.. none (but we count it as freed.. that's important). It
is the amount of memory we want desperately to free in the immediate
future. We count on it getting freed. The only way to free I/O bound
memory is to do the I/O.. as fast as the I/O subsystem can sync it.

This is the nut.. scan/deactivate percentages are fairly meaningless
unless we do something about these pages.

What the patch does is simply to push I/O as fast as we can.. we're
by definition I/O bound and _can't_ defer it under any circumstance,
for in this direction lies constipation. The only thing in the world
which will make it better is pushing I/O.

If you test the patch, you'll notice one very important thing. The
system no longer over-reacts.. as badly. That's a diagnostic point.
(On my system under my favorite page turnover rate load, I see my box
drowning in a pool of dirty pages.. which it's not allowed to drain)

What we do right now (as kswapd) is scan a tiny portion of the active
page list, and then push an arbitrary amount of swap because we can't
possibly deactivate enough pages if our shortage is larger than the
search area (nr_active_pages >> 6).. repeat until give-up time. In
practice here (test load, but still..), that leads to pushing soon
to be unneeded [supposition!] pages into swap a full 3/4 of the time.

> IMHO it would be good to not apply ANY code to the stable
> kernel tree unless we understand what it does and what the
> author meant the code to do...

Yes.. I agree 100%. I was not suggesting that this be blindly
integrated. (I know me.. can get all cornfoosed and fsck up;)

-Mike

2001-02-27 23:02:25

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

On Tue, 27 Feb 2001, Mike Galbraith wrote:

> On Tue, 27 Feb 2001, Rik van Riel wrote:
>
> > On Tue, 27 Feb 2001, Mike Galbraith wrote:
> >
> > > Attempting to avoid doing I/O has been harmful to throughput here
> > > ever since the queueing/elevator woes were fixed. Ever since then,
> > > tossing attempts at avoidance has improved throughput markedly.
> > >
> > > IMHO, any patch which claims to improve throughput via code deletion
> > > should be worth a little eyeball time.. and maybe even a test run ;-)
> > >
> > > Comments welcome.
> >
> > Before even thinking about testing this thing, I'd like to
> > see some (detailed?) explanation from you why exactly you
> > think the changes in this patch are good and how + why they
> > work.
>
> Ok.. quite reasonable ;-)
>
> First and foremost: What does refill_inactive_scan do? It places
> work to do on a list.. and nothing more. It frees no memory in and
> of itself.. none (but we count it as freed.. that's important). It
> is the amount of memory we want desperately to free in the immediate
> future. We count on it getting freed. The only way to free I/O bound
> memory is to do the I/O.. as fast as the I/O subsystem can sync it.
>
> This is the nut.. scan/deactivate percentages are fairly meaningless
> unless we do something about these pages.
>
> What the patch does is simply to push I/O as fast as we can.. we're
> by definition I/O bound and _can't_ defer it under any circumstance,
> for in this direction lies constipation. The only thing in the world
> which will make it better is pushing I/O.

In your I/O bound case, yes. But not in all cases.

> If you test the patch, you'll notice one very important thing. The
> system no longer over-reacts.. as badly. That's a diagnostic point.
> (On my system under my favorite page turnover rate load, I see my box
> drowning in a pool of dirty pages.. which it's not allowed to drain)
>
> What we do right now (as kswapd) is scan a tiny portion of the active
> page list, and then push an arbitrary amount of swap because we can't
> possibly deactivate enough pages if our shortage is larger than the
> search area (nr_active_pages >> 6).. repeat until give-up time. In
> practice here (test load, but still..), that leads to pushing soon
> to be unneeded [supposition!] pages into swap a full 3/4 of the time.

Have you tried to use SWAP_SHIFT as 4 instead of 5 on a stock 2.4.2-ac5 to
see if the system still swaps out too much?

2001-02-28 05:04:56

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

On Tue, 27 Feb 2001, Marcelo Tosatti wrote:

> On Tue, 27 Feb 2001, Mike Galbraith wrote:
>
> > What the patch does is simply to push I/O as fast as we can.. we're
> > by definition I/O bound and _can't_ defer it under any circumstance,
> > for in this direction lies constipation. The only thing in the world
> > which will make it better is pushing I/O.
>
> In your I/O bound case, yes. But not in all cases.

That's one reason I tossed it out. I don't _think_ it should have any
negative effect on other loads, but a test run might find otherwise.

> > What we do right now (as kswapd) is scan a tiny portion of the active
> > page list, and then push an arbitrary amount of swap because we can't
> > possibly deactivate enough pages if our shortage is larger than the
> > search area (nr_active_pages >> 6).. repeat until give-up time. In
> > practice here (test load, but still..), that leads to pushing soon
> > to be unneeded [supposition!] pages into swap a full 3/4 of the time.

(correction: it's 2/3 of the time not 3/4.. off by one bug in fingers;)

> Have you tried to use SWAP_SHIFT as 4 instead of 5 on a stock 2.4.2-ac5 to
> see if the system still swaps out too much?

Not yet, but will do.

-Mike

2001-02-28 08:11:37

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

> > Have you tried to use SWAP_SHIFT as 4 instead of 5 on a stock 2.4.2-ac5 to
> > see if the system still swaps out too much?
>
> Not yet, but will do.

Didn't help. (It actually reduced throughput a little)

-Mike

2001-02-28 08:45:10

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

On Wed, 28 Feb 2001, Mike Galbraith wrote:

> On Tue, 27 Feb 2001, Marcelo Tosatti wrote:
>
> > On Tue, 27 Feb 2001, Mike Galbraith wrote:
> >
> > > What the patch does is simply to push I/O as fast as we can.. we're
> > > by definition I/O bound and _can't_ defer it under any circumstance,
> > > for in this direction lies constipation. The only thing in the world
> > > which will make it better is pushing I/O.
> >
> > In your I/O bound case, yes. But not in all cases.
>
> That's one reason I tossed it out. I don't _think_ it should have any
> negative effect on other loads, but a test run might find otherwise.

Writes are more expensive than reads. Apart from the aggressive read
caching on the disk, writes have limited caching or no caching at all if
you need security (journalling, for example). (I'm not sure about write
caching details, any harddisk expert?)

On read intensive loads, doing IO to free memory (writing pages out) will
be horribly harmful for these reads (which you can free easily), so its
better to avoid the writes as much as possible.

I remember Matthew Dillon (FreeBSD VM guy) had a read intensive case were
using 20:1 clean/flush ratio to free pages in FreeBSD's launder routine
(at that time, IIRC, their launder routine was looping twice the inactive
dirty list looking for clean pages to throw away, and only on the third
loop it would do IO) was still being a problem for disk performance
because of the writes. Yes, it sounds weird.

I suppose you're running dbench.

2001-02-28 09:11:56

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

On Wed, 28 Feb 2001, Mike Galbraith wrote:

> > > Have you tried to use SWAP_SHIFT as 4 instead of 5 on a stock 2.4.2-ac5 to
> > > see if the system still swaps out too much?
> >
> > Not yet, but will do.

But what about swapping behaviour?

It still swaps too much?

2001-02-28 11:48:58

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

On Wed, 28 Feb 2001, Marcelo Tosatti wrote:

> On Wed, 28 Feb 2001, Mike Galbraith wrote:
>
> > > > Have you tried to use SWAP_SHIFT as 4 instead of 5 on a stock 2.4.2-ac5 to
> > > > see if the system still swaps out too much?
> > >
> > > Not yet, but will do.
>
> But what about swapping behaviour?
>
> It still swaps too much?

Yes.

(returning to study mode)

-Mike

2001-02-28 15:35:55

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

On Wed, 28 Feb 2001, Marcelo Tosatti wrote:
> On Wed, 28 Feb 2001, Mike Galbraith wrote:

> > That's one reason I tossed it out. I don't _think_ it should have any
> > negative effect on other loads, but a test run might find otherwise.
>
> Writes are more expensive than reads. Apart from the aggressive read
> caching on the disk, writes have limited caching or no caching at all if
> you need security (journalling, for example). (I'm not sure about write
> caching details, any harddisk expert?)

I suspect Mike needs to change his benchmark load a little
so that it dirties only 10% of the pages (might be realistic
for web and/or database loads).

At that point, you should be able to see that doing writes
all the time can really mess up read performance due to extra
introduced seeks.

We probably want some in-between solution (like FreeBSD has today).
The first time they see a dirty page, they mark it as seen, the
second time they come across it in the inactive list, they flush it.
This way IO is still delayed a bit and not done if there are enough
clean pages around.

Another solution would be to do some more explicit IO clustering and
only flush _large_ clusters ... no need to invoke extra disk seeks
just to free a single page, unless you only have single pages left.

regards,

Rik
--
Linux MM bugzilla: http://linux-mm.org/bugzilla.shtml

Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/
http://www.conectiva.com/ http://distro.conectiva.com/

2001-03-01 04:15:16

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

On Wed, 28 Feb 2001, Rik van Riel wrote:

> On Wed, 28 Feb 2001, Marcelo Tosatti wrote:
> > On Wed, 28 Feb 2001, Mike Galbraith wrote:
>
> > > That's one reason I tossed it out. I don't _think_ it should have any
> > > negative effect on other loads, but a test run might find otherwise.
> >
> > Writes are more expensive than reads. Apart from the aggressive read
> > caching on the disk, writes have limited caching or no caching at all if
> > you need security (journalling, for example). (I'm not sure about write
> > caching details, any harddisk expert?)
>
> I suspect Mike needs to change his benchmark load a little
> so that it dirties only 10% of the pages (might be realistic
> for web and/or database loads).

Asking the user to not dirty so many pages is wrong. My benchmark
load is many compute intensive tasks which each dirty a few pages
while doing real work. It would be unrealistic if it just dirtied
pages as fast as possible to intentionally jam up the vm, but it
doesn't do that.

> At that point, you should be able to see that doing writes
> all the time can really mess up read performance due to extra
> introduced seeks.

The fact that writes are painful doesn't change the fact that data
must be written in order to free memory and proceed. Besides, the
elevator is supposed to solve that not the allocator.. or?

> We probably want some in-between solution (like FreeBSD has today).
> The first time they see a dirty page, they mark it as seen, the
> second time they come across it in the inactive list, they flush it.
> This way IO is still delayed a bit and not done if there are enough
> clean pages around.

(delayed write is fine, but I'll be upset if vmlinux doesn't show up
after I buy more ram;)

> Another solution would be to do some more explicit IO clustering and
> only flush _large_ clusters ... no need to invoke extra disk seeks
> just to free a single page, unless you only have single pages left.

This sounds good.. except I keep thinking about the elevator. Clusters
disappear as soon as they hit the queues so clustering at the vm level
doesn't make any sense to me. Where pages actually land is a function
of the fs, and that gets torn down even further by the elevator. If
you submit pages one at a time, the plug will build clusters for you.

I don't think that the vm has the information needed to make decisions
like this nor the responsibility to do so. It's a customer of the I/O
layers beneath it.

-Mike

2001-03-01 15:37:40

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

On Thu, 1 Mar 2001, Mike Galbraith wrote:
> On Wed, 28 Feb 2001, Rik van Riel wrote:
> > On Wed, 28 Feb 2001, Marcelo Tosatti wrote:
> > > On Wed, 28 Feb 2001, Mike Galbraith wrote:
> >
> > > > That's one reason I tossed it out. I don't _think_ it should have any
> > > > negative effect on other loads, but a test run might find otherwise.
> > >
> > > Writes are more expensive than reads. Apart from the aggressive read
> > > caching on the disk, writes have limited caching or no caching at all if
> > > you need security (journalling, for example). (I'm not sure about write
> > > caching details, any harddisk expert?)
> >
> > I suspect Mike needs to change his benchmark load a little
> > so that it dirties only 10% of the pages (might be realistic
> > for web and/or database loads).
>
> Asking the user to not dirty so many pages is wrong. My benchmark
> load is many compute intensive tasks which each dirty a few pages
> while doing real work. It would be unrealistic if it just dirtied
> pages as fast as possible to intentionally jam up the vm, but it
> doesn't do that.

Asking you to test a different kind of workload is wrong ??

The kind of load I described _is_ realistic, think for example
about ftp/www/MySQL servers...

> > At that point, you should be able to see that doing writes
> > all the time can really mess up read performance due to extra
> > introduced seeks.
>
> The fact that writes are painful doesn't change the fact that data
> must be written in order to free memory and proceed. Besides, the
> elevator is supposed to solve that not the allocator.. or?

But if the amount of dirtied pages is _small_, it means that we can
allow the reads to continue uninterrupted for a while before we
flush all dirty pages in one go...

Also, the elevator can only try to optimise whatever you throw at
it. If you throw random requests at the elevator, you cannot expect
it to do ANY GOOD ...

The merging at the elevator level only works if the requests sent to
it are right next to each other on disk. This means that randomly
sending stuff to disk really DOES DESTROY PERFORMANCE and there's
nothing the elevator could ever hope to do about that.

> > We probably want some in-between solution (like FreeBSD has today).
> > The first time they see a dirty page, they mark it as seen, the
> > second time they come across it in the inactive list, they flush it.
> > This way IO is still delayed a bit and not done if there are enough
> > clean pages around.
>
> (delayed write is fine, but I'll be upset if vmlinux doesn't show up
> after I buy more ram;)

Writing out of old data is a task independent of the VM. This is a
job done by kupdate. The only thing the VM does is write pages out
earlier when it's under memory pressure.

> > Another solution would be to do some more explicit IO clustering and
> > only flush _large_ clusters ... no need to invoke extra disk seeks
> > just to free a single page, unless you only have single pages left.
>
> This sounds good.. except I keep thinking about the elevator.
> Clusters disappear as soon as they hit the queues so clustering
> at the vm level doesn't make any sense to me.

You should think about the elevator a bit more. Feel for the poor
thing and try to send it requests it can actually do something
useful with ;)

regards,

Rik
--
Linux MM bugzilla: http://linux-mm.org/bugzilla.shtml

Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/
http://www.conectiva.com/ http://distro.conectiva.com/

2001-03-01 20:35:58

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

On Thu, 1 Mar 2001, Rik van Riel wrote:

> On Thu, 1 Mar 2001, Mike Galbraith wrote:
> > On Wed, 28 Feb 2001, Rik van Riel wrote:
> > > On Wed, 28 Feb 2001, Marcelo Tosatti wrote:
> > > > On Wed, 28 Feb 2001, Mike Galbraith wrote:
> > >
> > > > > That's one reason I tossed it out. I don't _think_ it should have any
> > > > > negative effect on other loads, but a test run might find otherwise.
> > > >
> > > > Writes are more expensive than reads. Apart from the aggressive read
> > > > caching on the disk, writes have limited caching or no caching at all if
> > > > you need security (journalling, for example). (I'm not sure about write
> > > > caching details, any harddisk expert?)
> > >
> > > I suspect Mike needs to change his benchmark load a little
> > > so that it dirties only 10% of the pages (might be realistic
> > > for web and/or database loads).
> >
> > Asking the user to not dirty so many pages is wrong. My benchmark
> > load is many compute intensive tasks which each dirty a few pages
> > while doing real work. It would be unrealistic if it just dirtied
> > pages as fast as possible to intentionally jam up the vm, but it
> > doesn't do that.
>
> Asking you to test a different kind of workload is wrong ??

No no no and again no (perhaps I misread that bit). But otoh, you
haven't tested the patch I sent in good faith. I sent it because I
have thought about it. I may be wrong in my interpretation of the
results, but those results were thought about.. and they exist.

> The kind of load I described _is_ realistic, think for example
> about ftp/www/MySQL servers...

Yes. My favorite test load is also realistic.

> > > At that point, you should be able to see that doing writes
> > > all the time can really mess up read performance due to extra
> > > introduced seeks.
> >
> > The fact that writes are painful doesn't change the fact that data
> > must be written in order to free memory and proceed. Besides, the
> > elevator is supposed to solve that not the allocator.. or?
>
> But if the amount of dirtied pages is _small_, it means that we can
> allow the reads to continue uninterrupted for a while before we
> flush all dirty pages in one go...

"If wishes were horses, beggers would ride."

There is no mechanysm in place that ensures that dirty pages can't
get out of control, and they do in fact get out of control, and it
is exaserbated (mho) by attempting to define 'too much I/O' without
any information to base this definition upon.

> Also, the elevator can only try to optimise whatever you throw at
> it. If you throw random requests at the elevator, you cannot expect
> it to do ANY GOOD ...

This is a very good point (which I will think upon). I ask you this
in return. Why do you think that the random junk you throw at the
elevator is different than the random junk I throw at it? ;-) I see
no difference at all.. it's the same exact junk. (it's junk because
neither of us knows that it will be optimizable.. it really is a
random bunch of pages because we have ZERO information concerning
the origins, destinations nor informational content of the pages we're
pushing. We have no interest [only because we aren't clever enough
to be interested] in these things.)

> The merging at the elevator level only works if the requests sent to
> it are right next to each other on disk. This means that randomly
> sending stuff to disk really DOES DESTROY PERFORMANCE and there's
> nothing the elevator could ever hope to do about that.

True to some (very real) extent because of the limited buffering of
requests. However, I can not find any useful information that the
vm is using to guarantee the IT does not destroy performance by your
own definition. If it's there and I'm just missing it, I'd thank
you heartily if you'd hit me up side the head with a clue-x-4 ;-)

> > > We probably want some in-between solution (like FreeBSD has today).
> > > The first time they see a dirty page, they mark it as seen, the
> > > second time they come across it in the inactive list, they flush it.
> > > This way IO is still delayed a bit and not done if there are enough
> > > clean pages around.
> >
> > (delayed write is fine, but I'll be upset if vmlinux doesn't show up
> > after I buy more ram;)
>
> Writing out of old data is a task independent of the VM. This is a
> job done by kupdate. The only thing the VM does is write pages out
> earlier when it's under memory pressure.

I was joking.

> > > Another solution would be to do some more explicit IO clustering and
> > > only flush _large_ clusters ... no need to invoke extra disk seeks
> > > just to free a single page, unless you only have single pages left.
> >
> > This sounds good.. except I keep thinking about the elevator.
> > Clusters disappear as soon as they hit the queues so clustering
> > at the vm level doesn't make any sense to me.
>
> You should think about the elevator a bit more. Feel for the poor
> thing and try to send it requests it can actually do something
> useful with ;)

I will, and I hope you can help me out with a little more food for
thought.

-Mike

2001-03-01 21:08:02

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

On Thu, 1 Mar 2001, Mike Galbraith wrote:
> On Thu, 1 Mar 2001, Rik van Riel wrote:
> > On Thu, 1 Mar 2001, Mike Galbraith wrote:

> No no no and again no (perhaps I misread that bit). But otoh,
> you haven't tested the patch I sent in good faith. I sent it
> because I have thought about it. I may be wrong in my
> interpretation of the results, but those results were thought
> about.. and they exist.

I haven't tested it yet for a number of reasons. The most
important one is that the FreeBSD people have been playing
with this thing for a few years now and Matt Dillon has
told me the result of their tests ;)

> > But if the amount of dirtied pages is _small_, it means that we can
> > allow the reads to continue uninterrupted for a while before we
> > flush all dirty pages in one go...
>
> "If wishes were horses, beggers would ride."
>
> There is no mechanysm in place that ensures that dirty pages
> can't get out of control, and they do in fact get out of
> control, and it is exaserbated (mho) by attempting to define
> 'too much I/O' without any information to base this definition
> upon.

True. I think we want something in-between our ideas...

> > Also, the elevator can only try to optimise whatever you throw at
> > it. If you throw random requests at the elevator, you cannot expect
> > it to do ANY GOOD ...
>
> This is a very good point (which I will think upon). I ask you this
> in return. Why do you think that the random junk you throw at the
> elevator is different than the random junk I throw at it? ;-) I see
> no difference at all.. it's the same exact junk.

Except that your code throws the random junk at the elevator all
the time, while my code only bothers the elevator every once in
a while. This should make it possible for the disk reads to
continue with less interruptions.

> > The merging at the elevator level only works if the requests sent to
> > it are right next to each other on disk. This means that randomly
> > sending stuff to disk really DOES DESTROY PERFORMANCE and there's
> > nothing the elevator could ever hope to do about that.
>
> True to some (very real) extent because of the limited buffering
> of requests. However, I can not find any useful information
> that the vm is using to guarantee the IT does not destroy
> performance by your own definition.

Indeed. IMHO we should fix this by putting explicit IO
clustering in the ->writepage() functions.

Doing this, in combination with *WAITING* for dirty pages
to accumulate on the inactive list will give us the
possibility to do more writeout of dirty data with less
disk seeks (and less slowdown of the reads).

regards,

Rik
--
Linux MM bugzilla: http://linux-mm.org/bugzilla.shtml

Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/
http://www.conectiva.com/ http://distro.conectiva.com/

2001-03-01 22:04:47

by Rajagopal Ananthanarayanan

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

Rik van Riel wrote:

[ ... ]

> Except that your code throws the random junk at the elevator all
> the time, while my code only bothers the elevator every once in
> a while. This should make it possible for the disk reads to
> continue with less interruptions.
>

Couldn't agree with you more. The elevator does a decent job
these days, but higher level clustering could do more ...

[ ...]

> Indeed. IMHO we should fix this by putting explicit IO
> clustering in the ->writepage() functions.

Enhancing writepage() to perform clustering is the first step.
In addition you want entities (kupdated, kswapd, et. al)
that currently work with only buffers to invoke writepage()
at appropriate points. Just today I sent a patch that does this
and also combines delayed allocation out to Al Viro for comments.
If anyone else is interested I can send it out to the list.

ananth.

--------------------------------------------------------------------------
Rajagopal Ananthanarayanan ("ananth")
Member Technical Staff, SGI.
--------------------------------------------------------------------------

2001-03-01 22:25:31

by Alan

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

> There is no mechanysm in place that ensures that dirty pages can't
> get out of control, and they do in fact get out of control, and it
> is exaserbated (mho) by attempting to define 'too much I/O' without
> any information to base this definition upon.

I think this is a good point. If you do 'too much I/O' then the I/O gets
throttled by submit_bh(). The block I/O layer knows about 'too much I/O'.

2001-03-01 22:33:42

by Alan

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

> Except that your code throws the random junk at the elevator all
> the time, while my code only bothers the elevator every once in
> a while. This should make it possible for the disk reads to
> continue with less interruptions.

Think about it this way, throwing the stuff at the I/O layer is saying
'please make this go away'. Thats the VM decision. Scheduling the I/O is an
I/O and driver layer decision.

2001-03-01 22:35:51

by Chris Evans

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

On Thu, 1 Mar 2001, Rik van Riel wrote:

> True. I think we want something in-between our ideas...
^^^^^^^
> a while. This should make it possible for the disk reads to
^^^^^^

Oh dear.. not more "vm design by waving hands in the air". Come on people,
improve the vm by careful profiling, tweaking and benching, not by
throwing random patches in that seem cool in theory.

Cheers
Chris

2001-03-01 22:38:21

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

On Thu, 1 Mar 2001, Chris Evans wrote:

>
> On Thu, 1 Mar 2001, Rik van Riel wrote:
>
> > True. I think we want something in-between our ideas...
> ^^^^^^^
> > a while. This should make it possible for the disk reads to
> ^^^^^^
>
> Oh dear.. not more "vm design by waving hands in the air". Come on people,
> improve the vm by careful profiling, tweaking and benching, not by
> throwing random patches in that seem cool in theory.

OTOH, "careful profiling, tweaking and benching" are always limited to a
number workloads.

2001-03-01 22:39:21

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

On Thu, 1 Mar 2001, Chris Evans wrote:
> On Thu, 1 Mar 2001, Rik van Riel wrote:
>
> > True. I think we want something in-between our ideas...
> ^^^^^^^
> > a while. This should make it possible for the disk reads to
> ^^^^^^
>
> Oh dear.. not more "vm design by waving hands in the air". Come
> on people, improve the vm by careful profiling, tweaking and
> benching, not by throwing random patches in that seem cool in
> theory.

Actually, this was more of "vm design by looking at what
the FreeBSD folks did, why it didn't work and how they
fixed it after 2 years of testing various things".

Rik
--
Linux MM bugzilla: http://linux-mm.org/bugzilla.shtml

Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/
http://www.conectiva.com/ http://distro.conectiva.com/

2001-03-02 03:00:44

by Linus Torvalds

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

In article <[email protected]>,
Rik van Riel <[email protected]> wrote:
>
>I haven't tested it yet for a number of reasons. The most
>important one is that the FreeBSD people have been playing
>with this thing for a few years now and Matt Dillon has
>told me the result of their tests ;)

Note that the Linux VM is certainly different enough that I doubt the
comparisons are all that valid. Especially actual virtual memory mapping
is basically from another planet altogether, and heuristics that are
appropriate for *BSD may not really translate all that better.

I'll take numbers over talk any day. At least Mike had numbers, and
possible explanations for them. He also removed more code than he added,
which is always a good sign.

In short, please don't argue against numbers.

Linus

2001-03-02 04:41:17

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

On Thu, 1 Mar 2001, Chris Evans wrote:

> Oh dear.. not more "vm design by waving hands in the air". Come on people,
> improve the vm by careful profiling, tweaking and benching, not by
> throwing random patches in that seem cool in theory.

Excuse me.. we're trying to have a _constructive_ conversation here.

-Mike

2001-03-02 05:19:20

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

On Thu, 1 Mar 2001, Rik van Riel wrote:

> > > The merging at the elevator level only works if the requests sent to
> > > it are right next to each other on disk. This means that randomly
> > > sending stuff to disk really DOES DESTROY PERFORMANCE and there's
> > > nothing the elevator could ever hope to do about that.
> >
> > True to some (very real) extent because of the limited buffering
> > of requests. However, I can not find any useful information
> > that the vm is using to guarantee the IT does not destroy
> > performance by your own definition.
>
> Indeed. IMHO we should fix this by putting explicit IO
> clustering in the ->writepage() functions.

I notice there's a patch sitting in my mailbox.. think I'll go read
it and think (grunt grunt;) about this issue some more.

Thanks for the input Rik. I appreciate it.

-Mike

2001-03-02 16:12:31

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

On 1 Mar 2001, Linus Torvalds wrote:
> In article <[email protected]>,
> Rik van Riel <[email protected]> wrote:
> >
> >I haven't tested it yet for a number of reasons. The most
> >important one is that the FreeBSD people have been playing
> >with this thing for a few years now and Matt Dillon has
> >told me the result of their tests ;)
>
> Note that the Linux VM is certainly different enough that I
> doubt the comparisons are all that valid. Especially actual
> virtual memory mapping is basically from another planet
> altogether, and heuristics that are appropriate for *BSD may not
> really translate all that better.

The main difference is that under Linux the size of the
inactive list is dynamic, while under FreeBSD the system
always tries to keep a (very) large inactive list around.

I'm not sure if, or how, this would influence the percentage
of dirty pages on the inactive list or how often we'd need to
flush something to disk as opposed to reclaiming clean pages.

> I'll take numbers over talk any day. At least Mike had numbers,

The only number I saw when reading over this thread was that
Mike found that under one workload he tested the Linux kernel
ended up doing IO anyway about 2/3rds of the time.

This would also mean we'd be able to _avoid_ IO 1/3rd of the
time ;)

> In short, please don't argue against numbers.

I'm not arguing against his numbers, all I want to know is
if the patch has the same positive effect on other workloads
as well...

regards,

Rik
--
Linux MM bugzilla: http://linux-mm.org/bugzilla.shtml

Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/
http://www.conectiva.com/ http://distro.conectiva.com/

2001-03-07 00:08:09

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

On Fri, 2 Mar 2001, Mike Galbraith wrote:

> On Thu, 1 Mar 2001, Rik van Riel wrote:
>
> > > > The merging at the elevator level only works if the requests sent to
> > > > it are right next to each other on disk. This means that randomly
> > > > sending stuff to disk really DOES DESTROY PERFORMANCE and there's
> > > > nothing the elevator could ever hope to do about that.
> > >
> > > True to some (very real) extent because of the limited buffering
> > > of requests. However, I can not find any useful information
> > > that the vm is using to guarantee the IT does not destroy
> > > performance by your own definition.
> >
> > Indeed. IMHO we should fix this by putting explicit IO
> > clustering in the ->writepage() functions.
>
> I notice there's a patch sitting in my mailbox.. think I'll go read
> it and think (grunt grunt;) about this issue some more.

Mike,

One important information which is not being considered by
page_launder() now the dirty buffers watermark.

In general, it should not try to avoid writing dirty pages if we're above
the dirty buffers watermark.

2001-03-07 07:58:55

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

On Tue, 6 Mar 2001, Marcelo Tosatti wrote:

> On Fri, 2 Mar 2001, Mike Galbraith wrote:
>
> > On Thu, 1 Mar 2001, Rik van Riel wrote:
> >
> > > > > The merging at the elevator level only works if the requests sent to
> > > > > it are right next to each other on disk. This means that randomly
> > > > > sending stuff to disk really DOES DESTROY PERFORMANCE and there's
> > > > > nothing the elevator could ever hope to do about that.
> > > >
> > > > True to some (very real) extent because of the limited buffering
> > > > of requests. However, I can not find any useful information
> > > > that the vm is using to guarantee the IT does not destroy
> > > > performance by your own definition.
> > >
> > > Indeed. IMHO we should fix this by putting explicit IO
> > > clustering in the ->writepage() functions.
> >
> > I notice there's a patch sitting in my mailbox.. think I'll go read
> > it and think (grunt grunt;) about this issue some more.
>
> Mike,
>
> One important information which is not being considered by
> page_launder() now the dirty buffers watermark.
>
> In general, it should not try to avoid writing dirty pages if we're above
> the dirty buffers watermark.

Agreed in theory.. I'll go try to measure.

-Mike

2002-01-16 21:45:20

[permalink] [raw]

Subject: Re: [patch][rfc][rft] vm throughput 2.4.2-ac4

On Wed, 28 Feb 2001, Mike Galbraith wrote:

> On Wed, 28 Feb 2001, Marcelo Tosatti wrote:
>
> > On Wed, 28 Feb 2001, Mike Galbraith wrote:
> >
> > > > > Have you tried to use SWAP_SHIFT as 4 instead of 5 on a stock 2.4.2-ac5 to
> > > > > see if the system still swaps out too much?
> > > >
> > > > Not yet, but will do.
> >
> > But what about swapping behaviour?
> >
> > It still swaps too much?
>
> Yes.
>
> (returning to study mode)

Ok, I'm stupid. Changing SWAP_SHIFT will just balance the nr of tasks and
the per-task nr of scanned pte's, but the not (roughly) the total nr of
ptes scanned. I thought it would decrease the number of scanned ptes from
3% (which the current in -ac code does) to 0.3% (which Linus tree does) of
the total ptes.

The problem seems to be multiple users calling swap_out() with 3% of ptes
being scanned. This will unmap ptes way too heavily for common workloads.

Its magic number tuning.. but nothing better can be done for 2.4, I think.

2002-01-16 23:28:28