In an earlier post i mentioned a way of locking up my vm easily and
repeatedly but that has since been fixed in one way or another. I reran the
test and took vmstat 1 's of both runnings on a 2.4.14-pre6-preempt kernel
and a 2.4.13-ac5-preempt kernel. I began both vmstat's at the same time
(about 4 seconds before running each). What i did was run kghostview on a
postscript file located here http://safemode.homeip.net/test.ps . It is
224K. kmail was loaded previously in both trials so kdeinit was already
loaded as were all libs. After kghostview became responsive, i waited a few
seconds (again about 5) and then exited the app.
No other interaction or running programs were present while doing this.
I have 771580 KB of ram and 290740 KB of swap.
Now to explain the graphs.
The blue is AA's vm. The red is Rik's vm. Rik's vm finished in 66 seconds.
AA's vm finished in 52 seconds. Both start at 0 swap usage. Both from clean
boots.
Here is the graph http://safemode.homeip.net/vm_swapcomparison.png . It's
about 4.6K.
When you look at the graph it goes like this.
The left side is 0 seconds, the right side is 66 seconds. bottom is 0KB, top
is 290740KB.
These are generated from data from the orignal vmstat outputs. These are at
http://safemode.homeip.net/aa_vmstat
and
http://safemode.homeip.net/rik_vmstat
I'll leave the actual interpretation of the data of both the graph and raw
data up to those who actually know the code.
Neadless to say that while running the test on either box, the entire
computer became unresponsive multiple times for extended lengths of times.
No OOM was generated on either run.
On Wednesday 31 October 2001 22:41, Alex Pennace wrote:
> On Wed, Oct 31, 2001 at 10:18:29PM -0500, safemode wrote:
> > Here is the graph http://safemode.homeip.net/vm_swapcomparison.png .
> > It's about 4.6K.
>
> The server says, "Wrong file name fool."
Fixed it. That's what you get when you're creating the graph at the same
time as writing the email. Sorry to all. it will work now.
On Wed, Oct 31, 2001 at 10:18:29PM -0500, safemode wrote:
> Here is the graph http://safemode.homeip.net/vm_swapcomparison.png . It's
> about 4.6K.
The server says, "Wrong file name fool."
I decided it would be nice to see the actual swap io as well. since
allocating and writing and reading are two different things.
http://safemode.homeip.net/si-swap.png (max 24508)
http://safemode.homeip.net/so-swap.png (max 118132)
http://safemode.homeip.net/bi-swap.png (max 24536)
finally, i redid the swap one i have posted before. This time it has correct
scaling. Because of this you'll see a final dropoff of the aa's graph to 0,
disregard it, as well as all of the 0 data in the other graphs as they're
there to keep scale correct. Everything after 52 seconds in the aa graphs is
filler.
http://safemode.homeip.net/swap.png (max 290740)
All of these are larger than before due to the larger dimensions of the graph
files. feel free to mirror them if i end up not being fast enough.
On Wednesday 31 October 2001 22:18, safemode wrote:
> In an earlier post i mentioned a way of locking up my vm easily and
> repeatedly but that has since been fixed in one way or another. I reran
> the test and took vmstat 1 's of both runnings on a 2.4.14-pre6-preempt
> kernel and a 2.4.13-ac5-preempt kernel. I began both vmstat's at the same
> time (about 4 seconds before running each). What i did was run kghostview
> on a postscript file located here http://safemode.homeip.net/test.ps . It
> is 224K. kmail was loaded previously in both trials so kdeinit was already
> loaded as were all libs. After kghostview became responsive, i waited a
> few seconds (again about 5) and then exited the app.
>
> No other interaction or running programs were present while doing this.
> I have 771580 KB of ram and 290740 KB of swap.
>
> Now to explain the graphs.
> The blue is AA's vm. The red is Rik's vm. Rik's vm finished in 66
> seconds. AA's vm finished in 52 seconds. Both start at 0 swap usage. Both
> from clean boots.
>
> Here is the graph http://safemode.homeip.net/vm_swapcomparison.png .
> It's about 4.6K.
>
> When you look at the graph it goes like this.
> The left side is 0 seconds, the right side is 66 seconds. bottom is 0KB,
> top is 290740KB.
>
> These are generated from data from the orignal vmstat outputs. These are
> at http://safemode.homeip.net/aa_vmstat
> and
> http://safemode.homeip.net/rik_vmstat
>
> I'll leave the actual interpretation of the data of both the graph and raw
> data up to those who actually know the code.
>
> Neadless to say that while running the test on either box, the entire
> computer became unresponsive multiple times for extended lengths of times.
> No OOM was generated on either run.
> Here is the graph http://safemode.homeip.net/vm_swapcomparison.png . It's
here's my munge of the same data:
http://mhahn.mcmaster.ca/~hahn/foo.png
the measures I find interesting are the SI/SO rates. first, the most obvious
feature is that Rik-VM has a serious problem knowing when to *stop* swapping
out. but SO isn't a bad thing unless it's obsessive: it's when you see high
*swap-in* that you know the VM has previously chosen bad pages to SO.
and this is the second big difference: Rik-VM doesn't make nearly as many
mistakes - especially look at Andrea-VM thrashing out-in-out at ~ samples 26-32.
also, if you merely sum the SI and SO columns for each:
sum(SI) sum(SO) sum(SI+SO)
Rik-VM 43564 317448 290032
AA-VM 118284 171748 361012
to me, this looks like the same point: Rik being SO-happy,
Andrea having to SI a lot more. interesting also that Andrea wins the race,
in spite of poorer SO choices and more swap traffic overall.
> Neadless to say that while running the test on either box, the entire
> computer became unresponsive multiple times for extended lengths of times.
yes, unfortunately this corrupts the value of the data, since the timecourses
are not really comparable, and samples are only vaguely related to time...
regards, mark hahn.
On Thursday 01 November 2001 01:23, Mark Hahn wrote:
> > Here is the graph http://safemode.homeip.net/vm_swapcomparison.png .
> > It's
>
> here's my munge of the same data:
> http://mhahn.mcmaster.ca/~hahn/foo.png
> the measures I find interesting are the SI/SO rates. first, the most
> obvious feature is that Rik-VM has a serious problem knowing when to *stop*
> swapping out. but SO isn't a bad thing unless it's obsessive: it's when
> you see high *swap-in* that you know the VM has previously chosen bad pages
> to SO. and this is the second big difference: Rik-VM doesn't make nearly as
> many mistakes - especially look at Andrea-VM thrashing out-in-out at ~
> samples 26-32.
>
> also, if you merely sum the SI and SO columns for each:
> sum(SI) sum(SO) sum(SI+SO)
> Rik-VM 43564 317448 290032
> AA-VM 118284 171748 361012
> to me, this looks like the same point: Rik being SO-happy,
> Andrea having to SI a lot more. interesting also that Andrea wins the
> race, in spite of poorer SO choices and more swap traffic overall.
>
> > Neadless to say that while running the test on either box, the entire
> > computer became unresponsive multiple times for extended lengths of
> > times.
>
> yes, unfortunately this corrupts the value of the data, since the
> timecourses are not really comparable, and samples are only vaguely related
> to time...
If you miss anything you miss the plateu data that would be found when the
IO peaks. But i doubt much at all was really lost in this case.
> regards, mark hahn.
Actually i found that most if not all of the vmstat's that weren't being
displayed were displayed immediately after the "locks", ie. I got flooded
with vmstats upon the computer becoming responsive again.
Of course I should have been timing each run with a stopwatch, but that never
crossed my mind that the vmstat would be lost. In other words i thought of
it as analogous to having burst-lag on irc. Guess that's for tomorrow after
work.
On Thursday 01 November 2001 01:23, Mark Hahn wrote:
> > Here is the graph http://safemode.homeip.net/vm_swapcomparison.png .
> > It's
>
> here's my munge of the same data:
> http://mhahn.mcmaster.ca/~hahn/foo.png
> the measures I find interesting are the SI/SO rates. first, the most
> obvious feature is that Rik-VM has a serious problem knowing when to *stop*
> swapping out. but SO isn't a bad thing unless it's obsessive: it's when
> you see high *swap-in* that you know the VM has previously chosen bad pages
> to SO. and this is the second big difference: Rik-VM doesn't make nearly as
> many mistakes - especially look at Andrea-VM thrashing out-in-out at ~
> samples 26-32.
>
> also, if you merely sum the SI and SO columns for each:
> sum(SI) sum(SO) sum(SI+SO)
> Rik-VM 43564 317448 290032
> AA-VM 118284 171748 361012
> to me, this looks like the same point: Rik being SO-happy,
> Andrea having to SI a lot more. interesting also that Andrea wins the
> race, in spite of poorer SO choices and more swap traffic overall.
My guess is that rik's vm allocates memory too relaxed. It quickly grabbed
all the memory it thought it would need so it wouldn't have to waste time
increasing or shrinking it (i guess that's why) and in doing so it started to
strangle memory needed for other things, generally decreasing the overall
performance of the system. That could be why you see spikes increasing and
decreasing rapidly in rik's vm allocation of swap. He had allocated
everything and needed to shrink it to make room for something else (actual
generation of the kde window for kghostview?) which caused it to lose much
time and any advantage it had gained by not making actual swap mistakes.
AA's memory allocation is more minimalistic but it easily has room for the
memory needed to render the kde window and all once processing the ps file
was done. This would also back up what i visually saw during the test.
Rik's kernel got done the processing of the ps file before AA's did, but it
was stuck at a frozen looking kghostview window with nothing inside it for a
while before being able to actually render the contents. AA's was able to
render the contents almost immediately after the window showed up on the
screen.
Mark Hahn wrote:
>
> > Here is the graph http://safemode.homeip.net/vm_swapcomparison.png . It's
>
> here's my munge of the same data:
> http://mhahn.mcmaster.ca/~hahn/foo.png
> the measures I find interesting are the SI/SO rates. first, the most obvious
> feature is that Rik-VM has a serious problem knowing when to *stop* swapping
> out. but SO isn't a bad thing unless it's obsessive: it's when you see high
> *swap-in* that you know the VM has previously chosen bad pages to SO.
Sure. SO isn't bad for the benchmark, but think of the guy trying
to use the machine after the test finished. It probably swapped out
a lot of other processes which is why you didn't see it swap in again.
These things weren't needed for the bench, but daily use don't
look like that. If my big job takes a long time - no problem
if I can work on something else with nice performance.
> and this is the second big difference: Rik-VM doesn't make nearly as many
> mistakes - especially look at Andrea-VM thrashing out-in-out at ~ samples 26-32.
>
> also, if you merely sum the SI and SO columns for each:
> sum(SI) sum(SO) sum(SI+SO)
> Rik-VM 43564 317448 290032
> AA-VM 118284 171748 361012
> to me, this looks like the same point: Rik being SO-happy,
> Andrea having to SI a lot more. interesting also that Andrea wins the race,
> in spite of poorer SO choices and more swap traffic overall.
It'd be real interesting to know wether or not the "excessive" swapping
caused extra seeks. Readahead or simply reading more consecutive blocks
don't hurt - while seeks do. Perhaps this is why it didn't hurt so
much?
Also consider the multiuser aspect - punishing a memory pig with
extra swapin isn't necessarily so bad, if it keeps more memory around
for others. Could possibly be bad for a dedicated box, but Andrea won
on speed anyway.
Helge Hafting
On Thu, 1 Nov 2001, Mark Hahn wrote:
> also, if you merely sum the SI and SO columns for each:
> sum(SI) sum(SO) sum(SI+SO)
> Rik-VM 43564 317448 290032
> AA-VM 118284 171748 361012
> to me, this looks like the same point: Rik being SO-happy,
> Andrea having to SI a lot more. interesting also that Andrea wins the race,
> in spite of poorer SO choices and more swap traffic overall.
I think this is because in safemode's test, the swap space
gets exhausted. My VM works better when there is lots of
swap space available but degrades in the (rare) case where
swap space is exhausted.
Testing corner cases always gives interesting results ;)
regards,
Rik
--
DMCA, SSSCA, W3C? Who cares? http://thefreeworld.net/
http://www.surriel.com/ http://distro.conectiva.com/
On Thursday 01 November 2001 07:10, Rik van Riel wrote:
> On Thu, 1 Nov 2001, Mark Hahn wrote:
> > also, if you merely sum the SI and SO columns for each:
> > sum(SI) sum(SO) sum(SI+SO)
> > Rik-VM 43564 317448 290032
> > AA-VM 118284 171748 361012
> > to me, this looks like the same point: Rik being SO-happy,
> > Andrea having to SI a lot more. interesting also that Andrea wins the
> > race, in spite of poorer SO choices and more swap traffic overall.
>
> I think this is because in safemode's test, the swap space
> gets exhausted. My VM works better when there is lots of
> swap space available but degrades in the (rare) case where
> swap space is exhausted.
>
> Testing corner cases always gives interesting results ;)
>
> regards,
>
> Rik
In my previous post i mentioned something like that as to why your vm didn't
perform as well. The thing isn't that you use all of my available memory
(ram + swap), it's that you allocate it all, leaving nothing for the program
later on. I think anything that uses almost a gig of ram outside of
databases is going to be a corner case, but perhaps a better way to figure
out how much memory should be allocated is needed here. Andrea's vm seems
to do a good job at that. if only he could figure out a better way to swap
out pages correctly the first time (as some people say his made more mistakes
than yours) then i cant really find anything bad about it. And i'm trying
to.
Also as others pointed out. After the process was done. You had quite a lot
more swap still allocated. Why exacty?
> also, if you merely sum the SI and SO columns for each:
> sum(SI) sum(SO) sum(SI+SO)
> Rik-VM 43564 317448 290032
> AA-VM 118284 171748 361012
> to me, this looks like the same point: Rik being SO-happy,
> Andrea having to SI a lot more. interesting also that Andrea wins the race,
> in spite of poorer SO choices and more swap traffic overall.
Just cause I didn't see anybody mentioned it yet: your statement is not
valid you switched the SI+SO values. It's 361012 for Rik-VM and 290032 for
AA-VM (easy to see as SO for Rik-VM is higher than the sum for Rik-VM in
your table ;-) ). So AA makes less swap traffic overall which does
actually explain at least partially why it wins the race.
Regards,
Dirk Moerenhout ///// System Administrator ///// Planet Internet NV
I think the answer of why AA's kernel beat rik's has nothing to do with how
much swap rik is using or how much swap is being swapped back in. It has to
do with how rik decides what to swap. Apparently the algorithm used by rik
to play with memory is taking seriously too much cpu and it leaves little for
the actual process to work. Thus AA's less cpu intensive code allows the
program to actually run and despite making errors in what to swap-out, the
process finishes well before Rik's more intelligent code.
Unfortunately, the trailing columns in my aa vmstat somehow got lost during
the paste from terminal buffer to file. This means i'm going to have to redo
it all in order to get an accurate measurement to compare system cpu time to
the rik vm. But for now i think the rik vm system graph is sufficient. And
there are some numbers from the AA vmstat and those alone show a much lower
cpu usage than in rik's. MUCH.
I made an overlay of Rik's system ( kernel ) cpu usage on top of the so and
si graphs to illustrate this. Bottom being 0% top being 100% usage.
http://safemode.homeip.net/sys_so.png
Here we see that after every major write out, there is major kernel cpu
usage. This is serious usage, and this is the reason why rik's VM loses the
race even though it swapped out and in the right things the first try more
often than AA's.
http://safemode.homeip.net/sys_si.png
Of course after each major write out in Rik's vm there is a minor read in.
These happen to be directly under the cpu spikes so this could be the cause
of the cpu usage, perhaps determining where the page is? I dont know enough
about what's going on in the code to figure out if the VM does something
after writing out that could be using all that cpu or if whenever it needs to
read in. Although now that i look at it i'm tending to lean towards some bad
code dealing with swap -> ram.
This is truly where the simple vm design conquers the complex. Less cpu being
used by the kernel means more by the program, and sometimes the time gained
by not using a lot of cpu greatly outweighs the time lost by having to
correct mistakes with deciding what gets swapped in and out.
Maybe i'm wrong as to the cause of the kernel cpu usage, but from the numbers
i do have from AA's vmstat, they are much higher in Rik's vm than in
Andrea's. That and the fact that Rik's vm seems to be doing the right thing
whereas Andrea's is having to fix mistakes yet Rik's loses seems to tell you
that i'm not wrong in thinking that it's the vm's cpu usage that is the
culprit.
On Thu, 1 Nov 2001, safemode wrote:
> I think the answer of why AA's kernel beat rik's has nothing to do
> with how much swap rik is using or how much swap is being swapped back
> in. It has to do with how rik decides what to swap. Apparently the
> algorithm used by rik to play with memory is taking seriously too much
> cpu and it leaves little for the actual process to work.
Note that this is likely to be a side effect of running
completely out of swap, because that means many of the
"obvious candidates" of what to swap out cannot be swapped
out, meaning we have to scan more pages until we find
something which already has swap backing.
Before you draw conclusions like the one above, please test
again with more swap.
regards,
Rik
--
DMCA, SSSCA, W3C? Who cares? http://thefreeworld.net/
http://www.surriel.com/ http://distro.conectiva.com/
On Friday 02 November 2001 06:17, Rik van Riel wrote:
> On Thu, 1 Nov 2001, safemode wrote:
> > I think the answer of why AA's kernel beat rik's has nothing to do
> > with how much swap rik is using or how much swap is being swapped back
> > in. It has to do with how rik decides what to swap. Apparently the
> > algorithm used by rik to play with memory is taking seriously too much
> > cpu and it leaves little for the actual process to work.
>
> Note that this is likely to be a side effect of running
> completely out of swap, because that means many of the
> "obvious candidates" of what to swap out cannot be swapped
> out, meaning we have to scan more pages until we find
> something which already has swap backing.
>
> Before you draw conclusions like the one above, please test
> again with more swap.
>
> regards,
>
> Rik
I'll try it with more swap later on today after work. But realize, though,
the fact that you need much more swap to do the same thing (compared to aa's)
is not helping any adoption of the VM.
On Fri, 2 Nov 2001, safemode wrote:
> I'll try it with more swap later on today after work. But realize,
> though, the fact that you need much more swap to do the same thing
> (compared to aa's) is not helping any adoption of the VM.
Uhhh ... this is nothing but a classical speed/size tradeoff.
The fact that under my VM swap space stays reserved for the
program on swapin means that if the page isn't dirtied, we
can just drop it without having to write it to disk again.
In situations where there is enough swap available, this
should be a win (and it has traditionally been a big win).
Andrea's VM always frees swap space on swapin, so even if
the process doesn't write to its memory at all, the data
still needs to be written out to disk again.
Only in the one corner-case where my VM runs out of swap
space and Andrea's VM doesn't yet run out of swap you'll
find situations where the tactic used by Andrea's VM has
its advantages, but I consider this to be a rare situation.
regards,
Rik
--
DMCA, SSSCA, W3C? Who cares? http://thefreeworld.net/
http://www.surriel.com/ http://distro.conectiva.com/
So I added more swap, for 461549568 bytes total, and guess what happens? No,
Rik's vm did not beat AA's. You might say, "well then it must have came
close to tying it now since it has enough swap and wont fall victim to the
'speed/size tradeoff'". No, you would be wrong. Rik's VM completely locked
up in some kind of infinite swap loop like it did before in earlier kernels.
My server was swapping ( i saw disk activity) for over 8 hours before i
finally rebooted it. Needless to say that it angers me to unintentionally
lock the computer up. So what's going on here? rik, anyone? I've used this
test before to lock up Rik's VM and it is reproduceable on my machine. With
less swap, it seems to be gone in the latest kernels but when i added more
like Rik said to do, it locked up. For now i'll be running AA kernels until
this is figured out.
What i found insane from the little info provided by the vmstat i had running
at the time was that even with 450+MB of swap, Rik's VM still used it all up.
Why would it work better without that much ram but when i add more, it still
uses it all up and locks up. There's something seriously wrong here. I'm
going to test's andrea's with the new mem config too later. My bet is that
it doesn't lock up for over 8 hours trying to swap.
Since i'm now in Linus' kernel, i did the test again on it with the added
swap. It took about 7 minutes to finish. Since rik's vm locked up so
completly after only about a minute, i really have no data on what it was
doing during those 8 hours i had it running it's course. Andrea's was more
fruitful and vmstat ran quite nicely the entire time, probably only missing a
couple here and there.
The only thing i could tell from the two tests is that Rik's vm allocates all
of my memory and swap almost immediately and for extended periods of time.
Andrea's alocates memory gradually and more conservatively and only reaches
all of my swap for an instant. Other than that the only thing i can tell is
that my computer is usable when using andrea's vm after the test. I have to
reboot it after starting the test with Rik's vm.
On Friday 02 November 2001 18:48, safemode wrote:
> So I added more swap, for 461549568 bytes total, and guess what happens?
> No, Rik's vm did not beat AA's. You might say, "well then it must have
> came close to tying it now since it has enough swap and wont fall victim to
> the 'speed/size tradeoff'". No, you would be wrong. Rik's VM completely
> locked up in some kind of infinite swap loop like it did before in earlier
> kernels. My server was swapping ( i saw disk activity) for over 8 hours
> before i finally rebooted it. Needless to say that it angers me to
> unintentionally lock the computer up. So what's going on here? rik,
> anyone? I've used this test before to lock up Rik's VM and it is
> reproduceable on my machine. With less swap, it seems to be gone in the
> latest kernels but when i added more like Rik said to do, it locked up.
> For now i'll be running AA kernels until this is figured out.
> What i found insane from the little info provided by the vmstat i had
> running at the time was that even with 450+MB of swap, Rik's VM still used
> it all up. Why would it work better without that much ram but when i add
> more, it still uses it all up and locks up. There's something seriously
> wrong here. I'm going to test's andrea's with the new mem config too later.
> My bet is that it doesn't lock up for over 8 hours trying to swap.
