-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Rik,
I just found out about your desire to have some workstation testing
done to get feedback on your current VM patch.
I am currently using Kernel 2.4.9-ac18. I am still not happy about
some of the memory management. I love to try your patch. I would be
willing to do so against the 2.4.10-ac11 Kernel if a patch is
available. If Alan is going to implement this patch in a later
2.4.10-acxx patch I will wait patiently.
I also be most interested in knowing how I can log my system memory
usage. I can use top to log at say every 2-5 minutes depending on
how long I think my system will be up and running. What I do not
know how to do yet is log the number of KDE 1.1.2 or XFCE 3.8.0
windows open. I am basically using a SuSE 6.4 with the 2.4 kernel
and its updated element requirements and KDE 1.1.2. Also any other
memory use logging I can do may prove useful in understanding the
Workstation memory use and effect of the VM logic you are working on.
I recently started exploring other Window managers as I am trying to
isolate other memory issues, i.e. memory leaks. these leaks on a
workstation appear big time. At this point I am having difficulty in
collecting data that will tell me if the kernel, KDE, XFree, KFM,
Netscape, and/or KPanel are all leaking because of the kernel or if
this is "Team" effort of all or some of these. I know generally who
and what is getting out of hand, just not the who done it. Bottom
line, a 256MB Ram, with 256MB Swap file is just not holding up to
Workstation use, i.e. it is filling up!. I frequently have to shut
down applications just recover memory, and eventually I have to shut
down Linux as I cannot seem to get it to recover the memory.
I hate to say this, exact same system (I have SCSI trays to change
OS's) using NT 4.0 does generally never gets past 100 MB of swap
useage. The core of the cause is surfing the net. In Linux I seem
to reach a brick wall after 3-4 hours where swap and RAM are topped
off. In NT can go a few days with no problems if need be (system is
left overnight or while I am at work downloading files via my humble
56K (49.3K) connection. So ISO images do take a while ;).
Regards,
John L. Males
Willowdale, Ontario
Canada
12 October 2001 00:33
mailto:[email protected]
From: Rik van Riel <[email protected]>
To: <[email protected]>
Subject: [CFT][PATCH] smoother VM for -ac
Date: Wed, 10 Oct 2001 17:25:30 -0300 (BRST)
Cc: <[email protected]>, <[email protected]>,
Alan Cox <[email protected]>
Hi,
over the last week I've created a small patch which seems
to drastically improve VM performance and interactivity for
2.4.10-ac{9,10}. Initial test results mostly seem to suggest
that the system runs lots smoother for desktop use and doesn't
get into thrashing until the working set _really_ exceeds the
size of RAM.
People have already asked to have this patch integrated into
the -ac kernel, but it would be nice to have a few more test
results from this combined eatcache + stophog patch before
having it integrated ...
The patch implements the following things:
1) bypass page aging entirely for unused objects in
the cache
2) increase the distance between inactive_shortage
and inactive_plenty, so kswapd should spend less
time shuffling random pages around ... shouldn't
make a difference for most loads, but should add
some robustness in worst cases
3) does page aging _before_ the zone_inactive_plenty()
test, so old referenced bits get cleared
[not a big cpu eater, since the code won't run unless
we have a free or inactive shortage somewhere]
4) in page_alloc.c, the "slowdown" reschedule has been
made stronger by turning it into a try_to_free_pages(),
under memory load, this results in allocators calling
try_to_free_pages() when the amount of work to be done
isn't too bad yet and pretty much guarantees them they'll
get to do their allocation immediately afterwards ...
statistics make sure that the memory hogs are slowed down
much more than well-behaved programs
Please test this patch and tell Alan and me how it works for
you and whether there are loads where the system performs
worse with this patch than without...
regards,
Rik
- --
DMCA, SSSCA, W3C? Who cares? http://thefreeworld.net/ (volunteers
needed)
http://www.surriel.com/ http://distro.conectiva.com/
- --- linux-2.4.10-ac10/mm/page_alloc.c.orig Mon Oct 8 18:22:51
2001
+++ linux-2.4.10-ac10/mm/page_alloc.c Wed Oct 10 14:08:54 2001
@@ -346,22 +346,15 @@
* We wake up kswapd, in the hope that kswapd will
* resolve this situation before memory gets tight.
*
- - * We also yield the CPU, because that:
- - * - gives kswapd a chance to do something
- - * - slows down allocations, in particular the
- - * allocations from the fast allocator that's
- - * causing the problems ...
- - * - ... which minimises the impact the "bad guys"
- - * have on the rest of the system
- - * - if we don't have __GFP_IO set, kswapd may be
- - * able to free some memory we can't free ourselves
+ * We'll also help a bit trying to free pages, this
+ * way statistics will make sure really fast allocators
+ * are slowed down more than slow allocators and other
+ * programs in the system shouldn't be impacted as much
+ * by the hogs.
*/
wakeup_kswapd();
- - if (gfp_mask & __GFP_WAIT) {
- - __set_current_state(TASK_RUNNING);
- - current->policy |= SCHED_YIELD;
- - schedule();
- - }
+ if (gfp_mask & __GFP_WAIT)
+ try_to_free_pages(gfp_mask);
/*
* After waking up kswapd, we try to allocate a page
- --- linux-2.4.10-ac10/mm/vmscan.c.orig Mon Oct 8 18:22:51 2001
+++ linux-2.4.10-ac10/mm/vmscan.c Mon Oct 8 19:18:12 2001
@@ -50,7 +50,7 @@
inactive += zone->inactive_clean_pages;
inactive += zone->free_pages;
- - return (inactive > (zone->size / 3));
+ return (inactive > (zone->size * 2 / 5));
}
#define FREE_PLENTY_FACTOR 2
@@ -97,6 +97,24 @@
return pagecache > limit;
}
+static inline int page_mapping_notused(struct page * page)
+{
+ struct address_space * mapping = page->mapping;
+
+ if (!mapping)
+ return 0;
+
+ /* This mapping is really large and would monopolise the
pagecache. */
+ if (mapping->nrpages > atomic_read(&page_cache_size) / 20);
+ return 0;
+
+ /* File is mmaped by somebody */
+ if (mapping->i_mmap || mapping->i_mmap_shared)
+ return 1;
+
+ return 0;
+}
+
/*
* The swap-out function returns 1 if it successfully
* scanned all the pages it was asked to (`count').
@@ -826,14 +844,14 @@
}
/*
- - * Don't deactivate pages from zones which have
- - * plenty inactive pages.
+ * Do aging on the pages. Every time a page is
referenced,
+ * page->age gets incremented. If it wasn't
referenced, we
+ * decrement page->age. The page gets moved to the
inactive
+ * list when one of the following is true:
+ * - the page age reaches 0
+ * - the object the page belongs to isn't in active
use
+ * - the object the page belongs to is hogging the
cache
*/
- - if (zone_inactive_plenty(page->zone)) {
- - goto skip_page;
- - }
- -
- - /* Do aging on the pages. */
if (PageTestandClearReferenced(page)) {
age_page_up(page);
} else {
@@ -843,20 +861,26 @@
}
/*
- - * If the amount of buffer cache pages is too
- - * high we just move every buffer cache page we
- - * find to the inactive list. Eventually they'll
- - * be reclaimed there...
+ * Don't deactivate pages from zones which have
+ * plenty inactive pages.
+ */
+ if (zone_inactive_plenty(page->zone)) {
+ goto skip_page;
+ }
+
+ /*
+ * If the buffer cache is large, don't do page aging.
+ * If this page really is used, it'll be referenced
+ * again while on the inactive list.
*/
if (page->buffers && !page->mapping &&
too_many_buffers())
deactivate_page_nolock(page);
/*
- - * If the page cache is too large, we deactivate all
- - * page cache pages which are not in use by a
process.
+ * Deactivate pages from files which aren't in use,
busy
+ * pages will be referenced while on the inactive
list.
*/
- - if (pagecache_too_large() && page->mapping &&
- - page_count(page) <= (page->buffers ?
2 : 1))
+ if (page_mapping_notused(page))
deactivate_page_nolock(page);
/*
- --- linux-2.4.10-ac10/include/linux/swap.h.orig Mon Oct 8 18:23:03
2001
+++ linux-2.4.10-ac10/include/linux/swap.h Mon Oct 8 19:15:09
2001
@@ -261,7 +261,7 @@
if (vm_static_inactive_target)
return vm_static_inactive_target;
- - return num_physpages / 4;
+ return num_physpages / 5;
}
/*
-
On Fri, 2001-10-12 at 01:33, John L. Males wrote:
> I just found out about your desire to have some workstation testing
> done to get feedback on your current VM patch.
>
> I am currently using Kernel 2.4.9-ac18. I am still not happy about
> some of the memory management. I love to try your patch. I would be
> willing to do so against the 2.4.10-ac11 Kernel if a patch is
> available. If Alan is going to implement this patch in a later
> 2.4.10-acxx patch I will wait patiently.
Said patch and other VM work is in 2.4.10-ac12. Get that and report
back.
Robert Love
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Robert,
Great. Will download. I had not seen the 2.4.10-ac12 patch
announced. Thanks so kindly for the update. I expect I will have
some observations to make by Saturady evening here EDT.
Regards,
John L. Males
Willowdale, Ontario
Canada
12 October 2001 01:09
mailto:[email protected]
Subject: Re: Re[02]: [CFT][PATCH] smoother VM for -ac
From: Robert Love <[email protected]>
To: [email protected]
Copies to: Rik van Riel <[email protected]>,
[email protected],
[email protected], Alan Cox
<[email protected]>
Date sent: 12 Oct 2001 00:41:19 -0400
> On Fri, 2001-10-12 at 01:33, John L. Males wrote:
> > I just found out about your desire to have some workstation
> > testing done to get feedback on your current VM patch.
> >
> > I am currently using Kernel 2.4.9-ac18. I am still not happy
> > about some of the memory management. I love to try your patch.
> > I would be willing to do so against the 2.4.10-ac11 Kernel if a
> > patch is available. If Alan is going to implement this patch in
> > a later 2.4.10-acxx patch I will wait patiently.
>
> Said patch and other VM work is in 2.4.10-ac12. Get that and
> report back.
>
> Robert Love
>
>
-----BEGIN PGP SIGNATURE-----
Version: PGPfreeware 6.5.8 for non-commercial use
<http://www.pgp.com>
iQA/AwUBO8aJFeAqzTDdanI2EQJg/gCeK677eZLUFzesSNuHyVPih6eHeiUAniyH
VKENEZ1nMfqMmbmWG8TWNXGZ
=oZJr
-----END PGP SIGNATURE-----
"Boooomer ... Boom Boom, how are you Boom Boom" Boomer 1985 - February/2000
On Fri, Oct 12, 2001 at 12:41:19AM -0400, Robert Love wrote:
> On Fri, 2001-10-12 at 01:33, John L. Males wrote:
> > I just found out about your desire to have some workstation testing
> > done to get feedback on your current VM patch.
> >
> > I am currently using Kernel 2.4.9-ac18. I am still not happy about
> > some of the memory management. I love to try your patch. I would be
> > willing to do so against the 2.4.10-ac11 Kernel if a patch is
> > available. If Alan is going to implement this patch in a later
> > 2.4.10-acxx patch I will wait patiently.
>
> Said patch and other VM work is in 2.4.10-ac12. Get that and report
> back.
Later, if you've some time to test, I'd also be very interested in a
comparison with 2.4.12aa1.
Andrea
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
- -----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Andrea,
I can do. I see this is a VM is of keen interest. Question for you.
To really compare apples to apples I could spider a web site or two
just find. Then the challenge is to replay the "test" on the gui,
say KDE for example. Do you know of any good tools that would alow
me to do a GUI record/playback? I can then do an A vs B comparison.
Also, remind me, can I find your kernel to test on the SuSE FTP site
or via kernel.org. I had tried a few of the SuSE 2.4 kernels a few
levels back and I recall I was going to the people directory of the
FTP site and getting them from mantel I seem to recollect.
I will search about on internet to see if I can find a
record/playback too to get some sort of good A vs B comparison.
Regards,
John L. Males
Willowdale, Ontario
Canada
12 October 2001 01:33
mailto:[email protected]
Date sent: Fri, 12 Oct 2001 07:09:30 +0200
From: Andrea Arcangeli <[email protected]>
To: Robert Love <[email protected]>
Copies to: [email protected], Rik van Riel
<[email protected]>,
[email protected], [email protected],
Alan Cox <[email protected]>
Subject: Re: Re[02]: [CFT][PATCH] smoother VM for -ac
> On Fri, Oct 12, 2001 at 12:41:19AM -0400, Robert Love wrote:
> > On Fri, 2001-10-12 at 01:33, John L. Males wrote:
> > > I just found out about your desire to have some workstation
> > > testing done to get feedback on your current VM patch.
> > >
> > > I am currently using Kernel 2.4.9-ac18. I am still not happy
> > > about some of the memory management. I love to try your patch.
> > > I would be willing to do so against the 2.4.10-ac11 Kernel if
> > > a patch is available. If Alan is going to implement this patch
> > > in a later 2.4.10-acxx patch I will wait patiently.
> >
> > Said patch and other VM work is in 2.4.10-ac12. Get that and
> > report back.
>
> Later, if you've some time to test, I'd also be very interested in
> a comparison with 2.4.12aa1.
>
> Andrea
- -----BEGIN PGP SIGNATURE-----
Version: PGPfreeware 6.5.8 for non-commercial use
<http://www.pgp.com>
iQA/AwUBO8aOyOAqzTDdanI2EQLa4ACg8+jxwCGBvKXasN0skSmdwMEiXKsAoODH
m39x4FgquaNCVx0E8lHjHynA
=I7rv
- -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNATURE-----
Version: PGP 6.5.2 -- QDPGP 2.61a
Comment: .
iQA/AwUBO8aO2eAqzTDdanI2EQLRZQCfSgergtM1o0/4f+RIcD1tCvY3iJUAoL4f
WoHD5idV3ByDJzR3XWYigHmh
=dlmp
-----END PGP SIGNATURE-----
"Boooomer ... Boom Boom, how are you Boom Boom" Boomer 1985 - February/2000
On Fri, Oct 12, 2001 at 01:33:54AM -0500, John L. Males wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> - -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Andrea,
>
> I can do. I see this is a VM is of keen interest. Question for you.
> To really compare apples to apples I could spider a web site or two
> just find. Then the challenge is to replay the "test" on the gui,
> say KDE for example. Do you know of any good tools that would alow
> me to do a GUI record/playback? I can then do an A vs B comparison.
For testing the repsonsiveness I usually check the startup time of
applications like netscape with cold cache, later I just start an high
vm load on my desktop and I see how long can I keep working without
being too hurted. the first is certainly a measurable test, the second
isn't reliable since it doesn't generate raw numbers and it's too much
in function of the human feeling but it shows very well any patological
problem of the code. But they may not be the best tests.
> Also, remind me, can I find your kernel to test on the SuSE FTP site
> or via kernel.org. I had tried a few of the SuSE 2.4 kernels a few
> levels back and I recall I was going to the people directory of the
> FTP site and getting them from mantel I seem to recollect.
That's still fine procedure, only make sure to pick the latest 2.4.12
one based on 2.4.12aa1 before running the tests. thanks,
> I will search about on internet to see if I can find a
> record/playback too to get some sort of good A vs B comparison.
>
>
> Regards,
>
> John L. Males
> Willowdale, Ontario
> Canada
> 12 October 2001 01:33
> mailto:[email protected]
Andrea
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Andrea,
***** FYI, I am not on the kernel mailing list *****
I had a had a chance to do some testing with the unofficial SuSE
2.4.12 Kernel that I believe is based on your 2.4.12aa1.
I used:
ftp://ftp.suse.com/pub/people/mantel/next/RPM/k_i386-2.4.12-0.i386.rpm
I have an AMD K6-2-500 and find that the Pentium or "default" SuSE
kernels hang using the AMD K6-2. I was unable to compile the kernel
from the souce using:
ftp://ftp.suse.com/pub/people/mantel/next/linux-2.4.12.SuSE-0.tar.bz2
due some unknow technical problems. One was "make xconfig" would not
build, and using "make oldconfig" and the usual make commands to
build a kernel caused a error with a module a ways down. The kernel
image compiled fine.
Ok, enough of those side issues. The meat of the testing results is
that the k_i386-2.4.12-0.i386.rpm kernel seems to fair very well with
the testing I have done. The tests take about 20 minutes to complete
with the k_i386-2.4.12-0.i386.rpm kernel. The test mix is a bit
interesting, so I will only suggest it might be nice to shorten the
lapsed time of the test, but may not be possible due to I/O being the
bottleneck.
With respect to comparing the same test but using the 2.4.10-ac12
kernel that appears to have the both of Rik van Riel's patches:
http://lwn.net/2001/1011/a/cache-reclaim.php3
http://lwn.net/2001/1011/a/smooth.php3
The results were not great. The "exact" same test takes a little
over 3 hours to complete.
The part of the test that seems to cause problems with the
2.4.10-ac12 involves a 300MB working set. This 300MB working set was
on top of a basic 60MB (combined System, shared, cache and buffer)
after the initial system start up. The system used for testing has
256MB RAM and 256MB Swap file. It seems the 2.4.10-ac12 ends up with
extra memory (overhead??) allocated during this part of the test to
basically have next to nil shared+cache+buffer whilst having both RAM
and the cache full to the brim. If that is not enough the
2.4.10-ac12 seems to vary at times back and forth +-25MB while the
working set is still trying to be processed by the kernel. Along the
way the the 2.4.10-ac12 kernel also tends to kill or cause a signal 9
to some of the working set applications, but despite this the system
seems to churn on this 300MB working set for just about 3 hours
(other part of test brings total to just over 3 hours).
By comparison the k_i386-2.4.12-0.i386.rpm test during this same
300MB workig set showed little extra overhead. Hence 300 + 60 did
cause RAM to fill up and next to nil of share+cache+buffer, but the
swap file too the balance in a more expected manner, i.e about 120 MB
into the swap file. Hence the swap file was never pressured to it
full limit as would be expected.
In terms of workstation responsiveness, it was not great with the
k_i386-2.4.12-0.i386.rpm kernel, but was extremely, extremely to
ignoring workstation activity or taking in the order of 5+ minutes to
respond to simple things like launching qps, or changing directories
with kruiser and many big time problems getting into the screen saver
to unlock the workstation.
I do need to refine my tests a bit. One thing I am going to do is
move the detailed system information I am trying to log during the
test to a different physical drive than where the swap files are
located. I suspect this should ease the contention that may be
ensuing between the system's need to page to the swap file and the
need to keep logging the metric information.
I also need to find a way to collect certain metric information
during the test. Not being a developer (I am a QA/Testing Person) it
may take me a bit of effort to cut and carve what I need from qps and
xosview to get the metric information that is lacking for these
tests. I suspect I will not be able to look into the cutting and
carving of code until next weekend. I may try this test on the
2.4.9-ac18, maybe even the 2.2.19 for a feel if they are greatly
different in results to the two kernels tested earlier today.
Regards,
John L. Males
Willowdale, Ontario
Canada
14 October 2001 03:36
mailto:[email protected]
Date sent: Fri, 12 Oct 2001 07:54:29 +0200
From: Andrea Arcangeli <[email protected]>
To: "John L. Males" <[email protected]>
Copies to: Rik van Riel <[email protected]>,
[email protected],
[email protected], Alan Cox
<[email protected]>
Subject: Re: [CFT][PATCH] smoother VM for -ac
> On Fri, Oct 12, 2001 at 01:33:54AM -0500, John L. Males wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > - -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Andrea,
> >
> > I can do. I see this is a VM is of keen interest. Question for
> > you.
> > To really compare apples to apples I could spider a web site or
> > two just find. Then the challenge is to replay the "test" on the
> > gui, say KDE for example. Do you know of any good tools that
> > would alow me to do a GUI record/playback? I can then do an A vs
> > B comparison.
>
> For testing the repsonsiveness I usually check the startup time of
> applications like netscape with cold cache, later I just start an
> high vm load on my desktop and I see how long can I keep working
> without being too hurted. the first is certainly a measurable test,
> the second isn't reliable since it doesn't generate raw numbers and
> it's too much in function of the human feeling but it shows very
> well any
> patological problem of the code. But they may not be the best
> tests.
>
> > Also, remind me, can I find your kernel to test on the SuSE FTP
> > site or via kernel.org. I had tried a few of the SuSE 2.4
> > kernels a few levels back and I recall I was going to the people
> > directory of the FTP site and getting them from mantel I seem to
> > recollect.
>
> That's still fine procedure, only make sure to pick the latest
> 2.4.12 one based on 2.4.12aa1 before running the tests. thanks,
>
> > I will search about on internet to see if I can find a
> > record/playback too to get some sort of good A vs B comparison.
> >
> >
> > Regards,
> >
> > John L. Males
> > Willowdale, Ontario
> > Canada
> > 12 October 2001 01:33
> > mailto:[email protected]
>
> Andrea
-----BEGIN PGP SIGNATURE-----
Version: PGPfreeware 6.5.8 for non-commercial use
<http://www.pgp.com>
Comment: .
iQA/AwUBO8lOiuAqzTDdanI2EQLfrgCfTPXHDyEEoAWfTdWC28UaMzv1EAQAoKKT
PjOOkmCNggHekWz35GZugJnt
=FfXm
-----END PGP SIGNATURE-----
"Boooomer ... Boom Boom, how are you Boom Boom" Boomer 1985 - February/2000