Hi,
I have been making use of copy-on-write semantics of linux fork to
duplicate a process around 100+ times to generate client load against a
server. The copy-on-write allows me to run many more processes without
swap thrashing than I'd otherwise be able to. The client code is in
perl, so the process sizes are in the MBs. Using this technique I only
need about 2 MB per user, with around 5.5 MB shared.
What I've found is that everything works fine, so long as I don't run so
many clients that the kernel pages out part of a process. When this
happens, it seems (from looking at top output) that the shared memory is
copied when it is paged-out. What's worse is it seems that it is copied
for each process that is sharing it. The net effect is that one page is
gained, but many more pages are created in the other processes that were
sharing the memory. I typically see shared memory in each perl process
drop down to less than 2 MB when this occurs, so each process now
consumes about 6 MB of unshared memory (or so top tells me).
The upshot is the VM system thrashes really badly, because now it has to
swap more data out to disk. The sytem becomes unresponsive and I have
to wait minutes for keystrokes to be recognised (think CTRL-C ;) ). The
good news is that when the scripts complete, the system seems completely
normal.
My sytem is a PIII-550 intel BX chipset 7200 RPM IDE drive with 384 MB
RAM if anyone's interested.
Is this expected and reasonable behaviour? Is it possible for pages to
remain shared, even when they are swapped to disk? Does that already
happen anyway, meaning my analysis of the situation is off base?
I am currently running a vanilla 2.4.12, but have observed the same kind
of behaviour in all linux 2.4.x series kernel with the client test
application. I haven't tried it with 2.2.x or Alan Cox's latest kernel
series.
Thanks for listening and thanks to all the kernel hackers for all their
hard work.
Cheers,
Steve Butler
On Thu, 25 Oct 2001, Steven Butler wrote:
> I have been making use of copy-on-write semantics of linux fork to
> duplicate a process around 100+ times to generate client load against a
> server. The copy-on-write allows me to run many more processes without
> swap thrashing than I'd otherwise be able to. The client code is in
> perl, so the process sizes are in the MBs. Using this technique I only
> need about 2 MB per user, with around 5.5 MB shared.
[snip COW undone on swapout, leading to thrashing]
> Is this expected and reasonable behaviour?
Absolutely not, this is not supposed to happen.
> Is it possible for pages to remain shared, even when they are swapped
> to disk?
I think this already happens in the -ac kernels, probably
in -linus too (though I'm not 100% sure).
> Does that already happen anyway, meaning my analysis of the situation
> is off base?
Possible, but it's also possible you ran into a real bug,
it would be interesting to debug this further...
I wouldn't rule out a bug with the remove-from-swapcache
logic on swapin, either.
regards,
Rik
--
DMCA, SSSCA, W3C? Who cares? http://thefreeworld.net/
http://www.surriel.com/ http://distro.conectiva.com/
On Thu, 25 Oct 2001, Steven Butler wrote:
>
> I have been making use of copy-on-write semantics of linux fork to
> duplicate a process around 100+ times to generate client load against a
> server. The copy-on-write allows me to run many more processes without
> swap thrashing than I'd otherwise be able to. The client code is in
> perl, so the process sizes are in the MBs. Using this technique I only
> need about 2 MB per user, with around 5.5 MB shared.
>
> What I've found is that everything works fine, so long as I don't run so
> many clients that the kernel pages out part of a process. When this
> happens, it seems (from looking at top output) that the shared memory is
> copied when it is paged-out. What's worse is it seems that it is copied
> for each process that is sharing it. The net effect is that one page is
> gained, but many more pages are created in the other processes that were
> sharing the memory. I typically see shared memory in each perl process
> drop down to less than 2 MB when this occurs, so each process now
> consumes about 6 MB of unshared memory (or so top tells me).
It's to be expected that "top" (or 3rd field of /proc/pid/statm) will
tell you that under heavy swapout. It counts a page as "shared" if it's
currently mapped into more than one process (approx. description), and
swapout is doing its best to remove the mappings from processes, to free
up pages later; the usage count of the corresponding data, which would
include the swap usage count, is not being counted here (nor should it be).
If you were to swapoff, ideally you would find those shared counts going
back up to what they were; however, those processes do need that swap, so
in practice swapoff will probably fail or hang or cause OOM kills instead.
Hugh