Hello All,
I've got a box with 750MB of RAM.. for the sake of this test, I've
turned off swap. I also have a little program that creates and deletes
lots of little files on a RAMDISK with ext2 on it. The ramdisk is only
4MB max and the files are about 3K and there are never more than four
of them at a time - but the program is constantly creating and
deleting hundreds of thousands of them. I have no reason to think this
issue is unique to the ramdisk - it just makes files faster.
Just sitting back and watching vmstat while this runs my 'free' memory
drops from ~600MB to about ~16MB.. the buffers and cache remain roughly
constant.. at 16MB some sort of garbage collection kicks in - there is
a notable system pause and ~70MB moves from the used to the 'free'
column... this process repeats more or less in a steady state.
If, while this is going on, I run another little app that does
{x= malloc(300MB), memset (x,0,300MB), free (x)}.. suddenly I can move
300MB from the used to the 'free' state...
I assume there is some VFS structure that is growing (perhaps related
to the tremendous # of inodes I'm going through) and is only cleansed
when the number of free pages gets too low.. I'd be interested in any
details someone could provide.
My real question is about the commonly (?) used metric of
free+buffers+cached representing the size of an allocation that could
succeed.. my 300MB allocation above proves that doesn't apply
here.. although, if I turn on strict accounting, that 300MB does fail
even though there is only perhaps 150MB of userspace allocations
outstanding on the box.
Can anybody provide a better metric for "ram free for userspace
allocations"?
Thanks,
-Patrick
"Patrick R. McManus" wrote:
>
> ...
> Just sitting back and watching vmstat while this runs my 'free' memory
> drops from ~600MB to about ~16MB.. the buffers and cache remain roughly
> constant.. at 16MB some sort of garbage collection kicks in - there is
> a notable system pause and ~70MB moves from the used to the 'free'
> column... this process repeats more or less in a steady state.
Probably `negative dentries' - cached directory entries which say
"this file isn't there" so we don't need to go into the fs to
find that out.
If you could share your test apps that would help a lot.
> If, while this is going on, I run another little app that does
> {x= malloc(300MB), memset (x,0,300MB), free (x)}.. suddenly I can move
> 300MB from the used to the 'free' state...
The slab cache (general kernel memory allocator+cache) has mechanisms
for freeing cached objects when memory gets tight. That will recycle
all those negative dentries. If that's what you're seeing.
> ...
> Can anybody provide a better metric for "ram free for userspace
> allocations"?
On your machine it'll be "all of swap plus all of physical memory
minus whatever malloc'ed memory you're using now minus 8-12 megabytes".
There isn't much memory which cannot be reclaimed unless you have a
huge machine or you're doing odd things.
[Andrew Morton: Dec 11 16:09]
> "Patrick R. McManus" wrote:
> >
> > ...
> > Just sitting back and watching vmstat while this runs my 'free' memory
> > drops from ~600MB to about ~16MB.. the buffers and cache remain roughly
> > constant.. at 16MB some sort of garbage collection kicks in - there is
> > a notable system pause and ~70MB moves from the used to the 'free'
> > column... this process repeats more or less in a steady state.
>
> Probably `negative dentries' - cached directory entries which say
> "this file isn't there" so we don't need to go into the fs to
> find that out.
>
> If you could share your test apps that would help a lot.
sure! this is the "lots of files" program.
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#define BSZ 2000
unsigned char buf[BSZ+256];
main ()
{
int fd,i,j;
unsigned int ctr=0;
unsigned char r=0;
char f1[32],f2[32],f3[32],f4[32];
while (1)
{
sprintf (f1,"fxa_%06d",ctr++); sprintf (f2,"fxa_%06d",ctr++);
sprintf (f3,"fxa_%06d",ctr++); sprintf (f4,"fxa_%06d",ctr++);
fd = open (f1,O_CREAT | O_RDWR);
memset (buf,r++,BSZ+256); write (fd,buf,BSZ+r); close (fd);
fd = open (f2,O_CREAT | O_RDWR);
memset (buf,r++,BSZ+256); write (fd,buf,BSZ+r); close (fd);
fd = open (f3,O_CREAT | O_RDWR);
memset (buf,r++,BSZ+256); write (fd,buf,BSZ+r); close (fd);
fd = open (f4,O_CREAT | O_RDWR);
memset (buf,r++,BSZ+256); write (fd,buf,BSZ+r); close (fd);
unlink (f1); unlink (f2); unlink (f3); unlink (f4);
if (!r)
fprintf (stdout,".");fflush (stdout);
}
}
and this is the "gimme 300MB even though free sez I hardly have
anything" program:
#define BS (30*1024*1024)
main()
{
char *x;
x = malloc (BS);
if (!x)
printf ("heh - x is NULL\n");
else
{
printf ("allocd\n");
memset (x,0x31,BS);
printf ("set\n");
free (x);
printf ("freed\n");
}
}
> On your machine it'll be "all of swap plus all of physical memory
> minus whatever malloc'ed memory you're using now minus 8-12 megabytes".
> There isn't much memory which cannot be reclaimed unless you have a
> huge machine or you're doing odd things.
this is useful advice, thanks. Basically what the new procps does?
"Patrick R. McManus" wrote:
>
> ...
> > If you could share your test apps that would help a lot.
>
> sure! this is the "lots of files" program.
Yep, negative dentries:
dentry_cache: 149092KB 149092KB 100.0
ext3_inode_cache: 3425KB 10102KB 33.90
radix_tree_node: 594KB 1612KB 36.88
buffer_head: 543KB 1264KB 42.99
You can monitor these via /proc/slabinfo, or using Bill's bloatmeter
script from http://www.zip.com.au/~akpm/linux/patches/stuff/
> ...
>
> > On your machine it'll be "all of swap plus all of physical memory
> > minus whatever malloc'ed memory you're using now minus 8-12 megabytes".
> > There isn't much memory which cannot be reclaimed unless you have a
> > huge machine or you're doing odd things.
>
> this is useful advice, thanks. Basically what the new procps does?
I don't know what procps does.
But your fill-up-all-memory program will certainly do the trick.
Run it, create a huge swapstorm (or get oom-killed if there's no
swap) and then see what `free' says. That's as much memory as the
kernel will ever give you.
>Can anybody provide a better metric for "ram free for userspace
>allocations"?
This is a constant battle for programmers trying to develop sophisticated
applications for sophisticated operating systems.
If all of the applications are cooperating, you can do it fairly easily.
First, ask the kernel how much physical memory there is. Then fudge a
reduction based on kernel usage. Then subtract the amount of RAM each
application is 'really using' by asking it.
If you write all the applications, this isn't hard to do. If you have to
cooperate with other applications you didn't write, things get trickier. If
you're the only application, things are really easy.
There are any number of heuristics and guesstimates. Ultimately, I recommend
mostly leaving your memory usage user-configurable.
DS