I noticed that doing lots or file accesses causes the used memory to
increase, *after* subtracting buffers/cache. Here is an example:
$ free
total used free shared buffers cached
Mem: 773776 30024 743752 0 1992 10424
-/+ buffers/cache: 17608 756168
Swap: 81904 0 81904
$ du > /dev/null
$ free
total used free shared buffers cached
Mem: 773776 78008 695768 0 26328 10472
-/+ buffers/cache: 41208 732568
Swap: 81904 0 81904
Here 24 MB of memory have been used up. Repeating the du seems to have
little effect. This directory has ~3200 subdirs and 13400 files.
After a few hours use about 200 MB are used, apperently for
nothing. Killing all processed and unmounting file systems doesn't
help.
Is this a memory leak? I get the same results with ext2, ext3,
reiserfs and nfs.
--
M?ns Rullg?rd
[email protected]
On 21 Jul 2002 16:00:09 +0200
[email protected] (M) wrote:
>
> I noticed that doing lots or file accesses causes the used memory to
> increase, *after* subtracting buffers/cache. Here is an example:
>
> $ free
> total used free shared buffers
> cached
> Mem: 773776 30024 743752 0 1992
> 10424-/+ buffers/cache: 17608 756168
> Swap: 81904 0 81904
> $ du > /dev/null
> $ free
> total used free shared buffers
> cached
> Mem: 773776 78008 695768 0 26328
> 10472-/+ buffers/cache: 41208 732568
> Swap: 81904 0 81904
>
> Here 24 MB of memory have been used up. Repeating the du seems to have
> little effect. This directory has ~3200 subdirs and 13400 files.
>
> After a few hours use about 200 MB are used, apperently for
> nothing. Killing all processed and unmounting file systems doesn't
> help.
>
> Is this a memory leak? I get the same results with ext2, ext3,
> reiserfs and nfs.
wow!
I've been seeing this, too, but I thought I was just reading something
wrong. Especially after my nightly cron jobs (which involve a 'find
/') run, I'll often find myself with 80% of physical RAM used, and
nobody (as far as 'top' can see) using it. You didn't specify which
kernel you're using, but I'm running 2.4.19-rc1-ac1 plus some patches,
and I've seen it since at least about pre9-ac*. I might try to narrow it
down more if it could be useful.
--hobbs
Andrew Rodland <[email protected]> writes:
> On 21 Jul 2002 16:00:09 +0200
> [email protected] (M) wrote:
>
> >
> > I noticed that doing lots or file accesses causes the used memory to
> > increase, *after* subtracting buffers/cache. Here is an example:
> >
> > $ free
> > total used free shared buffers
> > cached
> > Mem: 773776 30024 743752 0 1992
> > 10424-/+ buffers/cache: 17608 756168
> > Swap: 81904 0 81904
> > $ du > /dev/null
> > $ free
> > total used free shared buffers
> > cached
> > Mem: 773776 78008 695768 0 26328
> > 10472-/+ buffers/cache: 41208 732568
> > Swap: 81904 0 81904
> >
> > Here 24 MB of memory have been used up. Repeating the du seems to have
> > little effect. This directory has ~3200 subdirs and 13400 files.
> >
> > After a few hours use about 200 MB are used, apperently for
> > nothing. Killing all processed and unmounting file systems doesn't
> > help.
> >
> > Is this a memory leak? I get the same results with ext2, ext3,
> > reiserfs and nfs.
>
> wow!
> I've been seeing this, too, but I thought I was just reading something
> wrong. Especially after my nightly cron jobs (which involve a 'find
> /') run, I'll often find myself with 80% of physical RAM used, and
> nobody (as far as 'top' can see) using it. You didn't specify which
> kernel you're using, but I'm running 2.4.19-rc1-ac1 plus some patches,
> and I've seen it since at least about pre9-ac*. I might try to narrow it
> down more if it could be useful.
I forgot to mention the kernel version. It's 2.4.19-rc3. It's been
going on a while, though, before I took the time start looking for it.
--
M?ns Rullg?rd
[email protected]
free don't know about slabcaches. take a look in /proc/slabinfo and see
what's using that memory. it's not a leak, the memory will be free'd
when the machine is under enough memory pressure.
--
/Martin
Never argue with an idiot. They drag you down to their level, then beat
you with experience.
On 21 Jul 2002, M?ns Rullg?rd wrote:
> I noticed that doing lots or file accesses causes the used memory to
> increase, *after* subtracting buffers/cache. Here is an example:
> Here 24 MB of memory have been used up. Repeating the du seems to have
> little effect. This directory has ~3200 subdirs and 13400 files.
> Is this a memory leak? I get the same results with ext2, ext3,
> reiserfs and nfs.
See /proc/slabinfo for the numbers, the memory is most likely
being used in the dentry_cache, the inode_cache and in buffer
heads.
This memory will be reclaimed when the system needs it.
regards,
Rik
--
Bravely reimplemented by the knights who say "NIH".
http://www.surriel.com/ http://distro.conectiva.com/
Rik van Riel <[email protected]> writes:
> > I noticed that doing lots or file accesses causes the used memory to
> > increase, *after* subtracting buffers/cache. Here is an example:
>
> > Here 24 MB of memory have been used up. Repeating the du seems to have
> > little effect. This directory has ~3200 subdirs and 13400 files.
>
> > Is this a memory leak? I get the same results with ext2, ext3,
> > reiserfs and nfs.
>
> See /proc/slabinfo for the numbers, the memory is most likely
> being used in the dentry_cache, the inode_cache and in buffer
> heads.
>
> This memory will be reclaimed when the system needs it.
Does this mean that free and /proc/meminfo are incorrect?
--
M?ns Rullg?rd
[email protected]
On 21 Jul 2002 16:20:39 +0200
Martin Josefsson <[email protected]> wrote:
> free don't know about slabcaches. take a look in /proc/slabinfo and
> see what's using that memory. it's not a leak, the memory will be
> free'd when the machine is under enough memory pressure.
>
Yeah... look at that. looks like I've got quite a bit of memory
invested in inode_cache and dentry_cache. There's no way to have them
reported as "cache" memory anymore?
On Mon, 22 Jul 2002, Andrew Rodland wrote:
> On 21 Jul 2002 16:20:39 +0200
> Martin Josefsson <[email protected]> wrote:
>
> > free don't know about slabcaches. take a look in /proc/slabinfo and
> > see what's using that memory. it's not a leak, the memory will be
> > free'd when the machine is under enough memory pressure.
>
> Yeah... look at that. looks like I've got quite a bit of memory
> invested in inode_cache and dentry_cache. There's no way to have them
> reported as "cache" memory anymore?
Submit patches for procps to make free examine /proc/slabinfo ;)
Rik
--
Bravely reimplemented by the knights who say "NIH".
http://www.surriel.com/ http://distro.conectiva.com/
> > This memory will be reclaimed when the system needs it.
>
> Does this mean that free and /proc/meminfo are incorrect?
By its own definition proc/meminfo is correct. top could go rummaging in
/proc/slabinfo but its questionable if it is meaningful to do so. The
actually "out of memory" case for a virtual memory system is not "no
memory pages free" nor "no memory or swap free" its closer to "working
set plus i/o buffers exceeds memory size".
That isnt something as easy to visualise or compute as "free"
Alan Cox <[email protected]> writes:
> > > This memory will be reclaimed when the system needs it.
> >
> > Does this mean that free and /proc/meminfo are incorrect?
>
> By its own definition proc/meminfo is correct. top could go rummaging in
> /proc/slabinfo but its questionable if it is meaningful to do so. The
> actually "out of memory" case for a virtual memory system is not "no
> memory pages free" nor "no memory or swap free" its closer to "working
> set plus i/o buffers exceeds memory size".
Why can't proc/meminfo report these caches as cached instead of plain
used? Would that be incorrect somehow?
--
M?ns Rullg?rd
[email protected]
On 21 Jul 2002, M?ns Rullg?rd wrote:
> Why can't proc/meminfo report these caches as cached instead of plain
> used? Would that be incorrect somehow?
The kernel exports the usage statistics in /proc/meminfo
and /proc/slabinfo. IMHO it's up to userland to present
this info in a meaningful way, no need to shove that piece
of functionality into the kernel.
regards,
Rik
--
Bravely reimplemented by the knights who say "NIH".
http://www.surriel.com/ http://distro.conectiva.com/
On Sunday 21 July 2002 16:20, Martin Josefsson wrote:
> free don't know about slabcaches. take a look in /proc/slabinfo and see
> what's using that memory. it's not a leak, the memory will be free'd
> when the machine is under enough memory pressure.
What happens when updatedb(1) runs? It does a `find /' and thus loads a lot of
fs related data into memory, which makes the kernel caches grow. But how
aggressive do these caches grow? Does this lead to any swapout because the
kernel likes it better to have some dentries and inodes in memory than
probably not-recently-used user pages?
If so, this would mean that a low-priority job like updatedb(1) makes pages
being swapped out that definitely have a higher priority. For updatedb and
similar things (largish ls -lR's) it would make sense to
load-and-quickly-forget all the inode and dentry cache stuff. Another
optimisation syscall, like madvise?
Stephan
--
"That's interesting. Can it be disabled?" -- someone on LKML, after being
told about the PIV hyperthreading features
On Sun, 21 Jul 2002, Rik van Riel wrote:
> On 21 Jul 2002, M?ns Rullg?rd wrote:
>
> > Why can't proc/meminfo report these caches as cached instead of plain
> > used? Would that be incorrect somehow?
>
> The kernel exports the usage statistics in /proc/meminfo
> and /proc/slabinfo. IMHO it's up to userland to present
> this info in a meaningful way, no need to shove that piece
> of functionality into the kernel.
What? The kernel is putting the data in /proc, it seems as easy to put all
cached memory in the cache bin as anywhere else. It's no more work to do
it right. No one is asking for new functionality, just the current
functionality being correct.
--
bill davidsen <[email protected]>
CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.
On Tue, 30 Jul 2002, Bill Davidsen wrote:
> What? The kernel is putting the data in /proc, it seems as easy to put
> all cached memory in the cache bin as anywhere else. It's no more work
> to do it right. No one is asking for new functionality, just the current
> functionality being correct.
Oh well ... send in those patches.
Rik
--
http://www.linuxsymposium.org/2002/
"You're one of those condescending OLS attendants"
"Here's a nickle kid. Go buy yourself a real t-shirt"
http://www.surriel.com/ http://distro.conectiva.com/