Not sure where to begin so here goes anway. Today I did an rsync backup
of a server with 2million+ files. Before doing so the used memory on the
server this was initiated from was under 200meg (excluding buffers and
cache). During the rsync the memory used grew to just shy of 1.6gig and
now, about 2 hours after the rsync has well and truly finished, the used
memory is at 1.23gig. This is what free reports:
total used free shared buffers cached
Mem: 2058128 1994468 63660 0 688604 11432
-/+ buffers/cache: 1294432 763696
Swap: 1048568 0 1048568
There are 75 processes on the box of which almost 47 are kernel
processes + init. Of the rest, the top 3 have an RSS of 9.4meg, 6.2meg
and 4.8meg, 7 are at around 2meg and the rest are below 2meg with the
majority below 1. So unless I'm misunderstanding something, processess
alone do not account for the amount of used memory.
The destination of the rsync was an ext3 filesystem over raid5 over ahci
sata.
I've included /proc/meminfo, /proc/slabinfo and config.gz. If there's
anything else please shout.
--
"To the extent that we overreact, we proffer the terrorists the
greatest tribute."
- High Court Judge Michael Kirby
Hi,
On Jan 17, 2008 8:34 AM, CaT <[email protected]> wrote:
> There are 75 processes on the box of which almost 47 are kernel
> processes + init. Of the rest, the top 3 have an RSS of 9.4meg, 6.2meg
> and 4.8meg, 7 are at around 2meg and the rest are below 2meg with the
> majority below 1. So unless I'm misunderstanding something, processess
> alone do not account for the amount of used memory.
Looking at /proc/slabinfo, 965 MB is used by the ext3 inode cache and
144 MB by the dentry cache.
Pekka
On Thu, 2008-01-17 at 17:34 +1100, CaT wrote:
> Not sure where to begin so here goes anway. Today I did an rsync backup
> of a server with 2million+ files. Before doing so the used memory on the
> server this was initiated from was under 200meg (excluding buffers and
> cache). During the rsync the memory used grew to just shy of 1.6gig and
> now, about 2 hours after the rsync has well and truly finished, the used
> memory is at 1.23gig. This is what free reports:
>
> total used free shared buffers cached
> Mem: 2058128 1994468 63660 0 688604 11432
> -/+ buffers/cache: 1294432 763696
> Swap: 1048568 0 1048568
>
> There are 75 processes on the box of which almost 47 are kernel
> processes + init. Of the rest, the top 3 have an RSS of 9.4meg, 6.2meg
> and 4.8meg, 7 are at around 2meg and the rest are below 2meg with the
> majority below 1. So unless I'm misunderstanding something, processess
> alone do not account for the amount of used memory.
>
> The destination of the rsync was an ext3 filesystem over raid5 over ahci
> sata.
>
> I've included /proc/meminfo, /proc/slabinfo and config.gz. If there's
> anything else please shout.
How much memory does:
echo 3 > /proc/sys/vm/drop_caches
gain you?
> ext3_inode_cache 1235577 1240565 768 5 1 : tunables 54 27 8 : slabdata 248113 248113 0
> dentry 703661 749797 200 19 1 : tunables 120 60 8 : slabdata 39463 39463 0
> buffer_head 174535 209087 104 37 1 : tunables 120 60 8 : slabdata 5651 5651 0
would get freed by doing that.
this one:
> size-64 537590 850249 64 59 1 : tunables 120 60 8 : slabdata 14411 14411 0
I'm unsure about, if that one sticks around that'd be something to worry
about. See if you can monitor this value and try to determine:
- if it ever drops
- what makes it grow (fastest)
I guess we could stick some instrumentation in there to track that
bucket.
On Thu, Jan 17, 2008 at 12:22:03PM +0100, Peter Zijlstra wrote:
> On Thu, 2008-01-17 at 17:34 +1100, CaT wrote:
> > cache). During the rsync the memory used grew to just shy of 1.6gig and
> > now, about 2 hours after the rsync has well and truly finished, the used
> > memory is at 1.23gig. This is what free reports:
> >
> > total used free shared buffers cached
> > Mem: 2058128 1994468 63660 0 688604 11432
> > -/+ buffers/cache: 1294432 763696
> > Swap: 1048568 0 1048568
>
> How much memory does:
>
> echo 3 > /proc/sys/vm/drop_caches
>
> gain you?
56M used now. Should all this cache usage not be counted towards the
'Cached' entry in meminfo rather then getting counted as part of used
ram. I assume that this would not cause an oom situation and would be
freed up if all that memory really did need to be used.
> > ext3_inode_cache 1235577 1240565 768 5 1 : tunables 54 27 8 : slabdata 248113 248113 0
> > dentry 703661 749797 200 19 1 : tunables 120 60 8 : slabdata 39463 39463 0
> > buffer_head 174535 209087 104 37 1 : tunables 120 60 8 : slabdata 5651 5651 0
>
> would get freed by doing that.
They were indeed.
> this one:
>
> > size-64 537590 850249 64 59 1 : tunables 120 60 8 : slabdata 14411 14411 0
>
> I'm unsure about, if that one sticks around that'd be something to worry
> about. See if you can monitor this value and try to determine:
This went away also.
> - if it ever drops
> - what makes it grow (fastest)
>
> I guess we could stick some instrumentation in there to track that
> bucket.
Might help prevent upraised eyebrows or worse. :)
--
"To the extent that we overreact, we proffer the terrorists the
greatest tribute."
- High Court Judge Michael Kirby
On Thu, 2008-01-17 at 22:40 +1100, CaT wrote:
> On Thu, Jan 17, 2008 at 12:22:03PM +0100, Peter Zijlstra wrote:
> > On Thu, 2008-01-17 at 17:34 +1100, CaT wrote:
> > > cache). During the rsync the memory used grew to just shy of 1.6gig and
> > > now, about 2 hours after the rsync has well and truly finished, the used
> > > memory is at 1.23gig. This is what free reports:
> > >
> > > total used free shared buffers cached
> > > Mem: 2058128 1994468 63660 0 688604 11432
> > > -/+ buffers/cache: 1294432 763696
> > > Swap: 1048568 0 1048568
> >
> > How much memory does:
> >
> > echo 3 > /proc/sys/vm/drop_caches
> >
> > gain you?
>
> 56M used now.
Good :-)
> Should all this cache usage not be counted towards the
> 'Cached' entry in meminfo rather then getting counted as part of used
> ram.
Cached is only the page-cache, not all the other caches we have..
This someones confuses people, but one gets used to it. slabinfo allows
one to easily view others.
> I assume that this would not cause an oom situation and would be
> freed up if all that memory really did need to be used.
Correct.
On Thu, Jan 17, 2008 at 01:07:12PM +0100, Peter Zijlstra wrote:
> On Thu, 2008-01-17 at 22:40 +1100, CaT wrote:
> > > How much memory does:
> > >
> > > echo 3 > /proc/sys/vm/drop_caches
> > >
> > > gain you?
> >
> > 56M used now.
>
> Good :-)
Indeed. :)
> > Should all this cache usage not be counted towards the
> > 'Cached' entry in meminfo rather then getting counted as part of used
> > ram.
>
> Cached is only the page-cache, not all the other caches we have..
> This someones confuses people, but one gets used to it. slabinfo allows
> one to easily view others.
So what would I have to do to find out the real amount of memory free?
Would that be the MemFree field in /proc/meminfo plus, from
/proc/slabinfo, *_cache and size-*?
If that's the case it would seem to be somewhat of a pain to get and
kind of out of left field as I'd say most people would expect MemFree to
indicate the amount of memory that's no longer freely available
(ignoring swapping it out for simplicities sake).
I'm wondering as I'm trying to monitor this stuff and I've just found
out that the easily accessible values are not as useful as I previously
thought.
--
"To the extent that we overreact, we proffer the terrorists the
greatest tribute."
- High Court Judge Michael Kirby
On Thu, 2008-01-17 at 23:14 +1100, CaT wrote:
> On Thu, Jan 17, 2008 at 01:07:12PM +0100, Peter Zijlstra wrote:
> > On Thu, 2008-01-17 at 22:40 +1100, CaT wrote:
> > > > How much memory does:
> > > >
> > > > echo 3 > /proc/sys/vm/drop_caches
> > > >
> > > > gain you?
> > >
> > > 56M used now.
> >
> > Good :-)
>
> Indeed. :)
>
> > > Should all this cache usage not be counted towards the
> > > 'Cached' entry in meminfo rather then getting counted as part of used
> > > ram.
> >
> > Cached is only the page-cache, not all the other caches we have..
> > This someones confuses people, but one gets used to it. slabinfo allows
> > one to easily view others.
>
> So what would I have to do to find out the real amount of memory free?
> Would that be the MemFree field in /proc/meminfo plus, from
> /proc/slabinfo, *_cache and size-*?
MemFree is the actual amount of free memory. Page reclaim and such
might, on memory pressure, free memory from cached, buffered and some
slabs that have shrinkers.
> If that's the case it would seem to be somewhat of a pain to get and
> kind of out of left field as I'd say most people would expect MemFree to
> indicate the amount of memory that's no longer freely available
> (ignoring swapping it out for simplicities sake).
I'm somewhat confused as to what you're saying. How would anyone expect
MemFree to be memory that is _not_ freely available?
Or are you asking how to compute the amount of freeable memory, that is
memory that isn't currently free, but could be freed up under pressure?
If that is indeed your question, then yes, thats rather hard as slabinfo
and the like don't indicate which buckets have shrinkers (but even if
they would have shrinkers there is no guarantee they'd be able to shrink
100%)
Hi
> > > Should all this cache usage not be counted towards the
> > > 'Cached' entry in meminfo rather then getting counted as part of used
> > > ram.
> >
> > Cached is only the page-cache, not all the other caches we have..
> > This someones confuses people, but one gets used to it. slabinfo allows
> > one to easily view others.
>
> So what would I have to do to find out the real amount of memory free?
> Would that be the MemFree field in /proc/meminfo plus, from
> /proc/slabinfo, *_cache and size-*?
SReclaimable field can't fill requilrement? :)
-kosaki
On Thu, Jan 17, 2008 at 01:21:51PM +0100, Peter Zijlstra wrote:
> > If that's the case it would seem to be somewhat of a pain to get and
> > kind of out of left field as I'd say most people would expect MemFree to
> > indicate the amount of memory that's no longer freely available
> > (ignoring swapping it out for simplicities sake).
>
> I'm somewhat confused as to what you're saying. How would anyone expect
> MemFree to be memory that is _not_ freely available?
I think we have conflicting ideas of what 'freely available' means here
which is causing the confusion. :)
> Or are you asking how to compute the amount of freeable memory, that is
> memory that isn't currently free, but could be freed up under pressure?
Yes. I think I am. (heh)
> If that is indeed your question, then yes, thats rather hard as slabinfo
> and the like don't indicate which buckets have shrinkers (but even if
Ah. Foo.
> they would have shrinkers there is no guarantee they'd be able to shrink
> 100%)
Whacky.
:/
--
"To the extent that we overreact, we proffer the terrorists the
greatest tribute."
- High Court Judge Michael Kirby
On Thu, Jan 17, 2008 at 09:22:50PM +0900, KOSAKI Motohiro wrote:
> Hi
>
> > > > Should all this cache usage not be counted towards the
> > > > 'Cached' entry in meminfo rather then getting counted as part of used
> > > > ram.
> > >
> > > Cached is only the page-cache, not all the other caches we have..
> > > This someones confuses people, but one gets used to it. slabinfo allows
> > > one to easily view others.
> >
> > So what would I have to do to find out the real amount of memory free?
> > Would that be the MemFree field in /proc/meminfo plus, from
> > /proc/slabinfo, *_cache and size-*?
>
> SReclaimable field can't fill requilrement? :)
Ah. Hmm. *pokes about* That does seem to correspond pretty closely to
what happens with the echo 3 >/proc/.... Though it never seems to get to
0 (~4 meg is as close as it gets on this box) which seems to jive with
what Peter said in the other email.
Hmm. I think tomorrow I'll be writing free.pl and modifying my graphing
and monitoring stuff to take this into account somehow. :)
Thanks!
--
"To the extent that we overreact, we proffer the terrorists the
greatest tribute."
- High Court Judge Michael Kirby