In a personal email, Mike Galbraith wrote to me:
| On Sat, 8 Dec 2001, Leigh Orf wrote:
|
| > inode_cache 439584 439586 512 62798 62798 1
| > dentry_cache 454136 454200 128 15140 15140 1
|
| I'd try moving shrink_[id]cache_memory to the very top of vmscan.c::shrink_caches.
|
| -Mike
Mike,
I tried what you suggested starting with a stock 2.4.16 kernel, and it
did fix the problem with 2.4.16 ENOMEM being returned.
Now with that change and after updatedb runs, here's what the memory
situation looks like. Note inode_cache and dentry_cache are almost
nothing. Dunno if that's a good thing or not, but I'd definitely
consider this for a patch.
home[1002]:/home/orf% free
total used free shared buffers cached
Mem: 1029828 1020316 9512 0 719596 145768
-/+ buffers/cache: 154952 874876
Swap: 2064344 0 2064344
home[1003]:/home/orf% cat /proc/meminfo
total: used: free: shared: buffers: cached:
Mem: 1054543872 1044979712 9564160 0 736870400 149282816
Swap: 2113888256 0 2113888256
MemTotal: 1029828 kB
MemFree: 9340 kB
MemShared: 0 kB
Buffers: 719600 kB
Cached: 145784 kB
SwapCached: 0 kB
Active: 462276 kB
Inactive: 481572 kB
HighTotal: 130992 kB
HighFree: 2044 kB
LowTotal: 898836 kB
LowFree: 7296 kB
SwapTotal: 2064344 kB
SwapFree: 2064344 kB
home[1001]:/home/orf% cat /proc/slabinfo
slabinfo - version: 1.1
kmem_cache 65 68 112 2 2 1
ip_conntrack 13 30 384 3 3 1
nfs_write_data 0 0 384 0 0 1
nfs_read_data 0 0 384 0 0 1
nfs_page 0 0 128 0 0 1
ip_fib_hash 10 112 32 1 1 1
urb_priv 0 0 64 0 0 1
clip_arp_cache 0 0 128 0 0 1
ip_mrt_cache 0 0 128 0 0 1
tcp_tw_bucket 0 30 128 0 1 1
tcp_bind_bucket 8 112 32 1 1 1
tcp_open_request 0 0 128 0 0 1
inet_peer_cache 1 59 64 1 1 1
ip_dst_cache 21 40 192 2 2 1
arp_cache 3 30 128 1 1 1
blkdev_requests 640 660 128 22 22 1
journal_head 0 0 48 0 0 1
revoke_table 0 0 12 0 0 1
revoke_record 0 0 32 0 0 1
dnotify cache 0 0 20 0 0 1
file lock cache 2 42 92 1 1 1
fasync cache 2 202 16 1 1 1
uid_cache 5 112 32 1 1 1
skbuff_head_cache 293 300 192 15 15 1
sock 209 213 1280 70 71 1
sigqueue 2 29 132 1 1 1
cdev_cache 21 295 64 3 5 1
bdev_cache 8 59 64 1 1 1
mnt_cache 19 59 64 1 1 1
inode_cache 7151 16905 512 2412 2415 1
dentry_cache 3043 12630 128 421 421 1
dquot 0 0 128 0 0 1
filp 1990 2010 128 67 67 1
names_cache 0 2 4096 0 2 1
buffer_head 220278 220350 128 7344 7345 1
mm_struct 64 80 192 4 4 1
vm_area_struct 2779 3180 128 93 106 1
fs_cache 63 118 64 2 2 1
files_cache 63 72 448 7 8 1
signal_act 71 72 1344 24 24 1
size-131072(DMA) 0 0 131072 0 0 32
size-131072 0 0 131072 0 0 32
size-65536(DMA) 0 0 65536 0 0 16
size-65536 1 1 65536 1 1 16
size-32768(DMA) 0 0 32768 0 0 8
size-32768 1 1 32768 1 1 8
size-16384(DMA) 0 0 16384 0 0 4
size-16384 1 5 16384 1 5 4
size-8192(DMA) 0 0 8192 0 0 2
size-8192 4 5 8192 4 5 2
size-4096(DMA) 0 0 4096 0 0 1
size-4096 72 78 4096 72 78 1
size-2048(DMA) 0 0 2048 0 0 1
size-2048 61 96 2048 32 48 1
size-1024(DMA) 0 0 1024 0 0 1
size-1024 1475 2988 1024 532 747 1
size-512(DMA) 0 0 512 0 0 1
size-512 1650 4168 512 351 521 1
size-256(DMA) 0 0 256 0 0 1
size-256 486 2340 256 45 156 1
size-128(DMA) 2 30 128 1 1 1
size-128 3483 10050 128 230 335 1
size-64(DMA) 0 0 64 0 0 1
size-64 1792 3304 64 56 56 1
size-32(DMA) 34 59 64 1 1 1
size-32 4159 11092 64 159 188 1
Leigh Orf
On Sun, 9 Dec 2001, Leigh Orf wrote:
> In a personal email, Mike Galbraith wrote to me:
>
> | On Sat, 8 Dec 2001, Leigh Orf wrote:
> |
> | > inode_cache 439584 439586 512 62798 62798 1
> | > dentry_cache 454136 454200 128 15140 15140 1
> |
> | I'd try moving shrink_[id]cache_memory to the very top of vmscan.c::shrink_caches.
> |
> | -Mike
>
> Mike,
>
> I tried what you suggested starting with a stock 2.4.16 kernel, and it
> did fix the problem with 2.4.16 ENOMEM being returned.
>
> Now with that change and after updatedb runs, here's what the memory
> situation looks like. Note inode_cache and dentry_cache are almost
> nothing. Dunno if that's a good thing or not, but I'd definitely
Almost nothing isn't particularly good after updatedb ;-)
> consider this for a patch.
No, but those do need faster pruning imho. The growth rate can be
really really fast at times.
-Mike
On Sun, 9 Dec 2001, Leigh Orf wrote:
> buffer_head 220278 220350 128 7344 7345 1
P.S.
These don't bother my box much.. a little though.
-Mike
What about moving the calls to shrink_[di]cache_memory() after the
nr_pages check after the call to kmem_cache_reap? Or perhaps keep it at
the beginning, but only call it after priority has gone a number of
notches down from DEF_PRIORITY?
Something like that seems like the only obvious way to balance how soon
these caches are flushed without over- or under-kill.
Also, shouldn't shrink_dqcache_memory use priority rather than
DEF_PRIORITY? I'm also not sure what the reasoning is behind the
nr_pages=chunk_size reset.
In the case that Leigh and I are seeing (and my readprofile runs) it
sounds like shrink_cache is getting called a ton, while the bloated d/i
caches are flushed too little, too late.
Just $0.02 from a newby. ;)
Thanks for the tip, Mike,
--
Ken.
[email protected]
On Sun, Dec 09, 2001 at 06:17:11PM +0100, Mike Galbraith wrote:
| On Sun, 9 Dec 2001, Leigh Orf wrote:
|
| > In a personal email, Mike Galbraith wrote to me:
| >
| > | On Sat, 8 Dec 2001, Leigh Orf wrote:
| > |
| > | > inode_cache 439584 439586 512 62798 62798 1
| > | > dentry_cache 454136 454200 128 15140 15140 1
| > |
| > | I'd try moving shrink_[id]cache_memory to the very top of vmscan.c::shrink_caches.
| > |
| > | -Mike
| >
| > Mike,
| >
| > I tried what you suggested starting with a stock 2.4.16 kernel, and it
| > did fix the problem with 2.4.16 ENOMEM being returned.
| >
| > Now with that change and after updatedb runs, here's what the memory
| > situation looks like. Note inode_cache and dentry_cache are almost
| > nothing. Dunno if that's a good thing or not, but I'd definitely
|
| Almost nothing isn't particularly good after updatedb ;-)
|
| > consider this for a patch.
|
| No, but those do need faster pruning imho. The growth rate can be
| really really fast at times.
|
| -Mike
On Mon, 10 Dec 2001, Ken Brownfield wrote:
> What about moving the calls to shrink_[di]cache_memory() after the
> nr_pages check after the call to kmem_cache_reap? Or perhaps keep it at
> the beginning, but only call it after priority has gone a number of
> notches down from DEF_PRIORITY?
>
> Something like that seems like the only obvious way to balance how soon
> these caches are flushed without over- or under-kill.
So obvious that it's been re-introduced 3 times now even though
it broke each time. ;)
The only way to get stuff balanced somewhat is to call the
shrink functions unconditionally. It's not optimally balanced,
but at least the cache will stay reasonably small while still
being able to grow under load.
Rik
--
DMCA, SSSCA, W3C? Who cares? http://thefreeworld.net/
http://www.surriel.com/ http://distro.conectiva.com/
Rik van Riel wrote:
| On Mon, 10 Dec 2001, Ken Brownfield wrote:
|
| > What about moving the calls to shrink_[di]cache_memory()
| > after the nr_pages check after the call to kmem_cache_reap?
| > Or perhaps keep it at the beginning, but only call it
| > after priority has gone a number of notches down from
| > DEF_PRIORITY?
| >
| > Something like that seems like the only obvious way to
| > balance how soon these caches are flushed without over- or
| > under-kill.
|
| So obvious that it's been re-introduced 3 times now even
| though it broke each time. ;)
And in fact, after furthur playing around with the "fixed" version
(moving shrink_[id]cache_memory to the top of vmscan.c::shrink_caches)
I find that I still will get ENOMEM after updatedb occasionally. Less
often than before, but it still happens.
| The only way to get stuff balanced somewhat is to call
| the shrink functions unconditionally. It's not optimally
| balanced, but at least the cache will stay reasonably small
| while still being able to grow under load.
I just can't understand why the kernel wouldn't tag application memory
as being more important tan buff/cache and free up some of that stuff
when an application calls for it. I mean, it won't even use the gobs of
swap I have. That just seems to be a plain ol' bug to me.
Leigh Orf
> -----Original Message-----
> I just can't understand why the kernel wouldn't tag application memory
> as being more important than buff/cache and free up some of that stuff
> when an application calls for it. I mean, it won't even use the gobs of
> swap I have. That just seems to be a plain ol' bug to me.
It's not strictly a bug ... it's a design decision that has unfortunate
consequences. A simple fix would be to allow the system administrator to set
an upper limit on the size of the page cache.
--
M. Edward Borasky
znmeb@@borasky-research.net
http://www.borasky-research.net
"M. Edward Borasky" wrote:
>
> > -----Original Message-----
> > I just can't understand why the kernel wouldn't tag application memory
> > as being more important than buff/cache and free up some of that stuff
> > when an application calls for it. I mean, it won't even use the gobs of
> > swap I have. That just seems to be a plain ol' bug to me.
>
> It's not strictly a bug ... it's a design decision that has unfortunate
> consequences. A simple fix would be to allow the system administrator to set
> an upper limit on the size of the page cache.
I'd say he has found a bug. Merely prioritizing cache over apps
so apps go to swap is a design desicion. Killing the app
for OOM reasons when there is free swap and/or cache
that can be freed up _is_ a bug.
Helge Hafting
On Mon, 10 Dec 2001, Leigh Orf wrote:
> And in fact, after furthur playing around with the "fixed" version
> (moving shrink_[id]cache_memory to the top of vmscan.c::shrink_caches)
> I find that I still will get ENOMEM after updatedb occasionally. Less
> often than before, but it still happens.
Yes.. reasonable.
-Mike