Hello, list,
I use 4 disk nodes with NBD.
All of my nodes have 2GB ram.
But the buffer cache newer rise over 830MB.
Is there some limit?
Where can i change this limit? (if it is)
Thanks,
Janos
[root@st-0001 root]# free
total used free shared buffers cached
Mem: 2073152 933188 1139964 0 836776 43416
-/+ buffers/cache: 52996 2020156
Swap: 0 0 0
[root@st-0001 root]# cat /proc/meminfo
MemTotal: 2073152 kB
MemFree: 1139012 kB
Buffers: 835928 kB
Cached: 43448 kB
SwapCached: 0 kB
Active: 12872 kB
Inactive: 871424 kB
HighTotal: 1179584 kB
HighFree: 1129764 kB
LowTotal: 893568 kB
LowFree: 9248 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
Mapped: 9104 kB
Slab: 30248 kB
CommitLimit: 1036576 kB
Committed_AS: 15428 kB
PageTables: 408 kB
VmallocTotal: 114680 kB
VmallocUsed: 196 kB
VmallocChunk: 114476 kB
[root@st-0001 root]#
> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]] On Behalf Of JaniD++
> Sent: Sunday, December 18, 2005 3:09 PM
> To: [email protected]
> Subject: buffer cache question
>
> Hello, list,
>
> I use 4 disk nodes with NBD.
> All of my nodes have 2GB ram.
>
> But the buffer cache newer rise over 830MB.
>
> Is there some limit?
For writes only 40% of the ram can be "dirty".
> Where can i change this limit? (if it is)
I believe there is a way to change it, I am pretty sure that it has
been discussed on the kernel list a couple of months ago, I
don't remember exactly what the change is, and I think the change
was more complicated that was obvious.
The previous subject on a similar thing was:
"kernel 2.6.13 buffer strangeness"
Roger
>
> Thanks,
> Janos
>
> [root@st-0001 root]# free
> total used free shared
> buffers cached
> Mem: 2073152 933188 1139964 0
> 836776 43416
> -/+ buffers/cache: 52996 2020156
> Swap: 0 0 0
> [root@st-0001 root]# cat /proc/meminfo
> MemTotal: 2073152 kB
> MemFree: 1139012 kB
> Buffers: 835928 kB
> Cached: 43448 kB
> SwapCached: 0 kB
> Active: 12872 kB
> Inactive: 871424 kB
> HighTotal: 1179584 kB
> HighFree: 1129764 kB
> LowTotal: 893568 kB
> LowFree: 9248 kB
> SwapTotal: 0 kB
> SwapFree: 0 kB
> Dirty: 0 kB
> Writeback: 0 kB
> Mapped: 9104 kB
> Slab: 30248 kB
> CommitLimit: 1036576 kB
> Committed_AS: 15428 kB
> PageTables: 408 kB
> VmallocTotal: 114680 kB
> VmallocUsed: 196 kB
> VmallocChunk: 114476 kB
> [root@st-0001 root]#
>
> -
> To unsubscribe from this list: send the line "unsubscribe
> linux-kernel" in the body of a message to
> [email protected] More majordomo info at
> http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
Hi,
I have read back in the kernel-archives, and found this messages, about the
same theme, but there is one difference!
On the old messages:
>Nate Diller wrote:
> just found the culprit. guess i should have read the code the first
> time. get_dirty_limits() in drivers/block/page_writeback.c has a
> hard-coded upper limit to dirty_ratio. it's capped to half of the
> unmapped pages, so maybe 30-40% of your system's memory. so if you are
> brave, just remove the "/ 2" parts from the 'if (dirty_ratio >
> unmapped_ratio / 2) dirty_ratio = unmapped_ratio / 2;' check, and you
> can have all the OOM goodness you want.
...
>I changed that bit of code to:
>
> if (dirty_ratio > unmapped_ratio - 10)
> dirty_ratio = unmapped_ratio - 10;
>
>and added a couple of sanity checks so that it couldn't get below 5 or
above 95.
>
>Then set /proc/sys/vm/dirty_ratio to 95 and dirty_background_ratio to 1.
In this case, this modification is only for the *dirty* memory buffer.
I want to use more buffer *cache*! :-)
The unwritten dirty memory ratio is good enough for me.
Anybody has an idea?
Cheers,
Janos
----- Original Message -----
From: "Roger Heflin" <[email protected]>
To: "'JaniD++'" <[email protected]>; <[email protected]>
Sent: Monday, December 19, 2005 7:15 PM
Subject: RE: buffer cache question
>
>
> > -----Original Message-----
> > From: [email protected]
> > [mailto:[email protected]] On Behalf Of JaniD++
> > Sent: Sunday, December 18, 2005 3:09 PM
> > To: [email protected]
> > Subject: buffer cache question
> >
> > Hello, list,
> >
> > I use 4 disk nodes with NBD.
> > All of my nodes have 2GB ram.
> >
> > But the buffer cache newer rise over 830MB.
> >
> > Is there some limit?
>
> For writes only 40% of the ram can be "dirty".
>
> > Where can i change this limit? (if it is)
>
> I believe there is a way to change it, I am pretty sure that it has
> been discussed on the kernel list a couple of months ago, I
> don't remember exactly what the change is, and I think the change
> was more complicated that was obvious.
>
> The previous subject on a similar thing was:
> "kernel 2.6.13 buffer strangeness"
>
> Roger
>
> >
> > Thanks,
> > Janos
> >
> > [root@st-0001 root]# free
> > total used free shared
> > buffers cached
> > Mem: 2073152 933188 1139964 0
> > 836776 43416
> > -/+ buffers/cache: 52996 2020156
> > Swap: 0 0 0
> > [root@st-0001 root]# cat /proc/meminfo
> > MemTotal: 2073152 kB
> > MemFree: 1139012 kB
> > Buffers: 835928 kB
> > Cached: 43448 kB
> > SwapCached: 0 kB
> > Active: 12872 kB
> > Inactive: 871424 kB
> > HighTotal: 1179584 kB
> > HighFree: 1129764 kB
> > LowTotal: 893568 kB
> > LowFree: 9248 kB
> > SwapTotal: 0 kB
> > SwapFree: 0 kB
> > Dirty: 0 kB
> > Writeback: 0 kB
> > Mapped: 9104 kB
> > Slab: 30248 kB
> > CommitLimit: 1036576 kB
> > Committed_AS: 15428 kB
> > PageTables: 408 kB
> > VmallocTotal: 114680 kB
> > VmallocUsed: 196 kB
> > VmallocChunk: 114476 kB
> > [root@st-0001 root]#
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe
> > linux-kernel" in the body of a message to
> > [email protected] More majordomo info at
> > http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at http://www.tux.org/lkml/
> >
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
----- Original Message -----
From: Nate Diller
To: JaniD++
Cc: Roger Heflin ; [email protected]
Sent: Friday, December 23, 2005 9:58 AM
Subject: Re: buffer cache question
On 11/21/05, JaniD++ <[email protected]> wrote:
Hi,
I have read back in the kernel-archives, and found this messages, about the
same theme, but there is one difference!
On the old messages:
>Nate Diller wrote:
> just found the culprit. guess i should have read the code the first
> time. get_dirty_limits() in drivers/block/page_writeback.c has a
> hard-coded upper limit to dirty_ratio. it's capped to half of the
> unmapped pages, so maybe 30-40% of your system's memory. so if you are
> brave, just remove the "/ 2" parts from the 'if (dirty_ratio >
> unmapped_ratio / 2) dirty_ratio = unmapped_ratio / 2;' check, and you
> can have all the OOM goodness you want.
...
>I changed that bit of code to:
>
> if (dirty_ratio > unmapped_ratio - 10)
> dirty_ratio = unmapped_ratio - 10;
>
>and added a couple of sanity checks so that it couldn't get below 5 or
above 95.
>
>Then set /proc/sys/vm/dirty_ratio to 95 and dirty_background_ratio to 1.
In this case, this modification is only for the *dirty* memory buffer.
I want to use more buffer *cache*! :-)
The unwritten dirty memory ratio is good enough for me.
Anybody has an idea?
<snip>
> > [root@st-0001 root]# free
> > total used free shared
> > buffers cached
> > Mem: 2073152 933188 1139964 0
> > 836776 43416
> > -/+ buffers/cache: 52996 2020156
> > Swap: 0 0 0
> > [root@st-0001 root]# cat /proc/meminfo
> > MemTotal: 2073152 kB
> > MemFree: 1139012 kB
> > Buffers: 835928 kB
> > Cached: 43448 kB
> > SwapCached: 0 kB
> > Active: 12872 kB
> > Inactive: 871424 kB
> > HighTotal: 1179584 kB
> > HighFree: 1129764 kB
> > LowTotal: 893568 kB
> > LowFree: 9248 kB
> > SwapTotal: 0 kB
> > SwapFree: 0 kB
> > Dirty: 0 kB
> > Writeback: 0 kB
> > Mapped: 9104 kB
> > Slab: 30248 kB
> > CommitLimit: 1036576 kB
> > Committed_AS: 15428 kB
> > PageTables: 408 kB
> > VmallocTotal: 114680 kB
> > VmallocUsed: 196 kB
> > VmallocChunk: 114476 kB
> > [root@st-0001 root]#
looks like you're barely using any of your high memory. maybe NBD doesn't
have highmem support. what file system are you using?
NATE
I cannot understant this.
NBD need to support highmem for buffering?
If know right, the kernel does buffering, not NBD!
But the kernel only use ~830MB for buffer cache instead of dinamically use
all free memory like page cache.
This is one raw disk node, independent from file system.
Cheers,
Janos
> ----- Original Message -----
> From: Nate Diller
> To: JaniD++
> Cc: Roger Heflin ; [email protected]
> Sent: Friday, December 23, 2005 9:58 AM
> Subject: Re: buffer cache question
>
> looks like you're barely using any of your high memory. maybe NBD doesn't
> have highmem support. what file system are you using?
>
> NATE
>
>
> I cannot understant this.
> NBD need to support highmem for buffering?
> If know right, the kernel does buffering, not NBD!
> But the kernel only use ~830MB for buffer cache instead of dinamically use
> all free memory like page cache.
>
> This is one raw disk node, independent from file system.
>
this is an NBD client, using CONFIG_BLK_DEV_NBD? on the 2.6 series
kernel? if so, then it, like any other kernel component using the
page cache, needs to explicity use kmap/kunmap to make use of memory
in the high memory zone. on a 32 bit machine, any pages above the 896
meg mark are treated specially inside the linux kernel (see
http://kerneltrap.org/node/2450).
if you don't have highmem support (CONFIG_HIGHMEM4G) enabled, then
enabling that should fix it. if you already have it enabled (it
looked like it to me, based on your /proc/meminfo) then there is a bug
somewhere.
it would seem from a brief inspection that the send/recv_bvec
functions in nbd (2.6.13) do use kmap. I don't know the nbd code very
well, it seems that Pavel Machek wrote the code, he or block layer
maintainer Jens Axboe may know something I don't. So if enabling
highmem in your .config doesn't help, try CC'ing them with your issue.
in the mean time, one of the memory split patches, such as the 4G:4G
patch, should get things working.
NATE
Hi,
----- Original Message -----
From: "Nate Diller" <[email protected]>
To: "JaniD++" <[email protected]>
Cc: <[email protected]>; <[email protected]>
Sent: Thursday, December 29, 2005 5:10 AM
Subject: Re: buffer cache question
> > ----- Original Message -----
> > From: Nate Diller
> > To: JaniD++
> > Cc: Roger Heflin ; [email protected]
> > Sent: Friday, December 23, 2005 9:58 AM
> > Subject: Re: buffer cache question
> >
> > looks like you're barely using any of your high memory. maybe NBD
doesn't
> > have highmem support. what file system are you using?
> >
> > NATE
> >
> >
> > I cannot understant this.
> > NBD need to support highmem for buffering?
> > If know right, the kernel does buffering, not NBD!
> > But the kernel only use ~830MB for buffer cache instead of dinamically
use
> > all free memory like page cache.
> >
> > This is one raw disk node, independent from file system.
> >
>
> this is an NBD client, using CONFIG_BLK_DEV_NBD? on the 2.6 series
> kernel? if so, then it, like any other kernel component using the
> page cache, needs to explicity use kmap/kunmap to make use of memory
> in the high memory zone. on a 32 bit machine, any pages above the 896
> meg mark are treated specially inside the linux kernel (see
> http://kerneltrap.org/node/2450).
Yes, i have read this helpful document, but there is about the memory usage,
mapping, and not exactly the buffering/caching.
Yes, i use the 2.6 series NBD.
>
> if you don't have highmem support (CONFIG_HIGHMEM4G) enabled, then
> enabling that should fix it. if you already have it enabled (it
> looked like it to me, based on your /proc/meminfo) then there is a bug
> somewhere.
Yes, it is enabled.
>
> it would seem from a brief inspection that the send/recv_bvec
> functions in nbd (2.6.13) do use kmap. I don't know the nbd code very
> well, it seems that Pavel Machek wrote the code, he or block layer
> maintainer Jens Axboe may know something I don't. So if enabling
> highmem in your .config doesn't help, try CC'ing them with your issue.
> in the mean time, one of the memory split patches, such as the 4G:4G
> patch, should get things working.
The NBD use buffer cache (i am sure), but the highest limit is about
830-840M.
Where can i get the split patch?
Thanks for helpful answers!
Cheers,
Janos
>
> NATE