hi,
this query arises from some small experiments i performed with my linux nfs client (2.4.18).
Conclusions first:- There is a limit of 8192 pages on the size of the local client cache and this space is shared between the various nfs mounts currently operating on this machine with this machine as the client.
Question 1. I haven't been able to trace this 8192 pages limit to any data structure or functionality - so please help
Question 2 : This question is regarding the management of this shared cache, space freeing mechanism in the cache to make space for fresh reads etc and arises from the following observations :-
I conducted the following experiments:- 'n' nfs clients (numbered 1, 2, 3) are currently mounted, reading 3 different non-overlapping directory trees of some server. The events proceed as follows when n=2,
client 1 reads 4096 4k pages from the server. Then, client 2 reads 4096 4k pages(different from those read by 1) from the server.
client 1 again reads 4096 pages from the server followed by client 2 repeating the same. Except for the first read whent the cache is cold, no later reads translate into an on-the-wire read.
<< just to add: i concluded of the 8192 pages limit from the following 3 observations: (1). when i made the two clients read 8192 pages each alternatively, every read all through caused an on-the-wire-lookup, (2). when only one client was repeatedly reading 8192 pages, the cache completely satisfied all but the first request, (3). when only one client was repeatedly reading 16384 pages, every read translated into an on-the-wire read.>>
Now i made n=3. (no. of pages being read by each client continue to be 4096)
I tried out two different orderings of client reads - <1,2,3,1,2,3> and <1,2,3,2,1>
In both these cases, __ALL__ reads translate into an on-the-wire read. I expected some requirement based freeing of space in the client cache to make space for say 3 after 1 and 2 have done their reads eg. if global lru based freeing then in <123123>, 3 would cause 1's 4096 pages to be thrown out of the cache to make space for its own 4096. And after that in the next cycle, 1 would cause the same to 2... which would mean all reads=on-the-wire-read. But this scenario would imply that in <12321>, the second time 2 is around it would find its data still in the cache, only 1 will suffer etc but this wasn't found to be the case. So when space is needed - the space freeing protocol recklessly frees all possible space ? no- global lru based freeing mechanism ? at least let me know where lies the corresponding code. thanks a lot.
Question 3: server->nr_requests is limited by the hard limit MAX_REQUEST_HARD. what is this server->nr_requests ? outstanding unsatisfied requests ? Again where is the limit for cache size ?
Thanks in advance
cheers
Abhishek
---------------------------------
Do you Yahoo!?
Yahoo! Web Hosting - establish your business online
hi-
you didn't tell us how large your client's RAM is,
what your mount options are, what other programs
are running on the clients, nor how long each client
takes to read (attribute cache invalidation might
cause each client to purge its cache, for instance).
as far as i know there is no 8192 page limit in the
NFS client.
-----Original Message-----
From: Abhishek Rai [mailto:[email protected]]=20
Sent: Tuesday, March 11, 2003 2:33 AM
To: [email protected]
Subject: [NFS] regarding size limit of local client cache
hi,=20
this query arises from some small experiments i performed with my linux
nfs client (2.4.18).=20
Conclusions first:- There is a limit of 8192 pages on the size of the
local client cache and this space is shared between the various nfs
mounts currently operating on this machine with this machine as the
client.
Question 1. I haven't been able to trace this 8192 pages limit to any
data structure or functionality - so please help
Question 2 : This question is regarding the management of this shared
cache, space freeing mechanism in the cache to make space for fresh
reads etc and arises from the following observations :-
I conducted the following experiments:- 'n' nfs clients (numbered 1, 2,
3) are currently mounted, reading 3 different non-overlapping directory
trees of some server. The events proceed as follows when n=3D2,
client 1 reads 4096 4k pages from the server. Then, client 2 reads 4096
4k pages(different from those read by 1) from the server.
client 1 again reads 4096 pages from the server followed by client 2
repeating the same. Except for the first read whent the cache is cold,
no later reads translate into an on-the-wire read.
<< just to add: i concluded of the 8192 pages limit from the following 3
observations: (1). when i made the two clients read 8192 pages each
alternatively, every read all through caused an on-the-wire-lookup, (2).
when only one client was repeatedly reading 8192 pages, the cache
completely satisfied all but the first request, (3). when only one
client was repeatedly reading 16384 pages, every read translated into an
on-the-wire read.>>
Now i made n=3D3. (no. of pages being read by each client continue to be
4096)
I tried out two different orderings of client reads - <1,2,3,1,2,3> and
<1,2,3,2,1>
In both these cases, __ALL__ reads translate into an on-the-wire read. I
expected some requirement based freeing of space in the client cache to
make space for say 3 after 1 and 2 have done their reads eg. if global
lru based freeing then in <123123>, 3 would cause 1's 4096 pages to be
thrown out of the cache to make space for its own 4096. And after that
in the next cycle, 1 would cause the same to 2... which would mean all
reads=3Don-the-wire-read. But this scenario would imply that in <12321>,
the second time 2 is around it would find its data still in the cache,
only 1 will suffer etc but this wasn't found to be the case. So when
space is needed - the space freeing protocol recklessly frees all
possible space ? no- global lru based freeing mechanism ? at least let
me know where lies the corresponding code. thanks a lot.
Question 3: server->nr_requests is limited by the hard limit
MAX_REQUEST_HARD. what is this server->nr_requests ? outstanding
unsatisfied requests ? Again where is the limit for cache size ?
Thanks in advance
cheers
Abhishek
Do you Yahoo!?
Yahoo! Web Hosting - establish your business online
-------------------------------------------------------
This SF.net email is sponsored by:Crypto Challenge is now open!
Get cracking and register here for some mind boggling fun and
the chance of winning an Apple iPod:
http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0031en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
hi,
my client RAM size is 128 MB. No special mount options => default values set. i performed this test with _NO_ other extra programs running(apart from linux regulars). This is one important information i missed out:- Each client reads all its required pages sequentially, and only after a minimum period of 40 seconds do I allow another client to start with its own reading exercises. So by the time a client comes around again and looks for its cached data it is at least 80 seconds since it was read.
Also, there isn't a question of attribute cache invalidation (causing the client to purge its cache etc) because the original files on the server are not modified by anyone(no nfs client or even server applications). <<So is it more a question of the nfs client managing a "cumulative" cache space for all the current mounts having this machine as the client?>>. In this scenario howz the observed 8192 pages limit explained and how hard/flexible this limit is depending on general load on the client machine.
thanks
"Lever, Charles" <[email protected]> wrote: hi-
you didn't tell us how large your client's RAM is,
what your mount options are, what other programs
are running on the clients, nor how long each client
takes to read (attribute cache invalidation might
cause each client to purge its cache, for instance).
as far as i know there is no 8192 page limit in the
NFS client.
-----Original Message-----
From: Abhishek Rai [mailto:[email protected]]
Sent: Tuesday, March 11, 2003 2:33 AM
To: [email protected]
Subject: [NFS] regarding size limit of local client cache
hi,
this query arises from some small experiments i performed with my linux
nfs client (2.4.18).
Conclusions first:- There is a limit of 8192 pages on the size of the
local client cache and this space is shared between the various nfs
mounts currently operating on this machine with this machine as the
client.
Question 1. I haven't been able to trace this 8192 pages limit to any
data structure or functionality - so please help
Question 2 : This question is regarding the management of this shared
cache, space freeing mechanism in the cache to make space for fresh
reads etc and arises from the following observations :-
I conducted the following experiments:- 'n' nfs clients (numbered 1, 2,
3) are currently mounted, reading 3 different non-overlapping directory
trees of some server. The events proceed as follows when n=2,
client 1 reads 4096 4k pages from the server. Then, client 2 reads 4096
4k pages(different from those read by 1) from the server.
client 1 again reads 4096 pages from the server followed by client 2
repeating the same. Except for the first read whent the cache is cold,
no later reads translate into an on-the-wire read.
<< just to add: i concluded of the 8192 pages limit from the following 3
observations: (1). when i made the two clients read 8192 pages each
alternatively, every read all through caused an on-the-wire-lookup, (2).
when only one client was repeatedly reading 8192 pages, the cache
completely satisfied all but the first request, (3). when only one
client was repeatedly reading 16384 pages, every read translated into an
on-the-wire read.>>
Now i made n=3. (no. of pages being read by each client continue to be
4096)
I tried out two different orderings of client reads - <1,2,3,1,2,3> and
<1,2,3,2,1>
In both these cases, __ALL__ reads translate into an on-the-wire read. I
expected some requirement based freeing of space in the client cache to
make space for say 3 after 1 and 2 have done their reads eg. if global
lru based freeing then in <123123>, 3 would cause 1's 4096 pages to be
thrown out of the cache to make space for its own 4096. And after that
in the next cycle, 1 would cause the same to 2... which would mean all
reads=on-the-wire-read. But this scenario would imply that in <12321>,
the second time 2 is around it would find its data still in the cache,
only 1 will suffer etc but this wasn't found to be the case. So when
space is needed - the space freeing protocol recklessly frees all
possible space ? no- global lru based freeing mechanism ? at least let
me know where lies the corresponding code. thanks a lot.
Question 3: server->nr_requests is limited by the hard limit
MAX_REQUEST_HARD. what is this server->nr_requests ? outstanding
unsatisfied requests ? Again where is the limit for cache size ?
Thanks in advance
cheers
Abhishek
Do you Yahoo!?
Yahoo! Web Hosting - establish your business online
-------------------------------------------------------
This SF.net email is sponsored by:Crypto Challenge is now open!
Get cracking and register here for some mind boggling fun and
the chance of winning an Apple iPod:
http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0031en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
---------------------------------
Do you Yahoo!?
Yahoo! Web Hosting - establish your business online
>>>>> " " == Charles Lever <Lever> writes:
> hi- you didn't tell us how large your client's RAM is, what
> your mount options are, what other programs are running on the
> clients, nor how long each client takes to read (attribute
> cache invalidation might cause each client to purge its cache,
> for instance).
> as far as i know there is no 8192 page limit in the NFS client.
Right. The size of the page cache is limited only by the available
free memory.
Cheers,
Trond
-------------------------------------------------------
This SF.net email is sponsored by:Crypto Challenge is now open!
Get cracking and register here for some mind boggling fun and
the chance of winning an Apple iPod:
http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0031en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
But in any case, I have a large swap partition(500 MB). So why doesn't the NFS client make use of the swap space to enlarge its available page cache space rather than restricting its page cache usage to 8192 -4k pages and thus paying the penalty of a very high numbe of on the wire reads.(As i mentioned before, there is negligible load on the client apart from the NFS)
Trond Myklebust <[email protected]> wrote:>>>>> " " == Charles Lever writes:
> hi- you didn't tell us how large your client's RAM is, what
> your mount options are, what other programs are running on the
> clients, nor how long each client takes to read (attribute
> cache invalidation might cause each client to purge its cache,
> for instance).
> as far as i know there is no 8192 page limit in the NFS client.
Right. The size of the page cache is limited only by the available
free memory.
Cheers,
Trond
---------------------------------
Do you Yahoo!?
Yahoo! Web Hosting - establish your business online
>>>>> " " == Abhishek Rai <[email protected]> writes:
> But in any case, I have a large swap partition(500 MB). So why
> doesn't the NFS client make use of the swap space to enlarge
> its available page cache space rather than restricting its page
> cache usage to 8192 -4k pages and thus paying the penalty of a
> very high numbe of on the wire reads.(As i mentioned before,
> there is negligible load on the client apart from the NFS)
There is no guarantee that swap will be faster than NFS. Think GigE or
faster...
In any case, this is an area that in entirely under the control of the
VM. NFS has no say whatsoever in what policy should be applied
w.r.t. caching.
Cheers,
Trond
-------------------------------------------------------
This SF.net email is sponsored by:Crypto Challenge is now open!
Get cracking and register here for some mind boggling fun and
the chance of winning an Apple iPod:
http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0031en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs