I have a HUGE directory structure on my NFS server. I host free websites and have literally 390K directories on my NFS server. Below is output from
my NFS box. It shows 99% of the clients time is taken by lookups (I believe going through the directories). I have heard there is a way through the
kernal or some other method to increase the cache associated with the DIRECTORY LOOKUPS so that I didn't have to hit the drive everytime to get
the directory. I have 1 GIG in the this machine and can easily upgrade it to 4 GIG to give it a huge directory lookup cache if someone can point me in
the right direction. My clients and my servers are both using kernal 2.4.7-10 (Redhat 7.2).
Thanks,
Matt Heaton
calls badcalls badauth badclnt xdrcall
73765732 0 0 0 0
Server nfs v2:
null getattr setattr root lookup readlink
0 0% 48449 0% 77378 0% 0 0% 5527260 51% 0 0%
read wrcache write create remove rename
3050391 28% 0 0% 1793260 16% 83335 0% 49376 0% 9001 0%
link symlink mkdir rmdir readdir fsstat
0 0% 0 0% 13721 0% 2258 0% 153840 1% 3 0%
Server nfs v3:
null getattr setattr lookup access readlink
0 0% 535201 0% 267339 0% 42856622 68% 167774 0% 0 0%
read write create mkdir symlink mknod
13981454 22% 2907986 4% 340093 0% 10591 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
153574 0% 5102 0% 189901 0% 0 0% 272374 0% 0 0%
fsstat fsinfo pathconf commit
42 0% 42 0% 0 0% 1269365 2%
Client rpc stats:
calls retrans authrefrsh
2043779 0 0
Client nfs v2:
null getattr setattr root lookup readlink
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
read wrcache write create remove rename
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
link symlink mkdir rmdir readdir fsstat
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
Client nfs v3:
null getattr setattr lookup access readlink
0 0% 26 0% 0 0% 2043741 99% 0 0% 0 0%
read write create mkdir symlink mknod
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
fsstat fsinfo pathconf commit
6 0% 6 0% 0 0% 0 0%
as your workload and performance issue may be fairly common, let me reply on
the list.
the Linux NFS client uses LOOKUP operations to verify cached filehandles.
the number of exported
directories may have nothing to do with how many on-the-wire lookup
operations you see.
since your workload is mostly reads, try Trond's CTO patch, with the "nocto"
mount option, and let us know the results.
-----Original Message-----
From: Matt Heaton [mailto:[email protected]]
Sent: Tuesday, March 19, 2002 12:27 AM
To: [email protected]
Subject: [NFS] NFSSTAT QUESTION.
I have a HUGE directory structure on my NFS server. I host free websites
and have literally 390K directories on my NFS server. Below is output from
my NFS box. It shows 99% of the clients time is taken by lookups (I believe
going through the directories). I have heard there is a way through the
kernal or some other method to increase the cache associated with the
DIRECTORY LOOKUPS so that I didn't have to hit the drive everytime to get
the directory. I have 1 GIG in the this machine and can easily upgrade it
to 4 GIG to give it a huge directory lookup cache if someone can point me in
the right direction. My clients and my servers are both using kernal
2.4.7-10 (Redhat 7.2).
Thanks,
Matt Heaton
calls badcalls badauth badclnt xdrcall
73765732 0 0 0 0
Server nfs v2:
null getattr setattr root lookup readlink
0 0% 48449 0% 77378 0% 0 0% 5527260 51% 0 0%
read wrcache write create remove rename
3050391 28% 0 0% 1793260 16% 83335 0% 49376 0% 9001 0%
link symlink mkdir rmdir readdir fsstat
0 0% 0 0% 13721 0% 2258 0% 153840 1% 3 0%
Server nfs v3:
null getattr setattr lookup access readlink
0 0% 535201 0% 267339 0% 42856622 68% 167774 0% 0 0%
read write create mkdir symlink mknod
13981454 22% 2907986 4% 340093 0% 10591 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
153574 0% 5102 0% 189901 0% 0 0% 272374 0% 0 0%
fsstat fsinfo pathconf commit
42 0% 42 0% 0 0% 1269365 2%
Client rpc stats:
calls retrans authrefrsh
2043779 0 0
Client nfs v2:
null getattr setattr root lookup readlink
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
read wrcache write create remove rename
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
link symlink mkdir rmdir readdir fsstat
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
Client nfs v3:
null getattr setattr lookup access readlink
0 0% 26 0% 0 0% 2043741 99% 0 0% 0 0%
read write create mkdir symlink mknod
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
fsstat fsinfo pathconf commit
6 0% 6 0% 0 0% 0 0%