Often when a cluster run (100+ nodes) is started my NFS server will get
very slow; users with their home directories on the server will lock up
for short periods, and the load average crawls higher.
The output of cat /proc/net/rpc/nfsd:
th 125 264502 142948.997 38666.037 22620.179 16494.577 10072.689
5700.113 2623.322 1094.785 474.752 1900.536
This would seem to mean that my nfsd threads are runnning at 100% for
much of the time, meaning I need more nfsd's, correct? But is there a
limit to the amount of nfsd's I should run? I had had it as high as 150
at one time, but it seemed to me this made the server less stable.
There may have been other reasons for this, however.
Anyway I have since doubled the ram on the server (to 8 gig). Machine
is a dual Xeon 2.6 Ghz machine running Fedora Core 2, kernel version
2.6.10-1.771_FC2smp, gigabit ethernet. Can I run more nfsd's, or should
I be looking for other problems?
Hugh
--
Hugh Caley | Unix Systems Administrator | CIS
AFFYMETRIX, INC. | 6550 Vallejo St. Ste 100 | Emeryville, CA 94608
Tel: 510-428-8537 | [email protected]
-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Tuesday June 21, [email protected] wrote:
> Often when a cluster run (100+ nodes) is started my NFS server will get
> very slow; users with their home directories on the server will lock up
> for short periods, and the load average crawls higher.
>
> The output of cat /proc/net/rpc/nfsd:
>
> th 125 264502 142948.997 38666.037 22620.179 16494.577 10072.689
> 5700.113 2623.322 1094.785 474.752 1900.536
>
> This would seem to mean that my nfsd threads are runnning at 100% for
> much of the time, meaning I need more nfsd's, correct? But is there a
> limit to the amount of nfsd's I should run? I had had it as high as 150
> at one time, but it seemed to me this made the server less stable.
> There may have been other reasons for this, however.
>
> Anyway I have since doubled the ram on the server (to 8 gig). Machine
> is a dual Xeon 2.6 Ghz machine running Fedora Core 2, kernel version
> 2.6.10-1.771_FC2smp, gigabit ethernet. Can I run more nfsd's, or should
> I be looking for other problems?
There should be very little cost in pushing the number of nfsd
substantially higher, but there might not be much gain.
So definitely run more nfsd - say 256 (you can just run "nfsd 256" and
it will increase the number of threads to that number).
But also look into how the filesystem and storage is performing.
What filesys and what storage are you using?
NeilBrown
-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
My experience is that above 16 or so I have not seen a gain
in the actual performance delivered to the network.
There also appears (on RHEL3 2.4 kernel) to be some sort
of kernel resource that gets overused by having too
many nfsds and causes other issues, this would not a
conclusive test, but appears to be a problem, we took
the nfsds back down from 256 to 64 because of this
perceived problem, and because it did not change any
of the performance issues that we had.
Roger
Atipa Technologies
> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]] On Behalf Of Hugh Caley
> Sent: Tuesday, June 21, 2005 9:02 PM
> To: [email protected]
> Subject: [NFS] How many nfsd's can I run?
>
> Often when a cluster run (100+ nodes) is started my NFS
> server will get very slow; users with their home directories
> on the server will lock up for short periods, and the load
> average crawls higher.
>
> The output of cat /proc/net/rpc/nfsd:
>
> th 125 264502 142948.997 38666.037 22620.179 16494.577 10072.689
> 5700.113 2623.322 1094.785 474.752 1900.536
>
> This would seem to mean that my nfsd threads are runnning at
> 100% for much of the time, meaning I need more nfsd's,
> correct? But is there a limit to the amount of nfsd's I
> should run? I had had it as high as 150 at one time, but it
> seemed to me this made the server less stable.
> There may have been other reasons for this, however.
>
> Anyway I have since doubled the ram on the server (to 8 gig).
> Machine is a dual Xeon 2.6 Ghz machine running Fedora Core
> 2, kernel version 2.6.10-1.771_FC2smp, gigabit ethernet. Can
> I run more nfsd's, or should I be looking for other problems?
>
> Hugh
>
> --
> Hugh Caley | Unix Systems Administrator | CIS AFFYMETRIX,
> INC. | 6550 Vallejo St. Ste 100 | Emeryville, CA 94608
> Tel: 510-428-8537 | [email protected]
>
>
>
> -------------------------------------------------------
> SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
> from IBM. Find simple to follow Roadmaps, straightforward articles,
> informative Webcasts and more! Get everything you need to get up to
> speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
> _______________________________________________
> NFS maillist - [email protected]
> https://lists.sourceforge.net/lists/listinfo/nfs
>
-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Thanks for the advice. So, if I run "nfsd 256", it will increase the
currently running 125 daemons to 256?
Filesystem is reiser, and the disk device is a Nexsan Atabeast.
Hugh
Neil Brown wrote:
>On Tuesday June 21, [email protected] wrote:
>
>
>>Often when a cluster run (100+ nodes) is started my NFS server will get
>>very slow; users with their home directories on the server will lock up
>>for short periods, and the load average crawls higher.
>>
>>The output of cat /proc/net/rpc/nfsd:
>>
>>th 125 264502 142948.997 38666.037 22620.179 16494.577 10072.689
>>5700.113 2623.322 1094.785 474.752 1900.536
>>
>>This would seem to mean that my nfsd threads are runnning at 100% for
>>much of the time, meaning I need more nfsd's, correct? But is there a
>>limit to the amount of nfsd's I should run? I had had it as high as 150
>>at one time, but it seemed to me this made the server less stable.
>>There may have been other reasons for this, however.
>>
>>Anyway I have since doubled the ram on the server (to 8 gig). Machine
>>is a dual Xeon 2.6 Ghz machine running Fedora Core 2, kernel version
>>2.6.10-1.771_FC2smp, gigabit ethernet. Can I run more nfsd's, or should
>>I be looking for other problems?
>>
>>
>
>There should be very little cost in pushing the number of nfsd
>substantially higher, but there might not be much gain.
>
>So definitely run more nfsd - say 256 (you can just run "nfsd 256" and
>it will increase the number of threads to that number).
>But also look into how the filesystem and storage is performing.
>
>What filesys and what storage are you using?
>
>NeilBrown
>
>
--
Hugh Caley | Unix Systems Administrator | CIS
AFFYMETRIX, INC. | 6550 Vallejo St. Ste 100 | Emeryville, CA 94608
Tel: 510-428-8537 | [email protected]
-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Wednesday June 22, [email protected] wrote:
> Thanks for the advice. So, if I run "nfsd 256", it will increase the
> currently running 125 daemons to 256?
Yes.
>
> Filesystem is reiser, and the disk device is a Nexsan Atabeast.
>
I don't know of any particular performance issues or tuning
possibilities with reiserfs, and I've never heard of an Atabeast..
So I don't think I can be any more helpful.
NeilBrown
-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs