Hi,
We recently moved one of our NFS servers from a Sun machine running
Solaris 7 to an Intel box running SuSE Linux 8.0 (kernel 2.4.18). We are
experiencing some (rather heavy) performance problems, more so for clients
accesing the service over our WAN link.
I'm starting to go through the excerise of tuning the NFS system, and
started first by trying to increase the number of nfsd process. It seems
that there is a hard coded limit of 128 processes, is that correct? If true,
is it because one shouldn't expect any substantial performance increase by
setting the number of nfsd threads to a number larger than 128?
Thanks in advance for your help. Regards,
---
Daniel Barbar
-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Wednesday July 31, [email protected] wrote:
> Hi,
>
> We recently moved one of our NFS servers from a Sun machine running
> Solaris 7 to an Intel box running SuSE Linux 8.0 (kernel 2.4.18). We are
> experiencing some (rather heavy) performance problems, more so for clients
> accesing the service over our WAN link.
> I'm starting to go through the excerise of tuning the NFS system, and
> started first by trying to increase the number of nfsd process. It seems
> that there is a hard coded limit of 128 processes, is that correct? If true,
> is it because one shouldn't expect any substantial performance increase by
> setting the number of nfsd threads to a number larger than 128?
This is correct in 2.4.18.
2.4.18 will have a substantially larger limit.
Just change the #define in fs/nfsd/nfssvc.c and recompile for a larger
limit.
I guess when the number was chosen, computers were smaller.
It could be that more than 128 threads will help you, but it would be
worth checking
/proc/net/rpc/nfsd
to see how many threads are really being used. Send me the content of
that file and I will interpret bits for you.
NeilBrown
-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
What OS are the clients running? If they are solaris boxes you might
want to consider the NFS/TCP patches for the linux NFS server -- NFS/UDP
over a WAN can have problems. TCP might help out some.
eric
Daniel Barbar wrote:
>
> Hi,
>
> We recently moved one of our NFS servers from a Sun machine running
> Solaris 7 to an Intel box running SuSE Linux 8.0 (kernel 2.4.18). We are
> experiencing some (rather heavy) performance problems, more so for clients
> accesing the service over our WAN link.
> I'm starting to go through the excerise of tuning the NFS system, and
> started first by trying to increase the number of nfsd process. It seems
> that there is a hard coded limit of 128 processes, is that correct? If true,
> is it because one shouldn't expect any substantial performance increase by
> setting the number of nfsd threads to a number larger than 128?
> Thanks in advance for your help. Regards,
>
> ---
> Daniel Barbar
>
> -------------------------------------------------------
> This sf.net email is sponsored by: Dice - The leading online job board
> for high-tech professionals. Search and apply for tech jobs today!
> http://seeker.dice.com/seeker.epl?rel_code=31
> _______________________________________________
> NFS maillist - [email protected]
> https://lists.sourceforge.net/lists/listinfo/nfs
-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Neil, Erich:
Thanks for the replies. Here is the content of /proc/net/rpc/nfsd:
fiona:~# cat /proc/net/rpc/nfsd
rc 19 123835 15603248
fh 10 15227728 0 14 188
io 3752095213 1826934055
th 128 0 34.710 2.230 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
ra 256 8316937 462671 10814 6530 3748 4498 2851 3290 1583 1881 402138
net 15727611 15727611 0 0
rpc 15727102 509 509 0 0
proc2 18 856 35104 6 0 9044 349 329927 0 0 83 0 0 0 0 223 0 1890 569
proc3 22 144698 1357961 50256 519080 2198920 744533 8887018 60150 5539 710 1
0 4486 383 127 0 268983 1005032 88344 6016 4533 2281
Most clients are Solaris (IS&T here is very Solaris centric). I was
considering installing the patch to enable NFS/TCP, I just wasn't sure
because of its "experimental" state. Have you guys gotten a feeling for its
stability? Thanks again,
---
Daniel Barbar
-----Original Message-----
From: Neil Brown [mailto:[email protected]]
Sent: Wednesday, July 31, 2002 9:27 PM
To: Daniel Barbar
Cc: '[email protected]'
Subject: Re: [NFS] Maximum number of nfsd daemons?
On Wednesday July 31, [email protected] wrote:
> Hi,
>
> We recently moved one of our NFS servers from a Sun machine running
> Solaris 7 to an Intel box running SuSE Linux 8.0 (kernel 2.4.18). We are
> experiencing some (rather heavy) performance problems, more so for clients
> accesing the service over our WAN link.
> I'm starting to go through the excerise of tuning the NFS system, and
> started first by trying to increase the number of nfsd process. It seems
> that there is a hard coded limit of 128 processes, is that correct? If
true,
> is it because one shouldn't expect any substantial performance increase by
> setting the number of nfsd threads to a number larger than 128?
This is correct in 2.4.18.
2.4.18 will have a substantially larger limit.
Just change the #define in fs/nfsd/nfssvc.c and recompile for a larger
limit.
I guess when the number was chosen, computers were smaller.
It could be that more than 128 threads will help you, but it would be
worth checking
/proc/net/rpc/nfsd
to see how many threads are really being used. Send me the content of
that file and I will interpret bits for you.
NeilBrown
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Hi Neil,
On Thu, 1 Aug 2002 14:27:03 +1000 (EST)
Neil Brown <[email protected]> wrote:
| Send me the content of that file and I will interpret bits for you.
|
| NeilBrown
|
I shamelessly take the opportunity to ask you to have at the following /proc/net/rpc/nfsd figures :o)
Quick description of our setup : NFS clients & server are Linux only (all at least
2.4.19pre6) , 100Mbs switched network, no router between server & clients,
clients and server are using NFSV3 UDP.
64 nfsd processes on the server ,around 40 to 50 clients.
(server is a 1GHz SMP 1go 1Go RAM)
# cat /proc/net/rpc/nfsd
rc 75545 57811921 1425095899
fh 841380 1483195871 0 3806 22127
io 947166859 1044180197
th 64 1701313 122616.020 45017.680 17551.390 8336.110 5302.600 2744.440
2160.200 1545.790 1213.580 5346.700
ra 128 517548125 17409328 11625583 8052404 5724124 5052943 4025961 3561989
2949267 2450055 227670791
net 1483010914 1483010914 0 0
rpc 1482983365 27549 27549 0 0
proc2 18 27540 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
proc3 22 27546 -1121099198 3805380 1366334119 342864898 352 806090436 42699347
6410085 439142 11 0 3627641 250110 655750 0 3473989 16097 2780770 2780468 0 21798882
We experience a lot of "NFS server not responding" so i wonder if we don't come close to a limit.
I see an average of 8 Mbs on eth0 so it doesn't look like the network is the problem.
( only one 100Mbs interface on the NFS server )
Thanks.
Philippe.
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Thursday August 1, [email protected] wrote:
> Neil, Erich:
>
> Thanks for the replies. Here is the content of /proc/net/rpc/nfsd:
>
> fiona:~# cat /proc/net/rpc/nfsd
> rc 19 123835 15603248
> fh 10 15227728 0 14 188
> io 3752095213 1826934055
> th 128 0 34.710 2.230 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
This line says
" You have 128 threads.
You have never used the last thread.
You have used between 1 and 12 threads (10%) for a total of 34.7 seconds.
You have used between 13 and 25 threads for a total of 2.23 seconds.
You have not noticably used more that 25 threads.
"
So if this load is typical, then you don't need more threads.
> ra 256 8316937 462671 10814 6530 3748 4498 2851 3290 1583 1881 402138
> net 15727611 15727611 0 0
> rpc 15727102 509 509 0 0
> proc2 18 856 35104 6 0 9044 349 329927 0 0 83 0 0 0 0 223 0 1890 569
> proc3 22 144698 1357961 50256 519080 2198920 744533 8887018 60150 5539 710 1
> 0 4486 383 127 0 268983 1005032 88344 6016 4533 2281
Everything else looks normal, but you haven't been running this server
for long. Maybe you need a longer period of data collection.
>
> Most clients are Solaris (IS&T here is very Solaris centric). I was
> considering installing the patch to enable NFS/TCP, I just wasn't sure
> because of its "experimental" state. Have you guys gotten a feeling for its
> stability? Thanks again,
I don't use it. It should only improve performance if you loose
packets on your network. A well switched lan with uniform interface
speeds should not notice. If you have a saturated lan, or interfaces
of different speeds (e.g. Gigabit client, 100M server) then TCP could
help.
NeilBrown
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Neil Brown wrote:
>>fiona:~# cat /proc/net/rpc/nfsd
>>rc 19 123835 15603248
>>fh 10 15227728 0 14 188
>>io 3752095213 1826934055
>>th 128 0 34.710 2.230 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000>
>>
> This line says
[...]
>
>>ra 256 8316937 462671 10814 6530 3748 4498 2851 3290 1583 1881 402138
>>net 15727611 15727611 0 0
>>rpc 15727102 509 509 0 0
>>proc2 18 856 35104 6 0 9044 349 329927 0 0 83 0 0 0 0 223 0 1890 569
>>proc3 22 144698 1357961 50256 519080 2198920 744533 8887018 60150 5539 710 1
>>0 4486 383 127 0 268983 1005032 88344 6016 4533 2281
>
> Everything else looks normal, but you haven't been running this server
> for long. Maybe you need a longer period of data collection.
[...]
It would be great if an explanation of /proc/net/rpc/nfsd could go
into the HOWTO ... :)
Regards,
Andreas
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Would you bother with more threads in this situation? The server's been
up 133 days; doesn't seem like it's worth messing with it if the last
thread has only been used for 11 seconds. I might increase it for the next
reboot (if ever :-), but I don't really want to bother with restarting nfsd:
rc 253405 247133819 2102736339
fh 164627 2388074384 0 0 0
io 1879707623 1245239505
th 20 802 499059.350 179740.290 95589.220 38896.840 13428.490 4192.740 1125.820 239.110 45.190 11.020
ra 40 205615873 1051844 560093 330858 220535 154846 109266 81350 66170 63103 0
net -1944843733 -1944843733 0 0
rpc -1944843733 0 0 0 0
proc2 18 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
proc3 22 0 42738275 24032239 215924238 1550591333 20714 233998756 172948575 24818081 13323 20691 0 24789604 13179 733268 18264 74781 0 171980 110430 0 59105832
--
Bruce Robertson, President/CEO +1-775-348-7299
Great Basin Internet Services, Inc. fax: +1-775-348-9412
http://www.greatbasin.net
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Andreas Behnert wrote:
>
> Neil Brown wrote:
> >>fiona:~# cat /proc/net/rpc/nfsd
> >>rc 19 123835 15603248
> >>fh 10 15227728 0 14 188
> >>io 3752095213 1826934055
> >>th 128 0 34.710 2.230 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000>
> >>
> > This line says
> [...]
> >
> >>ra 256 8316937 462671 10814 6530 3748 4498 2851 3290 1583 1881 402138
> >>net 15727611 15727611 0 0
> >>rpc 15727102 509 509 0 0
> >>proc2 18 856 35104 6 0 9044 349 329927 0 0 83 0 0 0 0 223 0 1890 569
> >>proc3 22 144698 1357961 50256 519080 2198920 744533 8887018 60150 5539 710 1
> >>0 4486 383 127 0 268983 1005032 88344 6016 4533 2281
> >
> > Everything else looks normal, but you haven't been running this server
> > for long. Maybe you need a longer period of data collection.
> [...]
>
> It would be great if an explanation of /proc/net/rpc/nfsd could go
> into the HOWTO ... :)
I'll look into it; there is already some discussion in the performance
section about some of the proc files, and the thread counts in
/proc/net/rpc/nfsd
Tom
--
------------------------------------------------------------
Tom McNeal [email protected] (650)906-0761 (cell)
------------------------------------------------------------
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Friday August 2, [email protected] wrote:
> Would you bother with more threads in this situation? The server's been
> up 133 days; doesn't seem like it's worth messing with it if the last
> thread has only been used for 11 seconds. I might increase it for the next
> reboot (if ever :-), but I don't really want to bother with restarting nfsd:
1/ It looks OK. I wouldn't bother increasing the number of threads.
2/ It is actualy 'any of the last 2 (10%)' that have been use for only
11 seconds.
3/ The second number is a good one to look at, 802 in this case.
It is the number of times that the last thread has been used.
Out of 200 million requests, 802 of them did use the last available
thread at the time. On those occasions it might have been helpful
to have another thread or 2, but it might not. So increasing the
number of threads wouldn't make a significant difference.
4/ In 2.4 you can increase or decrease the number of threads at any
time. You don't need to reboot. Just run
rpc.nfsd desired-number-of-threads
This will change the number of threads to the target.
e.g.
rpc.nfsd 0
will kill all threads.
rpc.nfsd 128
will give you the max.
NeilBrown
>
> rc 253405 247133819 2102736339
> fh 164627 2388074384 0 0 0
> io 1879707623 1245239505
> th 20 802 499059.350 179740.290 95589.220 38896.840 13428.490 4192.740 1125.820 239.110 45.190 11.020
> ra 40 205615873 1051844 560093 330858 220535 154846 109266 81350 66170 63103 0
> net -1944843733 -1944843733 0 0
> rpc -1944843733 0 0 0 0
> proc2 18 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> proc3 22 0 42738275 24032239 215924238 1550591333 20714 233998756 172948575 24818081 13323 20691 0 24789604 13179 733268 18264 74781 0 171980 110430 0 59105832
>
>
>
> --
> Bruce Robertson, President/CEO +1-775-348-7299
> Great Basin Internet Services, Inc. fax: +1-775-348-9412
> http://www.greatbasin.net
>
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Thursday August 1, [email protected] wrote:
>
> Hi Neil,
>
> On Thu, 1 Aug 2002 14:27:03 +1000 (EST)
> Neil Brown <[email protected]> wrote:
>
> | Send me the content of that file and I will interpret bits for you.
> |
> | NeilBrown
> |
>
> I shamelessly take the opportunity to ask you to have at the
> following /proc/net/rpc/nfsd figures :o)
Then you should address the email to me, not just to the list. I can
easily get behind on mailing list mail...
>
> Quick description of our setup : NFS clients & server are Linux only (all at least
> 2.4.19pre6) , 100Mbs switched network, no router between server & clients,
> clients and server are using NFSV3 UDP.
>
> 64 nfsd processes on the server ,around 40 to 50 clients.
> (server is a 1GHz SMP 1go 1Go RAM)
>
> # cat /proc/net/rpc/nfsd
> rc 75545 57811921 1425095899
> fh 841380 1483195871 0 3806 22127
> io 947166859 1044180197
> th 64 1701313 122616.020 45017.680 17551.390 8336.110 5302.600 2744.440
> 2160.200 1545.790 1213.580 5346.700
Looks a bit busy. I feel that the fact that the last number is bigger
than the preceding few is significant. You would probably expect some
sort of Poisson (spelling?) distribution here. However having a peek
at the end suggests that for a lot of those 5000 seconds the server
was over-loaded and really wanted to use more threads.
The fact that nearly two million requests used that last thread
suggests this as well. Now there were over a billion requests
altogether so most of the time your server was doing fine, but when a
peak load came it, it didn't cope.
I would increase the number of threads. Probably up to 128.
The usage counts don't get zeroed when you do that (maybe they
should..) so you need to take a copy of what they were just before you
up the thread count, and then subtract that from what you look at in a
few days..
> ra 128 517548125 17409328 11625583 8052404 5724124 5052943 4025961 3561989
> 2949267 2450055 227670791
> net 1483010914 1483010914 0 0
> rpc 1482983365 27549 27549 0 0
> proc2 18 27540 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> proc3 22 27546 -1121099198 3805380 1366334119 342864898 352 806090436 42699347
> 6410085 439142 11 0 3627641 250110 655750 0 3473989 16097 2780770 2780468 0 21798882
>
> We experience a lot of "NFS server not responding" so i wonder if we don't come close to a limit.
> I see an average of 8 Mbs on eth0 so it doesn't look like the network is the problem.
> ( only one 100Mbs interface on the NFS server )
"NFS server not responding" can be due to the receive queue filling up
on the NFS server.
while true; do netstat -nta | grep :2049 ; sleep 1; done
Watch the first column of numbers. If it frequently hits a ceiling at
65536 or near there, you are losing packets.
rpc.nfsd 0
echo 262144 > /proc/sys/net/core/rmem_default
echo 262144 > /proc/sys/net/core/rmem_max
rpc.nfsd 128
echo 65536 > /proc/sys/net/core/rmem_default
echo 65536 > /proc/sys/net/core/rmem_max
should fix that.
NeilBrown
>
> Thanks.
>
> Philippe.
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> NFS maillist - [email protected]
> https://lists.sourceforge.net/lists/listinfo/nfs
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Wed, 7 Aug 2002 21:04:22 +1000
Neil Brown <[email protected]> wrote:
| Then you should address the email to me, not just to the list. I can
| easily get behind on mailing list mail...
Ok, will do next time :o)
| > # cat /proc/net/rpc/nfsd
| > rc 75545 57811921 1425095899
| > fh 841380 1483195871 0 3806 22127
| > io 947166859 1044180197
| > th 64 1701313 122616.020 45017.680 17551.390 8336.110 5302.600 2744.440
| > 2160.200 1545.790 1213.580 5346.700
|
| Now there were over a billion requests
| altogether so most of the time your server was doing fine, but when a
| peak load came it, it didn't cope.
This is indeed what we observed.
|
| I would increase the number of threads. Probably up to 128.
This is what i did actually, and since then the servers are doing really good
even during peak hours, and no more warning messages.
|
| The usage counts don't get zeroed when you do that (maybe they
| should..) so you need to take a copy of what they were just before you
| up the thread count, and then subtract that from what you look at in a
| few days..
I've noticed that. Actually, i had to reboot one of them so i can read directly the figures
right now:
# uptime
14:40:33 up 22:45, 1 user, load average: 1.64, 1.92, 1.94
# cat /proc/net/rpc/nfsd
rc 218 486371 43878936
fh 1519133 42854538 0 659 1345
io 2725711409 2096612340
th 128 0 20.000 1.750 0.020 0.010 0.000 0.000 0.000 0.000 0.000 0.000
ra 256 609800 18945 8551 4061 2134 1456 1077 687 439 161 250781
net 44365742 44365802 0 0
rpc 44365507 229 229 0 0
proc2 18 229 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
proc3 22 229 25036678 24989 11489150 778515 0 6318090 360723 48596 5727 0 0 37396 4022 5136 0 27909 0 22837 22837 0 182478
|
| "NFS server not responding" can be due to the receive queue filling up
| on the NFS server.
| while true; do netstat -nta | grep :2049 ; sleep 1; done
i'm not using TCP right now :o) but will do very soon as i've just build a kernel with all
your patches.
|
| Watch the first column of numbers. If it frequently hits a ceiling at
| 65536 or near there, you are losing packets.
|
| rpc.nfsd 0
| echo 262144 > /proc/sys/net/core/rmem_default
| echo 262144 > /proc/sys/net/core/rmem_max
| rpc.nfsd 128
| echo 65536 > /proc/sys/net/core/rmem_default
| echo 65536 > /proc/sys/net/core/rmem_max
|
| should fix that.
|
| NeilBrown
well, i already added that in the nfs-kernel-server starting script as this was cleary said
in the NFS HOWTO on SF.
But when switching from 64 to 128 i forgot to do so. So i've just did again what's written above,
and everythin fine now.
Thanks,
Philippe.
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Wednesday August 7, [email protected] wrote:
>
>
> |
> | "NFS server not responding" can be due to the receive queue filling up
> | on the NFS server.
> | while true; do netstat -nta | grep :2049 ; sleep 1; done
>
> i'm not using TCP right now :o) but will do very soon as i've just build a kernel with all
> your patches.
>
I was actually thinking UDP. The 't' flag was a mistake.
If you have working TCP support, then this is all taken care of by the
kernel and you don't have to worry about it.
NeilBrown
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs