From: "Matt Heaton" Subject: Performance Woes... Date: Tue, 4 Jun 2002 22:29:56 -0600 Sender: nfs-admin@lists.sourceforge.net Message-ID: <0a1701c20c49$a57f6250$6701a8c0@c1886657a> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0A14_01C20C17.5A9B8D40" Return-path: Received: from rwcrmhc52.attbi.com ([216.148.227.88]) by usw-sf-list1.sourceforge.net with esmtp (Exim 3.31-VA-mm2 #1 (Debian)) id 17FSQz-00046S-00 for ; Tue, 04 Jun 2002 21:30:09 -0700 Received: from c1886657a ([12.254.138.126]) by rwcrmhc52.attbi.com (InterMail vM.4.01.03.27 201-229-121-127-20010626) with SMTP id <20020605042958.RGDU2751.rwcrmhc52.attbi.com@c1886657a> for ; Wed, 5 Jun 2002 04:29:58 +0000 To: Errors-To: nfs-admin@lists.sourceforge.net List-Help: List-Post: List-Subscribe: , List-Id: Discussion of NFS under Linux development, interoperability, and testing. List-Unsubscribe: , List-Archive: This is a multi-part message in MIME format. ------=_NextPart_000_0A14_01C20C17.5A9B8D40 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi All, I am having some trouble determining whether it is my NFS server = or Client that is the bottleneck. I have 10 web servers hooked up to 2 = NFS servers. The clients all run Redhat 7.2 with 2.4.18 with NFSALL patch applied, = and the servers are redhat 7.2 with the 2.4.18 kernel. I am using UDP. My problem is that my web servers when doing a "vmstat 1" get many = blocked processes. See below. I can't really get ANY kind of good reading to whether or not an NFS = server is overloaded or not. Besides iostat and nfsstat are there any = good tools?!?! I am going crazy trying to test. Please help. If you are an NFS guru = feel free to give me a call at (801) 361-1177 (Cell phone). I am happy = to pay! PS - The numbers below were taken at 10:30 PM so traffic is NOT at a = high during this time, you can expect 50% traffic increase at mid day. Any help would really be appreciated! Thanks! r b w swpd free buff cache si so bi bo in = cs us sy id 8 1 0 9668 103728 28624 456756 0 0 0 38 588 286 29 = 3 68 11 0 0 9668 103196 28624 457068 0 0 0 20 617 221 14 = 5 81 9 2 0 9668 103172 28624 456988 0 0 0 12 267 104 13 = 2 85 The 'r' at the top left is the problem. This happens when NFS traffic = goes way up. The nfsstat -c for the client look like this. Client nfs v3: null getattr setattr lookup access readlink =20 19 0% 29862505 67% 5171 0% 6976413 15% 16064 0% 0 0%=20 read write create mkdir symlink mknod =20 7697391 17% 2287 0% 10 0% 0 0% 0 0% 0 0%=20 remove rmdir rename link readdir readdirplus 0 0% 0 0% 0 0% 0 0% 0 0% 53 0%=20 fsstat fsinfo pathconf commit =20 5 0% 5 0% 0 0% 2287 0%=20 All clients are pretty much the same. The nfsstat -s for the two NFS = servers are below. Server nfs v2: null getattr setattr root lookup readlink =20 0 0% 4818 0% 457163 19% 0 0% 374516 16% 0 0%=20 read wrcache write create remove rename =20 1006 0% 0 0% 1338999 57% 147627 6% 550 0% 0 0%=20 link symlink mkdir rmdir readdir fsstat =20 0 0% 0 0% 6923 0% 73 0% 4795 0% 5 0%=20 Server nfs v3: null getattr setattr lookup access readlink =20 12 0% 2487236 38% 22870 0% 2451861 38% 19225 0% 0 0%=20 read write create mkdir symlink mknod =20 971932 15% 274775 4% 31984 0% 1212 0% 0 0% 0 0%=20 remove rmdir rename link readdir readdirplus 14422 0% 391 0% 10524 0% 0 0% 24138 0% 0 0%=20 fsstat fsinfo pathconf commit =20 8 0% 7 0% 0 0% 95129 1%=20 AND THE OTHER SERVER Server nfs v2: null getattr setattr root lookup readlink =20 0 0% 54088 0% 453322 1% 0 0% 23253032 55% 0 0%=20 read wrcache write create remove rename =20 1964341 4% 0 0% 13001439 31% 449191 1% 676267 1% 21247 0%=20 link symlink mkdir rmdir readdir fsstat =20 0 0% 0 0% 15479 0% 65541 0% 1647071 3% 51 0%=20 Server nfs v3: null getattr setattr lookup access readlink =20 0 0% 416318245 2% 2678145 0% 461314618 2% 127706435 3% 0 = 0%=20 read write create mkdir symlink mknod =20 184416498 1% 22576823 1% 2622952 0% 92490 0% 0 0% 0 0%=20 remove rmdir rename link readdir readdirplus 1550770 0% 61184 0% 1673675 0% 0 0% 5333994 0% 180 0%=20 fsstat fsinfo pathconf commit =20 187 0% 129 0% 0 0% 10053536 0%=20 IOSTAT 5 for the first NFS server below Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 1.00 0.00 30.40 0 152 dev8-0 48.60 273.60 516.80 1368 2584 IOSTAT 5 for the second NFS server below Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev3-0 0.40 0.00 17.60 0 88 dev8-0 70.60 752.00 132.80 3760 664 ------=_NextPart_000_0A14_01C20C17.5A9B8D40 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Hi All, I am having some trouble = determining=20 whether it is my NFS server or Client that is the bottleneck.  I = have 10=20 web servers hooked up to 2 NFS servers.
The clients all run Redhat 7.2 with = 2.4.18 with=20 NFSALL patch applied, and the servers are redhat 7.2 with the 2.4.18=20 kernel.  I am using UDP.
 
My problem is that my web servers when = doing a=20 "vmstat 1" get many blocked processes.  See below.
 
I can't really get ANY kind of good = reading to=20 whether or not an NFS server is overloaded or not.  Besides iostat = and=20 nfsstat are there any good tools?!?!
I am going crazy trying to test.  = Please=20 help.  If you are an NFS guru feel free to give me a call at (801) = 361-1177=20 (Cell phone).  I am happy to pay!
 
PS - The numbers below were taken at = 10:30 PM so=20 traffic is NOT at a high during this time, you can expect 50% traffic = increase=20 at mid day.
 
Any help would really be = appreciated! =20 Thanks!
 
r    b  = w  =20 swpd   free      = buff    =20 cache  si  so    bi    = bo  =20 in     cs   us   sy =20 id
8    1  = 0   9668=20 103728  28624 456756   0   = 0    =20 0    38  588   286  29   = 3 =20 68
11  0  0   9668 103196  28624 = 457068  =20 0   0     0    20 =20 617   221  14   5  = 81
9   =20 2  0   9668 103172  28624 456988   = 0  =20 0     0    12  267   = 104 =20 13   2  85
 
 
The 'r' at the top left is the = problem.  This=20 happens when NFS traffic goes way up.  The nfsstat -c for the = client look=20 like this.
 
Client nfs=20 v3:
null       = getattr   =20 setattr    lookup    =20 access     readlink  =20
19      0% 29862505 67% = 5171    0%=20 6976413 15% 16064   0% 0       = 0%=20
read       = write     =20 create     mkdir     =20 symlink    mknod      =
7697391 17%=20 2287    0% 10      0%=20 0       0% = 0      =20 0% 0       0% =
remove    =20 rmdir      rename    =20 link       readdir   =20 readdirplus
0       0%=20 0       0% = 0      =20 0% 0       0%=20 0       0% = 53      0%=20
fsstat     fsinfo    =20 pathconf   commit    =20
5       0%=20 5       0% = 0      =20 0% 2287    0%
 
 
All clients are pretty much the = same.  The=20 nfsstat -s for the two NFS servers are below.
 
Server nfs=20 v2:
null       = getattr   =20 setattr    root      =20 lookup     readlink  =20
0       0% 4818    0% = 457163=20 19% 0       0% 374516 16%=20 0       0%=20
read       wrcache   =20 write      create    =20 remove     rename    =20
1006    0% 0       0% = 1338999=20 57% 147627  6% 550     0%=20 0       0%=20
link       symlink   =20 mkdir      rmdir     =20 readdir    fsstat    =20
0       0%=20 0       0% 6923    0%=20 73      0% 4795    0%=20 5       0%
 
Server nfs=20 v3:
null       = getattr   =20 setattr    lookup    =20 access     readlink  =20
12      0% 2487236 38% 22870   0% = 2451861=20 38% 19225   0% 0       0%=20
read       = write     =20 create     mkdir     =20 symlink    mknod     
971932 = 15%=20 274775  4% 31984   0% 1212    0%=20 0       0% = 0      =20 0%
remove     = rmdir     =20 rename     link      =20 readdir    readdirplus
14422   0%=20 391     0% 10524   0%=20 0       0% 24138   0%=20 0       0% =
fsstat    =20 fsinfo     pathconf  =20 commit    
8       = 0%=20 7       0% = 0      =20 0% 95129   1%
 
 
AND THE OTHER SERVER
 
Server nfs=20 v2:
null       = getattr   =20 setattr    root      =20 lookup     readlink  =20
0       0% 54088   0% = 453322 =20 1% 0       0% 23253032 55%=20 0       0%=20
read       wrcache   =20 write      create    =20 remove     rename     =
1964341 =20 4% 0       0% 13001439 31% 449191  1% = 676267  1% 21247   0%=20
link       symlink   =20 mkdir      rmdir     =20 readdir    fsstat    =20
0       0%=20 0       0% 15479   0% = 65541  =20 0% 1647071  3% 51      0%
 
Server nfs=20 v3:
null       = getattr   =20 setattr    lookup    =20 access     readlink  =20
0       0% 416318245  2% = 2678145  0%=20 461314618  2% 127706435  3% = 0       0%=20
read       = write     =20 create     mkdir     =20 symlink    mknod     =20
184416498  1% 22576823  1% 2622952  0% = 92490   0%=20 0       0% = 0      =20 0%
remove     = rmdir     =20 rename     link      =20 readdir    readdirplus
1550770  0% = 61184   0%=20 1673675  0% 0       0% 5333994  = 0%=20 180     0%
fsstat    =20 fsinfo     pathconf  =20 commit    
187     0%=20 129     0% 0       0%=20 10053536  0%
 
IOSTAT 5 for the first NFS server=20 below
 
Device:       =20 tps   Blk_read/s   Blk_wrtn/s  =20 Blk_read  =20 Blk_wrtn
dev3-0       =20 1.00        =20 0.00       =20 30.40         =20 0       =20 152
dev8-0      =20 48.60      =20 273.60      =20 516.80      =20 1368       2584
 
IOSTAT 5 for the second NFS server=20 below
 
Device:       =20 tps   Blk_read/s   Blk_wrtn/s  =20 Blk_read  =20 Blk_wrtn
dev3-0       =20 0.40        =20 0.00       =20 17.60         =20 0        =20 88
dev8-0      =20 70.60      =20 752.00      =20 132.80      =20 3760        664
 
 
------=_NextPart_000_0A14_01C20C17.5A9B8D40-- _______________________________________________________________ Don't miss the 2002 Sprint PCS Application Developer's Conference August 25-28 in Las Vegas -- http://devcon.sprintpcs.com/adp/index.cfm _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs