2008-06-27 07:31:54

by Krishna Kumar2

[permalink] [raw]
Subject: Re: [NFS] NFS performance debugging

[email protected] wrote on 06/25/2008 07:29:16 PM:

> But please do not let this caveat stop you from trying this script and
> mountstats script. We definitely need feedback on how well they do or
> don't work (and as always... patches are welcomed! ;-) ) Once things
harden
> up (via any feedback), I'll added the install code to make these scripts
be
> installed (probably in /usr/sbin) which means they will be ready for
> prime time...

Hi Steve,

When I ran this util, it worked the first time:

[[email protected] ~]# /tmp/nfs-iostat

localhost:/local mounted on /nfs:

op/s rpc bklog
1.27 0.01
read: ops/s Kb/s Kb/op retrans
avg RTT (ms) avg exe (ms)
0.619 39.772 64.273 0 (0.0%)
16.889 16.927
write: ops/s Kb/s Kb/op retrans
avg RTT (ms) avg exe (ms)
0.626 39.112 62.523 0 (0.0%)
26.357 4039.241

8.124.111.71:/mnt mounted on /mnt:

op/s rpc bklog
3.00 0.00
read: ops/s Kb/s Kb/op retrans
avg RTT (ms) avg exe (ms)
0.000 0.000 0.000 0 (0.0%)
0.000 0.000
write: ops/s Kb/s Kb/op retrans
avg RTT (ms) avg exe (ms)
0.000 0.000 0.000 0 (0.0%)
0.000 0.000

(KK: for some reason, the real NFS mount gives all zero. Also, for
loopback, the read/write
BW is very poor: less than 40 Kb/s).

But from the next time I ran, and all subsequent runs (and even after a
reboot),
I get:

[[email protected] init.d]# /tmp/nfs-iostat
localhost:/local mounted on /nfs:

op/s rpc bklog
0.04 0.00
read: ops/s Kb/s Kb/op retrans
avg RTT (ms) avg exe (ms)
0.000 0.000 0.000 0 (0.0%)
0.000 0.000
write: ops/s Kb/s Kb/op retrans
avg RTT (ms) avg exe (ms)
0.000 0.000 0.000 0 (0.0%)
0.000 0.000

8.124.111.71:/mnt mounted on /mnt:

op/s rpc bklog
6.00 0.00
read: ops/s Kb/s Kb/op retrans
avg RTT (ms) avg exe (ms)
0.000 0.000 0.000 0 (0.0%)
0.000 0.000
write: ops/s Kb/s Kb/op retrans
avg RTT (ms) avg exe (ms)
0.000 0.000 0.000 0 (0.0%)
0.000 0.000

(KK: every filesystem gives zero for all parameters).

What could be wrong?

Thanks,

- KK


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs



2008-06-27 18:06:34

by Olga Kornievskaia

[permalink] [raw]
Subject: Re: [NFS] NFS performance debugging



Krishna Kumar2 wrote:
> [email protected] wrote on 06/25/2008 07:29:16 PM:
>
>
>> But please do not let this caveat stop you from trying this script and
>> mountstats script. We definitely need feedback on how well they do or
>> don't work (and as always... patches are welcomed! ;-) ) Once things
>>
> harden
>
>> up (via any feedback), I'll added the install code to make these scripts
>>
> be
>
>> installed (probably in /usr/sbin) which means they will be ready for
>> prime time...
>>
>
> Hi Steve,
>
> When I ran this util, it worked the first time:
>
> [[email protected] ~]# /tmp/nfs-iostat
>
> localhost:/local mounted on /nfs:
>
> op/s rpc bklog
> 1.27 0.01
> read: ops/s Kb/s Kb/op retrans
> avg RTT (ms) avg exe (ms)
> 0.619 39.772 64.273 0 (0.0%)
> 16.889 16.927
> write: ops/s Kb/s Kb/op retrans
> avg RTT (ms) avg exe (ms)
> 0.626 39.112 62.523 0 (0.0%)
> 26.357 4039.241
>
> 8.124.111.71:/mnt mounted on /mnt:
>
> op/s rpc bklog
> 3.00 0.00
> read: ops/s Kb/s Kb/op retrans
> avg RTT (ms) avg exe (ms)
> 0.000 0.000 0.000 0 (0.0%)
> 0.000 0.000
> write: ops/s Kb/s Kb/op retrans
> avg RTT (ms) avg exe (ms)
> 0.000 0.000 0.000 0 (0.0%)
> 0.000 0.000
>
> (KK: for some reason, the real NFS mount gives all zero. Also, for
> loopback, the read/write
> BW is very poor: less than 40 Kb/s).
>
> But from the next time I ran, and all subsequent runs (and even after a
> reboot),
> I get:
>
> [[email protected] init.d]# /tmp/nfs-iostat
> localhost:/local mounted on /nfs:
>
> op/s rpc bklog
> 0.04 0.00
> read: ops/s Kb/s Kb/op retrans
> avg RTT (ms) avg exe (ms)
> 0.000 0.000 0.000 0 (0.0%)
> 0.000 0.000
> write: ops/s Kb/s Kb/op retrans
> avg RTT (ms) avg exe (ms)
> 0.000 0.000 0.000 0 (0.0%)
> 0.000 0.000
>
> 8.124.111.71:/mnt mounted on /mnt:
>
> op/s rpc bklog
> 6.00 0.00
> read: ops/s Kb/s Kb/op retrans
> avg RTT (ms) avg exe (ms)
> 0.000 0.000 0.000 0 (0.0%)
> 0.000 0.000
> write: ops/s Kb/s Kb/op retrans
> avg RTT (ms) avg exe (ms)
> 0.000 0.000 0.000 0 (0.0%)
> 0.000 0.000
>
> (KK: every filesystem gives zero for all parameters).
>
> What could be wrong?
>
Are you running nfs-iostat just once? While doing your testing, you
should do "nfs-iostat 2" so that every 2secs nfs-iostat displays
information about the on-going test (you can choose any other polling
interval value).
> Thanks,
>
> - KK
>
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> _______________________________________________
> NFS maillist - [email protected]
> https://lists.sourceforge.net/lists/listinfo/nfs
> _______________________________________________
> Please note that [email protected] is being discontinued.
> Please subscribe to [email protected] instead.
> http://vger.kernel.org/vger-lists.html#linux-nfs
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs


2008-06-27 13:44:52

by Chuck Lever III

[permalink] [raw]
Subject: Re: [NFS] NFS performance debugging

On Fri, Jun 27, 2008 at 3:30 AM, Krishna Kumar2 <[email protected]> wrote:
> [email protected] wrote on 06/25/2008 07:29:16 PM:
>
>> But please do not let this caveat stop you from trying this script and
>> mountstats script. We definitely need feedback on how well they do or
>> don't work (and as always... patches are welcomed! ;-) ) Once things
> harden
>> up (via any feedback), I'll added the install code to make these scripts
> be
>> installed (probably in /usr/sbin) which means they will be ready for
>> prime time...
>
> Hi Steve,
>
> When I ran this util, it worked the first time:
>
> [[email protected] ~]# /tmp/nfs-iostat
>
> localhost:/local mounted on /nfs:
>
> op/s rpc bklog
> 1.27 0.01
> read: ops/s Kb/s Kb/op retrans
> avg RTT (ms) avg exe (ms)
> 0.619 39.772 64.273 0 (0.0%)
> 16.889 16.927
> write: ops/s Kb/s Kb/op retrans
> avg RTT (ms) avg exe (ms)
> 0.626 39.112 62.523 0 (0.0%)
> 26.357 4039.241
>
> 8.124.111.71:/mnt mounted on /mnt:
>
> op/s rpc bklog
> 3.00 0.00
> read: ops/s Kb/s Kb/op retrans
> avg RTT (ms) avg exe (ms)
> 0.000 0.000 0.000 0 (0.0%)
> 0.000 0.000
> write: ops/s Kb/s Kb/op retrans
> avg RTT (ms) avg exe (ms)
> 0.000 0.000 0.000 0 (0.0%)
> 0.000 0.000
>
> (KK: for some reason, the real NFS mount gives all zero. Also, for
> loopback, the read/write
> BW is very poor: less than 40 Kb/s).

As an aside, Jim Rees pointed out to me that that should be KB/s, not
Kb/s. I will post a patch to address that.

> But from the next time I ran, and all subsequent runs (and even after a
> reboot), I get:
>
> [[email protected] init.d]# /tmp/nfs-iostat
> localhost:/local mounted on /nfs:
>
> op/s rpc bklog
> 0.04 0.00
> read: ops/s Kb/s Kb/op retrans
> avg RTT (ms) avg exe (ms)
> 0.000 0.000 0.000 0 (0.0%)
> 0.000 0.000
> write: ops/s Kb/s Kb/op retrans
> avg RTT (ms) avg exe (ms)
> 0.000 0.000 0.000 0 (0.0%)
> 0.000 0.000
>
> 8.124.111.71:/mnt mounted on /mnt:
>
> op/s rpc bklog
> 6.00 0.00
> read: ops/s Kb/s Kb/op retrans
> avg RTT (ms) avg exe (ms)
> 0.000 0.000 0.000 0 (0.0%)
> 0.000 0.000
> write: ops/s Kb/s Kb/op retrans
> avg RTT (ms) avg exe (ms)
> 0.000 0.000 0.000 0 (0.0%)
> 0.000 0.000
>
> (KK: every filesystem gives zero for all parameters).
>
> What could be wrong?

At a guess, it's because the client is now caching your file data?

Can you post a copy of your mount command line, and your
/proc/self/mountstats file after a test run (but before you unmount)?

--
Chuck Lever

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs