2010-07-27 10:07:07

by Or Gerlitz

[permalink] [raw]
Subject: iostat "await" like counter for measuring nfs latency

Doing some iscsi and nfs performance testing, I realized that iostat(1)
doesn't support an "await" (*) like counter for measuring nfs latency.

Can tracking nfs latency be done in other means? is there a plan to
extend the nfs iostats code to include such/similar counter?

thanks,

Or.


(*) await - The average time (in milliseconds) for I/O requests issued
to the device to be served. This includes the time spent by the
requests in queue and the time spent servicing them.




2010-07-27 15:28:20

by Myklebust, Trond

[permalink] [raw]
Subject: Re: iostat "await" like counter for measuring nfs latency

On Tue, 2010-07-27 at 13:07 +0300, Or Gerlitz wrote:
> Doing some iscsi and nfs performance testing, I realized that iostat(1)
> doesn't support an "await" (*) like counter for measuring nfs latency.
>
> Can tracking nfs latency be done in other means? is there a plan to
> extend the nfs iostats code to include such/similar counter?
>
> thanks,
>
> Or.
>
>
> (*) await - The average time (in milliseconds) for I/O requests issued
> to the device to be served. This includes the time spent by the
> requests in queue and the time spent servicing them.
>
>

The information is already available in /proc/self/mountstats. It is up
to iostat to start using it.

Trond

2010-07-28 19:21:01

by Sebastien Godard

[permalink] [raw]
Subject: Re: iostat "await" like counter for measuring nfs latency

On 07/28/2010 07:40 PM, Chuck Lever wrote:
>
> On 07/28/10 05:45 AM, Or Gerlitz wrote:
>> Trond Myklebust wrote:
>>> On Tue, 2010-07-27 at 13:07 +0300, Or Gerlitz wrote:
>>>> Doing some iscsi and nfs performance testing, I realized that
>>>> iostat(1)
>>>> doesn't support an "await" like counter for measuring nfs latency.
>>>> Can tracking nfs latency be done in other means? is there a plan to
>>>> extend the nfs iostats code to include such/similar counter?
>>
>>> The information is already available in /proc/self/mountstats. It is up
>>> to iostat to start using it.
>>
>> Sebastian, is there a plan to support await like counter in sysstat?
>>
>> I didn't find a source repository (is there an open one? where?) so
>> downloaded the latest code (version 9.1.3) drop from the website and
>> couldn't see there such a counter, am I missing something?
>
> There are Python scripts in nfs-utils that report the statistics in
> /proc/self/mountstats. Nobody really likes the file format of
> /proc/self/mountstats, so the plan is to move these, at some point, to
> someplace where perf can report them.
>
> At that point, iostat support would also be nice.
>

I'm ready to accept a patch that would add an await like counter to
measure NFS latency.
BTW beginning with sysstat version 9.1.3, there is a specific command
("nfsiostat") aimed at displaying stats for network filesystems.

Regards,

--
Sébastien Godard (sysstat<at> orange.fr)
Help support sysstat...!
Visit http://pagesperso-orange.fr/sebastien.godard/





2010-07-28 17:41:08

by Chuck Lever III

[permalink] [raw]
Subject: Re: iostat "await" like counter for measuring nfs latency

On 07/28/10 05:45 AM, Or Gerlitz wrote:
> Trond Myklebust wrote:
>> On Tue, 2010-07-27 at 13:07 +0300, Or Gerlitz wrote:
>>> Doing some iscsi and nfs performance testing, I realized that iostat(1)
>>> doesn't support an "await" like counter for measuring nfs latency.
>>> Can tracking nfs latency be done in other means? is there a plan to
>>> extend the nfs iostats code to include such/similar counter?
>
>> The information is already available in /proc/self/mountstats. It is up
>> to iostat to start using it.
>
> Sebastian, is there a plan to support await like counter in sysstat?
>
> I didn't find a source repository (is there an open one? where?) so
> downloaded the latest code (version 9.1.3) drop from the website and
> couldn't see there such a counter, am I missing something?

There are Python scripts in nfs-utils that report the statistics in
/proc/self/mountstats. Nobody really likes the file format of
/proc/self/mountstats, so the plan is to move these, at some point, to
someplace where perf can report them.

At that point, iostat support would also be nice.

2010-07-28 09:45:19

by Or Gerlitz

[permalink] [raw]
Subject: Re: iostat "await" like counter for measuring nfs latency

Trond Myklebust wrote:
> On Tue, 2010-07-27 at 13:07 +0300, Or Gerlitz wrote:
>> Doing some iscsi and nfs performance testing, I realized that iostat(1)
>> doesn't support an "await" like counter for measuring nfs latency.
>> Can tracking nfs latency be done in other means? is there a plan to
>> extend the nfs iostats code to include such/similar counter?

> The information is already available in /proc/self/mountstats. It is up
> to iostat to start using it.

Sebastian, is there a plan to support await like counter in sysstat?

I didn't find a source repository (is there an open one? where?) so
downloaded the latest code (version 9.1.3) drop from the website and
couldn't see there such a counter, am I missing something?

Or.



2010-07-29 15:54:16

by Chuck Lever III

[permalink] [raw]
Subject: Re: iostat "await" like counter for measuring nfs latency

On 07/28/10 03:21 PM, Sebastien Godard wrote:
> On 07/28/2010 07:40 PM, Chuck Lever wrote:
>>
>> On 07/28/10 05:45 AM, Or Gerlitz wrote:
>>> Trond Myklebust wrote:
>>>> On Tue, 2010-07-27 at 13:07 +0300, Or Gerlitz wrote:
>>>>> Doing some iscsi and nfs performance testing, I realized that
>>>>> iostat(1)
>>>>> doesn't support an "await" like counter for measuring nfs latency.
>>>>> Can tracking nfs latency be done in other means? is there a plan to
>>>>> extend the nfs iostats code to include such/similar counter?
>>>
>>>> The information is already available in /proc/self/mountstats. It is up
>>>> to iostat to start using it.
>>>
>>> Sebastian, is there a plan to support await like counter in sysstat?
>>>
>>> I didn't find a source repository (is there an open one? where?) so
>>> downloaded the latest code (version 9.1.3) drop from the website and
>>> couldn't see there such a counter, am I missing something?
>>
>> There are Python scripts in nfs-utils that report the statistics in
>> /proc/self/mountstats. Nobody really likes the file format of
>> /proc/self/mountstats, so the plan is to move these, at some point, to
>> someplace where perf can report them.
>>
>> At that point, iostat support would also be nice.
>>
>
> I'm ready to accept a patch that would add an await like counter to
> measure NFS latency.

What is await, precisely?

And, by "NFS latency" do you mean read/write latency or something else?
For NFS, even stat() can require server interaction (ie is not served
from the system's local memory)

> BTW beginning with sysstat version 9.1.3, there is a specific command
> ("nfsiostat") aimed at displaying stats for network filesystems.