2009-06-05 13:50:31

by Tom Talpey

[permalink] [raw]
Subject: Re: Link performance over NFS degraded in RHEL5. -- was : Read/Write NFS I/O performance degraded by FLUSH_STABLE page flushing

On 6/5/2009 9:28 AM, Steve Dickson wrote:
>
> Tom Talpey wrote:
>> On 6/5/2009 7:35 AM, Steve Dickson wrote:
>>> Brian R Cowan wrote:
>>>> Trond Myklebust<[email protected]> wrote on 06/04/2009
>>>> 02:04:58
>>>> PM:
>>>>
>>>>> Did you try turning off write gathering on the server (i.e. add the
>>>>> 'no_wdelay' export option)? As I said earlier, that forces a delay of
>>>>> 10ms per RPC call, which might explain the FILE_SYNC slowness.
>>>> Just tried it, this seems to be a very useful workaround as well. The
>>>> FILE_SYNC write calls come back in about the same amount of time as the
>>>> write+commit pairs... Speeds up building regardless of the network
>>>> filesystem (ClearCase MVFS or straight NFS).
>>> Does anybody had the history as to why 'no_wdelay' is an
>>> export default?
>> Because "wdelay" is a complete crock?
>>
>> Adding 10ms to every write RPC only helps if there's a steady
>> single-file stream arriving at the server. In most other workloads
>> it only slows things down.
>>
>> The better solution is to continue tuning the clients to issue
>> writes in a more sequential and less all-or-nothing fashion.
>> There are plenty of other less crock-ful things to do in the
>> server, too.
> Ok... So do you think removing it as a default would cause
> any regressions?

I'm not 100% clear on what you mean by removing it. Since it's
a "no_" option, removing it means that "wdelay" becomes the
default? That would certainly cause a regression for many.

I think the big problem with tweaking the default in nfs_utils
is that there's little guarantee of the kernel behavior that
would result. Older kernels, NFSv2 mounts, etc will behave completely
differently from new ones, NFSv3, modified clients, etc. So
touching this option is quite risky, IMO, even though it's a
crock.

Tom.


2009-06-05 13:58:19

by Tom Talpey

[permalink] [raw]
Subject: Re: Link performance over NFS degraded in RHEL5. -- was : Read/Write NFS I/O performance degraded by FLUSH_STABLE page flushing

On 6/5/2009 9:54 AM, Trond Myklebust wrote:
> On Fri, 2009-06-05 at 09:50 -0400, Tom Talpey wrote:
>> I'm not 100% clear on what you mean by removing it. Since it's
>> a "no_" option, removing it means that "wdelay" becomes the
>> default? That would certainly cause a regression for many.
>
> You've misunderstood. The current default is to _set_ 'wdelay' on all
> exports that do not explicitly turn it off.

Ok, then turning it off will help some and hurt some. There's no
right setting for all. I do agree that fixing the server is the
best solution, not grabbing wildly at its crockful controls.

Tom.


2009-06-05 13:54:57

by Trond Myklebust

[permalink] [raw]
Subject: Re: Link performance over NFS degraded in RHEL5. -- was : Read/Write NFS I/O performance degraded by FLUSH_STABLE page flushing

On Fri, 2009-06-05 at 09:50 -0400, Tom Talpey wrote:
> I'm not 100% clear on what you mean by removing it. Since it's
> a "no_" option, removing it means that "wdelay" becomes the
> default? That would certainly cause a regression for many.

You've misunderstood. The current default is to _set_ 'wdelay' on all
exports that do not explicitly turn it off.

Trond