Hi,
we have a raid controller serving a file system that we export and
mount with NFS. Depending on the performance profile we chose for the
raid controller, the are huge performance differences on the NFS client
side between two profiles. Those two profiles both use write cache and
no journaling, but one honors the FUA bit if it is set, and the other
profile ignores it (there might be more differences, but the manual
doesn't state any others but the FUA bit).
With the profile ignoring the FUA bit, copying or deleting directories
of e.g. 10M with a about 1000 files is factor 5 faster than with the
profile honoring the FUA bit.
We export with the "sync" option. Does that option maybe set the FUA bit
for all write operations on the NFS server?
cu,
Frank
--
Dipl.-Inform. Frank Steiner Web: http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17 Phone: +49 89 2180-4049
80333 Muenchen, Germany Fax: +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
Thanks to the two of you for the detailed explanations! It did help me
understanding the impacts to my raid controllers profile!
cu,
Frank
--
Dipl.-Inform. Frank Steiner Web: http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17 Phone: +49 89 2180-4049
80333 Muenchen, Germany Fax: +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
On Jun. 10, 2008, 13:43 +0300, Frank Steiner <[email protected]> wrote:
> Hi,
>
> we have a raid controller serving a file system that we export and
> mount with NFS. Depending on the performance profile we chose for the
> raid controller, the are huge performance differences on the NFS client
> side between two profiles. Those two profiles both use write cache and
> no journaling, but one honors the FUA bit if it is set, and the other
> profile ignores it (there might be more differences, but the manual
> doesn't state any others but the FUA bit).
>
> With the profile ignoring the FUA bit, copying or deleting directories
> of e.g. 10M with a about 1000 files is factor 5 faster than with the
> profile honoring the FUA bit.
>
> We export with the "sync" option. Does that option maybe set the FUA bit
> for all write operations on the NFS server?
No, it's not supposed to that for *all* writes. It's just supposed
to honor the client's synchronous writes and commit calls.
However, the directory is synced after each change (you say above that
files are copied or deleted) with the sync flag and this is more
likely to cause the slowdown you see.
Benny
>
> cu,
> Frank
>
>
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
Frank Steiner wrote:
> With the profile ignoring the FUA bit, copying or deleting directories
> of e.g. 10M with a about 1000 files is factor 5 faster than with the
> profile honoring the FUA bit.
>
FUA bit is normally combined with write-thru scsi command that bypasses
storage write cache. I would imagine it needs to well synchronize
various pieces before issuing this command. It could hurt the
performance if not done well, particularly for meta data. So your result
is not surprising.
> We export with the "sync" option. Does that option maybe set the FUA bit
> for all write operations on the NFS server?
>
It depends on how the filesystem (and its associated disk subsystem) is
implemented. The "sync" export option itself has a heavy performance
impact, regardless how FUA bit is handled. Some vendors uses specialized
HW (e.g. NVRAM) to alleviate this performance hit. If your filesystem
doesn't have this type of support, you should expect "sync" option runs
much much slower than "async". It is a choice (or balance) between cost,
performance, and data reliability.
Don't you have the vendor's support group to go ? Better get this answer
from the storage vendor directly.
-- Wendy
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
Benny Halevy wrote:
> On Jun. 10, 2008, 18:44 +0300, Wendy Cheng <[email protected]> wrote:
>
>> Frank Steiner wrote:
>>
>>> With the profile ignoring the FUA bit, copying or deleting directories
>>> of e.g. 10M with a about 1000 files is factor 5 faster than with the
>>> profile honoring the FUA bit.
>>>
>>>
>> FUA bit is normally combined with write-thru scsi command that bypasses
>> storage write cache. I would imagine it needs to well synchronize
>> various pieces before issuing this command. It could hurt the
>> performance if not done well, particularly for meta data. So your result
>> is not surprising.
>>
>>> We export with the "sync" option. Does that option maybe set the FUA bit
>>> for all write operations on the NFS server?
>>>
>>>
>> It depends on how the filesystem (and its associated disk subsystem) is
>> implemented. The "sync" export option itself has a heavy performance
>> impact, regardless how FUA bit is handled. Some vendors uses specialized
>> HW (e.g. NVRAM) to alleviate this performance hit. If your filesystem
>> doesn't have this type of support, you should expect "sync" option runs
>> much much slower than "async". It is a choice (or balance) between cost,
>> performance, and data reliability.
>>
>
> Wendy, I *think* what you have in mind is the sync mount option
> rather than the sync export flag. The latter just tells the server
> not to cheat and do everything asynchronously. It should *not*
> have a heavy performance penalty for I/O intensive writes if the
> client is using async writes and commits.
>
>
No, I didn't get confused ... We can use Linux as an example :) .. check
out:
http://marc.info/?l=linux-nfs&m=119618886105337&w=2
-- quote
The default export might have been "async", but unless the option "sync"
in /etc/exports was being ignored I was already using "sync". Nevertheless
I will try to change to async and test if it makes a difference.
(one day later: )
I have now tried it and the load on the NFS server is much lower and KDE
logins seem to be reasonably fast now.
-- un-quote
-- Wendy
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
On Tue, Jun 10, 2008 at 01:52:08PM -0500, Wendy Cheng wrote:
> Benny Halevy wrote:
> > On Jun. 10, 2008, 18:44 +0300, Wendy Cheng <[email protected]> wrote:
> >
> >> Frank Steiner wrote:
> >>
> >>> With the profile ignoring the FUA bit, copying or deleting directories
> >>> of e.g. 10M with a about 1000 files is factor 5 faster than with the
> >>> profile honoring the FUA bit.
> >>>
> >>>
> >> FUA bit is normally combined with write-thru scsi command that bypasses
> >> storage write cache. I would imagine it needs to well synchronize
> >> various pieces before issuing this command. It could hurt the
> >> performance if not done well, particularly for meta data. So your result
> >> is not surprising.
> >>
> >>> We export with the "sync" option. Does that option maybe set the FUA bit
> >>> for all write operations on the NFS server?
> >>>
> >>>
> >> It depends on how the filesystem (and its associated disk subsystem) is
> >> implemented. The "sync" export option itself has a heavy performance
> >> impact, regardless how FUA bit is handled. Some vendors uses specialized
> >> HW (e.g. NVRAM) to alleviate this performance hit. If your filesystem
> >> doesn't have this type of support, you should expect "sync" option runs
> >> much much slower than "async". It is a choice (or balance) between cost,
> >> performance, and data reliability.
> >>
> >
> > Wendy, I *think* what you have in mind is the sync mount option
> > rather than the sync export flag. The latter just tells the server
> > not to cheat and do everything asynchronously. It should *not*
> > have a heavy performance penalty for I/O intensive writes if the
> > client is using async writes and commits.
> >
> >
> No, I didn't get confused ... We can use Linux as an example :) .. check
> out:
> http://marc.info/?l=linux-nfs&m=119618886105337&w=2
>
> -- quote
>
> The default export might have been "async", but unless the option "sync"
> in /etc/exports was being ignored I was already using "sync". Nevertheless
> I will try to change to async and test if it makes a difference.
>
> (one day later: )
>
> I have now tried it and the load on the NFS server is much lower and KDE
> logins seem to be reasonably fast now.
Yeah, the "async" export options is expected to improve performance on a
workload like that (with lots of directory creates/deletes). But it
does so by violating the promises that the protocol makes to the client.
--b.
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
Benny Halevy wrote:
> On Jun. 10, 2008, 21:52 +0300, Wendy Cheng <[email protected]> wrote:
>
>> Benny Halevy wrote:
>>
>>> On Jun. 10, 2008, 18:44 +0300, Wendy Cheng <[email protected]> wrote:
>>>
>>>
>>>> Frank Steiner wrote:
>>>>
>>>>
>>>>> With the profile ignoring the FUA bit, copying or deleting directories
>>>>> of e.g. 10M with a about 1000 files is factor 5 faster than with the
>>>>> profile honoring the FUA bit.
>>>>>
>>>>>
>>>>>
>>>> FUA bit is normally combined with write-thru scsi command that bypasses
>>>> storage write cache. I would imagine it needs to well synchronize
>>>> various pieces before issuing this command. It could hurt the
>>>> performance if not done well, particularly for meta data. So your result
>>>> is not surprising.
>>>>
>>>>
>>>>> We export with the "sync" option. Does that option maybe set the FUA bit
>>>>> for all write operations on the NFS server?
>>>>>
>>>>>
>>>>>
>>>> It depends on how the filesystem (and its associated disk subsystem) is
>>>> implemented. The "sync" export option itself has a heavy performance
>>>> impact, regardless how FUA bit is handled. Some vendors uses specialized
>>>> HW (e.g. NVRAM) to alleviate this performance hit. If your filesystem
>>>> doesn't have this type of support, you should expect "sync" option runs
>>>> much much slower than "async". It is a choice (or balance) between cost,
>>>> performance, and data reliability.
>>>>
>>>>
>>> Wendy, I *think* what you have in mind is the sync mount option
>>> rather than the sync export flag. The latter just tells the server
>>> not to cheat and do everything asynchronously. It should *not*
>>> have a heavy performance penalty for I/O intensive writes if the
>>> client is using async writes and commits.
>>>
>>>
>>>
>> No, I didn't get confused ... We can use Linux as an example :) .. check
>> out:
>> http://marc.info/?l=linux-nfs&m=119618886105337&w=2
>>
>> -- quote
>>
>> The default export might have been "async", but unless the option "sync"
>> in /etc/exports was being ignored I was already using "sync". Nevertheless
>> I will try to change to async and test if it makes a difference.
>>
>> (one day later: )
>>
>> I have now tried it and the load on the NFS server is much lower and KDE
>> logins seem to be reasonably fast now.
>>
>> -- un-quote
>>
>>
>> -- Wendy
>>
>
> I'm not sure how the problem you referred to is related to the
> one at hand. what I'm trying to say is that the sync _exports_
> flag has stronger affect on namespace modifying workloads, like
> what I believe Frank's workload might be ... [snip]
The referred thread showed "sync" export has performance impacts on
Linux server *too*.
-- Wendy
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued.
Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs