LinuxLists.cc - Linux NFS writes to Solaris very, very slow

On Monday 26 July 2004 19:21, you wrote:
> sorry i'm not copying the list... our local exchange server changed my
> outgoing To: address a couple of weeks ago, and i still haven't gotten
> the list moderator's attention to fix up my e-mail address.

No problem, I will send a full quote to the list.

>
> > Exactly, therefore I would like to enable the sync-export
> > option again.
> > However, reducing the speed to about 1/5 of async speed for
> > large files and
> > to about 1/20 for smaller files is not an option. The people already
> > complained that compiling their projects takes ages now...
> >
> > D'oh, sorry, I forgot to write that we use the tcp-mounts and
> > nfs3. So those
> > numbers really shouldn't happen, should they?
>
> if the server is working correctly, theoretically this should not
> happen.
>
> are you using strange mount options on the client, such as "noac" or
> "sync" ? "async" on the client is perfectly OK to use; it's the server
> side export option that is badness.

No, I think not. Here's the fstab line for /home:
hamilton:/home /home nfs rsize=8192,wsize=8192,hard,intr,tcp 0 0

Here's something that is for me pretty interesting (since I don't completely
understand whats causing it):

1.) The export option is changed on the server from sync to async,
'exportfs -rav' is executed.

2.) One client had mounted with the async-mount option, all other clients
without this option. No client remounts /home after the the export option was
changed on the server.

3.) All but the one client now can write to the server with 5MB/s for large
*and* small files. However the client which has mounted with the async-mount
option can now write with full 100MBit (so 11 MB/s) to the server.

4.) Of course, remounting /home on any client even without giving the
async-mount option, gives back full write speed to all clients now.

Is the behaviour from 3) intented to happen? Actually I thought that a remount
on the clients is necessary to gain back full write speed.

Thanks,
Bernd

Attachments:

(No filename) (1.98 kB)
(No filename) (189.00 B)
signature Download all attachments

2004-07-26 22:06:18

> >>[root@arvouin tmp]# mount cobra3:/p2v5f1 -o=20
> >>async,wsize=3D32768,rsize=3D32768,soft /mnt/cobra3
> >
> >um. you're not using NFS version 3?
> >
> I though it was a default !?

in Linux, only on more recent mount command / kernel combinations is
version 3 the default.

> >not to mention "soft" mounts are also truly the spawn of satan.
> > =20
> >
> OK I remove it, and force nfs v3

note: soft is probably not a performance issue here, but it will
definitely be a source of silent data corruption.

> However what stricks me is that although I asked for r&wsize=20
> of 32 K ,=20
> looking at /proc/mounts show :
> $cat /proc/mounts
> cobra3:/p2v5f1 /mnt/cobra3 nfs=20
> rw,v3,rsize=3D8192,wsize=3D8192,hard,tcp,lock,addr=3Dcobra3 0 0
>=20
> 8 K r&wsize ??? I also tried with 16K it's still shows 8K ?

that means your server supports only 8KB transfer sizes. the client and
server negotiate the maximum size of reads and writes at mount time.

have you read some of the excellent reference textbooks listed in the
NFS FAQ? that might help you to become a little more fluent with the
operation of NFS.

> howerver=20
> moving from soft to hard did shows up in /proc/mounts .
> Again, How can we check every options of an NFS mounted filesystem on=20
> the client side ? (other than /proc/mountd) .

some options don't show up when they are at their default settings.
no-one ever specifies "async" as a mount option, as it is the default,
so it is not included in /proc/mounts. NFS version isn't listed in
/proc/mounts unless it was specified on the original mount command.
otherwise, i quite agree with you that every setting should be spelled
out in /proc/mounts, but unfortunately that's not the way it works
today.

the only way to truly see what's going on is to capture a network trace
and load it up in ethereal. in fact, i recommend that as your next step
-- capture about 30 seconds of your test workload with tcpdump to see
what's going over the wire. ethereal also has a nice RPC round trip
average calculator.

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2004-11-22 22:20:57

Yes, I did made a note on this long thread to resume it. it is mainly in
french but mix a lot of english and captures a also in english so I
might be easy for you to read it .
Here it is ;

http://www.int-evry.fr/mci/user/procacci/Doc/nfs.html

check principaly from chapter 8 to the end
thanks everyone for your help.

Alex Mccoll wrote:

>Hi Jehan,
>I've been following your discussion with Neil Brown, thank you
>for your insights! I'm roughly in the same situation, and it's
>been very helpful for me to find your thread!
>Say, did you figure out how to create the journal on a seperate
>device? If you have any notes on that, could you send them
>my way?
>
>Thanks, in advance,
>Alex.
>
>
>On Tue, 23 Nov 2004, jehan procaccia wrote:
>
>
>
>>Date: Tue, 23 Nov 2004 22:46:44 +0100
>>From: jehan procaccia <[email protected]>
>>To: "Lever, Charles" <[email protected]>
>>Cc: [email protected]
>>Subject: Re: [NFS] async vs. sync
>>
>>Lever, Charles wrote:
>>
>>
>>
>>>>This is what I expect in term of performances . I will continue my
>>>>requests on the DEll/EMC hotline , but maybe the security of
>>>>that AX100
>>>>storage Processor (raid5, spare disk, double fiber attachement, UPS)
>>>>allows me to use async export mode in such a case ?
>>>>
>>>>
>>>>
>>>>
>>>the "async" export option changes the behavior of the NFS server
>>>daemons, not of the underlying local file system or storage subsystem.
>>>the problem is that changes made by clients will remain in your NFS
>>>server's memory and not get flushed onto permanent storage.
>>>
>>>so, i really don't think the storage subsystem will have any effect on
>>>the safety of your data before the data reaches permanent storage. as
>>>someone else pointed out earlier, the solution is to use battery-backed
>>>main memory when using "async" (prestoserve for solaris?).
>>>
>>>as trond said, if your users and backup facilities can tolerate the loss
>>>of data during a crash, then it is perfectly fine to use "async." most
>>>don't, however.
>>>
>>>btw, it is fairly well understood that RAID-5 and NFS servers don't mix
>>>well. RAID-5's weakest point is that it doesn't handle small random
>>>writes very well, and that's exactly what is required of it when
>>>handling NFS traffic that consists mostly of metadata changes (file
>>>creates, deletes, and so on). neil explained clearly how to make the
>>>best use of a RAID-5 with NFS: do your local file system journaling
>>>somewhere else.
>>>
>>>
>>>
>>>
>>No, not yet, but if it is safer and increase performances maybe I
>>should do it !
>>
>>Perhaps it's not the place to talk about ext3 here, but if someone on
>>the list did already put their journal on a separate device, please
>>confirm me those points:
>> From what I read on man mkefs for ext3 FS I can create a journal on a
>>separate FS :
>>mke2fs -O journal_dev external-journal
>>creates the journal FS, on which device ? -> internal scsi drive of my
>>server or better placed on the dell/EMC SP ?
>>
>>mke2fs -J device=/dev/external-journal /dev/emcpower
>>Format the FS and use the external journal just create above, but what is the recommended size of the
>>external journal ? when journal is internal it is said the size of the journal must be at least 1024 filesystem blocks
>>(in my case blocks a 4K size) so journal is at least 4 Mb, but should it be bigger ?
>>
>>Finally, can I "externalize" an already internal journal from production FS (convert journal from inside to outside without reformating the FS ) ?
>>
>>thanks.
>>
>>
>>
>>
>>>when trying your workload locally on the NFS server, realize that there
>>>are some optimizations that local file systems make, like caching and
>>>coalescing metadata updates, that the NFS protocol does not allow. this
>>>affects especially workloads with lots of metadata change operations,
>>>because the NFS protocol requires each metadata update to reside on
>>>permanent storage before the NFS server replies to the client,
>>>effectively serializing the workload with storage activity.
>>>
>>>
>>>
>>>
>>
>>-------------------------------------------------------
>>SF email is sponsored by - The IT Product Guide
>>Read honest & candid reviews on hundreds of IT Products from real users.
>>Discover which products truly live up to the hype. Start reading now.
>>http://productguide.itmanagersjournal.com/
>>_______________________________________________
>>NFS maillist - [email protected]
>>https://lists.sourceforge.net/lists/listinfo/nfs
>>
>>
>>

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs