Hi,
we are fighting with a strange performance problem when accessing a
Sun/Solaris-10 server from a Linux NFS client (HP DL380G4, x86_64,
RHEL4U3 user-space, various kernels from 2.6.9-34.ELsmp up to
2.9.21.1).
The underlying FS on the Sun is SAM-FS (4.5.33) which provides HSM
capabilities. The basic layout is that the "online cache" is relatively
small on "high-speed" FC disks, while the offline data are stored on
cheapish SATA disks and high-speed tapes.
The problem arises when accessing (reading) a file that is offline on
the Sun. In this case the so-called "stager" brings it back to the
online-cache first, while the client waits for the data to be
available.
This works fine (staging speed up-to or larger 50 MB/sec) on:
- administrative "stage" request
- accessing the offline file on the Sun-Server itself
- accessing the offline file from a Solaris-10 NFS client
When doing the same from the Linux Client performance drops below 10
MB/sec. Network performance itself looks good when accessing the file
"online".
One thing that looks different when doing tcpdump/snoop traces between
the systems is that in the Linux/Solaris case the client seems to send
more READ3 requests (factor > 3) that the Server acknowledges (both in
the online and offline case). In the Solaris/Solaris case the number of
requests and acknowledgements is about the same.
The remote filesystem is mounted NFS3/TCP with the following
parameters:
xxxx:/net/xxxx/fs03 on /net/xxxx/fs03 type nfs
(rw,hard,intr,bg,nfsvers=3,proto=tcp,timeo=600,rsize=32768,wsize=32768,addr=yy.yy.yy.yy)
We are already talking to the Sun people, but they cannot reproduce.
So I put my hope for some insight on this list.
Cheers
Martin
PS: Please CC me on replies, as I am only getting the digest
------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de
www: http://www.knobisoft.de
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
At 10:20 AM 5/14/2007, Martin Knoblauch wrote:
> not sure whether I should feel stupid. Inspecting the tcpdump log
>(snippet attached), I see a lot of "reply ERR 1448" packets coming from
>the Sun. Which of course could explain a few things.
Can you turn up the tcpdump reporting to show more NFS layer decoding?
The "1448" is just the packet length at this level (which is a bit odd in
itself - why such a large reply?), not the error code.
>12:11:03.201366 IP (tos 0x0, ttl 61, id 42986, offset 0, flags [DF],
>proto: TCP (6), length: 1500) 160.50.118.37.2049 >
>160.50.204.58.1090741696: reply ok 1448
>12:11:03.201442 IP (tos 0x0, ttl 61, id 42987, offset 0, flags [DF],
>proto: TCP (6), length: 1500) 160.50.118.37.2049 >
>160.50.204.58.1076887552: reply ERR 1448
>...
Tom.
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
>At 10:20 AM 5/14/2007, Martin Knoblauch wrote:
>> not sure whether I should feel stupid. Inspecting the tcpdump log
>>(snippet attached), I see a lot of "reply ERR 1448" packets coming
from
>>the Sun. Which of course could explain a few things.
>
>Can you turn up the tcpdump reporting to show more NFS layer decoding?
>The "1448" is just the packet length at this level (which is a bit odd
>in
>itself - why such a large reply?), not the error code.
Hi Tom,
How to turn NFS reporting up? "-vvv" does not give any more info?
I looked at the source to tcpdump/print-nfs.c. The "1448" is just the
length. The difference between "ok" and "ERR" is the value of the
SUNRPC_MSG_ACCEPTED bit in "rp->rm_reply.rp_stat".
>>12:11:03.201366 IP (tos 0x0, ttl 61, id 42986, offset 0, flags [DF],
>>proto: TCP (6), length: 1500) 160.50.118.37.2049 >
>>160.50.204.58.1090741696: reply ok 1448
>>12:11:03.201442 IP (tos 0x0, ttl 61, id 42987, offset 0, flags [DF],
>>proto: TCP (6), length: 1500) 160.50.118.37.2049 >
>>160.50.204.58.1076887552: reply ERR 1448
>>...
>
>Tom.
------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de
www: http://www.knobisoft.de
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
At 12:45 PM 5/14/2007, Martin Knoblauch wrote:
>>At 10:20 AM 5/14/2007, Martin Knoblauch wrote:
>>> not sure whether I should feel stupid. Inspecting the tcpdump log
>>>(snippet attached), I see a lot of "reply ERR 1448" packets coming
>from
>>>the Sun. Which of course could explain a few things.
>>
>>Can you turn up the tcpdump reporting to show more NFS layer decoding?
>>The "1448" is just the packet length at this level (which is a bit odd
>
>>in
>>itself - why such a large reply?), not the error code.
>Hi Tom,
>
> How to turn NFS reporting up? "-vvv" does not give any more info?
Actually, I don't use tcpdump any more - can you use wireshark (ethereal)?
If you've saved a captured trace, just open it with them. Then zoom in with
the gui.
Tom.
>
> I looked at the source to tcpdump/print-nfs.c. The "1448" is just the
>length. The difference between "ok" and "ERR" is the value of the
>SUNRPC_MSG_ACCEPTED bit in "rp->rm_reply.rp_stat".
>
>>>12:11:03.201366 IP (tos 0x0, ttl 61, id 42986, offset 0, flags [DF],
>
>>>proto: TCP (6), length: 1500) 160.50.118.37.2049 >
>>>160.50.204.58.1090741696: reply ok 1448
>>>12:11:03.201442 IP (tos 0x0, ttl 61, id 42987, offset 0, flags [DF],
>
>>>proto: TCP (6), length: 1500) 160.50.118.37.2049 >
>>>160.50.204.58.1076887552: reply ERR 1448
>>>...
>>
>>Tom.
>
>
>------------------------------------------------------
>Martin Knoblauch
>email: k n o b i AT knobisoft DOT de
>www: http://www.knobisoft.de
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs