2003-09-12 21:47:15

by Scott Mcdermott

[permalink] [raw]
Subject: "reply ERR" tcpdump output means what (TCP mounts over WAN)

I'm mounting a v3 export from a 2.4.22-rc2 server to a
2.4.22-pre4 client, over a WAN connection (it's over an
IPSEC tunnel, 1.5Mbits server -> client and 128Kbits client
-> server, 90ms round trip). I simply mounted -o rw,tcp. I
used a reliable transport because I thought it would be more
approporate over a link like this.

It works fine, but it's REALLY slow.

Here's some output from loading a Maildir-format spool (the
below is from the server)

17:22:31.910279 10.20.8.30.2049 > 10.0.0.8.1698577506: reply ERR 1376 (DF) [tos 0x2,ECT]
17:22:32.029639 10.20.8.30.2049 > 10.0.0.8.544434554: reply ERR 1376 (DF) [tos 0x2,ECT]
17:22:32.159093 10.20.8.30.2049 > 10.0.0.8.2848670243: reply ok 236 (DF) [tos 0x2,ECT]
17:22:32.260640 10.20.8.30.2049 > 10.0.0.8.2865447459: reply ok 1376 (DF) [tos 0x2,ECT]
17:22:32.260679 10.20.8.30.2049 > 10.0.0.8.1026770003: reply ERR 1376 (DF) [tos 0x2,ECT]
17:22:32.370613 10.20.8.30.2049 > 10.0.0.8.1634623304: reply ERR 1376 (DF) [tos 0x2,ECT]
17:22:32.370638 10.20.8.30.2049 > 10.0.0.8.2002875168: reply ERR 68 (DF) [tos 0x2,ECT]
17:22:32.589138 10.20.8.30.2049 > 10.0.0.8.2899001891: reply ok 1376 (DF) [tos 0x2,ECT]
17:22:32.809437 10.20.8.30.2049 > 10.0.0.8.2932556323: reply ok 1376 (DF) [tos 0x2,ECT]
17:22:32.809484 10.20.8.30.2049 > 10.0.0.8.1733515061: reply ERR 1376 (DF) [tos 0x2,ECT]
17:22:32.919132 10.20.8.30.2049 > 10.0.0.8.1229416566: reply ERR 100 (DF) [tos 0x2,ECT]
17:22:33.020512 10.20.8.30.2049 > 10.0.0.8.2949333539: reply ok 736 (DF) [tos 0x2,ECT]
17:22:33.121527 10.20.8.30.2049 > 10.0.0.8.2966110755: reply ok 236 (DF) [tos 0x2,ECT]
17:22:33.218590 10.20.8.30.2049 > 10.0.0.8.2982887971: reply ok 1376 (DF) [tos 0x2,ECT]
17:22:33.218627 10.20.8.30.2049 > 10.0.0.8.1932422757: reply ERR 816 (DF) [tos 0x2,ECT]
17:22:33.342968 10.20.8.30.2049 > 10.0.0.8.2999665187: reply ok 236 (DF) [tos 0x2,ECT]
17:22:33.437201 10.20.8.30.2049 > 10.0.0.8.3016442403: reply ok 1376 (DF) [tos 0x2,ECT]
17:22:33.437239 10.20.8.30.2049 > 10.0.0.8.812402511: reply ERR 1376 (DF) [tos 0x2,ECT]
17:22:33.547541 10.20.8.30.2049 > 10.0.0.8.896350316: reply ERR 1376 (DF) [tos 0x2,ECT]
17:22:33.547566 10.20.8.30.2049 > 10.0.0.8.1112298345: reply ERR 1376 (DF) [tos 0x2,ECT]
17:22:33.659641 10.20.8.30.2049 > 10.0.0.8.927545443: reply ERR 1376 (DF) [tos 0x2,ECT]
17:22:33.698638 10.20.8.30.2049 > 10.0.0.8.943937290: reply ERR 1376 (DF) [tos 0x2,ECT]
17:22:33.707383 10.20.8.30.nfs > 10.0.0.8.798: . ack 1967536866 win 22512 <nop,nop,timestamp 244760041 7925237> (DF)
17:22:33.718735 10.20.8.30.nfs > 10.0.0.8.798: . ack 129 win 22512 <nop,nop,timestamp 244760042 7925237> (DF)
17:22:33.769462 10.20.8.30.2049 > 10.0.0.8.1111057268: reply ERR 1376 (DF) [tos 0x2,ECT]
17:22:33.769492 10.20.8.30.2049 > 10.0.0.8.894593401: reply ERR 1376 (DF) [tos 0x2,ECT]
17:22:33.818440 10.20.8.30.2049 > 10.0.0.8.1295467855: reply ERR 1376 (DF) [tos 0x2,ECT]
17:22:33.881110 10.20.8.30.2049 > 10.0.0.8.1647340856: reply ERR 1376 (DF) [tos 0x2,ECT]
17:22:33.919503 10.20.8.30.2049 > 10.0.0.8.946622820: reply ERR 1376 (DF) [tos 0x2,ECT]
17:22:33.965985 10.20.8.30.2049 > 10.0.0.8.1314288944: reply ERR 1376 (DF) [tos 0x2,ECT]
17:22:33.990356 10.20.8.30.2049 > 10.0.0.8.1448767600: reply ERR 1376 (DF) [tos 0x2,ECT]
17:22:34.029968 10.20.8.30.2049 > 10.0.0.8.794908245: reply ERR 1376 (DF) [tos 0x2,ECT]
17:22:34.102846 10.20.8.30.2049 > 10.0.0.8.1181575987: reply ERR 608 (DF) [tos 0x2,ECT]
17:22:34.217498 10.20.8.30.nfs > 10.0.0.8.798: . ack 545 win 22512 <nop,nop,timestamp 244760092 7925288> (DF)
17:22:34.224557 10.20.8.30.2049 > 10.0.0.8.3117105699: reply ok 1376 (DF) [tos 0x2,ECT]
17:22:34.442064 10.20.8.30.2049 > 10.0.0.8.3150660131: reply ok 1376 (DF) [tos 0x2,ECT]
17:22:34.442103 10.20.8.30.2049 > 10.0.0.8.1935897193: reply ERR 1376 (DF) [tos 0x2,ECT]

what are these ERRs and how to avoid them? I assume they're
not normal?


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs


2003-09-15 15:50:07

by Lever, Charles

[permalink] [raw]
Subject: RE: "reply ERR" tcpdump output means what (TCP mounts over WAN)

hi scott-

can you provide a pcap format tcpdump? the text output below
omits most of the details we need to help you with your problem.

on your client:

su
tcpdump -s0 -w/tmp/dump.pcap host name-of-your-server-here

i've found that on slow asymmetric WAN links, using the "sync"
mount option on the client helps performance, and prevents
your NFS client from hogging all available network bandwidth
at the expense of other (interactive) sessions.

you can also try fiddling with your client's read ahead
parameters (see "sysctl" to tune vm.min-readahead and
vm.max-readahead).


> -----Original Message-----
> From: Scott Mcdermott [mailto:[email protected]]
> Sent: Friday, September 12, 2003 5:47 PM
> To: [email protected]
> Cc: Scott McDermott
> Subject: [NFS] "reply ERR" tcpdump output means what (TCP mounts over
> WAN)
>=20
>=20
> I'm mounting a v3 export from a 2.4.22-rc2 server to a
> 2.4.22-pre4 client, over a WAN connection (it's over an
> IPSEC tunnel, 1.5Mbits server -> client and 128Kbits client
> -> server, 90ms round trip). I simply mounted -o rw,tcp. I
> used a reliable transport because I thought it would be more
> approporate over a link like this.
>=20
> It works fine, but it's REALLY slow.
>=20
> Here's some output from loading a Maildir-format spool (the
> below is from the server)
>=20
> 17:22:31.910279 10.20.8.30.2049 > 10.0.0.8.1698577506: reply=20
> ERR 1376 (DF) [tos 0x2,ECT]
> 17:22:32.029639 10.20.8.30.2049 > 10.0.0.8.544434554: reply=20
> ERR 1376 (DF) [tos 0x2,ECT]
> 17:22:32.159093 10.20.8.30.2049 > 10.0.0.8.2848670243: reply=20
> ok 236 (DF) [tos 0x2,ECT]
> 17:22:32.260640 10.20.8.30.2049 > 10.0.0.8.2865447459: reply=20
> ok 1376 (DF) [tos 0x2,ECT]
> 17:22:32.260679 10.20.8.30.2049 > 10.0.0.8.1026770003: reply=20
> ERR 1376 (DF) [tos 0x2,ECT]
> 17:22:32.370613 10.20.8.30.2049 > 10.0.0.8.1634623304: reply=20
> ERR 1376 (DF) [tos 0x2,ECT]
> 17:22:32.370638 10.20.8.30.2049 > 10.0.0.8.2002875168: reply=20
> ERR 68 (DF) [tos 0x2,ECT]
> 17:22:32.589138 10.20.8.30.2049 > 10.0.0.8.2899001891: reply=20
> ok 1376 (DF) [tos 0x2,ECT]
> 17:22:32.809437 10.20.8.30.2049 > 10.0.0.8.2932556323: reply=20
> ok 1376 (DF) [tos 0x2,ECT]
> 17:22:32.809484 10.20.8.30.2049 > 10.0.0.8.1733515061: reply=20
> ERR 1376 (DF) [tos 0x2,ECT]
> 17:22:32.919132 10.20.8.30.2049 > 10.0.0.8.1229416566: reply=20
> ERR 100 (DF) [tos 0x2,ECT]
> 17:22:33.020512 10.20.8.30.2049 > 10.0.0.8.2949333539: reply=20
> ok 736 (DF) [tos 0x2,ECT]
> 17:22:33.121527 10.20.8.30.2049 > 10.0.0.8.2966110755: reply=20
> ok 236 (DF) [tos 0x2,ECT]
> 17:22:33.218590 10.20.8.30.2049 > 10.0.0.8.2982887971: reply=20
> ok 1376 (DF) [tos 0x2,ECT]
> 17:22:33.218627 10.20.8.30.2049 > 10.0.0.8.1932422757: reply=20
> ERR 816 (DF) [tos 0x2,ECT]
> 17:22:33.342968 10.20.8.30.2049 > 10.0.0.8.2999665187: reply=20
> ok 236 (DF) [tos 0x2,ECT]
> 17:22:33.437201 10.20.8.30.2049 > 10.0.0.8.3016442403: reply=20
> ok 1376 (DF) [tos 0x2,ECT]
> 17:22:33.437239 10.20.8.30.2049 > 10.0.0.8.812402511: reply=20
> ERR 1376 (DF) [tos 0x2,ECT]
> 17:22:33.547541 10.20.8.30.2049 > 10.0.0.8.896350316: reply=20
> ERR 1376 (DF) [tos 0x2,ECT]
> 17:22:33.547566 10.20.8.30.2049 > 10.0.0.8.1112298345: reply=20
> ERR 1376 (DF) [tos 0x2,ECT]
> 17:22:33.659641 10.20.8.30.2049 > 10.0.0.8.927545443: reply=20
> ERR 1376 (DF) [tos 0x2,ECT]
> 17:22:33.698638 10.20.8.30.2049 > 10.0.0.8.943937290: reply=20
> ERR 1376 (DF) [tos 0x2,ECT]
> 17:22:33.707383 10.20.8.30.nfs > 10.0.0.8.798: . ack=20
> 1967536866 win 22512 <nop,nop,timestamp 244760041 7925237> (DF)
> 17:22:33.718735 10.20.8.30.nfs > 10.0.0.8.798: . ack 129 win=20
> 22512 <nop,nop,timestamp 244760042 7925237> (DF)
> 17:22:33.769462 10.20.8.30.2049 > 10.0.0.8.1111057268: reply=20
> ERR 1376 (DF) [tos 0x2,ECT]
> 17:22:33.769492 10.20.8.30.2049 > 10.0.0.8.894593401: reply=20
> ERR 1376 (DF) [tos 0x2,ECT]
> 17:22:33.818440 10.20.8.30.2049 > 10.0.0.8.1295467855: reply=20
> ERR 1376 (DF) [tos 0x2,ECT]
> 17:22:33.881110 10.20.8.30.2049 > 10.0.0.8.1647340856: reply=20
> ERR 1376 (DF) [tos 0x2,ECT]
> 17:22:33.919503 10.20.8.30.2049 > 10.0.0.8.946622820: reply=20
> ERR 1376 (DF) [tos 0x2,ECT]
> 17:22:33.965985 10.20.8.30.2049 > 10.0.0.8.1314288944: reply=20
> ERR 1376 (DF) [tos 0x2,ECT]
> 17:22:33.990356 10.20.8.30.2049 > 10.0.0.8.1448767600: reply=20
> ERR 1376 (DF) [tos 0x2,ECT]
> 17:22:34.029968 10.20.8.30.2049 > 10.0.0.8.794908245: reply=20
> ERR 1376 (DF) [tos 0x2,ECT]
> 17:22:34.102846 10.20.8.30.2049 > 10.0.0.8.1181575987: reply=20
> ERR 608 (DF) [tos 0x2,ECT]
> 17:22:34.217498 10.20.8.30.nfs > 10.0.0.8.798: . ack 545 win=20
> 22512 <nop,nop,timestamp 244760092 7925288> (DF)
> 17:22:34.224557 10.20.8.30.2049 > 10.0.0.8.3117105699: reply=20
> ok 1376 (DF) [tos 0x2,ECT]
> 17:22:34.442064 10.20.8.30.2049 > 10.0.0.8.3150660131: reply=20
> ok 1376 (DF) [tos 0x2,ECT]
> 17:22:34.442103 10.20.8.30.2049 > 10.0.0.8.1935897193: reply=20
> ERR 1376 (DF) [tos 0x2,ECT]
>=20
> what are these ERRs and how to avoid them? I assume they're
> not normal?
>=20
>=20
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> NFS maillist - [email protected]
> https://lists.sourceforge.net/lists/listinfo/nfs
>=20


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs