2.4.20rc1 still exhibits the same identical problem as 2.4.19+kmap1 patch
which I identified a few weeks ago. I guess that patch was applied to
2.4.20, since it doesn't appear on Trond's patch page for 2.4.20.
I am still available to help duplicate or test fixes to this problem, if
anyone has any ideas. I can still use UDP but I'd really like to start
getting the speed of TCP.
thanks,
andrew
---------- Forwarded message ----------
Date: Mon, 14 Oct 2002 16:29:55 -0700 (PDT)
From: Andrew Ryan <[email protected]>
To: [email protected]
Subject: 2.4.19+RPC_ALL hangs running dbench 2.0
I've been running tests on the 2.4.19_NFS_ALL (the one from Oct 5) kernel
and seeing an easily reproducible hang on my machine (2x1.4 GHz PIII,
Compaq DL380G2, 4GB RAM), mounting a Netapp (F820 running 6.2R2) with the
mount options:
rw,tcp,nfsvers=3,rsize=32768,wsize=32768,intr,hard
The symptom is, I start a dbench run, and it starts up and runs for a
bit...
$ ~/dbench-2.0/dbench 16
clients started
16 23801 21.45 MB/sec
Then it gets hung up, and the dbench process is still running, and
the MB/sec number keeps dropping rapidly, approaching 0.
At this point:
* Any commands in other shells that are currently running (e.g. 'top') are
hung.
* My other shells are not hung, but if I try to execute any commands, the
commands hang forever.
* I can kill the dbench process with Ctrl-C, but that just gives me a
shell that cannot execute any commands (they all hang, like the other
shells).
* The nmi_watchdog is never triggered, even though the system is
completely unresponsive from a user level.
When I ctl-C the hung dbench process, sometimes the kernel generates
an oops, but other times not. If I have kdb on, I can get a backtrace, but
I was hoping there was an easier way to figure out what is causing this
bug. The one oops I get says something about 'kernel BUG at
highmem.c:159!'
Note I do *NOT* get this error if I run without the NFS_ALL. I also tested
this with just the RPC_ALL and I get the same error. So it definitely has
to be something in the RPC_ALL patchset. I'm confused though, bec. this is
the patchset which claims to have specific fixes for HIGHMEM.
All I really want is a fast, stable client for my 4GB, 2 CPU boxes. I'd
use the stock 2.4.19 but the RPC_ALL patchset leads me to believe that
there are HIGHMEM bugs in the stock 2.4.19 NFS client.
I'm willing to do some testing to chase this down, if it helps.
andrew
-------------------------------------------------------
This sf.net email is sponsored by: Influence the future
of Java(TM) technology. Join the Java Community
Process(SM) (JCP(SM)) program now.
http://ads.sourceforge.net/cgi-bin/redirect.pl?sunm0004en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
>>>>> " " == Andrew Ryan <[email protected]> writes:
> 2.4.20rc1 still exhibits the same identical problem as
> 2.4.19+kmap1 patch which I identified a few weeks ago. I guess
> that patch was applied to 2.4.20, since it doesn't appear on
> Trond's patch page for 2.4.20.
Does the following patch work for you?
Cheers,
Trond
diff -u --recursive --new-file linux-2.4.20-rc1/net/sunrpc/xdr.c linux-2.4.20-fix/net/sunrpc/xdr.c
--- linux-2.4.20-rc1/net/sunrpc/xdr.c 2002-08-14 09:00:35.000000000 -0400
+++ linux-2.4.20-fix/net/sunrpc/xdr.c 2002-11-01 12:08:52.000000000 -0500
@@ -244,6 +244,7 @@
pglen -= base;
base += xdr->page_base;
ppage += base >> PAGE_CACHE_SHIFT;
+ pglen += base & ~PAGE_CACHE_MASK;
}
for (;;) {
flush_dcache_page(*ppage);
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Friday 01 November 2002 12:44 pm, you wrote:
> " " == Andrew Ryan <[email protected]> writes:
> > 2.4.20rc1 still exhibits the same identical problem as
> > 2.4.19+kmap1 patch which I identified a few weeks ago. I guess
> > that patch was applied to 2.4.20, since it doesn't appear on
> > Trond's patch page for 2.4.20.
>
> Does the following patch work for you?
So far so good on the crashes. I'm able to get through a complete run of
dbench using TCP mounts on 2.4.20rc1, which I haven't been able to do before
this.
However, according to nfsstat -c's "Client rpc stats", I'm getting around a
10:1 ratio of retransmits to calls. That's bad. I get 0 retransmits when
using 2.4.19+kmap1, and any previous kernel I've ever tried TCP mounting
with. I checked 2.4.20rc1-vanilla and it does the same thing, so it's not the
latest patch.
andrewr
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
>>>>> " " == Andrew Ryan <[email protected]> writes:
> However, according to nfsstat -c's "Client rpc stats", I'm
> getting around a 10:1 ratio of retransmits to calls. That's
> bad. I get 0 retransmits when using 2.4.19+kmap1, and any
> previous kernel I've ever tried TCP mounting with. I checked
> 2.4.20rc1-vanilla and it does the same thing, so it's not the
> latest patch.
It is probably just the (known) accounting error in the 'retransmit'
statistics. Try applying the patch
http://www.fys.uio.no/~trondmy/src/2.4.20-pre9/linux-2.4.20-01-call_start.dif
if it bothers you.
Note: the TCP timeout + retransmits will (unlike UDP) be of the order
of 60 seconds, so if these were real retransmits, you'd be seeing a
*very* noticeable slowdown...
Cheers,
Trond
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
>>>>> Andrew Ryan <[email protected]> writes:
> So far so good on the crashes. I'm able to get through a
> complete run of dbench using TCP mounts on 2.4.20rc1, which I
> haven't been able to do before this.
Marcelo, Linus
We've uncovered yet another kmap imbalance in the new RPC code. This
looks like it might be the last one (my debugging printks have been
unable to unearth any more). One line fix + 4 line comment
appended. Please apply to both 2.4.20-rc1 and 2.5.45...
Cheers,
Trond
diff -u --recursive --new-file linux-2.4.20-rc1/net/sunrpc/xdr.c linux-2.4.20-kmap_fix/net/sunrpc/xdr.c
--- linux-2.4.20-rc1/net/sunrpc/xdr.c 2002-08-14 09:00:35.000000000 -0400
+++ linux-2.4.20-kmap_fix/net/sunrpc/xdr.c 2002-11-01 16:50:09.000000000 -0500
@@ -244,6 +244,11 @@
pglen -= base;
base += xdr->page_base;
ppage += base >> PAGE_CACHE_SHIFT;
+ /* Note: The offset means that the length of the first
+ * page is really (PAGE_CACHE_SIZE - (base & ~PAGE_CACHE_MASK)).
+ * In order to avoid an extra test inside the loop,
+ * we bump pglen here, and just subtract PAGE_CACHE_SIZE... */
+ pglen += base & ~PAGE_CACHE_MASK;
}
for (;;) {
flush_dcache_page(*ppage);
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
> >>>>> " " == Andrew Ryan <[email protected]> writes:
>
> > However, according to nfsstat -c's "Client rpc stats", I'm
> > getting around a 10:1 ratio of retransmits to calls. That's
> > bad. I get 0 retransmits when using 2.4.19+kmap1, and any
> > previous kernel I've ever tried TCP mounting with. I checked
> > 2.4.20rc1-vanilla and it does the same thing, so it's not the
> > latest patch.
>
> It is probably just the (known) accounting error in the
> 'retransmit' statistics. Try applying the patch
>
http://www.fys.uio.no/~trondmy/src/2.4.20-pre9/linux-2.4.20-01-call_start.dif
> if it bothers you.
call_start changes the way RPCs are counted, not how retransmits are counted.
i suspect andrew is seeing partial transmits -- in 2.4.20-pre, partial
TCP transmissions are counted as retransmits. in fact, it looks like the
patch i sent in august for 2.4.20-pre5 to fix xprt_write_space was never
applied to 2.4. trond, can you ping marcelo on this? it's a pretty
important performance enhancement that should be in 2.4.20.
> Note: the TCP timeout + retransmits will (unlike UDP) be of the order
> of 60 seconds, so if these were real retransmits, you'd be seeing a
> *very* noticeable slowdown...
except that mount is broken and sets a 6 second TCP retransmission default
rather than the standard value of 60 seconds.
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
>>>>> " " == Charles Lever <Lever> writes:
> call_start changes the way RPCs are counted, not how
> retransmits are counted.
Wrong. It fixes the following bug too:
@@ -645,7 +663,6 @@
case -ENOMEM:
case -EAGAIN:
task->tk_action = call_transmit;
- clnt->cl_stats->rpcretrans++;
break;
default:
if (clnt->cl_chatty)
Cheers,
Trond
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
> >>>>> " " == Charles Lever <Lever> writes:
>
> > call_start changes the way RPCs are counted, not how
> > retransmits are counted.
>
> Wrong. It fixes the following bug too:
>
> @@ -645,7 +663,6 @@
> case -ENOMEM:
> case -EAGAIN:
> task->tk_action = call_transmit;
> - clnt->cl_stats->rpcretrans++;
> break;
> default:
> if (clnt->cl_chatty)
oops, i forgot about that. it doesn't obviate the need for the
xprt_write_space fix, however.
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
At 10:58 PM 11/1/02 +0100, Trond Myklebust wrote:
>It is probably just the (known) accounting error in the 'retransmit'
>statistics. Try applying the patch
>
>
>http://www.fys.uio.no/~trondmy/src/2.4.20-pre9/linux-2.4.20-01-call_start.dif
>
>if it bothers you.
Yes, you are right, it seems the counter is broken, and performance is
fine. I did bonnie++ runs with 2.4.20rc1 kernels with and without the
call_start patch, and didn't see any significant difference.
But, it does bother me since I want to be able to diagnose network errors
if I ever have them, and with a busted counter it will be impossible to
tell what my real error rate is. Can the call_start patch go to Marcelo for
2.4.20rc2?
After a bonnie run with the call_start patch, I get this, which is much
better :)
$ nfsstat -c
Client rpc stats:
calls retrans authrefrsh
1648050 0 0
thanks,
andrew
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
>>>>> Andrew Ryan writes:
> At 10:58 PM 11/1/02 +0100, Trond Myklebust wrote:
>> It is probably just the (known) accounting error in the 'retransmit'
>> statistics. Try applying the patch
>>
>> http://www.fys.uio.no/~trondmy/src/2.4.20-pre9/linux-2.4.20-01-call_start.dif
>>
>> if it bothers you.
> Yes, you are right, it seems the counter is broken, and performance is
> fine. I did bonnie++ runs with 2.4.20rc1 kernels with and without the
> call_start patch, and didn't see any significant difference.
> But, it does bother me since I want to be able to diagnose network errors if
> I ever have them, and with a busted counter it will be impossible to tell
> what my real error rate is. Can the call_start patch go to Marcelo for
> 2.4.20rc2?
I'm an end user with approximately 3000 Linux NFS clients.
I will really appreciate correct counters when I'm debugging
nfs server, client and network problems. I've had to ignore
spurious counts of nfs retrans while debugging network issues.
I hope this patch will be in 2.4.20rc2. Thanks to all of you for
providing an excellent client-side implementation. And I am
serious.
> After a bonnie run with the call_start patch, I get this, which is much
> better :)
> $ nfsstat -c
> Client rpc stats:
> calls retrans authrefrsh
> 1648050 0 0
--
Quentin Fennessy [email protected]
Office: 512.602.3873
Cell: 512.694.7489
-------------------------------------------------------
This SF.net email is sponsored by: ApacheCon, November 18-21 in
Las Vegas (supported by COMDEX), the only Apache event to be
fully supported by the ASF. http://www.apachecon.com
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
> > >It is probably just the (known) accounting error in the 'retransmit'
> > >statistics. Try applying the patch
> > >
> > >
> > >http://www.fys.uio.no/~trondmy/src/2.4.20-pre9/linux-2.4.20-0
1-call_start.dif
> >
> >if it bothers you.
>
> Yes, you are right, it seems the counter is broken, and performance is
> fine. I did bonnie++ runs with 2.4.20rc1 kernels with and without the
> call_start patch, and didn't see any significant difference.
the call_start patch only changes RPC accounting. it should have no
effects (either positive or negative) on performance.
did you try the xprt_write_space patch? that fixes a real problem,
and addresses the root cause of the high retransmission counts you see.
> But, it does bother me since I want to be able to diagnose network errors
> if I ever have them, and with a busted counter it will be impossible to
> tell what my real error rate is.
the retransmit counter is supposed to count only whole RPC retransmissions,
not TCP partial transmissions, which are not due to network problems. if
the xprt_write_space patch was included, the false retransmit rate would
be much lower anyway (although not zero).
> Can the call_start patch go to Marcelo for 2.4.20rc2?
call_start is available in Trond's NFS_ALL patch for now. i think 2.4.20
is closed for changes like this.
-------------------------------------------------------
This SF.net email is sponsored by: ApacheCon, November 18-21 in
Las Vegas (supported by COMDEX), the only Apache event to be
fully supported by the ASF. http://www.apachecon.com
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs