2002-07-04 11:24:57

by Jeremy Sanders

[permalink] [raw]
Subject: Slow performance with 2.4.19pre1

Using a 2.4.19pre1 on the server (ext3 & raid1 on an IDE disk) I'm getting
very poor read and rewrite performance using nfs. There's no difference
using RedHat's 2.4.18-5 kernel on the server. The client is running
RedHat's 2.4.18-5 kernel. Here are the results from running bonnie++:

Version 1.02a ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
xpc1.ast.cam.ac. 1G 9242 52 9452 7 644 0 1624 8 1731 1 488.7 3
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 1207 3 182 96 2157 7 1224 3 4521 5 1766 4
xpc1.ast.cam.ac.uk,1G,9242,52,9452,7,644,0,1624,8,1731,1,488.7,3,16,1207,3,182,96,2157,7,1224,3,4521,5,1766,4

The machine is connected over 100Mps ethernet, full duplex. The mount
options are below:

xpc1:/<1>nfs/test> cat /proc/mounts
[...]
xserv1.ast.cam.ac.uk:/mnt/raid /mnt/nfs nfs
rw,v3,rsize=8192,wsize=8192,hard,intr,udp,lock,addr=xserv1.ast.cam.ac.uk 0 0

Local performance is very good (note different file size):

Version 1.02a ------Sequential Output------ --Sequential Input----Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
xserv1.ast.cam.a 2G 22311 96 40600 13 16150 4 24853 96 45830 5 323.0 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 2715 98 +++++ +++ +++++ +++ 2784 98 +++++ +++ 8798 99
xserv1.ast.cam.ac.uk,2G,22311,96,40600,13,16150,4,24853,96,45830,5,323.0,0,16,2715,98,+++++,+++,+++++,+++,2784,98,+++++,+++,8798,99

The network card and duplex/speed settings look okay (eepro100):

[root@xserv1 raid]# /root/mii-diag
Using the default interface 'eth0'.
Basic registers of MII PHY #1: 3000 782d 02a8 0154 05e1 45e1 0003 0000.
The autonegotiated capability is 01e0.
The autonegotiated media type is 100baseTx-FD.
Basic mode control register 0x3000: Auto-negotiation enabled.
You have link beat, and everything is working OK.
Your link partner advertised 45e1: Flow-control 100baseTx-FD 100baseTx
10baseT-FD 10baseT, w/ 802.3X flow control.
End of basic transceiver information.

However, if the client runs 2.4.17 instead of
2.4.18/2.4.19rc1/2.4.18-redhat then the read speed is much better:

Version 1.02a ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
xpc17.ast.cam.ac 1G 11255 74 10539 5 5239 4 11336 78 11260 3 575.6 3
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 1179 3 4176 14 2318 8 1230 4 4504 12 1755 6
xpc17.ast.cam.ac.uk,1G,11255,74,10539,5,5239,4,11336,78,11260,3,575.6,3,16,1179,3,4176,14,2318,8,1230,4,4504,12,1755,6
(same mount options as above)

Any ideas what problem there is in the 2.4.18+ nfs client? Is this a
networking thing showing up under 2.4.18?

This looks like RedHat's bug 67625.

Jeremy

--
Jeremy Sanders <[email protected]> http://www-xray.ast.cam.ac.uk/~jss/
X-Ray Group, Institute of Astronomy, University of Cambridge, UK.
Public Key Server PGP Key ID: E1AAE053











-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Caffeinated soap. No kidding.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs


2002-07-04 15:25:49

by Jeremy Sanders

[permalink] [raw]
Subject: Re: Slow performance with 2.4.19pre1

Sorry - I know it's bad form to reply to one's own emails. It seems that
the stock 2.4.19rc1 kernel doesn't have the problem as a client, and so
it's the RedHat patched 2.4.18 kernel which is a very slow client. This
looks like a RedHat bug.

Also the title of this mail is rather confusing!

Jeremy

--
Jeremy Sanders <[email protected]> http://www-xray.ast.cam.ac.uk/~jss/
X-Ray Group, Institute of Astronomy, University of Cambridge, UK.
Public Key Server PGP Key ID: E1AAE053



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Caffeinated soap. No kidding.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2002-07-04 20:33:54

by Bryan O'Sullivan

[permalink] [raw]
Subject: Re: Re: Slow performance with 2.4.19pre1

On Thu, 2002-07-04 at 08:25, Jeremy Sanders wrote:
> Sorry - I know it's bad form to reply to one's own emails. It seems that
> the stock 2.4.19rc1 kernel doesn't have the problem as a client, and so
> it's the RedHat patched 2.4.18 kernel which is a very slow client. This
> looks like a RedHat bug.

Yes, you're correct. The kernels Red Hat has been shipping with 7.3
have atrocious NFS client behaviour, and will bring some servers (such
as Netapp boxes) to a halt.

<b


Attachments:
signature.asc (232.00 B)
This is a digitally signed message part

2002-07-04 23:55:51

by H. J. Lu

[permalink] [raw]
Subject: Re: Re: Slow performance with 2.4.19pre1

On Thu, Jul 04, 2002 at 01:33:52PM -0700, Bryan O'Sullivan wrote:
> On Thu, 2002-07-04 at 08:25, Jeremy Sanders wrote:
> > Sorry - I know it's bad form to reply to one's own emails. It seems that
> > the stock 2.4.19rc1 kernel doesn't have the problem as a client, and so
> > it's the RedHat patched 2.4.18 kernel which is a very slow client. This
> > looks like a RedHat bug.
>=20
> Yes, you're correct. The kernels Red Hat has been shipping with 7.3
> have atrocious NFS client behaviour, and will bring some servers (such
> as Netapp boxes) to a halt.
>=20

Have you tried the 7.3 kernel update. It is 2.4.18-5.


H.J.


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Caffeinated soap. No kidding.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2002-07-05 07:53:38

by Bryan O'Sullivan

[permalink] [raw]
Subject: Re: Re: Slow performance with 2.4.19pre1

On Thu, 2002-07-04 at 16:55, H. J. Lu wrote:

> Have you tried the 7.3 kernel update. It is 2.4.18-5.

No, I usually run my own kernels. I only noticed the horror of the
2.4.18-3 NFS client performance as I was installing enough bits on a new
machine to replace the stock kernel with my own.

<b


Attachments:
signature.asc (232.00 B)
This is a digitally signed message part

2002-07-05 09:10:00

by Jeremy Sanders

[permalink] [raw]
Subject: Re: Re: Slow performance with 2.4.19pre1

On Thu, 4 Jul 2002, H. J. Lu wrote:

> On Thu, Jul 04, 2002 at 01:33:52PM -0700, Bryan O'Sullivan wrote:
> > On Thu, 2002-07-04 at 08:25, Jeremy Sanders wrote:
> > > Sorry - I know it's bad form to reply to one's own emails. It seems that
> > > the stock 2.4.19rc1 kernel doesn't have the problem as a client, and so
> > > it's the RedHat patched 2.4.18 kernel which is a very slow client. This
> > > looks like a RedHat bug.
> >
> > Yes, you're correct. The kernels Red Hat has been shipping with 7.3
> > have atrocious NFS client behaviour, and will bring some servers (such
> > as Netapp boxes) to a halt.
> >
>
> Have you tried the 7.3 kernel update. It is 2.4.18-5.

The tests were with 2.4.18-5 on an athlon.

Jeremy

--
Jeremy Sanders <[email protected]> http://www-xray.ast.cam.ac.uk/~jss/
X-Ray Group, Institute of Astronomy, University of Cambridge, UK.
Public Key Server PGP Key ID: E1AAE053



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Bringing you mounds of caffeinated joy.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2002-07-05 09:48:43

by Yusuf Goolamabbas

[permalink] [raw]
Subject: Re: Re: Slow performance with 2.4.19pre1

This thread on lkml may provide some insight

http://marc.theaimsgroup.com/?t=102519385600004&r=1&w=2

> > Have you tried the 7.3 kernel update. It is 2.4.18-5.
>
> The tests were with 2.4.18-5 on an athlon.
>
> Jeremy

--
Yusuf Goolamabbas
[email protected]


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Bringing you mounds of caffeinated joy.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs