Does anyone have any ideas about this?
We have been experiencing a number of issues regarding NFS serving on
SGI IRIX (6.5.19) servers which now appear to point back to Linux NFS
clients.
The server is an Origin 2000, quad processors, 1.5GB memory, serving
over Gigabit. This can be reproduced on connections whether or not they
are using jumbo frames. The disks that are served are hardware RAID3
capable of around 80-90MB/s (tested and optimised not theoretical).
Under certain circumstances will get a load average of 8 (or how ever
many nfsd's are running). The server itself is not that busy. Most of
the nfsd's show as sleeping but they are obviously waiting for
something.
Further investigation revealed that this situation arises when a Linux
client opens many files in quick succession (doing
open-read-close,open-read-close). We tend to use files in the range
3MB-13MB in size generally.
If we do the same test on an SGI client there is very little impact on
the load average.
We can recreate this by doing (on the client);
find . -type -f -print -exec grep jsdhgkhjfdk {} \;
The same effect appears to happen on Linux servers as well. A single
Linux client doing this will make the load average creep up towards 2.5.
Linux Version 2.4.20
nfs-utils-1.0.1-1
exports are exported with -32bitclients
Looking a bit deeper in a packet trace when the grep is running, Linux
seems to issue 16 read requests (16KB) at a time whilst an SGI will only
issue between 2 and 4. May or may not be relevant.
This information has also been passed back to SGI who are looking at it.
Packet traces and more detail is available if anyone wants them.
Any ideas or suggestions would be welcome.
******************************************************************
Iain Irwin-Powell (AKA Iain Powell)
Senior Systems Administrator
Cinesite Europe Limited
9 Carlisle Street
London
W1D 3BP
T: +442079734000
DDI: +442079734053
*******************************************************************
It's not broken, it just doesn't work the way you expected.
*******************************************************************
-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a
relational database is painful, don't do it! Check out ObjectStore.
Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Managed to get some quiet time to test the vm.max-readahead tweaking.
This definitely makes a difference to the load average on the SGI server
end. On this test I was only opening 1 file at a time.
Tuning vm.max-readahead down to 8 provided the best combination of
server load to client throughput, although the client throughput was
compromised by the down tuning (no real surprise there).
What I am waiting to get is a 'real world' situation so that I can tune
whilst I am seeing the problem to see if we can make it go away or at
least alleviate the symptoms.
The implication here (to me at least) is that the SGI is having problems
servicing all the requests from the Linux clients. I will feed this back
to SGI and see what they can come up with.
More when I know.
Iain
******************************************************************
Iain Irwin-Powell (AKA Iain Powell)
Senior Systems Administrator
Cinesite Europe Limited
9 Carlisle Street
London
W1D 3BP
T: +442079734000
DDI: +442079734053
*******************************************************************
It's not broken, it just doesn't work the way you expected.
*******************************************************************
> -----Original Message-----
> From: Danny Smith [mailto:[email protected]]
> Sent: 23 May 2003 13:12
> To: [email protected]
> Cc: Lever, Charles; Iain Irwin-Powell; [email protected]
> Subject: Re: [NFS] NFS Performance Between SGI Servers and Linux
Clients
>
> Trond Myklebust wrote:
>
> >>>>>>" " == Charles Lever <Lever> writes:
> >>>>>>
> >>>>>>
> >
> > > trond, isn't there a read ahead value in the client that can
be
> > > tweaked? he could trim down vm_max_readahead, unless there is
> > > a max_readahead[] entry for anonymous file systems.
> >
> >That should partly help to trim it down unless the client is reading
> >from > 1 file at a time. In that case the RPC layer will still try to
> >issue more reads (up to MIN(16,vm_max_readahead) requests per
process)
> >if the network and server permits it.
> >
> >Beware, though, that vm_max_readahead will effect not only
> >NFS. Performance on other tasks may suffer.
> >
> >
> I will give this a try in Iain's absence. In the real world we *are*
> likely
> to be reading multiple files, however for testing purposes we don't
have
> to.
>
> Danny
>
> --
> Danny Smith
> Senior Systems Administrator, Cinesite (Europe) Ltd
> 020 7973 4000 - x4055 / [email protected]
>
>
-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a
relational database is painful, don't do it! Check out ObjectStore.
Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
>>>>> " " == Iain Irwin-Powell <[email protected]> writes:
> Looking a bit deeper in a packet trace when the grep is
> running, Linux seems to issue 16 read requests (16KB) at a time
> whilst an SGI will only issue between 2 and 4. May or may not
> be relevant.
... and this is a problem because .... ???
The Linux client tries to read ahead as much as it can and as quickly
as it can. Why should it doing anything else?
Cheers,
Trond
-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a
relational database is painful, don't do it! Check out ObjectStore.
Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
> -----Original Message-----
> From: Trond Myklebust [mailto:[email protected]]
> Sent: Thursday, May 22, 2003 5:13 PM
> To: Iain Irwin-Powell
> Cc: [email protected]
> Subject: Re: [NFS] NFS Performance Between SGI Servers and=20
> Linux Clients
>=20
>=20
> >>>>> " " =3D=3D Iain Irwin-Powell <[email protected]> writes:
>=20
> > Looking a bit deeper in a packet trace when the grep is
> > running, Linux seems to issue 16 read requests (16KB) at a time
> > whilst an SGI will only issue between 2 and 4. May or may not
> > be relevant.
>=20
> ... and this is a problem because .... ???
>=20
> The Linux client tries to read ahead as much as it can and as quickly
> as it can. Why should it doing anything else?
well, this is one reason why NFS O_DIRECT is a desirable
feature. no readahead. certain applications don't need
it because the workload they generate is entirely random
small reads. in these cases, read ahead wastes I/O
bandwidth and memory space with data that is never used
by an application.
trond, isn't there a read ahead value in the client that
can be tweaked? he could trim down vm_max_readahead,=20
unless there is a max_readahead[] entry for anonymous
file systems.
either that will help iain, or prove that the problem
is not a result of read ahead.
-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a
relational database is painful, don't do it! Check out ObjectStore.
Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
>>>>> " " == Charles Lever <Lever> writes:
> trond, isn't there a read ahead value in the client that can be
> tweaked? he could trim down vm_max_readahead, unless there is
> a max_readahead[] entry for anonymous file systems.
That should partly help to trim it down unless the client is reading
from > 1 file at a time. In that case the RPC layer will still try to
issue more reads (up to MIN(16,vm_max_readahead) requests per process)
if the network and server permits it.
Beware, though, that vm_max_readahead will effect not only
NFS. Performance on other tasks may suffer.
Cheers,
Trond
-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a
relational database is painful, don't do it! Check out ObjectStore.
Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Iain,
6.5.19 (and 6.5.17/18/20 for that matter) have a number of known
fundamental problems [we've seen some nasty client side problems,
though I've obviously not been involved in server side issues]
NFS patch (5021 or 5022 [don't ask me which one is for 6.5.19,=20
I'm not *that* into irix!] contains a number of important fixes.
What you're describing is possibly, or is closely related to the
buf_relse_age bug 886112. you might try the workaround online:
sgi-hd 2# systune buf_relse_age 10
buf_relse_age =3D 0 (0x0)
Do you really want to change buf_relse_age to 10 (0xa)? (y/n) y
sgi-hd 3# systune buf_relse_age
buf_relse_age =3D 10 (0xa)
- Mike
-----Original Message-----
From: Iain Irwin-Powell [mailto:[email protected]]
Sent: 22 May 2003 18:32
To: [email protected]
Subject: [NFS] NFS Performance Between SGI Servers and Linux Clients
Does anyone have any ideas about this?
We have been experiencing a number of issues regarding NFS serving on
SGI IRIX (6.5.19) servers which now appear to point back to Linux NFS
clients.
The server is an Origin 2000, quad processors, 1.5GB memory, serving
over Gigabit. This can be reproduced on connections whether or not they
are using jumbo frames. The disks that are served are hardware RAID3
capable of around 80-90MB/s (tested and optimised not theoretical).=20
Under certain circumstances will get a load average of 8 (or how ever
many nfsd's are running). The server itself is not that busy. Most of
the nfsd's show as sleeping but they are obviously waiting for
something.
Further investigation revealed that this situation arises when a Linux
client opens many files in quick succession (doing
open-read-close,open-read-close). We tend to use files in the range
3MB-13MB in size generally.
If we do the same test on an SGI client there is very little impact on
the load average.
We can recreate this by doing (on the client);
find . -type -f -print -exec grep jsdhgkhjfdk {} \;
The same effect appears to happen on Linux servers as well. A single
Linux client doing this will make the load average creep up towards 2.5.
Linux Version 2.4.20
nfs-utils-1.0.1-1
exports are exported with -32bitclients
Looking a bit deeper in a packet trace when the grep is running, Linux
seems to issue 16 read requests (16KB) at a time whilst an SGI will only
issue between 2 and 4. May or may not be relevant.
This information has also been passed back to SGI who are looking at it.
Packet traces and more detail is available if anyone wants them.
Any ideas or suggestions would be welcome.
******************************************************************
Iain Irwin-Powell (AKA Iain Powell)
Senior Systems Administrator
Cinesite Europe Limited
9 Carlisle Street
London
W1D 3BP
=20
T: +442079734000
DDI: +442079734053
*******************************************************************
It's not broken, it just doesn't work the way you expected.
*******************************************************************
-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a
relational database is painful, don't do it! Check out ObjectStore.
Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a
relational database is painful, don't do it! Check out ObjectStore.
Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
People,
We have patch 5067, which appears from the release notes to obsolete 5022.
We've also got the buf_relse_age systune in place, and haven't seen an
appreciable difference.
Not sure if this is relevant here or not, but we also get infrequent
messages like the one below:
May 23 10:50:57 animal16 kernel: nfs_stat_to_errno: bad nfs status
return value: 11
My research in the list archives seems to indicate this is a pure server
issue.
Can anyone explain it in more detail? It would help us greatly in
feeding this problem
back to SGI.
Any help appreciated.
Danny
Kiernan, Michael wrote:
>Iain,
>
>6.5.19 (and 6.5.17/18/20 for that matter) have a number of known
>fundamental problems [we've seen some nasty client side problems,
>though I've obviously not been involved in server side issues]
>
>NFS patch (5021 or 5022 [don't ask me which one is for 6.5.19,
>I'm not *that* into irix!] contains a number of important fixes.
>What you're describing is possibly, or is closely related to the
>buf_relse_age bug 886112. you might try the workaround online:
>sgi-hd 2# systune buf_relse_age 10
> buf_relse_age = 0 (0x0)
> Do you really want to change buf_relse_age to 10 (0xa)? (y/n) y
>sgi-hd 3# systune buf_relse_age
> buf_relse_age = 10 (0xa)
>
>- Mike
>
>-----Original Message-----
>From: Iain Irwin-Powell [mailto:[email protected]]
>Sent: 22 May 2003 18:32
>To: [email protected]
>Subject: [NFS] NFS Performance Between SGI Servers and Linux Clients
>
>
>
>Does anyone have any ideas about this?
>
>We have been experiencing a number of issues regarding NFS serving on
>SGI IRIX (6.5.19) servers which now appear to point back to Linux NFS
>clients.
>
>The server is an Origin 2000, quad processors, 1.5GB memory, serving
>over Gigabit. This can be reproduced on connections whether or not they
>are using jumbo frames. The disks that are served are hardware RAID3
>capable of around 80-90MB/s (tested and optimised not theoretical).
>
>Under certain circumstances will get a load average of 8 (or how ever
>many nfsd's are running). The server itself is not that busy. Most of
>the nfsd's show as sleeping but they are obviously waiting for
>something.
>
>Further investigation revealed that this situation arises when a Linux
>client opens many files in quick succession (doing
>open-read-close,open-read-close). We tend to use files in the range
>3MB-13MB in size generally.
>
>If we do the same test on an SGI client there is very little impact on
>the load average.
>
>We can recreate this by doing (on the client);
>
>find . -type -f -print -exec grep jsdhgkhjfdk {} \;
>
>The same effect appears to happen on Linux servers as well. A single
>Linux client doing this will make the load average creep up towards 2.5.
>
>Linux Version 2.4.20
>nfs-utils-1.0.1-1
>exports are exported with -32bitclients
>
>Looking a bit deeper in a packet trace when the grep is running, Linux
>seems to issue 16 read requests (16KB) at a time whilst an SGI will only
>issue between 2 and 4. May or may not be relevant.
>
>This information has also been passed back to SGI who are looking at it.
>
>Packet traces and more detail is available if anyone wants them.
>
>Any ideas or suggestions would be welcome.
>
>******************************************************************
>Iain Irwin-Powell (AKA Iain Powell)
>Senior Systems Administrator
>Cinesite Europe Limited
>9 Carlisle Street
>London
>W1D 3BP
>
>T: +442079734000
>DDI: +442079734053
>*******************************************************************
>It's not broken, it just doesn't work the way you expected.
>*******************************************************************
>
>
>
>
>-------------------------------------------------------
>This SF.net email is sponsored by: ObjectStore.
>If flattening out C++ or Java code to make your application fit in a
>relational database is painful, don't do it! Check out ObjectStore.
>Now part of Progress Software. http://www.objectstore.net/sourceforge
>_______________________________________________
>NFS maillist - [email protected]
>https://lists.sourceforge.net/lists/listinfo/nfs
>
>
>-------------------------------------------------------
>This SF.net email is sponsored by: ObjectStore.
>If flattening out C++ or Java code to make your application fit in a
>relational database is painful, don't do it! Check out ObjectStore.
>Now part of Progress Software. http://www.objectstore.net/sourceforge
>_______________________________________________
>NFS maillist - [email protected]
>https://lists.sourceforge.net/lists/listinfo/nfs
>
>
>
--
Danny Smith
Senior Systems Administrator, Cinesite (Europe) Ltd
020 7973 4000 - x4055 / [email protected]
-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a
relational database is painful, don't do it! Check out ObjectStore.
Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Trond Myklebust wrote:
>>>>>>" " == Charles Lever <Lever> writes:
>>>>>>
>>>>>>
>
> > trond, isn't there a read ahead value in the client that can be
> > tweaked? he could trim down vm_max_readahead, unless there is
> > a max_readahead[] entry for anonymous file systems.
>
>That should partly help to trim it down unless the client is reading
>from > 1 file at a time. In that case the RPC layer will still try to
>issue more reads (up to MIN(16,vm_max_readahead) requests per process)
>if the network and server permits it.
>
>Beware, though, that vm_max_readahead will effect not only
>NFS. Performance on other tasks may suffer.
>
>
I will give this a try in Iain's absence. In the real world we *are* likely
to be reading multiple files, however for testing purposes we don't have to.
Danny
--
Danny Smith
Senior Systems Administrator, Cinesite (Europe) Ltd
020 7973 4000 - x4055 / [email protected]
-------------------------------------------------------
This SF.net email is sponsored by: ObjectStore.
If flattening out C++ or Java code to make your application fit in a
relational database is painful, don't do it! Check out ObjectStore.
Now part of Progress Software. http://www.objectstore.net/sourceforge
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs