Hi All
I have really slow working nfs.
Server is build on:
- dual pentium Xeon 2GHz,
- 2 3ware 7500 controllers,
- raid 5 on 7 160GB discs on controler 1
- raid 5 on 6 160GB disks on cotroller 2
Serwer is used for ~500k maildirs,
all clients (~20) are freebsd boxes, when nfs transfer is around 1MB/s for writing and 1.5MB/s for reading
listing user directories even with very few file stakes up to 15 seconds.
I tried nfs-ALL patch, but it didn' t help.
I run 256 nfs daemons,
echo 2097152 > /proc/sys/net/core/rmem_max
echo 2097152 > /proc/sys/net/core/rmem_default
nsfstat output:
Server rpc stats:
calls badcalls badauth badclnt xdrcall
25271043 17885061 0 17885061 0
Server nfs v3:
null getattr setattr lookup access readlink
25 0% 1159 0% 89718 0% 11702395 46% 10953437 43% 34868 0%
read write create mkdir symlink mknod
925727 3% 226250 0% 44964 0% 1089 0% 2299 0% 0 0%
remove rmdir rename link readdir readdirplus
35040 0% 42 0% 121064 0% 535 0% 534720 2% 0 0%
fsstat fsinfo pathconf commit
585179 2% 36 0% 0 0% 12493 0%
mount over tcp gives frequent errors on freebsd clients side:
kernel: nfs send error 32 from nfs server xxx.xxx.xxx.xxx:/export
thanks for any help,
rafal
On Tue, Nov 05, 2002 at 07:03:43PM +0100, poczta.dotcom.pl wrote:
> Hi All
>=20
> I have really slow working nfs.
> Server is build on:
> - dual pentium Xeon 2GHz,
> - 2 3ware 7500 controllers,
> - raid 5 on 7 160GB discs on controler 1=20
> - raid 5 on 6 160GB disks on cotroller 2=20
>=20
> Serwer is used for ~500k maildirs,=20
> all clients (~20) are freebsd boxes, when nfs transfer is around 1MB/s =
for writing and 1.5MB/s for reading=20
> listing user directories even with very few file stakes up to 15 second=
s.
>=20
> I tried nfs-ALL patch, but it didn' t help.
How is local performance?
What filesystem do you use?
What's the output from "iostat -x -d -k 60 3"?
--=20
Ragnar Kj=F8rstad
Big Storage
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
----- Original Message -----
From: "Ragnar Kj=F8rstad" <[email protected]>
To: "poczta.dotcom.pl" <[email protected]>
Cc: <[email protected]>
Sent: Tuesday, November 05, 2002 8:17 PM
Subject: Re: [NFS] nfs performance problem
> On Tue, Nov 05, 2002 at 07:03:43PM +0100, poczta.dotcom.pl wrote:
> > Hi All
> >
> > I have really slow working nfs.
> > Server is build on:
> > - dual pentium Xeon 2GHz,
> > - 2 3ware 7500 controllers,
> > - raid 5 on 7 160GB discs on controler 1
> > - raid 5 on 6 160GB disks on cotroller 2
> >
> > Serwer is used for ~500k maildirs,
> > all clients (~20) are freebsd boxes, when nfs transfer is around 1MB/=
s
for writing and 1.5MB/s for reading
> > listing user directories even with very few file stakes up to 15
seconds.
> >
> > I tried nfs-ALL patch, but it didn' t help.
>
> How is local performance?
local read/writes are ok
> What filesystem do you use?
reiserfs on top of lvm (to be able to get snapshots)
> What's the output from "iostat -x -d -k 60 3"?
>
below is output from 'iostat -d 60 3' (- my iostat does not take '-k'
option,
and '-d' gives me empty output) - mayby this is enough?
Linux 2.4.19 (hostname) 11/05/02
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev8-0 16.16 205.52 82.84 3481842 1403528
dev8-1 160.58 1266.39 1346.12 21455002 22805744
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev8-0 9.23 134.93 16.13 8096 968
dev8-1 150.92 753.20 1715.87 45192 102952
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev8-0 10.02 135.07 26.53 8104 1592
dev8-1 152.78 763.33 1767.20 45800 106032
rafal
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
I have 3ware cards as well (2 7500 Series 8 drive), just like yours. ALso
with 120 GIG drives (7200)
RPM. Whever I get 150-200 tps in iostat then my NFS runs SO SLOW. I also
only get 1 MB, to 1.5 MB
over NFS. The local speed seems to be ok, not great though. MY OPINION is
that this is because of seek time on the raid
array. I am serving very small files, just like you are. I am requesting
about 200-300 files per second from
each NFS server. So even though our throughput of only 1.5 MB isn't high.
The number of files per second is
actually quite high, and causes things to slow down because of seek time
issues. PLEASE GIVE US CACHEFS SOMEONE??
Does anyone have experience with IDE Raid arrays that get over 250 tps in
iostat that work fine? I would
be VERY VERY VERY interested to find out.
L8r...
Matt
> On Tue, Nov 05, 2002 at 07:03:43PM +0100, poczta.dotcom.pl wrote:
> > Hi All
> >
> > I have really slow working nfs.
> > Server is build on:
> > - dual pentium Xeon 2GHz,
> > - 2 3ware 7500 controllers,
> > - raid 5 on 7 160GB discs on controler 1
> > - raid 5 on 6 160GB disks on cotroller 2
> >
> > Serwer is used for ~500k maildirs,
> > all clients (~20) are freebsd boxes, when nfs transfer is around 1MB/s
for writing and 1.5MB/s for reading
> > listing user directories even with very few file stakes up to 15
seconds.
> >
> > I tried nfs-ALL patch, but it didn' t help.
>
> How is local performance?
local read/writes are ok
> What filesystem do you use?
reiserfs on top of lvm (to be able to get snapshots)
> What's the output from "iostat -x -d -k 60 3"?
>
below is output from 'iostat -d 60 3' (- my iostat does not take '-k'
option,
and '-d' gives me empty output) - mayby this is enough?
Linux 2.4.19 (hostname) 11/05/02
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev8-0 16.16 205.52 82.84 3481842 1403528
dev8-1 160.58 1266.39 1346.12 21455002 22805744
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev8-0 9.23 134.93 16.13 8096 968
dev8-1 150.92 753.20 1715.87 45192 102952
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev8-0 10.02 135.07 26.53 8104 1592
dev8-1 152.78 763.33 1767.20 45800 106032
rafal
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Tue, Nov 05, 2002 at 01:22:25PM -0700, Matt Heaton wrote:
> each NFS server. So even though our throughput of only 1.5 MB isn't high.
> The number of files per second is
> actually quite high, and causes things to slow down because of seek time
> issues. PLEASE GIVE US CACHEFS SOMEONE??
How is cachefs going to help? The kernel is already trying to cache data
as much as possible. Once you're trying to serve more data than you have
RAM, this are naturally going to degreate quite significantly as the system
becomes seek bound.
> Does anyone have experience with IDE Raid arrays that get over 250 tps in
> iostat that work fine? I would
> be VERY VERY VERY interested to find out.
Use raid1+0 and you'll be much happier, as read requests will be balanced
over multiple drives (mirroring means the same data can be read from all
of the mirrors). Additionally, you'll have much lower CPU utilization
and writes won't cause all disks in the array to seek for strip updates.
Read the archives for the past couple of weeks for another example of the
performance increase when going from raid5 to raid1+0.
-ben
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Cachefs will help quite a lot in my opinion because it doesn't just store
the files in RAM,
it uses the hard drive. So if you have an NFS client with an extra 5 gig
that you can
designate as cache then reads to the NFS server will go down DRAMATICALLY as
it will hit local cache on the NFS clients drive.
I agree raid 1+0 should be much faster for writes and a little for read, but
RAID 5 still
reads from all drives simultaneously (Has to read parity in too I know), but
can read
all 7 drives at once instead of only 4 drives at once in a raid 1+0
configuration with 8 drives
in the array. I have never used 1+0 so I am only talking about physical
drive layout rather
than any personal experience. Are my assumptions correct that raid 5 does
in fact read
from all drives at the same time? If so, reading might be a LITTLE faster
on raid 1+0 than
raid 5, but it shouldn't be HUGE. When I contacted 3ware, they basically
said the same thing.
I do agree that writes are MUCH faster on 1+0 than raid 5.
Any thoughts?
L8r...
Matt
> On Tue, Nov 05, 2002 at 01:22:25PM -0700, Matt Heaton wrote:
> > each NFS server. So even though our throughput of only 1.5 MB isn't
high.
> > The number of files per second is
> > actually quite high, and causes things to slow down because of seek time
> > issues. PLEASE GIVE US CACHEFS SOMEONE??
>
> How is cachefs going to help? The kernel is already trying to cache data
> as much as possible. Once you're trying to serve more data than you have
> RAM, this are naturally going to degreate quite significantly as the
system
> becomes seek bound.
>
> > Does anyone have experience with IDE Raid arrays that get over 250 tps
in
> > iostat that work fine? I would
> > be VERY VERY VERY interested to find out.
>
> Use raid1+0 and you'll be much happier, as read requests will be balanced
> over multiple drives (mirroring means the same data can be read from all
> of the mirrors). Additionally, you'll have much lower CPU utilization
> and writes won't cause all disks in the array to seek for strip updates.
> Read the archives for the past couple of weeks for another example of the
> performance increase when going from raid5 to raid1+0.
>
> -ben
>
>
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
> Cachefs will help quite a lot in my opinion because it
> doesn't just store
> the files in RAM,
> it uses the hard drive. So if you have an NFS client with an
> extra 5 gig
> that you can
> designate as cache then reads to the NFS server will go down
> DRAMATICALLY as
> it will hit local cache on the NFS clients drive.
yes, that's true, until you consider that without something
like v4 delegation, the client still has to contact the server
to keep its cache up to date.
matt, you really do want a faster server in this case.
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
You're wrong. Reads (not sequential, but scattered) on a 1+0 setup will be
faster as the raid1 driver optimizes requests somewhat, plus 2 (or more for
larger raid1s of disks) drives will be able to service requests for the same
stripe offset on different disks. There is no way a raid5 can service two
requests for the same stripe offset at different offsets in the array. And
even for a read-heavy workload, there are still writes to update metadata
(and journal unless you've got a separate journal device), and the impact
of those is that *all* reads to the array have to suffer from seeks when
even the smallest write is active.
On a side note, make sure you have the filesystem mounted with the noatime
flag if you can afford losing atimes.
-ben
On Tue, Nov 05, 2002 at 01:46:04PM -0700, Matt Heaton wrote:
> Cachefs will help quite a lot in my opinion because it doesn't just store
> the files in RAM,
> it uses the hard drive. So if you have an NFS client with an extra 5 gig
> that you can
> designate as cache then reads to the NFS server will go down DRAMATICALLY as
> it will hit local cache on the NFS clients drive.
>
> I agree raid 1+0 should be much faster for writes and a little for read, but
> RAID 5 still
> reads from all drives simultaneously (Has to read parity in too I know), but
> can read
> all 7 drives at once instead of only 4 drives at once in a raid 1+0
> configuration with 8 drives
> in the array. I have never used 1+0 so I am only talking about physical
> drive layout rather
> than any personal experience. Are my assumptions correct that raid 5 does
> in fact read
> from all drives at the same time? If so, reading might be a LITTLE faster
> on raid 1+0 than
> raid 5, but it shouldn't be HUGE. When I contacted 3ware, they basically
> said the same thing.
> I do agree that writes are MUCH faster on 1+0 than raid 5.
>
> Any thoughts?
>
> L8r...
>
> Matt
>
>
> > On Tue, Nov 05, 2002 at 01:22:25PM -0700, Matt Heaton wrote:
> > > each NFS server. So even though our throughput of only 1.5 MB isn't
> high.
> > > The number of files per second is
> > > actually quite high, and causes things to slow down because of seek time
> > > issues. PLEASE GIVE US CACHEFS SOMEONE??
> >
> > How is cachefs going to help? The kernel is already trying to cache data
> > as much as possible. Once you're trying to serve more data than you have
> > RAM, this are naturally going to degreate quite significantly as the
> system
> > becomes seek bound.
> >
> > > Does anyone have experience with IDE Raid arrays that get over 250 tps
> in
> > > iostat that work fine? I would
> > > be VERY VERY VERY interested to find out.
> >
> > Use raid1+0 and you'll be much happier, as read requests will be balanced
> > over multiple drives (mirroring means the same data can be read from all
> > of the mirrors). Additionally, you'll have much lower CPU utilization
> > and writes won't cause all disks in the array to seek for strip updates.
> > Read the archives for the past couple of weeks for another example of the
> > performance increase when going from raid5 to raid1+0.
> >
> > -ben
> >
> >
>
--
"Do you seek knowledge in time travel?"
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
> On a side note, make sure you have the filesystem mounted
> with the noatime flag if you can afford losing atimes.
to be scrupulously clear, that's on the server side, not on
the client side. the NFS client ignores the "noatime" mount
option, since file access timestamps are managed on the server.
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Tue, Nov 05, 2002 at 08:55:41PM +0100, poczta.dotcom.pl wrote:
> > How is local performance?
>=20
> local read/writes are ok
>=20
> > What filesystem do you use?
>=20
> reiserfs on top of lvm (to be able to get snapshots)
What reiserfs-version? 3.5 or 3.6?=20
What on-disk format? 3.5 or 3.6?
There should be an entry in the kernel-log from when the filesystem is
mounted.=20
Reiserfs-3.5 has some known performance-problems related to NFS - that
may be your problem.
> > What's the output from "iostat -x -d -k 60 3"?
> >
> below is output from 'iostat -d 60 3' (- my iostat does not take '-k'
> option,
> and '-d' gives me empty output) - mayby this is enough?
And not -x either?
iostat fr=E5 sysstat-4.0.3-2 (from RedHat) includes a "-x" option for
extended output. Most importantly, it will tell how much of the time
each device is busy.
It's unclear from your posts if this is a io-related problem or not.
Unnless you're running reiserfs-3.5 my guess is that it is the
IO-performance that is the problem. 3ware 7500 controllers have rather=20
poor performance on RAID5 - especially for writes.=20
If that's correct, then switching to a different RAID-level or replacing
the 3ware-controller should solve the problem. SCSI- or FC- RAIDS are
easily 10 or 20 times faster than the 3ware RAID for some types of
operations, and if you want something cheaper there is also different
types of IDE-RAIDs.
--=20
Ragnar Kj=F8rstad
Big Storage
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
----- Original Message -----
From: "Ragnar Kj=F8rstad" <[email protected]>
To: "poczta.dotcom.pl" <[email protected]>
Cc: <[email protected]>
Sent: Wednesday, November 06, 2002 12:32 AM
Subject: Re: [NFS] nfs performance problem
> On Tue, Nov 05, 2002 at 08:55:41PM +0100, poczta.dotcom.pl wrote:
> > > How is local performance?
> >
> > local read/writes are ok
> >
> > > What filesystem do you use?
> >
> > reiserfs on top of lvm (to be able to get snapshots)
>
> What reiserfs-version? 3.5 or 3.6?
> What on-disk format? 3.5 or 3.6?
3.6.25
>
> > > What's the output from "iostat -x -d -k 60 3"?
> > >
> > below is output from 'iostat -d 60 3' (- my iostat does not take '-k=
'
> > option,
> > and '-d' gives me empty output) - mayby this is enough?
>
> And not -x either?
now I have 4.0.6,
iostat -x -k 60 3 - outputs only column titles every 60 seconds but lin=
e
where numbers should be is blank,
the same I have on other machine (fresh gentoo linux)
> iostat fr=E5 sysstat-4.0.3-2 (from RedHat) includes a "-x" option for
> extended output. Most importantly, it will tell how much of the time
> each device is busy.
anyway local writes are ok
>
> It's unclear from your posts if this is a io-related problem or not.
> Unnless you're running reiserfs-3.5 my guess is that it is the
> IO-performance that is the problem. 3ware 7500 controllers have rather
> poor performance on RAID5 - especially for writes.
reiserfs 3.6.25, raid 5,
ok, I can understand raid 5 is not fast but getting below 2Mbytes/s
is really poor :-(
>
> If that's correct, then switching to a different RAID-level or replacin=
g
> the 3ware-controller should solve the problem. SCSI- or FC- RAIDS are
> easily 10 or 20 times faster than the 3ware RAID for some types of
> operations, and if you want something cheaper there is also different
> types of IDE-RAIDs.
>
what kind of IDE-RAID would You suggest?
thanks
rafal mycielski
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Wed, Nov 06, 2002 at 09:59:01AM +0100, myciel wrote:
> > > > What filesystem do you use?
> > >
> > > reiserfs on top of lvm (to be able to get snapshots)
> >
> > What reiserfs-version? 3.5 or 3.6?
> > What on-disk format? 3.5 or 3.6?
>=20
> 3.6.25
That's the reiserfs-version.
The on-disk format must be either "3.5" or "3.6".
There should be a message at mount-time telling you wich one, or you can
use "debugreiserfs <device>" to check it.
> > Unnless you're running reiserfs-3.5 my guess is that it is the
> > IO-performance that is the problem. 3ware 7500 controllers have rathe=
r
> > poor performance on RAID5 - especially for writes.
>=20
> reiserfs 3.6.25, raid 5,
> ok, I can understand raid 5 is not fast but getting below 2Mbytes/s
> is really poor :-(
Well, updating a single byte is a very very expensive operation on
raid5. I agree that 2MB/s is worse than one should expect though, so
there may be something else going on.
The fact that you say local writes are faster could indicate that the
problem is not only io-related.=20
Maybe there is some packet-loss? That kills performance on nfs.
Is there anything in the kernel-log on the clients to indicate the
problem?
> > If that's correct, then switching to a different RAID-level or replac=
ing
> > the 3ware-controller should solve the problem. SCSI- or FC- RAIDS are
> > easily 10 or 20 times faster than the 3ware RAID for some types of
> > operations, and if you want something cheaper there is also different
> > types of IDE-RAIDs.
>=20
> what kind of IDE-RAID would You suggest?
A BigStorage IDE-RAID of course :)
I'll get back to you off-list about that.
But it's still not clear if your problems are raid-related or
network-related. Possible a combination.
--=20
Ragnar Kj=F8rstad
Big Storage
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
> On Wed, Nov 06, 2002 at 09:59:01AM +0100, myciel wrote:
> > > > > What filesystem do you use?
> > > >
> > > > reiserfs on top of lvm (to be able to get snapshots)
> > >
> > > What reiserfs-version? 3.5 or 3.6?
> > > What on-disk format? 3.5 or 3.6?
> >
> > 3.6.25
>
> That's the reiserfs-version.
> The on-disk format must be either "3.5" or "3.6".
> There should be a message at mount-time telling you wich one, or you can
> use "debugreiserfs <device>" to check it.
>
format 3.6 with standard journal
> > > Unnless you're running reiserfs-3.5 my guess is that it is the
> > > IO-performance that is the problem. 3ware 7500 controllers have rather
> > > poor performance on RAID5 - especially for writes.
> >
> > reiserfs 3.6.25, raid 5,
> > ok, I can understand raid 5 is not fast but getting below 2Mbytes/s
> > is really poor :-(
>
> Well, updating a single byte is a very very expensive operation on
> raid5. I agree that 2MB/s is worse than one should expect though, so
> there may be something else going on.
>
> The fact that you say local writes are faster could indicate that the
> problem is not only io-related.
>
> Maybe there is some packet-loss?
I don't see any packet losses - at least as I can see with ping big packets
like 16k or 32k
>That kills performance on nfs.
> Is there anything in the kernel-log on the clients to indicate the
> problem?
>
yesterday evening I switched from udp to tcp,
performace is much better but I get a lot of kernel messages in syslog:
Nov 6 12:40:45 intler kernel: RPC request reserved 272 but used 276
Nov 6 12:40:50 intler kernel: RPC request reserved 240 but used 244
Nov 6 12:40:54 intler kernel: RPC request reserved 244 but used 248
what does above mean?
> >
> > what kind of IDE-RAID would You suggest?
>
> A BigStorage IDE-RAID of course :)
> I'll get back to you off-list about that.
ok
rafal mycielski
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
> From: =?iso-8859-1?Q?Ragnar_Kj=F8rstad?= <[email protected]>
> On Tue, Nov 05, 2002 at 08:55:41PM +0100, poczta.dotcom.pl wrote:
> > > How is local performance?
> > local read/writes are ok
> > > What filesystem do you use?
> > reiserfs on top of lvm (to be able to get snapshots)
I'm just going to throw my two cents in here. It sounds like the box is
getting a lot of random IO (~500k maildirs) which means he is most likely
seeing a LOT of seek issues, and RAID 5 is probably a really bad choice.
Cheap controllers and fast RAID 5 just don't go together, there's too much
math going on (the parity bit moves, remember) Matt, you really need to
trust the folks on the list on this and try it. You'll lose some capacity
(drop from 800GB to 480GB per array), but hopefully the performance boost
will be worth it.
Now, on the plus side, you're already using LVM, so I have a better idea
than RAID 1+0 for your situation. Build six RAID 1 arrays, then use LVM to
create a single logical volume out of the group. The randomness should
ensure that if your server receives 4 simultaneous read/write requests, they
will each have their own "spindle", an there will be no performance robbing
back and forth seeking, meaning the requests get handled faster, meaning
there's less chance for the system to start thrashing with seek requests.
I usually think of each RAID volume as a "virtual spindle", capable of
handling 1 request at a time. While each individual transaction might be
slower, (no striping to speed things up) there's six lanes to handle the
requests. Server thinking versus Workstation thinking.
Hopefully you can experiment with this, If this things production already it
may be too late.
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
We have a couple of NFS servers with 3ware cards. One is a 3ware 7850 (8
Maxtor 160GB - 5400RPM) and one is a 3ware 7500-12, both configured with
RAID 5 + Hot Spare. I have all the latest NFS patches applied. If I access
an NFS mounted directory on one of these systems and copy a bunch of little
files (1-2KB), I get 150-250 tps. If I access larger files (90% > 1MB each)
I will normally see 25-60MB/s transfer with 500-700 tps, with peaks of over
1000 tps - A good indication that disk seek is to blame for the problem with
the small files.
Thanks,
-Byran
From: "Matt Heaton" <[email protected]>
To: "poczta.dotcom.pl" <[email protected]>,
<[email protected]>
Subject: Re: [NFS] nfs performance problem
Date: Tue, 5 Nov 2002 13:22:25 -0700
I have 3ware cards as well (2 7500 Series 8 drive), just like yours. ALso
with 120 GIG drives (7200)
RPM. Whever I get 150-200 tps in iostat then my NFS runs SO SLOW. I also
only get 1 MB, to 1.5 MB
over NFS. The local speed seems to be ok, not great though. MY OPINION is
that this is because of seek time on the raid
array. I am serving very small files, just like you are. I am requesting
about 200-300 files per second from
each NFS server. So even though our throughput of only 1.5 MB isn't high.
The number of files per second is
actually quite high, and causes things to slow down because of seek time
issues. PLEASE GIVE US CACHEFS SOMEONE??
Does anyone have experience with IDE Raid arrays that get over 250 tps in
iostat that work fine? I would
be VERY VERY VERY interested to find out.
L8r...
Matt
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
I didn't understand? What is a good indication of poor seek time? The high
number for TPS? I thought a high number meant the drive was handling a huge
amount of transactions per second, and not necessarily slow. I would LOVE
to
have my NFS servers push out 500 TPS on iostat. I have 10 webservers that
each ask for between 10-25 files per second from our 3ware 7850 controllers.
We get about 200 TPS on the NFS server. Much more and the clients become
VERY SLUGGISH, and I get blocked R processes in vmstat on the client.
Sometimes
20-40 blocked R processes in vmstat. As soon as I offload some of files
from a busy
NFS server to another NFS server then the clients settle down and everything
works
great again, but I only get 1-1.5MB out of each NFS server before this
happens!
I can get really high TPS out of the server LOCALLY when I do copies etc,
its just
over NFS with a huge amount of really small files that everything stinks.
L8r...
Matt
> We have a couple of NFS servers with 3ware cards. One is a 3ware 7850 (8
> Maxtor 160GB - 5400RPM) and one is a 3ware 7500-12, both configured with
> RAID 5 + Hot Spare. I have all the latest NFS patches applied. If I
access
> an NFS mounted directory on one of these systems and copy a bunch of
little
> files (1-2KB), I get 150-250 tps. If I access larger files (90% > 1MB
each)
> I will normally see 25-60MB/s transfer with 500-700 tps, with peaks of
over
> 1000 tps - A good indication that disk seek is to blame for the problem
with
> the small files.
>
> Thanks,
> -Byran
>
> From: "Matt Heaton" <[email protected]>
> To: "poczta.dotcom.pl" <[email protected]>,
> <[email protected]>
> Subject: Re: [NFS] nfs performance problem
> Date: Tue, 5 Nov 2002 13:22:25 -0700
>
> I have 3ware cards as well (2 7500 Series 8 drive), just like yours. ALso
> with 120 GIG drives (7200)
> RPM. Whever I get 150-200 tps in iostat then my NFS runs SO SLOW. I also
> only get 1 MB, to 1.5 MB
> over NFS. The local speed seems to be ok, not great though. MY OPINION
is
> that this is because of seek time on the raid
> array. I am serving very small files, just like you are. I am
requesting
> about 200-300 files per second from
> each NFS server. So even though our throughput of only 1.5 MB isn't high.
> The number of files per second is
> actually quite high, and causes things to slow down because of seek time
> issues. PLEASE GIVE US CACHEFS SOMEONE??
>
> Does anyone have experience with IDE Raid arrays that get over 250 tps in
> iostat that work fine? I would
> be VERY VERY VERY interested to find out.
>
> L8r...
>
> Matt
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by: See the NEW Palm
> Tungsten T handheld. Power & Color in a compact size!
> http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
> _______________________________________________
> NFS maillist - [email protected]
> https://lists.sourceforge.net/lists/listinfo/nfs
>
>
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Thu, Nov 07, 2002 at 09:19:44AM -0600, Baker, Byran wrote:
> We have a couple of NFS servers with 3ware cards. One is a 3ware 7850 =
(8
> Maxtor 160GB - 5400RPM) and one is a 3ware 7500-12, both configured wit=
h
> RAID 5 + Hot Spare. I have all the latest NFS patches applied. If I a=
ccess
> an NFS mounted directory on one of these systems and copy a bunch of li=
ttle
> files (1-2KB), I get 150-250 tps. If I access larger files (90% > 1MB =
each)
> I will normally see 25-60MB/s transfer with 500-700 tps, with peaks of =
over
> 1000 tps - A good indication that disk seek is to blame for the problem=
with
> the small files.
Just to clarify here.
I bet when you transfer 1-2KB files you are able to transfer 150-250
files pr second, right?=20
When you transfer larger files the files pr second rate goes down, but
the tps-rate increases because each file is split into multiple scsi
requests.=20
And yes, I would agree with your conclution that disk seek is to blame.
Are those numbers for reads or writes? I would expect to see the same
tendency, but with stronger effect, for writes than reads.
--=20
Ragnar Kj=F8rstad
Big Storage
-------------------------------------------------------
This sf.net email is sponsored by: See the NEW Palm
Tungsten T handheld. Power & Color in a compact size!
http://ads.sourceforge.net/cgi-bin/redirect.pl?palm0001en
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
>... You might consider moving it forward to 2.6.23,
> as there is a significant readahead improvement/fix there.
>...
> Fengguang Wu's changes are merged in 2.6.23,
>...Tom,
Improving out-of-order nfsd requests is a great idea. Is there a
description of Fengguang Wu's improvements or maybe a link to the
specific patches. I tried searching through git and such and not sure
if I found the right set of patches.
Dean
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems? Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> NFS maillist - [email protected]
> https://lists.sourceforge.net/lists/listinfo/nfs
>
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
* Bernd Schubert ([email protected]) [071027 01:39]:
> Hello Andreas,
>
> On Thursday 25 October 2007 15:10:29 Andreas Schuldei wrote:
> > lotta:/var/disks/sda on /var/disks/sda type nfs
> > (ro,hard,intr,proto=tcp,rsize=32k,addr=217.213.5.44) lotta:/var/disks/sdb
>
> try to increase rsize and wsize as much as possible, the maximum can be
> adjusted in /proc/fs/nfsd/max_block_size on the nfs server.
i cant increase this to more then 1M. is that the hard limit?
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
At 09:35 PM 10/26/2007, dean hildebrand wrote:
>>... You might consider moving it forward to 2.6.23,
>> as there is a significant readahead improvement/fix there.
>>...
>> Fengguang Wu's changes are merged in 2.6.23,
>>...Tom,
>
>Improving out-of-order nfsd requests is a great idea. Is there a
>description of Fengguang Wu's improvements or maybe a link to the
>specific patches. I tried searching through git and such and not sure
>if I found the right set of patches.
There are quite a few of them. A big one with lots of background info
is in mm/readahead.c, commit 122a21d11cbfda6d1e33cbc8ae9e4c4ee2f1886e
Check out the "readahead thrashing" results at the end of the log, especially
the comment "the more overall read density, the more possible gain", i.e. it's
the density and not the actual order which now drives the readahead.
Good stuff.
Tom.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs