Hello,
For one of our projects we have a test program that measures file
system performance by writing up to 1000 files simultaneously. After
installing kernel v2.6.16 we noticed that XFS performance dropped by a
factor of 5 (tests that took around 4 minutes on kernel 2.6.15 now
take around 20 minutes to complete). We then checked all kernels
starting from 2.6.16 up to 2.6.19-rc5 with the same unpleasant result.
The funny thing about all this is that we chose XFS for that
particular project specifically because it was about 5 times faster
with the tests than the other file systems. Now they all take about
the same time.
I also noticed that I/O barriers were introduced in v2.6.16 and
thought they may be the cause, but mounting the file system with
'nobarrier' doesn't seem to affect the performance in any way.
Any thoughts on the matter are appreciated.
Thanks in advance,
--
Igor A. Valcov
Igor A. Valcov wrote:
> Hello,
>
> For one of our projects we have a test program that measures file
> system performance by writing up to 1000 files simultaneously. After
> installing kernel v2.6.16 we noticed that XFS performance dropped by a
> factor of 5 (tests that took around 4 minutes on kernel 2.6.15 now
> take around 20 minutes to complete).
Ouch.
> We then checked all kernels
> starting from 2.6.16 up to 2.6.19-rc5 with the same unpleasant result.
> The funny thing about all this is that we chose XFS for that
> particular project specifically because it was about 5 times faster
> with the tests than the other file systems. Now they all take about
> the same time.
>
> I also noticed that I/O barriers were introduced in v2.6.16 and
> thought they may be the cause, but mounting the file system with
> 'nobarrier' doesn't seem to affect the performance in any way.
>
> Any thoughts on the matter are appreciated.
Can you provide the test?
-Eric
On Thu, 2006-11-09 at 20:30 +0300, Igor A. Valcov wrote:
> Hello,
>
> For one of our projects we have a test program that measures file
> system performance by writing up to 1000 files simultaneously. After
> installing kernel v2.6.16 we noticed that XFS performance dropped by a
> factor of 5 (tests that took around 4 minutes on kernel 2.6.15 now
> take around 20 minutes to complete). We then checked all kernels
> starting from 2.6.16 up to 2.6.19-rc5 with the same unpleasant result.
> The funny thing about all this is that we chose XFS for that
> particular project specifically because it was about 5 times faster
> with the tests than the other file systems. Now they all take about
> the same time.
>
> I also noticed that I/O barriers were introduced in v2.6.16 and
> thought they may be the cause, but mounting the file system with
> 'nobarrier' doesn't seem to affect the performance in any way.
>
> Any thoughts on the matter are appreciated.
I would try verifying the problem on a non ide disk just
to confirm the write barrier theory.
Also file a bug.
http://oss/sgi.com/bugzilla
include test case and hard description if possible.
>
> Thanks in advance,
--
Russell Cattelan <[email protected]>
On Thu, Nov 09, 2006 at 12:08:35PM -0600, Russell Cattelan wrote:
> On Thu, 2006-11-09 at 20:30 +0300, Igor A. Valcov wrote:
> > Hello,
> >
> > For one of our projects we have a test program that measures file
> > system performance by writing up to 1000 files simultaneously. After
> > installing kernel v2.6.16 we noticed that XFS performance dropped by a
> > factor of 5 (tests that took around 4 minutes on kernel 2.6.15 now
> > take around 20 minutes to complete). We then checked all kernels
> > starting from 2.6.16 up to 2.6.19-rc5 with the same unpleasant result.
> > The funny thing about all this is that we chose XFS for that
> > particular project specifically because it was about 5 times faster
> > with the tests than the other file systems. Now they all take about
> > the same time.
> >
> > I also noticed that I/O barriers were introduced in v2.6.16 and
> > thought they may be the cause, but mounting the file system with
> > 'nobarrier' doesn't seem to affect the performance in any way.
> >
> > Any thoughts on the matter are appreciated.
> I would try verifying the problem on a non ide disk just
> to confirm the write barrier theory.
>
> Also file a bug.
> http://oss/sgi.com/bugzilla
> include test case and hard description if possible.
and cc [email protected] on XFS bug reports ;)
Cheers,
Dave.
--
Dave Chinner
Principal Engineer
SGI Australian Software Group
Igor A. Valcov wrote:
> I also noticed that I/O barriers were introduced in v2.6.16 and
> thought they may be the cause, but mounting the file system with
> 'nobarrier' doesn't seem to affect the performance in any way.
did this happen to be a remount with nobarrier, or a fresh mount?
-Eric
>> I also noticed that I/O barriers were introduced in v2.6.16 and
>> thought they may be the cause, but mounting the file system with
>> 'nobarrier' doesn't seem to affect the performance in any way.
>
>
> did this happen to be a remount with nobarrier, or a fresh mount?
For the barrier stuff, see
http://lkml.org/lkml/2006/5/19/33
-`J'
--
Below is a simplified version of the test program, and results of
testing different kernels/filesystems/mount options. The results are a
little different from the ones described in the initial post (this
time performance decreased "only" 2 times), but the general tendency
is clearly the same.
============ 2.6.19-rc5-git2 ============
mount -t xfs -o noatime,barrier /dev/sdc1 /mnt/disc
real 16m40.516s
user 0m17.989s
sys 9m36.320s
mount -t xfs -o noatime,nobarrier /dev/sdc1 /mnt/disc
real 15m40.212s
user 0m17.549s
sys 9m29.692s
mount -t ext3 -o noatime /dev/sdc1 /mnt/disc
real 49m44.728s
user 0m27.678s
sys 14m15.689s
============ 2.6.14.6 ============
mount -t xfs -o noatime /dev/sdc1 /mnt/disc
real 9m58.974s
user 0m17.373s
sys 8m4.850s
mount -t ext3 -o noatime /dev/sdc1 /mnt/disc
real 49m7.526s
user 0m26.278s
sys 12m37.627s
========================================
#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
#define __BYTES 8192
#define __FILES 1000
char buf [__BYTES];
int main ()
{
char fname [1024];
int nFiles [__FILES];
int f, i;
/* Fill buf */
for (i = 0; i < __BYTES; i++)
buf [i] = i % 128;
/* Create and open files */
for (f = 0; f < __FILES; f++) {
sprintf (fname, "/mnt/disc/storage/file-%d", f);
nFiles [f] = open (fname, O_WRONLY | O_CREAT | O_TRUNC, 0644);
}
for (i = 0; i < 262144; i++) {
/* Write data to a big file */
write (nFiles [0], buf, __BYTES);
/* Write data to small files */
for (f = 1; f < __FILES; f++)
write (nFiles [f], &f, sizeof (f));
}
for (f = 0; f < __FILES; f++) {
fsync (nFiles [f]);
close (nFiles [f]);
}
return 0;
}
========================================
--
Igor A. Valcov
Is a same test on ext4dev
============ 2.6.19-rc5-git2 ============
mount -t ext4dev -o extents /dev/sdc1 /mnt/disc
real 41m55.445s
user 0m24.242s
sys 14m20.302s
--
Igor A. Valcov
Igor A. Valcov wrote:
> Below is a simplified version of the test program, and results of
> testing different kernels/filesystems/mount options. The results are a
> little different from the ones described in the initial post (this
> time performance decreased "only" 2 times), but the general tendency
> is clearly the same.
I imagine that I know the answer, but to be sure you might put some time
checks into your test app to see -which- portion of the test is taking
the bulk of the time.
-Eric
On Fri, 10 Nov 2006 16:16:27 +0300
"Igor A. Valcov" <[email protected]> wrote:
> Below is a simplified version of the test program,
Boy, I hope not. The results of this test program are of very little interest.
> for (i = 0; i < 262144; i++) {
> /* Write data to a big file */
> write (nFiles [0], buf, __BYTES);
>
> /* Write data to small files */
> for (f = 1; f < __FILES; f++)
> write (nFiles [f], &f, sizeof (f));
> }
This sits in a loop doing write(fd, buf, 4). This is wildly inefficient -
you'd get a 10x throughput benefit and maybe 100x reduction in CPU cost
simply by switching to fwrite().
I suspect something went wrong here.
>> for (i = 0; i < 262144; i++) {
>> /* Write data to a big file */
>> write (nFiles [0], buf, __BYTES);
>>
>> /* Write data to small files */
>> for (f = 1; f < __FILES; f++)
>> write (nFiles [f], &f, sizeof (f));
>> }
>
>This sits in a loop doing write(fd, buf, 4). This is wildly inefficient -
>you'd get a 10x throughput benefit and maybe 100x reduction in CPU cost
>simply by switching to fwrite().
Well yes and no. The problem here is the syscall overhead. fwrite
buffers things, so needless syscalls are avoided.
The same could be done by changing the program logic and increasing the
size argument to read/write.
>I suspect something went wrong here.
Design error. :)
-`J'
--
On Thu, Nov 09, 2006 at 08:30:23PM +0300, Igor A. Valcov wrote:
> I also noticed that I/O barriers were introduced in v2.6.16 and
> thought they may be the cause, but mounting the file system with
> 'nobarrier' doesn't seem to affect the performance in any way.
I don't know if it's related, but i played with a few fs'es a few months ago
and both xfs and reiser4 were very slow. iirc, they both took about 45min
longer than reiserfs and jfs to copy 70 or 80gigs of files from another
drive.
I suspected delayed allocation was the culprit (both r4 and xfs use DA).
Never really looked into it much, however.
--
Tom Vier <[email protected]>
DSA Key ID 0x15741ECE