Hi list,
i'm developing an application (in C) which needs to write about
1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI.
It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7.
I buffer the data in (currently 4) 400Mb buffers and use write(2) in a
dedicated thread to write them to the raw disk (no fs).
The write(2) performance is not good enough, the writer threads take to
much time, and i ask you for ideas, howto to boost the write
performance.
Maybe mmaping the disk would work?
Cheers,
-Michael
PS. I would like to be cc'd as i usually don't read the list due to high
traffic.
El Tue, 09 Oct 2007 15:50:17 +0200
Michael Stiller <[email protected]> escribió:
> Hi list,
>
> i'm developing an application (in C) which needs to write about
> 1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI.
> It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7.
>
> I buffer the data in (currently 4) 400Mb buffers and use write(2) in a
> dedicated thread to write them to the raw disk (no fs).
>
> The write(2) performance is not good enough, the writer threads take
> to much time, and i ask you for ideas, howto to boost the write
> performance.
>
> Maybe mmaping the disk would work?
>
> Cheers,
>
> -Michael
>
> PS. I would like to be cc'd as i usually don't read the list due to
> high traffic.
>
> -
> To unsubscribe from this list: send the line "unsubscribe
> linux-kernel" in the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
Create a 1GB ram disk, write data there, and then backup into a hard
disk
--
Gustavo Chaín Dumit
Michael Stiller <[email protected]> writes:
>
> The write(2) performance is not good enough, the writer threads take to
> much time, and i ask you for ideas, howto to boost the write
> performance.
You could use an O_DIRECT write if the data is suitably aligned
and your IO sizes are big enough (O_DIRECT is usually a loss
on small IOs). It will also be synchronous, but if you do it
from a separate thread anyways that should be fine.
-Andi
On Tue, Oct 09 2007 at 16:56 +0200, Andi Kleen <[email protected]> wrote:
> Michael Stiller <[email protected]> writes:
>> The write(2) performance is not good enough, the writer threads take to
>> much time, and i ask you for ideas, howto to boost the write
>> performance.
>
> You could use an O_DIRECT write if the data is suitably aligned
> and your IO sizes are big enough (O_DIRECT is usually a loss
> on small IOs). It will also be synchronous, but if you do it
> from a separate thread anyways that should be fine.
>
> -Andi
If your target is a SCSI target you can gain up to 15% by using
sg. Search on the net for the "sg utils" package. The source code
of sg_dd and others are a grate example of how to do it.
Other wise O_DIRECT is your friend. Also look for asynchronous
I/O so you have a few pending IO buffers to keep the pipes full.
Boaz
Boaz Harrosh wrote:
[]
> If your target is a SCSI target you can gain up to 15% by using
> sg. Search on the net for the "sg utils" package. The source code
> of sg_dd and others are a grate example of how to do it.
>
> Other wise O_DIRECT is your friend. Also look for asynchronous
> I/O so you have a few pending IO buffers to keep the pipes full.
What's the advantage of sg_io over O_DIRECT to the block device?
I think sg devices are being (slowly?) obsoleted since most stuff
which has been in sg now works over normal block device, no?
At least, I haven't seen any significant difference between
sg_dd and dd with oflag=direct.
Thanks.
/mjt
On Tuesday 09 October 2007 23:50, Michael Stiller wrote:
> Hi list,
>
> i'm developing an application (in C) which needs to write about
> 1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI.
> It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7.
>
> I buffer the data in (currently 4) 400Mb buffers and use write(2) in a
> dedicated thread to write them to the raw disk (no fs).
>
> The write(2) performance is not good enough, the writer threads take to
> much time, and i ask you for ideas, howto to boost the write
> performance.
The kernel really cannot sustain 125MB/s? I assume the disk
array is capable?
Where is the bottleneck? Does it keep all disks busy, or are
the CPUs overloaded?
>
> The kernel really cannot sustain 125MB/s? I assume the disk
> array is capable?
Yes, the array should be capable, LSI controller U320, SATA disks inside
the array.
> Where is the bottleneck? Does it keep all disks busy, or are
> the CPUs overloaded?
I'm not sure where the bottleneck is. CPU Load goes up with
many pdflush processes in D state.
Thanks for the suggestions with sg and O_DIRECT so far,
will try this.
-Michael
On Tue, 2007-10-09 at 16:56 +0200, Andi Kleen wrote:
> Michael Stiller <[email protected]> writes:
> >
> > The write(2) performance is not good enough, the writer threads take to
> > much time, and i ask you for ideas, howto to boost the write
> > performance.
>
> You could use an O_DIRECT write if the data is suitably aligned
> and your IO sizes are big enough (O_DIRECT is usually a loss
> on small IOs). It will also be synchronous, but if you do it
> from a separate thread anyways that should be fine.
Thanks to all who answered and especially to Andi. Using O_DIRECT did
the trick.
Cheers,
Michael
Michael Stiller wrote:
> Hi list,
>
> i'm developing an application (in C) which needs to write about
> 1Gbit/s (125Mb/s) to a disk array attached via U320 SCSI.
> It runs on Dual Core 2 Xeons @2Ghz utilizing kernel 2.6.22.7.
People regularly report speeds higher than that on the RAID list, and I
can get that order of magnitude speed using dd with 1MB buffers to a
software RAID-0 array or cheap SATA drives. Are you using a decent
controller? Many "RAID" controllers have bandwidth limitations, buffer
size issues, etc. I had some chea "SCSI" arrays which were just SCSI
controllers in from of a bunch of cheap, slow, non-SCSI drives.
>
> I buffer the data in (currently 4) 400Mb buffers and use write(2) in a
> dedicated thread to write them to the raw disk (no fs).
I would limit the write size to a MB and see if that helps, regardless
of the buffer size. A circular queue of smaller buffers, like ptbuf, may
perform better.
>
> The write(2) performance is not good enough, the writer threads take to
> much time, and i ask you for ideas, howto to boost the write
> performance.
>
> Maybe mmaping the disk would work?
I don't think it would help, I'd really try limiting the size of the
write() calls first, assuming your hardware is adequate.
--
Bill Davidsen <[email protected]>
"We have more to fear from the bungling of the incompetent than from
the machinations of the wicked." - from Slashdot