Bart Van Assche, on 03/30/2009 10:06 PM wrote:
> On Mon, Mar 30, 2009 at 7:33 PM, Vladislav Bolkhovitin <[email protected]> wrote:
>> As part of 1.0.1 release preparations I made some performance tests to make
>> sure there are no performance regressions in SCST overall and iSCSI-SCST
>> particularly. Results were quite interesting, so I decided to publish them
>> together with the corresponding numbers for IET and STGT iSCSI targets. This
>> isn't a real performance comparison, it includes only few chosen tests,
>> because I don't have time for a complete comparison. But I hope somebody
>> will take up what I did and make it complete.
>>
>> Setup:
>>
>> Target: HT 2.4GHz Xeon, x86_32, 2GB of memory limited to 256MB by kernel
>> command line to have less test data footprint, 75GB 15K RPM SCSI disk as
>> backstorage, dual port 1Gbps E1000 Intel network card, 2.6.29 kernel.
>>
>> Initiator: 1.7GHz Xeon, x86_32, 1GB of memory limited to 256MB by kernel
>> command line to have less test data footprint, dual port 1Gbps E1000 Intel
>> network card, 2.6.27 kernel, open-iscsi 2.0-870-rc3.
>>
>> The target exported a 5GB file on XFS for FILEIO and 5GB partition for
>> BLOCKIO.
>>
>> All the tests were ran 3 times and average written. All the values are in
>> MB/s. The tests were ran with CFQ and deadline IO schedulers on the target.
>> All other parameters on both target and initiator were default.
>
> These are indeed interesting results. There are some aspects of the
> test setup I do not understand however:
> * All tests have been run with buffered I/O instead of direct I/O
> (iflag=direct / oflag=direct). My experience is that the results of
> tests with direct I/O are easier to reproduce (less variation between
> runs). So I have been wondering why the tests have been run with
> buffered I/O instead ?
Real applications use buffered I/O, hence it should be used in tests. It
evaluates all the storage stack on both initiator and target as a
whole. The results are very reproducible, variation is about 10%.
> * It is well known that having more memory in the target system
> improves performance because of read and write caching. What did you
> want to demonstrate by limiting the memory of the target system ?
If I had full 2GB on the target, I would have to spend on the
measurements 10 times more time, since the data footprint should be at
least 4x of the cache size. For sequential read/writes 256MB and 2GB of
the cache are the same.
Where it did matter (io_trash) I increased memory size to full 2GB.
> * Which SCST options were enabled on the target ? Was e.g. the
> NV_CACHE option enabled ?
Defaults, i.e. yes, enabled. But it didn't matter, since all the
filesystems where mounted on the initiator without data barriers enabled.
Thanks,
Vlad
P.S. Please don't drop CC.
On Mon, Mar 30, 2009 at 8:33 PM, Vladislav Bolkhovitin <[email protected]> wrote:
> Bart Van Assche, on 03/30/2009 10:06 PM wrote:
>> These are indeed interesting results. There are some aspects of the
>> test setup I do not understand however:
>> * All tests have been run with buffered I/O instead of direct I/O
>> (iflag=direct / oflag=direct). My experience is that the results of
>> tests with direct I/O are easier to reproduce (less variation between
>> runs). So I have been wondering why the tests have been run with
>> buffered I/O instead ?
>
> Real applications use buffered I/O, hence it should be used in tests. It
> ?evaluates all the storage stack on both initiator and target as a whole.
> The results are very reproducible, variation is about 10%.
Most applications do indeed use buffered I/O. Database software
however often uses direct I/O. It might be interesting to publish
performance results for both buffered I/O and direct I/O. A quote from
the paper "Asynchronous I/O Support in Linux 2.5" by Bhattacharya e.a.
(Linux Symposium, Ottawa, 2003):
Direct I/O (raw and O_DIRECT) transfers data between a user buffer and
a device without copying the data through the kernel?s buffer cache.
This mechanism can boost performance if the data is unlikely to be
used again in the short term (during a disk backup, for example), or
for applications such as large database management systems that
perform their own caching.
Bart.
Bart Van Assche, on 03/30/2009 10:53 PM wrote:
> On Mon, Mar 30, 2009 at 8:33 PM, Vladislav Bolkhovitin <[email protected]> wrote:
>> Bart Van Assche, on 03/30/2009 10:06 PM wrote:
>>> These are indeed interesting results. There are some aspects of the
>>> test setup I do not understand however:
>>> * All tests have been run with buffered I/O instead of direct I/O
>>> (iflag=direct / oflag=direct). My experience is that the results of
>>> tests with direct I/O are easier to reproduce (less variation between
>>> runs). So I have been wondering why the tests have been run with
>>> buffered I/O instead ?
>> Real applications use buffered I/O, hence it should be used in tests. It
>> evaluates all the storage stack on both initiator and target as a whole.
>> The results are very reproducible, variation is about 10%.
>
> Most applications do indeed use buffered I/O. Database software
> however often uses direct I/O. It might be interesting to publish
> performance results for both buffered I/O and direct I/O.
Yes, sure
> A quote from
> the paper "Asynchronous I/O Support in Linux 2.5" by Bhattacharya e.a.
> (Linux Symposium, Ottawa, 2003):
>
> Direct I/O (raw and O_DIRECT) transfers data between a user buffer and
> a device without copying the data through the kernel?s buffer cache.
> This mechanism can boost performance if the data is unlikely to be
> used again in the short term (during a disk backup, for example), or
> for applications such as large database management systems that
> perform their own caching.
Please don't misread phrase "unlikely to be used again in the short
term". If you have read-ahead, all your cached data is *likely* to be
used "again" in the near future after they were read from storage,
although only once in the first read by application. The same is true
for write-back caching, where data written to the cache once for each
command. Both read-ahead and write back are very important for good
performance and O_DIRECT throws them away. All the modern HDDs have a
memory buffer (cache) at least 2MB big on the cheapest ones. This cache
is essential for performance, although how can it make any difference if
the host computer has, say, 1000 times more memory?
Thus, to work effectively with O_DIRECT an application has to be very
smart to workaround the lack of read-ahead and write back.
I personally consider O_DIRECT (as well as BLOCKIO) as nothing more than
a workaround for possible flaws in the storage subsystem. If O_DIRECT
works better, then in 99+% cases there is something in the storage
subsystem, which should be fixed to perform better.
To be complete, there is one case where O_DIRECT and BLOCKIO have an
advantage: both of them transfer data zero-copy. So they are good if
your memory is too slow comparing to storage (InfiniBand case, for
instance) and additional data copy hurts performance noticeably.
> Bart.
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Scst-devel mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/scst-devel
>
Vladislav Bolkhovitin wrote:
> Bart Van Assche, on 03/30/2009 10:53 PM wrote:
> >
> > Most applications do indeed use buffered I/O. Database software
> > however often uses direct I/O. It might be interesting to publish
> > performance results for both buffered I/O and direct I/O.
>
> Yes, sure
>
> > A quote from
> > the paper "Asynchronous I/O Support in Linux 2.5" by Bhattacharya e.a.
> > (Linux Symposium, Ottawa, 2003):
> >
> > Direct I/O (raw and O_DIRECT) transfers data between a user buffer and
> > a device without copying the data through the kernel?s buffer cache.
> > This mechanism can boost performance if the data is unlikely to be
> > used again in the short term (during a disk backup, for example), or
> > for applications such as large database management systems that
> > perform their own caching.
>
> Please don't misread phrase "unlikely to be used again in the short
> term". If you have read-ahead, all your cached data is *likely* to be
> used "again" in the near future after they were read from storage,
> although only once in the first read by application. The same is true
> for write-back caching, where data written to the cache once for each
> command. Both read-ahead and write back are very important for good
> performance and O_DIRECT throws them away. All the modern HDDs have a
> memory buffer (cache) at least 2MB big on the cheapest ones.
> This cache is essential for performance, although how can it make any
> difference if the host computer has, say, 1000 times more memory?
>
> Thus, to work effectively with O_DIRECT an application has to be very
> smart to workaround the lack of read-ahead and write back.
True, the application has to perform it's own read-ahead and write-back.
Kind of like how a database does it, or maybe the page cache on the
iSCSI initiator's system ;-)
> I personally consider O_DIRECT (as well as BLOCKIO) as nothing more than
> a workaround for possible flaws in the storage subsystem. If O_DIRECT
> works better, then in 99+% cases there is something in the storage
> subsystem, which should be fixed to perform better.
That's not true, page cached I/O is broken into page sizes which limits
the I/O bandwidth of the storage hardware while imposing a higher CPU
overhead. Obviously page-cached I/O isn't ideal for all situations.
You could also have an amazing backend storage system with it's own
NVRAM cache. Why put the performance overhead onto the target system
when you can off-load it to the controller?
> To be complete, there is one case where O_DIRECT and BLOCKIO have an
> advantage: both of them transfer data zero-copy. So they are good if
> your memory is too slow comparing to storage (InfiniBand case, for
> instance) and additional data copy hurts performance noticeably.
The bottom line which will always be true:
Know your workload, configure your storage to match.
The best storage solutions allow the implementor the most flexibility
in configuring the storage, which I think both IET and SCST do.
IET just needs to fix how it does it workload with CFQ which
somehow SCST has overcome. Of course SCST tweaks the Linux kernel to
gain some extra speed.
Vlad, how about a comparison of SCST vs IET without those kernel hooks?
-Ross
______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.
On Tue, Mar 31, 2009 at 8:43 PM, Ross S. W. Walker
<[email protected]> wrote:
> IET just needs to fix how it does it workload with CFQ which
> somehow SCST has overcome. Of course SCST tweaks the Linux kernel to
> gain some extra speed.
I'm not familiar with the implementation details of CFQ, but I know
that one of the changes between SCST 1.0.0 and SCST 1.0.1 is that the
default number of kernel threads of the scst_vdisk kernel module has
been increased to 5. Could this explain the performance difference
between SCST and IET for FILEIO and BLOCKIO ?
Bart.
On Apr 1, 2009, at 2:29 AM, Bart Van Assche <[email protected]>
wrote:
> On Tue, Mar 31, 2009 at 8:43 PM, Ross S. W. Walker
> <[email protected]> wrote:
>> IET just needs to fix how it does it workload with CFQ which
>> somehow SCST has overcome. Of course SCST tweaks the Linux kernel to
>> gain some extra speed.
>
> I'm not familiar with the implementation details of CFQ, but I know
> that one of the changes between SCST 1.0.0 and SCST 1.0.1 is that the
> default number of kernel threads of the scst_vdisk kernel module has
> been increased to 5. Could this explain the performance difference
> between SCST and IET for FILEIO and BLOCKIO ?
Thank for the update. IET has used 8 threads per target for ages now,
I don't think it is that.
It may be how the I/O threads are forked in SCST that causes them to
be in the same I/O context with each other.
I'm pretty sure implementing a version of the patch that was used for
the dump command (found on the LKML) will fix this.
But thanks goes to Vlad for pointing this dificiency out so we can fix
it to help make IET even better.
-Ross
On Wed, 2009-04-01 at 08:20 -0400, Ross Walker wrote:
> On Apr 1, 2009, at 2:29 AM, Bart Van Assche <[email protected]>
> wrote:
>
> > On Tue, Mar 31, 2009 at 8:43 PM, Ross S. W. Walker
> > <[email protected]> wrote:
> >> IET just needs to fix how it does it workload with CFQ which
> >> somehow SCST has overcome. Of course SCST tweaks the Linux kernel to
> >> gain some extra speed.
> >
> > I'm not familiar with the implementation details of CFQ, but I know
> > that one of the changes between SCST 1.0.0 and SCST 1.0.1 is that the
> > default number of kernel threads of the scst_vdisk kernel module has
> > been increased to 5. Could this explain the performance difference
> > between SCST and IET for FILEIO and BLOCKIO ?
>
> Thank for the update. IET has used 8 threads per target for ages now,
> I don't think it is that.
>
> It may be how the I/O threads are forked in SCST that causes them to
> be in the same I/O context with each other.
>
> I'm pretty sure implementing a version of the patch that was used for
> the dump command (found on the LKML) will fix this.
>
> But thanks goes to Vlad for pointing this dificiency out so we can fix
> it to help make IET even better.
SCST explicitly fiddles with the io context to get this to happen. It
has a hack to block to export alloc_io_context:
http://marc.info/?t=122893564800003
James
James Bottomley, on 04/02/2009 12:23 AM wrote:
> On Wed, 2009-04-01 at 08:20 -0400, Ross Walker wrote:
>> On Apr 1, 2009, at 2:29 AM, Bart Van Assche <[email protected]>
>> wrote:
>>
>>> On Tue, Mar 31, 2009 at 8:43 PM, Ross S. W. Walker
>>> <[email protected]> wrote:
>>>> IET just needs to fix how it does it workload with CFQ which
>>>> somehow SCST has overcome. Of course SCST tweaks the Linux kernel to
>>>> gain some extra speed.
>>> I'm not familiar with the implementation details of CFQ, but I know
>>> that one of the changes between SCST 1.0.0 and SCST 1.0.1 is that the
>>> default number of kernel threads of the scst_vdisk kernel module has
>>> been increased to 5. Could this explain the performance difference
>>> between SCST and IET for FILEIO and BLOCKIO ?
>> Thank for the update. IET has used 8 threads per target for ages now,
>> I don't think it is that.
>>
>> It may be how the I/O threads are forked in SCST that causes them to
>> be in the same I/O context with each other.
>>
>> I'm pretty sure implementing a version of the patch that was used for
>> the dump command (found on the LKML) will fix this.
>>
>> But thanks goes to Vlad for pointing this dificiency out so we can fix
>> it to help make IET even better.
>
> SCST explicitly fiddles with the io context to get this to happen. It
> has a hack to block to export alloc_io_context:
>
> http://marc.info/?t=122893564800003
Correct, although I wouldn't call it "fiddle", rather "grouping" ;)
But that's not the only reason for good performance. Particularly, it
can't explain Bart's tmpfs results from the previous message, where the
majority of I/O done to/from RAM without any I/O scheduler involved. (Or
does I/O scheduler also involved with tmpfs?) Bart has 4GB RAM, if I
remember correctly, i.e. the test data set was 25% of RAM.
Thanks,
Vlad
Vladislav Bolkhovitin, on 04/02/2009 11:38 AM wrote:
> James Bottomley, on 04/02/2009 12:23 AM wrote:
>> On Wed, 2009-04-01 at 08:20 -0400, Ross Walker wrote:
>>> On Apr 1, 2009, at 2:29 AM, Bart Van Assche <[email protected]>
>>> wrote:
>>>
>>>> On Tue, Mar 31, 2009 at 8:43 PM, Ross S. W. Walker
>>>> <[email protected]> wrote:
>>>>> IET just needs to fix how it does it workload with CFQ which
>>>>> somehow SCST has overcome. Of course SCST tweaks the Linux kernel to
>>>>> gain some extra speed.
>>>> I'm not familiar with the implementation details of CFQ, but I know
>>>> that one of the changes between SCST 1.0.0 and SCST 1.0.1 is that the
>>>> default number of kernel threads of the scst_vdisk kernel module has
>>>> been increased to 5. Could this explain the performance difference
>>>> between SCST and IET for FILEIO and BLOCKIO ?
>>> Thank for the update. IET has used 8 threads per target for ages now,
>>> I don't think it is that.
>>>
>>> It may be how the I/O threads are forked in SCST that causes them to
>>> be in the same I/O context with each other.
>>>
>>> I'm pretty sure implementing a version of the patch that was used for
>>> the dump command (found on the LKML) will fix this.
>>>
>>> But thanks goes to Vlad for pointing this dificiency out so we can fix
>>> it to help make IET even better.
>> SCST explicitly fiddles with the io context to get this to happen. It
>> has a hack to block to export alloc_io_context:
>>
>> http://marc.info/?t=122893564800003
>
> Correct, although I wouldn't call it "fiddle", rather "grouping" ;)
>
> But that's not the only reason for good performance. Particularly, it
> can't explain Bart's tmpfs results from the previous message, where the
> majority of I/O done to/from RAM without any I/O scheduler involved. (Or
> does I/O scheduler also involved with tmpfs?) Bart has 4GB RAM, if I
> remember correctly, i.e. the test data set was 25% of RAM.
To remove any suspicions that I'm playing dirty games here I should note
that in many cases I can't say what exactly is responsible for good SCST
performance. I can say only something like "good design and
implementation", but, I guess, it wouldn't be counted too much.
SCST/iSCSI-SCST from the very beginning were designed and made with the
best performance in mind and that has brought the result. Sorry, but at
the moment I can't afford doing any "why it's so good?" kinds of
investigations, because I have a lot more important things to do, like
SCST procfs -> sysfs interface conversion.
Thanks,
Vlad
Vladislav Bolkhovitin wrote:
> Vladislav Bolkhovitin, on 04/02/2009 11:38 AM wrote:
> > James Bottomley, on 04/02/2009 12:23 AM wrote:
> >>
> >> SCST explicitly fiddles with the io context to get this to happen. It
> >> has a hack to block to export alloc_io_context:
> >>
> >> http://marc.info/?t=122893564800003
> >
> > Correct, although I wouldn't call it "fiddle", rather "grouping" ;)
Call it what you like,
Vladislav Bolkhovitin wrote:
> Ross S. W. Walker, on 03/30/2009 10:33 PM wrote:
>
> I would be interested in knowing how your code defeats CFQ's extremely
> high latency? Does your code reach into the io scheduler too? If not,
> some code hints would be great.
Hmm, CFQ doesn't have any extra processing latency, especially
"extremely", hence there is nothing to defeat. If it had, how could it
been chosen as the default?
----------
List: linux-scsi
Subject: [PATCH][RFC 13/23]: Export of alloc_io_context() function
From: Vladislav Bolkhovitin <vst () vlnb ! net>
Date: 2008-12-10 18:49:19
Message-ID: 49400F2F.4050603 () vlnb ! net
This patch exports alloc_io_context() function. For performance reasons
SCST queues commands using a pool of IO threads. It is considerably
better for performance (>30% increase on sequential reads) if threads in
a pool have the same IO context. Since SCST can be built as a module,
it needs alloc_io_context() function exported.
<snip>
----------
I call that lying.
> > But that's not the only reason for good performance. Particularly, it
> > can't explain Bart's tmpfs results from the previous message, where the
> > majority of I/O done to/from RAM without any I/O scheduler involved. (Or
> > does I/O scheduler also involved with tmpfs?) Bart has 4GB RAM, if I
> > remember correctly, i.e. the test data set was 25% of RAM.
>
> To remove any suspicions that I'm playing dirty games here I should note
<snip>
I don't know what games your playing at, but do me a favor, if your too
stupid enough to realize when your caught in a lie and to just shut up
then please do me the favor and leave me out of any further correspondence
from you.
Thank you,
-Ross
______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.
Ross S. W. Walker wrote:
> Vladislav Bolkhovitin wrote:
> > Vladislav Bolkhovitin, on 04/02/2009 11:38 AM wrote:
> > > James Bottomley, on 04/02/2009 12:23 AM wrote:
> > >>
> > >> SCST explicitly fiddles with the io context to get this to happen. It
> > >> has a hack to block to export alloc_io_context:
> > >>
> > >> http://marc.info/?t=122893564800003
> > >
> > > Correct, although I wouldn't call it "fiddle", rather "grouping" ;)
>
> Call it what you like,
>
> Vladislav Bolkhovitin wrote:
> > Ross S. W. Walker, on 03/30/2009 10:33 PM wrote:
> >
> > I would be interested in knowing how your code defeats CFQ's extremely
> > high latency? Does your code reach into the io scheduler too? If not,
> > some code hints would be great.
>
The above quoting was wrong, for accuracy, it should have read:
Vladislav Bolkhovitin wrote:
> Ross S. W. Walker, on 03/30/2009 10:33 PM wrote:
> >
> > I would be interested in knowing how your code defeats CFQ's extremely
> > high latency? Does your code reach into the io scheduler too? If not,
> > some code hints would be great.
>
> Hmm, CFQ doesn't have any extra processing latency, especially
> "extremely", hence there is nothing to defeat. If it had, how could it
> been chosen as the default?
Just so there is no misunderstanding who said what here.
-Ross
______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.
Ross S. W. Walker, on 04/02/2009 06:06 PM wrote:
> Vladislav Bolkhovitin wrote:
>> Vladislav Bolkhovitin, on 04/02/2009 11:38 AM wrote:
>>> James Bottomley, on 04/02/2009 12:23 AM wrote:
>>>> SCST explicitly fiddles with the io context to get this to happen. It
>>>> has a hack to block to export alloc_io_context:
>>>>
>>>> http://marc.info/?t=122893564800003
>>> Correct, although I wouldn't call it "fiddle", rather "grouping" ;)
>
> Call it what you like,
>
> Vladislav Bolkhovitin wrote:
>> Ross S. W. Walker, on 03/30/2009 10:33 PM wrote:
>>
>> I would be interested in knowing how your code defeats CFQ's extremely
>> high latency? Does your code reach into the io scheduler too? If not,
>> some code hints would be great.
>
> Hmm, CFQ doesn't have any extra processing latency, especially
> "extremely", hence there is nothing to defeat. If it had, how could it
> been chosen as the default?
>
> ----------
> List: linux-scsi
> Subject: [PATCH][RFC 13/23]: Export of alloc_io_context() function
> From: Vladislav Bolkhovitin <vst () vlnb ! net>
> Date: 2008-12-10 18:49:19
> Message-ID: 49400F2F.4050603 () vlnb ! net
>
> This patch exports alloc_io_context() function. For performance reasons
> SCST queues commands using a pool of IO threads. It is considerably
> better for performance (>30% increase on sequential reads) if threads in
> a pool have the same IO context. Since SCST can be built as a module,
> it needs alloc_io_context() function exported.
>
> <snip>
> ----------
>
> I call that lying.
>
>>> But that's not the only reason for good performance. Particularly, it
>>> can't explain Bart's tmpfs results from the previous message, where the
>>> majority of I/O done to/from RAM without any I/O scheduler involved. (Or
>>> does I/O scheduler also involved with tmpfs?) Bart has 4GB RAM, if I
>>> remember correctly, i.e. the test data set was 25% of RAM.
>> To remove any suspicions that I'm playing dirty games here I should note
> <snip>
>
> I don't know what games your playing at, but do me a favor, if your too
> stupid enough to realize when your caught in a lie and to just shut up
> then please do me the favor and leave me out of any further correspondence
> from you.
Think what you want and do what you want. You can even filter out all
e-mails from me, that's your right. But:
1. As I wrote grouping threads into a single IO context doesn't explain
all the performance difference and finding out reasons for other's
performance problems isn't something I can afford at the moment.
2. CFQ doesn't have any processing latency and has never had. Learn to
understand what are your writing about and how to correctly express
yourself at first. You asked about that latency and I replied that there
is nothing to defeat.
3. SCST doesn't have any hooks into CFQ and not going to have in the
considerable future.
> Thank you,
>
> -Ross
Vladislav Bolkhovitin wrote:
>
> Think what you want and do what you want. You can even filter out all
> e-mails from me, that's your right. But:
>
> 1. As I wrote grouping threads into a single IO context doesn't explain
> all the performance difference and finding out reasons for other's
> performance problems isn't something I can afford at the moment.
No, not all the performance, but a substantial part of it, enough
so to say IET has a real performance issue when using CFQ scheduler.
> 2. CFQ doesn't have any processing latency and has never had. Learn to
> understand what are your writing about and how to correctly express
> yourself at first. You asked about that latency and I replied that there
> is nothing to defeat.
CFQ pauses briefly before switching I/O contexts in order to make sure
it is giving as much bandwidth to a context before moving on. This is
documented. With a single I/O stream, or random I/O it won't be
noticeable, but for interleaved sequential I/O across multiple threads
with different I/O contexts it can be significant.
Not that Wikipedia is authorative: http://en.wikipedia.org/wiki/CFQ
It's right in the first paragraph:
"... While CFQ does not do explicit anticipatory IO scheduling, it
achieves the same effect of having good aggregate throughput for the
system as a whole, by allowing a process queue to idle at the end of
synchronous IO thereby "anticipating" further close IO from that
process. ..."
You can also check out the LXR:
This one in 2.6.18 kernels (RHEL) show a pause of HZ/10
http://lxr.linux.no/linux+v2.6.18/block/cfq-iosched.c#L30
So given a 10ms time slice, that would equate to ~1ms, in later
kernels it's defined as HZ/5 which can equate to ~2ms. These ms
delays can be an eternity for sequential I/O patterns.
> 3. SCST doesn't have any hooks into CFQ and not going to have in the
> considerable future.
True, SCST doesn't have any hooks into CFQ, but your code modifies
block/blk-ioc.c to export the alloc_io_context(), which by default
is a private function, to allow your kernel based threads to set
their I/O contexts to the same group, therefore avoiding the delay
CFQ imposes on the switching of the I/O contexts between these
threads.
-Ross
______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.