Any reason why submit_bh should turn on interrupts after being called by
a process with ints off in 2.4.20? I see it's possible to sleep during
elevatoring, but why does it need to leave interrupts on if the calling
state was with ints off.
I'm back BTW. Hope no one missed me. I can be reached at the email
address above.
:-)
Jeff
On Wed, Jun 02 2004, Jeff V. Merkey wrote:
>
> Any reason why submit_bh should turn on interrupts after being called by
> a process with ints off in 2.4.20? I see it's possible to sleep during
> elevatoring, but why does it need to leave interrupts on if the calling
> state was with ints off.
It's illegal to call it with interrupts off, so... __make_request()
doesn't save interrupt state, so you will always leave with interrupts
enabled.
--
Jens Axboe
Jens Axboe wrote:
>On Wed, Jun 02 2004, Jeff V. Merkey wrote:
>
>
>>Any reason why submit_bh should turn on interrupts after being called by
>>a process with ints off in 2.4.20? I see it's possible to sleep during
>>elevatoring, but why does it need to leave interrupts on if the calling
>>state was with ints off.
>>
>>
>
>It's illegal to call it with interrupts off, so... __make_request()
>doesn't save interrupt state, so you will always leave with interrupts
>enabled.
>
>
>
Jens
I noticed in the code it does not check for this when make_request is
called, so I altered the calling sequence to call with ints on. I don't
see much of a performance difference either way, so calling with ints on
was easy to instrument. I am posting about 80,000+ buffer heads per
second in with what I am doing, so filling out buffer_head structures
and submitting them ad hoc was causing some interrupt windows where the
chains were getting corrupted. I altered the calling sequence and added
atomic counters so I can submit and call with ints on to avoid the
corruption. One of the troublesome aspects of the manner in which
make_request is implemented in always needing a context of a thread for
sleeping to submit asynch I/O limits the ability to gang schedule large
disk I/O from the b_end_io callback. Would make performance a lot more
spectacular if it worked this way, but I am seeing good enough
performance with it left the way it is. 3Ware's 66Mhz ATA adapter in
this implementation is reaching almost 400 MB/S throughput on 2.4.20. I
have not tried this on 2.6 yet, but will later this month.
Also, I ported the kernel debugger from MANOS to Linux and made a lot of
significant enhancements and http://www.devicelogics.com is distributing it
from their website. If anyone wants a more pleasant debugger for the
kernel to work with, they are allowing downloads of the modules with a
patch. Not free (what is in this world) but very nice to work with.
Thanks for the response. Sorry I was out of touch for about a year. I
was going through a very nasty divorce with my wife of 24 years and I
discovered when something like that is happening in your life, you don't
have much attention for much else.
Jeff
On Thu, Jun 03 2004, Jeff V. Merkey wrote:
> Jens Axboe wrote:
>
> >On Wed, Jun 02 2004, Jeff V. Merkey wrote:
> >
> >
> >>Any reason why submit_bh should turn on interrupts after being called by
> >>a process with ints off in 2.4.20? I see it's possible to sleep during
> >>elevatoring, but why does it need to leave interrupts on if the calling
> >>state was with ints off.
> >>
> >>
> >
> >It's illegal to call it with interrupts off, so... __make_request()
> >doesn't save interrupt state, so you will always leave with interrupts
> >enabled.
> >
> >
> >
> Jens
>
> I noticed in the code it does not check for this when make_request is
> called, so I altered the calling sequence to call with ints on. I don't
> see much of a performance difference either way, so calling with ints on
> was easy to instrument. I am posting about 80,000+ buffer heads per
> second in with what I am doing, so filling out buffer_head structures
> and submitting them ad hoc was causing some interrupt windows where the
> chains were getting corrupted. I altered the calling sequence and added
> atomic counters so I can submit and call with ints on to avoid the
> corruption. One of the troublesome aspects of the manner in which
> make_request is implemented in always needing a context of a thread for
> sleeping to submit asynch I/O limits the ability to gang schedule large
> disk I/O from the b_end_io callback. Would make performance a lot more
> spectacular if it worked this way, but I am seeing good enough
> performance with it left the way it is. 3Ware's 66Mhz ATA adapter in
> this implementation is reaching almost 400 MB/S throughput on 2.4.20. I
> have not tried this on 2.6 yet, but will later this month.
Submitting large numbers of buffer_heads from b_end_io is _nasty_, 2.4
io scheduler runtime isn't exactly world champion and you are doing this
at hard irq time. Not a good idea. Definitely not the true path to
performance, unless you don't care about anything else in the system.
At least in 2.6 you have a much faster io scheduler and the additionally
large bio, so you wont spend nearly as much time there if you are
clever. You still need process context, though, that hasn't changed.
--
Jens Axboe
Jens Axboe wrote:
>Submitting large numbers of buffer_heads from b_end_io is _nasty_, 2.4
>io scheduler runtime isn't exactly world champion and you are doing this
>at hard irq time. Not a good idea. Definitely not the true path to
>performance, unless you don't care about anything else in the system.
>
>At least in 2.6 you have a much faster io scheduler and the additionally
>large bio, so you wont spend nearly as much time there if you are
>clever. You still need process context, though, that hasn't changed.
>
>
>
Sounds like I need to move to 2.6. I noticed the elevator is coalescing
quite well, and since I am posting mostly continguous runs of sectors,
what ends up at the adapter level would probably not change much much
between 2.4 and 2.6 since I am maxing out the driver request queues as
it is (255 pending requests of 32 scatter/gather elements of 256 sector
runs). 2.6 might help but I suspect it will only help alleviate the
submission overhead, and not make much difference on performance since
the 3Ware card does have an upward limit on outstanding I/O requests.
Jeff
On Thu, Jun 03 2004, Jeff V. Merkey wrote:
> Jens Axboe wrote:
>
> >Submitting large numbers of buffer_heads from b_end_io is _nasty_, 2.4
> >io scheduler runtime isn't exactly world champion and you are doing this
> >at hard irq time. Not a good idea. Definitely not the true path to
> >performance, unless you don't care about anything else in the system.
> >
> >At least in 2.6 you have a much faster io scheduler and the additionally
> >large bio, so you wont spend nearly as much time there if you are
> >clever. You still need process context, though, that hasn't changed.
> >
> >
> >
> Sounds like I need to move to 2.6. I noticed the elevator is coalescing
> quite well, and since I am posting mostly continguous runs of sectors,
> what ends up at the adapter level would probably not change much much
> between 2.4 and 2.6 since I am maxing out the driver request queues as
> it is (255 pending requests of 32 scatter/gather elements of 256 sector
> runs). 2.6 might help but I suspect it will only help alleviate the
> submission overhead, and not make much difference on performance since
> the 3Ware card does have an upward limit on outstanding I/O requests.
That's correct, it just helps you diminish the submission overhead by
pushing down 256 sector entities in one go. So as long as you're io
bound it won't give you better io performance, of course. If you are
doing 400MiB/sec it should help you out, though.
--
Jens Axboe
On Thu, 3 Jun 2004, Jens Axboe wrote:
> On Thu, Jun 03 2004, Jeff V. Merkey wrote:
> >
> > Sounds like I need to move to 2.6. I noticed the elevator is coalescing
> > quite well, and since I am posting mostly continguous runs of sectors,
> > what ends up at the adapter level would probably not change much much
> > between 2.4 and 2.6 since I am maxing out the driver request queues as
> > it is (255 pending requests of 32 scatter/gather elements of 256 sector
> > runs). 2.6 might help but I suspect it will only help alleviate the
> > submission overhead, and not make much difference on performance since
> > the 3Ware card does have an upward limit on outstanding I/O requests.
>
> That's correct, it just helps you diminish the submission overhead by
> pushing down 256 sector entities in one go. So as long as you're io
> bound it won't give you better io performance, of course. If you are
> doing 400MiB/sec it should help you out, though.
Well, if Jeff does almost exclusively contiguous stuff and submits them in
order, then the coalescing will make sure that even on 2.4.x the queues
don't get too long, and he probably won't see the pathological cases.
Linus
Jens Axboe wrote:
>On Thu, Jun 03 2004, Jeff V. Merkey wrote:
>
>
>>Jens Axboe wrote:
>>
>>
>>
>>>Submitting large numbers of buffer_heads from b_end_io is _nasty_, 2.4
>>>io scheduler runtime isn't exactly world champion and you are doing this
>>>at hard irq time. Not a good idea. Definitely not the true path to
>>>performance, unless you don't care about anything else in the system.
>>>
>>>At least in 2.6 you have a much faster io scheduler and the additionally
>>>large bio, so you wont spend nearly as much time there if you are
>>>clever. You still need process context, though, that hasn't changed.
>>>
>>>
>>>
>>>
>>>
>>Sounds like I need to move to 2.6. I noticed the elevator is coalescing
>>quite well, and since I am posting mostly continguous runs of sectors,
>>what ends up at the adapter level would probably not change much much
>>between 2.4 and 2.6 since I am maxing out the driver request queues as
>>it is (255 pending requests of 32 scatter/gather elements of 256 sector
>>runs). 2.6 might help but I suspect it will only help alleviate the
>>submission overhead, and not make much difference on performance since
>>the 3Ware card does have an upward limit on outstanding I/O requests.
>>
>>
>
>That's correct, it just helps you diminish the submission overhead by
>pushing down 256 sector entities in one go. So as long as you're io
>bound it won't give you better io performance, of course. If you are
>doing 400MiB/sec it should help you out, though.
>
>
>
I'll give it a try n 2.6.
:-)
Jeff
On Thu, Jun 03 2004, Linus Torvalds wrote:
>
>
> On Thu, 3 Jun 2004, Jens Axboe wrote:
>
> > On Thu, Jun 03 2004, Jeff V. Merkey wrote:
> > >
> > > Sounds like I need to move to 2.6. I noticed the elevator is coalescing
> > > quite well, and since I am posting mostly continguous runs of sectors,
> > > what ends up at the adapter level would probably not change much much
> > > between 2.4 and 2.6 since I am maxing out the driver request queues as
> > > it is (255 pending requests of 32 scatter/gather elements of 256 sector
> > > runs). 2.6 might help but I suspect it will only help alleviate the
> > > submission overhead, and not make much difference on performance since
> > > the 3Ware card does have an upward limit on outstanding I/O requests.
> >
> > That's correct, it just helps you diminish the submission overhead by
> > pushing down 256 sector entities in one go. So as long as you're io
> > bound it won't give you better io performance, of course. If you are
> > doing 400MiB/sec it should help you out, though.
>
> Well, if Jeff does almost exclusively contiguous stuff and submits them in
> order, then the coalescing will make sure that even on 2.4.x the queues
> don't get too long, and he probably won't see the pathological cases.
At 255 pending requests (see above), it will get somewhat bad. So even
if 2.4 copes, it will spend more sys time than 2.6 for this work load
for sure. The queues will get long even for mainly contig submissions,
but of course the time spent in the io scheduler will not be too bad
since it's scanned backwards. And call frequency will be 32 times as
big...
--
Jens Axboe
Linus Torvalds wrote:
>On Thu, 3 Jun 2004, Jens Axboe wrote:
>
>
>
>>On Thu, Jun 03 2004, Jeff V. Merkey wrote:
>>
>>
>>>Sounds like I need to move to 2.6. I noticed the elevator is coalescing
>>>quite well, and since I am posting mostly continguous runs of sectors,
>>>what ends up at the adapter level would probably not change much much
>>>between 2.4 and 2.6 since I am maxing out the driver request queues as
>>>it is (255 pending requests of 32 scatter/gather elements of 256 sector
>>>runs). 2.6 might help but I suspect it will only help alleviate the
>>>submission overhead, and not make much difference on performance since
>>>the 3Ware card does have an upward limit on outstanding I/O requests.
>>>
>>>
>>That's correct, it just helps you diminish the submission overhead by
>>pushing down 256 sector entities in one go. So as long as you're io
>>bound it won't give you better io performance, of course. If you are
>>doing 400MiB/sec it should help you out, though.
>>
>>
>
>Well, if Jeff does almost exclusively contiguous stuff and submits them in
>order, then the coalescing will make sure that even on 2.4.x the queues
>don't get too long, and he probably won't see the pathological cases.
>
> Linus
>-
>
Linus,
This seems to be the case.
Jeff
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to [email protected]
>More majordomo info at http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at http://www.tux.org/lkml/
>
>
>