Lukas Czerner <[email protected]> writes:
> Currently there is not limitation of number of requests in the loop bio
> list. This can lead into some nasty situations when the caller spawns
> tons of bio requests taking huge amount of memory. This is even more
> obvious with discard where blkdev_issue_discard() will submit all bios
> for the range and wait for them to finish afterwards. On really big loop
> devices this can lead to OOM situation as reported by Dave Chinner.
>
> With this patch we will wait in loop_make_request() if the number of
> bios in the loop bio list would exceed 'nr_requests' number of requests.
> We'll wake up the process as we process the bios form the list.
I think you might want to do something similar to what is done for
request_queues by implementing a congestion on and off threshold. As
Jens writes in this commit (predating the conversion to git):
Author: Jens Axboe <[email protected]>
Date: Wed Nov 3 15:47:37 2004 -0800
[PATCH] queue congestion threshold hysteresis
We need to open the gap between congestion on/off a little bit, or
we risk burning many cycles continually putting processes on a wait
queue only to wake them up again immediately. This was observed with
CFQ at least, which showed way excessive sys time.
Patch is from Arjan.
Signed-off-by: Jens Axboe <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
If you feel this isn't necessary, then I think you at least need to
justify it with testing. Perhaps Jens can shed some light on the exact
workload that triggered the pathological behaviour.
Cheers,
Jeff
On Mon, 1 Oct 2012, Jeff Moyer wrote:
> Date: Mon, 01 Oct 2012 12:52:19 -0400
> From: Jeff Moyer <[email protected]>
> To: Lukas Czerner <[email protected]>
> Cc: Jens Axboe <[email protected]>, [email protected],
> Dave Chinner <[email protected]>
> Subject: Re: [PATCH] loop: Limit the number of requests in the bio list
>
> Lukas Czerner <[email protected]> writes:
>
> > Currently there is not limitation of number of requests in the loop bio
> > list. This can lead into some nasty situations when the caller spawns
> > tons of bio requests taking huge amount of memory. This is even more
> > obvious with discard where blkdev_issue_discard() will submit all bios
> > for the range and wait for them to finish afterwards. On really big loop
> > devices this can lead to OOM situation as reported by Dave Chinner.
> >
> > With this patch we will wait in loop_make_request() if the number of
> > bios in the loop bio list would exceed 'nr_requests' number of requests.
> > We'll wake up the process as we process the bios form the list.
>
> I think you might want to do something similar to what is done for
> request_queues by implementing a congestion on and off threshold. As
> Jens writes in this commit (predating the conversion to git):
Right, I've had the same idea. However my first proof-of-concept
worked quite well without this and my simple performance testing did
not show any regression.
I've basically done just fstrim, and blkdiscard on huge loop device
measuring time to finish and dd bs=4k throughput. None of those showed
any performance regression. I've chosen those for being quite simple
and supposedly issuing quite a lot of bios. Any better
recommendation to test this ?
Also I am still unable to reproduce the problem Dave originally
experienced and I was hoping that he can test whether this helps or
not.
Dave could you give it a try please ? By creating huge (500T, 1000T,
1500T) loop device on machine with 2GB memory I was not able to reproduce
that. Maybe it's that xfs punch hole implementation is so damn fast
:). Please let me know.
Thanks!
-Lukas
>
> Author: Jens Axboe <[email protected]>
> Date: Wed Nov 3 15:47:37 2004 -0800
>
> [PATCH] queue congestion threshold hysteresis
>
> We need to open the gap between congestion on/off a little bit, or
> we risk burning many cycles continually putting processes on a wait
> queue only to wake them up again immediately. This was observed with
> CFQ at least, which showed way excessive sys time.
>
> Patch is from Arjan.
>
> Signed-off-by: Jens Axboe <[email protected]>
> Signed-off-by: Linus Torvalds <[email protected]>
>
> If you feel this isn't necessary, then I think you at least need to
> justify it with testing. Perhaps Jens can shed some light on the exact
> workload that triggered the pathological behaviour.
>
> Cheers,
> Jeff
>
On Tue, Oct 02, 2012 at 10:52:05AM +0200, Lukáš Czerner wrote:
> On Mon, 1 Oct 2012, Jeff Moyer wrote:
> > Date: Mon, 01 Oct 2012 12:52:19 -0400
> > From: Jeff Moyer <[email protected]>
> > To: Lukas Czerner <[email protected]>
> > Cc: Jens Axboe <[email protected]>, [email protected],
> > Dave Chinner <[email protected]>
> > Subject: Re: [PATCH] loop: Limit the number of requests in the bio list
> >
> > Lukas Czerner <[email protected]> writes:
> >
> > > Currently there is not limitation of number of requests in the loop bio
> > > list. This can lead into some nasty situations when the caller spawns
> > > tons of bio requests taking huge amount of memory. This is even more
> > > obvious with discard where blkdev_issue_discard() will submit all bios
> > > for the range and wait for them to finish afterwards. On really big loop
> > > devices this can lead to OOM situation as reported by Dave Chinner.
> > >
> > > With this patch we will wait in loop_make_request() if the number of
> > > bios in the loop bio list would exceed 'nr_requests' number of requests.
> > > We'll wake up the process as we process the bios form the list.
> >
> > I think you might want to do something similar to what is done for
> > request_queues by implementing a congestion on and off threshold. As
> > Jens writes in this commit (predating the conversion to git):
>
> Right, I've had the same idea. However my first proof-of-concept
> worked quite well without this and my simple performance testing did
> not show any regression.
>
> I've basically done just fstrim, and blkdiscard on huge loop device
> measuring time to finish and dd bs=4k throughput. None of those showed
> any performance regression. I've chosen those for being quite simple
> and supposedly issuing quite a lot of bios. Any better
> recommendation to test this ?
>
> Also I am still unable to reproduce the problem Dave originally
> experienced and I was hoping that he can test whether this helps or
> not.
>
> Dave could you give it a try please ? By creating huge (500T, 1000T,
> 1500T) loop device on machine with 2GB memory I was not able to reproduce
> that. Maybe it's that xfs punch hole implementation is so damn fast
> :). Please let me know.
Try a file with a few hundred thousand extents in it (preallocate
them). I found this while testing large block devices on loopback
devices, not with empty files.
Cheers,
Dave.
--
Dave Chinner
[email protected]
Dave Chinner <[email protected]> writes:
> On Tue, Oct 02, 2012 at 10:52:05AM +0200, Lukáš Czerner wrote:
>> On Mon, 1 Oct 2012, Jeff Moyer wrote:
>> > Date: Mon, 01 Oct 2012 12:52:19 -0400
>> > From: Jeff Moyer <[email protected]>
>> > To: Lukas Czerner <[email protected]>
>> > Cc: Jens Axboe <[email protected]>, [email protected],
>> > Dave Chinner <[email protected]>
>> > Subject: Re: [PATCH] loop: Limit the number of requests in the bio list
>> >
>> > Lukas Czerner <[email protected]> writes:
>> >
>> > > Currently there is not limitation of number of requests in the loop bio
>> > > list. This can lead into some nasty situations when the caller spawns
>> > > tons of bio requests taking huge amount of memory. This is even more
>> > > obvious with discard where blkdev_issue_discard() will submit all bios
>> > > for the range and wait for them to finish afterwards. On really big loop
>> > > devices this can lead to OOM situation as reported by Dave Chinner.
>> > >
>> > > With this patch we will wait in loop_make_request() if the number of
>> > > bios in the loop bio list would exceed 'nr_requests' number of requests.
>> > > We'll wake up the process as we process the bios form the list.
>> >
>> > I think you might want to do something similar to what is done for
>> > request_queues by implementing a congestion on and off threshold. As
>> > Jens writes in this commit (predating the conversion to git):
>>
>> Right, I've had the same idea. However my first proof-of-concept
>> worked quite well without this and my simple performance testing did
>> not show any regression.
Did you look at system time?
-Jeff
On Wed, 3 Oct 2012, Jeff Moyer wrote:
> Date: Wed, 03 Oct 2012 10:30:54 -0400
> From: Jeff Moyer <[email protected]>
> To: Dave Chinner <[email protected]>
> Cc: Lukáš Czerner <[email protected]>, Jens Axboe <[email protected]>,
> [email protected]
> Subject: Re: [PATCH] loop: Limit the number of requests in the bio list
>
> Dave Chinner <[email protected]> writes:
>
> > On Tue, Oct 02, 2012 at 10:52:05AM +0200, Lukáš Czerner wrote:
> >> On Mon, 1 Oct 2012, Jeff Moyer wrote:
> >> > Date: Mon, 01 Oct 2012 12:52:19 -0400
> >> > From: Jeff Moyer <[email protected]>
> >> > To: Lukas Czerner <[email protected]>
> >> > Cc: Jens Axboe <[email protected]>, [email protected],
> >> > Dave Chinner <[email protected]>
> >> > Subject: Re: [PATCH] loop: Limit the number of requests in the bio list
> >> >
> >> > Lukas Czerner <[email protected]> writes:
> >> >
> >> > > Currently there is not limitation of number of requests in the loop bio
> >> > > list. This can lead into some nasty situations when the caller spawns
> >> > > tons of bio requests taking huge amount of memory. This is even more
> >> > > obvious with discard where blkdev_issue_discard() will submit all bios
> >> > > for the range and wait for them to finish afterwards. On really big loop
> >> > > devices this can lead to OOM situation as reported by Dave Chinner.
> >> > >
> >> > > With this patch we will wait in loop_make_request() if the number of
> >> > > bios in the loop bio list would exceed 'nr_requests' number of requests.
> >> > > We'll wake up the process as we process the bios form the list.
> >> >
> >> > I think you might want to do something similar to what is done for
> >> > request_queues by implementing a congestion on and off threshold. As
> >> > Jens writes in this commit (predating the conversion to git):
> >>
> >> Right, I've had the same idea. However my first proof-of-concept
> >> worked quite well without this and my simple performance testing did
> >> not show any regression.
>
> Did you look at system time?
>
> -Jeff
Hi, none of the times showed any significant difference, there was
not any pattern suggesting a problem. Also the system time is included
in the real time, so it would show anyway I guess.
-Lukas
Lukáš Czerner <[email protected]> writes:
>> Did you look at system time?
>>
>> -Jeff
>
> Hi, none of the times showed any significant difference, there was
> not any pattern suggesting a problem. Also the system time is included
> in the real time, so it would show anyway I guess.
Only if you're cpu bound. If you aren't, elevated system time may not
increase overall runtime.
Cheers,
Jeff