Hi
Previously, I posted a patchset to add support of CFQ group hierarchical scheduling
in the way that it puts all CFQ queues in a hidden group and schedules with other
CFQ group under their parent. The patchset is available here,
http://lkml.org/lkml/2010/8/30/30
Vivek think this approach isn't so instinct that we should treat CFQ queues
and groups at the same level. Here is the new approach for hierarchical
scheduling based on Vivek's suggestion. The most big change of CFQ is that
it gets rid of cfq_slice_offset logic, and makes use of vdisktime for CFQ
queue scheduling just like CFQ group does. But I still give cfqq some jump
in vdisktime based on ioprio, thanks for Vivek to point out this. Now CFQ
queue and CFQ group use the same scheduling algorithm.
"use_hierarchy" interface is now added to switch between hierarchical mode
and flat mode. It works as memcg.
--
V3 -> V4 Changes:
- Take io class into account when calculating the boost value.
- Refine the vtime boosting logic as Vivek's Suggestion.
- Make the calculation of group slice cross all service trees under a group.
- Modify Documentation in terms of Vivek's comments.
--
V2 -> V3 Changes:
- Starting from cfqd->grp_service_tree for both hierarchical mode and flat mode
- Avoid recursion when allocating cfqg and force dispatch logic
- Fix a bug when boosting vdisktime
- Adjusting total_weight accordingly when changing weight
- Change group slice calculation into a hierarchical way
- Keep flat mode rather than deleting it first then adding it later
- kfree the parent cfqg if there nobody references to it
- Simplify select_queue logic by using some wrap function
- Make "use_hierarchy" interface work as memcg
- Make use of time_before() for vdisktime compare
- Update Document
- Fix some code style problems
--
V1 -> V2 Changes:
- Raname "struct io_sched_entity" to "struct cfq_entity" and don't differentiate
queue_entity and group_entity, just use cfqe instead.
- Give newly added cfqq a small vdisktime jump accord to its ioprio.
- Make flat mode as default CFQ group scheduling mode.
- Introduce "use_hierarchy" interface.
- Update blkio cgroup documents
Benchmarks:
I did some benchmarks by fio, I didn't see performance drop for throughtput or latency.
log is attached.
[PATCH 1/6 v4] cfq-iosched: Introduce cfq_entity for CFQ queue
[PATCH 2/6 v4] cfq-iosched: Introduce cfq_entity for CFQ group
[PATCH 3/6 v4] cfq-iosched: Introduce vdisktime and io weight for CFQ queue
[PATCH 4/6 v4] cfq-iosched: Extract some common code of service tree handling for CFQ queue and CFQ group
[PATCH 5/6 v4] cfq-iosched: CFQ group hierarchical scheduling and use_hierarchy interface
[PATCH 6/6 v4] blkio-cgroup: Document for blkio.use_hierarchy interface
Documentation/cgroups/blkio-controller.txt | 81 +-
block/blk-cgroup.c | 61 +
block/blk-cgroup.h | 3
block/cfq-iosched.c | 968 ++++++++++++++++++++--------
Thanks,
Gui
On Thu, Feb 10, 2011 at 03:45:45PM +0800, Gui Jianfeng wrote:
> Hi
>
> Previously, I posted a patchset to add support of CFQ group hierarchical scheduling
> in the way that it puts all CFQ queues in a hidden group and schedules with other
> CFQ group under their parent. The patchset is available here,
> http://lkml.org/lkml/2010/8/30/30
>
> Vivek think this approach isn't so instinct that we should treat CFQ queues
> and groups at the same level. Here is the new approach for hierarchical
> scheduling based on Vivek's suggestion. The most big change of CFQ is that
> it gets rid of cfq_slice_offset logic, and makes use of vdisktime for CFQ
> queue scheduling just like CFQ group does. But I still give cfqq some jump
> in vdisktime based on ioprio, thanks for Vivek to point out this. Now CFQ
> queue and CFQ group use the same scheduling algorithm.
>
> "use_hierarchy" interface is now added to switch between hierarchical mode
> and flat mode. It works as memcg.
>
> --
> V3 -> V4 Changes:
> - Take io class into account when calculating the boost value.
> - Refine the vtime boosting logic as Vivek's Suggestion.
Hi Gui,
What testing did you do to make sure that this vtime boosting logic is working
and is good replacement for slice_offset() logic for cfqq?
Secondly, did you get a chance to look at chad's patch of keeping track
of previous assigned vdisktime and keeping track of genrations. I think
his patch is going to coflict with yours, so one of you will have to
make adjustments. I think both the boost logic and keeping track of generation
logic can be combined.
if (entity->gneration_number > cfqd->active_generation)
Use_boost_logic;
else
Use_previously_assigned_vdisktime;
That way if we generation has changed then we really don't have a valid
vdisktime and we can use boost logic to come up with differential
vdisktime and if generation has not changed then we can continue to make
use of previous vdisktime.
Thanks
Vivek
Vivek Goyal wrote:
> On Thu, Feb 10, 2011 at 03:45:45PM +0800, Gui Jianfeng wrote:
>> Hi
>>
>> Previously, I posted a patchset to add support of CFQ group hierarchical scheduling
>> in the way that it puts all CFQ queues in a hidden group and schedules with other
>> CFQ group under their parent. The patchset is available here,
>> http://lkml.org/lkml/2010/8/30/30
>>
>> Vivek think this approach isn't so instinct that we should treat CFQ queues
>> and groups at the same level. Here is the new approach for hierarchical
>> scheduling based on Vivek's suggestion. The most big change of CFQ is that
>> it gets rid of cfq_slice_offset logic, and makes use of vdisktime for CFQ
>> queue scheduling just like CFQ group does. But I still give cfqq some jump
>> in vdisktime based on ioprio, thanks for Vivek to point out this. Now CFQ
>> queue and CFQ group use the same scheduling algorithm.
>>
>> "use_hierarchy" interface is now added to switch between hierarchical mode
>> and flat mode. It works as memcg.
>>
>> --
>> V3 -> V4 Changes:
>> - Take io class into account when calculating the boost value.
>> - Refine the vtime boosting logic as Vivek's Suggestion.
>
> Hi Gui,
>
> What testing did you do to make sure that this vtime boosting logic is working
> and is good replacement for slice_offset() logic for cfqq?
Hi Vivek,
I did fio test by runing 8 threads with different ioprio. It seems patched kernel
and vanilla kernel act the same behaviour. That is it doesn't play well. As I metioned
in previous mail, when I disable idling, all threads become SYNC_NOIDLE, and then
preempt each other, even if I start 8 threads. And reposition at the top of service
tree. This's why we can't see service differetiation.
>
> Secondly, did you get a chance to look at chad's patch of keeping track
> of previous assigned vdisktime and keeping track of genrations. I think
> his patch is going to coflict with yours, so one of you will have to
> make adjustments. I think both the boost logic and keeping track of generation
> logic can be combined.
>
> if (entity->gneration_number > cfqd->active_generation)
> Use_boost_logic;
> else
> Use_previously_assigned_vdisktime;
>
>
> That way if we generation has changed then we really don't have a valid
> vdisktime and we can use boost logic to come up with differential
> vdisktime and if generation has not changed then we can continue to make
> use of previous vdisktime.
Ok, I'd like to take a look chad's patch.
Thanks
Gui
>
> Thanks
> Vivek
>
Vivek Goyal wrote:
> On Thu, Feb 10, 2011 at 03:45:45PM +0800, Gui Jianfeng wrote:
>> Hi
>>
>> Previously, I posted a patchset to add support of CFQ group hierarchical scheduling
>> in the way that it puts all CFQ queues in a hidden group and schedules with other
>> CFQ group under their parent. The patchset is available here,
>> http://lkml.org/lkml/2010/8/30/30
>>
>> Vivek think this approach isn't so instinct that we should treat CFQ queues
>> and groups at the same level. Here is the new approach for hierarchical
>> scheduling based on Vivek's suggestion. The most big change of CFQ is that
>> it gets rid of cfq_slice_offset logic, and makes use of vdisktime for CFQ
>> queue scheduling just like CFQ group does. But I still give cfqq some jump
>> in vdisktime based on ioprio, thanks for Vivek to point out this. Now CFQ
>> queue and CFQ group use the same scheduling algorithm.
>>
>> "use_hierarchy" interface is now added to switch between hierarchical mode
>> and flat mode. It works as memcg.
>>
>> --
>> V3 -> V4 Changes:
>> - Take io class into account when calculating the boost value.
>> - Refine the vtime boosting logic as Vivek's Suggestion.
>
> Hi Gui,
>
> What testing did you do to make sure that this vtime boosting logic is working
> and is good replacement for slice_offset() logic for cfqq?
>
> Secondly, did you get a chance to look at chad's patch of keeping track
> of previous assigned vdisktime and keeping track of genrations. I think
> his patch is going to coflict with yours, so one of you will have to
> make adjustments. I think both the boost logic and keeping track of generation
> logic can be combined.
Hi Vivek,
For the time being, boosting logic only works for cfq queue, and keepinging track of
generation works for cfq group.
I think when we introduce boosting logic for cfq group, we can combine these two logic
together.
Thanks,
Gui
>
> if (entity->gneration_number > cfqd->active_generation)
> Use_boost_logic;
> else
> Use_previously_assigned_vdisktime;
>
>
> That way if we generation has changed then we really don't have a valid
> vdisktime and we can use boost logic to come up with differential
> vdisktime and if generation has not changed then we can continue to make
> use of previous vdisktime.
>
> Thanks
> Vivek
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
On Sat, Feb 12, 2011 at 02:08:54PM +0800, Gui Jianfeng wrote:
> Vivek Goyal wrote:
> > On Thu, Feb 10, 2011 at 03:45:45PM +0800, Gui Jianfeng wrote:
> >> Hi
> >>
> >> Previously, I posted a patchset to add support of CFQ group hierarchical scheduling
> >> in the way that it puts all CFQ queues in a hidden group and schedules with other
> >> CFQ group under their parent. The patchset is available here,
> >> http://lkml.org/lkml/2010/8/30/30
> >>
> >> Vivek think this approach isn't so instinct that we should treat CFQ queues
> >> and groups at the same level. Here is the new approach for hierarchical
> >> scheduling based on Vivek's suggestion. The most big change of CFQ is that
> >> it gets rid of cfq_slice_offset logic, and makes use of vdisktime for CFQ
> >> queue scheduling just like CFQ group does. But I still give cfqq some jump
> >> in vdisktime based on ioprio, thanks for Vivek to point out this. Now CFQ
> >> queue and CFQ group use the same scheduling algorithm.
> >>
> >> "use_hierarchy" interface is now added to switch between hierarchical mode
> >> and flat mode. It works as memcg.
> >>
> >> --
> >> V3 -> V4 Changes:
> >> - Take io class into account when calculating the boost value.
> >> - Refine the vtime boosting logic as Vivek's Suggestion.
> >
> > Hi Gui,
> >
> > What testing did you do to make sure that this vtime boosting logic is working
> > and is good replacement for slice_offset() logic for cfqq?
> >
> > Secondly, did you get a chance to look at chad's patch of keeping track
> > of previous assigned vdisktime and keeping track of genrations. I think
> > his patch is going to coflict with yours, so one of you will have to
> > make adjustments. I think both the boost logic and keeping track of generation
> > logic can be combined.
>
> Hi Vivek,
>
> For the time being, boosting logic only works for cfq queue, and keepinging track of
> generation works for cfq group.
> I think when we introduce boosting logic for cfq group, we can combine these two logic
> together.
Ok, I am assuming that now you are introducing another patch in the series
to get boosting logic working for groups also.
Also, it might be easier to test it with group_idle=0, as right now groups
don't preempt each other.
Thanks
Vivek
Vivek Goyal wrote:
> On Sat, Feb 12, 2011 at 02:08:54PM +0800, Gui Jianfeng wrote:
>> Vivek Goyal wrote:
>>> On Thu, Feb 10, 2011 at 03:45:45PM +0800, Gui Jianfeng wrote:
>>>> Hi
>>>>
>>>> Previously, I posted a patchset to add support of CFQ group hierarchical scheduling
>>>> in the way that it puts all CFQ queues in a hidden group and schedules with other
>>>> CFQ group under their parent. The patchset is available here,
>>>> http://lkml.org/lkml/2010/8/30/30
>>>>
>>>> Vivek think this approach isn't so instinct that we should treat CFQ queues
>>>> and groups at the same level. Here is the new approach for hierarchical
>>>> scheduling based on Vivek's suggestion. The most big change of CFQ is that
>>>> it gets rid of cfq_slice_offset logic, and makes use of vdisktime for CFQ
>>>> queue scheduling just like CFQ group does. But I still give cfqq some jump
>>>> in vdisktime based on ioprio, thanks for Vivek to point out this. Now CFQ
>>>> queue and CFQ group use the same scheduling algorithm.
>>>>
>>>> "use_hierarchy" interface is now added to switch between hierarchical mode
>>>> and flat mode. It works as memcg.
>>>>
>>>> --
>>>> V3 -> V4 Changes:
>>>> - Take io class into account when calculating the boost value.
>>>> - Refine the vtime boosting logic as Vivek's Suggestion.
>>> Hi Gui,
>>>
>>> What testing did you do to make sure that this vtime boosting logic is working
>>> and is good replacement for slice_offset() logic for cfqq?
>>>
>>> Secondly, did you get a chance to look at chad's patch of keeping track
>>> of previous assigned vdisktime and keeping track of genrations. I think
>>> his patch is going to coflict with yours, so one of you will have to
>>> make adjustments. I think both the boost logic and keeping track of generation
>>> logic can be combined.
>> Hi Vivek,
>>
>> For the time being, boosting logic only works for cfq queue, and keepinging track of
>> generation works for cfq group.
>> I think when we introduce boosting logic for cfq group, we can combine these two logic
>> together.
>
> Ok, I am assuming that now you are introducing another patch in the series
> to get boosting logic working for groups also.
>
> Also, it might be easier to test it with group_idle=0, as right now groups
> don't preempt each other.
Vivek,
If you don't object, I'd like to introduce this in a seperate patch when this series
gets merged. For I don't want to add more complexity for this seires. ;)
I'll post v5.
Thanks,
Gui
>
> Thanks
> Vivek
>
On Tue, Feb 15, 2011 at 11:13:48AM +0800, Gui Jianfeng wrote:
> Vivek Goyal wrote:
> > On Sat, Feb 12, 2011 at 02:08:54PM +0800, Gui Jianfeng wrote:
> >> Vivek Goyal wrote:
> >>> On Thu, Feb 10, 2011 at 03:45:45PM +0800, Gui Jianfeng wrote:
> >>>> Hi
> >>>>
> >>>> Previously, I posted a patchset to add support of CFQ group hierarchical scheduling
> >>>> in the way that it puts all CFQ queues in a hidden group and schedules with other
> >>>> CFQ group under their parent. The patchset is available here,
> >>>> http://lkml.org/lkml/2010/8/30/30
> >>>>
> >>>> Vivek think this approach isn't so instinct that we should treat CFQ queues
> >>>> and groups at the same level. Here is the new approach for hierarchical
> >>>> scheduling based on Vivek's suggestion. The most big change of CFQ is that
> >>>> it gets rid of cfq_slice_offset logic, and makes use of vdisktime for CFQ
> >>>> queue scheduling just like CFQ group does. But I still give cfqq some jump
> >>>> in vdisktime based on ioprio, thanks for Vivek to point out this. Now CFQ
> >>>> queue and CFQ group use the same scheduling algorithm.
> >>>>
> >>>> "use_hierarchy" interface is now added to switch between hierarchical mode
> >>>> and flat mode. It works as memcg.
> >>>>
> >>>> --
> >>>> V3 -> V4 Changes:
> >>>> - Take io class into account when calculating the boost value.
> >>>> - Refine the vtime boosting logic as Vivek's Suggestion.
> >>> Hi Gui,
> >>>
> >>> What testing did you do to make sure that this vtime boosting logic is working
> >>> and is good replacement for slice_offset() logic for cfqq?
> >>>
> >>> Secondly, did you get a chance to look at chad's patch of keeping track
> >>> of previous assigned vdisktime and keeping track of genrations. I think
> >>> his patch is going to coflict with yours, so one of you will have to
> >>> make adjustments. I think both the boost logic and keeping track of generation
> >>> logic can be combined.
> >> Hi Vivek,
> >>
> >> For the time being, boosting logic only works for cfq queue, and keepinging track of
> >> generation works for cfq group.
> >> I think when we introduce boosting logic for cfq group, we can combine these two logic
> >> together.
> >
> > Ok, I am assuming that now you are introducing another patch in the series
> > to get boosting logic working for groups also.
> >
> > Also, it might be easier to test it with group_idle=0, as right now groups
> > don't preempt each other.
>
> Vivek,
>
> If you don't object, I'd like to introduce this in a seperate patch when this series
> gets merged. For I don't want to add more complexity for this seires. ;)
> I'll post v5.
Ok, that's fine. We can implement boosting logic for groups later.
I think Chad's patch of remembering generation should help in getting service
differentiation even with idiling disabled.
Thanks
Vivek