Tasks might get more cpu than quota in persistent periods due to the
cpu burst introduced by commit f4183717b370 ("sched/fair: Introduce the
burstable CFS controller"). For example, one task group whose quota is
100ms per period and can get 100ms burst, and its avg utilization is
around 105ms per period. Once this group gets a free period which
leaves enough runtime, it has a chance to get computting power more
than its quota for 10 periods or more in common bandwidth configuration
(say, 100ms as period). It means tasks can 'steal' the bursted power to
do daily jobs because all tasks could be scheduled out or sleep to help
the group get free periods.
I believe the purpose of cpu burst is to help handling bursty worklod.
But if one task group can get computting power more than its quota for
persistent periods even there is no bursty workload, it's kinda broke.
This patch limits the burst to 2 periods so that it won't break the
quota limit for long. Permitting 2 periods can help on the scenario that
periods refresh lands in the middle of a burst workload. With this, we
can give task group more cpu burst power to handle the real burst
workload and don't worry about the 'stealing'.
Signed-off-by: Honglei Wang <[email protected]>
---
kernel/sched/fair.c | 13 ++++++++++---
kernel/sched/sched.h | 1 +
2 files changed, 11 insertions(+), 3 deletions(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 2cd626c22912..4e04cb4269ba 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4645,14 +4645,21 @@ void __refill_cfs_bandwidth_runtime(struct cfs_bandwidth *cfs_b)
return;
}
- cfs_b->runtime += cfs_b->quota;
- runtime = cfs_b->runtime_snap - cfs_b->runtime;
+ runtime = cfs_b->runtime_snap - cfs_b->runtime - cfs_b->quota;
if (runtime > 0) {
cfs_b->burst_time += runtime;
cfs_b->nr_burst++;
+ cfs_b->burst_periods++;
+ }
+
+ if (cfs_b->burst_periods > 1) {
+ cfs_b->runtime = cfs_b->quota;
+ cfs_b->burst_periods = 0;
+ } else {
+ cfs_b->runtime += cfs_b->quota;
+ cfs_b->runtime = min(cfs_b->runtime, cfs_b->quota + cfs_b->burst);
}
- cfs_b->runtime = min(cfs_b->runtime, cfs_b->quota + cfs_b->burst);
cfs_b->runtime_snap = cfs_b->runtime;
}
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 0e66749486e7..f42280bca3b2 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -370,6 +370,7 @@ struct cfs_bandwidth {
u64 burst;
u64 runtime_snap;
s64 hierarchical_quota;
+ u8 burst_periods;
u8 idle;
u8 period_active;
--
2.14.1
On Wed, Dec 08, 2021 at 10:50:38PM +0800, Honglei Wang wrote:
> Tasks might get more cpu than quota in persistent periods due to the
> cpu burst introduced by commit f4183717b370 ("sched/fair: Introduce the
> burstable CFS controller").
> For example, one task group whose quota is
> 100ms per period and can get 100ms burst, and its avg utilization is
> around 105ms per period.
That would be a mis-configuration, surely..
> Once this group gets a free period which
> leaves enough runtime, it has a chance to get computting power more
> than its quota for 10 periods or more in common bandwidth configuration
> (say, 100ms as period).
Sure, if it, for some miraculous reason, decides to sleep for a whole
period and then resume, it can indeed consume up to that 100ms extra,
which, if as per the above, done at 5ms per perios, would be 20 periods
until depleted.
> It means tasks can 'steal' the bursted power to
> do daily jobs because all tasks could be scheduled out or sleep to help
> the group get free periods.
That's the design,,
> I believe the purpose of cpu burst is to help handling bursty worklod.
> But if one task group can get computting power more than its quota for
> persistent periods even there is no bursty workload, it's kinda broke.
So if that was were bursty, it could consume that 100ms extra in a
single go and that would be fine, but spreading that same amount over 20
periods is somehow a problem? -- even though the interference is less.
> This patch limits the burst to 2 periods so that it won't break the
> quota limit for long. Permitting 2 periods can help on the scenario that
> periods refresh lands in the middle of a burst workload. With this, we
> can give task group more cpu burst power to handle the real burst
> workload and don't worry about the 'stealing'.
I've yet so see an actual reason for any of this...
On 2021/12/9 21:08, Peter Zijlstra wrote:
> On Wed, Dec 08, 2021 at 10:50:38PM +0800, Honglei Wang wrote:
>> Tasks might get more cpu than quota in persistent periods due to the
>> cpu burst introduced by commit f4183717b370 ("sched/fair: Introduce the
>> burstable CFS controller").
>
>> For example, one task group whose quota is
>> 100ms per period and can get 100ms burst, and its avg utilization is
>> around 105ms per period.
>
> That would be a mis-configuration, surely..
>
Well, it's a lame example to describe the spreading of the burst..
>> Once this group gets a free period which
>> leaves enough runtime, it has a chance to get computting power more
>> than its quota for 10 periods or more in common bandwidth configuration
>> (say, 100ms as period).
>
> Sure, if it, for some miraculous reason, decides to sleep for a whole
> period and then resume, it can indeed consume up to that 100ms extra,
> which, if as per the above, done at 5ms per perios, would be 20 periods
> until depleted.
>
>> It means tasks can 'steal' the bursted power to
>> do daily jobs because all tasks could be scheduled out or sleep to help
>> the group get free periods.
>
> That's the design,,
>
>> I believe the purpose of cpu burst is to help handling bursty worklod.
>> But if one task group can get computting power more than its quota for
>> persistent periods even there is no bursty workload, it's kinda broke.
>
> So if that was were bursty, it could consume that 100ms extra in a
> single go and that would be fine, but spreading that same amount over 20
> periods is somehow a problem? -- even though the interference is less.
>
The key thought I make the change is that If the spreading of burst
power always happen, it indicates the quota is not comfortable, the
better way is to do quota re-config for the container, but not always
get extra power from spreading the burst part which is not in the
consideration of high level container dispatcher such as k8s scheduler.
Container dispatcher might dispatch jobs oversale. The containers get
more power from the burst spreading is outside the sense of the
dispatcher. It might mislead the estimation of the entire ability of the
host.
IMO, cpu burst should be focus on the real burst workload, sharp and
short term.
Well, if 2 periods are a bit short for some huge cpu calculation jobs,
maybe we can add an option to define the burstable periods to let the
user make the decision based on the workload if you think this limit
periods idea make sense.
Thanks,
Honglei
>> This patch limits the burst to 2 periods so that it won't break the
>> quota limit for long. Permitting 2 periods can help on the scenario that
>> periods refresh lands in the middle of a burst workload. With this, we
>> can give task group more cpu burst power to handle the real burst
>> workload and don't worry about the 'stealing'.
>
> I've yet so see an actual reason for any of this...
>