This patchset is rebased upon v5.11-rc6.
Previous archive:
https://lore.kernel.org/lkml/[email protected]/
The CFS bandwidth controller limits CPU requests of a task group to
quota during each period. However, parallel workloads might be bursty
so that they get throttled. And they are latency sensitive at the same
time so that throttling them is undesired.
Scaling up period and quota allows greater burst capacity. But it might
cause longer stuck till next refill. We introduce "burst" to allow
accumulating unused quota from previous periods, and to be assigned when
a task group requests more CPU than quota during a specific period. Thus
allowing CPU time requests as long as the average requested CPU time is
below quota on the long run. The maximum accumulation is capped by burst
and is set 0 by default, thus the traditional behaviour remains.
A huge drop of 99th tail latency from more than 500ms to 27ms is seen for
real java workloads when using burst. Similar drops are seen when
testing with schbench too:
echo $$ > /sys/fs/cgroup/cpu/test/cgroup.procs
echo 700000 > /sys/fs/cgroup/cpu/test/cpu.cfs_quota_us
echo 100000 > /sys/fs/cgroup/cpu/test/cpu.cfs_period_us
echo 400000 > /sys/fs/cgroup/cpu/test/cpu.cfs_burst_us
# The average CPU usage is around 500%, which is 200ms CPU time
# every 40ms.
./schbench -m 1 -t 30 -r 60 -c 10000 -R 500
Without burst:
Latency percentiles (usec)
50.0000th: 7
75.0000th: 8
90.0000th: 9
95.0000th: 10
*99.0000th: 933
99.5000th: 981
99.9000th: 3068
min=0, max=20054
rps: 498.31 p95 (usec) 10 p99 (usec) 933 p95/cputime 0.10% p99/cputime 9.33%
With burst:
Latency percentiles (usec)
50.0000th: 7
75.0000th: 8
90.0000th: 9
95.0000th: 9
*99.0000th: 12
99.5000th: 13
99.9000th: 19
min=0, max=406
rps: 498.36 p95 (usec) 9 p99 (usec) 12 p95/cputime 0.09% p99/cputime 0.12%
How much workloads with benefit from burstable CFS bandwidth control
depends on how bursty and how latency sensitive they are.
Previously, Cong Wang and Konstantin Khlebnikov proposed similar
feature:
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/157476581065.5793.4518979877345136813.stgit@buzz/
This time we present more latency statistics and handle overflow while
accumulating.
Huaixin Chang (4):
sched/fair: Introduce primitives for CFS bandwidth burst
sched/fair: Make CFS bandwidth controller burstable
sched/fair: Add cfs bandwidth burst statistics
sched/fair: Add document for burstable CFS bandwidth control
Documentation/scheduler/sched-bwc.rst | 49 +++++++++++--
include/linux/sched/sysctl.h | 2 +
kernel/sched/core.c | 126 +++++++++++++++++++++++++++++-----
kernel/sched/fair.c | 58 +++++++++++++---
kernel/sched/sched.h | 9 ++-
kernel/sysctl.c | 18 +++++
6 files changed, 232 insertions(+), 30 deletions(-)
--
2.14.4.44.g2045bb6
Introduce statistics exports for the burstable cfs bandwidth
controller.
The following exports are included:
current_bw: current runtime in global pool
nr_burst: number of periods bandwidth burst occurs
burst_time: cumulative wall-time that any cpus has
used above quota in respective periods
Signed-off-by: Huaixin Chang <[email protected]>
Signed-off-by: Shanpei Chen <[email protected]>
---
kernel/sched/core.c | 6 ++++++
kernel/sched/fair.c | 12 +++++++++++-
kernel/sched/sched.h | 3 +++
3 files changed, 20 insertions(+), 1 deletion(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 9f1b05ad0411..d253903dbb4e 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -8681,6 +8681,8 @@ static int tg_set_cfs_bandwidth(struct task_group *tg, u64 period, u64 quota,
cfs_b->runtime = min(max_cfs_runtime, cfs_b->runtime);
}
+ cfs_b->previous_runtime = cfs_b->runtime;
+
/* Restart the period timer (if active) to handle new period expiry: */
if (runtime_enabled)
start_cfs_bandwidth(cfs_b, 1);
@@ -8929,6 +8931,10 @@ static int cpu_cfs_stat_show(struct seq_file *sf, void *v)
seq_printf(sf, "wait_sum %llu\n", ws);
}
+ seq_printf(sf, "current_bw %llu\n", cfs_b->runtime);
+ seq_printf(sf, "nr_burst %d\n", cfs_b->nr_burst);
+ seq_printf(sf, "burst_time %llu\n", cfs_b->burst_time);
+
return 0;
}
#endif /* CONFIG_CFS_BANDWIDTH */
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 6a7c261d206a..4cd3dc16659c 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4612,7 +4612,7 @@ static inline u64 sched_cfs_bandwidth_slice(void)
static void __refill_cfs_bandwidth_runtime(struct cfs_bandwidth *cfs_b,
u64 overrun)
{
- u64 refill;
+ u64 refill, runtime;
if (cfs_b->quota != RUNTIME_INF) {
@@ -4621,10 +4621,20 @@ static void __refill_cfs_bandwidth_runtime(struct cfs_bandwidth *cfs_b,
return;
}
+ if (cfs_b->previous_runtime > cfs_b->runtime) {
+ runtime = cfs_b->previous_runtime - cfs_b->runtime;
+ if (runtime > cfs_b->quota) {
+ cfs_b->burst_time += runtime - cfs_b->quota;
+ cfs_b->nr_burst++;
+ }
+ }
+
overrun = min(overrun, cfs_b->max_overrun);
refill = cfs_b->quota * overrun;
cfs_b->runtime += refill;
cfs_b->runtime = min(cfs_b->runtime, cfs_b->buffer);
+
+ cfs_b->previous_runtime = cfs_b->runtime;
}
}
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 4f53ea8e92ce..04b0a1ce4c89 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -360,6 +360,7 @@ struct cfs_bandwidth {
u64 burst;
u64 buffer;
u64 max_overrun;
+ u64 previous_runtime;
s64 hierarchical_quota;
u8 idle;
@@ -372,7 +373,9 @@ struct cfs_bandwidth {
/* Statistics: */
int nr_periods;
int nr_throttled;
+ int nr_burst;
u64 throttled_time;
+ u64 burst_time;
#endif
};
--
2.14.4.44.g2045bb6
In this patch, we introduce the notion of CFS bandwidth burst. Unused
"quota" from pervious "periods" might be accumulated and used in the
following "periods". The maximum amount of accumulated bandwidth is
bounded by "burst". And the maximun amount of CPU a group can consume in
a given period is "buffer" which is equivalent to "quota" + "burst in
case that this group has done enough accumulation.
Signed-off-by: Huaixin Chang <[email protected]>
Signed-off-by: Shanpei Chen <[email protected]>
---
kernel/sched/core.c | 91 ++++++++++++++++++++++++++++++++++++++++++++--------
kernel/sched/fair.c | 2 ++
kernel/sched/sched.h | 2 ++
3 files changed, 82 insertions(+), 13 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index ff74fca39ed2..28e3165c685b 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -8590,10 +8590,12 @@ static const u64 max_cfs_runtime = MAX_BW * NSEC_PER_USEC;
static int __cfs_schedulable(struct task_group *tg, u64 period, u64 runtime);
-static int tg_set_cfs_bandwidth(struct task_group *tg, u64 period, u64 quota)
+static int tg_set_cfs_bandwidth(struct task_group *tg, u64 period, u64 quota,
+ u64 burst)
{
int i, ret = 0, runtime_enabled, runtime_was_enabled;
struct cfs_bandwidth *cfs_b = &tg->cfs_bandwidth;
+ u64 buffer;
if (tg == &root_task_group)
return -EINVAL;
@@ -8620,6 +8622,16 @@ static int tg_set_cfs_bandwidth(struct task_group *tg, u64 period, u64 quota)
if (quota != RUNTIME_INF && quota > max_cfs_runtime)
return -EINVAL;
+ /*
+ * Bound burst to defend burst against overflow during bandwidth shift.
+ */
+ if (burst > max_cfs_runtime)
+ return -EINVAL;
+
+ if (quota == RUNTIME_INF)
+ buffer = RUNTIME_INF;
+ else
+ buffer = min(max_cfs_runtime, quota + burst);
/*
* Prevent race between setting of cfs_rq->runtime_enabled and
* unthrottle_offline_cfs_rqs().
@@ -8641,6 +8653,8 @@ static int tg_set_cfs_bandwidth(struct task_group *tg, u64 period, u64 quota)
raw_spin_lock_irq(&cfs_b->lock);
cfs_b->period = ns_to_ktime(period);
cfs_b->quota = quota;
+ cfs_b->burst = burst;
+ cfs_b->buffer = buffer;
__refill_cfs_bandwidth_runtime(cfs_b);
@@ -8674,9 +8688,10 @@ static int tg_set_cfs_bandwidth(struct task_group *tg, u64 period, u64 quota)
static int tg_set_cfs_quota(struct task_group *tg, long cfs_quota_us)
{
- u64 quota, period;
+ u64 quota, period, burst;
period = ktime_to_ns(tg->cfs_bandwidth.period);
+ burst = tg->cfs_bandwidth.burst;
if (cfs_quota_us < 0)
quota = RUNTIME_INF;
else if ((u64)cfs_quota_us <= U64_MAX / NSEC_PER_USEC)
@@ -8684,7 +8699,7 @@ static int tg_set_cfs_quota(struct task_group *tg, long cfs_quota_us)
else
return -EINVAL;
- return tg_set_cfs_bandwidth(tg, period, quota);
+ return tg_set_cfs_bandwidth(tg, period, quota, burst);
}
static long tg_get_cfs_quota(struct task_group *tg)
@@ -8702,15 +8717,16 @@ static long tg_get_cfs_quota(struct task_group *tg)
static int tg_set_cfs_period(struct task_group *tg, long cfs_period_us)
{
- u64 quota, period;
+ u64 quota, period, burst;
if ((u64)cfs_period_us > U64_MAX / NSEC_PER_USEC)
return -EINVAL;
period = (u64)cfs_period_us * NSEC_PER_USEC;
quota = tg->cfs_bandwidth.quota;
+ burst = tg->cfs_bandwidth.burst;
- return tg_set_cfs_bandwidth(tg, period, quota);
+ return tg_set_cfs_bandwidth(tg, period, quota, burst);
}
static long tg_get_cfs_period(struct task_group *tg)
@@ -8723,6 +8739,35 @@ static long tg_get_cfs_period(struct task_group *tg)
return cfs_period_us;
}
+static int tg_set_cfs_burst(struct task_group *tg, long cfs_burst_us)
+{
+ u64 quota, period, burst;
+
+ period = ktime_to_ns(tg->cfs_bandwidth.period);
+ quota = tg->cfs_bandwidth.quota;
+ if (cfs_burst_us < 0)
+ burst = RUNTIME_INF;
+ else if ((u64)cfs_burst_us <= U64_MAX / NSEC_PER_USEC)
+ burst = (u64)cfs_burst_us * NSEC_PER_USEC;
+ else
+ return -EINVAL;
+
+ return tg_set_cfs_bandwidth(tg, period, quota, burst);
+}
+
+static long tg_get_cfs_burst(struct task_group *tg)
+{
+ u64 burst_us;
+
+ if (tg->cfs_bandwidth.burst == RUNTIME_INF)
+ return -1;
+
+ burst_us = tg->cfs_bandwidth.burst;
+ do_div(burst_us, NSEC_PER_USEC);
+
+ return burst_us;
+}
+
static s64 cpu_cfs_quota_read_s64(struct cgroup_subsys_state *css,
struct cftype *cft)
{
@@ -8747,6 +8792,18 @@ static int cpu_cfs_period_write_u64(struct cgroup_subsys_state *css,
return tg_set_cfs_period(css_tg(css), cfs_period_us);
}
+static s64 cpu_cfs_burst_read_s64(struct cgroup_subsys_state *css,
+ struct cftype *cft)
+{
+ return tg_get_cfs_burst(css_tg(css));
+}
+
+static int cpu_cfs_burst_write_s64(struct cgroup_subsys_state *css,
+ struct cftype *cftype, s64 cfs_burst_us)
+{
+ return tg_set_cfs_burst(css_tg(css), cfs_burst_us);
+}
+
struct cfs_schedulable_data {
struct task_group *tg;
u64 period, quota;
@@ -8899,6 +8956,11 @@ static struct cftype cpu_legacy_files[] = {
.read_u64 = cpu_cfs_period_read_u64,
.write_u64 = cpu_cfs_period_write_u64,
},
+ {
+ .name = "cfs_burst_us",
+ .read_s64 = cpu_cfs_burst_read_s64,
+ .write_s64 = cpu_cfs_burst_write_s64,
+ },
{
.name = "stat",
.seq_show = cpu_cfs_stat_show,
@@ -9019,26 +9081,27 @@ static int cpu_weight_nice_write_s64(struct cgroup_subsys_state *css,
#endif
static void __maybe_unused cpu_period_quota_print(struct seq_file *sf,
- long period, long quota)
+ long period, long quota, long burst)
{
if (quota < 0)
seq_puts(sf, "max");
else
seq_printf(sf, "%ld", quota);
- seq_printf(sf, " %ld\n", period);
+ seq_printf(sf, " %ld %ld\n", period, burst);
}
-/* caller should put the current value in *@periodp before calling */
+/* caller should put the current value in *@periodp and *@burstp before calling */
static int __maybe_unused cpu_period_quota_parse(char *buf,
- u64 *periodp, u64 *quotap)
+ u64 *periodp, u64 *quotap, u64 *burstp)
{
char tok[21]; /* U64_MAX */
- if (sscanf(buf, "%20s %llu", tok, periodp) < 1)
+ if (sscanf(buf, "%20s %llu %llu", tok, periodp, burstp) < 1)
return -EINVAL;
*periodp *= NSEC_PER_USEC;
+ *burstp *= NSEC_PER_USEC;
if (sscanf(tok, "%llu", quotap))
*quotap *= NSEC_PER_USEC;
@@ -9055,7 +9118,8 @@ static int cpu_max_show(struct seq_file *sf, void *v)
{
struct task_group *tg = css_tg(seq_css(sf));
- cpu_period_quota_print(sf, tg_get_cfs_period(tg), tg_get_cfs_quota(tg));
+ cpu_period_quota_print(sf, tg_get_cfs_period(tg), tg_get_cfs_quota(tg),
+ tg_get_cfs_burst(tg));
return 0;
}
@@ -9064,12 +9128,13 @@ static ssize_t cpu_max_write(struct kernfs_open_file *of,
{
struct task_group *tg = css_tg(of_css(of));
u64 period = tg_get_cfs_period(tg);
+ u64 burst = tg_get_cfs_burst(tg);
u64 quota;
int ret;
- ret = cpu_period_quota_parse(buf, &period, "a);
+ ret = cpu_period_quota_parse(buf, &period, "a, &burst);
if (!ret)
- ret = tg_set_cfs_bandwidth(tg, period, quota);
+ ret = tg_set_cfs_bandwidth(tg, period, quota, burst);
return ret ?: nbytes;
}
#endif
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 04a3ce20da67..46945349f209 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -5262,6 +5262,8 @@ void init_cfs_bandwidth(struct cfs_bandwidth *cfs_b)
cfs_b->runtime = 0;
cfs_b->quota = RUNTIME_INF;
cfs_b->period = ns_to_ktime(default_cfs_period());
+ cfs_b->burst = 0;
+ cfs_b->buffer = RUNTIME_INF;
INIT_LIST_HEAD(&cfs_b->throttled_cfs_rq);
hrtimer_init(&cfs_b->period_timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_PINNED);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index bb09988451a0..2c0d8469c0fb 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -357,6 +357,8 @@ struct cfs_bandwidth {
ktime_t period;
u64 quota;
u64 runtime;
+ u64 burst;
+ u64 buffer;
s64 hierarchical_quota;
u8 idle;
--
2.14.4.44.g2045bb6
Basic description of usage and effect for CFS Bandwidth Control Burst.
Signed-off-by: Huaixin Chang <[email protected]>
Signed-off-by: Shanpei Chen <[email protected]>
---
Documentation/scheduler/sched-bwc.rst | 70 +++++++++++++++++++++++++++++++++--
1 file changed, 66 insertions(+), 4 deletions(-)
diff --git a/Documentation/scheduler/sched-bwc.rst b/Documentation/scheduler/sched-bwc.rst
index 9801d6b284b1..0933c66cc68b 100644
--- a/Documentation/scheduler/sched-bwc.rst
+++ b/Documentation/scheduler/sched-bwc.rst
@@ -21,18 +21,46 @@ cfs_quota units at each period boundary. As threads consume this bandwidth it
is transferred to cpu-local "silos" on a demand basis. The amount transferred
within each of these updates is tunable and described as the "slice".
+By default, CPU bandwidth consumption is strictly limited to quota within each
+given period. For the sequence of CPU usage u_i served under CFS bandwidth
+control, if for any j <= k N(j,k) is the number of periods from u_j to u_k:
+
+ u_j+...+u_k <= quota * N(j,k)
+
+For a bursty sequence among which interval u_j...u_k are at the peak, CPU
+requests might have to wait for more periods to replenish enough quota.
+Otherwise, larger quota is required.
+
+With "burst" buffer, CPU requests might be served as long as:
+
+ u_j+...+u_k <= B_j + quota * N(j,k)
+
+if for any j <= k N(j,k) is the number of periods from u_j to u_k and B_j is
+the accumulated quota from previous periods in burst buffer serving u_j.
+Burst buffer helps in that serving whole bursty CPU requests without throttling
+them can be done with moderate quota setting and accumulated quota in burst
+buffer, if:
+
+ u_0+...+u_n <= B_0 + quota * N(0,n)
+
+where B_0 is the initial state of burst buffer. The maximum accumulated quota in
+the burst buffer is capped by burst. With proper burst setting, the available
+bandwidth is still determined by quota and period on the long run.
+
Management
----------
-Quota and period are managed within the cpu subsystem via cgroupfs.
+Quota, period and burst are managed within the cpu subsystem via cgroupfs.
-cpu.cfs_quota_us: the total available run-time within a period (in microseconds)
+cpu.cfs_quota_us: run-time replenished within a period (in microseconds)
cpu.cfs_period_us: the length of a period (in microseconds)
+cpu.cfs_burst_us: the maximum accumulated run-time (in microseconds)
cpu.stat: exports throttling statistics [explained further below]
The default values are::
cpu.cfs_period_us=100ms
- cpu.cfs_quota=-1
+ cpu.cfs_quota_us=-1
+ cpu.cfs_burst_us=0
A value of -1 for cpu.cfs_quota_us indicates that the group does not have any
bandwidth restriction in place, such a group is described as an unconstrained
@@ -48,6 +76,11 @@ more detail below.
Writing any negative value to cpu.cfs_quota_us will remove the bandwidth limit
and return the group to an unconstrained state once more.
+A value of 0 for cpu.cfs_burst_us indicates that the group can not accumulate
+any unused bandwidth. It makes the traditional bandwidth control behavior for
+CFS unchanged. Writing any (valid) positive value(s) into cpu.cfs_burst_us
+will enact the cap on unused bandwidth accumulation.
+
Any updates to a group's bandwidth specification will result in it becoming
unthrottled if it is in a constrained state.
@@ -65,9 +98,21 @@ This is tunable via procfs::
Larger slice values will reduce transfer overheads, while smaller values allow
for more fine-grained consumption.
+There is also a global switch to turn off burst for all groups::
+ /proc/sys/kernel/sched_cfs_bw_burst_enabled (default=1)
+
+By default it is enabled. Writing a 0 value means no accumulated CPU time can be
+used for any group, even if cpu.cfs_burst_us is configured.
+
+Sometimes users might want a group to burst without accumulation. This is
+tunable via::
+ /proc/sys/kernel/sched_cfs_bw_burst_onset_percent (default=0)
+
+Up to 100% runtime of cpu.cfs_burst_us might be given on setting bandwidth.
+
Statistics
----------
-A group's bandwidth statistics are exported via 3 fields in cpu.stat.
+A group's bandwidth statistics are exported via 6 fields in cpu.stat.
cpu.stat:
@@ -75,6 +120,11 @@ cpu.stat:
- nr_throttled: Number of times the group has been throttled/limited.
- throttled_time: The total time duration (in nanoseconds) for which entities
of the group have been throttled.
+- current_bw: Current runtime in global pool.
+- nr_burst: Number of periods burst occurs.
+- burst_time: Cumulative wall-time that any CPUs has used above quota in
+ respective periods
+
This interface is read-only.
@@ -172,3 +222,15 @@ Examples
By using a small period here we are ensuring a consistent latency
response at the expense of burst capacity.
+
+4. Limit a group to 20% of 1 CPU, and allow accumulate up to 60% of 1 CPU
+ additionally, in case accumulation has been done.
+
+ With 50ms period, 10ms quota will be equivalent to 20% of 1 CPU.
+ And 30ms burst will be equivalent to 60% of 1 CPU.
+
+ # echo 10000 > cpu.cfs_quota_us /* quota = 10ms */
+ # echo 50000 > cpu.cfs_period_us /* period = 50ms */
+ # echo 30000 > cpu.cfs_burst_us /* burst = 30ms */
+
+ Larger buffer setting allows greater burst capacity.
--
2.14.4.44.g2045bb6