From: Tuukka Tikkanen <[email protected]>
This series of patches fixes some bugs, style issues and wild use of
variable types in cpuidle menu governor. One of the bugs is a logic
flaw where a detected previously recurring pattern is given priority
over known guaranteed earlier wakeup. The others involve value
overflows.
Tuukka Tikkanen (8):
Cpuidle: Ignore interval prediction result when timer is shorter
Cpuidle: Rearrange code and comments in get_typical_interval()
Cpuidle: Check called function parameter in get_typical_interval()
Cpuidle: CodingStyle: Break up multiple assignments on single line
Cpuidle: Fix menu_device->intervals type
Cpuidle: Fix variable domains in get_typical_interval()
Cpuidle: Add a comment warning about possible overflow
Cpuidle: Change struct menu_device field types
drivers/cpuidle/governors/menu.c | 96 ++++++++++++++++++++++++++------------
1 file changed, 65 insertions(+), 31 deletions(-)
--
1.7.9.5
From: Tuukka Tikkanen <[email protected]>
The menu governor has a number of tunable constants that may be changed
in the source. If certain combination of values are chosen, an overflow
is possible when the correction_factor is being recalculated.
This patch adds a warning regarding this possibility and describes the
change needed for fixing the issue. The change should not be permanently
enabled, as it will hurt performance when it is not needed.
Signed-off-by: Tuukka Tikkanen <[email protected]>
---
drivers/cpuidle/governors/menu.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/drivers/cpuidle/governors/menu.c b/drivers/cpuidle/governors/menu.c
index 96fd10d..f277c13 100644
--- a/drivers/cpuidle/governors/menu.c
+++ b/drivers/cpuidle/governors/menu.c
@@ -21,6 +21,15 @@
#include <linux/math64.h>
#include <linux/module.h>
+/*
+ * Please note when changing the tuning values:
+ * If (MAX_INTERESTING-1) * RESOLUTION > ULONG_MAX, the result of
+ * a scaling operation multiplication may overflow on 32 bit platforms.
+ * In that case, #define RESOLUTION as ULL to get 64 bit result:
+ * #define RESOLUTION 1024ULL
+ *
+ * The default values do not overflow.
+ */
#define BUCKETS 12
#define INTERVALS 8
#define RESOLUTION 1024
--
1.7.9.5
From: Tuukka Tikkanen <[email protected]>
get_typical_interval() uses int_sqrt() in calculation of standard
deviation. The formal parameter of int_sqrt() is unsigned long, which
may on some platforms be smaller than the 64 bit unsigned integer used
as the actual parameter. The overflow can occur frequently when actual
idle period lengths are in hundreds of milliseconds.
This patch adds a check for such overflow and rejects the candidate
average when an overflow would occur.
Signed-off-by: Tuukka Tikkanen <[email protected]>
---
drivers/cpuidle/governors/menu.c | 18 +++++++++++++-----
1 file changed, 13 insertions(+), 5 deletions(-)
diff --git a/drivers/cpuidle/governors/menu.c b/drivers/cpuidle/governors/menu.c
index b13d57c..c24b10b 100644
--- a/drivers/cpuidle/governors/menu.c
+++ b/drivers/cpuidle/governors/menu.c
@@ -226,18 +226,26 @@ again:
}
}
do_div(stddev, divisor);
- stddev = int_sqrt(stddev);
/*
* The typical interval is obtained when standard deviation is small
* or standard deviation is small compared to the average interval.
*
+ * int_sqrt() formal parameter type is unsigned long. When the
+ * greatest difference to an outlier exceeds ~65 ms * sqrt(divisor)
+ * the resulting squared standard deviation exceeds the input domain
+ * of int_sqrt on platforms where unsigned long is 32 bits in size.
+ * In such case reject the candidate average.
+ *
* Use this result only if there is no timer to wake us up sooner.
*/
- if (((avg > stddev * 6) && (divisor * 4 >= INTERVALS * 3))
+ if (likely(stddev <= ULONG_MAX)) {
+ stddev = int_sqrt(stddev);
+ if (((avg > stddev * 6) && (divisor * 4 >= INTERVALS * 3))
|| stddev <= 20) {
- if (data->expected_us > avg)
- data->predicted_us = avg;
- return;
+ if (data->expected_us > avg)
+ data->predicted_us = avg;
+ return;
+ }
}
/*
--
1.7.9.5
From: Tuukka Tikkanen <[email protected]>
This patch prevents cpuidle menu governor from using repeating interval
prediction result if the idle period predicted is longer than the one
allowed by shortest running timer.
Signed-off-by: Tuukka Tikkanen <[email protected]>
---
drivers/cpuidle/governors/menu.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/drivers/cpuidle/governors/menu.c b/drivers/cpuidle/governors/menu.c
index bc580b6..c8ab1c5 100644
--- a/drivers/cpuidle/governors/menu.c
+++ b/drivers/cpuidle/governors/menu.c
@@ -238,10 +238,13 @@ again:
*
* The typical interval is obtained when standard deviation is small
* or standard deviation is small compared to the average interval.
+ *
+ * Use this result only if there is no timer to wake us up sooner.
*/
if (((avg > stddev * 6) && (divisor * 4 >= INTERVALS * 3))
|| stddev <= 20) {
- data->predicted_us = avg;
+ if (data->expected_us > avg)
+ data->predicted_us = avg;
return;
} else if ((divisor * 4) > INTERVALS * 3) {
--
1.7.9.5
From: Tuukka Tikkanen <[email protected]>
This patch rearranges a if-return-elsif-goto-fi-return sequence into
if-return-fi-if-return-fi-goto sequence. The functionality remains the
same. Also, a lengthy comment that did not describe the functionality
in the order it occurs is split into half and top half is moved closer
to actual implementation it describes.
Signed-off-by: Tuukka Tikkanen <[email protected]>
---
drivers/cpuidle/governors/menu.c | 28 +++++++++++++++-------------
1 file changed, 15 insertions(+), 13 deletions(-)
diff --git a/drivers/cpuidle/governors/menu.c b/drivers/cpuidle/governors/menu.c
index c8ab1c5..b13d57c 100644
--- a/drivers/cpuidle/governors/menu.c
+++ b/drivers/cpuidle/governors/menu.c
@@ -228,14 +228,6 @@ again:
do_div(stddev, divisor);
stddev = int_sqrt(stddev);
/*
- * If we have outliers to the upside in our distribution, discard
- * those by setting the threshold to exclude these outliers, then
- * calculate the average and standard deviation again. Once we get
- * down to the bottom 3/4 of our samples, stop excluding samples.
- *
- * This can deal with workloads that have long pauses interspersed
- * with sporadic activity with a bunch of short pauses.
- *
* The typical interval is obtained when standard deviation is small
* or standard deviation is small compared to the average interval.
*
@@ -246,12 +238,22 @@ again:
if (data->expected_us > avg)
data->predicted_us = avg;
return;
-
- } else if ((divisor * 4) > INTERVALS * 3) {
- /* Exclude the max interval */
- thresh = max - 1;
- goto again;
}
+
+ /*
+ * If we have outliers to the upside in our distribution, discard
+ * those by setting the threshold to exclude these outliers, then
+ * calculate the average and standard deviation again. Once we get
+ * down to the bottom 3/4 of our samples, stop excluding samples.
+ *
+ * This can deal with workloads that have long pauses interspersed
+ * with sporadic activity with a bunch of short pauses.
+ */
+ if ((divisor * 4) <= INTERVALS * 3)
+ return;
+
+ thresh = max - 1;
+ goto again;
}
/**
--
1.7.9.5
From: Tuukka Tikkanen <[email protected]>
The function get_typical_interval() initializes a number of variables
that are immediately after declarations assigned constant values.
In addition, there are multiple assignments on a single line, which
is explicitly forbidden by Documentation/CodingStyle.
This patch removes redundant initial values for the variables and
breaks up the multiple assignment line.
Signed-off-by: Tuukka Tikkanen <[email protected]>
---
drivers/cpuidle/governors/menu.c | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/drivers/cpuidle/governors/menu.c b/drivers/cpuidle/governors/menu.c
index c24b10b..fe2dd04 100644
--- a/drivers/cpuidle/governors/menu.c
+++ b/drivers/cpuidle/governors/menu.c
@@ -199,14 +199,17 @@ static u64 div_round64(u64 dividend, u32 divisor)
*/
static void get_typical_interval(struct menu_device *data)
{
- int i = 0, divisor = 0;
- uint64_t max = 0, avg = 0, stddev = 0;
+ int i, divisor;
+ uint64_t max, avg, stddev;
int64_t thresh = LLONG_MAX; /* Discard outliers above this value. */
again:
/* first calculate average and standard deviation of the past */
- max = avg = divisor = stddev = 0;
+ max = 0;
+ avg = 0;
+ divisor = 0;
+ stddev = 0;
for (i = 0; i < INTERVALS; i++) {
int64_t value = data->intervals[i];
if (value <= thresh) {
--
1.7.9.5
From: Tuukka Tikkanen <[email protected]>
Struct menu_device member intervals is declared as u32, but the value
stored is (unsigned) int. The type is changed to match the value being
stored.
Signed-off-by: Tuukka Tikkanen <[email protected]>
---
drivers/cpuidle/governors/menu.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/cpuidle/governors/menu.c b/drivers/cpuidle/governors/menu.c
index fe2dd04..0b37ee4 100644
--- a/drivers/cpuidle/governors/menu.c
+++ b/drivers/cpuidle/governors/menu.c
@@ -118,7 +118,7 @@ struct menu_device {
unsigned int exit_us;
unsigned int bucket;
u64 correction_factor[BUCKETS];
- u32 intervals[INTERVALS];
+ unsigned int intervals[INTERVALS];
int interval_ptr;
};
--
1.7.9.5
From: Tuukka Tikkanen <[email protected]>
Field predicted_us value can never exceed expected_us value, but it has
a potentially larger type. As there is no need for additional 32 bits of
zeroes on 32 bit plaforms, change the type of predicted_us to match the
type of expected_us.
Field correction_factor is used to store a value that cannot exceed the
product of RESOLUTION and DECAY (default 1024*8 = 8192). The constants
cannot in practice be incremented to such values, that they'd overflow
unsigned int even on 32 bit systems, so the type is changed to avoid
unnecessary 64 bit arithmetic on 32 bit systems.
One multiplication of (now) 32 bit values needs an added cast to avoid
truncation of the result and has been added.
In order to avoid another multiplication from 32 bit domain to 64 bit
domain, the new correction_factor calculation has been changed from
new = old * (DECAY-1) / DECAY
to
new = old - old / DECAY,
which with infinite precision would yeild exactly the same result, but
now changes the direction of rounding. The impact is not significant as
the maximum accumulated difference cannot exceed the value of DECAY,
which is relatively small compared to product of RESOLUTION and DECAY
(8 / 8192).
Signed-off-by: Tuukka Tikkanen <[email protected]>
---
drivers/cpuidle/governors/menu.c | 28 +++++++++++++++++-----------
1 file changed, 17 insertions(+), 11 deletions(-)
diff --git a/drivers/cpuidle/governors/menu.c b/drivers/cpuidle/governors/menu.c
index f277c13..cbd9b6c 100644
--- a/drivers/cpuidle/governors/menu.c
+++ b/drivers/cpuidle/governors/menu.c
@@ -123,10 +123,10 @@ struct menu_device {
int needs_update;
unsigned int expected_us;
- u64 predicted_us;
+ unsigned int predicted_us;
unsigned int exit_us;
unsigned int bucket;
- u64 correction_factor[BUCKETS];
+ unsigned int correction_factor[BUCKETS];
unsigned int intervals[INTERVALS];
int interval_ptr;
};
@@ -321,8 +321,13 @@ static int menu_select(struct cpuidle_driver *drv, struct cpuidle_device *dev)
if (data->correction_factor[data->bucket] == 0)
data->correction_factor[data->bucket] = RESOLUTION * DECAY;
- /* Make sure to round up for half microseconds */
- data->predicted_us = div_round64(data->expected_us * data->correction_factor[data->bucket],
+ /*
+ * Force the result of multiplication to be 64 bits even if both
+ * operands are 32 bits.
+ * Make sure to round up for half microseconds.
+ */
+ data->predicted_us = div_round64((uint64_t)data->expected_us *
+ data->correction_factor[data->bucket],
RESOLUTION * DECAY);
get_typical_interval(data);
@@ -388,7 +393,7 @@ static void menu_update(struct cpuidle_driver *drv, struct cpuidle_device *dev)
unsigned int last_idle_us = cpuidle_get_last_residency(dev);
struct cpuidle_state *target = &drv->states[last_idx];
unsigned int measured_us;
- u64 new_factor;
+ unsigned int new_factor;
/*
* Ugh, this idle state doesn't support residency measurements, so we
@@ -409,10 +414,9 @@ static void menu_update(struct cpuidle_driver *drv, struct cpuidle_device *dev)
measured_us -= data->exit_us;
- /* update our correction ratio */
-
- new_factor = data->correction_factor[data->bucket]
- * (DECAY - 1) / DECAY;
+ /* Update our correction ratio */
+ new_factor = data->correction_factor[data->bucket];
+ new_factor -= new_factor / DECAY;
if (data->expected_us > 0 && measured_us < MAX_INTERESTING)
new_factor += RESOLUTION * measured_us / data->expected_us;
@@ -425,9 +429,11 @@ static void menu_update(struct cpuidle_driver *drv, struct cpuidle_device *dev)
/*
* We don't want 0 as factor; we always want at least
- * a tiny bit of estimated time.
+ * a tiny bit of estimated time. Fortunately, due to rounding,
+ * new_factor will stay nonzero regardless of measured_us values
+ * and the compiler can eliminate this test as long as DECAY > 1.
*/
- if (new_factor == 0)
+ if (DECAY == 1 && unlikely(new_factor == 0))
new_factor = 1;
data->correction_factor[data->bucket] = new_factor;
--
1.7.9.5
From: Tuukka Tikkanen <[email protected]>
The menu governor uses a static function get_typical_interval() to
try to detect a repeating pattern of wakeups. The previous interval
durations are stored as an array of unsigned ints, but the arithmetic
in the function is performed exclusively as 64 bit values, even when
the value stored in a variable is known not to exceed unsigned int,
which may be smaller and more efficient on some platforms.
This patch changes the types of varibles used to store some
intermediates, the maximum and and the cutoff threshold to unsigned
ints. Average and standard deviation are still treated as 64 bit values,
even when the values are known to be within the domain of unsigned int,
to avoid casts to ensure correct integer promotion for arithmetic
operations.
Signed-off-by: Tuukka Tikkanen <[email protected]>
---
drivers/cpuidle/governors/menu.c | 15 +++++++++------
1 file changed, 9 insertions(+), 6 deletions(-)
diff --git a/drivers/cpuidle/governors/menu.c b/drivers/cpuidle/governors/menu.c
index 0b37ee4..96fd10d 100644
--- a/drivers/cpuidle/governors/menu.c
+++ b/drivers/cpuidle/governors/menu.c
@@ -200,18 +200,19 @@ static u64 div_round64(u64 dividend, u32 divisor)
static void get_typical_interval(struct menu_device *data)
{
int i, divisor;
- uint64_t max, avg, stddev;
- int64_t thresh = LLONG_MAX; /* Discard outliers above this value. */
+ unsigned int max, thresh;
+ uint64_t avg, stddev;
+
+ thresh = ULONG_MAX; /* Discard outliers above this value */
again:
- /* first calculate average and standard deviation of the past */
+ /* First calculate the average of past intervals */
max = 0;
avg = 0;
divisor = 0;
- stddev = 0;
for (i = 0; i < INTERVALS; i++) {
- int64_t value = data->intervals[i];
+ unsigned int value = data->intervals[i];
if (value <= thresh) {
avg += value;
divisor++;
@@ -221,8 +222,10 @@ again:
}
do_div(avg, divisor);
+ /* Then try to determine standard deviation */
+ stddev = 0;
for (i = 0; i < INTERVALS; i++) {
- int64_t value = data->intervals[i];
+ unsigned int value = data->intervals[i];
if (value <= thresh) {
int64_t diff = value - avg;
stddev += diff * diff;
--
1.7.9.5