These patches introduce support for the Activity Monitors Unit (AMU)
CPU extension, an optional extension in ARMv8.4 CPUs. This provides
performance counters intended for system management use. Two of these
counters are then used to compute the frequency scale correction
factor needed to achieve frequency invariance.
With the CONFIG_ARM64_AMU_EXTN enabled the kernel is able to safely
run a mix of CPUs with and without support for the AMU extension.
The AMU capability is unconditionally enabled in the kernel as to
allow any late CPU to use the feature: the cpu_enable function will
be called for all CPUs that match the criteria, including secondary
and hotplugged CPUs, marking this feature as present on that
respective CPU.
To be noted that firmware must implement AMU support when running on
CPUs that present the activity monitors extension: allow access to
the registers from lower exception levels, enable the counters,
implement save and restore functionality. More details can be found
in the documentation.
Given that the activity counters inform on activity on the CPUs, and
that not all CPUs might implement the extension, for functional and
security reasons, it's best to disable access to the AMU registers
from userspace (EL0) and KVM guests.
In patch 6/7, two of the AMU counters are used to compute the frequency
scale factor needed to achieve frequency invariance of signals in the
scheduler, based on an interface added to support counter-based
frequency invariance - arch_scale_freq_tick. The interface and update
point for the counter-based frequency scale factor is based on the
similar approach in the patch that introduces frequency invariance
for x86 [1].
Testing:
- Build tested for multiple architectures and defconfigs.
- AMU feature detection, EL0 and KVM guest access to AMU registers,
feature support in firmware (version 1.5 and later of the ARM
Trusted Firmware) was tested on an Armv8-A Base Platform FVP:
Architecture Envelope Model [2] (supports version 8.0 to 8.5),
with the following configurations:
cluster0.has_arm_v8-4=1
cluster1.has_arm_v8-4=1
cluster0.has_amu=1
cluster1.has_amu=1
v5 -> v6
- v5 can be found at [7]
- The current series is based on arm64/master [8]. To be noted that
the use of counters for frequency invariance is dependent on a patch
in tip/sched/core [9] - it will build successfully without it but
counters will not be used for frequency invariance. This series
applies without issues on both [8] and [9] (or a merge of them)
- [1/7] kernel parameter removed as Catalin recommended
- [2/7, 4/7] changelog and documentation changed to remove information
about the removed kernel parameter
- [2/7, 3/7, 6/7] collected reviewed-by and acked-by from James,
Marc and Lukasz.
v4 -> v5:
- v4 can be found at [6]
- [1/7] rebased on top of latest tip/sched/core and fixed conflicts;
applied Reviewed-by from Valentin;
- [5/7] applied Reviewed-by from Valentin
- [6/7] applied Valentin and Pavan's suggested fixes; reworked
arch_cpu_freq_counters (now arch_freq_counters_available) and
confined it to the arch topology driver as recommended by Valentin
and Lukasz;
v3 -> v4:
- v3 can be found at [5]
- [1/7] renamed and changed format for disable_amu - now amu=<val> as
Suzuki and Vladimir recommended; removed dynamic allocation for
amu_cpus as Suzuki recommended;
- [2-4/7] collected Reviewed-by
- [5/7] modified changelog and collected Acked-by
- [6/7] removed cpu_get_max_freq as Lukasz recommended; improved debug
messages, warnings, and comments, added use of static key, replaced
validation for possible cpus with filterning of present CPUs - as per
Valentin's comments.
- [7/7] modified changelog to avoid confusion related to impact on
activity monitors use and collected Acked-by
v2 -> v3:
- v2 can be found at [4]
- [1/7] used cpumask instead of per-cpu variable to flag AMU presence
as; introduced disable_amu kernel parameter; removed ftr_id_pfr0 AMU
bits - recommended by Suzuki.
- [2/7] replaced obscure label as recommended by Valentin.
- [3/7] clarified activate_traps_vhe comment
- [4/7] dropped changes in arm64/cpu-feature-registers.txt; removed
use of variable names - recommended by Suzuki
- previous [5/6] - dropped as [1] as added to tip/sched/core
- [5/7] new patch introduced to cleanly obtain maximum hardware
frequency from cpufreq
- [6/7] (previously [6/6]):
- Removed use of workqueues by limiting the validation work done on
each cpu to the setting of the reference per-cpu counter variables.
This is now called directly from cpu_enable (cpufeature.c). Also,
further CPU, policy and system validation is done in a
late_initcall_sync function - waits for deferred probe work to
finish as well to ensure the maximum frequency is set by either
cpufreq drivers or platform drivers - recommended by Lukasz.
- Improved AMU use detection for CPUs in arch_set_freq_scale -
recommended by Lukasz.
- Properly validated arch_max_freq_scale and added detailed
documentation for how arch_max_freq_scale and freq_scale are
obtained based on counters - recommended by Valentin.
- Overall - limited tight coupling between AMU use and cpufreq
(use of maximum frequency information and policy validation).
- [7/7] introduced patch to warn if arch_timer_rate is too low
- functionality provided by Valentin.
v1 -> v2:
- v1 can be found at [3]
- Added patches that use the counters for the scheduler's frequency
invariance engine
- In patch arm64: add support for the AMU extension v1 -
- Defined an accessor function cpu_has_amu_feat to allow a read
of amu_feat only from the current CPU, to ensure the safe use
of the per-cpu variable for the current user (arm64 topology
driver) and future users.
- Modified type of amu_feat from bool to u8 to satisfy sparse
checker's warning 'expression using sizeof _Bool [sparse]',
as the size of bool is compiler dependent.
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://developer.arm.com/tools-and-software/simulation-models/fixed-virtual-platforms
[3] https://lore.kernel.org/lkml/[email protected]/
[4] https://lore.kernel.org/lkml/[email protected]/
[5] https://lore.kernel.org/lkml/[email protected]/
[6] https://lore.kernel.org/lkml/[email protected]/
[7] https://lore.kernel.org/lkml/[email protected]/
[8] https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git/commit/?id=bfdc6d91a25f4545bcd1b12e3219af4838142ef1
[9] https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?h=sched/core&id=a0f03b617c3b2644d3d47bf7d9e60aed01bd5b10
Ionela Voinescu (7):
arm64: add support for the AMU extension v1
arm64: trap to EL1 accesses to AMU counters from EL0
arm64/kvm: disable access to AMU registers from kvm guests
Documentation: arm64: document support for the AMU extension
cpufreq: add function to get the hardware max frequency
arm64: use activity monitors for frequency invariance
clocksource/drivers/arm_arch_timer: validate arch_timer_rate
Documentation/arm64/amu.rst | 112 +++++++++++++++++
Documentation/arm64/booting.rst | 14 +++
Documentation/arm64/index.rst | 1 +
arch/arm64/Kconfig | 27 ++++
arch/arm64/include/asm/assembler.h | 10 ++
arch/arm64/include/asm/cpucaps.h | 3 +-
arch/arm64/include/asm/cpufeature.h | 5 +
arch/arm64/include/asm/kvm_arm.h | 1 +
arch/arm64/include/asm/sysreg.h | 38 ++++++
arch/arm64/include/asm/topology.h | 9 ++
arch/arm64/kernel/cpufeature.c | 70 +++++++++++
arch/arm64/kernel/topology.c | 180 +++++++++++++++++++++++++++
arch/arm64/kvm/hyp/switch.c | 14 ++-
arch/arm64/kvm/sys_regs.c | 93 +++++++++++++-
arch/arm64/mm/proc.S | 3 +
drivers/base/arch_topology.c | 12 ++
drivers/clocksource/arm_arch_timer.c | 18 ++-
drivers/cpufreq/cpufreq.c | 20 +++
include/linux/arch_topology.h | 2 +
include/linux/cpufreq.h | 5 +
20 files changed, 630 insertions(+), 7 deletions(-)
create mode 100644 Documentation/arm64/amu.rst
base-commit: bfdc6d91a25f4545bcd1b12e3219af4838142ef1
--
2.17.1
The Frequency Invariance Engine (FIE) is providing a frequency
scaling correction factor that helps achieve more accurate
load-tracking.
So far, for arm and arm64 platforms, this scale factor has been
obtained based on the ratio between the current frequency and the
maximum supported frequency recorded by the cpufreq policy. The
setting of this scale factor is triggered from cpufreq drivers by
calling arch_set_freq_scale. The current frequency used in computation
is the frequency requested by a governor, but it may not be the
frequency that was implemented by the platform.
This correction factor can also be obtained using a core counter and a
constant counter to get information on the performance (frequency based
only) obtained in a period of time. This will more accurately reflect
the actual current frequency of the CPU, compared with the alternative
implementation that reflects the request of a performance level from
the OS.
Therefore, implement arch_scale_freq_tick to use activity monitors, if
present, for the computation of the frequency scale factor.
The use of AMU counters depends on:
- CONFIG_ARM64_AMU_EXTN - depents on the AMU extension being present
- CONFIG_CPU_FREQ - the current frequency obtained using counter
information is divided by the maximum frequency obtained from the
cpufreq policy.
While it is possible to have a combination of CPUs in the system with
and without support for activity monitors, the use of counters for
frequency invariance is only enabled for a CPU if all related CPUs
(CPUs in the same frequency domain) support and have enabled the core
and constant activity monitor counters. In this way, there is a clear
separation between the policies for which arch_set_freq_scale (cpufreq
based FIE) is used, and the policies for which arch_scale_freq_tick
(counter based FIE) is used to set the frequency scale factor. For
this purpose, a late_initcall_sync is registered to trigger validation
work for policies that will enable or disable the use of AMU counters
for frequency invariance. If CONFIG_CPU_FREQ is not defined, the use
of counters is enabled on all CPUs only if all possible CPUs correctly
support the necessary counters.
Signed-off-by: Ionela Voinescu <[email protected]>
Reviewed-by: Lukasz Luba <[email protected]>
Cc: Catalin Marinas <[email protected]>
Cc: Will Deacon <[email protected]>
Cc: Sudeep Holla <[email protected]>
---
arch/arm64/include/asm/topology.h | 9 ++
arch/arm64/kernel/cpufeature.c | 4 +
arch/arm64/kernel/topology.c | 180 ++++++++++++++++++++++++++++++
drivers/base/arch_topology.c | 12 ++
include/linux/arch_topology.h | 2 +
5 files changed, 207 insertions(+)
diff --git a/arch/arm64/include/asm/topology.h b/arch/arm64/include/asm/topology.h
index a4d945db95a2..21d4d40d6243 100644
--- a/arch/arm64/include/asm/topology.h
+++ b/arch/arm64/include/asm/topology.h
@@ -16,6 +16,15 @@ int pcibus_to_node(struct pci_bus *bus);
#include <linux/arch_topology.h>
+#ifdef CONFIG_ARM64_AMU_EXTN
+/*
+ * Replace task scheduler's default counter-based
+ * frequency-invariance scale factor setting.
+ */
+void topology_scale_freq_tick(void);
+#define arch_scale_freq_tick topology_scale_freq_tick
+#endif /* CONFIG_ARM64_AMU_EXTN */
+
/* Replace task scheduler's default frequency-invariant accounting */
#define arch_scale_freq_capacity topology_get_freq_scale
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 301538d3a197..bf98a2386534 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -1241,12 +1241,16 @@ bool cpu_has_amu_feat(int cpu)
return cpumask_test_cpu(cpu, &amu_cpus);
}
+/* Initialize the use of AMU counters for frequency invariance */
+extern void init_cpu_freq_invariance_counters(void);
+
static void cpu_amu_enable(struct arm64_cpu_capabilities const *cap)
{
if (has_cpuid_feature(cap, SCOPE_LOCAL_CPU)) {
pr_info("detected CPU%d: Activity Monitors Unit (AMU)\n",
smp_processor_id());
cpumask_set_cpu(smp_processor_id(), &amu_cpus);
+ init_cpu_freq_invariance_counters();
}
}
diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel/topology.c
index fa9528dfd0ce..0801a0f3c156 100644
--- a/arch/arm64/kernel/topology.c
+++ b/arch/arm64/kernel/topology.c
@@ -14,6 +14,7 @@
#include <linux/acpi.h>
#include <linux/arch_topology.h>
#include <linux/cacheinfo.h>
+#include <linux/cpufreq.h>
#include <linux/init.h>
#include <linux/percpu.h>
@@ -120,4 +121,183 @@ int __init parse_acpi_topology(void)
}
#endif
+#ifdef CONFIG_ARM64_AMU_EXTN
+#undef pr_fmt
+#define pr_fmt(fmt) "AMU: " fmt
+
+static DEFINE_PER_CPU_READ_MOSTLY(unsigned long, arch_max_freq_scale);
+static DEFINE_PER_CPU(u64, arch_const_cycles_prev);
+static DEFINE_PER_CPU(u64, arch_core_cycles_prev);
+static cpumask_var_t amu_fie_cpus;
+
+/* Initialize counter reference per-cpu variables for the current CPU */
+void init_cpu_freq_invariance_counters(void)
+{
+ this_cpu_write(arch_core_cycles_prev,
+ read_sysreg_s(SYS_AMEVCNTR0_CORE_EL0));
+ this_cpu_write(arch_const_cycles_prev,
+ read_sysreg_s(SYS_AMEVCNTR0_CONST_EL0));
+}
+
+static int validate_cpu_freq_invariance_counters(int cpu)
+{
+ u64 max_freq_hz, ratio;
+
+ if (!cpu_has_amu_feat(cpu)) {
+ pr_debug("CPU%d: counters are not supported.\n", cpu);
+ return -EINVAL;
+ }
+
+ if (unlikely(!per_cpu(arch_const_cycles_prev, cpu) ||
+ !per_cpu(arch_core_cycles_prev, cpu))) {
+ pr_debug("CPU%d: cycle counters are not enabled.\n", cpu);
+ return -EINVAL;
+ }
+
+ /* Convert maximum frequency from KHz to Hz and validate */
+ max_freq_hz = cpufreq_get_hw_max_freq(cpu) * 1000;
+ if (unlikely(!max_freq_hz)) {
+ pr_debug("CPU%d: invalid maximum frequency.\n", cpu);
+ return -EINVAL;
+ }
+
+ /*
+ * Pre-compute the fixed ratio between the frequency of the constant
+ * counter and the maximum frequency of the CPU.
+ *
+ * const_freq
+ * arch_max_freq_scale = ---------------- * SCHED_CAPACITY_SCALE²
+ * cpuinfo_max_freq
+ *
+ * We use a factor of 2 * SCHED_CAPACITY_SHIFT -> SCHED_CAPACITY_SCALE²
+ * in order to ensure a good resolution for arch_max_freq_scale for
+ * very low arch timer frequencies (down to the KHz range which should
+ * be unlikely).
+ */
+ ratio = (u64)arch_timer_get_rate() << (2 * SCHED_CAPACITY_SHIFT);
+ ratio = div64_u64(ratio, max_freq_hz);
+ if (!ratio) {
+ WARN_ONCE(1, "System timer frequency too low.\n");
+ return -EINVAL;
+ }
+
+ per_cpu(arch_max_freq_scale, cpu) = (unsigned long)ratio;
+
+ return 0;
+}
+
+static inline bool
+enable_policy_freq_counters(int cpu, cpumask_var_t valid_cpus)
+{
+ struct cpufreq_policy *policy = cpufreq_cpu_get(cpu);
+
+ if (!policy) {
+ pr_debug("CPU%d: No cpufreq policy found.\n", cpu);
+ return false;
+ }
+
+ if (cpumask_subset(policy->related_cpus, valid_cpus))
+ cpumask_or(amu_fie_cpus, policy->related_cpus,
+ amu_fie_cpus);
+
+ cpufreq_cpu_put(policy);
+
+ return true;
+}
+
+static DEFINE_STATIC_KEY_FALSE(amu_fie_key);
+#define amu_freq_invariant() static_branch_unlikely(&amu_fie_key)
+
+static int __init init_amu_fie(void)
+{
+ cpumask_var_t valid_cpus;
+ bool have_policy = false;
+ int ret = 0;
+ int cpu;
+
+ if (!zalloc_cpumask_var(&valid_cpus, GFP_KERNEL))
+ return -ENOMEM;
+
+ if (!zalloc_cpumask_var(&amu_fie_cpus, GFP_KERNEL)) {
+ ret = -ENOMEM;
+ goto free_valid_mask;
+ }
+
+ for_each_present_cpu(cpu) {
+ if (validate_cpu_freq_invariance_counters(cpu))
+ continue;
+ cpumask_set_cpu(cpu, valid_cpus);
+ have_policy |= enable_policy_freq_counters(cpu, valid_cpus);
+ }
+
+ /*
+ * If we are not restricted by cpufreq policies, we only enable
+ * the use of the AMU feature for FIE if all CPUs support AMU.
+ * Otherwise, enable_policy_freq_counters has already enabled
+ * policy cpus.
+ */
+ if (!have_policy && cpumask_equal(valid_cpus, cpu_present_mask))
+ cpumask_or(amu_fie_cpus, amu_fie_cpus, valid_cpus);
+
+ if (!cpumask_empty(amu_fie_cpus)) {
+ pr_info("CPUs[%*pbl]: counters will be used for FIE.",
+ cpumask_pr_args(amu_fie_cpus));
+ static_branch_enable(&amu_fie_key);
+ }
+
+free_valid_mask:
+ free_cpumask_var(valid_cpus);
+
+ return ret;
+}
+late_initcall_sync(init_amu_fie);
+
+bool arch_freq_counters_available(struct cpumask *cpus)
+{
+ return amu_freq_invariant() &&
+ cpumask_subset(cpus, amu_fie_cpus);
+}
+
+void topology_scale_freq_tick(void)
+{
+ u64 prev_core_cnt, prev_const_cnt;
+ u64 core_cnt, const_cnt, scale;
+ int cpu = smp_processor_id();
+
+ if (!amu_freq_invariant())
+ return;
+
+ if (!cpumask_test_cpu(cpu, amu_fie_cpus))
+ return;
+
+ const_cnt = read_sysreg_s(SYS_AMEVCNTR0_CONST_EL0);
+ core_cnt = read_sysreg_s(SYS_AMEVCNTR0_CORE_EL0);
+ prev_const_cnt = this_cpu_read(arch_const_cycles_prev);
+ prev_core_cnt = this_cpu_read(arch_core_cycles_prev);
+
+ if (unlikely(core_cnt <= prev_core_cnt ||
+ const_cnt <= prev_const_cnt))
+ goto store_and_exit;
+
+ /*
+ * /\core arch_max_freq_scale
+ * scale = ------- * --------------------
+ * /\const SCHED_CAPACITY_SCALE
+ *
+ * See validate_cpu_freq_invariance_counters() for details on
+ * arch_max_freq_scale and the use of SCHED_CAPACITY_SHIFT.
+ */
+ scale = core_cnt - prev_core_cnt;
+ scale *= this_cpu_read(arch_max_freq_scale);
+ scale = div64_u64(scale >> SCHED_CAPACITY_SHIFT,
+ const_cnt - prev_const_cnt);
+
+ scale = min_t(unsigned long, scale, SCHED_CAPACITY_SCALE);
+ this_cpu_write(freq_scale, (unsigned long)scale);
+
+store_and_exit:
+ this_cpu_write(arch_core_cycles_prev, core_cnt);
+ this_cpu_write(arch_const_cycles_prev, const_cnt);
+}
+#endif /* CONFIG_ARM64_AMU_EXTN */
diff --git a/drivers/base/arch_topology.c b/drivers/base/arch_topology.c
index 6119e11a9f95..8d63673c1689 100644
--- a/drivers/base/arch_topology.c
+++ b/drivers/base/arch_topology.c
@@ -21,6 +21,10 @@
#include <linux/sched.h>
#include <linux/smp.h>
+__weak bool arch_freq_counters_available(struct cpumask *cpus)
+{
+ return false;
+}
DEFINE_PER_CPU(unsigned long, freq_scale) = SCHED_CAPACITY_SCALE;
void arch_set_freq_scale(struct cpumask *cpus, unsigned long cur_freq,
@@ -29,6 +33,14 @@ void arch_set_freq_scale(struct cpumask *cpus, unsigned long cur_freq,
unsigned long scale;
int i;
+ /*
+ * If the use of counters for FIE is enabled, just return as we don't
+ * want to update the scale factor with information from CPUFREQ.
+ * Instead the scale factor will be updated from arch_scale_freq_tick.
+ */
+ if (arch_freq_counters_available(cpus))
+ return;
+
scale = (cur_freq << SCHED_CAPACITY_SHIFT) / max_freq;
for_each_cpu(i, cpus)
diff --git a/include/linux/arch_topology.h b/include/linux/arch_topology.h
index 3015ecbb90b1..1ccdddb541a7 100644
--- a/include/linux/arch_topology.h
+++ b/include/linux/arch_topology.h
@@ -33,6 +33,8 @@ unsigned long topology_get_freq_scale(int cpu)
return per_cpu(freq_scale, cpu);
}
+bool arch_freq_counters_available(struct cpumask *cpus);
+
struct cpu_topology {
int thread_id;
int core_id;
--
2.17.1
Using an arch timer with a frequency of less than 1MHz can potentially
result in incorrect functionality in systems that assume a reasonable
rate of the arch timer of 1 to 50MHz, described as typical in the
architecture specification.
Therefore, warn if the arch timer rate is below 1MHz, which is
considered atypical and worth emphasizing.
Signed-off-by: Ionela Voinescu <[email protected]>
Suggested-by: Valentin Schneider <[email protected]>
Acked-by: Marc Zyngier <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: Marc Zyngier <[email protected]>
---
drivers/clocksource/arm_arch_timer.c | 18 +++++++++++++++---
1 file changed, 15 insertions(+), 3 deletions(-)
diff --git a/drivers/clocksource/arm_arch_timer.c b/drivers/clocksource/arm_arch_timer.c
index 9a5464c625b4..4faa930eabf8 100644
--- a/drivers/clocksource/arm_arch_timer.c
+++ b/drivers/clocksource/arm_arch_timer.c
@@ -885,6 +885,17 @@ static int arch_timer_starting_cpu(unsigned int cpu)
return 0;
}
+static int validate_timer_rate(void)
+{
+ if (!arch_timer_rate)
+ return -EINVAL;
+
+ /* Arch timer frequency < 1MHz can cause trouble */
+ WARN_ON(arch_timer_rate < 1000000);
+
+ return 0;
+}
+
/*
* For historical reasons, when probing with DT we use whichever (non-zero)
* rate was probed first, and don't verify that others match. If the first node
@@ -900,7 +911,7 @@ static void arch_timer_of_configure_rate(u32 rate, struct device_node *np)
arch_timer_rate = rate;
/* Check the timer frequency. */
- if (arch_timer_rate == 0)
+ if (validate_timer_rate())
pr_warn("frequency not available\n");
}
@@ -1594,9 +1605,10 @@ static int __init arch_timer_acpi_init(struct acpi_table_header *table)
* CNTFRQ value. This *must* be correct.
*/
arch_timer_rate = arch_timer_get_cntfrq();
- if (!arch_timer_rate) {
+ ret = validate_timer_rate();
+ if (ret) {
pr_err(FW_BUG "frequency not available.\n");
- return -EINVAL;
+ return ret;
}
arch_timer_uses_ppi = arch_timer_select_ppi();
--
2.17.1
Add weak function to return the hardware maximum frequency of a CPU,
with the default implementation returning cpuinfo.max_freq, which is
the best information we can generically get from the cpufreq framework.
The default can be overwritten by a strong function in platforms
that want to provide an alternative implementation, with more accurate
information, obtained either from hardware or firmware.
Signed-off-by: Ionela Voinescu <[email protected]>
Acked-by: Viresh Kumar <[email protected]>
Reviewed-by: Valentin Schneider <[email protected]>
Cc: Rafael J. Wysocki <[email protected]>
Cc: Viresh Kumar <[email protected]>
---
drivers/cpufreq/cpufreq.c | 20 ++++++++++++++++++++
include/linux/cpufreq.h | 5 +++++
2 files changed, 25 insertions(+)
diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c
index cbe6c94bf158..985228aee46f 100644
--- a/drivers/cpufreq/cpufreq.c
+++ b/drivers/cpufreq/cpufreq.c
@@ -1725,6 +1725,26 @@ unsigned int cpufreq_quick_get_max(unsigned int cpu)
}
EXPORT_SYMBOL(cpufreq_quick_get_max);
+/**
+ * cpufreq_get_hw_max_freq - get the max hardware frequency of the CPU
+ * @cpu: CPU number
+ *
+ * The default return value is the max_freq field of cpuinfo.
+ */
+__weak unsigned int cpufreq_get_hw_max_freq(unsigned int cpu)
+{
+ struct cpufreq_policy *policy = cpufreq_cpu_get(cpu);
+ unsigned int ret_freq = 0;
+
+ if (policy) {
+ ret_freq = policy->cpuinfo.max_freq;
+ cpufreq_cpu_put(policy);
+ }
+
+ return ret_freq;
+}
+EXPORT_SYMBOL(cpufreq_get_hw_max_freq);
+
static unsigned int __cpufreq_get(struct cpufreq_policy *policy)
{
if (unlikely(policy_is_inactive(policy)))
diff --git a/include/linux/cpufreq.h b/include/linux/cpufreq.h
index 0fb561d1b524..f7240251a949 100644
--- a/include/linux/cpufreq.h
+++ b/include/linux/cpufreq.h
@@ -205,6 +205,7 @@ static inline bool policy_is_shared(struct cpufreq_policy *policy)
unsigned int cpufreq_get(unsigned int cpu);
unsigned int cpufreq_quick_get(unsigned int cpu);
unsigned int cpufreq_quick_get_max(unsigned int cpu);
+unsigned int cpufreq_get_hw_max_freq(unsigned int cpu);
void disable_cpufreq(void);
u64 get_cpu_idle_time(unsigned int cpu, u64 *wall, int io_busy);
@@ -232,6 +233,10 @@ static inline unsigned int cpufreq_quick_get_max(unsigned int cpu)
{
return 0;
}
+static inline unsigned int cpufreq_get_hw_max_freq(unsigned int cpu)
+{
+ return 0;
+}
static inline void disable_cpufreq(void) { }
#endif
--
2.17.1
Hi Ionela,
On Thu, Mar 05, 2020 at 09:06:20AM +0000, Ionela Voinescu wrote:
> v5 -> v6
> - v5 can be found at [7]
> - The current series is based on arm64/master [8].
arm64/master is just some random point in the Linus tree, I didn't
realise people are using it. I moved it to v5.6-rc3 as that's the base
for the final arm64 for-next/core.
> - [1/7] kernel parameter removed as Catalin recommended
> - [2/7, 4/7] changelog and documentation changed to remove information
> about the removed kernel parameter
> - [2/7, 3/7, 6/7] collected reviewed-by and acked-by from James,
> Marc and Lukasz.
The series looks fine. Once Sudeep reviews/acks the arch topology patch,
I'll queue the patches.
Thanks.
--
Catalin
On Thu, Mar 05, 2020 at 09:06:26AM +0000, Ionela Voinescu wrote:
> The Frequency Invariance Engine (FIE) is providing a frequency
> scaling correction factor that helps achieve more accurate
> load-tracking.
>
> So far, for arm and arm64 platforms, this scale factor has been
> obtained based on the ratio between the current frequency and the
> maximum supported frequency recorded by the cpufreq policy. The
> setting of this scale factor is triggered from cpufreq drivers by
> calling arch_set_freq_scale. The current frequency used in computation
> is the frequency requested by a governor, but it may not be the
> frequency that was implemented by the platform.
>
> This correction factor can also be obtained using a core counter and a
> constant counter to get information on the performance (frequency based
> only) obtained in a period of time. This will more accurately reflect
> the actual current frequency of the CPU, compared with the alternative
> implementation that reflects the request of a performance level from
> the OS.
>
> Therefore, implement arch_scale_freq_tick to use activity monitors, if
> present, for the computation of the frequency scale factor.
>
> The use of AMU counters depends on:
> - CONFIG_ARM64_AMU_EXTN - depents on the AMU extension being present
> - CONFIG_CPU_FREQ - the current frequency obtained using counter
> information is divided by the maximum frequency obtained from the
> cpufreq policy.
>
> While it is possible to have a combination of CPUs in the system with
> and without support for activity monitors, the use of counters for
> frequency invariance is only enabled for a CPU if all related CPUs
> (CPUs in the same frequency domain) support and have enabled the core
> and constant activity monitor counters. In this way, there is a clear
> separation between the policies for which arch_set_freq_scale (cpufreq
> based FIE) is used, and the policies for which arch_scale_freq_tick
> (counter based FIE) is used to set the frequency scale factor. For
> this purpose, a late_initcall_sync is registered to trigger validation
> work for policies that will enable or disable the use of AMU counters
> for frequency invariance. If CONFIG_CPU_FREQ is not defined, the use
> of counters is enabled on all CPUs only if all possible CPUs correctly
> support the necessary counters.
>
> Signed-off-by: Ionela Voinescu <[email protected]>
> Reviewed-by: Lukasz Luba <[email protected]>
> Cc: Catalin Marinas <[email protected]>
> Cc: Will Deacon <[email protected]>
> Cc: Sudeep Holla <[email protected]>
> ---
> arch/arm64/include/asm/topology.h | 9 ++
> arch/arm64/kernel/cpufeature.c | 4 +
> arch/arm64/kernel/topology.c | 180 ++++++++++++++++++++++++++++++
> drivers/base/arch_topology.c | 12 ++
> include/linux/arch_topology.h | 2 +
> 5 files changed, 207 insertions(+)
>
[...]
> diff --git a/drivers/base/arch_topology.c b/drivers/base/arch_topology.c
> index 6119e11a9f95..8d63673c1689 100644
> --- a/drivers/base/arch_topology.c
> +++ b/drivers/base/arch_topology.c
> @@ -21,6 +21,10 @@
> #include <linux/sched.h>
> #include <linux/smp.h>
>
> +__weak bool arch_freq_counters_available(struct cpumask *cpus)
> +{
> + return false;
> +}
> DEFINE_PER_CPU(unsigned long, freq_scale) = SCHED_CAPACITY_SCALE;
>
> void arch_set_freq_scale(struct cpumask *cpus, unsigned long cur_freq,
> @@ -29,6 +33,14 @@ void arch_set_freq_scale(struct cpumask *cpus, unsigned long cur_freq,
> unsigned long scale;
> int i;
>
> + /*
> + * If the use of counters for FIE is enabled, just return as we don't
> + * want to update the scale factor with information from CPUFREQ.
> + * Instead the scale factor will be updated from arch_scale_freq_tick.
> + */
> + if (arch_freq_counters_available(cpus))
> + return;
> +
> scale = (cur_freq << SCHED_CAPACITY_SHIFT) / max_freq;
>
> for_each_cpu(i, cpus)
> diff --git a/include/linux/arch_topology.h b/include/linux/arch_topology.h
> index 3015ecbb90b1..1ccdddb541a7 100644
> --- a/include/linux/arch_topology.h
> +++ b/include/linux/arch_topology.h
> @@ -33,6 +33,8 @@ unsigned long topology_get_freq_scale(int cpu)
> return per_cpu(freq_scale, cpu);
> }
>
> +bool arch_freq_counters_available(struct cpumask *cpus);
> +
> struct cpu_topology {
> int thread_id;
> int core_id;
Sorry for the delay. The arch_topology part looks fine to me. For that part:
Acked-by: Sudeep Holla <[email protected]>
--
Regards,
Sudeep
Hi Catalin, Sudeep,
On Friday 06 Mar 2020 at 11:21:34 (+0000), Catalin Marinas wrote:
> Hi Ionela,
>
> On Thu, Mar 05, 2020 at 09:06:20AM +0000, Ionela Voinescu wrote:
> > v5 -> v6
> > - v5 can be found at [7]
> > - The current series is based on arm64/master [8].
>
> arm64/master is just some random point in the Linus tree, I didn't
> realise people are using it. I moved it to v5.6-rc3 as that's the base
> for the final arm64 for-next/core.
>
> > - [1/7] kernel parameter removed as Catalin recommended
> > - [2/7, 4/7] changelog and documentation changed to remove information
> > about the removed kernel parameter
> > - [2/7, 3/7, 6/7] collected reviewed-by and acked-by from James,
> > Marc and Lukasz.
>
> The series looks fine. Once Sudeep reviews/acks the arch topology patch,
> I'll queue the patches.
>
Thank you both!
Ionela.
> Thanks.
>
> --
> Catalin