This patch series add sysfs interface to control CPU's hardware
prefetch behavior for performance tuning from userspace for arm64 and
x86 (on supported CPU).
Changes from v1:
- split the attribute file so that one-value-per-file
- example of old attribute file
/sys/devices/system/cpu/cpu*/cache/index[0,2]/prefetch_control
- example of new attribute file
/sys/devices/system/cpu/cpu*/cache/index[0,2]/prefetch_control/*_prefetcher_*
- remove the description of "default m" in arm64's Kconfig
https://lore.kernel.org/lkml/[email protected]/
[Background]
============
A64FX and some Intel processors have implementation-dependent register
for controlling CPU's hardware prefetch behavior. A64FX has
IMP_PF_STREAM_DETECT_CTRL_EL0[1], and Intel processors have MSR 0x1a4
(MSR_MISC_FEATURE_CONTROL)[2]. These registers cannot be accessed from
userspace.
[1]https://github.com/fujitsu/A64FX/tree/master/doc/
A64FX_Specification_HPC_Extension_v1_EN.pdf
[2]https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html
Volume 4
The advantage of using this is improved performance. As an example of
performance improvements, the results of running the Stream benchmark
on the A64FX are described in section [Merit].
For MSR 0x1a4, it is also possible to change the value from userspace
via the MSR driver. However, using MSR driver is not recommended, so
it needs a proper kernel interface[3].
[3]https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/about/
For these reasons, we provide a new proper kernel interface to control
both IMP_PF_STREAM_DETECT_CTRL_EL0 and MSR 0x1a4.
[Overall design]
================
The source code for this driver is divided into common parts
(driver/base/pfctl.c) and architecture parts (arch/XXX/XXX/pfctl.c).
Common parts is described architecture-independent processing, such as
creating sysfs.
Architecture parts is described architecture-dependent processing. It
must contain at least the what type of hardware prefetcher is supported
and how to read/write to the register. These information are set
through registration function in common parts.
This driver creates "prefetch_control" directory and some attribute
files in every CPU's cache/index[0,2] directory, if CPU supports
hardware prefetch control behavior. Each attribute file corresponds to
the cache level of the parent index directory.
Detailed description of this sysfs interface is in
Documentation/ABI/testing/sysfs-devices-system-cpu (patch8).
This driver needs cache sysfs directory and cache level/type
information. In ARM processor, these information can be obtained
from registers even without ACPI PPTT.
We add processing to create a cache/index directory using only the
information from the register if the machine does not support ACPI
PPTT and Kconfig for hardware prefetch control (CONFIG_HWPF_CONTROL)
is true in patch5.
This action caused a problem and is described in [Known problem].
[Examples]
==========
This section provides an example of using this sysfs interface at the
x86's model of INTEL_FAM6_BROADWELL_X.
This model has the following register specifications:
[0] L2 Hardware Prefetcher Disable (R/W)
[1] L2 Adjacent Cache Line Prefetcher Disable (R/W)
[2] DCU Hardware Prefetcher Disable (R/W)
[3] DCU IP Prefetcher Disable (R/W)
[63:4] Reserved
In this case, index0 (L1d cache) corresponds to bit[2,3] and index2
(L2 cache) corresponds to bit [0,1]. A list of attribute files of
index0 and index2 in CPU1 at BROADWELL_X is following:
```
# ls /sys/devices/system/cpu/cpu1/cache/index0/prefetch_control/
hardware_prefetcher_enable
ip_prefetcher_enable
# ls /sys/devices/system/cpu/cpu1/cache/index2/prefetch_control/
adjacent_cache_line_prefetcher_enable
hardware_prefetcher_enable
```
If user would like to disable the setting of "L2 Adjacent Cache Line
Prefetcher Disable (R/W)" in CPU1, do the following:
```
# echo 0 > /sys/devices/system/cpu/cpu1/cache/index2/prefetch_control/adjacent_cache_line_prefetcher_enable
```
In another example, a list of index0 at A64FX is following:
```
# ls /sys/devices/system/cpu/cpu1/cache/index0/prefetch_control/
stream_detect_prefetcher_dist
stream_detect_prefetcher_enable
stream_detect_prefetcher_strong
```
[Patch organizations]
=====================
This patch series add hardware prefetch control core driver for ARM64
and x86. Also, we add support for FUJITSU_CPU_PART_A64FX at ARM64 and
BROADWELL_X at x86.
- patch1: Add hardware prefetch core driver
This driver provides a register/unregister function to create the
"prefetch_control" directory and some attribute files in every CPU's
cache/index[0,2] directory.
If the architecture has control of the CPU's hardware prefetch
behavior, use this function to create sysfs. When registering, it
is necessary to provide what type of Hardware Prefetcher is
supported and how to read/write to the register.
- patch2: Add Kconfig/Makefile to build hardware prefetch control core
driver
- patch3: Add support for ARM64
This adds module init/exit code, and creates sysfs attribute file
"stream_detect_prefetcher_enable", "stream_detect_prefetcher_strong"
and "stream_detect_prefetcher_dist" for ARM64. This driver works only
if part number is FUJITSU_CPU_PART_A64FX at this point.
- patch4: Add Kconfig/Makefile to build driver for arm64
- patch5: Create cache sysfs directory without ACPI PPTT for hardware
prefetch control
Hardware Prefetch control driver needs cache sysfs directory and cache
level/type information. In ARM processor, these information can be
obtained from registers even without PPTT. Therefore, we set the
cpu_map_populated to true to create cache sysfs directory, if the
machine doesn't have PPTT.
- patch6: Add support for x86
This adds module init/exit code, and creates sysfs attribute file
"hardware_prefetcher_enable", "ip_prefetcher_enable" and
"adjacent_cache_line_prefetcher_enable" for x86. This driver works
only if the model is INTEL_FAM6_BROADWELL_X at this point.
- patch7: Add Kconfig/Makefile to build driver for x86
- patch8: Add documentation for the new sysfs interface
[Known problem]
===============
- `lscpu` command terminates with -ENOENT because cache/index directory
is exists but shared_cpu_map file does not exist. This is due to
patch5, which creates a cache/index directory containing only level
and type without ACPI PPTT.
[Merit]
=======
For reference, here is the result of STREAM Triad when tuning with
the "s file in L1 and L2 cache on A64FX.
| dist combination | Pattern A | Pattern B |
|-------------------|-------------|-------------|
| L1:256, L2:1024 | 234505.2144 | 114600.0801 |
| L1:1536, L2:1024 | 279172.8742 | 118979.4542 |
| L1:256, L2:10240 | 247716.7757 | 127364.1533 |
| L1:1536, L2:10240 | 283675.6625 | 125950.6847 |
In pattern A, we set the size of the array to 174720, which is about
half the size of the L1d cache. In pattern B, we set the size of the
array to 10485120, which is about twice the size of the L2 cache.
In pattern A, a change of dist at L1 has a larger effect. On the other
hand, in pattern B, the change of dist at L2 has a larger effect.
As described above, the optimal dist combination depends on the
characteristics of the application. Therefore, such a sysfs interface
is useful for performance tuning.
Best regards,
Kohei Tarumizu
Kohei Tarumizu (8):
drivers: base: Add hardware prefetch control core driver
drivers: base: Add Kconfig/Makefile to build hardware prefetch control
core driver
arm64: Add hardware prefetch control support for ARM64
arm64: Add Kconfig/Makefile to build hardware prefetch control driver
arm64: Create cache sysfs directory without ACPI PPTT for hardware
prefetch control
x86: Add hardware prefetch control support for x86
x86: Add Kconfig/Makefile to build hardware prefetch control driver
docs: ABI: Add sysfs documentation interface of hardware prefetch
control driver
.../ABI/testing/sysfs-devices-system-cpu | 89 ++++
MAINTAINERS | 8 +
arch/arm64/Kconfig | 7 +
arch/arm64/kernel/Makefile | 1 +
arch/arm64/kernel/cacheinfo.c | 29 ++
arch/arm64/kernel/pfctl.c | 368 ++++++++++++++++
arch/x86/Kconfig | 7 +
arch/x86/kernel/cpu/Makefile | 2 +
arch/x86/kernel/cpu/pfctl.c | 314 +++++++++++++
drivers/base/Kconfig | 13 +
drivers/base/Makefile | 1 +
drivers/base/pfctl.c | 412 ++++++++++++++++++
include/linux/pfctl.h | 41 ++
13 files changed, 1292 insertions(+)
create mode 100644 arch/arm64/kernel/pfctl.c
create mode 100644 arch/x86/kernel/cpu/pfctl.c
create mode 100644 drivers/base/pfctl.c
create mode 100644 include/linux/pfctl.h
--
2.27.0
This adds module init/exit code, and creates sysfs attribute file
"hardware_prefetcher_enable", "ip_prefetcher_enable" and
"adjacent_cache_line_prefetcher_enable" for x86. This driver works
only if the model is INTEL_FAM6_BROADWELL_X at this point.
If you would like to support a new model with the same register
specifications as INTEL_FAM6_BROADWELL_X, it is possible to add the
model settings to array of broadwell_cpu_ids[].
The details of the registers to be read and written in this patch are
described below:
"https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html"
Volume 4
Signed-off-by: Kohei Tarumizu <[email protected]>
---
arch/x86/kernel/cpu/pfctl.c | 314 ++++++++++++++++++++++++++++++++++++
1 file changed, 314 insertions(+)
create mode 100644 arch/x86/kernel/cpu/pfctl.c
diff --git a/arch/x86/kernel/cpu/pfctl.c b/arch/x86/kernel/cpu/pfctl.c
new file mode 100644
index 000000000000..be2dce644808
--- /dev/null
+++ b/arch/x86/kernel/cpu/pfctl.c
@@ -0,0 +1,314 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright 2022 FUJITSU LIMITED
+ *
+ * x86 Hardware Prefetch Control support
+ */
+
+#include <linux/bitfield.h>
+#include <linux/cacheinfo.h>
+#include <linux/pfctl.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <asm/cpu_device_id.h>
+#include <asm/intel-family.h>
+#include <asm/msr.h>
+
+struct pfctl_driver x86_pfctl_driver;
+
+/**************************************
+ * Intle BROADWELL support
+ **************************************/
+
+/*
+ * The register specification for each bits of Intel BROADWELL is as
+ * follow:
+ *
+ * [0] L2 Hardware Prefetcher Disable (R/W)
+ * [1] L2 Adjacent Cache Line Prefetcher Disable (R/W)
+ * [2] DCU Hardware Prefetcher Disable (R/W)
+ * [3] DCU IP Prefetcher Disable (R/W)
+ * [63:4] Reserved
+ *
+ * See "Intel 64 and IA-32 Architectures Software Developer's Manual"
+ * (https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html)
+ * for register specification details.
+ */
+#define BROADWELL_L2_HWPF_FIELD BIT_ULL(0)
+#define BROADWELL_L2_ACLPF_FIELD BIT_ULL(1)
+#define BROADWELL_DCU_HWPF_FIELD BIT_ULL(2)
+#define BROADWELL_DCU_IPPF_FIELD BIT_ULL(3)
+
+static int broadwell_get_hwpf_enable(u64 reg, unsigned int level)
+{
+ u64 val;
+
+ switch (level) {
+ case 1:
+ val = FIELD_GET(BROADWELL_DCU_HWPF_FIELD, reg);
+ break;
+ case 2:
+ val = FIELD_GET(BROADWELL_L2_HWPF_FIELD, reg);
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ if (val == 0)
+ return PFCTL_ENABLE_VAL;
+ else if (val == 1)
+ return PFCTL_DISABLE_VAL;
+ else
+ return -EINVAL;
+}
+
+static int broadwell_modify_hwpf_enable(u64 *reg, unsigned int level, u64 val)
+{
+ if (val == PFCTL_ENABLE_VAL)
+ val = 0;
+ else
+ val = 1;
+
+ switch (level) {
+ case 1:
+ *reg &= ~BROADWELL_DCU_HWPF_FIELD;
+ *reg |= FIELD_PREP(BROADWELL_DCU_HWPF_FIELD, val);
+ break;
+ case 2:
+ *reg &= ~BROADWELL_L2_HWPF_FIELD;
+ *reg |= FIELD_PREP(BROADWELL_L2_HWPF_FIELD, val);
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int broadwell_get_ippf_enable(u64 reg, unsigned int level)
+{
+ u64 val;
+
+ switch (level) {
+ case 1:
+ val = FIELD_GET(BROADWELL_DCU_IPPF_FIELD, reg);
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ if (val == 0)
+ return PFCTL_ENABLE_VAL;
+ else if (val == 1)
+ return PFCTL_DISABLE_VAL;
+ else
+ return -EINVAL;
+}
+
+static int broadwell_modify_ippf_enable(u64 *reg, unsigned int level, u64 val)
+{
+ if (val == PFCTL_ENABLE_VAL)
+ val = 0;
+ else
+ val = 1;
+
+ switch (level) {
+ case 1:
+ *reg &= ~BROADWELL_DCU_IPPF_FIELD;
+ *reg |= FIELD_PREP(BROADWELL_DCU_IPPF_FIELD, val);
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int broadwell_get_aclpf_enable(u64 reg, unsigned int level)
+{
+ u64 val;
+
+ switch (level) {
+ case 2:
+ val = FIELD_GET(BROADWELL_L2_ACLPF_FIELD, reg);
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ if (val == 0)
+ return PFCTL_ENABLE_VAL;
+ else if (val == 1)
+ return PFCTL_DISABLE_VAL;
+ else
+ return -EINVAL;
+}
+
+static int broadwell_modify_aclpf_enable(u64 *reg, unsigned int level, u64 val)
+{
+ if (val == PFCTL_ENABLE_VAL)
+ val = 0;
+ else
+ val = 1;
+
+ switch (level) {
+ case 2:
+ *reg &= ~BROADWELL_L2_ACLPF_FIELD;
+ *reg |= FIELD_PREP(BROADWELL_L2_ACLPF_FIELD, val);
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int broadwell_get_pfctl_params(enum pfctl_attr pattr, u64 reg,
+ unsigned int level, u64 *val)
+{
+ int ret;
+
+ switch (pattr) {
+ case HWPF_ENABLE:
+ ret = broadwell_get_hwpf_enable(reg, level);
+ break;
+ case IPPF_ENABLE:
+ ret = broadwell_get_ippf_enable(reg, level);
+ break;
+ case ACLPF_ENABLE:
+ ret = broadwell_get_aclpf_enable(reg, level);
+ break;
+ default:
+ return -ENOENT;
+ }
+
+ if (ret < 0)
+ return ret;
+ *val = ret;
+
+ return 0;
+}
+
+static int broadwell_modify_pfreg(enum pfctl_attr pattr, u64 *reg,
+ unsigned int level, u64 val)
+{
+ int ret;
+
+ switch (pattr) {
+ case HWPF_ENABLE:
+ ret = broadwell_modify_hwpf_enable(reg, level, val);
+ break;
+ case IPPF_ENABLE:
+ ret = broadwell_modify_ippf_enable(reg, level, val);
+ break;
+ case ACLPF_ENABLE:
+ ret = broadwell_modify_aclpf_enable(reg, level, val);
+ break;
+ default:
+ return -ENOENT;
+ }
+
+ if (ret < 0)
+ return ret;
+
+ return 0;
+}
+
+static int broadwell_read_pfreg(enum pfctl_attr pattr, unsigned int cpu,
+ unsigned int level, u64 *val)
+{
+ int ret;
+ u64 reg;
+
+ ret = rdmsrl_on_cpu(cpu, MSR_MISC_FEATURE_CONTROL, ®);
+ if (ret)
+ return ret;
+
+ ret = broadwell_get_pfctl_params(pattr, reg, level, val);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static int broadwell_write_pfreg(enum pfctl_attr pattr, unsigned int cpu,
+ unsigned int level, u64 val)
+{
+ int ret;
+ u64 reg;
+
+ ret = rdmsrl_on_cpu(cpu, MSR_MISC_FEATURE_CONTROL, ®);
+ if (ret)
+ return ret;
+
+ ret = broadwell_modify_pfreg(pattr, ®, level, val);
+ if (ret < 0)
+ return ret;
+
+ ret = wrmsrl_on_cpu(cpu, MSR_MISC_FEATURE_CONTROL, reg);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+/*
+ * In addition to BROADWELL_X, NEHALEM and others have same register
+ * specifications as those represented by BROADWELL_XXX_FIELD.
+ * If you want to add support for these processor, add the new target model
+ * here.
+ */
+static const struct x86_cpu_id broadwell_cpu_ids[] = {
+ X86_MATCH_INTEL_FAM6_MODEL(BROADWELL_X, NULL),
+ {}
+};
+
+/***** end of Intel BROADWELL support *****/
+
+/*
+ * This driver returns a negative value if it does not support the Hardware
+ * Prefetch Control or if it is running on a VM guest.
+ */
+static int __init setup_pfctl_driver_params(void)
+{
+ if (boot_cpu_has(X86_FEATURE_HYPERVISOR))
+ return -EINVAL;
+
+ if (x86_match_cpu(broadwell_cpu_ids)) {
+ x86_pfctl_driver.supported_l1d_prefetcher = HWPF|IPPF;
+ x86_pfctl_driver.supported_l2_prefetcher = HWPF|ACLPF;
+ x86_pfctl_driver.read_pfreg = broadwell_read_pfreg;
+ x86_pfctl_driver.write_pfreg = broadwell_write_pfreg;
+ } else {
+ return -ENODEV;
+ }
+
+ return 0;
+}
+
+static int __init x86_pfctl_init(void)
+{
+ int ret;
+
+ ret = setup_pfctl_driver_params();
+ if (ret < 0)
+ return ret;
+
+ ret = pfctl_register_driver(&x86_pfctl_driver);
+ if (ret < 0)
+ return ret;
+
+ return 0;
+}
+
+static void __exit x86_pfctl_exit(void)
+{
+ pfctl_unregister_driver(&x86_pfctl_driver);
+}
+
+late_initcall(x86_pfctl_init);
+module_exit(x86_pfctl_exit);
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("FUJITSU LIMITED");
+MODULE_DESCRIPTION("x86 Hardware Prefetch Control Driver");
--
2.27.0
This adds Kconfig/Makefile to build hardware prefetch control driver
for arm64 support. This also adds a MAINTAINERS entry.
Signed-off-by: Kohei Tarumizu <[email protected]>
---
MAINTAINERS | 1 +
arch/arm64/Kconfig | 7 +++++++
arch/arm64/kernel/Makefile | 1 +
3 files changed, 9 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index 213537cea2e2..7eb530f5b301 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -8455,6 +8455,7 @@ K: (devm_)?hwmon_device_(un)?register(|_with_groups|_with_info)
HARDWARE PREFETCH CONTROL DRIVERS
M: Kohei Tarumizu <[email protected]>
S: Maintained
+F: arch/arm64/kernel/pfctl.c
F: drivers/base/pfctl.c
F: include/linux/pfctl.h
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 09b885cc4db5..da6bf7e75df6 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -36,6 +36,7 @@ config ARM64
select ARCH_HAS_SET_DIRECT_MAP
select ARCH_HAS_SET_MEMORY
select ARCH_STACKWALK
+ select ARCH_HAS_HWPF_CONTROL
select ARCH_HAS_STRICT_KERNEL_RWX
select ARCH_HAS_STRICT_MODULE_RWX
select ARCH_HAS_SYNC_DMA_FOR_DEVICE
@@ -2027,6 +2028,12 @@ config STACKPROTECTOR_PER_TASK
def_bool y
depends on STACKPROTECTOR && CC_HAVE_STACKPROTECTOR_SYSREG
+config ARM64_HWPF_CONTROL
+ tristate "ARM64 Hardware Prefetch Control support"
+ depends on HWPF_CONTROL
+ help
+ This adds Hardware Prefetch driver control support for ARM64.
+
endmenu
menu "Boot options"
diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile
index 88b3e2a21408..d5eb1dc6bfa6 100644
--- a/arch/arm64/kernel/Makefile
+++ b/arch/arm64/kernel/Makefile
@@ -73,6 +73,7 @@ obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o
obj-$(CONFIG_ARM64_MTE) += mte.o
obj-y += vdso-wrap.o
obj-$(CONFIG_COMPAT_VDSO) += vdso32-wrap.o
+obj-$(CONFIG_ARM64_HWPF_CONTROL) += pfctl.o
obj-y += probes/
head-y := head.o
--
2.27.0
On 3/11/22 02:19, Kohei Tarumizu wrote:
> +static int broadwell_write_pfreg(enum pfctl_attr pattr, unsigned int cpu,
> + unsigned int level, u64 val)
> +{
> + int ret;
> + u64 reg;
> +
> + ret = rdmsrl_on_cpu(cpu, MSR_MISC_FEATURE_CONTROL, ®);
> + if (ret)
> + return ret;
> +
> + ret = broadwell_modify_pfreg(pattr, ®, level, val);
> + if (ret < 0)
> + return ret;
> +
> + ret = wrmsrl_on_cpu(cpu, MSR_MISC_FEATURE_CONTROL, reg);
> + if (ret)
> + return ret;
> +
> + return 0;
> +}
This needs to integrate _somehow_ with the pseudo_lock.c code. Right
now, I suspect that code would just overwrite any MSR changes made by
this code.
On 3/11/22 02:19, Kohei Tarumizu wrote:
> The advantage of using this is improved performance. As an example of
> performance improvements, the results of running the Stream benchmark
> on the A64FX are described in section [Merit].
I take it that there are users out there today that are sufficiently
motivated by the increased performance that they just do "wrmsr 0x1a4
0x1234".
You talked about this in the "[Merit]" section. But, that's a _little_
unconvincing. I don't doubt that there is *a* workload out there that
can benefit from hardware prefetcher tweaks.
Do we really expect end users to run their workloads and tweak these
values to find something optimal for them?
> > +static int broadwell_write_pfreg(enum pfctl_attr pattr, unsigned int cpu,
> > + unsigned int level, u64 val)
> > +{
> > + int ret;
> > + u64 reg;
> > +
> > + ret = rdmsrl_on_cpu(cpu, MSR_MISC_FEATURE_CONTROL, ®);
> > + if (ret)
> > + return ret;
> > +
> > + ret = broadwell_modify_pfreg(pattr, ®, level, val);
> > + if (ret < 0)
> > + return ret;
> > +
> > + ret = wrmsrl_on_cpu(cpu, MSR_MISC_FEATURE_CONTROL, reg);
> > + if (ret)
> > + return ret;
> > +
> > + return 0;
> > +}
>
> This needs to integrate _somehow_ with the pseudo_lock.c code. Right now, I
> suspect that code would just overwrite any MSR changes made by this code.
I lacked consideration for pseudo_lock.c code. I try to integration
with that code.
> I take it that there are users out there today that are sufficiently motivated by the
> increased performance that they just do "wrmsr 0x1a4 0x1234".
>
> You talked about this in the "[Merit]" section. But, that's a _little_ unconvincing.
> I don't doubt that there is *a* workload out there that can benefit from hardware
> prefetcher tweaks.
>
> Do we really expect end users to run their workloads and tweak these values to
> find something optimal for them?
In addition to the sample benchmarks in the [Merit] section, we assume
that some workloads will benefit from tweaking prefetches. We expect that
users can find the best parameters by using an tunable I/F from userspace.
I will find out the specific workload which improves the performance.