From: Matt Fleming <[email protected]>
Tony reports that booting his 144-cpu machine with maxcpus=10 triggers
the following WARN_ON(),
[ 21.045727] WARNING: CPU: 8 PID: 647 at arch/x86/kernel/cpu/perf_event_intel_cqm.c:1267 intel_cqm_cpu_prepare+0x75/0x90()
[ 21.045744] CPU: 8 PID: 647 Comm: systemd-udevd Not tainted 4.2.0-rc4 #1
[ 21.045745] Hardware name: Intel Corporation BRICKLAND/BRICKLAND, BIOS BRHSXSD1.86B.0066.R00.1506021730 06/02/2015
[ 21.045747] 0000000000000000 0000000082771b09 ffff880856333ba8 ffffffff81669b67
[ 21.045748] 0000000000000000 0000000000000000 ffff880856333be8 ffffffff8107b02a
[ 21.045750] ffff88085b789800 ffff88085f68a020 ffffffff819e2470 000000000000000a
[ 21.045750] Call Trace:
[ 21.045757] [<ffffffff81669b67>] dump_stack+0x45/0x57
[ 21.045759] [<ffffffff8107b02a>] warn_slowpath_common+0x8a/0xc0
[ 21.045761] [<ffffffff8107b15a>] warn_slowpath_null+0x1a/0x20
[ 21.045762] [<ffffffff81036725>] intel_cqm_cpu_prepare+0x75/0x90
[ 21.045764] [<ffffffff81036872>] intel_cqm_cpu_notifier+0x42/0x160
[ 21.045767] [<ffffffff8109a33d>] notifier_call_chain+0x4d/0x80
[ 21.045769] [<ffffffff8109a44e>] __raw_notifier_call_chain+0xe/0x10
[ 21.045770] [<ffffffff8107b538>] _cpu_up+0xe8/0x190
[ 21.045771] [<ffffffff8107b65a>] cpu_up+0x7a/0xa0
[ 21.045774] [<ffffffff8165e920>] cpu_subsys_online+0x40/0x90
[ 21.045777] [<ffffffff81433b37>] device_online+0x67/0x90
[ 21.045778] [<ffffffff81433bea>] online_store+0x8a/0xa0
[ 21.045782] [<ffffffff81430e78>] dev_attr_store+0x18/0x30
[ 21.045785] [<ffffffff8126b6ba>] sysfs_kf_write+0x3a/0x50
[ 21.045786] [<ffffffff8126ad40>] kernfs_fop_write+0x120/0x170
[ 21.045789] [<ffffffff811f0b77>] __vfs_write+0x37/0x100
[ 21.045791] [<ffffffff811f38b8>] ? __sb_start_write+0x58/0x110
[ 21.045795] [<ffffffff81296d2d>] ? security_file_permission+0x3d/0xc0
[ 21.045796] [<ffffffff811f1279>] vfs_write+0xa9/0x190
[ 21.045797] [<ffffffff811f2075>] SyS_write+0x55/0xc0
[ 21.045800] [<ffffffff81067300>] ? do_page_fault+0x30/0x80
[ 21.045804] [<ffffffff816709ae>] entry_SYSCALL_64_fastpath+0x12/0x71
[ 21.045805] ---[ end trace fe228b836d8af405 ]---
The root cause is that CPU_UP_PREPARE is completely the wrong notifier
action from which to access cpu_data() because smp_store_cpu_info()
won't have been executed by the target cpu at that point, which in turn
means that ->x86_cache_max_rmid and ->x86_cache_occ_scale haven't been
filled out.
Instead let's invoke our handler from CPU_STARTING and rename it
appropriately.
Reported-by: Tony Luck <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Vikas Shivappa <[email protected]>
Cc: Kanaka Juvva <[email protected]>
Cc: <[email protected]>
Cc: Ashok Raj <[email protected]>
Signed-off-by: Matt Fleming <[email protected]>
---
arch/x86/kernel/cpu/perf_event_intel_cqm.c | 8 +++-----
1 file changed, 3 insertions(+), 5 deletions(-)
diff --git a/arch/x86/kernel/cpu/perf_event_intel_cqm.c b/arch/x86/kernel/cpu/perf_event_intel_cqm.c
index 63eb68b73589..377e8f8ed391 100644
--- a/arch/x86/kernel/cpu/perf_event_intel_cqm.c
+++ b/arch/x86/kernel/cpu/perf_event_intel_cqm.c
@@ -1255,7 +1255,7 @@ static inline void cqm_pick_event_reader(int cpu)
cpumask_set_cpu(cpu, &cqm_cpumask);
}
-static void intel_cqm_cpu_prepare(unsigned int cpu)
+static void intel_cqm_cpu_starting(unsigned int cpu)
{
struct intel_pqr_state *state = &per_cpu(pqr_state, cpu);
struct cpuinfo_x86 *c = &cpu_data(cpu);
@@ -1296,13 +1296,11 @@ static int intel_cqm_cpu_notifier(struct notifier_block *nb,
unsigned int cpu = (unsigned long)hcpu;
switch (action & ~CPU_TASKS_FROZEN) {
- case CPU_UP_PREPARE:
- intel_cqm_cpu_prepare(cpu);
- break;
case CPU_DOWN_PREPARE:
intel_cqm_cpu_exit(cpu);
break;
case CPU_STARTING:
+ intel_cqm_cpu_starting(cpu);
cqm_pick_event_reader(cpu);
break;
}
@@ -1373,7 +1371,7 @@ static int __init intel_cqm_init(void)
goto out;
for_each_online_cpu(i) {
- intel_cqm_cpu_prepare(i);
+ intel_cqm_cpu_starting(i);
cqm_pick_event_reader(i);
}
--
2.1.0
Commit-ID: d7a702f0b1033cf402fef65bd6395072738f0844
Gitweb: http://git.kernel.org/tip/d7a702f0b1033cf402fef65bd6395072738f0844
Author: Matt Fleming <[email protected]>
AuthorDate: Thu, 6 Aug 2015 13:12:43 +0100
Committer: Ingo Molnar <[email protected]>
CommitDate: Wed, 12 Aug 2015 11:37:23 +0200
perf/x86/intel/cqm: Do not access cpu_data() from CPU_UP_PREPARE handler
Tony reports that booting his 144-cpu machine with maxcpus=10 triggers
the following WARN_ON():
[ 21.045727] WARNING: CPU: 8 PID: 647 at arch/x86/kernel/cpu/perf_event_intel_cqm.c:1267 intel_cqm_cpu_prepare+0x75/0x90()
[ 21.045744] CPU: 8 PID: 647 Comm: systemd-udevd Not tainted 4.2.0-rc4 #1
[ 21.045745] Hardware name: Intel Corporation BRICKLAND/BRICKLAND, BIOS BRHSXSD1.86B.0066.R00.1506021730 06/02/2015
[ 21.045747] 0000000000000000 0000000082771b09 ffff880856333ba8 ffffffff81669b67
[ 21.045748] 0000000000000000 0000000000000000 ffff880856333be8 ffffffff8107b02a
[ 21.045750] ffff88085b789800 ffff88085f68a020 ffffffff819e2470 000000000000000a
[ 21.045750] Call Trace:
[ 21.045757] [<ffffffff81669b67>] dump_stack+0x45/0x57
[ 21.045759] [<ffffffff8107b02a>] warn_slowpath_common+0x8a/0xc0
[ 21.045761] [<ffffffff8107b15a>] warn_slowpath_null+0x1a/0x20
[ 21.045762] [<ffffffff81036725>] intel_cqm_cpu_prepare+0x75/0x90
[ 21.045764] [<ffffffff81036872>] intel_cqm_cpu_notifier+0x42/0x160
[ 21.045767] [<ffffffff8109a33d>] notifier_call_chain+0x4d/0x80
[ 21.045769] [<ffffffff8109a44e>] __raw_notifier_call_chain+0xe/0x10
[ 21.045770] [<ffffffff8107b538>] _cpu_up+0xe8/0x190
[ 21.045771] [<ffffffff8107b65a>] cpu_up+0x7a/0xa0
[ 21.045774] [<ffffffff8165e920>] cpu_subsys_online+0x40/0x90
[ 21.045777] [<ffffffff81433b37>] device_online+0x67/0x90
[ 21.045778] [<ffffffff81433bea>] online_store+0x8a/0xa0
[ 21.045782] [<ffffffff81430e78>] dev_attr_store+0x18/0x30
[ 21.045785] [<ffffffff8126b6ba>] sysfs_kf_write+0x3a/0x50
[ 21.045786] [<ffffffff8126ad40>] kernfs_fop_write+0x120/0x170
[ 21.045789] [<ffffffff811f0b77>] __vfs_write+0x37/0x100
[ 21.045791] [<ffffffff811f38b8>] ? __sb_start_write+0x58/0x110
[ 21.045795] [<ffffffff81296d2d>] ? security_file_permission+0x3d/0xc0
[ 21.045796] [<ffffffff811f1279>] vfs_write+0xa9/0x190
[ 21.045797] [<ffffffff811f2075>] SyS_write+0x55/0xc0
[ 21.045800] [<ffffffff81067300>] ? do_page_fault+0x30/0x80
[ 21.045804] [<ffffffff816709ae>] entry_SYSCALL_64_fastpath+0x12/0x71
[ 21.045805] ---[ end trace fe228b836d8af405 ]---
The root cause is that CPU_UP_PREPARE is completely the wrong notifier
action from which to access cpu_data(), because smp_store_cpu_info()
won't have been executed by the target CPU at that point, which in turn
means that ->x86_cache_max_rmid and ->x86_cache_occ_scale haven't been
filled out.
Instead let's invoke our handler from CPU_STARTING and rename it
appropriately.
Reported-by: Tony Luck <[email protected]>
Signed-off-by: Matt Fleming <[email protected]>
Signed-off-by: Peter Zijlstra (Intel) <[email protected]>
Cc: Ashok Raj <[email protected]>
Cc: Kanaka Juvva <[email protected]>
Cc: Linus Torvalds <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Vikas Shivappa <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Ingo Molnar <[email protected]>
---
arch/x86/kernel/cpu/perf_event_intel_cqm.c | 8 +++-----
1 file changed, 3 insertions(+), 5 deletions(-)
diff --git a/arch/x86/kernel/cpu/perf_event_intel_cqm.c b/arch/x86/kernel/cpu/perf_event_intel_cqm.c
index 63eb68b..377e8f8 100644
--- a/arch/x86/kernel/cpu/perf_event_intel_cqm.c
+++ b/arch/x86/kernel/cpu/perf_event_intel_cqm.c
@@ -1255,7 +1255,7 @@ static inline void cqm_pick_event_reader(int cpu)
cpumask_set_cpu(cpu, &cqm_cpumask);
}
-static void intel_cqm_cpu_prepare(unsigned int cpu)
+static void intel_cqm_cpu_starting(unsigned int cpu)
{
struct intel_pqr_state *state = &per_cpu(pqr_state, cpu);
struct cpuinfo_x86 *c = &cpu_data(cpu);
@@ -1296,13 +1296,11 @@ static int intel_cqm_cpu_notifier(struct notifier_block *nb,
unsigned int cpu = (unsigned long)hcpu;
switch (action & ~CPU_TASKS_FROZEN) {
- case CPU_UP_PREPARE:
- intel_cqm_cpu_prepare(cpu);
- break;
case CPU_DOWN_PREPARE:
intel_cqm_cpu_exit(cpu);
break;
case CPU_STARTING:
+ intel_cqm_cpu_starting(cpu);
cqm_pick_event_reader(cpu);
break;
}
@@ -1373,7 +1371,7 @@ static int __init intel_cqm_init(void)
goto out;
for_each_online_cpu(i) {
- intel_cqm_cpu_prepare(i);
+ intel_cqm_cpu_starting(i);
cqm_pick_event_reader(i);
}