Hi
I tried to start my favorite tool, and discovered it wont fly on my HP BL460C G1
It samples few events then, seems to stop (no more NMI), other than that, machine
still running correctly.
(8 cpus like that one
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5450 @ 3.00GHz
stepping : 6
cpu MHz : 3000.253
cache size : 6144 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc arch_perfmon pebs bts pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm tpr_shadow vnmi flexpriority
bogomips : 6000.50
clflush size : 64
power management:
It is working correctly on 2.6.27
I tried a git bisect but quickly found a point where machine was not useable...
(partly because of the problem with the VPD problem on 5708S (PCI: Limit VPD length for Broadcom 5708S)
# opcontrol --deinit
Stopping profiling.
Killing daemon.
# rm -rf /var/lib/oprofile/samples/current
# opcontrol --vmlinux=vmlinux --start
Using 2.6+ OProfile kernel interface.
Reading module info.
Using log file /var/lib/oprofile/samples/oprofiled.log
Daemon started.
Profiler running.
# opcontrol --dump; opreport
CPU: Core 2, speed 3000.25 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 10000
CPU_CLK_UNHALT...|
samples| %|
------------------
17 80.9524 vmlinux
2 9.5238 libc-2.3.4.so
1 4.7619 bash
1 4.7619 ld-2.3.4.so
# opcontrol --dump; opreport
CPU: Core 2, speed 3000.25 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 10000
CPU_CLK_UNHALT...|
samples| %|
------------------
17 80.9524 vmlinux
2 9.5238 libc-2.3.4.so
1 4.7619 bash
1 4.7619 ld-2.3.4.so
# opcontrol --dump; opreport
CPU: Core 2, speed 3000.25 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 10000
CPU_CLK_UNHALT...|
samples| %|
------------------
17 80.9524 vmlinux
2 9.5238 libc-2.3.4.so
1 4.7619 bash
1 4.7619 ld-2.3.4.so
# grep NMI /proc/interrupts
NMI: 8 2 2 6 11 7 6 5 Non-maskable interrupts
# grep NMI /proc/interrupts
NMI: 8 2 2 6 11 7 6 5 Non-maskable interrupts
# grep NMI /proc/interrupts
NMI: 8 2 2 6 11 7 6 5 Non-maskable interrupts
# grep NMI /proc/interrupts
NMI: 8 2 2 6 11 7 6 5 Non-maskable interrupts
(adding some more cc's)
On Thu, Oct 30, 2008 at 8:01 PM, Eric Dumazet <[email protected]> wrote:
> I tried to start my favorite tool, and discovered it wont fly on my HP
> BL460C G1
>
> It samples few events then, seems to stop (no more NMI), other than that,
> machine
> still running correctly.
>
> (8 cpus like that one
>
> processor : 0
> vendor_id : GenuineIntel
> cpu family : 6
> model : 23
> model name : Intel(R) Xeon(R) CPU E5450 @ 3.00GHz
> stepping : 6
> cpu MHz : 3000.253
> cache size : 6144 KB
> physical id : 0
> siblings : 4
> core id : 0
> cpu cores : 4
> apicid : 0
> initial apicid : 0
> fdiv_bug : no
> hlt_bug : no
> f00f_bug : no
> coma_bug : no
> fpu : yes
> fpu_exception : yes
> cpuid level : 10
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
> cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm
> constant_tsc arch_perfmon pebs bts pni dtes64 monitor ds_cpl vmx est tm2
> ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm tpr_shadow vnmi flexpriority
> bogomips : 6000.50
> clflush size : 64
> power management:
>
> It is working correctly on 2.6.27
>
> I tried a git bisect but quickly found a point where machine was not
> useable...
> (partly because of the problem with the VPD problem on 5708S (PCI: Limit VPD
> length for Broadcom 5708S)
>
> # opcontrol --deinit
> Stopping profiling.
> Killing daemon.
> # rm -rf /var/lib/oprofile/samples/current
> # opcontrol --vmlinux=vmlinux --start
> Using 2.6+ OProfile kernel interface.
> Reading module info.
> Using log file /var/lib/oprofile/samples/oprofiled.log
> Daemon started.
> Profiler running.
>
> # opcontrol --dump; opreport
> CPU: Core 2, speed 3000.25 MHz (estimated)
> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit
> mask of 0x00 (Unhalted core cycles) count 10000
> CPU_CLK_UNHALT...|
> samples| %|
> ------------------
> 17 80.9524 vmlinux
> 2 9.5238 libc-2.3.4.so
> 1 4.7619 bash
> 1 4.7619 ld-2.3.4.so
> # opcontrol --dump; opreport
> CPU: Core 2, speed 3000.25 MHz (estimated)
> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit
> mask of 0x00 (Unhalted core cycles) count 10000
> CPU_CLK_UNHALT...|
> samples| %|
> ------------------
> 17 80.9524 vmlinux
> 2 9.5238 libc-2.3.4.so
> 1 4.7619 bash
> 1 4.7619 ld-2.3.4.so
> # opcontrol --dump; opreport
> CPU: Core 2, speed 3000.25 MHz (estimated)
> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit
> mask of 0x00 (Unhalted core cycles) count 10000
> CPU_CLK_UNHALT...|
> samples| %|
> ------------------
> 17 80.9524 vmlinux
> 2 9.5238 libc-2.3.4.so
> 1 4.7619 bash
> 1 4.7619 ld-2.3.4.so
>
> # grep NMI /proc/interrupts
> NMI: 8 2 2 6 11 7
> 6 5 Non-maskable interrupts
> # grep NMI /proc/interrupts
> NMI: 8 2 2 6 11 7
> 6 5 Non-maskable interrupts
> # grep NMI /proc/interrupts
> NMI: 8 2 2 6 11 7
> 6 5 Non-maskable interrupts
> # grep NMI /proc/interrupts
> NMI: 8 2 2 6 11 7
> 6 5 Non-maskable interrupts
I'm seeing the same thing on my 2-way x86-64. Here's my /proc/cpuinfo
in case it helps:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz
stepping : 6
cpu MHz : 2000.000
cache size : 4096 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
lm constant_tsc arch_perfmon pebs bts rep_good pni dtes64 monitor
ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm tpr_shadow
bogomips : 3989.97
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz
stepping : 6
cpu MHz : 2000.000
cache size : 4096 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx
lm constant_tsc arch_perfmon pebs bts rep_good pni dtes64 monitor
ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm tpr_shadow
bogomips : 3989.97
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
Hi Eric (and related people touching the code),
Just to confirm your observations (as you have a ? in subj);
oprofile is also broken for me on kernel 2.6.28-rc2.
(Related people Cc'ed
git log v2.6.27.. drivers/oprofile/ | grep Author: | sort |uniq)
Eric Dumazet wrote:
> I tried to start my favorite tool, and discovered it wont fly on my HP
> BL460C G1
My hardware: HP-proliant DL380-G5
CPU: Intel(R) Xeon(R) CPU E5335 @ 2.00GHz
> It samples few events then, seems to stop (no more NMI), other than that,
> machine still running correctly.
Same observation.
> It is working correctly on 2.6.27
Same observation.
Eric,
On 30.10.08 19:01:52, Eric Dumazet wrote:
> Hi
>
> I tried to start my favorite tool, and discovered it wont fly on my HP
> BL460C G1
>
> It samples few events then, seems to stop (no more NMI), other than that,
> machine
> still running correctly.
hmm, I can not reproduce this on my system. I am receiving samples and
the number is increasing. Andi, could you test this on some of your
systems?
-Robert
# opcontrol --dump; opreport
CPU: AMD64 family10, speed 2100 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit mask of 0x00 (No unit mask) count 100000
CPU_CLK_UNHALT...|
samples| %|
------------------
70897 98.6970 vmlinux
467 0.6501 libc-2.6.1.so
219 0.3049 bash
141 0.1963 ld-2.6.1.so
43 0.0599 oprofiled
15 0.0209 gawk-3.1.5
15 0.0209 opreport
9 0.0125 libstdc++.so.6.0.8
8 0.0111 grep
5 0.0070 sshd
4 0.0056 libcrypto.so.0.9.8
2 0.0028 rm
1 0.0014 cat
1 0.0014 dirname
1 0.0014 ls
1 0.0014 mkdir
1 0.0014 id
1 0.0014 ophelp
1 0.0014 libbfd-2.18.so
1 0.0014 ntpd
# grep NMI /proc/interrupts
NMI: 20357 14272 20151 9967 20140 13961 16291 20826 Non-maskable interrupts
# grep NMI /proc/interrupts
NMI: 20627 14458 20398 10080 20391 14204 16444 21016 Non-maskable interrupts
# cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 16
model : 2
model name : Quad-Core AMD Opteron(tm) Processor 2352
stepping : 3
cpu MHz : 2100.000
cache size : 512 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good pni monitor cx16 lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs
bogomips : 4189.26
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate
# uname -a
Linux paloma 2.6.28-rc2-oprofile-x86_64-standard #1 SMP Sat Nov 1 01:28:08 CET 2008 x86_64 Quad-Core AMD Opteron(tm) Processor 2352 AuthenticAMD GNU/Linux
--
Advanced Micro Devices, Inc.
Operating System Research Center
email: [email protected]
Andi,
your patchset breaks oprofile on Intel Core 2 cpus. I could bisect the
bug for c493756..5951290. Please take a look at this and send me a fix.
Thanks,
-Robert
Linux woglinde 2.6.27-oprofile-x86_64-standard-03189-g5951290 #5 SMP Tue Nov 4 18:16:41 CET 2008 x86_64 Intel(R) Core(TM)2 CPU 6300 @ 1.86GHz GenuineIntel GNU/Linux
IBS Fetch not supported - ignored
IBS OP not supported - ignored
Using default event: CPU_CLK_UNHALTED:100000:0:1:1
Daemon started.
Profiler running.
Oprofile started
Daemon running: pid 5845
Separate options: none
vmlinux file: none
Image filter: none
Call-graph depth: 0
number of samples #1: 11
Stopping profiling.
Killing daemon.
number of samples #2: 11
Linux woglinde 2.6.27-oprofile-x86_64-standard-03185-gc493756 #6 SMP Tue Nov 4 19:05:13 CET 2008 x86_64 Intel(R) Core(TM)2 CPU 6300 @ 1.86GHz GenuineIntel GNU/Linux
IBS Fetch not supported - ignored
IBS OP not supported - ignored
Using default event: CPU_CLK_UNHALTED:100000:0:1:1
Daemon started.
Profiler running.
Oprofile started
Daemon running: pid 5263
Separate options: none
vmlinux file: none
Image filter: none
Call-graph depth: 0
number of samples #1: 25
Stopping profiling.
Killing daemon.
number of samples #2: 64
Increasing sample number
commit 59512900baab03c5629f2ff5efad1d5d4e682ece
Author: Andi Kleen <[email protected]>
Date: Mon Sep 29 22:23:33 2008 +0200
oprofile: discover counters for op ppro too
Discover number of counters for all family 6 models even when not
in arch perfmon mode.
Signed-off-by: Andi Kleen <[email protected]>
Signed-off-by: Robert Richter <[email protected]>
arch/x86/oprofile/op_model_ppro.c | 8 +++++---
arch/x86/oprofile/op_x86_model.h | 2 +-
2 files changed, 6 insertions(+), 4 deletions(-)
commit b99170288421c79f0c2efa8b33e26e65f4bb7fb8
Author: Andi Kleen <[email protected]>
Date: Mon Aug 18 14:50:31 2008 +0200
oprofile: Implement Intel architectural perfmon support
Newer Intel CPUs (Core1+) have support for architectural
events described in CPUID 0xA. See the IA32 SDM Vol3b.18 for details.
The advantage of this is that it can be done without knowing about
the specific CPU, because the CPU describes by itself what
performance events are supported. This is only a fallback
because only a limited set of 6 events are supported.
This allows to do profiling on Nehalem and on Atom systems
(later not tested)
This patch implements support for that in oprofile's Intel
Family 6 profiling module. It also has the advantage of supporting
an arbitary number of events now as reported by the CPU.
Also allow arbitary counter widths >32bit while we're at it.
Requires a patched oprofile userland to support the new
architecture.
v2: update for latest oprofile tree
remove force_arch_perfmon
Signed-off-by: Andi Kleen <[email protected]>
Signed-off-by: Robert Richter <[email protected]>
arch/x86/oprofile/nmi_int.c | 23 ++++++--
arch/x86/oprofile/op_model_ppro.c | 104 +++++++++++++++++++++++++++++--------
arch/x86/oprofile/op_x86_model.h | 3 +
3 files changed, 102 insertions(+), 28 deletions(-)
commit f645f6406463a01869c50844befc76d528971690
Author: Andi Kleen <[email protected]>
Date: Wed Aug 20 17:22:02 2008 +0200
oprofile: Don't report Nehalem as core_2
This essentially reverts Linus' earlier 4b9f12a3779c548b68bc9af7d94030868ad3aa1b
commit. Nehalem is not core_2, so it shouldn't be reported as such.
However with the earlier arch perfmon patch it will fall back to
arch perfmon mode now, so there is no need to fake it as core_2.
The only drawback is that Linus will need to patch the arch perfmon
support into his oprofile binary now, but I think he can do that.
Signed-off-by: Andi Kleen <[email protected]>
Signed-off-by: Robert Richter <[email protected]>
arch/x86/oprofile/nmi_int.c | 3 ---
1 files changed, 0 insertions(+), 3 deletions(-)
commit 5d4488027d9cf3161c71566dfabb116bf69ab4d9
Author: Andi Kleen <[email protected]>
Date: Mon Aug 18 14:49:47 2008 +0200
oprofile: drop const in num counters field
allow to modify it at runtime
Signed-off-by: Andi Kleen <[email protected]>
Signed-off-by: Robert Richter <[email protected]>
arch/x86/oprofile/op_x86_model.h | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
--
Advanced Micro Devices, Inc.
Operating System Research Center
email: [email protected]
Hi Robert,
On Tue, 4 Nov 2008, Robert Richter wrote:
> your patchset breaks oprofile on Intel Core 2 cpus. I could bisect the
> bug for c493756..5951290. Please take a look at this and send me a fix.
Unfortunately I don't have my machine with me right now so I can't test
this, but don't we want something like this? Andi?
Pekka
>From 37d362c0262006be6a5d6a72b5993834afa776d7 Mon Sep 17 00:00:00 2001
From: Pekka Enberg <[email protected]>
Date: Tue, 4 Nov 2008 21:17:47 +0200
Subject: [PATCH] oprofile: make perfmon nethalem only
Commit b99170288421c79f0c2efa8b33e26e65f4bb7fb8 ("oprofile: Implement Intel
architectural perfmon support") added perfmon support to newer Intel CPUs
(Core1+). The problem there is that it requires a patched oprofile userland to
work properly.
Fix that up by making oprofile perfmon support nethalem only.
Signed-off-by: Pekka Enberg <[email protected]>
---
arch/x86/oprofile/nmi_int.c | 7 +++++++
arch/x86/oprofile/op_model_ppro.c | 4 +++-
2 files changed, 10 insertions(+), 1 deletions(-)
diff --git a/arch/x86/oprofile/nmi_int.c b/arch/x86/oprofile/nmi_int.c
index 022cd41..1f4448e 100644
--- a/arch/x86/oprofile/nmi_int.c
+++ b/arch/x86/oprofile/nmi_int.c
@@ -426,8 +426,15 @@ static int __init ppro_init(char **cpu_type)
static int __init arch_perfmon_init(char **cpu_type)
{
+ __u8 cpu_model = boot_cpu_data.x86_model;
+
if (!cpu_has_arch_perfmon)
return 0;
+
+ /* Nethalem only */
+ if (cpu_model != 26)
+ return 0;
+
*cpu_type = "i386/arch_perfmon";
model = &op_arch_perfmon_spec;
arch_perfmon_setup_counters();
diff --git a/arch/x86/oprofile/op_model_ppro.c b/arch/x86/oprofile/op_model_ppro.c
index 0620d6d..1c91ba2 100644
--- a/arch/x86/oprofile/op_model_ppro.c
+++ b/arch/x86/oprofile/op_model_ppro.c
@@ -66,6 +66,7 @@ static void ppro_fill_in_addresses(struct op_msrs * const msrs)
static void ppro_setup_ctrs(struct op_msrs const * const msrs)
{
+ __u8 cpu_model = boot_cpu_data.x86_model;
unsigned int low, high;
int i;
@@ -76,7 +77,8 @@ static void ppro_setup_ctrs(struct op_msrs const * const msrs)
return;
}
- if (cpu_has_arch_perfmon) {
+ /* Nethalem only */
+ if (cpu_has_arch_perfmon && cpu_model == 26) {
union cpuid10_eax eax;
eax.full = cpuid_eax(0xa);
if (counter_width < eax.split.bit_width)
--
1.5.3.7
Robert Richter wrote:
> Andi,
>
> your patchset breaks oprofile on Intel Core 2 cpus. I could bisect the
> bug for c493756..5951290. Please take a look at this and send me a fix.
I'll take a look but it'll be some time because my test setup is still
not fully functional after moving.
-Andi
Pekka J Enberg a ?crit :
> Hi Robert,
>
> On Tue, 4 Nov 2008, Robert Richter wrote:
>> your patchset breaks oprofile on Intel Core 2 cpus. I could bisect the
>> bug for c493756..5951290. Please take a look at this and send me a fix.
>
> Unfortunately I don't have my machine with me right now so I can't test
> this, but don't we want something like this? Andi?
Thanks a lot Pekka, this patch restores oprofile on my HP BL460c G1
Tested-by: Eric Dumazet <[email protected]>
>
> Pekka
>
>>From 37d362c0262006be6a5d6a72b5993834afa776d7 Mon Sep 17 00:00:00 2001
> From: Pekka Enberg <[email protected]>
> Date: Tue, 4 Nov 2008 21:17:47 +0200
> Subject: [PATCH] oprofile: make perfmon nethalem only
>
> Commit b99170288421c79f0c2efa8b33e26e65f4bb7fb8 ("oprofile: Implement Intel
> architectural perfmon support") added perfmon support to newer Intel CPUs
> (Core1+). The problem there is that it requires a patched oprofile userland to
> work properly.
>
> Fix that up by making oprofile perfmon support nethalem only.
>
> Signed-off-by: Pekka Enberg <[email protected]>
> ---
> arch/x86/oprofile/nmi_int.c | 7 +++++++
> arch/x86/oprofile/op_model_ppro.c | 4 +++-
> 2 files changed, 10 insertions(+), 1 deletions(-)
>
> diff --git a/arch/x86/oprofile/nmi_int.c b/arch/x86/oprofile/nmi_int.c
> index 022cd41..1f4448e 100644
> --- a/arch/x86/oprofile/nmi_int.c
> +++ b/arch/x86/oprofile/nmi_int.c
> @@ -426,8 +426,15 @@ static int __init ppro_init(char **cpu_type)
>
> static int __init arch_perfmon_init(char **cpu_type)
> {
> + __u8 cpu_model = boot_cpu_data.x86_model;
> +
> if (!cpu_has_arch_perfmon)
> return 0;
> +
> + /* Nethalem only */
> + if (cpu_model != 26)
> + return 0;
> +
> *cpu_type = "i386/arch_perfmon";
> model = &op_arch_perfmon_spec;
> arch_perfmon_setup_counters();
> diff --git a/arch/x86/oprofile/op_model_ppro.c b/arch/x86/oprofile/op_model_ppro.c
> index 0620d6d..1c91ba2 100644
> --- a/arch/x86/oprofile/op_model_ppro.c
> +++ b/arch/x86/oprofile/op_model_ppro.c
> @@ -66,6 +66,7 @@ static void ppro_fill_in_addresses(struct op_msrs * const msrs)
>
> static void ppro_setup_ctrs(struct op_msrs const * const msrs)
> {
> + __u8 cpu_model = boot_cpu_data.x86_model;
> unsigned int low, high;
> int i;
>
> @@ -76,7 +77,8 @@ static void ppro_setup_ctrs(struct op_msrs const * const msrs)
> return;
> }
>
> - if (cpu_has_arch_perfmon) {
> + /* Nethalem only */
> + if (cpu_has_arch_perfmon && cpu_model == 26) {
> union cpuid10_eax eax;
> eax.full = cpuid_eax(0xa);
> if (counter_width < eax.split.bit_width)
Pekka J Enberg wrote:
> Hi Robert,
>
> On Tue, 4 Nov 2008, Robert Richter wrote:
>> your patchset breaks oprofile on Intel Core 2 cpus. I could bisect the
>> bug for c493756..5951290. Please take a look at this and send me a fix.
>
> Unfortunately I don't have my machine with me right now so I can't test
> this, but don't we want something like this? Andi?
The patch is not the correct fix to whatever the problem is.
arch perfmon is not Nehalem specific.
-Andi
Eric Dumazet a ?crit :
> Pekka J Enberg a ?crit :
>> Hi Robert,
>>
>> On Tue, 4 Nov 2008, Robert Richter wrote:
>>> your patchset breaks oprofile on Intel Core 2 cpus. I could bisect the
>>> bug for c493756..5951290. Please take a look at this and send me a fix.
>>
>> Unfortunately I don't have my machine with me right now so I can't
>> test this, but don't we want something like this? Andi?
>
> Thanks a lot Pekka, this patch restores oprofile on my HP BL460c G1
>
> Tested-by: Eric Dumazet <[email protected]>
Update :
After a while, no more NMI are delivered. No anymore samples.
# grep NMI /proc/interrupts
NMI: 214337 2004224 183702 2121878 641711 793553 18841 306364 Non-maskable interrupts
# grep NMI /proc/interrupts
NMI: 214337 2004224 183702 2121878 641711 793553 18841 306364 Non-maskable interrupts
# grep NMI /proc/interrupts
NMI: 214337 2004224 183702 2121878 641711 793553 18841 306364 Non-maskable interrupts
[root@svivoipvnx001 /opt/src/linux-2.6]$
# ps aux|grep oprof
root 5336 0.4 0.0 3512 1704 ? S 07:02 0:01 /usr/local/bin/oprofiled --session-dir=/var/lib/oprofile --separate-lib=0 --separate-kernel=0 --separate-thread=0 --separate-cpu=0 --events=CPU_CLK_UNHALTED:60:0:10000:0:1:1, --vmlinux=vmlinux --kernel-range=c02000a0,c045e9c1
# opcontrol --stop
Stopping profiling.
# opcontrol --start
Profiler running.
# grep NMI /proc/interrupts
NMI: 214337 2004224 183702 2121878 641711 793553 18841 306364 Non-maskable interrupts
# grep NMI /proc/interrupts
NMI: 214337 2004224 183702 2121878 641711 793553 18841 306364 Non-maskable interrupts
>
>>
>> Pekka
>>
>>> From 37d362c0262006be6a5d6a72b5993834afa776d7 Mon Sep 17 00:00:00 2001
>> From: Pekka Enberg <[email protected]>
>> Date: Tue, 4 Nov 2008 21:17:47 +0200
>> Subject: [PATCH] oprofile: make perfmon nethalem only
>>
>> Commit b99170288421c79f0c2efa8b33e26e65f4bb7fb8 ("oprofile: Implement
>> Intel
>> architectural perfmon support") added perfmon support to newer Intel CPUs
>> (Core1+). The problem there is that it requires a patched oprofile
>> userland to
>> work properly.
>>
>> Fix that up by making oprofile perfmon support nethalem only.
>>
>> Signed-off-by: Pekka Enberg <[email protected]>
>> ---
>> arch/x86/oprofile/nmi_int.c | 7 +++++++
>> arch/x86/oprofile/op_model_ppro.c | 4 +++-
>> 2 files changed, 10 insertions(+), 1 deletions(-)
>>
>> diff --git a/arch/x86/oprofile/nmi_int.c b/arch/x86/oprofile/nmi_int.c
>> index 022cd41..1f4448e 100644
>> --- a/arch/x86/oprofile/nmi_int.c
>> +++ b/arch/x86/oprofile/nmi_int.c
>> @@ -426,8 +426,15 @@ static int __init ppro_init(char **cpu_type)
>>
>> static int __init arch_perfmon_init(char **cpu_type)
>> {
>> + __u8 cpu_model = boot_cpu_data.x86_model;
>> +
>> if (!cpu_has_arch_perfmon)
>> return 0;
>> +
>> + /* Nethalem only */
>> + if (cpu_model != 26)
>> + return 0;
>> +
>> *cpu_type = "i386/arch_perfmon";
>> model = &op_arch_perfmon_spec;
>> arch_perfmon_setup_counters();
>> diff --git a/arch/x86/oprofile/op_model_ppro.c
>> b/arch/x86/oprofile/op_model_ppro.c
>> index 0620d6d..1c91ba2 100644
>> --- a/arch/x86/oprofile/op_model_ppro.c
>> +++ b/arch/x86/oprofile/op_model_ppro.c
>> @@ -66,6 +66,7 @@ static void ppro_fill_in_addresses(struct op_msrs *
>> const msrs)
>>
>> static void ppro_setup_ctrs(struct op_msrs const * const msrs)
>> {
>> + __u8 cpu_model = boot_cpu_data.x86_model;
>> unsigned int low, high;
>> int i;
>>
>> @@ -76,7 +77,8 @@ static void ppro_setup_ctrs(struct op_msrs const *
>> const msrs)
>> return;
>> }
>>
>> - if (cpu_has_arch_perfmon) {
>> + /* Nethalem only */
>> + if (cpu_has_arch_perfmon && cpu_model == 26) {
>> union cpuid10_eax eax;
>> eax.full = cpuid_eax(0xa);
>> if (counter_width < eax.split.bit_width)
>
>
>
Eric Dumazet a ?crit :
> Eric Dumazet a ?crit :
>> Pekka J Enberg a ?crit :
>>> Hi Robert,
>>>
>>> On Tue, 4 Nov 2008, Robert Richter wrote:
>>>> your patchset breaks oprofile on Intel Core 2 cpus. I could bisect the
>>>> bug for c493756..5951290. Please take a look at this and send me a fix.
>>>
>>> Unfortunately I don't have my machine with me right now so I can't
>>> test this, but don't we want something like this? Andi?
>>
>> Thanks a lot Pekka, this patch restores oprofile on my HP BL460c G1
>>
>> Tested-by: Eric Dumazet <[email protected]>
>
> Update :
>
> After a while, no more NMI are delivered. No anymore samples.
>
Ingo asked me to test a revert of b99170288421c79f0c2efa8b33e26e65f4bb7fb8
(oprofile: Implement Intel architectural perfmon support)
Same problem than previous patch.
(doing a tbench 8 in background)
oprofile works a few seconds, then no more NMI / samples
# opcontrol --vmlinux=vmlinux --start
Using 2.6+ OProfile kernel interface.
Reading module info.
Using log file /var/lib/oprofile/samples/oprofiled.log
Daemon started.
Profiler running.
# grep NMI /proc/interrupts
NMI: 3102535 1009518 733819 2088857 1414146 703880 1229563 473603 Non-maskable interrupts
# grep NMI /proc/interrupts
NMI: 3490492 1009518 767878 2212917 1812791 1654826 2151141 473603 Non-maskable interrupts
# grep NMI /proc/interrupts
NMI: 3490492 1009518 767878 2212917 1812791 2188653 2671979 473603 Non-maskable interrupts
# grep NMI /proc/interrupts
NMI: 3490492 1009518 767878 2212917 1812791 2612789 2671979 473603 Non-maskable interrupts
# grep NMI /proc/interrupts
NMI: 3490492 1009518 767878 2212917 1812791 2612789 2671979 473603 Non-maskable interrupts
# opcontrol --deinit
Stopping profiling.
Killing daemon.
# grep NMI /proc/interrupts
NMI: 3490492 1009518 767878 2212917 1812791 2612789 2671979 473603 Non-maskable interrupts
# opcontrol --vmlinux=vmlinux --start
Using 2.6+ OProfile kernel interface.
Reading module info.
Using log file /var/lib/oprofile/samples/oprofiled.log
Daemon started.
Profiler running.
# grep NMI /proc/interrupts
NMI: 3730357 1265822 1174545 2485635 2257881 3082318 3078445 931068 Non-maskable interrupts
# grep NMI /proc/interrupts
NMI: 3730357 1265822 1573509 2485635 2469701 3512810 3477233 1143043 Non-maskable interrupts
# grep NMI /proc/interrupts
NMI: 3730357 1265822 2069149 2485635 2469701 3512810 3511350 1143043 Non-maskable interrupts
# grep NMI /proc/interrupts
NMI: 3730357 1265822 2069149 2485635 2469701 3512810 3511350 1143043 Non-maskable interrupts
# grep NMI /proc/interrupts
NMI: 3730357 1265822 2069149 2485635 2469701 3512810 3511350 1143043 Non-maskable interrupts
# grep . /dev/oprofile/*/* /dev/oprofile/*
/dev/oprofile/0/count:10000
/dev/oprofile/0/enabled:1
/dev/oprofile/0/event:60
/dev/oprofile/0/kernel:1
/dev/oprofile/0/unit_mask:0
/dev/oprofile/0/user:1
/dev/oprofile/1/count:0
/dev/oprofile/1/enabled:0
/dev/oprofile/1/event:0
/dev/oprofile/1/kernel:0
/dev/oprofile/1/unit_mask:0
/dev/oprofile/1/user:0
/dev/oprofile/stats/bt_lost_no_mapping:0
/dev/oprofile/stats/event_lost_overflow:83466
/dev/oprofile/stats/sample_lost_no_mapping:19174
/dev/oprofile/stats/sample_lost_no_mm:0
/dev/oprofile/backtrace_depth:0
grep: /dev/oprofile/buffer: Device or resource busy
/dev/oprofile/buffer_size:131072
/dev/oprofile/buffer_watershed:32768
/dev/oprofile/cpu_buffer_size:8192
/dev/oprofile/cpu_type:i386/core_2
grep: /dev/oprofile/dump: Invalid argument
/dev/oprofile/enable:1
/dev/oprofile/pointer_size:4
I tried with bigger buffer_size, same problem
opcontrol --vmlinux=vmlinux --cpu-buffer-size=65536 --buffer-size=1048576 --start
# grep . /dev/oprofile/*/* /dev/oprofile/*
/dev/oprofile/0/count:10000
/dev/oprofile/0/enabled:1
/dev/oprofile/0/event:60
/dev/oprofile/0/kernel:1
/dev/oprofile/0/unit_mask:0
/dev/oprofile/0/user:1
/dev/oprofile/1/count:0
/dev/oprofile/1/enabled:0
/dev/oprofile/1/event:0
/dev/oprofile/1/kernel:0
/dev/oprofile/1/unit_mask:0
/dev/oprofile/1/user:0
/dev/oprofile/stats/bt_lost_no_mapping:0
/dev/oprofile/stats/event_lost_overflow:1465392
/dev/oprofile/stats/sample_lost_no_mapping:117498
/dev/oprofile/stats/sample_lost_no_mm:20
/dev/oprofile/backtrace_depth:0
grep: /dev/oprofile/buffer: Device or resource busy
/dev/oprofile/buffer_size:1048576
/dev/oprofile/buffer_watershed:32768
/dev/oprofile/cpu_buffer_size:65536
/dev/oprofile/cpu_type:i386/core_2
grep: /dev/oprofile/dump: Invalid argument
/dev/oprofile/enable:1
/dev/oprofile/pointer_size:4
Can you please test this patch and see if it solves the problem?
Thanks,
-Andi
---
oprofile: Fix p6 counter overflow check
Fix the counter overflow check for CPUs with counter width > 32
I had a similar change in a different patch that I didn't submit
and I didn't notice the problem earlier because it was always
tested together.
Signed-off-by: Andi Kleen <[email protected]>
---
arch/x86/oprofile/op_model_ppro.c | 9 ++++-----
1 file changed, 4 insertions(+), 5 deletions(-)
Index: linux-2.6.28-rc3-test/arch/x86/oprofile/op_model_ppro.c
===================================================================
--- linux-2.6.28-rc3-test.orig/arch/x86/oprofile/op_model_ppro.c 2008-11-07 13:46:23.000000000 +0100
+++ linux-2.6.28-rc3-test/arch/x86/oprofile/op_model_ppro.c 2008-11-07 13:50:57.000000000 +0100
@@ -27,8 +27,7 @@
static int counter_width = 32;
#define CTR_IS_RESERVED(msrs, c) (msrs->counters[(c)].addr ? 1 : 0)
-#define CTR_READ(l, h, msrs, c) do {rdmsr(msrs->counters[(c)].addr, (l), (h)); } while (0)
-#define CTR_OVERFLOWED(n) (!((n) & (1U<<(counter_width-1))))
+#define CTR_OVERFLOWED(n) (!((n) & (1ULL<<(counter_width-1))))
#define CTRL_IS_RESERVED(msrs, c) (msrs->controls[(c)].addr ? 1 : 0)
#define CTRL_READ(l, h, msrs, c) do {rdmsr((msrs->controls[(c)].addr), (l), (h)); } while (0)
@@ -124,14 +123,14 @@
static int ppro_check_ctrs(struct pt_regs * const regs,
struct op_msrs const * const msrs)
{
- unsigned int low, high;
+ u64 val;
int i;
for (i = 0 ; i < num_counters; ++i) {
if (!reset_value[i])
continue;
- CTR_READ(low, high, msrs, i);
- if (CTR_OVERFLOWED(low)) {
+ rdmsrl(msrs->counters[i].addr, val);
+ if (CTR_OVERFLOWED(val)) {
oprofile_add_sample(regs, i);
wrmsrl(msrs->counters[i].addr, -reset_value[i]);
}
--
[email protected]
Andi Kleen a ?crit :
> Can you please test this patch and see if it solves the problem?
>
> Thanks,
> -Andi
>
> ---
>
> oprofile: Fix p6 counter overflow check
>
> Fix the counter overflow check for CPUs with counter width > 32
>
> I had a similar change in a different patch that I didn't submit
> and I didn't notice the problem earlier because it was always
> tested together.
>
> Signed-off-by: Andi Kleen <[email protected]>
>
Thanks Andi
Unfortunatly, there is still a freeze of NMIS after a while.
# grep NMI /proc/interrupts
NMI: 127869 439091 905577 6158 809531 54720 4402881 2411582 Non-maskable interrupts
# grep NMI /proc/interrupts
NMI: 294473 765839 1174243 282249 1190866 382543 4747495 2463270 Non-maskable interrupts
# grep NMI /proc/interrupts
NMI: 294473 899262 1174243 282249 1282407 515973 5139226 2463270 Non-maskable interrupts
# grep NMI /proc/interrupts
NMI: 294473 899262 1174243 282249 1282407 515973 5187081 2463270 Non-maskable interrupts
# grep NMI /proc/interrupts
NMI: 294473 899262 1174243 282249 1282407 515973 5187081 2463270 Non-maskable interrupts
# grep NMI /proc/interrupts
NMI: 294473 899262 1174243 282249 1282407 515973 5187081 2463270 Non-maskable interrupts
# grep NMI /proc/interrupts
NMI: 294473 899262 1174243 282249 1282407 515973 5187081 2463270 Non-maskable interrupts
#
# grep . /dev/oprofile/*/*
/dev/oprofile/0/count:10000
/dev/oprofile/0/enabled:1
/dev/oprofile/0/event:60
/dev/oprofile/0/kernel:1
/dev/oprofile/0/unit_mask:0
/dev/oprofile/0/user:1
/dev/oprofile/1/count:0
/dev/oprofile/1/enabled:0
/dev/oprofile/1/event:0
/dev/oprofile/1/kernel:0
/dev/oprofile/1/unit_mask:0
/dev/oprofile/1/user:0
/dev/oprofile/stats/bt_lost_no_mapping:0
/dev/oprofile/stats/event_lost_overflow:376941
/dev/oprofile/stats/sample_lost_no_mapping:43358
/dev/oprofile/stats/sample_lost_no_mm:26
# grep . /dev/oprofile/*
/dev/oprofile/backtrace_depth:0
grep: /dev/oprofile/buffer: Device or resource busy
/dev/oprofile/buffer_size:1048576
/dev/oprofile/buffer_watershed:32768
/dev/oprofile/cpu_buffer_size:65536
/dev/oprofile/cpu_type:i386/core_2
grep: /dev/oprofile/dump: Invalid argument
/dev/oprofile/enable:1
/dev/oprofile/pointer_size:4
> ---
> arch/x86/oprofile/op_model_ppro.c | 9 ++++-----
> 1 file changed, 4 insertions(+), 5 deletions(-)
>
> Index: linux-2.6.28-rc3-test/arch/x86/oprofile/op_model_ppro.c
> ===================================================================
> --- linux-2.6.28-rc3-test.orig/arch/x86/oprofile/op_model_ppro.c 2008-11-07 13:46:23.000000000 +0100
> +++ linux-2.6.28-rc3-test/arch/x86/oprofile/op_model_ppro.c 2008-11-07 13:50:57.000000000 +0100
> @@ -27,8 +27,7 @@
> static int counter_width = 32;
>
> #define CTR_IS_RESERVED(msrs, c) (msrs->counters[(c)].addr ? 1 : 0)
> -#define CTR_READ(l, h, msrs, c) do {rdmsr(msrs->counters[(c)].addr, (l), (h)); } while (0)
> -#define CTR_OVERFLOWED(n) (!((n) & (1U<<(counter_width-1))))
> +#define CTR_OVERFLOWED(n) (!((n) & (1ULL<<(counter_width-1))))
>
> #define CTRL_IS_RESERVED(msrs, c) (msrs->controls[(c)].addr ? 1 : 0)
> #define CTRL_READ(l, h, msrs, c) do {rdmsr((msrs->controls[(c)].addr), (l), (h)); } while (0)
> @@ -124,14 +123,14 @@
> static int ppro_check_ctrs(struct pt_regs * const regs,
> struct op_msrs const * const msrs)
> {
> - unsigned int low, high;
> + u64 val;
> int i;
>
> for (i = 0 ; i < num_counters; ++i) {
> if (!reset_value[i])
> continue;
> - CTR_READ(low, high, msrs, i);
> - if (CTR_OVERFLOWED(low)) {
> + rdmsrl(msrs->counters[i].addr, val);
> + if (CTR_OVERFLOWED(val)) {
> oprofile_add_sample(regs, i);
> wrmsrl(msrs->counters[i].addr, -reset_value[i]);
> }
>
>
On Fri, Nov 07, 2008 at 04:13:42PM +0100, Eric Dumazet wrote:
> Andi Kleen a ?crit :
> >Can you please test this patch and see if it solves the problem?
> >
> >Thanks,
> >-Andi
> >
> >---
> >
> >oprofile: Fix p6 counter overflow check
> >
> >Fix the counter overflow check for CPUs with counter width > 32
> >
> >I had a similar change in a different patch that I didn't submit
> >and I didn't notice the problem earlier because it was always
> >tested together.
> >
> >Signed-off-by: Andi Kleen <[email protected]>
> >
>
> Thanks Andi
>
> Unfortunatly, there is still a freeze of NMIS after a while.
How long is the while?
I don't see this here on a Merom when I let it run for
10 minutes or so.
You might need to bisect it.
-Andi
Andi Kleen a ?crit :
> On Fri, Nov 07, 2008 at 04:13:42PM +0100, Eric Dumazet wrote:
>> Andi Kleen a ?crit :
>>> Can you please test this patch and see if it solves the problem?
>>>
>>> Thanks,
>>> -Andi
>>>
>>> ---
>>>
>>> oprofile: Fix p6 counter overflow check
>>>
>>> Fix the counter overflow check for CPUs with counter width > 32
>>>
>>> I had a similar change in a different patch that I didn't submit
>>> and I didn't notice the problem earlier because it was always
>>> tested together.
>>>
>>> Signed-off-by: Andi Kleen <[email protected]>
>>>
>> Thanks Andi
>>
>> Unfortunatly, there is still a freeze of NMIS after a while.
>
> How long is the while?
>
Less than 20 seconds if fully loaded with "tbench 8" on my 8 cpus machine.
With your patch, many events are handled by oprofile (about 10.000.000
last time I tried) before freeze.
Without your patch, freeze was very very fast, no more than 10.000 NMI...
> I don't see this here on a Merom when I let it run for
> 10 minutes or so.
>
> You might need to bisect it.
Too many bugs during bisection unfortunatly, machine doesnt boot :(
On 07.11.08 14:02:49, Andi Kleen wrote:
>
> Can you please test this patch and see if it solves the problem?
>
> Thanks,
> -Andi
>
> ---
>
> oprofile: Fix p6 counter overflow check
>
> Fix the counter overflow check for CPUs with counter width > 32
>
> I had a similar change in a different patch that I didn't submit
> and I didn't notice the problem earlier because it was always
> tested together.
>
> Signed-off-by: Andi Kleen <[email protected]>
Andi,
your patch fixes the initial bug that stops sampling immediately after
starting. The issue Eric mentions seems to be a different problem. I
am also not able to reproduce this. Your patch on top of rc3 continues
sampling at least for several minutes.
I will apply your fix.
Thanks Andi,
-Robert
--
Advanced Micro Devices, Inc.
Operating System Research Center
email: [email protected]
On Fri, Nov 07, 2008 at 05:12:33PM +0100, Eric Dumazet wrote:
> Andi Kleen a ?crit :
> >On Fri, Nov 07, 2008 at 04:13:42PM +0100, Eric Dumazet wrote:
> >>Andi Kleen a ?crit :
> >>>Can you please test this patch and see if it solves the problem?
> >>>
> >>>Thanks,
> >>>-Andi
> >>>
> >>>---
> >>>
> >>>oprofile: Fix p6 counter overflow check
> >>>
> >>>Fix the counter overflow check for CPUs with counter width > 32
> >>>
> >>>I had a similar change in a different patch that I didn't submit
> >>>and I didn't notice the problem earlier because it was always
> >>>tested together.
> >>>
> >>>Signed-off-by: Andi Kleen <[email protected]>
> >>>
> >>Thanks Andi
> >>
> >>Unfortunatly, there is still a freeze of NMIS after a while.
> >
> >How long is the while?
> >
>
> Less than 20 seconds if fully loaded with "tbench 8" on my 8 cpus machine.
I don't have tbench on the test box, but when I load it with a couple
of while true ; do true ; done loops it also continues sampling.
Are you sure it worked in 2.6.27?
-Andi
Andi Kleen a ?crit :
> On Fri, Nov 07, 2008 at 05:12:33PM +0100, Eric Dumazet wrote:
>> Andi Kleen a ?crit :
>>> On Fri, Nov 07, 2008 at 04:13:42PM +0100, Eric Dumazet wrote:
>>>> Andi Kleen a ?crit :
>>>>> Can you please test this patch and see if it solves the problem?
>>>>>
>>>>> Thanks,
>>>>> -Andi
>>>>>
>>>>> ---
>>>>>
>>>>> oprofile: Fix p6 counter overflow check
>>>>>
>>>>> Fix the counter overflow check for CPUs with counter width > 32
>>>>>
>>>>> I had a similar change in a different patch that I didn't submit
>>>>> and I didn't notice the problem earlier because it was always
>>>>> tested together.
>>>>>
>>>>> Signed-off-by: Andi Kleen <[email protected]>
>>>>>
>>>> Thanks Andi
>>>>
>>>> Unfortunatly, there is still a freeze of NMIS after a while.
>>> How long is the while?
>>>
>> Less than 20 seconds if fully loaded with "tbench 8" on my 8 cpus machine.
>
> I don't have tbench on the test box, but when I load it with a couple
> of while true ; do true ; done loops it also continues sampling.
>
> Are you sure it worked in 2.6.27?
You are right... I got same problem with 2.6.27.4, it seems...
$ grep NMI /proc/interrupts
NMI: 423689 79703 1576236 580137 1405669 1187909 2923037 5163299 Non-maskable interrupts
$ grep NMI /proc/interrupts
NMI: 423689 79703 1576236 580137 1405669 1187909 2923037 5163299 Non-maskable interrupts
I'll try to find the faulty commit.
Thank you