Hi all,
These patch series provide a feature to stop Intel Processor Trace
(Intel PT) logging and save its registers in the memory on panic.
Intel PT is a new feature of Intel CPU "Broadwell", it captures
information about program execution flow. Here is a article about Intel
PT.
https://software.intel.com/en-us/blogs/2013/09/18/processor-tracing
Once Intel PT is enabled, the events which change program flow, like
branch instructions, exceptions, interruptions, traps and so on are
logged in the memory. This is very useful for debugging because we can
know the detailed behavior of software.
When kernel panic occurs while you are running a perf command with Intel
PT (with -e intel_pt// option), these patches disable Intel PT and save
its registers in the memory. After crash dump is captured by kdump, you
can retrieve Intel PT log buffer from vmcore and investigate kernel
behavior. I have not made a tool yet to salvage Intel PT log buffer from
vmcore, but I'll do once these patches are accepted.
changelog:
v2:
- Define function in intel_pt.h with static inline
v1:
https://lkml.org/lkml/2015/10/28/136
Background:
These patches are a part of patch series I posted before, the original
discussion is bellow.
x86: Intel Processor Trace Logger
v1: https://lkml.org/lkml/2015/7/29/6
v2: https://lkml.org/lkml/2015/9/8/24
The purpose of the original patches is introducing in-kernel logger
using Intel PT. To implement it I need to add some APIs to control perf
counter and ring buffer in kernel. Alexander Shishkin is working on such
APIs for his work to make use of Intel PT for process core dump. Apart
from such APIs, the feature to save Intel PT registers on panic is
helpful for normal perf command user as I described above, therefore I
separate the feature from original patches.
Takao Indoh (2):
perf/x86/intel/pt: Add interface to stop Intel PT logging
x86: Stop Intel PT before kdump starts
arch/x86/include/asm/intel_pt.h | 10 ++++++++++
arch/x86/kernel/cpu/perf_event_intel_pt.c | 9 +++++++++
arch/x86/kernel/crash.c | 11 +++++++++++
3 files changed, 30 insertions(+), 0 deletions(-)
create mode 100644 arch/x86/include/asm/intel_pt.h
This patch add a function for external component to stop Intel PT.
Basically this function is used when kernel panic occurs. When it is
called, intel_pt driver disables Intel PT and saves its registers using
pt_event_stop, which is also used by pmu.stop handler. This function
stops Intel PT on the cpu where it is working, therefore user need to
call it for each cpu to stop all logging.
Signed-off-by: Takao Indoh <[email protected]>
---
arch/x86/include/asm/intel_pt.h | 10 ++++++++++
arch/x86/kernel/cpu/perf_event_intel_pt.c | 9 +++++++++
2 files changed, 19 insertions(+), 0 deletions(-)
create mode 100644 arch/x86/include/asm/intel_pt.h
diff --git a/arch/x86/include/asm/intel_pt.h b/arch/x86/include/asm/intel_pt.h
new file mode 100644
index 0000000..e1a4117
--- /dev/null
+++ b/arch/x86/include/asm/intel_pt.h
@@ -0,0 +1,10 @@
+#ifndef _ASM_X86_INTEL_PT_H
+#define _ASM_X86_INTEL_PT_H
+
+#if defined(CONFIG_PERF_EVENTS) && defined(CONFIG_CPU_SUP_INTEL)
+void cpu_emergency_stop_pt(void);
+#else
+static inline void cpu_emergency_stop_pt(void) {}
+#endif
+
+#endif /* _ASM_X86_INTEL_PT_H */
diff --git a/arch/x86/kernel/cpu/perf_event_intel_pt.c b/arch/x86/kernel/cpu/perf_event_intel_pt.c
index 4216928..a638b5b 100644
--- a/arch/x86/kernel/cpu/perf_event_intel_pt.c
+++ b/arch/x86/kernel/cpu/perf_event_intel_pt.c
@@ -27,6 +27,7 @@
#include <asm/perf_event.h>
#include <asm/insn.h>
#include <asm/io.h>
+#include <asm/intel_pt.h>
#include "perf_event.h"
#include "intel_pt.h"
@@ -1125,6 +1126,14 @@ static int pt_event_init(struct perf_event *event)
return 0;
}
+void cpu_emergency_stop_pt(void)
+{
+ struct pt *pt = this_cpu_ptr(&pt_ctx);
+
+ if (pt->handle.event)
+ pt_event_stop(pt->handle.event, PERF_EF_UPDATE);
+}
+
static __init int pt_init(void)
{
int ret, cpu, prior_warn = 0;
--
1.7.1
This patch stops Intel PT logging and saves its registers in the memory
before kdump is started. This feature is needed to prevent Intel PT from
overwrite its log buffer after panic, and saved registers are needed to
find the last position where Intel PT wrote data. After crash dump is
captured by kdump, user can retrieve the log buffer from vmcore and use
it to investigate kernel behavior.
Signed-off-by: Takao Indoh <[email protected]>
---
arch/x86/kernel/crash.c | 11 +++++++++++
1 files changed, 11 insertions(+), 0 deletions(-)
diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
index 74ca2fe..5f383d2 100644
--- a/arch/x86/kernel/crash.c
+++ b/arch/x86/kernel/crash.c
@@ -35,6 +35,7 @@
#include <asm/cpu.h>
#include <asm/reboot.h>
#include <asm/virtext.h>
+#include <asm/intel_pt.h>
/* Alignment required for elf header segment */
#define ELF_CORE_HEADER_ALIGN 4096
@@ -127,6 +128,11 @@ static void kdump_nmi_callback(int cpu, struct pt_regs *regs)
cpu_emergency_vmxoff();
cpu_emergency_svm_disable();
+ /*
+ * Disable Intel PT to stop its logging
+ */
+ cpu_emergency_stop_pt();
+
disable_local_APIC();
}
@@ -172,6 +178,11 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
cpu_emergency_vmxoff();
cpu_emergency_svm_disable();
+ /*
+ * Disable Intel PT to stop its logging
+ */
+ cpu_emergency_stop_pt();
+
#ifdef CONFIG_X86_IO_APIC
/* Prevent crash_kexec() from deadlocking on ioapic_lock. */
ioapic_zap_locks();
--
1.7.1
Ping, any comments on these patches?
Thanks,
Takao Indoh
On 2015/11/04 14:22, Takao Indoh wrote:
> Hi all,
>
> These patch series provide a feature to stop Intel Processor Trace
> (Intel PT) logging and save its registers in the memory on panic.
>
> Intel PT is a new feature of Intel CPU "Broadwell", it captures
> information about program execution flow. Here is a article about Intel
> PT.
> https://software.intel.com/en-us/blogs/2013/09/18/processor-tracing
>
> Once Intel PT is enabled, the events which change program flow, like
> branch instructions, exceptions, interruptions, traps and so on are
> logged in the memory. This is very useful for debugging because we can
> know the detailed behavior of software.
>
> When kernel panic occurs while you are running a perf command with Intel
> PT (with -e intel_pt// option), these patches disable Intel PT and save
> its registers in the memory. After crash dump is captured by kdump, you
> can retrieve Intel PT log buffer from vmcore and investigate kernel
> behavior. I have not made a tool yet to salvage Intel PT log buffer from
> vmcore, but I'll do once these patches are accepted.
>
> changelog:
> v2:
> - Define function in intel_pt.h with static inline
>
> v1:
> https://lkml.org/lkml/2015/10/28/136
>
>
> Background:
> These patches are a part of patch series I posted before, the original
> discussion is bellow.
>
> x86: Intel Processor Trace Logger
> v1: https://lkml.org/lkml/2015/7/29/6
> v2: https://lkml.org/lkml/2015/9/8/24
>
> The purpose of the original patches is introducing in-kernel logger
> using Intel PT. To implement it I need to add some APIs to control perf
> counter and ring buffer in kernel. Alexander Shishkin is working on such
> APIs for his work to make use of Intel PT for process core dump. Apart
> from such APIs, the feature to save Intel PT registers on panic is
> helpful for normal perf command user as I described above, therefore I
> separate the feature from original patches.
>
> Takao Indoh (2):
> perf/x86/intel/pt: Add interface to stop Intel PT logging
> x86: Stop Intel PT before kdump starts
>
> arch/x86/include/asm/intel_pt.h | 10 ++++++++++
> arch/x86/kernel/cpu/perf_event_intel_pt.c | 9 +++++++++
> arch/x86/kernel/crash.c | 11 +++++++++++
> 3 files changed, 30 insertions(+), 0 deletions(-)
> create mode 100644 arch/x86/include/asm/intel_pt.h
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
On Thu, Nov 12, 2015 at 09:05:11PM +0900, Takao Indoh wrote:
> Ping, any comments on these patches?
>
I've taken them, they should appear in tip sometime after the merge
window closes.
On 2015/11/12 21:38, Peter Zijlstra wrote:
> On Thu, Nov 12, 2015 at 09:05:11PM +0900, Takao Indoh wrote:
>> Ping, any comments on these patches?
>>
>
> I've taken them, they should appear in tip sometime after the merge
> window closes.
>
Ok, thanks.
Thanks,
Takao Indoh
Commit-ID: 24cc12b17679f8e9046746f92fd377f589efc163
Gitweb: http://git.kernel.org/tip/24cc12b17679f8e9046746f92fd377f589efc163
Author: Takao Indoh <[email protected]>
AuthorDate: Wed, 4 Nov 2015 14:22:32 +0900
Committer: Ingo Molnar <[email protected]>
CommitDate: Mon, 23 Nov 2015 09:58:26 +0100
perf/x86/intel/pt: Add interface to stop Intel PT logging
This patch add a function for external components to stop Intel PT.
Basically this function is used when kernel panic occurs. When it is
called, the intel_pt driver disables Intel PT and saves its registers
using pt_event_stop(), which is also used by pmu.stop handler.
This function stops Intel PT on the CPU where it is working, therefore
users of it need to call it for each CPU to stop all logging.
Signed-off-by: Takao Indoh <[email protected]>
Signed-off-by: Peter Zijlstra (Intel) <[email protected]>
Cc: Alexander Shishkin<[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: H.Peter Anvin <[email protected]>
Cc: Jiri Olsa <[email protected]>
Cc: Linus Torvalds <[email protected]>
Cc: Mike Galbraith <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Stephane Eranian <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Vince Weaver <[email protected]>
Cc: Vivek Goyal <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Ingo Molnar <[email protected]>
---
arch/x86/include/asm/intel_pt.h | 10 ++++++++++
arch/x86/kernel/cpu/perf_event_intel_pt.c | 9 +++++++++
2 files changed, 19 insertions(+)
diff --git a/arch/x86/include/asm/intel_pt.h b/arch/x86/include/asm/intel_pt.h
new file mode 100644
index 0000000..e1a4117
--- /dev/null
+++ b/arch/x86/include/asm/intel_pt.h
@@ -0,0 +1,10 @@
+#ifndef _ASM_X86_INTEL_PT_H
+#define _ASM_X86_INTEL_PT_H
+
+#if defined(CONFIG_PERF_EVENTS) && defined(CONFIG_CPU_SUP_INTEL)
+void cpu_emergency_stop_pt(void);
+#else
+static inline void cpu_emergency_stop_pt(void) {}
+#endif
+
+#endif /* _ASM_X86_INTEL_PT_H */
diff --git a/arch/x86/kernel/cpu/perf_event_intel_pt.c b/arch/x86/kernel/cpu/perf_event_intel_pt.c
index 868e119..c0bbd10 100644
--- a/arch/x86/kernel/cpu/perf_event_intel_pt.c
+++ b/arch/x86/kernel/cpu/perf_event_intel_pt.c
@@ -27,6 +27,7 @@
#include <asm/perf_event.h>
#include <asm/insn.h>
#include <asm/io.h>
+#include <asm/intel_pt.h>
#include "perf_event.h"
#include "intel_pt.h"
@@ -1122,6 +1123,14 @@ static int pt_event_init(struct perf_event *event)
return 0;
}
+void cpu_emergency_stop_pt(void)
+{
+ struct pt *pt = this_cpu_ptr(&pt_ctx);
+
+ if (pt->handle.event)
+ pt_event_stop(pt->handle.event, PERF_EF_UPDATE);
+}
+
static __init int pt_init(void)
{
int ret, cpu, prior_warn = 0;
Commit-ID: da06a43d3f3f3df87416f654fe15d29fecb5e321
Gitweb: http://git.kernel.org/tip/da06a43d3f3f3df87416f654fe15d29fecb5e321
Author: Takao Indoh <[email protected]>
AuthorDate: Wed, 4 Nov 2015 14:22:33 +0900
Committer: Ingo Molnar <[email protected]>
CommitDate: Mon, 23 Nov 2015 09:58:26 +0100
perf, x86: Stop Intel PT before kdump starts
This patch stops Intel PT logging and saves its registers in memory
before kdump is started. This feature is needed to prevent Intel PT from
overwriting its log buffer after panic, and saved registers are needed to
find the last position where Intel PT wrote data.
After the crash dump is captured by kdump, users can retrieve the log buffer
from the vmcore and use it to investigate bad kernel behavior.
Signed-off-by: Takao Indoh <[email protected]>
Signed-off-by: Peter Zijlstra (Intel) <[email protected]>
Cc: Alexander Shishkin<[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: H.Peter Anvin <[email protected]>
Cc: Jiri Olsa <[email protected]>
Cc: Linus Torvalds <[email protected]>
Cc: Mike Galbraith <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Stephane Eranian <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Vince Weaver <[email protected]>
Cc: Vivek Goyal <[email protected]>
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Ingo Molnar <[email protected]>
---
arch/x86/kernel/crash.c | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
index 2c1910f..58f3431 100644
--- a/arch/x86/kernel/crash.c
+++ b/arch/x86/kernel/crash.c
@@ -35,6 +35,7 @@
#include <asm/cpu.h>
#include <asm/reboot.h>
#include <asm/virtext.h>
+#include <asm/intel_pt.h>
/* Alignment required for elf header segment */
#define ELF_CORE_HEADER_ALIGN 4096
@@ -125,6 +126,11 @@ static void kdump_nmi_callback(int cpu, struct pt_regs *regs)
cpu_emergency_vmxoff();
cpu_emergency_svm_disable();
+ /*
+ * Disable Intel PT to stop its logging
+ */
+ cpu_emergency_stop_pt();
+
disable_local_APIC();
}
@@ -169,6 +175,11 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
cpu_emergency_vmxoff();
cpu_emergency_svm_disable();
+ /*
+ * Disable Intel PT to stop its logging
+ */
+ cpu_emergency_stop_pt();
+
#ifdef CONFIG_X86_IO_APIC
/* Prevent crash_kexec() from deadlocking on ioapic_lock. */
ioapic_zap_locks();