2020-02-14 21:11:28

by Marco Elver

Subject: [PATCH v2] kcsan, trace: Make KCSAN compatible with tracing

Previously the system would lock up if ftrace was enabled together with
KCSAN. This is due to recursion on reporting if the tracer code is
instrumented with KCSAN.

To avoid this for all types of tracing, disable KCSAN instrumentation
for all of kernel/trace.

Furthermore, since KCSAN relies on udelay() to introduce delay, we have
to disable ftrace for udelay() (currently done for x86) in case KCSAN is
used together with lockdep and ftrace. The reason is that it may corrupt
lockdep IRQ flags tracing state due to a peculiar case of recursion
(details in Makefile comment).

Signed-off-by: Marco Elver <[email protected]>
Reported-by: Qian Cai <[email protected]>
Cc: Paul E. McKenney <[email protected]>
Cc: Steven Rostedt <[email protected]>
---
v2:
* Fix KCSAN+lockdep+ftrace compatibility.
---
arch/x86/lib/Makefile | 5 +++++
kernel/kcsan/Makefile | 2 ++
kernel/trace/Makefile | 3 +++
3 files changed, 10 insertions(+)

diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile
index 432a077056775..6110bce7237bd 100644
--- a/arch/x86/lib/Makefile
+++ b/arch/x86/lib/Makefile
@@ -8,6 +8,11 @@ KCOV_INSTRUMENT_delay.o := n

# KCSAN uses udelay for introducing watchpoint delay; avoid recursion.
KCSAN_SANITIZE_delay.o := n
+ifdef CONFIG_KCSAN
+# In case KCSAN+lockdep+ftrace are enabled, disable ftrace for delay.o to avoid
+# lockdep -> [other libs] -> KCSAN -> udelay -> ftrace -> lockdep recursion.
+CFLAGS_REMOVE_delay.o = $(CC_FLAGS_FTRACE)
+endif

# Early boot use of cmdline; don't instrument it
ifdef CONFIG_AMD_MEM_ENCRYPT
diff --git a/kernel/kcsan/Makefile b/kernel/kcsan/Makefile
index df6b7799e4927..d4999b38d1be5 100644
--- a/kernel/kcsan/Makefile
+++ b/kernel/kcsan/Makefile
@@ -4,6 +4,8 @@ KCOV_INSTRUMENT := n
UBSAN_SANITIZE := n

CFLAGS_REMOVE_core.o = $(CC_FLAGS_FTRACE)
+CFLAGS_REMOVE_debugfs.o = $(CC_FLAGS_FTRACE)
+CFLAGS_REMOVE_report.o = $(CC_FLAGS_FTRACE)

CFLAGS_core.o := $(call cc-option,-fno-conserve-stack,) \
$(call cc-option,-fno-stack-protector,)
diff --git a/kernel/trace/Makefile b/kernel/trace/Makefile
index f9dcd19165fa2..6b601d88bf71e 100644
--- a/kernel/trace/Makefile
+++ b/kernel/trace/Makefile
@@ -6,6 +6,9 @@ ifdef CONFIG_FUNCTION_TRACER
ORIG_CFLAGS := $(KBUILD_CFLAGS)
KBUILD_CFLAGS = $(subst $(CC_FLAGS_FTRACE),,$(ORIG_CFLAGS))

+# Avoid recursion due to instrumentation.
+KCSAN_SANITIZE := n
+
ifdef CONFIG_FTRACE_SELFTEST
# selftest needs instrumentation
CFLAGS_trace_selftest_dynamic.o = $(CC_FLAGS_FTRACE)
--
2.25.0.265.gbab2e86ba0-goog

2020-02-14 23:40:38

by Paul E. McKenney

Subject: Re: [PATCH v2] kcsan, trace: Make KCSAN compatible with tracing

On Fri, Feb 14, 2020 at 10:10:35PM +0100, Marco Elver wrote:
> Previously the system would lock up if ftrace was enabled together with
> KCSAN. This is due to recursion on reporting if the tracer code is
> instrumented with KCSAN.
>
> To avoid this for all types of tracing, disable KCSAN instrumentation
> for all of kernel/trace.
>
> Furthermore, since KCSAN relies on udelay() to introduce delay, we have
> to disable ftrace for udelay() (currently done for x86) in case KCSAN is
> used together with lockdep and ftrace. The reason is that it may corrupt
> lockdep IRQ flags tracing state due to a peculiar case of recursion
> (details in Makefile comment).
>
> Signed-off-by: Marco Elver <[email protected]>
> Reported-by: Qian Cai <[email protected]>
> Cc: Paul E. McKenney <[email protected]>
> Cc: Steven Rostedt <[email protected]>

Queued for review and further testing, thank you!

Qian, does this also fix things for you?

Thanx, Paul

> ---
> v2:
> * Fix KCSAN+lockdep+ftrace compatibility.
> ---
> arch/x86/lib/Makefile | 5 +++++
> kernel/kcsan/Makefile | 2 ++
> kernel/trace/Makefile | 3 +++
> 3 files changed, 10 insertions(+)
>
> diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile
> index 432a077056775..6110bce7237bd 100644
> --- a/arch/x86/lib/Makefile
> +++ b/arch/x86/lib/Makefile
> @@ -8,6 +8,11 @@ KCOV_INSTRUMENT_delay.o := n
>
> # KCSAN uses udelay for introducing watchpoint delay; avoid recursion.
> KCSAN_SANITIZE_delay.o := n
> +ifdef CONFIG_KCSAN
> +# In case KCSAN+lockdep+ftrace are enabled, disable ftrace for delay.o to avoid
> +# lockdep -> [other libs] -> KCSAN -> udelay -> ftrace -> lockdep recursion.
> +CFLAGS_REMOVE_delay.o = $(CC_FLAGS_FTRACE)
> +endif
>
> # Early boot use of cmdline; don't instrument it
> ifdef CONFIG_AMD_MEM_ENCRYPT
> diff --git a/kernel/kcsan/Makefile b/kernel/kcsan/Makefile
> index df6b7799e4927..d4999b38d1be5 100644
> --- a/kernel/kcsan/Makefile
> +++ b/kernel/kcsan/Makefile
> @@ -4,6 +4,8 @@ KCOV_INSTRUMENT := n
> UBSAN_SANITIZE := n
>
> CFLAGS_REMOVE_core.o = $(CC_FLAGS_FTRACE)
> +CFLAGS_REMOVE_debugfs.o = $(CC_FLAGS_FTRACE)
> +CFLAGS_REMOVE_report.o = $(CC_FLAGS_FTRACE)
>
> CFLAGS_core.o := $(call cc-option,-fno-conserve-stack,) \
> $(call cc-option,-fno-stack-protector,)
> diff --git a/kernel/trace/Makefile b/kernel/trace/Makefile
> index f9dcd19165fa2..6b601d88bf71e 100644
> --- a/kernel/trace/Makefile
> +++ b/kernel/trace/Makefile
> @@ -6,6 +6,9 @@ ifdef CONFIG_FUNCTION_TRACER
> ORIG_CFLAGS := $(KBUILD_CFLAGS)
> KBUILD_CFLAGS = $(subst $(CC_FLAGS_FTRACE),,$(ORIG_CFLAGS))
>
> +# Avoid recursion due to instrumentation.
> +KCSAN_SANITIZE := n
> +
> ifdef CONFIG_FTRACE_SELFTEST
> # selftest needs instrumentation
> CFLAGS_trace_selftest_dynamic.o = $(CC_FLAGS_FTRACE)
> --
> 2.25.0.265.gbab2e86ba0-goog
>

2020-02-17 17:09:13

by Qian Cai

Subject: Re: [PATCH v2] kcsan, trace: Make KCSAN compatible with tracing

On Fri, 2020-02-14 at 15:40 -0800, Paul E. McKenney wrote:
> On Fri, Feb 14, 2020 at 10:10:35PM +0100, Marco Elver wrote:
> > Previously the system would lock up if ftrace was enabled together with
> > KCSAN. This is due to recursion on reporting if the tracer code is
> > instrumented with KCSAN.
> >
> > To avoid this for all types of tracing, disable KCSAN instrumentation
> > for all of kernel/trace.
> >
> > Furthermore, since KCSAN relies on udelay() to introduce delay, we have
> > to disable ftrace for udelay() (currently done for x86) in case KCSAN is
> > used together with lockdep and ftrace. The reason is that it may corrupt
> > lockdep IRQ flags tracing state due to a peculiar case of recursion
> > (details in Makefile comment).
> >
> > Signed-off-by: Marco Elver <[email protected]>
> > Reported-by: Qian Cai <[email protected]>
> > Cc: Paul E. McKenney <[email protected]>
> > Cc: Steven Rostedt <[email protected]>
>
> Queued for review and further testing, thank you!
>
> Qian, does this also fix things for you?

It works fine. Feel free to use,

Tested-by: Qian Cai <[email protected]>

>
> Thanx, Paul
>
> > ---
> > v2:
> > * Fix KCSAN+lockdep+ftrace compatibility.
> > ---
> > arch/x86/lib/Makefile | 5 +++++
> > kernel/kcsan/Makefile | 2 ++
> > kernel/trace/Makefile | 3 +++
> > 3 files changed, 10 insertions(+)
> >
> > diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile
> > index 432a077056775..6110bce7237bd 100644
> > --- a/arch/x86/lib/Makefile
> > +++ b/arch/x86/lib/Makefile
> > @@ -8,6 +8,11 @@ KCOV_INSTRUMENT_delay.o := n
> >
> > # KCSAN uses udelay for introducing watchpoint delay; avoid recursion.
> > KCSAN_SANITIZE_delay.o := n
> > +ifdef CONFIG_KCSAN
> > +# In case KCSAN+lockdep+ftrace are enabled, disable ftrace for delay.o to avoid
> > +# lockdep -> [other libs] -> KCSAN -> udelay -> ftrace -> lockdep recursion.
> > +CFLAGS_REMOVE_delay.o = $(CC_FLAGS_FTRACE)
> > +endif
> >
> > # Early boot use of cmdline; don't instrument it
> > ifdef CONFIG_AMD_MEM_ENCRYPT
> > diff --git a/kernel/kcsan/Makefile b/kernel/kcsan/Makefile
> > index df6b7799e4927..d4999b38d1be5 100644
> > --- a/kernel/kcsan/Makefile
> > +++ b/kernel/kcsan/Makefile
> > @@ -4,6 +4,8 @@ KCOV_INSTRUMENT := n
> > UBSAN_SANITIZE := n
> >
> > CFLAGS_REMOVE_core.o = $(CC_FLAGS_FTRACE)
> > +CFLAGS_REMOVE_debugfs.o = $(CC_FLAGS_FTRACE)
> > +CFLAGS_REMOVE_report.o = $(CC_FLAGS_FTRACE)
> >
> > CFLAGS_core.o := $(call cc-option,-fno-conserve-stack,) \
> > $(call cc-option,-fno-stack-protector,)
> > diff --git a/kernel/trace/Makefile b/kernel/trace/Makefile
> > index f9dcd19165fa2..6b601d88bf71e 100644
> > --- a/kernel/trace/Makefile
> > +++ b/kernel/trace/Makefile
> > @@ -6,6 +6,9 @@ ifdef CONFIG_FUNCTION_TRACER
> > ORIG_CFLAGS := $(KBUILD_CFLAGS)
> > KBUILD_CFLAGS = $(subst $(CC_FLAGS_FTRACE),,$(ORIG_CFLAGS))
> >
> > +# Avoid recursion due to instrumentation.
> > +KCSAN_SANITIZE := n
> > +
> > ifdef CONFIG_FTRACE_SELFTEST
> > # selftest needs instrumentation
> > CFLAGS_trace_selftest_dynamic.o = $(CC_FLAGS_FTRACE)
> > --
> > 2.25.0.265.gbab2e86ba0-goog
> >

2020-02-17 23:15:13

by Paul E. McKenney

Subject: Re: [PATCH v2] kcsan, trace: Make KCSAN compatible with tracing

On Mon, Feb 17, 2020 at 12:06:14PM -0500, Qian Cai wrote:
> On Fri, 2020-02-14 at 15:40 -0800, Paul E. McKenney wrote:
> > On Fri, Feb 14, 2020 at 10:10:35PM +0100, Marco Elver wrote:
> > > Previously the system would lock up if ftrace was enabled together with
> > > KCSAN. This is due to recursion on reporting if the tracer code is
> > > instrumented with KCSAN.
> > >
> > > To avoid this for all types of tracing, disable KCSAN instrumentation
> > > for all of kernel/trace.
> > >
> > > Furthermore, since KCSAN relies on udelay() to introduce delay, we have
> > > to disable ftrace for udelay() (currently done for x86) in case KCSAN is
> > > used together with lockdep and ftrace. The reason is that it may corrupt
> > > lockdep IRQ flags tracing state due to a peculiar case of recursion
> > > (details in Makefile comment).
> > >
> > > Signed-off-by: Marco Elver <[email protected]>
> > > Reported-by: Qian Cai <[email protected]>
> > > Cc: Paul E. McKenney <[email protected]>
> > > Cc: Steven Rostedt <[email protected]>
> >
> > Queued for review and further testing, thank you!
> >
> > Qian, does this also fix things for you?
>
> It works fine. Feel free to use,
>
> Tested-by: Qian Cai <[email protected]>

Applied, thank you!

Thanx, Paul

> > > ---
> > > v2:
> > > * Fix KCSAN+lockdep+ftrace compatibility.
> > > ---
> > > arch/x86/lib/Makefile | 5 +++++
> > > kernel/kcsan/Makefile | 2 ++
> > > kernel/trace/Makefile | 3 +++
> > > 3 files changed, 10 insertions(+)
> > >
> > > diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile
> > > index 432a077056775..6110bce7237bd 100644
> > > --- a/arch/x86/lib/Makefile
> > > +++ b/arch/x86/lib/Makefile
> > > @@ -8,6 +8,11 @@ KCOV_INSTRUMENT_delay.o := n
> > >
> > > # KCSAN uses udelay for introducing watchpoint delay; avoid recursion.
> > > KCSAN_SANITIZE_delay.o := n
> > > +ifdef CONFIG_KCSAN
> > > +# In case KCSAN+lockdep+ftrace are enabled, disable ftrace for delay.o to avoid
> > > +# lockdep -> [other libs] -> KCSAN -> udelay -> ftrace -> lockdep recursion.
> > > +CFLAGS_REMOVE_delay.o = $(CC_FLAGS_FTRACE)
> > > +endif
> > >
> > > # Early boot use of cmdline; don't instrument it
> > > ifdef CONFIG_AMD_MEM_ENCRYPT
> > > diff --git a/kernel/kcsan/Makefile b/kernel/kcsan/Makefile
> > > index df6b7799e4927..d4999b38d1be5 100644
> > > --- a/kernel/kcsan/Makefile
> > > +++ b/kernel/kcsan/Makefile
> > > @@ -4,6 +4,8 @@ KCOV_INSTRUMENT := n
> > > UBSAN_SANITIZE := n
> > >
> > > CFLAGS_REMOVE_core.o = $(CC_FLAGS_FTRACE)
> > > +CFLAGS_REMOVE_debugfs.o = $(CC_FLAGS_FTRACE)
> > > +CFLAGS_REMOVE_report.o = $(CC_FLAGS_FTRACE)
> > >
> > > CFLAGS_core.o := $(call cc-option,-fno-conserve-stack,) \
> > > $(call cc-option,-fno-stack-protector,)
> > > diff --git a/kernel/trace/Makefile b/kernel/trace/Makefile
> > > index f9dcd19165fa2..6b601d88bf71e 100644
> > > --- a/kernel/trace/Makefile
> > > +++ b/kernel/trace/Makefile
> > > @@ -6,6 +6,9 @@ ifdef CONFIG_FUNCTION_TRACER
> > > ORIG_CFLAGS := $(KBUILD_CFLAGS)
> > > KBUILD_CFLAGS = $(subst $(CC_FLAGS_FTRACE),,$(ORIG_CFLAGS))
> > >
> > > +# Avoid recursion due to instrumentation.
> > > +KCSAN_SANITIZE := n
> > > +
> > > ifdef CONFIG_FTRACE_SELFTEST
> > > # selftest needs instrumentation
> > > CFLAGS_trace_selftest_dynamic.o = $(CC_FLAGS_FTRACE)
> > > --
> > > 2.25.0.265.gbab2e86ba0-goog
> > >