Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <20201123132300.1759342-1-elver@google.com> <20201123135512.GM3021@hirez.programming.kicks-ass.net>
In-Reply-To: <20201123135512.GM3021@hirez.programming.kicks-ass.net>
From:   Marco Elver <elver@google.com>
Date:   Mon, 23 Nov 2020 16:17:00 +0100
Message-ID: <CANpmjNPwuq8Hph3oOyJCVgWQ_d-gOTPEOT3BpbR2pnm5LBeJbw@mail.gmail.com>
Subject: Re: [PATCH v2] kcsan: Avoid scheduler recursion by using
 non-instrumented preempt_{disable,enable}()
To:     Peter Zijlstra <peterz@infradead.org>
Cc:     "Paul E. McKenney" <paulmck@kernel.org>,
        Will Deacon <will@kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@kernel.org>,
        Mark Rutland <mark.rutland@arm.com>,
        Boqun Feng <boqun.feng@gmail.com>,
        Dmitry Vyukov <dvyukov@google.com>,
        kasan-dev <kasan-dev@googlegroups.com>,
        LKML <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Mon, 23 Nov 2020 at 14:55, Peter Zijlstra <peterz@infradead.org> wrote:
> On Mon, Nov 23, 2020 at 02:23:00PM +0100, Marco Elver wrote:
> > When enabling KCSAN for kernel/sched (remove KCSAN_SANITIZE := n from
> > kernel/sched/Makefile), with CONFIG_DEBUG_PREEMPT=y, we can observe
> > recursion due to:
> >
> >       check_access() [via instrumentation]
> >         kcsan_setup_watchpoint()
> >           reset_kcsan_skip()
> >             kcsan_prandom_u32_max()
> >               get_cpu_var()
> >                 preempt_disable()
> >                   preempt_count_add() [in kernel/sched/core.c]
> >                     check_access() [via instrumentation]
> >
> > Avoid this by rewriting kcsan_prandom_u32_max() to only use safe
> > versions of preempt_disable() and preempt_enable() that do not call into
> > scheduler code.
> >
> > Note, while this currently does not affect an unmodified kernel, it'd be
> > good to keep a KCSAN kernel working when KCSAN_SANITIZE := n is removed
> > from kernel/sched/Makefile to permit testing scheduler code with KCSAN
> > if desired.
> >
> > Fixes: cd290ec24633 ("kcsan: Use tracing-safe version of prandom")
> > Signed-off-by: Marco Elver <elver@google.com>
> > ---
> > v2:
> > * Update comment to also point out preempt_enable().
> > ---
> >  kernel/kcsan/core.c | 15 ++++++++++++---
> >  1 file changed, 12 insertions(+), 3 deletions(-)
> >
> > diff --git a/kernel/kcsan/core.c b/kernel/kcsan/core.c
> > index 3994a217bde7..10513f3e2349 100644
> > --- a/kernel/kcsan/core.c
> > +++ b/kernel/kcsan/core.c
> > @@ -284,10 +284,19 @@ should_watch(const volatile void *ptr, size_t size, int type, struct kcsan_ctx *
> >   */
> >  static u32 kcsan_prandom_u32_max(u32 ep_ro)
> >  {
> > -     struct rnd_state *state = &get_cpu_var(kcsan_rand_state);
> > -     const u32 res = prandom_u32_state(state);
> > +     struct rnd_state *state;
> > +     u32 res;
> > +
> > +     /*
> > +      * Avoid recursion with scheduler by using non-tracing versions of
> > +      * preempt_disable() and preempt_enable() that do not call into
> > +      * scheduler code.
> > +      */
> > +     preempt_disable_notrace();
> > +     state = raw_cpu_ptr(&kcsan_rand_state);
> > +     res = prandom_u32_state(state);
> > +     preempt_enable_no_resched_notrace();
>
> This is a preemption bug. Does preempt_enable_notrace() not work?

No it didn't, because we end up calling preempt_schedule_notrace(),
which again might end in recursion.

Normally we could surround this by
kcsan_disable_current/kcsan_enable_current(), but that doesn't work
because we have this sequence:

     reset_kcsan_skip();
     if (!kcsan_is_enabled())
         ...

to avoid underflowing the skip counter if KCSAN is disabled. That
could be solved by writing to the skip-counter twice: once with a
non-random value, and if KCSAN is enabled with a random value. Would
that be better?

And I'd like to avoid adding __no_kcsan to scheduler functions.

Any recommendation?

Thanks,
-- Marco


>
> >
> > -     put_cpu_var(kcsan_rand_state);
> >       return (u32)(((u64) res * ep_ro) >> 32);
> >  }
> >
> > --
> > 2.29.2.454.gaff20da3a2-goog
> >