Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Tue, 10 Mar 2020 13:40:03 -0400 (EDT)
From:   Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
To:     Thomas Gleixner <tglx@linutronix.de>
Cc:     linux-kernel <linux-kernel@vger.kernel.org>,
        Peter Zijlstra <peterz@infradead.org>,
        rostedt <rostedt@goodmis.org>,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Alexei Starovoitov <ast@kernel.org>,
        paulmck <paulmck@kernel.org>,
        "Joel Fernandes, Google" <joel@joelfernandes.org>,
        Frederic Weisbecker <frederic@kernel.org>
Message-ID: <1666704263.23816.1583862003925.JavaMail.zimbra@efficios.com>
In-Reply-To: <87imjc5f6a.fsf@nanos.tec.linutronix.de>
References: <87mu8p797b.fsf@nanos.tec.linutronix.de> <1403546357.21810.1583779060302.JavaMail.zimbra@efficios.com> <871rq171ca.fsf@nanos.tec.linutronix.de> <1489283504.23399.1583852595008.JavaMail.zimbra@efficios.com> <87imjc5f6a.fsf@nanos.tec.linutronix.de>
Subject: Re: Instrumentation and RCU
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Thread-Topic: Instrumentation and RCU
Thread-Index: 0nTX/RKhF+4jMm6zLS3dy/aLVWymRA==
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

----- On Mar 10, 2020, at 12:48 PM, Thomas Gleixner tglx@linutronix.de wrote:

> Mathieu Desnoyers <mathieu.desnoyers@efficios.com> writes:
[...]
> See the entry series I'm working on. Aside of moving all this nonsense
> into C-code it splits lockdep and tracing so it looks like this:
> 
>            lockdep_hardirqs_off();
>            user_exit_irqsoff();
>            __trace_hardirqs_off();
> 
> The latter uses regular RCU and not the scru/rcu_irq dance.
> 

Awesome :)

> 
>>>> If a tracer recurses, or if a tracer attempts to trace another tracer, the
>>>> instrumentation would break the recursion chain by preventing instrumentation
>>>> from firing. If we end up caring about tracers tracing other tracers, we could
>>>> have one distinct flag per tracer and let each tracer break the recursion chain.
>>>>
>>>> Having this flag per kernel stack rather than per CPU or per thread would
>>>> allow tracing of nested interrupt handlers (and NMIs), but would break
>>>> call chains both within the same stack or going through a trap. I think
>>>> it could be a nice complementary safety net to handle mishaps in a non-fatal
>>>> way.
>>> 
>>> That works as long as none of this uses breakpoint based patching to
>>> dynamically disable/enable stuff.
>>
>> I'm clearly missing something here. I was expecting the "in_tracing" flag trick
>> to be able to fix the breakpoint recursion issue. What is the problem I'm
>> missing
>> here ?
> 
> How do you "fix" that when you can't reach the tracepoint because you
> trip over a breakpoint and then while trying to fixup that stuff you hit
> another one?

I may still be missing something, but if the fixup code (AFAIU the code performing
the out-of-line single-stepping of the original instruction) belongs to a section
hidden from instrumentation, it should not be an issue.

The basic idea would be, e.g. pseudo-code for int3:

<int3>  <---- in section which cannot be instrumented
if (recursion_ctx->in_tracer) {
   single-step original instruction
   iret
}
[...] prepare stuff
recursion_ctx->in_tracer = true;
instr_allowed()

call external kernel functions (which can be instrumented)

instr_disallowed()
recursion_ctx->in_tracer = false;
single-step original instruction
iret

The purpose of the "in_tracer" flag is to protect whatever is done within external
kernel functions (which can be instrumented) from triggering tracer recursion. It
needs to be combined with hiding of early/late low-level entry/exit functions from
instrumentation (as you propose) to work.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com