Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <CAP045Ar5CtqknH66i5ti6xOvo9cC9ib5v-5+3fFKcp_DW91hYw@mail.gmail.com>
 <20201026155521.GQ2594@hirez.programming.kicks-ass.net> <20201026160513.GC2651@hirez.programming.kicks-ass.net>
 <CAP045ApB_9h5Pp=a0L+taA6qFURrR6Se+W77Vb7A_VOWJNKfng@mail.gmail.com>
 <20201026163100.GR2594@hirez.programming.kicks-ass.net> <20201026165519.GD2651@hirez.programming.kicks-ass.net>
In-Reply-To: <20201026165519.GD2651@hirez.programming.kicks-ass.net>
From:   Kyle Huey <me@kylehuey.com>
Date:   Mon, 26 Oct 2020 10:12:30 -0700
Message-ID: <CAP045ArgMKSpt3Qa3TwzeE+xc78BJ5DPRazaP+bxNi570s+A=Q@mail.gmail.com>
Subject: Re: [REGRESSION] x86/debug: After PTRACE_SINGLESTEP DR_STEP is no
 longer reported in dr6
To:     Peter Zijlstra <peterz@infradead.org>
Cc:     open list <linux-kernel@vger.kernel.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        "maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)" <x86@kernel.org>,
        Linus Torvalds <torvalds@linux-foundation.org>,
        "Robert O'Callahan" <rocallahan@gmail.com>,
        Alexandre Chartre <alexandre.chartre@oracle.com>,
        "Paul E. McKenney" <paulmck@kernel.org>,
        Frederic Weisbecker <frederic@kernel.org>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Sean Christopherson <sean.j.christopherson@intel.com>,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Petr Mladek <pmladek@suse.com>,
        Joel Fernandes <joel@joelfernandes.org>,
        Steven Rostedt <rostedt@goodmis.org>,
        Boris Ostrovsky <boris.ostrovsky@oracle.com>,
        Juergen Gross <jgross@suse.com>,
        Brian Gerst <brgerst@gmail.com>,
        Andy Lutomirski <luto@kernel.org>,
        Josh Poimboeuf <jpoimboe@redhat.com>,
        Daniel Thompson <daniel.thompson@linaro.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Mon, Oct 26, 2020 at 9:55 AM Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Mon, Oct 26, 2020 at 05:31:00PM +0100, Peter Zijlstra wrote:
> > In that respect, I think the current virtual_dr6 = 0 is placed wrong, it
> > should only be in exc_debug_user(). The only 'problem' then is that we
> > seem to be able to loose BTF, but perhaps that is already an extant bug.
> >
> > Consider:
> >
> >  - perf: setup in-kernel #DB
> >  - tracer: ptrace(PTRACE_SINGLEBLOCK)
> >  - tracee: #DB on perf breakpoint, looses BTF
> >  - tracee .. never triggers actual blockstep
> >
> > Hmm ? Should we re-set BTF when TIF_BLOCKSTEP && !user_mode(regs) ?
>
> Something like so then.
>
> Or sould we also have the userspace #DB re-set BTF when it was !DR_STEP?
> I need to go untangle that ptrace stuff :/
>
> diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
> index 3c70fb34028b..31de8b0980ca 100644
> --- a/arch/x86/kernel/traps.c
> +++ b/arch/x86/kernel/traps.c
> @@ -793,19 +793,6 @@ static __always_inline unsigned long debug_read_clear_dr6(void)
>         set_debugreg(DR6_RESERVED, 6);
>         dr6 ^= DR6_RESERVED; /* Flip to positive polarity */
>
> -       /*
> -        * Clear the virtual DR6 value, ptrace routines will set bits here for
> -        * things we want signals for.
> -        */
> -       current->thread.virtual_dr6 = 0;
> -
> -       /*
> -        * The SDM says "The processor clears the BTF flag when it
> -        * generates a debug exception."  Clear TIF_BLOCKSTEP to keep
> -        * TIF_BLOCKSTEP in sync with the hardware BTF flag.
> -        */
> -       clear_thread_flag(TIF_BLOCKSTEP);
> -
>         return dr6;
>  }
>
> @@ -873,6 +860,20 @@ static __always_inline void exc_debug_kernel(struct pt_regs *regs,
>          */
>         WARN_ON_ONCE(user_mode(regs));
>
> +       if (test_thread_flag(TIF_BLOCKSTEP)) {
> +               /*
> +                * The SDM says "The processor clears the BTF flag when it
> +                * generates a debug exception." but PTRACE_BLOCKSTEP requested
> +                * it for userspace, but we just took a kernel #DB, so re-set
> +                * BTF.
> +                */
> +               unsigned long debugctl;
> +
> +               rdmsrl(MSR_IA32_DEBUGCTLMSR, debugctl);
> +               debugctl |= DEBUGCTLMSR_BTF;
> +               wrmsrl(MSR_IA32_DEBUGCTLMSR, debugctl);
> +       }
> +
>         /*
>          * Catch SYSENTER with TF set and clear DR_STEP. If this hit a
>          * watchpoint at the same time then that will still be handled.
> @@ -935,6 +936,26 @@ static __always_inline void exc_debug_user(struct pt_regs *regs,
>         irqentry_enter_from_user_mode(regs);
>         instrumentation_begin();
>
> +       /*
> +        * Clear the virtual DR6 value, ptrace routines will set bits here for
> +        * things we want signals for.
> +        */
> +       current->thread.virtual_dr6 = 0;
> +
> +       /*
> +        * If PTRACE requested SINGLE(BLOCK)STEP, make sure to reflect that in
> +        * the ptrace visible DR6 copy.
> +        */
> +       if (test_thread_flag(TIF_BLOCKSTEP) || test_thread_flag(TIF_SINGLESTEP))
> +               current->thread.virtual_dr6 |= (dr6 & DR_STEP);
> +
> +       /*
> +        * The SDM says "The processor clears the BTF flag when it
> +        * generates a debug exception."  Clear TIF_BLOCKSTEP to keep
> +        * TIF_BLOCKSTEP in sync with the hardware BTF flag.
> +        */
> +       clear_thread_flag(TIF_BLOCKSTEP);
> +
>         /*
>          * If dr6 has no reason to give us about the origin of this trap,
>          * then it's very likely the result of an icebp/int01 trap.

This looks good to me (at least the non BTF parts), and I'll test it
shortly, but especially now that clearing virtual_dr6 is moved to
exc_debug_user I still don't see why it's not ok to copy the entire
dr6 value into virtual_dr6 unconditionally.  Any extraneous dr6 state
from an in-kernel #DB would have been picked up and cleared already
when we entered exc_debug_kernel.

- Kyle