Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <c2bfe1aa390777b10d810d2b76a35b97fbd32a1c.camel@intel.com>
 <CALCETrUBDUKTcmmYD7BpZkL3869ELvha1PqOcScw4M-B_DQdiA@mail.gmail.com>
 <7653c6c74a4eee18b8bdc8262e0c0b5b95f9d771.camel@intel.com>
 <d4b9d6a1-d413-b044-1d4a-512bbbf06d78@intel.com> <14f3bee7569f229541852f61f0a1a88fcdec7249.camel@intel.com>
In-Reply-To: <14f3bee7569f229541852f61f0a1a88fcdec7249.camel@intel.com>
From:   "H.J. Lu" <hjl.tools@gmail.com>
Date:   Tue, 28 Jul 2020 17:35:37 -0700
Message-ID: <CAMe9rOr=WsOnKk3x8AK-iB5JLCjJ7FaDKXsFp01+3BbJvLZqMA@mail.gmail.com>
Subject: Re: Random shadow stack pointer corruption
To:     Yu-cheng Yu <yu-cheng.yu@intel.com>
Cc:     Dave Hansen <dave.hansen@intel.com>,
        Andy Lutomirski <luto@kernel.org>,
        LKML <linux-kernel@vger.kernel.org>, X86 ML <x86@kernel.org>,
        Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Ingo Molnar <mingo@redhat.com>,
        "Ravi V. Shankar" <ravi.v.shankar@intel.com>,
        Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
        Tony Luck <tony.luck@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Peter Zijlstra <peterz@infradead.org>,
        Weijiang Yang <weijiang.yang@intel.com>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Sat, Jul 18, 2020 at 4:35 PM Yu-cheng Yu <yu-cheng.yu@intel.com> wrote:
>
> On Sat, 2020-07-18 at 15:41 -0700, Dave Hansen wrote:
> > On 7/18/20 11:24 AM, Yu-cheng Yu wrote:
> > > On Sat, 2020-07-18 at 11:00 -0700, Andy Lutomirski wrote:
> > > > On Sat, Jul 18, 2020 at 10:58 AM Yu-cheng Yu <yu-cheng.yu@intel.com> wrote:
> > > > > Hi,
> > > > >
> > > > > My shadow stack tests start to have random shadow stack pointer corruption after
> > > > > v5.7 (excluding).  The symptom looks like some locking issue or the kernel is
> > > > > confused about which CPU a task is on.  In later tip/master, this can be
> > > > > triggered by creating two tasks and each does continuous
> > > > > pthread_create()/pthread_join().  If the kernel has max_cpus=1, the issue goes
> > > > > away.  I also checked XSAVES/XRSTORS, but this does not seem to be an issue
> > > > > coming from there.
> > > >
> > > > What do you mean "shadow stack pointer corruption"?  Is SSP itself
> > > > corrupt while running in the kernel?  Is one of the MSRs getting
> > > > corrupted?  Is the memory to which the shadow stack points getting
> > > > corrupted? Is the CPU rejecting an attempt to change SSP?
> > >
> > > What I see is, a new thread after ret_from_fork() and iret back to ring-3,
> > > its shadow stack pointer (MSR_IA32_PL3_SSP) is corrupted.
> >
> > Does corrupt mean random?  Or is it a valid stack address, just not for
> > _this_ thread?  Or NULL?  Or is it a kernel address?  Have you tried
> > tracing *ALL* the WRMSR's and XRSTOR's that write to the MSR?
>
> When a shadow stack address is changed, the address appears to be other task's.
> I traced all WRMSR's and XRSTOR's.  I also verified there have not been any
> XRSTORS from a wrong buffer.  When rc6 is tagged, I will re-base, test, and
> share current patches.
>

We have identified that

ommit 91eeafea1e4b7c95cc4f38af186d7d48fceef89a
Author: Thomas Gleixner <tglx@linutronix.de>
Date:   Thu May 21 22:05:28 2020 +0200

    x86/entry: Switch page fault exception to IDTENTRY_RAW

    Convert page fault exceptions to IDTENTRY_RAW:

      - Implement the C entry point with DEFINE_IDTENTRY_RAW
      - Add the CR2 read into the exception handler
      - Add the idtentry_enter/exit_cond_rcu() invocations in
        in the regular page fault handler and in the async PF
        part.
      - Emit the ASM stub with DECLARE_IDTENTRY_RAW
      - Remove the ASM idtentry in 64-bit
      - Remove the CR2 read from 64-bit
      - Remove the open coded ASM entry code in 32-bit
      - Fix up the XEN/PV code
      - Remove the old prototypes

    No functional change.

triggered the shadow stack corruption when the process returned from syscall.
SSP MSR somehow was changed between setting SSP MSR and IRET.    Could
there be a page fault between setting SSP MSR and IRET?

-- 
H.J.