Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
From:   Andy Lutomirski <luto@amacapital.net>
Mime-Version: 1.0 (1.0)
Subject: Re: Random shadow stack pointer corruption
Date:   Tue, 28 Jul 2020 17:56:20 -0700
Message-Id: <9CB08BDB-0245-474A-BE03-C529F659581F@amacapital.net>
References: <CAMe9rOr=WsOnKk3x8AK-iB5JLCjJ7FaDKXsFp01+3BbJvLZqMA@mail.gmail.com>
Cc:     Yu-cheng Yu <yu-cheng.yu@intel.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Andy Lutomirski <luto@kernel.org>,
        LKML <linux-kernel@vger.kernel.org>, X86 ML <x86@kernel.org>,
        Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Ingo Molnar <mingo@redhat.com>,
        "Ravi V. Shankar" <ravi.v.shankar@intel.com>,
        Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
        Tony Luck <tony.luck@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Peter Zijlstra <peterz@infradead.org>,
        Weijiang Yang <weijiang.yang@intel.com>
In-Reply-To: <CAMe9rOr=WsOnKk3x8AK-iB5JLCjJ7FaDKXsFp01+3BbJvLZqMA@mail.gmail.com>
To:     "H.J. Lu" <hjl.tools@gmail.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


> On Jul 28, 2020, at 5:36 PM, H.J. Lu <hjl.tools@gmail.com> wrote:
>=20
> =EF=BB=BFOn Sat, Jul 18, 2020 at 4:35 PM Yu-cheng Yu <yu-cheng.yu@intel.co=
m> wrote:
>>=20
>>> On Sat, 2020-07-18 at 15:41 -0700, Dave Hansen wrote:
>>> On 7/18/20 11:24 AM, Yu-cheng Yu wrote:
>>>> On Sat, 2020-07-18 at 11:00 -0700, Andy Lutomirski wrote:
>>>>> On Sat, Jul 18, 2020 at 10:58 AM Yu-cheng Yu <yu-cheng.yu@intel.com> w=
rote:
>>>>>> Hi,
>>>>>>=20
>>>>>> My shadow stack tests start to have random shadow stack pointer corru=
ption after
>>>>>> v5.7 (excluding).  The symptom looks like some locking issue or the k=
ernel is
>>>>>> confused about which CPU a task is on.  In later tip/master, this can=
 be
>>>>>> triggered by creating two tasks and each does continuous
>>>>>> pthread_create()/pthread_join().  If the kernel has max_cpus=3D1, the=
 issue goes
>>>>>> away.  I also checked XSAVES/XRSTORS, but this does not seem to be an=
 issue
>>>>>> coming from there.
>>>>>=20
>>>>> What do you mean "shadow stack pointer corruption"?  Is SSP itself
>>>>> corrupt while running in the kernel?  Is one of the MSRs getting
>>>>> corrupted?  Is the memory to which the shadow stack points getting
>>>>> corrupted? Is the CPU rejecting an attempt to change SSP?
>>>>=20
>>>> What I see is, a new thread after ret_from_fork() and iret back to ring=
-3,
>>>> its shadow stack pointer (MSR_IA32_PL3_SSP) is corrupted.
>>>=20
>>> Does corrupt mean random?  Or is it a valid stack address, just not for
>>> _this_ thread?  Or NULL?  Or is it a kernel address?  Have you tried
>>> tracing *ALL* the WRMSR's and XRSTOR's that write to the MSR?
>>=20
>> When a shadow stack address is changed, the address appears to be other t=
ask's.
>> I traced all WRMSR's and XRSTOR's.  I also verified there have not been a=
ny
>> XRSTORS from a wrong buffer.  When rc6 is tagged, I will re-base, test, a=
nd
>> share current patches.
>>=20
>=20
> We have identified that
>=20
> ommit 91eeafea1e4b7c95cc4f38af186d7d48fceef89a
> Author: Thomas Gleixner <tglx@linutronix.de>
> Date:   Thu May 21 22:05:28 2020 +0200
>=20
>    x86/entry: Switch page fault exception to IDTENTRY_RAW
>=20
>    Convert page fault exceptions to IDTENTRY_RAW:
>=20
>      - Implement the C entry point with DEFINE_IDTENTRY_RAW
>      - Add the CR2 read into the exception handler
>      - Add the idtentry_enter/exit_cond_rcu() invocations in
>        in the regular page fault handler and in the async PF
>        part.
>      - Emit the ASM stub with DECLARE_IDTENTRY_RAW
>      - Remove the ASM idtentry in 64-bit
>      - Remove the CR2 read from 64-bit
>      - Remove the open coded ASM entry code in 32-bit
>      - Fix up the XEN/PV code
>      - Remove the old prototypes
>=20
>    No functional change.
>=20
> triggered the shadow stack corruption when the process returned from sysca=
ll.
> SSP MSR somehow was changed between setting SSP MSR and IRET.    Could
> there be a page fault between setting SSP MSR and IRET?

Not upstream because there=E2=80=99s no SSP MSR.

>=20
> --=20
> H.J.