Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: [PATCH 1/2] x86/entry/64: Do not clear %rbx under Xen
From:   "M. Vefa Bicakci" <m.v.b@runbox.com>
To:     Andy Lutomirski <luto@kernel.org>
Cc:     LKML <linux-kernel@vger.kernel.org>,
        Dominik Brodowski <linux@dominikbrodowski.net>,
        Ingo Molnar <mingo@redhat.com>,
        "H. Peter Anvin" <hpa@zytor.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Boris Ostrovsky <boris.ostrovsky@oracle.com>,
        Juergen Gross <jgross@suse.com>,
        xen-devel@lists.xenproject.org, X86 ML <x86@kernel.org>,
        stable <stable@vger.kernel.org>,
        Dan Williams <dan.j.williams@intel.com>,
        Andi Kleen <ak@linux.intel.com>
References: <20180721194909.23903-1-m.v.b@runbox.com>
 <CALCETrX435cC7Hi2aTLVrBecZEEeJ3_TgQnTwbObNrt9MKvFzg@mail.gmail.com>
 <b9ad156c-45e9-da80-9cf0-eabbe5a1bbf2@runbox.com>
 <CALCETrXsNxYP7-w-OoiNdJWspTKCsr0i01e6NiwFk1O5jL6grA@mail.gmail.com>
 <ce0cf39e-102e-8a33-6d46-492cd34e9506@runbox.com>
Message-ID: <74f29b48-7522-c4f4-c2e7-d8e41fd88873@runbox.com>
Date:   Sat, 21 Jul 2018 21:20:35 -0400
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Thunderbird/52.9.1
MIME-Version: 1.0
In-Reply-To: <ce0cf39e-102e-8a33-6d46-492cd34e9506@runbox.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Language: en-CA
Content-Transfer-Encoding: 8bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On 07/21/2018 07:37 PM, M. Vefa Bicakci wrote:
> On 07/21/2018 07:30 PM, Andy Lutomirski wrote:
>> On Sat, Jul 21, 2018 at 4:19 PM, M. Vefa Bicakci <m.v.b@runbox.com> wrote:
>>> On 07/21/2018 05:45 PM, Andy Lutomirski wrote:
>>>>
>>>> On Sat, Jul 21, 2018 at 12:49 PM, M. Vefa Bicakci <m.v.b@runbox.com>
>>>> wrote:
>>>>>
>>>>> Commit 3ac6d8c787b8 ("x86/entry/64: Clear registers for
>>>>> exceptions/interrupts, to reduce speculation attack surface")
>>>>> unintendedly
>>>>> broke Xen PV virtual machines by clearing the %rbx register at the end of
>>>>> xen_failsafe_callback before the latter jumps to error_exit.
>>>>> error_exit expects the %rbx register to be a flag indicating whether
>>>>> there should be a return to kernel mode.
>>>>>
>>>>> This commit makes sure that the %rbx register is not cleared by
>>>>> the PUSH_AND_CLEAR_REGS macro, when the macro in question is instantiated
>>>>> by xen_failsafe_callback, to avoid the issue.
>>>>
>>>>
>>>> Seems like a genuine problem, but:
>>>>
>>>>> diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
>>>>> index c7449f377a77..96e8ff34129e 100644
>>>>> --- a/arch/x86/entry/entry_64.S
>>>>> +++ b/arch/x86/entry/entry_64.S
>>>>> @@ -1129,7 +1129,7 @@ ENTRY(xen_failsafe_callback)
>>>>> ????????? addq??? $0x30, %rsp
>>>>> ????????? UNWIND_HINT_IRET_REGS
>>>>> ????????? pushq?? $-1 /* orig_ax = -1 => not a system call */
>>>>> -?????? PUSH_AND_CLEAR_REGS
>>>>> +?????? PUSH_AND_CLEAR_REGS clear_rbx=0
>>>>> ????????? ENCODE_FRAME_POINTER
>>>>> ????????? jmp???? error_exit
>>>>
>>>>
>>>> The old code first set RBX to zero then, if frame pointers are on,
>>>> sets it to some special non-zero value, then crosses its fingers and
>>>> hopes for the best.? Your patched code just skips the zeroing part, so
>>>> RBX either contains the ENCODE_FRAME_POINTER result or is
>>>> uninitialized.
>>>>
>>>> How about actually initializing rbx to something sensible like, say, 1?
>>>
>>> Hello Andy,
>>>
>>> Thank you for the review! Apparently, I have not done my homework fully.
>>> I will test your suggestion and report back, most likely in a few hours.
>>>
>>> I have been testing with the next/linux-next tree's master branch
>>> (dated 20180720), and I noticed that ENCODE_FRAME_POINTER changes the
>>> frame pointer (i.e., RBP) register, as opposed to the RBX register,
>>> which the patch aims to avoid changing before jumping to error_exit.
>>> It is possible that I am missing something though -- I am not sure about
>>> the connection between the RBP and RBX registers.
>>
>> Sorry, brain fart on my part.
> 
> No problem! :-)
> 
>>> The change introduced by commit 3ac6d8c787b8 is in the excerpt below. Would
>>> it
>>> be valid to state that the original code had the same issue that you
>>> referred
>>> to (i.e., leaving the RBX register uninitialized)?
>>
>> Presumably.
>>
>> I would propose a rather different fix:
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/luto/linux.git/commit/?h=x86/pti&id=bb3d76b50c3bc78b67d79cf90d328f38a435c793
>>
>> Any chance you could test that and see if it fixes your problem?
> 
> Of course; I will report back with the result in a few hours.

Hello Andy,

I confirm that the commit at [1] resolves the issue in question as well.

To test, I first reverted my commit, applied your commit and verified that
the bug cannot be reproduced. Afterwards, I reverted your commit and
verified that the bug is reproducible.

I am not sure about the best way to document the bug I encountered in your
commit message, but in case you plan to have your commit merged, please
feel free to add a "Reported-and-tested-by: M. Vefa Bicakci <m.v.b@runbox.com>"
tag to the commit message, possibly with a link to this e-mail thread.

Finally, as I had mentioned in my commit message, this bug exists in all
kernel versions 4.14 and greater, so it would be nice if you could carbon-copy
"stable@vger.kernel.org" in the commit message as well.

Thank you,

Vefa

[1] https://git.kernel.org/pub/scm/linux/kernel/git/luto/linux.git/commit/?h=x86/pti&id=bb3d76b50c3bc78b67d79cf90d328f38a435c793