On Thu, Jan 18, 2018 at 02:48:23PM +0100, Peter Zijlstra wrote:
> From: Thomas Gleixner <[email protected]>
>
> Add the minimal infrastructure to control the speculation control feature.
>
> - Integrate it into the spectre_v2 coammand line parser and the mitigation
> selector function. The conditional selector function is a placeholder
> right now, which needs to be expanded with CPU specific decision
> functions.
>
> - Provide a static key for the actual code control.
>
> - Provide a init function which is called after jump label patching is
> functional.
>
> - Provide an interface for the late micro code loader to allow late
> discovery of the IBRS support. Not yet functional.
>
> [peterz: fixed Makefile]
>
> Signed-off-by: Thomas Gleixner <[email protected]>
> Signed-off-by: Peter Zijlstra (Intel) <[email protected]>
> ---
> Documentation/admin-guide/kernel-parameters.txt | 1
> arch/x86/include/asm/nospec-branch.h | 5 +++
> arch/x86/kernel/cpu/Makefile | 1
> arch/x86/kernel/cpu/bugs.c | 26 +++++++++++++++++-
> arch/x86/kernel/cpu/specctrl.c | 33 ++++++++++++++++++++++++
> 5 files changed, 64 insertions(+), 2 deletions(-)
>
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -3932,6 +3932,7 @@
> retpoline - replace indirect branches
> retpoline,generic - google's original retpoline
> retpoline,amd - AMD-specific minimal thunk
> + ibrs - Intel: Indirect Branch Restricted Speculation
Are there plans to add spectre_v2=ibrs_always to prevent SMT-based
attacks?
> --- /dev/null
> +++ b/arch/x86/kernel/cpu/specctrl.c
> @@ -0,0 +1,33 @@
> +// SPDX-License-Identifier: GPL-2.0
> +
> +#include <asm/cpufeature.h>
> +#include <asm/cpufeatures.h>
> +#include <asm/nospec-branch.h>
> +
> +static inline void specctrl_enable_ibrs(void)
> +{
> + setup_force_cpu_cap(X86_FEATURE_IBRS);
> +}
"spec_ctrl" seems much more readable than specctrl (for both function
and file names). And also more consistent with the SPEC_CTRL MSR and
FEATURE names.
--
Josh
On 01/18/2018 08:37 AM, Josh Poimboeuf wrote:
>>
>> --- a/Documentation/admin-guide/kernel-parameters.txt
>> +++ b/Documentation/admin-guide/kernel-parameters.txt
>> @@ -3932,6 +3932,7 @@
>> retpoline - replace indirect branches
>> retpoline,generic - google's original retpoline
>> retpoline,amd - AMD-specific minimal thunk
>> + ibrs - Intel: Indirect Branch Restricted Speculation
> Are there plans to add spectre_v2=ibrs_always to prevent SMT-based
> attacks?
What does "ibrs_always" mean to you?
There is a second bit in the MSR (STIBP) that is intended to keep
hyperthreads from influencing each-other. That is behavior is implicit
when IBRS is enabled.
I think ibrs_always *should* probably be kept to refer to the future
CPUs that can safely leave IBRS enabled all the time.
On 18/01/2018 18:08, Dave Hansen wrote:
> On 01/18/2018 08:37 AM, Josh Poimboeuf wrote:
>>>
>>> --- a/Documentation/admin-guide/kernel-parameters.txt
>>> +++ b/Documentation/admin-guide/kernel-parameters.txt
>>> @@ -3932,6 +3932,7 @@
>>> retpoline - replace indirect branches
>>> retpoline,generic - google's original retpoline
>>> retpoline,amd - AMD-specific minimal thunk
>>> + ibrs - Intel: Indirect Branch Restricted Speculation
>> Are there plans to add spectre_v2=ibrs_always to prevent SMT-based
>> attacks?
>
> What does "ibrs_always" mean to you?
>
> There is a second bit in the MSR (STIBP) that is intended to keep
> hyperthreads from influencing each-other. That is behavior is implicit
> when IBRS is enabled.
Yeah, I think we should have a mode to always leave that enabled, or
always set IBRS=1.
> I think ibrs_always *should* probably be kept to refer to the future
> CPUs that can safely leave IBRS enabled all the time.
Is that "safely" or "without throwing performance down the drain"?
Does "always IBRS=1" *hinder* the mitigation on existing processor, as
long as you reset IBRS=1 on kernel entry and vmexit? Or is it just slow?
Paolo
On Thu, Jan 18, 2018 at 06:12:36PM +0100, Paolo Bonzini wrote:
> On 18/01/2018 18:08, Dave Hansen wrote:
> > On 01/18/2018 08:37 AM, Josh Poimboeuf wrote:
> >>>
> >>> --- a/Documentation/admin-guide/kernel-parameters.txt
> >>> +++ b/Documentation/admin-guide/kernel-parameters.txt
> >>> @@ -3932,6 +3932,7 @@
> >>> retpoline - replace indirect branches
> >>> retpoline,generic - google's original retpoline
> >>> retpoline,amd - AMD-specific minimal thunk
> >>> + ibrs - Intel: Indirect Branch Restricted Speculation
> >> Are there plans to add spectre_v2=ibrs_always to prevent SMT-based
> >> attacks?
> >
> > What does "ibrs_always" mean to you?
Maybe ibrs_always isn't the best name. Basically we need an option to
protect user-user attacks via SMT.
It could be implemented with IBRS=1, or STIBP, or as part of the
mythical IBRS_ATT.
Maybe a 'user_smt' option, which could be appended to existing
'retpoline' or 'ibrs' options? Like spectre_v2=retpoline,user_smt or
spectre_v2=ibrs,user_smt?
> > There is a second bit in the MSR (STIBP) that is intended to keep
> > hyperthreads from influencing each-other. That is behavior is implicit
> > when IBRS is enabled.
Does this bit exist yet? I've never seen any patches for it.
> Yeah, I think we should have a mode to always leave that enabled, or
> always set IBRS=1.
>
> > I think ibrs_always *should* probably be kept to refer to the future
> > CPUs that can safely leave IBRS enabled all the time.
>
> Is that "safely" or "without throwing performance down the drain"?
>
> Does "always IBRS=1" *hinder* the mitigation on existing processor, as
> long as you reset IBRS=1 on kernel entry and vmexit? Or is it just slow?
Yes, enquiring minds want to know...
--
Josh
On Thu, Jan 18, 2018 at 12:24:31PM -0600, Josh Poimboeuf wrote:
> On Thu, Jan 18, 2018 at 06:12:36PM +0100, Paolo Bonzini wrote:
> > On 18/01/2018 18:08, Dave Hansen wrote:
> > > On 01/18/2018 08:37 AM, Josh Poimboeuf wrote:
> > >>>
> > >>> --- a/Documentation/admin-guide/kernel-parameters.txt
> > >>> +++ b/Documentation/admin-guide/kernel-parameters.txt
> > >>> @@ -3932,6 +3932,7 @@
> > >>> retpoline - replace indirect branches
> > >>> retpoline,generic - google's original retpoline
> > >>> retpoline,amd - AMD-specific minimal thunk
> > >>> + ibrs - Intel: Indirect Branch Restricted Speculation
> > >> Are there plans to add spectre_v2=ibrs_always to prevent SMT-based
> > >> attacks?
> > >
> > > What does "ibrs_always" mean to you?
>
> Maybe ibrs_always isn't the best name. Basically we need an option to
> protect user-user attacks via SMT.
>
> It could be implemented with IBRS=1, or STIBP, or as part of the
> mythical IBRS_ATT.
User stibp or user ibrs would be different things, both would be valid
for different use cases, and the user stibp should perform better.
Leaving ibrs on when returning from kernel to userland (or setting
ibrs if kernel used retpolines instead of ibrs) achieves stronger
semantics than just setting SPEC_CTRL with stibp when returning to
userland.
That is true no matter if kernel is using retpolines or ibrs.
IBRS is semantically equivalent to "STIBP; IBPB", so user_ibrs is
always inclusive of user_stibp.
Said that the CPU should better achieve such semantics without really
internally issuing an IBPB of course, but you can think at the current
IBRS as "STIBP; IBPB". That IBPB immediately after the STIBP makes a
difference to the non HT attacks possible on host userland.
user_smt wouldn't solve all cases that user_ibrs solves, but it'd be
ideal if critical user apps are built with retpolines and the only
concern left is a HT/SMT attack on those only need to care about
HT/SMT.
To begin with, user_ibrs would be more important than user_stibp.
On a side note: stibp isn't always available, it requires a new cpuid
check on bit 27 too, you can still write to it but it won't #gp, on
some CPUs it's simply implicit and you can write to it, but it's a
noop. I haven't figured exactly to differentiate when it's disabled or
implicitly enabled when not enumerated in cpuid.
On Thu, Jan 18, 2018 at 11:08 AM, Andrea Arcangeli <[email protected]> wrote:
> On Thu, Jan 18, 2018 at 12:24:31PM -0600, Josh Poimboeuf wrote:
>> On Thu, Jan 18, 2018 at 06:12:36PM +0100, Paolo Bonzini wrote:
>> > On 18/01/2018 18:08, Dave Hansen wrote:
>> > > On 01/18/2018 08:37 AM, Josh Poimboeuf wrote:
>> > >>>
>> > >>> --- a/Documentation/admin-guide/kernel-parameters.txt
>> > >>> +++ b/Documentation/admin-guide/kernel-parameters.txt
>> > >>> @@ -3932,6 +3932,7 @@
>> > >>> retpoline - replace indirect branches
>> > >>> retpoline,generic - google's original retpoline
>> > >>> retpoline,amd - AMD-specific minimal thunk
>> > >>> + ibrs - Intel: Indirect Branch Restricted Speculation
>> > >> Are there plans to add spectre_v2=ibrs_always to prevent SMT-based
>> > >> attacks?
>> > >
>> > > What does "ibrs_always" mean to you?
>>
>> Maybe ibrs_always isn't the best name. Basically we need an option to
>> protect user-user attacks via SMT.
>>
>> It could be implemented with IBRS=1, or STIBP, or as part of the
>> mythical IBRS_ATT.
>
> User stibp or user ibrs would be different things, both would be valid
> for different use cases, and the user stibp should perform better.
>
> Leaving ibrs on when returning from kernel to userland (or setting
> ibrs if kernel used retpolines instead of ibrs) achieves stronger
> semantics than just setting SPEC_CTRL with stibp when returning to
> userland.
I read the whitepaper that documented the new MSRs a couple days ago
and I'm now completely unable to find it. If anyone could send the
link, that would be great.
From memory, however, the docs were quite clear that setting leaving
IBRS set when entering user mode or guest mode does not guarantee any
particular protection unless an additional CPUID bit (the name of
which I forget) is set, and that current CPUs will *not* get that bit
set by microcode update. IOW the protection given is that, if you set
IBRS bit zero after entry to kernel mode, you are protected until you
re-enter user mode. When you're in user mode, you're not protected.
When you return back to kernel mode, you *still* aren't protected no
matter what value is in the MSR until you write 1 again.
>
> That is true no matter if kernel is using retpolines or ibrs.
>
> IBRS is semantically equivalent to "STIBP; IBPB", so user_ibrs is
> always inclusive of user_stibp.
Are you quite sure? I had the impression that IBPB was much slower
than writing 1 to IBRS and that writing 1 to IBRS has much more
limited effects.
On 18/01/2018 23:25, Andy Lutomirski wrote:
> On Thu, Jan 18, 2018 at 11:08 AM, Andrea Arcangeli <[email protected]> wrote:
>> On Thu, Jan 18, 2018 at 12:24:31PM -0600, Josh Poimboeuf wrote:
>>> On Thu, Jan 18, 2018 at 06:12:36PM +0100, Paolo Bonzini wrote:
>>>> On 18/01/2018 18:08, Dave Hansen wrote:
>>>>> On 01/18/2018 08:37 AM, Josh Poimboeuf wrote:
>>>>>>> --- a/Documentation/admin-guide/kernel-parameters.txt
>>>>>>> +++ b/Documentation/admin-guide/kernel-parameters.txt
>>>>>>> @@ -3932,6 +3932,7 @@
>>>>>>> retpoline - replace indirect branches
>>>>>>> retpoline,generic - google's original retpoline
>>>>>>> retpoline,amd - AMD-specific minimal thunk
>>>>>>> + ibrs - Intel: Indirect Branch Restricted Speculation
>>>>>> Are there plans to add spectre_v2=ibrs_always to prevent SMT-based
>>>>>> attacks?
>>>>> What does "ibrs_always" mean to you?
>>> Maybe ibrs_always isn't the best name. Basically we need an option to
>>> protect user-user attacks via SMT.
>>>
>>> It could be implemented with IBRS=1, or STIBP, or as part of the
>>> mythical IBRS_ATT.
>> User stibp or user ibrs would be different things, both would be valid
>> for different use cases, and the user stibp should perform better.
>>
>> Leaving ibrs on when returning from kernel to userland (or setting
>> ibrs if kernel used retpolines instead of ibrs) achieves stronger
>> semantics than just setting SPEC_CTRL with stibp when returning to
>> userland.
> I read the whitepaper that documented the new MSRs a couple days ago
> and I'm now completely unable to find it. If anyone could send the
> link, that would be great.
https://software.intel.com/sites/default/files/managed/c5/63/336996-Speculative-Execution-Side-Channel-Mitigations.pdf
~Andrew
Hello,
On Thu, Jan 18, 2018 at 03:25:25PM -0800, Andy Lutomirski wrote:
> I read the whitepaper that documented the new MSRs a couple days ago
> and I'm now completely unable to find it. If anyone could send the
> link, that would be great.
I see Andrew posted a link.
> From memory, however, the docs were quite clear that setting leaving
> IBRS set when entering user mode or guest mode does not guarantee any
> particular protection unless an additional CPUID bit (the name of
> which I forget) is set, and that current CPUs will *not* get that bit
My current understanding is that with SPEC_CTRL alone set in cpuid,
IBRS is meaningful, other bits don't matter.
> set by microcode update. IOW the protection given is that, if you set
> IBRS bit zero after entry to kernel mode, you are protected until you
> re-enter user mode. When you're in user mode, you're not protected.
If you leave IBRS set while in user mode, userland is protected as
strong as kernel mode.
Of course kernel can attack usermode through spectre variant#2 if it
wants :), but usermode has to trust the kernel to begin with, just
like guest mode has to trust the host kernel.
If you leave IBRS set, guest mode (including guest usermode) cannot
attack host userland, host userland cannot attack other host userland
and no HT is possible either.
Note guest kernel attack on host userland is not as concerning but
it's possible too, as long as you use host dm-crypt instead of qcow2
encryption on host, it's not a major concern. guest userland attack on
host userland is more of a concern because if it can be mounted
successfully, it would dump all guest memory making it impossible for
the guest to defend itself from its own userland (unless it uses ibpb
2 of course which calls IBPB at guest user to kernel entry instead of
IBRS). IBRS isn't enough in guest because like retpoline it doesn't
guarantee an IBPB.
Because these are only theoretical and there's no PoC IBRS is not
enabled by default in userland of course. However retpolining qemu
would be a low overhead sure solution to this that would avoid having
to set ibrs in userland to achieve the same.
> When you return back to kernel mode, you *still* aren't protected no
> matter what value is in the MSR until you write 1 again.
When you return to kernel mode you've to call IBRS again even if it
was left set, because there's a higher-privilege mode change. That's
equivalent to calling only IBPB and leaving STIBP set (only way to
understand the locations where IBRS has to be set is to imagine IBRS
as a strict "STIBP; IBPB").
specs are very explicit that it's very meaningful to set IBRS even if
already set.
> > That is true no matter if kernel is using retpolines or ibrs.
> >
> > IBRS is semantically equivalent to "STIBP; IBPB", so user_ibrs is
> > always inclusive of user_stibp.
>
> Are you quite sure? I had the impression that IBPB was much slower
Nothing in the specs says that IBRS is a "STIBP; IBPB" equivalent, but
if you want to find where to set IBRS, yes, I'm quite sure, you've to
think it like it.
> than writing 1 to IBRS and that writing 1 to IBRS has much more
> limited effects.
I think I already partly answered it already in the sentence that
followed the one you quoted, but I should elaborate it.
The semantics to use depending what you're trying to solve because it
can have both.
"STIBP; IBPB" could be called the coarse semantics and you have to
imagine IBRS as "STIBP; IBPB" whenever you are checking where to write
IBRS, or you risk missing places where you have to write IBRS even if
it is already set.
As opposed when you check when you need to leave IBRS set (note: after
it was already set in all places found with the coarse semantics),
or where you need to call IBPB, you can't rely on the coarse
semantics because of course the IBPB may not have really
happened... and so you've to imagine IBRS with finegrined semantics as
"temporary immunization from the poison including HT/SMT poison and
all poison is still left in the CPU and will return to affect the
runtime as soon as IBRS is cleared".
IBRS Q/A:
1) When to write IBRS in SPEC_CTRL? -> imagine it as "STIBP; IBPB"
2) When to leave IBRS set or when to call IBPB -> imagine the previous
setting of IBRS as temporarily disabling indirect branch prediction
without an IBPB implicit in IBRS
If you think it only like 1) you risk missing some places where you've
to write IBRS even if it was already set.
If you think it only like 2) you risk clearing it too early or you
risk missing a necessary IBPB.
It has to be thought simultaneously in both ways.
The sure thing I get from the specs is IBRS always implies STIBP (even
when STIBP is a noop), specs are pretty explicit about that.
So following the above two rules, assume you've retpolined host kernel
and you want to protect host userland from guest userland, IBRS set in
the host kernel to host user transition (user_ibrs) will achieve it
fine. Setting STIBP in the kernel (user_stibp) to user transition
won't help at all with this scenario.
If the kernel in host was using IBRS instead of retpolines, it already
had to set IBRS in vmexit to satisfy the coarse semantics and it would
just need to leave IBRS set in the kernel to user transition (it would
then need to set IBRS again in the user to kernel transition that
follows even if it was already set).
Thanks,
Andrea
On Thu, Jan 18, 2018 at 5:41 PM, Andrea Arcangeli <[email protected]> wrote:
> Hello,
>
> On Thu, Jan 18, 2018 at 03:25:25PM -0800, Andy Lutomirski wrote:
>> I read the whitepaper that documented the new MSRs a couple days ago
>> and I'm now completely unable to find it. If anyone could send the
>> link, that would be great.
>
> I see Andrew posted a link.
>
>> From memory, however, the docs were quite clear that setting leaving
>> IBRS set when entering user mode or guest mode does not guarantee any
>> particular protection unless an additional CPUID bit (the name of
>> which I forget) is set, and that current CPUs will *not* get that bit
>
> My current understanding is that with SPEC_CTRL alone set in cpuid,
> IBRS is meaningful, other bits don't matter.
>
>> set by microcode update. IOW the protection given is that, if you set
>> IBRS bit zero after entry to kernel mode, you are protected until you
>> re-enter user mode. When you're in user mode, you're not protected.
>
> If you leave IBRS set while in user mode, userland is protected as
> strong as kernel mode.
The link that Andrew posted says:
If software sets IA32_SPEC_CTRL.IBRS to 1 after a transition to a more
privileged predictor mode, predicted targets of indirect branches
executed in that predictor mode with IA32_SPEC_CTRL.IBRS = 1 cannot be
controlled by software that was executed in a less privileged
predictor mode or on another logical processor.
...
Enabling IBRS does not prevent software from controlling the predicted
targets of indirect branches of unrelated software executed later at
the same predictor mode (for example, between two different user
applications, or two different virtual machines). Such isolation can
be ensured through use of the IBPB command, described in Section
2.5.3, “Indirect Branch Predictor Barrier (IBPB)”.
So maybe it's poorly written, but I see nothing in that language that
suggests that IBRS=1 (on a CPU without enhanced IBRS) provides any
guarantees at all about who can or cannot control speculation of
indirect branches in user mode.
>
> When you return to kernel mode you've to call IBRS again even if it
> was left set, because there's a higher-privilege mode change. That's
> equivalent to calling only IBPB and leaving STIBP set (only way to
> understand the locations where IBRS has to be set is to imagine IBRS
> as a strict "STIBP; IBPB").
Then Intel should improve their spec to say so.
> IBRS Q/A:
>
> 1) When to write IBRS in SPEC_CTRL? -> imagine it as "STIBP; IBPB"
>
> 2) When to leave IBRS set or when to call IBPB -> imagine the previous
> setting of IBRS as temporarily disabling indirect branch prediction
> without an IBPB implicit in IBRS
>
> If you think it only like 1) you risk missing some places where you've
> to write IBRS even if it was already set.
>
> If you think it only like 2) you risk clearing it too early or you
> risk missing a necessary IBPB.
>
> It has to be thought simultaneously in both ways.
From your description, it sounds like what's actually happening is:
When IBRS is enabled, indirect branch speculation targets cannot be
controlled by code that executed on a different logical processor on
by code that executed prior to the most recent time that IBRS was set
to 1. Additionally, if the CPU supports enhanced IBRS, then indirect
branch speculation targets cannot be controlled by code that executed
at a less privileged predictor mode.
Is that actually correct? If so, could Intel please *say* so?
Because writing voodoo magic security code seriously sucks.
>
> The sure thing I get from the specs is IBRS always implies STIBP (even
> when STIBP is a noop), specs are pretty explicit about that.
Ah, it says:
As noted in Section 2.5.1, “Indirect Branch Restricted Speculation
(IBRS)”, enabling IBRS prevents software operating on one logical
processor from controlling the predicted targets of indirect branches
executed on another logical processor. For that reason, it is not
necessary to enable STIBP when IBRS is enabled.
So I guess we can write 1 when we enter the kernel, but we probably
want to write 2 instead of 0 when we exit.
> Enabling IBRS does not prevent software from controlling the predicted
> targets of indirect branches of unrelated software executed later at
> the same predictor mode (for example, between two different user
> applications, or two different virtual machines). Such isolation can
> be ensured through use of the IBPB command, described in Section
> 2.5.3, “Indirect Branch Predictor Barrier (IBPB)”.
>
> So maybe it's poorly written, but I see nothing in that language that
> suggests that IBRS=1 (on a CPU without enhanced IBRS) provides any
> guarantees at all about who can or cannot control speculation of
> indirect branches in user mode.
there is no such guarantee. Some of the IBRS implementations will actually flush rather than disable, or flush parts and disable other parts.
yes the wording is a bit cryptic, but it's also very explicit about what it covers (and the rest is not covered!) and had to allow a few different implementations unfortunately.
On Fri, Jan 19, 2018 at 04:15:33AM +0000, Van De Ven, Arjan wrote:
> there is no such guarantee. Some of the IBRS implementations will
> actually flush rather than disable, or flush parts and disable other
> parts.
To me it helps in order to memorize the spec to understand why the
spec is the way it is.
I tried to help explaining some of that, but I notice that I created
more confusion... I never intended IBPB can be skipped in user to user
switches if leaving IBRS set in userland, that's not what we do and it
wouldn't be ok with certain smarter CPUs.
> yes the wording is a bit cryptic, but it's also very explicit about
> what it covers (and the rest is not covered!) and had to allow a few
> different implementations unfortunately.
We already follow the spec to the letter and we only depend on what is
covered there.
Surely the specs already explain everything better than I could ever
do, so if anything wasn't clear in the two previous emails where I
failed to explain the difference between setting or leaving IBRS set
in userland (ibrs_user) and setting or leaving STIBP set in userland
(stibp_user) you'll find all answers in the very explicit spec per
above quote.
Thanks,
Andrea