Some cloud hypervisors do not provide IBPB on very recent CPU processors,
including AMD processors affected by Retbleed.
Using IBPB before firmware calls on such systems would cause a GPF at boot
like the one below. Do not enable such calls when IBPB support is not
present.
[ 0.997530] EFI Variables Facility v0.08 2004-May-17
[ 0.998866] general protection fault, maybe for address 0x1: 0000 [#1] PREEMPT SMP NOPTI
[ 1.000393] CPU: 0 PID: 24 Comm: kworker/u2:1 Not tainted 5.19.0-rc8+ #7
[ 1.000393] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 0.0.0 02/06/2015
[ 1.000393] Workqueue: efi_rts_wq efi_call_rts
[ 1.000393] RIP: 0010:efi_call_rts+0x66e/0x8a0
[ 1.000393] Code: e8 37 33 58 ff 41 bf 48 00 00 00 49 89 c0 44 89 f9 48 83 c8 01 4c 89 c2 48 c1 ea 20 66 90 b9 49 00 00 00 b8 01 00 00 00 31 d2 <0f> 30 e8 7b 9f 5d ff e8 f6 f8 ff ff 4c 89 f1 4c 89 ea 4c 89 e6 48
[ 1.000393] RSP: 0018:ffffb373800d7e38 EFLAGS: 00010246
[ 1.000393] RAX: 0000000000000001 RBX: 0000000000000006 RCX: 0000000000000049
[ 1.000393] RDX: 0000000000000000 RSI: ffff94fbc19d8fe0 RDI: ffff94fbc1b2b300
[ 1.000393] RBP: ffffb373800d7e70 R08: 0000000000000000 R09: 0000000000000000
[ 1.000393] R10: 000000000000000b R11: 000000000000000b R12: ffffb3738001fd78
[ 1.000393] R13: ffff94fbc2fcfc00 R14: ffffb3738001fd80 R15: 0000000000000048
[ 1.000393] FS: 0000000000000000(0000) GS:ffff94fc3da00000(0000) knlGS:0000000000000000
[ 1.000393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1.000393] CR2: ffff94fc30201000 CR3: 000000006f610000 CR4: 00000000000406f0
[ 1.000393] Call Trace:
[ 1.000393] <TASK>
[ 1.000393] ? __wake_up+0x13/0x20
[ 1.000393] process_one_work+0x21f/0x3f0
[ 1.000393] worker_thread+0x50/0x3e0
[ 1.000393] ? rescuer_thread+0x3a0/0x3a0
[ 1.000393] kthread+0xee/0x120
[ 1.000393] ? kthread_complete_and_exit+0x20/0x20
[ 1.000393] ret_from_fork+0x22/0x30
[ 1.000393] </TASK>
[ 1.000393] Modules linked in:
[ 1.037117] ---[ end trace 0000000000000000 ]---
[ 1.038324] RIP: 0010:efi_call_rts+0x66e/0x8a0
[ 1.039650] Code: e8 37 33 58 ff 41 bf 48 00 00 00 49 89 c0 44 89 f9 48 83 c8 01 4c 89 c2 48 c1 ea 20 66 90 b9 49 00 00 00 b8 01 00 00 00 31 d2 <0f> 30 e8 7b 9f 5d ff e8 f6 f8 ff ff 4c 89 f1 4c 89 ea 4c 89 e6 48
[ 1.044235] RSP: 0018:ffffb373800d7e38 EFLAGS: 00010246
[ 1.045513] RAX: 0000000000000001 RBX: 0000000000000006 RCX: 0000000000000049
[ 1.047260] RDX: 0000000000000000 RSI: ffff94fbc19d8fe0 RDI: ffff94fbc1b2b300
[ 1.049014] RBP: ffffb373800d7e70 R08: 0000000000000000 R09: 0000000000000000
[ 1.050762] R10: 000000000000000b R11: 000000000000000b R12: ffffb3738001fd78
[ 1.052521] R13: ffff94fbc2fcfc00 R14: ffffb3738001fd80 R15: 0000000000000048
[ 1.054243] FS: 0000000000000000(0000) GS:ffff94fc3da00000(0000) knlGS:0000000000000000
[ 1.056228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1.057632] CR2: ffff94fc30201000 CR3: 000000006f610000 CR4: 00000000000406f0
[ 1.059393] note: kworker/u2:1[24] exited with preempt_count 2
Fixes: 28a99e95f55c ("x86/amd: Use IBPB for firmware calls")
Reported-by: Dimitri John Ledkov <[email protected]>
Signed-off-by: Thadeu Lima de Souza Cascardo <[email protected]>
Cc: Peter Zijlstra (Intel) <[email protected]>
Cc: Borislav Petkov <[email protected]>
Cc: <[email protected]>
---
arch/x86/kernel/cpu/bugs.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 6454bc767f0f..6761668100b9 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -1520,6 +1520,7 @@ static void __init spectre_v2_select_mitigation(void)
* enable IBRS around firmware calls.
*/
if (boot_cpu_has_bug(X86_BUG_RETBLEED) &&
+ boot_cpu_has(X86_FEATURE_IBPB) &&
(boot_cpu_data.x86_vendor == X86_VENDOR_AMD ||
boot_cpu_data.x86_vendor == X86_VENDOR_HYGON)) {
--
2.34.1
On Thu, Jul 28, 2022 at 02:35:48PM +0200, Borislav Petkov wrote:
> drop stable@
>
> On Thu, Jul 28, 2022 at 09:26:02AM -0300, Thadeu Lima de Souza Cascardo wrote:
> > Some cloud hypervisors do not provide IBPB on very recent CPU processors,
> > including AMD processors affected by Retbleed.
>
> Which hypervisors are those? How relevant is that use case?
Azure for sure, and I guess AWS would be affected as well.
>
> How do I reproduce it here?
>
qemu -cpu host,ibpb=off on an affected AMD processor worked for me. With EFI,
so with something like -drive if=pflash,index=0,file=OVMF_CODE_4M.fd -drive
if=pflash,index=1,file=OVMF_VARS_4M.fd.
Cascardo.
> Thx.
>
> --
> Regards/Gruss,
> Boris.
>
> https://people.kernel.org/tglx/notes-about-netiquette
drop stable@
On Thu, Jul 28, 2022 at 09:26:02AM -0300, Thadeu Lima de Souza Cascardo wrote:
> Some cloud hypervisors do not provide IBPB on very recent CPU processors,
> including AMD processors affected by Retbleed.
Which hypervisors are those? How relevant is that use case?
How do I reproduce it here?
Thx.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
On Thu, 28 Jul 2022 at 13:35, Borislav Petkov <[email protected]> wrote:
>
> drop stable@
>
> On Thu, Jul 28, 2022 at 09:26:02AM -0300, Thadeu Lima de Souza Cascardo wrote:
> > Some cloud hypervisors do not provide IBPB on very recent CPU processors,
> > including AMD processors affected by Retbleed.
>
> Which hypervisors are those? How relevant is that use case?
>
> How do I reproduce it here?
Azure public cloud (so it is Azure custom hyper-v hypervisor) these
instance types https://docs.microsoft.com/en-us/azure/virtual-machines/dav4-dasv4-series
booted as gen2 (UEFI boot, so Dasv4-series instance types). A
particular one is chosen in our automated testing, and always fails. I
believe more than one instance type from that series of instance types
is affected.
I haven't tested but
https://docs.microsoft.com/en-us/azure/virtual-machines/dasv5-dadsv5-series
are probably affected too.
It's a class of popular-ish instance types, meaning that it could
potentially take out a class of users who due to availability,
performance, and/or pricing choose to run their workloads on those
instance types. Potentially causing them a major outage of being
unable to boot and/or reboot.
--
okurrr,
Dimitri
On Thu, Jul 28, 2022 at 09:26:02AM -0300, Thadeu Lima de Souza Cascardo wrote:
> Some cloud hypervisors do not provide IBPB on very recent CPU processors,
> including AMD processors affected by Retbleed.
That's a bug in the hypervisor.
> Fixes: 28a99e95f55c ("x86/amd: Use IBPB for firmware calls")
Fixes^WCreates-a-speculation-hole-in:
> Reported-by: Dimitri John Ledkov <[email protected]>
> Signed-off-by: Thadeu Lima de Souza Cascardo <[email protected]>
> Cc: Peter Zijlstra (Intel) <[email protected]>
> Cc: Borislav Petkov <[email protected]>
> Cc: <[email protected]>
> ---
> arch/x86/kernel/cpu/bugs.c | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
> index 6454bc767f0f..6761668100b9 100644
> --- a/arch/x86/kernel/cpu/bugs.c
> +++ b/arch/x86/kernel/cpu/bugs.c
> @@ -1520,6 +1520,7 @@ static void __init spectre_v2_select_mitigation(void)
> * enable IBRS around firmware calls.
> */
> if (boot_cpu_has_bug(X86_BUG_RETBLEED) &&
> + boot_cpu_has(X86_FEATURE_IBPB) &&
> (boot_cpu_data.x86_vendor == X86_VENDOR_AMD ||
> boot_cpu_data.x86_vendor == X86_VENDOR_HYGON)) {
At the very least we need a pr_warn() and something nasty in
retbleed_show_state() to warn the user their firmware calls are
vulnerable.
On Thu, Jul 28, 2022 at 03:33:35PM +0100, Dimitri John Ledkov wrote:
> Azure public cloud (so it is Azure custom hyper-v hypervisor) these
> instance types https://docs.microsoft.com/en-us/azure/virtual-machines/dav4-dasv4-series
Thank you both for the info.
Virt is an awful piece of sh*t when it goes and emulates all kinds of
imaginary CPUs. And AMD machine *without* an IBPB which is affected by
retbleed. Well, f*ck that.
Does that say somewhere on azure that those guests need to even enable
the mitigation or does the HV mitigate it for them?
Because I wouldn't mind to simply disable the mitigation when on a
hypervisor which doesn't support IBPB.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
+ Cooper to sanity-check me.
On Thu, Jul 28, 2022 at 05:18:31PM +0200, Borislav Petkov wrote:
> On Thu, Jul 28, 2022 at 03:33:35PM +0100, Dimitri John Ledkov wrote:
> > Azure public cloud (so it is Azure custom hyper-v hypervisor) these
> > instance types https://docs.microsoft.com/en-us/azure/virtual-machines/dav4-dasv4-series
>
> Thank you both for the info.
>
> Virt is an awful piece of sh*t when it goes and emulates all kinds of
> imaginary CPUs. And AMD machine *without* an IBPB which is affected by
> retbleed. Well, f*ck that.
>
> Does that say somewhere on azure that those guests need to even enable
> the mitigation or does the HV mitigate it for them?
>
> Because I wouldn't mind to simply disable the mitigation when on a
> hypervisor which doesn't support IBPB.
So for 5.19 we probably should take the one-liner just so that we
release with all known issues fixed.
Going forward, I'm thinking all that FW-mitigation selection should go
into a function called something like firmware_select_mitigations()
which gets called at the end of check_bugs(), after all mitigation
selectors have run.
And in there, the first check should be if X86_FEATURE_HYPERVISOR and if
set, not set any mitigations for firmware calls.
Because, frankly, is there any point in protecting against firmware
calls in the guest? The guest firmware is part of the hypervisor which
gets supplied by the guest owner or cloud provider or so.
In the former case you probably don't need protection and in the latter,
you don't have a choice.
But I'm unclear on the fw-in-the-guest thing - I'm sure Andy has a
better idea...
Thx.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
On Thu, Jul 28, 2022 at 05:50:06PM +0200, Borislav Petkov wrote:
> + Cooper to sanity-check me.
>
> On Thu, Jul 28, 2022 at 05:18:31PM +0200, Borislav Petkov wrote:
> > On Thu, Jul 28, 2022 at 03:33:35PM +0100, Dimitri John Ledkov wrote:
> > > Azure public cloud (so it is Azure custom hyper-v hypervisor) these
> > > instance types https://docs.microsoft.com/en-us/azure/virtual-machines/dav4-dasv4-series
> >
> > Thank you both for the info.
> >
> > Virt is an awful piece of sh*t when it goes and emulates all kinds of
> > imaginary CPUs. And AMD machine *without* an IBPB which is affected by
> > retbleed. Well, f*ck that.
> >
> > Does that say somewhere on azure that those guests need to even enable
> > the mitigation or does the HV mitigate it for them?
> >
> > Because I wouldn't mind to simply disable the mitigation when on a
> > hypervisor which doesn't support IBPB.
>
> So for 5.19 we probably should take the one-liner just so that we
> release with all known issues fixed.
>
> Going forward, I'm thinking all that FW-mitigation selection should go
> into a function called something like firmware_select_mitigations()
> which gets called at the end of check_bugs(), after all mitigation
> selectors have run.
>
> And in there, the first check should be if X86_FEATURE_HYPERVISOR and if
> set, not set any mitigations for firmware calls.
>
> Because, frankly, is there any point in protecting against firmware
> calls in the guest? The guest firmware is part of the hypervisor which
> gets supplied by the guest owner or cloud provider or so.
>
> In the former case you probably don't need protection and in the latter,
> you don't have a choice.
>
> But I'm unclear on the fw-in-the-guest thing - I'm sure Andy has a
> better idea...
>
I may be completely wrong here, so excuse me throwing out this idea.
But isn't it also possible that userspace attacks the kernel by leveraging
speculative execution when in firmware? So even when firmware is trusted, it
might not have mitigations like retpoline and rethunks. So userspace will train
the BTB in order to make a RET in the firmware speculate to a firmware gadget
that may spill out kernel bits to the cache.
Even though there is some limited mapping when doing the firmware calls, there
are still some kernel pages mapped.
Cascardo.
> Thx.
>
> --
> Regards/Gruss,
> Boris.
>
> https://people.kernel.org/tglx/notes-about-netiquette
The following commit has been merged into the x86/urgent branch of tip:
Commit-ID: 571c30b1a88465a1c85a6f7762609939b9085a15
Gitweb: https://git.kernel.org/tip/571c30b1a88465a1c85a6f7762609939b9085a15
Author: Thadeu Lima de Souza Cascardo <[email protected]>
AuthorDate: Thu, 28 Jul 2022 09:26:02 -03:00
Committer: Borislav Petkov <[email protected]>
CommitterDate: Fri, 29 Jul 2022 10:02:35 +02:00
x86/bugs: Do not enable IBPB at firmware entry when IBPB is not available
Some cloud hypervisors do not provide IBPB on very recent CPU processors,
including AMD processors affected by Retbleed.
Using IBPB before firmware calls on such systems would cause a GPF at boot
like the one below. Do not enable such calls when IBPB support is not
present.
EFI Variables Facility v0.08 2004-May-17
general protection fault, maybe for address 0x1: 0000 [#1] PREEMPT SMP NOPTI
CPU: 0 PID: 24 Comm: kworker/u2:1 Not tainted 5.19.0-rc8+ #7
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 0.0.0 02/06/2015
Workqueue: efi_rts_wq efi_call_rts
RIP: 0010:efi_call_rts
Code: e8 37 33 58 ff 41 bf 48 00 00 00 49 89 c0 44 89 f9 48 83 c8 01 4c 89 c2 48 c1 ea 20 66 90 b9 49 00 00 00 b8 01 00 00 00 31 d2 <0f> 30 e8 7b 9f 5d ff e8 f6 f8 ff ff 4c 89 f1 4c 89 ea 4c 89 e6 48
RSP: 0018:ffffb373800d7e38 EFLAGS: 00010246
RAX: 0000000000000001 RBX: 0000000000000006 RCX: 0000000000000049
RDX: 0000000000000000 RSI: ffff94fbc19d8fe0 RDI: ffff94fbc1b2b300
RBP: ffffb373800d7e70 R08: 0000000000000000 R09: 0000000000000000
R10: 000000000000000b R11: 000000000000000b R12: ffffb3738001fd78
R13: ffff94fbc2fcfc00 R14: ffffb3738001fd80 R15: 0000000000000048
FS: 0000000000000000(0000) GS:ffff94fc3da00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffff94fc30201000 CR3: 000000006f610000 CR4: 00000000000406f0
Call Trace:
<TASK>
? __wake_up
process_one_work
worker_thread
? rescuer_thread
kthread
? kthread_complete_and_exit
ret_from_fork
</TASK>
Modules linked in:
Fixes: 28a99e95f55c ("x86/amd: Use IBPB for firmware calls")
Reported-by: Dimitri John Ledkov <[email protected]>
Signed-off-by: Thadeu Lima de Souza Cascardo <[email protected]>
Signed-off-by: Borislav Petkov <[email protected]>
Cc: <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
---
arch/x86/kernel/cpu/bugs.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index 6454bc7..6761668 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -1520,6 +1520,7 @@ static void __init spectre_v2_select_mitigation(void)
* enable IBRS around firmware calls.
*/
if (boot_cpu_has_bug(X86_BUG_RETBLEED) &&
+ boot_cpu_has(X86_FEATURE_IBPB) &&
(boot_cpu_data.x86_vendor == X86_VENDOR_AMD ||
boot_cpu_data.x86_vendor == X86_VENDOR_HYGON)) {
On Thu, Jul 28, 2022 at 02:01:57PM -0300, Thadeu Lima de Souza Cascardo wrote:
> I may be completely wrong here, so excuse me throwing out this idea.
>
> But isn't it also possible that userspace attacks the kernel by leveraging
> speculative execution when in firmware? So even when firmware is trusted, it
> might not have mitigations like retpoline and rethunks. So userspace will train
> the BTB in order to make a RET in the firmware speculate to a firmware gadget
> that may spill out kernel bits to the cache.
>
> Even though there is some limited mapping when doing the firmware calls, there
> are still some kernel pages mapped.
Yah, I dunno. That's why I raised this and added Andy. I certainly see
your point tho.
And what I know is, I don't want to be dealing with imaginary virt guest
configurations just because some cloud providers wanna do whatever they
like.
I've put this mitigation selection spaghetti on my to-give-a-stern-look
list. Because it is looking insane already and it'll get even worse with
time.
Thx.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette