In the "Fast Short REP MOVSB" path of memmove, if we take the path where
the FSRM flag is enabled but the ERMS flag is not, there is no longer a
check for length >= 0x20 (both alternatives will be replaced with NOPs).
If a memmove() requiring a forward copy of less than 0x20 bytes happens
in this case, the `sub $0x20, %rdx` will cause the length to roll around
to a huge value and the copy will eventually hit a page fault.
This is not intended to happen, as the comment above the alternatives
mentions "FSRM implies ERMS".
However, there is a check in early_init_intel() that can disable ERMS,
so we should also be disabling FSRM in this path to maintain correctness
of the memmove() optimization.
Cc: [email protected]
Fixes: f444a5ff95dc ("x86/cpufeatures: Add support for fast short REP; MOVSB")
Signed-off-by: Daniel Verkamp <[email protected]>
---
arch/x86/kernel/cpu/intel.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/x86/kernel/cpu/intel.c b/arch/x86/kernel/cpu/intel.c
index 2d7ea5480ec3..71b412f820c7 100644
--- a/arch/x86/kernel/cpu/intel.c
+++ b/arch/x86/kernel/cpu/intel.c
@@ -328,6 +328,7 @@ static void early_init_intel(struct cpuinfo_x86 *c)
pr_info("Disabled fast string operations\n");
setup_clear_cpu_cap(X86_FEATURE_REP_GOOD);
setup_clear_cpu_cap(X86_FEATURE_ERMS);
+ setup_clear_cpu_cap(X86_FEATURE_FSRM);
}
}
--
2.37.3.998.g577e59143f-goog
On Thu, Sep 22, 2022 at 05:58:27PM -0700, Daniel Verkamp wrote:
> In the "Fast Short REP MOVSB" path of memmove, if we take the path where
> the FSRM flag is enabled but the ERMS flag is not, there is no longer a
> check for length >= 0x20 (both alternatives will be replaced with NOPs).
> If a memmove() requiring a forward copy of less than 0x20 bytes happens
> in this case, the `sub $0x20, %rdx` will cause the length to roll around
> to a huge value and the copy will eventually hit a page fault.
>
> This is not intended to happen, as the comment above the alternatives
> mentions "FSRM implies ERMS".
>
> However, there is a check in early_init_intel() that can disable ERMS,
> so we should also be disabling FSRM in this path to maintain correctness
> of the memmove() optimization.
Is this something you hit in a real-world scenario? If so, how exactly?
Thx.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
On Fri, Sep 23, 2022 at 4:13 AM Borislav Petkov <[email protected]> wrote:
>
> On Thu, Sep 22, 2022 at 05:58:27PM -0700, Daniel Verkamp wrote:
> > In the "Fast Short REP MOVSB" path of memmove, if we take the path where
> > the FSRM flag is enabled but the ERMS flag is not, there is no longer a
> > check for length >= 0x20 (both alternatives will be replaced with NOPs).
> > If a memmove() requiring a forward copy of less than 0x20 bytes happens
> > in this case, the `sub $0x20, %rdx` will cause the length to roll around
> > to a huge value and the copy will eventually hit a page fault.
> >
> > This is not intended to happen, as the comment above the alternatives
> > mentions "FSRM implies ERMS".
> >
> > However, there is a check in early_init_intel() that can disable ERMS,
> > so we should also be disabling FSRM in this path to maintain correctness
> > of the memmove() optimization.
>
> Is this something you hit in a real-world scenario? If so, how exactly?
>
> Thx.
Yes, we hit this in crosvm when booting the guest kernel with either
OVMF or u-boot on an Intel 12th Gen CPU. The guest kernel boots fine
when loaded directly (using the crosvm kernel loader and not running
any firmware setup in the guest), but it crashes when booting with
firmware inside the first forward memmove() after alternatives are set
up (which happens to be in printk). I haven't gotten to the bottom of
why exactly using firmware is causing this to be set up in an
inconsistent way, but this is a real-world situation, not just a
hypothetical.
Now that I look at it with fresh eyes again, maybe we should instead
directly patch the memmove FSRM alternative so that the flag-set
version just does the same jmp as the ERMS one. I can prepare a patch
for that instead of (or in addition to) this one if that sounds
better.
Thanks,
-- Daniel
On Fri, Sep 23, 2022 at 10:25:05AM -0700, Daniel Verkamp wrote:
> Yes, we hit this in crosvm when booting the guest kernel with either
> OVMF or u-boot on an Intel 12th Gen CPU. The guest kernel boots fine
> when loaded directly (using the crosvm kernel loader and not running
> any firmware setup in the guest), but it crashes when booting with
> firmware inside the first forward memmove() after alternatives are set
> up (which happens to be in printk). I haven't gotten to the bottom of
> why exactly using firmware is causing this to be set up in an
> inconsistent way, but this is a real-world situation, not just a
> hypothetical.
Sounds like broken virt firmware or so. And if that is not an issue on
baremetal, then the virt stack should be fixed - not the kernel.
> Now that I look at it with fresh eyes again, maybe we should instead
> directly patch the memmove FSRM alternative so that the flag-set
> version just does the same jmp as the ERMS one. I can prepare a patch
> for that instead of (or in addition to) this one if that sounds
> better.
So, if the virt firmware deviates from how the real hardware behaves,
then the kernel needs no fixing.
So you'd have to figure out why is the virt firmware causing this and
not baremetal.
Then we can talk about fixes.
Thx.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
On Fri, Sep 23, 2022 at 10:51 AM Borislav Petkov <[email protected]> wrote:
>
> On Fri, Sep 23, 2022 at 10:25:05AM -0700, Daniel Verkamp wrote:
> > Yes, we hit this in crosvm when booting the guest kernel with either
> > OVMF or u-boot on an Intel 12th Gen CPU. The guest kernel boots fine
> > when loaded directly (using the crosvm kernel loader and not running
> > any firmware setup in the guest), but it crashes when booting with
> > firmware inside the first forward memmove() after alternatives are set
> > up (which happens to be in printk). I haven't gotten to the bottom of
> > why exactly using firmware is causing this to be set up in an
> > inconsistent way, but this is a real-world situation, not just a
> > hypothetical.
>
> Sounds like broken virt firmware or so. And if that is not an issue on
> baremetal, then the virt stack should be fixed - not the kernel.
>
> > Now that I look at it with fresh eyes again, maybe we should instead
> > directly patch the memmove FSRM alternative so that the flag-set
> > version just does the same jmp as the ERMS one. I can prepare a patch
> > for that instead of (or in addition to) this one if that sounds
> > better.
>
> So, if the virt firmware deviates from how the real hardware behaves,
> then the kernel needs no fixing.
>
> So you'd have to figure out why is the virt firmware causing this and
> not baremetal.
>
> Then we can talk about fixes.
Hi Borislav,
We found that the IA32_MISC_ENABLE MSR setup was missing in the crosvm
firmware boot path (but not when directly booting a kernel, which is
why it did not get noticed for a while). Setting the fast string bit
in the MSR avoids the issue.
However, I still think it would be appropriate to apply this patch or
something like it, since there could be a CPU, microcode update, BIOS,
etc. that clears this bit while still having the CPUID flags for FSRM
and ERMS. The Intel SDM says: "Software can disable fast-string
operation by clearing the fast-string-enable bit (bit 0) of
IA32_MISC_ENABLE MSR", so it's not an invalid configuration for this
bit to be unset.
Additionally, something like this avoids the problem by making the
FSRM case jump directly to the REP MOVSB rather than falling through
to the ERMS jump in the next instruction, which seems like basically
free insurance (but if the FSRM flag gets used somewhere else in the
future, having it set consistently with ERMS is probably still a good
idea, per the original patch):
diff --git a/arch/x86/lib/memmove_64.S b/arch/x86/lib/memmove_64.S
index 724bbf83eb5b..8ac557409c7d 100644
--- a/arch/x86/lib/memmove_64.S
+++ b/arch/x86/lib/memmove_64.S
@@ -38,7 +38,7 @@ SYM_FUNC_START(__memmove)
/* FSRM implies ERMS => no length checks, do the copy directly */
.Lmemmove_begin_forward:
- ALTERNATIVE "cmp $0x20, %rdx; jb 1f", "", X86_FEATURE_FSRM
+ ALTERNATIVE "cmp $0x20, %rdx; jb 1f", "jmp .Lmemmove_erms",
X86_FEATURE_FSRM
ALTERNATIVE "", "jmp .Lmemmove_erms", X86_FEATURE_ERMS
And hey, this means one less instruction to execute in the FSRM path. :)
Thanks,
-- Daniel
On Fri, Oct 07, 2022 at 11:08:45AM -0700, Daniel Verkamp wrote:
> We found that the IA32_MISC_ENABLE MSR setup was missing in the crosvm
> firmware boot path
Sounds like crazy virtualization stuff.
> However, I still think it would be appropriate to apply this patch or
> something like it, since there could be a CPU, microcode update, BIOS,
> etc. that clears this bit while still having the CPUID flags for FSRM
> and ERMS.
You mean that "something" would be so silly so as to clear the MSR but
leave the CPUID bits set?
That sounds like a bug in that "something".
> The Intel SDM says: "Software can disable fast-string operation by
> clearing the fast-string-enable bit (bit 0) of IA32_MISC_ENABLE MSR",
> so it's not an invalid configuration for this bit to be unset.
Dunno, did Intel folks think about clearing the respective CPUID bits
when exposing IA32_MISC_ENABLE[0] to software? Tony?
> Additionally, something like this avoids the problem by making the
> FSRM case jump directly to the REP MOVSB rather than falling through
> to the ERMS jump in the next instruction, which seems like basically
> free insurance (but if the FSRM flag gets used somewhere else in the
> future,
I can't follow here.
> having it set consistently with ERMS is probably still a good
> idea, per the original patch):
>
> diff --git a/arch/x86/lib/memmove_64.S b/arch/x86/lib/memmove_64.S
> index 724bbf83eb5b..8ac557409c7d 100644
> --- a/arch/x86/lib/memmove_64.S
> +++ b/arch/x86/lib/memmove_64.S
> @@ -38,7 +38,7 @@ SYM_FUNC_START(__memmove)
>
> /* FSRM implies ERMS => no length checks, do the copy directly */
> .Lmemmove_begin_forward:
> - ALTERNATIVE "cmp $0x20, %rdx; jb 1f", "", X86_FEATURE_FSRM
> + ALTERNATIVE "cmp $0x20, %rdx; jb 1f", "jmp .Lmemmove_erms",
> X86_FEATURE_FSRM
> ALTERNATIVE "", "jmp .Lmemmove_erms", X86_FEATURE_ERMS
>
> And hey, this means one less instruction to execute in the FSRM path. :)
What one less instruction? After patching and since we assume FSRM =>
ERMS, we have only the JMP there if both flags are set. If ERMS only is
set, then we do the length check.
And you need the second alternative call for !ERMS, i.e., old rust.
So yes, your proposal is to do this because we assume if FSRM, then ERMS
but your diff above doesn't make it more readable but less.
Or I'm missing something ofc.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
>> The Intel SDM says: "Software can disable fast-string operation by
>> clearing the fast-string-enable bit (bit 0) of IA32_MISC_ENABLE MSR",
>> so it's not an invalid configuration for this bit to be unset.
>
> Dunno, did Intel folks think about clearing the respective CPUID bits
> when exposing IA32_MISC_ENABLE[0] to software? Tony?
I don't know if it was thought about. Experimentally clearing bit
0 of IA32_MISC_ENABLE does not affect the CPUID bit settings
for either ERMS or FSRM (on the one system I tried that supports
both of these bits).
-Tony
On Tue, Oct 11, 2022 at 05:09:19PM +0000, Luck, Tony wrote:
> >> The Intel SDM says: "Software can disable fast-string operation by
> >> clearing the fast-string-enable bit (bit 0) of IA32_MISC_ENABLE MSR",
> >> so it's not an invalid configuration for this bit to be unset.
> >
> > Dunno, did Intel folks think about clearing the respective CPUID bits
> > when exposing IA32_MISC_ENABLE[0] to software? Tony?
>
> I don't know if it was thought about. Experimentally clearing bit
> 0 of IA32_MISC_ENABLE does not affect the CPUID bit settings
> for either ERMS or FSRM (on the one system I tried that supports
> both of these bits).
Confirms my observation on a Coffeelake here too.
Oh well, probably not that important. I mean, I bet luserspace is
looking at those CPUID bits - glibc probably - and then probably selects
the proper memcpy or so routine, based on what's there.
If something clears MSR_IA32_MISC_ENABLE_FAST_STRING_BIT and we go and
clear our feature flags and luserspace still queries CPUID then oh well,
it'll be fun. It all depends on why something has cleared them tho. It
could be some performance thing or something a lot more funky. I guess
if stuff starts exploding left and right, there will soon be a microcode
patch after that. :-)
Might be prudent to check with hw folks just in case...
Thx.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
> If something clears MSR_IA32_MISC_ENABLE_FAST_STRING_BIT and we go and
> clear our feature flags and luserspace still queries CPUID then oh well,
> it'll be fun. It all depends on why something has cleared them tho. It
> could be some performance thing or something a lot more funky. I guess
> if stuff starts exploding left and right, there will soon be a microcode
> patch after that. :-)
Yes. ERMS CPUID bit was the early indicator to s/w that REP MOVS would
do some fancy speedups if certain conditions about source, destination and
count are all met. A hint to s/w to decide which memcpy() routine to use.
FSRM is a hint that the byte count restriction had pretty much been removed
so s/w can use REP MOVS in even more places.
I don't think Intel will deliberately release a CPU that has FSRM=1, ERMS=0.
-Tony
On Tue, Oct 11, 2022 at 07:08:51PM +0000, Luck, Tony wrote:
> I don't think Intel will deliberately release a CPU that has FSRM=1, ERMS=0.
Sure, but I don't mean that. Rather: if for some reason the kernel or
BIOS is supposed to fix an erratum related to those enhanced REP moving
routines and goes and clears the MSR bit.
That won't help because userspace will still use them since the CPUID
flags remain set.
I guess such a case is probably not going to happen in real life but if
it happened, that bit clearing is kinda useless.
I'd say.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
> That won't help because userspace will still use them since the CPUID
> flags remain set.
Even if writing the MSR did clear the CPUID bits, it wouldn't help with applications
that started before the kernel cleared the bits (assuming this was some run-time
update patch).
Worst case scenario is that the applications don't pick the most efficient memcpy().
-Tony
On 07. 10. 22, 20:08, Daniel Verkamp wrote:
> On Fri, Sep 23, 2022 at 10:51 AM Borislav Petkov <[email protected]> wrote:
>>
>> On Fri, Sep 23, 2022 at 10:25:05AM -0700, Daniel Verkamp wrote:
>>> Yes, we hit this in crosvm when booting the guest kernel with either
>>> OVMF or u-boot on an Intel 12th Gen CPU. The guest kernel boots fine
>>> when loaded directly (using the crosvm kernel loader and not running
>>> any firmware setup in the guest), but it crashes when booting with
>>> firmware inside the first forward memmove() after alternatives are set
>>> up (which happens to be in printk). I haven't gotten to the bottom of
>>> why exactly using firmware is causing this to be set up in an
>>> inconsistent way, but this is a real-world situation, not just a
>>> hypothetical.
>>
>> Sounds like broken virt firmware or so. And if that is not an issue on
>> baremetal, then the virt stack should be fixed - not the kernel.
>>
>>> Now that I look at it with fresh eyes again, maybe we should instead
>>> directly patch the memmove FSRM alternative so that the flag-set
>>> version just does the same jmp as the ERMS one. I can prepare a patch
>>> for that instead of (or in addition to) this one if that sounds
>>> better.
>>
>> So, if the virt firmware deviates from how the real hardware behaves,
>> then the kernel needs no fixing.
>>
>> So you'd have to figure out why is the virt firmware causing this and
>> not baremetal.
>>
>> Then we can talk about fixes.
>
> Hi Borislav,
>
> We found that the IA32_MISC_ENABLE MSR setup was missing in the crosvm
> firmware boot path (but not when directly booting a kernel, which is
> why it did not get noticed for a while). Setting the fast string bit
> in the MSR avoids the issue.
>
> However, I still think it would be appropriate to apply this patch or
> something like it, since there could be a CPU, microcode update, BIOS,
> etc. that clears this bit while still having the CPUID flags for FSRM
> and ERMS.
Let me resurrect this thread... Our customer has an AMD CPU which has
indeed both capabilities under normal circumstances. But they have a
cool UEFI BIOS too. They say:
"""
In AMD platform, while disalbe ERMS(Enhanced Rep MOVSB/STOSB) in UEFI
(system setup -> processor -> Enhanced Rep MOVSB/STOSB), the OS can't
boot normally.
"""
That is exactly the case here. So can we have the patch (the original
one, the one below or a better one) to fix this?
> The Intel SDM says: "Software can disable fast-string
> operation by clearing the fast-string-enable bit (bit 0) of
> IA32_MISC_ENABLE MSR", so it's not an invalid configuration for this
> bit to be unset.
>
> Additionally, something like this avoids the problem by making the
> FSRM case jump directly to the REP MOVSB rather than falling through
> to the ERMS jump in the next instruction, which seems like basically
> free insurance (but if the FSRM flag gets used somewhere else in the
> future, having it set consistently with ERMS is probably still a good
> idea, per the original patch):
>
> diff --git a/arch/x86/lib/memmove_64.S b/arch/x86/lib/memmove_64.S
> index 724bbf83eb5b..8ac557409c7d 100644
> --- a/arch/x86/lib/memmove_64.S
> +++ b/arch/x86/lib/memmove_64.S
> @@ -38,7 +38,7 @@ SYM_FUNC_START(__memmove)
>
> /* FSRM implies ERMS => no length checks, do the copy directly */
> .Lmemmove_begin_forward:
> - ALTERNATIVE "cmp $0x20, %rdx; jb 1f", "", X86_FEATURE_FSRM
> + ALTERNATIVE "cmp $0x20, %rdx; jb 1f", "jmp .Lmemmove_erms",
> X86_FEATURE_FSRM
> ALTERNATIVE "", "jmp .Lmemmove_erms", X86_FEATURE_ERMS
>
> And hey, this means one less instruction to execute in the FSRM path. :)
thanks,
--
js
suse labs
On Wed, Jan 04, 2023 at 08:43:51AM +0100, Jiri Slaby wrote:
> Let me resurrect this thread... Our customer has an AMD CPU which has indeed
> both capabilities under normal circumstances. But they have a cool UEFI BIOS
> too. They say:
>
> """
> In AMD platform, while disalbe ERMS(Enhanced Rep MOVSB/STOSB) in UEFI
> (system setup -> processor -> Enhanced Rep MOVSB/STOSB), the OS can't boot
> normally.
Any particular reason they're disabling ERMS?
What do they set FSRM to?
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
* Borislav Petkov <[email protected]> wrote:
> On Wed, Jan 04, 2023 at 08:43:51AM +0100, Jiri Slaby wrote:
> > Let me resurrect this thread... Our customer has an AMD CPU which has indeed
> > both capabilities under normal circumstances. But they have a cool UEFI BIOS
> > too. They say:
> >
> > """
> > In AMD platform, while disalbe ERMS(Enhanced Rep MOVSB/STOSB) in UEFI
> > (system setup -> processor -> Enhanced Rep MOVSB/STOSB), the OS can't boot
> > normally.
>
> Any particular reason they're disabling ERMS?
>
> What do they set FSRM to?
Nevertheless both Jiri and Daniel are making a valid argument: our x86
memcpy routines should not behave in an undefined fashion, *regardless* of
what CPUID environment we are in.
As practice has shown, both on virtual and on bare metal firmware can screw
things up enough so that the memcpy routines crash under Linux but under no
other OS...
So while you are technically correct that these are firmware bugs, I'm in
favor of robustifying our x86 memcpy routines against these bugs. Silently
not booting, where no other OS fails to boot, is poor form IMO.
Thanks,
Ingo
On Sat, Jan 14, 2023 at 10:19:40AM +0100, Ingo Molnar wrote:
> So while you are technically correct that these are firmware bugs, I'm in
> favor of robustifying our x86 memcpy routines against these bugs. Silently not
> booting, where no other OS fails to boot, is poor form IMO.
Yes, I know.
As I told you yesterday on IRC, I'll take care of it.
Thx.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
On 04. 01. 23, 12:39, Borislav Petkov wrote:
> On Wed, Jan 04, 2023 at 08:43:51AM +0100, Jiri Slaby wrote:
>> Let me resurrect this thread... Our customer has an AMD CPU which has indeed
>> both capabilities under normal circumstances. But they have a cool UEFI BIOS
>> too. They say:
>>
>> """
>> In AMD platform, while disalbe ERMS(Enhanced Rep MOVSB/STOSB) in UEFI
>> (system setup -> processor -> Enhanced Rep MOVSB/STOSB), the OS can't boot
>> normally.
>
> Any particular reason they're disabling ERMS?
Just so you know (I see you progressed with Ingo to fix this) -- despite
I asked the very same question, I received no answer quite yet. I
suppose it will sound like usual "because we can".
> What do they set FSRM to?
I suppose they keep it "enabled".
thanks,
--
js
suse labs
On Mon, Jan 16, 2023 at 06:26:50AM +0100, Jiri Slaby wrote:
> Just so you know (I see you progressed with Ingo to fix this) -- despite I
> asked the very same question, I received no answer quite yet. I suppose it
> will sound like usual "because we can".
Yah, tell them it is totally useless to disable ERMS on an AMD system. Just
leave the both settings on.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette