%RIP-relative addresses are nowadays correctly handled in alternative
instructions, so remove misleading comment and improve assembly to
use %RIP-relative address.
Also, explicitly using %gs: prefix will segfault for non-SMP builds.
Use macros from percpu.h which will DTRT with segment prefix register
as far as SMP/non-SMP builds are concerned.
Cc: Thomas Gleixner <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Borislav Petkov <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: "H. Peter Anvin" <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Signed-off-by: Uros Bizjak <[email protected]>
---
arch/x86/include/asm/uaccess_64.h | 11 ++++-------
1 file changed, 4 insertions(+), 7 deletions(-)
diff --git a/arch/x86/include/asm/uaccess_64.h b/arch/x86/include/asm/uaccess_64.h
index f2c02e4469cc..01455c0b070c 100644
--- a/arch/x86/include/asm/uaccess_64.h
+++ b/arch/x86/include/asm/uaccess_64.h
@@ -11,6 +11,7 @@
#include <asm/alternative.h>
#include <asm/cpufeatures.h>
#include <asm/page.h>
+#include <asm/percpu.h>
#ifdef CONFIG_ADDRESS_MASKING
/*
@@ -18,14 +19,10 @@
*/
static inline unsigned long __untagged_addr(unsigned long addr)
{
- /*
- * Refer tlbstate_untag_mask directly to avoid RIP-relative relocation
- * in alternative instructions. The relocation gets wrong when gets
- * copied to the target place.
- */
asm (ALTERNATIVE("",
- "and %%gs:tlbstate_untag_mask, %[addr]\n\t", X86_FEATURE_LAM)
- : [addr] "+r" (addr) : "m" (tlbstate_untag_mask));
+ "and " __percpu_arg([mask]) ", %[addr]", X86_FEATURE_LAM)
+ : [addr] "+r" (addr)
+ : [mask] "m" (__my_cpu_var(tlbstate_untag_mask)));
return addr;
}
--
2.41.0
On Thu, Nov 16, 2023 at 08:10:59PM +0100, Uros Bizjak wrote:
> %RIP-relative addresses are nowadays correctly handled in alternative
> instructions, so remove misleading comment and improve assembly to
> use %RIP-relative address.
Ha!, it might've been this exact case (and Kirill grumbling) that got me
to fix the alternative code :-)
> Also, explicitly using %gs: prefix will segfault for non-SMP builds.
> Use macros from percpu.h which will DTRT with segment prefix register
> as far as SMP/non-SMP builds are concerned.
> Signed-off-by: Uros Bizjak <[email protected]>
Acked-byL Peter Zijlstra (Intel) <[email protected]>
> ---
> arch/x86/include/asm/uaccess_64.h | 11 ++++-------
> 1 file changed, 4 insertions(+), 7 deletions(-)
>
> diff --git a/arch/x86/include/asm/uaccess_64.h b/arch/x86/include/asm/uaccess_64.h
> index f2c02e4469cc..01455c0b070c 100644
> --- a/arch/x86/include/asm/uaccess_64.h
> +++ b/arch/x86/include/asm/uaccess_64.h
> @@ -11,6 +11,7 @@
> #include <asm/alternative.h>
> #include <asm/cpufeatures.h>
> #include <asm/page.h>
> +#include <asm/percpu.h>
>
> #ifdef CONFIG_ADDRESS_MASKING
> /*
> @@ -18,14 +19,10 @@
> */
> static inline unsigned long __untagged_addr(unsigned long addr)
> {
> - /*
> - * Refer tlbstate_untag_mask directly to avoid RIP-relative relocation
> - * in alternative instructions. The relocation gets wrong when gets
> - * copied to the target place.
> - */
> asm (ALTERNATIVE("",
> - "and %%gs:tlbstate_untag_mask, %[addr]\n\t", X86_FEATURE_LAM)
> - : [addr] "+r" (addr) : "m" (tlbstate_untag_mask));
> + "and " __percpu_arg([mask]) ", %[addr]", X86_FEATURE_LAM)
> + : [addr] "+r" (addr)
> + : [mask] "m" (__my_cpu_var(tlbstate_untag_mask)));
>
> return addr;
> }
> --
> 2.41.0
>
On Fri, Nov 17, 2023 at 10:41:03AM +0100, Peter Zijlstra wrote:
> On Thu, Nov 16, 2023 at 08:10:59PM +0100, Uros Bizjak wrote:
> > %RIP-relative addresses are nowadays correctly handled in alternative
> > instructions, so remove misleading comment and improve assembly to
> > use %RIP-relative address.
>
> Ha!, it might've been this exact case (and Kirill grumbling) that got me
> to fix the alternative code :-)
Nice! :)
> > Also, explicitly using %gs: prefix will segfault for non-SMP builds.
> > Use macros from percpu.h which will DTRT with segment prefix register
> > as far as SMP/non-SMP builds are concerned.
>
> > Signed-off-by: Uros Bizjak <[email protected]>
>
> Acked-byL Peter Zijlstra (Intel) <[email protected]>
Acked-by: Kirill A. Shutemov <[email protected]>
--
Kiryl Shutsemau / Kirill A. Shutemov
On 11/16/23 11:10, Uros Bizjak wrote:
> %RIP-relative addresses are nowadays correctly handled in alternative
> instructions, so remove misleading comment and improve assembly to
> use %RIP-relative address.
>
> Also, explicitly using %gs: prefix will segfault for non-SMP builds.
> Use macros from percpu.h which will DTRT with segment prefix register
> as far as SMP/non-SMP builds are concerned.
OK, this is starting to feel silly. One could seriously question the use
case for supporting !SMP builds x86-64. It isn't like our performance
for SMP builds on UP systems is significantly worse, it is mostly just a
matter of code size, and the difference isn't huge, either, especially
considering that on systems of the x86-64 era the kernel is a rather
small part of system memory (unlike the very early i386 era, for those
of us who remember those ancient times.)
The number of UP x86-64 systems is really very small (since
multicore/SMT became ubiquitous at roughly the same time x86-64 was
introduced), and as far as I know none of them lack APIC which is really
the most fundamental difference between SMP and !SMP on x86.
Why don't we simply have %gs_base == 0 as an invariant for !SMP? If we
*REALLY* care to skip SWAPGS on !SMP systems, we could use alternatives
to patch out %gs: and lock (wouldn't even have to be explicit: this is
the kind of thing that objtool does really well.) We can use
alternatives without anything special, since it only matters after we
have entered user spae for the first time and would be concurrent with
patching out SWAPGS itself.
If we really *do* care about UP builds, we could teach objtool to do
this patching at compile time for the !SMP builds.
Also, didn't we at least use to have a way to mark a function as "init
on UP" so that it could be jettisoned with the init code if we find
ourselves on a uniprocessor system?
-hpa
On Fri, Nov 17, 2023 at 1:16 PM H. Peter Anvin <[email protected]> wrote:
>
> On 11/16/23 11:10, Uros Bizjak wrote:
> > %RIP-relative addresses are nowadays correctly handled in alternative
> > instructions, so remove misleading comment and improve assembly to
> > use %RIP-relative address.
> >
> > Also, explicitly using %gs: prefix will segfault for non-SMP builds.
> > Use macros from percpu.h which will DTRT with segment prefix register
> > as far as SMP/non-SMP builds are concerned.
>
> OK, this is starting to feel silly. One could seriously question the use
> case for supporting !SMP builds x86-64. It isn't like our performance
> for SMP builds on UP systems is significantly worse, it is mostly just a
> matter of code size, and the difference isn't huge, either, especially
> considering that on systems of the x86-64 era the kernel is a rather
> small part of system memory (unlike the very early i386 era, for those
> of us who remember those ancient times.)
>
> The number of UP x86-64 systems is really very small (since
> multicore/SMT became ubiquitous at roughly the same time x86-64 was
> introduced), and as far as I know none of them lack APIC which is really
> the most fundamental difference between SMP and !SMP on x86.
>
> Why don't we simply have %gs_base == 0 as an invariant for !SMP?
The reason is stack protector, which is still stuck at %gs:40. So
GSBASE has to point at fixed_percpu_data, even on a UP build. That is
corrected by the patch series I recently posted, though.
> If we
> *REALLY* care to skip SWAPGS on !SMP systems, we could use alternatives
> to patch out %gs: and lock (wouldn't even have to be explicit: this is
> the kind of thing that objtool does really well.) We can use
> alternatives without anything special, since it only matters after we
> have entered user spae for the first time and would be concurrent with
> patching out SWAPGS itself.
There is already support to patch out LOCK prefixes when running an
SMP build on a single CPU (.smp_locks section). Patching out the GS
prefix would only work if the initial percpu area is not freed.
Beyond that I don't think other optimizations are worth the effort,
and would get very little testing.
Brian Gerst
On 11/17/23 11:43, Brian Gerst wrote:>>
>> Why don't we simply have %gs_base == 0 as an invariant for !SMP?
>
> The reason is stack protector, which is still stuck at %gs:40. So
> GSBASE has to point at fixed_percpu_data, even on a UP build. That is
> corrected by the patch series I recently posted, though.
>
Right, that problem is gone.
>> If we
>> *REALLY* care to skip SWAPGS on !SMP systems, we could use alternativesYep, that is
>> to patch out %gs: and lock (wouldn't even have to be explicit: this is
>> the kind of thing that objtool does really well.) We can use
>> alternatives without anything special, since it only matters after we
>> have entered user spae for the first time and would be concurrent with
>> patching out SWAPGS itself.
>
> There is already support to patch out LOCK prefixes when running an
> SMP build on a single CPU (.smp_locks section). Patching out the GS
> prefix would only work if the initial percpu area is not freed.
> Beyond that I don't think other optimizations are worth the effort,
> and would get very little testing.
Yes, that is basically my point.
-hpa