MIME-Version: 1.0
References: <20230226110802.103134-1-usama.arif@bytedance.com>
 <20230226110802.103134-8-usama.arif@bytedance.com> <878rghmrn2.ffs@tglx>
In-Reply-To: <878rghmrn2.ffs@tglx>
From:   Brian Gerst <brgerst@gmail.com>
Date:   Tue, 28 Feb 2023 17:09:03 -0500
Message-ID: <CAMzpN2hwwZ64MycrCfqKw-Hu_8Xfvz_LdzMgiJ_Ho=x6HFYGVA@mail.gmail.com>
Subject: Re: [PATCH v12 07/11] x86/smpboot: Remove early_gdt_descr on 64-bit
To:     Thomas Gleixner <tglx@linutronix.de>
Cc:     Usama Arif <usama.arif@bytedance.com>, dwmw2@infradead.org,
        kim.phillips@amd.com, piotrgorski@cachyos.org,
        oleksandr@natalenko.name, arjan@linux.intel.com, mingo@redhat.com,
        bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com,
        x86@kernel.org, pbonzini@redhat.com, paulmck@kernel.org,
        linux-kernel@vger.kernel.org, kvm@vger.kernel.org,
        rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com,
        thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de,
        fam.zheng@bytedance.com, punit.agrawal@bytedance.com,
        simon.evans@bytedance.com, liangma@liangbit.com,
        David Woodhouse <dwmw@amazon.co.uk>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Tue, Feb 28, 2023 at 4:01=E2=80=AFPM Thomas Gleixner <tglx@linutronix.de=
> wrote:
>
> On Sun, Feb 26 2023 at 11:07, Usama Arif wrote:
> > @@ -265,7 +265,12 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SY=
M_L_GLOBAL)
> >        * addresses where we're currently running on. We have to do that=
 here
> >        * because in 32bit we couldn't load a 64bit linear address.
> >        */
> > -     lgdt    early_gdt_descr(%rip)
> > +     subq    $16, %rsp
> > +     movw    $(GDT_SIZE-1), (%rsp)
> > +     leaq    gdt_page(%rdx), %rax
>
> Even on !SMP gdt_page is in the 0...__per_cpu_end range. Which means
> that on !SMP this results in:
>
>       leaq    0xb000(%rdx),%rax
>
> and RDX is 0. That's not really a valid GDT pointer, right?

No.  On !SMP per-cpu variables are normal variables in the .data
section.  They are not gathered together in the per-cpu section and
are not accessed with the GS prefix.

ffffffff810000c9:       48 8d 82 00 10 81 82    lea    0x82811000(%rdx),%ra=
x
                        ffffffff810000cc: R_X86_64_32S  gdt_page

ffffffff82811000 D gdt_page

So RDX=3D0 is correct.

> > +     movq    %rax, 2(%rsp)
> > +     lgdt    (%rsp)
>
> and obviously that's equally broken for the task stack part:
>
> >       movq    pcpu_hot + X86_current_task(%rdx), %rax

Same as gdt_page:

ffffffff810000b1:       48 8b 82 00 88 a8 82    mov    0x82a88800(%rdx),%ra=
x
                        ffffffff810000b4: R_X86_64_32S  pcpu_hot

ffffffff82a88800 D pcpu_hot

> This needs:
>
> --- a/arch/x86/kernel/head_64.S
> +++ b/arch/x86/kernel/head_64.S
> @@ -239,7 +239,7 @@ SYM_INNER_LABEL(secondary_startup_64_no_
>         /* Get the per cpu offset for the given CPU# which is in ECX */
>         movq    __per_cpu_offset(,%rcx,8), %rdx
>  #else
> -       xorl    %edx, %edx
> +       leaq    INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx
>  #endif /* CONFIG_SMP */
>
>         /*
>
> in the initial_stack patch, which then allows to remove this hunk in the
> initial_gs patch:
>
> @@ -286,9 +286,6 @@ SYM_INNER_LABEL(secondary_startup_64_no_
>          * the per cpu areas are set up.
>          */
>         movl    $MSR_GS_BASE,%ecx
> -#ifndef CONFIG_SMP
> -       leaq    INIT_PER_CPU_VAR(fixed_percpu_data)(%rip), %rdx
> -#endif

On !SMP the only thing GSBASE is used for is the stack protector
canary, which is in fixed_percpu_data.  There is no per-cpu section.

FWIW, I posted a patch set a while back that switched x86-64 to use
the options added to newer compilers controlling where the canary is
located, allowing it to become a standard per-cpu variable and
removing the need to force the per-cpu section to be zero-based.
However it was not accepted at that time, due to removing support for
stack protector on older compilers (GCC < 8.1).

>         movl    %edx, %eax
>         shrq    $32, %rdx
>         wrmsr
>
> Maybe we should enforce CONFIG_SMP=3Dy first :)

Makes sense, only the earliest generations of x86-64 processors have a
single core/thread, and an SMP kernel can still run on them.

--
Brian Gerst