Message-ID: <16d9f738-18fd-a929-e711-f2a1e757e33f@mailbox.org>
Date:   Fri, 10 Mar 2023 19:20:56 +0000
MIME-Version: 1.0
Subject: Re: [PATCH v14 00/12] Parallel CPU bringup for x86_64
To:     Usama Arif <usama.arif@bytedance.com>, dwmw2@infradead.org,
        tglx@linutronix.de, kim.phillips@amd.com, brgerst@gmail.com
Cc:     piotrgorski@cachyos.org, oleksandr@natalenko.name,
        arjan@linux.intel.com, mingo@redhat.com, bp@alien8.de,
        dave.hansen@linux.intel.com, hpa@zytor.com, x86@kernel.org,
        pbonzini@redhat.com, paulmck@kernel.org,
        linux-kernel@vger.kernel.org, kvm@vger.kernel.org,
        rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com,
        thomas.lendacky@amd.com, seanjc@google.com, pmenzel@molgen.mpg.de,
        fam.zheng@bytedance.com, punit.agrawal@bytedance.com,
        simon.evans@bytedance.com, liangma@liangbit.com
References: <20230308171328.1562857-1-usama.arif@bytedance.com>
Content-Language: en-US
From:   Tor Vic <torvic9@mailbox.org>
In-Reply-To: <20230308171328.1562857-1-usama.arif@bytedance.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk


On 08.03.23 17:13, Usama Arif wrote:
> The main code change over v13 is to enable parallel bringup for SEV-ES guests.
> 
> Thanks,
> Usama
> 
> Changes across versions:
> v2: Cut it back to just INIT/SIPI/SIPI in parallel for now, nothing more
> v3: Clean up x2apic patch, add MTRR optimisation, lock topology update
>      in preparation for more parallelisation.
> v4: Fixes to the real mode parallelisation patch spotted by SeanC, to
>      avoid scribbling on initial_gs in common_cpu_up(), and to allow all
>      24 bits of the physical X2APIC ID to be used. That patch still needs
>      a Signed-off-by from its original author, who once claimed not to
>      remember writing it at all. But now we've fixed it, hopefully he'll
>      admit it now :)
> v5: rebase to v6.1 and remeasure performance, disable parallel bringup
>      for AMD CPUs.
> v6: rebase to v6.2-rc6, disabled parallel boot on amd as a cpu bug and
>      reused timer calibration for secondary CPUs.
> v7: [David Woodhouse] iterate over all possible CPUs to find any existing
>      cluster mask in alloc_clustermask. (patch 1/9)
>      Keep parallel AMD support enabled in AMD, using APIC ID in CPUID leaf
>      0x0B (for x2APIC mode) or CPUID leaf 0x01 where 8 bits are sufficient.
>      Included sanity checks for APIC id from 0x0B. (patch 6/9)
>      Removed patch for reusing timer calibration for secondary CPUs.
>      commit message and code improvements.
> v8: Fix CPU0 hotplug by setting up the initial_gs, initial_stack and
>      early_gdt_descr.
>      Drop trampoline lock and bail if APIC ID not found in find_cpunr.
>      Code comments improved and debug prints added.
> v9: Drop patch to avoid repeated saves of MTRR at boot time.
>      rebased and retested at v6.2-rc8.
>      added kernel doc for no_parallel_bringup and made do_parallel_bringup
>      __ro_after_init.
> v10: Fixed suspend/resume not working with parallel smpboot.
>       rebased and retested to 6.2.
>       fixed checkpatch errors.
> v11: Added patches from Brian Gerst to remove the global variables initial_gs,
>       initial_stack, and early_gdt_descr from the 64-bit boot code
>       (https://lore.kernel.org/all/20230222221301.245890-1-brgerst@gmail.com/).
> v12: Fixed compilation errors, acquire tr_lock for every stack setup in
>       trampoline_64.S.
>       Rearranged commits for a cleaner git history.
> v13: Fix build error with CONFIG_FORCE_NR_CPUS.
>       Commit message improved, typos fixed and extra comments added.
> v14: Enable parallel bringup for SEV-ES guests
>   
> Brian Gerst (3):
>    x86/smpboot: Remove initial_stack on 64-bit
>    x86/smpboot: Remove early_gdt_descr on 64-bit
>    x86/smpboot: Remove initial_gs
> 
> David Woodhouse (9):
>    x86/apic/x2apic: Allow CPU cluster_mask to be populated in parallel
>    cpu/hotplug: Move idle_thread_get() to <linux/smpboot.h>
>    cpu/hotplug: Add dynamic parallel bringup states before
>      CPUHP_BRINGUP_CPU
>    x86/smpboot: Reference count on smpboot_setup_warm_reset_vector()
>    x86/smpboot: Split up native_cpu_up into separate phases and document
>      them
>    x86/smpboot: Support parallel startup of secondary CPUs
>    x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel
>    x86/smpboot: Serialize topology updates for secondary bringup
>    x86/smpboot: Allow parallel bringup for SEV-ES
> 
>   .../admin-guide/kernel-parameters.txt         |   3 +
>   arch/x86/include/asm/cpu.h                    |   1 +
>   arch/x86/include/asm/processor.h              |   6 +-
>   arch/x86/include/asm/realmode.h               |   4 +-
>   arch/x86/include/asm/sev-common.h             |   3 +
>   arch/x86/include/asm/sev.h                    |   5 +
>   arch/x86/include/asm/smp.h                    |  18 +-
>   arch/x86/include/asm/topology.h               |   2 -
>   arch/x86/kernel/acpi/sleep.c                  |  30 +-
>   arch/x86/kernel/apic/apic.c                   |   2 +-
>   arch/x86/kernel/apic/x2apic_cluster.c         | 126 +++---
>   arch/x86/kernel/asm-offsets.c                 |   1 +
>   arch/x86/kernel/cpu/common.c                  |   6 +-
>   arch/x86/kernel/cpu/topology.c                |   2 +-
>   arch/x86/kernel/head_64.S                     | 162 ++++++--
>   arch/x86/kernel/smpboot.c                     | 366 +++++++++++++-----
>   arch/x86/realmode/init.c                      |   3 +
>   arch/x86/realmode/rm/trampoline_64.S          |  27 +-
>   arch/x86/xen/smp_pv.c                         |   4 +-
>   arch/x86/xen/xen-head.S                       |   2 +-
>   include/linux/cpuhotplug.h                    |   2 +
>   include/linux/smpboot.h                       |   7 +
>   kernel/cpu.c                                  |  31 +-
>   kernel/smpboot.h                              |   2 -
>   24 files changed, 614 insertions(+), 201 deletions(-)
> 

On Linux 6.2, Zen2 and Skylake, no issues or boot problems:

Tested-by: Tor Vic <torvic9@mailbox.org>