Date:   Wed, 01 Feb 2023 08:55:53 -0800
From:   "H. Peter Anvin" <hpa@zytor.com>
To:     Usama Arif <usama.arif@bytedance.com>,
        David Woodhouse <dwmw2@infradead.org>,
        Thomas Gleixner <tglx@linutronix.de>
CC:     Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>, x86@kernel.org,
        Paolo Bonzini <pbonzini@redhat.com>,
        "Paul E . McKenney" <paulmck@kernel.org>,
        linux-kernel@vger.kernel.org, kvm@vger.kernel.org,
        rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com,
        hushiyuan@huawei.com, luolongjun@huawei.com, hejingxian@huawei.com,
        Tom Lendacky <thomas.lendacky@amd.com>,
        Sean Christopherson <seanjc@google.com>,
        Paul Menzel <pmenzel@molgen.mpg.de>,
        Fam Zheng <fam.zheng@bytedance.com>,
        Punit Agrawal <punit.agrawal@bytedance.com>,
        simon.evans@bytedance.com, liangma@liangbit.com
Subject: Re: [External] Re: [PATCH v4 0/9] Parallel CPU bringup for x86_64
User-Agent: K-9 Mail for Android
In-Reply-To: <7fda05c9-9d4a-6005-0ce5-91bda1bb06c1@bytedance.com>
References: <20220201205328.123066-1-dwmw2@infradead.org> <ff876008-b642-4dbc-aa41-1639905e08b6@bytedance.com> <1d0ed92ab68409b62a14cd29d0021f92c6e2568a.camel@infradead.org> <7fda05c9-9d4a-6005-0ce5-91bda1bb06c1@bytedance.com>
Message-ID: <4C7F2481-0B0B-4399-A8E1-30731EFD02D2@zytor.com>
MIME-Version: 1.0
Content-Type: text/plain;
 charset=utf-8
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On February 1, 2023 8:38:14 AM PST, Usama Arif <usama=2Earif@bytedance=2Eco=
m> wrote:
>
>
>On 01/02/2023 15:08, David Woodhouse wrote:
>> On Wed, 2023-02-01 at 14:40 +0000, Usama Arif wrote:
>>> On 01/02/2022 20:53, David Woodhouse wrote:
>>>> Doing the INIT/SIPI/SIPI in parallel for all APs and *then* waiting f=
or
>>>> them shaves about 80% off the AP bringup time on a 96-thread 2-socket
>>>> Skylake box (EC2 c5=2Emetal) =E2=80=94 from about 500ms to 100ms=2E
>>>>=20
>>>> There are more wins to be had with further parallelisation, but this =
is
>>>> the simple part=2E
>>>>=20
>>>=20
>>> Hi,
>>>=20
>>> We are interested in reducing the boot time of servers (with kexec), a=
nd
>>> smpboot takes up a significant amount of time while booting=2E When
>>> testing the patch series (rebased to v6=2E1) on a server with 128 CPUs
>>> split across 2 NUMA nodes, it brought down the smpboot time from ~700m=
s
>>> to 100ms=2E Adding another cpuhp state for do_wait_cpu_initialized to =
make
>>> sure cpu_init is reached (as done in v1 of the series + using the
>>> cpu_finishup_mask) brought it down further to ~30ms=2E
>>>=20
>>> I just wanted to check what was needed to progress the patch series
>>> further for review? There weren't any comments on v4 of the patch so I
>>> couldn't figure out what more is needed=2E I think its quite useful to
>>> have this working so would be really glad help in anything needed to
>>> restart the review=2E
>>=20
>>=20
>> I believe the only thing holding it back was the fact that it broke on
>> some AMD CPUs=2E
>>=20
>> We don't *think* there are any remaining software issues; we think it's
>> hardware=2E Either an actual hardware race in CPU or chipset, or perhap=
s
>> even something as simple as a voltage regulator which can't cope with
>> an increase in power draw from *all* the CPUs at the same time=2E
>>=20
>> We have prodded AMD a few times to investigate, but so far to no avail=
=2E
>>=20
>> Last time I actually spoke to Thomas in person, I think he agreed that
>> we should just merge it and disable the parallel mode for the affected
>> AMD CPUs=2E
>>=20
>
>From the comments in v3, it seems to affect multiple generations, would i=
t be worth proceeding with the patches by disabling it on all AMD CPUs to b=
e on the safe side, until the actual issue is found and what causes it, and=
 then follow up later if the issue is found by disabling it only on affecte=
d cpus=2E Maybe simply do something like below?
>
>diff --git a/arch/x86/kernel/smpboot=2Ec b/arch/x86/kernel/smpboot=2Ec
>index 0f144773a7fc=2E=2E6b8884592341 100644
>--- a/arch/x86/kernel/smpboot=2Ec
>+++ b/arch/x86/kernel/smpboot=2Ec
>@@ -1575,7 +1575,8 @@ void __init native_smp_prepare_cpus(unsigned int ma=
x_cpus)
>         * for SEV-ES guests because they can't use CPUID that early=2E
>         */
>        if (IS_ENABLED(CONFIG_X86_32) || boot_cpu_data=2Ecpuid_level < 0x=
0B ||
>-           cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT))
>+           cc_platform_has(CC_ATTR_GUEST_STATE_ENCRYPT) ||
>+           boot_cpu_data=2Ex86_vendor =3D=3D X86_VENDOR_AMD)
>                do_parallel_bringup =3D false;
>
>        if (do_parallel_bringup) {
>
>
>
>
>> If you've already rebased to a newer kernel and tested it, perhaps now
>> is the time to do just that=2E
>
>If you would like me to repost the rebased patches to restart the reviews=
 (with do_parallel_bringup disabled for AMD), please let me know!
>
>Thanks,
>Usama

This should be a CPU bug flag in my option=2E