Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <ea03061e-14aa-8159-4935-869f694a4715@amd.com>
Date:   Mon, 7 Feb 2022 12:50:35 -0600
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.5.0
Subject: Re: [PATCH v4 0/9] Parallel CPU bringup for x86_64
Content-Language: en-US
To:     David Woodhouse <dwmw2@infradead.org>,
        Thomas Gleixner <tglx@linutronix.de>
Cc:     Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>, x86@kernel.org,
        "H . Peter Anvin" <hpa@zytor.com>,
        Paolo Bonzini <pbonzini@redhat.com>,
        "Paul E . McKenney" <paulmck@kernel.org>,
        linux-kernel@vger.kernel.org, kvm@vger.kernel.org,
        rcu@vger.kernel.org, mimoja@mimoja.de, hewenliang4@huawei.com,
        hushiyuan@huawei.com, luolongjun@huawei.com, hejingxian@huawei.com,
        Sean Christopherson <seanjc@google.com>,
        Paul Menzel <pmenzel@molgen.mpg.de>
References: <20220201205328.123066-1-dwmw2@infradead.org>
From:   Tom Lendacky <thomas.lendacky@amd.com>
In-Reply-To: <20220201205328.123066-1-dwmw2@infradead.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?V0FtQytSMFFjMnRXZ2xDd1RvUEJiMlAyMjhkWjNyV0gvTjQ4a0hQN3ljL1c2?=
 =?utf-8?B?TWhUOE9aQTJab3RCdWRRNkZpazRGVXl0NE85L0dNQW9VQ0hTUGRVVFlkUUM4?=
 =?utf-8?B?NlpiNGxJd2gxaUJRRU1Vc3E4eWZwN2V5L3NqOGJSZlVWUERsZi9meCt0N1V5?=
 =?utf-8?B?dERETmU1UmJxZzJINFdIWFIyQjR1ZzZtaWU4dHhjL2FDcEVtVFpBZnp2dERS?=
 =?utf-8?B?RDVhclFHV3pYemdsSTNaMmpZZUJ0M0hta2haOWFNc1ZMYlRKMjBrU1FDd21M?=
 =?utf-8?B?UmNMQnVOYnpUZkhweVhCZm92THBLZ2FES3U2ZER6YWpUdDVTSXVXcUJCUGdt?=
 =?utf-8?B?MFNNMnRxdFJyY1hXdCs2aDFlbVphSzk4Mk1lKzVnc1pHaGxmL2Q4NDhOSVYv?=
 =?utf-8?B?KzNxaW9yUXNqcGlsQklPQXdGTWhIR1dkaGk0M1o0MkYvZEp6bEFxT1lwUm50?=
 =?utf-8?B?bHhkUk5sVEQzdHZqckU0R1BlUVhzdGJ5VDZhdDR2SnhWcEhFQXpIRmxxNUZS?=
 =?utf-8?B?aVl5OGwrSzlMbm9CL1B0RGJIQXdNQjdvTmFSa0c1cDF4OGkyb296WkZWRUhQ?=
 =?utf-8?B?eDdnN3oyQmFJeFBUR1JQZ3dDUlNZanpsUTdHWVl2eE83YTVnOXpDMEtqT290?=
 =?utf-8?B?TVprcDJhUHlaOWUySDF4OXliMVZqRUo5RVdLSkRFVU9pRXliUXM2NXFocnAw?=
 =?utf-8?B?S3hDRGROSndXcG5rWVBFQkdZNitSczM5TldKSVV0eWZNWlMxbXJsOG9IQkhy?=
 =?utf-8?B?a2FvaVFWZW1VS0FvWHFtMks2dk0rRHIyelgyVk9JYzFUanNzeW5FSGczTFht?=
 =?utf-8?B?SHJXTzdOdWlUVkUzbmlnMmpuL1ZGOFVDbGh3TkxZaC9qYUYxWm81UWcxWnRU?=
 =?utf-8?B?WlhCdnJvS2ExSk5TQitOdGk3YVFCem5WZnZrZFNqY2duSGNaN3N1VXFhT1pE?=
 =?utf-8?B?RkpSNS9UVzQzcjJ1MlRscmF2bFlxRFBpNnhvWUpVblZaZTBWWXVoUzY5R3lL?=
 =?utf-8?B?SEErb1NZdytmRmUxWVJrczRVYjRoZnRUVmVNNDFPaDQ0MVBDemhSdmpobmhM?=
 =?utf-8?B?ZXRZNG5lV0tLcEVaWlIvTG1YUHhLOU1UQXRFNy9zeWNGR29sRmZ3MytGb2FY?=
 =?utf-8?B?TzVEcFc2bXJieXFCYUg4UytCaDhEMjlzd2tHZHI5TW9BN3FMUFZPUXJOWnht?=
 =?utf-8?B?bHB5dnZXemt5MUNRK2FNM0FVNzRVNjIwbVFUcENFak9UZ3NDeXdOdms1Mmlu?=
 =?utf-8?B?SHM2aVlCU0hZR0VWMTJuZHBRYU5QZzFDV2kxZUw5YmNXZmtuZ1FsbjJaNmZU?=
 =?utf-8?B?VU00VFFvVDd2SUtvYWJvbGRjc1JPQXFNU25GTGsxb2JHTHF5Tmg2Zi9sbzFx?=
 =?utf-8?B?aW0wZUpCRkNZTldRcER4OGJmSHNrNjYvb1NGSWlpMVMyTUxTRWJHbjJ4SXBn?=
 =?utf-8?B?TldYMW9FT3BxVFZRQzlCNVVrWVByaWMzSFk2a0F1eDFobjhsK2hNWHcyRTht?=
 =?utf-8?B?QVBrbmc5djlxVUFia0FZeDk0ZTN1ME1weVRUVC80UncyRmt3d3RNT0dKeFZQ?=
 =?utf-8?B?QWJDcHp1NVpRYjV6cXlsQ3NHNENycW1DLzFMRFpIMWgwRmxrcmpiZGdsR0hX?=
 =?utf-8?B?UmhwVDRDejkvY0drdktQSUhKVWQ4Q05ROXpsTnFOWjNCejUzZ2xzVlY3MGkz?=
 =?utf-8?B?K1lnRUZnM1hHWmR4WlJJZnAxYzVtbllweHljUmU1QW53VldSVXJCNmZ1bjBu?=
 =?utf-8?B?dVhySHh4VjhqQyt5S2lCa1l2YlR3VGMyS0dFcG9JMjhkeGt6Qk9SQ1dhUXFR?=
 =?utf-8?B?cjY1bDhqNW02V0VZcTJ1R1FQRGQ3OXBqMkg5VHJWQzAvaVFiTDRXRXZQc3A3?=
 =?utf-8?B?ZklPdnIwQkFXcXNpblBPd0ZTbWRwdDhKTmFjdEpEZGg5WnZIVnQxMTNtY2Zv?=
 =?utf-8?B?TjJHSk40OEpFeCs2bXI2L2RyVzNPT3FSWlRGQWZkTkFYNmwreTRuRWl1YWlB?=
 =?utf-8?B?dFU1NUdvYUN3OEphTHdEWmlTaHFESWVoOUhQendOaGZ3T3p0LzFvUEJiaUwy?=
 =?utf-8?B?ZUFDN1o5QVF5WGVuODNETEpmdEVjcUszdnR6NWppV2tsWExmRFZ5bkMvN0k2?=
 =?utf-8?B?MzQrRnE1N1lRVmR0K0RtL2NzMXBFdnJuUnBFV0lYS1RkUEJmNWZSblhzcjdu?=
 =?utf-8?Q?9l0s0FKnF9LrP/l3sZhSNAc=3D?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: bce39e9f-71cb-4991-5f80-08d9ea6abb70
X-MS-Exchange-CrossTenant-AuthSource: BL1PR12MB5221.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Feb 2022 18:50:38.6146
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ypIAvwLPSWKruux4a8HSoD9T3AIamnpzKtp/TwEVZ+BOV9kYdfjbvbTd6YyRM0+7V9s4jS4DLA4LpSN4th+Q3w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR12MB1429
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 2/1/22 14:53, David Woodhouse wrote:
> Doing the INIT/SIPI/SIPI in parallel for all APs and *then* waiting for
> them shaves about 80% off the AP bringup time on a 96-thread 2-socket
> Skylake box (EC2 c5.metal) — from about 500ms to 100ms.
> 
> There are more wins to be had with further parallelisation, but this is
> the simple part.
> 
> v2: Cut it back to just INIT/SIPI/SIPI in parallel for now, nothing more
> v3: Clean up x2apic patch, add MTRR optimisation, lock topology update
>      in preparation for more parallelisation.
> v4: Fixes to the real mode parallelisation patch spotted by SeanC, to
>      avoid scribbling on initial_gs in common_cpu_up(), and to allow all
>      24 bits of the physical X2APIC ID to be used. That patch still needs
>      a Signed-off-by from its original author, who once claimed not to
>      remember writing it at all. But now we've fixed it, hopefully he'll
>      admit it now :)

I'm no longer seeing crashes launching high vCPU-count guests with this 
series.

Thanks,
Tom

> 
> David Woodhouse (8):
>        x86/apic/x2apic: Fix parallel handling of cluster_mask
>        cpu/hotplug: Move idle_thread_get() to <linux/smpboot.h>
>        cpu/hotplug: Add dynamic parallel bringup states before CPUHP_BRINGUP_CPU
>        x86/smpboot: Reference count on smpboot_setup_warm_reset_vector()
>        x86/smpboot: Split up native_cpu_up into separate phases and document them
>        x86/smpboot: Send INIT/SIPI/SIPI to secondary CPUs in parallel
>        x86/mtrr: Avoid repeated save of MTRRs on boot-time CPU bringup
>        x86/smpboot: Serialize topology updates for secondary bringup
> 
> Thomas Gleixner (1):
>        x86/smpboot: Support parallel startup of secondary CPUs
> 
> [dwoodhou@i7 linux-2.6]$ git diff --stat  v5.17-rc2..share/parallel-5.17-part1
>   arch/x86/include/asm/realmode.h       |   3 +
>   arch/x86/include/asm/smp.h            |  13 +-
>   arch/x86/include/asm/topology.h       |   2 -
>   arch/x86/kernel/acpi/sleep.c          |   1 +
>   arch/x86/kernel/apic/apic.c           |   2 +-
>   arch/x86/kernel/apic/x2apic_cluster.c | 108 ++++++-----
>   arch/x86/kernel/cpu/common.c          |   6 +-
>   arch/x86/kernel/cpu/mtrr/mtrr.c       |   9 +
>   arch/x86/kernel/head_64.S             |  73 ++++++++
>   arch/x86/kernel/smpboot.c             | 325 ++++++++++++++++++++++++----------
>   arch/x86/realmode/init.c              |   3 +
>   arch/x86/realmode/rm/trampoline_64.S  |  14 ++
>   arch/x86/xen/smp_pv.c                 |   4 +-
>   include/linux/cpuhotplug.h            |   2 +
>   include/linux/smpboot.h               |   7 +
>   kernel/cpu.c                          |  27 ++-
>   kernel/smpboot.c                      |   2 +-
>   kernel/smpboot.h                      |   2 -
>   18 files changed, 442 insertions(+), 161 deletions(-)
> 
> 
>