Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2;
Message-ID: <02a80f45-496e-41dd-b17e-819ca82f27c5@amd.com>
Date:   Wed, 25 Oct 2023 14:04:27 -0500
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH 1/2] x86: Enable x2apic during resume from suspend if used
 previously
Content-Language: en-US
To:     Ingo Molnar <mingo@kernel.org>
Cc:     Tom Lendacky <thomas.lendacky@amd.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Borislav Petkov <bp@alien8.de>,
        Thomas Gleixner <tglx@linutronix.de>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        Sandipan Das <sandipan.das@amd.com>,
        "H . Peter Anvin" <hpa@zytor.com>, linux-kernel@vger.kernel.org,
        x86@kernel.org, linux-pm@vger.kernel.org, rafael@kernel.org,
        pavel@ucw.cz, linux-perf-users@vger.kernel.org,
        Ingo Molnar <mingo@redhat.com>,
        Arnaldo Carvalho de Melo <acme@kernel.org>,
        Mark Rutland <mark.rutland@arm.com>,
        Alexander Shishkin <alexander.shishkin@linux.intel.com>,
        Jiri Olsa <jolsa@kernel.org>,
        Namhyung Kim <namhyung@kernel.org>,
        Ian Rogers <irogers@google.com>,
        Adrian Hunter <adrian.hunter@intel.com>
References: <20231023160018.164054-1-mario.limonciello@amd.com>
 <20231023160018.164054-2-mario.limonciello@amd.com>
 <ZTeB9K4NYu1sRiZ1@gmail.com> <6894bebf-15ed-4bd2-aebc-f4ac0a30a395@amd.com>
 <ZTf4dtWBTjiKIntj@gmail.com>
From:   Mario Limonciello <mario.limonciello@amd.com>
In-Reply-To: <ZTf4dtWBTjiKIntj@gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?S3VXMTZyTWdoR3VJdFRlTGVpdXpwanNjTDVLbTdKaXVnNVJwR1hUNy9Md3M1?=
 =?utf-8?B?cGVHaHMyYmpwcGpZa1NOWlNxN1Nlb2RCWGVWK243cG5ZcXNQTHFKRFVxNnRq?=
 =?utf-8?B?L0h5OVA5RnVORkdVV0dDSEJvMEQwcEIvbzJiQzVFcEx2MkpJUkZ3NlhERXox?=
 =?utf-8?B?Mm0zNHhUZlBRMSt1QVh0N2d3a2xnakR2MUYxc1htc24xZGtRbVgvd3YyUGhO?=
 =?utf-8?B?WFFEVlk5LzBpS1NYMUVWRFp5QldWeXN3akV1eWlUWnBreFhHMjZmREE3YVhn?=
 =?utf-8?B?U3lGMGs2cWZObzd2Sldsc2lyRlRIYmpubDU4N29QZmRaeExWQm54UVNNZWJO?=
 =?utf-8?B?MjBJcWVRekpuNlZBUEtQVitVbG1pVS9HZldndy9XQ200dUwycWdXT05oSktL?=
 =?utf-8?B?MUtXTnJra3RuRVJLckVmQ1JPdmhiZVNtWml1TUoyVno1Q0x5R0RyTG9zZWlY?=
 =?utf-8?B?UGNqcXlxY0xhQ0lMSWlDZDk2YWJvNWlIZ2VoUDhxa0dIa2VVbDFSSzlWa25q?=
 =?utf-8?B?WlNicHNvR2RXVXZVWXRLeDRTWUsra0h2S3A1eUxZd2wyUWZkYnRIbDJsS1Yx?=
 =?utf-8?B?VW1WZWZtVzMyM0VTSEVjRWtwODRheHZLUlJvQmg0UTh4L0lUMGttTG40ZXVx?=
 =?utf-8?B?NGQveGZrQjBoYWNUcWRtZ0t6MEgrYnBnQ0ZKTlBTd2VpMlNlemN3cXg2Y0Q0?=
 =?utf-8?B?S3hYanphNjZVTU9nVzlPZ2poaG5XQXR1OVFMVUpiS0s4a2lqOUxMK09EMlJn?=
 =?utf-8?B?cXZoYlp6VEFwTDcrc0N4MmxjbmxSSlo4VmZiekVXRS9qcERhWHZZZW9KU0VN?=
 =?utf-8?B?bnFnRjZmbEkwNVFjcUFZTXV0SnpGc1ZZSThMQVJReTU5YUpmOWxRUkE5bS9C?=
 =?utf-8?B?aG9mcFBsN1BBVjhuU1h6R1k2VmdjeG5IbFZuRHd5Z3d2dlMxT0Q1WWNpcWU3?=
 =?utf-8?B?MjR3VVl3T1dGclNvMWhhaVBCcXA5VEdVQ29wcHBaQ0pab2VoMXp5TEFRTWNw?=
 =?utf-8?B?MmlWeTZJZEE1bFYzTUdOTGo4RlB4alV1cndMT2tGekxRcXpFeEY4cGFvYzgw?=
 =?utf-8?B?QThFaExKczVWalFMRXZZUHZ0NlBWMWY3T1VWZDFsTVBLemFRWkdlWDZBNWox?=
 =?utf-8?B?RTRpdElqbUFlOUh1UFBiNi8rN0pLeVVjZ0IrcmxSYXg3VzM1NHVEQ1JBdkZO?=
 =?utf-8?B?Y0pzR1puSndPaFZBeGNlNDJoOXFnOGpEczVFQUN3K2Vld0xtUHR0YzFGU2d3?=
 =?utf-8?B?WVFhVG9yV0dxUXIzbDNkdUFxK3VYN2dYbUI0RlBiU1VuRGl2N05IbHhicnpp?=
 =?utf-8?B?NHkvOW5yOGlXUllmZzNFTHcxeWN2M1ZKZ09yOFNTdUhnZHJyZGpUSEwvWlcv?=
 =?utf-8?B?cEFlZkFZK3N0Q2ZoQUdYbVdhdnl1Yis1UnRScnZ6NzZVUDlZT0xnVlg0VkhL?=
 =?utf-8?B?RTNNVjJjMlpGYmRNZVFmcEZDMHVEYmVWR3RjOHF3ZTRLY3pWUXVhZTBqc3ZU?=
 =?utf-8?B?N0Y3cm1sN2NWbGFRbm5VZXFkazM2aFFXR25HcDBZRE85STFFRkVXYm56UkxL?=
 =?utf-8?B?RStyNC9JQkdmcytoNEx4dmJLc3NVWTVLb1M3WG1mOGJtR1VFSGNxVncwdUlr?=
 =?utf-8?B?UmdCL3YvNGlrc01zV3BrWjUwWDlZWmpnNGtqOXpqSUt3TlR4OFJpRStBR0I1?=
 =?utf-8?B?R0ZPV2doSEppWjE3UnlLU1BBQUlDeU5JeCt5MFFOM0dzY3lMWDlHdXErZWFM?=
 =?utf-8?B?QUgxak5VT0FGTlhjOG14MU12UzN0UjlQOTRZQ2lScktPSkZDRm44WDk0VG82?=
 =?utf-8?B?N3FBdi84QzZZTkhzL1hlbGdaMHdteC9yYk8xZDZDZU1kMERVVHRDN3hkdkVx?=
 =?utf-8?B?L0dhbGszMHpzeWRJeGNGaUZDSWd4RGhpSEpoRkVlSnV2WGQ5SzEvMHF0K1hl?=
 =?utf-8?B?Z3RyWHM3REZRVE5lY2pmK1E2dWhNNXpUOWsvZE96ZWk4YnE0Y3R3TyttazEw?=
 =?utf-8?B?aXNnODZERCs1N2dwWEY1SFMyQUV4L3JXT2E1ejBuU2RFa2dlOWRJSklzeFlC?=
 =?utf-8?B?MDA5dURjMlNwc3pkOGdWUll0a0x3UTFGVEVzaFJPZEYra1NHM3FUOHlBeXFm?=
 =?utf-8?Q?MbeAJHtGN5ywR3Uvg66Drk/MA?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 98bd239d-8ccf-41da-e6c2-08dbd58d3776
X-MS-Exchange-CrossTenant-AuthSource: MN0PR12MB6101.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Oct 2023 19:04:30.2975
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 3wST7KqJZ8mqgBlXdbs0tZKRvnWiixcUSOoJCeIxALkxbp+pZBI/0cAycC+1+3U3PyjQ6ao39XY95wHuedSLVg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8573
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Wed, 25 Oct 2023 12:05:06 -0700 (PDT)

On 10/24/2023 12:01, Ingo Molnar wrote:
> 
> * Mario Limonciello <mario.limonciello@amd.com> wrote:
> 
>> +Tom
>>
>> On 10/24/2023 03:36, Ingo Molnar wrote:
>>>
>>> * Mario Limonciello <mario.limonciello@amd.com> wrote:
>>>
>>>> If x2apic was enabled during boot with parallel startup
>>>> it will be needed during resume from suspend to ram as well.
>>>>
>>>> Store whether to enable into the smpboot_control global variable
>>>> and during startup re-enable it if necessary.
>>>>
>>>> Cc: stable@vger.kernel.org # 6.5+
>>>> Fixes: 0c7ffa32dbd6 ("x86/smpboot/64: Implement arch_cpuhp_init_parallel_bringup() and enable it")
>>>> Signed-off-by: Mario Limonciello <mario.limonciello@amd.com>
>>>> ---
>>>>    arch/x86/include/asm/smp.h   |  1 +
>>>>    arch/x86/kernel/acpi/sleep.c | 12 ++++++++----
>>>>    arch/x86/kernel/head_64.S    | 15 +++++++++++++++
>>>>    3 files changed, 24 insertions(+), 4 deletions(-)
>>>>
>>>> diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h
>>>> index c31c633419fe..86584ffaebc3 100644
>>>> --- a/arch/x86/include/asm/smp.h
>>>> +++ b/arch/x86/include/asm/smp.h
>>>> @@ -190,6 +190,7 @@ extern unsigned long apic_mmio_base;
>>>>    #endif /* !__ASSEMBLY__ */
>>>>    /* Control bits for startup_64 */
>>>> +#define STARTUP_ENABLE_X2APIC	0x40000000
>>>>    #define STARTUP_READ_APICID	0x80000000
>>>>    /* Top 8 bits are reserved for control */
>>>> diff --git a/arch/x86/kernel/acpi/sleep.c b/arch/x86/kernel/acpi/sleep.c
>>>> index 6dfecb27b846..29734a1299f6 100644
>>>> --- a/arch/x86/kernel/acpi/sleep.c
>>>> +++ b/arch/x86/kernel/acpi/sleep.c
>>>> @@ -11,6 +11,7 @@
>>>>    #include <linux/dmi.h>
>>>>    #include <linux/cpumask.h>
>>>>    #include <linux/pgtable.h>
>>>> +#include <asm/apic.h>
>>>>    #include <asm/segment.h>
>>>>    #include <asm/desc.h>
>>>>    #include <asm/cacheflush.h>
>>>> @@ -129,11 +130,14 @@ int x86_acpi_suspend_lowlevel(void)
>>>>    	 */
>>>>    	current->thread.sp = (unsigned long)temp_stack + sizeof(temp_stack);
>>>>    	/*
>>>> -	 * Ensure the CPU knows which one it is when it comes back, if
>>>> -	 * it isn't in parallel mode and expected to work that out for
>>>> -	 * itself.
>>>> +	 * Ensure x2apic is re-enabled if necessary and the CPU knows which
>>>> +	 * one it is when it comes back, if it isn't in parallel mode and
>>>> +	 * expected to work that out for itself.
>>>>    	 */
>>>> -	if (!(smpboot_control & STARTUP_PARALLEL_MASK))
>>>> +	if (smpboot_control & STARTUP_PARALLEL_MASK) {
>>>> +		if (x2apic_enabled())
>>>> +			smpboot_control |= STARTUP_ENABLE_X2APIC;
>>>> +	} else
>>>>    		smpboot_control = smp_processor_id();
>>>
>>> Yeah, so instead of adding further kludges to the 'parallel bringup is
>>> possible' code path, which is arguably a functional feature that shouldn't
>>> have hardware-management coupled to it, would it be possible to fix
>>> parallel bringup to AMD-SEV systems, so that this code path isn't a
>>> quirk-dependent "parallel boot" codepath, but simply the "x86 SMP boot
>>> codepath", where all SMP x86 systems do a parallel bootup?
>>>
>>> The original commit by Thomas says:
>>>
>>>     0c7ffa32dbd6 ("x86/smpboot/64: Implement arch_cpuhp_init_parallel_bringup() and enable it")
>>>
>>>     | Unfortunately there is no RDMSR GHCB protocol at the moment, so enabling
>>>     | AMD-SEV guests for parallel startup needs some more thought.
>>>
>>> But that was half a year ago, isn't there RDMSR GHCB access code available now?
>>>
>>> This code would all read a lot more natural if it was the regular x86 SMP
>>> bootup path - which it is 'almost' today already, modulo quirk.
>>>
>>> Obviously coupling functional features with hardware quirks is fragile, for
>>> example your patch extending x86 SMP parallel bringup doesn't extend the
>>> AMD-SEV case, which may or may not matter in practice.
>>>
>>> So, if it's possible, it would be nice to fix AMD-SEV systems as well and
>>> remove this artificial coupling.
>>
>> It probably isn't clear since I didn't mention it in the commit message, but
>> this is not a system that supports AMD-SEV.  This is a workstation that
>> supports x2apic.  I'll clarify that for V2.
> 
> Yes, I suspected as much, but that's irrelevant to the arguments I
> outlined, that extending upon this quirk that makes SMP parallel bringup HW
> environment dependent, and then coupling s2ram x2apic re-enablement to that
> functional feature is inviting trouble in the long run.
> 

I spent some more time looking at ways to decouple this, and AFAICT 
thaw_secondary_cpus() doesn't actually bring CPUs back after resume in 
parallel mode.

To be symmetrical with that, another way to solve this that removes the 
"HW environment" aspect is to disable parallel boot for resume from 
sleep entirely.

Like this:

diff --git a/arch/x86/kernel/acpi/sleep.c b/arch/x86/kernel/acpi/sleep.c
index 6dfecb27b846..9265d97f497b 100644
--- a/arch/x86/kernel/acpi/sleep.c
+++ b/arch/x86/kernel/acpi/sleep.c
@@ -128,13 +128,12 @@ int x86_acpi_suspend_lowlevel(void)
          * value is in the actual %rsp register.
          */
         current->thread.sp = (unsigned long)temp_stack + 
sizeof(temp_stack);
-       /*
-        * Ensure the CPU knows which one it is when it comes back, if
-        * it isn't in parallel mode and expected to work that out for
-        * itself.
+       /*
+        * Don't use parallel startup for resume from sleep. This avoids
+        * hangs that may occur if x2apic was in use but platform
+        * has not enabled x2apic on it's own after resume.
          */
-       if (!(smpboot_control & STARTUP_PARALLEL_MASK))
-               smpboot_control = smp_processor_id();
+       smpboot_control = smp_processor_id();
  #endif
         initial_code = (unsigned long)wakeup_long64;
         saved_magic = 0x123456789abcdef0L;


> For example, what guarantees that the x2apic will be turned back on after
> suspend if a system is booted with maxcpus=1?

lapic_resume() will do this after the boot CPU makes it up.

> 
> Obviously something very close to your fix is needed.
> 

Given lapic_resume() handles this, I'd think with the style fixups you 
suggested my patch is appropriate.

>> I've looped Tom in to comment whether it's possible to improve AMD-SEV as
>> well.
> 
> Thanks!
> 
> 	Ingo