Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.32 as permitted sender) client-ip=23.128.96.32;
Message-ID: <53e60186-674f-449b-a99f-01cd759c65de@amd.com>
Date:   Mon, 25 Sep 2023 19:23:04 +1000
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH kernel] x86/compressed/64: reduce #VC nesting for
 intercepted CPUID for SEV-SNP guest
Content-Language: en-US
To:     Ingo Molnar <mingo@kernel.org>
Cc:     x86@kernel.org, linux-kernel@vger.kernel.org,
        Tom Lendacky <thomas.lendacky@amd.com>
References: <20230925042302.593317-1-aik@amd.com> <ZRFHtCCsL4kKajKF@gmail.com>
From:   Alexey Kardashevskiy <aik@amd.com>
In-Reply-To: <ZRFHtCCsL4kKajKF@gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?VnZkOGZRVHlhZEg0VkpXc2g3c3lTUERuVnUxekUwQngwNXFPUkxNa1R1SkFa?=
 =?utf-8?B?bzFoeHRlTWtJNDVrV01CaGRCMnJ4aW8xNEFxYkNuTUJVZHh2TmhhK3hFcE5J?=
 =?utf-8?B?MmJxVnRjdkpzM04rd1RqSFVER2RmWnFEQk1jZHg2bjdxMzV6TlZUQWx3SU5R?=
 =?utf-8?B?dUdlYzh2MWdqRXNrYmJPVUduT0tJK3k2SHlNYzJoMXpMTnkweHFFRjNUTEhT?=
 =?utf-8?B?MTVZMzZqOGxWcnUrN2xzamx6Q2N2SVBQc0VqcitqRFVDZjBuUHN5Q1RzUEIr?=
 =?utf-8?B?aXVsbzNGT1BLNjVFZXUzUGVGaG9hakFjZmN3d3ArRXNTb1Z2ZWtCcUlQbWVm?=
 =?utf-8?B?YzlBN0FHdEpDVldmOFpWRC9DdjhoUStGT2d3aGJmdGhnZHl1R1pwaFFyWkdq?=
 =?utf-8?B?UHU2eGppcUtzVmxuZ2R5Z25UeTlHWkVaaWhwUHlmKzIyeFFUbytTNXRHVjlO?=
 =?utf-8?B?T0lKbzlaWDMwWVdQUkZJWEhXZVNVcVFnWXRmSlZBZ0JIaXpFVHcvQ2lxNzBG?=
 =?utf-8?B?OEhBcDJOcXR5REtFaVByWldSS3J5R1F4QXRlYXhjY3hTQ0ZvRSsxSjZZYWgv?=
 =?utf-8?B?akgwblpZUFU1TnlFaWMzWXlMbEF3NlJONUpyNS9XQTNwTVY4WkJYREorbjRi?=
 =?utf-8?B?dms5cFJUSVR4SzhhN2FVUHdRZGZKbVZVeHhtMWlCdWRlYU5BN1IxeWJLa0p2?=
 =?utf-8?B?MXg0VE1MYWlPYUtLRnQ2RnhJN3hHRzFwMS8xdXQrNEZSZFAvbmdwZHhhNHJ0?=
 =?utf-8?B?VDE3cStMNUR5d3RqWlNKcmJ2OXZoWExETjc1a0ljKzJGcjY1V3doRFg2bU1W?=
 =?utf-8?B?NG1RN1FObk5vcnU3akNDc0MzcGFqbDdOR2ljTHUzTTVKQjNNdEp3TGxoNXEv?=
 =?utf-8?B?QmxJeGRPZFBnRFlIMDM2MXN1ZUQ4YUp2UmVSYit6TGlhNEdFelRFUTFZOEFt?=
 =?utf-8?B?aW5VcDQ4V2F2NU5QZmVpaDU1MFVzRE5KWEpGYWFEVy9hSDhvZDRPSm82UHlY?=
 =?utf-8?B?cDAvSnlNYmxGdGNDUFpDd1h6S0RkZTFxcGo0Q2xPcTBMdThTMDZMeGhBWEYw?=
 =?utf-8?B?bGp0Skdjazdrd1djQ0E4TVNOcTJBVnRyNXJlaUJrODRNWVd5Y0FYU21POWJs?=
 =?utf-8?B?ek5PanFWVEpGSFJUYnAvVStSZnJZTzJLYm5Bd3ZEY0tDbW1WRmlydFFaRmYr?=
 =?utf-8?B?TUowZmxyTCszZzVERkpscnRrdXhxc09oQUliSXpPMU5tVGJZZkJ6aE53STdl?=
 =?utf-8?B?VTljTnBpYzB4OXExL0VsaVlYbmtiN2FlcW41SGxBSGpDRzJ5ZjRyWEtpL0hM?=
 =?utf-8?B?dlM1N29DU2JQQmFlMU11MFpEdHlnNVpUTERhRXBYcFdtTU8wZnNXdlFmdHpM?=
 =?utf-8?B?VG1lb1R4MHhva1ZmYVdOTWhxcFRhUm4xNEVXaXNUZUJuL2dNTk9oZ21ZWTg5?=
 =?utf-8?B?b1JOcTBOUlcrdGcya1JPUmtzU0tHSksvUkF6MnpGQkdsNWVzSjBPY01lcnJq?=
 =?utf-8?B?cmtNcUNrS3ZzZmN1OHF5QjJRNnh4N1V6UzI1MEFlZjFZME9JTEVIKzFNbHBL?=
 =?utf-8?B?UGNpLzhPVFgwRXJEMUtwdWh3TkdldTN4REk3UUlySlB0aGxZWHhWaHl0blE1?=
 =?utf-8?B?Rkt1bkEyOHIvUm9ZQ3NmQ0JKbkJxMnNDcTZnZjBGc1VFWGxSTFpvVUJSZllj?=
 =?utf-8?B?UERlZFVvNkY4M3psRGdjcy9ZeVY5TjIrK2Z5V3RMZHRPdDZOSFB4RkZWZkVP?=
 =?utf-8?B?aU1mdENBVTdjaDJTS0ttTm8vRHQ3Wm83d09kajJIL2FPSXp5NnNTSE5YWGtp?=
 =?utf-8?B?czJBRnVZOXYzMGx6SVFLYXZNd1BKUmlJK2tXdjZXNVErZUkzelppenpndGs1?=
 =?utf-8?B?ejVPTEtjajZHM2ZvSkNiTG9YaDgxbXpIT1RPNlJ5OE9uNVB6WFY4WHkxT3g1?=
 =?utf-8?B?dEkvWUUvd0Q0SmRob3VQUWE1aDdzNUkyQ3NZekUwRFJJVFY3NFF6ZUVrZHhW?=
 =?utf-8?B?bkpZeUNhMnlpSHpEWGJuUitjbE9kREFPSGZvTHBZRzhlQit3NTJqZzk4ai9I?=
 =?utf-8?B?dUpzR3BFdGFuQ3EzVlA0cEdva2FIRFdITUhybVR5eFdjMXZZQUs4MStranNx?=
 =?utf-8?Q?9TOgAjLuq31RJQ0KWJnRUjseV?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 15ab9ffc-d1ff-4f61-5ed3-08dbbda909be
X-MS-Exchange-CrossTenant-AuthSource: CH3PR12MB9194.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Sep 2023 09:23:11.4835
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: upCvyIO/SJ96qcsVVz1ZBqNfKv5iR6Cai7fR6YQPTJXNUgdtvB8evwIiXj2UPVEDe7xFN98Mf7iYUzLl7gm47g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB6892
X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Mon, 25 Sep 2023 02:23:43 -0700 (PDT)


On 25/9/23 18:41, Ingo Molnar wrote:
> 
> * Alexey Kardashevskiy <aik@amd.com> wrote:
> 
>> For certain intercepts an SNP guest uses the GHCB protocol to talk to
>> the hypervisor from the #VC handler. The protocol requires a shared page so
>> there is one per vCPU. In case NMI arrives in a middle of #VC or the NMI
>> handler triggers a #VC, there is another "backup" GHCB page which stores
>> the content of the first one while SVM_VMGEXIT_NMI_COMPLETE is sent.
>> The vc_raw_handle_exception() handler manages main and backup GHCB pages
>> via __sev_get_ghcb/__sev_put_ghcb.
>>
>> This works fine for #VC and occasional NMIs. This does not work so fine if
>> the #VC handler causes intercept + another #VC, if NMI arrives during
>> the second #VC, there are no more pages for SVM_VMGEXIT_NMI_COMPLETE.
>> The problem place is the #VC CPUID handler. Running perf in the SNP guest
>> crashes with:
>>
>> Kernel panic - not syncing: Unable to handle #VC exception! GHCB and Backup GHCB are already in use
>>
>> vc_raw_handle_exception #1: exit_code 72 (CPUID) eax d ecx 1
>> We lock the main GHCB and while it is locked we get to
>> snp_cpuid_postprocess() which executes "rdmsr" of MSR_IA32_XSS==0xda0 which
>> triggers:
>>
>> vc_raw_handle_exception #2: exit_code 7c (MSR) ecx da0
>> Here we lock the backup ghcb.
>>
>> And then PMC NMI comes which cannot complete as there is no GHCB page left
>> to use:
>>
>> CPU: 5 PID: 566 Comm: touch Not tainted 6.5.0-rc2-aik-ad9c-g7413e71d3dcf-dirty #27
>> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown unknown
>> Call Trace:
>>   <NMI>
>>   dump_stack_lvl+0x44/0x60
>>   panic+0x222/0x310
>>   ____sev_get_ghcb+0x21e/0x220
>>   __sev_es_nmi_complete+0x28/0xf0
>>   exc_nmi+0x1ac/0x1c0
>>   end_repeat_nmi+0x16/0x67
>> ...
>>   </NMI>
>>   <TASK>
>>   vc_raw_handle_exception+0x9e/0x2c0
>>   kernel_exc_vmm_communication+0x4d/0xa0
>>   asm_exc_vmm_communication+0x31/0x60
>> RIP: 0010:snp_cpuid+0x2ad/0x420
>>
>> Drop one #VC by replacing "rdmsr" with GHCB's VMGEXIT to read the value from
>> the hypervisor.
>>
>> Fixes: ee0bfa08a345 ("x86/compressed/64: Add support for SEV-SNP CPUID table in #VC handlers")
>> Cc: x86@kernel.org
>> Signed-off-by: Alexey Kardashevskiy <aik@amd.com>
>> ---
>>
>> This is made on top of (which has the "efi/unaccepted: Make sure unaccepted table is mapped"
>> fix for booting SNP):
>> b996cbe1203c (tip/master) 15 hours ago Ingo Molnar Merge branch into tip/master: 'x86/tdx'
>>
>> plus:
>> https://lore.kernel.org/lkml/a5856fa1ebe3879de91a8f6298b6bbd901c61881.1690578565.git.thomas.lendacky@amd.com/
>> ---
>>   arch/x86/kernel/sev-shared.c | 18 +++++++++++++-----
>>   1 file changed, 13 insertions(+), 5 deletions(-)
>>
>> diff --git a/arch/x86/kernel/sev-shared.c b/arch/x86/kernel/sev-shared.c
>> index e73c90c9cc5b..399219de5a9b 100644
>> --- a/arch/x86/kernel/sev-shared.c
>> +++ b/arch/x86/kernel/sev-shared.c
>> @@ -477,11 +477,19 @@ static int snp_cpuid_postprocess(struct ghcb *ghcb, struct es_em_ctxt *ctxt,
>>   		if (leaf->subfn == 1) {
>>   			/* Get XSS value if XSAVES is enabled. */
>>   			if (leaf->eax & BIT(3)) {
>> -				unsigned long lo, hi;
>> -
>> -				asm volatile("rdmsr" : "=a" (lo), "=d" (hi)
>> -						     : "c" (MSR_IA32_XSS));
>> -				xss = (hi << 32) | lo;
>> +				/*
>> +				 * Since we're here, it is SNP and rdmsr will trigger
>> +				 * another #VC and waste one of just two GHCB pages.
>> +				 * Skip the intercept and do direct hypercall.
>> +				 */
>> +				ghcb_set_rcx(ghcb, MSR_IA32_XSS);
>> +				if (sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_MSR, 0, 0) != ES_OK)
>> +					return -EINVAL;
>> +
>> +				xss = (ghcb->save.rdx << 32) | ghcb->save.rax;
>> +
>> +				/* Invalidate qwords for likely another following GHCB call */
>> +				vc_ghcb_invalidate(ghcb);
> 
> Ok, so I agree with this fix, but could you please reduce the ugliness
> of this open-coded RDMSR by factoring out this sequence into a new
> helper, called rdmsr_GHCB() or so, with a similar signature as
> rdmsr(), where rdmsr_GHCB() emulates RDMSR behavior via a hypercall?
> 
> That makes this workaround to reduce nesting a lot easier to read & maintain
> in the longer run.

Currently it is:

#define rdmsr(msr, low, high)

Will the below signature do?

#define rdmsr_GHCB(msr, low, high, ghcb, ctxt) {( \
	int __ret; \
	ghcb_set_rcx((ghcb), (msr)); \
	__ret = sev_es_ghcb_hv_call((ghcb), (ctxt), SVM_EXIT_MSR, 0, 0); \
	if (__ret = ES_OK) { \
		low = (ghcb)->save.rax; \
		high = (ghcb)->save.rdx; \
	} \
	__ret; )}


rdmsr() does not return a value but rdmsr_GHCB() has to, can even make 
it static inline function, should I?


Thanks,

> 
> Thanks,
> 
> 	Ingo

-- 
Alexey