Received-SPF: pass (google.com: domain of linux-kernel+bounces-142952-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <c17d72ad-544f-0c3c-e250-a78041bfcc16@amd.com>
Date: Fri, 12 Apr 2024 17:28:12 +0200
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
Subject: Re: [PATCH v3 06/14] x86/sev: Use the SVSM to create a vCPU when not
 in VMPL0
Content-Language: en-US
To: Tom Lendacky <thomas.lendacky@amd.com>, linux-kernel@vger.kernel.org,
 x86@kernel.org, linux-coco@lists.linux.dev, svsm-devel@coconut-svsm.dev
Cc: Thomas Gleixner <tglx@linutronix.de>, Ingo Molnar <mingo@redhat.com>,
 Borislav Petkov <bp@alien8.de>, Dave Hansen <dave.hansen@linux.intel.com>,
 "H. Peter Anvin" <hpa@zytor.com>, Andy Lutomirski <luto@kernel.org>,
 Peter Zijlstra <peterz@infradead.org>,
 Dan Williams <dan.j.williams@intel.com>, Michael Roth
 <michael.roth@amd.com>, Ashish Kalra <ashish.kalra@amd.com>
References: <cover.1711405593.git.thomas.lendacky@amd.com>
 <b91122a1faf096ec89bb49653d7959158a614528.1711405593.git.thomas.lendacky@amd.com>
From: "Gupta, Pankaj" <pankaj.gupta@amd.com>
In-Reply-To: <b91122a1faf096ec89bb49653d7959158a614528.1711405593.git.thomas.lendacky@amd.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?cVNsbGRYYTZCUzA4bHlBcEFrQnZKQU81S0FvQVU0TFR3YlF2NHNsU3g4SXVv?=
 =?utf-8?B?dmlGYVlSc1N2cVA1bG52UW83RSs1NUJPMloyb2FkUzFTYmtOdUpDcTlGSGpT?=
 =?utf-8?B?UU9QK2laSk4yNDh1U2kyODUrNlBzbE9NOFEvYi9JNGFPYWRBV1NqRmZ1TDFO?=
 =?utf-8?B?Nmo5VGZPMGE5WmRnOERJS1pEM1JBQ0ZpdGtqRUcwR1J2VndoMlVoTHFOQnNj?=
 =?utf-8?B?OUE1cDNUaFRwbGdodThmR0lwYnVEcU9xRWd5b0NkaW5jcTNFdy80NHhPMm5s?=
 =?utf-8?B?ZTZxcUswM1ZuSFVWSC9xZG1EekZKSDJVSGFwdGZZSjlpL3d1RnBzSzV4Nmxo?=
 =?utf-8?B?TkRBN2xPa3hCRkN3NkNlMVpCQUYrVWQ1Z0NCVkJWa05qZ1k0RGVjMFpVQ1di?=
 =?utf-8?B?Mnd6KzBPYy9Gc1MzU0RwdVRFWlVmNm5oZWZ6Qk9jRTh2a2VpeG9MY0NkNisz?=
 =?utf-8?B?RmZTUXdZRWxvL21WaXVXTFhBN2MrZC95KytjNk4yR3c2TjlZUlhlQkEwNWEw?=
 =?utf-8?B?Z3M4dDk2cWJpNkZxcjlKTW9IV1hhRVcwOHV5R3plNEsycXJ4RmdxWEhCc3pP?=
 =?utf-8?B?RWVwTGx2d2p5SXdxOW5YbDZTak4wV21WZStYUWVENWViSnpFang0UVdCY2g1?=
 =?utf-8?B?QmJka3U1MXMyNjBLZkRhUjRsZFBCcERqekV6MC9seXc5UGF6a3BKeWs3cWZL?=
 =?utf-8?B?WkM2V0MzVmlwNmY3ZGt1UVNTdGhSZzN3NjhWZUM4MHJDNG9HN3diWFdHa0dR?=
 =?utf-8?B?NXpGZ0YzRXBZRWQ4RnV4TU5jams4L3RwQ3RvaWU3blJqYVNwQUZWeSt3YUJh?=
 =?utf-8?B?MEptV0tXbFFEaHVOVkRUOHhjczY5MmQwSWVIbFZxcUxNRG5GMHc5NjAwOE8y?=
 =?utf-8?B?L0NQaTlNeENxOGlTMm9ESUo4Wk9YMzhSczd6dE9vTlU5Ujd3eVNhdENnR25M?=
 =?utf-8?B?ZEx6YnJGUVBTWU0wUVZDNTR4SDVZQUk4dkgrVDcwUmNrRnFUY01CcmNJa0pv?=
 =?utf-8?B?Rjc3Y1BQN1NhNlh6NFJBTUNJL0RKMEdXbE9MWVlGYmNpYTdmQzlGK1pDU0gz?=
 =?utf-8?B?TnNzRk5MZUhtcjVoSTRzNzJtWVNCMXl3UFNyU2dNdkowTTB0YjZRMHpuTnFr?=
 =?utf-8?B?N1dnOTY0UVVSejZiK3Byd0Nsb0hkWnV4b09IRDJZdGszcDNvK1ZTUnVwU3ls?=
 =?utf-8?B?V1JOY1lBcHZjK0M2eWpJMC81YUNNSlB1bzRTd2xleHpNSDBVL1NSUnpEUUw0?=
 =?utf-8?B?ejlvZlgrV1VoY2JzY0hFanBRd1p4cThucUJzNHhTZHdEUVJ4WVlLa1gxNno3?=
 =?utf-8?B?V2phbjEvM2NMcUw0YTBtNWxDTlJkKzMwV1ZnSGFQYUpaNVh4MmFJakFzU2Fw?=
 =?utf-8?B?R3Y5blYyaTJRck9GcVpmMzlIVlVkb1RjYy9adzlZWDRPL0krZUJaQ0prcXpk?=
 =?utf-8?B?aUlIMGFKcUV5dVJQanA3citpczRxQ2t0VTgySDRFVzhpbFcxZks3QlJIY0Rh?=
 =?utf-8?B?S2hlZnJUWkxRb1pyL2g3SHFlQ2QybW02T2V2aUdBUXdRaHVzaXBXZ2Uxbmdx?=
 =?utf-8?B?RGhVM2NwWjZHckdGM3VzdnNaWkpGbjhVMmpxaDlieURxV0lyMURUWUsrRVdx?=
 =?utf-8?B?NllxZDgzaXorMVFPZXk3eUVoK1dJTEZTSWVRTzNzaXozNjJBUGhHdjdNSjgw?=
 =?utf-8?B?ZWRMMDh1MTkyeUJxUTRjRzVmVnNnRDVuL3Y5dStwdE9aNHpnUWxOaVNsY0ky?=
 =?utf-8?B?b1FvM25jZDVXM203dzhVTVZERUt1NExzTHJOeWZnc0FBMHpNL3RWY1FoOU5L?=
 =?utf-8?B?aTVoVVJsMWwxMkFRZklSTmVYN2pzbmg5SEhwdkVYSGN5TmdWaG5nV3Nld0p2?=
 =?utf-8?B?OHJMR0NadUExQ3JmWTNlZ1BVMC9RVUZCeU9YMzgzOE5rRTE1bjdRZGFiUkJv?=
 =?utf-8?B?S0ZYWTFza3A1czNFL2ZzYktjU2VLZXZ5MFlhMURHM0JVRlJWWkg1aUV6OElp?=
 =?utf-8?B?RGlZczJKdElwTGgycUtjR0pCL25RYXFUQ0F0dG9MVnFDM0dkbGhUUDZHb1FT?=
 =?utf-8?B?L3ZtZ1pKSzRqMDAxbml1SWdRalBlUzZVcXFKeTRwbUJMcThlaE9ERURtYXVH?=
 =?utf-8?Q?aMwzjTDFJKVe/0EdHYSKIcZib?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 398bee66-0022-4b61-468f-08dc5b053b8e
X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB2810.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Apr 2024 15:28:41.3405
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Y+lWIlFiQEvm9P0l9gISZZV9MV5916d5/+BYF1D1h6DQrz/GwzocIyMgMINntE+ThnvwNX9ifCwzeoaMJUW2Lg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYYPR12MB8854

On 3/25/2024 11:26 PM, Tom Lendacky wrote:
> Using the RMPADJUST instruction, the VSMA attribute can only be changed
> at VMPL0. An SVSM will be present when running at VMPL1 or a lower
> privilege level.
> 
> When an SVSM is present, use the SVSM_CORE_CREATE_VCPU call or the
> SVSM_CORE_DESTROY_VCPU call to perform VMSA attribute changes. Use the
> VMPL level supplied by the SVSM within the VMSA and when starting the
> AP.
> 
> Signed-off-by: Tom Lendacky <thomas.lendacky@amd.com>

Reviewed-by: Pankaj Gupta <pankaj.gupta@amd.com>

> ---
>   arch/x86/include/asm/sev.h |  2 ++
>   arch/x86/kernel/sev.c      | 60 +++++++++++++++++++++++++++++++++-----
>   2 files changed, 54 insertions(+), 8 deletions(-)
> 
> diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h
> index 204f0a4857d6..d7be613b7372 100644
> --- a/arch/x86/include/asm/sev.h
> +++ b/arch/x86/include/asm/sev.h
> @@ -229,6 +229,8 @@ struct svsm_call {
>   #define SVSM_CORE_CALL(x)		((0ULL << 32) | (x))
>   #define SVSM_CORE_REMAP_CA		0
>   #define SVSM_CORE_PVALIDATE		1
> +#define SVSM_CORE_CREATE_VCPU		2
> +#define SVSM_CORE_DELETE_VCPU		3
>   
>   #ifdef CONFIG_AMD_MEM_ENCRYPT
>   extern void __sev_es_ist_enter(struct pt_regs *regs);
> diff --git a/arch/x86/kernel/sev.c b/arch/x86/kernel/sev.c
> index d3e182d69d65..ea8b43a0f01b 100644
> --- a/arch/x86/kernel/sev.c
> +++ b/arch/x86/kernel/sev.c
> @@ -1016,7 +1016,7 @@ void snp_accept_memory(phys_addr_t start, phys_addr_t end)
>   	set_pages_state(vaddr, npages, SNP_PAGE_STATE_PRIVATE);
>   }
>   
> -static int snp_set_vmsa(void *va, bool vmsa)
> +static int base_snp_set_vmsa(void *va, bool vmsa)
>   {
>   	u64 attrs;
>   
> @@ -1034,6 +1034,40 @@ static int snp_set_vmsa(void *va, bool vmsa)
>   	return rmpadjust((unsigned long)va, RMP_PG_SIZE_4K, attrs);
>   }
>   
> +static int svsm_snp_set_vmsa(void *va, void *caa, int apic_id, bool vmsa)
> +{
> +	struct svsm_call call = {};
> +	unsigned long flags;
> +	int ret;
> +
> +	local_irq_save(flags);
> +
> +	call.caa = this_cpu_read(svsm_caa);
> +	call.rcx = __pa(va);
> +
> +	if (vmsa) {
> +		/* Protocol 0, Call ID 2 */
> +		call.rax = SVSM_CORE_CALL(SVSM_CORE_CREATE_VCPU);
> +		call.rdx = __pa(caa);
> +		call.r8  = apic_id;
> +	} else {
> +		/* Protocol 0, Call ID 3 */
> +		call.rax = SVSM_CORE_CALL(SVSM_CORE_DELETE_VCPU);
> +	}
> +
> +	ret = svsm_protocol(&call);
> +
> +	local_irq_restore(flags);
> +
> +	return ret;
> +}
> +
> +static int snp_set_vmsa(void *va, void *caa, int apic_id, bool vmsa)
> +{
> +	return vmpl ? svsm_snp_set_vmsa(va, caa, apic_id, vmsa)
> +		    : base_snp_set_vmsa(va, vmsa);
> +}
> +
>   #define __ATTR_BASE		(SVM_SELECTOR_P_MASK | SVM_SELECTOR_S_MASK)
>   #define INIT_CS_ATTRIBS		(__ATTR_BASE | SVM_SELECTOR_READ_MASK | SVM_SELECTOR_CODE_MASK)
>   #define INIT_DS_ATTRIBS		(__ATTR_BASE | SVM_SELECTOR_WRITE_MASK)
> @@ -1065,11 +1099,11 @@ static void *snp_alloc_vmsa_page(void)
>   	return page_address(p + 1);
>   }
>   
> -static void snp_cleanup_vmsa(struct sev_es_save_area *vmsa)
> +static void snp_cleanup_vmsa(struct sev_es_save_area *vmsa, int apic_id)
>   {
>   	int err;
>   
> -	err = snp_set_vmsa(vmsa, false);
> +	err = snp_set_vmsa(vmsa, NULL, apic_id, false);
>   	if (err)
>   		pr_err("clear VMSA page failed (%u), leaking page\n", err);
>   	else
> @@ -1080,6 +1114,7 @@ static int wakeup_cpu_via_vmgexit(u32 apic_id, unsigned long start_ip)
>   {
>   	struct sev_es_save_area *cur_vmsa, *vmsa;
>   	struct ghcb_state state;
> +	struct svsm_ca *caa;
>   	unsigned long flags;
>   	struct ghcb *ghcb;
>   	u8 sipi_vector;
> @@ -1126,6 +1161,12 @@ static int wakeup_cpu_via_vmgexit(u32 apic_id, unsigned long start_ip)
>   	if (!vmsa)
>   		return -ENOMEM;
>   
> +	/*
> +	 * If an SVSM is present, then the SVSM CAA per-CPU variable will
> +	 * have a value, otherwise it will be NULL.
> +	 */
> +	caa = per_cpu(svsm_caa, cpu);
> +
>   	/* CR4 should maintain the MCE value */
>   	cr4 = native_read_cr4() & X86_CR4_MCE;
>   
> @@ -1173,11 +1214,11 @@ static int wakeup_cpu_via_vmgexit(u32 apic_id, unsigned long start_ip)
>   	 *   VMPL level
>   	 *   SEV_FEATURES (matches the SEV STATUS MSR right shifted 2 bits)
>   	 */
> -	vmsa->vmpl		= 0;
> +	vmsa->vmpl		= vmpl;
>   	vmsa->sev_features	= sev_status >> 2;
>   
>   	/* Switch the page over to a VMSA page now that it is initialized */
> -	ret = snp_set_vmsa(vmsa, true);
> +	ret = snp_set_vmsa(vmsa, caa, apic_id, true);
>   	if (ret) {
>   		pr_err("set VMSA page failed (%u)\n", ret);
>   		free_page((unsigned long)vmsa);
> @@ -1193,7 +1234,10 @@ static int wakeup_cpu_via_vmgexit(u32 apic_id, unsigned long start_ip)
>   	vc_ghcb_invalidate(ghcb);
>   	ghcb_set_rax(ghcb, vmsa->sev_features);
>   	ghcb_set_sw_exit_code(ghcb, SVM_VMGEXIT_AP_CREATION);
> -	ghcb_set_sw_exit_info_1(ghcb, ((u64)apic_id << 32) | SVM_VMGEXIT_AP_CREATE);
> +	ghcb_set_sw_exit_info_1(ghcb,
> +				((u64)apic_id << 32)	|
> +				((u64)vmpl << 16)	|
> +				SVM_VMGEXIT_AP_CREATE);
>   	ghcb_set_sw_exit_info_2(ghcb, __pa(vmsa));
>   
>   	sev_es_wr_ghcb_msr(__pa(ghcb));
> @@ -1211,13 +1255,13 @@ static int wakeup_cpu_via_vmgexit(u32 apic_id, unsigned long start_ip)
>   
>   	/* Perform cleanup if there was an error */
>   	if (ret) {
> -		snp_cleanup_vmsa(vmsa);
> +		snp_cleanup_vmsa(vmsa, apic_id);
>   		vmsa = NULL;
>   	}
>   
>   	/* Free up any previous VMSA page */
>   	if (cur_vmsa)
> -		snp_cleanup_vmsa(cur_vmsa);
> +		snp_cleanup_vmsa(cur_vmsa, apic_id);
>   
>   	/* Record the current VMSA page */
>   	per_cpu(sev_vmsa, cpu) = vmsa;