Received-SPF: pass (google.com: domain of linux-kernel+bounces-15994-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Message-ID: <b82bb32b-3348-4c18-b07e-34f523ae93b5@amd.com>
Date: Wed, 3 Jan 2024 14:41:39 -0600
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH] x86/sev: Add support for allowing zero SEV ASIDs.
Content-Language: en-US
To: Sean Christopherson <seanjc@google.com>
Cc: pbonzini@redhat.com, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de,
 dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com,
 thomas.lendacky@amd.com, kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
 joro@8bytes.org
References: <20240102232136.38778-1-Ashish.Kalra@amd.com>
 <ZZSqkm5WNEUuuA_h@google.com>
From: "Kalra, Ashish" <ashish.kalra@amd.com>
In-Reply-To: <ZZSqkm5WNEUuuA_h@google.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?utf-8?B?ZU40TStQblltbHFleCtFWmx4ZjlEL0prRTBSYWFDTk9XY0ZLSmx3ZURqUUtM?=
 =?utf-8?B?aVg4V3NjUi9IazBVRUl6dlhNWHVlRGdjSm9mNkJVMVZxTGJDcmtKdHNyMDVX?=
 =?utf-8?B?cWswVmVJOVNXZG80RldtRUloMEM5dk9FOUFqVGF5RHRZNVQyejF4Wk9zdWc5?=
 =?utf-8?B?aTU4SVR3cWRPZzREU3VhNFRGSVVBdDNpa3lPaThRQmhBdnJ5dU1TdmxCS1pp?=
 =?utf-8?B?OXdEclFzbUVQajEvRjE4MG1sTE5Ba0R1d2l5U0o3N0hzd2VJVkl2TmZHdEt4?=
 =?utf-8?B?ZnNLSTU3aGxKdlRYTENqSEg0VFgwYlhLWi9Kd0NTZW9UY25icW96b1lvSjZ6?=
 =?utf-8?B?eEpHVkJyWk9rMFhFeHJxbHNuck11Ukx1SE9sQ2ltdFhMdG9sby90ZHR2ejZN?=
 =?utf-8?B?K2IvMFllOGN4blVOcTBwTGtyVzNYZFd2NkwxSVViNlB0ZkxIUFAzYmdoN1pi?=
 =?utf-8?B?SGhEVWVQQk9OMkMyOGZyL1ZTSGVIRnBGN2pSQ0Q0RXp5SThWN0oyTi93a2Rw?=
 =?utf-8?B?WTRkVVpleXZ4V21OS0R4Q0laQ041Q2hhN0ZYQUw3QjZST21kZnJZZ3F3MTd4?=
 =?utf-8?B?RTlta1JKNCtUc084M2Q4Tjl0N1J0THNCVUZyOU1uWUQ1Tm9YbjM5YXZrNHJa?=
 =?utf-8?B?ZnNZcllvUDNocUpZMXJyRFFJcXhkVlRzd3BVQVlTaDlEcjRscEMzUVRyRzBS?=
 =?utf-8?B?U2h1ZmtSeTlERzV6STZkWGdaa2tucFRvUjd0STRVeVBIajNkNlBsWWRXUXQx?=
 =?utf-8?B?MFZGMTU0MVNwRkZRcTJOY2VFZy96QkRFNHA5RFZJK3A1QzFQS3ZYQUY2RUJo?=
 =?utf-8?B?Y3NoZXdXemhGVEExQXIvYkVqUUtidGJNR0Z3dVJpcEZzZ3p2Y0NuSFN4azM0?=
 =?utf-8?B?SjJvR1F3VXFOTGF2WnNVcGZCWTBveEc5cS9YT1RZMUhNQmkwQVh0UlBVZXp1?=
 =?utf-8?B?dTZsNEVkRUlzbXRJbW1OTHhyMmZCVm5XSkJjRGVsWkNlQm1Na3NCMzh5c3hM?=
 =?utf-8?B?TEFKSTZwbFcyY05mcklQYzRML1p6Y29YSlZaNHF2bTJMKzRVOHFaTEd5REU4?=
 =?utf-8?B?YVlsTUF6MllDUk8rZTcxS3lpNlNSMGgvanpRNkpoTGxyaG1nMS90NXg0ME5Q?=
 =?utf-8?B?WmVsRFE3S0R2c2VRUWd3elZoeXRwZkhuL2hnS2xQanB0d2grcnFkMm8rUDlw?=
 =?utf-8?B?TlFydzZYQXdKWC9uY0tRdjg1UTVDS2VVY2J1czZHWkkrLzN0bmd1UDdYTElK?=
 =?utf-8?B?ZWtlb2ZRV25yLzk0dVl1WndRanR0WkdpbkFvUlg0K3pibGl2Ky9La0ErZHBq?=
 =?utf-8?B?MnFXTWJOR0NxdlhJY2Yrc1hreFFNQ254S1VTS09LdFNPWm5PNERHMFoxR2hx?=
 =?utf-8?B?YW5aYVBWU3I2TS9iVlhVaVk5L1I5YjR0NTg1eFhMOEhvU0JyWTFxNjYrb0VC?=
 =?utf-8?B?cGpQVm43SVQwL1JMMGttTEpQTk9FajVBZkxCUXpWMlNzUzdxM3B6U0xBNkRy?=
 =?utf-8?B?aGxlMVhrSzhDY3hkNWxLYUpqSThzbjdCblpvZ0NNQVRQNVZXdjR5MHlmalEx?=
 =?utf-8?B?WXlnR0NKTHBMSmdnRXlZRlYzVUp2UTkrVW1QMTZMMFdhc1ZHeGRHMHUwV1N0?=
 =?utf-8?B?bU90T1hXRVhMcUhHRmJFcWp2cjJKRUc3cDhzeHRPTnpxTVRZS001Tlpic1JN?=
 =?utf-8?B?cDY0TUY2Vkl6OHFRS2Z5VzgrUGF3YkR6RzlYeURBQWlBM0JnTzN4TGZNa2VL?=
 =?utf-8?B?dGpQR2Ywc3NoNjQvT1R5bWFnZzhqMGNKbHdTeW9IeWloM2IvVnM5S01MRy9j?=
 =?utf-8?B?ZnJTQk02eVRwU2RFRnp0bGFQWVVTSjNjazlPY01nQWdZQ2J2V2hoamdTeURD?=
 =?utf-8?B?QTRWK0FmcFhkT2FxeXFpSVViV1NJZ3ZRQ0J0S3lXYnhkcUExd2o0eUZNQXlH?=
 =?utf-8?B?Um5NVkliaTZKY2dRQ2hORjRKRGlGWWgyNW9DejNPUUt3eW1rRWVFNW5QTlFq?=
 =?utf-8?B?YlVwVGwzekFLYlN2NkR4UU9XUlJmcGo3ZUhRNjJiWVRJc0tnZTBGTlp2bUlm?=
 =?utf-8?B?OVh0elBHZEREQVNqUTlsNTlTT0JsazlpbmtHTWhtM2VRRG95UCtkd2YzeWk2?=
 =?utf-8?Q?3K2SAVLHeKe+epq9INjaftLBl?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 7f7e0a2e-486a-4029-3be7-08dc0c9c643c
X-MS-Exchange-CrossTenant-AuthSource: SN6PR12MB2767.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Jan 2024 20:41:41.7445
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: lWuGO6VQHqCAePDtLxHOwBrpfTs5gWv7G+75oNGkNYaEkv9c2OUPIBuNSBc54dHR2EU+Og5049rDaldmGbu+1w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB6837

Hello Sean,

On 1/2/2024 6:30 PM, Sean Christopherson wrote:
> On Tue, Jan 02, 2024, Ashish Kalra wrote:
>> @@ -2172,8 +2176,10 @@ void sev_vm_destroy(struct kvm *kvm)
>>   
>>   void __init sev_set_cpu_caps(void)
>>   {
>> -	if (!sev_enabled)
>> +	if (!sev_guests_enabled) {
> Ugh, what a mess.  The module param will show sev_enabled=false, but the caps
> and CPUID will show SEV=true.
>
> And this is doubly silly because "sev_enabled" is never actually checked, e.g.
> if misc cgroup support is disabled, KVM_SEV_INIT will try to reclaim ASIDs and
> eventually fail with -EBUSY, which is super confusing to users.

But this is what we expect that KVM_SEV_INIT will fail. In this case, 
sev_asid_new() will not actually

try to reclaim any ASIDs as sev_misc_cg_try_charge() will fail before 
any ASID bitmap walking/reclamation and

return an error which will eventually return -EBUSY to the user.

>
> The other weirdness is that KVM can cause sev_enabled=false && sev_es_enabled=true,
> but if *userspace* sets sev_enabled=false then sev_es_enabled is also forced off.
But that is already the behavior without this patch applied.
>
> In other words, the least awful option seems to be to keep sev_enabled true :-(
>
>>   		kvm_cpu_cap_clear(X86_FEATURE_SEV);
>> +		return;
> This is blatantly wrong, as it can result in KVM advertising SEV-ES if SEV is
> disabled by the user.
No, this ensures that we don't advertise any SEV capability if neither 
SEV/SEV-ES or in future SNP is enabled.
>
>> +	}
>>   	if (!sev_es_enabled)
>>   		kvm_cpu_cap_clear(X86_FEATURE_SEV_ES);
>>   }
>> @@ -2229,9 +2235,11 @@ void __init sev_hardware_setup(void)
>>   		goto out;
>>   	}
>>   
>> -	sev_asid_count = max_sev_asid - min_sev_asid + 1;
>> -	WARN_ON_ONCE(misc_cg_set_capacity(MISC_CG_RES_SEV, sev_asid_count));
>> -	sev_supported = true;
>> +	if (min_sev_asid <= max_sev_asid) {
>> +		sev_asid_count = max_sev_asid - min_sev_asid + 1;
>> +		WARN_ON_ONCE(misc_cg_set_capacity(MISC_CG_RES_SEV, sev_asid_count));
>> +		sev_supported = true;
>> +	}
>>   
>>   	/* SEV-ES support requested? */
>>   	if (!sev_es_enabled)
>> @@ -2262,7 +2270,8 @@ void __init sev_hardware_setup(void)
>>   	if (boot_cpu_has(X86_FEATURE_SEV))
>>   		pr_info("SEV %s (ASIDs %u - %u)\n",
>>   			sev_supported ? "enabled" : "disabled",
>> -			min_sev_asid, max_sev_asid);
>> +			sev_supported ? min_sev_asid : 0,
>> +			sev_supported ? max_sev_asid : 0);
> I honestly think we should print the "garbage" values.  The whole point of
> printing the min/max SEV ASIDs was to help users understand why SEV is disabled,
> i.e. printing zeroes is counterproductive.
>
>>   	if (boot_cpu_has(X86_FEATURE_SEV_ES))
>>   		pr_info("SEV-ES %s (ASIDs %u - %u)\n",
>>   			sev_es_supported ? "enabled" : "disabled",
> It's all a bit gross, but I think we want something like this (I'm definitely
> open to suggestions though):
>
> diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
> index d0c580607f00..bfac6d17462a 100644
> --- a/arch/x86/kvm/svm/sev.c
> +++ b/arch/x86/kvm/svm/sev.c
> @@ -143,8 +143,20 @@ static void sev_misc_cg_uncharge(struct kvm_sev_info *sev)
>   
>   static int sev_asid_new(struct kvm_sev_info *sev)
>   {
> -       int asid, min_asid, max_asid, ret;
> +       /*
> +        * SEV-enabled guests must use asid from min_sev_asid to max_sev_asid.
> +        * SEV-ES-enabled guest can use from 1 to min_sev_asid - 1.  Note, the
> +        * min ASID can end up larger than the max if basic SEV support is
> +        * effectively disabled by disallowing use of ASIDs for SEV guests.
> +        */
> +       unsigned int min_asid = sev->es_active ? 1 : min_sev_asid;
> +       unsigned int max_asid = sev->es_active ? min_sev_asid - 1 : max_sev_asid;
> +       unsigned int asid;
>          bool retry = true;
> +       int ret;
> +
> +       if (min_asid > max_asid)
> +               return -ENOTTY;
>   

This will still return -EBUSY to user. This check here or the failure 
return from sev_misc_cg_try_charge() are quite similar in that sense.

My point is that the same is achieved quite cleanly with 
sev_misc_cg_try_charge() too.

>          WARN_ON(sev->misc_cg);
>          sev->misc_cg = get_current_misc_cg();
> @@ -157,12 +169,6 @@ static int sev_asid_new(struct kvm_sev_info *sev)
>   
>          mutex_lock(&sev_bitmap_lock);
>   
> -       /*
> -        * SEV-enabled guests must use asid from min_sev_asid to max_sev_asid.
> -        * SEV-ES-enabled guest can use from 1 to min_sev_asid - 1.
> -        */
> -       min_asid = sev->es_active ? 1 : min_sev_asid;
> -       max_asid = sev->es_active ? min_sev_asid - 1 : max_sev_asid;
>   again:
>          asid = find_next_zero_bit(sev_asid_bitmap, max_asid + 1, min_asid);
>          if (asid > max_asid) {
> @@ -2232,8 +2238,10 @@ void __init sev_hardware_setup(void)
>                  goto out;
>          }
>   
> -       sev_asid_count = max_sev_asid - min_sev_asid + 1;
> -       WARN_ON_ONCE(misc_cg_set_capacity(MISC_CG_RES_SEV, sev_asid_count));
> +       if (min_sev_asid <= max_sev_asid) {
> +               sev_asid_count = max_sev_asid - min_sev_asid + 1;
> +               WARN_ON_ONCE(misc_cg_set_capacity(MISC_CG_RES_SEV, sev_asid_count));
> +       }
>          sev_supported = true;
>   
>          /* SEV-ES support requested? */
> @@ -2264,8 +2272,9 @@ void __init sev_hardware_setup(void)
>   out:
>          if (boot_cpu_has(X86_FEATURE_SEV))
>                  pr_info("SEV %s (ASIDs %u - %u)\n",
> -                       sev_supported ? "enabled" : "disabled",
> -                       min_sev_asid, max_sev_asid);
> +                       sev_supported ? (min_sev_asid <= max_sev_asid ? "enabled" : "unusable") : "disabled",
> +                       sev_supported ? min_sev_asid : 0,
> +                       sev_supported ? max_sev_asid : 0);

We are not showing min and max ASIDs for SEV as {0,0} with this patch as 
sev_supported is true ?

Thanks, Ashish

>          if (boot_cpu_has(X86_FEATURE_SEV_ES))
>                  pr_info("SEV-ES %s (ASIDs %u - %u)\n",
>                          sev_es_supported ? "enabled" : "disabled",