Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4;
Message-ID: <1d9a3a05-fbcb-49b5-955a-a3686ab3efdd@amd.com>
Date:   Thu, 5 Oct 2023 20:36:37 +1100
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH kernel v3] x86/compressed/64: reduce #VC nesting for
 intercepted CPUID for SEV-SNP guest
Content-Language: en-US
To:     Tom Lendacky <thomas.lendacky@amd.com>, x86@kernel.org
Cc:     linux-kernel@vger.kernel.org, Ingo Molnar <mingo@redhat.com>,
        Borislav Petkov <bp@alien8.de>,
        Sean Christopherson <seanjc@google.com>
References: <20231003073123.1763564-1-aik@amd.com>
 <32c0080f-1bd1-32c4-2bfa-910537a50ca8@amd.com>
 <14619d42-b319-4b5c-9363-c75096d27584@amd.com>
 <37f536c7-0b09-89f2-5556-d403a0726fdb@amd.com>
From:   Alexey Kardashevskiy <aik@amd.com>
In-Reply-To: <37f536c7-0b09-89f2-5556-d403a0726fdb@amd.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YmFNai9ua3l5WlcrWUJrU21BVWRxZkZ6empJUFJXTUpUaWpuczF3UGl4U1dT?=
 =?utf-8?B?V3BKVXE5K0VyWWdqVkkyMkR6Z09PNEpOVU9ZK0RoOWRVb0EzS1pxUkhEUktY?=
 =?utf-8?B?alpXNFFvZy9KMFIzYXF5aDN2VDJVSEQ3OU9hVjR1bVk4dnRRdExSQm5VTnU2?=
 =?utf-8?B?ekZzZG5jQmd4bzZUL1ppSVZyOXRzV1FuY2thMExxUTJsMEFIRVhBUFVTc3B3?=
 =?utf-8?B?dVFveFFGdjBabHhMSGI0b0dGemp0SkhqUkx4amZVOGNaSjZJVXhpT0tybnBM?=
 =?utf-8?B?NWZVKzNvZmxWVjR2M3FVYlM3M2FHRm93MzMvYXUvK2VGTEhDVFZxeVpocjJK?=
 =?utf-8?B?WmZIT2pkRGd5eFp3ZDlLU2UxUCtndXN1Yjh0bDJMTXhZcEhQeFpyWWpSaXpu?=
 =?utf-8?B?NUY2amhtU09pd0ZHbzk4c21RMWhBY1dNUy9CNkFKMFlTZHE1Q2IxbDhuTmFZ?=
 =?utf-8?B?dWtmcEhmWjhmQWEvNHZBU0wwamQ5VVFtbTcwM0FaTUhBQjRmMFljVDRoRDFI?=
 =?utf-8?B?ampkYktlSndPd1JnQ1BqMDdFYXFLdERjSERDNmsxeHNYRXlPQUdyMW1ObjJj?=
 =?utf-8?B?N2VCcVBKMXdwWlREZEdjcHRlRUlQdC9pbkRvOGpEMXlJK1NtcEZ0eEQvYXZy?=
 =?utf-8?B?QTVWSWtKdU44OTRtUDZzdjdBb3Y3UXZ3Z2xOTEdhVktLVUlDTG1nYlEvZXJs?=
 =?utf-8?B?QmZLZ2Rvai85U2dFazhxSUx1YllrRlExb0xaZnBPL3VqYXgydXlLTEdEbWQ1?=
 =?utf-8?B?Mmx5T1pTK2lWWXhtYXlEOTRkeHk4d2pzS1VxUlF5QVRTdFZOTW1NRjdJcVFL?=
 =?utf-8?B?ZmZKdVpHUitEcVhZMHFkMEl1N3IwNFFFS0pQNTZsVjlacFRPc2ZKVjRRclMy?=
 =?utf-8?B?NFlvbkRFSWkvWVg4TEs5VjRkOEo0YjNoVUxRM0l1R0krUzlHMklncEV4NGxk?=
 =?utf-8?B?ZnMySTAweTBuNlFPZzhUdzJLSlhsSlVlQ04ydGhBZ1lZVFVPVXFoeGxDazNM?=
 =?utf-8?B?aCtBcUZ6d1JCMGVsZnQ0b0ttT2ZXNGJBK0pCSmp2SCt4SXJwNWNDUG1zWnh2?=
 =?utf-8?B?bkZXd2dMYkZtMWJybDNUT0V6bzUzbzZEdFdwN1BZTiswTXIrbUN1RlpOLzYx?=
 =?utf-8?B?bTM4U2lQVGxCUysrRDl5eXR0SmRFckhGTVVrVytSNCtLYzl2VXB2eWJ4cmd6?=
 =?utf-8?B?OFUyY29BYUtURzdyWURra2ZURWUyMVkwMklvdHhScEpIVmhyK05qWEZUUDhI?=
 =?utf-8?B?Nmp4b3ZXZFBSUG5wUnZudzlhYis1N2Q0RGszZit4aTFMdVhIdFAvWGI0M2Vh?=
 =?utf-8?B?ODVMNThWWU44VzFQUHZGL0p2YTFRWSthb3o1bGFyQWExdkR0S1ZEVlNMb0VF?=
 =?utf-8?B?ZEg5eVVBUEpzeFB0dGM5S3V3aWxxbUtod2RNbVA5dUd0amZwanVtQWV4Z0o4?=
 =?utf-8?B?WGM4alRXeUc5RTdMd0l3dDNqcHUzUnd2ZGlWY2N0TnRJWWcrRnpBVXRvaWox?=
 =?utf-8?B?TzNqaHVYRzVUbXNTYWpqc2FpNTVtUEtaRks3UVFnQm5xbUF3UlhuZzEyZkM5?=
 =?utf-8?B?d2gzQ1AwY2V2Znd5RzhINk13cFcyUUF2S2RoMUVhcUpuSEY5VDE3VFdZYjNS?=
 =?utf-8?B?VU0vcUdEaHBkQlBhUklMRGlZUGE2TXlnVk4yakpDcE5qc1R3N2duTzBNOFNs?=
 =?utf-8?B?aU55ai9NVmczYVJ5RndIcnhaWlZ5bm1TL0dWc0dZWTRsUEM4aGsrcnJ2RWFF?=
 =?utf-8?B?SThXQXVvT0lJQ1BOckFpSmY0bnZ6bGdqNWdXQXlVeThxbEhUNVFzWi84MFRi?=
 =?utf-8?B?VHllU1V0VEwwd2lNek5oY3VvMDRTWXRJZkFINkdaWWxiZ0NJTVlqSTU4eXFs?=
 =?utf-8?B?RWRvNjRGL0pLVVFocW1WQ2hpcWZUczR5dEE4ZXFteUtGbmJkOTNpWExiWHRz?=
 =?utf-8?B?Mm1QVkQvU25WMDF0bVJJdzY5dWdaT25qOGFmT1pEMHhOQ01yTGdkNzc0TnRh?=
 =?utf-8?B?NkFHSDk2bXdxRFNRY29Ecy9tTWo2amtGaFVBaGpqNm9aWjFWbDREWHJ4UjI1?=
 =?utf-8?B?WTZCM1VscGxFVmlaWG5vVm1BSlFQcG9sNUI4eWg4WGk5eWltdjdCS0E1VjVr?=
 =?utf-8?Q?E65c36XZUIxHY0EcgJA3Axt8A?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f5ca49c0-4984-44f5-e9e8-08dbc58697b2
X-MS-Exchange-CrossTenant-AuthSource: CH3PR12MB9194.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Oct 2023 09:36:46.5750
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: /ABBx0Q3qk8r6pnpZp3ckkbscCJZF6XguAORgSVvorJvAmc1K5juhBXRfFoX5ykeRGIi0IDxaiBCxBCYP6JfJw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL1PR12MB5111
X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_BLOCKED,SPF_HELO_PASS,SPF_NONE,URIBL_BLOCKED
        autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Thu, 05 Oct 2023 07:23:56 -0700 (PDT)


On 5/10/23 00:53, Tom Lendacky wrote:
> On 10/3/23 18:22, Alexey Kardashevskiy wrote:
>>
>> On 4/10/23 04:21, Tom Lendacky wrote:
>>> On 10/3/23 02:31, Alexey Kardashevskiy wrote:
>>>> For certain intercepts an SNP guest uses the GHCB protocol to talk to
>>>> the hypervisor from the #VC handler. The protocol requires a shared 
>>>> page so
>>>> there is one per vCPU. In case NMI arrives in a middle of #VC or the 
>>>> NMI
>>>> handler triggers a #VC, there is another "backup" GHCB page which 
>>>> stores
>>>> the content of the first one while SVM_VMGEXIT_NMI_COMPLETE is sent.
>>>> The vc_raw_handle_exception() handler manages main and backup GHCB 
>>>> pages
>>>> via __sev_get_ghcb/__sev_put_ghcb.
>>>>
>>>> This works fine for #VC and occasional NMIs but not so fine when the 
>>>> #VC
>>>> handler causes intercept + another #VC. If NMI arrives during
>>>> the second #VC, there are no more pages for SVM_VMGEXIT_NMI_COMPLETE.
>>>> The problem place is the #VC CPUID handler which reads an MSR which
>>>> triggers another #VC and if "perf" was running, panic happens:
>>>>
>>>> Kernel panic - not syncing: Unable to handle #VC exception! GHCB and 
>>>> Backup GHCB are already in use
>>>>
>>>> Add a helper similar to native_read_msr_safe() for making a direct 
>>>> hypercall
>>>> in the SEV-ES environment. Use the new helper instead of the raw 
>>>> "rdmsr" to
>>>> avoid the extra #VC event.
>>>>
>>>> Fixes: ee0bfa08a345 ("x86/compressed/64: Add support for SEV-SNP 
>>>> CPUID table in #VC handlers")
>>>> Signed-off-by: Alexey Kardashevskiy <aik@amd.com>
>>>> ---
>>>>
>>>> Based on:
>>>> https://git.kernel.org/pub/scm/linux/kernel/git/bp/bp.git/log/?h=tip-x86-urgent
>>>> which top at the time was:
>>>> 62d5e970d022 "x86/sev: Change npages to unsigned long in 
>>>> snp_accept_memory()"
>>>>
>>>> ---
>>>> Changes:
>>>> v3:
>>>> * made it a function, mimic native_read_msr_safe() which 1) returns 
>>>> value 2) returns an error
>>>> * removed debug backtraces the commit log as these were added for 
>>>> debugging and never
>>>> appear with actual kernels
>>>>
>>>>
>>>> v2:
>>>> * de-uglify by defining rdmsr_safe_GHCB()
>>>> ---
>>>>   arch/x86/kernel/sev-shared.c | 27 +++++++++++++++++---
>>>>   1 file changed, 23 insertions(+), 4 deletions(-)
>>>>
>>>> diff --git a/arch/x86/kernel/sev-shared.c 
>>>> b/arch/x86/kernel/sev-shared.c
>>>> index dcf325b7b022..494d92a71986 100644
>>>> --- a/arch/x86/kernel/sev-shared.c
>>>> +++ b/arch/x86/kernel/sev-shared.c
>>>> @@ -241,6 +241,25 @@ static enum es_result 
>>>> sev_es_ghcb_hv_call(struct ghcb *ghcb,
>>>>       return verify_exception_info(ghcb, ctxt);
>>>>   }
>>>> +
>>>> +/* Paravirt SEV-ES rdmsr which avoids extra #VC event */
>>>> +static unsigned long long ghcb_prot_read_msr(unsigned int msr, 
>>>> struct ghcb *ghcb,
>>>> +                         struct es_em_ctxt *ctxt, int *err)
>>>
>>> Alternatively you could return enum es_result and take xss as a 
>>> parameter... six of one, half dozen of another I guess.
>>
>> How do we decide on this? :)
>>
>> and yeah, I need to s/int/enum es_result/
>>
>>>> +{
>>>> +    unsigned long long ret = 0;
>>>> +
>>>> +    ghcb_set_rcx(ghcb, msr);
>>>> +
>>>> +    *err = sev_es_ghcb_hv_call(ghcb, ctxt, SVM_EXIT_MSR, 0, 0);
>>>> +    if (*err == ES_OK)
>>>> +        ret = (ghcb->save.rdx << 32) | ghcb->save.rax;
>>>
>>> You should check ghcb_rax_is_valid(ghcb) and ghcb_rdx_is_valid(ghcb) 
>>> before using the values.
>>
>> Huh. v4 is coming then. Although what are the chances of *err == ES_OK 
>> and !ghcb_rax_is_valid() at the same time? What if *err == ES_OK and 
>> ghcb_rdx_is_valid()==true but ghcb_rax_is_valid()==false?
>>
>> return ((ghcb_rdx_is_valid(ghcb)?(ghcb->save.rdx << 32):0) |
>>      (ghcb_rax_is_valid(ghcb)?ghcb->save.rax:0;
>>
>> Or I can just drop *err, invalidate ghcb before sev_es_ghcb_hv_call() 
>> and only rely on (ghcb_rdx_is_valid() && ghcb_rax_is_valid)?
>>
>> Where should I stop with this? :)
> 
> No, you can't drop *err. The GHCB protocol specifically calls out how 
> errors can be returned and how register state is returned.
> 
> In this case, sev_es_ghcb_hv_call() will check for general errors being 
> returned from the hypervisor, e.g. non-zero SW_EXITINFO1[31:0] and that 
> is why you need to check *err.
> 
> Then you need to validate that the hypervisor set the proper registers, 
> hence the check for ghcb_rax/rdx_is_valid() (see __sev_cpuid_hv_ghcb() 
> as an example).


After an offline discussion, it turns out this intercepted rdmsr of XSS 
in this particular place (postprocessing of CPUID 0xd:0x1 bit3 == 
"XSAVES, XRSTOR, and XSS are supported") in the guest should not have 
been intercepted in the first place as it is virtualized and swapped as 
typeB, but it is intercepted as this is the default.


This applied to KVM fixes the guest crashing problem:

--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -4266,6 +4266,11 @@ static void svm_vcpu_after_set_cpuid(struct 
kvm_vcpu *vcpu)
         vcpu->arch.xsaves_enabled = guest_cpuid_has(vcpu, 
X86_FEATURE_XSAVE) &&
                                     boot_cpu_has(X86_FEATURE_XSAVE) &&
                                     boot_cpu_has(X86_FEATURE_XSAVES);
+       if (vcpu->arch.xsaves_enabled)
+               set_msr_interception(vcpu, svm->msrpm, MSR_IA32_XSS, 1, 1);


Sooo. I guess we want to fix the KVM but at least for now the guest 
needs the fix too, does not it?

And adding Sean in cc.

Thanks,


> 
> Thanks,
> Tom
> 
>>
>>>> +
>>>> +    /* Invalidate qwords for likely another following GHCB call */
>>>> +    vc_ghcb_invalidate(ghcb);
>>>
>>> We should probably call this on entry to the function, too, right? 
>>> Not sure it really matters though.
>>
>> The SVM_EXIT_MSR's handler in SVM/KVM only cares if RCX is valid in 
>> sev_es_validate_vmgexit() and the guest's ghcb_set_rcx() does that. 
>> Nothing in SVM enforces that other (unused) registers are not valid 
>> though. Thanks,
>>
>>
>>>
>>> Thanks,
>>> Tom
>>>
>>>> +
>>>> +    return ret;
>>>> +}
>>>> +
>>>>   static int __sev_cpuid_hv(u32 fn, int reg_idx, u32 *reg)
>>>>   {
>>>>       u64 val;
>>>> @@ -477,11 +496,11 @@ static int snp_cpuid_postprocess(struct ghcb 
>>>> *ghcb, struct es_em_ctxt *ctxt,
>>>>           if (leaf->subfn == 1) {
>>>>               /* Get XSS value if XSAVES is enabled. */
>>>>               if (leaf->eax & BIT(3)) {
>>>> -                unsigned long lo, hi;
>>>> +                int err = 0;
>>>> -                asm volatile("rdmsr" : "=a" (lo), "=d" (hi)
>>>> -                             : "c" (MSR_IA32_XSS));
>>>> -                xss = (hi << 32) | lo;
>>>> +                xss = ghcb_prot_read_msr(MSR_IA32_XSS, ghcb, ctxt, 
>>>> &err);
>>>> +                if (err != ES_OK)
>>>> +                    return -EINVAL;
>>>>               }
>>>>               /*
>>

-- 
Alexey