Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <c014fd8a-5456-478f-b643-44cfff27f931@intel.com>
Date:   Fri, 11 Aug 2023 11:03:53 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.14.0
Subject: Re: [PATCH v5 09/19] KVM:x86: Make guest supervisor states as
 non-XSAVE managed
To:     Paolo Bonzini <pbonzini@redhat.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>, <peterz@infradead.org>,
        "Sean Christopherson" <seanjc@google.com>
CC:     Chao Gao <chao.gao@intel.com>, <john.allen@amd.com>,
        <kvm@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        <rick.p.edgecombe@intel.com>, <binbin.wu@linux.intel.com>
References: <20230803042732.88515-1-weijiang.yang@intel.com>
 <20230803042732.88515-10-weijiang.yang@intel.com>
 <ZMuMN/8Qa1sjJR/n@chao-email>
 <bfc0b3cb-c17a-0ad6-6378-0c4e38f23024@intel.com>
 <ZM1jV3UPL0AMpVDI@google.com>
 <806e26c2-8d21-9cc9-a0b7-7787dd231729@intel.com>
 <c871cc44-b6a0-06e3-493b-33ddf4fa6e05@intel.com>
 <8396a9f6-fbc4-1e62-b6a9-3df568fd15a2@redhat.com>
Content-Language: en-US
From:   "Yang, Weijiang" <weijiang.yang@intel.com>
In-Reply-To: <8396a9f6-fbc4-1e62-b6a9-3df568fd15a2@redhat.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?K2NvWGd3NnEyQklMcktTdnlwSVlwa21mNlpickZacHVJK29Ibk9yb01wbTdF?=
 =?utf-8?B?K3IrY095dWVuTVRlZ2RoRkpVcTN3NWtReFB2QklyYlorRXJlbGpoRkYrdjRh?=
 =?utf-8?B?V1ZoUElkdkxmTUFBcTBkcWNFR0NBN0JwWGtReExoMjkvTFczMWt1TmZ1b2Ro?=
 =?utf-8?B?VXBmZ24xN2lzZzFmcjNHVm5WWnB3K3huNXFDUGRRaGVQemFEbTc0NWM4eDB2?=
 =?utf-8?B?cWhkM2dSaGxGS3JqVVR3ZjBKUWNoVWpqRVp0ZW85WkNjVmExUFdhVzlpVjRZ?=
 =?utf-8?B?bW03eThUOUVLK0JOVktlaG9DaElVUk5oSjIvWm1XVVU1ZXZLVFIvWlpLY0lr?=
 =?utf-8?B?RG1idU93c3ZsYU1HSlZXRFhHZkY5aWdIWEY5blUzbGYwNWtZWi9OcGRSSC9L?=
 =?utf-8?B?dWJSbXN4V21oTXJNN2FPNUdIZUhlSTZlUEVBWlZvUzVmK2tDcjAzeUc0MGMz?=
 =?utf-8?B?bmxaVDZIU0RiL2JhZGd3OGpsQ2oyUXRXbHg2MWpwdG9UcnNzN1hjeUpLaEl0?=
 =?utf-8?B?V21BbE93Sk51QnI1UW1QZjhPS2lZMTZ4RklYdXNZUk8zUkc1SzhaL1BzM2VM?=
 =?utf-8?B?WldxZ3Qra01NZVppYjNrNHNmRHZyRU12RGVlVURQeXlVNFFEZm1ZU0NqbnhU?=
 =?utf-8?B?K00yTGx5UXJycE8yZlNyZjl6WWI2dnEvbTZqRUtpQ0duV3BaU3BWM0EwNjl1?=
 =?utf-8?B?THhScXhVeHd6RkFlSGxYMUVmYkVlN09KMHFZTjdTYW9rZlR3cGtQSkVDMXJu?=
 =?utf-8?B?ZGc0NUlkbjVTWS9rc2R6bUNQTnpBa0pJcTF4eWFZWURkYjgxQWswekFPSXFp?=
 =?utf-8?B?c0NlWWUyMTNrb3h6eEZQaWF2dTJXcTJXRTlkNlpMU2w2OEkvaERYY3Vqd1lw?=
 =?utf-8?B?ZEhFYm9QQnVIK3NNK2JkN1Bvb2t2WUZxNm5TNk1MejdPMm1BRFJTdUFYRUpW?=
 =?utf-8?B?a01UaHhtcDNFYWp3WSt0WXE5TnQ0Y0xlSzlvYjNxUWdQOUNmVUROVHVBRVdw?=
 =?utf-8?B?eWkxWXFuak5QckttOWZtTkVDdkE3bHN1R2pkWWtseFBsTW5GUWpjZ1h3anFh?=
 =?utf-8?B?UU4zaCs5cmNmMTdkaVdQQWJ6R0JqaHJMeG9ZakEvZWU1UnlnOUFPSkpQUG5w?=
 =?utf-8?B?VURUaTdDTDdOVFpDT1p4dlZsVXY4c1IzZzhvcUE2YVk1SlpER0RNMnhWTXVv?=
 =?utf-8?B?cWJ0WlpGaHlITmg1cUN2NEhSYyttTnRMYnNQSHd6THBRbU9qTW9Nc1NicFFy?=
 =?utf-8?B?cVBwZ3Zad3JzUTR3S2E1QnhyVjh5eGJiUEErdlZ6Q0hTbUlWWlo3bFJRSEJ4?=
 =?utf-8?B?TU5iVHQwUlF0RXF2dGp3Zk1mZnI5SVB2ZEQxMHZ4UDFXRkJQaWdnNFpnSVo4?=
 =?utf-8?B?NDhIZUU1dDc0aGxFclZweFRxdFlCOXV2R2JLKzVUdjVEUk1HZHI3dTFsQkYr?=
 =?utf-8?B?RzlMNXpyaDg5azlCL0xpYm9sVUU5Skx4SG5lNVNLWGpRWG1QNkJHT25laitn?=
 =?utf-8?B?cm8vUGY0VG54NnZ5bmx0ZmRXcGRJRWxsRXppbGE1VmZ1emdOUklKMTNzNUpr?=
 =?utf-8?B?VEx6WUVHK2ozOGt4L2QzL0U0ZWEzRnpxYzJDSXZnTm82bENJa1dvZ2hDTFd2?=
 =?utf-8?B?NHpqZ1V0NFlISnh5eXJoKzFZSWtwYlRJQzd6TW1hOXpWdjcveUhhU3RYRi9J?=
 =?utf-8?B?QStRZHYyMmd5WVRoOUVrclhpd01kS1NnZm15dVFDVFdQRjBMc1JiVXFEQk56?=
 =?utf-8?B?bURKSnIrOWMvcFZSVmtJYmkyYmpmaElRblhkM1d4d0Jkc0tOZUd5Ly9oQnY1?=
 =?utf-8?B?TWVjRnlkb2lOWXN4bi9DVEI4akd5emxPT1RXWmlINzluM2lQL1hKbDJmaHll?=
 =?utf-8?B?dzZYYjcxOGsvY0tSMlVCazRtTVJnSGN4NnJXUkV3VFBqdHgrT0NaaWNTbGN5?=
 =?utf-8?B?eW9saU5QUGw4REFFR1psTWhLdTdMRnJNRUZqMk1zSHlMRHRwMHBSRmJyTSt4?=
 =?utf-8?B?NHlCMFFOWERPdXdIbnFZUHVBeFZoM2p1aDBIUWZ3a0k1WVlUbDVPQjQ1SGR1?=
 =?utf-8?B?bG1tVTRwcGYvaCtNN2Y5eEdPdE0wVS9YTGxkSGF1U0I1YjZQa2FCdTk4OUYv?=
 =?utf-8?B?NEU5QUg1MXJQQTdMSzV5cEJFbDdmemFWRDRleHlRamVLTkVCd2xIcjZJZ3B2?=
 =?utf-8?B?ckE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: aac0044c-df5f-449c-3925-08db9a179fdf
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB4965.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Aug 2023 03:04:06.1321
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Pvcl17g69BW9eF/TYyFooycGKJ5uJ1Iymu771xLGclxMYJfscodEndPhKokhRfftjqns0orfE4u71qPrzgRcRg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB7959
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,
        SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 8/10/2023 11:15 PM, Paolo Bonzini wrote:
> On 8/10/23 16:29, Dave Hansen wrote:
>> On 8/10/23 02:29, Yang, Weijiang wrote:
>> ...
>>> When KVM enumerates shadow stack support for guest in CPUID(0x7,
>>> 0).ECX[bit7], architecturally it claims both SS user and supervisor
>>> mode are supported. Although the latter is not supported in Linux,
>>> but in virtualization world, the guest OS could be non-Linux system,
>>> so KVM supervisor state support is necessary in this case.
>>
>> What actual OSes need this support?
>
> I think Xen could use it when running nested.  But KVM cannot expose support for CET in CPUID, and at the same time fake support for MSR_IA32_PL{0,1,2}_SSP (e.g. inject a #GP if it's ever written to a nonzero value).
>
> I suppose we could invent our own paravirtualized CPUID bit for "supervisor IBT works but supervisor SHSTK doesn't".  Linux could check that but I don't think it's a good idea.
>
> So... do, or do not.  There is no try. :)
>
>>> Two solutions are on the table:
>>> 1) Enable CET supervisor support in Linux kernel like user mode support.
>>
>> We _will_ do this eventually, but not until FRED is merged.  The core
>> kernel also probably won't be managing the MSRs on non-FRED hardware.
>>
>> I think what you're really talking about here is that the kernel would
>> enable CET_S XSAVE state management so that CET_S state could be managed
>> by the core kernel's FPU code.
>
> Yes, I understand it that way too.

Sorry for confusion, I missed the word "state" here.

>> That is, frankly, *NOT* like the user mode support at all.
>
> I agree.
>
>>> 2) Enable support in KVM domain.
>>>
>>> Problem:
>>> The Pros/Cons for each solution(my individual thoughts):
>>> In kernel solution:
>>> Pros:
>>> - Avoid saving/restoring 3 supervisor MSRs(PL{0,1,2}_SSP) at vCPU
>>>    execution path.
>>> - Easy for KVM to manage guest CET xstate bits for guest.
>>> Cons:
>>> - Unnecessary supervisor state xsaves/xrstors operation for non-vCPU
>>>    thread.
>>
>> What operations would be unnecessary exactly?
>
> Saving/restoring PL0/1/2_SSP when switching from one usermode task's fpstate to another.
>
>>> KVM solution:
>>> Pros:
>>> - Not touch current kernel FPU management framework and logic.
>>> - No extra space and operation for non-vCPU thread.
>>> Cons:
>>> - Manually saving/restoring 3 supervisor MSRs is a performance burden to
>>>    KVM.
>>> - It looks more like a hack method for KVM, and some handling logic
>>>    seems a bit awkward.
>>
>> In a perfect world, we'd just allocate space for CET_S in the KVM
>> fpstates.  The core kernel fpstates would have
>> XSTATE_BV[13]==XCOMP_BV[13]==0.  An XRSTOR of the core kernel fpstates
>> would just set CET_S to its init state.
>
> Yep.  I don't think it's a lot of work to implement.  The basic idea as you point out below is something like
>
> #define XFEATURE_MASK_USER_DYNAMIC XFEATURE_MASK_XTILE_DATA
> #define XFEATURE_MASK_USER_OPTIONAL \
>     (XFEATURE_MASK_DYNAMIC | XFEATURE_MASK_CET_KERNEL)
>
> where XFEATURE_MASK_USER_DYNAMIC is used for xfd-related tasks (including the ARCH_GET_XCOMP_SUPP arch_prctl) but everything else uses XFEATURE_MASK_USER_OPTIONAL.
>
> KVM would enable the feature by hand when allocating the guest fpstate. Disabled features would be cleared from EDX:EAX when calling XSAVE/XSAVEC/XSAVES.

OK, I'll move ahead in that direction.

>> But I suspect that would be too much work to implement in practice.  It
>> would be akin to a new lesser kind of dynamic xstate, one that didn't
>> interact with XFD and *NEVER* gets allocated in the core kernel
>> fpstates, even on demand.
>>
>> I want to hear more about who is going to use CET_S state under KVM in
>> practice.  I don't want to touch it if this is some kind of purely
>> academic exercise.  But it's also silly to hack some kind of temporary
>> solution into KVM that we'll rip out in a year when real supervisor
>> shadow stack support comes along.
>>
>> If it's actually necessary, we should probably just eat the 24 bytes in
>> the fpstates, flip the bit in IA32_XSS and move on.  There shouldn't be
>> any other meaningful impact to the core kernel.
>
> If that's good to you, why not.

Thanks to all of you for quickly helping out!

> Paolo
>