Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <ffd84a94-abea-a813-a6da-59d45a5ac468@intel.com>
Date:   Mon, 10 Jul 2023 08:28:46 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.13.0
From:   "Yang, Weijiang" <weijiang.yang@intel.com>
Subject: Re: [PATCH v3 00/21] Enable CET Virtualization
To:     Sean Christopherson <seanjc@google.com>
CC:     <pbonzini@redhat.com>, <kvm@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>, <peterz@infradead.org>,
        <rppt@kernel.org>, <binbin.wu@linux.intel.com>,
        <rick.p.edgecombe@intel.com>, <john.allen@amd.com>,
        "Neiger, Gil" <gil.neiger@intel.com>
References: <20230511040857.6094-1-weijiang.yang@intel.com>
 <ZIufL7p/ZvxjXwK5@google.com>
 <147246fc-79a2-3bb5-f51f-93dfc1cffcc0@intel.com>
 <ZIyiWr4sR+MqwmAo@google.com>
Content-Language: en-US
In-Reply-To: <ZIyiWr4sR+MqwmAo@google.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NnlDcmdSbjZmQ2ovN0ZBS1hYT3dmbXAxcm9UQ2ZnUzE3ekYvcE9nV0xnVHRv?=
 =?utf-8?B?ZjR5citmM2pROTFMb0dVbGQvSS9MeVJxc0tUMmhaSEpOazZsaDR1MW44Nzht?=
 =?utf-8?B?THNXSnp4MXZKaC9vUlp3eVlLWk5rV1UzWjgxTi9CRE5DcmcyRzBnNWRxdzV4?=
 =?utf-8?B?QXBtL1NCQ2wvOWlscFllbTQyVkRKQlJFa2lINVdrdTdTd1AxRTZHbW1VRmgw?=
 =?utf-8?B?bHc3c2RjdTN5VTR0bU13dE5OeXM3N3FrSmxFZS9pTnFpNDExWHdGMEk0bHBY?=
 =?utf-8?B?KzBJZzFSYVZQejJQeDNJSFIveHE3Y3F2dmVMbjl0RzZLWkpVaTVsVFEwWXFH?=
 =?utf-8?B?M1dyd0pwRVIxdGhHUFZHcnRlK1Y2R3o1SmhsaHVaTi9pQTExNlhqc2plWTZl?=
 =?utf-8?B?Z0p0cVFXdHRYamdkcUZteE9RdmlLMytDT0hKVmVwSlVjV0NSWXlIb1JWTmhO?=
 =?utf-8?B?QkVqKzZyUVVsRFc4ZE1UQlJ0NFFBc0JjOW96L2dkbXNhNkhVUzVBTXZwM3Ja?=
 =?utf-8?B?eDRvNDVOb0VWa3BzTmI1Rit3VVhQd2o3TXJUdzhXdEh6MWd5V1NhdlRqRlp5?=
 =?utf-8?B?Qmpqakxha1lkVHhrYncxZkRkWG9SR2h3ekdoMnc5VWUwWVBqd0RxNC9kL1Uv?=
 =?utf-8?B?eVlXazAzMmZBOU5WaGZqamxhNUpzcWUwTWZUUVNoSHRLdnB0bkgyR0ZPd2Zr?=
 =?utf-8?B?WnQrMzE2NU9OczlwNU0rYVhKYnZObmxzU3phbCtQOS9neHJJUkNvdngxS09a?=
 =?utf-8?B?RjdTRE83VmlSNmlsRmIyWmNNOUFnR1hnWDBhaVk1RTkwdFNpczUwZVdkUFlZ?=
 =?utf-8?B?RTZ5K283WEc3U1l5ZFZTblZKZjJOb3JPTGMvRW1GMFJzWE4xQnFSdUhlWmc4?=
 =?utf-8?B?Ry9XS2kvQURWd1BwK0RQdDlETnNOQ3FZc0E2ZGsyMW1QNHJIN3dUK0JGeUQy?=
 =?utf-8?B?ZEV0VkxZd3JvK1dyV21nQkpYYlpYQUp3SlQwbW9Gb0sxUVpqb2prUFRseXVO?=
 =?utf-8?B?WGdIMVlHUDVCU0x5c2E2UVRMK1JCREgxa0FhY2ZOSVI4elIvYWZYdjkwWWln?=
 =?utf-8?B?ZFY0eXVoU1kvRFl4Ym56N0hycy9VKzhlZlorNWNIZm1GdHE2UkVuZ2M1UGFP?=
 =?utf-8?B?NlBDcjdnZnUrdzZZVDBLNldNVHJrRTNHcE0wQ04zeVpYNk9KNElQSWtoUis3?=
 =?utf-8?B?OE1RT1U2MGxoaTI0bHE0NUVoWklqS3FMRTllTS82c3VRbzNlL0djUjJsQnBN?=
 =?utf-8?B?N2lRNkJQMmhhOEw0RUJqMjVGSUJNcElPTkhpTC9vZ1owSVRzWjBlTkoxMThT?=
 =?utf-8?B?L2o5dDdYcVFaaWlWV2x2NkduclFTSE9FYzluYWVBWkM1MjJwWVhBM2FzTitj?=
 =?utf-8?B?RmRGMmIzbFoySUhmZFdEV2lUUDFIMTZKYjF6ZEpZb3E3eHNldjJFTk9oZk1h?=
 =?utf-8?B?ZnVLeWpWRCtveEJDSnBOY2ordWgrZCtFQmNrSURvN1lKUFROVVR2YU54UnIy?=
 =?utf-8?B?VjQrcnViL0JBaTdCT0psTGVpVjhoTk1mNGgvYVFMUXM3ZVRtejZqaTFnZWVV?=
 =?utf-8?B?cS9EOWZwRk43elptaGxRSW1qY1h0VDhMY2lZSXZNQzRvUG9YRHM4aTkxOEY5?=
 =?utf-8?B?NXhjVGQyWE1TMTNqQ0FYMDJNYldwQllGQkwzVGFqRFBpV0FscisxT0pUTUNa?=
 =?utf-8?B?dTNqZThvRzBzaGJiT244OUFnRlBxSHZScHMybkZBd0Mva25vd2VVRDFOV0E2?=
 =?utf-8?B?cmVYTDl0dlhjUEM0OXM2czVZaWlXTTNqVlRQbmNkanp0b3lPMGxYMFRKV2pX?=
 =?utf-8?B?VlBDUmhod2ZJSm14YWxWRkxDK2ljeWovamNWczBla3czdmYvbVJ2WDd2V21Z?=
 =?utf-8?B?L3p1a2ZhWTBpT1RIMDN0dDBxY2QxR3VpVmYxNm9GaU43Vjk1bkw2WVFEQUJZ?=
 =?utf-8?B?YW44Q3AxVkcyMXhkSUxSMnlUZXphTHRtbzBhaWV4czNDL3ZxN3NSTlJDY3dX?=
 =?utf-8?B?aUlmYXBOSENXdkEzR1gycFBuTkFVSHIzNUdBUTQvejRKd0VGU1VEcjd2dE1S?=
 =?utf-8?B?amtYMzJqQ2czcWM3Z05XMExVTUhqNGxpV3BMQ1ZWZlVxOThzNEZoMjUyZjhx?=
 =?utf-8?B?WlF1N1REeis5UnVRVTNCRjZPRjJZRitFUFVZc0gzcWZXSW5CWktFRkhBWUkw?=
 =?utf-8?B?MXc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 524e29a7-dd93-4cfd-8c82-08db80dca57c
X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB4965.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jul 2023 00:28:56.3855
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: JRM8V2QxVa5yqT8gVH6bn3hnMq9uAuCURHsKtarhmFaAsaK1EasexATkAA4XXAH8N00gQ2vxfwQUk2zhRQ/ErQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB6075
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,
        RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


> *sigh*
>
> I got filled in on the details offlist.
>
> 1) In the next version of this series, please rework it to reincorporate Supervisor
>     Shadow Stack support into the main series, i.e. pretend Intel's implemenation
>     isn't horribly flawed.  KVM can't guarantee that a VM-Exit won't occur, i.e.
>     can't advertise CET_SS, but I want the baseline support to be implemented,
>     otherwise the series as a whole is a big confusing mess with unanswered question
>     left, right, and center.  And more importantly, architecturally SSS exists if
>     X86_FEATURE_SHSTK is enumerated, i.e. the guest should be allowed to utilize
>     SSS if it so chooses, with the obvious caveat that there's a non-zero chance
>     the guest risks death by doing so.  Or if userspace can ensure no VM-Exit will
>     occur, which is difficult but feasible (ignoring #MC), e.g. by statically
>     partitioning memory, prefaulting all memory in guest firmware, and not dirty
>     logging SSS pages.  In such an extreme setup, userspace can enumerate CET_SSS
>     to the guest, and KVM should support that.
>   
> 2) Add the below patch to document exactly why KVM doesn't advertise CET_SSS.
>     While Intel is apparently ok with treating KVM developers like mushrooms, I
>     am not.
>
> ---
> From: Sean Christopherson<seanjc@google.com>
> Date: Fri, 16 Jun 2023 10:04:37 -0700
> Subject: [PATCH] KVM: x86: Explicitly document that KVM must not advertise
>   CET_SSS
>
> Explicitly call out that KVM must NOT advertise CET_SSS to userspace,
> i.e. must not tell userspace and thus the guest that it is safe for the
> guest to enable Supervisor Shadow Stacks (SSS).
>
> Intel's implementation of SSS is fatally flawed for virtualized
> environments, as despite wording in the SDM that suggests otherwise,
> Intel CPUs' handling of shadow stack switches are NOT fully atomic.  Only
> the check-and-update of the supervisor shadow stack token's busy bit is
> atomic.  Per the SDM:
>
>    If the far CALL or event delivery pushes a stack frame after the token
>    is acquired and any of the pushes causes a fault or VM exit, the
>    processor will revert to the old shadow stack and the busy bit in the
>    new shadow stack's token remains set.
>
> Or more bluntly, any fault or VM-Exit that occurs when pushing to the
> shadow stack after the busy bit is set is fatal to the kernel, i.e. to
> the guest in KVM's case.  The (guest) kernel can protect itself against
> faults, e.g. by ensuring that the shadow stack always has a valid mapping,
> but a guest kernel obviously has no control over, or even knowledge of,
> VM-Exits due to host activity.
>
> To help software determine when it is safe to use SSS, Intel defined
> CPUID.0x7.1.EDX bit (CET_SSS) and updated Intel CPUs to enumerate CET_SS,
> i.e. bare metal Intel CPUs advertise to software that it is safe to enable
> SSS.
>
>    If CPUID.(EAX=07H,ECX=1H):EDX[bit 18] is enumerated as 1, it is
>    sufficient for an operating system to ensure that none of the pushes can
>    cause a page fault.
>
> But CET_SS also comes with an major caveat that is kinda sorta documented
> in the SDM:
>
>    When emulating the CPUID instruction, a virtual-machine monitor should
>    return this bit as 0 if those pushes can cause VM exits.
>
> In other words, CET_SSS (bit 18) does NOT enumerate that the underlying
> CPU prevents VM-Exits, only that the environment in which the software is
> running will not generate VM-Exits.  I.e. CET_SSS is a stopgap to stem the
> bleeding and allow kernels to enable SSS, not an indication that the
> underlying CPU is immune to the VM-Exit problem.
>
> And unfortunately, KVM itself effectively has zero chance of ensuring that
> a shadow stack switch can't trigger a VM-Exit, e.g. KVM zaps *all* SPTEs
> when any memslot is deleted, enabling dirty logging write-protects SPTEs,
> etc.  A sufficiently motivated userspace can, at least in theory, provide
> a safe environment for SSS, e.g. by statically partitioning and
> prefaulting (in guest firmware) all memory, disabling PML, never
> write-protecting guest shadow stacks, etc.  But such a setup is far, far
> beyond typical KVM deployments.
>
> Note, AMD CPUs have a similar erratum, but AMD CPUs *DO* perform the full
> shadow stack switch atomically so long as the stack is mapped WB and does
> not cross a page boundary, i.e. a "normal" KVM setup and a well-behaved
> guest play nice with SSS without additional shenanigans.
>
> Signed-off-by: Sean Christopherson<seanjc@google.com>
> ---
>   arch/x86/kvm/cpuid.c | 10 +++++++++-
>   1 file changed, 9 insertions(+), 1 deletion(-)
>
> diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
> index 1e3ee96c879b..ecf4a68aaa08 100644
> --- a/arch/x86/kvm/cpuid.c
> +++ b/arch/x86/kvm/cpuid.c
> @@ -658,7 +658,15 @@ void kvm_set_cpu_caps(void)
>   	);
>   
>   	kvm_cpu_cap_init_kvm_defined(CPUID_7_1_EDX,
> -		F(AVX_VNNI_INT8) | F(AVX_NE_CONVERT) | F(PREFETCHITI)
> +		F(AVX_VNNI_INT8) | F(AVX_NE_CONVERT) | F(PREFETCHITI) |
> +
> +		/*
> +		 * Do NOT advertise CET_SSS, i.e. do not tell userspace and the
> +		 * guest that it is safe to use Supervisor Shadow Stacks under
> +		 * KVM when running on Intel CPUs.  KVM itself cannot guarantee
> +		 * that a VM-Exit won't occur during a shadow stack update.
> +		 */
> +		0 /* F(CET_SSS) */
>   	);
>   
>   	kvm_cpu_cap_mask(CPUID_D_1_EAX,
>
> base-commit: 9305c14847719870e9e08294034861360577ce08

Hi, Sean,

Gil reminded me SDM has been updated CET SSS related topics 
recently(June release):

======================================================================

Section 17.2.3 (Supervisor Shadow Stack Token) in Volume 1 of the SDM:
     If the far CALL or event delivery pushes a stack frame after the 
token is acquired and any of the pushes causes a
     fault or VM exit, the processor will revert to the old shadow stack 
and the busy bit in the new shadow stack's token
     remains set. The new shadow stack is said to be prematurely busy. 
Software should enable supervisor shadow
     stacks only if it is certain that this situation cannot occur. If 
CPUID.(EAX=07H,ECX=1H):EDX[bit 18] is enumerated
     as 1, it is sufficient for an operating system to ensure that none 
of the pushes can cause a page fault.

Volume 2A: CPUID.(EAX=07H,ECX=1H):EDX[bit 18] as follows:
     CET_SSS. If 1, indicates that an operating system can enable 
supervisor shadow stacks as long as
     it ensures that a supervisor shadow stack cannot become prematurely 
busy due to page faults (see Section
     17.2.3 of the Intel® 64 and IA-32 Architectures Software 
Developer’s Manual, Volume 1). When
     emulating the CPUID instruction, a virtual-machine monitor (VMM) 
should return this bit as 1 only if it
     ensures that VM exits cannot cause a guest supervisor shadow stack 
to appear to be prematurely busy.
     Such a VMM could set the “prematurely busy shadow stack” VM-exit 
control and use the additional information
     that it provides.

Volume 3C: new “prematurely busy shadow stack” VM-exit control.

========================================================================

And Gil told me additional information was planed to be released later 
in the summer.

Maybe you need modify above changelog a bit per the update.

Given the updated parts are technical forecast, I don't plan to 
implement it in this series and still enumerate

CET_SSS ==0 for guest. What's your thoughts?