Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 30 Nov 2022 14:29:37 +0800
From:   Chao Gao <chao.gao@intel.com>
To:     Jon Kohler <jon@nutanix.com>
CC:     Sean Christopherson <seanjc@google.com>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>, <x86@kernel.org>,
        "H. Peter Anvin" <hpa@zytor.com>, <kvm@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] KVM: X86: set EXITING_GUEST_MODE as soon as vCPU exits
Message-ID: <Y4b4UURccshroVV9@gao-cwp>
References: <20221129182226.82087-1-jon@nutanix.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20221129182226.82087-1-jon@nutanix.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?sWSH45gECvE0zjjVqXnQc7ZcBoAn0a4p5b666qmMmgA8AkIbr2ADBvOEur/U?=
 =?us-ascii?Q?8vUa595z0BAPd6wBmsJES3Q52P7ENyMyaJ1+OCkIHPLC+M4chlvby2CqdU0/?=
 =?us-ascii?Q?wHNEVW1T7nYPeC/FkmmK8nGgKP7JAwZ5iWXm6CuBR+erYuo/a1ks4AoPKZ4s?=
 =?us-ascii?Q?GSqXG/J3+Mj0m6sFSIPBiv7mEe8fksUWJh2z4V/ePxdFz9XZX6TOxh1eeGn8?=
 =?us-ascii?Q?Usnfv7eBB0KfUVrrVkahClLA2dF9hEWCdkBrWSj2+1KRXscXyzBz1L42ixF0?=
 =?us-ascii?Q?rw+miHwKT/T+umUdoof1f7zErVIsQy9pxDfOadaRxK+E2dpsHEbSrUVthVFO?=
 =?us-ascii?Q?o9UwyvpGHRUowBkaGsDFSWXaKtMoIK3zZC9qDwgTYECEXzLUEVw4N6hwVOcq?=
 =?us-ascii?Q?1W2F/xFuG/atVnwncip5hfe30H9uz9g6nrdxJDueMF7R7Drt4vrSvluWSD/5?=
 =?us-ascii?Q?3fHHIB8Y4gChKrLxrUgMOorPVLmjY3fEp3DSPrEx/YjS88t/SdIktpM0PJDG?=
 =?us-ascii?Q?HK0yR5foLO3qCMtBX8brgyYULq/xcpmikYRkigSjgijH0lvfiRRqhE9z/L5v?=
 =?us-ascii?Q?WBvbk9B47RaUEwsVJpnyx0n4RTMhxFBLe4RLSHeQWeRMUB69Dtg5/MsSSHKL?=
 =?us-ascii?Q?239lkrSgC4eqMKLaSTvIm3gz1t0Xwy947mPq4o4jGY6SiVdbk5jf6kLy+aLW?=
 =?us-ascii?Q?2cmHuh+VHV9HaDjJDmUE1btsEy7ftKiixVHO9V54900v2sPLY8WK1B1lR7JR?=
 =?us-ascii?Q?EQtZktAwTShJDsGm2ie4MvuXI3u1qibdJevWWVXX9CzRYQn/QBpwJ9C69Te1?=
 =?us-ascii?Q?KvUBciTvqSjJ7VIGIPlHijGhFbjLeEN3UQWOAKdmSSsmmfGpYRoAkR4Cg350?=
 =?us-ascii?Q?vxLDIJXSHIRAHsokr0mO6JsywJkhSVcXsXtEC0RgKM+u6JKtYq6wxobcuY9e?=
 =?us-ascii?Q?/M2vw7EKikFhtxgho5WO25IWz5xtv6lMS1b4nKr24WkdGnARfYs4P05yZlDV?=
 =?us-ascii?Q?VKJTkYkx0MPZlLQ4tWrUCEY9JEYkvY5eS0upGsA/v0HE1fZtuL6N9M9AipmL?=
 =?us-ascii?Q?T1EueRZGmNlrTJblTU0OzfGsA4NtzxCjJRg8tXeAKrdf/oUdHwvjQzOTRhBo?=
 =?us-ascii?Q?i08nN2cQgjZHOZvzgafitk8gqB29+T50a+M5AAPYzUf+Aj8mITF+Reshw4C8?=
 =?us-ascii?Q?7DGab5/fLYZLZsYYoyoHzSHOMn52Dh0PJYJTfoIoMlNhIKwcPcOQQpIwkNOK?=
 =?us-ascii?Q?06M1Wa0xIMGI+A49ROma80F9df1pC1Ji6NpaxkY+HRgYfzb/AGPZWlxVp4VC?=
 =?us-ascii?Q?N/cTF8vX3EJjUG8BddBSpa9JJq4jWk846NSdis4sK6/jIBZQ+MSlJGRbHyOd?=
 =?us-ascii?Q?EJ7TBgdaj5JwelhNNJh3ToQOBMXf2moBgsMYBDUArrwfhZBxpMjs7CWNzkS/?=
 =?us-ascii?Q?MIQTBg95SGcUcRt+btRr/6b/SLsRCOIRl3URBWw6Nvxzm3MEk9Qh0q61SuEz?=
 =?us-ascii?Q?ZQ5A+j2Iw+HOf2N7hbO1V9/g66mbxpGFonGUDa/DxvGaCBL3qPOwMUUN5lP6?=
 =?us-ascii?Q?cU79RVNn00OfIlAy36oYDQcx/oyXGwRvBSoWB6cG?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 2bc0a009-c191-48ac-2e48-08dad29c4108
X-MS-Exchange-CrossTenant-AuthSource: MWHPR1101MB2221.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Nov 2022 06:29:37.6945
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: EU/6qRzSDg4dqSitO5LE481hRYGOfJqjpPK2o21ZFyn0gRKM7BiRUtfff2KMkYWCaWRclzVv/HOpGRSaaKN5uw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB7432
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_PASS,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Nov 29, 2022 at 01:22:25PM -0500, Jon Kohler wrote:
>@@ -7031,6 +7042,18 @@ void noinstr vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp)
> void noinstr vmx_spec_ctrl_restore_host(struct vcpu_vmx *vmx,
> 					unsigned int flags)
> {
>+	struct kvm_vcpu *vcpu = &vmx->vcpu;
>+
>+	/* Optimize IPI reduction by setting mode immediately after vmexit
>+	 * without a memmory barrier as this as not paired anywhere. vcpu->mode
>+	 * is will be set to OUTSIDE_GUEST_MODE in x86 common code with a memory
>+	 * barrier, after the host is done fully restoring various host states.
>+	 * Since the rdmsr and wrmsr below are expensive, this must be done
>+	 * first, so that the IPI suppression window covers the time dealing
>+	 * with fixing up SPEC_CTRL.
>+	 */
>+	vcpu->mode = EXITING_GUEST_MODE;

Does this break kvm_vcpu_kick()? IIUC, kvm_vcpu_kick() does nothing if
vcpu->mode is already EXITING_GUEST_MODE, expecting the vCPU will exit
guest mode. But ...

>+
> 	u64 hostval = this_cpu_read(x86_spec_ctrl_current);
> 
> 	if (!cpu_feature_enabled(X86_FEATURE_MSR_SPEC_CTRL))
>diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
>index 2835bd796639..0e0d228f3fa5 100644
>--- a/arch/x86/kvm/x86.c
>+++ b/arch/x86/kvm/x86.c
>@@ -2160,6 +2160,14 @@ fastpath_t handle_fastpath_set_msr_irqoff(struct kvm_vcpu *vcpu)
> 		data = kvm_read_edx_eax(vcpu);
> 		if (!handle_fastpath_set_tscdeadline(vcpu, data)) {
> 			kvm_skip_emulated_instruction(vcpu);
>+			/* Reset IN_GUEST_MODE since we're going to reenter
>+			 * guest as part of this fast path. This is done as
>+			 * an optimization without a memory barrier since
>+			 * EXITING_GUEST_MODE is also set without a memory
>+			 * barrier.
>+			 */
>+			if (vcpu->mode == EXITING_GUEST_MODE)
>+				vcpu->mode = IN_GUEST_MODE;

... the vCPU enters guest mode again.

I believe mode transition from EXITING_GUEST_MODE to IN_GUEST_MODE
directly isn't valid for current KVM.