Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 1 Dec 2022 12:55:47 +0800
From:   Chao Gao <chao.gao@intel.com>
To:     Jon Kohler <jon@nutanix.com>
CC:     Sean Christopherson <seanjc@google.com>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        X86 ML <x86@kernel.org>, "H. Peter Anvin" <hpa@zytor.com>,
        "kvm @ vger . kernel . org" <kvm@vger.kernel.org>,
        LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] KVM: X86: set EXITING_GUEST_MODE as soon as vCPU exits
Message-ID: <Y4gz02jGZpNFbnAE@gao-cwp>
References: <20221129182226.82087-1-jon@nutanix.com>
 <Y4b4UURccshroVV9@gao-cwp>
 <7248AC8C-FB3A-48D0-8DB2-F96CB6C15C17@nutanix.com>
Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <7248AC8C-FB3A-48D0-8DB2-F96CB6C15C17@nutanix.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ek1IWmtqcFJkek9FcWJ4SWJSQ0JSVGt1cDh5b0NheUs4UXZOSVpubWFiSlIr?=
 =?utf-8?B?eUFwMDhrd3JEdTZUcHdZT0RkN0JMR0NTOTFxVDgrZk5CU21rZndRMVhUY21Q?=
 =?utf-8?B?eFprQlJkRlBXQlg0cGFPdE5CZU1LMTFsY09sMTlYVmtKMUh2UXVPb0JjRURK?=
 =?utf-8?B?eWFFWHVZbDZoVktMWGxEaFRZY1o1UUE0KzNMbFdZYnh0K3QxRy9wQnplZzlu?=
 =?utf-8?B?VHlKTVduZG9JTnJvNmIwSVZma0dpVnQvd05ORFBJcitLUklWb1o2S2F0dnpz?=
 =?utf-8?B?eU83T1lMZVkvMlNrcnU5QTlaZ2Y0b2NlNTh3cDF6MTZDckxhMkMwamg0K3VP?=
 =?utf-8?B?MVRJcnNuRTRHenlqSWhScnhEdjFjSkYrK0o3SFpvbVdSMVBsajF4RFVEVkcy?=
 =?utf-8?B?S2cyUGxDZFV6a3hwdi9QOFVxaGpNaE9vZDdJcUkzNFZrL0ptN1pRamxXVUZO?=
 =?utf-8?B?MTdqUU5jWFpyRVBRQnNsWEFJRVhFcjhaWFExdzZ1SzJTMUI3dmRtWldsTW0x?=
 =?utf-8?B?Mll4ZEltV0RGZytvK2ZLV2hqNlFuTENOZlQ5U2NNT2VxcFVDZ29xYWpvN3I5?=
 =?utf-8?B?ZTlPM25CZ2hkNkVEdVN0V3ZFaEQwTkp3TURMMzNVWW5JOUljaHpQT1k0c2V3?=
 =?utf-8?B?TDJrTkF3eTh5cFBKTnN1Zlg0V1RETituVXVrUE1XdnE4SmZRdkNPQXNkbXpV?=
 =?utf-8?B?UUhzU0lMckVlUlZpeEIrZUJUUGJ1Q2wzSXc0dGIrbWVSODVUYzdEZm9jL1Ba?=
 =?utf-8?B?S2U4Q0ppNzg4S0tQSHhLY0lYMXpsRUJ5WFZaWERQWUNpK2gvY21oTFdwUW51?=
 =?utf-8?B?eVNkVlZDOFEybVM1SGRDNUpVdk1IY0lGekVTSXlZdGFJMzdtQTFtZ3lxVXdX?=
 =?utf-8?B?UkNmTjhVTlk1WlhaWmZoMTZFMHhGRXRkK293TWJ0U0xHd2t2RThReFhKcEww?=
 =?utf-8?B?ZE1YWVBIU2ZYdGVLTkVyeEtyTUEwdjZVcDFVbVV6TEZhVXBqa1FSTG5XWEoy?=
 =?utf-8?B?OXljWTdmYW54RnZzMFMrY0VHeFpuZFFwUVpncG1oU0tycWxPcnNST2w3Tkk3?=
 =?utf-8?B?aFpZTjRYdVllbFJyUUhpWUMyNUhOKzVpazBjLzd2T1hZUEZ0TGUxajBPRU0r?=
 =?utf-8?B?bXVSeEV0d0Z3a0F2c1UxbG5Sbkhub2tyck9sR3RuQ3ZHZjg3TlZJNTVGMm1U?=
 =?utf-8?B?cVlXSDc4d21sOWgyWjhVNzJYWUNVdk00cytYQ0VDbEpEdSt1WHFoU0s2cVJK?=
 =?utf-8?B?Q1lLcnU3MkkxeTI4YS8wSXNXbGNGc2xHaWUwc2dIQS9jWnI2bllJTkdQTURE?=
 =?utf-8?B?d3h6OTJCMUdJN3hvd3BmNGZjYks5QmtiMEFTVFVwWFcrb0R4YyswRk1ON1pa?=
 =?utf-8?B?cWxSaGJDRGJjSFVwNzAvSk8rYVNSYnNkTytoUnNuRHpubE5hN3NIRUZiMjJ3?=
 =?utf-8?B?Nk52UmFmdThUNGNDVlpNZERheGllM2NkZ1EwLzZwTnJWRXpiN2VObHFSdU5W?=
 =?utf-8?B?WGJ1Y3UrSUtrejVtbzl4d2YrU2tpYys2b1cwMHpYbUtGbWMyOUdVcWtKdk1r?=
 =?utf-8?B?cmJMdU9RVHkyQjVCVDM5eDNWWjNvMjY5UDhaQitwVTFOM1pqSXBQUHJxRDN5?=
 =?utf-8?B?ZnRBTHNJK2k5VWlFejJGYmdXaENUcytINjBrc1Rhd0cyWklYQmk5TGRydm9o?=
 =?utf-8?B?cGF3Z1pXWXg5RDh3cXQzcFg4NEpBZ3BNLzh2ckVRVWlnTnJONWd1QU5ueDlt?=
 =?utf-8?B?anRUVm05c01xc2orcHNCdHJOMVZPa0JXcXNramtrcmhHV2pMTnFpZm5OL2xG?=
 =?utf-8?B?ZlgyZ21WQVVtZCtkQ1A1T3R6algwdFV0blRpWit5RDRUdHFRSlNvM0hHWU9J?=
 =?utf-8?B?ZTNnOUlmU3BsTUxSOXd6RE9PT04ydVREQi9NdjhYVTBjdld1cnhyaCsvTzdz?=
 =?utf-8?B?NjE4cDFjZjl2RmlKWXRTdCtBR2tOcDdqZlhkVkxvSXljWG94aEc1TXg5Mm1O?=
 =?utf-8?B?N2g4RnhjWlpVeC9idnFzcTNwbDBYSWZWK1V1dnFKWnIwVExVd24vU0dkeUdE?=
 =?utf-8?B?bGxXajY4QytBK3R5aFY3UjFSSjgvVjc2Z3NsLzk0dC9tQXN3ODdvNWdzSVVI?=
 =?utf-8?Q?e7spruH5mXut/Oszzz+1Q4IU/?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 27191b25-484e-404c-9fac-08dad358501c
X-MS-Exchange-CrossTenant-AuthSource: CY4PR1101MB2214.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Dec 2022 04:55:49.3481
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: AU2SrkqzICFdwKSrI+u98KRNn6c/YAnGakRK40itz4jbpw2PPGG8NSXsD0DjkaGtKXVdTFlvmK+LqGjUpounbA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR11MB5514
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_PASS,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Nov 30, 2022 at 02:07:57PM +0000, Jon Kohler wrote:
>
>
>> On Nov 30, 2022, at 1:29 AM, Chao Gao <chao.gao@intel.com> wrote:
>> 
>
>Chao while I’ve got you here, I was inspired to tune up the software side here based
>on the VTD suppress notifications change we had been talking about. Any chance
>we could get the v4 of that? Seemed like it was almost done, yea? Would love to 

I didn't post a new version because there is no feedback on v3. But
considering there is a mistake in v3, I will fix it and post v4.

>get our hands on that to help accelerate the VTD path.
>
>
>> On Tue, Nov 29, 2022 at 01:22:25PM -0500, Jon Kohler wrote:
>>> @@ -7031,6 +7042,18 @@ void noinstr vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp)
>>> void noinstr vmx_spec_ctrl_restore_host(struct vcpu_vmx *vmx,
>>> 					unsigned int flags)
>>> {
>>> +	struct kvm_vcpu *vcpu = &vmx->vcpu;
>>> +
>>> +	/* Optimize IPI reduction by setting mode immediately after vmexit
>>> +	 * without a memmory barrier as this as not paired anywhere. vcpu->mode
>>> +	 * is will be set to OUTSIDE_GUEST_MODE in x86 common code with a memory
>>> +	 * barrier, after the host is done fully restoring various host states.
>>> +	 * Since the rdmsr and wrmsr below are expensive, this must be done
>>> +	 * first, so that the IPI suppression window covers the time dealing
>>> +	 * with fixing up SPEC_CTRL.
>>> +	 */
>>> +	vcpu->mode = EXITING_GUEST_MODE;
>> 
>> Does this break kvm_vcpu_kick()? IIUC, kvm_vcpu_kick() does nothing if
>> vcpu->mode is already EXITING_GUEST_MODE, expecting the vCPU will exit
>> guest mode. But ...
>
>IIRC that’d only be a problem for fast path exits that reenter guest (like TSC Deadline)
>everything else *will* eventually exit out to kernel mode to pickup whatever other
>requests may be pending. In this sense, this patch is actually even better for kick
>because we will send incrementally less spurious kicks.

Yes. I agree.

>
>Even then, for fast path reentry exits, a guest is likely to exit all the way out eventually
>for something else soon enough, so worst case something gets a wee bit more delayed
>than it should. Small price to pay for clawing back cycles on the IPI send side I think.

Thanks for above clarification. On second thoughts, for fastpath, there is a
call of kvm_vcpu_exit_request() before re-entry. This call guarantees that
vCPUs will exit guest mode if any request pending. So, this change actually
won't lead to a delay in handling pending events.