Received-SPF: pass (google.com: domain of linux-kernel+bounces-19088-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Mon, 8 Jan 2024 13:09:59 +0800
From: Chao Gao <chao.gao@intel.com>
To: Sean Christopherson <seanjc@google.com>
CC: <isaku.yamahata@intel.com>, <kvm@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>, <isaku.yamahata@gmail.com>, Paolo Bonzini
	<pbonzini@redhat.com>, <erdemaktas@google.com>, Sagi Shahar
	<sagis@google.com>, David Matlack <dmatlack@google.com>, Kai Huang
	<kai.huang@intel.com>, Zhi Wang <zhi.wang.linux@gmail.com>,
	<chen.bo@intel.com>, <hang.yuan@intel.com>, <tina.zhang@intel.com>
Subject: Re: [PATCH v17 092/116] KVM: TDX: Handle TDX PV HLT hypercall
Message-ID: <ZZuDp+Pl0BHKEfPt@chao-email>
References: <cover.1699368322.git.isaku.yamahata@intel.com>
 <7ca4b7af33646e3f5693472b4394ba0179b550e1.1699368322.git.isaku.yamahata@intel.com>
 <ZZiLKKobVcmvrPmb@google.com>
Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <ZZiLKKobVcmvrPmb@google.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?TEIxSlJvaTE5QjhYWVVaeDhCQWQ5VXcrS3RYdG92cjI4QXdYRnZPRUxCQ3dY?=
 =?utf-8?B?b0FQVnZPNHdqcldHbjQ5T1Q5Sjh6WlF5aWlWSFBZbmMwUmc2b3NpQWtSYkx1?=
 =?utf-8?B?VzNwMG9ZQ1BBck5kTXJNZzAweDV5dFBxMmhNWG1XNWF3MDFoZDBnV0V0MzM0?=
 =?utf-8?B?VUp4N1VadWtjQTQybE9YUXJma2NhMTVlUDJrWW5wWG56ZUhUT0Y2NEUxL0pl?=
 =?utf-8?B?V0pKMVlzbldTWWMrWlNvVHNVQUpRdWdPTHhRSURiOWEvSE80b0xqVHVYdVZ0?=
 =?utf-8?B?V0xUeTV3WnUxOWUyQldmcHRUTXVoL3hKTTdUeVVEQ3ppUzZQOE4rY0crYmpk?=
 =?utf-8?B?YXpEMEd1QnE4MnZCRnRTbExvT3Rvc2NzQTU0dldlekVRNjhnbFBZcEJKRnJH?=
 =?utf-8?B?bmJna3JoeEdXWGxJU3JKWHlWQWU5aTJzbjN5NTRIVVFQMUJRM2tBd05RNXJB?=
 =?utf-8?B?bWdFK0Fuc1VIZHhPQk9YRGQ5ZGI5b1RFdGZlTmhsMEZyaHhPZ0VKUVJ1bk1Q?=
 =?utf-8?B?M09YYVVIMW01UG9BSzcxSXp5VDQydUt1TGRGWlRxYUF3YXZmVkdrS3NSMWlo?=
 =?utf-8?B?T0Y0bW1hZzIzekFaV1BrV3VHWHVmay92ZXdWNWRzRHlQbTlPd2FyNUVnVGNp?=
 =?utf-8?B?dVBzK1c1a1B6L29KL3puelZYY1pxREk0Z0xxUFg1WWtkUjdtakJuWjNsQnE5?=
 =?utf-8?B?ZFZ4dG9XTDFSbVBqdTY3TVF1bkNrR0ZrWUovWTFwQzQvSU5sWHNPUDRGd3pM?=
 =?utf-8?B?VTNhTXUrUnV1bU9XZ1JiWkVZZnVwMlJJZVdWWEJwcUgvcEdvVzNtMituNHNo?=
 =?utf-8?B?L0FKdzArejZKd2tHbTFUT2tiaDQvNndjVnUrQTZjTit1ajNiY0NDTjNhSHR3?=
 =?utf-8?B?TFhzU1djbXVYZUhQQ2lPb0RrbW8xaHBOL2tHODMwVDZEUkw5RDA0azlpWlI5?=
 =?utf-8?B?OGFEdnRQazZkZGhNdUVTckZOWVZPRVBlUFZPOEoxUEgyblZ6aTJudmJibzhj?=
 =?utf-8?B?U3MzYytGNGFHVzA2Zk1JTDBMaFN4azJuUWtRWWRxbkVQZzl2YWZWQUY0WWQr?=
 =?utf-8?B?NFNWWFk5d0hEOFZXN2xYOHlUbmMwRkRSSmN3NmJxdHdQakt0aHhDV1pnR0ZI?=
 =?utf-8?B?VHZoRmQyQml2L1JEcFVpSldVUDQ1aFBxS1ErbmJtRjhvaXgzQk1xQUZSWnpZ?=
 =?utf-8?B?YTJ4VTRTeEIwYk1SNC9ncFNseTRSYlZRaG5NUW9nWDFEZ1hCam9aT2dxMlJp?=
 =?utf-8?B?TGovM3pRN0g3TVIwRFc2SEdNNHZzQ0Y1bzI0YTVsVktjaW5nNktFTGY5RFJs?=
 =?utf-8?B?R2JwRE14V2JXT1ZPSWJXWVk5TVdXMnpoaVppMGZVUFA5RmJpTGNIT3cyaTNu?=
 =?utf-8?B?c0FEdGRkUllkSXRNaWZrS1BONFViWDljKzkwVHhZNHdQV1gzVEFrY2lzRXpl?=
 =?utf-8?B?dFlNT3RtRzRlSnc5amU4ZDZtV3hDcXYrNWQ5Y0RtYkJacXpxeTFmM3o4bVBy?=
 =?utf-8?B?UVhWRU9Fd0hIYjhGN1FuTFA2Tm1xWXFjN2pUOFFzNlFpSDFobS84akh6cVVV?=
 =?utf-8?B?cGZzZEJDOEdOcmtCR3pRZW5QWWNpeHVvN0VRZjZESUpoQWppVU11dFp1Y3hS?=
 =?utf-8?B?anNJSnFOQ2hXSmxSK0duSXJoWjVNN2lydVcyeVBxUE5RaGtBazU4UXMvaUhs?=
 =?utf-8?B?SG91alVPZGEzd2lxalFPQlN6TkM2K1NsUFdjbVdGTFpNdFVrb0dySGxicmN5?=
 =?utf-8?B?REQ5MDhUeWwxbXRydElSRnlzaEdzVllWK0JQRHJvTVRKOUVpcFdqZEUxTlBC?=
 =?utf-8?B?bWJwM2Exa285Snp2RjRvSFJEQ0MybHR1bVdyZklYSGpzWm1GMHhLNTd1NTYx?=
 =?utf-8?B?Vm4yLzhsdFpJMElrd3dnTTdCRE9weHpNODdoRjFZR1hPWUV3U1RucHFmWEdZ?=
 =?utf-8?B?SjRyanFCcGtVaFVDalZIUFNHdlg4UDF5YStQY2s3UmxiNHdUK3N4Sm9kUTZs?=
 =?utf-8?B?bnBRVEl0bEIzREFqSWo4dlZjeUhSTXpvNjJvZUJObWFDMGNwOGtnZ1l4WFh6?=
 =?utf-8?B?eFd4dEgvZVVvWm94NWpCQkRJKzZJUzgyQ2gvL0FiblNJeFljNUt3SW9wVThr?=
 =?utf-8?Q?x9gBT8sqRwBWBOZz8JapRzyUg?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 7f8c1e62-a3d7-4328-684a-08dc100815e5
X-MS-Exchange-CrossTenant-AuthSource: CH3PR11MB8660.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Jan 2024 05:10:09.6507
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: NtgFCLBTlZLeCfhKM+w4qz81IrQaNyND6wMONgsOGkQQb+qg6+40A7Y5o+iyLGZNF7PptntTI8XqshbeV7wEnA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR11MB8403
X-OriginatorOrg: intel.com

On Fri, Jan 05, 2024 at 03:05:12PM -0800, Sean Christopherson wrote:
>On Tue, Nov 07, 2023, isaku.yamahata@intel.com wrote:
>> From: Isaku Yamahata <isaku.yamahata@intel.com>
>> 
>> Wire up TDX PV HLT hypercall to the KVM backend function.
>> 
>> Signed-off-by: Isaku Yamahata <isaku.yamahata@intel.com>
>> ---
>>  arch/x86/kvm/vmx/tdx.c | 42 +++++++++++++++++++++++++++++++++++++++++-
>>  arch/x86/kvm/vmx/tdx.h |  3 +++
>>  2 files changed, 44 insertions(+), 1 deletion(-)
>> 
>> diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c
>> index 3a1fe74b95c3..4e48989d364f 100644
>> --- a/arch/x86/kvm/vmx/tdx.c
>> +++ b/arch/x86/kvm/vmx/tdx.c
>> @@ -662,7 +662,32 @@ void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
>>  
>>  bool tdx_protected_apic_has_interrupt(struct kvm_vcpu *vcpu)
>>  {
>> -	return pi_has_pending_interrupt(vcpu);
>> +	bool ret = pi_has_pending_interrupt(vcpu);
>> +	struct vcpu_tdx *tdx = to_tdx(vcpu);
>> +
>> +	if (ret || vcpu->arch.mp_state != KVM_MP_STATE_HALTED)
>> +		return true;
>> +
>> +	if (tdx->interrupt_disabled_hlt)
>> +		return false;
>> +
>> +	/*
>> +	 * This is for the case where the virtual interrupt is recognized,
>> +	 * i.e. set in vmcs.RVI, between the STI and "HLT".  KVM doesn't have
>> +	 * access to RVI and the interrupt is no longer in the PID (because it
>> +	 * was "recognized".  It doesn't get delivered in the guest because the
>> +	 * TDCALL completes before interrupts are enabled.
>> +	 *
>> +	 * TDX modules sets RVI while in an STI interrupt shadow.
>> +	 * - TDExit(typically TDG.VP.VMCALL<HLT>) from the guest to TDX module.
>> +	 *   The interrupt shadow at this point is gone.
>> +	 * - It knows that there is an interrupt that can be delivered
>> +	 *   (RVI > PPR && EFLAGS.IF=1, the other conditions of 29.2.2 don't
>> +	 *    matter)
>> +	 * - It forwards the TDExit nevertheless, to a clueless hypervisor that
>> +	 *   has no way to glean either RVI or PPR.
>
>WTF.  Seriously, what in the absolute hell is going on.  I reported this internally
>four ***YEARS*** ago.  This is not some obscure theoretical edge case, this is core
>functionality and it's completely broken garbage.
>
>NAK.  Hard NAK.  Fix the TDX module, full stop.
>
>Even worse, TDX 1.5 apparently _already_ has the necessary logic for dealing with
>interrupts that are pending in RVI when handling NESTED VM-Enter.  Really!?!?!
>Y'all went and added nested virtualization support of some kind, but can't find
>the time to get the basics right?

We actually fixed the TDX module. See 11.9.5. Pending Virtual Interrupt
Delivery Indication in TDX module 1.5 spec [1]

  The host VMM can detect whether a virtual interrupt is pending delivery to a
  VCPU in the Virtual APIC page, using TDH.VP.RD to read the VCPU_STATE_DETAILS
  TDVPS field.
  
  The typical use case is when the guest TD VCPU indicates to the host VMM, using
  TDG.VP.VMCALL, that it has no work to do and can be halted. The guest TD is
  expected to pass an “interrupt blocked” flag. The guest TD is expected to set
  this flag to 0 if and only if RFLAGS.IF is 1 or the TDCALL instruction that
  invokes TDG.VP.VMCALL immediately follows an STI instruction. If the “interrupt
  blocked” flag is 0, the host VMM can determine whether to re-schedule the guest
  TD VCPU based on VCPU_STATE_DETAILS.

Isaku, this patch didn't read VCPU_STATE_DETAILS. Maybe you missed something
during rebase? Regarding buggy_hlt_workaround, do you aim to avoid reading
VCPU_STATE_DETAILS as much as possible (because reading it via SEAMCALL is
costly, ~3-4K cycles)? if so, please make it clear in the changelog/comment.

[1]: https://cdrdv2.intel.com/v1/dl/getContent/733575

>