Received-SPF: pass (google.com: domain of linux-kernel+bounces-149381-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <f1097d63-2266-4e2d-996a-a387f497b4c7@intel.com>
Date: Thu, 18 Apr 2024 13:09:31 +1200
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v19 087/130] KVM: TDX: handle vcpu migration over logical
 processor
To: Isaku Yamahata <isaku.yamahata@intel.com>
CC: Sean Christopherson <seanjc@google.com>, "Chatre, Reinette"
	<reinette.chatre@intel.com>, "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"isaku.yamahata@gmail.com" <isaku.yamahata@gmail.com>, Paolo Bonzini
	<pbonzini@redhat.com>, "Aktas, Erdem" <erdemaktas@google.com>, Sagi Shahar
	<sagis@google.com>, "Chen, Bo2" <chen.bo@intel.com>, "Yuan, Hang"
	<hang.yuan@intel.com>, "Zhang, Tina" <tina.zhang@intel.com>,
	"isaku.yamahata@linux.intel.com" <isaku.yamahata@linux.intel.com>
References: <cover.1708933498.git.isaku.yamahata@intel.com>
 <b9fe57ceeaabe650f0aecb21db56ef2b1456dcfe.1708933498.git.isaku.yamahata@intel.com>
 <0c3efffa-8dd5-4231-8e90-e0241f058a20@intel.com>
 <20240412214201.GO3039520@ls.amr.corp.intel.com>
 <Zhm5rYA8eSWIUi36@google.com>
 <20240413004031.GQ3039520@ls.amr.corp.intel.com>
 <Zh0wGQ_FfPRENgb0@google.com>
 <20240415224828.GS3039520@ls.amr.corp.intel.com>
 <a552a48d-81b6-441a-88cf-63301f6968a2@intel.com>
 <20240416164432.GZ3039520@ls.amr.corp.intel.com>
Content-Language: en-US
From: "Huang, Kai" <kai.huang@intel.com>
In-Reply-To: <20240416164432.GZ3039520@ls.amr.corp.intel.com>
Content-Type: text/plain; charset="UTF-8"; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bjhTWlVESkZodURWRnJNaTI2U2JSZWptM3FPbVNMM1pvZHEvVjJlRjQ5WFRF?=
 =?utf-8?B?YTdRb29CanBsNXRBbXFVYU80WWtWV2llYURQZFNJT3B4RXZFU2x0Rnk1SElH?=
 =?utf-8?B?UUYxNmRIYXczT3BwQ3pWSVkrc003dlhVaG95aXo2WVBGRm9RNGZnN1lvSFdv?=
 =?utf-8?B?RFM1WjQ2Ly81VVE1NDFDbzJ4QnhzMEQ2VTNMQmpYdjNkYXA3UnYvam1XS2JJ?=
 =?utf-8?B?aUQwQnNYa09EQzZzRGVNbWF6YjQ4REhCLy9oM3BYdXJrN25xcUJEaWVMc2l2?=
 =?utf-8?B?S0l2U2ZXeTY4NWFBUm5LdVRCZUtBMUV1dkhYVStFcU5RT1ErTzh1TFdxZU4x?=
 =?utf-8?B?cXFwUThYK2Zzc2tDZzZCNlFXWmZKQ0VxMm5TbGErNFhjb0d6NGh5K09jaHFk?=
 =?utf-8?B?aEN0aHhjRUdVRkthQmpVZDZqTnRLU2NCaHoyOTZpcHpaNmYvTzI3MnUvSXJh?=
 =?utf-8?B?dzJib0djb0piVkdNQ2E3UWMwaGRyc1poSUkybEl0WjU5aUZWUUtNMHFxR1VZ?=
 =?utf-8?B?OGZVTkttMVdOR1lxUUxEbzd2a3lNSGdieTQ1WG1RdTZ6YTNQNEhuait6VGZt?=
 =?utf-8?B?QnlzUUtiK3RkRjlOdXZwdFk5WkZqV2pTRGVIQVpISklrdE1tVEo4ekFzZ0w4?=
 =?utf-8?B?U1F0WlliWHNteE5XUEZLbDdTUXluSEkrTEpxaGVQanlQbHJNelRoR3JmcVVv?=
 =?utf-8?B?QnJ5UkFtWEhObUxRZkNCR2lmaFI4QkZlQnVrQkdmZkgxUWhsTWowQUZkTi9P?=
 =?utf-8?B?eVBONjhPK25vandBWU15S0NYMXVEN2lBODVTNk93b1llYWt6aTNuUi9VZEdj?=
 =?utf-8?B?SFFoV09LZlNQL2hsZG5MVXVnbVA4MHpLaUJ2S2RpcEdYZ2tPU3orbnRid3Q3?=
 =?utf-8?B?V0hGWTZBY29FQXhLSFJzcjV0NDJvN1o2MDBYcGhyaVFFSk5FQWRBTXBEeXVN?=
 =?utf-8?B?ZnhSbGhTam5Kb3RnSFZiS0FlUWFTVkhESU54cUpnZnhlRlJRa24rWHg4dTdo?=
 =?utf-8?B?ZllwUTk4djE0c2h4R3J0ck94QkZVYUJMS21hdm95dmdrQlNGczJGZEtpeVZs?=
 =?utf-8?B?ZlB6blJTZEJHMVZwS3FYVHdYQXhpWlJhUFBLT09lWU9qKytTRXR0emNERzFR?=
 =?utf-8?B?NkdGYWNXVXN4dGp3UklMeHJDd3ZJRmdMWmFrRTRKZWxNcXUyN0JLTDVtamhx?=
 =?utf-8?B?RTB6L0diL3NMUUhvS0pFYWxZZCtKb3JMRXRkYUJWWUdBMEdjckd0SUI2dnJT?=
 =?utf-8?B?dmtnZ1owZGw4ZFFJcmVwNkVWaTMwN1JOMDVPZ3NtUEVaLzhLZFNRbEV5VU9I?=
 =?utf-8?B?UGYvUEl4VXdHSWJ2VXBqRVpvOTZqNzQ4UXR0aXN4YWV0c0dYcXQ1ZlRDWEF2?=
 =?utf-8?B?cGRQTTNrWnZLM3hRUnZ6K0Q0VktOdWJ4b1p0L0dwM3lUM2dXdnlwVEZoV2p3?=
 =?utf-8?B?N2QwaElJWGhQTmFVM2Y4WGdVc0JsOHdHSlYrejd5d3VQd0l3eEVueFcvNE9Y?=
 =?utf-8?B?cG1ucVFlbU8rZXZ4SXd1a2hBdXhIQkxybEQxb1I5NkFvWmVTOENwV0dFVmM3?=
 =?utf-8?B?ZXE3M0FHWEpVcmNBMW91WUdyRW9tUTg5ejI2UGVoUERTL0ttTXBtSTVKU3Zx?=
 =?utf-8?B?WFFjcHRmVkFINTFWYzdoNEpiVUJTZzhicFZ3eFpOUmRkMWpkSUZDRlByd1gz?=
 =?utf-8?B?ZTJaVG1WN0NzNzFFUVFPb1E0aUZKUHh2UzlFZkdWYmhQcWQxcWR2amovYTBY?=
 =?utf-8?B?dnEvaENuRmhwWEJrbmJ1bXNDRFVUWXd2RDhkUmEwZDd6Zkl5YTM3Z0RFN04x?=
 =?utf-8?B?OXZ4VmxVeDcrTVZrbVFYS0Y2T04ySGF0eE55cERvMWVEZlVyUXB1OTQwRzNX?=
 =?utf-8?B?YUI4Z0FpVEVXNWpoeTN1RUNpbzBiWlErS3lFNExZMGh6VWFFVmpNczhYajdQ?=
 =?utf-8?B?cTFsZFllRDBlZ3lEYjVEUnQxRWlGTEk3dm5qZmRaNzlSWktlU2NwTFYwS3hh?=
 =?utf-8?B?a0FrZXVJODFNVjR5ZlBJWTFxeU9FSW95eXJyVGE3Y2ZQa3FyR2tLakFLaGt3?=
 =?utf-8?B?bjJzZ0RaQ1lWSnVnaXJoVGNMaTZadnV2cGUzaWdUMGlMU2tmQmh3cVRsVFh2?=
 =?utf-8?Q?s7Odcjzr7krrJezeFZ2MVwZsM?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 205d5ba5-6d6e-4dde-4ff5-08dc5f443a83
X-MS-Exchange-CrossTenant-AuthSource: BL1PR11MB5978.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 Apr 2024 01:09:42.5868
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 1EDelQdHvMhC9k+447idzrjGQsme+RemwIIJXfK3rRcP59LpJmwvpHcTM1TPl4hqUAHJdcld5R2h3unrXTxL6w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR11MB5778
X-OriginatorOrg: intel.com


On 17/04/2024 4:44 am, Isaku Yamahata wrote:
> On Tue, Apr 16, 2024 at 12:05:31PM +1200,
> "Huang, Kai" <kai.huang@intel.com> wrote:
> 
>>
>>
>> On 16/04/2024 10:48 am, Yamahata, Isaku wrote:
>>> On Mon, Apr 15, 2024 at 06:49:35AM -0700,
>>> Sean Christopherson <seanjc@google.com> wrote:
>>>
>>>> On Fri, Apr 12, 2024, Isaku Yamahata wrote:
>>>>> On Fri, Apr 12, 2024 at 03:46:05PM -0700,
>>>>> Sean Christopherson <seanjc@google.com> wrote:
>>>>>
>>>>>> On Fri, Apr 12, 2024, Isaku Yamahata wrote:
>>>>>>> On Fri, Apr 12, 2024 at 09:15:29AM -0700, Reinette Chatre <reinette.chatre@intel.com> wrote:
>>>>>>>>> +void tdx_mmu_release_hkid(struct kvm *kvm)
>>>>>>>>> +{
>>>>>>>>> +	while (__tdx_mmu_release_hkid(kvm) == -EBUSY)
>>>>>>>>> +		;
>>>>>>>>>    }
>>>>>>>>
>>>>>>>> As I understand, __tdx_mmu_release_hkid() returns -EBUSY
>>>>>>>> after TDH.VP.FLUSH has been sent for every vCPU followed by
>>>>>>>> TDH.MNG.VPFLUSHDONE, which returns TDX_FLUSHVP_NOT_DONE.
>>>>>>>>
>>>>>>>> Considering earlier comment that a retry of TDH.VP.FLUSH is not
>>>>>>>> needed, why is this while() loop here that sends the
>>>>>>>> TDH.VP.FLUSH again to all vCPUs instead of just a loop within
>>>>>>>> __tdx_mmu_release_hkid() to _just_ resend TDH.MNG.VPFLUSHDONE?
>>>>>>>>
>>>>>>>> Could it be possible for a vCPU to appear during this time, thus
>>>>>>>> be missed in one TDH.VP.FLUSH cycle, to require a new cycle of
>>>>>>>> TDH.VP.FLUSH?
>>>>>>>
>>>>>>> Yes. There is a race between closing KVM vCPU fd and MMU notifier release hook.
>>>>>>> When KVM vCPU fd is closed, vCPU context can be loaded again.
>>>>>>
>>>>>> But why is _loading_ a vCPU context problematic?
>>>>>
>>>>> It's nothing problematic.  It becomes a bit harder to understand why
>>>>> tdx_mmu_release_hkid() issues IPI on each loop.  I think it's reasonable
>>>>> to make the normal path easy and to complicate/penalize the destruction path.
>>>>> Probably I should've added comment on the function.
>>>>
>>>> By "problematic", I meant, why can that result in a "missed in one TDH.VP.FLUSH
>>>> cycle"?  AFAICT, loading a vCPU shouldn't cause that vCPU to be associated from
>>>> the TDX module's perspective, and thus shouldn't trigger TDX_FLUSHVP_NOT_DONE.
>>>>
>>>> I.e. looping should be unnecessary, no?
>>>
>>> The loop is unnecessary with the current code.
>>>
>>> The possible future optimization is to reduce destruction time of Secure-EPT
>>> somehow.  One possible option is to release HKID while vCPUs are still alive and
>>> destruct Secure-EPT with multiple vCPU context.  Because that's future
>>> optimization, we can ignore it at this phase.
>>
>> I kinda lost here.
>>
>> I thought in the current v19 code, you have already implemented this
>> optimization?
>>
>> Or is this optimization totally different from what we discussed in an
>> earlier patch?
>>
>> https://lore.kernel.org/lkml/8feaba8f8ef249950b629f3a8300ddfb4fbcf11c.camel@intel.com/
> 
> That's only the first step.  We can optimize it further with multiple vCPUs
> context.

OK.  Let's put aside how important these optimizations are and whether 
they should be done in the initial TDX support, I think the right way to 
organize the patches is to bring functionality first and then put 
performance optimization later.

That can make both writing code and code review easier.

And more importantly, the "performance optimization" can be discussed 
_separately_.

For example, as mentioned in the link above, I think the optimization of 
"releasing HKID in the MMU notifier release to improve TD teardown 
latency" complicates things a lot, e.g., not only to the TD 
creation/teardown sequence, but also here -- w/o it we don't even need 
to consider the race between vCPU load and MMU notifier release:

https://lore.kernel.org/kvm/20240412214201.GO3039520@ls.amr.corp.intel.com/

So I think we should start with implementing these sequences in "normal 
way" first, and then do the optimization(s) later.

And to me the "normal way" for TD creation/destruction we can just do:

1) Use normal SEPT sequence to teardown private EPT page table;
2) Do VP.FLUSH when vCPU is destroyed
3) Do VPFLUSHDONE after all vCPUs are destroyed
4) release HKID at last stage of destroying VM.

For vCPU migration, you do VP.FLUSH on the old pCPU before you load the 
vCPU to the new pCPU, as shown in this patch.

Then you don't need to cover the silly code change around 
tdx_mmu_release_hkid() in this patch.

Am I missing anything?