Received-SPF: pass (google.com: domain of linux-kernel+bounces-157720-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Message-ID: <a76a430a-2a9d-4ff6-88c2-bdfafff92ae4@intel.com>
Date: Wed, 24 Apr 2024 14:07:28 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH V4 0/4] KVM: x86: Make bus clock frequency for vAPIC timer
 configurable
To: "Edgecombe, Rick P" <rick.p.edgecombe@intel.com>, "seanjc@google.com"
	<seanjc@google.com>
CC: "jmattson@google.com" <jmattson@google.com>, "Gao, Chao"
	<chao.gao@intel.com>, "linux-kernel@vger.kernel.org"
	<linux-kernel@vger.kernel.org>, "vkuznets@redhat.com" <vkuznets@redhat.com>,
	"Annapurve, Vishal" <vannapurve@google.com>, "Li, Xiaoyao"
	<xiaoyao.li@intel.com>, "Aktas, Erdem" <erdemaktas@google.com>,
	"pbonzini@redhat.com" <pbonzini@redhat.com>, "mlevitsk@redhat.com"
	<mlevitsk@redhat.com>, "Yamahata, Isaku" <isaku.yamahata@intel.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>, Yuan Yao <yuan.yao@intel.com>
References: <cover.1711035400.git.reinette.chatre@intel.com>
 <6fae9b07de98d7f56b903031be4490490042ff90.camel@intel.com>
 <Ziku9m_1hQhJgm_m@google.com>
 <26073e608fc450c6c0dcfe1f5cb1590f14c71e96.camel@intel.com>
Content-Language: en-US
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <26073e608fc450c6c0dcfe1f5cb1590f14c71e96.camel@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OWtqWXp1b0VGT1d0dENOY21ydzV5OG1IYldYZjBROEZraTBVNVlKY0pzSCtn?=
 =?utf-8?B?UmFjWGlRUk9WcncvQS82MVBBRTJCKzRJWDlGSVVVdy9lMHhwWUl6dTBtOWor?=
 =?utf-8?B?TWF5Rnl2YWFoeW5LNnR6WktpQzVYV3JJdkpjOGxlaWpQRFg2SFczYS9meDA3?=
 =?utf-8?B?bWsyMThuS2drTTdvS21rTERuWERzOURVMmpKTEx6OU9oRWpqTXJpWjVGc1N6?=
 =?utf-8?B?ZVlRRjlHSTcrcUs3eFhKYjkvaU1YM3d2VTAycVJ6R2tuc2FDVldMdDZoc0s2?=
 =?utf-8?B?amg5bW9rMGs2SEpKRDB6WXcvWWtBbnFyTW5qUnVRNzNncXRpcmd5cEUyZktt?=
 =?utf-8?B?QUlTSkFSaHpFSzY2c1pieThldXpSRlJyT09jbEM1eGpzNW0welYyYjJ6S0hW?=
 =?utf-8?B?d0NnbnBZUERVRTlXdGJvWU9OZmg0VS83VktBZEYxYjRtM21QcTZxcFBCalYy?=
 =?utf-8?B?UkhMYnF2akhVOGlraDY1ZW44Z2hidzZhWHN4bzdzdG1xWDVPMTRHMHpoZjdh?=
 =?utf-8?B?cForZ2ljeXFEY1pmNU8zMTRLYWFWK0ZsRC9ibXhacEluTTltSTZaU0JZNGI3?=
 =?utf-8?B?V0VPREVMNnQ5TUQzbGhPNHRhK3FHdWZubk81QXZVT2VsSFFocjhINTQzenNz?=
 =?utf-8?B?VGxCSzA4Y3IrcXlCeEpGOEFxMkVkR0c5RmRqRU9ralJQMHQ4eTNReTduUEhY?=
 =?utf-8?B?RXRuaS8yMWo0Sm9WdC9lR3JPendxYis4dHh0bGtJNVNnWW1wdVdldXBLVmdl?=
 =?utf-8?B?M2ZiR3pUMXJBdWFvUCszQi8zUzNlQi9TUlcwQVJ3R1MvMEpVdFlWUUN6U1BS?=
 =?utf-8?B?b2ZBOXdtTmt6akZCZ2xJQ1BKT1NuSVVoRDltc21LelROdm94Zy94RXgvckY4?=
 =?utf-8?B?UEc5YWNiQ1lFd2ZlNzhidDMzTWFVclRlMEpqdFFRM2YxRmk4d0wzYnVWQjNv?=
 =?utf-8?B?U0F4N1JBb3RwOEdDeE5yaWI3WVF4WXF2SlVSMEN6Skt6cnV0ZXBGN01HYXZH?=
 =?utf-8?B?UklKK09XY3NzaytJMkZJQWdLMFExYno2TTNZTEc1djdvRE9HamhxWUljeGJT?=
 =?utf-8?B?T3gzNllhazZhdzZSdlFEUmJHMGdpSEdrYzlWRHdWQlR1L2MxcUVibU9wSmZs?=
 =?utf-8?B?YjRsb3lURkJIWlY0aXhsaFJ4TnhIWFQ2MzZwRzk0WEJxWkZ0MFljUnF4Uzdw?=
 =?utf-8?B?b3gyd3U3aG8wcWdCRnBGVjcza2VPeTNBVXNHSTA4eFlPcmpyK3pUUUdJSnhZ?=
 =?utf-8?B?N2pjdmJrSm5oOThadjFENnRWOCtvZVVscjJQNXdnT0Y0ajhiZ2xaVjhhQS9p?=
 =?utf-8?B?N0crNzdsbk5rK1V1bkN5OFNtVTVUTzFtTDUyTUFvbkZpanp3bWhoaURtak5j?=
 =?utf-8?B?alBaeFF6NG5QWGRUL2ZINFFRUTVIcjR0VWVMc3o1cmNPUVFlWFJicXRQOTBO?=
 =?utf-8?B?QXUyUG9SVysrT3ZYWFVnbEZieDhuUlBxVitMUDhxUEFoWlM3UTlUWFNHTnl2?=
 =?utf-8?B?bkRpM2EzdUlSSjBRSkFLdW9YODY1SFA1RjhoY0pNQWJoanFsQlYzTTR1UTUx?=
 =?utf-8?B?MmlJVVExRW9oNURYMXg4Z2xjOE9rQTRBT2prajR1OWdTcUs0TUJKWUpNa013?=
 =?utf-8?B?aTlIczNTOWZidWN0QitQeGUwMEI0dU81Q3hwS2FYcWxXRGtCSjJLMGFYYWc5?=
 =?utf-8?B?aFhRQ056WnA5TXNQZW5jbzRnUVd5RmJrM0VnSmpVRTk0STZva2pPdjZJcGdM?=
 =?utf-8?B?NEVUZTU4dGdraEN1SU81VllXcHoxYkhVanpyWndHVEh5MEFxWTZBdDZiZXlj?=
 =?utf-8?B?WXhFS01XM3g3K3RQZWdsUUR6UE5EYldBUFJ0bVpKRUljMk1TWlhlNGQ3WlFn?=
 =?utf-8?B?bnhGTTdWUlh1WVlFeDFBMFZDNjd6cWRXajkwVi9DQkNEU3NsblpmRWhNK1VI?=
 =?utf-8?B?SnhBd1NYSjN1Wi96dWpNRlhJcFlpSkx6NDhLOFdEeFZReWhoU3QyTklqcmFP?=
 =?utf-8?B?UVZvcnhWckI1MDZYdlRrZ3FBbDJneWltcGU1cW04S0FEQVQxNlRMei9Ja043?=
 =?utf-8?B?MHNxdlM2UkN1UzJ1bWhVTXNVU3pCb0p6T29HNmJzS2pNV1NjK1dSN3JkNDJI?=
 =?utf-8?B?R3h3c3VLWWd1TzBBZVNJK29EUFFVZTJHbXFvSHFiT0NuQmx5TmpvdElJU1dO?=
 =?utf-8?B?TEE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 539bc3f5-db01-44dc-d01e-08dc64a28e12
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Apr 2024 21:07:31.1945
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: GOkcnK8Y+4uY+24BiuwjMOGbAR1MSh+LgIV2NpRpO2klx4NbkEoqbMGZCRxblTr/s0klpSxWdr4xTELroH2+I3+IKUnZAXgWueXB/5paVpM=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR11MB8350
X-OriginatorOrg: intel.com

(+Yao Yuan)

Hi Sean and Rick,

On 4/24/2024 9:38 AM, Edgecombe, Rick P wrote:
> On Wed, 2024-04-24 at 09:13 -0700, Sean Christopherson wrote:
>> On Tue, Apr 16, 2024, Rick P Edgecombe wrote:
>>> On Thu, 2024-03-21 at 09:37 -0700, Reinette Chatre wrote:
>>>>
>>>> Summary
>>>> -------
>>>> Add KVM_CAP_X86_APIC_BUS_FREQUENCY capability to configure the APIC
>>>> bus clock frequency for APIC timer emulation.
>>>> Allow KVM_ENABLE_CAPABILITY(KVM_CAP_X86_APIC_BUS_FREQUENCY) to set the
>>>> frequency in nanoseconds. When using this capability, the user space
>>>> VMM should configure CPUID leaf 0x15 to advertise the frequency.
>>>
>>> Looks good to me and...
>>> Tested-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
>>>
>>> The only thing missing is actually integrating it into TDX qemu patches and
>>> testing the resulting TD. I think we are making a fair assumption that the
>>> problem should be resolved based on the analysis, but we have not actually
>>> tested that part. Is that right?
>>
>> Please tell me that Rick is wrong, and that this actually has been tested with
>> a TDX guest.  I don't care _who_ tested it, or with what VMM it has been
>> tested,
>> but _someone_ needs to verify that this actually fixes the TDX issue.
> 
> It is in the process of getting a TDX test developed (or rather updated).
> Agreed, it requires verification that it fixes the original TDX issue. That is
> why I raised it.
> 
> Reinette was working on this internally and some iterations were happening, but
> we are trying to work on the public list as much as possible per your earlier
> comments. So that is why she posted it.
> 
> There was at least some level of TDX integration in the past. I'm not sure what
> exactly was tested, but we are going to re-verify it with the latest everything.

Apologies for the delay. I am the one needing to do this testing and it took me a while
to ramp up on all the parts (and I am still learning).

I encountered quite the roadblock (for me) along the way that was caused by a lingering
timer (presumably left by TDVF). Thank you so much to Isaku and Yao Yuan for helping me
to root cause this. I believe that this is unique to the kvm-unit-tests that does
not reset the environment like the OS.

A modified x86/apic.c:test_apic_timer_one_shot() was used to test this feature. Below I
provide the diff of essential parts against
https://github.com/intel/kvm-unit-tests-tdx/blob/tdx/x86/apic.c for your reference. With
these modifications it can be confirmed that the test within a TD fails without the work
in this series, and passes with it. This was tested against a host kernel running a
snapshot of the ongoing KVM TDX work and corresponding QEMU changes (including a QEMU
change that enables the new capability introduced in this series).

Below are the core changes made to the existing APIC test. The two major changes are:
(a) stop any lingering timers before the test starts, (b) use CPUID 0x15 in TDX to
accurately determine the TSC and APIC frequencies instead of making 1GHz assumption
and use similar check as the kselftest test introduced in this series (I did have to
increase the amount with which the frequency is allowed to deviate by 1% in my testing).

Please note that there are some more changes needed to run this test in TDX since all
APIC tests are not appropriate for TDX. This snippet was used in my testing and I
will work with kvm-unit-test folks on the next steps to have it integrated.


@@ -477,11 +478,29 @@ static void lvtt_handler(isr_regs_t *regs)
 
 static void test_apic_timer_one_shot(void)
 {
-	uint64_t tsc1, tsc2;
 	static const uint32_t interval = 0x10000;
+	uint64_t measured_apic_freq, tsc2, tsc1;
+	uint32_t tsc_freq = 0, apic_freq = 0;
+	struct cpuid cpuid_tsc = {};
 
 #define APIC_LVT_TIMER_VECTOR    (0xee)
 
+	/*
+	 * CPUID 0x15 is not available in VMX, can use it to obtain
+	 * TSC and APIC frequency for accurate testing
+	 */
+	if (is_tdx_guest()) {
+		cpuid_tsc = raw_cpuid(0x15, 0);
+		tsc_freq = cpuid_tsc.c * cpuid_tsc.b / cpuid_tsc.a;
+		apic_freq = cpuid_tsc.c;
+	}
+	/*
+	   stop already fired local timer
+	   the test case can be negative failure if the timer fired
+	   after installed lvtt_handler but *before*
+	   write to TIMICT again.
+	 */
+	apic_write(APIC_TMICT, 0);
 	handle_irq(APIC_LVT_TIMER_VECTOR, lvtt_handler);
 
 	/* One shot mode */
@@ -503,8 +522,16 @@ static void test_apic_timer_one_shot(void)
 	 * cases, the following should satisfy on all modern
 	 * processors.
 	 */
-	report((lvtt_counter == 1) && (tsc2 - tsc1 >= interval),
-	       "APIC LVT timer one shot");
+	if (is_tdx_guest()) {
+		measured_apic_freq = interval * (tsc_freq / (tsc2 - tsc1));
+		report((lvtt_counter == 1) &&
+		       (measured_apic_freq < apic_freq * 102 / 100) &&
+		       (measured_apic_freq > apic_freq * 98 / 100),
+		       "APIC LVT timer one shot");
+	} else {
+		report((lvtt_counter == 1) && (tsc2 - tsc1 >= interval),
+		"APIC LVT timer one shot");
+	}
 }
 

Reinette