Received-SPF: pass (google.com: domain of linux-kernel+bounces-147643-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <ad651a0b-1c91-473f-b6a5-74c3c0de5e08@intel.com>
Date: Tue, 16 Apr 2024 14:28:31 -0700
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH V4 3/4] KVM: x86: Add a capability to configure bus
 frequency for APIC timer
To: "Edgecombe, Rick P" <rick.p.edgecombe@intel.com>, "jmattson@google.com"
	<jmattson@google.com>, "Gao, Chao" <chao.gao@intel.com>, "seanjc@google.com"
	<seanjc@google.com>, "vkuznets@redhat.com" <vkuznets@redhat.com>, "Annapurve,
 Vishal" <vannapurve@google.com>, "Li, Xiaoyao" <xiaoyao.li@intel.com>,
	"Aktas, Erdem" <erdemaktas@google.com>, "pbonzini@redhat.com"
	<pbonzini@redhat.com>, "mlevitsk@redhat.com" <mlevitsk@redhat.com>,
	"Yamahata, Isaku" <isaku.yamahata@intel.com>
CC: "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
References: <cover.1711035400.git.reinette.chatre@intel.com>
 <6146ef9f9e5a17a1940b0efb571c5143b0e9ef8f.1711035400.git.reinette.chatre@intel.com>
 <7b9651b233e43af66be47bd5a20297ca2d7c7e4a.camel@intel.com>
Content-Language: en-US
From: Reinette Chatre <reinette.chatre@intel.com>
In-Reply-To: <7b9651b233e43af66be47bd5a20297ca2d7c7e4a.camel@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?M3FPQkYxRE1KUFJuVjhxV09SRnIySXhFeXpDUjE3RElzZzJzUXNOTjA3OG03?=
 =?utf-8?B?L20zb1ZJMVBXZ2l3eHlWd2dFNXpFcXhsb1RDWEZmN2p5RGRUZklJdENEWGw2?=
 =?utf-8?B?STVuKzB2ckZGWlZzOVd6Y0kxUVdXYnFrUDArc1U0Zzk3WWFseStuZG1YSVNC?=
 =?utf-8?B?QkRDS1l5cHNYeU4vZFcvdnFEMkNKUFlUYmc0aWlWRE9MMzBscWVFVzYySjFk?=
 =?utf-8?B?UTRBYzV6M05KdWdGeks3OUhxSWZhcktZcTF5b1k5djJUZFRHcUpUM1NNdEFv?=
 =?utf-8?B?aTZNSlFVYjBZbStUa0U3a2ROSXl1bVhudWszcThHeXJEYmhDTGYzUTZKVXNq?=
 =?utf-8?B?cjN1TjVKZGJ0SDRFUkpiTWI0VlJtcVlqZGpMQm5WSHdvWVl0MW5Td1RkWFN1?=
 =?utf-8?B?MjFxUklSODhGQUdPRVlkODRuOUlKZVJ5RzlqUDdvSXNGUEh5K0NqZEYxTEpx?=
 =?utf-8?B?L2wvWWRHMG93Q29TR1lSMEdWS3UrWW8yWHVyV245Mk05VVVvbXpBbFJqdldF?=
 =?utf-8?B?RUorK1E2cHBldEtPSzNRdU90ZjNEdDNpcGR0alM4M1JIM001VnVBR2pzb05q?=
 =?utf-8?B?ZXBJS0dtblEvcEtlc2k1R2xSenZUSlgzM3VLSXJKVkVKVEZ1SUtCYUt2NEZI?=
 =?utf-8?B?R0hRZjBDU29Tb2RoNTRaYzVGbG9kenRXUjNEYm5OQ1dhWEl2a0poMjBYTDZH?=
 =?utf-8?B?SC9zRkNmS24zYXg3dXhtVFVNcXVzdDlnV1Q0djFEWkRySU03dU1YMk9yNnY4?=
 =?utf-8?B?UGNnQnFqOXdQdlRubGJZYVVRMDgxR3E3ejV5VEcxUmVQZUR0cVRnM1BwRG9C?=
 =?utf-8?B?VGxLNnZGblpLcXY4NGpCUklUZzlBVHVWNlRieEJFWVZNVmdJQk5yRVF5VlUz?=
 =?utf-8?B?dWNVVUp5cXdNdTRLYnljUmFzbVRDSWRHZUhNSklkdW9acWcrb0FtamMrdTFt?=
 =?utf-8?B?SVdMUHhxRnNRek1TZnJoREpJYnd1bS9KNE9NRkpGNy9jMHhWRTNtOGFoUk1J?=
 =?utf-8?B?ZXZmYWhyeDQreHVPQTJGb3Z3OHJmM0YyZmdYS0dNdjVaSlFvMk95Vk9ZMTdm?=
 =?utf-8?B?T1Q1bno3bVpjd0pBaWVJWXN3a1BQQVgxTEhHc3Zkc0cvQllYQzIzVWtDdlhw?=
 =?utf-8?B?V0NYWlVqZDluNlI5RWVWS05YTE1GVEVMYm1QZUIrY0xHZVYxQWU2N08zTkYr?=
 =?utf-8?B?Tk9WYVl4VVQvSDNhY2FrYi9DK3RESGZNaUFnajFvRTc4RVRRaDEraVlycGxP?=
 =?utf-8?B?S0huMmpBU0ZXK2xSZG1FOWpHRHdzTjBqb2I4L1U2RTJOR0hBR1JnQUJjVU8y?=
 =?utf-8?B?NDZnM2l0dFhNbEkvSXdVaUZVNk9tSWNCYUx2dkVMa2FTa1hxTkJwejhaT2FY?=
 =?utf-8?B?UXhSTyttTmhpdFAwUGRibVQ3clhhei96QWdpWCtTSHZMclJMQWdYMFBTbUxG?=
 =?utf-8?B?cWl3S1p4K2Mvc0ZwYTBJZHdIMlB5KzZOU2tWUmRzSldFRE5lWUovZVExZTV1?=
 =?utf-8?B?RTUxblY5MkZhZnZkM3c3Q1VXM1k0ZjdSN2lxWGxQZ0VodXVOa0JXNDdNelBU?=
 =?utf-8?B?M0NydnVobFU1TVhxNWs4dVgxbW5zMCtCWml4cUNseXphOS9EMHZEVWtjb1FG?=
 =?utf-8?B?S3M4amlQeWJwbUlFL3hlMncyTitYN09pN1BjUnZMSDNkZnFMT1JaaGhVTTM3?=
 =?utf-8?B?TXQzMGFIMWtoNm02L3gvRklrRzJsQXZQOGY0NSszS095VFQyeHF3alcyT01O?=
 =?utf-8?B?cmVVd2hPc0I5Yk9wdUpnelNuUzFNZk5LaS9WS1Qydm42Unltbk1lV0lzcktG?=
 =?utf-8?B?Nm42QkFjakRXUXB3bnNTcTRpZHBsWE5LZ29nNTBraWN6ci83Njl4U2JUeXVt?=
 =?utf-8?B?cVdwN1FZZndleEN6V1drRUQwZzdJOEhMT3A2RVVESVY5TmlCR1RtM3NnclA2?=
 =?utf-8?B?Q3Ywenl6Z0s4aWxuYjlrVXd1aTJhZitqeWg3bi9EU2xTSitMenZtSTlHZ0Zy?=
 =?utf-8?B?N01oUm1KMW1hc2xjanBYdkx3Szd6SkxUbUJwazZmWHd0dFR5QU83dEkxZUVw?=
 =?utf-8?B?Y1pEbnpXRndsbExPcHUvczMzOEQvU1BjNE5xalNNWkMyNGRSV1B5Y2tTTjBm?=
 =?utf-8?B?MkxCZ3pYSjJFL01zbXlWVlROMWtXYlJWNkh2RURpZnE3bSs5NlBzQ1VWWE1a?=
 =?utf-8?B?Zmc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: a33ba62c-3741-446c-1000-08dc5e5c2bc9
X-MS-Exchange-CrossTenant-AuthSource: SJ2PR11MB7573.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Apr 2024 21:28:34.6110
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: OShWFyxmpsJq1Eu0mg/B6tSG+nMl6YFaibq28XpZzcapyn+Ju6JTW2v0Fzdz6w47dOKPx1XdNx/8RXfE0AfhVPfPnxKHmvpfXlaI1N2tuDM=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB7246
X-OriginatorOrg: intel.com

Hi Rick,

On 4/16/2024 10:08 AM, Edgecombe, Rick P wrote:
> On Thu, 2024-03-21 at 09:37 -0700, Reinette Chatre wrote:
>> From: Isaku Yamahata <isaku.yamahata@intel.com>
>>
>> Add KVM_CAP_X86_APIC_BUS_FREQUENCY capability to configure the APIC
>> bus clock frequency for APIC timer emulation.
>> Allow KVM_ENABLE_CAPABILITY(KVM_CAP_X86_APIC_BUS_FREQUENCY) to set the
>> frequency in nanoseconds. When using this capability, the user space
>> VMM should configure CPUID leaf 0x15 to advertise the frequency.
>>
>> Vishal reported that the TDX guest kernel expects a 25MHz APIC bus
>> frequency but ends up getting interrupts at a significantly higher rate.
>>
>> The TDX architecture hard-codes the core crystal clock frequency to
>> 25MHz and mandates exposing it via CPUID leaf 0x15. The TDX architecture
>> does not allow the VMM to override the value.
>>
>> In addition, per Intel SDM:
>>     "The APIC timer frequency will be the processor’s bus clock or core
>>      crystal clock frequency (when TSC/core crystal clock ratio is
>>      enumerated in CPUID leaf 0x15) divided by the value specified in
>>      the divide configuration register."
>>
>> The resulting 25MHz APIC bus frequency conflicts with the KVM hardcoded
>> APIC bus frequency of 1GHz.
>>
>> The KVM doesn't enumerate CPUID leaf 0x15 to the guest unless the user
>> space VMM sets it using KVM_SET_CPUID. If the CPUID leaf 0x15 is
>> enumerated, the guest kernel uses it as the APIC bus frequency. If not,
>> the guest kernel measures the frequency based on other known timers like
>> the ACPI timer or the legacy PIT. As reported by Vishal the TDX guest
>> kernel expects a 25MHz timer frequency but gets timer interrupt more
>> frequently due to the 1GHz frequency used by KVM.
>>
>> To ensure that the guest doesn't have a conflicting view of the APIC bus
>> frequency, allow the userspace to tell KVM to use the same frequency that
>> TDX mandates instead of the default 1Ghz.
>>
>> There are several options to address this:
>> 1. Make the KVM able to configure APIC bus frequency (this series).
>>    Pro: It resembles the existing hardware.  The recent Intel CPUs
>>         adapts 25MHz.
>>    Con: Require the VMM to emulate the APIC timer at 25MHz.
>> 2. Make the TDX architecture enumerate CPUID leaf 0x15 to configurable
>>    frequency or not enumerate it.
>>    Pro: Any APIC bus frequency is allowed.
>>    Con: Deviates from TDX architecture.
>> 3. Make the TDX guest kernel use 1GHz when it's running on KVM.
>>    Con: The kernel ignores CPUID leaf 0x15.
>> 4. Change CPUID leaf 0x15 under TDX to report the crystal clock frequency
>>    as 1 GHz.
>>    Pro: This has been the virtual APIC frequency for KVM guests for 13
>>         years.
>>    Pro: This requires changing only one hard-coded constant in TDX.
>>    Con: It doesn't work with other VMMs as TDX isn't specific to KVM.
>>    Con: Core crystal clock frequency is also used to calculate TSC
>>         frequency.
>>    Con: If it is configured to value different from hardware, it will
>>         break the correctness of INTEL-PT Mini Time Count (MTC) packets
>>         in TDs.
>>
>> Reported-by: Vishal Annapurve <vannapurve@google.com>
>> Closes:
>> https://lore.kernel.org/lkml/20231006011255.4163884-1-vannapurve@google.com/
> 
> Is Closes appropriate, given the issue Vishal hit was on non-upstream code?

The issue that Vishal encountered was root-caused to the APIC bus frequency
conflict between KVM and TDX that has not changed since the original report.

It would be great if this submission can get confirmation that this series addresses
the issues encountered originally. From what I can tell there has not been
any formal accept/reject of this solution (apart from agreeing that it is the
right approach [1]) since the original report.

Reinette

[1] https://lore.kernel.org/lkml/CAGtprH8-jiC+wsy2LgmZimrRUT6kuntD6EJso2Mvx5Y42za9Dw@mail.gmail.com/