Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <1eb6048b-bf23-78a0-9c3c-54bbd12c3864@suse.com>
Date:   Mon, 12 Dec 2022 17:46:29 +0100
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.5.1
Subject: Re: [PATCH linux-next v2] x86/xen/time: prefer tsc as clocksource
 when it is invariant
Content-Language: en-US
To:     Krister Johansen <kjlx@templeofstupid.com>
Cc:     Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>, x86@kernel.org,
        "H. Peter Anvin" <hpa@zytor.com>, xen-devel@lists.xenproject.org,
        linux-kernel@vger.kernel.org,
        Marcelo Tosatti <mtosatti@redhat.com>,
        Anthony Liguori <aliguori@amazon.com>,
        David Reaver <me@davidreaver.com>,
        Brendan Gregg <brendan@intel.com>,
        Juergen Gross <jgross@suse.com>,
        Boris Ostrovsky <boris.ostrovsky@oracle.com>
References: <20221208163650.GA3225@templeofstupid.com>
 <1e6c1b08-d573-fba9-61fd-d40a74427d46@oracle.com>
 <20221212155730.GA1973@templeofstupid.com>
 <20221212160524.GB1973@templeofstupid.com>
From:   Jan Beulich <jbeulich@suse.com>
In-Reply-To: <20221212160524.GB1973@templeofstupid.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cWlMMlk0NjI3enRmQ2VIY3ozTFR3eXpRV2w0NWw5WVBtZk9MQlFaOTU5dmlz?=
 =?utf-8?B?VlVlZEVUWC9FSDIwNS9YVDRLSkl2SmNyV3FJWWd0cTFUcEpWZmhGRWswcGps?=
 =?utf-8?B?YXZWZDJVMGt2elJSSnNkamZ6VWUydXNWcytQYXp4VHNmRERoTnpjRUxEU0Qw?=
 =?utf-8?B?MUtEa0dUTDBzNWl6cnRSckUwY2taamJCNVozOFRab3VEVlgvZzFFNWtvaVB6?=
 =?utf-8?B?cXlMeGx2SHFYTG82bktZcE9EQVYzb1V6S3RrTmpnUWs1Rmh1bUdKakhCUk5F?=
 =?utf-8?B?VS9uN1liRzlRbHMvaGdMZ01jcFlQZG5tRUwxbDJwdDg5UjZDSkozZWI3K2NE?=
 =?utf-8?B?Q2tLUkhhUjF2M2dZSUFQSjdKUVQxV2cvSzM2eDhEb0l4NHhncGxZeFhibGtH?=
 =?utf-8?B?WnJJeTE1UU54R0h2OU5lcXBBV3VWLy9hekFFbDZJL2VzemJBNlBQSUVBQ1hC?=
 =?utf-8?B?OUJzZStvVE1rZ3NzR0Q3aUVqalZaZUZhTUhOdkFxb2YxWUdDWklKWVdjOUQ5?=
 =?utf-8?B?YXdQcWdnQTZhdDBnRXJWSVRjNlVIcjdFMUNhU1Erb21uMGQvTGZHMjVGN1lD?=
 =?utf-8?B?d3R2SUQ5dWRic1VRMGlUZEltRVpjWXJBWFZvcFR6N3pZcGlraHVQd0w1K0N6?=
 =?utf-8?B?N3cyU1lOZHdYK2p4YVp2bFVuMUtKZS9kMkNPKzk0MWV3MDFTSmZGNmk0TmRK?=
 =?utf-8?B?Q0JpNFdBUDBLUHNRd256aU5COXZkMW9CRVBnMnVvUi9iRC9PTWFpdHp6ZHNk?=
 =?utf-8?B?UmZKWVZCVERCa25BZHJhTkR0MDh6R3dNZnptS3Y4YmV2bThRY1RrVmRKNE83?=
 =?utf-8?B?YTNyT1NjQ1NYWUJBazI0ZlRYa1MrdThmM28xd3N6YzVtVnZzQm14NG1ya1Z4?=
 =?utf-8?B?Uk5ZcGVGOWJUNnU1VUR1aktuQlBvREVsMW9UUDJpbGR2dkNOUjdTSEZNc2JX?=
 =?utf-8?B?eHBOd3pjTmo2T2NJQ0thYW1oU3RIYnBXejlNTXhJajdXL3pZOEptQ0VRdzg1?=
 =?utf-8?B?MzdCNFVIMG9uWkR4eDF1NHNPTDA5aHJsOHhNQjh3bmxwdndkcFRXN2k1Q2RW?=
 =?utf-8?B?dnlSR3c1UDFUQTlEcHU5YlFtVlhybXhtMWV6emNZWHpGS093dWNzQ2dhU3dB?=
 =?utf-8?B?emJIYk82c3pOUTJSb29rVjlNVERhNVk5QkFsVk9LcDhzNHZjbmRMVDlWWmU5?=
 =?utf-8?B?b1p0cWR4Rk9zWi9qRjcrOHJ4aEw2Kys3MjN2M09ySVhSNWYvTVhyZWFQWDVQ?=
 =?utf-8?B?T0F2dmlOdE9rM3QybGlvVm5vZmU3ZEhiNlgwaUNPWlliNjFkVmw2cCs4Z295?=
 =?utf-8?B?NGZqbjZUMGxVM0JQWnZGbWR1MjlKWWxRYWVrclpkaU1MdUU1Y2FEYW93amIz?=
 =?utf-8?B?VDNGUUdLU0huY3pnWWtoNzlJL0prQ2FkcjAybXRrM2hRNndWNjVrWUFFYm9k?=
 =?utf-8?B?T1Aza1NNcGR1MnhLNjJVK3RmajIwYzFXSGlORVFnWXJJOFlzTzV1NVdyckUx?=
 =?utf-8?B?aUFwRm1IRXg1VFpGUXEwM3pCTEFtU0xGSVZ4NDB5L2UwQnBoUnpBRjBUK0VH?=
 =?utf-8?B?My83cGJsSmM3THdsMFVqaS9BZnVwQ2JIakdBdThBa0lBN210dGNLRE0wQnJv?=
 =?utf-8?B?eEQvaUdsU1pqQmVHMUVZcUl1YWF0bktqZ2tiM0NFWmZNZU5WRUJvUHpDTmFa?=
 =?utf-8?B?ODRNMmlCSDg5bWtDM3paVndjSG85RmlzSHZnL1NpbjN6ZlZTRU80V0Q0ajNE?=
 =?utf-8?B?RENmQTRVWjJZWmtUTVIyWnlWeHdQWkxjMGNpbmREekdzOWN1dThsc2FldzBq?=
 =?utf-8?B?V2VEOUZRSDVQdjhORnJUclRrQlp1ajNSNUhDemZpYWRDZHhLeCtQakNKbjFR?=
 =?utf-8?B?NEhORHprMVJlMGUvRm5nSkEwRjdxcEhUbEhidEg5ZzRNdHJaRDZDVjNaZFNY?=
 =?utf-8?B?YnQ2bkhnSXoxTzZRRFVSajNmb3VteG9qYXE4K2Z1VktKaitLeC9hcW9WYmNk?=
 =?utf-8?B?dEtXTzNmZW9aVk9sMWxxdFVFZWphZVlTclNwVnNwdjhzWS9DeFlWV1A5YVpV?=
 =?utf-8?B?RjF4T2N3a3crZlFsN0tCUzZDNTBWMjE3K21PMDJiRVZvOHQrUkJnejMrekhL?=
 =?utf-8?Q?hvopEQM4DL6M0ARYfnZVhWvVn?=
X-OriginatorOrg: suse.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f2694101-7397-4e03-59d2-08dadc606bc9
X-MS-Exchange-CrossTenant-AuthSource: VE1PR04MB6560.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Dec 2022 16:46:31.0769
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: f7a17af6-1c5c-4a36-aa8b-f5be247aa4ba
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 3EKwO1MJAXn/FvzmtOINtW5CN1Hwabtw8Gyhp1B69ydXTROCqcG91d+F3HjPl3PCj0O6eqjy+WqKzQ5h1wxOUQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR04MB6798
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On 12.12.2022 17:05, Krister Johansen wrote:
> Kvm elects to use tsc instead of kvm-clock when it can detect that the
> TSC is invariant.
> 
> (As of commit 7539b174aef4 ("x86: kvmguest: use TSC clocksource if
> invariant TSC is exposed")).
> 
> Notable cloud vendors[1] and performance engineers[2] recommend that Xen
> users preferentially select tsc over xen-clocksource due the performance
> penalty incurred by the latter.  These articles are persuasive and
> tailored to specific use cases.  In order to understand the tradeoffs
> around this choice more fully, this author had to reference the
> documented[3] complexities around the Xen configuration, as well as the
> kernel's clocksource selection algorithm.  Many users may not attempt
> this to correctly configure the right clock source in their guest.
> 
> The approach taken in the kvm-clock module spares users this confusion,
> where possible.
> 
> Both the Intel SDM[4] and the Xen tsc documentation explain that marking
> a tsc as invariant means that it should be considered stable by the OS
> and is elibile to be used as a wall clock source.  The Xen documentation
> further clarifies that this is only reliable on HVM and PVH because PV
> cannot intercept a cpuid instruction.

Without meaning to express a view on the argumentation as a whole, this
PV aspect is suspicious. Unless you open-code a use of the CPUID insn
in the kernel, all uses of CPUID are going to be processed by Xen by
virtue of the respective pvops hook. Documentation says what it says
for environments where this might not be the case.

> @@ -474,15 +475,55 @@ static void xen_setup_vsyscall_time_info(void)
>  	xen_clocksource.vdso_clock_mode = VDSO_CLOCKMODE_PVCLOCK;
>  }
>  
> +/*
> + * Check if it is possible to safely use the tsc as a clocksource.  This is only
> + * true if the domain is HVM or PVH, the hypervisor notifies the guest that its
> + * tsc is invariant, and the tsc instruction is not going to be emulated.
> + */
> +static int __init xen_tsc_safe_clocksource(void)
> +{
> +	u32 eax, ebx, ecx, edx;
> +
> +	if (!(xen_hvm_domain() || xen_pvh_domain()))
> +		return 0;
> +
> +	if (!(boot_cpu_has(X86_FEATURE_CONSTANT_TSC)))
> +		return 0;
> +
> +	if (!(boot_cpu_has(X86_FEATURE_NONSTOP_TSC)))
> +		return 0;
> +
> +	if (check_tsc_unstable())
> +		return 0;
> +
> +	cpuid(xen_cpuid_base() + 3, &eax, &ebx, &ecx, &edx);

Xen leaf 3 has sub-leaves, so I think you need to set ecx to zero before
this call.

Jan