Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Thomas Gleixner <tglx@linutronix.de>
To:     Muhammad Usama Anjum <usama.anjum@collabora.com>,
        Jonathan Corbet <corbet@lwn.net>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        "maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)" <x86@kernel.org>,
        "H. Peter Anvin" <hpa@zytor.com>,
        "open list:DOCUMENTATION" <linux-doc@vger.kernel.org>,
        open list <linux-kernel@vger.kernel.org>,
        "Guilherme G. Piccoli" <gpiccoli@igalia.com>
Cc:     Muhammad Usama Anjum <usama.anjum@collabora.com>,
        Steven Noonan <steven@uplinklabs.net>, kernel@collabora.com
Subject: Re: Direct rdtsc call side-effect
In-Reply-To: <87mt1jeax1.ffs@tglx>
References: <6719fb05-382c-8ec4-ccda-72798906a54b@collabora.com>
 <87mt1jeax1.ffs@tglx>
Date:   Thu, 01 Jun 2023 20:20:39 +0200
Message-ID: <87h6rrdoy0.ffs@tglx>
MIME-Version: 1.0
Content-Type: text/plain
Precedence: bulk

On Thu, Jun 01 2023 at 12:26, Thomas Gleixner wrote:
> On Thu, Jun 01 2023 at 13:45, Muhammad Usama Anjum wrote:
>> We are thinking of saving and restoring the timestamp counter at suspend
>> and resume time respectively.

I assume you talk about suspend to disk here, right? Suspend to RAM
definitely does not have the problem at least not on any halfways
contemporary CPU.

>>  In theory it can work on Intel because of
>> TSC_ADJUST register. But it'll never work on AMD until:
>> * AMD supports the same kind of adjust register. (AMD has said that the
>> adjust register cannot be implemented in their firmware. They'll have to
>> add it to their hardware.)
>> * by manual synchronization in kernel (I know you don't like this idea. But
>> there is something Windows is doing to save/restore and sync the TSC)
>
> Synchronizing TSC by writing the TSC MSR is fragile as hell. This has
> been tried so often and never reliably passed all synchronization tests
> on a wide range of systems.
>
> It kinda works on single socket, but not on larger systems.

Here is an example where it falls flat on its nose.

 One of the early Ryzen laptops had a broken BIOS which came up with
 unsynchronized TSCs. I tried to fix that up, but couldn't get it to sync
 on all CPUs because for some stupid reason the TSC write got
 arbritrarily delayed (assumably by SMI/SMM).

 After the vendor fixed the BIOS, I tried again and the problem
 persisted.

 So on such a machine the 'fixup time' mechanism would simply render an
 otherwise perfectly fine TSC unusable for timekeeping.

We asked both Intel and AMD to add TSC_ADJUST probably 15 years
ago. Intel added it with some HSW variants (IIRC) and since SKL all CPUs
have it. I don't know why AMD thought it's not required. That could have
spared a gazillion of bugzilla entries vs. the early Ryzen machines.

Thanks,

        tglx