Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Sat, 3 Dec 2022 10:51:48 +0800
From:   Feng Tang <feng.tang@intel.com>
To:     "Paul E. McKenney" <paulmck@kernel.org>
CC:     Thomas Gleixner <tglx@linutronix.de>,
        <linux-kernel@vger.kernel.org>, <sboyd@kernel.org>,
        <corbet@lwn.net>, <Mark.Rutland@arm.com>, <maz@kernel.org>,
        <kernel-team@meta.com>, <ak@linux.intel.com>,
        <zhengjun.xing@intel.com>, Chris Mason <clm@meta.com>,
        John Stultz <jstultz@google.com>,
        Waiman Long <longman@redhat.com>
Subject: Re: [PATCH clocksource 1/3] clocksource: Reject bogus watchdog
 clocksource measurements
Message-ID: <Y4q5xObZ4JYIqZ6x@feng-clx>
References: <Y4bg1H/HLRLfucNO@feng-clx>
 <20221130051600.GL4001@paulmck-ThinkPad-P17-Gen-1>
 <Y4brjD/xUDlzJ4v7@feng-clx>
 <20221130055024.GM4001@paulmck-ThinkPad-P17-Gen-1>
 <Y4bxZCEAmPnALsRV@feng-clx>
 <20221201172405.GI4001@paulmck-ThinkPad-P17-Gen-1>
 <Y4lQdKaKh2P7eLBG@feng-clx>
 <20221202014428.GN4001@paulmck-ThinkPad-P17-Gen-1>
 <Y4lc04C6KqYUG5lf@feng-clx>
 <20221202222402.GI4001@paulmck-ThinkPad-P17-Gen-1>
Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20221202222402.GI4001@paulmck-ThinkPad-P17-Gen-1>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QWZOdzY5N2lBam4wMlBFZkExa1VrOVhmU1BpVCt2MGpNdndhUkVPMHgzb2I1?=
 =?utf-8?B?aWFCaHZOM0dHclBSeWlaUkhRRk1rZEVnbjB2T0gyT3VpcjBRZ3NIUldyaTJD?=
 =?utf-8?B?M0RVZTRBN2doSnVORGo4OEVjc0hraGFtTkI0cmN3M0dJcHBoRnk5dEtYT0Y5?=
 =?utf-8?B?ZkptaHkyYlJ0YkxmYzdQVnZ6UW1GVmVic3g4YzlZcmZpVkRQVkNoVHRMT05G?=
 =?utf-8?B?VDRjNkpqYVZVMkFuY0NxdEMwa1lQZTQyUllYd3B3dzJHbEltQXdFRkJaazhS?=
 =?utf-8?B?OVYrSTM4ZVVMUlFVVCtBc2RseHFHa3J0MzA0V3c4aHVoMm40dSttU0d6ekpM?=
 =?utf-8?B?eVdKTDZDVGNoZVdsUkFPMDY0TlRFWEtaWE5KbFZMRWJKSDRZM0JXTFg1dUFv?=
 =?utf-8?B?NGZtTXdwUCtzZVdMUGU1SkxlKzJWZjZ5SXp3SU5ERDV1N2xLa2lMbVpXVW9H?=
 =?utf-8?B?Ni9CdnU2bG1RNXJkdjhuZXZXQmd6cHUxYjVVNlhnajk4bDJsMmZDZDYwdG1S?=
 =?utf-8?B?aVRKYytnbU02YUJ2anV5KzQrVi9zZDBibWE1THNwdzVvWE1RSUhKTlUzRk5I?=
 =?utf-8?B?TC8vVjBnNVJVVGpiSUYzQ3RTZ0FuY1ZCS0VUK3pIMGJKckt5dXRPeWN4Wlgr?=
 =?utf-8?B?RkVBampvWE9lekVqQnpkeDc4c1FrMVJQQkY2MHhFNm5JM1lBdTcyeVYrcm1z?=
 =?utf-8?B?d3lWbXpEM3drSmkyUE1VcVFQRCtnZkhXQ21PTmowZEdSWjJqTm5nWnczQ0Nn?=
 =?utf-8?B?L285RWNwSWlMRmYxV2N1VjE5bHpnbUZZbm1Fc3hoSU1ZY2djaVAxdUZOeERQ?=
 =?utf-8?B?dXNRSkplS3ZIWWRPTFBKejdNR3pSKzFZeHRBbU1wYU9URnl4dWEzSDlrUVlj?=
 =?utf-8?B?dHFKQkM3cC90RDBFeWxFOHhEcnBvam11b3lQOGZFeHVqWXVWa003TDRNVlAy?=
 =?utf-8?B?TUVRNGRxWGpxQWZzNksxb0xDdjVwREVwWGJqTm9mUnhZYlFJMVNwVUdZNHJ2?=
 =?utf-8?B?eDROa0wzODZCSlRId0xibW9mRUxLa2psUVpXNXNyMzVGcUtXaXBYQkVJOTVv?=
 =?utf-8?B?K0Y2Ky9TTTZhcXQ1Z1dVakF3QzJBSEVrVFd2bDVzaVJpb3FKRWMxWGhYcEtT?=
 =?utf-8?B?bG9EcDlJOUtZcS9NaGtSZkZEUzRUWnVDZUVUKzhJV2V4ZnBtSFhjd3c3QllW?=
 =?utf-8?B?RHJUY2dvWEFPVlprSVpMUU8rbWxIZGVHMHB6Wk53SllVTjhuZUdmTzJ6SjJZ?=
 =?utf-8?B?UGhsUGdERXJUVjBxUnNLV0U0a3c5dTl3MmJyaVBWK2Zvc2QzU3QzMWk4aHNq?=
 =?utf-8?B?SnZiTW9UUGk0RmZha3ZhdXlPUGM1NG9xKyt6TXJuTnNNTnAwK3AzZlZoTERh?=
 =?utf-8?B?enFUWG9US3NNQTRmMXd6UmpjOGt2ckx6VU5PZS96QnFsNkF3Kzh1RXZwNGlC?=
 =?utf-8?B?UEIyOWxLeElTSHZYQW9GdU5jKzFlN2dDYk0xYXFhWUxJTWxyNElVWmx4NElW?=
 =?utf-8?B?cUVhdkJXN2lyWVlpbytzY3dvVzdpZE5zNW1tZnl6Q08vVVJZVEpVcjNJdUVr?=
 =?utf-8?B?N0ZyWUxWdXd3NFRSL2RBVDdXZ3RLNFM5dFhFSEFIWkNGZTJ0R0N2b2xHUmRx?=
 =?utf-8?B?c2ZWQytQTUY2akgxMy8rT0VPVm5KNWtzVFA0VlFodFlLVC80OUlFU0twWHpP?=
 =?utf-8?B?REZNS0dmdlBHbkxCQllKTXp0UUVQaVVqc3RZWHBVU3pyWUZuYjdXVEx0ZDFu?=
 =?utf-8?B?QkhOcGFIOWFjUEcvUWdJVklmbWRjcjcyNEtKR0hUYzVteERwaldmMjhjZzJR?=
 =?utf-8?B?ckJyVi9pR0p6QjNBbzB2K2lNVUEycjZObFgyQS8vMjJxNnM1N3l4VDJzVGlx?=
 =?utf-8?B?eDc0dkZYc2hWWmdLRWlEczRJc3RrS1lBQjZnNTJsZEhZYW1OYUUrNHFKd0Vu?=
 =?utf-8?B?QTBLajB5anNnS0Q3dzhkMm04UkhYS1k1QlRNWGkrOVYyTzc3aERURnlZV2Ji?=
 =?utf-8?B?aXpOSHYvNi9yNUVhZGhUTE1iS3RMZUY0L1VwVjVCK0ZIZE95dDlnOXFXbS94?=
 =?utf-8?B?T0VCUHpucmkweTJqS04ycTlwL2V1RDYweGwvWjJhd1dvSW5oM2tWUFlORzFs?=
 =?utf-8?Q?WDXvC7vUoudYM9mJQacKUTqh7?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 7a94705e-12fa-4ec6-0611-08dad4d9c0e5
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6304.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Dec 2022 02:54:53.7135
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: YgdMYENCDJC7owjeBb9TT0ArGQuGkvA4gSL7wkJCRIlfx36bdv1nq/D5oGEHHqE/LPsWao8wkui5h3dkJtW+6Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR11MB6958
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Dec 02, 2022 at 02:24:02PM -0800, Paul E. McKenney wrote:
[...]
> > > > > > > (My next step involves a pillow, but will follow up tomorrow morning
> > > > > > > Pacific Time.)
> > > > > >  
> > > > > > Really thanks for checking this through late night!
> > > > > 
> > > > > No problem, and I guess it is instead the day after tomorrow, but
> > > > > I thought you might be interested in chronyd's opinion:
> > > > > 
> > > > > [root@rtptest1029.snc8 ~]# cat /var/lib/chrony/drift
> > > > >         40001.074911             0.002098
> > > > > 
> > > > > In contrast, on my Fedora laptop:
> > > > > 
> > > > > $ sudo cat /var/lib/chrony/drift
> > > > >             2.074313             0.186606
> > > > > 
> > > > > I am (perhaps incorrectly) taking this to indicate that TSC is in fact
> > > > > drifting with respect to standard time.
> > > >  
> > > > This info is very useful! It further confirms the CPUID(0x15) gave
> > > > the wrong frequency info. 
> > > 
> > > So the TSC is just doing what it is told.  ;-)
> > > 
> > > This indicates a firmware problem?
> > > 
> > > > Also I don't think TSC itself is drifting, and the drift some from
> > > > the wrong match calculation(1896 MHz), if we give it the correct
> > > > number (likely 1975 MHz here), there shouldn't be big chrony drift
> > > > like your Fedora laptop.
> > > 
> > > Resetting so that the clocksource watchdog once again gets rid of TSC,
> > > but leaving nohpet:
> > > 
> > > [    0.000000] tsc: using CPUID[0x15] crystal_khz= 24000 kHz ebx=158 eax=2
> > > [    0.000000] tsc: Detected 1900.000 MHz processor
> > > [    0.000000] tsc: Detected 1896.000 MHz TSC
> > > [    5.287750] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
> > > [   17.963947] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x36a8d32ce31, max_idle_ns: 881590731004 ns
> > > [   19.996287] clocksource: timekeeping watchdog on CPU3: Marking clocksource 'tsc-early' as unstable because the skew is too large:
> > > [   20.040287] clocksource:                       'refined-jiffies' wd_nsec: 503923392 wd_now: fffb73f8 wd_last: fffb7200 mask: ffffffff
> > > [   20.067286] clocksource:                       'tsc-early' cs_nsec: 588021368 cs_now: 581c1eb378 cs_last: 57d9aad9e8 mask: ffffffffffffffff
> > >  [   20.096286] clocksource:                       No current clocksource.
> > >  [   20.111286] tsc: Marking TSC unstable due to clocksource watchdog
> > >  [   24.582541] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
> > >  [   49.739301] clocksource: Switched to clocksource refined-jiffies
> > >  [   50.046356] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
> > >  [   50.066475] clocksource: Switched to clocksource acpi_pm
> > > 
> > > # cat /var/lib/chrony/drift 
> > >             1.372570             0.020049
> > > 
> > > I interpret this to mean that acpi_pm (and thus from prior observations,
> > > HPET as well) are counting at the correct rate.
> > 
> > Correct. And this is a good news! that 1975 MHz seems to be the right
> > number.
> > 
> > Could you try below patch, it should override the CPUID calculation
> > and forced to use HW timer calibrated number:
> > 
> > ---
> > diff --git a/arch/x86/kernel/tsc.c b/arch/x86/kernel/tsc.c
> > index a78e73da4a74..68a2fea4961d 100644
> > --- a/arch/x86/kernel/tsc.c
> > +++ b/arch/x86/kernel/tsc.c
> > @@ -1417,7 +1417,8 @@ static int __init init_tsc_clocksource(void)
> >  	 * When TSC frequency is known (retrieved via MSR or CPUID), we skip
> >  	 * the refined calibration and directly register it as a clocksource.
> >  	 */
> > -	if (boot_cpu_has(X86_FEATURE_TSC_KNOWN_FREQ)) {
> > +//	if (boot_cpu_has(X86_FEATURE_TSC_KNOWN_FREQ)) {
> > +	if (false) {
> >  		if (boot_cpu_has(X86_FEATURE_ART))
> >  			art_related_clocksource = &clocksource_tsc;
> >  		clocksource_register_khz(&clocksource_tsc, tsc_khz);
> > ---
> > 
> > And together with b50db7095fe0 "x86/tsc: Disable clocksource watchdog
> > for TSC on qualified platorms". I assume this will tell TSC to use
> > 1975 MHZ as its frequency.
> 
> This did not change things,

Maybe the kernel is with the tsc reclibrate patch? it will not use
the reclibrated frequency for a KNOWN_FREQ case, but just print
out the info. Anyway this doesn't matter, as it's just for debug,
and your way already proved the 1975 MHz is the right number :)

> but when I hardcoded tsc_khz to 1975, the
> clocksource watchdog no longer disables TSC and chrony shows drifts of
> less than 2.0.  (As opposed to about 40,000 without the hardcoding.)
> 
> So yes, forcing 1975 makes TSC work nicely.  Yay!  ;-)

Great to know it works and we have some workaround before a final
solution :)

Thanks,
Feng

> 							Thanx, Paul