Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 23 Nov 2022 10:36:04 +0800
From:   Feng Tang <feng.tang@intel.com>
To:     "Paul E. McKenney" <paulmck@kernel.org>
CC:     Thomas Gleixner <tglx@linutronix.de>,
        <linux-kernel@vger.kernel.org>, <john.stultz@linaro.org>,
        <sboyd@kernel.org>, <corbet@lwn.net>, <Mark.Rutland@arm.com>,
        <maz@kernel.org>, <kernel-team@meta.com>, <neeraju@codeaurora.org>,
        <ak@linux.intel.com>, <zhengjun.xing@intel.com>,
        Chris Mason <clm@meta.com>, John Stultz <jstultz@google.com>,
        Waiman Long <longman@redhat.com>
Subject: Re: [PATCH clocksource 1/3] clocksource: Reject bogus watchdog
 clocksource measurements
Message-ID: <Y32HFE+BaATh9+l/@feng-clx>
References: <20221114232807.GA834337@paulmck-ThinkPad-P17-Gen-1>
 <20221114232827.835599-1-paulmck@kernel.org>
 <87mt8pkzw1.ffs@tglx>
 <20221117230910.GI4001@paulmck-ThinkPad-P17-Gen-1>
 <Y3rMc2VbgVLHN9db@feng-clx>
 <20221121181449.GA3774542@paulmck-ThinkPad-P17-Gen-1>
 <Y3zxB6r1kin8pSH1@feng-clx>
 <20221122220712.GP4001@paulmck-ThinkPad-P17-Gen-1>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20221122220712.GP4001@paulmck-ThinkPad-P17-Gen-1>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?Wnmjx1k1RLRzVIiPqvQF0kY88pThGlfTX4NtEIAFJrby8XDN8/atWMRKG1og?=
 =?us-ascii?Q?MtmJeZ/rLtiA1/hvThkOTmiffOysz/IIEj4hSl6mCDymelFxgZfBQIc/kl/3?=
 =?us-ascii?Q?6c3e1z4DIf//DN9k5C/jTSN7LX7HiqZOAmgwM3sGzbHSMdNT/C05oJkeGep5?=
 =?us-ascii?Q?v0yVEAN61uPOCvLNCKI6e7etEY6AhXkGG/NEMoCxpoT1CnWubNG02pgTdc0E?=
 =?us-ascii?Q?0UEFm0TAkOSfTxg3Zd4fIduthdF+tEUvEyRFzYlpM6TpQq5UUD52nuRqvAkW?=
 =?us-ascii?Q?aNbQ/01fSNaB+DX08mQLgrLJSzrwU/1Rj4nmEQ3cHzOIa53KLkd2agx1xZn6?=
 =?us-ascii?Q?P5viKiHPbGnjMAh6JviGiB7X5kU8s+YDxigl8i6i+s2xyLb/+I5PNTRbP+I8?=
 =?us-ascii?Q?gOlnUX0ilDHtKOgx8E39PYgocKvrY60Z1ucrjdafxlA9UVVe5hABiBRz4le7?=
 =?us-ascii?Q?5zSlvnpvvTk2oyxUxV2li16PSJv7ocr0OJGK/BPsMBgHox1PEf1SPKrCJFCL?=
 =?us-ascii?Q?h8S5eLzObtugoNv78+ipFPIwAzcFyUzINoJNkmR+8fs4sh4x0x1L8S/3Xzzk?=
 =?us-ascii?Q?kB4fngZf3ZapbD7ogDyRPwB53k/JTstKRKR+mF3TTzsaMbq9U2+CpxbvZGr2?=
 =?us-ascii?Q?TradSNpnXNgVDnPkrXGeOrTq85KxATyIK/+AVkuC9gozCyRcjh5hbR7o2ZK/?=
 =?us-ascii?Q?1eWJCOjuWpNZiphGj1dZSKuubdh91/Cs3FuH4id134ju5o66onBveYpn+v4m?=
 =?us-ascii?Q?B6orq+fyvnwep5Fkf71xDe+wKFuPbOLYrP+j+paaatNchM1H+Ryb9PCglK5a?=
 =?us-ascii?Q?Yy/W3oGy27hWKMDSWyUdDHuvNfgG6ZdHV85jKatfhKEFZDscExx+AbimhA2B?=
 =?us-ascii?Q?b8TFidlJOMnC5IBkPUn6OHrZvHtGZRDRKsy7YmIPl0OTOibxr8N/CgZdi2dj?=
 =?us-ascii?Q?wBjHkTyzZ8Y13NuCUw1wS3rbEBn6AN4umAylKIDw9hmH5ZFMHExRklEyK68s?=
 =?us-ascii?Q?KM6O2xP4w9yO1wppmAtjxwAXeZrDxTNng7QQe5rBfjvas40j2ABk7+zKcDXr?=
 =?us-ascii?Q?mJUN5RvF/EPiq2Y11khKaqyH97j0AztJD/oIFBdCbVgQapUPB70TPNwHxub7?=
 =?us-ascii?Q?HY7QAe2qs/jP+1DF1IarsI+84L2Uok2XYt/P8Bl10x2sx9S6TUYXgS/uAeWA?=
 =?us-ascii?Q?iY5U8M0zjlloR13suX0NA5mby5GOATc2/0sGpAdl3lTe78WEvP8hONrlPYez?=
 =?us-ascii?Q?o+YcXfN6vs+OIk3qmvgV/ltVfSjj+NhFStx8qs+WgiRc12oKU+27bvpmpC5d?=
 =?us-ascii?Q?ykeIM5SARRgQInyMOFP4QXnUK4ordbUVFYBc217NiGIXSuH/t1axCY+LZIjg?=
 =?us-ascii?Q?lHZpueW1FK1TtN4RjcPqvmQLJSlO73vFXhgLEWoj4CEGjARrJZYoGjpPF/0s?=
 =?us-ascii?Q?jX/8n7tHl/iUH1+tGEfWV6jlJSztw90YPtYFV5J+RivV0rZP36iOt6tGs8+u?=
 =?us-ascii?Q?0yr+PBLrAAz4cWMlBCcytNg8UyUYYLd3dA/OAT1CCzZ7EHt3DQD1glKO46ky?=
 =?us-ascii?Q?dAs7kjxVy7orOsg8DotAlS+egHbyj5LoLKP5P8dS?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 49a88736-a272-4180-d5dd-08daccfbedc5
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6304.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Nov 2022 02:39:22.5619
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: F7X/KdG8Trs+3gy+by+I3xrQTyV7QV0MVuTsxQYUxD7ku66A19yTQ5OeHblYGiKoWA8r7xQ1oFNLdPtV0K6+6g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR11MB4766
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_PASS,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Tue, Nov 22, 2022 at 02:07:12PM -0800, Paul E. McKenney wrote:
[...] 
> > > If PM_TIMER was involved, I would expect 'acpi_pm' instead of
> > > refined-jiffies.  Or am I misinterpreting the output and/or code?
> > 
> > It's about timing. On a typical server platform, the clocksources
> > init order could be:
> >   refined-jiffies --> hpet --> tsc-early --> acpi_pm --> tsc 
> > 
> > >From your log, TSC('tsc-early') is disabled before 'acpi_pm' get
> > initialized, so 'acpi_pm' timer (if exist) had no chance to watchdog
> > the tsc.
> > 
> > > Either way, would it make sense to add CLOCK_SOURCE_MUST_VERIFY to
> > > clocksource_hpet.flags?
> > 
> > Maybe try below patch, which will skip watchdog for 'tsc-early',
> > while giving 'acpi_pm' timer a chance to watchdog 'tsc'.
> > 
> > diff --git a/arch/x86/kernel/tsc.c b/arch/x86/kernel/tsc.c
> > index cafacb2e58cc..9840f0131764 100644
> > --- a/arch/x86/kernel/tsc.c
> > +++ b/arch/x86/kernel/tsc.c
> > @@ -1131,8 +1131,7 @@ static struct clocksource clocksource_tsc_early = {
> >  	.uncertainty_margin	= 32 * NSEC_PER_MSEC,
> >  	.read			= read_tsc,
> >  	.mask			= CLOCKSOURCE_MASK(64),
> > -	.flags			= CLOCK_SOURCE_IS_CONTINUOUS |
> > -				  CLOCK_SOURCE_MUST_VERIFY,
> > +	.flags			= CLOCK_SOURCE_IS_CONTINUOUS,
> >  	.vdso_clock_mode	= VDSO_CLOCKMODE_TSC,
> >  	.enable			= tsc_cs_enable,
> >  	.resume			= tsc_resume,
> 
> Your mainline patch b50db7095fe0 ("x86/tsc: Disable clocksource watchdog
> for TSC on qualified platorms") mitigates the issue so we are good for
> the immediate future, at least assuming reliable TSC.
> 
> But it also disables checking against HPET, hence my question about
> marking clocksource_hpet.flags with CLOCK_SOURCE_MUST_VERIFY at boot time
> on systems whose CPUs have constant_tsc, nonstop_tsc, and tsc_adjust.

IIUC, this will make TSC to watchdog HPET every 500 ms. We have got
report that the 500ms watchdog timer had big impact on some parallel
workload on big servers, that was another factor for us to seek
stopping the timer.

Is this about the concern of possible TSC frequency calibration
issue, as the 40 ms per second drift between HPET and TSC? With 
b50db7095fe0 backported, we also have another patch to force TSC
calibration for those platforms which get the TSC freq directly
from CPUID or MSR and don't have such info in dmesg:
 "tsc: Refined TSC clocksource calibration: 2693.509 MHz" 

https://lore.kernel.org/lkml/20220509144110.9242-1-feng.tang@intel.com/

We did met tsc calibration issue due to some firmware issue, and
this can help to catch it. You can try it if you think it's relevant.

Thanks,
Feng

> Would that make sense?
> 
> 							Thanx, Paul
> 
> > > I am sending the full console output off-list.  Hey, you asked for it!  ;-)
> > 
> > Thanks for sharing!
> > 
> > Thanks,
> > Feng
> > 
> > > 
> > > 							Thanx, Paul