Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 22 Dec 2022 14:37:24 +0800
From:   Feng Tang <feng.tang@intel.com>
To:     "Paul E. McKenney" <paulmck@kernel.org>
CC:     Waiman Long <longman@redhat.com>, John Stultz <jstultz@google.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Stephen Boyd <sboyd@kernel.org>, <x86@kernel.org>,
        Peter Zijlstra <peterz@infradead.org>,
        <linux-kernel@vger.kernel.org>, Tim Chen <tim.c.chen@intel.com>
Subject: Re: [RFC PATCH] clocksource: Suspend the watchdog temporarily when
 high read lantency detected
Message-ID: <Y6P7JPHf9wDse0XK@feng-clx>
References: <20221220082512.186283-1-feng.tang@intel.com>
 <6fb04ee9-ce77-4835-2ad1-b7f8419cfb77@redhat.com>
 <20221220183400.GY4001@paulmck-ThinkPad-P17-Gen-1>
 <Y6Ja+kYQAi4pppV6@feng-clx>
 <8a9bed0d-c166-37e9-24c3-8cea7a336c76@redhat.com>
 <20221222004032.GI4001@paulmck-ThinkPad-P17-Gen-1>
 <a82092f5-abc8-584f-b2ba-f06c82ffbe7d@redhat.com>
 <20221222055515.GJ4001@paulmck-ThinkPad-P17-Gen-1>
 <Y6PyisHYYtde/6Xk@feng-clx>
 <20221222061429.GL4001@paulmck-ThinkPad-P17-Gen-1>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20221222061429.GL4001@paulmck-ThinkPad-P17-Gen-1>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?2LC2vJp9PoF8oug6sa3CnmdPI63jv3gZwJyhFFJ0K/kXM8oEH4wSOyucGzEx?=
 =?us-ascii?Q?d+bASxXCQ6aN6rTgDxNw2WNcPcPclM0ynjlD4XDpD3URgFhEweefHDGG/NL3?=
 =?us-ascii?Q?+wgGHaqTlP0sbwJEJFx04YHkJs1w3/fwR+8+9nqE5gyl4xpPHG6lV2AowZ/v?=
 =?us-ascii?Q?z2oNI+hU66yOAJp5nujaOxiqfzMIOqx+ure0TbK79aaYto7Mt+JIDXEdEqwq?=
 =?us-ascii?Q?GAPqJqmmtYg8NBuqG5cpy7OfW0syKGi4GjHyRXHZriLNaZugqQoPT14JIpkf?=
 =?us-ascii?Q?60xJVXmRUoszep2nVtWT7dynBd91oSyevKxs8T8Mjeel3nVSklAK15HxQOE9?=
 =?us-ascii?Q?AET8F4GorwuET6Dr0+PXhMnv62+94kniRsUBiPM+4yaT6gaAFL4BMmYIqNqK?=
 =?us-ascii?Q?0M7RfxES3k8A1FQzRwjqu9ik5ZAJhjbwAuHpmiAr1VtfL3s3YNfMPYZ3Wnjq?=
 =?us-ascii?Q?ep23Y9Gh8IBENNRgeUsFLN7d5WjJ+YP2dd3HrLdFyInBS+hi9NaiQhYPAyio?=
 =?us-ascii?Q?B2nCim5ZmeIKfLXIHzf5XpsrfKXUhG7M1eWHmVeb2qDk0+V2bQGkSYAXABwT?=
 =?us-ascii?Q?MECd8yRPbdZ3xMEcbHFTzL+livRcX44tQ9Y8pKl8MPnn9wZ6EIfKqMwgXMfg?=
 =?us-ascii?Q?nFSMOwJIsRvR0j5KaQxwwCTBiAG5+0b+YYWQ1dJ4yDE2lyd8DJE/XFQppzE8?=
 =?us-ascii?Q?CD+WzuF4tqXwlA8HAoPlfo4COsHiagbQJmV+OsM9v/QWJaAyq4NDEGTqp9Mn?=
 =?us-ascii?Q?EmMdQrizTzDbeiPdv79+73cXLutIskOtG9jnU2Ei7GJRMRYEohRAGajfuUfb?=
 =?us-ascii?Q?dMnJl0Fs7UhEhrtHnmzAdNdw0e+ETp9cl8Oswb2RNSHxW8/WC/tD8SyqEHPV?=
 =?us-ascii?Q?fQLK3E0Z8dn6Jw+1mf0zfO/6ALx5iVTE9QkaZJTdvsJX349sCoMVcvUm/S+U?=
 =?us-ascii?Q?yE4SGriZUge1TNJxP+i0NsKsaVGz4aTfqP6HDVPs7pPBIf2FLqEJY1DP9Fpn?=
 =?us-ascii?Q?cUFc1DCBQV9DQsnzf1UAoaxSfwo1D+i+WujE8M9VxBZ9nWCODkJVFAx26KV+?=
 =?us-ascii?Q?3uEFBWxDYOCNckx3e8CJeUqQpyXejbofoVO3ydVpYGXiEj/rdCDV/Zs0U0/O?=
 =?us-ascii?Q?x57WFM1nGu/UNMZjhAQqcwmyVSAkVSpGvNOaXc88uS8XNhaqLRMIKxcJ1Vap?=
 =?us-ascii?Q?ThYU1AzaKWA1rly+DYqnAA/bjWuj/El/ESkNQTgJgDWxfSyhMLSPqU52I9x3?=
 =?us-ascii?Q?gM7uXwuIpZXEHRabxWmkw/6j4d61CE5azdM9B72oFIhgFEBvlC7cHjTI0sYa?=
 =?us-ascii?Q?EpK+sjbMD6WxN3gw9MLA+/JThuv84kd7VIagHk6OhGxqOaiI2iWZLusP2t2I?=
 =?us-ascii?Q?3zXk8aso2oSqDsg0I8FMF/oah2EFhZQFhooBu2+uUp08KC4T06ZNHeLumGtN?=
 =?us-ascii?Q?soXpF/oI9mGf0NDQeM4UmsRdR28+4VlgElE6E7n4pCTF5uiGYSxsQRxgrJnr?=
 =?us-ascii?Q?S2drnZOjL6U6OVT4jfyHRiWFf5UZccbYgg3AfAzorVl+y6JNbmDh7bKGKwWw?=
 =?us-ascii?Q?PGJ6cWvF1ttwZXbkLG5A8NDNSYk2MUbjj7NxXHHg?=
X-MS-Exchange-CrossTenant-Network-Message-Id: ab95350b-37c1-4762-82b7-08dae3e76378
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6304.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Dec 2022 06:40:17.4794
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Th25qFOG3l1dmVoKvpdGTrvavnvusgqx27nMBkf2VZukXz8P8jswxkeW0IBzjRqtFPZMq+o8hpuB3Im8i/ilBw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB6006
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Dec 21, 2022 at 10:14:29PM -0800, Paul E. McKenney wrote:
> On Thu, Dec 22, 2022 at 02:00:42PM +0800, Feng Tang wrote:
> > On Wed, Dec 21, 2022 at 09:55:15PM -0800, Paul E. McKenney wrote:
> > > On Wed, Dec 21, 2022 at 10:39:53PM -0500, Waiman Long wrote:
> > > > On 12/21/22 19:40, Paul E. McKenney wrote:
> > > > > commit 199dfa2ba23dd0d650b1482a091e2e15457698b7
> > > > > Author: Paul E. McKenney<paulmck@kernel.org>
> > > > > Date:   Wed Dec 21 16:20:25 2022 -0800
> > > > > 
> > > > >      clocksource: Verify HPET and PMTMR when TSC unverified
> > > > >      On systems with two or fewer sockets, when the boot CPU has CONSTANT_TSC,
> > > > >      NONSTOP_TSC, and TSC_ADJUST, clocksource watchdog verification of the
> > > > >      TSC is disabled.  This works well much of the time, but there is the
> > > > >      occasional system that meets all of these criteria, but which still
> > > > >      has a TSC that skews significantly from atomic-clock time.  This is
> > > > >      usually attributed to a firmware or hardware fault.  Yes, the various
> > > > >      NTP daemons do express their opinions of userspace-to-atomic-clock time
> > > > >      skew, but they put them in various places, depending on the daemon and
> > > > >      distro in question.  It would therefore be good for the kernel to have
> > > > >      some clue that there is a problem.
> > > > >      The old behavior of marking the TSC unstable is a non-starter because a
> > > > >      great many workloads simply cannot tolerate the overheads and latencies
> > > > >      of the various non-TSC clocksources.  In addition, NTP-corrected systems
> > > > >      often seem to be able to tolerate significant kernel-space time skew as
> > > > >      long as the userspace time sources are within epsilon of atomic-clock
> > > > >      time.
> > > > >      Therefore, when watchdog verification of TSC is disabled, enable it for
> > > > >      HPET and PMTMR (AKA ACPI PM timer).  This provides the needed in-kernel
> > > > >      time-skew diagnostic without degrading the system's performance.
> > > > >      Signed-off-by: Paul E. McKenney<paulmck@kernel.org>
> > > > >      Cc: Thomas Gleixner<tglx@linutronix.de>
> > > > >      Cc: Ingo Molnar<mingo@redhat.com>
> > > > >      Cc: Borislav Petkov<bp@alien8.de>
> > > > >      Cc: Dave Hansen<dave.hansen@linux.intel.com>
> > > > >      Cc: "H. Peter Anvin"<hpa@zytor.com>
> > > > >      Cc: Daniel Lezcano<daniel.lezcano@linaro.org>
> > > > >      Cc: Feng Tang<feng.tang@intel.com>
> > > > >      Cc: Waiman Long <longman@redhat.com
> > > > >      Cc:<x86@kernel.org>
> > > > 
> > > > As I currently understand, you are trying to use TSC as a watchdog to check
> > > > against HPET and PMTMR. I do have 2 questions about this patch.
> > > > 
> > > > First of all, why you need to use both HPET and PMTMR? Can you just use one
> > > > of those that are available. Secondly, is it possible to enable this
> > > > time-skew diagnostic for a limit amount of time instead running
> > > > indefinitely? The running of the clocksource watchdog itself will still
> > > > consume a tiny amount of CPU cycles.
> > > 
> > > I could certainly do something so that only the first of HPET and PMTMR
> > > is checked.  Could you give me a quick run-through of the advantages of
> > > using only one?  I would need to explain that in the commit log.
> > > 
> > > Would it make sense to have a kernel boot variable giving the number of
> > > minutes for which the watchdog was to run, with a default of zero
> > > meaning "indefinitely"?
> > 
> > We've discussed about the "os noise", which customer may really care.
> > IIUC, this patch intends to test if HPET/PMTIMER HW is broken, so how
> > about making it run for a number of minutes the default behavior.   
> 
> It is also intended to determine if TSC is broken, with NTP drift rates
> used to determine which timer is at fault.
> 
> OK, how about a Kconfig option for the number of minutes, set to whatever
> you guys tell me?  (Three minutes?  Five minutes?  Something else?)
> People wanting to run it continuously could then build their kernels
> with that Kconfig option set to zero.
 
I don't have specific preference for 5 or 10 minutes, as long as it
is a one time deal :) 

> > Also I've run the patch on a Alderlake system, with a fine acpi pm_timer
> > and a fake broken pm_timer, and they both works without errors.
> 
> Thank you!  Did it correctly identify the fake broken pm_timer as being
> broken?  If so, may I have your Tested-by?

On that Alderlake system, HPET will be disabled by kernel, and I
manually increased the tsc frequency about 1/256 to make pm_timer
look to have 1/256 deviation. And got dmesg like:

[    2.738554] clocksource: timekeeping watchdog on CPU3: Marking clocksource 'acpi_pm' as unstable because the skew is too large:
[    2.738558] clocksource:                       'tsc' wd_nsec: 275956624 wd_now: 13aab38d0d wd_last: 1382c1144d mask: ffffffffffffffff
[    2.738564] clocksource:                       'acpi_pm' cs_nsec: 277034651 cs_now: 731575 cs_last: 63f3cb mask: ffffff
[    2.738568] clocksource:                       'tsc' (not 'acpi_pm') is current clocksource.

The deviation is indeed about 1/256. And pm_timer won't be shown in /sys/:

/sys/devices/system/clocksource/clocksource0/available_clocksource:tsc 
/sys/devices/system/clocksource/clocksource0/current_clocksource:tsc

So feel free to add:

	Tested-by: Feng Tang <feng.tang@intel.com>

Thanks,
Feng

> 
> 							Thanx, Paul