Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 23 Dec 2022 10:09:04 +0800
From:   Feng Tang <feng.tang@intel.com>
To:     "Paul E. McKenney" <paulmck@kernel.org>
CC:     Waiman Long <longman@redhat.com>, John Stultz <jstultz@google.com>,
        "Thomas Gleixner" <tglx@linutronix.de>,
        Stephen Boyd <sboyd@kernel.org>, <x86@kernel.org>,
        Peter Zijlstra <peterz@infradead.org>,
        <linux-kernel@vger.kernel.org>, Tim Chen <tim.c.chen@intel.com>
Subject: Re: [RFC PATCH] clocksource: Suspend the watchdog temporarily when
 high read lantency detected
Message-ID: <Y6UNwAOQIGWyDfsX@feng-clx>
References: <Y6Ja+kYQAi4pppV6@feng-clx>
 <8a9bed0d-c166-37e9-24c3-8cea7a336c76@redhat.com>
 <20221222004032.GI4001@paulmck-ThinkPad-P17-Gen-1>
 <a82092f5-abc8-584f-b2ba-f06c82ffbe7d@redhat.com>
 <20221222055515.GJ4001@paulmck-ThinkPad-P17-Gen-1>
 <Y6PyisHYYtde/6Xk@feng-clx>
 <20221222061429.GL4001@paulmck-ThinkPad-P17-Gen-1>
 <Y6P7JPHf9wDse0XK@feng-clx>
 <20221222182446.GQ4001@paulmck-ThinkPad-P17-Gen-1>
 <20221222214219.GA1875767@paulmck-ThinkPad-P17-Gen-1>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20221222214219.GA1875767@paulmck-ThinkPad-P17-Gen-1>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?wVmYG4AQEKMRwKlU9fEjTo3ZWWQ+vgKjxaxE21XaNsisKCp+vJQllwofmcVV?=
 =?us-ascii?Q?BCYpqXuCd3PTD0oQu6SN6B8b1Ik2nBFP3JUFeP/Enys4eanSQyw4PY7ahrP8?=
 =?us-ascii?Q?97Np6cb63R3TgA6R9iG/+sOqQWu/+sxQxgjda1Ejjam+p70Dk4lYQtgBqSoR?=
 =?us-ascii?Q?AHcU814qcKP7IpQEtcbC7zGl/cOhC1snv06eOzIOOn9v18o59UBKjiipYOjs?=
 =?us-ascii?Q?zM/YyBqzNPb4xa5GwOpdwgFtdy4zWLzddNgnw/LyF6qWCVTV1cmITP3cbrAH?=
 =?us-ascii?Q?m0gulwiGBOEjWvGMURa5pjnLXHhoARLuLfdU4nga9zA07J6HSeDOOy9wW4sw?=
 =?us-ascii?Q?G3XOU+jtnPyQ9w//zdJaPyFufrpaDlb7KAIHOOm6u859MclC2ETM+xcUyr/C?=
 =?us-ascii?Q?uILqIfwlGxVcdD/sZh/8pClTrDKSI8npCijdnD9B6pKS6o2paGrFSmd3+Cmy?=
 =?us-ascii?Q?f25+HJEtbWnG3FrtZ6FdxF5xzrQjdoiR/MH16CHWauB3tghGjGdpQS05OvOw?=
 =?us-ascii?Q?LDKx1uUaqTb+HyAj4JVtpHdpt8Oo7ySvOI+mw732un/Ni0QEV8r1H2oyhoY8?=
 =?us-ascii?Q?7pkOv76cRWcDbaKc2n63U1QE+wziTBlI95VZIuVkkfOvYYVAR4UMxNuUaDrE?=
 =?us-ascii?Q?P0c6YbcCnT95uydSOf03OzAV/DtVcKTth205Qtv9XiGlyU8n5C90cQBLDar0?=
 =?us-ascii?Q?9AP5EgbMzSn6/jUIQjD695e24zSnpzeBheF+CIbblilXGUqdXkW9DP7liHLb?=
 =?us-ascii?Q?8wzMBVBq7YyVrks63i7UugqyieQ4tC3Gb27Tj7heCnMYriC/s3cQpcqmz/kQ?=
 =?us-ascii?Q?JN201oYc1qv7E7svcr3X/sAPuEuShUxqpNTQTnQ0E7hgOGOyap8XOTp3YOfA?=
 =?us-ascii?Q?NZxUGx+kC/jVrtHsm3Jwg2iBdR7F9By+VCWGyF/9H9dH/sKg9t2UqqV1SciF?=
 =?us-ascii?Q?ji2fM0LMdAemijnIRTSWqFV+j5Nbm0AEuUguPSOG3EV8+6N31PMRQBF5RG5C?=
 =?us-ascii?Q?eakfiUO7EFSrDUy20Xufkwikmhq6wazIWH7rpqHZPJQRU6HG1siYxnDihZpZ?=
 =?us-ascii?Q?soiOHfaZ6wMIbYTeIqMRRCIRB/rLb1QUD2ecobDGcGVJqmPvoZ9zeycWcbzL?=
 =?us-ascii?Q?DhJuvc9tnzVZmLGwc2k1cttq+7fg9lAA+Z4DPPHqDQ+zYcJg9Tq2LlD7jRXa?=
 =?us-ascii?Q?Q735KX7nXA2fHFer9nITMIYSLsv3J9N8RuAfxRrldRlaBzQLkfsYhw1mixEQ?=
 =?us-ascii?Q?B5456xeNu1vjGzZWaUHwr42mIXn5/gKoNnlDuR2j+nZ3TRSY4RzhcYCKWSAD?=
 =?us-ascii?Q?jJhLaw3T40UIZySkMpD+rDmXJCZxAwT9wv4ldOUX+vq1WXNyhMC0FI/2stcz?=
 =?us-ascii?Q?/MkUp2KwDZfwQBMh+UCDr2Ck/Aj2J55HADmbwDiqKiQx6uKNU7bryVHxBfbS?=
 =?us-ascii?Q?6Q6d6ptkwORXCKKsaPteCTVMAhOIG6dH69wBpQ807Fwb5d6/nBrV5N8OhPUq?=
 =?us-ascii?Q?RhE9iCH39DXUxJtbRqY0aGQP5DXJO/vyMVj7NVw4HCiWpZP/sgMN9KcBvWtf?=
 =?us-ascii?Q?VaBdhNTg2NJN3hLJISgP/NkvLrg+Ni65PeVbhZyY?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 1ec7bf2b-67ab-448f-5a9e-08dae48b10f9
X-MS-Exchange-CrossTenant-AuthSource: MN0PR11MB6304.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Dec 2022 02:11:56.4897
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: fvxTvuqJj+jGtOpUvVSqpKmqc/fH09A5Au65eZ1MXHn00cOiEC/4THT3JiazELa259lrdJ5ad69jaEVyx4AXDg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR11MB6479
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Thu, Dec 22, 2022 at 01:42:19PM -0800, Paul E. McKenney wrote:
[...]
> > > 
> > > > > Also I've run the patch on a Alderlake system, with a fine acpi pm_timer
> > > > > and a fake broken pm_timer, and they both works without errors.
> > > > 
> > > > Thank you!  Did it correctly identify the fake broken pm_timer as being
> > > > broken?  If so, may I have your Tested-by?
> > > 
> > > On that Alderlake system, HPET will be disabled by kernel, and I
> > > manually increased the tsc frequency about 1/256 to make pm_timer
> > > look to have 1/256 deviation. And got dmesg like:
> > > 
> > > [    2.738554] clocksource: timekeeping watchdog on CPU3: Marking clocksource 'acpi_pm' as unstable because the skew is too large:
> > > [    2.738558] clocksource:                       'tsc' wd_nsec: 275956624 wd_now: 13aab38d0d wd_last: 1382c1144d mask: ffffffffffffffff
> > > [    2.738564] clocksource:                       'acpi_pm' cs_nsec: 277034651 cs_now: 731575 cs_last: 63f3cb mask: ffffff
> > > [    2.738568] clocksource:                       'tsc' (not 'acpi_pm') is current clocksource.
> > > 
> > > The deviation is indeed about 1/256. And pm_timer won't be shown in /sys/:
> > > 
> > > /sys/devices/system/clocksource/clocksource0/available_clocksource:tsc 
> > > /sys/devices/system/clocksource/clocksource0/current_clocksource:tsc
> > > 
> > > So feel free to add:
> > > 
> > > 	Tested-by: Feng Tang <feng.tang@intel.com>
> > 
> > Thank you very much!  I will apply this on my next rebase.
> 
> But first, here is a prototype of the limited-time clocksource watchdog.
> Thoughts?
> 
> 						Thanx, Paul
> 
> ------------------------------------------------------------------------
> 
> commit dfbf67806c4c7f2bdd79cdefe86a2bea6e7afcab
> Author: Paul E. McKenney <paulmck@kernel.org>
> Date:   Thu Dec 22 13:21:47 2022 -0800
> 
>     clocksource: Permit limited-duration clocksource watchdogging
>     
>     Some systems want regular clocksource checking, but their workloads
>     cannot tolerate the overhead of full-time clocksource checking.
>     Therefore, add a CLOCKSOURCE_WATCHDOG_DURATION Kconfig option and a
>     clocksource.watchdog_duration kernel-boot parameter that limits the
>     clocksource watchdog to the specified number of minutes past boot.
>     Values of zero disable checking, and a value of -1 restores the
>     traditional behavior of always checking.
>     
>     This does change behavior compared to older kernels, but recent kernels
>     disable the clocksource watchdog completely in the common case where the
>     TSC is judged to be trustworthy.  This change in behavior is therefore
>     not a real issue.
    
Yes, this changes the general semantics. Last year, I've posted a
patch to limit the watchdog to run for 10 minutes, and at that time
Thomas mentioned one of his machine may show tsc issue after running
for one day depending on work load [1].

As the intention is to validate HPET/PMTIMER, which are not as
delicate as TSC, maybe we can add a per-clocksource verify-period
field, and only set it for HPET/PMTIMER?

[1]. https://lore.kernel.org/lkml/875z286xtk.fsf@nanos.tec.linutronix.de/

Thanks,
Feng

>     Link: https://lore.kernel.org/all/a82092f5-abc8-584f-b2ba-f06c82ffbe7d@redhat.com/
>     Reported-by: Waiman Long <longman@redhat.com>
>     Reported-by: Feng Tang <feng.tang@intel.com>
>     Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
> 
> diff --git a/kernel/time/Kconfig b/kernel/time/Kconfig
> index bae8f11070bef..2676e011673d5 100644
> --- a/kernel/time/Kconfig
> +++ b/kernel/time/Kconfig
> @@ -209,5 +209,17 @@ config CLOCKSOURCE_WATCHDOG_MAX_SKEW_US
>  	  per million.	If the clocksource is good enough for NTP,
>  	  it is good enough for the clocksource watchdog!
>  
> +config CLOCKSOURCE_WATCHDOG_DURATION
> +	int "Default time to run clocksource watchdog (in minutes)"
> +	depends on CLOCKSOURCE_WATCHDOG
> +	range -1 1440
> +	default 10
> +	help
> +	  Specify the default number of minutes that the clocksource
> +	  watchdog should run after being started.  Specify -1 to run
> +	  indefinitely or zero to not run at all.  This value may be
> +	  overridden using the clocksource.watchdog_duration kernel
> +	  boot parameter.
> +
>  endmenu
>  endif
> diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c
> index b153fce8c0e4b..c920c6c22e0fb 100644
> --- a/kernel/time/clocksource.c
> +++ b/kernel/time/clocksource.c
> @@ -213,6 +213,9 @@ module_param(max_cswd_read_retries, ulong, 0644);
>  EXPORT_SYMBOL_GPL(max_cswd_read_retries);
>  static int verify_n_cpus = 8;
>  module_param(verify_n_cpus, int, 0644);
> +static int watchdog_duration = CONFIG_CLOCKSOURCE_WATCHDOG_DURATION;
> +module_param(watchdog_duration, int, 0444);
> +static unsigned long watchdog_end_jiffies;
>  
>  enum wd_read_status {
>  	WD_READ_SUCCESS,
> @@ -549,7 +552,9 @@ static void clocksource_watchdog(struct timer_list *unused)
>  	 * Arm timer if not already pending: could race with concurrent
>  	 * pair clocksource_stop_watchdog() clocksource_start_watchdog().
>  	 */
> -	if (!timer_pending(&watchdog_timer)) {
> +	if (!timer_pending(&watchdog_timer) &&
> +	    (watchdog_duration < 0 ||
> +	     (watchdog_duration >= 0 && time_before(jiffies, watchdog_end_jiffies)))) {
>  		watchdog_timer.expires += WATCHDOG_INTERVAL + extra_wait;
>  		add_timer_on(&watchdog_timer, next_cpu);
>  	}
> @@ -559,8 +564,10 @@ static void clocksource_watchdog(struct timer_list *unused)
>  
>  static inline void clocksource_start_watchdog(void)
>  {
> -	if (watchdog_running || !watchdog || list_empty(&watchdog_list))
> +	if (watchdog_running || !watchdog || list_empty(&watchdog_list) || !watchdog_duration)
>  		return;
> +	if (watchdog_duration > 0)
> +		watchdog_end_jiffies = jiffies + watchdog_duration * 60 * HZ;
>  	timer_setup(&watchdog_timer, clocksource_watchdog, 0);
>  	watchdog_timer.expires = jiffies + WATCHDOG_INTERVAL;
>  	add_timer_on(&watchdog_timer, cpumask_first(cpu_online_mask));