Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755459AbbHQQPK (ORCPT ); Mon, 17 Aug 2015 12:15:10 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:20871 "EHLO mx0b-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751317AbbHQQPI (ORCPT ); Mon, 17 Aug 2015 12:15:08 -0400 Date: Mon, 17 Aug 2015 09:14:51 -0700 From: Shaohua Li To: CC: , John Stultz , Thomas Gleixner , Ingo Molnar Subject: Re: [PATCH 1/2] clocksource: improve unstable clocksource detection Message-ID: <20150817161450.GA4079968@devbig257.prn2.facebook.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.20 (2009-12-10) X-Originating-IP: [192.168.52.123] X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.14.151,1.0.33,0.0.0000 definitions=2015-08-17_03:2015-08-17,2015-08-17,1970-01-01 signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2835 Lines: 65 ping, any comments? On Wed, Aug 05, 2015 at 11:12:53AM -0700, Shaohua Li wrote: > From time to time we saw TSC is marked as unstable in our systems, while > the CPUs declare to have stable TSC. Looking at the clocksource unstable > detection, there are two problems: > - watchdog clock source wrap. HPET is the most common watchdog clock > source. It's 32-bit and runs in 14.3Mhz. That means the hpet counter > can wrap in about 5 minutes. > - threshold isn't scaled against interval. The threshold is 0.0625s in > 0.5s interval. What if the actual interval is bigger than 0.5s? > > The watchdog runs in a timer bh, so hard/soft irq can defer its running. > Heavy network stack softirq can hog a cpu. IPMI driver can disable > interrupt for a very long time. The first problem is mostly we are > suffering I think. > > Here is a simple patch to fix the issues. If the waterdog doesn't run > for a long time, we ignore the detection. This should work for the two > problems. For the second one, we probably doen't need to scale if the > interval isn't very long. > > Cc: John Stultz > Cc: Thomas Gleixner > Cc: Ingo Molnar > Signed-off-by: Shaohua Li > --- > kernel/time/clocksource.c | 7 +++++-- > 1 file changed, 5 insertions(+), 2 deletions(-) > > diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c > index 841b72f..8417c83 100644 > --- a/kernel/time/clocksource.c > +++ b/kernel/time/clocksource.c > @@ -122,9 +122,10 @@ static int clocksource_watchdog_kthread(void *data); > static void __clocksource_change_rating(struct clocksource *cs, int rating); > > /* > - * Interval: 0.5sec Threshold: 0.0625s > + * Interval: 0.5sec MaxInterval: 1s Threshold: 0.0625s > */ > #define WATCHDOG_INTERVAL (HZ >> 1) > +#define WATCHDOG_MAX_INTERVAL_NS (NSEC_PER_SEC) > #define WATCHDOG_THRESHOLD (NSEC_PER_SEC >> 4) > > static void clocksource_watchdog_work(struct work_struct *work) > @@ -217,7 +218,9 @@ static void clocksource_watchdog(unsigned long data) > continue; > > /* Check the deviation from the watchdog clocksource. */ > - if ((abs(cs_nsec - wd_nsec) > WATCHDOG_THRESHOLD)) { > + if ((abs(cs_nsec - wd_nsec) > WATCHDOG_THRESHOLD) && > + cs_nsec < WATCHDOG_MAX_INTERVAL_NS && > + wd_nsec < WATCHDOG_MAX_INTERVAL_NS) { > pr_warn("timekeeping watchdog: Marking clocksource '%s' as unstable because the skew is too large:\n", > cs->name); > pr_warn(" '%s' wd_now: %llx wd_last: %llx mask: %llx\n", > -- > 1.8.5.6 > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/