Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752610AbdFUOmC (ORCPT ); Wed, 21 Jun 2017 10:42:02 -0400 Received: from mga01.intel.com ([192.55.52.88]:16302 "EHLO mga01.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752136AbdFUOmA (ORCPT ); Wed, 21 Jun 2017 10:42:00 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.39,369,1493708400"; d="scan'208";a="1143454838" From: kan.liang@intel.com To: linux-kernel@vger.kernel.org Cc: dzickus@redhat.com, mingo@kernel.org, akpm@linux-foundation.org, babu.moger@oracle.com, atomlin@redhat.com, prarit@redhat.com, torvalds@linux-foundation.org, peterz@infradead.org, tglx@linutronix.de, eranian@google.com, acme@redhat.com, ak@linux.intel.com, Kan Liang , stable@vger.kernel.org Subject: [PATCH V2] kernel/watchdog: fix spurious hard lockups Date: Wed, 21 Jun 2017 07:41:18 -0700 Message-Id: <20170621144118.5939-1-kan.liang@intel.com> X-Mailer: git-send-email 2.11.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2325 Lines: 63 From: Kan Liang Some users reported spurious NMI watchdog timeouts. We now have more and more systems where the Turbo range is wide enough that the NMI watchdog expires faster than the soft watchdog timer that updates the interrupt tick the NMI watchdog relies on. This problem was originally added by commit 58687acba592 ("lockup_detector: Combine nmi_watchdog and softlockup detector"). Previously the NMI watchdog would always check jiffies, which were ticking fast enough. But now the backing is quite slow so the expire time becomes more sensitive. For mainline the right fix is to switch the NMI watchdog to reference cycles, which tick always at the same rate independent of turbo mode. But this is requires some complicated changes in perf, which are too difficult to backport. Since we need a stable fix to just increase the NMI watchdog rate here to avoid the spurious timeouts. This is not an ideal fix because a 3x as large Turbo range could still fail, but for now that's not likely. Signed-off-by: Kan Liang Cc: stable@vger.kernel.org Fixes: 58687acba592 ("lockup_detector: Combine nmi_watchdog and softlockup detector") --- The right fix for mainline can be found here. perf/x86/intel: enable CPU ref_cycles for GP counter perf/x86/intel, watchdog: Switch NMI watchdog to ref cycles on x86 https://patchwork.kernel.org/patch/9779087/ https://patchwork.kernel.org/patch/9779089/ Change since V1: - Restrict the period in hw_nmi.c for Intel platform. (Don Zickus) arch/x86/kernel/apic/hw_nmi.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/arch/x86/kernel/apic/hw_nmi.c b/arch/x86/kernel/apic/hw_nmi.c index c73c9fb281e1..716d44e986f9 100644 --- a/arch/x86/kernel/apic/hw_nmi.c +++ b/arch/x86/kernel/apic/hw_nmi.c @@ -20,9 +20,15 @@ #include #ifdef CONFIG_HARDLOCKUP_DETECTOR +/* + * The NMI watchdog relies on PERF_COUNT_HW_CPU_CYCLES event, which + * can tick faster than the measured CPU Frequency due to Turbo mode. + * That can lead to spurious timeouts. + * To workaround the issue, extending the period by 3 times. + */ u64 hw_nmi_get_sample_period(int watchdog_thresh) { - return (u64)(cpu_khz) * 1000 * watchdog_thresh; + return (u64)(cpu_khz) * 1000 * watchdog_thresh * 3; } #endif -- 2.11.0