Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752145AbdF1OaB (ORCPT ); Wed, 28 Jun 2017 10:30:01 -0400 Received: from mx2.suse.de ([195.135.220.15]:34705 "EHLO mx1.suse.de" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751583AbdF1O35 (ORCPT ); Wed, 28 Jun 2017 10:29:57 -0400 Date: Wed, 28 Jun 2017 16:29:52 +0200 From: Michal Hocko To: "Liang, Kan" Cc: "linux-kernel@vger.kernel.org" , "dzickus@redhat.com" , "mingo@kernel.org" , "akpm@linux-foundation.org" , "babu.moger@oracle.com" , "atomlin@redhat.com" , "prarit@redhat.com" , "torvalds@linux-foundation.org" , "peterz@infradead.org" , "tglx@linutronix.de" , "eranian@google.com" , "acme@redhat.com" , "ak@linux.intel.com" , "stable@vger.kernel.org" Subject: Re: [PATCH] kernel/watchdog: fix spurious hard lockups Message-ID: <20170628142952.GI5225@dhcp22.suse.cz> References: <20170620213309.30051-1-kan.liang@intel.com> <20170628114058.GB5234@dhcp22.suse.cz> <37D7C6CF3E00A74B8858931C1DB2F07753713A22@SHSMSX103.ccr.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <37D7C6CF3E00A74B8858931C1DB2F07753713A22@SHSMSX103.ccr.corp.intel.com> User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 1483 Lines: 34 On Wed 28-06-17 13:24:08, Liang, Kan wrote: > > > > From: Kan Liang > > > > > > Some users reported spurious NMI watchdog timeouts. > > > > > > We now have more and more systems where the Turbo range is wide > > enough > > > that the NMI watchdog expires faster than the soft watchdog timer that > > > updates the interrupt tick the NMI watchdog relies on. > > > > AFAIR the watchdog doesn't rely on deferred timers so this would suggest > > that a standard hrtimer can expire much later than programmed, right? > > The softlockup watchdog relies on hrtimers. > The hardlockup watchdog (NMI watchdog) relies on perf subsystem and > using unhalted CPU cycles. > When the softlockup watchdog expires, it updates the hrtimer_interrupts. > When the NMI watchdog expires, it will check the hrtimer_interrupts, and > determine if it's a hardlockup. > The design was to make the softlockup watchdog runs with 2.5 times the > rate of NMI watchdog. So it guarantees that the hrtimer_interrupts is > updated before the NMI watchdog expires. > That works well if Turbo-Mode is disabled. > However, when Turbo-Mode is enabled, unhalted CPU cycles might run > much faster than expected, even faster than softlockup watchdog. > So the softlockup watchdog will not get a chance to update the > hrtimer_interrupts, which will trigger false positives. So it is not the hrtimer which doesn't fire but rather the NMI events fire too quickly, right? -- Michal Hocko SUSE Labs