Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752547AbdCEXUA (ORCPT ); Sun, 5 Mar 2017 18:20:00 -0500 Received: from mail-lf0-f66.google.com ([209.85.215.66]:34766 "EHLO mail-lf0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752398AbdCEXT6 (ORCPT ); Sun, 5 Mar 2017 18:19:58 -0500 Date: Mon, 6 Mar 2017 00:19:54 +0100 From: Frederic Weisbecker To: Meelis Roos Cc: Thomas Gleixner , Linux Kernel list , x86@kernel.org, Peter Zijlstra , Ingo Molnar Subject: Re: PPro arch_cpu_idle: NMI watchdog: Watchdog detected hard LOCKUP on cpu 1 Message-ID: <20170305231952.GA2686@lerouge> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3702 Lines: 86 On Sun, Mar 05, 2017 at 10:48:50PM +0200, Meelis Roos wrote: > Added some CC-s because of bisect find. Whole context should be still > here. > > > > > > > > This is on my trusty IBM PC365, dual Pentium Pro. 4.10 worked fine, > > > > > > > 4.10.0-09686-g9e314890292c and 4.10.0-10770-g2d6be4abf514 exhibit a > > > > > > > problem. Ocassionally NMI watchdog kicks in and discovers one of the > > > > > > > CPUs in LOCKUP. The system keeps running fine. The first lockup was > > > > > > > different, all the others were from arch_cpu_idle. Sometime ecey couple > > > > > > > of seconds (after some activity), sometimes nothing for a long time > > > > > > > (idle, no SSH logins). > > > > > > > > > > > > The only watchdog related patch which hit after 4.10 is: > > > > > > > > > > > > 8dcde9def5a1 kernel/watchdog.c: do not hardcode CPU 0 as the initial thread > > > > > > > > > > > > Can you try to revert that for a start? I'm not seeing why it should be the > > > > > > culprit from a quick glance, but ... > > > > > > > > > > Reverting this patch does not help. > > > > > > > > I did not expect that, but excluding it was a valid shot in the > > > > dark. Thanmks for trying. > > > > > > > > To be honest, I have no idea what causes that at the moment, but I will > > > > come back to you tomorrow after thinking it through (with brain awake) how > > > > to debug this. > > > > > > Went through the related changes which came in during the merge window. One > > > which affects the per cpu timers is: 914122c389d0 > > > > > > Can you try to revert that one please? > > > > Running out of obvious culprits. Any chance that you can do a bisect or > > this too painful on that box? > > Done on a P4 where the problem also appeared. The bisecting resulted in > this commit. Does it seem realistic? I will also try if this help son > the old PPro. > > 93825f2ec736f30e034ab7c9d56b42849c5b00da is the first bad commit > commit 93825f2ec736f30e034ab7c9d56b42849c5b00da > Author: Frederic Weisbecker > Date: Tue Jan 31 04:09:16 2017 +0100 > > jiffies: Reuse TICK_NSEC instead of NSEC_PER_JIFFY > > NSEC_PER_JIFFY is an ad-hoc redefinition of TICK_NSEC. Let's rather > use a unique and well maintained version. > > Signed-off-by: Frederic Weisbecker > Cc: Benjamin Herrenschmidt > Cc: Paul Mackerras > Cc: Michael Ellerman > Cc: Heiko Carstens > Cc: Martin Schwidefsky > Cc: Tony Luck > Cc: Fenghua Yu > Cc: Peter Zijlstra > Cc: Rik van Riel > Cc: Stanislaw Gruszka > Cc: Wanpeng Li > Link: http://lkml.kernel.org/r/1485832191-26889-1-git-send-email-fweisbec@gmail.com > Signed-off-by: Ingo Molnar Ouch, looking at that patch again, I probably had a delusional moment when I wrote this: diff --git a/kernel/time/jiffies.c b/kernel/time/jiffies.c index a4a0e47..7906b3f 100644 --- a/kernel/time/jiffies.c +++ b/kernel/time/jiffies.c @@ -125,7 +125,7 @@ int register_refined_jiffies(long cycles_per_second) shift_hz += cycles_per_tick/2; do_div(shift_hz, cycles_per_tick); /* Calculate nsec_per_tick using shift_hz */ - nsec_per_tick = (u64)NSEC_PER_SEC << 8; + nsec_per_tick = (u64)TICK_NSEC << 8; nsec_per_tick += (u32)shift_hz/2; do_div(nsec_per_tick, (u32)shift_hz); Could you please retry after reverting this specific chunk? (that would be the very fix to apply). Thanks!