Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Zhenzhong Duan <zhenzhong.duan@oracle.com>
To:     linux-kernel@vger.kernel.org
Cc:     Zhenzhong Duan <zhenzhong.duan@oracle.com>,
        John Stultz <john.stultz@linaro.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Stephen Boyd <sboyd@kernel.org>,
        Waiman Long <longman@redhat.com>,
        Srinivas Eeda <srinivas.eeda@oracle.com>
Subject: [PATCH 1/2] acpi_pm: Fix bootup softlockup due to PMTMR counter read contention
Date:   Thu, 14 Mar 2019 16:42:11 +0800
Message-Id: <1552552932-21821-1-git-send-email-zhenzhong.duan@oracle.com>
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

During bootup stage of a large system with many CPUs, with nohpet, PMTMR
is temporarily selected as the clock source which can lead to a softlockup
because of the following reasons:
 1) There is a single PMTMR counter shared by all the CPUs.
 2) PMTMR counter reading is a very slow operation.

At bootup stage tick device is firstly initialized in periodic mode and
then switch to one-shot mode when a high resolution clocksource is
initialized. Between clocksoure initialization and switching to one-shot
mode, there is small window where timer interrupt triggers.

Due to PMTMR read contention, the 1ms(HZ=1000) interval isn't enough
for all the CPUs to process timer interrupt in periodic mode.
Then CPUs are busy processing interrupt one by one without a break,
tick_clock_notify() have no chance to be called and we never switch
to one-shot mode. Finally the system may crash because of a NMI
watchdog soft lockup, logs:

[   20.181521] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff,
max_idle_ns: 2085701024 ns
[   44.273786] BUG: soft lockup - CPU#48 stuck for 23s! [swapper/48:0]
[   44.279992] BUG: soft lockup - CPU#49 stuck for 23s! [migration/49:307]
[   44.285169] BUG: soft lockup - CPU#50 stuck for 23s! [migration/50:313]

In one-shot mode, the contention is still there but next event is always
set with a future value. We may missed some ticks, but the timer code is
smart enough to pick up those missed ticks.

By moving tick_clock_notify() into stop_machine, kernel changes to one-shot
mode early before the contention accumulate and lockup system.

This patch also address the same issue of commit f99fd22e4d4b ("x86/hpet:
Reduce HPET counter read contention") in a simple way, so that commit could
be reverted.

Signed-off-by: Zhenzhong Duan <zhenzhong.duan@oracle.com>
Tested-by: Kin Cho <kin.cho@oracle.com>
Cc: John Stultz <john.stultz@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Stephen Boyd <sboyd@kernel.org>
Cc: Waiman Long <longman@redhat.com>
Cc: Srinivas Eeda <srinivas.eeda@oracle.com>
---
 kernel/time/timekeeping.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c
index f986e19..815c92d 100644
--- a/kernel/time/timekeeping.c
+++ b/kernel/time/timekeeping.c
@@ -1378,6 +1378,7 @@ static int change_clocksource(void *data)
 
 	write_seqcount_end(&tk_core.seq);
 	raw_spin_unlock_irqrestore(&timekeeper_lock, flags);
+	tick_clock_notify();
 
 	return 0;
 }
@@ -1396,7 +1397,6 @@ int timekeeping_notify(struct clocksource *clock)
 	if (tk->tkr_mono.clock == clock)
 		return 0;
 	stop_machine(change_clocksource, clock, NULL);
-	tick_clock_notify();
 	return tk->tkr_mono.clock == clock ? 0 : -1;
 }
 
-- 
1.8.3.1