Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751507AbdFIAtH (ORCPT ); Thu, 8 Jun 2017 20:49:07 -0400 Received: from mail-wm0-f67.google.com ([74.125.82.67]:35740 "EHLO mail-wm0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751428AbdFIAtF (ORCPT ); Thu, 8 Jun 2017 20:49:05 -0400 Date: Fri, 9 Jun 2017 02:48:57 +0200 From: Frederic Weisbecker To: "Levin, Alexander (Sasha Levin)" Cc: Thomas Gleixner , Ingo Molnar , LKML , Peter Zijlstra , Rik van Riel , James Hartsock , "stable@vger.kernel.org" , Tim Wright , Pavel Machek Subject: Re: [PATCH] nohz: Fix spurious warning when hrtimer and clockevent get out of sync Message-ID: <20170609004853.GA27190@lerouge> References: <1492783255-5051-3-git-send-email-fweisbec@gmail.com> <20170603080638.7okm4ztp7zuphc4b@sasha-lappy> <20170603124237.GA25077@lerouge> <20170603125259.scyapxq2fftqiiz3@sasha-lappy> <20170606145227.GB22016@lerouge> <20170607040801.4r7iqfzynf3d3xom@sasha-lappy> <20170607141400.GA29571@lerouge> <20170607213349.wtc2xhljlx5zjv5f@sasha-lappy> <20170608190702.GB27757@lerouge> <20170608221336.vem2i3364sljme4n@sasha-lappy> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20170608221336.vem2i3364sljme4n@sasha-lappy> User-Agent: Mutt/1.5.24 (2015-08-30) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3770 Lines: 100 On Thu, Jun 08, 2017 at 10:13:38PM +0000, Levin, Alexander (Sasha Levin) wrote: > On Thu, Jun 08, 2017 at 09:07:05PM +0200, Frederic Weisbecker wrote: > > Awesome, these traces have been very helpful! So now I think I get what's going on. > > Can you please test the following fix? > > With the patch, I hit the warning early on boot: > > [ 1.423727] clocksource: Switched to clocksource kvm-clock > [ 1.429326] ------------[ cut here ]------------ > [ 1.430234] WARNING: CPU: 1 PID: 0 at kernel/time/tick-sched.c:792 __tick_nohz_idle_enter+0xe1c/0x15c0 Oh sorry, I inverted some conditional. It warns as soon as the tick is completely stopped and not just deferred, which seldom happen on my testbox. I need to learn programming again. Here is the fixed version: --- >From f80041b5209aaf9d02ac25a29a248d0f214ba19f Mon Sep 17 00:00:00 2001 From: Frederic Weisbecker Date: Thu, 8 Jun 2017 16:32:58 +0200 Subject: [PATCH] nohz: Fix spurious warning when hrtimer and clocksource get out of sync The sanity check ensuring that the tick expiry cache (ts->next_tick) is actually in sync with the hardware clock (dev->next_event) makes the wrong assumption that the clock can't be programmed later than the hrtimer deadline. In fact the clock hardware can be programmed later on some conditions such as: * The hrtimer deadline is already in the past. * The hrtimer deadline is earlier than the minimum delay supported by the hardware. Such conditions can be met when we program the tick, for example if the last jiffies update hasn't been seen by the current CPU yet, we may program the hrtimer to a deadline that is earlier than ktime_get() because last_jiffies_update is our timestamp base to compute the next tick. As a result, we can randomly observe such warning: WARNING: CPU: 5 PID: 0 at kernel/time/tick-sched.c:794 tick_nohz_stop_sched_tick kernel/time/tick-sched.c:791 [inline] Call Trace: tick_nohz_irq_exit tick_irq_exit irq_exit exiting_irq smp_call_function_interrupt smp_call_function_single_interrupt call_function_single_interrupt Therefore, let's rather make sure that the tick expiry cache is sync'ed with the tick hrtimer deadline, against which it is not supposed to drift away. The clock hardware instead has its own will and can't be used as a reliable comparison point. Reported-by: Sasha Levin Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Rik van Riel Cc: James Hartsock Cc: Tim Wright Signed-off-by: Frederic Weisbecker --- kernel/time/tick-sched.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c index 9d31f1e..83c788e 100644 --- a/kernel/time/tick-sched.c +++ b/kernel/time/tick-sched.c @@ -768,7 +768,8 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts, /* Skip reprogram of event if its not changed */ if (ts->tick_stopped && (expires == ts->next_tick)) { /* Sanity check: make sure clockevent is actually programmed */ - if (likely(dev->next_event <= ts->next_tick)) + if (tick == KTIME_MAX || + ts->next_tick == hrtimer_get_expires(&ts->sched_timer)) goto out; WARN_ON_ONCE(1); @@ -806,8 +807,10 @@ static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts, goto out; } + hrtimer_set_expires(&ts->sched_timer, tick); + if (ts->nohz_mode == NOHZ_MODE_HIGHRES) - hrtimer_start(&ts->sched_timer, tick, HRTIMER_MODE_ABS_PINNED); + hrtimer_start_expires(&ts->sched_timer, HRTIMER_MODE_ABS_PINNED); else tick_program_event(tick, 1); out: -- 2.7.4