From: "Liang, Kan" <kan.liang@intel.com>
To: "'Don Zickus'" <dzickus@redhat.com>,
        "'Thomas Gleixner'" <tglx@linutronix.de>
CC: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "mingo@kernel.org" <mingo@kernel.org>,
        "akpm@linux-foundation.org" <akpm@linux-foundation.org>,
        "babu.moger@oracle.com" <babu.moger@oracle.com>,
        "atomlin@redhat.com" <atomlin@redhat.com>,
        "prarit@redhat.com" <prarit@redhat.com>,
        "torvalds@linux-foundation.org" <torvalds@linux-foundation.org>,
        "peterz@infradead.org" <peterz@infradead.org>,
        "eranian@google.com" <eranian@google.com>,
        "acme@redhat.com" <acme@redhat.com>,
        "ak@linux.intel.com" <ak@linux.intel.com>,
        "stable@vger.kernel.org" <stable@vger.kernel.org>
Subject: RE: [PATCH V2] kernel/watchdog: fix spurious hard lockups
Thread-Topic: [PATCH V2] kernel/watchdog: fix spurious hard lockups
Thread-Index: AQHS6pyX93nZMscYGUu+rlSjZMkkxKIvVm6AgAErMwCAARD/gIAAjbWAgABZxoCABJ2VAIABkHmAgB6zMiCAAGBLgIAc+e0w
Date: Tue, 15 Aug 2017 01:16:51 +0000
Message-ID: <37D7C6CF3E00A74B8858931C1DB2F07753784A2B@SHSMSX103.ccr.corp.intel.com>
References: <20170621144118.5939-1-kan.liang@intel.com>
 <alpine.DEB.2.20.1706212235550.2152@nanos>
 <20170622154450.2lua7fdmigcixldw@redhat.com>
 <alpine.DEB.2.20.1706230952190.2647@nanos>
 <20170623162907.l6inpxgztwwkeaoi@redhat.com>
 <alpine.DEB.2.20.1706232348020.2234@nanos>
 <20170626201927.3ak7fk3yvdzbb4ay@redhat.com>
 <20170627201249.ll34ecwhpme3vh2u@redhat.com>
 <37D7C6CF3E00A74B8858931C1DB2F0775371D43E@SHSMSX103.ccr.corp.intel.com>
 <20170717144637.34umykrccvjma3fl@redhat.com>
In-Reply-To: <20170717144637.34umykrccvjma3fl@redhat.com>
Accept-Language: zh-CN, en-US
Content-Language: en-US
dlp-product: dlpe-windows
dlp-version: 10.0.102.7
dlp-reaction: no-action
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 8BIT
MIME-Version: 1.0
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 2213
Lines: 61


> On Mon, Jul 17, 2017 at 01:24:23AM +0000, Liang, Kan wrote:
> > Hi Don & Thomas,
> >
> > Sorry for the late response. We just finished the tests for all proposed
> patches.
> >
> > There are three proposed patches so far.
> > Patch 1: The patch as above which speed up the hrtimer.
> > Patch 2: Thomas's first proposal.
> > https://patchwork.kernel.org/patch/9803033/
> > https://patchwork.kernel.org/patch/9805903/
> > Patch 3: my original proposal which increase the NMI watchdog timeout
> > by 3X https://patchwork.kernel.org/patch/9802053/
> >
> > According to our test, only patch 3 works well.
> > The other two patches will hang the system eventually.
> > For patch 1, the system hang after running our test case for ~1 hour.
> > For patch 2, the system hang in running the overnight test.
> > There is no error message shown when the system hang. So I don't know
> > the root cause yet.
> 
> Hi Kan,
> 
> Thanks for the feedback.  Odd that the different patches had different results.
> What is more odd to me is the hang.  I thought these were all false lockups
> that prematurely panic'd and rebooted the box.
> 
> Is the machine configured to panic on hardlockup and reboot?  Perhaps
> kdump is enabled to store the console log for review upon reboot?
> 
> It almost implies that a hardlockup did happen but isnt' being detected until
> later??
> >
> > BTW: We set 1 to watchdog_thresh when we did the test.
> > It's believed that can speed up the failure.
> 
> Sure, you/they look for 1 second hangs instead of 10 second ones.  But with
> patch3 it is more like 3 seconds'ish vs 30 second'ish.
> 
> As Thomas asked, I would also be interested in the way the test works.  The
> hang doesn't make sense.
> 

Hi Don and Thomas, 

Sorry for the late response.

We have confirmed that the hardlock with "speed up the hrtimer" patch is
actually another issue. Tim has already proposed a patch to fix it. 
Here is his patch. https://lkml.org/lkml/2017/8/14/1000

This patch which speed up the hrtimer (https://lkml.org/lkml/2017/6/26/685)
is decent to fix the spurious hard lockups.
Tested-by: Kan Liang <kan.liang@intel.com>

Please consider to merge it into both mainline and stable tree.

Thanks,
Kan