DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 154B7C0587EA
Date: Mon, 17 Jul 2017 10:46:37 -0400
From: Don Zickus <dzickus@redhat.com>
To: "Liang, Kan" <kan.liang@intel.com>
Cc: Thomas Gleixner <tglx@linutronix.de>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "mingo@kernel.org" <mingo@kernel.org>,
        "akpm@linux-foundation.org" <akpm@linux-foundation.org>,
        "babu.moger@oracle.com" <babu.moger@oracle.com>,
        "atomlin@redhat.com" <atomlin@redhat.com>,
        "prarit@redhat.com" <prarit@redhat.com>,
        "torvalds@linux-foundation.org" <torvalds@linux-foundation.org>,
        "peterz@infradead.org" <peterz@infradead.org>,
        "eranian@google.com" <eranian@google.com>,
        "acme@redhat.com" <acme@redhat.com>,
        "ak@linux.intel.com" <ak@linux.intel.com>,
        "stable@vger.kernel.org" <stable@vger.kernel.org>
Subject: Re: [PATCH V2] kernel/watchdog: fix spurious hard lockups
Message-ID: <20170717144637.34umykrccvjma3fl@redhat.com>
References: <20170621144118.5939-1-kan.liang@intel.com>
 <alpine.DEB.2.20.1706212235550.2152@nanos>
 <20170622154450.2lua7fdmigcixldw@redhat.com>
 <alpine.DEB.2.20.1706230952190.2647@nanos>
 <20170623162907.l6inpxgztwwkeaoi@redhat.com>
 <alpine.DEB.2.20.1706232348020.2234@nanos>
 <20170626201927.3ak7fk3yvdzbb4ay@redhat.com>
 <20170627201249.ll34ecwhpme3vh2u@redhat.com>
 <37D7C6CF3E00A74B8858931C1DB2F0775371D43E@SHSMSX103.ccr.corp.intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <37D7C6CF3E00A74B8858931C1DB2F0775371D43E@SHSMSX103.ccr.corp.intel.com>
User-Agent: NeoMutt/20170428-dirty (1.8.2)
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 1652
Lines: 43

On Mon, Jul 17, 2017 at 01:24:23AM +0000, Liang, Kan wrote:
> Hi Don & Thomas,
> 
> Sorry for the late response. We just finished the tests for all proposed patches.
> 
> There are three proposed patches so far.
> Patch 1: The patch as above which speed up the hrtimer.
> Patch 2: Thomas's first proposal.
> https://patchwork.kernel.org/patch/9803033/
> https://patchwork.kernel.org/patch/9805903/
> Patch 3: my original proposal which increase the NMI watchdog timeout by 3X
> https://patchwork.kernel.org/patch/9802053/
> 
> According to our test, only patch 3 works well.
> The other two patches will hang the system eventually.
> For patch 1, the system hang after running our test case for ~1 hour.
> For patch 2, the system hang in running the overnight test.
> There is no error message shown when the system hang. So I don't know the
> root cause yet.

Hi Kan,

Thanks for the feedback.  Odd that the different patches had different
results.  What is more odd to me is the hang.  I thought these were all
false lockups that prematurely panic'd and rebooted the box.

Is the machine configured to panic on hardlockup and reboot?  Perhaps kdump
is enabled to store the console log for review upon reboot?

It almost implies that a hardlockup did happen but isnt' being detected
until later??
> 
> BTW: We set 1 to watchdog_thresh when we did the test.
> It's believed that can speed up the failure.

Sure, you/they look for 1 second hangs instead of 10 second ones.  But with
patch3 it is more like 3 seconds'ish vs 30 second'ish.

As Thomas asked, I would also be interested in the way the test works.  The
hang doesn't make sense.

Cheers,
Don