Date: Mon, 18 Apr 2016 02:32:54 +0100 (BST)
From: "Maciej W. Rozycki" <macro@linux-mips.org>
To: Bob Tracy <rct@gherkin.frus.com>
cc: linux-kernel@vger.kernel.org, debian-alpha@lists.debian.org,
        mcree@orcon.net.nz, jay.estabrook@gmail.com, mattst88@gmail.com
Subject: Re: [BUG] machine check Oops on Alpha
In-Reply-To: <20160417210532.GA27208@gherkin.frus.com>
Message-ID: <alpine.LFD.2.20.1604172254540.28036@eddie.linux-mips.org>
References: <20160417210532.GA27208@gherkin.frus.com>
User-Agent: Alpine 2.20 (LFD 67 2015-01-07)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 3164
Lines: 80

On Sun, 17 Apr 2016, Bob Tracy wrote:

> While a "machine check" is normally indicative of an underlying hardware
> issue, the fact this is a one-time-per-boot issue has me thinking
> otherwise.  I suspect a code path being traversed prior to the Oops that
> gets bypassed afterward.  As previously mentioned, there have been months-
> long intervals in the past where the issue has either been masked or non-
> existent.  Currently, the issue has persisted through several 4.X kernel
> release candidates and releases.

 It may or may not be a hardware issue it would seem, there's this comment 
in `process_mcheck_info':

	/*
	 * See if the machine check is due to a badaddr() and if so,
	 * ignore it.
	 */

> Attached is an example of precisely what I'm talking about as far as a
> "good" Oops.  It occurred within a day of the last reboot, and the
> machine has been running fine since.  Been flogging the devil out of it,
> too: lots of updates (hundreds of megabytes), kernel builds, etc.

 So from this dump it looks like the immediate problem is not the machine 
check itself but rather a null pointer dereference (offset by 0x10, so 
likely a structure member access):

Unable to handle kernel paging request at virtual address 0000000000000010

which happens at:

pc is at process_mcheck_info+0x54/0x370

and the offending instruction is:

	10 00 89 a2 	ldl	a4,16(s0)

and s0 is indeed null.  To me it looks like we're here:

	printk(KERN_CRIT "%s machine check: vector=0x%lx pc=0x%lx code=0x%x\n",
	       machine, vector, get_irq_regs()->pc, mchk_header->code);

(so not a benign MCE after all) trying to fetch `mchk_header->code', which 
means `la_ptr' is null for some reason.  This value is passed down from 
`cia_machine_check', from `do_entInt', and originally comes from PALcode, 
supposed to point to the logout area.

 The SCB vector, still present in a0 it would seem, is 630, which looks 
legitimate, means "Processor correctable machine check" and is used for 
signalling Istream or Dstream correctable ECC errors.  These are dealt 
with IIUC by PALcode before the machine check is dispatched, which would 
explain why, except for the Oops observed, the system continues to operate 
normally.

 So question is whether it's PALcode doing something weird or is it a 
register getting corrupted due to a bug somewhere, either in our code or 
GCC.  Hmm...

 I'd be tempted to run with the patch below to see what's the value of 
`la_ptr' early on in processing (`entInt' code in entry.S looks sane to 
me, doesn't touch a2).  NB a rebuild doesn't have to be costly if you only 
poke at a single file or a few which aren't e.g. headers included from 
everywhere.

  Maciej

diff --git a/arch/alpha/kernel/irq_alpha.c b/arch/alpha/kernel/irq_alpha.c
index 1c8625c..6773bab 100644
--- a/arch/alpha/kernel/irq_alpha.c
+++ b/arch/alpha/kernel/irq_alpha.c
@@ -46,6 +46,9 @@ do_entInt(unsigned long type, unsigned long vector,
 {
 	struct pt_regs *old_regs;
 
+	if (type == 2)
+		printk(KERN_CRIT "machine check: LA: %016lx\n", la_ptr);
+
 	/*
 	 * Disable interrupts during IRQ handling.
 	 * Note that there is no matching local_irq_enable() due to