Date: Sat, 23 Apr 2016 20:43:41 +0200
From: Marc Haber <mh+linux-kernel@zugschlus.de>
To: Borislav Petkov <bp@alien8.de>
Cc: Paolo Bonzini <pbonzini@redhat.com>, linux-kernel@vger.kernel.org,
        kvm ML <kvm@vger.kernel.org>
Subject: Re: Major KVM issues with kernel 4.5 on the host
Message-ID: <20160423184341.GA21755@torres.zugschlus.de>
References: <570EADD2.8030300@redhat.com>
 <20160413222942.GD7600@torres.zugschlus.de>
 <570EEF6D.40307@redhat.com>
 <20160414052220.GE7600@torres.zugschlus.de>
 <20160421083948.GF21755@torres.zugschlus.de>
 <20160421123711.GD28821@pd.tnic>
 <20160421145005.GI21755@torres.zugschlus.de>
 <20160421165106.GK28821@pd.tnic>
 <20160421200433.GL21755@torres.zugschlus.de>
 <20160423160429.GL8531@pd.tnic>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
In-Reply-To: <20160423160429.GL8531@pd.tnic>
User-Agent: Mutt/1.5.23 (2014-03-12)
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 2566
Lines: 63

On Sat, Apr 23, 2016 at 06:04:29PM +0200, Borislav Petkov wrote:
> On Thu, Apr 21, 2016 at 10:04:33PM +0200, Marc Haber wrote:
> > Yes, but there are two symptoms. The VM either suffers file system
> > issues (garbage read from files, or an aborted ext4 journal and
> > following ro remount) or it stops dead in its tracks.
> 
> Stops dead? What does that mean exactly? Box is wedged solid and it
> doesn't react to any key presses?

No ping, no reaction on serial console, no reaction on virtual
console, no syslog entries.

> Because if so, this could really be a DRAM going bad and a correctable
> error turning into an uncorrectable. How old is the DRAM in that box?
> Judging by your CPU, it should be a couple of years...

Uncorrectable errors would still be identified by the ECC hardware,
and the box wouldn't be perfectly fine with an "old" kernel.

> > The box reports about one correctable error per week, so I probably
> > have a faulty DIMM, but since the issue only surfaces in VMs while the
> > host system is in perfect working order...
> 
> So it could be that correctable error turns into an uncorrectable one at
> some point. But then you should be getting an exception...

Yes, that would be in the logs.

> > And yes, I am pondering to simply replace the box with an Intel CPU.
> 
> Your CPU is fine, from what I've seen so far.

But we still postulate that the issue does only show on older AMD
CPUs. Otherwise, I wouldn't be the only one making this experience.

> > I go the way of Debian packages since it is easier to handle the
> > crypto file systems when the machine is booting up.
> 
> As long as you're testing the correct bisection kernels...

I am reasonably sure about that, yes.

> > And yes, I think about doing a test reinstall on unencrypted disk to
> > find out whether encryption plays a role, but I currently need the
> > machine to urgently to take it out of serice for half a month, and,
> > again, the host system is in perfect working order, it is just VMs
> > that barf.
> 
> Yeah, I can't reproduce it here and I have a very similar box to yours
> which is otherwise idle, more or less.
> 
> Another fact which points to potentially DIMM going bad...

Do you want me to memtest for 24 hours?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421