Date: Sat, 23 Apr 2016 19:52:46 +0100
From: "Dr. David Alan Gilbert" <dave@treblig.org>
To: Marc Haber <mh+linux-kernel@zugschlus.de>
Cc: Borislav Petkov <bp@alien8.de>, Paolo Bonzini <pbonzini@redhat.com>,
        linux-kernel@vger.kernel.org, kvm ML <kvm@vger.kernel.org>
Subject: Re: Major KVM issues with kernel 4.5 on the host
Message-ID: <20160423185246.GC8376@gallifrey>
References: <20160413222942.GD7600@torres.zugschlus.de>
 <570EEF6D.40307@redhat.com>
 <20160414052220.GE7600@torres.zugschlus.de>
 <20160421083948.GF21755@torres.zugschlus.de>
 <20160421123711.GD28821@pd.tnic>
 <20160421145005.GI21755@torres.zugschlus.de>
 <20160421165106.GK28821@pd.tnic>
 <20160421200433.GL21755@torres.zugschlus.de>
 <20160423160429.GL8531@pd.tnic>
 <20160423184341.GA21755@torres.zugschlus.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160423184341.GA21755@torres.zugschlus.de>
User-Agent: Mutt/1.5.21 (2010-09-15)
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 3251
Lines: 76

* Marc Haber (mh+linux-kernel@zugschlus.de) wrote:
> On Sat, Apr 23, 2016 at 06:04:29PM +0200, Borislav Petkov wrote:
> > On Thu, Apr 21, 2016 at 10:04:33PM +0200, Marc Haber wrote:
> > > Yes, but there are two symptoms. The VM either suffers file system
> > > issues (garbage read from files, or an aborted ext4 journal and
> > > following ro remount) or it stops dead in its tracks.
> > 
> > Stops dead? What does that mean exactly? Box is wedged solid and it
> > doesn't react to any key presses?
> 
> No ping, no reaction on serial console, no reaction on virtual
> console, no syslog entries.
> 
> > Because if so, this could really be a DRAM going bad and a correctable
> > error turning into an uncorrectable. How old is the DRAM in that box?
> > Judging by your CPU, it should be a couple of years...
> 
> Uncorrectable errors would still be identified by the ECC hardware,
> and the box wouldn't be perfectly fine with an "old" kernel.

Hmm, your problem does sound like bad hardware, but....
If you've got a nice reliable crash, can you try turning transparent huge pages
off on the host;
   echo never > /sys/kernel/mm/transparent_hugepage/enabled

Dave

> > > The box reports about one correctable error per week, so I probably
> > > have a faulty DIMM, but since the issue only surfaces in VMs while the
> > > host system is in perfect working order...
> > 
> > So it could be that correctable error turns into an uncorrectable one at
> > some point. But then you should be getting an exception...
> 
> Yes, that would be in the logs.
> 
> > > And yes, I am pondering to simply replace the box with an Intel CPU.
> > 
> > Your CPU is fine, from what I've seen so far.
> 
> But we still postulate that the issue does only show on older AMD
> CPUs. Otherwise, I wouldn't be the only one making this experience.
> 
> > > I go the way of Debian packages since it is easier to handle the
> > > crypto file systems when the machine is booting up.
> > 
> > As long as you're testing the correct bisection kernels...
> 
> I am reasonably sure about that, yes.
> 
> > > And yes, I think about doing a test reinstall on unencrypted disk to
> > > find out whether encryption plays a role, but I currently need the
> > > machine to urgently to take it out of serice for half a month, and,
> > > again, the host system is in perfect working order, it is just VMs
> > > that barf.
> > 
> > Yeah, I can't reproduce it here and I have a very similar box to yours
> > which is otherwise idle, more or less.
> > 
> > Another fact which points to potentially DIMM going bad...
> 
> Do you want me to memtest for 24 hours?
> 
> Greetings
> Marc
> 
> -- 
> -----------------------------------------------------------------------------
> Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
> Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
> Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421
-- 
 -----Open up your eyes, open up your mind, open up your code -------   
/ Dr. David Alan Gilbert    |       Running GNU/Linux       | Happy  \ 
\        dave @ treblig.org |                               | In Hex /
 \ _________________________|_____ http://www.treblig.org   |_______/