Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <CAD4b4WK9W+dNZZ_WPt-9ZNpSHvyxdwPW86Rtq4AKOONuvyR37A@mail.gmail.com>
 <20200529131452.pgj7mx7xaz7n4kz3@linutronix.de>
In-Reply-To: <20200529131452.pgj7mx7xaz7n4kz3@linutronix.de>
From:   Mark Marshall <markmarshall14@gmail.com>
Date:   Fri, 29 May 2020 17:38:39 +0200
Message-ID: <CAD4b4WLS36JKepz31m98Z3Ve05d63GnfBGsuuWRXpjXZqPoBrA@mail.gmail.com>
Subject: Re: Kernel crash due to memory corruption with v5.4.26-rt17 and
 PowerPC e500
To:     Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc:     linux-rt-users <linux-rt-users@vger.kernel.org>,
        Mark Marshall <mark.marshall@omicronenergy.com>,
        thomas.graziadei@omicronenergy.com,
        Thomas Gleixner <tglx@linutronix.de>,
        linux-kernel@vger.kernel.org, rostedt@goodmis.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

Hi Sebastian & list,

I had assumed that my e-mail had got lost or overlooked, I was meaning to
post a follow up message this week...

All I could find from the debugging and tracing that we added was that
something was going wrong with the mm data structures somewhere in the
exec code.  In the end I just spent a week or two pouring over the diffs
of this code between the versions that I new worked and didn't work.

I eventually found the culprit.  On the working kernel versions there is
a patch called "mm: Protect activate_mm() by preempt_[disable&enable]_rt()"=
.
This is commit f0b4a9cb253a on the V4.19.82-rt30 branch, for instance.
Although the commit message talks about ARM, it seems that we need this for
PowerPC too (I guess, any PowerPC with the "nohash" MMU?).

Could you please add this commit back to the RT branch?  I'm not sure how
to find out the history of this commit.  For instance, why has it been
removed from the RT patchset?  How are these things tracked, generally?

Best regards,
Mark

On Fri, 29 May 2020 at 15:14, Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
>
> On 2020-05-04 11:40:08 [+0200], Mark Marshall wrote:
> > The easiest way we have found to reproduce the crash is to repeatedly
> > insert and then remove a module.  The crash then appears to be related
> > to either paging in the module or in exiting the mdev process.  (The
> > crash does also happen at other times, but it is hard to reproduce
> > reliably then).  This simple script will almost always crash:
> >
> >    for i in $(seq 1000) ; do echo $i ; modprobe crc7 ; rmmod crc7 ; don=
e
>
> So I tried that on 5.6.14-rt7 with the qemu version of e500 (the SMP and
> UP version). No luck. I don't have anything with real hardware.
> Could you share the .config in case this is related?
>
> > (The crc7 module is chosen as it is small and simple.  Any module will
> > work / crash).
> >
> > We have tried kernels v5.0, v5.2 and v5.6.  The v5.0 and v5.2 kernels
> > do not show the problem.  The v5.6 kernel does show the problem.
> > Switching of RT fixes the problem.
> >
> > I have reduced the functionality in the kernel to a bare minimum
> > (removing networking, USB and PCI, as we have some out-of-tree patches
> > in those areas) and we still get the crash.
> =E2=80=A6
> > I have added some debugging code where the mm_struct and
> > vma_area_struct have "poision" values at the start and the end, and
> > this seems to show that the vma_area_struct is getting corrupted, but
> > I'm not able to see where.
>
> oh.
>
> > We have switched on all of the debugging that we can, including
> > KASAN, and this shows nothing.
> >
> >
> > Can anyone help us?  What can we try next?  Is anyone using the e500
> > with the RT kernel?  Does anyone have any idea how to debug problems
> > related to the error message "Bad rss-counter state"?
> >
> > Any help or advice would be most gratefully received.
>
> I don't have any ideas. You could try to apply only a part of the RT
> patch and see if it problem is still there. If you are lucky you find
> the patch that introduces the problem. If not, the problem appears with
> the RT switch=E2=80=A6
>
> > Many thanks,
> > Mark Marshall and Thomas Graziadei
>
> Sebastian