Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 9 Mar 2022 17:11:40 +0000
From:   "Russell King (Oracle)" <linux@armlinux.org.uk>
To:     Naresh Kamboju <naresh.kamboju@linaro.org>
Cc:     Ard Biesheuvel <ardb@kernel.org>,
        open list <linux-kernel@vger.kernel.org>,
        Linux-Next Mailing List <linux-next@vger.kernel.org>,
        Linux ARM <linux-arm-kernel@lists.infradead.org>,
        Linus Walleij <linus.walleij@linaro.org>,
        Arnd Bergmann <arnd@arndb.de>,
        Corentin Labbe <clabbe.montjoie@gmail.com>,
        Stephen Rothwell <sfr@canb.auug.org.au>
Subject: Re: [next] arm: Internal error: Oops: 5 PC is at
 __read_once_word_nocheck
Message-ID: <YijfzAhWAgfGGqCO@shell.armlinux.org.uk>
References: <CA+G9fYtpy8VgK+ag6OsA9TDrwi5YGU4hu7GM8xwpO7v6LrCD4Q@mail.gmail.com>
 <YiiDZ7jjG38gqP+Q@shell.armlinux.org.uk>
 <CAMj1kXHTdk1Abm7ShoZzrW6EpM9eyFMPSdaa58Ziie4ZMecCnQ@mail.gmail.com>
 <CA+G9fYvCvBBi+dZ+CnUy=ZK6GhCFhBw72_==Cav=Q8QP5T1r5w@mail.gmail.com>
 <CA+G9fYt73AYs=z-BeZh22RBp==sf73pKky6m4iPSH7a4FssK7w@mail.gmail.com>
 <CAMj1kXEFZVeWLaRQJmwO+Nn6uW4q6vXJOaNNTVKju1p2bMQksA@mail.gmail.com>
 <YijCkHHhpleeADAO@shell.armlinux.org.uk>
 <CA+G9fYtjrAMg8TykZdRyZEfRthyqom_73x87F-60C_QFeEL8Bg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CA+G9fYtjrAMg8TykZdRyZEfRthyqom_73x87F-60C_QFeEL8Bg@mail.gmail.com>
Sender: Russell King (Oracle) <linux@armlinux.org.uk>
Precedence: bulk

On Wed, Mar 09, 2022 at 10:08:25PM +0530, Naresh Kamboju wrote:
> Hi Russell,
> 
> On Wed, 9 Mar 2022 at 20:37, Russell King (Oracle)
> <linux@armlinux.org.uk> wrote:
> >
> > On Wed, Mar 09, 2022 at 03:57:32PM +0100, Ard Biesheuvel wrote:
> > > On Wed, 9 Mar 2022 at 15:44, Naresh Kamboju <naresh.kamboju@linaro.org> wrote:
> > > >
> <trim>
> > Well, we unwound until:
> >
> >  __irq_svc from migrate_disable+0x0/0x70
> >
> > and then crashed - and the key thing there is that we're at the start
> > of migrate_disable() when we took an interrupt.
> >
> > For some reason, this triggers an access to address 0x10, which faults.
> > We then try unwinding again, and successfully unwind all the way back
> > to the same point (the line above) which then causes the unwinder to
> > again access address 0x10, and the cycle repeats with the stack
> > growing bigger and bigger.
> >
> > I'd suggest also testing without the revert but with my patch.
> 
> I have tested your patch on top of linux next-20220309 and still see kernel
> crash as below [1]. build link [2].
> 
> [   26.812060] 8<--- cut here ---
> [   26.813459] Unhandled fault: page domain fault (0x01b) at 0xb6a3ab70
> [   26.816139] [b6a3ab70] *pgd=fb28a835
> [   26.817770] Internal error: : 1b [#1] SMP ARM
> [   26.819636] Modules linked in:
> [   26.820956] CPU: 0 PID: 211 Comm: haveged Not tainted
> 5.17.0-rc7-next-20220309 #1
> [   26.824519] Hardware name: Generic DT based system
> [   26.827148] PC is at __read_once_word_nocheck+0x0/0x8
> [   26.829856] LR is at unwind_frame+0x7dc/0xab4
> 
> - Naresh
> 
> [1] https://lkft.validation.linaro.org/scheduler/job/4688599#L596
> [2] https://builds.tuxbuild.com/269gYLGuAdmltuLhIUDAjS2fg1Q/

I think the problem has just moved:

[   27.113085]  __irq_svc from __copy_to_user_std+0x24/0x378

The code at the start of __copy_to_user_std is:

   0:   e3a034bf        mov     r3, #-1090519040        ; 0xbf000000
   4:   e243c001        sub     ip, r3, #1
   8:   e05cc000        subs    ip, ip, r0
   c:   228cc001        addcs   ip, ip, #1
  10:   205cc002        subscs  ip, ip, r2
  14:   33a00000        movcc   r0, #0
  18:   e320f014        csdb
  1c:   e3a03000        mov     r3, #0
  20:   e92d481d        push    {r0, r2, r3, r4, fp, lr}
  24:   e1a0b00d        mov     fp, sp

and the unwind information will be:

0xc056f14c <arm_copy_to_user+0x1c>: @0xc0b89b84
  Compact model index: 1
  0x9b      vsp = r11
  0xb1 0x0d pop {r0, r2, r3}
  0x84 0x81 pop {r4, r11, r14}
  0xb0      finish

The problem is that the unwind information says "starting at offset
0x1c, to unwind do the following operations". The first of which is
to move r11 (fp) to the stack pointer. However, r11 isn't setup
until function offset 0x24. You've hit that instruction, which hasn't
executed yet, but the stack has been modified by pushing r0, r2-r4,
fp and lr onto it.

Given this, there is no way that the unwinder (as it currently stands)
can do its job properly between 0x1c and 0x24.

I don't think this is specifically caused by Ard's patches, but by
the addition of KASAN, which has the effect of calling the unwinder
at random points in the kernel (when an interrupt happens) and it's
clear from the above that there are windows in the code where, if
we attempt to unwind using the unwind information, we faill fail
because the program state is not consistent with the unwind
information.

Ard's patch that changes:

	ctrl->vrs[reg] = READ_ONCE_NOCHECK(*(*vsp));

to use get_kernel_nofault() should have the effect of protecting
against the oops, but the side effect is that it is fundamentally not
possible with the way these things are to unwind at these points -
which means its not possible to get a stacktrace there.

So, I don't think this is a "new" problem, but a weakness of using
the unwinder to get a backtrace for KASAN.

Do you have any way to work out exactly when this problem first
appeared?

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!