LinuxLists.cc - [next] arm: Internal error: Oops: 5 PC is at __read_once_word

[permalink] [raw]

Subject: Re: [next] arm: Internal error: Oops: 5 PC is at __read_once_word_nocheck

On Wed, Mar 09, 2022 at 03:18:12PM +0530, Naresh Kamboju wrote:
> While boting linux next-20220308 on BeagleBoard-X15 and qemu arm the following
> kernel crash reported which is CONFIG_KASAN enabled build [1] & [2].

The unwinder is currently broken in linux-next. Please try reverting
532319b9c418 ("ARM: unwind: disregard unwind info before stack frame is
set up")

Thanks.

--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

2022-03-09 16:09:55

[permalink] [raw]

Subject: Re: [next] arm: Internal error: Oops: 5 PC is at __read_once_word_nocheck

On Wed, 9 Mar 2022 at 16:07, Russell King (Oracle)
<[email protected]> wrote:
>
> On Wed, Mar 09, 2022 at 03:57:32PM +0100, Ard Biesheuvel wrote:
> > On Wed, 9 Mar 2022 at 15:44, Naresh Kamboju <[email protected]> wrote:
> > >
> > > On Wed, 9 Mar 2022 at 19:37, Naresh Kamboju <[email protected]> wrote:
> > > >
> > > > On Wed, 9 Mar 2022 at 16:16, Ard Biesheuvel <[email protected]> wrote:
> > > > >
> > > > > On Wed, 9 Mar 2022 at 11:37, Russell King (Oracle)
> > > > > <[email protected]> wrote:
> > > > > >
> > > > > > On Wed, Mar 09, 2022 at 03:18:12PM +0530, Naresh Kamboju wrote:
> > > > > > > While boting linux next-20220308 on BeagleBoard-X15 and qemu arm the following
> > > > > > > kernel crash reported which is CONFIG_KASAN enabled build [1] & [2].
> > > > > >
> > > > > > The unwinder is currently broken in linux-next. Please try reverting
> > > > > > 532319b9c418 ("ARM: unwind: disregard unwind info before stack frame is
> > > > > > set up")
> > >
> > > I have reverted the suggested commit and built and boot failed due to reported
> > > kernel crash [1].
> > >
> > > - Naresh
> > >
> >
> > Thanks Naresh,
> >
> > This looks like it might be related to the issue Russell just sent a fix for:
> > https://lore.kernel.org/linux-arm-kernel/CAMj1kXEqp2UmsyUe1eWErtpMk3dGEFZyyno3nqydC_ML0bwTLw@mail.gmail.com/T/#t
> >
> > Could you please try that?
>
> Well, we unwound until:
>
> __irq_svc from migrate_disable+0x0/0x70
>
> and then crashed - and the key thing there is that we're at the start
> of migrate_disable() when we took an interrupt.
>
> For some reason, this triggers an access to address 0x10, which faults.
> We then try unwinding again, and successfully unwind all the way back
> to the same point (the line above) which then causes the unwinder to
> again access address 0x10, and the cycle repeats with the stack
> growing bigger and bigger.
>
> I'd suggest also testing without the revert but with my patch.
>

Indeed.

And as I suggested the other day, maybe it wouldn't be so bad to
harden the vsp dereference, like below:

--- a/arch/arm/kernel/unwind.c
+++ b/arch/arm/kernel/unwind.c
@@ -27,6 +27,7 @@
#include <linux/sched.h>
#include <linux/slab.h>
#include <linux/spinlock.h>
+#include <linux/uaccess.h>
#include <linux/list.h>

#include <asm/sections.h>
@@ -236,10 +237,11 @@ static int unwind_pop_register(struct
unwind_ctrl_block *ctrl,
if (*vsp >= (unsigned long *)ctrl->sp_high)
return -URC_FAILURE;

- /* Use READ_ONCE_NOCHECK here to avoid this memory access
- * from being tracked by KASAN.
+ /* Use get_kernel_nofault() here to avoid this memory access
+ * from causing a fatal fault, and from being tracked by KASAN.
*/
- ctrl->vrs[reg] = READ_ONCE_NOCHECK(*(*vsp));
+ if (get_kernel_nofault(ctrl->vrs[reg], *vsp))
+ return -URC_FAILURE;
if (reg == 14)
ctrl->lr_addr = *vsp;
(*vsp)++;

2022-03-09 16:16:02

[permalink] [raw]

Subject: Re: [next] arm: Internal error: Oops: 5 PC is at __read_once_word_nocheck

On Wed, Mar 09, 2022 at 03:57:32PM +0100, Ard Biesheuvel wrote:
> On Wed, 9 Mar 2022 at 15:44, Naresh Kamboju <[email protected]> wrote:
> >
> > On Wed, 9 Mar 2022 at 19:37, Naresh Kamboju <[email protected]> wrote:
> > >
> > > On Wed, 9 Mar 2022 at 16:16, Ard Biesheuvel <[email protected]> wrote:
> > > >
> > > > On Wed, 9 Mar 2022 at 11:37, Russell King (Oracle)
> > > > <[email protected]> wrote:
> > > > >
> > > > > On Wed, Mar 09, 2022 at 03:18:12PM +0530, Naresh Kamboju wrote:
> > > > > > While boting linux next-20220308 on BeagleBoard-X15 and qemu arm the following
> > > > > > kernel crash reported which is CONFIG_KASAN enabled build [1] & [2].
> > > > >
> > > > > The unwinder is currently broken in linux-next. Please try reverting
> > > > > 532319b9c418 ("ARM: unwind: disregard unwind info before stack frame is
> > > > > set up")
> >
> > I have reverted the suggested commit and built and boot failed due to reported
> > kernel crash [1].
> >
> > - Naresh
> >
>
> Thanks Naresh,
>
> This looks like it might be related to the issue Russell just sent a fix for:
> https://lore.kernel.org/linux-arm-kernel/CAMj1kXEqp2UmsyUe1eWErtpMk3dGEFZyyno3nqydC_ML0bwTLw@mail.gmail.com/T/#t
>
> Could you please try that?

Well, we unwound until:

__irq_svc from migrate_disable+0x0/0x70

and then crashed - and the key thing there is that we're at the start
of migrate_disable() when we took an interrupt.

For some reason, this triggers an access to address 0x10, which faults.
We then try unwinding again, and successfully unwind all the way back
to the same point (the line above) which then causes the unwinder to
again access address 0x10, and the cycle repeats with the stack
growing bigger and bigger.

I'd suggest also testing without the revert but with my patch.

Thanks.

--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

2022-03-09 17:15:30

[permalink] [raw]

Subject: Re: [next] arm: Internal error: Oops: 5 PC is at __read_once_word_nocheck

Hi Russell,

On Wed, 9 Mar 2022 at 20:37, Russell King (Oracle)
<[email protected]> wrote:
>
> On Wed, Mar 09, 2022 at 03:57:32PM +0100, Ard Biesheuvel wrote:
> > On Wed, 9 Mar 2022 at 15:44, Naresh Kamboju <[email protected]> wrote:
> > >
<trim>
> Well, we unwound until:
>
> __irq_svc from migrate_disable+0x0/0x70
>
> and then crashed - and the key thing there is that we're at the start
> of migrate_disable() when we took an interrupt.
>
> For some reason, this triggers an access to address 0x10, which faults.
> We then try unwinding again, and successfully unwind all the way back
> to the same point (the line above) which then causes the unwinder to
> again access address 0x10, and the cycle repeats with the stack
> growing bigger and bigger.
>
> I'd suggest also testing without the revert but with my patch.

I have tested your patch on top of linux next-20220309 and still see kernel
crash as below [1]. build link [2].

[ 26.812060] 8<--- cut here ---
[ 26.813459] Unhandled fault: page domain fault (0x01b) at 0xb6a3ab70
[ 26.816139] [b6a3ab70] *pgd=fb28a835
[ 26.817770] Internal error: : 1b [#1] SMP ARM
[ 26.819636] Modules linked in:
[ 26.820956] CPU: 0 PID: 211 Comm: haveged Not tainted
5.17.0-rc7-next-20220309 #1
[ 26.824519] Hardware name: Generic DT based system
[ 26.827148] PC is at __read_once_word_nocheck+0x0/0x8
[ 26.829856] LR is at unwind_frame+0x7dc/0xab4

- Naresh

[1] https://lkft.validation.linaro.org/scheduler/job/4688599#L596
[2] https://builds.tuxbuild.com/269gYLGuAdmltuLhIUDAjS2fg1Q/

2022-03-09 20:58:15

[permalink] [raw]

Subject: Re: [next] arm: Internal error: Oops: 5 PC is at __read_once_word_nocheck

On Wed, 9 Mar 2022 at 20:39, Russell King (Oracle)
<[email protected]> wrote:
>
> On Wed, Mar 09, 2022 at 08:14:30PM +0100, Ard Biesheuvel wrote:
> > On Wed, 9 Mar 2022 at 19:48, Russell King (Oracle)
> > <[email protected]> wrote:
> > >
> > > On Wed, Mar 09, 2022 at 06:43:42PM +0100, Ard Biesheuvel wrote:
> > > > On Wed, 9 Mar 2022 at 18:11, Russell King (Oracle)
> > > > <[email protected]> wrote:
> > > > >
> > > > > On Wed, Mar 09, 2022 at 10:08:25PM +0530, Naresh Kamboju wrote:
> > > > > > Hi Russell,
> > > > > >
> > > > > > On Wed, 9 Mar 2022 at 20:37, Russell King (Oracle)
> > > > > > <[email protected]> wrote:
> > > > > > >
> > > > > > > On Wed, Mar 09, 2022 at 03:57:32PM +0100, Ard Biesheuvel wrote:
> > > > > > > > On Wed, 9 Mar 2022 at 15:44, Naresh Kamboju <[email protected]> wrote:
> > > > > > > > >
> > > > > > <trim>
> > > > > > > Well, we unwound until:
> > > > > > >
> > > > > > > __irq_svc from migrate_disable+0x0/0x70
> > > > > > >
> > > > > > > and then crashed - and the key thing there is that we're at the start
> > > > > > > of migrate_disable() when we took an interrupt.
> > > > > > >
> > > > > > > For some reason, this triggers an access to address 0x10, which faults.
> > > > > > > We then try unwinding again, and successfully unwind all the way back
> > > > > > > to the same point (the line above) which then causes the unwinder to
> > > > > > > again access address 0x10, and the cycle repeats with the stack
> > > > > > > growing bigger and bigger.
> > > > > > >
> > > > > > > I'd suggest also testing without the revert but with my patch.
> > > > > >
> > > > > > I have tested your patch on top of linux next-20220309 and still see kernel
> > > > > > crash as below [1]. build link [2].
> > > > > >
> > > > > > [ 26.812060] 8<--- cut here ---
> > > > > > [ 26.813459] Unhandled fault: page domain fault (0x01b) at 0xb6a3ab70
> > > > > > [ 26.816139] [b6a3ab70] *pgd=fb28a835
> > > > > > [ 26.817770] Internal error: : 1b [#1] SMP ARM
> > > > > > [ 26.819636] Modules linked in:
> > > > > > [ 26.820956] CPU: 0 PID: 211 Comm: haveged Not tainted
> > > > > > 5.17.0-rc7-next-20220309 #1
> > > > > > [ 26.824519] Hardware name: Generic DT based system
> > > > > > [ 26.827148] PC is at __read_once_word_nocheck+0x0/0x8
> > > > > > [ 26.829856] LR is at unwind_frame+0x7dc/0xab4
> > > > > >
> > > > > > - Naresh
> > > > > >
> > > > > > [1] https://lkft.validation.linaro.org/scheduler/job/4688599#L596
> > > > > > [2] https://builds.tuxbuild.com/269gYLGuAdmltuLhIUDAjS2fg1Q/
> > > > >
> > > > > I think the problem has just moved:
> > > > >
> > > > > [ 27.113085] __irq_svc from __copy_to_user_std+0x24/0x378
> > > > >
> > > > > The code at the start of __copy_to_user_std is:
> > > > >
> > > > > 0: e3a034bf mov r3, #-1090519040 ; 0xbf000000
> > > > > 4: e243c001 sub ip, r3, #1
> > > > > 8: e05cc000 subs ip, ip, r0
> > > > > c: 228cc001 addcs ip, ip, #1
> > > > > 10: 205cc002 subscs ip, ip, r2
> > > > > 14: 33a00000 movcc r0, #0
> > > > > 18: e320f014 csdb
> > > > > 1c: e3a03000 mov r3, #0
> > > > > 20: e92d481d push {r0, r2, r3, r4, fp, lr}
> > > > > 24: e1a0b00d mov fp, sp
> > > > >
> > > > > and the unwind information will be:
> > > > >
> > > > > 0xc056f14c <arm_copy_to_user+0x1c>: @0xc0b89b84
> > > > > Compact model index: 1
> > > > > 0x9b vsp = r11
> > > > > 0xb1 0x0d pop {r0, r2, r3}
> > > > > 0x84 0x81 pop {r4, r11, r14}
> > > > > 0xb0 finish
> > > > >
> > > > > The problem is that the unwind information says "starting at offset
> > > > > 0x1c, to unwind do the following operations". The first of which is
> > > > > to move r11 (fp) to the stack pointer. However, r11 isn't setup
> > > > > until function offset 0x24. You've hit that instruction, which hasn't
> > > > > executed yet, but the stack has been modified by pushing r0, r2-r4,
> > > > > fp and lr onto it.
> > > > >
> > > > > Given this, there is no way that the unwinder (as it currently stands)
> > > > > can do its job properly between 0x1c and 0x24.
> > > > >
> > > > > I don't think this is specifically caused by Ard's patches, but by
> > > > > the addition of KASAN, which has the effect of calling the unwinder
> > > > > at random points in the kernel (when an interrupt happens) and it's
> > > > > clear from the above that there are windows in the code where, if
> > > > > we attempt to unwind using the unwind information, we faill fail
> > > > > because the program state is not consistent with the unwind
> > > > > information.
> > > > >
> > > > > Ard's patch that changes:
> > > > >
> > > > > ctrl->vrs[reg] = READ_ONCE_NOCHECK(*(*vsp));
> > > > >
> > > > > to use get_kernel_nofault() should have the effect of protecting
> > > > > against the oops, but the side effect is that it is fundamentally not
> > > > > possible with the way these things are to unwind at these points -
> > > > > which means its not possible to get a stacktrace there.
> > > > >
> > > > > So, I don't think this is a "new" problem, but a weakness of using
> > > > > the unwinder to get a backtrace for KASAN.
> > > > >
> > > >
> > > > It essentially means that we cannot unwind through asynchronous
> > > > exceptions, and so we should probably make the svc_entry macro
> > > > .nounwind, instead of pretending that we can reliably unwind through
> > > > it.
> > >
> > > Doesn't that impact the ability to debug the kernel over things like
> > > oopses and the like?
> > >
> >
> > The backtrace dumped by __die() uses the pt_regs from the exception
> > context as the starting point, so the exception entry code that deals
> > with the condition that triggered the oops is omitted, and does not
> > have to be unwound.
>
> That is true, but that's not really the case I was thinking about.
> I was thinking more about cases such as RCU stalls, soft lockups,
> etc.
>
> For example:
>
> https://www.linuxquestions.org/questions/linux-kernel-70/kenel-v4-4-60-panic-in-igmp6_send-and-and-__neigh_create-4175704721/
>
> In that stack trace, the interesting bits are not the beginning of
> the stack trace down to __irq_svc, but everything beyond __irq_svc,
> since the lockup is probably caused by being stuck in
> _raw_write_lock_bh().
>
> It's these situations that we will totally destroy debuggability for,
> and the only way around that would be to force frame pointers and
> ARM builds (not Thumb-2 as that requires the unwinder... which means
> a Thumb-2 kernel soft lockup would be undebuggable.
>

Indeed.

But that means that the only other choice we have is to retain the
imprecise nature of the current solution (which usually works fine
btw), and simply deal with the faulting double dereference of vsp in
the unwinder code. We simply don't know whether the exception was
taken at a point where the stack frame is consistent with the unwind
data.

2022-03-09 20:58:47

[permalink] [raw]

Subject: Re: [next] arm: Internal error: Oops: 5 PC is at __read_once_word_nocheck

On Wed, Mar 09, 2022 at 06:43:42PM +0100, Ard Biesheuvel wrote:
> On Wed, 9 Mar 2022 at 18:11, Russell King (Oracle)
> <[email protected]> wrote:
> >
> > On Wed, Mar 09, 2022 at 10:08:25PM +0530, Naresh Kamboju wrote:
> > > Hi Russell,
> > >
> > > On Wed, 9 Mar 2022 at 20:37, Russell King (Oracle)
> > > <[email protected]> wrote:
> > > >
> > > > On Wed, Mar 09, 2022 at 03:57:32PM +0100, Ard Biesheuvel wrote:
> > > > > On Wed, 9 Mar 2022 at 15:44, Naresh Kamboju <[email protected]> wrote:
> > > > > >
> > > <trim>
> > > > Well, we unwound until:
> > > >
> > > > __irq_svc from migrate_disable+0x0/0x70
> > > >
> > > > and then crashed - and the key thing there is that we're at the start
> > > > of migrate_disable() when we took an interrupt.
> > > >
> > > > For some reason, this triggers an access to address 0x10, which faults.
> > > > We then try unwinding again, and successfully unwind all the way back
> > > > to the same point (the line above) which then causes the unwinder to
> > > > again access address 0x10, and the cycle repeats with the stack
> > > > growing bigger and bigger.
> > > >
> > > > I'd suggest also testing without the revert but with my patch.
> > >
> > > I have tested your patch on top of linux next-20220309 and still see kernel
> > > crash as below [1]. build link [2].
> > >
> > > [ 26.812060] 8<--- cut here ---
> > > [ 26.813459] Unhandled fault: page domain fault (0x01b) at 0xb6a3ab70
> > > [ 26.816139] [b6a3ab70] *pgd=fb28a835
> > > [ 26.817770] Internal error: : 1b [#1] SMP ARM
> > > [ 26.819636] Modules linked in:
> > > [ 26.820956] CPU: 0 PID: 211 Comm: haveged Not tainted
> > > 5.17.0-rc7-next-20220309 #1
> > > [ 26.824519] Hardware name: Generic DT based system
> > > [ 26.827148] PC is at __read_once_word_nocheck+0x0/0x8
> > > [ 26.829856] LR is at unwind_frame+0x7dc/0xab4
> > >
> > > - Naresh
> > >
> > > [1] https://lkft.validation.linaro.org/scheduler/job/4688599#L596
> > > [2] https://builds.tuxbuild.com/269gYLGuAdmltuLhIUDAjS2fg1Q/
> >
> > I think the problem has just moved:
> >
> > [ 27.113085] __irq_svc from __copy_to_user_std+0x24/0x378
> >
> > The code at the start of __copy_to_user_std is:
> >
> > 0: e3a034bf mov r3, #-1090519040 ; 0xbf000000
> > 4: e243c001 sub ip, r3, #1
> > 8: e05cc000 subs ip, ip, r0
> > c: 228cc001 addcs ip, ip, #1
> > 10: 205cc002 subscs ip, ip, r2
> > 14: 33a00000 movcc r0, #0
> > 18: e320f014 csdb
> > 1c: e3a03000 mov r3, #0
> > 20: e92d481d push {r0, r2, r3, r4, fp, lr}
> > 24: e1a0b00d mov fp, sp
> >
> > and the unwind information will be:
> >
> > 0xc056f14c <arm_copy_to_user+0x1c>: @0xc0b89b84
> > Compact model index: 1
> > 0x9b vsp = r11
> > 0xb1 0x0d pop {r0, r2, r3}
> > 0x84 0x81 pop {r4, r11, r14}
> > 0xb0 finish
> >
> > The problem is that the unwind information says "starting at offset
> > 0x1c, to unwind do the following operations". The first of which is
> > to move r11 (fp) to the stack pointer. However, r11 isn't setup
> > until function offset 0x24. You've hit that instruction, which hasn't
> > executed yet, but the stack has been modified by pushing r0, r2-r4,
> > fp and lr onto it.
> >
> > Given this, there is no way that the unwinder (as it currently stands)
> > can do its job properly between 0x1c and 0x24.
> >
> > I don't think this is specifically caused by Ard's patches, but by
> > the addition of KASAN, which has the effect of calling the unwinder
> > at random points in the kernel (when an interrupt happens) and it's
> > clear from the above that there are windows in the code where, if
> > we attempt to unwind using the unwind information, we faill fail
> > because the program state is not consistent with the unwind
> > information.
> >
> > Ard's patch that changes:
> >
> > ctrl->vrs[reg] = READ_ONCE_NOCHECK(*(*vsp));
> >
> > to use get_kernel_nofault() should have the effect of protecting
> > against the oops, but the side effect is that it is fundamentally not
> > possible with the way these things are to unwind at these points -
> > which means its not possible to get a stacktrace there.
> >
> > So, I don't think this is a "new" problem, but a weakness of using
> > the unwinder to get a backtrace for KASAN.
> >
>
> It essentially means that we cannot unwind through asynchronous
> exceptions, and so we should probably make the svc_entry macro
> .nounwind, instead of pretending that we can reliably unwind through
> it.

Doesn't that impact the ability to debug the kernel over things like
oopses and the like?

--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

2022-03-09 22:36:37

[permalink] [raw]

Subject: Re: [next] arm: Internal error: Oops: 5 PC is at __read_once_word_nocheck

On Wed, 9 Mar 2022 at 17:38, Naresh Kamboju <[email protected]> wrote:
>
> Hi Russell,
>
> On Wed, 9 Mar 2022 at 20:37, Russell King (Oracle)
> <[email protected]> wrote:
> >
> > On Wed, Mar 09, 2022 at 03:57:32PM +0100, Ard Biesheuvel wrote:
> > > On Wed, 9 Mar 2022 at 15:44, Naresh Kamboju <[email protected]> wrote:
> > > >
> <trim>
> > Well, we unwound until:
> >
> > __irq_svc from migrate_disable+0x0/0x70
> >
> > and then crashed - and the key thing there is that we're at the start
> > of migrate_disable() when we took an interrupt.
> >
> > For some reason, this triggers an access to address 0x10, which faults.
> > We then try unwinding again, and successfully unwind all the way back
> > to the same point (the line above) which then causes the unwinder to
> > again access address 0x10, and the cycle repeats with the stack
> > growing bigger and bigger.
> >
> > I'd suggest also testing without the revert but with my patch.
>
> I have tested your patch on top of linux next-20220309 and still see kernel
> crash as below [1]. build link [2].
>
> [ 26.812060] 8<--- cut here ---
> [ 26.813459] Unhandled fault: page domain fault (0x01b) at 0xb6a3ab70
> [ 26.816139] [b6a3ab70] *pgd=fb28a835
> [ 26.817770] Internal error: : 1b [#1] SMP ARM
> [ 26.819636] Modules linked in:
> [ 26.820956] CPU: 0 PID: 211 Comm: haveged Not tainted
> 5.17.0-rc7-next-20220309 #1
> [ 26.824519] Hardware name: Generic DT based system
> [ 26.827148] PC is at __read_once_word_nocheck+0x0/0x8
> [ 26.829856] LR is at unwind_frame+0x7dc/0xab4
>

Thanks Naresh,

Is this something that could be bisected? With the unwind change
reverted, it is not obvious where the problem originated. Also, could
you check whether it is reproducible on the ARM tree? (for-next on
git://git.armlinux.org.uk/~rmk/linux-arm.git)

Unfortunately, after having been able to reproduce this locally, the
issue went away when I did the revert, so I am no longer seeing the
problem.

--
Ard.

2022-03-09 23:55:12

[permalink] [raw]

Subject: Re: [next] arm: Internal error: Oops: 5 PC is at __read_once_word_nocheck

On Wed, Mar 09, 2022 at 08:14:30PM +0100, Ard Biesheuvel wrote:
> On Wed, 9 Mar 2022 at 19:48, Russell King (Oracle)
> <[email protected]> wrote:
> >
> > On Wed, Mar 09, 2022 at 06:43:42PM +0100, Ard Biesheuvel wrote:
> > > On Wed, 9 Mar 2022 at 18:11, Russell King (Oracle)
> > > <[email protected]> wrote:
> > > >
> > > > On Wed, Mar 09, 2022 at 10:08:25PM +0530, Naresh Kamboju wrote:
> > > > > Hi Russell,
> > > > >
> > > > > On Wed, 9 Mar 2022 at 20:37, Russell King (Oracle)
> > > > > <[email protected]> wrote:
> > > > > >
> > > > > > On Wed, Mar 09, 2022 at 03:57:32PM +0100, Ard Biesheuvel wrote:
> > > > > > > On Wed, 9 Mar 2022 at 15:44, Naresh Kamboju <[email protected]> wrote:
> > > > > > > >
> > > > > <trim>
> > > > > > Well, we unwound until:
> > > > > >
> > > > > > __irq_svc from migrate_disable+0x0/0x70
> > > > > >
> > > > > > and then crashed - and the key thing there is that we're at the start
> > > > > > of migrate_disable() when we took an interrupt.
> > > > > >
> > > > > > For some reason, this triggers an access to address 0x10, which faults.
> > > > > > We then try unwinding again, and successfully unwind all the way back
> > > > > > to the same point (the line above) which then causes the unwinder to
> > > > > > again access address 0x10, and the cycle repeats with the stack
> > > > > > growing bigger and bigger.
> > > > > >
> > > > > > I'd suggest also testing without the revert but with my patch.
> > > > >
> > > > > I have tested your patch on top of linux next-20220309 and still see kernel
> > > > > crash as below [1]. build link [2].
> > > > >
> > > > > [ 26.812060] 8<--- cut here ---
> > > > > [ 26.813459] Unhandled fault: page domain fault (0x01b) at 0xb6a3ab70
> > > > > [ 26.816139] [b6a3ab70] *pgd=fb28a835
> > > > > [ 26.817770] Internal error: : 1b [#1] SMP ARM
> > > > > [ 26.819636] Modules linked in:
> > > > > [ 26.820956] CPU: 0 PID: 211 Comm: haveged Not tainted
> > > > > 5.17.0-rc7-next-20220309 #1
> > > > > [ 26.824519] Hardware name: Generic DT based system
> > > > > [ 26.827148] PC is at __read_once_word_nocheck+0x0/0x8
> > > > > [ 26.829856] LR is at unwind_frame+0x7dc/0xab4
> > > > >
> > > > > - Naresh
> > > > >
> > > > > [1] https://lkft.validation.linaro.org/scheduler/job/4688599#L596
> > > > > [2] https://builds.tuxbuild.com/269gYLGuAdmltuLhIUDAjS2fg1Q/
> > > >
> > > > I think the problem has just moved:
> > > >
> > > > [ 27.113085] __irq_svc from __copy_to_user_std+0x24/0x378
> > > >
> > > > The code at the start of __copy_to_user_std is:
> > > >
> > > > 0: e3a034bf mov r3, #-1090519040 ; 0xbf000000
> > > > 4: e243c001 sub ip, r3, #1
> > > > 8: e05cc000 subs ip, ip, r0
> > > > c: 228cc001 addcs ip, ip, #1
> > > > 10: 205cc002 subscs ip, ip, r2
> > > > 14: 33a00000 movcc r0, #0
> > > > 18: e320f014 csdb
> > > > 1c: e3a03000 mov r3, #0
> > > > 20: e92d481d push {r0, r2, r3, r4, fp, lr}
> > > > 24: e1a0b00d mov fp, sp
> > > >
> > > > and the unwind information will be:
> > > >
> > > > 0xc056f14c <arm_copy_to_user+0x1c>: @0xc0b89b84
> > > > Compact model index: 1
> > > > 0x9b vsp = r11
> > > > 0xb1 0x0d pop {r0, r2, r3}
> > > > 0x84 0x81 pop {r4, r11, r14}
> > > > 0xb0 finish
> > > >
> > > > The problem is that the unwind information says "starting at offset
> > > > 0x1c, to unwind do the following operations". The first of which is
> > > > to move r11 (fp) to the stack pointer. However, r11 isn't setup
> > > > until function offset 0x24. You've hit that instruction, which hasn't
> > > > executed yet, but the stack has been modified by pushing r0, r2-r4,
> > > > fp and lr onto it.
> > > >
> > > > Given this, there is no way that the unwinder (as it currently stands)
> > > > can do its job properly between 0x1c and 0x24.
> > > >
> > > > I don't think this is specifically caused by Ard's patches, but by
> > > > the addition of KASAN, which has the effect of calling the unwinder
> > > > at random points in the kernel (when an interrupt happens) and it's
> > > > clear from the above that there are windows in the code where, if
> > > > we attempt to unwind using the unwind information, we faill fail
> > > > because the program state is not consistent with the unwind
> > > > information.
> > > >
> > > > Ard's patch that changes:
> > > >
> > > > ctrl->vrs[reg] = READ_ONCE_NOCHECK(*(*vsp));
> > > >
> > > > to use get_kernel_nofault() should have the effect of protecting
> > > > against the oops, but the side effect is that it is fundamentally not
> > > > possible with the way these things are to unwind at these points -
> > > > which means its not possible to get a stacktrace there.
> > > >
> > > > So, I don't think this is a "new" problem, but a weakness of using
> > > > the unwinder to get a backtrace for KASAN.
> > > >
> > >
> > > It essentially means that we cannot unwind through asynchronous
> > > exceptions, and so we should probably make the svc_entry macro
> > > .nounwind, instead of pretending that we can reliably unwind through
> > > it.
> >
> > Doesn't that impact the ability to debug the kernel over things like
> > oopses and the like?
> >
>
> The backtrace dumped by __die() uses the pt_regs from the exception
> context as the starting point, so the exception entry code that deals
> with the condition that triggered the oops is omitted, and does not
> have to be unwound.

That is true, but that's not really the case I was thinking about.
I was thinking more about cases such as RCU stalls, soft lockups,
etc.

For example:

https://www.linuxquestions.org/questions/linux-kernel-70/kenel-v4-4-60-panic-in-igmp6_send-and-and-__neigh_create-4175704721/

In that stack trace, the interesting bits are not the beginning of
the stack trace down to __irq_svc, but everything beyond __irq_svc,
since the lockup is probably caused by being stuck in
_raw_write_lock_bh().

It's these situations that we will totally destroy debuggability for,
and the only way around that would be to force frame pointers and
ARM builds (not Thumb-2 as that requires the unwinder... which means
a Thumb-2 kernel soft lockup would be undebuggable.

--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

2022-03-10 09:25:01

[permalink] [raw]

Subject: Re: [next] arm: Internal error: Oops: 5 PC is at __read_once_word_nocheck

On Wed, 9 Mar 2022 at 19:48, Russell King (Oracle)
<[email protected]> wrote:
>
> On Wed, Mar 09, 2022 at 06:43:42PM +0100, Ard Biesheuvel wrote:
> > On Wed, 9 Mar 2022 at 18:11, Russell King (Oracle)
> > <[email protected]> wrote:
> > >
> > > On Wed, Mar 09, 2022 at 10:08:25PM +0530, Naresh Kamboju wrote:
> > > > Hi Russell,
> > > >
> > > > On Wed, 9 Mar 2022 at 20:37, Russell King (Oracle)
> > > > <[email protected]> wrote:
> > > > >
> > > > > On Wed, Mar 09, 2022 at 03:57:32PM +0100, Ard Biesheuvel wrote:
> > > > > > On Wed, 9 Mar 2022 at 15:44, Naresh Kamboju <[email protected]> wrote:
> > > > > > >
> > > > <trim>
> > > > > Well, we unwound until:
> > > > >
> > > > > __irq_svc from migrate_disable+0x0/0x70
> > > > >
> > > > > and then crashed - and the key thing there is that we're at the start
> > > > > of migrate_disable() when we took an interrupt.
> > > > >
> > > > > For some reason, this triggers an access to address 0x10, which faults.
> > > > > We then try unwinding again, and successfully unwind all the way back
> > > > > to the same point (the line above) which then causes the unwinder to
> > > > > again access address 0x10, and the cycle repeats with the stack
> > > > > growing bigger and bigger.
> > > > >
> > > > > I'd suggest also testing without the revert but with my patch.
> > > >
> > > > I have tested your patch on top of linux next-20220309 and still see kernel
> > > > crash as below [1]. build link [2].
> > > >
> > > > [ 26.812060] 8<--- cut here ---
> > > > [ 26.813459] Unhandled fault: page domain fault (0x01b) at 0xb6a3ab70
> > > > [ 26.816139] [b6a3ab70] *pgd=fb28a835
> > > > [ 26.817770] Internal error: : 1b [#1] SMP ARM
> > > > [ 26.819636] Modules linked in:
> > > > [ 26.820956] CPU: 0 PID: 211 Comm: haveged Not tainted
> > > > 5.17.0-rc7-next-20220309 #1
> > > > [ 26.824519] Hardware name: Generic DT based system
> > > > [ 26.827148] PC is at __read_once_word_nocheck+0x0/0x8
> > > > [ 26.829856] LR is at unwind_frame+0x7dc/0xab4
> > > >
> > > > - Naresh
> > > >
> > > > [1] https://lkft.validation.linaro.org/scheduler/job/4688599#L596
> > > > [2] https://builds.tuxbuild.com/269gYLGuAdmltuLhIUDAjS2fg1Q/
> > >
> > > I think the problem has just moved:
> > >
> > > [ 27.113085] __irq_svc from __copy_to_user_std+0x24/0x378
> > >
> > > The code at the start of __copy_to_user_std is:
> > >
> > > 0: e3a034bf mov r3, #-1090519040 ; 0xbf000000
> > > 4: e243c001 sub ip, r3, #1
> > > 8: e05cc000 subs ip, ip, r0
> > > c: 228cc001 addcs ip, ip, #1
> > > 10: 205cc002 subscs ip, ip, r2
> > > 14: 33a00000 movcc r0, #0
> > > 18: e320f014 csdb
> > > 1c: e3a03000 mov r3, #0
> > > 20: e92d481d push {r0, r2, r3, r4, fp, lr}
> > > 24: e1a0b00d mov fp, sp
> > >
> > > and the unwind information will be:
> > >
> > > 0xc056f14c <arm_copy_to_user+0x1c>: @0xc0b89b84
> > > Compact model index: 1
> > > 0x9b vsp = r11
> > > 0xb1 0x0d pop {r0, r2, r3}
> > > 0x84 0x81 pop {r4, r11, r14}
> > > 0xb0 finish
> > >
> > > The problem is that the unwind information says "starting at offset
> > > 0x1c, to unwind do the following operations". The first of which is
> > > to move r11 (fp) to the stack pointer. However, r11 isn't setup
> > > until function offset 0x24. You've hit that instruction, which hasn't
> > > executed yet, but the stack has been modified by pushing r0, r2-r4,
> > > fp and lr onto it.
> > >
> > > Given this, there is no way that the unwinder (as it currently stands)
> > > can do its job properly between 0x1c and 0x24.
> > >
> > > I don't think this is specifically caused by Ard's patches, but by
> > > the addition of KASAN, which has the effect of calling the unwinder
> > > at random points in the kernel (when an interrupt happens) and it's
> > > clear from the above that there are windows in the code where, if
> > > we attempt to unwind using the unwind information, we faill fail
> > > because the program state is not consistent with the unwind
> > > information.
> > >
> > > Ard's patch that changes:
> > >
> > > ctrl->vrs[reg] = READ_ONCE_NOCHECK(*(*vsp));
> > >
> > > to use get_kernel_nofault() should have the effect of protecting
> > > against the oops, but the side effect is that it is fundamentally not
> > > possible with the way these things are to unwind at these points -
> > > which means its not possible to get a stacktrace there.
> > >
> > > So, I don't think this is a "new" problem, but a weakness of using
> > > the unwinder to get a backtrace for KASAN.
> > >
> >
> > It essentially means that we cannot unwind through asynchronous
> > exceptions, and so we should probably make the svc_entry macro
> > .nounwind, instead of pretending that we can reliably unwind through
> > it.
>
> Doesn't that impact the ability to debug the kernel over things like
> oopses and the like?
>

The backtrace dumped by __die() uses the pt_regs from the exception
context as the starting point, so the exception entry code that deals
with the condition that triggered the oops is omitted, and does not
have to be unwound.

There may be other cases where less context is being provided with
this change, but it is basically the conclusion drawn in this thread
that the ARM EHABI unwinder cannot reliably do so anyway.

2022-03-10 10:27:07

[permalink] [raw]

Subject: Re: [next] arm: Internal error: Oops: 5 PC is at __read_once_word_nocheck

On Wed, 9 Mar 2022 at 18:11, Russell King (Oracle)
<[email protected]> wrote:
>
> On Wed, Mar 09, 2022 at 10:08:25PM +0530, Naresh Kamboju wrote:
> > Hi Russell,
> >
> > On Wed, 9 Mar 2022 at 20:37, Russell King (Oracle)
> > <[email protected]> wrote:
> > >
> > > On Wed, Mar 09, 2022 at 03:57:32PM +0100, Ard Biesheuvel wrote:
> > > > On Wed, 9 Mar 2022 at 15:44, Naresh Kamboju <[email protected]> wrote:
> > > > >
> > <trim>
> > > Well, we unwound until:
> > >
> > > __irq_svc from migrate_disable+0x0/0x70
> > >
> > > and then crashed - and the key thing there is that we're at the start
> > > of migrate_disable() when we took an interrupt.
> > >
> > > For some reason, this triggers an access to address 0x10, which faults.
> > > We then try unwinding again, and successfully unwind all the way back
> > > to the same point (the line above) which then causes the unwinder to
> > > again access address 0x10, and the cycle repeats with the stack
> > > growing bigger and bigger.
> > >
> > > I'd suggest also testing without the revert but with my patch.
> >
> > I have tested your patch on top of linux next-20220309 and still see kernel
> > crash as below [1]. build link [2].
> >
> > [ 26.812060] 8<--- cut here ---
> > [ 26.813459] Unhandled fault: page domain fault (0x01b) at 0xb6a3ab70
> > [ 26.816139] [b6a3ab70] *pgd=fb28a835
> > [ 26.817770] Internal error: : 1b [#1] SMP ARM
> > [ 26.819636] Modules linked in:
> > [ 26.820956] CPU: 0 PID: 211 Comm: haveged Not tainted
> > 5.17.0-rc7-next-20220309 #1
> > [ 26.824519] Hardware name: Generic DT based system
> > [ 26.827148] PC is at __read_once_word_nocheck+0x0/0x8
> > [ 26.829856] LR is at unwind_frame+0x7dc/0xab4
> >
> > - Naresh
> >
> > [1] https://lkft.validation.linaro.org/scheduler/job/4688599#L596
> > [2] https://builds.tuxbuild.com/269gYLGuAdmltuLhIUDAjS2fg1Q/
>
> I think the problem has just moved:
>
> [ 27.113085] __irq_svc from __copy_to_user_std+0x24/0x378
>
> The code at the start of __copy_to_user_std is:
>
> 0: e3a034bf mov r3, #-1090519040 ; 0xbf000000
> 4: e243c001 sub ip, r3, #1
> 8: e05cc000 subs ip, ip, r0
> c: 228cc001 addcs ip, ip, #1
> 10: 205cc002 subscs ip, ip, r2
> 14: 33a00000 movcc r0, #0
> 18: e320f014 csdb
> 1c: e3a03000 mov r3, #0
> 20: e92d481d push {r0, r2, r3, r4, fp, lr}
> 24: e1a0b00d mov fp, sp
>
> and the unwind information will be:
>
> 0xc056f14c <arm_copy_to_user+0x1c>: @0xc0b89b84
> Compact model index: 1
> 0x9b vsp = r11
> 0xb1 0x0d pop {r0, r2, r3}
> 0x84 0x81 pop {r4, r11, r14}
> 0xb0 finish
>
> The problem is that the unwind information says "starting at offset
> 0x1c, to unwind do the following operations". The first of which is
> to move r11 (fp) to the stack pointer. However, r11 isn't setup
> until function offset 0x24. You've hit that instruction, which hasn't
> executed yet, but the stack has been modified by pushing r0, r2-r4,
> fp and lr onto it.
>
> Given this, there is no way that the unwinder (as it currently stands)
> can do its job properly between 0x1c and 0x24.
>
> I don't think this is specifically caused by Ard's patches, but by
> the addition of KASAN, which has the effect of calling the unwinder
> at random points in the kernel (when an interrupt happens) and it's
> clear from the above that there are windows in the code where, if
> we attempt to unwind using the unwind information, we faill fail
> because the program state is not consistent with the unwind
> information.
>
> Ard's patch that changes:
>
> ctrl->vrs[reg] = READ_ONCE_NOCHECK(*(*vsp));
>
> to use get_kernel_nofault() should have the effect of protecting
> against the oops, but the side effect is that it is fundamentally not
> possible with the way these things are to unwind at these points -
> which means its not possible to get a stacktrace there.
>
> So, I don't think this is a "new" problem, but a weakness of using
> the unwinder to get a backtrace for KASAN.
>

It essentially means that we cannot unwind through asynchronous
exceptions, and so we should probably make the svc_entry macro
.nounwind, instead of pretending that we can reliably unwind through
it.

For these annotation purposes, where the interrupt was taken is not
terribly interesting anyway, terminating the stacktrace earlier might
even recover some performance lost to KASAN overhead.

Naresh, please try the patch below.

diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S
index 5609ca8ae46a..0d8ae1a14643 100644
--- a/arch/arm/kernel/entry-armv.S
+++ b/arch/arm/kernel/entry-armv.S
@@ -184,12 +184,12 @@ ENDPROC(__und_invalid)

.macro svc_entry, stack_hole=0, trace=1, uaccess=1, overflow_check=1
UNWIND(.fnstart )
+ UNWIND(.cantunwind )
sub sp, sp, #(SVC_REGS_SIZE + \stack_hole)
THUMB( add sp, r1 ) @ get SP in a GPR without
THUMB( sub r1, sp, r1 ) @ using a temp register

.if \overflow_check
- UNWIND(.save {r0 - pc} )
do_overflow_check (SVC_REGS_SIZE + \stack_hole)
.endif

2022-03-10 13:38:30

[permalink] [raw]

Subject: Re: [next] arm: Internal error: Oops: 5 PC is at __read_once_word_nocheck

On Thu, 10 Mar 2022 at 14:01, Russell King (Oracle)
<[email protected]> wrote:
>
> On Thu, Mar 10, 2022 at 12:35:55PM +0000, Russell King (Oracle) wrote:
> > On Wed, Mar 09, 2022 at 09:42:29PM +0100, Ard Biesheuvel wrote:
> > > On Wed, 9 Mar 2022 at 20:39, Russell King (Oracle)
> > > <[email protected]> wrote:
> > > >
> > > > On Wed, Mar 09, 2022 at 08:14:30PM +0100, Ard Biesheuvel wrote:
> > > > > The backtrace dumped by __die() uses the pt_regs from the exception
> > > > > context as the starting point, so the exception entry code that deals
> > > > > with the condition that triggered the oops is omitted, and does not
> > > > > have to be unwound.
> > > >
> > > > That is true, but that's not really the case I was thinking about.
> > > > I was thinking more about cases such as RCU stalls, soft lockups,
> > > > etc.
> > > >
> > > > For example:
> > > >
> > > > https://www.linuxquestions.org/questions/linux-kernel-70/kenel-v4-4-60-panic-in-igmp6_send-and-and-__neigh_create-4175704721/
> > > >
> > > > In that stack trace, the interesting bits are not the beginning of
> > > > the stack trace down to __irq_svc, but everything beyond __irq_svc,
> > > > since the lockup is probably caused by being stuck in
> > > > _raw_write_lock_bh().
> > > >
> > > > It's these situations that we will totally destroy debuggability for,
> > > > and the only way around that would be to force frame pointers and
> > > > ARM builds (not Thumb-2 as that requires the unwinder... which means
> > > > a Thumb-2 kernel soft lockup would be undebuggable.
> > > >
> > >
> > > Indeed.
> > >
> > > But that means that the only other choice we have is to retain the
> > > imprecise nature of the current solution (which usually works fine
> > > btw), and simply deal with the faulting double dereference of vsp in
> > > the unwinder code. We simply don't know whether the exception was
> > > taken at a point where the stack frame is consistent with the unwind
> > > data.
> >
> > Okay, further analysis (for the record, since I've said much of this on
> > IRC):
> >
> > What we have currently is a robust unwinder that will cope when things
> > go wrong, such as an interrupt taken during the prologue of a function.
> > The way it copes is by two mechanisms:
> >
> > /* store the highest address on the stack to avoid crossing it*/
> > low = frame->sp;
> > ctrl.sp_high = ALIGN(low, THREAD_SIZE);
> >
> > These two represent the allowable bounds of the kernel stack. When we
> > run the unwinder, before each unwind instruction we check whether the
> > current SP value is getting close to the top of the kernel stack, and
> > if so, turn on additional checking:
> >
> > if ((ctrl.sp_high - ctrl.vrs[SP]) < sizeof(ctrl.vrs))
> > ctrl.check_each_pop = 1;
> >
> > that will ensure if we go off the top of the kernel stack, the
> > unwinder will report failure, and not access those addresses.
> >
> > After each instruction, we check whether the SP value is within the
> > above bounds:
> >
> > if (ctrl.vrs[SP] < low || ctrl.vrs[SP] >= ctrl.sp_high)
> > return -URC_FAILURE;
> >
> > This means that the unwinder can never modify SP to point outside of
> > the kernel stack region identified by low..ctrl.sp_high, thereby
> > protecting the load inside unwind_pop_register() from ever
> > dereferencing something outside of the kernel stack. Moreover, it also
> > prevents the unwinder modifying SP to point below the current stack
> > frame.
> >
> > The problem has been introduced by trying to make the unwinder cope
> > with IRQ stacks in b6506981f880 ("ARM: unwind: support unwinding across
> > multiple stacks"):
> >
> > - if (!load_sp)
> > + if (!load_sp) {
> > ctrl->vrs[SP] = (unsigned long)vsp;
> > + } else {
> > + ctrl->sp_low = ctrl->vrs[SP];
> > + ctrl->sp_high = ALIGN(ctrl->sp_low, THREAD_SIZE);
> > + }
> >
> > Now, whenever SP is loaded, we reset the allowable range for the SP
> > value, and this completely defeats the protections we previously had
> > which were ensuring that:
> >
> > 1) the SP value doesn't jump back _down_ the kernel stack resulting
> > in an infinite unwind loop.
> > 2) the SP value doesn't end up outside the kernel stack.
> >
> > We need those protections to prevent these problems that are being
> > reported - and the most efficient way I can think of doing that is to
> > somehow valudate the new SP value _before_ we modify sp_low and
> > sp_high, so these two limits are always valid.
> >
> > Merely changing the READ_ONCE_NOCHECK() to be get_kernel_nocheck()
> > will only partly fix this problem - it will stop the unwinder oopsing
> > the kernel, but definitely doesn't protect against (1) and doesn't
> > protect against SP pointing at some thing that is accessible (e.g.
> > a device or other kernel memory.)
> >
> > We're yet again at Thursday, with the last linux-next prior to the
> > merge window being created this evening, which really doesn't leave
> > much time to get this sorted... and we can't say "this code should
> > have been in earlier in the cycle" this time around, because these
> > changes to the unwinder have been present in linux-next prior to
> > 5.17-rc2. Annoyingly, it seems merging stuff earlier in the cycle
> > doesn't actually solve the problem of these last minute debugging
> > panics.
> >
> > Any suggestions for what we should do? Can we come up with some way
> > to validate the new SP value before 6pm UTC this evening?
>
> Also, looking deeper at the last linaro job report:
>
> https://lkft.validation.linaro.org/scheduler/job/4688599#L684
>
> the dumped memory doesn't look like an exception stack. If it was,
> e82aab40 would be the saved CPSR value and c388eb80 would be the PC
> value, both of which are nonsense.
>
> The stack that we were in (and we dumped out in full) was:
>
> Stack: (0xc381bb30 to 0xc381c000)
>

This is the IRQ stack that we switch to in

[ 29.198048] irq_exit from __irq_svc+0x70/0x8c
[ 29.200896] Exception stack(0xc5fcfc98 to 0xc5fcfce0)

> and the exception stack (the saved pt_regs) is:
>
> r0 r1 r2 r3 r4
> bfa0: c2ba47c0 0000000a c2ba1358 ffff9537 c2c05d00 00400140 c62d5624 c1948b20
> r5 r6 r7 r8 r9 r10 fp ip
> bfc0: e82ab498 c62d5400 c35377a0 c62d5404 25706000 c381bfe8 c62d5400 00000001
> sp lr pc cpsr orig_r0
> bfe0: c5fcfc48 c036251c c0995a14 20040013 ffffffff c5fcfc7c c62d5400 c0300bf0
>

I don't follow this - which pt_regs is this?

> but, we end up dumping out:
>
> Exception stack(0xc5fcfc98 to 0xc5fcfce0)
> fc80: b6a3ab70 00000004
> fca0: 00000000 00000004 b6a3ab70 c055f928 c388eb80 c5fcfd40 00000000 c5fcfd50
> fcc0: 00000005 00000051 e82aad1c c03ae570 00000000 c388eb80 c3512a20 e82aab40
>
> Firstly, that's in the wrong stack to be dumping for the exception
> stack, and secondly, why is it 0x50 bytes above the saved SP value from
> the real exception stack - that makes no sense in itself.
>

This is the exception stack that we were on when taking an IRQ and
subsequently switching to the IRQ stack.

2022-03-10 14:10:53

[permalink] [raw]

Subject: Re: [next] arm: Internal error: Oops: 5 PC is at __read_once_word_nocheck

On Wed, Mar 09, 2022 at 09:42:29PM +0100, Ard Biesheuvel wrote:
> On Wed, 9 Mar 2022 at 20:39, Russell King (Oracle)
> <[email protected]> wrote:
> >
> > On Wed, Mar 09, 2022 at 08:14:30PM +0100, Ard Biesheuvel wrote:
> > > The backtrace dumped by __die() uses the pt_regs from the exception
> > > context as the starting point, so the exception entry code that deals
> > > with the condition that triggered the oops is omitted, and does not
> > > have to be unwound.
> >
> > That is true, but that's not really the case I was thinking about.
> > I was thinking more about cases such as RCU stalls, soft lockups,
> > etc.
> >
> > For example:
> >
> > https://www.linuxquestions.org/questions/linux-kernel-70/kenel-v4-4-60-panic-in-igmp6_send-and-and-__neigh_create-4175704721/
> >
> > In that stack trace, the interesting bits are not the beginning of
> > the stack trace down to __irq_svc, but everything beyond __irq_svc,
> > since the lockup is probably caused by being stuck in
> > _raw_write_lock_bh().
> >
> > It's these situations that we will totally destroy debuggability for,
> > and the only way around that would be to force frame pointers and
> > ARM builds (not Thumb-2 as that requires the unwinder... which means
> > a Thumb-2 kernel soft lockup would be undebuggable.
> >
>
> Indeed.
>
> But that means that the only other choice we have is to retain the
> imprecise nature of the current solution (which usually works fine
> btw), and simply deal with the faulting double dereference of vsp in
> the unwinder code. We simply don't know whether the exception was
> taken at a point where the stack frame is consistent with the unwind
> data.

Okay, further analysis (for the record, since I've said much of this on
IRC):

What we have currently is a robust unwinder that will cope when things
go wrong, such as an interrupt taken during the prologue of a function.
The way it copes is by two mechanisms:

/* store the highest address on the stack to avoid crossing it*/
low = frame->sp;
ctrl.sp_high = ALIGN(low, THREAD_SIZE);

These two represent the allowable bounds of the kernel stack. When we
run the unwinder, before each unwind instruction we check whether the
current SP value is getting close to the top of the kernel stack, and
if so, turn on additional checking:

if ((ctrl.sp_high - ctrl.vrs[SP]) < sizeof(ctrl.vrs))
ctrl.check_each_pop = 1;

that will ensure if we go off the top of the kernel stack, the
unwinder will report failure, and not access those addresses.

After each instruction, we check whether the SP value is within the
above bounds:

if (ctrl.vrs[SP] < low || ctrl.vrs[SP] >= ctrl.sp_high)
return -URC_FAILURE;

This means that the unwinder can never modify SP to point outside of
the kernel stack region identified by low..ctrl.sp_high, thereby
protecting the load inside unwind_pop_register() from ever
dereferencing something outside of the kernel stack. Moreover, it also
prevents the unwinder modifying SP to point below the current stack
frame.

The problem has been introduced by trying to make the unwinder cope
with IRQ stacks in b6506981f880 ("ARM: unwind: support unwinding across
multiple stacks"):

- if (!load_sp)
+ if (!load_sp) {
ctrl->vrs[SP] = (unsigned long)vsp;
+ } else {
+ ctrl->sp_low = ctrl->vrs[SP];
+ ctrl->sp_high = ALIGN(ctrl->sp_low, THREAD_SIZE);
+ }

Now, whenever SP is loaded, we reset the allowable range for the SP
value, and this completely defeats the protections we previously had
which were ensuring that:

1) the SP value doesn't jump back _down_ the kernel stack resulting
in an infinite unwind loop.
2) the SP value doesn't end up outside the kernel stack.

We need those protections to prevent these problems that are being
reported - and the most efficient way I can think of doing that is to
somehow valudate the new SP value _before_ we modify sp_low and
sp_high, so these two limits are always valid.

Merely changing the READ_ONCE_NOCHECK() to be get_kernel_nocheck()
will only partly fix this problem - it will stop the unwinder oopsing
the kernel, but definitely doesn't protect against (1) and doesn't
protect against SP pointing at some thing that is accessible (e.g.
a device or other kernel memory.)

We're yet again at Thursday, with the last linux-next prior to the
merge window being created this evening, which really doesn't leave
much time to get this sorted... and we can't say "this code should
have been in earlier in the cycle" this time around, because these
changes to the unwinder have been present in linux-next prior to
5.17-rc2. Annoyingly, it seems merging stuff earlier in the cycle
doesn't actually solve the problem of these last minute debugging
panics.

Any suggestions for what we should do? Can we come up with some way
to validate the new SP value before 6pm UTC this evening?

--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

2022-03-10 22:43:35

[permalink] [raw]

Subject: Re: [next] arm: Internal error: Oops: 5 PC is at __read_once_word_nocheck

Hi Ard and Russell,

The boot test pass on linux next-20220310 tag with KASAN=y on BeagleBoard x15
device. but LTP cve tests reproduced the reported kernel crash [1].
From the available historical data I can confirm that this is an
intermittent issue on
BeagleBoard x15 devices.

OTOH, the kernel crash is always reproducible on qemu-arm with KASAN=y
while booting which has been known to fail for a long time.

From the Ardb tree I have boot tested qemu-arm with KASAN=y the reported
kernel crash is always reproducible.

The build steps [3] and extra Kconfigs.

- Naresh
[1] https://lkft.validation.linaro.org/scheduler/job/4701310
[2] https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/
[3] https://builds.tuxbuild.com/2661dIAPUjE2DMJvye91He2gus0/tuxmake_reproducer.sh

2022-03-11 00:28:14

[permalink] [raw]

Subject: Re: [next] arm: Internal error: Oops: 5 PC is at __read_once_word_nocheck

On Thu, 10 Mar 2022 at 22:50, Naresh Kamboju <[email protected]> wrote:
>
> On Fri, 11 Mar 2022 at 02:55, Ard Biesheuvel <[email protected]> wrote:
> >
> > On Thu, 10 Mar 2022 at 22:18, Naresh Kamboju <[email protected]> wrote:
> > >
> > > Hi Ard and Russell,
> > >
> > > The boot test pass on linux next-20220310 tag with KASAN=y on BeagleBoard x15
> > > device. but LTP cve tests reproduced the reported kernel crash [1].
> > > From the available historical data I can confirm that this is an
> > > intermittent issue on
> > > BeagleBoard x15 devices.
> > >
> > > OTOH, the kernel crash is always reproducible on qemu-arm with KASAN=y
> > > while booting which has been known to fail for a long time.
> > >
> > > From the Ardb tree I have boot tested qemu-arm with KASAN=y the reported
> > > kernel crash is always reproducible.
> > >
> > > The build steps [3] and extra Kconfigs.
> > >
> > > - Naresh
> > > [1] https://lkft.validation.linaro.org/scheduler/job/4701310
> > > [2] https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/
> > > [3] https://builds.tuxbuild.com/2661dIAPUjE2DMJvye91He2gus0/tuxmake_reproducer.sh
> >
> > Thanks Naresh. I'm having trouble to make sense of this, though. The
> > linked output log appears to be from a build that lacks my 'ARM:
> > entry: fix unwinder problems caused by IRQ stacks' patch, as it
> > doesn't show any occurrences of call_with_stack() on any of the call
> > stacks.
> >
> > Do you have a link to the vmlinux and zImage files for this build?
>
> Yes.
>
> vmlinux.xz: https://builds.tuxbuild.com/26BmIasJnAyCii0SkgbKarkF369/vmlinux.xz
> zImage: https://builds.tuxbuild.com/26BmIasJnAyCii0SkgbKarkF369/zImage
> System.map: https://builds.tuxbuild.com/26BmIasJnAyCii0SkgbKarkF369/System.map
> Build log: https://builds.tuxbuild.com/26BmIasJnAyCii0SkgbKarkF369/
>

This kernel does not appear to have

ARM: unwind: set frame.pc correctly for current-thread unwinding
ARM: entry: fix unwinder problems caused by IRQ stacks
ARM: Revert "unwind: dump exception stack from calling frame"

so it is expected that the same issue is still being observed.

Could you please try -next with those patches applied?

2022-03-11 02:55:06

[permalink] [raw]

Subject: Re: [next] arm: Internal error: Oops: 5 PC is at __read_once_word_nocheck

On Wed, Mar 09, 2022 at 10:08:25PM +0530, Naresh Kamboju wrote:
> Hi Russell,
>
> On Wed, 9 Mar 2022 at 20:37, Russell King (Oracle)
> <[email protected]> wrote:
> >
> > On Wed, Mar 09, 2022 at 03:57:32PM +0100, Ard Biesheuvel wrote:
> > > On Wed, 9 Mar 2022 at 15:44, Naresh Kamboju <[email protected]> wrote:
> > > >
> <trim>
> > Well, we unwound until:
> >
> > __irq_svc from migrate_disable+0x0/0x70
> >
> > and then crashed - and the key thing there is that we're at the start
> > of migrate_disable() when we took an interrupt.
> >
> > For some reason, this triggers an access to address 0x10, which faults.
> > We then try unwinding again, and successfully unwind all the way back
> > to the same point (the line above) which then causes the unwinder to
> > again access address 0x10, and the cycle repeats with the stack
> > growing bigger and bigger.
> >
> > I'd suggest also testing without the revert but with my patch.
>
> I have tested your patch on top of linux next-20220309 and still see kernel
> crash as below [1]. build link [2].
>
> [ 26.812060] 8<--- cut here ---
> [ 26.813459] Unhandled fault: page domain fault (0x01b) at 0xb6a3ab70
> [ 26.816139] [b6a3ab70] *pgd=fb28a835
> [ 26.817770] Internal error: : 1b [#1] SMP ARM
> [ 26.819636] Modules linked in:
> [ 26.820956] CPU: 0 PID: 211 Comm: haveged Not tainted
> 5.17.0-rc7-next-20220309 #1
> [ 26.824519] Hardware name: Generic DT based system
> [ 26.827148] PC is at __read_once_word_nocheck+0x0/0x8
> [ 26.829856] LR is at unwind_frame+0x7dc/0xab4
>
> - Naresh
>
> [1] https://lkft.validation.linaro.org/scheduler/job/4688599#L596
> [2] https://builds.tuxbuild.com/269gYLGuAdmltuLhIUDAjS2fg1Q/

I think the problem has just moved:

[ 27.113085] __irq_svc from __copy_to_user_std+0x24/0x378

The code at the start of __copy_to_user_std is:

0: e3a034bf mov r3, #-1090519040 ; 0xbf000000
4: e243c001 sub ip, r3, #1
8: e05cc000 subs ip, ip, r0
c: 228cc001 addcs ip, ip, #1
10: 205cc002 subscs ip, ip, r2
14: 33a00000 movcc r0, #0
18: e320f014 csdb
1c: e3a03000 mov r3, #0
20: e92d481d push {r0, r2, r3, r4, fp, lr}
24: e1a0b00d mov fp, sp

and the unwind information will be:

0xc056f14c <arm_copy_to_user+0x1c>: @0xc0b89b84
Compact model index: 1
0x9b vsp = r11
0xb1 0x0d pop {r0, r2, r3}
0x84 0x81 pop {r4, r11, r14}
0xb0 finish

The problem is that the unwind information says "starting at offset
0x1c, to unwind do the following operations". The first of which is
to move r11 (fp) to the stack pointer. However, r11 isn't setup
until function offset 0x24. You've hit that instruction, which hasn't
executed yet, but the stack has been modified by pushing r0, r2-r4,
fp and lr onto it.

Given this, there is no way that the unwinder (as it currently stands)
can do its job properly between 0x1c and 0x24.

I don't think this is specifically caused by Ard's patches, but by
the addition of KASAN, which has the effect of calling the unwinder
at random points in the kernel (when an interrupt happens) and it's
clear from the above that there are windows in the code where, if
we attempt to unwind using the unwind information, we faill fail
because the program state is not consistent with the unwind
information.

Ard's patch that changes:

ctrl->vrs[reg] = READ_ONCE_NOCHECK(*(*vsp));

to use get_kernel_nofault() should have the effect of protecting
against the oops, but the side effect is that it is fundamentally not
possible with the way these things are to unwind at these points -
which means its not possible to get a stacktrace there.

So, I don't think this is a "new" problem, but a weakness of using
the unwinder to get a backtrace for KASAN.

Do you have any way to work out exactly when this problem first
appeared?

--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

2022-03-11 16:25:32

[permalink] [raw]

Subject: Re: [next] arm: Internal error: Oops: 5 PC is at __read_once_word_nocheck

On Thu, 10 Mar 2022 at 22:18, Naresh Kamboju <[email protected]> wrote:
>
> Hi Ard and Russell,
>
> The boot test pass on linux next-20220310 tag with KASAN=y on BeagleBoard x15
> device. but LTP cve tests reproduced the reported kernel crash [1].
> From the available historical data I can confirm that this is an
> intermittent issue on
> BeagleBoard x15 devices.
>
> OTOH, the kernel crash is always reproducible on qemu-arm with KASAN=y
> while booting which has been known to fail for a long time.
>
> From the Ardb tree I have boot tested qemu-arm with KASAN=y the reported
> kernel crash is always reproducible.
>
> The build steps [3] and extra Kconfigs.
>
> - Naresh
> [1] https://lkft.validation.linaro.org/scheduler/job/4701310
> [2] https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/
> [3] https://builds.tuxbuild.com/2661dIAPUjE2DMJvye91He2gus0/tuxmake_reproducer.sh

Thanks Naresh. I'm having trouble to make sense of this, though. The
linked output log appears to be from a build that lacks my 'ARM:
entry: fix unwinder problems caused by IRQ stacks' patch, as it
doesn't show any occurrences of call_with_stack() on any of the call
stacks.

Do you have a link to the vmlinux and zImage files for this build?

2022-03-11 18:38:17

[permalink] [raw]

Subject: Re: [next] arm: Internal error: Oops: 5 PC is at __read_once_word_nocheck

On Thu, Mar 10, 2022 at 12:35:55PM +0000, Russell King (Oracle) wrote:
> On Wed, Mar 09, 2022 at 09:42:29PM +0100, Ard Biesheuvel wrote:
> > On Wed, 9 Mar 2022 at 20:39, Russell King (Oracle)
> > <[email protected]> wrote:
> > >
> > > On Wed, Mar 09, 2022 at 08:14:30PM +0100, Ard Biesheuvel wrote:
> > > > The backtrace dumped by __die() uses the pt_regs from the exception
> > > > context as the starting point, so the exception entry code that deals
> > > > with the condition that triggered the oops is omitted, and does not
> > > > have to be unwound.
> > >
> > > That is true, but that's not really the case I was thinking about.
> > > I was thinking more about cases such as RCU stalls, soft lockups,
> > > etc.
> > >
> > > For example:
> > >
> > > https://www.linuxquestions.org/questions/linux-kernel-70/kenel-v4-4-60-panic-in-igmp6_send-and-and-__neigh_create-4175704721/
> > >
> > > In that stack trace, the interesting bits are not the beginning of
> > > the stack trace down to __irq_svc, but everything beyond __irq_svc,
> > > since the lockup is probably caused by being stuck in
> > > _raw_write_lock_bh().
> > >
> > > It's these situations that we will totally destroy debuggability for,
> > > and the only way around that would be to force frame pointers and
> > > ARM builds (not Thumb-2 as that requires the unwinder... which means
> > > a Thumb-2 kernel soft lockup would be undebuggable.
> > >
> >
> > Indeed.
> >
> > But that means that the only other choice we have is to retain the
> > imprecise nature of the current solution (which usually works fine
> > btw), and simply deal with the faulting double dereference of vsp in
> > the unwinder code. We simply don't know whether the exception was
> > taken at a point where the stack frame is consistent with the unwind
> > data.
>
> Okay, further analysis (for the record, since I've said much of this on
> IRC):
>
> What we have currently is a robust unwinder that will cope when things
> go wrong, such as an interrupt taken during the prologue of a function.
> The way it copes is by two mechanisms:
>
> /* store the highest address on the stack to avoid crossing it*/
> low = frame->sp;
> ctrl.sp_high = ALIGN(low, THREAD_SIZE);
>
> These two represent the allowable bounds of the kernel stack. When we
> run the unwinder, before each unwind instruction we check whether the
> current SP value is getting close to the top of the kernel stack, and
> if so, turn on additional checking:
>
> if ((ctrl.sp_high - ctrl.vrs[SP]) < sizeof(ctrl.vrs))
> ctrl.check_each_pop = 1;
>
> that will ensure if we go off the top of the kernel stack, the
> unwinder will report failure, and not access those addresses.
>
> After each instruction, we check whether the SP value is within the
> above bounds:
>
> if (ctrl.vrs[SP] < low || ctrl.vrs[SP] >= ctrl.sp_high)
> return -URC_FAILURE;
>
> This means that the unwinder can never modify SP to point outside of
> the kernel stack region identified by low..ctrl.sp_high, thereby
> protecting the load inside unwind_pop_register() from ever
> dereferencing something outside of the kernel stack. Moreover, it also
> prevents the unwinder modifying SP to point below the current stack
> frame.
>
> The problem has been introduced by trying to make the unwinder cope
> with IRQ stacks in b6506981f880 ("ARM: unwind: support unwinding across
> multiple stacks"):
>
> - if (!load_sp)
> + if (!load_sp) {
> ctrl->vrs[SP] = (unsigned long)vsp;
> + } else {
> + ctrl->sp_low = ctrl->vrs[SP];
> + ctrl->sp_high = ALIGN(ctrl->sp_low, THREAD_SIZE);
> + }
>
> Now, whenever SP is loaded, we reset the allowable range for the SP
> value, and this completely defeats the protections we previously had
> which were ensuring that:
>
> 1) the SP value doesn't jump back _down_ the kernel stack resulting
> in an infinite unwind loop.
> 2) the SP value doesn't end up outside the kernel stack.
>
> We need those protections to prevent these problems that are being
> reported - and the most efficient way I can think of doing that is to
> somehow valudate the new SP value _before_ we modify sp_low and
> sp_high, so these two limits are always valid.
>
> Merely changing the READ_ONCE_NOCHECK() to be get_kernel_nocheck()
> will only partly fix this problem - it will stop the unwinder oopsing
> the kernel, but definitely doesn't protect against (1) and doesn't
> protect against SP pointing at some thing that is accessible (e.g.
> a device or other kernel memory.)
>
> We're yet again at Thursday, with the last linux-next prior to the
> merge window being created this evening, which really doesn't leave
> much time to get this sorted... and we can't say "this code should
> have been in earlier in the cycle" this time around, because these
> changes to the unwinder have been present in linux-next prior to
> 5.17-rc2. Annoyingly, it seems merging stuff earlier in the cycle
> doesn't actually solve the problem of these last minute debugging
> panics.
>
> Any suggestions for what we should do? Can we come up with some way
> to validate the new SP value before 6pm UTC this evening?

Also, looking deeper at the last linaro job report:

https://lkft.validation.linaro.org/scheduler/job/4688599#L684

the dumped memory doesn't look like an exception stack. If it was,
e82aab40 would be the saved CPSR value and c388eb80 would be the PC
value, both of which are nonsense.

The stack that we were in (and we dumped out in full) was:

Stack: (0xc381bb30 to 0xc381c000)

and the exception stack (the saved pt_regs) is:

r0 r1 r2 r3 r4
bfa0: c2ba47c0 0000000a c2ba1358 ffff9537 c2c05d00 00400140 c62d5624 c1948b20
r5 r6 r7 r8 r9 r10 fp ip
bfc0: e82ab498 c62d5400 c35377a0 c62d5404 25706000 c381bfe8 c62d5400 00000001
sp lr pc cpsr orig_r0
bfe0: c5fcfc48 c036251c c0995a14 20040013 ffffffff c5fcfc7c c62d5400 c0300bf0

but, we end up dumping out:

Exception stack(0xc5fcfc98 to 0xc5fcfce0)
fc80: b6a3ab70 00000004
fca0: 00000000 00000004 b6a3ab70 c055f928 c388eb80 c5fcfd40 00000000 c5fcfd50
fcc0: 00000005 00000051 e82aad1c c03ae570 00000000 c388eb80 c3512a20 e82aab40

Firstly, that's in the wrong stack to be dumping for the exception
stack, and secondly, why is it 0x50 bytes above the saved SP value from
the real exception stack - that makes no sense in itself.

--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

2022-03-11 21:15:24

[permalink] [raw]

Subject: Re: [next] arm: Internal error: Oops: 5 PC is at __read_once_word_nocheck

On Thu, Mar 10, 2022 at 11:06:17PM +0100, Ard Biesheuvel wrote:
> On Thu, 10 Mar 2022 at 22:50, Naresh Kamboju <[email protected]> wrote:
> >
> > On Fri, 11 Mar 2022 at 02:55, Ard Biesheuvel <[email protected]> wrote:
> > >
> > > On Thu, 10 Mar 2022 at 22:18, Naresh Kamboju <[email protected]> wrote:
> > > >
> > > > Hi Ard and Russell,
> > > >
> > > > The boot test pass on linux next-20220310 tag with KASAN=y on BeagleBoard x15
> > > > device. but LTP cve tests reproduced the reported kernel crash [1].
> > > > From the available historical data I can confirm that this is an
> > > > intermittent issue on
> > > > BeagleBoard x15 devices.
> > > >
> > > > OTOH, the kernel crash is always reproducible on qemu-arm with KASAN=y
> > > > while booting which has been known to fail for a long time.
> > > >
> > > > From the Ardb tree I have boot tested qemu-arm with KASAN=y the reported
> > > > kernel crash is always reproducible.
> > > >
> > > > The build steps [3] and extra Kconfigs.
> > > >
> > > > - Naresh
> > > > [1] https://lkft.validation.linaro.org/scheduler/job/4701310
> > > > [2] https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/
> > > > [3] https://builds.tuxbuild.com/2661dIAPUjE2DMJvye91He2gus0/tuxmake_reproducer.sh
> > >
> > > Thanks Naresh. I'm having trouble to make sense of this, though. The
> > > linked output log appears to be from a build that lacks my 'ARM:
> > > entry: fix unwinder problems caused by IRQ stacks' patch, as it
> > > doesn't show any occurrences of call_with_stack() on any of the call
> > > stacks.
> > >
> > > Do you have a link to the vmlinux and zImage files for this build?
> >
> > Yes.
> >
> > vmlinux.xz: https://builds.tuxbuild.com/26BmIasJnAyCii0SkgbKarkF369/vmlinux.xz
> > zImage: https://builds.tuxbuild.com/26BmIasJnAyCii0SkgbKarkF369/zImage
> > System.map: https://builds.tuxbuild.com/26BmIasJnAyCii0SkgbKarkF369/System.map
> > Build log: https://builds.tuxbuild.com/26BmIasJnAyCii0SkgbKarkF369/
> >
>
> This kernel does not appear to have
>
> ARM: unwind: set frame.pc correctly for current-thread unwinding
> ARM: entry: fix unwinder problems caused by IRQ stacks
> ARM: Revert "unwind: dump exception stack from calling frame"
>
> so it is expected that the same issue is still being observed.
>
> Could you please try -next with those patches applied?

I concur, from my inspection of the above referenced vmlinux file.

--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

2022-03-11 23:21:15

[permalink] [raw]

Subject: Re: [next] arm: Internal error: Oops: 5 PC is at __read_once_word_nocheck

On Fri, 11 Mar 2022 at 02:55, Ard Biesheuvel <[email protected]> wrote:
>
> On Thu, 10 Mar 2022 at 22:18, Naresh Kamboju <[email protected]> wrote:
> >
> > Hi Ard and Russell,
> >
> > The boot test pass on linux next-20220310 tag with KASAN=y on BeagleBoard x15
> > device. but LTP cve tests reproduced the reported kernel crash [1].
> > From the available historical data I can confirm that this is an
> > intermittent issue on
> > BeagleBoard x15 devices.
> >
> > OTOH, the kernel crash is always reproducible on qemu-arm with KASAN=y
> > while booting which has been known to fail for a long time.
> >
> > From the Ardb tree I have boot tested qemu-arm with KASAN=y the reported
> > kernel crash is always reproducible.
> >
> > The build steps [3] and extra Kconfigs.
> >
> > - Naresh
> > [1] https://lkft.validation.linaro.org/scheduler/job/4701310
> > [2] https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/
> > [3] https://builds.tuxbuild.com/2661dIAPUjE2DMJvye91He2gus0/tuxmake_reproducer.sh
>
> Thanks Naresh. I'm having trouble to make sense of this, though. The
> linked output log appears to be from a build that lacks my 'ARM:
> entry: fix unwinder problems caused by IRQ stacks' patch, as it
> doesn't show any occurrences of call_with_stack() on any of the call
> stacks.
>
> Do you have a link to the vmlinux and zImage files for this build?

Yes.

vmlinux.xz: https://builds.tuxbuild.com/26BmIasJnAyCii0SkgbKarkF369/vmlinux.xz
zImage: https://builds.tuxbuild.com/26BmIasJnAyCii0SkgbKarkF369/zImage
System.map: https://builds.tuxbuild.com/26BmIasJnAyCii0SkgbKarkF369/System.map
Build log: https://builds.tuxbuild.com/26BmIasJnAyCii0SkgbKarkF369/

- Naresh

2022-03-14 14:47:42

[permalink] [raw]

Subject: Re: [next] arm: Internal error: Oops: 5 PC is at __read_once_word_nocheck

Hi Ard and Russell,

Your three patches applied and tested on x15 tested by Daniel and reported
kernel crash did not find it after multiple iterations.

ARM: unwind: set frame.pc correctly for current-thread unwinding
ARM: entry: fix unwinder problems caused by IRQ stacks
ARM: Revert "unwind: dump exception stack from calling frame"

Tested-by: Linux Kernel Functional Testing <[email protected]>

Build url:
https://builds.tuxbuild.com/26DZbOeAxshMvtU2FhS3Fytr7NS/

vmlinux: https://builds.tuxbuild.com/26DZbOeAxshMvtU2FhS3Fytr7NS/vmlinux.xz
zImage: https://builds.tuxbuild.com/26DZbOeAxshMvtU2FhS3Fytr7NS/zImage
System.map: https://builds.tuxbuild.com/26DZbOeAxshMvtU2FhS3Fytr7NS/System.map

LAVA test jobs ran on x15 device:
https://lkft.validation.linaro.org/scheduler/job/4702341
https://lkft.validation.linaro.org/scheduler/job/4702344
https://lkft.validation.linaro.org/scheduler/job/4702348
https://lkft.validation.linaro.org/scheduler/job/4702350
https://lkft.validation.linaro.org/scheduler/job/4702352
https://lkft.validation.linaro.org/scheduler/job/4702354

- Naresh

2022-03-14 17:23:42