As the following log:
where we experience a CPU hard lockup. The assembly code (disassembled by gdb)
0xc06c6e90 <__tcp_select_window+148>: beq 0xc06c6eb0<__tcp_select_window+180>
0xc06c6e94 <__tcp_select_window+152>: mov r2, #1008; 0x3f0
0xc06c6e98 <__tcp_select_window+156>: ldr r5, [r0,#1004] ; 0x3ec
0xc06c6e9c <__tcp_select_window+160>: ldrh r2, [r0,r2]
....
0xc06c6ee0 <__tcp_select_window+228>: addne r0, r0, #1
0xc06c6ee4 <__tcp_select_window+232>: lslne r0, r0, r2
0xc06c6ee8 <__tcp_select_window+236>: ldmne sp, {r4, r5,r11, sp,pc}
Could either the “strhi”/”strlo” pair, or the lslne/ldmne pair, be
tripping over errata 818325, or a similar errata?
0xc06c6eec <__tcp_select_window+240>: b 0xc06c6f40<__tcp_select_window+324>
This is patch can fix the *hard lock* in some case.
As the Russell said:
"in other words, which can be handled by updating a control register in the firmware or
boot loader"
Maybe the better solution is in firmware.
Others, I'm no sure this workaround patch if can be accepted.
I resend this patch for getting some suggestion from you.
--
Thanks!
Huang Tao (1):
ARM: errata: Workaround for Cortex-A12 erratum 818325
arch/arm/Kconfig | 13 +++++++++++++
arch/arm/mm/proc-v7.S | 12 ++++++++++++
2 files changed, 25 insertions(+)
--
1.9.1
From: Huang Tao <[email protected]>
On Cortex-A12 (r0p0..r0p1-00lac0-rc11), when a CPU executes a sequence of
two conditional store instructions with opposite condition code and
updating the same register, the system might enter a deadlock if the
second conditional instruction is an UNPREDICTABLE STR or STM
instruction. This workaround setting bit[12] of the Feature Register
prevents the erratum. This bit disables an optimisation applied to a
sequence of 2 instructions that use opposing condition codes.
Signed-off-by: Huang Tao <[email protected]>
Signed-off-by: Kever Yang <[email protected]>
Signed-off-by: Caesar Wang <[email protected]>
---
arch/arm/Kconfig | 13 +++++++++++++
arch/arm/mm/proc-v7.S | 12 ++++++++++++
2 files changed, 25 insertions(+)
diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 639411f..554b57a 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -1263,6 +1263,19 @@ config ARM_ERRATA_773022
loop buffer may deliver incorrect instructions. This
workaround disables the loop buffer to avoid the erratum.
+config ARM_ERRATA_818325
+ bool "ARM errata: Execution of an UNPREDICTABLE STR or STM instruction might deadlock"
+ depends on CPU_V7
+ help
+ This option enables the workaround for the 818325 Cortex-A12
+ (r0p0..r0p1-00lac0-rc11) erratum. When a CPU executes a sequence of
+ two conditional store instructions with opposite condition code and
+ updating the same register, the system might enter a deadlock if the
+ second conditional instruction is an UNPREDICTABLE STR or STM
+ instruction. This workaround setting bit[12] of the Feature Register
+ prevents the erratum. This bit disables an optimisation applied to a
+ sequence of 2 instructions that use opposing condition codes.
+
endmenu
source "arch/arm/common/Kconfig"
diff --git a/arch/arm/mm/proc-v7.S b/arch/arm/mm/proc-v7.S
index de2b246..2b338ec 100644
--- a/arch/arm/mm/proc-v7.S
+++ b/arch/arm/mm/proc-v7.S
@@ -439,6 +439,18 @@ __v7_setup_cont:
teq r0, r10
beq __ca9_errata
+ /* Cortex-A12 Errata */
+ ldr r10, =0x00000c0d @ Cortex-A12 primary part number
+ teq r0, r10
+ bne 5f
+#ifdef CONFIG_ARM_ERRATA_818325
+ teq r6, #0x00 @ present in r0p0
+ teqne r6, #0x01 @ present in r0p1-00lac0-rc11
+ mrceq p15, 0, r10, c15, c0, 1 @ read diagnostic register
+ orreq r10, r10, #1 << 12 @ set bit #12
+ mcreq p15, 0, r10, c15, c0, 1 @ write diagnostic register
+ isb
+#endif
/* Cortex-A15 Errata */
ldr r10, =0x00000c0f @ Cortex-A15 primary part number
teq r0, r10
--
1.9.1
On Tuesday 03 November 2015 16:10:09 Caesar Wang wrote:
>
> + /* Cortex-A12 Errata */
> + ldr r10, =0x00000c0d @ Cortex-A12 primary part number
> + teq r0, r10
> + bne 5f
> +#ifdef CONFIG_ARM_ERRATA_818325
> + teq r6, #0x00 @ present in r0p0
> + teqne r6, #0x01 @ present in r0p1-00lac0-rc11
> + mrceq p15, 0, r10, c15, c0, 1 @ read diagnostic register
> + orreq r10, r10, #1 << 12 @ set bit #12
> + mcreq p15, 0, r10, c15, c0, 1 @ write diagnostic register
> + isb
> +#endif
> /* Cortex-A15 Errata */
>
Does this still build? You seem to have lost the '5:' label.
Arnd
在 2015年11月03日 16:45, Arnd Bergmann 写道:
> On Tuesday 03 November 2015 16:10:09 Caesar Wang wrote:
>> + /* Cortex-A12 Errata */
>> + ldr r10, =0x00000c0d @ Cortex-A12 primary part number
>> + teq r0, r10
>> + bne 5f
beq __ca15_errata:
>> +#ifdef CONFIG_ARM_ERRATA_818325
>> + teq r6, #0x00 @ present in r0p0
>> + teqne r6, #0x01 @ present in r0p1-00lac0-rc11
>> + mrceq p15, 0, r10, c15, c0, 1 @ read diagnostic register
>> + orreq r10, r10, #1 << 12 @ set bit #12
>> + mcreq p15, 0, r10, c15, c0, 1 @ write diagnostic register
>> + isb
>> +#endif
>> /* Cortex-A15 Errata */
>>
> Does this still build? You seem to have lost the '5:' label.
No, I didn't have build in next kernel.
Yup, the patch need a bit change from the message.
commit 17e7bf86690eaad4906d2295f0bd171cc194633b
Author: Russell King <[email protected]>
Date: Sat Apr 4 21:34:33 2015 +0100
ARM: proc-v7: move CPU errata out of line
-----
Original patch:
https://patchwork.kernel.org/patch/4735341/
Applied and verified on kernel V3.14.
>
> Arnd
>
> _______________________________________________
> linux-arm-kernel mailing list
> [email protected]
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
--
Thanks,
Caesar
Hi Huang,
[auto build test ERROR on mvebu/for-next -- if it's inappropriate base, please suggest rules for selecting the more suitable base]
url: https://github.com/0day-ci/linux/commits/Caesar-Wang/ARM-errata-Workaround-for-Cortex-A12-erratum-818325/20151103-163417
config: arm-prima2_defconfig (attached as .config)
reproduce:
wget https://git.kernel.org/cgit/linux/kernel/git/wfg/lkp-tests.git/plain/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=arm
All errors (new ones prefixed by >>):
/tmp/ccjF0uyl.s: Assembler messages:
>> /tmp/ccjF0uyl.s: Error: local label `"5" (instance number 1 of a fb label)' is not defined
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
From: Huang Tao <[email protected]>
On Cortex-A12 (r0p0..r0p1-00lac0-rc11), when a CPU executes a sequence of
two conditional store instructions with opposite condition code and
updating the same register, the system might enter a deadlock if the
second conditional instruction is an UNPREDICTABLE STR or STM
instruction. This workaround setting bit[12] of the Feature Register
prevents the erratum. This bit disables an optimisation applied to a
sequence of 2 instructions that use opposing condition codes.
Signed-off-by: Huang Tao <[email protected]>
Signed-off-by: Kever Yang <[email protected]>
Signed-off-by: Caesar Wang <[email protected]>
---
Changes in v1:
- fix the build error.
arch/arm/Kconfig | 13 +++++++++++++
arch/arm/mm/proc-v7.S | 16 ++++++++++++++++
2 files changed, 29 insertions(+)
diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 639411f..554b57a 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -1263,6 +1263,19 @@ config ARM_ERRATA_773022
loop buffer may deliver incorrect instructions. This
workaround disables the loop buffer to avoid the erratum.
+config ARM_ERRATA_818325
+ bool "ARM errata: Execution of an UNPREDICTABLE STR or STM instruction might deadlock"
+ depends on CPU_V7
+ help
+ This option enables the workaround for the 818325 Cortex-A12
+ (r0p0..r0p1-00lac0-rc11) erratum. When a CPU executes a sequence of
+ two conditional store instructions with opposite condition code and
+ updating the same register, the system might enter a deadlock if the
+ second conditional instruction is an UNPREDICTABLE STR or STM
+ instruction. This workaround setting bit[12] of the Feature Register
+ prevents the erratum. This bit disables an optimisation applied to a
+ sequence of 2 instructions that use opposing condition codes.
+
endmenu
source "arch/arm/common/Kconfig"
diff --git a/arch/arm/mm/proc-v7.S b/arch/arm/mm/proc-v7.S
index de2b246..e95c83c 100644
--- a/arch/arm/mm/proc-v7.S
+++ b/arch/arm/mm/proc-v7.S
@@ -351,6 +351,17 @@ __ca9_errata:
#endif
b __errata_finish
+__ca12_errata:
+#ifdef CONFIG_ARM_ERRATA_818325
+ teq r6, #0x00 @ present in r0p0
+ teqne r6, #0x01 @ present in r0p1-00lac0-rc11
+ mrceq p15, 0, r10, c15, c0, 1 @ read diagnostic register
+ orreq r10, r10, #1 << 12 @ set bit #12
+ mcreq p15, 0, r10, c15, c0, 1 @ write diagnostic register
+ isb
+#endif
+ b __errata_finish
+
__ca15_errata:
#ifdef CONFIG_ARM_ERRATA_773022
cmp r6, #0x4 @ only present up to r0p4
@@ -439,6 +450,11 @@ __v7_setup_cont:
teq r0, r10
beq __ca9_errata
+ /* Cortex-A12 Errata */
+ ldr r10, =0x00000c0d @ Cortex-A12 primary part number
+ teq r0, r10
+ beq __ca12_errata
+
/* Cortex-A15 Errata */
ldr r10, =0x00000c0f @ Cortex-A15 primary part number
teq r0, r10
--
1.9.1
On Tue, Nov 03, 2015 at 04:10:08PM +0800, Caesar Wang wrote:
> As the Russell said:
> "in other words, which can be handled by updating a control register in
> the firmware or boot loader"
> Maybe the better solution is in firmware.
The full quote is:
"I think we're at the point where we start insisting that workarounds
which are simple enable/disable feature bit operations (in other words,
which can be handled by updating a control register in the firmware or
boot loader) must be done that way, and we are not going to add such
workarounds to the kernel anymore."
The position hasn't changed. Workarounds such as this should be handled
in the firmware/boot loader before control is passed to the kernel.
The reason is very simple: if the C compiler can generate code which
triggers the bug, it can generate code which triggers the bug in the
boot loader. So, the only place such workarounds can be done is before
any C code gets executed. Putting such workarounds in the kernel is
completely inappropriate.
Sorry, I'm not going to accept this workaround into the kernel.
--
FTTC broadband for 0.8mile line: currently at 9.6Mbps down 400kbps up
according to speedtest.net.
On Tue, Nov 03, 2015 at 04:10:08PM +0800, Caesar Wang wrote:
> As the following log:
> where we experience a CPU hard lockup. The assembly code (disassembled by gdb)
>
> 0xc06c6e90 <__tcp_select_window+148>: beq 0xc06c6eb0<__tcp_select_window+180>
> 0xc06c6e94 <__tcp_select_window+152>: mov r2, #1008; 0x3f0
> 0xc06c6e98 <__tcp_select_window+156>: ldr r5, [r0,#1004] ; 0x3ec
> 0xc06c6e9c <__tcp_select_window+160>: ldrh r2, [r0,r2]
> ....
>
> 0xc06c6ee0 <__tcp_select_window+228>: addne r0, r0, #1
> 0xc06c6ee4 <__tcp_select_window+232>: lslne r0, r0, r2
> 0xc06c6ee8 <__tcp_select_window+236>: ldmne sp, {r4, r5,r11, sp,pc}
>
> Could either the “strhi”/”strlo” pair, or the lslne/ldmne pair, be
> tripping over errata 818325, or a similar errata?
No. One of the conditions for #818325 is:
The second instruction is an UNPREDICTABLE STR or STM (maximum two2
registers in the list) with write-back and the write-back register is
in the list of stored registers.
I don't see either of those in your code snippet above, but then I don't
see your strhi/strlo either. What's going on?
> 0xc06c6eec <__tcp_select_window+240>: b 0xc06c6f40<__tcp_select_window+324>
>
> This is patch can fix the *hard lock* in some case.
>
> As the Russell said:
> "in other words, which can be handled by updating a control register in the firmware or
> boot loader"
Russell is completely correct: this should be worked around in firmware.
There are a number of reasons for that:
(1) You want the workaround enabled for all privilege and security
levels, which means applying it before you enter the kernel.
(2) If Linux boots in non-secure, then the workaround may silently
fail to apply.
(3) The CPU may have an ECO fix, in which case we wouldn't want to
enable the workaround.
(4) Some workarounds (albeit not this one, afaict) require changing
CPU configuration that can only be done very early on, e.g. whilst
"the memory system is idle".
Now, I appreciate that doing this in the kernel may be the easiest thing
for your particular SoC, but that doesn't necessarily mean that it's the
best thing to do in the mainline kernel. Whilst there *is* precedent for
this already, we've been trying to move away from setting these bits in
the kernel for the reasons mentioned above.
Will
Hello Russell:
在 2015年11月03日 19:14, Russell King - ARM Linux 写道:
> On Tue, Nov 03, 2015 at 04:10:08PM +0800, Caesar Wang wrote:
>> As the Russell said:
>> "in other words, which can be handled by updating a control register in
>> the firmware or boot loader"
>> Maybe the better solution is in firmware.
>
> The full quote is:
>
> "I think we're at the point where we start insisting that workarounds
> which are simple enable/disable feature bit operations (in other words,
> which can be handled by updating a control register in the firmware or
> boot loader) must be done that way, and we are not going to add such
> workarounds to the kernel anymore."
>
> The position hasn't changed. Workarounds such as this should be handled
> in the firmware/boot loader before control is passed to the kernel.
>
> The reason is very simple: if the C compiler can generate code which
> triggers the bug, it can generate code which triggers the bug in the
> boot loader. So, the only place such workarounds can be done is before
> any C code gets executed. Putting such workarounds in the kernel is
> completely inappropriate.
I agree with your reason for CPU0. But how about CPU1~3 if we don't use
any firmware such as ARM Trusted Firmware to take control of CPU power
on? If the CPU1~3 will run on Linux when its first instruction is running?
BTW I don't want to argue with you the workaround is right or wrong
because I know the errata just happen on r0p0 not r0p1.
>
> Sorry, I'm not going to accept this workaround into the kernel.
It seems we should introduce some code outside the kernel to do such
initialization?
Hi,
On Tue, Nov 3, 2015 at 3:30 AM, Will Deacon <[email protected]> wrote:
> On Tue, Nov 03, 2015 at 04:10:08PM +0800, Caesar Wang wrote:
>> As the following log:
>> where we experience a CPU hard lockup. The assembly code (disassembled by gdb)
>>
>> 0xc06c6e90 <__tcp_select_window+148>: beq 0xc06c6eb0<__tcp_select_window+180>
>> 0xc06c6e94 <__tcp_select_window+152>: mov r2, #1008; 0x3f0
>> 0xc06c6e98 <__tcp_select_window+156>: ldr r5, [r0,#1004] ; 0x3ec
>> 0xc06c6e9c <__tcp_select_window+160>: ldrh r2, [r0,r2]
>> ....
>>
>> 0xc06c6ee0 <__tcp_select_window+228>: addne r0, r0, #1
>> 0xc06c6ee4 <__tcp_select_window+232>: lslne r0, r0, r2
>> 0xc06c6ee8 <__tcp_select_window+236>: ldmne sp, {r4, r5,r11, sp,pc}
>>
>> Could either the “strhi”/”strlo” pair, or the lslne/ldmne pair, be
>> tripping over errata 818325, or a similar errata?
>
> No. One of the conditions for #818325 is:
>
> The second instruction is an UNPREDICTABLE STR or STM (maximum two2
> registers in the list) with write-back and the write-back register is
> in the list of stored registers.
>
> I don't see either of those in your code snippet above, but then I don't
> see your strhi/strlo either. What's going on?
It looks like Caesar is proposing that this errata is the root cause
for some hard lockups we're seeing on rk3288 Chromebooks. I agree
with folks here that say this isn't terribly likely, but I always like
to be proven wrong. ;)
We've got code that samples / prints CPU_DBGPCSR at the time of a hard
lockup. That register isn't 100% accurate about where a CPU is, but
it's better than nothing (technically there may be ways to actually
use the DBG registers to stop the remote CPU and maybe give more info,
but I digress).
When CPUs are hard locked up, they are often found at:
<c0117c8c> v7_coherent_kern_range+0x58/0x74
or
<c0118278> v7wbi_flush_user_tlb_range+0x30/0x38
That made me think that an errata might be the root cause of our hard
lockups, since ARM errata often trigger in cache/tlb functions. I
think Caesar dug up this old errata fix in response to my suggestion.
If you know of any ARM errata that might trigger hard lockups like
this, I'd certainly be all ears. It's also possible that we've got
something running at too low of a voltage or we've got clock dividers
or cache timings programmed incorrectly somewhere. To give a more
full disassembly of one of the crashes:
<4>[ 1623.480846] SMP: failed to stop secondary CPUs
<3>[ 1623.480862] CPU1 PC: <c01827e8> __unqueue_futex+0x68/0x88
<3>[ 1623.480879] CPU2 PC: <c0117c8c> v7_coherent_kern_range+0x58/0x74
<3>[ 1623.480895] CPU3 PC: <c0118268> v7wbi_flush_user_tlb_range+0x20/0x38
---
c01827dc: e2841010 add r1, r4, #16
c01827e0: e2445004 sub r5, r4, #4
c01827e4: eb068d33 bl c0325cb8 <plist_del> (File
Offset: 0x235cb8)
=> c01827e8: f595f000 pldw [r5]
c01827ec: e1953f9f ldrex r3, [r5]
c01827f0: e2433001 sub r3, r3, #1
c01827f4: e1852f93 strex r2, r3, [r5]
c01827f8: e3320000 teq r2, #0
c01827fc: 1afffffa bne c01827ec
<__unqueue_futex+0x6c> (File Offset: 0x927ec)
c0182800: e89da830 ldm sp, {r4, r5, fp, sp, pc}
---
c0117c80: e08cc002 add ip, ip, r2
c0117c84: e15c0001 cmp ip, r1
c0117c88: 3afffffb bcc c0117c7c
<v7_coherent_kern_range+0x48> (File Offset: 0x27c7c)
=> c0117c8c: e3a00000 mov r0, #0
c0117c90: ee070fd1 mcr 15, 0, r0, cr7, cr1, {6}
c0117c94: f57ff04a dsb ishst
c0117c98: f57ff06f isb sy
c0117c9c: e1a0f00e mov pc, lr
---
c0118260: e1830600 orr r0, r3, r0, lsl #12
c0118264: e1a01601 lsl r1, r1, #12
=> c0118268: ee080f33 mcr 15, 0, r0, cr8, cr3, {1}
c011826c: e2800a01 add r0, r0, #4096 ; 0x1000
c0118270: e1500001 cmp r0, r1
c0118274: 3afffffb bcc c0118268
<v7wbi_flush_user_tlb_range+0x20> (File Offset: 0x28268)
c0118278: f57ff04b dsb ish
c011827c: e1a0f00e mov pc, lr
On Tue, Nov 03, 2015 at 11:00:20AM -0800, Doug Anderson wrote:
> Hi,
Hey Doug,
> When CPUs are hard locked up, they are often found at:
>
> <c0117c8c> v7_coherent_kern_range+0x58/0x74
> or
> <c0118278> v7wbi_flush_user_tlb_range+0x30/0x38
>
> That made me think that an errata might be the root cause of our hard
> lockups, since ARM errata often trigger in cache/tlb functions. I
> think Caesar dug up this old errata fix in response to my suggestion.
I still don't see how 818325 is related, since there aren't any conditional
stores in the sequences below.
> If you know of any ARM errata that might trigger hard lockups like
> this, I'd certainly be all ears. It's also possible that we've got
> something running at too low of a voltage or we've got clock dividers
> or cache timings programmed incorrectly somewhere. To give a more
> full disassembly of one of the crashes:
>
> <4>[ 1623.480846] SMP: failed to stop secondary CPUs
> <3>[ 1623.480862] CPU1 PC: <c01827e8> __unqueue_futex+0x68/0x88
> <3>[ 1623.480879] CPU2 PC: <c0117c8c> v7_coherent_kern_range+0x58/0x74
> <3>[ 1623.480895] CPU3 PC: <c0118268> v7wbi_flush_user_tlb_range+0x20/0x38
>
> ---
Do you have any register values for these CPUs?
> c01827dc: e2841010 add r1, r4, #16
> c01827e0: e2445004 sub r5, r4, #4
> c01827e4: eb068d33 bl c0325cb8 <plist_del> (File
> Offset: 0x235cb8)
> => c01827e8: f595f000 pldw [r5]
> c01827ec: e1953f9f ldrex r3, [r5]
> c01827f0: e2433001 sub r3, r3, #1
> c01827f4: e1852f93 strex r2, r3, [r5]
> c01827f8: e3320000 teq r2, #0
> c01827fc: 1afffffa bne c01827ec
> <__unqueue_futex+0x6c> (File Offset: 0x927ec)
> c0182800: e89da830 ldm sp, {r4, r5, fp, sp, pc}
For example, the futex address in r5 ...
> c0117c80: e08cc002 add ip, ip, r2
> c0117c84: e15c0001 cmp ip, r1
> c0117c88: 3afffffb bcc c0117c7c
> <v7_coherent_kern_range+0x48> (File Offset: 0x27c7c)
> => c0117c8c: e3a00000 mov r0, #0
> c0117c90: ee070fd1 mcr 15, 0, r0, cr7, cr1, {6}
> c0117c94: f57ff04a dsb ishst
> c0117c98: f57ff06f isb sy
> c0117c9c: e1a0f00e mov pc, lr
... the address in r0 for the cache maintenance ...
> c0118260: e1830600 orr r0, r3, r0, lsl #12
> c0118264: e1a01601 lsl r1, r1, #12
> => c0118268: ee080f33 mcr 15, 0, r0, cr8, cr3, {1}
> c011826c: e2800a01 add r0, r0, #4096 ; 0x1000
> c0118270: e1500001 cmp r0, r1
> c0118274: 3afffffb bcc c0118268
> <v7wbi_flush_user_tlb_range+0x20> (File Offset: 0x28268)
> c0118278: f57ff04b dsb ish
> c011827c: e1a0f00e mov pc, lr
... and the address in r0 for the TLBI.
Are the cores executing instructions at this point, or by "hard LOCKUP"
do you mean that they're deadlocked in hardware?
Will
Will,
On Fri, Nov 6, 2015 at 4:17 AM, Will Deacon <[email protected]> wrote:
> On Tue, Nov 03, 2015 at 11:00:20AM -0800, Doug Anderson wrote:
>> Hi,
>
> Hey Doug,
>
>> When CPUs are hard locked up, they are often found at:
>>
>> <c0117c8c> v7_coherent_kern_range+0x58/0x74
>> or
>> <c0118278> v7wbi_flush_user_tlb_range+0x30/0x38
>>
>> That made me think that an errata might be the root cause of our hard
>> lockups, since ARM errata often trigger in cache/tlb functions. I
>> think Caesar dug up this old errata fix in response to my suggestion.
>
> I still don't see how 818325 is related, since there aren't any conditional
> stores in the sequences below.
>
>> If you know of any ARM errata that might trigger hard lockups like
>> this, I'd certainly be all ears. It's also possible that we've got
>> something running at too low of a voltage or we've got clock dividers
>> or cache timings programmed incorrectly somewhere. To give a more
>> full disassembly of one of the crashes:
>>
>> <4>[ 1623.480846] SMP: failed to stop secondary CPUs
>> <3>[ 1623.480862] CPU1 PC: <c01827e8> __unqueue_futex+0x68/0x88
>> <3>[ 1623.480879] CPU2 PC: <c0117c8c> v7_coherent_kern_range+0x58/0x74
>> <3>[ 1623.480895] CPU3 PC: <c0118268> v7wbi_flush_user_tlb_range+0x20/0x38
>>
>> ---
>
> Do you have any register values for these CPUs?
No, unfortunately not. The only reason I have the PCs is because we
have code to sample CPU_DBGPCSR at hard lockup time (actually any
panic time). There's no equivalent for other registers. The code
does try to sample a number of times, so the fact that we only have
one PC value for each of the other CPUs implies that they are either
totally stuck or running in a very tight loop (from experimentation,
if you are running in a tight loop of just a few instructions the
CPU_DBGPCSR for a CPU may or may not update).
If you're curious, you can see rockchip_panic_notify() in
<https://chromium.googlesource.com/chromiumos/third_party/kernel/+/chromeos-3.14/arch/arm/mach-rockchip/rockchip.c>.
It's basically some code that's been ported forward from code in an
old Android tree and it's not beautiful, but it's better than nothing.
The code only runs if the panic notifier failed to stop the other CPUs
in a normal way.
Technically (I think) I saw something in the CPU debug registers that
would actually allow me to force another CPU to stop. That might let
me gain control over it and inspect the other registers. Doing that
is probably beyond what I have time for right now, though.
>> c01827dc: e2841010 add r1, r4, #16
>> c01827e0: e2445004 sub r5, r4, #4
>> c01827e4: eb068d33 bl c0325cb8 <plist_del> (File
>> Offset: 0x235cb8)
>> => c01827e8: f595f000 pldw [r5]
>> c01827ec: e1953f9f ldrex r3, [r5]
>> c01827f0: e2433001 sub r3, r3, #1
>> c01827f4: e1852f93 strex r2, r3, [r5]
>> c01827f8: e3320000 teq r2, #0
>> c01827fc: 1afffffa bne c01827ec
>> <__unqueue_futex+0x6c> (File Offset: 0x927ec)
>> c0182800: e89da830 ldm sp, {r4, r5, fp, sp, pc}
>
> For example, the futex address in r5 ...
>
>> c0117c80: e08cc002 add ip, ip, r2
>> c0117c84: e15c0001 cmp ip, r1
>> c0117c88: 3afffffb bcc c0117c7c
>> <v7_coherent_kern_range+0x48> (File Offset: 0x27c7c)
>> => c0117c8c: e3a00000 mov r0, #0
>> c0117c90: ee070fd1 mcr 15, 0, r0, cr7, cr1, {6}
>> c0117c94: f57ff04a dsb ishst
>> c0117c98: f57ff06f isb sy
>> c0117c9c: e1a0f00e mov pc, lr
>
> ... the address in r0 for the cache maintenance ...
>
>> c0118260: e1830600 orr r0, r3, r0, lsl #12
>> c0118264: e1a01601 lsl r1, r1, #12
>> => c0118268: ee080f33 mcr 15, 0, r0, cr8, cr3, {1}
>> c011826c: e2800a01 add r0, r0, #4096 ; 0x1000
>> c0118270: e1500001 cmp r0, r1
>> c0118274: 3afffffb bcc c0118268
>> <v7wbi_flush_user_tlb_range+0x20> (File Offset: 0x28268)
>> c0118278: f57ff04b dsb ish
>> c011827c: e1a0f00e mov pc, lr
>
> ... and the address in r0 for the TLBI.
>
> Are the cores executing instructions at this point, or by "hard LOCKUP"
> do you mean that they're deadlocked in hardware?
If they are executing, they aren't executing much. That means it's
likely they're deadlocked in hardware.
-Doug