We got many crashs in for_each_frame+0x18 arch/arm/lib/backtrace.S
1003: ldr r2, [sv_pc, #-4]
The backtrace is
dump_backtrace
show_stack
sched_show_task
show_state_filter
sysrq_handle_showstate_blocked
__handle_sysrq
write_sysrq_trigger
proc_reg_write
__vfs_write
vfs_write
sys_write
Related Kernel config
CONFIG_CPU_SW_DOMAIN_PAN=y
# CONFIG_ARM_UNWIND is not set
CONFIG_FRAME_POINTER=y
The task A was dumping the stack of an UN task B. However, the task B
scheduled to run on another CPU, which cause it stack content changed.
Then, task A may hit a page domain fault and die().
[520.661314] Unhandled fault: page domain fault (0x01b) at 0x32848c02
The addr 0x32848c02 is a valid user-space address.
PAGE DIRECTORY: d1854000
PGD: d1854ca0 => bb21e835
PMD: d1854ca0 => bb21e835
PTE: bb21e120 => afffa79f
With CONFIG_CPU_SW_DOMAIN_PAN=y, a page domain fault occurred.
{ do_bad, SIGSEGV, SEGV_ACCERR, "page domain fault"},
Without check the __ex_table entry, do_bad() just return fault and die().
.pushsection __ex_table,"a"
.long 1003b, 1006b
This patch try __ex_table in do_bad(), the same as in __do_kernel_fault().
Signed-off-by: Lvqiang <[email protected]>
---
arch/arm/mm/fault.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
index bd0f482..22f45df 100644
--- a/arch/arm/mm/fault.c
+++ b/arch/arm/mm/fault.c
@@ -487,11 +487,14 @@ static inline bool access_error(unsigned int fsr, struct vm_area_struct *vma)
#endif /* CONFIG_ARM_LPAE */
/*
- * This abort handler always returns "fault".
+ * Checks __ex_table before returns "fault".
*/
static int
do_bad(unsigned long addr, unsigned int fsr, struct pt_regs *regs)
{
+ if (fixup_exception(regs))
+ return 0;
+
return 1;
}
--
1.7.9.5
On Thu, Nov 07, 2019 at 03:45:13PM +0800, Lvqiang wrote:
>
> We got many crashs in for_each_frame+0x18 arch/arm/lib/backtrace.S
> 1003: ldr r2, [sv_pc, #-4]
>
> The backtrace is
> dump_backtrace
> show_stack
> sched_show_task
> show_state_filter
> sysrq_handle_showstate_blocked
> __handle_sysrq
> write_sysrq_trigger
> proc_reg_write
> __vfs_write
> vfs_write
> sys_write
>
> Related Kernel config
> CONFIG_CPU_SW_DOMAIN_PAN=y
> # CONFIG_ARM_UNWIND is not set
> CONFIG_FRAME_POINTER=y
>
> The task A was dumping the stack of an UN task B. However, the task B
What is "an UN task B"?
> scheduled to run on another CPU, which cause it stack content changed.
> Then, task A may hit a page domain fault and die().
> [520.661314] Unhandled fault: page domain fault (0x01b) at 0x32848c02
So, the backtrace code is trying to access userspace. It isn't supposed
to be accessing userspace - there are no guarantees that userspace will
be using frame pointers. That is the bug.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up
Hi Russell,
Thanks a lot for the reply!
UN means TASK_INTERRUPTIBLE.
Task A found the Task B was in TASK_INTERRUPTIBLE.
But just during try to get the backtrace of Task B, the Task B changed to TASK_RUNNING
Task B push and pop to it stack during executing, so the stack context of task B changed a lot.
But Task A calculated and pop a value as sv_fp of Task B.
1002: ldr sv_fp, [frame, #-12] @ get saved fp
But, the task B had been TASK_RUNNING, sv_fp Task A get can be any value chaned by the executing of Task B.
It can be an accessible user-space address of Task A's address space.
If we enable the CONFIG_ARM_UNWIND, the crash is gone.
-----Original Message-----
From: Russell King - ARM Linux admin [mailto:[email protected]]
Sent: Thursday, November 07, 2019 5:24 PM
To: ????ǿ (Lvqiang Huang)
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH] ARM: check __ex_table in do_bad()
On Thu, Nov 07, 2019 at 03:45:13PM +0800, Lvqiang wrote:
>
> We got many crashs in for_each_frame+0x18 arch/arm/lib/backtrace.S
> 1003: ldr r2, [sv_pc, #-4]
>
> The backtrace is
> dump_backtrace
> show_stack
> sched_show_task
> show_state_filter
> sysrq_handle_showstate_blocked
> __handle_sysrq
> write_sysrq_trigger
> proc_reg_write
> __vfs_write
> vfs_write
> sys_write
>
> Related Kernel config
> CONFIG_CPU_SW_DOMAIN_PAN=y
> # CONFIG_ARM_UNWIND is not set
> CONFIG_FRAME_POINTER=y
>
> The task A was dumping the stack of an UN task B. However, the task B
What is "an UN task B"?
> scheduled to run on another CPU, which cause it stack content changed.
> Then, task A may hit a page domain fault and die().
> [520.661314] Unhandled fault: page domain fault (0x01b) at
> 0x32848c02
So, the backtrace code is trying to access userspace. It isn't supposed to be accessing userspace - there are no guarantees that userspace will be using frame pointers. That is the bug.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up According to speedtest.net: 11.9Mbps down 500kbps up
============================================================================
This email (including its attachments) is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. Unauthorized use, dissemination, distribution or copying of this email or the information herein or taking any action in reliance on the contents of this email or the information herein, by anyone other than the intended recipient, or an employee or agent responsible for delivering the message to the intended recipient, is strictly prohibited. If you are not the intended recipient, please do not read, copy, use or disclose any part of this e-mail to others. Please notify the sender immediately and permanently delete this e-mail and any attachments if you received it in error. Internet communications cannot be guaranteed to be timely, secure, error-free or virus-free. The sender does not accept liability for any errors or omissions.
???ʼ????丽?????б??????ʣ??ܷ??ɱ???????й¶???????????ʼ???ָ?ض??ռ??ˡ??Ͻ??Ǿ???Ȩʹ?á????????????????Ʊ??ʼ????????ݡ????Ǹ??ض??ռ??ˣ??????Ķ??????ơ? ʹ?û???¶???ʼ????κ????ݡ??????ձ??ʼ???????ϵͳ????????ɾ?????ʼ??????и????????Իظ??ʼ??ķ?ʽ???̸?֪?????ˡ?????֤??????ͨ?ż?ʱ????ȫ???????????????????˶??κδ?©?????е????Ρ?
> ?? 2019??11??7?գ?17:24??Russell King - ARM Linux admin <[email protected]> д????
>
>> On Thu, Nov 07, 2019 at 03:45:13PM +0800, Lvqiang wrote:
>>
>> We got many crashs in for_each_frame+0x18 arch/arm/lib/backtrace.S
>> 1003: ldr r2, [sv_pc, #-4]
>>
>> The backtrace is
>> dump_backtrace
>> show_stack
>> sched_show_task
>> show_state_filter
>> sysrq_handle_showstate_blocked
>> __handle_sysrq
>> write_sysrq_trigger
>> proc_reg_write
>> __vfs_write
>> vfs_write
>> sys_write
>>
>> Related Kernel config
>> CONFIG_CPU_SW_DOMAIN_PAN=y
>> # CONFIG_ARM_UNWIND is not set
>> CONFIG_FRAME_POINTER=y
>>
>> The task A was dumping the stack of an UN task B. However, the task B
>
> What is "an UN task B"?
UN means TASK_UNINTERRUPTIBLE.
(Sorry for the typo in the last reply)
>> scheduled to run on another CPU, which cause it stack content changed.
>> Then, task A may hit a page domain fault and die().
>> [520.661314] Unhandled fault: page domain fault (0x01b) at 0x32848c02
>
> So, the backtrace code is trying to access userspace. It isn't supposed
> to be accessing userspace - there are no guarantees that userspace will
> be using frame pointers. That is the bug.
>
There is a race condition when try to get the backtrace of another task??whose frames may totally changed during the execution.
> --
> RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
> FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
> According to speedtest.net: 11.9Mbps down 500kbps up
============================================================================
This email (including its attachments) is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. Unauthorized use, dissemination, distribution or copying of this email or the information herein or taking any action in reliance on the contents of this email or the information herein, by anyone other than the intended recipient, or an employee or agent responsible for delivering the message to the intended recipient, is strictly prohibited. If you are not the intended recipient, please do not read, copy, use or disclose any part of this e-mail to others. Please notify the sender immediately and permanently delete this e-mail and any attachments if you received it in error. Internet communications cannot be guaranteed to be timely, secure, error-free or virus-free. The sender does not accept liability for any errors or omissions.
???ʼ????丽?????б??????ʣ??ܷ??ɱ???????й¶???????????ʼ???ָ?ض??ռ??ˡ??Ͻ??Ǿ???Ȩʹ?á????????????????Ʊ??ʼ????????ݡ????Ǹ??ض??ռ??ˣ??????Ķ??????ơ? ʹ?û???¶???ʼ????κ????ݡ??????ձ??ʼ???????ϵͳ????????ɾ?????ʼ??????и????????Իظ??ʼ??ķ?ʽ???̸?֪?????ˡ?????֤??????ͨ?ż?ʱ????ȫ???????????????????˶??κδ?©?????е????Ρ?
Sorry for not having described it clearly, please let me add some more information.
The kernel log for the scenario
[20461.271374] sysrq: SysRq : Show Blocked State
[20461.271405] task PC stack pid father
[20461.271436] mbox-send-threa D c08cfad8 0 38 2 0x00000000
/*and ignore some logs abort the backtrace dump of some TASK_UNINTERRUPTIBLE tasks */
[20461.273387] fsck.exfat D c08cfad8 0 6221 2276 0x00000000
[20461.273408] Backtrace:
[20461.273430] [<c08cf5d0>] (__schedule) from [<c08cff84>] (schedule+0x90/0xa8)
[20461.273442] r10:ce009ef0 r9:ce009df4 r8:c0d0790c r7:00000082 r6:7fffffff r5:00000000
[20461.273477] r4:ce008000
[20461.273497] [<c08cfef4>] (schedule) from [<c08d2b90>] (schedule_timeout+0x2c/0x26c)
[20461.273509] r4:7fffffff r3:dc8ba693
[20461.273561] Unhandled fault: page domain fault (0x01b) at 0x32848c02
[20461.273576] pgd = d1854000
[20461.273587] [32848c02] *pgd=bb21e835
[20461.273607] Internal error: : 1b [#1] PREEMPT SMP ARM
[20461.278903] CPU: 2 PID: 5917 Comm: watchdog Tainted: G W O 4.4.147+ #1
[20461.278929] task: e9beecc0 task.stack: e30a4000
[20461.278949] PC is at for_each_frame+0x18/0x88
[20461.278965] LR is at vprintk_emit+0x470/0x4ec
The Task A: finally crashed task, PID: 5917 Comm: watchdog, running on CPU 2, dumping backtrace of all UN tasks.
The Task B: TASK_UNINTERRUPTIBLE to TASK_RUNNING when Task A is trying to dump its backtrace.
The first 2 frames dump for task B are ok, see
[20461.273430] [<c08cf5d0>] (__schedule) from [<c08cff84>] (schedule+0x90/0xa8)
[20461.273497] [<c08cfef4>] (schedule) from [<c08d2b90>] (schedule_timeout+0x2c/0x26c)
Then task A crashed:
[20461.273561] Unhandled fault: page domain fault (0x01b) at 0x32848c02
From the RAM dump after kernel crash, we can see Task B had been scheduled to running on CPU 0.
crash_arm> ps 6221
PID PPID CPU TASK ST %MEM VSZ RSS COMM
> 6221 2276 0 cde04880 RU 0.4 17784 13596 fsck.exfat
And the backtrace should changed, which cause the crash of Task A.
crash_arm> bt 6221
PID: 6221 TASK: cde04880 CPU: 0 COMMAND: "fsck.exfat"
#0 [<c0117a5c>] (__kunmap_atomic) from [<c0413ae8>]
#1 [<c0413894>] (copy_page_to_iter) from [<c01f4788>]
#2 [<c01f439c>] (generic_file_read_iter) from [<c02725e8>]
#3 [<c027257c>] (blkdev_read_iter) from [<c023b5b0>]
#4 [<c023b4f8>] (__vfs_read) from [<c023bd04>]
#5 [<c023bc78>] (vfs_read) from [<c023c7e0>]
#6 [<c023c76c>] (sys_pread64) from [<c01079a0>]
This is the race condition, try to backtrace another task is not safe. We can't assume the task won't be scheduled to execution during the backtrace dump. The stack frame should totally change once execute again.
The __ex_table entry in @for_each_frame should adding for this scenario. But with CONFIG_CPU_SW_DOMAIN_PAN=y, page domain fault may hit and go the do_bad() instead of do_page_fault().
The path may not an optimal solution, I just want to point out the problem, and is there any concern if we check __ex_table in do_bad()?
Now, our project had enabled CONFIG_ARM_UNWIND=y, it will fail to get an unwind_idx when get a wrong sv_pc, then the unwind abort without kernel crash.
-----Original Message-----
From: ????ǿ (Lvqiang Huang)
Sent: Friday, November 08, 2019 1:23 AM
To: Russell King - ARM Linux admin
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH] ARM: check __ex_table in do_bad()
> ?? 2019??11??7?գ?17:24??Russell King - ARM Linux admin
> <[email protected]> д????
>
>> On Thu, Nov 07, 2019 at 03:45:13PM +0800, Lvqiang wrote:
>>
>> We got many crashs in for_each_frame+0x18 arch/arm/lib/backtrace.S
>> 1003: ldr r2, [sv_pc, #-4]
>>
>> The backtrace is
>> dump_backtrace
>> show_stack
>> sched_show_task
>> show_state_filter
>> sysrq_handle_showstate_blocked
>> __handle_sysrq
>> write_sysrq_trigger
>> proc_reg_write
>> __vfs_write
>> vfs_write
>> sys_write
>>
>> Related Kernel config
>> CONFIG_CPU_SW_DOMAIN_PAN=y
>> # CONFIG_ARM_UNWIND is not set
>> CONFIG_FRAME_POINTER=y
>>
>> The task A was dumping the stack of an UN task B. However, the task B
>
> What is "an UN task B"?
UN means TASK_UNINTERRUPTIBLE.
(Sorry for the typo in the last reply)
>> scheduled to run on another CPU, which cause it stack content changed.
>> Then, task A may hit a page domain fault and die().
>> [520.661314] Unhandled fault: page domain fault (0x01b) at
>> 0x32848c02
>
> So, the backtrace code is trying to access userspace. It isn't
> supposed to be accessing userspace - there are no guarantees that
> userspace will be using frame pointers. That is the bug.
>
There is a race condition when try to get the backtrace of another task??whose frames may totally changed during the execution.
> --
> RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
> FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down
> 622kbps up According to speedtest.net: 11.9Mbps down 500kbps up
============================================================================
This email (including its attachments) is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. Unauthorized use, dissemination, distribution or copying of this email or the information herein or taking any action in reliance on the contents of this email or the information herein, by anyone other than the intended recipient, or an employee or agent responsible for delivering the message to the intended recipient, is strictly prohibited. If you are not the intended recipient, please do not read, copy, use or disclose any part of this e-mail to others. Please notify the sender immediately and permanently delete this e-mail and any attachments if you received it in error. Internet communications cannot be guaranteed to be timely, secure, error-free or virus-free. The sender does not accept liability for any errors or omissions.
???ʼ????丽?????б??????ʣ??ܷ??ɱ???????й¶???????????ʼ???ָ?ض??ռ??ˡ??Ͻ??Ǿ???Ȩʹ?á????????????????Ʊ??ʼ????????ݡ????Ǹ??ض??ռ??ˣ??????Ķ??????ơ? ʹ?û???¶???ʼ????κ????ݡ??????ձ??ʼ???????ϵͳ????????ɾ?????ʼ??????и????????Իظ??ʼ??ķ?ʽ???̸?֪?????ˡ?????֤??????ͨ?ż?ʱ????ȫ???????????????????˶??κδ?©?????е????Ρ?