2019-11-12 13:35:06

by Mark-PK Tsai

[permalink] [raw]
Subject: [PATCH] ARM: fix race in for_each_frame

The sv_pc, which is saved in the stack, may be an invalid address
if the target thread is running on another processor in the meantime.
It will cause kernel crash at `ldr r2, [sv_pc, #-4]`.

Check if sv_pc is valid before use it like unwind_frame in
arch/arm/kernel/unwind.c.

Signed-off-by: Mike-SL Lin <[email protected]>
Signed-off-by: Mark-PK Tsai <[email protected]>
---
arch/arm/lib/backtrace.S | 5 +++++
1 file changed, 5 insertions(+)

diff --git a/arch/arm/lib/backtrace.S b/arch/arm/lib/backtrace.S
index 582925238d65..84f06381bbfb 100644
--- a/arch/arm/lib/backtrace.S
+++ b/arch/arm/lib/backtrace.S
@@ -64,6 +64,11 @@ for_each_frame: tst frame, mask @ Check for address exceptions
sub sv_pc, sv_pc, offset @ Correct PC for prefetching
bic sv_pc, sv_pc, mask @ mask PC/LR for the mode

+ mov r0, sv_pc
+ bl kernel_text_address @ check if sv_pc is valid
+ cmp r0, #0 @ if sv_pc is not kernel text
+ beq 1006f @ address, abort backtrace
+
1003: ldr r2, [sv_pc, #-4] @ if stmfd sp!, {args} exists,
ldr r3, .Ldsi+4 @ adjust saved 'pc' back one
teq r3, r2, lsr #11 @ instruction
--
2.18.0


Subject: Re: [PATCH] ARM: fix race in for_each_frame


> ?? 2019??11??12?գ?21:31??Mark-PK Tsai <[email protected]> д????
>
> The sv_pc, which is saved in the stack, may be an invalid address
> if the target thread is running on another processor in the meantime.
> It will cause kernel crash at `ldr r2, [sv_pc, #-4]`.
>
> Check if sv_pc is valid before use it like unwind_frame in
> arch/arm/kernel/unwind.c.
>
> Signed-off-by: Mike-SL Lin <[email protected]>
> Signed-off-by: Mark-PK Tsai <[email protected]>
> ---
> arch/arm/lib/backtrace.S | 5 +++++
> 1 file changed, 5 insertions(+)
>
> diff --git a/arch/arm/lib/backtrace.S b/arch/arm/lib/backtrace.S
> index 582925238d65..84f06381bbfb 100644
> --- a/arch/arm/lib/backtrace.S
> +++ b/arch/arm/lib/backtrace.S
> @@ -64,6 +64,11 @@ for_each_frame: tst frame, mask @ Check for address exceptions
> sub sv_pc, sv_pc, offset @ Correct PC for prefetching
> bic sv_pc, sv_pc, mask @ mask PC/LR for the mode
>
> + mov r0, sv_pc
> + bl kernel_text_address @ check if sv_pc is valid
> + cmp r0, #0 @ if sv_pc is not kernel text
> + beq 1006f @ address, abort backtrace
> +

The sv_pc can be a kernel module text.

> 1003: ldr r2, [sv_pc, #-4] @ if stmfd sp!, {args} exists,
> ldr r3, .Ldsi+4 @ adjust saved 'pc' back one
> teq r3, r2, lsr #11 @ instruction
> --
> 2.18.0


============================================================================
This email (including its attachments) is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. Unauthorized use, dissemination, distribution or copying of this email or the information herein or taking any action in reliance on the contents of this email or the information herein, by anyone other than the intended recipient, or an employee or agent responsible for delivering the message to the intended recipient, is strictly prohibited. If you are not the intended recipient, please do not read, copy, use or disclose any part of this e-mail to others. Please notify the sender immediately and permanently delete this e-mail and any attachments if you received it in error. Internet communications cannot be guaranteed to be timely, secure, error-free or virus-free. The sender does not accept liability for any errors or omissions.
???ʼ????丽?????б??????ʣ??ܷ??ɱ???????й¶???????͸????ʼ???ָ?ض??ռ??ˡ??Ͻ??Ǿ???Ȩʹ?á????????????????Ʊ??ʼ????????ݡ????Ǹ??ض??ռ??ˣ??????Ķ??????ơ? ʹ?û???¶???ʼ????κ????ݡ??????ձ??ʼ???????ϵͳ????????ɾ?????ʼ??????и????????Իظ??ʼ??ķ?ʽ???̸?֪?????ˡ??޷???֤??????ͨ?ż?ʱ????ȫ???????????????????˶??κδ?©?????е????Ρ?

Subject: Re: [PATCH] ARM: fix race in for_each_frame

Dear Mark,
Thanks a lot for the reply.

As said in last reply, sv_pc can be a module text, then more check needed.

And beside crash at 1003, we may also get crash at 1001, the frame is invalid. (The last sv_pv is valid and sv_frame is invalid), then more check needed.

And we often show_data around the general proposal registers when kernel crash. When they contain an address mapping for a hw register but can??t access because clock gated, it will crash again because do_bad() is involved. (continuous crash in arm and hang at die_lock in arm64)

So, why not check the __ex_table in do_bad() ?

> ?? 2019??11??12?գ?21:31??Mark-PK Tsai <[email protected]> д????
>
> The sv_pc, which is saved in the stack, may be an invalid address
> if the target thread is running on another processor in the meantime.
> It will cause kernel crash at `ldr r2, [sv_pc, #-4]`.
>
> Check if sv_pc is valid before use it like unwind_frame in
> arch/arm/kernel/unwind.c.
>
> Signed-off-by: Mike-SL Lin <[email protected]>
> Signed-off-by: Mark-PK Tsai <[email protected]>
> ---
> arch/arm/lib/backtrace.S | 5 +++++
> 1 file changed, 5 insertions(+)
>
> diff --git a/arch/arm/lib/backtrace.S b/arch/arm/lib/backtrace.S
> index 582925238d65..84f06381bbfb 100644
> --- a/arch/arm/lib/backtrace.S
> +++ b/arch/arm/lib/backtrace.S
> @@ -64,6 +64,11 @@ for_each_frame: tst frame, mask @ Check for address exceptions
> sub sv_pc, sv_pc, offset @ Correct PC for prefetching
> bic sv_pc, sv_pc, mask @ mask PC/LR for the mode
>
> + mov r0, sv_pc
> + bl kernel_text_address @ check if sv_pc is valid
> + cmp r0, #0 @ if sv_pc is not kernel text
> + beq 1006f @ address, abort backtrace
> +
> 1003: ldr r2, [sv_pc, #-4] @ if stmfd sp!, {args} exists,
> ldr r3, .Ldsi+4 @ adjust saved 'pc' back one
> teq r3, r2, lsr #11 @ instruction
> --
> 2.18.0


============================================================================
This email (including its attachments) is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. Unauthorized use, dissemination, distribution or copying of this email or the information herein or taking any action in reliance on the contents of this email or the information herein, by anyone other than the intended recipient, or an employee or agent responsible for delivering the message to the intended recipient, is strictly prohibited. If you are not the intended recipient, please do not read, copy, use or disclose any part of this e-mail to others. Please notify the sender immediately and permanently delete this e-mail and any attachments if you received it in error. Internet communications cannot be guaranteed to be timely, secure, error-free or virus-free. The sender does not accept liability for any errors or omissions.
???ʼ????丽?????б??????ʣ??ܷ??ɱ???????й¶???????͸????ʼ???ָ?ض??ռ??ˡ??Ͻ??Ǿ???Ȩʹ?á????????????????Ʊ??ʼ????????ݡ????Ǹ??ض??ռ??ˣ??????Ķ??????ơ? ʹ?û???¶???ʼ????κ????ݡ??????ձ??ʼ???????ϵͳ????????ɾ?????ʼ??????и????????Իظ??ʼ??ķ?ʽ???̸?֪?????ˡ??޷???֤??????ͨ?ż?ʱ????ȫ???????????????????˶??κδ?©?????е????Ρ?

2019-11-12 15:24:21

by Mark-PK Tsai

[permalink] [raw]
Subject: Re: [PATCH] ARM: fix race in for_each_frame


> . 2019.11.12..21:31.Mark-PK Tsai <[email protected]> ...
>
> The sv_pc, which is saved in the stack, may be an invalid address
> if the target thread is running on another processor in the meantime.
> It will cause kernel crash at `ldr r2, [sv_pc, #-4]`.
>
> Check if sv_pc is valid before use it like unwind_frame in
> arch/arm/kernel/unwind.c.
>
> Signed-off-by: Mike-SL Lin <[email protected]>
> Signed-off-by: Mark-PK Tsai <[email protected]>
> ---
> arch/arm/lib/backtrace.S | 5 +++++
> 1 file changed, 5 insertions(+)
>
> diff --git a/arch/arm/lib/backtrace.S b/arch/arm/lib/backtrace.S
> index 582925238d65..84f06381bbfb 100644
> --- a/arch/arm/lib/backtrace.S
> +++ b/arch/arm/lib/backtrace.S
> @@ -64,6 +64,11 @@ for_each_frame: tst frame, mask @ Check for address exceptions
> sub sv_pc, sv_pc, offset @ Correct PC for prefetching
> bic sv_pc, sv_pc, mask @ mask PC/LR for the mode
>
> + mov r0, sv_pc
> + bl kernel_text_address @ check if sv_pc is valid
> + cmp r0, #0 @ if sv_pc is not kernel text
> + beq 1006f @ address, abort backtrace
> +

The sv_pc can be a kernel module text.

The module text area is ok for kernel_text_address().

> 1003: ldr r2, [sv_pc, #-4] @ if stmfd sp!, {args} exists,
> ldr r3, .Ldsi+4 @ adjust saved 'pc' back one
> teq r3, r2, lsr #11 @ instruction
> --
> 2.18.0

2019-11-13 03:27:08

by Mark-PK Tsai

[permalink] [raw]
Subject: Re: [PATCH] ARM: fix race in for_each_frame

> Dear Mark,
> Thanks a lot for the reply.
>
> As said in last reply, sv_pc can be a module text, then more check needed.

If sv_pc is in module text area, kernel_text_address() returns true.

>
> And beside crash at 1003, we may also get crash at 1001, the frame is invalid. (The last sv_pv is valid and sv_frame is invalid), then more check needed.

There's a basic check for sv_fp at the end of 1004.
But I'm not sure is it enough to prevent the 1001 crash you mentioned.
Should we add a verify_stack for sv_fp?

>
> And we often show_data around the general proposal registers when kernel crash. When they contain an address mapping for a hw register but can’t access because clock gated, it will crash again because do_bad() is involved. (continuous crash in arm and hang at die_lock in arm64)
>
> So, why not check the __ex_table in do_bad() ?
>

On our arm platform, kernel just die becuase the Unhandled fault in for_each_frame.
So I'd rather to fix it before the continuous crash happen.