2020-04-28 14:45:45

by Rasmus Villemoes

[permalink] [raw]
Subject: [PATCH -rt] hrtimer: fix logic for when grabbing softirq_expiry_lock can be elided

Commit

hrtimer: Add a missing bracket and hide `migration_base' on !SMP

which is 47b6de0b7f22 in 5.2-rt and 40aae5708e7a in 4.19-rt,
inadvertently changed the logic from base != &migration_base to base
== &migration_base.

On !CONFIG_SMP, the effect was to effectively always elide this
lock/unlock pair (since is_migration_base() is unconditionally false),
which for me consistently causes lockups during reboot, and reportedly
also often causes a hang during boot.

Adding this logical negation (or, what is effectively the same thing
on !CONFIG_SMP, reverting the above commit as well as "hrtimer:
Prevent using hrtimer_grab_expiry_lock() on migration_base") fixes
that lockup.

Fixes: 40aae5708e7a (hrtimer: Add a missing bracket and hide `migration_base' on !SMP) # 4.19-rt
Fixes: 47b6de0b7f22 (hrtimer: Add a missing bracket and hide `migration_base' on !SMP) # 5.2-rt
Signed-off-by: Rasmus Villemoes <[email protected]>
---
Something like this? I wasn't sure what Fixes: tag(s) to include, if
any. It's quite possible the same fix is needed on earlier -rt
kernels, I didn't check.

kernel/time/hrtimer.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index e54a95de8b79..c3966c090246 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -953,7 +953,7 @@ void hrtimer_grab_expiry_lock(const struct hrtimer *timer)
{
struct hrtimer_clock_base *base = READ_ONCE(timer->base);

- if (timer->is_soft && is_migration_base(base)) {
+ if (timer->is_soft && !is_migration_base(base)) {
spin_lock(&base->cpu_base->softirq_expiry_lock);
spin_unlock(&base->cpu_base->softirq_expiry_lock);
}
--
2.23.0


2020-04-28 14:56:18

by Steven Rostedt

[permalink] [raw]
Subject: Re: [PATCH -rt] hrtimer: fix logic for when grabbing softirq_expiry_lock can be elided

On Tue, 28 Apr 2020 16:40:26 +0200
Rasmus Villemoes <[email protected]> wrote:

> Commit
>
> hrtimer: Add a missing bracket and hide `migration_base' on !SMP
>
> which is 47b6de0b7f22 in 5.2-rt and 40aae5708e7a in 4.19-rt,
> inadvertently changed the logic from base != &migration_base to base
> == &migration_base.
>
> On !CONFIG_SMP, the effect was to effectively always elide this
> lock/unlock pair (since is_migration_base() is unconditionally false),
> which for me consistently causes lockups during reboot, and reportedly
> also often causes a hang during boot.
>
> Adding this logical negation (or, what is effectively the same thing
> on !CONFIG_SMP, reverting the above commit as well as "hrtimer:
> Prevent using hrtimer_grab_expiry_lock() on migration_base") fixes
> that lockup.
>
> Fixes: 40aae5708e7a (hrtimer: Add a missing bracket and hide `migration_base' on !SMP) # 4.19-rt
> Fixes: 47b6de0b7f22 (hrtimer: Add a missing bracket and hide `migration_base' on !SMP) # 5.2-rt
> Signed-off-by: Rasmus Villemoes <[email protected]>
> ---
> Something like this? I wasn't sure what Fixes: tag(s) to include, if
> any. It's quite possible the same fix is needed on earlier -rt
> kernels, I didn't check.
>
> kernel/time/hrtimer.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
> index e54a95de8b79..c3966c090246 100644
> --- a/kernel/time/hrtimer.c
> +++ b/kernel/time/hrtimer.c
> @@ -953,7 +953,7 @@ void hrtimer_grab_expiry_lock(const struct hrtimer *timer)
> {
> struct hrtimer_clock_base *base = READ_ONCE(timer->base);
>
> - if (timer->is_soft && is_migration_base(base)) {
> + if (timer->is_soft && !is_migration_base(base)) {

That was my sloppiness in not seeing that 5.2-rt had == and 4.19 had !=.

Thanks for tracking this down!

Reviewed-by: Steven Rostedt (VMware) <[email protected]>

-- Steve

> spin_lock(&base->cpu_base->softirq_expiry_lock);
> spin_unlock(&base->cpu_base->softirq_expiry_lock);
> }

2020-04-28 15:23:36

by Tom Zanussi

[permalink] [raw]
Subject: Re: [PATCH -rt] hrtimer: fix logic for when grabbing softirq_expiry_lock can be elided

On Tue, 2020-04-28 at 16:40 +0200, Rasmus Villemoes wrote:
> Commit
>
> hrtimer: Add a missing bracket and hide `migration_base' on !SMP
>
> which is 47b6de0b7f22 in 5.2-rt and 40aae5708e7a in 4.19-rt,
> inadvertently changed the logic from base != &migration_base to base
> == &migration_base.
>
> On !CONFIG_SMP, the effect was to effectively always elide this
> lock/unlock pair (since is_migration_base() is unconditionally
> false),
> which for me consistently causes lockups during reboot, and
> reportedly
> also often causes a hang during boot.
>
> Adding this logical negation (or, what is effectively the same thing
> on !CONFIG_SMP, reverting the above commit as well as "hrtimer:
> Prevent using hrtimer_grab_expiry_lock() on migration_base") fixes
> that lockup.
>
> Fixes: 40aae5708e7a (hrtimer: Add a missing bracket and hide
> `migration_base' on !SMP) # 4.19-rt
> Fixes: 47b6de0b7f22 (hrtimer: Add a missing bracket and hide
> `migration_base' on !SMP) # 5.2-rt
> Signed-off-by: Rasmus Villemoes <[email protected]>
> ---
> Something like this? I wasn't sure what Fixes: tag(s) to include, if
> any. It's quite possible the same fix is needed on earlier -rt
> kernels, I didn't check.
>

Yeah, looks good, I'll post a new update with this shortly.

Thanks,

Tom

> kernel/time/hrtimer.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
> index e54a95de8b79..c3966c090246 100644
> --- a/kernel/time/hrtimer.c
> +++ b/kernel/time/hrtimer.c
> @@ -953,7 +953,7 @@ void hrtimer_grab_expiry_lock(const struct
> hrtimer *timer)
> {
> struct hrtimer_clock_base *base = READ_ONCE(timer->base);
>
> - if (timer->is_soft && is_migration_base(base)) {
> + if (timer->is_soft && !is_migration_base(base)) {
> spin_lock(&base->cpu_base->softirq_expiry_lock);
> spin_unlock(&base->cpu_base->softirq_expiry_lock);
> }