If a spinner is present, there is a chance that the load of
rwsem_has_spinner() in rwsem_wake() can be reordered with
respect to decrement of rwsem count in __up_write() leading
to wakeup being missed.
spinning writer up_write caller
--------------- -----------------------
[S] osq_unlock() [L] osq
spin_lock(wait_lock)
sem->count=0xFFFFFFFF00000001
+0xFFFFFFFF00000000
count=sem->count
MB
sem->count=0xFFFFFFFE00000001
-0xFFFFFFFF00000001
RMB
spin_trylock(wait_lock)
return
rwsem_try_write_lock(count)
spin_unlock(wait_lock)
schedule()
Reordering of atomic_long_sub_return_release() in __up_write()
and rwsem_has_spinner() in rwsem_wake() can cause missing of
wakeup in up_write() context. In spinning writer, sem->count
and local variable count is 0XFFFFFFFE00000001. It would result
in rwsem_try_write_lock() failing to acquire rwsem and spinning
writer going to sleep in rwsem_down_write_failed().
The smp_rmb() will make sure that the spinner state is
consulted after sem->count is updated in up_write context.
Signed-off-by: Prateek Sood <[email protected]>
---
kernel/locking/rwsem-xadd.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 45 insertions(+)
diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
index 34e727f..5c687f6 100644
--- a/kernel/locking/rwsem-xadd.c
+++ b/kernel/locking/rwsem-xadd.c
@@ -586,6 +586,51 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem)
DEFINE_WAKE_Q(wake_q);
/*
+ * If a spinner is present, there is a chance that the load of
+ * rwsem_has_spinner() in rwsem_wake() can be reordered with
+ * respect to decrement of rwsem count in __up_write() leading
+ * to wakeup being missed.
+ *
+ * spinning writer up_write caller
+ * --------------- -----------------------
+ * [S] osq_unlock() [L] osq
+ * spin_lock(wait_lock)
+ * sem->count=0xFFFFFFFF00000001
+ * +0xFFFFFFFF00000000
+ * count=sem->count
+ * MB
+ * sem->count=0xFFFFFFFE00000001
+ * -0xFFFFFFFF00000001
+ * RMB
+ * spin_trylock(wait_lock)
+ * return
+ * rwsem_try_write_lock(count)
+ * spin_unlock(wait_lock)
+ * schedule()
+ *
+ * Reordering of atomic_long_sub_return_release() in __up_write()
+ * and rwsem_has_spinner() in rwsem_wake() can cause missing of
+ * wakeup in up_write() context. In spinning writer, sem->count
+ * and local variable count is 0XFFFFFFFE00000001. It would result
+ * in rwsem_try_write_lock() failing to acquire rwsem and spinning
+ * writer going to sleep in rwsem_down_write_failed().
+ *
+ *
+ * The RMB in below example is to make sure that the spinner state is
+ * consulted after sem->count is updated in up_write context.
+ * This would guarantee trylock on rwsem is successful
+ * in rwsem_down_write_failed().
+ * spinning writer up_write caller
+ * --------------- -----------------------
+ * [S] osq_unlock() atomic_update(sem->count)
+ * RMB
+ * atomic_update(sem->count) [L] osq
+ * MB
+ * rwsem_try_write_lock(count)
+ */
+ smp_rmb();
+
+ /*
* If a spinner is present, it is not necessary to do the wakeup.
* Try to do wakeup only if the trylock succeeds to minimize
* spinlock contention which may introduce too much delay in the
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center, Inc.,
is a member of Code Aurora Forum, a Linux Foundation Collaborative Project.
WTH did you not Cc the people that commented on your patch last time?
On Wed, Aug 23, 2017 at 04:58:55PM +0530, Prateek Sood wrote:
> If a spinner is present, there is a chance that the load of
> rwsem_has_spinner() in rwsem_wake() can be reordered with
> respect to decrement of rwsem count in __up_write() leading
> to wakeup being missed.
> spinning writer up_write caller
> --------------- -----------------------
> [S] osq_unlock() [L] osq
> spin_lock(wait_lock)
> sem->count=0xFFFFFFFF00000001
> +0xFFFFFFFF00000000
> count=sem->count
> MB
> sem->count=0xFFFFFFFE00000001
> -0xFFFFFFFF00000001
> RMB
This doesn't make sense, it appears to order a STORE against something
else.
> spin_trylock(wait_lock)
> return
> rwsem_try_write_lock(count)
> spin_unlock(wait_lock)
> schedule()
On Thu, Aug 24, 2017 at 01:29:27PM +0200, Peter Zijlstra wrote:
>
> WTH did you not Cc the people that commented on your patch last time?
>
> On Wed, Aug 23, 2017 at 04:58:55PM +0530, Prateek Sood wrote:
> > If a spinner is present, there is a chance that the load of
> > rwsem_has_spinner() in rwsem_wake() can be reordered with
> > respect to decrement of rwsem count in __up_write() leading
> > to wakeup being missed.
>
> > spinning writer up_write caller
> > --------------- -----------------------
> > [S] osq_unlock() [L] osq
> > spin_lock(wait_lock)
> > sem->count=0xFFFFFFFF00000001
> > +0xFFFFFFFF00000000
> > count=sem->count
> > MB
> > sem->count=0xFFFFFFFE00000001
> > -0xFFFFFFFF00000001
> > RMB
>
> This doesn't make sense, it appears to order a STORE against something
> else.
>
> > spin_trylock(wait_lock)
> > return
> > rwsem_try_write_lock(count)
> > spin_unlock(wait_lock)
> > schedule()
Is this what you wanted to write?
---
kernel/locking/rwsem-xadd.c | 27 +++++++++++++++++++++++++++
1 file changed, 27 insertions(+)
diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
index 02f660666ab8..813b5d3654ce 100644
--- a/kernel/locking/rwsem-xadd.c
+++ b/kernel/locking/rwsem-xadd.c
@@ -613,6 +613,33 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem)
DEFINE_WAKE_Q(wake_q);
/*
+ * __rwsem_down_write_failed_common(sem)
+ * rwsem_optimistic_spin(sem)
+ * osq_unlock(sem->osq)
+ * ...
+ * atomic_long_add_return(&sem->count)
+ *
+ * - VS -
+ *
+ * __up_write()
+ * if (atomic_long_sub_return_release(&sem->count) < 0)
+ * rwsem_wake(sem)
+ * osq_is_locked(&sem->osq)
+ *
+ * And __up_write() must observe !osq_is_locked() when it observes the
+ * atomic_long_add_return() in order to not miss a wakeup.
+ *
+ * This boils down to:
+ *
+ * [S.rel] X = 1 [RmW] r0 = (Y += 0)
+ * MB RMB
+ * [RmW] Y += 1 [L] r1 = X
+ *
+ * exists (r0=1 /\ r1=0)
+ */
+ smp_rmb();
+
+ /*
* If a spinner is present, it is not necessary to do the wakeup.
* Try to do wakeup only if the trylock succeeds to minimize
* spinlock contention which may introduce too much delay in the
On Thu, Aug 24, 2017 at 02:33:04PM +0200, Peter Zijlstra wrote:
> On Thu, Aug 24, 2017 at 01:29:27PM +0200, Peter Zijlstra wrote:
> >
> > WTH did you not Cc the people that commented on your patch last time?
> >
> > On Wed, Aug 23, 2017 at 04:58:55PM +0530, Prateek Sood wrote:
> > > If a spinner is present, there is a chance that the load of
> > > rwsem_has_spinner() in rwsem_wake() can be reordered with
> > > respect to decrement of rwsem count in __up_write() leading
> > > to wakeup being missed.
> >
> > > spinning writer up_write caller
> > > --------------- -----------------------
> > > [S] osq_unlock() [L] osq
> > > spin_lock(wait_lock)
> > > sem->count=0xFFFFFFFF00000001
> > > +0xFFFFFFFF00000000
> > > count=sem->count
> > > MB
> > > sem->count=0xFFFFFFFE00000001
> > > -0xFFFFFFFF00000001
> > > RMB
> >
> > This doesn't make sense, it appears to order a STORE against something
> > else.
> >
> > > spin_trylock(wait_lock)
> > > return
> > > rwsem_try_write_lock(count)
> > > spin_unlock(wait_lock)
> > > schedule()
>
> Is this what you wanted to write?
And ideally there should be a comment near the atomic_long_add_return()
in __rwsem_down_write_failed_common() to indicate we rely on the implied
smp_mb() before it -- just in case someone goes and makes it
atomic_long_add_return_relaxed().
And I suppose someone should look at the waiting branch of that thing
too.. because I'm not sure what happens if waiting is true but count
isn't big enough.
I bloody hate the rwsem code, that BIAS stuff forever confuses me. I
have a start at rewriting the thing to put the owner in the lock word
just like we now do for mutex, but never seem to get around to finishing
it.
> ---
> kernel/locking/rwsem-xadd.c | 27 +++++++++++++++++++++++++++
> 1 file changed, 27 insertions(+)
>
> diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
> index 02f660666ab8..813b5d3654ce 100644
> --- a/kernel/locking/rwsem-xadd.c
> +++ b/kernel/locking/rwsem-xadd.c
> @@ -613,6 +613,33 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem)
> DEFINE_WAKE_Q(wake_q);
>
> /*
> + * __rwsem_down_write_failed_common(sem)
> + * rwsem_optimistic_spin(sem)
> + * osq_unlock(sem->osq)
> + * ...
> + * atomic_long_add_return(&sem->count)
> + *
> + * - VS -
> + *
> + * __up_write()
> + * if (atomic_long_sub_return_release(&sem->count) < 0)
> + * rwsem_wake(sem)
> + * osq_is_locked(&sem->osq)
> + *
> + * And __up_write() must observe !osq_is_locked() when it observes the
> + * atomic_long_add_return() in order to not miss a wakeup.
> + *
> + * This boils down to:
> + *
> + * [S.rel] X = 1 [RmW] r0 = (Y += 0)
> + * MB RMB
> + * [RmW] Y += 1 [L] r1 = X
> + *
> + * exists (r0=1 /\ r1=0)
> + */
> + smp_rmb();
> +
> + /*
> * If a spinner is present, it is not necessary to do the wakeup.
> * Try to do wakeup only if the trylock succeeds to minimize
> * spinlock contention which may introduce too much delay in the
On 08/24/2017 06:22 PM, Peter Zijlstra wrote:
> On Thu, Aug 24, 2017 at 02:33:04PM +0200, Peter Zijlstra wrote:
>> On Thu, Aug 24, 2017 at 01:29:27PM +0200, Peter Zijlstra wrote:
>>>
>>> WTH did you not Cc the people that commented on your patch last time?
>>>
>>> On Wed, Aug 23, 2017 at 04:58:55PM +0530, Prateek Sood wrote:
>>>> If a spinner is present, there is a chance that the load of
>>>> rwsem_has_spinner() in rwsem_wake() can be reordered with
>>>> respect to decrement of rwsem count in __up_write() leading
>>>> to wakeup being missed.
>>>
>>>> spinning writer up_write caller
>>>> --------------- -----------------------
>>>> [S] osq_unlock() [L] osq
>>>> spin_lock(wait_lock)
>>>> sem->count=0xFFFFFFFF00000001
>>>> +0xFFFFFFFF00000000
>>>> count=sem->count
>>>> MB
>>>> sem->count=0xFFFFFFFE00000001
>>>> -0xFFFFFFFF00000001
>>>> RMB
>>>
>>> This doesn't make sense, it appears to order a STORE against something
>>> else.
>>>
>>>> spin_trylock(wait_lock)
>>>> return
>>>> rwsem_try_write_lock(count)
>>>> spin_unlock(wait_lock)
>>>> schedule()
>>
>> Is this what you wanted to write?
>
> And ideally there should be a comment near the atomic_long_add_return()
> in __rwsem_down_write_failed_common() to indicate we rely on the implied
> smp_mb() before it -- just in case someone goes and makes it
> atomic_long_add_return_relaxed().
>
> And I suppose someone should look at the waiting branch of that thing
> too.. because I'm not sure what happens if waiting is true but count
> isn't big enough.
>
> I bloody hate the rwsem code, that BIAS stuff forever confuses me. I
> have a start at rewriting the thing to put the owner in the lock word
> just like we now do for mutex, but never seem to get around to finishing
> it.
>
>> ---
>> kernel/locking/rwsem-xadd.c | 27 +++++++++++++++++++++++++++
>> 1 file changed, 27 insertions(+)
>>
>> diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
>> index 02f660666ab8..813b5d3654ce 100644
>> --- a/kernel/locking/rwsem-xadd.c
>> +++ b/kernel/locking/rwsem-xadd.c
>> @@ -613,6 +613,33 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem)
>> DEFINE_WAKE_Q(wake_q);
>>
>> /*
>> + * __rwsem_down_write_failed_common(sem)
>> + * rwsem_optimistic_spin(sem)
>> + * osq_unlock(sem->osq)
>> + * ...
>> + * atomic_long_add_return(&sem->count)
>> + *
>> + * - VS -
>> + *
>> + * __up_write()
>> + * if (atomic_long_sub_return_release(&sem->count) < 0)
>> + * rwsem_wake(sem)
>> + * osq_is_locked(&sem->osq)
>> + *
>> + * And __up_write() must observe !osq_is_locked() when it observes the
>> + * atomic_long_add_return() in order to not miss a wakeup.
>> + *
>> + * This boils down to:
>> + *
>> + * [S.rel] X = 1 [RmW] r0 = (Y += 0)
>> + * MB RMB
>> + * [RmW] Y += 1 [L] r1 = X
>> + *
>> + * exists (r0=1 /\ r1=0)
>> + */
>> + smp_rmb();
>> +
>> + /*
>> * If a spinner is present, it is not necessary to do the wakeup.
>> * Try to do wakeup only if the trylock succeeds to minimize
>> * spinlock contention which may introduce too much delay in the
Thanks Peter for your suggestion on comments.
I will resend the patch with updated comments
--
Qualcomm India Private Limited, on behalf of Qualcomm Innovation
Center, Inc., is a member of Code Aurora Forum, a Linux Foundation
Collaborative Project