This patchset contains a few modifications to mutex_spin_on_owner().
The first patch makes the optimistic spinner continue spinning whenever
the owner changes, and the second patch refactors mutex_spin_on_owner()
to micro optimize the code as well as make it simpler.
Jason Low (2):
mutex: In mutex_spin_on_owner(), return true when owner changes
mutex: Refactor mutex_spin_on_owner
kernel/locking/mutex.c | 47 ++++++++++++++++++++++-------------------------
1 files changed, 22 insertions(+), 25 deletions(-)
--
1.7.2.5
In the mutex_spin_on_owner(), we return true only if lock->owner == NULL.
This was beneficial in situations where there were multiple threads
simultaneously spinning for the mutex. If another thread got the lock
while other spinner(s) were also doing mutex_spin_on_owner(), then the
other spinners would stop spinning. This workaround helped reduce the
chance that many spinners were simultaneously spinning for the mutex
which can help reduce contention in highly contended cases.
However, recent changes were made to the optimistic spinning code such
that instead of having all spinners simultaneously spin for the mutex,
we queue the spinners with an MCS lock such that only one thread spins
for the mutex at a time. Furthermore, the OSQ optimizations ensure that
spinners in the queue will stop waiting if it needs to reschedule.
Now, we don't have to worry about multiple threads spinning on owner
at the same time, and if lock->owner is not NULL at this point, it likely
means another thread happens to obtain the lock in the fastpath. In this
case, it would make sense for the spinner to continue spinning as long
as the spinner doesn't need to schedule and the mutex owner is running.
This patch changes this so that mutex_spin_on_owner() returns true when
the lock owner changes, which means a thread will only stop spinning
if it either needs to reschedule or if the lock owner is not running.
We saw up to a 5% performance improvement in the fserver workload with
this patch.
Signed-off-by: Jason Low <[email protected]>
---
kernel/locking/mutex.c | 8 ++++----
1 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c
index 4541951..04ea9ce 100644
--- a/kernel/locking/mutex.c
+++ b/kernel/locking/mutex.c
@@ -237,11 +237,11 @@ int mutex_spin_on_owner(struct mutex *lock, struct task_struct *owner)
rcu_read_unlock();
/*
- * We break out the loop above on need_resched() and when the
- * owner changed, which is a sign for heavy contention. Return
- * success only when lock->owner is NULL.
+ * We break out of the loop above on either need_resched(), when
+ * the owner is not running, or when the lock owner changed.
+ * Return success only when the lock owner changed.
*/
- return lock->owner == NULL;
+ return lock->owner != owner;
}
/*
--
1.7.2.5
As suggested by Davidlohr, we could refactor mutex_spin_on_owner().
Currently, we split up owner_running() with mutex_spin_on_owner().
When the owner changes, we make duplicate owner checks which are not
necessary. It also makes the code a bit obscure as we are using a
second check to figure out why we broke out of the loop.
This patch modifies it such that we remove the owner_running() function
and the mutex_spin_on_owner loop directly checks for if the owner changes,
if the owner is not running, or if we need to reschedule. If the owner
changes, we break out of the loop and return true. If the owner is not
running or if we need to reschedule, then break out of the loop and return
false.
Cc: Davidlohr Bueso <[email protected]>
Signed-off-by: Jason Low <[email protected]>
---
kernel/locking/mutex.c | 47 ++++++++++++++++++++++-------------------------
1 files changed, 22 insertions(+), 25 deletions(-)
diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c
index 04ea9ce..218c561 100644
--- a/kernel/locking/mutex.c
+++ b/kernel/locking/mutex.c
@@ -204,44 +204,41 @@ ww_mutex_set_context_fastpath(struct ww_mutex *lock,
* Mutex spinning code migrated from kernel/sched/core.c
*/
-static inline bool owner_running(struct mutex *lock, struct task_struct *owner)
-{
- if (lock->owner != owner)
- return false;
-
- /*
- * Ensure we emit the owner->on_cpu, dereference _after_ checking
- * lock->owner still matches owner, if that fails, owner might
- * point to free()d memory, if it still matches, the rcu_read_lock()
- * ensures the memory stays valid.
- */
- barrier();
-
- return owner->on_cpu;
-}
-
/*
* Look out! "owner" is an entirely speculative pointer
* access and not reliable.
*/
static noinline
-int mutex_spin_on_owner(struct mutex *lock, struct task_struct *owner)
+bool mutex_spin_on_owner(struct mutex *lock, struct task_struct *owner)
{
+ bool ret;
+
rcu_read_lock();
- while (owner_running(lock, owner)) {
- if (need_resched())
+ while (true) {
+ /* Return success when the lock owner changed */
+ if (lock->owner != owner) {
+ ret = true;
break;
+ }
+
+ /*
+ * Ensure we emit the owner->on_cpu, dereference _after_
+ * checking lock->owner still matches owner, if that fails,
+ * owner might point to free()d memory, if it still matches,
+ * the rcu_read_lock() ensures the memory stays valid.
+ */
+ barrier();
+
+ if (!owner->on_cpu || need_resched()) {
+ ret = false;
+ break;
+ }
cpu_relax_lowlatency();
}
rcu_read_unlock();
- /*
- * We break out of the loop above on either need_resched(), when
- * the owner is not running, or when the lock owner changed.
- * Return success only when the lock owner changed.
- */
- return lock->owner != owner;
+ return ret;
}
/*
--
1.7.2.5
On Mon, 2015-02-02 at 13:59 -0800, Jason Low wrote:
> In the mutex_spin_on_owner(), we return true only if lock->owner == NULL.
> This was beneficial in situations where there were multiple threads
> simultaneously spinning for the mutex. If another thread got the lock
> while other spinner(s) were also doing mutex_spin_on_owner(), then the
> other spinners would stop spinning. This workaround helped reduce the
> chance that many spinners were simultaneously spinning for the mutex
> which can help reduce contention in highly contended cases.
>
> However, recent changes were made to the optimistic spinning code such
> that instead of having all spinners simultaneously spin for the mutex,
> we queue the spinners with an MCS lock such that only one thread spins
> for the mutex at a time. Furthermore, the OSQ optimizations ensure that
> spinners in the queue will stop waiting if it needs to reschedule.
>
> Now, we don't have to worry about multiple threads spinning on owner
> at the same time, and if lock->owner is not NULL at this point, it likely
> means another thread happens to obtain the lock in the fastpath. In this
> case, it would make sense for the spinner to continue spinning as long
> as the spinner doesn't need to schedule and the mutex owner is running.
>
> This patch changes this so that mutex_spin_on_owner() returns true when
> the lock owner changes, which means a thread will only stop spinning
> if it either needs to reschedule or if the lock owner is not running.
>
> We saw up to a 5% performance improvement in the fserver workload with
> this patch.
As pointed out, this fits in nicely with what we're doing with rwsems.
Based on our recent discussions:
> Signed-off-by: Jason Low <[email protected]>
Acked-by: Davidlohr Bueso <[email protected]>
Commit-ID: 07d2413a61db6500f58e614e873eed79d7f2ed72
Gitweb: http://git.kernel.org/tip/07d2413a61db6500f58e614e873eed79d7f2ed72
Author: Jason Low <[email protected]>
AuthorDate: Mon, 2 Feb 2015 13:59:26 -0800
Committer: Ingo Molnar <[email protected]>
CommitDate: Wed, 18 Feb 2015 16:57:07 +0100
locking/mutex: In mutex_spin_on_owner(), return true when owner changes
In the mutex_spin_on_owner(), we return true only if lock->owner == NULL.
This was beneficial in situations where there were multiple threads
simultaneously spinning for the mutex. If another thread got the lock
while other spinner(s) were also doing mutex_spin_on_owner(), then the
other spinners would stop spinning. This workaround helped reduce the
chance that many spinners were simultaneously spinning for the mutex
which can help reduce contention in highly contended cases.
However, recent changes were made to the optimistic spinning code such
that instead of having all spinners simultaneously spin for the mutex,
we queue the spinners with an MCS lock such that only one thread spins
for the mutex at a time. Furthermore, the OSQ optimizations ensure that
spinners in the queue will stop waiting if it needs to reschedule.
Now, we don't have to worry about multiple threads spinning on owner
at the same time, and if lock->owner is not NULL at this point, it likely
means another thread happens to obtain the lock in the fastpath. In this
case, it would make sense for the spinner to continue spinning as long
as the spinner doesn't need to schedule and the mutex owner is running.
This patch changes this so that mutex_spin_on_owner() returns true when
the lock owner changes, which means a thread will only stop spinning
if it either needs to reschedule or if the lock owner is not running.
We saw up to a 5% performance improvement in the fserver workload with
this patch.
Signed-off-by: Jason Low <[email protected]>
Signed-off-by: Peter Zijlstra (Intel) <[email protected]>
Acked-by: Davidlohr Bueso <[email protected]>
Cc: Aswin Chandramouleeswaran <[email protected]>
Cc: Linus Torvalds <[email protected]>
Cc: Paul E. McKenney <[email protected]>
Cc: Tim Chen <[email protected]>
Cc: [email protected]
Cc: [email protected]
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Ingo Molnar <[email protected]>
---
kernel/locking/mutex.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c
index 94674e5..49cce44 100644
--- a/kernel/locking/mutex.c
+++ b/kernel/locking/mutex.c
@@ -250,11 +250,11 @@ int mutex_spin_on_owner(struct mutex *lock, struct task_struct *owner)
rcu_read_unlock();
/*
- * We break out the loop above on need_resched() and when the
- * owner changed, which is a sign for heavy contention. Return
- * success only when lock->owner is NULL.
+ * We break out of the loop above on either need_resched(), when
+ * the owner is not running, or when the lock owner changed.
+ * Return success only when the lock owner changed.
*/
- return lock->owner == NULL;
+ return lock->owner != owner;
}
/*
Commit-ID: be1f7bf217ebb1e42190d7d0b332c89ea7871378
Gitweb: http://git.kernel.org/tip/be1f7bf217ebb1e42190d7d0b332c89ea7871378
Author: Jason Low <[email protected]>
AuthorDate: Mon, 2 Feb 2015 13:59:27 -0800
Committer: Ingo Molnar <[email protected]>
CommitDate: Wed, 18 Feb 2015 16:57:08 +0100
locking/mutex: Refactor mutex_spin_on_owner()
As suggested by Davidlohr, we could refactor mutex_spin_on_owner().
Currently, we split up owner_running() with mutex_spin_on_owner().
When the owner changes, we make duplicate owner checks which are not
necessary. It also makes the code a bit obscure as we are using a
second check to figure out why we broke out of the loop.
This patch modifies it such that we remove the owner_running() function
and the mutex_spin_on_owner() loop directly checks for if the owner changes,
if the owner is not running, or if we need to reschedule. If the owner
changes, we break out of the loop and return true. If the owner is not
running or if we need to reschedule, then break out of the loop and return
false.
Suggested-by: Davidlohr Bueso <[email protected]>
Signed-off-by: Jason Low <[email protected]>
Signed-off-by: Peter Zijlstra (Intel) <[email protected]>
Cc: Aswin Chandramouleeswaran <[email protected]>
Cc: Linus Torvalds <[email protected]>
Cc: Paul E. McKenney <[email protected]>
Cc: Tim Chen <[email protected]>
Cc: [email protected]
Cc: [email protected]
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Ingo Molnar <[email protected]>
---
kernel/locking/mutex.c | 47 ++++++++++++++++++++++-------------------------
1 file changed, 22 insertions(+), 25 deletions(-)
diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c
index 49cce44..59cd6c3 100644
--- a/kernel/locking/mutex.c
+++ b/kernel/locking/mutex.c
@@ -217,44 +217,41 @@ ww_mutex_set_context_slowpath(struct ww_mutex *lock,
}
#ifdef CONFIG_MUTEX_SPIN_ON_OWNER
-static inline bool owner_running(struct mutex *lock, struct task_struct *owner)
-{
- if (lock->owner != owner)
- return false;
-
- /*
- * Ensure we emit the owner->on_cpu, dereference _after_ checking
- * lock->owner still matches owner, if that fails, owner might
- * point to free()d memory, if it still matches, the rcu_read_lock()
- * ensures the memory stays valid.
- */
- barrier();
-
- return owner->on_cpu;
-}
-
/*
* Look out! "owner" is an entirely speculative pointer
* access and not reliable.
*/
static noinline
-int mutex_spin_on_owner(struct mutex *lock, struct task_struct *owner)
+bool mutex_spin_on_owner(struct mutex *lock, struct task_struct *owner)
{
+ bool ret;
+
rcu_read_lock();
- while (owner_running(lock, owner)) {
- if (need_resched())
+ while (true) {
+ /* Return success when the lock owner changed */
+ if (lock->owner != owner) {
+ ret = true;
break;
+ }
+
+ /*
+ * Ensure we emit the owner->on_cpu, dereference _after_
+ * checking lock->owner still matches owner, if that fails,
+ * owner might point to free()d memory, if it still matches,
+ * the rcu_read_lock() ensures the memory stays valid.
+ */
+ barrier();
+
+ if (!owner->on_cpu || need_resched()) {
+ ret = false;
+ break;
+ }
cpu_relax_lowlatency();
}
rcu_read_unlock();
- /*
- * We break out of the loop above on either need_resched(), when
- * the owner is not running, or when the lock owner changed.
- * Return success only when the lock owner changed.
- */
- return lock->owner != owner;
+ return ret;
}
/*