Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758866AbcLAOHT (ORCPT ); Thu, 1 Dec 2016 09:07:19 -0500 Received: from mail-wj0-f193.google.com ([209.85.210.193]:33027 "EHLO mail-wj0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1758450AbcLAOHO (ORCPT ); Thu, 1 Dec 2016 09:07:14 -0500 From: =?UTF-8?q?Nicolai=20H=C3=A4hnle?= To: linux-kernel@vger.kernel.org Cc: =?UTF-8?q?Nicolai=20H=C3=A4hnle?= , Peter Zijlstra , Ingo Molnar , Maarten Lankhorst , Daniel Vetter , Chris Wilson , dri-devel@lists.freedesktop.org Subject: [PATCH v2 11/11] [rfc] locking/ww_mutex: Always spin optimistically for the first waiter Date: Thu, 1 Dec 2016 15:06:54 +0100 Message-Id: <1480601214-26583-12-git-send-email-nhaehnle@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1480601214-26583-1-git-send-email-nhaehnle@gmail.com> References: <1480601214-26583-1-git-send-email-nhaehnle@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2799 Lines: 93 From: Nicolai Hähnle Check the current owner's context once against our stamp. If our stamp is lower, we continue to spin optimistically instead of backing off. This is correct with respect to deadlock detection because while the (owner, ww_ctx) pair may re-appear if the owner task manages to unlock and re-acquire the lock while we're spinning, the context could only have been re-initialized with an even higher stamp. We also still detect when we have to back off for other waiters that join the list while we're spinning. But taking the wait_lock in mutex_spin_on_owner feels iffy, even if it is done only once. Median timings taken of a contention-heavy GPU workload: Before: real 0m53.086s user 0m7.360s sys 1m46.204s After: real 0m52.577s user 0m7.544s sys 1m49.200s Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Maarten Lankhorst Cc: Daniel Vetter Cc: Chris Wilson Cc: dri-devel@lists.freedesktop.org Signed-off-by: Nicolai Hähnle --- kernel/locking/mutex.c | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index 38d173c..9216239 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -378,6 +378,28 @@ bool mutex_spin_on_owner(struct mutex *lock, struct task_struct *owner, struct mutex_waiter *waiter) { bool ret = true; + struct ww_acquire_ctx *owner_ww_ctx = NULL; + + if (use_ww_ctx && ww_ctx && ww_ctx->acquired > 0) { + struct ww_mutex *ww; + unsigned long flags; + + ww = container_of(lock, struct ww_mutex, base); + + /* + * Check the stamp of the current owner once. This allows us + * to spin optimistically in the case where the current owner + * has a higher stamp than us. + */ + spin_lock_mutex(&lock->wait_lock, flags); + owner_ww_ctx = ww->ctx; + if (owner_ww_ctx && + __ww_mutex_stamp_after(ww_ctx, owner_ww_ctx)) { + spin_unlock_mutex(&lock->wait_lock, flags); + return false; + } + spin_unlock_mutex(&lock->wait_lock, flags); + } rcu_read_lock(); while (__mutex_owner(lock) == owner) { @@ -414,9 +436,16 @@ bool mutex_spin_on_owner(struct mutex *lock, struct task_struct *owner, * Check this in every inner iteration because we may * be racing against another thread's ww_mutex_lock. */ - if (ww_ctx->acquired > 0 && READ_ONCE(ww->ctx)) { - ret = false; - break; + if (ww_ctx->acquired > 0) { + struct ww_acquire_ctx *current_ctx; + + current_ctx = READ_ONCE(ww->ctx); + + if (current_ctx && + current_ctx != owner_ww_ctx) { + ret = false; + break; + } } /* -- 2.7.4