Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751982AbbD3VNO (ORCPT ); Thu, 30 Apr 2015 17:13:14 -0400 Received: from g1t5425.austin.hp.com ([15.216.225.55]:42915 "EHLO g1t5425.austin.hp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751348AbbD3VNG (ORCPT ); Thu, 30 Apr 2015 17:13:06 -0400 From: Waiman Long To: Peter Zijlstra , Ingo Molnar Cc: linux-kernel@vger.kernel.org, Jason Low , Davidlohr Bueso , Scott J Norton , Douglas Hatch , Waiman Long Subject: [PATCH v4 2/2] locking/rwsem: check for active writer before wakeup Date: Thu, 30 Apr 2015 17:12:17 -0400 Message-Id: <1430428337-16802-3-git-send-email-Waiman.Long@hp.com> X-Mailer: git-send-email 1.7.1 In-Reply-To: <1430428337-16802-1-git-send-email-Waiman.Long@hp.com> References: <1430428337-16802-1-git-send-email-Waiman.Long@hp.com> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2266 Lines: 69 On a highly contended rwsem, spinlock contention due to the slow rwsem_wake() call can be a significant portion of the total CPU cycles used. With writer lock stealing and writer optimistic spinning, there is also a chance that the lock may have been stolen by the time that the wait_lock is acquired. This patch adds a low cost checking code after acquiring the wait_lock to look for active writer. The presence of an active writer will abort the wakeup operation. Signed-off-by: Waiman Long --- kernel/locking/rwsem-xadd.c | 21 +++++++++++++++++++-- 1 files changed, 19 insertions(+), 2 deletions(-) diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c index 2bb25e2..815f0cc 100644 --- a/kernel/locking/rwsem-xadd.c +++ b/kernel/locking/rwsem-xadd.c @@ -399,6 +399,15 @@ static inline bool rwsem_has_spinner(struct rw_semaphore *sem) return osq_is_locked(&sem->osq); } +/* + * Return true if there is an active writer by checking the owner field which + * should be set if there is one. + */ +static inline bool rwsem_has_active_writer(struct rw_semaphore *sem) +{ + return READ_ONCE(sem->owner) != NULL; +} + #else static bool rwsem_optimistic_spin(struct rw_semaphore *sem) { @@ -409,6 +418,11 @@ static inline bool rwsem_has_spinner(struct rw_semaphore *sem) { return false; } + +static inline bool rwsem_has_active_writer(struct rw_semaphore *sem) +{ + return false; /* Assume it has no active writer */ +} #endif /* @@ -524,8 +538,11 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem) raw_spin_lock_irqsave(&sem->wait_lock, flags); locked: - /* do nothing if list empty */ - if (!list_empty(&sem->wait_list)) + /* + * Do nothing if list empty or the lock has just been stolen by a + * writer after a possibly long wait in getting the wait_lock. + */ + if (!list_empty(&sem->wait_list) && !rwsem_has_active_writer(sem)) sem = __rwsem_do_wake(sem, RWSEM_WAKE_ANY); raw_spin_unlock_irqrestore(&sem->wait_lock, flags); -- 1.7.1 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/