Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933581Ab3EGNq3 (ORCPT ); Tue, 7 May 2013 09:46:29 -0400 Received: from mail-da0-f41.google.com ([209.85.210.41]:45722 "EHLO mail-da0-f41.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S933529Ab3EGNqW (ORCPT ); Tue, 7 May 2013 09:46:22 -0400 From: Michel Lespinasse To: Linus Torvalds , David Howells , Peter Zijlstra , Ingo Molnar , Rik van Riel , Davidlohr Bueso , Peter Hurley Cc: Alex Shi , Yuanhan Liu , Andrew Morton , linux-kernel@vger.kernel.org Subject: [PATCH v3 06/14] rwsem: more agressive lock stealing in rwsem_down_write_failed Date: Tue, 7 May 2013 06:45:54 -0700 Message-Id: <1367934362-10433-7-git-send-email-walken@google.com> X-Mailer: git-send-email 1.8.2.1 In-Reply-To: <1367934362-10433-1-git-send-email-walken@google.com> References: <1367934362-10433-1-git-send-email-walken@google.com> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3583 Lines: 89 Some small code simplifications can be achieved by doing more agressive lock stealing: - When rwsem_down_write_failed() notices that there are no active locks (and thus no thread to wake us if we decided to sleep), it used to wake the first queued process. However, stealing the lock is also sufficient to deal with this case, so we don't need this check anymore. - In try_get_writer_sem(), we can steal the lock even when the first waiter is a reader. This is correct because the code path that wakes readers is protected by the wait_lock. As to the performance effects of this change, they are expected to be minimal: readers are still granted the lock (rather than having to acquire it themselves) when they reach the front of the wait queue, so we have essentially the same behavior as in rwsem-spinlock. Signed-off-by: Michel Lespinasse Reviewed-by: Rik van Riel Reviewed-by: Peter Hurley Acked-by: Davidlohr Bueso --- lib/rwsem.c | 29 ++++++++--------------------- 1 file changed, 8 insertions(+), 21 deletions(-) diff --git a/lib/rwsem.c b/lib/rwsem.c index c73bd96dc30c..2360bf204098 100644 --- a/lib/rwsem.c +++ b/lib/rwsem.c @@ -143,20 +143,12 @@ __rwsem_do_wake(struct rw_semaphore *sem, int wake_type) } /* Try to get write sem, caller holds sem->wait_lock: */ -static int try_get_writer_sem(struct rw_semaphore *sem, - struct rwsem_waiter *waiter) +static int try_get_writer_sem(struct rw_semaphore *sem) { - struct rwsem_waiter *fwaiter; long oldcount, adjustment; - /* only steal when first waiter is writing */ - fwaiter = list_entry(sem->wait_list.next, struct rwsem_waiter, list); - if (fwaiter->type != RWSEM_WAITING_FOR_WRITE) - return 0; - adjustment = RWSEM_ACTIVE_WRITE_BIAS; - /* Only one waiter in the queue: */ - if (fwaiter == waiter && waiter->list.next == &sem->wait_list) + if (list_is_singular(&sem->wait_list)) adjustment -= RWSEM_WAITING_BIAS; try_again_write: @@ -233,23 +225,18 @@ struct rw_semaphore __sched *rwsem_down_write_failed(struct rw_semaphore *sem) /* we're now waiting on the lock, but no longer actively locking */ count = rwsem_atomic_update(adjustment, sem); - /* If there are no active locks, wake the front queued process(es) up. - * - * Alternatively, if we're called from a failed down_write(), there - * were already threads queued before us and there are no active - * writers, the lock must be read owned; so we try to wake any read - * locks that were queued ahead of us. */ - if (count == RWSEM_WAITING_BIAS) - sem = __rwsem_do_wake(sem, RWSEM_WAKE_NO_ACTIVE); - else if (count > RWSEM_WAITING_BIAS && - adjustment == -RWSEM_ACTIVE_WRITE_BIAS) + /* If there were already threads queued before us and there are no + * active writers, the lock must be read owned; so we try to wake + * any read locks that were queued ahead of us. */ + if (count > RWSEM_WAITING_BIAS && + adjustment == -RWSEM_ACTIVE_WRITE_BIAS) sem = __rwsem_do_wake(sem, RWSEM_WAKE_READ_OWNED); /* wait until we successfully acquire the lock */ while (true) { set_task_state(tsk, TASK_UNINTERRUPTIBLE); - if (try_get_writer_sem(sem, &waiter)) + if (try_get_writer_sem(sem)) break; raw_spin_unlock_irq(&sem->wait_lock); -- 1.8.2.1 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/