Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754188AbcKRSzl (ORCPT ); Fri, 18 Nov 2016 13:55:41 -0500 Received: from smtp2.provo.novell.com ([137.65.250.81]:40494 "EHLO smtp2.provo.novell.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753868AbcKRSzc (ORCPT ); Fri, 18 Nov 2016 13:55:32 -0500 From: Davidlohr Bueso To: mingo@kernel.org, peterz@infradead.org, oleg@redhat.com Cc: john.stultz@linaro.org, dimitrysh@google.com, linux-kernel@vger.kernel.org, dave@stgolabs.net, Davidlohr Bueso Subject: [PATCH 3/3] locking/percpu-rwsem: Avoid unnecessary writer wakeups Date: Fri, 18 Nov 2016 10:54:37 -0800 Message-Id: <1479495277-9075-4-git-send-email-dave@stgolabs.net> X-Mailer: git-send-email 2.6.6 In-Reply-To: <1479495277-9075-1-git-send-email-dave@stgolabs.net> References: <1479495277-9075-1-git-send-email-dave@stgolabs.net> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3431 Lines: 113 There is obviously no point in doing the wakeup if the reader wakee task can do the active readers check on behalf of the writer on its way to unlocking itself. The downside is that when readers_active_check() does return true we end up iterating the per-CPU counter twice. This trade-off, however, does seem reasonable in that if we are here, (i) we have already lost any hope for reader side performance and; (ii) because this lock is used mainly for sharing, it is not crazy to expect to have readers incoming for the lock during this window -- therefore sending writers right back to sleep for every reader up(). Signed-off-by: Davidlohr Bueso --- kernel/locking/percpu-rwsem.c | 72 ++++++++++++++++++++++++------------------- 1 file changed, 40 insertions(+), 32 deletions(-) diff --git a/kernel/locking/percpu-rwsem.c b/kernel/locking/percpu-rwsem.c index cb71201855f2..8e6fbf117f14 100644 --- a/kernel/locking/percpu-rwsem.c +++ b/kernel/locking/percpu-rwsem.c @@ -8,6 +8,44 @@ #include #include +#define per_cpu_sum(var) \ +({ \ + typeof(var) __sum = 0; \ + int cpu; \ + compiletime_assert_atomic_type(__sum); \ + for_each_possible_cpu(cpu) \ + __sum += per_cpu(var, cpu); \ + __sum; \ +}) + +/* + * Return true if the modular sum of the sem->read_count per-CPU variable is + * zero. If this sum is zero, then it is stable due to the fact that if any + * newly arriving readers increment a given counter, they will immediately + * decrement that same counter. + */ +static bool readers_active_check(struct percpu_rw_semaphore *sem) +{ + if (per_cpu_sum(*sem->read_count) != 0) + return false; + + /* + * If we observed the decrement; ensure we see the entire critical + * section. In the case of __readers_active_check we avoid the + * critical section sync, as the writer wakee will fully re-check + * to continue. + */ + + smp_mb(); /* C matches B */ + + return true; +} + +static bool __readers_active_check(struct percpu_rw_semaphore *sem) +{ + return !(per_cpu_sum(*sem->read_count) !=0); +} + int __percpu_init_rwsem(struct percpu_rw_semaphore *sem, const char *name, struct lock_class_key *rwsem_key) { @@ -103,41 +141,11 @@ void __percpu_up_read(struct percpu_rw_semaphore *sem) __this_cpu_dec(*sem->read_count); /* Prod writer to recheck readers_active */ - swake_up(&sem->writer); + if (__readers_active_check(sem)) + swake_up(&sem->writer); } EXPORT_SYMBOL_GPL(__percpu_up_read); -#define per_cpu_sum(var) \ -({ \ - typeof(var) __sum = 0; \ - int cpu; \ - compiletime_assert_atomic_type(__sum); \ - for_each_possible_cpu(cpu) \ - __sum += per_cpu(var, cpu); \ - __sum; \ -}) - -/* - * Return true if the modular sum of the sem->read_count per-CPU variable is - * zero. If this sum is zero, then it is stable due to the fact that if any - * newly arriving readers increment a given counter, they will immediately - * decrement that same counter. - */ -static bool readers_active_check(struct percpu_rw_semaphore *sem) -{ - if (per_cpu_sum(*sem->read_count) != 0) - return false; - - /* - * If we observed the decrement; ensure we see the entire critical - * section. - */ - - smp_mb(); /* C matches B */ - - return true; -} - void percpu_down_write(struct percpu_rw_semaphore *sem) { /* Notify readers to take the slow path. */ -- 2.6.6