Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754554AbbHCRBF (ORCPT ); Mon, 3 Aug 2015 13:01:05 -0400 Received: from terminus.zytor.com ([198.137.202.10]:40122 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754131AbbHCRBC (ORCPT ); Mon, 3 Aug 2015 13:01:02 -0400 Date: Mon, 3 Aug 2015 10:00:08 -0700 From: tip-bot for Waiman Long Message-ID: Cc: paulmck@linux.vnet.ibm.com, torvalds@linux-foundation.org, Waiman.Long@hp.com, hpa@zytor.com, scott.norton@hp.com, doug.hatch@hp.com, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, will.deacon@arm.com, tglx@linutronix.de, mingo@kernel.org, arnd@arndb.de, peterz@infradead.org Reply-To: mingo@kernel.org, arnd@arndb.de, peterz@infradead.org, linux-kernel@vger.kernel.org, tglx@linutronix.de, will.deacon@arm.com, akpm@linux-foundation.org, doug.hatch@hp.com, scott.norton@hp.com, paulmck@linux.vnet.ibm.com, torvalds@linux-foundation.org, Waiman.Long@hp.com, hpa@zytor.com In-Reply-To: <1436459543-29126-2-git-send-email-Waiman.Long@hp.com> References: <1436459543-29126-2-git-send-email-Waiman.Long@hp.com> To: linux-tip-commits@vger.kernel.org Subject: [tip:locking/core] locking/qrwlock: Reduce reader/ writer to reader lock transfer latency Git-Commit-ID: ffffeaf318bd8da036eb8eb784b025a9f829201b X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3154 Lines: 71 Commit-ID: ffffeaf318bd8da036eb8eb784b025a9f829201b Gitweb: http://git.kernel.org/tip/ffffeaf318bd8da036eb8eb784b025a9f829201b Author: Waiman Long AuthorDate: Thu, 9 Jul 2015 12:32:22 -0400 Committer: Ingo Molnar CommitDate: Mon, 3 Aug 2015 10:57:10 +0200 locking/qrwlock: Reduce reader/writer to reader lock transfer latency Currently, a reader will check first to make sure that the writer mode byte is cleared before incrementing the reader count. That waiting is not really necessary. It increases the latency in the reader/writer to reader transition and reduces readers performance. This patch eliminates that waiting. It also has the side effect of reducing the chance of writer lock stealing and improving the fairness of the lock. Using a locking microbenchmark, a 10-threads 5M locking loop of mostly readers (RW ratio = 10,000:1) has the following performance numbers in a Haswell-EX box: Kernel Locking Rate (Kops/s) ------ --------------------- 4.1.1 15,063,081 4.1.1+patch 17,241,552 (+14.4%) Signed-off-by: Waiman Long Signed-off-by: Peter Zijlstra (Intel) Cc: Andrew Morton Cc: Arnd Bergmann Cc: Douglas Hatch Cc: Linus Torvalds Cc: Paul E. McKenney Cc: Peter Zijlstra Cc: Scott J Norton Cc: Thomas Gleixner Cc: Will Deacon Link: http://lkml.kernel.org/r/1436459543-29126-2-git-send-email-Waiman.Long@hp.com Signed-off-by: Ingo Molnar --- kernel/locking/qrwlock.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/kernel/locking/qrwlock.c b/kernel/locking/qrwlock.c index d9c36c5..6a7a3b8 100644 --- a/kernel/locking/qrwlock.c +++ b/kernel/locking/qrwlock.c @@ -88,15 +88,11 @@ void queued_read_lock_slowpath(struct qrwlock *lock, u32 cnts) arch_spin_lock(&lock->lock); /* - * At the head of the wait queue now, wait until the writer state - * goes to 0 and then try to increment the reader count and get - * the lock. It is possible that an incoming writer may steal the - * lock in the interim, so it is necessary to check the writer byte - * to make sure that the write lock isn't taken. + * At the head of the wait queue now, increment the reader count + * and wait until the writer, if it has the lock, has gone away. + * At ths stage, it is not possible for a writer to remain in the + * waiting state (_QW_WAITING). So there won't be any deadlock. */ - while (atomic_read(&lock->cnts) & _QW_WMASK) - cpu_relax_lowlatency(); - cnts = atomic_add_return(_QR_BIAS, &lock->cnts) - _QR_BIAS; rspin_until_writer_unlock(lock, cnts); -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/