Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754264AbbGFPgJ (ORCPT ); Mon, 6 Jul 2015 11:36:09 -0400 Received: from terminus.zytor.com ([198.137.202.10]:49026 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753725AbbGFPgE (ORCPT ); Mon, 6 Jul 2015 11:36:04 -0400 Date: Mon, 6 Jul 2015 08:35:20 -0700 From: tip-bot for Waiman Long Message-ID: Cc: will.deacon@arm.com, mingo@kernel.org, scott.norton@hp.com, arnd@arndb.de, torvalds@linux-foundation.org, tglx@linutronix.de, peterz@infradead.org, linux-kernel@vger.kernel.org, hpa@zytor.com, doug.hatch@hp.com, Waiman.Long@hp.com Reply-To: Waiman.Long@hp.com, doug.hatch@hp.com, hpa@zytor.com, linux-kernel@vger.kernel.org, peterz@infradead.org, tglx@linutronix.de, arnd@arndb.de, torvalds@linux-foundation.org, scott.norton@hp.com, mingo@kernel.org, will.deacon@arm.com In-Reply-To: <1434729002-57724-3-git-send-email-Waiman.Long@hp.com> References: <1434729002-57724-3-git-send-email-Waiman.Long@hp.com> To: linux-tip-commits@vger.kernel.org Subject: [tip:locking/urgent] locking/qrwlock: Better optimization for interrupt context readers Git-Commit-ID: 0e06e5be70d392aa842c1455ec2d0baf62aeed48 X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4133 Lines: 102 Commit-ID: 0e06e5be70d392aa842c1455ec2d0baf62aeed48 Gitweb: http://git.kernel.org/tip/0e06e5be70d392aa842c1455ec2d0baf62aeed48 Author: Waiman Long AuthorDate: Fri, 19 Jun 2015 11:50:01 -0400 Committer: Ingo Molnar CommitDate: Mon, 6 Jul 2015 14:11:28 +0200 locking/qrwlock: Better optimization for interrupt context readers The qrwlock is fair in the process context, but becoming unfair when in the interrupt context to support use cases like the tasklist_lock. The current code isn't that well-documented on what happens when in the interrupt context. The rspin_until_writer_unlock() will only spin if the writer has gotten the lock. If the writer is still in the waiting state, the increment in the reader count will cause the writer to remain in the waiting state and the new interrupt context reader will get the lock and return immediately. The current code, however, does an additional read of the lock value which is not necessary as the information has already been there in the fast path. This may sometime cause an additional cacheline transfer when the lock is highly contended. This patch passes the lock value information gotten in the fast path to the slow path to eliminate the additional read. It also documents the action for the interrupt context readers more clearly. Signed-off-by: Waiman Long Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Will Deacon Cc: Arnd Bergmann Cc: Douglas Hatch Cc: Linus Torvalds Cc: Peter Zijlstra Cc: Scott J Norton Cc: Thomas Gleixner Link: http://lkml.kernel.org/r/1434729002-57724-3-git-send-email-Waiman.Long@hp.com Signed-off-by: Ingo Molnar --- include/asm-generic/qrwlock.h | 4 ++-- kernel/locking/qrwlock.c | 13 +++++++------ 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/include/asm-generic/qrwlock.h b/include/asm-generic/qrwlock.h index 55e3ee1..deb9e8b 100644 --- a/include/asm-generic/qrwlock.h +++ b/include/asm-generic/qrwlock.h @@ -36,7 +36,7 @@ /* * External function declarations */ -extern void queued_read_lock_slowpath(struct qrwlock *lock); +extern void queued_read_lock_slowpath(struct qrwlock *lock, u32 cnts); extern void queued_write_lock_slowpath(struct qrwlock *lock); /** @@ -105,7 +105,7 @@ static inline void queued_read_lock(struct qrwlock *lock) return; /* The slowpath will decrement the reader count, if necessary. */ - queued_read_lock_slowpath(lock); + queued_read_lock_slowpath(lock, cnts); } /** diff --git a/kernel/locking/qrwlock.c b/kernel/locking/qrwlock.c index 49057d4..d9c36c5 100644 --- a/kernel/locking/qrwlock.c +++ b/kernel/locking/qrwlock.c @@ -62,20 +62,21 @@ rspin_until_writer_unlock(struct qrwlock *lock, u32 cnts) /** * queued_read_lock_slowpath - acquire read lock of a queue rwlock * @lock: Pointer to queue rwlock structure + * @cnts: Current qrwlock lock value */ -void queued_read_lock_slowpath(struct qrwlock *lock) +void queued_read_lock_slowpath(struct qrwlock *lock, u32 cnts) { - u32 cnts; - /* * Readers come here when they cannot get the lock without waiting */ if (unlikely(in_interrupt())) { /* - * Readers in interrupt context will spin until the lock is - * available without waiting in the queue. + * Readers in interrupt context will get the lock immediately + * if the writer is just waiting (not holding the lock yet). + * The rspin_until_writer_unlock() function returns immediately + * in this case. Otherwise, they will spin until the lock + * is available without waiting in the queue. */ - cnts = smp_load_acquire((u32 *)&lock->cnts); rspin_until_writer_unlock(lock, cnts); return; } -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/