Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754578AbbKWQ21 (ORCPT ); Mon, 23 Nov 2015 11:28:27 -0500 Received: from terminus.zytor.com ([198.137.202.10]:44696 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753747AbbKWQ2S (ORCPT ); Mon, 23 Nov 2015 11:28:18 -0500 Date: Mon, 23 Nov 2015 08:27:13 -0800 From: tip-bot for Waiman Long Message-ID: Cc: tglx@linutronix.de, scott.norton@hpe.com, Waiman.Long@hpe.com, mingo@kernel.org, linux-kernel@vger.kernel.org, paulmck@linux.vnet.ibm.com, torvalds@linux-foundation.org, akpm@linux-foundation.org, peterz@infradead.org, doug.hatch@hpe.com, hpa@zytor.com, dave@stgolabs.net Reply-To: dave@stgolabs.net, hpa@zytor.com, doug.hatch@hpe.com, peterz@infradead.org, akpm@linux-foundation.org, torvalds@linux-foundation.org, paulmck@linux.vnet.ibm.com, linux-kernel@vger.kernel.org, Waiman.Long@hpe.com, mingo@kernel.org, tglx@linutronix.de, scott.norton@hpe.com In-Reply-To: <1447114167-47185-3-git-send-email-Waiman.Long@hpe.com> References: <1447114167-47185-3-git-send-email-Waiman.Long@hpe.com> To: linux-tip-commits@vger.kernel.org Subject: [tip:locking/core] locking/qspinlock: Prefetch the next node cacheline Git-Commit-ID: 81b5598665a24083dd889fbd8cb08b0d8de4b8ad X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 2807 Lines: 70 Commit-ID: 81b5598665a24083dd889fbd8cb08b0d8de4b8ad Gitweb: http://git.kernel.org/tip/81b5598665a24083dd889fbd8cb08b0d8de4b8ad Author: Waiman Long AuthorDate: Mon, 9 Nov 2015 19:09:22 -0500 Committer: Ingo Molnar CommitDate: Mon, 23 Nov 2015 10:01:59 +0100 locking/qspinlock: Prefetch the next node cacheline A queue head CPU, after acquiring the lock, will have to notify the next CPU in the wait queue that it has became the new queue head. This involves loading a new cacheline from the MCS node of the next CPU. That operation can be expensive and add to the latency of locking operation. This patch addes code to optmistically prefetch the next MCS node cacheline if the next pointer is defined and it has been spinning for the MCS lock for a while. This reduces the locking latency and improves the system throughput. The performance change will depend on whether the prefetch overhead can be hidden within the latency of the lock spin loop. On really short critical section, there may not be performance gain at all. With longer critical section, however, it was found to have a performance boost of 5-10% over a range of different queue depths with a spinlock loop microbenchmark. Signed-off-by: Waiman Long Signed-off-by: Peter Zijlstra (Intel) Cc: Andrew Morton Cc: Davidlohr Bueso Cc: Douglas Hatch Cc: H. Peter Anvin Cc: Linus Torvalds Cc: Paul E. McKenney Cc: Peter Zijlstra Cc: Scott J Norton Cc: Thomas Gleixner Link: http://lkml.kernel.org/r/1447114167-47185-3-git-send-email-Waiman.Long@hpe.com Signed-off-by: Ingo Molnar --- kernel/locking/qspinlock.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c index 7868418..365b203 100644 --- a/kernel/locking/qspinlock.c +++ b/kernel/locking/qspinlock.c @@ -407,6 +407,16 @@ queue: pv_wait_node(node); arch_mcs_spin_lock_contended(&node->locked); + + /* + * While waiting for the MCS lock, the next pointer may have + * been set by another lock waiter. We optimistically load + * the next pointer & prefetch the cacheline for writing + * to reduce latency in the upcoming MCS unlock operation. + */ + next = READ_ONCE(node->next); + if (next) + prefetchw(next); } /* -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/