Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752258AbdI2Jdr (ORCPT ); Fri, 29 Sep 2017 05:33:47 -0400 Received: from terminus.zytor.com ([65.50.211.136]:56535 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751901AbdI2Jdp (ORCPT ); Fri, 29 Sep 2017 05:33:45 -0400 Date: Fri, 29 Sep 2017 02:30:13 -0700 From: tip-bot for Prateek Sood Message-ID: Cc: tglx@linutronix.de, hpa@zytor.com, torvalds@linux-foundation.org, prsood@codeaurora.org, mingo@kernel.org, peterz@infradead.org, linux-kernel@vger.kernel.org Reply-To: tglx@linutronix.de, prsood@codeaurora.org, hpa@zytor.com, torvalds@linux-foundation.org, mingo@kernel.org, peterz@infradead.org, linux-kernel@vger.kernel.org In-Reply-To: <1504794658-15397-1-git-send-email-prsood@codeaurora.org> References: <1504794658-15397-1-git-send-email-prsood@codeaurora.org> To: linux-tip-commits@vger.kernel.org Subject: [tip:locking/urgent] locking/rwsem-xadd: Fix missed wakeup due to reordering of load Git-Commit-ID: 9c29c31830a4eca724e137a9339137204bbb31be X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3410 Lines: 93 Commit-ID: 9c29c31830a4eca724e137a9339137204bbb31be Gitweb: https://git.kernel.org/tip/9c29c31830a4eca724e137a9339137204bbb31be Author: Prateek Sood AuthorDate: Thu, 7 Sep 2017 20:00:58 +0530 Committer: Ingo Molnar CommitDate: Fri, 29 Sep 2017 10:10:20 +0200 locking/rwsem-xadd: Fix missed wakeup due to reordering of load If a spinner is present, there is a chance that the load of rwsem_has_spinner() in rwsem_wake() can be reordered with respect to decrement of rwsem count in __up_write() leading to wakeup being missed: spinning writer up_write caller --------------- ----------------------- [S] osq_unlock() [L] osq spin_lock(wait_lock) sem->count=0xFFFFFFFF00000001 +0xFFFFFFFF00000000 count=sem->count MB sem->count=0xFFFFFFFE00000001 -0xFFFFFFFF00000001 spin_trylock(wait_lock) return rwsem_try_write_lock(count) spin_unlock(wait_lock) schedule() Reordering of atomic_long_sub_return_release() in __up_write() and rwsem_has_spinner() in rwsem_wake() can cause missing of wakeup in up_write() context. In spinning writer, sem->count and local variable count is 0XFFFFFFFE00000001. It would result in rwsem_try_write_lock() failing to acquire rwsem and spinning writer going to sleep in rwsem_down_write_failed(). The smp_rmb() will make sure that the spinner state is consulted after sem->count is updated in up_write context. Signed-off-by: Prateek Sood Signed-off-by: Peter Zijlstra (Intel) Cc: Linus Torvalds Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: dave@stgolabs.net Cc: longman@redhat.com Cc: parri.andrea@gmail.com Cc: sramana@codeaurora.org Link: http://lkml.kernel.org/r/1504794658-15397-1-git-send-email-prsood@codeaurora.org Signed-off-by: Ingo Molnar --- kernel/locking/rwsem-xadd.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c index 02f6606..1fefe6d 100644 --- a/kernel/locking/rwsem-xadd.c +++ b/kernel/locking/rwsem-xadd.c @@ -613,6 +613,33 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem) DEFINE_WAKE_Q(wake_q); /* + * __rwsem_down_write_failed_common(sem) + * rwsem_optimistic_spin(sem) + * osq_unlock(sem->osq) + * ... + * atomic_long_add_return(&sem->count) + * + * - VS - + * + * __up_write() + * if (atomic_long_sub_return_release(&sem->count) < 0) + * rwsem_wake(sem) + * osq_is_locked(&sem->osq) + * + * And __up_write() must observe !osq_is_locked() when it observes the + * atomic_long_add_return() in order to not miss a wakeup. + * + * This boils down to: + * + * [S.rel] X = 1 [RmW] r0 = (Y += 0) + * MB RMB + * [RmW] Y += 1 [L] r1 = X + * + * exists (r0=1 /\ r1=0) + */ + smp_rmb(); + + /* * If a spinner is present, it is not necessary to do the wakeup. * Try to do wakeup only if the trylock succeeds to minimize * spinlock contention which may introduce too much delay in the