Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932234AbbFRUbf (ORCPT ); Thu, 18 Jun 2015 16:31:35 -0400 Received: from terminus.zytor.com ([198.137.202.10]:37262 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751044AbbFRUb1 (ORCPT ); Thu, 18 Jun 2015 16:31:27 -0400 Date: Thu, 18 Jun 2015 13:30:35 -0700 From: tip-bot for Davidlohr Bueso Message-ID: Cc: umgwanakikbuti@gmail.com, rostedt@goodmis.org, hpa@zytor.com, tglx@linutronix.de, bigeasy@linutronix.de, linux-kernel@vger.kernel.org, peterz@infradead.org, paulmck@linux.vnet.ibm.com, dave@stgolabs.net, dbueso@suse.de, mingo@kernel.org Reply-To: linux-kernel@vger.kernel.org, peterz@infradead.org, rostedt@goodmis.org, umgwanakikbuti@gmail.com, hpa@zytor.com, bigeasy@linutronix.de, tglx@linutronix.de, mingo@kernel.org, paulmck@linux.vnet.ibm.com, dave@stgolabs.net, dbueso@suse.de In-Reply-To: <1432056298-18738-2-git-send-email-dave@stgolabs.net> References: <1432056298-18738-2-git-send-email-dave@stgolabs.net> To: linux-tip-commits@vger.kernel.org Subject: [tip:sched/core] locking/rtmutex: Implement lockless top-waiter wakeup Git-Commit-ID: 45ab4effc3bee6f8a5cb05652b7bb895ec5b6a7a X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3521 Lines: 97 Commit-ID: 45ab4effc3bee6f8a5cb05652b7bb895ec5b6a7a Gitweb: http://git.kernel.org/tip/45ab4effc3bee6f8a5cb05652b7bb895ec5b6a7a Author: Davidlohr Bueso AuthorDate: Tue, 19 May 2015 10:24:55 -0700 Committer: Thomas Gleixner CommitDate: Thu, 18 Jun 2015 22:27:46 +0200 locking/rtmutex: Implement lockless top-waiter wakeup Mark the task for later wakeup after the wait_lock has been released. This way, once the next task is awoken, it will have a better chance to of finding the wait_lock free when continuing executing in __rt_mutex_slowlock() when trying to acquire the rtmutex, calling try_to_take_rt_mutex(). Upon contended scenarios, other tasks attempting take the lock may acquire it first, right after the wait_lock is released, but (a) this can also occur with the current code, as it relies on the spinlock fairness, and (b) we are dealing with the top-waiter anyway, so it will always take the lock next. Signed-off-by: Davidlohr Bueso Cc: Steven Rostedt Cc: Mike Galbraith Cc: Paul E. McKenney Cc: Sebastian Andrzej Siewior Cc: Davidlohr Bueso Cc: Peter Zijlstra Link: http://lkml.kernel.org/r/1432056298-18738-2-git-send-email-dave@stgolabs.net Signed-off-by: Thomas Gleixner --- kernel/locking/rtmutex.c | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c index b025295..44ee8f8 100644 --- a/kernel/locking/rtmutex.c +++ b/kernel/locking/rtmutex.c @@ -955,14 +955,13 @@ static int task_blocks_on_rt_mutex(struct rt_mutex *lock, } /* - * Wake up the next waiter on the lock. - * * Remove the top waiter from the current tasks pi waiter list and - * wake it up. + * queue it up. * * Called with lock->wait_lock held. */ -static void wakeup_next_waiter(struct rt_mutex *lock) +static void mark_wakeup_next_waiter(struct wake_q_head *wake_q, + struct rt_mutex *lock) { struct rt_mutex_waiter *waiter; unsigned long flags; @@ -991,12 +990,7 @@ static void wakeup_next_waiter(struct rt_mutex *lock) raw_spin_unlock_irqrestore(¤t->pi_lock, flags); - /* - * It's safe to dereference waiter as it cannot go away as - * long as we hold lock->wait_lock. The waiter task needs to - * acquire it in order to dequeue the waiter. - */ - wake_up_process(waiter->task); + wake_q_add(wake_q, waiter->task); } /* @@ -1258,6 +1252,8 @@ static inline int rt_mutex_slowtrylock(struct rt_mutex *lock) static void __sched rt_mutex_slowunlock(struct rt_mutex *lock) { + WAKE_Q(wake_q); + raw_spin_lock(&lock->wait_lock); debug_rt_mutex_unlock(lock); @@ -1306,10 +1302,13 @@ rt_mutex_slowunlock(struct rt_mutex *lock) /* * The wakeup next waiter path does not suffer from the above * race. See the comments there. + * + * Queue the next waiter for wakeup once we release the wait_lock. */ - wakeup_next_waiter(lock); + mark_wakeup_next_waiter(&wake_q, lock); raw_spin_unlock(&lock->wait_lock); + wake_up_q(&wake_q); /* Undo pi boosting if necessary: */ rt_mutex_adjust_prio(current); -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/