Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753245AbdHJMRK (ORCPT ); Thu, 10 Aug 2017 08:17:10 -0400 Received: from terminus.zytor.com ([65.50.211.136]:51597 "EHLO terminus.zytor.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752091AbdHJMRI (ORCPT ); Thu, 10 Aug 2017 08:17:08 -0400 Date: Thu, 10 Aug 2017 05:12:28 -0700 From: tip-bot for Paolo Bonzini Message-ID: Cc: jbaron@akamai.com, tglx@linutronix.de, pbonzini@redhat.com, hpa@zytor.com, peterz@infradead.org, torvalds@linux-foundation.org, linux-kernel@vger.kernel.org, mingo@kernel.org, eric.dumazet@gmail.com Reply-To: jbaron@akamai.com, tglx@linutronix.de, pbonzini@redhat.com, hpa@zytor.com, peterz@infradead.org, torvalds@linux-foundation.org, linux-kernel@vger.kernel.org, mingo@kernel.org, eric.dumazet@gmail.com In-Reply-To: <1501601046-35683-2-git-send-email-pbonzini@redhat.com> References: <1501601046-35683-2-git-send-email-pbonzini@redhat.com> To: linux-tip-commits@vger.kernel.org Subject: [tip:locking/core] jump_label: Fix concurrent static_key_enable/disable() Git-Commit-ID: 1dbb6704de91b169a58d0c8221624afd6a95cfc7 X-Mailer: tip-git-log-daemon Robot-ID: Robot-Unsubscribe: Contact to get blacklisted from these emails MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Content-Disposition: inline Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4269 Lines: 148 Commit-ID: 1dbb6704de91b169a58d0c8221624afd6a95cfc7 Gitweb: http://git.kernel.org/tip/1dbb6704de91b169a58d0c8221624afd6a95cfc7 Author: Paolo Bonzini AuthorDate: Tue, 1 Aug 2017 17:24:04 +0200 Committer: Ingo Molnar CommitDate: Thu, 10 Aug 2017 12:28:56 +0200 jump_label: Fix concurrent static_key_enable/disable() static_key_enable/disable are trying to cap the static key count to 0/1. However, their use of key->enabled is outside jump_label_lock so they do not really ensure that. Rewrite them to do a quick check for an already enabled (respectively, already disabled), and then recheck under the jump label lock. Unlike static_key_slow_inc/dec, a failed check under the jump label lock does not modify key->enabled. Signed-off-by: Paolo Bonzini Signed-off-by: Peter Zijlstra (Intel) Cc: Eric Dumazet Cc: Jason Baron Cc: Linus Torvalds Cc: Peter Zijlstra Cc: Thomas Gleixner Link: http://lkml.kernel.org/r/1501601046-35683-2-git-send-email-pbonzini@redhat.com Signed-off-by: Ingo Molnar --- include/linux/jump_label.h | 22 +++++++++-------- kernel/jump_label.c | 59 +++++++++++++++++++++++++++++----------------- 2 files changed, 49 insertions(+), 32 deletions(-) diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h index 2afd74b..740a42e 100644 --- a/include/linux/jump_label.h +++ b/include/linux/jump_label.h @@ -234,22 +234,24 @@ static inline int jump_label_apply_nops(struct module *mod) static inline void static_key_enable(struct static_key *key) { - int count = static_key_count(key); - - WARN_ON_ONCE(count < 0 || count > 1); + STATIC_KEY_CHECK_USE(); - if (!count) - static_key_slow_inc(key); + if (atomic_read(&key->enabled) != 0) { + WARN_ON_ONCE(atomic_read(&key->enabled) != 1); + return; + } + atomic_set(&key->enabled, 1); } static inline void static_key_disable(struct static_key *key) { - int count = static_key_count(key); - - WARN_ON_ONCE(count < 0 || count > 1); + STATIC_KEY_CHECK_USE(); - if (count) - static_key_slow_dec(key); + if (atomic_read(&key->enabled) != 1) { + WARN_ON_ONCE(atomic_read(&key->enabled) != 0); + return; + } + atomic_set(&key->enabled, 0); } #define STATIC_KEY_INIT_TRUE { .enabled = ATOMIC_INIT(1) } diff --git a/kernel/jump_label.c b/kernel/jump_label.c index d11c506..833eeca 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -79,28 +79,6 @@ int static_key_count(struct static_key *key) } EXPORT_SYMBOL_GPL(static_key_count); -void static_key_enable(struct static_key *key) -{ - int count = static_key_count(key); - - WARN_ON_ONCE(count < 0 || count > 1); - - if (!count) - static_key_slow_inc(key); -} -EXPORT_SYMBOL_GPL(static_key_enable); - -void static_key_disable(struct static_key *key) -{ - int count = static_key_count(key); - - WARN_ON_ONCE(count < 0 || count > 1); - - if (count) - static_key_slow_dec(key); -} -EXPORT_SYMBOL_GPL(static_key_disable); - void static_key_slow_inc(struct static_key *key) { int v, v1; @@ -139,6 +117,43 @@ void static_key_slow_inc(struct static_key *key) } EXPORT_SYMBOL_GPL(static_key_slow_inc); +void static_key_enable(struct static_key *key) +{ + STATIC_KEY_CHECK_USE(); + if (atomic_read(&key->enabled) > 0) { + WARN_ON_ONCE(atomic_read(&key->enabled) != 1); + return; + } + + cpus_read_lock(); + jump_label_lock(); + if (atomic_read(&key->enabled) == 0) { + atomic_set(&key->enabled, -1); + jump_label_update(key); + atomic_set(&key->enabled, 1); + } + jump_label_unlock(); + cpus_read_unlock(); +} +EXPORT_SYMBOL_GPL(static_key_enable); + +void static_key_disable(struct static_key *key) +{ + STATIC_KEY_CHECK_USE(); + if (atomic_read(&key->enabled) != 1) { + WARN_ON_ONCE(atomic_read(&key->enabled) != 0); + return; + } + + cpus_read_lock(); + jump_label_lock(); + if (atomic_cmpxchg(&key->enabled, 1, 0)) + jump_label_update(key); + jump_label_unlock(); + cpus_read_unlock(); +} +EXPORT_SYMBOL_GPL(static_key_disable); + static void __static_key_slow_dec(struct static_key *key, unsigned long rate_limit, struct delayed_work *work) {