Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33;
Feedback-ID: iad51458e:Fastmail
Date:   Wed, 27 Sep 2023 23:06:09 -0700
From:   Boqun Feng <boqun.feng@gmail.com>
To:     Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc:     linux-kernel@vger.kernel.org, rcu@vger.kernel.org,
        "Paul E. McKenney" <paulmck@kernel.org>,
        Frederic Weisbecker <frederic@kernel.org>,
        Ingo Molnar <mingo@redhat.com>,
        Joel Fernandes <joel@joelfernandes.org>,
        John Ogness <john.ogness@linutronix.de>,
        Josh Triplett <josh@joshtriplett.org>,
        Lai Jiangshan <jiangshanlai@gmail.com>,
        Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
        Neeraj Upadhyay <quic_neeraju@quicinc.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Steven Rostedt <rostedt@goodmis.org>,
        Thomas Gleixner <tglx@linutronix.de>,
        Waiman Long <longman@redhat.com>,
        Will Deacon <will@kernel.org>,
        Zqiang <qiang.zhang1211@gmail.com>
Subject: Re: [RFC PATCH] srcu: Use try-lock lockdep annotation for NMI-safe
 access.
Message-ID: <ZRUX0YUrXfepRGKE@Boquns-Mac-mini.home>
References: <20230927160231.XRCDDSK4@linutronix.de>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20230927160231.XRCDDSK4@linutronix.de>
Precedence: bulk

On Wed, Sep 27, 2023 at 06:02:31PM +0200, Sebastian Andrzej Siewior wrote:
> It is claimed that srcu_read_lock_nmisafe() NMI-safe. However it
> triggers a lockdep if used from NMI because lockdep expects a deadlock
> since nothing disables NMIs while the lock is acquired.
> 
> Use a try-lock annotation for srcu_read_lock_nmisafe() to avoid lockdep
> complains if used from NMI.
> 
> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> ---
> 
> The splat:
> | ================================
> | WARNING: inconsistent lock state
> | 6.6.0-rc3-rt5+ #85 Not tainted
> | --------------------------------
> | inconsistent {INITIAL USE} -> {IN-NMI} usage.
> | swapper/0/0 [HC1[1]:SC0[0]:HE0:SE1] takes:
> | ffffffff828e6c90 (console_srcu){....}-{0:0}, at: console_srcu_read_lock+0x3a/0x50
> | {INITIAL USE} state was registered at:
> …
> |        CPU0
> |        ----
> |   lock(console_srcu);
> |   <Interrupt>
> |     lock(console_srcu);
> |
> |  *** DEADLOCK ***
> |
> 
> My guess is that trylock annotation should not apply to
> rcu_lock_acquire(). This would distinguish it from from non-NMI safe
> srcu_read_lock_nmisafe() and NMI check in rcu_read_unlock() is only
> there to survive if accidentally used in-NMI.

I think this is a "side-effect" of commit f0f44752f5f6 ("rcu: Annotate
SRCU's update-side lockdep dependencies"). In verify_lock_unused(), i.e.
the checking for NMI lock usages, the logic is that

1)	read lock usages in NMI conflicts with write lock usage in
	normal context (i.e. LOCKF_USED)

2)	write lock usage in NMI conflicts with read and write lock usage
	in normal context (i.e. LOCKF_USED | LOCKF_USED_READ)

before that commit, only read-side of SRCU is annotated, in other words,
SRCU only has read lock usage from lockdep PoV, but after that commit,
we annotate synchronize_srcu() as a write lock usage, so that we can
detect deadlocks between *normal* srcu_read_lock() and
synchronize_srcu(), however the side effect is now SRCU has a write lock
usage from lockdep PoV.

Actually in the above commit, I explicitly leave
srcu_read_lock_nmisafe() alone since its locking rules may be different
compared to srcu_read_lock(). In lockdep terms, srcu_read_lock_nmisafe()
is a !check read lock and srcu_read_lock() is a check read lock. Maybe
instead of using the trylock trick, we change lockdep to igore !check
locks for NMI context detection? Untested code as below:

diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c
index e85b5ad3e206..1af8d44e5eb4 100644
--- a/kernel/locking/lockdep.c
+++ b/kernel/locking/lockdep.c
@@ -5727,8 +5727,9 @@ void lock_acquire(struct lockdep_map *lock, unsigned int subclass,
                return;

        if (unlikely(!lockdep_enabled())) {
+               /* Only do NMI context checking if it's a check lock */
                /* XXX allow trylock from NMI ?!? */
-               if (lockdep_nmi() && !trylock) {
+               if (check && lockdep_nmi() && !trylock) {
                        struct held_lock hlock;

                        hlock.acquire_ip = ip;

Peter, thoughts?

Of course, either way, we need

Fixes: f0f44752f5f6 ("rcu: Annotate SRCU's update-side lockdep dependencies")

Regards,
Boqun

> 
>  include/linux/rcupdate.h | 6 ++++++
>  include/linux/srcu.h     | 2 +-
>  2 files changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> index 5e5f920ade909..44aab5c0bd2c1 100644
> --- a/include/linux/rcupdate.h
> +++ b/include/linux/rcupdate.h
> @@ -303,6 +303,11 @@ static inline void rcu_lock_acquire(struct lockdep_map *map)
>  	lock_acquire(map, 0, 0, 2, 0, NULL, _THIS_IP_);
>  }
>  
> +static inline void rcu_try_lock_acquire(struct lockdep_map *map)
> +{
> +	lock_acquire(map, 0, 1, 2, 0, NULL, _THIS_IP_);
> +}
> +
>  static inline void rcu_lock_release(struct lockdep_map *map)
>  {
>  	lock_release(map, _THIS_IP_);
> @@ -317,6 +322,7 @@ int rcu_read_lock_any_held(void);
>  #else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
>  
>  # define rcu_lock_acquire(a)		do { } while (0)
> +# define rcu_try_lock_acquire(a)	do { } while (0)
>  # define rcu_lock_release(a)		do { } while (0)
>  
>  static inline int rcu_read_lock_held(void)
> diff --git a/include/linux/srcu.h b/include/linux/srcu.h
> index 127ef3b2e6073..236610e4a8fa5 100644
> --- a/include/linux/srcu.h
> +++ b/include/linux/srcu.h
> @@ -229,7 +229,7 @@ static inline int srcu_read_lock_nmisafe(struct srcu_struct *ssp) __acquires(ssp
>  
>  	srcu_check_nmi_safety(ssp, true);
>  	retval = __srcu_read_lock_nmisafe(ssp);
> -	rcu_lock_acquire(&ssp->dep_map);
> +	rcu_try_lock_acquire(&ssp->dep_map);
>  	return retval;
>  }
>  
> -- 
> 2.40.1
>