Received-SPF: pass (google.com: domain of linux-kernel+bounces-29113-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Message-ID: <5a758416-8c70-4a0e-834d-fecf6c95c7d8@redhat.com>
Date: Wed, 17 Jan 2024 09:58:02 -0500
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH] lockdep: fix deadlock issue between lockdep and rcu
Content-Language: en-US
To: Xuewen Yan <xuewen.yan94@gmail.com>, Boqun Feng <boqun.feng@gmail.com>
Cc: Zhiguo Niu <zhiguo.niu@unisoc.com>, peterz@infradead.org,
 mingo@redhat.com, will@kernel.org, linux-kernel@vger.kernel.org,
 niuzhiguo84@gmail.com, ke.wang@unisoc.com, xuewen.yan@unisoc.com
References: <1705308796-13547-1-git-send-email-zhiguo.niu@unisoc.com>
 <ZabBHHwZd70IDDxP@boqun-archlinux>
 <CAB8ipk--3-+K2bhQS_YMTgpxCkZRbX21Lv_x-nT298yx5YTJeg@mail.gmail.com>
From: Waiman Long <longman@redhat.com>
In-Reply-To: <CAB8ipk--3-+K2bhQS_YMTgpxCkZRbX21Lv_x-nT298yx5YTJeg@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit


On 1/16/24 23:35, Xuewen Yan wrote:
> On Wed, Jan 17, 2024 at 1:47 AM Boqun Feng <boqun.feng@gmail.com> wrote:
>> On Mon, Jan 15, 2024 at 04:53:16PM +0800, Zhiguo Niu wrote:
>>> There is a deadlock scenario between lockdep and rcu when
>>> rcu nocb feature is enabled, just as following call stack:
>>>
>>>       rcuop/x
>>> -000|queued_spin_lock_slowpath(lock = 0xFFFFFF817F2A8A80, val = ?)
>>> -001|queued_spin_lock(inline) // try to hold nocb_gp_lock
>>> -001|do_raw_spin_lock(lock = 0xFFFFFF817F2A8A80)
>>> -002|__raw_spin_lock_irqsave(inline)
>>> -002|_raw_spin_lock_irqsave(lock = 0xFFFFFF817F2A8A80)
>>> -003|wake_nocb_gp_defer(inline)
>>> -003|__call_rcu_nocb_wake(rdp = 0xFFFFFF817F30B680)
>>> -004|__call_rcu_common(inline)
>>> -004|call_rcu(head = 0xFFFFFFC082EECC28, func = ?)
>>> -005|call_rcu_zapped(inline)
>>> -005|free_zapped_rcu(ch = ?)// hold graph lock
>>> -006|rcu_do_batch(rdp = 0xFFFFFF817F245680)
>>> -007|nocb_cb_wait(inline)
>>> -007|rcu_nocb_cb_kthread(arg = 0xFFFFFF817F245680)
>>> -008|kthread(_create = 0xFFFFFF80803122C0)
>>> -009|ret_from_fork(asm)
>>>
>>>       rcuop/y
>>> -000|queued_spin_lock_slowpath(lock = 0xFFFFFFC08291BBC8, val = 0)
>>> -001|queued_spin_lock()
>>> -001|lockdep_lock()
>>> -001|graph_lock() // try to hold graph lock
>>> -002|lookup_chain_cache_add()
>>> -002|validate_chain()
>>> -003|lock_acquire
>>> -004|_raw_spin_lock_irqsave(lock = 0xFFFFFF817F211D80)
>>> -005|lock_timer_base(inline)
>>> -006|mod_timer(inline)
>>> -006|wake_nocb_gp_defer(inline)// hold nocb_gp_lock
>>> -006|__call_rcu_nocb_wake(rdp = 0xFFFFFF817F2A8680)
>>> -007|__call_rcu_common(inline)
>>> -007|call_rcu(head = 0xFFFFFFC0822E0B58, func = ?)
>>> -008|call_rcu_hurry(inline)
>>> -008|rcu_sync_call(inline)
>>> -008|rcu_sync_func(rhp = 0xFFFFFFC0822E0B58)
>>> -009|rcu_do_batch(rdp = 0xFFFFFF817F266680)
>>> -010|nocb_cb_wait(inline)
>>> -010|rcu_nocb_cb_kthread(arg = 0xFFFFFF817F266680)
>>> -011|kthread(_create = 0xFFFFFF8080363740)
>>> -012|ret_from_fork(asm)
>>>
>>> rcuop/x and rcuop/y are rcu nocb threads with the same nocb gp thread.
>>>
>> Nice! Looks like you find the root cause ;-) nocb_gp_lock and graph_lock
>> have an ABBA deadlock due to lockdep's dependency on RCU. I assume this
>> actually fixes the problem you saw?
>>
>> However, I want to suggest a different fix, please see below:
>>
>>> This patch release the graph lock before lockdep call_rcu.
>>>
>>> Signed-off-by: Zhiguo Niu <zhiguo.niu@unisoc.com>
>>> Signed-off-by: Xuewen Yan <xuewen.yan@unisoc.com>
>>> ---
>>>   kernel/locking/lockdep.c | 38 +++++++++++++++++++++++++-------------
>>>   1 file changed, 25 insertions(+), 13 deletions(-)
>>>
>>> diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c
>>> index 151bd3d..c1d432a 100644
>>> --- a/kernel/locking/lockdep.c
>>> +++ b/kernel/locking/lockdep.c
>>> @@ -6186,23 +6186,29 @@ static struct pending_free *get_pending_free(void)
>>>   /*
>>>    * Schedule an RCU callback if no RCU callback is pending. Must be called with
>>>    * the graph lock held.
>>> + *
>>> + * Return true if graph lock need be released by the caller, otherwise false
>>> + * means graph lock is released by itself.
>>>    */
>>> -static void call_rcu_zapped(struct pending_free *pf)
>>> +static bool call_rcu_zapped(struct pending_free *pf)
>>>   {
>>>        WARN_ON_ONCE(inside_selftest());
>>>
>>>        if (list_empty(&pf->zapped))
>>> -             return;
>>> +             return true;
>>>
>>>        if (delayed_free.scheduled)
>>> -             return;
>>> +             return true;
>>>
>>>        delayed_free.scheduled = true;
>>>
>>>        WARN_ON_ONCE(delayed_free.pf + delayed_free.index != pf);
>>>        delayed_free.index ^= 1;
>>>
>>> +     lockdep_unlock();
>>>        call_rcu(&delayed_free.rcu_head, free_zapped_rcu);
>>> +
>>> +     return false;
>>>   }
>>>
>>>   /* The caller must hold the graph lock. May be called from RCU context. */
>>> @@ -6228,6 +6234,7 @@ static void free_zapped_rcu(struct rcu_head *ch)
>>>   {
>>>        struct pending_free *pf;
>>>        unsigned long flags;
>>> +     bool need_unlock;
>>>
>>>        if (WARN_ON_ONCE(ch != &delayed_free.rcu_head))
>>>                return;
>>> @@ -6243,9 +6250,9 @@ static void free_zapped_rcu(struct rcu_head *ch)
>>>        /*
>>>         * If there's anything on the open list, close and start a new callback.
>>>         */
>>> -     call_rcu_zapped(delayed_free.pf + delayed_free.index);
>>> -
>>> -     lockdep_unlock();
>>> +     need_unlock = call_rcu_zapped(delayed_free.pf + delayed_free.index);
>>> +     if (need_unlock)
>>> +             lockdep_unlock();
>> Instead of returning a bool to control the unlock, I think it's better
>> that we refactor the call_rcu_zapped() a bit, so it becomes a
>> prepare_call_rcu_zapped():
>>
>>          // See if we need to queue an RCU callback, must called with
>>          // the lockdep lock held, returns false if either we don't have
>>          // any pending free or the callback is already scheduled.
>>          // Otherwise, a call_rcu() must follow this function call.
>>          static bool prepare_call_rcu_zapped(struct pending_free *pf)
>>          {
>>                  WARN_ON_ONCE(inside_selftest());
>>
>>                  if (list_empty(&pf->zapped))
>>                          return false;
>>
>>                  if (delayed_free.scheduled)
>>                          return false;
>>
>>                  delayed_free.scheduled = true;
>>
>>                  WARN_ON_ONCE(delayed_free.pf + delayed_free.index != pf);
>>                  delayed_free.index ^= 1;
>>
>>                  return true;
>>          }
>>
>> , and here we can:
>>
>>          <lockdep_lock() is called previous>
>>          need_callback = prepare_call_rcu_zapped(...);
>>          lockdep_unlock();
>>          raw_local_irq_restore(flags);
>>
>>          if (need_callback)
>>                  call_rcu(&delayed_free.rcu_head, free_zapped_rcu);
> Would there be any problems if call_rcu is placed outside the shutdown
> interrupt?

call_rcu() doesn't need to be called with interrupt disabled. In fact, 
it calls local_irq_save() itself when necessary. So that is perfectly fine.

Cheers,
Longman