Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 5 Jul 2023 08:52:38 +0300
From:   Leon Romanovsky <leon@kernel.org>
To:     Chengfeng Ye <dg573847474@gmail.com>
Cc:     dennis.dalessandro@cornelisnetworks.com, jgg@ziepe.ca,
        linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: [PATCH] IB/hfi1: Fix potential deadlock on &sde->flushlist_lock
Message-ID: <20230705055238.GG6455@unreal>
References: <20230628045925.5261-1-dg573847474@gmail.com>
 <20230704114849.GA6455@unreal>
 <CAAo+4rXkMM87OJzim=8dACdV=kWK_1yXeD=W5GZzHoJ2Gz6rtw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAAo+4rXkMM87OJzim=8dACdV=kWK_1yXeD=W5GZzHoJ2Gz6rtw@mail.gmail.com>
Precedence: bulk

On Wed, Jul 05, 2023 at 01:42:31AM +0800, Chengfeng Ye wrote:
> > Plus, we already in context where interrupts are stopped.
> 
> Indeed they can be called from .ndo_start_xmit callback and
> the document said it is with bh disabled.
> 
> But I found some call chain from the user process that seems could
> be called from irq disabled context. For sdma_send_txlist(),
> there is a call chain.
> 
> -> hfi1_write_iter()  (.write_iter callback)
> -> hfi1_user_sdma_process_request()
> -> user_sdma_send_pkts()
> -> sdma_send_txlist()
> 
> The .write_iter seems not to disable irq by default, as mentioned by
> https://www.kernel.org/doc/Documentation/filesystems/vfs.txt
> And I didn't find any explicit disabling or bh or irq along the call path,
> and also see several  copy_from_usr() which cannot be invoked under
> irq context.
> 
> 
> For sdma_send_txreq(), there is a call chain.
> 
> -> qp_priv_alloc()
> -> iowait_init() (register _hfi1_do_tid_send() as a work queue)
> -> _hfi1_do_tid_send() (workqueue)
> -> hfi1_do_tid_send()
> -> hfi1_verbs_send()
> -> sr(qp, ps, 0) (sr could points to hfi1_verbs_send_dm())
> -> hfi1_verbs_send_dma()
> -> sdma_send_txreq()
> 
> _hfi1_do_tid_send() is a work queue without irq disabled by default,
> I also check the remaining call path and also found that there is no explicit
> irq disable, instead the call site of hfi1_verbs_send() is exactly after
> spin_lock_irq_restore(), seems like a hint that it is probably called withirq
> enable.

Right, that path is called in process context and can sleep, there is no
need in irq disabled variant there.

> 
> Another hint is that the lock acquisition of
> spin_lock_irqsave(&sde->tail_lock, flags);
> just before my patch in the same function also disable irq, seems like another
> hint that this function could be called with interrupt disable,

Exactly, we already called to spin_lock_irqsave(), there is no value in
doing it twice.
void f() {
	spin_lock_irqsave(...)
	spin_lock_irqsave(...)
	....
	spin_unlock_irqrestore(...)
	spin_unlock_irqrestore(...)
}

is exactly the same as
void f() {
	spin_lock_irqsave(...)
	spin_lock(...)
	....
	spin_unlock(...)
	spin_unlock_irqrestore(...)
}

Thanks