Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
From:   "Saleem, Shiraz" <shiraz.saleem@intel.com>
To:     Dan Carpenter <dan.carpenter@oracle.com>,
        "duoming@zju.edu.cn" <duoming@zju.edu.cn>
CC:     "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "chris@zankel.net" <chris@zankel.net>,
        "jcmvbkbc@gmail.com" <jcmvbkbc@gmail.com>,
        "Ismail, Mustafa" <mustafa.ismail@intel.com>,
        "jgg@ziepe.ca" <jgg@ziepe.ca>,
        "wg@grandegger.com" <wg@grandegger.com>,
        "mkl@pengutronix.de" <mkl@pengutronix.de>,
        "davem@davemloft.net" <davem@davemloft.net>,
        "kuba@kernel.org" <kuba@kernel.org>,
        "pabeni@redhat.com" <pabeni@redhat.com>,
        "jes@trained-monkey.org" <jes@trained-monkey.org>,
        "gregkh@linuxfoundation.org" <gregkh@linuxfoundation.org>,
        "jirislaby@kernel.org" <jirislaby@kernel.org>,
        "alexander.deucher@amd.com" <alexander.deucher@amd.com>,
        "linux-xtensa@linux-xtensa.org" <linux-xtensa@linux-xtensa.org>,
        "linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
        "linux-can@vger.kernel.org" <linux-can@vger.kernel.org>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        "linux-hippi@sunsite.dk" <linux-hippi@sunsite.dk>,
        "linux-staging@lists.linux.dev" <linux-staging@lists.linux.dev>,
        "linux-serial@vger.kernel.org" <linux-serial@vger.kernel.org>,
        "linux-usb@vger.kernel.org" <linux-usb@vger.kernel.org>
Subject: RE: Re: [PATCH 09/11] drivers: infiniband: hw: Fix deadlock in
 irdma_cleanup_cm_core()
Thread-Topic: Re: [PATCH 09/11] drivers: infiniband: hw: Fix deadlock in
 irdma_cleanup_cm_core()
Thread-Index: AQHYSkn4HJrzWeSt/0qrCBQuiFUlDqzkT6MAgAAY8oCAACnjAIAAHSXQ
Date:   Thu, 7 Apr 2022 17:36:12 +0000
Message-ID: <MWHPR11MB00293D107510E728769874DFE9E69@MWHPR11MB0029.namprd11.prod.outlook.com>
References: <cover.1649310812.git.duoming@zju.edu.cn>
 <4069b99042d28c8e51b941d9e698b99d1656ed33.1649310812.git.duoming@zju.edu.cn>
 <20220407112455.GK3293@kadam>
 <1be0c02d.3f701.1800416ef60.Coremail.duoming@zju.edu.cn>
 <20220407142908.GO12805@kadam>
In-Reply-To: <20220407142908.GO12805@kadam>
Accept-Language: en-US
Content-Language: en-US
dlp-product: dlpe-windows
dlp-reaction: no-action
dlp-version: 11.6.401.20
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?dd7LHsxYuGsynP3Q2dy5PnnzKLE3MUAuU4pUhQrDecbK4oKnGpfjyf57ZQVr?=
 =?us-ascii?Q?GdRzmRzFSLoxxDZm0gp0cf0z9P7878bW0xIxunKZK0wJq+lRnt0ne4mRSbVr?=
 =?us-ascii?Q?GTFsrWpvhUoKpFjYIoloP2PH9wjLLGm7pmTXDNfaOnSw8YA/WQa/R7tgcUrH?=
 =?us-ascii?Q?INJxDVSPHT5Wx5cyDu+8AMIpjdyXq9D1EULjBzl1eX4FvfyJaOrhB19+ULVV?=
 =?us-ascii?Q?y5nTDwBxrmzg6DELOiecB3vBKBnHBoBTpWFRrOezGCRAgi6fNir7GT4L8ivd?=
 =?us-ascii?Q?NlvNm7qy7eXgvyQFAy3hOzq6Ur4NAS6vcfhQCOw0RiU4pmwgHlD8Ke/I+xFs?=
 =?us-ascii?Q?nbCP9PEMYDWz+X/EVfO60jTPCQGNzp2thbU+T5wSu1DPU1hdvIC5LnzHVPo2?=
 =?us-ascii?Q?Xzw/H8XAGbz638QsfZt43oKFnlXB3NihSm2vp39oFJQ0p3dxpECh4UKbHq3g?=
 =?us-ascii?Q?hevwAC0dDfufeY40QEahr5G04u31uSwyffYSGrNbaFgZl90m8rRgqiBwXi3O?=
 =?us-ascii?Q?Rwrhx+MLU/MaWlWEKJwPth9ZI86gJmVezUZKJvmx8rUUqFTFGlpSaX+MgEpa?=
 =?us-ascii?Q?G2VdWy0JdxvdTG83dsRe1bhwt/yHbnGU6P0FKFXFIw4Q29eo/uJDZ3ydNn/5?=
 =?us-ascii?Q?uE3JhgH38OyBBKC7Dqj6qmxsKKQKbHSy3LYUFzQHSHuPCjahA6r1H51kw7AD?=
 =?us-ascii?Q?U+9NBQrt1phNBWdCdZBvzOMxyHxMpuDDMIBWgsMO4pvyttm3vrnVUdhyaXdV?=
 =?us-ascii?Q?H8hoZWcI37Wsw63scC6K6YAmRrl01pKFzh2hi/89MwOP+WnC+9b3y8kd4rtw?=
 =?us-ascii?Q?3fRSVc2jYAS/vbPIqZ4x/i+kOZP2R0tVcT1z5RhKIPDx+x9T6E82OrQ0IpSD?=
 =?us-ascii?Q?fCMqa2BgJPbF2vKFy5OZGJFJss9YxVx3SHMQ3wcYbnpfI331PPmyR1OoFzx+?=
 =?us-ascii?Q?1Cr50xgGDHD7SL8/7aF9PSHtvn38S1lMuuCAQX8KTIiVUqaXieQ8ivDB8GIh?=
 =?us-ascii?Q?LNO+Q7DIpNg8XIanH+XKrMZrvaqBJhRKszwECJ6kUINr5Op33naoM/n1Zgbb?=
 =?us-ascii?Q?BIxwWk8e8qfo+cRqNfAQgyK/301GquARSHgy4YbC7/0CCbqYpSYDtCqCiPid?=
 =?us-ascii?Q?JDH20NuuE0LZCpoq0dFgXKaQG70VkCA5WZ4wpjcuDmfaCJxUjcvkfMldP+Bq?=
 =?us-ascii?Q?zqR211rL3Rrkv5qB6SYD0d7teolClN47XCMKIQrah+hjTF/SjZz58aqXEng8?=
 =?us-ascii?Q?j7jP9im+nO/ynRZRi7laQlGc6+KIiR43sXkcAlfoqrgI7fD4ubi5XPoHMQHB?=
 =?us-ascii?Q?ErBFlyHbMiRCk2zRwJlZ939E3JF7gDOQVifh2QUcsXOedKNtOe2bbHqMUCoc?=
 =?us-ascii?Q?Q3gEL0l9CWXt2cqJErfS5aQ/BTzrA0eP+QOs1Ncg70bwvyALNuG/PEHFfNSg?=
 =?us-ascii?Q?qII2ewZb00IcO6G0x0EgI6m3QUqSgJMC5mk8rAmW0MQq+ccQme7tzf5HU/d0?=
 =?us-ascii?Q?K0EAnd3a2jylSOhjTwgVrBsHw4hV0NIGG77++CSPrNo7Rvgc9/IZlGn10OAN?=
 =?us-ascii?Q?GzVbfdFI4Xd9ZyvNte3PH1+63b5+2FGnkmJ3TVIgsgsZTWEb23n3mOMTNXvD?=
 =?us-ascii?Q?bfbKGLLKwwwOy3iLjvk2rMM4t4Bb5jqbLxzhzyBRvBJ/mkKsGypSVTlMxX0P?=
 =?us-ascii?Q?HhqqtjHdVlZPqUTM+rRjzls4B14rQTKLnCbtIoDhbj0PUZ/tMaI8hpXAI0E0?=
 =?us-ascii?Q?OS06+xnJ+A=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: MWHPR11MB0029.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b0f5c653-50d9-4c36-9c06-08da18bd1c22
X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Apr 2022 17:36:12.4856
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: fTknXxz9EF5b34m5PtrxcgRi2WwbtyC6F+eqk5cFtjCEwlgQj0Odvbv28Fbn6mgyyRlFdE4nkXX5Ef1MVFYJAw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR11MB4210
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE,
	URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

> Subject: Re: Re: [PATCH 09/11] drivers: infiniband: hw: Fix deadlock in
> irdma_cleanup_cm_core()
>=20
> On Thu, Apr 07, 2022 at 08:54:13PM +0800, duoming@zju.edu.cn wrote:
> > Hello,
> >
> > On Thu, 7 Apr 2022 14:24:56 +0300 Dan Carpenter wrote:
> >
> > > > There is a deadlock in irdma_cleanup_cm_core(), which is shown
> > > > below:
> > > >
> > > >    (Thread 1)              |      (Thread 2)
> > > >                            | irdma_schedule_cm_timer()
> > > > irdma_cleanup_cm_core()    |  add_timer()
> > > >  spin_lock_irqsave() //(1) |  (wait a time)
> > > >  ...                       | irdma_cm_timer_tick()
> > > >  del_timer_sync()          |  spin_lock_irqsave() //(2)
> > > >  (wait timer to stop)      |  ...
> > > >
> > > > We hold cm_core->ht_lock in position (1) of thread 1 and use
> > > > del_timer_sync() to wait timer to stop, but timer handler also
> > > > need cm_core->ht_lock in position (2) of thread 2.
> > > > As a result, irdma_cleanup_cm_core() will block forever.
> > > >
> > > > This patch extracts del_timer_sync() from the protection of
> > > > spin_lock_irqsave(), which could let timer handler to obtain the
> > > > needed lock.
> > > >
> > > > Signed-off-by: Duoming Zhou <duoming@zju.edu.cn>
> > > > ---
> > > >  drivers/infiniband/hw/irdma/cm.c | 5 ++++-
> > > >  1 file changed, 4 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/drivers/infiniband/hw/irdma/cm.c
> > > > b/drivers/infiniband/hw/irdma/cm.c
> > > > index dedb3b7edd8..019dd8bfe08 100644
> > > > --- a/drivers/infiniband/hw/irdma/cm.c
> > > > +++ b/drivers/infiniband/hw/irdma/cm.c
> > > > @@ -3252,8 +3252,11 @@ void irdma_cleanup_cm_core(struct
> irdma_cm_core *cm_core)
> > > >  		return;
> > > >
> > > >  	spin_lock_irqsave(&cm_core->ht_lock, flags);
> > > > -	if (timer_pending(&cm_core->tcp_timer))
> > > > +	if (timer_pending(&cm_core->tcp_timer)) {
> > > > +		spin_unlock_irqrestore(&cm_core->ht_lock, flags);
> > > >  		del_timer_sync(&cm_core->tcp_timer);
> > > > +		spin_lock_irqsave(&cm_core->ht_lock, flags);
> > > > +	}
> > > >  	spin_unlock_irqrestore(&cm_core->ht_lock, flags);
> > >
> > > This lock doesn't seem to be protecting anything.  Also do we need
> > > to check timer_pending()?  I think the del_timer_sync() function
> > > will just return directly if there isn't a pending lock?
> >
> > Thanks a lot for your advice, I will remove the timer_pending() and
> > the redundant lock.
>=20
> I didn't give any advice. :P I only ask questions when I don't know the a=
nswers.
> Someone probably needs to look at &cm_core->ht_lock and figure out what i=
t's
> protecting.
>=20
Agreed on this fix.

We should not lock around del_timer_sync or need to check on timer_pending.

However, we do need serialize addition of a timer which can be called from =
multiple paths, i.e. the timer handler and irdma_schedule_cm_timer.

Shiraz