Received: by 2002:ad5:474a:0:0:0:0:0 with SMTP id i10csp3478171imu; Sun, 11 Nov 2018 16:01:55 -0800 (PST) X-Google-Smtp-Source: AJdET5dNFpljIkiCTL3tXHtziuHyaVIahnxpycTJKTWeS0LFJ91CGXW52xHGw1lxwKohIAmNJ6sI X-Received: by 2002:a65:65c9:: with SMTP id y9mr15935780pgv.438.1541980915708; Sun, 11 Nov 2018 16:01:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1541980915; cv=none; d=google.com; s=arc-20160816; b=Es0NPihYV8d67TVqGofYwc/hzDmxzC9yE9jI9S0S9knFoKFV1ZU1sQGvdpAlq1/0fy ZFYhjgSz5+/kYNBGWvyuYotqABq9Yv3fXKCe3LHcZ0PygqkpQmJMYzbQZhZZ+7oiiJgB sGOW1jdwGK3K5uI14acvn3m3USdLjGfcHTFl9uchEdY3cuPdn0tXhMaZOS/0WGN4xjGE l1SkgJPzolF3dVQxttqz4sByaHhIKoh3q7k7KoL9nsHSLBpauPJsNuh359Pz8qDaSxeO p3KVB75NEX/h0e9nnUJyrm3o54x/zwkaGdbWBaPSr+SSuIN+zVfdkfere346nNrymvr5 H3fg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=FADr2ec8jdSXvFWajayEGcikg8061+biGk9awCw4rEY=; b=vFjpAUt+daZWk7CLRWO5y6rIlXmImKt8s71/6zbWNi6Jua2O2McgAHQQ5jYak7kfWw g4uFrfQYor1HTan5vmH4ychU5XOmxMjMLZGZFvCimGPbGsaocINh551cIOLbwNbM4w3u N9ypCldW+4ynf85Jktc55GCk3Fyw6LuCE15zy3VD/g/5r6ezWULwkFQiEsALDlKxvfM7 cwDst4BhEJFqfUGisAYLTMteApn6pcRFrE5F9sXg29/EhkH/1p39pIqAwvmqop7xRhG0 LboBlqQldFyRKM7thAQfTy5jWaFjocE3OxmfcS/X10xxXIZfxL+UGW9IupYQj+f5YU+l KzRQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=SWQZEvHl; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id a34-v6si17049822pld.249.2018.11.11.16.01.40; Sun, 11 Nov 2018 16:01:55 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=SWQZEvHl; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732263AbeKLIR5 (ORCPT + 99 others); Mon, 12 Nov 2018 03:17:57 -0500 Received: from mail.kernel.org ([198.145.29.99]:34210 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732069AbeKLIR5 (ORCPT ); Mon, 12 Nov 2018 03:17:57 -0500 Received: from localhost (unknown [206.108.79.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 6127A208A3; Sun, 11 Nov 2018 22:28:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1541975281; bh=K7zCT4v+Nn2ocX3mAF2b+vbr2lqMfkJeOI7Ir4eb56Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SWQZEvHl+EuNFpfN3Tpri1BTZi80pG/sjGBvNtSSo+kTIL/ZLzzRjPSHxRxJuolPA 2J7PNhpJDKJSlXtYx92JOmXl2+ZK1y0G1qOusNpJlXzknCAaSuKa1Hg+sNMzWw3sTv kkL+q2bHqOI9z0UcQz6pGXo96nl0wwSmqpHlJ1Ic= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Waiman Long , Dave Jiang , Vinod Koul , Sasha Levin Subject: [PATCH 4.19 175/361] driver/dma/ioat: Call del_timer_sync() without holding prep_lock Date: Sun, 11 Nov 2018 14:18:42 -0800 Message-Id: <20181111221644.800626655@linuxfoundation.org> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20181111221619.915519183@linuxfoundation.org> References: <20181111221619.915519183@linuxfoundation.org> User-Agent: quilt/0.65 X-stable: review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org 4.19-stable review patch. If anyone has any objections, please let me know. ------------------ From: Waiman Long [ Upstream commit cfb03be6c7e8a1591285849c361d67b09f5149f7 ] The following lockdep splat was observed: [ 1222.241750] ====================================================== [ 1222.271301] WARNING: possible circular locking dependency detected [ 1222.301060] 4.16.0-10.el8+5.x86_64+debug #1 Not tainted [ 1222.326659] ------------------------------------------------------ [ 1222.356565] systemd-shutdow/1 is trying to acquire lock: [ 1222.382660] ((&ioat_chan->timer)){+.-.}, at: [<00000000f71e1a28>] del_timer_sync+0x5/0xf0 [ 1222.422928] [ 1222.422928] but task is already holding lock: [ 1222.451743] (&(&ioat_chan->prep_lock)->rlock){+.-.}, at: [<000000008ea98b12>] ioat_shutdown+0x86/0x100 [ioatdma] : [ 1223.524987] Chain exists of: [ 1223.524987] (&ioat_chan->timer) --> &(&ioat_chan->cleanup_lock)->rlock --> &(&ioat_chan->prep_lock)->rlock [ 1223.524987] [ 1223.594082] Possible unsafe locking scenario: [ 1223.594082] [ 1223.622630] CPU0 CPU1 [ 1223.645080] ---- ---- [ 1223.667404] lock(&(&ioat_chan->prep_lock)->rlock); [ 1223.691535] lock(&(&ioat_chan->cleanup_lock)->rlock); [ 1223.728657] lock(&(&ioat_chan->prep_lock)->rlock); [ 1223.765122] lock((&ioat_chan->timer)); [ 1223.784095] [ 1223.784095] *** DEADLOCK *** [ 1223.784095] [ 1223.813492] 4 locks held by systemd-shutdow/1: [ 1223.834677] #0: (reboot_mutex){+.+.}, at: [<0000000056d33456>] SYSC_reboot+0x10f/0x300 [ 1223.873310] #1: (&dev->mutex){....}, at: [<00000000258dfdd7>] device_shutdown+0x1c8/0x660 [ 1223.913604] #2: (&dev->mutex){....}, at: [<0000000068331147>] device_shutdown+0x1d6/0x660 [ 1223.954000] #3: (&(&ioat_chan->prep_lock)->rlock){+.-.}, at: [<000000008ea98b12>] ioat_shutdown+0x86/0x100 [ioatdma] In the ioat_shutdown() function: spin_lock_bh(&ioat_chan->prep_lock); set_bit(IOAT_CHAN_DOWN, &ioat_chan->state); del_timer_sync(&ioat_chan->timer); spin_unlock_bh(&ioat_chan->prep_lock); According to the synchronization rule for the del_timer_sync() function, the caller must not hold locks which would prevent completion of the timer's handler. The timer structure has its own lock that manages its synchronization. Setting the IOAT_CHAN_DOWN bit should prevent other CPUs from trying to use that device anyway, there is probably no need to call del_timer_sync() while holding the prep_lock. So the del_timer_sync() call is now moved outside of the prep_lock critical section to prevent the circular lock dependency. Signed-off-by: Waiman Long Reviewed-by: Dave Jiang Signed-off-by: Vinod Koul Signed-off-by: Sasha Levin Signed-off-by: Greg Kroah-Hartman --- drivers/dma/ioat/init.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) --- a/drivers/dma/ioat/init.c +++ b/drivers/dma/ioat/init.c @@ -1205,8 +1205,15 @@ static void ioat_shutdown(struct pci_dev spin_lock_bh(&ioat_chan->prep_lock); set_bit(IOAT_CHAN_DOWN, &ioat_chan->state); - del_timer_sync(&ioat_chan->timer); spin_unlock_bh(&ioat_chan->prep_lock); + /* + * Synchronization rule for del_timer_sync(): + * - The caller must not hold locks which would prevent + * completion of the timer's handler. + * So prep_lock cannot be held before calling it. + */ + del_timer_sync(&ioat_chan->timer); + /* this should quiesce then reset */ ioat_reset_hw(ioat_chan); }