Received: by 2002:ad5:474a:0:0:0:0:0 with SMTP id i10csp2956640imu; Mon, 19 Nov 2018 08:36:05 -0800 (PST) X-Google-Smtp-Source: AJdET5fDUsvpnfU02QRW3VG2Nwnyh+tYt3+c7gFjUarwJt9EC0yVxc7UyItK3n9M/IeHYrWUgCJe X-Received: by 2002:a63:b218:: with SMTP id x24mr20122304pge.223.1542645365570; Mon, 19 Nov 2018 08:36:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542645365; cv=none; d=google.com; s=arc-20160816; b=lXEuRTkwwhNeb88zNl8vR5V+ONSPDlhgId27RYMlIVxUucTiSATB5z+HU19ujX2Xrv iI5PPGPvlLCjxHCZbyoV0iDh1jfQDn3JzTA7NZhF/QOV9yLpB6Ex6/ZZBA5QA/i3w2fW gBHSz5flXSaKAks47+k33UUL1t/zEF/iJcROTpm9JDC0+whgKCg6Lz1VokZDCnJJMI6d Us+4WuBklR7lpVrC1FGpOk4dacfooAZsb2AWxQ2bSpBFrJL3e++DHmg6jCMEMwyuXa3C TfQzaSiaCEmtfn2UseNtHdNfUA3lOIKtMGMZNby8x9zd9sr1T7DqvVoxH/h+TtwBexvv YlWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=Ne2+nHIP+7RJQR8Dd7srX7uf+VluYpxWfCyNC455PFw=; b=aZKjxg7LHmhYPPE2RMouFisn+dYAworPFpnfBO8dbcHOZvmqPbHnizwo03OpgHZLMV 4Z6zEUDF3aYJXpPmIXBHJ9i+VnIBbB5FJ/ty4SEf3qxmiPlqqFT9T3UkIfRZLe90HERY 100vtO/Z6O8Vp9B7oyVyXjyZ29rRUDh9GRvxG7SslhC8i64kz18J69aa9QJRE1hWDNvs 6eSQYuW08cqLTkeN2nh4o7FtwB/qHYLQP+wuXuuhO1R3tKo2IJDZ5la/MzEQzPo/Fzcl XNvYMOeUqkVhIG8QhDaY7oIIzZ7ov+q+gQ7mGpgO1sE+hOGXuGCYbAAAwRBHPB/1jqww LwzQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=lzFiVWgC; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id a2si39636015pgm.154.2018.11.19.08.35.50; Mon, 19 Nov 2018 08:36:05 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=lzFiVWgC; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731400AbeKTC6g (ORCPT + 99 others); Mon, 19 Nov 2018 21:58:36 -0500 Received: from mail.kernel.org ([198.145.29.99]:57670 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731389AbeKTC6f (ORCPT ); Mon, 19 Nov 2018 21:58:35 -0500 Received: from localhost (5356596B.cm-6-7b.dynamic.ziggo.nl [83.86.89.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id B74F32089F; Mon, 19 Nov 2018 16:34:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1542645267; bh=WTO9ZRcuQlc4XEam7Z50CQC/adgNeartxlFpDYeqmsk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lzFiVWgCkg6EuAGQrMHvtE8QMHvlJfbdQcp3HcSZ9VqSF4mM7pi6Bxc9UkSeUz6m9 YCqxU3ExG7/DhvMAkJZt6qjBfdIBfZQnnZtTUR/UKCn8KcNuRKcRbZZVDimMQrtp6Y wM56xX8Z550KRhn3LlJQSAB55qclNg/hYiafKMSw= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Andrew Jones , Bart Van Assche , linux-scsi@vger.kernel.org, "Martin K. Petersen" , Christoph Hellwig , "James E.J. Bottomley" , "jianchao.wang" , Ming Lei , Jens Axboe Subject: [PATCH 4.19 098/205] SCSI: fix queue cleanup race before queue initialization is done Date: Mon, 19 Nov 2018 17:26:45 +0100 Message-Id: <20181119162633.027739711@linuxfoundation.org> X-Mailer: git-send-email 2.19.1 In-Reply-To: <20181119162616.586062722@linuxfoundation.org> References: <20181119162616.586062722@linuxfoundation.org> User-Agent: quilt/0.65 X-stable: review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org 4.19-stable review patch. If anyone has any objections, please let me know. ------------------ From: Ming Lei commit 8dc765d438f1e42b3e8227b3b09fad7d73f4ec9a upstream. c2856ae2f315d ("blk-mq: quiesce queue before freeing queue") has already fixed this race, however the implied synchronize_rcu() in blk_mq_quiesce_queue() can slow down LUN probe a lot, so caused performance regression. Then 1311326cf4755c7 ("blk-mq: avoid to synchronize rcu inside blk_cleanup_queue()") tried to quiesce queue for avoiding unnecessary synchronize_rcu() only when queue initialization is done, because it is usual to see lots of inexistent LUNs which need to be probed. However, turns out it isn't safe to quiesce queue only when queue initialization is done. Because when one SCSI command is completed, the user of sending command can be waken up immediately, then the scsi device may be removed, meantime the run queue in scsi_end_request() is still in-progress, so kernel panic can be caused. In Red Hat QE lab, there are several reports about this kind of kernel panic triggered during kernel booting. This patch tries to address the issue by grabing one queue usage counter during freeing one request and the following run queue. Fixes: 1311326cf4755c7 ("blk-mq: avoid to synchronize rcu inside blk_cleanup_queue()") Cc: Andrew Jones Cc: Bart Van Assche Cc: linux-scsi@vger.kernel.org Cc: Martin K. Petersen Cc: Christoph Hellwig Cc: James E.J. Bottomley Cc: stable Cc: jianchao.wang Signed-off-by: Ming Lei Signed-off-by: Jens Axboe Signed-off-by: Greg Kroah-Hartman --- block/blk-core.c | 5 ++--- drivers/scsi/scsi_lib.c | 8 ++++++++ 2 files changed, 10 insertions(+), 3 deletions(-) --- a/block/blk-core.c +++ b/block/blk-core.c @@ -793,9 +793,8 @@ void blk_cleanup_queue(struct request_qu * dispatch may still be in-progress since we dispatch requests * from more than one contexts. * - * No need to quiesce queue if it isn't initialized yet since - * blk_freeze_queue() should be enough for cases of passthrough - * request. + * We rely on driver to deal with the race in case that queue + * initialization isn't done. */ if (q->mq_ops && blk_queue_init_done(q)) blk_mq_quiesce_queue(q); --- a/drivers/scsi/scsi_lib.c +++ b/drivers/scsi/scsi_lib.c @@ -697,6 +697,12 @@ static bool scsi_end_request(struct requ */ scsi_mq_uninit_cmd(cmd); + /* + * queue is still alive, so grab the ref for preventing it + * from being cleaned up during running queue. + */ + percpu_ref_get(&q->q_usage_counter); + __blk_mq_end_request(req, error); if (scsi_target(sdev)->single_lun || @@ -704,6 +710,8 @@ static bool scsi_end_request(struct requ kblockd_schedule_work(&sdev->requeue_work); else blk_mq_run_hw_queues(q, true); + + percpu_ref_put(&q->q_usage_counter); } else { unsigned long flags;