From: Bart Van Assche <Bart.VanAssche@sandisk.com>
To: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "hch@infradead.org" <hch@infradead.org>,
        "linux-block@vger.kernel.org" <linux-block@vger.kernel.org>,
        "tom.leiming@gmail.com" <tom.leiming@gmail.com>,
        "axboe@fb.com" <axboe@fb.com>
CC: "yizhan@redhat.com" <yizhan@redhat.com>,
        "tj@kernel.org" <tj@kernel.org>
Subject: Re: [PATCH 2/2] blk-mq: start to freeze queue just after setting
 dying
Thread-Topic: [PATCH 2/2] blk-mq: start to freeze queue just after setting
 dying
Thread-Index: AQHSmPZe5nYT78/Ep0GJ+9WTi9E4LQ==
Date: Thu, 9 Mar 2017 16:58:26 +0000
Message-ID: <1489078694.2597.5.camel@sandisk.com>
References: <1489064578-17305-1-git-send-email-tom.leiming@gmail.com>
         <1489064578-17305-4-git-send-email-tom.leiming@gmail.com>
In-Reply-To: <1489064578-17305-4-git-send-email-tom.leiming@gmail.com>
Accept-Language: nl-NL, en-US
Content-Language: en-US
Content-Type: text/plain; charset="iso-8859-1"
Content-ID: <B161DBF1B6873D4BA246D24357138861@sandisk.com>
MIME-Version: 1.0
WDCIPOUTBOUND: EOP-TRUE
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Mar 2017 16:58:31.7992
 (UTC)
X-MS-Exchange-CrossTenant-Id: b61c8803-16f3-4c35-9b17-6f65f441df86
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=b61c8803-16f3-4c35-9b17-6f65f441df86;Ip=[74.221.232.54];Helo=[sacsmgep14.sandisk.com]
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR0401MB1492
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by mail.home.local id v29GxeJl004328
Content-Length: 2497
Lines: 76

On Thu, 2017-03-09 at 21:02 +0800, Ming Lei wrote:
> Before commit 780db2071a(blk-mq: decouble blk-mq freezing
> from generic bypassing), the dying flag is checked before
> entering queue, and Tejun converts the checking into .mq_freeze_depth,
> and assumes the counter is increased just after dying flag
> is set. Unfortunately we doesn't do that in blk_set_queue_dying().
> 
> This patch calls blk_mq_freeze_queue_start() for blk-mq in
> blk_set_queue_dying(), so that we can block new I/O coming
> once the queue is set as dying.
> 
> Given blk_set_queue_dying() is always called in remove path
> of block device, and queue will be cleaned up later, we don't
> need to worry about undo of the counter.
> 
> Cc: Tejun Heo <tj@kernel.org>
> Signed-off-by: Ming Lei <tom.leiming@gmail.com>
> ---
>  block/blk-core.c | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/block/blk-core.c b/block/blk-core.c
> index 0eeb99ef654f..559487e58296 100644
> --- a/block/blk-core.c
> +++ b/block/blk-core.c
> @@ -500,9 +500,12 @@ void blk_set_queue_dying(struct request_queue *q)
>  	queue_flag_set(QUEUE_FLAG_DYING, q);
>  	spin_unlock_irq(q->queue_lock);
>  
> -	if (q->mq_ops)
> +	if (q->mq_ops) {
>  		blk_mq_wake_waiters(q);
> -	else {
> +
> +		/* block new I/O coming */
> +		blk_mq_freeze_queue_start(q);
> +	} else {
>  		struct request_list *rl;
>  
>  		spin_lock_irq(q->queue_lock);

The comment above blk_mq_freeze_queue_start() should explain more clearly
why that call is needed. Additionally, I think this patch makes the
blk_freeze_queue() call in blk_cleanup_queue() superfluous. How about the
(entirely untested) patch below?

diff --git a/block/blk-core.c b/block/blk-core.c
index 1086dac8724c..3ce48f2d65cf 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -500,6 +500,12 @@ void blk_set_queue_dying(struct request_queue *q)
?	queue_flag_set(QUEUE_FLAG_DYING, q);
?	spin_unlock_irq(q->queue_lock);
?
+	/*
+	?* Force blk_queue_enter() and blk_mq_queue_enter() to check the
+	?* "dying" flag.
+	?*/
+	blk_freeze_queue(q);
+
?	if (q->mq_ops)
?		blk_mq_wake_waiters(q);
?	else {
@@ -555,7 +561,7 @@ void blk_cleanup_queue(struct request_queue *q)
?	?* Drain all requests queued before DYING marking. Set DEAD flag to
?	?* prevent that q->request_fn() gets invoked after draining finished.
?	?*/
-	blk_freeze_queue(q);
+	WARN_ON_ONCE(!atomic_read(&q->mq_freeze_depth));
?	spin_lock_irq(lock);
?	if (!q->mq_ops)
?		__blk_drain_queue(q, true);

Thanks,

Bart.