Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Sat, 20 Aug 2022 15:00:39 +0800
From:   Ming Lei <ming.lei@redhat.com>
To:     Chris Murphy <lists@colorremedies.com>
Cc:     Nikolay Borisov <nborisov@suse.com>, Jens Axboe <axboe@kernel.dk>,
        Jan Kara <jack@suse.cz>,
        Paolo Valente <paolo.valente@linaro.org>,
        Btrfs BTRFS <linux-btrfs@vger.kernel.org>,
        Linux-RAID <linux-raid@vger.kernel.org>,
        linux-block <linux-block@vger.kernel.org>,
        linux-kernel <linux-kernel@vger.kernel.org>,
        Josef Bacik <josef@toxicpanda.com>
Subject: Re: stalling IO regression since linux 5.12, through 5.18
Message-ID: <YwCGlyDMhWubqKoL@T590>
References: <Yv0KmT8UYos2/4SX@T590>
 <35f0d608-7448-4276-8922-19a23d8f9049@www.fastmail.com>
 <Yv2P0zyoVvz35w/m@T590>
 <568465de-5c3b-4d94-a74b-5b83ce2f942f@www.fastmail.com>
 <Yv2w+Tuhw1RAoXI5@T590>
 <9f2f608a-cd5f-4736-9e6d-07ccc2eca12c@www.fastmail.com>
 <a817431f-276f-4aab-9ff8-c3e397494339@www.fastmail.com>
 <5426d0f9-6539-477d-8feb-2b49136b960f@www.fastmail.com>
 <Yv3NIQlDL0T3lstU@T590>
 <0f731b0a-fbd5-4e7b-a3df-0ed63360c1e0@www.fastmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <0f731b0a-fbd5-4e7b-a3df-0ed63360c1e0@www.fastmail.com>
Precedence: bulk

On Fri, Aug 19, 2022 at 03:20:25PM -0400, Chris Murphy wrote:
> 
> 
> On Thu, Aug 18, 2022, at 1:24 AM, Ming Lei wrote:
> > On Thu, Aug 18, 2022 at 12:27:04AM -0400, Chris Murphy wrote:
> >> 
> >> 
> >> On Thu, Aug 18, 2022, at 12:18 AM, Chris Murphy wrote:
> >> > On Thu, Aug 18, 2022, at 12:12 AM, Chris Murphy wrote:
> >> >> On Wed, Aug 17, 2022, at 11:41 PM, Ming Lei wrote:
> >> >>
> >> >>> OK, can you post the blk-mq debugfs log after you trigger it on v5.17?
> >> 
> >> Same boot, 3rd log. But the load is above 300 so I kinda need to sysrq+b soon.
> >> 
> >> https://drive.google.com/file/d/1375H558kqPTdng439rvG6LuXXWPXLToo/view?usp=sharing
> >> 
> >
> > Also please test the following one too:
> >
> >
> > diff --git a/block/blk-mq.c b/block/blk-mq.c
> > index 5ee62b95f3e5..d01c64be08e2 100644
> > --- a/block/blk-mq.c
> > +++ b/block/blk-mq.c
> > @@ -1991,7 +1991,8 @@ bool blk_mq_dispatch_rq_list(struct blk_mq_hw_ctx 
> > *hctx, struct list_head *list,
> >  		if (!needs_restart ||
> >  		    (no_tag && list_empty_careful(&hctx->dispatch_wait.entry)))
> >  			blk_mq_run_hw_queue(hctx, true);
> > -		else if (needs_restart && needs_resource)
> > +		else if (needs_restart && (needs_resource ||
> > +					blk_mq_is_shared_tags(hctx->flags)))
> >  			blk_mq_delay_run_hw_queue(hctx, BLK_MQ_RESOURCE_DELAY);
> > 
> >  		blk_mq_update_dispatch_busy(hctx, true);
> >
> 
> 
> With just this patch on top of 5.17.0, it still hangs. I've captured block debugfs log:
> https://drive.google.com/file/d/1ic4YHxoL9RrCdy_5FNdGfh_q_J3d_Ft0/view?usp=sharing

The log is similar with before, and the only difference is RESTART not
set.

Also follows another patch merged to v5.18 and it fixes io stall too, feel free to test it:

8f5fea65b06d blk-mq: avoid extending delays of active hctx from blk_mq_delay_run_hw_queues


Thanks, 
Ming