Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Subject: Re: stalling IO regression since linux 5.12, through 5.18
To:     Ming Lei <ming.lei@redhat.com>,
        Chris Murphy <lists@colorremedies.com>, Jan Kara <jack@suse.cz>
Cc:     Nikolay Borisov <nborisov@suse.com>, Jens Axboe <axboe@kernel.dk>,
        Paolo Valente <paolo.valente@linaro.org>,
        Btrfs BTRFS <linux-btrfs@vger.kernel.org>,
        Linux-RAID <linux-raid@vger.kernel.org>,
        linux-block <linux-block@vger.kernel.org>,
        linux-kernel <linux-kernel@vger.kernel.org>,
        Josef Bacik <josef@toxicpanda.com>,
        "yukuai (C)" <yukuai3@huawei.com>
References: <Yv0KmT8UYos2/4SX@T590>
 <35f0d608-7448-4276-8922-19a23d8f9049@www.fastmail.com>
 <Yv2P0zyoVvz35w/m@T590>
 <568465de-5c3b-4d94-a74b-5b83ce2f942f@www.fastmail.com>
 <Yv2w+Tuhw1RAoXI5@T590>
 <9f2f608a-cd5f-4736-9e6d-07ccc2eca12c@www.fastmail.com>
 <a817431f-276f-4aab-9ff8-c3e397494339@www.fastmail.com>
 <5426d0f9-6539-477d-8feb-2b49136b960f@www.fastmail.com>
 <Yv3NIQlDL0T3lstU@T590>
 <0f731b0a-fbd5-4e7b-a3df-0ed63360c1e0@www.fastmail.com>
 <YwCGlyDMhWubqKoL@T590>
From:   Yu Kuai <yukuai1@huaweicloud.com>
Message-ID: <297cbb87-87aa-2e1d-1fc3-8e96c241f28f@huaweicloud.com>
Date:   Thu, 1 Sep 2022 15:02:03 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101
 Thunderbird/60.8.0
MIME-Version: 1.0
In-Reply-To: <YwCGlyDMhWubqKoL@T590>
Content-Type: text/plain; charset=gbk; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk

Hi, Chris

?? 2022/08/20 15:00, Ming Lei д??:
> On Fri, Aug 19, 2022 at 03:20:25PM -0400, Chris Murphy wrote:
>>
>>
>> On Thu, Aug 18, 2022, at 1:24 AM, Ming Lei wrote:
>>> On Thu, Aug 18, 2022 at 12:27:04AM -0400, Chris Murphy wrote:
>>>>
>>>>
>>>> On Thu, Aug 18, 2022, at 12:18 AM, Chris Murphy wrote:
>>>>> On Thu, Aug 18, 2022, at 12:12 AM, Chris Murphy wrote:
>>>>>> On Wed, Aug 17, 2022, at 11:41 PM, Ming Lei wrote:
>>>>>>
>>>>>>> OK, can you post the blk-mq debugfs log after you trigger it on v5.17?
>>>>
>>>> Same boot, 3rd log. But the load is above 300 so I kinda need to sysrq+b soon.
>>>>
>>>> https://drive.google.com/file/d/1375H558kqPTdng439rvG6LuXXWPXLToo/view?usp=sharing
>>>>
>>>
>>> Also please test the following one too:
>>>
>>>
>>> diff --git a/block/blk-mq.c b/block/blk-mq.c
>>> index 5ee62b95f3e5..d01c64be08e2 100644
>>> --- a/block/blk-mq.c
>>> +++ b/block/blk-mq.c
>>> @@ -1991,7 +1991,8 @@ bool blk_mq_dispatch_rq_list(struct blk_mq_hw_ctx
>>> *hctx, struct list_head *list,
>>>   		if (!needs_restart ||
>>>   		    (no_tag && list_empty_careful(&hctx->dispatch_wait.entry)))
>>>   			blk_mq_run_hw_queue(hctx, true);
>>> -		else if (needs_restart && needs_resource)
>>> +		else if (needs_restart && (needs_resource ||
>>> +					blk_mq_is_shared_tags(hctx->flags)))
>>>   			blk_mq_delay_run_hw_queue(hctx, BLK_MQ_RESOURCE_DELAY);
>>>
>>>   		blk_mq_update_dispatch_busy(hctx, true);
>>>
>>
>>
>> With just this patch on top of 5.17.0, it still hangs. I've captured block debugfs log:
>> https://drive.google.com/file/d/1ic4YHxoL9RrCdy_5FNdGfh_q_J3d_Ft0/view?usp=sharing
> 
> The log is similar with before, and the only difference is RESTART not
> set.
> 
> Also follows another patch merged to v5.18 and it fixes io stall too, feel free to test it:
> 
> 8f5fea65b06d blk-mq: avoid extending delays of active hctx from blk_mq_delay_run_hw_queues

Have you tried this patch?

We meet a similar problem in our test, and I'm pretty sure about the
situation at the scene,

Our test environment??nvme with bfq ioscheduler,

How io is stalled:

1. hctx1 dispatch rq from bfq in service queue, bfqq becomes empty,
dispatch somehow fails and rq is inserted to hctx1->dispatch, new run
work is queued.

2. other hctx tries to dispatch rq, however, in service bfqq is
empty, bfq_dispatch_request return NULL, thus
blk_mq_delay_run_hw_queues is called.

3. for the problem described in above patch??run work from "hctx1"
can be stalled.

Above patch should fix this io stall, however, it seems to me bfq do
have some problems that in service bfqq doesn't expire under following
situation:

1. dispatched rqs don't complete
2. no new rq is issued to bfq

Thanks,
Kuai
> 
> 
> 
> Thanks,
> Ming
> 
> .
>