Subject: Re: Write I/O queue hangup at random on recent Linus' kernels
To:     "Kenneth R. Crudup" <kenny@panix.com>
Cc:     linux-bcache@vger.kernel.org, linux-block@vger.kernel.org,
        linux-ext4@vger.kernel.org, linux-nvme@lists.infradead.org
References: <b3ba57a7-d363-9c17-c4be-9dbe86875@panix.com>
 <b9c2681f-e63a-4d3b-913d-d8a75e2c2ea0@kernel.dk>
 <be6a783-97db-c3bf-b16f-e8c62b14755d@panix.com>
From:   Jens Axboe <axboe@kernel.dk>
Message-ID: <17206ea6-506d-b1de-09e8-c935ff308bd6@kernel.dk>
Date:   Thu, 25 Nov 2021 14:07:29 -0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.10.0
MIME-Version: 1.0
In-Reply-To: <be6a783-97db-c3bf-b16f-e8c62b14755d@panix.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 11/25/21 2:05 PM, Kenneth R. Crudup wrote:
> 
> On Tue, 23 Nov 2021, Jens Axboe wrote:
> 
>> It looks like some missed accounting. You can just disable wbt for now, would
>> be a useful data point to see if that fixes it. Just do:
> 
>> echo 0 > /sys/block/nvme0n1/queue/wbt_lat_usec
> 
>> and that will disable writeback throttling on that device.
> 
> It's been about 48 hours and haven't seen the issue since doing this.

Great, thanks for verifying. From your report 5.16-rc2 has the issue, is
5.15 fine?

-- 
Jens Axboe