When the inflight IOs are slow and no new IOs are issued, we expect
iostat could manifest the IO hang problem. However after
commit 5b18b5a73760 ("block: delete part_round_stats and switch to less
precise counting"), io_tick and time_in_queue will not be updated until
the end of IO, and the avgqu-sz and %util columns of iostat will be zero.
Because it has using stat.nsecs accumulation to express time_in_queue
which is not suitable to change, and may %util will express the status
better when io hang occur. To fix io_ticks, we use update_io_ticks and
inflight to update io_ticks when diskstats_show and part_stat_show
been called.
Fixes: 5b18b5a73760 ("block: delete part_round_stats and switch to less precise counting")
Signed-off-by: Zhang Wensheng <[email protected]>
---
block/genhd.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/block/genhd.c b/block/genhd.c
index 626c8406f21a..df0656ffb4ad 100644
--- a/block/genhd.c
+++ b/block/genhd.c
@@ -913,12 +913,14 @@ ssize_t part_stat_show(struct device *dev,
struct disk_stats stat;
unsigned int inflight;
- part_stat_read_all(bdev, &stat);
if (queue_is_mq(q))
inflight = blk_mq_in_flight(q, bdev);
else
inflight = part_in_flight(bdev);
+ if (inflight)
+ update_io_ticks(bdev, jiffies, true);
+ part_stat_read_all(bdev, &stat);
return sprintf(buf,
"%8lu %8lu %8llu %8u "
"%8lu %8lu %8llu %8u "
@@ -1174,12 +1176,14 @@ static int diskstats_show(struct seq_file *seqf, void *v)
xa_for_each(&gp->part_tbl, idx, hd) {
if (bdev_is_partition(hd) && !bdev_nr_sectors(hd))
continue;
- part_stat_read_all(hd, &stat);
if (queue_is_mq(gp->queue))
inflight = blk_mq_in_flight(gp->queue, hd);
else
inflight = part_in_flight(hd);
+ if (inflight)
+ update_io_ticks(hd, jiffies, true);
+ part_stat_read_all(hd, &stat);
seq_printf(seqf, "%4d %7d %pg "
"%lu %lu %lu %u "
"%lu %lu %lu %u "
--
2.31.1
friendly ping...
在 2022/1/28 14:22, zhangwensheng (E) 写道:
> friendly ping...
>
> 在 2022/1/25 17:19, Zhang Wensheng 写道:
>> When the inflight IOs are slow and no new IOs are issued, we expect
>> iostat could manifest the IO hang problem. However after
>> commit 5b18b5a73760 ("block: delete part_round_stats and switch to less
>> precise counting"), io_tick and time_in_queue will not be updated until
>> the end of IO, and the avgqu-sz and %util columns of iostat will be
>> zero.
>>
>> Because it has using stat.nsecs accumulation to express time_in_queue
>> which is not suitable to change, and may %util will express the status
>> better when io hang occur. To fix io_ticks, we use update_io_ticks and
>> inflight to update io_ticks when diskstats_show and part_stat_show
>> been called.
>>
>> Fixes: 5b18b5a73760 ("block: delete part_round_stats and switch to
>> less precise counting")
>> Signed-off-by: Zhang Wensheng <[email protected]>
>> ---
>> block/genhd.c | 8 ++++++--
>> 1 file changed, 6 insertions(+), 2 deletions(-)
>>
>> diff --git a/block/genhd.c b/block/genhd.c
>> index 626c8406f21a..df0656ffb4ad 100644
>> --- a/block/genhd.c
>> +++ b/block/genhd.c
>> @@ -913,12 +913,14 @@ ssize_t part_stat_show(struct device *dev,
>> struct disk_stats stat;
>> unsigned int inflight;
>> - part_stat_read_all(bdev, &stat);
>> if (queue_is_mq(q))
>> inflight = blk_mq_in_flight(q, bdev);
>> else
>> inflight = part_in_flight(bdev);
>> + if (inflight)
>> + update_io_ticks(bdev, jiffies, true);
>> + part_stat_read_all(bdev, &stat);
>> return sprintf(buf,
>> "%8lu %8lu %8llu %8u "
>> "%8lu %8lu %8llu %8u "
>> @@ -1174,12 +1176,14 @@ static int diskstats_show(struct seq_file
>> *seqf, void *v)
>> xa_for_each(&gp->part_tbl, idx, hd) {
>> if (bdev_is_partition(hd) && !bdev_nr_sectors(hd))
>> continue;
>> - part_stat_read_all(hd, &stat);
>> if (queue_is_mq(gp->queue))
>> inflight = blk_mq_in_flight(gp->queue, hd);
>> else
>> inflight = part_in_flight(hd);
>> + if (inflight)
>> + update_io_ticks(hd, jiffies, true);
>> + part_stat_read_all(hd, &stat);
>> seq_printf(seqf, "%4d %7d %pg "
>> "%lu %lu %lu %u "
>> "%lu %lu %lu %u "
friendly ping...
在 2022/2/8 10:12, zhangwensheng (E) 写道:
> friendly ping...
>
> 在 2022/1/28 14:22, zhangwensheng (E) 写道:
>> friendly ping...
>>
>> 在 2022/1/25 17:19, Zhang Wensheng 写道:
>>> When the inflight IOs are slow and no new IOs are issued, we expect
>>> iostat could manifest the IO hang problem. However after
>>> commit 5b18b5a73760 ("block: delete part_round_stats and switch to less
>>> precise counting"), io_tick and time_in_queue will not be updated until
>>> the end of IO, and the avgqu-sz and %util columns of iostat will be
>>> zero.
>>>
>>> Because it has using stat.nsecs accumulation to express time_in_queue
>>> which is not suitable to change, and may %util will express the status
>>> better when io hang occur. To fix io_ticks, we use update_io_ticks and
>>> inflight to update io_ticks when diskstats_show and part_stat_show
>>> been called.
>>>
>>> Fixes: 5b18b5a73760 ("block: delete part_round_stats and switch to
>>> less precise counting")
>>> Signed-off-by: Zhang Wensheng <[email protected]>
>>> ---
>>> block/genhd.c | 8 ++++++--
>>> 1 file changed, 6 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/block/genhd.c b/block/genhd.c
>>> index 626c8406f21a..df0656ffb4ad 100644
>>> --- a/block/genhd.c
>>> +++ b/block/genhd.c
>>> @@ -913,12 +913,14 @@ ssize_t part_stat_show(struct device *dev,
>>> struct disk_stats stat;
>>> unsigned int inflight;
>>> - part_stat_read_all(bdev, &stat);
>>> if (queue_is_mq(q))
>>> inflight = blk_mq_in_flight(q, bdev);
>>> else
>>> inflight = part_in_flight(bdev);
>>> + if (inflight)
>>> + update_io_ticks(bdev, jiffies, true);
>>> + part_stat_read_all(bdev, &stat);
>>> return sprintf(buf,
>>> "%8lu %8lu %8llu %8u "
>>> "%8lu %8lu %8llu %8u "
>>> @@ -1174,12 +1176,14 @@ static int diskstats_show(struct seq_file
>>> *seqf, void *v)
>>> xa_for_each(&gp->part_tbl, idx, hd) {
>>> if (bdev_is_partition(hd) && !bdev_nr_sectors(hd))
>>> continue;
>>> - part_stat_read_all(hd, &stat);
>>> if (queue_is_mq(gp->queue))
>>> inflight = blk_mq_in_flight(gp->queue, hd);
>>> else
>>> inflight = part_in_flight(hd);
>>> + if (inflight)
>>> + update_io_ticks(hd, jiffies, true);
>>> + part_stat_read_all(hd, &stat);
>>> seq_printf(seqf, "%4d %7d %pg "
>>> "%lu %lu %lu %u "
>>> "%lu %lu %lu %u "
On 2/16/22 8:07 PM, zhangwensheng (E) wrote:
> hi, Jens:
>
> I am sorry that there are some problems with the patch below:
>
>
> @@ -913,12 +913,14 @@ ssize_t part_stat_show(struct device *dev,
> struct disk_stats stat;
> unsigned int inflight;
> - part_stat_read_all(bdev, &stat);
> if (queue_is_mq(q))
> inflight = blk_mq_in_flight(q, bdev);
> else
> inflight = part_in_flight(bdev);
> + if (inflight)
> + update_io_ticks(bdev, jiffies, true);
> + part_stat_read_all(bdev, &stat);
> return sprintf(buf,
> "%8lu %8lu %8llu %8u "
> "%8lu %8lu %8llu %8u "
>
> It shoule use part_stat_lock() & part_stat_unlock() to protect update_io_ticks(),
>
> because it operates on per-cpu variables. And I will fix it in V2.
I'll drop it, please resend a fixed and tested one.
--
Jens Axboe
On Tue, 25 Jan 2022 17:19:38 +0800, Zhang Wensheng wrote:
> When the inflight IOs are slow and no new IOs are issued, we expect
> iostat could manifest the IO hang problem. However after
> commit 5b18b5a73760 ("block: delete part_round_stats and switch to less
> precise counting"), io_tick and time_in_queue will not be updated until
> the end of IO, and the avgqu-sz and %util columns of iostat will be zero.
>
> Because it has using stat.nsecs accumulation to express time_in_queue
> which is not suitable to change, and may %util will express the status
> better when io hang occur. To fix io_ticks, we use update_io_ticks and
> inflight to update io_ticks when diskstats_show and part_stat_show
> been called.
>
> [...]
Applied, thanks!
[1/1] block: update io_ticks when io hang
commit: 62847731488f59971413484005a7fb2772cb9249
Best regards,
--
Jens Axboe