2022-01-26 20:58:18

by David Decotigny

[permalink] [raw]
Subject: [PATCH RFC v3 1/1] mtd_blkdevs: avoid soft lockups with some mtd/spi devices

From: David Decotigny <[email protected]>

With some spi devices, the heavy cpu usage due to polling the spi
registers may lead to netdev timeouts, RCU complaints, etc. This can
be acute in the absence of CONFIG_PREEMPT. This patch allows to give
enough breathing room to avoid those incorrectly detected netdev
timeouts for example.

Example splat on 5.10.92:
[ 828.399306] rcu: INFO: rcu_sched self-detected stall on CPU
...
[ 828.419245] Task dump for CPU 1:
[ 828.422465] task:kworker/1:1H state:R running task on cpu 1 stack: 0 pid: 76 ppid: 2 flags:0x0000002a
[ 828.433132] Workqueue: kblockd blk_mq_run_work_fn
[ 828.437820] Call trace:
...
[ 828.512267] spi_mem_exec_op+0x4d0/0xde0
[ 828.516184] spi_mem_dirmap_read+0x180/0x39c
[ 828.520443] spi_nor_read_data+0x428/0x7e8
[ 828.524523] spi_nor_read+0x154/0x214
[ 828.528172] mtd_read_oob+0x440/0x714
[ 828.531815] mtd_read+0xac/0x120
[ 828.535030] mtdblock_readsect+0x178/0x230
[ 828.539102] mtd_blktrans_work+0x9fc/0xf28
[ 828.543177] mtd_queue_rq+0x1ac/0x2e4
[ 828.546827] blk_mq_dispatch_rq_list+0x2cc/0xa44
[ 828.551419] blk_mq_do_dispatch_sched+0xb0/0x7cc
[ 828.556010] __blk_mq_sched_dispatch_requests+0x350/0x494
[ 828.561372] blk_mq_sched_dispatch_requests+0xac/0xe4
[ 828.566387] __blk_mq_run_hw_queue+0x130/0x254
[ 828.570806] blk_mq_run_work_fn+0x50/0x60
[ 828.574814] process_one_work+0x578/0xf1c
[ 828.578814] worker_thread+0x5dc/0xea0
[ 828.582547] kthread+0x270/0x2d4
[ 828.585765] ret_from_fork+0x10/0x30


Signed-off-by: David Decotigny <[email protected]>
---
drivers/mtd/mtd_blkdevs.c | 1 +
1 file changed, 1 insertion(+)

diff --git a/drivers/mtd/mtd_blkdevs.c b/drivers/mtd/mtd_blkdevs.c
index 243f28a3206b4..64d2b093f114b 100644
--- a/drivers/mtd/mtd_blkdevs.c
+++ b/drivers/mtd/mtd_blkdevs.c
@@ -158,6 +158,7 @@ static void mtd_blktrans_work(struct mtd_blktrans_dev *dev)
}

background_done = 0;
+ cond_resched();
spin_lock_irq(&dev->queue_lock);
}
}
--
2.35.0.rc0.227.g00780c9af4-goog


2022-01-26 20:59:33

by Richard Weinberger

[permalink] [raw]
Subject: Re: [PATCH RFC v3 1/1] mtd_blkdevs: avoid soft lockups with some mtd/spi devices

----- Ursprüngliche Mail -----
> Von: "David Decotigny" <[email protected]>
> An: "linux-kernel" <[email protected]>
> CC: "Miquel Raynal" <[email protected]>, "richard" <[email protected]>, "Vignesh Raghavendra" <[email protected]>,
> "linux-mtd" <[email protected]>, "David Decotigny" <[email protected]>
> Gesendet: Mittwoch, 26. Januar 2022 11:11:20
> Betreff: [PATCH RFC v3 1/1] mtd_blkdevs: avoid soft lockups with some mtd/spi devices

> From: David Decotigny <[email protected]>
>
> With some spi devices, the heavy cpu usage due to polling the spi
> registers may lead to netdev timeouts, RCU complaints, etc. This can
> be acute in the absence of CONFIG_PREEMPT. This patch allows to give
> enough breathing room to avoid those incorrectly detected netdev
> timeouts for example.
>
> Example splat on 5.10.92:
> [ 828.399306] rcu: INFO: rcu_sched self-detected stall on CPU
> ...
> [ 828.419245] Task dump for CPU 1:
> [ 828.422465] task:kworker/1:1H state:R running task on cpu 1 stack:
> 0 pid: 76 ppid: 2 flags:0x0000002a
> [ 828.433132] Workqueue: kblockd blk_mq_run_work_fn
> [ 828.437820] Call trace:
> ...
> [ 828.512267] spi_mem_exec_op+0x4d0/0xde0
> [ 828.516184] spi_mem_dirmap_read+0x180/0x39c
> [ 828.520443] spi_nor_read_data+0x428/0x7e8
> [ 828.524523] spi_nor_read+0x154/0x214
> [ 828.528172] mtd_read_oob+0x440/0x714
> [ 828.531815] mtd_read+0xac/0x120
> [ 828.535030] mtdblock_readsect+0x178/0x230
> [ 828.539102] mtd_blktrans_work+0x9fc/0xf28
> [ 828.543177] mtd_queue_rq+0x1ac/0x2e4
> [ 828.546827] blk_mq_dispatch_rq_list+0x2cc/0xa44
> [ 828.551419] blk_mq_do_dispatch_sched+0xb0/0x7cc
> [ 828.556010] __blk_mq_sched_dispatch_requests+0x350/0x494
> [ 828.561372] blk_mq_sched_dispatch_requests+0xac/0xe4
> [ 828.566387] __blk_mq_run_hw_queue+0x130/0x254
> [ 828.570806] blk_mq_run_work_fn+0x50/0x60
> [ 828.574814] process_one_work+0x578/0xf1c
> [ 828.578814] worker_thread+0x5dc/0xea0
> [ 828.582547] kthread+0x270/0x2d4
> [ 828.585765] ret_from_fork+0x10/0x30
>
>
> Signed-off-by: David Decotigny <[email protected]>

Reviewed-by: Richard Weinberger <[email protected]>

Thanks,
//richard

2022-01-26 21:06:42

by Miquel Raynal

[permalink] [raw]
Subject: Re: [PATCH RFC v3 1/1] mtd_blkdevs: avoid soft lockups with some mtd/spi devices

On Wed, 2022-01-26 at 10:11:20 UTC, David Decotigny wrote:
> From: David Decotigny <[email protected]>
>
> With some spi devices, the heavy cpu usage due to polling the spi
> registers may lead to netdev timeouts, RCU complaints, etc. This can
> be acute in the absence of CONFIG_PREEMPT. This patch allows to give
> enough breathing room to avoid those incorrectly detected netdev
> timeouts for example.
>
> Example splat on 5.10.92:
> [ 828.399306] rcu: INFO: rcu_sched self-detected stall on CPU
> ...
> [ 828.419245] Task dump for CPU 1:
> [ 828.422465] task:kworker/1:1H state:R running task on cpu 1 stack: 0 pid: 76 ppid: 2 flags:0x0000002a
> [ 828.433132] Workqueue: kblockd blk_mq_run_work_fn
> [ 828.437820] Call trace:
> ...
> [ 828.512267] spi_mem_exec_op+0x4d0/0xde0
> [ 828.516184] spi_mem_dirmap_read+0x180/0x39c
> [ 828.520443] spi_nor_read_data+0x428/0x7e8
> [ 828.524523] spi_nor_read+0x154/0x214
> [ 828.528172] mtd_read_oob+0x440/0x714
> [ 828.531815] mtd_read+0xac/0x120
> [ 828.535030] mtdblock_readsect+0x178/0x230
> [ 828.539102] mtd_blktrans_work+0x9fc/0xf28
> [ 828.543177] mtd_queue_rq+0x1ac/0x2e4
> [ 828.546827] blk_mq_dispatch_rq_list+0x2cc/0xa44
> [ 828.551419] blk_mq_do_dispatch_sched+0xb0/0x7cc
> [ 828.556010] __blk_mq_sched_dispatch_requests+0x350/0x494
> [ 828.561372] blk_mq_sched_dispatch_requests+0xac/0xe4
> [ 828.566387] __blk_mq_run_hw_queue+0x130/0x254
> [ 828.570806] blk_mq_run_work_fn+0x50/0x60
> [ 828.574814] process_one_work+0x578/0xf1c
> [ 828.578814] worker_thread+0x5dc/0xea0
> [ 828.582547] kthread+0x270/0x2d4
> [ 828.585765] ret_from_fork+0x10/0x30
>
>
> Signed-off-by: David Decotigny <[email protected]>
> Reviewed-by: Richard Weinberger <[email protected]>

Applied to https://git.kernel.org/pub/scm/linux/kernel/git/mtd/linux.git mtd/next, thanks.

Miquel