2023-02-27 06:23:16

by Sanan Hasanov

[permalink] [raw]
Subject: INFO: task hung in io_ring_exit_work

Good day, dear maintainers,

We found a bug using a modified kernel configuration file used by syzbot.

We enhanced the coverage of the configuration file using our tool, klocalizer.

Kernel Branch: 6.2.0-next-20230225
Kernel config:?https://drive.google.com/file/d/1NS9N8rvftQ7BouImn2OVnC96qrNhAeuO/view?usp=share_link
C Reproducer:?Unfortunately, there is no reproducer for this bug yet.

Thank you!

Best regards,
Sanan Hasanov

INFO: task kworker/u16:20:839 blocked for more than 143 seconds.
? ? ? Not tainted 6.2.0-next-20230225 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/u16:20 ?state:D stack:25504 pid:839 ? ppid:2 ? ? ?flags:0x00004000
Workqueue: events_unbound io_ring_exit_work
Call Trace:
?<TASK>
?context_switch kernel/sched/core.c:5304 [inline]
?__schedule+0x24bc/0x5a60 kernel/sched/core.c:6622
?schedule+0xe7/0x1b0 kernel/sched/core.c:6698
?schedule_timeout+0x26e/0x2b0 kernel/time/timer.c:2143
?do_wait_for_common kernel/sched/completion.c:85 [inline]
?__wait_for_common+0x1ce/0x5d0 kernel/sched/completion.c:106
?io_ring_exit_work+0x543/0x13c0 io_uring/io_uring.c:3027
?process_one_work+0x9ba/0x1820 kernel/workqueue.c:2390
?worker_thread+0x669/0x1090 kernel/workqueue.c:2537
?kthread+0x2e8/0x3a0 kernel/kthread.c:376
?ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308
?</TASK>
INFO: task kworker/u16:24:842 blocked for more than 143 seconds.
? ? ? Not tainted 6.2.0-next-20230225 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/u16:24 ?state:D stack:25472 pid:842 ? ppid:2 ? ? ?flags:0x00004000
Workqueue: events_unbound io_ring_exit_work
Call Trace:
?<TASK>
?context_switch kernel/sched/core.c:5304 [inline]
?__schedule+0x24bc/0x5a60 kernel/sched/core.c:6622
?schedule+0xe7/0x1b0 kernel/sched/core.c:6698
?schedule_timeout+0x26e/0x2b0 kernel/time/timer.c:2143
?do_wait_for_common kernel/sched/completion.c:85 [inline]
?__wait_for_common+0x1ce/0x5d0 kernel/sched/completion.c:106
?io_ring_exit_work+0x543/0x13c0 io_uring/io_uring.c:3027
?process_one_work+0x9ba/0x1820 kernel/workqueue.c:2390
?worker_thread+0x669/0x1090 kernel/workqueue.c:2537
?kthread+0x2e8/0x3a0 kernel/kthread.c:376
?ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308
?</TASK>
INFO: task kworker/u16:27:845 blocked for more than 143 seconds.
? ? ? Not tainted 6.2.0-next-20230225 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/u16:27 ?state:D stack:25120 pid:845 ? ppid:2 ? ? ?flags:0x00004000
Workqueue: events_unbound io_ring_exit_work
Call Trace:
?<TASK>
?context_switch kernel/sched/core.c:5304 [inline]
?__schedule+0x24bc/0x5a60 kernel/sched/core.c:6622
?schedule+0xe7/0x1b0 kernel/sched/core.c:6698
?schedule_timeout+0x26e/0x2b0 kernel/time/timer.c:2143
?do_wait_for_common kernel/sched/completion.c:85 [inline]
?__wait_for_common+0x1ce/0x5d0 kernel/sched/completion.c:106
?io_ring_exit_work+0x543/0x13c0 io_uring/io_uring.c:3027
?process_one_work+0x9ba/0x1820 kernel/workqueue.c:2390
?worker_thread+0x669/0x1090 kernel/workqueue.c:2537
?kthread+0x2e8/0x3a0 kernel/kthread.c:376
?ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308
?</TASK>
INFO: task kworker/u16:57:870 blocked for more than 143 seconds.
? ? ? Not tainted 6.2.0-next-20230225 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/u16:57 ?state:D stack:25120 pid:870 ? ppid:2 ? ? ?flags:0x00004000
Workqueue: events_unbound io_ring_exit_work
Call Trace:
?<TASK>
?context_switch kernel/sched/core.c:5304 [inline]
?__schedule+0x24bc/0x5a60 kernel/sched/core.c:6622
?schedule+0xe7/0x1b0 kernel/sched/core.c:6698
?schedule_timeout+0x26e/0x2b0 kernel/time/timer.c:2143
?do_wait_for_common kernel/sched/completion.c:85 [inline]
?__wait_for_common+0x1ce/0x5d0 kernel/sched/completion.c:106
?io_ring_exit_work+0x543/0x13c0 io_uring/io_uring.c:3027
?process_one_work+0x9ba/0x1820 kernel/workqueue.c:2390
?worker_thread+0x669/0x1090 kernel/workqueue.c:2537
?kthread+0x2e8/0x3a0 kernel/kthread.c:376
?ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308
?</TASK>
INFO: task kworker/u16:58:871 blocked for more than 143 seconds.
? ? ? Not tainted 6.2.0-next-20230225 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/u16:58 ?state:D stack:24624 pid:871 ? ppid:2 ? ? ?flags:0x00004000
Workqueue: events_unbound io_ring_exit_work
Call Trace:
?<TASK>
?context_switch kernel/sched/core.c:5304 [inline]
?__schedule+0x24bc/0x5a60 kernel/sched/core.c:6622
?schedule+0xe7/0x1b0 kernel/sched/core.c:6698
?schedule_timeout+0x26e/0x2b0 kernel/time/timer.c:2143
?do_wait_for_common kernel/sched/completion.c:85 [inline]
?__wait_for_common+0x1ce/0x5d0 kernel/sched/completion.c:106
?io_ring_exit_work+0x543/0x13c0 io_uring/io_uring.c:3027
?process_one_work+0x9ba/0x1820 kernel/workqueue.c:2390
?worker_thread+0x669/0x1090 kernel/workqueue.c:2537
?kthread+0x2e8/0x3a0 kernel/kthread.c:376
?ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308
?</TASK>
INFO: task kworker/u16:60:874 blocked for more than 143 seconds.
? ? ? Not tainted 6.2.0-next-20230225 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/u16:60 ?state:D stack:24784 pid:874 ? ppid:2 ? ? ?flags:0x00004000
Workqueue: events_unbound io_ring_exit_work
Call Trace:
?<TASK>
?context_switch kernel/sched/core.c:5304 [inline]
?__schedule+0x24bc/0x5a60 kernel/sched/core.c:6622
?schedule+0xe7/0x1b0 kernel/sched/core.c:6698
?schedule_timeout+0x26e/0x2b0 kernel/time/timer.c:2143
?do_wait_for_common kernel/sched/completion.c:85 [inline]
?__wait_for_common+0x1ce/0x5d0 kernel/sched/completion.c:106
?io_ring_exit_work+0x543/0x13c0 io_uring/io_uring.c:3027
?process_one_work+0x9ba/0x1820 kernel/workqueue.c:2390
?worker_thread+0x669/0x1090 kernel/workqueue.c:2537
?kthread+0x2e8/0x3a0 kernel/kthread.c:376
?ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308
?</TASK>
INFO: task kworker/u16:62:875 blocked for more than 143 seconds.
? ? ? Not tainted 6.2.0-next-20230225 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/u16:62 ?state:D stack:24864 pid:875 ? ppid:2 ? ? ?flags:0x00004000
Workqueue: events_unbound io_ring_exit_work
Call Trace:
?<TASK>
?context_switch kernel/sched/core.c:5304 [inline]
?__schedule+0x24bc/0x5a60 kernel/sched/core.c:6622
?schedule+0xe7/0x1b0 kernel/sched/core.c:6698
?schedule_timeout+0x26e/0x2b0 kernel/time/timer.c:2143
?do_wait_for_common kernel/sched/completion.c:85 [inline]
?__wait_for_common+0x1ce/0x5d0 kernel/sched/completion.c:106
?io_ring_exit_work+0x543/0x13c0 io_uring/io_uring.c:3027
?process_one_work+0x9ba/0x1820 kernel/workqueue.c:2390
?worker_thread+0x669/0x1090 kernel/workqueue.c:2537
?kthread+0x2e8/0x3a0 kernel/kthread.c:376
?ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308
?</TASK>
INFO: task kworker/u16:63:876 blocked for more than 143 seconds.
? ? ? Not tainted 6.2.0-next-20230225 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/u16:63 ?state:D stack:24272 pid:876 ? ppid:2 ? ? ?flags:0x00004000
Workqueue: events_unbound io_ring_exit_work
Call Trace:
?<TASK>
?context_switch kernel/sched/core.c:5304 [inline]
?__schedule+0x24bc/0x5a60 kernel/sched/core.c:6622
?schedule+0xe7/0x1b0 kernel/sched/core.c:6698
?schedule_timeout+0x26e/0x2b0 kernel/time/timer.c:2143
?do_wait_for_common kernel/sched/completion.c:85 [inline]
?__wait_for_common+0x1ce/0x5d0 kernel/sched/completion.c:106
?io_ring_exit_work+0x543/0x13c0 io_uring/io_uring.c:3027
?process_one_work+0x9ba/0x1820 kernel/workqueue.c:2390
?worker_thread+0x669/0x1090 kernel/workqueue.c:2537
?kthread+0x2e8/0x3a0 kernel/kthread.c:376
?ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308
?</TASK>
INFO: task kworker/u16:64:877 blocked for more than 143 seconds.
? ? ? Not tainted 6.2.0-next-20230225 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/u16:64 ?state:D stack:25008 pid:877 ? ppid:2 ? ? ?flags:0x00004000
Workqueue: events_unbound io_ring_exit_work
Call Trace:
?<TASK>
?context_switch kernel/sched/core.c:5304 [inline]
?__schedule+0x24bc/0x5a60 kernel/sched/core.c:6622
?schedule+0xe7/0x1b0 kernel/sched/core.c:6698
?schedule_timeout+0x26e/0x2b0 kernel/time/timer.c:2143
?do_wait_for_common kernel/sched/completion.c:85 [inline]
?__wait_for_common+0x1ce/0x5d0 kernel/sched/completion.c:106
?io_ring_exit_work+0x543/0x13c0 io_uring/io_uring.c:3027
?process_one_work+0x9ba/0x1820 kernel/workqueue.c:2390
?worker_thread+0x669/0x1090 kernel/workqueue.c:2537
?kthread+0x2e8/0x3a0 kernel/kthread.c:376
?ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308
?</TASK>
INFO: task kworker/u16:0:28836 blocked for more than 143 seconds.
? ? ? Not tainted 6.2.0-next-20230225 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/u16:0 ? state:D stack:25472 pid:28836 ppid:2 ? ? ?flags:0x00004000
Workqueue: loop2 loop_rootcg_workfn
Call Trace:
?<TASK>
?context_switch kernel/sched/core.c:5304 [inline]
?__schedule+0x24bc/0x5a60 kernel/sched/core.c:6622
?schedule+0xe7/0x1b0 kernel/sched/core.c:6698
?io_schedule+0xbe/0x130 kernel/sched/core.c:8884
?folio_wait_bit_common+0x390/0x9b0 mm/filemap.c:1301
?__folio_lock mm/filemap.c:1664 [inline]
?folio_lock include/linux/pagemap.h:952 [inline]
?folio_lock include/linux/pagemap.h:948 [inline]
?__filemap_get_folio+0xb13/0xd20 mm/filemap.c:1936
?shmem_get_folio_gfp+0x41e/0x1960 mm/shmem.c:1880
?shmem_get_folio mm/shmem.c:2071 [inline]
?shmem_file_read_iter+0x569/0xa50 mm/shmem.c:2748
?call_read_iter include/linux/fs.h:1845 [inline]
?do_iter_readv_writev+0x2df/0x3b0 fs/read_write.c:733
?do_iter_read+0x2f2/0x750 fs/read_write.c:796
?vfs_iter_read+0x74/0xa0 fs/read_write.c:838
?lo_read_simple drivers/block/loop.c:290 [inline]
?do_req_filebacked drivers/block/loop.c:500 [inline]
?loop_handle_cmd drivers/block/loop.c:1879 [inline]
?loop_process_work+0x15a8/0x2130 drivers/block/loop.c:1914
?process_one_work+0x9ba/0x1820 kernel/workqueue.c:2390
?worker_thread+0x669/0x1090 kernel/workqueue.c:2537
?kthread+0x2e8/0x3a0 kernel/kthread.c:376
?ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308
?</TASK>
Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings

Showing all locks held in the system:
1 lock held by rcu_tasks_kthre/12:
?#0: ffffffff8bf915f0 (rcu_tasks.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x2e/0xd90 kernel/rcu/tasks.h:510
1 lock held by rcu_tasks_trace/13:
?#0: ffffffff8bf912f0 (rcu_tasks_trace.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x2e/0xd90 kernel/rcu/tasks.h:510
1 lock held by khungtaskd/59:
?#0: ffffffff8bf92140 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x55/0x340 kernel/locking/lockdep.c:6495
1 lock held by in:imklog/7366:
?#0: ffff88811488a168 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0xe7/0x100 fs/file.c:1046
1 lock held by syz-fuzzer/7263:
2 locks held by kworker/u16:20/839:
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x868/0x1820 kernel/workqueue.c:2361
?#1: ffffc90007487da8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x89c/0x1820 kernel/workqueue.c:2365
2 locks held by kworker/u16:24/842:
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x868/0x1820 kernel/workqueue.c:2361
?#1: ffffc900074b7da8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x89c/0x1820 kernel/workqueue.c:2365
2 locks held by kworker/u16:27/845:
?#0:
ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x868/0x1820 kernel/workqueue.c:2361
?#1: ffffc900074e7da8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x89c/0x1820 kernel/workqueue.c:2365
2 locks held by kworker/u16:57/870:
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x868/0x1820 kernel/workqueue.c:2361
?#1: ffffc900076bfda8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x89c/0x1820 kernel/workqueue.c:2365
2 locks held by kworker/u16:58/871:
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x868/0x1820 kernel/workqueue.c:2361
?#1: ffffc900076cfda8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x89c/0x1820 kernel/workqueue.c:2365
2 locks held by kworker/u16:60/874:
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x868/0x1820 kernel/workqueue.c:2361
?#1: ffffc9000775fda8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x89c/0x1820 kernel/workqueue.c:2365
2 locks held by kworker/u16:62/875:
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x868/0x1820 kernel/workqueue.c:2361
?#1: ffffc9000776fda8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x89c/0x1820 kernel/workqueue.c:2365
2 locks held by kworker/u16:63/876:
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x868/0x1820 kernel/workqueue.c:2361
?#1: ffffc9000777fda8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x89c/0x1820 kernel/workqueue.c:2365
2 locks held by kworker/u16:64/877:
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x868/0x1820 kernel/workqueue.c:2361
?#1: ffffc9000781fda8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x89c/0x1820 kernel/workqueue.c:2365
2 locks held by kworker/u16:0/28836:
?#0: ffff8880458e9938 ((wq_completion)loop2){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
?#0: ffff8880458e9938 ((wq_completion)loop2){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
?#0: ffff8880458e9938 ((wq_completion)loop2){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
?#0: ffff8880458e9938 ((wq_completion)loop2){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
?#0: ffff8880458e9938 ((wq_completion)loop2){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
?#0: ffff8880458e9938 ((wq_completion)loop2){+.+.}-{0:0}, at: process_one_work+0x868/0x1820 kernel/workqueue.c:2361
?#1: ffffc90009aa7da8 ((work_completion)(&lo->rootcg_work)){+.+.}-{0:0}, at: process_one_work+0x89c/0x1820 kernel/workqueue.c:2365
2 locks held by syz-executor.2/6115:
?#0: ffff8881128c00e0 (&type->s_umount_key#73/1){+.+.}-{3:3}, at: alloc_super+0x22e/0xb60 fs/super.c:228
?#1: ffff8881128b5090 (&nilfs->ns_sem){++++}-{3:3}, at: init_nilfs+0x7d/0x1300 fs/nilfs2/the_nilfs.c:630
2 locks held by kworker/u16:1/6137:
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x868/0x1820 kernel/workqueue.c:2361
?#1: ffffc90003a0fda8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x89c/0x1820 kernel/workqueue.c:2365
2 locks held by kworker/u16:2/6138:
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x868/0x1820 kernel/workqueue.c:2361
?#1: ffffc90003887da8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x89c/0x1820 kernel/workqueue.c:2365
2 locks held by kworker/u16:3/6139:
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x868/0x1820 kernel/workqueue.c:2361
?#1: ffffc90003a2fda8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x89c/0x1820 kernel/workqueue.c:2365
2 locks held by kworker/u16:4/6140:
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x868/0x1820 kernel/workqueue.c:2361
?#1: ffffc90003677da8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x89c/0x1820 kernel/workqueue.c:2365
2 locks held by kworker/u16:5/6141:
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x868/0x1820 kernel/workqueue.c:2361
?#1: ffffc90003847da8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x89c/0x1820 kernel/workqueue.c:2365
2 locks held by kworker/u16:6/6142:
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x868/0x1820 kernel/workqueue.c:2361
?#1: ffffc90003a3fda8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x89c/0x1820 kernel/workqueue.c:2365
2 locks held by kworker/u16:7/6143:
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x868/0x1820 kernel/workqueue.c:2361
?#1: ffffc90003a4fda8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x89c/0x1820 kernel/workqueue.c:2365
2 locks held by kworker/u16:8/6144:
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x868/0x1820 kernel/workqueue.c:2361
?#1: ffffc90003a5fda8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x89c/0x1820 kernel/workqueue.c:2365
2 locks held by kworker/u16:9/6145:
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x868/0x1820 kernel/workqueue.c:2361
?#1: ffffc90003a6fda8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x89c/0x1820 kernel/workqueue.c:2365
2 locks held by kworker/u16:10/6146:
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x868/0x1820 kernel/workqueue.c:2361
?#1: ffffc90003a7fda8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x89c/0x1820 kernel/workqueue.c:2365
2 locks held by kworker/u16:11/6147:
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:639 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:666 [inline]
?#0: ffff888100081138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x868/0x1820 kernel/workqueue.c:2361
?#1: ffffc90003a8fda8 ((work_completion)(&ctx->exit_work)){+.+.}-{0:0}, at: process_one_work+0x89c/0x1820 kernel/workqueue.c:2365

=============================================


2023-03-01 14:54:05

by Jens Axboe

[permalink] [raw]
Subject: Re: INFO: task hung in io_ring_exit_work

On 2/26/23 11:23 PM, Sanan Hasanov wrote:
> Good day, dear maintainers,
>
> We found a bug using a modified kernel configuration file used by syzbot.
>
> We enhanced the coverage of the configuration file using our tool, klocalizer.
>
> Kernel Branch: 6.2.0-next-20230225
> Kernel config: https://drive.google.com/file/d/1NS9N8rvftQ7BouImn2OVnC96qrNhAeuO/view?usp=share_link
> C Reproducer: Unfortunately, there is no reproducer for this bug yet.

Not sure what we're supposed to do with this, without a syzbot or
c reproducer? You can get into this condition if you SIGSTOP the
task so it can't complete its task_work, for example. But impossible
to glean anything from this report.

--
Jens Axboe