Dear Linux kernel maintainers,
Syzkaller reports this previously unknown bug on Linux
6.8.0-rc3-00043-ga69d20885494-dirty #4. Seems like the bug was
silently or unintendedly fixed in the latest version.
```
Syzkaller hit 'WARNING in io_fill_cqe_req_aux' bug.
------------[ cut here ]------------
WARNING: CPU: 7 PID: 8369 at io_uring/io_uring.h:132
io_lockdep_assert_cq_locked+0x2c7/0x340 io_uring/io_uring.h:132
Modules linked in:
CPU: 7 PID: 8369 Comm: syz-executor263 Not tainted
6.8.0-rc3-00043-ga69d20885494-dirty #4
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
RIP: 0010:io_lockdep_assert_cq_locked+0x2c7/0x340 io_uring/io_uring.h:132
Code: 48 8d bb 98 03 00 00 be ff ff ff ff e8 52 45 4b 06 31 ff 89 c3
89 c6 e8 b7 e2 2d fd 85 db 0f 85 d5 fe ff ff e8 0a e7 2d fd 90 <0f> 0b
90 e9 c7 fe ff ff e8 fc e6 2d fd e8 c7 38 fa fc 48 85 c0 0f
RSP: 0018:ffffc90012af79a8 EFLAGS: 00010293
RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff845cf059
RDX: ffff8880252ea440 RSI: ffffffff845cf066 RDI: 0000000000000005
RBP: 0000000000000000 R08: 0000000000000005 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001
FS: 00005555570e13c0(0000) GS:ffff88823bd80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1bdbcae020 CR3: 0000000022624000 CR4: 0000000000750ef0
PKRU: 55555554
Call Trace:
<TASK>
io_fill_cqe_req_aux+0xd6/0x1f0 io_uring/io_uring.c:925
io_poll_check_events io_uring/poll.c:325 [inline]
io_poll_task_func+0x16f/0x1000 io_uring/poll.c:357
io_handle_tw_list+0x172/0x560 io_uring/io_uring.c:1154
tctx_task_work_run+0xaa/0x330 io_uring/io_uring.c:1226
tctx_task_work+0x7b/0xd0 io_uring/io_uring.c:1244
task_work_run+0x16d/0x260 kernel/task_work.c:180
get_signal+0x1cb/0x25a0 kernel/signal.c:2669
arch_do_signal_or_restart+0x81/0x7e0 arch/x86/kernel/signal.c:310
exit_to_user_mode_loop kernel/entry/common.c:105 [inline]
exit_to_user_mode_prepare include/linux/entry-common.h:328 [inline]
__syscall_exit_to_user_mode_work kernel/entry/common.c:201 [inline]
syscall_exit_to_user_mode+0x156/0x2b0 kernel/entry/common.c:212
do_syscall_64+0xe5/0x270 arch/x86/entry/common.c:89
entry_SYSCALL_64_after_hwframe+0x6f/0x77
RIP: 0033:0x7f1bdbc2d88d
Code: c3 e8 a7 1f 00 00 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 89 f8 48
89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d
01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007ffd12f6fa18 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
RAX: 0000000000000001 RBX: 000000000000220b RCX: 00007f1bdbc2d88d
RDX: 0000000000000000 RSI: 0000000000005012 RDI: 0000000000000003
RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000001
R13: 431bde82d7b634db R14: 00007f1bdbcaa4f0 R15: 0000000000000001
</TASK>
Syzkaller reproducer:
# {Threaded:false Repeat:true RepeatTimes:0 Procs:1 Slowdown:1
Sandbox: SandboxArg:0 Leak:false NetInjection:false NetDevices:false
NetReset:false Cgroups:false BinfmtMisc:false CloseFDs:false
KCSAN:false DevlinkPCI:false NicVF:false USB:false VhciInjection:false
Wifi:false IEEE802154:false Sysctl:false Swap:false UseTmpDir:false
HandleSegv:false Repro:false Trace:false LegacyOptions:{Collide:false
Fault:false FaultCall:0 FaultNth:0}}
r0 = syz_io_uring_setup(0x220b, &(0x7f0000000000)={0x0, 0x63db,
0x10000, 0x800}, &(0x7f0000000080)=<r1=>0x0,
&(0x7f0000000200)=<r2=>0x0)
r3 = socket$inet(0x2, 0x1, 0x0)
syz_io_uring_submit(r1, r2,
&(0x7f0000000a80)=@IORING_OP_POLL_ADD={0x6, 0x0, 0x0, @fd=r3, 0x0,
0x0, 0x1})
io_uring_enter(r0, 0x5012, 0x0, 0x0, 0x0, 0x0)
```
crepro is in the attachment.
Best Regards
Xdchase
On 6/7/24 18:07, chase xd wrote:
> Dear Linux kernel maintainers,
>
> Syzkaller reports this previously unknown bug on Linux
> 6.8.0-rc3-00043-ga69d20885494-dirty #4. Seems like the bug was
> silently or unintendedly fixed in the latest version.
That branch you're using is confusing, apart from being
dirty and rc3, apparently it has never been merged. The
patch the test fails on looks different upstream:
commit 902ce82c2aa130bea5e3feca2d4ae62781865da7
Author: Pavel Begunkov <[email protected]>
Date: Mon Mar 18 22:00:32 2024 +0000
io_uring: get rid of intermediate aux cqe caches
It reproduces with your version but not with anything
upstream
> ```
> Syzkaller hit 'WARNING in io_fill_cqe_req_aux' bug.
>
> ------------[ cut here ]------------
> WARNING: CPU: 7 PID: 8369 at io_uring/io_uring.h:132
> io_lockdep_assert_cq_locked+0x2c7/0x340 io_uring/io_uring.h:132
> Modules linked in:
> CPU: 7 PID: 8369 Comm: syz-executor263 Not tainted
> 6.8.0-rc3-00043-ga69d20885494-dirty #4
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
> RIP: 0010:io_lockdep_assert_cq_locked+0x2c7/0x340 io_uring/io_uring.h:132
> Code: 48 8d bb 98 03 00 00 be ff ff ff ff e8 52 45 4b 06 31 ff 89 c3
> 89 c6 e8 b7 e2 2d fd 85 db 0f 85 d5 fe ff ff e8 0a e7 2d fd 90 <0f> 0b
> 90 e9 c7 fe ff ff e8 fc e6 2d fd e8 c7 38 fa fc 48 85 c0 0f
> RSP: 0018:ffffc90012af79a8 EFLAGS: 00010293
> RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff845cf059
> RDX: ffff8880252ea440 RSI: ffffffff845cf066 RDI: 0000000000000005
> RBP: 0000000000000000 R08: 0000000000000005 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
> R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001
> FS: 00005555570e13c0(0000) GS:ffff88823bd80000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007f1bdbcae020 CR3: 0000000022624000 CR4: 0000000000750ef0
> PKRU: 55555554
> Call Trace:
> <TASK>
> io_fill_cqe_req_aux+0xd6/0x1f0 io_uring/io_uring.c:925
> io_poll_check_events io_uring/poll.c:325 [inline]
> io_poll_task_func+0x16f/0x1000 io_uring/poll.c:357
> io_handle_tw_list+0x172/0x560 io_uring/io_uring.c:1154
> tctx_task_work_run+0xaa/0x330 io_uring/io_uring.c:1226
> tctx_task_work+0x7b/0xd0 io_uring/io_uring.c:1244
> task_work_run+0x16d/0x260 kernel/task_work.c:180
> get_signal+0x1cb/0x25a0 kernel/signal.c:2669
> arch_do_signal_or_restart+0x81/0x7e0 arch/x86/kernel/signal.c:310
> exit_to_user_mode_loop kernel/entry/common.c:105 [inline]
> exit_to_user_mode_prepare include/linux/entry-common.h:328 [inline]
> __syscall_exit_to_user_mode_work kernel/entry/common.c:201 [inline]
> syscall_exit_to_user_mode+0x156/0x2b0 kernel/entry/common.c:212
> do_syscall_64+0xe5/0x270 arch/x86/entry/common.c:89
> entry_SYSCALL_64_after_hwframe+0x6f/0x77
> RIP: 0033:0x7f1bdbc2d88d
> Code: c3 e8 a7 1f 00 00 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 89 f8 48
> 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d
> 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
> RSP: 002b:00007ffd12f6fa18 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
> RAX: 0000000000000001 RBX: 000000000000220b RCX: 00007f1bdbc2d88d
> RDX: 0000000000000000 RSI: 0000000000005012 RDI: 0000000000000003
> RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000001
> R13: 431bde82d7b634db R14: 00007f1bdbcaa4f0 R15: 0000000000000001
> </TASK>
>
>
> Syzkaller reproducer:
> # {Threaded:false Repeat:true RepeatTimes:0 Procs:1 Slowdown:1
> Sandbox: SandboxArg:0 Leak:false NetInjection:false NetDevices:false
> NetReset:false Cgroups:false BinfmtMisc:false CloseFDs:false
> KCSAN:false DevlinkPCI:false NicVF:false USB:false VhciInjection:false
> Wifi:false IEEE802154:false Sysctl:false Swap:false UseTmpDir:false
> HandleSegv:false Repro:false Trace:false LegacyOptions:{Collide:false
> Fault:false FaultCall:0 FaultNth:0}}
> r0 = syz_io_uring_setup(0x220b, &(0x7f0000000000)={0x0, 0x63db,
> 0x10000, 0x800}, &(0x7f0000000080)=<r1=>0x0,
> &(0x7f0000000200)=<r2=>0x0)
> r3 = socket$inet(0x2, 0x1, 0x0)
> syz_io_uring_submit(r1, r2,
> &(0x7f0000000a80)=@IORING_OP_POLL_ADD={0x6, 0x0, 0x0, @fd=r3, 0x0,
> 0x0, 0x1})
> io_uring_enter(r0, 0x5012, 0x0, 0x0, 0x0, 0x0)
> ```
>
> crepro is in the attachment.
>
> Best Regards
> Xdchase
--
Pavel Begunkov
Sorry now I'm also a bit confused by the branch choosing. I checked
out branch "for-6.9/io_uring" and started testing on that branch. I
assume that was the latest version of io_uring at that time, even now
I check out that branch and the bug still exists. How should I know
whether the branch will be merged, and which branch do you think I
should test on? Thanks.
Pavel Begunkov <[email protected]> 于2024年6月12日周三 03:11写道：
>
> On 6/7/24 18:07, chase xd wrote:
> > Dear Linux kernel maintainers,
> >
> > Syzkaller reports this previously unknown bug on Linux
> > 6.8.0-rc3-00043-ga69d20885494-dirty #4. Seems like the bug was
> > silently or unintendedly fixed in the latest version.
>
> That branch you're using is confusing, apart from being
> dirty and rc3, apparently it has never been merged. The
> patch the test fails on looks different upstream:
>
>
> commit 902ce82c2aa130bea5e3feca2d4ae62781865da7
> Author: Pavel Begunkov <[email protected]>
> Date: Mon Mar 18 22:00:32 2024 +0000
>
> io_uring: get rid of intermediate aux cqe caches
>
>
> It reproduces with your version but not with anything
> upstream
>
>
> > ```
> > Syzkaller hit 'WARNING in io_fill_cqe_req_aux' bug.
> >
> > ------------[ cut here ]------------
> > WARNING: CPU: 7 PID: 8369 at io_uring/io_uring.h:132
> > io_lockdep_assert_cq_locked+0x2c7/0x340 io_uring/io_uring.h:132
> > Modules linked in:
> > CPU: 7 PID: 8369 Comm: syz-executor263 Not tainted
> > 6.8.0-rc3-00043-ga69d20885494-dirty #4
> > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
> > RIP: 0010:io_lockdep_assert_cq_locked+0x2c7/0x340 io_uring/io_uring.h:132
> > Code: 48 8d bb 98 03 00 00 be ff ff ff ff e8 52 45 4b 06 31 ff 89 c3
> > 89 c6 e8 b7 e2 2d fd 85 db 0f 85 d5 fe ff ff e8 0a e7 2d fd 90 <0f> 0b
> > 90 e9 c7 fe ff ff e8 fc e6 2d fd e8 c7 38 fa fc 48 85 c0 0f
> > RSP: 0018:ffffc90012af79a8 EFLAGS: 00010293
> > RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff845cf059
> > RDX: ffff8880252ea440 RSI: ffffffff845cf066 RDI: 0000000000000005
> > RBP: 0000000000000000 R08: 0000000000000005 R09: 0000000000000000
> > R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
> > R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001
> > FS: 00005555570e13c0(0000) GS:ffff88823bd80000(0000) knlGS:0000000000000000
> > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 00007f1bdbcae020 CR3: 0000000022624000 CR4: 0000000000750ef0
> > PKRU: 55555554
> > Call Trace:
> > <TASK>
> > io_fill_cqe_req_aux+0xd6/0x1f0 io_uring/io_uring.c:925
> > io_poll_check_events io_uring/poll.c:325 [inline]
> > io_poll_task_func+0x16f/0x1000 io_uring/poll.c:357
> > io_handle_tw_list+0x172/0x560 io_uring/io_uring.c:1154
> > tctx_task_work_run+0xaa/0x330 io_uring/io_uring.c:1226
> > tctx_task_work+0x7b/0xd0 io_uring/io_uring.c:1244
> > task_work_run+0x16d/0x260 kernel/task_work.c:180
> > get_signal+0x1cb/0x25a0 kernel/signal.c:2669
> > arch_do_signal_or_restart+0x81/0x7e0 arch/x86/kernel/signal.c:310
> > exit_to_user_mode_loop kernel/entry/common.c:105 [inline]
> > exit_to_user_mode_prepare include/linux/entry-common.h:328 [inline]
> > __syscall_exit_to_user_mode_work kernel/entry/common.c:201 [inline]
> > syscall_exit_to_user_mode+0x156/0x2b0 kernel/entry/common.c:212
> > do_syscall_64+0xe5/0x270 arch/x86/entry/common.c:89
> > entry_SYSCALL_64_after_hwframe+0x6f/0x77
> > RIP: 0033:0x7f1bdbc2d88d
> > Code: c3 e8 a7 1f 00 00 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 89 f8 48
> > 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d
> > 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
> > RSP: 002b:00007ffd12f6fa18 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
> > RAX: 0000000000000001 RBX: 000000000000220b RCX: 00007f1bdbc2d88d
> > RDX: 0000000000000000 RSI: 0000000000005012 RDI: 0000000000000003
> > RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
> > R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000001
> > R13: 431bde82d7b634db R14: 00007f1bdbcaa4f0 R15: 0000000000000001
> > </TASK>
> >
> >
> > Syzkaller reproducer:
> > # {Threaded:false Repeat:true RepeatTimes:0 Procs:1 Slowdown:1
> > Sandbox: SandboxArg:0 Leak:false NetInjection:false NetDevices:false
> > NetReset:false Cgroups:false BinfmtMisc:false CloseFDs:false
> > KCSAN:false DevlinkPCI:false NicVF:false USB:false VhciInjection:false
> > Wifi:false IEEE802154:false Sysctl:false Swap:false UseTmpDir:false
> > HandleSegv:false Repro:false Trace:false LegacyOptions:{Collide:false
> > Fault:false FaultCall:0 FaultNth:0}}
> > r0 = syz_io_uring_setup(0x220b, &(0x7f0000000000)={0x0, 0x63db,
> > 0x10000, 0x800}, &(0x7f0000000080)=<r1=>0x0,
> > &(0x7f0000000200)=<r2=>0x0)
> > r3 = socket$inet(0x2, 0x1, 0x0)
> > syz_io_uring_submit(r1, r2,
> > &(0x7f0000000a80)=@IORING_OP_POLL_ADD={0x6, 0x0, 0x0, @fd=r3, 0x0,
> > 0x0, 0x1})
> > io_uring_enter(r0, 0x5012, 0x0, 0x0, 0x0, 0x0)
> > ```
> >
> > crepro is in the attachment.
> >
> > Best Regards
> > Xdchase
>
> --
> Pavel Begunkov
On 6/12/24 08:10, chase xd wrote:
> Sorry now I'm also a bit confused by the branch choosing. I checked
> out branch "for-6.9/io_uring" and started testing on that branch. I
> assume that was the latest version of io_uring at that time, even now
> I check out that branch and the bug still exists. How should I know
> whether the branch will be merged, and which branch do you think I
> should test on? Thanks.
# git show a69d20885494:io_uring/io_uring.c | grep -A 13 io_fill_cqe_req_aux
bool io_fill_cqe_req_aux(struct io_kiocb *req, bool defer, s32 res, u32 cflags)
{
struct io_ring_ctx *ctx = req->ctx;
u64 user_data = req->cqe.user_data;
if (!defer)
return __io_post_aux_cqe(ctx, user_data, res, cflags, false);
lockdep_assert_held(&ctx->uring_lock);
io_lockdep_assert_cq_locked(ctx);
ctx->submit_state.flush_cqes = true;
return io_fill_cqe_aux(ctx, user_data, res, cflags);
}
That's the buggy version from the hash you're testing, IIRC it
was in the tree for longer than necessary, presumably which is
why you found it, but it was never sent to Linus. Below is
current state of for-6.9 and what it was replaced with
respectively. Let me separately check for-6.9/io_uring if you're
concerned about it.
# git show for-6.9/io_uring:io_uring/io_uring.c | grep -A 30 io_fill_cqe_req_aux
bool io_fill_cqe_req_aux(struct io_kiocb *req, bool defer, s32 res, u32 cflags)
{
struct io_ring_ctx *ctx = req->ctx;
u64 user_data = req->cqe.user_data;
struct io_uring_cqe *cqe;
lockdep_assert(!io_wq_current_is_worker());
if (!defer)
return __io_post_aux_cqe(ctx, user_data, res, cflags, false);
lockdep_assert_held(&ctx->uring_lock);
if (ctx->submit_state.cqes_count == ARRAY_SIZE(ctx->completion_cqes)) {
...
# git show origin/for-6.10/io_uring:io_uring/io_uring.c | grep -A 13 io_req_post_cqe
bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags)
{
struct io_ring_ctx *ctx = req->ctx;
bool posted;
lockdep_assert(!io_wq_current_is_worker());
lockdep_assert_held(&ctx->uring_lock);
__io_cq_lock(ctx);
posted = io_fill_cqe_aux(ctx, req->cqe.user_data, res, cflags);
ctx->submit_state.cq_flush = true;
__io_cq_unlock_post(ctx);
return posted;
}
--
Pavel Begunkov
On 6/12/24 13:35, Pavel Begunkov wrote:
> On 6/12/24 08:10, chase xd wrote:
>> Sorry now I'm also a bit confused by the branch choosing. I checked
>> out branch "for-6.9/io_uring" and started testing on that branch. I
>> assume that was the latest version of io_uring at that time, even now
>> I check out that branch and the bug still exists. How should I know
>> whether the branch will be merged, and which branch do you think I
>> should test on? Thanks.
>
> # git show a69d20885494:io_uring/io_uring.c | grep -A 13 io_fill_cqe_req_aux
> bool io_fill_cqe_req_aux(struct io_kiocb *req, bool defer, s32 res, u32 cflags)
> {
> struct io_ring_ctx *ctx = req->ctx;
> u64 user_data = req->cqe.user_data;
>
> if (!defer)
> return __io_post_aux_cqe(ctx, user_data, res, cflags, false);
>
> lockdep_assert_held(&ctx->uring_lock);
> io_lockdep_assert_cq_locked(ctx);
>
> ctx->submit_state.flush_cqes = true;
> return io_fill_cqe_aux(ctx, user_data, res, cflags);
> }
>
> That's the buggy version from the hash you're testing, IIRC it
> was in the tree for longer than necessary, presumably which is
> why you found it, but it was never sent to Linus. Below is
> current state of for-6.9 and what it was replaced with
> respectively. Let me separately check for-6.9/io_uring if you're
> concerned about it.
In other words, it happens that bugs appear in the branches
but get rooted out before it gets anywhere. The main confusion
is that the version you're looking at was fixed up back somewhere
in March. That's fine, I'd just recommend fetch the repo and
update your base.
I can't hit the problem with for-6.9/io_uring, which make sense
because it's lacking the patch I'd blame it to. I'm confused
how you see it there.
> # git show for-6.9/io_uring:io_uring/io_uring.c | grep -A 30 io_fill_cqe_req_aux
> bool io_fill_cqe_req_aux(struct io_kiocb *req, bool defer, s32 res, u32 cflags)
> {
> struct io_ring_ctx *ctx = req->ctx;
> u64 user_data = req->cqe.user_data;
> struct io_uring_cqe *cqe;
>
> lockdep_assert(!io_wq_current_is_worker());
>
> if (!defer)
> return __io_post_aux_cqe(ctx, user_data, res, cflags, false);
>
> lockdep_assert_held(&ctx->uring_lock);
>
> if (ctx->submit_state.cqes_count == ARRAY_SIZE(ctx->completion_cqes)) {
> ...
>
> # git show origin/for-6.10/io_uring:io_uring/io_uring.c | grep -A 13 io_req_post_cqe
> bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags)
> {
> struct io_ring_ctx *ctx = req->ctx;
> bool posted;
>
> lockdep_assert(!io_wq_current_is_worker());
> lockdep_assert_held(&ctx->uring_lock);
>
> __io_cq_lock(ctx);
> posted = io_fill_cqe_aux(ctx, req->cqe.user_data, res, cflags);
> ctx->submit_state.cq_flush = true;
> __io_cq_unlock_post(ctx);
> return posted;
> }
>
--
Pavel Begunkov