Hello,
syzbot found the following crash on:
HEAD commit: 4e6d47206c32 tls: Add support for inplace records encryption
git tree: net-next
console output: https://syzkaller.appspot.com/x/log.txt?x=13834b81400000
kernel config: https://syzkaller.appspot.com/x/.config?x=e569aa5632ebd436
dashboard link: https://syzkaller.appspot.com/bug?extid=c7dd55d7aec49d48e49a
compiler: gcc (GCC) 8.0.1 20180413 (experimental)
Unfortunately, I don't have any reproducer for this crash yet.
IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: [email protected]
netlink: 'syz-executor1': attribute type 1 has an invalid length.
==================================================================
BUG: KASAN: use-after-free in sctp_id2assoc+0x3a7/0x3e0
net/sctp/socket.c:276
Read of size 8 at addr ffff880195b3eb20 by task syz-executor2/15454
CPU: 1 PID: 15454 Comm: syz-executor2 Not tainted 4.19.0-rc5+ #242
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
Google 01/01/2011
Call Trace:
__dump_stack lib/dump_stack.c:77 [inline]
dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113
print_address_description.cold.8+0x9/0x1ff mm/kasan/report.c:256
kasan_report_error mm/kasan/report.c:354 [inline]
kasan_report.cold.9+0x242/0x309 mm/kasan/report.c:412
__asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433
sctp_id2assoc+0x3a7/0x3e0 net/sctp/socket.c:276
sctp_do_peeloff+0x88/0x530 net/sctp/socket.c:5327
sctp_getsockopt_peeloff_common.isra.24+0xb5/0x2f0 net/sctp/socket.c:5376
sctp_getsockopt_peeloff_flags net/sctp/socket.c:5452 [inline]
sctp_getsockopt+0x17f1/0x7c78 net/sctp/socket.c:7474
sock_common_getsockopt+0x9a/0xe0 net/core/sock.c:2997
__sys_getsockopt+0x1ad/0x390 net/socket.c:1939
__do_sys_getsockopt net/socket.c:1950 [inline]
__se_sys_getsockopt net/socket.c:1947 [inline]
__x64_sys_getsockopt+0xbe/0x150 net/socket.c:1947
do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290
entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x457579
Code: 1d b4 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7
48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff
ff 0f 83 eb b3 fb ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007f86d7f8bc78 EFLAGS: 00000246 ORIG_RAX: 0000000000000037
RAX: ffffffffffffffda RBX: 0000000000000005 RCX: 0000000000457579
RDX: 000000000000007a RSI: 0000000000000084 RDI: 0000000000000007
RBP: 000000000072c040 R08: 00000000200001c0 R09: 0000000000000000
R10: 0000000020000080 R11: 0000000000000246 R12: 00007f86d7f8c6d4
R13: 00000000004c79b8 R14: 00000000004cdb70 R15: 00000000ffffffff
Allocated by task 15404:
save_stack+0x43/0xd0 mm/kasan/kasan.c:448
set_track mm/kasan/kasan.c:460 [inline]
kasan_kmalloc+0xc7/0xe0 mm/kasan/kasan.c:553
kmem_cache_alloc_trace+0x152/0x750 mm/slab.c:3620
kmalloc include/linux/slab.h:513 [inline]
kzalloc include/linux/slab.h:707 [inline]
sctp_association_new+0x12b/0x22a0 net/sctp/associola.c:311
__sctp_connect+0x6ad/0xda0 net/sctp/socket.c:1209
__sctp_setsockopt_connectx+0x134/0x190 net/sctp/socket.c:1368
sctp_setsockopt_connectx net/sctp/socket.c:1400 [inline]
sctp_setsockopt+0x18e2/0x6da0 net/sctp/socket.c:4359
sock_common_setsockopt+0x9a/0xe0 net/core/sock.c:3038
__sys_setsockopt+0x1ba/0x3c0 net/socket.c:1902
__do_sys_setsockopt net/socket.c:1913 [inline]
__se_sys_setsockopt net/socket.c:1910 [inline]
__x64_sys_setsockopt+0xbe/0x150 net/socket.c:1910
do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290
entry_SYSCALL_64_after_hwframe+0x49/0xbe
Freed by task 15404:
save_stack+0x43/0xd0 mm/kasan/kasan.c:448
set_track mm/kasan/kasan.c:460 [inline]
__kasan_slab_free+0x102/0x150 mm/kasan/kasan.c:521
kasan_slab_free+0xe/0x10 mm/kasan/kasan.c:528
__cache_free mm/slab.c:3498 [inline]
kfree+0xcf/0x230 mm/slab.c:3813
sctp_association_destroy net/sctp/associola.c:437 [inline]
sctp_association_put+0x264/0x350 net/sctp/associola.c:885
sctp_wait_for_connect+0x444/0x640 net/sctp/socket.c:8677
__sctp_connect+0xbb3/0xda0 net/sctp/socket.c:1260
__sctp_setsockopt_connectx+0x134/0x190 net/sctp/socket.c:1368
sctp_setsockopt_connectx net/sctp/socket.c:1400 [inline]
sctp_setsockopt+0x18e2/0x6da0 net/sctp/socket.c:4359
sock_common_setsockopt+0x9a/0xe0 net/core/sock.c:3038
__sys_setsockopt+0x1ba/0x3c0 net/socket.c:1902
__do_sys_setsockopt net/socket.c:1913 [inline]
__se_sys_setsockopt net/socket.c:1910 [inline]
__x64_sys_setsockopt+0xbe/0x150 net/socket.c:1910
do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290
entry_SYSCALL_64_after_hwframe+0x49/0xbe
The buggy address belongs to the object at ffff880195b3eb00
which belongs to the cache kmalloc-4096 of size 4096
The buggy address is located 32 bytes inside of
4096-byte region [ffff880195b3eb00, ffff880195b3fb00)
The buggy address belongs to the page:
page:ffffea000656cf80 count:1 mapcount:0 mapping:ffff8801da800dc0 index:0x0
compound_mapcount: 0
flags: 0x2fffc0000008100(slab|head)
raw: 02fffc0000008100 ffffea0006e66188 ffffea00065b5d08 ffff8801da800dc0
raw: 0000000000000000 ffff880195b3eb00 0000000100000001 0000000000000000
page dumped because: kasan: bad access detected
Memory state around the buggy address:
ffff880195b3ea00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
ffff880195b3ea80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
> ffff880195b3eb00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
^
ffff880195b3eb80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff880195b3ec00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================
---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at [email protected].
syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with
syzbot.
On Thu, Oct 04, 2018 at 01:48:03AM -0700, syzbot wrote:
> Hello,
>
> syzbot found the following crash on:
>
> HEAD commit: 4e6d47206c32 tls: Add support for inplace records encryption
> git tree: net-next
> console output: https://syzkaller.appspot.com/x/log.txt?x=13834b81400000
> kernel config: https://syzkaller.appspot.com/x/.config?x=e569aa5632ebd436
> dashboard link: https://syzkaller.appspot.com/bug?extid=c7dd55d7aec49d48e49a
> compiler: gcc (GCC) 8.0.1 20180413 (experimental)
>
> Unfortunately, I don't have any reproducer for this crash yet.
>
> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> Reported-by: [email protected]
>
> netlink: 'syz-executor1': attribute type 1 has an invalid length.
> ==================================================================
> BUG: KASAN: use-after-free in sctp_id2assoc+0x3a7/0x3e0
> net/sctp/socket.c:276
> Read of size 8 at addr ffff880195b3eb20 by task syz-executor2/15454
>
> CPU: 1 PID: 15454 Comm: syz-executor2 Not tainted 4.19.0-rc5+ #242
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
> Google 01/01/2011
> Call Trace:
> __dump_stack lib/dump_stack.c:77 [inline]
> dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113
> print_address_description.cold.8+0x9/0x1ff mm/kasan/report.c:256
> kasan_report_error mm/kasan/report.c:354 [inline]
> kasan_report.cold.9+0x242/0x309 mm/kasan/report.c:412
> __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433
> sctp_id2assoc+0x3a7/0x3e0 net/sctp/socket.c:276
I'm not seeing yet how this could happen.
All sockopts here are serialized by sock_lock.
do_peeloff here would create another socket, but the issue was
triggered before that.
The same function that freed this memory, also removes the entry from
idr mapping, so this entry shouldn't be there anymore.
I have only two theories so far:
- an issue with IDR/RCU.
- something else happened that just the call stacks are not revealing.
On Fri, Oct 5, 2018 at 4:58 PM, Marcelo Ricardo Leitner
<[email protected]> wrote:
> On Thu, Oct 04, 2018 at 01:48:03AM -0700, syzbot wrote:
>> Hello,
>>
>> syzbot found the following crash on:
>>
>> HEAD commit: 4e6d47206c32 tls: Add support for inplace records encryption
>> git tree: net-next
>> console output: https://syzkaller.appspot.com/x/log.txt?x=13834b81400000
>> kernel config: https://syzkaller.appspot.com/x/.config?x=e569aa5632ebd436
>> dashboard link: https://syzkaller.appspot.com/bug?extid=c7dd55d7aec49d48e49a
>> compiler: gcc (GCC) 8.0.1 20180413 (experimental)
>>
>> Unfortunately, I don't have any reproducer for this crash yet.
>>
>> IMPORTANT: if you fix the bug, please add the following tag to the commit:
>> Reported-by: [email protected]
>>
>> netlink: 'syz-executor1': attribute type 1 has an invalid length.
>> ==================================================================
>> BUG: KASAN: use-after-free in sctp_id2assoc+0x3a7/0x3e0
>> net/sctp/socket.c:276
>> Read of size 8 at addr ffff880195b3eb20 by task syz-executor2/15454
>>
>> CPU: 1 PID: 15454 Comm: syz-executor2 Not tainted 4.19.0-rc5+ #242
>> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
>> Google 01/01/2011
>> Call Trace:
>> __dump_stack lib/dump_stack.c:77 [inline]
>> dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113
>> print_address_description.cold.8+0x9/0x1ff mm/kasan/report.c:256
>> kasan_report_error mm/kasan/report.c:354 [inline]
>> kasan_report.cold.9+0x242/0x309 mm/kasan/report.c:412
>> __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433
>> sctp_id2assoc+0x3a7/0x3e0 net/sctp/socket.c:276
>
> I'm not seeing yet how this could happen.
> All sockopts here are serialized by sock_lock.
> do_peeloff here would create another socket, but the issue was
> triggered before that.
> The same function that freed this memory, also removes the entry from
> idr mapping, so this entry shouldn't be there anymore.
>
> I have only two theories so far:
> - an issue with IDR/RCU.
> - something else happened that just the call stacks are not revealing.
The "asoc->base.sk != sk" check after idr_find suggests that we don't
actually know what sock it belongs to. And if we don't know then
locking this sock can't help keeping another sock association alive.
Am I missing something obvious here? Should we take assoc ref while we
are still holding sctp_assocs_id_lock?
On Wed, Oct 10, 2018 at 05:28:12PM +0200, Dmitry Vyukov wrote:
> On Fri, Oct 5, 2018 at 4:58 PM, Marcelo Ricardo Leitner
> <[email protected]> wrote:
> > On Thu, Oct 04, 2018 at 01:48:03AM -0700, syzbot wrote:
> >> Hello,
> >>
> >> syzbot found the following crash on:
> >>
> >> HEAD commit: 4e6d47206c32 tls: Add support for inplace records encryption
> >> git tree: net-next
> >> console output: https://syzkaller.appspot.com/x/log.txt?x=13834b81400000
> >> kernel config: https://syzkaller.appspot.com/x/.config?x=e569aa5632ebd436
> >> dashboard link: https://syzkaller.appspot.com/bug?extid=c7dd55d7aec49d48e49a
> >> compiler: gcc (GCC) 8.0.1 20180413 (experimental)
> >>
> >> Unfortunately, I don't have any reproducer for this crash yet.
> >>
> >> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> >> Reported-by: [email protected]
> >>
> >> netlink: 'syz-executor1': attribute type 1 has an invalid length.
> >> ==================================================================
> >> BUG: KASAN: use-after-free in sctp_id2assoc+0x3a7/0x3e0
> >> net/sctp/socket.c:276
> >> Read of size 8 at addr ffff880195b3eb20 by task syz-executor2/15454
> >>
> >> CPU: 1 PID: 15454 Comm: syz-executor2 Not tainted 4.19.0-rc5+ #242
> >> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
> >> Google 01/01/2011
> >> Call Trace:
> >> __dump_stack lib/dump_stack.c:77 [inline]
> >> dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113
> >> print_address_description.cold.8+0x9/0x1ff mm/kasan/report.c:256
> >> kasan_report_error mm/kasan/report.c:354 [inline]
> >> kasan_report.cold.9+0x242/0x309 mm/kasan/report.c:412
> >> __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433
> >> sctp_id2assoc+0x3a7/0x3e0 net/sctp/socket.c:276
> >
> > I'm not seeing yet how this could happen.
> > All sockopts here are serialized by sock_lock.
> > do_peeloff here would create another socket, but the issue was
> > triggered before that.
> > The same function that freed this memory, also removes the entry from
> > idr mapping, so this entry shouldn't be there anymore.
> >
> > I have only two theories so far:
> > - an issue with IDR/RCU.
> > - something else happened that just the call stacks are not revealing.
>
> The "asoc->base.sk != sk" check after idr_find suggests that we don't
> actually know what sock it belongs to. And if we don't know then
Right. The check is more because the IDR is global and not per socket
(and we don't want sockets accessing asocs from other sockets), and not
that the asoc may move to another socket in between, but it also
protects from such cases, yes.
> locking this sock can't help keeping another sock association alive.
> Am I missing something obvious here? Should we take assoc ref while we
Not sure. Maybe I am. Thanks for looking into this, btw.
> are still holding sctp_assocs_id_lock?
Shouldn't be needed.
Solely by the call stacks:
- we tried to establish a new asoc from a sctp_connect() call,
blocking one.
- it slept waiting for the connect
- (something closed the asoc in between the sleeps, because it freed
the asoc right when waking up on sctp_wait_for_connect())
- it freed the asoc after sleeping on it on sctp_wait_for_connect [A]
- another thread tried to peeloff that asoc [B]
For [B] to access the asoc in question, it had to take the same sock
lock [A] had taken, and then the idr should not return an asoc in
sctp_i2asoc(). Note that we can't peeloff an asoc twice, thus why
the certainty here.
If [B] actually kicked in before the sleep resumed, that should have
been fine because it took the same sock lock [A] would have to
re-take. In this case an asoc would have been returned by
sctp_id2asoc(), the asoc would have been moved to a new socket, but
all while holding the original socket sock lock.
On Wed, Oct 10, 2018 at 8:13 PM, Marcelo Ricardo Leitner
<[email protected]> wrote:
> On Wed, Oct 10, 2018 at 05:28:12PM +0200, Dmitry Vyukov wrote:
>> On Fri, Oct 5, 2018 at 4:58 PM, Marcelo Ricardo Leitner
>> <[email protected]> wrote:
>> > On Thu, Oct 04, 2018 at 01:48:03AM -0700, syzbot wrote:
>> >> Hello,
>> >>
>> >> syzbot found the following crash on:
>> >>
>> >> HEAD commit: 4e6d47206c32 tls: Add support for inplace records encryption
>> >> git tree: net-next
>> >> console output: https://syzkaller.appspot.com/x/log.txt?x=13834b81400000
>> >> kernel config: https://syzkaller.appspot.com/x/.config?x=e569aa5632ebd436
>> >> dashboard link: https://syzkaller.appspot.com/bug?extid=c7dd55d7aec49d48e49a
>> >> compiler: gcc (GCC) 8.0.1 20180413 (experimental)
>> >>
>> >> Unfortunately, I don't have any reproducer for this crash yet.
>> >>
>> >> IMPORTANT: if you fix the bug, please add the following tag to the commit:
>> >> Reported-by: [email protected]
>> >>
>> >> netlink: 'syz-executor1': attribute type 1 has an invalid length.
>> >> ==================================================================
>> >> BUG: KASAN: use-after-free in sctp_id2assoc+0x3a7/0x3e0
>> >> net/sctp/socket.c:276
>> >> Read of size 8 at addr ffff880195b3eb20 by task syz-executor2/15454
>> >>
>> >> CPU: 1 PID: 15454 Comm: syz-executor2 Not tainted 4.19.0-rc5+ #242
>> >> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
>> >> Google 01/01/2011
>> >> Call Trace:
>> >> __dump_stack lib/dump_stack.c:77 [inline]
>> >> dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113
>> >> print_address_description.cold.8+0x9/0x1ff mm/kasan/report.c:256
>> >> kasan_report_error mm/kasan/report.c:354 [inline]
>> >> kasan_report.cold.9+0x242/0x309 mm/kasan/report.c:412
>> >> __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433
>> >> sctp_id2assoc+0x3a7/0x3e0 net/sctp/socket.c:276
>> >
>> > I'm not seeing yet how this could happen.
>> > All sockopts here are serialized by sock_lock.
>> > do_peeloff here would create another socket, but the issue was
>> > triggered before that.
>> > The same function that freed this memory, also removes the entry from
>> > idr mapping, so this entry shouldn't be there anymore.
>> >
>> > I have only two theories so far:
>> > - an issue with IDR/RCU.
>> > - something else happened that just the call stacks are not revealing.
>>
>> The "asoc->base.sk != sk" check after idr_find suggests that we don't
>> actually know what sock it belongs to. And if we don't know then
>
> Right. The check is more because the IDR is global and not per socket
> (and we don't want sockets accessing asocs from other sockets), and not
> that the asoc may move to another socket in between, but it also
> protects from such cases, yes.
>
>> locking this sock can't help keeping another sock association alive.
>> Am I missing something obvious here? Should we take assoc ref while we
>
> Not sure. Maybe I am. Thanks for looking into this, btw.
>
>> are still holding sctp_assocs_id_lock?
>
> Shouldn't be needed.
>
> Solely by the call stacks:
> - we tried to establish a new asoc from a sctp_connect() call,
> blocking one.
> - it slept waiting for the connect
> - (something closed the asoc in between the sleeps, because it freed
> the asoc right when waking up on sctp_wait_for_connect())
> - it freed the asoc after sleeping on it on sctp_wait_for_connect [A]
> - another thread tried to peeloff that asoc [B]
>
> For [B] to access the asoc in question, it had to take the same sock
> lock [A] had taken, and then the idr should not return an asoc in
> sctp_i2asoc(). Note that we can't peeloff an asoc twice, thus why
> the certainty here.
>
> If [B] actually kicked in before the sleep resumed, that should have
> been fine because it took the same sock lock [A] would have to
> re-take. In this case an asoc would have been returned by
> sctp_id2asoc(), the asoc would have been moved to a new socket, but
> all while holding the original socket sock lock.
But why A and B use the same lock?
sctp_assocs_id is global, so it contains asocs from all sockets, right?
assoc id comes straight from userspaces.
So isn't it possible that B uses completely different sock but passes
assoc id from the A sock? Then B should find assoc in sctp_assocs_id,
and at the point of "asoc->base.sk != sk" check the assoc can be
already freed.
On Wed, Oct 10, 2018 at 08:28:22PM +0200, Dmitry Vyukov wrote:
> On Wed, Oct 10, 2018 at 8:13 PM, Marcelo Ricardo Leitner
> <[email protected]> wrote:
> > On Wed, Oct 10, 2018 at 05:28:12PM +0200, Dmitry Vyukov wrote:
> >> On Fri, Oct 5, 2018 at 4:58 PM, Marcelo Ricardo Leitner
> >> <[email protected]> wrote:
> >> > On Thu, Oct 04, 2018 at 01:48:03AM -0700, syzbot wrote:
> >> >> Hello,
> >> >>
> >> >> syzbot found the following crash on:
> >> >>
> >> >> HEAD commit: 4e6d47206c32 tls: Add support for inplace records encryption
> >> >> git tree: net-next
> >> >> console output: https://syzkaller.appspot.com/x/log.txt?x=13834b81400000
> >> >> kernel config: https://syzkaller.appspot.com/x/.config?x=e569aa5632ebd436
> >> >> dashboard link: https://syzkaller.appspot.com/bug?extid=c7dd55d7aec49d48e49a
> >> >> compiler: gcc (GCC) 8.0.1 20180413 (experimental)
> >> >>
> >> >> Unfortunately, I don't have any reproducer for this crash yet.
> >> >>
> >> >> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> >> >> Reported-by: [email protected]
> >> >>
> >> >> netlink: 'syz-executor1': attribute type 1 has an invalid length.
> >> >> ==================================================================
> >> >> BUG: KASAN: use-after-free in sctp_id2assoc+0x3a7/0x3e0
> >> >> net/sctp/socket.c:276
> >> >> Read of size 8 at addr ffff880195b3eb20 by task syz-executor2/15454
> >> >>
> >> >> CPU: 1 PID: 15454 Comm: syz-executor2 Not tainted 4.19.0-rc5+ #242
> >> >> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
> >> >> Google 01/01/2011
> >> >> Call Trace:
> >> >> __dump_stack lib/dump_stack.c:77 [inline]
> >> >> dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113
> >> >> print_address_description.cold.8+0x9/0x1ff mm/kasan/report.c:256
> >> >> kasan_report_error mm/kasan/report.c:354 [inline]
> >> >> kasan_report.cold.9+0x242/0x309 mm/kasan/report.c:412
> >> >> __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433
> >> >> sctp_id2assoc+0x3a7/0x3e0 net/sctp/socket.c:276
> >> >
> >> > I'm not seeing yet how this could happen.
> >> > All sockopts here are serialized by sock_lock.
> >> > do_peeloff here would create another socket, but the issue was
> >> > triggered before that.
> >> > The same function that freed this memory, also removes the entry from
> >> > idr mapping, so this entry shouldn't be there anymore.
> >> >
> >> > I have only two theories so far:
> >> > - an issue with IDR/RCU.
> >> > - something else happened that just the call stacks are not revealing.
> >>
> >> The "asoc->base.sk != sk" check after idr_find suggests that we don't
> >> actually know what sock it belongs to. And if we don't know then
> >
> > Right. The check is more because the IDR is global and not per socket
> > (and we don't want sockets accessing asocs from other sockets), and not
> > that the asoc may move to another socket in between, but it also
> > protects from such cases, yes.
> >
> >> locking this sock can't help keeping another sock association alive.
> >> Am I missing something obvious here? Should we take assoc ref while we
> >
> > Not sure. Maybe I am. Thanks for looking into this, btw.
> >
> >> are still holding sctp_assocs_id_lock?
> >
> > Shouldn't be needed.
> >
> > Solely by the call stacks:
> > - we tried to establish a new asoc from a sctp_connect() call,
> > blocking one.
> > - it slept waiting for the connect
> > - (something closed the asoc in between the sleeps, because it freed
> > the asoc right when waking up on sctp_wait_for_connect())
> > - it freed the asoc after sleeping on it on sctp_wait_for_connect [A]
> > - another thread tried to peeloff that asoc [B]
> >
> > For [B] to access the asoc in question, it had to take the same sock
> > lock [A] had taken, and then the idr should not return an asoc in
> > sctp_i2asoc(). Note that we can't peeloff an asoc twice, thus why
> > the certainty here.
> >
> > If [B] actually kicked in before the sleep resumed, that should have
> > been fine because it took the same sock lock [A] would have to
> > re-take. In this case an asoc would have been returned by
> > sctp_id2asoc(), the asoc would have been moved to a new socket, but
> > all while holding the original socket sock lock.
>
> But why A and B use the same lock?
>
> sctp_assocs_id is global, so it contains asocs from all sockets, right?
> assoc id comes straight from userspaces.
> So isn't it possible that B uses completely different sock but passes
> assoc id from the A sock? Then B should find assoc in sctp_assocs_id,
> and at the point of "asoc->base.sk != sk" check the assoc can be
> already freed.
That explains it. Somehow I was thinking the issue was for reading
->dead instead. Now it's pretty clear. Then this should be the patch
we want. Can you please give it a spin? (only compile tested)
While holding the spinlock, an entry cannot be removed from the idr
and thus it cannot be freed. So even if the app uses an id from
another socket, it will still be there.
---8<---
diff --git a/net/sctp/socket.c b/net/sctp/socket.c
index f73e9d38d5ba..a7722f43aa69 100644
--- a/net/sctp/socket.c
+++ b/net/sctp/socket.c
@@ -271,11 +271,10 @@ struct sctp_association *sctp_id2assoc(struct sock *sk, sctp_assoc_t id)
spin_lock_bh(&sctp_assocs_id_lock);
asoc = (struct sctp_association *)idr_find(&sctp_assocs_id, (int)id);
+ if (asoc && (asoc->base.sk != sk || asoc->base.dead))
+ asoc = NULL;
spin_unlock_bh(&sctp_assocs_id_lock);
- if (!asoc || (asoc->base.sk != sk) || asoc->base.dead)
- return NULL;
-
return asoc;
}
On Wed, Oct 10, 2018 at 8:40 PM, Marcelo Ricardo Leitner
<[email protected]> wrote:
> On Wed, Oct 10, 2018 at 08:28:22PM +0200, Dmitry Vyukov wrote:
>> On Wed, Oct 10, 2018 at 8:13 PM, Marcelo Ricardo Leitner
>> <[email protected]> wrote:
>> > On Wed, Oct 10, 2018 at 05:28:12PM +0200, Dmitry Vyukov wrote:
>> >> On Fri, Oct 5, 2018 at 4:58 PM, Marcelo Ricardo Leitner
>> >> <[email protected]> wrote:
>> >> > On Thu, Oct 04, 2018 at 01:48:03AM -0700, syzbot wrote:
>> >> >> Hello,
>> >> >>
>> >> >> syzbot found the following crash on:
>> >> >>
>> >> >> HEAD commit: 4e6d47206c32 tls: Add support for inplace records encryption
>> >> >> git tree: net-next
>> >> >> console output: https://syzkaller.appspot.com/x/log.txt?x=13834b81400000
>> >> >> kernel config: https://syzkaller.appspot.com/x/.config?x=e569aa5632ebd436
>> >> >> dashboard link: https://syzkaller.appspot.com/bug?extid=c7dd55d7aec49d48e49a
>> >> >> compiler: gcc (GCC) 8.0.1 20180413 (experimental)
>> >> >>
>> >> >> Unfortunately, I don't have any reproducer for this crash yet.
>> >> >>
>> >> >> IMPORTANT: if you fix the bug, please add the following tag to the commit:
>> >> >> Reported-by: [email protected]
>> >> >>
>> >> >> netlink: 'syz-executor1': attribute type 1 has an invalid length.
>> >> >> ==================================================================
>> >> >> BUG: KASAN: use-after-free in sctp_id2assoc+0x3a7/0x3e0
>> >> >> net/sctp/socket.c:276
>> >> >> Read of size 8 at addr ffff880195b3eb20 by task syz-executor2/15454
>> >> >>
>> >> >> CPU: 1 PID: 15454 Comm: syz-executor2 Not tainted 4.19.0-rc5+ #242
>> >> >> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
>> >> >> Google 01/01/2011
>> >> >> Call Trace:
>> >> >> __dump_stack lib/dump_stack.c:77 [inline]
>> >> >> dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113
>> >> >> print_address_description.cold.8+0x9/0x1ff mm/kasan/report.c:256
>> >> >> kasan_report_error mm/kasan/report.c:354 [inline]
>> >> >> kasan_report.cold.9+0x242/0x309 mm/kasan/report.c:412
>> >> >> __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433
>> >> >> sctp_id2assoc+0x3a7/0x3e0 net/sctp/socket.c:276
>> >> >
>> >> > I'm not seeing yet how this could happen.
>> >> > All sockopts here are serialized by sock_lock.
>> >> > do_peeloff here would create another socket, but the issue was
>> >> > triggered before that.
>> >> > The same function that freed this memory, also removes the entry from
>> >> > idr mapping, so this entry shouldn't be there anymore.
>> >> >
>> >> > I have only two theories so far:
>> >> > - an issue with IDR/RCU.
>> >> > - something else happened that just the call stacks are not revealing.
>> >>
>> >> The "asoc->base.sk != sk" check after idr_find suggests that we don't
>> >> actually know what sock it belongs to. And if we don't know then
>> >
>> > Right. The check is more because the IDR is global and not per socket
>> > (and we don't want sockets accessing asocs from other sockets), and not
>> > that the asoc may move to another socket in between, but it also
>> > protects from such cases, yes.
>> >
>> >> locking this sock can't help keeping another sock association alive.
>> >> Am I missing something obvious here? Should we take assoc ref while we
>> >
>> > Not sure. Maybe I am. Thanks for looking into this, btw.
>> >
>> >> are still holding sctp_assocs_id_lock?
>> >
>> > Shouldn't be needed.
>> >
>> > Solely by the call stacks:
>> > - we tried to establish a new asoc from a sctp_connect() call,
>> > blocking one.
>> > - it slept waiting for the connect
>> > - (something closed the asoc in between the sleeps, because it freed
>> > the asoc right when waking up on sctp_wait_for_connect())
>> > - it freed the asoc after sleeping on it on sctp_wait_for_connect [A]
>> > - another thread tried to peeloff that asoc [B]
>> >
>> > For [B] to access the asoc in question, it had to take the same sock
>> > lock [A] had taken, and then the idr should not return an asoc in
>> > sctp_i2asoc(). Note that we can't peeloff an asoc twice, thus why
>> > the certainty here.
>> >
>> > If [B] actually kicked in before the sleep resumed, that should have
>> > been fine because it took the same sock lock [A] would have to
>> > re-take. In this case an asoc would have been returned by
>> > sctp_id2asoc(), the asoc would have been moved to a new socket, but
>> > all while holding the original socket sock lock.
>>
>> But why A and B use the same lock?
>>
>> sctp_assocs_id is global, so it contains asocs from all sockets, right?
>> assoc id comes straight from userspaces.
>> So isn't it possible that B uses completely different sock but passes
>> assoc id from the A sock? Then B should find assoc in sctp_assocs_id,
>> and at the point of "asoc->base.sk != sk" check the assoc can be
>> already freed.
>
> That explains it. Somehow I was thinking the issue was for reading
> ->dead instead. Now it's pretty clear. Then this should be the patch
> we want. Can you please give it a spin? (only compile tested)
syzbot can only test patches for bug with reproducers, this one
doesn't have one (not surprising taking into account subtliness of the
race):
https://github.com/google/syzkaller/blob/master/docs/syzbot.md#testing-patches
It's not possible squeeze in custom patches either:
https://github.com/google/syzkaller/blob/master/docs/syzbot.md#no-custom-patches
But the change looks good to me.
Acked-by: Dmitry Vyukov <[email protected]>
> While holding the spinlock, an entry cannot be removed from the idr
> and thus it cannot be freed. So even if the app uses an id from
> another socket, it will still be there.
>
> ---8<---
>
> diff --git a/net/sctp/socket.c b/net/sctp/socket.c
> index f73e9d38d5ba..a7722f43aa69 100644
> --- a/net/sctp/socket.c
> +++ b/net/sctp/socket.c
> @@ -271,11 +271,10 @@ struct sctp_association *sctp_id2assoc(struct sock *sk, sctp_assoc_t id)
>
> spin_lock_bh(&sctp_assocs_id_lock);
> asoc = (struct sctp_association *)idr_find(&sctp_assocs_id, (int)id);
> + if (asoc && (asoc->base.sk != sk || asoc->base.dead))
> + asoc = NULL;
> spin_unlock_bh(&sctp_assocs_id_lock);
>
> - if (!asoc || (asoc->base.sk != sk) || asoc->base.dead)
> - return NULL;
> -
> return asoc;
> }
>
On Wed, Oct 10, 2018 at 03:40:11PM -0300, Marcelo Ricardo Leitner wrote:
> On Wed, Oct 10, 2018 at 08:28:22PM +0200, Dmitry Vyukov wrote:
> > On Wed, Oct 10, 2018 at 8:13 PM, Marcelo Ricardo Leitner
> > <[email protected]> wrote:
> > > On Wed, Oct 10, 2018 at 05:28:12PM +0200, Dmitry Vyukov wrote:
> > >> On Fri, Oct 5, 2018 at 4:58 PM, Marcelo Ricardo Leitner
> > >> <[email protected]> wrote:
> > >> > On Thu, Oct 04, 2018 at 01:48:03AM -0700, syzbot wrote:
> > >> >> Hello,
> > >> >>
> > >> >> syzbot found the following crash on:
> > >> >>
> > >> >> HEAD commit: 4e6d47206c32 tls: Add support for inplace records encryption
> > >> >> git tree: net-next
> > >> >> console output: https://syzkaller.appspot.com/x/log.txt?x=13834b81400000
> > >> >> kernel config: https://syzkaller.appspot.com/x/.config?x=e569aa5632ebd436
> > >> >> dashboard link: https://syzkaller.appspot.com/bug?extid=c7dd55d7aec49d48e49a
> > >> >> compiler: gcc (GCC) 8.0.1 20180413 (experimental)
> > >> >>
> > >> >> Unfortunately, I don't have any reproducer for this crash yet.
> > >> >>
> > >> >> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > >> >> Reported-by: [email protected]
> > >> >>
> > >> >> netlink: 'syz-executor1': attribute type 1 has an invalid length.
> > >> >> ==================================================================
> > >> >> BUG: KASAN: use-after-free in sctp_id2assoc+0x3a7/0x3e0
> > >> >> net/sctp/socket.c:276
> > >> >> Read of size 8 at addr ffff880195b3eb20 by task syz-executor2/15454
> > >> >>
> > >> >> CPU: 1 PID: 15454 Comm: syz-executor2 Not tainted 4.19.0-rc5+ #242
> > >> >> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
> > >> >> Google 01/01/2011
> > >> >> Call Trace:
> > >> >> __dump_stack lib/dump_stack.c:77 [inline]
> > >> >> dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113
> > >> >> print_address_description.cold.8+0x9/0x1ff mm/kasan/report.c:256
> > >> >> kasan_report_error mm/kasan/report.c:354 [inline]
> > >> >> kasan_report.cold.9+0x242/0x309 mm/kasan/report.c:412
> > >> >> __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433
> > >> >> sctp_id2assoc+0x3a7/0x3e0 net/sctp/socket.c:276
> > >> >
> > >> > I'm not seeing yet how this could happen.
> > >> > All sockopts here are serialized by sock_lock.
> > >> > do_peeloff here would create another socket, but the issue was
> > >> > triggered before that.
> > >> > The same function that freed this memory, also removes the entry from
> > >> > idr mapping, so this entry shouldn't be there anymore.
> > >> >
> > >> > I have only two theories so far:
> > >> > - an issue with IDR/RCU.
> > >> > - something else happened that just the call stacks are not revealing.
> > >>
> > >> The "asoc->base.sk != sk" check after idr_find suggests that we don't
> > >> actually know what sock it belongs to. And if we don't know then
> > >
> > > Right. The check is more because the IDR is global and not per socket
> > > (and we don't want sockets accessing asocs from other sockets), and not
> > > that the asoc may move to another socket in between, but it also
> > > protects from such cases, yes.
> > >
> > >> locking this sock can't help keeping another sock association alive.
> > >> Am I missing something obvious here? Should we take assoc ref while we
> > >
> > > Not sure. Maybe I am. Thanks for looking into this, btw.
> > >
> > >> are still holding sctp_assocs_id_lock?
> > >
> > > Shouldn't be needed.
> > >
> > > Solely by the call stacks:
> > > - we tried to establish a new asoc from a sctp_connect() call,
> > > blocking one.
> > > - it slept waiting for the connect
> > > - (something closed the asoc in between the sleeps, because it freed
> > > the asoc right when waking up on sctp_wait_for_connect())
> > > - it freed the asoc after sleeping on it on sctp_wait_for_connect [A]
> > > - another thread tried to peeloff that asoc [B]
> > >
> > > For [B] to access the asoc in question, it had to take the same sock
> > > lock [A] had taken, and then the idr should not return an asoc in
> > > sctp_i2asoc(). Note that we can't peeloff an asoc twice, thus why
> > > the certainty here.
> > >
> > > If [B] actually kicked in before the sleep resumed, that should have
> > > been fine because it took the same sock lock [A] would have to
> > > re-take. In this case an asoc would have been returned by
> > > sctp_id2asoc(), the asoc would have been moved to a new socket, but
> > > all while holding the original socket sock lock.
> >
> > But why A and B use the same lock?
> >
> > sctp_assocs_id is global, so it contains asocs from all sockets, right?
> > assoc id comes straight from userspaces.
> > So isn't it possible that B uses completely different sock but passes
> > assoc id from the A sock? Then B should find assoc in sctp_assocs_id,
> > and at the point of "asoc->base.sk != sk" check the assoc can be
> > already freed.
>
> That explains it. Somehow I was thinking the issue was for reading
> ->dead instead. Now it's pretty clear. Then this should be the patch
> we want. Can you please give it a spin? (only compile tested)
>
> While holding the spinlock, an entry cannot be removed from the idr
> and thus it cannot be freed. So even if the app uses an id from
> another socket, it will still be there.
>
> ---8<---
>
> diff --git a/net/sctp/socket.c b/net/sctp/socket.c
> index f73e9d38d5ba..a7722f43aa69 100644
> --- a/net/sctp/socket.c
> +++ b/net/sctp/socket.c
> @@ -271,11 +271,10 @@ struct sctp_association *sctp_id2assoc(struct sock *sk, sctp_assoc_t id)
>
> spin_lock_bh(&sctp_assocs_id_lock);
> asoc = (struct sctp_association *)idr_find(&sctp_assocs_id, (int)id);
> + if (asoc && (asoc->base.sk != sk || asoc->base.dead))
> + asoc = NULL;
> spin_unlock_bh(&sctp_assocs_id_lock);
>
> - if (!asoc || (asoc->base.sk != sk) || asoc->base.dead)
> - return NULL;
> -
> return asoc;
> }
>
>
Marcello, can you post this with a proper changelog commit please? Based on the
bug report, and description of the problem, I think we can all agree this is a
sane fix
Thanks
Neil
On Tue, Oct 16, 2018 at 07:28:17AM -0400, Neil Horman wrote:
> On Wed, Oct 10, 2018 at 03:40:11PM -0300, Marcelo Ricardo Leitner wrote:
> > On Wed, Oct 10, 2018 at 08:28:22PM +0200, Dmitry Vyukov wrote:
> > > On Wed, Oct 10, 2018 at 8:13 PM, Marcelo Ricardo Leitner
> > > <[email protected]> wrote:
> > > > On Wed, Oct 10, 2018 at 05:28:12PM +0200, Dmitry Vyukov wrote:
> > > >> On Fri, Oct 5, 2018 at 4:58 PM, Marcelo Ricardo Leitner
> > > >> <[email protected]> wrote:
> > > >> > On Thu, Oct 04, 2018 at 01:48:03AM -0700, syzbot wrote:
> > > >> >> Hello,
> > > >> >>
> > > >> >> syzbot found the following crash on:
> > > >> >>
> > > >> >> HEAD commit: 4e6d47206c32 tls: Add support for inplace records encryption
> > > >> >> git tree: net-next
> > > >> >> console output: https://syzkaller.appspot.com/x/log.txt?x=13834b81400000
> > > >> >> kernel config: https://syzkaller.appspot.com/x/.config?x=e569aa5632ebd436
> > > >> >> dashboard link: https://syzkaller.appspot.com/bug?extid=c7dd55d7aec49d48e49a
> > > >> >> compiler: gcc (GCC) 8.0.1 20180413 (experimental)
> > > >> >>
> > > >> >> Unfortunately, I don't have any reproducer for this crash yet.
> > > >> >>
> > > >> >> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > > >> >> Reported-by: [email protected]
> > > >> >>
> > > >> >> netlink: 'syz-executor1': attribute type 1 has an invalid length.
> > > >> >> ==================================================================
> > > >> >> BUG: KASAN: use-after-free in sctp_id2assoc+0x3a7/0x3e0
> > > >> >> net/sctp/socket.c:276
> > > >> >> Read of size 8 at addr ffff880195b3eb20 by task syz-executor2/15454
> > > >> >>
> > > >> >> CPU: 1 PID: 15454 Comm: syz-executor2 Not tainted 4.19.0-rc5+ #242
> > > >> >> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
> > > >> >> Google 01/01/2011
> > > >> >> Call Trace:
> > > >> >> __dump_stack lib/dump_stack.c:77 [inline]
> > > >> >> dump_stack+0x1c4/0x2b4 lib/dump_stack.c:113
> > > >> >> print_address_description.cold.8+0x9/0x1ff mm/kasan/report.c:256
> > > >> >> kasan_report_error mm/kasan/report.c:354 [inline]
> > > >> >> kasan_report.cold.9+0x242/0x309 mm/kasan/report.c:412
> > > >> >> __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433
> > > >> >> sctp_id2assoc+0x3a7/0x3e0 net/sctp/socket.c:276
> > > >> >
> > > >> > I'm not seeing yet how this could happen.
> > > >> > All sockopts here are serialized by sock_lock.
> > > >> > do_peeloff here would create another socket, but the issue was
> > > >> > triggered before that.
> > > >> > The same function that freed this memory, also removes the entry from
> > > >> > idr mapping, so this entry shouldn't be there anymore.
> > > >> >
> > > >> > I have only two theories so far:
> > > >> > - an issue with IDR/RCU.
> > > >> > - something else happened that just the call stacks are not revealing.
> > > >>
> > > >> The "asoc->base.sk != sk" check after idr_find suggests that we don't
> > > >> actually know what sock it belongs to. And if we don't know then
> > > >
> > > > Right. The check is more because the IDR is global and not per socket
> > > > (and we don't want sockets accessing asocs from other sockets), and not
> > > > that the asoc may move to another socket in between, but it also
> > > > protects from such cases, yes.
> > > >
> > > >> locking this sock can't help keeping another sock association alive.
> > > >> Am I missing something obvious here? Should we take assoc ref while we
> > > >
> > > > Not sure. Maybe I am. Thanks for looking into this, btw.
> > > >
> > > >> are still holding sctp_assocs_id_lock?
> > > >
> > > > Shouldn't be needed.
> > > >
> > > > Solely by the call stacks:
> > > > - we tried to establish a new asoc from a sctp_connect() call,
> > > > blocking one.
> > > > - it slept waiting for the connect
> > > > - (something closed the asoc in between the sleeps, because it freed
> > > > the asoc right when waking up on sctp_wait_for_connect())
> > > > - it freed the asoc after sleeping on it on sctp_wait_for_connect [A]
> > > > - another thread tried to peeloff that asoc [B]
> > > >
> > > > For [B] to access the asoc in question, it had to take the same sock
> > > > lock [A] had taken, and then the idr should not return an asoc in
> > > > sctp_i2asoc(). Note that we can't peeloff an asoc twice, thus why
> > > > the certainty here.
> > > >
> > > > If [B] actually kicked in before the sleep resumed, that should have
> > > > been fine because it took the same sock lock [A] would have to
> > > > re-take. In this case an asoc would have been returned by
> > > > sctp_id2asoc(), the asoc would have been moved to a new socket, but
> > > > all while holding the original socket sock lock.
> > >
> > > But why A and B use the same lock?
> > >
> > > sctp_assocs_id is global, so it contains asocs from all sockets, right?
> > > assoc id comes straight from userspaces.
> > > So isn't it possible that B uses completely different sock but passes
> > > assoc id from the A sock? Then B should find assoc in sctp_assocs_id,
> > > and at the point of "asoc->base.sk != sk" check the assoc can be
> > > already freed.
> >
> > That explains it. Somehow I was thinking the issue was for reading
> > ->dead instead. Now it's pretty clear. Then this should be the patch
> > we want. Can you please give it a spin? (only compile tested)
> >
> > While holding the spinlock, an entry cannot be removed from the idr
> > and thus it cannot be freed. So even if the app uses an id from
> > another socket, it will still be there.
> >
> > ---8<---
> >
> > diff --git a/net/sctp/socket.c b/net/sctp/socket.c
> > index f73e9d38d5ba..a7722f43aa69 100644
> > --- a/net/sctp/socket.c
> > +++ b/net/sctp/socket.c
> > @@ -271,11 +271,10 @@ struct sctp_association *sctp_id2assoc(struct sock *sk, sctp_assoc_t id)
> >
> > spin_lock_bh(&sctp_assocs_id_lock);
> > asoc = (struct sctp_association *)idr_find(&sctp_assocs_id, (int)id);
> > + if (asoc && (asoc->base.sk != sk || asoc->base.dead))
> > + asoc = NULL;
> > spin_unlock_bh(&sctp_assocs_id_lock);
> >
> > - if (!asoc || (asoc->base.sk != sk) || asoc->base.dead)
> > - return NULL;
> > -
> > return asoc;
> > }
> >
> >
> Marcello, can you post this with a proper changelog commit please? Based on the
> bug report, and description of the problem, I think we can all agree this is a
> sane fix
Yes, in a few. The patch should be ready, but ahm.. I had destroyed by
test environment (disk failures). I'm seizing the moment to bring it
up.
Thanks,
Marcelo
>
>
> Thanks
> Neil
>