Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
MIME-Version: 1.0
In-Reply-To: <20180513062131.GB654@sol.localdomain>
References: <001a1143e716efd3a9055cc8696b@google.com> <001a1146f1146fe0b505605e42c1@google.com>
 <CAOLK0pxR1qJC3sT2QoHgj-ys7FgFqY8724Xo9CLjWxi439ZrTw@mail.gmail.com>
 <91a32ec1-64e7-8b6b-9464-6599dd6735ad@intel.com> <20180126232755.z3gtqrm2oz6sp2t6@gmail.com>
 <d630a4c9-04ba-9e41-1f7b-11c59b2cd8e4@gmail.com> <20180406031524.GB21382@sol.localdomain>
 <20180513062131.GB654@sol.localdomain>
From:   Dmitry Vyukov <dvyukov@google.com>
Date:   Sat, 26 May 2018 11:24:01 +0200
Message-ID: <CACT4Y+a-3sP2DEbAp5G4YpKtu9jei-V_eyprMGjGyy60cZhPjA@mail.gmail.com>
Subject: Re: KASAN: use-after-free Write in irq_bypass_register_consumer
To:     Eric Biggers <ebiggers3@gmail.com>
Cc:     Tianyu Lan <lantianyu1986@gmail.com>,
        "Lan, Tianyu" <tianyu.lan@intel.com>,
        syzbot 
        <bot+7ad82526b98c64c90062275162dacfa48c20dbd3@syzkaller.appspotmail.com>,
        Christian Borntraeger <borntraeger@de.ibm.com>,
        Cornelia Huck <cornelia.huck@de.ibm.com>,
        David Hildenbrand <david@redhat.com>,
        "H. Peter Anvin" <hpa@zytor.com>, jschoenh@amazon.de,
        kvm <kvm@vger.kernel.org>,
        "linux-kernel@vger kernel org" <linux-kernel@vger.kernel.org>,
        Ingo Molnar <mingo@redhat.com>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Radim Krcmar <rkrcmar@redhat.com>,
        syzkaller-bugs <syzkaller-bugs@googlegroups.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        "the arch/x86 maintainers" <x86@kernel.org>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On Sun, May 13, 2018 at 8:21 AM, Eric Biggers <ebiggers3@gmail.com> wrote:
> On Thu, Apr 05, 2018 at 08:15:24PM -0700, Eric Biggers wrote:
>> On Mon, Jan 29, 2018 at 01:29:48PM +0800, Tianyu Lan wrote:
>> >
>> >
>> > On 1/27/2018 7:27 AM, Eric Biggers wrote:
>> > > On Sat, Dec 16, 2017 at 04:37:02PM +0800, Lan, Tianyu wrote:
>> > > > The root cause is that kvm_irqfd_assign() and kvm_irqfd_deassign() can't
>> > > > be run in parallel. Some data structure(e.g, irqfd->consumer) will be
>> > > > crashed because irqfd may be freed in deassign path before they are used
>> > > > in assign path. The other data maybe used in deassign path before
>> > > > initialization. Syzbot test hit such case. Add mutx between
>> > > > kvm_irqfd_assign() and kvm_irqfd_deassign() can fix such issue. Will
>> > > > send patch to fix it.
>> > > >
>> > > > On 12/16/2017 12:53 PM, Tianyu Lan wrote:
>> > > > > I reproduced the issue. Will have a look.
>> > > > >
>> > > > > -- Best regards Tianyu Lan 2017-12-15 18:14 GMT+08:00 syzbot
>> > > > > <bot+7ad82526b98c64c90062275162dacfa48c20dbd3@syzkaller.appspotmail.com>:
>> > > > > > syzkaller has found reproducer for the following crash on
>> > > > > > 82bcf1def3b5f1251177ad47c44f7e17af039b4b
>> > > > > > git://git.cmpxchg.org/linux-mmots.git/master
>> > > > > > compiler: gcc (GCC) 7.1.1 20170620
>> > > > > > .config is attached
>> > > > > > Raw console output is attached.
>> > > > > > C reproducer is attached
>> > > > > > syzkaller reproducer is attached. Seehttps://goo.gl/kgGztJ
>> > > > > > for information about syzkaller reproducers
>> > > > > >
>> > > > > >
>> > > > > > ==================================================================
>> > > > > > BUG: KASAN: use-after-free in __list_add include/linux/list.h:64 [inline]
>> > > > > > BUG: KASAN: use-after-free in list_add include/linux/list.h:79 [inline]
>> > > > > > BUG: KASAN: use-after-free in irq_bypass_register_consumer+0x4b4/0x500
>> > > > > > virt/lib/irqbypass.c:217
>> > > > > > Write of size 8 at addr ffff8801cdf51180 by task syzkaller436086/15031
>> > > > > >
>> > > > > > CPU: 1 PID: 15031 Comm: syzkaller436086 Not tainted 4.15.0-rc2-mm1+ #39
>> > > > > > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS
>> > > > > > Google 01/01/2011
>> > > > > > Call Trace:
>> > > > > >    __dump_stack lib/dump_stack.c:17 [inline]
>> > > > > >    dump_stack+0x194/0x257 lib/dump_stack.c:53
>> > > > > >    print_address_description+0x73/0x250 mm/kasan/report.c:252
>> > > > > >    kasan_report_error mm/kasan/report.c:351 [inline]
>> > > > > >    kasan_report+0x25b/0x340 mm/kasan/report.c:409
>> > > > > >    __asan_report_store8_noabort+0x17/0x20 mm/kasan/report.c:435
>> > > > > >    __list_add include/linux/list.h:64 [inline]
>> > > > > >    list_add include/linux/list.h:79 [inline]
>> > > > > >    irq_bypass_register_consumer+0x4b4/0x500 virt/lib/irqbypass.c:217
>> > > > > >    kvm_irqfd_assign arch/x86/kvm/../../../virt/kvm/eventfd.c:417 [inline]
>> > > > > >    kvm_irqfd+0x137f/0x1d50 arch/x86/kvm/../../../virt/kvm/eventfd.c:572
>> > > > > >    kvm_vm_ioctl+0x1079/0x1c40 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2992
>> > > > > >    vfs_ioctl fs/ioctl.c:46 [inline]
>> > > > > >    do_vfs_ioctl+0x1b1/0x1530 fs/ioctl.c:686
>> > > > > >    SYSC_ioctl fs/ioctl.c:701 [inline]
>> > > > > >    SyS_ioctl+0x8f/0xc0 fs/ioctl.c:692
>> > > > > >    entry_SYSCALL_64_fastpath+0x1f/0x96
>> > > > > > RIP: 0033:0x44d379
>> > > > > > RSP: 002b:00007fc5ff9a9d08 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
>> > > > > > RAX: ffffffffffffffda RBX: 00007fc5ff9aa700 RCX: 000000000044d379
>> > > > > > RDX: 0000000020080fe0 RSI: 000000004020ae76 RDI: 0000000000000005
>> > > > > > RBP: 00000000007ff900 R08: 00007fc5ff9aa700 R09: 00007fc5ff9aa700
>> > > > > > R10: 00007fc5ff9aa700 R11: 0000000000000246 R12: 0000000000000000
>> > > > > > R13: 00000000007ff8ff R14: 00007fc5ff9aa9c0 R15: 0000000000000000
>> > > > > >
>> > > > > > Allocated by task 15031:
>> > > > > >    save_stack+0x43/0xd0 mm/kasan/kasan.c:447
>> > > > > >    set_track mm/kasan/kasan.c:459 [inline]
>> > > > > >    kasan_kmalloc+0xad/0xe0 mm/kasan/kasan.c:551
>> > > > > >    kmem_cache_alloc_trace+0x136/0x750 mm/slab.c:3614
>> > > > > >    kmalloc include/linux/slab.h:516 [inline]
>> > > > > >    kzalloc include/linux/slab.h:705 [inline]
>> > > > > >    kvm_irqfd_assign arch/x86/kvm/../../../virt/kvm/eventfd.c:296 [inline]
>> > > > > >    kvm_irqfd+0x16c/0x1d50 arch/x86/kvm/../../../virt/kvm/eventfd.c:572
>> > > > > >    kvm_vm_ioctl+0x1079/0x1c40 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2992
>> > > > > >    vfs_ioctl fs/ioctl.c:46 [inline]
>> > > > > >    do_vfs_ioctl+0x1b1/0x1530 fs/ioctl.c:686
>> > > > > >    SYSC_ioctl fs/ioctl.c:701 [inline]
>> > > > > >    SyS_ioctl+0x8f/0xc0 fs/ioctl.c:692
>> > > > > >    entry_SYSCALL_64_fastpath+0x1f/0x96
>> > > > > >
>> > > > > > Freed by task 1402:
>> > > > > >    save_stack+0x43/0xd0 mm/kasan/kasan.c:447
>> > > > > >    set_track mm/kasan/kasan.c:459 [inline]
>> > > > > >    kasan_slab_free+0x71/0xc0 mm/kasan/kasan.c:524
>> > > > > >    __cache_free mm/slab.c:3492 [inline]
>> > > > > >    kfree+0xca/0x250 mm/slab.c:3807
>> > > > > >    irqfd_shutdown+0x13c/0x1a0 arch/x86/kvm/../../../virt/kvm/eventfd.c:148
>> > > > > >    process_one_work+0xbfd/0x1bc0 kernel/workqueue.c:2113
>> > > > > >    worker_thread+0x223/0x1990 kernel/workqueue.c:2247
>> > > > > >    kthread+0x37a/0x440 kernel/kthread.c:238
>> > > > > >    ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:524
>> > > > > >
>> > > > > > The buggy address belongs to the object at ffff8801cdf51000
>> > > > > >    which belongs to the cache kmalloc-512 of size 512
>> > > > > > The buggy address is located 384 bytes inside of
>> > > > > >    512-byte region [ffff8801cdf51000, ffff8801cdf51200)
>> > > > > > The buggy address belongs to the page:
>> > > > > > page:00000000d08a0d63 count:1 mapcount:0 mapping:00000000d54c7be6 index:0x0
>> > > > > > flags: 0x2fffc0000000100(slab)
>> > > > > > raw: 02fffc0000000100 ffff8801cdf51000 0000000000000000 0000000100000006
>> > > > > > raw: ffffea00073a7660 ffffea000737f3a0 ffff8801dac00940 0000000000000000
>> > > > > > page dumped because: kasan: bad access detected
>> > > > > >
>> > > > > > Memory state around the buggy address:
>> > > > > >    ffff8801cdf51080: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>> > > > > >    ffff8801cdf51100: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>> > > > > > > ffff8801cdf51180: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>> > > > > >                      ^
>> > > > > >    ffff8801cdf51200: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
>> > > > > >    ffff8801cdf51280: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>> > > > > > ==================================================================
>> > > > > >
>> > > Tianyu, are you still planning to send a patch for this?  This is still
>> > > happening.
>> > >
>> > > - Eric
>> > Yes, I sent out fix patch and wait for review.
>> >  https://patchwork.kernel.org/patch/10128925/
>> >
>>
>> This is still happening.  Can you please resend your patch?
>>
>
> Still happening; it's pretty easily reproducible, using the program below.
> (Sometimes it crashes in different ways too.  It also spams the log, ignore
> that.)  Tianyu's patch at https://patchwork.kernel.org/patch/10128925/ fixes it,
> though the locking is difficult to understand so I'm not sure it's correct yet.
> Probably the comment that says "do not drop the file until the irqfd is fully
> initialized, otherwise we might race against the EPOLLHUP" would be obsoleted
> and should be removed?  Can the KVM maintainers please review the patch too?
>
> #include <fcntl.h>
> #include <linux/kvm.h>
> #include <pthread.h>
> #include <sys/eventfd.h>
> #include <sys/ioctl.h>
>
> static int dev, vm, event;
>
> static void *irqfd_thrproc(void *_flags)
> {
>         struct kvm_irqfd irqfd = { .fd = event, .flags = (long)_flags };
>
>         for (;;)
>                 ioctl(vm, KVM_IRQFD, &irqfd);
> }
>
> int main()
> {
>         pthread_t t;
>
>         event = eventfd(0, 0);
>         dev = open("/dev/kvm", 0);
>         vm = ioctl(dev, KVM_CREATE_VM, 0);
>         ioctl(vm, KVM_CREATE_IRQCHIP, 0);
>         pthread_create(&t, NULL, irqfd_thrproc, (void *)(long)0);
>         irqfd_thrproc((void *)(long)KVM_IRQFD_FLAG_DEASSIGN);
> }


Tentative duplicates of this bug:

BUG: unable to handle kernel NULL pointer dereference in
irq_bypass_register_consumer
KASAN: use-after-free Read in irq_bypass_unregister_consumer
general protection fault in eventfd_ctx_remove_wait_queue
KASAN: use-after-free Read in irq_bypass_register_consumer

https://syzkaller.appspot.com/bug?id=45591ae3053c59fb50169401fb61cb596735f9d1