Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933368AbdCLLVL (ORCPT ); Sun, 12 Mar 2017 07:21:11 -0400 Received: from mail-vk0-f53.google.com ([209.85.213.53]:35188 "EHLO mail-vk0-f53.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755697AbdCLLVE (ORCPT ); Sun, 12 Mar 2017 07:21:04 -0400 MIME-Version: 1.0 In-Reply-To: References: <3e72461c-7197-e941-1d35-1aca34df2f8e@redhat.com> From: Dmitry Vyukov Date: Sun, 12 Mar 2017 12:20:42 +0100 Message-ID: Subject: Re: kvm: WARNING in mmu_spte_clear_track_bits To: Paolo Bonzini Cc: =?UTF-8?B?UmFkaW0gS3LEjW3DocWZ?= , Thomas Gleixner , Ingo Molnar , "H. Peter Anvin" , "x86@kernel.org" , KVM list , LKML , Alan Stern , Steve Rutherford , Xiao Guangrong , Haozhong Zhang , syzkaller Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 6482 Lines: 132 On Tue, Jan 17, 2017 at 5:00 PM, Dmitry Vyukov wrote: > On Tue, Jan 17, 2017 at 4:20 PM, Paolo Bonzini wrote: >> >> >> On 13/01/2017 12:15, Dmitry Vyukov wrote: >>> >>> I've commented out the WARNING for now, but I am seeing lots of >>> use-after-free's and rcu stalls involving mmu_spte_clear_track_bits: >>> >>> >>> BUG: KASAN: use-after-free in mmu_spte_clear_track_bits+0x186/0x190 >>> arch/x86/kvm/mmu.c:597 at addr ffff880068ae2008 >>> Read of size 8 by task syz-executor2/16715 >>> page:ffffea00016e6170 count:0 mapcount:0 mapping: (null) index:0x0 >>> flags: 0x500000000000000() >>> raw: 0500000000000000 0000000000000000 0000000000000000 00000000ffffffff >>> raw: ffffea00017ec5a0 ffffea0001783d48 ffff88006aec5d98 >>> page dumped because: kasan: bad access detected >>> CPU: 2 PID: 16715 Comm: syz-executor2 Not tainted 4.10.0-rc3+ #163 >>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 >>> Call Trace: >>> __dump_stack lib/dump_stack.c:15 [inline] >>> dump_stack+0x292/0x3a2 lib/dump_stack.c:51 >>> kasan_report_error mm/kasan/report.c:213 [inline] >>> kasan_report+0x42d/0x460 mm/kasan/report.c:307 >>> __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:333 >>> mmu_spte_clear_track_bits+0x186/0x190 arch/x86/kvm/mmu.c:597 >>> drop_spte+0x24/0x280 arch/x86/kvm/mmu.c:1182 >>> kvm_zap_rmapp+0x119/0x260 arch/x86/kvm/mmu.c:1401 >>> kvm_unmap_rmapp+0x1d/0x30 arch/x86/kvm/mmu.c:1412 >>> kvm_handle_hva_range+0x54a/0x7d0 arch/x86/kvm/mmu.c:1565 >>> kvm_unmap_hva_range+0x2e/0x40 arch/x86/kvm/mmu.c:1591 >>> kvm_mmu_notifier_invalidate_range_start+0xae/0x140 >>> arch/x86/kvm/../../../virt/kvm/kvm_main.c:360 >>> __mmu_notifier_invalidate_range_start+0x1f8/0x300 mm/mmu_notifier.c:199 >>> mmu_notifier_invalidate_range_start include/linux/mmu_notifier.h:282 [inline] >>> unmap_vmas+0x14b/0x1b0 mm/memory.c:1368 >>> unmap_region+0x2f8/0x560 mm/mmap.c:2460 >>> do_munmap+0x7b8/0xfa0 mm/mmap.c:2657 >>> mmap_region+0x68f/0x18e0 mm/mmap.c:1612 >>> do_mmap+0x6a2/0xd40 mm/mmap.c:1450 >>> do_mmap_pgoff include/linux/mm.h:2031 [inline] >>> vm_mmap_pgoff+0x1a9/0x200 mm/util.c:305 >>> SYSC_mmap_pgoff mm/mmap.c:1500 [inline] >>> SyS_mmap_pgoff+0x22c/0x5d0 mm/mmap.c:1458 >>> SYSC_mmap arch/x86/kernel/sys_x86_64.c:95 [inline] >>> SyS_mmap+0x16/0x20 arch/x86/kernel/sys_x86_64.c:86 >>> entry_SYSCALL_64_fastpath+0x1f/0xc2 >>> RIP: 0033:0x445329 >>> RSP: 002b:00007fb33933cb58 EFLAGS: 00000282 ORIG_RAX: 0000000000000009 >>> RAX: ffffffffffffffda RBX: 0000000020000000 RCX: 0000000000445329 >>> RDX: 0000000000000003 RSI: 0000000000af1000 RDI: 0000000020000000 >>> RBP: 00000000006dfe90 R08: ffffffffffffffff R09: 0000000000000000 >>> R10: 0000000000000032 R11: 0000000000000282 R12: 0000000000700000 >>> R13: 0000000000000006 R14: ffffffffffffffff R15: 0000000020001000 >>> Memory state around the buggy address: >>> ffff880068ae1f00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >>> ffff880068ae1f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 >>>> ffff880068ae2000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff >>> ^ >>> ffff880068ae2080: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff >>> ffff880068ae2100: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff >>> ================================================================== >> >> This could be related to the gfn_to_rmap issues. > > > Humm... That's possible. Potentially I am not seeing any more of > spte-related crashes after I applied the following patch: > > --- a/virt/kvm/kvm_main.c > +++ b/virt/kvm/kvm_main.c > @@ -968,8 +968,7 @@ int __kvm_set_memory_region(struct kvm *kvm, > /* Check for overlaps */ > r = -EEXIST; > kvm_for_each_memslot(slot, __kvm_memslots(kvm, as_id)) { > - if ((slot->id >= KVM_USER_MEM_SLOTS) || > - (slot->id == id)) > + if (slot->id == id) > continue; > if (!((base_gfn + npages <= slot->base_gfn) || > (base_gfn >= slot->base_gfn + slot->npages))) Friendly ping. Just hit it on mmotm/86292b33d4b79ee03e2f43ea0381ef85f077c760 (without the above change): ------------[ cut here ]------------ WARNING: CPU: 1 PID: 31060 at arch/x86/kvm/mmu.c:682 mmu_spte_clear_track_bits+0x3a1/0x420 arch/x86/kvm/mmu.c:682 CPU: 1 PID: 31060 Comm: syz-executor0 Not tainted 4.11.0-rc1+ #328 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:16 [inline] dump_stack+0x1a7/0x26a lib/dump_stack.c:52 panic+0x1f8/0x40f kernel/panic.c:180 __warn+0x1c4/0x1e0 kernel/panic.c:541 warn_slowpath_null+0x2c/0x40 kernel/panic.c:584 mmu_spte_clear_track_bits+0x3a1/0x420 arch/x86/kvm/mmu.c:682 drop_spte+0x24/0x280 arch/x86/kvm/mmu.c:1323 mmu_page_zap_pte+0x223/0x350 arch/x86/kvm/mmu.c:2438 kvm_mmu_page_unlink_children arch/x86/kvm/mmu.c:2460 [inline] kvm_mmu_prepare_zap_page+0x1ce/0x13d0 arch/x86/kvm/mmu.c:2504 kvm_zap_obsolete_pages arch/x86/kvm/mmu.c:5134 [inline] kvm_mmu_invalidate_zap_all_pages+0x4d4/0x6b0 arch/x86/kvm/mmu.c:5175 kvm_arch_flush_shadow_all+0x15/0x20 arch/x86/kvm/x86.c:8364 kvm_mmu_notifier_release+0x71/0xb0 arch/x86/kvm/../../../virt/kvm/kvm_main.c:472 __mmu_notifier_release+0x1e5/0x6b0 mm/mmu_notifier.c:75 mmu_notifier_release include/linux/mmu_notifier.h:235 [inline] exit_mmap+0x3a3/0x470 mm/mmap.c:2941 __mmput kernel/fork.c:890 [inline] mmput+0x228/0x700 kernel/fork.c:912 exit_mm kernel/exit.c:558 [inline] do_exit+0x9e8/0x1c20 kernel/exit.c:866 do_group_exit+0x149/0x400 kernel/exit.c:983 get_signal+0x6d9/0x1840 kernel/signal.c:2318 do_signal+0x94/0x1f30 arch/x86/kernel/signal.c:808 exit_to_usermode_loop+0x1e5/0x2d0 arch/x86/entry/common.c:157 prepare_exit_to_usermode arch/x86/entry/common.c:191 [inline] syscall_return_slowpath+0x3bd/0x460 arch/x86/entry/common.c:260 entry_SYSCALL_64_fastpath+0xc0/0xc2 RIP: 0033:0x4458d9 RSP: 002b:00007ffa472c3b58 EFLAGS: 00000286 ORIG_RAX: 00000000000000ce RAX: fffffffffffffff4 RBX: 0000000000708000 RCX: 00000000004458d9 RDX: 0000000000000000 RSI: 000000002006bff8 RDI: 000000000000a05b RBP: 0000000000000fe0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000286 R12: 00000000006df0a0 R13: 000000000000a05b R14: 000000002006bff8 R15: 0000000000000000