Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753050Ab3ITRvZ (ORCPT ); Fri, 20 Sep 2013 13:51:25 -0400 Received: from mail.windriver.com ([147.11.1.11]:58677 "EHLO mail.windriver.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752540Ab3ITRvX (ORCPT ); Fri, 20 Sep 2013 13:51:23 -0400 Date: Fri, 20 Sep 2013 13:51:50 -0400 From: Paul Gortmaker To: Paolo Bonzini CC: , , , , Subject: Re: [PATCH 0/3] KVM: Make kvm_lock non-raw Message-ID: <20130920175150.GA24770@windriver.com> References: <1379340373-5135-1-git-send-email-pbonzini@redhat.com> <5237824E.5090802@windriver.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <5237824E.5090802@windriver.com> User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 8879 Lines: 194 [Re: [PATCH 0/3] KVM: Make kvm_lock non-raw] On 16/09/2013 (Mon 18:12) Paul Gortmaker wrote: > On 13-09-16 10:06 AM, Paolo Bonzini wrote: > > Paul Gortmaker reported a BUG on preempt-rt kernels, due to taking the > > mmu_lock within the raw kvm_lock in mmu_shrink_scan. He provided a > > patch that shrunk the kvm_lock critical section so that the mmu_lock > > critical section does not nest with it, but in the end there is no reason > > for the vm_list to be protected by a raw spinlock. Only manipulations > > of kvm_usage_count and the consequent hardware_enable/disable operations > > are not preemptable. > > > > This small series thus splits the kvm_lock in the "raw" part and the > > "non-raw" part. > > > > Paul, could you please provide your Tested-by? > > Sure, I'll go back and see if I can find what triggered it in the > original report, and give the patches a spin on 3.4.x-rt (and probably > 3.10.x-rt, since that is where rt-current is presently). Seems fine on 3.4-rt. On 3.10.10-rt7 it looks like there are other issues, probably not explicitly related to this patchset (see below). Paul. -- e1000e 0000:00:19.0 eth1: removed PHC assign device 0:0:19.0 pci 0000:00:19.0: irq 43 for MSI/MSI-X pci 0000:00:19.0: irq 43 for MSI/MSI-X pci 0000:00:19.0: irq 43 for MSI/MSI-X pci 0000:00:19.0: irq 43 for MSI/MSI-X BUG: sleeping function called from invalid context at /home/paul/git/linux-rt/kernel/rtmutex.c:659 in_atomic(): 1, irqs_disabled(): 1, pid: 0, name: swapper/0 2 locks held by swapper/0/0: #0: (rcu_read_lock){.+.+.+}, at: [] kvm_set_irq_inatomic+0x2a/0x4a0 #1: (rcu_read_lock){.+.+.+}, at: [] kvm_irq_delivery_to_apic_fast+0x60/0x3d0 irq event stamp: 6121390 hardirqs last enabled at (6121389): [] restore_args+0x0/0x30 hardirqs last disabled at (6121390): [] common_interrupt+0x6a/0x6f softirqs last enabled at (0): [< (null)>] (null) softirqs last disabled at (0): [< (null)>] (null) Preemption disabled at:[] cpu_startup_entry+0x1ba/0x430 CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.10-rt7 #2 Hardware name: Dell Inc. OptiPlex 990/0VNP2H, BIOS A17 03/14/2013 ffffffff8201c440 ffff880223603cf0 ffffffff819f177d ffff880223603d18 ffffffff810c90d3 ffff880214a50110 0000000000000001 0000000000000001 ffff880223603d38 ffffffff819f89a4 ffff880214a50110 ffff880214a50110 Call Trace: [] dump_stack+0x19/0x1b [] __might_sleep+0x153/0x250 [] rt_spin_lock+0x24/0x60 [] __wake_up+0x36/0x70 [] kvm_vcpu_kick+0x3b/0xd0 [] __apic_accept_irq+0x2b2/0x3a0 [] kvm_apic_set_irq+0x27/0x30 [] kvm_irq_delivery_to_apic_fast+0x1ae/0x3d0 [] ? kvm_irq_delivery_to_apic_fast+0x60/0x3d0 [] kvm_set_irq_inatomic+0x12b/0x4a0 [] ? kvm_set_irq_inatomic+0x2a/0x4a0 [] kvm_assigned_dev_msi+0x23/0x40 [] handle_irq_event_percpu+0x88/0x3d0 [] ? cpu_startup_entry+0x19c/0x430 [] handle_irq_event+0x48/0x70 [] handle_edge_irq+0x77/0x120 [] handle_irq+0x1e/0x30 [] do_IRQ+0x5a/0xd0 [] common_interrupt+0x6f/0x6f [] ? retint_restore_args+0xe/0xe [] ? cpu_startup_entry+0x19c/0x430 [] ? cpu_startup_entry+0x158/0x430 [] rest_init+0x137/0x140 [] ? rest_init+0x5/0x140 [] start_kernel+0x3af/0x3bc [] ? repair_env_string+0x5e/0x5e [] x86_64_start_reservations+0x2a/0x2c [] x86_64_start_kernel+0xcc/0xcf ================================= [ INFO: inconsistent lock state ] 3.10.10-rt7 #2 Not tainted --------------------------------- inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage. swapper/0/0 [HC1[1]:SC0[0]:HE0:SE1] takes: (&(&(&q->lock)->lock)->wait_lock){?.+.-.}, at: [] rt_spin_lock_slowlock+0x48/0x370 {HARDIRQ-ON-W} state was registered at: [] __lock_acquire+0x69d/0x20e0 [] lock_acquire+0x9e/0x1f0 [] _raw_spin_lock+0x40/0x80 [] rt_spin_lock_slowlock+0x48/0x370 [] rt_spin_lock+0x2c/0x60 [] __wake_up+0x36/0x70 [] run_timer_softirq+0x1be/0x390 [] do_current_softirqs+0x239/0x5b0 [] run_ksoftirqd+0x38/0x60 [] smpboot_thread_fn+0x22c/0x340 [] kthread+0xcd/0xe0 [] ret_from_fork+0x7c/0xb0 irq event stamp: 6121390 hardirqs last enabled at (6121389): [] restore_args+0x0/0x30 hardirqs last disabled at (6121390): [] common_interrupt+0x6a/0x6f softirqs last enabled at (0): [< (null)>] (null) softirqs last disabled at (0): [< (null)>] (null) other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(&(&(&q->lock)->lock)->wait_lock); lock(&(&(&q->lock)->lock)->wait_lock); *** DEADLOCK *** 2 locks held by swapper/0/0: #0: (rcu_read_lock){.+.+.+}, at: [] kvm_set_irq_inatomic+0x2a/0x4a0 #1: (rcu_read_lock){.+.+.+}, at: [] kvm_irq_delivery_to_apic_fast+0x60/0x3d0 stack backtrace: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.10-rt7 #2 Hardware name: Dell Inc. OptiPlex 990/0VNP2H, BIOS A17 03/14/2013 ffffffff8262b550 ffff880223603a40 ffffffff819f177d ffff880223603a90 ffffffff819ec532 0000000000000000 ffffffff00000000 ffff880200000001 0000000000000002 ffffffff8201ccc0 ffffffff810f9040 0000000000000000 Call Trace: [] dump_stack+0x19/0x1b [] print_usage_bug.part.36+0x28b/0x29a [] ? check_usage_backwards+0x150/0x150 [] mark_lock+0x28b/0x6a0 [] __lock_acquire+0x949/0x20e0 [] ? __module_text_address+0x12/0x60 [] ? is_module_text_address+0x2f/0x60 [] ? __kernel_text_address+0x58/0x80 [] ? print_context_stack+0x62/0xf0 [] lock_acquire+0x9e/0x1f0 [] ? rt_spin_lock_slowlock+0x48/0x370 [] _raw_spin_lock+0x40/0x80 [] ? rt_spin_lock_slowlock+0x48/0x370 [] rt_spin_lock_slowlock+0x48/0x370 [] rt_spin_lock+0x2c/0x60 [] __wake_up+0x36/0x70 [] kvm_vcpu_kick+0x3b/0xd0 [] __apic_accept_irq+0x2b2/0x3a0 [] kvm_apic_set_irq+0x27/0x30 [] kvm_irq_delivery_to_apic_fast+0x1ae/0x3d0 [] ? kvm_irq_delivery_to_apic_fast+0x60/0x3d0 [] kvm_set_irq_inatomic+0x12b/0x4a0 [] ? kvm_set_irq_inatomic+0x2a/0x4a0 [] kvm_assigned_dev_msi+0x23/0x40 [] handle_irq_event_percpu+0x88/0x3d0 [] ? cpu_startup_entry+0x19c/0x430 [] handle_irq_event+0x48/0x70 [] handle_edge_irq+0x77/0x120 [] handle_irq+0x1e/0x30 [] do_IRQ+0x5a/0xd0 [] common_interrupt+0x6f/0x6f [] ? retint_restore_args+0xe/0xe [] ? cpu_startup_entry+0x19c/0x430 [] ? cpu_startup_entry+0x158/0x430 [] rest_init+0x137/0x140 [] ? rest_init+0x5/0x140 [] start_kernel+0x3af/0x3bc [] ? repair_env_string+0x5e/0x5e [] x86_64_start_reservations+0x2a/0x2c [] x86_64_start_kernel+0xcc/0xcf > > Paul. > -- > > > > > Thanks, > > > > Paolo > > > > Paolo Bonzini (3): > > KVM: cleanup (physical) CPU hotplug > > KVM: protect kvm_usage_count with its own spinlock > > KVM: Convert kvm_lock back to non-raw spinlock > > > > Documentation/virtual/kvm/locking.txt | 8 ++++-- > > arch/x86/kvm/mmu.c | 4 +-- > > arch/x86/kvm/x86.c | 8 +++--- > > include/linux/kvm_host.h | 2 +- > > virt/kvm/kvm_main.c | 51 ++++++++++++++++++----------------- > > 5 files changed, 40 insertions(+), 33 deletions(-) > > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/