Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1760062Ab2EJPUJ (ORCPT ); Thu, 10 May 2012 11:20:09 -0400 Received: from e23smtp02.au.ibm.com ([202.81.31.144]:57893 "EHLO e23smtp02.au.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757805Ab2EJPUC (ORCPT ); Thu, 10 May 2012 11:20:02 -0400 From: Nikunj A Dadhania To: Avi Kivity Cc: Andrea Arcangeli , KVM , LKML , Ingo Molnar , Peter Zijlstra Subject: Re: KVM: Softlockups in guests while running kernbench In-Reply-To: <4FAB88B8.5060203@redhat.com> References: <87wr4kjvtk.fsf@abhimanyu.in.ibm.com> <4FAB88B8.5060203@redhat.com> User-Agent: Notmuch/0.10.2+70~gf0e0053 (http://notmuchmail.org) Emacs/24.0.95.1 (x86_64-unknown-linux-gnu) Date: Thu, 10 May 2012 15:39:04 +0530 Message-ID: <87pqace4an.fsf@abhimanyu.in.ibm.com> MIME-Version: 1.0 Content-Type: text/plain x-cbid: 12050923-5490-0000-0000-0000015B2267 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 103595 Lines: 1977 On Thu, 10 May 2012 12:22:00 +0300, Avi Kivity wrote: > On 05/10/2012 11:15 AM, Nikunj A Dadhania wrote: > > I am running a 3.4.0-rc4 based kernel(commit: af3a3ab2), guest config > > attached. > > > > During my tests, I saw few softlockups inside the guests while running > > kernbench inside the guest. I can reproduce this repeatedly. > > > > Test Setup: > > > > - Create 4 VMs (8vcpu, 1GB RAM) > > - Run kernbench inside each guests (kernbench -f -M -H -o 16) in parallel > > > > How overcommitted are you? 8 Physical CPU and 32 vCPUs > What does the host hardware look like? Machine : IBM xSeries with Intel(R) Xeon(R) x5570 2.93GHz CPU with 8 core , 64GB RAM Threading is disabled. > Is there any load on the host? No > > The traces themselves always point a couple of instructions after > interrupts are enabled. Can you post a few more? > Console logs of one VM: ---------------------------------------------------------------- BUG: soft lockup - CPU#1 stuck for 22s! [khugepaged:52] Modules linked in: CPU 1 Modules linked in: Pid: 52, comm: khugepaged Not tainted 3.4.0-rc4-base+ #1 Red Hat KVM RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x19/0x30 RSP: 0000:ffff880038189a38 EFLAGS: 00000286 RAX: 0000000000000000 RBX: ffff88003df90000 RCX: ffff88003d981680 RDX: ffffea00005647d8 RSI: 0000000000000286 RDI: 0000000000000286 RBP: ffff880038189a40 R08: ffffea0000562ae0 R09: f600000000000000 R10: ffff88003d981680 R11: 0000000000000000 R12: ffff880038189a00 R13: ffff88003d655258 R14: 0000000000000297 R15: ffffea000055ebf8 FS: 0000000000000000(0000) GS:ffff88003d640000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007ff586ffc000 CR3: 0000000001a0b000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process khugepaged (pid: 52, threadinfo ffff880038188000, task ffff880038182b30) Stack: BUG: soft lockup - CPU#3 stuck for 22s! [cc1:5177] Modules linked in: CPU 3 Modules linked in: Pid: 5177, comm: cc1 Not tainted 3.4.0-rc4-base+ #1 Red Hat KVM RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x19/0x30 RSP: 0018:ffff88003806fcc0 EFLAGS: 00000286 RAX: ffffea0000a5ef68 RBX: ffffffff8110d7f8 RCX: ffff88003d981680 RDX: ffffea0000a5e1a0 RSI: 0000000000000286 RDI: 0000000000000286 RBP: ffff88003806fcc8 R08: ffffea0000a5e1a0 R09: 0000000000000000 R10: ffff88003e7fdfb8 R11: 0000000000000000 R12: 0000000000000297 R13: ffff88003806fc88 R14: ffff88003d6d5258 R15: 0000000000000297 FS: 0000000000000000(0000) GS:ffff88003d6c0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000003cddcaae90 CR3: 0000000001a0b000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process cc1 (pid: 5177, threadinfo ffff88003806e000, task ffff88003dc914a0) Stack: 000000000000000e ffff88003806fd48 ffffffff8110d7ed ffff88003806fd78 ffffea0000a5ef68 ffff88003d981680 0000000000000005 0000000000000286 0000000200000000 ffffea0000a5ef68 ffffea0000a35908 0000000000000000 Call Trace: [] release_pages+0x1ad/0x210 [] free_pages_and_swap_cache+0xb4/0xe0 [] ? cpumask_any_but+0x31/0x50 [] tlb_flush_mmu+0x6c/0x90 [] tlb_finish_mmu+0x14/0x40 [] exit_mmap+0xd8/0x140 [] mmput+0x73/0x120 [] exit_mm+0x10a/0x140 [] do_exit+0x16b/0x450 [] do_group_exit+0x55/0xd0 [] sys_exit_group+0x17/0x20 [] system_call_fastpath+0x16/0x1b Code: 66 90 e8 db b9 a4 ff 66 90 c9 c3 0f 1f 80 00 00 00 00 55 48 89 e5 53 66 66 66 66 90 48 89 f3 e8 1e ba a4 ff 66 90 48 89 df 57 9d <66> 66 90 66 90 5b c9 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 Call Trace: [] release_pages+0x1ad/0x210 [] free_pages_and_swap_cache+0xb4/0xe0 [] ? cpumask_any_but+0x31/0x50 [] tlb_flush_mmu+0x6c/0x90 [] tlb_finish_mmu+0x14/0x40 [] exit_mmap+0xd8/0x140 [] mmput+0x73/0x120 [] exit_mm+0x10a/0x140 [] do_exit+0x16b/0x450 [] do_group_exit+0x55/0xd0 [] sys_exit_group+0x17/0x20 [] system_call_fastpath+0x16/0x1b 000000000000000e ffff880038189ac0 ffffffff8110d7ed 0000000000000004 ffffea00005c1238 ffff88003d981680 0000000000000009 0000000000000286 0000000200000282 ffffea00005c1238 ffffea0000562878 0000000000000010 Call Trace: [] release_pages+0x1ad/0x210 [] ? add_page_to_unevictable_list+0xd0/0xd0 [] pagevec_lru_move_fn+0xcf/0xf0 [] __pagevec_lru_add+0x17/0x20 [] __lru_cache_add+0x68/0x90 [] ? __mem_cgroup_uncharge_common+0xf9/0x280 [] lru_cache_add_lru+0x2d/0x50 [] putback_lru_page+0x69/0xe0 [] release_pte_page+0x2b/0x40 [] __collapse_huge_page_copy+0x185/0x200 [] ? __collapse_huge_page_isolate+0x116/0x1c0 [] collapse_huge_page+0x327/0x500 [] khugepaged_scan_pmd+0x1c5/0x2f0 [] ? __schedule+0x3de/0x710 [] khugepaged_scan_mm_slot+0x17e/0x280 [] khugepaged_do_scan+0xbc/0xe0 [] khugepaged_loop+0x5f/0x3c0 [] ? wake_up_bit+0x40/0x40 [] ? khugepaged_loop+0x3c0/0x3c0 [] khugepaged+0x4a/0xb0 [] kthread+0x9e/0xb0 [] kernel_thread_helper+0x4/0x10 [] ? kthread_freezable_should_stop+0x70/0x70 [] ? gs_change+0x13/0x13 Code: 66 90 e8 db b9 a4 ff 66 90 c9 c3 0f 1f 80 00 00 00 00 55 48 89 e5 53 66 66 66 66 90 48 89 f3 e8 1e ba a4 ff 66 90 48 89 df 57 9d <66> 66 90 66 90 5b c9 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 Call Trace: [] release_pages+0x1ad/0x210 [] ? add_page_to_unevictable_list+0xd0/0xd0 [] pagevec_lru_move_fn+0xcf/0xf0 [] __pagevec_lru_add+0x17/0x20 [] __lru_cache_add+0x68/0x90 [] ? __mem_cgroup_uncharge_common+0xf9/0x280 [] lru_cache_add_lru+0x2d/0x50 [] putback_lru_page+0x69/0xe0 [] release_pte_page+0x2b/0x40 [] __collapse_huge_page_copy+0x185/0x200 [] ? __collapse_huge_page_isolate+0x116/0x1c0 [] collapse_huge_page+0x327/0x500 [] khugepaged_scan_pmd+0x1c5/0x2f0 [] ? __schedule+0x3de/0x710 [] khugepaged_scan_mm_slot+0x17e/0x280 [] khugepaged_do_scan+0xbc/0xe0 [] khugepaged_loop+0x5f/0x3c0 [] ? wake_up_bit+0x40/0x40 [] ? khugepaged_loop+0x3c0/0x3c0 [] khugepaged+0x4a/0xb0 [] kthread+0x9e/0xb0 [] kernel_thread_helper+0x4/0x10 [] ? kthread_freezable_should_stop+0x70/0x70 [] ? gs_change+0x13/0x13 hdparm: sending ioctl 330 to a partition! hdparm: sending ioctl 330 to a partition! hdparm: sending ioctl 330 to a partition! hdparm: sending ioctl 330 to a partition! hdparm: sending ioctl 330 to a partition! hdparm: sending ioctl 330 to a partition! hdparm: sending ioctl 330 to a partition! hdparm: sending ioctl 330 to a partition! hdparm: sending ioctl 330 to a partition! hdparm: sending ioctl 330 to a partition! BUG: soft lockup - CPU#6 stuck for 22s! [khugepaged:52] Modules linked in: CPU 6 Modules linked in: Pid: 52, comm: khugepaged Not tainted 3.4.0-rc4-base+ #1 Red Hat KVM RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x19/0x30 RSP: 0018:ffff880038189ab8 EFLAGS: 00000282 RAX: 0000000000000282 RBX: 000000000000000e RCX: ffffc90000182000 RDX: 0000000000000009 RSI: 0000000000000282 RDI: 0000000000000282 RBP: ffff880038189ac0 R08: 000000000000000a R09: 9e00000000000000 R10: ffff88003d981680 R11: 0000000000000000 R12: ffff88003d78f1e0 R13: 000000000000000e R14: 0000000000000010 R15: ffffffff81109eaf FS: 0000000000000000(0000) GS:ffff88003d780000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f6ffe87f000 CR3: 0000000001a0b000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process khugepaged (pid: 52, threadinfo ffff880038188000, task ffff880038182b30) Stack: ffff88003d78f160 ffff880038189b10 ffffffff8110d90f 0000000000000282 0000000000000001 ffffea00009e05e0 ffffea00007efa78 ffff88003d78f160 0000000000000000 00000000ffffffff 000000000067ddb8 ffff880038189b20 Call Trace: [] pagevec_lru_move_fn+0xbf/0xf0 [] __pagevec_lru_add+0x17/0x20 [] __lru_cache_add+0x68/0x90 [] ? __mem_cgroup_uncharge_common+0xf9/0x280 [] lru_cache_add_lru+0x2d/0x50 [] putback_lru_page+0x69/0xe0 [] release_pte_page+0x2b/0x40 [] __collapse_huge_page_copy+0x185/0x200 [] ? __collapse_huge_page_isolate+0x116/0x1c0 [] collapse_huge_page+0x327/0x500 [] ? rb_erase+0xce/0x150 [] khugepaged_scan_pmd+0x1c5/0x2f0 [] khugepaged_scan_mm_slot+0x17e/0x280 [] khugepaged_do_scan+0xbc/0xe0 [] khugepaged_loop+0x5f/0x3c0 [] ? wake_up_bit+0x40/0x40 [] ? khugepaged_loop+0x3c0/0x3c0 [] khugepaged+0x4a/0xb0 [] kthread+0x9e/0xb0 [] kernel_thread_helper+0x4/0x10 [] ? kthread_freezable_should_stop+0x70/0x70 [] ? gs_change+0x13/0x13 Code: 66 90 e8 db b9 a4 ff 66 90 c9 c3 0f 1f 80 00 00 00 00 55 48 89 e5 53 66 66 66 66 90 48 89 f3 e8 1e ba a4 ff 66 90 48 89 df 57 9d <66> 66 90 66 90 5b c9 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 Call Trace: [] pagevec_lru_move_fn+0xbf/0xf0 [] __pagevec_lru_add+0x17/0x20 [] __lru_cache_add+0x68/0x90 [] ? __mem_cgroup_uncharge_common+0xf9/0x280 [] lru_cache_add_lru+0x2d/0x50 [] putback_lru_page+0x69/0xe0 [] release_pte_page+0x2b/0x40 [] __collapse_huge_page_copy+0x185/0x200 [] ? __collapse_huge_page_isolate+0x116/0x1c0 [] collapse_huge_page+0x327/0x500 [] ? rb_erase+0xce/0x150 [] khugepaged_scan_pmd+0x1c5/0x2f0 [] khugepaged_scan_mm_slot+0x17e/0x280 [] khugepaged_do_scan+0xbc/0xe0 [] khugepaged_loop+0x5f/0x3c0 [] ? wake_up_bit+0x40/0x40 [] ? khugepaged_loop+0x3c0/0x3c0 [] khugepaged+0x4a/0xb0 [] kthread+0x9e/0xb0 [] kernel_thread_helper+0x4/0x10 [] ? kthread_freezable_should_stop+0x70/0x70 [] ? gs_change+0x13/0x13 BUG: soft lockup - CPU#2 stuck for 22s! [cc1:5311] Modules linked in: CPU 2 Modules linked in: Pid: 5311, comm: cc1 Not tainted 3.4.0-rc4-base+ #1 Red Hat KVM RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x19/0x30 RSP: 0018:ffff880038a99cc0 EFLAGS: 00000286 RAX: ffffea0000ad9d50 RBX: ffffffff8110d7f8 RCX: ffff88003d981680 RDX: ffffea000054b348 RSI: 0000000000000286 RDI: 0000000000000286 RBP: ffff880038a99cc8 R08: ffffea000054b348 R09: 0000000000000000 R10: ffff88003e7fdf88 R11: 0000000000000000 R12: 0000000000000297 R13: ffff880038a99c88 R14: ffff88003d695258 R15: 0000000000000297 FS: 0000000000000000(0000) GS:ffff88003d680000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000003cddcaae90 CR3: 0000000001a0b000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process cc1 (pid: 5311, threadinfo ffff880038a98000, task ffff88003dc90a70) Stack: 000000000000000e ffff880038a99d48 ffffffff8110d7ed ffff880038a99d78 ffffea0000ad9d50 ffff88003d981680 0000000000000005 0000000000000286 0000000200000000 ffffea0000ad9d50 ffffea000096e760 0000000000000000 Call Trace: [] release_pages+0x1ad/0x210 [] free_pages_and_swap_cache+0xb4/0xe0 [] ? cpumask_any_but+0x31/0x50 [] tlb_flush_mmu+0x6c/0x90 [] tlb_finish_mmu+0x14/0x40 [] exit_mmap+0xd8/0x140 [] mmput+0x73/0x120 [] exit_mm+0x10a/0x140 [] do_exit+0x16b/0x450 [] do_group_exit+0x55/0xd0 [] sys_exit_group+0x17/0x20 [] system_call_fastpath+0x16/0x1b Code: 66 90 e8 db b9 a4 ff 66 90 c9 c3 0f 1f 80 00 00 00 00 55 48 89 e5 53 66 66 66 66 90 48 89 f3 e8 1e ba a4 ff 66 90 48 89 df 57 9d <66> 66 90 66 90 5b c9 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 Call Trace: [] release_pages+0x1ad/0x210 [] free_pages_and_swap_cache+0xb4/0xe0 [] ? cpumask_any_but+0x31/0x50 [] tlb_flush_mmu+0x6c/0x90 [] tlb_finish_mmu+0x14/0x40 [] exit_mmap+0xd8/0x140 [] mmput+0x73/0x120 [] exit_mm+0x10a/0x140 [] do_exit+0x16b/0x450 [] do_group_exit+0x55/0xd0 [] sys_exit_group+0x17/0x20 [] system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#4 stuck for 22s! [cc1:5296] Modules linked in: CPU 4 Modules linked in: Pid: 5296, comm: cc1 Not tainted 3.4.0-rc4-base+ #1 Red Hat KVM RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x19/0x30 RSP: 0018:ffff88003b201cc0 EFLAGS: 00000286 RAX: ffffea0000540090 RBX: 0000000001a03025 RCX: ffff88003d981680 RDX: ffffea0000540058 RSI: 0000000000000286 RDI: 0000000000000286 RBP: ffff88003b201cc8 R08: ffffea0000540058 R09: 0000000000000000 R10: ffff88003e7fdf70 R11: 0000000000000020 R12: 0000000000000297 R13: ffff88003b201c88 R14: ffff88003d715258 R15: 0000000000000297 BUG: soft lockup - CPU#6 stuck for 22s! [khugepaged:52] Modules linked in: CPU 6 Modules linked in: Pid: 52, comm: khugepaged Not tainted 3.4.0-rc4-base+ #1 Red Hat KVM RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x19/0x30 RSP: 0018:ffff880038189a38 EFLAGS: 00000286 RAX: 0000000000000000 RBX: 0000000000000002 RCX: ffff88003d981680 RDX: ffffea0000592940 RSI: 0000000000000286 RDI: 0000000000000286 RBP: ffff880038189a40 R08: ffffea00007d1fe8 R09: 5000000000000000 R10: ffff88003d981680 R11: 0000000000000000 R12: ffff880038189a00 R13: ffff88003d795258 R14: 0000000000000297 R15: ffffea00009f4460 FS: 0000000000000000(0000) GS:ffff88003d780000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f6ffe87f000 CR3: 0000000001a0b000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process khugepaged (pid: 52, threadinfo ffff880038188000, task ffff880038182b30) Stack: 000000000000000e ffff880038189ac0 ffffffff8110d7ed 0000000000000008 ffffea0000856920 ffff88003d981680 0000000000000009 0000000000000286 0000000200000282 ffffea0000856920 ffffea00007670d8 0000000000000010 Call Trace: [] release_pages+0x1ad/0x210 [] ? add_page_to_unevictable_list+0xd0/0xd0 [] pagevec_lru_move_fn+0xcf/0xf0 [] __pagevec_lru_add+0x17/0x20 [] __lru_cache_add+0x68/0x90 [] ? __mem_cgroup_uncharge_common+0xf9/0x280 [] lru_cache_add_lru+0x2d/0x50 [] putback_lru_page+0x69/0xe0 [] release_pte_page+0x2b/0x40 [] __collapse_huge_page_copy+0x185/0x200 [] ? __collapse_huge_page_isolate+0x116/0x1c0 [] collapse_huge_page+0x327/0x500 [] ? rb_erase+0xce/0x150 [] khugepaged_scan_pmd+0x1c5/0x2f0 [] ? __schedule+0x3de/0x710 [] khugepaged_scan_mm_slot+0x17e/0x280 [] khugepaged_do_scan+0xbc/0xe0 [] khugepaged_loop+0x5f/0x3c0 [] ? wake_up_bit+0x40/0x40 [] ? khugepaged_loop+0x3c0/0x3c0 [] khugepaged+0x4a/0xb0 [] kthread+0x9e/0xb0 [] kernel_thread_helper+0x4/0x10 [] ? kthread_freezable_should_stop+0x70/0x70 [] ? gs_change+0x13/0x13 Code: 66 90 e8 db b9 a4 ff 66 90 c9 c3 0f 1f 80 00 00 00 00 55 48 89 e5 53 66 66 66 66 90 48 89 f3 e8 1e ba a4 ff 66 90 48 89 df 57 9d <66> 66 90 66 90 5b c9 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 Call Trace: [] release_pages+0x1ad/0x210 [] ? add_page_to_unevictable_list+0xd0/0xd0 [] pagevec_lru_move_fn+0xcf/0xf0 [] __pagevec_lru_add+0x17/0x20 [] __lru_cache_add+0x68/0x90 [] ? __mem_cgroup_uncharge_common+0xf9/0x280 [] lru_cache_add_lru+0x2d/0x50 [] putback_lru_page+0x69/0xe0 [] release_pte_page+0x2b/0x40 [] __collapse_huge_page_copy+0x185/0x200 [] ? __collapse_huge_page_isolate+0x116/0x1c0 [] collapse_huge_page+0x327/0x500 [] ? rb_erase+0xce/0x150 [] khugepaged_scan_pmd+0x1c5/0x2f0 [] ? __schedule+0x3de/0x710 [] khugepaged_scan_mm_slot+0x17e/0x280 [] khugepaged_do_scan+0xbc/0xe0 [] khugepaged_loop+0x5f/0x3c0 [] ? wake_up_bit+0x40/0x40 [] ? khugepaged_loop+0x3c0/0x3c0 [] khugepaged+0x4a/0xb0 [] kthread+0x9e/0xb0 [] kernel_thread_helper+0x4/0x10 [] ? kthread_freezable_should_stop+0x70/0x70 [] ? gs_change+0x13/0x13 FS: 0000000000000000(0000) GS:ffff88003d700000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000003cddcaae90 CR3: 0000000001a0b000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process cc1 (pid: 5296, threadinfo ffff88003b200000, task ffff88003b1ed4a0) Stack: 000000000000000e ffff88003b201d48 ffffffff8110d7ed ffff88003b201d78 ffffea0000540090 ffff88003d981680 00000000ffffffff 0000000000000286 0000000200000000 ffffea0000540090 ffffea00004fe3f0 0000000000000000 Call Trace: [] release_pages+0x1ad/0x210 [] free_pages_and_swap_cache+0xb4/0xe0 [] ? cpumask_any_but+0x31/0x50 [] tlb_flush_mmu+0x6c/0x90 [] tlb_finish_mmu+0x14/0x40 [] exit_mmap+0xd8/0x140 [] mmput+0x73/0x120 [] exit_mm+0x10a/0x140 [] do_exit+0x16b/0x450 [] do_group_exit+0x55/0xd0 [] sys_exit_group+0x17/0x20 [] system_call_fastpath+0x16/0x1b Code: 66 90 e8 db b9 a4 ff 66 90 c9 c3 0f 1f 80 00 00 00 00 55 48 89 e5 53 66 66 66 66 90 48 89 f3 e8 1e ba a4 ff 66 90 48 89 df 57 9d <66> 66 90 66 90 5b c9 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 Call Trace: [] release_pages+0x1ad/0x210 [] free_pages_and_swap_cache+0xb4/0xe0 [] ? cpumask_any_but+0x31/0x50 [] tlb_flush_mmu+0x6c/0x90 [] tlb_finish_mmu+0x14/0x40 [] exit_mmap+0xd8/0x140 [] mmput+0x73/0x120 [] exit_mm+0x10a/0x140 [] do_exit+0x16b/0x450 [] do_group_exit+0x55/0xd0 [] sys_exit_group+0x17/0x20 [] system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#3 stuck for 22s! [cc1:5346] Modules linked in: CPU 3 Modules linked in: Pid: 5346, comm: cc1 Not tainted 3.4.0-rc4-base+ #1 Red Hat KVM RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x19/0x30 RSP: 0018:ffff880038a1dcc0 EFLAGS: 00000286 BUG: soft lockup - CPU#7 stuck for 22s! [cc1:5292] Modules linked in: CPU 7 Modules linked in: Pid: 5292, comm: cc1 Not tainted 3.4.0-rc4-base+ #1 Red Hat KVM RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x19/0x30 RSP: 0018:ffff88003b22fcc0 EFLAGS: 00000286 RAX: ffffea000092aa98 RBX: ffffffff8110d7f8 RCX: ffff88003d981680 RDX: ffffea000092aa60 RSI: 0000000000000286 RDI: 0000000000000286 RBP: ffff88003b22fcc8 R08: ffffea000092aa60 R09: 0000000000000000 R10: ffff88003e7fdfa0 R11: 0000000000000000 R12: 0000000000000297 R13: ffff88003b22fc88 R14: ffff88003d7d5258 R15: 0000000000000297 FS: 0000000000000000(0000) GS:ffff88003d7c0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000003cddcaae90 CR3: 0000000001a0b000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process cc1 (pid: 5292, threadinfo ffff88003b22e000, task ffff88003b22d560) Stack: 000000000000000e ffff88003b22fd48 ffffffff8110d7ed ffff88003b22fd78 ffffea000092aa98 ffff88003d981680 0000000000000005 0000000000000286 0000000200000000 ffffea000092aa98 ffffea0000609b90 0000000000000000 Call Trace: [] release_pages+0x1ad/0x210 [] free_pages_and_swap_cache+0xb4/0xe0 [] ? cpumask_any_but+0x31/0x50 [] tlb_flush_mmu+0x6c/0x90 [] tlb_finish_mmu+0x14/0x40 [] exit_mmap+0xd8/0x140 [] mmput+0x73/0x120 [] exit_mm+0x10a/0x140 [] do_exit+0x16b/0x450 [] do_group_exit+0x55/0xd0 [] sys_exit_group+0x17/0x20 [] system_call_fastpath+0x16/0x1b Code: 66 90 e8 db b9 a4 ff 66 90 c9 c3 0f 1f 80 00 00 00 00 55 48 89 e5 53 66 66 66 66 90 48 89 f3 e8 1e ba a4 ff 66 90 48 89 df 57 9d <66> 66 90 66 90 5b c9 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 Call Trace: [] release_pages+0x1ad/0x210 [] free_pages_and_swap_cache+0xb4/0xe0 [] ? cpumask_any_but+0x31/0x50 [] tlb_flush_mmu+0x6c/0x90 [] tlb_finish_mmu+0x14/0x40 [] exit_mmap+0xd8/0x140 [] mmput+0x73/0x120 [] exit_mm+0x10a/0x140 [] do_exit+0x16b/0x450 [] do_group_exit+0x55/0xd0 [] sys_exit_group+0x17/0x20 [] system_call_fastpath+0x16/0x1b RAX: ffffea000086a4c8 RBX: 0000000001a03025 RCX: ffff88003d981680 RDX: ffffea000086a490 RSI: 0000000000000286 RDI: 0000000000000286 RBP: ffff880038a1dcc8 R08: ffffea000086a490 R09: 0000000000000000 R10: ffff88003e7fdfa0 R11: 0000000000000020 R12: 0000000000000297 R13: ffff880038a1dc88 R14: ffff88003d6d5258 R15: 0000000000000297 FS: 00007fc542663700(0000) GS:ffff88003d6c0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000003cddcaae90 CR3: 0000000001a0b000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process cc1 (pid: 5346, threadinfo ffff880038a1c000, task ffff880038200040) Stack: 000000000000000e ffff880038a1dd48 ffffffff8110d7ed 0000000000000020 ffffea000086a4c8 ffff88003d981680 0000000000813c88 0000000000000286 000000023810ce10 ffffea000086a4c8 ffffea000072f030 0000000000000297 Call Trace: [] release_pages+0x1ad/0x210 [] free_pages_and_swap_cache+0xb4/0xe0 [] ? cpumask_any_but+0x31/0x50 [] tlb_flush_mmu+0x6c/0x90 [] tlb_finish_mmu+0x14/0x40 [] exit_mmap+0xd8/0x140 [] mmput+0x73/0x120 [] exit_mm+0x10a/0x140 [] do_exit+0x16b/0x450 [] do_group_exit+0x55/0xd0 [] sys_exit_group+0x17/0x20 [] system_call_fastpath+0x16/0x1b Code: 66 90 e8 db b9 a4 ff 66 90 c9 c3 0f 1f 80 00 00 00 00 55 48 89 e5 53 66 66 66 66 90 48 89 f3 e8 1e ba a4 ff 66 90 48 89 df 57 9d <66> 66 90 66 90 5b c9 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 Call Trace: [] release_pages+0x1ad/0x210 [] free_pages_and_swap_cache+0xb4/0xe0 [] ? cpumask_any_but+0x31/0x50 [] tlb_flush_mmu+0x6c/0x90 [] tlb_finish_mmu+0x14/0x40 [] exit_mmap+0xd8/0x140 [] mmput+0x73/0x120 [] exit_mm+0x10a/0x140 [] do_exit+0x16b/0x450 [] do_group_exit+0x55/0xd0 [] sys_exit_group+0x17/0x20 [] system_call_fastpath+0x16/0x1b BUG: soft lockup - CPU#5 stuck for 22s! [khugepaged:52] Modules linked in: CPU 5 Modules linked in: Pid: 52, comm: khugepaged Not tainted 3.4.0-rc4-base+ #1 Red Hat KVM RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x19/0x30 RSP: 0000:ffff880038189b18 EFLAGS: 00000282 RAX: 000000000001529a RBX: 0000000000000010 RCX: ffffffffffffffe0 RDX: fffffffffffffff5 RSI: 0000000000000282 RDI: 0000000000000282 RBP: ffff880038189b20 R08: 00000000fffffff6 R09: 0000000000000000 R10: ffff88003e7fdfb8 R11: 0000000000000000 R12: ffffea000073b858 R13: 0000000200000282 R14: 0000000000000286 R15: 0000000000000009 FS: 0000000000000000(0000) GS:ffff88003d740000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007ff4e3946000 CR3: 0000000001a0b000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process khugepaged (pid: 52, threadinfo ffff880038188000, task ffff880038182b30) Stack: ffff88003d981680 ffff880038189b60 ffffffff8110d450 ffff880038189b70 ffffea0000831ad0 ffffea0000831ad0 ffff88003df5b538 0000000000000538 0000000000511488 ffff880038189b80 ffffffff8110d4c6 ffff880038189b80 Call Trace: [] __page_cache_release+0x100/0x130 [] __put_single_page+0x16/0x30 [] put_page+0x25/0x40 [] free_page_and_swap_cache+0x2a/0x60 [] __collapse_huge_page_copy+0x1d1/0x200 [] ? __collapse_huge_page_isolate+0x116/0x1c0 [] collapse_huge_page+0x327/0x500 [] ? rb_erase+0xce/0x150 [] khugepaged_scan_pmd+0x1c5/0x2f0 [] ? __schedule+0x3de/0x710 [] khugepaged_scan_mm_slot+0x17e/0x280 [] khugepaged_do_scan+0xbc/0xe0 [] khugepaged_loop+0x5f/0x3c0 [] ? wake_up_bit+0x40/0x40 [] ? khugepaged_loop+0x3c0/0x3c0 [] khugepaged+0x4a/0xb0 [] kthread+0x9e/0xb0 [] kernel_thread_helper+0x4/0x10 [] ? kthread_freezable_should_stop+0x70/0x70 [] ? gs_change+0x13/0x13 Code: 66 90 e8 db b9 a4 ff 66 90 c9 c3 0f 1f 80 00 00 00 00 55 48 89 e5 53 66 66 66 66 90 48 89 f3 e8 1e ba a4 ff 66 90 48 89 df 57 9d <66> 66 90 66 90 5b c9 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 Call Trace: [] __page_cache_release+0x100/0x130 [] __put_single_page+0x16/0x30 [] put_page+0x25/0x40 [] free_page_and_swap_cache+0x2a/0x60 [] __collapse_huge_page_copy+0x1d1/0x200 [] ? __collapse_huge_page_isolate+0x116/0x1c0 [] collapse_huge_page+0x327/0x500 [] ? rb_erase+0xce/0x150 [] khugepaged_scan_pmd+0x1c5/0x2f0 [] ? __schedule+0x3de/0x710 [] khugepaged_scan_mm_slot+0x17e/0x280 [] khugepaged_do_scan+0xbc/0xe0 [] khugepaged_loop+0x5f/0x3c0 [] ? wake_up_bit+0x40/0x40 [] ? khugepaged_loop+0x3c0/0x3c0 [] khugepaged+0x4a/0xb0 [] kthread+0x9e/0xb0 [] kernel_thread_helper+0x4/0x10 [] ? kthread_freezable_should_stop+0x70/0x70 [] ? gs_change+0x13/0x13 parted: sending ioctl 1261 to a partition! parted: sending ioctl 1261 to a partition! parted: sending ioctl 1261 to a partition! parted: sending ioctl 1261 to a partition! parted: sending ioctl 1261 to a partition! parted: sending ioctl 1261 to a partition! parted: sending ioctl 1261 to a partition! parted: sending ioctl 1261 to a partition! parted: sending ioctl 1261 to a partition! BUG: soft lockup - CPU#3 stuck for 23s! [cc1:5744] Modules linked in: CPU 3 Modules linked in: Pid: 5744, comm: cc1 Not tainted 3.4.0-rc4-base+ #1 Red Hat KVM RIP: 0010:[] BUG: soft lockup - CPU#5 stuck for 23s! [khugepaged:52] Modules linked in: CPU 5 Modules linked in: Pid: 52, comm: khugepaged Not tainted 3.4.0-rc4-base+ #1 Red Hat KVM RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x19/0x30 RSP: 0000:ffff880038189ab8 EFLAGS: 00000282 RAX: 0000000000000282 RBX: 000000000000000e RCX: ffffc90000182000 RDX: 0000000000000009 RSI: 0000000000000282 RDI: 0000000000000282 RBP: ffff880038189ac0 R08: 00000000fffffff2 R09: fa00000000000000 R10: ffff88003d981680 R11: 0000000000000000 R12: ffff88003d74f1e0 R13: 000000000000000e R14: 0000000000000010 R15: ffffffff81109eaf FS: 0000000000000000(0000) GS:ffff88003d740000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007fff8827c908 CR3: 0000000001a0b000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process khugepaged (pid: 52, threadinfo ffff880038188000, task ffff880038182b30) Stack: ffff88003d74f160 ffff880038189b10 ffffffff8110d90f 0000000000000282 0000000000000001 ffffea0000629798 ffffea00008d0fe8 ffff88003d74f160 0000000000000000 00000000ffffffff 000000000048b8b8 ffff880038189b20 Call Trace: [] pagevec_lru_move_fn+0xbf/0xf0 [] __pagevec_lru_add+0x17/0x20 [] __lru_cache_add+0x68/0x90 [] ? __mem_cgroup_uncharge_common+0xf9/0x280 [] lru_cache_add_lru+0x2d/0x50 [] putback_lru_page+0x69/0xe0 [] release_pte_page+0x2b/0x40 [] __collapse_huge_page_copy+0x185/0x200 [] ? __collapse_huge_page_isolate+0x116/0x1c0 [] collapse_huge_page+0x327/0x500 [] ? rb_erase+0xce/0x150 [] khugepaged_scan_pmd+0x1c5/0x2f0 [] khugepaged_scan_mm_slot+0x17e/0x280 [] khugepaged_do_scan+0xbc/0xe0 [] khugepaged_loop+0x5f/0x3c0 [] ? wake_up_bit+0x40/0x40 [] ? khugepaged_loop+0x3c0/0x3c0 [] khugepaged+0x4a/0xb0 [] kthread+0x9e/0xb0 [] kernel_thread_helper+0x4/0x10 [] ? kthread_freezable_should_stop+0x70/0x70 [] ? gs_change+0x13/0x13 Code: 66 90 e8 db b9 a4 ff 66 90 c9 c3 0f 1f 80 00 00 00 00 55 48 89 e5 53 66 66 66 66 90 48 89 f3 e8 1e ba a4 ff 66 90 48 89 df 57 9d <66> 66 90 66 90 5b c9 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 Call Trace: [] pagevec_lru_move_fn+0xbf/0xf0 [] __pagevec_lru_add+0x17/0x20 [] __lru_cache_add+0x68/0x90 [] ? __mem_cgroup_uncharge_common+0xf9/0x280 [] lru_cache_add_lru+0x2d/0x50 [] putback_lru_page+0x69/0xe0 [] release_pte_page+0x2b/0x40 [] __collapse_huge_page_copy+0x185/0x200 [] ? __collapse_huge_page_isolate+0x116/0x1c0 [] collapse_huge_page+0x327/0x500 [] ? rb_erase+0xce/0x150 [] khugepaged_scan_pmd+0x1c5/0x2f0 [] khugepaged_scan_mm_slot+0x17e/0x280 [] khugepaged_do_scan+0xbc/0xe0 [] khugepaged_loop+0x5f/0x3c0 [] ? wake_up_bit+0x40/0x40 [] ? khugepaged_loop+0x3c0/0x3c0 [] khugepaged+0x4a/0xb0 [] kthread+0x9e/0xb0 [] kernel_thread_helper+0x4/0x10 [] ? kthread_freezable_should_stop+0x70/0x70 [] ? gs_change+0x13/0x13 [] _raw_spin_unlock_irqrestore+0x19/0x30 RSP: 0018:ffff88003928bcc0 EFLAGS: 00000286 RAX: ffffea00007849c8 RBX: 0000000001a03025 RCX: ffff88003d981680 RDX: ffffea0000784990 RSI: 0000000000000286 RDI: 0000000000000286 RBP: ffff88003928bcc8 R08: ffffea0000784990 R09: 0000000000000000 R10: ffff88003e7fdf88 R11: 0000000000000020 R12: 0000000000000297 R13: ffff88003928bc88 R14: ffff88003d6d5258 R15: 0000000000000297 FS: 0000000000000000(0000) GS:ffff88003d6c0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000003cddcaae90 CR3: 0000000001a0b000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process cc1 (pid: 5744, threadinfo ffff88003928a000, task ffff88003dd46040) Stack: 000000000000000e ffff88003928bd48 ffffffff8110d7ed ffff88003928bd78 ffffea00007849c8 ffff88003d981680 00000000ffffffff 0000000000000286 0000000200000000 ffffea00007849c8 ffffea00006ce028 0000000000000000 Call Trace: [] release_pages+0x1ad/0x210 [] free_pages_and_swap_cache+0xb4/0xe0 [] ? cpumask_any_but+0x31/0x50 [] tlb_flush_mmu+0x6c/0x90 [] tlb_finish_mmu+0x14/0x40 [] exit_mmap+0xd8/0x140 [] mmput+0x73/0x120 [] exit_mm+0x10a/0x140 [] do_exit+0x16b/0x450 [] do_group_exit+0x55/0xd0 [] sys_exit_group+0x17/0x20 [] system_call_fastpath+0x16/0x1b Code: 66 90 e8 db b9 a4 ff 66 90 c9 c3 0f 1f 80 00 00 00 00 55 48 89 e5 53 66 66 66 66 90 48 89 f3 e8 1e ba a4 ff 66 90 48 89 df 57 9d <66> 66 90 66 90 5b c9 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 Call Trace: [] release_pages+0x1ad/0x210 [] free_pages_and_swap_cache+0xb4/0xe0 [] ? cpumask_any_but+0x31/0x50 [] tlb_flush_mmu+0x6c/0x90 [] tlb_finish_mmu+0x14/0x40 [] exit_mmap+0xd8/0x140 [] mmput+0x73/0x120 [] exit_mm+0x10a/0x140 [