Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751503AbaLSO5V (ORCPT ); Fri, 19 Dec 2014 09:57:21 -0500 Received: from mx1.redhat.com ([209.132.183.28]:43482 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751279AbaLSO5N (ORCPT ); Fri, 19 Dec 2014 09:57:13 -0500 Date: Fri, 19 Dec 2014 09:55:28 -0500 From: Dave Jones To: Linus Torvalds Cc: Chris Mason , Mike Galbraith , Ingo Molnar , Peter Zijlstra , =?iso-8859-1?Q?D=E2niel?= Fraga , Sasha Levin , "Paul E. McKenney" , Linux Kernel Mailing List , Suresh Siddha , Oleg Nesterov , Peter Anvin Subject: Re: frequent lockups in 3.18rc4 Message-ID: <20141219145528.GC13404@redhat.com> Mail-Followup-To: Dave Jones , Linus Torvalds , Chris Mason , Mike Galbraith , Ingo Molnar , Peter Zijlstra , =?iso-8859-1?Q?D=E2niel?= Fraga , Sasha Levin , "Paul E. McKenney" , Linux Kernel Mailing List , Suresh Siddha , Oleg Nesterov , Peter Anvin References: <20141218051327.GA31988@redhat.com> <1418918059.17358.6@mail.thefacebook.com> <20141218161230.GA6042@redhat.com> <20141219024549.GB1671@redhat.com> <20141219035859.GA20022@redhat.com> <20141219040308.GB20022@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Dec 18, 2014 at 08:48:24PM -0800, Linus Torvalds wrote: > On Thu, Dec 18, 2014 at 8:03 PM, Dave Jones wrote: > > > > So the only thing that was on that could cause spinlock overhead > > was DEBUG_SPINLOCK (and LOCK_STAT, though iirc that's not huge either) > > So DEBUG_SPINLOCK does have one big downside if I recall correctly - > the debugging spinlocks are very much not fair. So they don't work > like the real ticket spinlocks. That might have serious effects on the > contention case, with some thread not making any progress due to just > the implementation of the debug spinlocks. Wish DEBUG_SPINLOCK disabled, I see the same behaviour. Lots of traces spewed, but it seems to run and run (at least so far). Dave [24053.092097] trinity-c206 (26510) used greatest stack depth: 8888 bytes left [24998.017355] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [trinity-c205:636] [24998.017457] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc [24998.019523] CPU: 1 PID: 636 Comm: trinity-c205 Not tainted 3.18.0+ #108 [24998.021412] task: ffff8800962fed60 ti: ffff880242678000 task.ti: ffff880242678000 [24998.022367] RIP: 0010:[] [] generic_exec_single+0xea/0x1b0 [24998.023332] RSP: 0018:ffff88024267b9c8 EFLAGS: 00000202 [24998.024287] RAX: 0000000000000008 RBX: ffffffff81799e0d RCX: 0000000000000038 [24998.025245] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000 [24998.026194] RBP: ffff88024267ba28 R08: ffff8802444f43f0 R09: 0000000000000000 [24998.027143] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88024267b938 [24998.028144] R13: 0000000000406040 R14: ffff880242678000 R15: ffff8800962fed60 [24998.029095] FS: 00007ff08e78d740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000 [24998.030050] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24998.031005] CR2: 0000000000000000 CR3: 0000000228031000 CR4: 00000000001407e0 [24998.031967] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [24998.032930] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [24998.033898] Stack: [24998.034855] ffffffff8124055d ffff8801b4eff080 ffffffff8124055d 0000000000000000 [24998.035843] ffffffff81047bb0 ffff88024267bad8 0000000000000003 0000000077930768 [24998.036832] ffff88024267ba48 00000000ffffffff 0000000000000000 ffffffff81047bb0 [24998.037855] Call Trace: [24998.038837] [] ? proc_alloc_inode+0x1d/0xb0 [24998.039826] [] ? proc_alloc_inode+0x1d/0xb0 [24998.040804] [] ? do_flush_tlb_all+0x60/0x60 [24998.041779] [] ? do_flush_tlb_all+0x60/0x60 [24998.042748] [] smp_call_function_single+0x70/0xd0 [24998.043705] [] ? release_pages+0x1c2/0x280 [24998.044647] [] ? do_flush_tlb_all+0x60/0x60 [24998.045585] [] smp_call_function_many+0x2b9/0x320 [24998.046528] [] flush_tlb_mm_range+0x90/0x1b0 [24998.047510] [] tlb_flush_mmu_tlbonly+0x42/0x50 [24998.048456] [] unmap_single_vma+0x6d8/0x900 [24998.049409] [] ? __free_pages+0x37/0x50 [24998.050361] [] zap_page_range_single+0xfc/0x160 [24998.051309] [] unmap_mapping_range+0x12f/0x190 [24998.052260] [] shmem_fallocate+0x38d/0x4c0 [24998.053209] [] ? cmpxchg_double_slab.isra.49+0xd8/0x122 [24998.054156] [] do_fallocate+0x12f/0x1d0 [24998.055100] [] SyS_madvise+0x385/0x860 [24998.056044] [] ? syscall_trace_enter_phase2+0xa7/0x1e0 [24998.056992] [] tracesys_phase2+0xd4/0xd9 [24998.057963] Code: 00 2b 01 00 48 89 de 48 03 14 c5 e0 bf cf 81 48 89 df e8 5a eb 26 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 43 18 01 75 f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 [24998.059993] sending NMI to other CPUs: [24998.060963] NMI backtrace for cpu 0 [24998.061989] CPU: 0 PID: 2940 Comm: trinity-c150 Not tainted 3.18.0+ #108 [24998.064073] task: ffff8801bf3536b0 ti: ffff880197e0c000 task.ti: ffff880197e0c000 [24998.065137] RIP: 0010:[] [] read_hpet+0x16/0x20 [24998.066226] RSP: 0018:ffff88024e203e38 EFLAGS: 00000046 [24998.067310] RAX: 0000000061fece8a RBX: 0000000000510792 RCX: 0000000000000000 [24998.068407] RDX: 0000000000000000 RSI: ffff88024e20c710 RDI: ffffffff81c26f40 [24998.069491] RBP: ffff88024e203e38 R08: 0000000000000000 R09: 000000000000000f [24998.070569] R10: 0000000000000526 R11: 000000000000000f R12: 000016bf99600917 [24998.071642] R13: 0000000000000000 R14: ffff88024e20c700 R15: 000016eacc784674 [24998.072713] FS: 00007ff08e78d740(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000 [24998.073771] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24998.074813] CR2: 0000000000000001 CR3: 0000000229d22000 CR4: 00000000001407f0 [24998.075855] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [24998.076891] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [24998.077903] Stack: [24998.078886] ffff88024e203e68 ffffffff810e0d3e ffff88024e20ca80 ffff88024e20ca80 [24998.079872] ffff880197e0fe38 ffff88024e20c6c0 ffff88024e203e98 ffffffff810e9cd3 [24998.080832] ffff88024e203f28 ffff88024e20ca80 ffff88024e203f28 ffff88024e20c6c0 [24998.081769] Call Trace: [24998.082680] [24998.083577] [] ktime_get+0x3e/0xa0 [24998.084450] [] tick_sched_timer+0x23/0x160 [24998.085315] [] __run_hrtimer+0x76/0x1f0 [24998.086173] [] ? tick_init_highres+0x20/0x20 [24998.087025] [] hrtimer_interrupt+0x107/0x260 [24998.087877] [] local_apic_timer_interrupt+0x3b/0x70 [24998.088732] [] smp_apic_timer_interrupt+0x45/0x60 [24998.089583] [] apic_timer_interrupt+0x6f/0x80 [24998.090435] [24998.091279] [] ? __remove_hrtimer+0x4e/0xa0 [24998.092118] [] ? ipcget+0x8a/0x1e0 [24998.092951] [] ? ipcget+0x7c/0x1e0 [24998.093779] [] SyS_msgget+0x4d/0x70 [24998.094605] [] tracesys_phase2+0xd4/0xd9 [24998.095420] Code: 00 29 c7 ba 00 00 00 00 b8 c2 ff ff ff 83 ff 7f 5d 0f 4f c2 c3 0f 1f 44 00 00 55 48 8b 05 d3 c8 ec 00 48 89 e5 8b 80 f0 00 00 00 <89> c0 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 8b 0d 29 c8 ec 00 [24998.097220] NMI backtrace for cpu 3 [24998.098074] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.18.0+ #108 [24998.099812] task: ffff8802444fe270 ti: ffff880244718000 task.ti: ffff880244718000 [24998.100708] RIP: 0010:[] [] intel_idle+0xdb/0x180 [24998.101617] RSP: 0018:ffff88024471be08 EFLAGS: 00000046 [24998.102529] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001 [24998.103448] RDX: 0000000000000000 RSI: ffff88024471bfd8 RDI: 0000000000000003 [24998.104364] RBP: ffff88024471be38 R08: 0000000000000010 R09: 000000000000265a [24998.105283] R10: 0000000000007d5b R11: 00000000000003ff R12: 0000000000000005 [24998.106201] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880244718000 [24998.107115] FS: 0000000000000000(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000 [24998.108039] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24998.108962] CR2: 00007fda8919e000 CR3: 0000000001c11000 CR4: 00000000001407e0 [24998.109894] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [24998.110820] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [24998.111747] Stack: [24998.112668] 000000034471be38 5fbbf5c12849ac78 ffffe8ffffcc1ba8 0000000000000005 [24998.113610] ffffffff81c9e440 0000000000000003 ffff88024471be88 ffffffff8163a995 [24998.114553] 000016bfb0c185fc ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330 [24998.115501] Call Trace: [24998.116440] [] cpuidle_enter_state+0x55/0x190 [24998.117389] [] cpuidle_enter+0x17/0x20 [24998.118309] [] cpu_startup_entry+0x355/0x410 [24998.119208] [] start_secondary+0x1aa/0x230 [24998.120106] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 [24998.122074] NMI backtrace for cpu 2 [24998.122981] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.18.0+ #108 [24998.124771] task: ffff8802444f8af0 ti: ffff88024470c000 task.ti: ffff88024470c000 [24998.125679] RIP: 0010:[] [] intel_idle+0xdb/0x180 [24998.126593] RSP: 0018:ffff88024470fe08 EFLAGS: 00000046 [24998.127528] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001 [24998.128444] RDX: 0000000000000000 RSI: ffff88024470ffd8 RDI: 0000000000000002 [24998.129357] RBP: ffff88024470fe38 R08: ffff88024e28ecf0 R09: 00000000ffffffff [24998.130267] R10: 000000000000265c R11: 00000000000003ff R12: 0000000000000005 [24998.131172] R13: 0000000000000032 R14: 0000000000000004 R15: ffff88024470c000 [24998.132071] FS: 0000000000000000(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000 [24998.132974] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [24998.133876] CR2: 00007fda8919e000 CR3: 0000000001c11000 CR4: 00000000001407e0 [24998.134789] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [24998.135694] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [24998.136589] Stack: [24998.137505] 000000024470fe38 c6aa8015cfd0a8b5 ffffe8ffffc81ba8 0000000000000005 [24998.138424] ffffffff81c9e440 0000000000000002 ffff88024470fe88 ffffffff8163a995 [24998.139340] 000016bfb0c18159 ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330 [24998.140262] Call Trace: [24998.141176] [] cpuidle_enter_state+0x55/0x190 [24998.142103] [] cpuidle_enter+0x17/0x20 [24998.143026] [] cpu_startup_entry+0x355/0x410 [24998.143948] [] start_secondary+0x1aa/0x230 [24998.144868] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 [25026.001132] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [trinity-c205:636] [25026.002121] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc [25026.007365] CPU: 1 PID: 636 Comm: trinity-c205 Tainted: G L 3.18.0+ #108 [25026.009524] task: ffff8800962fed60 ti: ffff880242678000 task.ti: ffff880242678000 [25026.010591] RIP: 0010:[] [] generic_exec_single+0xee/0x1b0 [25026.011710] RSP: 0018:ffff88024267b9c8 EFLAGS: 00000202 [25026.012787] RAX: 0000000000000008 RBX: ffffffff81799e0d RCX: 0000000000000038 [25026.013854] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000 [25026.014902] RBP: ffff88024267ba28 R08: ffff8802444f43f0 R09: 0000000000000000 [25026.015941] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88024267b938 [25026.016982] R13: 0000000000406040 R14: ffff880242678000 R15: ffff8800962fed60 [25026.018019] FS: 00007ff08e78d740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000 [25026.019065] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25026.020113] CR2: 0000000000000000 CR3: 0000000228031000 CR4: 00000000001407e0 [25026.021205] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25026.022263] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25026.023310] Stack: [25026.024349] ffffffff8124055d ffff8801b4eff080 ffffffff8124055d 0000000000000000 [25026.025411] ffffffff81047bb0 ffff88024267bad8 0000000000000003 0000000077930768 [25026.026472] ffff88024267ba48 00000000ffffffff 0000000000000000 ffffffff81047bb0 [25026.027535] Call Trace: [25026.028585] [] ? proc_alloc_inode+0x1d/0xb0 [25026.029640] [] ? proc_alloc_inode+0x1d/0xb0 [25026.030683] [] ? do_flush_tlb_all+0x60/0x60 [25026.031747] [] ? do_flush_tlb_all+0x60/0x60 [25026.032771] [] smp_call_function_single+0x70/0xd0 [25026.033800] [] ? release_pages+0x1c2/0x280 [25026.034826] [] ? do_flush_tlb_all+0x60/0x60 [25026.035854] [] smp_call_function_many+0x2b9/0x320 [25026.036883] [] flush_tlb_mm_range+0x90/0x1b0 [25026.037896] [] tlb_flush_mmu_tlbonly+0x42/0x50 [25026.038886] [] unmap_single_vma+0x6d8/0x900 [25026.039875] [] ? __free_pages+0x37/0x50 [25026.040865] [] zap_page_range_single+0xfc/0x160 [25026.041867] [] unmap_mapping_range+0x12f/0x190 [25026.042818] [] shmem_fallocate+0x38d/0x4c0 [25026.043763] [] ? cmpxchg_double_slab.isra.49+0xd8/0x122 [25026.044707] [] do_fallocate+0x12f/0x1d0 [25026.045647] [] SyS_madvise+0x385/0x860 [25026.046587] [] ? syscall_trace_enter_phase2+0xa7/0x1e0 [25026.047529] [] tracesys_phase2+0xd4/0xd9 [25026.048469] Code: 48 89 de 48 03 14 c5 e0 bf cf 81 48 89 df e8 5a eb 26 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 f6 43 18 01 <75> f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 0f 85 98 00 [25026.050509] sending NMI to other CPUs: [25026.051516] NMI backtrace for cpu 0 [25026.052541] CPU: 0 PID: 2940 Comm: trinity-c150 Tainted: G L 3.18.0+ #108 [25026.054635] task: ffff8801bf3536b0 ti: ffff880197e0c000 task.ti: ffff880197e0c000 [25026.055696] RIP: 0010:[] [] read_hpet+0x16/0x20 [25026.056761] RSP: 0018:ffff88024e203e38 EFLAGS: 00000046 [25026.057822] RAX: 0000000079e588fc RBX: 0000000000511d6e RCX: 0000000000000000 [25026.058886] RDX: 0000000000000000 RSI: ffff88024e20c710 RDI: ffffffff81c26f40 [25026.059947] RBP: ffff88024e203e38 R08: 0000000000000000 R09: 000000000000000f [25026.061009] R10: 0000000000000526 R11: 000000000000000f R12: 000016c61e4e2117 [25026.062073] R13: 0000000000000000 R14: ffff88024e20c700 R15: 000016eacc784674 [25026.063139] FS: 00007ff08e78d740(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000 [25026.064215] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25026.065271] CR2: 0000000000000001 CR3: 0000000229d22000 CR4: 00000000001407f0 [25026.066314] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25026.067349] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25026.068359] Stack: [25026.069339] ffff88024e203e68 ffffffff810e0d3e ffff88024e20ca80 ffff88024e20ca80 [25026.070326] ffff880197e0fe38 ffff88024e20c6c0 ffff88024e203e98 ffffffff810e9cd3 [25026.071289] ffff88024e203f28 ffff88024e20ca80 ffff88024e203f28 ffff88024e20c6c0 [25026.072225] Call Trace: [25026.073133] [25026.074028] [] ktime_get+0x3e/0xa0 [25026.074902] [] tick_sched_timer+0x23/0x160 [25026.075767] [] __run_hrtimer+0x76/0x1f0 [25026.076622] [] ? tick_init_highres+0x20/0x20 [25026.077476] [] hrtimer_interrupt+0x107/0x260 [25026.078328] [] local_apic_timer_interrupt+0x3b/0x70 [25026.079183] [] smp_apic_timer_interrupt+0x45/0x60 [25026.080035] [] apic_timer_interrupt+0x6f/0x80 [25026.080885] [25026.081728] [] ? __remove_hrtimer+0x4e/0xa0 [25026.082565] [] ? ipcget+0x8a/0x1e0 [25026.083397] [] ? ipcget+0x7c/0x1e0 [25026.084226] [] SyS_msgget+0x4d/0x70 [25026.085050] [] tracesys_phase2+0xd4/0xd9 [25026.085867] Code: 00 29 c7 ba 00 00 00 00 b8 c2 ff ff ff 83 ff 7f 5d 0f 4f c2 c3 0f 1f 44 00 00 55 48 8b 05 d3 c8 ec 00 48 89 e5 8b 80 f0 00 00 00 <89> c0 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 8b 0d 29 c8 ec 00 [25026.087667] NMI backtrace for cpu 3 [25026.088525] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G L 3.18.0+ #108 [25026.090268] task: ffff8802444fe270 ti: ffff880244718000 task.ti: ffff880244718000 [25026.091168] RIP: 0010:[] [] intel_idle+0xdb/0x180 [25026.092081] RSP: 0018:ffff88024471be08 EFLAGS: 00000046 [25026.092994] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001 [25026.093915] RDX: 0000000000000000 RSI: ffff88024471bfd8 RDI: 0000000000000003 [25026.094834] RBP: ffff88024471be38 R08: ffff88024e2ced08 R09: 00000000ffffffff [25026.095754] R10: 000000000000265a R11: 00000000000003ff R12: 0000000000000005 [25026.096677] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880244718000 [25026.097598] FS: 0000000000000000(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000 [25026.098526] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25026.099454] CR2: 00007fffcba6fea0 CR3: 0000000001c11000 CR4: 00000000001407e0 [25026.100392] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25026.101324] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25026.102257] Stack: [25026.103183] 000000034471be38 5fbbf5c12849ac78 ffffe8ffffcc1ba8 0000000000000005 [25026.104133] ffffffff81c9e440 0000000000000003 ffff88024471be88 ffffffff8163a995 [25026.105083] 000016c63648391d ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330 [25026.106036] Call Trace: [25026.106984] [] cpuidle_enter_state+0x55/0x190 [25026.107936] [] cpuidle_enter+0x17/0x20 [25026.108865] [] cpu_startup_entry+0x355/0x410 [25026.109771] [] start_secondary+0x1aa/0x230 [25026.110674] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 [25026.112654] NMI backtrace for cpu 2 [25026.113565] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G L 3.18.0+ #108 [25026.115375] task: ffff8802444f8af0 ti: ffff88024470c000 task.ti: ffff88024470c000 [25026.116284] RIP: 0010:[] [] intel_idle+0xdb/0x180 [25026.117207] RSP: 0018:ffff88024470fe08 EFLAGS: 00000046 [25026.118123] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001 [25026.119041] RDX: 0000000000000000 RSI: ffff88024470ffd8 RDI: 0000000000000002 [25026.119958] RBP: ffff88024470fe38 R08: ffff88024e28ed08 R09: 00000000ffffffff [25026.120875] R10: 0000000000002659 R11: 00000000000003ff R12: 0000000000000005 [25026.121814] R13: 0000000000000032 R14: 0000000000000004 R15: ffff88024470c000 [25026.122719] FS: 0000000000000000(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000 [25026.123625] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25026.124533] CR2: 00007fda8919e000 CR3: 0000000001c11000 CR4: 00000000001407e0 [25026.125450] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25026.126361] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25026.127262] Stack: [25026.128157] 000000024470fe38 c6aa8015cfd0a8b5 ffffe8ffffc81ba8 0000000000000005 [25026.129079] ffffffff81c9e440 0000000000000002 ffff88024470fe88 ffffffff8163a995 [25026.129996] 000016c636483734 ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330 [25026.130920] Call Trace: [25026.131864] [] cpuidle_enter_state+0x55/0x190 [25026.132794] [] cpuidle_enter+0x17/0x20 [25026.133725] [] cpu_startup_entry+0x355/0x410 [25026.134652] [] start_secondary+0x1aa/0x230 [25026.135579] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 [25037.184648] INFO: rcu_sched detected stalls on CPUs/tasks: [25037.185608] (detected by 0, t=6002 jiffies, g=777861, c=777860, q=0) [25037.186536] INFO: Stall ended before state dump start [25053.984908] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c205:636] [25053.985884] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc [25053.991028] CPU: 1 PID: 636 Comm: trinity-c205 Tainted: G L 3.18.0+ #108 [25053.993097] task: ffff8800962fed60 ti: ffff880242678000 task.ti: ffff880242678000 [25053.994131] RIP: 0010:[] [] generic_exec_single+0xea/0x1b0 [25053.995202] RSP: 0018:ffff88024267b9c8 EFLAGS: 00000202 [25053.996223] RAX: 0000000000000008 RBX: ffffffff81799e0d RCX: 0000000000000038 [25053.997247] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000 [25053.998270] RBP: ffff88024267ba28 R08: ffff8802444f43f0 R09: 0000000000000000 [25053.999294] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88024267b938 [25054.000323] R13: 0000000000406040 R14: ffff880242678000 R15: ffff8800962fed60 [25054.001349] FS: 00007ff08e78d740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000 [25054.002384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25054.003420] CR2: 0000000000000000 CR3: 0000000228031000 CR4: 00000000001407e0 [25054.004455] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25054.005522] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25054.006552] Stack: [25054.007575] ffffffff8124055d ffff8801b4eff080 ffffffff8124055d 0000000000000000 [25054.008619] ffffffff81047bb0 ffff88024267bad8 0000000000000003 0000000077930768 [25054.009658] ffff88024267ba48 00000000ffffffff 0000000000000000 ffffffff81047bb0 [25054.010692] Call Trace: [25054.011716] [] ? proc_alloc_inode+0x1d/0xb0 [25054.012743] [] ? proc_alloc_inode+0x1d/0xb0 [25054.013763] [] ? do_flush_tlb_all+0x60/0x60 [25054.014779] [] ? do_flush_tlb_all+0x60/0x60 [25054.015830] [] smp_call_function_single+0x70/0xd0 [25054.016846] [] ? release_pages+0x1c2/0x280 [25054.017846] [] ? do_flush_tlb_all+0x60/0x60 [25054.018823] [] smp_call_function_many+0x2b9/0x320 [25054.019798] [] flush_tlb_mm_range+0x90/0x1b0 [25054.020774] [] tlb_flush_mmu_tlbonly+0x42/0x50 [25054.021732] [] unmap_single_vma+0x6d8/0x900 [25054.022671] [] ? __free_pages+0x37/0x50 [25054.023604] [] zap_page_range_single+0xfc/0x160 [25054.024534] [] unmap_mapping_range+0x12f/0x190 [25054.025495] [] shmem_fallocate+0x38d/0x4c0 [25054.026421] [] ? cmpxchg_double_slab.isra.49+0xd8/0x122 [25054.027353] [] do_fallocate+0x12f/0x1d0 [25054.028281] [] SyS_madvise+0x385/0x860 [25054.029206] [] ? syscall_trace_enter_phase2+0xa7/0x1e0 [25054.030136] [] tracesys_phase2+0xd4/0xd9 [25054.031054] Code: 00 2b 01 00 48 89 de 48 03 14 c5 e0 bf cf 81 48 89 df e8 5a eb 26 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 43 18 01 75 f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 [25054.033040] sending NMI to other CPUs: [25054.033992] NMI backtrace for cpu 0 [25054.035027] CPU: 0 PID: 2940 Comm: trinity-c150 Tainted: G L 3.18.0+ #108 [25054.037117] task: ffff8801bf3536b0 ti: ffff880197e0c000 task.ti: ffff880197e0c000 [25054.038172] RIP: 0010:[] [] read_hpet+0x16/0x20 [25054.039238] RSP: 0018:ffff88024e203e38 EFLAGS: 00000046 [25054.040297] RAX: 0000000091ca7f65 RBX: 0000000000513346 RCX: 0000000000000000 [25054.041365] RDX: 0000000000000000 RSI: ffff88024e20c710 RDI: ffffffff81c26f40 [25054.042436] RBP: ffff88024e203e38 R08: 0000000000000000 R09: 000000000000000f [25054.043514] R10: 0000000000000526 R11: 0000000000aaaaaa R12: 000016cca33c3917 [25054.044572] R13: 0000000000000000 R14: ffff88024e20c700 R15: 000016eacc784674 [25054.045606] FS: 00007ff08e78d740(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000 [25054.046645] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25054.047685] CR2: 0000000000000001 CR3: 0000000229d22000 CR4: 00000000001407f0 [25054.048728] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25054.049763] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25054.050774] Stack: [25054.051757] ffff88024e203e68 ffffffff810e0d3e ffff88024e20ca80 ffff88024e20ca80 [25054.052744] ffff880197e0fe38 ffff88024e20c6c0 ffff88024e203e98 ffffffff810e9cd3 [25054.053709] ffff88024e203f28 ffff88024e20ca80 ffff88024e203f28 ffff88024e20c6c0 [25054.054649] Call Trace: [25054.055559] [25054.056455] [] ktime_get+0x3e/0xa0 [25054.057326] [] tick_sched_timer+0x23/0x160 [25054.058194] [] __run_hrtimer+0x76/0x1f0 [25054.059050] [] ? tick_init_highres+0x20/0x20 [25054.059904] [] hrtimer_interrupt+0x107/0x260 [25054.060759] [] local_apic_timer_interrupt+0x3b/0x70 [25054.061614] [] smp_apic_timer_interrupt+0x45/0x60 [25054.062466] [] apic_timer_interrupt+0x6f/0x80 [25054.063317] [25054.064159] [] ? __remove_hrtimer+0x4e/0xa0 [25054.064994] [] ? ipcget+0x8a/0x1e0 [25054.065826] [] ? ipcget+0x7c/0x1e0 [25054.066656] [] SyS_msgget+0x4d/0x70 [25054.067480] [] tracesys_phase2+0xd4/0xd9 [25054.068297] Code: 00 29 c7 ba 00 00 00 00 b8 c2 ff ff ff 83 ff 7f 5d 0f 4f c2 c3 0f 1f 44 00 00 55 48 8b 05 d3 c8 ec 00 48 89 e5 8b 80 f0 00 00 00 <89> c0 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 8b 0d 29 c8 ec 00 [25054.070095] NMI backtrace for cpu 2 [25054.070949] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G L 3.18.0+ #108 [25054.072692] task: ffff8802444f8af0 ti: ffff88024470c000 task.ti: ffff88024470c000 [25054.073589] RIP: 0010:[] [] intel_idle+0xdb/0x180 [25054.074498] RSP: 0018:ffff88024470fe08 EFLAGS: 00000046 [25054.075411] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001 [25054.076329] RDX: 0000000000000000 RSI: ffff88024470ffd8 RDI: 0000000000000002 [25054.077248] RBP: ffff88024470fe38 R08: ffff88024e28ed08 R09: 00000000ffffffff [25054.078167] R10: 000000000000265b R11: 00000000000003ff R12: 0000000000000005 [25054.079087] R13: 0000000000000032 R14: 0000000000000004 R15: ffff88024470c000 [25054.080007] FS: 0000000000000000(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000 [25054.080934] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25054.081856] CR2: 00007fda8919e000 CR3: 0000000001c11000 CR4: 00000000001407e0 [25054.082790] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25054.083721] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25054.084651] Stack: [25054.085578] 000000024470fe38 c6aa8015cfd0a8b5 ffffe8ffffc81ba8 0000000000000005 [25054.086525] ffffffff81c9e440 0000000000000002 ffff88024470fe88 ffffffff8163a995 [25054.087477] 000016ccba9dadfd ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330 [25054.088429] Call Trace: [25054.089371] [] cpuidle_enter_state+0x55/0x190 [25054.090324] [] cpuidle_enter+0x17/0x20 [25054.091251] [] cpu_startup_entry+0x355/0x410 [25054.092158] [] start_secondary+0x1aa/0x230 [25054.093059] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 [25054.095041] NMI backtrace for cpu 3 [25054.095952] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G L 3.18.0+ #108 [25054.097763] task: ffff8802444fe270 ti: ffff880244718000 task.ti: ffff880244718000 [25054.098676] RIP: 0010:[] [] intel_idle+0xdb/0x180 [25054.099598] RSP: 0018:ffff88024471be08 EFLAGS: 00000046 [25054.100517] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001 [25054.101441] RDX: 0000000000000000 RSI: ffff88024471bfd8 RDI: 0000000000000003 [25054.102360] RBP: ffff88024471be38 R08: 0000000000000018 R09: 0000000000002655 [25054.103278] R10: 000000000000be78 R11: 00000000000003ff R12: 0000000000000005 [25054.104189] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880244718000 [25054.105118] FS: 0000000000000000(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000 [25054.106026] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25054.106933] CR2: 00007fda8919e000 CR3: 0000000001c11000 CR4: 00000000001407e0 [25054.107847] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25054.108756] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25054.109656] Stack: [25054.110546] 000000034471be38 5fbbf5c12849ac78 ffffe8ffffcc1ba8 0000000000000005 [25054.111466] ffffffff81c9e440 0000000000000003 ffff88024471be88 ffffffff8163a995 [25054.112383] 000016ccba9dc98b ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330 [25054.113305] Call Trace: [25054.114222] [] cpuidle_enter_state+0x55/0x190 [25054.115173] [] cpuidle_enter+0x17/0x20 [25054.116098] [] cpu_startup_entry+0x355/0x410 [25054.117022] [] start_secondary+0x1aa/0x230 [25054.117945] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 [25081.968684] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c205:636] [25081.969644] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc [25081.974748] CPU: 1 PID: 636 Comm: trinity-c205 Tainted: G L 3.18.0+ #108 [25081.976852] task: ffff8800962fed60 ti: ffff880242678000 task.ti: ffff880242678000 [25081.977892] RIP: 0010:[] [] generic_exec_single+0xea/0x1b0 [25081.978981] RSP: 0018:ffff88024267b9c8 EFLAGS: 00000202 [25081.980030] RAX: 0000000000000008 RBX: ffffffff81799e0d RCX: 0000000000000038 [25081.981068] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000 [25081.982090] RBP: ffff88024267ba28 R08: ffff8802444f43f0 R09: 0000000000000000 [25081.983107] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88024267b938 [25081.984121] R13: 0000000000406040 R14: ffff880242678000 R15: ffff8800962fed60 [25081.985132] FS: 00007ff08e78d740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000 [25081.986155] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25081.987179] CR2: 0000000000000000 CR3: 0000000228031000 CR4: 00000000001407e0 [25081.988208] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25081.989271] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25081.990294] Stack: [25081.991306] ffffffff8124055d ffff8801b4eff080 ffffffff8124055d 0000000000000000 [25081.992338] ffffffff81047bb0 ffff88024267bad8 0000000000000003 0000000077930768 [25081.993369] ffff88024267ba48 00000000ffffffff 0000000000000000 ffffffff81047bb0 [25081.994406] Call Trace: [25081.995430] [] ? proc_alloc_inode+0x1d/0xb0 [25081.996456] [] ? proc_alloc_inode+0x1d/0xb0 [25081.997470] [] ? do_flush_tlb_all+0x60/0x60 [25081.998481] [] ? do_flush_tlb_all+0x60/0x60 [25081.999512] [] smp_call_function_single+0x70/0xd0 [25082.000509] [] ? release_pages+0x1c2/0x280 [25082.001510] [] ? do_flush_tlb_all+0x60/0x60 [25082.002510] [] smp_call_function_many+0x2b9/0x320 [25082.003514] [] flush_tlb_mm_range+0x90/0x1b0 [25082.004498] [] tlb_flush_mmu_tlbonly+0x42/0x50 [25082.005463] [] unmap_single_vma+0x6d8/0x900 [25082.006425] [] ? __free_pages+0x37/0x50 [25082.007386] [] zap_page_range_single+0xfc/0x160 [25082.008335] [] unmap_mapping_range+0x12f/0x190 [25082.009300] [] shmem_fallocate+0x38d/0x4c0 [25082.010222] [] ? cmpxchg_double_slab.isra.49+0xd8/0x122 [25082.011144] [] do_fallocate+0x12f/0x1d0 [25082.012057] [] SyS_madvise+0x385/0x860 [25082.012971] [] ? syscall_trace_enter_phase2+0xa7/0x1e0 [25082.013889] [] tracesys_phase2+0xd4/0xd9 [25082.014807] Code: 00 2b 01 00 48 89 de 48 03 14 c5 e0 bf cf 81 48 89 df e8 5a eb 26 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 43 18 01 75 f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 [25082.016796] sending NMI to other CPUs: [25082.017742] NMI backtrace for cpu 0 [25082.018768] CPU: 0 PID: 2940 Comm: trinity-c150 Tainted: G L 3.18.0+ #108 [25082.020865] task: ffff8801bf3536b0 ti: ffff880197e0c000 task.ti: ffff880197e0c000 [25082.021929] RIP: 0010:[] [] read_hpet+0x16/0x20 [25082.022996] RSP: 0018:ffff88024e203e38 EFLAGS: 00000046 [25082.024057] RAX: 00000000a9afbd0d RBX: 000000000051491e RCX: 0000000000000000 [25082.025125] RDX: 0000000000000000 RSI: ffff88024e20c710 RDI: ffffffff81c26f40 [25082.026185] RBP: ffff88024e203e38 R08: 0000000000000000 R09: 000000000000000f [25082.027248] R10: 0000000000000526 R11: 0000000000aaaaaa R12: 000016d3282a5117 [25082.028317] R13: 0000000000000000 R14: ffff88024e20c700 R15: 000016eacc784674 [25082.029383] FS: 00007ff08e78d740(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000 [25082.030459] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25082.031518] CR2: 0000000000000001 CR3: 0000000229d22000 CR4: 00000000001407f0 [25082.032563] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25082.033598] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25082.034612] Stack: [25082.035595] ffff88024e203e68 ffffffff810e0d3e ffff88024e20ca80 ffff88024e20ca80 [25082.036582] ffff880197e0fe38 ffff88024e20c6c0 ffff88024e203e98 ffffffff810e9cd3 [25082.037546] ffff88024e203f28 ffff88024e20ca80 ffff88024e203f28 ffff88024e20c6c0 [25082.038485] Call Trace: [25082.039397] [25082.040290] [] ktime_get+0x3e/0xa0 [25082.041164] [] tick_sched_timer+0x23/0x160 [25082.042032] [] __run_hrtimer+0x76/0x1f0 [25082.042890] [] ? tick_init_highres+0x20/0x20 [25082.043748] [] hrtimer_interrupt+0x107/0x260 [25082.044603] [] local_apic_timer_interrupt+0x3b/0x70 [25082.045460] [] smp_apic_timer_interrupt+0x45/0x60 [25082.046317] [] apic_timer_interrupt+0x6f/0x80 [25082.047170] [25082.048018] [] ? __remove_hrtimer+0x4e/0xa0 [25082.048856] [] ? ipcget+0x8a/0x1e0 [25082.049691] [] ? ipcget+0x7c/0x1e0 [25082.050520] [] SyS_msgget+0x4d/0x70 [25082.051347] [] tracesys_phase2+0xd4/0xd9 [25082.052166] Code: 00 29 c7 ba 00 00 00 00 b8 c2 ff ff ff 83 ff 7f 5d 0f 4f c2 c3 0f 1f 44 00 00 55 48 8b 05 d3 c8 ec 00 48 89 e5 8b 80 f0 00 00 00 <89> c0 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 8b 0d 29 c8 ec 00 [25082.053966] NMI backtrace for cpu 3 [25082.054824] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G L 3.18.0+ #108 [25082.056568] task: ffff8802444fe270 ti: ffff880244718000 task.ti: ffff880244718000 [25082.057464] RIP: 0010:[] [] intel_idle+0xdb/0x180 [25082.058376] RSP: 0018:ffff88024471be08 EFLAGS: 00000046 [25082.059292] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001 [25082.060214] RDX: 0000000000000000 RSI: ffff88024471bfd8 RDI: 0000000000000003 [25082.061132] RBP: ffff88024471be38 R08: ffff88024e2ced08 R09: 00000000ffffffff [25082.062054] R10: 0000000000002655 R11: 00000000000003ff R12: 0000000000000005 [25082.062977] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880244718000 [25082.063899] FS: 0000000000000000(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000 [25082.064825] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25082.065751] CR2: 00007fda8919e000 CR3: 0000000001c11000 CR4: 00000000001407e0 [25082.066685] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25082.067618] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25082.068552] Stack: [25082.069477] 000000034471be38 5fbbf5c12849ac78 ffffe8ffffcc1ba8 0000000000000005 [25082.070426] ffffffff81c9e440 0000000000000003 ffff88024471be88 ffffffff8163a995 [25082.071378] 000016d33f8be858 ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330 [25082.072334] Call Trace: [25082.073279] [] cpuidle_enter_state+0x55/0x190 [25082.074233] [] cpuidle_enter+0x17/0x20 [25082.075161] [] cpu_startup_entry+0x355/0x410 [25082.076066] [] start_secondary+0x1aa/0x230 [25082.076968] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 [25082.078949] NMI backtrace for cpu 2 [25082.079865] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G L 3.18.0+ #108 [25082.081672] task: ffff8802444f8af0 ti: ffff88024470c000 task.ti: ffff88024470c000 [25082.082586] RIP: 0010:[] [] intel_idle+0xdb/0x180 [25082.083507] RSP: 0018:ffff88024470fe08 EFLAGS: 00000046 [25082.084424] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001 [25082.085345] RDX: 0000000000000000 RSI: ffff88024470ffd8 RDI: 0000000000000002 [25082.086261] RBP: ffff88024470fe38 R08: ffff88024e28ed08 R09: 00000000ffffffff [25082.087178] R10: 000000000000265b R11: 00000000000003ff R12: 0000000000000005 [25082.088088] R13: 0000000000000032 R14: 0000000000000004 R15: ffff88024470c000 [25082.089016] FS: 0000000000000000(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000 [25082.089919] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25082.090826] CR2: 00007f274c4061c0 CR3: 0000000001c11000 CR4: 00000000001407e0 [25082.091740] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25082.092652] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25082.093556] Stack: [25082.094451] 000000024470fe38 c6aa8015cfd0a8b5 ffffe8ffffc81ba8 0000000000000005 [25082.095371] ffffffff81c9e440 0000000000000002 ffff88024470fe88 ffffffff8163a995 [25082.096293] 000016d33f8bcc84 ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330 [25082.097220] Call Trace: [25082.098141] [] cpuidle_enter_state+0x55/0x190 [25082.099095] [] cpuidle_enter+0x17/0x20 [25082.100026] [] cpu_startup_entry+0x355/0x410 [25082.100956] [] start_secondary+0x1aa/0x230 [25082.101882] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 [25097.149884] INFO: rcu_sched self-detected stall on CPU [25097.149885] INFO: rcu_sched self-detected stall on CPU [25097.149886] 1: (5973 ticks this GP) idle=6b3/140000000000001/0 softirq=1672732/1672732 [25097.149897] [25097.149898] (t=6000 jiffies g=777862 c=777861 q=0) [25097.149899] Task dump for CPU 0: [25097.149899] trinity-c150 R [25097.149900] running task [25097.149908] 13312 2940 29562 0x10000008 [25097.149909] ffff880197e0c000 [25097.149909] 000000002e7fd874 [25097.149909] ffff880197e0fb88 [25097.149910] ffffffff810f1c86 [25097.149910] ffff8801bf3536b0 [25097.149910] 00007fffa4bfd708 [25097.149911] ffff880197e0fb98 [25097.149911] 000000002e7fd874 [25097.149911] ffff880197e0fbb8 [25097.149911] ffffffff810f1c86 [25097.149912] ffff880197e0fc08 [25097.149912] 00007fffa4bfd708 [25097.149912] Call Trace: [25097.149915] [] ? __module_text_address+0x16/0x80 [25097.149917] [] ? __module_text_address+0x16/0x80 [25097.149919] [] ? is_module_text_address+0x16/0x30 [25097.149921] [] ? __kernel_text_address+0x58/0x80 [25097.149923] [] ? print_context_stack+0x8f/0x100 [25097.149925] [] ? dump_trace+0x16f/0x350 [25097.149926] [] ? link_path_walk+0x266/0x850 [25097.149927] [] ? getname_flags+0x4f/0x1a0 [25097.149929] [] ? final_putname+0x26/0x50 [25097.149930] [] ? save_stack_trace+0x2f/0x50 [25097.149941] [] ? set_track+0x70/0x140 [25097.149943] [] ? free_debug_processing+0x1de/0x232 [25097.149944] [] ? __slab_free+0x122/0x2e9 [25097.149946] [] ? timerqueue_del+0x24/0x70 [25097.149948] [] ? __remove_hrtimer+0x4e/0xa0 [25097.149950] [] ? ipcget+0x8a/0x1e0 [25097.149951] [] ? ipcget+0x7c/0x1e0 [25097.149953] [] ? SyS_msgget+0x4d/0x70 [25097.149955] [] ? tracesys_phase2+0xd4/0xd9 [25097.149955] Task dump for CPU 1: [25097.149956] trinity-c205 R [25097.149956] running task [25097.149958] 13312 636 29562 0x10000008 [25097.149959] ffff8800962fed60 [25097.149959] 0000000077930768 [25097.149959] ffff88024e243d78 [25097.149959] ffffffff810a3932 [25097.149960] 0000000000000001 [25097.149960] ffffffff81c4bf80 [25097.149960] ffff88024e243d98 [25097.149960] ffffffff810a76dd [25097.149961] ffff88024e24c440 [25097.149961] 0000000000000002 [25097.149961] ffff88024e243dc8 [25097.149961] ffffffff810ce920 [25097.149962] Call Trace: [25097.149962] [25097.149964] [] sched_show_task+0xd2/0x140 [25097.149965] [] dump_cpu_task+0x3d/0x50 [25097.149969] [] rcu_dump_cpu_stacks+0x90/0xd0 [25097.149970] [] rcu_check_callbacks+0x4c3/0x730 [25097.149973] [] ? acct_account_cputime+0x1c/0x20 [25097.149974] [] ? account_system_time+0x8e/0x190 [25097.149976] [] update_process_times+0x4b/0x80 [25097.149977] [] tick_sched_timer+0x4f/0x160 [25097.149978] [] __run_hrtimer+0x76/0x1f0 [25097.149979] [] ? tick_init_highres+0x20/0x20 [25097.149980] [] hrtimer_interrupt+0x107/0x260 [25097.149982] [] local_apic_timer_interrupt+0x3b/0x70 [25097.149983] [] smp_apic_timer_interrupt+0x45/0x60 [25097.149985] [] apic_timer_interrupt+0x6f/0x80 [25097.149985] [25097.149987] [] ? retint_restore_args+0xe/0xe [25097.149989] [] ? generic_exec_single+0xee/0x1b0 [25097.149990] [] ? generic_exec_single+0x129/0x1b0 [25097.149991] [] ? proc_alloc_inode+0x1d/0xb0 [25097.149992] [] ? proc_alloc_inode+0x1d/0xb0 [25097.149993] [] ? do_flush_tlb_all+0x60/0x60 [25097.149994] [] ? do_flush_tlb_all+0x60/0x60 [25097.149995] [] smp_call_function_single+0x70/0xd0 [25097.149998] [] ? release_pages+0x1c2/0x280 [25097.149999] [] ? do_flush_tlb_all+0x60/0x60 [25097.150000] [] smp_call_function_many+0x2b9/0x320 [25097.150001] [] flush_tlb_mm_range+0x90/0x1b0 [25097.150003] [] tlb_flush_mmu_tlbonly+0x42/0x50 [25097.150004] [] unmap_single_vma+0x6d8/0x900 [25097.150006] [] ? __free_pages+0x37/0x50 [25097.150007] [] zap_page_range_single+0xfc/0x160 [25097.150009] [] unmap_mapping_range+0x12f/0x190 [25097.150011] [] shmem_fallocate+0x38d/0x4c0 [25097.150012] [] ? cmpxchg_double_slab.isra.49+0xd8/0x122 [25097.150013] [] do_fallocate+0x12f/0x1d0 [25097.150015] [] SyS_madvise+0x385/0x860 [25097.150016] [] ? syscall_trace_enter_phase2+0xa7/0x1e0 [25097.150018] [] tracesys_phase2+0xd4/0xd9 [25097.206563] 0: (1 GPs behind) idle=845/140000000000001/0 softirq=1616166/1616167 [25097.207170] (t=6000 jiffies g=777862 c=777861 q=0) [25097.207778] Task dump for CPU 0: [25097.208382] trinity-c150 R running task 13312 2940 29562 0x10000008 [25097.209002] ffff8801bf3536b0 000000002e7fd874 ffff88024e203d78 ffffffff810a3932 [25097.209635] 0000000000000000 ffffffff81c4bf80 ffff88024e203d98 ffffffff810a76dd [25097.210304] ffff88024e20c440 0000000000000001 ffff88024e203dc8 ffffffff810ce920 [25097.210965] Call Trace: [25097.211614] [] sched_show_task+0xd2/0x140 [25097.212284] [] dump_cpu_task+0x3d/0x50 [25097.212955] [] rcu_dump_cpu_stacks+0x90/0xd0 [25097.213628] [] rcu_check_callbacks+0x4c3/0x730 [25097.214300] [] ? acct_account_cputime+0x1c/0x20 [25097.214973] [] ? account_system_time+0x8e/0x190 [25097.215645] [] update_process_times+0x4b/0x80 [25097.216317] [] tick_sched_timer+0x4f/0x160 [25097.216996] [] __run_hrtimer+0x76/0x1f0 [25097.217676] [] ? tick_init_highres+0x20/0x20 [25097.218359] [] hrtimer_interrupt+0x107/0x260 [25097.219040] [] local_apic_timer_interrupt+0x3b/0x70 [25097.219719] [] smp_apic_timer_interrupt+0x45/0x60 [25097.220415] [] apic_timer_interrupt+0x6f/0x80 [25097.221091] [] ? __remove_hrtimer+0x4e/0xa0 [25097.221778] [] ? ipcget+0x8a/0x1e0 [25097.222454] [] ? ipcget+0x7c/0x1e0 [25097.223125] [] SyS_msgget+0x4d/0x70 [25097.223793] [] tracesys_phase2+0xd4/0xd9 [25097.224461] Task dump for CPU 1: [25097.225124] trinity-c205 R running task 13312 636 29562 0x10000008 [25097.225806] 00000000458261c0 ffff880066bae1b8 ffff880066bae1f0 ffff88021cbef200 [25097.226496] ffff88024267bd20 0000000000000001 ffff88024267bd78 ffffffff811927cf [25097.227190] 0000000000000000 0000000000000000 ffff880066bae1b8 0000000000000000 [25097.227889] Call Trace: [25097.228578] [] ? unmap_mapping_range+0x12f/0x190 [25097.229279] [] ? shmem_fallocate+0x38d/0x4c0 [25097.230004] [] ? cmpxchg_double_slab.isra.49+0xd8/0x122 [25097.230708] [] ? do_fallocate+0x12f/0x1d0 [25097.231409] [] ? SyS_madvise+0x385/0x860 [25097.232106] [] ? syscall_trace_enter_phase2+0xa7/0x1e0 [25097.232810] [] ? tracesys_phase2+0xd4/0xd9 [25121.945509] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c205:636] [25121.946236] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc [25121.950189] CPU: 1 PID: 636 Comm: trinity-c205 Tainted: G L 3.18.0+ #108 [25121.951854] task: ffff8800962fed60 ti: ffff880242678000 task.ti: ffff880242678000 [25121.952705] RIP: 0010:[] [] generic_exec_single+0xee/0x1b0 [25121.953566] RSP: 0018:ffff88024267b9c8 EFLAGS: 00000202 [25121.954428] RAX: 0000000000000008 RBX: ffffffff81799e0d RCX: 0000000000000038 [25121.955296] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000 [25121.956199] RBP: ffff88024267ba28 R08: ffff8802444f43f0 R09: 0000000000000000 [25121.957065] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88024267b938 [25121.957937] R13: 0000000000406040 R14: ffff880242678000 R15: ffff8800962fed60 [25121.958811] FS: 00007ff08e78d740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000 [25121.959692] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25121.960571] CR2: 0000000000000000 CR3: 0000000228031000 CR4: 00000000001407e0 [25121.961462] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25121.962346] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25121.963231] Stack: [25121.964105] ffffffff8124055d ffff8801b4eff080 ffffffff8124055d 0000000000000000 [25121.965010] ffffffff81047bb0 ffff88024267bad8 0000000000000003 0000000077930768 [25121.965945] ffff88024267ba48 00000000ffffffff 0000000000000000 ffffffff81047bb0 [25121.966838] Call Trace: [25121.967723] [] ? proc_alloc_inode+0x1d/0xb0 [25121.968615] [] ? proc_alloc_inode+0x1d/0xb0 [25121.969494] [] ? do_flush_tlb_all+0x60/0x60 [25121.970371] [] ? do_flush_tlb_all+0x60/0x60 [25121.971239] [] smp_call_function_single+0x70/0xd0 [25121.972107] [] ? release_pages+0x1c2/0x280 [25121.972968] [] ? do_flush_tlb_all+0x60/0x60 [25121.973829] [] smp_call_function_many+0x2b9/0x320 [25121.974690] [] flush_tlb_mm_range+0x90/0x1b0 [25121.975592] [] tlb_flush_mmu_tlbonly+0x42/0x50 [25121.976454] [] unmap_single_vma+0x6d8/0x900 [25121.977319] [] ? __free_pages+0x37/0x50 [25121.978178] [] zap_page_range_single+0xfc/0x160 [25121.979044] [] unmap_mapping_range+0x12f/0x190 [25121.979905] [] shmem_fallocate+0x38d/0x4c0 [25121.980768] [] ? cmpxchg_double_slab.isra.49+0xd8/0x122 [25121.981636] [] do_fallocate+0x12f/0x1d0 [25121.982501] [] SyS_madvise+0x385/0x860 [25121.983362] [] ? syscall_trace_enter_phase2+0xa7/0x1e0 [25121.984228] [] tracesys_phase2+0xd4/0xd9 [25121.985086] Code: 48 89 de 48 03 14 c5 e0 bf cf 81 48 89 df e8 5a eb 26 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 f6 43 18 01 <75> f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 0f 85 98 00 [25121.986993] sending NMI to other CPUs: [25121.987891] NMI backtrace for cpu 0 [25121.988871] CPU: 0 PID: 2940 Comm: trinity-c150 Tainted: G L 3.18.0+ #108 [25121.990863] task: ffff8801bf3536b0 ti: ffff880197e0c000 task.ti: ffff880197e0c000 [25121.991871] RIP: 0010:[] [] read_hpet+0x16/0x20 [25121.992886] RSP: 0018:ffff88024e203e38 EFLAGS: 00000046 [25121.993898] RAX: 00000000cbd1340c RBX: 000000000051684a RCX: 0000000000000000 [25121.994911] RDX: 0000000000000000 RSI: ffff88024e20c710 RDI: ffffffff81c26f40 [25121.995923] RBP: ffff88024e203e38 R08: 0000000000000000 R09: 000000000000000f [25121.996942] R10: 0000000000000526 R11: 0000000000aaaaaa R12: 000016dc7859e117 [25121.997961] R13: 0000000000000000 R14: ffff88024e20c700 R15: 000016eacc784674 [25121.998975] FS: 00007ff08e78d740(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000 [25122.000003] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25122.001032] CR2: 0000000000000001 CR3: 0000000229d22000 CR4: 00000000001407f0 [25122.002066] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25122.003094] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25122.004099] Stack: [25122.005070] ffff88024e203e68 ffffffff810e0d3e ffff88024e20ca80 ffff88024e20ca80 [25122.006050] ffff880197e0fe38 ffff88024e20c6c0 ffff88024e203e98 ffffffff810e9cd3 [25122.007005] ffff88024e203f28 ffff88024e20ca80 ffff88024e203f28 ffff88024e20c6c0 [25122.007936] Call Trace: [25122.008841] [25122.009729] [] ktime_get+0x3e/0xa0 [25122.010594] [] tick_sched_timer+0x23/0x160 [25122.011453] [] __run_hrtimer+0x76/0x1f0 [25122.012301] [] ? tick_init_highres+0x20/0x20 [25122.013148] [] hrtimer_interrupt+0x107/0x260 [25122.013996] [] local_apic_timer_interrupt+0x3b/0x70 [25122.014843] [] smp_apic_timer_interrupt+0x45/0x60 [25122.015690] [] apic_timer_interrupt+0x6f/0x80 [25122.016534] [25122.017374] [] ? __remove_hrtimer+0x4e/0xa0 [25122.018207] [] ? ipcget+0x8a/0x1e0 [25122.019037] [] ? ipcget+0x7c/0x1e0 [25122.019859] [] SyS_msgget+0x4d/0x70 [25122.020679] [] tracesys_phase2+0xd4/0xd9 [25122.021490] Code: 00 29 c7 ba 00 00 00 00 b8 c2 ff ff ff 83 ff 7f 5d 0f 4f c2 c3 0f 1f 44 00 00 55 48 8b 05 d3 c8 ec 00 48 89 e5 8b 80 f0 00 00 00 <89> c0 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 8b 0d 29 c8 ec 00 [25122.023285] NMI backtrace for cpu 2 [25122.024138] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G L 3.18.0+ #108 [25122.025877] task: ffff8802444f8af0 ti: ffff88024470c000 task.ti: ffff88024470c000 [25122.026772] RIP: 0010:[] [] intel_idle+0xdb/0x180 [25122.027680] RSP: 0018:ffff88024470fe08 EFLAGS: 00000046 [25122.028590] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001 [25122.029511] RDX: 0000000000000000 RSI: ffff88024470ffd8 RDI: 0000000000000002 [25122.030427] RBP: ffff88024470fe38 R08: ffff88024e28ed08 R09: 00000000ffffffff [25122.031345] R10: 000000000000265b R11: 00000000000003ff R12: 0000000000000005 [25122.032268] R13: 0000000000000032 R14: 0000000000000004 R15: ffff88024470c000 [25122.033189] FS: 0000000000000000(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000 [25122.034115] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25122.035042] CR2: 00007f6b1f71a000 CR3: 0000000001c11000 CR4: 00000000001407e0 [25122.035977] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25122.036911] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25122.037844] Stack: [25122.038768] 000000024470fe38 c6aa8015cfd0a8b5 ffffe8ffffc81ba8 0000000000000005 [25122.039719] ffffffff81c9e440 0000000000000002 ffff88024470fe88 ffffffff8163a995 [25122.040671] 000016dc8fbb58a7 ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330 [25122.041629] Call Trace: [25122.042576] [] cpuidle_enter_state+0x55/0x190 [25122.043530] [] cpuidle_enter+0x17/0x20 [25122.044462] [] cpu_startup_entry+0x355/0x410 [25122.045370] [] start_secondary+0x1aa/0x230 [25122.046275] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 [25122.048261] NMI backtrace for cpu 3 [25122.049177] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G L 3.18.0+ #108 [25122.050992] task: ffff8802444fe270 ti: ffff880244718000 task.ti: ffff880244718000 [25122.051911] RIP: 0010:[] [] intel_idle+0xdb/0x180 [25122.052836] RSP: 0018:ffff88024471be08 EFLAGS: 00000046 [25122.053757] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001 [25122.054682] RDX: 0000000000000000 RSI: ffff88024471bfd8 RDI: 0000000000000003 [25122.055628] RBP: ffff88024471be38 R08: ffff88024e2ced08 R09: 00000000ffffffff [25122.056550] R10: 0000000000002655 R11: 00000000000003ff R12: 0000000000000005 [25122.057466] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880244718000 [25122.058374] FS: 0000000000000000(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000 [25122.059282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25122.060191] CR2: 00007fda8919e000 CR3: 0000000001c11000 CR4: 00000000001407e0 [25122.061107] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25122.062020] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25122.062925] Stack: [25122.063822] 000000034471be38 5fbbf5c12849ac78 ffffe8ffffcc1ba8 0000000000000005 [25122.064744] ffffffff81c9e440 0000000000000003 ffff88024471be88 ffffffff8163a995 [25122.065690] 000016dc8fbb74c1 ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330 [25122.066617] Call Trace: [25122.067536] [] cpuidle_enter_state+0x55/0x190 [25122.068470] [] cpuidle_enter+0x17/0x20 [25122.069400] [] cpu_startup_entry+0x355/0x410 [25122.070329] [] start_secondary+0x1aa/0x230 [25122.071258] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 [25149.929286] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c205:636] [25149.930256] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc [25149.935393] CPU: 1 PID: 636 Comm: trinity-c205 Tainted: G L 3.18.0+ #108 [25149.937508] task: ffff8800962fed60 ti: ffff880242678000 task.ti: ffff880242678000 [25149.938555] RIP: 0010:[] [] generic_exec_single+0xee/0x1b0 [25149.939652] RSP: 0018:ffff88024267b9c8 EFLAGS: 00000202 [25149.940706] RAX: 0000000000000008 RBX: ffffffff81799e0d RCX: 0000000000000038 [25149.941751] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000 [25149.942780] RBP: ffff88024267ba28 R08: ffff8802444f43f0 R09: 0000000000000000 [25149.943803] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88024267b938 [25149.944822] R13: 0000000000406040 R14: ffff880242678000 R15: ffff8800962fed60 [25149.945839] FS: 00007ff08e78d740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000 [25149.946862] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25149.947891] CR2: 0000000000000000 CR3: 0000000228031000 CR4: 00000000001407e0 [25149.948928] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25149.949997] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25149.951027] Stack: [25149.952045] ffffffff8124055d ffff8801b4eff080 ffffffff8124055d 0000000000000000 [25149.953082] ffffffff81047bb0 ffff88024267bad8 0000000000000003 0000000077930768 [25149.954119] ffff88024267ba48 00000000ffffffff 0000000000000000 ffffffff81047bb0 [25149.955159] Call Trace: [25149.956184] [] ? proc_alloc_inode+0x1d/0xb0 [25149.957219] [] ? proc_alloc_inode+0x1d/0xb0 [25149.958239] [] ? do_flush_tlb_all+0x60/0x60 [25149.959258] [] ? do_flush_tlb_all+0x60/0x60 [25149.960299] [] smp_call_function_single+0x70/0xd0 [25149.961305] [] ? release_pages+0x1c2/0x280 [25149.962310] [] ? do_flush_tlb_all+0x60/0x60 [25149.963313] [] smp_call_function_many+0x2b9/0x320 [25149.964320] [] flush_tlb_mm_range+0x90/0x1b0 [25149.965312] [] tlb_flush_mmu_tlbonly+0x42/0x50 [25149.966281] [] unmap_single_vma+0x6d8/0x900 [25149.967249] [] ? __free_pages+0x37/0x50 [25149.968212] [] zap_page_range_single+0xfc/0x160 [25149.969163] [] unmap_mapping_range+0x12f/0x190 [25149.970131] [] shmem_fallocate+0x38d/0x4c0 [25149.971056] [] ? cmpxchg_double_slab.isra.49+0xd8/0x122 [25149.971977] [] do_fallocate+0x12f/0x1d0 [25149.972895] [] SyS_madvise+0x385/0x860 [25149.973812] [] ? syscall_trace_enter_phase2+0xa7/0x1e0 [25149.974735] [] tracesys_phase2+0xd4/0xd9 [25149.975654] Code: 48 89 de 48 03 14 c5 e0 bf cf 81 48 89 df e8 5a eb 26 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 f6 43 18 01 <75> f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 0f 85 98 00 [25149.977642] sending NMI to other CPUs: [25149.978587] NMI backtrace for cpu 0 [25149.979615] CPU: 0 PID: 2940 Comm: trinity-c150 Tainted: G L 3.18.0+ #108 [25149.981704] task: ffff8801bf3536b0 ti: ffff880197e0c000 task.ti: ffff880197e0c000 [25149.982766] RIP: 0010:[] [] invoke_rcu_core+0x2b/0x50 [25149.983834] RSP: 0018:ffff88024e203db8 EFLAGS: 00000093 [25149.984894] RAX: ffffffff81cfe330 RBX: 0000000000000000 RCX: 00000000000bde85 [25149.985959] RDX: 00000000000bde86 RSI: ffff88024e20c440 RDI: ffffffff81c4bf80 [25149.987021] RBP: ffff88024e203dc8 R08: 0000000000000000 R09: 000000000000000f [25149.988084] R10: 0000000000000526 R11: 0000000000aaaaaa R12: ffffffff81c4bf80 [25149.989155] R13: ffffffff81c4bf80 R14: 0000000000000000 R15: 000016eacc784674 [25149.990227] FS: 00007ff08e78d740(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000 [25149.991310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25149.992374] CR2: 0000000000000001 CR3: 0000000229d22000 CR4: 00000000001407f0 [25149.993420] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25149.994462] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25149.995477] Stack: [25149.996463] ffff8801bf3536b0 ffff88024e20c440 ffff88024e203e38 ffffffff810d48f3 [25149.997456] ffff88024e203de8 ffffffff81120b4c ffff88024e203e28 ffffffff810a800e [25149.998421] ffff88024e203e08 ffffffff81cfe338 ffff88024e203e38 ffff8801bf3536b0 [25149.999364] Call Trace: [25150.000280] [25150.001179] [] rcu_check_callbacks+0x183/0x730 [25150.002063] [] ? acct_account_cputime+0x1c/0x20 [25150.002939] [] ? account_system_time+0x8e/0x190 [25150.003805] [] update_process_times+0x4b/0x80 [25150.004669] [] tick_sched_timer+0x4f/0x160 [25150.005530] [] __run_hrtimer+0x76/0x1f0 [25150.006391] [] ? tick_init_highres+0x20/0x20 [25150.007248] [] hrtimer_interrupt+0x107/0x260 [25150.008106] [] local_apic_timer_interrupt+0x3b/0x70 [25150.008958] [] smp_apic_timer_interrupt+0x45/0x60 [25150.009803] [] apic_timer_interrupt+0x6f/0x80 [25150.010645] [25150.011487] [] ? __remove_hrtimer+0x4e/0xa0 [25150.012328] [] ? ipcget+0x8a/0x1e0 [25150.013162] [] ? ipcget+0x7c/0x1e0 [25150.013991] [] SyS_msgget+0x4d/0x70 [25150.014809] [] tracesys_phase2+0xd4/0xd9 [25150.015627] Code: 0f 1f 44 00 00 55 48 89 e5 53 48 83 ec 08 65 8b 1c 25 2c a0 00 00 3b 1d e8 02 c3 00 73 24 89 db 48 8b 05 31 3e 74 00 48 0f a3 18 <19> db 85 db 74 0a bf 09 00 00 00 e8 b5 be fa ff 48 83 c4 08 5b [25150.017421] NMI backtrace for cpu 2 [25150.018283] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G L 3.18.0+ #108 [25150.020063] task: ffff8802444f8af0 ti: ffff88024470c000 task.ti: ffff88024470c000 [25150.020972] RIP: 0010:[] [] intel_idle+0xdb/0x180 [25150.021897] RSP: 0018:ffff88024470fe08 EFLAGS: 00000046 [25150.022819] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001 [25150.023745] RDX: 0000000000000000 RSI: ffff88024470ffd8 RDI: 0000000000000002 [25150.024669] RBP: ffff88024470fe38 R08: ffff88024e28ed08 R09: 00000000ffffffff [25150.025595] R10: 000000000000265b R11: 00000000000003ff R12: 0000000000000005 [25150.026523] R13: 0000000000000032 R14: 0000000000000004 R15: ffff88024470c000 [25150.027445] FS: 0000000000000000(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000 [25150.028378] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25150.029311] CR2: 00007f6b1f71a000 CR3: 0000000001c11000 CR4: 00000000001407e0 [25150.030250] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25150.031190] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25150.032125] Stack: [25150.033056] 000000024470fe38 c6aa8015cfd0a8b5 ffffe8ffffc81ba8 0000000000000005 [25150.034012] ffffffff81c9e440 0000000000000002 ffff88024470fe88 ffffffff8163a995 [25150.034966] 000016e314a978e8 ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330 [25150.035903] Call Trace: [25150.036807] [] cpuidle_enter_state+0x55/0x190 [25150.037724] [] cpuidle_enter+0x17/0x20 [25150.038637] [] cpu_startup_entry+0x355/0x410 [25150.039548] [] start_secondary+0x1aa/0x230 [25150.040438] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 [25150.042345] NMI backtrace for cpu 3 [25150.043231] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G L 3.18.0+ #108 [25150.045019] task: ffff8802444fe270 ti: ffff880244718000 task.ti: ffff880244718000 [25150.045926] RIP: 0010:[] [] intel_idle+0xdb/0x180 [25150.046843] RSP: 0018:ffff88024471be08 EFLAGS: 00000046 [25150.047755] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001 [25150.048670] RDX: 0000000000000000 RSI: ffff88024471bfd8 RDI: 0000000000000003 [25150.049598] RBP: ffff88024471be38 R08: ffff88024e2ced08 R09: 00000000ffffffff [25150.050500] R10: 0000000000002655 R11: 00000000000003ff R12: 0000000000000005 [25150.051406] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880244718000 [25150.052313] FS: 0000000000000000(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000 [25150.053216] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25150.054114] CR2: 00007fda8919e000 CR3: 0000000001c11000 CR4: 00000000001407e0 [25150.055021] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25150.055929] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25150.056829] Stack: [25150.057727] 000000034471be38 5fbbf5c12849ac78 ffffe8ffffcc1ba8 0000000000000005 [25150.058649] ffffffff81c9e440 0000000000000003 ffff88024471be88 ffffffff8163a995 [25150.059600] 000016e314a99619 ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330 [25150.060530] Call Trace: [25150.061450] [] cpuidle_enter_state+0x55/0x190 [25150.062380] [] cpuidle_enter+0x17/0x20 [25150.063303] [] cpu_startup_entry+0x355/0x410 [25150.064231] [] start_secondary+0x1aa/0x230 [25150.065157] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 [25177.913063] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c205:636] [25177.914027] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc [25177.919143] CPU: 1 PID: 636 Comm: trinity-c205 Tainted: G L 3.18.0+ #108 [25177.921249] task: ffff8800962fed60 ti: ffff880242678000 task.ti: ffff880242678000 [25177.922289] RIP: 0010:[] [] generic_exec_single+0xea/0x1b0 [25177.923382] RSP: 0018:ffff88024267b9c8 EFLAGS: 00000202 [25177.924433] RAX: 0000000000000008 RBX: ffffffff81799e0d RCX: 0000000000000038 [25177.925476] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000 [25177.926504] RBP: ffff88024267ba28 R08: ffff8802444f43f0 R09: 0000000000000000 [25177.927525] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88024267b938 [25177.928542] R13: 0000000000406040 R14: ffff880242678000 R15: ffff8800962fed60 [25177.929556] FS: 00007ff08e78d740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000 [25177.930580] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25177.931606] CR2: 0000000000000000 CR3: 0000000228031000 CR4: 00000000001407e0 [25177.932638] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25177.933707] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25177.934735] Stack: [25177.935750] ffffffff8124055d ffff8801b4eff080 ffffffff8124055d 0000000000000000 [25177.936786] ffffffff81047bb0 ffff88024267bad8 0000000000000003 0000000077930768 [25177.937819] ffff88024267ba48 00000000ffffffff 0000000000000000 ffffffff81047bb0 [25177.938856] Call Trace: [25177.939882] [] ? proc_alloc_inode+0x1d/0xb0 [25177.940911] [] ? proc_alloc_inode+0x1d/0xb0 [25177.941928] [] ? do_flush_tlb_all+0x60/0x60 [25177.942941] [] ? do_flush_tlb_all+0x60/0x60 [25177.943980] [] smp_call_function_single+0x70/0xd0 [25177.944982] [] ? release_pages+0x1c2/0x280 [25177.945985] [] ? do_flush_tlb_all+0x60/0x60 [25177.946988] [] smp_call_function_many+0x2b9/0x320 [25177.947993] [] flush_tlb_mm_range+0x90/0x1b0 [25177.948984] [] tlb_flush_mmu_tlbonly+0x42/0x50 [25177.949951] [] unmap_single_vma+0x6d8/0x900 [25177.950920] [] ? __free_pages+0x37/0x50 [25177.951886] [] zap_page_range_single+0xfc/0x160 [25177.952837] [] unmap_mapping_range+0x12f/0x190 [25177.953805] [] shmem_fallocate+0x38d/0x4c0 [25177.954730] [] ? cmpxchg_double_slab.isra.49+0xd8/0x122 [25177.955653] [] do_fallocate+0x12f/0x1d0 [25177.956573] [] SyS_madvise+0x385/0x860 [25177.957491] [] ? syscall_trace_enter_phase2+0xa7/0x1e0 [25177.958414] [] tracesys_phase2+0xd4/0xd9 [25177.959332] Code: 00 2b 01 00 48 89 de 48 03 14 c5 e0 bf cf 81 48 89 df e8 5a eb 26 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 43 18 01 75 f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 [25177.961321] sending NMI to other CPUs: [25177.962268] NMI backtrace for cpu 0 [25177.963295] CPU: 0 PID: 2940 Comm: trinity-c150 Tainted: G L 3.18.0+ #108 [25177.965391] task: ffff8801bf3536b0 ti: ffff880197e0c000 task.ti: ffff880197e0c000 [25177.966456] RIP: 0010:[] [] read_hpet+0x16/0x20 [25177.967525] RSP: 0018:ffff88024e203e38 EFLAGS: 00000046 [25177.968586] RAX: 00000000fb9d303f RBX: 00000000005193fc RCX: 0000000000000000 [25177.969651] RDX: 0000000000000000 RSI: ffff88024e20c710 RDI: ffffffff81c26f40 [25177.970712] RBP: ffff88024e203e38 R08: 0000000000000000 R09: 000000000000000f [25177.971777] R10: 0000000000000526 R11: 0000000000aaaaaa R12: 000016e982361117 [25177.972846] R13: 0000000000000000 R14: ffff88024e20c700 R15: 000016eacc784674 [25177.973915] FS: 00007ff08e78d740(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000 [25177.974998] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25177.976058] CR2: 0000000000000001 CR3: 0000000229d22000 CR4: 00000000001407f0 [25177.977103] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25177.978139] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25177.979151] Stack: [25177.980134] ffff88024e203e68 ffffffff810e0d3e ffff88024e20ca80 ffff88024e20ca80 [25177.981123] ffff880197e0fe38 ffff88024e20c6c0 ffff88024e203e98 ffffffff810e9cd3 [25177.982089] ffff88024e203f28 ffff88024e20ca80 ffff88024e203f28 ffff88024e20c6c0 [25177.983029] Call Trace: [25177.983943] [25177.984839] [] ktime_get+0x3e/0xa0 [25177.985714] [] tick_sched_timer+0x23/0x160 [25177.986581] [] __run_hrtimer+0x76/0x1f0 [25177.987439] [] ? tick_init_highres+0x20/0x20 [25177.988296] [] hrtimer_interrupt+0x107/0x260 [25177.989149] [] local_apic_timer_interrupt+0x3b/0x70 [25177.990007] [] smp_apic_timer_interrupt+0x45/0x60 [25177.990862] [] apic_timer_interrupt+0x6f/0x80 [25177.991714] [25177.992558] [] ? __remove_hrtimer+0x4e/0xa0 [25177.993396] [] ? ipcget+0x8a/0x1e0 [25177.994233] [] ? ipcget+0x7c/0x1e0 [25177.995063] [] SyS_msgget+0x4d/0x70 [25177.995891] [] tracesys_phase2+0xd4/0xd9 [25177.996712] Code: 00 29 c7 ba 00 00 00 00 b8 c2 ff ff ff 83 ff 7f 5d 0f 4f c2 c3 0f 1f 44 00 00 55 48 8b 05 d3 c8 ec 00 48 89 e5 8b 80 f0 00 00 00 <89> c0 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 8b 0d 29 c8 ec 00 [25177.998518] NMI backtrace for cpu 2 [25177.999377] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G L 3.18.0+ #108 [25178.001121] task: ffff8802444f8af0 ti: ffff88024470c000 task.ti: ffff88024470c000 [25178.002019] RIP: 0010:[] [] intel_idle+0xdb/0x180 [25178.002932] RSP: 0018:ffff88024470fe08 EFLAGS: 00000046 [25178.003849] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001 [25178.004771] RDX: 0000000000000000 RSI: ffff88024470ffd8 RDI: 0000000000000002 [25178.005688] RBP: ffff88024470fe38 R08: ffff88024e28ed08 R09: 00000000ffffffff [25178.006609] R10: 000000000000265b R11: 00000000000003ff R12: 0000000000000005 [25178.007532] R13: 0000000000000032 R14: 0000000000000004 R15: ffff88024470c000 [25178.008453] FS: 0000000000000000(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000 [25178.009379] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25178.010307] CR2: 00007f6b1f71a000 CR3: 0000000001c11000 CR4: 00000000001407e0 [25178.011246] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25178.012181] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25178.013112] Stack: [25178.014036] 000000024470fe38 c6aa8015cfd0a8b5 ffffe8ffffc81ba8 0000000000000005 [25178.014985] ffffffff81c9e440 0000000000000002 ffff88024470fe88 ffffffff8163a995 [25178.015936] 000016e99997935a ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330 [25178.016893] Call Trace: [25178.017836] [] cpuidle_enter_state+0x55/0x190 [25178.018791] [] cpuidle_enter+0x17/0x20 [25178.019719] [] cpu_startup_entry+0x355/0x410 [25178.020625] [] start_secondary+0x1aa/0x230 [25178.021526] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 [25178.023504] NMI backtrace for cpu 3 [25178.024417] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G L 3.18.0+ #108 [25178.026229] task: ffff8802444fe270 ti: ffff880244718000 task.ti: ffff880244718000 [25178.027145] RIP: 0010:[] [] intel_idle+0xdb/0x180 [25178.028067] RSP: 0018:ffff88024471be08 EFLAGS: 00000046 [25178.028983] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001 [25178.029906] RDX: 0000000000000000 RSI: ffff88024471bfd8 RDI: 0000000000000003 [25178.030826] RBP: ffff88024471be38 R08: ffff88024e2ced08 R09: 00000000ffffffff [25178.031744] R10: 0000000000002656 R11: 00000000000003ff R12: 0000000000000005 [25178.032657] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880244718000 [25178.033586] FS: 0000000000000000(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000 [25178.034496] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25178.035401] CR2: 00007fda8919e000 CR3: 0000000001c11000 CR4: 00000000001407e0 [25178.036319] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25178.037229] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25178.038130] Stack: [25178.039023] 000000034471be38 5fbbf5c12849ac78 ffffe8ffffcc1ba8 0000000000000005 [25178.039948] ffffffff81c9e440 0000000000000003 ffff88024471be88 ffffffff8163a995 [25178.040867] 000016e99997b046 ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330 [25178.041793] Call Trace: [25178.042713] [] cpuidle_enter_state+0x55/0x190 [25178.043663] [] cpuidle_enter+0x17/0x20 [25178.044589] [] cpu_startup_entry+0x355/0x410 [25178.045515] [] start_secondary+0x1aa/0x230 [25178.046441] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 [25183.090076] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 88.570 msecs [25183.090087] sched: RT throttling activated [25205.896861] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [trinity-c65:8051] [25205.897959] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc [25205.904284] CPU: 1 PID: 8051 Comm: trinity-c65 Tainted: G L 3.18.0+ #108 [25205.906737] task: ffff8800a1b98000 ti: ffff880071a70000 task.ti: ffff880071a70000 [25205.907986] RIP: 0010:[] [] __slab_alloc+0x4e5/0x53b [25205.909296] RSP: 0018:ffff880071a739f8 EFLAGS: 00000246 [25205.910532] RAX: 0000000000000001 RBX: ffff88022f966c40 RCX: 0000000180240022 [25205.911789] RDX: 0000000000000024 RSI: ffffea0008be5800 RDI: ffff880245827cc0 [25205.913018] RBP: ffff880071a73ac8 R08: ffff88022f965138 R09: 0000000000000000 [25205.914222] R10: 0000000000000092 R11: 0000000000000000 R12: ffffffff810133af [25205.915459] R13: ffff880071a73978 R14: 0000000100240021 R15: ffffffff8134f019 [25205.916728] FS: 00007f2805635740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000 [25205.917955] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25205.919158] CR2: 0000000001a86a10 CR3: 0000000097832000 CR4: 00000000001407e0 [25205.920348] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25205.921514] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25205.922678] Stack: [25205.923828] ffff880071a73a68 ffffffff811b7057 ffff8800a1b98000 ffff8800a1b98000 [25205.925006] 0000000000000000 ffff88022f9669f0 0000000200000000 ffffffff8134f019 [25205.926179] ffffffff00000000 0000000000000246 0000000000000000 0000000000000000 [25205.927394] Call Trace: [25205.928579] [] ? alloc_pages_vma+0x97/0x160 [25205.929762] [] ? __radix_tree_preload+0x49/0xc0 [25205.930972] [] ? shmem_alloc_page+0x6e/0xc0 [25205.932185] [] ? __radix_tree_create+0x85/0x220 [25205.933398] [] ? __radix_tree_preload+0x49/0xc0 [25205.934603] [] kmem_cache_alloc+0x1bb/0x1e0 [25205.935825] [] __radix_tree_preload+0x49/0xc0 [25205.937011] [] radix_tree_maybe_preload+0x21/0x30 [25205.938175] [] shmem_getpage_gfp+0x466/0x810 [25205.939323] [] ? finish_task_switch+0x4a/0x100 [25205.940453] [] shmem_write_begin+0x3f/0x60 [25205.941549] [] generic_perform_write+0xd4/0x1f0 [25205.942627] [] __generic_file_write_iter+0x15f/0x350 [25205.943694] [] ? new_sync_read+0xd0/0xd0 [25205.944754] [] generic_file_write_iter+0x3d/0xb0 [25205.945813] [] ? __generic_file_write_iter+0x350/0x350 [25205.946885] [] do_iter_readv_writev+0x78/0xc0 [25205.948034] [] do_readv_writev+0xd8/0x2a0 [25205.949150] [] ? __generic_file_write_iter+0x350/0x350 [25205.950297] [] ? __generic_file_write_iter+0x350/0x350 [25205.951463] [] ? __hrtimer_start_range_ns+0x252/0x380 [25205.952586] [] ? hrtimer_start+0x18/0x20 [25205.953678] [] vfs_writev+0x39/0x50 [25205.954747] [] SyS_writev+0x59/0xf0 [25205.955854] [] tracesys_phase2+0xd4/0xd9 [25205.956941] Code: 00 02 00 00 49 c7 45 00 00 00 00 00 75 11 ff b5 78 ff ff ff 9d e8 07 b4 9a ff 4c 89 e0 eb 0f e8 fd b4 9a ff ff b5 78 ff ff ff 9d <4c> 89 e0 48 8b 75 c8 65 48 33 34 25 28 00 00 00 74 32 e8 ff 79 [25205.959247] sending NMI to other CPUs: [25205.960438] NMI backtrace for cpu 2 [25205.961454] CPU: 2 PID: 7491 Comm: trinity-c14 Tainted: G L 3.18.0+ #108 [25205.963519] task: ffff88022b98a0d0 ti: ffff880096408000 task.ti: ffff880096408000 [25205.964574] RIP: 0010:[] [] __lru_cache_add+0x4a/0x90 [25205.965645] RSP: 0018:ffff88009640bb38 EFLAGS: 00000293 [25205.966690] RAX: 000000000000000c RBX: ffff88024e28e0c0 RCX: 000000000001602a [25205.967724] RDX: 000000000000000d RSI: 0000000000000018 RDI: ffffea00042410c0 [25205.968745] RBP: ffff88009640bb48 R08: ffff8801020dae90 R09: 0000000000000000 [25205.969761] R10: 0000000000000001 R11: 0000000000000000 R12: ffffea00042410c0 [25205.970773] R13: 0000000000000000 R14: ffff880096339f18 R15: 0000000000000000 [25205.971763] FS: 00007f2805635740(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000 [25205.972737] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25205.973689] CR2: 0000003370219050 CR3: 000000009ab9e000 CR4: 00000000001407e0 [25205.974628] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25205.975543] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25205.976440] Stack: [25205.977315] 0000000000000003 0000000000000000 ffff88009640bb58 ffffffff81174919 [25205.978198] ffff88009640bc18 ffffffff8117fa80 ffff88009640bb98 ffffffff8109bdb0 [25205.979071] ffff8802292c8980 ffff88009640bc80 0000000000000000 00000000000000d0 [25205.979939] Call Trace: [25205.980791] [] lru_cache_add_anon+0x19/0x20 [25205.981659] [] shmem_getpage_gfp+0x4a0/0x810 [25205.982526] [] ? finish_task_switch+0xe0/0x100 [25205.983385] [] shmem_write_begin+0x3f/0x60 [25205.984245] [] generic_perform_write+0xd4/0x1f0 [25205.985098] [] __generic_file_write_iter+0x15f/0x350 [25205.985946] [] ? new_sync_read+0xd0/0xd0 [25205.986787] [] generic_file_write_iter+0x3d/0xb0 [25205.987629] [] ? __generic_file_write_iter+0x350/0x350 [25205.988479] [] do_iter_readv_writev+0x78/0xc0 [25205.989323] [] do_readv_writev+0xd8/0x2a0 [25205.990156] [] ? __generic_file_write_iter+0x350/0x350 [25205.990994] [] ? __generic_file_write_iter+0x350/0x350 [25205.991822] [] ? __hrtimer_start_range_ns+0x252/0x380 [25205.992644] [] ? hrtimer_start+0x18/0x20 [25205.993461] [] vfs_writev+0x39/0x50 [25205.994277] [] SyS_writev+0x59/0xf0 [25205.995089] [] tracesys_phase2+0xd4/0xd9 [25205.995900] Code: 48 03 1c 25 68 bd 00 00 48 8b 07 f6 c4 80 75 47 41 8b 44 24 1c 85 c0 7e 4c f0 41 ff 44 24 1c 48 8b 03 83 f8 0e 74 1a 48 8d 50 01 <48> 89 13 4c 89 64 c3 10 5b 65 ff 0c 25 60 a8 00 00 41 5c 5d c3 [25205.997698] NMI backtrace for cpu 3 [25205.998563] CPU: 3 PID: 10185 Comm: trinity-c162 Tainted: G L 3.18.0+ #108 [25206.000332] task: ffff880197ea2bc0 ti: ffff8801c6410000 task.ti: ffff8801c6410000 [25206.001236] RIP: 0010:[] [] generic_exec_single+0xee/0x1b0 [25206.002161] RSP: 0018:ffff8801c64139c8 EFLAGS: 00000202 [25206.003077] RAX: 0000000000000008 RBX: ffff8801c64139e0 RCX: 0000000000000038 [25206.004003] RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000000 [25206.004926] RBP: ffff8801c6413a28 R08: ffff8802444f47e0 R09: 0000000000000000 [25206.005848] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001 [25206.006769] R13: 0000000000000001 R14: ffff880229f27a00 R15: 0000000000000003 [25206.007683] FS: 00007f2805635740(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000 [25206.008612] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25206.009539] CR2: 00007f28048e2220 CR3: 000000021ca6c000 CR4: 00000000001407e0 [25206.010484] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25206.011426] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25206.012367] Stack: [25206.013300] ffffffff8124055d ffff8801e640ddc0 ffffffff8124055d 0000000000000000 [25206.014263] ffffffff81047bb0 ffff8801c6413ad8 0000000000000003 00000000e731055f [25206.015228] ffff8801c6413a48 00000000ffffffff 0000000000000001 ffffffff81047bb0 [25206.016172] Call Trace: [25206.017090] [] ? proc_alloc_inode+0x1d/0xb0 [25206.018026] [] ? proc_alloc_inode+0x1d/0xb0 [25206.018958] [] ? do_flush_tlb_all+0x60/0x60 [25206.019879] [] ? do_flush_tlb_all+0x60/0x60 [25206.020766] [] smp_call_function_single+0x70/0xd0 [25206.021637] [] ? do_flush_tlb_all+0x60/0x60 [25206.022498] [] smp_call_function_many+0x2b9/0x320 [25206.023356] [] ? zone_statistics+0x81/0xa0 [25206.024210] [] flush_tlb_mm_range+0x90/0x1b0 [25206.025067] [] tlb_flush_mmu_tlbonly+0x42/0x50 [25206.025914] [] unmap_single_vma+0x6d8/0x900 [25206.026757] [] zap_page_range_single+0xfc/0x160 [25206.027601] [] unmap_mapping_range+0x12f/0x190 [25206.028439] [] shmem_fallocate+0x38d/0x4c0 [25206.029270] [] do_fallocate+0x12f/0x1d0 [25206.030089] [] SyS_madvise+0x385/0x860 [25206.030910] [] ? kmem_cache_free+0x14e/0x1e0 [25206.031733] [] ? putname+0x29/0x40 [25206.032546] [] ? syscall_trace_enter_phase2+0xa7/0x1e0 [25206.033361] [] tracesys_phase2+0xd4/0xd9 [25206.034170] Code: 48 89 de 48 03 14 c5 e0 bf cf 81 48 89 df e8 5a eb 26 00 84 c0 75 46 45 85 ed 74 11 f6 43 18 01 74 0b 0f 1f 00 f3 90 f6 43 18 01 <75> f8 31 c0 48 8b 7d d8 65 48 33 3c 25 28 00 00 00 0f 85 98 00 [25206.035960] NMI backtrace for cpu 0 [25206.036822] CPU: 0 PID: 9587 Comm: trinity-c129 Tainted: G L 3.18.0+ #108 [25206.038638] task: ffff88009a8695e0 ti: ffff880039f48000 task.ti: ffff880039f48000 [25206.039565] RIP: 0033:[<000000336ee891e5>] [<000000336ee891e5>] 0x336ee891e5 [25206.040510] RSP: 002b:00007fffdd789658 EFLAGS: 00000202 [25206.041437] RAX: 0000000001b33250 RBX: 0000000001b33240 RCX: 000000000202fe00 [25206.042378] RDX: 0000000002533240 RSI: 0000000000000054 RDI: 0000000001b33250 [25206.043317] RBP: 0000000000a00010 R08: 0000000001b33240 R09: 000000000000002e [25206.044255] R10: 0000000000003666 R11: 0000000000000246 R12: 0000000002533250 [25206.045186] R13: 0000000000020db0 R14: 0000000000000000 R15: 0000000000000000 [25206.046117] FS: 00007f2805635740(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000 [25206.047047] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25206.047954] CR2: 00000000019fcad0 CR3: 000000009edfa000 CR4: 00000000001407f0 [25206.048866] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25206.049779] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25206.051604] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 91.159 msecs [25233.880621] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c251:17822] [25233.881750] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc [25233.887204] CPU: 1 PID: 17822 Comm: trinity-c251 Tainted: G L 3.18.0+ #108 [25233.889315] task: ffff8801c65ec1a0 ti: ffff88009b7b0000 task.ti: ffff88009b7b0000 [25233.890406] RIP: 0010:[] [] copy_user_enhanced_fast_string+0x5/0x10 [25233.891548] RSP: 0018:ffff88009b7b3be0 EFLAGS: 00010206 [25233.892666] RAX: 00007f95d93d2239 RBX: ffffffff8117faa1 RCX: 00000000000000a0 [25233.893801] RDX: 0000000000001000 RSI: 00007f95d93d3199 RDI: ffff88019049ef60 [25233.894943] RBP: ffff88009b7b3c28 R08: ffff88022f8d1668 R09: 0000000000000000 [25233.896102] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000 [25233.897252] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000003 [25233.898395] FS: 00007f95dc296740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000 [25233.899559] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25233.900733] CR2: 00007f95da9e0bbe CR3: 0000000095a94000 CR4: 00000000001407e0 [25233.901878] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25233.903009] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25233.904151] Stack: [25233.905253] ffffffff8118e606 ffff88009b7b3bf8 000000001bb22000 0000000000001000 [25233.906364] 000000001bb22000 0000000000001000 ffff88009b7b3d60 0000000000000000 [25233.907478] ffff8800a2518d28 ffff88009b7b3cc8 ffffffff81165307 ffff88009b7b3c88 [25233.908613] Call Trace: [25233.909759] [] ? iov_iter_copy_from_user_atomic+0x156/0x180 [25233.910938] [] generic_perform_write+0xf7/0x1f0 [25233.912164] [] __generic_file_write_iter+0x15f/0x350 [25233.913376] [] ? new_sync_read+0xd0/0xd0 [25233.914574] [] generic_file_write_iter+0x3d/0xb0 [25233.915730] [] ? __generic_file_write_iter+0x350/0x350 [25233.916890] [] do_iter_readv_writev+0x78/0xc0 [25233.918061] [] do_readv_writev+0xd8/0x2a0 [25233.919202] [] ? __generic_file_write_iter+0x350/0x350 [25233.920376] [] ? __generic_file_write_iter+0x350/0x350 [25233.921584] [] ? __hrtimer_start_range_ns+0x252/0x380 [25233.922742] [] ? hrtimer_start+0x18/0x20 [25233.923883] [] vfs_writev+0x39/0x50 [25233.925034] [] SyS_writev+0x59/0xf0 [25233.926189] [] system_call_fastpath+0x12/0x17 [25233.927320] Code: 48 ff c6 48 ff c7 ff c9 75 f2 89 d1 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 31 c0 0f 1f 00 c3 0f 1f 80 00 00 00 00 0f 1f 00 89 d1 a4 31 c0 0f 1f 00 c3 90 90 90 0f 1f 00 83 fa 08 0f 82 95 00 [25233.929775] sending NMI to other CPUs: [25233.930989] NMI backtrace for cpu 0 [25233.932037] CPU: 0 PID: 17755 Comm: trinity-c184 Tainted: G L 3.18.0+ #108 [25233.934172] task: ffff88022d25cc90 ti: ffff880229f78000 task.ti: ffff880229f78000 [25233.935239] RIP: 0010:[] [] usb_unanchor_urb+0x12/0x60 [25233.936302] RSP: 0018:ffff88024e203d50 EFLAGS: 00000082 [25233.937352] RAX: 0000000000000000 RBX: ffff880242a2ecb0 RCX: 0000000000000000 [25233.938412] RDX: 000000000000000e RSI: 0000000000000046 RDI: ffff880242a2ecb0 [25233.939464] RBP: ffff88024e203d68 R08: ffffffff81f5dea8 R09: 00000000000fffe2 [25233.940527] R10: fffffffff8000000 R11: 0000000000000004 R12: ffff88009f014f38 [25233.941592] R13: 0000000000000000 R14: 0000000000000000 R15: ffff880242080000 [25233.942660] FS: 00007f95dc296740(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000 [25233.943738] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25233.944808] CR2: 0000003370219050 CR3: 000000022e902000 CR4: 00000000001407f0 [25233.945886] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25233.946955] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25233.948024] Stack: [25233.949088] ffff88009f014f38 0000000000000000 0000000000000000 ffff88024e203d98 [25233.950171] ffffffff8158fc9c ffff880241d1d6c0 ffff880242a2ecb0 ffff8802416a4558 [25233.951243] ffff880241d1d6c0 ffff88024e203dc8 ffffffff8158fec0 ffff88024e203dc8 [25233.952305] Call Trace: [25233.953331] [25233.954339] [] __usb_hcd_giveback_urb+0x5c/0x120 [25233.955323] [] usb_hcd_giveback_urb+0x40/0xf0 [25233.956290] [] xhci_irq+0xccf/0x1bc0 [25233.957234] [] xhci_msi_irq+0x11/0x20 [25233.958161] [] handle_irq_event_percpu+0x3e/0x1c0 [25233.959072] [] handle_irq_event+0x42/0x70 [25233.959958] [] handle_edge_irq+0x7f/0x150 [25233.960829] [] handle_irq+0xb1/0x140 [25233.961691] [] do_IRQ+0x53/0x100 [25233.962544] [] common_interrupt+0x6f/0x6f [25233.963393] [25233.964240] [] ? unlock_page+0x18/0x90 [25233.965088] [] shmem_undo_range+0x1b2/0x710 [25233.965941] [] shmem_truncate_range+0x18/0x40 [25233.966786] [] shmem_setattr+0x110/0x1a0 [25233.967622] [] notify_change+0x241/0x390 [25233.968453] [] do_truncate+0x73/0xc0 [25233.969279] [] do_sys_ftruncate.constprop.14+0x10c/0x160 [25233.970113] [] ? trace_hardirqs_on_thunk+0x3a/0x3c [25233.970943] [] SyS_ftruncate+0xe/0x10 [25233.971761] [] system_call_fastpath+0x12/0x17 [25233.972581] Code: 00 49 8d 7c 24 10 31 c9 ba 01 00 00 00 be 03 00 00 00 e8 42 50 b2 ff eb cd 0f 1f 44 00 00 55 48 85 ff 48 89 e5 41 56 41 55 41 54 <53> 48 89 fb 74 35 4c 8b 67 40 4d 85 e4 74 2c 4d 8d 6c 24 28 4c [25233.974377] NMI backtrace for cpu 3 [25233.975270] CPU: 3 PID: 17777 Comm: trinity-c206 Tainted: G L 3.18.0+ #108 [25233.977090] task: ffff880228aa2bc0 ti: ffff88022936c000 task.ti: ffff88022936c000 [25233.978013] RIP: 0010:[] [] unlock_page+0x25/0x90 [25233.978949] RSP: 0018:ffff88022936fbf8 EFLAGS: 00000202 [25233.979880] RAX: ffffea0008ee2d80 RBX: ffffea0008ee2d80 RCX: 00000000146c9000 [25233.980816] RDX: 002ffe000008001c RSI: 9e37fffffffc0001 RDI: ffffea0008ee2d80 [25233.981761] RBP: ffff88022936fbf8 R08: 0000000000001000 R09: ffffea0008ee2d80 [25233.982702] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000001000 [25233.983653] R13: ffff88022936fd60 R14: 0000000000000000 R15: ffff8800a251da08 [25233.984603] FS: 00007f95dc296740(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000 [25233.985565] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25233.986528] CR2: 0000000000000008 CR3: 000000022d223000 CR4: 00000000001407e0 [25233.987505] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25233.988484] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25233.989436] Stack: [25233.990361] ffff88022936fc28 ffffffff8117f380 00000000146c8000 0000000000001000 [25233.991308] ffff88022936fd60 0000000000000000 ffff88022936fcc8 ffffffff8116532a [25233.992263] ffff88022936fd68 0000000000001000 ffff880228aa2bc0 00000000811f1681 [25233.993212] Call Trace: [25233.994140] [] shmem_write_end+0x40/0xf0 [25233.995056] [] generic_perform_write+0x11a/0x1f0 [25233.995969] [] __generic_file_write_iter+0x15f/0x350 [25233.996873] [] ? new_sync_read+0xd0/0xd0 [25233.997776] [] generic_file_write_iter+0x3d/0xb0 [25233.998673] [] ? __generic_file_write_iter+0x350/0x350 [25233.999570] [] do_iter_readv_writev+0x78/0xc0 [25234.000462] [] do_readv_writev+0xd8/0x2a0 [25234.001357] [] ? __generic_file_write_iter+0x350/0x350 [25234.002277] [] ? __generic_file_write_iter+0x350/0x350 [25234.003155] [] ? __hrtimer_start_range_ns+0x252/0x380 [25234.004045] [] ? hrtimer_start+0x18/0x20 [25234.004942] [] vfs_writev+0x39/0x50 [25234.005827] [] SyS_writev+0x59/0xf0 [25234.006705] [] system_call_fastpath+0x12/0x17 [25234.007581] Code: 1f 80 00 00 00 00 0f 1f 44 00 00 55 48 89 f8 48 8b 17 48 89 e5 83 e2 01 74 60 f0 80 27 fe 48 be 01 00 fc ff ff ff 37 9e 48 8b 17 <48> 0f af f7 48 89 d1 48 c1 ea 34 83 e2 03 48 c1 e9 36 48 8d 14 [25234.009513] NMI backtrace for cpu 2 [25234.010438] CPU: 2 PID: 17723 Comm: trinity-c152 Tainted: G L 3.18.0+ #108 [25234.012415] task: ffff880229a50af0 ti: ffff88022e918000 task.ti: ffff88022e918000 [25234.013417] RIP: 0010:[] [] read_hpet+0x16/0x20 [25234.014445] RSP: 0018:ffff88022e91be58 EFLAGS: 00000046 [25234.015465] RAX: 000000002b67efe4 RBX: 000000000051c224 RCX: 0000000000000004 [25234.016505] RDX: 0000000000004949 RSI: ffff88022e91be98 RDI: ffffffff81c26f40 [25234.017547] RBP: ffff88022e91be58 R08: 0000000000000004 R09: 00000000004323f1 [25234.018587] R10: 0000000000000004 R11: 0000000000000202 R12: 000016f68c124117 [25234.019640] R13: 0000000000000000 R14: ffff880229825108 R15: 0000000000000000 [25234.020693] FS: 00007f95dc296740(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000 [25234.021740] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25234.022761] CR2: 00007f95dc055220 CR3: 000000014fecd000 CR4: 00000000001407e0 [25234.023813] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25234.024844] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25234.025871] Stack: [25234.026890] ffff88022e91be88 ffffffff810e0d3e ffff88022e91be98 ffff880229825108 [25234.027926] 000016f6dfac1843 ffff88022e91bf18 ffff88022e91beb8 ffffffff810da9da [25234.028977] 0000000000000086 00000000749a29d5 ffff880229825108 ffff88022e91bf38 [25234.030035] Call Trace: [25234.031074] [] ktime_get+0x3e/0xa0 [25234.032113] [] hrtimer_get_remaining+0x3a/0x70 [25234.033154] [] itimer_get_remtime+0x16/0x60 [25234.034172] [] do_setitimer+0xb2/0x260 [25234.035175] [] alarm_setitimer+0x49/0x90 [25234.036173] [] SyS_alarm+0xe/0x20 [25234.037171] [] system_call_fastpath+0x12/0x17 [25234.038136] Code: 00 29 c7 ba 00 00 00 00 b8 c2 ff ff ff 83 ff 7f 5d 0f 4f c2 c3 0f 1f 44 00 00 55 48 8b 05 d3 c8 ec 00 48 89 e5 8b 80 f0 00 00 00 <89> c0 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 8b 0d 29 c8 ec 00 [25234.040197] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 109.194 msecs [25261.864386] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trinity-c24:30558] [25261.865435] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc [25261.871111] CPU: 1 PID: 30558 Comm: trinity-c24 Tainted: G L 3.18.0+ #108 [25261.873609] task: ffff88009b63abc0 ti: ffff8802283d4000 task.ti: ffff8802283d4000 [25261.874875] RIP: 0033:[<0000000000412fc8>] [<0000000000412fc8>] 0x412fc8 [25261.876181] RSP: 002b:00007fff73bbaf70 EFLAGS: 00000202 [25261.877378] RAX: 00000000759aa432 RBX: ffffffff81799dfa RCX: 0000000000000038 [25261.878584] RDX: 000000000bc2aa05 RSI: 00007fff73bbaf4c RDI: 000000336f1b76e0 [25261.879738] RBP: 0000000000000002 R08: 000000336f1b70e8 R09: 000000336f1b7140 [25261.880885] R10: 0000000000000000 R11: 0000000000000206 R12: ffff8802283d7f78 [25261.882058] R13: 0000000000000000 R14: ffff8802283d4000 R15: ffff88009b63abc0 [25261.883248] FS: 00007f95dc296740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000 [25261.884396] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25261.885546] CR2: 0000000000000001 CR3: 0000000235a45000 CR4: 00000000001407e0 [25261.886701] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25261.887903] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25261.890230] sending NMI to other CPUs: [25261.891374] NMI backtrace for cpu 0 [25261.892452] CPU: 0 PID: 30551 Comm: trinity-c4 Tainted: G L 3.18.0+ #108 [25261.894667] task: ffff880229a515e0 ti: ffff88000e014000 task.ti: ffff88000e014000 [25261.895791] RIP: 0010:[] [] system_call+0x0/0x3 [25261.896927] RSP: 0018:00007fff73bbafa8 EFLAGS: 00000046 [25261.898061] RAX: 000000000000003d RBX: 0000000000007780 RCX: 000000336eebc2fc [25261.899206] RDX: 000000000000000b RSI: 00007fff73bbafb0 RDI: 0000000000007780 [25261.900327] RBP: 0000000000000000 R08: 00007f95dc296740 R09: 0000000000000000 [25261.901427] R10: 0000000000000000 R11: 0000000000000246 R12: 00007f95dc0cb000 [25261.902524] R13: 00007f95dc0cb068 R14: 0000000000000000 R15: 0000000000000000 [25261.903615] FS: 00007f95dc296740(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000 [25261.904695] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25261.905755] CR2: 000000336f1b7740 CR3: 0000000095b7d000 CR4: 00000000001407f0 [25261.906819] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25261.907870] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25261.908922] Stack: [25261.909964] 000000000041401b 00007f95dc216768 00007f95dc0cb000 0000000000000000 [25261.911029] 00007f95dc0cb07c 00007f95dc0cb000 0000000000000000 00007f95dc0cb07c [25261.912090] 0000000000416c03 00000000000042bc 0000000000000017 00000000cccccccd [25261.913152] Call Trace: [25261.914193] [25261.914200] Code: [25261.915236] 8b 3c 24 4c 8b 74 24 08 4c 8b 6c 24 10 4c 8b 64 24 18 48 8b 6c 24 20 48 8b 5c 24 28 48 83 c4 30 e9 6f 02 00 00 66 0f 1f 44 00 00 <0f> 01 f8 65 48 89 24 25 80 a0 00 00 65 48 8b 24 25 88 a8 00 00 [25261.917470] NMI backtrace for cpu 3 [25261.918589] CPU: 3 PID: 30572 Comm: trinity-c230 Tainted: G L 3.18.0+ #108 [25261.920852] task: ffff8800967736b0 ti: ffff88022cbc0000 task.ti: ffff88022cbc0000 [25261.922001] RIP: 0010:[] [] iov_iter_fault_in_readable+0x64/0x80 [25261.923165] RSP: 0018:ffff88022cbc3c18 EFLAGS: 00000206 [25261.924316] RAX: 0000000000000000 RBX: 00000000029ef000 RCX: 0000000000001007 [25261.925475] RDX: 00007f95d952b800 RSI: 0000000000001000 RDI: 000000000001e800 [25261.926633] RBP: ffff88022cbc3c28 R08: ffff880066aca360 R09: 0000000000094524 [25261.927794] R10: ffff88022cbc3af8 R11: 00000000000003ff R12: 0000000000001000 [25261.928954] R13: ffff88022cbc3d60 R14: 0000000000001000 R15: ffff8800a2518d28 [25261.930112] FS: 00007f95dc296740(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000 [25261.931280] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25261.932458] CR2: 00007f95d9a2036e CR3: 000000022b2d6000 CR4: 00000000001407e0 [25261.933618] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25261.934753] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25261.935873] Stack: [25261.936974] 0000000000001000 00ff8800a2518d28 ffff88022cbc3cc8 ffffffff811652b8 [25261.938081] ffff88022cbc3c98 0000000000001000 ffff8800967736b0 00000000811f1681 [25261.939164] 0000000000f60000 0000000000000000 ffff88009f3c9400 ffffffff81828380 [25261.940227] Call Trace: [25261.941261] [] generic_perform_write+0xa8/0x1f0 [25261.942290] [] __generic_file_write_iter+0x15f/0x350 [25261.943300] [] ? new_sync_read+0xd0/0xd0 [25261.944291] [] generic_file_write_iter+0x3d/0xb0 [25261.945270] [] ? __generic_file_write_iter+0x350/0x350 [25261.946243] [] do_iter_readv_writev+0x78/0xc0 [25261.947210] [] do_readv_writev+0xd8/0x2a0 [25261.948174] [] ? __generic_file_write_iter+0x350/0x350 [25261.949141] [] ? __generic_file_write_iter+0x350/0x350 [25261.950106] [] ? __hrtimer_start_range_ns+0x252/0x380 [25261.951068] [] ? hrtimer_start+0x18/0x20 [25261.952027] [] vfs_writev+0x39/0x50 [25261.952982] [] SyS_writev+0x59/0xf0 [25261.953928] [] system_call_fastpath+0x12/0x17 [25261.954865] Code: 10 0f 1f 00 40 8a 39 0f 1f 00 85 d2 40 88 7d ff 75 2d 48 63 f6 48 8d 54 31 ff 48 31 d1 48 f7 c1 00 f0 ff ff 74 0f 0f 1f 00 8a 12 <0f> 1f 00 88 55 ff 0f b6 55 ff c9 f3 c3 0f 1f 80 00 00 00 00 c9 [25261.956947] NMI backtrace for cpu 2 [25261.957928] CPU: 2 PID: 30581 Comm: trinity-c42 Tainted: G L 3.18.0+ #108 [25261.959978] task: ffff88022c1f2bc0 ti: ffff8800a2604000 task.ti: ffff8800a2604000 [25261.961020] RIP: 0033:[<00007fff73bd3bfe>] [<00007fff73bd3bfe>] 0x7fff73bd3bfe [25261.962065] RSP: 002b:00007fff73bbaf20 EFLAGS: 00000246 [25261.963081] RAX: 0000000043481221 RBX: 00007f95dbedd068 RCX: 000000336eeef4b9 [25261.964103] RDX: 0000000000000002 RSI: 0000000000000000 RDI: 00007f95dbedd000 [25261.965115] RBP: 00007fff73bbaf20 R08: 002b34bd7fef2fdf R09: 00000000004d1e4c [25261.966125] R10: 0000000000000002 R11: 0000000000000246 R12: 00007f95dbedd000 [25261.967135] R13: 00000000000001ff R14: 0000000000000000 R15: 00007f95dc2966a0 [25261.968161] FS: 00007f95dc296740(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000 [25261.969187] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25261.970198] CR2: 0000000000000008 CR3: 0000000040239000 CR4: 00000000001407e0 [25261.971236] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25261.972260] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25289.848160] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [trinity-c27:31317] [25289.849258] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc [25289.855194] CPU: 1 PID: 31317 Comm: trinity-c27 Tainted: G L 3.18.0+ #108 [25289.857533] task: ffff8802415a0af0 ti: ffff8802290e4000 task.ti: ffff8802290e4000 [25289.858723] RIP: 0010:[] [] clear_page_c_e+0x7/0x10 [25289.859915] RSP: 0000:ffff8802290e7ce0 EFLAGS: 00010246 [25289.861099] RAX: 0000000000000000 RBX: ffff8802415a0af0 RCX: 00000000000001c0 [25289.862267] RDX: 0000000000007a55 RSI: 0000000000000ead RDI: ffff880123d7ae40 [25289.863466] RBP: ffff8802290e7d18 R08: ffffea00048f0000 R09: 00000000000160f0 [25289.864660] R10: ffff88024e5d4d80 R11: 0000000000000000 R12: 0000000000000000 [25289.865808] R13: ffff8802415a0af0 R14: ffff88024e5d5b00 R15: 0000000200000000 [25289.866970] FS: 00007fd3c5236740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000 [25289.868134] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25289.869294] CR2: 0000000001c00000 CR3: 0000000227866000 CR4: 00000000001407e0 [25289.870491] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25289.871683] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25289.872843] Stack: [25289.874017] ffffffff81196633 ffff880235814070 0000000001c00000 0000000000000000 [25289.875201] ffff8800967dcc00 ffff880235814070 ffff8801c64b4600 ffff8802290e7d78 [25289.876372] ffffffff811ca951 ffff8802290e7e18 ffffffff8115f002 ffffea00048f0000 [25289.877537] Call Trace: [25289.878705] [] ? clear_huge_page+0xa3/0x160 [25289.879878] [] do_huge_pmd_anonymous_page+0x151/0x3b0 [25289.881085] [] ? __perf_sw_event+0x1b2/0x200 [25289.882274] [] handle_mm_fault+0x14b/0xe90 [25289.883439] [] __do_page_fault+0x20c/0x610 [25289.884554] [] ? do_brk+0x24d/0x350 [25289.885663] [] ? trace_hardirqs_off_thunk+0x3a/0x6c [25289.886786] [] do_page_fault+0xc/0x10 [25289.887911] [] page_fault+0x22/0x30 [25289.889033] Code: bc 0f 1f 00 e8 fb fa d1 ff 90 90 90 90 90 90 90 90 90 90 90 b9 00 02 00 00 31 c0 f3 48 ab c3 0f 1f 44 00 00 b9 00 10 00 00 31 c0 aa c3 66 0f 1f 44 00 00 eb ee 0f 1f 84 00 00 00 00 00 0f 1f [25289.891387] sending NMI to other CPUs: [25289.892488] NMI backtrace for cpu 2 [25289.893575] CPU: 2 PID: 350 Comm: trinity-c163 Tainted: G L 3.18.0+ #108 [25289.895771] task: ffff8802293236b0 ti: ffff88009ab34000 task.ti: ffff88009ab34000 [25289.896886] RIP: 0010:[] [] __list_add+0x1a/0xc0 [25289.898020] RSP: 0000:ffff88009ab37b08 EFLAGS: 00000092 [25289.899149] RAX: 0000000000000030 RBX: ffffea00079e7c20 RCX: 000000000000000d [25289.900265] RDX: ffffea00079e7c60 RSI: ffff88024e296120 RDI: ffffea00079e7c20 [25289.901361] RBP: ffff88009ab37b28 R08: ffff88024e296120 R09: 00000000001e79f0 [25289.902457] R10: 0000000000000002 R11: 0000000000000002 R12: ffffea00079e7c60 [25289.903549] R13: ffff88024e296120 R14: 0000000000000002 R15: 0000000000000202 [25289.904623] FS: 00007fd3c5236740(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000 [25289.905686] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25289.906746] CR2: 000000336ef629f2 CR3: 00000002359ea000 CR4: 00000000001407e0 [25289.907809] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25289.908869] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25289.909925] Stack: [25289.910974] 002ffe000008000c ffffea00079e7c00 ffff88024e2960f0 0000000000000000 [25289.912045] ffff88009ab37b78 ffffffff8116d218 00000000001e79f0 ffff88024e5d4d80 [25289.913122] ffff880238414300 ffffea00079e7cc0 ffffea00079e7c00 0000000000000000 [25289.914193] Call Trace: [25289.915250] [] free_hot_cold_page+0x148/0x1b0 [25289.916316] [] free_hot_cold_page_list+0x4e/0xc0 [25289.917375] [] ? locks_alloc_lock+0x1f/0x70 [25289.918438] [] release_pages+0x1c2/0x280 [25289.919489] [] __pagevec_release+0x43/0x60 [25289.920534] [] shmem_undo_range+0x460/0x710 [25289.921557] [] shmem_truncate_range+0x18/0x40 [25289.922553] [] shmem_setattr+0x110/0x1a0 [25289.923527] [] notify_change+0x241/0x390 [25289.924478] [] do_truncate+0x73/0xc0 [25289.925401] [] do_sys_ftruncate.constprop.14+0x10c/0x160 [25289.926315] [] ? trace_hardirqs_on_thunk+0x3a/0x3c [25289.927214] [] compat_SyS_ftruncate+0x10/0x20 [25289.928093] [] ia32_do_call+0x13/0x13 [25289.928958] Code: ff ff e9 3b ff ff ff b8 f4 ff ff ff e9 31 ff ff ff 55 48 89 e5 41 55 49 89 f5 41 54 49 89 d4 53 48 89 fb 48 83 ec 08 4c 8b 42 08 <49> 39 f0 75 2e 4d 8b 45 00 4d 39 c4 75 6c 4c 39 e3 74 42 4c 39 [25289.930848] NMI backtrace for cpu 3 [25289.931762] CPU: 3 PID: 31454 Comm: trinity-c164 Tainted: G L 3.18.0+ #108 [25289.933619] task: ffff88009f3f95e0 ti: ffff880229bf4000 task.ti: ffff880229bf4000 [25289.934573] RIP: 0010:[] [] cancel_dirty_page+0x39/0xc0 [25289.935536] RSP: 0000:ffff880229bf7c28 EFLAGS: 00000286 [25289.936482] RAX: 00000000fffffffb RBX: ffff8800a2518d28 RCX: 0000000000001fdd [25289.937434] RDX: ffffea0008433d00 RSI: 0000000000001000 RDI: ffffea0008433d00 [25289.938386] RBP: ffff880229bf7c48 R08: 000000000001182a R09: ffff880121d39fc8 [25289.939345] R10: ffff880229bf7bd8 R11: 0000000000000220 R12: 0000000000001000 [25289.940295] R13: ffff880229bf7d40 R14: 0000000000000000 R15: 00000000000118ee [25289.941237] FS: 00007fd3c5236740(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000 [25289.942185] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25289.943137] CR2: 00000000014f4fd8 CR3: 000000010750d000 CR4: 00000000001407e0 [25289.944088] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25289.945036] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25289.945982] Stack: [25289.946922] ffff880229bf7d40 ffffea0008433d00 ffff8800a2518d28 ffff880229bf7d40 [25289.947887] ffff880229bf7c68 ffffffff811758fe 000000000000000a ffff880229bf7cd0 [25289.948859] ffff880229bf7df8 ffffffff81180fab ffffea00027bbe00 ffff8800a2518be0 [25289.949826] Call Trace: [25289.950786] [] truncate_inode_page+0x4e/0x90 [25289.951752] [] shmem_undo_range+0x41b/0x710 [25289.952721] [] shmem_truncate_range+0x18/0x40 [25289.953666] [] shmem_setattr+0x110/0x1a0 [25289.954589] [] notify_change+0x241/0x390 [25289.955510] [] do_truncate+0x73/0xc0 [25289.956428] [] do_sys_ftruncate.constprop.14+0x10c/0x160 [25289.957345] [] ? trace_hardirqs_on_thunk+0x3a/0x3c [25289.958247] [] SyS_ftruncate+0xe/0x10 [25289.959129] [] system_call_fastpath+0x12/0x17 [25289.960005] Code: 89 f4 53 48 83 ec 08 f0 0f ba 37 04 72 14 48 83 c4 08 5b 41 5c 41 5d 5d c3 66 0f 1f 84 00 00 00 00 00 48 8b 5f 08 48 85 db 74 e3 <48> 8b 83 88 00 00 00 f6 40 20 01 75 d6 be 0b 00 00 00 e8 f0 e9 [25289.961909] NMI backtrace for cpu 0 [25289.962833] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G L 3.18.0+ #108 [25289.964723] task: ffffffff81c16460 ti: ffffffff81c00000 task.ti: ffffffff81c00000 [25289.965679] RIP: 0010:[] [] intel_idle+0xdb/0x180 [25289.966648] RSP: 0018:ffffffff81c03e58 EFLAGS: 00000046 [25289.967596] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001 [25289.968548] RDX: 0000000000000000 RSI: ffffffff81c03fd8 RDI: 0000000000000000 [25289.969493] RBP: ffffffff81c03e88 R08: 0000000000000018 R09: 00000000000006c9 [25289.970443] R10: 0000000000000fc9 R11: 0000000000000400 R12: 0000000000000005 [25289.971384] R13: 0000000000000032 R14: 0000000000000004 R15: ffffffff81c00000 [25289.972314] FS: 0000000000000000(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000 [25289.973248] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25289.974183] CR2: 0000000000000008 CR3: 0000000001c11000 CR4: 00000000001407f0 [25289.975117] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25289.976047] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25289.976979] Stack: [25289.977905] 0000000081c03e88 b7fee4166800c9e8 ffffe8ffffc01ba8 0000000000000005 [25289.978865] ffffffff81c9e440 0000000000000000 ffffffff81c03ed8 ffffffff8163a995 [25289.979820] 00001703ad828700 ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330 [25289.980783] Call Trace: [25289.981737] [] cpuidle_enter_state+0x55/0x190 [25289.982705] [] cpuidle_enter+0x17/0x20 [25289.983669] [] cpu_startup_entry+0x355/0x410 [25289.984636] [] rest_init+0x87/0x90 [25289.985602] [] start_kernel+0x483/0x4a4 [25289.986565] [] ? set_init_arg+0x55/0x55 [25289.987521] [] x86_64_start_reservations+0x2a/0x2c [25289.988488] [] x86_64_start_kernel+0xf2/0xf6 [25289.989425] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 [25317.831994] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [swapper/1:0] [25317.833213] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc [25317.839376] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G L 3.18.0+ #108 [25317.841869] task: ffff8802444fed60 ti: ffff880244704000 task.ti: ffff880244704000 [25317.843147] RIP: 0010:[] [] cpuidle_enter_state+0x79/0x190 [25317.844445] RSP: 0018:ffff880244707e48 EFLAGS: 00000246 [25317.845727] RAX: 0000170a2fdbc2c9 RBX: ffffffff81799e0d RCX: 0000000000000019 [25317.847008] RDX: 20c49ba5e353f7cf RSI: ffff880244707fd8 RDI: ffffffff81c26f40 [25317.848288] RBP: ffff880244707e88 R08: ffff88024e24ecf0 R09: 00000000ffffffff [25317.849570] R10: 000000000000262f R11: 0000000000aaaaaa R12: ffff880244707db8 [25317.850860] R13: 0000000000000000 R14: ffff880244704000 R15: ffff8802444fed60 [25317.852157] FS: 0000000000000000(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000 [25317.853393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25317.854633] CR2: 00007fda8919e000 CR3: 0000000001c11000 CR4: 00000000001407e0 [25317.855885] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25317.857121] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25317.858353] Stack: [25317.859581] 0000170a2f44020c ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330 [25317.860841] ffffe8ffffc41ba8 ffff880244704000 ffffffff81c9e440 ffff880244704000 [25317.862108] ffff880244707e98 ffffffff8163ab87 ffff880244707f08 ffffffff810b9095 [25317.863330] Call Trace: [25317.864545] [] cpuidle_enter+0x17/0x20 [25317.865774] [] cpu_startup_entry+0x355/0x410 [25317.866999] [] start_secondary+0x1aa/0x230 [25317.868218] Code: c8 48 89 df ff 50 48 41 89 c5 e8 63 63 aa ff 44 8b 63 04 49 89 c7 0f 1f 44 00 00 e8 12 f8 af ff fb 48 ba cf f7 53 e3 a5 9b c4 20 <4c> 2b 7d c0 4c 89 f8 49 c1 ff 3f 48 f7 ea b8 ff ff ff 7f 48 c1 [25317.870820] sending NMI to other CPUs: [25317.872055] NMI backtrace for cpu 0 [25317.873204] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G L 3.18.0+ #108 [25317.875524] task: ffffffff81c16460 ti: ffffffff81c00000 task.ti: ffffffff81c00000 [25317.876676] RIP: 0010:[] [] intel_idle+0xdb/0x180 [25317.877832] RSP: 0018:ffffffff81c03e58 EFLAGS: 00000046 [25317.878979] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001 [25317.880133] RDX: 0000000000000000 RSI: ffffffff81c03fd8 RDI: 0000000000000000 [25317.881280] RBP: ffffffff81c03e88 R08: 0000000000000018 R09: 0000000000000401 [25317.882431] R10: 0000000000000f4c R11: 0000000000000000 R12: 0000000000000005 [25317.883583] R13: 0000000000000032 R14: 0000000000000004 R15: ffffffff81c00000 [25317.884735] FS: 0000000000000000(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000 [25317.885886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25317.887030] CR2: 00000000006392c8 CR3: 0000000001c11000 CR4: 00000000001407f0 [25317.888181] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25317.889331] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25317.890475] Stack: [25317.891613] 0000000081c03e88 b7fee4166800c9e8 ffffe8ffffc01ba8 0000000000000005 [25317.892770] ffffffff81c9e440 0000000000000000 ffffffff81c03ed8 ffffffff8163a995 [25317.893926] 0000170a323f875b ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330 [25317.895082] Call Trace: [25317.896217] [] cpuidle_enter_state+0x55/0x190 [25317.897361] [] cpuidle_enter+0x17/0x20 [25317.898503] [] cpu_startup_entry+0x355/0x410 [25317.899646] [] rest_init+0x87/0x90 [25317.900788] [] start_kernel+0x483/0x4a4 [25317.901931] [] ? set_init_arg+0x55/0x55 [25317.903073] [] x86_64_start_reservations+0x2a/0x2c [25317.904216] [] x86_64_start_kernel+0xf2/0xf6 [25317.905357] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 [25317.907755] NMI backtrace for cpu 3 [25317.908965] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G L 3.18.0+ #108 [25317.911350] task: ffff8802444fe270 ti: ffff880244718000 task.ti: ffff880244718000 [25317.912485] RIP: 0010:[] [] intel_idle+0xdb/0x180 [25317.913548] RSP: 0018:ffff88024471be08 EFLAGS: 00000046 [25317.914602] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001 [25317.915637] RDX: 0000000000000000 RSI: ffff88024471bfd8 RDI: 0000000000000003 [25317.916642] RBP: ffff88024471be38 R08: ffff88024e2cecf0 R09: 00000000ffffffff [25317.917629] R10: 000000000000263b R11: 0000000000000400 R12: 0000000000000005 [25317.918611] R13: 0000000000000032 R14: 0000000000000004 R15: ffff880244718000 [25317.919581] FS: 0000000000000000(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000 [25317.920557] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25317.921528] CR2: 0000000000000024 CR3: 0000000001c11000 CR4: 00000000001407e0 [25317.922527] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25317.923502] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25317.924477] Stack: [25317.925437] 000000034471be38 5fbbf5c12849ac78 ffffe8ffffcc1ba8 0000000000000005 [25317.926419] ffffffff81c9e440 0000000000000003 ffff88024471be88 ffffffff8163a995 [25317.927423] 0000170a323e077e ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330 [25317.928430] Call Trace: [25317.929409] [] cpuidle_enter_state+0x55/0x190 [25317.930397] [] cpuidle_enter+0x17/0x20 [25317.931370] [] cpu_startup_entry+0x355/0x410 [25317.932373] [] start_secondary+0x1aa/0x230 [25317.933346] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 [25317.935489] NMI backtrace for cpu 2 [25317.936503] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G L 3.18.0+ #108 [25317.938500] task: ffff8802444f8af0 ti: ffff88024470c000 task.ti: ffff88024470c000 [25317.939486] RIP: 0010:[] [] intel_idle+0xdb/0x180 [25317.940485] RSP: 0018:ffff88024470fe08 EFLAGS: 00000046 [25317.941483] RAX: 0000000000000032 RBX: 0000000000000010 RCX: 0000000000000001 [25317.942501] RDX: 0000000000000000 RSI: ffff88024470ffd8 RDI: 0000000000000002 [25317.943457] RBP: ffff88024470fe38 R08: ffff88024e28ecf0 R09: 00000000ffffffff [25317.944395] R10: 0000000000002639 R11: 0000000000000400 R12: 0000000000000005 [25317.945329] R13: 0000000000000032 R14: 0000000000000004 R15: ffff88024470c000 [25317.946259] FS: 0000000000000000(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000 [25317.947190] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25317.948124] CR2: 0000000000639058 CR3: 0000000001c11000 CR4: 00000000001407e0 [25317.949063] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25317.950009] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25317.950953] Stack: [25317.951915] 000000024470fe38 c6aa8015cfd0a8b5 ffffe8ffffc81ba8 0000000000000005 [25317.952893] ffffffff81c9e440 0000000000000002 ffff88024470fe88 ffffffff8163a995 [25317.953885] 0000170a323df4ab ffffffff81c9e610 ffffffff81c9e440 ffffffff81cfe330 [25317.954862] Call Trace: [25317.955831] [] cpuidle_enter_state+0x55/0x190 [25317.956807] [] cpuidle_enter+0x17/0x20 [25317.957776] [] cpu_startup_entry+0x355/0x410 [25317.958741] [] start_secondary+0x1aa/0x230 [25317.959702] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 [25345.815706] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [trinity-c103:12391] [25345.816962] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc [25345.823837] CPU: 1 PID: 12391 Comm: trinity-c103 Tainted: G L 3.18.0+ #108 [25345.826600] task: ffff88022911abc0 ti: ffff880143340000 task.ti: ffff880143340000 [25345.827931] RIP: 0010:[] [] __slab_alloc+0x4e5/0x53b [25345.829277] RSP: 0018:ffff880143343ca0 EFLAGS: 00000246 [25345.830607] RAX: 0000000000000001 RBX: ffff880143343c60 RCX: 000000018020001e [25345.831928] RDX: 0000000000000020 RSI: ffffea00076ef700 RDI: ffff880245825680 [25345.833232] RBP: ffff880143343d70 R08: ffff8801dbbdc000 R09: 0000000000000000 [25345.834528] R10: 0000000000000082 R11: 00003ffffffff000 R12: 00000000000000bb [25345.835836] R13: ffffffff810133af R14: ffff880143343c20 R15: 000000010020001d [25345.837084] FS: 00007f5c9bc2f740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000 [25345.838337] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25345.839599] CR2: 00000000020d4968 CR3: 00000002299b4000 CR4: 00000000001407e0 [25345.840857] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25345.842117] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25345.843379] Stack: [25345.844620] 01ffffff00000000 ffff88022ea27288 00007f5c97aa4fff 00007f5c97a24000 [25345.845915] 0000000000000000 ffff8801dbbdc600 0000000227852b98 ffffffff810750f5 [25345.847139] ffffffff00000000 0000000000000246 00007f5c97aa5000 00007f5c97aa4fff [25345.848385] Call Trace: [25345.849599] [] ? copy_process.part.29+0x12a5/0x1970 [25345.850835] [] ? copy_process.part.29+0x12a5/0x1970 [25345.852052] [] kmem_cache_alloc+0x1bb/0x1e0 [25345.853258] [] copy_process.part.29+0x12a5/0x1970 [25345.854460] [] ? get_empty_filp+0xdd/0x1f0 [25345.855664] [] do_fork+0xe1/0x3d0 [25345.856837] [] ? syscall_trace_enter_phase2+0xa7/0x1e0 [25345.858003] [] SyS_clone+0x16/0x20 [25345.859172] [] stub_clone+0x69/0x90 [25345.860312] [] ? tracesys_phase2+0xd4/0xd9 [25345.861429] Code: 00 02 00 00 49 c7 45 00 00 00 00 00 75 11 ff b5 78 ff ff ff 9d e8 07 b4 9a ff 4c 89 e0 eb 0f e8 fd b4 9a ff ff b5 78 ff ff ff 9d <4c> 89 e0 48 8b 75 c8 65 48 33 34 25 28 00 00 00 74 32 e8 ff 79 [25345.863833] sending NMI to other CPUs: [25345.864970] NMI backtrace for cpu 2 [25345.866051] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G L 3.18.0+ #108 [25345.868240] task: ffff8802444f8af0 ti: ffff88024470c000 task.ti: ffff88024470c000 [25345.869350] RIP: 0010:[] [] intel_idle+0xdb/0x180 [25345.870475] RSP: 0018:ffff88024470fe08 EFLAGS: 00000046 [25345.871594] RAX: 0000000000000001 RBX: 0000000000000002 RCX: 0000000000000001 [25345.872721] RDX: 0000000000000000 RSI: ffff88024470ffd8 RDI: 0000000000000002 [25345.873846] RBP: ffff88024470fe38 R08: 0000000000000018 R09: 0000000000001033 [25345.874967] R10: 000000000000331d R11: 0000000000000002 R12: 0000000000000002 [25345.876083] R13: 0000000000000001 R14: 0000000000000001 R15: ffff88024470c000 [25345.877191] FS: 0000000000000000(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000 [25345.878306] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25345.879419] CR2: 0000003370219050 CR3: 0000000229371000 CR4: 00000000001407e0 [25345.880537] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25345.881648] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25345.882753] Stack: [25345.883844] 000000024470fe38 c6aa8015cfd0a8b5 ffffe8ffffc81ba8 0000000000000002 [25345.884958] ffffffff81c9e440 0000000000000002 ffff88024470fe88 ffffffff8163a995 [25345.886071] 00001710b7b5ad88 ffffffff81c9e508 ffffffff81c9e440 ffffffff81cfe330 [25345.887189] Call Trace: [25345.888301] [] cpuidle_enter_state+0x55/0x190 [25345.889424] [] cpuidle_enter+0x17/0x20 [25345.890517] [] cpu_startup_entry+0x355/0x410 [25345.891590] [] start_secondary+0x1aa/0x230 [25345.892652] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 [25345.894882] NMI backtrace for cpu 0 [25345.895925] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G L 3.18.0+ #108 [25345.897992] task: ffffffff81c16460 ti: ffffffff81c00000 task.ti: ffffffff81c00000 [25345.899014] RIP: 0010:[] [] intel_idle+0xdb/0x180 [25345.900033] RSP: 0018:ffffffff81c03e58 EFLAGS: 00000046 [25345.901026] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000001 [25345.902020] RDX: 0000000000000000 RSI: ffffffff81c03fd8 RDI: 0000000000000000 [25345.903001] RBP: ffffffff81c03e88 R08: 0000000000000018 R09: 000000000000032a [25345.903982] R10: 00000000000003f6 R11: 00000000000fff9d R12: 0000000000000001 [25345.904960] R13: 0000000000000000 R14: 0000000000000001 R15: ffffffff81c00000 [25345.905936] FS: 0000000000000000(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000 [25345.906947] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25345.907962] CR2: 0000000000000001 CR3: 00000000973c9000 CR4: 00000000001407f0 [25345.909015] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25345.910072] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25345.911128] Stack: [25345.912132] 0000000081c03e88 b7fee4166800c9e8 ffffe8ffffc01ba8 0000000000000001 [25345.913130] ffffffff81c9e440 0000000000000000 ffffffff81c03ed8 ffffffff8163a995 [25345.914121] 00001710b7b5d674 ffffffff81c9e4b0 ffffffff81c9e440 ffffffff81cfe330 [25345.915112] Call Trace: [25345.916097] [] cpuidle_enter_state+0x55/0x190 [25345.917082] [] cpuidle_enter+0x17/0x20 [25345.918056] [] cpu_startup_entry+0x355/0x410 [25345.919028] [] rest_init+0x87/0x90 [25345.920015] [] start_kernel+0x483/0x4a4 [25345.920988] [] ? set_init_arg+0x55/0x55 [25345.921960] [] x86_64_start_reservations+0x2a/0x2c [25345.922929] [] x86_64_start_kernel+0xf2/0xf6 [25345.923900] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 [25345.926016] NMI backtrace for cpu 3 [25345.927071] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G L 3.18.0+ #108 [25345.929211] task: ffff8802444fe270 ti: ffff880244718000 task.ti: ffff880244718000 [25345.930264] RIP: 0010:[] [] intel_idle+0xdb/0x180 [25345.931337] RSP: 0018:ffff88024471be08 EFLAGS: 00000046 [25345.932420] RAX: 0000000000000001 RBX: 0000000000000002 RCX: 0000000000000001 [25345.933563] RDX: 0000000000000000 RSI: ffff88024471bfd8 RDI: 0000000000000003 [25345.934692] RBP: ffff88024471be38 R08: 0000000000000018 R09: 0000000000000d4b [25345.935804] R10: 0000000000002da4 R11: 0000000000000002 R12: 0000000000000002 [25345.936797] R13: 0000000000000001 R14: 0000000000000001 R15: ffff880244718000 [25345.937778] FS: 0000000000000000(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000 [25345.938766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25345.939752] CR2: 0000000001e81930 CR3: 00000001f9d90000 CR4: 00000000001407e0 [25345.940747] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25345.941745] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25345.942743] Stack: [25345.943728] 000000034471be38 5fbbf5c12849ac78 ffffe8ffffcc1ba8 0000000000000002 [25345.944753] ffffffff81c9e440 0000000000000003 ffff88024471be88 ffffffff8163a995 [25345.945764] 00001710b7b5d25c ffffffff81c9e508 ffffffff81c9e440 ffffffff81cfe330 [25345.946777] Call Trace: [25345.947789] [] cpuidle_enter_state+0x55/0x190 [25345.948807] [] cpuidle_enter+0x17/0x20 [25345.949812] [] cpu_startup_entry+0x355/0x410 [25345.950811] [] start_secondary+0x1aa/0x230 [25345.951826] Code: 31 d2 65 48 8b 34 25 88 a8 00 00 48 8d 86 38 c0 ff ff 48 89 d1 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e8 0f 01 c9 <65> 48 8b 0c 25 88 a8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48 [25373.799486] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [trinity-c17:29947] [25373.800608] Modules linked in: dlci bridge snd_seq_dummy fuse 8021q garp stp tun bnep rfcomm scsi_transport_iscsi af_key llc2 hidp can_bcm sctp libcrc32c can_raw nfnetlink nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep snd_seq e1000e shpchp snd_seq_device usb_debug coretemp hwmon x86_pkg_temp_thermal ptp pps_core snd_pcm nfsd snd_timer snd soundcore auth_rpcgss kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr serio_raw oid_registry nfs_acl lockd grace sunrpc [25373.806511] CPU: 1 PID: 29947 Comm: trinity-c17 Tainted: G L 3.18.0+ #108 [25373.808876] task: ffff88022908abc0 ti: ffff8801bf164000 task.ti: ffff8801bf164000 [25373.810057] RIP: 0010:[] [] find_get_pid+0x1e/0x30 [25373.811282] RSP: 0018:ffff8801bf167ee8 EFLAGS: 00000202 [25373.812535] RAX: ffff8801bf33a140 RBX: ffffffff810850e0 RCX: ffff8801bf33a170 [25373.813781] RDX: 0000000000000030 RSI: ffffffff81c44840 RDI: 0000000000007d64 [25373.815031] RBP: ffff8801bf167ee8 R08: 00007f0b34598740 R09: 0000000000000000 [25373.816283] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [25373.817515] R13: ffff8801bf167eb0 R14: ffff8801bf33a140 R15: ffff8801bf167e68 [25373.818738] FS: 00007f0b34598740(0000) GS:ffff88024e240000(0000) knlGS:0000000000000000 [25373.819941] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25373.821149] CR2: 000000336f1b7740 CR3: 0000000242a70000 CR4: 00000000001407e0 [25373.822379] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25373.823593] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25373.824799] Stack: [25373.825983] ffff8801bf167f78 ffffffff81079935 0000000f00000000 ffff8801bf33a140 [25373.827203] 0000000000000000 00007fff74741110 0000000000000000 0000000000000000 [25373.828434] ffff88022908abc0 ffffffff8135862e 0000000000000246 0000000000000000 [25373.829660] Call Trace: [25373.830833] [] SyS_wait4+0x55/0x110 [25373.832014] [] ? trace_hardirqs_on_thunk+0x3a/0x3c [25373.833206] [] system_call_fastpath+0x12/0x17 [25373.834409] Code: 80 00 00 00 00 31 c0 eb eb 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 65 ff 04 25 60 a8 00 00 e8 ca fe ff ff 48 85 c0 74 03 f0 ff 00 <65> ff 0c 25 60 a8 00 00 5d c3 0f 1f 84 00 00 00 00 00 0f 1f 44 [25373.837037] sending NMI to other CPUs: [25373.838287] NMI backtrace for cpu 0 [25373.839429] CPU: 0 PID: 32111 Comm: modprobe Tainted: G L 3.18.0+ #108 [25373.841745] task: ffff8802293be270 ti: ffff880103704000 task.ti: ffff880103704000 [25373.842890] RIP: 0010:[] [] __lookup_mnt+0x64/0x80 [25373.844045] RSP: 0018:ffff880103707bb8 EFLAGS: 00000286 [25373.845190] RAX: ffff880241670780 RBX: ffff880103707da8 RCX: 000000000000000e [25373.846327] RDX: ffff880244b56300 RSI: ffff880241b03ae8 RDI: ffff8802416707a0 [25373.847446] RBP: ffff880103707bb8 R08: 70646f6d2f6e7572 R09: ffff880103707bdc [25373.848558] R10: ffff88014331e921 R11: 0000000000000003 R12: ffff880241b03ae8 [25373.849665] R13: ffff880103707c60 R14: ffff880103707c50 R15: ffff8802416707a0 [25373.850770] FS: 00007fdddcfb2740(0000) GS:ffff88024e200000(0000) knlGS:0000000000000000 [25373.851886] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25373.853000] CR2: 00000000016211d8 CR3: 00000001c6569000 CR4: 00000000001407f0 [25373.854123] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25373.855252] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25373.856371] Stack: [25373.857476] ffff880103707c18 ffffffff811e082a 0000000000000081 ffff880200000001 [25373.858600] 0000000203707be8 00000000d85f4923 ffff880103707c08 ffff88014331e925 [25373.859730] ffff8802293be270 6f636e6c2e6d7471 00000000ffffff9c ffff880103707da8 [25373.860865] Call Trace: [25373.861983] [] lookup_fast+0xca/0x2e0 [25373.863101] [] link_path_walk+0x1a5/0x850 [25373.864213] [] path_lookupat+0x64/0x770 [25373.865321] [] ? kmem_cache_alloc+0xff/0x1e0 [25373.866424] [] ? getname_flags+0x4f/0x1a0 [25373.867522] [] filename_lookup+0x2b/0xc0 [25373.868622] [] user_path_at_empty+0x63/0xc0 [25373.869721] [] ? fput+0xa0/0xa0 [25373.870814] [] ? ext4_release_dir+0x2a/0x40 [25373.871890] [] user_path_at+0x11/0x20 [25373.872942] [] vfs_fstatat+0x63/0xc0 [25373.873985] [] ? __fput+0x188/0x1f0 [25373.875004] [] SYSC_newstat+0x2e/0x60 [25373.876000] [] ? task_work_run+0xc4/0xe0 [25373.876971] [] ? do_notify_resume+0x59/0x80 [25373.877921] [] ? trace_hardirqs_on_thunk+0x3a/0x3c [25373.878846] [] SyS_newstat+0xe/0x10 [25373.879757] [] system_call_fastpath+0x12/0x17 [25373.880651] Code: c2 48 8b 10 31 c0 48 85 d2 75 1c 5d c3 66 2e 0f 1f 84 00 00 00 00 00 48 39 72 18 74 23 48 8b 12 48 85 d2 74 17 66 90 48 8b 42 10 <48> 83 c0 20 48 39 c7 74 e3 48 8b 12 48 85 d2 75 eb 31 c0 5d c3 [25373.882565] NMI backtrace for cpu 2 [25373.883509] CPU: 2 PID: 31661 Comm: trinity-c90 Tainted: G L 3.18.0+ #108 [25373.885431] task: ffff88022e86a0d0 ti: ffff880039fc0000 task.ti: ffff880039fc0000 [25373.886415] RIP: 0010:[] [] iov_iter_fault_in_readable+0x64/0x80 [25373.887422] RSP: 0018:ffff880039fc3c18 EFLAGS: 00000206 [25373.888422] RAX: 0000000000000000 RBX: 000000000468e000 RCX: 0000000000003001 [25373.889432] RDX: 00007f0b31a7ea00 RSI: 0000000000001000 RDI: 000000000026fa00 [25373.890454] RBP: ffff880039fc3c28 R08: ffff8800978c0cf0 R09: 00000000008b89e2 [25373.891462] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000001000 [25373.892470] R13: ffff880039fc3d60 R14: 0000000000001000 R15: ffff88022e9718b0 [25373.893546] FS: 00007f0b34598740(0000) GS:ffff88024e280000(0000) knlGS:0000000000000000 [25373.894591] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25373.895602] CR2: 00007f0b33620294 CR3: 000000009f3b0000 CR4: 00000000001407e0 [25373.896666] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25373.897705] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25373.898733] Stack: [25373.899757] 0000000000001000 00ff88022e9718b0 ffff880039fc3cc8 ffffffff811652b8 [25373.900776] 0000000054940ba3 0000000000001000 ffff88022e86a0d0 0000000000000a01 [25373.901816] 000000000468dffa 0000000000000000 ffff880071a82580 ffffffff81828380 [25373.902857] Call Trace: [25373.903836] [] generic_perform_write+0xa8/0x1f0 [25373.904819] [] __generic_file_write_iter+0x15f/0x350 [25373.905807] [] ? new_sync_read+0xd0/0xd0 [25373.906776] [] generic_file_write_iter+0x3d/0xb0 [25373.907731] [] ? __generic_file_write_iter+0x350/0x350 [25373.908671] [] do_iter_readv_writev+0x78/0xc0 [25373.909602] [] do_readv_writev+0xd8/0x2a0 [25373.910526] [] ? __generic_file_write_iter+0x350/0x350 [25373.911458] [] ? __generic_file_write_iter+0x350/0x350 [25373.912427] [] ? __hrtimer_start_range_ns+0x252/0x380 [25373.913397] [] ? hrtimer_start+0x18/0x20 [25373.914320] [] vfs_writev+0x39/0x50 [25373.915231] [] SyS_writev+0x59/0xf0 [25373.916128] [] system_call_fastpath+0x12/0x17 [25373.917017] Code: 10 0f 1f 00 40 8a 39 0f 1f 00 85 d2 40 88 7d ff 75 2d 48 63 f6 48 8d 54 31 ff 48 31 d1 48 f7 c1 00 f0 ff ff 74 0f 0f 1f 00 8a 12 <0f> 1f 00 88 55 ff 0f b6 55 ff c9 f3 c3 0f 1f 80 00 00 00 00 c9 [25373.918994] NMI backtrace for cpu 3 [25373.919961] CPU: 3 PID: 29575 Comm: trinity-c135 Tainted: G L 3.18.0+ #108 [25373.921902] task: ffff88022d2fa0d0 ti: ffff88009aa0c000 task.ti: ffff88009aa0c000 [25373.922890] RIP: 0010:[] [] unlock_page+0x18/0x90 [25373.923900] RSP: 0018:ffff88009aa0fbf8 EFLAGS: 00000202 [25373.924888] RAX: ffffea0001505d00 RBX: ffffea0001505d00 RCX: 007c8fc09347b000 [25373.925868] RDX: 0000000000000001 RSI: ffff88022e973770 RDI: ffffea0001505d00 [25373.926844] RBP: ffff88009aa0fbf8 R08: 0000000000001000 R09: ffffea0001505d00 [25373.927819] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000001000 [25373.928790] R13: ffff88009aa0fd60 R14: 0000000000000000 R15: ffff88022e973770 [25373.929770] FS: 00007f0b34598740(0000) GS:ffff88024e2c0000(0000) knlGS:0000000000000000 [25373.930759] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [25373.931746] CR2: 00007f0b3320417d CR3: 0000000039ddb000 CR4: 00000000001407e0 [25373.932739] DR0: 00007fade9372000 DR1: 00007fe6ed782000 DR2: 0000000000000000 [25373.933800] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [25373.934848] Stack: [25373.935836] ffff88009aa0fc28 ffffffff8117f380 007c8fc09347a000 0000000000001000 [25373.936902] ffff88009aa0fd60 0000000000000000 ffff88009aa0fcc8 ffffffff8116532a [25373.938049] ffff88009aa0fd68 0000000000001000 ffff88022d2fa0d0 0000000045806f40 [25373.939153] Call Trace: [25373.940209] [] shmem_write_end+0x40/0xf0 [25373.941247] [] generic_perform_write+0x11a/0x1f0 [25373.942283] [] __generic_file_write_iter+0x15f/0x350 [25373.943317] [] ? new_sync_read+0xd0/0xd0 [25373.944323] [] generic_file_write_iter+0x3d/0xb0 [25373.945306] [] ? __generic_file_write_iter+0x350/0x350 [25373.946298] [] do_iter_readv_writev+0x78/0xc0 [25373.947279] [] do_readv_writev+0xd8/0x2a0 [25373.948237] [] ? __generic_file_write_iter+0x350/0x350 [25373.949185] [] ? __generic_file_write_iter+0x350/0x350 [25373.950126] [] ? __hrtimer_start_range_ns+0x252/0x380 [25373.951061] [] ? hrtimer_start+0x18/0x20 [25373.951985] [] vfs_writev+0x39/0x50 [25373.952958] [] SyS_writev+0x59/0xf0 [25373.953919] [] system_call_fastpath+0x12/0x17 [25373.954946] Code: 41 f6 c4 01 0f 84 7d ff ff ff eb eb 0f 1f 80 00 00 00 00 0f 1f 44 00 00 55 48 89 f8 48 8b 17 48 89 e5 83 e2 01 74 60 f0 80 27 fe <48> be 01 00 fc ff ff ff 37 9e 48 8b 17 48 0f af f7 48 89 d1 48 [25373.957052] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 118.758 msecs -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/