Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754181Ab3FQRyF (ORCPT ); Mon, 17 Jun 2013 13:54:05 -0400 Received: from mx1.redhat.com ([209.132.183.28]:26549 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753951Ab3FQReS (ORCPT ); Mon, 17 Jun 2013 13:34:18 -0400 Date: Mon, 17 Jun 2013 10:26:11 -0400 From: Dave Jones To: Linux Kernel Cc: Linus Torvalds Subject: [3.10-rc6] soft lockups from sync() Message-ID: <20130617142611.GA16732@redhat.com> Mail-Followup-To: Dave Jones , Linux Kernel , Linus Torvalds MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 72019 Lines: 833 Haven't seen this happen before.. 3.10-rc6 + a bunch of patches to survive other trinity induced bugs + some debug 'helpers' (the loadavg-at-oops for eg), but nothing that should be responsible for this. [15208.612347] BUG: soft lockup - CPU#0 stuck for 23s! [trinity-child0:19193] [15208.612350] BUG: soft lockup - CPU#1 stuck for 23s! [trinity-child1:15177] [15208.612373] Modules linked in: bridge 8021q garp stp tun fuse snd_seq_dummy rfcomm bnep nfnetlink hidp ipt_ULOG scsi_transport_iscsi can_bcm ipx p8023 p8022 x25 netrom nfc rose llc2 ax25 caif_socket irda caif pppoe pppox ppp_generic crc_ccitt slhc appletalk psnap can_raw llc can af_rxrpc atm af_key af_802154 rds phonet bluetooth rfkill coretemp hwmon kvm_intel kvm snd_hda_codec_realtek crc32c_intel ghash_clmulni_intel microcode snd_hda_codec_hdmi pcspkr snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device usb_debug snd_pcm e1000e snd_page_alloc snd_timer snd ptp pps_core soundcore xfs libcrc32c [15208.612374] irq event stamp: 1206048 [15208.612379] hardirqs last enabled at (1206047): [] restore_args+0x0/0x30 [15208.612381] hardirqs last disabled at (1206048): [] apic_timer_interrupt+0x6a/0x80 [15208.612385] softirqs last enabled at (1206046): [] __do_softirq+0x194/0x440 [15208.612387] softirqs last disabled at (1206041): [] irq_exit+0xcd/0xe0 [15208.612390] CPU: 1 PID: 15177 Comm: trinity-child1 Not tainted 3.10.0-rc6+ #15 [loadavg: 21.98 9.76 7.20 41/174 19195] [15208.612391] task: ffff88023e25ca40 ti: ffff880217668000 task.ti: ffff880217668000 [15208.612395] RIP: 0010:[] [] btree_merge+0x100/0x100 [15208.612396] RSP: 0018:ffff880217669d48 EFLAGS: 00000297 [15208.612396] RAX: ffff880217669fd8 RBX: 0000000000000000 RCX: 000000000000b910 [15208.612397] RDX: 000000000000d5d4 RSI: 0000000000000001 RDI: 0000000000000001 [15208.612397] RBP: ffff880217669d78 R08: 0000000000000001 R09: 0000000000000000 [15208.612398] R10: 0000000000000000 R11: 0000000000000001 R12: ffffffff816ed220 [15208.612398] R13: ffffffff811038de R14: ffff880217669ca8 R15: 0000000000000046 [15208.612399] FS: 00007ff7647d2740(0000) GS:ffff880244a00000(0000) knlGS:0000000000000000 [15208.612400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15208.612400] CR2: 00007fc6d3a67000 CR3: 0000000211c8f000 CR4: 00000000001407e0 [15208.612401] DR0: 0000000002ceeac0 DR1: 0000000000000000 DR2: 0000000000000000 [15208.612401] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [15208.612402] Stack: [15208.612403] ffffffff81309544 ffffffff81c04640 0000000030f05c80 0000000088c820e8 [15208.612405] ffff88023b48d540 ffff88023b48d750 ffff880217669d88 ffffffff8130947f [15208.612406] ffff880217669db0 ffffffff81311a91 ffffffff81c04640 ffffffff81c04658 [15208.612407] Call Trace: [15208.612409] [] ? delay_tsc+0x24/0xe0 [15208.612411] [] __delay+0xf/0x20 [15208.612413] [] do_raw_spin_lock+0xe1/0x130 [15208.612414] [] _raw_spin_lock+0x60/0x80 [15208.612417] [] ? sync_inodes_sb+0x228/0x2a0 [15208.612419] [] sync_inodes_sb+0x228/0x2a0 [15208.612421] [] ? wait_for_completion+0xdf/0x110 [15208.612424] [] ? get_parent_ip+0xd/0x50 [15208.612425] [] ? generic_write_sync+0x70/0x70 [15208.612426] [] sync_inodes_one_sb+0x19/0x20 [15208.612429] [] iterate_supers+0xb2/0x110 [15208.612430] [] sys_sync+0x35/0x90 [15208.612432] [] tracesys+0xdd/0xe2 [15208.612447] Code: c7 46 10 00 00 00 00 48 8d 65 d8 5b 41 5c 41 5d 41 5e 41 5f 5d c3 48 8d 65 d8 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 0b 66 90 <55> 48 89 e5 53 65 48 8b 04 25 f0 b9 00 00 8b 80 44 e0 ff ff 65 [15208.642330] BUG: soft lockup - CPU#2 stuck for 23s! [trinity-child2:18845] [15208.642354] Modules linked in: bridge 8021q garp stp tun fuse snd_seq_dummy rfcomm bnep nfnetlink hidp ipt_ULOG scsi_transport_iscsi can_bcm ipx p8023 p8022 x25 netrom nfc rose llc2 ax25 caif_socket irda caif pppoe pppox ppp_generic crc_ccitt slhc appletalk psnap can_raw llc can af_rxrpc atm af_key af_802154 rds phonet bluetooth rfkill coretemp hwmon kvm_intel kvm snd_hda_codec_realtek crc32c_intel ghash_clmulni_intel microcode snd_hda_codec_hdmi pcspkr snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device usb_debug snd_pcm e1000e snd_page_alloc snd_timer snd ptp pps_core soundcore xfs libcrc32c [15208.642356] irq event stamp: 162644 [15208.642360] hardirqs last enabled at (162643): [] restore_args+0x0/0x30 [15208.642362] hardirqs last disabled at (162644): [] apic_timer_interrupt+0x6a/0x80 [15208.642364] softirqs last enabled at (162642): [] __do_softirq+0x194/0x440 [15208.642366] softirqs last disabled at (162639): [] irq_exit+0xcd/0xe0 [15208.642369] CPU: 2 PID: 18845 Comm: trinity-child2 Not tainted 3.10.0-rc6+ #15 [loadavg: 21.98 9.76 7.20 41/174 19195] [15208.642370] task: ffff88023c5e8000 ti: ffff880234656000 task.ti: ffff880234656000 [15208.642373] RIP: 0010:[] [] delay_tsc+0xd0/0xe0 [15208.642373] RSP: 0018:ffff880234657d50 EFLAGS: 00000202 [15208.642374] RAX: 0000000000080008 RBX: ffffffff816ed220 RCX: 000000000000b910 [15208.642375] RDX: 0000000000000018 RSI: 0000000000000001 RDI: 0000000000000001 [15208.642375] RBP: ffff880234657d78 R08: 0000000000000001 R09: 0000000000000000 [15208.642376] R10: 0000000000000000 R11: 0000000000000001 R12: ffff880234657cc8 [15208.642376] R13: 0000000000000046 R14: ffff880234656000 R15: ffff88023c5e8000 [15208.642377] FS: 00007f87fde40740(0000) GS:ffff880244c00000(0000) knlGS:0000000000000000 [15208.642378] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15208.642378] CR2: 0000000000000018 CR3: 00000001bc382000 CR4: 00000000001407e0 [15208.642379] DR0: 0000000002ceeac0 DR1: 0000000000000000 DR2: 0000000000000000 [15208.642379] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [15208.642379] Stack: [15208.642381] ffffffff81c04640 0000000030f5e7e1 0000000088c820e8 ffff88023b5fc4c0 [15208.642382] ffff88023b5fc6d0 ffff880234657d88 ffffffff8130947f ffff880234657db0 [15208.642384] ffffffff81311a91 ffffffff81c04640 ffffffff81c04658 ffff88023c4be8c0 [15208.642384] Call Trace: [15208.642386] [] __delay+0xf/0x20 [15208.642389] [] do_raw_spin_lock+0xe1/0x130 [15208.642391] [] _raw_spin_lock+0x60/0x80 [15208.642393] [] ? sync_inodes_sb+0x228/0x2a0 [15208.642394] [] sync_inodes_sb+0x228/0x2a0 [15208.642396] [] ? wait_for_completion+0xdf/0x110 [15208.642398] [] ? get_parent_ip+0xd/0x50 [15208.642400] [] ? generic_write_sync+0x70/0x70 [15208.642401] [] sync_inodes_one_sb+0x19/0x20 [15208.642403] [] iterate_supers+0xb2/0x110 [15208.642404] [] sys_sync+0x35/0x90 [15208.642406] [] tracesys+0xdd/0xe2 [15208.642420] Code: 41 5c 41 5d 41 5e 41 5f 5d c3 e8 ec 1a 3e 00 eb 9f e8 35 9e 00 00 44 29 f3 41 01 dc 41 89 c5 0f 1f 00 0f ae e8 0f 31 89 c3 eb 9b cb 1a 3e 00 eb cd 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 [15208.672313] BUG: soft lockup - CPU#3 stuck for 23s! [trinity-child0:17147] [15208.672335] Modules linked in: bridge 8021q garp stp tun fuse snd_seq_dummy rfcomm bnep nfnetlink hidp ipt_ULOG scsi_transport_iscsi can_bcm ipx p8023 p8022 x25 netrom nfc rose llc2 ax25 caif_socket irda caif pppoe pppox ppp_generic crc_ccitt slhc appletalk psnap can_raw llc can af_rxrpc atm af_key af_802154 rds phonet bluetooth rfkill coretemp hwmon kvm_intel kvm snd_hda_codec_realtek crc32c_intel ghash_clmulni_intel microcode snd_hda_codec_hdmi pcspkr snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device usb_debug snd_pcm e1000e snd_page_alloc snd_timer snd ptp pps_core soundcore xfs libcrc32c [15208.672335] irq event stamp: 4183277 [15208.672338] hardirqs last enabled at (4183276): [] restore_args+0x0/0x30 [15208.672340] hardirqs last disabled at (4183277): [] apic_timer_interrupt+0x6a/0x80 [15208.672342] softirqs last enabled at (4182502): [] __do_softirq+0x194/0x440 [15208.672344] softirqs last disabled at (4182505): [] irq_exit+0xcd/0xe0 [15208.672346] CPU: 3 PID: 17147 Comm: trinity-child0 Not tainted 3.10.0-rc6+ #15 [loadavg: 21.98 9.76 7.20 41/174 19195] [15208.672348] task: ffff88023c7ba520 ti: ffff88022b4c6000 task.ti: ffff88022b4c6000 [15208.672350] RIP: 0010:[] [] __do_softirq+0xb1/0x440 [15208.672351] RSP: 0000:ffff880244e03f08 EFLAGS: 00000202 [15208.672351] RAX: ffff88023c7ba520 RBX: ffffffff816ed220 RCX: 0000000000000002 [15208.672352] RDX: 0000000000000450 RSI: ffff88023c7bacd8 RDI: ffff88023c7ba520 [15208.672352] RBP: ffff880244e03f70 R08: 0000000000000002 R09: 0000000000000000 [15208.672353] R10: 0000000000000001 R11: 0000000000000000 R12: ffff880244e03e78 [15208.672353] R13: ffffffff816f5d2f R14: ffff880244e03f70 R15: 0000000000000000 [15208.672354] FS: 00007f87fde40740(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000 [15208.672355] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15208.672355] CR2: 00000000052da178 CR3: 0000000172dca000 CR4: 00000000001407e0 [15208.672356] DR0: 0000000002ceeac0 DR1: 0000000000000000 DR2: 0000000000000000 [15208.672356] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [15208.672357] Stack: [15208.672358] 0000000a00406040 000000010016c2d5 ffff88022b4c7fd8 ffff88022b4c7fd8 [15208.672360] ffff88022b4c7fd8 ffff88023c7ba918 ffff88022b4c7fd8 ffffffff00000003 [15208.672361] ffff88023c7ba520 0000000000000000 ffff88023bc1b4e8 0000000000000000 [15208.672361] Call Trace: [15208.672363] [15208.672364] [] irq_exit+0xcd/0xe0 [15208.672366] [] smp_apic_timer_interrupt+0x6b/0x9b [15208.672368] [] apic_timer_interrupt+0x6f/0x80 [15208.672368] [15208.672370] [] ? retint_restore_args+0xe/0xe [15208.672372] [] ? sync_inodes_sb+0x19f/0x2a0 [15208.672373] [] ? sync_inodes_sb+0x198/0x2a0 [15208.672375] [] ? wait_for_completion+0xdf/0x110 [15208.672377] [] ? get_parent_ip+0xd/0x50 [15208.672378] [] ? generic_write_sync+0x70/0x70 [15208.672379] [] sync_inodes_one_sb+0x19/0x20 [15208.672381] [] iterate_supers+0xb2/0x110 [15208.672382] [] sys_sync+0x35/0x90 [15208.672383] [] tracesys+0xdd/0xe2 [15208.672398] Code: 48 89 45 b8 48 89 45 b0 48 89 45 a8 66 0f 1f 44 00 00 65 c7 04 25 80 0f 1d 00 00 00 00 00 e8 b7 31 06 00 fb 49 c7 c6 00 41 c0 81 0e 0f 1f 44 00 00 49 83 c6 08 41 d1 ef 74 6c 41 f6 c7 01 74 [15208.871052] Modules linked in: bridge 8021q garp stp tun fuse snd_seq_dummy rfcomm bnep nfnetlink hidp ipt_ULOG scsi_transport_iscsi can_bcm ipx p8023 p8022 x25 netrom nfc rose llc2 ax25 caif_socket irda caif pppoe pppox ppp_generic crc_ccitt slhc appletalk psnap can_raw llc can af_rxrpc atm af_key af_802154 rds phonet bluetooth rfkill coretemp hwmon kvm_intel kvm snd_hda_codec_realtek crc32c_intel ghash_clmulni_intel microcode snd_hda_codec_hdmi pcspkr snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device usb_debug snd_pcm e1000e snd_page_alloc snd_timer snd ptp pps_core soundcore xfs libcrc32c [15208.882510] irq event stamp: 31508 [15208.884037] hardirqs last enabled at (31507): [] restore_args+0x0/0x30 [15208.886155] hardirqs last disabled at (31508): [] apic_timer_interrupt+0x6a/0x80 [15208.888370] softirqs last enabled at (31506): [] __do_softirq+0x194/0x440 [15208.890526] softirqs last disabled at (31501): [] irq_exit+0xcd/0xe0 [15208.892623] CPU: 0 PID: 19193 Comm: trinity-child0 Not tainted 3.10.0-rc6+ #15 [loadavg: 21.98 9.76 7.20 41/174 19195] [15208.897735] task: ffff880215eb2520 ti: ffff880229d02000 task.ti: ffff880229d02000 [15208.899824] RIP: 0010:[] [] delay_tsc+0x73/0xe0 [15208.901971] RSP: 0018:ffff880229d03b08 EFLAGS: 00000202 [15208.903825] RAX: 000000004f8e861b RBX: ffffffff816ed220 RCX: 000000000000b910 [15208.905880] RDX: 0000000000001fcf RSI: 0000000000000001 RDI: 0000000000000001 [15208.907910] RBP: ffff880229d03b30 R08: 0000000000000001 R09: 0000000000000000 [15208.909930] R10: 0000000000000000 R11: 0000000000000001 R12: ffff880229d03a78 [15208.911921] R13: 0000000000000046 R14: ffff880229d02000 R15: ffff880215eb2520 [15208.913885] FS: 00007f4c8fca9740(0000) GS:ffff880244800000(0000) knlGS:0000000000000000 [15208.915945] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15208.917711] CR2: 0000003850ae6500 CR3: 000000017b1a9000 CR4: 00000000001407f0 [15208.919628] DR0: 0000000002ceeac0 DR1: 0000000000000000 DR2: 0000000000000000 [15208.921529] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [15208.923431] Stack: [15208.924728] ffffffff81c04640 0000000030df1abe 0000000088c820e8 0000000000004af9 [15208.926720] 0000000000000000 ffff880229d03b40 ffffffff8130947f ffff880229d03b68 [15208.928705] ffffffff81311a91 ffffffff81c04640 ffffffff81c04658 0000000000000000 [15208.930670] Call Trace: [15208.931993] [] __delay+0xf/0x20 [15208.933583] [] do_raw_spin_lock+0xe1/0x130 [15208.935284] [] _raw_spin_lock+0x60/0x80 [15208.936939] [] ? inode_sb_list_add+0x19/0x50 [15208.938649] [] inode_sb_list_add+0x19/0x50 [15208.940326] [] new_inode+0x29/0x30 [15208.941924] [] proc_pid_make_inode+0x1f/0x250 [15208.943636] [] proc_pid_instantiate+0x1b/0xd0 [15208.945329] [] proc_pid_lookup+0x13c/0x200 [15208.946991] [] ? proc_pid_lookup+0x8e/0x200 [15208.948657] [] proc_root_lookup+0x2f/0x40 [15208.950301] [] lookup_real+0x1d/0x50 [15208.951898] [] __lookup_hash+0x33/0x40 [15208.953510] [] lookup_slow+0x44/0xa9 [15208.955091] [] link_path_walk+0x733/0x900 [15208.956722] [] path_openat+0x94/0x530 [15208.958310] [] ? native_sched_clock+0x24/0x80 [15208.959977] [] ? sched_clock_cpu+0xb5/0x100 [15208.961628] [] ? sched_clock_cpu+0xb5/0x100 [15208.963271] [] do_filp_open+0x38/0x80 [15208.964852] [] ? _raw_spin_unlock+0x31/0x60 [15208.966493] [] ? __alloc_fd+0xaf/0x200 [15208.968079] [] do_sys_open+0xe9/0x1c0 [15208.969655] [] SyS_open+0x1e/0x20 [15208.971185] [] tracesys+0xdd/0xe2 [15208.972716] Code: 3e 00 49 8b 87 38 e0 ff ff a8 08 75 5a f3 90 bf 01 00 00 00 e8 ef 76 3e 00 e8 8a 9e 00 00 41 39 c5 75 4b 0f 1f 00 0f ae e8 0f 31 <48> c1 e2 20 89 c0 48 09 c2 41 89 d6 29 da 44 39 e2 72 ba bf 01 [15236.596709] BUG: soft lockup - CPU#0 stuck for 23s! [trinity-child0:19193] [15236.596712] BUG: soft lockup - CPU#1 stuck for 23s! [trinity-child1:15177] [15236.596732] Modules linked in: bridge 8021q garp stp tun fuse snd_seq_dummy rfcomm bnep nfnetlink hidp ipt_ULOG scsi_transport_iscsi can_bcm ipx p8023 p8022 x25 netrom nfc rose llc2 ax25 caif_socket irda caif pppoe pppox ppp_generic crc_ccitt slhc appletalk psnap can_raw llc can af_rxrpc atm af_key af_802154 rds phonet bluetooth rfkill coretemp hwmon kvm_intel kvm snd_hda_codec_realtek crc32c_intel ghash_clmulni_intel microcode snd_hda_codec_hdmi pcspkr snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device usb_debug snd_pcm e1000e snd_page_alloc snd_timer snd ptp pps_core soundcore xfs libcrc32c [15236.596732] irq event stamp: 1228866 [15236.596735] hardirqs last enabled at (1228865): [] restore_args+0x0/0x30 [15236.596736] hardirqs last disabled at (1228866): [] apic_timer_interrupt+0x6a/0x80 [15236.596738] softirqs last enabled at (1228864): [] __do_softirq+0x194/0x440 [15236.596740] softirqs last disabled at (1228859): [] irq_exit+0xcd/0xe0 [15236.596742] CPU: 1 PID: 15177 Comm: trinity-child1 Not tainted 3.10.0-rc6+ #15 [loadavg: 31.01 12.86 8.30 45/174 19195] [15236.596743] task: ffff88023e25ca40 ti: ffff880217668000 task.ti: ffff880217668000 [15236.596745] RIP: 0010:[] [] delay_tsc+0x2f/0xe0 [15236.596746] RSP: 0018:ffff880217669d50 EFLAGS: 00000202 [15236.596746] RAX: 00000000453865bb RBX: 0000000000000001 RCX: 000000000000b910 [15236.596747] RDX: 0000000000001fde RSI: 0000000000000001 RDI: 0000000000000001 [15236.596747] RBP: ffff880217669d78 R08: 0000000000000001 R09: 0000000000000000 [15236.596748] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000 [15236.596748] R13: 0000000000000001 R14: ffffffff816ed220 R15: ffffffff811038de [15236.596749] FS: 00007ff7647d2740(0000) GS:ffff880244a00000(0000) knlGS:0000000000000000 [15236.596750] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15236.596750] CR2: 00007fc6d3a67000 CR3: 0000000211c8f000 CR4: 00000000001407e0 [15236.596751] DR0: 0000000002ceeac0 DR1: 0000000000000000 DR2: 0000000000000000 [15236.596751] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [15236.596751] Stack: [15236.596753] ffffffff81c04640 0000000062eeaf74 0000000088c820e8 ffff88023b48d540 [15236.596754] ffff88023b48d750 ffff880217669d88 ffffffff8130947f ffff880217669db0 [15236.596756] ffffffff81311a91 ffffffff81c04640 ffffffff81c04658 ffff88023c4be8c0 [15236.596756] Call Trace: [15236.596758] [] __delay+0xf/0x20 [15236.596759] [] do_raw_spin_lock+0xe1/0x130 [15236.596761] [] _raw_spin_lock+0x60/0x80 [15236.596763] [] ? sync_inodes_sb+0x228/0x2a0 [15236.596764] [] sync_inodes_sb+0x228/0x2a0 [15236.596766] [] ? wait_for_completion+0xdf/0x110 [15236.596768] [] ? get_parent_ip+0xd/0x50 [15236.596769] [] ? generic_write_sync+0x70/0x70 [15236.596770] [] sync_inodes_one_sb+0x19/0x20 [15236.596772] [] iterate_supers+0xb2/0x110 [15236.596773] [] sys_sync+0x35/0x90 [15236.596774] [] tracesys+0xdd/0xe2 [15236.596789] Code: 00 55 48 89 e5 41 57 41 56 41 55 41 54 41 89 fc bf 01 00 00 00 53 e8 31 77 3e 00 e8 cc 9e 00 00 41 89 c5 0f 1f 00 0f ae e8 0f 31 <65> 4c 8b 3c 25 f0 b9 00 00 89 c3 eb 2f 0f 1f 40 00 bf 01 00 00 [15236.626693] BUG: soft lockup - CPU#2 stuck for 23s! [trinity-child2:18845] [15236.626714] Modules linked in: bridge 8021q garp stp tun fuse snd_seq_dummy rfcomm bnep nfnetlink hidp ipt_ULOG scsi_transport_iscsi can_bcm ipx p8023 p8022 x25 netrom nfc rose llc2 ax25 caif_socket irda caif pppoe pppox ppp_generic crc_ccitt slhc appletalk psnap can_raw llc can af_rxrpc atm af_key af_802154 rds phonet bluetooth rfkill coretemp hwmon kvm_intel kvm snd_hda_codec_realtek crc32c_intel ghash_clmulni_intel microcode snd_hda_codec_hdmi pcspkr snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device usb_debug snd_pcm e1000e snd_page_alloc snd_timer snd ptp pps_core soundcore xfs libcrc32c [15236.626714] irq event stamp: 185294 [15236.626718] hardirqs last enabled at (185293): [] restore_args+0x0/0x30 [15236.626720] hardirqs last disabled at (185294): [] apic_timer_interrupt+0x6a/0x80 [15236.626722] softirqs last enabled at (185292): [] __do_softirq+0x194/0x440 [15236.626723] softirqs last disabled at (185289): [] irq_exit+0xcd/0xe0 [15236.626726] CPU: 2 PID: 18845 Comm: trinity-child2 Not tainted 3.10.0-rc6+ #15 [loadavg: 31.01 12.86 8.30 45/174 19195] [15236.626728] task: ffff88023c5e8000 ti: ffff880234656000 task.ti: ffff880234656000 [15236.626730] RIP: 0010:[] [] sub_preempt_count+0x51/0x100 [15236.626730] RSP: 0018:ffff880234657d38 EFLAGS: 00000202 [15236.626731] RAX: ffff880234657fd8 RBX: ffffffff8130a8ee RCX: 000000000000b910 [15236.626731] RDX: 0000000000000015 RSI: 0000000000000001 RDI: 0000000000000001 [15236.626732] RBP: ffff880234657d40 R08: 0000000000000001 R09: 0000000000000000 [15236.626732] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000062ed582a [15236.626733] R13: ffffffff81c04640 R14: ffffffff810b72dd R15: ffff880234657ca8 [15236.626733] FS: 00007f87fde40740(0000) GS:ffff880244c00000(0000) knlGS:0000000000000000 [15236.626734] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15236.626734] CR2: 0000000000000018 CR3: 00000001bc382000 CR4: 00000000001407e0 [15236.626735] DR0: 0000000002ceeac0 DR1: 0000000000000000 DR2: 0000000000000000 [15236.626735] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [15236.626736] Stack: [15236.626737] 000000004952d18a ffff880234657d78 ffffffff813095b0 ffffffff81c04640 [15236.626738] 0000000062f203bd 0000000088c820e8 ffff88023b5fc4c0 ffff88023b5fc6d0 [15236.626740] ffff880234657d88 ffffffff8130947f ffff880234657db0 ffffffff81311a91 [15236.626740] Call Trace: [15236.626742] [] delay_tsc+0x90/0xe0 [15236.626743] [] __delay+0xf/0x20 [15236.626745] [] do_raw_spin_lock+0xe1/0x130 [15236.626747] [] _raw_spin_lock+0x60/0x80 [15236.626749] [] ? sync_inodes_sb+0x228/0x2a0 [15236.626750] [] sync_inodes_sb+0x228/0x2a0 [15236.626753] [] ? wait_for_completion+0xdf/0x110 [15236.626755] [] ? get_parent_ip+0xd/0x50 [15236.626756] [] ? generic_write_sync+0x70/0x70 [15236.626757] [] sync_inodes_one_sb+0x19/0x20 [15236.626759] [] iterate_supers+0xb2/0x110 [15236.626760] [] sys_sync+0x35/0x90 [15236.626761] [] tracesys+0xdd/0xe2 [15236.626776] Code: ff 7c 4a 81 ff fe 00 00 00 76 7d 65 48 8b 04 25 f0 b9 00 00 3b 98 44 e0 ff ff 74 16 65 48 8b 04 25 f0 b9 00 00 29 98 44 e0 ff ff <5b> 5d c3 0f 1f 40 00 48 8b 45 00 48 8b 78 08 e8 5b c0 99 ff 48 [15236.656665] BUG: soft lockup - CPU#3 stuck for 23s! [trinity-child0:17147] [15236.656685] Modules linked in: bridge 8021q garp stp tun fuse snd_seq_dummy rfcomm bnep nfnetlink hidp ipt_ULOG scsi_transport_iscsi can_bcm ipx p8023 p8022 x25 netrom nfc rose llc2 ax25 caif_socket irda caif pppoe pppox ppp_generic crc_ccitt slhc appletalk psnap can_raw llc can af_rxrpc atm af_key af_802154 rds phonet bluetooth rfkill coretemp hwmon kvm_intel kvm snd_hda_codec_realtek crc32c_intel ghash_clmulni_intel microcode snd_hda_codec_hdmi pcspkr snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device usb_debug snd_pcm e1000e snd_page_alloc snd_timer snd ptp pps_core soundcore xfs libcrc32c [15236.656685] irq event stamp: 6367699 [15236.656687] hardirqs last enabled at (6367698): [] restore_args+0x0/0x30 [15236.656689] hardirqs last disabled at (6367699): [] apic_timer_interrupt+0x6a/0x80 [15236.656691] softirqs last enabled at (6366924): [] __do_softirq+0x194/0x440 [15236.656692] softirqs last disabled at (6366927): [] irq_exit+0xcd/0xe0 [15236.656695] CPU: 3 PID: 17147 Comm: trinity-child0 Not tainted 3.10.0-rc6+ #15 [loadavg: 31.01 12.86 8.30 45/174 19195] [15236.656696] task: ffff88023c7ba520 ti: ffff88022b4c6000 task.ti: ffff88022b4c6000 [15236.656698] RIP: 0010:[] [] __do_softirq+0xb1/0x440 [15236.656699] RSP: 0000:ffff880244e03f08 EFLAGS: 00000212 [15236.656699] RAX: ffff88023c7ba520 RBX: ffffffff816ed220 RCX: 0000000000000002 [15236.656700] RDX: 00000000000031c0 RSI: ffff88023c7bad10 RDI: ffff88023c7ba520 [15236.656700] RBP: ffff880244e03f70 R08: 0000000000000003 R09: 0000000000000000 [15236.656700] R10: 0000000000000001 R11: 0000000000000000 R12: ffff880244e03e78 [15236.656701] R13: ffffffff816f5d2f R14: ffff880244e03f70 R15: 0000000000000000 [15236.656702] FS: 00007f87fde40740(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000 [15236.656702] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15236.656703] CR2: 00000000052da178 CR3: 0000000172dca000 CR4: 00000000001407e0 [15236.656703] DR0: 0000000002ceeac0 DR1: 0000000000000000 DR2: 0000000000000000 [15236.656704] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [15236.656704] Stack: [15236.656705] 0000000a00406040 000000010016cdc5 ffff88022b4c7fd8 ffff88022b4c7fd8 [15236.656707] ffff88022b4c7fd8 ffff88023c7ba918 ffff88022b4c7fd8 ffffffff00000003 [15236.656708] ffff88023c7ba520 0000000000000000 0000000000000000 0000000000000002 [15236.656708] Call Trace: [15236.656709] [15236.656711] [] irq_exit+0xcd/0xe0 [15236.656713] [] smp_apic_timer_interrupt+0x6b/0x9b [15236.656714] [] apic_timer_interrupt+0x6f/0x80 [15236.656715] [15236.656716] [] ? retint_restore_args+0xe/0xe [15236.656718] [] ? lock_acquire+0xa6/0x1f0 [15236.656720] [] ? sync_inodes_sb+0x1c2/0x2a0 [15236.656722] [] _raw_spin_lock+0x40/0x80 [15236.656723] [] ? sync_inodes_sb+0x1c2/0x2a0 [15236.656725] [] sync_inodes_sb+0x1c2/0x2a0 [15236.656726] [] ? wait_for_completion+0xdf/0x110 [15236.656728] [] ? get_parent_ip+0xd/0x50 [15236.656729] [] ? generic_write_sync+0x70/0x70 [15236.656730] [] sync_inodes_one_sb+0x19/0x20 [15236.656732] [] iterate_supers+0xb2/0x110 [15236.656733] [] sys_sync+0x35/0x90 [15236.656734] [] tracesys+0xdd/0xe2 [15236.656749] Code: 48 89 45 b8 48 89 45 b0 48 89 45 a8 66 0f 1f 44 00 00 65 c7 04 25 80 0f 1d 00 00 00 00 00 e8 b7 31 06 00 fb 49 c7 c6 00 41 c0 81 0e 0f 1f 44 00 00 49 83 c6 08 41 d1 ef 74 6c 41 f6 c7 01 74 [15236.857644] Modules linked in: bridge 8021q garp stp tun fuse snd_seq_dummy rfcomm bnep nfnetlink hidp ipt_ULOG scsi_transport_iscsi can_bcm ipx p8023 p8022 x25 netrom nfc rose llc2 ax25 caif_socket irda caif pppoe pppox ppp_generic crc_ccitt slhc appletalk psnap can_raw llc can af_rxrpc atm af_key af_802154 rds phonet bluetooth rfkill coretemp hwmon kvm_intel kvm snd_hda_codec_realtek crc32c_intel ghash_clmulni_intel microcode snd_hda_codec_hdmi pcspkr snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device usb_debug snd_pcm e1000e snd_page_alloc snd_timer snd ptp pps_core soundcore xfs libcrc32c [15236.868730] irq event stamp: 55072 [15236.870164] hardirqs last enabled at (55071): [] restore_args+0x0/0x30 [15236.872193] hardirqs last disabled at (55072): [] apic_timer_interrupt+0x6a/0x80 [15236.874315] softirqs last enabled at (55070): [] __do_softirq+0x194/0x440 [15236.876379] softirqs last disabled at (55065): [] irq_exit+0xcd/0xe0 [15236.878406] CPU: 0 PID: 19193 Comm: trinity-child0 Not tainted 3.10.0-rc6+ #15 [loadavg: 31.01 12.86 8.30 11/182 19203] [15236.883338] task: ffff880215eb2520 ti: ffff880229d02000 task.ti: ffff880229d02000 [15236.885317] RIP: 0010:[] [] delay_tsc+0x2f/0xe0 [15236.887342] RSP: 0018:ffff880229d03b08 EFLAGS: 00000202 [15236.889040] RAX: 00000000453865be RBX: ffffffff816ed220 RCX: 000000000000b910 [15236.890938] RDX: 0000000000001fde RSI: 0000000000000001 RDI: 0000000000000001 [15236.892815] RBP: ffff880229d03b30 R08: 0000000000000001 R09: 0000000000000000 [15236.894663] R10: 0000000000000000 R11: 0000000000000001 R12: ffff880229d03a78 [15236.896496] R13: 0000000000000046 R14: ffff880229d02000 R15: ffff880215eb2520 [15236.898309] FS: 00007f4c8fca9740(0000) GS:ffff880244800000(0000) knlGS:0000000000000000 [15236.900232] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15236.901881] CR2: 0000003850ae6500 CR3: 000000017b1a9000 CR4: 00000000001407f0 [15236.903696] DR0: 0000000002ceeac0 DR1: 0000000000000000 DR2: 0000000000000000 [15236.905510] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [15236.907358] Stack: [15236.908545] ffffffff81c04640 000000006222166a 0000000088c820e8 0000000000004af9 [15236.910401] 0000000000000000 ffff880229d03b40 ffffffff8130947f ffff880229d03b68 [15236.912257] ffffffff81311a91 ffffffff81c04640 ffffffff81c04658 0000000000000000 [15236.914110] Call Trace: [15236.915338] [] __delay+0xf/0x20 [15236.916863] [] do_raw_spin_lock+0xe1/0x130 [15236.918471] [] _raw_spin_lock+0x60/0x80 [15236.920043] [] ? inode_sb_list_add+0x19/0x50 [15236.921668] [] inode_sb_list_add+0x19/0x50 [15236.923262] [] new_inode+0x29/0x30 [15236.924770] [] proc_pid_make_inode+0x1f/0x250 [15236.926386] [] proc_pid_instantiate+0x1b/0xd0 [15236.928015] [] proc_pid_lookup+0x13c/0x200 [15236.929599] [] ? proc_pid_lookup+0x8e/0x200 [15236.931192] [] proc_root_lookup+0x2f/0x40 [15236.932755] [] lookup_real+0x1d/0x50 [15236.934265] [] __lookup_hash+0x33/0x40 [15236.935795] [] lookup_slow+0x44/0xa9 [15236.937351] [] link_path_walk+0x733/0x900 [15236.938907] [] path_openat+0x94/0x530 [15236.940422] [] ? native_sched_clock+0x24/0x80 [15236.942021] [] ? sched_clock_cpu+0xb5/0x100 [15236.943597] [] ? sched_clock_cpu+0xb5/0x100 [15236.945164] [] do_filp_open+0x38/0x80 [15236.946694] [] ? _raw_spin_unlock+0x31/0x60 [15236.948260] [] ? __alloc_fd+0xaf/0x200 [15236.949777] [] do_sys_open+0xe9/0x1c0 [15236.951291] [] SyS_open+0x1e/0x20 [15236.952757] [] tracesys+0xdd/0xe2 [15236.954222] Code: 00 55 48 89 e5 41 57 41 56 41 55 41 54 41 89 fc bf 01 00 00 00 53 e8 31 77 3e 00 e8 cc 9e 00 00 41 89 c5 0f 1f 00 0f ae e8 0f 31 <65> 4c 8b 3c 25 f0 b9 00 00 89 c3 eb 2f 0f 1f 40 00 bf 01 00 00 [15260.643263] BUG: soft lockup - CPU#3 stuck for 22s! [trinity-child0:17147] [15260.645155] Modules linked in: bridge 8021q garp stp tun fuse snd_seq_dummy rfcomm bnep nfnetlink hidp ipt_ULOG scsi_transport_iscsi can_bcm ipx p8023 p8022 x25 netrom nfc rose llc2 ax25 caif_socket irda caif pppoe pppox ppp_generic crc_ccitt slhc appletalk psnap can_raw llc can af_rxrpc atm af_key af_802154 rds phonet bluetooth rfkill coretemp hwmon kvm_intel kvm snd_hda_codec_realtek crc32c_intel ghash_clmulni_intel microcode snd_hda_codec_hdmi pcspkr snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device usb_debug snd_pcm e1000e snd_page_alloc snd_timer snd ptp pps_core soundcore xfs libcrc32c [15260.658187] irq event stamp: 8234507 [15260.659770] hardirqs last enabled at (8234506): [] restore_args+0x0/0x30 [15260.661955] hardirqs last disabled at (8234507): [] apic_timer_interrupt+0x6a/0x80 [15260.664223] softirqs last enabled at (8233736): [] __do_softirq+0x194/0x440 [15260.666386] softirqs last disabled at (8233739): [] irq_exit+0xcd/0xe0 [15260.668466] CPU: 3 PID: 17147 Comm: trinity-child0 Not tainted 3.10.0-rc6+ #15 [loadavg: 37.37 15.78 9.38 40/184 19207] [15260.673429] task: ffff88023c7ba520 ti: ffff88022b4c6000 task.ti: ffff88022b4c6000 [15260.675400] RIP: 0010:[] [] __do_softirq+0xb1/0x440 [15260.677506] RSP: 0000:ffff880244e03f08 EFLAGS: 00000206 [15260.679237] RAX: ffff88023c7ba520 RBX: ffffffff816ed220 RCX: 0000000000000002 [15260.681190] RDX: 0000000000000450 RSI: ffff88023c7bacd8 RDI: ffff88023c7ba520 [15260.683131] RBP: ffff880244e03f70 R08: 0000000000000003 R09: 0000000000000000 [15260.685049] R10: 0000000000000001 R11: 0000000000000000 R12: ffff880244e03e78 [15260.686941] R13: ffffffff816f5d2f R14: ffff880244e03f70 R15: 0000000000000000 [15260.688835] FS: 00007f87fde40740(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000 [15260.690851] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15260.692578] CR2: 0000000009647168 CR3: 0000000172dca000 CR4: 00000000001407e0 [15260.694478] DR0: 0000000002ceeac0 DR1: 0000000000000000 DR2: 0000000000000000 [15260.696376] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [15260.698272] Stack: [15260.699565] 0000000a00406040 000000010016d72b ffff88022b4c7fd8 ffff88022b4c7fd8 [15260.701657] ffff88022b4c7fd8 ffff88023c7ba918 ffff88022b4c7fd8 ffffffff00000003 [15260.703759] ffff88023c7ba520 0000000000000000 ffffffff811da678 0000000000000246 [15260.705863] Call Trace: [15260.707213] [15260.708776] [] ? sync_inodes_sb+0x198/0x2a0 [15260.710389] [] irq_exit+0xcd/0xe0 [15260.712034] [] smp_apic_timer_interrupt+0x6b/0x9b [15260.713845] [] apic_timer_interrupt+0x6f/0x80 [15260.715611] [15260.717116] [] ? retint_restore_args+0xe/0xe [15260.718717] [] ? lock_release+0xb0/0x320 [15260.720428] [] _raw_spin_unlock+0x1f/0x60 [15260.722152] [] sync_inodes_sb+0x198/0x2a0 [15260.723880] [] ? wait_for_completion+0xdf/0x110 [15260.725659] [] ? get_parent_ip+0xd/0x50 [15260.727361] [] ? generic_write_sync+0x70/0x70 [15260.729125] [] sync_inodes_one_sb+0x19/0x20 [15260.730876] [] iterate_supers+0xb2/0x110 [15260.732588] [] sys_sync+0x35/0x90 [15260.734228] [] tracesys+0xdd/0xe2 [15260.735870] Code: 48 89 45 b8 48 89 45 b0 48 89 45 a8 66 0f 1f 44 00 00 65 c7 04 25 80 0f 1d 00 00 00 00 00 e8 b7 31 06 00 fb 49 c7 c6 00 41 c0 81 0e 0f 1f 44 00 00 49 83 c6 08 41 d1 ef 74 6c 41 f6 c7 01 74 [15288.567669] BUG: soft lockup - CPU#1 stuck for 22s! [trinity-child2:19176] [15288.567672] BUG: soft lockup - CPU#0 stuck for 22s! [trinity-child0:18971] [15288.567694] Modules linked in: bridge 8021q garp stp tun fuse snd_seq_dummy rfcomm bnep nfnetlink hidp ipt_ULOG scsi_transport_iscsi can_bcm ipx p8023 p8022 x25 netrom nfc rose llc2 ax25 caif_socket irda caif pppoe pppox ppp_generic crc_ccitt slhc appletalk psnap can_raw llc can af_rxrpc atm af_key af_802154 rds phonet bluetooth rfkill coretemp hwmon kvm_intel kvm snd_hda_codec_realtek crc32c_intel ghash_clmulni_intel microcode snd_hda_codec_hdmi pcspkr snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device usb_debug snd_pcm e1000e snd_page_alloc snd_timer snd ptp pps_core soundcore xfs libcrc32c [15288.567694] irq event stamp: 142242 [15288.567700] hardirqs last enabled at (142241): [] restore_args+0x0/0x30 [15288.567703] hardirqs last disabled at (142242): [] apic_timer_interrupt+0x6a/0x80 [15288.567706] softirqs last enabled at (142240): [] __do_softirq+0x194/0x440 [15288.567708] softirqs last disabled at (142235): [] irq_exit+0xcd/0xe0 [15288.567711] CPU: 0 PID: 18971 Comm: trinity-child0 Not tainted 3.10.0-rc6+ #15 [loadavg: 41.18 18.72 10.56 37/176 19247] [15288.567713] task: ffff880227f5a520 ti: ffff880172e18000 task.ti: ffff880172e18000 [15288.567716] RIP: 0010:[] [] delay_tsc+0x2f/0xe0 [15288.567717] RSP: 0018:ffff880172e19d50 EFLAGS: 00000202 [15288.567718] RAX: 000000000d73e038 RBX: ffffffff816ed220 RCX: 000000000000b910 [15288.567718] RDX: 0000000000001ffa RSI: 0000000000000001 RDI: 0000000000000001 [15288.567719] RBP: ffff880172e19d78 R08: 0000000000000001 R09: 0000000000000000 [15288.567719] R10: 0000000000000000 R11: 0000000000000001 R12: ffff880172e19cc8 [15288.567720] R13: 0000000000000046 R14: ffff880172e18000 R15: ffff880227f5a520 [15288.567721] FS: 00007f1557d68740(0000) GS:ffff880244800000(0000) knlGS:0000000000000000 [15288.567721] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15288.567722] CR2: 00007f8813f2a8e8 CR3: 000000023c414000 CR4: 00000000001407f0 [15288.567722] DR0: 0000000002ceeac0 DR1: 0000000000000000 DR2: 0000000000000000 [15288.567723] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [15288.567723] Stack: [15288.567725] ffffffff81c04640 000000002b4442bd 0000000088c820e8 0000000000000000 [15288.567726] ffffffff00000000 ffff880172e19d88 ffffffff8130947f ffff880172e19db0 [15288.567728] ffffffff81311a91 ffffffff81c04640 ffffffff81c04658 ffff88023c4be8c0 [15288.567728] Call Trace: [15288.567730] [] __delay+0xf/0x20 [15288.567733] [] do_raw_spin_lock+0xe1/0x130 [15288.567734] [] _raw_spin_lock+0x60/0x80 [15288.567737] [] ? sync_inodes_sb+0x174/0x2a0 [15288.567739] [] sync_inodes_sb+0x174/0x2a0 [15288.567741] [] ? wait_for_completion+0xdf/0x110 [15288.567744] [] ? get_parent_ip+0xd/0x50 [15288.567745] [] ? generic_write_sync+0x70/0x70 [15288.567746] [] sync_inodes_one_sb+0x19/0x20 [15288.567749] [] iterate_supers+0xb2/0x110 [15288.567750] [] sys_sync+0x35/0x90 [15288.567752] [] tracesys+0xdd/0xe2 [15288.567766] Code: 00 55 48 89 e5 41 57 41 56 41 55 41 54 41 89 fc bf 01 00 00 00 53 e8 31 77 3e 00 e8 cc 9e 00 00 41 89 c5 0f 1f 00 0f ae e8 0f 31 <65> 4c 8b 3c 25 f0 b9 00 00 89 c3 eb 2f 0f 1f 40 00 bf 01 00 00 [15288.597656] BUG: soft lockup - CPU#2 stuck for 22s! [trinity-child3:14411] [15288.597681] Modules linked in: bridge 8021q garp stp tun fuse snd_seq_dummy rfcomm bnep nfnetlink hidp ipt_ULOG scsi_transport_iscsi can_bcm ipx p8023 p8022 x25 netrom nfc rose llc2 ax25 caif_socket irda caif pppoe pppox ppp_generic crc_ccitt slhc appletalk psnap can_raw llc can af_rxrpc atm af_key af_802154 rds phonet bluetooth rfkill coretemp hwmon kvm_intel kvm snd_hda_codec_realtek crc32c_intel ghash_clmulni_intel microcode snd_hda_codec_hdmi pcspkr snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device usb_debug snd_pcm e1000e snd_page_alloc snd_timer snd ptp pps_core soundcore xfs libcrc32c [15288.597681] irq event stamp: 1864048 [15288.597685] hardirqs last enabled at (1864047): [] restore_args+0x0/0x30 [15288.597687] hardirqs last disabled at (1864048): [] apic_timer_interrupt+0x6a/0x80 [15288.597690] softirqs last enabled at (1864046): [] __do_softirq+0x194/0x440 [15288.597692] softirqs last disabled at (1864041): [] irq_exit+0xcd/0xe0 [15288.597695] CPU: 2 PID: 14411 Comm: trinity-child3 Not tainted 3.10.0-rc6+ #15 [loadavg: 41.18 18.72 10.56 37/176 19247] [15288.597696] task: ffff8802270e0000 ti: ffff880176ef2000 task.ti: ffff880176ef2000 [15288.597699] RIP: 0010:[] [] delay_tsc+0x73/0xe0 [15288.597700] RSP: 0018:ffff880176ef3d18 EFLAGS: 00000202 [15288.597701] RAX: 00000000118e4c49 RBX: ffffffff816ed220 RCX: 000000000000b910 [15288.597701] RDX: 0000000000001ffa RSI: 0000000000000001 RDI: 0000000000000001 [15288.597702] RBP: ffff880176ef3d40 R08: 0000000000000001 R09: 0000000000000000 [15288.597702] R10: 0000000000000000 R11: 0000000000000001 R12: ffff880176ef3c88 [15288.597703] R13: 0000000000000046 R14: ffff880176ef2000 R15: ffff8802270e0000 [15288.597704] FS: 00007f87fde40740(0000) GS:ffff880244c00000(0000) knlGS:0000000000000000 [15288.597704] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15288.597705] CR2: 0000000000000001 CR3: 0000000240d0a000 CR4: 00000000001407e0 [15288.597705] DR0: 0000000002ceeac0 DR1: 0000000000000000 DR2: 0000000000000000 [15288.597706] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [15288.597706] Stack: [15288.597708] ffffffff81c04640 000000002c0280c5 0000000088c820e8 ffffffff81814880 [15288.597709] ffff8802434ac520 ffff880176ef3d50 ffffffff8130947f ffff880176ef3d78 [15288.597710] ffffffff81311a91 ffffffff81c04640 ffffffff81c04658 ffffffff81814880 [15288.597711] Call Trace: [15288.597713] [] __delay+0xf/0x20 [15288.597715] [] do_raw_spin_lock+0xe1/0x130 [15288.597717] [] _raw_spin_lock+0x60/0x80 [15288.597719] [] ? evict+0x6b/0x1a0 [15288.597720] [] evict+0x6b/0x1a0 [15288.597721] [] iput+0xf5/0x190 [15288.597724] [] dput+0x208/0x2f0 [15288.597726] [] __fput+0x198/0x2d0 [15288.597727] [] ____fput+0xe/0x10 [15288.597729] [] task_work_run+0xac/0xe0 [15288.597731] [] do_exit+0x2bb/0xcd0 [15288.597733] [] ? trace_hardirqs_on_caller+0xfd/0x1c0 [15288.597735] [] ? trace_hardirqs_on+0xd/0x10 [15288.597736] [] do_group_exit+0x4c/0xc0 [15288.597738] [] SyS_exit_group+0x14/0x20 [15288.597739] [] tracesys+0xdd/0xe2 [15288.597755] Code: 3e 00 49 8b 87 38 e0 ff ff a8 08 75 5a f3 90 bf 01 00 00 00 e8 ef 76 3e 00 e8 8a 9e 00 00 41 39 c5 75 4b 0f 1f 00 0f ae e8 0f 31 <48> c1 e2 20 89 c0 48 09 c2 41 89 d6 29 da 44 39 e2 72 ba bf 01 [15288.627643] BUG: soft lockup - CPU#3 stuck for 22s! [trinity-child0:17147] [15288.627665] Modules linked in: bridge 8021q garp stp tun fuse snd_seq_dummy rfcomm bnep nfnetlink hidp ipt_ULOG scsi_transport_iscsi can_bcm ipx p8023 p8022 x25 netrom nfc rose llc2 ax25 caif_socket irda caif pppoe pppox ppp_generic crc_ccitt slhc appletalk psnap can_raw llc can af_rxrpc atm af_key af_802154 rds phonet bluetooth rfkill coretemp hwmon kvm_intel kvm snd_hda_codec_realtek crc32c_intel ghash_clmulni_intel microcode snd_hda_codec_hdmi pcspkr snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device usb_debug snd_pcm e1000e snd_page_alloc snd_timer snd ptp pps_core soundcore xfs libcrc32c [15288.627666] irq event stamp: 10406231 [15288.627669] hardirqs last enabled at (10406230): [] restore_args+0x0/0x30 [15288.627671] hardirqs last disabled at (10406231): [] apic_timer_interrupt+0x6a/0x80 [15288.627673] softirqs last enabled at (10405456): [] __do_softirq+0x194/0x440 [15288.627675] softirqs last disabled at (10405459): [] irq_exit+0xcd/0xe0 [15288.627677] CPU: 3 PID: 17147 Comm: trinity-child0 Not tainted 3.10.0-rc6+ #15 [loadavg: 41.18 18.72 10.56 37/176 19247] [15288.627678] task: ffff88023c7ba520 ti: ffff88022b4c6000 task.ti: ffff88022b4c6000 [15288.627681] RIP: 0010:[] [] __do_softirq+0xb1/0x440 [15288.627681] RSP: 0018:ffff880244e03f08 EFLAGS: 00000202 [15288.627682] RAX: ffff88023c7ba520 RBX: ffffffff816ed220 RCX: 0000000000000002 [15288.627682] RDX: 00000000000031c0 RSI: ffff88023c7bad10 RDI: ffff88023c7ba520 [15288.627683] RBP: ffff880244e03f70 R08: 0000000000000003 R09: 0000000000000000 [15288.627683] R10: 0000000000000001 R11: 0000000000000000 R12: ffff880244e03e78 [15288.627684] R13: ffffffff816f5d2f R14: ffff880244e03f70 R15: 0000000000000000 [15288.627685] FS: 00007f87fde40740(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000 [15288.627685] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15288.627686] CR2: 0000000000000008 CR3: 0000000172dca000 CR4: 00000000001407e0 [15288.627687] DR0: 0000000002ceeac0 DR1: 0000000000000000 DR2: 0000000000000000 [15288.627687] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [15288.627687] Stack: [15288.627689] 0000000a00406040 000000010016e21b ffff88022b4c7fd8 ffff88022b4c7fd8 [15288.627690] ffff88022b4c7fd8 ffff88023c7ba918 ffff88022b4c7fd8 ffffffff00000003 [15288.627692] ffff88023c7ba520 0000000000000000 0000000000000000 0000000000000002 [15288.627692] Call Trace: [15288.627693] [15288.627695] [] irq_exit+0xcd/0xe0 [15288.627697] [] smp_apic_timer_interrupt+0x6b/0x9b [15288.627698] [] apic_timer_interrupt+0x6f/0x80 [15288.627699] [15288.627701] [] ? retint_restore_args+0xe/0xe [15288.627703] [] ? lock_acquire+0xa6/0x1f0 [15288.627705] [] ? sync_inodes_sb+0x1c2/0x2a0 [15288.627707] [] _raw_spin_lock+0x40/0x80 [15288.627708] [] ? sync_inodes_sb+0x1c2/0x2a0 [15288.627710] [] sync_inodes_sb+0x1c2/0x2a0 [15288.627712] [] ? wait_for_completion+0xdf/0x110 [15288.627714] [] ? get_parent_ip+0xd/0x50 [15288.627715] [] ? generic_write_sync+0x70/0x70 [15288.627717] [] sync_inodes_one_sb+0x19/0x20 [15288.627719] [] iterate_supers+0xb2/0x110 [15288.627720] [] sys_sync+0x35/0x90 [15288.627721] [] tracesys+0xdd/0xe2 [15288.627736] Code: 48 89 45 b8 48 89 45 b0 48 89 45 a8 66 0f 1f 44 00 00 65 c7 04 25 80 0f 1d 00 00 00 00 00 e8 b7 31 06 00 fb 49 c7 c6 00 41 c0 81 0e 0f 1f 44 00 00 49 83 c6 08 41 d1 ef 74 6c 41 f6 c7 01 74 [15288.669173] NOHZ: local_softirq_pending 100 [15288.834639] Modules linked in: bridge 8021q garp stp tun fuse snd_seq_dummy rfcomm bnep nfnetlink hidp ipt_ULOG scsi_transport_iscsi can_bcm ipx p8023 p8022 x25 netrom nfc rose llc2 ax25 caif_socket irda caif pppoe pppox ppp_generic crc_ccitt slhc appletalk psnap can_raw llc can af_rxrpc atm af_key af_802154 rds phonet bluetooth rfkill coretemp hwmon kvm_intel kvm snd_hda_codec_realtek crc32c_intel ghash_clmulni_intel microcode snd_hda_codec_hdmi pcspkr snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device usb_debug snd_pcm e1000e snd_page_alloc snd_timer snd ptp pps_core soundcore xfs libcrc32c [15288.845739] irq event stamp: 68222 [15288.847174] hardirqs last enabled at (68221): [] restore_args+0x0/0x30 [15288.849205] hardirqs last disabled at (68222): [] apic_timer_interrupt+0x6a/0x80 [15288.851330] softirqs last enabled at (68220): [] __do_softirq+0x194/0x440 [15288.853395] softirqs last disabled at (68215): [] irq_exit+0xcd/0xe0 [15288.855402] CPU: 1 PID: 19176 Comm: trinity-child2 Not tainted 3.10.0-rc6+ #15 [loadavg: 41.18 18.72 10.56 17/176 19247] [15288.860325] task: ffff880235662520 ti: ffff8801ad19a000 task.ti: ffff8801ad19a000 [15288.862278] RIP: 0010:[] [] delay_tsc+0x73/0xe0 [15288.864273] RSP: 0018:ffff8801ad19bd50 EFLAGS: 00000202 [15288.865948] RAX: 000000000d73e038 RBX: ffffffff816ed220 RCX: 000000000000b910 [15288.867822] RDX: 0000000000001ffa RSI: 0000000000000001 RDI: 0000000000000001 [15288.869677] RBP: ffff8801ad19bd78 R08: 0000000000000001 R09: 0000000000000000 [15288.871504] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8801ad19bcc8 [15288.873310] R13: 0000000000000046 R14: ffff8801ad19a000 R15: ffff880235662520 [15288.875117] FS: 00007f4c11586740(0000) GS:ffff880244a00000(0000) knlGS:0000000000000000 [15288.877039] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15288.878701] CR2: 0000000003d52648 CR3: 000000021db18000 CR4: 00000000001407e0 [15288.880523] DR0: 0000000002ceeac0 DR1: 0000000000000000 DR2: 0000000000000000 [15288.882325] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [15288.884108] Stack: [15288.885273] ffffffff81c04640 000000002b4703f5 0000000088c820e8 0000000000000000 [15288.887126] ffffffff00000000 ffff8801ad19bd88 ffffffff8130947f ffff8801ad19bdb0 [15288.888974] ffffffff81311a91 ffffffff81c04640 ffffffff81c04658 ffff88023c4be8c0 [15288.890814] Call Trace: [15288.892029] [] __delay+0xf/0x20 [15288.893518] [] do_raw_spin_lock+0xe1/0x130 [15288.895124] [] _raw_spin_lock+0x60/0x80 [15288.896695] [] ? sync_inodes_sb+0x174/0x2a0 [15288.898297] [] sync_inodes_sb+0x174/0x2a0 [15288.899869] [] ? wait_for_completion+0xdf/0x110 [15288.901507] [] ? get_parent_ip+0xd/0x50 [15288.903057] [] ? generic_write_sync+0x70/0x70 [15288.904669] [] sync_inodes_one_sb+0x19/0x20 [15288.906260] [] iterate_supers+0xb2/0x110 [15288.907842] [] sys_sync+0x35/0x90 [15288.909320] [] tracesys+0xdd/0xe2 [15288.910793] Code: 3e 00 49 8b 87 38 e0 ff ff a8 08 75 5a f3 90 bf 01 00 00 00 e8 ef 76 3e 00 e8 8a 9e 00 00 41 39 c5 75 4b 0f 1f 00 0f ae e8 0f 31 <48> c1 e2 20 89 c0 48 09 c2 41 89 d6 29 da 44 39 e2 72 ba bf 01 [15336.600829] BUG: soft lockup - CPU#3 stuck for 23s! [trinity-child1:18506] [15336.602729] Modules linked in: bridge 8021q garp stp tun fuse snd_seq_dummy rfcomm bnep nfnetlink hidp ipt_ULOG scsi_transport_iscsi can_bcm ipx p8023 p8022 x25 netrom nfc rose llc2 ax25 caif_socket irda caif pppoe pppox ppp_generic crc_ccitt slhc appletalk psnap can_raw llc can af_rxrpc atm af_key af_802154 rds phonet bluetooth rfkill coretemp hwmon kvm_intel kvm snd_hda_codec_realtek crc32c_intel ghash_clmulni_intel microcode snd_hda_codec_hdmi pcspkr snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device usb_debug snd_pcm e1000e snd_page_alloc snd_timer snd ptp pps_core soundcore xfs libcrc32c [15336.615595] irq event stamp: 456216 [15336.617196] hardirqs last enabled at (456215): [] restore_args+0x0/0x30 [15336.619412] hardirqs last disabled at (456216): [] apic_timer_interrupt+0x6a/0x80 [15336.621723] softirqs last enabled at (456214): [] __do_softirq+0x194/0x440 [15336.623987] softirqs last disabled at (456209): [] irq_exit+0xcd/0xe0 [15336.626179] CPU: 3 PID: 18506 Comm: trinity-child1 Not tainted 3.10.0-rc6+ #15 [loadavg: 38.73 20.95 11.68 37/176 19867] [15336.631485] task: ffff88023b8b0000 ti: ffff88022706c000 task.ti: ffff88022706c000 [15336.633643] RIP: 0010:[] [] debug_smp_processor_id+0x1c/0xf0 [15336.636020] RSP: 0018:ffff88022706dd38 EFLAGS: 00000297 [15336.637939] RAX: 0000000000000002 RBX: ffff88022706c000 RCX: 000000000000b910 [15336.640076] RDX: 0000000000000c0b RSI: 0000000000000001 RDI: 0000000000000001 [15336.642222] RBP: ffff88022706dd40 R08: 0000000000000001 R09: 0000000000000000 [15336.644349] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000 [15336.646445] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001 [15336.648537] FS: 00007f1557d68740(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000 [15336.650717] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15336.652600] CR2: 0000003850abc720 CR3: 0000000227df8000 CR4: 00000000001407e0 [15336.654629] DR0: 0000000002ceeac0 DR1: 0000000000000000 DR2: 0000000000000000 [15336.656631] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [15336.658616] Stack: [15336.659980] ffffffff81c04640 ffff88022706dd78 ffffffff81309544 ffffffff81c04640 [15336.662177] 0000000028fb87d0 0000000088c820e8 0000000000000000 ffffffff00000000 [15336.664330] ffff88022706dd88 ffffffff8130947f ffff88022706ddb0 ffffffff81311a91 [15336.666517] Call Trace: [15336.667965] [] delay_tsc+0x24/0xe0 [15336.669686] [] __delay+0xf/0x20 [15336.671357] [] do_raw_spin_lock+0xe1/0x130 [15336.673132] [] _raw_spin_lock+0x60/0x80 [15336.674864] [] ? sync_inodes_sb+0x174/0x2a0 [15336.676632] [] sync_inodes_sb+0x174/0x2a0 [15336.678380] [] ? wait_for_completion+0xdf/0x110 [15336.680210] [] ? get_parent_ip+0xd/0x50 [15336.681954] [] ? generic_write_sync+0x70/0x70 [15336.683748] [] sync_inodes_one_sb+0x19/0x20 [15336.685519] [] iterate_supers+0xb2/0x110 [15336.687245] [] sys_sync+0x35/0x90 [15336.688896] [] tracesys+0xdd/0xe2 [15336.690538] Code: 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 0b 66 90 55 48 89 e5 53 65 48 8b 04 25 f0 b9 00 00 8b 80 44 e0 ff ff 65 8b 1c 25 1c b0 00 00 <85> c0 74 05 89 d8 5b 5d c3 9c 58 f6 c4 02 74 f4 89 d8 8b 15 a4 [15340.538629] BUG: soft lockup - CPU#0 stuck for 22s! [trinity-child0:18971] [15340.538639] BUG: soft lockup - CPU#1 stuck for 22s! [trinity-child0:17147] [15340.538661] Modules linked in: bridge 8021q garp stp tun fuse snd_seq_dummy rfcomm bnep nfnetlink hidp ipt_ULOG scsi_transport_iscsi can_bcm ipx p8023 p8022 x25 netrom nfc rose llc2 ax25 caif_socket irda caif pppoe pppox ppp_generic crc_ccitt slhc appletalk psnap can_raw llc can af_rxrpc atm af_key af_802154 rds phonet bluetooth rfkill coretemp hwmon kvm_intel kvm snd_hda_codec_realtek crc32c_intel ghash_clmulni_intel microcode snd_hda_codec_hdmi pcspkr snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device usb_debug snd_pcm e1000e snd_page_alloc snd_timer snd ptp pps_core soundcore xfs libcrc32c [15340.538661] irq event stamp: 14310743 [15340.538664] hardirqs last enabled at (14310742): [] restore_args+0x0/0x30 [15340.538667] hardirqs last disabled at (14310743): [] apic_timer_interrupt+0x6a/0x80 [15340.538669] softirqs last enabled at (14309968): [] __do_softirq+0x194/0x440 [15340.538671] softirqs last disabled at (14309971): [] irq_exit+0xcd/0xe0 [15340.538674] CPU: 1 PID: 17147 Comm: trinity-child0 Not tainted 3.10.0-rc6+ #15 [loadavg: 39.63 21.43 11.89 39/176 19867] [15340.538675] task: ffff88023c7ba520 ti: ffff88022b4c6000 task.ti: ffff88022b4c6000 [15340.538677] RIP: 0010:[] [] __do_softirq+0xb1/0x440 [15340.538678] RSP: 0018:ffff880244a03f08 EFLAGS: 00000202 [15340.538678] RAX: ffff88023c7ba520 RBX: ffffffff816ed220 RCX: 0000000000000002 [15340.538679] RDX: 00000000000031c0 RSI: ffff88023c7bad10 RDI: ffff88023c7ba520 [15340.538679] RBP: ffff880244a03f70 R08: 0000000000000003 R09: 0000000000000000 [15340.538680] R10: 0000000000000001 R11: 0000000000000000 R12: ffff880244a03e78 [15340.538680] R13: ffffffff816f5d2f R14: ffff880244a03f70 R15: 0000000000000000 [15340.538681] FS: 00007f87fde40740(0000) GS:ffff880244a00000(0000) knlGS:0000000000000000 [15340.538682] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15340.538682] CR2: 0000000000000000 CR3: 0000000172dca000 CR4: 00000000001407e0 [15340.538683] DR0: 0000000002ceeac0 DR1: 0000000000000000 DR2: 0000000000000000 [15340.538683] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [15340.538684] Stack: [15340.538685] 0000000a00406040 000000010016f665 ffff88022b4c7fd8 ffff88022b4c7fd8 [15340.538687] ffff88022b4c7fd8 ffff88023c7ba918 ffff88022b4c7fd8 ffffffff00000001 [15340.538688] ffff88023c7ba520 0000000000000000 ffff880241d60ba0 0000000000000246 [15340.538689] Call Trace: [15340.538690] [15340.538691] [] irq_exit+0xcd/0xe0 [15340.538693] [] smp_apic_timer_interrupt+0x6b/0x9b [15340.538695] [] apic_timer_interrupt+0x6f/0x80 [15340.538696] [15340.538697] [] ? retint_restore_args+0xe/0xe [15340.538699] [] ? lock_acquired+0x105/0x3f0 [15340.538701] [] ? sync_inodes_sb+0x1c2/0x2a0 [15340.538703] [] _raw_spin_lock+0x6c/0x80 [15340.538704] [] ? sync_inodes_sb+0x1c2/0x2a0 [15340.538706] [] sync_inodes_sb+0x1c2/0x2a0 [15340.538708] [] ? wait_for_completion+0xdf/0x110 [15340.538710] [] ? get_parent_ip+0xd/0x50 [15340.538711] [] ? generic_write_sync+0x70/0x70 [15340.538712] [] sync_inodes_one_sb+0x19/0x20 [15340.538714] [] iterate_supers+0xb2/0x110 [15340.538716] [] sys_sync+0x35/0x90 [15340.538717] [] tracesys+0xdd/0xe2 [15340.538732] Code: 48 89 45 b8 48 89 45 b0 48 89 45 a8 66 0f 1f 44 00 00 65 c7 04 25 80 0f 1d 00 00 00 00 00 e8 b7 31 06 00 fb 49 c7 c6 00 41 c0 81 0e 0f 1f 44 00 00 49 83 c6 08 41 d1 ef 74 6c 41 f6 c7 01 74 [15340.636008] Modules linked in: bridge 8021q garp stp tun fuse snd_seq_dummy rfcomm bnep nfnetlink hidp ipt_ULOG scsi_transport_iscsi can_bcm ipx p8023 p8022 x25 netrom nfc rose llc2 ax25 caif_socket irda caif pppoe pppox ppp_generic crc_ccitt slhc appletalk psnap can_raw llc can af_rxrpc atm af_key af_802154 rds phonet bluetooth rfkill coretemp hwmon kvm_intel kvm snd_hda_codec_realtek crc32c_intel ghash_clmulni_intel microcode snd_hda_codec_hdmi pcspkr snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device usb_debug snd_pcm e1000e snd_page_alloc snd_timer snd ptp pps_core soundcore xfs libcrc32c [15340.647482] irq event stamp: 407368 [15340.649023] hardirqs last enabled at (407367): [] restore_args+0x0/0x30 [15340.651157] hardirqs last disabled at (407368): [] apic_timer_interrupt+0x6a/0x80 [15340.653378] softirqs last enabled at (407366): [] __do_softirq+0x194/0x440 [15340.655540] softirqs last disabled at (407361): [] irq_exit+0xcd/0xe0 [15340.657646] CPU: 0 PID: 18971 Comm: trinity-child0 Not tainted 3.10.0-rc6+ #15 [loadavg: 39.63 21.43 11.89 21/176 19867] [15340.662750] task: ffff880227f5a520 ti: ffff880172e18000 task.ti: ffff880172e18000 [15340.664794] RIP: 0010:[] [] delay_tsc+0x2f/0xe0 [15340.666882] RSP: 0018:ffff880172e19d50 EFLAGS: 00000202 [15340.668677] RAX: 00000000d5af5ab1 RBX: ffffffff816ed220 RCX: 000000000000b910 [15340.670670] RDX: 0000000000002015 RSI: 0000000000000001 RDI: 0000000000000001 [15340.672641] RBP: ffff880172e19d78 R08: 0000000000000001 R09: 0000000000000000 [15340.674583] R10: 0000000000000000 R11: 0000000000000001 R12: ffff880172e19cc8 [15340.676496] R13: 0000000000000046 R14: ffff880172e18000 R15: ffff880227f5a520 [15340.678387] FS: 00007f1557d68740(0000) GS:ffff880244800000(0000) knlGS:0000000000000000 [15340.680390] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [15340.682128] CR2: 0000000000000000 CR3: 000000023c414000 CR4: 00000000001407f0 [15340.684036] DR0: 0000000002ceeac0 DR1: 0000000000000000 DR2: 0000000000000000 [15340.685935] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [15340.687825] Stack: [15340.689084] ffffffff81c04640 000000002ff51e5c 0000000088c820e8 0000000000000000 [15340.691024] ffffffff00000000 ffff880172e19d88 ffffffff8130947f ffff880172e19db0 [15340.692961] ffffffff81311a91 ffffffff81c04640 ffffffff81c04658 ffff88023c4be8c0 [15340.694885] Call Trace: [15340.696181] [] __delay+0xf/0x20 [15340.697750] [] do_raw_spin_lock+0xe1/0x130 [15340.699427] [] _raw_spin_lock+0x60/0x80 [15340.701072] [] ? sync_inodes_sb+0x174/0x2a0 [15340.702758] [] sync_inodes_sb+0x174/0x2a0 [15340.704417] [] ? wait_for_completion+0xdf/0x110 [15340.706130] [] ? get_parent_ip+0xd/0x50 [15340.707761] [] ? generic_write_sync+0x70/0x70 [15340.709448] [] sync_inodes_one_sb+0x19/0x20 [15340.711115] [] iterate_supers+0xb2/0x110 [15340.712748] [] sys_sync+0x35/0x90 [15340.714303] [] tracesys+0xdd/0xe2 [15340.715855] Code: 00 55 48 89 e5 41 57 41 56 41 55 41 54 41 89 fc bf 01 00 00 00 53 e8 31 77 3e 00 e8 cc 9e 00 00 41 89 c5 0f 1f 00 0f ae e8 0f 31 <65> 4c 8b 3c 25 f0 b9 00 00 89 c3 eb 2f 0f 1f 40 00 bf 01 00 00 [16915.716746] BUG: soft lockup - CPU#3 stuck for 22s! [trinity-child3:27517] [16915.718657] Modules linked in: bridge 8021q garp stp tun fuse snd_seq_dummy rfcomm bnep nfnetlink hidp ipt_ULOG scsi_transport_iscsi can_bcm ipx p8023 p8022 x25 netrom nfc rose llc2 ax25 caif_socket irda caif pppoe pppox ppp_generic crc_ccitt slhc appletalk psnap can_raw llc can af_rxrpc atm af_key af_802154 rds phonet bluetooth rfkill coretemp hwmon kvm_intel kvm snd_hda_codec_realtek crc32c_intel ghash_clmulni_intel microcode snd_hda_codec_hdmi pcspkr snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device usb_debug snd_pcm e1000e snd_page_alloc snd_timer snd ptp pps_core soundcore xfs libcrc32c [16915.731584] irq event stamp: 9669609 [16915.733194] hardirqs last enabled at (9669608): [] restore_args+0x0/0x30 [16915.735433] hardirqs last disabled at (9669609): [] apic_timer_interrupt+0x6a/0x80 [16915.737774] softirqs last enabled at (9668852): [] __do_softirq+0x194/0x440 [16915.740063] softirqs last disabled at (9668855): [] irq_exit+0xcd/0xe0 [16915.742271] CPU: 3 PID: 27517 Comm: trinity-child3 Not tainted 3.10.0-rc6+ #15 [loadavg: 28.58 14.79 11.37 49/189 3213] [16915.747582] task: ffff8801f1000000 ti: ffff880233958000 task.ti: ffff880233958000 [16915.749746] RIP: 0010:[] [] __do_softirq+0xb1/0x440 [16915.752035] RSP: 0018:ffff880244e03f08 EFLAGS: 00000202 [16915.753953] RAX: ffff8801f1000000 RBX: ffffffff816ed220 RCX: 0000000000000002 [16915.756099] RDX: 00000000000031c0 RSI: ffff8801f10007f0 RDI: ffff8801f1000000 [16915.758243] RBP: ffff880244e03f70 R08: 0000000000000003 R09: 0000000000000000 [16915.760404] R10: 0000000000000001 R11: 0000000000000000 R12: ffff880244e03e78 [16915.762533] R13: ffffffff816f5d2f R14: ffff880244e03f70 R15: 0000000000000000 [16915.764648] FS: 00007faecbd33740(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000 [16915.766862] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16915.768776] CR2: 0000000000000001 CR3: 00000002057de000 CR4: 00000000001407e0 [16915.770835] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16915.772855] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [16915.774855] Stack: [16915.776249] 0000000a00406040 0000000100195e0b ffff880233959fd8 ffff880233959fd8 [16915.778399] ffff880233959fd8 ffff8801f10003f8 ffff880233959fd8 ffffffff00000003 [16915.780578] ffff8801f1000000 0000000000000000 ffff8802221076e0 0000000000000246 [16915.782777] Call Trace: [16915.784228] [16915.785863] [] irq_exit+0xcd/0xe0 [16915.787410] [] smp_apic_timer_interrupt+0x6b/0x9b [16915.789263] [] apic_timer_interrupt+0x6f/0x80 [16915.791078] [16915.792628] [] ? lock_acquired+0x105/0x3f0 [16915.794252] [] ? sync_inodes_sb+0x1c2/0x2a0 [16915.796044] [] _raw_spin_lock+0x6c/0x80 [16915.797789] [] ? sync_inodes_sb+0x1c2/0x2a0 [16915.799577] [] sync_inodes_sb+0x1c2/0x2a0 [16915.801346] [] ? wait_for_completion+0x36/0x110 [16915.803157] [] ? get_parent_ip+0xd/0x50 [16915.804901] [] ? generic_write_sync+0x70/0x70 [16915.806696] [] sync_inodes_one_sb+0x19/0x20 [16915.808467] [] iterate_supers+0xb2/0x110 [16915.810214] [] sys_sync+0x35/0x90 [16915.811880] [] tracesys+0xdd/0xe2 [16915.813523] Code: 48 89 45 b8 48 89 45 b0 48 89 45 a8 66 0f 1f 44 00 00 65 c7 04 25 80 0f 1d 00 00 00 00 00 e8 b7 31 06 00 fb 49 c7 c6 00 41 c0 81 0e 0f 1f 44 00 00 49 83 c6 08 41 d1 ef 74 6c 41 f6 c7 01 74 [16939.703310] BUG: soft lockup - CPU#3 stuck for 22s! [trinity-child3:27517] [16939.705133] Modules linked in: bridge 8021q garp stp tun fuse snd_seq_dummy rfcomm bnep nfnetlink hidp ipt_ULOG scsi_transport_iscsi can_bcm ipx p8023 p8022 x25 netrom nfc rose llc2 ax25 caif_socket irda caif pppoe pppox ppp_generic crc_ccitt slhc appletalk psnap can_raw llc can af_rxrpc atm af_key af_802154 rds phonet bluetooth rfkill coretemp hwmon kvm_intel kvm snd_hda_codec_realtek crc32c_intel ghash_clmulni_intel microcode snd_hda_codec_hdmi pcspkr snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device usb_debug snd_pcm e1000e snd_page_alloc snd_timer snd ptp pps_core soundcore xfs libcrc32c [16939.718151] irq event stamp: 11403791 [16939.719810] hardirqs last enabled at (11403790): [] restore_args+0x0/0x30 [16939.722081] hardirqs last disabled at (11403791): [] apic_timer_interrupt+0x6a/0x80 [16939.724438] softirqs last enabled at (11403034): [] __do_softirq+0x194/0x440 [16939.726746] softirqs last disabled at (11403037): [] irq_exit+0xcd/0xe0 [16939.728991] CPU: 3 PID: 27517 Comm: trinity-child3 Not tainted 3.10.0-rc6+ #15 [loadavg: 36.37 17.42 12.30 47/197 3394] [16939.734329] task: ffff8801f1000000 ti: ffff880233958000 task.ti: ffff880233958000 [16939.736489] RIP: 0010:[] [] __do_softirq+0xb1/0x440 [16939.738762] RSP: 0018:ffff880244e03f08 EFLAGS: 00000206 [16939.740654] RAX: ffff8801f1000000 RBX: ffffffff816ed220 RCX: 0000000000000002 [16939.742738] RDX: 00000000000031c0 RSI: ffff8801f10007f0 RDI: ffff8801f1000000 [16939.744793] RBP: ffff880244e03f70 R08: 0000000000000003 R09: 0000000000000000 [16939.746828] R10: 0000000000000001 R11: 0000000000000000 R12: ffff880244e03e78 [16939.748837] R13: ffffffff816f5d2f R14: ffff880244e03f70 R15: 0000000000000000 [16939.750832] FS: 00007faecbd33740(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000 [16939.752943] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [16939.754778] CR2: 0000000000000001 CR3: 00000002057de000 CR4: 00000000001407e0 [16939.756790] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [16939.758794] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600 [16939.760769] Stack: [16939.762144] 0000000a00406040 000000010019676b ffff880233959fd8 ffff880233959fd8 [16939.764307] ffff880233959fd8 ffff8801f10003f8 ffff880233959fd8 ffffffff00000003 [16939.766457] ffff8801f1000000 0000000000000000 0000000000000000 0000000000000002 [16939.768554] Call Trace: [16939.769940] [16939.771528] [] irq_exit+0xcd/0xe0 [16939.773062] [] smp_apic_timer_interrupt+0x6b/0x9b [16939.774909] [] apic_timer_interrupt+0x6f/0x80 [16939.776719] [16939.778270] [] ? retint_restore_args+0xe/0xe [16939.779910] [] ? lock_acquire+0xa6/0x1f0 [16939.781660] [