Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753220AbbG0MyM (ORCPT ); Mon, 27 Jul 2015 08:54:12 -0400 Received: from mail-wi0-f181.google.com ([209.85.212.181]:32869 "EHLO mail-wi0-f181.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752515AbbG0MyJ (ORCPT ); Mon, 27 Jul 2015 08:54:09 -0400 To: "Linux-Kernel@Vger. Kernel. Org" Cc: oleg@redhat.com, riel@redhat.com, Peter Zijlstra , Steven Rostedt From: Nikolay Borisov Subject: HARD LOCKUP: Strange hard lock up on spin_lock(&sighand->siglock); Message-ID: <55B629E9.6020207@kyup.com> Date: Mon, 27 Jul 2015 15:54:01 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 73050 Lines: 837 Hello, I have a machine running kernel 3.13.3, in fact I have 3 such machines which are connected via corosync in a cluster. On one of the machines I observed the following lock-up: Jul 26 06:01:14 shrek kernel: ------------[ cut here ]------------ Jul 26 06:01:14 shrek kernel: WARNING: CPU: 6 PID: 30236 at kernel/watchdog.c:245 watchdog_overflow_callback+0x98/0xc0() Jul 26 06:01:14 shrek kernel: Watchdog detected hard LOCKUP on cpu 6 Jul 26 06:01:14 shrek kernel: Modules linked in: dm_snapshot netconsole configfs fuse bonding ipv6 xt_state iptable_mangle xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_raw dm_mirror dm_region_hash dm_log lpc_ich mfd_core ioapic ioatdma ipmi_devintf ipmi_si ipmi_msghandler igb(O) dca Jul 26 06:01:14 shrek kernel: CPU: 6 PID: 30236 Comm: pidof Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: 00000000000000f5 ffff88087fc06bb8 ffffffff815a6c8e 00000000000000f5 Jul 26 06:01:14 shrek kernel: ffff88087fc06c08 ffff88087fc06bf8 ffffffff81044202 ffff88087fc06bd8 Jul 26 06:01:14 shrek kernel: ffff880869940000 0000000000000000 ffff88087fc06d48 0000000000000000 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] dump_stack+0x49/0x5b Jul 26 06:01:14 shrek kernel: [] warn_slowpath_common+0x82/0xb0 Jul 26 06:01:14 shrek kernel: [] warn_slowpath_fmt+0x41/0x50 Jul 26 06:01:14 shrek kernel: [] watchdog_overflow_callback+0x98/0xc0 Jul 26 06:01:14 shrek kernel: [] __perf_event_overflow+0x98/0x240 Jul 26 06:01:14 shrek kernel: [] perf_event_overflow+0x14/0x20 Jul 26 06:01:14 shrek kernel: [] intel_pmu_handle_irq+0x1cc/0x3b0 Jul 26 06:01:14 shrek kernel: [] ? _raw_spin_lock+0x1c/0x30 Jul 26 06:01:14 shrek kernel: [] ? apei_read+0x60/0x90 Jul 26 06:01:14 shrek kernel: [] perf_event_nmi_handler+0x34/0x60 Jul 26 06:01:14 shrek kernel: [] nmi_handle+0x76/0x110 Jul 26 06:01:14 shrek kernel: [] do_nmi+0xfd/0x360 Jul 26 06:01:14 shrek kernel: [] end_repeat_nmi+0x1e/0x32 Jul 26 06:01:14 shrek kernel: [] ? _raw_spin_lock+0x1c/0x30 Jul 26 06:01:14 shrek kernel: [] ? _raw_spin_lock+0x1c/0x30 Jul 26 06:01:14 shrek kernel: [] ? _raw_spin_lock+0x1c/0x30 Jul 26 06:01:14 shrek kernel: <> [] __lock_task_sighand+0x44/0x80 Jul 26 06:01:14 shrek kernel: [] do_task_stat+0x102/0xbb0 Jul 26 06:01:14 shrek kernel: [] proc_tgid_stat+0xf/0x20 Jul 26 06:01:14 shrek kernel: [] proc_single_show+0x5f/0xa0 Jul 26 06:01:14 shrek kernel: [] seq_read+0x192/0x670 Jul 26 06:01:14 shrek kernel: [] vfs_read+0xd5/0x230 Jul 26 06:01:14 shrek kernel: [] SyS_read+0x5a/0xa0 Jul 26 06:01:14 shrek kernel: [] system_call_fastpath+0x16/0x1b Jul 26 06:01:14 shrek kernel: ---[ end trace 13447e4cb40ed84d ]--- Jul 26 06:01:14 shrek kernel: ------------[ cut here ]------------ Jul 26 06:01:14 shrek kernel: WARNING: CPU: 9 PID: 30215 at kernel/watchdog.c:245 watchdog_overflow_callback+0x98/0xc0() Jul 26 06:01:14 shrek kernel: Watchdog detected hard LOCKUP on cpu 9 Jul 26 06:01:14 shrek kernel: Modules linked in: dm_snapshot netconsole configfs fuse bonding ipv6 xt_state iptable_mangle xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_raw dm_mirror dm_region_hash dm_log lpc_ich mfd_core ioapic ioatdma ipmi_devintf ipmi_si ipmi_msghandler igb(O) dca Jul 26 06:01:14 shrek kernel: CPU: 9 PID: 30215 Comm: pkill Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: 00000000000000f5 ffff88087fc66bb8 ffffffff815a6c8e 00000000000000f5 Jul 26 06:01:14 shrek kernel: ffff88087fc66c08 ffff88087fc66bf8 ffffffff81044202 ffff88087fc66b10 Jul 26 06:01:14 shrek kernel: ffff880869990000 0000000000000000 ffff88087fc66d48 0000000000000000 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] dump_stack+0x49/0x5b Jul 26 06:01:14 shrek kernel: [] warn_slowpath_common+0x82/0xb0 Jul 26 06:01:14 shrek kernel: [] warn_slowpath_fmt+0x41/0x50 Jul 26 06:01:14 shrek kernel: [] watchdog_overflow_callback+0x98/0xc0 Jul 26 06:01:14 shrek kernel: [] __perf_event_overflow+0x98/0x240 Jul 26 06:01:14 shrek kernel: [] ? hw_nmi_get_sample_period+0x20/0x20 Jul 26 06:01:14 shrek kernel: [] perf_event_overflow+0x14/0x20 Jul 26 06:01:14 shrek kernel: [] intel_pmu_handle_irq+0x1cc/0x3b0 Jul 26 06:01:14 shrek kernel: [] ? apei_read+0x60/0x90 Jul 26 06:01:14 shrek kernel: [] perf_event_nmi_handler+0x34/0x60 Jul 26 06:01:14 shrek kernel: [] nmi_handle+0x76/0x110 Jul 26 06:01:14 shrek kernel: [] do_nmi+0xfd/0x360 Jul 26 06:01:14 shrek kernel: [] end_repeat_nmi+0x1e/0x32 Jul 26 06:01:14 shrek kernel: [] ? _raw_spin_lock+0x1f/0x30 Jul 26 06:01:14 shrek kernel: [] ? _raw_spin_lock+0x1f/0x30 Jul 26 06:01:14 shrek kernel: [] ? _raw_spin_lock+0x1f/0x30 Jul 26 06:01:14 shrek kernel: <> [] __lock_task_sighand+0x44/0x80 Jul 26 06:01:14 shrek kernel: [] proc_pid_status+0x3ea/0x7c0 Jul 26 06:01:14 shrek kernel: [] proc_single_show+0x5f/0xa0 Jul 26 06:01:14 shrek kernel: [] seq_read+0x192/0x670 Jul 26 06:01:14 shrek kernel: [] vfs_read+0xd5/0x230 Jul 26 06:01:14 shrek kernel: [] SyS_read+0x5a/0xa0 Jul 26 06:01:14 shrek kernel: [] system_call_fastpath+0x16/0x1b Jul 26 06:01:14 shrek kernel: [] ? sysret_check+0x26/0x65 Jul 26 06:01:14 shrek kernel: ---[ end trace 13447e4cb40ed84e ]--- Jul 26 06:01:14 shrek kernel: ------------[ cut here ]------------ Jul 26 06:01:14 shrek kernel: WARNING: CPU: 10 PID: 30390 at kernel/watchdog.c:245 watchdog_overflow_callback+0x98/0xc0() Jul 26 06:01:14 shrek kernel: Watchdog detected hard LOCKUP on cpu 10 Jul 26 06:01:14 shrek kernel: Modules linked in: dm_snapshot netconsole configfs fuse bonding ipv6 xt_state iptable_mangle xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_raw dm_mirror dm_region_hash dm_log lpc_ich mfd_core ioapic ioatdma ipmi_devintf ipmi_si ipmi_msghandler igb(O) dca Jul 26 06:01:14 shrek kernel: CPU: 10 PID: 30390 Comm: pidof Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: 00000000000000f5 ffff88087fc86bb8 ffffffff815a6c8e 00000000000000f5 Jul 26 06:01:14 shrek kernel: ffff88087fc86c08 ffff88087fc86bf8 ffffffff81044202 ffff88087fc86bd8 Jul 26 06:01:14 shrek kernel: ffff8808699b0000 0000000000000000 ffff88087fc86d48 0000000000000000 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] dump_stack+0x49/0x5b Jul 26 06:01:14 shrek kernel: [] warn_slowpath_common+0x82/0xb0 Jul 26 06:01:14 shrek kernel: [] warn_slowpath_fmt+0x41/0x50 Jul 26 06:01:14 shrek kernel: [] watchdog_overflow_callback+0x98/0xc0 Jul 26 06:01:14 shrek kernel: [] __perf_event_overflow+0x98/0x240 Jul 26 06:01:14 shrek kernel: [] perf_event_overflow+0x14/0x20 Jul 26 06:01:14 shrek kernel: [] intel_pmu_handle_irq+0x1cc/0x3b0 Jul 26 06:01:14 shrek kernel: [] ? _raw_spin_lock+0x1c/0x30 Jul 26 06:01:14 shrek kernel: [] ? apei_read+0x60/0x90 Jul 26 06:01:14 shrek kernel: [] perf_event_nmi_handler+0x34/0x60 Jul 26 06:01:14 shrek kernel: [] nmi_handle+0x76/0x110 Jul 26 06:01:14 shrek kernel: [] do_nmi+0xfd/0x360 Jul 26 06:01:14 shrek kernel: [] end_repeat_nmi+0x1e/0x32 Jul 26 06:01:14 shrek kernel: [] ? _raw_spin_lock+0x1c/0x30 Jul 26 06:01:14 shrek kernel: [] ? _raw_spin_lock+0x1c/0x30 Jul 26 06:01:14 shrek kernel: [] ? _raw_spin_lock+0x1c/0x30 Jul 26 06:01:14 shrek kernel: <> [] __lock_task_sighand+0x44/0x80 Jul 26 06:01:14 shrek kernel: [] do_task_stat+0x102/0xbb0 Jul 26 06:01:14 shrek kernel: [] proc_tgid_stat+0xf/0x20 Jul 26 06:01:14 shrek kernel: [] proc_single_show+0x5f/0xa0 Jul 26 06:01:14 shrek kernel: [] seq_read+0x192/0x670 Jul 26 06:01:14 shrek kernel: [] vfs_read+0xd5/0x230 Jul 26 06:01:14 shrek kernel: [] SyS_read+0x5a/0xa0 Jul 26 06:01:14 shrek kernel: [] system_call_fastpath+0x16/0x1b Jul 26 06:01:14 shrek kernel: [] ? sysret_check+0x26/0x65 Jul 26 06:01:14 shrek kernel: ---[ end trace 13447e4cb40ed84f ]--- Jul 26 06:01:14 shrek kernel: ------------[ cut here ]------------ Jul 26 06:01:14 shrek kernel: WARNING: CPU: 2 PID: 30144 at kernel/watchdog.c:245 watchdog_overflow_callback+0x98/0xc0() Jul 26 06:01:14 shrek kernel: Watchdog detected hard LOCKUP on cpu 2 Jul 26 06:01:14 shrek kernel: Modules linked in: dm_snapshot netconsole configfs fuse bonding ipv6 xt_state iptable_mangle xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_raw dm_mirror dm_region_hash dm_log lpc_ich mfd_core ioapic ioatdma ipmi_devintf ipmi_si ipmi_msghandler igb(O) dca Jul 26 06:01:14 shrek kernel: CPU: 2 PID: 30144 Comm: pidof Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: 00000000000000f5 ffff88046fc46bb8 ffffffff815a6c8e 00000000000000f5 Jul 26 06:01:14 shrek kernel: ffff88046fc46c08 ffff88046fc46bf8 ffffffff81044202 ffff880867bd2510 Jul 26 06:01:14 shrek kernel: ffff880469de0000 0000000000000000 ffff88046fc46d48 0000000000000000 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] dump_stack+0x49/0x5b Jul 26 06:01:14 shrek kernel: [] warn_slowpath_common+0x82/0xb0 Jul 26 06:01:14 shrek kernel: [] warn_slowpath_fmt+0x41/0x50 Jul 26 06:01:14 shrek kernel: [] watchdog_overflow_callback+0x98/0xc0 Jul 26 06:01:14 shrek kernel: [] __perf_event_overflow+0x98/0x240 Jul 26 06:01:14 shrek kernel: [] ? hw_nmi_get_sample_period+0x20/0x20 Jul 26 06:01:14 shrek kernel: [] perf_event_overflow+0x14/0x20 Jul 26 06:01:14 shrek kernel: [] intel_pmu_handle_irq+0x1cc/0x3b0 Jul 26 06:01:14 shrek kernel: [] ? apei_read+0x60/0x90 Jul 26 06:01:14 shrek kernel: [] perf_event_nmi_handler+0x34/0x60 Jul 26 06:01:14 shrek kernel: [] nmi_handle+0x76/0x110 Jul 26 06:01:14 shrek kernel: [] do_nmi+0xfd/0x360 Jul 26 06:01:14 shrek kernel: [] end_repeat_nmi+0x1e/0x32 Jul 26 06:01:14 shrek kernel: [] ? _raw_spin_lock+0x1c/0x30 Jul 26 06:01:14 shrek kernel: [] ? _raw_spin_lock+0x1c/0x30 Jul 26 06:01:14 shrek kernel: [] ? _raw_spin_lock+0x1c/0x30 Jul 26 06:01:14 shrek kernel: <> [] __lock_task_sighand+0x44/0x80 Jul 26 06:01:14 shrek kernel: [] do_task_stat+0x102/0xbb0 Jul 26 06:01:14 shrek kernel: [] proc_tgid_stat+0xf/0x20 Jul 26 06:01:14 shrek kernel: [] proc_single_show+0x5f/0xa0 Jul 26 06:01:14 shrek kernel: [] seq_read+0x192/0x670 Jul 26 06:01:14 shrek kernel: [] vfs_read+0xd5/0x230 Jul 26 06:01:14 shrek kernel: [] SyS_read+0x5a/0xa0 Jul 26 06:01:14 shrek kernel: [] system_call_fastpath+0x16/0x1b Jul 26 06:01:14 shrek kernel: ---[ end trace 13447e4cb40ed850 ]--- Jul 26 06:01:14 shrek kernel: INFO: rcu_sched detected stalls on CPUs/tasks: { 8 18} (detected by 12, t=14712 jiffies, g=115985071, c=115985070, q=186073) Jul 26 06:01:14 shrek kernel: sending NMI to all CPUs: Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 1 Jul 26 06:01:14 shrek kernel: CPU: 1 PID: 30726 Comm: pidof Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff88013d551260 ti: ffff88013d5517b8 task.ti: ffff88013d5517b8 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff815ab15c Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff880300eb5b08 EFLAGS: 00000093 Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000 Jul 26 06:01:14 shrek kernel: RDX: 000000000000000a RSI: ffff880300eb5c30 RDI: ffff880621f44a08 Jul 26 06:01:14 shrek kernel: RBP: ffff880300eb5b08 R08: 0000000000000001 R09: 0000000000000000 Jul 26 06:01:14 shrek kernel: R10: 0000000000000022 R11: 0000000000000001 R12: ffff880621f44200 Jul 26 06:01:14 shrek kernel: R13: ffff880300eb5c30 R14: ffff880622797420 R15: ffff8801fd476300 Jul 26 06:01:14 shrek kernel: FS: 0000034a18fe5700(0000) GS:ffff88046fc20000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: 00000000019bfc18 CR3: 00000000015ba000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff880300eb5b38 ffffffff81053474 ffff880300ff2e10 ffff880622796e40 Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff880300eb5c98 ffffffff8119dd02 Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] __lock_task_sighand+0x44/0x80 Jul 26 06:01:14 shrek kernel: [] do_task_stat+0x102/0xbb0 Jul 26 06:01:14 shrek kernel: [] proc_tgid_stat+0xf/0x20 Jul 26 06:01:14 shrek kernel: [] proc_single_show+0x5f/0xa0 Jul 26 06:01:14 shrek kernel: [] seq_read+0x192/0x670 Jul 26 06:01:14 shrek kernel: [] vfs_read+0xd5/0x230 Jul 26 06:01:14 shrek kernel: [] SyS_read+0x5a/0xa0 Jul 26 06:01:14 shrek kernel: [] system_call_fastpath+0x16/0x1b Jul 26 06:01:14 shrek kernel: [] ? sysret_check+0x26/0x65 Jul 26 06:01:14 shrek kernel: Code: 38 c2 75 f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 <0f> b6 07 38 c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55 Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 0 Jul 26 06:01:14 shrek kernel: CPU: 0 PID: 0 Comm: swapper/0 Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffffffff81a0a920 ti: ffffffff81a0ae78 task.ti: ffffffff81a0ae78 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff8100c456 Jul 26 06:01:14 shrek kernel: RSP: 0000:ffffffff81a01ec8 EFLAGS: 00000286 Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffffffff81a0ae80 RCX: 0000000000000000 Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fc0c200 Jul 26 06:01:14 shrek kernel: RBP: ffffffff81a01ec8 R08: 0000000000000000 R09: 0000000000000000 Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81a0ae80 Jul 26 06:01:14 shrek kernel: R13: ffffffff81a0ae80 R14: ffffffff81afa5c8 R15: ffffffff81a0ae80 Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fc00000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: 000002dd81723ae0 CR3: 00000000015b8000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffffffff81a01ed8 ffffffff8100c6b6 ffffffff81a01f48 ffffffff810897b9 Jul 26 06:01:14 shrek kernel: 0000000000000286 ffffffff81a0ae78 ffffffff81a0ae80 ffffffff81a0ae78 Jul 26 06:01:14 shrek kernel: ffffffff81885200 77e46a1d0cffc667 00000000000937e0 ffffffff81c940c0 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] arch_cpu_idle+0x16/0x20 Jul 26 06:01:14 shrek kernel: [] cpu_startup_entry+0xc9/0x1b0 Jul 26 06:01:14 shrek kernel: [] rest_init+0x72/0x80 Jul 26 06:01:14 shrek kernel: [] start_kernel+0x3f5/0x3fc Jul 26 06:01:14 shrek kernel: [] ? repair_env_string+0x62/0x62 Jul 26 06:01:14 shrek kernel: [] ? memblock_reserve+0x46/0x4b Jul 26 06:01:14 shrek kernel: [] ? early_idt_handlers+0x120/0x120 Jul 26 06:01:14 shrek kernel: [] x86_64_start_reservations+0x2a/0x2c Jul 26 06:01:14 shrek kernel: [] x86_64_start_kernel+0x154/0x15f Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65 Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 2 Jul 26 06:01:14 shrek kernel: CPU: 2 PID: 30144 Comm: pidof Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff880867824980 ti: ffff880867824ed8 task.ti: ffff880867824ed8 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff815ab15f Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff88063e911bc8 EFLAGS: 00000093 Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000 Jul 26 06:01:14 shrek kernel: RDX: 0000000000000004 RSI: ffff88063e911cf0 RDI: ffff880621f44a08 Jul 26 06:01:14 shrek kernel: RBP: ffff88063e911bc8 R08: 0000000000000001 R09: 0000000000000000 Jul 26 06:01:14 shrek kernel: R10: 0000000000000022 R11: 0000000000000001 R12: ffff880621f44200 Jul 26 06:01:14 shrek kernel: R13: ffff88063e911cf0 R14: ffff880622797420 R15: ffff8801fd476300 Jul 26 06:01:14 shrek kernel: FS: 000002d271e24700(0000) GS:ffff88046fc40000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: 00000000010da108 CR3: 00000000015bc000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff88063e911bf8 ffffffff81053474 ffff88037e541ce8 ffff880622796e40 Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff88063e911d58 ffffffff8119dd02 Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] __lock_task_sighand+0x44/0x80 Jul 26 06:01:14 shrek kernel: [] do_task_stat+0x102/0xbb0 Jul 26 06:01:14 shrek kernel: [] proc_tgid_stat+0xf/0x20 Jul 26 06:01:14 shrek kernel: [] proc_single_show+0x5f/0xa0 Jul 26 06:01:14 shrek kernel: [] seq_read+0x192/0x670 Jul 26 06:01:14 shrek kernel: [] vfs_read+0xd5/0x230 Jul 26 06:01:14 shrek kernel: [] SyS_read+0x5a/0xa0 Jul 26 06:01:14 shrek kernel: [] system_call_fastpath+0x16/0x1b Jul 26 06:01:14 shrek kernel: Code: f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 0f b6 07 <38> c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55 48 89 e5 Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 3 Jul 26 06:01:14 shrek kernel: CPU: 3 PID: 1270 Comm: pidof Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff8806883ad2b0 ti: ffff8806883ad808 task.ti: ffff8806883ad808 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff815ab15a Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff8805b3375b48 EFLAGS: 00000097 Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000 Jul 26 06:01:14 shrek kernel: RDX: 000000000000000b RSI: ffff8805b3375c70 RDI: ffff880621f44a08 Jul 26 06:01:14 shrek kernel: RBP: ffff8805b3375b48 R08: 0000000000000001 R09: 0000000000000000 Jul 26 06:01:14 shrek kernel: R10: 0000000000000022 R11: 0000000000000001 R12: ffff880621f44200 Jul 26 06:01:14 shrek kernel: R13: ffff8805b3375c70 R14: ffff880622797420 R15: ffff8801fd476300 Jul 26 06:01:14 shrek kernel: FS: 0000029bd8f09700(0000) GS:ffff88046fc60000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: 0000000003ca0498 CR3: 00000000015be000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff8805b3375b78 ffffffff81053474 ffff88010261a258 ffff880622796e40 Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff8805b3375cd8 ffffffff8119dd02 Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] __lock_task_sighand+0x44/0x80 Jul 26 06:01:14 shrek kernel: [] do_task_stat+0x102/0xbb0 Jul 26 06:01:14 shrek kernel: [] proc_tgid_stat+0xf/0x20 Jul 26 06:01:14 shrek kernel: [] proc_single_show+0x5f/0xa0 Jul 26 06:01:14 shrek kernel: [] seq_read+0x192/0x670 Jul 26 06:01:14 shrek kernel: [] vfs_read+0xd5/0x230 Jul 26 06:01:14 shrek kernel: [] SyS_read+0x5a/0xa0 Jul 26 06:01:14 shrek kernel: [] system_call_fastpath+0x16/0x1b Jul 26 06:01:14 shrek kernel: [] ? sysret_check+0x26/0x65 Jul 26 06:01:14 shrek kernel: Code: b6 03 38 c2 75 f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 90 0f b6 07 38 c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 4 Jul 26 06:01:14 shrek kernel: CPU: 4 PID: 0 Comm: swapper/4 Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff880469d70930 ti: ffff880469d70e88 task.ti: ffff880469d70e88 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff8100c456 Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d79e88 EFLAGS: 00000286 Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d70e90 RCX: 0000000000000000 Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fc8c200 Jul 26 06:01:14 shrek kernel: RBP: ffff880469d79e88 R08: 0000000000000000 R09: 0000000000000000 Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d70e90 Jul 26 06:01:14 shrek kernel: R13: ffff880469d70e90 R14: ffffffff81afa5c8 R15: ffff880469d70e90 Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fc80000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: 000002e423d730a0 CR3: 00000000015c0000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff880469d79e98 ffffffff8100c6b6 ffff880469d79f08 ffffffff810897b9 Jul 26 06:01:14 shrek kernel: 0000000000000004 ffff880469d70e88 ffff880469d70e90 ffff880469d70e88 Jul 26 06:01:14 shrek kernel: 000000000043fd9f b6726a4c2c676c66 ffff880469d79ef8 0000000000000004 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] arch_cpu_idle+0x16/0x20 Jul 26 06:01:14 shrek kernel: [] cpu_startup_entry+0xc9/0x1b0 Jul 26 06:01:14 shrek kernel: [] start_secondary+0x238/0x350 Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65 Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 5 Jul 26 06:01:14 shrek kernel: CPU: 5 PID: 0 Comm: swapper/5 Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff880469d71260 ti: ffff880469d717b8 task.ti: ffff880469d717b8 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff8100c456 Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d7be88 EFLAGS: 00000286 Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d717c0 RCX: 0000000000000000 Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fcac200 Jul 26 06:01:14 shrek kernel: RBP: ffff880469d7be88 R08: 0000000000000000 R09: 0000000000000000 Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d717c0 Jul 26 06:01:14 shrek kernel: R13: ffff880469d717c0 R14: ffffffff81afa5c8 R15: ffff880469d717c0 Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fca0000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: ffffffffff600400 CR3: 00000000015c2000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff880469d7be98 ffffffff8100c6b6 ffff880469d7bf08 ffffffff810897b9 Jul 26 06:01:14 shrek kernel: 0000000000000005 ffff880469d717b8 ffff880469d717c0 ffff880469d717b8 Jul 26 06:01:14 shrek kernel: 0000000000000206 a8d39d71d1e5d619 ffff880469d7bef8 0000000000000005 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] arch_cpu_idle+0x16/0x20 Jul 26 06:01:14 shrek kernel: [] cpu_startup_entry+0xc9/0x1b0 Jul 26 06:01:14 shrek kernel: [] start_secondary+0x238/0x350 Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65 Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 6 Jul 26 06:01:14 shrek kernel: INFO: NMI handler (ffffffff8102d460) took too long to run: 45.273 msecs Jul 26 06:01:14 shrek kernel: CPU: 6 PID: 30236 Comm: pidof Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff8806a1f78930 ti: ffff8806a1f78e88 task.ti: ffff8806a1f78e88 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff815ab15c Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff880801077be8 EFLAGS: 00000093 Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000 Jul 26 06:01:14 shrek kernel: RDX: 0000000000000008 RSI: ffff880801077d10 RDI: ffff880621f44a08 Jul 26 06:01:14 shrek kernel: RBP: ffff880801077be8 R08: 0000000000000001 R09: 0000000000000000 Jul 26 06:01:14 shrek kernel: R10: 0000000000000022 R11: 0000000000000001 R12: ffff880621f44200 Jul 26 06:01:14 shrek kernel: R13: ffff880801077d10 R14: ffff880622797420 R15: ffff8801fd476300 Jul 26 06:01:14 shrek kernel: FS: 000002fb353c6700(0000) GS:ffff88087fc00000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: 00000000021c82a8 CR3: 00000000015c4000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff880801077c18 ffffffff81053474 ffff88012c217b58 ffff880622796e40 Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff880801077d78 ffffffff8119dd02 Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] __lock_task_sighand+0x44/0x80 Jul 26 06:01:14 shrek kernel: [] do_task_stat+0x102/0xbb0 Jul 26 06:01:14 shrek kernel: [] proc_tgid_stat+0xf/0x20 Jul 26 06:01:14 shrek kernel: [] proc_single_show+0x5f/0xa0 Jul 26 06:01:14 shrek kernel: [] seq_read+0x192/0x670 Jul 26 06:01:14 shrek kernel: [] vfs_read+0xd5/0x230 Jul 26 06:01:14 shrek kernel: [] SyS_read+0x5a/0xa0 Jul 26 06:01:14 shrek kernel: [] system_call_fastpath+0x16/0x1b Jul 26 06:01:14 shrek kernel: Code: 38 c2 75 f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 <0f> b6 07 38 c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55 Jul 26 06:01:14 shrek kernel: INFO: NMI handler (ffffffff8102d460) took too long to run: 52.833 msecs Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 7 Jul 26 06:01:14 shrek kernel: CPU: 7 PID: 29973 Comm: ps Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff880867821b90 ti: ffff8808678220e8 task.ti: ffff8808678220e8 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff815ab15c Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff8807ed2fdb08 EFLAGS: 00000097 Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000 Jul 26 06:01:14 shrek kernel: RDX: 0000000000000002 RSI: ffff8807ed2fdc30 RDI: ffff880621f44a08 Jul 26 06:01:14 shrek kernel: RBP: ffff8807ed2fdb08 R08: 0000000000000001 R09: 0000000000000000 Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000001 R12: ffff880621f44200 Jul 26 06:01:14 shrek kernel: R13: ffff8807ed2fdc30 R14: ffff880622797420 R15: ffff8801fd476300 Jul 26 06:01:14 shrek kernel: FS: 00000344c1e647a0(0000) GS:ffff88087fc20000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: 00000344c1e011dc CR3: 00000000015c6000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff8807ed2fdb38 ffffffff81053474 ffff880122b384b0 ffff880622796e40 Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff8807ed2fdc98 ffffffff8119dd02 Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] __lock_task_sighand+0x44/0x80 Jul 26 06:01:14 shrek kernel: [] do_task_stat+0x102/0xbb0 Jul 26 06:01:14 shrek kernel: [] proc_tgid_stat+0xf/0x20 Jul 26 06:01:14 shrek kernel: [] proc_single_show+0x5f/0xa0 Jul 26 06:01:14 shrek kernel: [] seq_read+0x192/0x670 Jul 26 06:01:14 shrek kernel: [] vfs_read+0xd5/0x230 Jul 26 06:01:14 shrek kernel: [] SyS_read+0x5a/0xa0 Jul 26 06:01:14 shrek kernel: [] system_call_fastpath+0x16/0x1b Jul 26 06:01:14 shrek kernel: [] ? sysret_check+0x26/0x65 Jul 26 06:01:14 shrek kernel: [] ? sysret_check+0x26/0x65 Jul 26 06:01:14 shrek kernel: Code: 38 c2 75 f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 <0f> b6 07 38 c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55 Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 8 Jul 26 06:01:14 shrek kernel: CPU: 8 PID: 29979 Comm: sg-queue Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff880622796e40 ti: ffff880622797398 task.ti: ffff880622797398 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff815ab0bd Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff88067e6a1e68 EFLAGS: 00000097 Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: ffff880622796e40 Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: ffff88067e6a1eb8 RDI: ffff880621f44a08 Jul 26 06:01:14 shrek kernel: RBP: ffff88067e6a1e68 R08: ffff88067e6a1eb8 R09: 000000000000751a Jul 26 06:01:14 shrek kernel: R10: 0000000000000008 R11: 0000000000000246 R12: ffff88067e6a1e98 Jul 26 06:01:14 shrek kernel: R13: ffff88067e6a1eb8 R14: 0000000002d7c680 R15: 0000000002d7ccb0 Jul 26 06:01:14 shrek kernel: FS: 0000037e10229700(0000) GS:ffff88087fc40000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: 0000037e0f8978d0 CR3: 00000000015c8000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff88067e6a1e88 ffffffff81053620 0000000000000000 0000000000000002 Jul 26 06:01:14 shrek kernel: ffff88067e6a1ea8 ffffffff8105369a 0000000000000000 0000000002d7ccb0 Jul 26 06:01:14 shrek kernel: ffff88067e6a1ef8 ffffffff81056fcb 0000000000000000 ffffffffffff4111 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] __set_current_blocked+0x30/0x60 Jul 26 06:01:14 shrek kernel: [] sigprocmask+0x4a/0x80 Jul 26 06:01:14 shrek kernel: [] SyS_rt_sigprocmask+0xbb/0xd0 Jul 26 06:01:14 shrek kernel: [] system_call_fastpath+0x16/0x1b Jul 26 06:01:14 shrek kernel: [] ? retint_swapgs+0xe/0x11 Jul 26 06:01:14 shrek kernel: Code: 48 83 c4 08 b8 01 00 00 00 5b c9 c3 66 90 55 48 89 e5 fa b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 <0f> b6 07 38 c2 75 f7 c9 c3 66 2e 0f 1f 84 00 00 00 00 00 55 48 Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 9 Jul 26 06:01:14 shrek kernel: CPU: 9 PID: 30215 Comm: pkill Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff88072c811260 ti: ffff88072c8117b8 task.ti: ffff88072c8117b8 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff815ab15c Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff8807ed29fbc8 EFLAGS: 00000093 Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000004 Jul 26 06:01:14 shrek kernel: RDX: 0000000000000007 RSI: ffff8807ed29fd00 RDI: ffff880621f44a08 Jul 26 06:01:14 shrek kernel: RBP: ffff8807ed29fbc8 R08: 0000000000000001 R09: 000000000000fffd Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000006 R12: ffff880621f44200 Jul 26 06:01:14 shrek kernel: R13: ffff8807ed29fd00 R14: ffff880622797420 R15: 0000000000000000 Jul 26 06:01:14 shrek kernel: FS: 000002e88ef8a700(0000) GS:ffff88087fc60000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: 000000000459c008 CR3: 00000000015ca000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff8807ed29fbf8 ffffffff81053474 ffff88010a1a6640 ffff880622796e40 Jul 26 06:01:14 shrek kernel: ffff880488a07500 ffffffff81ab4400 ffff8807ed29fd58 ffffffff8119ecca Jul 26 06:01:14 shrek kernel: ffffffff0000751a ffffffff00000000 ffffffff00000000 ffffffff00000000 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] __lock_task_sighand+0x44/0x80 Jul 26 06:01:14 shrek kernel: [] proc_pid_status+0x3ea/0x7c0 Jul 26 06:01:14 shrek kernel: [] proc_single_show+0x5f/0xa0 Jul 26 06:01:14 shrek kernel: [] seq_read+0x192/0x670 Jul 26 06:01:14 shrek kernel: [] vfs_read+0xd5/0x230 Jul 26 06:01:14 shrek kernel: [] SyS_read+0x5a/0xa0 Jul 26 06:01:14 shrek kernel: [] system_call_fastpath+0x16/0x1b Jul 26 06:01:14 shrek kernel: [] ? sysret_check+0x26/0x65 Jul 26 06:01:14 shrek kernel: Code: 38 c2 75 f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 <0f> b6 07 38 c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55 Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 10 Jul 26 06:01:14 shrek kernel: CPU: 10 PID: 30390 Comm: pidof Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff8804b4c60930 ti: ffff8804b4c60e88 task.ti: ffff8804b4c60e88 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff815ab15f Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff880594293b08 EFLAGS: 00000097 Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000 Jul 26 06:01:14 shrek kernel: RDX: 0000000000000009 RSI: ffff880594293c30 RDI: ffff880621f44a08 Jul 26 06:01:14 shrek kernel: RBP: ffff880594293b08 R08: 0000000000000001 R09: 0000000000000000 Jul 26 06:01:14 shrek kernel: R10: 0000000000000022 R11: 0000000000000001 R12: ffff880621f44200 Jul 26 06:01:14 shrek kernel: R13: ffff880594293c30 R14: ffff880622797420 R15: ffff8801fd476300 Jul 26 06:01:14 shrek kernel: FS: 0000031905858700(0000) GS:ffff88087fc80000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: 00000000009fc3c8 CR3: 00000000015cc000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff880594293b38 ffffffff81053474 ffff880101077450 ffff880622796e40 Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff880594293c98 ffffffff8119dd02 Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] __lock_task_sighand+0x44/0x80 Jul 26 06:01:14 shrek kernel: [] do_task_stat+0x102/0xbb0 Jul 26 06:01:14 shrek kernel: [] proc_tgid_stat+0xf/0x20 Jul 26 06:01:14 shrek kernel: [] proc_single_show+0x5f/0xa0 Jul 26 06:01:14 shrek kernel: [] seq_read+0x192/0x670 Jul 26 06:01:14 shrek kernel: [] vfs_read+0xd5/0x230 Jul 26 06:01:14 shrek kernel: [] SyS_read+0x5a/0xa0 Jul 26 06:01:14 shrek kernel: [] system_call_fastpath+0x16/0x1b Jul 26 06:01:14 shrek kernel: [] ? sysret_check+0x26/0x65 Jul 26 06:01:14 shrek kernel: Code: f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 0f b6 07 <38> c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55 48 89 e5 Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 11 Jul 26 06:01:14 shrek kernel: CPU: 11 PID: 30213 Comm: pkill Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff8804a95b1260 ti: ffff8804a95b17b8 task.ti: ffff8804a95b17b8 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff815ab15f Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff880696bf3ba8 EFLAGS: 00000097 Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000004 Jul 26 06:01:14 shrek kernel: RDX: 0000000000000006 RSI: ffff880696bf3ce0 RDI: ffff880621f44a08 Jul 26 06:01:14 shrek kernel: RBP: ffff880696bf3ba8 R08: 0000000000000001 R09: 000000000000fffd Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000006 R12: ffff880621f44200 Jul 26 06:01:14 shrek kernel: R13: ffff880696bf3ce0 R14: ffff880622797420 R15: 0000000000000000 Jul 26 06:01:14 shrek kernel: FS: 000002bb0925b700(0000) GS:ffff88087fca0000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: 0000000000be3018 CR3: 00000000015ce000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff880696bf3bd8 ffffffff81053474 ffff88007993a3e8 ffff880622796e40 Jul 26 06:01:14 shrek kernel: ffff880488a07500 ffffffff81ab4400 ffff880696bf3d38 ffffffff8119ecca Jul 26 06:01:14 shrek kernel: ffffffff0000751a ffffffff00000000 ffffffff00000000 ffffffff00000000 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] __lock_task_sighand+0x44/0x80 Jul 26 06:01:14 shrek kernel: [] proc_pid_status+0x3ea/0x7c0 Jul 26 06:01:14 shrek kernel: [] proc_single_show+0x5f/0xa0 Jul 26 06:01:14 shrek kernel: [] seq_read+0x192/0x670 Jul 26 06:01:14 shrek kernel: [] vfs_read+0xd5/0x230 Jul 26 06:01:14 shrek kernel: [] SyS_read+0x5a/0xa0 Jul 26 06:01:14 shrek kernel: [] system_call_fastpath+0x16/0x1b Jul 26 06:01:14 shrek kernel: [] ? sysret_check+0x26/0x65 Jul 26 06:01:14 shrek kernel: [] ? sysret_check+0x26/0x65 Jul 26 06:01:14 shrek kernel: Code: f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 0f b6 07 <38> c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55 48 89 e5 Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 12 Jul 26 06:01:14 shrek kernel: CPU: 12 PID: 0 Comm: swapper/12 Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff880469d752b0 ti: ffff880469d75808 task.ti: ffff880469d75808 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff812ab2c4 Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff88046fcc3d38 EFLAGS: 00000046 Jul 26 06:01:14 shrek kernel: RAX: 00000000a7bd4e81 RBX: 0000000000000000 RCX: 00000000a7bd4e5d Jul 26 06:01:14 shrek kernel: RDX: 000000000000000c RSI: 000000000000000c RDI: 0000000000033452 Jul 26 06:01:14 shrek kernel: RBP: ffff88046fcc3d38 R08: ffffffff81afa5c8 R09: 0000000000000000 Jul 26 06:01:14 shrek kernel: R10: 0000000000001848 R11: 000000002b300043 R12: 0000000000001000 Jul 26 06:01:14 shrek kernel: R13: ffffffff81afa5c8 R14: 0000000000000400 R15: 0000000000000096 Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fcc0000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: ffffffffff600400 CR3: 00000000015d0000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff88046fcc3d48 ffffffff812ab289 ffff88046fcc3d68 ffffffff8102ca62 Jul 26 06:01:14 shrek kernel: 0000000000000002 000000000000a024 ffff88046fcc3db8 ffffffff8102d428 Jul 26 06:01:14 shrek kernel: ffffffff00000003 000000000000000d ffff88046fcc3d98 0000000000000000 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: Jul 26 06:01:14 shrek kernel: [] __const_udelay+0x29/0x30 Jul 26 06:01:14 shrek kernel: [] native_safe_apic_wait_icr_idle+0x22/0x60 Jul 26 06:01:14 shrek kernel: [] default_send_IPI_mask_sequence_phys+0xb8/0xd0 Jul 26 06:01:14 shrek kernel: [] physflat_send_IPI_all+0x12/0x20 Jul 26 06:01:14 shrek kernel: [] arch_trigger_all_cpu_backtrace+0x47/0x80 Jul 26 06:01:14 shrek kernel: [] rcu_check_callbacks+0x617/0x640 Jul 26 06:01:14 shrek kernel: [] update_process_times+0x43/0x80 Jul 26 06:01:14 shrek kernel: [] tick_sched_timer+0x51/0x130 Jul 26 06:01:14 shrek kernel: [] __run_hrtimer+0x5f/0x130 Jul 26 06:01:14 shrek kernel: [] hrtimer_interrupt+0x106/0x250 Jul 26 06:01:14 shrek kernel: [] local_apic_timer_interrupt+0x36/0x60 Jul 26 06:01:14 shrek kernel: [] smp_apic_timer_interrupt+0x3e/0x60 Jul 26 06:01:14 shrek kernel: [] apic_timer_interrupt+0x83/0x90 Jul 26 06:01:14 shrek kernel: Jul 26 06:01:14 shrek kernel: [] ? default_idle+0x6/0x10 Jul 26 06:01:14 shrek kernel: [] arch_cpu_idle+0x16/0x20 Jul 26 06:01:14 shrek kernel: [] cpu_startup_entry+0xc9/0x1b0 Jul 26 06:01:14 shrek kernel: [] start_secondary+0x238/0x350 Jul 26 06:01:14 shrek kernel: Code: a0 00 00 0f 1f 00 0f ae e8 0f 31 89 c1 eb 16 0f 1f 84 00 00 00 00 00 f3 90 65 8b 14 25 20 a0 00 00 39 d6 75 12 0f 1f 00 0f ae e8 <0f> 31 89 c2 29 ca 39 fa 72 e2 c9 c3 29 c1 89 d6 01 cf 0f 1f 00 Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 13 Jul 26 06:01:14 shrek kernel: CPU: 13 PID: 0 Comm: swapper/13 Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff880469d75be0 ti: ffff880469d76138 task.ti: ffff880469d76138 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff8100c456 Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d8be88 EFLAGS: 00000286 Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d76140 RCX: 0000000000000000 Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fcec200 Jul 26 06:01:14 shrek kernel: RBP: ffff880469d8be88 R08: 0000000000000000 R09: 0000000000000000 Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d76140 Jul 26 06:01:14 shrek kernel: R13: ffff880469d76140 R14: ffffffff81afa5c8 R15: ffff880469d76140 Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fce0000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: ffffffffff600400 CR3: 00000000015d2000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff880469d8be98 ffffffff8100c6b6 ffff880469d8bf08 ffffffff810897b9 Jul 26 06:01:14 shrek kernel: 000000000000000d ffff880469d76138 ffff880469d76140 ffff880469d76138 Jul 26 06:01:14 shrek kernel: 0043fd9fa00c6708 5e3321e5c10994cc ffff880469d8bef8 000000000000000d Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] arch_cpu_idle+0x16/0x20 Jul 26 06:01:14 shrek kernel: [] cpu_startup_entry+0xc9/0x1b0 Jul 26 06:01:14 shrek kernel: [] start_secondary+0x238/0x350 Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65 Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 14 Jul 26 06:01:14 shrek kernel: CPU: 14 PID: 0 Comm: swapper/14 Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff880469d76510 ti: ffff880469d76a68 task.ti: ffff880469d76a68 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff8100c456 Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d8de88 EFLAGS: 00000286 Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d76a70 RCX: 0000000000000000 Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fd0c200 Jul 26 06:01:14 shrek kernel: RBP: ffff880469d8de88 R08: 0000000000000000 R09: 0000000000000000 Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d76a70 Jul 26 06:01:14 shrek kernel: R13: ffff880469d76a70 R14: ffffffff81afa5c8 R15: ffff880469d76a70 Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fd00000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: 0000000008b70820 CR3: 00000000015d4000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff880469d8de98 ffffffff8100c6b6 ffff880469d8df08 ffffffff810897b9 Jul 26 06:01:14 shrek kernel: 000000000000000e ffff880469d76a68 ffff880469d76a70 ffff880469d76a68 Jul 26 06:01:14 shrek kernel: 0043fd9fa1d85012 617db263e885420d ffff880469d8def8 000000000000000e Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] arch_cpu_idle+0x16/0x20 Jul 26 06:01:14 shrek kernel: [] cpu_startup_entry+0xc9/0x1b0 Jul 26 06:01:14 shrek kernel: [] start_secondary+0x238/0x350 Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65 Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 15 Jul 26 06:01:14 shrek kernel: CPU: 15 PID: 0 Comm: swapper/15 Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff880469d76e40 ti: ffff880469d77398 task.ti: ffff880469d77398 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff8100c456 Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d8fe88 EFLAGS: 00000286 Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d773a0 RCX: 0000000000000000 Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fd2c200 Jul 26 06:01:14 shrek kernel: RBP: ffff880469d8fe88 R08: 0000000000000000 R09: 0000000000000000 Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d773a0 Jul 26 06:01:14 shrek kernel: R13: ffff880469d773a0 R14: ffffffff81afa5c8 R15: ffff880469d773a0 Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fd20000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: ffffffffff600400 CR3: 00000000015d6000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff880469d8fe98 ffffffff8100c6b6 ffff880469d8ff08 ffffffff810897b9 Jul 26 06:01:14 shrek kernel: 000000000000000f ffff880469d77398 ffff880469d773a0 ffff880469d77398 Jul 26 06:01:14 shrek kernel: 0043fd9fa3a15351 a19db38607e2294d ffff880469d8fef8 000000000000000f Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] arch_cpu_idle+0x16/0x20 Jul 26 06:01:14 shrek kernel: [] cpu_startup_entry+0xc9/0x1b0 Jul 26 06:01:14 shrek kernel: [] start_secondary+0x238/0x350 Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65 Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 16 Jul 26 06:01:14 shrek kernel: CPU: 16 PID: 0 Comm: swapper/16 Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff880469d90000 ti: ffff880469d90558 task.ti: ffff880469d90558 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff8100c456 Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d99e88 EFLAGS: 00000286 Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d90560 RCX: 0000000000000000 Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fd4c200 Jul 26 06:01:14 shrek kernel: RBP: ffff880469d99e88 R08: 0000000000000000 R09: 0000000000000000 Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d90560 Jul 26 06:01:14 shrek kernel: R13: ffff880469d90560 R14: ffffffff81afa5c8 R15: ffff880469d90560 Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fd40000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: 0000000001758000 CR3: 00000000015d8000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff880469d99e98 ffffffff8100c6b6 ffff880469d99f08 ffffffff810897b9 Jul 26 06:01:14 shrek kernel: 0000000000000010 ffff880469d90558 ffff880469d90560 ffff880469d90558 Jul 26 06:01:14 shrek kernel: 0043fd9fa56a1bdf 2e70ed886688fb1c ffff880469d99ef8 0000000000000010 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] arch_cpu_idle+0x16/0x20 Jul 26 06:01:14 shrek kernel: [] cpu_startup_entry+0xc9/0x1b0 Jul 26 06:01:14 shrek kernel: [] start_secondary+0x238/0x350 Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65 Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 17 Jul 26 06:01:14 shrek kernel: CPU: 17 PID: 0 Comm: swapper/17 Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff880469d90930 ti: ffff880469d90e88 task.ti: ffff880469d90e88 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff8100c456 Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d9be88 EFLAGS: 00000286 Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d90e90 RCX: 0000000000000000 Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fd6c200 Jul 26 06:01:14 shrek kernel: RBP: ffff880469d9be88 R08: 0000000000000000 R09: 0000000000000000 Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d90e90 Jul 26 06:01:14 shrek kernel: R13: ffff880469d90e90 R14: ffffffff81afa5c8 R15: ffff880469d90e90 Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fd60000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: 0000034fa60bf398 CR3: 00000000015da000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff880469d9be98 ffffffff8100c6b6 ffff880469d9bf08 ffffffff810897b9 Jul 26 06:01:14 shrek kernel: 0000000000000011 ffff880469d90e88 ffff880469d90e90 ffff880469d90e88 Jul 26 06:01:14 shrek kernel: ffff880469d9beb2 05735dbd652beb34 ffff880469d9bef8 0000000000000011 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] arch_cpu_idle+0x16/0x20 Jul 26 06:01:14 shrek kernel: [] cpu_startup_entry+0xc9/0x1b0 Jul 26 06:01:14 shrek kernel: [] start_secondary+0x238/0x350 Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65 Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 18 Jul 26 06:01:14 shrek kernel: CPU: 18 PID: 29972 Comm: ps Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff880622792df0 ti: ffff880622793348 task.ti: ffff880622793348 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff815ab15c Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff8806db9e3b38 EFLAGS: 00000093 Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000 Jul 26 06:01:14 shrek kernel: RDX: 0000000000000001 RSI: ffff8806db9e3c60 RDI: ffff880621f44a08 Jul 26 06:01:14 shrek kernel: RBP: ffff8806db9e3b38 R08: 0000000000000001 R09: 0000000000000000 Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000001 R12: ffff880621f44200 Jul 26 06:01:14 shrek kernel: R13: ffff8806db9e3c60 R14: ffff880622797420 R15: ffff8801fd476300 Jul 26 06:01:14 shrek kernel: FS: 000002969e6097a0(0000) GS:ffff88087fcc0000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: 000002969e5e01dc CR3: 00000000015dc000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff8806db9e3b68 ffffffff81053474 ffff880129ded2c0 ffff880622796e40 Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff8806db9e3cc8 ffffffff8119dd02 Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] __lock_task_sighand+0x44/0x80 Jul 26 06:01:14 shrek kernel: [] do_task_stat+0x102/0xbb0 Jul 26 06:01:14 shrek kernel: [] proc_tgid_stat+0xf/0x20 Jul 26 06:01:14 shrek kernel: [] proc_single_show+0x5f/0xa0 Jul 26 06:01:14 shrek kernel: [] seq_read+0x192/0x670 Jul 26 06:01:14 shrek kernel: [] vfs_read+0xd5/0x230 Jul 26 06:01:14 shrek kernel: [] SyS_read+0x5a/0xa0 Jul 26 06:01:14 shrek kernel: [] system_call_fastpath+0x16/0x1b Jul 26 06:01:14 shrek kernel: [] ? sysret_check+0x26/0x65 Jul 26 06:01:14 shrek kernel: Code: 38 c2 75 f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 <0f> b6 07 38 c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55 Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 19 Jul 26 06:01:14 shrek kernel: CPU: 19 PID: 0 Comm: swapper/19 Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff880469d91b90 ti: ffff880469d920e8 task.ti: ffff880469d920e8 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff8100c456 Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d9fe88 EFLAGS: 00000286 Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d920f0 RCX: 0000000000000000 Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88087fcec200 Jul 26 06:01:14 shrek kernel: RBP: ffff880469d9fe88 R08: 0000000000000000 R09: 0000000000000000 Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d920f0 Jul 26 06:01:14 shrek kernel: R13: ffff880469d920f0 R14: ffffffff81afa5c8 R15: ffff880469d920f0 Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88087fce0000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: 0000000001816cf8 CR3: 00000000015de000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff880469d9fe98 ffffffff8100c6b6 ffff880469d9ff08 ffffffff810897b9 Jul 26 06:01:14 shrek kernel: 0000000000000013 ffff880469d920e8 ffff880469d920f0 ffff880469d920e8 Jul 26 06:01:14 shrek kernel: 0043fd9faac3c713 e2b091c5f11a3308 ffff880469d9fef8 0000000000000013 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] arch_cpu_idle+0x16/0x20 Jul 26 06:01:14 shrek kernel: [] cpu_startup_entry+0xc9/0x1b0 Jul 26 06:01:14 shrek kernel: [] start_secondary+0x238/0x350 Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65 Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 20 Jul 26 06:01:14 shrek kernel: CPU: 20 PID: 0 Comm: swapper/20 Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff880469d924c0 ti: ffff880469d92a18 task.ti: ffff880469d92a18 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff8100c456 Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469da1e88 EFLAGS: 00000286 Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d92a20 RCX: 0000000000000000 Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88087fd0c200 Jul 26 06:01:14 shrek kernel: RBP: ffff880469da1e88 R08: 0000000000000000 R09: 0000000000000000 Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d92a20 Jul 26 06:01:14 shrek kernel: R13: ffff880469d92a20 R14: ffffffff81afa5c8 R15: ffff880469d92a20 Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88087fd00000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: ffffffffff600400 CR3: 00000000015e0000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff880469da1e98 ffffffff8100c6b6 ffff880469da1f08 ffffffff810897b9 Jul 26 06:01:14 shrek kernel: 0000000000000014 ffff880469d92a18 ffff880469d92a20 ffff880469d92a18 Jul 26 06:01:14 shrek kernel: 0043fd9fac8b8e6d 0fa0aa77f886bbaa ffff880469da1ef8 0000000000000014 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] arch_cpu_idle+0x16/0x20 Jul 26 06:01:14 shrek kernel: [] cpu_startup_entry+0xc9/0x1b0 Jul 26 06:01:14 shrek kernel: [] start_secondary+0x238/0x350 Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65 Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 21 Jul 26 06:01:14 shrek kernel: CPU: 21 PID: 30214 Comm: pkill Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff880867a2ee40 ti: ffff880867a2f398 task.ti: ffff880867a2f398 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff815ab15c Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff8804b4dc9b68 EFLAGS: 00000097 Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000004 Jul 26 06:01:14 shrek kernel: RDX: 0000000000000005 RSI: ffff8804b4dc9ca0 RDI: ffff880621f44a08 Jul 26 06:01:14 shrek kernel: RBP: ffff8804b4dc9b68 R08: 0000000000000001 R09: 000000000000fffd Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000006 R12: ffff880621f44200 Jul 26 06:01:14 shrek kernel: R13: ffff8804b4dc9ca0 R14: ffff880622797420 R15: 0000000000000000 Jul 26 06:01:14 shrek kernel: FS: 0000032a2ff54700(0000) GS:ffff88087fd20000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: 0000000001319008 CR3: 00000000015e2000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff8804b4dc9b98 ffffffff81053474 ffff8800794e9c20 ffff880622796e40 Jul 26 06:01:14 shrek kernel: ffff880488a07500 ffffffff81ab4400 ffff8804b4dc9cf8 ffffffff8119ecca Jul 26 06:01:14 shrek kernel: ffffffff0000751a ffffffff00000000 ffffffff00000000 ffffffff00000000 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] __lock_task_sighand+0x44/0x80 Jul 26 06:01:14 shrek kernel: [] proc_pid_status+0x3ea/0x7c0 Jul 26 06:01:14 shrek kernel: [] proc_single_show+0x5f/0xa0 Jul 26 06:01:14 shrek kernel: [] seq_read+0x192/0x670 Jul 26 06:01:14 shrek kernel: [] vfs_read+0xd5/0x230 Jul 26 06:01:14 shrek kernel: [] SyS_read+0x5a/0xa0 Jul 26 06:01:14 shrek kernel: [] system_call_fastpath+0x16/0x1b Jul 26 06:01:14 shrek kernel: [] ? sysret_check+0x26/0x65 Jul 26 06:01:14 shrek kernel: Code: 38 c2 75 f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 <0f> b6 07 38 c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55 Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 22 Jul 26 06:01:14 shrek kernel: CPU: 22 PID: 30017 Comm: pidof Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff880490800930 ti: ffff880490800e88 task.ti: ffff880490800e88 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff815ab15f Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff8807f1d03b08 EFLAGS: 00000093 Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000 Jul 26 06:01:14 shrek kernel: RDX: 0000000000000003 RSI: ffff8807f1d03c30 RDI: ffff880621f44a08 Jul 26 06:01:14 shrek kernel: RBP: ffff8807f1d03b08 R08: 0000000000000001 R09: 0000000000000000 Jul 26 06:01:14 shrek kernel: R10: 0000000000000022 R11: 0000000000000001 R12: ffff880621f44200 Jul 26 06:01:14 shrek kernel: R13: ffff8807f1d03c30 R14: ffff880622797420 R15: ffff8801fd476300 Jul 26 06:01:14 shrek kernel: FS: 0000025ccaa4c700(0000) GS:ffff88087fd40000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: 0000000003f2a4f8 CR3: 00000000015e4000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff8807f1d03b38 ffffffff81053474 ffff88013bb36190 ffff880622796e40 Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff8807f1d03c98 ffffffff8119dd02 Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] __lock_task_sighand+0x44/0x80 Jul 26 06:01:14 shrek kernel: [] do_task_stat+0x102/0xbb0 Jul 26 06:01:14 shrek kernel: [] proc_tgid_stat+0xf/0x20 Jul 26 06:01:14 shrek kernel: [] proc_single_show+0x5f/0xa0 Jul 26 06:01:14 shrek kernel: [] seq_read+0x192/0x670 Jul 26 06:01:14 shrek kernel: [] vfs_read+0xd5/0x230 Jul 26 06:01:14 shrek kernel: [] SyS_read+0x5a/0xa0 Jul 26 06:01:14 shrek kernel: [] system_call_fastpath+0x16/0x1b Jul 26 06:01:14 shrek kernel: [] ? sysret_check+0x26/0x65 Jul 26 06:01:14 shrek kernel: Code: f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 0f b6 07 <38> c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55 48 89 e5 Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 23 Jul 26 06:01:14 shrek kernel: CPU: 23 PID: 0 Comm: swapper/23 Tainted: W O 3.13.3-sg1 #6 Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013 Jul 26 06:01:14 shrek kernel: task: ffff880469d94050 ti: ffff880469d945a8 task.ti: ffff880469d945a8 Jul 26 06:01:14 shrek kernel: RIP: 0010:[] [] ffffffff8100c456 Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469da7e88 EFLAGS: 00000286 Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d945b0 RCX: 0000000000000000 Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88087fd6c200 Jul 26 06:01:14 shrek kernel: RBP: ffff880469da7e88 R08: 0000000000000000 R09: 0000000000000000 Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d945b0 Jul 26 06:01:14 shrek kernel: R13: ffff880469d945b0 R14: ffffffff81afa5c8 R15: ffff880469d945b0 Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88087fd60000(0000) knlGS:0000000000000000 Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jul 26 06:01:14 shrek kernel: CR2: ffffffffff600400 CR3: 00000000015e6000 CR4: 00000000001607f0 Jul 26 06:01:14 shrek kernel: Stack: Jul 26 06:01:14 shrek kernel: ffff880469da7e98 ffffffff8100c6b6 ffff880469da7f08 ffffffff810897b9 Jul 26 06:01:14 shrek kernel: 0000000000000017 ffff880469d945a8 ffff880469d945b0 ffff880469d945a8 Jul 26 06:01:14 shrek kernel: 0043fd9fb1e28232 a8f790adc213c71a ffff880469da7ef8 0000000000000017 Jul 26 06:01:14 shrek kernel: Call Trace: Jul 26 06:01:14 shrek kernel: [] arch_cpu_idle+0x16/0x20 Jul 26 06:01:14 shrek kernel: [] cpu_startup_entry+0xc9/0x1b0 Jul 26 06:01:14 shrek kernel: [] start_secondary+0x238/0x350 Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65 My investigation showed up that the CPUs are in the following state: CPUS 1, 2, 3, 6, 7, 8, 9, 10, 11, 18, 21, 22 are locked on the spinlock inside __lock_task_sighand, in a rcu_read_lock() section. Whereas CPUS: 4, 5, 13, 14, 15, 16, 17, 19, 20, 23 are staying idle. >From the logs I gather that 2 different subsystems detected that: the NMI watchdog and the rcu stall detector (cpu's 8 and 18). While looking at the code and googling around I discovered the following commit from PeterZ: https://lkml.org/lkml/2008/11/13/157 Based on that I think could be happening is that the sighand itself is being freed while we are in the grace period inside __lock_task_sighand but the slab page itself is not freed as per the semantics of SLAB_DESTROY_BY_RCU. I looked up the source of this function in the latest kernels and saw that Oleg had put a comment clarifying the semantics but I'm still not convinced that it is safe. What if we are trying to lock the spinlock before this particular slab is initialised with sighand_ctor? Looking at the code and reading Peter's explanation of SLAB_RCU I cannot identify which portion of __lock_task_sighand implements the following logic: if (!try_get_ref(obj)) // might fail for free objects goto again; Also what's more peculiar was the fact that when "Shrek" experienced this issue, resources were migrated to the 2nd machine and it also experienced the same behavior - CPUs locked on the sighand spinlock. That's the first time I've seen this particular issue. And I'm not really able to come up with a way where a task's sighand could be corrupted in a such a way as to prevent acquiring the lock. Regards, Nikolay -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/