Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S964997Ab2HWReR (ORCPT ); Thu, 23 Aug 2012 13:34:17 -0400 Received: from mail-wi0-f170.google.com ([209.85.212.170]:43535 "EHLO mail-wi0-f170.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S964886Ab2HWReM (ORCPT ); Thu, 23 Aug 2012 13:34:12 -0400 MIME-Version: 1.0 Date: Thu, 23 Aug 2012 13:34:09 -0400 Message-ID: Subject: Possible mptsas regression post 3.5.0 From: John Drescher To: LKML Content-Type: text/plain; charset=ISO-8859-1 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 134423 Lines: 2284 Over the last few weeks I have done some reliability testing with mdraid6 on a machine with 2 lsi mptsas controllers and 13 SATA I drives. My testing involved physically hot removing a drive forcing the raid to grab a spare and rebuild. This worked great for the 5 or so times I did this on gentoo-sources-3.5.0 and lower. However any attempt to do this on gentoo-sources-3.5.1 or even the 3.6-rc2 git resulted in a total lockup of the array. I originally thought this was a mdadm regression and posted about that last week here: https://lkml.org/lkml/2012/8/17/503 I have since enabled additional kernel debugging options and get the following output: [ 1629.132138] md: data-check of RAID array md0 [ 1629.132152] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. [ 1629.132165] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. [ 1629.132192] md: using 128k window, over a total of 1048512k. [ 1649.849873] md: md0: data-check done. [ 2170.325379] mptbase: ioc1: LogInfo(0x31110d00): Originator={PL}, Code={Reset} , SubCode(0x0d00) cb_idx mptbase_reply [ 2174.695292] sd 1:0:3:0: [sdk] Synchronizing SCSI cache [ 2174.695335] sd 1:0:3:0: [sdk] [ 2174.695342] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK [ 2185.534172] md: super_written gets error=-19, uptodate=0 [ 2185.534191] md/raid:md1: Disk failure on sdk2, disabling device. [ 2185.534191] md/raid:md1: Operation continuing on 11 devices. [ 2199.154901] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u:3:6089] [ 2199.154916] Modules linked in: netconsole configfs w83627ehf hwmon_vid ipmi_s i ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_intel kvm e1000e lpc_i ch mfd_core i2c_i801 pcspkr microcode i2c_core video button xts gf128mul aes_x86 _64 aes_generic cbc sha256_generic dm_snapshot dm_crypt dm_mirror dm_region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_hcd usb_storage ehci_hcd megaraid _sas megaraid_mbox megaraid_mm megaraid sr_mod cdrom sd_mod sata_mv ata_piix ahc i libahci pata_marvell pata_mpiix libata [ 2199.155324] CPU 0 [ 2199.155330] Modules linked in:[ 2199.156397] netconsole configfs w83627ehf h wmon_vid ipmi_si ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_intel k vm e1000e lpc_ich mfd_core i2c_i801 pcspkr microcode i2c_core video button xts g f128mul aes_x86_64 aes_generic cbc sha256_generic e1000 nfs lockd fscache auth_r pcgss nfs_acl sunrpc reiserfs multipath linear raid0 dm_raid dm_snapshot dm_cryp t dm_mirror dm_region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_hcd usb _storage ehci_hcd megaraid_sas megaraid_mbox megaraid_mm megaraid sr_mod cdrom s d_mod sata_mv ata_piix ahci libahci pata_marvell pata_mpiix libata [ 2199.157086] Pid: 6089, comm: kworker/u:3 Not tainted 3.5.1-gentoo-dbg-lock-1 #2 To be filled by O.E.M. To be filled by O.E.M./P8B-X series [ 2199.157221] RIP: 0010:[] [] _raw_spin_un lock_irqrestore+0x32/0x40 [ 2199.157344] RSP: 0018:ffff880221dadaa0 EFLAGS: 00000282 [ 2199.157405] RAX: 0000000000000000 RBX: ffff880221dada50 RCX: 000000000000b828 [ 2199.157471] RDX: 0000000000002e40 RSI: ffff880226600000 RDI: ffff8802233f2090 [ 2199.157536] RBP: ffff880221dadab0 R08: 0000000000000001 R09: 0000000000000000 [ 2199.157602] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88021e1bc460 [ 2199.157670] R13: 0000000000000082 R14: ffff880221dada20 R15: ffff8802233f20a8 [ 2199.157737] FS: 0000000000000000(0000) GS:ffff880226600000(0000) knlGS:00000 00000000000 [ 2199.157854] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 2199.157917] CR2: 00007f42a5fb2070 CR3: 0000000001a0b000 CR4: 00000000000407f0 [ 2199.157983] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2199.158048] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 2199.158114] Process kworker/u:3 (pid: 6089, threadinfo ffff880221dac000, task ffff88021e1bc460) [ 2199.158230] Stack: [ 2199.158285] ffff88021eb50000 ffff8802233f1ff0 ffff880221dadb00 ffffffff81417 98a [ 2199.158408] 0000000000000282 ffff8802233f2000 0000000000000046 ffff8802221e6 800 [ 2199.158533] 0000000000000008 ffff880222156630 ffff88021eb0b000 1221000005000 000 [ 2199.158662] Call Trace: [ 2199.158725] [] scsi_remove_target+0xda/0x1f0 [ 2199.158793] [] sas_rphy_remove+0x55/0x60 [ 2199.158859] [] sas_rphy_delete+0x11/0x20 [ 2199.158925] [] sas_port_delete+0x25/0x160 [ 2199.158989] [] mptsas_del_end_device+0x183/0x270 [ 2199.159054] [] mptsas_hotplug_work+0x1ec/0x920 [ 2199.159120] [] ? mptsas_free_fw_event+0x6b/0xb0 [ 2199.159187] [] ? sched_clock_cpu+0xc5/0x120 [ 2199.159251] [] mptsas_firmware_event_work+0xbc0/0xfa0 [ 2199.159319] [] ? __lock_acquire.isra.27+0x29f/0xb30 [ 2199.159388] [] ? mptsas_expander_add+0x140/0x140 [ 2199.159456] [] ? mptsas_expander_add+0x140/0x140 [ 2199.159524] [] process_one_work+0x184/0x460 [ 2199.159591] [] ? process_one_work+0x126/0x460 [ 2199.159658] [] worker_thread+0x15e/0x350 [ 2199.159724] [] ? manage_workers.isra.32+0x220/0x220 [ 2199.159792] [] kthread+0x9d/0xb0 [ 2199.159854] [] kernel_thread_helper+0x4/0x10 [ 2199.159917] [] ? __init_kthread_worker+0x70/0x70 [ 2199.159981] [] ? gs_change+0xb/0xb [ 2199.160040] Code: 10 48 8b 55 08 48 89 5d f0 48 89 f3 4c 89 65 f8 be 01 00 00 00 49 89 fc 48 8d 7f 18 e8 58 a6 ab ff 4c 89 e7 e8 d0 8f d8 ff 53 9d <48> 8b 5d f0 4c 8b 65 f8 c9 c3 0f 1f 40 00 55 be 01 00 00 00 48 [ 2227.091437] BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u:3:6089] [ 2227.091506] Modules linked in: netconsole configfs w83627ehf hwmon_vid ipmi_s i ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_intel kvm e1000e lpc_i ch mfd_core i2c_i801 pcspkr microcode i2c_core video button xts gf128mul aes_x86 _64 aes_generic cbc sha256_generic e1000 nfs lockd fscache auth_rpcgss nfs_acl s unrpc reiserfs multipath linear raid0 dm_raid dm_snapshot dm_crypt dm_mirror dm_ region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_hcd usb_storage ehci_h cd megaraid_sas megaraid_mbox megaraid_mm megaraid sr_mod cdrom sd_mod sata_mv a ta_piix ahci libahci pata_marvell pata_mpiix libata [ 2227.092148] CPU 0 [ 2227.092153] Modules linked in:[ 2227.092209] netconsole configfs w83627ehf h wmon_vid ipmi_si ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_intel k vm e1000e lpc_ich mfd_core i2c_i801 pcspkr microcode i2c_core video button xts g f128mul aes_x86_64 aes_generic cbc sha256_generic e1000 nfs lockd fscache auth_r pcgss nfs_acl sunrpc reiserfs multipath linear raid0 dm_raid dm_snapshot dm_cryp t dm_mirror dm_region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_hcd usb _storage ehci_hcd megaraid_sas megaraid_mbox megaraid_mm megaraid sr_mod cdrom s d_mod sata_mv ata_piix ahci libahci pata_marvell pata_mpiix libata [ 2227.092869] Pid: 6089, comm: kworker/u:3 Not tainted 3.5.1-gentoo-dbg-lock-1 #2 To be filled by O.E.M. To be filled by O.E.M./P8B-X series [ 2227.092997] RIP: 0010:[] [] scsi_is_host _device+0xa/0x20 [ 2227.093112] RSP: 0018:ffff880221dada88 EFLAGS: 00000246 [ 2227.093173] RAX: 0000000000000000 RBX: ffff880221dada20 RCX: 000000000000b828 [ 2227.093238] RDX: 0000000000002e40 RSI: ffff880226600000 RDI: ffff8802233f22c8 [ 2227.093302] RBP: ffff880221dadab0 R08: 0000000000000001 R09: 0000000000000000 [ 2227.093367] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88021e1bcb48 [ 2227.093431] R13: 0000000000000082 R14: ffff88021e1bc460 R15: ffffffff81061ea5 [ 2227.093496] FS: 0000000000000000(0000) GS:ffff880226600000(0000) knlGS:00000 00000000000 [ 2227.093616] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 2227.093679] CR2: 00007f42a5fb2070 CR3: 0000000001a0b000 CR4: 00000000000407f0 [ 2227.093744] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2227.093808] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 2227.093874] Process kworker/u:3 (pid: 6089, threadinfo ffff880221dac000, task ffff88021e1bc460) [ 2227.093992] Stack: [ 2227.094048] ffffffff81415171 ffff880221dadab0 ffff88021eb50000 ffff8802233f1 ff0 [ 2227.094176] ffff8802233f2000 ffff880221dadb00 ffffffff81417a74 0000000000000 282 [ 2227.094304] ffff8802233f2000 0000000000000046 ffff8802221e6800 0000000000000 008 [ 2227.094434] Call Trace: [ 2227.094493] [] ? scsi_target_reap+0x31/0x100 [ 2227.094558] [] scsi_remove_target+0x1c4/0x1f0 [ 2227.094623] [] sas_rphy_remove+0x55/0x60 [ 2227.094687] [] sas_rphy_delete+0x11/0x20 [ 2227.094751] [] sas_port_delete+0x25/0x160 [ 2227.094815] [] mptsas_del_end_device+0x183/0x270 [ 2227.094881] [] mptsas_hotplug_work+0x1ec/0x920 [ 2227.094949] [] ? mptsas_free_fw_event+0x6b/0xb0 [ 2227.095017] [] ? sched_clock_cpu+0xc5/0x120 [ 2227.095083] [] mptsas_firmware_event_work+0xbc0/0xfa0 [ 2227.095151] [] ? __lock_acquire.isra.27+0x29f/0xb30 [ 2227.095220] [] ? mptsas_expander_add+0x140/0x140 [ 2227.095288] [] ? mptsas_expander_add+0x140/0x140 [ 2227.095353] [] process_one_work+0x184/0x460 [ 2227.095415] [] ? process_one_work+0x126/0x460 [ 2227.095478] [] worker_thread+0x15e/0x350 [ 2227.095541] [] ? manage_workers.isra.32+0x220/0x220 [ 2227.095608] [] kthread+0x9d/0xb0 [ 2227.095672] [] kernel_thread_helper+0x4/0x10 [ 2227.095739] [] ? __init_kthread_worker+0x70/0x70 [ 2227.095807] [] ? gs_change+0xb/0xb [ 2227.095870] Code: 90 0f b7 06 55 48 89 e5 5d 39 87 74 fc ff ff 0f 94 c0 0f b6 c0 c3 66 66 2e 0f 1f 84 00 00 00 00 00 31 c0 48 81 7f 58 20 d0 a4 81 <55> 48 89 e5 5d 0f 94 c0 c3 66 66 66 66 2e 0f 1f 84 00 00 00 00 [ 2234.564238] INFO: rcu_sched self-detected stall on CPU [ 2234.564302] 0: (18001 ticks this GP) idle=c6b/140000000000001/0 [ 2234.564364] (t=18001 jiffies) [ 2234.564422] sending NMI to all CPUs: [ 2234.564480] NMI backtrace for cpu 0 [ 2234.564537] CPU 0 [ 2234.564543] Modules linked in:[ 2234.564599] netconsole configfs w83627ehf h wmon_vid ipmi_si ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_intel k vm e1000e lpc_ich mfd_core i2c_i801 pcspkr microcode i2c_core video button xts g f128mul aes_x86_64 aes_generic cbc sha256_generic e1000 nfs lockd fscache auth_r pcgss linear raid0 dm_raid dm_snapshot dm_crypt dm_mirror dm_region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_hcd usb_storage ehci_hcd megaraid_sas meg araid_mbox megaraid_mm megaraid sr_mod cdrom sd_mod sata_mv ata_piix ahci libahc i pata_marvell pata_mpiix libata [ 2234.565384] Pid: 6089, comm: kworker/u:3 Not tainted 3.5.1-gentoo-dbg-lock-1 #2 To be filled by O.E.M. To be filled by O.E.M./P8B-X series [ 2234.565524] RIP: 0010:[] [] delay_tsc+0x 2f/0x60 [ 2234.565646] RSP: 0018:ffff880226603dd8 EFLAGS: 00000046 [ 2234.565709] RAX: 00000000d36df40c RBX: 0000000000002710 RCX: 0000000000000000 [ 2234.565778] RDX: 0000000000000020 RSI: 00000000d36df3ec RDI: 000000000030b803 [ 2234.565847] RBP: ffff880226603dd8 R08: 0000000000000000 R09: 0000000000000000 [ 2234.565916] R10: 00000000000005f6 R11: 0000000000000001 R12: ffffffff81a27d00 [ 2234.565985] R13: ffffffff81a27e00 R14: ffffffff8107efb0 R15: ffff88022660d978 [ 2234.566053] FS: 0000000000000000(0000) GS:ffff880226600000(0000) knlGS:00000 00000000000 [ 2234.566174] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 2234.566240] CR2: 00007f42a5fb2070 CR3: 0000000001a0b000 CR4: 00000000000407f0 [ 2234.566310] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2234.566379] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 2234.566447] Process kworker/u:3 (pid: 6089, threadinfo ffff880221dac000, task ffff88021e1bc460) [ 2234.566569] Stack: [ 2234.566627] ffff880226603de8 ffffffff8134a24a ffff880226603df8 ffffffff8134a 27e [ 2234.566754] ffff880226603e18 ffffffff8101fcaa 0000000000000001 ffff8802267cd dc0 [ 2234.566882] ffff880226603e68 ffffffff810b24f9 ffff880226603e78 ffffffff8105e 120 [ 2234.567006] Call Trace: [ 2234.567062] [ 2234.567068] [] __delay+0xa/0x10 [ 2234.567187] [] __const_udelay+0x2e/0x30 [ 2234.567255] [] arch_trigger_all_cpu_backtrace+0x6a/0x90 [ 2234.567413] [] __rcu_pending+0x199/0x4d0 [ 2234.567481] [] ? account_system_time+0x130/0x200 [ 2234.567550] [] ? account_system_time+0xaa/0x200 [ 2234.567618] [] ? tick_init_highres+0x20/0x20 [ 2234.567686] [] rcu_check_callbacks+0xb8/0xf0 [ 2234.567754] [] update_process_times+0x43/0x80 [ 2234.567822] [] tick_sched_timer+0x58/0x150 [ 2234.567890] [] __run_hrtimer.isra.32+0x5c/0x100 [ 2234.568972] [] hrtimer_interrupt+0xfb/0x250 [ 2234.569036] [] smp_apic_timer_interrupt+0x64/0xa0 [ 2234.569101] [] apic_timer_interrupt+0x67/0x70 [ 2234.569164] [ 2234.569170] [] ? _raw_spin_unlock_irqrestore+0x32/0x40 [ 2234.569295] [] scsi_target_reap+0x7d/0x100 [ 2234.569362] [] scsi_remove_target+0x1c4/0x1f0 [ 2234.569429] [] sas_rphy_remove+0x55/0x60 [ 2234.569495] [] sas_rphy_delete+0x11/0x20 [ 2234.569562] [] sas_port_delete+0x25/0x160 [ 2234.569628] [] mptsas_del_end_device+0x183/0x270 [ 2234.569694] [] mptsas_hotplug_work+0x1ec/0x920 [ 2234.569761] [] ? mptsas_free_fw_event+0x6b/0xb0 [ 2234.569827] [] ? sched_clock_cpu+0xc5/0x120 [ 2234.569891] [] mptsas_firmware_event_work+0xbc0/0xfa0 [ 2234.569957] [] ? __lock_acquire.isra.27+0x29f/0xb30 [ 2234.570024] [] ? mptsas_expander_add+0x140/0x140 [ 2234.570090] [] ? mptsas_expander_add+0x140/0x140 [ 2234.570156] [] process_one_work+0x184/0x460 [ 2234.570220] [] ? process_one_work+0x126/0x460 [ 2234.570287] [] worker_thread+0x15e/0x350 [ 2234.570353] [] ? manage_workers.isra.32+0x220/0x220 [ 2234.570422] [] kthread+0x9d/0xb0 [ 2234.570487] [] kernel_thread_helper+0x4/0x10 [ 2234.570556] [] ? __init_kthread_worker+0x70/0x70 [ 2234.570625] [] ? gs_change+0xb/0xb [ 2234.570687] Code: 04 25 08 ce 00 00 48 89 e5 0f 1f 00 0f ae e8 0f 31 89 c6 eb 16 0f 1f 80 00 00 00 00 f3 90 65 8b 0c 25 08 ce 00 00 41 39 c8 75 12 <0f> 1f 00 0f ae e8 0f 31 89 c2 29 f2 39 fa 72 e1 5d c3 29 c6 01 [ 2234.571087] NMI backtrace for cpu 4 [ 2234.571149] CPU 4 [ 2234.571155] Modules linked in:[ 2234.571213] netconsole configfs w83627ehf h wmon_vid ipmi_si ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_intel k vm e1000e lpc_ich mfd_core i2c_i801 pcspkr microcode i2c_core video button xts g f128mul aes_x86_64 aes_generic cbc sha256_generic e1000 nfs lockd fscache auth_r pcgss nfs_acl sunrpc reiserfs multipath linear raid0 dm_raid dm_snapshot dm_cryp t dm_mirror dm_region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_hcd usb _storage ehci_hcd megaraid_sas megaraid_mbox megaraid_mm megaraid sr_mod cdrom s d_mod sata_mv ata_piix ahci libahci pata_marvell pata_mpiix libata [ 2234.572022] Pid: 0, comm: swapper/4 Not tainted 3.5.1-gentoo-dbg-lock-1 #2 To be filled by O.E.M. To be filled by O.E.M./P8B-X series [ 2234.572157] RIP: 0010:[] [] mwait_idle_w ith_hints+0x58/0x70 [ 2234.572279] RSP: 0018:ffff8802235a3e58 EFLAGS: 00000046 [ 2234.572342] RAX: 0000000000000020 RBX: ffff8802232452e8 RCX: 0000000000000001 [ 2234.572411] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000020 [ 2234.572481] RBP: ffff8802235a3e58 R08: ffff8802235a3fd8 R09: 0000000000000000 [ 2234.572550] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000003 [ 2234.572618] R13: ffff880223245000 R14: ffff880223245020 R15: 12ad09af1b03210f [ 2234.572688] FS: 0000000000000000(0000) GS:ffff880226e00000(0000) knlGS:00000 00000000000 [ 2234.572808] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 2234.572874] CR2: 00007fab1b8e3000 CR3: 0000000001a0b000 CR4: 00000000000407e0 [ 2234.572943] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2234.573011] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 2234.573078] Process swapper/4 (pid: 0, threadinfo ffff8802235a2000, task ffff 88022359a230) [ 2234.573200] Stack: [ 2234.573258] ffff8802235a3e68 ffffffff8101d54d ffff8802235a3ec8 ffffffff813d8 3bd [ 2234.573386] ffffffff81a17b40 00000000ffffffff 0000000000000000 0000000000000 001 [ 2234.573512] ffff8802235a3ee8 ffff880223245020 ffffffff81a48b10 00000000fffff ff0 [ 2234.573636] Call Trace: [ 2234.573693] [] acpi_processor_ffh_cstate_enter+0x2d/0x30 [ 2234.573761] [] acpi_idle_enter_bm+0x1dc/0x293 [ 2234.573830] [] cpuidle_enter+0x19/0x20 [ 2234.573897] [] cpuidle_idle_call+0x8b/0xf0 [ 2234.573966] [] cpu_idle+0x5a/0xc0 [ 2234.574034] [] start_secondary+0x212/0x214 [ 2234.574099] Code: 8b 04 25 f0 b8 00 00 49 8d 80 38 e0 ff ff 48 89 d1 0f 01 c8 0f ae f0 49 8b 80 38 e0 ff ff a8 08 75 09 48 89 f8 48 89 f1 0f 01 c9 <5d> c3 66 0f 1f 44 00 00 0f ae ba 38 e0 ff ff eb c0 0f 1f 80 00 [ 2234.578736] NMI backtrace for cpu 5 [ 2234.578797] CPU 5 [ 2234.578803] Modules linked in:[ 2234.578859] netconsole configfs w83627ehf h wmon_vid ipmi_si ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_intel k vm e1000e lpc_ich mfd_core i2c_i801 pcspkr microcode i2c_core video button xts g f128mul aes_x86_64 aes_generic cbc sha256_generic e1000 nfs lockd fscache auth_r pcgss nfs_acl sunrpc reiserfs multipath linear raid0 dm_raid dm_snapshot dm_cryp t dm_mirror dm_region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_hcd usb _storage ehci_hcd megaraid_sas megaraid_mbox megaraid_mm megaraid sr_mod cdrom s d_mod sata_mv ata_piix ahci libahci pata_marvell pata_mpiix libata [ 2234.579587] Pid: 0, comm: swapper/5 Not tainted 3.5.1-gentoo-dbg-lock-1 #2 To be filled by O.E.M. To be filled by O.E.M./P8B-X series [ 2234.579721] RIP: 0010:[] [] mwait_idle_w ith_hints+0x58/0x70 [ 2234.579846] RSP: 0018:ffff8802235a5e58 EFLAGS: 00000046 [ 2234.579908] RAX: 0000000000000020 RBX: ffff880223245ae8 RCX: 0000000000000001 [ 2234.579975] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000020 [ 2234.580041] RBP: ffff8802235a5e58 R08: ffff8802235a5fd8 R09: 0000000000000000 [ 2234.580108] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000003 [ 2234.580175] R13: ffff880223245800 R14: ffff880223245820 R15: 12ad09af1b034d34 [ 2234.580241] FS: 0000000000000000(0000) GS:ffff880227000000(0000) knlGS:00000 00000000000 [ 2234.580361] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 2234.580427] CR2: 00007f62e1b73000 CR3: 0000000001a0b000 CR4: 00000000000407e0 [ 2234.580495] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2234.580563] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 2234.580631] Process swapper/5 (pid: 0, threadinfo ffff8802235a4000, task ffff 88022359c460) [ 2234.580751] Stack: [ 2234.580809] ffff8802235a5e68 ffffffff8101d54d ffff8802235a5ec8 ffffffff813d8 3bd [ 2234.580937] ffffffff81a17b40 00000000ffffffff 0000000000000000 0000000000000 001 [ 2234.581062] ffff8802235a5ee8 ffff880223245820 ffffffff81a48b10 00000000fffff ff0 [ 2234.581188] Call Trace: [ 2234.581248] [] acpi_processor_ffh_cstate_enter+0x2d/0x30 [ 2234.581318] [] acpi_idle_enter_bm+0x1dc/0x293 [ 2234.581387] [] cpuidle_enter+0x19/0x20 [ 2234.581454] [] cpuidle_idle_call+0x8b/0xf0 [ 2234.581521] [] cpu_idle+0x5a/0xc0 [ 2234.581586] [] start_secondary+0x212/0x214 [ 2234.581652] Code: 8b 04 25 f0 b8 00 00 49 8d 80 38 e0 ff ff 48 89 d1 0f 01 c8 0f ae f0 49 8b 80 38 e0 ff ff a8 08 75 09 48 89 f8 48 89 f1 0f 01 c9 <5d> c3 66 0f 1f 44 00 00 0f ae ba 38 e0 ff ff eb c0 0f 1f 80 00 [ 2234.582056] NMI backtrace for cpu 1 [ 2234.582116] CPU 1 [ 2234.582121] Modules linked in:[ 2234.582180] netconsole configfs w83627ehf h wmon_vid ipmi_si ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_intel k vm e1000e lpc_ich mfd_core i2c_i801 pcspkr microcode i2c_core video button xts g f128mul aes_x86_64 aes_generic cbc sha256_generic e1000 nfs lockd fscache auth_r pcgss nfs_acl sunrpc reiserfs multipath linear raid0 dm_raid dm_snapshot dm_cryp t dm_mirror dm_region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_hcd usb _storage ehci_hcd megaraid_sas megaraid_mbox megaraid_mm megaraid sr_mod cdrom s d_mod sata_mv ata_piix ahci libahci pata_marvell pata_mpiix libata [ 2234.582880] Pid: 0, comm: swapper/1 Not tainted 3.5.1-gentoo-dbg-lock-1 #2 To be filled by O.E.M. To be filled by O.E.M./P8B-X series [ 2234.583014] RIP: 0010:[] [] mwait_idle_w ith_hints+0x58/0x70 [ 2234.583138] RSP: 0000:ffff88022350de58 EFLAGS: 00000046 [ 2234.583202] RAX: 0000000000000020 RBX: ffff8802226f02e8 RCX: 0000000000000001 [ 2234.583271] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000020 [ 2234.583336] RBP: ffff88022350de58 R08: ffff88022350dfd8 R09: 0000000000000000 [ 2234.583401] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000003 [ 2234.583466] R13: ffff8802226f0000 R14: ffff8802226f0020 R15: 12ad09af1b0364fa [ 2234.583531] FS: 0000000000000000(0000) GS:ffff880226800000(0000) knlGS:00000 00000000000 [ 2234.583644] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 2234.583710] CR2: 00007fab1b4b89e8 CR3: 0000000001a0b000 CR4: 00000000000407e0 [ 2234.583778] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2234.583843] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 2234.583911] Process swapper/1 (pid: 0, threadinfo ffff88022350c000, task ffff 880223592230) [ 2234.584031] Stack: [ 2234.584088] ffff88022350de68 ffffffff8101d54d ffff88022350dec8 ffffffff813d8 3bd [ 2234.584213] ffffffff81a17b40 00000000ffffffff 0000000000000000 0000000000000 001 [ 2234.584336] ffff88022350dee8 ffff8802226f0020 ffffffff81a48b10 00000000fffff ff0 [ 2234.584457] Call Trace: [ 2234.584514] [] acpi_processor_ffh_cstate_enter+0x2d/0x30 [ 2234.584579] [] acpi_idle_enter_bm+0x1dc/0x293 [ 2234.584643] [] cpuidle_enter+0x19/0x20 [ 2234.584709] [] cpuidle_idle_call+0x8b/0xf0 [ 2234.584775] [] cpu_idle+0x5a/0xc0 [ 2234.584840] [] start_secondary+0x212/0x214 [ 2234.584905] Code: 8b 04 25 f0 b8 00 00 49 8d 80 38 e0 ff ff 48 89 d1 0f 01 c8 0f ae f0 49 8b 80 38 e0 ff ff a8 08 75 09 48 89 f8 48 89 f1 0f 01 c9 <5d> c3 66 0f 1f 44 00 00 0f ae ba 38 e0 ff ff eb c0 0f 1f 80 00 [ 2234.585263] NMI backtrace for cpu 2 [ 2234.585268] 0: (18001 ticks this GP) idle=c6b/140000000000001/0 [ 2234.585269] (detected by 1, t=18007 jiffies) [ 2234.585270] INFO: Stall ended before state dump start [ 2234.585266] INFO: rcu_sched detected stalls on CPUs/tasks: [ 2234.585508] CPU 2 [ 2234.585575] Modules linked in: netconsole configfs w8362 7ehf hwmon_vid ipmi_si ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_i ntel kvm e1000e lpc_ich mfd_core i2c_i801 pcspkr microcode i2c_core video button xts gf128mul aes_x86_64 aes_generic cbc sha256_generic e1000 nfs lockd fscache auth_rpcgss nfs_acl sunrpc reiserfs multipath linear raid0 dm_raid dm_snapshot d m_crypt dm_mirror dm_region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_h cd usb_storage ehci_hcd megaraid_sas megaraid_mbox megaraid_mm megaraid sr_mod c drom sd_mod sata_mv ata_piix ahci libahci pata_marvell pata_mpiix libata [ 2234.587328] Pid: 0, comm: swapper/2 Not tainted 3.5.1-gentoo-dbg-lock-1 #2 To be filled by O.E.M. To be filled by O.E.M./P8B-X series [ 2234.587464] RIP: 0010:[] [] mwait_idle_w ith_hints+0x58/0x70 [ 2234.587588] RSP: 0018:ffff88022350fe58 EFLAGS: 00000046 [ 2234.587651] RAX: 0000000000000020 RBX: ffff8802232202e8 RCX: 0000000000000001 [ 2234.587719] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000020 [ 2234.587785] RBP: ffff88022350fe58 R08: ffff88022350ffd8 R09: 0000000000000000 [ 2234.587853] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000003 [ 2234.587920] R13: ffff880223220000 R14: ffff880223220020 R15: 12ad09af1b03638a [ 2234.587987] FS: 0000000000000000(0000) GS:ffff880226a00000(0000) knlGS:00000 00000000000 [ 2234.588102] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 2234.588170] CR2: 00007fab1b4b89e8 CR3: 0000000001a0b000 CR4: 00000000000407e0 [ 2234.588239] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2234.588308] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 2234.588376] Process swapper/2 (pid: 0, threadinfo ffff88022350e000, task ffff 880223594460) [ 2234.588492] Stack: [ 2234.588548] ffff88022350fe68 ffffffff8101d54d ffff88022350fec8 ffffffff813d8 3bd [ 2234.588672] ffffffff81a17b40 00000000ffffffff 0000000000000000 0000000000000 001 [ 2234.588797] ffff88022350fee8 ffff880223220020 ffffffff81a48b10 00000000fffff ff0 [ 2234.588922] Call Trace: [ 2234.588981] [] acpi_processor_ffh_cstate_enter+0x2d/0x30 [ 2234.589048] [] acpi_idle_enter_bm+0x1dc/0x293 [ 2234.589115] [] cpuidle_enter+0x19/0x20 [ 2234.589182] [] cpuidle_idle_call+0x8b/0xf0 [ 2234.589250] [] cpu_idle+0x5a/0xc0 [ 2234.589315] [] start_secondary+0x212/0x214 [ 2234.589381] Code: 8b 04 25 f0 b8 00 00 49 8d 80 38 e0 ff ff 48 89 d1 0f 01 c8 0f ae f0 49 8b 80 38 e0 ff ff a8 08 75 09 48 89 f8 48 89 f1 0f 01 c9 <5d> c3 66 0f 1f 44 00 00 0f ae ba 38 e0 ff ff eb c0 0f 1f 80 00 [ 2234.589784] NMI backtrace for cpu 6 [ 2234.589846] CPU 6 [ 2234.589851] Modules linked in:[ 2234.589910] netconsole configfs w83627ehf h wmon_vid ipmi_si ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_intel k vm e1000e lpc_ich mfd_core i2c_i801 pcspkr microcode i2c_core video button xts g f128mul aes_x86_64 aes_generic cbc sha256_generic e1000 nfs lockd fscache auth_r pcgss nfs_acl sunrpc reiserfs multipath linear raid0 dm_raid dm_snapshot dm_cryp t dm_mirror dm_region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_hcd usb _storage ehci_hcd megaraid_sas megaraid_mbox megaraid_mm megaraid sr_mod cdrom s d_mod sata_mv ata_piix ahci libahci pata_marvell pata_mpiix libata [ 2234.590608] Pid: 0, comm: swapper/6 Not tainted 3.5.1-gentoo-dbg-lock-1 #2 To be filled by O.E.M. To be filled by O.E.M./P8B-X series [ 2234.590743] RIP: 0010:[] [] mwait_idle_w ith_hints+0x58/0x70 [ 2234.590867] RSP: 0018:ffff8802235a7e58 EFLAGS: 00000046 [ 2234.590929] RAX: 0000000000000020 RBX: ffff8802232462e8 RCX: 0000000000000001 [ 2234.590995] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000020 [ 2234.591060] RBP: ffff8802235a7e58 R08: ffff8802235a7fd8 R09: 0000000000000000 [ 2234.591126] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000003 [ 2234.591193] R13: ffff880223246000 R14: ffff880223246020 R15: 12ad09af1b0354ad [ 2234.591259] FS: 0000000000000000(0000) GS:ffff880227200000(0000) knlGS:00000 00000000000 [ 2234.591379] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 2234.591445] CR2: 00007fab1b4b89e8 CR3: 0000000001a0b000 CR4: 00000000000407e0 [ 2234.591513] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2234.591580] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 2234.591649] Process swapper/6 (pid: 0, threadinfo ffff8802235a6000, task ffff 8802235a8000) [ 2234.591765] Stack: [ 2234.591821] ffff8802235a7e68 ffffffff8101d54d ffff8802235a7ec8 ffffffff813d8 3bd [ 2234.591941] ffffffff81a17b40 00000000ffffffff 0000000000000000 0000000000000 001 [ 2234.592061] ffff8802235a7ee8 ffff880223246020 ffffffff81a48b10 00000000fffff ff0 [ 2234.592180] Call Trace: [ 2234.592236] [] acpi_processor_ffh_cstate_enter+0x2d/0x30 [ 2234.592301] [] acpi_idle_enter_bm+0x1dc/0x293 [ 2234.592365] [] cpuidle_enter+0x19/0x20 [ 2234.592427] [] cpuidle_idle_call+0x8b/0xf0 [ 2234.592489] [] cpu_idle+0x5a/0xc0 [ 2234.592550] [] start_secondary+0x212/0x214 [ 2234.592612] Code: 8b 04 25 f0 b8 00 00 49 8d 80 38 e0 ff ff 48 89 d1 0f 01 c8 0f ae f0 49 8b 80 38 e0 ff ff a8 08 75 09 48 89 f8 48 89 f1 0f 01 c9 <5d> c3 66 0f 1f 44 00 00 0f ae ba 38 e0 ff ff eb c0 0f 1f 80 00 [ 2234.592963] NMI backtrace for cpu 3 [ 2234.593023] CPU 3 [ 2234.593029] Modules linked in:[ 2234.593086] netconsole configfs w83627ehf h wmon_vid ipmi_si ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_intel k vm e1000e lpc_ich mfd_core i2c_i801 pcspkr microcode i2c_core video button xts g f128mul aes_x86_64 aes_generic cbc sha256_generic e1000 nfs lockd fscache auth_r pcgss nfs_acl sunrpc reiserfs multipath linear raid0 dm_raid dm_snapshot dm_cryp t dm_mirror dm_region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_hcd usb _storage ehci_hcd megaraid_sas megaraid_mbox megaraid_mm megaraid sr_mod cdrom s d_mod sata_mv ata_piix ahci libahci pata_marvell pata_mpiix libata [ 2234.593806] Pid: 0, comm: swapper/3 Not tainted 3.5.1-gentoo-dbg-lock-1 #2 To be filled by O.E.M. To be filled by O.E.M./P8B-X series [ 2234.593936] RIP: 0010:[] [] mwait_idle_w ith_hints+0x58/0x70 [ 2234.594059] RSP: 0018:ffff8802235a1e58 EFLAGS: 00000046 [ 2234.594124] RAX: 0000000000000020 RBX: ffff8802233a82e8 RCX: 0000000000000001 [ 2234.594192] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000020 [ 2234.594260] RBP: ffff8802235a1e58 R08: ffff8802235a1fd8 R09: 0000000000000000 [ 2234.594330] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000003 [ 2234.594399] R13: ffff8802233a8000 R14: ffff8802233a8020 R15: 12ad09af1b036dbc [ 2234.594468] FS: 0000000000000000(0000) GS:ffff880226c00000(0000) knlGS:00000 00000000000 [ 2234.594588] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 2234.594653] CR2: 00007fab1b4b89e8 CR3: 0000000001a0b000 CR4: 00000000000407e0 [ 2234.594719] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2234.594787] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 2234.594856] Process swapper/3 (pid: 0, threadinfo ffff8802235a0000, task ffff 880223598000) [ 2234.594975] Stack: [ 2234.595033] ffff8802235a1e68 ffffffff8101d54d ffff8802235a1ec8 ffffffff813d8 3bd [ 2234.595162] ffffffff81a17b40 00000000ffffffff 0000000000000000 0000000000000 001 [ 2234.595290] ffff8802235a1ee8 ffff8802233a8020 ffffffff81a48b10 00000000fffff ff0 [ 2234.595418] Call Trace: [ 2234.595477] [] acpi_processor_ffh_cstate_enter+0x2d/0x30 [ 2234.595546] [] acpi_idle_enter_bm+0x1dc/0x293 [ 2234.595612] [] cpuidle_enter+0x19/0x20 [ 2234.595678] [] cpuidle_idle_call+0x8b/0xf0 [ 2234.595744] [] cpu_idle+0x5a/0xc0 [ 2234.595809] [] start_secondary+0x212/0x214 [ 2234.595873] Code: 8b 04 25 f0 b8 00 00 49 8d 80 38 e0 ff ff 48 89 d1 0f 01 c8 0f ae f0 49 8b 80 38 e0 ff ff a8 08 75 09 48 89 f8 48 89 f1 0f 01 c9 <5d> c3 66 0f 1f 44 00 00 0f ae ba 38 e0 ff ff eb c0 0f 1f 80 00 [ 2234.596280] NMI backtrace for cpu 7 [ 2234.596339] CPU 7 [ 2234.596344] Modules linked in:[ 2234.596402] netconsole configfs w83627ehf h wmon_vid ipmi_si ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_intel k vm e1000e lpc_ich mfd_core i2c_i801 pcspkr microcode i2c_core video button xts g f128mul aes_x86_64 aes_generic cbc sha256_generic e1000 nfs lockd fscache auth_r pcgss nfs_acl sunrpc reiserfs multipath linear raid0 dm_raid dm_snapshot dm_cryp t dm_mirror dm_region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_hcd usb _storage ehci_hcd megaraid_sas megaraid_mbox megaraid_mm megaraid sr_mod cdrom s d_mod sata_mv ata_piix ahci libahci pata_marvell pata_mpiix libata [ 2234.597108] Pid: 0, comm: swapper/7 Not tainted 3.5.1-gentoo-dbg-lock-1 #2 To be filled by O.E.M. To be filled by O.E.M./P8B-X series [ 2234.597236] RIP: 0010:[] [] mwait_idle_w ith_hints+0x58/0x70 [ 2234.597354] RSP: 0018:ffff8802235b1e58 EFLAGS: 00000046 [ 2234.597415] RAX: 0000000000000020 RBX: ffff880223246ae8 RCX: 0000000000000001 [ 2234.597482] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000020 [ 2234.597550] RBP: ffff8802235b1e58 R08: ffff8802235b1fd8 R09: 0000000000000000 [ 2234.597618] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000003 [ 2234.597686] R13: ffff880223246800 R14: ffff880223246820 R15: 12ad09af1b035836 [ 2234.597754] FS: 0000000000000000(0000) GS:ffff880227400000(0000) knlGS:00000 00000000000 [ 2234.597873] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 2234.597938] CR2: 000000000062b2c8 CR3: 0000000001a0b000 CR4: 00000000000407e0 [ 2234.598005] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2234.598073] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 2234.598138] Process swapper/7 (pid: 0, threadinfo ffff8802235b0000, task ffff 8802235aa230) [ 2234.598254] Stack: [ 2234.598310] ffff8802235b1e68 ffffffff8101d54d ffff8802235b1ec8 ffffffff813d8 3bd [ 2234.598434] ffffffff81a17b40 00000000ffffffff 0000000000000000 0000000000000 001 [ 2234.598561] ffff8802235b1ee8 ffff880223246820 ffffffff81a48b10 00000000fffff ff0 [ 2234.598688] Call Trace: [ 2234.598747] [] acpi_processor_ffh_cstate_enter+0x2d/0x30 [ 2234.598815] [] acpi_idle_enter_bm+0x1dc/0x293 [ 2234.598882] [] cpuidle_enter+0x19/0x20 [ 2234.598946] [] cpuidle_idle_call+0x8b/0xf0 [ 2234.599010] [] cpu_idle+0x5a/0xc0 [ 2234.599074] [] start_secondary+0x212/0x214 [ 2234.599137] Code: 8b 04 25 f0 b8 00 00 49 8d 80 38 e0 ff ff 48 89 d1 0f 01 c8 0f ae f0 49 8b 80 38 e0 ff ff a8 08 75 09 48 89 f8 48 89 f1 0f 01 c9 <5d> c3 66 0f 1f 44 00 00 0f ae ba 38 e0 ff ff eb c0 0f 1f 80 00 [ 2234.594787] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 2234.594856] Process swapper/3 (pid: 0, threadinfo ffff8802235a0000, task ffff880223598000) [ 2234.594975] Stack: [ 2234.595033] ffff8802235a1e68 ffffffff8101d54d ffff8802235a1ec8 ffffffff813d83bd [ 2234.595162] ffffffff81a17b40 00000000ffffffff 0000000000000000 0000000000000001 [ 2234.595290] ffff8802235a1ee8 ffff8802233a8020 ffffffff81a48b10 00000000fffffff0 [ 2234.595418] Call Trace: [ 2234.595477] [] acpi_processor_ffh_cstate_enter+0x2d/0x30 [ 2234.595546] [] acpi_idle_enter_bm+0x1dc/0x293 [ 2234.595612] [] cpuidle_enter+0x19/0x20 [ 2234.595678] [] cpuidle_idle_call+0x8b/0xf0 [ 2234.595744] [] cpu_idle+0x5a/0xc0 [ 2234.595809] [] start_secondary+0x212/0x214 [ 2234.595873] Code: 8b 04 25 f0 b8 00 00 49 8d 80 38 e0 ff ff 48 89 d1 0f 01 c8 0f ae f0 49 8b 80 38 e0 ff ff a8 08 75 09 48 89 f8 48 89 f1 0f 01 c9 <5d> c3 66 0f 1f 44 00 00 0f ae ba 38 e0 ff ff eb c0 0f 1f 80 00 [ 2234.596280] NMI backtrace for cpu 7 [ 2234.596339] CPU 7 [ 2234.596344] Modules linked in:[ 2234.596402] netconsole configfs w83627ehf hwmon_vid ipmi_si ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_intel kvm e1000e lpc_ich mfd_core i2c_i801 pcspkr microcode i2c_core video button xts gf128mul aes_x86_64 aes_generic cbc sha256_generic e1000 nfs lockd fscache auth_rpcgss nfs_acl sunrpc reiserfs multipath linear raid0 dm_raid dm_snapshot dm_crypt dm_mirror dm_region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_hcd usb_storage ehci_hcd megaraid_sas megaraid_mbox megaraid_mm megaraid sr_mod cdrom sd_mod sata_mv ata_piix ahci libahci pata_marvell pata_mpiix libata [ 2234.597108] Pid: 0, comm: swapper/7 Not tainted 3.5.1-gentoo-dbg-lock-1 #2 To be filled by O.E.M. To be filled by O.E.M./P8B-X series [ 2234.597236] RIP: 0010:[] [] mwait_idle_with_hints+0x58/0x70 [ 2234.597354] RSP: 0018:ffff8802235b1e58 EFLAGS: 00000046 [ 2234.597415] RAX: 0000000000000020 RBX: ffff880223246ae8 RCX: 0000000000000001 [ 2234.597482] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000020 [ 2234.597550] RBP: ffff8802235b1e58 R08: ffff8802235b1fd8 R09: 0000000000000000 [ 2234.597618] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000003 [ 2234.597686] R13: ffff880223246800 R14: ffff880223246820 R15: 12ad09af1b035836 [ 2234.597754] FS: 0000000000000000(0000) GS:ffff880227400000(0000) knlGS:0000000000000000 [ 2234.597873] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 2234.597938] CR2: 000000000062b2c8 CR3: 0000000001a0b000 CR4: 00000000000407e0 [ 2234.598005] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2234.598073] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 2234.598138] Process swapper/7 (pid: 0, threadinfo ffff8802235b0000, task ffff8802235aa230) [ 2234.598254] Stack: [ 2234.598310] ffff8802235b1e68 ffffffff8101d54d ffff8802235b1ec8 ffffffff813d83bd [ 2234.598434] ffffffff81a17b40 00000000ffffffff 0000000000000000 0000000000000001 [ 2234.598561] ffff8802235b1ee8 ffff880223246820 ffffffff81a48b10 00000000fffffff0 [ 2234.598688] Call Trace: [ 2234.598747] [] acpi_processor_ffh_cstate_enter+0x2d/0x30 [ 2234.598815] [] acpi_idle_enter_bm+0x1dc/0x293 [ 2234.598882] [] cpuidle_enter+0x19/0x20 [ 2234.598946] [] cpuidle_idle_call+0x8b/0xf0 [ 2234.599010] [] cpu_idle+0x5a/0xc0 [ 2234.599074] [] start_secondary+0x212/0x214 [ 2234.599137] Code: 8b 04 25 f0 b8 00 00 49 8d 80 38 e0 ff ff 48 89 d1 0f 01 c8 0f ae f0 49 8b 80 38 e0 ff ff a8 08 75 09 48 89 f8 48 89 f1 0f 01 c9 <5d> c3 66 0f 1f 44 00 00 0f ae ba 38 e0 ff ff eb c0 0f 1f 80 00 [ 2259.018905] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u:3:6089] [ 2259.018973] Modules linked in: netconsole configfs w83627ehf hwmon_vid ipmi_si ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_intel kvm e1000e lpc_ich mfd_core i2c_i801 pcspkr microcode i2c_core video button xts gf128mul aes_x86_64 aes_generic cbc sha256_generic e1000 nfs lockd fscache auth_rpcgss nfs_acl dm_mirror dm_region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_hcd usb_storage ehci_hcd megaraid_sas megaraid_mbox megaraid_mm megaraid sr_mod cdrom sd_mod sata_mv ata_piix ahci libahci pata_marvell pata_mpiix libata [ 2259.020627] CPU 0 [ 2259.020632] Modules linked in:[ 2259.020689] netconsole configfs w83627ehf hwmon_vid ipmi_si ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_intel kvm e1000e lpc_ich mfd_core i2c_i801 pcspkr microcode i2c_core video button xts gf128mul aes_x86_64 aes_generic cbc sha256_generic e1000 nfs lockd fscache auth_rpcgss nfs_acl sunrpc reiserfs multipath linear raid0 dm_raid dm_snapshot dm_crypt dm_mirror dm_region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_hcd usb_storage ehci_hcd megaraid_sas megaraid_mbox megaraid_mm megaraid sr_mod cdrom sd_mod sata_mv ata_piix ahci libahci pata_marvell pata_mpiix libata [ 2259.021357] Pid: 6089, comm: kworker/u:3 Not tainted 3.5.1-gentoo-dbg-lock-1 #2 To be filled by O.E.M. To be filled by O.E.M./P8B-X series [ 2259.021493] RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x32/0x40 [ 2259.021615] RSP: 0018:ffff880221dada70 EFLAGS: 00000286 [ 2259.021678] RAX: 0000000000000000 RBX: ffff880221dada20 RCX: 000000000000b828 [ 2259.021744] RDX: 0000000000002e40 RSI: ffff880226600000 RDI: ffff8802233f2090 [ 2259.021811] RBP: ffff880221dada80 R08: 0000000000000001 R09: 0000000000000000 [ 2259.021877] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88021e1bc460 [ 2259.021943] R13: 0000000000000082 R14: ffff880221dad9f0 R15: ffff8802233f20a8 [ 2259.022012] FS: 0000000000000000(0000) GS:ffff880226600000(0000) knlGS:0000000000000000 [ 2259.022132] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 2259.022195] CR2: 00007f42a5fb2070 CR3: 0000000001a0b000 CR4: 00000000000407f0 [ 2259.022260] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2259.022326] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 2259.022392] Process kworker/u:3 (pid: 6089, threadinfo ffff880221dac000, task ffff88021e1bc460) [ 2259.022512] Stack: [ 2259.022569] ffff88021eb50000 ffff8802233f2000 ffff880221dadab0 ffffffff814151bd [ 2259.022695] ffff880221dadab0 ffff88021eb50000 ffff8802233f1ff0 ffff8802233f2000 [ 2259.022820] ffff880221dadb00 ffffffff81417a74 0000000000000282 ffff8802233f2000 [ 2259.023045] Call Trace: [ 2259.023103] [] scsi_target_reap+0x7d/0x100 [ 2259.023165] [] scsi_remove_target+0x1c4/0x1f0 [ 2259.023228] [] sas_rphy_remove+0x55/0x60 [ 2259.023291] [] sas_rphy_delete+0x11/0x20 [ 2259.023357] [] sas_port_delete+0x25/0x160 [ 2259.023422] [] mptsas_del_end_device+0x183/0x270 [ 2259.023488] [] mptsas_hotplug_work+0x1ec/0x920 [ 2259.023556] [] ? mptsas_free_fw_event+0x6b/0xb0 [ 2259.023623] [] ? sched_clock_cpu+0xc5/0x120 [ 2259.023690] [] mptsas_firmware_event_work+0xbc0/0xfa0 [ 2259.023758] [] ? __lock_acquire.isra.27+0x29f/0xb30 [ 2259.023825] [] ? mptsas_expander_add+0x140/0x140 [ 2259.023891] [] ? mptsas_expander_add+0x140/0x140 [ 2259.023955] [] process_one_work+0x184/0x460 [ 2259.024018] [] ? process_one_work+0x126/0x460 [ 2259.024081] [] worker_thread+0x15e/0x350 [ 2259.024143] [] ? manage_workers.isra.32+0x220/0x220 [ 2259.024208] [] kthread+0x9d/0xb0 [ 2259.024270] [] kernel_thread_helper+0x4/0x10 [ 2259.024336] [] ? __init_kthread_worker+0x70/0x70 [ 2259.024404] [] ? gs_change+0xb/0xb [ 2259.024468] Code: 10 48 8b 55 08 48 89 5d f0 48 89 f3 4c 89 65 f8 be 01 00 00 00 49 89 fc 48 8d 7f 18 e8 58 a6 ab ff 4c 89 e7 e8 d0 8f d8 [ 2286.955440] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u:3:6089] [ 2286.955508] Modules linked in: netconsole configfs w83627ehf hwmon_vid ipmi_si ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_intel kvm e1000e lpc_ich mfd_core i2c_i801 pcspkr microcode i2c_core video button xts gf128mul aes_x86_64 aes_generic cbc sha256_generic e1000 nfs lockd fscache auth_rpcgss nfs_acl dm_mirror dm_region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_hcd usb_storage ehci_hcd megaraid_sas megaraid_mbox megaraid_mm megaraid sr_mod cdrom sd_mod sata_mv ata_piix ahci libahci pata_marvell pata_mpiix libata [ 2286.956139] CPU 0 [ 2286.956144] Modules linked in:[ 2286.956203] netconsole configfs w83627ehf hwmon_vid ipmi_si ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_intel kvm e1000e lpc_ich mfd_core i2c_i801 pcspkr microcode i2c_core video button xts gf128mul aes_x86_64 aes_generic cbc sha256_generic e1000 nfs lockd fscache auth_rpcgss nfs_acl sunrpc reiserfs multipath linear raid0 dm_raid dm_snapshot dm_crypt dm_mirror dm_region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_hcd usb_storage ehci_hcd megaraid_sas megaraid_mbox megaraid_mm megaraid sr_mod cdrom sd_mod sata_mv ata_piix ahci libahci pata_marvell pata_mpiix libata [ 2286.956879] Pid: 6089, comm: kworker/u:3 Not tainted 3.5.1-gentoo-dbg-lock-1 #2 To be filled by O.E.M. To be filled by O.E.M./P8B-X series [ 2286.957014] RIP: 0010:[] [] scsi_target_reap+0x8/0x100 [ 2286.957138] RSP: 0018:ffff880221dadaa0 EFLAGS: 00000282 [ 2286.957203] RAX: 0000000000000000 RBX: ffff880221dada50 RCX: 000000000000b828 [ 2286.957271] RDX: 0000000000002e40 RSI: ffff880226600000 RDI: ffff88021eb50000 [ 2286.957338] RBP: ffff880221dadab0 R08: 0000000000000001 R09: 0000000000000000 [ 2286.957405] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88021e1bc460 [ 2286.957474] R13: 0000000000000082 R14: ffff880221dada20 R15: ffff8802233f20a8 [ 2286.957542] FS: 0000000000000000(0000) GS:ffff880226600000(0000) knlGS:0000000000000000 [ 2286.957661] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 2286.957724] CR2: 00007f42a5fb2070 CR3: 0000000001a0b000 CR4: 00000000000407f0 [ 2286.957791] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2286.957859] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 2286.957927] Process kworker/u:3 (pid: 6089, threadinfo ffff880221dac000, task ffff88021e1bc460) [ 2286.958047] Stack: [ 2286.958105] ffff8802233f1ff0 ffff8802233f2000 ffff880221dadb00 ffffffff81417a74 [ 2286.958231] 0000000000000282 ffff8802233f2000 0000000000000046 ffff8802221e6800 [ 2286.958356] 0000000000000008 ffff880222156630 ffff88021eb0b000 1221000005000000 [ 2286.958483] Call Trace: [ 2286.958542] [] scsi_remove_target+0x1c4/0x1f0 [ 2286.958610] [] sas_rphy_remove+0x55/0x60 [ 2286.958677] [] sas_rphy_delete+0x11/0x20 [ 2286.958742] [] sas_port_delete+0x25/0x160 [ 2286.958810] [] mptsas_del_end_device+0x183/0x270 [ 2286.958876] [] mptsas_hotplug_work+0x1ec/0x920 [ 2286.958944] [] ? mptsas_free_fw_event+0x6b/0xb0 [ 2286.959011] [] ? sched_clock_cpu+0xc5/0x120 [ 2286.959075] [] mptsas_firmware_event_work+0xbc0/0xfa0 [ 2286.959140] [] ? __lock_acquire.isra.27+0x29f/0xb30 [ 2286.959205] [] ? mptsas_expander_add+0x140/0x140 [ 2286.959272] [] ? mptsas_expander_add+0x140/0x140 [ 2286.959339] [] process_one_work+0x184/0x460 [ 2286.959405] [] ? process_one_work+0x126/0x460 [ 2286.959473] [] worker_thread+0x15e/0x350 [ 2286.959539] [] ? manage_workers.isra.32+0x220/0x220 [ 2286.959606] [] kthread+0x9d/0xb0 [ 2286.959672] [] kernel_thread_helper+0x4/0x10 [ 2286.959740] [] ? __init_kthread_worker+0x70/0x70 [ 2286.959807] [] ? gs_change+0xb/0xb [ 2286.959870] Code: 95 50 ff ff ff e8 ec 81 1a 00 b9 01 00 00 00 4c 8b 95 50 ff ff ff e9 56 f5 ff ff 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 55 41 54 <53> 48 89 fb 48 83 ec 08 4c 8b 67 28 eb 13 66 2e 0f 1f 84 00 00 [ 2314.891975] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u:3:6089] [ 2314.892043] Modules linked in: netconsole configfs w83627ehf hwmon_vid ipmi_si ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_intel kvm e1000e lpc_ich mfd_core i2c_i801 pcspkr microcode i2c_core video button xts gf128mul aes_x86_64 aes_generic cbc sha256_generic e1000 nfs lockd fscache auth_rpcgss nfs_acl sunrpc reiserfs multipath linear raid0 dm_raid dm_snapshot dm_crypt dm_mirror dm_region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_hcd usb_storage ehci_hcd megaraid_sas megaraid_mbox megaraid_mm megaraid sr_mod cdrom sd_mod sata_mv ata_piix ahci libahci pata_marvell pata_mpiix libata [ 2314.892761] CPU 0 [ 2314.892766] Modules linked in:[ 2314.892897] netconsole configfs w83627ehf hwmon_vid ipmi_si ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_intel kvm e1000e lpc_ich mfd_core i2c_i801 pcspkr microcode i2c_core video button xts gf128mul aes_x86_64 aes_generic cbc sha256_generic e1000 nfs lockd fscache auth_rpcgss nfs_acl sunrpc reiserfs multipath linear raid0 dm_raid dm_snapshot dm_crypt dm_mirror dm_region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_hcd usb_storage ehci_hcd megaraid_sas megaraid_mbox megaraid_mm megaraid sr_mod cdrom sd_mod sata_mv ata_piix ahci libahci pata_marvell pata_mpiix libata [ 2314.893578] Pid: 6089, comm: kworker/u:3 Not tainted 3.5.1-gentoo-dbg-lock-1 #2 To be filled by O.E.M. To be filled by O.E.M./P8B-X series [ 2314.893792] RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x32/0x40 [ 2314.893912] RSP: 0018:ffff880221dada70 EFLAGS: 00000286 [ 2314.893974] RAX: 0000000000000000 RBX: ffff880221dada20 RCX: 000000000000b828 [ 2314.894040] RDX: 0000000000002e40 RSI: ffff880226600000 RDI: ffff8802233f2090 [ 2314.894104] RBP: ffff880221dada80 R08: 0000000000000001 R09: 0000000000000000 [ 2314.894170] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88021e1bc460 [ 2314.894236] R13: 0000000000000082 R14: ffff880221dad9f0 R15: ffff8802233f20a8 [ 2314.894302] FS: 0000000000000000(0000) GS:ffff880226600000(0000) knlGS:0000000000000000 [ 2314.894423] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 2314.894489] CR2: 00007f42a5fb2070 CR3: 0000000001a0b000 CR4: 00000000000407f0 [ 2314.894558] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2314.894626] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 2314.894694] Process kworker/u:3 (pid: 6089, threadinfo ffff880221dac000, task ffff88021e1bc460) [ 2314.894813] Stack: [ 2314.894869] ffff88021eb50000 ffff8802233f2000 ffff880221dadab0 ffffffff814151bd [ 2314.894995] ffff880221dadab0 ffff88021eb50000 ffff8802233f1ff0 ffff8802233f2000 [ 2314.895121] ffff880221dadb00 ffffffff81417a74 0000000000000282 ffff8802233f2000 [ 2314.895240] Call Trace: [ 2314.895296] [] scsi_target_reap+0x7d/0x100 [ 2314.895358] [] scsi_remove_target+0x1c4/0x1f0 [ 2314.895420] [] sas_rphy_remove+0x55/0x60 [ 2314.895482] [] sas_rphy_delete+0x11/0x20 [ 2314.896521] [] sas_port_delete+0x25/0x160 [ 2314.896583] [] mptsas_del_end_device+0x183/0x270 [ 2314.896647] [] mptsas_hotplug_work+0x1ec/0x920 [ 2314.896783] [] ? mptsas_free_fw_event+0x6b/0xb0 [ 2314.896850] [] ? sched_clock_cpu+0xc5/0x120 [ 2314.896915] [] mptsas_firmware_event_work+0xbc0/0xfa0 [ 2314.896982] [] ? __lock_acquire.isra.27+0x29f/0xb30 [ 2314.897047] [] ? mptsas_expander_add+0x140/0x140 [ 2314.897113] [] ? mptsas_expander_add+0x140/0x140 [ 2314.897178] [] process_one_work+0x184/0x460 [ 2314.897242] [] ? process_one_work+0x126/0x460 [ 2314.897305] [] worker_thread+0x15e/0x350 [ 2314.897367] [] ? manage_workers.isra.32+0x220/0x220 [ 2314.897432] [] kthread+0x9d/0xb0 [ 2314.897496] [] kernel_thread_helper+0x4/0x10 [ 2314.897563] [] ? __init_kthread_worker+0x70/0x70 [ 2314.897631] [] ? gs_change+0xb/0xb [ 2314.897691] Code: 10 48 8b 55 08 48 89 5d f0 48 89 f3 4c 89 65 f8 be 01 00 00 00 49 89 fc 48 8d 7f 18 e8 58 a6 ab ff 4c 89 e7 e8 d0 8f d8 ff 53 9d <48> 8b 5d f0 4c 8b 65 f8 c9 c3 0f 1f 40 00 55 be 01 00 00 00 48 [ 2342.828510] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u:3:6089] [ 2342.828574] Modules linked in: netconsole configfs w83627ehf hwmon_vid ipmi_si ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_intel kvm e1000e lpc_ich mfd_core i2c_i801 pcspkr microcode i2c_core video button xts gf128mul aes_x86_64 aes_generic cbc sha256_generic dm_mirror dm_region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_hcd usb_storage ehci_hcd megaraid_sas megaraid_mbox megaraid_mm megaraid sr_mod cdrom sd_mod sata_mv ata_piix ahci libahci pata_marvell pata_mpiix libata [ 2342.829192] CPU 0 [ 2342.829197] Modules linked in:[ 2342.829254] netconsole configfs w83627ehf hwmon_vid ipmi_si ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_intel kvm e1000e lpc_ich mfd_core i2c_i801 pcspkr microcode i2c_core video button xts gf128mul aes_x86_64 aes_generic cbc sha256_generic e1000 nfs lockd fscache auth_rpcgss nfs_acl sunrpc reiserfs multipath linear raid0 dm_raid dm_snapshot dm_crypt dm_mirror dm_region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_hcd usb_storage ehci_hcd megaraid_sas megaraid_mbox megaraid_mm megaraid sr_mod cdrom sd_mod sata_mv ata_piix ahci libahci pata_marvell pata_mpiix libata [ 2342.830039] Pid: 6089, comm: kworker/u:3 Not tainted 3.5.1-gentoo-dbg-lock-1 #2 To be filled by O.E.M. To be filled by O.E.M./P8B-X series [ 2342.830168] RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x32/0x40 [ 2342.830290] RSP: 0018:ffff880221dadaa0 EFLAGS: 00000282 [ 2342.830353] RAX: 0000000000000000 RBX: ffff880221dada50 RCX: 000000000000b828 [ 2342.830420] RDX: 0000000000002e40 RSI: ffff880226600000 RDI: ffff8802233f2090 [ 2342.830488] RBP: ffff880221dadab0 R08: 0000000000000001 R09: 0000000000000000 [ 2342.830556] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88021e1bc460 [ 2342.830623] R13: 0000000000000082 R14: ffff880221dada20 R15: ffff8802233f20a8 [ 2342.830691] FS: 0000000000000000(0000) GS:ffff880226600000(0000) knlGS:0000000000000000 [ 2342.830812] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 2342.830874] CR2: 00007f42a5fb2070 CR3: 0000000001a0b000 CR4: 00000000000407f0 [ 2342.830939] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2342.831005] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 2342.831071] Process kworker/u:3 (pid: 6089, threadinfo ffff880221dac000, task ffff88021e1bc460) [ 2342.831188] Stack: [ 2342.831244] ffff88021eb50000 ffff8802233f1ff0 ffff880221dadb00 ffffffff81417a6c [ 2342.831370] 0000000000000282 ffff8802233f2000 0000000000000046 ffff8802221e6800 [ 2342.831495] 0000000000000008 ffff880222156630 ffff88021eb0b000 1221000005000000 [ 2342.831623] Call Trace: [ 2342.831683] [] scsi_remove_target+0x1bc/0x1f0 [ 2342.831749] [] sas_rphy_remove+0x55/0x60 [ 2342.831814] [] sas_rphy_delete+0x11/0x20 [ 2342.831878] [] sas_port_delete+0x25/0x160 [ 2342.831942] [] mptsas_del_end_device+0x183/0x270 [ 2342.832009] [] mptsas_hotplug_work+0x1ec/0x920 [ 2342.832076] [] ? mptsas_free_fw_event+0x6b/0xb0 [ 2342.832144] [] ? sched_clock_cpu+0xc5/0x120 [ 2342.832210] [] mptsas_firmware_event_work+0xbc0/0xfa0 [ 2342.832277] [] ? __lock_acquire.isra.27+0x29f/0xb30 [ 2342.832345] [] ? mptsas_expander_add+0x140/0x140 [ 2342.832412] [] ? mptsas_expander_add+0x140/0x140 [ 2342.832479] [] process_one_work+0x184/0x460 [ 2342.832545] [] ? process_one_work+0x126/0x460 [ 2342.832610] [] worker_thread+0x15e/0x350 [ 2342.832676] [] ? manage_workers.isra.32+0x220/0x220 [ 2342.832744] [] kthread+0x9d/0xb0 [ 2342.832808] [] kernel_thread_helper+0x4/0x10 [ 2342.832875] [] ? __init_kthread_worker+0x70/0x70 [ 2342.832943] [] ? gs_change+0xb/0xb [ 2342.833007] Code: 10 48 8b 55 08 48 89 5d f0 48 89 f3 4c 89 65 f8 be 01 00 00 00 49 89 fc 48 8d 7f 18 e8 58 a6 ab ff 4c 89 e7 e8 d0 8f d8 ff 53 9d <48> 8b 5d f0 4c 8b 65 f8 c9 c3 0f 1f 40 00 55 be 01 00 00 00 48 [ 2370.765046] BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u:3:6089] [ 2370.765113] Modules linked in: netconsole configfs w83627ehf hwmon_vid ipmi_si ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_intel kvm e1000e lpc_ich mfd_core i2c_i801 pcspkr microcode i2c_core video button xts gf128mul aes_x86_64 aes_generic cbc sha256_generic e1000 nfs lockd fscache auth_rpcgss nfs_acl sunrpc reiserfs multipath linear raid0 dm_raid dm_snapshot dm_crypt dm_mirror dm_region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_hcd usb_storage ehci_hcd megaraid_sas megaraid_mbox megaraid_mm megaraid sr_mod cdrom sd_mod sata_mv ata_piix ahci libahci pata_marvell pata_mpiix libata [ 2370.765747] CPU 0 [ 2370.765752] Modules linked in:[ 2370.765810] netconsole configfs w83627ehf hwmon_vid ipmi_si ipmi_devintf ipmi_msghandler autofs4 coretemp hwmon kvm_intel kvm e1000e lpc_ich mfd_core i2c_i801 pcspkr microcode i2c_core video button xts gf128mul aes_x86_64 aes_generic cbc sha256_generic e1000 nfs lockd fscache auth_rpcgss nfs_acl sunrpc reiserfs multipath linear raid0 dm_raid dm_snapshot dm_crypt dm_mirror dm_region_hash dm_log scsi_wait_scan sl811_hcd ohci_hcd uhci_hcd usb_storage ehci_hcd megaraid_sas megaraid_mbox megaraid_mm megaraid sr_mod cdrom sd_mod sata_mv ata_piix ahci libahci pata_marvell pata_mpiix libata [ 2370.766490] Pid: 6089, comm: kworker/u:3 Not tainted 3.5.1-gentoo-dbg-lock-1 #2 To be filled by O.E.M. To be filled by O.E.M./P8B-X series [ 2370.766625] RIP: 0010:[] [] _raw_spin_unlock_irqrestore+0x32/0x40 [ 2370.766750] RSP: 0018:ffff880221dada70 EFLAGS: 00000286 [ 2370.766815] RAX: 0000000000000000 RBX: ffff880221dada20 RCX: 000000000000b828 [ 2370.766884] RDX: 0000000000002e40 RSI: ffff880226600000 RDI: ffff8802233f2090 [ 2370.766954] RBP: ffff880221dada80 R08: 0000000000000001 R09: 0000000000000000 [ 2370.767021] R10: 0000000000000001 R11: 0000000000000000 R12: ffff88021e1bc460 [ 2370.767087] R13: 0000000000000082 R14: ffff880221dad9f0 R15: ffff8802233f20a8 [ 2370.767152] FS: 0000000000000000(0000) GS:ffff880226600000(0000) knlGS:0000000000000000 [ 2370.767267] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 2370.767331] CR2: 00007f42a5fb2070 CR3: 0000000001a0b000 CR4: 00000000000407f0 [ 2370.767398] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 2370.767465] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 2370.767534] Process kworker/u:3 (pid: 6089, threadinfo ffff880221dac000, task ffff88021e1bc460) [ 2370.767655] Stack: [ 2370.767712] ffff88021eb50000 ffff8802233f2000 ffff880221dadab0 ffffffff814151bd [ 2370.767837] ffff880221dadab0 ffff88021eb50000 ffff8802233f1ff0 ffff8802233f2000 [ 2370.767960] ffff880221dadb00 ffffffff81417a74 0000000000000282 ffff8802233f2000 [ 2370.768084] Call Trace: [ 2370.768143] [] scsi_target_reap+0x7d/0x100 [ 2370.768209] [] scsi_remove_target+0x1c4/0x1f0 [ 2370.768275] [] sas_rphy_remove+0x55/0x60 [ 2370.768341] [] sas_rphy_delete+0x11/0x20 [ 2370.768405] [] sas_port_delete+0x25/0x160 [ 2370.768470] [] mptsas_del_end_device+0x183/0x270 [ 2370.768537] [] mptsas_hotplug_work+0x1ec/0x920 [ 2370.768603] [] ? mptsas_free_fw_event+0x6b/0xb0 [ 2370.768669] [] ? sched_clock_cpu+0xc5/0x120 [ 2370.768736] [] mptsas_firmware_event_work+0xbc0/0xfa0 [ 2370.768805] [] ? __lock_acquire.isra.27+0x29f/0xb30 [ 2370.768873] [] ? mptsas_expander_add+0x140/0x140 [ 2370.768938] [] ? mptsas_expander_add+0x140/0x140 [ 2370.769004] [] process_one_work+0x184/0x460 [ 2370.769068] [] ? process_one_work+0x126/0x460 [ 2370.769135] [] worker_thread+0x15e/0x350 [ 2370.769201] [] ? manage_workers.isra.32+0x220/0x220 [ 2370.769269] [] kthread+0x9d/0xb0 [ 2370.769333] [] kernel_thread_helper+0x4/0x10 [ 2370.769399] [] ? __init_kthread_worker+0x70/0x70 [ 2370.769465] [] ? gs_change+0xb/0xb [ 2370.769528] Code: 10 48 8b 55 08 48 89 5d f0 48 89 f3 4c 89 65 f8 be 01 00 00 00 49 89 fc 48 8d 7f 18 e8 58 a6 ab ff 4c 89 e7 e8 d0 8f d8 ff 53 9d <48> 8b 5d f0 4c 8b 65 f8 c9 c3 0f 1f 40 00 55 be 01 00 00 00 48 [ 2395.531904] INFO: task md1_raid6:4897 blocked for more than 120 seconds. [ 2395.531972] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2395.532089] md1_raid6 D ffff88022229a230 0 4897 2 0x00000000 [ 2395.532162] ffff8802201eba60 0000000000000046 ffff880226dd16d8 ffff8802201ebfd8 [ 2395.532290] ffff8802201ebfd8 0000000000004000 ffff880223598000 ffff88022229a230 [ 2395.532417] ffff8802201eba00 ffffffff81348f1c ffff8802201effff 0000000000000086 [ 2395.532543] Call Trace: [ 2395.532600] [] ? vsnprintf+0x35c/0x600 [ 2395.532663] [] ? put_dec+0x72/0x90 [ 2395.532726] [] ? sched_clock_cpu+0xc5/0x120 [ 2395.532792] [] schedule+0x24/0x70 [ 2395.532856] [] schedule_timeout+0x1a5/0x220 [ 2395.532922] [] ? lock_release_holdtime.part.9+0x1c/0x1a0 [ 2395.532991] [] wait_for_common+0xc8/0x160 [ 2395.533057] [] ? try_to_wake_up+0x300/0x300 [ 2395.533124] [] ? call_rcu_bh+0x20/0x20 [ 2395.533189] [] wait_for_completion+0x18/0x20 [ 2395.533256] [] wait_rcu_gp+0x48/0x50 [ 2395.533320] [] ? wait_rcu_gp+0x50/0x50 [ 2395.534390] [] ? wait_for_common+0x2b/0x160 [ 2395.534455] [] synchronize_sched+0x29/0x30 [ 2395.534519] [] raid5_remove_disk+0x118/0x190 [ 2395.534583] [] md_check_recovery+0x4d7/0x720 [ 2395.534649] [] raid5d+0x3e/0x6c0 [ 2395.534715] [] ? lock_release_holdtime.part.9+0x1c/0x1a0 [ 2395.534784] [] md_thread+0x10e/0x140 [ 2395.534849] [] ? __init_waitqueue_head+0x60/0x60 [ 2395.534915] [] ? md_register_thread+0x100/0x100 [ 2395.534980] [] kthread+0x9d/0xb0 [ 2395.535043] [] kernel_thread_helper+0x4/0x10 [ 2395.535108] [] ? __init_kthread_worker+0x70/0x70 [ 2395.535173] [] ? gs_change+0xb/0xb [ 2395.535240] 1 lock held by md1_raid6/4897: [ 2395.535298] #0: (&mddev->reconfig_mutex){......}, at: [] md_check_recovery+0xe4/0x720 [ 2395.535427] INFO: task btrfs-transacti:4919 blocked for more than 120 seconds. [ 2395.535543] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2395.535659] btrfs-transacti D 0000000000000002 0 4919 2 0x00000000 [ 2395.535730] ffff8802213d98c0 0000000000000046 ffff8802213d97d0 ffff8802213d9fd8 [ 2395.535857] ffff8802213d9fd8 0000000000004000 ffff880223592230 ffff88021eb5c460 [ 2395.535982] ffff8802213d9810 ffffffff81061ea5 ffff8802201ec2f0 0000000000000082 [ 2395.536104] Call Trace: [ 2395.536161] [] ? sched_clock_cpu+0xc5/0x120 [ 2395.536226] [] ? lock_release_holdtime.part.9+0x1c/0x1a0 [ 2395.536293] [] ? ktime_get_ts+0xa8/0xe0 [ 2395.536358] [] ? __lock_page+0x70/0x70 [ 2395.536422] [] schedule+0x24/0x70 [ 2395.536488] [] io_schedule+0x8a/0xd0 [ 2395.536552] [] sleep_on_page+0x9/0x10 [ 2395.536617] [] __wait_on_bit+0x57/0x80 [ 2395.536683] [] ? find_get_pages_tag+0x29/0x1c0 [ 2395.536750] [] wait_on_page_bit+0x70/0x80 [ 2395.536816] [] ? autoremove_wake_function+0x40/0x40 [ 2395.536882] [] filemap_fdatawait_range+0x104/0x190 [ 2395.536947] [] btrfs_wait_ordered_range+0x66/0x120 [ 2395.537011] [] __btrfs_write_out_cache+0x84b/0xa40 [ 2395.537077] [] btrfs_write_out_cache+0x93/0xf0 [ 2395.537143] [] ? _raw_spin_unlock+0x26/0x30 [ 2395.537209] [] btrfs_write_dirty_block_groups+0x503/0x5f0 [ 2395.537278] [] commit_cowonly_roots+0x10d/0x1de [ 2395.537346] [] ? btrfs_commit_transaction+0x5bf/0xa40 [ 2395.537414] [] btrfs_commit_transaction+0x61e/0xa40 [ 2395.537482] [] ? __init_waitqueue_head+0x60/0x60 [ 2395.537550] [] ? start_transaction+0x8b/0x2a0 [ 2395.537618] [] transaction_kthread+0x255/0x2d0 [ 2395.537685] [] ? cleaner_kthread+0x1c0/0x1c0 [ 2395.537749] [] kthread+0x9d/0xb0 [ 2395.537810] [] kernel_thread_helper+0x4/0x10 [ 2395.537874] [] ? __init_kthread_worker+0x70/0x70 [ 2395.537939] [] ? gs_change+0xb/0xb [ 2395.538002] 3 locks held by btrfs-transacti/4919: [ 2395.538064] #0: (&fs_info->transaction_kthread_mutex){......}, at: [] transaction_kthread+0xd7/0x2d0 [ 2395.538200] #1: (&fs_info->reloc_mutex){......}, at: [] btrfs_commit_transaction+0x541/0xa40 [ 2395.538337] #2: (&fs_info->tree_log_mutex){......}, at: [] btrfs_commit_transaction+0x5bf/0xa40 [ 2395.538475] INFO: task syslog-ng:6102 blocked for more than 120 seconds. [ 2395.538546] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2395.538662] syslog-ng D ffff8802226fb120 0 6102 5632 0x00000000 [ 2395.538732] ffff8801e7cc5a98 0000000000000046 ffff8801e7cc59b8 ffff8801e7cc5fd8 [ 2395.538855] ffff8801e7cc5fd8 0000000000004000 ffff88022359c460 ffff88021da40000 [ 2395.538979] ffff8801e7cc59d8 0000000000000092 ffff88021da40000 ffff88021da40000 [ 2395.539108] Call Trace: [ 2395.539168] [] ? lock_release_holdtime.part.9+0x1c/0x1a0 [ 2395.539235] [] schedule+0x24/0x70 [ 2395.539300] [] wait_current_trans.isra.31+0xa5/0xf0 [ 2395.539368] [] ? __init_waitqueue_head+0x60/0x60 [ 2395.539436] [] start_transaction+0x1e8/0x2a0 [ 2395.539503] [] btrfs_join_transaction+0x10/0x20 [ 2395.539569] [] btrfs_dirty_inode+0x50/0xd0 [ 2395.539632] [] btrfs_update_time+0x57/0x90 [ 2395.539697] [] ? current_fs_time+0x11/0x50 [ 2395.539761] [] update_time+0x20/0xc0 [ 2395.539826] [] ? mnt_clone_write+0xd/0x30 [ 2395.539891] [] file_update_time+0x94/0xf0 [ 2395.539958] [] btrfs_file_aio_write+0x1c0/0x4b0 [ 2395.540026] [] ? sched_clock_cpu+0xc5/0x120 [ 2395.540092] [] ? __lock_acquire.isra.27+0x29f/0xb30 [ 2395.540159] [] ? sched_clock_cpu+0xc5/0x120 [ 2395.540224] [] ? sched_clock_cpu+0xc5/0x120 [ 2395.540289] [] ? __btrfs_buffered_write+0x320/0x320 [ 2395.540356] [] do_sync_readv_writev+0xd3/0x110 [ 2395.540421] [] ? __lock_acquire.isra.27+0x29f/0xb30 [ 2395.540490] [] ? lock_release_holdtime.part.9+0x1c/0x1a0 [ 2395.540558] [] ? rw_copy_check_uvector+0x7a/0x130 [ 2395.540624] [] do_readv_writev+0xc8/0x1d0 [ 2395.540686] [] ? lock_release_holdtime.part.9+0x1c/0x1a0 [ 2395.540751] [] vfs_writev+0x3a/0x50 [ 2395.540812] [] sys_writev+0x48/0xc0 [ 2395.540874] [] system_call_fastpath+0x16/0x1b [ 2395.540936] 1 lock held by syslog-ng/6102: [ 2395.540994] #0: (&sb->s_type->i_mutex_key#9){......}, at: [] btrfs_file_aio_write+0xcf/0x4b0 [ 2395.544657] INFO: task flush-btrfs-1:5733 blocked for more than 120 seconds. [ 2395.544722] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2395.544836] flush-btrfs-1 D ffff8802226fb120 0 5733 2 0x00000000 [ 2395.544904] ffff88021b8356d0 0000000000000046 ffff88021b8355f0 ffff88021b835fd8 [ 2395.545028] ffff88021b835fd8 0000000000004000 ffff880223592230 ffff88021ecda230 [ 2395.545155] ffff88021b835610 0000000000000086 ffff88021ecda230 ffff88021ecda230 [ 2395.548997] Call Trace: [ 2395.549056] [] ? lock_release_holdtime.part.9+0x1c/0x1a0 [ 2395.549121] [] schedule+0x24/0x70 [ 2395.549182] [] wait_current_trans.isra.31+0xa5/0xf0 [ 2395.549248] [] ? __init_waitqueue_head+0x60/0x60 [ 2395.549313] [] start_transaction+0x1e8/0x2a0 [ 2395.549379] [] btrfs_join_transaction+0x10/0x20 [ 2395.549443] [] cow_file_range+0x7f/0x480 [ 2395.549508] [] ? lock_release_holdtime.part.9+0x1c/0x1a0 [ 2395.549578] [] ? _raw_spin_unlock+0x26/0x30 [ 2395.549644] [] run_delalloc_range+0x34c/0x370 [ 2395.549712] [] __extent_writepage+0x5d0/0x710 [ 2395.549778] [] ? find_get_pages_tag+0x11f/0x1c0 [ 2395.554072] [] ? find_get_pages_tag+0x29/0x1c0 [ 2395.554138] [] extent_write_cache_pages.isra.37.constprop.50+0x2ba/0x410 [ 2395.554254] [] ? writeback_sb_inodes+0x12e/0x380 [ 2395.554317] [] ? writeback_sb_inodes+0x12e/0x380 [ 2395.554383] [] extent_writepages+0x44/0x60 [ 2395.554447] [] ? btrfs_writepage+0x70/0x70 [ 2395.554512] [] btrfs_writepages+0x23/0x30 [ 2395.554579] [] do_writepages+0x1e/0x30 [ 2395.554645] [] __writeback_single_inode.isra.28+0x3e/0x120 [ 2395.554714] [] writeback_sb_inodes+0x22b/0x380 [ 2395.554782] [] __writeback_inodes_wb+0x96/0xc0 [ 2395.554851] [] wb_writeback+0x1cb/0x1e0 [ 2395.554914] [] wb_check_old_data_flush+0x97/0xa0 [ 2395.554977] [] wb_do_writeback+0xc8/0x140 [ 2395.555039] [] bdi_writeback_thread+0x72/0x150 [ 2395.555103] [] ? wb_do_writeback+0x140/0x140 [ 2395.555171] [] kthread+0x9d/0xb0 [ 2395.555232] [] kernel_thread_helper+0x4/0x10 [ 2395.555295] [] ? __init_kthread_worker+0x70/0x70 [ 2395.555358] [] ? gs_change+0xb/0xb [ 2395.555418] 1 lock held by flush-btrfs-1/5733: [ 2395.555476] #0: (&type->s_umount_key#26){......}, at: [] grab_super_passive+0x3f/0x90 [ 2395.555609] INFO: task systemd-udevd:6100 blocked for more than 120 seconds. [ 2395.555673] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2395.555784] systemd-udevd D 0000000000000002 0 6100 1 0x00000004 [ 2395.555852] ffff8801e7cdb958 0000000000000046 ffff8801e7cdb868 ffff8801e7cdbfd8 [ 2395.555971] ffff8801e7cdbfd8 0000000000004000 ffff8802235a8000 ffff88021da42230 [ 2395.556090] ffff8801e7cdb8a8 ffffffff81061ea5 ffff8802201ec2f0 0000000000000096 [ 2395.556209] Call Trace: [ 2395.556265] [] ? sched_clock_cpu+0xc5/0x120 [ 2395.556328] [] ? lock_release_holdtime.part.9+0x1c/0x1a0 [ 2395.556393] [] ? ktime_get_ts+0xa8/0xe0 [ 2395.556455] [] ? __lock_page+0x70/0x70 [ 2395.556522] [] schedule+0x24/0x70 [ 2395.556585] [] io_schedule+0x8a/0xd0 [ 2395.556647] [] sleep_on_page+0x9/0x10 [ 2395.556710] [