Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932808Ab3CEAB3 (ORCPT ); Mon, 4 Mar 2013 19:01:29 -0500 Received: from rydia.net ([69.46.88.68]:39806 "EHLO mail.rydia.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932182Ab3CEAB2 (ORCPT ); Mon, 4 Mar 2013 19:01:28 -0500 Date: Mon, 4 Mar 2013 16:01:31 -0800 (PST) From: dormando X-X-Sender: dormando@localhost6.localdomain6 To: linux-kernel@vger.kernel.org, linux-netdev@vger.kernel.org Subject: BUG: IPv4: Attempt to release TCP socket in state 1 Message-ID: User-Agent: Alpine 2.02 (DEB 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 23462 Lines: 424 Hi! I have a (core lockup?) with 3.7.6+ and 3.8.2 which appears to be under ixgbe. The machine appears to still be up but network stays in a severely hobbled state. Either lagging or not responding to the network at all. On a new box the hang happens within 8-24 hours of giving it production network traffic. On an older machine (6 cores instead of 8, etc) it can run for a week or more before hanging. The hang from 3.7 might be slightly different than 3.8. They seem to be mostly the same aside from 3.8 hanging in the GRO path. Don't see anything obvious in 3.9-rc1 that would fix it, and haven't tried 3.9-rc1. I've not yet figured out how to reproduce outside of production (as always, sigh). This doesn't seem to happen with 3.6.6, but we have different and less frequent kernel panics there. >From 3.7: [21934.669780] IPv4: Attempt to release TCP socket in state 1 ffff882785e3db00 [21969.265883] ------------[ cut here ]------------ [21969.265898] WARNING: at net/sched/sch_generic.c:255 dev_watchdog+0x258/0x270() [21969.265900] Hardware name: X9DR3-F [21969.265902] NETDEV WATCHDOG: eth2 (ixgbe): transmit queue 11 timed out [21969.265903] Modules linked in: macvlan bridge ipmi_watchdog ipmi_devintf coretemp ghash_clmulni_intel gpio_ich microcode ixgbe sb_edac mdio lpc_ich edac_core mei mfd_core ipmi_si ipmi_msghandler isci libsas igb [21969.265930] Pid: 0, comm: swapper/10 Not tainted 3.7.8 #1 [21969.265931] Call Trace: [21969.265933] [] warn_slowpath_common+0x7f/0xc0 [21969.265945] [] ? ip_local_deliver_finish+0xde/0x290 [21969.265948] [] warn_slowpath_fmt+0x46/0x50 [21969.265950] [] ? ip_rcv_finish+0x119/0x360 [21969.265953] [] dev_watchdog+0x258/0x270 [21969.265956] [] ? __netdev_watchdog_up+0x80/0x80 [21969.265960] [] call_timer_fn+0x49/0x130 [21969.265963] [] ? scheduler_tick+0x15f/0x190 [21969.265965] [] run_timer_softirq+0x224/0x290 [21969.265967] [] ? update_process_times+0x76/0x90 [21969.265969] [] ? __netdev_watchdog_up+0x80/0x80 [21969.265974] [] ? ktime_get+0x54/0xe0 [21969.265977] [] __do_softirq+0xc7/0x230 [21969.265990] [] call_softirq+0x1c/0x30 [21969.265995] [] do_softirq+0x55/0x90 [21969.265997] [] irq_exit+0x85/0xa0 [21969.265999] [] smp_apic_timer_interrupt+0x6e/0x99 [21969.266002] [] apic_timer_interrupt+0x6a/0x70 [21969.266003] [] ? __schedule+0x3aa/0x750 [21969.266011] [] ? mwait_idle+0xad/0x1f0 [21969.266013] [] cpu_idle+0xb3/0x100 [21969.266017] [] start_secondary+0x1c9/0x1d0 [21969.266019] ---[ end trace 0739ad788910e77e ]--- [21969.266059] ixgbe 0000:83:00.0 eth2: Reset adapter [22019.676899] INFO: rcu_sched self-detected stall on CPU { 30} (t=15001 jiffies) [22019.676963] Pid: 0, comm: swapper/30 Tainted: G W 3.7.8 #1 [22019.676966] Call Trace: [22019.676968] [] rcu_check_callbacks+0x1b4/0x600 [22019.676985] [] ? account_system_time+0xe8/0x1e0 [22019.676988] [] update_process_times+0x48/0x90 [22019.676993] [] tick_sched_timer+0x77/0x160 [22019.677006] [] __run_hrtimer+0x7d/0x1c0 [22019.677008] [] ? tick_setup_sched_timer+0x110/0x110 [22019.677010] [] hrtimer_interrupt+0xf6/0x230 [22019.677015] [] smp_apic_timer_interrupt+0x69/0x99 [22019.677018] [] apic_timer_interrupt+0x6a/0x70 [22019.677023] [] ? __inet_lookup_established+0xc0/0x280 [22019.677026] [] ? inet_del_protocol+0x40/0x40 [22019.677030] [] tcp_v4_early_demux+0xa3/0x170 [22019.677033] [] ip_rcv_finish+0x14d/0x360 [22019.677035] [] ip_rcv+0x226/0x310 [22019.677041] [] __netif_receive_skb+0x492/0x640 [22019.677043] [] ? __wake_up_common+0x59/0x90 [22019.677051] [] ? ixgbe_poll+0xe3b/0x1140 [ixgbe] [22019.677054] [] process_backlog+0xf4/0x1e0 [22019.677056] [] net_rx_action+0xf5/0x260 [22019.677070] [] __do_softirq+0xc7/0x230 [22019.677072] [] call_softirq+0x1c/0x30 [22019.677076] [] do_softirq+0x55/0x90 [22019.677078] [] irq_exit+0x85/0xa0 [22019.677080] [] do_IRQ+0x66/0xe0 [22019.677084] [] common_interrupt+0x6a/0x6a [22019.677085] [] ? __schedule+0x3aa/0x750 [22019.677090] [] ? mwait_idle+0xad/0x1f0 [22019.677092] [] cpu_idle+0xb3/0x100 [22019.677096] [] start_secondary+0x1c9/0x1d0 [22188.695704] INFO: task kworker/10:2:676 blocked for more than 120 seconds. [22188.695750] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [22188.695807] kworker/10:2 D ffffffff81806e40 0 676 2 0x00000000 [22188.695813] ffff882ff9dadad8 0000000000000046 ffff882ff9082d80 00000000000126c0 [22188.695816] ffff882ff9dadfd8 ffff882ff9dac010 00000000000126c0 00000000000126c0 [22188.695818] ffff882ff9dadfd8 00000000000126c0 ffff882ffb185b00 ffff882ff9082d80 [22188.695820] Call Trace: [22188.695830] [] schedule+0x29/0x70 [22188.695833] [] schedule_timeout+0x165/0x200 [22188.695838] [] ? ttwu_do_wakeup+0x45/0x100 [22188.695840] [] ? T.1871+0x49/0x60 [22188.695843] [] ? try_to_wake_up+0x23e/0x2b0 [22188.695845] [] wait_for_common+0xc8/0x160 [22188.695847] [] ? try_to_wake_up+0x2b0/0x2b0 [22188.695852] [] ? rcu_cpu_stall_reset+0x60/0x60 [22188.695854] [] wait_for_completion+0x1d/0x20 [22188.695859] [] __stop_cpus+0x56/0x80 [22188.695861] [] ? rcu_cpu_stall_reset+0x60/0x60 [22188.695864] [] try_stop_cpus+0x4d/0x80 [22188.695867] [] synchronize_sched_expedited+0x9a/0x120 [22188.695869] [] synchronize_rcu_expedited+0xe/0x10 [22188.695874] [] synchronize_net+0x25/0x30 [22188.695880] [] dev_deactivate_many+0x254/0x260 [22188.695882] [] dev_deactivate+0x2d/0x40 [22188.695886] [] linkwatch_do_dev+0x34/0x60 [22188.695888] [] __linkwatch_run_queue+0xf3/0x1e0 [22188.695891] [] linkwatch_event+0x25/0x30 [22188.695894] [] process_one_work+0x160/0x460 [22188.695896] [] ? __linkwatch_run_queue+0x1e0/0x1e0 [22188.695899] [] worker_thread+0x12b/0x3d0 [22188.695901] [] ? manage_workers+0x300/0x300 [22188.695904] [] kthread+0xce/0xe0 [22188.695907] [] ? kthread_freezable_should_stop+0x70/0x70 [22188.695911] [] ret_from_fork+0x7c/0xb0 [22188.695913] [] ? kthread_freezable_should_stop+0x70/0x70 [tons of processes hung in a similar way] Then every few hundred seconds swapper bails: [22919.239167] INFO: rcu_sched self-detected stall on CPU { 30} (t=240021 jiffies) [22919.239409] Pid: 0, comm: swapper/30 Tainted: G W 3.7.8 #1 [22919.239411] Call Trace: [22919.239413] [] rcu_check_callbacks+0x1b4/0x600 [22919.239430] [] ? account_system_time+0xe8/0x1e0 [22919.239434] [] update_process_times+0x48/0x90 [22919.239439] [] tick_sched_timer+0x77/0x160 [22919.239442] [] __run_hrtimer+0x7d/0x1c0 [22919.239445] [] ? tick_setup_sched_timer+0x110/0x110 [22919.239447] [] hrtimer_interrupt+0xf6/0x230 [22919.239453] [] smp_apic_timer_interrupt+0x69/0x99 [22919.239455] [] apic_timer_interrupt+0x6a/0x70 [22919.239461] [] ? __inet_lookup_established+0xcb/0x280 [22919.239463] [] ? inet_del_protocol+0x40/0x40 [22919.239468] [] tcp_v4_early_demux+0xa3/0x170 [22919.239470] [] ip_rcv_finish+0x14d/0x360 [22919.239472] [] ip_rcv+0x226/0x310 [22919.239478] [] __netif_receive_skb+0x492/0x640 [22919.239481] [] ? __wake_up_common+0x59/0x90 [22919.239490] [] ? ixgbe_poll+0xe3b/0x1140 [ixgbe] [22919.239493] [] process_backlog+0xf4/0x1e0 [22919.239495] [] net_rx_action+0xf5/0x260 [22919.239499] [] __do_softirq+0xc7/0x230 [22919.239501] [] call_softirq+0x1c/0x30 [22919.239505] [] do_softirq+0x55/0x90 [22919.239507] [] irq_exit+0x85/0xa0 [22919.239509] [] do_IRQ+0x66/0xe0 [22919.239513] [] common_interrupt+0x6a/0x6a [22919.239514] [] ? __schedule+0x3aa/0x750 [22919.239520] [] ? mwait_idle+0xad/0x1f0 [22919.239522] [] cpu_idle+0xb3/0x100 [22919.239526] [] start_secondary+0x1c9/0x1d0 [23099.151590] INFO: rcu_sched self-detected stall on CPU { 30} (t=285025 jiffies) [23099.151823] Pid: 0, comm: swapper/30 Tainted: G W [23099.151825] Call Trace: [23099.151827] [] rcu_check_callbacks+0x1b4/0x600 [23099.151841] [] ? account_system_time+0xe8/0x1e0 [23099.151845] [] update_process_times+0x48/0x90 [23099.151849] [] tick_sched_timer+0x77/0x160 [23099.151853] [] __run_hrtimer+0x7d/0x1c0 [23099.151856] [] ? tick_setup_sched_timer+0x110/0x110 [23099.151857] [] hrtimer_interrupt+0xf6/0x230 [23099.151863] [] smp_apic_timer_interrupt+0x69/0x99 [23099.151865] [] apic_timer_interrupt+0x6a/0x70 [23099.151870] [] ? __inet_lookup_established+0x173/0x280 [23099.151873] [] ? inet_del_protocol+0x40/0x40 [23099.151877] [] tcp_v4_early_demux+0xa3/0x170 [23099.151880] [] ip_rcv_finish+0x14d/0x360 [23099.151882] [] ip_rcv+0x226/0x310 [23099.151887] [] __netif_receive_skb+0x492/0x640 [23099.151890] [] ? __wake_up_common+0x59/0x90 [23099.151897] [] ? ixgbe_poll+0xe3b/0x1140 [ixgbe] [23099.151900] [] process_backlog+0xf4/0x1e0 [23099.151902] [] net_rx_action+0xf5/0x260 [23099.151906] [] __do_softirq+0xc7/0x230 [23099.151908] [] call_softirq+0x1c/0x30 [23099.151912] [] do_softirq+0x55/0x90 [23099.151914] [] irq_exit+0x85/0xa0 [23099.151916] [] do_IRQ+0x66/0xe0 [23099.151920] [] common_interrupt+0x6a/0x6a [23099.151920] [] ? __schedule+0x3aa/0x750 [23099.151926] [] ? mwait_idle+0xad/0x1f0 [23099.151928] [] cpu_idle+0xb3/0x100 [23099.151931] [] start_secondary+0x1c9/0x1d0 Under 3.8.2: [33486.326977] IPv4: Attempt to release TCP socket in state 1 ffff883269ea2300 [33486.342971] IPv4: Attempt to release TCP socket in state 1 ffff8835efccbf00 [33505.595925] ------------[ cut here ]------------ [33505.595934] WARNING: at net/sched/sch_generic.c:254 dev_watchdog+0x258/0x270() [33505.595935] Hardware name: X9DR3-F [33505.595937] NETDEV WATCHDOG: eth2 (ixgbe): transmit queue 0 timed out [33505.595938] Modules linked in: macvlan iptable_nat nf_nat_ipv4 nf_nat bridge coretemp ghash_clmulni_intel gpio_ich ixgbe microcode sb_edac mei lpc_ich edac_core mfd_core mdio isci libsas igb ptp pps_core [33505.595951] Pid: 0, comm: swapper/4 Not tainted 3.8.2 #2 [33505.595952] Call Trace: [33505.595954] [] warn_slowpath_common+0x7f/0xc0 [33505.595960] [] warn_slowpath_fmt+0x46/0x50 [33505.595962] [] dev_watchdog+0x258/0x270 [33505.595965] [] ? __netdev_watchdog_up+0x80/0x80 [33505.595968] [] call_timer_fn+0x49/0x130 [33505.595972] [] ? scheduler_tick+0x15f/0x190 [33505.595974] [] run_timer_softirq+0x224/0x290 [33505.595976] [] ? update_process_times+0x76/0x90 [33505.595978] [] ? __netdev_watchdog_up+0x80/0x80 [33505.595981] [] ? ktime_get+0x54/0xe0 [33505.595983] [] __do_softirq+0xc7/0x230 [33505.595987] [] call_softirq+0x1c/0x30 [33505.595990] [] do_softirq+0x55/0x90 [33505.595993] [] irq_exit+0x85/0xa0 [33505.595996] [] smp_apic_timer_interrupt+0x6e/0x99 [33505.596000] [] apic_timer_interrupt+0x6a/0x70 [33505.596002] [] ? __schedule+0x3ac/0x750 [33505.596009] [] ? mwait_idle+0xad/0x1f0 [33505.596011] [] cpu_idle+0xb3/0x100 [33505.596014] [] start_secondary+0x1d7/0x1de [33505.596015] ---[ end trace 3d817d7c7ae67386 ]--- [33505.596064] ixgbe 0000:83:00.0 eth2: Reset adapter [33556.011932] INFO: rcu_sched self-detected stall on CPU { 24} (t=15001 jiffies g=1985385 c=1985384 q=270786) [33556.011968] Pid: 0, comm: swapper/24 Tainted: G W 3.8.2 #2 [33556.011970] Call Trace: [33556.011972] [] rcu_check_callbacks+0x21e/0x7c0 [33556.011986] [] ? account_system_time+0xe8/0x1e0 [33556.011992] [] update_process_times+0x48/0x90 [33556.011996] [] tick_sched_timer+0x56/0x130 [33556.012000] [] __run_hrtimer+0x7d/0x1c0 [33556.012002] [] ? tick_setup_sched_timer+0x110/0x110 [33556.012004] [] hrtimer_interrupt+0xf6/0x230 [33556.012010] [] smp_apic_timer_interrupt+0x69/0x99 [33556.012013] [] apic_timer_interrupt+0x6a/0x70 [33556.012017] [] ? __inet_lookup_established+0xcb/0x2d0 [33556.012020] [] ? inet_del_protocol+0x40/0x40 [33556.012024] [] tcp_v4_early_demux+0xac/0x170 [33556.012025] [] ip_rcv_finish+0x14d/0x360 [33556.012027] [] ip_rcv+0x226/0x310 [33556.012032] [] __netif_receive_skb+0x492/0x640 [33556.012034] [] netif_receive_skb+0x2d/0x90 [33556.012036] [] ? tcp4_gro_receive+0xb0/0x130 [33556.012038] [] napi_gro_complete+0x95/0xe0 [33556.012040] [] dev_gro_receive+0x2b6/0x3b0 [33556.012043] [] napi_gro_receive+0x5b/0x130 [33556.012051] [] ixgbe_poll+0x54a/0x1180 [ixgbe] [33556.012054] [] ? enqueue_task+0x6a/0x80 [33556.012056] [] net_rx_action+0xf5/0x260 [33556.012058] [] __do_softirq+0xc7/0x230 [33556.012061] [] call_softirq+0x1c/0x30 [33556.012064] [] do_softirq+0x55/0x90 [33556.012066] [] irq_exit+0x85/0xa0 [33556.012068] [] do_IRQ+0x66/0xe0 [33556.012071] [] common_interrupt+0x6a/0x6a [33556.012073] [] ? __schedule+0x3ac/0x750 [33556.012078] [] ? mwait_idle+0xad/0x1f0 [33556.012080] [] cpu_idle+0xb3/0x100 [33556.012082] [] start_secondary+0x1d7/0x1de [33716.090584] INFO: task kworker/4:2:882 blocked for more than 120 seconds. [33716.090602] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [33716.090618] kworker/4:2 D ffffffff81807160 0 882 2 0x00000000 [33716.090622] ffff881fd2547ad8 0000000000000046 ffff881fd0ac2dc0 0000000000012700 [33716.090624] ffff881fd2547fd8 ffff881fd2546010 0000000000012700 0000000000012700 [33716.090626] ffff881fd2547fd8 0000000000012700 ffff881fd3655b80 ffff881fd0ac2dc0 [33716.090628] Call Trace: [33716.090639] [] schedule+0x29/0x70 [33716.090642] [] schedule_timeout+0x165/0x200 [33716.090647] [] ? physflat_send_IPI_mask+0xe/0x10 [33716.090650] [] ? try_to_wake_up+0x23e/0x2b0 [33716.090653] [] wait_for_common+0xc8/0x160 [33716.090654] [] ? try_to_wake_up+0x2b0/0x2b0 [33716.090660] [] ? rcu_cpu_stall_reset+0x60/0x60 [33716.090662] [] wait_for_completion+0x1d/0x20 [33716.090665] [] __stop_cpus+0x56/0x80 [33716.090667] [] ? rcu_cpu_stall_reset+0x60/0x60 [33716.090669] [] try_stop_cpus+0x4d/0x80 [33716.090672] [] synchronize_sched_expedited+0xfb/0x1d0 [33716.090674] [] synchronize_rcu_expedited+0xe/0x10 [33716.090678] [] synchronize_net+0x25/0x30 [33716.090683] [] dev_deactivate_many+0x254/0x260 [33716.090685] [] dev_deactivate+0x2d/0x40 [33716.090688] [] linkwatch_do_dev+0x34/0x60 [33716.090690] [] __linkwatch_run_queue+0xf3/0x1e0 [33716.090692] [] linkwatch_event+0x25/0x30 [33716.090696] [] process_one_work+0x168/0x450 [33716.090699] [] worker_thread+0x12b/0x3d0 [33716.090702] [] ? manage_workers+0x300/0x300 [33716.090704] [] kthread+0xce/0xe0 [33716.090706] [] ? kthread_freezable_should_stop+0x70/0x70 [33716.090709] [] ret_from_fork+0x7c/0xb0 [33716.090711] [] ? kthread_freezable_should_stop+0x70/0x70 [more hung processes bailing] [37335.739761] INFO: rcu_sched self-detected stall on CPU { 24} (t=960083 jiffies g=1985385 c=1985384 q=19390495) [37335.739828] Pid: 0, comm: swapper/24 Tainted: G W 3.8.2 #2 [37335.739830] Call Trace: [37335.739832] [] rcu_check_callbacks+0x21e/0x7c0 [37335.739847] [] ? account_system_time+0xe8/0x1e0 [37335.739853] [] update_process_times+0x48/0x90 [37335.739857] [] tick_sched_timer+0x56/0x130 [37335.739860] [] __run_hrtimer+0x7d/0x1c0 [37335.739863] [] ? tick_setup_sched_timer+0x110/0x110 [37335.739865] [] hrtimer_interrupt+0xf6/0x230 [37335.739871] [] smp_apic_timer_interrupt+0x69/0x99 [37335.739874] [] apic_timer_interrupt+0x6a/0x70 [37335.739878] [] ? __inet_lookup_established+0xcf/0x2d0 [37335.739880] [] ? inet_del_protocol+0x40/0x40 [37335.739884] [] tcp_v4_early_demux+0xac/0x170 [37335.739886] [] ip_rcv_finish+0x14d/0x360 [37335.739888] [] ip_rcv+0x226/0x310 [37335.739892] [] __netif_receive_skb+0x492/0x640 [37335.739895] [] netif_receive_skb+0x2d/0x90 [37335.739897] [] ? tcp4_gro_receive+0xb0/0x130 [37335.739899] [] napi_gro_complete+0x95/0xe0 [37335.739901] [] dev_gro_receive+0x2b6/0x3b0 [37335.739903] [] napi_gro_receive+0x5b/0x130 [37335.739911] [] ixgbe_poll+0x54a/0x1180 [ixgbe] [37335.739915] [] ? enqueue_task+0x6a/0x80 [37335.739917] [] net_rx_action+0xf5/0x260 [37335.739919] [] __do_softirq+0xc7/0x230 [37335.739922] [] call_softirq+0x1c/0x30 [37335.739927] [] do_softirq+0x55/0x90 [37335.739928] [] irq_exit+0x85/0xa0 [37335.739931] [] do_IRQ+0x66/0xe0 [37335.739937] [] common_interrupt+0x6a/0x6a [37335.739938] [] ? __schedule+0x3ac/0x750 [37335.739943] [] ? mwait_idle+0xad/0x1f0 [37335.739945] [] cpu_idle+0xb3/0x100 [37335.739948] [] start_secondary+0x1d7/0x1de [37515.727179] INFO: rcu_sched self-detected stall on CPU { 24} (t=1005087 jiffies g=1985385 c=1985384 q=20855557) [37515.727246] Pid: 0, comm: swapper/24 Tainted: G W 3.8.2 #2 [37515.727249] Call Trace: [37515.727251] [] rcu_check_callbacks+0x21e/0x7c0 [37515.727265] [] ? account_system_time+0xe8/0x1e0 [37515.727271] [] update_process_times+0x48/0x90 [37515.727275] [] tick_sched_timer+0x56/0x130 [37515.727279] [] __run_hrtimer+0x7d/0x1c0 [37515.727281] [] ? tick_setup_sched_timer+0x110/0x110 [37515.727283] [] hrtimer_interrupt+0xf6/0x230 [37515.727289] [] smp_apic_timer_interrupt+0x69/0x99 [37515.727292] [] apic_timer_interrupt+0x6a/0x70 [37515.727296] [] ? __inet_lookup_established+0xcb/0x2d0 [37515.727298] [] ? inet_del_protocol+0x40/0x40 [37515.727302] [] tcp_v4_early_demux+0xac/0x170 [37515.727304] [] ip_rcv_finish+0x14d/0x360 [37515.727306] [] ip_rcv+0x226/0x310 [37515.727310] [] __netif_receive_skb+0x492/0x640 [37515.727312] [] netif_receive_skb+0x2d/0x90 [37515.727315] [] ? tcp4_gro_receive+0xb0/0x130 [37515.727317] [] napi_gro_complete+0x95/0xe0 [37515.727319] [] dev_gro_receive+0x2b6/0x3b0 [37515.727322] [] napi_gro_receive+0x5b/0x130 [37515.727330] [] ixgbe_poll+0x54a/0x1180 [ixgbe] [37515.727334] [] ? enqueue_task+0x6a/0x80 [37515.727336] [] net_rx_action+0xf5/0x260 [37515.727338] [] __do_softirq+0xc7/0x230 [37515.727341] [] call_softirq+0x1c/0x30 [37515.727345] [] do_softirq+0x55/0x90 [37515.727346] [] irq_exit+0x85/0xa0 [37515.727349] [] do_IRQ+0x66/0xe0 [37515.727354] [] common_interrupt+0x6a/0x6a [37515.727355] [] ? __schedule+0x3ac/0x750 [37515.727360] [] ? mwait_idle+0xad/0x1f0 [37515.727362] [] cpu_idle+0xb3/0x100 [37515.727365] [] start_secondary+0x1d7/0x1de ... then swapped just does this until someone reboots the box. Apologies for the ugly paste. Thanks, -Dormando -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/