Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757679AbZIQPr2 (ORCPT ); Thu, 17 Sep 2009 11:47:28 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1755406AbZIQPr1 (ORCPT ); Thu, 17 Sep 2009 11:47:27 -0400 Received: from alpha.arachsys.com ([91.203.57.7]:58936 "EHLO alpha.arachsys.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756590AbZIQPr0 (ORCPT ); Thu, 17 Sep 2009 11:47:26 -0400 Date: Thu, 17 Sep 2009 16:47:24 +0100 From: Chris Webb To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Transmit timeouts with forcedeth card in 2.6.30.4 Message-ID: <20090917154724.GC16620@arachsys.com> References: <4ab017c4.uBg+e5AMK7RN0BGz%hostmaster@elastichosts.com> <20090915224925.GA4630@alpha.arachsys.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20090915224925.GA4630@alpha.arachsys.com> User-Agent: Mutt/1.5.20 (2009-06-14) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 13155 Lines: 175 Hi. I'm occasionally seeing the following transmit time-out from a forcedeth ethernet interface with linux 2.6.30.4. We have a cluster of identical machines with Supermicro H8DMT motherboards on which there are two such interfaces integrated, and these errors appear on random machines across the cluster, about once every two or three weeks per machine. Following this error, it is impossible to ping in or out, but it is restored back to life by a simple ip link set eth0 down ip link set eth0 up The eth0 is enslaved to a bridge and has the mtu set at the default 1500. A second forcedeth nic in the same machine with mtu set at 9000 and not joined to bridge doesn't seem to suffer from this problem, although it may have much lower traffic in any case. forcedeth: Reverse Engineered nForce ethernet driver. Version 0.64. forcedeth 0000:00:08.0: setting latency timer to 64 nv_probe: set workaround bit for reversed mac addr forcedeth 0000:00:08.0: ifname eth0, PHY OUI 0x5043 @ 2, addr 00:30:48:67:7d:7c forcedeth 0000:00:08.0: highdma csum vlan pwrctl mgmt gbit lnktim msi desc-v3 [...] WARNING: at net/sched/sch_generic.c:226 dev_watchdog+0x130/0x1da() Hardware name: H8DMT NETDEV WATCHDOG: eth0 (forcedeth): transmit timed out Modules linked in: Pid: 0, comm: swapper Tainted: G M 2.6.30.4-elastic-lon-b #2 Call Trace: [] warn_slowpath_common+0x77/0x8f [] warn_slowpath_fmt+0x3c/0x3e [] ? lock_timer_base+0x27/0x4d [] dev_watchdog+0x130/0x1da [] ? ptrace_detach+0x86/0x95 [] run_timer_softirq+0x164/0x1dc [] ? dev_watchdog+0x0/0x1da [] __do_softirq+0xdc/0x1e3 [] call_softirq+0x1c/0x28 [] do_softirq+0x34/0x72 [] irq_exit+0x3f/0x8b [] smp_apic_timer_interrupt+0x89/0x97 [] apic_timer_interrupt+0x13/0x20 [] ? default_idle+0x9d/0x118 [] ? c1e_idle+0xe3/0xea [] ? atomic_notifier_call_chain+0xf/0x11 [] ? cpu_idle+0x52/0xb6 [] ? start_secondary+0x1a4/0x1a8 ---[ end trace 97f305223326b99b ]--- eth0: Got tx_timeout. irq: 00000036 eth0: Ring at 42596c000 eth0: Dumping tx registers 0: 00002036 00000000 00000003 0006000d 00000000 00000000 00000000 00000000 20: 00000000 00000000 00000000 00000000 00000001 00000000 00000000 00000000 40: 0420e20e 0000a855 00002e20 00000000 00000000 00000000 00000000 00000000 60: 00000000 00000000 00000000 0000ffff 0000ffff 0000ffff 0000ffff 00000000 80: 003b0f3c 00040001 00000000 007f0080 0000061c 00000001 00200000 80007f30 a0: 0016070f 00000016 67483000 00007c7d 00000001 00000000 00000000 00000000 c0: 10000001 00000001 00000001 00000001 00000001 00000001 00000001 00000001 e0: 00000001 00000001 00000001 00000001 00000001 00000001 00000001 00000001 100: 2596e000 2596c000 01ff00ff 00008000 00010064 00000000 0000004f 2596e150 120: 2596cb50 204308c0 a000fff2 00000000 00000000 2596e15c 2596cb50 01e08000 140: 00304120 80c02600 00000004 00000004 00000008 00000000 00000000 00000000 160: 00000004 00000004 00000004 00000004 01ff0080 0000c000 00000000 00000000 180: 00000006 00000008 0294796d 00008103 0000004a 00004000 00000591 0000c183 1a0: 00000006 00000008 0294796d 00008103 0000004a 00004000 0000059e 0000c583 1c0: 00000006 00000008 0294796d 00008103 0000004a 00004000 00000596 0000c583 1e0: 00000006 00000008 0294796d 00008103 0000004a 00004000 0000059b 0000c583 200: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 220: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 240: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 260: 00000000 00000000 fe027001 00000100 00000011 000000a3 fe027011 000001a3 280: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 2a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 2c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 2e0: 00000000 00000000 00000000 00000000 00000000 00000001 00000001 00000001 300: 80212000 00000000 00000000 00000000 00000000 00002000 00000000 00000000 320: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 340: 00000000 00000000 00000000 00000000 00000000 00000020 d104438f 00000000 360: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 380: 00000000 00000000 00000000 00000000 00000000 00000000 00000002 00000000 3a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 3c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 3e0: 06255300 00701365 00000000 00000000 00000032 00000000 00000000 00000000 400: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 420: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 440: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 460: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 480: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 4a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 4c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 4e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 500: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 520: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 540: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 560: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 580: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 5a0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 5c0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 5e0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 600: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 eth0: Dumping tx ring 000: 00000005 d4ea5802 20000029 // 00000006 92398802 20000029 // 00000004 423a0c02 20000029 // 00000 004: 00000005 dc2bf422 20000040 // 00000005 dc2b8422 20000062 // 00000005 dc2b8c22 2000005c // 00000 008: 00000008 20437c22 20000095 // 00000008 20432022 20000095 // 00000008 20435c22 20000095 // 00000 00c: 00000006 0aa3d822 2000006e // 00000004 fa8d3022 200005c6 // 00000005 7806f022 2000059e // 00000 010: 00000008 20435022 20000095 // 00000008 20437022 20000095 // 00000008 20435822 20000095 // 00000 014: 00000008 20430822 20000095 // 00000006 0aa38822 20000069 // 00000005 61eb3022 200005e9 // 00000 018: 00000005 7806d822 20000599 // 00000008 20436c22 20000090 // 00000008 20432c22 20000090 // 00000 01c: 00000008 20436822 20000090 // 00000005 7806f822 200005b7 // 00000008 206c8c22 2000003c // 00000 020: 00000008 206cc822 20000090 // 00000008 206ce022 20000090 // 00000008 206cc422 20000090 // 00000 024: 00000008 206c9022 20000090 // 00000008 206c8422 20000090 // 00000008 206cb422 2000003d // 00000 028: 00000005 0c077022 200005d5 // 00000005 7806a822 20000599 // 00000008 206ca822 20000090 // 00000 02c: 00000008 206cbc22 20000090 // 00000008 206c9c22 20000090 // 00000006 0aa3ac02 20000029 // 00000 030: 00000006 0aa3dc22 20000090 // 00000006 0aa3cc22 20000090 // 00000006 0aa3d422 20000090 // 00000 034: 00000006 0aa3bc22 20000090 // 00000006 0aa3a022 20000090 // 00000006 0aa39402 20000029 // 00000 038: 00000001 c34fe402 20000029 // 00000004 3683e022 200005e1 // 00000006 0aa3e422 200000a0 // 00000 03c: 00000006 0aa3c422 20000090 // 00000006 0aa3f822 20000090 // 00000006 0aa3b422 20000090 // 00000 040: 00000001 c34fdc22 20000069 // 00000001 c34fd402 20000029 // 00000008 25736022 200000a0 // 00000 044: 00000008 25731022 20000090 // 00000008 25734422 20000090 // 00000008 25735c22 20000090 // 00000 048: 00000008 25732422 20000090 // 00000005 dc2b9402 2c000059 // 00000008 25735002 20000029 // 00000 04c: 00000006 000fbc02 2c000059 // 00000008 25abc002 20000029 // 00000008 25abb002 20000029 // 00000 050: 00000005 dc2bd002 20000029 // 00000008 25ab8c02 20000029 // 00000006 249f5402 20000029 // 00000 054: 00000008 25abfc02 20000029 // 00000006 249f5802 20000029 // 00000005 dc2ba002 20000029 // 00000 058: 00000006 249f0402 20000029 // 00000008 254de402 20000029 // 00000008 254df802 20000029 // 00000 05c: 00000005 dc2ba822 20000041 // 00000006 25091c02 20000029 // 00000006 25096422 20000041 // 00000 060: 00000005 dc2b9002 20000029 // 00000006 25091402 20000029 // 00000006 249f6c22 20000041 // 00000 064: 00000008 2022b002 20000029 // 00000006 25090c02 20000029 // 00000005 e597f802 20000029 // 00000 068: 00000006 25093402 20000029 // 00000006 25093002 20000029 // 00000006 000f9802 20000029 // 00000 06c: 00000006 249f2422 2000003f // 00000006 000fcc02 20000029 // 00000006 249f1802 20000029 // 00000 070: 00000005 dc2ba402 20000029 // 00000005 dc2bc002 20000029 // 00000005 e597f402 20000029 // 00000 074: 00000008 206cd802 20000029 // 00000005 e580d402 20000029 // 00000006 000ff002 20000029 // 00000 078: 00000006 249f2c02 20000029 // 00000006 249f7802 20000029 // 00000008 2022c802 20000029 // 00000 07c: 00000006 000f9c02 20000029 // 00000006 249f3802 20000029 // 00000005 dc2bec02 20000029 // 00000 080: 00000005 e597b802 20000029 // 00000003 9a19cc02 20000029 // 00000003 9a19f802 20000029 // 00000 084: 00000004 25992c02 20000029 // 00000006 249f2802 20000029 // 00000004 25994c02 20000029 // 00000 088: 00000006 000f8402 20000029 // 00000005 dc2b9c02 20000029 // 00000006 25091002 20000029 // 00000 08c: 00000006 000f9422 2000003f // 00000006 000fa822 2000003f // 00000006 9239ac02 20000029 // 00000 090: 00000006 0aa38402 20000029 // 00000001 c34fc002 20000029 // 00000008 2022ac02 20000029 // 00000 094: 00000001 c34f9802 20000029 // 00000008 20433c02 20000029 // 00000005 e580d802 20000029 // 00000 098: 00000005 e5978002 20000029 // 00000001 c34fbc22 2000003f // 00000001 c34fcc02 20000029 // 00000 09c: 00000006 25096802 20000029 // 00000001 c34ff802 20000029 // 00000005 e580f402 20000029 // 00000 0a0: 00000005 e580b802 20000029 // 00000006 000fe802 20000029 // 00000001 c34fa402 20000029 // 00000 0a4: 00000005 e580b402 20000029 // 00000001 c34ffc02 20000029 // 00000001 c34f9002 20000029 // 00000 0a8: 00000001 c34fe002 20000029 // 00000005 dc2b9802 20000029 // 00000001 c34ff002 20000029 // 00000 0ac: 00000006 9239dc02 20000029 // 00000001 c34fe802 20000029 // 00000006 9239c802 20000029 // 00000 0b0: 00000001 c34f9402 20000029 // 00000006 9239d002 20000029 // 00000005 dc2bc402 20000029 // 00000 0b4: 00000006 249f1c02 20000029 // 00000006 000fc402 20000029 // 00000001 c34fd802 20000029 // 00000 0b8: 00000006 249f5c02 20000029 // 00000001 c34f8002 20000029 // 00000006 92398c02 20000029 // 00000 0bc: 00000001 c34f8402 20000029 // 00000001 c34ff402 20000029 // 00000008 206cb802 20000029 // 00000 0c0: 00000005 e5978c02 20000029 // 00000005 e5978802 20000029 // 00000001 c34fc402 20000029 // 00000 0c4: 00000005 e597e002 20000029 // 00000003 9a199c02 20000029 // 00000005 dc2bb402 20000029 // 00000 0c8: 00000005 e597fc02 20000029 // 00000003 9a19d002 20000029 // 00000005 e597d402 20000029 // 00000 0cc: 00000005 e580fc02 20000029 // 00000003 9a198c02 20000029 // 00000005 e580a402 20000029 // 00000 0d0: 00000003 9a19b002 20000029 // 00000004 25652002 20000029 // 00000004 25656002 20000029 // 00000 0d4: 00000005 e580dc02 20000029 // 00000004 25650802 20000029 // 00000005 e580ec02 20000029 // 00000 0d8: 00000006 000fe402 20000029 // 00000004 23942802 20000029 // 00000004 25997c02 20000029 // 00000 0dc: 00000004 25994802 20000029 // 00000004 25995802 20000029 // 00000006 92399802 20000029 // 00000 0e0: 00000006 92398002 20000029 // 00000004 23940802 20000029 // 00000006 9239d402 20000029 // 00000 0e4: 00000004 25993802 20000029 // 00000004 25995c02 20000029 // 00000004 25996002 20000029 // 00000 0e8: 00000004 25997402 20000029 // 00000003 9a19dc02 20000029 // 00000004 25917c02 20000029 // 00000 0ec: 00000004 25914c02 20000029 // 00000004 25914802 20000029 // 00000004 25911402 20000029 // 00000 0f0: 00000006 9239b402 20000029 // 00000001 c34fac02 20000029 // 00000006 92399002 20000029 // 00000 0f4: 00000006 92398402 20000029 // 00000001 c34fb002 20000029 // 00000005 d4ea4802 20000029 // 00000 0f8: 00000003 f5818c02 20000029 // 00000008 206ccc02 20000029 // 00000003 f581ec02 20000029 // 00000 0fc: 00000008 206cd402 20000029 // 00000003 f581c002 20000029 // 00000006 9239f802 20000029 // 00000 Best wishes, Chris. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/