Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932430AbaAaN3L (ORCPT ); Fri, 31 Jan 2014 08:29:11 -0500 Received: from smtp02.citrix.com ([66.165.176.63]:59969 "EHLO SMTP02.CITRIX.COM" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932249AbaAaN3J (ORCPT ); Fri, 31 Jan 2014 08:29:09 -0500 X-IronPort-AV: E=Sophos;i="4.95,757,1384300800"; d="scan'208";a="96519208" Message-ID: <52EBA51E.808@citrix.com> Date: Fri, 31 Jan 2014 14:29:02 +0100 From: Zoltan Kiss User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Michael Chan , Zoltan Kiss CC: Jeff Kirsher , Jesse Brandeburg , Bruce Allan , Carolyn Wyborny , Don Skidmore , Greg Rose , Peter P Waskiewicz Jr , Alex Duyck , John Ronciak , Tushar Dave , Akeem G Abodunrin , "David S. Miller" , , "netdev@vger.kernel.org" , , "xen-devel@lists.xenproject.org" Subject: Re: igb and bnx2: "NETDEV WATCHDOG: transmit queue timed out" when skb has huge linear buffer References: <52EAA31B.1090606@schaman.hu> <1391114048.4804.2.camel@LTIRV-MCHAN1.corp.ad.broadcom.com> In-Reply-To: <1391114048.4804.2.camel@LTIRV-MCHAN1.corp.ad.broadcom.com> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [10.68.14.36] X-DLP: MIA2 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 30/01/14 21:34, Michael Chan wrote: > On Thu, 2014-01-30 at 19:08 +0000, Zoltan Kiss wrote: >> I've experienced some queue timeout problems mentioned in the subject >> with igb and bnx2 cards. > Please provide the full tx timeout dmesg. bnx2 dumps some diagnostic > information during tx timeout that may be useful. Thanks. Hi, Here is some: [ 5417.275463] ------------[ cut here ]------------ [ 5417.275472] WARNING: at net/sched/sch_generic.c:255 dev_watchdog+0x156/0x1f0() [ 5417.275474] NETDEV WATCHDOG: eth1 (bnx2): transmit queue 2 timed out [ 5417.275476] Modules linked in: tun nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss oid_registry nfsv4 nfs fscache lockd sunrpc ipv6 openvswitch(O) ipt_REJECT nf_conntrack_ipv rack xt_tcpudp iptable_filter ip_tables x_tables nls_utf8 isofs dm_multipath scsi_dh dm_mod dcdbas coretemp microcode psmouse serio_raw lpc_ich mfd_core hid_generic ehci_p sg hed bnx2 usbhid hid sr_mod cdrom sd_mod pata_acpi ata_generic ata_piix libata uhci_hcd mptsas mptscsih mptbase scsi_transport_sas scsi_mod [ 5417.275517] CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G O 3.10.11-0.xs1.8.50.170.377582 #1 [ 5417.275518] Hardware name: Dell Inc. PowerEdge R710/00W9X3, BIOS 1.2.6 07/17/2009 [ 5417.275520] 000000ff f008be08 c1488c53 f008be30 c1046664 c1658a88 f008be5c 000000ff [ 5417.275525] c13fc146 c13fc146 ee96a000 00000002 00137d44 f008be48 c1046723 00000009 [ 5417.275530] f008be40 c1658a88 f008be5c f008be80 c13fc146 c16556e1 000000ff c1658a88 [ 5417.275535] Call Trace: [ 5417.275539] [] dump_stack+0x16/0x1b [ 5417.275544] [] warn_slowpath_common+0x64/0x80 [ 5417.275546] [] ? dev_watchdog+0x156/0x1f0 [ 5417.275549] [] ? dev_watchdog+0x156/0x1f0 [ 5417.275551] [] warn_slowpath_fmt+0x33/0x40 [ 5417.275554] [] dev_watchdog+0x156/0x1f0 [ 5417.275559] [] call_timer_fn+0x3e/0xf0 [ 5417.275563] [] ? finish_task_switch+0x4e/0xb0 [ 5417.275565] [] ? __netdev_watchdog_up+0x60/0x60 [ 5417.275568] [] run_timer_softirq+0x1ab/0x210 [ 5417.275571] [] ? __netdev_watchdog_up+0x60/0x60 [ 5417.275574] [] __do_softirq+0xc4/0x200 [ 5417.275577] [] ? xen_do_upcall+0x7/0xc [ 5417.275579] [] run_ksoftirqd+0x20/0x50 [ 5417.275582] [] smpboot_thread_fn+0x142/0x150 [ 5417.275586] [] kthread+0x9b/0xa0 [ 5417.275589] [] ? smpboot_create_threads+0x60/0x60 [ 5417.275591] [] ? cpu_rt_runtime_read+0x40/0x80 [ 5417.275594] [] ret_from_kernel_thread+0x1b/0x28 [ 5417.275596] [] ? kthread_freezable_should_stop+0x60/0x60 [ 5417.275599] ---[ end trace 691f572d388226ca ]--- [ 5417.275602] bnx2 0000:01:00.1 eth1: <--- start FTQ dump ---> [ 5417.275622] bnx2 0000:01:00.1 eth1: RV2P_PFTQ_CTL 00010000 [ 5417.275629] bnx2 0000:01:00.1 eth1: RV2P_TFTQ_CTL 00020000 [ 5417.275636] bnx2 0000:01:00.1 eth1: RV2P_MFTQ_CTL 00004000 [ 5417.275643] bnx2 0000:01:00.1 eth1: TBDR_FTQ_CTL 00004002 [ 5417.275650] bnx2 0000:01:00.1 eth1: TDMA_FTQ_CTL 00010002 [ 5417.275657] bnx2 0000:01:00.1 eth1: TXP_FTQ_CTL 00010000 [ 5417.275663] bnx2 0000:01:00.1 eth1: TXP_FTQ_CTL 00010000 [ 5417.275670] bnx2 0000:01:00.1 eth1: TPAT_FTQ_CTL 00010000 [ 5417.275677] bnx2 0000:01:00.1 eth1: RXP_CFTQ_CTL 00008000 [ 5417.275684] bnx2 0000:01:00.1 eth1: RXP_FTQ_CTL 00100000 [ 5417.275690] bnx2 0000:01:00.1 eth1: COM_COMXQ_FTQ_CTL 00010000 [ 5417.275698] bnx2 0000:01:00.1 eth1: COM_COMTQ_FTQ_CTL 00020000 [ 5417.275705] bnx2 0000:01:00.1 eth1: COM_COMQ_FTQ_CTL 00010000 [ 5417.275712] bnx2 0000:01:00.1 eth1: CP_CPQ_FTQ_CTL 00004000 [ 5417.275718] bnx2 0000:01:00.1 eth1: CPU states: [ 5417.275730] bnx2 0000:01:00.1 eth1: 045000 mode b84c state 80001000 evt_mask 500 pc 8001284 pc 8001284 instr 1440fffc [ 5417.275746] bnx2 0000:01:00.1 eth1: 085000 mode b84c state 80005000 evt_mask 500 pc 8000a54 pc 8000a5c instr 10400016 [ 5417.275785] bnx2 0000:01:00.1 eth1: 0c5000 mode b84c state 80001000 evt_mask 500 pc 8004c20 pc 8004c20 instr 32050003 [ 5417.275801] bnx2 0000:01:00.1 eth1: 105000 mode b8cc state 80000000 evt_mask 500 pc 8000a8c pc 8000a94 instr 8c420020 [ 5417.275817] bnx2 0000:01:00.1 eth1: 145000 mode b880 state 80000000 evt_mask 500 pc 8000ab0 pc 800d1e8 instr 27bd0020 [ 5417.275834] bnx2 0000:01:00.1 eth1: 185000 mode b8cc state 80000000 evt_mask 500 pc 8000cb0 pc 8000930 instr 8ce800e8 [ 5417.275845] bnx2 0000:01:00.1 eth1: <--- end FTQ dump ---> [ 5417.275851] bnx2 0000:01:00.1 eth1: <--- start TBDC dump ---> [ 5417.275858] bnx2 0000:01:00.1 eth1: TBDC free cnt: 32 [ 5417.275864] bnx2 0000:01:00.1 eth1: LINE CID BIDX CMD VALIDS [ 5417.275875] bnx2 0000:01:00.1 eth1: 00 001080 17c8 00 [0] [ 5417.275886] bnx2 0000:01:00.1 eth1: 01 001080 17e0 00 [0] [ 5417.275897] bnx2 0000:01:00.1 eth1: 02 001080 17e8 00 [0] [ 5417.275907] bnx2 0000:01:00.1 eth1: 03 001080 17f8 00 [0] [ 5417.275918] bnx2 0000:01:00.1 eth1: 04 001080 1800 00 [0] [ 5417.275929] bnx2 0000:01:00.1 eth1: 05 001080 17d0 00 [0] [ 5417.275940] bnx2 0000:01:00.1 eth1: 06 001080 17d8 00 [0] [ 5417.275951] bnx2 0000:01:00.1 eth1: 07 001080 17f0 00 [0] [ 5417.275961] bnx2 0000:01:00.1 eth1: 08 001080 1620 00 [0] [ 5417.275972] bnx2 0000:01:00.1 eth1: 09 17de00 fbf8 78 [0] [ 5417.275983] bnx2 0000:01:00.1 eth1: 0a 1bbf80 fef8 9f [0] [ 5417.275994] bnx2 0000:01:00.1 eth1: 0b 1d2d80 f7f8 7f [0] [ 5417.276005] bnx2 0000:01:00.1 eth1: 0c 148f00 f7b8 88 [0] [ 5417.276016] bnx2 0000:01:00.1 eth1: 0d 16af80 f7d0 75 [0] [ 5417.276026] bnx2 0000:01:00.1 eth1: 0e 1adf80 bfb0 26 [0] [ 5417.276037] bnx2 0000:01:00.1 eth1: 0f 1ebf80 dd68 3c [0] [ 5417.276048] bnx2 0000:01:00.1 eth1: 10 1cf700 d1f0 fc [0] [ 5417.276059] bnx2 0000:01:00.1 eth1: 11 1cdc00 fbf0 7d [0] [ 5417.276069] bnx2 0000:01:00.1 eth1: 12 15c900 f7f8 ef [0] [ 5417.276081] bnx2 0000:01:00.1 eth1: 13 17cf00 d7d8 3f [0] [ 5417.276093] bnx2 0000:01:00.1 eth1: 14 1ecf80 ffb0 b7 [0] [ 5417.276107] bnx2 0000:01:00.1 eth1: 15 1cbd80 f3e8 bf [0] [ 5417.276119] bnx2 0000:01:00.1 eth1: 16 179b80 d7f8 d7 [0] [ 5417.276130] bnx2 0000:01:00.1 eth1: 17 1fdf00 f3e8 7e [0] [ 5417.276141] bnx2 0000:01:00.1 eth1: 18 1f9780 b578 af [0] [ 5417.276152] bnx2 0000:01:00.1 eth1: 19 1d7d80 fef0 ff [0] [ 5417.276163] bnx2 0000:01:00.1 eth1: 1a 1d9e80 5fe8 d7 [0] [ 5417.276174] bnx2 0000:01:00.1 eth1: 1b 1fff80 ebf8 f8 [0] [ 5417.276186] bnx2 0000:01:00.1 eth1: 1c 1fbd80 f7d8 7f [0] [ 5417.276200] bnx2 0000:01:00.1 eth1: 1d 16da80 2ef8 ff [0] [ 5417.276211] bnx2 0000:01:00.1 eth1: 1e 1f9b80 bf50 8e [0] [ 5417.276224] bnx2 0000:01:00.1 eth1: 1f 1bdf00 faf8 75 [0] [ 5417.276231] bnx2 0000:01:00.1 eth1: <--- end TBDC dump ---> [ 5417.276246] bnx2 0000:01:00.1 eth1: DEBUG: intr_sem[0] PCI_CMD[00100406] [ 5417.276258] bnx2 0000:01:00.1 eth1: DEBUG: PCI_PM[19002008] PCI_MISC_CFG[92000088] [ 5417.276269] bnx2 0000:01:00.1 eth1: DEBUG: EMAC_TX_STATUS[00000008] EMAC_RX_STATUS[00000000] [ 5417.276280] bnx2 0000:01:00.1 eth1: DEBUG: RPM_MGMT_PKT_CTRL[40000088] [ 5417.276288] bnx2 0000:01:00.1 eth1: DEBUG: HC_STATS_INTERRUPT_STATUS[01fb0004] [ 5417.276298] bnx2 0000:01:00.1 eth1: DEBUG: PBA[00000000] [ 5417.276304] bnx2 0000:01:00.1 eth1: <--- start MCP states dump ---> [ 5417.276314] bnx2 0000:01:00.1 eth1: DEBUG: MCP_STATE_P0[0003610e] MCP_STATE_P1[0003610e] [ 5417.276326] bnx2 0000:01:00.1 eth1: DEBUG: MCP mode[0000b880] state[80000000] evt_mask[00000500] [ 5417.276339] bnx2 0000:01:00.1 eth1: DEBUG: pc[0800d7b8] pc[08000cdc] instr[00041880] [ 5417.276349] bnx2 0000:01:00.1 eth1: DEBUG: shmem states: [ 5417.276358] bnx2 0000:01:00.1 eth1: DEBUG: drv_mb[0d000004] fw_mb[00000004] link_status[0000006f] [ 5417.276369] drv_pulse_mb[00001485] [ 5417.276373] bnx2 0000:01:00.1 eth1: DEBUG: dev_info_signature[44564903] reset_type[01005254] [ 5417.276383] condition[0003610e] [ 5417.276389] bnx2 0000:01:00.1 eth1: DEBUG: 000001c0: 01005254 42530000 0003610e 00000000 [ 5417.276402] bnx2 0000:01:00.1 eth1: DEBUG: 000003cc: 44444444 44444444 44444444 00000a28 [ 5417.276416] bnx2 0000:01:00.1 eth1: DEBUG: 000003dc: 0004ffff 00000000 00000000 00000000 [ 5417.276430] bnx2 0000:01:00.1 eth1: DEBUG: 000003ec: 00000000 00000000 00000000 00000000 [ 5417.276440] bnx2 0000:01:00.1 eth1: DEBUG: 0x3fc[0000ffff] -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/