Jesse Brandeburg wrote:
> On Sun, Feb 14, 2010 at 8:29 AM, Jesper Krogh <[email protected]> wrote:
>> Hi List.
>>
>> I have tried to get a dual bond of 2 x 10G NICs using the
>> Intel Corporation 82598EB 10-Gigabit AT2 Server Adapter (rev 01)
>> going. As first it looked like it "just worked" but when tried to fill
>> the links with data one of the NIC's (eth7) hang and did a reset of
>> itself, so all data was pushed through the other NIC in the bond (eth8)
>>
>> Full dmesg below, but I think the important part is this:
>>
>> [ 2162.745354] ixgbe: eth7: ixgbe_check_tx_hang: Detected Tx Unit Hang
>> [ 2162.745356] Tx Queue <4>
>> [ 2162.745356] TDH, TDT <e1>, <cc>
>> [ 2162.745357] next_to_use <cc>
>> [ 2162.745358] next_to_clean <e1>
>> [ 2162.745359] tx_buffer_info[next_to_clean]
>> [ 2162.745359] time_stamp <1000713d3>
>> [ 2162.745360] jiffies <10007152e>
>> [ 2163.162478] ixgbe: eth7: ixgbe_clean_tx_irq: tx hang 1 detected,
>> resetting adapter
>> [ 2163.357333] bonding: bond0: link status definitely down for interface
>> eth7, disabling it
>> [ 2168.670342] ixgbe: eth7 NIC Link is Up 10 Gbps, Flow Control: None
>
> Hi Jesper, my first thought was flow control, but I can see you have it off.
I didnt change it, so its default.
> Can we get some more details on the hardware and bios version?
Sun X4600, 128GB, 16 cores..
> What
> about some dmidecode output. I'm checking here if we have any
> hardware like this.
http://shrek.krogh.cc/~jesper/dmidecode.txt
> are you running ubuntu 9.10 or something else?
We're on Ubuntu 8.04 but we basically "only" use a filesystem, kernel
with a well-performing NFS-server and some NIC that work well :-)
> Wow, thats a monster machine, 8 nodes, 128GB ram. Can we get a full
> lspci -vvv output, as well as ethtool -e eth7 and eth8
http://shrek.krogh.cc/~jesper/lspci.txt
http://shrek.krogh.cc/~jesper/ethtool-eth7.txt
http://shrek.krogh.cc/~jesper/ethtool-eth8.txt
> 32 has ixgbe with a known issue of multiple mappings on transmit
> possibly causing some problems, could it be that you're running into
> this? can you apply commit e5a43549f7a58509a91b299a51337d386697b92c
> and see if it fixes your issue?
I'll do that the next time I can push a reboot through.
--
Jesper
On Mon, 2010-02-22 at 10:40 -0800, Jesper Krogh wrote:
> Jesse Brandeburg wrote:
> > On Sun, Feb 14, 2010 at 8:29 AM, Jesper Krogh <[email protected]> wrote:
> >> Hi List.
> >>
> >> I have tried to get a dual bond of 2 x 10G NICs using the
> >> Intel Corporation 82598EB 10-Gigabit AT2 Server Adapter (rev 01)
> >> going. As first it looked like it "just worked" but when tried to fill
> >> the links with data one of the NIC's (eth7) hang and did a reset of
> >> itself, so all data was pushed through the other NIC in the bond (eth8)
> >>
> >> Full dmesg below, but I think the important part is this:
> >>
> >> [ 2162.745354] ixgbe: eth7: ixgbe_check_tx_hang: Detected Tx Unit Hang
> >> [ 2162.745356] Tx Queue <4>
> >> [ 2162.745356] TDH, TDT <e1>, <cc>
> >> [ 2162.745357] next_to_use <cc>
> >> [ 2162.745358] next_to_clean <e1>
> >> [ 2162.745359] tx_buffer_info[next_to_clean]
> >> [ 2162.745359] time_stamp <1000713d3>
> >> [ 2162.745360] jiffies <10007152e>
> >> [ 2163.162478] ixgbe: eth7: ixgbe_clean_tx_irq: tx hang 1 detected,
> >> resetting adapter
> >> [ 2163.357333] bonding: bond0: link status definitely down for interface
> >> eth7, disabling it
> >> [ 2168.670342] ixgbe: eth7 NIC Link is Up 10 Gbps, Flow Control: None
> >
> > Hi Jesper, my first thought was flow control, but I can see you have it off.
>
> I didnt change it, so its default.
>
> > Can we get some more details on the hardware and bios version?
>
> Sun X4600, 128GB, 16 cores..
>
> > What
> > about some dmidecode output. I'm checking here if we have any
> > hardware like this.
>
> http://shrek.krogh.cc/~jesper/dmidecode.txt
>
>
> > are you running ubuntu 9.10 or something else?
>
> We're on Ubuntu 8.04 but we basically "only" use a filesystem, kernel
> with a well-performing NFS-server and some NIC that work well :-)
>
> > Wow, thats a monster machine, 8 nodes, 128GB ram. Can we get a full
> > lspci -vvv output, as well as ethtool -e eth7 and eth8
>
> http://shrek.krogh.cc/~jesper/lspci.txt
>
> http://shrek.krogh.cc/~jesper/ethtool-eth7.txt
> http://shrek.krogh.cc/~jesper/ethtool-eth8.txt
>
> > 32 has ixgbe with a known issue of multiple mappings on transmit
> > possibly causing some problems, could it be that you're running into
> > this? can you apply commit e5a43549f7a58509a91b299a51337d386697b92c
> > and see if it fixes your issue?
>
> I'll do that the next time I can push a reboot through.
>
including full email for context, did this solve your issue?
--
Jesse Brandeburg
This email sent via Evolution, powered by Linux