Andrew Morton wrote:
> Another customer..
>
> Begin forwarded message:
>
> Date: Fri, 29 Sep 2006 00:44:01 +0200
> From: Matthias Hentges <[email protected]>
> To: Andrew Morton <[email protected]>
> Cc: [email protected]
> Subject: Re: 2.6.18-mm2
>
>
> Hello all,
>
> I've just tested -mm2 on my C2D system and I'm getting a lot of these
> messages:
>
> "[ 139.143807] printk: 131 messages suppressed.
> [ 139.148235] sky2 0000:03:00.0: pci express error (0x500547)"
>
> Please note that the "sky2" driver has always been the black sheep on
> that system due to regular full lock-ups of the driver, requiring a
> rmmod sky2 + modprobe sky2 cycle.
>
> This happens often enough to warrant writing a cronjob checking the
> network and auto-rmmod'ing the module.....
>
> While the above is bloody annoying at times (heh), the driver never
> caused any messages like the ones I now get with -mm2 .
sky2 just turned on PCI Express error reporting, so it makes sense that
messages would appear. The better question is whether this is a driver
problem, or a hardware problem. With your "black sheep" comment, I
wonder if it isn't a hardware problem that's been hidden.
Jeff
On Thu, 28 Sep 2006 19:07:05 -0400
Jeff Garzik <[email protected]> wrote:
> Andrew Morton wrote:
> > Another customer..
> >
> > Begin forwarded message:
> >
> > Date: Fri, 29 Sep 2006 00:44:01 +0200
> > From: Matthias Hentges <[email protected]>
> > To: Andrew Morton <[email protected]>
> > Cc: [email protected]
> > Subject: Re: 2.6.18-mm2
> >
> >
> > Hello all,
> >
> > I've just tested -mm2 on my C2D system and I'm getting a lot of these
> > messages:
> >
> > "[ 139.143807] printk: 131 messages suppressed.
> > [ 139.148235] sky2 0000:03:00.0: pci express error (0x500547)"
> >
> > Please note that the "sky2" driver has always been the black sheep on
> > that system due to regular full lock-ups of the driver, requiring a
> > rmmod sky2 + modprobe sky2 cycle.
> >
> > This happens often enough to warrant writing a cronjob checking the
> > network and auto-rmmod'ing the module.....
> >
> > While the above is bloody annoying at times (heh), the driver never
> > caused any messages like the ones I now get with -mm2 .
>
> sky2 just turned on PCI Express error reporting, so it makes sense that
> messages would appear. The better question is whether this is a driver
> problem, or a hardware problem. With your "black sheep" comment, I
> wonder if it isn't a hardware problem that's been hidden.
Here is the debug patch I sent to the first reporter of the problem.
I know what the offset is supposed to be, so if the PCI subsystem is
wrong, this will show.
--- sky2.orig/drivers/net/sky2.c 2006-09-28 08:45:27.000000000 -0700
+++ sky2/drivers/net/sky2.c 2006-09-28 08:51:24.000000000 -0700
@@ -2463,6 +2463,7 @@
sky2_write8(hw, B0_CTST, CS_MRST_CLR);
+#define PEX_UNC_ERR_STAT 0x104 /* PCI extended error capablity */
/* clear any PEX errors */
if (pci_find_capability(hw->pdev, PCI_CAP_ID_EXP)) {
hw->err_cap = pci_find_ext_capability(hw->pdev, PCI_EXT_CAP_ID_ERR);
@@ -2470,6 +2471,15 @@
sky2_pci_write32(hw,
hw->err_cap + PCI_ERR_UNCOR_STATUS,
0xffffffffUL);
+ else
+ printk(KERN_ERR PFX "pci express found but not extended error support?\n");
+
+ if (hw->err_cap + PCI_ERR_UNCOR_STATUS != PEX_UNC_ERR_STAT) {
+
+ printk(KERN_ERR PFX "pci express error status register fixed from %#x to %#x\n",
+ hw->err_cap, PEX_UNC_ERR_STAT - PCI_ERR_UNCOR_STATUS);
+ hw->err_cap = PEX_UNC_ERR_STAT - PCI_ERR_UNCOR_STATUS;
+ }
}
hw->pmd_type = sky2_read8(hw, B2_PMD_TYP);
On Thu, 28 Sep 2006 19:07:05 -0400
Jeff Garzik <[email protected]> wrote:
> Andrew Morton wrote:
> > Another customer..
> >
> > Begin forwarded message:
> >
> > Date: Fri, 29 Sep 2006 00:44:01 +0200
> > From: Matthias Hentges <[email protected]>
> > To: Andrew Morton <[email protected]>
> > Cc: [email protected]
> > Subject: Re: 2.6.18-mm2
> >
> >
> > Hello all,
> >
> > I've just tested -mm2 on my C2D system and I'm getting a lot of these
> > messages:
> >
> > "[ 139.143807] printk: 131 messages suppressed.
> > [ 139.148235] sky2 0000:03:00.0: pci express error (0x500547)"
> >
> > Please note that the "sky2" driver has always been the black sheep on
> > that system due to regular full lock-ups of the driver, requiring a
> > rmmod sky2 + modprobe sky2 cycle.
> >
> > This happens often enough to warrant writing a cronjob checking the
> > network and auto-rmmod'ing the module.....
> >
> > While the above is bloody annoying at times (heh), the driver never
> > caused any messages like the ones I now get with -mm2 .
>
> sky2 just turned on PCI Express error reporting, so it makes sense that
> messages would appear. The better question is whether this is a driver
> problem, or a hardware problem. With your "black sheep" comment, I
> wonder if it isn't a hardware problem that's been hidden.
>
See also http://bugzilla.kernel.org/show_bug.cgi?id=7222
That's two reports in 18 hours, from amongst the presumably-small population
of sky2-owning -mm testers.
>
> See also http://bugzilla.kernel.org/show_bug.cgi?id=7222
>
> That's two reports in 18 hours, from amongst the presumably-small population
> of sky2-owning -mm testers.
I'll back it out if we don't get a simple resolution. It was just trying to
use the pci facilities as intended.
Note: I know what is causing all the sky2 problems, there is something wrong that
is causing flow control negotiation not to propagate back to all the multiple levels
of the chip. Unclear how to fix it, the documentation is not helpful on this.
If not resolved soon, I'll just force Tx flow control off for now.
--
Stephen Hemminger <[email protected]>
On Thu, Sep 28, 2006 at 04:30:23PM -0700, Stephen Hemminger wrote:
> Note: I know what is causing all the sky2 problems, there is something wrong that
> is causing flow control negotiation not to propagate back to all the multiple levels
> of the chip. Unclear how to fix it, the documentation is not helpful on this.
> If not resolved soon, I'll just force Tx flow control off for now.
just for the record, same problem here.
I mean, with my Sony Vaio VGN-SZ1VP (here[1] you can find all hardware
details), it's enough some mega of udp traffic, usually nfs, to "freeze"
the network. Well, no complain from the kernel. It's enough to rmmod and
modprobe sky2 to fix the problem. I already tried -mm1, but nothing
changed. In the meanwhile I will continue to use my usb network card.
Thanks a lot for your time,
Andrea Gelmini
-------
[1] http://groups.google.it/group/linux.kernel/msg/ceff3014c410bea6
Hello Stephen,
Am Donnerstag, den 28.09.2006, 16:19 -0700 schrieb Stephen Hemminger:
[...]
> Here is the debug patch I sent to the first reporter of the problem.
> I know what the offset is supposed to be, so if the PCI subsystem is
> wrong, this will show.
>
> --- sky2.orig/drivers/net/sky2.c 2006-09-28 08:45:27.000000000 -0700
> +++ sky2/drivers/net/sky2.c 2006-09-28 08:51:24.000000000 -0700
[...]
Thanks for the patch:
+root@mhcln01:~ >dmesg |grep -i expres
[ 36.814288] sky2 pci express found but not extended error support?
[ 36.820494] sky2 pci express error status register fixed from 0x0 to
0x100
[ 36.833769] sky2 0000:04:00.0: pci express error (0x100000)
[ 36.851296] sky2 pci express found but not extended error support?
[ 36.857499] sky2 pci express error status register fixed from 0x0 to
0x100
[ 36.870756] sky2 0000:03:00.0: pci express error (0x100000)
+root@mhcln01:~ >
The annoying messages are gone =)
Attached is a complete dmesg of a fresh boot.
Thanks for your time.
--
Matthias Hentges
My OS: Debian SID. Geek by Nature, Linux by Choice
Hello Stephen,
Am Donnerstag, den 28.09.2006, 16:19 -0700 schrieb Stephen Hemminger:
> Here is the debug patch I sent to the first reporter of the problem.
> I know what the offset is supposed to be, so if the PCI subsystem is
> wrong, this will show.
>
> --- sky2.orig/drivers/net/sky2.c 2006-09-28 08:45:27.000000000 -0700
> +++ sky2/drivers/net/sky2.c 2006-09-28 08:51:24.000000000 -0700
> @@ -2463,6 +2463,7 @@
>
> sky2_write8(hw, B0_CTST, CS_MRST_CLR);
>
> +#define PEX_UNC_ERR_STAT 0x104 /* PCI extended error capablity */
> /* clear any PEX errors */
> if (pci_find_capability(hw->pdev, PCI_CAP_ID_EXP)) {
> hw->err_cap = pci_find_ext_capability(hw->pdev, PCI_EXT_CAP_ID_ERR);
> @@ -2470,6 +2471,15 @@
> sky2_pci_write32(hw,
> hw->err_cap + PCI_ERR_UNCOR_STATUS,
> 0xffffffffUL);
> + else
> + printk(KERN_ERR PFX "pci express found but not extended error support?\n");
> +
> + if (hw->err_cap + PCI_ERR_UNCOR_STATUS != PEX_UNC_ERR_STAT) {
> +
> + printk(KERN_ERR PFX "pci express error status register fixed from %#x to %#x\n",
> + hw->err_cap, PEX_UNC_ERR_STAT - PCI_ERR_UNCOR_STATUS);
> + hw->err_cap = PEX_UNC_ERR_STAT - PCI_ERR_UNCOR_STATUS;
> + }
> }
>
> hw->pmd_type = sky2_read8(hw, B2_PMD_TYP);
while the above patch indeed removes the error messages from my previous
mail, I have since seen random but reproduceable freezes of the box in
question. I believe they are sky2 related since the freeze can be
triggered by continuous network traffic (like playing a movie over NFS
etc.).
The freezes only happen with 2.6.18-mm2 and 2.6.18-mm3. 2.6.18-mm1 works
perfectly fine.
I've hooked up the box to my laptop via a serial cable and captured all
kernel messages from booting up the machine to the freeze. You'll note
that the last messages are from the sky2 driver ;)
Once frozen the network is dead, the screen won't wake up from suspend
and CAPSLOCK can not be toggled. SYSRQ (sp?) still works tho.
Any help in debugging this problem would be appreciated =)
--
Matthias 'CoreDump' Hentges
My OS: Debian SID. Geek by Nature, Linux by Choice
On Wed, 04 Oct 2006 04:57:08 +0200
Matthias Hentges <[email protected]> wrote:
> Hello Stephen,
>
> Am Donnerstag, den 28.09.2006, 16:19 -0700 schrieb Stephen Hemminger:
>
> > Here is the debug patch I sent to the first reporter of the problem.
> > I know what the offset is supposed to be, so if the PCI subsystem is
> > wrong, this will show.
> >
> > --- sky2.orig/drivers/net/sky2.c 2006-09-28 08:45:27.000000000 -0700
> > +++ sky2/drivers/net/sky2.c 2006-09-28 08:51:24.000000000 -0700
> > @@ -2463,6 +2463,7 @@
> >
> > sky2_write8(hw, B0_CTST, CS_MRST_CLR);
> >
> > +#define PEX_UNC_ERR_STAT 0x104 /* PCI extended error capablity */
> > /* clear any PEX errors */
> > if (pci_find_capability(hw->pdev, PCI_CAP_ID_EXP)) {
> > hw->err_cap = pci_find_ext_capability(hw->pdev, PCI_EXT_CAP_ID_ERR);
> > @@ -2470,6 +2471,15 @@
> > sky2_pci_write32(hw,
> > hw->err_cap + PCI_ERR_UNCOR_STATUS,
> > 0xffffffffUL);
> > + else
> > + printk(KERN_ERR PFX "pci express found but not extended error support?\n");
> > +
> > + if (hw->err_cap + PCI_ERR_UNCOR_STATUS != PEX_UNC_ERR_STAT) {
> > +
> > + printk(KERN_ERR PFX "pci express error status register fixed from %#x to %#x\n",
> > + hw->err_cap, PEX_UNC_ERR_STAT - PCI_ERR_UNCOR_STATUS);
> > + hw->err_cap = PEX_UNC_ERR_STAT - PCI_ERR_UNCOR_STATUS;
> > + }
> > }
> >
> > hw->pmd_type = sky2_read8(hw, B2_PMD_TYP);
>
> while the above patch indeed removes the error messages from my previous
> mail, I have since seen random but reproduceable freezes of the box in
> question. I believe they are sky2 related since the freeze can be
> triggered by continuous network traffic (like playing a movie over NFS
> etc.).
When it fixes what does the log say. I'm probably going to back out
the PCI express extended error using the pci_XXX functions.
> The freezes only happen with 2.6.18-mm2 and 2.6.18-mm3. 2.6.18-mm1 works
> perfectly fine.
> I've hooked up the box to my laptop via a serial cable and captured all
> kernel messages from booting up the machine to the freeze. You'll note
> that the last messages are from the sky2 driver ;)
>
Does it still happen with linus git tree. If so, a git bisect might
help. It might not be sky2 related at all, there has been lots of changes.
> Once frozen the network is dead, the screen won't wake up from suspend
> and CAPSLOCK can not be toggled. SYSRQ (sp?) still works tho.
>
> Any help in debugging this problem would be appreciated =)
The TX timeout is a symptom of a common bug still not fixed where
the transmitter stops. I'm working on reproducing it on my hardware and switches,
because without a reproducible test, its just shooting in the dark and
that isn't working.
Hello Stephen,
Am Dienstag, den 03.10.2006, 20:26 -0700 schrieb Stephen Hemminger:
[...]
> > while the above patch indeed removes the error messages from my previous
> > mail, I have since seen random but reproduceable freezes of the box in
> > question. I believe they are sky2 related since the freeze can be
> > triggered by continuous network traffic (like playing a movie over NFS
> > etc.).
>
> When it fixes what does the log say. I'm probably going to back out
> the PCI express extended error using the pci_XXX functions.
> > The freezes only happen with 2.6.18-mm2 and 2.6.18-mm3. 2.6.18-mm1 works
> > perfectly fine.
> > I've hooked up the box to my laptop via a serial cable and captured all
> > kernel messages from booting up the machine to the freeze. You'll note
> > that the last messages are from the sky2 driver ;)
> >
>
> Does it still happen with linus git tree. If so, a git bisect might
> help. It might not be sky2 related at all, there has been lots of changes.
I am doing a bisect right now which is kind of a PITA as the freeze is
completely random.
I have since noticed that the freeze happens shortly after the network
dies, possibly during the "rrmod sky2 / moprobe sky2" my script is
doing.
> > Once frozen the network is dead, the screen won't wake up from suspend
> > and CAPSLOCK can not be toggled. SYSRQ (sp?) still works tho.
> >
> > Any help in debugging this problem would be appreciated =)
>
> The TX timeout is a symptom of a common bug still not fixed where
> the transmitter stops. I'm working on reproducing it on my hardware and switches,
> because without a reproducible test, its just shooting in the dark and
> that isn't working.
I'd be happy to assist with that as I have his bug up to 5 times a day :
\
--
Matthias 'CoreDump' Hentges
Webmaster of hentges.net and OpenZaurus developer.
You can reach me in #openzaurus on Freenode.
My OS: Debian SID. Geek by Nature, Linux by Choice
Hi Stephen,
I believe I have identified the problem. The freeze only happens when
your debug patch to work around sky2 PCIe error messages is applied.
Without your patch (attached) I get _tons_ of error messages and the NIC
dies every few seconds / minutes (reproduceable!), but the system
recovers just fine from a NIC crash.
I have verified this behavior (works fine w/o debug patch, freezes with
patch applied) with:
- 2.6.19-rc1-git4
- 2.6.18-git something
- 2.6.18-mm3
--
Matthias 'CoreDump' Hentges
My OS: Debian SID. Geek by Nature, Linux by Choice
On Sun, 08 Oct 2006 15:41:45 +0200
Matthias Hentges <[email protected]> wrote:
> Hi Stephen,
>
> I believe I have identified the problem. The freeze only happens when
> your debug patch to work around sky2 PCIe error messages is applied.
> Without your patch (attached) I get _tons_ of error messages and the NIC
> dies every few seconds / minutes (reproduceable!), but the system
> recovers just fine from a NIC crash.
>
> I have verified this behavior (works fine w/o debug patch, freezes with
> patch applied) with:
> - 2.6.19-rc1-git4
> - 2.6.18-git something
> - 2.6.18-mm3
>
Does 2.6.18 work?
What is the PCI config of the device (lspci -vvvx)?
What is the chip version (dmesg | grep sky2)?
Am Sonntag, den 08.10.2006, 09:20 -0700 schrieb Stephen Hemminger:
> On Sun, 08 Oct 2006 15:41:45 +0200
> Matthias Hentges <[email protected]> wrote:
[...]
> > I have verified this behavior (works fine w/o debug patch, freezes with
> > patch applied) with:
> > - 2.6.19-rc1-git4
> > - 2.6.18-git something
> > - 2.6.18-mm3
> >
>
> Does 2.6.18 work?
2.6.18 does not require the patch ( in fact, the patch won't apply at
all) and thus does not freeze when the NIC goes down.
> What is the PCI config of the device (lspci -vvvx)?
I've attached the lspci dump of both onboard sky2 NICs.
> What is the chip version (dmesg | grep sky2)?
>
+root@mhcln01:..patches/generic/coretemp >dmesg |grep sky2
[ 34.833453] sky2 v1.7 addr 0xfa9fc000 irq 17 Yukon-EC (0xb6) rev 2
[ 34.833527] sky2 eth0: addr 00:17:31:f4:f1:8c
[ 34.833663] sky2 v1.7 addr 0xfa8fc000 irq 16 Yukon-EC (0xb6) rev 2
[ 34.833729] sky2 eth1: addr 00:17:31:f4:f7:cc
[ 42.165687] sky2 eth1: enabling interface
[ 43.847972] sky2 eth1: Link is up at 100 Mbps, full duplex, flow
control both
--
Matthias 'CoreDump' Hentges
My OS: Debian SID. Geek by Nature, Linux by Choice
Oops, I forgot the "x" in lspci -vvvx, new dumps are attached.
--
Matthias 'CoreDump' Hentges
My OS: Debian SID. Geek by Nature, Linux by Choice
On Sun, 08 Oct 2006 20:31:36 +0200
Matthias Hentges <[email protected]> wrote:
>
> Oops, I forgot the "x" in lspci -vvvx, new dumps are attached.
I think I know what the problem is. The PCI access routines to access pci express
registers (ie reg > 256), only work if using MMCONFIG access. For some reason
your configuration doesn't want to use/allow that.
When it happened before, I ended up just not using the pci_read_config_XXX
routines and using the device map. I'll revert the patch that started using
pci_find_ext_capabablity.
--
Stephen Hemminger <[email protected]>
On 2006-10-09, Stephen Hemminger <[email protected]> wrote:
> On Sun, 08 Oct 2006 20:31:36 +0200
> Matthias Hentges <[email protected]> wrote:
>
>>
>> Oops, I forgot the "x" in lspci -vvvx, new dumps are attached.
>
>
> I think I know what the problem is. The PCI access routines to access pci express
> registers (ie reg > 256), only work if using MMCONFIG access. For some reason
> your configuration doesn't want to use/allow that.
In case you didn't read, here's top of thread about MMCONFIG 4 days ago:
<http://article.gmane.org/gmane.linux.ports.x86-64.general/1794>
In short: due to Intel BIOS bug, mmconfig doesn't used in kernel.
And kernel developers mainly do not care much (yet) about drivers, until
vi$ta certified hardware will be in run.
____
Le Sun, 8 Oct 2006 09:20:01 -0700, Stephen Hemminger
<[email protected]> a écrit :
> On Sun, 08 Oct 2006 15:41:45 +0200
> Matthias Hentges <[email protected]> wrote:
>
> > Hi Stephen,
> >
> > I believe I have identified the problem. The freeze only happens when
> > your debug patch to work around sky2 PCIe error messages is applied.
> > Without your patch (attached) I get _tons_ of error messages and the NIC
> > dies every few seconds / minutes (reproduceable!), but the system
> > recovers just fine from a NIC crash.
> >
> > I have verified this behavior (works fine w/o debug patch, freezes with
> > patch applied) with:
> > - 2.6.19-rc1-git4
> > - 2.6.18-git something
> > - 2.6.18-mm3
> >
>
> Does 2.6.18 work?
>
> What is the PCI config of the device (lspci -vvvx)?
>
> What is the chip version (dmesg | grep sky2)?
I've got the same issue here since a while (See Message-ID:
<[email protected]> from 18 Apr 2006)
I steel get freeze (with non tainted kernel) and getting these messages
at boot :
~ % dmesg | grep -i sky
sky2 v1.7 addr 0xff3fc000 irq 233 Yukon-EC (0xb6) rev 2
sky2 eth0: addr 00:15:f2:a9:6a:65
sky2 0000:02:00.0: No interrupt was generated using MSI, switching to INTx mode. Please report this failure to the PCI maintainer and include system chipset information.
sky2 eth0: enabling interface
sky2 eth0: disabling interface
lspci -vvvx attached
--
Beber - E-Mail / Jabber (+GMail) : beber_AT_meleeweb.net
http://www.meleeweb.net
Hello Stephen,
Am Montag, den 09.10.2006, 09:45 -0700 schrieb Stephen Hemminger:
> On Sun, 08 Oct 2006 20:31:36 +0200
> Matthias Hentges <[email protected]> wrote:
>
> >
> > Oops, I forgot the "x" in lspci -vvvx, new dumps are attached.
>
>
> I think I know what the problem is. The PCI access routines to access pci express
> registers (ie reg > 256), only work if using MMCONFIG access. For some reason
> your configuration doesn't want to use/allow that.
>
> When it happened before, I ended up just not using the pci_read_config_XXX
> routines and using the device map. I'll revert the patch that started using
> pci_find_ext_capabablity.
the new patch still freezes the box on network loss :\
Tested w/ 2.6.19-rc1-git5
The last kernel message is "eth1: tx timeout"
--
Matthias 'CoreDump' Hentges
My OS: Debian SID. Geek by Nature, Linux by Choice
On 2006-10-09, Oleg Verych wrote:
> On 2006-10-09, Stephen Hemminger wrote:
>> On Sun, 08 Oct 2006 20:31:36 +0200
>> Matthias Hentges <[email protected]> wrote:
>>
>>>
>>> Oops, I forgot the "x" in lspci -vvvx, new dumps are attached.
>>
>>
>> I think I know what the problem is. The PCI access routines to access pci express
>> registers (ie reg > 256), only work if using MMCONFIG access. For some reason
>> your configuration doesn't want to use/allow that.
>
> In case you didn't read, here's top of thread about MMCONFIG 4 days ago:
><http://article.gmane.org/gmane.linux.ports.x86-64.general/1794>
>
> In short: due to Intel BIOS bug, mmconfig doesn't used in kernel.
> And kernel developers mainly do not care much (yet) about drivers, until
> vi$ta certified hardware will be in run.
> ____
[and, due to my dumb, not copied here Stephen's answer was:]
> Then I would argue that PCI express support is broken in the kernel.
____