dann frazier wrote:
> On Tue, Jun 30, 2009 at 11:35:45AM -0700, Yinghai Lu wrote:
>> dann frazier wrote:
>>> hey,
>>> I'm finding problems when booting a dl585g2 w/ an LSISAS1068
>>> controller. In brief, this system system stopped booting once MSI
>>> became disabled by default in mptbase. Passing the parameter
>>> 'mpt_msi_enable_sas=1' to mptbase allows it to work again (tested w/
>>> 2.6.30).
>>>
>>> The symptom is that the system starts up, and we see the following:
>>>
>>> ======== SNIP =========
>>> [ 6.941489] hub 3-2:1.0: USB hub found
>>> [ 6.943705] hub 3-2:1.0: 7 ports detected
>>> [ 6.992639] Copyright (c) 1999-2008 LSI Corporation
>>> [ 7.014262] Fusion MPT SAS Host driver 3.04.07
>>> [ 7.021685] mptsas 0000:42:01.0: PCI INT A -> GSI 24 (level, low) -> IRQ 24
>>> [ 7.026597] usb 1-6.3: new full speed USB device using ehci_hcd and address 3
>>> [ 7.071143] mptbase: ioc0: Initiating bringup
>>> [ 7.102198] usb 1-6.3: New USB device found, idVendor=0e34, idProduct=0204
>>> [ 7.119107] usb 1-6.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
>>> [ 7.128171] usb 1-6.3: Product: iPort/USB I2C Host Adapter
>>> [ 7.160994] usb 1-6.3: Manufacturer: Silicon Labs
>>> [ 7.171468] usb 1-6.3: SerialNumber: 0001248
>>> [ 7.174931] usb 1-6.3: configuration #1 chosen from 1 choice
>>> [ 7.360010] ioc0: LSISAS1068 B0: Capabilities={Initiator}
>>> [ 40.581020] mptbase: ioc0: Initiating recovery
>>> [ 71.190560] Clocksource tsc unstable (delta = 4398041105588 ns)
>>> [ 81.190160] mptbase: ioc0: Initiating recovery
>>> ======== SNIP =========
>>>
>>> These "Initiating recovery" messages are printed periodically, and the
>>> system never recovers.
>> Dann, that could be : ioapic routing in ACPI DSDT is not right somehow.
>>
>> please try to move the card to other slot to get some luck.
>> or pci=routeirq could help too sometimes.
>
> For the record, I tried moving the card to the other PCI-X slot and
> the pci=routeirq boot param, but neither helped.
>
so 2.6.22.1 works on that system with pci=nomsi?
maybe your BIOS is too new, you may try the old BIOS.
updating BIOS is not always a option to deployed systems. maybe sustaining team could break sth.
YH
On Sat, Jul 04, 2009 at 12:30:36PM -0700, Yinghai Lu wrote:
> dann frazier wrote:
> > On Tue, Jun 30, 2009 at 11:35:45AM -0700, Yinghai Lu wrote:
> >> dann frazier wrote:
> >>> hey,
> >>> I'm finding problems when booting a dl585g2 w/ an LSISAS1068
> >>> controller. In brief, this system system stopped booting once MSI
> >>> became disabled by default in mptbase. Passing the parameter
> >>> 'mpt_msi_enable_sas=1' to mptbase allows it to work again (tested w/
> >>> 2.6.30).
> >>>
> >>> The symptom is that the system starts up, and we see the following:
> >>>
> >>> ======== SNIP =========
> >>> [ 6.941489] hub 3-2:1.0: USB hub found
> >>> [ 6.943705] hub 3-2:1.0: 7 ports detected
> >>> [ 6.992639] Copyright (c) 1999-2008 LSI Corporation
> >>> [ 7.014262] Fusion MPT SAS Host driver 3.04.07
> >>> [ 7.021685] mptsas 0000:42:01.0: PCI INT A -> GSI 24 (level, low) -> IRQ 24
> >>> [ 7.026597] usb 1-6.3: new full speed USB device using ehci_hcd and address 3
> >>> [ 7.071143] mptbase: ioc0: Initiating bringup
> >>> [ 7.102198] usb 1-6.3: New USB device found, idVendor=0e34, idProduct=0204
> >>> [ 7.119107] usb 1-6.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
> >>> [ 7.128171] usb 1-6.3: Product: iPort/USB I2C Host Adapter
> >>> [ 7.160994] usb 1-6.3: Manufacturer: Silicon Labs
> >>> [ 7.171468] usb 1-6.3: SerialNumber: 0001248
> >>> [ 7.174931] usb 1-6.3: configuration #1 chosen from 1 choice
> >>> [ 7.360010] ioc0: LSISAS1068 B0: Capabilities={Initiator}
> >>> [ 40.581020] mptbase: ioc0: Initiating recovery
> >>> [ 71.190560] Clocksource tsc unstable (delta = 4398041105588 ns)
> >>> [ 81.190160] mptbase: ioc0: Initiating recovery
> >>> ======== SNIP =========
> >>>
> >>> These "Initiating recovery" messages are printed periodically, and the
> >>> system never recovers.
> >> Dann, that could be : ioapic routing in ACPI DSDT is not right somehow.
> >>
> >> please try to move the card to other slot to get some luck.
> >> or pci=routeirq could help too sometimes.
> >
> > For the record, I tried moving the card to the other PCI-X slot and
> > the pci=routeirq boot param, but neither helped.
> >
>
> so 2.6.22.1 works on that system with pci=nomsi?
yes
> maybe your BIOS is too new, you may try the old BIOS.
>
> updating BIOS is not always a option to deployed systems. maybe
> sustaining team could break sth.
Thanks - I'll see if I can find the right pair of eyes internally to
look into this.
>
> YH
>
--
dann frazier
On Sat, Jul 04, 2009 at 12:30:36PM -0700, Yinghai Lu wrote:
> dann frazier wrote:
> > On Tue, Jun 30, 2009 at 11:35:45AM -0700, Yinghai Lu wrote:
> >> dann frazier wrote:
> >>> hey,
> >>> I'm finding problems when booting a dl585g2 w/ an LSISAS1068
> >>> controller. In brief, this system system stopped booting once MSI
> >>> became disabled by default in mptbase. Passing the parameter
> >>> 'mpt_msi_enable_sas=1' to mptbase allows it to work again (tested w/
> >>> 2.6.30).
> >>>
> >>> The symptom is that the system starts up, and we see the following:
> >>>
> >>> ======== SNIP =========
> >>> [ 6.941489] hub 3-2:1.0: USB hub found
> >>> [ 6.943705] hub 3-2:1.0: 7 ports detected
> >>> [ 6.992639] Copyright (c) 1999-2008 LSI Corporation
> >>> [ 7.014262] Fusion MPT SAS Host driver 3.04.07
> >>> [ 7.021685] mptsas 0000:42:01.0: PCI INT A -> GSI 24 (level, low) -> IRQ 24
> >>> [ 7.026597] usb 1-6.3: new full speed USB device using ehci_hcd and address 3
> >>> [ 7.071143] mptbase: ioc0: Initiating bringup
> >>> [ 7.102198] usb 1-6.3: New USB device found, idVendor=0e34, idProduct=0204
> >>> [ 7.119107] usb 1-6.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
> >>> [ 7.128171] usb 1-6.3: Product: iPort/USB I2C Host Adapter
> >>> [ 7.160994] usb 1-6.3: Manufacturer: Silicon Labs
> >>> [ 7.171468] usb 1-6.3: SerialNumber: 0001248
> >>> [ 7.174931] usb 1-6.3: configuration #1 chosen from 1 choice
> >>> [ 7.360010] ioc0: LSISAS1068 B0: Capabilities={Initiator}
> >>> [ 40.581020] mptbase: ioc0: Initiating recovery
> >>> [ 71.190560] Clocksource tsc unstable (delta = 4398041105588 ns)
> >>> [ 81.190160] mptbase: ioc0: Initiating recovery
> >>> ======== SNIP =========
> >>>
> >>> These "Initiating recovery" messages are printed periodically, and the
> >>> system never recovers.
> >> Dann, that could be : ioapic routing in ACPI DSDT is not right somehow.
> >>
> >> please try to move the card to other slot to get some luck.
> >> or pci=routeirq could help too sometimes.
> >
> > For the record, I tried moving the card to the other PCI-X slot and
> > the pci=routeirq boot param, but neither helped.
> >
>
> so 2.6.22.1 works on that system with pci=nomsi?
>
> maybe your BIOS is too new, you may try the old BIOS.
>
> updating BIOS is not always a option to deployed systems. maybe sustaining team could break sth.
Back with more info.
Both of the original bisects I did with cciss and mptsas appeared to
land on commits that just masked the issue. e.g., a cciss hang caused
by a bug w/ no logical drives, and mptsas switching toggling the MSI
default.
I tried to avoid these issues by doing another bisect w/ storage
attached to the cciss controller, and landed on this commit:
--------------------------------------------------------------------
commit 8d539108560ec121d59eee05160236488266221c
Author: Linus Torvalds <[email protected]>
Date: Thu May 8 18:41:48 2008 -0700
Revert "PCI: remove default PCI expansion ROM memory allocation"
This reverts commit 9f8daccaa05c14e5643bdd4faf5aed9cc8e6f11e, which
was reported to break X startup (xf86-video-ati-6.8.0).
--------------------------------------------------------------------
>From that, I found that pci=norom avoids the problem. But, again, this
is just a masking commit - things used to work before 9f8dacc was
there.
I did another bisect, this time keeping the norom patches out
entirely. This landed on:
--------------------------------------------------------------------
commit 30a18d6c3f1e774de656ebd8ff219d53e2ba4029
Author: Yinghai Lu <[email protected]>
Date: Tue Feb 19 03:21:20 2008 -0800
x86: multi pci root bus with different io resource range, on
64-bit
--------------------------------------------------------------------
This appears to be the commit that actually introduced the issue.
I've attached dmesg w/ PCI DEBUG enabled from both sides of this
changeset, as well as a diff w/o printk timestamps.
Also note that #ifdef'ing out x86_pci_root_bus_res_quirks() in
amd_bus.c seems to avoid the problem as well.
--
dann frazier
dann frazier wrote:
>
> --------------------------------------------------------------------
> commit 30a18d6c3f1e774de656ebd8ff219d53e2ba4029
> Author: Yinghai Lu <[email protected]>
> Date: Tue Feb 19 03:21:20 2008 -0800
>
> x86: multi pci root bus with different io resource range, on
> 64-bit
> --------------------------------------------------------------------
>
> This appears to be the commit that actually introduced the issue.
> I've attached dmesg w/ PCI DEBUG enabled from both sides of this
> changeset, as well as a diff w/o printk timestamps.
>
> Also note that #ifdef'ing out x86_pci_root_bus_res_quirks() in
> amd_bus.c seems to avoid the problem as well.
your BIOS didn't allocate io resources to some device under HT chain on node1/link2
aka under peer root bus.
Kernel try to allocate resource to them.
and the patch did right thing according to range
bus: 40 index 1 mmio: [d9f00000, dfefffff]
YH
+bus: [00,3f] on node 0 link 1
+bus: 00 index 0 io port: [0, 2fff]
+bus: 00 index 1 mmio: [80000000, d9efffff]
+bus: 00 index 2 mmio: [dff00000, e3ffffff]
+bus: 00 index 3 mmio: [a0000, bffff]
+bus: 00 index 4 mmio: [e8000000, ffffffff]
+bus: 00 index 5 mmio: [480000000, fbffffffff]
+bus: [40,7f] on node 1 link 2
+bus: 40 index 0 io port: [3000, ffff]
+bus: 40 index 1 mmio: [d9f00000, dfefffff]
+bus: 40 index 2 mmio: [e4000000, e7ffffff]
ACPI: bus type pci registered
PCI: Using configuration type 1 for base access
ACPI: EC: Look up EC in DSDT
@@ -646,18 +659,18 @@
IO window: disabled.
MEM window: disabled.
PREFETCH window: disabled.
- got res [88000000:8801ffff] bus [88000000:8801ffff] flags 27200 for BAR 6 of 0000:41:01.0
- got res [88020000:8803ffff] bus [88020000:8803ffff] flags 27200 for BAR 6 of 0000:41:02.0
+ got res [d9f00000:d9f1ffff] bus [d9f00000:d9f1ffff] flags 27200 for BAR 6 of 0000:41:01.0
+ got res [d9f20000:d9f3ffff] bus [d9f20000:d9f3ffff] flags 27200 for BAR 6 of 0000:41:02.0
PCI: Bridge: 0000:40:10.0
IO window: disabled.
MEM window: 0xda000000-0xddffffff
- PREFETCH window: 0x0000000088000000-0x00000000880fffff
- got res [88200000:883fffff] bus [88200000:883fffff] flags 27200 for BAR 6 of 0000:42:01.0
- got res [88100000:8813ffff] bus [88100000:8813ffff] flags 27200 for BAR 6 of 0000:42:02.0
+ PREFETCH window: 0x00000000d9f00000-0x00000000d9ffffff
+ got res [de000000:de1fffff] bus [de000000:de1fffff] flags 27200 for BAR 6 of 0000:42:01.0
+ got res [de200000:de23ffff] bus [de200000:de23ffff] flags 27200 for BAR 6 of 0000:42:02.0
PCI: Bridge: 0000:40:11.0
IO window: 3000-3fff
MEM window: 0xdfe00000-0xdfefffff
- PREFETCH window: 0x0000000088100000-0x00000000883fffff
+ PREFETCH window: 0x00000000de000000-0x00000000de2fffff
Yinghai Lu wrote:
> dann frazier wrote:
>> --------------------------------------------------------------------
>> commit 30a18d6c3f1e774de656ebd8ff219d53e2ba4029
>> Author: Yinghai Lu <[email protected]>
>> Date: Tue Feb 19 03:21:20 2008 -0800
>>
>> x86: multi pci root bus with different io resource range, on
>> 64-bit
>> --------------------------------------------------------------------
>>
>> This appears to be the commit that actually introduced the issue.
>> I've attached dmesg w/ PCI DEBUG enabled from both sides of this
>> changeset, as well as a diff w/o printk timestamps.
>>
>> Also note that #ifdef'ing out x86_pci_root_bus_res_quirks() in
>> amd_bus.c seems to avoid the problem as well.
>
> your BIOS didn't allocate io resources to some device under HT chain on node1/link2
> aka under peer root bus.
>
> Kernel try to allocate resource to them.
>
> and the patch did right thing according to range
> bus: 40 index 1 mmio: [d9f00000, dfefffff]
>
> YH
>
>
> +bus: [00,3f] on node 0 link 1
> +bus: 00 index 0 io port: [0, 2fff]
> +bus: 00 index 1 mmio: [80000000, d9efffff]
> +bus: 00 index 2 mmio: [dff00000, e3ffffff]
> +bus: 00 index 3 mmio: [a0000, bffff]
> +bus: 00 index 4 mmio: [e8000000, ffffffff]
> +bus: 00 index 5 mmio: [480000000, fbffffffff]
> +bus: [40,7f] on node 1 link 2
> +bus: 40 index 0 io port: [3000, ffff]
> +bus: 40 index 1 mmio: [d9f00000, dfefffff]
> +bus: 40 index 2 mmio: [e4000000, e7ffffff]
> ACPI: bus type pci registered
> PCI: Using configuration type 1 for base access
> ACPI: EC: Look up EC in DSDT
> @@ -646,18 +659,18 @@
> IO window: disabled.
> MEM window: disabled.
> PREFETCH window: disabled.
> - got res [88000000:8801ffff] bus [88000000:8801ffff] flags 27200 for BAR 6 of 0000:41:01.0
> - got res [88020000:8803ffff] bus [88020000:8803ffff] flags 27200 for BAR 6 of 0000:41:02.0
> + got res [d9f00000:d9f1ffff] bus [d9f00000:d9f1ffff] flags 27200 for BAR 6 of 0000:41:01.0
> + got res [d9f20000:d9f3ffff] bus [d9f20000:d9f3ffff] flags 27200 for BAR 6 of 0000:41:02.0
> PCI: Bridge: 0000:40:10.0
> IO window: disabled.
> MEM window: 0xda000000-0xddffffff
> - PREFETCH window: 0x0000000088000000-0x00000000880fffff
> - got res [88200000:883fffff] bus [88200000:883fffff] flags 27200 for BAR 6 of 0000:42:01.0
> - got res [88100000:8813ffff] bus [88100000:8813ffff] flags 27200 for BAR 6 of 0000:42:02.0
> + PREFETCH window: 0x00000000d9f00000-0x00000000d9ffffff
> + got res [de000000:de1fffff] bus [de000000:de1fffff] flags 27200 for BAR 6 of 0000:42:01.0
> + got res [de200000:de23ffff] bus [de200000:de23ffff] flags 27200 for BAR 6 of 0000:42:02.0
> PCI: Bridge: 0000:40:11.0
> IO window: 3000-3fff
> MEM window: 0xdfe00000-0xdfefffff
> - PREFETCH window: 0x0000000088100000-0x00000000883fffff
> + PREFETCH window: 0x00000000de000000-0x00000000de2fffff
>
>
and those range are not overlapping with IOAPIC BAR allocating
[ 0.000000] ACPI: IOAPIC (id[0x08] address[0xd9cf0000] gsi_base[0])
[ 0.000000] IOAPIC[0]: apic_id 8, version 0, address 0xd9cf0000, GSI 0-23
[ 0.000000] ACPI: IOAPIC (id[0x09] address[0xd9fd0000] gsi_base[24])
[ 0.000000] IOAPIC[1]: apic_id 9, version 0, address 0xd9fd0000, GSI 24-30
[ 0.000000] ACPI: IOAPIC (id[0x0a] address[0xd9fe0000] gsi_base[31])
[ 0.000000] IOAPIC[2]: apic_id 10, version 0, address 0xd9fe0000, GSI 31-37
[ 0.000000] ACPI: IOAPIC (id[0x0b] address[0xd9ff0000] gsi_base[38])
[ 0.000000] IOAPIC[3]: apic_id 11, version 0, address 0xd9ff0000, GSI 38-61
YH
Yinghai Lu wrote:
> Yinghai Lu wrote:
>> dann frazier wrote:
>>> --------------------------------------------------------------------
>>> commit 30a18d6c3f1e774de656ebd8ff219d53e2ba4029
>>> Author: Yinghai Lu <[email protected]>
>>> Date: Tue Feb 19 03:21:20 2008 -0800
>>>
>>> x86: multi pci root bus with different io resource range, on
>>> 64-bit
>>> --------------------------------------------------------------------
>>>
>>> This appears to be the commit that actually introduced the issue.
>>> I've attached dmesg w/ PCI DEBUG enabled from both sides of this
>>> changeset, as well as a diff w/o printk timestamps.
>>>
>>> Also note that #ifdef'ing out x86_pci_root_bus_res_quirks() in
>>> amd_bus.c seems to avoid the problem as well.
>> your BIOS didn't allocate io resources to some device under HT chain on node1/link2
>> aka under peer root bus.
>>
>> Kernel try to allocate resource to them.
>>
>> and the patch did right thing according to range
>> bus: 40 index 1 mmio: [d9f00000, dfefffff]
>>
>> YH
>>
>>
>> +bus: [00,3f] on node 0 link 1
>> +bus: 00 index 0 io port: [0, 2fff]
>> +bus: 00 index 1 mmio: [80000000, d9efffff]
>> +bus: 00 index 2 mmio: [dff00000, e3ffffff]
>> +bus: 00 index 3 mmio: [a0000, bffff]
>> +bus: 00 index 4 mmio: [e8000000, ffffffff]
>> +bus: 00 index 5 mmio: [480000000, fbffffffff]
>> +bus: [40,7f] on node 1 link 2
>> +bus: 40 index 0 io port: [3000, ffff]
>> +bus: 40 index 1 mmio: [d9f00000, dfefffff]
>> +bus: 40 index 2 mmio: [e4000000, e7ffffff]
>> ACPI: bus type pci registered
>> PCI: Using configuration type 1 for base access
>> ACPI: EC: Look up EC in DSDT
>> @@ -646,18 +659,18 @@
>> IO window: disabled.
>> MEM window: disabled.
>> PREFETCH window: disabled.
>> - got res [88000000:8801ffff] bus [88000000:8801ffff] flags 27200 for BAR 6 of 0000:41:01.0
>> - got res [88020000:8803ffff] bus [88020000:8803ffff] flags 27200 for BAR 6 of 0000:41:02.0
>> + got res [d9f00000:d9f1ffff] bus [d9f00000:d9f1ffff] flags 27200 for BAR 6 of 0000:41:01.0
>> + got res [d9f20000:d9f3ffff] bus [d9f20000:d9f3ffff] flags 27200 for BAR 6 of 0000:41:02.0
>> PCI: Bridge: 0000:40:10.0
>> IO window: disabled.
>> MEM window: 0xda000000-0xddffffff
>> - PREFETCH window: 0x0000000088000000-0x00000000880fffff
>> - got res [88200000:883fffff] bus [88200000:883fffff] flags 27200 for BAR 6 of 0000:42:01.0
>> - got res [88100000:8813ffff] bus [88100000:8813ffff] flags 27200 for BAR 6 of 0000:42:02.0
>> + PREFETCH window: 0x00000000d9f00000-0x00000000d9ffffff
>> + got res [de000000:de1fffff] bus [de000000:de1fffff] flags 27200 for BAR 6 of 0000:42:01.0
>> + got res [de200000:de23ffff] bus [de200000:de23ffff] flags 27200 for BAR 6 of 0000:42:02.0
>> PCI: Bridge: 0000:40:11.0
>> IO window: 3000-3fff
>> MEM window: 0xdfe00000-0xdfefffff
>> - PREFETCH window: 0x0000000088100000-0x00000000883fffff
>> + PREFETCH window: 0x00000000de000000-0x00000000de2fffff
>>
>>
>
>
> and those range are not overlapping with IOAPIC BAR allocating
> [ 0.000000] ACPI: IOAPIC (id[0x08] address[0xd9cf0000] gsi_base[0])
> [ 0.000000] IOAPIC[0]: apic_id 8, version 0, address 0xd9cf0000, GSI 0-23
> [ 0.000000] ACPI: IOAPIC (id[0x09] address[0xd9fd0000] gsi_base[24])
> [ 0.000000] IOAPIC[1]: apic_id 9, version 0, address 0xd9fd0000, GSI 24-30
> [ 0.000000] ACPI: IOAPIC (id[0x0a] address[0xd9fe0000] gsi_base[31])
> [ 0.000000] IOAPIC[2]: apic_id 10, version 0, address 0xd9fe0000, GSI 31-37
> [ 0.000000] ACPI: IOAPIC (id[0x0b] address[0xd9ff0000] gsi_base[38])
> [ 0.000000] IOAPIC[3]: apic_id 11, version 0, address 0xd9ff0000, GSI 38-61
>
with the new allocation:
[ 4.279139] got res [d9f00000:d9f1ffff] bus [d9f00000:d9f1ffff] flags 27200 for BAR 6 of 0000:41:01.0
[ 4.279142] got res [d9f20000:d9f3ffff] bus [d9f20000:d9f3ffff] flags 27200 for BAR 6 of 0000:41:02.0
[ 4.279144] PCI: Bridge: 0000:40:10.0
[ 4.299135] IO window: disabled.
[ 4.319148] MEM window: 0xda000000-0xddffffff
[ 4.333140] PREFETCH window: 0x00000000d9f00000-0x00000000d9ffffff
[ 4.370170] got res [de000000:de1fffff] bus [de000000:de1fffff] flags 27200 for BAR 6 of 0000:42:01.0
[ 4.370173] got res [de200000:de23ffff] bus [de200000:de23ffff] flags 27200 for BAR 6 of 0000:42:02.0
[ 4.370175] PCI: Bridge: 0000:40:11.0
[ 4.390143] IO window: 3000-3fff
[ 4.410177] MEM window: 0xdfe00000-0xdfefffff
[ 4.435157] PREFETCH window: 0x00000000de000000-0x00000000de2fffff
[ 4.333140] PREFETCH window: 0x00000000d9f00000-0x00000000d9ffffff
are all directed to one 8132 PCI-X bridge.
the the IOAPIC for 9, 10, 11... falling that range...
we need one better to allocate range for that pci-x bridge to aovide ioapic BAR allocated...
anyway your BIOS sucks.
YH
On Thu, 2009-07-09 at 12:48 -0700, Yinghai Lu wrote:
> [ 4.333140] PREFETCH window: 0x00000000d9f00000-0x00000000d9ffffff
> are all directed to one 8132 PCI-X bridge.
>
> the the IOAPIC for 9, 10, 11... falling that range...
>
> we need one better to allocate range for that pci-x bridge to aovide ioapic BAR allocated...
>
> anyway your BIOS sucks.
It's a sad and very annoying fact of life that most BIOSs suck. If they
didn't suck, our bios handling code wouldn't be a nasty mess of
exceptions, work arounds and quirks.
Can we fix this so that both the DL585 and the opteron multi root stuff
can work? There's a large population of DL585's out there, so we can't
release 2.6.32 like this ... and I'd rather not have to choose between
breaking the DL585's and breaking the Sun Opterons.
James
On Thu, Jul 9, 2009 at 1:40 PM, James
Bottomley<[email protected]> wrote:
> On Thu, 2009-07-09 at 12:48 -0700, Yinghai Lu wrote:
>> [ ? ?4.333140] ? PREFETCH window: 0x00000000d9f00000-0x00000000d9ffffff
>> are all directed to one 8132 PCI-X bridge.
>>
>> the the IOAPIC for 9, 10, 11... falling that range...
>>
>> we need one better to allocate range for that pci-x bridge to aovide ioapic BAR allocated...
>>
>> anyway your BIOS sucks.
>
> It's a sad and very annoying fact of life that most BIOSs suck. ?If they
> didn't suck, our bios handling code wouldn't be a nasty mess of
> exceptions, work arounds and quirks.
>
> Can we fix this so that both the DL585 and the opteron multi root stuff
> can work? ?There's a large population of DL585's out there, so we can't
> release 2.6.32 like this ... and I'd rather not have to choose between
> breaking the DL585's and breaking the Sun Opterons.
sure. will send out one patch after get some sanity test here.
YH
Stephen reported that his DL585 G2 need noapic after 2.6.22 (?)
Bann bisected
--------------------------------------------------------------------
commit 30a18d6c3f1e774de656ebd8ff219d53e2ba4029
Date: Tue Feb 19 03:21:20 2008 -0800
x86: multi pci root bus with different io resource range, on
64-bit
--------------------------------------------------------------------
cause the problem.
it turns out that
1. that system have two HT chains.
2. BIOS doesn't allocate resource for BAR 6 under 8132 etc
3. that patches will try to split root resource to peer root resources
according to pci conf of NB
4. pci assign unassign path, assign some range to pci-x bridge, but it
is overlapping with ioapic addr of io4 and 8132.
the reason is at that point ioapic address and BAR is not inserted yet.
the BAR for io4 ioapic is not in regular position.
the BAR for 8132 is sitting to func1, and pci-x bridge is func0.
aka that patch is not the cause, and it just uncover other problems.
solution is trying to insert ioapic_resource early a little bit.
Reported-by: Stephen Frost <[email protected]>
Reported-by: dann frazier <[email protected]>
Signed-off-by: Yinghai Lu <[email protected]>
---
arch/x86/include/asm/io_apic.h | 2 ++
arch/x86/kernel/apic/io_apic.c | 14 +++-----------
arch/x86/pci/i386.c | 7 +++++++
3 files changed, 12 insertions(+), 11 deletions(-)
Index: linux-2.6/arch/x86/include/asm/io_apic.h
===================================================================
--- linux-2.6.orig/arch/x86/include/asm/io_apic.h
+++ linux-2.6/arch/x86/include/asm/io_apic.h
@@ -161,6 +161,7 @@ extern int io_apic_set_pci_routing(struc
struct io_apic_irq_attr *irq_attr);
extern int (*ioapic_renumber_irq)(int ioapic, int irq);
extern void ioapic_init_mappings(void);
+extern void ioapic_insert_resources(void);
extern struct IO_APIC_route_entry **alloc_ioapic_entries(void);
extern void free_ioapic_entries(struct IO_APIC_route_entry **ioapic_entries);
@@ -180,6 +181,7 @@ extern void ioapic_write_entry(int apic,
#define io_apic_assign_pci_irqs 0
static const int timer_through_8259 = 0;
static inline void ioapic_init_mappings(void) { }
+static inline void ioapic_insert_resources(void) { }
static inline void probe_nr_irqs_gsi(void) { }
#endif
Index: linux-2.6/arch/x86/kernel/apic/io_apic.c
===================================================================
--- linux-2.6.orig/arch/x86/kernel/apic/io_apic.c
+++ linux-2.6/arch/x86/kernel/apic/io_apic.c
@@ -4182,28 +4182,20 @@ fake_ioapic_page:
}
}
-static int __init ioapic_insert_resources(void)
+void __init ioapic_insert_resources(void)
{
int i;
struct resource *r = ioapic_resources;
if (!r) {
- if (nr_ioapics > 0) {
+ if (nr_ioapics > 0)
printk(KERN_ERR
"IO APIC resources couldn't be allocated.\n");
- return -1;
- }
- return 0;
+ return;
}
for (i = 0; i < nr_ioapics; i++) {
insert_resource(&iomem_resource, r);
r++;
}
-
- return 0;
}
-
-/* Insert the IO APIC resources after PCI initialization has occured to handle
- * IO APICS that are mapped in on a BAR in PCI space. */
-late_initcall(ioapic_insert_resources);
Index: linux-2.6/arch/x86/pci/i386.c
===================================================================
--- linux-2.6.orig/arch/x86/pci/i386.c
+++ linux-2.6/arch/x86/pci/i386.c
@@ -35,6 +35,7 @@
#include <asm/pat.h>
#include <asm/e820.h>
#include <asm/pci_x86.h>
+#include <asm/io_apic.h>
static int
@@ -227,6 +228,12 @@ void __init pcibios_resource_survey(void
pcibios_allocate_resources(1);
e820_reserve_resources_late();
+ /*
+ * Insert the IO APIC resources after PCI initialization has
+ * occured to handle IO APICS that are mapped in on a BAR in
+ * PCI space, but before we trying to assign unassigned pci res.
+ */
+ ioapic_insert_resources();
}
/**
Thanks Yinghai!
Tested-by: dann frazier <[email protected]>
On Thu, Jul 09, 2009 at 02:41:47PM -0700, Yinghai Lu wrote:
>
> Stephen reported that his DL585 G2 need noapic after 2.6.22 (?)
>
> Bann bisected
s/Bann/Dann/ :)
> --------------------------------------------------------------------
> commit 30a18d6c3f1e774de656ebd8ff219d53e2ba4029
> Date: Tue Feb 19 03:21:20 2008 -0800
>
> x86: multi pci root bus with different io resource range, on
> 64-bit
> --------------------------------------------------------------------
> cause the problem.
>
> it turns out that
> 1. that system have two HT chains.
> 2. BIOS doesn't allocate resource for BAR 6 under 8132 etc
> 3. that patches will try to split root resource to peer root resources
> according to pci conf of NB
> 4. pci assign unassign path, assign some range to pci-x bridge, but it
> is overlapping with ioapic addr of io4 and 8132.
>
> the reason is at that point ioapic address and BAR is not inserted yet.
> the BAR for io4 ioapic is not in regular position.
> the BAR for 8132 is sitting to func1, and pci-x bridge is func0.
>
> aka that patch is not the cause, and it just uncover other problems.
>
> solution is trying to insert ioapic_resource early a little bit.
>
> Reported-by: Stephen Frost <[email protected]>
> Reported-by: dann frazier <[email protected]>
> Signed-off-by: Yinghai Lu <[email protected]>
>
> ---
> arch/x86/include/asm/io_apic.h | 2 ++
> arch/x86/kernel/apic/io_apic.c | 14 +++-----------
> arch/x86/pci/i386.c | 7 +++++++
> 3 files changed, 12 insertions(+), 11 deletions(-)
>
> Index: linux-2.6/arch/x86/include/asm/io_apic.h
> ===================================================================
> --- linux-2.6.orig/arch/x86/include/asm/io_apic.h
> +++ linux-2.6/arch/x86/include/asm/io_apic.h
> @@ -161,6 +161,7 @@ extern int io_apic_set_pci_routing(struc
> struct io_apic_irq_attr *irq_attr);
> extern int (*ioapic_renumber_irq)(int ioapic, int irq);
> extern void ioapic_init_mappings(void);
> +extern void ioapic_insert_resources(void);
>
> extern struct IO_APIC_route_entry **alloc_ioapic_entries(void);
> extern void free_ioapic_entries(struct IO_APIC_route_entry **ioapic_entries);
> @@ -180,6 +181,7 @@ extern void ioapic_write_entry(int apic,
> #define io_apic_assign_pci_irqs 0
> static const int timer_through_8259 = 0;
> static inline void ioapic_init_mappings(void) { }
> +static inline void ioapic_insert_resources(void) { }
>
> static inline void probe_nr_irqs_gsi(void) { }
> #endif
> Index: linux-2.6/arch/x86/kernel/apic/io_apic.c
> ===================================================================
> --- linux-2.6.orig/arch/x86/kernel/apic/io_apic.c
> +++ linux-2.6/arch/x86/kernel/apic/io_apic.c
> @@ -4182,28 +4182,20 @@ fake_ioapic_page:
> }
> }
>
> -static int __init ioapic_insert_resources(void)
> +void __init ioapic_insert_resources(void)
> {
> int i;
> struct resource *r = ioapic_resources;
>
> if (!r) {
> - if (nr_ioapics > 0) {
> + if (nr_ioapics > 0)
> printk(KERN_ERR
> "IO APIC resources couldn't be allocated.\n");
> - return -1;
> - }
> - return 0;
> + return;
> }
>
> for (i = 0; i < nr_ioapics; i++) {
> insert_resource(&iomem_resource, r);
> r++;
> }
> -
> - return 0;
> }
> -
> -/* Insert the IO APIC resources after PCI initialization has occured to handle
> - * IO APICS that are mapped in on a BAR in PCI space. */
> -late_initcall(ioapic_insert_resources);
> Index: linux-2.6/arch/x86/pci/i386.c
> ===================================================================
> --- linux-2.6.orig/arch/x86/pci/i386.c
> +++ linux-2.6/arch/x86/pci/i386.c
> @@ -35,6 +35,7 @@
> #include <asm/pat.h>
> #include <asm/e820.h>
> #include <asm/pci_x86.h>
> +#include <asm/io_apic.h>
>
>
> static int
> @@ -227,6 +228,12 @@ void __init pcibios_resource_survey(void
> pcibios_allocate_resources(1);
>
> e820_reserve_resources_late();
> + /*
> + * Insert the IO APIC resources after PCI initialization has
> + * occured to handle IO APICS that are mapped in on a BAR in
> + * PCI space, but before we trying to assign unassigned pci res.
> + */
> + ioapic_insert_resources();
> }
>
> /**
>
--
dann frazier
Stephen reported that his DL585 G2 need noapic after 2.6.22 (?)
Dann bisected
--------------------------------------------------------------------
commit 30a18d6c3f1e774de656ebd8ff219d53e2ba4029
Date: Tue Feb 19 03:21:20 2008 -0800
x86: multi pci root bus with different io resource range, on
64-bit
--------------------------------------------------------------------
caused the problem.
it turns out that
1. that AMD-based system has two HT chains.
2. BIOS doesn't allocate resource for BAR 6 of devices under 8132 etc
3. that multi-peer-root patche will try to split root resource to peer
root resources according to pci conf of NB
4. pci assign unassigned code assign some range to pci-x bridge, but it
is overlapping BAR that are used by ioapic addr of io4 and 8132.
the reason: at that point ioapic address are not inserted yet.
aka that patch is not the cause, and it just uncover other problems.
solution is trying to insert ioapic_resource early a little bit.
v2: update comments
Reported-by: Stephen Frost <[email protected]>
Reported-and-Tested-by: dann frazier <[email protected]>
Signed-off-by: Yinghai Lu <[email protected]>
Cc: [email protected]
---
arch/x86/include/asm/io_apic.h | 2 ++
arch/x86/kernel/apic/io_apic.c | 14 +++-----------
arch/x86/pci/i386.c | 7 +++++++
3 files changed, 12 insertions(+), 11 deletions(-)
Index: linux-2.6/arch/x86/include/asm/io_apic.h
===================================================================
--- linux-2.6.orig/arch/x86/include/asm/io_apic.h
+++ linux-2.6/arch/x86/include/asm/io_apic.h
@@ -161,6 +161,7 @@ extern int io_apic_set_pci_routing(struc
struct io_apic_irq_attr *irq_attr);
extern int (*ioapic_renumber_irq)(int ioapic, int irq);
extern void ioapic_init_mappings(void);
+extern void ioapic_insert_resources(void);
extern struct IO_APIC_route_entry **alloc_ioapic_entries(void);
extern void free_ioapic_entries(struct IO_APIC_route_entry **ioapic_entries);
@@ -180,6 +181,7 @@ extern void ioapic_write_entry(int apic,
#define io_apic_assign_pci_irqs 0
static const int timer_through_8259 = 0;
static inline void ioapic_init_mappings(void) { }
+static inline void ioapic_insert_resources(void) { }
static inline void probe_nr_irqs_gsi(void) { }
#endif
Index: linux-2.6/arch/x86/kernel/apic/io_apic.c
===================================================================
--- linux-2.6.orig/arch/x86/kernel/apic/io_apic.c
+++ linux-2.6/arch/x86/kernel/apic/io_apic.c
@@ -4181,28 +4181,20 @@ fake_ioapic_page:
}
}
-static int __init ioapic_insert_resources(void)
+void __init ioapic_insert_resources(void)
{
int i;
struct resource *r = ioapic_resources;
if (!r) {
- if (nr_ioapics > 0) {
+ if (nr_ioapics > 0)
printk(KERN_ERR
"IO APIC resources couldn't be allocated.\n");
- return -1;
- }
- return 0;
+ return;
}
for (i = 0; i < nr_ioapics; i++) {
insert_resource(&iomem_resource, r);
r++;
}
-
- return 0;
}
-
-/* Insert the IO APIC resources after PCI initialization has occured to handle
- * IO APICS that are mapped in on a BAR in PCI space. */
-late_initcall(ioapic_insert_resources);
Index: linux-2.6/arch/x86/pci/i386.c
===================================================================
--- linux-2.6.orig/arch/x86/pci/i386.c
+++ linux-2.6/arch/x86/pci/i386.c
@@ -35,6 +35,7 @@
#include <asm/pat.h>
#include <asm/e820.h>
#include <asm/pci_x86.h>
+#include <asm/io_apic.h>
static int
@@ -227,6 +228,12 @@ void __init pcibios_resource_survey(void
pcibios_allocate_resources(1);
e820_reserve_resources_late();
+ /*
+ * Insert the IO APIC resources after PCI initialization has
+ * occured to handle IO APICS that are mapped in on a BAR in
+ * PCI space, but before trying to assign unassigned pci res.
+ */
+ ioapic_insert_resources();
}
/**
On Fri, 10 Jul 2009 09:36:20 -0700
Yinghai Lu <[email protected]> wrote:
>
> Stephen reported that his DL585 G2 need noapic after 2.6.22 (?)
>
> Dann bisected
> --------------------------------------------------------------------
> commit 30a18d6c3f1e774de656ebd8ff219d53e2ba4029
> Date: Tue Feb 19 03:21:20 2008 -0800
>
> x86: multi pci root bus with different io resource range, on
> 64-bit
> --------------------------------------------------------------------
> caused the problem.
>
Applied to my for-linus tree, thanks Yinghai.
--
Jesse Barnes, Intel Open Source Technology Center
On Fri, 10 Jul 2009, Yinghai Lu wrote:
>
> solution is trying to insert ioapic_resource early a little bit.
>
> v2: update comments
Ack, looks sane to me.
Linus