Hi All,
I'm in the process of updating the kernel version used on our products
from 4.4 -> 5.1.
We have one product that uses a Kirkwood CPU, IDT PCI bridge and Marvell
Switch ASIC. The Switch ASIC presents as multiple PCI devices.
The hardware setup looks like this
__________
[ Kirkwood ] --- [ IDT 5T5 ] ---+--- | |
+--- | Switch |
+--- | |
+--- |__________|
On the 4.4 based kernel things are fine
[root@awplus flash]# lspci -t
-[0000:00]---01.0-[01-06]----00.0-[02-06]--+-02.0-[03]----00.0
+-03.0-[04]----00.0
+-04.0-[05]----00.0
\-05.0-[06]----00.0
But on the 5.1 based kernel things get a little weird
[root@awplus flash]# lspci -t
-[0000:00]---01.0-[01-06]--+-00.0-[02-06]--
+-01.0
+-02.0-[02-06]--
+-03.0-[02-06]--
+-04.0-[02-06]--
+-05.0-[02-06]--
+-06.0-[02-06]--
+-07.0-[02-06]--
+-08.0-[02-06]--
+-09.0-[02-06]--
+-0a.0-[02-06]--
+-0b.0-[02-06]--
+-0c.0-[02-06]--
+-0d.0-[02-06]--
+-0e.0-[02-06]--
+-0f.0-[02-06]--
+-10.0-[02-06]--
+-11.0-[02-06]--
+-12.0-[02-06]--
+-13.0-[02-06]--
+-14.0-[02-06]--
+-15.0-[02-06]--
+-16.0-[02-06]--
+-17.0-[02-06]--
+-18.0-[02-06]--
+-19.0-[02-06]--
+-1a.0-[02-06]--
+-1b.0-[02-06]--
+-1c.0-[02-06]--
+-1d.0-[02-06]--
+-1e.0-[02-06]--
\-1f.0-[02-06]--+-02.0-[03]----00.0
+-03.0-[04]----00.0
+-04.0-[05]----00.0
\-05.0-[06]----00.0
I'll start bisecting to see where things started going wrong. I just
wondered if this rings any bells for anyone.
The startup output also seems to be quite unhappy
Detected board: alliedtelesis,SBx81GC40
Booting into Linux kernel ...
** 143 printk messages dropped **
pci 0000:01:19.0: PME# supported from D0 D3hot D3cold
pci 0000:01:1a.0: [111d:803c] type 01 class 0x060400
pci 0000:01:1a.0: PME# supported from D0 D3hot D3cold
pci 0000:01:1b.0: [111d:803c] type 01 class 0x060400
pci 0000:01:1b.0: PME# supported from D0 D3hot D3cold
pci 0000:01:1c.0: [111d:803c] type 01 class 0x060400
pci 0000:01:1c.0: PME# supported from D0 D3hot D3cold
pci 0000:01:1d.0: [111d:803c] type 01 class 0x060400
pci 0000:01:1d.0: PME# supported from D0 D3hot D3cold
pci 0000:01:1e.0: [111d:803c] type 01 class 0x060400
pci 0000:01:1e.0: PME# supported from D0 D3hot D3cold
pci 0000:01:1f.0: [111d:803c] type 01 class 0x060400
pci 0000:01:1f.0: PME# supported from D0 D3hot D3cold
PCI: bus1: Fast back to back transfers disabled
pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:04.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:05.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:06.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:07.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:08.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:09.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:0a.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:0b.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:0c.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:0d.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:0e.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:0f.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:10.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:11.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:12.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:13.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:14.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:15.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:16.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:17.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:18.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:19.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:1a.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:1b.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:1c.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:1d.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:1e.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:01:1f.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:02:02.0: [111d:803c] type 01 class 0x060400
pci 0000:02:02.0: enabling Extended Tags
pci 0000:02:02.0: PME# supported from D0 D3hot D3cold
pci 0000:02:03.0: [111d:803c] type 01 class 0x060400
pci 0000:02:03.0: enabling Extended Tags
pci 0000:02:03.0: PME# supported from D0 D3hot D3cold
pci 0000:02:04.0: [111d:803c] type 01 class 0x060400
pci 0000:02:04.0: enabling Extended Tags
pci 0000:02:04.0: PME# supported from D0 D3hot D3cold
pci 0000:02:05.0: [111d:803c] type 01 class 0x060400
pci 0000:02:05.0: enabling Extended Tags
pci 0000:02:05.0: PME# supported from D0 D3hot D3cold
PCI: bus2: Fast back to back transfers disabled
pci 0000:02:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:02:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:02:04.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:02:05.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:03:00.0: [11ab:e023] type 00 class 0x058000
pci 0000:03:00.0: reg 0x10: [mem 0xd0000000-0xd00fffff 64bit pref]
pci 0000:03:00.0: reg 0x18: [mem 0x00000000-0x03ffffff 64bit]
PCI: bus3: Fast back to back transfers disabled
pci_bus 0000:03: busn_res: [bus 03-ff] end is updated to 03
pci 0000:04:00.0: [11ab:e023] type 00 class 0x058000
pci 0000:04:00.0: reg 0x10: [mem 0xd0000000-0xd00fffff 64bit pref]
pci 0000:04:00.0: reg 0x18: [mem 0x00000000-0x03ffffff 64bit]
PCI: bus4: Fast back to back transfers disabled
pci_bus 0000:04: busn_res: [bus 04-ff] end is updated to 04
pci 0000:05:00.0: [11ab:e023] type 00 class 0x058000
pci 0000:05:00.0: reg 0x10: [mem 0xd0000000-0xd00fffff 64bit pref]
pci 0000:05:00.0: reg 0x18: [mem 0x00000000-0x03ffffff 64bit]
PCI: bus5: Fast back to back transfers disabled
pci_bus 0000:05: busn_res: [bus 05-ff] end is updated to 05
pci 0000:06:00.0: [11ab:e023] type 00 class 0x058000
pci 0000:06:00.0: reg 0x10: [mem 0xd0000000-0xd00fffff 64bit pref]
pci 0000:06:00.0: reg 0x18: [mem 0x00000000-0x03ffffff 64bit]
PCI: bus6: Fast back to back transfers disabled
pci_bus 0000:06: busn_res: [bus 06-ff] end is updated to 06
pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 06
pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 06
pci 0000:00:01.0: BAR 8: no space for [mem size 0x1c000000]
pci 0000:00:01.0: BAR 8: failed to assign [mem size 0x1c000000]
pci 0000:00:01.0: BAR 6: assigned [mem 0xe0000000-0xe00007ff pref]
pci 0000:01:01.0: BAR 2: no space for [mem size 0x08000000]
pci 0000:01:01.0: BAR 2: failed to assign [mem size 0x08000000]
pci 0000:01:00.0: BAR 8: no space for [mem size 0x10000000]
pci 0000:01:00.0: BAR 8: failed to assign [mem size 0x10000000]
pci 0000:01:00.0: BAR 9: no space for [mem size 0x00400000 64bit pref]
pci 0000:01:00.0: BAR 9: failed to assign [mem size 0x00400000 64bit pref]
pci 0000:01:01.0: BAR 0: no space for [mem size 0x00100000 64bit pref]
pci 0000:01:01.0: BAR 0: failed to assign [mem size 0x00100000 64bit pref]
pci 0000:02:02.0: BAR 8: no space for [mem size 0x04000000]
pci 0000:02:02.0: BAR 8: failed to assign [mem size 0x04000000]
pci 0000:02:03.0: BAR 8: no space for [mem size 0x04000000]
pci 0000:02:03.0: BAR 8: failed to assign [mem size 0x04000000]
pci 0000:02:04.0: BAR 8: no space for [mem size 0x04000000]
pci 0000:02:04.0: BAR 8: failed to assign [mem size 0x04000000]
pci 0000:02:05.0: BAR 8: no space for [mem size 0x04000000]
pci 0000:02:05.0: BAR 8: failed to assign [mem size 0x04000000]
pci 0000:02:02.0: BAR 9: no space for [mem size 0x00100000 64bit pref]
pci 0000:02:02.0: BAR 9: failed to assign [mem size 0x00100000 64bit pref]
pci 0000:02:03.0: BAR 9: no space for [mem size 0x00100000 64bit pref]
pci 0000:02:03.0: BAR 9: failed to assign [mem size 0x00100000 64bit pref]
pci 0000:02:04.0: BAR 9: no space for [mem size 0x00100000 64bit pref]
pci 0000:02:04.0: BAR 9: failed to assign [mem size 0x00100000 64bit pref]
pci 0000:02:05.0: BAR 9: no space for [mem size 0x00100000 64bit pref]
pci 0000:02:05.0: BAR 9: failed to assign [mem size 0x00100000 64bit pref]
pci 0000:03:00.0: BAR 2: no space for [mem size 0x04000000 64bit]
pci 0000:03:00.0: BAR 2: failed to assign [mem size 0x04000000 64bit]
pci 0000:03:00.0: BAR 0: no space for [mem size 0x00100000 64bit pref]
pci 0000:03:00.0: BAR 0: failed to assign [mem size 0x00100000 64bit pref]
pci 0000:02:02.0: PCI bridge to [bus 03]
pci 0000:04:00.0: BAR 2: no space for [mem size 0x04000000 64bit]
pci 0000:04:00.0: BAR 2: failed to assign [mem size 0x04000000 64bit]
pci 0000:04:00.0: BAR 0: no space for [mem size 0x00100000 64bit pref]
pci 0000:04:00.0: BAR 0: failed to assign [mem size 0x00100000 64bit pref]
pci 0000:02:03.0: PCI bridge to [bus 04]
pci 0000:05:00.0: BAR 2: no space for [mem size 0x04000000 64bit]
pci 0000:05:00.0: BAR 2: failed to assign [mem size 0x04000000 64bit]
pci 0000:05:00.0: BAR 0: no space for [mem size 0x00100000 64bit pref]
pci 0000:05:00.0: BAR 0: failed to assign [mem size 0x00100000 64bit pref]
pci 0000:02:04.0: PCI bridge to [bus 05]
pci 0000:06:00.0: BAR 2: no space for [mem size 0x04000000 64bit]
pci 0000:06:00.0: BAR 2: failed to assign [mem size 0x04000000 64bit]
pci 0000:06:00.0: BAR 0: no space for [mem size 0x00100000 64bit pref]
pci 0000:06:00.0: BAR 0: failed to assign [mem size 0x00100000 64bit pref]
pci 0000:02:05.0: PCI bridge to [bus 06]
pci 0000:01:00.0: PCI bridge to [bus 02-06]
pci 0000:00:01.0: PCI bridge to [bus 01-06]
Hello Chris,
On Fri, 21 Jun 2019 04:03:27 +0000
Chris Packham <[email protected]> wrote:
> I'm in the process of updating the kernel version used on our products
> from 4.4 -> 5.1.
>
> We have one product that uses a Kirkwood CPU, IDT PCI bridge and Marvell
> Switch ASIC. The Switch ASIC presents as multiple PCI devices.
>
> The hardware setup looks like this
> __________
> [ Kirkwood ] --- [ IDT 5T5 ] ---+--- | |
> +--- | Switch |
> +--- | |
> +--- |__________|
>
> On the 4.4 based kernel things are fine
>
> [root@awplus flash]# lspci -t
> -[0000:00]---01.0-[01-06]----00.0-[02-06]--+-02.0-[03]----00.0
> +-03.0-[04]----00.0
> +-04.0-[05]----00.0
> \-05.0-[06]----00.0
>
> But on the 5.1 based kernel things get a little weird
>
> [root@awplus flash]# lspci -t
> -[0000:00]---01.0-[01-06]--+-00.0-[02-06]--
> +-01.0
> +-02.0-[02-06]--
> +-03.0-[02-06]--
> +-04.0-[02-06]--
> +-05.0-[02-06]--
> +-06.0-[02-06]--
> +-07.0-[02-06]--
> +-08.0-[02-06]--
> +-09.0-[02-06]--
> +-0a.0-[02-06]--
> +-0b.0-[02-06]--
> +-0c.0-[02-06]--
> +-0d.0-[02-06]--
> +-0e.0-[02-06]--
> +-0f.0-[02-06]--
> +-10.0-[02-06]--
> +-11.0-[02-06]--
> +-12.0-[02-06]--
> +-13.0-[02-06]--
> +-14.0-[02-06]--
> +-15.0-[02-06]--
> +-16.0-[02-06]--
> +-17.0-[02-06]--
> +-18.0-[02-06]--
> +-19.0-[02-06]--
> +-1a.0-[02-06]--
> +-1b.0-[02-06]--
> +-1c.0-[02-06]--
> +-1d.0-[02-06]--
> +-1e.0-[02-06]--
> \-1f.0-[02-06]--+-02.0-[03]----00.0
> +-03.0-[04]----00.0
> +-04.0-[05]----00.0
> \-05.0-[06]----00.0
>
>
> I'll start bisecting to see where things started going wrong. I just
> wondered if this rings any bells for anyone.
I am almost sure that the culprit is
1f08673eef1236f7d02d93fcf596bb8531ef0d12 ("PCI: mvebu: Convert to PCI
emulated bridge config space").
I still think it makes sense to share the bridge emulation code between
the mvebu and aardvark drivers, but this sharing has required making
the code very different, with lots of subtle differences in behavior in
how registers are emulated.
Unfortunately, I don't have access to one of these complicated PCI
setup with a HW switch on the way, so I couldn't test this kind of
setups.
Do you mind helping with figuring out what the issues are ? That would
be really nice.
Best regards,
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
Hi Thomas,
On 21/06/19 6:17 PM, Thomas Petazzoni wrote:
> Hello Chris,
>
> On Fri, 21 Jun 2019 04:03:27 +0000
> Chris Packham <[email protected]> wrote:
>
>> I'm in the process of updating the kernel version used on our products
>> from 4.4 -> 5.1.
>>
>> We have one product that uses a Kirkwood CPU, IDT PCI bridge and Marvell
>> Switch ASIC. The Switch ASIC presents as multiple PCI devices.
>>
>> The hardware setup looks like this
>> __________
>> [ Kirkwood ] --- [ IDT 5T5 ] ---+--- | |
>> +--- | Switch |
>> +--- | |
>> +--- |__________|
>>
>> On the 4.4 based kernel things are fine
>>
>> [root@awplus flash]# lspci -t
>> -[0000:00]---01.0-[01-06]----00.0-[02-06]--+-02.0-[03]----00.0
>> +-03.0-[04]----00.0
>> +-04.0-[05]----00.0
>> \-05.0-[06]----00.0
>>
>> But on the 5.1 based kernel things get a little weird
>>
>> [root@awplus flash]# lspci -t
>> -[0000:00]---01.0-[01-06]--+-00.0-[02-06]--
>> +-01.0
>> +-02.0-[02-06]--
>> +-03.0-[02-06]--
>> +-04.0-[02-06]--
>> +-05.0-[02-06]--
>> +-06.0-[02-06]--
>> +-07.0-[02-06]--
>> +-08.0-[02-06]--
>> +-09.0-[02-06]--
>> +-0a.0-[02-06]--
>> +-0b.0-[02-06]--
>> +-0c.0-[02-06]--
>> +-0d.0-[02-06]--
>> +-0e.0-[02-06]--
>> +-0f.0-[02-06]--
>> +-10.0-[02-06]--
>> +-11.0-[02-06]--
>> +-12.0-[02-06]--
>> +-13.0-[02-06]--
>> +-14.0-[02-06]--
>> +-15.0-[02-06]--
>> +-16.0-[02-06]--
>> +-17.0-[02-06]--
>> +-18.0-[02-06]--
>> +-19.0-[02-06]--
>> +-1a.0-[02-06]--
>> +-1b.0-[02-06]--
>> +-1c.0-[02-06]--
>> +-1d.0-[02-06]--
>> +-1e.0-[02-06]--
>> \-1f.0-[02-06]--+-02.0-[03]----00.0
>> +-03.0-[04]----00.0
>> +-04.0-[05]----00.0
>> \-05.0-[06]----00.0
>>
>>
>> I'll start bisecting to see where things started going wrong. I just
>> wondered if this rings any bells for anyone.
>
> I am almost sure that the culprit is
> 1f08673eef1236f7d02d93fcf596bb8531ef0d12 ("PCI: mvebu: Convert to PCI
> emulated bridge config space").
The problem seems to pre-date this commit. I've gone back as far as 4.18
and the problem still exists (in fact there are more duplicate devices).
I'll keep going back (unfortunately due to out platform being out of
tree it's not a simple bisect).
> I still think it makes sense to share the bridge emulation code between
> the mvebu and aardvark drivers, but this sharing has required making
> the code very different, with lots of subtle differences in behavior in
> how registers are emulated.
Agreed. Bugs love to hide in duplicated code.
I will admit to being ignorant about the need for an emulated bridge. I
know it has something to do with the type of transaction used for the
downstream devices. I also know that these systems won't work without an
emulated bridge.
> Unfortunately, I don't have access to one of these complicated PCI
> setup with a HW switch on the way, so I couldn't test this kind of
> setups.
>
> Do you mind helping with figuring out what the issues are ? That would
> be really nice.
No problem. As I said I'll keep going to find a point where behaviour
turns bad for me. I suspect we might find other problems along the way.
On 24/06/19 4:08 PM, Chris Packham wrote:
> Hi Thomas,
>
> On 21/06/19 6:17 PM, Thomas Petazzoni wrote:
>> Hello Chris,
>>
>> On Fri, 21 Jun 2019 04:03:27 +0000
>> Chris Packham <[email protected]> wrote:
>>
>>> I'm in the process of updating the kernel version used on our products
>>> from 4.4 -> 5.1.
>>>
>>> We have one product that uses a Kirkwood CPU, IDT PCI bridge and Marvell
>>> Switch ASIC. The Switch ASIC presents as multiple PCI devices.
>>>
>>> The hardware setup looks like this
>>> __________
>>> [ Kirkwood ] --- [ IDT 5T5 ] ---+--- | |
>>> +--- | Switch |
>>> +--- | |
>>> +--- |__________|
>>>
>>> On the 4.4 based kernel things are fine
>>>
>>> [root@awplus flash]# lspci -t
>>> -[0000:00]---01.0-[01-06]----00.0-[02-06]--+-02.0-[03]----00.0
>>> +-03.0-[04]----00.0
>>> +-04.0-[05]----00.0
>>> \-05.0-[06]----00.0
>>>
>>> But on the 5.1 based kernel things get a little weird
>>>
>>> [root@awplus flash]# lspci -t
>>> -[0000:00]---01.0-[01-06]--+-00.0-[02-06]--
>>> +-01.0
>>> +-02.0-[02-06]--
>>> +-03.0-[02-06]--
>>> +-04.0-[02-06]--
>>> +-05.0-[02-06]--
>>> +-06.0-[02-06]--
>>> +-07.0-[02-06]--
>>> +-08.0-[02-06]--
>>> +-09.0-[02-06]--
>>> +-0a.0-[02-06]--
>>> +-0b.0-[02-06]--
>>> +-0c.0-[02-06]--
>>> +-0d.0-[02-06]--
>>> +-0e.0-[02-06]--
>>> +-0f.0-[02-06]--
>>> +-10.0-[02-06]--
>>> +-11.0-[02-06]--
>>> +-12.0-[02-06]--
>>> +-13.0-[02-06]--
>>> +-14.0-[02-06]--
>>> +-15.0-[02-06]--
>>> +-16.0-[02-06]--
>>> +-17.0-[02-06]--
>>> +-18.0-[02-06]--
>>> +-19.0-[02-06]--
>>> +-1a.0-[02-06]--
>>> +-1b.0-[02-06]--
>>> +-1c.0-[02-06]--
>>> +-1d.0-[02-06]--
>>> +-1e.0-[02-06]--
>>> \-1f.0-[02-06]--+-02.0-[03]----00.0
>>> +-03.0-[04]----00.0
>>> +-04.0-[05]----00.0
>>> \-05.0-[06]----00.0
>>>
>>>
>>> I'll start bisecting to see where things started going wrong. I just
>>> wondered if this rings any bells for anyone.
>>
>> I am almost sure that the culprit is
>> 1f08673eef1236f7d02d93fcf596bb8531ef0d12 ("PCI: mvebu: Convert to PCI
>> emulated bridge config space").
>
> The problem seems to pre-date this commit. I've gone back as far as 4.18
> and the problem still exists (in fact there are more duplicate devices).
> I'll keep going back (unfortunately due to out platform being out of
> tree it's not a simple bisect).
>
>> I still think it makes sense to share the bridge emulation code between
>> the mvebu and aardvark drivers, but this sharing has required making
>> the code very different, with lots of subtle differences in behavior in
>> how registers are emulated.
>
> Agreed. Bugs love to hide in duplicated code.
>
> I will admit to being ignorant about the need for an emulated bridge. I
> know it has something to do with the type of transaction used for the
> downstream devices. I also know that these systems won't work without an
> emulated bridge.
>
>> Unfortunately, I don't have access to one of these complicated PCI
>> setup with a HW switch on the way, so I couldn't test this kind of
>> setups.
>>
>> Do you mind helping with figuring out what the issues are ? That would
>> be really nice.
>
> No problem. As I said I'll keep going to find a point where behaviour
> turns bad for me. I suspect we might find other problems along the way.
>
Some progress. Our defconfig had CONFIG_CMDLINE="pci=pcie_scan_all" in
it. This dated back to before we were using a devicetree with our
kirkwood platforms. At some point this started having an effect on the
emulated bridge.