Hello,
I am trying to figure-out an issue on our i.MX6Q platform based design
where PCIe interface goes bad.
We have a Phytec i.MX6Q eMMC SOM, attached to our custom designed
board. PCIe root-complex from i.MX6Q is attached to PLX switch
(PEX8605).
Linux kernel version is 4.19.9x and also 4.14.134 (from phytec's
linux-mainline repo). Kernel do not have PCIe hot-plug and PNP enabled
in config.
PLX switch #PERST is attached to a GPIO pin and stays in disable state
until Linux is booted. So at boot time only PCIe root-complex is
initialized by kernel.
After boot if I do "lspci -v" and see everything good from PCIe
root-complex (below):
~ # lspci -v
00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
[Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 295
Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: None
Memory behind bridge: None
Prefetchable memory behind bridge: None
[virtual] Expansion ROM at 01100000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Kernel driver in use: pcieport
Then I enable the #PERST pin of PLX switch, everything is still good
(no rescan on Linux is done yet)
~ # echo 139 > /sys/class/gpio/export
~ # echo out > /sys/class/gpio/gpio139/direction
~ # echo 1 > /sys/class/gpio/gpio139/value
~ # lspci -v
00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
[Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 295
Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: None
Memory behind bridge: None
Prefetchable memory behind bridge: None
[virtual] Expansion ROM at 01100000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Kernel driver in use: pcieport
Now just disable/put-in-reset the PLX switch (Linux don't see the
switch yet, as no rescan on PCIe was done). Now "lspci -v" and
root-complex goes bad.
~ # echo 0 > /sys/class/gpio/gpio139/value
~ # lspci -v
00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
[Normal decode])
Flags: fast devsel, IRQ 295
Memory at 01000000 (64-bit, prefetchable) [disabled] [size=1M]
Bus: primary=00, secondary=00, subordinate=00, sec-latency=0
I/O behind bridge: 00000000-00000fff [size=4K]
Memory behind bridge: 00000000-000fffff [size=1M]
Prefetchable memory behind bridge: 00000000-000fffff [size=1M]
[virtual] Expansion ROM at 01100000 [disabled] [size=64K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Kernel driver in use: pcieport
~ # uname -a
Linux buildroot-2019.08-imx6 4.14.134-phy2 #1 SMP Thu Feb 20 12:13:33
UTC 2020 armv7l GNU/Linux
~ #
I am really not sure what is going wrong here. Did I am missing
something basic?
Thanks in advance,
-- Fawad Lateef
Hi again,
Can someone guide me what is going on?
Thanks,
-- Fawad Lateef
On Sat, 22 Feb 2020 at 16:25, Fawad Lateef <[email protected]> wrote:
>
> Hello,
>
> I am trying to figure-out an issue on our i.MX6Q platform based design
> where PCIe interface goes bad.
>
> We have a Phytec i.MX6Q eMMC SOM, attached to our custom designed
> board. PCIe root-complex from i.MX6Q is attached to PLX switch
> (PEX8605).
>
> Linux kernel version is 4.19.9x and also 4.14.134 (from phytec's
> linux-mainline repo). Kernel do not have PCIe hot-plug and PNP enabled
> in config.
>
> PLX switch #PERST is attached to a GPIO pin and stays in disable state
> until Linux is booted. So at boot time only PCIe root-complex is
> initialized by kernel.
>
> After boot if I do "lspci -v" and see everything good from PCIe
> root-complex (below):
>
> ~ # lspci -v
> 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> [Normal decode])
> Flags: bus master, fast devsel, latency 0, IRQ 295
> Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
> Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> I/O behind bridge: None
> Memory behind bridge: None
> Prefetchable memory behind bridge: None
> [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> Capabilities: [40] Power Management version 3
> Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
> Capabilities: [70] Express Root Port (Slot-), MSI 00
> Capabilities: [100] Advanced Error Reporting
> Capabilities: [140] Virtual Channel
> Kernel driver in use: pcieport
>
>
> Then I enable the #PERST pin of PLX switch, everything is still good
> (no rescan on Linux is done yet)
>
> ~ # echo 139 > /sys/class/gpio/export
> ~ # echo out > /sys/class/gpio/gpio139/direction
> ~ # echo 1 > /sys/class/gpio/gpio139/value
> ~ # lspci -v
> 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> [Normal decode])
> Flags: bus master, fast devsel, latency 0, IRQ 295
> Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
> Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> I/O behind bridge: None
> Memory behind bridge: None
> Prefetchable memory behind bridge: None
> [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> Capabilities: [40] Power Management version 3
> Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
> Capabilities: [70] Express Root Port (Slot-), MSI 00
> Capabilities: [100] Advanced Error Reporting
> Capabilities: [140] Virtual Channel
> Kernel driver in use: pcieport
>
>
> Now just disable/put-in-reset the PLX switch (Linux don't see the
> switch yet, as no rescan on PCIe was done). Now "lspci -v" and
> root-complex goes bad.
>
> ~ # echo 0 > /sys/class/gpio/gpio139/value
> ~ # lspci -v
> 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> [Normal decode])
> Flags: fast devsel, IRQ 295
> Memory at 01000000 (64-bit, prefetchable) [disabled] [size=1M]
> Bus: primary=00, secondary=00, subordinate=00, sec-latency=0
> I/O behind bridge: 00000000-00000fff [size=4K]
> Memory behind bridge: 00000000-000fffff [size=1M]
> Prefetchable memory behind bridge: 00000000-000fffff [size=1M]
> [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> Capabilities: [40] Power Management version 3
> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
> Capabilities: [70] Express Root Port (Slot-), MSI 00
> Capabilities: [100] Advanced Error Reporting
> Capabilities: [140] Virtual Channel
> Kernel driver in use: pcieport
>
> ~ # uname -a
> Linux buildroot-2019.08-imx6 4.14.134-phy2 #1 SMP Thu Feb 20 12:13:33
> UTC 2020 armv7l GNU/Linux
> ~ #
>
>
> I am really not sure what is going wrong here. Did I am missing
> something basic?
>
> Thanks in advance,
>
> -- Fawad Lateef
On Sat, Feb 22, 2020 at 04:25:41PM +0100, Fawad Lateef wrote:
> Hello,
>
> I am trying to figure-out an issue on our i.MX6Q platform based design
> where PCIe interface goes bad.
>
> We have a Phytec i.MX6Q eMMC SOM, attached to our custom designed
> board. PCIe root-complex from i.MX6Q is attached to PLX switch
> (PEX8605).
>
> Linux kernel version is 4.19.9x and also 4.14.134 (from phytec's
> linux-mainline repo). Kernel do not have PCIe hot-plug and PNP enabled
> in config.
>
> PLX switch #PERST is attached to a GPIO pin and stays in disable state
> until Linux is booted. So at boot time only PCIe root-complex is
> initialized by kernel.
>
> After boot if I do "lspci -v" and see everything good from PCIe
> root-complex (below):
>
> ~ # lspci -v
> 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> [Normal decode])
> Flags: bus master, fast devsel, latency 0, IRQ 295
> Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
> Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> I/O behind bridge: None
> Memory behind bridge: None
> Prefetchable memory behind bridge: None
> [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> Capabilities: [40] Power Management version 3
> Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
> Capabilities: [70] Express Root Port (Slot-), MSI 00
> Capabilities: [100] Advanced Error Reporting
> Capabilities: [140] Virtual Channel
> Kernel driver in use: pcieport
>
>
> Then I enable the #PERST pin of PLX switch, everything is still good
> (no rescan on Linux is done yet)
>
> ~ # echo 139 > /sys/class/gpio/export
> ~ # echo out > /sys/class/gpio/gpio139/direction
> ~ # echo 1 > /sys/class/gpio/gpio139/value
> ~ # lspci -v
> 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> [Normal decode])
> Flags: bus master, fast devsel, latency 0, IRQ 295
> Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
> Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> I/O behind bridge: None
> Memory behind bridge: None
> Prefetchable memory behind bridge: None
> [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> Capabilities: [40] Power Management version 3
> Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
> Capabilities: [70] Express Root Port (Slot-), MSI 00
> Capabilities: [100] Advanced Error Reporting
> Capabilities: [140] Virtual Channel
> Kernel driver in use: pcieport
>
>
> Now just disable/put-in-reset the PLX switch (Linux don't see the
> switch yet, as no rescan on PCIe was done). Now "lspci -v" and
> root-complex goes bad.
>
> ~ # echo 0 > /sys/class/gpio/gpio139/value
> ~ # lspci -v
> 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> [Normal decode])
> Flags: fast devsel, IRQ 295
> Memory at 01000000 (64-bit, prefetchable) [disabled] [size=1M]
> Bus: primary=00, secondary=00, subordinate=00, sec-latency=0
> I/O behind bridge: 00000000-00000fff [size=4K]
> Memory behind bridge: 00000000-000fffff [size=1M]
> Prefetchable memory behind bridge: 00000000-000fffff [size=1M]
> [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> Capabilities: [40] Power Management version 3
> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
> Capabilities: [70] Express Root Port (Slot-), MSI 00
> Capabilities: [100] Advanced Error Reporting
> Capabilities: [140] Virtual Channel
> Kernel driver in use: pcieport
>
> ~ # uname -a
> Linux buildroot-2019.08-imx6 4.14.134-phy2 #1 SMP Thu Feb 20 12:13:33
> UTC 2020 armv7l GNU/Linux
> ~ #
>
>
> I am really not sure what is going wrong here. Did I am missing
> something basic?
I agree, it looks like something's wrong, but I really don't have any
ideas.
I would start by using "lspci -xxxx" to see the actual values we get
from config space. It looks like we're reading zeros from at least
the bus and window registers.
You could also instrument the i.MX config accessors in case there's
something strange going on there. Maybe try to reproduce this on a
current upstream kernel?
Bjorn
[+cc Richard, Lucas]
On Wed, Feb 26, 2020 at 05:25:52PM -0600, Bjorn Helgaas wrote:
> On Sat, Feb 22, 2020 at 04:25:41PM +0100, Fawad Lateef wrote:
> > Hello,
> >
> > I am trying to figure-out an issue on our i.MX6Q platform based design
> > where PCIe interface goes bad.
> >
> > We have a Phytec i.MX6Q eMMC SOM, attached to our custom designed
> > board. PCIe root-complex from i.MX6Q is attached to PLX switch
> > (PEX8605).
> >
> > Linux kernel version is 4.19.9x and also 4.14.134 (from phytec's
> > linux-mainline repo). Kernel do not have PCIe hot-plug and PNP enabled
> > in config.
> >
> > PLX switch #PERST is attached to a GPIO pin and stays in disable state
> > until Linux is booted. So at boot time only PCIe root-complex is
> > initialized by kernel.
> >
> > After boot if I do "lspci -v" and see everything good from PCIe
> > root-complex (below):
> >
> > ~ # lspci -v
> > 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> > [Normal decode])
> > Flags: bus master, fast devsel, latency 0, IRQ 295
> > Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
> > Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> > I/O behind bridge: None
> > Memory behind bridge: None
> > Prefetchable memory behind bridge: None
> > [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> > Capabilities: [40] Power Management version 3
> > Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
> > Capabilities: [70] Express Root Port (Slot-), MSI 00
> > Capabilities: [100] Advanced Error Reporting
> > Capabilities: [140] Virtual Channel
> > Kernel driver in use: pcieport
> >
> >
> > Then I enable the #PERST pin of PLX switch, everything is still good
> > (no rescan on Linux is done yet)
> >
> > ~ # echo 139 > /sys/class/gpio/export
> > ~ # echo out > /sys/class/gpio/gpio139/direction
> > ~ # echo 1 > /sys/class/gpio/gpio139/value
> > ~ # lspci -v
> > 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> > [Normal decode])
> > Flags: bus master, fast devsel, latency 0, IRQ 295
> > Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
> > Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> > I/O behind bridge: None
> > Memory behind bridge: None
> > Prefetchable memory behind bridge: None
> > [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> > Capabilities: [40] Power Management version 3
> > Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
> > Capabilities: [70] Express Root Port (Slot-), MSI 00
> > Capabilities: [100] Advanced Error Reporting
> > Capabilities: [140] Virtual Channel
> > Kernel driver in use: pcieport
> >
> >
> > Now just disable/put-in-reset the PLX switch (Linux don't see the
> > switch yet, as no rescan on PCIe was done). Now "lspci -v" and
> > root-complex goes bad.
> >
> > ~ # echo 0 > /sys/class/gpio/gpio139/value
> > ~ # lspci -v
> > 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> > [Normal decode])
> > Flags: fast devsel, IRQ 295
> > Memory at 01000000 (64-bit, prefetchable) [disabled] [size=1M]
> > Bus: primary=00, secondary=00, subordinate=00, sec-latency=0
> > I/O behind bridge: 00000000-00000fff [size=4K]
> > Memory behind bridge: 00000000-000fffff [size=1M]
> > Prefetchable memory behind bridge: 00000000-000fffff [size=1M]
> > [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> > Capabilities: [40] Power Management version 3
> > Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
> > Capabilities: [70] Express Root Port (Slot-), MSI 00
> > Capabilities: [100] Advanced Error Reporting
> > Capabilities: [140] Virtual Channel
> > Kernel driver in use: pcieport
> >
> > ~ # uname -a
> > Linux buildroot-2019.08-imx6 4.14.134-phy2 #1 SMP Thu Feb 20 12:13:33
> > UTC 2020 armv7l GNU/Linux
> > ~ #
> >
> >
> > I am really not sure what is going wrong here. Did I am missing
> > something basic?
>
> I agree, it looks like something's wrong, but I really don't have any
> ideas.
>
> I would start by using "lspci -xxxx" to see the actual values we get
> from config space. It looks like we're reading zeros from at least
> the bus and window registers.
>
> You could also instrument the i.MX config accessors in case there's
> something strange going on there. Maybe try to reproduce this on a
> current upstream kernel?
>
> Bjorn
Hi Fawad,
On Sat, Feb 22, 2020 at 12:26 PM Fawad Lateef <[email protected]> wrote:
>
> Hello,
>
> I am trying to figure-out an issue on our i.MX6Q platform based design
> where PCIe interface goes bad.
>
> We have a Phytec i.MX6Q eMMC SOM, attached to our custom designed
> board. PCIe root-complex from i.MX6Q is attached to PLX switch
> (PEX8605).
>
> Linux kernel version is 4.19.9x and also 4.14.134 (from phytec's
Does it happen with 5.4 or 5.5 too?
Which dts are you using?
> Then I enable the #PERST pin of PLX switch, everything is still good
> (no rescan on Linux is done yet)
>
> ~ # echo 139 > /sys/class/gpio/export
> ~ # echo out > /sys/class/gpio/gpio139/direction
> ~ # echo 1 > /sys/class/gpio/gpio139/value
Not sure why you toggle the PERST pin from userspace.
You should do it via reset-gpio property in the device tree.
Hi Bjorn,
Thanks for your reply. Please see my comments below.
By the way, I have another development kit from "Embedded Artists"
with i.MX6Q SOM. I did similar test quickly (with WLAN attached to
PCIe root-complex _not_ PLX switch). This one also showed same
behavior though I have to confirm this properly (working on it). Then
at-least I can say its not exactly issue of Phytec SOM.
On Thu, 27 Feb 2020 at 00:27, Bjorn Helgaas <[email protected]> wrote:
>
> [+cc Richard, Lucas]
>
> On Wed, Feb 26, 2020 at 05:25:52PM -0600, Bjorn Helgaas wrote:
> > On Sat, Feb 22, 2020 at 04:25:41PM +0100, Fawad Lateef wrote:
> > > Hello,
> > >
> > > I am trying to figure-out an issue on our i.MX6Q platform based design
> > > where PCIe interface goes bad.
> > >
> > > We have a Phytec i.MX6Q eMMC SOM, attached to our custom designed
> > > board. PCIe root-complex from i.MX6Q is attached to PLX switch
> > > (PEX8605).
> > >
> > > Linux kernel version is 4.19.9x and also 4.14.134 (from phytec's
> > > linux-mainline repo). Kernel do not have PCIe hot-plug and PNP enabled
> > > in config.
> > >
> > > PLX switch #PERST is attached to a GPIO pin and stays in disable state
> > > until Linux is booted. So at boot time only PCIe root-complex is
> > > initialized by kernel.
> > >
> > > After boot if I do "lspci -v" and see everything good from PCIe
> > > root-complex (below):
> > >
> > > ~ # lspci -v
> > > 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> > > [Normal decode])
> > > Flags: bus master, fast devsel, latency 0, IRQ 295
> > > Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
> > > Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> > > I/O behind bridge: None
> > > Memory behind bridge: None
> > > Prefetchable memory behind bridge: None
> > > [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> > > Capabilities: [40] Power Management version 3
> > > Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
> > > Capabilities: [70] Express Root Port (Slot-), MSI 00
> > > Capabilities: [100] Advanced Error Reporting
> > > Capabilities: [140] Virtual Channel
> > > Kernel driver in use: pcieport
> > >
> > >
> > > Then I enable the #PERST pin of PLX switch, everything is still good
> > > (no rescan on Linux is done yet)
> > >
> > > ~ # echo 139 > /sys/class/gpio/export
> > > ~ # echo out > /sys/class/gpio/gpio139/direction
> > > ~ # echo 1 > /sys/class/gpio/gpio139/value
> > > ~ # lspci -v
> > > 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> > > [Normal decode])
> > > Flags: bus master, fast devsel, latency 0, IRQ 295
> > > Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
> > > Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> > > I/O behind bridge: None
> > > Memory behind bridge: None
> > > Prefetchable memory behind bridge: None
> > > [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> > > Capabilities: [40] Power Management version 3
> > > Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
> > > Capabilities: [70] Express Root Port (Slot-), MSI 00
> > > Capabilities: [100] Advanced Error Reporting
> > > Capabilities: [140] Virtual Channel
> > > Kernel driver in use: pcieport
> > >
> > >
> > > Now just disable/put-in-reset the PLX switch (Linux don't see the
> > > switch yet, as no rescan on PCIe was done). Now "lspci -v" and
> > > root-complex goes bad.
> > >
> > > ~ # echo 0 > /sys/class/gpio/gpio139/value
> > > ~ # lspci -v
> > > 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> > > [Normal decode])
> > > Flags: fast devsel, IRQ 295
> > > Memory at 01000000 (64-bit, prefetchable) [disabled] [size=1M]
> > > Bus: primary=00, secondary=00, subordinate=00, sec-latency=0
> > > I/O behind bridge: 00000000-00000fff [size=4K]
> > > Memory behind bridge: 00000000-000fffff [size=1M]
> > > Prefetchable memory behind bridge: 00000000-000fffff [size=1M]
> > > [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> > > Capabilities: [40] Power Management version 3
> > > Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
> > > Capabilities: [70] Express Root Port (Slot-), MSI 00
> > > Capabilities: [100] Advanced Error Reporting
> > > Capabilities: [140] Virtual Channel
> > > Kernel driver in use: pcieport
> > >
> > > ~ # uname -a
> > > Linux buildroot-2019.08-imx6 4.14.134-phy2 #1 SMP Thu Feb 20 12:13:33
> > > UTC 2020 armv7l GNU/Linux
> > > ~ #
> > >
> > >
> > > I am really not sure what is going wrong here. Did I am missing
> > > something basic?
> >
> > I agree, it looks like something's wrong, but I really don't have any
> > ideas.
> >
> > I would start by using "lspci -xxxx" to see the actual values we get
> > from config space. It looks like we're reading zeros from at least
> > the bus and window registers.
Somehow "lspci -xxxx" generate kernel crash ("imprecise external
abort") on both Phytec and Embedded Artists SOMs. lspci with -xxx (3
x) works but not 4 x. Seems like i.MX6 general issue?
> >
> > You could also instrument the i.MX config accessors in case there's
> > something strange going on there. Maybe try to reproduce this on a
> > current upstream kernel?
I will try to read i.MX PCIe config registers, but I think those will
be read through PCIe interface and when it goes bad, devmem or any
other access to root-complex memory-address hangs the full SOM, not
even sys-rq works.
I was playing with 5.2.xx kernel earlier, but didn't try it on
recently. Will do a clean build with it again and see if I can face
similar situation.
Thanks,
Fawad Lateef
> >
> > Bjorn
Hi Fabio,
On Thu, 27 Feb 2020 at 00:40, Fabio Estevam <[email protected]> wrote:
>
> Hi Fawad,
>
> On Sat, Feb 22, 2020 at 12:26 PM Fawad Lateef <[email protected]> wrote:
> >
> > Hello,
> >
> > I am trying to figure-out an issue on our i.MX6Q platform based design
> > where PCIe interface goes bad.
> >
> > We have a Phytec i.MX6Q eMMC SOM, attached to our custom designed
> > board. PCIe root-complex from i.MX6Q is attached to PLX switch
> > (PEX8605).
> >
> > Linux kernel version is 4.19.9x and also 4.14.134 (from phytec's
>
> Does it happen with 5.4 or 5.5 too?
I had 5.2.xx kernel working earlier but then due to other issues I
switched back to Phytec provided kernel. I will give 5.2.xx (as its
patched for SOM already on my system) a try again and see if its
better.
>
> Which dts are you using?
I attached the dts files by mail. Though I tried on another OEM som
"Embedded ARtists" i.MX6Q with their own dts (but reset-gpio setting
for mPCIe card commented out) quickly and saw similar behaviour.
>
> > Then I enable the #PERST pin of PLX switch, everything is still good
> > (no rescan on Linux is done yet)
> >
> > ~ # echo 139 > /sys/class/gpio/export
> > ~ # echo out > /sys/class/gpio/gpio139/direction
> > ~ # echo 1 > /sys/class/gpio/gpio139/value
>
> Not sure why you toggle the PERST pin from userspace.
I am trying to do this later from user-space as we are battery
operated WLAN AP device and only want to enable devices based on
different "modes"; like on batteries we do not want USB3 controller
active and also might just enable one of the two WLANs etc.
>
> You should do it via reset-gpio property in the device tree.
I tried to enable only PLX switch with reset-gpio and I see that later
when I try to enable WLANs and do pci->rescan then WLANs unable to
allocate memory in BAR regions. Likely as we do not have PCIe hot-plug
enabled. On ARM without BIOS/bootloader doing enumeration it might be
even useful, right?
~ # echo 1 > /sys/bus/pci/rescan
[ 2280.186261] pcieport 0000:02:02.0: BAR 8: no space for [mem size 0x00300000]
[ 2280.193409] pcieport 0000:02:02.0: BAR 8: failed to assign [mem
size 0x00300000]
[ 2280.200834] pcieport 0000:02:03.0: BAR 8: no space for [mem size 0x00300000]
[ 2280.207948] pcieport 0000:02:03.0: BAR 8: failed to assign [mem
size 0x00300000]
[ 2280.215456] pci 0000:04:00.0: BAR 0: no space for [mem size 0x00200000 64bit]
[ 2280.222690] pci 0000:04:00.0: BAR 0: failed to assign [mem size
0x00200000 64bit]
[ 2280.230206] pci 0000:04:00.0: BAR 6: no space for [mem size 0x00010000 pref]
[ 2280.237321] pci 0000:04:00.0: BAR 6: failed to assign [mem size
0x00010000 pref]
[ 2280.244886] pci 0000:05:00.0: BAR 0: no space for [mem size 0x00200000 64bit]
[ 2280.252115] pci 0000:05:00.0: BAR 0: failed to assign [mem size
0x00200000 64bit]
[ 2280.259623] pci 0000:05:00.0: BAR 6: no space for [mem size 0x00010000 pref]
[ 2280.266729] pci 0000:05:00.0: BAR 6: failed to assign [mem size
0x00010000 pref]
By the way is there way to specify multiple "gpio-reset" pins in
device tree? Is reset-gpio property can have multiple pins OR
reset-gpios is to be used and its similar to gpio-reset (without 's')
property?
Thanks,
Fawad Lateef
On Fri, Feb 28, 2020 at 11:16:59AM +0100, Fawad Lateef wrote:
> Hi Bjorn,
>
> Thanks for your reply. Please see my comments below.
>
> By the way, I have another development kit from "Embedded Artists"
> with i.MX6Q SOM. I did similar test quickly (with WLAN attached to
> PCIe root-complex _not_ PLX switch). This one also showed same
> behavior though I have to confirm this properly (working on it). Then
> at-least I can say its not exactly issue of Phytec SOM.
>
> On Thu, 27 Feb 2020 at 00:27, Bjorn Helgaas <[email protected]> wrote:
> > On Wed, Feb 26, 2020 at 05:25:52PM -0600, Bjorn Helgaas wrote:
> > > On Sat, Feb 22, 2020 at 04:25:41PM +0100, Fawad Lateef wrote:
> > > > Hello,
> > > >
> > > > I am trying to figure-out an issue on our i.MX6Q platform based design
> > > > where PCIe interface goes bad.
> > > >
> > > > We have a Phytec i.MX6Q eMMC SOM, attached to our custom designed
> > > > board. PCIe root-complex from i.MX6Q is attached to PLX switch
> > > > (PEX8605).
> > > >
> > > > Linux kernel version is 4.19.9x and also 4.14.134 (from phytec's
> > > > linux-mainline repo). Kernel do not have PCIe hot-plug and PNP enabled
> > > > in config.
> > > >
> > > > PLX switch #PERST is attached to a GPIO pin and stays in disable state
> > > > until Linux is booted. So at boot time only PCIe root-complex is
> > > > initialized by kernel.
> > > >
> > > > After boot if I do "lspci -v" and see everything good from PCIe
> > > > root-complex (below):
> > > >
> > > > ~ # lspci -v
> > > > 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> > > > [Normal decode])
> > > > Flags: bus master, fast devsel, latency 0, IRQ 295
> > > > Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
> > > > Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> > > > I/O behind bridge: None
> > > > Memory behind bridge: None
> > > > Prefetchable memory behind bridge: None
> > > > [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> > > > Capabilities: [40] Power Management version 3
> > > > Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
> > > > Capabilities: [70] Express Root Port (Slot-), MSI 00
> > > > Capabilities: [100] Advanced Error Reporting
> > > > Capabilities: [140] Virtual Channel
> > > > Kernel driver in use: pcieport
> > > >
> > > >
> > > > Then I enable the #PERST pin of PLX switch, everything is still good
> > > > (no rescan on Linux is done yet)
> > > >
> > > > ~ # echo 139 > /sys/class/gpio/export
> > > > ~ # echo out > /sys/class/gpio/gpio139/direction
> > > > ~ # echo 1 > /sys/class/gpio/gpio139/value
> > > > ~ # lspci -v
> > > > 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> > > > [Normal decode])
> > > > Flags: bus master, fast devsel, latency 0, IRQ 295
> > > > Memory at 01000000 (32-bit, non-prefetchable) [size=1M]
> > > > Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
> > > > I/O behind bridge: None
> > > > Memory behind bridge: None
> > > > Prefetchable memory behind bridge: None
> > > > [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> > > > Capabilities: [40] Power Management version 3
> > > > Capabilities: [50] MSI: Enable+ Count=1/1 Maskable+ 64bit+
> > > > Capabilities: [70] Express Root Port (Slot-), MSI 00
> > > > Capabilities: [100] Advanced Error Reporting
> > > > Capabilities: [140] Virtual Channel
> > > > Kernel driver in use: pcieport
> > > >
> > > >
> > > > Now just disable/put-in-reset the PLX switch (Linux don't see the
> > > > switch yet, as no rescan on PCIe was done). Now "lspci -v" and
> > > > root-complex goes bad.
> > > >
> > > > ~ # echo 0 > /sys/class/gpio/gpio139/value
> > > > ~ # lspci -v
> > > > 00:00.0 PCI bridge: Synopsys, Inc. Device abcd (rev 01) (prog-if 00
> > > > [Normal decode])
> > > > Flags: fast devsel, IRQ 295
> > > > Memory at 01000000 (64-bit, prefetchable) [disabled] [size=1M]
> > > > Bus: primary=00, secondary=00, subordinate=00, sec-latency=0
> > > > I/O behind bridge: 00000000-00000fff [size=4K]
> > > > Memory behind bridge: 00000000-000fffff [size=1M]
> > > > Prefetchable memory behind bridge: 00000000-000fffff [size=1M]
> > > > [virtual] Expansion ROM at 01100000 [disabled] [size=64K]
> > > > Capabilities: [40] Power Management version 3
> > > > Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
> > > > Capabilities: [70] Express Root Port (Slot-), MSI 00
> > > > Capabilities: [100] Advanced Error Reporting
> > > > Capabilities: [140] Virtual Channel
> > > > Kernel driver in use: pcieport
> > > >
> > > > ~ # uname -a
> > > > Linux buildroot-2019.08-imx6 4.14.134-phy2 #1 SMP Thu Feb 20 12:13:33
> > > > UTC 2020 armv7l GNU/Linux
> > > > ~ #
> > > >
> > > >
> > > > I am really not sure what is going wrong here. Did I am missing
> > > > something basic?
> > >
> > > I agree, it looks like something's wrong, but I really don't have any
> > > ideas.
> > >
> > > I would start by using "lspci -xxxx" to see the actual values we get
> > > from config space. It looks like we're reading zeros from at least
> > > the bus and window registers.
>
> Somehow "lspci -xxxx" generate kernel crash ("imprecise external
> abort") on both Phytec and Embedded Artists SOMs. lspci with -xxx (3
> x) works but not 4 x. Seems like i.MX6 general issue?
Sounds like i.MX6 doesn't handle PCIe errors correctly. "lspci -xxx"
reads the 256-byte PCI config space, while "lspci -xxxx" reads the
entire 4K extended config space. If we read config space that a
device doesn't implement, I think we'll get an Unsupported Request
completion on PCIe. That *should* be handled nicely (without causing
a kernel crash) and turned into a ~0 response to the read. If that
doesn't work, it needs to be solved somewhere in the i.MX6 or ARM arch
code.
Bjorn