2011-04-28 23:06:54

by Xianghua Xiao

[permalink] [raw]
Subject: PCIe/PCI rescan for new FPGA devices

I have a FPGA-PCIe device and a FPGA-PCI device that are empty at
kernel boot time, they're only loaded after kernel is up. After I load
FPGA images, "lspci" can not show the device, "echo 1 >
/sys/bus/pci/rescan" also does nothing about it. With a reboot I can
see the devices are allocated correctly though, but I want to avoid
the reboot.

I checked at fakephp(which is to be deprecated) and hotplug/rescan,
none of that worked for this scenario.

Can 'rescan' handle cases like this? or, is there a way that I tell
the kernel to reserve a few BARs somehow in the PCI topology at
bootime so I can "insert" the FPGAs later?

Thanks,
Xianghua


2011-04-29 02:28:56

by Greg KH

[permalink] [raw]
Subject: Re: PCIe/PCI rescan for new FPGA devices

On Thu, Apr 28, 2011 at 06:06:50PM -0500, Xianghua Xiao wrote:
> I have a FPGA-PCIe device and a FPGA-PCI device that are empty at
> kernel boot time, they're only loaded after kernel is up. After I load
> FPGA images, "lspci" can not show the device, "echo 1 >
> /sys/bus/pci/rescan" also does nothing about it. With a reboot I can
> see the devices are allocated correctly though, but I want to avoid
> the reboot.
>
> I checked at fakephp(which is to be deprecated) and hotplug/rescan,
> none of that worked for this scenario.
>
> Can 'rescan' handle cases like this? or, is there a way that I tell
> the kernel to reserve a few BARs somehow in the PCI topology at
> bootime so I can "insert" the FPGAs later?

Your firmware has to do this, see the PCI hotplug specification for
details as to the proper procedure for this.

Best of luck,

greg k-h

2011-04-29 06:41:00

by Yinghai Lu

[permalink] [raw]
Subject: Re: PCIe/PCI rescan for new FPGA devices

On Thu, Apr 28, 2011 at 4:06 PM, Xianghua Xiao <[email protected]> wrote:
> I have a FPGA-PCIe device and a FPGA-PCI device that are empty at
> kernel boot time, they're only loaded after kernel is up. After I load
> FPGA images, "lspci" can not show the device, "echo 1 >
> /sys/bus/pci/rescan" also does nothing about it. With a reboot I can
> see the devices are allocated correctly though, but I want to avoid
> the reboot.
>
> I checked at fakephp(which is to be deprecated) and hotplug/rescan,
> none of that worked for this scenario.
>
> Can 'rescan' handle cases like this? or, is there a way that I tell
> the kernel to reserve a few BARs somehow in the PCI topology at
> bootime so I can "insert" the FPGAs later?

well, you did not send out the lspci -tv yet.

assume lspci tree like this

+-[0000:c0]-+-00.0-
| |
| +-01.0-[c3]--
| +-03.0-[c4-c9]----00.0-[c5-c7]--+-02.0-[c6]--+-00.0 your FPGA

so c0:03.0 is pcie root port,
c4:00.0 is your pcie switch upstream port
c5:02.0 is your pcie switch downstream port
c6:0 is your FPGA controller.

so you could remove c0:03.0 or c4:00.0 or c5:02.0, then do rescan.
--- depend if your BIOS allocate the right value to the c0:03.0

Yinghai