Hello,
This V1 of patch series is to provide the required pci OF interfaces for
the PCIe device which uses flattened device tree to describe apertures in
its PCIe BARs. e.g, Xilinx Alveo PCIe accelerator. This requires a base
device tree which contains nodes for PCIe devices. A PCIe device driver
can then overlay a flattened device tree on the PCIe device tree node.
There are two separate parts for this to work. First, not all system has
a base device tree created by default. Thus, a patch to create an empty
device tree root node has been submitted.
https://lore.kernel.org/lkml/[email protected]/
Second, PCIe is self discoverable bus and there might not be a device tree
node created for PCIe device. This patch provides a new interface to create
a ‘pci-ep-bus’ node under the base device tree root node. PCIe device
driver may call this interface in its probe routine to create device tree
node, then overlays its device tree to the node.
For the overlayed device tree nodes, each node presents a hardware aperture
implemented in its PCIe BARs. The aperture register address consists of BAR
index and offset. It uses the following encoding:
0xIooooooo 0xoooooooo
Where:
I = BAR index
ooooooo oooooooo = BAR offset
The ‘pci-ep-bus’ node been created is compatible with ‘simple-bus’ and
contains ‘ranges’ property for translating aperture address to CPU address.
The last patch enhances of_overlay_fdt_apply(). The ‘pci-ep-bus’ device
node is created dynamically. The flattened device tree may not specify an
fixed target overlay path in front. Instead, a relative path to the
‘pci-ep-bus’ node is specified in the flattened tree. Thus, a new
parameter is added to point the target base node which is ‘pci-ep-bus’
node in this case. Then the entire overlay target path is target base node
path plus the relative path specified in the flattened device tree.
Lizhi Hou (4):
pci: add interface to create pci-ep device tree node
Documentation: devicetree: bindings: add binding for PCIe endpoint bus
fpga: xrt: management physical function driver
of: enhance overlay applying interface to specific target base node
.../devicetree/bindings/bus/pci-ep-bus.yaml | 72 +++++++
drivers/fpga/Kconfig | 3 +
drivers/fpga/Makefile | 3 +
drivers/fpga/xrt/Kconfig | 24 +++
drivers/fpga/xrt/Makefile | 8 +
drivers/fpga/xrt/mgmt/Makefile | 13 ++
drivers/fpga/xrt/mgmt/dt-test.dts | 15 ++
drivers/fpga/xrt/mgmt/dt-test.h | 15 ++
drivers/fpga/xrt/mgmt/xmgmt-drv.c | 102 ++++++++++
drivers/gpu/drm/rcar-du/rcar_du_of.c | 2 +-
drivers/of/overlay.c | 37 ++--
drivers/of/unittest.c | 2 +-
drivers/pci/of.c | 180 ++++++++++++++++++
include/linux/of.h | 2 +-
include/linux/of_pci.h | 15 ++
15 files changed, 479 insertions(+), 14 deletions(-)
create mode 100644 Documentation/devicetree/bindings/bus/pci-ep-bus.yaml
create mode 100644 drivers/fpga/xrt/Kconfig
create mode 100644 drivers/fpga/xrt/Makefile
create mode 100644 drivers/fpga/xrt/mgmt/Makefile
create mode 100644 drivers/fpga/xrt/mgmt/dt-test.dts
create mode 100644 drivers/fpga/xrt/mgmt/dt-test.h
create mode 100644 drivers/fpga/xrt/mgmt/xmgmt-drv.c
--
2.27.0
The PCIe device driver which attaches to management function on Alveo
devices. The first version of this driver demonstrates calling PCIe
interface to create device tree node.
Signed-off-by: Sonal Santan <[email protected]>
Signed-off-by: Max Zhen <[email protected]>
Signed-off-by: Lizhi Hou <[email protected]>
---
drivers/fpga/Kconfig | 3 ++
drivers/fpga/Makefile | 3 ++
drivers/fpga/xrt/Kconfig | 24 ++++++++++++
drivers/fpga/xrt/Makefile | 8 ++++
drivers/fpga/xrt/mgmt/Makefile | 12 ++++++
drivers/fpga/xrt/mgmt/xmgmt-drv.c | 63 +++++++++++++++++++++++++++++++
6 files changed, 113 insertions(+)
create mode 100644 drivers/fpga/xrt/Kconfig
create mode 100644 drivers/fpga/xrt/Makefile
create mode 100644 drivers/fpga/xrt/mgmt/Makefile
create mode 100644 drivers/fpga/xrt/mgmt/xmgmt-drv.c
diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
index 991b3f361ec9..93ae387c97c5 100644
--- a/drivers/fpga/Kconfig
+++ b/drivers/fpga/Kconfig
@@ -243,4 +243,7 @@ config FPGA_MGR_VERSAL_FPGA
configure the programmable logic(PL).
To compile this as a module, choose M here.
+
+source "drivers/fpga/xrt/Kconfig"
+
endif # FPGA
diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
index 0bff783d1b61..81ea43c40c64 100644
--- a/drivers/fpga/Makefile
+++ b/drivers/fpga/Makefile
@@ -49,3 +49,6 @@ obj-$(CONFIG_FPGA_DFL_NIOS_INTEL_PAC_N3000) += dfl-n3000-nios.o
# Drivers for FPGAs which implement DFL
obj-$(CONFIG_FPGA_DFL_PCI) += dfl-pci.o
+
+# XRT drivers for Xilinx Alveo platforms
+obj-$(CONFIG_FPGA_XRT) += xrt/
diff --git a/drivers/fpga/xrt/Kconfig b/drivers/fpga/xrt/Kconfig
new file mode 100644
index 000000000000..47efc8f71cec
--- /dev/null
+++ b/drivers/fpga/xrt/Kconfig
@@ -0,0 +1,24 @@
+
+# XRT Alveo FPGA device configuration
+#
+
+config FPGA_XRT
+ tristate "XRT Alveo Drivers"
+ depends on OF
+ select OF_EMPTY_ROOT
+ select OF_OVERLAY
+ help
+ Select this option to enable Xilinx XRT Alveo drivers. Xilinx Alveo
+ card is PCIe device and has two PCIe functions. The first function
+ performs board manangement and XRT management driver will be attached
+ to it. The second function performs data movement, compute unit
+ scheduling etc. And an XRT user driver will be attached to it.
+
+config FPGA_XRT_XMGMT
+ tristate "Xilinx Alveo Management Driver"
+ depends on FPGA_XRT
+ help
+ Select this option to enable XRT PCIe driver for Xilinx Alveo FPGA.
+ This driver provides interfaces for userspace application to access
+ Alveo FPGA device, such as: downloading FPGA bitstream, query card
+ information, hot reset card etc.
diff --git a/drivers/fpga/xrt/Makefile b/drivers/fpga/xrt/Makefile
new file mode 100644
index 000000000000..2d251b5653bb
--- /dev/null
+++ b/drivers/fpga/xrt/Makefile
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# Copyright (C) 2020-2022 Xilinx, Inc. All rights reserved.
+#
+# Authors: [email protected]
+#
+
+obj-$(CONFIG_FPGA_XRT_XMGMT) += mgmt/
diff --git a/drivers/fpga/xrt/mgmt/Makefile b/drivers/fpga/xrt/mgmt/Makefile
new file mode 100644
index 000000000000..b893c7293d70
--- /dev/null
+++ b/drivers/fpga/xrt/mgmt/Makefile
@@ -0,0 +1,12 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# Copyright (C) 2020-2022 Xilinx, Inc. All rights reserved.
+#
+# Authors: [email protected]
+# [email protected]
+#
+
+obj-$(CONFIG_FPGA_XRT_XMGMT) += xrt-mgmt.o
+
+xrt-mgmt-objs := \
+ xmgmt-drv.o
diff --git a/drivers/fpga/xrt/mgmt/xmgmt-drv.c b/drivers/fpga/xrt/mgmt/xmgmt-drv.c
new file mode 100644
index 000000000000..60742a478a43
--- /dev/null
+++ b/drivers/fpga/xrt/mgmt/xmgmt-drv.c
@@ -0,0 +1,63 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Xilinx Alveo Management Function Driver
+ *
+ * Copyright (C) 2020-2022 Xilinx, Inc.
+ *
+ * Authors:
+ * Cheng Zhen <[email protected]>
+ * Lizhi Hou <[email protected]>
+ */
+
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/aer.h>
+#include <linux/vmalloc.h>
+#include <linux/delay.h>
+#include <linux/of_pci.h>
+
+#define XMGMT_MODULE_NAME "xrt-mgmt"
+
+/* PCI Device IDs */
+#define PCI_DEVICE_ID_U50 0x5020
+static const struct pci_device_id xmgmt_pci_ids[] = {
+ { PCI_DEVICE(PCI_VENDOR_ID_XILINX, PCI_DEVICE_ID_U50), }, /* Alveo U50 */
+ { 0, }
+};
+
+static int xmgmt_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+{
+ devm_of_pci_create_bus_endpoint(pdev);
+
+ return 0;
+}
+
+static struct pci_driver xmgmt_driver = {
+ .name = XMGMT_MODULE_NAME,
+ .id_table = xmgmt_pci_ids,
+ .probe = xmgmt_probe,
+};
+
+static int __init xmgmt_init(void)
+{
+ int res;
+
+ res = pci_register_driver(&xmgmt_driver);
+ if (res)
+ return res;
+
+ return 0;
+}
+
+static __exit void xmgmt_exit(void)
+{
+ pci_unregister_driver(&xmgmt_driver);
+}
+
+module_init(xmgmt_init);
+module_exit(xmgmt_exit);
+
+MODULE_DEVICE_TABLE(pci, xmgmt_pci_ids);
+MODULE_AUTHOR("XRT Team <[email protected]>");
+MODULE_DESCRIPTION("Xilinx Alveo management function driver");
+MODULE_LICENSE("GPL v2");
--
2.27.0
On Fri, Mar 04, 2022 at 09:23:00PM -0800, Lizhi Hou wrote:
> Hello,
Why is this a resend? I see
https://lore.kernel.org/r/[email protected],
which looks like it was posted just a few minutes before this. I
assume this "RESEND" is identical?
> This V1 of patch series is to provide the required pci OF interfaces for
> the PCIe device which uses flattened device tree to describe apertures in
> its PCIe BARs. e.g, Xilinx Alveo PCIe accelerator. This requires a base
> device tree which contains nodes for PCIe devices. A PCIe device driver
> can then overlay a flattened device tree on the PCIe device tree node.
> There are two separate parts for this to work. First, not all system has
> a base device tree created by default. Thus, a patch to create an empty
> device tree root node has been submitted.
> https://lore.kernel.org/lkml/[email protected]/
> Second, PCIe is self discoverable bus and there might not be a device tree
> node created for PCIe device. This patch provides a new interface to create
> a ‘pci-ep-bus’ node under the base device tree root node. PCIe device
> driver may call this interface in its probe routine to create device tree
> node, then overlays its device tree to the node.
> For the overlayed device tree nodes, each node presents a hardware aperture
> implemented in its PCIe BARs. The aperture register address consists of BAR
> index and offset. It uses the following encoding:
> 0xIooooooo 0xoooooooo
> Where:
> I = BAR index
> ooooooo oooooooo = BAR offset
> The ‘pci-ep-bus’ node been created is compatible with ‘simple-bus’ and
> contains ‘ranges’ property for translating aperture address to CPU address.
> The last patch enhances of_overlay_fdt_apply(). The ‘pci-ep-bus’ device
> node is created dynamically. The flattened device tree may not specify an
> fixed target overlay path in front. Instead, a relative path to the
> ‘pci-ep-bus’ node is specified in the flattened tree. Thus, a new
> parameter is added to point the target base node which is ‘pci-ep-bus’
> node in this case. Then the entire overlay target path is target base node
> path plus the relative path specified in the flattened device tree.
s/pci/PCI/ (capitalize acronyms above, also in other subjects, commit
logs, and code comments)
s/PCIe/PCI/ (in most cases, the above is not PCIe-specific)
Please add blank lines between paragraphs to make this easier to read.
The above tells *what* this series does, but not *why* we need it.
Apparently you want to describe PCI BARs in DT. Normally the PCI core
discovers devices and BARs using the PCI enumeration process (read
config space looking for a Device ID, read standard BAR locations
(unimplemented BARs are hardwired to zero)). Obviously you know all
of this. What we need here (and in the commit log for the relevant
patch) is some explanation about why this standard process doesn't
work and you need to do something via DT.
I'm guessing this is for the case where Linux is running *on* the
endpoint, so instead of enumerating devices from the perspective of a
PCI host controller, it's on the "other" side of the device, e.g., as
described in Documentation/PCI/endpoint/pci-endpoint.rst
So the commit log should mention that and explain why we need this new
DT support. The endpoint support has been around for a while, so this
should explain what's different about Xilinx Alveo and why it needs
this new stuff.
Bjorn
+ Kishon
On Fri, Mar 04, 2022 at 09:23:03PM -0800, Lizhi Hou wrote:
> The PCIe device driver which attaches to management function on Alveo
> devices. The first version of this driver demonstrates calling PCIe
> interface to create device tree node.
>
I'm assuming that this driver implements the PCI endpoint functions. Then this
driver should be under drivers/pci/endpoint/functions/ making use of the
existing PCI endpoint subsystem.
Thanks,
Mani
> Signed-off-by: Sonal Santan <[email protected]>
> Signed-off-by: Max Zhen <[email protected]>
> Signed-off-by: Lizhi Hou <[email protected]>
> ---
> drivers/fpga/Kconfig | 3 ++
> drivers/fpga/Makefile | 3 ++
> drivers/fpga/xrt/Kconfig | 24 ++++++++++++
> drivers/fpga/xrt/Makefile | 8 ++++
> drivers/fpga/xrt/mgmt/Makefile | 12 ++++++
> drivers/fpga/xrt/mgmt/xmgmt-drv.c | 63 +++++++++++++++++++++++++++++++
> 6 files changed, 113 insertions(+)
> create mode 100644 drivers/fpga/xrt/Kconfig
> create mode 100644 drivers/fpga/xrt/Makefile
> create mode 100644 drivers/fpga/xrt/mgmt/Makefile
> create mode 100644 drivers/fpga/xrt/mgmt/xmgmt-drv.c
>
> diff --git a/drivers/fpga/Kconfig b/drivers/fpga/Kconfig
> index 991b3f361ec9..93ae387c97c5 100644
> --- a/drivers/fpga/Kconfig
> +++ b/drivers/fpga/Kconfig
> @@ -243,4 +243,7 @@ config FPGA_MGR_VERSAL_FPGA
> configure the programmable logic(PL).
>
> To compile this as a module, choose M here.
> +
> +source "drivers/fpga/xrt/Kconfig"
> +
> endif # FPGA
> diff --git a/drivers/fpga/Makefile b/drivers/fpga/Makefile
> index 0bff783d1b61..81ea43c40c64 100644
> --- a/drivers/fpga/Makefile
> +++ b/drivers/fpga/Makefile
> @@ -49,3 +49,6 @@ obj-$(CONFIG_FPGA_DFL_NIOS_INTEL_PAC_N3000) += dfl-n3000-nios.o
>
> # Drivers for FPGAs which implement DFL
> obj-$(CONFIG_FPGA_DFL_PCI) += dfl-pci.o
> +
> +# XRT drivers for Xilinx Alveo platforms
> +obj-$(CONFIG_FPGA_XRT) += xrt/
> diff --git a/drivers/fpga/xrt/Kconfig b/drivers/fpga/xrt/Kconfig
> new file mode 100644
> index 000000000000..47efc8f71cec
> --- /dev/null
> +++ b/drivers/fpga/xrt/Kconfig
> @@ -0,0 +1,24 @@
> +
> +# XRT Alveo FPGA device configuration
> +#
> +
> +config FPGA_XRT
> + tristate "XRT Alveo Drivers"
> + depends on OF
> + select OF_EMPTY_ROOT
> + select OF_OVERLAY
> + help
> + Select this option to enable Xilinx XRT Alveo drivers. Xilinx Alveo
> + card is PCIe device and has two PCIe functions. The first function
> + performs board manangement and XRT management driver will be attached
> + to it. The second function performs data movement, compute unit
> + scheduling etc. And an XRT user driver will be attached to it.
> +
> +config FPGA_XRT_XMGMT
> + tristate "Xilinx Alveo Management Driver"
> + depends on FPGA_XRT
> + help
> + Select this option to enable XRT PCIe driver for Xilinx Alveo FPGA.
> + This driver provides interfaces for userspace application to access
> + Alveo FPGA device, such as: downloading FPGA bitstream, query card
> + information, hot reset card etc.
> diff --git a/drivers/fpga/xrt/Makefile b/drivers/fpga/xrt/Makefile
> new file mode 100644
> index 000000000000..2d251b5653bb
> --- /dev/null
> +++ b/drivers/fpga/xrt/Makefile
> @@ -0,0 +1,8 @@
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# Copyright (C) 2020-2022 Xilinx, Inc. All rights reserved.
> +#
> +# Authors: [email protected]
> +#
> +
> +obj-$(CONFIG_FPGA_XRT_XMGMT) += mgmt/
> diff --git a/drivers/fpga/xrt/mgmt/Makefile b/drivers/fpga/xrt/mgmt/Makefile
> new file mode 100644
> index 000000000000..b893c7293d70
> --- /dev/null
> +++ b/drivers/fpga/xrt/mgmt/Makefile
> @@ -0,0 +1,12 @@
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# Copyright (C) 2020-2022 Xilinx, Inc. All rights reserved.
> +#
> +# Authors: [email protected]
> +# [email protected]
> +#
> +
> +obj-$(CONFIG_FPGA_XRT_XMGMT) += xrt-mgmt.o
> +
> +xrt-mgmt-objs := \
> + xmgmt-drv.o
> diff --git a/drivers/fpga/xrt/mgmt/xmgmt-drv.c b/drivers/fpga/xrt/mgmt/xmgmt-drv.c
> new file mode 100644
> index 000000000000..60742a478a43
> --- /dev/null
> +++ b/drivers/fpga/xrt/mgmt/xmgmt-drv.c
> @@ -0,0 +1,63 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Xilinx Alveo Management Function Driver
> + *
> + * Copyright (C) 2020-2022 Xilinx, Inc.
> + *
> + * Authors:
> + * Cheng Zhen <[email protected]>
> + * Lizhi Hou <[email protected]>
> + */
> +
> +#include <linux/module.h>
> +#include <linux/pci.h>
> +#include <linux/aer.h>
> +#include <linux/vmalloc.h>
> +#include <linux/delay.h>
> +#include <linux/of_pci.h>
> +
> +#define XMGMT_MODULE_NAME "xrt-mgmt"
> +
> +/* PCI Device IDs */
> +#define PCI_DEVICE_ID_U50 0x5020
> +static const struct pci_device_id xmgmt_pci_ids[] = {
> + { PCI_DEVICE(PCI_VENDOR_ID_XILINX, PCI_DEVICE_ID_U50), }, /* Alveo U50 */
> + { 0, }
> +};
> +
> +static int xmgmt_probe(struct pci_dev *pdev, const struct pci_device_id *id)
> +{
> + devm_of_pci_create_bus_endpoint(pdev);
> +
> + return 0;
> +}
> +
> +static struct pci_driver xmgmt_driver = {
> + .name = XMGMT_MODULE_NAME,
> + .id_table = xmgmt_pci_ids,
> + .probe = xmgmt_probe,
> +};
> +
> +static int __init xmgmt_init(void)
> +{
> + int res;
> +
> + res = pci_register_driver(&xmgmt_driver);
> + if (res)
> + return res;
> +
> + return 0;
> +}
> +
> +static __exit void xmgmt_exit(void)
> +{
> + pci_unregister_driver(&xmgmt_driver);
> +}
> +
> +module_init(xmgmt_init);
> +module_exit(xmgmt_exit);
> +
> +MODULE_DEVICE_TABLE(pci, xmgmt_pci_ids);
> +MODULE_AUTHOR("XRT Team <[email protected]>");
> +MODULE_DESCRIPTION("Xilinx Alveo management function driver");
> +MODULE_LICENSE("GPL v2");
> --
> 2.27.0
>
--
மணிவண்ணன் சதாசிவம்
On Fri, Mar 04, 2022 at 09:23:03PM -0800, Lizhi Hou wrote:
> The PCIe device driver which attaches to management function on Alveo
> devices. The first version of this driver demonstrates calling PCIe
> interface to create device tree node.
> ...
> +static int __init xmgmt_init(void)
> +{
> + int res;
> +
> + res = pci_register_driver(&xmgmt_driver);
> + if (res)
> + return res;
> +
> + return 0;
This is the same as:
return pci_register_driver(&xmgmt_driver);
Bjorn