Hi all,
On Microchip PolarFire SoC (MPFS), the PCIe controller is connected to the
CPU via one of three Fabric Interface Connectors (FICs). Each FIC present
to the CPU complex as 64-bit AXI-M and 64-bit AXI-S. To preserve
compatibility with other PolarFire family members, the PCIe controller is
connected to its encapsulating FIC via a 32-bit AXI-M and 32-bit AXI-S
interface.
Each FIC is implemented in FPGA logic and can incorporate logic along its 64-bit
AXI-M to 32-bit AXI-M chain (including address translation) and, likewise, along
its 32-bit AXI-S to 64-bit AXI-S chain (again including address translation).
In order to reduce the potential support space for the PCIe controller in
this environment, MPFS supports certain reference designs for these address
translations: reference designs for cache-coherent memory accesses
and reference designs for non-cache-coherent memory accesses. The precise
details of these reference designs and associated customer guidelines
recommending that customers adhere to the addressing schemes used in those
reference designs are available from Microchip, but the implication for the
PCIe controller address translation between CPU-space and PCIe-space are:
For outbound address translation, the PCIe controller address translation tables
are treated as if they are 32-bit only. Any further address translation must
be done in FPGA fabric.
For inbound address translation, the PCIe controller is configurable for two
cases:
* In the case of cache-coherent designs, the base of the AXI-S side of the
address translation must be set to 0 and the size should be 4 GiB wide. The
FPGA fabric must complete any address translations based on that 0-based
address translation.
* In the case of non-cache coherent designs, the base of AXI-S side of the
address translation must be set to 0x8000'0000 and the size shall be 2 GiB
wide. The FPGA fabric must complete any address translation based on that
0x80000000 base.
So, for example, in the non-cache-coherent case, with a device tree property
that maps an inbound range from 0x10'0000'0000 in PCIe space to 0x10'0000'0000
in CPU space, the PCIe rootport will translate a PCIe address of 0x10'0000'0000
to an intermediate 32-bit AXI-S address of 0x8000'0000 and the FIC is
responsible for translating that intermediate 32-bit AXI-S address of
0x8000'0000 to a 64-bit AXI-S address of 0x10'0000'0000.
And similarly, for example, in the cache-coherent case, with a device tree
property that maps an inbound range from 0x10'0000'0000 in PCIe space to
0x10'0000'0000 in CPU space, the PCIe rootport will translate a PCIe address
of 0x10'0000'0000 to an intermediate 32-bit AXI-S address of 0x0000'0000 and
the FIC is responsible for translating that intermediate 32-bit AXI-S address
of 0x0000'0000 to a 64-bit AXI-S address of 0x10'0000'0000.
See https://lore.kernel.org/all/[email protected]/T/
for backstory.
Daire McNamara (2):
PCI: microchip: Fix outbound address translation tables
PCI: microchip: Fix inbound address translation tables
drivers/pci/controller/pcie-microchip-host.c | 104 +++++++++++++++++--
1 file changed, 96 insertions(+), 8 deletions(-)
base-commit: a38297e3fb012ddfa7ce0321a7e5a8daeb1872b6
--
2.34.1
On Microchip PolarFire SoC the PCIe Root Port can be behind one of three
general purpose Fabric Interface Controller (FIC) buses that encapsulates
an AXI-S bus. Depending on which FIC(s) the Root Port is connected
through to CPU space, and what address translation is done by that FIC,
the Root Port driver's inbound address translation may vary.
For all current supported designs and all future expected designs,
inbound address translation done by a FIC on PolarFire SoC varies
depending on whether PolarFire SoC in operating in dma-coherent mode or
dma-noncoherent mode.
The setup of the outbound address translation tables in the root port
driver only needs to handle these two cases.
Setup the inbound address translation tables to one of two address
translations, depending on whether the rootport is marked as dma-coherent or
dma-noncoherent.
Fixes: 6f15a9c9f941 ("PCI: microchip: Add Microchip Polarfire PCIe controller driver")
Signed-off-by: Daire McNamara <[email protected]>
---
drivers/pci/controller/pcie-microchip-host.c | 97 +++++++++++++++++++-
1 file changed, 92 insertions(+), 5 deletions(-)
diff --git a/drivers/pci/controller/pcie-microchip-host.c b/drivers/pci/controller/pcie-microchip-host.c
index 0795cd122a4a..3042e00cc44e 100644
--- a/drivers/pci/controller/pcie-microchip-host.c
+++ b/drivers/pci/controller/pcie-microchip-host.c
@@ -30,6 +30,9 @@
#define MC_PCIE_BRIDGE_ADDR (MC_PCIE1_BRIDGE_ADDR)
#define MC_PCIE_CTRL_ADDR (MC_PCIE1_CTRL_ADDR)
+#define MC_MAX_NUM_INBOUND_WINDOWS 8
+#define MPFS_NC_BOUNCE_ADDR 0x80000000
+
/* PCIe Bridge Phy Regs */
#define PCIE_PCI_IRQ_DW0 0xa8
#define MSIX_CAP_MASK BIT(31)
@@ -105,6 +108,7 @@
#define ATR0_AXI4_SLV0_TRSL_PARAM 0x810u
#define PCIE_TX_RX_INTERFACE 0x00000000u
#define PCIE_CONFIG_INTERFACE 0x00000001u
+#define TRSL_ID_AXI4_MASTER_0 0x00000004u
#define ATR_ENTRY_SIZE 32
@@ -931,6 +935,89 @@ static int mc_pcie_init_irq_domains(struct mc_pcie *port)
return mc_allocate_msi_domains(port);
}
+static void mc_pcie_setup_inbound_atr(int window_index, u64 axi_addr, u64 pcie_addr, size_t size)
+{
+ void __iomem *bridge_base_addr = port->axi_base_addr + MC_PCIE_BRIDGE_ADDR;
+ u32 table_offset = window_index * ATR_ENTRY_SIZE;
+ u32 atr_sz;
+ u32 val;
+
+ atr_sz = ilog2(size) - 1;
+ atr_sz &= GENMASK(5, 0);
+
+ val = lower_32_bits(pcie_addr) & GENMASK(31, 12);
+ val |= (atr_sz << ATR_SIZE_SHIFT);
+ val |= ATR_IMPL_ENABLE;
+ writel(val, bridge_base_addr + table_offset + ATR0_PCIE_WIN0_SRCADDR_PARAM);
+
+ writel(upper_32_bits(pcie_addr), bridge_base_addr + table_offset +
+ ATR0_PCIE_WIN0_SRC_ADDR);
+
+ writel(lower_32_bits(axi_addr), bridge_base_addr + table_offset +
+ ATR0_PCIE_WIN0_TRSL_ADDR_LSB);
+ writel(upper_32_bits(axi_addr), bridge_base_addr + table_offset +
+ ATR0_PCIE_WIN0_TRSL_ADDR_UDW);
+
+ writel(TRSL_ID_AXI4_MASTER_0, bridge_base_addr + table_offset +
+ ATR0_PCIE_WIN0_TRSL_PARAM);
+}
+
+static int mc_pcie_setup_inbound_ranges(struct platform_device *pdev, struct mc_pcie *port)
+{
+ struct device *dev = &pdev->dev;
+ struct device_node *dn = dev->of_node;
+ struct of_range_parser parser;
+ struct of_range range;
+ int atr_index = 0;
+
+ /*
+ * MPFS PCIe root port is 32-bit only, behind a Fabric Interface
+ * Controller FPGA logic block which contains the AXI-S interface.
+ *
+ * From the point of view of the PCIe root port, There are only
+ * two supported Root Port configurations
+ *
+ * Configuration 1: for use with fully coherent designs; supports a
+ * window from 0x0 (CPU space) to specified PCIe space.
+ *
+ * Configuration 2: for use with non-coherent designs; supports two
+ * 1 Gb wide windows to CPU space; one mapping cpu space 0 to pcie
+ * space 0x80000000 and mapping cpu space 0x40000000 to pcie
+ * space 0xc0000000. This cfg needs two windows because of how
+ * the MSI space is allocated in the AXI-S range on MPFS.
+ *
+ * The FIC interface outside the PCIe block *must* complete the inbound
+ * address translation as per MCHP MPFS FPGA design guidelines.
+ */
+ if (device_property_read_bool(dev, "dma-noncoherent")) {
+ /*
+ * Always need same two tables in this case. Need two tables
+ * due to hardware interactions between address and size.
+ */
+ mc_pcie_setup_inbound_atr(0, 0, MPFS_NC_BOUNCE_ADDR, SZ_1G);
+ mc_pcie_setup_inbound_atr(1, SZ_1G, MPFS_NC_BOUNCE_ADDR + SZ_1G, SZ_1G);
+ } else {
+ /* Find any dma-ranges */
+ if (of_pci_dma_range_parser_init(&parser, dn)) {
+ /* No dma-range property - setup default */
+ mc_pcie_setup_inbound_atr(0, 0, 0, SZ_4G);
+ return 0;
+ }
+
+ for_each_of_range(&parser, &range) {
+ if (atr_index >= MC_MAX_NUM_INBOUND_WINDOWS) {
+ dev_err(dev, "too many inbound ranges; %d available tables\n",
+ MC_MAX_NUM_INBOUND_WINDOWS);
+ return -EINVAL;
+ }
+ mc_pcie_setup_inbound_atr(atr_index, 0, range.pci_addr, range.size);
+ atr_index++;
+ }
+ }
+
+ return 0;
+}
+
static void mc_pcie_setup_window(void __iomem *bridge_base_addr, u32 index,
phys_addr_t axi_addr, phys_addr_t pci_addr,
size_t size)
@@ -962,11 +1049,6 @@ static void mc_pcie_setup_window(void __iomem *bridge_base_addr, u32 index,
val = upper_32_bits(pci_addr);
writel(val, bridge_base_addr + (index * ATR_ENTRY_SIZE) +
ATR0_AXI4_SLV0_TRSL_ADDR_UDW);
-
- val = readl(bridge_base_addr + ATR0_PCIE_WIN0_SRCADDR_PARAM);
- val |= (ATR0_PCIE_ATR_SIZE << ATR0_PCIE_ATR_SIZE_SHIFT);
- writel(val, bridge_base_addr + ATR0_PCIE_WIN0_SRCADDR_PARAM);
- writel(0, bridge_base_addr + ATR0_PCIE_WIN0_SRC_ADDR);
}
static int mc_pcie_setup_windows(struct platform_device *pdev,
@@ -1130,6 +1212,11 @@ static int mc_platform_init(struct pci_config_window *cfg)
if (ret)
return ret;
+ /* Configure inbound translation tables */
+ ret = mc_pcie_setup_inbound_ranges(pdev, port);
+ if (ret)
+ return ret;
+
/* Address translation is up; safe to enable interrupts */
ret = mc_init_interrupts(pdev, port);
if (ret)
--
2.34.1
On Microchip PolarFire SoC (MPFS) the PCIe Root Port can be behind one of
three general-purpose Fabric Interface Controller (FIC) buses that
encapsulate an AXI-M interface. That FIC is responsible for managing
the translations of the upper 32-bits of the AXI-M address. On MPFS,
the Root Port driver needs to take account of that outbound address
translation done by the parent FIC bus before setting up its own
outbound address translation tables. In all cases on MPFS,
the remaining outbound address translation tables are 32-bit only.
Limit the outbound address translation tables to 32-bit only.
Fixes: 6f15a9c9f941 ("PCI: microchip: Add Microchip Polarfire PCIe controller driver")
Signed-off-by: Daire McNamara <[email protected]>
---
drivers/pci/controller/pcie-microchip-host.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/drivers/pci/controller/pcie-microchip-host.c b/drivers/pci/controller/pcie-microchip-host.c
index 137fb8570ba2..0795cd122a4a 100644
--- a/drivers/pci/controller/pcie-microchip-host.c
+++ b/drivers/pci/controller/pcie-microchip-host.c
@@ -983,7 +983,8 @@ static int mc_pcie_setup_windows(struct platform_device *pdev,
if (resource_type(entry->res) == IORESOURCE_MEM) {
pci_addr = entry->res->start - entry->offset;
mc_pcie_setup_window(bridge_base_addr, index,
- entry->res->start, pci_addr,
+ entry->res->start & 0xffffffff,
+ pci_addr & 0xffffffff,
resource_size(entry->res));
index++;
}
@@ -1117,8 +1118,8 @@ static int mc_platform_init(struct pci_config_window *cfg)
int ret;
/* Configure address translation table 0 for PCIe config space */
- mc_pcie_setup_window(bridge_base_addr, 0, cfg->res.start,
- cfg->res.start,
+ mc_pcie_setup_window(bridge_base_addr, 0, cfg->res.start & 0xffffffff,
+ cfg->res.start & 0xffffffff,
resource_size(&cfg->res));
/* Need some fixups in config space */
--
2.34.1
On Fri, May 31, 2024 at 09:53:32AM +0100, Daire McNamara wrote:
> On Microchip PolarFire SoC (MPFS) the PCIe Root Port can be behind one of
> three general-purpose Fabric Interface Controller (FIC) buses that
> encapsulate an AXI-M interface. That FIC is responsible for managing
> the translations of the upper 32-bits of the AXI-M address. On MPFS,
> the Root Port driver needs to take account of that outbound address
> translation done by the parent FIC bus before setting up its own
> outbound address translation tables. In all cases on MPFS,
> the remaining outbound address translation tables are 32-bit only.
>
> Limit the outbound address translation tables to 32-bit only.
>
> Fixes: 6f15a9c9f941 ("PCI: microchip: Add Microchip Polarfire PCIe controller driver")
>
> Signed-off-by: Daire McNamara <[email protected]>
> ---
> drivers/pci/controller/pcie-microchip-host.c | 7 ++++---
> 1 file changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/pci/controller/pcie-microchip-host.c b/drivers/pci/controller/pcie-microchip-host.c
> index 137fb8570ba2..0795cd122a4a 100644
> --- a/drivers/pci/controller/pcie-microchip-host.c
> +++ b/drivers/pci/controller/pcie-microchip-host.c
> @@ -983,7 +983,8 @@ static int mc_pcie_setup_windows(struct platform_device *pdev,
> if (resource_type(entry->res) == IORESOURCE_MEM) {
> pci_addr = entry->res->start - entry->offset;
> mc_pcie_setup_window(bridge_base_addr, index,
> - entry->res->start, pci_addr,
> + entry->res->start & 0xffffffff,
> + pci_addr & 0xffffffff,
> resource_size(entry->res));
Is this masking something that the PCI core needs to be aware of when
it allocates address space for BARs?
The PCI core knows about the CPU physical address range of each bridge
window and the corresponding PCI address range. From this patch, it
looks like only the low 32 bits of the CPU address are used by the
Root Port. That might not be a problem as long as the windows
described by DT are correct and none of them overlap after masking out
the upper 32 bits. But for example, if DT has windows like this:
[mem 0x1'0000'0000-0x1'8000'0000]
[mem 0x2'0000'0000-0x2'8000'0000]
the PCI core will assume they are valid and non-overlapping, when
IIUC, they *do* overlap.
But also only the low 32 bits of the PCI address are used, and it
seems like the PCI core will need to know that so it doesn't program a
64-bit BAR with a value above 4GB?
> index++;
> }
> @@ -1117,8 +1118,8 @@ static int mc_platform_init(struct pci_config_window *cfg)
> int ret;
>
> /* Configure address translation table 0 for PCIe config space */
> - mc_pcie_setup_window(bridge_base_addr, 0, cfg->res.start,
> - cfg->res.start,
> + mc_pcie_setup_window(bridge_base_addr, 0, cfg->res.start & 0xffffffff,
> + cfg->res.start & 0xffffffff,
> resource_size(&cfg->res));
>
> /* Need some fixups in config space */
> --
> 2.34.1
>
On Fri, May 31, 2024 at 09:53:33AM +0100, Daire McNamara wrote:
> On Microchip PolarFire SoC the PCIe Root Port can be behind one of three
> general purpose Fabric Interface Controller (FIC) buses that encapsulates
> an AXI-S bus. Depending on which FIC(s) the Root Port is connected
> through to CPU space, and what address translation is done by that FIC,
> the Root Port driver's inbound address translation may vary.
>
> For all current supported designs and all future expected designs,
> inbound address translation done by a FIC on PolarFire SoC varies
> depending on whether PolarFire SoC in operating in dma-coherent mode or
> dma-noncoherent mode.
>
> The setup of the outbound address translation tables in the root port
> driver only needs to handle these two cases.
>
> Setup the inbound address translation tables to one of two address
> translations, depending on whether the rootport is marked as dma-coherent or
> dma-noncoherent.
Since we're talking about dma-noncoherent here, I think this series
should contain a patch that adds the property to the binding for PCIe:
-- >8 --
From af066543b8f8b8b0b37e0844979f0c3e28f30513 Mon Sep 17 00:00:00 2001
From: Conor Dooley <[email protected]>
Date: Mon, 20 Mar 2023 11:02:11 +0000
Subject: [PATCH] dt-bindings: PCI: microchip,pcie-host: allow dma-noncoherent
PolarFire SoC may be configured in a way that requires non-coherent DMA
handling. On RISC-V, buses are coherent by default & the dma-noncoherent
property is required to denote buses or devices that are non-coherent.
Signed-off-by: Conor Dooley <[email protected]>
---
Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml | 2 ++
1 file changed, 2 insertions(+)
diff --git a/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml b/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml
index 45c14b6e4aa4..2f21109c3580 100644
--- a/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml
+++ b/Documentation/devicetree/bindings/pci/microchip,pcie-host.yaml
@@ -53,6 +53,8 @@ properties:
items:
pattern: '^fic[0-3]$'
+ dma-noncoherent: true
+
interrupts:
minItems: 1
items:
--
2.43.2
On Mon, Jun 03, 2024 at 01:45:16PM -0500, Bjorn Helgaas wrote:
> On Fri, May 31, 2024 at 09:53:32AM +0100, Daire McNamara wrote:
> > On Microchip PolarFire SoC (MPFS) the PCIe Root Port can be behind one of
> > three general-purpose Fabric Interface Controller (FIC) buses that
> > encapsulate an AXI-M interface. That FIC is responsible for managing
> > the translations of the upper 32-bits of the AXI-M address. On MPFS,
> > the Root Port driver needs to take account of that outbound address
> > translation done by the parent FIC bus before setting up its own
> > outbound address translation tables. In all cases on MPFS,
> > the remaining outbound address translation tables are 32-bit only.
> >
> > Limit the outbound address translation tables to 32-bit only.
> >
> > Fixes: 6f15a9c9f941 ("PCI: microchip: Add Microchip Polarfire PCIe controller driver")
> >
> > Signed-off-by: Daire McNamara <[email protected]>
> > ---
> > drivers/pci/controller/pcie-microchip-host.c | 7 ++++---
> > 1 file changed, 4 insertions(+), 3 deletions(-)
> >
> > diff --git a/drivers/pci/controller/pcie-microchip-host.c b/drivers/pci/controller/pcie-microchip-host.c
> > index 137fb8570ba2..0795cd122a4a 100644
> > --- a/drivers/pci/controller/pcie-microchip-host.c
> > +++ b/drivers/pci/controller/pcie-microchip-host.c
> > @@ -983,7 +983,8 @@ static int mc_pcie_setup_windows(struct platform_device *pdev,
> > if (resource_type(entry->res) == IORESOURCE_MEM) {
> > pci_addr = entry->res->start - entry->offset;
> > mc_pcie_setup_window(bridge_base_addr, index,
> > - entry->res->start, pci_addr,
> > + entry->res->start & 0xffffffff,
> > + pci_addr & 0xffffffff,
> > resource_size(entry->res));
>
> Is this masking something that the PCI core needs to be aware of when
> it allocates address space for BARs?
I don't believe so.
>
> The PCI core knows about the CPU physical address range of each bridge
> window and the corresponding PCI address range. From this patch, it
> looks like only the low 32 bits of the CPU address are used by the
> Root Port. That might not be a problem as long as the windows
> described by DT are correct and none of them overlap after masking out
> the upper 32 bits. But for example, if DT has windows like this:
>
> [mem 0x1'0000'0000-0x1'8000'0000]
> [mem 0x2'0000'0000-0x2'8000'0000]
>
> the PCI core will assume they are valid and non-overlapping, when
> IIUC, they *do* overlap.
True, but I can't see how that could happen on any real system - in my mind,
a PolarFire Soc designer (or indeed any designer on any chip) will know where
its rootport is actually attached in its memory map. On PolarFire SoC, for
example, a designer can only reach a rootport over a FIC, and - if they were
to attach to the rootport over two FICs at the same time, that would be a
blunder and would be picked up during design phase. I can't imagine any
reason anyone would release a product with that arrangement.
>
> But also only the low 32 bits of the PCI address are used, and it
> seems like the PCI core will need to know that so it doesn't program a
> 64-bit BAR with a value above 4GB?
Yeah, I'll send around a v2 shortly to address this - I was rather
over-zealous when I prevented that.
>
> > index++;
> > }
> > @@ -1117,8 +1118,8 @@ static int mc_platform_init(struct pci_config_window *cfg)
> > int ret;
> >
> > /* Configure address translation table 0 for PCIe config space */
> > - mc_pcie_setup_window(bridge_base_addr, 0, cfg->res.start,
> > - cfg->res.start,
> > + mc_pcie_setup_window(bridge_base_addr, 0, cfg->res.start & 0xffffffff,
> > + cfg->res.start & 0xffffffff,
> > resource_size(&cfg->res));
> >
> > /* Need some fixups in config space */
> > --
> > 2.34.1
> >
>