From: Hou Zhiqiang <[email protected]>
This patch set is to recode the Mobiveil driver and add
PCIe support for NXP Layerscape series SoCs integrated
Mobiveil's PCIe Gen4 controller.
Hou Zhiqiang (12):
PCI: mobiveil: Re-abstract the private structure
PCI: mobiveil: Move the host initialization into a routine
PCI: mobiveil: Collect the interrupt related operations into a routine
PCI: mobiveil: Modularize the Mobiveil PCIe Host Bridge IP driver
PCI: mobiveil: Add callback function for interrupt initialization
PCI: mobiveil: Add callback function for link up check
PCI: mobiveil: Make mobiveil_host_init() can be used to re-init host
PCI: mobiveil: Add 8-bit and 16-bit CSR register accessors
dt-bindings: PCI: Add NXP Layerscape SoCs PCIe Gen4 controller
PCI: mobiveil: Add PCIe Gen4 RC driver for NXP Layerscape SoCs
arm64: dts: lx2160a: Add PCIe controller DT nodes
arm64: defconfig: Enable CONFIG_PCIE_LAYERSCAPE_GEN4
.../bindings/pci/layerscape-pcie-gen4.txt | 52 ++
MAINTAINERS | 10 +-
.../arm64/boot/dts/freescale/fsl-lx2160a.dtsi | 163 ++++++
arch/arm64/configs/defconfig | 1 +
drivers/pci/controller/Kconfig | 11 +-
drivers/pci/controller/Makefile | 2 +-
drivers/pci/controller/mobiveil/Kconfig | 34 ++
drivers/pci/controller/mobiveil/Makefile | 5 +
.../mobiveil/pcie-layerscape-gen4.c | 274 +++++++++
.../pcie-mobiveil-host.c} | 544 ++++--------------
.../controller/mobiveil/pcie-mobiveil-plat.c | 60 ++
.../pci/controller/mobiveil/pcie-mobiveil.c | 230 ++++++++
.../pci/controller/mobiveil/pcie-mobiveil.h | 226 ++++++++
13 files changed, 1157 insertions(+), 455 deletions(-)
create mode 100644 Documentation/devicetree/bindings/pci/layerscape-pcie-gen4.txt
create mode 100644 drivers/pci/controller/mobiveil/Kconfig
create mode 100644 drivers/pci/controller/mobiveil/Makefile
create mode 100644 drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c
rename drivers/pci/controller/{pcie-mobiveil.c => mobiveil/pcie-mobiveil-host.c} (54%)
create mode 100644 drivers/pci/controller/mobiveil/pcie-mobiveil-plat.c
create mode 100644 drivers/pci/controller/mobiveil/pcie-mobiveil.c
create mode 100644 drivers/pci/controller/mobiveil/pcie-mobiveil.h
--
2.17.1
From: Hou Zhiqiang <[email protected]>
Move the host initialization related operations into a new
routine to make it can be reused by other incoming platform's
PCIe host driver, in which the Mobiveil GPEX is integrated.
Signed-off-by: Hou Zhiqiang <[email protected]>
---
V9:
- New patch splited from the #1 of V8 patches to make it easy to review.
drivers/pci/controller/pcie-mobiveil.c | 38 +++++++++++++++-----------
1 file changed, 22 insertions(+), 16 deletions(-)
diff --git a/drivers/pci/controller/pcie-mobiveil.c b/drivers/pci/controller/pcie-mobiveil.c
index 5fd26e376af2..97f682ca7c7a 100644
--- a/drivers/pci/controller/pcie-mobiveil.c
+++ b/drivers/pci/controller/pcie-mobiveil.c
@@ -873,27 +873,15 @@ static int mobiveil_pcie_init_irq_domain(struct mobiveil_pcie *pcie)
return 0;
}
-static int mobiveil_pcie_probe(struct platform_device *pdev)
+int mobiveil_pcie_host_probe(struct mobiveil_pcie *pcie)
{
- struct mobiveil_pcie *pcie;
+ struct root_port *rp = &pcie->rp;
+ struct pci_host_bridge *bridge = rp->bridge;
+ struct device *dev = &pcie->pdev->dev;
struct pci_bus *bus;
struct pci_bus *child;
- struct pci_host_bridge *bridge;
- struct device *dev = &pdev->dev;
- struct root_port *rp;
int ret;
- /* allocate the PCIe port */
- bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
- if (!bridge)
- return -ENOMEM;
-
- pcie = pci_host_bridge_priv(bridge);
- rp = &pcie->rp;
- rp->bridge = bridge;
-
- pcie->pdev = pdev;
-
ret = mobiveil_pcie_parse_dt(pcie);
if (ret) {
dev_err(dev, "Parsing DT failed, ret: %x\n", ret);
@@ -956,6 +944,24 @@ static int mobiveil_pcie_probe(struct platform_device *pdev)
return 0;
}
+static int mobiveil_pcie_probe(struct platform_device *pdev)
+{
+ struct mobiveil_pcie *pcie;
+ struct pci_host_bridge *bridge;
+ struct device *dev = &pdev->dev;
+
+ bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
+ if (!bridge)
+ return -ENOMEM;
+
+ pcie = pci_host_bridge_priv(bridge);
+ pcie->rp.bridge = bridge;
+
+ pcie->pdev = pdev;
+
+ return mobiveil_pcie_host_probe(pcie);
+}
+
static const struct of_device_id mobiveil_pcie_of_match[] = {
{.compatible = "mbvl,gpex40-pcie",},
{},
--
2.17.1
From: Hou Zhiqiang <[email protected]>
Collect the interrupt initialization related operations into
a new routine to make it more readable.
Signed-off-by: Hou Zhiqiang <[email protected]>
---
V9:
- New patch splited from the #1 of V8 patches to make it easy to review.
drivers/pci/controller/pcie-mobiveil.c | 65 +++++++++++++++++---------
1 file changed, 42 insertions(+), 23 deletions(-)
diff --git a/drivers/pci/controller/pcie-mobiveil.c b/drivers/pci/controller/pcie-mobiveil.c
index 97f682ca7c7a..512b27a0536e 100644
--- a/drivers/pci/controller/pcie-mobiveil.c
+++ b/drivers/pci/controller/pcie-mobiveil.c
@@ -454,12 +454,6 @@ static int mobiveil_pcie_parse_dt(struct mobiveil_pcie *pcie)
return PTR_ERR(pcie->csr_axi_slave_base);
pcie->pcie_reg_base = res->start;
- /* map MSI config resource */
- res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "apb_csr");
- pcie->apb_csr_base = devm_pci_remap_cfg_resource(dev, res);
- if (IS_ERR(pcie->apb_csr_base))
- return PTR_ERR(pcie->apb_csr_base);
-
/* read the number of windows requested */
if (of_property_read_u32(node, "apio-wins", &pcie->apio_wins))
pcie->apio_wins = MAX_PIO_WINDOWS;
@@ -467,12 +461,6 @@ static int mobiveil_pcie_parse_dt(struct mobiveil_pcie *pcie)
if (of_property_read_u32(node, "ppio-wins", &pcie->ppio_wins))
pcie->ppio_wins = MAX_PIO_WINDOWS;
- rp->irq = platform_get_irq(pdev, 0);
- if (rp->irq <= 0) {
- dev_err(dev, "failed to map IRQ: %d\n", rp->irq);
- return -ENODEV;
- }
-
return 0;
}
@@ -618,9 +606,6 @@ static int mobiveil_host_init(struct mobiveil_pcie *pcie)
pab_ctrl |= (1 << AMBA_PIO_ENABLE_SHIFT) | (1 << PEX_PIO_ENABLE_SHIFT);
mobiveil_csr_writel(pcie, pab_ctrl, PAB_CTRL);
- mobiveil_csr_writel(pcie, (PAB_INTP_INTX_MASK | PAB_INTP_MSI_MASK),
- PAB_INTP_AMBA_MISC_ENB);
-
/*
* program PIO Enable Bit to 1 and Config Window Enable Bit to 1 in
* PAB_AXI_PIO_CTRL Register
@@ -670,9 +655,6 @@ static int mobiveil_host_init(struct mobiveil_pcie *pcie)
value |= (PCI_CLASS_BRIDGE_PCI << 16);
mobiveil_csr_writel(pcie, value, PAB_INTP_AXI_PIO_CLASS);
- /* setup MSI hardware registers */
- mobiveil_pcie_enable_msi(pcie);
-
return 0;
}
@@ -873,6 +855,46 @@ static int mobiveil_pcie_init_irq_domain(struct mobiveil_pcie *pcie)
return 0;
}
+static int mobiveil_pcie_interrupt_init(struct mobiveil_pcie *pcie)
+{
+ struct platform_device *pdev = pcie->pdev;
+ struct device *dev = &pdev->dev;
+ struct root_port *rp = &pcie->rp;
+ struct resource *res;
+ int ret;
+
+ /* map MSI config resource */
+ res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "apb_csr");
+ pcie->apb_csr_base = devm_pci_remap_cfg_resource(dev, res);
+ if (IS_ERR(pcie->apb_csr_base))
+ return PTR_ERR(pcie->apb_csr_base);
+
+ /* setup MSI hardware registers */
+ mobiveil_pcie_enable_msi(pcie);
+
+ rp->irq = platform_get_irq(pdev, 0);
+ if (rp->irq <= 0) {
+ dev_err(dev, "failed to map IRQ: %d\n", rp->irq);
+ return -ENODEV;
+ }
+
+ /* initialize the IRQ domains */
+ ret = mobiveil_pcie_init_irq_domain(pcie);
+ if (ret) {
+ dev_err(dev, "Failed creating IRQ Domain\n");
+ return ret;
+ }
+
+ irq_set_chained_handler_and_data(rp->irq, mobiveil_pcie_isr, pcie);
+
+ /* Enable interrupts */
+ mobiveil_csr_writel(pcie, (PAB_INTP_INTX_MASK | PAB_INTP_MSI_MASK),
+ PAB_INTP_AMBA_MISC_ENB);
+
+
+ return 0;
+}
+
int mobiveil_pcie_host_probe(struct mobiveil_pcie *pcie)
{
struct root_port *rp = &pcie->rp;
@@ -906,15 +928,12 @@ int mobiveil_pcie_host_probe(struct mobiveil_pcie *pcie)
return ret;
}
- /* initialize the IRQ domains */
- ret = mobiveil_pcie_init_irq_domain(pcie);
+ ret = mobiveil_pcie_interrupt_init(pcie);
if (ret) {
- dev_err(dev, "Failed creating IRQ Domain\n");
+ dev_err(dev, "Interrupt init failed\n");
return ret;
}
- irq_set_chained_handler_and_data(rp->irq, mobiveil_pcie_isr, pcie);
-
/* Initialize bridge */
bridge->dev.parent = dev;
bridge->sysdata = pcie;
--
2.17.1
From: Hou Zhiqiang <[email protected]>
Enable the PCIe Gen4 controller driver for Layerscape SoCs.
Signed-off-by: Hou Zhiqiang <[email protected]>
Reviewed-by: Minghuan Lian <[email protected]>
---
V9:
- No change
arch/arm64/configs/defconfig | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig
index 7fa92defb964..0e51207b5ed5 100644
--- a/arch/arm64/configs/defconfig
+++ b/arch/arm64/configs/defconfig
@@ -199,6 +199,7 @@ CONFIG_PCI_HOST_THUNDER_PEM=y
CONFIG_PCI_HOST_THUNDER_ECAM=y
CONFIG_PCIE_ROCKCHIP_HOST=m
CONFIG_PCI_LAYERSCAPE=y
+CONFIG_PCIE_LAYERSCAPE_GEN4=y
CONFIG_PCI_HISI=y
CONFIG_PCIE_QCOM=y
CONFIG_PCIE_ARMADA_8K=y
--
2.17.1
From: Hou Zhiqiang <[email protected]>
Make the mobiveil_host_init() function can be used to re-init
host controller's PAB and GPEX CSR register block, as NXP
integrated Mobiveil IP has to reset and then re-init the PAB
and GPEX CSR registers upon hot-reset.
Signed-off-by: Hou Zhiqiang <[email protected]>
Reviewed-by: Subrahmanya Lingappa <[email protected]>
---
V9:
- No change
.../controller/mobiveil/pcie-mobiveil-host.c | 19 ++++++++++++-------
.../pci/controller/mobiveil/pcie-mobiveil.h | 1 +
2 files changed, 13 insertions(+), 7 deletions(-)
diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c b/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
index 3cd93df6fe6e..9bc3da036720 100644
--- a/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
+++ b/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
@@ -221,18 +221,23 @@ static void mobiveil_pcie_enable_msi(struct mobiveil_pcie *pcie)
writel_relaxed(1, pcie->apb_csr_base + MSI_ENABLE_OFFSET);
}
-static int mobiveil_host_init(struct mobiveil_pcie *pcie)
+int mobiveil_host_init(struct mobiveil_pcie *pcie, bool reinit)
{
struct root_port *rp = &pcie->rp;
struct pci_host_bridge *bridge = rp->bridge;
u32 value, pab_ctrl, type;
struct resource_entry *win;
- /* setup bus numbers */
- value = mobiveil_csr_readl(pcie, PCI_PRIMARY_BUS);
- value &= 0xff000000;
- value |= 0x00ff0100;
- mobiveil_csr_writel(pcie, value, PCI_PRIMARY_BUS);
+ pcie->ib_wins_configured = 0;
+ pcie->ob_wins_configured = 0;
+
+ if (!reinit) {
+ /* setup bus numbers */
+ value = mobiveil_csr_readl(pcie, PCI_PRIMARY_BUS);
+ value &= 0xff000000;
+ value |= 0x00ff0100;
+ mobiveil_csr_writel(pcie, value, PCI_PRIMARY_BUS);
+ }
/*
* program Bus Master Enable Bit in Command Register in PAB Config
@@ -569,7 +574,7 @@ int mobiveil_pcie_host_probe(struct mobiveil_pcie *pcie)
* configure all inbound and outbound windows and prepare the RC for
* config access
*/
- ret = mobiveil_host_init(pcie);
+ ret = mobiveil_host_init(pcie, false);
if (ret) {
dev_err(dev, "Failed to initialize host\n");
return ret;
diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.h b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
index 95d2e7c809b8..37116c2a19fe 100644
--- a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
+++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
@@ -166,6 +166,7 @@ struct mobiveil_pcie {
};
int mobiveil_pcie_host_probe(struct mobiveil_pcie *pcie);
+int mobiveil_host_init(struct mobiveil_pcie *pcie, bool reinit);
bool mobiveil_pcie_link_up(struct mobiveil_pcie *pcie);
int mobiveil_bringup_link(struct mobiveil_pcie *pcie);
void program_ob_windows(struct mobiveil_pcie *pcie, int win_num, u64 cpu_addr,
--
2.17.1
From: Hou Zhiqiang <[email protected]>
Add PCIe Gen4 controller DT bindings of NXP Layerscape SoCs.
Signed-off-by: Hou Zhiqiang <[email protected]>
Reviewed-by: Rob Herring <[email protected]>
---
V9:
- No change
.../bindings/pci/layerscape-pcie-gen4.txt | 52 +++++++++++++++++++
MAINTAINERS | 8 +++
2 files changed, 60 insertions(+)
create mode 100644 Documentation/devicetree/bindings/pci/layerscape-pcie-gen4.txt
diff --git a/Documentation/devicetree/bindings/pci/layerscape-pcie-gen4.txt b/Documentation/devicetree/bindings/pci/layerscape-pcie-gen4.txt
new file mode 100644
index 000000000000..b40fb5d15d3d
--- /dev/null
+++ b/Documentation/devicetree/bindings/pci/layerscape-pcie-gen4.txt
@@ -0,0 +1,52 @@
+NXP Layerscape PCIe Gen4 controller
+
+This PCIe controller is based on the Mobiveil PCIe IP and thus inherits all
+the common properties defined in mobiveil-pcie.txt.
+
+Required properties:
+- compatible: should contain the platform identifier such as:
+ "fsl,lx2160a-pcie"
+- reg: base addresses and lengths of the PCIe controller register blocks.
+ "csr_axi_slave": Bridge config registers
+ "config_axi_slave": PCIe controller registers
+- interrupts: A list of interrupt outputs of the controller. Must contain an
+ entry for each entry in the interrupt-names property.
+- interrupt-names: It could include the following entries:
+ "intr": The interrupt that is asserted for controller interrupts
+ "aer": Asserted for aer interrupt when chip support the aer interrupt with
+ none MSI/MSI-X/INTx mode,but there is interrupt line for aer.
+ "pme": Asserted for pme interrupt when chip support the pme interrupt with
+ none MSI/MSI-X/INTx mode,but there is interrupt line for pme.
+- dma-coherent: Indicates that the hardware IP block can ensure the coherency
+ of the data transferred from/to the IP block. This can avoid the software
+ cache flush/invalid actions, and improve the performance significantly.
+- msi-parent : See the generic MSI binding described in
+ Documentation/devicetree/bindings/interrupt-controller/msi.txt.
+
+Example:
+
+ pcie@3400000 {
+ compatible = "fsl,lx2160a-pcie";
+ reg = <0x00 0x03400000 0x0 0x00100000 /* controller registers */
+ 0x80 0x00000000 0x0 0x00001000>; /* configuration space */
+ reg-names = "csr_axi_slave", "config_axi_slave";
+ interrupts = <GIC_SPI 108 IRQ_TYPE_LEVEL_HIGH>, /* AER interrupt */
+ <GIC_SPI 108 IRQ_TYPE_LEVEL_HIGH>, /* PME interrupt */
+ <GIC_SPI 108 IRQ_TYPE_LEVEL_HIGH>; /* controller interrupt */
+ interrupt-names = "aer", "pme", "intr";
+ #address-cells = <3>;
+ #size-cells = <2>;
+ device_type = "pci";
+ apio-wins = <8>;
+ ppio-wins = <8>;
+ dma-coherent;
+ bus-range = <0x0 0xff>;
+ msi-parent = <&its>;
+ ranges = <0x82000000 0x0 0x40000000 0x80 0x40000000 0x0 0x40000000>;
+ #interrupt-cells = <1>;
+ interrupt-map-mask = <0 0 0 7>;
+ interrupt-map = <0000 0 0 1 &gic 0 0 GIC_SPI 109 IRQ_TYPE_LEVEL_HIGH>,
+ <0000 0 0 2 &gic 0 0 GIC_SPI 110 IRQ_TYPE_LEVEL_HIGH>,
+ <0000 0 0 3 &gic 0 0 GIC_SPI 111 IRQ_TYPE_LEVEL_HIGH>,
+ <0000 0 0 4 &gic 0 0 GIC_SPI 112 IRQ_TYPE_LEVEL_HIGH>;
+ };
diff --git a/MAINTAINERS b/MAINTAINERS
index a4ad99619e53..2f68f71896c7 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -12569,6 +12569,14 @@ L: [email protected]
S: Maintained
F: drivers/pci/controller/dwc/*layerscape*
+PCI DRIVER FOR NXP LAYERSCAPE GEN4 CONTROLLER
+M: Hou Zhiqiang <[email protected]>
+L: [email protected]
+L: [email protected]
+S: Maintained
+F: Documentation/devicetree/bindings/pci/layerscape-pcie-gen4.txt
+F: drivers/pci/controller/mobibeil/pcie-layerscape-gen4.c
+
PCI DRIVER FOR GENERIC OF HOSTS
M: Will Deacon <[email protected]>
L: [email protected]
--
2.17.1
From: Hou Zhiqiang <[email protected]>
The Mobiveil GPEX internal MSI/INTx controller may not be used
by other platforms in which the Mobiveil GPEX is integrated.
This patch is to allow these platforms to implement their
specific interrupt initialization.
Signed-off-by: Hou Zhiqiang <[email protected]>
---
V9:
- New patch splited from the #1 of V8 patches to make it easy to review.
drivers/pci/controller/mobiveil/pcie-mobiveil-host.c | 3 +++
drivers/pci/controller/mobiveil/pcie-mobiveil.h | 7 +++++++
2 files changed, 10 insertions(+)
diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c b/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
index 2cc424e78d33..3cd93df6fe6e 100644
--- a/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
+++ b/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
@@ -507,6 +507,9 @@ static int mobiveil_pcie_interrupt_init(struct mobiveil_pcie *pcie)
struct resource *res;
int ret;
+ if (rp->ops->interrupt_init)
+ return rp->ops->interrupt_init(pcie);
+
/* map MSI config resource */
res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "apb_csr");
pcie->apb_csr_base = devm_pci_remap_cfg_resource(dev, res);
diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.h b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
index e3148078e9dd..18d85806a7fc 100644
--- a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
+++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
@@ -130,10 +130,17 @@ struct mobiveil_msi { /* MSI information */
DECLARE_BITMAP(msi_irq_in_use, PCI_NUM_MSI);
};
+struct mobiveil_pcie;
+
+struct mobiveil_rp_ops {
+ int (*interrupt_init)(struct mobiveil_pcie *pcie);
+};
+
struct root_port {
char root_bus_nr;
void __iomem *config_axi_slave_base; /* endpoint config base */
struct resource *ob_io_res;
+ struct mobiveil_rp_ops *ops;
int irq;
raw_spinlock_t intx_mask_lock;
struct irq_domain *intx_domain;
--
2.17.1
From: Hou Zhiqiang <[email protected]>
The LX2160A integrated 6 PCIe Gen4 controllers.
Signed-off-by: Hou Zhiqiang <[email protected]>
Reviewed-by: Minghuan Lian <[email protected]>
---
V9:
- No change
.../arm64/boot/dts/freescale/fsl-lx2160a.dtsi | 163 ++++++++++++++++++
1 file changed, 163 insertions(+)
diff --git a/arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi b/arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi
index e883fe0fc1b7..ecc5bd90aa5d 100644
--- a/arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi
+++ b/arch/arm64/boot/dts/freescale/fsl-lx2160a.dtsi
@@ -1055,5 +1055,168 @@
};
};
};
+
+ pcie@3400000 {
+ compatible = "fsl,lx2160a-pcie";
+ reg = <0x00 0x03400000 0x0 0x00100000 /* controller registers */
+ 0x80 0x00000000 0x0 0x00001000>; /* configuration space */
+ reg-names = "csr_axi_slave", "config_axi_slave";
+ interrupts = <GIC_SPI 108 IRQ_TYPE_LEVEL_HIGH>, /* AER interrupt */
+ <GIC_SPI 108 IRQ_TYPE_LEVEL_HIGH>, /* PME interrupt */
+ <GIC_SPI 108 IRQ_TYPE_LEVEL_HIGH>; /* controller interrupt */
+ interrupt-names = "aer", "pme", "intr";
+ #address-cells = <3>;
+ #size-cells = <2>;
+ device_type = "pci";
+ dma-coherent;
+ apio-wins = <8>;
+ ppio-wins = <8>;
+ bus-range = <0x0 0xff>;
+ ranges = <0x82000000 0x0 0x40000000 0x80 0x40000000 0x0 0x40000000>; /* non-prefetchable memory */
+ msi-parent = <&its>;
+ #interrupt-cells = <1>;
+ interrupt-map-mask = <0 0 0 7>;
+ interrupt-map = <0000 0 0 1 &gic 0 0 GIC_SPI 109 IRQ_TYPE_LEVEL_HIGH>,
+ <0000 0 0 2 &gic 0 0 GIC_SPI 110 IRQ_TYPE_LEVEL_HIGH>,
+ <0000 0 0 3 &gic 0 0 GIC_SPI 111 IRQ_TYPE_LEVEL_HIGH>,
+ <0000 0 0 4 &gic 0 0 GIC_SPI 112 IRQ_TYPE_LEVEL_HIGH>;
+ status = "disabled";
+ };
+
+ pcie@3500000 {
+ compatible = "fsl,lx2160a-pcie";
+ reg = <0x00 0x03500000 0x0 0x00100000 /* controller registers */
+ 0x88 0x00000000 0x0 0x00001000>; /* configuration space */
+ reg-names = "csr_axi_slave", "config_axi_slave";
+ interrupts = <GIC_SPI 113 IRQ_TYPE_LEVEL_HIGH>, /* AER interrupt */
+ <GIC_SPI 113 IRQ_TYPE_LEVEL_HIGH>, /* PME interrupt */
+ <GIC_SPI 113 IRQ_TYPE_LEVEL_HIGH>; /* controller interrupt */
+ interrupt-names = "aer", "pme", "intr";
+ #address-cells = <3>;
+ #size-cells = <2>;
+ device_type = "pci";
+ dma-coherent;
+ apio-wins = <8>;
+ ppio-wins = <8>;
+ bus-range = <0x0 0xff>;
+ ranges = <0x82000000 0x0 0x40000000 0x88 0x40000000 0x0 0x40000000>; /* non-prefetchable memory */
+ msi-parent = <&its>;
+ #interrupt-cells = <1>;
+ interrupt-map-mask = <0 0 0 7>;
+ interrupt-map = <0000 0 0 1 &gic 0 0 GIC_SPI 114 IRQ_TYPE_LEVEL_HIGH>,
+ <0000 0 0 2 &gic 0 0 GIC_SPI 115 IRQ_TYPE_LEVEL_HIGH>,
+ <0000 0 0 3 &gic 0 0 GIC_SPI 116 IRQ_TYPE_LEVEL_HIGH>,
+ <0000 0 0 4 &gic 0 0 GIC_SPI 117 IRQ_TYPE_LEVEL_HIGH>;
+ status = "disabled";
+ };
+
+ pcie@3600000 {
+ compatible = "fsl,lx2160a-pcie";
+ reg = <0x00 0x03600000 0x0 0x00100000 /* controller registers */
+ 0x90 0x00000000 0x0 0x00001000>; /* configuration space */
+ reg-names = "csr_axi_slave", "config_axi_slave";
+ interrupts = <GIC_SPI 118 IRQ_TYPE_LEVEL_HIGH>, /* AER interrupt */
+ <GIC_SPI 118 IRQ_TYPE_LEVEL_HIGH>, /* PME interrupt */
+ <GIC_SPI 118 IRQ_TYPE_LEVEL_HIGH>; /* controller interrupt */
+ interrupt-names = "aer", "pme", "intr";
+ #address-cells = <3>;
+ #size-cells = <2>;
+ device_type = "pci";
+ dma-coherent;
+ apio-wins = <256>;
+ ppio-wins = <24>;
+ bus-range = <0x0 0xff>;
+ ranges = <0x82000000 0x0 0x40000000 0x90 0x40000000 0x0 0x40000000>; /* non-prefetchable memory */
+ msi-parent = <&its>;
+ #interrupt-cells = <1>;
+ interrupt-map-mask = <0 0 0 7>;
+ interrupt-map = <0000 0 0 1 &gic 0 0 GIC_SPI 119 IRQ_TYPE_LEVEL_HIGH>,
+ <0000 0 0 2 &gic 0 0 GIC_SPI 120 IRQ_TYPE_LEVEL_HIGH>,
+ <0000 0 0 3 &gic 0 0 GIC_SPI 121 IRQ_TYPE_LEVEL_HIGH>,
+ <0000 0 0 4 &gic 0 0 GIC_SPI 122 IRQ_TYPE_LEVEL_HIGH>;
+ status = "disabled";
+ };
+
+ pcie@3700000 {
+ compatible = "fsl,lx2160a-pcie";
+ reg = <0x00 0x03700000 0x0 0x00100000 /* controller registers */
+ 0x98 0x00000000 0x0 0x00001000>; /* configuration space */
+ reg-names = "csr_axi_slave", "config_axi_slave";
+ interrupts = <GIC_SPI 123 IRQ_TYPE_LEVEL_HIGH>, /* AER interrupt */
+ <GIC_SPI 123 IRQ_TYPE_LEVEL_HIGH>, /* PME interrupt */
+ <GIC_SPI 123 IRQ_TYPE_LEVEL_HIGH>; /* controller interrupt */
+ interrupt-names = "aer", "pme", "intr";
+ #address-cells = <3>;
+ #size-cells = <2>;
+ device_type = "pci";
+ dma-coherent;
+ apio-wins = <8>;
+ ppio-wins = <8>;
+ bus-range = <0x0 0xff>;
+ ranges = <0x82000000 0x0 0x40000000 0x98 0x40000000 0x0 0x40000000>; /* non-prefetchable memory */
+ msi-parent = <&its>;
+ #interrupt-cells = <1>;
+ interrupt-map-mask = <0 0 0 7>;
+ interrupt-map = <0000 0 0 1 &gic 0 0 GIC_SPI 124 IRQ_TYPE_LEVEL_HIGH>,
+ <0000 0 0 2 &gic 0 0 GIC_SPI 125 IRQ_TYPE_LEVEL_HIGH>,
+ <0000 0 0 3 &gic 0 0 GIC_SPI 126 IRQ_TYPE_LEVEL_HIGH>,
+ <0000 0 0 4 &gic 0 0 GIC_SPI 127 IRQ_TYPE_LEVEL_HIGH>;
+ status = "disabled";
+ };
+
+ pcie@3800000 {
+ compatible = "fsl,lx2160a-pcie";
+ reg = <0x00 0x03800000 0x0 0x00100000 /* controller registers */
+ 0xa0 0x00000000 0x0 0x00001000>; /* configuration space */
+ reg-names = "csr_axi_slave", "config_axi_slave";
+ interrupts = <GIC_SPI 128 IRQ_TYPE_LEVEL_HIGH>, /* AER interrupt */
+ <GIC_SPI 128 IRQ_TYPE_LEVEL_HIGH>, /* PME interrupt */
+ <GIC_SPI 128 IRQ_TYPE_LEVEL_HIGH>; /* controller interrupt */
+ interrupt-names = "aer", "pme", "intr";
+ #address-cells = <3>;
+ #size-cells = <2>;
+ device_type = "pci";
+ dma-coherent;
+ apio-wins = <256>;
+ ppio-wins = <24>;
+ bus-range = <0x0 0xff>;
+ ranges = <0x82000000 0x0 0x40000000 0xa0 0x40000000 0x0 0x40000000>; /* non-prefetchable memory */
+ msi-parent = <&its>;
+ #interrupt-cells = <1>;
+ interrupt-map-mask = <0 0 0 7>;
+ interrupt-map = <0000 0 0 1 &gic 0 0 GIC_SPI 129 IRQ_TYPE_LEVEL_HIGH>,
+ <0000 0 0 2 &gic 0 0 GIC_SPI 130 IRQ_TYPE_LEVEL_HIGH>,
+ <0000 0 0 3 &gic 0 0 GIC_SPI 131 IRQ_TYPE_LEVEL_HIGH>,
+ <0000 0 0 4 &gic 0 0 GIC_SPI 132 IRQ_TYPE_LEVEL_HIGH>;
+ status = "disabled";
+ };
+
+ pcie@3900000 {
+ compatible = "fsl,lx2160a-pcie";
+ reg = <0x00 0x03900000 0x0 0x00100000 /* controller registers */
+ 0xa8 0x00000000 0x0 0x00001000>; /* configuration space */
+ reg-names = "csr_axi_slave", "config_axi_slave";
+ interrupts = <GIC_SPI 103 IRQ_TYPE_LEVEL_HIGH>, /* AER interrupt */
+ <GIC_SPI 103 IRQ_TYPE_LEVEL_HIGH>, /* PME interrupt */
+ <GIC_SPI 103 IRQ_TYPE_LEVEL_HIGH>; /* controller interrupt */
+ interrupt-names = "aer", "pme", "intr";
+ #address-cells = <3>;
+ #size-cells = <2>;
+ device_type = "pci";
+ dma-coherent;
+ apio-wins = <8>;
+ ppio-wins = <8>;
+ bus-range = <0x0 0xff>;
+ ranges = <0x82000000 0x0 0x40000000 0xa8 0x40000000 0x0 0x40000000>; /* non-prefetchable memory */
+ msi-parent = <&its>;
+ #interrupt-cells = <1>;
+ interrupt-map-mask = <0 0 0 7>;
+ interrupt-map = <0000 0 0 1 &gic 0 0 GIC_SPI 104 IRQ_TYPE_LEVEL_HIGH>,
+ <0000 0 0 2 &gic 0 0 GIC_SPI 105 IRQ_TYPE_LEVEL_HIGH>,
+ <0000 0 0 3 &gic 0 0 GIC_SPI 106 IRQ_TYPE_LEVEL_HIGH>,
+ <0000 0 0 4 &gic 0 0 GIC_SPI 107 IRQ_TYPE_LEVEL_HIGH>;
+ status = "disabled";
+ };
+
};
};
--
2.17.1
From: Hou Zhiqiang <[email protected]>
This PCIe controller is based on the Mobiveil GPEX IP, which is
compatible with the PCI Express™ Base Specification, Revision 4.0.
Signed-off-by: Hou Zhiqiang <[email protected]>
Reviewed-by: Minghuan Lian <[email protected]>
---
drivers/pci/controller/mobiveil/Kconfig | 10 +
drivers/pci/controller/mobiveil/Makefile | 1 +
.../mobiveil/pcie-layerscape-gen4.c | 274 ++++++++++++++++++
.../pci/controller/mobiveil/pcie-mobiveil.h | 16 +-
4 files changed, 299 insertions(+), 2 deletions(-)
create mode 100644 drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c
diff --git a/drivers/pci/controller/mobiveil/Kconfig b/drivers/pci/controller/mobiveil/Kconfig
index 64343c07bfed..c823be8dab1c 100644
--- a/drivers/pci/controller/mobiveil/Kconfig
+++ b/drivers/pci/controller/mobiveil/Kconfig
@@ -21,4 +21,14 @@ config PCIE_MOBIVEIL_PLAT
Soft IP. It has up to 8 outbound and inbound windows
for address translation and it is a PCIe Gen4 IP.
+config PCIE_LAYERSCAPE_GEN4
+ bool "Freescale Layerscape PCIe Gen4 controller"
+ depends on PCI
+ depends on OF && (ARM64 || ARCH_LAYERSCAPE)
+ depends on PCI_MSI_IRQ_DOMAIN
+ select PCIE_MOBIVEIL_HOST
+ help
+ Say Y here if you want PCIe Gen4 controller support on
+ Layerscape SoCs. The PCIe controller can work in RC or
+ EP mode according to RCW[HOST_AGT_PEX] setting.
endmenu
diff --git a/drivers/pci/controller/mobiveil/Makefile b/drivers/pci/controller/mobiveil/Makefile
index 9fb6d1c6504d..99d879de32d6 100644
--- a/drivers/pci/controller/mobiveil/Makefile
+++ b/drivers/pci/controller/mobiveil/Makefile
@@ -2,3 +2,4 @@
obj-$(CONFIG_PCIE_MOBIVEIL) += pcie-mobiveil.o
obj-$(CONFIG_PCIE_MOBIVEIL_HOST) += pcie-mobiveil-host.o
obj-$(CONFIG_PCIE_MOBIVEIL_PLAT) += pcie-mobiveil-plat.o
+obj-$(CONFIG_PCIE_LAYERSCAPE_GEN4) += pcie-layerscape-gen4.o
diff --git a/drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c b/drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c
new file mode 100644
index 000000000000..6c0d3e2532db
--- /dev/null
+++ b/drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c
@@ -0,0 +1,274 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * PCIe Gen4 host controller driver for NXP Layerscape SoCs
+ *
+ * Copyright 2019 NXP
+ *
+ * Author: Zhiqiang Hou <[email protected]>
+ */
+
+#include <linux/kernel.h>
+#include <linux/interrupt.h>
+#include <linux/init.h>
+#include <linux/of_pci.h>
+#include <linux/of_platform.h>
+#include <linux/of_irq.h>
+#include <linux/of_address.h>
+#include <linux/pci.h>
+#include <linux/platform_device.h>
+#include <linux/resource.h>
+#include <linux/mfd/syscon.h>
+#include <linux/regmap.h>
+
+#include "pcie-mobiveil.h"
+
+/* LUT and PF control registers */
+#define PCIE_LUT_OFF 0x80000
+#define PCIE_PF_OFF 0xc0000
+#define PCIE_PF_INT_STAT 0x18
+#define PF_INT_STAT_PABRST BIT(31)
+
+#define PCIE_PF_DBG 0x7fc
+#define PF_DBG_LTSSM_MASK 0x3f
+#define PF_DBG_LTSSM_L0 0x2d /* L0 state */
+#define PF_DBG_WE BIT(31)
+#define PF_DBG_PABR BIT(27)
+
+#define to_ls_pcie_g4(x) platform_get_drvdata((x)->pdev)
+
+struct ls_pcie_g4 {
+ struct mobiveil_pcie pci;
+ struct delayed_work dwork;
+ int irq;
+};
+
+static inline u32 ls_pcie_g4_lut_readl(struct ls_pcie_g4 *pcie, u32 off)
+{
+ return ioread32(pcie->pci.csr_axi_slave_base + PCIE_LUT_OFF + off);
+}
+
+static inline void ls_pcie_g4_lut_writel(struct ls_pcie_g4 *pcie,
+ u32 off, u32 val)
+{
+ iowrite32(val, pcie->pci.csr_axi_slave_base + PCIE_LUT_OFF + off);
+}
+
+static inline u32 ls_pcie_g4_pf_readl(struct ls_pcie_g4 *pcie, u32 off)
+{
+ return ioread32(pcie->pci.csr_axi_slave_base + PCIE_PF_OFF + off);
+}
+
+static inline void ls_pcie_g4_pf_writel(struct ls_pcie_g4 *pcie,
+ u32 off, u32 val)
+{
+ iowrite32(val, pcie->pci.csr_axi_slave_base + PCIE_PF_OFF + off);
+}
+
+static bool ls_pcie_g4_is_bridge(struct ls_pcie_g4 *pcie)
+{
+ struct mobiveil_pcie *mv_pci = &pcie->pci;
+ u32 header_type;
+
+ header_type = mobiveil_csr_readb(mv_pci, PCI_HEADER_TYPE);
+ header_type &= 0x7f;
+
+ return header_type == PCI_HEADER_TYPE_BRIDGE;
+}
+
+static int ls_pcie_g4_link_up(struct mobiveil_pcie *pci)
+{
+ struct ls_pcie_g4 *pcie = to_ls_pcie_g4(pci);
+ u32 state;
+
+ state = ls_pcie_g4_pf_readl(pcie, PCIE_PF_DBG);
+ state = state & PF_DBG_LTSSM_MASK;
+
+ if (state == PF_DBG_LTSSM_L0)
+ return 1;
+
+ return 0;
+}
+
+static void ls_pcie_g4_disable_interrupt(struct ls_pcie_g4 *pcie)
+{
+ struct mobiveil_pcie *mv_pci = &pcie->pci;
+
+ mobiveil_csr_writel(mv_pci, 0, PAB_INTP_AMBA_MISC_ENB);
+}
+
+static void ls_pcie_g4_enable_interrupt(struct ls_pcie_g4 *pcie)
+{
+ struct mobiveil_pcie *mv_pci = &pcie->pci;
+ u32 val;
+
+ /* Clear the interrupt status */
+ mobiveil_csr_writel(mv_pci, 0xffffffff, PAB_INTP_AMBA_MISC_STAT);
+
+ val = PAB_INTP_INTX_MASK | PAB_INTP_MSI | PAB_INTP_RESET |
+ PAB_INTP_PCIE_UE | PAB_INTP_IE_PMREDI | PAB_INTP_IE_EC;
+ mobiveil_csr_writel(mv_pci, val, PAB_INTP_AMBA_MISC_ENB);
+}
+
+static void ls_pcie_g4_reinit_hw(struct ls_pcie_g4 *pcie)
+{
+ struct mobiveil_pcie *mv_pci = &pcie->pci;
+ struct device *dev = &mv_pci->pdev->dev;
+ u32 val, act_stat;
+ int to = 100;
+
+ /* Poll for pab_csb_reset to set and PAB activity to clear */
+ do {
+ usleep_range(10, 15);
+ val = ls_pcie_g4_pf_readl(pcie, PCIE_PF_INT_STAT);
+ act_stat = mobiveil_csr_readl(mv_pci, PAB_ACTIVITY_STAT);
+ } while (((val & PF_INT_STAT_PABRST) == 0 || act_stat) && to--);
+ if (to < 0) {
+ dev_err(dev, "Poll PABRST&PABACT timeout\n");
+ return;
+ }
+
+ /* clear PEX_RESET bit in PEX_PF0_DBG register */
+ val = ls_pcie_g4_pf_readl(pcie, PCIE_PF_DBG);
+ val |= PF_DBG_WE;
+ ls_pcie_g4_pf_writel(pcie, PCIE_PF_DBG, val);
+
+ val = ls_pcie_g4_pf_readl(pcie, PCIE_PF_DBG);
+ val |= PF_DBG_PABR;
+ ls_pcie_g4_pf_writel(pcie, PCIE_PF_DBG, val);
+
+ val = ls_pcie_g4_pf_readl(pcie, PCIE_PF_DBG);
+ val &= ~PF_DBG_WE;
+ ls_pcie_g4_pf_writel(pcie, PCIE_PF_DBG, val);
+
+ mobiveil_host_init(mv_pci, true);
+
+ to = 100;
+ while (!ls_pcie_g4_link_up(mv_pci) && to--)
+ usleep_range(200, 250);
+ if (to < 0)
+ dev_err(dev, "PCIe link training timeout\n");
+}
+
+static irqreturn_t ls_pcie_g4_isr(int irq, void *dev_id)
+{
+ struct ls_pcie_g4 *pcie = (struct ls_pcie_g4 *)dev_id;
+ struct mobiveil_pcie *mv_pci = &pcie->pci;
+ u32 val;
+
+ val = mobiveil_csr_readl(mv_pci, PAB_INTP_AMBA_MISC_STAT);
+ if (!val)
+ return IRQ_NONE;
+
+ if (val & PAB_INTP_RESET) {
+ ls_pcie_g4_disable_interrupt(pcie);
+ schedule_delayed_work(&pcie->dwork, msecs_to_jiffies(1));
+ }
+
+ mobiveil_csr_writel(mv_pci, val, PAB_INTP_AMBA_MISC_STAT);
+
+ return IRQ_HANDLED;
+}
+
+static int ls_pcie_g4_interrupt_init(struct mobiveil_pcie *mv_pci)
+{
+ struct ls_pcie_g4 *pcie = to_ls_pcie_g4(mv_pci);
+ struct platform_device *pdev = mv_pci->pdev;
+ struct device *dev = &pdev->dev;
+ int ret;
+
+ pcie->irq = platform_get_irq_byname(pdev, "intr");
+ if (pcie->irq < 0) {
+ dev_err(dev, "Can't get 'intr' IRQ, errno = %d\n", pcie->irq);
+ return pcie->irq;
+ }
+ ret = devm_request_irq(dev, pcie->irq, ls_pcie_g4_isr,
+ IRQF_SHARED, pdev->name, pcie);
+ if (ret) {
+ dev_err(dev, "Can't register PCIe IRQ, errno = %d\n", ret);
+ return ret;
+ }
+
+ return 0;
+}
+
+static void ls_pcie_g4_reset(struct work_struct *work)
+{
+ struct delayed_work *dwork = container_of(work, struct delayed_work,
+ work);
+ struct ls_pcie_g4 *pcie = container_of(dwork, struct ls_pcie_g4, dwork);
+ struct mobiveil_pcie *mv_pci = &pcie->pci;
+ u16 ctrl;
+
+ ctrl = mobiveil_csr_readw(mv_pci, PCI_BRIDGE_CONTROL);
+ ctrl &= ~PCI_BRIDGE_CTL_BUS_RESET;
+ mobiveil_csr_writew(mv_pci, ctrl, PCI_BRIDGE_CONTROL);
+ ls_pcie_g4_reinit_hw(pcie);
+ ls_pcie_g4_enable_interrupt(pcie);
+}
+
+static struct mobiveil_rp_ops ls_pcie_g4_rp_ops = {
+ .interrupt_init = ls_pcie_g4_interrupt_init,
+};
+
+static const struct mobiveil_pab_ops ls_pcie_g4_pab_ops = {
+ .link_up = ls_pcie_g4_link_up,
+};
+
+static int __init ls_pcie_g4_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ struct pci_host_bridge *bridge;
+ struct mobiveil_pcie *mv_pci;
+ struct ls_pcie_g4 *pcie;
+ struct device_node *np = dev->of_node;
+ int ret;
+
+ if (!of_parse_phandle(np, "msi-parent", 0)) {
+ dev_err(dev, "Failed to find msi-parent\n");
+ return -EINVAL;
+ }
+
+ bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
+ if (!bridge)
+ return -ENOMEM;
+
+ pcie = pci_host_bridge_priv(bridge);
+ mv_pci = &pcie->pci;
+
+ mv_pci->pdev = pdev;
+ mv_pci->ops = &ls_pcie_g4_pab_ops;
+ mv_pci->rp.ops = &ls_pcie_g4_rp_ops;
+ mv_pci->rp.bridge = bridge;
+
+ platform_set_drvdata(pdev, pcie);
+
+ INIT_DELAYED_WORK(&pcie->dwork, ls_pcie_g4_reset);
+
+ ret = mobiveil_pcie_host_probe(mv_pci);
+ if (ret) {
+ dev_err(dev, "Fail to probe\n");
+ return ret;
+ }
+
+ if (!ls_pcie_g4_is_bridge(pcie))
+ return -ENODEV;
+
+ ls_pcie_g4_enable_interrupt(pcie);
+
+ return 0;
+}
+
+static const struct of_device_id ls_pcie_g4_of_match[] = {
+ { .compatible = "fsl,lx2160a-pcie", },
+ { },
+};
+
+static struct platform_driver ls_pcie_g4_driver = {
+ .driver = {
+ .name = "layerscape-pcie-gen4",
+ .of_match_table = ls_pcie_g4_of_match,
+ .suppress_bind_attrs = true,
+ },
+};
+
+builtin_platform_driver_probe(ls_pcie_g4_driver, ls_pcie_g4_probe);
diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.h b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
index 750a7fd95bc1..c57a68d2bac4 100644
--- a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
+++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
@@ -43,6 +43,8 @@
#define PAGE_LO_MASK 0x3ff
#define PAGE_SEL_OFFSET_SHIFT 10
+#define PAB_ACTIVITY_STAT 0x81c
+
#define PAB_AXI_PIO_CTRL 0x0840
#define APIO_EN_MASK 0xf
@@ -51,8 +53,18 @@
#define PAB_INTP_AMBA_MISC_ENB 0x0b0c
#define PAB_INTP_AMBA_MISC_STAT 0x0b1c
-#define PAB_INTP_INTX_MASK 0x01e0
-#define PAB_INTP_MSI_MASK 0x8
+#define PAB_INTP_RESET BIT(1)
+#define PAB_INTP_MSI BIT(3)
+#define PAB_INTP_INTA BIT(5)
+#define PAB_INTP_INTB BIT(6)
+#define PAB_INTP_INTC BIT(7)
+#define PAB_INTP_INTD BIT(8)
+#define PAB_INTP_PCIE_UE BIT(9)
+#define PAB_INTP_IE_PMREDI BIT(29)
+#define PAB_INTP_IE_EC BIT(30)
+#define PAB_INTP_MSI_MASK PAB_INTP_MSI
+#define PAB_INTP_INTX_MASK (PAB_INTP_INTA | PAB_INTP_INTB |\
+ PAB_INTP_INTC | PAB_INTP_INTD)
#define PAB_AXI_AMAP_CTRL(win) PAB_REG_ADDR(0x0ba0, win)
#define WIN_ENABLE_SHIFT 0
--
2.17.1
From: Hou Zhiqiang <[email protected]>
The Mobiveil PCIe controller can work in either Root Complex
mode or Endpoint mode. So introduce a new structure root_port,
and abstract the RC related members into it.
Signed-off-by: Hou Zhiqiang <[email protected]>
---
V9:
- New patch splited from the #1 of V8 patches to make it easy to review.
drivers/pci/controller/pcie-mobiveil.c | 99 ++++++++++++++++----------
1 file changed, 60 insertions(+), 39 deletions(-)
diff --git a/drivers/pci/controller/pcie-mobiveil.c b/drivers/pci/controller/pcie-mobiveil.c
index 3a696ca45bfa..5fd26e376af2 100644
--- a/drivers/pci/controller/pcie-mobiveil.c
+++ b/drivers/pci/controller/pcie-mobiveil.c
@@ -3,7 +3,10 @@
* PCIe host controller driver for Mobiveil PCIe Host controller
*
* Copyright (c) 2018 Mobiveil Inc.
+ * Copyright 2019 NXP
+ *
* Author: Subrahmanya Lingappa <[email protected]>
+ * Recode: Hou Zhiqiang <[email protected]>
*/
#include <linux/delay.h>
@@ -138,22 +141,27 @@ struct mobiveil_msi { /* MSI information */
DECLARE_BITMAP(msi_irq_in_use, PCI_NUM_MSI);
};
+struct root_port {
+ char root_bus_nr;
+ void __iomem *config_axi_slave_base; /* endpoint config base */
+ struct resource *ob_io_res;
+ int irq;
+ raw_spinlock_t intx_mask_lock;
+ struct irq_domain *intx_domain;
+ struct mobiveil_msi msi;
+ struct pci_host_bridge *bridge;
+};
+
struct mobiveil_pcie {
struct platform_device *pdev;
- void __iomem *config_axi_slave_base; /* endpoint config base */
void __iomem *csr_axi_slave_base; /* root port config base */
void __iomem *apb_csr_base; /* MSI register base */
phys_addr_t pcie_reg_base; /* Physical PCIe Controller Base */
- struct irq_domain *intx_domain;
- raw_spinlock_t intx_mask_lock;
- int irq;
int apio_wins;
int ppio_wins;
int ob_wins_configured; /* configured outbound windows */
int ib_wins_configured; /* configured inbound windows */
- struct resource *ob_io_res;
- char root_bus_nr;
- struct mobiveil_msi msi;
+ struct root_port rp;
};
/*
@@ -281,16 +289,17 @@ static bool mobiveil_pcie_link_up(struct mobiveil_pcie *pcie)
static bool mobiveil_pcie_valid_device(struct pci_bus *bus, unsigned int devfn)
{
struct mobiveil_pcie *pcie = bus->sysdata;
+ struct root_port *rp = &pcie->rp;
/* Only one device down on each root port */
- if ((bus->number == pcie->root_bus_nr) && (devfn > 0))
+ if ((bus->number == rp->root_bus_nr) && (devfn > 0))
return false;
/*
* Do not read more than one device on the bus directly
* attached to RC
*/
- if ((bus->primary == pcie->root_bus_nr) && (PCI_SLOT(devfn) > 0))
+ if ((bus->primary == rp->root_bus_nr) && (PCI_SLOT(devfn) > 0))
return false;
return true;
@@ -304,13 +313,14 @@ static void __iomem *mobiveil_pcie_map_bus(struct pci_bus *bus,
unsigned int devfn, int where)
{
struct mobiveil_pcie *pcie = bus->sysdata;
+ struct root_port *rp = &pcie->rp;
u32 value;
if (!mobiveil_pcie_valid_device(bus, devfn))
return NULL;
/* RC config access */
- if (bus->number == pcie->root_bus_nr)
+ if (bus->number == rp->root_bus_nr)
return pcie->csr_axi_slave_base + where;
/*
@@ -325,7 +335,7 @@ static void __iomem *mobiveil_pcie_map_bus(struct pci_bus *bus,
mobiveil_csr_writel(pcie, value, PAB_AXI_AMAP_PEX_WIN_L(WIN_NUM_0));
- return pcie->config_axi_slave_base + where;
+ return rp->config_axi_slave_base + where;
}
static struct pci_ops mobiveil_pcie_ops = {
@@ -339,7 +349,8 @@ static void mobiveil_pcie_isr(struct irq_desc *desc)
struct irq_chip *chip = irq_desc_get_chip(desc);
struct mobiveil_pcie *pcie = irq_desc_get_handler_data(desc);
struct device *dev = &pcie->pdev->dev;
- struct mobiveil_msi *msi = &pcie->msi;
+ struct root_port *rp = &pcie->rp;
+ struct mobiveil_msi *msi = &rp->msi;
u32 msi_data, msi_addr_lo, msi_addr_hi;
u32 intr_status, msi_status;
unsigned long shifted_status;
@@ -365,7 +376,7 @@ static void mobiveil_pcie_isr(struct irq_desc *desc)
shifted_status >>= PAB_INTX_START;
do {
for_each_set_bit(bit, &shifted_status, PCI_NUM_INTX) {
- virq = irq_find_mapping(pcie->intx_domain,
+ virq = irq_find_mapping(rp->intx_domain,
bit + 1);
if (virq)
generic_handle_irq(virq);
@@ -424,15 +435,16 @@ static int mobiveil_pcie_parse_dt(struct mobiveil_pcie *pcie)
struct device *dev = &pcie->pdev->dev;
struct platform_device *pdev = pcie->pdev;
struct device_node *node = dev->of_node;
+ struct root_port *rp = &pcie->rp;
struct resource *res;
/* map config resource */
res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
"config_axi_slave");
- pcie->config_axi_slave_base = devm_pci_remap_cfg_resource(dev, res);
- if (IS_ERR(pcie->config_axi_slave_base))
- return PTR_ERR(pcie->config_axi_slave_base);
- pcie->ob_io_res = res;
+ rp->config_axi_slave_base = devm_pci_remap_cfg_resource(dev, res);
+ if (IS_ERR(rp->config_axi_slave_base))
+ return PTR_ERR(rp->config_axi_slave_base);
+ rp->ob_io_res = res;
/* map csr resource */
res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
@@ -455,9 +467,9 @@ static int mobiveil_pcie_parse_dt(struct mobiveil_pcie *pcie)
if (of_property_read_u32(node, "ppio-wins", &pcie->ppio_wins))
pcie->ppio_wins = MAX_PIO_WINDOWS;
- pcie->irq = platform_get_irq(pdev, 0);
- if (pcie->irq <= 0) {
- dev_err(dev, "failed to map IRQ: %d\n", pcie->irq);
+ rp->irq = platform_get_irq(pdev, 0);
+ if (rp->irq <= 0) {
+ dev_err(dev, "failed to map IRQ: %d\n", rp->irq);
return -ENODEV;
}
@@ -564,9 +576,9 @@ static int mobiveil_bringup_link(struct mobiveil_pcie *pcie)
static void mobiveil_pcie_enable_msi(struct mobiveil_pcie *pcie)
{
phys_addr_t msg_addr = pcie->pcie_reg_base;
- struct mobiveil_msi *msi = &pcie->msi;
+ struct mobiveil_msi *msi = &pcie->rp.msi;
- pcie->msi.num_of_vectors = PCI_NUM_MSI;
+ msi->num_of_vectors = PCI_NUM_MSI;
msi->msi_pages_phys = (phys_addr_t)msg_addr;
writel_relaxed(lower_32_bits(msg_addr),
@@ -579,7 +591,8 @@ static void mobiveil_pcie_enable_msi(struct mobiveil_pcie *pcie)
static int mobiveil_host_init(struct mobiveil_pcie *pcie)
{
- struct pci_host_bridge *bridge = pci_host_bridge_from_priv(pcie);
+ struct root_port *rp = &pcie->rp;
+ struct pci_host_bridge *bridge = rp->bridge;
u32 value, pab_ctrl, type;
struct resource_entry *win;
@@ -629,8 +642,8 @@ static int mobiveil_host_init(struct mobiveil_pcie *pcie)
*/
/* config outbound translation window */
- program_ob_windows(pcie, WIN_NUM_0, pcie->ob_io_res->start, 0,
- CFG_WINDOW_TYPE, resource_size(pcie->ob_io_res));
+ program_ob_windows(pcie, WIN_NUM_0, rp->ob_io_res->start, 0,
+ CFG_WINDOW_TYPE, resource_size(rp->ob_io_res));
/* memory inbound translation window */
program_ib_windows(pcie, WIN_NUM_0, 0, 0, MEM_WINDOW_TYPE, IB_WIN_SIZE);
@@ -667,32 +680,36 @@ static void mobiveil_mask_intx_irq(struct irq_data *data)
{
struct irq_desc *desc = irq_to_desc(data->irq);
struct mobiveil_pcie *pcie;
+ struct root_port *rp;
unsigned long flags;
u32 mask, shifted_val;
pcie = irq_desc_get_chip_data(desc);
+ rp = &pcie->rp;
mask = 1 << ((data->hwirq + PAB_INTX_START) - 1);
- raw_spin_lock_irqsave(&pcie->intx_mask_lock, flags);
+ raw_spin_lock_irqsave(&rp->intx_mask_lock, flags);
shifted_val = mobiveil_csr_readl(pcie, PAB_INTP_AMBA_MISC_ENB);
shifted_val &= ~mask;
mobiveil_csr_writel(pcie, shifted_val, PAB_INTP_AMBA_MISC_ENB);
- raw_spin_unlock_irqrestore(&pcie->intx_mask_lock, flags);
+ raw_spin_unlock_irqrestore(&rp->intx_mask_lock, flags);
}
static void mobiveil_unmask_intx_irq(struct irq_data *data)
{
struct irq_desc *desc = irq_to_desc(data->irq);
struct mobiveil_pcie *pcie;
+ struct root_port *rp;
unsigned long flags;
u32 shifted_val, mask;
pcie = irq_desc_get_chip_data(desc);
+ rp = &pcie->rp;
mask = 1 << ((data->hwirq + PAB_INTX_START) - 1);
- raw_spin_lock_irqsave(&pcie->intx_mask_lock, flags);
+ raw_spin_lock_irqsave(&rp->intx_mask_lock, flags);
shifted_val = mobiveil_csr_readl(pcie, PAB_INTP_AMBA_MISC_ENB);
shifted_val |= mask;
mobiveil_csr_writel(pcie, shifted_val, PAB_INTP_AMBA_MISC_ENB);
- raw_spin_unlock_irqrestore(&pcie->intx_mask_lock, flags);
+ raw_spin_unlock_irqrestore(&rp->intx_mask_lock, flags);
}
static struct irq_chip intx_irq_chip = {
@@ -760,7 +777,7 @@ static int mobiveil_irq_msi_domain_alloc(struct irq_domain *domain,
unsigned int nr_irqs, void *args)
{
struct mobiveil_pcie *pcie = domain->host_data;
- struct mobiveil_msi *msi = &pcie->msi;
+ struct mobiveil_msi *msi = &pcie->rp.msi;
unsigned long bit;
WARN_ON(nr_irqs != 1);
@@ -787,7 +804,7 @@ static void mobiveil_irq_msi_domain_free(struct irq_domain *domain,
{
struct irq_data *d = irq_domain_get_irq_data(domain, virq);
struct mobiveil_pcie *pcie = irq_data_get_irq_chip_data(d);
- struct mobiveil_msi *msi = &pcie->msi;
+ struct mobiveil_msi *msi = &pcie->rp.msi;
mutex_lock(&msi->lock);
@@ -808,9 +825,9 @@ static int mobiveil_allocate_msi_domains(struct mobiveil_pcie *pcie)
{
struct device *dev = &pcie->pdev->dev;
struct fwnode_handle *fwnode = of_node_to_fwnode(dev->of_node);
- struct mobiveil_msi *msi = &pcie->msi;
+ struct mobiveil_msi *msi = &pcie->rp.msi;
- mutex_init(&pcie->msi.lock);
+ mutex_init(&msi->lock);
msi->dev_domain = irq_domain_add_linear(NULL, msi->num_of_vectors,
&msi_domain_ops, pcie);
if (!msi->dev_domain) {
@@ -834,18 +851,19 @@ static int mobiveil_pcie_init_irq_domain(struct mobiveil_pcie *pcie)
{
struct device *dev = &pcie->pdev->dev;
struct device_node *node = dev->of_node;
+ struct root_port *rp = &pcie->rp;
int ret;
/* setup INTx */
- pcie->intx_domain = irq_domain_add_linear(node, PCI_NUM_INTX,
- &intx_domain_ops, pcie);
+ rp->intx_domain = irq_domain_add_linear(node, PCI_NUM_INTX,
+ &intx_domain_ops, pcie);
- if (!pcie->intx_domain) {
+ if (!rp->intx_domain) {
dev_err(dev, "Failed to get a INTx IRQ domain\n");
return -ENOMEM;
}
- raw_spin_lock_init(&pcie->intx_mask_lock);
+ raw_spin_lock_init(&rp->intx_mask_lock);
/* setup MSI */
ret = mobiveil_allocate_msi_domains(pcie);
@@ -862,6 +880,7 @@ static int mobiveil_pcie_probe(struct platform_device *pdev)
struct pci_bus *child;
struct pci_host_bridge *bridge;
struct device *dev = &pdev->dev;
+ struct root_port *rp;
int ret;
/* allocate the PCIe port */
@@ -870,6 +889,8 @@ static int mobiveil_pcie_probe(struct platform_device *pdev)
return -ENOMEM;
pcie = pci_host_bridge_priv(bridge);
+ rp = &pcie->rp;
+ rp->bridge = bridge;
pcie->pdev = pdev;
@@ -904,12 +925,12 @@ static int mobiveil_pcie_probe(struct platform_device *pdev)
return ret;
}
- irq_set_chained_handler_and_data(pcie->irq, mobiveil_pcie_isr, pcie);
+ irq_set_chained_handler_and_data(rp->irq, mobiveil_pcie_isr, pcie);
/* Initialize bridge */
bridge->dev.parent = dev;
bridge->sysdata = pcie;
- bridge->busnr = pcie->root_bus_nr;
+ bridge->busnr = rp->root_bus_nr;
bridge->ops = &mobiveil_pcie_ops;
bridge->map_irq = of_irq_parse_and_map_pci;
bridge->swizzle_irq = pci_common_swizzle;
--
2.17.1
From: Hou Zhiqiang <[email protected]>
Modularize the Mobiveil PCIe host driver according to the
abstraction of Root Complex and Endpoint and move it into
a new directory.
Split the RC related routines into pcie-mobiveil-host.c,
and common routines into pcie-mobiveil.c, move the macro
definitions and function declarations into pcie-mobiveil.h,
and the Mobiveil platform reference code into
pcie-mobiveil-plat.c. So that it is easy to reuse the
extracted routines to add a new host driver, which
integrated Mobiveil PCIe GPEX IP.
Signed-off-by: Hou Zhiqiang <[email protected]>
---
V9:
- New patch splited from the #1 of V8 patches to make it easy to review.
MAINTAINERS | 2 +-
drivers/pci/controller/Kconfig | 11 +-
drivers/pci/controller/Makefile | 2 +-
drivers/pci/controller/mobiveil/Kconfig | 24 ++
drivers/pci/controller/mobiveil/Makefile | 4 +
.../pcie-mobiveil-host.c} | 398 +-----------------
.../controller/mobiveil/pcie-mobiveil-plat.c | 60 +++
.../pci/controller/mobiveil/pcie-mobiveil.c | 227 ++++++++++
.../pci/controller/mobiveil/pcie-mobiveil.h | 178 ++++++++
9 files changed, 497 insertions(+), 409 deletions(-)
create mode 100644 drivers/pci/controller/mobiveil/Kconfig
create mode 100644 drivers/pci/controller/mobiveil/Makefile
rename drivers/pci/controller/{pcie-mobiveil.c => mobiveil/pcie-mobiveil-host.c} (61%)
create mode 100644 drivers/pci/controller/mobiveil/pcie-mobiveil-plat.c
create mode 100644 drivers/pci/controller/mobiveil/pcie-mobiveil.c
create mode 100644 drivers/pci/controller/mobiveil/pcie-mobiveil.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 3f7f8cdbc471..a4ad99619e53 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -12612,7 +12612,7 @@ M: Hou Zhiqiang <[email protected]>
L: [email protected]
S: Supported
F: Documentation/devicetree/bindings/pci/mobiveil-pcie.txt
-F: drivers/pci/controller/pcie-mobiveil.c
+F: drivers/pci/controller/mobiveil/pcie-mobiveil*
PCI DRIVER FOR MVEBU (Marvell Armada 370 and Armada XP SOC support)
M: Thomas Petazzoni <[email protected]>
diff --git a/drivers/pci/controller/Kconfig b/drivers/pci/controller/Kconfig
index f5de9119e8d3..74fd332755ae 100644
--- a/drivers/pci/controller/Kconfig
+++ b/drivers/pci/controller/Kconfig
@@ -241,16 +241,6 @@ config PCIE_MEDIATEK
Say Y here if you want to enable PCIe controller support on
MediaTek SoCs.
-config PCIE_MOBIVEIL
- bool "Mobiveil AXI PCIe controller"
- depends on ARCH_ZYNQMP || COMPILE_TEST
- depends on OF
- depends on PCI_MSI_IRQ_DOMAIN
- help
- Say Y here if you want to enable support for the Mobiveil AXI PCIe
- Soft IP. It has up to 8 outbound and inbound windows
- for address translation and it is a PCIe Gen4 IP.
-
config PCIE_TANGO_SMP8759
bool "Tango SMP8759 PCIe controller (DANGEROUS)"
depends on ARCH_TANGO && PCI_MSI && OF
@@ -289,4 +279,5 @@ config PCI_HYPERV_INTERFACE
have a common interface with the Hyper-V PCI frontend driver.
source "drivers/pci/controller/dwc/Kconfig"
+source "drivers/pci/controller/mobiveil/Kconfig"
endmenu
diff --git a/drivers/pci/controller/Makefile b/drivers/pci/controller/Makefile
index a2a22c9d91af..44414cfd45ea 100644
--- a/drivers/pci/controller/Makefile
+++ b/drivers/pci/controller/Makefile
@@ -27,11 +27,11 @@ obj-$(CONFIG_PCIE_ROCKCHIP) += pcie-rockchip.o
obj-$(CONFIG_PCIE_ROCKCHIP_EP) += pcie-rockchip-ep.o
obj-$(CONFIG_PCIE_ROCKCHIP_HOST) += pcie-rockchip-host.o
obj-$(CONFIG_PCIE_MEDIATEK) += pcie-mediatek.o
-obj-$(CONFIG_PCIE_MOBIVEIL) += pcie-mobiveil.o
obj-$(CONFIG_PCIE_TANGO_SMP8759) += pcie-tango.o
obj-$(CONFIG_VMD) += vmd.o
# pcie-hisi.o quirks are needed even without CONFIG_PCIE_DW
obj-y += dwc/
+obj-y += mobiveil/
# The following drivers are for devices that use the generic ACPI
diff --git a/drivers/pci/controller/mobiveil/Kconfig b/drivers/pci/controller/mobiveil/Kconfig
new file mode 100644
index 000000000000..64343c07bfed
--- /dev/null
+++ b/drivers/pci/controller/mobiveil/Kconfig
@@ -0,0 +1,24 @@
+# SPDX-License-Identifier: GPL-2.0
+
+menu "Mobiveil PCIe Core Support"
+ depends on PCI
+
+config PCIE_MOBIVEIL
+ bool
+
+config PCIE_MOBIVEIL_HOST
+ bool
+ depends on PCI_MSI_IRQ_DOMAIN
+ select PCIE_MOBIVEIL
+
+config PCIE_MOBIVEIL_PLAT
+ bool "Mobiveil AXI PCIe controller"
+ depends on ARCH_ZYNQMP || COMPILE_TEST
+ depends on OF
+ select PCIE_MOBIVEIL_HOST
+ help
+ Say Y here if you want to enable support for the Mobiveil AXI PCIe
+ Soft IP. It has up to 8 outbound and inbound windows
+ for address translation and it is a PCIe Gen4 IP.
+
+endmenu
diff --git a/drivers/pci/controller/mobiveil/Makefile b/drivers/pci/controller/mobiveil/Makefile
new file mode 100644
index 000000000000..9fb6d1c6504d
--- /dev/null
+++ b/drivers/pci/controller/mobiveil/Makefile
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_PCIE_MOBIVEIL) += pcie-mobiveil.o
+obj-$(CONFIG_PCIE_MOBIVEIL_HOST) += pcie-mobiveil-host.o
+obj-$(CONFIG_PCIE_MOBIVEIL_PLAT) += pcie-mobiveil-plat.o
diff --git a/drivers/pci/controller/pcie-mobiveil.c b/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
similarity index 61%
rename from drivers/pci/controller/pcie-mobiveil.c
rename to drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
index 512b27a0536e..2cc424e78d33 100644
--- a/drivers/pci/controller/pcie-mobiveil.c
+++ b/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
@@ -9,7 +9,6 @@
* Recode: Hou Zhiqiang <[email protected]>
*/
-#include <linux/delay.h>
#include <linux/init.h>
#include <linux/interrupt.h>
#include <linux/irq.h>
@@ -26,265 +25,7 @@
#include <linux/platform_device.h>
#include <linux/slab.h>
-#include "../pci.h"
-
-/* register offsets and bit positions */
-
-/*
- * translation tables are grouped into windows, each window registers are
- * grouped into blocks of 4 or 16 registers each
- */
-#define PAB_REG_BLOCK_SIZE 16
-#define PAB_EXT_REG_BLOCK_SIZE 4
-
-#define PAB_REG_ADDR(offset, win) \
- (offset + (win * PAB_REG_BLOCK_SIZE))
-#define PAB_EXT_REG_ADDR(offset, win) \
- (offset + (win * PAB_EXT_REG_BLOCK_SIZE))
-
-#define LTSSM_STATUS 0x0404
-#define LTSSM_STATUS_L0_MASK 0x3f
-#define LTSSM_STATUS_L0 0x2d
-
-#define PAB_CTRL 0x0808
-#define AMBA_PIO_ENABLE_SHIFT 0
-#define PEX_PIO_ENABLE_SHIFT 1
-#define PAGE_SEL_SHIFT 13
-#define PAGE_SEL_MASK 0x3f
-#define PAGE_LO_MASK 0x3ff
-#define PAGE_SEL_OFFSET_SHIFT 10
-
-#define PAB_AXI_PIO_CTRL 0x0840
-#define APIO_EN_MASK 0xf
-
-#define PAB_PEX_PIO_CTRL 0x08c0
-#define PIO_ENABLE_SHIFT 0
-
-#define PAB_INTP_AMBA_MISC_ENB 0x0b0c
-#define PAB_INTP_AMBA_MISC_STAT 0x0b1c
-#define PAB_INTP_INTX_MASK 0x01e0
-#define PAB_INTP_MSI_MASK 0x8
-
-#define PAB_AXI_AMAP_CTRL(win) PAB_REG_ADDR(0x0ba0, win)
-#define WIN_ENABLE_SHIFT 0
-#define WIN_TYPE_SHIFT 1
-#define WIN_TYPE_MASK 0x3
-#define WIN_SIZE_MASK 0xfffffc00
-
-#define PAB_EXT_AXI_AMAP_SIZE(win) PAB_EXT_REG_ADDR(0xbaf0, win)
-
-#define PAB_EXT_AXI_AMAP_AXI_WIN(win) PAB_EXT_REG_ADDR(0x80a0, win)
-#define PAB_AXI_AMAP_AXI_WIN(win) PAB_REG_ADDR(0x0ba4, win)
-#define AXI_WINDOW_ALIGN_MASK 3
-
-#define PAB_AXI_AMAP_PEX_WIN_L(win) PAB_REG_ADDR(0x0ba8, win)
-#define PAB_BUS_SHIFT 24
-#define PAB_DEVICE_SHIFT 19
-#define PAB_FUNCTION_SHIFT 16
-
-#define PAB_AXI_AMAP_PEX_WIN_H(win) PAB_REG_ADDR(0x0bac, win)
-#define PAB_INTP_AXI_PIO_CLASS 0x474
-
-#define PAB_PEX_AMAP_CTRL(win) PAB_REG_ADDR(0x4ba0, win)
-#define AMAP_CTRL_EN_SHIFT 0
-#define AMAP_CTRL_TYPE_SHIFT 1
-#define AMAP_CTRL_TYPE_MASK 3
-
-#define PAB_EXT_PEX_AMAP_SIZEN(win) PAB_EXT_REG_ADDR(0xbef0, win)
-#define PAB_EXT_PEX_AMAP_AXI_WIN(win) PAB_EXT_REG_ADDR(0xb4a0, win)
-#define PAB_PEX_AMAP_AXI_WIN(win) PAB_REG_ADDR(0x4ba4, win)
-#define PAB_PEX_AMAP_PEX_WIN_L(win) PAB_REG_ADDR(0x4ba8, win)
-#define PAB_PEX_AMAP_PEX_WIN_H(win) PAB_REG_ADDR(0x4bac, win)
-
-/* starting offset of INTX bits in status register */
-#define PAB_INTX_START 5
-
-/* supported number of MSI interrupts */
-#define PCI_NUM_MSI 16
-
-/* MSI registers */
-#define MSI_BASE_LO_OFFSET 0x04
-#define MSI_BASE_HI_OFFSET 0x08
-#define MSI_SIZE_OFFSET 0x0c
-#define MSI_ENABLE_OFFSET 0x14
-#define MSI_STATUS_OFFSET 0x18
-#define MSI_DATA_OFFSET 0x20
-#define MSI_ADDR_L_OFFSET 0x24
-#define MSI_ADDR_H_OFFSET 0x28
-
-/* outbound and inbound window definitions */
-#define WIN_NUM_0 0
-#define WIN_NUM_1 1
-#define CFG_WINDOW_TYPE 0
-#define IO_WINDOW_TYPE 1
-#define MEM_WINDOW_TYPE 2
-#define IB_WIN_SIZE ((u64)256 * 1024 * 1024 * 1024)
-#define MAX_PIO_WINDOWS 8
-
-/* Parameters for the waiting for link up routine */
-#define LINK_WAIT_MAX_RETRIES 10
-#define LINK_WAIT_MIN 90000
-#define LINK_WAIT_MAX 100000
-
-#define PAGED_ADDR_BNDRY 0xc00
-#define OFFSET_TO_PAGE_ADDR(off) \
- ((off & PAGE_LO_MASK) | PAGED_ADDR_BNDRY)
-#define OFFSET_TO_PAGE_IDX(off) \
- ((off >> PAGE_SEL_OFFSET_SHIFT) & PAGE_SEL_MASK)
-
-struct mobiveil_msi { /* MSI information */
- struct mutex lock; /* protect bitmap variable */
- struct irq_domain *msi_domain;
- struct irq_domain *dev_domain;
- phys_addr_t msi_pages_phys;
- int num_of_vectors;
- DECLARE_BITMAP(msi_irq_in_use, PCI_NUM_MSI);
-};
-
-struct root_port {
- char root_bus_nr;
- void __iomem *config_axi_slave_base; /* endpoint config base */
- struct resource *ob_io_res;
- int irq;
- raw_spinlock_t intx_mask_lock;
- struct irq_domain *intx_domain;
- struct mobiveil_msi msi;
- struct pci_host_bridge *bridge;
-};
-
-struct mobiveil_pcie {
- struct platform_device *pdev;
- void __iomem *csr_axi_slave_base; /* root port config base */
- void __iomem *apb_csr_base; /* MSI register base */
- phys_addr_t pcie_reg_base; /* Physical PCIe Controller Base */
- int apio_wins;
- int ppio_wins;
- int ob_wins_configured; /* configured outbound windows */
- int ib_wins_configured; /* configured inbound windows */
- struct root_port rp;
-};
-
-/*
- * mobiveil_pcie_sel_page - routine to access paged register
- *
- * Registers whose address greater than PAGED_ADDR_BNDRY (0xc00) are paged,
- * for this scheme to work extracted higher 6 bits of the offset will be
- * written to pg_sel field of PAB_CTRL register and rest of the lower 10
- * bits enabled with PAGED_ADDR_BNDRY are used as offset of the register.
- */
-static void mobiveil_pcie_sel_page(struct mobiveil_pcie *pcie, u8 pg_idx)
-{
- u32 val;
-
- val = readl(pcie->csr_axi_slave_base + PAB_CTRL);
- val &= ~(PAGE_SEL_MASK << PAGE_SEL_SHIFT);
- val |= (pg_idx & PAGE_SEL_MASK) << PAGE_SEL_SHIFT;
-
- writel(val, pcie->csr_axi_slave_base + PAB_CTRL);
-}
-
-static void *mobiveil_pcie_comp_addr(struct mobiveil_pcie *pcie, u32 off)
-{
- if (off < PAGED_ADDR_BNDRY) {
- /* For directly accessed registers, clear the pg_sel field */
- mobiveil_pcie_sel_page(pcie, 0);
- return pcie->csr_axi_slave_base + off;
- }
-
- mobiveil_pcie_sel_page(pcie, OFFSET_TO_PAGE_IDX(off));
- return pcie->csr_axi_slave_base + OFFSET_TO_PAGE_ADDR(off);
-}
-
-static int mobiveil_pcie_read(void __iomem *addr, int size, u32 *val)
-{
- if ((uintptr_t)addr & (size - 1)) {
- *val = 0;
- return PCIBIOS_BAD_REGISTER_NUMBER;
- }
-
- switch (size) {
- case 4:
- *val = readl(addr);
- break;
- case 2:
- *val = readw(addr);
- break;
- case 1:
- *val = readb(addr);
- break;
- default:
- *val = 0;
- return PCIBIOS_BAD_REGISTER_NUMBER;
- }
-
- return PCIBIOS_SUCCESSFUL;
-}
-
-static int mobiveil_pcie_write(void __iomem *addr, int size, u32 val)
-{
- if ((uintptr_t)addr & (size - 1))
- return PCIBIOS_BAD_REGISTER_NUMBER;
-
- switch (size) {
- case 4:
- writel(val, addr);
- break;
- case 2:
- writew(val, addr);
- break;
- case 1:
- writeb(val, addr);
- break;
- default:
- return PCIBIOS_BAD_REGISTER_NUMBER;
- }
-
- return PCIBIOS_SUCCESSFUL;
-}
-
-static u32 mobiveil_csr_read(struct mobiveil_pcie *pcie, u32 off, size_t size)
-{
- void *addr;
- u32 val;
- int ret;
-
- addr = mobiveil_pcie_comp_addr(pcie, off);
-
- ret = mobiveil_pcie_read(addr, size, &val);
- if (ret)
- dev_err(&pcie->pdev->dev, "read CSR address failed\n");
-
- return val;
-}
-
-static void mobiveil_csr_write(struct mobiveil_pcie *pcie, u32 val, u32 off,
- size_t size)
-{
- void *addr;
- int ret;
-
- addr = mobiveil_pcie_comp_addr(pcie, off);
-
- ret = mobiveil_pcie_write(addr, size, val);
- if (ret)
- dev_err(&pcie->pdev->dev, "write CSR address failed\n");
-}
-
-static u32 mobiveil_csr_readl(struct mobiveil_pcie *pcie, u32 off)
-{
- return mobiveil_csr_read(pcie, off, 0x4);
-}
-
-static void mobiveil_csr_writel(struct mobiveil_pcie *pcie, u32 val, u32 off)
-{
- mobiveil_csr_write(pcie, val, off, 0x4);
-}
-
-static bool mobiveil_pcie_link_up(struct mobiveil_pcie *pcie)
-{
- return (mobiveil_csr_readl(pcie, LTSSM_STATUS) &
- LTSSM_STATUS_L0_MASK) == LTSSM_STATUS_L0;
-}
+#include "pcie-mobiveil.h"
static bool mobiveil_pcie_valid_device(struct pci_bus *bus, unsigned int devfn)
{
@@ -464,103 +205,6 @@ static int mobiveil_pcie_parse_dt(struct mobiveil_pcie *pcie)
return 0;
}
-static void program_ib_windows(struct mobiveil_pcie *pcie, int win_num,
- u64 cpu_addr, u64 pci_addr, u32 type, u64 size)
-{
- u32 value;
- u64 size64 = ~(size - 1);
-
- if (win_num >= pcie->ppio_wins) {
- dev_err(&pcie->pdev->dev,
- "ERROR: max inbound windows reached !\n");
- return;
- }
-
- value = mobiveil_csr_readl(pcie, PAB_PEX_AMAP_CTRL(win_num));
- value &= ~(AMAP_CTRL_TYPE_MASK << AMAP_CTRL_TYPE_SHIFT | WIN_SIZE_MASK);
- value |= type << AMAP_CTRL_TYPE_SHIFT | 1 << AMAP_CTRL_EN_SHIFT |
- (lower_32_bits(size64) & WIN_SIZE_MASK);
- mobiveil_csr_writel(pcie, value, PAB_PEX_AMAP_CTRL(win_num));
-
- mobiveil_csr_writel(pcie, upper_32_bits(size64),
- PAB_EXT_PEX_AMAP_SIZEN(win_num));
-
- mobiveil_csr_writel(pcie, lower_32_bits(cpu_addr),
- PAB_PEX_AMAP_AXI_WIN(win_num));
- mobiveil_csr_writel(pcie, upper_32_bits(cpu_addr),
- PAB_EXT_PEX_AMAP_AXI_WIN(win_num));
-
- mobiveil_csr_writel(pcie, lower_32_bits(pci_addr),
- PAB_PEX_AMAP_PEX_WIN_L(win_num));
- mobiveil_csr_writel(pcie, upper_32_bits(pci_addr),
- PAB_PEX_AMAP_PEX_WIN_H(win_num));
-
- pcie->ib_wins_configured++;
-}
-
-/*
- * routine to program the outbound windows
- */
-static void program_ob_windows(struct mobiveil_pcie *pcie, int win_num,
- u64 cpu_addr, u64 pci_addr, u32 type, u64 size)
-{
- u32 value;
- u64 size64 = ~(size - 1);
-
- if (win_num >= pcie->apio_wins) {
- dev_err(&pcie->pdev->dev,
- "ERROR: max outbound windows reached !\n");
- return;
- }
-
- /*
- * program Enable Bit to 1, Type Bit to (00) base 2, AXI Window Size Bit
- * to 4 KB in PAB_AXI_AMAP_CTRL register
- */
- value = mobiveil_csr_readl(pcie, PAB_AXI_AMAP_CTRL(win_num));
- value &= ~(WIN_TYPE_MASK << WIN_TYPE_SHIFT | WIN_SIZE_MASK);
- value |= 1 << WIN_ENABLE_SHIFT | type << WIN_TYPE_SHIFT |
- (lower_32_bits(size64) & WIN_SIZE_MASK);
- mobiveil_csr_writel(pcie, value, PAB_AXI_AMAP_CTRL(win_num));
-
- mobiveil_csr_writel(pcie, upper_32_bits(size64),
- PAB_EXT_AXI_AMAP_SIZE(win_num));
-
- /*
- * program AXI window base with appropriate value in
- * PAB_AXI_AMAP_AXI_WIN0 register
- */
- mobiveil_csr_writel(pcie,
- lower_32_bits(cpu_addr) & (~AXI_WINDOW_ALIGN_MASK),
- PAB_AXI_AMAP_AXI_WIN(win_num));
- mobiveil_csr_writel(pcie, upper_32_bits(cpu_addr),
- PAB_EXT_AXI_AMAP_AXI_WIN(win_num));
-
- mobiveil_csr_writel(pcie, lower_32_bits(pci_addr),
- PAB_AXI_AMAP_PEX_WIN_L(win_num));
- mobiveil_csr_writel(pcie, upper_32_bits(pci_addr),
- PAB_AXI_AMAP_PEX_WIN_H(win_num));
-
- pcie->ob_wins_configured++;
-}
-
-static int mobiveil_bringup_link(struct mobiveil_pcie *pcie)
-{
- int retries;
-
- /* check if the link is up or not */
- for (retries = 0; retries < LINK_WAIT_MAX_RETRIES; retries++) {
- if (mobiveil_pcie_link_up(pcie))
- return 0;
-
- usleep_range(LINK_WAIT_MIN, LINK_WAIT_MAX);
- }
-
- dev_err(&pcie->pdev->dev, "link never came up\n");
-
- return -ETIMEDOUT;
-}
-
static void mobiveil_pcie_enable_msi(struct mobiveil_pcie *pcie)
{
phys_addr_t msg_addr = pcie->pcie_reg_base;
@@ -962,43 +606,3 @@ int mobiveil_pcie_host_probe(struct mobiveil_pcie *pcie)
return 0;
}
-
-static int mobiveil_pcie_probe(struct platform_device *pdev)
-{
- struct mobiveil_pcie *pcie;
- struct pci_host_bridge *bridge;
- struct device *dev = &pdev->dev;
-
- bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
- if (!bridge)
- return -ENOMEM;
-
- pcie = pci_host_bridge_priv(bridge);
- pcie->rp.bridge = bridge;
-
- pcie->pdev = pdev;
-
- return mobiveil_pcie_host_probe(pcie);
-}
-
-static const struct of_device_id mobiveil_pcie_of_match[] = {
- {.compatible = "mbvl,gpex40-pcie",},
- {},
-};
-
-MODULE_DEVICE_TABLE(of, mobiveil_pcie_of_match);
-
-static struct platform_driver mobiveil_pcie_driver = {
- .probe = mobiveil_pcie_probe,
- .driver = {
- .name = "mobiveil-pcie",
- .of_match_table = mobiveil_pcie_of_match,
- .suppress_bind_attrs = true,
- },
-};
-
-builtin_platform_driver(mobiveil_pcie_driver);
-
-MODULE_LICENSE("GPL v2");
-MODULE_DESCRIPTION("Mobiveil PCIe host controller driver");
-MODULE_AUTHOR("Subrahmanya Lingappa <[email protected]>");
diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil-plat.c b/drivers/pci/controller/mobiveil/pcie-mobiveil-plat.c
new file mode 100644
index 000000000000..64c85f852869
--- /dev/null
+++ b/drivers/pci/controller/mobiveil/pcie-mobiveil-plat.c
@@ -0,0 +1,60 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * PCIe host controller driver for Mobiveil PCIe Host controller
+ *
+ * Copyright (c) 2018 Mobiveil Inc.
+ * Copyright 2019 NXP
+ *
+ * Author: Subrahmanya Lingappa <[email protected]>
+ * Recode: Hou Zhiqiang <[email protected]>
+ */
+
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/of_pci.h>
+#include <linux/pci.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+
+#include "pcie-mobiveil.h"
+
+static int mobiveil_pcie_probe(struct platform_device *pdev)
+{
+ struct mobiveil_pcie *pcie;
+ struct pci_host_bridge *bridge;
+ struct device *dev = &pdev->dev;
+
+ bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
+ if (!bridge)
+ return -ENOMEM;
+
+ pcie = pci_host_bridge_priv(bridge);
+ pcie->rp.bridge = bridge;
+
+ pcie->pdev = pdev;
+
+ return mobiveil_pcie_host_probe(pcie);
+}
+
+static const struct of_device_id mobiveil_pcie_of_match[] = {
+ {.compatible = "mbvl,gpex40-pcie",},
+ {},
+};
+
+MODULE_DEVICE_TABLE(of, mobiveil_pcie_of_match);
+
+static struct platform_driver mobiveil_pcie_driver = {
+ .probe = mobiveil_pcie_probe,
+ .driver = {
+ .name = "mobiveil-pcie",
+ .of_match_table = mobiveil_pcie_of_match,
+ .suppress_bind_attrs = true,
+ },
+};
+
+builtin_platform_driver(mobiveil_pcie_driver);
+
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("Mobiveil PCIe host controller driver");
+MODULE_AUTHOR("Subrahmanya Lingappa <[email protected]>");
diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.c b/drivers/pci/controller/mobiveil/pcie-mobiveil.c
new file mode 100644
index 000000000000..2773f823c9ea
--- /dev/null
+++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.c
@@ -0,0 +1,227 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * PCIe host controller driver for Mobiveil PCIe Host controller
+ *
+ * Copyright (c) 2018 Mobiveil Inc.
+ * Copyright 2019 NXP
+ *
+ * Author: Subrahmanya Lingappa <[email protected]>
+ * Recode: Hou Zhiqiang <[email protected]>
+ */
+
+#include <linux/delay.h>
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/pci.h>
+#include <linux/platform_device.h>
+
+#include "pcie-mobiveil.h"
+
+/*
+ * mobiveil_pcie_sel_page - routine to access paged register
+ *
+ * Registers whose address greater than PAGED_ADDR_BNDRY (0xc00) are paged,
+ * for this scheme to work extracted higher 6 bits of the offset will be
+ * written to pg_sel field of PAB_CTRL register and rest of the lower 10
+ * bits enabled with PAGED_ADDR_BNDRY are used as offset of the register.
+ */
+static void mobiveil_pcie_sel_page(struct mobiveil_pcie *pcie, u8 pg_idx)
+{
+ u32 val;
+
+ val = readl(pcie->csr_axi_slave_base + PAB_CTRL);
+ val &= ~(PAGE_SEL_MASK << PAGE_SEL_SHIFT);
+ val |= (pg_idx & PAGE_SEL_MASK) << PAGE_SEL_SHIFT;
+
+ writel(val, pcie->csr_axi_slave_base + PAB_CTRL);
+}
+
+static void *mobiveil_pcie_comp_addr(struct mobiveil_pcie *pcie, u32 off)
+{
+ if (off < PAGED_ADDR_BNDRY) {
+ /* For directly accessed registers, clear the pg_sel field */
+ mobiveil_pcie_sel_page(pcie, 0);
+ return pcie->csr_axi_slave_base + off;
+ }
+
+ mobiveil_pcie_sel_page(pcie, OFFSET_TO_PAGE_IDX(off));
+ return pcie->csr_axi_slave_base + OFFSET_TO_PAGE_ADDR(off);
+}
+
+static int mobiveil_pcie_read(void __iomem *addr, int size, u32 *val)
+{
+ if ((uintptr_t)addr & (size - 1)) {
+ *val = 0;
+ return PCIBIOS_BAD_REGISTER_NUMBER;
+ }
+
+ switch (size) {
+ case 4:
+ *val = readl(addr);
+ break;
+ case 2:
+ *val = readw(addr);
+ break;
+ case 1:
+ *val = readb(addr);
+ break;
+ default:
+ *val = 0;
+ return PCIBIOS_BAD_REGISTER_NUMBER;
+ }
+
+ return PCIBIOS_SUCCESSFUL;
+}
+
+static int mobiveil_pcie_write(void __iomem *addr, int size, u32 val)
+{
+ if ((uintptr_t)addr & (size - 1))
+ return PCIBIOS_BAD_REGISTER_NUMBER;
+
+ switch (size) {
+ case 4:
+ writel(val, addr);
+ break;
+ case 2:
+ writew(val, addr);
+ break;
+ case 1:
+ writeb(val, addr);
+ break;
+ default:
+ return PCIBIOS_BAD_REGISTER_NUMBER;
+ }
+
+ return PCIBIOS_SUCCESSFUL;
+}
+
+u32 mobiveil_csr_read(struct mobiveil_pcie *pcie, u32 off, size_t size)
+{
+ void *addr;
+ u32 val;
+ int ret;
+
+ addr = mobiveil_pcie_comp_addr(pcie, off);
+
+ ret = mobiveil_pcie_read(addr, size, &val);
+ if (ret)
+ dev_err(&pcie->pdev->dev, "read CSR address failed\n");
+
+ return val;
+}
+
+void mobiveil_csr_write(struct mobiveil_pcie *pcie, u32 val, u32 off,
+ size_t size)
+{
+ void *addr;
+ int ret;
+
+ addr = mobiveil_pcie_comp_addr(pcie, off);
+
+ ret = mobiveil_pcie_write(addr, size, val);
+ if (ret)
+ dev_err(&pcie->pdev->dev, "write CSR address failed\n");
+}
+
+bool mobiveil_pcie_link_up(struct mobiveil_pcie *pcie)
+{
+ return (mobiveil_csr_readl(pcie, LTSSM_STATUS) &
+ LTSSM_STATUS_L0_MASK) == LTSSM_STATUS_L0;
+}
+
+void program_ib_windows(struct mobiveil_pcie *pcie, int win_num,
+ u64 cpu_addr, u64 pci_addr, u32 type, u64 size)
+{
+ u32 value;
+ u64 size64 = ~(size - 1);
+
+ if (win_num >= pcie->ppio_wins) {
+ dev_err(&pcie->pdev->dev,
+ "ERROR: max inbound windows reached !\n");
+ return;
+ }
+
+ value = mobiveil_csr_readl(pcie, PAB_PEX_AMAP_CTRL(win_num));
+ value &= ~(AMAP_CTRL_TYPE_MASK << AMAP_CTRL_TYPE_SHIFT | WIN_SIZE_MASK);
+ value |= type << AMAP_CTRL_TYPE_SHIFT | 1 << AMAP_CTRL_EN_SHIFT |
+ (lower_32_bits(size64) & WIN_SIZE_MASK);
+ mobiveil_csr_writel(pcie, value, PAB_PEX_AMAP_CTRL(win_num));
+
+ mobiveil_csr_writel(pcie, upper_32_bits(size64),
+ PAB_EXT_PEX_AMAP_SIZEN(win_num));
+
+ mobiveil_csr_writel(pcie, lower_32_bits(cpu_addr),
+ PAB_PEX_AMAP_AXI_WIN(win_num));
+ mobiveil_csr_writel(pcie, upper_32_bits(cpu_addr),
+ PAB_EXT_PEX_AMAP_AXI_WIN(win_num));
+
+ mobiveil_csr_writel(pcie, lower_32_bits(pci_addr),
+ PAB_PEX_AMAP_PEX_WIN_L(win_num));
+ mobiveil_csr_writel(pcie, upper_32_bits(pci_addr),
+ PAB_PEX_AMAP_PEX_WIN_H(win_num));
+
+ pcie->ib_wins_configured++;
+}
+
+/*
+ * routine to program the outbound windows
+ */
+void program_ob_windows(struct mobiveil_pcie *pcie, int win_num,
+ u64 cpu_addr, u64 pci_addr, u32 type, u64 size)
+{
+ u32 value;
+ u64 size64 = ~(size - 1);
+
+ if (win_num >= pcie->apio_wins) {
+ dev_err(&pcie->pdev->dev,
+ "ERROR: max outbound windows reached !\n");
+ return;
+ }
+
+ /*
+ * program Enable Bit to 1, Type Bit to (00) base 2, AXI Window Size Bit
+ * to 4 KB in PAB_AXI_AMAP_CTRL register
+ */
+ value = mobiveil_csr_readl(pcie, PAB_AXI_AMAP_CTRL(win_num));
+ value &= ~(WIN_TYPE_MASK << WIN_TYPE_SHIFT | WIN_SIZE_MASK);
+ value |= 1 << WIN_ENABLE_SHIFT | type << WIN_TYPE_SHIFT |
+ (lower_32_bits(size64) & WIN_SIZE_MASK);
+ mobiveil_csr_writel(pcie, value, PAB_AXI_AMAP_CTRL(win_num));
+
+ mobiveil_csr_writel(pcie, upper_32_bits(size64),
+ PAB_EXT_AXI_AMAP_SIZE(win_num));
+
+ /*
+ * program AXI window base with appropriate value in
+ * PAB_AXI_AMAP_AXI_WIN0 register
+ */
+ mobiveil_csr_writel(pcie,
+ lower_32_bits(cpu_addr) & (~AXI_WINDOW_ALIGN_MASK),
+ PAB_AXI_AMAP_AXI_WIN(win_num));
+ mobiveil_csr_writel(pcie, upper_32_bits(cpu_addr),
+ PAB_EXT_AXI_AMAP_AXI_WIN(win_num));
+
+ mobiveil_csr_writel(pcie, lower_32_bits(pci_addr),
+ PAB_AXI_AMAP_PEX_WIN_L(win_num));
+ mobiveil_csr_writel(pcie, upper_32_bits(pci_addr),
+ PAB_AXI_AMAP_PEX_WIN_H(win_num));
+
+ pcie->ob_wins_configured++;
+}
+
+int mobiveil_bringup_link(struct mobiveil_pcie *pcie)
+{
+ int retries;
+
+ /* check if the link is up or not */
+ for (retries = 0; retries < LINK_WAIT_MAX_RETRIES; retries++) {
+ if (mobiveil_pcie_link_up(pcie))
+ return 0;
+
+ usleep_range(LINK_WAIT_MIN, LINK_WAIT_MAX);
+ }
+
+ dev_err(&pcie->pdev->dev, "link never came up\n");
+
+ return -ETIMEDOUT;
+}
diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.h b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
new file mode 100644
index 000000000000..e3148078e9dd
--- /dev/null
+++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
@@ -0,0 +1,178 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * PCIe host controller driver for Mobiveil PCIe Host controller
+ *
+ * Copyright (c) 2018 Mobiveil Inc.
+ * Copyright 2019 NXP
+ *
+ * Author: Subrahmanya Lingappa <[email protected]>
+ * Recode: Hou Zhiqiang <[email protected]>
+ */
+
+#ifndef _PCIE_MOBIVEIL_H
+#define _PCIE_MOBIVEIL_H
+
+#include <linux/pci.h>
+#include <linux/irq.h>
+#include <linux/msi.h>
+#include "../../pci.h"
+
+/* register offsets and bit positions */
+
+/*
+ * translation tables are grouped into windows, each window registers are
+ * grouped into blocks of 4 or 16 registers each
+ */
+#define PAB_REG_BLOCK_SIZE 16
+#define PAB_EXT_REG_BLOCK_SIZE 4
+
+#define PAB_REG_ADDR(offset, win) \
+ (offset + (win * PAB_REG_BLOCK_SIZE))
+#define PAB_EXT_REG_ADDR(offset, win) \
+ (offset + (win * PAB_EXT_REG_BLOCK_SIZE))
+
+#define LTSSM_STATUS 0x0404
+#define LTSSM_STATUS_L0_MASK 0x3f
+#define LTSSM_STATUS_L0 0x2d
+
+#define PAB_CTRL 0x0808
+#define AMBA_PIO_ENABLE_SHIFT 0
+#define PEX_PIO_ENABLE_SHIFT 1
+#define PAGE_SEL_SHIFT 13
+#define PAGE_SEL_MASK 0x3f
+#define PAGE_LO_MASK 0x3ff
+#define PAGE_SEL_OFFSET_SHIFT 10
+
+#define PAB_AXI_PIO_CTRL 0x0840
+#define APIO_EN_MASK 0xf
+
+#define PAB_PEX_PIO_CTRL 0x08c0
+#define PIO_ENABLE_SHIFT 0
+
+#define PAB_INTP_AMBA_MISC_ENB 0x0b0c
+#define PAB_INTP_AMBA_MISC_STAT 0x0b1c
+#define PAB_INTP_INTX_MASK 0x01e0
+#define PAB_INTP_MSI_MASK 0x8
+
+#define PAB_AXI_AMAP_CTRL(win) PAB_REG_ADDR(0x0ba0, win)
+#define WIN_ENABLE_SHIFT 0
+#define WIN_TYPE_SHIFT 1
+#define WIN_TYPE_MASK 0x3
+#define WIN_SIZE_MASK 0xfffffc00
+
+#define PAB_EXT_AXI_AMAP_SIZE(win) PAB_EXT_REG_ADDR(0xbaf0, win)
+
+#define PAB_EXT_AXI_AMAP_AXI_WIN(win) PAB_EXT_REG_ADDR(0x80a0, win)
+#define PAB_AXI_AMAP_AXI_WIN(win) PAB_REG_ADDR(0x0ba4, win)
+#define AXI_WINDOW_ALIGN_MASK 3
+
+#define PAB_AXI_AMAP_PEX_WIN_L(win) PAB_REG_ADDR(0x0ba8, win)
+#define PAB_BUS_SHIFT 24
+#define PAB_DEVICE_SHIFT 19
+#define PAB_FUNCTION_SHIFT 16
+
+#define PAB_AXI_AMAP_PEX_WIN_H(win) PAB_REG_ADDR(0x0bac, win)
+#define PAB_INTP_AXI_PIO_CLASS 0x474
+
+#define PAB_PEX_AMAP_CTRL(win) PAB_REG_ADDR(0x4ba0, win)
+#define AMAP_CTRL_EN_SHIFT 0
+#define AMAP_CTRL_TYPE_SHIFT 1
+#define AMAP_CTRL_TYPE_MASK 3
+
+#define PAB_EXT_PEX_AMAP_SIZEN(win) PAB_EXT_REG_ADDR(0xbef0, win)
+#define PAB_EXT_PEX_AMAP_AXI_WIN(win) PAB_EXT_REG_ADDR(0xb4a0, win)
+#define PAB_PEX_AMAP_AXI_WIN(win) PAB_REG_ADDR(0x4ba4, win)
+#define PAB_PEX_AMAP_PEX_WIN_L(win) PAB_REG_ADDR(0x4ba8, win)
+#define PAB_PEX_AMAP_PEX_WIN_H(win) PAB_REG_ADDR(0x4bac, win)
+
+/* starting offset of INTX bits in status register */
+#define PAB_INTX_START 5
+
+/* supported number of MSI interrupts */
+#define PCI_NUM_MSI 16
+
+/* MSI registers */
+#define MSI_BASE_LO_OFFSET 0x04
+#define MSI_BASE_HI_OFFSET 0x08
+#define MSI_SIZE_OFFSET 0x0c
+#define MSI_ENABLE_OFFSET 0x14
+#define MSI_STATUS_OFFSET 0x18
+#define MSI_DATA_OFFSET 0x20
+#define MSI_ADDR_L_OFFSET 0x24
+#define MSI_ADDR_H_OFFSET 0x28
+
+/* outbound and inbound window definitions */
+#define WIN_NUM_0 0
+#define WIN_NUM_1 1
+#define CFG_WINDOW_TYPE 0
+#define IO_WINDOW_TYPE 1
+#define MEM_WINDOW_TYPE 2
+#define IB_WIN_SIZE ((u64)256 * 1024 * 1024 * 1024)
+#define MAX_PIO_WINDOWS 8
+
+/* Parameters for the waiting for link up routine */
+#define LINK_WAIT_MAX_RETRIES 10
+#define LINK_WAIT_MIN 90000
+#define LINK_WAIT_MAX 100000
+
+#define PAGED_ADDR_BNDRY 0xc00
+#define OFFSET_TO_PAGE_ADDR(off) \
+ ((off & PAGE_LO_MASK) | PAGED_ADDR_BNDRY)
+#define OFFSET_TO_PAGE_IDX(off) \
+ ((off >> PAGE_SEL_OFFSET_SHIFT) & PAGE_SEL_MASK)
+
+struct mobiveil_msi { /* MSI information */
+ struct mutex lock; /* protect bitmap variable */
+ struct irq_domain *msi_domain;
+ struct irq_domain *dev_domain;
+ phys_addr_t msi_pages_phys;
+ int num_of_vectors;
+ DECLARE_BITMAP(msi_irq_in_use, PCI_NUM_MSI);
+};
+
+struct root_port {
+ char root_bus_nr;
+ void __iomem *config_axi_slave_base; /* endpoint config base */
+ struct resource *ob_io_res;
+ int irq;
+ raw_spinlock_t intx_mask_lock;
+ struct irq_domain *intx_domain;
+ struct mobiveil_msi msi;
+ struct pci_host_bridge *bridge;
+};
+
+struct mobiveil_pcie {
+ struct platform_device *pdev;
+ void __iomem *csr_axi_slave_base; /* root port config base */
+ void __iomem *apb_csr_base; /* MSI register base */
+ phys_addr_t pcie_reg_base; /* Physical PCIe Controller Base */
+ int apio_wins;
+ int ppio_wins;
+ int ob_wins_configured; /* configured outbound windows */
+ int ib_wins_configured; /* configured inbound windows */
+ struct root_port rp;
+};
+
+int mobiveil_pcie_host_probe(struct mobiveil_pcie *pcie);
+bool mobiveil_pcie_link_up(struct mobiveil_pcie *pcie);
+int mobiveil_bringup_link(struct mobiveil_pcie *pcie);
+void program_ob_windows(struct mobiveil_pcie *pcie, int win_num, u64 cpu_addr,
+ u64 pci_addr, u32 type, u64 size);
+void program_ib_windows(struct mobiveil_pcie *pcie, int win_num, u64 cpu_addr,
+ u64 pci_addr, u32 type, u64 size);
+u32 mobiveil_csr_read(struct mobiveil_pcie *pcie, u32 off, size_t size);
+void mobiveil_csr_write(struct mobiveil_pcie *pcie, u32 val, u32 off,
+ size_t size);
+
+static inline u32 mobiveil_csr_readl(struct mobiveil_pcie *pcie, u32 off)
+{
+ return mobiveil_csr_read(pcie, off, 0x4);
+}
+
+static inline void mobiveil_csr_writel(struct mobiveil_pcie *pcie, u32 val,
+ u32 off)
+{
+ mobiveil_csr_write(pcie, val, off, 0x4);
+}
+
+#endif /* _PCIE_MOBIVEIL_H */
--
2.17.1
From: Hou Zhiqiang <[email protected]>
There are some 8-bit and 16-bit registers in PCIe configuration
space, so add these accessors accordingly.
Signed-off-by: Hou Zhiqiang <[email protected]>
Reviewed-by: Minghuan Lian <[email protected]>
Reviewed-by: Subrahmanya Lingappa <[email protected]>
---
V9:
- No change
.../pci/controller/mobiveil/pcie-mobiveil.h | 23 +++++++++++++++++++
1 file changed, 23 insertions(+)
diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.h b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
index 37116c2a19fe..750a7fd95bc1 100644
--- a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
+++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
@@ -182,10 +182,33 @@ static inline u32 mobiveil_csr_readl(struct mobiveil_pcie *pcie, u32 off)
return mobiveil_csr_read(pcie, off, 0x4);
}
+static inline u32 mobiveil_csr_readw(struct mobiveil_pcie *pcie, u32 off)
+{
+ return mobiveil_csr_read(pcie, off, 0x2);
+}
+
+static inline u32 mobiveil_csr_readb(struct mobiveil_pcie *pcie, u32 off)
+{
+ return mobiveil_csr_read(pcie, off, 0x1);
+}
+
+
static inline void mobiveil_csr_writel(struct mobiveil_pcie *pcie, u32 val,
u32 off)
{
mobiveil_csr_write(pcie, val, off, 0x4);
}
+static inline void mobiveil_csr_writew(struct mobiveil_pcie *pcie, u32 val,
+ u32 off)
+{
+ mobiveil_csr_write(pcie, val, off, 0x2);
+}
+
+static inline void mobiveil_csr_writeb(struct mobiveil_pcie *pcie, u32 val,
+ u32 off)
+{
+ mobiveil_csr_write(pcie, val, off, 0x1);
+}
+
#endif /* _PCIE_MOBIVEIL_H */
--
2.17.1
From: Hou Zhiqiang <[email protected]>
The platforms, in which the Mobiveil GPEX is integrated,
may have their specific mechanism to check link up status.
This patch is to enable these platforms to implement theirs.
Signed-off-by: Hou Zhiqiang <[email protected]>
---
V9:
- New patch splited from the #1 of V8 patches to make it easy to review.
drivers/pci/controller/mobiveil/pcie-mobiveil.c | 3 +++
drivers/pci/controller/mobiveil/pcie-mobiveil.h | 5 +++++
2 files changed, 8 insertions(+)
diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.c b/drivers/pci/controller/mobiveil/pcie-mobiveil.c
index 2773f823c9ea..b9ed2d95641c 100644
--- a/drivers/pci/controller/mobiveil/pcie-mobiveil.c
+++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.c
@@ -125,6 +125,9 @@ void mobiveil_csr_write(struct mobiveil_pcie *pcie, u32 val, u32 off,
bool mobiveil_pcie_link_up(struct mobiveil_pcie *pcie)
{
+ if (pcie->ops->link_up)
+ return pcie->ops->link_up(pcie);
+
return (mobiveil_csr_readl(pcie, LTSSM_STATUS) &
LTSSM_STATUS_L0_MASK) == LTSSM_STATUS_L0;
}
diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.h b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
index 18d85806a7fc..95d2e7c809b8 100644
--- a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
+++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
@@ -148,6 +148,10 @@ struct root_port {
struct pci_host_bridge *bridge;
};
+struct mobiveil_pab_ops {
+ int (*link_up)(struct mobiveil_pcie *pcie);
+};
+
struct mobiveil_pcie {
struct platform_device *pdev;
void __iomem *csr_axi_slave_base; /* root port config base */
@@ -157,6 +161,7 @@ struct mobiveil_pcie {
int ppio_wins;
int ob_wins_configured; /* configured outbound windows */
int ib_wins_configured; /* configured inbound windows */
+ const struct mobiveil_pab_ops *ops;
struct root_port rp;
};
--
2.17.1
On Wed, Nov 20, 2019 at 03:45:17AM +0000, Z.q. Hou wrote:
> From: Hou Zhiqiang <[email protected]>
>
> This patch set is to recode the Mobiveil driver and add
> PCIe support for NXP Layerscape series SoCs integrated
> Mobiveil's PCIe Gen4 controller.
How many PCIe cards have been tested to work/don't work with this?
I need:
PCI: mobiveil: ls_pcie_g4: fix SError when accessing config space
PCI: mobiveil: ls_pcie_g4: add Workaround for A-011451
PCI: mobiveil: ls_pcie_g4: add Workaround for A-011577
to successfully boot with a Mellanox card plugged in with a previous
revision of these patches.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up
Hi Russell,
> -----Original Message-----
> From: Russell King - ARM Linux admin <[email protected]>
> Sent: 2019??11??20?? 17:57
> To: Z.q. Hou <[email protected]>
> Cc: [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; Leo Li
> <[email protected]>; [email protected];
> [email protected]; [email protected];
> [email protected]; M.h. Lian <[email protected]>; Xiaowei
> Bao <[email protected]>; Mingkai Hu <[email protected]>
> Subject: Re: [PATCHv9 00/12] PCI: Recode Mobiveil driver and add PCIe Gen4
> driver for NXP Layerscape SoCs
>
> On Wed, Nov 20, 2019 at 03:45:17AM +0000, Z.q. Hou wrote:
> > From: Hou Zhiqiang <[email protected]>
> >
> > This patch set is to recode the Mobiveil driver and add PCIe support
> > for NXP Layerscape series SoCs integrated Mobiveil's PCIe Gen4
> > controller.
>
> How many PCIe cards have been tested to work/don't work with this?
>
> I need:
>
> PCI: mobiveil: ls_pcie_g4: fix SError when accessing config space
> PCI: mobiveil: ls_pcie_g4: add Workaround for A-011451
> PCI: mobiveil: ls_pcie_g4: add Workaround for A-011577
>
> to successfully boot with a Mellanox card plugged in with a previous revision
> of these patches.
>
Yes, we need to apply these NXP internal maintained workarounds on top of
this series. I only tested Intel e1000e NIC with this patch set + these 3
workarounds.
Thanks,
Zhiqiang
> --
> RMK's Patch system:
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww
> .armlinux.org.uk%2Fdeveloper%2Fpatches%2F&data=02%7C01%7Czhiq
> iang.hou%40nxp.com%7C69f6fb1f4fd44f3fca3808d76da01440%7C686ea1d
> 3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C637098406606503361&sd
> ata=wOLWzKfZZoiP%2FZpTOw5zr4enpuNImz45RM8Hy80aUdI%3D&res
> erved=0
> FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps
> up According to speedtest.net: 11.9Mbps down 500kbps up
Hi!
On Tue, Nov 19, 2019 at 7:45 PM Z.q. Hou <[email protected]> wrote:
>
> From: Hou Zhiqiang <[email protected]>
>
> This patch set is to recode the Mobiveil driver and add
> PCIe support for NXP Layerscape series SoCs integrated
> Mobiveil's PCIe Gen4 controller.
Can we get a respin for this on top of the 5.5 merge window material?
Given that it's a bunch of refactorings, many of them don't apply on
top of the material that was merged.
I'd love to see these go in sooner rather than later so I can start
getting -next running on ls2160a here.
-Olof
On Tue, Dec 17, 2019 at 02:50:15AM +0000, Z.q. Hou wrote:
> Hi Lorenzo,
>
> The v9 patches have addressed the comments from Andrew, and it has
> been dried about 1 month, can you help to apply them?
We shall have a look beginning of next week, sorry for the delay
in getting back to you.
Lorenzo
> Thanks,
> Zhiqiang
>
> > -----Original Message-----
> > From: Olof Johansson <[email protected]>
> > Sent: 2019年12月14日 2:37
> > To: Z.q. Hou <[email protected]>; [email protected]
> > Cc: [email protected]; [email protected];
> > [email protected]; [email protected];
> > [email protected]; [email protected]; [email protected];
> > [email protected]; [email protected];
> > [email protected]; Leo Li <[email protected]>;
> > [email protected]; [email protected];
> > [email protected]; [email protected]; Mingkai Hu
> > <[email protected]>; M.h. Lian <[email protected]>; Xiaowei Bao
> > <[email protected]>
> > Subject: Re: [PATCHv9 00/12] PCI: Recode Mobiveil driver and add PCIe Gen4
> > driver for NXP Layerscape SoCs
> >
> > Hi!
> >
> > On Tue, Nov 19, 2019 at 7:45 PM Z.q. Hou <[email protected]> wrote:
> > >
> > > From: Hou Zhiqiang <[email protected]>
> > >
> > > This patch set is to recode the Mobiveil driver and add PCIe support
> > > for NXP Layerscape series SoCs integrated Mobiveil's PCIe Gen4
> > > controller.
> >
> > Can we get a respin for this on top of the 5.5 merge window material?
> > Given that it's a bunch of refactorings, many of them don't apply on top of
> > the material that was merged.
> >
> > I'd love to see these go in sooner rather than later so I can start getting -next
> > running on ls2160a here.
> >
> >
> > -Olof
On Fri, Jan 10, 2020 at 7:33 AM Lorenzo Pieralisi
<[email protected]> wrote:
>
> On Tue, Dec 17, 2019 at 02:50:15AM +0000, Z.q. Hou wrote:
> > Hi Lorenzo,
> >
> > The v9 patches have addressed the comments from Andrew, and it has
> > been dried about 1 month, can you help to apply them?
>
> We shall have a look beginning of next week, sorry for the delay
> in getting back to you.
Note that the patch set no longer applies since the refactorings
conflict with new development by others.
Zhiqiang, can you rebase and post a new version of the patch set?
-Olof
On Wed, Nov 20, 2019 at 03:45:23AM +0000, Z.q. Hou wrote:
> From: Hou Zhiqiang <[email protected]>
>
> The Mobiveil PCIe controller can work in either Root Complex
> mode or Endpoint mode. So introduce a new structure root_port,
> and abstract the RC related members into it.
The first sentence explains the trigger for this work, the second
explains what you are changing, it would be helpful to also describe
why you need to make this change. You could do this by extending the
last sentence, e.g.
"So introduce a new structure root_port, and abstract the RC
related members into it such that it can be used by both ..."
As this series doesn't actually add a EP driver, this abstraction
isn't needed now - but it is nice to have - it may be helpful to explain
this.
The email subject could also more precisely explain what this patch
does.
>
> Signed-off-by: Hou Zhiqiang <[email protected]>
> ---
> V9:
> - New patch splited from the #1 of V8 patches to make it easy to review.
Indeed, it's much nicer to review - thanks.
>
> drivers/pci/controller/pcie-mobiveil.c | 99 ++++++++++++++++----------
> 1 file changed, 60 insertions(+), 39 deletions(-)
>
> diff --git a/drivers/pci/controller/pcie-mobiveil.c b/drivers/pci/controller/pcie-mobiveil.c
> index 3a696ca45bfa..5fd26e376af2 100644
> --- a/drivers/pci/controller/pcie-mobiveil.c
> +++ b/drivers/pci/controller/pcie-mobiveil.c
> @@ -3,7 +3,10 @@
> * PCIe host controller driver for Mobiveil PCIe Host controller
> *
> * Copyright (c) 2018 Mobiveil Inc.
> + * Copyright 2019 NXP
> + *
> * Author: Subrahmanya Lingappa <[email protected]>
> + * Recode: Hou Zhiqiang <[email protected]>
As per my previous feedback, I'm not sure the value of using the term refactor
or a synonym of it. And I certaintly don't want to encourage anyone that
modifies this file to add a similar tag when the information is easily visible
via GIT and the get_maintainers script.
> */
>
> #include <linux/delay.h>
> @@ -138,22 +141,27 @@ struct mobiveil_msi { /* MSI information */
> DECLARE_BITMAP(msi_irq_in_use, PCI_NUM_MSI);
> };
>
> +struct root_port {
> + char root_bus_nr;
> + void __iomem *config_axi_slave_base; /* endpoint config base */
> + struct resource *ob_io_res;
> + int irq;
> + raw_spinlock_t intx_mask_lock;
> + struct irq_domain *intx_domain;
> + struct mobiveil_msi msi;
> + struct pci_host_bridge *bridge;
> +};
Please prefix with mobiveil given we have mobiveil related structures
inside it.
(Also on your respin, please rebase as per Olof's feedback).
Thanks,
Andrew Murray
> +
> struct mobiveil_pcie {
> struct platform_device *pdev;
> - void __iomem *config_axi_slave_base; /* endpoint config base */
> void __iomem *csr_axi_slave_base; /* root port config base */
> void __iomem *apb_csr_base; /* MSI register base */
> phys_addr_t pcie_reg_base; /* Physical PCIe Controller Base */
> - struct irq_domain *intx_domain;
> - raw_spinlock_t intx_mask_lock;
> - int irq;
> int apio_wins;
> int ppio_wins;
> int ob_wins_configured; /* configured outbound windows */
> int ib_wins_configured; /* configured inbound windows */
> - struct resource *ob_io_res;
> - char root_bus_nr;
> - struct mobiveil_msi msi;
> + struct root_port rp;
> };
>
> /*
> @@ -281,16 +289,17 @@ static bool mobiveil_pcie_link_up(struct mobiveil_pcie *pcie)
> static bool mobiveil_pcie_valid_device(struct pci_bus *bus, unsigned int devfn)
> {
> struct mobiveil_pcie *pcie = bus->sysdata;
> + struct root_port *rp = &pcie->rp;
>
> /* Only one device down on each root port */
> - if ((bus->number == pcie->root_bus_nr) && (devfn > 0))
> + if ((bus->number == rp->root_bus_nr) && (devfn > 0))
> return false;
>
> /*
> * Do not read more than one device on the bus directly
> * attached to RC
> */
> - if ((bus->primary == pcie->root_bus_nr) && (PCI_SLOT(devfn) > 0))
> + if ((bus->primary == rp->root_bus_nr) && (PCI_SLOT(devfn) > 0))
> return false;
>
> return true;
> @@ -304,13 +313,14 @@ static void __iomem *mobiveil_pcie_map_bus(struct pci_bus *bus,
> unsigned int devfn, int where)
> {
> struct mobiveil_pcie *pcie = bus->sysdata;
> + struct root_port *rp = &pcie->rp;
> u32 value;
>
> if (!mobiveil_pcie_valid_device(bus, devfn))
> return NULL;
>
> /* RC config access */
> - if (bus->number == pcie->root_bus_nr)
> + if (bus->number == rp->root_bus_nr)
> return pcie->csr_axi_slave_base + where;
>
> /*
> @@ -325,7 +335,7 @@ static void __iomem *mobiveil_pcie_map_bus(struct pci_bus *bus,
>
> mobiveil_csr_writel(pcie, value, PAB_AXI_AMAP_PEX_WIN_L(WIN_NUM_0));
>
> - return pcie->config_axi_slave_base + where;
> + return rp->config_axi_slave_base + where;
> }
>
> static struct pci_ops mobiveil_pcie_ops = {
> @@ -339,7 +349,8 @@ static void mobiveil_pcie_isr(struct irq_desc *desc)
> struct irq_chip *chip = irq_desc_get_chip(desc);
> struct mobiveil_pcie *pcie = irq_desc_get_handler_data(desc);
> struct device *dev = &pcie->pdev->dev;
> - struct mobiveil_msi *msi = &pcie->msi;
> + struct root_port *rp = &pcie->rp;
> + struct mobiveil_msi *msi = &rp->msi;
> u32 msi_data, msi_addr_lo, msi_addr_hi;
> u32 intr_status, msi_status;
> unsigned long shifted_status;
> @@ -365,7 +376,7 @@ static void mobiveil_pcie_isr(struct irq_desc *desc)
> shifted_status >>= PAB_INTX_START;
> do {
> for_each_set_bit(bit, &shifted_status, PCI_NUM_INTX) {
> - virq = irq_find_mapping(pcie->intx_domain,
> + virq = irq_find_mapping(rp->intx_domain,
> bit + 1);
> if (virq)
> generic_handle_irq(virq);
> @@ -424,15 +435,16 @@ static int mobiveil_pcie_parse_dt(struct mobiveil_pcie *pcie)
> struct device *dev = &pcie->pdev->dev;
> struct platform_device *pdev = pcie->pdev;
> struct device_node *node = dev->of_node;
> + struct root_port *rp = &pcie->rp;
> struct resource *res;
>
> /* map config resource */
> res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
> "config_axi_slave");
> - pcie->config_axi_slave_base = devm_pci_remap_cfg_resource(dev, res);
> - if (IS_ERR(pcie->config_axi_slave_base))
> - return PTR_ERR(pcie->config_axi_slave_base);
> - pcie->ob_io_res = res;
> + rp->config_axi_slave_base = devm_pci_remap_cfg_resource(dev, res);
> + if (IS_ERR(rp->config_axi_slave_base))
> + return PTR_ERR(rp->config_axi_slave_base);
> + rp->ob_io_res = res;
>
> /* map csr resource */
> res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
> @@ -455,9 +467,9 @@ static int mobiveil_pcie_parse_dt(struct mobiveil_pcie *pcie)
> if (of_property_read_u32(node, "ppio-wins", &pcie->ppio_wins))
> pcie->ppio_wins = MAX_PIO_WINDOWS;
>
> - pcie->irq = platform_get_irq(pdev, 0);
> - if (pcie->irq <= 0) {
> - dev_err(dev, "failed to map IRQ: %d\n", pcie->irq);
> + rp->irq = platform_get_irq(pdev, 0);
> + if (rp->irq <= 0) {
> + dev_err(dev, "failed to map IRQ: %d\n", rp->irq);
> return -ENODEV;
> }
>
> @@ -564,9 +576,9 @@ static int mobiveil_bringup_link(struct mobiveil_pcie *pcie)
> static void mobiveil_pcie_enable_msi(struct mobiveil_pcie *pcie)
> {
> phys_addr_t msg_addr = pcie->pcie_reg_base;
> - struct mobiveil_msi *msi = &pcie->msi;
> + struct mobiveil_msi *msi = &pcie->rp.msi;
>
> - pcie->msi.num_of_vectors = PCI_NUM_MSI;
> + msi->num_of_vectors = PCI_NUM_MSI;
> msi->msi_pages_phys = (phys_addr_t)msg_addr;
>
> writel_relaxed(lower_32_bits(msg_addr),
> @@ -579,7 +591,8 @@ static void mobiveil_pcie_enable_msi(struct mobiveil_pcie *pcie)
>
> static int mobiveil_host_init(struct mobiveil_pcie *pcie)
> {
> - struct pci_host_bridge *bridge = pci_host_bridge_from_priv(pcie);
> + struct root_port *rp = &pcie->rp;
> + struct pci_host_bridge *bridge = rp->bridge;
> u32 value, pab_ctrl, type;
> struct resource_entry *win;
>
> @@ -629,8 +642,8 @@ static int mobiveil_host_init(struct mobiveil_pcie *pcie)
> */
>
> /* config outbound translation window */
> - program_ob_windows(pcie, WIN_NUM_0, pcie->ob_io_res->start, 0,
> - CFG_WINDOW_TYPE, resource_size(pcie->ob_io_res));
> + program_ob_windows(pcie, WIN_NUM_0, rp->ob_io_res->start, 0,
> + CFG_WINDOW_TYPE, resource_size(rp->ob_io_res));
>
> /* memory inbound translation window */
> program_ib_windows(pcie, WIN_NUM_0, 0, 0, MEM_WINDOW_TYPE, IB_WIN_SIZE);
> @@ -667,32 +680,36 @@ static void mobiveil_mask_intx_irq(struct irq_data *data)
> {
> struct irq_desc *desc = irq_to_desc(data->irq);
> struct mobiveil_pcie *pcie;
> + struct root_port *rp;
> unsigned long flags;
> u32 mask, shifted_val;
>
> pcie = irq_desc_get_chip_data(desc);
> + rp = &pcie->rp;
> mask = 1 << ((data->hwirq + PAB_INTX_START) - 1);
> - raw_spin_lock_irqsave(&pcie->intx_mask_lock, flags);
> + raw_spin_lock_irqsave(&rp->intx_mask_lock, flags);
> shifted_val = mobiveil_csr_readl(pcie, PAB_INTP_AMBA_MISC_ENB);
> shifted_val &= ~mask;
> mobiveil_csr_writel(pcie, shifted_val, PAB_INTP_AMBA_MISC_ENB);
> - raw_spin_unlock_irqrestore(&pcie->intx_mask_lock, flags);
> + raw_spin_unlock_irqrestore(&rp->intx_mask_lock, flags);
> }
>
> static void mobiveil_unmask_intx_irq(struct irq_data *data)
> {
> struct irq_desc *desc = irq_to_desc(data->irq);
> struct mobiveil_pcie *pcie;
> + struct root_port *rp;
> unsigned long flags;
> u32 shifted_val, mask;
>
> pcie = irq_desc_get_chip_data(desc);
> + rp = &pcie->rp;
> mask = 1 << ((data->hwirq + PAB_INTX_START) - 1);
> - raw_spin_lock_irqsave(&pcie->intx_mask_lock, flags);
> + raw_spin_lock_irqsave(&rp->intx_mask_lock, flags);
> shifted_val = mobiveil_csr_readl(pcie, PAB_INTP_AMBA_MISC_ENB);
> shifted_val |= mask;
> mobiveil_csr_writel(pcie, shifted_val, PAB_INTP_AMBA_MISC_ENB);
> - raw_spin_unlock_irqrestore(&pcie->intx_mask_lock, flags);
> + raw_spin_unlock_irqrestore(&rp->intx_mask_lock, flags);
> }
>
> static struct irq_chip intx_irq_chip = {
> @@ -760,7 +777,7 @@ static int mobiveil_irq_msi_domain_alloc(struct irq_domain *domain,
> unsigned int nr_irqs, void *args)
> {
> struct mobiveil_pcie *pcie = domain->host_data;
> - struct mobiveil_msi *msi = &pcie->msi;
> + struct mobiveil_msi *msi = &pcie->rp.msi;
> unsigned long bit;
>
> WARN_ON(nr_irqs != 1);
> @@ -787,7 +804,7 @@ static void mobiveil_irq_msi_domain_free(struct irq_domain *domain,
> {
> struct irq_data *d = irq_domain_get_irq_data(domain, virq);
> struct mobiveil_pcie *pcie = irq_data_get_irq_chip_data(d);
> - struct mobiveil_msi *msi = &pcie->msi;
> + struct mobiveil_msi *msi = &pcie->rp.msi;
>
> mutex_lock(&msi->lock);
>
> @@ -808,9 +825,9 @@ static int mobiveil_allocate_msi_domains(struct mobiveil_pcie *pcie)
> {
> struct device *dev = &pcie->pdev->dev;
> struct fwnode_handle *fwnode = of_node_to_fwnode(dev->of_node);
> - struct mobiveil_msi *msi = &pcie->msi;
> + struct mobiveil_msi *msi = &pcie->rp.msi;
>
> - mutex_init(&pcie->msi.lock);
> + mutex_init(&msi->lock);
> msi->dev_domain = irq_domain_add_linear(NULL, msi->num_of_vectors,
> &msi_domain_ops, pcie);
> if (!msi->dev_domain) {
> @@ -834,18 +851,19 @@ static int mobiveil_pcie_init_irq_domain(struct mobiveil_pcie *pcie)
> {
> struct device *dev = &pcie->pdev->dev;
> struct device_node *node = dev->of_node;
> + struct root_port *rp = &pcie->rp;
> int ret;
>
> /* setup INTx */
> - pcie->intx_domain = irq_domain_add_linear(node, PCI_NUM_INTX,
> - &intx_domain_ops, pcie);
> + rp->intx_domain = irq_domain_add_linear(node, PCI_NUM_INTX,
> + &intx_domain_ops, pcie);
>
> - if (!pcie->intx_domain) {
> + if (!rp->intx_domain) {
> dev_err(dev, "Failed to get a INTx IRQ domain\n");
> return -ENOMEM;
> }
>
> - raw_spin_lock_init(&pcie->intx_mask_lock);
> + raw_spin_lock_init(&rp->intx_mask_lock);
>
> /* setup MSI */
> ret = mobiveil_allocate_msi_domains(pcie);
> @@ -862,6 +880,7 @@ static int mobiveil_pcie_probe(struct platform_device *pdev)
> struct pci_bus *child;
> struct pci_host_bridge *bridge;
> struct device *dev = &pdev->dev;
> + struct root_port *rp;
> int ret;
>
> /* allocate the PCIe port */
> @@ -870,6 +889,8 @@ static int mobiveil_pcie_probe(struct platform_device *pdev)
> return -ENOMEM;
>
> pcie = pci_host_bridge_priv(bridge);
> + rp = &pcie->rp;
> + rp->bridge = bridge;
>
> pcie->pdev = pdev;
>
> @@ -904,12 +925,12 @@ static int mobiveil_pcie_probe(struct platform_device *pdev)
> return ret;
> }
>
> - irq_set_chained_handler_and_data(pcie->irq, mobiveil_pcie_isr, pcie);
> + irq_set_chained_handler_and_data(rp->irq, mobiveil_pcie_isr, pcie);
>
> /* Initialize bridge */
> bridge->dev.parent = dev;
> bridge->sysdata = pcie;
> - bridge->busnr = pcie->root_bus_nr;
> + bridge->busnr = rp->root_bus_nr;
> bridge->ops = &mobiveil_pcie_ops;
> bridge->map_irq = of_irq_parse_and_map_pci;
> bridge->swizzle_irq = pci_common_swizzle;
> --
> 2.17.1
>
On Wed, Nov 20, 2019 at 03:45:30AM +0000, Z.q. Hou wrote:
> From: Hou Zhiqiang <[email protected]>
>
> Move the host initialization related operations into a new
> routine to make it can be reused by other incoming platform's
s/to make/such that/
'function' is probably a better word than 'routine'.
> PCIe host driver, in which the Mobiveil GPEX is integrated.
>
> Signed-off-by: Hou Zhiqiang <[email protected]>
> ---
> V9:
> - New patch splited from the #1 of V8 patches to make it easy to review.
>
> drivers/pci/controller/pcie-mobiveil.c | 38 +++++++++++++++-----------
> 1 file changed, 22 insertions(+), 16 deletions(-)
>
> diff --git a/drivers/pci/controller/pcie-mobiveil.c b/drivers/pci/controller/pcie-mobiveil.c
> index 5fd26e376af2..97f682ca7c7a 100644
> --- a/drivers/pci/controller/pcie-mobiveil.c
> +++ b/drivers/pci/controller/pcie-mobiveil.c
> @@ -873,27 +873,15 @@ static int mobiveil_pcie_init_irq_domain(struct mobiveil_pcie *pcie)
> return 0;
> }
>
> -static int mobiveil_pcie_probe(struct platform_device *pdev)
> +int mobiveil_pcie_host_probe(struct mobiveil_pcie *pcie)
This is no longer static - but do you need to add a header file somewhere?
> {
> - struct mobiveil_pcie *pcie;
> + struct root_port *rp = &pcie->rp;
> + struct pci_host_bridge *bridge = rp->bridge;
> + struct device *dev = &pcie->pdev->dev;
> struct pci_bus *bus;
> struct pci_bus *child;
> - struct pci_host_bridge *bridge;
> - struct device *dev = &pdev->dev;
> - struct root_port *rp;
> int ret;
>
> - /* allocate the PCIe port */
> - bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
> - if (!bridge)
> - return -ENOMEM;
> -
> - pcie = pci_host_bridge_priv(bridge);
> - rp = &pcie->rp;
> - rp->bridge = bridge;
> -
> - pcie->pdev = pdev;
> -
> ret = mobiveil_pcie_parse_dt(pcie);
> if (ret) {
> dev_err(dev, "Parsing DT failed, ret: %x\n", ret);
> @@ -956,6 +944,24 @@ static int mobiveil_pcie_probe(struct platform_device *pdev)
> return 0;
> }
>
> +static int mobiveil_pcie_probe(struct platform_device *pdev)
> +{
> + struct mobiveil_pcie *pcie;
> + struct pci_host_bridge *bridge;
> + struct device *dev = &pdev->dev;
> +
> + bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
You've lost the comment that was above this.
Thanks,
Andrew Murray
> + if (!bridge)
> + return -ENOMEM;
> +
> + pcie = pci_host_bridge_priv(bridge);
> + pcie->rp.bridge = bridge;
> +
> + pcie->pdev = pdev;
> +
> + return mobiveil_pcie_host_probe(pcie);
> +}
> +
> static const struct of_device_id mobiveil_pcie_of_match[] = {
> {.compatible = "mbvl,gpex40-pcie",},
> {},
> --
> 2.17.1
>
On Wed, Nov 20, 2019 at 03:45:37AM +0000, Z.q. Hou wrote:
> From: Hou Zhiqiang <[email protected]>
>
> Collect the interrupt initialization related operations into
> a new routine to make it more readable.
I prefer the word 'function' instead of routine. Also indicate why, not only
is it nicer but it is in preparation for EP support.
>
> Signed-off-by: Hou Zhiqiang <[email protected]>
> ---
> V9:
> - New patch splited from the #1 of V8 patches to make it easy to review.
>
> drivers/pci/controller/pcie-mobiveil.c | 65 +++++++++++++++++---------
> 1 file changed, 42 insertions(+), 23 deletions(-)
>
> diff --git a/drivers/pci/controller/pcie-mobiveil.c b/drivers/pci/controller/pcie-mobiveil.c
> index 97f682ca7c7a..512b27a0536e 100644
> --- a/drivers/pci/controller/pcie-mobiveil.c
> +++ b/drivers/pci/controller/pcie-mobiveil.c
> @@ -454,12 +454,6 @@ static int mobiveil_pcie_parse_dt(struct mobiveil_pcie *pcie)
> return PTR_ERR(pcie->csr_axi_slave_base);
> pcie->pcie_reg_base = res->start;
>
> - /* map MSI config resource */
> - res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "apb_csr");
> - pcie->apb_csr_base = devm_pci_remap_cfg_resource(dev, res);
> - if (IS_ERR(pcie->apb_csr_base))
> - return PTR_ERR(pcie->apb_csr_base);
> -
> /* read the number of windows requested */
> if (of_property_read_u32(node, "apio-wins", &pcie->apio_wins))
> pcie->apio_wins = MAX_PIO_WINDOWS;
> @@ -467,12 +461,6 @@ static int mobiveil_pcie_parse_dt(struct mobiveil_pcie *pcie)
> if (of_property_read_u32(node, "ppio-wins", &pcie->ppio_wins))
> pcie->ppio_wins = MAX_PIO_WINDOWS;
>
> - rp->irq = platform_get_irq(pdev, 0);
> - if (rp->irq <= 0) {
> - dev_err(dev, "failed to map IRQ: %d\n", rp->irq);
> - return -ENODEV;
> - }
> -
> return 0;
> }
>
> @@ -618,9 +606,6 @@ static int mobiveil_host_init(struct mobiveil_pcie *pcie)
> pab_ctrl |= (1 << AMBA_PIO_ENABLE_SHIFT) | (1 << PEX_PIO_ENABLE_SHIFT);
> mobiveil_csr_writel(pcie, pab_ctrl, PAB_CTRL);
>
> - mobiveil_csr_writel(pcie, (PAB_INTP_INTX_MASK | PAB_INTP_MSI_MASK),
> - PAB_INTP_AMBA_MISC_ENB);
> -
> /*
> * program PIO Enable Bit to 1 and Config Window Enable Bit to 1 in
> * PAB_AXI_PIO_CTRL Register
> @@ -670,9 +655,6 @@ static int mobiveil_host_init(struct mobiveil_pcie *pcie)
> value |= (PCI_CLASS_BRIDGE_PCI << 16);
> mobiveil_csr_writel(pcie, value, PAB_INTP_AXI_PIO_CLASS);
>
> - /* setup MSI hardware registers */
> - mobiveil_pcie_enable_msi(pcie);
> -
> return 0;
> }
>
> @@ -873,6 +855,46 @@ static int mobiveil_pcie_init_irq_domain(struct mobiveil_pcie *pcie)
> return 0;
> }
>
> +static int mobiveil_pcie_interrupt_init(struct mobiveil_pcie *pcie)
> +{
> + struct platform_device *pdev = pcie->pdev;
> + struct device *dev = &pdev->dev;
> + struct root_port *rp = &pcie->rp;
> + struct resource *res;
> + int ret;
> +
> + /* map MSI config resource */
> + res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "apb_csr");
> + pcie->apb_csr_base = devm_pci_remap_cfg_resource(dev, res);
> + if (IS_ERR(pcie->apb_csr_base))
> + return PTR_ERR(pcie->apb_csr_base);
> +
> + /* setup MSI hardware registers */
> + mobiveil_pcie_enable_msi(pcie);
Does this need to come after mobiveil_pcie_init_irq_domain - given that
this function sets up the irq domain for MSI?
Thanks,
Andrew Murray
> +
> + rp->irq = platform_get_irq(pdev, 0);
> + if (rp->irq <= 0) {
> + dev_err(dev, "failed to map IRQ: %d\n", rp->irq);
> + return -ENODEV;
> + }
> +
> + /* initialize the IRQ domains */
> + ret = mobiveil_pcie_init_irq_domain(pcie);
> + if (ret) {
> + dev_err(dev, "Failed creating IRQ Domain\n");
> + return ret;
> + }
> +
> + irq_set_chained_handler_and_data(rp->irq, mobiveil_pcie_isr, pcie);
> +
> + /* Enable interrupts */
> + mobiveil_csr_writel(pcie, (PAB_INTP_INTX_MASK | PAB_INTP_MSI_MASK),
> + PAB_INTP_AMBA_MISC_ENB);
> +
> +
> + return 0;
> +}
> +
> int mobiveil_pcie_host_probe(struct mobiveil_pcie *pcie)
> {
> struct root_port *rp = &pcie->rp;
> @@ -906,15 +928,12 @@ int mobiveil_pcie_host_probe(struct mobiveil_pcie *pcie)
> return ret;
> }
>
> - /* initialize the IRQ domains */
> - ret = mobiveil_pcie_init_irq_domain(pcie);
> + ret = mobiveil_pcie_interrupt_init(pcie);
> if (ret) {
> - dev_err(dev, "Failed creating IRQ Domain\n");
> + dev_err(dev, "Interrupt init failed\n");
> return ret;
> }
>
> - irq_set_chained_handler_and_data(rp->irq, mobiveil_pcie_isr, pcie);
> -
> /* Initialize bridge */
> bridge->dev.parent = dev;
> bridge->sysdata = pcie;
> --
> 2.17.1
>
On Wed, Nov 20, 2019 at 03:45:43AM +0000, Z.q. Hou wrote:
> From: Hou Zhiqiang <[email protected]>
>
> Modularize the Mobiveil PCIe host driver according to the
> abstraction of Root Complex and Endpoint and move it into
> a new directory.
>
> Split the RC related routines into pcie-mobiveil-host.c,
> and common routines into pcie-mobiveil.c, move the macro
> definitions and function declarations into pcie-mobiveil.h,
> and the Mobiveil platform reference code into
> pcie-mobiveil-plat.c. So that it is easy to reuse the
> extracted routines to add a new host driver, which
> integrated Mobiveil PCIe GPEX IP.
>
> Signed-off-by: Hou Zhiqiang <[email protected]>
Reviewed-by: Andrew Murray <[email protected]>
> ---
> V9:
> - New patch splited from the #1 of V8 patches to make it easy to review.
>
> MAINTAINERS | 2 +-
> drivers/pci/controller/Kconfig | 11 +-
> drivers/pci/controller/Makefile | 2 +-
> drivers/pci/controller/mobiveil/Kconfig | 24 ++
> drivers/pci/controller/mobiveil/Makefile | 4 +
> .../pcie-mobiveil-host.c} | 398 +-----------------
> .../controller/mobiveil/pcie-mobiveil-plat.c | 60 +++
> .../pci/controller/mobiveil/pcie-mobiveil.c | 227 ++++++++++
> .../pci/controller/mobiveil/pcie-mobiveil.h | 178 ++++++++
> 9 files changed, 497 insertions(+), 409 deletions(-)
> create mode 100644 drivers/pci/controller/mobiveil/Kconfig
> create mode 100644 drivers/pci/controller/mobiveil/Makefile
> rename drivers/pci/controller/{pcie-mobiveil.c => mobiveil/pcie-mobiveil-host.c} (61%)
> create mode 100644 drivers/pci/controller/mobiveil/pcie-mobiveil-plat.c
> create mode 100644 drivers/pci/controller/mobiveil/pcie-mobiveil.c
> create mode 100644 drivers/pci/controller/mobiveil/pcie-mobiveil.h
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 3f7f8cdbc471..a4ad99619e53 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -12612,7 +12612,7 @@ M: Hou Zhiqiang <[email protected]>
> L: [email protected]
> S: Supported
> F: Documentation/devicetree/bindings/pci/mobiveil-pcie.txt
> -F: drivers/pci/controller/pcie-mobiveil.c
> +F: drivers/pci/controller/mobiveil/pcie-mobiveil*
>
> PCI DRIVER FOR MVEBU (Marvell Armada 370 and Armada XP SOC support)
> M: Thomas Petazzoni <[email protected]>
> diff --git a/drivers/pci/controller/Kconfig b/drivers/pci/controller/Kconfig
> index f5de9119e8d3..74fd332755ae 100644
> --- a/drivers/pci/controller/Kconfig
> +++ b/drivers/pci/controller/Kconfig
> @@ -241,16 +241,6 @@ config PCIE_MEDIATEK
> Say Y here if you want to enable PCIe controller support on
> MediaTek SoCs.
>
> -config PCIE_MOBIVEIL
> - bool "Mobiveil AXI PCIe controller"
> - depends on ARCH_ZYNQMP || COMPILE_TEST
> - depends on OF
> - depends on PCI_MSI_IRQ_DOMAIN
> - help
> - Say Y here if you want to enable support for the Mobiveil AXI PCIe
> - Soft IP. It has up to 8 outbound and inbound windows
> - for address translation and it is a PCIe Gen4 IP.
> -
> config PCIE_TANGO_SMP8759
> bool "Tango SMP8759 PCIe controller (DANGEROUS)"
> depends on ARCH_TANGO && PCI_MSI && OF
> @@ -289,4 +279,5 @@ config PCI_HYPERV_INTERFACE
> have a common interface with the Hyper-V PCI frontend driver.
>
> source "drivers/pci/controller/dwc/Kconfig"
> +source "drivers/pci/controller/mobiveil/Kconfig"
> endmenu
> diff --git a/drivers/pci/controller/Makefile b/drivers/pci/controller/Makefile
> index a2a22c9d91af..44414cfd45ea 100644
> --- a/drivers/pci/controller/Makefile
> +++ b/drivers/pci/controller/Makefile
> @@ -27,11 +27,11 @@ obj-$(CONFIG_PCIE_ROCKCHIP) += pcie-rockchip.o
> obj-$(CONFIG_PCIE_ROCKCHIP_EP) += pcie-rockchip-ep.o
> obj-$(CONFIG_PCIE_ROCKCHIP_HOST) += pcie-rockchip-host.o
> obj-$(CONFIG_PCIE_MEDIATEK) += pcie-mediatek.o
> -obj-$(CONFIG_PCIE_MOBIVEIL) += pcie-mobiveil.o
> obj-$(CONFIG_PCIE_TANGO_SMP8759) += pcie-tango.o
> obj-$(CONFIG_VMD) += vmd.o
> # pcie-hisi.o quirks are needed even without CONFIG_PCIE_DW
> obj-y += dwc/
> +obj-y += mobiveil/
>
>
> # The following drivers are for devices that use the generic ACPI
> diff --git a/drivers/pci/controller/mobiveil/Kconfig b/drivers/pci/controller/mobiveil/Kconfig
> new file mode 100644
> index 000000000000..64343c07bfed
> --- /dev/null
> +++ b/drivers/pci/controller/mobiveil/Kconfig
> @@ -0,0 +1,24 @@
> +# SPDX-License-Identifier: GPL-2.0
> +
> +menu "Mobiveil PCIe Core Support"
> + depends on PCI
> +
> +config PCIE_MOBIVEIL
> + bool
> +
> +config PCIE_MOBIVEIL_HOST
> + bool
> + depends on PCI_MSI_IRQ_DOMAIN
> + select PCIE_MOBIVEIL
> +
> +config PCIE_MOBIVEIL_PLAT
> + bool "Mobiveil AXI PCIe controller"
> + depends on ARCH_ZYNQMP || COMPILE_TEST
> + depends on OF
> + select PCIE_MOBIVEIL_HOST
> + help
> + Say Y here if you want to enable support for the Mobiveil AXI PCIe
> + Soft IP. It has up to 8 outbound and inbound windows
> + for address translation and it is a PCIe Gen4 IP.
> +
> +endmenu
> diff --git a/drivers/pci/controller/mobiveil/Makefile b/drivers/pci/controller/mobiveil/Makefile
> new file mode 100644
> index 000000000000..9fb6d1c6504d
> --- /dev/null
> +++ b/drivers/pci/controller/mobiveil/Makefile
> @@ -0,0 +1,4 @@
> +# SPDX-License-Identifier: GPL-2.0
> +obj-$(CONFIG_PCIE_MOBIVEIL) += pcie-mobiveil.o
> +obj-$(CONFIG_PCIE_MOBIVEIL_HOST) += pcie-mobiveil-host.o
> +obj-$(CONFIG_PCIE_MOBIVEIL_PLAT) += pcie-mobiveil-plat.o
> diff --git a/drivers/pci/controller/pcie-mobiveil.c b/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
> similarity index 61%
> rename from drivers/pci/controller/pcie-mobiveil.c
> rename to drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
> index 512b27a0536e..2cc424e78d33 100644
> --- a/drivers/pci/controller/pcie-mobiveil.c
> +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
> @@ -9,7 +9,6 @@
> * Recode: Hou Zhiqiang <[email protected]>
> */
>
> -#include <linux/delay.h>
> #include <linux/init.h>
> #include <linux/interrupt.h>
> #include <linux/irq.h>
> @@ -26,265 +25,7 @@
> #include <linux/platform_device.h>
> #include <linux/slab.h>
>
> -#include "../pci.h"
> -
> -/* register offsets and bit positions */
> -
> -/*
> - * translation tables are grouped into windows, each window registers are
> - * grouped into blocks of 4 or 16 registers each
> - */
> -#define PAB_REG_BLOCK_SIZE 16
> -#define PAB_EXT_REG_BLOCK_SIZE 4
> -
> -#define PAB_REG_ADDR(offset, win) \
> - (offset + (win * PAB_REG_BLOCK_SIZE))
> -#define PAB_EXT_REG_ADDR(offset, win) \
> - (offset + (win * PAB_EXT_REG_BLOCK_SIZE))
> -
> -#define LTSSM_STATUS 0x0404
> -#define LTSSM_STATUS_L0_MASK 0x3f
> -#define LTSSM_STATUS_L0 0x2d
> -
> -#define PAB_CTRL 0x0808
> -#define AMBA_PIO_ENABLE_SHIFT 0
> -#define PEX_PIO_ENABLE_SHIFT 1
> -#define PAGE_SEL_SHIFT 13
> -#define PAGE_SEL_MASK 0x3f
> -#define PAGE_LO_MASK 0x3ff
> -#define PAGE_SEL_OFFSET_SHIFT 10
> -
> -#define PAB_AXI_PIO_CTRL 0x0840
> -#define APIO_EN_MASK 0xf
> -
> -#define PAB_PEX_PIO_CTRL 0x08c0
> -#define PIO_ENABLE_SHIFT 0
> -
> -#define PAB_INTP_AMBA_MISC_ENB 0x0b0c
> -#define PAB_INTP_AMBA_MISC_STAT 0x0b1c
> -#define PAB_INTP_INTX_MASK 0x01e0
> -#define PAB_INTP_MSI_MASK 0x8
> -
> -#define PAB_AXI_AMAP_CTRL(win) PAB_REG_ADDR(0x0ba0, win)
> -#define WIN_ENABLE_SHIFT 0
> -#define WIN_TYPE_SHIFT 1
> -#define WIN_TYPE_MASK 0x3
> -#define WIN_SIZE_MASK 0xfffffc00
> -
> -#define PAB_EXT_AXI_AMAP_SIZE(win) PAB_EXT_REG_ADDR(0xbaf0, win)
> -
> -#define PAB_EXT_AXI_AMAP_AXI_WIN(win) PAB_EXT_REG_ADDR(0x80a0, win)
> -#define PAB_AXI_AMAP_AXI_WIN(win) PAB_REG_ADDR(0x0ba4, win)
> -#define AXI_WINDOW_ALIGN_MASK 3
> -
> -#define PAB_AXI_AMAP_PEX_WIN_L(win) PAB_REG_ADDR(0x0ba8, win)
> -#define PAB_BUS_SHIFT 24
> -#define PAB_DEVICE_SHIFT 19
> -#define PAB_FUNCTION_SHIFT 16
> -
> -#define PAB_AXI_AMAP_PEX_WIN_H(win) PAB_REG_ADDR(0x0bac, win)
> -#define PAB_INTP_AXI_PIO_CLASS 0x474
> -
> -#define PAB_PEX_AMAP_CTRL(win) PAB_REG_ADDR(0x4ba0, win)
> -#define AMAP_CTRL_EN_SHIFT 0
> -#define AMAP_CTRL_TYPE_SHIFT 1
> -#define AMAP_CTRL_TYPE_MASK 3
> -
> -#define PAB_EXT_PEX_AMAP_SIZEN(win) PAB_EXT_REG_ADDR(0xbef0, win)
> -#define PAB_EXT_PEX_AMAP_AXI_WIN(win) PAB_EXT_REG_ADDR(0xb4a0, win)
> -#define PAB_PEX_AMAP_AXI_WIN(win) PAB_REG_ADDR(0x4ba4, win)
> -#define PAB_PEX_AMAP_PEX_WIN_L(win) PAB_REG_ADDR(0x4ba8, win)
> -#define PAB_PEX_AMAP_PEX_WIN_H(win) PAB_REG_ADDR(0x4bac, win)
> -
> -/* starting offset of INTX bits in status register */
> -#define PAB_INTX_START 5
> -
> -/* supported number of MSI interrupts */
> -#define PCI_NUM_MSI 16
> -
> -/* MSI registers */
> -#define MSI_BASE_LO_OFFSET 0x04
> -#define MSI_BASE_HI_OFFSET 0x08
> -#define MSI_SIZE_OFFSET 0x0c
> -#define MSI_ENABLE_OFFSET 0x14
> -#define MSI_STATUS_OFFSET 0x18
> -#define MSI_DATA_OFFSET 0x20
> -#define MSI_ADDR_L_OFFSET 0x24
> -#define MSI_ADDR_H_OFFSET 0x28
> -
> -/* outbound and inbound window definitions */
> -#define WIN_NUM_0 0
> -#define WIN_NUM_1 1
> -#define CFG_WINDOW_TYPE 0
> -#define IO_WINDOW_TYPE 1
> -#define MEM_WINDOW_TYPE 2
> -#define IB_WIN_SIZE ((u64)256 * 1024 * 1024 * 1024)
> -#define MAX_PIO_WINDOWS 8
> -
> -/* Parameters for the waiting for link up routine */
> -#define LINK_WAIT_MAX_RETRIES 10
> -#define LINK_WAIT_MIN 90000
> -#define LINK_WAIT_MAX 100000
> -
> -#define PAGED_ADDR_BNDRY 0xc00
> -#define OFFSET_TO_PAGE_ADDR(off) \
> - ((off & PAGE_LO_MASK) | PAGED_ADDR_BNDRY)
> -#define OFFSET_TO_PAGE_IDX(off) \
> - ((off >> PAGE_SEL_OFFSET_SHIFT) & PAGE_SEL_MASK)
> -
> -struct mobiveil_msi { /* MSI information */
> - struct mutex lock; /* protect bitmap variable */
> - struct irq_domain *msi_domain;
> - struct irq_domain *dev_domain;
> - phys_addr_t msi_pages_phys;
> - int num_of_vectors;
> - DECLARE_BITMAP(msi_irq_in_use, PCI_NUM_MSI);
> -};
> -
> -struct root_port {
> - char root_bus_nr;
> - void __iomem *config_axi_slave_base; /* endpoint config base */
> - struct resource *ob_io_res;
> - int irq;
> - raw_spinlock_t intx_mask_lock;
> - struct irq_domain *intx_domain;
> - struct mobiveil_msi msi;
> - struct pci_host_bridge *bridge;
> -};
> -
> -struct mobiveil_pcie {
> - struct platform_device *pdev;
> - void __iomem *csr_axi_slave_base; /* root port config base */
> - void __iomem *apb_csr_base; /* MSI register base */
> - phys_addr_t pcie_reg_base; /* Physical PCIe Controller Base */
> - int apio_wins;
> - int ppio_wins;
> - int ob_wins_configured; /* configured outbound windows */
> - int ib_wins_configured; /* configured inbound windows */
> - struct root_port rp;
> -};
> -
> -/*
> - * mobiveil_pcie_sel_page - routine to access paged register
> - *
> - * Registers whose address greater than PAGED_ADDR_BNDRY (0xc00) are paged,
> - * for this scheme to work extracted higher 6 bits of the offset will be
> - * written to pg_sel field of PAB_CTRL register and rest of the lower 10
> - * bits enabled with PAGED_ADDR_BNDRY are used as offset of the register.
> - */
> -static void mobiveil_pcie_sel_page(struct mobiveil_pcie *pcie, u8 pg_idx)
> -{
> - u32 val;
> -
> - val = readl(pcie->csr_axi_slave_base + PAB_CTRL);
> - val &= ~(PAGE_SEL_MASK << PAGE_SEL_SHIFT);
> - val |= (pg_idx & PAGE_SEL_MASK) << PAGE_SEL_SHIFT;
> -
> - writel(val, pcie->csr_axi_slave_base + PAB_CTRL);
> -}
> -
> -static void *mobiveil_pcie_comp_addr(struct mobiveil_pcie *pcie, u32 off)
> -{
> - if (off < PAGED_ADDR_BNDRY) {
> - /* For directly accessed registers, clear the pg_sel field */
> - mobiveil_pcie_sel_page(pcie, 0);
> - return pcie->csr_axi_slave_base + off;
> - }
> -
> - mobiveil_pcie_sel_page(pcie, OFFSET_TO_PAGE_IDX(off));
> - return pcie->csr_axi_slave_base + OFFSET_TO_PAGE_ADDR(off);
> -}
> -
> -static int mobiveil_pcie_read(void __iomem *addr, int size, u32 *val)
> -{
> - if ((uintptr_t)addr & (size - 1)) {
> - *val = 0;
> - return PCIBIOS_BAD_REGISTER_NUMBER;
> - }
> -
> - switch (size) {
> - case 4:
> - *val = readl(addr);
> - break;
> - case 2:
> - *val = readw(addr);
> - break;
> - case 1:
> - *val = readb(addr);
> - break;
> - default:
> - *val = 0;
> - return PCIBIOS_BAD_REGISTER_NUMBER;
> - }
> -
> - return PCIBIOS_SUCCESSFUL;
> -}
> -
> -static int mobiveil_pcie_write(void __iomem *addr, int size, u32 val)
> -{
> - if ((uintptr_t)addr & (size - 1))
> - return PCIBIOS_BAD_REGISTER_NUMBER;
> -
> - switch (size) {
> - case 4:
> - writel(val, addr);
> - break;
> - case 2:
> - writew(val, addr);
> - break;
> - case 1:
> - writeb(val, addr);
> - break;
> - default:
> - return PCIBIOS_BAD_REGISTER_NUMBER;
> - }
> -
> - return PCIBIOS_SUCCESSFUL;
> -}
> -
> -static u32 mobiveil_csr_read(struct mobiveil_pcie *pcie, u32 off, size_t size)
> -{
> - void *addr;
> - u32 val;
> - int ret;
> -
> - addr = mobiveil_pcie_comp_addr(pcie, off);
> -
> - ret = mobiveil_pcie_read(addr, size, &val);
> - if (ret)
> - dev_err(&pcie->pdev->dev, "read CSR address failed\n");
> -
> - return val;
> -}
> -
> -static void mobiveil_csr_write(struct mobiveil_pcie *pcie, u32 val, u32 off,
> - size_t size)
> -{
> - void *addr;
> - int ret;
> -
> - addr = mobiveil_pcie_comp_addr(pcie, off);
> -
> - ret = mobiveil_pcie_write(addr, size, val);
> - if (ret)
> - dev_err(&pcie->pdev->dev, "write CSR address failed\n");
> -}
> -
> -static u32 mobiveil_csr_readl(struct mobiveil_pcie *pcie, u32 off)
> -{
> - return mobiveil_csr_read(pcie, off, 0x4);
> -}
> -
> -static void mobiveil_csr_writel(struct mobiveil_pcie *pcie, u32 val, u32 off)
> -{
> - mobiveil_csr_write(pcie, val, off, 0x4);
> -}
> -
> -static bool mobiveil_pcie_link_up(struct mobiveil_pcie *pcie)
> -{
> - return (mobiveil_csr_readl(pcie, LTSSM_STATUS) &
> - LTSSM_STATUS_L0_MASK) == LTSSM_STATUS_L0;
> -}
> +#include "pcie-mobiveil.h"
>
> static bool mobiveil_pcie_valid_device(struct pci_bus *bus, unsigned int devfn)
> {
> @@ -464,103 +205,6 @@ static int mobiveil_pcie_parse_dt(struct mobiveil_pcie *pcie)
> return 0;
> }
>
> -static void program_ib_windows(struct mobiveil_pcie *pcie, int win_num,
> - u64 cpu_addr, u64 pci_addr, u32 type, u64 size)
> -{
> - u32 value;
> - u64 size64 = ~(size - 1);
> -
> - if (win_num >= pcie->ppio_wins) {
> - dev_err(&pcie->pdev->dev,
> - "ERROR: max inbound windows reached !\n");
> - return;
> - }
> -
> - value = mobiveil_csr_readl(pcie, PAB_PEX_AMAP_CTRL(win_num));
> - value &= ~(AMAP_CTRL_TYPE_MASK << AMAP_CTRL_TYPE_SHIFT | WIN_SIZE_MASK);
> - value |= type << AMAP_CTRL_TYPE_SHIFT | 1 << AMAP_CTRL_EN_SHIFT |
> - (lower_32_bits(size64) & WIN_SIZE_MASK);
> - mobiveil_csr_writel(pcie, value, PAB_PEX_AMAP_CTRL(win_num));
> -
> - mobiveil_csr_writel(pcie, upper_32_bits(size64),
> - PAB_EXT_PEX_AMAP_SIZEN(win_num));
> -
> - mobiveil_csr_writel(pcie, lower_32_bits(cpu_addr),
> - PAB_PEX_AMAP_AXI_WIN(win_num));
> - mobiveil_csr_writel(pcie, upper_32_bits(cpu_addr),
> - PAB_EXT_PEX_AMAP_AXI_WIN(win_num));
> -
> - mobiveil_csr_writel(pcie, lower_32_bits(pci_addr),
> - PAB_PEX_AMAP_PEX_WIN_L(win_num));
> - mobiveil_csr_writel(pcie, upper_32_bits(pci_addr),
> - PAB_PEX_AMAP_PEX_WIN_H(win_num));
> -
> - pcie->ib_wins_configured++;
> -}
> -
> -/*
> - * routine to program the outbound windows
> - */
> -static void program_ob_windows(struct mobiveil_pcie *pcie, int win_num,
> - u64 cpu_addr, u64 pci_addr, u32 type, u64 size)
> -{
> - u32 value;
> - u64 size64 = ~(size - 1);
> -
> - if (win_num >= pcie->apio_wins) {
> - dev_err(&pcie->pdev->dev,
> - "ERROR: max outbound windows reached !\n");
> - return;
> - }
> -
> - /*
> - * program Enable Bit to 1, Type Bit to (00) base 2, AXI Window Size Bit
> - * to 4 KB in PAB_AXI_AMAP_CTRL register
> - */
> - value = mobiveil_csr_readl(pcie, PAB_AXI_AMAP_CTRL(win_num));
> - value &= ~(WIN_TYPE_MASK << WIN_TYPE_SHIFT | WIN_SIZE_MASK);
> - value |= 1 << WIN_ENABLE_SHIFT | type << WIN_TYPE_SHIFT |
> - (lower_32_bits(size64) & WIN_SIZE_MASK);
> - mobiveil_csr_writel(pcie, value, PAB_AXI_AMAP_CTRL(win_num));
> -
> - mobiveil_csr_writel(pcie, upper_32_bits(size64),
> - PAB_EXT_AXI_AMAP_SIZE(win_num));
> -
> - /*
> - * program AXI window base with appropriate value in
> - * PAB_AXI_AMAP_AXI_WIN0 register
> - */
> - mobiveil_csr_writel(pcie,
> - lower_32_bits(cpu_addr) & (~AXI_WINDOW_ALIGN_MASK),
> - PAB_AXI_AMAP_AXI_WIN(win_num));
> - mobiveil_csr_writel(pcie, upper_32_bits(cpu_addr),
> - PAB_EXT_AXI_AMAP_AXI_WIN(win_num));
> -
> - mobiveil_csr_writel(pcie, lower_32_bits(pci_addr),
> - PAB_AXI_AMAP_PEX_WIN_L(win_num));
> - mobiveil_csr_writel(pcie, upper_32_bits(pci_addr),
> - PAB_AXI_AMAP_PEX_WIN_H(win_num));
> -
> - pcie->ob_wins_configured++;
> -}
> -
> -static int mobiveil_bringup_link(struct mobiveil_pcie *pcie)
> -{
> - int retries;
> -
> - /* check if the link is up or not */
> - for (retries = 0; retries < LINK_WAIT_MAX_RETRIES; retries++) {
> - if (mobiveil_pcie_link_up(pcie))
> - return 0;
> -
> - usleep_range(LINK_WAIT_MIN, LINK_WAIT_MAX);
> - }
> -
> - dev_err(&pcie->pdev->dev, "link never came up\n");
> -
> - return -ETIMEDOUT;
> -}
> -
> static void mobiveil_pcie_enable_msi(struct mobiveil_pcie *pcie)
> {
> phys_addr_t msg_addr = pcie->pcie_reg_base;
> @@ -962,43 +606,3 @@ int mobiveil_pcie_host_probe(struct mobiveil_pcie *pcie)
>
> return 0;
> }
> -
> -static int mobiveil_pcie_probe(struct platform_device *pdev)
> -{
> - struct mobiveil_pcie *pcie;
> - struct pci_host_bridge *bridge;
> - struct device *dev = &pdev->dev;
> -
> - bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
> - if (!bridge)
> - return -ENOMEM;
> -
> - pcie = pci_host_bridge_priv(bridge);
> - pcie->rp.bridge = bridge;
> -
> - pcie->pdev = pdev;
> -
> - return mobiveil_pcie_host_probe(pcie);
> -}
> -
> -static const struct of_device_id mobiveil_pcie_of_match[] = {
> - {.compatible = "mbvl,gpex40-pcie",},
> - {},
> -};
> -
> -MODULE_DEVICE_TABLE(of, mobiveil_pcie_of_match);
> -
> -static struct platform_driver mobiveil_pcie_driver = {
> - .probe = mobiveil_pcie_probe,
> - .driver = {
> - .name = "mobiveil-pcie",
> - .of_match_table = mobiveil_pcie_of_match,
> - .suppress_bind_attrs = true,
> - },
> -};
> -
> -builtin_platform_driver(mobiveil_pcie_driver);
> -
> -MODULE_LICENSE("GPL v2");
> -MODULE_DESCRIPTION("Mobiveil PCIe host controller driver");
> -MODULE_AUTHOR("Subrahmanya Lingappa <[email protected]>");
> diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil-plat.c b/drivers/pci/controller/mobiveil/pcie-mobiveil-plat.c
> new file mode 100644
> index 000000000000..64c85f852869
> --- /dev/null
> +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil-plat.c
> @@ -0,0 +1,60 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * PCIe host controller driver for Mobiveil PCIe Host controller
> + *
> + * Copyright (c) 2018 Mobiveil Inc.
> + * Copyright 2019 NXP
> + *
> + * Author: Subrahmanya Lingappa <[email protected]>
> + * Recode: Hou Zhiqiang <[email protected]>
Same feedback as before with the recode tag.
> + */
> +
> +#include <linux/init.h>
> +#include <linux/kernel.h>
> +#include <linux/module.h>
> +#include <linux/of_pci.h>
> +#include <linux/pci.h>
> +#include <linux/platform_device.h>
> +#include <linux/slab.h>
> +
> +#include "pcie-mobiveil.h"
> +
> +static int mobiveil_pcie_probe(struct platform_device *pdev)
> +{
> + struct mobiveil_pcie *pcie;
> + struct pci_host_bridge *bridge;
> + struct device *dev = &pdev->dev;
> +
> + bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
> + if (!bridge)
> + return -ENOMEM;
> +
> + pcie = pci_host_bridge_priv(bridge);
> + pcie->rp.bridge = bridge;
> +
> + pcie->pdev = pdev;
> +
> + return mobiveil_pcie_host_probe(pcie);
> +}
> +
> +static const struct of_device_id mobiveil_pcie_of_match[] = {
> + {.compatible = "mbvl,gpex40-pcie",},
> + {},
> +};
> +
> +MODULE_DEVICE_TABLE(of, mobiveil_pcie_of_match);
> +
> +static struct platform_driver mobiveil_pcie_driver = {
> + .probe = mobiveil_pcie_probe,
> + .driver = {
> + .name = "mobiveil-pcie",
> + .of_match_table = mobiveil_pcie_of_match,
> + .suppress_bind_attrs = true,
> + },
> +};
> +
> +builtin_platform_driver(mobiveil_pcie_driver);
> +
> +MODULE_LICENSE("GPL v2");
> +MODULE_DESCRIPTION("Mobiveil PCIe host controller driver");
> +MODULE_AUTHOR("Subrahmanya Lingappa <[email protected]>");
> diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.c b/drivers/pci/controller/mobiveil/pcie-mobiveil.c
> new file mode 100644
> index 000000000000..2773f823c9ea
> --- /dev/null
> +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.c
> @@ -0,0 +1,227 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * PCIe host controller driver for Mobiveil PCIe Host controller
> + *
> + * Copyright (c) 2018 Mobiveil Inc.
> + * Copyright 2019 NXP
> + *
> + * Author: Subrahmanya Lingappa <[email protected]>
> + * Recode: Hou Zhiqiang <[email protected]>
> + */
> +
> +#include <linux/delay.h>
> +#include <linux/init.h>
> +#include <linux/kernel.h>
> +#include <linux/pci.h>
> +#include <linux/platform_device.h>
> +
> +#include "pcie-mobiveil.h"
> +
> +/*
> + * mobiveil_pcie_sel_page - routine to access paged register
> + *
> + * Registers whose address greater than PAGED_ADDR_BNDRY (0xc00) are paged,
> + * for this scheme to work extracted higher 6 bits of the offset will be
> + * written to pg_sel field of PAB_CTRL register and rest of the lower 10
> + * bits enabled with PAGED_ADDR_BNDRY are used as offset of the register.
> + */
> +static void mobiveil_pcie_sel_page(struct mobiveil_pcie *pcie, u8 pg_idx)
> +{
> + u32 val;
> +
> + val = readl(pcie->csr_axi_slave_base + PAB_CTRL);
> + val &= ~(PAGE_SEL_MASK << PAGE_SEL_SHIFT);
> + val |= (pg_idx & PAGE_SEL_MASK) << PAGE_SEL_SHIFT;
> +
> + writel(val, pcie->csr_axi_slave_base + PAB_CTRL);
> +}
> +
> +static void *mobiveil_pcie_comp_addr(struct mobiveil_pcie *pcie, u32 off)
> +{
> + if (off < PAGED_ADDR_BNDRY) {
> + /* For directly accessed registers, clear the pg_sel field */
> + mobiveil_pcie_sel_page(pcie, 0);
> + return pcie->csr_axi_slave_base + off;
> + }
> +
> + mobiveil_pcie_sel_page(pcie, OFFSET_TO_PAGE_IDX(off));
> + return pcie->csr_axi_slave_base + OFFSET_TO_PAGE_ADDR(off);
> +}
> +
> +static int mobiveil_pcie_read(void __iomem *addr, int size, u32 *val)
> +{
> + if ((uintptr_t)addr & (size - 1)) {
> + *val = 0;
> + return PCIBIOS_BAD_REGISTER_NUMBER;
> + }
> +
> + switch (size) {
> + case 4:
> + *val = readl(addr);
> + break;
> + case 2:
> + *val = readw(addr);
> + break;
> + case 1:
> + *val = readb(addr);
> + break;
> + default:
> + *val = 0;
> + return PCIBIOS_BAD_REGISTER_NUMBER;
> + }
> +
> + return PCIBIOS_SUCCESSFUL;
> +}
> +
> +static int mobiveil_pcie_write(void __iomem *addr, int size, u32 val)
> +{
> + if ((uintptr_t)addr & (size - 1))
> + return PCIBIOS_BAD_REGISTER_NUMBER;
> +
> + switch (size) {
> + case 4:
> + writel(val, addr);
> + break;
> + case 2:
> + writew(val, addr);
> + break;
> + case 1:
> + writeb(val, addr);
> + break;
> + default:
> + return PCIBIOS_BAD_REGISTER_NUMBER;
> + }
> +
> + return PCIBIOS_SUCCESSFUL;
> +}
> +
> +u32 mobiveil_csr_read(struct mobiveil_pcie *pcie, u32 off, size_t size)
> +{
> + void *addr;
> + u32 val;
> + int ret;
> +
> + addr = mobiveil_pcie_comp_addr(pcie, off);
> +
> + ret = mobiveil_pcie_read(addr, size, &val);
> + if (ret)
> + dev_err(&pcie->pdev->dev, "read CSR address failed\n");
> +
> + return val;
> +}
> +
> +void mobiveil_csr_write(struct mobiveil_pcie *pcie, u32 val, u32 off,
> + size_t size)
> +{
> + void *addr;
> + int ret;
> +
> + addr = mobiveil_pcie_comp_addr(pcie, off);
> +
> + ret = mobiveil_pcie_write(addr, size, val);
> + if (ret)
> + dev_err(&pcie->pdev->dev, "write CSR address failed\n");
> +}
> +
> +bool mobiveil_pcie_link_up(struct mobiveil_pcie *pcie)
> +{
> + return (mobiveil_csr_readl(pcie, LTSSM_STATUS) &
> + LTSSM_STATUS_L0_MASK) == LTSSM_STATUS_L0;
> +}
> +
> +void program_ib_windows(struct mobiveil_pcie *pcie, int win_num,
> + u64 cpu_addr, u64 pci_addr, u32 type, u64 size)
> +{
> + u32 value;
> + u64 size64 = ~(size - 1);
> +
> + if (win_num >= pcie->ppio_wins) {
> + dev_err(&pcie->pdev->dev,
> + "ERROR: max inbound windows reached !\n");
> + return;
> + }
> +
> + value = mobiveil_csr_readl(pcie, PAB_PEX_AMAP_CTRL(win_num));
> + value &= ~(AMAP_CTRL_TYPE_MASK << AMAP_CTRL_TYPE_SHIFT | WIN_SIZE_MASK);
> + value |= type << AMAP_CTRL_TYPE_SHIFT | 1 << AMAP_CTRL_EN_SHIFT |
> + (lower_32_bits(size64) & WIN_SIZE_MASK);
> + mobiveil_csr_writel(pcie, value, PAB_PEX_AMAP_CTRL(win_num));
> +
> + mobiveil_csr_writel(pcie, upper_32_bits(size64),
> + PAB_EXT_PEX_AMAP_SIZEN(win_num));
> +
> + mobiveil_csr_writel(pcie, lower_32_bits(cpu_addr),
> + PAB_PEX_AMAP_AXI_WIN(win_num));
> + mobiveil_csr_writel(pcie, upper_32_bits(cpu_addr),
> + PAB_EXT_PEX_AMAP_AXI_WIN(win_num));
> +
> + mobiveil_csr_writel(pcie, lower_32_bits(pci_addr),
> + PAB_PEX_AMAP_PEX_WIN_L(win_num));
> + mobiveil_csr_writel(pcie, upper_32_bits(pci_addr),
> + PAB_PEX_AMAP_PEX_WIN_H(win_num));
> +
> + pcie->ib_wins_configured++;
> +}
> +
> +/*
> + * routine to program the outbound windows
> + */
> +void program_ob_windows(struct mobiveil_pcie *pcie, int win_num,
> + u64 cpu_addr, u64 pci_addr, u32 type, u64 size)
> +{
> + u32 value;
> + u64 size64 = ~(size - 1);
> +
> + if (win_num >= pcie->apio_wins) {
> + dev_err(&pcie->pdev->dev,
> + "ERROR: max outbound windows reached !\n");
> + return;
> + }
> +
> + /*
> + * program Enable Bit to 1, Type Bit to (00) base 2, AXI Window Size Bit
> + * to 4 KB in PAB_AXI_AMAP_CTRL register
> + */
> + value = mobiveil_csr_readl(pcie, PAB_AXI_AMAP_CTRL(win_num));
> + value &= ~(WIN_TYPE_MASK << WIN_TYPE_SHIFT | WIN_SIZE_MASK);
> + value |= 1 << WIN_ENABLE_SHIFT | type << WIN_TYPE_SHIFT |
> + (lower_32_bits(size64) & WIN_SIZE_MASK);
> + mobiveil_csr_writel(pcie, value, PAB_AXI_AMAP_CTRL(win_num));
> +
> + mobiveil_csr_writel(pcie, upper_32_bits(size64),
> + PAB_EXT_AXI_AMAP_SIZE(win_num));
> +
> + /*
> + * program AXI window base with appropriate value in
> + * PAB_AXI_AMAP_AXI_WIN0 register
> + */
> + mobiveil_csr_writel(pcie,
> + lower_32_bits(cpu_addr) & (~AXI_WINDOW_ALIGN_MASK),
> + PAB_AXI_AMAP_AXI_WIN(win_num));
> + mobiveil_csr_writel(pcie, upper_32_bits(cpu_addr),
> + PAB_EXT_AXI_AMAP_AXI_WIN(win_num));
> +
> + mobiveil_csr_writel(pcie, lower_32_bits(pci_addr),
> + PAB_AXI_AMAP_PEX_WIN_L(win_num));
> + mobiveil_csr_writel(pcie, upper_32_bits(pci_addr),
> + PAB_AXI_AMAP_PEX_WIN_H(win_num));
> +
> + pcie->ob_wins_configured++;
> +}
> +
> +int mobiveil_bringup_link(struct mobiveil_pcie *pcie)
> +{
> + int retries;
> +
> + /* check if the link is up or not */
> + for (retries = 0; retries < LINK_WAIT_MAX_RETRIES; retries++) {
> + if (mobiveil_pcie_link_up(pcie))
> + return 0;
> +
> + usleep_range(LINK_WAIT_MIN, LINK_WAIT_MAX);
> + }
> +
> + dev_err(&pcie->pdev->dev, "link never came up\n");
> +
> + return -ETIMEDOUT;
> +}
> diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.h b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> new file mode 100644
> index 000000000000..e3148078e9dd
> --- /dev/null
> +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> @@ -0,0 +1,178 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * PCIe host controller driver for Mobiveil PCIe Host controller
> + *
> + * Copyright (c) 2018 Mobiveil Inc.
> + * Copyright 2019 NXP
> + *
> + * Author: Subrahmanya Lingappa <[email protected]>
> + * Recode: Hou Zhiqiang <[email protected]>
> + */
> +
> +#ifndef _PCIE_MOBIVEIL_H
> +#define _PCIE_MOBIVEIL_H
> +
> +#include <linux/pci.h>
> +#include <linux/irq.h>
> +#include <linux/msi.h>
> +#include "../../pci.h"
> +
> +/* register offsets and bit positions */
> +
> +/*
> + * translation tables are grouped into windows, each window registers are
> + * grouped into blocks of 4 or 16 registers each
> + */
> +#define PAB_REG_BLOCK_SIZE 16
> +#define PAB_EXT_REG_BLOCK_SIZE 4
> +
> +#define PAB_REG_ADDR(offset, win) \
> + (offset + (win * PAB_REG_BLOCK_SIZE))
> +#define PAB_EXT_REG_ADDR(offset, win) \
> + (offset + (win * PAB_EXT_REG_BLOCK_SIZE))
> +
> +#define LTSSM_STATUS 0x0404
> +#define LTSSM_STATUS_L0_MASK 0x3f
> +#define LTSSM_STATUS_L0 0x2d
> +
> +#define PAB_CTRL 0x0808
> +#define AMBA_PIO_ENABLE_SHIFT 0
> +#define PEX_PIO_ENABLE_SHIFT 1
> +#define PAGE_SEL_SHIFT 13
> +#define PAGE_SEL_MASK 0x3f
> +#define PAGE_LO_MASK 0x3ff
> +#define PAGE_SEL_OFFSET_SHIFT 10
> +
> +#define PAB_AXI_PIO_CTRL 0x0840
> +#define APIO_EN_MASK 0xf
> +
> +#define PAB_PEX_PIO_CTRL 0x08c0
> +#define PIO_ENABLE_SHIFT 0
> +
> +#define PAB_INTP_AMBA_MISC_ENB 0x0b0c
> +#define PAB_INTP_AMBA_MISC_STAT 0x0b1c
> +#define PAB_INTP_INTX_MASK 0x01e0
> +#define PAB_INTP_MSI_MASK 0x8
> +
> +#define PAB_AXI_AMAP_CTRL(win) PAB_REG_ADDR(0x0ba0, win)
> +#define WIN_ENABLE_SHIFT 0
> +#define WIN_TYPE_SHIFT 1
> +#define WIN_TYPE_MASK 0x3
> +#define WIN_SIZE_MASK 0xfffffc00
> +
> +#define PAB_EXT_AXI_AMAP_SIZE(win) PAB_EXT_REG_ADDR(0xbaf0, win)
> +
> +#define PAB_EXT_AXI_AMAP_AXI_WIN(win) PAB_EXT_REG_ADDR(0x80a0, win)
> +#define PAB_AXI_AMAP_AXI_WIN(win) PAB_REG_ADDR(0x0ba4, win)
> +#define AXI_WINDOW_ALIGN_MASK 3
> +
> +#define PAB_AXI_AMAP_PEX_WIN_L(win) PAB_REG_ADDR(0x0ba8, win)
> +#define PAB_BUS_SHIFT 24
> +#define PAB_DEVICE_SHIFT 19
> +#define PAB_FUNCTION_SHIFT 16
> +
> +#define PAB_AXI_AMAP_PEX_WIN_H(win) PAB_REG_ADDR(0x0bac, win)
> +#define PAB_INTP_AXI_PIO_CLASS 0x474
> +
> +#define PAB_PEX_AMAP_CTRL(win) PAB_REG_ADDR(0x4ba0, win)
> +#define AMAP_CTRL_EN_SHIFT 0
> +#define AMAP_CTRL_TYPE_SHIFT 1
> +#define AMAP_CTRL_TYPE_MASK 3
> +
> +#define PAB_EXT_PEX_AMAP_SIZEN(win) PAB_EXT_REG_ADDR(0xbef0, win)
> +#define PAB_EXT_PEX_AMAP_AXI_WIN(win) PAB_EXT_REG_ADDR(0xb4a0, win)
> +#define PAB_PEX_AMAP_AXI_WIN(win) PAB_REG_ADDR(0x4ba4, win)
> +#define PAB_PEX_AMAP_PEX_WIN_L(win) PAB_REG_ADDR(0x4ba8, win)
> +#define PAB_PEX_AMAP_PEX_WIN_H(win) PAB_REG_ADDR(0x4bac, win)
> +
> +/* starting offset of INTX bits in status register */
> +#define PAB_INTX_START 5
> +
> +/* supported number of MSI interrupts */
> +#define PCI_NUM_MSI 16
> +
> +/* MSI registers */
> +#define MSI_BASE_LO_OFFSET 0x04
> +#define MSI_BASE_HI_OFFSET 0x08
> +#define MSI_SIZE_OFFSET 0x0c
> +#define MSI_ENABLE_OFFSET 0x14
> +#define MSI_STATUS_OFFSET 0x18
> +#define MSI_DATA_OFFSET 0x20
> +#define MSI_ADDR_L_OFFSET 0x24
> +#define MSI_ADDR_H_OFFSET 0x28
> +
> +/* outbound and inbound window definitions */
> +#define WIN_NUM_0 0
> +#define WIN_NUM_1 1
> +#define CFG_WINDOW_TYPE 0
> +#define IO_WINDOW_TYPE 1
> +#define MEM_WINDOW_TYPE 2
> +#define IB_WIN_SIZE ((u64)256 * 1024 * 1024 * 1024)
> +#define MAX_PIO_WINDOWS 8
> +
> +/* Parameters for the waiting for link up routine */
> +#define LINK_WAIT_MAX_RETRIES 10
> +#define LINK_WAIT_MIN 90000
> +#define LINK_WAIT_MAX 100000
> +
> +#define PAGED_ADDR_BNDRY 0xc00
> +#define OFFSET_TO_PAGE_ADDR(off) \
> + ((off & PAGE_LO_MASK) | PAGED_ADDR_BNDRY)
> +#define OFFSET_TO_PAGE_IDX(off) \
> + ((off >> PAGE_SEL_OFFSET_SHIFT) & PAGE_SEL_MASK)
> +
> +struct mobiveil_msi { /* MSI information */
> + struct mutex lock; /* protect bitmap variable */
> + struct irq_domain *msi_domain;
> + struct irq_domain *dev_domain;
> + phys_addr_t msi_pages_phys;
> + int num_of_vectors;
> + DECLARE_BITMAP(msi_irq_in_use, PCI_NUM_MSI);
> +};
> +
> +struct root_port {
> + char root_bus_nr;
> + void __iomem *config_axi_slave_base; /* endpoint config base */
> + struct resource *ob_io_res;
> + int irq;
> + raw_spinlock_t intx_mask_lock;
> + struct irq_domain *intx_domain;
> + struct mobiveil_msi msi;
> + struct pci_host_bridge *bridge;
> +};
> +
> +struct mobiveil_pcie {
> + struct platform_device *pdev;
> + void __iomem *csr_axi_slave_base; /* root port config base */
> + void __iomem *apb_csr_base; /* MSI register base */
> + phys_addr_t pcie_reg_base; /* Physical PCIe Controller Base */
> + int apio_wins;
> + int ppio_wins;
> + int ob_wins_configured; /* configured outbound windows */
> + int ib_wins_configured; /* configured inbound windows */
> + struct root_port rp;
> +};
> +
> +int mobiveil_pcie_host_probe(struct mobiveil_pcie *pcie);
> +bool mobiveil_pcie_link_up(struct mobiveil_pcie *pcie);
> +int mobiveil_bringup_link(struct mobiveil_pcie *pcie);
> +void program_ob_windows(struct mobiveil_pcie *pcie, int win_num, u64 cpu_addr,
> + u64 pci_addr, u32 type, u64 size);
> +void program_ib_windows(struct mobiveil_pcie *pcie, int win_num, u64 cpu_addr,
> + u64 pci_addr, u32 type, u64 size);
> +u32 mobiveil_csr_read(struct mobiveil_pcie *pcie, u32 off, size_t size);
> +void mobiveil_csr_write(struct mobiveil_pcie *pcie, u32 val, u32 off,
> + size_t size);
> +
> +static inline u32 mobiveil_csr_readl(struct mobiveil_pcie *pcie, u32 off)
> +{
> + return mobiveil_csr_read(pcie, off, 0x4);
> +}
> +
> +static inline void mobiveil_csr_writel(struct mobiveil_pcie *pcie, u32 val,
> + u32 off)
> +{
> + mobiveil_csr_write(pcie, val, off, 0x4);
> +}
> +
> +#endif /* _PCIE_MOBIVEIL_H */
> --
> 2.17.1
>
On Wed, Nov 20, 2019 at 03:45:50AM +0000, Z.q. Hou wrote:
> From: Hou Zhiqiang <[email protected]>
>
> The Mobiveil GPEX internal MSI/INTx controller may not be used
> by other platforms in which the Mobiveil GPEX is integrated.
> This patch is to allow these platforms to implement their
> specific interrupt initialization.
>
> Signed-off-by: Hou Zhiqiang <[email protected]>
> ---
> V9:
> - New patch splited from the #1 of V8 patches to make it easy to review.
>
> drivers/pci/controller/mobiveil/pcie-mobiveil-host.c | 3 +++
> drivers/pci/controller/mobiveil/pcie-mobiveil.h | 7 +++++++
> 2 files changed, 10 insertions(+)
>
> diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c b/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
> index 2cc424e78d33..3cd93df6fe6e 100644
> --- a/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
> +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
> @@ -507,6 +507,9 @@ static int mobiveil_pcie_interrupt_init(struct mobiveil_pcie *pcie)
> struct resource *res;
> int ret;
>
> + if (rp->ops->interrupt_init)
> + return rp->ops->interrupt_init(pcie);
> +
This may be cleaner if you have a helper function named
"mobiveil_pcie_interrupt_init" where it either calls interrupt_init if present
or calls this current function renamed to "mobiveil_pcie_integrated_interrupt_init"
or similar.
A bit like the DWC dw_pcie_rd_own_conf function.
Thanks,
Andrew Murray
> /* map MSI config resource */
> res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "apb_csr");
> pcie->apb_csr_base = devm_pci_remap_cfg_resource(dev, res);
> diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.h b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> index e3148078e9dd..18d85806a7fc 100644
> --- a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> @@ -130,10 +130,17 @@ struct mobiveil_msi { /* MSI information */
> DECLARE_BITMAP(msi_irq_in_use, PCI_NUM_MSI);
> };
>
> +struct mobiveil_pcie;
> +
> +struct mobiveil_rp_ops {
> + int (*interrupt_init)(struct mobiveil_pcie *pcie);
> +};
> +
> struct root_port {
> char root_bus_nr;
> void __iomem *config_axi_slave_base; /* endpoint config base */
> struct resource *ob_io_res;
> + struct mobiveil_rp_ops *ops;
> int irq;
> raw_spinlock_t intx_mask_lock;
> struct irq_domain *intx_domain;
> --
> 2.17.1
>
On Wed, Nov 20, 2019 at 03:45:57AM +0000, Z.q. Hou wrote:
> From: Hou Zhiqiang <[email protected]>
>
> The platforms, in which the Mobiveil GPEX is integrated,
> may have their specific mechanism to check link up status.
> This patch is to enable these platforms to implement theirs.
>
> Signed-off-by: Hou Zhiqiang <[email protected]>
> ---
> V9:
> - New patch splited from the #1 of V8 patches to make it easy to review.
>
> drivers/pci/controller/mobiveil/pcie-mobiveil.c | 3 +++
> drivers/pci/controller/mobiveil/pcie-mobiveil.h | 5 +++++
> 2 files changed, 8 insertions(+)
>
> diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.c b/drivers/pci/controller/mobiveil/pcie-mobiveil.c
> index 2773f823c9ea..b9ed2d95641c 100644
> --- a/drivers/pci/controller/mobiveil/pcie-mobiveil.c
> +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.c
> @@ -125,6 +125,9 @@ void mobiveil_csr_write(struct mobiveil_pcie *pcie, u32 val, u32 off,
>
> bool mobiveil_pcie_link_up(struct mobiveil_pcie *pcie)
> {
> + if (pcie->ops->link_up)
> + return pcie->ops->link_up(pcie);
> +
> return (mobiveil_csr_readl(pcie, LTSSM_STATUS) &
> LTSSM_STATUS_L0_MASK) == LTSSM_STATUS_L0;
On the previous patch I suggested that we don't mix up the link_up logic
with the logic that decides which function to call. In this case the link_up
logic is trivial. So this is probably OK.
Reviewed-by: Andrew Murray <[email protected]>
> }
> diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.h b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> index 18d85806a7fc..95d2e7c809b8 100644
> --- a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> @@ -148,6 +148,10 @@ struct root_port {
> struct pci_host_bridge *bridge;
> };
>
> +struct mobiveil_pab_ops {
> + int (*link_up)(struct mobiveil_pcie *pcie);
> +};
> +
> struct mobiveil_pcie {
> struct platform_device *pdev;
> void __iomem *csr_axi_slave_base; /* root port config base */
> @@ -157,6 +161,7 @@ struct mobiveil_pcie {
> int ppio_wins;
> int ob_wins_configured; /* configured outbound windows */
> int ib_wins_configured; /* configured inbound windows */
> + const struct mobiveil_pab_ops *ops;
> struct root_port rp;
> };
>
> --
> 2.17.1
>
On Wed, Nov 20, 2019 at 03:46:03AM +0000, Z.q. Hou wrote:
> From: Hou Zhiqiang <[email protected]>
>
> Make the mobiveil_host_init() function can be used to re-init
Perhaps reword to "Allow the mobiveil_host_init() function to be
used to ...
> host controller's PAB and GPEX CSR register block, as NXP
> integrated Mobiveil IP has to reset and then re-init the PAB
> and GPEX CSR registers upon hot-reset.
>
> Signed-off-by: Hou Zhiqiang <[email protected]>
> Reviewed-by: Subrahmanya Lingappa <[email protected]>
> ---
> V9:
> - No change
>
> .../controller/mobiveil/pcie-mobiveil-host.c | 19 ++++++++++++-------
> .../pci/controller/mobiveil/pcie-mobiveil.h | 1 +
> 2 files changed, 13 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c b/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
> index 3cd93df6fe6e..9bc3da036720 100644
> --- a/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
> +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
> @@ -221,18 +221,23 @@ static void mobiveil_pcie_enable_msi(struct mobiveil_pcie *pcie)
> writel_relaxed(1, pcie->apb_csr_base + MSI_ENABLE_OFFSET);
> }
>
> -static int mobiveil_host_init(struct mobiveil_pcie *pcie)
> +int mobiveil_host_init(struct mobiveil_pcie *pcie, bool reinit)
> {
> struct root_port *rp = &pcie->rp;
> struct pci_host_bridge *bridge = rp->bridge;
> u32 value, pab_ctrl, type;
> struct resource_entry *win;
>
> - /* setup bus numbers */
> - value = mobiveil_csr_readl(pcie, PCI_PRIMARY_BUS);
> - value &= 0xff000000;
> - value |= 0x00ff0100;
> - mobiveil_csr_writel(pcie, value, PCI_PRIMARY_BUS);
> + pcie->ib_wins_configured = 0;
> + pcie->ob_wins_configured = 0;
This works so long as the number of bridge->windows never reduces. I
think this assumption holds true.
Thanks,
Andrew Murray
> +
> + if (!reinit) {
> + /* setup bus numbers */
> + value = mobiveil_csr_readl(pcie, PCI_PRIMARY_BUS);
> + value &= 0xff000000;
> + value |= 0x00ff0100;
> + mobiveil_csr_writel(pcie, value, PCI_PRIMARY_BUS);
> + }
>
> /*
> * program Bus Master Enable Bit in Command Register in PAB Config
> @@ -569,7 +574,7 @@ int mobiveil_pcie_host_probe(struct mobiveil_pcie *pcie)
> * configure all inbound and outbound windows and prepare the RC for
> * config access
> */
> - ret = mobiveil_host_init(pcie);
> + ret = mobiveil_host_init(pcie, false);
> if (ret) {
> dev_err(dev, "Failed to initialize host\n");
> return ret;
> diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.h b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> index 95d2e7c809b8..37116c2a19fe 100644
> --- a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> @@ -166,6 +166,7 @@ struct mobiveil_pcie {
> };
>
> int mobiveil_pcie_host_probe(struct mobiveil_pcie *pcie);
> +int mobiveil_host_init(struct mobiveil_pcie *pcie, bool reinit);
> bool mobiveil_pcie_link_up(struct mobiveil_pcie *pcie);
> int mobiveil_bringup_link(struct mobiveil_pcie *pcie);
> void program_ob_windows(struct mobiveil_pcie *pcie, int win_num, u64 cpu_addr,
> --
> 2.17.1
>
On Wed, Nov 20, 2019 at 03:46:10AM +0000, Z.q. Hou wrote:
> From: Hou Zhiqiang <[email protected]>
>
> There are some 8-bit and 16-bit registers in PCIe configuration
> space, so add these accessors accordingly.
>
> Signed-off-by: Hou Zhiqiang <[email protected]>
> Reviewed-by: Minghuan Lian <[email protected]>
> Reviewed-by: Subrahmanya Lingappa <[email protected]>
> ---
> V9:
> - No change
>
> .../pci/controller/mobiveil/pcie-mobiveil.h | 23 +++++++++++++++++++
> 1 file changed, 23 insertions(+)
>
> diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.h b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> index 37116c2a19fe..750a7fd95bc1 100644
> --- a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> @@ -182,10 +182,33 @@ static inline u32 mobiveil_csr_readl(struct mobiveil_pcie *pcie, u32 off)
> return mobiveil_csr_read(pcie, off, 0x4);
> }
>
> +static inline u32 mobiveil_csr_readw(struct mobiveil_pcie *pcie, u32 off)
> +{
> + return mobiveil_csr_read(pcie, off, 0x2);
> +}
> +
> +static inline u32 mobiveil_csr_readb(struct mobiveil_pcie *pcie, u32 off)
> +{
> + return mobiveil_csr_read(pcie, off, 0x1);
> +}
Do you think the above two return types should reflect the size of the access?
Thanks,
Andrew Murray
> +
> +
> static inline void mobiveil_csr_writel(struct mobiveil_pcie *pcie, u32 val,
> u32 off)
> {
> mobiveil_csr_write(pcie, val, off, 0x4);
> }
>
> +static inline void mobiveil_csr_writew(struct mobiveil_pcie *pcie, u32 val,
> + u32 off)
> +{
> + mobiveil_csr_write(pcie, val, off, 0x2);
> +}
> +
> +static inline void mobiveil_csr_writeb(struct mobiveil_pcie *pcie, u32 val,
> + u32 off)
> +{
> + mobiveil_csr_write(pcie, val, off, 0x1);
> +}
> +
> #endif /* _PCIE_MOBIVEIL_H */
> --
> 2.17.1
>
On Wed, Nov 20, 2019 at 03:46:23AM +0000, Z.q. Hou wrote:
> From: Hou Zhiqiang <[email protected]>
>
> This PCIe controller is based on the Mobiveil GPEX IP, which is
> compatible with the PCI Express™ Base Specification, Revision 4.0.
>
> Signed-off-by: Hou Zhiqiang <[email protected]>
> Reviewed-by: Minghuan Lian <[email protected]>
> ---
> drivers/pci/controller/mobiveil/Kconfig | 10 +
> drivers/pci/controller/mobiveil/Makefile | 1 +
> .../mobiveil/pcie-layerscape-gen4.c | 274 ++++++++++++++++++
> .../pci/controller/mobiveil/pcie-mobiveil.h | 16 +-
> 4 files changed, 299 insertions(+), 2 deletions(-)
> create mode 100644 drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c
>
> diff --git a/drivers/pci/controller/mobiveil/Kconfig b/drivers/pci/controller/mobiveil/Kconfig
> index 64343c07bfed..c823be8dab1c 100644
> --- a/drivers/pci/controller/mobiveil/Kconfig
> +++ b/drivers/pci/controller/mobiveil/Kconfig
> @@ -21,4 +21,14 @@ config PCIE_MOBIVEIL_PLAT
> Soft IP. It has up to 8 outbound and inbound windows
> for address translation and it is a PCIe Gen4 IP.
>
> +config PCIE_LAYERSCAPE_GEN4
> + bool "Freescale Layerscape PCIe Gen4 controller"
> + depends on PCI
> + depends on OF && (ARM64 || ARCH_LAYERSCAPE)
> + depends on PCI_MSI_IRQ_DOMAIN
> + select PCIE_MOBIVEIL_HOST
> + help
> + Say Y here if you want PCIe Gen4 controller support on
> + Layerscape SoCs. The PCIe controller can work in RC or
> + EP mode according to RCW[HOST_AGT_PEX] setting.
I think you can remove the last sentence - it doesn't give any value
to users of KConfig.
> endmenu
> diff --git a/drivers/pci/controller/mobiveil/Makefile b/drivers/pci/controller/mobiveil/Makefile
> index 9fb6d1c6504d..99d879de32d6 100644
> --- a/drivers/pci/controller/mobiveil/Makefile
> +++ b/drivers/pci/controller/mobiveil/Makefile
> @@ -2,3 +2,4 @@
> obj-$(CONFIG_PCIE_MOBIVEIL) += pcie-mobiveil.o
> obj-$(CONFIG_PCIE_MOBIVEIL_HOST) += pcie-mobiveil-host.o
> obj-$(CONFIG_PCIE_MOBIVEIL_PLAT) += pcie-mobiveil-plat.o
> +obj-$(CONFIG_PCIE_LAYERSCAPE_GEN4) += pcie-layerscape-gen4.o
> diff --git a/drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c b/drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c
> new file mode 100644
> index 000000000000..6c0d3e2532db
> --- /dev/null
> +++ b/drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c
> @@ -0,0 +1,274 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * PCIe Gen4 host controller driver for NXP Layerscape SoCs
> + *
> + * Copyright 2019 NXP
> + *
> + * Author: Zhiqiang Hou <[email protected]>
> + */
> +
> +#include <linux/kernel.h>
> +#include <linux/interrupt.h>
> +#include <linux/init.h>
> +#include <linux/of_pci.h>
> +#include <linux/of_platform.h>
> +#include <linux/of_irq.h>
> +#include <linux/of_address.h>
> +#include <linux/pci.h>
> +#include <linux/platform_device.h>
> +#include <linux/resource.h>
> +#include <linux/mfd/syscon.h>
> +#include <linux/regmap.h>
> +
> +#include "pcie-mobiveil.h"
> +
> +/* LUT and PF control registers */
> +#define PCIE_LUT_OFF 0x80000
> +#define PCIE_PF_OFF 0xc0000
> +#define PCIE_PF_INT_STAT 0x18
> +#define PF_INT_STAT_PABRST BIT(31)
> +
> +#define PCIE_PF_DBG 0x7fc
> +#define PF_DBG_LTSSM_MASK 0x3f
> +#define PF_DBG_LTSSM_L0 0x2d /* L0 state */
> +#define PF_DBG_WE BIT(31)
> +#define PF_DBG_PABR BIT(27)
> +
> +#define to_ls_pcie_g4(x) platform_get_drvdata((x)->pdev)
> +
> +struct ls_pcie_g4 {
> + struct mobiveil_pcie pci;
> + struct delayed_work dwork;
> + int irq;
> +};
> +
> +static inline u32 ls_pcie_g4_lut_readl(struct ls_pcie_g4 *pcie, u32 off)
> +{
> + return ioread32(pcie->pci.csr_axi_slave_base + PCIE_LUT_OFF + off);
> +}
> +
> +static inline void ls_pcie_g4_lut_writel(struct ls_pcie_g4 *pcie,
> + u32 off, u32 val)
> +{
> + iowrite32(val, pcie->pci.csr_axi_slave_base + PCIE_LUT_OFF + off);
> +}
> +
> +static inline u32 ls_pcie_g4_pf_readl(struct ls_pcie_g4 *pcie, u32 off)
> +{
> + return ioread32(pcie->pci.csr_axi_slave_base + PCIE_PF_OFF + off);
> +}
> +
> +static inline void ls_pcie_g4_pf_writel(struct ls_pcie_g4 *pcie,
> + u32 off, u32 val)
> +{
> + iowrite32(val, pcie->pci.csr_axi_slave_base + PCIE_PF_OFF + off);
> +}
> +
> +static bool ls_pcie_g4_is_bridge(struct ls_pcie_g4 *pcie)
> +{
> + struct mobiveil_pcie *mv_pci = &pcie->pci;
> + u32 header_type;
> +
> + header_type = mobiveil_csr_readb(mv_pci, PCI_HEADER_TYPE);
> + header_type &= 0x7f;
> +
> + return header_type == PCI_HEADER_TYPE_BRIDGE;
> +}
> +
> +static int ls_pcie_g4_link_up(struct mobiveil_pcie *pci)
> +{
> + struct ls_pcie_g4 *pcie = to_ls_pcie_g4(pci);
> + u32 state;
> +
> + state = ls_pcie_g4_pf_readl(pcie, PCIE_PF_DBG);
> + state = state & PF_DBG_LTSSM_MASK;
> +
> + if (state == PF_DBG_LTSSM_L0)
> + return 1;
> +
> + return 0;
> +}
> +
> +static void ls_pcie_g4_disable_interrupt(struct ls_pcie_g4 *pcie)
> +{
> + struct mobiveil_pcie *mv_pci = &pcie->pci;
> +
> + mobiveil_csr_writel(mv_pci, 0, PAB_INTP_AMBA_MISC_ENB);
> +}
> +
> +static void ls_pcie_g4_enable_interrupt(struct ls_pcie_g4 *pcie)
> +{
> + struct mobiveil_pcie *mv_pci = &pcie->pci;
> + u32 val;
> +
> + /* Clear the interrupt status */
> + mobiveil_csr_writel(mv_pci, 0xffffffff, PAB_INTP_AMBA_MISC_STAT);
> +
> + val = PAB_INTP_INTX_MASK | PAB_INTP_MSI | PAB_INTP_RESET |
> + PAB_INTP_PCIE_UE | PAB_INTP_IE_PMREDI | PAB_INTP_IE_EC;
> + mobiveil_csr_writel(mv_pci, val, PAB_INTP_AMBA_MISC_ENB);
> +}
> +
> +static void ls_pcie_g4_reinit_hw(struct ls_pcie_g4 *pcie)
> +{
> + struct mobiveil_pcie *mv_pci = &pcie->pci;
> + struct device *dev = &mv_pci->pdev->dev;
> + u32 val, act_stat;
> + int to = 100;
> +
> + /* Poll for pab_csb_reset to set and PAB activity to clear */
> + do {
> + usleep_range(10, 15);
> + val = ls_pcie_g4_pf_readl(pcie, PCIE_PF_INT_STAT);
> + act_stat = mobiveil_csr_readl(mv_pci, PAB_ACTIVITY_STAT);
> + } while (((val & PF_INT_STAT_PABRST) == 0 || act_stat) && to--);
> + if (to < 0) {
> + dev_err(dev, "Poll PABRST&PABACT timeout\n");
> + return;
If a timeout happens here - the caller has no idea this has happened
and yet the following work doesn't get done. Isn't this a problem?
> + }
> +
> + /* clear PEX_RESET bit in PEX_PF0_DBG register */
> + val = ls_pcie_g4_pf_readl(pcie, PCIE_PF_DBG);
> + val |= PF_DBG_WE;
> + ls_pcie_g4_pf_writel(pcie, PCIE_PF_DBG, val);
> +
> + val = ls_pcie_g4_pf_readl(pcie, PCIE_PF_DBG);
> + val |= PF_DBG_PABR;
> + ls_pcie_g4_pf_writel(pcie, PCIE_PF_DBG, val);
> +
> + val = ls_pcie_g4_pf_readl(pcie, PCIE_PF_DBG);
> + val &= ~PF_DBG_WE;
> + ls_pcie_g4_pf_writel(pcie, PCIE_PF_DBG, val);
> +
> + mobiveil_host_init(mv_pci, true);
Can mobiveil_host_init fail?
> +
> + to = 100;
> + while (!ls_pcie_g4_link_up(mv_pci) && to--)
> + usleep_range(200, 250);
> + if (to < 0)
> + dev_err(dev, "PCIe link training timeout\n");
> +}
> +
> +static irqreturn_t ls_pcie_g4_isr(int irq, void *dev_id)
> +{
> + struct ls_pcie_g4 *pcie = (struct ls_pcie_g4 *)dev_id;
> + struct mobiveil_pcie *mv_pci = &pcie->pci;
> + u32 val;
> +
> + val = mobiveil_csr_readl(mv_pci, PAB_INTP_AMBA_MISC_STAT);
> + if (!val)
> + return IRQ_NONE;
> +
> + if (val & PAB_INTP_RESET) {
Can you explain why this is needed (perhaps also in the cover letter)?
> + ls_pcie_g4_disable_interrupt(pcie);
> + schedule_delayed_work(&pcie->dwork, msecs_to_jiffies(1));
> + }
> +
> + mobiveil_csr_writel(mv_pci, val, PAB_INTP_AMBA_MISC_STAT);
> +
> + return IRQ_HANDLED;
> +}
> +
> +static int ls_pcie_g4_interrupt_init(struct mobiveil_pcie *mv_pci)
> +{
> + struct ls_pcie_g4 *pcie = to_ls_pcie_g4(mv_pci);
> + struct platform_device *pdev = mv_pci->pdev;
> + struct device *dev = &pdev->dev;
> + int ret;
> +
> + pcie->irq = platform_get_irq_byname(pdev, "intr");
> + if (pcie->irq < 0) {
> + dev_err(dev, "Can't get 'intr' IRQ, errno = %d\n", pcie->irq);
> + return pcie->irq;
> + }
> + ret = devm_request_irq(dev, pcie->irq, ls_pcie_g4_isr,
> + IRQF_SHARED, pdev->name, pcie);
> + if (ret) {
> + dev_err(dev, "Can't register PCIe IRQ, errno = %d\n", ret);
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +static void ls_pcie_g4_reset(struct work_struct *work)
> +{
> + struct delayed_work *dwork = container_of(work, struct delayed_work,
> + work);
> + struct ls_pcie_g4 *pcie = container_of(dwork, struct ls_pcie_g4, dwork);
> + struct mobiveil_pcie *mv_pci = &pcie->pci;
> + u16 ctrl;
> +
> + ctrl = mobiveil_csr_readw(mv_pci, PCI_BRIDGE_CONTROL);
> + ctrl &= ~PCI_BRIDGE_CTL_BUS_RESET;
> + mobiveil_csr_writew(mv_pci, ctrl, PCI_BRIDGE_CONTROL);
> + ls_pcie_g4_reinit_hw(pcie);
> + ls_pcie_g4_enable_interrupt(pcie);
> +}
> +
> +static struct mobiveil_rp_ops ls_pcie_g4_rp_ops = {
> + .interrupt_init = ls_pcie_g4_interrupt_init,
> +};
> +
> +static const struct mobiveil_pab_ops ls_pcie_g4_pab_ops = {
> + .link_up = ls_pcie_g4_link_up,
> +};
> +
> +static int __init ls_pcie_g4_probe(struct platform_device *pdev)
> +{
> + struct device *dev = &pdev->dev;
> + struct pci_host_bridge *bridge;
> + struct mobiveil_pcie *mv_pci;
> + struct ls_pcie_g4 *pcie;
> + struct device_node *np = dev->of_node;
> + int ret;
> +
> + if (!of_parse_phandle(np, "msi-parent", 0)) {
> + dev_err(dev, "Failed to find msi-parent\n");
> + return -EINVAL;
> + }
> +
> + bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
> + if (!bridge)
> + return -ENOMEM;
> +
> + pcie = pci_host_bridge_priv(bridge);
> + mv_pci = &pcie->pci;
> +
> + mv_pci->pdev = pdev;
> + mv_pci->ops = &ls_pcie_g4_pab_ops;
> + mv_pci->rp.ops = &ls_pcie_g4_rp_ops;
> + mv_pci->rp.bridge = bridge;
> +
> + platform_set_drvdata(pdev, pcie);
> +
> + INIT_DELAYED_WORK(&pcie->dwork, ls_pcie_g4_reset);
> +
> + ret = mobiveil_pcie_host_probe(mv_pci);
> + if (ret) {
> + dev_err(dev, "Fail to probe\n");
> + return ret;
> + }
> +
> + if (!ls_pcie_g4_is_bridge(pcie))
Is this a check that could apply to all host bridge drivers and thus live
in mobiveil_pcie_host_probe?
> + return -ENODEV;
> +
> + ls_pcie_g4_enable_interrupt(pcie);
Is there an issue here in that we enable interrupts *after* telling the kernel
about our controller? (Same applies for bailing if the IP isn't a bridge).
Thanks,
Andrew Murray
> +
> + return 0;
> +}
> +
> +static const struct of_device_id ls_pcie_g4_of_match[] = {
> + { .compatible = "fsl,lx2160a-pcie", },
> + { },
> +};
> +
> +static struct platform_driver ls_pcie_g4_driver = {
> + .driver = {
> + .name = "layerscape-pcie-gen4",
> + .of_match_table = ls_pcie_g4_of_match,
> + .suppress_bind_attrs = true,
> + },
> +};
> +
> +builtin_platform_driver_probe(ls_pcie_g4_driver, ls_pcie_g4_probe);
> diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.h b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> index 750a7fd95bc1..c57a68d2bac4 100644
> --- a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> @@ -43,6 +43,8 @@
> #define PAGE_LO_MASK 0x3ff
> #define PAGE_SEL_OFFSET_SHIFT 10
>
> +#define PAB_ACTIVITY_STAT 0x81c
> +
> #define PAB_AXI_PIO_CTRL 0x0840
> #define APIO_EN_MASK 0xf
>
> @@ -51,8 +53,18 @@
>
> #define PAB_INTP_AMBA_MISC_ENB 0x0b0c
> #define PAB_INTP_AMBA_MISC_STAT 0x0b1c
> -#define PAB_INTP_INTX_MASK 0x01e0
> -#define PAB_INTP_MSI_MASK 0x8
> +#define PAB_INTP_RESET BIT(1)
> +#define PAB_INTP_MSI BIT(3)
> +#define PAB_INTP_INTA BIT(5)
> +#define PAB_INTP_INTB BIT(6)
> +#define PAB_INTP_INTC BIT(7)
> +#define PAB_INTP_INTD BIT(8)
> +#define PAB_INTP_PCIE_UE BIT(9)
> +#define PAB_INTP_IE_PMREDI BIT(29)
> +#define PAB_INTP_IE_EC BIT(30)
> +#define PAB_INTP_MSI_MASK PAB_INTP_MSI
> +#define PAB_INTP_INTX_MASK (PAB_INTP_INTA | PAB_INTP_INTB |\
> + PAB_INTP_INTC | PAB_INTP_INTD)
>
> #define PAB_AXI_AMAP_CTRL(win) PAB_REG_ADDR(0x0ba0, win)
> #define WIN_ENABLE_SHIFT 0
> --
> 2.17.1
>
Hi Olof,
Thanks a lot for your comments!
And sorry for my delay respond!
> -----Original Message-----
> From: Olof Johansson <[email protected]>
> Sent: 2020年1月11日 1:06
> To: Lorenzo Pieralisi <[email protected]>
> Cc: Z.q. Hou <[email protected]>; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; Leo Li <[email protected]>;
> [email protected]; [email protected]; [email protected];
> Mingkai Hu <[email protected]>; M.h. Lian <[email protected]>;
> Xiaowei Bao <[email protected]>
> Subject: Re: [PATCHv9 00/12] PCI: Recode Mobiveil driver and add PCIe Gen4
> driver for NXP Layerscape SoCs
>
> On Fri, Jan 10, 2020 at 7:33 AM Lorenzo Pieralisi <[email protected]>
> wrote:
> >
> > On Tue, Dec 17, 2019 at 02:50:15AM +0000, Z.q. Hou wrote:
> > > Hi Lorenzo,
> > >
> > > The v9 patches have addressed the comments from Andrew, and it has
> > > been dried about 1 month, can you help to apply them?
> >
> > We shall have a look beginning of next week, sorry for the delay in
> > getting back to you.
>
> Note that the patch set no longer applies since the refactorings conflict with
> new development by others.
>
> Zhiqiang, can you rebase and post a new version of the patch set?
Yes, I will rebase the patches to the latest code base.
Thanks,
Zhiqiang
>
> -Olof
Hi Andrew,
Thanks a lot for your comments!
Sorry for my delay respond!
> -----Original Message-----
> From: Andrew Murray <[email protected]>
> Sent: 2020??1??13?? 18:10
> To: Z.q. Hou <[email protected]>
> Cc: [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; Leo Li
> <[email protected]>; [email protected];
> [email protected]; [email protected]; Mingkai Hu
> <[email protected]>; M.h. Lian <[email protected]>; Xiaowei Bao
> <[email protected]>
> Subject: Re: [PATCHv9 01/12] PCI: mobiveil: Re-abstract the private structure
>
> On Wed, Nov 20, 2019 at 03:45:23AM +0000, Z.q. Hou wrote:
> > From: Hou Zhiqiang <[email protected]>
> >
> > The Mobiveil PCIe controller can work in either Root Complex mode or
> > Endpoint mode. So introduce a new structure root_port, and abstract
> > the RC related members into it.
>
> The first sentence explains the trigger for this work, the second explains what
> you are changing, it would be helpful to also describe why you need to make
> this change. You could do this by extending the last sentence, e.g.
>
> "So introduce a new structure root_port, and abstract the RC related
> members into it such that it can be used by both ..."
>
> As this series doesn't actually add a EP driver, this abstraction isn't needed
> now - but it is nice to have - it may be helpful to explain this.
>
> The email subject could also more precisely explain what this patch does.
Thanks for the good suggestions! Will change in v10.
>
>
> >
> > Signed-off-by: Hou Zhiqiang <[email protected]>
> > ---
> > V9:
> > - New patch splited from the #1 of V8 patches to make it easy to review.
>
> Indeed, it's much nicer to review - thanks.
>
>
> >
> > drivers/pci/controller/pcie-mobiveil.c | 99
> > ++++++++++++++++----------
> > 1 file changed, 60 insertions(+), 39 deletions(-)
> >
> > diff --git a/drivers/pci/controller/pcie-mobiveil.c
> > b/drivers/pci/controller/pcie-mobiveil.c
> > index 3a696ca45bfa..5fd26e376af2 100644
> > --- a/drivers/pci/controller/pcie-mobiveil.c
> > +++ b/drivers/pci/controller/pcie-mobiveil.c
> > @@ -3,7 +3,10 @@
> > * PCIe host controller driver for Mobiveil PCIe Host controller
> > *
> > * Copyright (c) 2018 Mobiveil Inc.
> > + * Copyright 2019 NXP
> > + *
> > * Author: Subrahmanya Lingappa <[email protected]>
> > + * Recode: Hou Zhiqiang <[email protected]>
>
> As per my previous feedback, I'm not sure the value of using the term refactor
> or a synonym of it. And I certaintly don't want to encourage anyone that
> modifies this file to add a similar tag when the information is easily visible via
> GIT and the get_maintainers script.
Will remove the recode tag in v10.
>
> > */
> >
> > #include <linux/delay.h>
> > @@ -138,22 +141,27 @@ struct mobiveil_msi { /* MSI
> information */
> > DECLARE_BITMAP(msi_irq_in_use, PCI_NUM_MSI); };
> >
> > +struct root_port {
> > + char root_bus_nr;
> > + void __iomem *config_axi_slave_base; /* endpoint config base */
> > + struct resource *ob_io_res;
> > + int irq;
> > + raw_spinlock_t intx_mask_lock;
> > + struct irq_domain *intx_domain;
> > + struct mobiveil_msi msi;
> > + struct pci_host_bridge *bridge;
> > +};
>
> Please prefix with mobiveil given we have mobiveil related structures inside it.
Do you mean s/root_port/mobiveil_root_port/ ?
Thanks,
Zhiqiang
>
> (Also on your respin, please rebase as per Olof's feedback).
>
> Thanks,
>
> Andrew Murray
>
> > +
> > struct mobiveil_pcie {
> > struct platform_device *pdev;
> > - void __iomem *config_axi_slave_base; /* endpoint config base */
> > void __iomem *csr_axi_slave_base; /* root port config base */
> > void __iomem *apb_csr_base; /* MSI register base */
> > phys_addr_t pcie_reg_base; /* Physical PCIe Controller Base */
> > - struct irq_domain *intx_domain;
> > - raw_spinlock_t intx_mask_lock;
> > - int irq;
> > int apio_wins;
> > int ppio_wins;
> > int ob_wins_configured; /* configured outbound windows */
> > int ib_wins_configured; /* configured inbound windows */
> > - struct resource *ob_io_res;
> > - char root_bus_nr;
> > - struct mobiveil_msi msi;
> > + struct root_port rp;
> > };
> >
> > /*
> > @@ -281,16 +289,17 @@ static bool mobiveil_pcie_link_up(struct
> > mobiveil_pcie *pcie) static bool mobiveil_pcie_valid_device(struct
> > pci_bus *bus, unsigned int devfn) {
> > struct mobiveil_pcie *pcie = bus->sysdata;
> > + struct root_port *rp = &pcie->rp;
> >
> > /* Only one device down on each root port */
> > - if ((bus->number == pcie->root_bus_nr) && (devfn > 0))
> > + if ((bus->number == rp->root_bus_nr) && (devfn > 0))
> > return false;
> >
> > /*
> > * Do not read more than one device on the bus directly
> > * attached to RC
> > */
> > - if ((bus->primary == pcie->root_bus_nr) && (PCI_SLOT(devfn) > 0))
> > + if ((bus->primary == rp->root_bus_nr) && (PCI_SLOT(devfn) > 0))
> > return false;
> >
> > return true;
> > @@ -304,13 +313,14 @@ static void __iomem
> *mobiveil_pcie_map_bus(struct pci_bus *bus,
> > unsigned int devfn, int where) {
> > struct mobiveil_pcie *pcie = bus->sysdata;
> > + struct root_port *rp = &pcie->rp;
> > u32 value;
> >
> > if (!mobiveil_pcie_valid_device(bus, devfn))
> > return NULL;
> >
> > /* RC config access */
> > - if (bus->number == pcie->root_bus_nr)
> > + if (bus->number == rp->root_bus_nr)
> > return pcie->csr_axi_slave_base + where;
> >
> > /*
> > @@ -325,7 +335,7 @@ static void __iomem
> *mobiveil_pcie_map_bus(struct
> > pci_bus *bus,
> >
> > mobiveil_csr_writel(pcie, value,
> PAB_AXI_AMAP_PEX_WIN_L(WIN_NUM_0));
> >
> > - return pcie->config_axi_slave_base + where;
> > + return rp->config_axi_slave_base + where;
> > }
> >
> > static struct pci_ops mobiveil_pcie_ops = { @@ -339,7 +349,8 @@
> > static void mobiveil_pcie_isr(struct irq_desc *desc)
> > struct irq_chip *chip = irq_desc_get_chip(desc);
> > struct mobiveil_pcie *pcie = irq_desc_get_handler_data(desc);
> > struct device *dev = &pcie->pdev->dev;
> > - struct mobiveil_msi *msi = &pcie->msi;
> > + struct root_port *rp = &pcie->rp;
> > + struct mobiveil_msi *msi = &rp->msi;
> > u32 msi_data, msi_addr_lo, msi_addr_hi;
> > u32 intr_status, msi_status;
> > unsigned long shifted_status;
> > @@ -365,7 +376,7 @@ static void mobiveil_pcie_isr(struct irq_desc *desc)
> > shifted_status >>= PAB_INTX_START;
> > do {
> > for_each_set_bit(bit, &shifted_status, PCI_NUM_INTX) {
> > - virq = irq_find_mapping(pcie->intx_domain,
> > + virq = irq_find_mapping(rp->intx_domain,
> > bit + 1);
> > if (virq)
> > generic_handle_irq(virq);
> > @@ -424,15 +435,16 @@ static int mobiveil_pcie_parse_dt(struct
> mobiveil_pcie *pcie)
> > struct device *dev = &pcie->pdev->dev;
> > struct platform_device *pdev = pcie->pdev;
> > struct device_node *node = dev->of_node;
> > + struct root_port *rp = &pcie->rp;
> > struct resource *res;
> >
> > /* map config resource */
> > res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
> > "config_axi_slave");
> > - pcie->config_axi_slave_base = devm_pci_remap_cfg_resource(dev, res);
> > - if (IS_ERR(pcie->config_axi_slave_base))
> > - return PTR_ERR(pcie->config_axi_slave_base);
> > - pcie->ob_io_res = res;
> > + rp->config_axi_slave_base = devm_pci_remap_cfg_resource(dev, res);
> > + if (IS_ERR(rp->config_axi_slave_base))
> > + return PTR_ERR(rp->config_axi_slave_base);
> > + rp->ob_io_res = res;
> >
> > /* map csr resource */
> > res = platform_get_resource_byname(pdev, IORESOURCE_MEM, @@
> -455,9
> > +467,9 @@ static int mobiveil_pcie_parse_dt(struct mobiveil_pcie *pcie)
> > if (of_property_read_u32(node, "ppio-wins", &pcie->ppio_wins))
> > pcie->ppio_wins = MAX_PIO_WINDOWS;
> >
> > - pcie->irq = platform_get_irq(pdev, 0);
> > - if (pcie->irq <= 0) {
> > - dev_err(dev, "failed to map IRQ: %d\n", pcie->irq);
> > + rp->irq = platform_get_irq(pdev, 0);
> > + if (rp->irq <= 0) {
> > + dev_err(dev, "failed to map IRQ: %d\n", rp->irq);
> > return -ENODEV;
> > }
> >
> > @@ -564,9 +576,9 @@ static int mobiveil_bringup_link(struct
> > mobiveil_pcie *pcie) static void mobiveil_pcie_enable_msi(struct
> > mobiveil_pcie *pcie) {
> > phys_addr_t msg_addr = pcie->pcie_reg_base;
> > - struct mobiveil_msi *msi = &pcie->msi;
> > + struct mobiveil_msi *msi = &pcie->rp.msi;
> >
> > - pcie->msi.num_of_vectors = PCI_NUM_MSI;
> > + msi->num_of_vectors = PCI_NUM_MSI;
> > msi->msi_pages_phys = (phys_addr_t)msg_addr;
> >
> > writel_relaxed(lower_32_bits(msg_addr),
> > @@ -579,7 +591,8 @@ static void mobiveil_pcie_enable_msi(struct
> > mobiveil_pcie *pcie)
> >
> > static int mobiveil_host_init(struct mobiveil_pcie *pcie) {
> > - struct pci_host_bridge *bridge = pci_host_bridge_from_priv(pcie);
> > + struct root_port *rp = &pcie->rp;
> > + struct pci_host_bridge *bridge = rp->bridge;
> > u32 value, pab_ctrl, type;
> > struct resource_entry *win;
> >
> > @@ -629,8 +642,8 @@ static int mobiveil_host_init(struct mobiveil_pcie
> *pcie)
> > */
> >
> > /* config outbound translation window */
> > - program_ob_windows(pcie, WIN_NUM_0, pcie->ob_io_res->start, 0,
> > - CFG_WINDOW_TYPE, resource_size(pcie->ob_io_res));
> > + program_ob_windows(pcie, WIN_NUM_0, rp->ob_io_res->start, 0,
> > + CFG_WINDOW_TYPE, resource_size(rp->ob_io_res));
> >
> > /* memory inbound translation window */
> > program_ib_windows(pcie, WIN_NUM_0, 0, 0, MEM_WINDOW_TYPE,
> > IB_WIN_SIZE); @@ -667,32 +680,36 @@ static void
> > mobiveil_mask_intx_irq(struct irq_data *data) {
> > struct irq_desc *desc = irq_to_desc(data->irq);
> > struct mobiveil_pcie *pcie;
> > + struct root_port *rp;
> > unsigned long flags;
> > u32 mask, shifted_val;
> >
> > pcie = irq_desc_get_chip_data(desc);
> > + rp = &pcie->rp;
> > mask = 1 << ((data->hwirq + PAB_INTX_START) - 1);
> > - raw_spin_lock_irqsave(&pcie->intx_mask_lock, flags);
> > + raw_spin_lock_irqsave(&rp->intx_mask_lock, flags);
> > shifted_val = mobiveil_csr_readl(pcie, PAB_INTP_AMBA_MISC_ENB);
> > shifted_val &= ~mask;
> > mobiveil_csr_writel(pcie, shifted_val, PAB_INTP_AMBA_MISC_ENB);
> > - raw_spin_unlock_irqrestore(&pcie->intx_mask_lock, flags);
> > + raw_spin_unlock_irqrestore(&rp->intx_mask_lock, flags);
> > }
> >
> > static void mobiveil_unmask_intx_irq(struct irq_data *data) {
> > struct irq_desc *desc = irq_to_desc(data->irq);
> > struct mobiveil_pcie *pcie;
> > + struct root_port *rp;
> > unsigned long flags;
> > u32 shifted_val, mask;
> >
> > pcie = irq_desc_get_chip_data(desc);
> > + rp = &pcie->rp;
> > mask = 1 << ((data->hwirq + PAB_INTX_START) - 1);
> > - raw_spin_lock_irqsave(&pcie->intx_mask_lock, flags);
> > + raw_spin_lock_irqsave(&rp->intx_mask_lock, flags);
> > shifted_val = mobiveil_csr_readl(pcie, PAB_INTP_AMBA_MISC_ENB);
> > shifted_val |= mask;
> > mobiveil_csr_writel(pcie, shifted_val, PAB_INTP_AMBA_MISC_ENB);
> > - raw_spin_unlock_irqrestore(&pcie->intx_mask_lock, flags);
> > + raw_spin_unlock_irqrestore(&rp->intx_mask_lock, flags);
> > }
> >
> > static struct irq_chip intx_irq_chip = { @@ -760,7 +777,7 @@ static
> > int mobiveil_irq_msi_domain_alloc(struct irq_domain *domain,
> > unsigned int nr_irqs, void *args) {
> > struct mobiveil_pcie *pcie = domain->host_data;
> > - struct mobiveil_msi *msi = &pcie->msi;
> > + struct mobiveil_msi *msi = &pcie->rp.msi;
> > unsigned long bit;
> >
> > WARN_ON(nr_irqs != 1);
> > @@ -787,7 +804,7 @@ static void mobiveil_irq_msi_domain_free(struct
> > irq_domain *domain, {
> > struct irq_data *d = irq_domain_get_irq_data(domain, virq);
> > struct mobiveil_pcie *pcie = irq_data_get_irq_chip_data(d);
> > - struct mobiveil_msi *msi = &pcie->msi;
> > + struct mobiveil_msi *msi = &pcie->rp.msi;
> >
> > mutex_lock(&msi->lock);
> >
> > @@ -808,9 +825,9 @@ static int mobiveil_allocate_msi_domains(struct
> > mobiveil_pcie *pcie) {
> > struct device *dev = &pcie->pdev->dev;
> > struct fwnode_handle *fwnode = of_node_to_fwnode(dev->of_node);
> > - struct mobiveil_msi *msi = &pcie->msi;
> > + struct mobiveil_msi *msi = &pcie->rp.msi;
> >
> > - mutex_init(&pcie->msi.lock);
> > + mutex_init(&msi->lock);
> > msi->dev_domain = irq_domain_add_linear(NULL,
> msi->num_of_vectors,
> > &msi_domain_ops, pcie);
> > if (!msi->dev_domain) {
> > @@ -834,18 +851,19 @@ static int mobiveil_pcie_init_irq_domain(struct
> > mobiveil_pcie *pcie) {
> > struct device *dev = &pcie->pdev->dev;
> > struct device_node *node = dev->of_node;
> > + struct root_port *rp = &pcie->rp;
> > int ret;
> >
> > /* setup INTx */
> > - pcie->intx_domain = irq_domain_add_linear(node, PCI_NUM_INTX,
> > - &intx_domain_ops, pcie);
> > + rp->intx_domain = irq_domain_add_linear(node, PCI_NUM_INTX,
> > + &intx_domain_ops, pcie);
> >
> > - if (!pcie->intx_domain) {
> > + if (!rp->intx_domain) {
> > dev_err(dev, "Failed to get a INTx IRQ domain\n");
> > return -ENOMEM;
> > }
> >
> > - raw_spin_lock_init(&pcie->intx_mask_lock);
> > + raw_spin_lock_init(&rp->intx_mask_lock);
> >
> > /* setup MSI */
> > ret = mobiveil_allocate_msi_domains(pcie);
> > @@ -862,6 +880,7 @@ static int mobiveil_pcie_probe(struct
> platform_device *pdev)
> > struct pci_bus *child;
> > struct pci_host_bridge *bridge;
> > struct device *dev = &pdev->dev;
> > + struct root_port *rp;
> > int ret;
> >
> > /* allocate the PCIe port */
> > @@ -870,6 +889,8 @@ static int mobiveil_pcie_probe(struct
> platform_device *pdev)
> > return -ENOMEM;
> >
> > pcie = pci_host_bridge_priv(bridge);
> > + rp = &pcie->rp;
> > + rp->bridge = bridge;
> >
> > pcie->pdev = pdev;
> >
> > @@ -904,12 +925,12 @@ static int mobiveil_pcie_probe(struct
> platform_device *pdev)
> > return ret;
> > }
> >
> > - irq_set_chained_handler_and_data(pcie->irq, mobiveil_pcie_isr, pcie);
> > + irq_set_chained_handler_and_data(rp->irq, mobiveil_pcie_isr, pcie);
> >
> > /* Initialize bridge */
> > bridge->dev.parent = dev;
> > bridge->sysdata = pcie;
> > - bridge->busnr = pcie->root_bus_nr;
> > + bridge->busnr = rp->root_bus_nr;
> > bridge->ops = &mobiveil_pcie_ops;
> > bridge->map_irq = of_irq_parse_and_map_pci;
> > bridge->swizzle_irq = pci_common_swizzle;
> > --
> > 2.17.1
> >
Hi Andrew,
Thanks a lot for your comments!
> -----Original Message-----
> From: Andrew Murray <[email protected]>
> Sent: 2020??1??13?? 18:35
> To: Z.q. Hou <[email protected]>
> Cc: [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; Leo Li
> <[email protected]>; [email protected];
> [email protected]; [email protected]; Mingkai Hu
> <[email protected]>; M.h. Lian <[email protected]>; Xiaowei Bao
> <[email protected]>
> Subject: Re: [PATCHv9 03/12] PCI: mobiveil: Collect the interrupt related
> operations into a routine
>
> On Wed, Nov 20, 2019 at 03:45:37AM +0000, Z.q. Hou wrote:
> > From: Hou Zhiqiang <[email protected]>
> >
> > Collect the interrupt initialization related operations into a new
> > routine to make it more readable.
>
> I prefer the word 'function' instead of routine. Also indicate why, not only is it
> nicer but it is in preparation for EP support.
Will replace the 'routine' with 'function', it is only used in RC mode, not used in EP mode.
>
> >
> > Signed-off-by: Hou Zhiqiang <[email protected]>
> > ---
> > V9:
> > - New patch splited from the #1 of V8 patches to make it easy to review.
> >
> > drivers/pci/controller/pcie-mobiveil.c | 65
> > +++++++++++++++++---------
> > 1 file changed, 42 insertions(+), 23 deletions(-)
> >
> > diff --git a/drivers/pci/controller/pcie-mobiveil.c
> > b/drivers/pci/controller/pcie-mobiveil.c
> > index 97f682ca7c7a..512b27a0536e 100644
> > --- a/drivers/pci/controller/pcie-mobiveil.c
> > +++ b/drivers/pci/controller/pcie-mobiveil.c
> > @@ -454,12 +454,6 @@ static int mobiveil_pcie_parse_dt(struct
> mobiveil_pcie *pcie)
> > return PTR_ERR(pcie->csr_axi_slave_base);
> > pcie->pcie_reg_base = res->start;
> >
> > - /* map MSI config resource */
> > - res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
> "apb_csr");
> > - pcie->apb_csr_base = devm_pci_remap_cfg_resource(dev, res);
> > - if (IS_ERR(pcie->apb_csr_base))
> > - return PTR_ERR(pcie->apb_csr_base);
> > -
> > /* read the number of windows requested */
> > if (of_property_read_u32(node, "apio-wins", &pcie->apio_wins))
> > pcie->apio_wins = MAX_PIO_WINDOWS;
> > @@ -467,12 +461,6 @@ static int mobiveil_pcie_parse_dt(struct
> mobiveil_pcie *pcie)
> > if (of_property_read_u32(node, "ppio-wins", &pcie->ppio_wins))
> > pcie->ppio_wins = MAX_PIO_WINDOWS;
> >
> > - rp->irq = platform_get_irq(pdev, 0);
> > - if (rp->irq <= 0) {
> > - dev_err(dev, "failed to map IRQ: %d\n", rp->irq);
> > - return -ENODEV;
> > - }
> > -
> > return 0;
> > }
> >
> > @@ -618,9 +606,6 @@ static int mobiveil_host_init(struct mobiveil_pcie
> *pcie)
> > pab_ctrl |= (1 << AMBA_PIO_ENABLE_SHIFT) | (1 <<
> PEX_PIO_ENABLE_SHIFT);
> > mobiveil_csr_writel(pcie, pab_ctrl, PAB_CTRL);
> >
> > - mobiveil_csr_writel(pcie, (PAB_INTP_INTX_MASK |
> PAB_INTP_MSI_MASK),
> > - PAB_INTP_AMBA_MISC_ENB);
> > -
> > /*
> > * program PIO Enable Bit to 1 and Config Window Enable Bit to 1 in
> > * PAB_AXI_PIO_CTRL Register
> > @@ -670,9 +655,6 @@ static int mobiveil_host_init(struct mobiveil_pcie
> *pcie)
> > value |= (PCI_CLASS_BRIDGE_PCI << 16);
> > mobiveil_csr_writel(pcie, value, PAB_INTP_AXI_PIO_CLASS);
> >
> > - /* setup MSI hardware registers */
> > - mobiveil_pcie_enable_msi(pcie);
> > -
> > return 0;
> > }
> >
> > @@ -873,6 +855,46 @@ static int mobiveil_pcie_init_irq_domain(struct
> mobiveil_pcie *pcie)
> > return 0;
> > }
> >
> > +static int mobiveil_pcie_interrupt_init(struct mobiveil_pcie *pcie) {
> > + struct platform_device *pdev = pcie->pdev;
> > + struct device *dev = &pdev->dev;
> > + struct root_port *rp = &pcie->rp;
> > + struct resource *res;
> > + int ret;
> > +
> > + /* map MSI config resource */
> > + res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
> "apb_csr");
> > + pcie->apb_csr_base = devm_pci_remap_cfg_resource(dev, res);
> > + if (IS_ERR(pcie->apb_csr_base))
> > + return PTR_ERR(pcie->apb_csr_base);
> > +
> > + /* setup MSI hardware registers */
> > + mobiveil_pcie_enable_msi(pcie);
>
> Does this need to come after mobiveil_pcie_init_irq_domain - given that this
> function sets up the irq domain for MSI?
No, I don't think so, because I didn't change the relative order of the 2 functions
you mentioned.
Thanks,
Zhiqiang
>
> Thanks,
>
> Andrew Murray
>
> > +
> > + rp->irq = platform_get_irq(pdev, 0);
> > + if (rp->irq <= 0) {
> > + dev_err(dev, "failed to map IRQ: %d\n", rp->irq);
> > + return -ENODEV;
> > + }
> > +
> > + /* initialize the IRQ domains */
> > + ret = mobiveil_pcie_init_irq_domain(pcie);
> > + if (ret) {
> > + dev_err(dev, "Failed creating IRQ Domain\n");
> > + return ret;
> > + }
> > +
> > + irq_set_chained_handler_and_data(rp->irq, mobiveil_pcie_isr, pcie);
> > +
> > + /* Enable interrupts */
> > + mobiveil_csr_writel(pcie, (PAB_INTP_INTX_MASK |
> PAB_INTP_MSI_MASK),
> > + PAB_INTP_AMBA_MISC_ENB);
> > +
> > +
> > + return 0;
> > +}
> > +
> > int mobiveil_pcie_host_probe(struct mobiveil_pcie *pcie) {
> > struct root_port *rp = &pcie->rp;
> > @@ -906,15 +928,12 @@ int mobiveil_pcie_host_probe(struct
> mobiveil_pcie *pcie)
> > return ret;
> > }
> >
> > - /* initialize the IRQ domains */
> > - ret = mobiveil_pcie_init_irq_domain(pcie);
> > + ret = mobiveil_pcie_interrupt_init(pcie);
> > if (ret) {
> > - dev_err(dev, "Failed creating IRQ Domain\n");
> > + dev_err(dev, "Interrupt init failed\n");
> > return ret;
> > }
> >
> > - irq_set_chained_handler_and_data(rp->irq, mobiveil_pcie_isr, pcie);
> > -
> > /* Initialize bridge */
> > bridge->dev.parent = dev;
> > bridge->sysdata = pcie;
> > --
> > 2.17.1
> >
On Thu, Feb 06, 2020 at 11:04:29AM +0000, Z.q. Hou wrote:
> Hi Andrew,
>
> Thanks a lot for your comments!
> Sorry for my delay respond!
>
> > -----Original Message-----
> > From: Andrew Murray <[email protected]>
> > Sent: 2020年1月13日 18:10
> > To: Z.q. Hou <[email protected]>
> > Cc: [email protected]; [email protected];
> > [email protected]; [email protected];
> > [email protected]; [email protected]; [email protected];
> > [email protected]; [email protected];
> > [email protected]; [email protected]; Leo Li
> > <[email protected]>; [email protected];
> > [email protected]; [email protected]; Mingkai Hu
> > <[email protected]>; M.h. Lian <[email protected]>; Xiaowei Bao
> > <[email protected]>
> > Subject: Re: [PATCHv9 01/12] PCI: mobiveil: Re-abstract the private structure
> >
> > On Wed, Nov 20, 2019 at 03:45:23AM +0000, Z.q. Hou wrote:
> > > From: Hou Zhiqiang <[email protected]>
> > >
> > > The Mobiveil PCIe controller can work in either Root Complex mode or
> > > Endpoint mode. So introduce a new structure root_port, and abstract
> > > the RC related members into it.
> >
> > The first sentence explains the trigger for this work, the second explains what
> > you are changing, it would be helpful to also describe why you need to make
> > this change. You could do this by extending the last sentence, e.g.
> >
> > "So introduce a new structure root_port, and abstract the RC related
> > members into it such that it can be used by both ..."
> >
> > As this series doesn't actually add a EP driver, this abstraction isn't needed
> > now - but it is nice to have - it may be helpful to explain this.
> >
> > The email subject could also more precisely explain what this patch does.
>
> Thanks for the good suggestions! Will change in v10.
>
> >
> >
> > >
> > > Signed-off-by: Hou Zhiqiang <[email protected]>
> > > ---
> > > V9:
> > > - New patch splited from the #1 of V8 patches to make it easy to review.
> >
> > Indeed, it's much nicer to review - thanks.
> >
> >
> > >
> > > drivers/pci/controller/pcie-mobiveil.c | 99
> > > ++++++++++++++++----------
> > > 1 file changed, 60 insertions(+), 39 deletions(-)
> > >
> > > diff --git a/drivers/pci/controller/pcie-mobiveil.c
> > > b/drivers/pci/controller/pcie-mobiveil.c
> > > index 3a696ca45bfa..5fd26e376af2 100644
> > > --- a/drivers/pci/controller/pcie-mobiveil.c
> > > +++ b/drivers/pci/controller/pcie-mobiveil.c
> > > @@ -3,7 +3,10 @@
> > > * PCIe host controller driver for Mobiveil PCIe Host controller
> > > *
> > > * Copyright (c) 2018 Mobiveil Inc.
> > > + * Copyright 2019 NXP
> > > + *
> > > * Author: Subrahmanya Lingappa <[email protected]>
> > > + * Recode: Hou Zhiqiang <[email protected]>
> >
> > As per my previous feedback, I'm not sure the value of using the term refactor
> > or a synonym of it. And I certaintly don't want to encourage anyone that
> > modifies this file to add a similar tag when the information is easily visible via
> > GIT and the get_maintainers script.
>
> Will remove the recode tag in v10.
>
> >
> > > */
> > >
> > > #include <linux/delay.h>
> > > @@ -138,22 +141,27 @@ struct mobiveil_msi { /* MSI
> > information */
> > > DECLARE_BITMAP(msi_irq_in_use, PCI_NUM_MSI); };
> > >
> > > +struct root_port {
> > > + char root_bus_nr;
> > > + void __iomem *config_axi_slave_base; /* endpoint config base */
> > > + struct resource *ob_io_res;
> > > + int irq;
> > > + raw_spinlock_t intx_mask_lock;
> > > + struct irq_domain *intx_domain;
> > > + struct mobiveil_msi msi;
> > > + struct pci_host_bridge *bridge;
> > > +};
> >
> > Please prefix with mobiveil given we have mobiveil related structures inside it.
>
> Do you mean s/root_port/mobiveil_root_port/ ?
Yes!
Thanks,
Andrew Murray
>
> Thanks,
> Zhiqiang
>
> >
> > (Also on your respin, please rebase as per Olof's feedback).
> >
> > Thanks,
> >
> > Andrew Murray
> >
> > > +
> > > struct mobiveil_pcie {
> > > struct platform_device *pdev;
> > > - void __iomem *config_axi_slave_base; /* endpoint config base */
> > > void __iomem *csr_axi_slave_base; /* root port config base */
> > > void __iomem *apb_csr_base; /* MSI register base */
> > > phys_addr_t pcie_reg_base; /* Physical PCIe Controller Base */
> > > - struct irq_domain *intx_domain;
> > > - raw_spinlock_t intx_mask_lock;
> > > - int irq;
> > > int apio_wins;
> > > int ppio_wins;
> > > int ob_wins_configured; /* configured outbound windows */
> > > int ib_wins_configured; /* configured inbound windows */
> > > - struct resource *ob_io_res;
> > > - char root_bus_nr;
> > > - struct mobiveil_msi msi;
> > > + struct root_port rp;
> > > };
> > >
> > > /*
> > > @@ -281,16 +289,17 @@ static bool mobiveil_pcie_link_up(struct
> > > mobiveil_pcie *pcie) static bool mobiveil_pcie_valid_device(struct
> > > pci_bus *bus, unsigned int devfn) {
> > > struct mobiveil_pcie *pcie = bus->sysdata;
> > > + struct root_port *rp = &pcie->rp;
> > >
> > > /* Only one device down on each root port */
> > > - if ((bus->number == pcie->root_bus_nr) && (devfn > 0))
> > > + if ((bus->number == rp->root_bus_nr) && (devfn > 0))
> > > return false;
> > >
> > > /*
> > > * Do not read more than one device on the bus directly
> > > * attached to RC
> > > */
> > > - if ((bus->primary == pcie->root_bus_nr) && (PCI_SLOT(devfn) > 0))
> > > + if ((bus->primary == rp->root_bus_nr) && (PCI_SLOT(devfn) > 0))
> > > return false;
> > >
> > > return true;
> > > @@ -304,13 +313,14 @@ static void __iomem
> > *mobiveil_pcie_map_bus(struct pci_bus *bus,
> > > unsigned int devfn, int where) {
> > > struct mobiveil_pcie *pcie = bus->sysdata;
> > > + struct root_port *rp = &pcie->rp;
> > > u32 value;
> > >
> > > if (!mobiveil_pcie_valid_device(bus, devfn))
> > > return NULL;
> > >
> > > /* RC config access */
> > > - if (bus->number == pcie->root_bus_nr)
> > > + if (bus->number == rp->root_bus_nr)
> > > return pcie->csr_axi_slave_base + where;
> > >
> > > /*
> > > @@ -325,7 +335,7 @@ static void __iomem
> > *mobiveil_pcie_map_bus(struct
> > > pci_bus *bus,
> > >
> > > mobiveil_csr_writel(pcie, value,
> > PAB_AXI_AMAP_PEX_WIN_L(WIN_NUM_0));
> > >
> > > - return pcie->config_axi_slave_base + where;
> > > + return rp->config_axi_slave_base + where;
> > > }
> > >
> > > static struct pci_ops mobiveil_pcie_ops = { @@ -339,7 +349,8 @@
> > > static void mobiveil_pcie_isr(struct irq_desc *desc)
> > > struct irq_chip *chip = irq_desc_get_chip(desc);
> > > struct mobiveil_pcie *pcie = irq_desc_get_handler_data(desc);
> > > struct device *dev = &pcie->pdev->dev;
> > > - struct mobiveil_msi *msi = &pcie->msi;
> > > + struct root_port *rp = &pcie->rp;
> > > + struct mobiveil_msi *msi = &rp->msi;
> > > u32 msi_data, msi_addr_lo, msi_addr_hi;
> > > u32 intr_status, msi_status;
> > > unsigned long shifted_status;
> > > @@ -365,7 +376,7 @@ static void mobiveil_pcie_isr(struct irq_desc *desc)
> > > shifted_status >>= PAB_INTX_START;
> > > do {
> > > for_each_set_bit(bit, &shifted_status, PCI_NUM_INTX) {
> > > - virq = irq_find_mapping(pcie->intx_domain,
> > > + virq = irq_find_mapping(rp->intx_domain,
> > > bit + 1);
> > > if (virq)
> > > generic_handle_irq(virq);
> > > @@ -424,15 +435,16 @@ static int mobiveil_pcie_parse_dt(struct
> > mobiveil_pcie *pcie)
> > > struct device *dev = &pcie->pdev->dev;
> > > struct platform_device *pdev = pcie->pdev;
> > > struct device_node *node = dev->of_node;
> > > + struct root_port *rp = &pcie->rp;
> > > struct resource *res;
> > >
> > > /* map config resource */
> > > res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
> > > "config_axi_slave");
> > > - pcie->config_axi_slave_base = devm_pci_remap_cfg_resource(dev, res);
> > > - if (IS_ERR(pcie->config_axi_slave_base))
> > > - return PTR_ERR(pcie->config_axi_slave_base);
> > > - pcie->ob_io_res = res;
> > > + rp->config_axi_slave_base = devm_pci_remap_cfg_resource(dev, res);
> > > + if (IS_ERR(rp->config_axi_slave_base))
> > > + return PTR_ERR(rp->config_axi_slave_base);
> > > + rp->ob_io_res = res;
> > >
> > > /* map csr resource */
> > > res = platform_get_resource_byname(pdev, IORESOURCE_MEM, @@
> > -455,9
> > > +467,9 @@ static int mobiveil_pcie_parse_dt(struct mobiveil_pcie *pcie)
> > > if (of_property_read_u32(node, "ppio-wins", &pcie->ppio_wins))
> > > pcie->ppio_wins = MAX_PIO_WINDOWS;
> > >
> > > - pcie->irq = platform_get_irq(pdev, 0);
> > > - if (pcie->irq <= 0) {
> > > - dev_err(dev, "failed to map IRQ: %d\n", pcie->irq);
> > > + rp->irq = platform_get_irq(pdev, 0);
> > > + if (rp->irq <= 0) {
> > > + dev_err(dev, "failed to map IRQ: %d\n", rp->irq);
> > > return -ENODEV;
> > > }
> > >
> > > @@ -564,9 +576,9 @@ static int mobiveil_bringup_link(struct
> > > mobiveil_pcie *pcie) static void mobiveil_pcie_enable_msi(struct
> > > mobiveil_pcie *pcie) {
> > > phys_addr_t msg_addr = pcie->pcie_reg_base;
> > > - struct mobiveil_msi *msi = &pcie->msi;
> > > + struct mobiveil_msi *msi = &pcie->rp.msi;
> > >
> > > - pcie->msi.num_of_vectors = PCI_NUM_MSI;
> > > + msi->num_of_vectors = PCI_NUM_MSI;
> > > msi->msi_pages_phys = (phys_addr_t)msg_addr;
> > >
> > > writel_relaxed(lower_32_bits(msg_addr),
> > > @@ -579,7 +591,8 @@ static void mobiveil_pcie_enable_msi(struct
> > > mobiveil_pcie *pcie)
> > >
> > > static int mobiveil_host_init(struct mobiveil_pcie *pcie) {
> > > - struct pci_host_bridge *bridge = pci_host_bridge_from_priv(pcie);
> > > + struct root_port *rp = &pcie->rp;
> > > + struct pci_host_bridge *bridge = rp->bridge;
> > > u32 value, pab_ctrl, type;
> > > struct resource_entry *win;
> > >
> > > @@ -629,8 +642,8 @@ static int mobiveil_host_init(struct mobiveil_pcie
> > *pcie)
> > > */
> > >
> > > /* config outbound translation window */
> > > - program_ob_windows(pcie, WIN_NUM_0, pcie->ob_io_res->start, 0,
> > > - CFG_WINDOW_TYPE, resource_size(pcie->ob_io_res));
> > > + program_ob_windows(pcie, WIN_NUM_0, rp->ob_io_res->start, 0,
> > > + CFG_WINDOW_TYPE, resource_size(rp->ob_io_res));
> > >
> > > /* memory inbound translation window */
> > > program_ib_windows(pcie, WIN_NUM_0, 0, 0, MEM_WINDOW_TYPE,
> > > IB_WIN_SIZE); @@ -667,32 +680,36 @@ static void
> > > mobiveil_mask_intx_irq(struct irq_data *data) {
> > > struct irq_desc *desc = irq_to_desc(data->irq);
> > > struct mobiveil_pcie *pcie;
> > > + struct root_port *rp;
> > > unsigned long flags;
> > > u32 mask, shifted_val;
> > >
> > > pcie = irq_desc_get_chip_data(desc);
> > > + rp = &pcie->rp;
> > > mask = 1 << ((data->hwirq + PAB_INTX_START) - 1);
> > > - raw_spin_lock_irqsave(&pcie->intx_mask_lock, flags);
> > > + raw_spin_lock_irqsave(&rp->intx_mask_lock, flags);
> > > shifted_val = mobiveil_csr_readl(pcie, PAB_INTP_AMBA_MISC_ENB);
> > > shifted_val &= ~mask;
> > > mobiveil_csr_writel(pcie, shifted_val, PAB_INTP_AMBA_MISC_ENB);
> > > - raw_spin_unlock_irqrestore(&pcie->intx_mask_lock, flags);
> > > + raw_spin_unlock_irqrestore(&rp->intx_mask_lock, flags);
> > > }
> > >
> > > static void mobiveil_unmask_intx_irq(struct irq_data *data) {
> > > struct irq_desc *desc = irq_to_desc(data->irq);
> > > struct mobiveil_pcie *pcie;
> > > + struct root_port *rp;
> > > unsigned long flags;
> > > u32 shifted_val, mask;
> > >
> > > pcie = irq_desc_get_chip_data(desc);
> > > + rp = &pcie->rp;
> > > mask = 1 << ((data->hwirq + PAB_INTX_START) - 1);
> > > - raw_spin_lock_irqsave(&pcie->intx_mask_lock, flags);
> > > + raw_spin_lock_irqsave(&rp->intx_mask_lock, flags);
> > > shifted_val = mobiveil_csr_readl(pcie, PAB_INTP_AMBA_MISC_ENB);
> > > shifted_val |= mask;
> > > mobiveil_csr_writel(pcie, shifted_val, PAB_INTP_AMBA_MISC_ENB);
> > > - raw_spin_unlock_irqrestore(&pcie->intx_mask_lock, flags);
> > > + raw_spin_unlock_irqrestore(&rp->intx_mask_lock, flags);
> > > }
> > >
> > > static struct irq_chip intx_irq_chip = { @@ -760,7 +777,7 @@ static
> > > int mobiveil_irq_msi_domain_alloc(struct irq_domain *domain,
> > > unsigned int nr_irqs, void *args) {
> > > struct mobiveil_pcie *pcie = domain->host_data;
> > > - struct mobiveil_msi *msi = &pcie->msi;
> > > + struct mobiveil_msi *msi = &pcie->rp.msi;
> > > unsigned long bit;
> > >
> > > WARN_ON(nr_irqs != 1);
> > > @@ -787,7 +804,7 @@ static void mobiveil_irq_msi_domain_free(struct
> > > irq_domain *domain, {
> > > struct irq_data *d = irq_domain_get_irq_data(domain, virq);
> > > struct mobiveil_pcie *pcie = irq_data_get_irq_chip_data(d);
> > > - struct mobiveil_msi *msi = &pcie->msi;
> > > + struct mobiveil_msi *msi = &pcie->rp.msi;
> > >
> > > mutex_lock(&msi->lock);
> > >
> > > @@ -808,9 +825,9 @@ static int mobiveil_allocate_msi_domains(struct
> > > mobiveil_pcie *pcie) {
> > > struct device *dev = &pcie->pdev->dev;
> > > struct fwnode_handle *fwnode = of_node_to_fwnode(dev->of_node);
> > > - struct mobiveil_msi *msi = &pcie->msi;
> > > + struct mobiveil_msi *msi = &pcie->rp.msi;
> > >
> > > - mutex_init(&pcie->msi.lock);
> > > + mutex_init(&msi->lock);
> > > msi->dev_domain = irq_domain_add_linear(NULL,
> > msi->num_of_vectors,
> > > &msi_domain_ops, pcie);
> > > if (!msi->dev_domain) {
> > > @@ -834,18 +851,19 @@ static int mobiveil_pcie_init_irq_domain(struct
> > > mobiveil_pcie *pcie) {
> > > struct device *dev = &pcie->pdev->dev;
> > > struct device_node *node = dev->of_node;
> > > + struct root_port *rp = &pcie->rp;
> > > int ret;
> > >
> > > /* setup INTx */
> > > - pcie->intx_domain = irq_domain_add_linear(node, PCI_NUM_INTX,
> > > - &intx_domain_ops, pcie);
> > > + rp->intx_domain = irq_domain_add_linear(node, PCI_NUM_INTX,
> > > + &intx_domain_ops, pcie);
> > >
> > > - if (!pcie->intx_domain) {
> > > + if (!rp->intx_domain) {
> > > dev_err(dev, "Failed to get a INTx IRQ domain\n");
> > > return -ENOMEM;
> > > }
> > >
> > > - raw_spin_lock_init(&pcie->intx_mask_lock);
> > > + raw_spin_lock_init(&rp->intx_mask_lock);
> > >
> > > /* setup MSI */
> > > ret = mobiveil_allocate_msi_domains(pcie);
> > > @@ -862,6 +880,7 @@ static int mobiveil_pcie_probe(struct
> > platform_device *pdev)
> > > struct pci_bus *child;
> > > struct pci_host_bridge *bridge;
> > > struct device *dev = &pdev->dev;
> > > + struct root_port *rp;
> > > int ret;
> > >
> > > /* allocate the PCIe port */
> > > @@ -870,6 +889,8 @@ static int mobiveil_pcie_probe(struct
> > platform_device *pdev)
> > > return -ENOMEM;
> > >
> > > pcie = pci_host_bridge_priv(bridge);
> > > + rp = &pcie->rp;
> > > + rp->bridge = bridge;
> > >
> > > pcie->pdev = pdev;
> > >
> > > @@ -904,12 +925,12 @@ static int mobiveil_pcie_probe(struct
> > platform_device *pdev)
> > > return ret;
> > > }
> > >
> > > - irq_set_chained_handler_and_data(pcie->irq, mobiveil_pcie_isr, pcie);
> > > + irq_set_chained_handler_and_data(rp->irq, mobiveil_pcie_isr, pcie);
> > >
> > > /* Initialize bridge */
> > > bridge->dev.parent = dev;
> > > bridge->sysdata = pcie;
> > > - bridge->busnr = pcie->root_bus_nr;
> > > + bridge->busnr = rp->root_bus_nr;
> > > bridge->ops = &mobiveil_pcie_ops;
> > > bridge->map_irq = of_irq_parse_and_map_pci;
> > > bridge->swizzle_irq = pci_common_swizzle;
> > > --
> > > 2.17.1
> > >
Hi Andrew,
Thanks a lot for your comments!
> -----Original Message-----
> From: Andrew Murray <[email protected]>
> Sent: 2020??1??13?? 18:19
> To: Z.q. Hou <[email protected]>
> Cc: [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; Leo Li
> <[email protected]>; [email protected];
> [email protected]; [email protected]; Mingkai Hu
> <[email protected]>; M.h. Lian <[email protected]>; Xiaowei Bao
> <[email protected]>
> Subject: Re: [PATCHv9 02/12] PCI: mobiveil: Move the host initialization into a
> routine
>
> On Wed, Nov 20, 2019 at 03:45:30AM +0000, Z.q. Hou wrote:
> > From: Hou Zhiqiang <[email protected]>
> >
> > Move the host initialization related operations into a new routine to
> > make it can be reused by other incoming platform's
>
> s/to make/such that/
>
> 'function' is probably a better word than 'routine'.
>
Will change in v10.
>
> > PCIe host driver, in which the Mobiveil GPEX is integrated.
> >
> > Signed-off-by: Hou Zhiqiang <[email protected]>
> > ---
> > V9:
> > - New patch splited from the #1 of V8 patches to make it easy to review.
> >
> > drivers/pci/controller/pcie-mobiveil.c | 38
> > +++++++++++++++-----------
> > 1 file changed, 22 insertions(+), 16 deletions(-)
> >
> > diff --git a/drivers/pci/controller/pcie-mobiveil.c
> > b/drivers/pci/controller/pcie-mobiveil.c
> > index 5fd26e376af2..97f682ca7c7a 100644
> > --- a/drivers/pci/controller/pcie-mobiveil.c
> > +++ b/drivers/pci/controller/pcie-mobiveil.c
> > @@ -873,27 +873,15 @@ static int mobiveil_pcie_init_irq_domain(struct
> mobiveil_pcie *pcie)
> > return 0;
> > }
> >
> > -static int mobiveil_pcie_probe(struct platform_device *pdev)
> > +int mobiveil_pcie_host_probe(struct mobiveil_pcie *pcie)
>
> This is no longer static - but do you need to add a header file somewhere?
The function mobiveil_pcie_probe() is still static below, the
mobiveil_pcie_host_probe() is new introduced one, it is now only used in this
.c file, so I'll change it to a static function in v10.
>
>
> > {
> > - struct mobiveil_pcie *pcie;
> > + struct root_port *rp = &pcie->rp;
> > + struct pci_host_bridge *bridge = rp->bridge;
> > + struct device *dev = &pcie->pdev->dev;
> > struct pci_bus *bus;
> > struct pci_bus *child;
> > - struct pci_host_bridge *bridge;
> > - struct device *dev = &pdev->dev;
> > - struct root_port *rp;
> > int ret;
> >
> > - /* allocate the PCIe port */
> > - bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
> > - if (!bridge)
> > - return -ENOMEM;
> > -
> > - pcie = pci_host_bridge_priv(bridge);
> > - rp = &pcie->rp;
> > - rp->bridge = bridge;
> > -
> > - pcie->pdev = pdev;
> > -
> > ret = mobiveil_pcie_parse_dt(pcie);
> > if (ret) {
> > dev_err(dev, "Parsing DT failed, ret: %x\n", ret); @@ -956,6
> > +944,24 @@ static int mobiveil_pcie_probe(struct platform_device *pdev)
> > return 0;
> > }
> >
> > +static int mobiveil_pcie_probe(struct platform_device *pdev) {
> > + struct mobiveil_pcie *pcie;
> > + struct pci_host_bridge *bridge;
> > + struct device *dev = &pdev->dev;
> > +
> > + bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
>
> You've lost the comment that was above this.
Will add it back.
Thanks,
Zhiqiang
>
> Thanks,
>
> Andrew Murray
>
> > + if (!bridge)
> > + return -ENOMEM;
> > +
> > + pcie = pci_host_bridge_priv(bridge);
> > + pcie->rp.bridge = bridge;
> > +
> > + pcie->pdev = pdev;
> > +
> > + return mobiveil_pcie_host_probe(pcie); }
> > +
> > static const struct of_device_id mobiveil_pcie_of_match[] = {
> > {.compatible = "mbvl,gpex40-pcie",},
> > {},
> > --
> > 2.17.1
> >
Hi Andrew,
Thanks a lot for your review!
B.R,
Zhiqiang
> -----Original Message-----
> From: Andrew Murray <[email protected]>
> Sent: 2020??1??13?? 19:06
> To: Z.q. Hou <[email protected]>
> Cc: [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; Leo Li
> <[email protected]>; [email protected];
> [email protected]; [email protected]; Mingkai Hu
> <[email protected]>; M.h. Lian <[email protected]>; Xiaowei Bao
> <[email protected]>
> Subject: Re: [PATCHv9 04/12] PCI: mobiveil: Modularize the Mobiveil PCIe
> Host Bridge IP driver
>
> On Wed, Nov 20, 2019 at 03:45:43AM +0000, Z.q. Hou wrote:
> > From: Hou Zhiqiang <[email protected]>
> >
> > Modularize the Mobiveil PCIe host driver according to the abstraction
> > of Root Complex and Endpoint and move it into a new directory.
> >
> > Split the RC related routines into pcie-mobiveil-host.c, and common
> > routines into pcie-mobiveil.c, move the macro definitions and function
> > declarations into pcie-mobiveil.h, and the Mobiveil platform reference
> > code into pcie-mobiveil-plat.c. So that it is easy to reuse the
> > extracted routines to add a new host driver, which integrated Mobiveil
> > PCIe GPEX IP.
> >
> > Signed-off-by: Hou Zhiqiang <[email protected]>
>
> Reviewed-by: Andrew Murray <[email protected]>
>
> > ---
> > V9:
> > - New patch splited from the #1 of V8 patches to make it easy to review.
> >
> > MAINTAINERS | 2 +-
> > drivers/pci/controller/Kconfig | 11 +-
> > drivers/pci/controller/Makefile | 2 +-
> > drivers/pci/controller/mobiveil/Kconfig | 24 ++
> > drivers/pci/controller/mobiveil/Makefile | 4 +
> > .../pcie-mobiveil-host.c} | 398 +-----------------
> > .../controller/mobiveil/pcie-mobiveil-plat.c | 60 +++
> > .../pci/controller/mobiveil/pcie-mobiveil.c | 227 ++++++++++
> > .../pci/controller/mobiveil/pcie-mobiveil.h | 178 ++++++++
> > 9 files changed, 497 insertions(+), 409 deletions(-) create mode
> > 100644 drivers/pci/controller/mobiveil/Kconfig
> > create mode 100644 drivers/pci/controller/mobiveil/Makefile
> > rename drivers/pci/controller/{pcie-mobiveil.c =>
> > mobiveil/pcie-mobiveil-host.c} (61%) create mode 100644
> > drivers/pci/controller/mobiveil/pcie-mobiveil-plat.c
> > create mode 100644 drivers/pci/controller/mobiveil/pcie-mobiveil.c
> > create mode 100644 drivers/pci/controller/mobiveil/pcie-mobiveil.h
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS index
> > 3f7f8cdbc471..a4ad99619e53 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -12612,7 +12612,7 @@ M: Hou Zhiqiang
> <[email protected]>
> > L: [email protected]
> > S: Supported
> > F: Documentation/devicetree/bindings/pci/mobiveil-pcie.txt
> > -F: drivers/pci/controller/pcie-mobiveil.c
> > +F: drivers/pci/controller/mobiveil/pcie-mobiveil*
> >
> > PCI DRIVER FOR MVEBU (Marvell Armada 370 and Armada XP SOC
> support)
> > M: Thomas Petazzoni <[email protected]>
> > diff --git a/drivers/pci/controller/Kconfig
> > b/drivers/pci/controller/Kconfig index f5de9119e8d3..74fd332755ae
> > 100644
> > --- a/drivers/pci/controller/Kconfig
> > +++ b/drivers/pci/controller/Kconfig
> > @@ -241,16 +241,6 @@ config PCIE_MEDIATEK
> > Say Y here if you want to enable PCIe controller support on
> > MediaTek SoCs.
> >
> > -config PCIE_MOBIVEIL
> > - bool "Mobiveil AXI PCIe controller"
> > - depends on ARCH_ZYNQMP || COMPILE_TEST
> > - depends on OF
> > - depends on PCI_MSI_IRQ_DOMAIN
> > - help
> > - Say Y here if you want to enable support for the Mobiveil AXI PCIe
> > - Soft IP. It has up to 8 outbound and inbound windows
> > - for address translation and it is a PCIe Gen4 IP.
> > -
> > config PCIE_TANGO_SMP8759
> > bool "Tango SMP8759 PCIe controller (DANGEROUS)"
> > depends on ARCH_TANGO && PCI_MSI && OF @@ -289,4 +279,5 @@
> config
> > PCI_HYPERV_INTERFACE
> > have a common interface with the Hyper-V PCI frontend driver.
> >
> > source "drivers/pci/controller/dwc/Kconfig"
> > +source "drivers/pci/controller/mobiveil/Kconfig"
> > endmenu
> > diff --git a/drivers/pci/controller/Makefile
> > b/drivers/pci/controller/Makefile index a2a22c9d91af..44414cfd45ea
> > 100644
> > --- a/drivers/pci/controller/Makefile
> > +++ b/drivers/pci/controller/Makefile
> > @@ -27,11 +27,11 @@ obj-$(CONFIG_PCIE_ROCKCHIP) += pcie-rockchip.o
> > obj-$(CONFIG_PCIE_ROCKCHIP_EP) += pcie-rockchip-ep.o
> > obj-$(CONFIG_PCIE_ROCKCHIP_HOST) += pcie-rockchip-host.o
> > obj-$(CONFIG_PCIE_MEDIATEK) += pcie-mediatek.o
> > -obj-$(CONFIG_PCIE_MOBIVEIL) += pcie-mobiveil.o
> > obj-$(CONFIG_PCIE_TANGO_SMP8759) += pcie-tango.o
> > obj-$(CONFIG_VMD) += vmd.o
> > # pcie-hisi.o quirks are needed even without CONFIG_PCIE_DW
> > obj-y += dwc/
> > +obj-y += mobiveil/
> >
> >
> > # The following drivers are for devices that use the generic ACPI
> > diff --git a/drivers/pci/controller/mobiveil/Kconfig
> > b/drivers/pci/controller/mobiveil/Kconfig
> > new file mode 100644
> > index 000000000000..64343c07bfed
> > --- /dev/null
> > +++ b/drivers/pci/controller/mobiveil/Kconfig
> > @@ -0,0 +1,24 @@
> > +# SPDX-License-Identifier: GPL-2.0
> > +
> > +menu "Mobiveil PCIe Core Support"
> > + depends on PCI
> > +
> > +config PCIE_MOBIVEIL
> > + bool
> > +
> > +config PCIE_MOBIVEIL_HOST
> > + bool
> > + depends on PCI_MSI_IRQ_DOMAIN
> > + select PCIE_MOBIVEIL
> > +
> > +config PCIE_MOBIVEIL_PLAT
> > + bool "Mobiveil AXI PCIe controller"
> > + depends on ARCH_ZYNQMP || COMPILE_TEST
> > + depends on OF
> > + select PCIE_MOBIVEIL_HOST
> > + help
> > + Say Y here if you want to enable support for the Mobiveil AXI PCIe
> > + Soft IP. It has up to 8 outbound and inbound windows
> > + for address translation and it is a PCIe Gen4 IP.
> > +
> > +endmenu
> > diff --git a/drivers/pci/controller/mobiveil/Makefile
> > b/drivers/pci/controller/mobiveil/Makefile
> > new file mode 100644
> > index 000000000000..9fb6d1c6504d
> > --- /dev/null
> > +++ b/drivers/pci/controller/mobiveil/Makefile
> > @@ -0,0 +1,4 @@
> > +# SPDX-License-Identifier: GPL-2.0
> > +obj-$(CONFIG_PCIE_MOBIVEIL) += pcie-mobiveil.o
> > +obj-$(CONFIG_PCIE_MOBIVEIL_HOST) += pcie-mobiveil-host.o
> > +obj-$(CONFIG_PCIE_MOBIVEIL_PLAT) += pcie-mobiveil-plat.o
> > diff --git a/drivers/pci/controller/pcie-mobiveil.c
> > b/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
> > similarity index 61%
> > rename from drivers/pci/controller/pcie-mobiveil.c
> > rename to drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
> > index 512b27a0536e..2cc424e78d33 100644
> > --- a/drivers/pci/controller/pcie-mobiveil.c
> > +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
> > @@ -9,7 +9,6 @@
> > * Recode: Hou Zhiqiang <[email protected]>
> > */
> >
> > -#include <linux/delay.h>
> > #include <linux/init.h>
> > #include <linux/interrupt.h>
> > #include <linux/irq.h>
> > @@ -26,265 +25,7 @@
> > #include <linux/platform_device.h>
> > #include <linux/slab.h>
> >
> > -#include "../pci.h"
> > -
> > -/* register offsets and bit positions */
> > -
> > -/*
> > - * translation tables are grouped into windows, each window registers
> > are
> > - * grouped into blocks of 4 or 16 registers each
> > - */
> > -#define PAB_REG_BLOCK_SIZE 16
> > -#define PAB_EXT_REG_BLOCK_SIZE 4
> > -
> > -#define PAB_REG_ADDR(offset, win) \
> > - (offset + (win * PAB_REG_BLOCK_SIZE))
> > -#define PAB_EXT_REG_ADDR(offset, win) \
> > - (offset + (win * PAB_EXT_REG_BLOCK_SIZE))
> > -
> > -#define LTSSM_STATUS 0x0404
> > -#define LTSSM_STATUS_L0_MASK 0x3f
> > -#define LTSSM_STATUS_L0 0x2d
> > -
> > -#define PAB_CTRL 0x0808
> > -#define AMBA_PIO_ENABLE_SHIFT 0
> > -#define PEX_PIO_ENABLE_SHIFT 1
> > -#define PAGE_SEL_SHIFT 13
> > -#define PAGE_SEL_MASK 0x3f
> > -#define PAGE_LO_MASK 0x3ff
> > -#define PAGE_SEL_OFFSET_SHIFT 10
> > -
> > -#define PAB_AXI_PIO_CTRL 0x0840
> > -#define APIO_EN_MASK 0xf
> > -
> > -#define PAB_PEX_PIO_CTRL 0x08c0
> > -#define PIO_ENABLE_SHIFT 0
> > -
> > -#define PAB_INTP_AMBA_MISC_ENB 0x0b0c
> > -#define PAB_INTP_AMBA_MISC_STAT 0x0b1c
> > -#define PAB_INTP_INTX_MASK 0x01e0
> > -#define PAB_INTP_MSI_MASK 0x8
> > -
> > -#define PAB_AXI_AMAP_CTRL(win) PAB_REG_ADDR(0x0ba0, win)
> > -#define WIN_ENABLE_SHIFT 0
> > -#define WIN_TYPE_SHIFT 1
> > -#define WIN_TYPE_MASK 0x3
> > -#define WIN_SIZE_MASK 0xfffffc00
> > -
> > -#define PAB_EXT_AXI_AMAP_SIZE(win) PAB_EXT_REG_ADDR(0xbaf0,
> win)
> > -
> > -#define PAB_EXT_AXI_AMAP_AXI_WIN(win)
> PAB_EXT_REG_ADDR(0x80a0, win)
> > -#define PAB_AXI_AMAP_AXI_WIN(win) PAB_REG_ADDR(0x0ba4, win)
> > -#define AXI_WINDOW_ALIGN_MASK 3
> > -
> > -#define PAB_AXI_AMAP_PEX_WIN_L(win) PAB_REG_ADDR(0x0ba8,
> win)
> > -#define PAB_BUS_SHIFT 24
> > -#define PAB_DEVICE_SHIFT 19
> > -#define PAB_FUNCTION_SHIFT 16
> > -
> > -#define PAB_AXI_AMAP_PEX_WIN_H(win) PAB_REG_ADDR(0x0bac,
> win)
> > -#define PAB_INTP_AXI_PIO_CLASS 0x474
> > -
> > -#define PAB_PEX_AMAP_CTRL(win) PAB_REG_ADDR(0x4ba0, win)
> > -#define AMAP_CTRL_EN_SHIFT 0
> > -#define AMAP_CTRL_TYPE_SHIFT 1
> > -#define AMAP_CTRL_TYPE_MASK 3
> > -
> > -#define PAB_EXT_PEX_AMAP_SIZEN(win) PAB_EXT_REG_ADDR(0xbef0,
> win)
> > -#define PAB_EXT_PEX_AMAP_AXI_WIN(win)
> PAB_EXT_REG_ADDR(0xb4a0, win)
> > -#define PAB_PEX_AMAP_AXI_WIN(win) PAB_REG_ADDR(0x4ba4, win)
> > -#define PAB_PEX_AMAP_PEX_WIN_L(win) PAB_REG_ADDR(0x4ba8,
> win)
> > -#define PAB_PEX_AMAP_PEX_WIN_H(win) PAB_REG_ADDR(0x4bac,
> win)
> > -
> > -/* starting offset of INTX bits in status register */
> > -#define PAB_INTX_START 5
> > -
> > -/* supported number of MSI interrupts */
> > -#define PCI_NUM_MSI 16
> > -
> > -/* MSI registers */
> > -#define MSI_BASE_LO_OFFSET 0x04
> > -#define MSI_BASE_HI_OFFSET 0x08
> > -#define MSI_SIZE_OFFSET 0x0c
> > -#define MSI_ENABLE_OFFSET 0x14
> > -#define MSI_STATUS_OFFSET 0x18
> > -#define MSI_DATA_OFFSET 0x20
> > -#define MSI_ADDR_L_OFFSET 0x24
> > -#define MSI_ADDR_H_OFFSET 0x28
> > -
> > -/* outbound and inbound window definitions */
> > -#define WIN_NUM_0 0
> > -#define WIN_NUM_1 1
> > -#define CFG_WINDOW_TYPE 0
> > -#define IO_WINDOW_TYPE 1
> > -#define MEM_WINDOW_TYPE 2
> > -#define IB_WIN_SIZE ((u64)256 * 1024 * 1024 * 1024)
> > -#define MAX_PIO_WINDOWS 8
> > -
> > -/* Parameters for the waiting for link up routine */
> > -#define LINK_WAIT_MAX_RETRIES 10
> > -#define LINK_WAIT_MIN 90000
> > -#define LINK_WAIT_MAX 100000
> > -
> > -#define PAGED_ADDR_BNDRY 0xc00
> > -#define OFFSET_TO_PAGE_ADDR(off) \
> > - ((off & PAGE_LO_MASK) | PAGED_ADDR_BNDRY)
> > -#define OFFSET_TO_PAGE_IDX(off) \
> > - ((off >> PAGE_SEL_OFFSET_SHIFT) & PAGE_SEL_MASK)
> > -
> > -struct mobiveil_msi { /* MSI information */
> > - struct mutex lock; /* protect bitmap variable */
> > - struct irq_domain *msi_domain;
> > - struct irq_domain *dev_domain;
> > - phys_addr_t msi_pages_phys;
> > - int num_of_vectors;
> > - DECLARE_BITMAP(msi_irq_in_use, PCI_NUM_MSI);
> > -};
> > -
> > -struct root_port {
> > - char root_bus_nr;
> > - void __iomem *config_axi_slave_base; /* endpoint config base */
> > - struct resource *ob_io_res;
> > - int irq;
> > - raw_spinlock_t intx_mask_lock;
> > - struct irq_domain *intx_domain;
> > - struct mobiveil_msi msi;
> > - struct pci_host_bridge *bridge;
> > -};
> > -
> > -struct mobiveil_pcie {
> > - struct platform_device *pdev;
> > - void __iomem *csr_axi_slave_base; /* root port config base */
> > - void __iomem *apb_csr_base; /* MSI register base */
> > - phys_addr_t pcie_reg_base; /* Physical PCIe Controller Base */
> > - int apio_wins;
> > - int ppio_wins;
> > - int ob_wins_configured; /* configured outbound windows */
> > - int ib_wins_configured; /* configured inbound windows */
> > - struct root_port rp;
> > -};
> > -
> > -/*
> > - * mobiveil_pcie_sel_page - routine to access paged register
> > - *
> > - * Registers whose address greater than PAGED_ADDR_BNDRY (0xc00) are
> > paged,
> > - * for this scheme to work extracted higher 6 bits of the offset will
> > be
> > - * written to pg_sel field of PAB_CTRL register and rest of the lower
> > 10
> > - * bits enabled with PAGED_ADDR_BNDRY are used as offset of the
> register.
> > - */
> > -static void mobiveil_pcie_sel_page(struct mobiveil_pcie *pcie, u8
> > pg_idx) -{
> > - u32 val;
> > -
> > - val = readl(pcie->csr_axi_slave_base + PAB_CTRL);
> > - val &= ~(PAGE_SEL_MASK << PAGE_SEL_SHIFT);
> > - val |= (pg_idx & PAGE_SEL_MASK) << PAGE_SEL_SHIFT;
> > -
> > - writel(val, pcie->csr_axi_slave_base + PAB_CTRL);
> > -}
> > -
> > -static void *mobiveil_pcie_comp_addr(struct mobiveil_pcie *pcie, u32
> > off) -{
> > - if (off < PAGED_ADDR_BNDRY) {
> > - /* For directly accessed registers, clear the pg_sel field */
> > - mobiveil_pcie_sel_page(pcie, 0);
> > - return pcie->csr_axi_slave_base + off;
> > - }
> > -
> > - mobiveil_pcie_sel_page(pcie, OFFSET_TO_PAGE_IDX(off));
> > - return pcie->csr_axi_slave_base + OFFSET_TO_PAGE_ADDR(off);
> > -}
> > -
> > -static int mobiveil_pcie_read(void __iomem *addr, int size, u32 *val)
> > -{
> > - if ((uintptr_t)addr & (size - 1)) {
> > - *val = 0;
> > - return PCIBIOS_BAD_REGISTER_NUMBER;
> > - }
> > -
> > - switch (size) {
> > - case 4:
> > - *val = readl(addr);
> > - break;
> > - case 2:
> > - *val = readw(addr);
> > - break;
> > - case 1:
> > - *val = readb(addr);
> > - break;
> > - default:
> > - *val = 0;
> > - return PCIBIOS_BAD_REGISTER_NUMBER;
> > - }
> > -
> > - return PCIBIOS_SUCCESSFUL;
> > -}
> > -
> > -static int mobiveil_pcie_write(void __iomem *addr, int size, u32 val)
> > -{
> > - if ((uintptr_t)addr & (size - 1))
> > - return PCIBIOS_BAD_REGISTER_NUMBER;
> > -
> > - switch (size) {
> > - case 4:
> > - writel(val, addr);
> > - break;
> > - case 2:
> > - writew(val, addr);
> > - break;
> > - case 1:
> > - writeb(val, addr);
> > - break;
> > - default:
> > - return PCIBIOS_BAD_REGISTER_NUMBER;
> > - }
> > -
> > - return PCIBIOS_SUCCESSFUL;
> > -}
> > -
> > -static u32 mobiveil_csr_read(struct mobiveil_pcie *pcie, u32 off,
> > size_t size) -{
> > - void *addr;
> > - u32 val;
> > - int ret;
> > -
> > - addr = mobiveil_pcie_comp_addr(pcie, off);
> > -
> > - ret = mobiveil_pcie_read(addr, size, &val);
> > - if (ret)
> > - dev_err(&pcie->pdev->dev, "read CSR address failed\n");
> > -
> > - return val;
> > -}
> > -
> > -static void mobiveil_csr_write(struct mobiveil_pcie *pcie, u32 val, u32 off,
> > - size_t size)
> > -{
> > - void *addr;
> > - int ret;
> > -
> > - addr = mobiveil_pcie_comp_addr(pcie, off);
> > -
> > - ret = mobiveil_pcie_write(addr, size, val);
> > - if (ret)
> > - dev_err(&pcie->pdev->dev, "write CSR address failed\n");
> > -}
> > -
> > -static u32 mobiveil_csr_readl(struct mobiveil_pcie *pcie, u32 off) -{
> > - return mobiveil_csr_read(pcie, off, 0x4);
> > -}
> > -
> > -static void mobiveil_csr_writel(struct mobiveil_pcie *pcie, u32 val,
> > u32 off) -{
> > - mobiveil_csr_write(pcie, val, off, 0x4);
> > -}
> > -
> > -static bool mobiveil_pcie_link_up(struct mobiveil_pcie *pcie) -{
> > - return (mobiveil_csr_readl(pcie, LTSSM_STATUS) &
> > - LTSSM_STATUS_L0_MASK) == LTSSM_STATUS_L0;
> > -}
> > +#include "pcie-mobiveil.h"
> >
> > static bool mobiveil_pcie_valid_device(struct pci_bus *bus, unsigned
> > int devfn) { @@ -464,103 +205,6 @@ static int
> > mobiveil_pcie_parse_dt(struct mobiveil_pcie *pcie)
> > return 0;
> > }
> >
> > -static void program_ib_windows(struct mobiveil_pcie *pcie, int win_num,
> > - u64 cpu_addr, u64 pci_addr, u32 type, u64 size)
> > -{
> > - u32 value;
> > - u64 size64 = ~(size - 1);
> > -
> > - if (win_num >= pcie->ppio_wins) {
> > - dev_err(&pcie->pdev->dev,
> > - "ERROR: max inbound windows reached !\n");
> > - return;
> > - }
> > -
> > - value = mobiveil_csr_readl(pcie, PAB_PEX_AMAP_CTRL(win_num));
> > - value &= ~(AMAP_CTRL_TYPE_MASK << AMAP_CTRL_TYPE_SHIFT |
> WIN_SIZE_MASK);
> > - value |= type << AMAP_CTRL_TYPE_SHIFT | 1 <<
> AMAP_CTRL_EN_SHIFT |
> > - (lower_32_bits(size64) & WIN_SIZE_MASK);
> > - mobiveil_csr_writel(pcie, value, PAB_PEX_AMAP_CTRL(win_num));
> > -
> > - mobiveil_csr_writel(pcie, upper_32_bits(size64),
> > - PAB_EXT_PEX_AMAP_SIZEN(win_num));
> > -
> > - mobiveil_csr_writel(pcie, lower_32_bits(cpu_addr),
> > - PAB_PEX_AMAP_AXI_WIN(win_num));
> > - mobiveil_csr_writel(pcie, upper_32_bits(cpu_addr),
> > - PAB_EXT_PEX_AMAP_AXI_WIN(win_num));
> > -
> > - mobiveil_csr_writel(pcie, lower_32_bits(pci_addr),
> > - PAB_PEX_AMAP_PEX_WIN_L(win_num));
> > - mobiveil_csr_writel(pcie, upper_32_bits(pci_addr),
> > - PAB_PEX_AMAP_PEX_WIN_H(win_num));
> > -
> > - pcie->ib_wins_configured++;
> > -}
> > -
> > -/*
> > - * routine to program the outbound windows
> > - */
> > -static void program_ob_windows(struct mobiveil_pcie *pcie, int win_num,
> > - u64 cpu_addr, u64 pci_addr, u32 type, u64 size)
> > -{
> > - u32 value;
> > - u64 size64 = ~(size - 1);
> > -
> > - if (win_num >= pcie->apio_wins) {
> > - dev_err(&pcie->pdev->dev,
> > - "ERROR: max outbound windows reached !\n");
> > - return;
> > - }
> > -
> > - /*
> > - * program Enable Bit to 1, Type Bit to (00) base 2, AXI Window Size Bit
> > - * to 4 KB in PAB_AXI_AMAP_CTRL register
> > - */
> > - value = mobiveil_csr_readl(pcie, PAB_AXI_AMAP_CTRL(win_num));
> > - value &= ~(WIN_TYPE_MASK << WIN_TYPE_SHIFT | WIN_SIZE_MASK);
> > - value |= 1 << WIN_ENABLE_SHIFT | type << WIN_TYPE_SHIFT |
> > - (lower_32_bits(size64) & WIN_SIZE_MASK);
> > - mobiveil_csr_writel(pcie, value, PAB_AXI_AMAP_CTRL(win_num));
> > -
> > - mobiveil_csr_writel(pcie, upper_32_bits(size64),
> > - PAB_EXT_AXI_AMAP_SIZE(win_num));
> > -
> > - /*
> > - * program AXI window base with appropriate value in
> > - * PAB_AXI_AMAP_AXI_WIN0 register
> > - */
> > - mobiveil_csr_writel(pcie,
> > - lower_32_bits(cpu_addr) &
> (~AXI_WINDOW_ALIGN_MASK),
> > - PAB_AXI_AMAP_AXI_WIN(win_num));
> > - mobiveil_csr_writel(pcie, upper_32_bits(cpu_addr),
> > - PAB_EXT_AXI_AMAP_AXI_WIN(win_num));
> > -
> > - mobiveil_csr_writel(pcie, lower_32_bits(pci_addr),
> > - PAB_AXI_AMAP_PEX_WIN_L(win_num));
> > - mobiveil_csr_writel(pcie, upper_32_bits(pci_addr),
> > - PAB_AXI_AMAP_PEX_WIN_H(win_num));
> > -
> > - pcie->ob_wins_configured++;
> > -}
> > -
> > -static int mobiveil_bringup_link(struct mobiveil_pcie *pcie) -{
> > - int retries;
> > -
> > - /* check if the link is up or not */
> > - for (retries = 0; retries < LINK_WAIT_MAX_RETRIES; retries++) {
> > - if (mobiveil_pcie_link_up(pcie))
> > - return 0;
> > -
> > - usleep_range(LINK_WAIT_MIN, LINK_WAIT_MAX);
> > - }
> > -
> > - dev_err(&pcie->pdev->dev, "link never came up\n");
> > -
> > - return -ETIMEDOUT;
> > -}
> > -
> > static void mobiveil_pcie_enable_msi(struct mobiveil_pcie *pcie) {
> > phys_addr_t msg_addr = pcie->pcie_reg_base; @@ -962,43 +606,3 @@
> int
> > mobiveil_pcie_host_probe(struct mobiveil_pcie *pcie)
> >
> > return 0;
> > }
> > -
> > -static int mobiveil_pcie_probe(struct platform_device *pdev) -{
> > - struct mobiveil_pcie *pcie;
> > - struct pci_host_bridge *bridge;
> > - struct device *dev = &pdev->dev;
> > -
> > - bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
> > - if (!bridge)
> > - return -ENOMEM;
> > -
> > - pcie = pci_host_bridge_priv(bridge);
> > - pcie->rp.bridge = bridge;
> > -
> > - pcie->pdev = pdev;
> > -
> > - return mobiveil_pcie_host_probe(pcie);
> > -}
> > -
> > -static const struct of_device_id mobiveil_pcie_of_match[] = {
> > - {.compatible = "mbvl,gpex40-pcie",},
> > - {},
> > -};
> > -
> > -MODULE_DEVICE_TABLE(of, mobiveil_pcie_of_match);
> > -
> > -static struct platform_driver mobiveil_pcie_driver = {
> > - .probe = mobiveil_pcie_probe,
> > - .driver = {
> > - .name = "mobiveil-pcie",
> > - .of_match_table = mobiveil_pcie_of_match,
> > - .suppress_bind_attrs = true,
> > - },
> > -};
> > -
> > -builtin_platform_driver(mobiveil_pcie_driver);
> > -
> > -MODULE_LICENSE("GPL v2");
> > -MODULE_DESCRIPTION("Mobiveil PCIe host controller driver");
> > -MODULE_AUTHOR("Subrahmanya Lingappa
> <[email protected]>");
> > diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil-plat.c
> > b/drivers/pci/controller/mobiveil/pcie-mobiveil-plat.c
> > new file mode 100644
> > index 000000000000..64c85f852869
> > --- /dev/null
> > +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil-plat.c
> > @@ -0,0 +1,60 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * PCIe host controller driver for Mobiveil PCIe Host controller
> > + *
> > + * Copyright (c) 2018 Mobiveil Inc.
> > + * Copyright 2019 NXP
> > + *
> > + * Author: Subrahmanya Lingappa <[email protected]>
> > + * Recode: Hou Zhiqiang <[email protected]>
>
> Same feedback as before with the recode tag.
>
> > + */
> > +
> > +#include <linux/init.h>
> > +#include <linux/kernel.h>
> > +#include <linux/module.h>
> > +#include <linux/of_pci.h>
> > +#include <linux/pci.h>
> > +#include <linux/platform_device.h>
> > +#include <linux/slab.h>
> > +
> > +#include "pcie-mobiveil.h"
> > +
> > +static int mobiveil_pcie_probe(struct platform_device *pdev) {
> > + struct mobiveil_pcie *pcie;
> > + struct pci_host_bridge *bridge;
> > + struct device *dev = &pdev->dev;
> > +
> > + bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
> > + if (!bridge)
> > + return -ENOMEM;
> > +
> > + pcie = pci_host_bridge_priv(bridge);
> > + pcie->rp.bridge = bridge;
> > +
> > + pcie->pdev = pdev;
> > +
> > + return mobiveil_pcie_host_probe(pcie); }
> > +
> > +static const struct of_device_id mobiveil_pcie_of_match[] = {
> > + {.compatible = "mbvl,gpex40-pcie",},
> > + {},
> > +};
> > +
> > +MODULE_DEVICE_TABLE(of, mobiveil_pcie_of_match);
> > +
> > +static struct platform_driver mobiveil_pcie_driver = {
> > + .probe = mobiveil_pcie_probe,
> > + .driver = {
> > + .name = "mobiveil-pcie",
> > + .of_match_table = mobiveil_pcie_of_match,
> > + .suppress_bind_attrs = true,
> > + },
> > +};
> > +
> > +builtin_platform_driver(mobiveil_pcie_driver);
> > +
> > +MODULE_LICENSE("GPL v2");
> > +MODULE_DESCRIPTION("Mobiveil PCIe host controller driver");
> > +MODULE_AUTHOR("Subrahmanya Lingappa
> <[email protected]>");
> > diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.c
> > b/drivers/pci/controller/mobiveil/pcie-mobiveil.c
> > new file mode 100644
> > index 000000000000..2773f823c9ea
> > --- /dev/null
> > +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.c
> > @@ -0,0 +1,227 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * PCIe host controller driver for Mobiveil PCIe Host controller
> > + *
> > + * Copyright (c) 2018 Mobiveil Inc.
> > + * Copyright 2019 NXP
> > + *
> > + * Author: Subrahmanya Lingappa <[email protected]>
> > + * Recode: Hou Zhiqiang <[email protected]> */
> > +
> > +#include <linux/delay.h>
> > +#include <linux/init.h>
> > +#include <linux/kernel.h>
> > +#include <linux/pci.h>
> > +#include <linux/platform_device.h>
> > +
> > +#include "pcie-mobiveil.h"
> > +
> > +/*
> > + * mobiveil_pcie_sel_page - routine to access paged register
> > + *
> > + * Registers whose address greater than PAGED_ADDR_BNDRY (0xc00) are
> > +paged,
> > + * for this scheme to work extracted higher 6 bits of the offset will
> > +be
> > + * written to pg_sel field of PAB_CTRL register and rest of the lower
> > +10
> > + * bits enabled with PAGED_ADDR_BNDRY are used as offset of the
> register.
> > + */
> > +static void mobiveil_pcie_sel_page(struct mobiveil_pcie *pcie, u8
> > +pg_idx) {
> > + u32 val;
> > +
> > + val = readl(pcie->csr_axi_slave_base + PAB_CTRL);
> > + val &= ~(PAGE_SEL_MASK << PAGE_SEL_SHIFT);
> > + val |= (pg_idx & PAGE_SEL_MASK) << PAGE_SEL_SHIFT;
> > +
> > + writel(val, pcie->csr_axi_slave_base + PAB_CTRL); }
> > +
> > +static void *mobiveil_pcie_comp_addr(struct mobiveil_pcie *pcie, u32
> > +off) {
> > + if (off < PAGED_ADDR_BNDRY) {
> > + /* For directly accessed registers, clear the pg_sel field */
> > + mobiveil_pcie_sel_page(pcie, 0);
> > + return pcie->csr_axi_slave_base + off;
> > + }
> > +
> > + mobiveil_pcie_sel_page(pcie, OFFSET_TO_PAGE_IDX(off));
> > + return pcie->csr_axi_slave_base + OFFSET_TO_PAGE_ADDR(off); }
> > +
> > +static int mobiveil_pcie_read(void __iomem *addr, int size, u32 *val)
> > +{
> > + if ((uintptr_t)addr & (size - 1)) {
> > + *val = 0;
> > + return PCIBIOS_BAD_REGISTER_NUMBER;
> > + }
> > +
> > + switch (size) {
> > + case 4:
> > + *val = readl(addr);
> > + break;
> > + case 2:
> > + *val = readw(addr);
> > + break;
> > + case 1:
> > + *val = readb(addr);
> > + break;
> > + default:
> > + *val = 0;
> > + return PCIBIOS_BAD_REGISTER_NUMBER;
> > + }
> > +
> > + return PCIBIOS_SUCCESSFUL;
> > +}
> > +
> > +static int mobiveil_pcie_write(void __iomem *addr, int size, u32 val)
> > +{
> > + if ((uintptr_t)addr & (size - 1))
> > + return PCIBIOS_BAD_REGISTER_NUMBER;
> > +
> > + switch (size) {
> > + case 4:
> > + writel(val, addr);
> > + break;
> > + case 2:
> > + writew(val, addr);
> > + break;
> > + case 1:
> > + writeb(val, addr);
> > + break;
> > + default:
> > + return PCIBIOS_BAD_REGISTER_NUMBER;
> > + }
> > +
> > + return PCIBIOS_SUCCESSFUL;
> > +}
> > +
> > +u32 mobiveil_csr_read(struct mobiveil_pcie *pcie, u32 off, size_t
> > +size) {
> > + void *addr;
> > + u32 val;
> > + int ret;
> > +
> > + addr = mobiveil_pcie_comp_addr(pcie, off);
> > +
> > + ret = mobiveil_pcie_read(addr, size, &val);
> > + if (ret)
> > + dev_err(&pcie->pdev->dev, "read CSR address failed\n");
> > +
> > + return val;
> > +}
> > +
> > +void mobiveil_csr_write(struct mobiveil_pcie *pcie, u32 val, u32 off,
> > + size_t size)
> > +{
> > + void *addr;
> > + int ret;
> > +
> > + addr = mobiveil_pcie_comp_addr(pcie, off);
> > +
> > + ret = mobiveil_pcie_write(addr, size, val);
> > + if (ret)
> > + dev_err(&pcie->pdev->dev, "write CSR address failed\n"); }
> > +
> > +bool mobiveil_pcie_link_up(struct mobiveil_pcie *pcie) {
> > + return (mobiveil_csr_readl(pcie, LTSSM_STATUS) &
> > + LTSSM_STATUS_L0_MASK) == LTSSM_STATUS_L0; }
> > +
> > +void program_ib_windows(struct mobiveil_pcie *pcie, int win_num,
> > + u64 cpu_addr, u64 pci_addr, u32 type, u64 size) {
> > + u32 value;
> > + u64 size64 = ~(size - 1);
> > +
> > + if (win_num >= pcie->ppio_wins) {
> > + dev_err(&pcie->pdev->dev,
> > + "ERROR: max inbound windows reached !\n");
> > + return;
> > + }
> > +
> > + value = mobiveil_csr_readl(pcie, PAB_PEX_AMAP_CTRL(win_num));
> > + value &= ~(AMAP_CTRL_TYPE_MASK << AMAP_CTRL_TYPE_SHIFT |
> WIN_SIZE_MASK);
> > + value |= type << AMAP_CTRL_TYPE_SHIFT | 1 <<
> AMAP_CTRL_EN_SHIFT |
> > + (lower_32_bits(size64) & WIN_SIZE_MASK);
> > + mobiveil_csr_writel(pcie, value, PAB_PEX_AMAP_CTRL(win_num));
> > +
> > + mobiveil_csr_writel(pcie, upper_32_bits(size64),
> > + PAB_EXT_PEX_AMAP_SIZEN(win_num));
> > +
> > + mobiveil_csr_writel(pcie, lower_32_bits(cpu_addr),
> > + PAB_PEX_AMAP_AXI_WIN(win_num));
> > + mobiveil_csr_writel(pcie, upper_32_bits(cpu_addr),
> > + PAB_EXT_PEX_AMAP_AXI_WIN(win_num));
> > +
> > + mobiveil_csr_writel(pcie, lower_32_bits(pci_addr),
> > + PAB_PEX_AMAP_PEX_WIN_L(win_num));
> > + mobiveil_csr_writel(pcie, upper_32_bits(pci_addr),
> > + PAB_PEX_AMAP_PEX_WIN_H(win_num));
> > +
> > + pcie->ib_wins_configured++;
> > +}
> > +
> > +/*
> > + * routine to program the outbound windows */ void
> > +program_ob_windows(struct mobiveil_pcie *pcie, int win_num,
> > + u64 cpu_addr, u64 pci_addr, u32 type, u64 size) {
> > + u32 value;
> > + u64 size64 = ~(size - 1);
> > +
> > + if (win_num >= pcie->apio_wins) {
> > + dev_err(&pcie->pdev->dev,
> > + "ERROR: max outbound windows reached !\n");
> > + return;
> > + }
> > +
> > + /*
> > + * program Enable Bit to 1, Type Bit to (00) base 2, AXI Window Size Bit
> > + * to 4 KB in PAB_AXI_AMAP_CTRL register
> > + */
> > + value = mobiveil_csr_readl(pcie, PAB_AXI_AMAP_CTRL(win_num));
> > + value &= ~(WIN_TYPE_MASK << WIN_TYPE_SHIFT | WIN_SIZE_MASK);
> > + value |= 1 << WIN_ENABLE_SHIFT | type << WIN_TYPE_SHIFT |
> > + (lower_32_bits(size64) & WIN_SIZE_MASK);
> > + mobiveil_csr_writel(pcie, value, PAB_AXI_AMAP_CTRL(win_num));
> > +
> > + mobiveil_csr_writel(pcie, upper_32_bits(size64),
> > + PAB_EXT_AXI_AMAP_SIZE(win_num));
> > +
> > + /*
> > + * program AXI window base with appropriate value in
> > + * PAB_AXI_AMAP_AXI_WIN0 register
> > + */
> > + mobiveil_csr_writel(pcie,
> > + lower_32_bits(cpu_addr) &
> (~AXI_WINDOW_ALIGN_MASK),
> > + PAB_AXI_AMAP_AXI_WIN(win_num));
> > + mobiveil_csr_writel(pcie, upper_32_bits(cpu_addr),
> > + PAB_EXT_AXI_AMAP_AXI_WIN(win_num));
> > +
> > + mobiveil_csr_writel(pcie, lower_32_bits(pci_addr),
> > + PAB_AXI_AMAP_PEX_WIN_L(win_num));
> > + mobiveil_csr_writel(pcie, upper_32_bits(pci_addr),
> > + PAB_AXI_AMAP_PEX_WIN_H(win_num));
> > +
> > + pcie->ob_wins_configured++;
> > +}
> > +
> > +int mobiveil_bringup_link(struct mobiveil_pcie *pcie) {
> > + int retries;
> > +
> > + /* check if the link is up or not */
> > + for (retries = 0; retries < LINK_WAIT_MAX_RETRIES; retries++) {
> > + if (mobiveil_pcie_link_up(pcie))
> > + return 0;
> > +
> > + usleep_range(LINK_WAIT_MIN, LINK_WAIT_MAX);
> > + }
> > +
> > + dev_err(&pcie->pdev->dev, "link never came up\n");
> > +
> > + return -ETIMEDOUT;
> > +}
> > diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > new file mode 100644
> > index 000000000000..e3148078e9dd
> > --- /dev/null
> > +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > @@ -0,0 +1,178 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/*
> > + * PCIe host controller driver for Mobiveil PCIe Host controller
> > + *
> > + * Copyright (c) 2018 Mobiveil Inc.
> > + * Copyright 2019 NXP
> > + *
> > + * Author: Subrahmanya Lingappa <[email protected]>
> > + * Recode: Hou Zhiqiang <[email protected]> */
> > +
> > +#ifndef _PCIE_MOBIVEIL_H
> > +#define _PCIE_MOBIVEIL_H
> > +
> > +#include <linux/pci.h>
> > +#include <linux/irq.h>
> > +#include <linux/msi.h>
> > +#include "../../pci.h"
> > +
> > +/* register offsets and bit positions */
> > +
> > +/*
> > + * translation tables are grouped into windows, each window registers
> > +are
> > + * grouped into blocks of 4 or 16 registers each */
> > +#define PAB_REG_BLOCK_SIZE 16
> > +#define PAB_EXT_REG_BLOCK_SIZE 4
> > +
> > +#define PAB_REG_ADDR(offset, win) \
> > + (offset + (win * PAB_REG_BLOCK_SIZE))
> > +#define PAB_EXT_REG_ADDR(offset, win) \
> > + (offset + (win * PAB_EXT_REG_BLOCK_SIZE))
> > +
> > +#define LTSSM_STATUS 0x0404
> > +#define LTSSM_STATUS_L0_MASK 0x3f
> > +#define LTSSM_STATUS_L0 0x2d
> > +
> > +#define PAB_CTRL 0x0808
> > +#define AMBA_PIO_ENABLE_SHIFT 0
> > +#define PEX_PIO_ENABLE_SHIFT 1
> > +#define PAGE_SEL_SHIFT 13
> > +#define PAGE_SEL_MASK 0x3f
> > +#define PAGE_LO_MASK 0x3ff
> > +#define PAGE_SEL_OFFSET_SHIFT 10
> > +
> > +#define PAB_AXI_PIO_CTRL 0x0840
> > +#define APIO_EN_MASK 0xf
> > +
> > +#define PAB_PEX_PIO_CTRL 0x08c0
> > +#define PIO_ENABLE_SHIFT 0
> > +
> > +#define PAB_INTP_AMBA_MISC_ENB 0x0b0c
> > +#define PAB_INTP_AMBA_MISC_STAT 0x0b1c
> > +#define PAB_INTP_INTX_MASK 0x01e0
> > +#define PAB_INTP_MSI_MASK 0x8
> > +
> > +#define PAB_AXI_AMAP_CTRL(win) PAB_REG_ADDR(0x0ba0, win)
> > +#define WIN_ENABLE_SHIFT 0
> > +#define WIN_TYPE_SHIFT 1
> > +#define WIN_TYPE_MASK 0x3
> > +#define WIN_SIZE_MASK 0xfffffc00
> > +
> > +#define PAB_EXT_AXI_AMAP_SIZE(win) PAB_EXT_REG_ADDR(0xbaf0,
> win)
> > +
> > +#define PAB_EXT_AXI_AMAP_AXI_WIN(win)
> PAB_EXT_REG_ADDR(0x80a0, win)
> > +#define PAB_AXI_AMAP_AXI_WIN(win) PAB_REG_ADDR(0x0ba4, win)
> > +#define AXI_WINDOW_ALIGN_MASK 3
> > +
> > +#define PAB_AXI_AMAP_PEX_WIN_L(win) PAB_REG_ADDR(0x0ba8,
> win)
> > +#define PAB_BUS_SHIFT 24
> > +#define PAB_DEVICE_SHIFT 19
> > +#define PAB_FUNCTION_SHIFT 16
> > +
> > +#define PAB_AXI_AMAP_PEX_WIN_H(win) PAB_REG_ADDR(0x0bac,
> win)
> > +#define PAB_INTP_AXI_PIO_CLASS 0x474
> > +
> > +#define PAB_PEX_AMAP_CTRL(win) PAB_REG_ADDR(0x4ba0,
> win)
> > +#define AMAP_CTRL_EN_SHIFT 0
> > +#define AMAP_CTRL_TYPE_SHIFT 1
> > +#define AMAP_CTRL_TYPE_MASK 3
> > +
> > +#define PAB_EXT_PEX_AMAP_SIZEN(win) PAB_EXT_REG_ADDR(0xbef0,
> win)
> > +#define PAB_EXT_PEX_AMAP_AXI_WIN(win)
> PAB_EXT_REG_ADDR(0xb4a0, win)
> > +#define PAB_PEX_AMAP_AXI_WIN(win) PAB_REG_ADDR(0x4ba4, win)
> > +#define PAB_PEX_AMAP_PEX_WIN_L(win) PAB_REG_ADDR(0x4ba8,
> win)
> > +#define PAB_PEX_AMAP_PEX_WIN_H(win) PAB_REG_ADDR(0x4bac,
> win)
> > +
> > +/* starting offset of INTX bits in status register */
> > +#define PAB_INTX_START 5
> > +
> > +/* supported number of MSI interrupts */
> > +#define PCI_NUM_MSI 16
> > +
> > +/* MSI registers */
> > +#define MSI_BASE_LO_OFFSET 0x04
> > +#define MSI_BASE_HI_OFFSET 0x08
> > +#define MSI_SIZE_OFFSET 0x0c
> > +#define MSI_ENABLE_OFFSET 0x14
> > +#define MSI_STATUS_OFFSET 0x18
> > +#define MSI_DATA_OFFSET 0x20
> > +#define MSI_ADDR_L_OFFSET 0x24
> > +#define MSI_ADDR_H_OFFSET 0x28
> > +
> > +/* outbound and inbound window definitions */
> > +#define WIN_NUM_0 0
> > +#define WIN_NUM_1 1
> > +#define CFG_WINDOW_TYPE 0
> > +#define IO_WINDOW_TYPE 1
> > +#define MEM_WINDOW_TYPE 2
> > +#define IB_WIN_SIZE ((u64)256 * 1024 * 1024 * 1024)
> > +#define MAX_PIO_WINDOWS 8
> > +
> > +/* Parameters for the waiting for link up routine */
> > +#define LINK_WAIT_MAX_RETRIES 10
> > +#define LINK_WAIT_MIN 90000
> > +#define LINK_WAIT_MAX 100000
> > +
> > +#define PAGED_ADDR_BNDRY 0xc00
> > +#define OFFSET_TO_PAGE_ADDR(off) \
> > + ((off & PAGE_LO_MASK) | PAGED_ADDR_BNDRY)
> > +#define OFFSET_TO_PAGE_IDX(off) \
> > + ((off >> PAGE_SEL_OFFSET_SHIFT) & PAGE_SEL_MASK)
> > +
> > +struct mobiveil_msi { /* MSI information */
> > + struct mutex lock; /* protect bitmap variable */
> > + struct irq_domain *msi_domain;
> > + struct irq_domain *dev_domain;
> > + phys_addr_t msi_pages_phys;
> > + int num_of_vectors;
> > + DECLARE_BITMAP(msi_irq_in_use, PCI_NUM_MSI); };
> > +
> > +struct root_port {
> > + char root_bus_nr;
> > + void __iomem *config_axi_slave_base; /* endpoint config base */
> > + struct resource *ob_io_res;
> > + int irq;
> > + raw_spinlock_t intx_mask_lock;
> > + struct irq_domain *intx_domain;
> > + struct mobiveil_msi msi;
> > + struct pci_host_bridge *bridge;
> > +};
> > +
> > +struct mobiveil_pcie {
> > + struct platform_device *pdev;
> > + void __iomem *csr_axi_slave_base; /* root port config base */
> > + void __iomem *apb_csr_base; /* MSI register base */
> > + phys_addr_t pcie_reg_base; /* Physical PCIe Controller Base */
> > + int apio_wins;
> > + int ppio_wins;
> > + int ob_wins_configured; /* configured outbound windows */
> > + int ib_wins_configured; /* configured inbound windows */
> > + struct root_port rp;
> > +};
> > +
> > +int mobiveil_pcie_host_probe(struct mobiveil_pcie *pcie); bool
> > +mobiveil_pcie_link_up(struct mobiveil_pcie *pcie); int
> > +mobiveil_bringup_link(struct mobiveil_pcie *pcie); void
> > +program_ob_windows(struct mobiveil_pcie *pcie, int win_num, u64
> cpu_addr,
> > + u64 pci_addr, u32 type, u64 size); void
> program_ib_windows(struct
> > +mobiveil_pcie *pcie, int win_num, u64 cpu_addr,
> > + u64 pci_addr, u32 type, u64 size);
> > +u32 mobiveil_csr_read(struct mobiveil_pcie *pcie, u32 off, size_t
> > +size); void mobiveil_csr_write(struct mobiveil_pcie *pcie, u32 val, u32 off,
> > + size_t size);
> > +
> > +static inline u32 mobiveil_csr_readl(struct mobiveil_pcie *pcie, u32
> > +off) {
> > + return mobiveil_csr_read(pcie, off, 0x4); }
> > +
> > +static inline void mobiveil_csr_writel(struct mobiveil_pcie *pcie, u32 val,
> > + u32 off)
> > +{
> > + mobiveil_csr_write(pcie, val, off, 0x4); }
> > +
> > +#endif /* _PCIE_MOBIVEIL_H */
> > --
> > 2.17.1
> >
Hi Andrew,
Thanks a lot for your comments!
> -----Original Message-----
> From: Andrew Murray <[email protected]>
> Sent: 2020??1??13?? 19:20
> To: Z.q. Hou <[email protected]>
> Cc: [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; Leo Li
> <[email protected]>; [email protected];
> [email protected]; [email protected]; Mingkai Hu
> <[email protected]>; M.h. Lian <[email protected]>; Xiaowei Bao
> <[email protected]>
> Subject: Re: [PATCHv9 05/12] PCI: mobiveil: Add callback function for
> interrupt initialization
>
> On Wed, Nov 20, 2019 at 03:45:50AM +0000, Z.q. Hou wrote:
> > From: Hou Zhiqiang <[email protected]>
> >
> > The Mobiveil GPEX internal MSI/INTx controller may not be used by
> > other platforms in which the Mobiveil GPEX is integrated.
> > This patch is to allow these platforms to implement their specific
> > interrupt initialization.
> >
> > Signed-off-by: Hou Zhiqiang <[email protected]>
> > ---
> > V9:
> > - New patch splited from the #1 of V8 patches to make it easy to review.
> >
> > drivers/pci/controller/mobiveil/pcie-mobiveil-host.c | 3 +++
> > drivers/pci/controller/mobiveil/pcie-mobiveil.h | 7 +++++++
> > 2 files changed, 10 insertions(+)
> >
> > diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
> > b/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
> > index 2cc424e78d33..3cd93df6fe6e 100644
> > --- a/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
> > +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
> > @@ -507,6 +507,9 @@ static int mobiveil_pcie_interrupt_init(struct
> mobiveil_pcie *pcie)
> > struct resource *res;
> > int ret;
> >
> > + if (rp->ops->interrupt_init)
> > + return rp->ops->interrupt_init(pcie);
> > +
>
> This may be cleaner if you have a helper function named
> "mobiveil_pcie_interrupt_init" where it either calls interrupt_init if present or
> calls this current function renamed to
> "mobiveil_pcie_integrated_interrupt_init"
> or similar.
>
> A bit like the DWC dw_pcie_rd_own_conf function.
Good suggestion! Will change in v10.
Thanks,
Zhiqiang
>
> Thanks,
>
> Andrew Murray
>
> > /* map MSI config resource */
> > res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
> "apb_csr");
> > pcie->apb_csr_base = devm_pci_remap_cfg_resource(dev, res); diff
> > --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > index e3148078e9dd..18d85806a7fc 100644
> > --- a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > @@ -130,10 +130,17 @@ struct mobiveil_msi { /* MSI
> information */
> > DECLARE_BITMAP(msi_irq_in_use, PCI_NUM_MSI); };
> >
> > +struct mobiveil_pcie;
> > +
> > +struct mobiveil_rp_ops {
> > + int (*interrupt_init)(struct mobiveil_pcie *pcie); };
> > +
> > struct root_port {
> > char root_bus_nr;
> > void __iomem *config_axi_slave_base; /* endpoint config base */
> > struct resource *ob_io_res;
> > + struct mobiveil_rp_ops *ops;
> > int irq;
> > raw_spinlock_t intx_mask_lock;
> > struct irq_domain *intx_domain;
> > --
> > 2.17.1
> >
Hi Andrew,
Thanks a lot for your comments!
> -----Original Message-----
> From: Andrew Murray <[email protected]>
> Sent: 2020??1??13?? 19:32
> To: Z.q. Hou <[email protected]>
> Cc: [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; Leo Li
> <[email protected]>; [email protected];
> [email protected]; [email protected]; Mingkai Hu
> <[email protected]>; M.h. Lian <[email protected]>; Xiaowei Bao
> <[email protected]>
> Subject: Re: [PATCHv9 08/12] PCI: mobiveil: Add 8-bit and 16-bit CSR register
> accessors
>
> On Wed, Nov 20, 2019 at 03:46:10AM +0000, Z.q. Hou wrote:
> > From: Hou Zhiqiang <[email protected]>
> >
> > There are some 8-bit and 16-bit registers in PCIe configuration space,
> > so add these accessors accordingly.
> >
> > Signed-off-by: Hou Zhiqiang <[email protected]>
> > Reviewed-by: Minghuan Lian <[email protected]>
> > Reviewed-by: Subrahmanya Lingappa <[email protected]>
> > ---
> > V9:
> > - No change
> >
> > .../pci/controller/mobiveil/pcie-mobiveil.h | 23
> +++++++++++++++++++
> > 1 file changed, 23 insertions(+)
> >
> > diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > index 37116c2a19fe..750a7fd95bc1 100644
> > --- a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > @@ -182,10 +182,33 @@ static inline u32 mobiveil_csr_readl(struct
> mobiveil_pcie *pcie, u32 off)
> > return mobiveil_csr_read(pcie, off, 0x4); }
> >
> > +static inline u32 mobiveil_csr_readw(struct mobiveil_pcie *pcie, u32
> > +off) {
> > + return mobiveil_csr_read(pcie, off, 0x2); }
> > +
> > +static inline u32 mobiveil_csr_readb(struct mobiveil_pcie *pcie, u32
> > +off) {
> > + return mobiveil_csr_read(pcie, off, 0x1); }
>
> Do you think the above two return types should reflect the size of the access?
Will change in v10.
Thanks,
Zhiqiang
>
> Thanks,
>
> Andrew Murray
>
> > +
> > +
> > static inline void mobiveil_csr_writel(struct mobiveil_pcie *pcie, u32 val,
> > u32 off)
> > {
> > mobiveil_csr_write(pcie, val, off, 0x4); }
> >
> > +static inline void mobiveil_csr_writew(struct mobiveil_pcie *pcie, u32 val,
> > + u32 off)
> > +{
> > + mobiveil_csr_write(pcie, val, off, 0x2); }
> > +
> > +static inline void mobiveil_csr_writeb(struct mobiveil_pcie *pcie, u32 val,
> > + u32 off)
> > +{
> > + mobiveil_csr_write(pcie, val, off, 0x1); }
> > +
> > #endif /* _PCIE_MOBIVEIL_H */
> > --
> > 2.17.1
> >
Hi Andrew,
Thanks a lot for your comments!
> -----Original Message-----
> From: Andrew Murray <[email protected]>
> Sent: 2020年1月13日 20:03
> To: Z.q. Hou <[email protected]>
> Cc: [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; Leo Li
> <[email protected]>; [email protected];
> [email protected]; [email protected]; Mingkai Hu
> <[email protected]>; M.h. Lian <[email protected]>; Xiaowei Bao
> <[email protected]>
> Subject: Re: [PATCHv9 10/12] PCI: mobiveil: Add PCIe Gen4 RC driver for NXP
> Layerscape SoCs
>
> On Wed, Nov 20, 2019 at 03:46:23AM +0000, Z.q. Hou wrote:
> > From: Hou Zhiqiang <[email protected]>
> >
> > This PCIe controller is based on the Mobiveil GPEX IP, which is
> > compatible with the PCI Express™ Base Specification, Revision 4.0.
> >
> > Signed-off-by: Hou Zhiqiang <[email protected]>
> > Reviewed-by: Minghuan Lian <[email protected]>
> > ---
> > drivers/pci/controller/mobiveil/Kconfig | 10 +
> > drivers/pci/controller/mobiveil/Makefile | 1 +
> > .../mobiveil/pcie-layerscape-gen4.c | 274
> ++++++++++++++++++
> > .../pci/controller/mobiveil/pcie-mobiveil.h | 16 +-
> > 4 files changed, 299 insertions(+), 2 deletions(-) create mode
> > 100644 drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c
> >
> > diff --git a/drivers/pci/controller/mobiveil/Kconfig
> > b/drivers/pci/controller/mobiveil/Kconfig
> > index 64343c07bfed..c823be8dab1c 100644
> > --- a/drivers/pci/controller/mobiveil/Kconfig
> > +++ b/drivers/pci/controller/mobiveil/Kconfig
> > @@ -21,4 +21,14 @@ config PCIE_MOBIVEIL_PLAT
> > Soft IP. It has up to 8 outbound and inbound windows
> > for address translation and it is a PCIe Gen4 IP.
> >
> > +config PCIE_LAYERSCAPE_GEN4
> > + bool "Freescale Layerscape PCIe Gen4 controller"
> > + depends on PCI
> > + depends on OF && (ARM64 || ARCH_LAYERSCAPE)
> > + depends on PCI_MSI_IRQ_DOMAIN
> > + select PCIE_MOBIVEIL_HOST
> > + help
> > + Say Y here if you want PCIe Gen4 controller support on
> > + Layerscape SoCs. The PCIe controller can work in RC or
> > + EP mode according to RCW[HOST_AGT_PEX] setting.
>
> I think you can remove the last sentence - it doesn't give any value to users of
> KConfig.
OK, will remove it in v10.
>
>
> > endmenu
> > diff --git a/drivers/pci/controller/mobiveil/Makefile
> > b/drivers/pci/controller/mobiveil/Makefile
> > index 9fb6d1c6504d..99d879de32d6 100644
> > --- a/drivers/pci/controller/mobiveil/Makefile
> > +++ b/drivers/pci/controller/mobiveil/Makefile
> > @@ -2,3 +2,4 @@
> > obj-$(CONFIG_PCIE_MOBIVEIL) += pcie-mobiveil.o
> > obj-$(CONFIG_PCIE_MOBIVEIL_HOST) += pcie-mobiveil-host.o
> > obj-$(CONFIG_PCIE_MOBIVEIL_PLAT) += pcie-mobiveil-plat.o
> > +obj-$(CONFIG_PCIE_LAYERSCAPE_GEN4) += pcie-layerscape-gen4.o
> > diff --git a/drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c
> > b/drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c
> > new file mode 100644
> > index 000000000000..6c0d3e2532db
> > --- /dev/null
> > +++ b/drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c
> > @@ -0,0 +1,274 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * PCIe Gen4 host controller driver for NXP Layerscape SoCs
> > + *
> > + * Copyright 2019 NXP
> > + *
> > + * Author: Zhiqiang Hou <[email protected]> */
> > +
> > +#include <linux/kernel.h>
> > +#include <linux/interrupt.h>
> > +#include <linux/init.h>
> > +#include <linux/of_pci.h>
> > +#include <linux/of_platform.h>
> > +#include <linux/of_irq.h>
> > +#include <linux/of_address.h>
> > +#include <linux/pci.h>
> > +#include <linux/platform_device.h>
> > +#include <linux/resource.h>
> > +#include <linux/mfd/syscon.h>
> > +#include <linux/regmap.h>
> > +
> > +#include "pcie-mobiveil.h"
> > +
> > +/* LUT and PF control registers */
> > +#define PCIE_LUT_OFF 0x80000
> > +#define PCIE_PF_OFF 0xc0000
> > +#define PCIE_PF_INT_STAT 0x18
> > +#define PF_INT_STAT_PABRST BIT(31)
> > +
> > +#define PCIE_PF_DBG 0x7fc
> > +#define PF_DBG_LTSSM_MASK 0x3f
> > +#define PF_DBG_LTSSM_L0 0x2d /* L0 state */
> > +#define PF_DBG_WE BIT(31)
> > +#define PF_DBG_PABR BIT(27)
> > +
> > +#define to_ls_pcie_g4(x) platform_get_drvdata((x)->pdev)
> > +
> > +struct ls_pcie_g4 {
> > + struct mobiveil_pcie pci;
> > + struct delayed_work dwork;
> > + int irq;
> > +};
> > +
> > +static inline u32 ls_pcie_g4_lut_readl(struct ls_pcie_g4 *pcie, u32
> > +off) {
> > + return ioread32(pcie->pci.csr_axi_slave_base + PCIE_LUT_OFF + off);
> > +}
> > +
> > +static inline void ls_pcie_g4_lut_writel(struct ls_pcie_g4 *pcie,
> > + u32 off, u32 val)
> > +{
> > + iowrite32(val, pcie->pci.csr_axi_slave_base + PCIE_LUT_OFF + off); }
> > +
> > +static inline u32 ls_pcie_g4_pf_readl(struct ls_pcie_g4 *pcie, u32
> > +off) {
> > + return ioread32(pcie->pci.csr_axi_slave_base + PCIE_PF_OFF + off); }
> > +
> > +static inline void ls_pcie_g4_pf_writel(struct ls_pcie_g4 *pcie,
> > + u32 off, u32 val)
> > +{
> > + iowrite32(val, pcie->pci.csr_axi_slave_base + PCIE_PF_OFF + off); }
> > +
> > +static bool ls_pcie_g4_is_bridge(struct ls_pcie_g4 *pcie) {
> > + struct mobiveil_pcie *mv_pci = &pcie->pci;
> > + u32 header_type;
> > +
> > + header_type = mobiveil_csr_readb(mv_pci, PCI_HEADER_TYPE);
> > + header_type &= 0x7f;
> > +
> > + return header_type == PCI_HEADER_TYPE_BRIDGE; }
> > +
> > +static int ls_pcie_g4_link_up(struct mobiveil_pcie *pci) {
> > + struct ls_pcie_g4 *pcie = to_ls_pcie_g4(pci);
> > + u32 state;
> > +
> > + state = ls_pcie_g4_pf_readl(pcie, PCIE_PF_DBG);
> > + state = state & PF_DBG_LTSSM_MASK;
> > +
> > + if (state == PF_DBG_LTSSM_L0)
> > + return 1;
> > +
> > + return 0;
> > +}
> > +
> > +static void ls_pcie_g4_disable_interrupt(struct ls_pcie_g4 *pcie) {
> > + struct mobiveil_pcie *mv_pci = &pcie->pci;
> > +
> > + mobiveil_csr_writel(mv_pci, 0, PAB_INTP_AMBA_MISC_ENB); }
> > +
> > +static void ls_pcie_g4_enable_interrupt(struct ls_pcie_g4 *pcie) {
> > + struct mobiveil_pcie *mv_pci = &pcie->pci;
> > + u32 val;
> > +
> > + /* Clear the interrupt status */
> > + mobiveil_csr_writel(mv_pci, 0xffffffff, PAB_INTP_AMBA_MISC_STAT);
> > +
> > + val = PAB_INTP_INTX_MASK | PAB_INTP_MSI | PAB_INTP_RESET |
> > + PAB_INTP_PCIE_UE | PAB_INTP_IE_PMREDI | PAB_INTP_IE_EC;
> > + mobiveil_csr_writel(mv_pci, val, PAB_INTP_AMBA_MISC_ENB); }
> > +
> > +static void ls_pcie_g4_reinit_hw(struct ls_pcie_g4 *pcie) {
> > + struct mobiveil_pcie *mv_pci = &pcie->pci;
> > + struct device *dev = &mv_pci->pdev->dev;
> > + u32 val, act_stat;
> > + int to = 100;
> > +
> > + /* Poll for pab_csb_reset to set and PAB activity to clear */
> > + do {
> > + usleep_range(10, 15);
> > + val = ls_pcie_g4_pf_readl(pcie, PCIE_PF_INT_STAT);
> > + act_stat = mobiveil_csr_readl(mv_pci, PAB_ACTIVITY_STAT);
> > + } while (((val & PF_INT_STAT_PABRST) == 0 || act_stat) && to--);
> > + if (to < 0) {
> > + dev_err(dev, "Poll PABRST&PABACT timeout\n");
> > + return;
>
> If a timeout happens here - the caller has no idea this has happened and yet
> the following work doesn't get done. Isn't this a problem?
Will change the return value type to 'int' in v10, so that the caller can know the fail.
>
> > + }
> > +
> > + /* clear PEX_RESET bit in PEX_PF0_DBG register */
> > + val = ls_pcie_g4_pf_readl(pcie, PCIE_PF_DBG);
> > + val |= PF_DBG_WE;
> > + ls_pcie_g4_pf_writel(pcie, PCIE_PF_DBG, val);
> > +
> > + val = ls_pcie_g4_pf_readl(pcie, PCIE_PF_DBG);
> > + val |= PF_DBG_PABR;
> > + ls_pcie_g4_pf_writel(pcie, PCIE_PF_DBG, val);
> > +
> > + val = ls_pcie_g4_pf_readl(pcie, PCIE_PF_DBG);
> > + val &= ~PF_DBG_WE;
> > + ls_pcie_g4_pf_writel(pcie, PCIE_PF_DBG, val);
> > +
> > + mobiveil_host_init(mv_pci, true);
>
> Can mobiveil_host_init fail?
It should not fail, only register programming operations were left in this function.
>
> > +
> > + to = 100;
> > + while (!ls_pcie_g4_link_up(mv_pci) && to--)
> > + usleep_range(200, 250);
> > + if (to < 0)
> > + dev_err(dev, "PCIe link training timeout\n"); }
> > +
> > +static irqreturn_t ls_pcie_g4_isr(int irq, void *dev_id) {
> > + struct ls_pcie_g4 *pcie = (struct ls_pcie_g4 *)dev_id;
> > + struct mobiveil_pcie *mv_pci = &pcie->pci;
> > + u32 val;
> > +
> > + val = mobiveil_csr_readl(mv_pci, PAB_INTP_AMBA_MISC_STAT);
> > + if (!val)
> > + return IRQ_NONE;
> > +
> > + if (val & PAB_INTP_RESET) {
>
> Can you explain why this is needed (perhaps also in the cover letter)?
The hot reset will result in the RC crash, so need the ISR to reset the RC.
>
> > + ls_pcie_g4_disable_interrupt(pcie);
> > + schedule_delayed_work(&pcie->dwork, msecs_to_jiffies(1));
> > + }
> > +
> > + mobiveil_csr_writel(mv_pci, val, PAB_INTP_AMBA_MISC_STAT);
> > +
> > + return IRQ_HANDLED;
> > +}
> > +
> > +static int ls_pcie_g4_interrupt_init(struct mobiveil_pcie *mv_pci) {
> > + struct ls_pcie_g4 *pcie = to_ls_pcie_g4(mv_pci);
> > + struct platform_device *pdev = mv_pci->pdev;
> > + struct device *dev = &pdev->dev;
> > + int ret;
> > +
> > + pcie->irq = platform_get_irq_byname(pdev, "intr");
> > + if (pcie->irq < 0) {
> > + dev_err(dev, "Can't get 'intr' IRQ, errno = %d\n", pcie->irq);
> > + return pcie->irq;
> > + }
> > + ret = devm_request_irq(dev, pcie->irq, ls_pcie_g4_isr,
> > + IRQF_SHARED, pdev->name, pcie);
> > + if (ret) {
> > + dev_err(dev, "Can't register PCIe IRQ, errno = %d\n", ret);
> > + return ret;
> > + }
> > +
> > + return 0;
> > +}
> > +
> > +static void ls_pcie_g4_reset(struct work_struct *work) {
> > + struct delayed_work *dwork = container_of(work, struct delayed_work,
> > + work);
> > + struct ls_pcie_g4 *pcie = container_of(dwork, struct ls_pcie_g4, dwork);
> > + struct mobiveil_pcie *mv_pci = &pcie->pci;
> > + u16 ctrl;
> > +
> > + ctrl = mobiveil_csr_readw(mv_pci, PCI_BRIDGE_CONTROL);
> > + ctrl &= ~PCI_BRIDGE_CTL_BUS_RESET;
> > + mobiveil_csr_writew(mv_pci, ctrl, PCI_BRIDGE_CONTROL);
> > + ls_pcie_g4_reinit_hw(pcie);
> > + ls_pcie_g4_enable_interrupt(pcie);
> > +}
> > +
> > +static struct mobiveil_rp_ops ls_pcie_g4_rp_ops = {
> > + .interrupt_init = ls_pcie_g4_interrupt_init, };
> > +
> > +static const struct mobiveil_pab_ops ls_pcie_g4_pab_ops = {
> > + .link_up = ls_pcie_g4_link_up,
> > +};
> > +
> > +static int __init ls_pcie_g4_probe(struct platform_device *pdev) {
> > + struct device *dev = &pdev->dev;
> > + struct pci_host_bridge *bridge;
> > + struct mobiveil_pcie *mv_pci;
> > + struct ls_pcie_g4 *pcie;
> > + struct device_node *np = dev->of_node;
> > + int ret;
> > +
> > + if (!of_parse_phandle(np, "msi-parent", 0)) {
> > + dev_err(dev, "Failed to find msi-parent\n");
> > + return -EINVAL;
> > + }
> > +
> > + bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
> > + if (!bridge)
> > + return -ENOMEM;
> > +
> > + pcie = pci_host_bridge_priv(bridge);
> > + mv_pci = &pcie->pci;
> > +
> > + mv_pci->pdev = pdev;
> > + mv_pci->ops = &ls_pcie_g4_pab_ops;
> > + mv_pci->rp.ops = &ls_pcie_g4_rp_ops;
> > + mv_pci->rp.bridge = bridge;
> > +
> > + platform_set_drvdata(pdev, pcie);
> > +
> > + INIT_DELAYED_WORK(&pcie->dwork, ls_pcie_g4_reset);
> > +
> > + ret = mobiveil_pcie_host_probe(mv_pci);
> > + if (ret) {
> > + dev_err(dev, "Fail to probe\n");
> > + return ret;
> > + }
> > +
> > + if (!ls_pcie_g4_is_bridge(pcie))
>
> Is this a check that could apply to all host bridge drivers and thus live in
> mobiveil_pcie_host_probe?
Yes, will do in v10.
>
> > + return -ENODEV;
> > +
> > + ls_pcie_g4_enable_interrupt(pcie);
>
> Is there an issue here in that we enable interrupts *after* telling the kernel
> about our controller? (Same applies for bailing if the IP isn't a bridge).
Andrew, I don't understand the issue, can you help to explain?
Thanks,
Zhiqiang
>
> Thanks,
>
> Andrew Murray
>
> > +
> > + return 0;
> > +}
> > +
> > +static const struct of_device_id ls_pcie_g4_of_match[] = {
> > + { .compatible = "fsl,lx2160a-pcie", },
> > + { },
> > +};
> > +
> > +static struct platform_driver ls_pcie_g4_driver = {
> > + .driver = {
> > + .name = "layerscape-pcie-gen4",
> > + .of_match_table = ls_pcie_g4_of_match,
> > + .suppress_bind_attrs = true,
> > + },
> > +};
> > +
> > +builtin_platform_driver_probe(ls_pcie_g4_driver, ls_pcie_g4_probe);
> > diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > index 750a7fd95bc1..c57a68d2bac4 100644
> > --- a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > @@ -43,6 +43,8 @@
> > #define PAGE_LO_MASK 0x3ff
> > #define PAGE_SEL_OFFSET_SHIFT 10
> >
> > +#define PAB_ACTIVITY_STAT 0x81c
> > +
> > #define PAB_AXI_PIO_CTRL 0x0840
> > #define APIO_EN_MASK 0xf
> >
> > @@ -51,8 +53,18 @@
> >
> > #define PAB_INTP_AMBA_MISC_ENB 0x0b0c
> > #define PAB_INTP_AMBA_MISC_STAT 0x0b1c
> > -#define PAB_INTP_INTX_MASK 0x01e0
> > -#define PAB_INTP_MSI_MASK 0x8
> > +#define PAB_INTP_RESET BIT(1)
> > +#define PAB_INTP_MSI BIT(3)
> > +#define PAB_INTP_INTA BIT(5)
> > +#define PAB_INTP_INTB BIT(6)
> > +#define PAB_INTP_INTC BIT(7)
> > +#define PAB_INTP_INTD BIT(8)
> > +#define PAB_INTP_PCIE_UE BIT(9)
> > +#define PAB_INTP_IE_PMREDI BIT(29)
> > +#define PAB_INTP_IE_EC BIT(30)
> > +#define PAB_INTP_MSI_MASK PAB_INTP_MSI
> > +#define PAB_INTP_INTX_MASK (PAB_INTP_INTA |
> PAB_INTP_INTB |\
> > + PAB_INTP_INTC | PAB_INTP_INTD)
> >
> > #define PAB_AXI_AMAP_CTRL(win) PAB_REG_ADDR(0x0ba0,
> win)
> > #define WIN_ENABLE_SHIFT 0
> > --
> > 2.17.1
> >
On Thu, Feb 06, 2020 at 01:45:57PM +0000, Z.q. Hou wrote:
> Hi Andrew,
>
> Thanks a lot for your comments!
>
> > -----Original Message-----
> > From: Andrew Murray <[email protected]>
> > Sent: 2020年1月13日 20:03
> > To: Z.q. Hou <[email protected]>
> > Cc: [email protected]; [email protected];
> > [email protected]; [email protected];
> > [email protected]; [email protected]; [email protected];
> > [email protected]; [email protected];
> > [email protected]; [email protected]; Leo Li
> > <[email protected]>; [email protected];
> > [email protected]; [email protected]; Mingkai Hu
> > <[email protected]>; M.h. Lian <[email protected]>; Xiaowei Bao
> > <[email protected]>
> > Subject: Re: [PATCHv9 10/12] PCI: mobiveil: Add PCIe Gen4 RC driver for NXP
> > Layerscape SoCs
> >
> > On Wed, Nov 20, 2019 at 03:46:23AM +0000, Z.q. Hou wrote:
> > > From: Hou Zhiqiang <[email protected]>
> > >
> > > This PCIe controller is based on the Mobiveil GPEX IP, which is
> > > compatible with the PCI Express™ Base Specification, Revision 4.0.
> > >
> > > Signed-off-by: Hou Zhiqiang <[email protected]>
> > > Reviewed-by: Minghuan Lian <[email protected]>
> > > ---
> > > drivers/pci/controller/mobiveil/Kconfig | 10 +
> > > drivers/pci/controller/mobiveil/Makefile | 1 +
> > > .../mobiveil/pcie-layerscape-gen4.c | 274
> > ++++++++++++++++++
> > > .../pci/controller/mobiveil/pcie-mobiveil.h | 16 +-
> > > 4 files changed, 299 insertions(+), 2 deletions(-) create mode
> > > 100644 drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c
> > >
> > > diff --git a/drivers/pci/controller/mobiveil/Kconfig
> > > b/drivers/pci/controller/mobiveil/Kconfig
> > > index 64343c07bfed..c823be8dab1c 100644
> > > --- a/drivers/pci/controller/mobiveil/Kconfig
> > > +++ b/drivers/pci/controller/mobiveil/Kconfig
> > > @@ -21,4 +21,14 @@ config PCIE_MOBIVEIL_PLAT
> > > Soft IP. It has up to 8 outbound and inbound windows
> > > for address translation and it is a PCIe Gen4 IP.
> > >
> > > +config PCIE_LAYERSCAPE_GEN4
> > > + bool "Freescale Layerscape PCIe Gen4 controller"
> > > + depends on PCI
> > > + depends on OF && (ARM64 || ARCH_LAYERSCAPE)
> > > + depends on PCI_MSI_IRQ_DOMAIN
> > > + select PCIE_MOBIVEIL_HOST
> > > + help
> > > + Say Y here if you want PCIe Gen4 controller support on
> > > + Layerscape SoCs. The PCIe controller can work in RC or
> > > + EP mode according to RCW[HOST_AGT_PEX] setting.
> >
> > I think you can remove the last sentence - it doesn't give any value to users of
> > KConfig.
>
> OK, will remove it in v10.
>
> >
> >
> > > endmenu
> > > diff --git a/drivers/pci/controller/mobiveil/Makefile
> > > b/drivers/pci/controller/mobiveil/Makefile
> > > index 9fb6d1c6504d..99d879de32d6 100644
> > > --- a/drivers/pci/controller/mobiveil/Makefile
> > > +++ b/drivers/pci/controller/mobiveil/Makefile
> > > @@ -2,3 +2,4 @@
> > > obj-$(CONFIG_PCIE_MOBIVEIL) += pcie-mobiveil.o
> > > obj-$(CONFIG_PCIE_MOBIVEIL_HOST) += pcie-mobiveil-host.o
> > > obj-$(CONFIG_PCIE_MOBIVEIL_PLAT) += pcie-mobiveil-plat.o
> > > +obj-$(CONFIG_PCIE_LAYERSCAPE_GEN4) += pcie-layerscape-gen4.o
> > > diff --git a/drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c
> > > b/drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c
> > > new file mode 100644
> > > index 000000000000..6c0d3e2532db
> > > --- /dev/null
> > > +++ b/drivers/pci/controller/mobiveil/pcie-layerscape-gen4.c
> > > @@ -0,0 +1,274 @@
> > > +// SPDX-License-Identifier: GPL-2.0
> > > +/*
> > > + * PCIe Gen4 host controller driver for NXP Layerscape SoCs
> > > + *
> > > + * Copyright 2019 NXP
> > > + *
> > > + * Author: Zhiqiang Hou <[email protected]> */
> > > +
> > > +#include <linux/kernel.h>
> > > +#include <linux/interrupt.h>
> > > +#include <linux/init.h>
> > > +#include <linux/of_pci.h>
> > > +#include <linux/of_platform.h>
> > > +#include <linux/of_irq.h>
> > > +#include <linux/of_address.h>
> > > +#include <linux/pci.h>
> > > +#include <linux/platform_device.h>
> > > +#include <linux/resource.h>
> > > +#include <linux/mfd/syscon.h>
> > > +#include <linux/regmap.h>
> > > +
> > > +#include "pcie-mobiveil.h"
> > > +
> > > +/* LUT and PF control registers */
> > > +#define PCIE_LUT_OFF 0x80000
> > > +#define PCIE_PF_OFF 0xc0000
> > > +#define PCIE_PF_INT_STAT 0x18
> > > +#define PF_INT_STAT_PABRST BIT(31)
> > > +
> > > +#define PCIE_PF_DBG 0x7fc
> > > +#define PF_DBG_LTSSM_MASK 0x3f
> > > +#define PF_DBG_LTSSM_L0 0x2d /* L0 state */
> > > +#define PF_DBG_WE BIT(31)
> > > +#define PF_DBG_PABR BIT(27)
> > > +
> > > +#define to_ls_pcie_g4(x) platform_get_drvdata((x)->pdev)
> > > +
> > > +struct ls_pcie_g4 {
> > > + struct mobiveil_pcie pci;
> > > + struct delayed_work dwork;
> > > + int irq;
> > > +};
> > > +
> > > +static inline u32 ls_pcie_g4_lut_readl(struct ls_pcie_g4 *pcie, u32
> > > +off) {
> > > + return ioread32(pcie->pci.csr_axi_slave_base + PCIE_LUT_OFF + off);
> > > +}
> > > +
> > > +static inline void ls_pcie_g4_lut_writel(struct ls_pcie_g4 *pcie,
> > > + u32 off, u32 val)
> > > +{
> > > + iowrite32(val, pcie->pci.csr_axi_slave_base + PCIE_LUT_OFF + off); }
> > > +
> > > +static inline u32 ls_pcie_g4_pf_readl(struct ls_pcie_g4 *pcie, u32
> > > +off) {
> > > + return ioread32(pcie->pci.csr_axi_slave_base + PCIE_PF_OFF + off); }
> > > +
> > > +static inline void ls_pcie_g4_pf_writel(struct ls_pcie_g4 *pcie,
> > > + u32 off, u32 val)
> > > +{
> > > + iowrite32(val, pcie->pci.csr_axi_slave_base + PCIE_PF_OFF + off); }
> > > +
> > > +static bool ls_pcie_g4_is_bridge(struct ls_pcie_g4 *pcie) {
> > > + struct mobiveil_pcie *mv_pci = &pcie->pci;
> > > + u32 header_type;
> > > +
> > > + header_type = mobiveil_csr_readb(mv_pci, PCI_HEADER_TYPE);
> > > + header_type &= 0x7f;
> > > +
> > > + return header_type == PCI_HEADER_TYPE_BRIDGE; }
> > > +
> > > +static int ls_pcie_g4_link_up(struct mobiveil_pcie *pci) {
> > > + struct ls_pcie_g4 *pcie = to_ls_pcie_g4(pci);
> > > + u32 state;
> > > +
> > > + state = ls_pcie_g4_pf_readl(pcie, PCIE_PF_DBG);
> > > + state = state & PF_DBG_LTSSM_MASK;
> > > +
> > > + if (state == PF_DBG_LTSSM_L0)
> > > + return 1;
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static void ls_pcie_g4_disable_interrupt(struct ls_pcie_g4 *pcie) {
> > > + struct mobiveil_pcie *mv_pci = &pcie->pci;
> > > +
> > > + mobiveil_csr_writel(mv_pci, 0, PAB_INTP_AMBA_MISC_ENB); }
> > > +
> > > +static void ls_pcie_g4_enable_interrupt(struct ls_pcie_g4 *pcie) {
> > > + struct mobiveil_pcie *mv_pci = &pcie->pci;
> > > + u32 val;
> > > +
> > > + /* Clear the interrupt status */
> > > + mobiveil_csr_writel(mv_pci, 0xffffffff, PAB_INTP_AMBA_MISC_STAT);
> > > +
> > > + val = PAB_INTP_INTX_MASK | PAB_INTP_MSI | PAB_INTP_RESET |
> > > + PAB_INTP_PCIE_UE | PAB_INTP_IE_PMREDI | PAB_INTP_IE_EC;
> > > + mobiveil_csr_writel(mv_pci, val, PAB_INTP_AMBA_MISC_ENB); }
> > > +
> > > +static void ls_pcie_g4_reinit_hw(struct ls_pcie_g4 *pcie) {
> > > + struct mobiveil_pcie *mv_pci = &pcie->pci;
> > > + struct device *dev = &mv_pci->pdev->dev;
> > > + u32 val, act_stat;
> > > + int to = 100;
> > > +
> > > + /* Poll for pab_csb_reset to set and PAB activity to clear */
> > > + do {
> > > + usleep_range(10, 15);
> > > + val = ls_pcie_g4_pf_readl(pcie, PCIE_PF_INT_STAT);
> > > + act_stat = mobiveil_csr_readl(mv_pci, PAB_ACTIVITY_STAT);
> > > + } while (((val & PF_INT_STAT_PABRST) == 0 || act_stat) && to--);
> > > + if (to < 0) {
> > > + dev_err(dev, "Poll PABRST&PABACT timeout\n");
> > > + return;
> >
> > If a timeout happens here - the caller has no idea this has happened and yet
> > the following work doesn't get done. Isn't this a problem?
>
> Will change the return value type to 'int' in v10, so that the caller can know the fail.
>
> >
> > > + }
> > > +
> > > + /* clear PEX_RESET bit in PEX_PF0_DBG register */
> > > + val = ls_pcie_g4_pf_readl(pcie, PCIE_PF_DBG);
> > > + val |= PF_DBG_WE;
> > > + ls_pcie_g4_pf_writel(pcie, PCIE_PF_DBG, val);
> > > +
> > > + val = ls_pcie_g4_pf_readl(pcie, PCIE_PF_DBG);
> > > + val |= PF_DBG_PABR;
> > > + ls_pcie_g4_pf_writel(pcie, PCIE_PF_DBG, val);
> > > +
> > > + val = ls_pcie_g4_pf_readl(pcie, PCIE_PF_DBG);
> > > + val &= ~PF_DBG_WE;
> > > + ls_pcie_g4_pf_writel(pcie, PCIE_PF_DBG, val);
> > > +
> > > + mobiveil_host_init(mv_pci, true);
> >
> > Can mobiveil_host_init fail?
>
> It should not fail, only register programming operations were left in this function.
>
> >
> > > +
> > > + to = 100;
> > > + while (!ls_pcie_g4_link_up(mv_pci) && to--)
> > > + usleep_range(200, 250);
> > > + if (to < 0)
> > > + dev_err(dev, "PCIe link training timeout\n"); }
> > > +
> > > +static irqreturn_t ls_pcie_g4_isr(int irq, void *dev_id) {
> > > + struct ls_pcie_g4 *pcie = (struct ls_pcie_g4 *)dev_id;
> > > + struct mobiveil_pcie *mv_pci = &pcie->pci;
> > > + u32 val;
> > > +
> > > + val = mobiveil_csr_readl(mv_pci, PAB_INTP_AMBA_MISC_STAT);
> > > + if (!val)
> > > + return IRQ_NONE;
> > > +
> > > + if (val & PAB_INTP_RESET) {
> >
> > Can you explain why this is needed (perhaps also in the cover letter)?
>
> The hot reset will result in the RC crash, so need the ISR to reset the RC.
>
> >
> > > + ls_pcie_g4_disable_interrupt(pcie);
> > > + schedule_delayed_work(&pcie->dwork, msecs_to_jiffies(1));
> > > + }
> > > +
> > > + mobiveil_csr_writel(mv_pci, val, PAB_INTP_AMBA_MISC_STAT);
> > > +
> > > + return IRQ_HANDLED;
> > > +}
> > > +
> > > +static int ls_pcie_g4_interrupt_init(struct mobiveil_pcie *mv_pci) {
> > > + struct ls_pcie_g4 *pcie = to_ls_pcie_g4(mv_pci);
> > > + struct platform_device *pdev = mv_pci->pdev;
> > > + struct device *dev = &pdev->dev;
> > > + int ret;
> > > +
> > > + pcie->irq = platform_get_irq_byname(pdev, "intr");
> > > + if (pcie->irq < 0) {
> > > + dev_err(dev, "Can't get 'intr' IRQ, errno = %d\n", pcie->irq);
> > > + return pcie->irq;
> > > + }
> > > + ret = devm_request_irq(dev, pcie->irq, ls_pcie_g4_isr,
> > > + IRQF_SHARED, pdev->name, pcie);
> > > + if (ret) {
> > > + dev_err(dev, "Can't register PCIe IRQ, errno = %d\n", ret);
> > > + return ret;
> > > + }
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static void ls_pcie_g4_reset(struct work_struct *work) {
> > > + struct delayed_work *dwork = container_of(work, struct delayed_work,
> > > + work);
> > > + struct ls_pcie_g4 *pcie = container_of(dwork, struct ls_pcie_g4, dwork);
> > > + struct mobiveil_pcie *mv_pci = &pcie->pci;
> > > + u16 ctrl;
> > > +
> > > + ctrl = mobiveil_csr_readw(mv_pci, PCI_BRIDGE_CONTROL);
> > > + ctrl &= ~PCI_BRIDGE_CTL_BUS_RESET;
> > > + mobiveil_csr_writew(mv_pci, ctrl, PCI_BRIDGE_CONTROL);
> > > + ls_pcie_g4_reinit_hw(pcie);
> > > + ls_pcie_g4_enable_interrupt(pcie);
> > > +}
> > > +
> > > +static struct mobiveil_rp_ops ls_pcie_g4_rp_ops = {
> > > + .interrupt_init = ls_pcie_g4_interrupt_init, };
> > > +
> > > +static const struct mobiveil_pab_ops ls_pcie_g4_pab_ops = {
> > > + .link_up = ls_pcie_g4_link_up,
> > > +};
> > > +
> > > +static int __init ls_pcie_g4_probe(struct platform_device *pdev) {
> > > + struct device *dev = &pdev->dev;
> > > + struct pci_host_bridge *bridge;
> > > + struct mobiveil_pcie *mv_pci;
> > > + struct ls_pcie_g4 *pcie;
> > > + struct device_node *np = dev->of_node;
> > > + int ret;
> > > +
> > > + if (!of_parse_phandle(np, "msi-parent", 0)) {
> > > + dev_err(dev, "Failed to find msi-parent\n");
> > > + return -EINVAL;
> > > + }
> > > +
> > > + bridge = devm_pci_alloc_host_bridge(dev, sizeof(*pcie));
> > > + if (!bridge)
> > > + return -ENOMEM;
> > > +
> > > + pcie = pci_host_bridge_priv(bridge);
> > > + mv_pci = &pcie->pci;
> > > +
> > > + mv_pci->pdev = pdev;
> > > + mv_pci->ops = &ls_pcie_g4_pab_ops;
> > > + mv_pci->rp.ops = &ls_pcie_g4_rp_ops;
> > > + mv_pci->rp.bridge = bridge;
> > > +
> > > + platform_set_drvdata(pdev, pcie);
> > > +
> > > + INIT_DELAYED_WORK(&pcie->dwork, ls_pcie_g4_reset);
> > > +
> > > + ret = mobiveil_pcie_host_probe(mv_pci);
> > > + if (ret) {
> > > + dev_err(dev, "Fail to probe\n");
> > > + return ret;
> > > + }
> > > +
> > > + if (!ls_pcie_g4_is_bridge(pcie))
> >
> > Is this a check that could apply to all host bridge drivers and thus live in
> > mobiveil_pcie_host_probe?
>
> Yes, will do in v10.
>
> >
> > > + return -ENODEV;
> > > +
> > > + ls_pcie_g4_enable_interrupt(pcie);
> >
> > Is there an issue here in that we enable interrupts *after* telling the kernel
> > about our controller? (Same applies for bailing if the IP isn't a bridge).
>
> Andrew, I don't understand the issue, can you help to explain?
If I recall correctly mobiveil_pcie_host_probe tells the kernel there is a PCI host
bridge and allows it to start enumerating the tree - at this point surely interrupts
may be expected - however we don't enable them until after this point. I'd assume
we'd need to get the hardware into a state where it can handle interrupts before
telling the kernel it can use this host bridge.
Likewise is ls_pcie_g4_is_bridge returns false we fail the probe yet the kernel may
already be enumerating the bus.
Thanks,
Andrew Murray
>
> Thanks,
> Zhiqiang
>
> >
> > Thanks,
> >
> > Andrew Murray
> >
> > > +
> > > + return 0;
> > > +}
> > > +
> > > +static const struct of_device_id ls_pcie_g4_of_match[] = {
> > > + { .compatible = "fsl,lx2160a-pcie", },
> > > + { },
> > > +};
> > > +
> > > +static struct platform_driver ls_pcie_g4_driver = {
> > > + .driver = {
> > > + .name = "layerscape-pcie-gen4",
> > > + .of_match_table = ls_pcie_g4_of_match,
> > > + .suppress_bind_attrs = true,
> > > + },
> > > +};
> > > +
> > > +builtin_platform_driver_probe(ls_pcie_g4_driver, ls_pcie_g4_probe);
> > > diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > > b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > > index 750a7fd95bc1..c57a68d2bac4 100644
> > > --- a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > > +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > > @@ -43,6 +43,8 @@
> > > #define PAGE_LO_MASK 0x3ff
> > > #define PAGE_SEL_OFFSET_SHIFT 10
> > >
> > > +#define PAB_ACTIVITY_STAT 0x81c
> > > +
> > > #define PAB_AXI_PIO_CTRL 0x0840
> > > #define APIO_EN_MASK 0xf
> > >
> > > @@ -51,8 +53,18 @@
> > >
> > > #define PAB_INTP_AMBA_MISC_ENB 0x0b0c
> > > #define PAB_INTP_AMBA_MISC_STAT 0x0b1c
> > > -#define PAB_INTP_INTX_MASK 0x01e0
> > > -#define PAB_INTP_MSI_MASK 0x8
> > > +#define PAB_INTP_RESET BIT(1)
> > > +#define PAB_INTP_MSI BIT(3)
> > > +#define PAB_INTP_INTA BIT(5)
> > > +#define PAB_INTP_INTB BIT(6)
> > > +#define PAB_INTP_INTC BIT(7)
> > > +#define PAB_INTP_INTD BIT(8)
> > > +#define PAB_INTP_PCIE_UE BIT(9)
> > > +#define PAB_INTP_IE_PMREDI BIT(29)
> > > +#define PAB_INTP_IE_EC BIT(30)
> > > +#define PAB_INTP_MSI_MASK PAB_INTP_MSI
> > > +#define PAB_INTP_INTX_MASK (PAB_INTP_INTA |
> > PAB_INTP_INTB |\
> > > + PAB_INTP_INTC | PAB_INTP_INTD)
> > >
> > > #define PAB_AXI_AMAP_CTRL(win) PAB_REG_ADDR(0x0ba0,
> > win)
> > > #define WIN_ENABLE_SHIFT 0
> > > --
> > > 2.17.1
> > >
Hi Andrew,
Thanks a lot for your review!
B.R,
Zhiqiang
> -----Original Message-----
> From: Andrew Murray <[email protected]>
> Sent: 2020??1??13?? 19:22
> To: Z.q. Hou <[email protected]>
> Cc: [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; Leo Li
> <[email protected]>; [email protected];
> [email protected]; [email protected]; Mingkai Hu
> <[email protected]>; M.h. Lian <[email protected]>; Xiaowei Bao
> <[email protected]>
> Subject: Re: [PATCHv9 06/12] PCI: mobiveil: Add callback function for link up
> check
>
> On Wed, Nov 20, 2019 at 03:45:57AM +0000, Z.q. Hou wrote:
> > From: Hou Zhiqiang <[email protected]>
> >
> > The platforms, in which the Mobiveil GPEX is integrated, may have
> > their specific mechanism to check link up status.
> > This patch is to enable these platforms to implement theirs.
> >
> > Signed-off-by: Hou Zhiqiang <[email protected]>
> > ---
> > V9:
> > - New patch splited from the #1 of V8 patches to make it easy to review.
> >
> > drivers/pci/controller/mobiveil/pcie-mobiveil.c | 3 +++
> > drivers/pci/controller/mobiveil/pcie-mobiveil.h | 5 +++++
> > 2 files changed, 8 insertions(+)
> >
> > diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.c
> > b/drivers/pci/controller/mobiveil/pcie-mobiveil.c
> > index 2773f823c9ea..b9ed2d95641c 100644
> > --- a/drivers/pci/controller/mobiveil/pcie-mobiveil.c
> > +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.c
> > @@ -125,6 +125,9 @@ void mobiveil_csr_write(struct mobiveil_pcie
> > *pcie, u32 val, u32 off,
> >
> > bool mobiveil_pcie_link_up(struct mobiveil_pcie *pcie) {
> > + if (pcie->ops->link_up)
> > + return pcie->ops->link_up(pcie);
> > +
> > return (mobiveil_csr_readl(pcie, LTSSM_STATUS) &
> > LTSSM_STATUS_L0_MASK) == LTSSM_STATUS_L0;
>
> On the previous patch I suggested that we don't mix up the link_up logic with
> the logic that decides which function to call. In this case the link_up logic is
> trivial. So this is probably OK.
>
> Reviewed-by: Andrew Murray <[email protected]>
>
> > }
> > diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > index 18d85806a7fc..95d2e7c809b8 100644
> > --- a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > @@ -148,6 +148,10 @@ struct root_port {
> > struct pci_host_bridge *bridge;
> > };
> >
> > +struct mobiveil_pab_ops {
> > + int (*link_up)(struct mobiveil_pcie *pcie); };
> > +
> > struct mobiveil_pcie {
> > struct platform_device *pdev;
> > void __iomem *csr_axi_slave_base; /* root port config base */
> > @@ -157,6 +161,7 @@ struct mobiveil_pcie {
> > int ppio_wins;
> > int ob_wins_configured; /* configured outbound windows */
> > int ib_wins_configured; /* configured inbound windows */
> > + const struct mobiveil_pab_ops *ops;
> > struct root_port rp;
> > };
> >
> > --
> > 2.17.1
> >
Hi Andrew,
Thanks a lot for your comments!
> -----Original Message-----
> From: Andrew Murray <[email protected]>
> Sent: 2020??1??13?? 19:27
> To: Z.q. Hou <[email protected]>
> Cc: [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; [email protected];
> [email protected]; [email protected];
> [email protected]; [email protected]; Leo Li
> <[email protected]>; [email protected];
> [email protected]; [email protected]; Mingkai Hu
> <[email protected]>; M.h. Lian <[email protected]>; Xiaowei Bao
> <[email protected]>
> Subject: Re: [PATCHv9 07/12] PCI: mobiveil: Make mobiveil_host_init() can be
> used to re-init host
>
> On Wed, Nov 20, 2019 at 03:46:03AM +0000, Z.q. Hou wrote:
> > From: Hou Zhiqiang <[email protected]>
> >
> > Make the mobiveil_host_init() function can be used to re-init
>
> Perhaps reword to "Allow the mobiveil_host_init() function to be used to ...
>
Thanks a lot for your suggestion, will change in v10.
Thanks,
Zhiqiang
> > host controller's PAB and GPEX CSR register block, as NXP integrated
> > Mobiveil IP has to reset and then re-init the PAB and GPEX CSR
> > registers upon hot-reset.
> >
> > Signed-off-by: Hou Zhiqiang <[email protected]>
> > Reviewed-by: Subrahmanya Lingappa <[email protected]>
> > ---
> > V9:
> > - No change
> >
> > .../controller/mobiveil/pcie-mobiveil-host.c | 19 ++++++++++++-------
> > .../pci/controller/mobiveil/pcie-mobiveil.h | 1 +
> > 2 files changed, 13 insertions(+), 7 deletions(-)
> >
> > diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
> > b/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
> > index 3cd93df6fe6e..9bc3da036720 100644
> > --- a/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
> > +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil-host.c
> > @@ -221,18 +221,23 @@ static void mobiveil_pcie_enable_msi(struct
> mobiveil_pcie *pcie)
> > writel_relaxed(1, pcie->apb_csr_base + MSI_ENABLE_OFFSET); }
> >
> > -static int mobiveil_host_init(struct mobiveil_pcie *pcie)
> > +int mobiveil_host_init(struct mobiveil_pcie *pcie, bool reinit)
> > {
> > struct root_port *rp = &pcie->rp;
> > struct pci_host_bridge *bridge = rp->bridge;
> > u32 value, pab_ctrl, type;
> > struct resource_entry *win;
> >
> > - /* setup bus numbers */
> > - value = mobiveil_csr_readl(pcie, PCI_PRIMARY_BUS);
> > - value &= 0xff000000;
> > - value |= 0x00ff0100;
> > - mobiveil_csr_writel(pcie, value, PCI_PRIMARY_BUS);
> > + pcie->ib_wins_configured = 0;
> > + pcie->ob_wins_configured = 0;
>
> This works so long as the number of bridge->windows never reduces. I think
> this assumption holds true.
>
> Thanks,
>
> Andrew Murray
>
> > +
> > + if (!reinit) {
> > + /* setup bus numbers */
> > + value = mobiveil_csr_readl(pcie, PCI_PRIMARY_BUS);
> > + value &= 0xff000000;
> > + value |= 0x00ff0100;
> > + mobiveil_csr_writel(pcie, value, PCI_PRIMARY_BUS);
> > + }
> >
> > /*
> > * program Bus Master Enable Bit in Command Register in PAB Config
> > @@ -569,7 +574,7 @@ int mobiveil_pcie_host_probe(struct mobiveil_pcie
> *pcie)
> > * configure all inbound and outbound windows and prepare the RC for
> > * config access
> > */
> > - ret = mobiveil_host_init(pcie);
> > + ret = mobiveil_host_init(pcie, false);
> > if (ret) {
> > dev_err(dev, "Failed to initialize host\n");
> > return ret;
> > diff --git a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > index 95d2e7c809b8..37116c2a19fe 100644
> > --- a/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > +++ b/drivers/pci/controller/mobiveil/pcie-mobiveil.h
> > @@ -166,6 +166,7 @@ struct mobiveil_pcie { };
> >
> > int mobiveil_pcie_host_probe(struct mobiveil_pcie *pcie);
> > +int mobiveil_host_init(struct mobiveil_pcie *pcie, bool reinit);
> > bool mobiveil_pcie_link_up(struct mobiveil_pcie *pcie); int
> > mobiveil_bringup_link(struct mobiveil_pcie *pcie); void
> > program_ob_windows(struct mobiveil_pcie *pcie, int win_num, u64
> > cpu_addr,
> > --
> > 2.17.1
> >
On Thu, Feb 6, 2020 at 11:57 AM Z.q. Hou <[email protected]> wrote:
>
> Hi Olof,
>
> Thanks a lot for your comments!
> And sorry for my delay respond!
Actually, they apply with only minor conflicts on top of current -next.
Bjorn, any chance we can get you to pick these up pretty soon? They
enable full use of a promising ARM developer system, the SolidRun
HoneyComb, and would be quite valuable for me and others to be able to
use with mainline or -next without any additional patches applied --
which this patchset achieves.
I know there are pending revisions based on feedback. I'll leave it up
to you and others to determine if that can be done with incremental
patches on top, or if it should be fixed before the initial patchset
is applied. But all in all, it's holding up adaption by me and surely
others of a very interesting platform -- I'm looking to replace my
aging MacchiatoBin with one of these and would need PCIe/NVMe to work
before I do.
Thanks!
-Olof
On Mon, Feb 10, 2020 at 04:12:30PM +0100, Olof Johansson wrote:
> On Thu, Feb 6, 2020 at 11:57 AM Z.q. Hou <[email protected]> wrote:
> >
> > Hi Olof,
> >
> > Thanks a lot for your comments!
> > And sorry for my delay respond!
>
> Actually, they apply with only minor conflicts on top of current -next.
>
> Bjorn, any chance we can get you to pick these up pretty soon? They
> enable full use of a promising ARM developer system, the SolidRun
> HoneyComb, and would be quite valuable for me and others to be able to
> use with mainline or -next without any additional patches applied --
> which this patchset achieves.
>
> I know there are pending revisions based on feedback. I'll leave it up
> to you and others to determine if that can be done with incremental
> patches on top, or if it should be fixed before the initial patchset
> is applied. But all in all, it's holding up adaption by me and surely
> others of a very interesting platform -- I'm looking to replace my
> aging MacchiatoBin with one of these and would need PCIe/NVMe to work
> before I do.
If you're going to be using NVMe, make sure you use a power-fail safe
version; I've already had one instance where ext4 failed to mount
because of a corrupted journal using an XPG SX8200 after the Honeycomb
Serror'd, and then I powered it down after a few hours before later
booting it back up.
EXT4-fs (nvme0n1p2): INFO: recovery required on readonly filesystem
EXT4-fs (nvme0n1p2): write access will be enabled during recovery
JBD2: journal transaction 80849 on nvme0n1p2-8 is corrupt.
EXT4-fs (nvme0n1p2): error loading journal
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up
On Mon, Feb 10, 2020 at 4:23 PM Russell King - ARM Linux admin
<[email protected]> wrote:
>
> On Mon, Feb 10, 2020 at 04:12:30PM +0100, Olof Johansson wrote:
> > On Thu, Feb 6, 2020 at 11:57 AM Z.q. Hou <[email protected]> wrote:
> > >
> > > Hi Olof,
> > >
> > > Thanks a lot for your comments!
> > > And sorry for my delay respond!
> >
> > Actually, they apply with only minor conflicts on top of current -next.
> >
> > Bjorn, any chance we can get you to pick these up pretty soon? They
> > enable full use of a promising ARM developer system, the SolidRun
> > HoneyComb, and would be quite valuable for me and others to be able to
> > use with mainline or -next without any additional patches applied --
> > which this patchset achieves.
> >
> > I know there are pending revisions based on feedback. I'll leave it up
> > to you and others to determine if that can be done with incremental
> > patches on top, or if it should be fixed before the initial patchset
> > is applied. But all in all, it's holding up adaption by me and surely
> > others of a very interesting platform -- I'm looking to replace my
> > aging MacchiatoBin with one of these and would need PCIe/NVMe to work
> > before I do.
>
> If you're going to be using NVMe, make sure you use a power-fail safe
> version; I've already had one instance where ext4 failed to mount
> because of a corrupted journal using an XPG SX8200 after the Honeycomb
> Serror'd, and then I powered it down after a few hours before later
> booting it back up.
>
> EXT4-fs (nvme0n1p2): INFO: recovery required on readonly filesystem
> EXT4-fs (nvme0n1p2): write access will be enabled during recovery
> JBD2: journal transaction 80849 on nvme0n1p2-8 is corrupt.
> EXT4-fs (nvme0n1p2): error loading journal
Hmm, using btrfs on mine, not sure if the exposure is similar or not.
Do you know if the SErr was due to a known issue and/or if it's
something that's fixed in production silicon?
(I still can't enable SMMU since across a warm reboot it fails
*completely*, with nothing coming up and working. NXP folks, you
listening? :)
-Olof
On Mon, Feb 10, 2020 at 04:12:30PM +0100, Olof Johansson wrote:
> On Thu, Feb 6, 2020 at 11:57 AM Z.q. Hou <[email protected]> wrote:
> >
> > Hi Olof,
> >
> > Thanks a lot for your comments!
> > And sorry for my delay respond!
>
> Actually, they apply with only minor conflicts on top of current -next.
>
> Bjorn, any chance we can get you to pick these up pretty soon? They
> enable full use of a promising ARM developer system, the SolidRun
> HoneyComb, and would be quite valuable for me and others to be able to
> use with mainline or -next without any additional patches applied --
> which this patchset achieves.
>
> I know there are pending revisions based on feedback. I'll leave it up
> to you and others to determine if that can be done with incremental
> patches on top, or if it should be fixed before the initial patchset
> is applied. But all in all, it's holding up adaption by me and surely
> others of a very interesting platform -- I'm looking to replace my
> aging MacchiatoBin with one of these and would need PCIe/NVMe to work
> before I do.
We should be able to merge them for v5.7, I don't know when they
will land in -next.
Thanks,
Lorenzo
On Mon, Feb 10, 2020 at 04:28:23PM +0100, Olof Johansson wrote:
> On Mon, Feb 10, 2020 at 4:23 PM Russell King - ARM Linux admin
> <[email protected]> wrote:
> >
> > On Mon, Feb 10, 2020 at 04:12:30PM +0100, Olof Johansson wrote:
> > > On Thu, Feb 6, 2020 at 11:57 AM Z.q. Hou <[email protected]> wrote:
> > > >
> > > > Hi Olof,
> > > >
> > > > Thanks a lot for your comments!
> > > > And sorry for my delay respond!
> > >
> > > Actually, they apply with only minor conflicts on top of current -next.
> > >
> > > Bjorn, any chance we can get you to pick these up pretty soon? They
> > > enable full use of a promising ARM developer system, the SolidRun
> > > HoneyComb, and would be quite valuable for me and others to be able to
> > > use with mainline or -next without any additional patches applied --
> > > which this patchset achieves.
> > >
> > > I know there are pending revisions based on feedback. I'll leave it up
> > > to you and others to determine if that can be done with incremental
> > > patches on top, or if it should be fixed before the initial patchset
> > > is applied. But all in all, it's holding up adaption by me and surely
> > > others of a very interesting platform -- I'm looking to replace my
> > > aging MacchiatoBin with one of these and would need PCIe/NVMe to work
> > > before I do.
> >
> > If you're going to be using NVMe, make sure you use a power-fail safe
> > version; I've already had one instance where ext4 failed to mount
> > because of a corrupted journal using an XPG SX8200 after the Honeycomb
> > Serror'd, and then I powered it down after a few hours before later
> > booting it back up.
> >
> > EXT4-fs (nvme0n1p2): INFO: recovery required on readonly filesystem
> > EXT4-fs (nvme0n1p2): write access will be enabled during recovery
> > JBD2: journal transaction 80849 on nvme0n1p2-8 is corrupt.
> > EXT4-fs (nvme0n1p2): error loading journal
>
> Hmm, using btrfs on mine, not sure if the exposure is similar or not.
As I understand the problem, it isn't a filesystem issue. It's a data
integrity issue with the NVMe over power fail, how they cache the data,
and ultimately write it to the nand flash.
Have a read of:
https://www.kingston.com/en/solutions/servers-data-centers/ssd-power-loss-protection
As NVMe and SSD are basically the same underlying technology (the host
interface is different) and the issues I've heard, and now experienced
with my NVMe, I think the above is a good pointer to the problems of
flash mass storage.
As I understand it, the problem occurs when the mapping table has not
been written back to flash, power is lost without the Standby Immediate
command being sent, and there is no way for the firmware to quickly
save the table. On subsequent power up, the firmware has to
reconstruct the mapping table, and depending on how that is done,
incorrect (old?) data may be returned for some blocks.
That can happen to any blocks on the drive, which means any data can
be at risk from a power loss event, whether that is a power failure
or after a crash.
> Do you know if the SErr was due to a known issue and/or if it's
> something that's fixed in production silicon?
The SError is triggered by something on the PCIe side of things; if I
leave the Mellanox PCIe card out, then I don't get them. The errata
patches I have merged into my tree help a bit, turning the code from
being unable to boot without a SError with the card plugged in, to
being able to boot and last a while - but the SErrors still eventually
come, maybe taking a few days... and that's without the Mellanox
ethernet interface being up.
> (I still can't enable SMMU since across a warm reboot it fails
> *completely*, with nothing coming up and working. NXP folks, you
> listening? :)
Is it just a warm reboot? I thought I saw SMMU activity on a cold
boot as well, implying that there were devices active that Linux
did not know about.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up
On Mon, Feb 10, 2020 at 04:15:53PM +0000, Russell King - ARM Linux admin wrote:
> On Mon, Feb 10, 2020 at 04:28:23PM +0100, Olof Johansson wrote:
> > On Mon, Feb 10, 2020 at 4:23 PM Russell King - ARM Linux admin
> > <[email protected]> wrote:
> > >
> > > On Mon, Feb 10, 2020 at 04:12:30PM +0100, Olof Johansson wrote:
> > > > On Thu, Feb 6, 2020 at 11:57 AM Z.q. Hou <[email protected]> wrote:
> > > > >
> > > > > Hi Olof,
> > > > >
> > > > > Thanks a lot for your comments!
> > > > > And sorry for my delay respond!
> > > >
> > > > Actually, they apply with only minor conflicts on top of current -next.
> > > >
> > > > Bjorn, any chance we can get you to pick these up pretty soon? They
> > > > enable full use of a promising ARM developer system, the SolidRun
> > > > HoneyComb, and would be quite valuable for me and others to be able to
> > > > use with mainline or -next without any additional patches applied --
> > > > which this patchset achieves.
> > > >
> > > > I know there are pending revisions based on feedback. I'll leave it up
> > > > to you and others to determine if that can be done with incremental
> > > > patches on top, or if it should be fixed before the initial patchset
> > > > is applied. But all in all, it's holding up adaption by me and surely
> > > > others of a very interesting platform -- I'm looking to replace my
> > > > aging MacchiatoBin with one of these and would need PCIe/NVMe to work
> > > > before I do.
> > >
> > > If you're going to be using NVMe, make sure you use a power-fail safe
> > > version; I've already had one instance where ext4 failed to mount
> > > because of a corrupted journal using an XPG SX8200 after the Honeycomb
> > > Serror'd, and then I powered it down after a few hours before later
> > > booting it back up.
> > >
> > > EXT4-fs (nvme0n1p2): INFO: recovery required on readonly filesystem
> > > EXT4-fs (nvme0n1p2): write access will be enabled during recovery
> > > JBD2: journal transaction 80849 on nvme0n1p2-8 is corrupt.
> > > EXT4-fs (nvme0n1p2): error loading journal
> >
> > Hmm, using btrfs on mine, not sure if the exposure is similar or not.
>
> As I understand the problem, it isn't a filesystem issue. It's a data
> integrity issue with the NVMe over power fail, how they cache the data,
> and ultimately write it to the nand flash.
>
> Have a read of:
>
> https://www.kingston.com/en/solutions/servers-data-centers/ssd-power-loss-protection
This was the link I was actually looking for:
http://industrial.adata.com/en/technology/92
but there's also:
http://industrial.adata.com/en/technology/26
ADATA make the XPG SX8200:
NVME Identify Controller:
vid : 0x1cc1
ssvid : 0x1cc1
mn : ADATA SX8200PNP
fr : R0906I
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up
[cc:ing honeycomb-users, didn't think of that earlier]
On Mon, Feb 10, 2020 at 5:16 PM Russell King - ARM Linux admin
<[email protected]> wrote:
>
> On Mon, Feb 10, 2020 at 04:28:23PM +0100, Olof Johansson wrote:
> > On Mon, Feb 10, 2020 at 4:23 PM Russell King - ARM Linux admin
> > <[email protected]> wrote:
> > >
> > > On Mon, Feb 10, 2020 at 04:12:30PM +0100, Olof Johansson wrote:
> > > > On Thu, Feb 6, 2020 at 11:57 AM Z.q. Hou <[email protected]> wrote:
> > > > >
> > > > > Hi Olof,
> > > > >
> > > > > Thanks a lot for your comments!
> > > > > And sorry for my delay respond!
> > > >
> > > > Actually, they apply with only minor conflicts on top of current -next.
> > > >
> > > > Bjorn, any chance we can get you to pick these up pretty soon? They
> > > > enable full use of a promising ARM developer system, the SolidRun
> > > > HoneyComb, and would be quite valuable for me and others to be able to
> > > > use with mainline or -next without any additional patches applied --
> > > > which this patchset achieves.
> > > >
> > > > I know there are pending revisions based on feedback. I'll leave it up
> > > > to you and others to determine if that can be done with incremental
> > > > patches on top, or if it should be fixed before the initial patchset
> > > > is applied. But all in all, it's holding up adaption by me and surely
> > > > others of a very interesting platform -- I'm looking to replace my
> > > > aging MacchiatoBin with one of these and would need PCIe/NVMe to work
> > > > before I do.
> > >
> > > If you're going to be using NVMe, make sure you use a power-fail safe
> > > version; I've already had one instance where ext4 failed to mount
> > > because of a corrupted journal using an XPG SX8200 after the Honeycomb
> > > Serror'd, and then I powered it down after a few hours before later
> > > booting it back up.
> > >
> > > EXT4-fs (nvme0n1p2): INFO: recovery required on readonly filesystem
> > > EXT4-fs (nvme0n1p2): write access will be enabled during recovery
> > > JBD2: journal transaction 80849 on nvme0n1p2-8 is corrupt.
> > > EXT4-fs (nvme0n1p2): error loading journal
> >
> > Hmm, using btrfs on mine, not sure if the exposure is similar or not.
>
> As I understand the problem, it isn't a filesystem issue. It's a data
> integrity issue with the NVMe over power fail, how they cache the data,
> and ultimately write it to the nand flash.
>
> Have a read of:
>
> https://www.kingston.com/en/solutions/servers-data-centers/ssd-power-loss-protection
>
> As NVMe and SSD are basically the same underlying technology (the host
> interface is different) and the issues I've heard, and now experienced
> with my NVMe, I think the above is a good pointer to the problems of
> flash mass storage.
>
> As I understand it, the problem occurs when the mapping table has not
> been written back to flash, power is lost without the Standby Immediate
> command being sent, and there is no way for the firmware to quickly
> save the table. On subsequent power up, the firmware has to
> reconstruct the mapping table, and depending on how that is done,
> incorrect (old?) data may be returned for some blocks.
>
> That can happen to any blocks on the drive, which means any data can
> be at risk from a power loss event, whether that is a power failure
> or after a crash.
Makes me suspect if there's some board-level power/reset sequencing
issue, or if there's a problem with one card going down disabling
others. I haven't read the specs enough to know what's expected
behavior but I've seen similar issues on other platforms so take it
with a grain of salt.
> > Do you know if the SErr was due to a known issue and/or if it's
> > something that's fixed in production silicon?
>
> The SError is triggered by something on the PCIe side of things; if I
> leave the Mellanox PCIe card out, then I don't get them. The errata
> patches I have merged into my tree help a bit, turning the code from
> being unable to boot without a SError with the card plugged in, to
> being able to boot and last a while - but the SErrors still eventually
> come, maybe taking a few days... and that's without the Mellanox
> ethernet interface being up.
>
> > (I still can't enable SMMU since across a warm reboot it fails
> > *completely*, with nothing coming up and working. NXP folks, you
> > listening? :)
>
> Is it just a warm reboot? I thought I saw SMMU activity on a cold
> boot as well, implying that there were devices active that Linux
> did not know about.
Yeah, 100% reproducible on warm reboot -- every single time. Not on
cold boot though (100% success rate as far as I remember). I boot with
kernel on NVMe on PCIe, native 1GbE for networking. u-boot from SD
card.
This is with the SolidRun u-boot from GitHub.
-Olof
On Mon, Feb 10, 2020 at 9:32 AM Olof Johansson <[email protected]> wrote:
>
> On Mon, Feb 10, 2020 at 4:23 PM Russell King - ARM Linux admin
> <[email protected]> wrote:
> >
> > On Mon, Feb 10, 2020 at 04:12:30PM +0100, Olof Johansson wrote:
> > > On Thu, Feb 6, 2020 at 11:57 AM Z.q. Hou <[email protected]> wrote:
> > > >
> > > > Hi Olof,
> > > >
> > > > Thanks a lot for your comments!
> > > > And sorry for my delay respond!
> > >
> > > Actually, they apply with only minor conflicts on top of current -next.
> > >
> > > Bjorn, any chance we can get you to pick these up pretty soon? They
> > > enable full use of a promising ARM developer system, the SolidRun
> > > HoneyComb, and would be quite valuable for me and others to be able to
> > > use with mainline or -next without any additional patches applied --
> > > which this patchset achieves.
> > >
> > > I know there are pending revisions based on feedback. I'll leave it up
> > > to you and others to determine if that can be done with incremental
> > > patches on top, or if it should be fixed before the initial patchset
> > > is applied. But all in all, it's holding up adaption by me and surely
> > > others of a very interesting platform -- I'm looking to replace my
> > > aging MacchiatoBin with one of these and would need PCIe/NVMe to work
> > > before I do.
> >
> > If you're going to be using NVMe, make sure you use a power-fail safe
> > version; I've already had one instance where ext4 failed to mount
> > because of a corrupted journal using an XPG SX8200 after the Honeycomb
> > Serror'd, and then I powered it down after a few hours before later
> > booting it back up.
> >
> > EXT4-fs (nvme0n1p2): INFO: recovery required on readonly filesystem
> > EXT4-fs (nvme0n1p2): write access will be enabled during recovery
> > JBD2: journal transaction 80849 on nvme0n1p2-8 is corrupt.
> > EXT4-fs (nvme0n1p2): error loading journal
>
> Hmm, using btrfs on mine, not sure if the exposure is similar or not.
>
> Do you know if the SErr was due to a known issue and/or if it's
> something that's fixed in production silicon?
>
> (I still can't enable SMMU since across a warm reboot it fails
> *completely*, with nothing coming up and working. NXP folks, you
> listening? :)
This is a known issue about DPAA2 MC bus not working well with SMMU
based IO mapping. Adding Laurentiu to the chain who has been looking
into this issue.
Regards,
Leo
On Mon, Feb 10, 2020 at 12:41 PM Li Yang <[email protected]> wrote:
>
> On Mon, Feb 10, 2020 at 9:32 AM Olof Johansson <[email protected]> wrote:
> >
> > On Mon, Feb 10, 2020 at 4:23 PM Russell King - ARM Linux admin
> > <[email protected]> wrote:
> > >
> > > On Mon, Feb 10, 2020 at 04:12:30PM +0100, Olof Johansson wrote:
> > > > On Thu, Feb 6, 2020 at 11:57 AM Z.q. Hou <[email protected]> wrote:
> > > > >
> > > > > Hi Olof,
> > > > >
> > > > > Thanks a lot for your comments!
> > > > > And sorry for my delay respond!
> > > >
> > > > Actually, they apply with only minor conflicts on top of current -next.
> > > >
> > > > Bjorn, any chance we can get you to pick these up pretty soon? They
> > > > enable full use of a promising ARM developer system, the SolidRun
> > > > HoneyComb, and would be quite valuable for me and others to be able to
> > > > use with mainline or -next without any additional patches applied --
> > > > which this patchset achieves.
> > > >
> > > > I know there are pending revisions based on feedback. I'll leave it up
> > > > to you and others to determine if that can be done with incremental
> > > > patches on top, or if it should be fixed before the initial patchset
> > > > is applied. But all in all, it's holding up adaption by me and surely
> > > > others of a very interesting platform -- I'm looking to replace my
> > > > aging MacchiatoBin with one of these and would need PCIe/NVMe to work
> > > > before I do.
> > >
> > > If you're going to be using NVMe, make sure you use a power-fail safe
> > > version; I've already had one instance where ext4 failed to mount
> > > because of a corrupted journal using an XPG SX8200 after the Honeycomb
> > > Serror'd, and then I powered it down after a few hours before later
> > > booting it back up.
> > >
> > > EXT4-fs (nvme0n1p2): INFO: recovery required on readonly filesystem
> > > EXT4-fs (nvme0n1p2): write access will be enabled during recovery
> > > JBD2: journal transaction 80849 on nvme0n1p2-8 is corrupt.
> > > EXT4-fs (nvme0n1p2): error loading journal
> >
> > Hmm, using btrfs on mine, not sure if the exposure is similar or not.
> >
> > Do you know if the SErr was due to a known issue and/or if it's
> > something that's fixed in production silicon?
> >
> > (I still can't enable SMMU since across a warm reboot it fails
> > *completely*, with nothing coming up and working. NXP folks, you
> > listening? :)
>
> This is a known issue about DPAA2 MC bus not working well with SMMU
> based IO mapping. Adding Laurentiu to the chain who has been looking
> into this issue.
Forgot to mention that you can workaround the issue by setting
CONFIG_ARM_SMMU_DISABLE_BYPASS_BY_DEFAULT=n or adding
"arm-smmu.disable_bypass=0" to boot parameters.
Regards,
Leo
On 10.02.2020 20:41, Li Yang wrote:
> On Mon, Feb 10, 2020 at 9:32 AM Olof Johansson <[email protected]> wrote:
>>
>> On Mon, Feb 10, 2020 at 4:23 PM Russell King - ARM Linux admin
>> <[email protected]> wrote:
>>>
>>> On Mon, Feb 10, 2020 at 04:12:30PM +0100, Olof Johansson wrote:
>>>> On Thu, Feb 6, 2020 at 11:57 AM Z.q. Hou <[email protected]> wrote:
>>>>>
>>>>> Hi Olof,
>>>>>
>>>>> Thanks a lot for your comments!
>>>>> And sorry for my delay respond!
>>>>
>>>> Actually, they apply with only minor conflicts on top of current -next.
>>>>
>>>> Bjorn, any chance we can get you to pick these up pretty soon? They
>>>> enable full use of a promising ARM developer system, the SolidRun
>>>> HoneyComb, and would be quite valuable for me and others to be able to
>>>> use with mainline or -next without any additional patches applied --
>>>> which this patchset achieves.
>>>>
>>>> I know there are pending revisions based on feedback. I'll leave it up
>>>> to you and others to determine if that can be done with incremental
>>>> patches on top, or if it should be fixed before the initial patchset
>>>> is applied. But all in all, it's holding up adaption by me and surely
>>>> others of a very interesting platform -- I'm looking to replace my
>>>> aging MacchiatoBin with one of these and would need PCIe/NVMe to work
>>>> before I do.
>>>
>>> If you're going to be using NVMe, make sure you use a power-fail safe
>>> version; I've already had one instance where ext4 failed to mount
>>> because of a corrupted journal using an XPG SX8200 after the Honeycomb
>>> Serror'd, and then I powered it down after a few hours before later
>>> booting it back up.
>>>
>>> EXT4-fs (nvme0n1p2): INFO: recovery required on readonly filesystem
>>> EXT4-fs (nvme0n1p2): write access will be enabled during recovery
>>> JBD2: journal transaction 80849 on nvme0n1p2-8 is corrupt.
>>> EXT4-fs (nvme0n1p2): error loading journal
>>
>> Hmm, using btrfs on mine, not sure if the exposure is similar or not.
>>
>> Do you know if the SErr was due to a known issue and/or if it's
>> something that's fixed in production silicon?
>>
>> (I still can't enable SMMU since across a warm reboot it fails
>> *completely*, with nothing coming up and working. NXP folks, you
>> listening? :)
>
> This is a known issue about DPAA2 MC bus not working well with SMMU
> based IO mapping. Adding Laurentiu to the chain who has been looking
> into this issue.
Yes, I'm closely following the issue. I actually have a workaround
(attached) but haven't submitted as it will probably raise a lot of
eyebrows. In the mean time I'm following some discussions [1][2][3] on
the iommu list which seem to try to tackle what appears to be a similar
issue but with framebuffers. My hope is that we will be able to leverage
whatever turns out.
In the mean time, can you try the workaround Leo suggested?
[1] https://patchwork.kernel.org/patch/11327667/
[2] https://patchwork.kernel.org/patch/10967729/
[3] https://patchwork.kernel.org/cover/11279577/
---
Best Regards, Laurentiu
On 2020-02-11 12:13 pm, Laurentiu Tudor wrote:
[...]
>> This is a known issue about DPAA2 MC bus not working well with SMMU
>> based IO mapping. Adding Laurentiu to the chain who has been looking
>> into this issue.
>
> Yes, I'm closely following the issue. I actually have a workaround
> (attached) but haven't submitted as it will probably raise a lot of
> eyebrows. In the mean time I'm following some discussions [1][2][3] on
> the iommu list which seem to try to tackle what appears to be a similar
> issue but with framebuffers. My hope is that we will be able to leverage
> whatever turns out.
Indeed it's more general than framebuffers - in fact there was a
specific requirement from the IORT side to accommodate network/storage
controllers with in-memory firmware/configuration data/whatever set up
by the bootloader that want to be handed off 'live' to Linux because the
overhead of stopping and restarting them is impractical. Thus this DPAA2
setup is very much within scope of the desired solution, so please feel
free to join in (particularly on the DT parts) :)
As for right now, note that your patch would only be a partial
mitigation to slightly reduce the fault window but not remove it
entirely. To be robust the SMMU driver *has* to know about live streams
before the first arm_smmu_reset() - hence the need for generic firmware
bindings - so doing anything from the MC driver is already too late (and
indeed the current iommu_request_dm_for_dev() mechanism is itself a
microcosm of the same problem).
> In the mean time, can you try the workaround Leo suggested?
Agreed, I'd imagine the command-line option is probably the best choice
for these platforms, since it's likely to be easier to set that by
default in the bootloader than faff with rebuilding generic kernel configs.
Robin.
> [1] https://patchwork.kernel.org/patch/11327667/
> [2] https://patchwork.kernel.org/patch/10967729/
> [3] https://patchwork.kernel.org/cover/11279577/
>
> ---
> Best Regards, Laurentiu
>
> _______________________________________________
> linux-arm-kernel mailing list
> [email protected]
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>
On 11.02.2020 15:04, Robin Murphy wrote:
> On 2020-02-11 12:13 pm, Laurentiu Tudor wrote:
> [...]
>>> This is a known issue about DPAA2 MC bus not working well with SMMU
>>> based IO mapping. Adding Laurentiu to the chain who has been looking
>>> into this issue.
>>
>> Yes, I'm closely following the issue. I actually have a workaround
>> (attached) but haven't submitted as it will probably raise a lot of
>> eyebrows. In the mean time I'm following some discussions [1][2][3] on
>> the iommu list which seem to try to tackle what appears to be a
>> similar issue but with framebuffers. My hope is that we will be able
>> to leverage whatever turns out.
>
> Indeed it's more general than framebuffers - in fact there was a
> specific requirement from the IORT side to accommodate network/storage
> controllers with in-memory firmware/configuration data/whatever set up
> by the bootloader that want to be handed off 'live' to Linux because the
> overhead of stopping and restarting them is impractical. Thus this DPAA2
> setup is very much within scope of the desired solution, so please feel
> free to join in (particularly on the DT parts) :)
Will sure do. Seems that the 2nd approach (the one with list of
compatibles in arm-smmu) fits really well with our scenario. Will this
be the way to go forward?
> As for right now, note that your patch would only be a partial
> mitigation to slightly reduce the fault window but not remove it
> entirely. To be robust the SMMU driver *has* to know about live streams
> before the first arm_smmu_reset() - hence the need for generic firmware
> bindings - so doing anything from the MC driver is already too late (and
> indeed the current iommu_request_dm_for_dev() mechanism is itself a
> microcosm of the same problem).
I think you might have missed in the patch that it pauses the firmware
at early boot, in its driver init and it resumes it only after
iommu_request_dm_for_dev() has completed. :)
---
Best Regards, Laurentiu
On Tue, Feb 11, 2020 at 5:04 AM Robin Murphy <[email protected]> wrote:
>
> On 2020-02-11 12:13 pm, Laurentiu Tudor wrote:
> [...]
> >> This is a known issue about DPAA2 MC bus not working well with SMMU
> >> based IO mapping. Adding Laurentiu to the chain who has been looking
> >> into this issue.
> >
> > Yes, I'm closely following the issue. I actually have a workaround
> > (attached) but haven't submitted as it will probably raise a lot of
> > eyebrows. In the mean time I'm following some discussions [1][2][3] on
> > the iommu list which seem to try to tackle what appears to be a similar
> > issue but with framebuffers. My hope is that we will be able to leverage
> > whatever turns out.
>
> Indeed it's more general than framebuffers - in fact there was a
> specific requirement from the IORT side to accommodate network/storage
> controllers with in-memory firmware/configuration data/whatever set up
> by the bootloader that want to be handed off 'live' to Linux because the
> overhead of stopping and restarting them is impractical. Thus this DPAA2
> setup is very much within scope of the desired solution, so please feel
> free to join in (particularly on the DT parts) :)
That's a real problem that nees a solution, but that's not what's
happening here, since cold boots works fine.
Isn't it a whole lot more likely that something isn't
reset/reinitialized properly in u-boot, such that there is lingering
state in the setup, causing this?
> As for right now, note that your patch would only be a partial
> mitigation to slightly reduce the fault window but not remove it
> entirely. To be robust the SMMU driver *has* to know about live streams
> before the first arm_smmu_reset() - hence the need for generic firmware
> bindings - so doing anything from the MC driver is already too late (and
> indeed the current iommu_request_dm_for_dev() mechanism is itself a
> microcosm of the same problem).
This is more likely a live stream that's left behind from the previous
kernel (there are some error messages about being unable to detach
domains, but the errors make it hard to tell what driver didn't unbind
enough).
*BUT*, even with that bug, the system should reboot reliably and come
up clean. So, something isn't clearing up the state *on boot*.
> > In the mean time, can you try the workaround Leo suggested?
>
> Agreed, I'd imagine the command-line option is probably the best choice
> for these platforms, since it's likely to be easier to set that by
> default in the bootloader than faff with rebuilding generic kernel configs.
For the generic user, definitely. I'll give it a go later this week
when I have a bit more spare time with the device physically present.
-Olof
On 11/02/2020 1:55 pm, Laurentiu Tudor wrote:
>
>
> On 11.02.2020 15:04, Robin Murphy wrote:
>> On 2020-02-11 12:13 pm, Laurentiu Tudor wrote:
>> [...]
>>>> This is a known issue about DPAA2 MC bus not working well with SMMU
>>>> based IO mapping. Adding Laurentiu to the chain who has been looking
>>>> into this issue.
>>>
>>> Yes, I'm closely following the issue. I actually have a workaround
>>> (attached) but haven't submitted as it will probably raise a lot of
>>> eyebrows. In the mean time I'm following some discussions [1][2][3]
>>> on the iommu list which seem to try to tackle what appears to be a
>>> similar issue but with framebuffers. My hope is that we will be able
>>> to leverage whatever turns out.
>>
>> Indeed it's more general than framebuffers - in fact there was a
>> specific requirement from the IORT side to accommodate network/storage
>> controllers with in-memory firmware/configuration data/whatever set up
>> by the bootloader that want to be handed off 'live' to Linux because
>> the overhead of stopping and restarting them is impractical. Thus this
>> DPAA2 setup is very much within scope of the desired solution, so
>> please feel free to join in (particularly on the DT parts) :)
>
> Will sure do. Seems that the 2nd approach (the one with list of
> compatibles in arm-smmu) fits really well with our scenario. Will this
> be the way to go forward?
I'm hoping that Thierry's proposal can be made to work out, since it's
closer to how the ACPI version should work, which means we would be able
to do a lot more in shared common code rather than baking magic
knowledge and duplicated functionality into individual IOMMU drivers.
>> As for right now, note that your patch would only be a partial
>> mitigation to slightly reduce the fault window but not remove it
>> entirely. To be robust the SMMU driver *has* to know about live
>> streams before the first arm_smmu_reset() - hence the need for generic
>> firmware bindings - so doing anything from the MC driver is already
>> too late (and indeed the current iommu_request_dm_for_dev() mechanism
>> is itself a microcosm of the same problem).
>
> I think you might have missed in the patch that it pauses the firmware
> at early boot, in its driver init and it resumes it only after
> iommu_request_dm_for_dev() has completed. :)
Ah, from the context I missed that that was non-modular and relying on
initcall trickery... fair enough, in that case I'll downgrade my "it's
insufficient" to "it's ugly and somewhat fragile" :P
Thanks,
Robin.
On 11.02.2020 16:48, Olof Johansson wrote:
> On Tue, Feb 11, 2020 at 5:04 AM Robin Murphy <[email protected]> wrote:
>>
>> On 2020-02-11 12:13 pm, Laurentiu Tudor wrote:
>> [...]
>>>> This is a known issue about DPAA2 MC bus not working well with SMMU
>>>> based IO mapping. Adding Laurentiu to the chain who has been looking
>>>> into this issue.
>>>
>>> Yes, I'm closely following the issue. I actually have a workaround
>>> (attached) but haven't submitted as it will probably raise a lot of
>>> eyebrows. In the mean time I'm following some discussions [1][2][3] on
>>> the iommu list which seem to try to tackle what appears to be a similar
>>> issue but with framebuffers. My hope is that we will be able to leverage
>>> whatever turns out.
>>
>> Indeed it's more general than framebuffers - in fact there was a
>> specific requirement from the IORT side to accommodate network/storage
>> controllers with in-memory firmware/configuration data/whatever set up
>> by the bootloader that want to be handed off 'live' to Linux because the
>> overhead of stopping and restarting them is impractical. Thus this DPAA2
>> setup is very much within scope of the desired solution, so please feel
>> free to join in (particularly on the DT parts) :)
>
> That's a real problem that nees a solution, but that's not what's
> happening here, since cold boots works fine.
>
> Isn't it a whole lot more likely that something isn't
> reset/reinitialized properly in u-boot, such that there is lingering
> state in the setup, causing this?
Ok, so this is completely something else. I don't think our u-boots are
designed to run in ways other than coming from hard reset.
>> As for right now, note that your patch would only be a partial
>> mitigation to slightly reduce the fault window but not remove it
>> entirely. To be robust the SMMU driver *has* to know about live streams
>> before the first arm_smmu_reset() - hence the need for generic firmware
>> bindings - so doing anything from the MC driver is already too late (and
>> indeed the current iommu_request_dm_for_dev() mechanism is itself a
>> microcosm of the same problem).
>
> This is more likely a live stream that's left behind from the previous
> kernel (there are some error messages about being unable to detach
> domains, but the errors make it hard to tell what driver didn't unbind
> enough).
I also noticed those messages. Perhaps our PCI driver doesn't do all the
required cleanup.
> *BUT*, even with that bug, the system should reboot reliably and come
> up clean. So, something isn't clearing up the state *on boot*.
We do test some kexec based "soft-reset" scenarios, didn't hit your
issue but instead we hit this:
https://lkml.org/lkml/2018/9/21/1066
Can you please provide some more info on your scenario?
---
Best Regards, Laurentiu
On Mon, Feb 10, 2020 at 03:22:57PM +0000, Russell King - ARM Linux admin wrote:
> On Mon, Feb 10, 2020 at 04:12:30PM +0100, Olof Johansson wrote:
> > On Thu, Feb 6, 2020 at 11:57 AM Z.q. Hou <[email protected]> wrote:
> > >
> > > Hi Olof,
> > >
> > > Thanks a lot for your comments!
> > > And sorry for my delay respond!
> >
> > Actually, they apply with only minor conflicts on top of current -next.
> >
> > Bjorn, any chance we can get you to pick these up pretty soon? They
> > enable full use of a promising ARM developer system, the SolidRun
> > HoneyComb, and would be quite valuable for me and others to be able to
> > use with mainline or -next without any additional patches applied --
> > which this patchset achieves.
> >
> > I know there are pending revisions based on feedback. I'll leave it up
> > to you and others to determine if that can be done with incremental
> > patches on top, or if it should be fixed before the initial patchset
> > is applied. But all in all, it's holding up adaption by me and surely
> > others of a very interesting platform -- I'm looking to replace my
> > aging MacchiatoBin with one of these and would need PCIe/NVMe to work
> > before I do.
>
> If you're going to be using NVMe, make sure you use a power-fail safe
> version; I've already had one instance where ext4 failed to mount
> because of a corrupted journal using an XPG SX8200 after the Honeycomb
> Serror'd, and then I powered it down after a few hours before later
> booting it back up.
>
> EXT4-fs (nvme0n1p2): INFO: recovery required on readonly filesystem
> EXT4-fs (nvme0n1p2): write access will be enabled during recovery
> JBD2: journal transaction 80849 on nvme0n1p2-8 is corrupt.
> EXT4-fs (nvme0n1p2): error loading journal
... and last night, I just got more ext4fs errors on the NVMe, without
any unclean power cycles:
[73729.556544] EXT4-fs error (device nvme0n1p2): ext4_lookup:1700: inode #917524: comm rm: iget: checksum invalid
[73729.565354] Aborting journal on device nvme0n1p2-8.
[73729.568995] EXT4-fs (nvme0n1p2): Remounting filesystem read-only
[73729.569077] EXT4-fs error (device nvme0n1p2): ext4_journal_check_start:61: Detected aborted journal
[73729.573741] EXT4-fs error (device nvme0n1p2): ext4_lookup:1700: inode #917524: comm rm: iget: checksum invalid
[73729.593330] EXT4-fs error (device nvme0n1p2): ext4_lookup:1700: inode #917524: comm mv: iget: checksum invalid
The affected file is /var/backups/dpkg.status.6.gz
It was cleanly shut down and powered off on the 22nd February, booted
yesterday morning followed by another reboot a few minutes later.
What worries me is the fact that corruption has happened - and if that
happens to a file rather than an inode, it will likely go unnoticed
for a considerably longer time.
I think I'm getting to the point of deciding NVMe or the LX2160A to be
just too unreliable for serious use. I hadn't noticed any issues when
using the rootfs on the eMMC, so it suggests either the NVMe is
unreliable, or there's a problem with PCIe on this platform (which we
kind of know about with Jon's GPU rendering issues.)
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up
On Sat, Feb 29, 2020 at 09:55:50AM +0000, Russell King - ARM Linux admin wrote:
> On Mon, Feb 10, 2020 at 03:22:57PM +0000, Russell King - ARM Linux admin wrote:
> > On Mon, Feb 10, 2020 at 04:12:30PM +0100, Olof Johansson wrote:
> > > On Thu, Feb 6, 2020 at 11:57 AM Z.q. Hou <[email protected]> wrote:
> > > >
> > > > Hi Olof,
> > > >
> > > > Thanks a lot for your comments!
> > > > And sorry for my delay respond!
> > >
> > > Actually, they apply with only minor conflicts on top of current -next.
> > >
> > > Bjorn, any chance we can get you to pick these up pretty soon? They
> > > enable full use of a promising ARM developer system, the SolidRun
> > > HoneyComb, and would be quite valuable for me and others to be able to
> > > use with mainline or -next without any additional patches applied --
> > > which this patchset achieves.
> > >
> > > I know there are pending revisions based on feedback. I'll leave it up
> > > to you and others to determine if that can be done with incremental
> > > patches on top, or if it should be fixed before the initial patchset
> > > is applied. But all in all, it's holding up adaption by me and surely
> > > others of a very interesting platform -- I'm looking to replace my
> > > aging MacchiatoBin with one of these and would need PCIe/NVMe to work
> > > before I do.
> >
> > If you're going to be using NVMe, make sure you use a power-fail safe
> > version; I've already had one instance where ext4 failed to mount
> > because of a corrupted journal using an XPG SX8200 after the Honeycomb
> > Serror'd, and then I powered it down after a few hours before later
> > booting it back up.
> >
> > EXT4-fs (nvme0n1p2): INFO: recovery required on readonly filesystem
> > EXT4-fs (nvme0n1p2): write access will be enabled during recovery
> > JBD2: journal transaction 80849 on nvme0n1p2-8 is corrupt.
> > EXT4-fs (nvme0n1p2): error loading journal
>
> ... and last night, I just got more ext4fs errors on the NVMe, without
> any unclean power cycles:
>
> [73729.556544] EXT4-fs error (device nvme0n1p2): ext4_lookup:1700: inode #917524: comm rm: iget: checksum invalid
> [73729.565354] Aborting journal on device nvme0n1p2-8.
> [73729.568995] EXT4-fs (nvme0n1p2): Remounting filesystem read-only
> [73729.569077] EXT4-fs error (device nvme0n1p2): ext4_journal_check_start:61: Detected aborted journal
> [73729.573741] EXT4-fs error (device nvme0n1p2): ext4_lookup:1700: inode #917524: comm rm: iget: checksum invalid
> [73729.593330] EXT4-fs error (device nvme0n1p2): ext4_lookup:1700: inode #917524: comm mv: iget: checksum invalid
>
> The affected file is /var/backups/dpkg.status.6.gz
>
> It was cleanly shut down and powered off on the 22nd February, booted
> yesterday morning followed by another reboot a few minutes later.
>
> What worries me is the fact that corruption has happened - and if that
> happens to a file rather than an inode, it will likely go unnoticed
> for a considerably longer time.
>
> I think I'm getting to the point of deciding NVMe or the LX2160A to be
> just too unreliable for serious use. I hadn't noticed any issues when
> using the rootfs on the eMMC, so it suggests either the NVMe is
> unreliable, or there's a problem with PCIe on this platform (which we
> kind of know about with Jon's GPU rendering issues.)
Adding Ted and Andreas...
Here's the debugfs -n "id" output for dpkg.status.5.gz (which is fine,
and probably a similar size):
debugfs: id <917527>
0000 a481 0000 30ff 0300 bd8e 475e bd77 4f5e ....0.....G^.wO^
0020 29ca 345e 0000 0000 0000 0100 0002 0000 ).4^............
0040 0000 0800 0100 0000 0af3 0100 0400 0000 ................
0060 0000 0000 0000 0000 4000 0000 8087 3800 [email protected].
0100 0000 0000 0000 0000 0000 0000 0000 0000 ................
*
0140 0000 0000 c40b 4c0a 0000 0000 0000 0000 ......L.........
0160 0000 0000 0000 0000 0000 0000 3884 0000 ............8...
0200 2000 95f2 44b8 bdc9 a4d2 9883 c861 dc92 ...D........a..
0220 bd31 4a5e ecc5 260c 0000 0000 0000 0000 .1J^..&.........
0240 0000 0000 0000 0000 0000 0000 0000 0000 ................
*
and for the affected inode:
debugfs: id <917524>
0000 a481 0000 30ff 0300 3d3d 465e bd77 4f5e ....0...==F^.wO^
0020 29ca 345e 0000 0000 0000 0100 0002 0000 ).4^............
0040 0000 0800 0100 0000 0af3 0100 0400 0000 ................
0060 0000 0000 0000 0000 4000 0000 c088 3800 [email protected].
0100 0000 0000 0000 0000 0000 0000 0000 0000 ................
*
0140 0000 0000 5fc4 cfb4 0000 0000 0000 0000 ...._...........
0160 0000 0000 0000 0000 0000 0000 af23 0000 .............#..
0200 2000 1cc3 ac95 c9c8 a4d2 9883 583e addf ...........X>..
0220 3de0 485e b04d 7151 0000 0000 0000 0000 =.H^.MqQ........
0240 0000 0000 0000 0000 0000 0000 0000 0000 ................
*
and "stat" output:
debugfs: stat <917527>
Inode: 917527 Type: regular Mode: 0644 Flags: 0x80000
Generation: 172755908 Version: 0x00000000:00000001
User: 0 Group: 0 Project: 0 Size: 261936
File ACL: 0
Links: 1 Blockcount: 512
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x5e4f77bd:c9bdb844 -- Fri Feb 21 06:25:01 2020
atime: 0x5e478ebd:92dc61c8 -- Sat Feb 15 06:25:01 2020
mtime: 0x5e34ca29:8398d2a4 -- Sat Feb 1 00:45:29 2020
crtime: 0x5e4a31bd:0c26c5ec -- Mon Feb 17 06:25:01 2020
Size of extra inode fields: 32
Inode checksum: 0xf2958438
EXTENTS:
(0-63):3704704-3704767
debugfs: stat <917524>
Inode: 917524 Type: regular Mode: 0644 Flags: 0x80000
Generation: 3033515103 Version: 0x00000000:00000001
User: 0 Group: 0 Project: 0 Size: 261936
File ACL: 0
Links: 1 Blockcount: 512
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x5e4f77bd:c8c995ac -- Fri Feb 21 06:25:01 2020
atime: 0x5e463d3d:dfad3e58 -- Fri Feb 14 06:25:01 2020
mtime: 0x5e34ca29:8398d2a4 -- Sat Feb 1 00:45:29 2020
crtime: 0x5e48e03d:51714db0 -- Sun Feb 16 06:25:01 2020
Size of extra inode fields: 32
Inode checksum: 0xc31c23af
EXTENTS:
(0-63):3705024-3705087
When using sif (set_inode_info) to re-set the UID to 0 on this (so
provoke the checksum to be updated):
debugfs: id <917524>
0000 a481 0000 30ff 0300 3d3d 465e bd77 4f5e ....0...==F^.wO^
0020 29ca 345e 0000 0000 0000 0100 0002 0000 ).4^............
0040 0000 0800 0100 0000 0af3 0100 0400 0000 ................
0060 0000 0000 0000 0000 4000 0000 c088 3800 [email protected].
0100 0000 0000 0000 0000 0000 0000 0000 0000 ................
*
0140 0000 0000 5fc4 cfb4 0000 0000 0000 0000 ...._...........
0160 0000 0000 0000 0000 0000 0000 b61f 0000 ................
^^^^
0200 2000 aa15 ac95 c9c8 a4d2 9883 583e addf ...........X>..
^^^^
0220 3de0 485e b04d 7151 0000 0000 0000 0000 =.H^.MqQ........
0240 0000 0000 0000 0000 0000 0000 0000 0000 ................
*
The values with "^^^^" are the checksum, which are the only values
that have changed here - the checksum is now 0x15aa1fb6 rather than
0xc31c23af.
With that changed, running e2fsck -n on the filesystem results in a
pass:
root@cex7:~# e2fsck -n /dev/nvme0n1p2
e2fsck 1.44.5 (15-Dec-2018)
Warning: skipping journal recovery because doing a read-only filesystem check.
/dev/nvme0n1p2 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/nvme0n1p2: 121163/2097152 files (0.1% non-contiguous), 1349227/8388608 blocks
and the file now appears to be intact (being a gzip file, gzip verifies
that the contents are now as it expects.)
So, it looks like the _only_ issue is that the checksum on the inode
became invalid, which seems to suggest that it *isn't* a NVMe nor PCIe
issue.
I wonder whether the journal would contain anything useful, but I don't
know how to use debugfs to find that out - while I can dump the journal,
I'd need to know which block contains the inode, and then work out where
in the journal that block was going to be written. If that would help,
let me know ASAP as I'll hold off rebooting the platform for a while
(which means the filesystem will remain as-is - and yes, I have the
debugfs file for e2undo to put stuff back.) Maybe it's possible to pull
the block number out of the e2undo file?
tune2fs says:
Checksum type: crc32c
Checksum: 0x682f91b9
I guess this is what is used to checksum the inodes? If so, it's using
the kernel's crc32c-generic driver (according to /proc/crypto).
Could it be a race condition, or some problem that's specific to the
ARM64 kernel that's provoking this corruption?
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up
On Sat, Feb 29, 2020 at 11:04:56AM +0000, Russell King - ARM Linux admin wrote:
> On Sat, Feb 29, 2020 at 09:55:50AM +0000, Russell King - ARM Linux admin wrote:
> > On Mon, Feb 10, 2020 at 03:22:57PM +0000, Russell King - ARM Linux admin wrote:
> > > On Mon, Feb 10, 2020 at 04:12:30PM +0100, Olof Johansson wrote:
> > > > On Thu, Feb 6, 2020 at 11:57 AM Z.q. Hou <[email protected]> wrote:
> > > > >
> > > > > Hi Olof,
> > > > >
> > > > > Thanks a lot for your comments!
> > > > > And sorry for my delay respond!
> > > >
> > > > Actually, they apply with only minor conflicts on top of current -next.
> > > >
> > > > Bjorn, any chance we can get you to pick these up pretty soon? They
> > > > enable full use of a promising ARM developer system, the SolidRun
> > > > HoneyComb, and would be quite valuable for me and others to be able to
> > > > use with mainline or -next without any additional patches applied --
> > > > which this patchset achieves.
> > > >
> > > > I know there are pending revisions based on feedback. I'll leave it up
> > > > to you and others to determine if that can be done with incremental
> > > > patches on top, or if it should be fixed before the initial patchset
> > > > is applied. But all in all, it's holding up adaption by me and surely
> > > > others of a very interesting platform -- I'm looking to replace my
> > > > aging MacchiatoBin with one of these and would need PCIe/NVMe to work
> > > > before I do.
> > >
> > > If you're going to be using NVMe, make sure you use a power-fail safe
> > > version; I've already had one instance where ext4 failed to mount
> > > because of a corrupted journal using an XPG SX8200 after the Honeycomb
> > > Serror'd, and then I powered it down after a few hours before later
> > > booting it back up.
> > >
> > > EXT4-fs (nvme0n1p2): INFO: recovery required on readonly filesystem
> > > EXT4-fs (nvme0n1p2): write access will be enabled during recovery
> > > JBD2: journal transaction 80849 on nvme0n1p2-8 is corrupt.
> > > EXT4-fs (nvme0n1p2): error loading journal
> >
> > ... and last night, I just got more ext4fs errors on the NVMe, without
> > any unclean power cycles:
> >
> > [73729.556544] EXT4-fs error (device nvme0n1p2): ext4_lookup:1700: inode #917524: comm rm: iget: checksum invalid
> > [73729.565354] Aborting journal on device nvme0n1p2-8.
> > [73729.568995] EXT4-fs (nvme0n1p2): Remounting filesystem read-only
> > [73729.569077] EXT4-fs error (device nvme0n1p2): ext4_journal_check_start:61: Detected aborted journal
> > [73729.573741] EXT4-fs error (device nvme0n1p2): ext4_lookup:1700: inode #917524: comm rm: iget: checksum invalid
> > [73729.593330] EXT4-fs error (device nvme0n1p2): ext4_lookup:1700: inode #917524: comm mv: iget: checksum invalid
> >
> > The affected file is /var/backups/dpkg.status.6.gz
> >
> > It was cleanly shut down and powered off on the 22nd February, booted
> > yesterday morning followed by another reboot a few minutes later.
> >
> > What worries me is the fact that corruption has happened - and if that
> > happens to a file rather than an inode, it will likely go unnoticed
> > for a considerably longer time.
> >
> > I think I'm getting to the point of deciding NVMe or the LX2160A to be
> > just too unreliable for serious use. I hadn't noticed any issues when
> > using the rootfs on the eMMC, so it suggests either the NVMe is
> > unreliable, or there's a problem with PCIe on this platform (which we
> > kind of know about with Jon's GPU rendering issues.)
>
> Adding Ted and Andreas...
>
> Here's the debugfs -n "id" output for dpkg.status.5.gz (which is fine,
> and probably a similar size):
>
> debugfs: id <917527>
> 0000 a481 0000 30ff 0300 bd8e 475e bd77 4f5e ....0.....G^.wO^
> 0020 29ca 345e 0000 0000 0000 0100 0002 0000 ).4^............
> 0040 0000 0800 0100 0000 0af3 0100 0400 0000 ................
> 0060 0000 0000 0000 0000 4000 0000 8087 3800 [email protected].
> 0100 0000 0000 0000 0000 0000 0000 0000 0000 ................
> *
> 0140 0000 0000 c40b 4c0a 0000 0000 0000 0000 ......L.........
> 0160 0000 0000 0000 0000 0000 0000 3884 0000 ............8...
> 0200 2000 95f2 44b8 bdc9 a4d2 9883 c861 dc92 ...D........a..
> 0220 bd31 4a5e ecc5 260c 0000 0000 0000 0000 .1J^..&.........
> 0240 0000 0000 0000 0000 0000 0000 0000 0000 ................
> *
>
> and for the affected inode:
> debugfs: id <917524>
> 0000 a481 0000 30ff 0300 3d3d 465e bd77 4f5e ....0...==F^.wO^
> 0020 29ca 345e 0000 0000 0000 0100 0002 0000 ).4^............
> 0040 0000 0800 0100 0000 0af3 0100 0400 0000 ................
> 0060 0000 0000 0000 0000 4000 0000 c088 3800 [email protected].
> 0100 0000 0000 0000 0000 0000 0000 0000 0000 ................
> *
> 0140 0000 0000 5fc4 cfb4 0000 0000 0000 0000 ...._...........
> 0160 0000 0000 0000 0000 0000 0000 af23 0000 .............#..
> 0200 2000 1cc3 ac95 c9c8 a4d2 9883 583e addf ...........X>..
> 0220 3de0 485e b04d 7151 0000 0000 0000 0000 =.H^.MqQ........
> 0240 0000 0000 0000 0000 0000 0000 0000 0000 ................
> *
>
> and "stat" output:
> debugfs: stat <917527>
> Inode: 917527 Type: regular Mode: 0644 Flags: 0x80000
> Generation: 172755908 Version: 0x00000000:00000001
> User: 0 Group: 0 Project: 0 Size: 261936
> File ACL: 0
> Links: 1 Blockcount: 512
> Fragment: Address: 0 Number: 0 Size: 0
> ctime: 0x5e4f77bd:c9bdb844 -- Fri Feb 21 06:25:01 2020
> atime: 0x5e478ebd:92dc61c8 -- Sat Feb 15 06:25:01 2020
> mtime: 0x5e34ca29:8398d2a4 -- Sat Feb 1 00:45:29 2020
> crtime: 0x5e4a31bd:0c26c5ec -- Mon Feb 17 06:25:01 2020
> Size of extra inode fields: 32
> Inode checksum: 0xf2958438
> EXTENTS:
> (0-63):3704704-3704767
> debugfs: stat <917524>
> Inode: 917524 Type: regular Mode: 0644 Flags: 0x80000
> Generation: 3033515103 Version: 0x00000000:00000001
> User: 0 Group: 0 Project: 0 Size: 261936
> File ACL: 0
> Links: 1 Blockcount: 512
> Fragment: Address: 0 Number: 0 Size: 0
> ctime: 0x5e4f77bd:c8c995ac -- Fri Feb 21 06:25:01 2020
> atime: 0x5e463d3d:dfad3e58 -- Fri Feb 14 06:25:01 2020
> mtime: 0x5e34ca29:8398d2a4 -- Sat Feb 1 00:45:29 2020
> crtime: 0x5e48e03d:51714db0 -- Sun Feb 16 06:25:01 2020
> Size of extra inode fields: 32
> Inode checksum: 0xc31c23af
> EXTENTS:
> (0-63):3705024-3705087
>
> When using sif (set_inode_info) to re-set the UID to 0 on this (so
> provoke the checksum to be updated):
>
> debugfs: id <917524>
> 0000 a481 0000 30ff 0300 3d3d 465e bd77 4f5e ....0...==F^.wO^
> 0020 29ca 345e 0000 0000 0000 0100 0002 0000 ).4^............
> 0040 0000 0800 0100 0000 0af3 0100 0400 0000 ................
> 0060 0000 0000 0000 0000 4000 0000 c088 3800 [email protected].
> 0100 0000 0000 0000 0000 0000 0000 0000 0000 ................
> *
> 0140 0000 0000 5fc4 cfb4 0000 0000 0000 0000 ...._...........
> 0160 0000 0000 0000 0000 0000 0000 b61f 0000 ................
> ^^^^
> 0200 2000 aa15 ac95 c9c8 a4d2 9883 583e addf ...........X>..
> ^^^^
> 0220 3de0 485e b04d 7151 0000 0000 0000 0000 =.H^.MqQ........
> 0240 0000 0000 0000 0000 0000 0000 0000 0000 ................
> *
>
> The values with "^^^^" are the checksum, which are the only values
> that have changed here - the checksum is now 0x15aa1fb6 rather than
> 0xc31c23af.
>
> With that changed, running e2fsck -n on the filesystem results in a
> pass:
>
> root@cex7:~# e2fsck -n /dev/nvme0n1p2
> e2fsck 1.44.5 (15-Dec-2018)
> Warning: skipping journal recovery because doing a read-only filesystem check.
> /dev/nvme0n1p2 contains a file system with errors, check forced.
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Pass 3: Checking directory connectivity
> Pass 4: Checking reference counts
> Pass 5: Checking group summary information
> /dev/nvme0n1p2: 121163/2097152 files (0.1% non-contiguous), 1349227/8388608 blocks
>
> and the file now appears to be intact (being a gzip file, gzip verifies
> that the contents are now as it expects.)
>
> So, it looks like the _only_ issue is that the checksum on the inode
> became invalid, which seems to suggest that it *isn't* a NVMe nor PCIe
> issue.
>
> I wonder whether the journal would contain anything useful, but I don't
> know how to use debugfs to find that out - while I can dump the journal,
> I'd need to know which block contains the inode, and then work out where
> in the journal that block was going to be written. If that would help,
> let me know ASAP as I'll hold off rebooting the platform for a while
> (which means the filesystem will remain as-is - and yes, I have the
> debugfs file for e2undo to put stuff back.) Maybe it's possible to pull
> the block number out of the e2undo file?
Okay, the inode was stored in block 3670049, and the journal appears
to contains no entries for that block.
> tune2fs says:
>
> Checksum type: crc32c
> Checksum: 0x682f91b9
>
> I guess this is what is used to checksum the inodes? If so, it's using
> the kernel's crc32c-generic driver (according to /proc/crypto).
>
> Could it be a race condition, or some problem that's specific to the
> ARM64 kernel that's provoking this corruption?
Something else occurs to me:
root@cex7:~# ls -li --time=ctime --full-time /var/backups/dpkg.status*
917622 -rw-r--r-- 1 root root 999052 2020-02-29 06:25:01.852231277 +0000 /var/backups/dpkg.status
917583 -rw-r--r-- 1 root root 999052 2020-02-21 06:25:01.958160960 +0000 /var/backups/dpkg.status.0
917520 -rw-r--r-- 1 root root 261936 2020-02-21 06:25:01.954161050 +0000 /var/backups/dpkg.status.1.gz
917531 -rw-r--r-- 1 root root 261936 2020-02-21 06:25:01.854163293 +0000 /var/backups/dpkg.status.2.gz
917532 -rw-r--r-- 1 root root 261936 2020-02-21 06:25:01.850163383 +0000 /var/backups/dpkg.status.3.gz
917509 -rw-r--r-- 1 root root 261936 2020-02-21 06:25:01.850163383 +0000 /var/backups/dpkg.status.4.gz
917527 -rw-r--r-- 1 root root 261936 2020-02-21 06:25:01.846163473 +0000 /var/backups/dpkg.status.5.gz
917524 -rw-r--r-- 1 root root 261936 2020-02-21 06:25:01.842163563 +0000 /var/backups/dpkg.status.6.gz
So the last time that the kernel changed inode 917524 was on the 21th
of February, probably when it was last renamed by logrotate, and like
several other files stored in the same inode block. Yet, _only_ the
checksum for 917524 was corrupted, the rest were fine.
I would guess that logrotate behaves as follows:
- remove /var/backups/dpkg.status.6.gz
- rename /var/backups/dpkg.status.5.gz to /var/backups/dpkg.status.6.gz
- repeat for other dpkg.status.*.gz files
- gzip /var/backups/dpkg.status.0 to /var/backups/dpkg.status.1.gz
- rename /var/backups/dpkg.status to /var/backups/dpkg.status.0
- create new /var/backups/dpkg.status
Looking at the inode block in the e2undo file, inode 917524 is at
offset 0x300 into the block, which means the first inode in the
block is 917521 and the last is 917536, which means we have several
of the dpkg.status.* files that are stored in this inode block.
That would've meant that the inode for /var/backups/dpkg.status.6.gz
would have been updated just before the inode for
/var/backups/dpkg.status.5.gz. I wonder if the inode block was
written out somehow out of order, with the ctime for
/var/backups/dpkg.status.6.gz having been updated but not the checksum
as a result of the later changes - maybe as a result of having
executed on a different CPU? That would suggest a weakness in the
ARM64 locking implementation, coherency issues, or interconnect issues.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up
On Sat, Feb 29, 2020 at 12:08:28PM +0000, Russell King - ARM Linux admin wrote:
> On Sat, Feb 29, 2020 at 11:04:56AM +0000, Russell King - ARM Linux admin wrote:
> > On Sat, Feb 29, 2020 at 09:55:50AM +0000, Russell King - ARM Linux admin wrote:
> > > On Mon, Feb 10, 2020 at 03:22:57PM +0000, Russell King - ARM Linux admin wrote:
> > > > On Mon, Feb 10, 2020 at 04:12:30PM +0100, Olof Johansson wrote:
> > > > > On Thu, Feb 6, 2020 at 11:57 AM Z.q. Hou <[email protected]> wrote:
> > > > > >
> > > > > > Hi Olof,
> > > > > >
> > > > > > Thanks a lot for your comments!
> > > > > > And sorry for my delay respond!
> > > > >
> > > > > Actually, they apply with only minor conflicts on top of current -next.
> > > > >
> > > > > Bjorn, any chance we can get you to pick these up pretty soon? They
> > > > > enable full use of a promising ARM developer system, the SolidRun
> > > > > HoneyComb, and would be quite valuable for me and others to be able to
> > > > > use with mainline or -next without any additional patches applied --
> > > > > which this patchset achieves.
> > > > >
> > > > > I know there are pending revisions based on feedback. I'll leave it up
> > > > > to you and others to determine if that can be done with incremental
> > > > > patches on top, or if it should be fixed before the initial patchset
> > > > > is applied. But all in all, it's holding up adaption by me and surely
> > > > > others of a very interesting platform -- I'm looking to replace my
> > > > > aging MacchiatoBin with one of these and would need PCIe/NVMe to work
> > > > > before I do.
> > > >
> > > > If you're going to be using NVMe, make sure you use a power-fail safe
> > > > version; I've already had one instance where ext4 failed to mount
> > > > because of a corrupted journal using an XPG SX8200 after the Honeycomb
> > > > Serror'd, and then I powered it down after a few hours before later
> > > > booting it back up.
> > > >
> > > > EXT4-fs (nvme0n1p2): INFO: recovery required on readonly filesystem
> > > > EXT4-fs (nvme0n1p2): write access will be enabled during recovery
> > > > JBD2: journal transaction 80849 on nvme0n1p2-8 is corrupt.
> > > > EXT4-fs (nvme0n1p2): error loading journal
> > >
> > > ... and last night, I just got more ext4fs errors on the NVMe, without
> > > any unclean power cycles:
> > >
> > > [73729.556544] EXT4-fs error (device nvme0n1p2): ext4_lookup:1700: inode #917524: comm rm: iget: checksum invalid
> > > [73729.565354] Aborting journal on device nvme0n1p2-8.
> > > [73729.568995] EXT4-fs (nvme0n1p2): Remounting filesystem read-only
> > > [73729.569077] EXT4-fs error (device nvme0n1p2): ext4_journal_check_start:61: Detected aborted journal
> > > [73729.573741] EXT4-fs error (device nvme0n1p2): ext4_lookup:1700: inode #917524: comm rm: iget: checksum invalid
> > > [73729.593330] EXT4-fs error (device nvme0n1p2): ext4_lookup:1700: inode #917524: comm mv: iget: checksum invalid
> > >
> > > The affected file is /var/backups/dpkg.status.6.gz
> > >
> > > It was cleanly shut down and powered off on the 22nd February, booted
> > > yesterday morning followed by another reboot a few minutes later.
> > >
> > > What worries me is the fact that corruption has happened - and if that
> > > happens to a file rather than an inode, it will likely go unnoticed
> > > for a considerably longer time.
> > >
> > > I think I'm getting to the point of deciding NVMe or the LX2160A to be
> > > just too unreliable for serious use. I hadn't noticed any issues when
> > > using the rootfs on the eMMC, so it suggests either the NVMe is
> > > unreliable, or there's a problem with PCIe on this platform (which we
> > > kind of know about with Jon's GPU rendering issues.)
> >
> > Adding Ted and Andreas...
> >
> > Here's the debugfs -n "id" output for dpkg.status.5.gz (which is fine,
> > and probably a similar size):
> >
> > debugfs: id <917527>
> > 0000 a481 0000 30ff 0300 bd8e 475e bd77 4f5e ....0.....G^.wO^
> > 0020 29ca 345e 0000 0000 0000 0100 0002 0000 ).4^............
> > 0040 0000 0800 0100 0000 0af3 0100 0400 0000 ................
> > 0060 0000 0000 0000 0000 4000 0000 8087 3800 [email protected].
> > 0100 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > *
> > 0140 0000 0000 c40b 4c0a 0000 0000 0000 0000 ......L.........
> > 0160 0000 0000 0000 0000 0000 0000 3884 0000 ............8...
> > 0200 2000 95f2 44b8 bdc9 a4d2 9883 c861 dc92 ...D........a..
> > 0220 bd31 4a5e ecc5 260c 0000 0000 0000 0000 .1J^..&.........
> > 0240 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > *
> >
> > and for the affected inode:
> > debugfs: id <917524>
> > 0000 a481 0000 30ff 0300 3d3d 465e bd77 4f5e ....0...==F^.wO^
> > 0020 29ca 345e 0000 0000 0000 0100 0002 0000 ).4^............
> > 0040 0000 0800 0100 0000 0af3 0100 0400 0000 ................
> > 0060 0000 0000 0000 0000 4000 0000 c088 3800 [email protected].
> > 0100 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > *
> > 0140 0000 0000 5fc4 cfb4 0000 0000 0000 0000 ...._...........
> > 0160 0000 0000 0000 0000 0000 0000 af23 0000 .............#..
> > 0200 2000 1cc3 ac95 c9c8 a4d2 9883 583e addf ...........X>..
> > 0220 3de0 485e b04d 7151 0000 0000 0000 0000 =.H^.MqQ........
> > 0240 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > *
> >
> > and "stat" output:
> > debugfs: stat <917527>
> > Inode: 917527 Type: regular Mode: 0644 Flags: 0x80000
> > Generation: 172755908 Version: 0x00000000:00000001
> > User: 0 Group: 0 Project: 0 Size: 261936
> > File ACL: 0
> > Links: 1 Blockcount: 512
> > Fragment: Address: 0 Number: 0 Size: 0
> > ctime: 0x5e4f77bd:c9bdb844 -- Fri Feb 21 06:25:01 2020
> > atime: 0x5e478ebd:92dc61c8 -- Sat Feb 15 06:25:01 2020
> > mtime: 0x5e34ca29:8398d2a4 -- Sat Feb 1 00:45:29 2020
> > crtime: 0x5e4a31bd:0c26c5ec -- Mon Feb 17 06:25:01 2020
> > Size of extra inode fields: 32
> > Inode checksum: 0xf2958438
> > EXTENTS:
> > (0-63):3704704-3704767
> > debugfs: stat <917524>
> > Inode: 917524 Type: regular Mode: 0644 Flags: 0x80000
> > Generation: 3033515103 Version: 0x00000000:00000001
> > User: 0 Group: 0 Project: 0 Size: 261936
> > File ACL: 0
> > Links: 1 Blockcount: 512
> > Fragment: Address: 0 Number: 0 Size: 0
> > ctime: 0x5e4f77bd:c8c995ac -- Fri Feb 21 06:25:01 2020
> > atime: 0x5e463d3d:dfad3e58 -- Fri Feb 14 06:25:01 2020
> > mtime: 0x5e34ca29:8398d2a4 -- Sat Feb 1 00:45:29 2020
> > crtime: 0x5e48e03d:51714db0 -- Sun Feb 16 06:25:01 2020
> > Size of extra inode fields: 32
> > Inode checksum: 0xc31c23af
> > EXTENTS:
> > (0-63):3705024-3705087
> >
> > When using sif (set_inode_info) to re-set the UID to 0 on this (so
> > provoke the checksum to be updated):
> >
> > debugfs: id <917524>
> > 0000 a481 0000 30ff 0300 3d3d 465e bd77 4f5e ....0...==F^.wO^
> > 0020 29ca 345e 0000 0000 0000 0100 0002 0000 ).4^............
> > 0040 0000 0800 0100 0000 0af3 0100 0400 0000 ................
> > 0060 0000 0000 0000 0000 4000 0000 c088 3800 [email protected].
> > 0100 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > *
> > 0140 0000 0000 5fc4 cfb4 0000 0000 0000 0000 ...._...........
> > 0160 0000 0000 0000 0000 0000 0000 b61f 0000 ................
> > ^^^^
> > 0200 2000 aa15 ac95 c9c8 a4d2 9883 583e addf ...........X>..
> > ^^^^
> > 0220 3de0 485e b04d 7151 0000 0000 0000 0000 =.H^.MqQ........
> > 0240 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > *
> >
> > The values with "^^^^" are the checksum, which are the only values
> > that have changed here - the checksum is now 0x15aa1fb6 rather than
> > 0xc31c23af.
> >
> > With that changed, running e2fsck -n on the filesystem results in a
> > pass:
> >
> > root@cex7:~# e2fsck -n /dev/nvme0n1p2
> > e2fsck 1.44.5 (15-Dec-2018)
> > Warning: skipping journal recovery because doing a read-only filesystem check.
> > /dev/nvme0n1p2 contains a file system with errors, check forced.
> > Pass 1: Checking inodes, blocks, and sizes
> > Pass 2: Checking directory structure
> > Pass 3: Checking directory connectivity
> > Pass 4: Checking reference counts
> > Pass 5: Checking group summary information
> > /dev/nvme0n1p2: 121163/2097152 files (0.1% non-contiguous), 1349227/8388608 blocks
> >
> > and the file now appears to be intact (being a gzip file, gzip verifies
> > that the contents are now as it expects.)
> >
> > So, it looks like the _only_ issue is that the checksum on the inode
> > became invalid, which seems to suggest that it *isn't* a NVMe nor PCIe
> > issue.
> >
> > I wonder whether the journal would contain anything useful, but I don't
> > know how to use debugfs to find that out - while I can dump the journal,
> > I'd need to know which block contains the inode, and then work out where
> > in the journal that block was going to be written. If that would help,
> > let me know ASAP as I'll hold off rebooting the platform for a while
> > (which means the filesystem will remain as-is - and yes, I have the
> > debugfs file for e2undo to put stuff back.) Maybe it's possible to pull
> > the block number out of the e2undo file?
>
> Okay, the inode was stored in block 3670049, and the journal appears
> to contains no entries for that block.
>
> > tune2fs says:
> >
> > Checksum type: crc32c
> > Checksum: 0x682f91b9
> >
> > I guess this is what is used to checksum the inodes? If so, it's using
> > the kernel's crc32c-generic driver (according to /proc/crypto).
> >
> > Could it be a race condition, or some problem that's specific to the
> > ARM64 kernel that's provoking this corruption?
>
> Something else occurs to me:
>
> root@cex7:~# ls -li --time=ctime --full-time /var/backups/dpkg.status*
> 917622 -rw-r--r-- 1 root root 999052 2020-02-29 06:25:01.852231277 +0000 /var/backups/dpkg.status
> 917583 -rw-r--r-- 1 root root 999052 2020-02-21 06:25:01.958160960 +0000 /var/backups/dpkg.status.0
> 917520 -rw-r--r-- 1 root root 261936 2020-02-21 06:25:01.954161050 +0000 /var/backups/dpkg.status.1.gz
> 917531 -rw-r--r-- 1 root root 261936 2020-02-21 06:25:01.854163293 +0000 /var/backups/dpkg.status.2.gz
> 917532 -rw-r--r-- 1 root root 261936 2020-02-21 06:25:01.850163383 +0000 /var/backups/dpkg.status.3.gz
> 917509 -rw-r--r-- 1 root root 261936 2020-02-21 06:25:01.850163383 +0000 /var/backups/dpkg.status.4.gz
> 917527 -rw-r--r-- 1 root root 261936 2020-02-21 06:25:01.846163473 +0000 /var/backups/dpkg.status.5.gz
> 917524 -rw-r--r-- 1 root root 261936 2020-02-21 06:25:01.842163563 +0000 /var/backups/dpkg.status.6.gz
>
> So the last time that the kernel changed inode 917524 was on the 21th
> of February, probably when it was last renamed by logrotate, and like
> several other files stored in the same inode block. Yet, _only_ the
> checksum for 917524 was corrupted, the rest were fine.
>
> I would guess that logrotate behaves as follows:
> - remove /var/backups/dpkg.status.6.gz
> - rename /var/backups/dpkg.status.5.gz to /var/backups/dpkg.status.6.gz
> - repeat for other dpkg.status.*.gz files
> - gzip /var/backups/dpkg.status.0 to /var/backups/dpkg.status.1.gz
> - rename /var/backups/dpkg.status to /var/backups/dpkg.status.0
> - create new /var/backups/dpkg.status
>
> Looking at the inode block in the e2undo file, inode 917524 is at
> offset 0x300 into the block, which means the first inode in the
> block is 917521 and the last is 917536, which means we have several
> of the dpkg.status.* files that are stored in this inode block.
>
> That would've meant that the inode for /var/backups/dpkg.status.6.gz
> would have been updated just before the inode for
> /var/backups/dpkg.status.5.gz. I wonder if the inode block was
> written out somehow out of order, with the ctime for
> /var/backups/dpkg.status.6.gz having been updated but not the checksum
> as a result of the later changes - maybe as a result of having
> executed on a different CPU? That would suggest a weakness in the
> ARM64 locking implementation, coherency issues, or interconnect issues.
Looking at the errata configuration, I have:
# ARM errata workarounds via the alternatives framework
#
CONFIG_ARM64_WORKAROUND_CLEAN_CACHE=y
CONFIG_ARM64_ERRATUM_826319=y
CONFIG_ARM64_ERRATUM_827319=y
CONFIG_ARM64_ERRATUM_824069=y
CONFIG_ARM64_ERRATUM_819472=y
CONFIG_ARM64_ERRATUM_832075=y
CONFIG_ARM64_ERRATUM_834220=y
CONFIG_ARM64_ERRATUM_845719=y
CONFIG_ARM64_ERRATUM_843419=y
CONFIG_ARM64_ERRATUM_1024718=y
CONFIG_ARM64_ERRATUM_1418040=y
CONFIG_ARM64_ERRATUM_1165522=y
CONFIG_ARM64_ERRATUM_1286807=y
CONFIG_ARM64_ERRATUM_1319367=y
CONFIG_ARM64_ERRATUM_1463225=y
# CONFIG_ARM64_ERRATUM_1542419 is not set
# CONFIG_CAVIUM_ERRATUM_22375 is not set
# CONFIG_CAVIUM_ERRATUM_23154 is not set
# CONFIG_CAVIUM_ERRATUM_27456 is not set
# CONFIG_CAVIUM_ERRATUM_30115 is not set
# CONFIG_CAVIUM_TX2_ERRATUM_219 is not set
CONFIG_QCOM_FALKOR_ERRATUM_1003=y
CONFIG_ARM64_WORKAROUND_REPEAT_TLBI=y
CONFIG_QCOM_FALKOR_ERRATUM_1009=y
CONFIG_QCOM_QDF2400_ERRATUM_0065=y
# CONFIG_SOCIONEXT_SYNQUACER_PREITS is not set
# CONFIG_HISILICON_ERRATUM_161600802 is not set
CONFIG_QCOM_FALKOR_ERRATUM_E1041=y
# CONFIG_FUJITSU_ERRATUM_010001 is not set
# end of ARM errata workarounds via the alternatives framework
...
CONFIG_FSL_ERRATUM_A008585=y
CONFIG_HISILICON_ERRATUM_161010101=y
CONFIG_ARM64_ERRATUM_858921=y
so I don't think it's a missing errata kconfig setting, unless there's
an erratum that isn't in v5.5 that's necessary.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up
On Sat, Feb 29, 2020 at 11:04:56AM +0000, Russell King - ARM Linux admin wrote:
>
> Could it be a race condition, or some problem that's specific to the
> ARM64 kernel that's provoking this corruption?
Since I got brought in mid-way through this discussion, can someone
summarize the vital details of the bughunt? What kernel version is
involved, and is this a regression? If so, what's the last version of
the kernel where you didn't have a problem on this hardware?
Can you trigger this failure reliably?
Unfortunately, while I'm regularly running xfstests on x86_64 on a
Google Compute Engine VM, I'm not doing any runs on arm64. I can
certainly build an arm-64.
There's a test-appliance designed to be run on ARM64 here[1].
[1] https://kernel.org/pub/linux/kernel/people/tytso/kvm-xfstests/xfstests-amd64.tar.xz
which is a Debian chroot, designed to be run via android-xfstests[2], but
if you unpack it, it should be possible to enter the chroot and
trigger the xfstests run manually on any arm64 system.
[2] https://thunk.org/android-xfstests
Does anyone know if kernel CI is running xfstests regularly?
Cheers,
- Ted
On Sat, Feb 29, 2020 at 10:19:07AM -0500, Theodore Y. Ts'o wrote:
> On Sat, Feb 29, 2020 at 11:04:56AM +0000, Russell King - ARM Linux admin wrote:
> > Could it be a race condition, or some problem that's specific to the
> > ARM64 kernel that's provoking this corruption?
>
> Since I got brought in mid-way through this discussion, can someone
> summarize the vital details of the bughunt? What kernel version is
> involved, and is this a regression? If so, what's the last version of
> the kernel where you didn't have a problem on this hardware?
It's a new platform, I've run most 5.x kernels on it, but only recently
have I had a NVMe. Currently running a 5.5 based kernel (for which I
have to patch in support for the platform), and I've no idea if it is
a regression or not.
> Can you trigger this failure reliably?
No - the very first time I ended up with a corrupted ext4 fs was on the
8th February, and at that time it was put down to the NVMe not being
power-off safe: the machine had crashed sometime over night, resulting
in a section of my network going offline (due to a pause frame storm).
So, I powered it down from crashed state - and from what people tell me,
NVMe _may_ keep blocks unwritten to safe media for a considerable time.
I never bothered to investigate it because the explanation seemed
reasonable, and manually running e2fsck fixed the filesystem.
The system was then booted back into using the NVMe rootfs, and
continued to do so without apparent issue until the 21st Feb, when I
cleanly shut it down, and powered it off. During the time it was
running, it likely saw many reboots of the 5.5 kernel.
I powered it back on yesterday morning, and this morning it found the
fs corruption while trying to do a logrotate.
As I say in my last email, I suspect it isn't an ext4 bug, but either
a locking implementation issue, coherency issue, or interconnect issue.
The 4k block with the affected inode looks perfectly reasonable with
the only exception that the checksum is incorrect for that one inode -
and other inodes stored in the same 4k block were modified afterwards.
It suggests to me that the writes to update the two 16-bit words
containing the checksum were somehow lost for this particular inode.
> Unfortunately, while I'm regularly running xfstests on x86_64 on a
> Google Compute Engine VM, I'm not doing any runs on arm64. I can
> certainly build an arm-64.
>
> There's a test-appliance designed to be run on ARM64 here[1].
>
> [1] https://kernel.org/pub/linux/kernel/people/tytso/kvm-xfstests/xfstests-amd64.tar.xz
The filename seems to say "amd64" not "arm64" ?
> which is a Debian chroot, designed to be run via android-xfstests[2], but
> if you unpack it, it should be possible to enter the chroot and
> trigger the xfstests run manually on any arm64 system.
>
> [2] https://thunk.org/android-xfstests
>
> Does anyone know if kernel CI is running xfstests regularly?
I don't know...
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up
On Sat, Feb 29, 2020 at 05:03:28PM +0000, Russell King - ARM Linux admin wrote:
> > There's a test-appliance designed to be run on ARM64 here[1].
> >
> > [1] https://kernel.org/pub/linux/kernel/people/tytso/kvm-xfstests/xfstests-amd64.tar.xz
>
> The filename seems to say "amd64" not "arm64" ?
Sorry, I cut and pasted the wrong link: s/amd64/arm64/
If there are arm64-specific locking issues, we can probably flush them
out if we could figure out some way of running some of the stress
tests in xfstests. I don't know a whole lot about arm-64
architectures; would running xfstests on, say, an Amazon AWS arm-based
VM be representative of your new architecture? Or are there a lot of
sub-architecture differences in the arm-64 world?
- Ted
On Sat, Feb 29, 2020 at 11:04:56AM +0000, Russell King - ARM Linux admin wrote:
> Adding Ted and Andreas...
>
> Here's the debugfs -n "id" output for dpkg.status.5.gz (which is fine,
> and probably a similar size):
>
> debugfs: id <917527>
> 0000 a481 0000 30ff 0300 bd8e 475e bd77 4f5e ....0.....G^.wO^
> 0020 29ca 345e 0000 0000 0000 0100 0002 0000 ).4^............
> 0040 0000 0800 0100 0000 0af3 0100 0400 0000 ................
> 0060 0000 0000 0000 0000 4000 0000 8087 3800 [email protected].
> 0100 0000 0000 0000 0000 0000 0000 0000 0000 ................
> *
> 0140 0000 0000 c40b 4c0a 0000 0000 0000 0000 ......L.........
> 0160 0000 0000 0000 0000 0000 0000 3884 0000 ............8...
> 0200 2000 95f2 44b8 bdc9 a4d2 9883 c861 dc92 ...D........a..
> 0220 bd31 4a5e ecc5 260c 0000 0000 0000 0000 .1J^..&.........
> 0240 0000 0000 0000 0000 0000 0000 0000 0000 ................
> *
>
> and for the affected inode:
> debugfs: id <917524>
> 0000 a481 0000 30ff 0300 3d3d 465e bd77 4f5e ....0...==F^.wO^
> 0020 29ca 345e 0000 0000 0000 0100 0002 0000 ).4^............
> 0040 0000 0800 0100 0000 0af3 0100 0400 0000 ................
> 0060 0000 0000 0000 0000 4000 0000 c088 3800 [email protected].
> 0100 0000 0000 0000 0000 0000 0000 0000 0000 ................
> *
> 0140 0000 0000 5fc4 cfb4 0000 0000 0000 0000 ...._...........
> 0160 0000 0000 0000 0000 0000 0000 af23 0000 .............#..
> 0200 2000 1cc3 ac95 c9c8 a4d2 9883 583e addf ...........X>..
> 0220 3de0 485e b04d 7151 0000 0000 0000 0000 =.H^.MqQ........
> 0240 0000 0000 0000 0000 0000 0000 0000 0000 ................
> *
>
> and "stat" output:
> debugfs: stat <917527>
> Inode: 917527 Type: regular Mode: 0644 Flags: 0x80000
> Generation: 172755908 Version: 0x00000000:00000001
> User: 0 Group: 0 Project: 0 Size: 261936
> File ACL: 0
> Links: 1 Blockcount: 512
> Fragment: Address: 0 Number: 0 Size: 0
> ctime: 0x5e4f77bd:c9bdb844 -- Fri Feb 21 06:25:01 2020
> atime: 0x5e478ebd:92dc61c8 -- Sat Feb 15 06:25:01 2020
> mtime: 0x5e34ca29:8398d2a4 -- Sat Feb 1 00:45:29 2020
> crtime: 0x5e4a31bd:0c26c5ec -- Mon Feb 17 06:25:01 2020
> Size of extra inode fields: 32
> Inode checksum: 0xf2958438
> EXTENTS:
> (0-63):3704704-3704767
> debugfs: stat <917524>
> Inode: 917524 Type: regular Mode: 0644 Flags: 0x80000
> Generation: 3033515103 Version: 0x00000000:00000001
> User: 0 Group: 0 Project: 0 Size: 261936
> File ACL: 0
> Links: 1 Blockcount: 512
> Fragment: Address: 0 Number: 0 Size: 0
> ctime: 0x5e4f77bd:c8c995ac -- Fri Feb 21 06:25:01 2020
> atime: 0x5e463d3d:dfad3e58 -- Fri Feb 14 06:25:01 2020
> mtime: 0x5e34ca29:8398d2a4 -- Sat Feb 1 00:45:29 2020
> crtime: 0x5e48e03d:51714db0 -- Sun Feb 16 06:25:01 2020
> Size of extra inode fields: 32
> Inode checksum: 0xc31c23af
> EXTENTS:
> (0-63):3705024-3705087
>
> When using sif (set_inode_info) to re-set the UID to 0 on this (so
> provoke the checksum to be updated):
>
> debugfs: id <917524>
> 0000 a481 0000 30ff 0300 3d3d 465e bd77 4f5e ....0...==F^.wO^
> 0020 29ca 345e 0000 0000 0000 0100 0002 0000 ).4^............
> 0040 0000 0800 0100 0000 0af3 0100 0400 0000 ................
> 0060 0000 0000 0000 0000 4000 0000 c088 3800 [email protected].
> 0100 0000 0000 0000 0000 0000 0000 0000 0000 ................
> *
> 0140 0000 0000 5fc4 cfb4 0000 0000 0000 0000 ...._...........
> 0160 0000 0000 0000 0000 0000 0000 b61f 0000 ................
> ^^^^
> 0200 2000 aa15 ac95 c9c8 a4d2 9883 583e addf ...........X>..
> ^^^^
> 0220 3de0 485e b04d 7151 0000 0000 0000 0000 =.H^.MqQ........
> 0240 0000 0000 0000 0000 0000 0000 0000 0000 ................
> *
>
> The values with "^^^^" are the checksum, which are the only values
> that have changed here - the checksum is now 0x15aa1fb6 rather than
> 0xc31c23af.
>
> With that changed, running e2fsck -n on the filesystem results in a
> pass:
>
> root@cex7:~# e2fsck -n /dev/nvme0n1p2
> e2fsck 1.44.5 (15-Dec-2018)
> Warning: skipping journal recovery because doing a read-only filesystem check.
> /dev/nvme0n1p2 contains a file system with errors, check forced.
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Pass 3: Checking directory connectivity
> Pass 4: Checking reference counts
> Pass 5: Checking group summary information
> /dev/nvme0n1p2: 121163/2097152 files (0.1% non-contiguous), 1349227/8388608 blocks
>
> and the file now appears to be intact (being a gzip file, gzip verifies
> that the contents are now as it expects.)
>
> So, it looks like the _only_ issue is that the checksum on the inode
> became invalid, which seems to suggest that it *isn't* a NVMe nor PCIe
> issue.
>
> I wonder whether the journal would contain anything useful, but I don't
> know how to use debugfs to find that out - while I can dump the journal,
> I'd need to know which block contains the inode, and then work out where
> in the journal that block was going to be written. If that would help,
> let me know ASAP as I'll hold off rebooting the platform for a while
> (which means the filesystem will remain as-is - and yes, I have the
> debugfs file for e2undo to put stuff back.) Maybe it's possible to pull
> the block number out of the e2undo file?
>
> tune2fs says:
>
> Checksum type: crc32c
> Checksum: 0x682f91b9
>
> I guess this is what is used to checksum the inodes? If so, it's using
> the kernel's crc32c-generic driver (according to /proc/crypto).
>
> Could it be a race condition, or some problem that's specific to the
> ARM64 kernel that's provoking this corruption?
Hi,
The corruption has returned this evening:
[25094.614718] EXT4-fs error (device nvme0n1p2): ext4_lookup:1707: inode #271688: comm mandb: iget: checksum invalid
[25094.623781] Aborting journal on device nvme0n1p2-8.
[25094.627419] EXT4-fs (nvme0n1p2): Remounting filesystem read-only
[25094.628206] EXT4-fs error (device nvme0n1p2):
ext4_journal_check_start:83: Detected aborted journal
root@cex7:[~]:<506> debugfs /dev/nvme0n1p2
debugfs 1.44.5 (15-Dec-2018)
debugfs: id <271688>
0000 a481 0000 f108 0000 2518 fd5d 2518 fd5d ........%..]%..]
0020 9f49 715c 0000 0000 0000 0100 0800 0000 .Iq\............
0040 0000 0800 0100 0000 0af3 0100 0400 0000 ................
0060 0000 0000 0000 0000 0100 0000 ed19 1100 ................
0100 0000 0000 0000 0000 0000 0000 0000 0000 ................
*
0140 0000 0000 b42f 4f06 0000 0000 0000 0000 ...../O.........
0160 0000 0000 0000 0000 0000 0000 c9cf 0000 ................
0200 2000 8d83 086d bebf 0000 0000 086d bebf ....m.......m..
0220 2518 fd5d 086d bebf 0000 0000 0000 0000 %..].m..........
0240 0000 0000 0000 0000 0000 0000 0000 0000 ................
*
debugfs: stat <271688>
Inode: 271688 Type: regular Mode: 0644 Flags: 0x80000
Generation: 105852852 Version: 0x00000000:00000001
User: 0 Group: 0 Project: 0 Size: 2289
File ACL: 0
Links: 1 Blockcount: 8
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x5dfd1825:bfbe6d08 -- Fri Dec 20 18:51:17 2019
atime: 0x5dfd1825:bfbe6d08 -- Fri Dec 20 18:51:17 2019
mtime: 0x5c71499f:00000000 -- Sat Feb 23 13:24:47 2019
crtime: 0x5dfd1825:bfbe6d08 -- Fri Dec 20 18:51:17 2019
Size of extra inode fields: 32
Inode checksum: 0x838dcfc9
EXTENTS:
(0):1120749
debugfs:
root@cex7:[~]:<509> e2fsck -n /dev/nvme0n1p2
e2fsck 1.44.5 (15-Dec-2018)
Warning: skipping journal recovery because doing a read-only filesystem check.
/dev/nvme0n1p2 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/nvme0n1p2: 147476/2097152 files (0.1% non-contiguous), 1542719/8388608 blocks
This time, the machine has not been powered down for a very long time,
although I've booted 5.7 (plus the additional patches including several
workarounds in the PCIe driver so my Mellanox card works) on it earlier
today. I did notice that debian decided to run a fsck on the filesystem
at reboot, which is a little weird as it's ext4, and found nothing wrong.
Hmm, I just tried:
root@cex7:[~]:<514> hdparm -f /dev/nvme0n1p2
root@cex7:[~]:<515> hdparm -f /dev/nvme0n1
root@cex7:[~]:<517> e2fsck -n /dev/nvme0n1p2
e2fsck 1.44.5 (15-Dec-2018)
Warning: skipping journal recovery because doing a read-only filesystem
check.
/dev/nvme0n1p2 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Entry 'mainlog.2.gz' in /var/log/exim4 (917613) has deleted/unused inode 922603. Clear? no
Entry 'mainlog.2.gz' in /var/log/exim4 (917613) has an incorrect filetype (was 1, should be 0).
Fix? no
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Unattached inode 920748
Connect to /lost+found? no
Pass 5: Checking group summary information
Block bitmap differences: +(9259--9280) -3703011 -3703044 -3703053 +3736187 -3827722 -3830272 +3906363 +3911697 +3911699 +3911701 +3911703 +3913228
Fix? no
Free blocks count wrong for group #113 (12615, counted=12606).
Fix? no
Free blocks count wrong (6845889, counted=6845880).
Fix? no
Inode bitmap differences: Group 112 inode bitmap does not match checksum.
IGNORED.
Block bitmap differences: Group 113 block bitmap does not match checksum.
IGNORED.
/dev/nvme0n1p2: ********** WARNING: Filesystem still has errors **********
/dev/nvme0n1p2: 147476/2097152 files (0.1% non-contiguous), 1542719/8388608 blocks
which looks less good, and is likely to be e2fsck reading off the media
rather than using what was in the kernel cache. However, still nothing
for the offending inode, who's raw data remains unchanged from what I've
quoted above from debugfs.
It /seems/ to be pointing at the data on the media changing, possibly
buggy firmware on the nvme (ADATA SX8200PNP) drive, maybe? Or maybe
undiscovered bugs in the Mobiveil PCIe hardware corrupting transfers
to the nvme?
The problem is, this is rather undebuggable as it happens so rarely. :(
I'm becoming very discouraged to touch nvme ever again by this, as this
is my first and only experience of that technology. I'm considering
getting some conventional SATA HDDs and junking nvme on the basis of
it being an unreliable technology.
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC for 0.8m (est. 1762m) line in suburbia: sync at 13.1Mbps down 424kbps up
On Sat, Jun 06, 2020 at 12:53:43AM +0100, Russell King - ARM Linux admin wrote:
> On Sat, Feb 29, 2020 at 11:04:56AM +0000, Russell King - ARM Linux admin wrote:
> > Adding Ted and Andreas...
> >
> > Here's the debugfs -n "id" output for dpkg.status.5.gz (which is fine,
> > and probably a similar size):
> >
> > debugfs: id <917527>
> > 0000 a481 0000 30ff 0300 bd8e 475e bd77 4f5e ....0.....G^.wO^
> > 0020 29ca 345e 0000 0000 0000 0100 0002 0000 ).4^............
> > 0040 0000 0800 0100 0000 0af3 0100 0400 0000 ................
> > 0060 0000 0000 0000 0000 4000 0000 8087 3800 [email protected].
> > 0100 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > *
> > 0140 0000 0000 c40b 4c0a 0000 0000 0000 0000 ......L.........
> > 0160 0000 0000 0000 0000 0000 0000 3884 0000 ............8...
> > 0200 2000 95f2 44b8 bdc9 a4d2 9883 c861 dc92 ...D........a..
> > 0220 bd31 4a5e ecc5 260c 0000 0000 0000 0000 .1J^..&.........
> > 0240 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > *
> >
> > and for the affected inode:
> > debugfs: id <917524>
> > 0000 a481 0000 30ff 0300 3d3d 465e bd77 4f5e ....0...==F^.wO^
> > 0020 29ca 345e 0000 0000 0000 0100 0002 0000 ).4^............
> > 0040 0000 0800 0100 0000 0af3 0100 0400 0000 ................
> > 0060 0000 0000 0000 0000 4000 0000 c088 3800 [email protected].
> > 0100 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > *
> > 0140 0000 0000 5fc4 cfb4 0000 0000 0000 0000 ...._...........
> > 0160 0000 0000 0000 0000 0000 0000 af23 0000 .............#..
> > 0200 2000 1cc3 ac95 c9c8 a4d2 9883 583e addf ...........X>..
> > 0220 3de0 485e b04d 7151 0000 0000 0000 0000 =.H^.MqQ........
> > 0240 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > *
> >
> > and "stat" output:
> > debugfs: stat <917527>
> > Inode: 917527 Type: regular Mode: 0644 Flags: 0x80000
> > Generation: 172755908 Version: 0x00000000:00000001
> > User: 0 Group: 0 Project: 0 Size: 261936
> > File ACL: 0
> > Links: 1 Blockcount: 512
> > Fragment: Address: 0 Number: 0 Size: 0
> > ctime: 0x5e4f77bd:c9bdb844 -- Fri Feb 21 06:25:01 2020
> > atime: 0x5e478ebd:92dc61c8 -- Sat Feb 15 06:25:01 2020
> > mtime: 0x5e34ca29:8398d2a4 -- Sat Feb 1 00:45:29 2020
> > crtime: 0x5e4a31bd:0c26c5ec -- Mon Feb 17 06:25:01 2020
> > Size of extra inode fields: 32
> > Inode checksum: 0xf2958438
> > EXTENTS:
> > (0-63):3704704-3704767
> > debugfs: stat <917524>
> > Inode: 917524 Type: regular Mode: 0644 Flags: 0x80000
> > Generation: 3033515103 Version: 0x00000000:00000001
> > User: 0 Group: 0 Project: 0 Size: 261936
> > File ACL: 0
> > Links: 1 Blockcount: 512
> > Fragment: Address: 0 Number: 0 Size: 0
> > ctime: 0x5e4f77bd:c8c995ac -- Fri Feb 21 06:25:01 2020
> > atime: 0x5e463d3d:dfad3e58 -- Fri Feb 14 06:25:01 2020
> > mtime: 0x5e34ca29:8398d2a4 -- Sat Feb 1 00:45:29 2020
> > crtime: 0x5e48e03d:51714db0 -- Sun Feb 16 06:25:01 2020
> > Size of extra inode fields: 32
> > Inode checksum: 0xc31c23af
> > EXTENTS:
> > (0-63):3705024-3705087
> >
> > When using sif (set_inode_info) to re-set the UID to 0 on this (so
> > provoke the checksum to be updated):
> >
> > debugfs: id <917524>
> > 0000 a481 0000 30ff 0300 3d3d 465e bd77 4f5e ....0...==F^.wO^
> > 0020 29ca 345e 0000 0000 0000 0100 0002 0000 ).4^............
> > 0040 0000 0800 0100 0000 0af3 0100 0400 0000 ................
> > 0060 0000 0000 0000 0000 4000 0000 c088 3800 [email protected].
> > 0100 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > *
> > 0140 0000 0000 5fc4 cfb4 0000 0000 0000 0000 ...._...........
> > 0160 0000 0000 0000 0000 0000 0000 b61f 0000 ................
> > ^^^^
> > 0200 2000 aa15 ac95 c9c8 a4d2 9883 583e addf ...........X>..
> > ^^^^
> > 0220 3de0 485e b04d 7151 0000 0000 0000 0000 =.H^.MqQ........
> > 0240 0000 0000 0000 0000 0000 0000 0000 0000 ................
> > *
> >
> > The values with "^^^^" are the checksum, which are the only values
> > that have changed here - the checksum is now 0x15aa1fb6 rather than
> > 0xc31c23af.
> >
> > With that changed, running e2fsck -n on the filesystem results in a
> > pass:
> >
> > root@cex7:~# e2fsck -n /dev/nvme0n1p2
> > e2fsck 1.44.5 (15-Dec-2018)
> > Warning: skipping journal recovery because doing a read-only filesystem check.
> > /dev/nvme0n1p2 contains a file system with errors, check forced.
> > Pass 1: Checking inodes, blocks, and sizes
> > Pass 2: Checking directory structure
> > Pass 3: Checking directory connectivity
> > Pass 4: Checking reference counts
> > Pass 5: Checking group summary information
> > /dev/nvme0n1p2: 121163/2097152 files (0.1% non-contiguous), 1349227/8388608 blocks
> >
> > and the file now appears to be intact (being a gzip file, gzip verifies
> > that the contents are now as it expects.)
> >
> > So, it looks like the _only_ issue is that the checksum on the inode
> > became invalid, which seems to suggest that it *isn't* a NVMe nor PCIe
> > issue.
> >
> > I wonder whether the journal would contain anything useful, but I don't
> > know how to use debugfs to find that out - while I can dump the journal,
> > I'd need to know which block contains the inode, and then work out where
> > in the journal that block was going to be written. If that would help,
> > let me know ASAP as I'll hold off rebooting the platform for a while
> > (which means the filesystem will remain as-is - and yes, I have the
> > debugfs file for e2undo to put stuff back.) Maybe it's possible to pull
> > the block number out of the e2undo file?
> >
> > tune2fs says:
> >
> > Checksum type: crc32c
> > Checksum: 0x682f91b9
> >
> > I guess this is what is used to checksum the inodes? If so, it's using
> > the kernel's crc32c-generic driver (according to /proc/crypto).
> >
> > Could it be a race condition, or some problem that's specific to the
> > ARM64 kernel that's provoking this corruption?
>
> Hi,
>
> The corruption has returned this evening:
>
> [25094.614718] EXT4-fs error (device nvme0n1p2): ext4_lookup:1707: inode #271688: comm mandb: iget: checksum invalid
> [25094.623781] Aborting journal on device nvme0n1p2-8.
> [25094.627419] EXT4-fs (nvme0n1p2): Remounting filesystem read-only
> [25094.628206] EXT4-fs error (device nvme0n1p2):
> ext4_journal_check_start:83: Detected aborted journal
> root@cex7:[~]:<506> debugfs /dev/nvme0n1p2
> debugfs 1.44.5 (15-Dec-2018)
> debugfs: id <271688>
> 0000 a481 0000 f108 0000 2518 fd5d 2518 fd5d ........%..]%..]
> 0020 9f49 715c 0000 0000 0000 0100 0800 0000 .Iq\............
> 0040 0000 0800 0100 0000 0af3 0100 0400 0000 ................
> 0060 0000 0000 0000 0000 0100 0000 ed19 1100 ................
> 0100 0000 0000 0000 0000 0000 0000 0000 0000 ................
> *
> 0140 0000 0000 b42f 4f06 0000 0000 0000 0000 ...../O.........
> 0160 0000 0000 0000 0000 0000 0000 c9cf 0000 ................
> 0200 2000 8d83 086d bebf 0000 0000 086d bebf ....m.......m..
> 0220 2518 fd5d 086d bebf 0000 0000 0000 0000 %..].m..........
> 0240 0000 0000 0000 0000 0000 0000 0000 0000 ................
> *
>
> debugfs: stat <271688>
> Inode: 271688 Type: regular Mode: 0644 Flags: 0x80000
> Generation: 105852852 Version: 0x00000000:00000001
> User: 0 Group: 0 Project: 0 Size: 2289
> File ACL: 0
> Links: 1 Blockcount: 8
> Fragment: Address: 0 Number: 0 Size: 0
> ctime: 0x5dfd1825:bfbe6d08 -- Fri Dec 20 18:51:17 2019
> atime: 0x5dfd1825:bfbe6d08 -- Fri Dec 20 18:51:17 2019
> mtime: 0x5c71499f:00000000 -- Sat Feb 23 13:24:47 2019
> crtime: 0x5dfd1825:bfbe6d08 -- Fri Dec 20 18:51:17 2019
> Size of extra inode fields: 32
> Inode checksum: 0x838dcfc9
> EXTENTS:
> (0):1120749
> debugfs:
> root@cex7:[~]:<509> e2fsck -n /dev/nvme0n1p2
> e2fsck 1.44.5 (15-Dec-2018)
> Warning: skipping journal recovery because doing a read-only filesystem check.
> /dev/nvme0n1p2 contains a file system with errors, check forced.
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Pass 3: Checking directory connectivity
> Pass 4: Checking reference counts
> Pass 5: Checking group summary information
> /dev/nvme0n1p2: 147476/2097152 files (0.1% non-contiguous), 1542719/8388608 blocks
>
> This time, the machine has not been powered down for a very long time,
> although I've booted 5.7 (plus the additional patches including several
> workarounds in the PCIe driver so my Mellanox card works) on it earlier
> today. I did notice that debian decided to run a fsck on the filesystem
> at reboot, which is a little weird as it's ext4, and found nothing wrong.
>
> Hmm, I just tried:
>
> root@cex7:[~]:<514> hdparm -f /dev/nvme0n1p2
> root@cex7:[~]:<515> hdparm -f /dev/nvme0n1
> root@cex7:[~]:<517> e2fsck -n /dev/nvme0n1p2
> e2fsck 1.44.5 (15-Dec-2018)
> Warning: skipping journal recovery because doing a read-only filesystem
> check.
> /dev/nvme0n1p2 contains a file system with errors, check forced.
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Entry 'mainlog.2.gz' in /var/log/exim4 (917613) has deleted/unused inode 922603. Clear? no
>
> Entry 'mainlog.2.gz' in /var/log/exim4 (917613) has an incorrect filetype (was 1, should be 0).
> Fix? no
>
> Pass 3: Checking directory connectivity
> Pass 4: Checking reference counts
> Unattached inode 920748
> Connect to /lost+found? no
>
> Pass 5: Checking group summary information
> Block bitmap differences: +(9259--9280) -3703011 -3703044 -3703053 +3736187 -3827722 -3830272 +3906363 +3911697 +3911699 +3911701 +3911703 +3913228
> Fix? no
>
> Free blocks count wrong for group #113 (12615, counted=12606).
> Fix? no
>
> Free blocks count wrong (6845889, counted=6845880).
> Fix? no
>
> Inode bitmap differences: Group 112 inode bitmap does not match checksum.
> IGNORED.
> Block bitmap differences: Group 113 block bitmap does not match checksum.
> IGNORED.
>
> /dev/nvme0n1p2: ********** WARNING: Filesystem still has errors **********
>
> /dev/nvme0n1p2: 147476/2097152 files (0.1% non-contiguous), 1542719/8388608 blocks
>
> which looks less good, and is likely to be e2fsck reading off the media
> rather than using what was in the kernel cache. However, still nothing
> for the offending inode, who's raw data remains unchanged from what I've
> quoted above from debugfs.
>
> It /seems/ to be pointing at the data on the media changing, possibly
> buggy firmware on the nvme (ADATA SX8200PNP) drive, maybe? Or maybe
> undiscovered bugs in the Mobiveil PCIe hardware corrupting transfers
> to the nvme?
>
> The problem is, this is rather undebuggable as it happens so rarely. :(
>
> I'm becoming very discouraged to touch nvme ever again by this, as this
> is my first and only experience of that technology. I'm considering
> getting some conventional SATA HDDs and junking nvme on the basis of
> it being an unreliable technology.
Okay, now I'm confused. I haven't rebooted the platform (I was just
about to) but because of the issues I've had in the past with the
filesystem not being mountable, I thought I ought to run e2fsck on
the now read-only root filesystem before rebooting to ensure that it
is consistent.
root@cex7:[~]:<587> e2fsck -n /dev/nvme0n1p2
e2fsck 1.44.5 (15-Dec-2018)
/dev/nvme0n1p2 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (6845930, counted=6845886).
Fix? no
Free inodes count wrong (1949681, counted=1949673).
Fix? no
/dev/nvme0n1p2: 147471/2097152 files (0.1% non-contiguous),
1542678/8388608 blocks
but but but, the filesystem is still mounted read-only, so how can it
have changed?
--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC for 0.8m (est. 1762m) line in suburbia: sync at 13.1Mbps down 424kbps up