From: Lad Prabhakar <[email protected]>
Hi All,
RISC-V non-coherent function pointer based cache management operations
=======================================================================
As based on the previous discussion [0]. [1], It was suggested upon the below:
* Move out cache drivers from drivers/soc/folder
* Implement function pointers for CMO instead of ALTERNATIVE_x() macro
* Add cache drivers to drivers/cache folder
[0] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/linux-riscv/Y62nOqzyuUKqYDpq@spud/#R
This v6 version of the patch series add support to use function pointers for CMO
and switches the current CMO implementations for zicbom and T-HEAD to use function
pointers.
Note, with the above approach we might see some performance degradation due to additional
checks.
----------x---------------------x--------------------x---------------x--------------
non-coherent DMA support for AX45MP
====================================
On the Andes AX45MP core, cache coherency is a specification option so it
may not be supported. In this case DMA will fail. To get around with this
issue this patch series does the below:
1] Andes alternative ports is implemented as errata which checks if the IOCP
is missing and only then applies to CMO errata. One vendor specific SBI EXT
(RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND) is implemented as part of errata. If we could
access the DTB in errata I can get rid of this EXT ID from OpenSBI. Is there any
approach we can access the DTB in patch callback?
Below are the configs which Andes port provides (and are selected by RZ/Five):
- ERRATA_ANDES
- ERRATA_ANDES_CMO
2] Andes AX45MP core has a Programmable Physical Memory Attributes (PMA)
block that allows dynamic adjustment of memory attributes in the runtime.
It contains a configurable amount of PMA entries implemented as CSR
registers to control the attributes of memory locations in interest.
OpenSBI configures the PMA regions as required and creates a reserve memory
node and propagates it to the higher boot stack.
An RFC patch for OpenSBI is posted here:
https://patchwork.ozlabs.org/project/opensbi/patch/[email protected]/
reserved-memory {
#address-cells = <2>;
#size-cells = <2>;
ranges;
pma_resv0@58000000 {
compatible = "shared-dma-pool";
reg = <0x0 0x58000000 0x0 0x08000000>;
no-map;
linux,dma-default;
};
};
The above shared DMA pool gets appended to Linux DTB so the DMA memory
requests go through this region.
3] We provide callbacks to synchronize specific content between memory and
cache.
- arch_sync_dma_for_device()
- arch_sync_dma_for_cpu()
4] RZ/Five SoC selects the below configs
- AX45MP_L2_CACHE
- DMA_GLOBAL_POOL
- ERRATA_ANDES
- ERRATA_ANDES_CMO
OpenSBI implementation patches can be found here:
1] https://patchwork.ozlabs.org/project/opensbi/cover/[email protected]/
2]https://patchwork.ozlabs.org/project/opensbi/patch/[email protected]/
----------x---------------------x--------------------x---------------x--------------
Note,
- This series requires testing on Cores with zicbom and T-Head SoCs
- Ive used GCC 12.2.0 for compilation (tested with allmodconfig)
- Tested all the IP blocks on RZ/Five which use DMA
v5.1 -> v6
* Dropped use of ALTERNATIVE_x() macro
* Now switched to used function pointers for CMO
* Moved driver to drivers/cache folder
v5 -> v5.1
* https://patchwork.kernel.org/project/linux-riscv/list/?series=708610&state=%2A&archive=both
v4 -> v5
* Rebased ALTERNATIVE_3() macro on top of Andrew's patches
* Rebased the changes on top of Heiko's alternative call patches
* Dropped configuring the PMA from Linux
* Dropped configuring the L2 cache from Linux and dropped the binding for same
* Now using runtime patching mechanism instead of compile time config
RFC v3 -> v4
* Implemented ALTERNATIVE_3() macro
* Now using runtime patching mechanism instead of compile time config
* Added Andes CMO as and errata
* Fixed comments pointed by Geert
RFC v2-> RFC v3
* Fixed review comments pointed by Conor
* Move DT binding into cache folder
* Fixed DT binding check issue
* Added andestech,ax45mp-cache.h header file
* Now passing the flags for the PMA setup as part of andestech,pma-regions
property.
* Added andestech,inst/data-prefetch and andestech,tag/data-ram-ctl
properties to configure the L2 cache.
* Registered the cache driver as platform driver
RFC v1-> RFC v2
* Moved out the code from arc/riscv to drivers/soc/renesas
* Now handling the PMA setup as part of the L2 cache
* Now making use of dma-noncoherent.c instead SoC specific implementation.
* Dropped arch_dma_alloc() and arch_dma_free()
* Switched to RISCV_DMA_NONCOHERENT
* Included DT binding doc
RFC v2: https://patchwork.kernel.org/project/linux-renesas-soc/cover/[email protected]/
RFC v1: https://patchwork.kernel.org/project/linux-renesas-soc/cover/[email protected]/
Cheers,
Prabhakar
Lad Prabhakar (6):
riscv: mm: dma-noncoherent: Switch using function pointers for cache
management
riscv: asm: vendorid_list: Add Andes Technology to the vendors list
riscv: errata: Add Andes alternative ports
dt-bindings: cache: r9a07g043f-l2-cache: Add DT binding documentation
for L2 cache controller
cache: Add L2 cache management for Andes AX45MP RISC-V core
soc: renesas: Kconfig: Select the required configs for RZ/Five SoC
.../cache/andestech,ax45mp-cache.yaml | 81 +++++
arch/riscv/Kconfig.erratas | 22 ++
arch/riscv/errata/Makefile | 1 +
arch/riscv/errata/andes/Makefile | 1 +
arch/riscv/errata/andes/errata.c | 71 +++++
arch/riscv/errata/thead/errata.c | 71 +++++
arch/riscv/include/asm/alternative.h | 3 +
arch/riscv/include/asm/dma-noncoherent.h | 83 ++++++
arch/riscv/include/asm/errata_list.h | 53 ----
arch/riscv/include/asm/vendorid_list.h | 1 +
arch/riscv/kernel/alternative.c | 5 +
arch/riscv/kernel/cpufeature.c | 2 +
arch/riscv/mm/dma-noncoherent.c | 94 +++++-
arch/riscv/mm/pmem.c | 18 +-
drivers/Kconfig | 2 +
drivers/Makefile | 1 +
drivers/cache/Kconfig | 10 +
drivers/cache/Makefile | 3 +
drivers/cache/ax45mp_cache.c | 279 ++++++++++++++++++
drivers/soc/renesas/Kconfig | 4 +
20 files changed, 744 insertions(+), 61 deletions(-)
create mode 100644 Documentation/devicetree/bindings/cache/andestech,ax45mp-cache.yaml
create mode 100644 arch/riscv/errata/andes/Makefile
create mode 100644 arch/riscv/errata/andes/errata.c
create mode 100644 arch/riscv/include/asm/dma-noncoherent.h
create mode 100644 drivers/cache/Kconfig
create mode 100644 drivers/cache/Makefile
create mode 100644 drivers/cache/ax45mp_cache.c
--
2.25.1
From: Lad Prabhakar <[email protected]>
I/O Coherence Port (IOCP) provides an AXI interface for connecting
external non-caching masters, such as DMA controllers. The accesses
from IOCP are coherent with D-Caches and L2 Cache.
IOCP is a specification option and is disabled on the Renesas RZ/Five
SoC due to this reason IP blocks using DMA will fail.
The Andes AX45MP core has a Programmable Physical Memory Attributes (PMA)
block that allows dynamic adjustment of memory attributes in the runtime.
It contains a configurable amount of PMA entries implemented as CSR
registers to control the attributes of memory locations in interest.
Below are the memory attributes supported:
* Device, Non-bufferable
* Device, bufferable
* Memory, Non-cacheable, Non-bufferable
* Memory, Non-cacheable, Bufferable
* Memory, Write-back, No-allocate
* Memory, Write-back, Read-allocate
* Memory, Write-back, Write-allocate
* Memory, Write-back, Read and Write-allocate
More info about PMA (section 10.3):
Link: http://www.andestech.com/wp-content/uploads/AX45MP-1C-Rev.-5.0.0-Datasheet.pdf
As a workaround for SoCs with IOCP disabled CMO needs to be handled by
software. Firstly OpenSBI configures the memory region as
"Memory, Non-cacheable, Bufferable" and passes this region as a global
shared dma pool as a DT node. With DMA_GLOBAL_POOL enabled all DMA
allocations happen from this region and synchronization callbacks are
implemented to synchronize when doing DMA transactions.
Example PMA region passes as a DT node from OpenSBI:
reserved-memory {
#address-cells = <2>;
#size-cells = <2>;
ranges;
pma_resv0@58000000 {
compatible = "shared-dma-pool";
reg = <0x0 0x58000000 0x0 0x08000000>;
no-map;
linux,dma-default;
};
};
Signed-off-by: Lad Prabhakar <[email protected]>
---
v5 -> v6
* Moved driver to cache folder
* Switched to new API for CMO
v4 -> v5
* Dropped code for configuring L2 cache
* Dropped code for configuring PMA
* Updated commit message
* Added comments
* Changed static branch enable/disable order
RFC v3 -> v4
* Made use of runtime patching instead of compile time
* Now just exposing single function ax45mp_no_iocp_cmo() for CMO handling
* Added a check to make sure cache line size is always 64 bytes
* Renamed folder rzf -> rzfive
* Improved Kconfig description
* Dropped L2 cache configuration
* Dropped unnecessary casts
* Fixed comments pointed by Geert.
---
drivers/Kconfig | 2 +
drivers/Makefile | 1 +
drivers/cache/Kconfig | 10 ++
drivers/cache/Makefile | 3 +
drivers/cache/ax45mp_cache.c | 279 +++++++++++++++++++++++++++++++++++
5 files changed, 295 insertions(+)
create mode 100644 drivers/cache/Kconfig
create mode 100644 drivers/cache/Makefile
create mode 100644 drivers/cache/ax45mp_cache.c
diff --git a/drivers/Kconfig b/drivers/Kconfig
index 968bd0a6fd78..44abd2cba3a3 100644
--- a/drivers/Kconfig
+++ b/drivers/Kconfig
@@ -15,6 +15,8 @@ source "drivers/base/Kconfig"
source "drivers/bus/Kconfig"
+source "drivers/cache/Kconfig"
+
source "drivers/connector/Kconfig"
source "drivers/firmware/Kconfig"
diff --git a/drivers/Makefile b/drivers/Makefile
index bdf1c66141c9..9fc91d2bbe4c 100644
--- a/drivers/Makefile
+++ b/drivers/Makefile
@@ -6,6 +6,7 @@
# Rewritten to use lists instead of if-statements.
#
+obj-y += cache/
obj-y += irqchip/
obj-y += bus/
diff --git a/drivers/cache/Kconfig b/drivers/cache/Kconfig
new file mode 100644
index 000000000000..5478adff3d88
--- /dev/null
+++ b/drivers/cache/Kconfig
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: GPL-2.0
+menu "Cache Drivers"
+
+config AX45MP_L2_CACHE
+ bool "Andes Technology AX45MP L2 Cache controller"
+ depends on RISCV && RISCV_DMA_NONCOHERENT
+ help
+ Support for the L2 cache controller on Andes Technology AX45MP platforms.
+
+endmenu
diff --git a/drivers/cache/Makefile b/drivers/cache/Makefile
new file mode 100644
index 000000000000..2012e7fb978d
--- /dev/null
+++ b/drivers/cache/Makefile
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_AX45MP_L2_CACHE) += ax45mp_cache.o
diff --git a/drivers/cache/ax45mp_cache.c b/drivers/cache/ax45mp_cache.c
new file mode 100644
index 000000000000..556e6875627c
--- /dev/null
+++ b/drivers/cache/ax45mp_cache.c
@@ -0,0 +1,279 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * non-coherent cache functions for Andes AX45MP
+ *
+ * Copyright (C) 2022 Renesas Electronics Corp.
+ */
+
+#include <asm/dma-noncoherent.h>
+#include <linux/cacheflush.h>
+#include <linux/cacheinfo.h>
+#include <linux/dma-direction.h>
+#include <linux/of_address.h>
+#include <linux/of_platform.h>
+
+#include <asm/cacheflush.h>
+#include <asm/sbi.h>
+
+/* L2 cache registers */
+#define AX45MP_L2C_REG_CTL_OFFSET 0x8
+
+#define AX45MP_L2C_REG_C0_CMD_OFFSET 0x40
+#define AX45MP_L2C_REG_C0_ACC_OFFSET 0x48
+#define AX45MP_L2C_REG_STATUS_OFFSET 0x80
+
+/* D-cache operation */
+#define AX45MP_CCTL_L1D_VA_INVAL 0
+#define AX45MP_CCTL_L1D_VA_WB 1
+
+/* L2 CCTL status */
+#define AX45MP_CCTL_L2_STATUS_IDLE 0
+
+/* L2 CCTL status cores mask */
+#define AX45MP_CCTL_L2_STATUS_C0_MASK 0xf
+
+/* L2 cache operation */
+#define AX45MP_CCTL_L2_PA_INVAL 0x8
+#define AX45MP_CCTL_L2_PA_WB 0x9
+
+#define AX45MP_L2C_REG_PER_CORE_OFFSET 0x10
+#define AX45MP_CCTL_L2_STATUS_PER_CORE_OFFSET 4
+
+#define AX45MP_L2C_REG_CN_CMD_OFFSET(n) \
+ (AX45MP_L2C_REG_C0_CMD_OFFSET + ((n) * AX45MP_L2C_REG_PER_CORE_OFFSET))
+#define AX45MP_L2C_REG_CN_ACC_OFFSET(n) \
+ (AX45MP_L2C_REG_C0_ACC_OFFSET + ((n) * AX45MP_L2C_REG_PER_CORE_OFFSET))
+#define AX45MP_CCTL_L2_STATUS_CN_MASK(n) \
+ (AX45MP_CCTL_L2_STATUS_C0_MASK << ((n) * AX45MP_CCTL_L2_STATUS_PER_CORE_OFFSET))
+
+#define AX45MP_CCTL_REG_UCCTLBEGINADDR_NUM 0x80b
+#define AX45MP_CCTL_REG_UCCTLCOMMAND_NUM 0x80c
+
+#define AX45MP_CACHE_LINE_SIZE 64
+
+struct ax45mp_priv {
+ void __iomem *l2c_base;
+ u32 ax45mp_cache_line_size;
+};
+
+static struct ax45mp_priv *ax45mp_priv;
+
+/* L2 Cache operations */
+static inline uint32_t ax45mp_cpu_l2c_get_cctl_status(void)
+{
+ return readl(ax45mp_priv->l2c_base + AX45MP_L2C_REG_STATUS_OFFSET);
+}
+
+/*
+ * Software trigger CCTL operation (cache maintenance operations) by writing
+ * to ucctlcommand and ucctlbeginaddr registers and write-back an L2 cache
+ * entry.
+ */
+static void ax45mp_cpu_dcache_wb_range(void *start, void *end, int line_size)
+{
+ void __iomem *base = ax45mp_priv->l2c_base;
+ int mhartid = smp_processor_id();
+ unsigned long pa;
+
+ while (end > start) {
+ csr_write(AX45MP_CCTL_REG_UCCTLBEGINADDR_NUM, start);
+ csr_write(AX45MP_CCTL_REG_UCCTLCOMMAND_NUM, AX45MP_CCTL_L1D_VA_WB);
+
+ pa = virt_to_phys(start);
+ writel(pa, base + AX45MP_L2C_REG_CN_ACC_OFFSET(mhartid));
+ writel(AX45MP_CCTL_L2_PA_WB,
+ base + AX45MP_L2C_REG_CN_CMD_OFFSET(mhartid));
+ while ((ax45mp_cpu_l2c_get_cctl_status() &
+ AX45MP_CCTL_L2_STATUS_CN_MASK(mhartid)) !=
+ AX45MP_CCTL_L2_STATUS_IDLE)
+ ;
+
+ start += line_size;
+ }
+}
+
+/*
+ * Software trigger CCTL operation by writing to ucctlcommand and ucctlbeginaddr
+ * registers and invalidate the L2 cache entry.
+ */
+static void ax45mp_cpu_dcache_inval_range(void *start, void *end, int line_size)
+{
+ void __iomem *base = ax45mp_priv->l2c_base;
+ int mhartid = smp_processor_id();
+ unsigned long pa;
+
+ while (end > start) {
+ csr_write(AX45MP_CCTL_REG_UCCTLBEGINADDR_NUM, start);
+ csr_write(AX45MP_CCTL_REG_UCCTLCOMMAND_NUM, AX45MP_CCTL_L1D_VA_INVAL);
+
+ pa = virt_to_phys(start);
+ writel(pa, base + AX45MP_L2C_REG_CN_ACC_OFFSET(mhartid));
+ writel(AX45MP_CCTL_L2_PA_INVAL,
+ base + AX45MP_L2C_REG_CN_CMD_OFFSET(mhartid));
+ while ((ax45mp_cpu_l2c_get_cctl_status() &
+ AX45MP_CCTL_L2_STATUS_CN_MASK(mhartid)) !=
+ AX45MP_CCTL_L2_STATUS_IDLE)
+ ;
+
+ start += line_size;
+ }
+}
+
+static void ax45mp_cpu_dma_inval_range(void *vaddr, size_t size)
+{
+ char cache_buf[2][AX45MP_CACHE_LINE_SIZE];
+ unsigned long start = (unsigned long)vaddr;
+ unsigned long end = start + size;
+ unsigned long old_start = start;
+ unsigned long old_end = end;
+ unsigned long line_size;
+ unsigned long flags;
+
+ if (unlikely(start == end))
+ return;
+
+ line_size = ax45mp_priv->ax45mp_cache_line_size;
+
+ memset(&cache_buf, 0x0, sizeof(cache_buf));
+ start = start & (~(line_size - 1));
+ end = ((end + line_size - 1) & (~(line_size - 1)));
+
+ local_irq_save(flags);
+ if (unlikely(start != old_start))
+ memcpy(&cache_buf[0][0], (void *)start, line_size);
+
+ if (unlikely(end != old_end))
+ memcpy(&cache_buf[1][0], (void *)(old_end & (~(line_size - 1))), line_size);
+
+ ax45mp_cpu_dcache_inval_range(vaddr, (void *)end, line_size);
+
+ if (unlikely(start != old_start))
+ memcpy((void *)start, &cache_buf[0][0], (old_start & (line_size - 1)));
+
+ local_irq_restore(flags);
+}
+
+static void ax45mp_cpu_dma_wb_range(void *vaddr, size_t size)
+{
+ unsigned long start = (unsigned long)vaddr;
+ unsigned long end = start + size;
+ unsigned long line_size;
+ unsigned long flags;
+
+ line_size = ax45mp_priv->ax45mp_cache_line_size;
+ local_irq_save(flags);
+ start = start & (~(line_size - 1));
+ ax45mp_cpu_dcache_wb_range(vaddr, (void *)end, line_size);
+ local_irq_restore(flags);
+}
+
+static void ax45mp_no_iocp_cmo(void *vaddr, size_t size,
+ enum dma_data_direction dir,
+ enum dma_noncoherent_ops ops)
+{
+ /* We have nothing to do in case of NON_COHERENT_DMA_PREP */
+ if (ops != NON_COHERENT_SYNC_DMA_FOR_DEVICE &&
+ ops != NON_COHERENT_SYNC_DMA_FOR_CPU)
+ return;
+
+ /*
+ * In case of DMA_FROM_DEVICE invalidate the L2 cache entries and
+ * in case of DMA_TO_DEVICE and DMA_BIDIRECTIONAL write-back an L2
+ * cache entries.
+ */
+ if (ops == NON_COHERENT_SYNC_DMA_FOR_DEVICE) {
+ switch (dir) {
+ case DMA_FROM_DEVICE:
+ ax45mp_cpu_dma_inval_range(vaddr, size);
+ break;
+ case DMA_TO_DEVICE:
+ case DMA_BIDIRECTIONAL:
+ ax45mp_cpu_dma_wb_range(vaddr, size);
+ break;
+ default:
+ break;
+ }
+
+ return;
+ }
+
+ /*
+ * ops == NON_COHERENT_SYNC_DMA_FOR_CPU
+ *
+ * in case of DMA_BIDIRECTIONAL and DMA_FROM_DEVICE invalidate the L2
+ * cache entries.
+ */
+ if (dir == DMA_BIDIRECTIONAL || dir == DMA_FROM_DEVICE)
+ ax45mp_cpu_dma_inval_range(vaddr, size);
+}
+
+static void ax45mp_get_l2_line_size(struct platform_device *pdev)
+{
+ struct device_node *np = pdev->dev.of_node;
+ struct device *dev = &pdev->dev;
+ int ret;
+
+ ret = of_property_read_u32(np, "cache-line-size", &ax45mp_priv->ax45mp_cache_line_size);
+ if (ret) {
+ dev_err(dev, "Failed to get cache-line-size, defaulting to 64 bytes\n");
+ ax45mp_priv->ax45mp_cache_line_size = AX45MP_CACHE_LINE_SIZE;
+ }
+
+ if (ax45mp_priv->ax45mp_cache_line_size != AX45MP_CACHE_LINE_SIZE) {
+ dev_err(dev, "Expected cache-line-size to be 64 bytes (found:%u). Defaulting to 64 bytes\n",
+ ax45mp_priv->ax45mp_cache_line_size);
+ ax45mp_priv->ax45mp_cache_line_size = AX45MP_CACHE_LINE_SIZE;
+ }
+}
+
+static int ax45mp_l2c_probe(struct platform_device *pdev)
+{
+ struct riscv_cache_ops ax45mp_cmo_ops;
+
+ /*
+ * riscv_cbom_block_size is set very much earlier so we can
+ * definitely rely on it and only if its being set we continue
+ * further in the probe path.
+ */
+ if (!riscv_cbom_block_size)
+ return 0;
+
+ ax45mp_priv = devm_kzalloc(&pdev->dev, sizeof(*ax45mp_priv), GFP_KERNEL);
+ if (!ax45mp_priv)
+ return -ENOMEM;
+
+ ax45mp_priv->l2c_base = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(ax45mp_priv->l2c_base))
+ return PTR_ERR(ax45mp_priv->l2c_base);
+
+ ax45mp_get_l2_line_size(pdev);
+
+ memset(&ax45mp_cmo_ops, 0x0, sizeof(ax45mp_cmo_ops));
+ ax45mp_cmo_ops.riscv_dma_noncoherent_cmo_ops = &ax45mp_no_iocp_cmo;
+ riscv_noncoherent_register_cache_ops(&ax45mp_cmo_ops);
+
+ return 0;
+}
+
+static const struct of_device_id ax45mp_cache_ids[] = {
+ { .compatible = "andestech,ax45mp-cache" },
+ { /* sentinel */ }
+};
+
+static struct platform_driver ax45mp_l2c_driver = {
+ .driver = {
+ .name = "ax45mp-l2c",
+ .of_match_table = ax45mp_cache_ids,
+ },
+ .probe = ax45mp_l2c_probe,
+};
+
+static int __init ax45mp_cache_init(void)
+{
+ return platform_driver_register(&ax45mp_l2c_driver);
+}
+arch_initcall(ax45mp_cache_init);
+
+MODULE_AUTHOR("Lad Prabhakar <[email protected]>");
+MODULE_DESCRIPTION("Andes AX45MP L2 cache driver");
+MODULE_LICENSE("GPL");
--
2.25.1
From: Lad Prabhakar <[email protected]>
Add Andes Technology to the vendors list.
Signed-off-by: Lad Prabhakar <[email protected]>
Reviewed-by: Heiko Stuebner <[email protected]>
Reviewed-by: Conor Dooley <[email protected]>
---
v5 -> v6
* No change
v4 -> v5
* Included RB tags
RFC v3 -> v4
* New patch
---
arch/riscv/include/asm/vendorid_list.h | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/riscv/include/asm/vendorid_list.h b/arch/riscv/include/asm/vendorid_list.h
index cb89af3f0704..e55407ace0c3 100644
--- a/arch/riscv/include/asm/vendorid_list.h
+++ b/arch/riscv/include/asm/vendorid_list.h
@@ -5,6 +5,7 @@
#ifndef ASM_VENDOR_LIST_H
#define ASM_VENDOR_LIST_H
+#define ANDESTECH_VENDOR_ID 0x31e
#define SIFIVE_VENDOR_ID 0x489
#define THEAD_VENDOR_ID 0x5b7
--
2.25.1
From: Lad Prabhakar <[email protected]>
Add required ports of the Alternative scheme for Andes CPU cores.
I/O Coherence Port (IOCP) provides an AXI interface for connecting external
non-caching masters, such as DMA controllers. IOCP is a specification
option and is disabled on the Renesas RZ/Five SoC due to this reason cache
management needs a software workaround.
Signed-off-by: Lad Prabhakar <[email protected]>
---
v5 -> v6
* Dropped patching alternative and now just probing IOCP
v4 -> v5
* Sorted the Kconfig/Makefile/Switch based on Core name
* Added a comments
* Introduced RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND SBI EXT ID to check if
CMO needs to be applied. Is there a way we can access the DTB while patching
as we can drop this SBI EXT ID and add a DT property instead for cmo?
RFC v3 -> v4
* New patch
---
arch/riscv/Kconfig.erratas | 22 +++++++++
arch/riscv/errata/Makefile | 1 +
arch/riscv/errata/andes/Makefile | 1 +
arch/riscv/errata/andes/errata.c | 71 ++++++++++++++++++++++++++++
arch/riscv/include/asm/alternative.h | 3 ++
arch/riscv/kernel/alternative.c | 5 ++
6 files changed, 103 insertions(+)
create mode 100644 arch/riscv/errata/andes/Makefile
create mode 100644 arch/riscv/errata/andes/errata.c
diff --git a/arch/riscv/Kconfig.erratas b/arch/riscv/Kconfig.erratas
index 69621ae6d647..f0f0c1abd52b 100644
--- a/arch/riscv/Kconfig.erratas
+++ b/arch/riscv/Kconfig.erratas
@@ -1,5 +1,27 @@
menu "CPU errata selection"
+config ERRATA_ANDES
+ bool "Andes AX45MP errata"
+ depends on !XIP_KERNEL
+ select RISCV_ALTERNATIVE
+ help
+ All Andes errata Kconfig depend on this Kconfig. Disabling
+ this Kconfig will disable all Andes errata. Please say "Y"
+ here if your platform uses Andes CPU cores.
+
+ Otherwise, please say "N" here to avoid unnecessary overhead.
+
+config ERRATA_ANDES_CMO
+ bool "Apply Andes cache management errata"
+ depends on ERRATA_ANDES && MMU && ARCH_R9A07G043
+ select RISCV_DMA_NONCOHERENT
+ default y
+ help
+ This will apply the cache management errata to handle the
+ non-standard handling on non-coherent operations on Andes cores.
+
+ If you don't know what to do here, say "Y".
+
config ERRATA_SIFIVE
bool "SiFive errata"
depends on !XIP_KERNEL
diff --git a/arch/riscv/errata/Makefile b/arch/riscv/errata/Makefile
index a1055965fbee..6f1c693af92d 100644
--- a/arch/riscv/errata/Makefile
+++ b/arch/riscv/errata/Makefile
@@ -1,2 +1,3 @@
+obj-$(CONFIG_ERRATA_ANDES) += andes/
obj-$(CONFIG_ERRATA_SIFIVE) += sifive/
obj-$(CONFIG_ERRATA_THEAD) += thead/
diff --git a/arch/riscv/errata/andes/Makefile b/arch/riscv/errata/andes/Makefile
new file mode 100644
index 000000000000..2d644e19caef
--- /dev/null
+++ b/arch/riscv/errata/andes/Makefile
@@ -0,0 +1 @@
+obj-y += errata.o
diff --git a/arch/riscv/errata/andes/errata.c b/arch/riscv/errata/andes/errata.c
new file mode 100644
index 000000000000..d893c0aabd23
--- /dev/null
+++ b/arch/riscv/errata/andes/errata.c
@@ -0,0 +1,71 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Erratas to be applied for Andes CPU cores
+ *
+ * Copyright (C) 2022 Renesas Electronics Corporation.
+ *
+ * Author: Lad Prabhakar <[email protected]>
+ */
+
+#include <asm/cacheflush.h>
+#include <asm/dma-noncoherent.h>
+#include <asm/errata_list.h>
+#include <asm/patch.h>
+#include <asm/sbi.h>
+#include <asm/vendorid_list.h>
+
+#define ANDESTECH_AX45MP_MARCHID 0x8000000000008a45UL
+#define ANDESTECH_AX45MP_MIMPID 0x500UL
+#define ANDESTECH_SBI_EXT_ANDES 0x0900031E
+
+#define RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND 0
+
+static long ax45mp_iocp_sw_workaround(void)
+{
+ struct sbiret ret;
+
+ /*
+ * RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND SBI EXT checks if the IOCP is missing and
+ * cache is controllable only then CMO will be applied to the platform.
+ */
+ ret = sbi_ecall(ANDESTECH_SBI_EXT_ANDES, RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND,
+ 0, 0, 0, 0, 0, 0);
+
+ return ret.error ? 0 : ret.value;
+}
+
+static bool errata_probe_iocp(unsigned int stage, unsigned long arch_id, unsigned long impid)
+{
+ if (!IS_ENABLED(CONFIG_ERRATA_ANDES_CMO))
+ return false;
+
+ if (arch_id != ANDESTECH_AX45MP_MARCHID || impid != ANDESTECH_AX45MP_MIMPID)
+ return false;
+
+ if (!ax45mp_iocp_sw_workaround())
+ return false;
+
+ /* Set this just to make core cbo code happy */
+ riscv_cbom_block_size = 1;
+ riscv_noncoherent_supported();
+
+ return true;
+}
+
+static void andes_errata_probe(unsigned int stage, unsigned long archid, unsigned long impid)
+{
+ /*
+ * In the absence of the I/O Coherency Port, access to certain peripherals
+ * requires vendor specific DMA handling.
+ */
+ errata_probe_iocp(stage, archid, impid);
+}
+
+void __init_or_module andes_errata_patch_func(struct alt_entry *begin, struct alt_entry *end,
+ unsigned long archid, unsigned long impid,
+ unsigned int stage)
+{
+ andes_errata_probe(stage, archid, impid);
+
+ /* we have nothing to patch here ATM so just return back */
+}
diff --git a/arch/riscv/include/asm/alternative.h b/arch/riscv/include/asm/alternative.h
index 1bd4027d34ca..e3a8e603eb5a 100644
--- a/arch/riscv/include/asm/alternative.h
+++ b/arch/riscv/include/asm/alternative.h
@@ -43,6 +43,9 @@ struct errata_checkfunc_id {
bool (*func)(struct alt_entry *alt);
};
+void andes_errata_patch_func(struct alt_entry *begin, struct alt_entry *end,
+ unsigned long archid, unsigned long impid,
+ unsigned int stage);
void sifive_errata_patch_func(struct alt_entry *begin, struct alt_entry *end,
unsigned long archid, unsigned long impid,
unsigned int stage);
diff --git a/arch/riscv/kernel/alternative.c b/arch/riscv/kernel/alternative.c
index 6212ea0eed72..d7027a977ee6 100644
--- a/arch/riscv/kernel/alternative.c
+++ b/arch/riscv/kernel/alternative.c
@@ -40,6 +40,11 @@ static void __init_or_module riscv_fill_cpu_mfr_info(struct cpu_manufacturer_inf
#endif
switch (cpu_mfr_info->vendor_id) {
+#ifdef CONFIG_ERRATA_ANDES
+ case ANDESTECH_VENDOR_ID:
+ cpu_mfr_info->patch_func = andes_errata_patch_func;
+ break;
+#endif
#ifdef CONFIG_ERRATA_SIFIVE
case SIFIVE_VENDOR_ID:
cpu_mfr_info->patch_func = sifive_errata_patch_func;
--
2.25.1
From: Lad Prabhakar <[email protected]>
Add DT binding documentation for L2 cache controller found on RZ/Five SoC.
The Renesas RZ/Five microprocessor includes a RISC-V CPU Core (AX45MP
Single) from Andes. The AX45MP core has an L2 cache controller, this patch
describes the L2 cache block.
Signed-off-by: Lad Prabhakar <[email protected]>
Reviewed-by: Rob Herring <[email protected]>
---
v5 -> v6
* Included RB tag from Geert
v4 -> v5
* Dropped L2 cache configuration properties
* Dropped PMA configuration properties
* Ordered the required list to match the properties list
RFC v3 -> v4
* Dropped l2 cache configuration parameters
* s/larger/large
* Added minItems/maxItems for andestech,pma-regions
---
.../cache/andestech,ax45mp-cache.yaml | 81 +++++++++++++++++++
1 file changed, 81 insertions(+)
create mode 100644 Documentation/devicetree/bindings/cache/andestech,ax45mp-cache.yaml
diff --git a/Documentation/devicetree/bindings/cache/andestech,ax45mp-cache.yaml b/Documentation/devicetree/bindings/cache/andestech,ax45mp-cache.yaml
new file mode 100644
index 000000000000..9f0be4835ad7
--- /dev/null
+++ b/Documentation/devicetree/bindings/cache/andestech,ax45mp-cache.yaml
@@ -0,0 +1,81 @@
+# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
+# Copyright (C) 2022 Renesas Electronics Corp.
+%YAML 1.2
+---
+$id: http://devicetree.org/schemas/cache/andestech,ax45mp-cache.yaml#
+$schema: http://devicetree.org/meta-schemas/core.yaml#
+
+title: Andestech AX45MP L2 Cache Controller
+
+maintainers:
+ - Lad Prabhakar <[email protected]>
+
+description:
+ A level-2 cache (L2C) is used to improve the system performance by providing
+ a large amount of cache line entries and reasonable access delays. The L2C
+ is shared between cores, and a non-inclusive non-exclusive policy is used.
+
+select:
+ properties:
+ compatible:
+ contains:
+ enum:
+ - andestech,ax45mp-cache
+
+ required:
+ - compatible
+
+properties:
+ compatible:
+ items:
+ - const: andestech,ax45mp-cache
+ - const: cache
+
+ reg:
+ maxItems: 1
+
+ interrupts:
+ maxItems: 1
+
+ cache-line-size:
+ const: 64
+
+ cache-level:
+ const: 2
+
+ cache-sets:
+ const: 1024
+
+ cache-size:
+ enum: [131072, 262144, 524288, 1048576, 2097152]
+
+ cache-unified: true
+
+ next-level-cache: true
+
+additionalProperties: false
+
+required:
+ - compatible
+ - reg
+ - interrupts
+ - cache-line-size
+ - cache-level
+ - cache-sets
+ - cache-size
+ - cache-unified
+
+examples:
+ - |
+ #include <dt-bindings/interrupt-controller/irq.h>
+
+ cache-controller@2010000 {
+ compatible = "andestech,ax45mp-cache", "cache";
+ reg = <0x13400000 0x100000>;
+ interrupts = <508 IRQ_TYPE_LEVEL_HIGH>;
+ cache-line-size = <64>;
+ cache-level = <2>;
+ cache-sets = <1024>;
+ cache-size = <262144>;
+ cache-unified;
+ };
--
2.25.1
From: Lad Prabhakar <[email protected]>
Explicitly select the required Cache management and Errata configs
required for the RZ/Five SoC.
Signed-off-by: Lad Prabhakar <[email protected]>
---
v5->v6
* New patch
---
drivers/soc/renesas/Kconfig | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/drivers/soc/renesas/Kconfig b/drivers/soc/renesas/Kconfig
index 660498252ec5..c3475c084685 100644
--- a/drivers/soc/renesas/Kconfig
+++ b/drivers/soc/renesas/Kconfig
@@ -340,6 +340,10 @@ if RISCV
config ARCH_R9A07G043
bool "RISC-V Platform support for RZ/Five"
select ARCH_RZG2L
+ select AX45MP_L2_CACHE
+ select DMA_GLOBAL_POOL
+ select ERRATA_ANDES
+ select ERRATA_ANDES_CMO
help
This enables support for the Renesas RZ/Five SoC.
--
2.25.1
On Fri, Jan 06, 2023 at 06:55:24PM +0000, Prabhakar wrote:
> From: Lad Prabhakar <[email protected]>
>
> Add DT binding documentation for L2 cache controller found on RZ/Five SoC.
>
> The Renesas RZ/Five microprocessor includes a RISC-V CPU Core (AX45MP
> Single) from Andes. The AX45MP core has an L2 cache controller, this patch
> describes the L2 cache block.
>
> Signed-off-by: Lad Prabhakar <[email protected]>
> Reviewed-by: Rob Herring <[email protected]>
> ---
> v5 -> v6
> * Included RB tag from Geert
I think not!
> v4 -> v5
> * Dropped L2 cache configuration properties
> * Dropped PMA configuration properties
> * Ordered the required list to match the properties list
>
> RFC v3 -> v4
> * Dropped l2 cache configuration parameters
> * s/larger/large
> * Added minItems/maxItems for andestech,pma-regions
> ---
> .../cache/andestech,ax45mp-cache.yaml | 81 +++++++++++++++++++
> 1 file changed, 81 insertions(+)
> create mode 100644 Documentation/devicetree/bindings/cache/andestech,ax45mp-cache.yaml
>
> diff --git a/Documentation/devicetree/bindings/cache/andestech,ax45mp-cache.yaml b/Documentation/devicetree/bindings/cache/andestech,ax45mp-cache.yaml
> new file mode 100644
> index 000000000000..9f0be4835ad7
> --- /dev/null
> +++ b/Documentation/devicetree/bindings/cache/andestech,ax45mp-cache.yaml
> @@ -0,0 +1,81 @@
> +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
> +# Copyright (C) 2022 Renesas Electronics Corp.
> +%YAML 1.2
> +---
> +$id: http://devicetree.org/schemas/cache/andestech,ax45mp-cache.yaml#
> +$schema: http://devicetree.org/meta-schemas/core.yaml#
> +
> +title: Andestech AX45MP L2 Cache Controller
> +
> +maintainers:
> + - Lad Prabhakar <[email protected]>
> +
> +description:
> + A level-2 cache (L2C) is used to improve the system performance by providing
> + a large amount of cache line entries and reasonable access delays. The L2C
> + is shared between cores, and a non-inclusive non-exclusive policy is used.
> +
> +select:
> + properties:
> + compatible:
> + contains:
> + enum:
> + - andestech,ax45mp-cache
> +
> + required:
> + - compatible
> +
> +properties:
> + compatible:
> + items:
> + - const: andestech,ax45mp-cache
> + - const: cache
You might find value in a specific compatible for your SoC & enforce
constraints for it. Or you might not & I don't care either way :)
Reviewed-by: Conor Dooley <[email protected]>
Thanks,
Conor.
On Fri, Jan 06, 2023 at 06:55:23PM +0000, Prabhakar wrote:
> From: Lad Prabhakar <[email protected]>
>
> Add required ports of the Alternative scheme for Andes CPU cores.
>
> I/O Coherence Port (IOCP) provides an AXI interface for connecting external
> non-caching masters, such as DMA controllers. IOCP is a specification
> option and is disabled on the Renesas RZ/Five SoC due to this reason cache
> management needs a software workaround.
>
> Signed-off-by: Lad Prabhakar <[email protected]>
> ---
> v5 -> v6
> * Dropped patching alternative and now just probing IOCP
>
> v4 -> v5
> * Sorted the Kconfig/Makefile/Switch based on Core name
> * Added a comments
> * Introduced RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND SBI EXT ID to check if
> CMO needs to be applied. Is there a way we can access the DTB while patching
> as we can drop this SBI EXT ID and add a DT property instead for cmo?
>
> RFC v3 -> v4
> * New patch
> ---
> arch/riscv/Kconfig.erratas | 22 +++++++++
> arch/riscv/errata/Makefile | 1 +
> arch/riscv/errata/andes/Makefile | 1 +
> arch/riscv/errata/andes/errata.c | 71 ++++++++++++++++++++++++++++
> arch/riscv/include/asm/alternative.h | 3 ++
> arch/riscv/kernel/alternative.c | 5 ++
> 6 files changed, 103 insertions(+)
> create mode 100644 arch/riscv/errata/andes/Makefile
> create mode 100644 arch/riscv/errata/andes/errata.c
>
> diff --git a/arch/riscv/Kconfig.erratas b/arch/riscv/Kconfig.erratas
> index 69621ae6d647..f0f0c1abd52b 100644
> --- a/arch/riscv/Kconfig.erratas
> +++ b/arch/riscv/Kconfig.erratas
> @@ -1,5 +1,27 @@
> menu "CPU errata selection"
>
> +config ERRATA_ANDES
> + bool "Andes AX45MP errata"
> + depends on !XIP_KERNEL
> + select RISCV_ALTERNATIVE
> + help
> + All Andes errata Kconfig depend on this Kconfig. Disabling
> + this Kconfig will disable all Andes errata. Please say "Y"
> + here if your platform uses Andes CPU cores.
> +
> + Otherwise, please say "N" here to avoid unnecessary overhead.
> +
> +config ERRATA_ANDES_CMO
> + bool "Apply Andes cache management errata"
> + depends on ERRATA_ANDES && MMU && ARCH_R9A07G043
> + select RISCV_DMA_NONCOHERENT
> + default y
> + help
> + This will apply the cache management errata to handle the
> + non-standard handling on non-coherent operations on Andes cores.
> +
> + If you don't know what to do here, say "Y".
Ideally we would not need errata to turn this stuff on at all, but, as
you pointed out to me off-list, arch_setup_dma_ops() complains if we
have not set up.
I'm happy to commit to trying to sort that out in follow on work w/ MPFS,
since in that case it really isn't errata, and not require it for this
series as you do fit that particular bill IMO.
Reviewed-by: Conor Dooley <[email protected]>
Thanks,
Conor.
FWIW:
Reviewed-by: Conor Dooley <[email protected]>
On Fri, Jan 06, 2023 at 06:55:25PM +0000, Prabhakar wrote:
> From: Lad Prabhakar <[email protected]>
>
> I/O Coherence Port (IOCP) provides an AXI interface for connecting
> external non-caching masters, such as DMA controllers. The accesses
> from IOCP are coherent with D-Caches and L2 Cache.
>
> IOCP is a specification option and is disabled on the Renesas RZ/Five
> SoC due to this reason IP blocks using DMA will fail.
>
> The Andes AX45MP core has a Programmable Physical Memory Attributes (PMA)
> block that allows dynamic adjustment of memory attributes in the runtime.
> It contains a configurable amount of PMA entries implemented as CSR
> registers to control the attributes of memory locations in interest.
> Below are the memory attributes supported:
> * Device, Non-bufferable
> * Device, bufferable
> * Memory, Non-cacheable, Non-bufferable
> * Memory, Non-cacheable, Bufferable
> * Memory, Write-back, No-allocate
> * Memory, Write-back, Read-allocate
> * Memory, Write-back, Write-allocate
> * Memory, Write-back, Read and Write-allocate
>
> More info about PMA (section 10.3):
> Link: http://www.andestech.com/wp-content/uploads/AX45MP-1C-Rev.-5.0.0-Datasheet.pdf
>
> As a workaround for SoCs with IOCP disabled CMO needs to be handled by
> software. Firstly OpenSBI configures the memory region as
> "Memory, Non-cacheable, Bufferable" and passes this region as a global
> shared dma pool as a DT node. With DMA_GLOBAL_POOL enabled all DMA
> allocations happen from this region and synchronization callbacks are
> implemented to synchronize when doing DMA transactions.
>
> Example PMA region passes as a DT node from OpenSBI:
> reserved-memory {
> #address-cells = <2>;
> #size-cells = <2>;
> ranges;
>
> pma_resv0@58000000 {
> compatible = "shared-dma-pool";
> reg = <0x0 0x58000000 0x0 0x08000000>;
> no-map;
> linux,dma-default;
> };
> };
>
> Signed-off-by: Lad Prabhakar <[email protected]>
> diff --git a/drivers/cache/ax45mp_cache.c b/drivers/cache/ax45mp_cache.c
> new file mode 100644
> index 000000000000..556e6875627c
> --- /dev/null
> +++ b/drivers/cache/ax45mp_cache.c
> @@ -0,0 +1,279 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * non-coherent cache functions for Andes AX45MP
> + *
> + * Copyright (C) 2022 Renesas Electronics Corp.
> + */
> +
> +#include <asm/dma-noncoherent.h>
> +#include <linux/cacheflush.h>
> +#include <linux/cacheinfo.h>
> +#include <linux/dma-direction.h>
> +#include <linux/of_address.h>
> +#include <linux/of_platform.h>
> +
> +#include <asm/cacheflush.h>
> +#include <asm/sbi.h>
You don't actually need this anymore, do you?
> +static int ax45mp_l2c_probe(struct platform_device *pdev)
> +{
> + struct riscv_cache_ops ax45mp_cmo_ops;
> +
> + /*
> + * riscv_cbom_block_size is set very much earlier so we can
> + * definitely rely on it and only if its being set we continue
> + * further in the probe path.
> + */
> + if (!riscv_cbom_block_size)
> + return 0;
Return 0? That's because we may actually have the IOCP & do not want to
install ops, right?
If so, please add that to the comment.
> +
> + ax45mp_priv = devm_kzalloc(&pdev->dev, sizeof(*ax45mp_priv), GFP_KERNEL);
> + if (!ax45mp_priv)
> + return -ENOMEM;
> +
> + ax45mp_priv->l2c_base = devm_platform_ioremap_resource(pdev, 0);
> + if (IS_ERR(ax45mp_priv->l2c_base))
> + return PTR_ERR(ax45mp_priv->l2c_base);
> +
> + ax45mp_get_l2_line_size(pdev);
> +
> + memset(&ax45mp_cmo_ops, 0x0, sizeof(ax45mp_cmo_ops));
> + ax45mp_cmo_ops.riscv_dma_noncoherent_cmo_ops = &ax45mp_no_iocp_cmo;
Yah, drop this dance and use a static struct foo_ops construct please.
With those two, I'm happy with this I guess..
Reviewed-by: Conor Dooley <[email protected]>
> + riscv_noncoherent_register_cache_ops(&ax45mp_cmo_ops);
> +
> + return 0;
> +}
> +
> +static const struct of_device_id ax45mp_cache_ids[] = {
> + { .compatible = "andestech,ax45mp-cache" },
> + { /* sentinel */ }
> +};
> +
> +static struct platform_driver ax45mp_l2c_driver = {
> + .driver = {
> + .name = "ax45mp-l2c",
> + .of_match_table = ax45mp_cache_ids,
> + },
> + .probe = ax45mp_l2c_probe,
> +};
> +
> +static int __init ax45mp_cache_init(void)
> +{
> + return platform_driver_register(&ax45mp_l2c_driver);
> +}
> +arch_initcall(ax45mp_cache_init);
> +
> +MODULE_AUTHOR("Lad Prabhakar <[email protected]>");
> +MODULE_DESCRIPTION("Andes AX45MP L2 cache driver");
> +MODULE_LICENSE("GPL");
BTW, I think these are surplus-to-requirements since this is never going
to be built as a module.
If you resurrect the directory level maintainers entry from my v5.1, you
can also add:
Acked-by: Conor Dooley <[email protected]>
Thanks,
Conor.
Hi Conor,
Thank you for the review.
On Fri, Jan 6, 2023 at 9:53 PM Conor Dooley <[email protected]> wrote:
>
> On Fri, Jan 06, 2023 at 06:55:24PM +0000, Prabhakar wrote:
> > From: Lad Prabhakar <[email protected]>
> >
> > Add DT binding documentation for L2 cache controller found on RZ/Five SoC.
> >
> > The Renesas RZ/Five microprocessor includes a RISC-V CPU Core (AX45MP
> > Single) from Andes. The AX45MP core has an L2 cache controller, this patch
> > describes the L2 cache block.
> >
> > Signed-off-by: Lad Prabhakar <[email protected]>
> > Reviewed-by: Rob Herring <[email protected]>
> > ---
> > v5 -> v6
> > * Included RB tag from Geert
>
> I think not!
>
Sorry that was a typo. I meant Rob :)
> > v4 -> v5
> > * Dropped L2 cache configuration properties
> > * Dropped PMA configuration properties
> > * Ordered the required list to match the properties list
> >
> > RFC v3 -> v4
> > * Dropped l2 cache configuration parameters
> > * s/larger/large
> > * Added minItems/maxItems for andestech,pma-regions
> > ---
> > .../cache/andestech,ax45mp-cache.yaml | 81 +++++++++++++++++++
> > 1 file changed, 81 insertions(+)
> > create mode 100644 Documentation/devicetree/bindings/cache/andestech,ax45mp-cache.yaml
> >
> > diff --git a/Documentation/devicetree/bindings/cache/andestech,ax45mp-cache.yaml b/Documentation/devicetree/bindings/cache/andestech,ax45mp-cache.yaml
> > new file mode 100644
> > index 000000000000..9f0be4835ad7
> > --- /dev/null
> > +++ b/Documentation/devicetree/bindings/cache/andestech,ax45mp-cache.yaml
> > @@ -0,0 +1,81 @@
> > +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
> > +# Copyright (C) 2022 Renesas Electronics Corp.
> > +%YAML 1.2
> > +---
> > +$id: http://devicetree.org/schemas/cache/andestech,ax45mp-cache.yaml#
> > +$schema: http://devicetree.org/meta-schemas/core.yaml#
> > +
> > +title: Andestech AX45MP L2 Cache Controller
> > +
> > +maintainers:
> > + - Lad Prabhakar <[email protected]>
> > +
> > +description:
> > + A level-2 cache (L2C) is used to improve the system performance by providing
> > + a large amount of cache line entries and reasonable access delays. The L2C
> > + is shared between cores, and a non-inclusive non-exclusive policy is used.
> > +
> > +select:
> > + properties:
> > + compatible:
> > + contains:
> > + enum:
> > + - andestech,ax45mp-cache
> > +
> > + required:
> > + - compatible
> > +
> > +properties:
> > + compatible:
> > + items:
> > + - const: andestech,ax45mp-cache
> > + - const: cache
>
> You might find value in a specific compatible for your SoC & enforce
> constraints for it. Or you might not & I don't care either way :)
>
Good point actually. Geert what do you think?
Cheers,
Prabhakar
Hi Conor,
Thank you for the review.
On Sat, Jan 7, 2023 at 12:09 AM Conor Dooley <[email protected]> wrote:
>
> On Fri, Jan 06, 2023 at 06:55:25PM +0000, Prabhakar wrote:
> > From: Lad Prabhakar <[email protected]>
> >
> > I/O Coherence Port (IOCP) provides an AXI interface for connecting
> > external non-caching masters, such as DMA controllers. The accesses
> > from IOCP are coherent with D-Caches and L2 Cache.
> >
> > IOCP is a specification option and is disabled on the Renesas RZ/Five
> > SoC due to this reason IP blocks using DMA will fail.
> >
> > The Andes AX45MP core has a Programmable Physical Memory Attributes (PMA)
> > block that allows dynamic adjustment of memory attributes in the runtime.
> > It contains a configurable amount of PMA entries implemented as CSR
> > registers to control the attributes of memory locations in interest.
> > Below are the memory attributes supported:
> > * Device, Non-bufferable
> > * Device, bufferable
> > * Memory, Non-cacheable, Non-bufferable
> > * Memory, Non-cacheable, Bufferable
> > * Memory, Write-back, No-allocate
> > * Memory, Write-back, Read-allocate
> > * Memory, Write-back, Write-allocate
> > * Memory, Write-back, Read and Write-allocate
> >
> > More info about PMA (section 10.3):
> > Link: http://www.andestech.com/wp-content/uploads/AX45MP-1C-Rev.-5.0.0-Datasheet.pdf
> >
> > As a workaround for SoCs with IOCP disabled CMO needs to be handled by
> > software. Firstly OpenSBI configures the memory region as
> > "Memory, Non-cacheable, Bufferable" and passes this region as a global
> > shared dma pool as a DT node. With DMA_GLOBAL_POOL enabled all DMA
> > allocations happen from this region and synchronization callbacks are
> > implemented to synchronize when doing DMA transactions.
> >
> > Example PMA region passes as a DT node from OpenSBI:
> > reserved-memory {
> > #address-cells = <2>;
> > #size-cells = <2>;
> > ranges;
> >
> > pma_resv0@58000000 {
> > compatible = "shared-dma-pool";
> > reg = <0x0 0x58000000 0x0 0x08000000>;
> > no-map;
> > linux,dma-default;
> > };
> > };
> >
> > Signed-off-by: Lad Prabhakar <[email protected]>
>
> > diff --git a/drivers/cache/ax45mp_cache.c b/drivers/cache/ax45mp_cache.c
> > new file mode 100644
> > index 000000000000..556e6875627c
> > --- /dev/null
> > +++ b/drivers/cache/ax45mp_cache.c
> > @@ -0,0 +1,279 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * non-coherent cache functions for Andes AX45MP
> > + *
> > + * Copyright (C) 2022 Renesas Electronics Corp.
> > + */
> > +
> > +#include <asm/dma-noncoherent.h>
> > +#include <linux/cacheflush.h>
> > +#include <linux/cacheinfo.h>
> > +#include <linux/dma-direction.h>
> > +#include <linux/of_address.h>
> > +#include <linux/of_platform.h>
> > +
> > +#include <asm/cacheflush.h>
> > +#include <asm/sbi.h>
>
> You don't actually need this anymore, do you?
>
Agreed these can be dropped now.
> > +static int ax45mp_l2c_probe(struct platform_device *pdev)
> > +{
> > + struct riscv_cache_ops ax45mp_cmo_ops;
> > +
> > + /*
> > + * riscv_cbom_block_size is set very much earlier so we can
> > + * definitely rely on it and only if its being set we continue
> > + * further in the probe path.
> > + */
> > + if (!riscv_cbom_block_size)
> > + return 0;
>
> Return 0? That's because we may actually have the IOCP & do not want to
> install ops, right?
> If so, please add that to the comment.
>
OK, I will add that.
> > +
> > + ax45mp_priv = devm_kzalloc(&pdev->dev, sizeof(*ax45mp_priv), GFP_KERNEL);
> > + if (!ax45mp_priv)
> > + return -ENOMEM;
> > +
> > + ax45mp_priv->l2c_base = devm_platform_ioremap_resource(pdev, 0);
> > + if (IS_ERR(ax45mp_priv->l2c_base))
> > + return PTR_ERR(ax45mp_priv->l2c_base);
> > +
> > + ax45mp_get_l2_line_size(pdev);
> > +
> > + memset(&ax45mp_cmo_ops, 0x0, sizeof(ax45mp_cmo_ops));
> > + ax45mp_cmo_ops.riscv_dma_noncoherent_cmo_ops = &ax45mp_no_iocp_cmo;
>
> Yah, drop this dance and use a static struct foo_ops construct please.
>
Will do.
> With those two, I'm happy with this I guess..
> Reviewed-by: Conor Dooley <[email protected]>
>
> > + riscv_noncoherent_register_cache_ops(&ax45mp_cmo_ops);
> > +
> > + return 0;
> > +}
> > +
> > +static const struct of_device_id ax45mp_cache_ids[] = {
> > + { .compatible = "andestech,ax45mp-cache" },
> > + { /* sentinel */ }
> > +};
> > +
> > +static struct platform_driver ax45mp_l2c_driver = {
> > + .driver = {
> > + .name = "ax45mp-l2c",
> > + .of_match_table = ax45mp_cache_ids,
> > + },
> > + .probe = ax45mp_l2c_probe,
> > +};
> > +
> > +static int __init ax45mp_cache_init(void)
> > +{
> > + return platform_driver_register(&ax45mp_l2c_driver);
> > +}
> > +arch_initcall(ax45mp_cache_init);
> > +
> > +MODULE_AUTHOR("Lad Prabhakar <[email protected]>");
> > +MODULE_DESCRIPTION("Andes AX45MP L2 cache driver");
> > +MODULE_LICENSE("GPL");
>
> BTW, I think these are surplus-to-requirements since this is never going
> to be built as a module.
>
Agreed, I will drop it.
> If you resurrect the directory level maintainers entry from my v5.1, you
> can also add:
Sure I'll add it in the next version..
> Acked-by: Conor Dooley <[email protected]>
>
... and also include your ACK.
Cheers,
Prabhakar
Hi Prabhakar,
On Sat, Jan 7, 2023 at 9:47 PM Lad, Prabhakar
<[email protected]> wrote:
> On Fri, Jan 6, 2023 at 9:53 PM Conor Dooley <[email protected]> wrote:
> > On Fri, Jan 06, 2023 at 06:55:24PM +0000, Prabhakar wrote:
> > > From: Lad Prabhakar <[email protected]>
> > >
> > > Add DT binding documentation for L2 cache controller found on RZ/Five SoC.
> > >
> > > The Renesas RZ/Five microprocessor includes a RISC-V CPU Core (AX45MP
> > > Single) from Andes. The AX45MP core has an L2 cache controller, this patch
> > > describes the L2 cache block.
> > >
> > > Signed-off-by: Lad Prabhakar <[email protected]>
> > > Reviewed-by: Rob Herring <[email protected]>
> > > --- /dev/null
> > > +++ b/Documentation/devicetree/bindings/cache/andestech,ax45mp-cache.yaml
> > > @@ -0,0 +1,81 @@
> > > +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
> > > +# Copyright (C) 2022 Renesas Electronics Corp.
> > > +%YAML 1.2
> > > +---
> > > +$id: http://devicetree.org/schemas/cache/andestech,ax45mp-cache.yaml#
> > > +$schema: http://devicetree.org/meta-schemas/core.yaml#
> > > +
> > > +title: Andestech AX45MP L2 Cache Controller
> > > +
> > > +maintainers:
> > > + - Lad Prabhakar <[email protected]>
> > > +
> > > +description:
> > > + A level-2 cache (L2C) is used to improve the system performance by providing
> > > + a large amount of cache line entries and reasonable access delays. The L2C
> > > + is shared between cores, and a non-inclusive non-exclusive policy is used.
> > > +
> > > +select:
> > > + properties:
> > > + compatible:
> > > + contains:
> > > + enum:
> > > + - andestech,ax45mp-cache
> > > +
> > > + required:
> > > + - compatible
> > > +
> > > +properties:
> > > + compatible:
> > > + items:
> > > + - const: andestech,ax45mp-cache
> > > + - const: cache
> >
> > You might find value in a specific compatible for your SoC & enforce
> > constraints for it. Or you might not & I don't care either way :)
> >
> Good point actually. Geert what do you think?
That might be prudent, to cater for the way the standard AX45MP cache
block is integrated into the RZ/Five (or any other) SoC.
Still, in the absence of an SoC-specific compatible value, you can
handle integration issues using soc_device_match().
Gr{oetje,eeting}s,
Geert
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]
In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
-- Linus Torvalds
Hi Geert,
Thanks for the feedback.
On Mon, Jan 9, 2023 at 12:15 PM Geert Uytterhoeven <[email protected]> wrote:
>
> Hi Prabhakar,
>
> On Sat, Jan 7, 2023 at 9:47 PM Lad, Prabhakar
> <[email protected]> wrote:
> > On Fri, Jan 6, 2023 at 9:53 PM Conor Dooley <[email protected]> wrote:
> > > On Fri, Jan 06, 2023 at 06:55:24PM +0000, Prabhakar wrote:
> > > > From: Lad Prabhakar <[email protected]>
> > > >
> > > > Add DT binding documentation for L2 cache controller found on RZ/Five SoC.
> > > >
> > > > The Renesas RZ/Five microprocessor includes a RISC-V CPU Core (AX45MP
> > > > Single) from Andes. The AX45MP core has an L2 cache controller, this patch
> > > > describes the L2 cache block.
> > > >
> > > > Signed-off-by: Lad Prabhakar <[email protected]>
> > > > Reviewed-by: Rob Herring <[email protected]>
>
> > > > --- /dev/null
> > > > +++ b/Documentation/devicetree/bindings/cache/andestech,ax45mp-cache.yaml
> > > > @@ -0,0 +1,81 @@
> > > > +# SPDX-License-Identifier: (GPL-2.0 OR BSD-2-Clause)
> > > > +# Copyright (C) 2022 Renesas Electronics Corp.
> > > > +%YAML 1.2
> > > > +---
> > > > +$id: http://devicetree.org/schemas/cache/andestech,ax45mp-cache.yaml#
> > > > +$schema: http://devicetree.org/meta-schemas/core.yaml#
> > > > +
> > > > +title: Andestech AX45MP L2 Cache Controller
> > > > +
> > > > +maintainers:
> > > > + - Lad Prabhakar <[email protected]>
> > > > +
> > > > +description:
> > > > + A level-2 cache (L2C) is used to improve the system performance by providing
> > > > + a large amount of cache line entries and reasonable access delays. The L2C
> > > > + is shared between cores, and a non-inclusive non-exclusive policy is used.
> > > > +
> > > > +select:
> > > > + properties:
> > > > + compatible:
> > > > + contains:
> > > > + enum:
> > > > + - andestech,ax45mp-cache
> > > > +
> > > > + required:
> > > > + - compatible
> > > > +
> > > > +properties:
> > > > + compatible:
> > > > + items:
> > > > + - const: andestech,ax45mp-cache
> > > > + - const: cache
> > >
> > > You might find value in a specific compatible for your SoC & enforce
> > > constraints for it. Or you might not & I don't care either way :)
> > >
> > Good point actually. Geert what do you think?
>
> That might be prudent, to cater for the way the standard AX45MP cache
> block is integrated into the RZ/Five (or any other) SoC.
>
> Still, in the absence of an SoC-specific compatible value, you can
> handle integration issues using soc_device_match().
>
Agreed, I'll continue with the current DT bindings.
Cheers,
Prabhakar