Hi Conor,

On Thu, Apr 13, 2023 at 8:06 AM Conor Dooley <[email protected]> wrote:
>
> On Wed, Apr 12, 2023 at 09:25:34PM +0100, Conor Dooley wrote:
> > On Wed, Apr 12, 2023 at 12:08:58PM +0100, Prabhakar wrote:
> > > From: Lad Prabhakar <[email protected]>
> > >
> > > I/O Coherence Port (IOCP) provides an AXI interface for connecting
> > > external non-caching masters, such as DMA controllers. The accesses
> > > from IOCP are coherent with D-Caches and L2 Cache.
> > >
> > > IOCP is a specification option and is disabled on the Renesas RZ/Five
> > > SoC due to this reason IP blocks using DMA will fail.
> > >
> > > The Andes AX45MP core has a Programmable Physical Memory Attributes (PMA)
> > > block that allows dynamic adjustment of memory attributes in the runtime.
> > > It contains a configurable amount of PMA entries implemented as CSR
> > > registers to control the attributes of memory locations in interest.
> > > Below are the memory attributes supported:
> > > * Device, Non-bufferable
> > > * Device, bufferable
> > > * Memory, Non-cacheable, Non-bufferable
> > > * Memory, Non-cacheable, Bufferable
> > > * Memory, Write-back, No-allocate
> > > * Memory, Write-back, Read-allocate
> > > * Memory, Write-back, Write-allocate
> > > * Memory, Write-back, Read and Write-allocate
> > >
> > > More info about PMA (section 10.3):
> > > Link: http://www.andestech.com/wp-content/uploads/AX45MP-1C-Rev.-5.0.0-Datasheet.pdf
> > >
> > > As a workaround for SoCs with IOCP disabled CMO needs to be handled by
> > > software. Firstly OpenSBI configures the memory region as
> > > "Memory, Non-cacheable, Bufferable" and passes this region as a global
> > > shared dma pool as a DT node. With DMA_GLOBAL_POOL enabled all DMA
> > > allocations happen from this region and synchronization callbacks are
> > > implemented to synchronize when doing DMA transactions.
> > >
> > > Example PMA region passes as a DT node from OpenSBI:
> > > reserved-memory {
> > > #address-cells = <2>;
> > > #size-cells = <2>;
> > > ranges;
> > >
> > > pma_resv0@58000000 {
> > > compatible = "shared-dma-pool";
> > > reg = <0x0 0x58000000 0x0 0x08000000>;
> > > no-map;
> > > linux,dma-default;
> > > };
> > > };
> > >
> > > Signed-off-by: Lad Prabhakar <[email protected]>
> > > ---
> > > v7 -> v8
> > > * Dropped function pointer usage
> > > * Now exporting the functions for clean/inval/flush
> > > * Switched to using early_initcall instead of arch_initcall
> > > * Dropped entry for "include/cache" from MAINTAINERS
> > > * Dropped dependency of RISCV on AX45MP_L2_CACHE
> > > * Returning error in case of cache line mismatch
> >
> > > * Renamed clean/inval/flush functions
> >
> > I kinda screwed you with that request given Hellwig's NAK on the
> > function pointer based stuff. Ah well, I prefer matching the proposed
> > naming of the dma core to what RVI chose for the instructions.
> >
> > Reviewed-by: Conor Dooley <[email protected]>
> >
> > I suppose this will need a resubmission once Arnd's stuff gets applied,
> > but I would like to see it have a run through the build bots etc.
>
> So apparently my build bot did actually run against this series?
> https://patchwork.kernel.org/project/linux-riscv/list/?series=739109
>
> To be quite honest, I am not sure at all how it managed to apply the
> series w/ Arnd's pre-reqs. Perhaps it has achieved some from of
> sentience. There's a build failure for 32-bit that appeared on the final
> patch, but is not really its fault:
> ../arch/riscv/mm/dma-noncoherent.c: Assembler messages:
> ../arch/riscv/mm/dma-noncoherent.c:104: Error: unrecognized opcode `sd s0,0(sp)'
> ../arch/riscv/mm/dma-noncoherent.c:105: Error: unrecognized opcode `sd ra,8(sp)'
> ../arch/riscv/mm/dma-noncoherent.c:110: Error: unrecognized opcode `ld ra,8(sp)'
> ../arch/riscv/mm/dma-noncoherent.c:111: Error: unrecognized opcode `ld s0,0(sp)'
> ../arch/riscv/mm/dma-noncoherent.c:111: Error: unrecognized opcode `sd s0,0(sp)'
> ../arch/riscv/mm/dma-noncoherent.c:112: Error: unrecognized opcode `sd ra,8(sp)'
> ../arch/riscv/mm/dma-noncoherent.c:117: Error: unrecognized opcode `ld ra,8(sp)'
> ../arch/riscv/mm/dma-noncoherent.c:118: Error: unrecognized opcode `ld s0,0(sp)'
> ../arch/riscv/mm/pmem.c: Assembler messages:
> ../arch/riscv/mm/pmem.c:98: Error: unrecognized opcode `sd s0,0(sp)'
> ../arch/riscv/mm/pmem.c:99: Error: unrecognized opcode `sd ra,8(sp)'
> ../arch/riscv/mm/pmem.c:104: Error: unrecognized opcode `ld ra,8(sp)'
> ../arch/riscv/mm/pmem.c:105: Error: unrecognized opcode `ld s0,0(sp)'
> ../arch/riscv/mm/dma-noncoherent.c:138: Error: unrecognized opcode `sd s0,0(sp)'
> ../arch/riscv/mm/dma-noncoherent.c:139: Error: unrecognized opcode `sd ra,8(sp)'
> ../arch/riscv/mm/dma-noncoherent.c:144: Error: unrecognized opcode `ld ra,8(sp)'
> ../arch/riscv/mm/dma-noncoherent.c:145: Error: unrecognized opcode `ld s0,0(sp)'
> ../arch/riscv/mm/pmem.c:104: Error: unrecognized opcode `sd s0,0(sp)'
> ../arch/riscv/mm/pmem.c:105: Error: unrecognized opcode `sd ra,8(sp)'
> ../arch/riscv/mm/pmem.c:110: Error: unrecognized opcode `ld ra,8(sp)'
> ../arch/riscv/mm/pmem.c:111: Error: unrecognized opcode `ld s0,0(sp)'
> ../arch/riscv/mm/pmem.c:110: Error: attempt to move .org backwards
> ../arch/riscv/mm/pmem.c:116: Error: attempt to move .org backwards
> ../arch/riscv/mm/dma-noncoherent.c:116: Error: attempt to move .org backwards
> ../arch/riscv/mm/dma-noncoherent.c:123: Error: attempt to move .org backwards
> ../arch/riscv/mm/dma-noncoherent.c:150: Error: attempt to move .org backwards
> make[4]: *** [../scripts/Makefile.build:252: arch/riscv/mm/pmem.o] Error 1
> make[4]: *** [../scripts/Makefile.build:252: arch/riscv/mm/dma-noncoherent.o] Error 1
> make[4]: Target 'arch/riscv/mm/' not remade because of errors.
> make[3]: *** [../scripts/Makefile.build:494: arch/riscv/mm] Error 2
> make[3]: Target 'arch/riscv/' not remade because of errors.
> make[2]: *** [../scripts/Makefile.build:494: arch/riscv] Error 2
>
> The simplest solution may to just be making the erratum depend on 64BIT?
>
I dont think this will work, as pmem.c is compiled unconditionally. Is
dma-noncoherent.c also valid for RISCV-32? If not then we can make
pmem.c compile conditionally if DMA non-coherenet is enabled and we
make DMA non-coherent depend on 64bit.

Cheers,
Prabhakar

2023-04-13 18:48:28

On Wed, Apr 19, 2023 at 08:59:03AM -0700, Palmer Dabbelt wrote:
> IMO we should just take the support in Linux: trying to hide stuff behind the
> SBI leads to more more headaches than it's worth, we end up with a bunch of
> broken firmware to try and work around. We've already got a mess here because
> of the D1 support, we might as well just live with it.

I strongly disagree. Adding more and more per-vendor ops simply does
not scale. We're getting us into a giant long-time mess that will be
unfixable.