Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
MIME-Version: 1.0
References: <20210807145537.124744-1-xianting.tian@linux.alibaba.com>
 <20210809003044.6692ddce@xhacker> <c09fc86d-1e6a-3315-7489-9e269935ba55@linux.alibaba.com>
 <CAK8P3a1wyhLp1hUUotzwqc1pyCyfWhO7O_pvt5O_U=qZp-ZhnA@mail.gmail.com>
In-Reply-To: <CAK8P3a1wyhLp1hUUotzwqc1pyCyfWhO7O_pvt5O_U=qZp-ZhnA@mail.gmail.com>
From:   Atish Patra <atishp@atishpatra.org>
Date:   Mon, 9 Aug 2021 12:19:20 -0700
Message-ID: <CAOnJCU+YQOF2+_FNa5Hpig_dnymNA-ZvtgU4-B5eisTyB9GT6Q@mail.gmail.com>
Subject: Re: [PATCH] riscv: add ARCH_DMA_MINALIGN support
To:     Arnd Bergmann <arnd@kernel.org>
Cc:     Xianting TIan <xianting.tian@linux.alibaba.com>,
        Jisheng Zhang <jszhang3@mail.ustc.edu.cn>,
        Paul Walmsley <paul.walmsley@sifive.com>,
        Palmer Dabbelt <palmer@dabbelt.com>,
        Albert Ou <aou@eecs.berkeley.edu>,
        linux-riscv <linux-riscv@lists.infradead.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Mon, Aug 9, 2021 at 12:50 AM Arnd Bergmann <arnd@kernel.org> wrote:
>
> On Mon, Aug 9, 2021 at 8:20 AM Xianting TIan
> <xianting.tian@linux.alibaba.com> wrote:
> >
> > >> +#define ARCH_DMA_MINALIGN   L1_CACHE_BYTES
> > > It's not a good idea to blindly set this for all riscv. For "coherent"
> > > platforms, this is not necessary and will waste memory.
> >
> > I checked ARCH_DMA_MINALIGN definition,  "If an architecture isn't fully
> > DMA-coherent, ARCH_DMA_MINALIGN must be set".
> >
> > so that the memory allocator makes sure that kmalloc'ed buffer doesn't
> > share a cache line with the others.
> >
> > Documentation/core-api/dma-api-howto.rst
> >
> > 2) ARCH_DMA_MINALIGN
> >
> >     Architectures must ensure that kmalloc'ed buffer is
> >     DMA-safe. Drivers and subsystems depend on it. If an architecture
> >     isn't fully DMA-coherent (i.e. hardware doesn't ensure that data in
> >     the CPU cache is identical to data in main memory),
> >     ARCH_DMA_MINALIGN must be set so that the memory allocator
> >     makes sure that kmalloc'ed buffer doesn't share a cache line with
> >     the others. See arch/arm/include/asm/cache.h as an example.
> >
> >     Note that ARCH_DMA_MINALIGN is about DMA memory alignment
> >     constraints. You don't need to worry about the architecture data
> >     alignment constraints (e.g. the alignment constraints about 64-bit
> >     objects).
>
> The platform spec [1] says about this:
>
> | Memory accesses by I/O masters can be coherent or non-coherent
> | with respect to all hart-related caches.
>
> So the kernel in its default configuration can not assume that DMA is
> cache coherent on RISC-V. Making this configurable implies that
> a kernel that is configured for cache-coherent machines can no longer
> run on all hardware that follows the platform spec.
>
> We have the same problem on arm64, where most of the server parts
> are cache coherent, but the majority of the low-end embedded devices
> are not, and we require that a single kernel ran run on all of the above.
>
> One idea that we have discussed several times is to start the kernel
> without the small kmalloc caches and defer their creation until a
> later point in the boot process after determining whether any
> non-coherent devices have been discovered. Any in-kernel structures
> that have an explicit ARCH_DMA_MINALIGN alignment won't
> benefit from this, but any subsequent kmalloc() calls can use the
> smaller caches. The tricky bit is finding out whether /everything/ on
> the system is cache-coherent or not, since we do not have a global
> flag for that in the DT. See [2] for a recent discussion.

Can we add a new DT property to indicate the system is fully cache-coherent ?
That will be helpful for RISC-V as well. We already have platforms like hifive
unleashed/unmatched that are coherent while beagleV is not.

The workaround to support both platforms in a single image was not
very pretty [1].

[1] https://patchwork.kernel.org/project/linux-riscv/list/?series=520541

>
>        Arnd
>
> [1] https://github.com/riscv/riscv-platform-specs/blob/main/riscv-platform-spec.adoc#architecture
> [2] https://lore.kernel.org/linux-arm-kernel/20210527124356.22367-1-will@kernel.org/
>
> _______________________________________________
> linux-riscv mailing list
> linux-riscv@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-riscv


-- 
Regards,
Atish