Currently, riscv defines ARCH_DMA_MINALIGN as L1_CACHE_BYTES, I.E
64Bytes, if CONFIG_RISCV_DMA_NONCOHERENT=y. To support unified kernel
Image, usually we have to enable CONFIG_RISCV_DMA_NONCOHERENT, thus
it brings some bad effects to for coherent platforms:
Firstly, it wastes memory, kmalloc-96, kmalloc-32, kmalloc-16 and
kmalloc-8 slab caches don't exist any more, they are replaced with
either kmalloc-128 or kmalloc-64.
Secondly, larger than necessary kmalloc aligned allocations results
in unnecessary cache/TLB pressure.
This issue also exists on arm64 platforms. From last year, Catalin
tried to solve this issue by decoupling ARCH_KMALLOC_MINALIGN from
ARCH_DMA_MINALIGN, limiting kmalloc() minimum alignment to
dma_get_cache_alignment() and replacing ARCH_KMALLOC_MINALIGN usage
in various drivers with ARCH_DMA_MINALIGN etc.
One fact we can make use of for riscv: if the CPU doesn't support
ZICBOM or T-HEAD CMO, we know the platform is coherent. Based on
Catalin's work and above fact, we can easily solve the kmalloc align
issue for riscv: we can override dma_get_cache_alignment(), then let
it return ARCH_DMA_MINALIGN at the beginning and return 1 once we know
the underlying HW neither supports ZICBOM nor supports T-HEAD CMO.
So what about if the CPU supports ZICBOM and T-HEAD CMO, but all the
devices are dma coherent? Well, we use ARCH_DMA_MINALIGN as the
kmalloc minimum alignment, nothing changed in this case. This case
can be improved in the future.
After this patch, a simple test of booting to a small buildroot rootfs
on qemu shows:
kmalloc-96 5041 5041 96 ...
kmalloc-64 9606 9606 64 ...
kmalloc-32 5128 5128 32 ...
kmalloc-16 7682 7682 16 ...
kmalloc-8 10246 10246 8 ...
So we save about 1268KB memory. The saving will be much larger in normal
OS env on real HW platforms.
[1] Link: https://lore.kernel.org/linux-arm-kernel/[email protected]/
Signed-off-by: Jisheng Zhang <[email protected]>
---
arch/riscv/include/asm/cache.h | 14 ++++++++++++++
arch/riscv/mm/dma-noncoherent.c | 4 ++++
2 files changed, 18 insertions(+)
diff --git a/arch/riscv/include/asm/cache.h b/arch/riscv/include/asm/cache.h
index d3036df23ccb..2174fe7bac9a 100644
--- a/arch/riscv/include/asm/cache.h
+++ b/arch/riscv/include/asm/cache.h
@@ -13,6 +13,7 @@
#ifdef CONFIG_RISCV_DMA_NONCOHERENT
#define ARCH_DMA_MINALIGN L1_CACHE_BYTES
+#define ARCH_KMALLOC_MINALIGN (8)
#endif
/*
@@ -23,4 +24,17 @@
#define ARCH_SLAB_MINALIGN 16
#endif
+#ifndef __ASSEMBLY__
+
+#ifdef CONFIG_RISCV_DMA_NONCOHERENT
+extern int dma_cache_alignment;
+#define dma_get_cache_alignment dma_get_cache_alignment
+static inline int dma_get_cache_alignment(void)
+{
+ return dma_cache_alignment;
+}
+#endif
+
+#endif /* __ASSEMBLY__ */
+
#endif /* _ASM_RISCV_CACHE_H */
diff --git a/arch/riscv/mm/dma-noncoherent.c b/arch/riscv/mm/dma-noncoherent.c
index 0e172e2b4751..21b553c299db 100644
--- a/arch/riscv/mm/dma-noncoherent.c
+++ b/arch/riscv/mm/dma-noncoherent.c
@@ -11,6 +11,8 @@
#include <asm/cacheflush.h>
static bool noncoherent_supported __ro_after_init;
+int dma_cache_alignment __ro_after_init = ARCH_DMA_MINALIGN;
+EXPORT_SYMBOL(dma_cache_alignment);
void arch_sync_dma_for_device(phys_addr_t paddr, size_t size,
enum dma_data_direction dir)
@@ -78,5 +80,7 @@ void riscv_noncoherent_supported(bool cmo)
WARN(!riscv_cbom_block_size,
"Non-coherent DMA support enabled without a block size\n");
noncoherent_supported = true;
+ } else {
+ dma_cache_alignment = 1;
}
}
--
2.40.1
On Sat, May 27, 2023 at 12:59:57AM +0800, Jisheng Zhang wrote:
> Currently, riscv defines ARCH_DMA_MINALIGN as L1_CACHE_BYTES, I.E
> 64Bytes, if CONFIG_RISCV_DMA_NONCOHERENT=y. To support unified kernel
> Image, usually we have to enable CONFIG_RISCV_DMA_NONCOHERENT, thus
> it brings some bad effects to for coherent platforms:
>
> Firstly, it wastes memory, kmalloc-96, kmalloc-32, kmalloc-16 and
> kmalloc-8 slab caches don't exist any more, they are replaced with
> either kmalloc-128 or kmalloc-64.
>
> Secondly, larger than necessary kmalloc aligned allocations results
> in unnecessary cache/TLB pressure.
>
> This issue also exists on arm64 platforms. From last year, Catalin
> tried to solve this issue by decoupling ARCH_KMALLOC_MINALIGN from
> ARCH_DMA_MINALIGN, limiting kmalloc() minimum alignment to
> dma_get_cache_alignment() and replacing ARCH_KMALLOC_MINALIGN usage
> in various drivers with ARCH_DMA_MINALIGN etc.
>
> One fact we can make use of for riscv: if the CPU doesn't support
> ZICBOM or T-HEAD CMO, we know the platform is coherent. Based on
> Catalin's work and above fact, we can easily solve the kmalloc align
> issue for riscv: we can override dma_get_cache_alignment(), then let
> it return ARCH_DMA_MINALIGN at the beginning and return 1 once we know
> the underlying HW neither supports ZICBOM nor supports T-HEAD CMO.
>
> So what about if the CPU supports ZICBOM and T-HEAD CMO, but all the
> devices are dma coherent? Well, we use ARCH_DMA_MINALIGN as the
> kmalloc minimum alignment, nothing changed in this case. This case
> can be improved in the future.
>
> After this patch, a simple test of booting to a small buildroot rootfs
> on qemu shows:
>
> kmalloc-96 5041 5041 96 ...
> kmalloc-64 9606 9606 64 ...
> kmalloc-32 5128 5128 32 ...
> kmalloc-16 7682 7682 16 ...
> kmalloc-8 10246 10246 8 ...
>
> So we save about 1268KB memory. The saving will be much larger in normal
> OS env on real HW platforms.
>
> [1] Link: https://lore.kernel.org/linux-arm-kernel/[email protected]/
>
> Signed-off-by: Jisheng Zhang <[email protected]>
Fails to build chief, with loads of:
linux/dma-mapping.h:546:19: error: redefinition of 'dma_get_cache_alignment'
And for 32-bit there's also a rake of:
include/linux/slab.h:239:9: warning: 'ARCH_KMALLOC_MINALIGN' macro redefined [-Wmacro-redefined]
At the very least, reproducable with rv32_defconfig.
Cheers,
Conor.
On Mon, May 29, 2023 at 12:17:46PM +0100, Conor Dooley wrote:
> On Sat, May 27, 2023 at 12:59:57AM +0800, Jisheng Zhang wrote:
> > After this patch, a simple test of booting to a small buildroot rootfs
> > on qemu shows:
> >
> > kmalloc-96 5041 5041 96 ...
> > kmalloc-64 9606 9606 64 ...
> > kmalloc-32 5128 5128 32 ...
> > kmalloc-16 7682 7682 16 ...
> > kmalloc-8 10246 10246 8 ...
> >
> > So we save about 1268KB memory. The saving will be much larger in normal
> > OS env on real HW platforms.
> >
> > [1] Link: https://lore.kernel.org/linux-arm-kernel/[email protected]/
> >
> > Signed-off-by: Jisheng Zhang <[email protected]>
>
> Fails to build chief, with loads of:
> linux/dma-mapping.h:546:19: error: redefinition of 'dma_get_cache_alignment'
>
> And for 32-bit there's also a rake of:
> include/linux/slab.h:239:9: warning: 'ARCH_KMALLOC_MINALIGN' macro redefined [-Wmacro-redefined]
>
> At the very least, reproducable with rv32_defconfig.
Have you this it on top of the KMALLOC_MINALIGN preparation series?
https://lore.kernel.org/r/[email protected]/
--
Catalin
On Tue, May 30, 2023 at 10:59:41AM +0100, Catalin Marinas wrote:
> On Mon, May 29, 2023 at 12:17:46PM +0100, Conor Dooley wrote:
> > On Sat, May 27, 2023 at 12:59:57AM +0800, Jisheng Zhang wrote:
> > > After this patch, a simple test of booting to a small buildroot rootfs
> > > on qemu shows:
> > >
> > > kmalloc-96 5041 5041 96 ...
> > > kmalloc-64 9606 9606 64 ...
> > > kmalloc-32 5128 5128 32 ...
> > > kmalloc-16 7682 7682 16 ...
> > > kmalloc-8 10246 10246 8 ...
> > >
> > > So we save about 1268KB memory. The saving will be much larger in normal
> > > OS env on real HW platforms.
> > >
> > > [1] Link: https://lore.kernel.org/linux-arm-kernel/[email protected]/
While I think of it, Link: goes at the start of the line, the [1] should
go at the end (although I don't think you actually reference the link
anywhere in the text & it'll probably not be particularly relevant if a
subsequent revision of that patchset is applied.
> > >
> > > Signed-off-by: Jisheng Zhang <[email protected]>
> >
> > Fails to build chief, with loads of:
> > linux/dma-mapping.h:546:19: error: redefinition of 'dma_get_cache_alignment'
> >
> > And for 32-bit there's also a rake of:
> > include/linux/slab.h:239:9: warning: 'ARCH_KMALLOC_MINALIGN' macro redefined [-Wmacro-redefined]
> >
> > At the very least, reproducable with rv32_defconfig.
>
> Have you this it on top of the KMALLOC_MINALIGN preparation series?
>
> https://lore.kernel.org/r/[email protected]/
Oh, no. Thanks for pointing that out.
Our automation stuff only uses what is in riscv/{for-next,master,fixes}.
Unless my reading comprehension is particularly bad of late it was
non-obvious that this depended on something that had not yet been
applied - it sounded like your series had already been merged last year.
Apologies for the noise then on this patch, but please try to be more
clear about what the dependencies actually are Jisheng.
Cheers,
Conor.
On Tue, May 30, 2023 at 11:34:06AM +0100, Conor Dooley wrote:
> On Tue, May 30, 2023 at 10:59:41AM +0100, Catalin Marinas wrote:
> > On Mon, May 29, 2023 at 12:17:46PM +0100, Conor Dooley wrote:
> > > On Sat, May 27, 2023 at 12:59:57AM +0800, Jisheng Zhang wrote:
> > > > After this patch, a simple test of booting to a small buildroot rootfs
> > > > on qemu shows:
> > > >
> > > > kmalloc-96 5041 5041 96 ...
> > > > kmalloc-64 9606 9606 64 ...
> > > > kmalloc-32 5128 5128 32 ...
> > > > kmalloc-16 7682 7682 16 ...
> > > > kmalloc-8 10246 10246 8 ...
> > > >
> > > > So we save about 1268KB memory. The saving will be much larger in normal
> > > > OS env on real HW platforms.
> > > >
> > > > [1] Link: https://lore.kernel.org/linux-arm-kernel/[email protected]/
>
> While I think of it, Link: goes at the start of the line, the [1] should
> go at the end (although I don't think you actually reference the link
> anywhere in the text & it'll probably not be particularly relevant if a
> subsequent revision of that patchset is applied.
I plan to post at least one more. I'd suggest the risc-v patchset to
only go in once my series landed.
> > > > Signed-off-by: Jisheng Zhang <[email protected]>
> > >
> > > Fails to build chief, with loads of:
> > > linux/dma-mapping.h:546:19: error: redefinition of 'dma_get_cache_alignment'
> > >
> > > And for 32-bit there's also a rake of:
> > > include/linux/slab.h:239:9: warning: 'ARCH_KMALLOC_MINALIGN' macro redefined [-Wmacro-redefined]
> > >
> > > At the very least, reproducable with rv32_defconfig.
> >
> > Have you this it on top of the KMALLOC_MINALIGN preparation series?
> >
> > https://lore.kernel.org/r/[email protected]/
>
> Oh, no. Thanks for pointing that out.
> Our automation stuff only uses what is in riscv/{for-next,master,fixes}.
> Unless my reading comprehension is particularly bad of late it was
> non-obvious that this depended on something that had not yet been
> applied - it sounded like your series had already been merged last year.
Yeah, it was only obvious to me since it was my series ;).
--
Catalin
On Tue, May 30, 2023 at 02:08:10PM +0100, Catalin Marinas wrote:
> On Tue, May 30, 2023 at 11:34:06AM +0100, Conor Dooley wrote:
> > On Tue, May 30, 2023 at 10:59:41AM +0100, Catalin Marinas wrote:
> > > On Mon, May 29, 2023 at 12:17:46PM +0100, Conor Dooley wrote:
> > > > On Sat, May 27, 2023 at 12:59:57AM +0800, Jisheng Zhang wrote:
> > > > > After this patch, a simple test of booting to a small buildroot rootfs
> > > > > on qemu shows:
> > > > >
> > > > > kmalloc-96 5041 5041 96 ...
> > > > > kmalloc-64 9606 9606 64 ...
> > > > > kmalloc-32 5128 5128 32 ...
> > > > > kmalloc-16 7682 7682 16 ...
> > > > > kmalloc-8 10246 10246 8 ...
> > > > >
> > > > > So we save about 1268KB memory. The saving will be much larger in normal
> > > > > OS env on real HW platforms.
> > > > >
> > > > > [1] Link: https://lore.kernel.org/linux-arm-kernel/[email protected]/
> >
> > While I think of it, Link: goes at the start of the line, the [1] should
> > go at the end (although I don't think you actually reference the link
> > anywhere in the text & it'll probably not be particularly relevant if a
> > subsequent revision of that patchset is applied.
>
> I plan to post at least one more. I'd suggest the risc-v patchset to
> only go in once my series landed.
Sure I will wait for your series landing in linus tree firstly.
>
> > > > > Signed-off-by: Jisheng Zhang <[email protected]>
> > > >
> > > > Fails to build chief, with loads of:
> > > > linux/dma-mapping.h:546:19: error: redefinition of 'dma_get_cache_alignment'
> > > >
> > > > And for 32-bit there's also a rake of:
> > > > include/linux/slab.h:239:9: warning: 'ARCH_KMALLOC_MINALIGN' macro redefined [-Wmacro-redefined]
> > > >
> > > > At the very least, reproducable with rv32_defconfig.
> > >
> > > Have you this it on top of the KMALLOC_MINALIGN preparation series?
> > >
> > > https://lore.kernel.org/r/[email protected]/
> >
> > Oh, no. Thanks for pointing that out.
> > Our automation stuff only uses what is in riscv/{for-next,master,fixes}.
> > Unless my reading comprehension is particularly bad of late it was
Aha I dunno this mechanism before.
> > non-obvious that this depended on something that had not yet been
Your reading comprehension is good ;) I just listed the dependency but
didn't explictly mention its merge status.
I will wait for Catalin's series being merged.