Received: by 2002:ac0:a5a6:0:0:0:0:0 with SMTP id m35-v6csp3729745imm; Wed, 5 Sep 2018 05:12:48 -0700 (PDT) X-Google-Smtp-Source: ANB0Vdam8EFZUQY3ALyym5vW1vEnxJKTlUijuL8sdRc+/H6yDX3BrDAuCDtdWu2v5rGKmOVDqK89 X-Received: by 2002:a17:902:127:: with SMTP id 36-v6mr39429792plb.178.1536149568792; Wed, 05 Sep 2018 05:12:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536149568; cv=none; d=google.com; s=arc-20160816; b=Ays6ZV9FTXJWu3ocmYzO5gR8RFO1/jYGFmZVw7X/kEfH78E62XrrIWPID4a/83YKYA 1MUftI/P14Y5RO2cjgKDg933EyjNVdtONeYpmAC4yCsFTgSdRW+0JWeELPoYaFcs+I2h 1gW+kIXiZYc4NvNKh7Ivbi3Hn+5heXlinOoywDpDMd/IGmo04QVJvV7qNUWv3+bQQUHq GM20k9H+CJN6AMlVen490mPgJR4bW45XKGqKVJvzfEl6PGLjRE41X/9qBuMxCqL64jpK UIbWNQ70BoEXVVz81dcLPBQTWkghm49weV+xEZETw5gwavTLwfPEb5eIpXY9fiFika4G fRFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:references :in-reply-to:message-id:date:subject:cc:to:from; bh=YYpWL0hx1mvXed70NWWbPXn0AF6z9fwio2pTmnKXWkI=; b=qer63Fi5e1HY+rfWtD1FB6utdyj4wn74a8CqvTOvzV6YDo0ji/Aehujbw5McEONaUW ECvTwE1BeiQ0L29/f26zzX7iiXMpjvZJmed3NtOJJDWAh5m1z1KXafPxkYNhYketLb34 071wXzRVlbMfEjfZb5xTIoH3rTCeBY1gRHguf60FY49JGt6VtrAIzLr2wRYtG0VhQPY0 I0tipEj7Hde87A7sAObByXiMu1wy1K3Fve+NDq2zP8OKLeUKrcKr+BI0OYnx0EjdO4Ps L2Bggn5XBHbhqnbaLABJAoqit8FXS0ibDMOXRraqQ0OQLLCPuKM6n1+x/I9gJmmazGyQ VF9Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id 34-v6si1874802plm.205.2018.09.05.05.12.20; Wed, 05 Sep 2018 05:12:48 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727776AbeIEQic (ORCPT + 99 others); Wed, 5 Sep 2018 12:38:32 -0400 Received: from smtp2200-217.mail.aliyun.com ([121.197.200.217]:46550 "EHLO smtp2200-217.mail.aliyun.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727104AbeIEQic (ORCPT ); Wed, 5 Sep 2018 12:38:32 -0400 X-Alimail-AntiSpam: AC=CONTINUE;BC=0.07437673|-1;CH=green;FP=0|0|0|0|0|-1|-1|-1;HT=e01e01546;MF=ren_guo@c-sky.com;NM=1;PH=DS;RN=12;RT=12;SR=0;TI=SMTPD_---.CmssqGg_1536149297; Received: from localhost(mailfrom:ren_guo@c-sky.com fp:SMTPD_---.CmssqGg_1536149297) by smtp.aliyun-inc.com(10.147.41.158); Wed, 05 Sep 2018 20:08:17 +0800 From: Guo Ren To: linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org, tglx@linutronix.de, daniel.lezcano@linaro.org, jason@lakedaemon.net, arnd@arndb.de Cc: c-sky_gcc_upstream@c-sky.com, gnu-csky@mentor.com, thomas.petazzoni@bootlin.com, wbx@uclibc-ng.org, ren_guo@c-sky.com, green.hu@gmail.com Subject: [PATCH V3 06/26] csky: Cache and TLB routines Date: Wed, 5 Sep 2018 20:07:45 +0800 Message-Id: <16105a3e54f1c4bb65a5ec81d77af7c176e705c6.1536138304.git.ren_guo@c-sky.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: References: In-Reply-To: References: Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Signed-off-by: Guo Ren --- arch/csky/abiv1/cacheflush.c | 50 ++++++++ arch/csky/abiv1/inc/abi/cacheflush.h | 41 +++++++ arch/csky/abiv1/inc/abi/tlb.h | 11 ++ arch/csky/abiv2/cacheflush.c | 54 +++++++++ arch/csky/abiv2/inc/abi/cacheflush.h | 38 ++++++ arch/csky/abiv2/inc/abi/tlb.h | 12 ++ arch/csky/include/asm/barrier.h | 45 +++++++ arch/csky/include/asm/cache.h | 28 +++++ arch/csky/include/asm/cacheflush.h | 8 ++ arch/csky/include/asm/io.h | 23 ++++ arch/csky/include/asm/tlb.h | 19 +++ arch/csky/include/asm/tlbflush.h | 22 ++++ arch/csky/include/uapi/asm/cachectl.h | 13 +++ arch/csky/mm/cachev1.c | 126 ++++++++++++++++++++ arch/csky/mm/cachev2.c | 79 +++++++++++++ arch/csky/mm/syscache.c | 28 +++++ arch/csky/mm/tlb.c | 214 ++++++++++++++++++++++++++++++++++ 17 files changed, 811 insertions(+) create mode 100644 arch/csky/abiv1/cacheflush.c create mode 100644 arch/csky/abiv1/inc/abi/cacheflush.h create mode 100644 arch/csky/abiv1/inc/abi/tlb.h create mode 100644 arch/csky/abiv2/cacheflush.c create mode 100644 arch/csky/abiv2/inc/abi/cacheflush.h create mode 100644 arch/csky/abiv2/inc/abi/tlb.h create mode 100644 arch/csky/include/asm/barrier.h create mode 100644 arch/csky/include/asm/cache.h create mode 100644 arch/csky/include/asm/cacheflush.h create mode 100644 arch/csky/include/asm/io.h create mode 100644 arch/csky/include/asm/tlb.h create mode 100644 arch/csky/include/asm/tlbflush.h create mode 100644 arch/csky/include/uapi/asm/cachectl.h create mode 100644 arch/csky/mm/cachev1.c create mode 100644 arch/csky/mm/cachev2.c create mode 100644 arch/csky/mm/syscache.c create mode 100644 arch/csky/mm/tlb.c diff --git a/arch/csky/abiv1/cacheflush.c b/arch/csky/abiv1/cacheflush.c new file mode 100644 index 0000000..4c6fede --- /dev/null +++ b/arch/csky/abiv1/cacheflush.c @@ -0,0 +1,50 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2018 Hangzhou C-SKY Microsystems co.,ltd. +#include +#include +#include +#include +#include +#include +#include +#include +#include + +void flush_dcache_page(struct page *page) +{ + struct address_space *mapping = page_mapping(page); + unsigned long addr; + + if (mapping && !mapping_mapped(mapping)) { + set_bit(PG_arch_1, &(page)->flags); + return; + } + + /* + * We could delay the flush for the !page_mapping case too. But that + * case is for exec env/arg pages and those are %99 certainly going to + * get faulted into the tlb (and thus flushed) anyways. + */ + addr = (unsigned long) page_address(page); + dcache_wb_range(addr, addr + PAGE_SIZE); +} + +void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, pte_t *pte) +{ + unsigned long addr; + struct page *page; + unsigned long pfn; + + pfn = pte_pfn(*pte); + if (unlikely(!pfn_valid(pfn))) + return; + + page = pfn_to_page(pfn); + addr = (unsigned long) page_address(page); + + if (vma->vm_flags & VM_EXEC || + pages_do_alias(addr, address & PAGE_MASK)) + cache_wbinv_all(); + + clear_bit(PG_arch_1, &(page)->flags); +} diff --git a/arch/csky/abiv1/inc/abi/cacheflush.h b/arch/csky/abiv1/inc/abi/cacheflush.h new file mode 100644 index 0000000..ba5071e --- /dev/null +++ b/arch/csky/abiv1/inc/abi/cacheflush.h @@ -0,0 +1,41 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2018 Hangzhou C-SKY Microsystems co.,ltd. +#ifndef __ABI_CSKY_CACHEFLUSH_H +#define __ABI_CSKY_CACHEFLUSH_H + +#include +#include +#include + +#define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 1 +extern void flush_dcache_page(struct page *); + +#define flush_cache_mm(mm) cache_wbinv_all() +#define flush_cache_page(vma,page,pfn) cache_wbinv_all() +#define flush_cache_dup_mm(mm) cache_wbinv_all() + +#define flush_cache_range(mm,start,end) cache_wbinv_range(start, end) +#define flush_cache_vmap(start, end) cache_wbinv_range(start, end) +#define flush_cache_vunmap(start, end) cache_wbinv_range(start, end) + +#define flush_icache_page(vma, page) cache_wbinv_all() +#define flush_icache_range(start, end) cache_wbinv_range(start, end) +#define flush_icache_user_range(vma,pg,adr,len) cache_wbinv_range(adr, adr + len) + +#define copy_from_user_page(vma, page, vaddr, dst, src, len) \ +do{ \ + cache_wbinv_all(); \ + memcpy(dst, src, len); \ + icache_inv_all(); \ +}while(0) + +#define copy_to_user_page(vma, page, vaddr, dst, src, len) \ +do{ \ + cache_wbinv_all(); \ + memcpy(dst, src, len); \ +}while(0) + +#define flush_dcache_mmap_lock(mapping) do{}while(0) +#define flush_dcache_mmap_unlock(mapping) do{}while(0) + +#endif /* __ABI_CSKY_CACHEFLUSH_H */ diff --git a/arch/csky/abiv1/inc/abi/tlb.h b/arch/csky/abiv1/inc/abi/tlb.h new file mode 100644 index 0000000..cf0c8d9 --- /dev/null +++ b/arch/csky/abiv1/inc/abi/tlb.h @@ -0,0 +1,11 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2018 Hangzhou C-SKY Microsystems co.,ltd. +#ifndef __ABI_CSKY_TLB_H +#define __ABI_CSKY_TLB_H + +#define tlb_start_vma(tlb, vma) \ + do { \ + if (!tlb->fullmm) \ + cache_wbinv_all(); \ + } while (0) +#endif /* __ABI_CSKY_TLB_H */ diff --git a/arch/csky/abiv2/cacheflush.c b/arch/csky/abiv2/cacheflush.c new file mode 100644 index 0000000..794e9ce --- /dev/null +++ b/arch/csky/abiv2/cacheflush.c @@ -0,0 +1,54 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2018 Hangzhou C-SKY Microsystems co.,ltd. +#include +#include +#include +#include + +void flush_icache_page(struct vm_area_struct *vma, struct page *page) +{ + void *va; + unsigned long start, end; + + va = page_address(page); + start = (unsigned long) va; + + if (va == NULL && PageHighMem(page)) + start = (unsigned long) kmap_atomic(page); + + end = start + PAGE_SIZE; + + cache_wbinv_range(start, end); + + if (va == NULL && PageHighMem(page)) + kunmap_atomic((void *)start); +} + +void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, pte_t *pte) +{ + unsigned long addr, pfn; + struct page *page; + void *va; + + if (!(vma->vm_flags & VM_EXEC)) + return; + + pfn = pte_pfn(*pte); + if (unlikely(!pfn_valid(pfn))) + return; + + page = pfn_to_page(pfn); + if (page == ZERO_PAGE(0)) + return; + + va = page_address(page); + addr = (unsigned long) va; + + if (va == NULL && PageHighMem(page)) + addr = (unsigned long) kmap_atomic(page); + + cache_wbinv_range(addr, addr + PAGE_SIZE); + + if (va == NULL && PageHighMem(page)) + kunmap_atomic((void *) addr); +} diff --git a/arch/csky/abiv2/inc/abi/cacheflush.h b/arch/csky/abiv2/inc/abi/cacheflush.h new file mode 100644 index 0000000..696ca87 --- /dev/null +++ b/arch/csky/abiv2/inc/abi/cacheflush.h @@ -0,0 +1,38 @@ +// SPDX-License-Identifier: GPL-2.0 + +#ifndef __ABI_CSKY_CACHEFLUSH_H +#define __ABI_CSKY_CACHEFLUSH_H + +/* Keep includes the same across arches. */ +#include + +/* + * The cache doesn't need to be flushed when TLB entries change when + * the cache is mapped to physical memory, not virtual memory + */ +#define flush_cache_all() do { } while (0) +#define flush_cache_mm(mm) do { } while (0) +#define flush_cache_dup_mm(mm) do { } while (0) +#define flush_cache_range(vma, start, end) do { } while (0) +#define flush_cache_page(vma, vmaddr, pfn) do { } while (0) +#define ARCH_IMPLEMENTS_FLUSH_DCACHE_PAGE 0 +#define flush_dcache_page(page) do { } while (0) +#define flush_dcache_mmap_lock(mapping) do { } while (0) +#define flush_dcache_mmap_unlock(mapping) do { } while (0) + +#define flush_icache_range(start, end) cache_wbinv_range(start, end) +#define flush_icache_user_range(vma,pg,adr,len) cache_wbinv_range(adr, adr + len) +void flush_icache_page(struct vm_area_struct *vma, struct page *page); + +#define flush_cache_vmap(start, end) do { } while (0) +#define flush_cache_vunmap(start, end) do { } while (0) + +#define copy_to_user_page(vma, page, vaddr, dst, src, len) \ + do { \ + memcpy(dst, src, len); \ + flush_icache_user_range(vma, page, vaddr, len); \ + } while (0) +#define copy_from_user_page(vma, page, vaddr, dst, src, len) \ + memcpy(dst, src, len) + +#endif /* __ABI_CSKY_CACHEFLUSH_H */ diff --git a/arch/csky/abiv2/inc/abi/tlb.h b/arch/csky/abiv2/inc/abi/tlb.h new file mode 100644 index 0000000..0faa4e9 --- /dev/null +++ b/arch/csky/abiv2/inc/abi/tlb.h @@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2018 Hangzhou C-SKY Microsystems co.,ltd. +#ifndef __ABI_CSKY_TLB_H +#define __ABI_CSKY_TLB_H + +#define tlb_start_vma(tlb, vma) \ + do { \ + if (!tlb->fullmm && (vma->vm_flags & VM_EXEC)) \ + icache_inv_all(); \ + } while (0) + +#endif /* __ABI_CSKY_TLB_H */ diff --git a/arch/csky/include/asm/barrier.h b/arch/csky/include/asm/barrier.h new file mode 100644 index 0000000..ecf4a02 --- /dev/null +++ b/arch/csky/include/asm/barrier.h @@ -0,0 +1,45 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2018 Hangzhou C-SKY Microsystems co.,ltd. + +#ifndef __ASM_CSKY_BARRIER_H +#define __ASM_CSKY_BARRIER_H + +#ifndef __ASSEMBLY__ + +#define nop() asm volatile ("nop") + +/* + * sync: completion barrier + * sync.s: completion barrier and shareable to other cores + * sync.i: completion barrier with flush cpu pipeline + * sync.is: completion barrier with flush cpu pipeline and shareable to other cores + * + * bar.brwarw: ordering barrier for all load/store instructions before it + * bar.brwarws: ordering barrier for all load/store instructions before it and shareable to other cores + * bar.brar: ordering barrier for all load instructions before it + * bar.brars: ordering barrier for all load instructions before it and shareable to other cores + * bar.bwaw: ordering barrier for all store instructions before it + * bar.bwaws: ordering barrier for all store instructions before it and shareable to other cores + */ + +#ifdef CONFIG_CPU_HAS_CACHEV2 +#define mb() asm volatile ("bar.brwarw":::"memory") +#define rmb() asm volatile ("bar.brar":::"memory") +#define wmb() asm volatile ("bar.bwaw":::"memory") + +#ifdef CONFIG_SMP +#define __smp_mb() asm volatile ("bar.brwarws":::"memory") +#define __smp_rmb() asm volatile ("bar.brars":::"memory") +#define __smp_wmb() asm volatile ("bar.bwaws":::"memory") +#endif /* CONFIG_SMP */ + +#define sync_is() asm volatile ("sync.is":::"memory") + +#else /* !CONFIG_CPU_HAS_CACHEV2 */ +#define mb() asm volatile ("sync":::"memory") +#endif + +#include + +#endif /* __ASSEMBLY__ */ +#endif /* __ASM_CSKY_BARRIER_H */ diff --git a/arch/csky/include/asm/cache.h b/arch/csky/include/asm/cache.h new file mode 100644 index 0000000..9bb3cca --- /dev/null +++ b/arch/csky/include/asm/cache.h @@ -0,0 +1,28 @@ +#ifndef __ASM_CSKY_CACHE_H +#define __ASM_CSKY_CACHE_H + +/* bytes per L1 cache line */ +#define L1_CACHE_SHIFT CONFIG_L1_CACHE_SHIFT + +#define L1_CACHE_BYTES (1 << L1_CACHE_SHIFT) + +#define ARCH_DMA_MINALIGN L1_CACHE_BYTES + +#ifndef __ASSEMBLY__ + +void dcache_wb_line(unsigned long start); + +void icache_inv_range(unsigned long start, unsigned long end); +void icache_inv_all(void); + +void dcache_wb_range(unsigned long start, unsigned long end); +void dcache_wbinv_all(void); + +void cache_wbinv_range(unsigned long start, unsigned long end); +void cache_wbinv_all(void); + +void dma_wbinv_range(unsigned long start, unsigned long end); +void dma_wb_range(unsigned long start, unsigned long end); + +#endif +#endif /* __ASM_CSKY_CACHE_H */ diff --git a/arch/csky/include/asm/cacheflush.h b/arch/csky/include/asm/cacheflush.h new file mode 100644 index 0000000..297fd9c --- /dev/null +++ b/arch/csky/include/asm/cacheflush.h @@ -0,0 +1,8 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2018 Hangzhou C-SKY Microsystems co.,ltd. +#ifndef __ASM_CSKY_CACHEFLUSH_H +#define __ASM_CSKY_CACHEFLUSH_H + +#include + +#endif /* __ASM_CSKY_CACHEFLUSH_H */ diff --git a/arch/csky/include/asm/io.h b/arch/csky/include/asm/io.h new file mode 100644 index 0000000..fcb2142 --- /dev/null +++ b/arch/csky/include/asm/io.h @@ -0,0 +1,23 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2018 Hangzhou C-SKY Microsystems co.,ltd. +#ifndef __ASM_CSKY_IO_H +#define __ASM_CSKY_IO_H + +#include +#include +#include + +extern void __iomem *ioremap(phys_addr_t offset, size_t size); + +extern void iounmap(void *addr); + +extern int remap_area_pages(unsigned long address, phys_addr_t phys_addr, + size_t size, unsigned long flags); + +#define ioremap_nocache(phy, sz) ioremap(phy, sz) +#define ioremap_wc ioremap_nocache +#define ioremap_wt ioremap_nocache + +#include + +#endif /* __ASM_CSKY_IO_H */ diff --git a/arch/csky/include/asm/tlb.h b/arch/csky/include/asm/tlb.h new file mode 100644 index 0000000..b2ea919 --- /dev/null +++ b/arch/csky/include/asm/tlb.h @@ -0,0 +1,19 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2018 Hangzhou C-SKY Microsystems co.,ltd. +#ifndef __ASM_CSKY_TLB_H +#define __ASM_CSKY_TLB_H + +#include +#include + +#define tlb_end_vma(tlb, vma) \ + do { \ + if (!tlb->fullmm) \ + flush_tlb_range(vma, vma->vm_start, vma->vm_end); \ + } while (0) + +#define tlb_flush(tlb) flush_tlb_mm((tlb)->mm) + +#include + +#endif /* __ASM_CSKY_TLB_H */ diff --git a/arch/csky/include/asm/tlbflush.h b/arch/csky/include/asm/tlbflush.h new file mode 100644 index 0000000..d44cb07 --- /dev/null +++ b/arch/csky/include/asm/tlbflush.h @@ -0,0 +1,22 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2018 Hangzhou C-SKY Microsystems co.,ltd. +#ifndef __ASM_TLBFLUSH_H +#define __ASM_TLBFLUSH_H +/* + * TLB flushing: + * + * - flush_tlb_all() flushes all processes TLB entries + * - flush_tlb_mm(mm) flushes the specified mm context TLB entries + * - flush_tlb_page(vma, vmaddr) flushes one page + * - flush_tlb_range(vma, start, end) flushes a range of pages + * - flush_tlb_kernel_range(start, end) flushes a range of kernel pages + */ +extern void flush_tlb_all(void); +extern void flush_tlb_mm(struct mm_struct *mm); +extern void flush_tlb_page(struct vm_area_struct *vma, unsigned long page); +extern void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); +extern void flush_tlb_kernel_range(unsigned long start, unsigned long end); + +extern void flush_tlb_one(unsigned long vaddr); + +#endif diff --git a/arch/csky/include/uapi/asm/cachectl.h b/arch/csky/include/uapi/asm/cachectl.h new file mode 100644 index 0000000..38a1a28 --- /dev/null +++ b/arch/csky/include/uapi/asm/cachectl.h @@ -0,0 +1,13 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2018 Hangzhou C-SKY Microsystems co.,ltd. +#ifndef __ASM_CSKY_CACHECTL_H +#define __ASM_CSKY_CACHECTL_H + +/* + * See "man cacheflush" + */ +#define ICACHE (1<<0) +#define DCACHE (1<<1) +#define BCACHE (ICACHE|DCACHE) + +#endif /* __ASM_CSKY_CACHECTL_H */ diff --git a/arch/csky/mm/cachev1.c b/arch/csky/mm/cachev1.c new file mode 100644 index 0000000..c50e9a2 --- /dev/null +++ b/arch/csky/mm/cachev1.c @@ -0,0 +1,126 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2018 Hangzhou C-SKY Microsystems co.,ltd. +#include +#include +#include + +/* for L1-cache */ +#define INS_CACHE (1 << 0) +#define DATA_CACHE (1 << 1) +#define CACHE_INV (1 << 4) +#define CACHE_CLR (1 << 5) +#define CACHE_OMS (1 << 6) +#define CACHE_ITS (1 << 7) +#define CACHE_LICF (1 << 31) + +/* for L2-cache */ +#define CR22_LEVEL_SHIFT (1) +#define CR22_SET_SHIFT (7) +#define CR22_WAY_SHIFT (30) +#define CR22_WAY_SHIFT_L2 (29) + +#define SYNC asm volatile("sync\n") + +static DEFINE_SPINLOCK(cache_lock); + +static inline void cache_op_line(unsigned long i, unsigned int val) +{ + mtcr("cr22", i); + mtcr("cr17", val); +} + +#define CCR2_L2E (1 << 3) +static void cache_op_all(unsigned int value, unsigned int l2) +{ + mtcr("cr17", value | CACHE_CLR); + SYNC; + + if (l2 && (mfcr_ccr2() & CCR2_L2E)) { + mtcr("cr24", value | CACHE_CLR); + SYNC; + } +} + +static void cache_op_range( + unsigned int start, + unsigned int end, + unsigned int value, + unsigned int l2) +{ + unsigned long i, flags; + unsigned int val = value | CACHE_CLR | CACHE_OMS; + bool l2_sync; + + if (unlikely((end - start) >= PAGE_SIZE) || + unlikely(start < PAGE_OFFSET) || + unlikely(start >= PAGE_OFFSET + LOWMEM_LIMIT)) { + cache_op_all(value, l2); + return; + } + + if ((mfcr_ccr2() & CCR2_L2E) && l2) + l2_sync = 1; + else + l2_sync = 0; + + spin_lock_irqsave(&cache_lock, flags); + + i = start & ~(L1_CACHE_BYTES - 1); + for(; i < end; i += L1_CACHE_BYTES) { + cache_op_line(i, val); + if (l2_sync) { + SYNC; + mtcr("cr24", val); + } + } + spin_unlock_irqrestore(&cache_lock, flags); + + SYNC; +} + +void dcache_wb_line(unsigned long start) +{ + asm volatile("idly4\n":::"memory"); + cache_op_line(start, DATA_CACHE|CACHE_CLR); + SYNC; +} + +void icache_inv_range(unsigned long start, unsigned long end) +{ + cache_op_range(start, end, INS_CACHE|CACHE_INV, 0); +} + +void icache_inv_all(void) +{ + cache_op_all(INS_CACHE|CACHE_INV, 0); +} + +void dcache_wb_range(unsigned long start, unsigned long end) +{ + cache_op_range(start, end, DATA_CACHE|CACHE_CLR, 0); +} + +void dcache_wbinv_all(void) +{ + cache_op_all(DATA_CACHE|CACHE_CLR|CACHE_INV, 0); +} + +void cache_wbinv_range(unsigned long start, unsigned long end) +{ + cache_op_range(start, end, INS_CACHE|DATA_CACHE|CACHE_CLR|CACHE_INV, 0); +} + +void cache_wbinv_all(void) +{ + cache_op_all(INS_CACHE|DATA_CACHE|CACHE_CLR|CACHE_INV, 0); +} + +void dma_wbinv_range(unsigned long start, unsigned long end) +{ + cache_op_range(start, end, DATA_CACHE|CACHE_CLR|CACHE_INV, 1); +} + +void dma_wb_range(unsigned long start, unsigned long end) +{ + cache_op_range(start, end, DATA_CACHE|CACHE_INV, 1); +} diff --git a/arch/csky/mm/cachev2.c b/arch/csky/mm/cachev2.c new file mode 100644 index 0000000..9cd1363 --- /dev/null +++ b/arch/csky/mm/cachev2.c @@ -0,0 +1,79 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2018 Hangzhou C-SKY Microsystems co.,ltd. +#include +#include +#include +#include + +void inline dcache_wb_line(unsigned long start) +{ + asm volatile("dcache.cval1 %0\n"::"r"(start):"memory"); + sync_is(); +} + +void icache_inv_range(unsigned long start, unsigned long end) +{ + unsigned long i = start & ~(L1_CACHE_BYTES - 1); + + for (;i < end; i += L1_CACHE_BYTES) + asm volatile("icache.iva %0\n"::"r"(i):"memory"); + sync_is(); +} + +void icache_inv_all(void) +{ + asm volatile("icache.ialls\n":::"memory"); + sync_is(); +} + +void dcache_wb_range(unsigned long start, unsigned long end) +{ + unsigned long i = start & ~(L1_CACHE_BYTES - 1); + + for (;i < end; i += L1_CACHE_BYTES) + asm volatile("dcache.cval1 %0\n"::"r"(i):"memory"); + sync_is(); +} + +void dcache_inv_range(unsigned long start, unsigned long end) +{ + unsigned long i = start & ~(L1_CACHE_BYTES - 1); + + for (;i < end; i += L1_CACHE_BYTES) + asm volatile("dcache.civa %0\n"::"r"(i):"memory"); + sync_is(); +} + +void cache_wbinv_range(unsigned long start, unsigned long end) +{ + unsigned long i = start & ~(L1_CACHE_BYTES - 1); + + for (;i < end; i += L1_CACHE_BYTES) { + asm volatile("dcache.cval1 %0\n"::"r"(i):"memory"); + } + sync_is(); + + i = start & ~(L1_CACHE_BYTES - 1); + for (;i < end; i += L1_CACHE_BYTES) { + asm volatile("icache.iva %0\n"::"r"(i):"memory"); + } + sync_is(); +} + +void dma_wbinv_range(unsigned long start, unsigned long end) +{ + unsigned long i = start & ~(L1_CACHE_BYTES - 1); + + for (; i < end; i += L1_CACHE_BYTES) + asm volatile("dcache.civa %0\n"::"r"(i):"memory"); + sync_is(); +} + +void dma_wb_range(unsigned long start, unsigned long end) +{ + unsigned long i = start & ~(L1_CACHE_BYTES - 1); + + for (; i < end; i += L1_CACHE_BYTES) + asm volatile("dcache.civa %0\n"::"r"(i):"memory"); + sync_is(); +} diff --git a/arch/csky/mm/syscache.c b/arch/csky/mm/syscache.c new file mode 100644 index 0000000..ae91704 --- /dev/null +++ b/arch/csky/mm/syscache.c @@ -0,0 +1,28 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2018 Hangzhou C-SKY Microsystems co.,ltd. +#include +#include +#include +#include + +SYSCALL_DEFINE3(cacheflush, + void __user *, addr, + unsigned long, bytes, + int, cache) +{ + switch(cache) { + case ICACHE: + icache_inv_range((unsigned long)addr, (unsigned long)addr + bytes); + break; + case DCACHE: + dcache_wb_range((unsigned long)addr, (unsigned long)addr + bytes); + break; + case BCACHE: + cache_wbinv_range((unsigned long)addr, (unsigned long)addr + bytes); + break; + default: + return -EINVAL; + } + + return 0; +} diff --git a/arch/csky/mm/tlb.c b/arch/csky/mm/tlb.c new file mode 100644 index 0000000..37d94c7 --- /dev/null +++ b/arch/csky/mm/tlb.c @@ -0,0 +1,214 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright (C) 2018 Hangzhou C-SKY Microsystems co.,ltd. +#include +#include +#include +#include + +#include +#include +#include + +#define CSKY_TLB_SIZE CONFIG_CPU_TLB_SIZE + +void flush_tlb_all(void) +{ + tlb_invalid_all(); +} + +void flush_tlb_mm(struct mm_struct *mm) +{ + int cpu = smp_processor_id(); + + if (cpu_context(cpu, mm) != 0) + drop_mmu_context(mm,cpu); + + tlb_invalid_all(); +} + +#define restore_asid_inv_utlb(oldpid, newpid) \ +do { \ + if((oldpid & ASID_MASK) == newpid) \ + write_mmu_entryhi(oldpid +1); \ + write_mmu_entryhi(oldpid); \ +} while(0) + +void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, + unsigned long end) +{ + struct mm_struct *mm = vma->vm_mm; + int cpu = smp_processor_id(); + + if (cpu_context(cpu, mm) != 0) { + unsigned long size, flags; + int newpid = cpu_asid(cpu, mm); + + local_irq_save(flags); + size = (end - start + (PAGE_SIZE - 1)) >> PAGE_SHIFT; + size = (size + 1) >> 1; + if (size <= CSKY_TLB_SIZE/2) { + start &= (PAGE_MASK << 1); + end += ((PAGE_SIZE << 1) - 1); + end &= (PAGE_MASK << 1); +#ifdef CONFIG_CPU_HAS_TLBI + while (start < end) { + asm volatile("tlbi.vaas %0"::"r"(start | newpid)); + start += (PAGE_SIZE << 1); + } + sync_is(); +#else + { + int oldpid = read_mmu_entryhi(); + + while (start < end) { + int idx; + + write_mmu_entryhi(start | newpid); + start += (PAGE_SIZE << 1); + tlb_probe(); + idx = read_mmu_index(); + if (idx >= 0) + tlb_invalid_indexed(); + } + restore_asid_inv_utlb(oldpid, newpid); + } +#endif + } else { + drop_mmu_context(mm, cpu); + } + local_irq_restore(flags); + } +} + +void flush_tlb_kernel_range(unsigned long start, unsigned long end) +{ + unsigned long size, flags; + + local_irq_save(flags); + size = (end - start + (PAGE_SIZE - 1)) >> PAGE_SHIFT; + if (size <= CSKY_TLB_SIZE) { + start &= (PAGE_MASK << 1); + end += ((PAGE_SIZE << 1) - 1); + end &= (PAGE_MASK << 1); +#ifdef CONFIG_CPU_HAS_TLBI + while (start < end) { + asm volatile("tlbi.vaas %0"::"r"(start)); + start += (PAGE_SIZE << 1); + } + sync_is(); +#else + { + int oldpid = read_mmu_entryhi(); + while (start < end) { + int idx; + write_mmu_entryhi(start); + start += (PAGE_SIZE << 1); + tlb_probe(); + idx = read_mmu_index(); + if (idx >= 0) + tlb_invalid_indexed(); + } + restore_asid_inv_utlb(oldpid, 0); + } +#endif + } else + flush_tlb_all(); + + local_irq_restore(flags); +} + +void flush_tlb_page(struct vm_area_struct *vma, unsigned long page) +{ + int cpu = smp_processor_id(); + int newpid = cpu_asid(cpu, vma->vm_mm); + + if (!vma || cpu_context(cpu, vma->vm_mm) != 0) { + page &= (PAGE_MASK << 1); + +#ifdef CONFIG_CPU_HAS_TLBI + asm volatile("tlbi.vaas %0"::"r"(page | newpid)); + sync_is(); +#else + { + int oldpid, idx; + unsigned long flags; + local_irq_save(flags); + oldpid = read_mmu_entryhi(); + write_mmu_entryhi(page | newpid); + tlb_probe(); + idx = read_mmu_index(); + if(idx >= 0) + tlb_invalid_indexed(); + + restore_asid_inv_utlb(oldpid, newpid); + local_irq_restore(flags); + } +#endif + } +} + +/* + * Remove one kernel space TLB entry. This entry is assumed to be marked + * global so we don't do the ASID thing. + */ +void flush_tlb_one(unsigned long page) +{ + int oldpid; + oldpid = read_mmu_entryhi(); + page &= (PAGE_MASK << 1); + +#ifdef CONFIG_CPU_HAS_TLBI + page = page | (oldpid & 0xfff); + asm volatile("tlbi.vaas %0"::"r"(page)); + sync_is(); +#else + { + int idx; + unsigned long flags; + + page = page | (oldpid & 0xff); + + local_irq_save(flags); + write_mmu_entryhi(page); + tlb_probe(); + idx = read_mmu_index(); + if (idx >= 0) + tlb_invalid_indexed(); + restore_asid_inv_utlb(oldpid, oldpid); + local_irq_restore(flags); + } +#endif +} + +EXPORT_SYMBOL(flush_tlb_one); + +/* show current 32 jtlbs */ +void show_jtlb_table(void) +{ + unsigned long flags; + int entryhi, entrylo0, entrylo1; + int entry; + int oldpid; + + local_irq_save(flags); + entry = 0; + pr_info("\n\n\n"); + + oldpid = read_mmu_entryhi(); + while (entry < CSKY_TLB_SIZE) + { + write_mmu_index(entry); + tlb_read(); + entryhi = read_mmu_entryhi(); + entrylo0 = read_mmu_entrylo0(); + entrylo0 = entrylo0; + entrylo1 = read_mmu_entrylo1(); + entrylo1 = entrylo1; + pr_info("jtlb[%d]: entryhi - 0x%x; entrylo0 - 0x%x;" + " entrylo1 - 0x%x\n", + entry, entryhi, entrylo0, entrylo1); + entry++; + } + write_mmu_entryhi(oldpid); + local_irq_restore(flags); +} -- 2.7.4