Received: by 10.192.165.148 with SMTP id m20csp727757imm; Fri, 4 May 2018 05:35:42 -0700 (PDT) X-Google-Smtp-Source: AB8JxZpAmESxmGSxfpD8liJlG8UreTztER9QxAfbOMBhZe1FziiBaI3rkWFFrIsnyvRb2K8usXsh X-Received: by 2002:a17:902:14d:: with SMTP id 71-v6mr2942203plb.275.1525437342855; Fri, 04 May 2018 05:35:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1525437342; cv=none; d=google.com; s=arc-20160816; b=PTHAY8r+Ynl/rXLZThapuNB35conjvw3FTe+QVzVllm3EjrsKHIyqgHuZIrdgb2WXN I+sLqBBcaucFspCxdy2b1uRY1kTDI+fAQsoWtTYlgsb59HY+4ajkSWKFVMA/FuyIBChF TNo0BPzyk36tXn5u3bXYPJDBVxFec85LO2Aa5zG4wSKDN+ilO8eDA6VDvwYVontOPVwB XqhQXrafLjXDA5FUaV120daBGIt0fduWFO/by/AqyffHEZyRfRwG+HS58WEkvvn+ne89 VDFf9xkoP+E1eW8dsbuLZLdWNrcZWSMOMLI0dqDatyy8KXj2B8uh4jgJkZRM2Nv9t+c5 kApA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:date:cc:to:subject:from:references :in-reply-to:message-id:arc-authentication-results; bh=59qogPuPZplQLkNmpdNqIGZfnKU0gqlmLQTsfzojd/U=; b=NcUKXitKAPm/48ZBJuziOD3tGjSRLTTux2wA0g7nXyzOEWKzW+0iCyzBalFYfNWLfo qOlpnDCBxERBJYWoLD0R1Nuj4k7eoNByxm4xItOKkZc7dm3VzAtP5sGMxE1v0UROMHIe jB1bp6CiVXdUwPL24N+Xgu+E3fFi0cweiLGxhH5KaguxDptbe0hG+BsghBawKpFAZ3pq Q5ooWz72c5U8yfGihr4+YV7kHDvorT63RvUkrs2O8UTPPJcyhNK7v8puFlVmNbwxv67Q 0eCx8Equ0JVF/EwcQcDnmrog9WptSdL8oFoehgByKOpioTzWhAQV+0CftfnhRp6vOKXw DoOQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id k16-v6si17187663pli.171.2018.05.04.05.35.28; Fri, 04 May 2018 05:35:42 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751742AbeEDMeW (ORCPT + 99 others); Fri, 4 May 2018 08:34:22 -0400 Received: from pegase1.c-s.fr ([93.17.236.30]:16584 "EHLO pegase1.c-s.fr" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751684AbeEDMeS (ORCPT ); Fri, 4 May 2018 08:34:18 -0400 Received: from localhost (mailhub1-int [192.168.12.234]) by localhost (Postfix) with ESMTP id 40crzn3l2Qz9txgD; Fri, 4 May 2018 14:34:13 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at c-s.fr Received: from pegase1.c-s.fr ([192.168.12.234]) by localhost (pegase1.c-s.fr [192.168.12.234]) (amavisd-new, port 10024) with ESMTP id 8G6StAcTcA-6; Fri, 4 May 2018 14:34:13 +0200 (CEST) Received: from messagerie.si.c-s.fr (messagerie.si.c-s.fr [192.168.25.192]) by pegase1.c-s.fr (Postfix) with ESMTP id 40crzn377sz9ttC1; Fri, 4 May 2018 14:34:13 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 3B4D78B977; Fri, 4 May 2018 14:34:17 +0200 (CEST) X-Virus-Scanned: amavisd-new at c-s.fr Received: from messagerie.si.c-s.fr ([127.0.0.1]) by localhost (messagerie.si.c-s.fr [127.0.0.1]) (amavisd-new, port 10023) with ESMTP id 7KLalOvWKlsA; Fri, 4 May 2018 14:34:17 +0200 (CEST) Received: from po14934vm.idsi0.si.c-s.fr (po15451.idsi0.si.c-s.fr [172.25.231.2]) by messagerie.si.c-s.fr (Postfix) with ESMTP id 130608B975; Fri, 4 May 2018 14:34:17 +0200 (CEST) Received: by po14934vm.idsi0.si.c-s.fr (Postfix, from userid 0) id D99386CF2D; Fri, 4 May 2018 14:34:16 +0200 (CEST) Message-Id: In-Reply-To: References: From: Christophe Leroy Subject: [PATCH 13/17] powerpc/mm: Use hardware assistance in TLB handlers on the 8xx To: Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , aneesh.kumar@linux.vnet.ibm.com Cc: linux-kernel@vger.kernel.org, linuxppc-dev@lists.ozlabs.org Date: Fri, 4 May 2018 14:34:16 +0200 (CEST) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Today, on the 8xx the TLB handlers do SW tablewalk by doing all the calculation in ASM, in order to match with the Linux page table structure. The 8xx offers hardware assistance which allows significant size reduction of the TLB handlers, hence also reduces the time spent in the handlers. However, using this HW assistance implies some constraints on the page table structure: - Regardless of the main page size used (4k or 16k), the level 1 table (PGD) contains 1024 entries and each PGD entry covers a 4Mbytes area which is managed by a level 2 table (PTE) containing also 1024 entries each describing a 4k page. - 16k pages require 4 identifical entries in the L2 table - 512k pages PTE have to be spread every 128 bytes in the L2 table - 8M pages PTE are at the address pointed by the L1 entry and each 8M page require 2 identical entries in the PGD. In order to use hardware assistance, this patch does the following modifications: - Make PGD size independant of the main page size - In 16k pages mode, redefine pte_t as a struct with 4 elements, and populate those 4 elements in __set_pte_at() and pte_update() - Modify the TLB handlers to use HW assistance - Adapt the size of the hugepage tables. Signed-off-by: Christophe Leroy --- arch/powerpc/include/asm/hugetlb.h | 4 +- arch/powerpc/include/asm/nohash/32/pgtable.h | 16 +- arch/powerpc/include/asm/nohash/pgtable.h | 4 + arch/powerpc/include/asm/pgtable-types.h | 4 + arch/powerpc/kernel/head_8xx.S | 227 +++++++++------------------ arch/powerpc/mm/8xx_mmu.c | 10 +- arch/powerpc/mm/hugetlbpage.c | 12 ++ 7 files changed, 112 insertions(+), 165 deletions(-) diff --git a/arch/powerpc/include/asm/hugetlb.h b/arch/powerpc/include/asm/hugetlb.h index 78540c074d70..6d29be6bac74 100644 --- a/arch/powerpc/include/asm/hugetlb.h +++ b/arch/powerpc/include/asm/hugetlb.h @@ -77,7 +77,9 @@ static inline pte_t *hugepte_offset(hugepd_t hpd, unsigned long addr, unsigned long idx = 0; pte_t *dir = hugepd_page(hpd); -#ifndef CONFIG_PPC_FSL_BOOK3E +#ifdef CONFIG_PPC_8xx + idx = (addr & ((1UL << pdshift) - 1)) >> PAGE_SHIFT; +#elif !defined(CONFIG_PPC_FSL_BOOK3E) idx = (addr & ((1UL << pdshift) - 1)) >> hugepd_shift(hpd); #endif diff --git a/arch/powerpc/include/asm/nohash/32/pgtable.h b/arch/powerpc/include/asm/nohash/32/pgtable.h index 009a5b3d3192..3efd616bbc80 100644 --- a/arch/powerpc/include/asm/nohash/32/pgtable.h +++ b/arch/powerpc/include/asm/nohash/32/pgtable.h @@ -18,7 +18,11 @@ extern int icache_44x_need_flush; #endif /* __ASSEMBLY__ */ +#if defined(CONFIG_PPC_8xx) && defined(CONFIG_PPC_16K_PAGES) +#define PTE_INDEX_SIZE (PTE_SHIFT - 2) +#else #define PTE_INDEX_SIZE PTE_SHIFT +#endif #define PMD_INDEX_SIZE 0 #define PUD_INDEX_SIZE 0 #define PGD_INDEX_SIZE (32 - PGDIR_SHIFT) @@ -47,7 +51,11 @@ extern int icache_44x_need_flush; * -Matt */ /* PGDIR_SHIFT determines what a top-level page table entry can map */ +#ifdef CONFIG_PPC_8xx +#define PGDIR_SHIFT 22 +#else #define PGDIR_SHIFT (PAGE_SHIFT + PTE_INDEX_SIZE) +#endif #define PGDIR_SIZE (1UL << PGDIR_SHIFT) #define PGDIR_MASK (~(PGDIR_SIZE-1)) @@ -190,7 +198,13 @@ static inline unsigned long pte_update(pte_t *p, : "cc" ); #else /* PTE_ATOMIC_UPDATES */ unsigned long old = pte_val(*p); - *p = __pte((old & ~clr) | set); + unsigned long new = (old & ~clr) | set; + +#if defined(CONFIG_PPC_8xx) && defined(CONFIG_PPC_16K_PAGES) + p->pte = p->pte1 = p->pte2 = p->pte3 = new; +#else + *p = __pte(new); +#endif #endif /* !PTE_ATOMIC_UPDATES */ #ifdef CONFIG_44x diff --git a/arch/powerpc/include/asm/nohash/pgtable.h b/arch/powerpc/include/asm/nohash/pgtable.h index 077472640b35..e4b6c084be5c 100644 --- a/arch/powerpc/include/asm/nohash/pgtable.h +++ b/arch/powerpc/include/asm/nohash/pgtable.h @@ -165,7 +165,11 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long addr, /* Anything else just stores the PTE normally. That covers all 64-bit * cases, and 32-bit non-hash with 32-bit PTEs. */ +#if defined(CONFIG_PPC_8xx) && defined(CONFIG_PPC_16K_PAGES) + ptep->pte = ptep->pte1 = ptep->pte2 = ptep->pte3 = pte_val(pte); +#else *ptep = pte; +#endif /* * With hardware tablewalk, a sync is needed to ensure that diff --git a/arch/powerpc/include/asm/pgtable-types.h b/arch/powerpc/include/asm/pgtable-types.h index eccb30b38b47..3b0edf041b2e 100644 --- a/arch/powerpc/include/asm/pgtable-types.h +++ b/arch/powerpc/include/asm/pgtable-types.h @@ -3,7 +3,11 @@ #define _ASM_POWERPC_PGTABLE_TYPES_H /* PTE level */ +#if defined(CONFIG_PPC_8xx) && defined(CONFIG_PPC_16K_PAGES) +typedef struct { pte_basic_t pte, pte1, pte2, pte3; } pte_t; +#else typedef struct { pte_basic_t pte; } pte_t; +#endif #define __pte(x) ((pte_t) { (x) }) static inline pte_basic_t pte_val(pte_t x) { diff --git a/arch/powerpc/kernel/head_8xx.S b/arch/powerpc/kernel/head_8xx.S index 85b017c67e11..4855d5a36f70 100644 --- a/arch/powerpc/kernel/head_8xx.S +++ b/arch/powerpc/kernel/head_8xx.S @@ -275,7 +275,7 @@ SystemCall: . = 0x1100 /* * For the MPC8xx, this is a software tablewalk to load the instruction - * TLB. The task switch loads the M_TW register with the pointer to the first + * TLB. The task switch loads the M_TWB register with the pointer to the first * level table. * If we discover there is no second level table (value is zero) or if there * is an invalid pte, we load that into the TLB, which causes another fault @@ -285,106 +285,100 @@ SystemCall: */ #ifdef CONFIG_8xx_CPU15 -#define INVALIDATE_ADJACENT_PAGES_CPU15(tmp, addr) \ - addi tmp, addr, PAGE_SIZE; \ - tlbie tmp; \ - addi tmp, addr, -PAGE_SIZE; \ - tlbie tmp +#define INVALIDATE_ADJACENT_PAGES_CPU15(addr) \ + addi addr, addr, PAGE_SIZE; \ + tlbie addr; \ + addi addr, addr, -(PAGE_SIZE << 1); \ + tlbie addr; \ + addi addr, addr, PAGE_SIZE #else -#define INVALIDATE_ADJACENT_PAGES_CPU15(tmp, addr) +#define INVALIDATE_ADJACENT_PAGES_CPU15(addr) #endif InstructionTLBMiss: mtspr SPRN_SPRG_SCRATCH0, r10 +#if defined(ITLB_MISS_KERNEL) || defined(CONFIG_SWAP) mtspr SPRN_SPRG_SCRATCH1, r11 -#if defined(ITLB_MISS_KERNEL) || defined(CONFIG_HUGETLB_PAGE) - mtspr SPRN_SPRG_SCRATCH2, r12 +#ifdef ITLB_MISS_KERNEL + mfcr r11 +#endif #endif /* If we are faulting a kernel address, we have to use the * kernel page tables. */ mfspr r10, SPRN_SRR0 /* Get effective address of fault */ - INVALIDATE_ADJACENT_PAGES_CPU15(r11, r10) + INVALIDATE_ADJACENT_PAGES_CPU15(r10) /* Only modules will cause ITLB Misses as we always * pin the first 8MB of kernel memory */ -#if defined(ITLB_MISS_KERNEL) || defined(CONFIG_HUGETLB_PAGE) - mfcr r12 -#endif + mtspr SPRN_MD_EPN, r10 #ifdef ITLB_MISS_KERNEL #if defined(SIMPLE_KERNEL_ADDRESS) && defined(CONFIG_PIN_TLB_TEXT) - andis. r11, r10, 0x8000 /* Address >= 0x80000000 */ + cmpi cr0, r10, 0 /* Address >= 0x80000000 */ #else - rlwinm r11, r10, 16, 0xfff8 - cmpli cr0, r11, PAGE_OFFSET@h + rlwinm r10, r10, 16, 0xfff8 + cmpli cr0, r10, PAGE_OFFSET@h #ifndef CONFIG_PIN_TLB_TEXT /* It is assumed that kernel code fits into the first 8M page */ _ENTRY(ITLBMiss_cmp) - cmpli cr7, r11, (PAGE_OFFSET + 0x0800000)@h + cmpli cr7, r10, (PAGE_OFFSET + 0x0800000)@h #endif #endif #endif - mfspr r11, SPRN_M_TW /* Get level 1 table */ + mfspr r10, SPRN_M_TWB /* Get level 1 table */ #ifdef ITLB_MISS_KERNEL #if defined(SIMPLE_KERNEL_ADDRESS) && defined(CONFIG_PIN_TLB_TEXT) - beq+ 3f + bge+ 3f #else blt+ 3f #endif #ifndef CONFIG_PIN_TLB_TEXT blt cr7, ITLBMissLinear #endif - lis r11, (swapper_pg_dir-PAGE_OFFSET)@ha + rlwinm r10, r10, 0, 20, 31 + oris r10, r10, (swapper_pg_dir - PAGE_OFFSET)@h + ori r10, r10, (swapper_pg_dir - PAGE_OFFSET)@l 3: #endif +#ifdef ITLB_MISS_KERNEL + mfcr r11 +#endif /* Insert level 1 index */ - rlwimi r11, r10, 32 - ((PAGE_SHIFT - 2) << 1), (PAGE_SHIFT - 2) << 1, 29 - lwz r11, (swapper_pg_dir-PAGE_OFFSET)@l(r11) /* Get the level 1 entry */ + lwz r10, 0(r10) /* Get the level 1 entry */ + mtspr SPRN_MI_TWC, r10 /* Set segment attributes */ + mtspr SPRN_MD_TWC, r10 - /* Extract level 2 index */ - rlwinm r10, r10, 32 - (PAGE_SHIFT - 2), 32 - PAGE_SHIFT, 29 -#ifdef CONFIG_HUGETLB_PAGE - mtcr r11 -#endif - /* Load the MI_TWC with the attributes for this "segment." */ - mtspr SPRN_MI_TWC, r11 /* Set segment attributes */ -#ifdef CONFIG_HUGETLB_PAGE - bt- 28, 10f /* bit 28 = Large page (8M) */ - bt- 29, 20f /* bit 29 = Large page (8M or 512k) */ -#endif - rlwimi r10, r11, 0, 0, 32 - PAGE_SHIFT - 1 /* Add level 2 base */ + mfspr r10, SPRN_MD_TWC lwz r10, 0(r10) /* Get the pte */ -4: -#if defined(ITLB_MISS_KERNEL) || defined(CONFIG_HUGETLB_PAGE) - mtcr r12 -#endif #ifdef CONFIG_SWAP rlwinm r11, r10, 32-5, _PAGE_PRESENT and r11, r11, r10 rlwimi r10, r11, 0, _PAGE_PRESENT #endif - li r11, RPN_PATTERN | 0x200 /* The Linux PTE won't go exactly into the MMU TLB. * Software indicator bits 20 and 23 must be clear. * Software indicator bits 22, 24, 25, 26, and 27 must be * set. All other Linux PTE bits control the behavior * of the MMU. */ - rlwimi r11, r10, 4, 0x0400 /* Copy _PAGE_EXEC into bit 21 */ - rlwimi r10, r11, 0, 0x0ff0 /* Set 22, 24-27, clear 20,23 */ + rlwimi r10, r10, 0, 0x0f00 /* Clear bits 20-23 */ + rlwimi r10, r10, 4, 0x0400 /* Copy _PAGE_EXEC into bit 21 */ + ori r10, r10, RPN_PATTERN | 0x200 /* Set 22 and 24-27 */ mtspr SPRN_MI_RPN, r10 /* Update TLB entry */ /* Restore registers */ _ENTRY(itlb_miss_exit_1) mfspr r10, SPRN_SPRG_SCRATCH0 +#if defined(ITLB_MISS_KERNEL) || defined(CONFIG_SWAP) mfspr r11, SPRN_SPRG_SCRATCH1 -#if defined(ITLB_MISS_KERNEL) || defined(CONFIG_HUGETLB_PAGE) - mfspr r12, SPRN_SPRG_SCRATCH2 #endif rfi #ifdef CONFIG_PERF_EVENTS _ENTRY(itlb_miss_perf) +#if !defined(ITLB_MISS_KERNEL) && !defined(CONFIG_SWAP) + mtspr SPRN_SPRG_SCRATCH1, r11 +#endif lis r10, (itlb_miss_counter - PAGE_OFFSET)@ha lwz r11, (itlb_miss_counter - PAGE_OFFSET)@l(r10) addi r11, r11, 1 @@ -392,83 +386,42 @@ _ENTRY(itlb_miss_perf) #endif mfspr r10, SPRN_SPRG_SCRATCH0 mfspr r11, SPRN_SPRG_SCRATCH1 -#if defined(ITLB_MISS_KERNEL) || defined(CONFIG_HUGETLB_PAGE) - mfspr r12, SPRN_SPRG_SCRATCH2 -#endif rfi -#ifdef CONFIG_HUGETLB_PAGE -10: /* 8M pages */ -#ifdef CONFIG_PPC_16K_PAGES - /* Extract level 2 index */ - rlwinm r10, r10, 32 - (PAGE_SHIFT_8M - PAGE_SHIFT), 32 + PAGE_SHIFT_8M - (PAGE_SHIFT << 1), 29 - /* Add level 2 base */ - rlwimi r10, r11, 0, 0, 32 + PAGE_SHIFT_8M - (PAGE_SHIFT << 1) - 1 -#else - /* Level 2 base */ - rlwinm r10, r11, 0, ~HUGEPD_SHIFT_MASK -#endif - lwz r10, 0(r10) /* Get the pte */ - b 4b - -20: /* 512k pages */ - /* Extract level 2 index */ - rlwinm r10, r10, 32 - (PAGE_SHIFT_512K - PAGE_SHIFT), 32 + PAGE_SHIFT_512K - (PAGE_SHIFT << 1), 29 - /* Add level 2 base */ - rlwimi r10, r11, 0, 0, 32 + PAGE_SHIFT_512K - (PAGE_SHIFT << 1) - 1 - lwz r10, 0(r10) /* Get the pte */ - b 4b -#endif - . = 0x1200 DataStoreTLBMiss: mtspr SPRN_SPRG_SCRATCH0, r10 mtspr SPRN_SPRG_SCRATCH1, r11 - mtspr SPRN_SPRG_SCRATCH2, r12 - mfcr r12 + mfcr r11 /* If we are faulting a kernel address, we have to use the * kernel page tables. */ mfspr r10, SPRN_MD_EPN - rlwinm r11, r10, 16, 0xfff8 - cmpli cr0, r11, PAGE_OFFSET@h - mfspr r11, SPRN_M_TW /* Get level 1 table */ - blt+ 3f - rlwinm r11, r10, 16, 0xfff8 + rlwinm r10, r10, 16, 0xfff8 + cmpli cr0, r10, PAGE_OFFSET@h #ifndef CONFIG_PIN_TLB_IMMR - cmpli cr0, r11, VIRT_IMMR_BASE@h + cmpli cr6, r10, VIRT_IMMR_BASE@h #endif _ENTRY(DTLBMiss_cmp) - cmpli cr7, r11, (PAGE_OFFSET + 0x1800000)@h + cmpli cr7, r10, (PAGE_OFFSET + 0x1800000)@h + mfspr r10, SPRN_M_TWB /* Get level 1 table */ + blt+ 3f #ifndef CONFIG_PIN_TLB_IMMR _ENTRY(DTLBMiss_jmp) - beq- DTLBMissIMMR + beq- cr6, DTLBMissIMMR #endif blt cr7, DTLBMissLinear - lis r11, (swapper_pg_dir-PAGE_OFFSET)@ha + rlwinm r10, r10, 0, 20, 31 + oris r10, r10, (swapper_pg_dir - PAGE_OFFSET)@h + ori r10, r10, (swapper_pg_dir - PAGE_OFFSET)@l 3: - - /* Insert level 1 index */ - rlwimi r11, r10, 32 - ((PAGE_SHIFT - 2) << 1), (PAGE_SHIFT - 2) << 1, 29 - lwz r11, (swapper_pg_dir-PAGE_OFFSET)@l(r11) /* Get the level 1 entry */ - - /* We have a pte table, so load fetch the pte from the table. - */ - /* Extract level 2 index */ - rlwinm r10, r10, 32 - (PAGE_SHIFT - 2), 32 - PAGE_SHIFT, 29 -#ifdef CONFIG_HUGETLB_PAGE + lwz r10, 0(r10) /* Get the level 1 entry */ mtcr r11 -#endif - mtspr SPRN_MD_TWC, r11 -#ifdef CONFIG_HUGETLB_PAGE - bt- 28, 10f /* bit 28 = Large page (8M) */ - bt- 29, 20f /* bit 29 = Large page (8M or 512k) */ -#endif - rlwimi r10, r11, 0, 0, 32 - PAGE_SHIFT - 1 /* Add level 2 base */ + + mtspr SPRN_MD_TWC, r10 + mfspr r10, SPRN_MD_TWC lwz r10, 0(r10) /* Get the pte */ -4: - mtcr r12 /* Both _PAGE_ACCESSED and _PAGE_PRESENT has to be set. * We also need to know if the insn is a load/store, so: @@ -498,7 +451,6 @@ _ENTRY(DTLBMiss_jmp) _ENTRY(dtlb_miss_exit_1) mfspr r10, SPRN_SPRG_SCRATCH0 mfspr r11, SPRN_SPRG_SCRATCH1 - mfspr r12, SPRN_SPRG_SCRATCH2 rfi #ifdef CONFIG_PERF_EVENTS _ENTRY(dtlb_miss_perf) @@ -509,32 +461,8 @@ _ENTRY(dtlb_miss_perf) #endif mfspr r10, SPRN_SPRG_SCRATCH0 mfspr r11, SPRN_SPRG_SCRATCH1 - mfspr r12, SPRN_SPRG_SCRATCH2 rfi -#ifdef CONFIG_HUGETLB_PAGE -10: /* 8M pages */ - /* Extract level 2 index */ -#ifdef CONFIG_PPC_16K_PAGES - rlwinm r10, r10, 32 - (PAGE_SHIFT_8M - PAGE_SHIFT), 32 + PAGE_SHIFT_8M - (PAGE_SHIFT << 1), 29 - /* Add level 2 base */ - rlwimi r10, r11, 0, 0, 32 + PAGE_SHIFT_8M - (PAGE_SHIFT << 1) - 1 -#else - /* Level 2 base */ - rlwinm r10, r11, 0, ~HUGEPD_SHIFT_MASK -#endif - lwz r10, 0(r10) /* Get the pte */ - b 4b - -20: /* 512k pages */ - /* Extract level 2 index */ - rlwinm r10, r10, 32 - (PAGE_SHIFT_512K - PAGE_SHIFT), 32 + PAGE_SHIFT_512K - (PAGE_SHIFT << 1), 29 - /* Add level 2 base */ - rlwimi r10, r11, 0, 0, 32 + PAGE_SHIFT_512K - (PAGE_SHIFT << 1) - 1 - lwz r10, 0(r10) /* Get the pte */ - b 4b -#endif - /* This is an instruction TLB error on the MPC8xx. This could be due * to many reasons, such as executing guarded memory or illegal instruction * addresses. There is nothing to do but handle a big time error fault. @@ -642,7 +570,7 @@ InstructionBreakpoint: * not enough space in the DataStoreTLBMiss area. */ DTLBMissIMMR: - mtcr r12 + mtcr r11 /* Set 512k byte guarded page and mark it valid */ li r10, MD_PS512K | MD_GUARDED | MD_SVALID mtspr SPRN_MD_TWC, r10 @@ -657,15 +585,14 @@ DTLBMissIMMR: _ENTRY(dtlb_miss_exit_2) mfspr r10, SPRN_SPRG_SCRATCH0 mfspr r11, SPRN_SPRG_SCRATCH1 - mfspr r12, SPRN_SPRG_SCRATCH2 rfi DTLBMissLinear: - mtcr r12 + mtcr r11 /* Set 8M byte page and mark it valid */ li r11, MD_PS8MEG | MD_SVALID mtspr SPRN_MD_TWC, r11 - rlwinm r10, r10, 0, 0x0f800000 /* 8xx supports max 256Mb RAM */ + rlwinm r10, r10, 20, 0x0f800000 /* 8xx supports max 256Mb RAM */ ori r10, r10, 0xf0 | MD_SPS16K | _PAGE_PRIVILEGED | _PAGE_DIRTY | \ _PAGE_PRESENT mtspr SPRN_MD_RPN, r10 /* Update TLB entry */ @@ -675,16 +602,15 @@ DTLBMissLinear: _ENTRY(dtlb_miss_exit_3) mfspr r10, SPRN_SPRG_SCRATCH0 mfspr r11, SPRN_SPRG_SCRATCH1 - mfspr r12, SPRN_SPRG_SCRATCH2 rfi #ifndef CONFIG_PIN_TLB_TEXT ITLBMissLinear: - mtcr r12 + mtcr r11 /* Set 8M byte page and mark it valid */ li r11, MI_PS8MEG | MI_SVALID mtspr SPRN_MI_TWC, r11 - rlwinm r10, r10, 0, 0x0f800000 /* 8xx supports max 256Mb RAM */ + rlwinm r10, r10, 20, 0x0f800000 /* 8xx supports max 256Mb RAM */ ori r10, r10, 0xf0 | MI_SPS16K | _PAGE_PRIVILEGED | _PAGE_DIRTY | \ _PAGE_PRESENT mtspr SPRN_MI_RPN, r10 /* Update TLB entry */ @@ -692,7 +618,6 @@ ITLBMissLinear: _ENTRY(itlb_miss_exit_2) mfspr r10, SPRN_SPRG_SCRATCH0 mfspr r11, SPRN_SPRG_SCRATCH1 - mfspr r12, SPRN_SPRG_SCRATCH2 rfi #endif @@ -706,9 +631,10 @@ FixupDAR:/* Entry point for dcbx workaround. */ mtspr SPRN_SPRG_SCRATCH2, r10 /* fetch instruction from memory. */ mfspr r10, SPRN_SRR0 + mtspr SPRN_MD_EPN, r10 rlwinm r11, r10, 16, 0xfff8 cmpli cr0, r11, PAGE_OFFSET@h - mfspr r11, SPRN_M_TW /* Get level 1 table */ + mfspr r11, SPRN_M_TWB /* Get level 1 table */ blt+ 3f rlwinm r11, r10, 16, 0xfff8 _ENTRY(FixupDAR_cmp) @@ -716,19 +642,20 @@ _ENTRY(FixupDAR_cmp) /* create physical page address from effective address */ tophys(r11, r10) blt- cr7, 201f - lis r11, (swapper_pg_dir-PAGE_OFFSET)@ha - /* Insert level 1 index */ -3: rlwimi r11, r10, 32 - ((PAGE_SHIFT - 2) << 1), (PAGE_SHIFT - 2) << 1, 29 - lwz r11, (swapper_pg_dir-PAGE_OFFSET)@l(r11) /* Get the level 1 entry */ + mfspr r11, SPRN_M_TWB /* Get level 1 table */ + rlwinm r11, r11, 0, 20, 31 + oris r11, r11, (swapper_pg_dir - PAGE_OFFSET)@h + ori r11, r11, (swapper_pg_dir - PAGE_OFFSET)@l +3: + lwz r11, 0(r11) /* Get the level 1 entry */ + mtspr SPRN_MD_TWC, r11 mtcr r11 + mfspr r11, SPRN_MD_TWC + lwz r11, 0(r11) /* Get the pte */ bt 28,200f /* bit 28 = Large page (8M) */ bt 29,202f /* bit 29 = Large page (8M or 512K) */ - rlwinm r11, r11,0,0,19 /* Extract page descriptor page address */ - /* Insert level 2 index */ - rlwimi r11, r10, 32 - (PAGE_SHIFT - 2), 32 - PAGE_SHIFT, 29 - lwz r11, 0(r11) /* Get the pte */ /* concat physical page address(r11) and page offset(r10) */ - rlwimi r11, r10, 0, 32 - PAGE_SHIFT, 31 + rlwimi r11, r10, 0, 20, 31 201: lwz r11,0(r11) /* Check if it really is a dcbx instruction. */ /* dcbt and dcbtst does not generate DTLB Misses/Errors, @@ -748,23 +675,12 @@ _ENTRY(FixupDAR_cmp) 141: mfspr r10,SPRN_SPRG_SCRATCH2 b DARFixed /* Nope, go back to normal TLB processing */ - /* concat physical page address(r11) and page offset(r10) */ 200: -#ifdef CONFIG_PPC_16K_PAGES - rlwinm r11, r11, 0, 0, 32 + PAGE_SHIFT_8M - (PAGE_SHIFT << 1) - 1 - rlwimi r11, r10, 32 - (PAGE_SHIFT_8M - 2), 32 + PAGE_SHIFT_8M - (PAGE_SHIFT << 1), 29 -#else - rlwinm r11, r10, 0, ~HUGEPD_SHIFT_MASK -#endif - lwz r11, 0(r11) /* Get the pte */ /* concat physical page address(r11) and page offset(r10) */ rlwimi r11, r10, 0, 32 - PAGE_SHIFT_8M, 31 b 201b 202: - rlwinm r11, r11, 0, 0, 32 + PAGE_SHIFT_512K - (PAGE_SHIFT << 1) - 1 - rlwimi r11, r10, 32 - (PAGE_SHIFT_512K - 2), 32 + PAGE_SHIFT_512K - (PAGE_SHIFT << 1), 29 - lwz r11, 0(r11) /* Get the pte */ /* concat physical page address(r11) and page offset(r10) */ rlwimi r11, r10, 0, 32 - PAGE_SHIFT_512K, 31 b 201b @@ -898,9 +814,10 @@ start_here: * init's THREAD like the context switch code does, but this is * easier......until someone changes init's static structures. */ - lis r6, swapper_pg_dir@ha + lis r6, swapper_pg_dir@h + ori r6, r6, swapper_pg_dir@l tophys(r6,r6) - mtspr SPRN_M_TW, r6 + mtspr SPRN_M_TWB, r6 lis r4,2f@h ori r4,r4,2f@l tophys(r4,r4) diff --git a/arch/powerpc/mm/8xx_mmu.c b/arch/powerpc/mm/8xx_mmu.c index 5d53684c2ebd..54a02b8e21ec 100644 --- a/arch/powerpc/mm/8xx_mmu.c +++ b/arch/powerpc/mm/8xx_mmu.c @@ -173,8 +173,6 @@ void __init setup_initial_memory_limit(phys_addr_t first_memblock_base, */ void set_context(unsigned long id, pgd_t *pgd) { - s16 offset = (s16)(__pa(swapper_pg_dir)); - #ifdef CONFIG_BDI_SWITCH pgd_t **ptr = *(pgd_t ***)(KERNELBASE + 0xf0); @@ -184,12 +182,8 @@ void set_context(unsigned long id, pgd_t *pgd) *(ptr + 1) = pgd; #endif - /* Register M_TW will contain base address of level 1 table minus the - * lower part of the kernel PGDIR base address, so that all accesses to - * level 1 table are done relative to lower part of kernel PGDIR base - * address. - */ - mtspr(SPRN_M_TW, __pa(pgd) - offset); + /* Register M_TWB will contain base address of level 1 table */ + mtspr(SPRN_M_TWB, __pa(pgd)); /* Update context */ mtspr(SPRN_M_CASID, id - 1); diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c index f1153f8254e3..6c07d40eed0f 100644 --- a/arch/powerpc/mm/hugetlbpage.c +++ b/arch/powerpc/mm/hugetlbpage.c @@ -61,7 +61,11 @@ static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp, cachep = hugepte_cache; num_hugepd = 1 << (pshift - pdshift); } else { +#ifdef CONFIG_PPC_8xx + cachep = PGT_CACHE(PTE_SHIFT); +#else cachep = PGT_CACHE(pdshift - pshift); +#endif num_hugepd = 1; } @@ -326,7 +330,11 @@ static void free_hugepd_range(struct mmu_gather *tlb, hugepd_t *hpdp, int pdshif if (shift >= pdshift) hugepd_free(tlb, hugepte); else +#ifdef CONFIG_PPC_8xx + pgtable_free_tlb(tlb, hugepte, PTE_SHIFT); +#else pgtable_free_tlb(tlb, hugepte, pdshift - shift); +#endif } static void hugetlb_free_pmd_range(struct mmu_gather *tlb, pud_t *pud, @@ -689,7 +697,11 @@ static int __init hugetlbpage_init(void) * use pgt cache for hugepd. */ if (pdshift > shift) +#ifdef CONFIG_PPC_8xx + pgtable_cache_add(PTE_SHIFT, NULL); +#else pgtable_cache_add(pdshift - shift, NULL); +#endif #if defined(CONFIG_PPC_FSL_BOOK3E) || defined(CONFIG_PPC_8xx) else if (!hugepte_cache) { /* -- 2.13.3