Svpbmt is an extension defining "Supervisor-mode: page-based memory types"
for things like non-cacheable pages or I/O memory pages.
So this is my 2nd try at implementing Svpbmt (and the diverging D1 memory
types) using the alternatives framework.
This includes a number of changes to the alternatives mechanism itself.
The biggest one being the move to a more central location, as I expect
in the future, nearly every chip needing some sort of patching, be it
either for erratas or for optional features (svpbmt or others).
Detection of the svpbmt functionality is done via Atish's isa extension
handling series [0] and thus does not need any dt-parsing of its own
anymore.
The series also introduces support for the memory types of the D1
which are implemented differently to svpbmt. But when patching anyway
it's pretty clean to add the D1 variant via ALTERNATIVE_2 to the same
location.
The only slightly bigger difference is that the "normal" type is not 0
as with svpbmt, so kernel patches for this PMA type need to be applied
even before the MMU is brought up, so the series introduces a separate
stage for that.
In theory this series is 2 parts:
- alternatives improvements
- svpbmt+d1
I picked the recipient list from the previous versions, hopefully
I didn't forget anybody.
I tested the series on:
- qemu-rv32 + buildroot rootfs
- qemu-rv64 + debian roots
- Allwinner D1-Nezha
- BeagleV - it at least reached the same point as without the series
changes in v9:
- rebase onto 5.18-rc1
- drop the sbi null-ptr patch
While I still think this to be non-ideal as is, it isn't really
necessary for svpbmt support anymore
- merge cpufeature + svpbmt patch, as otherwise some empty shells
cause build warnings when a bisection stops between these two
patches
- address review comments from Christoph Hellwig:
- keep alternatives optional, they now get selected by its
users (erratas and also the newly introduced svpbmt kconfig)
- wrap long lines and keep things below 80 characters
- restyle svpbmt + thead errata assembly
- introduce a helper for the repeated calls to
(val & _PAGE_PFN_MASK) >> _PAGE_PFN_SHIFT
changes in v8:
- rebase onto 5.17-final + isa extension series
We're halfway through the merge-window, so this series
should be merge after that
- adapt to fix limiting alternatives to non-xip-kernels
- add .norelax option for alternatives
- fix unused cpu_apply_errata in thead errata
- don't use static globals to store cpu-manufacturer data
as it makes machines hang if done too early
changes in v7:
- fix typo in patch1 (Atish)
- moved to Atish's isa-extension framework
- and therefore move regular boot-alternatives directly behind fill_hwcaps
- change T-Head errata Kconfig text (Atish)
changes in v6:
- rebase onto 5.17-rc1
- handle sbi null-ptr differently
- improve commit messages
- use riscv,mmu as property name
changes in v5:
- move to use alternatives for runtime-patching
- add D1 variant
[0] https://lore.kernel.org/r/[email protected]
Heiko Stuebner (12):
riscv: integrate alternatives better into the main architecture
riscv: allow different stages with alternatives
riscv: implement module alternatives
riscv: implement ALTERNATIVE_2 macro
riscv: extend concatenated alternatives-lines to the same length
riscv: prevent compressed instructions in alternatives
riscv: move boot alternatives to after fill_hwcap
riscv: Fix accessing pfn bits in PTEs for non-32bit variants
riscv: add RISC-V Svpbmt extension support
riscv: remove FIXMAP_PAGE_IO and fall back to its default value
riscv: don't use global static vars to store alternative data
riscv: add memory-type errata for T-Head
arch/riscv/Kconfig | 22 ++++
arch/riscv/Kconfig.erratas | 33 +++--
arch/riscv/Kconfig.socs | 1 -
arch/riscv/Makefile | 2 +-
arch/riscv/errata/Makefile | 2 +-
arch/riscv/errata/alternative.c | 75 ------------
arch/riscv/errata/sifive/errata.c | 20 ++-
arch/riscv/errata/thead/Makefile | 1 +
arch/riscv/errata/thead/errata.c | 82 +++++++++++++
arch/riscv/include/asm/alternative-macros.h | 129 +++++++++++++++-----
arch/riscv/include/asm/alternative.h | 25 +++-
arch/riscv/include/asm/errata_list.h | 59 +++++++++
arch/riscv/include/asm/fixmap.h | 2 -
arch/riscv/include/asm/hwcap.h | 1 +
arch/riscv/include/asm/pgtable-32.h | 17 +++
arch/riscv/include/asm/pgtable-64.h | 79 +++++++++++-
arch/riscv/include/asm/pgtable-bits.h | 10 --
arch/riscv/include/asm/pgtable.h | 55 +++++++--
arch/riscv/include/asm/vendorid_list.h | 1 +
arch/riscv/kernel/Makefile | 1 +
arch/riscv/kernel/alternative.c | 104 ++++++++++++++++
arch/riscv/kernel/cpu.c | 1 +
arch/riscv/kernel/cpufeature.c | 80 +++++++++++-
arch/riscv/kernel/module.c | 29 +++++
arch/riscv/kernel/setup.c | 2 +
arch/riscv/kernel/smpboot.c | 4 -
arch/riscv/kernel/traps.c | 2 +-
arch/riscv/mm/init.c | 1 +
28 files changed, 679 insertions(+), 161 deletions(-)
delete mode 100644 arch/riscv/errata/alternative.c
create mode 100644 arch/riscv/errata/thead/Makefile
create mode 100644 arch/riscv/errata/thead/errata.c
create mode 100644 arch/riscv/kernel/alternative.c
--
2.35.1
Future features may need to be applied at a different
time during boot, so allow defining stages for alternatives
and handling them differently depending on the stage.
Also make the alternatives-location more flexible so that
future stages may provide their own location.
Signed-off-by: Heiko Stuebner <[email protected]>
---
arch/riscv/errata/sifive/errata.c | 3 ++-
arch/riscv/include/asm/alternative.h | 5 ++++-
arch/riscv/kernel/alternative.c | 26 +++++++++++++++++---------
3 files changed, 23 insertions(+), 11 deletions(-)
diff --git a/arch/riscv/errata/sifive/errata.c b/arch/riscv/errata/sifive/errata.c
index f5e5ae70e829..4fe03ac41fd7 100644
--- a/arch/riscv/errata/sifive/errata.c
+++ b/arch/riscv/errata/sifive/errata.c
@@ -80,7 +80,8 @@ static void __init warn_miss_errata(u32 miss_errata)
}
void __init sifive_errata_patch_func(struct alt_entry *begin, struct alt_entry *end,
- unsigned long archid, unsigned long impid)
+ unsigned long archid, unsigned long impid,
+ unsigned int stage)
{
struct alt_entry *alt;
u32 cpu_req_errata = sifive_errata_probe(archid, impid);
diff --git a/arch/riscv/include/asm/alternative.h b/arch/riscv/include/asm/alternative.h
index 7b42bcef0ecf..0ff550667e94 100644
--- a/arch/riscv/include/asm/alternative.h
+++ b/arch/riscv/include/asm/alternative.h
@@ -19,6 +19,8 @@
#include <linux/stddef.h>
#include <asm/hwcap.h>
+#define RISCV_ALTERNATIVES_BOOT 0 /* alternatives applied during regular boot */
+
void __init apply_boot_alternatives(void);
struct alt_entry {
@@ -35,7 +37,8 @@ struct errata_checkfunc_id {
};
void sifive_errata_patch_func(struct alt_entry *begin, struct alt_entry *end,
- unsigned long archid, unsigned long impid);
+ unsigned long archid, unsigned long impid,
+ unsigned int stage);
#else /* CONFIG_RISCV_ALTERNATIVE */
diff --git a/arch/riscv/kernel/alternative.c b/arch/riscv/kernel/alternative.c
index e8b4a0fe488c..02db62f55bac 100644
--- a/arch/riscv/kernel/alternative.c
+++ b/arch/riscv/kernel/alternative.c
@@ -22,8 +22,8 @@ static struct cpu_manufacturer_info_t {
} cpu_mfr_info;
static void (*vendor_patch_func)(struct alt_entry *begin, struct alt_entry *end,
- unsigned long archid,
- unsigned long impid) __initdata;
+ unsigned long archid, unsigned long impid,
+ unsigned int stage) __initdata;
static inline void __init riscv_fill_cpu_mfr_info(void)
{
@@ -58,6 +58,18 @@ static void __init init_alternative(void)
* a feature detect on the boot CPU). No need to worry about other CPUs
* here.
*/
+static void __init _apply_alternatives(struct alt_entry *begin,
+ struct alt_entry *end,
+ unsigned int stage)
+{
+ if (!vendor_patch_func)
+ return;
+
+ vendor_patch_func(begin, end,
+ cpu_mfr_info.arch_id, cpu_mfr_info.imp_id,
+ stage);
+}
+
void __init apply_boot_alternatives(void)
{
/* If called on non-boot cpu things could go wrong */
@@ -65,11 +77,7 @@ void __init apply_boot_alternatives(void)
init_alternative();
- if (!vendor_patch_func)
- return;
-
- vendor_patch_func((struct alt_entry *)__alt_start,
- (struct alt_entry *)__alt_end,
- cpu_mfr_info.arch_id, cpu_mfr_info.imp_id);
+ _apply_alternatives((struct alt_entry *)__alt_start,
+ (struct alt_entry *)__alt_end,
+ RISCV_ALTERNATIVES_BOOT);
}
-
--
2.35.1
Instructions are opportunistically compressed by the RISC-V assembler
when possible, but in alternatives-blocks both the old and new content
need to be the same size, so having the toolchain do somewhat random
optimizations will cause strange side-effects like
"attempt to move .org backwards" compile-time errors.
Already a simple "and" used in alternatives assembly will cause these
mismatched code sizes.
So prevent compressed instructions to be generated in alternatives-
code and use option-push and -pop to only limit this to the relevant
code blocks
Signed-off-by: Heiko Stuebner <[email protected]>
Reviewed-by: Christoph Hellwig <[email protected]>
---
arch/riscv/include/asm/alternative-macros.h | 24 +++++++++++++++++++++
1 file changed, 24 insertions(+)
diff --git a/arch/riscv/include/asm/alternative-macros.h b/arch/riscv/include/asm/alternative-macros.h
index 8c2bbc7bbe50..e13b1f6bb400 100644
--- a/arch/riscv/include/asm/alternative-macros.h
+++ b/arch/riscv/include/asm/alternative-macros.h
@@ -21,7 +21,11 @@
.popsection
.subsection 1
888 :
+ .option push
+ .option norvc
+ .option norelax
\new_c
+ .option pop
889 :
.previous
.org . - (889b - 888b) + (887b - 886b)
@@ -31,7 +35,11 @@
.macro __ALTERNATIVE_CFG old_c, new_c, vendor_id, errata_id, enable
886 :
+ .option push
+ .option norvc
+ .option norelax
\old_c
+ .option pop
887 :
ALT_NEW_CONTENT \vendor_id, \errata_id, \enable, \new_c
.endm
@@ -42,7 +50,11 @@
.macro __ALTERNATIVE_CFG_2 old_c, new_c_1, vendor_id_1, errata_id_1, enable_1, \
new_c_2, vendor_id_2, errata_id_2, enable_2
886 :
+ .option push
+ .option norvc
+ .option norelax
\old_c
+ .option pop
887 :
ALT_NEW_CONTENT \vendor_id_1, \errata_id_1, \enable_1, \new_c_1
ALT_NEW_CONTENT \vendor_id_2, \errata_id_2, \enable_2, \new_c_2
@@ -76,7 +88,11 @@
".popsection\n" \
".subsection 1\n" \
"888 :\n" \
+ ".option push\n" \
+ ".option norvc\n" \
+ ".option norelax\n" \
new_c "\n" \
+ ".option pop\n" \
"889 :\n" \
".previous\n" \
".org . - (887b - 886b) + (889b - 888b)\n" \
@@ -85,7 +101,11 @@
#define __ALTERNATIVE_CFG(old_c, new_c, vendor_id, errata_id, enable) \
"886 :\n" \
+ ".option push\n" \
+ ".option norvc\n" \
+ ".option norelax\n" \
old_c "\n" \
+ ".option pop\n" \
"887 :\n" \
ALT_NEW_CONTENT(vendor_id, errata_id, enable, new_c)
@@ -97,7 +117,11 @@
new_c_2, vendor_id_2, errata_id_2, \
enable_2) \
"886 :\n" \
+ ".option push\n" \
+ ".option norvc\n" \
+ ".option norelax\n" \
old_c "\n" \
+ ".option pop\n" \
"887 :\n" \
ALT_NEW_CONTENT(vendor_id_1, errata_id_1, enable_1, new_c_1) \
ALT_NEW_CONTENT(vendor_id_2, errata_id_2, enable_2, new_c_2)
--
2.35.1
On rv32 the PFN part of PTEs is defined to use bits [xlen-1:10]
while on rv64 it is defined to use bits [53:10], leaving [63:54]
as reserved.
With upcoming optional extensions like svpbmt these previously
reserved bits will get used so simply right-shifting the PTE
to get the PFN won't be enough.
So introduce a _PAGE_PFN_MASK constant to mask the correct bits
for both rv32 and rv64 before shifting.
Signed-off-by: Heiko Stuebner <[email protected]>
---
arch/riscv/include/asm/pgtable-32.h | 8 ++++++++
arch/riscv/include/asm/pgtable-64.h | 14 +++++++++++---
arch/riscv/include/asm/pgtable-bits.h | 6 ------
arch/riscv/include/asm/pgtable.h | 8 +++++---
4 files changed, 24 insertions(+), 12 deletions(-)
diff --git a/arch/riscv/include/asm/pgtable-32.h b/arch/riscv/include/asm/pgtable-32.h
index 5b2e79e5bfa5..e266a4fe7f43 100644
--- a/arch/riscv/include/asm/pgtable-32.h
+++ b/arch/riscv/include/asm/pgtable-32.h
@@ -7,6 +7,7 @@
#define _ASM_RISCV_PGTABLE_32_H
#include <asm-generic/pgtable-nopmd.h>
+#include <linux/bits.h>
#include <linux/const.h>
/* Size of region mapped by a page global directory */
@@ -16,4 +17,11 @@
#define MAX_POSSIBLE_PHYSMEM_BITS 34
+/*
+ * rv32 PTE format:
+ * | XLEN-1 10 | 9 8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0
+ * PFN reserved for SW D A G U X W R V
+ */
+#define _PAGE_PFN_MASK GENMASK(31, 10)
+
#endif /* _ASM_RISCV_PGTABLE_32_H */
diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/pgtable-64.h
index 7e246e9f8d70..15f3ad5aee4f 100644
--- a/arch/riscv/include/asm/pgtable-64.h
+++ b/arch/riscv/include/asm/pgtable-64.h
@@ -6,6 +6,7 @@
#ifndef _ASM_RISCV_PGTABLE_64_H
#define _ASM_RISCV_PGTABLE_64_H
+#include <linux/bits.h>
#include <linux/const.h>
extern bool pgtable_l4_enabled;
@@ -65,6 +66,13 @@ typedef struct {
#define PTRS_PER_PMD (PAGE_SIZE / sizeof(pmd_t))
+/*
+ * rv64 PTE format:
+ * | 63 | 62 61 | 60 54 | 53 10 | 9 8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0
+ * N MT RSV PFN reserved for SW D A G U X W R V
+ */
+#define _PAGE_PFN_MASK GENMASK(53, 10)
+
static inline int pud_present(pud_t pud)
{
return (pud_val(pud) & _PAGE_PRESENT);
@@ -108,12 +116,12 @@ static inline unsigned long _pud_pfn(pud_t pud)
static inline pmd_t *pud_pgtable(pud_t pud)
{
- return (pmd_t *)pfn_to_virt(pud_val(pud) >> _PAGE_PFN_SHIFT);
+ return (pmd_t *)pfn_to_virt(__page_val_to_pfn(pud_val(pud)));
}
static inline struct page *pud_page(pud_t pud)
{
- return pfn_to_page(pud_val(pud) >> _PAGE_PFN_SHIFT);
+ return pfn_to_page(__page_val_to_pfn(pud_val(pud)));
}
#define mm_p4d_folded mm_p4d_folded
@@ -143,7 +151,7 @@ static inline pmd_t pfn_pmd(unsigned long pfn, pgprot_t prot)
static inline unsigned long _pmd_pfn(pmd_t pmd)
{
- return pmd_val(pmd) >> _PAGE_PFN_SHIFT;
+ return __page_val_to_pfn(pmd_val(pmd));
}
#define mk_pmd(page, prot) pfn_pmd(page_to_pfn(page), prot)
diff --git a/arch/riscv/include/asm/pgtable-bits.h b/arch/riscv/include/asm/pgtable-bits.h
index a6b0c89824c2..e571fa954afc 100644
--- a/arch/riscv/include/asm/pgtable-bits.h
+++ b/arch/riscv/include/asm/pgtable-bits.h
@@ -6,12 +6,6 @@
#ifndef _ASM_RISCV_PGTABLE_BITS_H
#define _ASM_RISCV_PGTABLE_BITS_H
-/*
- * PTE format:
- * | XLEN-1 10 | 9 8 | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0
- * PFN reserved for SW D A G U X W R V
- */
-
#define _PAGE_ACCESSED_OFFSET 6
#define _PAGE_PRESENT (1 << 0)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index 046b44225623..faba543e2b08 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -108,6 +108,8 @@
#include <asm/tlbflush.h>
#include <linux/mm_types.h>
+#define __page_val_to_pfn(_val) (((_val) & _PAGE_PFN_MASK) >> _PAGE_PFN_SHIFT)
+
#ifdef CONFIG_64BIT
#include <asm/pgtable-64.h>
#else
@@ -261,12 +263,12 @@ static inline unsigned long _pgd_pfn(pgd_t pgd)
static inline struct page *pmd_page(pmd_t pmd)
{
- return pfn_to_page(pmd_val(pmd) >> _PAGE_PFN_SHIFT);
+ return pfn_to_page(__page_val_to_pfn(pmd_val(pmd)));
}
static inline unsigned long pmd_page_vaddr(pmd_t pmd)
{
- return (unsigned long)pfn_to_virt(pmd_val(pmd) >> _PAGE_PFN_SHIFT);
+ return (unsigned long)pfn_to_virt(__page_val_to_pfn(pmd_val(pmd)));
}
static inline pte_t pmd_pte(pmd_t pmd)
@@ -282,7 +284,7 @@ static inline pte_t pud_pte(pud_t pud)
/* Yields the page frame number (PFN) of a page table entry */
static inline unsigned long pte_pfn(pte_t pte)
{
- return (pte_val(pte) >> _PAGE_PFN_SHIFT);
+ return __page_val_to_pfn(pte_val(pte));
}
#define pte_page(x) pfn_to_page(pte_pfn(x))
--
2.35.1
If not defined in the arch, FIXMAP_PAGE_IO defaults to PAGE_KERNEL_IO,
which we defined when adding the svpbmt implementation.
So drop the FIXMAP_PAGE_IO riscv define.
Signed-off-by: Heiko Stuebner <[email protected]>
Reviewed-by: Christoph Hellwig <[email protected]>
---
arch/riscv/include/asm/fixmap.h | 2 --
1 file changed, 2 deletions(-)
diff --git a/arch/riscv/include/asm/fixmap.h b/arch/riscv/include/asm/fixmap.h
index 3cfece8b6568..5c3e7b97fcc6 100644
--- a/arch/riscv/include/asm/fixmap.h
+++ b/arch/riscv/include/asm/fixmap.h
@@ -45,8 +45,6 @@ enum fixed_addresses {
__end_of_fixed_addresses
};
-#define FIXMAP_PAGE_IO PAGE_KERNEL
-
#define __early_set_fixmap __set_fixmap
#define __late_set_fixmap __set_fixmap
--
2.35.1
Some current cpus based on T-Head cores implement memory-types
way different than described in the svpbmt spec even going
so far as using PTE bits marked as reserved.
Add the T-Head vendor-id and necessary errata code to
replace the affected instructions.
Signed-off-by: Heiko Stuebner <[email protected]>
Tested-by: Samuel Holland <[email protected]>
---
arch/riscv/Kconfig.erratas | 20 +++++++
arch/riscv/errata/Makefile | 1 +
arch/riscv/errata/sifive/errata.c | 7 ++-
arch/riscv/errata/thead/Makefile | 1 +
arch/riscv/errata/thead/errata.c | 82 ++++++++++++++++++++++++++
arch/riscv/include/asm/alternative.h | 6 ++
arch/riscv/include/asm/errata_list.h | 50 +++++++++++++++-
arch/riscv/include/asm/pgtable-64.h | 18 +++++-
arch/riscv/include/asm/pgtable.h | 18 +++++-
arch/riscv/include/asm/vendorid_list.h | 1 +
arch/riscv/kernel/alternative.c | 12 ++++
arch/riscv/kernel/cpufeature.c | 7 ++-
arch/riscv/mm/init.c | 1 +
13 files changed, 215 insertions(+), 9 deletions(-)
create mode 100644 arch/riscv/errata/thead/Makefile
create mode 100644 arch/riscv/errata/thead/errata.c
diff --git a/arch/riscv/Kconfig.erratas b/arch/riscv/Kconfig.erratas
index c521c2ae2de2..12036e65648e 100644
--- a/arch/riscv/Kconfig.erratas
+++ b/arch/riscv/Kconfig.erratas
@@ -33,4 +33,24 @@ config ERRATA_SIFIVE_CIP_1200
If you don't know what to do here, say "Y".
+config ERRATA_THEAD
+ bool "T-HEAD errata"
+ select RISCV_ALTERNATIVE
+ help
+ All T-HEAD errata Kconfig depend on this Kconfig. Disabling
+ this Kconfig will disable all T-HEAD errata. Please say "Y"
+ here if your platform uses T-HEAD CPU cores.
+
+ Otherwise, please say "N" here to avoid unnecessary overhead.
+
+config ERRATA_THEAD_PBMT
+ bool "Apply T-Head memory type errata"
+ depends on ERRATA_THEAD && 64BIT
+ default y
+ help
+ This will apply the memory type errata to handle the non-standard
+ memory type bits in page-table-entries on T-Head SoCs.
+
+ If you don't know what to do here, say "Y".
+
endmenu
diff --git a/arch/riscv/errata/Makefile b/arch/riscv/errata/Makefile
index 0ca1c5281a2d..a1055965fbee 100644
--- a/arch/riscv/errata/Makefile
+++ b/arch/riscv/errata/Makefile
@@ -1 +1,2 @@
obj-$(CONFIG_ERRATA_SIFIVE) += sifive/
+obj-$(CONFIG_ERRATA_THEAD) += thead/
diff --git a/arch/riscv/errata/sifive/errata.c b/arch/riscv/errata/sifive/errata.c
index 3e39587a49dc..672f02b21ce0 100644
--- a/arch/riscv/errata/sifive/errata.c
+++ b/arch/riscv/errata/sifive/errata.c
@@ -88,10 +88,15 @@ void __init_or_module sifive_errata_patch_func(struct alt_entry *begin,
unsigned int stage)
{
struct alt_entry *alt;
- u32 cpu_req_errata = sifive_errata_probe(archid, impid);
+ u32 cpu_req_errata;
u32 cpu_apply_errata = 0;
u32 tmp;
+ if (stage == RISCV_ALTERNATIVES_EARLY_BOOT)
+ return;
+
+ cpu_req_errata = sifive_errata_probe(archid, impid);
+
for (alt = begin; alt < end; alt++) {
if (alt->vendor_id != SIFIVE_VENDOR_ID)
continue;
diff --git a/arch/riscv/errata/thead/Makefile b/arch/riscv/errata/thead/Makefile
new file mode 100644
index 000000000000..2d644e19caef
--- /dev/null
+++ b/arch/riscv/errata/thead/Makefile
@@ -0,0 +1 @@
+obj-y += errata.o
diff --git a/arch/riscv/errata/thead/errata.c b/arch/riscv/errata/thead/errata.c
new file mode 100644
index 000000000000..e5d75270b99c
--- /dev/null
+++ b/arch/riscv/errata/thead/errata.c
@@ -0,0 +1,82 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2021 Heiko Stuebner <[email protected]>
+ */
+
+#include <linux/bug.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/string.h>
+#include <linux/uaccess.h>
+#include <asm/alternative.h>
+#include <asm/cacheflush.h>
+#include <asm/errata_list.h>
+#include <asm/patch.h>
+#include <asm/vendorid_list.h>
+
+struct errata_info {
+ char name[ERRATA_STRING_LENGTH_MAX];
+ bool (*check_func)(unsigned long arch_id, unsigned long impid);
+ unsigned int stage;
+};
+
+static bool errata_mt_check_func(unsigned long arch_id, unsigned long impid)
+{
+ if (arch_id != 0 || impid != 0)
+ return false;
+ return true;
+}
+
+static const struct errata_info errata_list[ERRATA_THEAD_NUMBER] = {
+ {
+ .name = "memory-types",
+ .stage = RISCV_ALTERNATIVES_EARLY_BOOT,
+ .check_func = errata_mt_check_func
+ },
+};
+
+static u32 thead_errata_probe(unsigned int stage, unsigned long archid, unsigned long impid)
+{
+ const struct errata_info *info;
+ u32 cpu_req_errata = 0;
+ int idx;
+
+ for (idx = 0; idx < ERRATA_THEAD_NUMBER; idx++) {
+ info = &errata_list[idx];
+
+ if ((stage == RISCV_ALTERNATIVES_MODULE ||
+ info->stage == stage) && info->check_func(archid, impid))
+ cpu_req_errata |= (1U << idx);
+ }
+
+ return cpu_req_errata;
+}
+
+void __init_or_module thead_errata_patch_func(struct alt_entry *begin, struct alt_entry *end,
+ unsigned long archid, unsigned long impid,
+ unsigned int stage)
+{
+ struct alt_entry *alt;
+ u32 cpu_req_errata = thead_errata_probe(stage, archid, impid);
+ u32 tmp;
+
+ for (alt = begin; alt < end; alt++) {
+ if (alt->vendor_id != THEAD_VENDOR_ID)
+ continue;
+ if (alt->errata_id >= ERRATA_THEAD_NUMBER)
+ continue;
+
+ tmp = (1U << alt->errata_id);
+ if (cpu_req_errata & tmp) {
+ /* On vm-alternatives, the mmu isn't running yet */
+ if (stage == RISCV_ALTERNATIVES_EARLY_BOOT)
+ memcpy((void *)__pa_symbol(alt->old_ptr),
+ (void *)__pa_symbol(alt->alt_ptr), alt->alt_len);
+ else
+ patch_text_nosync(alt->old_ptr, alt->alt_ptr, alt->alt_len);
+ }
+ }
+
+ if (stage == RISCV_ALTERNATIVES_EARLY_BOOT)
+ local_flush_icache_all();
+}
diff --git a/arch/riscv/include/asm/alternative.h b/arch/riscv/include/asm/alternative.h
index 64936356c37c..6511dd73e812 100644
--- a/arch/riscv/include/asm/alternative.h
+++ b/arch/riscv/include/asm/alternative.h
@@ -21,8 +21,10 @@
#define RISCV_ALTERNATIVES_BOOT 0 /* alternatives applied during regular boot */
#define RISCV_ALTERNATIVES_MODULE 1 /* alternatives applied during module-init */
+#define RISCV_ALTERNATIVES_EARLY_BOOT 2 /* alternatives applied before mmu start */
void __init apply_boot_alternatives(void);
+void __init apply_early_boot_alternatives(void);
void apply_module_alternatives(void *start, size_t length);
struct alt_entry {
@@ -41,6 +43,9 @@ struct errata_checkfunc_id {
void sifive_errata_patch_func(struct alt_entry *begin, struct alt_entry *end,
unsigned long archid, unsigned long impid,
unsigned int stage);
+void thead_errata_patch_func(struct alt_entry *begin, struct alt_entry *end,
+ unsigned long archid, unsigned long impid,
+ unsigned int stage);
void riscv_cpufeature_patch_func(struct alt_entry *begin, struct alt_entry *end,
unsigned int stage);
@@ -48,6 +53,7 @@ void riscv_cpufeature_patch_func(struct alt_entry *begin, struct alt_entry *end,
#else /* CONFIG_RISCV_ALTERNATIVE */
static inline void apply_boot_alternatives(void) { }
+static inline void apply_early_boot_alternatives(void) { }
static inline void apply_module_alternatives(void *start, size_t length) { }
#endif /* CONFIG_RISCV_ALTERNATIVE */
diff --git a/arch/riscv/include/asm/errata_list.h b/arch/riscv/include/asm/errata_list.h
index dbfcd9b72bd8..9e2888dbb5b1 100644
--- a/arch/riscv/include/asm/errata_list.h
+++ b/arch/riscv/include/asm/errata_list.h
@@ -14,6 +14,11 @@
#define ERRATA_SIFIVE_NUMBER 2
#endif
+#ifdef CONFIG_ERRATA_THEAD
+#define ERRATA_THEAD_PBMT 0
+#define ERRATA_THEAD_NUMBER 1
+#endif
+
#define CPUFEATURE_SVPBMT 0
#define CPUFEATURE_NUMBER 1
@@ -42,12 +47,51 @@ asm(ALTERNATIVE("sfence.vma %0", "sfence.vma", SIFIVE_VENDOR_ID, \
* in the default case.
*/
#define ALT_SVPBMT_SHIFT 61
+#define ALT_THEAD_PBMT_SHIFT 59
#define ALT_SVPBMT(_val, prot) \
-asm(ALTERNATIVE("li %0, 0\t\nnop", "li %0, %1\t\nslli %0,%0,%2", 0, \
- CPUFEATURE_SVPBMT, CONFIG_RISCV_ISA_SVPBMT) \
+asm(ALTERNATIVE_2("li %0, 0\t\nnop", \
+ "li %0, %1\t\nslli %0,%0,%3", 0, \
+ CPUFEATURE_SVPBMT, CONFIG_RISCV_ISA_SVPBMT, \
+ "li %0, %2\t\nslli %0,%0,%4", THEAD_VENDOR_ID, \
+ ERRATA_THEAD_PBMT, CONFIG_ERRATA_THEAD_PBMT) \
: "=r"(_val) \
: "I"(prot##_SVPBMT >> ALT_SVPBMT_SHIFT), \
- "I"(ALT_SVPBMT_SHIFT))
+ "I"(prot##_THEAD >> ALT_THEAD_PBMT_SHIFT), \
+ "I"(ALT_SVPBMT_SHIFT), \
+ "I"(ALT_THEAD_PBMT_SHIFT))
+
+#ifdef CONFIG_ERRATA_THEAD_PBMT
+/*
+ * IO/NOCACHE memory types are handled together with svpbmt,
+ * so on T-Head chips, check if no other memory type is set,
+ * and set the non-0 PMA type if applicable.
+ */
+#define ALT_THEAD_PMA(_val) \
+asm volatile(ALTERNATIVE( \
+ "nop\n\t" \
+ "nop\n\t" \
+ "nop\n\t" \
+ "nop\n\t" \
+ "nop\n\t" \
+ "nop\n\t" \
+ "nop", \
+ "li t3, %2\n\t" \
+ "slli t3, t3, %4\n\t" \
+ "and t3, %0, t3\n\t" \
+ "bne t3, zero, 2f\n\t" \
+ "li t3, %3\n\t" \
+ "slli t3, t3, %4\n\t" \
+ "or %0, %0, t3\n\t" \
+ "2:", THEAD_VENDOR_ID, \
+ ERRATA_THEAD_PBMT, CONFIG_ERRATA_THEAD_PBMT) \
+ : "+r"(_val) \
+ : "0"(_val), \
+ "I"(_PAGE_MTMASK_THEAD >> ALT_THEAD_PBMT_SHIFT), \
+ "I"(_PAGE_PMA_THEAD >> ALT_THEAD_PBMT_SHIFT), \
+ "I"(ALT_THEAD_PBMT_SHIFT))
+#else
+#define ALT_THEAD_PMA(_val)
+#endif
#endif /* __ASSEMBLY__ */
diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/pgtable-64.h
index 2354501f0203..e4ff3e0ab887 100644
--- a/arch/riscv/include/asm/pgtable-64.h
+++ b/arch/riscv/include/asm/pgtable-64.h
@@ -86,6 +86,18 @@ typedef struct {
#define _PAGE_IO_SVPBMT (1UL << 62)
#define _PAGE_MTMASK_SVPBMT (_PAGE_NOCACHE_SVPBMT | _PAGE_IO_SVPBMT)
+/*
+ * [63:59] T-Head Memory Type definitions:
+ *
+ * 00000 - NC Weakly-ordered, Non-cacheable, Non-bufferable, Non-shareable, Non-trustable
+ * 01110 - PMA Weakly-ordered, Cacheable, Bufferable, Shareable, Non-trustable
+ * 10000 - IO Strongly-ordered, Non-cacheable, Non-bufferable, Non-shareable, Non-trustable
+ */
+#define _PAGE_PMA_THEAD ((1UL << 62) | (1UL << 61) | (1UL << 60))
+#define _PAGE_NOCACHE_THEAD 0UL
+#define _PAGE_IO_THEAD (1UL << 63)
+#define _PAGE_MTMASK_THEAD (_PAGE_PMA_THEAD | _PAGE_IO_THEAD | (1UL << 59))
+
static inline u64 riscv_page_mtmask(void)
{
u64 val;
@@ -193,7 +205,11 @@ static inline bool mm_pud_folded(struct mm_struct *mm)
static inline pmd_t pfn_pmd(unsigned long pfn, pgprot_t prot)
{
- return __pmd((pfn << _PAGE_PFN_SHIFT) | pgprot_val(prot));
+ unsigned long prot_val = pgprot_val(prot);
+
+ ALT_THEAD_PMA(prot_val);
+
+ return __pmd((pfn << _PAGE_PFN_SHIFT) | prot_val);
}
static inline unsigned long _pmd_pfn(pmd_t pmd)
diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index c55341b72de1..6f0a260d3f2c 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -250,7 +250,11 @@ static inline void pmd_clear(pmd_t *pmdp)
static inline pgd_t pfn_pgd(unsigned long pfn, pgprot_t prot)
{
- return __pgd((pfn << _PAGE_PFN_SHIFT) | pgprot_val(prot));
+ unsigned long prot_val = pgprot_val(prot);
+
+ ALT_THEAD_PMA(prot_val);
+
+ return __pgd((pfn << _PAGE_PFN_SHIFT) | prot_val);
}
static inline unsigned long _pgd_pfn(pgd_t pgd)
@@ -289,7 +293,11 @@ static inline unsigned long pte_pfn(pte_t pte)
/* Constructs a page table entry */
static inline pte_t pfn_pte(unsigned long pfn, pgprot_t prot)
{
- return __pte((pfn << _PAGE_PFN_SHIFT) | pgprot_val(prot));
+ unsigned long prot_val = pgprot_val(prot);
+
+ ALT_THEAD_PMA(prot_val);
+
+ return __pte((pfn << _PAGE_PFN_SHIFT) | prot_val);
}
#define mk_pte(page, prot) pfn_pte(page_to_pfn(page), prot)
@@ -398,7 +406,11 @@ static inline int pmd_protnone(pmd_t pmd)
/* Modify page protection bits */
static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
{
- return __pte((pte_val(pte) & _PAGE_CHG_MASK) | pgprot_val(newprot));
+ unsigned long newprot_val = pgprot_val(newprot);
+
+ ALT_THEAD_PMA(newprot_val);
+
+ return __pte((pte_val(pte) & _PAGE_CHG_MASK) | newprot_val);
}
#define pgd_ERROR(e) \
diff --git a/arch/riscv/include/asm/vendorid_list.h b/arch/riscv/include/asm/vendorid_list.h
index 9d934215b3c8..cb89af3f0704 100644
--- a/arch/riscv/include/asm/vendorid_list.h
+++ b/arch/riscv/include/asm/vendorid_list.h
@@ -6,5 +6,6 @@
#define ASM_VENDOR_LIST_H
#define SIFIVE_VENDOR_ID 0x489
+#define THEAD_VENDOR_ID 0x5b7
#endif
diff --git a/arch/riscv/kernel/alternative.c b/arch/riscv/kernel/alternative.c
index 27f722ae452b..e7980bf17e8b 100644
--- a/arch/riscv/kernel/alternative.c
+++ b/arch/riscv/kernel/alternative.c
@@ -42,6 +42,11 @@ static void __init_or_module riscv_fill_cpu_mfr_info(struct cpu_manufacturer_inf
case SIFIVE_VENDOR_ID:
cpu_mfr_info->vendor_patch_func = sifive_errata_patch_func;
break;
+#endif
+#ifdef CONFIG_ERRATA_THEAD
+ case THEAD_VENDOR_ID:
+ cpu_mfr_info->vendor_patch_func = thead_errata_patch_func;
+ break;
#endif
default:
cpu_mfr_info->vendor_patch_func = NULL;
@@ -82,6 +87,13 @@ void __init apply_boot_alternatives(void)
RISCV_ALTERNATIVES_BOOT);
}
+void __init apply_early_boot_alternatives(void)
+{
+ _apply_alternatives((struct alt_entry *)__alt_start,
+ (struct alt_entry *)__alt_end,
+ RISCV_ALTERNATIVES_EARLY_BOOT);
+}
+
#ifdef CONFIG_MODULES
void apply_module_alternatives(void *start, size_t length)
{
diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c
index f514b949c6a7..dea3ea19deee 100644
--- a/arch/riscv/kernel/cpufeature.c
+++ b/arch/riscv/kernel/cpufeature.c
@@ -254,7 +254,12 @@ struct cpufeature_info {
static bool __init_or_module cpufeature_svpbmt_check_func(unsigned int stage)
{
#ifdef CONFIG_RISCV_ISA_SVPBMT
- return riscv_isa_extension_available(NULL, SVPBMT);
+ switch (stage) {
+ case RISCV_ALTERNATIVES_EARLY_BOOT:
+ return false;
+ default:
+ return riscv_isa_extension_available(NULL, SVPBMT);
+ }
#endif
return false;
diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
index 9535bea8688c..1d35a0667db3 100644
--- a/arch/riscv/mm/init.c
+++ b/arch/riscv/mm/init.c
@@ -935,6 +935,7 @@ asmlinkage void __init setup_vm(uintptr_t dtb_pa)
BUG_ON((kernel_map.virt_addr + kernel_map.size) > ADDRESS_SPACE_END - SZ_4K);
#endif
+ apply_early_boot_alternatives();
pt_ops_set_early();
/* Setup early PGD for fixmap */
--
2.35.1
On Wed, 13 Apr 2022 at 05:03, Heiko Stuebner <[email protected]> wrote:
>
> Future features may need to be applied at a different
> time during boot, so allow defining stages for alternatives
> and handling them differently depending on the stage.
>
> Also make the alternatives-location more flexible so that
> future stages may provide their own location.
>
> Signed-off-by: Heiko Stuebner <[email protected]>
Reviewed-by: Philipp Tomsich <[email protected]>
On Wed, 13 Apr 2022 at 05:06, Heiko Stuebner <[email protected]> wrote:
>
> Some current cpus based on T-Head cores implement memory-types
> way different than described in the svpbmt spec even going
> so far as using PTE bits marked as reserved.
>
> Add the T-Head vendor-id and necessary errata code to
> replace the affected instructions.
>
> Signed-off-by: Heiko Stuebner <[email protected]>
> Tested-by: Samuel Holland <[email protected]>
Reviewed-by: Philipp Tomsich <[email protected]>
On Wed, 13 Apr 2022 at 05:04, Heiko Stuebner <[email protected]> wrote:
>
> On rv32 the PFN part of PTEs is defined to use bits [xlen-1:10]
> while on rv64 it is defined to use bits [53:10], leaving [63:54]
> as reserved.
>
> With upcoming optional extensions like svpbmt these previously
> reserved bits will get used so simply right-shifting the PTE
> to get the PFN won't be enough.
>
> So introduce a _PAGE_PFN_MASK constant to mask the correct bits
> for both rv32 and rv64 before shifting.
>
> Signed-off-by: Heiko Stuebner <[email protected]>
Reviewed-by: Philipp Tomsich <[email protected]>
On Wed, 13 Apr 2022 at 05:03, Heiko Stuebner <[email protected]> wrote:
>
> Instructions are opportunistically compressed by the RISC-V assembler
> when possible, but in alternatives-blocks both the old and new content
> need to be the same size, so having the toolchain do somewhat random
> optimizations will cause strange side-effects like
> "attempt to move .org backwards" compile-time errors.
>
> Already a simple "and" used in alternatives assembly will cause these
> mismatched code sizes.
>
> So prevent compressed instructions to be generated in alternatives-
> code and use option-push and -pop to only limit this to the relevant
> code blocks
>
> Signed-off-by: Heiko Stuebner <[email protected]>
> Reviewed-by: Christoph Hellwig <[email protected]>
Reviewed-by: Philipp Tomsich <[email protected]>
On Wed, 13 Apr 2022 at 05:05, Heiko Stuebner <[email protected]> wrote:
>
> If not defined in the arch, FIXMAP_PAGE_IO defaults to PAGE_KERNEL_IO,
> which we defined when adding the svpbmt implementation.
>
> So drop the FIXMAP_PAGE_IO riscv define.
>
> Signed-off-by: Heiko Stuebner <[email protected]>
> Reviewed-by: Christoph Hellwig <[email protected]>
Reviewed-by: Philipp Tomsich <[email protected]>