From: Guo Ren <[email protected]>
This patch series adds independent irq/softirq stacks to decrease the
press of the thread stack. Also, add a thread STACK_SIZE config for
users to adjust the proper size during compile time.
This patch series belonged to the generic entry, which has been merged
to for-next now.
v11:
- Rebase on palmer/for-next (20230324)
- Separate from generic entry patch series.
v10:
https://lore.kernel.org/linux-riscv/[email protected]/
- Rebase on palmer/for-next branch (20221208)
- Remove unrelated patches from the series (Suggested-by: Bjorn)
- Fixup Typos.
v9:
https://lore.kernel.org/linux-riscv/[email protected]/
- Fixup NR_syscalls check (by Ben Hutchings)
- Add Tested-by: Jisheng Zhang
v8:
https://lore.kernel.org/linux-riscv/[email protected]/
- Rebase on palmer/for-next branch (20221102)
- Add save/restore_from_x5_to_x31 .macro (JishengZhang)
- Consolidate ret_from_kernel_thread into ret_from_fork (JishengZhang)
- Optimize __noinstr_section comment (JiangshanLai)
v7:
https://lore.kernel.org/linux-riscv/[email protected]/
- Fixup regs_irqs_disabled with SR_PIE
- Optimize stackleak_erase -> stackleak_erase_on_task_stack (Thx Mark
Rutland)
- Add BUG_ON(!irqs_disabled()) in trap handlers
- Using regs_irqs_disabled in __do_page_fault
- Remove unnecessary irq disable in ret_from_exception and add comment
v6:
https://lore.kernel.org/linux-riscv/[email protected]/
- Use THEAD_SIZE_ORDER for thread size adjustment in kconfig (Thx Arnd)
- Move call_on_stack to inline style (Thx Peter Zijlstra)
- Fixup fp chain broken (Thx Chen Zhongjin)
- Remove common entry modification, and fixup page_fault entry (Thx
Peter Zijlstra)
- Treat some traps as nmi entry (Thx Peter Zijlstra)
v5:
https://lore.kernel.org/linux-riscv/[email protected]/
- Add riscv own stackleak patch instead of generic entry modification
(by Mark Rutland)
- Add EXPERT dependency for THREAD_SIZE (by Arnd)
- Add EXPERT dependency for IRQ_STACK (by Sebastian, David Laight)
- Corrected __trap_section (by Peter Zijlstra)
- Add Tested-by (Yipeng Zou)
- Use CONFIG_SOFTIRQ_ON_OWN_STACK replace "#ifndef CONFIG_PREEMPT_RT"
- Fixup systrace_enter compile error
- Fixup exit_to_user_mode_prepare preempt_disable warning
V4:
https://lore.kernel.org/linux-riscv/[email protected]/
- Fixup entry.S with "la" bug (by Conor.Dooley)
- Fixup missing noinstr bug (by Peter Zijlstra)
V3:
https://lore.kernel.org/linux-riscv/[email protected]/
- Fixup CONFIG_COMPAT=n compile error
- Add THREAD_SIZE_ORDER config
- Optimize elf_kexec.c warning fixup
- Add static to irq_stack_ptr definition
V2:
https://lore.kernel.org/linux-riscv/[email protected]/
- Fixup compile error by include "riscv: ptrace: Remove duplicate
operation"
- Fixup compile warning
Reported-by: kernel test robot <[email protected]>
- Add test repo link in cover letter
V1:
https://lore.kernel.org/linux-riscv/[email protected]/
Guo Ren (3):
riscv: stack: Support HAVE_IRQ_EXIT_ON_IRQ_STACK
riscv: stack: Support HAVE_SOFTIRQ_ON_OWN_STACK
riscv: stack: Add config of thread stack size
arch/riscv/Kconfig | 19 ++++++++
arch/riscv/include/asm/thread_info.h | 14 ++----
arch/riscv/include/asm/vmap_stack.h | 28 ++++++++++++
arch/riscv/kernel/irq.c | 66 ++++++++++++++++++++++++++++
arch/riscv/kernel/traps.c | 38 +++++++++++++++-
5 files changed, 152 insertions(+), 13 deletions(-)
create mode 100644 arch/riscv/include/asm/vmap_stack.h
--
2.36.1
From: Guo Ren <[email protected]>
Add independent irq stacks for percpu to prevent kernel stack overflows.
It is also compatible with VMAP_STACK by implementing
arch_alloc_vmap_stack. Many architectures have supported
HAVE_IRQ_EXIT_ON_IRQ_STACK, riscv should follow up.
Tested-by: Jisheng Zhang <[email protected]>
Signed-off-by: Guo Ren <[email protected]>
Signed-off-by: Guo Ren <[email protected]>
---
arch/riscv/Kconfig | 8 ++++++
arch/riscv/include/asm/thread_info.h | 2 ++
arch/riscv/include/asm/vmap_stack.h | 28 ++++++++++++++++++++
arch/riscv/kernel/irq.c | 32 +++++++++++++++++++++++
arch/riscv/kernel/traps.c | 38 ++++++++++++++++++++++++++--
5 files changed, 106 insertions(+), 2 deletions(-)
create mode 100644 arch/riscv/include/asm/vmap_stack.h
diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index e6df999f08cc..eb3c40d3a21b 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -493,6 +493,14 @@ config FPU
If you don't know what to do here, say Y.
+config IRQ_STACKS
+ bool "Independent irq stacks" if EXPERT
+ default y
+ select HAVE_IRQ_EXIT_ON_IRQ_STACK
+ help
+ Add independent irq stacks for percpu to prevent kernel stack overflows.
+ We may save some memory footprint by disabling IRQ_STACKS.
+
endmenu # "Platform type"
menu "Kernel features"
diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h
index e0d202134b44..ab60593eed99 100644
--- a/arch/riscv/include/asm/thread_info.h
+++ b/arch/riscv/include/asm/thread_info.h
@@ -40,6 +40,8 @@
#define OVERFLOW_STACK_SIZE SZ_4K
#define SHADOW_OVERFLOW_STACK_SIZE (1024)
+#define IRQ_STACK_SIZE THREAD_SIZE
+
#ifndef __ASSEMBLY__
extern long shadow_stack[SHADOW_OVERFLOW_STACK_SIZE / sizeof(long)];
diff --git a/arch/riscv/include/asm/vmap_stack.h b/arch/riscv/include/asm/vmap_stack.h
new file mode 100644
index 000000000000..3fbf481abf4f
--- /dev/null
+++ b/arch/riscv/include/asm/vmap_stack.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+// Copied from arch/arm64/include/asm/vmap_stack.h.
+#ifndef _ASM_RISCV_VMAP_STACK_H
+#define _ASM_RISCV_VMAP_STACK_H
+
+#include <linux/bug.h>
+#include <linux/gfp.h>
+#include <linux/kconfig.h>
+#include <linux/vmalloc.h>
+#include <linux/pgtable.h>
+#include <asm/thread_info.h>
+
+/*
+ * To ensure that VMAP'd stack overflow detection works correctly, all VMAP'd
+ * stacks need to have the same alignment.
+ */
+static inline unsigned long *arch_alloc_vmap_stack(size_t stack_size, int node)
+{
+ void *p;
+
+ BUILD_BUG_ON(!IS_ENABLED(CONFIG_VMAP_STACK));
+
+ p = __vmalloc_node(stack_size, THREAD_ALIGN, THREADINFO_GFP, node,
+ __builtin_return_address(0));
+ return kasan_reset_tag(p);
+}
+
+#endif /* _ASM_RISCV_VMAP_STACK_H */
diff --git a/arch/riscv/kernel/irq.c b/arch/riscv/kernel/irq.c
index 7207fa08d78f..52f2fa44a9bb 100644
--- a/arch/riscv/kernel/irq.c
+++ b/arch/riscv/kernel/irq.c
@@ -9,6 +9,37 @@
#include <linux/irqchip.h>
#include <linux/seq_file.h>
#include <asm/smp.h>
+#include <asm/vmap_stack.h>
+
+#ifdef CONFIG_IRQ_STACKS
+DEFINE_PER_CPU(ulong *, irq_stack_ptr);
+
+#ifdef CONFIG_VMAP_STACK
+static void init_irq_stacks(void)
+{
+ int cpu;
+ ulong *p;
+
+ for_each_possible_cpu(cpu) {
+ p = arch_alloc_vmap_stack(IRQ_STACK_SIZE, cpu_to_node(cpu));
+ per_cpu(irq_stack_ptr, cpu) = p;
+ }
+}
+#else
+/* irq stack only needs to be 16 byte aligned - not IRQ_STACK_SIZE aligned. */
+DEFINE_PER_CPU_ALIGNED(ulong [IRQ_STACK_SIZE/sizeof(ulong)], irq_stack);
+
+static void init_irq_stacks(void)
+{
+ int cpu;
+
+ for_each_possible_cpu(cpu)
+ per_cpu(irq_stack_ptr, cpu) = per_cpu(irq_stack, cpu);
+}
+#endif /* CONFIG_VMAP_STACK */
+#else
+static void init_irq_stacks(void) {}
+#endif /* CONFIG_IRQ_STACKS */
int arch_show_interrupts(struct seq_file *p, int prec)
{
@@ -18,6 +49,7 @@ int arch_show_interrupts(struct seq_file *p, int prec)
void __init init_IRQ(void)
{
+ init_irq_stacks();
irqchip_init();
if (!handle_arch_irq)
panic("No interrupt controller found.");
diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c
index 1f4e37be7eb3..b69933ab6bf8 100644
--- a/arch/riscv/kernel/traps.c
+++ b/arch/riscv/kernel/traps.c
@@ -305,16 +305,50 @@ asmlinkage __visible noinstr void do_page_fault(struct pt_regs *regs)
}
#endif
-asmlinkage __visible noinstr void do_irq(struct pt_regs *regs)
+static void noinstr handle_riscv_irq(struct pt_regs *regs)
{
struct pt_regs *old_regs;
- irqentry_state_t state = irqentry_enter(regs);
irq_enter_rcu();
old_regs = set_irq_regs(regs);
handle_arch_irq(regs);
set_irq_regs(old_regs);
irq_exit_rcu();
+}
+
+#ifdef CONFIG_IRQ_STACKS
+DECLARE_PER_CPU(ulong *, irq_stack_ptr);
+#endif
+
+asmlinkage void noinstr do_irq(struct pt_regs *regs)
+{
+ irqentry_state_t state = irqentry_enter(regs);
+#ifdef CONFIG_IRQ_STACKS
+ if (on_thread_stack()) {
+ ulong *sp = per_cpu(irq_stack_ptr, smp_processor_id())
+ + IRQ_STACK_SIZE/sizeof(ulong);
+ __asm__ __volatile(
+ "addi sp, sp, -"RISCV_SZPTR "\n"
+ REG_S" ra, (sp) \n"
+ "addi sp, sp, -"RISCV_SZPTR "\n"
+ REG_S" s0, (sp) \n"
+ "addi s0, sp, 2*"RISCV_SZPTR "\n"
+ "move sp, %[sp] \n"
+ "move a0, %[regs] \n"
+ "call handle_riscv_irq \n"
+ "addi sp, s0, -2*"RISCV_SZPTR"\n"
+ REG_L" s0, (sp) \n"
+ "addi sp, sp, "RISCV_SZPTR "\n"
+ REG_L" ra, (sp) \n"
+ "addi sp, sp, "RISCV_SZPTR "\n"
+ :
+ : [sp] "r" (sp), [regs] "r" (regs)
+ : "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7",
+ "t0", "t1", "t2", "t3", "t4", "t5", "t6",
+ "memory");
+ } else
+#endif
+ handle_riscv_irq(regs);
irqentry_exit(regs, state);
}
--
2.36.1
From: Guo Ren <[email protected]>
Add the HAVE_SOFTIRQ_ON_OWN_STACK feature for the IRQ_STACKS config. The
irq and softirq use the same independent irq_stack of percpu by time
division multiplexing.
Tested-by: Jisheng Zhang <[email protected]>
Signed-off-by: Guo Ren <[email protected]>
Signed-off-by: Guo Ren <[email protected]>
---
arch/riscv/Kconfig | 7 ++++---
arch/riscv/kernel/irq.c | 34 ++++++++++++++++++++++++++++++++++
2 files changed, 38 insertions(+), 3 deletions(-)
diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index eb3c40d3a21b..7b10af7d2479 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -494,12 +494,13 @@ config FPU
If you don't know what to do here, say Y.
config IRQ_STACKS
- bool "Independent irq stacks" if EXPERT
+ bool "Independent irq & softirq stacks" if EXPERT
default y
select HAVE_IRQ_EXIT_ON_IRQ_STACK
+ select HAVE_SOFTIRQ_ON_OWN_STACK
help
- Add independent irq stacks for percpu to prevent kernel stack overflows.
- We may save some memory footprint by disabling IRQ_STACKS.
+ Add independent irq & softirq stacks for percpu to prevent kernel stack
+ overflows. We may save some memory footprint by disabling IRQ_STACKS.
endmenu # "Platform type"
diff --git a/arch/riscv/kernel/irq.c b/arch/riscv/kernel/irq.c
index 52f2fa44a9bb..0592c2e99b5f 100644
--- a/arch/riscv/kernel/irq.c
+++ b/arch/riscv/kernel/irq.c
@@ -10,6 +10,8 @@
#include <linux/seq_file.h>
#include <asm/smp.h>
#include <asm/vmap_stack.h>
+#include <asm/softirq_stack.h>
+#include <asm/stacktrace.h>
#ifdef CONFIG_IRQ_STACKS
DEFINE_PER_CPU(ulong *, irq_stack_ptr);
@@ -37,6 +39,38 @@ static void init_irq_stacks(void)
per_cpu(irq_stack_ptr, cpu) = per_cpu(irq_stack, cpu);
}
#endif /* CONFIG_VMAP_STACK */
+
+#ifdef CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK
+void do_softirq_own_stack(void)
+{
+#ifdef CONFIG_IRQ_STACKS
+ if (on_thread_stack()) {
+ ulong *sp = per_cpu(irq_stack_ptr, smp_processor_id())
+ + IRQ_STACK_SIZE/sizeof(ulong);
+ __asm__ __volatile(
+ "addi sp, sp, -"RISCV_SZPTR "\n"
+ REG_S" ra, (sp) \n"
+ "addi sp, sp, -"RISCV_SZPTR "\n"
+ REG_S" s0, (sp) \n"
+ "addi s0, sp, 2*"RISCV_SZPTR "\n"
+ "move sp, %[sp] \n"
+ "call __do_softirq \n"
+ "addi sp, s0, -2*"RISCV_SZPTR"\n"
+ REG_L" s0, (sp) \n"
+ "addi sp, sp, "RISCV_SZPTR "\n"
+ REG_L" ra, (sp) \n"
+ "addi sp, sp, "RISCV_SZPTR "\n"
+ :
+ : [sp] "r" (sp)
+ : "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7",
+ "t0", "t1", "t2", "t3", "t4", "t5", "t6",
+ "memory");
+ } else
+#endif
+ __do_softirq();
+}
+#endif /* CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK */
+
#else
static void init_irq_stacks(void) {}
#endif /* CONFIG_IRQ_STACKS */
--
2.36.1
From: Guo Ren <[email protected]>
The commit 0cac21b02ba5 ("riscv: use 16KB kernel stack on 64-bit")
increases the thread size mandatory, but some scenarios, such as D1 with
a small memory footprint, would suffer from that. After independent irq
stack support, let's give users a choice to determine their custom stack
size.
Link: https://lore.kernel.org/linux-riscv/[email protected]/
Suggested-by: Arnd Bergmann <[email protected]>
Tested-by: Jisheng Zhang <[email protected]>
Signed-off-by: Guo Ren <[email protected]>
Signed-off-by: Guo Ren <[email protected]>
---
arch/riscv/Kconfig | 10 ++++++++++
arch/riscv/include/asm/thread_info.h | 12 +-----------
2 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index 7b10af7d2479..f58a8e37f1d5 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -502,6 +502,16 @@ config IRQ_STACKS
Add independent irq & softirq stacks for percpu to prevent kernel stack
overflows. We may save some memory footprint by disabling IRQ_STACKS.
+config THREAD_SIZE_ORDER
+ int "Kernel stack size (in power-of-two numbers of page size)" if VMAP_STACK && EXPERT
+ range 0 4
+ default 1 if 32BIT && !KASAN
+ default 3 if 64BIT && KASAN
+ default 2
+ help
+ Specify the Pages of thread stack size (from 4KB to 64KB), which also
+ affects irq stack size, which is equal to thread stack size.
+
endmenu # "Platform type"
menu "Kernel features"
diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h
index ab60593eed99..41556ee84290 100644
--- a/arch/riscv/include/asm/thread_info.h
+++ b/arch/riscv/include/asm/thread_info.h
@@ -11,18 +11,8 @@
#include <asm/page.h>
#include <linux/const.h>
-#ifdef CONFIG_KASAN
-#define KASAN_STACK_ORDER 1
-#else
-#define KASAN_STACK_ORDER 0
-#endif
-
/* thread information allocation */
-#ifdef CONFIG_64BIT
-#define THREAD_SIZE_ORDER (2 + KASAN_STACK_ORDER)
-#else
-#define THREAD_SIZE_ORDER (1 + KASAN_STACK_ORDER)
-#endif
+#define THREAD_SIZE_ORDER CONFIG_THREAD_SIZE_ORDER
#define THREAD_SIZE (PAGE_SIZE << THREAD_SIZE_ORDER)
/*
--
2.36.1
On Fri, Mar 24, 2023 at 03:12:37AM -0400, [email protected] wrote:
> From: Guo Ren <[email protected]>
>
> Add independent irq stacks for percpu to prevent kernel stack overflows.
> It is also compatible with VMAP_STACK by implementing
> arch_alloc_vmap_stack. Many architectures have supported
> HAVE_IRQ_EXIT_ON_IRQ_STACK, riscv should follow up.
>
> Tested-by: Jisheng Zhang <[email protected]>
> Signed-off-by: Guo Ren <[email protected]>
> Signed-off-by: Guo Ren <[email protected]>
> --- a/arch/riscv/kernel/irq.c
> +++ b/arch/riscv/kernel/irq.c
> @@ -9,6 +9,37 @@
> #include <linux/irqchip.h>
> #include <linux/seq_file.h>
> #include <asm/smp.h>
> +#include <asm/vmap_stack.h>
> +
> +#ifdef CONFIG_IRQ_STACKS
> +DEFINE_PER_CPU(ulong *, irq_stack_ptr);
btw, sparse is complaining about this variable:
../arch/riscv/kernel/irq.c:15:1: warning: symbol '__pcpu_scope_irq_stack_ptr' was not declared. Should it be static?
I'm not immediately sure why that is the case, but should be
reproducible with gcc-12 allmodconfig.
Thanks,
Conor.
> +
> +#ifdef CONFIG_VMAP_STACK
> +static void init_irq_stacks(void)
> +{
> + int cpu;
> + ulong *p;
> +
> + for_each_possible_cpu(cpu) {
> + p = arch_alloc_vmap_stack(IRQ_STACK_SIZE, cpu_to_node(cpu));
> + per_cpu(irq_stack_ptr, cpu) = p;
> + }
> +}
> +#else
> +/* irq stack only needs to be 16 byte aligned - not IRQ_STACK_SIZE aligned. */
> +DEFINE_PER_CPU_ALIGNED(ulong [IRQ_STACK_SIZE/sizeof(ulong)], irq_stack);
> +
> +static void init_irq_stacks(void)
> +{
> + int cpu;
> +
> + for_each_possible_cpu(cpu)
> + per_cpu(irq_stack_ptr, cpu) = per_cpu(irq_stack, cpu);
> +}
> +#endif /* CONFIG_VMAP_STACK */
> +#else
> +static void init_irq_stacks(void) {}
> +#endif /* CONFIG_IRQ_STACKS */
>
> int arch_show_interrupts(struct seq_file *p, int prec)
> {
> @@ -18,6 +49,7 @@ int arch_show_interrupts(struct seq_file *p, int prec)
>
> void __init init_IRQ(void)
> {
> + init_irq_stacks();
> irqchip_init();
> if (!handle_arch_irq)
> panic("No interrupt controller found.");
> diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c
> index 1f4e37be7eb3..b69933ab6bf8 100644
> --- a/arch/riscv/kernel/traps.c
> +++ b/arch/riscv/kernel/traps.c
> @@ -305,16 +305,50 @@ asmlinkage __visible noinstr void do_page_fault(struct pt_regs *regs)
> }
> #endif
>
> -asmlinkage __visible noinstr void do_irq(struct pt_regs *regs)
> +static void noinstr handle_riscv_irq(struct pt_regs *regs)
> {
> struct pt_regs *old_regs;
> - irqentry_state_t state = irqentry_enter(regs);
>
> irq_enter_rcu();
> old_regs = set_irq_regs(regs);
> handle_arch_irq(regs);
> set_irq_regs(old_regs);
> irq_exit_rcu();
> +}
> +
> +#ifdef CONFIG_IRQ_STACKS
> +DECLARE_PER_CPU(ulong *, irq_stack_ptr);
> +#endif
> +
> +asmlinkage void noinstr do_irq(struct pt_regs *regs)
> +{
> + irqentry_state_t state = irqentry_enter(regs);
> +#ifdef CONFIG_IRQ_STACKS
> + if (on_thread_stack()) {
> + ulong *sp = per_cpu(irq_stack_ptr, smp_processor_id())
> + + IRQ_STACK_SIZE/sizeof(ulong);
> + __asm__ __volatile(
> + "addi sp, sp, -"RISCV_SZPTR "\n"
> + REG_S" ra, (sp) \n"
> + "addi sp, sp, -"RISCV_SZPTR "\n"
> + REG_S" s0, (sp) \n"
> + "addi s0, sp, 2*"RISCV_SZPTR "\n"
> + "move sp, %[sp] \n"
> + "move a0, %[regs] \n"
> + "call handle_riscv_irq \n"
> + "addi sp, s0, -2*"RISCV_SZPTR"\n"
> + REG_L" s0, (sp) \n"
> + "addi sp, sp, "RISCV_SZPTR "\n"
> + REG_L" ra, (sp) \n"
> + "addi sp, sp, "RISCV_SZPTR "\n"
> + :
> + : [sp] "r" (sp), [regs] "r" (regs)
> + : "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7",
> + "t0", "t1", "t2", "t3", "t4", "t5", "t6",
> + "memory");
> + } else
> +#endif
> + handle_riscv_irq(regs);
>
> irqentry_exit(regs, state);
> }
> --
> 2.36.1
>
>
On Mon, Mar 27, 2023 at 7:30 PM Conor Dooley <[email protected]> wrote:
>
> On Fri, Mar 24, 2023 at 03:12:37AM -0400, [email protected] wrote:
> > From: Guo Ren <[email protected]>
> >
> > Add independent irq stacks for percpu to prevent kernel stack overflows.
> > It is also compatible with VMAP_STACK by implementing
> > arch_alloc_vmap_stack. Many architectures have supported
> > HAVE_IRQ_EXIT_ON_IRQ_STACK, riscv should follow up.
> >
> > Tested-by: Jisheng Zhang <[email protected]>
> > Signed-off-by: Guo Ren <[email protected]>
> > Signed-off-by: Guo Ren <[email protected]>
>
> > --- a/arch/riscv/kernel/irq.c
> > +++ b/arch/riscv/kernel/irq.c
> > @@ -9,6 +9,37 @@
> > #include <linux/irqchip.h>
> > #include <linux/seq_file.h>
> > #include <asm/smp.h>
> > +#include <asm/vmap_stack.h>
> > +
> > +#ifdef CONFIG_IRQ_STACKS
> > +DEFINE_PER_CPU(ulong *, irq_stack_ptr);
>
> btw, sparse is complaining about this variable:
> ../arch/riscv/kernel/irq.c:15:1: warning: symbol '__pcpu_scope_irq_stack_ptr' was not declared. Should it be static?
I declared it in traps.c, maybe I should put it in the vmap_stack.h.
>
> I'm not immediately sure why that is the case, but should be
> reproducible with gcc-12 allmodconfig.
>
> Thanks,
> Conor.
>
> > +
> > +#ifdef CONFIG_VMAP_STACK
> > +static void init_irq_stacks(void)
> > +{
> > + int cpu;
> > + ulong *p;
> > +
> > + for_each_possible_cpu(cpu) {
> > + p = arch_alloc_vmap_stack(IRQ_STACK_SIZE, cpu_to_node(cpu));
> > + per_cpu(irq_stack_ptr, cpu) = p;
> > + }
> > +}
> > +#else
> > +/* irq stack only needs to be 16 byte aligned - not IRQ_STACK_SIZE aligned. */
> > +DEFINE_PER_CPU_ALIGNED(ulong [IRQ_STACK_SIZE/sizeof(ulong)], irq_stack);
> > +
> > +static void init_irq_stacks(void)
> > +{
> > + int cpu;
> > +
> > + for_each_possible_cpu(cpu)
> > + per_cpu(irq_stack_ptr, cpu) = per_cpu(irq_stack, cpu);
> > +}
> > +#endif /* CONFIG_VMAP_STACK */
> > +#else
> > +static void init_irq_stacks(void) {}
> > +#endif /* CONFIG_IRQ_STACKS */
> >
> > int arch_show_interrupts(struct seq_file *p, int prec)
> > {
> > @@ -18,6 +49,7 @@ int arch_show_interrupts(struct seq_file *p, int prec)
> >
> > void __init init_IRQ(void)
> > {
> > + init_irq_stacks();
> > irqchip_init();
> > if (!handle_arch_irq)
> > panic("No interrupt controller found.");
> > diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c
> > index 1f4e37be7eb3..b69933ab6bf8 100644
> > --- a/arch/riscv/kernel/traps.c
> > +++ b/arch/riscv/kernel/traps.c
> > @@ -305,16 +305,50 @@ asmlinkage __visible noinstr void do_page_fault(struct pt_regs *regs)
> > }
> > #endif
> >
> > -asmlinkage __visible noinstr void do_irq(struct pt_regs *regs)
> > +static void noinstr handle_riscv_irq(struct pt_regs *regs)
> > {
> > struct pt_regs *old_regs;
> > - irqentry_state_t state = irqentry_enter(regs);
> >
> > irq_enter_rcu();
> > old_regs = set_irq_regs(regs);
> > handle_arch_irq(regs);
> > set_irq_regs(old_regs);
> > irq_exit_rcu();
> > +}
> > +
> > +#ifdef CONFIG_IRQ_STACKS
> > +DECLARE_PER_CPU(ulong *, irq_stack_ptr);
> > +#endif
I declared it here.
> > +
> > +asmlinkage void noinstr do_irq(struct pt_regs *regs)
> > +{
> > + irqentry_state_t state = irqentry_enter(regs);
> > +#ifdef CONFIG_IRQ_STACKS
> > + if (on_thread_stack()) {
> > + ulong *sp = per_cpu(irq_stack_ptr, smp_processor_id())
> > + + IRQ_STACK_SIZE/sizeof(ulong);
> > + __asm__ __volatile(
> > + "addi sp, sp, -"RISCV_SZPTR "\n"
> > + REG_S" ra, (sp) \n"
> > + "addi sp, sp, -"RISCV_SZPTR "\n"
> > + REG_S" s0, (sp) \n"
> > + "addi s0, sp, 2*"RISCV_SZPTR "\n"
> > + "move sp, %[sp] \n"
> > + "move a0, %[regs] \n"
> > + "call handle_riscv_irq \n"
> > + "addi sp, s0, -2*"RISCV_SZPTR"\n"
> > + REG_L" s0, (sp) \n"
> > + "addi sp, sp, "RISCV_SZPTR "\n"
> > + REG_L" ra, (sp) \n"
> > + "addi sp, sp, "RISCV_SZPTR "\n"
> > + :
> > + : [sp] "r" (sp), [regs] "r" (regs)
> > + : "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7",
> > + "t0", "t1", "t2", "t3", "t4", "t5", "t6",
> > + "memory");
> > + } else
> > +#endif
> > + handle_riscv_irq(regs);
> >
> > irqentry_exit(regs, state);
> > }
> > --
> > 2.36.1
> >
> >
--
Best Regards
Guo Ren
On Mon, Mar 27, 2023 at 09:32:51PM +0800, Guo Ren wrote:
> On Mon, Mar 27, 2023 at 7:30 PM Conor Dooley <[email protected]> wrote:
> >
> > On Fri, Mar 24, 2023 at 03:12:37AM -0400, [email protected] wrote:
> > > From: Guo Ren <[email protected]>
> > >
> > > Add independent irq stacks for percpu to prevent kernel stack overflows.
> > > It is also compatible with VMAP_STACK by implementing
> > > arch_alloc_vmap_stack. Many architectures have supported
> > > HAVE_IRQ_EXIT_ON_IRQ_STACK, riscv should follow up.
> > >
> > > Tested-by: Jisheng Zhang <[email protected]>
> > > Signed-off-by: Guo Ren <[email protected]>
> > > Signed-off-by: Guo Ren <[email protected]>
> >
> > > --- a/arch/riscv/kernel/irq.c
> > > +++ b/arch/riscv/kernel/irq.c
> > > @@ -9,6 +9,37 @@
> > > #include <linux/irqchip.h>
> > > #include <linux/seq_file.h>
> > > #include <asm/smp.h>
> > > +#include <asm/vmap_stack.h>
> > > +
> > > +#ifdef CONFIG_IRQ_STACKS
> > > +DEFINE_PER_CPU(ulong *, irq_stack_ptr);
> >
> > btw, sparse is complaining about this variable:
> > ../arch/riscv/kernel/irq.c:15:1: warning: symbol '__pcpu_scope_irq_stack_ptr' was not declared. Should it be static?
> I declared it in traps.c, maybe I should put it in the vmap_stack.h.
Ahh, I was distracted by the DEFINE_PER_CPU above and didn't look at
where the actual declaration was.. Moving it to a header sounds good to
me, thanks.