This series implements C syscall entry/exit for PPC32. It reuses
the work already done for PPC64.
This series is based on today's merge-test (b6f72fc05389e3fc694bf5a5fa1bbd33f61879e0)
In terms on performance we have the following number of cycles on an
8xx running null_syscall benchmark:
- mainline: 296 cycles
- after patch 4: 283 cycles
- after patch 16: 304 cycles
- after patch 17: 348 cycles
- at the end of the series: 320 cycles
So in summary, we have a degradation of performance of 8% on null_syscall.
I think it is not a big degradation, it is worth it.
v4 was the first mature version.
v5:
- Comments from Nick
- Converted booke DBCR0 handling in C
- Removed convertion of KUAP restore in C (will be done as part of interrupt entry/exit porting to C)
- Added a few more changes in preparatory patches to prepare for interrupt entry/exit in C which will follow
Christophe Leroy (22):
powerpc/32s: Add missing call to kuep_lock on syscall entry
powerpc/32: Always enable data translation on syscall entry
powerpc/32: On syscall entry, enable instruction translation at the
same time as data
powerpc/32: Reorder instructions to avoid using CTR in syscall entry
powerpc/irq: Add helper to set regs->softe
powerpc/irq: Rework helpers that manipulate MSR[EE/RI]
powerpc/irq: Add stub irq_soft_mask_return() for PPC32
powerpc/syscall: Rename syscall_64.c into interrupt.c
powerpc/syscall: Make interrupt.c buildable on PPC32
powerpc/syscall: Use is_compat_task()
powerpc/syscall: Save r3 in regs->orig_r3
powerpc/syscall: Change condition to check MSR_RI
powerpc/32: Always save non volatile GPRs at syscall entry
powerpc/syscall: implement system call entry/exit logic in C for PPC32
powerpc/32: Remove verification of MSR_PR on syscall in the ASM entry
powerpc/syscall: Avoid stack frame in likely part of
system_call_exception()
powerpc/syscall: Do not check unsupported scv vector on PPC32
powerpc/syscall: Remove FULL_REGS verification in
system_call_exception
powerpc/syscall: Optimise checks in beginning of
system_call_exception()
powerpc/syscall: Avoid storing 'current' in another pointer
powerpc/32: Remove the counter in global_dbcr0
powerpc/32: Handle bookE debugging in C in syscall entry/exit
arch/powerpc/include/asm/hw_irq.h | 91 +++--
arch/powerpc/include/asm/ptrace.h | 5 +
arch/powerpc/include/asm/reg.h | 1 +
arch/powerpc/include/asm/reg_booke.h | 3 +
arch/powerpc/kernel/Makefile | 4 +-
arch/powerpc/kernel/entry_32.S | 321 ++----------------
arch/powerpc/kernel/entry_64.S | 2 -
arch/powerpc/kernel/head_32.h | 96 +-----
arch/powerpc/kernel/head_booke.h | 51 +--
.../kernel/{syscall_64.c => interrupt.c} | 120 +++++--
arch/powerpc/kernel/syscalls/syscall.tbl | 20 +-
11 files changed, 218 insertions(+), 496 deletions(-)
rename arch/powerpc/kernel/{syscall_64.c => interrupt.c} (80%)
--
2.25.0
Userspace Execution protection and fast syscall entry were implemented
independently from each other and were both merged in kernel 5.2,
leading to syscall entry missing userspace execution protection.
On syscall entry, execution of user space memory must be
locked in the same way as on exception entry.
Fixes: b86fb88855ea ("powerpc/32: implement fast entry for syscalls on non BOOKE")
Cc: [email protected]
Signed-off-by: Christophe Leroy <[email protected]>
---
arch/powerpc/kernel/entry_32.S | 3 +++
1 file changed, 3 insertions(+)
diff --git a/arch/powerpc/kernel/entry_32.S b/arch/powerpc/kernel/entry_32.S
index b102b40c4988..b1e36602c013 100644
--- a/arch/powerpc/kernel/entry_32.S
+++ b/arch/powerpc/kernel/entry_32.S
@@ -351,6 +351,9 @@ trace_syscall_entry_irq_off:
.globl transfer_to_syscall
transfer_to_syscall:
+#ifdef CONFIG_PPC_BOOK3S_32
+ kuep_lock r11, r12
+#endif
#ifdef CONFIG_TRACE_IRQFLAGS
andi. r12,r9,MSR_EE
beq- trace_syscall_entry_irq_off
--
2.25.0
regs->softe doesn't exist on PPC32.
Add irq_soft_mask_regs_set_state() helper to set regs->softe.
This helper will void on PPC32.
Signed-off-by: Christophe Leroy <[email protected]>
---
arch/powerpc/include/asm/hw_irq.h | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/include/asm/hw_irq.h b/arch/powerpc/include/asm/hw_irq.h
index 614957f74cee..ed0c3b049dfd 100644
--- a/arch/powerpc/include/asm/hw_irq.h
+++ b/arch/powerpc/include/asm/hw_irq.h
@@ -38,6 +38,8 @@
#define PACA_IRQ_MUST_HARD_MASK (PACA_IRQ_EE)
#endif
+#endif /* CONFIG_PPC64 */
+
/*
* flags for paca->irq_soft_mask
*/
@@ -46,8 +48,6 @@
#define IRQS_PMI_DISABLED 2
#define IRQS_ALL_DISABLED (IRQS_DISABLED | IRQS_PMI_DISABLED)
-#endif /* CONFIG_PPC64 */
-
#ifndef __ASSEMBLY__
#ifdef CONFIG_PPC64
@@ -287,6 +287,10 @@ extern void irq_set_pending_from_srr1(unsigned long srr1);
extern void force_external_irq_replay(void);
+static inline void irq_soft_mask_regs_set_state(struct pt_regs *regs, unsigned long val)
+{
+ regs->softe = val;
+}
#else /* CONFIG_PPC64 */
static inline unsigned long arch_local_save_flags(void)
@@ -355,6 +359,9 @@ static inline bool arch_irq_disabled_regs(struct pt_regs *regs)
static inline void may_hard_irq_enable(void) { }
+static inline void irq_soft_mask_regs_set_state(struct pt_regs *regs, unsigned long val)
+{
+}
#endif /* CONFIG_PPC64 */
#define ARCH_IRQ_INIT_FLAGS IRQ_NOREQUEST
--
2.25.0
syscall_64.c will be reused almost as is for PPC32.
As this file also contains functions to handle other types
of interrupts rename it interrupt.c
Signed-off-by: Christophe Leroy <[email protected]>
---
arch/powerpc/kernel/Makefile | 2 +-
arch/powerpc/kernel/{syscall_64.c => interrupt.c} | 0
2 files changed, 1 insertion(+), 1 deletion(-)
rename arch/powerpc/kernel/{syscall_64.c => interrupt.c} (100%)
diff --git a/arch/powerpc/kernel/Makefile b/arch/powerpc/kernel/Makefile
index c173efd66c00..26ff8c6e06b7 100644
--- a/arch/powerpc/kernel/Makefile
+++ b/arch/powerpc/kernel/Makefile
@@ -60,7 +60,7 @@ obj-y := cputable.o syscalls.o \
hw_breakpoint_constraints.o
obj-y += ptrace/
obj-$(CONFIG_PPC64) += setup_64.o \
- paca.o nvram_64.o note.o syscall_64.o
+ paca.o nvram_64.o note.o interrupt.o
obj-$(CONFIG_COMPAT) += sys_ppc32.o signal_32.o
obj-$(CONFIG_VDSO32) += vdso32_wrapper.o
obj-$(CONFIG_PPC_WATCHDOG) += watchdog.o
diff --git a/arch/powerpc/kernel/syscall_64.c b/arch/powerpc/kernel/interrupt.c
similarity index 100%
rename from arch/powerpc/kernel/syscall_64.c
rename to arch/powerpc/kernel/interrupt.c
--
2.25.0
In preparation for porting syscall entry/exit to C, inconditionally
save non volatile general purpose registers.
Commit 965dd3ad3076 ("powerpc/64/syscall: Remove non-volatile GPR save
optimisation") provides detailed explanation.
This increases the number of cycles by 24 cycles on 8xx with
null_syscall benchmark (280 => 304 cycles)
Signed-off-by: Christophe Leroy <[email protected]>
---
arch/powerpc/kernel/entry_32.S | 46 +-----------------------
arch/powerpc/kernel/head_32.h | 2 +-
arch/powerpc/kernel/head_booke.h | 2 +-
arch/powerpc/kernel/syscalls/syscall.tbl | 20 +++--------
4 files changed, 8 insertions(+), 62 deletions(-)
diff --git a/arch/powerpc/kernel/entry_32.S b/arch/powerpc/kernel/entry_32.S
index b1e36602c013..97dc28a68465 100644
--- a/arch/powerpc/kernel/entry_32.S
+++ b/arch/powerpc/kernel/entry_32.S
@@ -351,6 +351,7 @@ trace_syscall_entry_irq_off:
.globl transfer_to_syscall
transfer_to_syscall:
+ SAVE_NVGPRS(r1)
#ifdef CONFIG_PPC_BOOK3S_32
kuep_lock r11, r12
#endif
@@ -614,51 +615,6 @@ ret_from_kernel_syscall:
#endif
_ASM_NOKPROBE_SYMBOL(ret_from_kernel_syscall)
-/*
- * The fork/clone functions need to copy the full register set into
- * the child process. Therefore we need to save all the nonvolatile
- * registers (r13 - r31) before calling the C code.
- */
- .globl ppc_fork
-ppc_fork:
- SAVE_NVGPRS(r1)
- lwz r0,_TRAP(r1)
- rlwinm r0,r0,0,0,30 /* clear LSB to indicate full */
- stw r0,_TRAP(r1) /* register set saved */
- b sys_fork
-
- .globl ppc_vfork
-ppc_vfork:
- SAVE_NVGPRS(r1)
- lwz r0,_TRAP(r1)
- rlwinm r0,r0,0,0,30 /* clear LSB to indicate full */
- stw r0,_TRAP(r1) /* register set saved */
- b sys_vfork
-
- .globl ppc_clone
-ppc_clone:
- SAVE_NVGPRS(r1)
- lwz r0,_TRAP(r1)
- rlwinm r0,r0,0,0,30 /* clear LSB to indicate full */
- stw r0,_TRAP(r1) /* register set saved */
- b sys_clone
-
- .globl ppc_clone3
-ppc_clone3:
- SAVE_NVGPRS(r1)
- lwz r0,_TRAP(r1)
- rlwinm r0,r0,0,0,30 /* clear LSB to indicate full */
- stw r0,_TRAP(r1) /* register set saved */
- b sys_clone3
-
- .globl ppc_swapcontext
-ppc_swapcontext:
- SAVE_NVGPRS(r1)
- lwz r0,_TRAP(r1)
- rlwinm r0,r0,0,0,30 /* clear LSB to indicate full */
- stw r0,_TRAP(r1) /* register set saved */
- b sys_swapcontext
-
/*
* Top-level page fault handling.
* This is in assembler because if do_page_fault tells us that
diff --git a/arch/powerpc/kernel/head_32.h b/arch/powerpc/kernel/head_32.h
index 24dc326e0d56..7b12736ec546 100644
--- a/arch/powerpc/kernel/head_32.h
+++ b/arch/powerpc/kernel/head_32.h
@@ -148,7 +148,7 @@
stw r2,GPR2(r11)
addi r10,r10,STACK_FRAME_REGS_MARKER@l
stw r9,_MSR(r11)
- li r2, \trapno + 1
+ li r2, \trapno
stw r10,8(r11)
stw r2,_TRAP(r11)
SAVE_GPR(0, r11)
diff --git a/arch/powerpc/kernel/head_booke.h b/arch/powerpc/kernel/head_booke.h
index b3c502c503a0..626e716576ce 100644
--- a/arch/powerpc/kernel/head_booke.h
+++ b/arch/powerpc/kernel/head_booke.h
@@ -124,7 +124,7 @@ ALT_FTR_SECTION_END_IFSET(CPU_FTR_EMB_HV)
stw r2,GPR2(r11)
addi r12, r12, STACK_FRAME_REGS_MARKER@l
stw r9,_MSR(r11)
- li r2, \trapno + 1
+ li r2, \trapno
stw r12, 8(r11)
stw r2,_TRAP(r11)
SAVE_GPR(0, r11)
diff --git a/arch/powerpc/kernel/syscalls/syscall.tbl b/arch/powerpc/kernel/syscalls/syscall.tbl
index f744eb5cba88..96b2157f0371 100644
--- a/arch/powerpc/kernel/syscalls/syscall.tbl
+++ b/arch/powerpc/kernel/syscalls/syscall.tbl
@@ -9,9 +9,7 @@
#
0 nospu restart_syscall sys_restart_syscall
1 nospu exit sys_exit
-2 32 fork ppc_fork sys_fork
-2 64 fork sys_fork
-2 spu fork sys_ni_syscall
+2 nospu fork sys_fork
3 common read sys_read
4 common write sys_write
5 common open sys_open compat_sys_open
@@ -160,9 +158,7 @@
119 32 sigreturn sys_sigreturn compat_sys_sigreturn
119 64 sigreturn sys_ni_syscall
119 spu sigreturn sys_ni_syscall
-120 32 clone ppc_clone sys_clone
-120 64 clone sys_clone
-120 spu clone sys_ni_syscall
+120 nospu clone sys_clone
121 common setdomainname sys_setdomainname
122 common uname sys_newuname
123 common modify_ldt sys_ni_syscall
@@ -244,9 +240,7 @@
186 spu sendfile sys_sendfile64
187 common getpmsg sys_ni_syscall
188 common putpmsg sys_ni_syscall
-189 32 vfork ppc_vfork sys_vfork
-189 64 vfork sys_vfork
-189 spu vfork sys_ni_syscall
+189 nospu vfork sys_vfork
190 common ugetrlimit sys_getrlimit compat_sys_getrlimit
191 common readahead sys_readahead compat_sys_readahead
192 32 mmap2 sys_mmap2 compat_sys_mmap2
@@ -322,9 +316,7 @@
248 32 clock_nanosleep sys_clock_nanosleep_time32
248 64 clock_nanosleep sys_clock_nanosleep
248 spu clock_nanosleep sys_clock_nanosleep
-249 32 swapcontext ppc_swapcontext compat_sys_swapcontext
-249 64 swapcontext sys_swapcontext
-249 spu swapcontext sys_ni_syscall
+249 nospu swapcontext sys_swapcontext compat_sys_swapcontext
250 common tgkill sys_tgkill
251 32 utimes sys_utimes_time32
251 64 utimes sys_utimes
@@ -522,9 +514,7 @@
432 common fsmount sys_fsmount
433 common fspick sys_fspick
434 common pidfd_open sys_pidfd_open
-435 32 clone3 ppc_clone3 sys_clone3
-435 64 clone3 sys_clone3
-435 spu clone3 sys_ni_syscall
+435 nospu clone3 sys_clone3
436 common close_range sys_close_range
437 common openat2 sys_openat2
438 common pidfd_getfd sys_pidfd_getfd
--
2.25.0
Instead of hard comparing task flags with _TIF_32BIT, use
is_compat_task(). The advantage is that it returns 0 on PPC32
allthough _TIF_32BIT is always set.
Signed-off-by: Christophe Leroy <[email protected]>
---
arch/powerpc/kernel/interrupt.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
index 2dac4d2bb1cf..46fd195ca659 100644
--- a/arch/powerpc/kernel/interrupt.c
+++ b/arch/powerpc/kernel/interrupt.c
@@ -2,6 +2,8 @@
#include <linux/context_tracking.h>
#include <linux/err.h>
+#include <linux/compat.h>
+
#include <asm/asm-prototypes.h>
#include <asm/kup.h>
#include <asm/cputime.h>
@@ -118,7 +120,7 @@ notrace long system_call_exception(long r3, long r4, long r5,
/* May be faster to do array_index_nospec? */
barrier_nospec();
- if (unlikely(is_32bit_task())) {
+ if (unlikely(is_compat_task())) {
f = (void *)compat_sys_call_table[r0];
r3 &= 0x00000000ffffffffULL;
--
2.25.0
In system_call_exception(), MSR_RI also needs to be checked on 8xx.
Only booke and 40x doesn't have MSR_RI.
Signed-off-by: Christophe Leroy <[email protected]>
---
v5: Also in interrupt exit prepare
---
arch/powerpc/kernel/interrupt.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
index 1a2dec49f811..107ec39f05cb 100644
--- a/arch/powerpc/kernel/interrupt.c
+++ b/arch/powerpc/kernel/interrupt.c
@@ -39,7 +39,7 @@ notrace long system_call_exception(long r3, long r4, long r5,
trace_hardirqs_off(); /* finish reconciling */
- if (IS_ENABLED(CONFIG_PPC_BOOK3S))
+ if (!IS_ENABLED(CONFIG_BOOKE) && !IS_ENABLED(CONFIG_40x))
BUG_ON(!(regs->msr & MSR_RI));
BUG_ON(!(regs->msr & MSR_PR));
BUG_ON(!FULL_REGS(regs));
@@ -338,7 +338,7 @@ notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs, unsigned
unsigned long flags;
unsigned long ret = 0;
- if (IS_ENABLED(CONFIG_PPC_BOOK3S))
+ if (!IS_ENABLED(CONFIG_BOOKE) && !IS_ENABLED(CONFIG_40x))
BUG_ON(!(regs->msr & MSR_RI));
BUG_ON(!(regs->msr & MSR_PR));
BUG_ON(!FULL_REGS(regs));
@@ -436,7 +436,8 @@ notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs, unsign
unsigned long amr;
#endif
- if (IS_ENABLED(CONFIG_PPC_BOOK3S) && unlikely(!(regs->msr & MSR_RI)))
+ if (!IS_ENABLED(CONFIG_BOOKE) && !IS_ENABLED(CONFIG_40x) &&
+ unlikely(!(regs->msr & MSR_RI)))
unrecoverable_exception(regs);
BUG_ON(regs->msr & MSR_PR);
BUG_ON(!FULL_REGS(regs));
--
2.25.0
Now that we are using rfi instead of mtmsr to reactivate MMU, it is
possible to reorder instructions and avoid the need to use CTR for
stashing SRR0.
null_syscall on 8xx is reduced by 3 cycles (283 => 280 cycles).
Signed-off-by: Christophe Leroy <[email protected]>
---
arch/powerpc/kernel/head_32.h | 22 ++++++++++------------
1 file changed, 10 insertions(+), 12 deletions(-)
diff --git a/arch/powerpc/kernel/head_32.h b/arch/powerpc/kernel/head_32.h
index 4029c51dce5d..24dc326e0d56 100644
--- a/arch/powerpc/kernel/head_32.h
+++ b/arch/powerpc/kernel/head_32.h
@@ -116,30 +116,28 @@
.endm
.macro SYSCALL_ENTRY trapno
- mfspr r12,SPRN_SPRG_THREAD
mfspr r9, SPRN_SRR1
- mfspr r11, SPRN_SRR0
- mtctr r11
+ mfspr r10, SPRN_SRR0
andi. r11, r9, MSR_PR
+ beq- 99f
+ LOAD_REG_IMMEDIATE(r11, MSR_KERNEL) /* can take exceptions */
+ lis r12, 1f@h
+ ori r12, r12, 1f@l
+ mtspr SPRN_SRR1, r11
+ mtspr SPRN_SRR0, r12
+ mfspr r12,SPRN_SPRG_THREAD
mr r11, r1
lwz r1,TASK_STACK-THREAD(r12)
- beq- 99f
+ tovirt(r12, r12)
addi r1, r1, THREAD_SIZE - INT_FRAME_SIZE
- LOAD_REG_IMMEDIATE(r10, MSR_KERNEL) /* can take exceptions */
- mtspr SPRN_SRR1, r10
- lis r10, 1f@h
- ori r10, r10, 1f@l
- mtspr SPRN_SRR0, r10
rfi
1:
- tovirt(r12, r12)
stw r11,GPR1(r1)
stw r11,0(r1)
mr r11, r1
+ stw r10,_NIP(r11)
mflr r10
stw r10, _LINK(r11)
- mfctr r10
- stw r10,_NIP(r11)
mfcr r10
rlwinm r10,r10,0,4,2 /* Clear SO bit in CR */
stw r10,_CCR(r11) /* save registers */
--
2.25.0
To allow building interrupt.c on PPC32, ifdef out specific PPC64
code or use helpers which are available on both PP32 and PPC64
Modify Makefile to always build interrupt.o
Signed-off-by: Christophe Leroy <[email protected]>
---
v5:
- Also for interrupt exit preparation
- Opted out kuap related code, ppc32 keeps it in ASM for the time being
---
arch/powerpc/kernel/Makefile | 4 ++--
arch/powerpc/kernel/interrupt.c | 31 ++++++++++++++++++++++++-------
2 files changed, 26 insertions(+), 9 deletions(-)
diff --git a/arch/powerpc/kernel/Makefile b/arch/powerpc/kernel/Makefile
index 26ff8c6e06b7..163755b1cef4 100644
--- a/arch/powerpc/kernel/Makefile
+++ b/arch/powerpc/kernel/Makefile
@@ -57,10 +57,10 @@ obj-y := cputable.o syscalls.o \
prom.o traps.o setup-common.o \
udbg.o misc.o io.o misc_$(BITS).o \
of_platform.o prom_parse.o firmware.o \
- hw_breakpoint_constraints.o
+ hw_breakpoint_constraints.o interrupt.o
obj-y += ptrace/
obj-$(CONFIG_PPC64) += setup_64.o \
- paca.o nvram_64.o note.o interrupt.o
+ paca.o nvram_64.o note.o
obj-$(CONFIG_COMPAT) += sys_ppc32.o signal_32.o
obj-$(CONFIG_VDSO32) += vdso32_wrapper.o
obj-$(CONFIG_PPC_WATCHDOG) += watchdog.o
diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
index d6be4f9a67e5..2dac4d2bb1cf 100644
--- a/arch/powerpc/kernel/interrupt.c
+++ b/arch/powerpc/kernel/interrupt.c
@@ -39,7 +39,7 @@ notrace long system_call_exception(long r3, long r4, long r5,
BUG_ON(!(regs->msr & MSR_RI));
BUG_ON(!(regs->msr & MSR_PR));
BUG_ON(!FULL_REGS(regs));
- BUG_ON(regs->softe != IRQS_ENABLED);
+ BUG_ON(arch_irq_disabled_regs(regs));
#ifdef CONFIG_PPC_PKEY
if (mmu_has_feature(MMU_FTR_PKEY)) {
@@ -65,7 +65,9 @@ notrace long system_call_exception(long r3, long r4, long r5,
isync();
} else
#endif
+#ifdef CONFIG_PPC64
kuap_check_amr();
+#endif
account_cpu_user_entry();
@@ -77,7 +79,7 @@ notrace long system_call_exception(long r3, long r4, long r5,
* frame, or if the unwinder was taught the first stack frame always
* returns to user with IRQS_ENABLED, this store could be avoided!
*/
- regs->softe = IRQS_ENABLED;
+ irq_soft_mask_regs_set_state(regs, IRQS_ENABLED);
local_irq_enable();
@@ -151,6 +153,7 @@ static notrace inline bool __prep_irq_for_enabled_exit(bool clear_ri)
__hard_EE_RI_disable();
else
__hard_irq_disable();
+#ifdef CONFIG_PPC64
if (unlikely(lazy_irq_pending_nocheck())) {
/* Took an interrupt, may have more exit work to do. */
if (clear_ri)
@@ -162,7 +165,7 @@ static notrace inline bool __prep_irq_for_enabled_exit(bool clear_ri)
}
local_paca->irq_happened = 0;
irq_soft_mask_set(IRQS_ENABLED);
-
+#endif
return true;
}
@@ -216,7 +219,9 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,
CT_WARN_ON(ct_state() == CONTEXT_USER);
+#ifdef CONFIG_PPC64
kuap_check_amr();
+#endif
regs->result = r3;
@@ -309,7 +314,7 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,
account_cpu_user_exit();
-#ifdef CONFIG_PPC_BOOK3S /* BOOK3E not yet using this */
+#ifdef CONFIG_PPC_BOOK3S_64 /* BOOK3E and ppc32 not using this */
/*
* We do this at the end so that we do context switch with KERNEL AMR
*/
@@ -318,7 +323,7 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,
return ret;
}
-#ifdef CONFIG_PPC_BOOK3S /* BOOK3E not yet using this */
+#ifndef CONFIG_PPC_BOOK3E_64 /* BOOK3E not yet using this */
notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs, unsigned long msr)
{
#ifdef CONFIG_PPC_BOOK3E
@@ -333,14 +338,16 @@ notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs, unsigned
BUG_ON(!(regs->msr & MSR_RI));
BUG_ON(!(regs->msr & MSR_PR));
BUG_ON(!FULL_REGS(regs));
- BUG_ON(regs->softe != IRQS_ENABLED);
+ BUG_ON(arch_irq_disabled_regs(regs));
CT_WARN_ON(ct_state() == CONTEXT_USER);
/*
* We don't need to restore AMR on the way back to userspace for KUAP.
* AMR can only have been unlocked if we interrupted the kernel.
*/
+#ifdef CONFIG_PPC64
kuap_check_amr();
+#endif
local_irq_save(flags);
@@ -407,7 +414,9 @@ notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs, unsigned
/*
* We do this at the end so that we do context switch with KERNEL AMR
*/
+#ifdef CONFIG_PPC64
kuap_user_restore(regs);
+#endif
return ret;
}
@@ -419,7 +428,9 @@ notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs, unsign
unsigned long *ti_flagsp = ¤t_thread_info()->flags;
unsigned long flags;
unsigned long ret = 0;
+#ifdef CONFIG_PPC64
unsigned long amr;
+#endif
if (IS_ENABLED(CONFIG_PPC_BOOK3S) && unlikely(!(regs->msr & MSR_RI)))
unrecoverable_exception(regs);
@@ -432,7 +443,9 @@ notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs, unsign
if (TRAP(regs) != 0x700)
CT_WARN_ON(ct_state() == CONTEXT_USER);
+#ifdef CONFIG_PPC64
amr = kuap_get_and_check_amr();
+#endif
if (unlikely(*ti_flagsp & _TIF_EMULATE_STACK_STORE)) {
clear_bits(_TIF_EMULATE_STACK_STORE, ti_flagsp);
@@ -441,7 +454,7 @@ notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs, unsign
local_irq_save(flags);
- if (regs->softe == IRQS_ENABLED) {
+ if (!arch_irq_disabled_regs(regs)) {
/* Returning to a kernel context with local irqs enabled. */
WARN_ON_ONCE(!(regs->msr & MSR_EE));
again:
@@ -458,8 +471,10 @@ notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs, unsign
} else {
/* Returning to a kernel context with local irqs disabled. */
__hard_EE_RI_disable();
+#ifdef CONFIG_PPC64
if (regs->msr & MSR_EE)
local_paca->irq_happened &= ~PACA_IRQ_HARD_DIS;
+#endif
}
@@ -472,7 +487,9 @@ notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs, unsign
* which would cause Read-After-Write stalls. Hence, we take the AMR
* value from the check above.
*/
+#ifdef CONFIG_PPC64
kuap_kernel_restore(regs, amr);
+#endif
return ret;
}
--
2.25.0
To allow building syscall_64.c smoothly on PPC32, add stub version
of irq_soft_mask_return().
Signed-off-by: Christophe Leroy <[email protected]>
---
arch/powerpc/include/asm/hw_irq.h | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/arch/powerpc/include/asm/hw_irq.h b/arch/powerpc/include/asm/hw_irq.h
index 4739f61e632c..56a98936a6a9 100644
--- a/arch/powerpc/include/asm/hw_irq.h
+++ b/arch/powerpc/include/asm/hw_irq.h
@@ -330,6 +330,11 @@ static inline void irq_soft_mask_regs_set_state(struct pt_regs *regs, unsigned l
}
#else /* CONFIG_PPC64 */
+static inline notrace unsigned long irq_soft_mask_return(void)
+{
+ return 0;
+}
+
static inline unsigned long arch_local_save_flags(void)
{
return mfmsr();
--
2.25.0
Save r3 in regs->orig_r3 in system_call_exception()
Signed-off-by: Christophe Leroy <[email protected]>
---
v5: Removed the assembly one on SCV type system call
---
arch/powerpc/kernel/entry_64.S | 2 --
arch/powerpc/kernel/interrupt.c | 2 ++
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/kernel/entry_64.S b/arch/powerpc/kernel/entry_64.S
index 33ddfeef4fe9..a91c2def165d 100644
--- a/arch/powerpc/kernel/entry_64.S
+++ b/arch/powerpc/kernel/entry_64.S
@@ -108,7 +108,6 @@ END_FTR_SECTION_IFSET(CPU_FTR_TM)
li r11,\trapnr
std r11,_TRAP(r1)
std r12,_CCR(r1)
- std r3,ORIG_GPR3(r1)
addi r10,r1,STACK_FRAME_OVERHEAD
ld r11,exception_marker@toc(r2)
std r11,-16(r10) /* "regshere" marker */
@@ -278,7 +277,6 @@ END_BTB_FLUSH_SECTION
std r10,_LINK(r1)
std r11,_TRAP(r1)
std r12,_CCR(r1)
- std r3,ORIG_GPR3(r1)
addi r10,r1,STACK_FRAME_OVERHEAD
ld r11,exception_marker@toc(r2)
std r11,-16(r10) /* "regshere" marker */
diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
index 46fd195ca659..1a2dec49f811 100644
--- a/arch/powerpc/kernel/interrupt.c
+++ b/arch/powerpc/kernel/interrupt.c
@@ -29,6 +29,8 @@ notrace long system_call_exception(long r3, long r4, long r5,
{
syscall_fn f;
+ regs->orig_gpr3 = r3;
+
if (IS_ENABLED(CONFIG_PPC_IRQ_SOFT_MASK_DEBUG))
BUG_ON(irq_soft_mask_return() != IRQS_ALL_DISABLED);
--
2.25.0
In preparation of porting PPC32 to C syscall entry/exit,
rewrite the following helpers as static inline functions and
add support for PPC32 in them:
__hard_irq_enable()
__hard_irq_disable()
__hard_EE_RI_disable()
__hard_RI_enable()
Then use them in PPC32 version of arch_local_irq_disable()
and arch_local_irq_enable() to avoid code duplication.
Signed-off-by: Christophe Leroy <[email protected]>
---
arch/powerpc/include/asm/hw_irq.h | 75 +++++++++++++++++++++----------
arch/powerpc/include/asm/reg.h | 1 +
2 files changed, 52 insertions(+), 24 deletions(-)
diff --git a/arch/powerpc/include/asm/hw_irq.h b/arch/powerpc/include/asm/hw_irq.h
index ed0c3b049dfd..4739f61e632c 100644
--- a/arch/powerpc/include/asm/hw_irq.h
+++ b/arch/powerpc/include/asm/hw_irq.h
@@ -50,6 +50,55 @@
#ifndef __ASSEMBLY__
+static inline void __hard_irq_enable(void)
+{
+ if (IS_ENABLED(CONFIG_BOOKE) || IS_ENABLED(CONFIG_40x))
+ wrtee(MSR_EE);
+ else if (IS_ENABLED(CONFIG_PPC_8xx))
+ wrtspr(SPRN_EIE);
+ else if (IS_ENABLED(CONFIG_PPC_BOOK3S_64))
+ __mtmsrd(MSR_EE | MSR_RI, 1);
+ else
+ mtmsr(mfmsr() | MSR_EE);
+}
+
+static inline void __hard_irq_disable(void)
+{
+ if (IS_ENABLED(CONFIG_BOOKE) || IS_ENABLED(CONFIG_40x))
+ wrtee(0);
+ else if (IS_ENABLED(CONFIG_PPC_8xx))
+ wrtspr(SPRN_EID);
+ else if (IS_ENABLED(CONFIG_PPC_BOOK3S_64))
+ __mtmsrd(MSR_RI, 1);
+ else
+ mtmsr(mfmsr() & ~MSR_EE);
+}
+
+static inline void __hard_EE_RI_disable(void)
+{
+ if (IS_ENABLED(CONFIG_BOOKE) || IS_ENABLED(CONFIG_40x))
+ wrtee(0);
+ else if (IS_ENABLED(CONFIG_PPC_8xx))
+ wrtspr(SPRN_NRI);
+ else if (IS_ENABLED(CONFIG_PPC_BOOK3S_64))
+ __mtmsrd(0, 1);
+ else
+ mtmsr(mfmsr() & ~(MSR_EE | MSR_RI));
+}
+
+static inline void __hard_RI_enable(void)
+{
+ if (IS_ENABLED(CONFIG_BOOKE) || IS_ENABLED(CONFIG_40x))
+ return;
+
+ if (IS_ENABLED(CONFIG_PPC_8xx))
+ wrtspr(SPRN_EID);
+ else if (IS_ENABLED(CONFIG_PPC_BOOK3S_64))
+ __mtmsrd(MSR_RI, 1);
+ else
+ mtmsr(mfmsr() | MSR_RI);
+}
+
#ifdef CONFIG_PPC64
#include <asm/paca.h>
@@ -212,18 +261,6 @@ static inline bool arch_irqs_disabled(void)
#endif /* CONFIG_PPC_BOOK3S */
-#ifdef CONFIG_PPC_BOOK3E
-#define __hard_irq_enable() wrtee(MSR_EE)
-#define __hard_irq_disable() wrtee(0)
-#define __hard_EE_RI_disable() wrtee(0)
-#define __hard_RI_enable() do { } while (0)
-#else
-#define __hard_irq_enable() __mtmsrd(MSR_EE|MSR_RI, 1)
-#define __hard_irq_disable() __mtmsrd(MSR_RI, 1)
-#define __hard_EE_RI_disable() __mtmsrd(0, 1)
-#define __hard_RI_enable() __mtmsrd(MSR_RI, 1)
-#endif
-
#define hard_irq_disable() do { \
unsigned long flags; \
__hard_irq_disable(); \
@@ -322,22 +359,12 @@ static inline unsigned long arch_local_irq_save(void)
static inline void arch_local_irq_disable(void)
{
- if (IS_ENABLED(CONFIG_BOOKE))
- wrtee(0);
- else if (IS_ENABLED(CONFIG_PPC_8xx))
- wrtspr(SPRN_EID);
- else
- mtmsr(mfmsr() & ~MSR_EE);
+ __hard_irq_disable();
}
static inline void arch_local_irq_enable(void)
{
- if (IS_ENABLED(CONFIG_BOOKE))
- wrtee(MSR_EE);
- else if (IS_ENABLED(CONFIG_PPC_8xx))
- wrtspr(SPRN_EIE);
- else
- mtmsr(mfmsr() | MSR_EE);
+ __hard_irq_enable();
}
static inline bool arch_irqs_disabled_flags(unsigned long flags)
diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/reg.h
index c5a3e856191c..bc4305ba00d0 100644
--- a/arch/powerpc/include/asm/reg.h
+++ b/arch/powerpc/include/asm/reg.h
@@ -1375,6 +1375,7 @@
#define mtmsr(v) asm volatile("mtmsr %0" : \
: "r" ((unsigned long)(v)) \
: "memory")
+#define __mtmsrd(v, l) BUILD_BUG()
#define __MTMSR "mtmsr"
#endif
--
2.25.0
On 40x and 8xx, kernel text is pinned.
On book3s/32, kernel text is mapped by BATs.
Enable instruction translation at the same time as data translation, it
makes things simpler.
MSR_RI can also be set at the same time because srr0/srr1 are already
saved and r1 is set properly.
On booke, translation is always on, so at the end all PPC32
have translation on early.
This reduces null_syscall benchmark by 13 cycles on 8xx
(296 ==> 283 cycles).
Signed-off-by: Christophe Leroy <[email protected]>
---
arch/powerpc/kernel/head_32.h | 26 +++++++++-----------------
arch/powerpc/kernel/head_booke.h | 7 ++-----
2 files changed, 11 insertions(+), 22 deletions(-)
diff --git a/arch/powerpc/kernel/head_32.h b/arch/powerpc/kernel/head_32.h
index fdc07beab844..4029c51dce5d 100644
--- a/arch/powerpc/kernel/head_32.h
+++ b/arch/powerpc/kernel/head_32.h
@@ -125,9 +125,13 @@
lwz r1,TASK_STACK-THREAD(r12)
beq- 99f
addi r1, r1, THREAD_SIZE - INT_FRAME_SIZE
- LOAD_REG_IMMEDIATE(r10, MSR_KERNEL & ~(MSR_IR | MSR_RI)) /* can take DTLB miss */
- mtmsr r10
- isync
+ LOAD_REG_IMMEDIATE(r10, MSR_KERNEL) /* can take exceptions */
+ mtspr SPRN_SRR1, r10
+ lis r10, 1f@h
+ ori r10, r10, 1f@l
+ mtspr SPRN_SRR0, r10
+ rfi
+1:
tovirt(r12, r12)
stw r11,GPR1(r1)
stw r11,0(r1)
@@ -141,9 +145,6 @@
stw r10,_CCR(r11) /* save registers */
#ifdef CONFIG_40x
rlwinm r9,r9,0,14,12 /* clear MSR_WE (necessary?) */
-#else
- LOAD_REG_IMMEDIATE(r10, MSR_KERNEL & ~MSR_IR) /* can take exceptions */
- mtmsr r10 /* (except for mach check in rtas) */
#endif
lis r10,STACK_FRAME_REGS_MARKER@ha /* exception frame marker */
stw r2,GPR2(r11)
@@ -180,8 +181,6 @@
#endif
3:
- lis r11, transfer_to_syscall@h
- ori r11, r11, transfer_to_syscall@l
#ifdef CONFIG_TRACE_IRQFLAGS
/*
* If MSR is changing we need to keep interrupts disabled at this point
@@ -193,15 +192,8 @@
#else
LOAD_REG_IMMEDIATE(r10, MSR_KERNEL | MSR_EE)
#endif
-#if defined(CONFIG_PPC_8xx) && defined(CONFIG_PERF_EVENTS)
- mtspr SPRN_NRI, r0
-#endif
- mtspr SPRN_SRR1,r10
- mtspr SPRN_SRR0,r11
- rfi /* jump to handler, enable MMU */
-#ifdef CONFIG_40x
- b . /* Prevent prefetch past rfi */
-#endif
+ mtmsr r10
+ b transfer_to_syscall /* jump to handler */
99: b ret_from_kernel_syscall
.endm
diff --git a/arch/powerpc/kernel/head_booke.h b/arch/powerpc/kernel/head_booke.h
index 706cd9368992..b3c502c503a0 100644
--- a/arch/powerpc/kernel/head_booke.h
+++ b/arch/powerpc/kernel/head_booke.h
@@ -157,8 +157,6 @@ ALT_FTR_SECTION_END_IFSET(CPU_FTR_EMB_HV)
stw r12,4(r11)
3:
- lis r11, transfer_to_syscall@h
- ori r11, r11, transfer_to_syscall@l
#ifdef CONFIG_TRACE_IRQFLAGS
/*
* If MSR is changing we need to keep interrupts disabled at this point
@@ -172,9 +170,8 @@ ALT_FTR_SECTION_END_IFSET(CPU_FTR_EMB_HV)
lis r10, (MSR_KERNEL | MSR_EE)@h
ori r10, r10, (MSR_KERNEL | MSR_EE)@l
#endif
- mtspr SPRN_SRR1,r10
- mtspr SPRN_SRR0,r11
- rfi /* jump to handler, enable MMU */
+ mtmsr r10
+ b transfer_to_syscall /* jump to handler */
99: b ret_from_kernel_syscall
.endm
--
2.25.0
That's port of PPC64 syscall entry/exit logic in C to PPC32.
Performancewise on 8xx:
Before : 304 cycles on null_syscall
After : 348 cycles on null_syscall
Signed-off-by: Christophe Leroy <[email protected]>
---
arch/powerpc/kernel/entry_32.S | 224 +++++--------------------------
arch/powerpc/kernel/head_32.h | 18 ---
arch/powerpc/kernel/head_booke.h | 17 ---
3 files changed, 30 insertions(+), 229 deletions(-)
diff --git a/arch/powerpc/kernel/entry_32.S b/arch/powerpc/kernel/entry_32.S
index 97dc28a68465..bbf7ecea6fe0 100644
--- a/arch/powerpc/kernel/entry_32.S
+++ b/arch/powerpc/kernel/entry_32.S
@@ -329,117 +329,23 @@ stack_ovf:
_ASM_NOKPROBE_SYMBOL(stack_ovf)
#endif
-#ifdef CONFIG_TRACE_IRQFLAGS
-trace_syscall_entry_irq_off:
- /*
- * Syscall shouldn't happen while interrupts are disabled,
- * so let's do a warning here.
- */
-0: trap
- EMIT_BUG_ENTRY 0b,__FILE__,__LINE__, BUGFLAG_WARNING
- bl trace_hardirqs_on
-
- /* Now enable for real */
- LOAD_REG_IMMEDIATE(r10, MSR_KERNEL | MSR_EE)
- mtmsr r10
-
- REST_GPR(0, r1)
- REST_4GPRS(3, r1)
- REST_2GPRS(7, r1)
- b DoSyscall
-#endif /* CONFIG_TRACE_IRQFLAGS */
-
.globl transfer_to_syscall
transfer_to_syscall:
SAVE_NVGPRS(r1)
#ifdef CONFIG_PPC_BOOK3S_32
kuep_lock r11, r12
#endif
-#ifdef CONFIG_TRACE_IRQFLAGS
- andi. r12,r9,MSR_EE
- beq- trace_syscall_entry_irq_off
-#endif /* CONFIG_TRACE_IRQFLAGS */
-/*
- * Handle a system call.
- */
- .stabs "arch/powerpc/kernel/",N_SO,0,0,0f
- .stabs "entry_32.S",N_SO,0,0,0f
-0:
-
-_GLOBAL(DoSyscall)
- stw r3,ORIG_GPR3(r1)
- li r12,0
- stw r12,RESULT(r1)
-#ifdef CONFIG_TRACE_IRQFLAGS
- /* Make sure interrupts are enabled */
- mfmsr r11
- andi. r12,r11,MSR_EE
- /* We came in with interrupts disabled, we WARN and mark them enabled
- * for lockdep now */
-0: tweqi r12, 0
- EMIT_BUG_ENTRY 0b,__FILE__,__LINE__, BUGFLAG_WARNING
-#endif /* CONFIG_TRACE_IRQFLAGS */
- lwz r11,TI_FLAGS(r2)
- andi. r11,r11,_TIF_SYSCALL_DOTRACE
- bne- syscall_dotrace
-syscall_dotrace_cont:
- cmplwi 0,r0,NR_syscalls
- lis r10,sys_call_table@h
- ori r10,r10,sys_call_table@l
- slwi r0,r0,2
- bge- 66f
-
- barrier_nospec_asm
- /*
- * Prevent the load of the handler below (based on the user-passed
- * system call number) being speculatively executed until the test
- * against NR_syscalls and branch to .66f above has
- * committed.
- */
+ /* Calling convention has r9 = orig r0, r10 = regs */
+ addi r10,r1,STACK_FRAME_OVERHEAD
+ mr r9,r0
+ stw r10,THREAD+PT_REGS(r2)
+ bl system_call_exception
- lwzx r10,r10,r0 /* Fetch system call handler [ptr] */
- mtlr r10
- addi r9,r1,STACK_FRAME_OVERHEAD
- PPC440EP_ERR42
- blrl /* Call handler */
- .globl ret_from_syscall
ret_from_syscall:
-#ifdef CONFIG_DEBUG_RSEQ
- /* Check whether the syscall is issued inside a restartable sequence */
- stw r3,GPR3(r1)
- addi r3,r1,STACK_FRAME_OVERHEAD
- bl rseq_syscall
- lwz r3,GPR3(r1)
-#endif
- mr r6,r3
- /* disable interrupts so current_thread_info()->flags can't change */
- LOAD_REG_IMMEDIATE(r10,MSR_KERNEL) /* doesn't include MSR_EE */
- /* Note: We don't bother telling lockdep about it */
- mtmsr r10
- lwz r9,TI_FLAGS(r2)
- li r8,-MAX_ERRNO
- andi. r0,r9,(_TIF_SYSCALL_DOTRACE|_TIF_SINGLESTEP|_TIF_USER_WORK_MASK|_TIF_PERSYSCALL_MASK)
- bne- syscall_exit_work
- cmplw 0,r3,r8
- blt+ syscall_exit_cont
- lwz r11,_CCR(r1) /* Load CR */
- neg r3,r3
- oris r11,r11,0x1000 /* Set SO bit in CR */
- stw r11,_CCR(r1)
-syscall_exit_cont:
- lwz r8,_MSR(r1)
-#ifdef CONFIG_TRACE_IRQFLAGS
- /* If we are going to return from the syscall with interrupts
- * off, we trace that here. It shouldn't normally happen.
- */
- andi. r10,r8,MSR_EE
- bne+ 1f
- stw r3,GPR3(r1)
- bl trace_hardirqs_off
- lwz r3,GPR3(r1)
-1:
-#endif /* CONFIG_TRACE_IRQFLAGS */
+ addi r4,r1,STACK_FRAME_OVERHEAD
+ li r5,0
+ bl syscall_exit_prepare
#if defined(CONFIG_4xx) || defined(CONFIG_BOOKE)
/* If the process has its own DBCR0 value, load it up. The internal
debug mode bit tells us that dbcr0 should be loaded. */
@@ -453,12 +359,6 @@ syscall_exit_cont:
cmplwi cr0,r5,0
bne- 2f
#endif /* CONFIG_PPC_47x */
-1:
-BEGIN_FTR_SECTION
- lwarx r7,0,r1
-END_FTR_SECTION_IFSET(CPU_FTR_NEED_PAIRED_STWCX)
- stwcx. r0,0,r1 /* to clear the reservation */
- ACCOUNT_CPU_USER_EXIT(r2, r5, r7)
#ifdef CONFIG_PPC_BOOK3S_32
kuep_unlock r5, r7
#endif
@@ -466,21 +366,36 @@ END_FTR_SECTION_IFSET(CPU_FTR_NEED_PAIRED_STWCX)
lwz r4,_LINK(r1)
lwz r5,_CCR(r1)
mtlr r4
- mtcr r5
lwz r7,_NIP(r1)
- lwz r2,GPR2(r1)
- lwz r1,GPR1(r1)
+ lwz r8,_MSR(r1)
+ cmpwi r3,0
+ lwz r3,GPR3(r1)
syscall_exit_finish:
-#if defined(CONFIG_PPC_8xx) && defined(CONFIG_PERF_EVENTS)
- mtspr SPRN_NRI, r0
-#endif
mtspr SPRN_SRR0,r7
mtspr SPRN_SRR1,r8
+
+ bne 3f
+ mtcr r5
+
+1: lwz r2,GPR2(r1)
+ lwz r1,GPR1(r1)
rfi
#ifdef CONFIG_40x
b . /* Prevent prefetch past rfi */
#endif
-_ASM_NOKPROBE_SYMBOL(syscall_exit_finish)
+
+3: mtcr r5
+ lwz r4,_CTR(r1)
+ lwz r5,_XER(r1)
+ REST_NVGPRS(r1)
+ mtctr r4
+ mtxer r5
+ lwz r0,GPR0(r1)
+ lwz r3,GPR3(r1)
+ REST_8GPRS(4,r1)
+ lwz r12,GPR12(r1)
+ b 1b
+
#ifdef CONFIG_44x
2: li r7,0
iccci r0,r0
@@ -488,9 +403,6 @@ _ASM_NOKPROBE_SYMBOL(syscall_exit_finish)
b 1b
#endif /* CONFIG_44x */
-66: li r3,-ENOSYS
- b ret_from_syscall
-
.globl ret_from_fork
ret_from_fork:
REST_NVGPRS(r1)
@@ -509,82 +421,6 @@ ret_from_kernel_thread:
li r3,0
b ret_from_syscall
-/* Traced system call support */
-syscall_dotrace:
- SAVE_NVGPRS(r1)
- li r0,0xc00
- stw r0,_TRAP(r1)
- addi r3,r1,STACK_FRAME_OVERHEAD
- bl do_syscall_trace_enter
- /*
- * Restore argument registers possibly just changed.
- * We use the return value of do_syscall_trace_enter
- * for call number to look up in the table (r0).
- */
- mr r0,r3
- lwz r3,GPR3(r1)
- lwz r4,GPR4(r1)
- lwz r5,GPR5(r1)
- lwz r6,GPR6(r1)
- lwz r7,GPR7(r1)
- lwz r8,GPR8(r1)
- REST_NVGPRS(r1)
-
- cmplwi r0,NR_syscalls
- /* Return code is already in r3 thanks to do_syscall_trace_enter() */
- bge- ret_from_syscall
- b syscall_dotrace_cont
-
-syscall_exit_work:
- andi. r0,r9,_TIF_RESTOREALL
- beq+ 0f
- REST_NVGPRS(r1)
- b 2f
-0: cmplw 0,r3,r8
- blt+ 1f
- andi. r0,r9,_TIF_NOERROR
- bne- 1f
- lwz r11,_CCR(r1) /* Load CR */
- neg r3,r3
- oris r11,r11,0x1000 /* Set SO bit in CR */
- stw r11,_CCR(r1)
-
-1: stw r6,RESULT(r1) /* Save result */
- stw r3,GPR3(r1) /* Update return value */
-2: andi. r0,r9,(_TIF_PERSYSCALL_MASK)
- beq 4f
-
- /* Clear per-syscall TIF flags if any are set. */
-
- li r11,_TIF_PERSYSCALL_MASK
- addi r12,r2,TI_FLAGS
-3: lwarx r8,0,r12
- andc r8,r8,r11
- stwcx. r8,0,r12
- bne- 3b
-
-4: /* Anything which requires enabling interrupts? */
- andi. r0,r9,(_TIF_SYSCALL_DOTRACE|_TIF_SINGLESTEP)
- beq ret_from_except
-
- /* Re-enable interrupts. There is no need to trace that with
- * lockdep as we are supposed to have IRQs on at this point
- */
- ori r10,r10,MSR_EE
- mtmsr r10
-
- /* Save NVGPRS if they're not saved already */
- lwz r4,_TRAP(r1)
- andi. r4,r4,1
- beq 5f
- SAVE_NVGPRS(r1)
- li r4,0xc00
- stw r4,_TRAP(r1)
-5:
- addi r3,r1,STACK_FRAME_OVERHEAD
- bl do_syscall_trace_leave
- b ret_from_except_full
-
/*
* System call was called from kernel. We get here with SRR1 in r9.
* Mark the exception as recoverable once we have retrieved SRR0,
diff --git a/arch/powerpc/kernel/head_32.h b/arch/powerpc/kernel/head_32.h
index 7b12736ec546..fea7fe00a690 100644
--- a/arch/powerpc/kernel/head_32.h
+++ b/arch/powerpc/kernel/head_32.h
@@ -154,17 +154,12 @@
SAVE_GPR(0, r11)
SAVE_4GPRS(3, r11)
SAVE_2GPRS(7, r11)
- addi r11,r1,STACK_FRAME_OVERHEAD
addi r2,r12,-THREAD
- stw r11,PT_REGS(r12)
#if defined(CONFIG_40x)
/* Check to see if the dbcr0 register is set up to debug. Use the
internal debug mode bit to do this. */
lwz r12,THREAD_DBCR0(r12)
andis. r12,r12,DBCR0_IDM@h
-#endif
- ACCOUNT_CPU_USER_ENTRY(r2, r11, r12)
-#if defined(CONFIG_40x)
beq+ 3f
/* From user and task is ptraced - load up global dbcr0 */
li r12,-1 /* clear all pending debug events */
@@ -176,21 +171,8 @@
lwz r12,4(r11)
addi r12,r12,-1
stw r12,4(r11)
-#endif
-
3:
-#ifdef CONFIG_TRACE_IRQFLAGS
- /*
- * If MSR is changing we need to keep interrupts disabled at this point
- * otherwise we might risk taking an interrupt before we tell lockdep
- * they are enabled.
- */
- LOAD_REG_IMMEDIATE(r10, MSR_KERNEL)
- rlwimi r10, r9, 0, MSR_EE
-#else
- LOAD_REG_IMMEDIATE(r10, MSR_KERNEL | MSR_EE)
#endif
- mtmsr r10
b transfer_to_syscall /* jump to handler */
99: b ret_from_kernel_syscall
.endm
diff --git a/arch/powerpc/kernel/head_booke.h b/arch/powerpc/kernel/head_booke.h
index 626e716576ce..db931f1167aa 100644
--- a/arch/powerpc/kernel/head_booke.h
+++ b/arch/powerpc/kernel/head_booke.h
@@ -131,14 +131,11 @@ ALT_FTR_SECTION_END_IFSET(CPU_FTR_EMB_HV)
SAVE_4GPRS(3, r11)
SAVE_2GPRS(7, r11)
- addi r11,r1,STACK_FRAME_OVERHEAD
addi r2,r10,-THREAD
- stw r11,PT_REGS(r10)
/* Check to see if the dbcr0 register is set up to debug. Use the
internal debug mode bit to do this. */
lwz r12,THREAD_DBCR0(r10)
andis. r12,r12,DBCR0_IDM@h
- ACCOUNT_CPU_USER_ENTRY(r2, r11, r12)
beq+ 3f
/* From user and task is ptraced - load up global dbcr0 */
li r12,-1 /* clear all pending debug events */
@@ -157,20 +154,6 @@ ALT_FTR_SECTION_END_IFSET(CPU_FTR_EMB_HV)
stw r12,4(r11)
3:
-#ifdef CONFIG_TRACE_IRQFLAGS
- /*
- * If MSR is changing we need to keep interrupts disabled at this point
- * otherwise we might risk taking an interrupt before we tell lockdep
- * they are enabled.
- */
- lis r10, MSR_KERNEL@h
- ori r10, r10, MSR_KERNEL@l
- rlwimi r10, r9, 0, MSR_EE
-#else
- lis r10, (MSR_KERNEL | MSR_EE)@h
- ori r10, r10, (MSR_KERNEL | MSR_EE)@l
-#endif
- mtmsr r10
b transfer_to_syscall /* jump to handler */
99: b ret_from_kernel_syscall
.endm
--
2.25.0
global_dbcr0 has two parts, 4 bytes to save/restore the
value of SPRN_DBCR0, and 4 bytes that are incremented/decremented
everytime something is saving/loading the above value.
This counter is only incremented/decremented, its value is never
used and never read.
Remove the counter and devide the size of global_dbcr0 by 2.
Signed-off-by: Christophe Leroy <[email protected]>
---
v5: New
---
arch/powerpc/kernel/entry_32.S | 12 +++---------
arch/powerpc/kernel/head_32.h | 3 ---
arch/powerpc/kernel/head_booke.h | 5 +----
3 files changed, 4 insertions(+), 16 deletions(-)
diff --git a/arch/powerpc/kernel/entry_32.S b/arch/powerpc/kernel/entry_32.S
index 7c824e8928d0..a574201b0eb6 100644
--- a/arch/powerpc/kernel/entry_32.S
+++ b/arch/powerpc/kernel/entry_32.S
@@ -175,14 +175,11 @@ transfer_to_handler:
addi r11,r11,global_dbcr0@l
#ifdef CONFIG_SMP
lwz r9,TASK_CPU(r2)
- slwi r9,r9,3
+ slwi r9,r9,2
add r11,r11,r9
#endif
lwz r12,0(r11)
mtspr SPRN_DBCR0,r12
- lwz r12,4(r11)
- addi r12,r12,-1
- stw r12,4(r11)
#endif
b 3f
@@ -980,14 +977,11 @@ load_dbcr0:
addi r11,r11,global_dbcr0@l
#ifdef CONFIG_SMP
lwz r9,TASK_CPU(r2)
- slwi r9,r9,3
+ slwi r9,r9,2
add r11,r11,r9
#endif
stw r10,0(r11)
mtspr SPRN_DBCR0,r0
- lwz r10,4(r11)
- addi r10,r10,1
- stw r10,4(r11)
li r11,-1
mtspr SPRN_DBSR,r11 /* clear all pending debug events */
blr
@@ -996,7 +990,7 @@ load_dbcr0:
.align 4
.global global_dbcr0
global_dbcr0:
- .space 8*NR_CPUS
+ .space 4*NR_CPUS
.previous
#endif /* !(CONFIG_4xx || CONFIG_BOOKE) */
diff --git a/arch/powerpc/kernel/head_32.h b/arch/powerpc/kernel/head_32.h
index 282d8fd443a9..5001c6ecc3ec 100644
--- a/arch/powerpc/kernel/head_32.h
+++ b/arch/powerpc/kernel/head_32.h
@@ -166,9 +166,6 @@
addi r11,r11,global_dbcr0@l
lwz r12,0(r11)
mtspr SPRN_DBCR0,r12
- lwz r12,4(r11)
- addi r12,r12,-1
- stw r12,4(r11)
3:
#endif
b transfer_to_syscall /* jump to handler */
diff --git a/arch/powerpc/kernel/head_booke.h b/arch/powerpc/kernel/head_booke.h
index bfbd240cc8a2..5f565232b99d 100644
--- a/arch/powerpc/kernel/head_booke.h
+++ b/arch/powerpc/kernel/head_booke.h
@@ -142,14 +142,11 @@ ALT_FTR_SECTION_END_IFSET(CPU_FTR_EMB_HV)
addi r11,r11,global_dbcr0@l
#ifdef CONFIG_SMP
lwz r10, TASK_CPU(r2)
- slwi r10, r10, 3
+ slwi r10, r10, 2
add r11, r11, r10
#endif
lwz r12,0(r11)
mtspr SPRN_DBCR0,r12
- lwz r12,4(r11)
- addi r12,r12,-1
- stw r12,4(r11)
3:
b transfer_to_syscall /* jump to handler */
--
2.25.0
The handling of SPRN_DBCR0 and other registers can easily
be done in C instead of ASM.
Signed-off-by: Christophe Leroy <[email protected]>
---
v5: New
---
arch/powerpc/include/asm/reg_booke.h | 3 +++
arch/powerpc/kernel/entry_32.S | 7 -------
arch/powerpc/kernel/head_32.h | 15 --------------
arch/powerpc/kernel/head_booke.h | 19 ------------------
arch/powerpc/kernel/interrupt.c | 29 +++++++++++++++++++++++++++-
5 files changed, 31 insertions(+), 42 deletions(-)
diff --git a/arch/powerpc/include/asm/reg_booke.h b/arch/powerpc/include/asm/reg_booke.h
index 262782f08fd4..17b8dcd9a40d 100644
--- a/arch/powerpc/include/asm/reg_booke.h
+++ b/arch/powerpc/include/asm/reg_booke.h
@@ -691,6 +691,9 @@
#define mttmr(rn, v) asm volatile(MTTMR(rn, %0) : \
: "r" ((unsigned long)(v)) \
: "memory")
+
+extern unsigned long global_dbcr0[];
+
#endif /* !__ASSEMBLY__ */
#endif /* __ASM_POWERPC_REG_BOOKE_H__ */
diff --git a/arch/powerpc/kernel/entry_32.S b/arch/powerpc/kernel/entry_32.S
index a574201b0eb6..8dea4d3b1d06 100644
--- a/arch/powerpc/kernel/entry_32.S
+++ b/arch/powerpc/kernel/entry_32.S
@@ -342,13 +342,6 @@ transfer_to_syscall:
ret_from_syscall:
addi r4,r1,STACK_FRAME_OVERHEAD
bl syscall_exit_prepare
-#if defined(CONFIG_4xx) || defined(CONFIG_BOOKE)
- /* If the process has its own DBCR0 value, load it up. The internal
- debug mode bit tells us that dbcr0 should be loaded. */
- lwz r0,THREAD+THREAD_DBCR0(r2)
- andis. r10,r0,DBCR0_IDM@h
- bnel- load_dbcr0
-#endif
#ifdef CONFIG_PPC_47x
lis r4,icache_44x_need_flush@ha
lwz r5,icache_44x_need_flush@l(r4)
diff --git a/arch/powerpc/kernel/head_32.h b/arch/powerpc/kernel/head_32.h
index 5001c6ecc3ec..961b1ce3b6bf 100644
--- a/arch/powerpc/kernel/head_32.h
+++ b/arch/powerpc/kernel/head_32.h
@@ -153,21 +153,6 @@
SAVE_4GPRS(3, r11)
SAVE_2GPRS(7, r11)
addi r2,r12,-THREAD
-#if defined(CONFIG_40x)
- /* Check to see if the dbcr0 register is set up to debug. Use the
- internal debug mode bit to do this. */
- lwz r12,THREAD_DBCR0(r12)
- andis. r12,r12,DBCR0_IDM@h
- beq+ 3f
- /* From user and task is ptraced - load up global dbcr0 */
- li r12,-1 /* clear all pending debug events */
- mtspr SPRN_DBSR,r12
- lis r11,global_dbcr0@ha
- addi r11,r11,global_dbcr0@l
- lwz r12,0(r11)
- mtspr SPRN_DBCR0,r12
-3:
-#endif
b transfer_to_syscall /* jump to handler */
.endm
diff --git a/arch/powerpc/kernel/head_booke.h b/arch/powerpc/kernel/head_booke.h
index 5f565232b99d..47857795f50a 100644
--- a/arch/powerpc/kernel/head_booke.h
+++ b/arch/powerpc/kernel/head_booke.h
@@ -130,25 +130,6 @@ ALT_FTR_SECTION_END_IFSET(CPU_FTR_EMB_HV)
SAVE_2GPRS(7, r11)
addi r2,r10,-THREAD
- /* Check to see if the dbcr0 register is set up to debug. Use the
- internal debug mode bit to do this. */
- lwz r12,THREAD_DBCR0(r10)
- andis. r12,r12,DBCR0_IDM@h
- beq+ 3f
- /* From user and task is ptraced - load up global dbcr0 */
- li r12,-1 /* clear all pending debug events */
- mtspr SPRN_DBSR,r12
- lis r11,global_dbcr0@ha
- addi r11,r11,global_dbcr0@l
-#ifdef CONFIG_SMP
- lwz r10, TASK_CPU(r2)
- slwi r10, r10, 2
- add r11, r11, r10
-#endif
- lwz r12,0(r11)
- mtspr SPRN_DBCR0,r12
-
-3:
b transfer_to_syscall /* jump to handler */
.endm
diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
index c89a8eac3e24..6111acf61373 100644
--- a/arch/powerpc/kernel/interrupt.c
+++ b/arch/powerpc/kernel/interrupt.c
@@ -76,6 +76,13 @@ notrace long system_call_exception(long r3, long r4, long r5,
kuap_check_amr();
#endif
+#ifdef CONFIG_PPC_ADV_DEBUG_REGS
+ if (IS_ENABLED(CONFIG_PPC32) && unlikely(current->thread.debug.dbcr0 & DBCR0_IDM)) {
+ mtspr(SPRN_DBSR, -1);
+ mtspr(SPRN_DBCR0, global_dbcr0[smp_processor_id()]);
+ }
+#endif
+
account_cpu_user_entry();
account_stolen_time();
@@ -324,6 +331,22 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,
local_paca->tm_scratch = regs->msr;
#endif
+#ifdef CONFIG_PPC_ADV_DEBUG_REGS
+ if (unlikely(current->thread.debug.dbcr0 & DBCR0_IDM)) {
+ /*
+ * Check to see if the dbcr0 register is set up to debug.
+ * Use the internal debug mode bit to do this.
+ */
+ mtmsr(mfmsr() & ~MSR_DE);
+ if (IS_ENABLED(CONFIG_PPC32)) {
+ isync();
+ global_dbcr0[smp_processor_id()] = mfspr(SPRN_DBCR0);
+ }
+ mtspr(SPRN_DBCR0, current->thread.debug.dbcr0);
+ mtspr(SPRN_DBSR, -1);
+ }
+#endif
+
account_cpu_user_exit();
#ifdef CONFIG_PPC_BOOK3S_64 /* BOOK3E and ppc32 not using this */
@@ -401,13 +424,17 @@ notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs, unsigned
goto again;
}
-#ifdef CONFIG_PPC_BOOK3E
+#ifdef CONFIG_PPC_ADV_DEBUG_REGS
if (unlikely(current->thread.debug.dbcr0 & DBCR0_IDM)) {
/*
* Check to see if the dbcr0 register is set up to debug.
* Use the internal debug mode bit to do this.
*/
mtmsr(mfmsr() & ~MSR_DE);
+ if (IS_ENABLED(CONFIG_PPC32)) {
+ isync();
+ global_dbcr0[smp_processor_id()] = mfspr(SPRN_DBCR0);
+ }
mtspr(SPRN_DBCR0, current->thread.debug.dbcr0);
mtspr(SPRN_DBSR, -1);
}
--
2.25.0
system_call_exception() checks MSR_PR and BUGs if a syscall
is issued from kernel mode.
No need to handle it anymore from the ASM entry code.
null_syscall reduction 2 cycles (348 => 346 cycles)
Signed-off-by: Christophe Leroy <[email protected]>
---
arch/powerpc/kernel/entry_32.S | 30 ------------------------------
arch/powerpc/kernel/head_32.h | 3 ---
arch/powerpc/kernel/head_booke.h | 3 ---
3 files changed, 36 deletions(-)
diff --git a/arch/powerpc/kernel/entry_32.S b/arch/powerpc/kernel/entry_32.S
index bbf7ecea6fe0..cffe58e63356 100644
--- a/arch/powerpc/kernel/entry_32.S
+++ b/arch/powerpc/kernel/entry_32.S
@@ -421,36 +421,6 @@ ret_from_kernel_thread:
li r3,0
b ret_from_syscall
- /*
- * System call was called from kernel. We get here with SRR1 in r9.
- * Mark the exception as recoverable once we have retrieved SRR0,
- * trap a warning and return ENOSYS with CR[SO] set.
- */
- .globl ret_from_kernel_syscall
-ret_from_kernel_syscall:
- mfspr r9, SPRN_SRR0
- mfspr r10, SPRN_SRR1
-#if !defined(CONFIG_4xx) && !defined(CONFIG_BOOKE)
- LOAD_REG_IMMEDIATE(r11, MSR_KERNEL & ~(MSR_IR|MSR_DR))
- mtmsr r11
-#endif
-
-0: trap
- EMIT_BUG_ENTRY 0b,__FILE__,__LINE__, BUGFLAG_WARNING
-
- li r3, ENOSYS
- crset so
-#if defined(CONFIG_PPC_8xx) && defined(CONFIG_PERF_EVENTS)
- mtspr SPRN_NRI, r0
-#endif
- mtspr SPRN_SRR0, r9
- mtspr SPRN_SRR1, r10
- rfi
-#ifdef CONFIG_40x
- b . /* Prevent prefetch past rfi */
-#endif
-_ASM_NOKPROBE_SYMBOL(ret_from_kernel_syscall)
-
/*
* Top-level page fault handling.
* This is in assembler because if do_page_fault tells us that
diff --git a/arch/powerpc/kernel/head_32.h b/arch/powerpc/kernel/head_32.h
index fea7fe00a690..282d8fd443a9 100644
--- a/arch/powerpc/kernel/head_32.h
+++ b/arch/powerpc/kernel/head_32.h
@@ -118,8 +118,6 @@
.macro SYSCALL_ENTRY trapno
mfspr r9, SPRN_SRR1
mfspr r10, SPRN_SRR0
- andi. r11, r9, MSR_PR
- beq- 99f
LOAD_REG_IMMEDIATE(r11, MSR_KERNEL) /* can take exceptions */
lis r12, 1f@h
ori r12, r12, 1f@l
@@ -174,7 +172,6 @@
3:
#endif
b transfer_to_syscall /* jump to handler */
-99: b ret_from_kernel_syscall
.endm
.macro save_dar_dsisr_on_stack reg1, reg2, sp
diff --git a/arch/powerpc/kernel/head_booke.h b/arch/powerpc/kernel/head_booke.h
index db931f1167aa..bfbd240cc8a2 100644
--- a/arch/powerpc/kernel/head_booke.h
+++ b/arch/powerpc/kernel/head_booke.h
@@ -106,10 +106,8 @@ ALT_FTR_SECTION_END_IFSET(CPU_FTR_EMB_HV)
#endif
mfspr r9, SPRN_SRR1
BOOKE_CLEAR_BTB(r11)
- andi. r11, r9, MSR_PR
lwz r11, TASK_STACK - THREAD(r10)
rlwinm r12,r12,0,4,2 /* Clear SO bit in CR */
- beq- 99f
ALLOC_STACK_FRAME(r11, THREAD_SIZE - INT_FRAME_SIZE)
stw r12, _CCR(r11) /* save various registers */
mflr r12
@@ -155,7 +153,6 @@ ALT_FTR_SECTION_END_IFSET(CPU_FTR_EMB_HV)
3:
b transfer_to_syscall /* jump to handler */
-99: b ret_from_kernel_syscall
.endm
/* To handle the additional exception priority levels on 40x and Book-E
--
2.25.0
For book3s/64, FULL_REGS() is 'true' at all time, so the test voids.
For others, non volatile registers are saved inconditionally.
So the verification is pointless.
Should one fail to do it, it would anyway be caught by the
CHECK_FULL_REGS() in copy_thread() as we have removed the
special versions ppc_fork() and friends.
null_syscall benchmark reduction 4 cycles (332 => 328 cycles)
Signed-off-by: Christophe Leroy <[email protected]>
---
arch/powerpc/kernel/interrupt.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
index 8fafca727b8b..55e1aa18cdb9 100644
--- a/arch/powerpc/kernel/interrupt.c
+++ b/arch/powerpc/kernel/interrupt.c
@@ -42,7 +42,6 @@ notrace long system_call_exception(long r3, long r4, long r5,
if (!IS_ENABLED(CONFIG_BOOKE) && !IS_ENABLED(CONFIG_40x))
BUG_ON(!(regs->msr & MSR_RI));
BUG_ON(!(regs->msr & MSR_PR));
- BUG_ON(!FULL_REGS(regs));
BUG_ON(arch_irq_disabled_regs(regs));
#ifdef CONFIG_PPC_PKEY
--
2.25.0
When r3 is not modified, reload it from regs->orig_r3 to free
volatile registers. This avoids a stack frame for the likely part
of system_call_exception()
Before the patch:
c000b4d4 <system_call_exception>:
c000b4d4: 7c 08 02 a6 mflr r0
c000b4d8: 94 21 ff e0 stwu r1,-32(r1)
c000b4dc: 93 e1 00 1c stw r31,28(r1)
c000b4e0: 90 01 00 24 stw r0,36(r1)
c000b4e4: 90 6a 00 88 stw r3,136(r10)
c000b4e8: 81 6a 00 84 lwz r11,132(r10)
c000b4ec: 69 6b 00 02 xori r11,r11,2
c000b4f0: 55 6b ff fe rlwinm r11,r11,31,31,31
c000b4f4: 0f 0b 00 00 twnei r11,0
c000b4f8: 81 6a 00 a0 lwz r11,160(r10)
c000b4fc: 55 6b 07 fe clrlwi r11,r11,31
c000b500: 0f 0b 00 00 twnei r11,0
c000b504: 7c 0c 42 e6 mftb r0
c000b508: 83 e2 00 08 lwz r31,8(r2)
c000b50c: 81 82 00 28 lwz r12,40(r2)
c000b510: 90 02 00 24 stw r0,36(r2)
c000b514: 7d 8c f8 50 subf r12,r12,r31
c000b518: 7c 0c 02 14 add r0,r12,r0
c000b51c: 90 02 00 08 stw r0,8(r2)
c000b520: 7c 10 13 a6 mtspr 80,r0
c000b524: 81 62 00 70 lwz r11,112(r2)
c000b528: 71 60 86 91 andi. r0,r11,34449
c000b52c: 40 82 00 34 bne c000b560 <system_call_exception+0x8c>
c000b530: 2b 89 01 b6 cmplwi cr7,r9,438
c000b534: 41 9d 00 64 bgt cr7,c000b598 <system_call_exception+0xc4>
c000b538: 3d 40 c0 5c lis r10,-16292
c000b53c: 55 29 10 3a rlwinm r9,r9,2,0,29
c000b540: 39 4a 41 e8 addi r10,r10,16872
c000b544: 80 01 00 24 lwz r0,36(r1)
c000b548: 7d 2a 48 2e lwzx r9,r10,r9
c000b54c: 7c 08 03 a6 mtlr r0
c000b550: 7d 29 03 a6 mtctr r9
c000b554: 83 e1 00 1c lwz r31,28(r1)
c000b558: 38 21 00 20 addi r1,r1,32
c000b55c: 4e 80 04 20 bctr
After the patch:
c000b4d4 <system_call_exception>:
c000b4d4: 81 6a 00 84 lwz r11,132(r10)
c000b4d8: 90 6a 00 88 stw r3,136(r10)
c000b4dc: 69 6b 00 02 xori r11,r11,2
c000b4e0: 55 6b ff fe rlwinm r11,r11,31,31,31
c000b4e4: 0f 0b 00 00 twnei r11,0
c000b4e8: 80 6a 00 a0 lwz r3,160(r10)
c000b4ec: 54 63 07 fe clrlwi r3,r3,31
c000b4f0: 0f 03 00 00 twnei r3,0
c000b4f4: 7d 6c 42 e6 mftb r11
c000b4f8: 81 82 00 08 lwz r12,8(r2)
c000b4fc: 80 02 00 28 lwz r0,40(r2)
c000b500: 91 62 00 24 stw r11,36(r2)
c000b504: 7c 00 60 50 subf r0,r0,r12
c000b508: 7d 60 5a 14 add r11,r0,r11
c000b50c: 91 62 00 08 stw r11,8(r2)
c000b510: 7c 10 13 a6 mtspr 80,r0
c000b514: 80 62 00 70 lwz r3,112(r2)
c000b518: 70 6b 86 91 andi. r11,r3,34449
c000b51c: 40 82 00 28 bne c000b544 <system_call_exception+0x70>
c000b520: 2b 89 01 b6 cmplwi cr7,r9,438
c000b524: 41 9d 00 84 bgt cr7,c000b5a8 <system_call_exception+0xd4>
c000b528: 80 6a 00 88 lwz r3,136(r10)
c000b52c: 3d 40 c0 5c lis r10,-16292
c000b530: 55 29 10 3a rlwinm r9,r9,2,0,29
c000b534: 39 4a 41 e4 addi r10,r10,16868
c000b538: 7d 2a 48 2e lwzx r9,r10,r9
c000b53c: 7d 29 03 a6 mtctr r9
c000b540: 4e 80 04 20 bctr
Signed-off-by: Christophe Leroy <[email protected]>
---
arch/powerpc/kernel/interrupt.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
index 107ec39f05cb..205902052112 100644
--- a/arch/powerpc/kernel/interrupt.c
+++ b/arch/powerpc/kernel/interrupt.c
@@ -117,6 +117,9 @@ notrace long system_call_exception(long r3, long r4, long r5,
return regs->gpr[3];
}
return -ENOSYS;
+ } else {
+ /* Restore r3 from orig_gpr3 to free up a volatile reg */
+ r3 = regs->orig_gpr3;
}
/* May be faster to do array_index_nospec? */
--
2.25.0
By saving the pointer pointing to thread_info.flags, gcc copies r2
in a non-volatile register.
We know 'current' doesn't change, so avoid that intermediaite pointer.
Reduces null_syscall benchmark by 2 cycles (322 => 320 cycles)
On PPC64, gcc seems to know that 'current' is not changing, and it keeps
it in a non volatile register to avoid multiple read of 'current' in paca.
Signed-off-by: Christophe Leroy <[email protected]>
---
v5: Also in interrupt exit prepare
---
arch/powerpc/kernel/interrupt.c | 28 +++++++++++-----------------
1 file changed, 11 insertions(+), 17 deletions(-)
diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
index 8c38e8c95be2..c89a8eac3e24 100644
--- a/arch/powerpc/kernel/interrupt.c
+++ b/arch/powerpc/kernel/interrupt.c
@@ -223,7 +223,6 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,
struct pt_regs *regs,
long scv)
{
- unsigned long *ti_flagsp = ¤t_thread_info()->flags;
unsigned long ti_flags;
unsigned long ret = 0;
@@ -241,7 +240,7 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,
/* Check whether the syscall is issued inside a restartable sequence */
rseq_syscall(regs);
- ti_flags = *ti_flagsp;
+ ti_flags = current_thread_info()->flags;
if (unlikely(r3 >= (unsigned long)-MAX_ERRNO) && !scv) {
if (likely(!(ti_flags & (_TIF_NOERROR | _TIF_RESTOREALL)))) {
@@ -255,7 +254,7 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,
ret = _TIF_RESTOREALL;
else
regs->gpr[3] = r3;
- clear_bits(_TIF_PERSYSCALL_MASK, ti_flagsp);
+ clear_bits(_TIF_PERSYSCALL_MASK, ¤t_thread_info()->flags);
} else {
regs->gpr[3] = r3;
}
@@ -268,7 +267,7 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,
local_irq_disable();
again:
- ti_flags = READ_ONCE(*ti_flagsp);
+ ti_flags = READ_ONCE(current_thread_info()->flags);
while (unlikely(ti_flags & (_TIF_USER_WORK_MASK & ~_TIF_RESTORE_TM))) {
local_irq_enable();
if (ti_flags & _TIF_NEED_RESCHED) {
@@ -284,7 +283,7 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,
do_notify_resume(regs, ti_flags);
}
local_irq_disable();
- ti_flags = READ_ONCE(*ti_flagsp);
+ ti_flags = READ_ONCE(current_thread_info()->flags);
}
if (IS_ENABLED(CONFIG_PPC_BOOK3S) && IS_ENABLED(CONFIG_PPC_FPU)) {
@@ -339,10 +338,6 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,
#ifndef CONFIG_PPC_BOOK3E_64 /* BOOK3E not yet using this */
notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs, unsigned long msr)
{
-#ifdef CONFIG_PPC_BOOK3E
- struct thread_struct *ts = ¤t->thread;
-#endif
- unsigned long *ti_flagsp = ¤t_thread_info()->flags;
unsigned long ti_flags;
unsigned long flags;
unsigned long ret = 0;
@@ -365,7 +360,7 @@ notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs, unsigned
local_irq_save(flags);
again:
- ti_flags = READ_ONCE(*ti_flagsp);
+ ti_flags = READ_ONCE(current_thread_info()->flags);
while (unlikely(ti_flags & (_TIF_USER_WORK_MASK & ~_TIF_RESTORE_TM))) {
local_irq_enable(); /* returning to user: may enable */
if (ti_flags & _TIF_NEED_RESCHED) {
@@ -376,7 +371,7 @@ notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs, unsigned
do_notify_resume(regs, ti_flags);
}
local_irq_disable();
- ti_flags = READ_ONCE(*ti_flagsp);
+ ti_flags = READ_ONCE(current_thread_info()->flags);
}
if (IS_ENABLED(CONFIG_PPC_BOOK3S) && IS_ENABLED(CONFIG_PPC_FPU)) {
@@ -407,13 +402,13 @@ notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs, unsigned
}
#ifdef CONFIG_PPC_BOOK3E
- if (unlikely(ts->debug.dbcr0 & DBCR0_IDM)) {
+ if (unlikely(current->thread.debug.dbcr0 & DBCR0_IDM)) {
/*
* Check to see if the dbcr0 register is set up to debug.
* Use the internal debug mode bit to do this.
*/
mtmsr(mfmsr() & ~MSR_DE);
- mtspr(SPRN_DBCR0, ts->debug.dbcr0);
+ mtspr(SPRN_DBCR0, current->thread.debug.dbcr0);
mtspr(SPRN_DBSR, -1);
}
#endif
@@ -438,7 +433,6 @@ void preempt_schedule_irq(void);
notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs, unsigned long msr)
{
- unsigned long *ti_flagsp = ¤t_thread_info()->flags;
unsigned long flags;
unsigned long ret = 0;
#ifdef CONFIG_PPC64
@@ -461,8 +455,8 @@ notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs, unsign
amr = kuap_get_and_check_amr();
#endif
- if (unlikely(*ti_flagsp & _TIF_EMULATE_STACK_STORE)) {
- clear_bits(_TIF_EMULATE_STACK_STORE, ti_flagsp);
+ if (unlikely(current_thread_info()->flags & _TIF_EMULATE_STACK_STORE)) {
+ clear_bits(_TIF_EMULATE_STACK_STORE, ¤t_thread_info()->flags);
ret = 1;
}
@@ -474,7 +468,7 @@ notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs, unsign
again:
if (IS_ENABLED(CONFIG_PREEMPT)) {
/* Return to preemptible kernel context */
- if (unlikely(*ti_flagsp & _TIF_NEED_RESCHED)) {
+ if (unlikely(current_thread_info()->flags & _TIF_NEED_RESCHED)) {
if (preempt_count() == 0)
preempt_schedule_irq();
}
--
2.25.0
Combine all tests of regs->msr into a single logical one.
Before the patch:
0: 81 6a 00 84 lwz r11,132(r10)
4: 90 6a 00 88 stw r3,136(r10)
8: 69 60 00 02 xori r0,r11,2
c: 54 00 ff fe rlwinm r0,r0,31,31,31
10: 0f 00 00 00 twnei r0,0
14: 69 63 40 00 xori r3,r11,16384
18: 54 63 97 fe rlwinm r3,r3,18,31,31
1c: 0f 03 00 00 twnei r3,0
20: 69 6b 80 00 xori r11,r11,32768
24: 55 6b 8f fe rlwinm r11,r11,17,31,31
28: 0f 0b 00 00 twnei r11,0
After the patch:
0: 81 6a 00 84 lwz r11,132(r10)
4: 90 6a 00 88 stw r3,136(r10)
8: 7d 6b 58 f8 not r11,r11
c: 71 6b c0 02 andi. r11,r11,49154
10: 0f 0b 00 00 twnei r11,0
6 cycles less on powerpc 8xx (328 => 322 cycles).
Signed-off-by: Christophe Leroy <[email protected]>
---
arch/powerpc/kernel/interrupt.c | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
index 55e1aa18cdb9..8c38e8c95be2 100644
--- a/arch/powerpc/kernel/interrupt.c
+++ b/arch/powerpc/kernel/interrupt.c
@@ -28,6 +28,7 @@ notrace long system_call_exception(long r3, long r4, long r5,
unsigned long r0, struct pt_regs *regs)
{
syscall_fn f;
+ unsigned long expected_msr;
regs->orig_gpr3 = r3;
@@ -39,10 +40,13 @@ notrace long system_call_exception(long r3, long r4, long r5,
trace_hardirqs_off(); /* finish reconciling */
+ expected_msr = MSR_PR;
if (!IS_ENABLED(CONFIG_BOOKE) && !IS_ENABLED(CONFIG_40x))
- BUG_ON(!(regs->msr & MSR_RI));
- BUG_ON(!(regs->msr & MSR_PR));
- BUG_ON(arch_irq_disabled_regs(regs));
+ expected_msr |= MSR_RI;
+ if (IS_ENABLED(CONFIG_PPC32))
+ expected_msr |= MSR_EE;
+ BUG_ON((regs->msr & expected_msr) ^ expected_msr);
+ BUG_ON(IS_ENABLED(CONFIG_PPC64) && arch_irq_disabled_regs(regs));
#ifdef CONFIG_PPC_PKEY
if (mmu_has_feature(MMU_FTR_PKEY)) {
--
2.25.0
Only PPC64 has scv. No need to check the 0x7ff0 trap on PPC32.
For that, add a helper trap_is_unsupported_scv() similar to
trap_is_scv().
And ignore the scv parameter in syscall_exit_prepare (Save 14 cycles
346 => 332 cycles)
Signed-off-by: Christophe Leroy <[email protected]>
---
v5: Added a helper trap_is_unsupported_scv()
---
arch/powerpc/include/asm/ptrace.h | 5 +++++
arch/powerpc/kernel/entry_32.S | 1 -
arch/powerpc/kernel/interrupt.c | 7 +++++--
3 files changed, 10 insertions(+), 3 deletions(-)
diff --git a/arch/powerpc/include/asm/ptrace.h b/arch/powerpc/include/asm/ptrace.h
index 58f9dc060a7b..2c842b11a924 100644
--- a/arch/powerpc/include/asm/ptrace.h
+++ b/arch/powerpc/include/asm/ptrace.h
@@ -229,6 +229,11 @@ static inline bool trap_is_scv(struct pt_regs *regs)
return (IS_ENABLED(CONFIG_PPC_BOOK3S_64) && TRAP(regs) == 0x3000);
}
+static inline bool trap_is_unsupported_scv(struct pt_regs *regs)
+{
+ return (IS_ENABLED(CONFIG_PPC_BOOK3S_64) && TRAP(regs) == 0x7ff0);
+}
+
static inline bool trap_is_syscall(struct pt_regs *regs)
{
return (trap_is_scv(regs) || TRAP(regs) == 0xc00);
diff --git a/arch/powerpc/kernel/entry_32.S b/arch/powerpc/kernel/entry_32.S
index cffe58e63356..7c824e8928d0 100644
--- a/arch/powerpc/kernel/entry_32.S
+++ b/arch/powerpc/kernel/entry_32.S
@@ -344,7 +344,6 @@ transfer_to_syscall:
ret_from_syscall:
addi r4,r1,STACK_FRAME_OVERHEAD
- li r5,0
bl syscall_exit_prepare
#if defined(CONFIG_4xx) || defined(CONFIG_BOOKE)
/* If the process has its own DBCR0 value, load it up. The internal
diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
index 205902052112..8fafca727b8b 100644
--- a/arch/powerpc/kernel/interrupt.c
+++ b/arch/powerpc/kernel/interrupt.c
@@ -88,7 +88,7 @@ notrace long system_call_exception(long r3, long r4, long r5,
local_irq_enable();
if (unlikely(current_thread_info()->flags & _TIF_SYSCALL_DOTRACE)) {
- if (unlikely(regs->trap == 0x7ff0)) {
+ if (unlikely(trap_is_unsupported_scv(regs))) {
/* Unsupported scv vector */
_exception(SIGILL, regs, ILL_ILLOPC, regs->nip);
return regs->gpr[3];
@@ -111,7 +111,7 @@ notrace long system_call_exception(long r3, long r4, long r5,
r8 = regs->gpr[8];
} else if (unlikely(r0 >= NR_syscalls)) {
- if (unlikely(regs->trap == 0x7ff0)) {
+ if (unlikely(trap_is_unsupported_scv(regs))) {
/* Unsupported scv vector */
_exception(SIGILL, regs, ILL_ILLOPC, regs->nip);
return regs->gpr[3];
@@ -224,6 +224,9 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,
unsigned long ti_flags;
unsigned long ret = 0;
+ if (IS_ENABLED(CONFIG_PPC32))
+ scv = 0;
+
CT_WARN_ON(ct_state() == CONTEXT_USER);
#ifdef CONFIG_PPC64
--
2.25.0
Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
> This series implements C syscall entry/exit for PPC32. It reuses
> the work already done for PPC64.
>
> This series is based on today's merge-test (b6f72fc05389e3fc694bf5a5fa1bbd33f61879e0)
>
> In terms on performance we have the following number of cycles on an
> 8xx running null_syscall benchmark:
> - mainline: 296 cycles
> - after patch 4: 283 cycles
> - after patch 16: 304 cycles
> - after patch 17: 348 cycles
> - at the end of the series: 320 cycles
>
> So in summary, we have a degradation of performance of 8% on null_syscall.
>
> I think it is not a big degradation, it is worth it.
I guess it's 13% from 283. But it's very nice to use the shared C code.
There might be a few more percent speedup in there we can find later.
Thanks,
Nick
Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
> regs->softe doesn't exist on PPC32.
>
> Add irq_soft_mask_regs_set_state() helper to set regs->softe.
> This helper will void on PPC32.
>
> Signed-off-by: Christophe Leroy <[email protected]>
> ---
> arch/powerpc/include/asm/hw_irq.h | 11 +++++++++--
> 1 file changed, 9 insertions(+), 2 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/hw_irq.h b/arch/powerpc/include/asm/hw_irq.h
> index 614957f74cee..ed0c3b049dfd 100644
> --- a/arch/powerpc/include/asm/hw_irq.h
> +++ b/arch/powerpc/include/asm/hw_irq.h
> @@ -38,6 +38,8 @@
> #define PACA_IRQ_MUST_HARD_MASK (PACA_IRQ_EE)
> #endif
>
> +#endif /* CONFIG_PPC64 */
> +
> /*
> * flags for paca->irq_soft_mask
> */
> @@ -46,8 +48,6 @@
> #define IRQS_PMI_DISABLED 2
> #define IRQS_ALL_DISABLED (IRQS_DISABLED | IRQS_PMI_DISABLED)
>
> -#endif /* CONFIG_PPC64 */
> -
> #ifndef __ASSEMBLY__
>
> #ifdef CONFIG_PPC64
> @@ -287,6 +287,10 @@ extern void irq_set_pending_from_srr1(unsigned long srr1);
>
> extern void force_external_irq_replay(void);
>
> +static inline void irq_soft_mask_regs_set_state(struct pt_regs *regs, unsigned long val)
> +{
> + regs->softe = val;
> +}
> #else /* CONFIG_PPC64 */
>
> static inline unsigned long arch_local_save_flags(void)
> @@ -355,6 +359,9 @@ static inline bool arch_irq_disabled_regs(struct pt_regs *regs)
>
> static inline void may_hard_irq_enable(void) { }
>
> +static inline void irq_soft_mask_regs_set_state(struct pt_regs *regs, unsigned long val)
> +{
> +}
> #endif /* CONFIG_PPC64 */
>
> #define ARCH_IRQ_INIT_FLAGS IRQ_NOREQUEST
What I don't like about this where you use it is it kind of pollutes
the ppc32 path with this function which is not valid to use.
I would prefer if you had this purely so it could compile with:
if (IS_ENABLED(CONFIG_PPC64)))
irq_soft_mask_regs_set_state(regs, blah);
And then you could make the ppc32 cause a link error if it did not
get eliminated at compile time (e.g., call an undefined function).
You could do the same with the kuap_ functions to change some ifdefs
to IS_ENABLED.
That's just my preference but if you prefer this way I guess that's
okay.
Thanks,
Nick
Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
> In preparation of porting PPC32 to C syscall entry/exit,
> rewrite the following helpers as static inline functions and
> add support for PPC32 in them:
> __hard_irq_enable()
> __hard_irq_disable()
> __hard_EE_RI_disable()
> __hard_RI_enable()
>
> Then use them in PPC32 version of arch_local_irq_disable()
> and arch_local_irq_enable() to avoid code duplication.
>
Reviewed-by: Nicholas Piggin <[email protected]>
> Signed-off-by: Christophe Leroy <[email protected]>
> ---
> arch/powerpc/include/asm/hw_irq.h | 75 +++++++++++++++++++++----------
> arch/powerpc/include/asm/reg.h | 1 +
> 2 files changed, 52 insertions(+), 24 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/hw_irq.h b/arch/powerpc/include/asm/hw_irq.h
> index ed0c3b049dfd..4739f61e632c 100644
> --- a/arch/powerpc/include/asm/hw_irq.h
> +++ b/arch/powerpc/include/asm/hw_irq.h
> @@ -50,6 +50,55 @@
>
> #ifndef __ASSEMBLY__
>
> +static inline void __hard_irq_enable(void)
> +{
> + if (IS_ENABLED(CONFIG_BOOKE) || IS_ENABLED(CONFIG_40x))
> + wrtee(MSR_EE);
> + else if (IS_ENABLED(CONFIG_PPC_8xx))
> + wrtspr(SPRN_EIE);
> + else if (IS_ENABLED(CONFIG_PPC_BOOK3S_64))
> + __mtmsrd(MSR_EE | MSR_RI, 1);
> + else
> + mtmsr(mfmsr() | MSR_EE);
> +}
> +
> +static inline void __hard_irq_disable(void)
> +{
> + if (IS_ENABLED(CONFIG_BOOKE) || IS_ENABLED(CONFIG_40x))
> + wrtee(0);
> + else if (IS_ENABLED(CONFIG_PPC_8xx))
> + wrtspr(SPRN_EID);
> + else if (IS_ENABLED(CONFIG_PPC_BOOK3S_64))
> + __mtmsrd(MSR_RI, 1);
> + else
> + mtmsr(mfmsr() & ~MSR_EE);
> +}
> +
> +static inline void __hard_EE_RI_disable(void)
> +{
> + if (IS_ENABLED(CONFIG_BOOKE) || IS_ENABLED(CONFIG_40x))
> + wrtee(0);
> + else if (IS_ENABLED(CONFIG_PPC_8xx))
> + wrtspr(SPRN_NRI);
> + else if (IS_ENABLED(CONFIG_PPC_BOOK3S_64))
> + __mtmsrd(0, 1);
> + else
> + mtmsr(mfmsr() & ~(MSR_EE | MSR_RI));
> +}
> +
> +static inline void __hard_RI_enable(void)
> +{
> + if (IS_ENABLED(CONFIG_BOOKE) || IS_ENABLED(CONFIG_40x))
> + return;
> +
> + if (IS_ENABLED(CONFIG_PPC_8xx))
> + wrtspr(SPRN_EID);
> + else if (IS_ENABLED(CONFIG_PPC_BOOK3S_64))
> + __mtmsrd(MSR_RI, 1);
> + else
> + mtmsr(mfmsr() | MSR_RI);
> +}
> +
> #ifdef CONFIG_PPC64
> #include <asm/paca.h>
>
> @@ -212,18 +261,6 @@ static inline bool arch_irqs_disabled(void)
>
> #endif /* CONFIG_PPC_BOOK3S */
>
> -#ifdef CONFIG_PPC_BOOK3E
> -#define __hard_irq_enable() wrtee(MSR_EE)
> -#define __hard_irq_disable() wrtee(0)
> -#define __hard_EE_RI_disable() wrtee(0)
> -#define __hard_RI_enable() do { } while (0)
> -#else
> -#define __hard_irq_enable() __mtmsrd(MSR_EE|MSR_RI, 1)
> -#define __hard_irq_disable() __mtmsrd(MSR_RI, 1)
> -#define __hard_EE_RI_disable() __mtmsrd(0, 1)
> -#define __hard_RI_enable() __mtmsrd(MSR_RI, 1)
> -#endif
> -
> #define hard_irq_disable() do { \
> unsigned long flags; \
> __hard_irq_disable(); \
> @@ -322,22 +359,12 @@ static inline unsigned long arch_local_irq_save(void)
>
> static inline void arch_local_irq_disable(void)
> {
> - if (IS_ENABLED(CONFIG_BOOKE))
> - wrtee(0);
> - else if (IS_ENABLED(CONFIG_PPC_8xx))
> - wrtspr(SPRN_EID);
> - else
> - mtmsr(mfmsr() & ~MSR_EE);
> + __hard_irq_disable();
> }
>
> static inline void arch_local_irq_enable(void)
> {
> - if (IS_ENABLED(CONFIG_BOOKE))
> - wrtee(MSR_EE);
> - else if (IS_ENABLED(CONFIG_PPC_8xx))
> - wrtspr(SPRN_EIE);
> - else
> - mtmsr(mfmsr() | MSR_EE);
> + __hard_irq_enable();
> }
>
> static inline bool arch_irqs_disabled_flags(unsigned long flags)
> diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/reg.h
> index c5a3e856191c..bc4305ba00d0 100644
> --- a/arch/powerpc/include/asm/reg.h
> +++ b/arch/powerpc/include/asm/reg.h
> @@ -1375,6 +1375,7 @@
> #define mtmsr(v) asm volatile("mtmsr %0" : \
> : "r" ((unsigned long)(v)) \
> : "memory")
> +#define __mtmsrd(v, l) BUILD_BUG()
> #define __MTMSR "mtmsr"
> #endif
>
> --
> 2.25.0
>
>
Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
> To allow building syscall_64.c smoothly on PPC32, add stub version
> of irq_soft_mask_return().
>
> Signed-off-by: Christophe Leroy <[email protected]>
Same kind of comment as the other soft mask stuff. Again not a big deal
but there might be a way to improve it. For example make a
debug_syscall_entry(regs) function that ppc64 could put the soft mask
checks into.
No big deal, if you don't make any changes now I might see about doing
something like that after your series goes in.
Reviewed-by: Nicholas Piggin <[email protected]>
> ---
> arch/powerpc/include/asm/hw_irq.h | 5 +++++
> 1 file changed, 5 insertions(+)
>
> diff --git a/arch/powerpc/include/asm/hw_irq.h b/arch/powerpc/include/asm/hw_irq.h
> index 4739f61e632c..56a98936a6a9 100644
> --- a/arch/powerpc/include/asm/hw_irq.h
> +++ b/arch/powerpc/include/asm/hw_irq.h
> @@ -330,6 +330,11 @@ static inline void irq_soft_mask_regs_set_state(struct pt_regs *regs, unsigned l
> }
> #else /* CONFIG_PPC64 */
>
> +static inline notrace unsigned long irq_soft_mask_return(void)
> +{
> + return 0;
> +}
> +
> static inline unsigned long arch_local_save_flags(void)
> {
> return mfmsr();
> --
> 2.25.0
>
>
Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
> syscall_64.c will be reused almost as is for PPC32.
>
> As this file also contains functions to handle other types
> of interrupts rename it interrupt.c
>
> Signed-off-by: Christophe Leroy <[email protected]>
Reviewed-by: Nicholas Piggin <[email protected]>
> ---
> arch/powerpc/kernel/Makefile | 2 +-
> arch/powerpc/kernel/{syscall_64.c => interrupt.c} | 0
> 2 files changed, 1 insertion(+), 1 deletion(-)
> rename arch/powerpc/kernel/{syscall_64.c => interrupt.c} (100%)
>
> diff --git a/arch/powerpc/kernel/Makefile b/arch/powerpc/kernel/Makefile
> index c173efd66c00..26ff8c6e06b7 100644
> --- a/arch/powerpc/kernel/Makefile
> +++ b/arch/powerpc/kernel/Makefile
> @@ -60,7 +60,7 @@ obj-y := cputable.o syscalls.o \
> hw_breakpoint_constraints.o
> obj-y += ptrace/
> obj-$(CONFIG_PPC64) += setup_64.o \
> - paca.o nvram_64.o note.o syscall_64.o
> + paca.o nvram_64.o note.o interrupt.o
> obj-$(CONFIG_COMPAT) += sys_ppc32.o signal_32.o
> obj-$(CONFIG_VDSO32) += vdso32_wrapper.o
> obj-$(CONFIG_PPC_WATCHDOG) += watchdog.o
> diff --git a/arch/powerpc/kernel/syscall_64.c b/arch/powerpc/kernel/interrupt.c
> similarity index 100%
> rename from arch/powerpc/kernel/syscall_64.c
> rename to arch/powerpc/kernel/interrupt.c
> --
> 2.25.0
>
>
Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
> To allow building interrupt.c on PPC32, ifdef out specific PPC64
> code or use helpers which are available on both PP32 and PPC64
>
> Modify Makefile to always build interrupt.o
>
> Signed-off-by: Christophe Leroy <[email protected]>
> ---
> v5:
> - Also for interrupt exit preparation
> - Opted out kuap related code, ppc32 keeps it in ASM for the time being
> ---
> arch/powerpc/kernel/Makefile | 4 ++--
> arch/powerpc/kernel/interrupt.c | 31 ++++++++++++++++++++++++-------
> 2 files changed, 26 insertions(+), 9 deletions(-)
>
> diff --git a/arch/powerpc/kernel/Makefile b/arch/powerpc/kernel/Makefile
> index 26ff8c6e06b7..163755b1cef4 100644
> --- a/arch/powerpc/kernel/Makefile
> +++ b/arch/powerpc/kernel/Makefile
> @@ -57,10 +57,10 @@ obj-y := cputable.o syscalls.o \
> prom.o traps.o setup-common.o \
> udbg.o misc.o io.o misc_$(BITS).o \
> of_platform.o prom_parse.o firmware.o \
> - hw_breakpoint_constraints.o
> + hw_breakpoint_constraints.o interrupt.o
> obj-y += ptrace/
> obj-$(CONFIG_PPC64) += setup_64.o \
> - paca.o nvram_64.o note.o interrupt.o
> + paca.o nvram_64.o note.o
> obj-$(CONFIG_COMPAT) += sys_ppc32.o signal_32.o
> obj-$(CONFIG_VDSO32) += vdso32_wrapper.o
> obj-$(CONFIG_PPC_WATCHDOG) += watchdog.o
> diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
> index d6be4f9a67e5..2dac4d2bb1cf 100644
> --- a/arch/powerpc/kernel/interrupt.c
> +++ b/arch/powerpc/kernel/interrupt.c
> @@ -39,7 +39,7 @@ notrace long system_call_exception(long r3, long r4, long r5,
> BUG_ON(!(regs->msr & MSR_RI));
> BUG_ON(!(regs->msr & MSR_PR));
> BUG_ON(!FULL_REGS(regs));
> - BUG_ON(regs->softe != IRQS_ENABLED);
> + BUG_ON(arch_irq_disabled_regs(regs));
>
> #ifdef CONFIG_PPC_PKEY
> if (mmu_has_feature(MMU_FTR_PKEY)) {
> @@ -65,7 +65,9 @@ notrace long system_call_exception(long r3, long r4, long r5,
> isync();
> } else
> #endif
> +#ifdef CONFIG_PPC64
> kuap_check_amr();
> +#endif
Wouldn't mind trying to get rid of these ifdefs at some point, but
there's some kuap / keys changes going on recently so I'm happy enough
to let this settle then look at whether we can refactor.
>
> account_cpu_user_entry();
>
> @@ -77,7 +79,7 @@ notrace long system_call_exception(long r3, long r4, long r5,
> * frame, or if the unwinder was taught the first stack frame always
> * returns to user with IRQS_ENABLED, this store could be avoided!
> */
> - regs->softe = IRQS_ENABLED;
> + irq_soft_mask_regs_set_state(regs, IRQS_ENABLED);
>
> local_irq_enable();
>
> @@ -151,6 +153,7 @@ static notrace inline bool __prep_irq_for_enabled_exit(bool clear_ri)
> __hard_EE_RI_disable();
> else
> __hard_irq_disable();
> +#ifdef CONFIG_PPC64
> if (unlikely(lazy_irq_pending_nocheck())) {
> /* Took an interrupt, may have more exit work to do. */
> if (clear_ri)
> @@ -162,7 +165,7 @@ static notrace inline bool __prep_irq_for_enabled_exit(bool clear_ri)
> }
> local_paca->irq_happened = 0;
> irq_soft_mask_set(IRQS_ENABLED);
> -
> +#endif
> return true;
> }
>
Do we prefer space before return except in trivial cases?
> @@ -216,7 +219,9 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,
>
> CT_WARN_ON(ct_state() == CONTEXT_USER);
>
> +#ifdef CONFIG_PPC64
> kuap_check_amr();
> +#endif
>
> regs->result = r3;
>
> @@ -309,7 +314,7 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,
>
> account_cpu_user_exit();
>
> -#ifdef CONFIG_PPC_BOOK3S /* BOOK3E not yet using this */
> +#ifdef CONFIG_PPC_BOOK3S_64 /* BOOK3E and ppc32 not using this */
> /*
> * We do this at the end so that we do context switch with KERNEL AMR
> */
> @@ -318,7 +323,7 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,
> return ret;
> }
>
> -#ifdef CONFIG_PPC_BOOK3S /* BOOK3E not yet using this */
> +#ifndef CONFIG_PPC_BOOK3E_64 /* BOOK3E not yet using this */
> notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs, unsigned long msr)
> {
> #ifdef CONFIG_PPC_BOOK3E
Why are you building this for 32? I don't mind if it's just to keep
things similar and make it build for now, but you're not using it yet,
right?
Reviewed-by: Nicholas Piggin <[email protected]>
> @@ -333,14 +338,16 @@ notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs, unsigned
> BUG_ON(!(regs->msr & MSR_RI));
> BUG_ON(!(regs->msr & MSR_PR));
> BUG_ON(!FULL_REGS(regs));
> - BUG_ON(regs->softe != IRQS_ENABLED);
> + BUG_ON(arch_irq_disabled_regs(regs));
> CT_WARN_ON(ct_state() == CONTEXT_USER);
>
> /*
> * We don't need to restore AMR on the way back to userspace for KUAP.
> * AMR can only have been unlocked if we interrupted the kernel.
> */
> +#ifdef CONFIG_PPC64
> kuap_check_amr();
> +#endif
>
> local_irq_save(flags);
>
> @@ -407,7 +414,9 @@ notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs, unsigned
> /*
> * We do this at the end so that we do context switch with KERNEL AMR
> */
> +#ifdef CONFIG_PPC64
> kuap_user_restore(regs);
> +#endif
> return ret;
> }
>
> @@ -419,7 +428,9 @@ notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs, unsign
> unsigned long *ti_flagsp = ¤t_thread_info()->flags;
> unsigned long flags;
> unsigned long ret = 0;
> +#ifdef CONFIG_PPC64
> unsigned long amr;
> +#endif
>
> if (IS_ENABLED(CONFIG_PPC_BOOK3S) && unlikely(!(regs->msr & MSR_RI)))
> unrecoverable_exception(regs);
> @@ -432,7 +443,9 @@ notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs, unsign
> if (TRAP(regs) != 0x700)
> CT_WARN_ON(ct_state() == CONTEXT_USER);
>
> +#ifdef CONFIG_PPC64
> amr = kuap_get_and_check_amr();
> +#endif
>
> if (unlikely(*ti_flagsp & _TIF_EMULATE_STACK_STORE)) {
> clear_bits(_TIF_EMULATE_STACK_STORE, ti_flagsp);
> @@ -441,7 +454,7 @@ notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs, unsign
>
> local_irq_save(flags);
>
> - if (regs->softe == IRQS_ENABLED) {
> + if (!arch_irq_disabled_regs(regs)) {
> /* Returning to a kernel context with local irqs enabled. */
> WARN_ON_ONCE(!(regs->msr & MSR_EE));
> again:
> @@ -458,8 +471,10 @@ notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs, unsign
> } else {
> /* Returning to a kernel context with local irqs disabled. */
> __hard_EE_RI_disable();
> +#ifdef CONFIG_PPC64
> if (regs->msr & MSR_EE)
> local_paca->irq_happened &= ~PACA_IRQ_HARD_DIS;
> +#endif
> }
>
>
> @@ -472,7 +487,9 @@ notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs, unsign
> * which would cause Read-After-Write stalls. Hence, we take the AMR
> * value from the check above.
> */
> +#ifdef CONFIG_PPC64
> kuap_kernel_restore(regs, amr);
> +#endif
>
> return ret;
> }
> --
> 2.25.0
>
>
Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
> Instead of hard comparing task flags with _TIF_32BIT, use
> is_compat_task(). The advantage is that it returns 0 on PPC32
> allthough _TIF_32BIT is always set.
>
> Signed-off-by: Christophe Leroy <[email protected]>
Reviewed-by: Nicholas Piggin <[email protected]>
> ---
> arch/powerpc/kernel/interrupt.c | 4 +++-
> 1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
> index 2dac4d2bb1cf..46fd195ca659 100644
> --- a/arch/powerpc/kernel/interrupt.c
> +++ b/arch/powerpc/kernel/interrupt.c
> @@ -2,6 +2,8 @@
>
> #include <linux/context_tracking.h>
> #include <linux/err.h>
> +#include <linux/compat.h>
> +
> #include <asm/asm-prototypes.h>
> #include <asm/kup.h>
> #include <asm/cputime.h>
> @@ -118,7 +120,7 @@ notrace long system_call_exception(long r3, long r4, long r5,
> /* May be faster to do array_index_nospec? */
> barrier_nospec();
>
> - if (unlikely(is_32bit_task())) {
> + if (unlikely(is_compat_task())) {
> f = (void *)compat_sys_call_table[r0];
>
> r3 &= 0x00000000ffffffffULL;
> --
> 2.25.0
>
>
Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
> Save r3 in regs->orig_r3 in system_call_exception()
>
> Signed-off-by: Christophe Leroy <[email protected]>
Reviewed-by: Nicholas Piggin <[email protected]>
> ---
> v5: Removed the assembly one on SCV type system call
> ---
> arch/powerpc/kernel/entry_64.S | 2 --
> arch/powerpc/kernel/interrupt.c | 2 ++
> 2 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/arch/powerpc/kernel/entry_64.S b/arch/powerpc/kernel/entry_64.S
> index 33ddfeef4fe9..a91c2def165d 100644
> --- a/arch/powerpc/kernel/entry_64.S
> +++ b/arch/powerpc/kernel/entry_64.S
> @@ -108,7 +108,6 @@ END_FTR_SECTION_IFSET(CPU_FTR_TM)
> li r11,\trapnr
> std r11,_TRAP(r1)
> std r12,_CCR(r1)
> - std r3,ORIG_GPR3(r1)
> addi r10,r1,STACK_FRAME_OVERHEAD
> ld r11,exception_marker@toc(r2)
> std r11,-16(r10) /* "regshere" marker */
> @@ -278,7 +277,6 @@ END_BTB_FLUSH_SECTION
> std r10,_LINK(r1)
> std r11,_TRAP(r1)
> std r12,_CCR(r1)
> - std r3,ORIG_GPR3(r1)
> addi r10,r1,STACK_FRAME_OVERHEAD
> ld r11,exception_marker@toc(r2)
> std r11,-16(r10) /* "regshere" marker */
> diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
> index 46fd195ca659..1a2dec49f811 100644
> --- a/arch/powerpc/kernel/interrupt.c
> +++ b/arch/powerpc/kernel/interrupt.c
> @@ -29,6 +29,8 @@ notrace long system_call_exception(long r3, long r4, long r5,
> {
> syscall_fn f;
>
> + regs->orig_gpr3 = r3;
> +
> if (IS_ENABLED(CONFIG_PPC_IRQ_SOFT_MASK_DEBUG))
> BUG_ON(irq_soft_mask_return() != IRQS_ALL_DISABLED);
>
> --
> 2.25.0
>
>
Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
> In system_call_exception(), MSR_RI also needs to be checked on 8xx.
> Only booke and 40x doesn't have MSR_RI.
Reviewed-by: Nicholas Piggin <[email protected]>
...
>
> Signed-off-by: Christophe Leroy <[email protected]>
> ---
> v5: Also in interrupt exit prepare
> ---
> arch/powerpc/kernel/interrupt.c | 7 ++++---
> 1 file changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
> index 1a2dec49f811..107ec39f05cb 100644
> --- a/arch/powerpc/kernel/interrupt.c
> +++ b/arch/powerpc/kernel/interrupt.c
> @@ -39,7 +39,7 @@ notrace long system_call_exception(long r3, long r4, long r5,
>
> trace_hardirqs_off(); /* finish reconciling */
>
> - if (IS_ENABLED(CONFIG_PPC_BOOK3S))
> + if (!IS_ENABLED(CONFIG_BOOKE) && !IS_ENABLED(CONFIG_40x))
> BUG_ON(!(regs->msr & MSR_RI));
> BUG_ON(!(regs->msr & MSR_PR));
> BUG_ON(!FULL_REGS(regs));
> @@ -338,7 +338,7 @@ notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs, unsigned
> unsigned long flags;
> unsigned long ret = 0;
>
> - if (IS_ENABLED(CONFIG_PPC_BOOK3S))
> + if (!IS_ENABLED(CONFIG_BOOKE) && !IS_ENABLED(CONFIG_40x))
> BUG_ON(!(regs->msr & MSR_RI));
> BUG_ON(!(regs->msr & MSR_PR));
> BUG_ON(!FULL_REGS(regs));
> @@ -436,7 +436,8 @@ notrace unsigned long interrupt_exit_kernel_prepare(struct pt_regs *regs, unsign
> unsigned long amr;
> #endif
>
> - if (IS_ENABLED(CONFIG_PPC_BOOK3S) && unlikely(!(regs->msr & MSR_RI)))
> + if (!IS_ENABLED(CONFIG_BOOKE) && !IS_ENABLED(CONFIG_40x) &&
> + unlikely(!(regs->msr & MSR_RI)))
> unrecoverable_exception(regs);
> BUG_ON(regs->msr & MSR_PR);
> BUG_ON(!FULL_REGS(regs));
> --
> 2.25.0
>
>
Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
> When r3 is not modified, reload it from regs->orig_r3 to free
> volatile registers. This avoids a stack frame for the likely part
> of system_call_exception()
This doesn't on my 64s build, but it does reduce one non volatile
register save/restore. With quite a bit more register pressure
reduction 64s can avoid the stack frame as well.
It's a cool trick but quite code and compiler specific so I don't know
how worthwhile it is to keep considering we're calling out into random
kernel C code after this.
Maybe just keep it PPC32 specific for the moment, will have to do more
tuning for 64 and we have other stuff to do there first.
If you are happy to make it 32-bit only then
Reviewed-by: Nicholas Piggin <[email protected]>
>
> Before the patch:
>
> c000b4d4 <system_call_exception>:
> c000b4d4: 7c 08 02 a6 mflr r0
> c000b4d8: 94 21 ff e0 stwu r1,-32(r1)
> c000b4dc: 93 e1 00 1c stw r31,28(r1)
> c000b4e0: 90 01 00 24 stw r0,36(r1)
> c000b4e4: 90 6a 00 88 stw r3,136(r10)
> c000b4e8: 81 6a 00 84 lwz r11,132(r10)
> c000b4ec: 69 6b 00 02 xori r11,r11,2
> c000b4f0: 55 6b ff fe rlwinm r11,r11,31,31,31
> c000b4f4: 0f 0b 00 00 twnei r11,0
> c000b4f8: 81 6a 00 a0 lwz r11,160(r10)
> c000b4fc: 55 6b 07 fe clrlwi r11,r11,31
> c000b500: 0f 0b 00 00 twnei r11,0
> c000b504: 7c 0c 42 e6 mftb r0
> c000b508: 83 e2 00 08 lwz r31,8(r2)
> c000b50c: 81 82 00 28 lwz r12,40(r2)
> c000b510: 90 02 00 24 stw r0,36(r2)
> c000b514: 7d 8c f8 50 subf r12,r12,r31
> c000b518: 7c 0c 02 14 add r0,r12,r0
> c000b51c: 90 02 00 08 stw r0,8(r2)
> c000b520: 7c 10 13 a6 mtspr 80,r0
> c000b524: 81 62 00 70 lwz r11,112(r2)
> c000b528: 71 60 86 91 andi. r0,r11,34449
> c000b52c: 40 82 00 34 bne c000b560 <system_call_exception+0x8c>
> c000b530: 2b 89 01 b6 cmplwi cr7,r9,438
> c000b534: 41 9d 00 64 bgt cr7,c000b598 <system_call_exception+0xc4>
> c000b538: 3d 40 c0 5c lis r10,-16292
> c000b53c: 55 29 10 3a rlwinm r9,r9,2,0,29
> c000b540: 39 4a 41 e8 addi r10,r10,16872
> c000b544: 80 01 00 24 lwz r0,36(r1)
> c000b548: 7d 2a 48 2e lwzx r9,r10,r9
> c000b54c: 7c 08 03 a6 mtlr r0
> c000b550: 7d 29 03 a6 mtctr r9
> c000b554: 83 e1 00 1c lwz r31,28(r1)
> c000b558: 38 21 00 20 addi r1,r1,32
> c000b55c: 4e 80 04 20 bctr
>
> After the patch:
>
> c000b4d4 <system_call_exception>:
> c000b4d4: 81 6a 00 84 lwz r11,132(r10)
> c000b4d8: 90 6a 00 88 stw r3,136(r10)
> c000b4dc: 69 6b 00 02 xori r11,r11,2
> c000b4e0: 55 6b ff fe rlwinm r11,r11,31,31,31
> c000b4e4: 0f 0b 00 00 twnei r11,0
> c000b4e8: 80 6a 00 a0 lwz r3,160(r10)
> c000b4ec: 54 63 07 fe clrlwi r3,r3,31
> c000b4f0: 0f 03 00 00 twnei r3,0
> c000b4f4: 7d 6c 42 e6 mftb r11
> c000b4f8: 81 82 00 08 lwz r12,8(r2)
> c000b4fc: 80 02 00 28 lwz r0,40(r2)
> c000b500: 91 62 00 24 stw r11,36(r2)
> c000b504: 7c 00 60 50 subf r0,r0,r12
> c000b508: 7d 60 5a 14 add r11,r0,r11
> c000b50c: 91 62 00 08 stw r11,8(r2)
> c000b510: 7c 10 13 a6 mtspr 80,r0
> c000b514: 80 62 00 70 lwz r3,112(r2)
> c000b518: 70 6b 86 91 andi. r11,r3,34449
> c000b51c: 40 82 00 28 bne c000b544 <system_call_exception+0x70>
> c000b520: 2b 89 01 b6 cmplwi cr7,r9,438
> c000b524: 41 9d 00 84 bgt cr7,c000b5a8 <system_call_exception+0xd4>
> c000b528: 80 6a 00 88 lwz r3,136(r10)
> c000b52c: 3d 40 c0 5c lis r10,-16292
> c000b530: 55 29 10 3a rlwinm r9,r9,2,0,29
> c000b534: 39 4a 41 e4 addi r10,r10,16868
> c000b538: 7d 2a 48 2e lwzx r9,r10,r9
> c000b53c: 7d 29 03 a6 mtctr r9
> c000b540: 4e 80 04 20 bctr
>
> Signed-off-by: Christophe Leroy <[email protected]>
> ---
> arch/powerpc/kernel/interrupt.c | 3 +++
> 1 file changed, 3 insertions(+)
>
> diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
> index 107ec39f05cb..205902052112 100644
> --- a/arch/powerpc/kernel/interrupt.c
> +++ b/arch/powerpc/kernel/interrupt.c
> @@ -117,6 +117,9 @@ notrace long system_call_exception(long r3, long r4, long r5,
> return regs->gpr[3];
> }
> return -ENOSYS;
> + } else {
> + /* Restore r3 from orig_gpr3 to free up a volatile reg */
> + r3 = regs->orig_gpr3;
> }
>
> /* May be faster to do array_index_nospec? */
> --
> 2.25.0
>
>
Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
> Only PPC64 has scv. No need to check the 0x7ff0 trap on PPC32.
> For that, add a helper trap_is_unsupported_scv() similar to
> trap_is_scv().
>
> And ignore the scv parameter in syscall_exit_prepare (Save 14 cycles
> 346 => 332 cycles)
>
> Signed-off-by: Christophe Leroy <[email protected]>
> ---
> v5: Added a helper trap_is_unsupported_scv()
> ---
> arch/powerpc/include/asm/ptrace.h | 5 +++++
> arch/powerpc/kernel/entry_32.S | 1 -
> arch/powerpc/kernel/interrupt.c | 7 +++++--
> 3 files changed, 10 insertions(+), 3 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/ptrace.h b/arch/powerpc/include/asm/ptrace.h
> index 58f9dc060a7b..2c842b11a924 100644
> --- a/arch/powerpc/include/asm/ptrace.h
> +++ b/arch/powerpc/include/asm/ptrace.h
> @@ -229,6 +229,11 @@ static inline bool trap_is_scv(struct pt_regs *regs)
> return (IS_ENABLED(CONFIG_PPC_BOOK3S_64) && TRAP(regs) == 0x3000);
> }
>
> +static inline bool trap_is_unsupported_scv(struct pt_regs *regs)
> +{
> + return (IS_ENABLED(CONFIG_PPC_BOOK3S_64) && TRAP(regs) == 0x7ff0);
> +}
This change is good.
> +
> static inline bool trap_is_syscall(struct pt_regs *regs)
> {
> return (trap_is_scv(regs) || TRAP(regs) == 0xc00);
> diff --git a/arch/powerpc/kernel/entry_32.S b/arch/powerpc/kernel/entry_32.S
> index cffe58e63356..7c824e8928d0 100644
> --- a/arch/powerpc/kernel/entry_32.S
> +++ b/arch/powerpc/kernel/entry_32.S
> @@ -344,7 +344,6 @@ transfer_to_syscall:
>
> ret_from_syscall:
> addi r4,r1,STACK_FRAME_OVERHEAD
> - li r5,0
> bl syscall_exit_prepare
For this one, I think it would be nice to do the "right" thing and make
the function prototypes different on !64S. They could then declare a
local const bool scv = 0.
We could have syscall_exit_prepare and syscall_exit_prepare_maybe_scv
or something like that, 64s can use the latter one and the former can be
a wrapper that passes constant 0 for scv. Then we don't have different
prototypes for the same function, but you just have to make the 32-bit
version static inline and the 64-bit version exported to asm.
Thanks,
Nick
> #if defined(CONFIG_4xx) || defined(CONFIG_BOOKE)
> /* If the process has its own DBCR0 value, load it up. The internal
> diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
> index 205902052112..8fafca727b8b 100644
> --- a/arch/powerpc/kernel/interrupt.c
> +++ b/arch/powerpc/kernel/interrupt.c
> @@ -88,7 +88,7 @@ notrace long system_call_exception(long r3, long r4, long r5,
> local_irq_enable();
>
> if (unlikely(current_thread_info()->flags & _TIF_SYSCALL_DOTRACE)) {
> - if (unlikely(regs->trap == 0x7ff0)) {
> + if (unlikely(trap_is_unsupported_scv(regs))) {
> /* Unsupported scv vector */
> _exception(SIGILL, regs, ILL_ILLOPC, regs->nip);
> return regs->gpr[3];
> @@ -111,7 +111,7 @@ notrace long system_call_exception(long r3, long r4, long r5,
> r8 = regs->gpr[8];
>
> } else if (unlikely(r0 >= NR_syscalls)) {
> - if (unlikely(regs->trap == 0x7ff0)) {
> + if (unlikely(trap_is_unsupported_scv(regs))) {
> /* Unsupported scv vector */
> _exception(SIGILL, regs, ILL_ILLOPC, regs->nip);
> return regs->gpr[3];
> @@ -224,6 +224,9 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,
> unsigned long ti_flags;
> unsigned long ret = 0;
>
> + if (IS_ENABLED(CONFIG_PPC32))
> + scv = 0;
> +
> CT_WARN_ON(ct_state() == CONTEXT_USER);
>
> #ifdef CONFIG_PPC64
> --
> 2.25.0
>
>
Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
> For book3s/64, FULL_REGS() is 'true' at all time, so the test voids.
> For others, non volatile registers are saved inconditionally.
>
> So the verification is pointless.
>
> Should one fail to do it, it would anyway be caught by the
> CHECK_FULL_REGS() in copy_thread() as we have removed the
> special versions ppc_fork() and friends.
>
> null_syscall benchmark reduction 4 cycles (332 => 328 cycles)
I wonder if we rather make a CONFIG option for a bunch of these simpler
debug checks here (and also in interrupt exit, wrappers, etc) rather
than remove them entirely.
Thanks,
Nick
>
> Signed-off-by: Christophe Leroy <[email protected]>
> ---
> arch/powerpc/kernel/interrupt.c | 1 -
> 1 file changed, 1 deletion(-)
>
> diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
> index 8fafca727b8b..55e1aa18cdb9 100644
> --- a/arch/powerpc/kernel/interrupt.c
> +++ b/arch/powerpc/kernel/interrupt.c
> @@ -42,7 +42,6 @@ notrace long system_call_exception(long r3, long r4, long r5,
> if (!IS_ENABLED(CONFIG_BOOKE) && !IS_ENABLED(CONFIG_40x))
> BUG_ON(!(regs->msr & MSR_RI));
> BUG_ON(!(regs->msr & MSR_PR));
> - BUG_ON(!FULL_REGS(regs));
> BUG_ON(arch_irq_disabled_regs(regs));
>
> #ifdef CONFIG_PPC_PKEY
> --
> 2.25.0
>
>
Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
> Combine all tests of regs->msr into a single logical one.
Okay by me unless we choose to do the config option and put these all
under it. I think I would prefer that because sometimes the registers
are in a state you can't easily see what the values in the expression
were. In this case it doesn't matter so much because they should be in
regs in the interrupt frame.
Thanks,
Nick
>
> Before the patch:
>
> 0: 81 6a 00 84 lwz r11,132(r10)
> 4: 90 6a 00 88 stw r3,136(r10)
> 8: 69 60 00 02 xori r0,r11,2
> c: 54 00 ff fe rlwinm r0,r0,31,31,31
> 10: 0f 00 00 00 twnei r0,0
> 14: 69 63 40 00 xori r3,r11,16384
> 18: 54 63 97 fe rlwinm r3,r3,18,31,31
> 1c: 0f 03 00 00 twnei r3,0
> 20: 69 6b 80 00 xori r11,r11,32768
> 24: 55 6b 8f fe rlwinm r11,r11,17,31,31
> 28: 0f 0b 00 00 twnei r11,0
>
> After the patch:
>
> 0: 81 6a 00 84 lwz r11,132(r10)
> 4: 90 6a 00 88 stw r3,136(r10)
> 8: 7d 6b 58 f8 not r11,r11
> c: 71 6b c0 02 andi. r11,r11,49154
> 10: 0f 0b 00 00 twnei r11,0
>
> 6 cycles less on powerpc 8xx (328 => 322 cycles).
>
> Signed-off-by: Christophe Leroy <[email protected]>
> ---
> arch/powerpc/kernel/interrupt.c | 10 +++++++---
> 1 file changed, 7 insertions(+), 3 deletions(-)
>
> diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
> index 55e1aa18cdb9..8c38e8c95be2 100644
> --- a/arch/powerpc/kernel/interrupt.c
> +++ b/arch/powerpc/kernel/interrupt.c
> @@ -28,6 +28,7 @@ notrace long system_call_exception(long r3, long r4, long r5,
> unsigned long r0, struct pt_regs *regs)
> {
> syscall_fn f;
> + unsigned long expected_msr;
>
> regs->orig_gpr3 = r3;
>
> @@ -39,10 +40,13 @@ notrace long system_call_exception(long r3, long r4, long r5,
>
> trace_hardirqs_off(); /* finish reconciling */
>
> + expected_msr = MSR_PR;
> if (!IS_ENABLED(CONFIG_BOOKE) && !IS_ENABLED(CONFIG_40x))
> - BUG_ON(!(regs->msr & MSR_RI));
> - BUG_ON(!(regs->msr & MSR_PR));
> - BUG_ON(arch_irq_disabled_regs(regs));
> + expected_msr |= MSR_RI;
> + if (IS_ENABLED(CONFIG_PPC32))
> + expected_msr |= MSR_EE;
> + BUG_ON((regs->msr & expected_msr) ^ expected_msr);
> + BUG_ON(IS_ENABLED(CONFIG_PPC64) && arch_irq_disabled_regs(regs));
>
> #ifdef CONFIG_PPC_PKEY
> if (mmu_has_feature(MMU_FTR_PKEY)) {
> --
> 2.25.0
>
>
Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
> By saving the pointer pointing to thread_info.flags, gcc copies r2
> in a non-volatile register.
>
> We know 'current' doesn't change, so avoid that intermediaite pointer.
>
> Reduces null_syscall benchmark by 2 cycles (322 => 320 cycles)
>
> On PPC64, gcc seems to know that 'current' is not changing, and it keeps
> it in a non volatile register to avoid multiple read of 'current' in paca.
>
> Signed-off-by: Christophe Leroy <[email protected]>
What if you did this?
---
arch/powerpc/include/asm/current.h | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)
diff --git a/arch/powerpc/include/asm/current.h b/arch/powerpc/include/asm/current.h
index bbfb94800415..59ab327972a5 100644
--- a/arch/powerpc/include/asm/current.h
+++ b/arch/powerpc/include/asm/current.h
@@ -23,16 +23,19 @@ static inline struct task_struct *get_current(void)
return task;
}
-#define current get_current()
#else
-/*
- * We keep `current' in r2 for speed.
- */
-register struct task_struct *current asm ("r2");
+static inline struct task_struct *get_current(void)
+{
+ register struct task_struct *task asm ("r2");
+
+ return task;
+}
#endif
+#define current get_current()
+
#endif /* __KERNEL__ */
#endif /* _ASM_POWERPC_CURRENT_H */
--
Le 09/02/2021 à 02:11, Nicholas Piggin a écrit :
> Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
>> regs->softe doesn't exist on PPC32.
>>
>> Add irq_soft_mask_regs_set_state() helper to set regs->softe.
>> This helper will void on PPC32.
>>
>> Signed-off-by: Christophe Leroy <[email protected]>
>> ---
>> arch/powerpc/include/asm/hw_irq.h | 11 +++++++++--
>> 1 file changed, 9 insertions(+), 2 deletions(-)
>>
>> diff --git a/arch/powerpc/include/asm/hw_irq.h b/arch/powerpc/include/asm/hw_irq.h
>> index 614957f74cee..ed0c3b049dfd 100644
>> --- a/arch/powerpc/include/asm/hw_irq.h
>> +++ b/arch/powerpc/include/asm/hw_irq.h
>> @@ -38,6 +38,8 @@
>> #define PACA_IRQ_MUST_HARD_MASK (PACA_IRQ_EE)
>> #endif
>>
>> +#endif /* CONFIG_PPC64 */
>> +
>> /*
>> * flags for paca->irq_soft_mask
>> */
>> @@ -46,8 +48,6 @@
>> #define IRQS_PMI_DISABLED 2
>> #define IRQS_ALL_DISABLED (IRQS_DISABLED | IRQS_PMI_DISABLED)
>>
>> -#endif /* CONFIG_PPC64 */
>> -
>> #ifndef __ASSEMBLY__
>>
>> #ifdef CONFIG_PPC64
>> @@ -287,6 +287,10 @@ extern void irq_set_pending_from_srr1(unsigned long srr1);
>>
>> extern void force_external_irq_replay(void);
>>
>> +static inline void irq_soft_mask_regs_set_state(struct pt_regs *regs, unsigned long val)
>> +{
>> + regs->softe = val;
>> +}
>> #else /* CONFIG_PPC64 */
>>
>> static inline unsigned long arch_local_save_flags(void)
>> @@ -355,6 +359,9 @@ static inline bool arch_irq_disabled_regs(struct pt_regs *regs)
>>
>> static inline void may_hard_irq_enable(void) { }
>>
>> +static inline void irq_soft_mask_regs_set_state(struct pt_regs *regs, unsigned long val)
>> +{
>> +}
>> #endif /* CONFIG_PPC64 */
>>
>> #define ARCH_IRQ_INIT_FLAGS IRQ_NOREQUEST
>
> What I don't like about this where you use it is it kind of pollutes
> the ppc32 path with this function which is not valid to use.
>
> I would prefer if you had this purely so it could compile with:
>
> if (IS_ENABLED(CONFIG_PPC64)))
> irq_soft_mask_regs_set_state(regs, blah);
>
> And then you could make the ppc32 cause a link error if it did not
> get eliminated at compile time (e.g., call an undefined function).
>
> You could do the same with the kuap_ functions to change some ifdefs
> to IS_ENABLED.
>
> That's just my preference but if you prefer this way I guess that's
> okay.
I see you didn't change your mind since last April :)
I'll see what I can do.
Christophe
Le 09/02/2021 à 02:27, Nicholas Piggin a écrit :
> Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
>> To allow building interrupt.c on PPC32, ifdef out specific PPC64
>> code or use helpers which are available on both PP32 and PPC64
>>
>> Modify Makefile to always build interrupt.o
>>
>> Signed-off-by: Christophe Leroy <[email protected]>
>> ---
>> v5:
>> - Also for interrupt exit preparation
>> - Opted out kuap related code, ppc32 keeps it in ASM for the time being
>> ---
>> arch/powerpc/kernel/Makefile | 4 ++--
>> arch/powerpc/kernel/interrupt.c | 31 ++++++++++++++++++++++++-------
>> 2 files changed, 26 insertions(+), 9 deletions(-)
>>
>> diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
>> index d6be4f9a67e5..2dac4d2bb1cf 100644
>> --- a/arch/powerpc/kernel/interrupt.c
>> +++ b/arch/powerpc/kernel/interrupt.c
>> @@ -39,7 +39,7 @@ notrace long system_call_exception(long r3, long r4, long r5,
>> BUG_ON(!(regs->msr & MSR_RI));
>> BUG_ON(!(regs->msr & MSR_PR));
>> BUG_ON(!FULL_REGS(regs));
>> - BUG_ON(regs->softe != IRQS_ENABLED);
>> + BUG_ON(arch_irq_disabled_regs(regs));
>>
>> #ifdef CONFIG_PPC_PKEY
>> if (mmu_has_feature(MMU_FTR_PKEY)) {
>> @@ -65,7 +65,9 @@ notrace long system_call_exception(long r3, long r4, long r5,
>> isync();
>> } else
>> #endif
>> +#ifdef CONFIG_PPC64
>> kuap_check_amr();
>> +#endif
>
> Wouldn't mind trying to get rid of these ifdefs at some point, but
> there's some kuap / keys changes going on recently so I'm happy enough
> to let this settle then look at whether we can refactor.
I have a follow up series that implements interrupts entries/exits in C and that removes all kuap
assembly, I will likely release it as RFC later today.
>
>>
>> account_cpu_user_entry();
>>
>> @@ -318,7 +323,7 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,
>> return ret;
>> }
>>
>> -#ifdef CONFIG_PPC_BOOK3S /* BOOK3E not yet using this */
>> +#ifndef CONFIG_PPC_BOOK3E_64 /* BOOK3E not yet using this */
>> notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs, unsigned long msr)
>> {
>> #ifdef CONFIG_PPC_BOOK3E
>
> Why are you building this for 32? I don't mind if it's just to keep
> things similar and make it build for now, but you're not using it yet,
> right?
The series using that will follow, I thought it would be worth doing this at once.
>
> Reviewed-by: Nicholas Piggin <[email protected]>
>
Le 09/02/2021 à 03:00, Nicholas Piggin a écrit :
> Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
>> Only PPC64 has scv. No need to check the 0x7ff0 trap on PPC32.
>> For that, add a helper trap_is_unsupported_scv() similar to
>> trap_is_scv().
>>
>> And ignore the scv parameter in syscall_exit_prepare (Save 14 cycles
>> 346 => 332 cycles)
>>
>> Signed-off-by: Christophe Leroy <[email protected]>
>> ---
>> v5: Added a helper trap_is_unsupported_scv()
>> ---
>> arch/powerpc/include/asm/ptrace.h | 5 +++++
>> arch/powerpc/kernel/entry_32.S | 1 -
>> arch/powerpc/kernel/interrupt.c | 7 +++++--
>> 3 files changed, 10 insertions(+), 3 deletions(-)
>>
>> diff --git a/arch/powerpc/include/asm/ptrace.h b/arch/powerpc/include/asm/ptrace.h
>> index 58f9dc060a7b..2c842b11a924 100644
>> --- a/arch/powerpc/include/asm/ptrace.h
>> +++ b/arch/powerpc/include/asm/ptrace.h
>> @@ -229,6 +229,11 @@ static inline bool trap_is_scv(struct pt_regs *regs)
>> return (IS_ENABLED(CONFIG_PPC_BOOK3S_64) && TRAP(regs) == 0x3000);
>> }
>>
>> +static inline bool trap_is_unsupported_scv(struct pt_regs *regs)
>> +{
>> + return (IS_ENABLED(CONFIG_PPC_BOOK3S_64) && TRAP(regs) == 0x7ff0);
>> +}
>
> This change is good.
>
>> +
>> static inline bool trap_is_syscall(struct pt_regs *regs)
>> {
>> return (trap_is_scv(regs) || TRAP(regs) == 0xc00);
>> diff --git a/arch/powerpc/kernel/entry_32.S b/arch/powerpc/kernel/entry_32.S
>> index cffe58e63356..7c824e8928d0 100644
>> --- a/arch/powerpc/kernel/entry_32.S
>> +++ b/arch/powerpc/kernel/entry_32.S
>> @@ -344,7 +344,6 @@ transfer_to_syscall:
>>
>> ret_from_syscall:
>> addi r4,r1,STACK_FRAME_OVERHEAD
>> - li r5,0
>> bl syscall_exit_prepare
>
> For this one, I think it would be nice to do the "right" thing and make
> the function prototypes different on !64S. They could then declare a
> local const bool scv = 0.
>
> We could have syscall_exit_prepare and syscall_exit_prepare_maybe_scv
> or something like that, 64s can use the latter one and the former can be
> a wrapper that passes constant 0 for scv. Then we don't have different
> prototypes for the same function, but you just have to make the 32-bit
> version static inline and the 64-bit version exported to asm.
You can't call a static inline function from ASM, I don't understand you.
What is wrong for you really here ? Is that the fact we leave scv random, or is that the below
IS_ENABLED() ?
I don't mind keeping the 'li r5,0' before calling the function if you find it cleaner, the real
performance gain is with setting scv to 0 below for PPC32 (and maybe it should be set to zero for
book3e/64 too ?).
Other solution would be to do replace (!scv) by (!scv || !IS_ENABLED(CONFIG_PPC_BOOK3S_64)) in the
two places it is used in syscall_exit_prepare().
Any preference ?
Thanks
Christophe
>> @@ -224,6 +224,9 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,
>> unsigned long ti_flags;
>> unsigned long ret = 0;
>>
>> + if (IS_ENABLED(CONFIG_PPC32))
>> + scv = 0;
>> +
>> CT_WARN_ON(ct_state() == CONTEXT_USER);
>>
>> #ifdef CONFIG_PPC64
>> --
>> 2.25.0
>>
>>
Le 09/02/2021 à 02:11, Nicholas Piggin a écrit :
> Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
>> regs->softe doesn't exist on PPC32.
>>
>> Add irq_soft_mask_regs_set_state() helper to set regs->softe.
>> This helper will void on PPC32.
>>
>> Signed-off-by: Christophe Leroy <[email protected]>
>> ---
>
> You could do the same with the kuap_ functions to change some ifdefs
> to IS_ENABLED.
>
> That's just my preference but if you prefer this way I guess that's
> okay.
>
That's also my preference on the long term.
Here it is ephemeral, I have a follow up series implementing interrupt exit/entry in C and getting
rid of all the assembly kuap hence getting rid of those ifdefs.
The issue I see when using IS_ENABLED() is that you have to indent to the right, then you interfere
with the file history and 'git blame'
Thanks for reviewing my series and looking forward to your feedback on my series on the interrupt
entry/exit that I will likely release later today.
Christophe
Excerpts from Christophe Leroy's message of February 9, 2021 3:57 pm:
>
>
> Le 09/02/2021 à 02:11, Nicholas Piggin a écrit :
>> Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
>>> regs->softe doesn't exist on PPC32.
>>>
>>> Add irq_soft_mask_regs_set_state() helper to set regs->softe.
>>> This helper will void on PPC32.
>>>
>>> Signed-off-by: Christophe Leroy <[email protected]>
>>> ---
>>> arch/powerpc/include/asm/hw_irq.h | 11 +++++++++--
>>> 1 file changed, 9 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/arch/powerpc/include/asm/hw_irq.h b/arch/powerpc/include/asm/hw_irq.h
>>> index 614957f74cee..ed0c3b049dfd 100644
>>> --- a/arch/powerpc/include/asm/hw_irq.h
>>> +++ b/arch/powerpc/include/asm/hw_irq.h
>>> @@ -38,6 +38,8 @@
>>> #define PACA_IRQ_MUST_HARD_MASK (PACA_IRQ_EE)
>>> #endif
>>>
>>> +#endif /* CONFIG_PPC64 */
>>> +
>>> /*
>>> * flags for paca->irq_soft_mask
>>> */
>>> @@ -46,8 +48,6 @@
>>> #define IRQS_PMI_DISABLED 2
>>> #define IRQS_ALL_DISABLED (IRQS_DISABLED | IRQS_PMI_DISABLED)
>>>
>>> -#endif /* CONFIG_PPC64 */
>>> -
>>> #ifndef __ASSEMBLY__
>>>
>>> #ifdef CONFIG_PPC64
>>> @@ -287,6 +287,10 @@ extern void irq_set_pending_from_srr1(unsigned long srr1);
>>>
>>> extern void force_external_irq_replay(void);
>>>
>>> +static inline void irq_soft_mask_regs_set_state(struct pt_regs *regs, unsigned long val)
>>> +{
>>> + regs->softe = val;
>>> +}
>>> #else /* CONFIG_PPC64 */
>>>
>>> static inline unsigned long arch_local_save_flags(void)
>>> @@ -355,6 +359,9 @@ static inline bool arch_irq_disabled_regs(struct pt_regs *regs)
>>>
>>> static inline void may_hard_irq_enable(void) { }
>>>
>>> +static inline void irq_soft_mask_regs_set_state(struct pt_regs *regs, unsigned long val)
>>> +{
>>> +}
>>> #endif /* CONFIG_PPC64 */
>>>
>>> #define ARCH_IRQ_INIT_FLAGS IRQ_NOREQUEST
>>
>> What I don't like about this where you use it is it kind of pollutes
>> the ppc32 path with this function which is not valid to use.
>>
>> I would prefer if you had this purely so it could compile with:
>>
>> if (IS_ENABLED(CONFIG_PPC64)))
>> irq_soft_mask_regs_set_state(regs, blah);
>>
>> And then you could make the ppc32 cause a link error if it did not
>> get eliminated at compile time (e.g., call an undefined function).
>>
>> You could do the same with the kuap_ functions to change some ifdefs
>> to IS_ENABLED.
>>
>> That's just my preference but if you prefer this way I guess that's
>> okay.
>
> I see you didn't change your mind since last April :)
>
> I'll see what I can do.
If you have more patches in the works and will do some cleanup passes I
don't mind so much.
Thanks,
Nick
Excerpts from Christophe Leroy's message of February 9, 2021 4:18 pm:
>
>
> Le 09/02/2021 à 02:11, Nicholas Piggin a écrit :
>> Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
>>> regs->softe doesn't exist on PPC32.
>>>
>>> Add irq_soft_mask_regs_set_state() helper to set regs->softe.
>>> This helper will void on PPC32.
>>>
>>> Signed-off-by: Christophe Leroy <[email protected]>
>>> ---
>>
>> You could do the same with the kuap_ functions to change some ifdefs
>> to IS_ENABLED.
>>
>> That's just my preference but if you prefer this way I guess that's
>> okay.
>>
>
>
> That's also my preference on the long term.
>
> Here it is ephemeral, I have a follow up series implementing interrupt exit/entry in C and getting
> rid of all the assembly kuap hence getting rid of those ifdefs.
I thought it might have been because you hate ifdef more tha most :)
> The issue I see when using IS_ENABLED() is that you have to indent to the right, then you interfere
> with the file history and 'git blame'
Valid point if it's just going to indent back the other way in your next
series.
> Thanks for reviewing my series and looking forward to your feedback on my series on the interrupt
> entry/exit that I will likely release later today.
Cool, I'm eager to see them.
Thanks,
Nick
Excerpts from Christophe Leroy's message of February 9, 2021 4:02 pm:
>
>
> Le 09/02/2021 à 02:27, Nicholas Piggin a écrit :
>> Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
>>> To allow building interrupt.c on PPC32, ifdef out specific PPC64
>>> code or use helpers which are available on both PP32 and PPC64
>>>
>>> Modify Makefile to always build interrupt.o
>>>
>>> Signed-off-by: Christophe Leroy <[email protected]>
>>> ---
>>> v5:
>>> - Also for interrupt exit preparation
>>> - Opted out kuap related code, ppc32 keeps it in ASM for the time being
>>> ---
>>> arch/powerpc/kernel/Makefile | 4 ++--
>>> arch/powerpc/kernel/interrupt.c | 31 ++++++++++++++++++++++++-------
>>> 2 files changed, 26 insertions(+), 9 deletions(-)
>>>
>
>>> diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
>>> index d6be4f9a67e5..2dac4d2bb1cf 100644
>>> --- a/arch/powerpc/kernel/interrupt.c
>>> +++ b/arch/powerpc/kernel/interrupt.c
>>> @@ -39,7 +39,7 @@ notrace long system_call_exception(long r3, long r4, long r5,
>>> BUG_ON(!(regs->msr & MSR_RI));
>>> BUG_ON(!(regs->msr & MSR_PR));
>>> BUG_ON(!FULL_REGS(regs));
>>> - BUG_ON(regs->softe != IRQS_ENABLED);
>>> + BUG_ON(arch_irq_disabled_regs(regs));
>>>
>>> #ifdef CONFIG_PPC_PKEY
>>> if (mmu_has_feature(MMU_FTR_PKEY)) {
>>> @@ -65,7 +65,9 @@ notrace long system_call_exception(long r3, long r4, long r5,
>>> isync();
>>> } else
>>> #endif
>>> +#ifdef CONFIG_PPC64
>>> kuap_check_amr();
>>> +#endif
>>
>> Wouldn't mind trying to get rid of these ifdefs at some point, but
>> there's some kuap / keys changes going on recently so I'm happy enough
>> to let this settle then look at whether we can refactor.
>
> I have a follow up series that implements interrupts entries/exits in C and that removes all kuap
> assembly, I will likely release it as RFC later today.
>
>>
>>>
>>> account_cpu_user_entry();
>>>
>>> @@ -318,7 +323,7 @@ notrace unsigned long syscall_exit_prepare(unsigned long r3,
>>> return ret;
>>> }
>>>
>>> -#ifdef CONFIG_PPC_BOOK3S /* BOOK3E not yet using this */
>>> +#ifndef CONFIG_PPC_BOOK3E_64 /* BOOK3E not yet using this */
>>> notrace unsigned long interrupt_exit_user_prepare(struct pt_regs *regs, unsigned long msr)
>>> {
>>> #ifdef CONFIG_PPC_BOOK3E
>>
>> Why are you building this for 32? I don't mind if it's just to keep
>> things similar and make it build for now, but you're not using it yet,
>> right?
>
> The series using that will follow, I thought it would be worth doing this at once.
Yeah that's fine by me then.
Thanks,
Nick
Excerpts from Christophe Leroy's message of February 9, 2021 4:13 pm:
>
>
> Le 09/02/2021 à 03:00, Nicholas Piggin a écrit :
>> Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
>>> Only PPC64 has scv. No need to check the 0x7ff0 trap on PPC32.
>>> For that, add a helper trap_is_unsupported_scv() similar to
>>> trap_is_scv().
>>>
>>> And ignore the scv parameter in syscall_exit_prepare (Save 14 cycles
>>> 346 => 332 cycles)
>>>
>>> Signed-off-by: Christophe Leroy <[email protected]>
>>> ---
>>> v5: Added a helper trap_is_unsupported_scv()
>>> ---
>>> arch/powerpc/include/asm/ptrace.h | 5 +++++
>>> arch/powerpc/kernel/entry_32.S | 1 -
>>> arch/powerpc/kernel/interrupt.c | 7 +++++--
>>> 3 files changed, 10 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/arch/powerpc/include/asm/ptrace.h b/arch/powerpc/include/asm/ptrace.h
>>> index 58f9dc060a7b..2c842b11a924 100644
>>> --- a/arch/powerpc/include/asm/ptrace.h
>>> +++ b/arch/powerpc/include/asm/ptrace.h
>>> @@ -229,6 +229,11 @@ static inline bool trap_is_scv(struct pt_regs *regs)
>>> return (IS_ENABLED(CONFIG_PPC_BOOK3S_64) && TRAP(regs) == 0x3000);
>>> }
>>>
>>> +static inline bool trap_is_unsupported_scv(struct pt_regs *regs)
>>> +{
>>> + return (IS_ENABLED(CONFIG_PPC_BOOK3S_64) && TRAP(regs) == 0x7ff0);
>>> +}
>>
>> This change is good.
>>
>>> +
>>> static inline bool trap_is_syscall(struct pt_regs *regs)
>>> {
>>> return (trap_is_scv(regs) || TRAP(regs) == 0xc00);
>>> diff --git a/arch/powerpc/kernel/entry_32.S b/arch/powerpc/kernel/entry_32.S
>>> index cffe58e63356..7c824e8928d0 100644
>>> --- a/arch/powerpc/kernel/entry_32.S
>>> +++ b/arch/powerpc/kernel/entry_32.S
>>> @@ -344,7 +344,6 @@ transfer_to_syscall:
>>>
>>> ret_from_syscall:
>>> addi r4,r1,STACK_FRAME_OVERHEAD
>>> - li r5,0
>>> bl syscall_exit_prepare
>>
>> For this one, I think it would be nice to do the "right" thing and make
>> the function prototypes different on !64S. They could then declare a
>> local const bool scv = 0.
>>
>> We could have syscall_exit_prepare and syscall_exit_prepare_maybe_scv
>> or something like that, 64s can use the latter one and the former can be
>> a wrapper that passes constant 0 for scv. Then we don't have different
>> prototypes for the same function, but you just have to make the 32-bit
>> version static inline and the 64-bit version exported to asm.
>
> You can't call a static inline function from ASM, I don't understand you.
I mean
#ifdef CONFIG_PPC_BOOK3S_64
notrace unsigned long syscall_exit_prepare_scv(unsigned long r3,
struct pt_regs *regs,
long scv)
#else
static inline long syscall_exit_prepare_scv(unsigned long r3,
struct pt_regs *regs,
long scv)
#endif
#ifndef CONFIG_PPC_BOOK3S_64
notrace unsigned long syscall_exit_prepare(unsigned long r3,
struct pt_regs *regs)
{
return syscall_exit_prepare_scv(r3, regs, 0);
}
#endif
>
> What is wrong for you really here ? Is that the fact we leave scv random, or is that the below
> IS_ENABLED() ?
That scv arg is random. I know generated code essentially would be no
different and no possibility of tracing, but would just prefer to call
the C "correctly" if possible.
> I don't mind keeping the 'li r5,0' before calling the function if you find it cleaner, the real
> performance gain is with setting scv to 0 below for PPC32 (and maybe it should be set to zero for
> book3e/64 too ?).
Yes 64e would like this optimisation.
Thanks,
Nick
On Tue, Feb 09, 2021 at 12:36:20PM +1000, Nicholas Piggin wrote:
> What if you did this?
> +static inline struct task_struct *get_current(void)
> +{
> + register struct task_struct *task asm ("r2");
> +
> + return task;
> +}
Local register asm variables are *only* guaranteed to live in that
register as operands to an asm. See
https://gcc.gnu.org/onlinedocs/gcc/Local-Register-Variables.html#Local-Register-Variables
("The only supported use" etc.)
You can do something like
static inline struct task_struct *get_current(void)
{
register struct task_struct *task asm ("r2");
asm("" : "+r"(task));
return task;
}
which makes sure that "task" actually is in r2 at the point of that asm.
Segher
Le 09/02/2021 à 03:02, Nicholas Piggin a écrit :
> Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
>> For book3s/64, FULL_REGS() is 'true' at all time, so the test voids.
>> For others, non volatile registers are saved inconditionally.
>>
>> So the verification is pointless.
>>
>> Should one fail to do it, it would anyway be caught by the
>> CHECK_FULL_REGS() in copy_thread() as we have removed the
>> special versions ppc_fork() and friends.
>>
>> null_syscall benchmark reduction 4 cycles (332 => 328 cycles)
>
> I wonder if we rather make a CONFIG option for a bunch of these simpler
> debug checks here (and also in interrupt exit, wrappers, etc) rather
> than remove them entirely.
We can drop this patch if you prefer. Anyway, like book3s/64, once ppc32 also do interrupt
entry/exit in C, FULL_REGS() will already return true.
Christophe
>
> Thanks,
> Nick
>
>>
>> Signed-off-by: Christophe Leroy <[email protected]>
>> ---
>> arch/powerpc/kernel/interrupt.c | 1 -
>> 1 file changed, 1 deletion(-)
>>
>> diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
>> index 8fafca727b8b..55e1aa18cdb9 100644
>> --- a/arch/powerpc/kernel/interrupt.c
>> +++ b/arch/powerpc/kernel/interrupt.c
>> @@ -42,7 +42,6 @@ notrace long system_call_exception(long r3, long r4, long r5,
>> if (!IS_ENABLED(CONFIG_BOOKE) && !IS_ENABLED(CONFIG_40x))
>> BUG_ON(!(regs->msr & MSR_RI));
>> BUG_ON(!(regs->msr & MSR_PR));
>> - BUG_ON(!FULL_REGS(regs));
>> BUG_ON(arch_irq_disabled_regs(regs));
>>
>> #ifdef CONFIG_PPC_PKEY
>> --
>> 2.25.0
>>
>>
From: Segher Boessenkool
> Sent: 09 February 2021 13:51
>
> On Tue, Feb 09, 2021 at 12:36:20PM +1000, Nicholas Piggin wrote:
> > What if you did this?
>
> > +static inline struct task_struct *get_current(void)
> > +{
> > + register struct task_struct *task asm ("r2");
> > +
> > + return task;
> > +}
>
> Local register asm variables are *only* guaranteed to live in that
> register as operands to an asm. See
> https://gcc.gnu.org/onlinedocs/gcc/Local-Register-Variables.html#Local-Register-Variables
> ("The only supported use" etc.)
>
> You can do something like
>
> static inline struct task_struct *get_current(void)
> {
> register struct task_struct *task asm ("r2");
>
> asm("" : "+r"(task));
>
> return task;
> }
>
> which makes sure that "task" actually is in r2 at the point of that asm.
If "r2" always contains current (and is never assigned by the compiler)
why not use a global register variable for it?
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
Le 09/02/2021 à 03:06, Nicholas Piggin a écrit :
> Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
>> Combine all tests of regs->msr into a single logical one.
>
> Okay by me unless we choose to do the config option and put these all
> under it. I think I would prefer that because sometimes the registers
> are in a state you can't easily see what the values in the expression
> were. In this case it doesn't matter so much because they should be in
> regs in the interrupt frame.
Yes indeed. I reword the commit log and tell that.
>
> Thanks,
> Nick
>
>>
>> Before the patch:
>>
>> 0: 81 6a 00 84 lwz r11,132(r10)
>> 4: 90 6a 00 88 stw r3,136(r10)
>> 8: 69 60 00 02 xori r0,r11,2
>> c: 54 00 ff fe rlwinm r0,r0,31,31,31
>> 10: 0f 00 00 00 twnei r0,0
>> 14: 69 63 40 00 xori r3,r11,16384
>> 18: 54 63 97 fe rlwinm r3,r3,18,31,31
>> 1c: 0f 03 00 00 twnei r3,0
>> 20: 69 6b 80 00 xori r11,r11,32768
>> 24: 55 6b 8f fe rlwinm r11,r11,17,31,31
>> 28: 0f 0b 00 00 twnei r11,0
>>
>> After the patch:
>>
>> 0: 81 6a 00 84 lwz r11,132(r10)
>> 4: 90 6a 00 88 stw r3,136(r10)
>> 8: 7d 6b 58 f8 not r11,r11
>> c: 71 6b c0 02 andi. r11,r11,49154
>> 10: 0f 0b 00 00 twnei r11,0
>>
>> 6 cycles less on powerpc 8xx (328 => 322 cycles).
>>
>> Signed-off-by: Christophe Leroy <[email protected]>
>> ---
>> arch/powerpc/kernel/interrupt.c | 10 +++++++---
>> 1 file changed, 7 insertions(+), 3 deletions(-)
>>
>> diff --git a/arch/powerpc/kernel/interrupt.c b/arch/powerpc/kernel/interrupt.c
>> index 55e1aa18cdb9..8c38e8c95be2 100644
>> --- a/arch/powerpc/kernel/interrupt.c
>> +++ b/arch/powerpc/kernel/interrupt.c
>> @@ -28,6 +28,7 @@ notrace long system_call_exception(long r3, long r4, long r5,
>> unsigned long r0, struct pt_regs *regs)
>> {
>> syscall_fn f;
>> + unsigned long expected_msr;
>>
>> regs->orig_gpr3 = r3;
>>
>> @@ -39,10 +40,13 @@ notrace long system_call_exception(long r3, long r4, long r5,
>>
>> trace_hardirqs_off(); /* finish reconciling */
>>
>> + expected_msr = MSR_PR;
>> if (!IS_ENABLED(CONFIG_BOOKE) && !IS_ENABLED(CONFIG_40x))
>> - BUG_ON(!(regs->msr & MSR_RI));
>> - BUG_ON(!(regs->msr & MSR_PR));
>> - BUG_ON(arch_irq_disabled_regs(regs));
>> + expected_msr |= MSR_RI;
>> + if (IS_ENABLED(CONFIG_PPC32))
>> + expected_msr |= MSR_EE;
>> + BUG_ON((regs->msr & expected_msr) ^ expected_msr);
>> + BUG_ON(IS_ENABLED(CONFIG_PPC64) && arch_irq_disabled_regs(regs));
>>
>> #ifdef CONFIG_PPC_PKEY
>> if (mmu_has_feature(MMU_FTR_PKEY)) {
>> --
>> 2.25.0
>>
>>
From: Christophe Leroy <[email protected]>
> Sent: 09 February 2021 17:04
>
> Le 09/02/2021 à 15:31, David Laight a écrit :
> > From: Segher Boessenkool
> >> Sent: 09 February 2021 13:51
> >>
> >> On Tue, Feb 09, 2021 at 12:36:20PM +1000, Nicholas Piggin wrote:
> >>> What if you did this?
> >>
> >>> +static inline struct task_struct *get_current(void)
> >>> +{
> >>> + register struct task_struct *task asm ("r2");
> >>> +
> >>> + return task;
> >>> +}
> >>
> >> Local register asm variables are *only* guaranteed to live in that
> >> register as operands to an asm. See
> >> https://gcc.gnu.org/onlinedocs/gcc/Local-Register-Variables.html#Local-Register-Variables
> >> ("The only supported use" etc.)
> >>
> >> You can do something like
> >>
> >> static inline struct task_struct *get_current(void)
> >> {
> >> register struct task_struct *task asm ("r2");
> >>
> >> asm("" : "+r"(task));
> >>
> >> return task;
> >> }
> >>
> >> which makes sure that "task" actually is in r2 at the point of that asm.
> >
> > If "r2" always contains current (and is never assigned by the compiler)
> > why not use a global register variable for it?
> >
>
>
> The change proposed by Nick doesn't solve the issue.
>
> The problem is that at the begining of the function we have:
>
> unsigned long *ti_flagsp = ¤t_thread_info()->flags;
>
> When the function uses ti_flagsp for the first time, it does use 112(r2)
>
> Then the function calls some other functions.
>
> Most likely because the function could update 'current', GCC copies r2 into r30, so that if r2 get
> changed by the called function, ti_flagsp is still based on the previous value of current.
>
> Allthough we know r2 wont change, GCC doesn't know it. And in order to save r2 into r30, it needs to
> save r30 in the stack.
>
>
> By using ¤t_thread_info()->flags directly instead of this intermediaite ti_flagsp pointer, GCC
> uses r2 instead instead of doing a copy.
Does marking current_thread_info() 'pure' (I think that the right one)
work - so that gcc knows its result doesn't depend on external data
and that it doesn't change external data.
Although I'm not 100% how well those attributes actually work.
> Nick, I don't understand the reason why you need that 'ti_flagsp' local var.
Probably to save typing.
I sometimes reload locals after function calls.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)
Excerpts from Christophe Leroy's message of February 10, 2021 2:13 am:
>
>
> Le 09/02/2021 à 02:55, Nicholas Piggin a écrit :
>> Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
>>> When r3 is not modified, reload it from regs->orig_r3 to free
>>> volatile registers. This avoids a stack frame for the likely part
>>> of system_call_exception()
>>
>> This doesn't on my 64s build, but it does reduce one non volatile
>> register save/restore. With quite a bit more register pressure
>> reduction 64s can avoid the stack frame as well.
>
> The stack frame is not due to the registers because on PPC64 you have the redzone that you don't
> have on PPC32.
>
> As far as I can see, this is due to a call to .arch_local_irq_restore().
>
> On ppc32 arch_local_irq_restore() is just a write to MSR.
Oh you're right there. We can actually inline fast paths of that I have
a patch somewhere, but not sure if it's worthwhile.
>> It's a cool trick but quite code and compiler specific so I don't know
>> how worthwhile it is to keep considering we're calling out into random
>> kernel C code after this.
>>
>> Maybe just keep it PPC32 specific for the moment, will have to do more
>> tuning for 64 and we have other stuff to do there first.
>>
>> If you are happy to make it 32-bit only then
>
> I think we can leave without this, that's only one or two cycles won.
Okay for this round let's drop it for now.
Thanks,
Nick
Excerpts from Christophe Leroy's message of February 10, 2021 12:31 am:
>
>
> Le 09/02/2021 à 03:02, Nicholas Piggin a écrit :
>> Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
>>> For book3s/64, FULL_REGS() is 'true' at all time, so the test voids.
>>> For others, non volatile registers are saved inconditionally.
>>>
>>> So the verification is pointless.
>>>
>>> Should one fail to do it, it would anyway be caught by the
>>> CHECK_FULL_REGS() in copy_thread() as we have removed the
>>> special versions ppc_fork() and friends.
>>>
>>> null_syscall benchmark reduction 4 cycles (332 => 328 cycles)
>>
>> I wonder if we rather make a CONFIG option for a bunch of these simpler
>> debug checks here (and also in interrupt exit, wrappers, etc) rather
>> than remove them entirely.
>
> We can drop this patch if you prefer. Anyway, like book3s/64, once ppc32 also do interrupt
> entry/exit in C, FULL_REGS() will already return true.
Sure let's do that.
Thanks,
Nick
Excerpts from Christophe Leroy's message of February 10, 2021 3:03 am:
>
>
> Le 09/02/2021 à 15:31, David Laight a écrit :
>> From: Segher Boessenkool
>>> Sent: 09 February 2021 13:51
>>>
>>> On Tue, Feb 09, 2021 at 12:36:20PM +1000, Nicholas Piggin wrote:
>>>> What if you did this?
>>>
>>>> +static inline struct task_struct *get_current(void)
>>>> +{
>>>> + register struct task_struct *task asm ("r2");
>>>> +
>>>> + return task;
>>>> +}
>>>
>>> Local register asm variables are *only* guaranteed to live in that
>>> register as operands to an asm. See
>>> https://gcc.gnu.org/onlinedocs/gcc/Local-Register-Variables.html#Local-Register-Variables
>>> ("The only supported use" etc.)
>>>
>>> You can do something like
>>>
>>> static inline struct task_struct *get_current(void)
>>> {
>>> register struct task_struct *task asm ("r2");
>>>
>>> asm("" : "+r"(task));
>>>
>>> return task;
>>> }
>>>
>>> which makes sure that "task" actually is in r2 at the point of that asm.
>>
>> If "r2" always contains current (and is never assigned by the compiler)
>> why not use a global register variable for it?
>>
>
>
> The change proposed by Nick doesn't solve the issue.
It seemed to change code generation in a simple test case, oh well.
>
> The problem is that at the begining of the function we have:
>
> unsigned long *ti_flagsp = ¤t_thread_info()->flags;
>
> When the function uses ti_flagsp for the first time, it does use 112(r2)
>
> Then the function calls some other functions.
>
> Most likely because the function could update 'current', GCC copies r2 into r30, so that if r2 get
> changed by the called function, ti_flagsp is still based on the previous value of current.
>
> Allthough we know r2 wont change, GCC doesn't know it. And in order to save r2 into r30, it needs to
> save r30 in the stack.
>
>
> By using ¤t_thread_info()->flags directly instead of this intermediaite ti_flagsp pointer, GCC
> uses r2 instead instead of doing a copy.
>
>
> Nick, I don't understand the reason why you need that 'ti_flagsp' local var.
Just to save typing, I don't mind your patch I was just wondering if
current could be improved in general.
Thanks,
Nick
Le 09/02/2021 à 02:55, Nicholas Piggin a écrit :
> Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
>> When r3 is not modified, reload it from regs->orig_r3 to free
>> volatile registers. This avoids a stack frame for the likely part
>> of system_call_exception()
>
> This doesn't on my 64s build, but it does reduce one non volatile
> register save/restore. With quite a bit more register pressure
> reduction 64s can avoid the stack frame as well.
The stack frame is not due to the registers because on PPC64 you have the redzone that you don't
have on PPC32.
As far as I can see, this is due to a call to .arch_local_irq_restore().
On ppc32 arch_local_irq_restore() is just a write to MSR.
>
> It's a cool trick but quite code and compiler specific so I don't know
> how worthwhile it is to keep considering we're calling out into random
> kernel C code after this.
>
> Maybe just keep it PPC32 specific for the moment, will have to do more
> tuning for 64 and we have other stuff to do there first.
>
> If you are happy to make it 32-bit only then
I think we can leave without this, that's only one or two cycles won.
>
> Reviewed-by: Nicholas Piggin <[email protected]>
>
Le 09/02/2021 à 15:31, David Laight a écrit :
> From: Segher Boessenkool
>> Sent: 09 February 2021 13:51
>>
>> On Tue, Feb 09, 2021 at 12:36:20PM +1000, Nicholas Piggin wrote:
>>> What if you did this?
>>
>>> +static inline struct task_struct *get_current(void)
>>> +{
>>> + register struct task_struct *task asm ("r2");
>>> +
>>> + return task;
>>> +}
>>
>> Local register asm variables are *only* guaranteed to live in that
>> register as operands to an asm. See
>> https://gcc.gnu.org/onlinedocs/gcc/Local-Register-Variables.html#Local-Register-Variables
>> ("The only supported use" etc.)
>>
>> You can do something like
>>
>> static inline struct task_struct *get_current(void)
>> {
>> register struct task_struct *task asm ("r2");
>>
>> asm("" : "+r"(task));
>>
>> return task;
>> }
>>
>> which makes sure that "task" actually is in r2 at the point of that asm.
>
> If "r2" always contains current (and is never assigned by the compiler)
> why not use a global register variable for it?
>
The change proposed by Nick doesn't solve the issue.
The problem is that at the begining of the function we have:
unsigned long *ti_flagsp = ¤t_thread_info()->flags;
When the function uses ti_flagsp for the first time, it does use 112(r2)
Then the function calls some other functions.
Most likely because the function could update 'current', GCC copies r2 into r30, so that if r2 get
changed by the called function, ti_flagsp is still based on the previous value of current.
Allthough we know r2 wont change, GCC doesn't know it. And in order to save r2 into r30, it needs to
save r30 in the stack.
By using ¤t_thread_info()->flags directly instead of this intermediaite ti_flagsp pointer, GCC
uses r2 instead instead of doing a copy.
Nick, I don't understand the reason why you need that 'ti_flagsp' local var.
Christophe
Le 10/02/2021 à 03:00, Nicholas Piggin a écrit :
> Excerpts from Christophe Leroy's message of February 10, 2021 3:03 am:
>>
>>
>> Le 09/02/2021 à 15:31, David Laight a écrit :
>>> From: Segher Boessenkool
>>>> Sent: 09 February 2021 13:51
>>>>
>>>> On Tue, Feb 09, 2021 at 12:36:20PM +1000, Nicholas Piggin wrote:
>>>>> What if you did this?
>>>>
>>>>> +static inline struct task_struct *get_current(void)
>>>>> +{
>>>>> + register struct task_struct *task asm ("r2");
>>>>> +
>>>>> + return task;
>>>>> +}
>>>>
>>>> Local register asm variables are *only* guaranteed to live in that
>>>> register as operands to an asm. See
>>>> https://gcc.gnu.org/onlinedocs/gcc/Local-Register-Variables.html#Local-Register-Variables
>>>> ("The only supported use" etc.)
>>>>
>>>> You can do something like
>>>>
>>>> static inline struct task_struct *get_current(void)
>>>> {
>>>> register struct task_struct *task asm ("r2");
>>>>
>>>> asm("" : "+r"(task));
>>>>
>>>> return task;
>>>> }
>>>>
>>>> which makes sure that "task" actually is in r2 at the point of that asm.
>>>
>>> If "r2" always contains current (and is never assigned by the compiler)
>>> why not use a global register variable for it?
>>>
>>
>>
>> The change proposed by Nick doesn't solve the issue.
>
> It seemed to change code generation in a simple test case, oh well.
>
>>
>> The problem is that at the begining of the function we have:
>>
>> unsigned long *ti_flagsp = ¤t_thread_info()->flags;
>>
>> When the function uses ti_flagsp for the first time, it does use 112(r2)
>>
>> Then the function calls some other functions.
>>
>> Most likely because the function could update 'current', GCC copies r2 into r30, so that if r2 get
>> changed by the called function, ti_flagsp is still based on the previous value of current.
>>
>> Allthough we know r2 wont change, GCC doesn't know it. And in order to save r2 into r30, it needs to
>> save r30 in the stack.
>>
>>
>> By using ¤t_thread_info()->flags directly instead of this intermediaite ti_flagsp pointer, GCC
>> uses r2 instead instead of doing a copy.
>>
>>
>> Nick, I don't understand the reason why you need that 'ti_flagsp' local var.
>
> Just to save typing, I don't mind your patch I was just wondering if
> current could be improved in general.
>
Thanks,
I'll post v6 of it as a follow-up of yesterday's two remaining follow-up patches.
Christophe
On Mon, 8 Feb 2021 15:10:19 +0000 (UTC), Christophe Leroy wrote:
> This series implements C syscall entry/exit for PPC32. It reuses
> the work already done for PPC64.
>
> This series is based on today's merge-test (b6f72fc05389e3fc694bf5a5fa1bbd33f61879e0)
>
> In terms on performance we have the following number of cycles on an
> 8xx running null_syscall benchmark:
> - mainline: 296 cycles
> - after patch 4: 283 cycles
> - after patch 16: 304 cycles
> - after patch 17: 348 cycles
> - at the end of the series: 320 cycles
>
> [...]
Patches 1-15 and 21 applied to powerpc/next.
[01/22] powerpc/32s: Add missing call to kuep_lock on syscall entry
https://git.kernel.org/powerpc/c/57fdfbce89137ae85cd5cef48be168040a47dd13
[02/22] powerpc/32: Always enable data translation on syscall entry
https://git.kernel.org/powerpc/c/eca2411040c1ee15b8882c6427fb4eb5a48ada69
[03/22] powerpc/32: On syscall entry, enable instruction translation at the same time as data
https://git.kernel.org/powerpc/c/76249ddc27080b6b835a89cedcc4185b3b5a6b23
[04/22] powerpc/32: Reorder instructions to avoid using CTR in syscall entry
https://git.kernel.org/powerpc/c/2c59e5104821c5720e88bafa9e522f8bea9ce8fa
[05/22] powerpc/irq: Add helper to set regs->softe
https://git.kernel.org/powerpc/c/fb5608fd117a8b48752d2b5a7e70847c1ed33d33
[06/22] powerpc/irq: Rework helpers that manipulate MSR[EE/RI]
https://git.kernel.org/powerpc/c/08353779f2889305f64e04de3e46ed59ed60f859
[07/22] powerpc/irq: Add stub irq_soft_mask_return() for PPC32
https://git.kernel.org/powerpc/c/6650c4782d5788346a25a4f698880d124f2699a0
[08/22] powerpc/syscall: Rename syscall_64.c into interrupt.c
https://git.kernel.org/powerpc/c/ab1a517d55b01b54ba70f5d54f926f5ab4b18339
[09/22] powerpc/syscall: Make interrupt.c buildable on PPC32
https://git.kernel.org/powerpc/c/344bb20b159dd0996e521c0d4c131a6ae10c322a
[10/22] powerpc/syscall: Use is_compat_task()
https://git.kernel.org/powerpc/c/72b7a9e56b25babfe4c90bf3ce88285c7fb62ab9
[11/22] powerpc/syscall: Save r3 in regs->orig_r3
https://git.kernel.org/powerpc/c/8875f47b7681aa4e4484a9b612577b044725f839
[12/22] powerpc/syscall: Change condition to check MSR_RI
https://git.kernel.org/powerpc/c/c01b916658150e98f00a4981750c37a3224c8735
[13/22] powerpc/32: Always save non volatile GPRs at syscall entry
https://git.kernel.org/powerpc/c/fbcee2ebe8edbb6a93316f0a189ae7fcfaa7094f
[14/22] powerpc/syscall: implement system call entry/exit logic in C for PPC32
https://git.kernel.org/powerpc/c/6f76a01173ccaa363739f913394d4e138d92d718
[15/22] powerpc/32: Remove verification of MSR_PR on syscall in the ASM entry
https://git.kernel.org/powerpc/c/4d67facbcbdb3d9e3c9cb82e4ec47fc63d298dd8
[21/22] powerpc/32: Remove the counter in global_dbcr0
https://git.kernel.org/powerpc/c/eb595eca74067b78d36fb188b555e30f28686fc7
cheers
Le 09/02/2021 à 08:49, Nicholas Piggin a écrit :
> Excerpts from Christophe Leroy's message of February 9, 2021 4:18 pm:
>>
>>
>> Le 09/02/2021 à 02:11, Nicholas Piggin a écrit :
>>> Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
>>>> regs->softe doesn't exist on PPC32.
>>>>
>>>> Add irq_soft_mask_regs_set_state() helper to set regs->softe.
>>>> This helper will void on PPC32.
>>>>
>>>> Signed-off-by: Christophe Leroy <[email protected]>
>>>> ---
>>>
>>> You could do the same with the kuap_ functions to change some ifdefs
>>> to IS_ENABLED.
>>>
>>> That's just my preference but if you prefer this way I guess that's
>>> okay.
>>>
>>
>>
>> That's also my preference on the long term.
>>
>> Here it is ephemeral, I have a follow up series implementing interrupt exit/entry in C and getting
>> rid of all the assembly kuap hence getting rid of those ifdefs.
>
> I thought it might have been because you hate ifdef more tha most :)
>
>> The issue I see when using IS_ENABLED() is that you have to indent to the right, then you interfere
>> with the file history and 'git blame'
>
> Valid point if it's just going to indent back the other way in your next
> series.
>
>> Thanks for reviewing my series and looking forward to your feedback on my series on the interrupt
>> entry/exit that I will likely release later today.
>
> Cool, I'm eager to see them.
>
Hi Nick, have you been able to look at it ?
https://patchwork.ozlabs.org/project/linuxppc-dev/cover/[email protected]/
Thanks
Christophe
Excerpts from Christophe Leroy's message of March 5, 2021 6:54 pm:
>
>
> Le 09/02/2021 à 08:49, Nicholas Piggin a écrit :
>> Excerpts from Christophe Leroy's message of February 9, 2021 4:18 pm:
>>>
>>>
>>> Le 09/02/2021 à 02:11, Nicholas Piggin a écrit :
>>>> Excerpts from Christophe Leroy's message of February 9, 2021 1:10 am:
>>>>> regs->softe doesn't exist on PPC32.
>>>>>
>>>>> Add irq_soft_mask_regs_set_state() helper to set regs->softe.
>>>>> This helper will void on PPC32.
>>>>>
>>>>> Signed-off-by: Christophe Leroy <[email protected]>
>>>>> ---
>>>>
>>>> You could do the same with the kuap_ functions to change some ifdefs
>>>> to IS_ENABLED.
>>>>
>>>> That's just my preference but if you prefer this way I guess that's
>>>> okay.
>>>>
>>>
>>>
>>> That's also my preference on the long term.
>>>
>>> Here it is ephemeral, I have a follow up series implementing interrupt exit/entry in C and getting
>>> rid of all the assembly kuap hence getting rid of those ifdefs.
>>
>> I thought it might have been because you hate ifdef more tha most :)
>>
>>> The issue I see when using IS_ENABLED() is that you have to indent to the right, then you interfere
>>> with the file history and 'git blame'
>>
>> Valid point if it's just going to indent back the other way in your next
>> series.
>>
>>> Thanks for reviewing my series and looking forward to your feedback on my series on the interrupt
>>> entry/exit that I will likely release later today.
>>
>> Cool, I'm eager to see them.
>>
>
> Hi Nick, have you been able to look at it ?
>
> https://patchwork.ozlabs.org/project/linuxppc-dev/cover/[email protected]/
Hi Christophe,
I had a look at it, it's mostly ppc32 code which I don't know well but
it looks like a very nice cleanup and it's good to be sharing the C
code here. All the common code changes look fine to me.
I'll take a closer look if you can rebase and repost the series I need
to create a tree and base 64e conversion on top of yours as they touch
the same common places.
Thanks,
Nick