This series implements MIPS support for restartable sequences, hooks up
the rseq syscall & implements MIPS support in the rseq selftests.
Applies atop Linus' master as of 2837461dbe6f ("Merge tag 'scsi-fixes'
of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi").
Thanks,
Paul
Paul Burton (4):
MIPS: Add support for restartable sequences
MIPS: Add syscall detection for restartable sequences
MIPS: Wire up the restartable sequences (rseq) syscall
rseq/selftests: Implement MIPS support
arch/mips/Kconfig | 1 +
arch/mips/include/uapi/asm/unistd.h | 15 +-
arch/mips/kernel/entry.S | 8 +
arch/mips/kernel/scall32-o32.S | 1 +
arch/mips/kernel/scall64-64.S | 1 +
arch/mips/kernel/scall64-n32.S | 1 +
arch/mips/kernel/scall64-o32.S | 1 +
arch/mips/kernel/signal.c | 3 +
tools/testing/selftests/rseq/param_test.c | 24 +
tools/testing/selftests/rseq/rseq-mips.h | 725 ++++++++++++++++++++++
tools/testing/selftests/rseq/rseq.h | 2 +
11 files changed, 776 insertions(+), 6 deletions(-)
create mode 100644 tools/testing/selftests/rseq/rseq-mips.h
--
2.17.1
Implement support for restartable sequences on MIPS, which requires 3
simple things:
- Call rseq_handle_notify_resume() on return to userspace if
TIF_NOTIFY_RESUME is set.
- Call rseq_signal_deliver() to fixup the pre-signal stack frame when
a signal is delivered whilst executing a restartable sequence
critical section.
- Select CONFIG_HAVE_RSEQ.
Signed-off-by: Paul Burton <[email protected]>
Cc: James Hogan <[email protected]>
Cc: Ralf Baechle <[email protected]>
Cc: Mathieu Desnoyers <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Paul E. McKenney <[email protected]>
Cc: Boqun Feng <[email protected]>
Cc: [email protected]
Cc: [email protected]
---
arch/mips/Kconfig | 1 +
arch/mips/kernel/signal.c | 3 +++
2 files changed, 4 insertions(+)
diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index fe98e459a416..afe2b0c867ac 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -66,6 +66,7 @@ config MIPS
select HAVE_OPROFILE
select HAVE_PERF_EVENTS
select HAVE_REGS_AND_STACK_ACCESS_API
+ select HAVE_RSEQ
select HAVE_SYSCALL_TRACEPOINTS
select HAVE_VIRT_CPU_ACCOUNTING_GEN if 64BIT || !SMP
select IRQ_FORCED_THREADING
diff --git a/arch/mips/kernel/signal.c b/arch/mips/kernel/signal.c
index 9e224469c788..00f2535d2226 100644
--- a/arch/mips/kernel/signal.c
+++ b/arch/mips/kernel/signal.c
@@ -801,6 +801,8 @@ static void handle_signal(struct ksignal *ksig, struct pt_regs *regs)
regs->regs[0] = 0; /* Don't deal with this again. */
}
+ rseq_signal_deliver(regs);
+
if (sig_uses_siginfo(&ksig->ka, abi))
ret = abi->setup_rt_frame(vdso + abi->vdso->off_rt_sigreturn,
ksig, regs, oldset);
@@ -868,6 +870,7 @@ asmlinkage void do_notify_resume(struct pt_regs *regs, void *unused,
if (thread_info_flags & _TIF_NOTIFY_RESUME) {
clear_thread_flag(TIF_NOTIFY_RESUME);
tracehook_notify_resume(regs);
+ rseq_handle_notify_resume(regs);
}
user_enter();
--
2.17.1
Syscalls are not allowed inside restartable sequences, so add a call to
rseq_syscall() at the very beginning of the system call exit path when
CONFIG_DEBUG_RSEQ=y. This will help us to detect whether there is a
syscall issued erroneously inside a restartable sequence.
Signed-off-by: Paul Burton <[email protected]>
Cc: James Hogan <[email protected]>
Cc: Ralf Baechle <[email protected]>
Cc: Mathieu Desnoyers <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Paul E. McKenney <[email protected]>
Cc: Boqun Feng <[email protected]>
Cc: [email protected]
Cc: [email protected]
---
arch/mips/kernel/entry.S | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/arch/mips/kernel/entry.S b/arch/mips/kernel/entry.S
index 38a302919e6b..d7de8adcfcc8 100644
--- a/arch/mips/kernel/entry.S
+++ b/arch/mips/kernel/entry.S
@@ -79,6 +79,10 @@ FEXPORT(ret_from_fork)
jal schedule_tail # a0 = struct task_struct *prev
FEXPORT(syscall_exit)
+#ifdef CONFIG_DEBUG_RSEQ
+ move a0, sp
+ jal rseq_syscall
+#endif
local_irq_disable # make sure need_resched and
# signals dont change between
# sampling and return
@@ -141,6 +145,10 @@ work_notifysig: # deal with pending signals and
j resume_userspace_check
FEXPORT(syscall_exit_partial)
+#ifdef CONFIG_DEBUG_RSEQ
+ move a0, sp
+ jal rseq_syscall
+#endif
local_irq_disable # make sure need_resched doesn't
# change between and return
LONG_L a2, TI_FLAGS($28) # current->work
--
2.17.1
Implement support for both MIPS32 & MIPS64 in the rseq selftests, in
order to sanity check the recently enabled rseq syscall.
The tests all pass on a MIPS Boston development board running either a
MIPS32r2 interAptiv CPU & a MIPS64r6 I6500 CPU, both of which were
configured with 2 cores each of which have 2 hardware threads (VP(E)s) -
ie. 4 CPUs.
Signed-off-by: Paul Burton <[email protected]>
Cc: James Hogan <[email protected]>
Cc: Ralf Baechle <[email protected]>
Cc: Mathieu Desnoyers <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Paul E. McKenney <[email protected]>
Cc: Boqun Feng <[email protected]>
Cc: [email protected]
Cc: [email protected]
---
tools/testing/selftests/rseq/param_test.c | 24 +
tools/testing/selftests/rseq/rseq-mips.h | 725 ++++++++++++++++++++++
tools/testing/selftests/rseq/rseq.h | 2 +
3 files changed, 751 insertions(+)
create mode 100644 tools/testing/selftests/rseq/rseq-mips.h
diff --git a/tools/testing/selftests/rseq/param_test.c b/tools/testing/selftests/rseq/param_test.c
index 6a9f602a8718..615252331813 100644
--- a/tools/testing/selftests/rseq/param_test.c
+++ b/tools/testing/selftests/rseq/param_test.c
@@ -137,6 +137,30 @@ unsigned int yield_mod_cnt, nr_abort;
"subic. %%" INJECT_ASM_REG ", %%" INJECT_ASM_REG ", 1\n\t" \
"bne 222b\n\t" \
"333:\n\t"
+
+#elif defined(__mips__)
+
+#define RSEQ_INJECT_INPUT \
+ , [loop_cnt_1]"m"(loop_cnt[1]) \
+ , [loop_cnt_2]"m"(loop_cnt[2]) \
+ , [loop_cnt_3]"m"(loop_cnt[3]) \
+ , [loop_cnt_4]"m"(loop_cnt[4]) \
+ , [loop_cnt_5]"m"(loop_cnt[5]) \
+ , [loop_cnt_6]"m"(loop_cnt[6])
+
+#define INJECT_ASM_REG "$5"
+
+#define RSEQ_INJECT_CLOBBER \
+ , INJECT_ASM_REG
+
+#define RSEQ_INJECT_ASM(n) \
+ "lw " INJECT_ASM_REG ", %[loop_cnt_" #n "]\n\t" \
+ "beqz " INJECT_ASM_REG ", 333f\n\t" \
+ "222:\n\t" \
+ "addiu " INJECT_ASM_REG ", -1\n\t" \
+ "bnez " INJECT_ASM_REG ", 222b\n\t" \
+ "333:\n\t"
+
#else
#error unsupported target
#endif
diff --git a/tools/testing/selftests/rseq/rseq-mips.h b/tools/testing/selftests/rseq/rseq-mips.h
new file mode 100644
index 000000000000..63131352fbdd
--- /dev/null
+++ b/tools/testing/selftests/rseq/rseq-mips.h
@@ -0,0 +1,725 @@
+/* SPDX-License-Identifier: LGPL-2.1 OR MIT */
+/*
+ * Author: Paul Burton <[email protected]>
+ * (C) Copyright 2018 MIPS Tech LLC
+ *
+ * Based on rseq-arm.h:
+ * (C) Copyright 2016-2018 - Mathieu Desnoyers <[email protected]>
+ */
+
+#define RSEQ_SIG 0x53053053
+
+#define rseq_smp_mb() __asm__ __volatile__ ("sync" ::: "memory")
+#define rseq_smp_rmb() rseq_smp_mb()
+#define rseq_smp_wmb() rseq_smp_mb()
+
+#define rseq_smp_load_acquire(p) \
+__extension__ ({ \
+ __typeof(*p) ____p1 = RSEQ_READ_ONCE(*p); \
+ rseq_smp_mb(); \
+ ____p1; \
+})
+
+#define rseq_smp_acquire__after_ctrl_dep() rseq_smp_rmb()
+
+#define rseq_smp_store_release(p, v) \
+do { \
+ rseq_smp_mb(); \
+ RSEQ_WRITE_ONCE(*p, v); \
+} while (0)
+
+#ifdef RSEQ_SKIP_FASTPATH
+#include "rseq-skip.h"
+#else /* !RSEQ_SKIP_FASTPATH */
+
+#if _MIPS_SZLONG == 64
+# define LONG ".dword"
+# define LONG_LA "dla"
+# define LONG_L "ld"
+# define LONG_S "sd"
+# define LONG_ADDI "daddiu"
+# define U32_U64_PAD(x) x
+#elif _MIPS_SZLONG == 32
+# define LONG ".word"
+# define LONG_LA "la"
+# define LONG_L "lw"
+# define LONG_S "sw"
+# define LONG_ADDI "addiu"
+# ifdef __BIG_ENDIAN
+# define U32_U64_PAD(x) "0x0, " x
+# else
+# define U32_U64_PAD(x) x ", 0x0"
+# endif
+#else
+# error unsupported _MIPS_SZLONG
+#endif
+
+#define __RSEQ_ASM_DEFINE_TABLE(version, flags, start_ip, \
+ post_commit_offset, abort_ip) \
+ ".pushsection __rseq_table, \"aw\"\n\t" \
+ ".balign 32\n\t" \
+ ".word " __rseq_str(version) ", " __rseq_str(flags) "\n\t" \
+ LONG " " U32_U64_PAD(__rseq_str(start_ip)) "\n\t" \
+ LONG " " U32_U64_PAD(__rseq_str(post_commit_offset)) "\n\t" \
+ LONG " " U32_U64_PAD(__rseq_str(abort_ip)) "\n\t" \
+ ".popsection\n\t"
+
+#define RSEQ_ASM_DEFINE_TABLE(start_ip, post_commit_ip, abort_ip) \
+ __RSEQ_ASM_DEFINE_TABLE(0x0, 0x0, start_ip, \
+ (post_commit_ip - start_ip), abort_ip)
+
+#define RSEQ_ASM_STORE_RSEQ_CS(label, cs_label, rseq_cs) \
+ RSEQ_INJECT_ASM(1) \
+ LONG_LA " $4, " __rseq_str(cs_label) "\n\t" \
+ LONG_S " $4, %[" __rseq_str(rseq_cs) "]\n\t" \
+ __rseq_str(label) ":\n\t"
+
+#define RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, label) \
+ RSEQ_INJECT_ASM(2) \
+ "lw $4, %[" __rseq_str(current_cpu_id) "]\n\t" \
+ "bne $4, %[" __rseq_str(cpu_id) "], " __rseq_str(label) "\n\t"
+
+#define __RSEQ_ASM_DEFINE_ABORT(table_label, label, teardown, \
+ abort_label, version, flags, \
+ start_ip, post_commit_offset, abort_ip) \
+ ".balign 32\n\t" \
+ __rseq_str(table_label) ":\n\t" \
+ ".word " __rseq_str(version) ", " __rseq_str(flags) "\n\t" \
+ LONG " " U32_U64_PAD(__rseq_str(start_ip)) "\n\t" \
+ LONG " " U32_U64_PAD(__rseq_str(post_commit_offset)) "\n\t" \
+ LONG " " U32_U64_PAD(__rseq_str(abort_ip)) "\n\t" \
+ ".word " __rseq_str(RSEQ_SIG) "\n\t" \
+ __rseq_str(label) ":\n\t" \
+ teardown \
+ "b %l[" __rseq_str(abort_label) "]\n\t"
+
+#define RSEQ_ASM_DEFINE_ABORT(table_label, label, teardown, abort_label, \
+ start_ip, post_commit_ip, abort_ip) \
+ __RSEQ_ASM_DEFINE_ABORT(table_label, label, teardown, \
+ abort_label, 0x0, 0x0, start_ip, \
+ (post_commit_ip - start_ip), abort_ip)
+
+#define RSEQ_ASM_DEFINE_CMPFAIL(label, teardown, cmpfail_label) \
+ __rseq_str(label) ":\n\t" \
+ teardown \
+ "b %l[" __rseq_str(cmpfail_label) "]\n\t"
+
+#define rseq_workaround_gcc_asm_size_guess() __asm__ __volatile__("")
+
+static inline __attribute__((always_inline))
+int rseq_cmpeqv_storev(intptr_t *v, intptr_t expect, intptr_t newv, int cpu)
+{
+ RSEQ_INJECT_C(9)
+
+ rseq_workaround_gcc_asm_size_guess();
+ __asm__ __volatile__ goto (
+ RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */
+ /* Start rseq by storing table entry pointer into rseq_cs. */
+ RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
+ RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
+ RSEQ_INJECT_ASM(3)
+ LONG_L " $4, %[v]\n\t"
+ "bne $4, %[expect], %l[cmpfail]\n\t"
+ RSEQ_INJECT_ASM(4)
+#ifdef RSEQ_COMPARE_TWICE
+ RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
+ LONG_L " $4, %[v]\n\t"
+ "bne $4, %[expect], %l[error2]\n\t"
+#endif
+ /* final store */
+ LONG_S " %[newv], %[v]\n\t"
+ "2:\n\t"
+ RSEQ_INJECT_ASM(5)
+ "b 5f\n\t"
+ RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
+ "5:\n\t"
+ : /* gcc asm goto does not allow outputs */
+ : [cpu_id] "r" (cpu),
+ [current_cpu_id] "m" (__rseq_abi.cpu_id),
+ [rseq_cs] "m" (__rseq_abi.rseq_cs),
+ [v] "m" (*v),
+ [expect] "r" (expect),
+ [newv] "r" (newv)
+ RSEQ_INJECT_INPUT
+ : "$4", "memory"
+ RSEQ_INJECT_CLOBBER
+ : abort, cmpfail
+#ifdef RSEQ_COMPARE_TWICE
+ , error1, error2
+#endif
+ );
+ rseq_workaround_gcc_asm_size_guess();
+ return 0;
+abort:
+ rseq_workaround_gcc_asm_size_guess();
+ RSEQ_INJECT_FAILED
+ return -1;
+cmpfail:
+ rseq_workaround_gcc_asm_size_guess();
+ return 1;
+#ifdef RSEQ_COMPARE_TWICE
+error1:
+ rseq_bug("cpu_id comparison failed");
+error2:
+ rseq_bug("expected value comparison failed");
+#endif
+}
+
+static inline __attribute__((always_inline))
+int rseq_cmpnev_storeoffp_load(intptr_t *v, intptr_t expectnot,
+ off_t voffp, intptr_t *load, int cpu)
+{
+ RSEQ_INJECT_C(9)
+
+ rseq_workaround_gcc_asm_size_guess();
+ __asm__ __volatile__ goto (
+ RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */
+ /* Start rseq by storing table entry pointer into rseq_cs. */
+ RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
+ RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
+ RSEQ_INJECT_ASM(3)
+ LONG_L " $4, %[v]\n\t"
+ "beq $4, %[expectnot], %l[cmpfail]\n\t"
+ RSEQ_INJECT_ASM(4)
+#ifdef RSEQ_COMPARE_TWICE
+ RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
+ LONG_L " $4, %[v]\n\t"
+ "beq $4, %[expectnot], %l[error2]\n\t"
+#endif
+ LONG_S " $4, %[load]\n\t"
+ LONG_ADDI " $4, %[voffp]\n\t"
+ LONG_L " $4, 0($4)\n\t"
+ /* final store */
+ LONG_S " $4, %[v]\n\t"
+ "2:\n\t"
+ RSEQ_INJECT_ASM(5)
+ "b 5f\n\t"
+ RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
+ "5:\n\t"
+ : /* gcc asm goto does not allow outputs */
+ : [cpu_id] "r" (cpu),
+ [current_cpu_id] "m" (__rseq_abi.cpu_id),
+ [rseq_cs] "m" (__rseq_abi.rseq_cs),
+ /* final store input */
+ [v] "m" (*v),
+ [expectnot] "r" (expectnot),
+ [voffp] "Ir" (voffp),
+ [load] "m" (*load)
+ RSEQ_INJECT_INPUT
+ : "$4", "memory"
+ RSEQ_INJECT_CLOBBER
+ : abort, cmpfail
+#ifdef RSEQ_COMPARE_TWICE
+ , error1, error2
+#endif
+ );
+ rseq_workaround_gcc_asm_size_guess();
+ return 0;
+abort:
+ rseq_workaround_gcc_asm_size_guess();
+ RSEQ_INJECT_FAILED
+ return -1;
+cmpfail:
+ rseq_workaround_gcc_asm_size_guess();
+ return 1;
+#ifdef RSEQ_COMPARE_TWICE
+error1:
+ rseq_bug("cpu_id comparison failed");
+error2:
+ rseq_bug("expected value comparison failed");
+#endif
+}
+
+static inline __attribute__((always_inline))
+int rseq_addv(intptr_t *v, intptr_t count, int cpu)
+{
+ RSEQ_INJECT_C(9)
+
+ rseq_workaround_gcc_asm_size_guess();
+ __asm__ __volatile__ goto (
+ RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */
+ /* Start rseq by storing table entry pointer into rseq_cs. */
+ RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
+ RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
+ RSEQ_INJECT_ASM(3)
+#ifdef RSEQ_COMPARE_TWICE
+ RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
+#endif
+ LONG_L " $4, %[v]\n\t"
+ LONG_ADDI " $4, %[count]\n\t"
+ /* final store */
+ LONG_S " $4, %[v]\n\t"
+ "2:\n\t"
+ RSEQ_INJECT_ASM(4)
+ "b 5f\n\t"
+ RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
+ "5:\n\t"
+ : /* gcc asm goto does not allow outputs */
+ : [cpu_id] "r" (cpu),
+ [current_cpu_id] "m" (__rseq_abi.cpu_id),
+ [rseq_cs] "m" (__rseq_abi.rseq_cs),
+ [v] "m" (*v),
+ [count] "Ir" (count)
+ RSEQ_INJECT_INPUT
+ : "$4", "memory"
+ RSEQ_INJECT_CLOBBER
+ : abort
+#ifdef RSEQ_COMPARE_TWICE
+ , error1
+#endif
+ );
+ rseq_workaround_gcc_asm_size_guess();
+ return 0;
+abort:
+ rseq_workaround_gcc_asm_size_guess();
+ RSEQ_INJECT_FAILED
+ return -1;
+#ifdef RSEQ_COMPARE_TWICE
+error1:
+ rseq_bug("cpu_id comparison failed");
+#endif
+}
+
+static inline __attribute__((always_inline))
+int rseq_cmpeqv_trystorev_storev(intptr_t *v, intptr_t expect,
+ intptr_t *v2, intptr_t newv2,
+ intptr_t newv, int cpu)
+{
+ RSEQ_INJECT_C(9)
+
+ rseq_workaround_gcc_asm_size_guess();
+ __asm__ __volatile__ goto (
+ RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */
+ /* Start rseq by storing table entry pointer into rseq_cs. */
+ RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
+ RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
+ RSEQ_INJECT_ASM(3)
+ LONG_L " $4, %[v]\n\t"
+ "bne $4, %[expect], %l[cmpfail]\n\t"
+ RSEQ_INJECT_ASM(4)
+#ifdef RSEQ_COMPARE_TWICE
+ RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
+ LONG_L " $4, %[v]\n\t"
+ "bne $4, %[expect], %l[error2]\n\t"
+#endif
+ /* try store */
+ LONG_S " %[newv2], %[v2]\n\t"
+ RSEQ_INJECT_ASM(5)
+ /* final store */
+ LONG_S " %[newv], %[v]\n\t"
+ "2:\n\t"
+ RSEQ_INJECT_ASM(6)
+ "b 5f\n\t"
+ RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
+ "5:\n\t"
+ : /* gcc asm goto does not allow outputs */
+ : [cpu_id] "r" (cpu),
+ [current_cpu_id] "m" (__rseq_abi.cpu_id),
+ [rseq_cs] "m" (__rseq_abi.rseq_cs),
+ /* try store input */
+ [v2] "m" (*v2),
+ [newv2] "r" (newv2),
+ /* final store input */
+ [v] "m" (*v),
+ [expect] "r" (expect),
+ [newv] "r" (newv)
+ RSEQ_INJECT_INPUT
+ : "$4", "memory"
+ RSEQ_INJECT_CLOBBER
+ : abort, cmpfail
+#ifdef RSEQ_COMPARE_TWICE
+ , error1, error2
+#endif
+ );
+ rseq_workaround_gcc_asm_size_guess();
+ return 0;
+abort:
+ rseq_workaround_gcc_asm_size_guess();
+ RSEQ_INJECT_FAILED
+ return -1;
+cmpfail:
+ rseq_workaround_gcc_asm_size_guess();
+ return 1;
+#ifdef RSEQ_COMPARE_TWICE
+error1:
+ rseq_bug("cpu_id comparison failed");
+error2:
+ rseq_bug("expected value comparison failed");
+#endif
+}
+
+static inline __attribute__((always_inline))
+int rseq_cmpeqv_trystorev_storev_release(intptr_t *v, intptr_t expect,
+ intptr_t *v2, intptr_t newv2,
+ intptr_t newv, int cpu)
+{
+ RSEQ_INJECT_C(9)
+
+ rseq_workaround_gcc_asm_size_guess();
+ __asm__ __volatile__ goto (
+ RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */
+ /* Start rseq by storing table entry pointer into rseq_cs. */
+ RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
+ RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
+ RSEQ_INJECT_ASM(3)
+ LONG_L " $4, %[v]\n\t"
+ "bne $4, %[expect], %l[cmpfail]\n\t"
+ RSEQ_INJECT_ASM(4)
+#ifdef RSEQ_COMPARE_TWICE
+ RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
+ LONG_L " $4, %[v]\n\t"
+ "bne $4, %[expect], %l[error2]\n\t"
+#endif
+ /* try store */
+ LONG_S " %[newv2], %[v2]\n\t"
+ RSEQ_INJECT_ASM(5)
+ "sync\n\t" /* full sync provides store-release */
+ /* final store */
+ LONG_S " %[newv], %[v]\n\t"
+ "2:\n\t"
+ RSEQ_INJECT_ASM(6)
+ "b 5f\n\t"
+ RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
+ "5:\n\t"
+ : /* gcc asm goto does not allow outputs */
+ : [cpu_id] "r" (cpu),
+ [current_cpu_id] "m" (__rseq_abi.cpu_id),
+ [rseq_cs] "m" (__rseq_abi.rseq_cs),
+ /* try store input */
+ [v2] "m" (*v2),
+ [newv2] "r" (newv2),
+ /* final store input */
+ [v] "m" (*v),
+ [expect] "r" (expect),
+ [newv] "r" (newv)
+ RSEQ_INJECT_INPUT
+ : "$4", "memory"
+ RSEQ_INJECT_CLOBBER
+ : abort, cmpfail
+#ifdef RSEQ_COMPARE_TWICE
+ , error1, error2
+#endif
+ );
+ rseq_workaround_gcc_asm_size_guess();
+ return 0;
+abort:
+ rseq_workaround_gcc_asm_size_guess();
+ RSEQ_INJECT_FAILED
+ return -1;
+cmpfail:
+ rseq_workaround_gcc_asm_size_guess();
+ return 1;
+#ifdef RSEQ_COMPARE_TWICE
+error1:
+ rseq_bug("cpu_id comparison failed");
+error2:
+ rseq_bug("expected value comparison failed");
+#endif
+}
+
+static inline __attribute__((always_inline))
+int rseq_cmpeqv_cmpeqv_storev(intptr_t *v, intptr_t expect,
+ intptr_t *v2, intptr_t expect2,
+ intptr_t newv, int cpu)
+{
+ RSEQ_INJECT_C(9)
+
+ rseq_workaround_gcc_asm_size_guess();
+ __asm__ __volatile__ goto (
+ RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */
+ /* Start rseq by storing table entry pointer into rseq_cs. */
+ RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
+ RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
+ RSEQ_INJECT_ASM(3)
+ LONG_L " $4, %[v]\n\t"
+ "bne $4, %[expect], %l[cmpfail]\n\t"
+ RSEQ_INJECT_ASM(4)
+ LONG_L " $4, %[v2]\n\t"
+ "bne $4, %[expect2], %l[cmpfail]\n\t"
+ RSEQ_INJECT_ASM(5)
+#ifdef RSEQ_COMPARE_TWICE
+ RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, %l[error1])
+ LONG_L " $4, %[v]\n\t"
+ "bne $4, %[expect], %l[error2]\n\t"
+ LONG_L " $4, %[v2]\n\t"
+ "bne $4, %[expect2], %l[error3]\n\t"
+#endif
+ /* final store */
+ LONG_S " %[newv], %[v]\n\t"
+ "2:\n\t"
+ RSEQ_INJECT_ASM(6)
+ "b 5f\n\t"
+ RSEQ_ASM_DEFINE_ABORT(3, 4, "", abort, 1b, 2b, 4f)
+ "5:\n\t"
+ : /* gcc asm goto does not allow outputs */
+ : [cpu_id] "r" (cpu),
+ [current_cpu_id] "m" (__rseq_abi.cpu_id),
+ [rseq_cs] "m" (__rseq_abi.rseq_cs),
+ /* cmp2 input */
+ [v2] "m" (*v2),
+ [expect2] "r" (expect2),
+ /* final store input */
+ [v] "m" (*v),
+ [expect] "r" (expect),
+ [newv] "r" (newv)
+ RSEQ_INJECT_INPUT
+ : "$4", "memory"
+ RSEQ_INJECT_CLOBBER
+ : abort, cmpfail
+#ifdef RSEQ_COMPARE_TWICE
+ , error1, error2, error3
+#endif
+ );
+ rseq_workaround_gcc_asm_size_guess();
+ return 0;
+abort:
+ rseq_workaround_gcc_asm_size_guess();
+ RSEQ_INJECT_FAILED
+ return -1;
+cmpfail:
+ rseq_workaround_gcc_asm_size_guess();
+ return 1;
+#ifdef RSEQ_COMPARE_TWICE
+error1:
+ rseq_bug("cpu_id comparison failed");
+error2:
+ rseq_bug("1st expected value comparison failed");
+error3:
+ rseq_bug("2nd expected value comparison failed");
+#endif
+}
+
+static inline __attribute__((always_inline))
+int rseq_cmpeqv_trymemcpy_storev(intptr_t *v, intptr_t expect,
+ void *dst, void *src, size_t len,
+ intptr_t newv, int cpu)
+{
+ uintptr_t rseq_scratch[3];
+
+ RSEQ_INJECT_C(9)
+
+ rseq_workaround_gcc_asm_size_guess();
+ __asm__ __volatile__ goto (
+ RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */
+ LONG_S " %[src], %[rseq_scratch0]\n\t"
+ LONG_S " %[dst], %[rseq_scratch1]\n\t"
+ LONG_S " %[len], %[rseq_scratch2]\n\t"
+ /* Start rseq by storing table entry pointer into rseq_cs. */
+ RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
+ RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
+ RSEQ_INJECT_ASM(3)
+ LONG_L " $4, %[v]\n\t"
+ "bne $4, %[expect], 5f\n\t"
+ RSEQ_INJECT_ASM(4)
+#ifdef RSEQ_COMPARE_TWICE
+ RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 6f)
+ LONG_L " $4, %[v]\n\t"
+ "bne $4, %[expect], 7f\n\t"
+#endif
+ /* try memcpy */
+ "beqz %[len], 333f\n\t" \
+ "222:\n\t" \
+ "lb $4, 0(%[src])\n\t" \
+ "sb $4, 0(%[dst])\n\t" \
+ LONG_ADDI " %[src], 1\n\t" \
+ LONG_ADDI " %[dst], 1\n\t" \
+ LONG_ADDI " %[len], -1\n\t" \
+ "bnez %[len], 222b\n\t" \
+ "333:\n\t" \
+ RSEQ_INJECT_ASM(5)
+ /* final store */
+ LONG_S " %[newv], %[v]\n\t"
+ "2:\n\t"
+ RSEQ_INJECT_ASM(6)
+ /* teardown */
+ LONG_L " %[len], %[rseq_scratch2]\n\t"
+ LONG_L " %[dst], %[rseq_scratch1]\n\t"
+ LONG_L " %[src], %[rseq_scratch0]\n\t"
+ "b 8f\n\t"
+ RSEQ_ASM_DEFINE_ABORT(3, 4,
+ /* teardown */
+ LONG_L " %[len], %[rseq_scratch2]\n\t"
+ LONG_L " %[dst], %[rseq_scratch1]\n\t"
+ LONG_L " %[src], %[rseq_scratch0]\n\t",
+ abort, 1b, 2b, 4f)
+ RSEQ_ASM_DEFINE_CMPFAIL(5,
+ /* teardown */
+ LONG_L " %[len], %[rseq_scratch2]\n\t"
+ LONG_L " %[dst], %[rseq_scratch1]\n\t"
+ LONG_L " %[src], %[rseq_scratch0]\n\t",
+ cmpfail)
+#ifdef RSEQ_COMPARE_TWICE
+ RSEQ_ASM_DEFINE_CMPFAIL(6,
+ /* teardown */
+ LONG_L " %[len], %[rseq_scratch2]\n\t"
+ LONG_L " %[dst], %[rseq_scratch1]\n\t"
+ LONG_L " %[src], %[rseq_scratch0]\n\t",
+ error1)
+ RSEQ_ASM_DEFINE_CMPFAIL(7,
+ /* teardown */
+ LONG_L " %[len], %[rseq_scratch2]\n\t"
+ LONG_L " %[dst], %[rseq_scratch1]\n\t"
+ LONG_L " %[src], %[rseq_scratch0]\n\t",
+ error2)
+#endif
+ "8:\n\t"
+ : /* gcc asm goto does not allow outputs */
+ : [cpu_id] "r" (cpu),
+ [current_cpu_id] "m" (__rseq_abi.cpu_id),
+ [rseq_cs] "m" (__rseq_abi.rseq_cs),
+ /* final store input */
+ [v] "m" (*v),
+ [expect] "r" (expect),
+ [newv] "r" (newv),
+ /* try memcpy input */
+ [dst] "r" (dst),
+ [src] "r" (src),
+ [len] "r" (len),
+ [rseq_scratch0] "m" (rseq_scratch[0]),
+ [rseq_scratch1] "m" (rseq_scratch[1]),
+ [rseq_scratch2] "m" (rseq_scratch[2])
+ RSEQ_INJECT_INPUT
+ : "$4", "memory"
+ RSEQ_INJECT_CLOBBER
+ : abort, cmpfail
+#ifdef RSEQ_COMPARE_TWICE
+ , error1, error2
+#endif
+ );
+ rseq_workaround_gcc_asm_size_guess();
+ return 0;
+abort:
+ rseq_workaround_gcc_asm_size_guess();
+ RSEQ_INJECT_FAILED
+ return -1;
+cmpfail:
+ rseq_workaround_gcc_asm_size_guess();
+ return 1;
+#ifdef RSEQ_COMPARE_TWICE
+error1:
+ rseq_workaround_gcc_asm_size_guess();
+ rseq_bug("cpu_id comparison failed");
+error2:
+ rseq_workaround_gcc_asm_size_guess();
+ rseq_bug("expected value comparison failed");
+#endif
+}
+
+static inline __attribute__((always_inline))
+int rseq_cmpeqv_trymemcpy_storev_release(intptr_t *v, intptr_t expect,
+ void *dst, void *src, size_t len,
+ intptr_t newv, int cpu)
+{
+ uintptr_t rseq_scratch[3];
+
+ RSEQ_INJECT_C(9)
+
+ rseq_workaround_gcc_asm_size_guess();
+ __asm__ __volatile__ goto (
+ RSEQ_ASM_DEFINE_TABLE(1f, 2f, 4f) /* start, commit, abort */
+ LONG_S " %[src], %[rseq_scratch0]\n\t"
+ LONG_S " %[dst], %[rseq_scratch1]\n\t"
+ LONG_S " %[len], %[rseq_scratch2]\n\t"
+ /* Start rseq by storing table entry pointer into rseq_cs. */
+ RSEQ_ASM_STORE_RSEQ_CS(1, 3f, rseq_cs)
+ RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 4f)
+ RSEQ_INJECT_ASM(3)
+ LONG_L " $4, %[v]\n\t"
+ "bne $4, %[expect], 5f\n\t"
+ RSEQ_INJECT_ASM(4)
+#ifdef RSEQ_COMPARE_TWICE
+ RSEQ_ASM_CMP_CPU_ID(cpu_id, current_cpu_id, 6f)
+ LONG_L " $4, %[v]\n\t"
+ "bne $4, %[expect], 7f\n\t"
+#endif
+ /* try memcpy */
+ "beqz %[len], 333f\n\t" \
+ "222:\n\t" \
+ "lb $4, 0(%[src])\n\t" \
+ "sb $4, 0(%[dst])\n\t" \
+ LONG_ADDI " %[src], 1\n\t" \
+ LONG_ADDI " %[dst], 1\n\t" \
+ LONG_ADDI " %[len], -1\n\t" \
+ "bnez %[len], 222b\n\t" \
+ "333:\n\t" \
+ RSEQ_INJECT_ASM(5)
+ "sync\n\t" /* full sync provides store-release */
+ /* final store */
+ LONG_S " %[newv], %[v]\n\t"
+ "2:\n\t"
+ RSEQ_INJECT_ASM(6)
+ /* teardown */
+ LONG_L " %[len], %[rseq_scratch2]\n\t"
+ LONG_L " %[dst], %[rseq_scratch1]\n\t"
+ LONG_L " %[src], %[rseq_scratch0]\n\t"
+ "b 8f\n\t"
+ RSEQ_ASM_DEFINE_ABORT(3, 4,
+ /* teardown */
+ LONG_L " %[len], %[rseq_scratch2]\n\t"
+ LONG_L " %[dst], %[rseq_scratch1]\n\t"
+ LONG_L " %[src], %[rseq_scratch0]\n\t",
+ abort, 1b, 2b, 4f)
+ RSEQ_ASM_DEFINE_CMPFAIL(5,
+ /* teardown */
+ LONG_L " %[len], %[rseq_scratch2]\n\t"
+ LONG_L " %[dst], %[rseq_scratch1]\n\t"
+ LONG_L " %[src], %[rseq_scratch0]\n\t",
+ cmpfail)
+#ifdef RSEQ_COMPARE_TWICE
+ RSEQ_ASM_DEFINE_CMPFAIL(6,
+ /* teardown */
+ LONG_L " %[len], %[rseq_scratch2]\n\t"
+ LONG_L " %[dst], %[rseq_scratch1]\n\t"
+ LONG_L " %[src], %[rseq_scratch0]\n\t",
+ error1)
+ RSEQ_ASM_DEFINE_CMPFAIL(7,
+ /* teardown */
+ LONG_L " %[len], %[rseq_scratch2]\n\t"
+ LONG_L " %[dst], %[rseq_scratch1]\n\t"
+ LONG_L " %[src], %[rseq_scratch0]\n\t",
+ error2)
+#endif
+ "8:\n\t"
+ : /* gcc asm goto does not allow outputs */
+ : [cpu_id] "r" (cpu),
+ [current_cpu_id] "m" (__rseq_abi.cpu_id),
+ [rseq_cs] "m" (__rseq_abi.rseq_cs),
+ /* final store input */
+ [v] "m" (*v),
+ [expect] "r" (expect),
+ [newv] "r" (newv),
+ /* try memcpy input */
+ [dst] "r" (dst),
+ [src] "r" (src),
+ [len] "r" (len),
+ [rseq_scratch0] "m" (rseq_scratch[0]),
+ [rseq_scratch1] "m" (rseq_scratch[1]),
+ [rseq_scratch2] "m" (rseq_scratch[2])
+ RSEQ_INJECT_INPUT
+ : "$4", "memory"
+ RSEQ_INJECT_CLOBBER
+ : abort, cmpfail
+#ifdef RSEQ_COMPARE_TWICE
+ , error1, error2
+#endif
+ );
+ rseq_workaround_gcc_asm_size_guess();
+ return 0;
+abort:
+ rseq_workaround_gcc_asm_size_guess();
+ RSEQ_INJECT_FAILED
+ return -1;
+cmpfail:
+ rseq_workaround_gcc_asm_size_guess();
+ return 1;
+#ifdef RSEQ_COMPARE_TWICE
+error1:
+ rseq_workaround_gcc_asm_size_guess();
+ rseq_bug("cpu_id comparison failed");
+error2:
+ rseq_workaround_gcc_asm_size_guess();
+ rseq_bug("expected value comparison failed");
+#endif
+}
+
+#endif /* !RSEQ_SKIP_FASTPATH */
diff --git a/tools/testing/selftests/rseq/rseq.h b/tools/testing/selftests/rseq/rseq.h
index 0a808575cbc4..a4684112676c 100644
--- a/tools/testing/selftests/rseq/rseq.h
+++ b/tools/testing/selftests/rseq/rseq.h
@@ -73,6 +73,8 @@ extern __thread volatile struct rseq __rseq_abi;
#include <rseq-arm.h>
#elif defined(__PPC__)
#include <rseq-ppc.h>
+#elif defined(__mips__)
+#include <rseq-mips.h>
#else
#error unsupported target
#endif
--
2.17.1
Wire up the restartable sequences (rseq) syscall for MIPS. This was
introduced by commit d7822b1e24f2 ("rseq: Introduce restartable
sequences system call") & MIPS now supports the prerequisites.
Signed-off-by: Paul Burton <[email protected]>
Cc: James Hogan <[email protected]>
Cc: Ralf Baechle <[email protected]>
Cc: Mathieu Desnoyers <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Paul E. McKenney <[email protected]>
Cc: Boqun Feng <[email protected]>
Cc: [email protected]
Cc: [email protected]
---
arch/mips/include/uapi/asm/unistd.h | 15 +++++++++------
arch/mips/kernel/scall32-o32.S | 1 +
arch/mips/kernel/scall64-64.S | 1 +
arch/mips/kernel/scall64-n32.S | 1 +
arch/mips/kernel/scall64-o32.S | 1 +
5 files changed, 13 insertions(+), 6 deletions(-)
diff --git a/arch/mips/include/uapi/asm/unistd.h b/arch/mips/include/uapi/asm/unistd.h
index bb05e9916a5f..170bf0b5b250 100644
--- a/arch/mips/include/uapi/asm/unistd.h
+++ b/arch/mips/include/uapi/asm/unistd.h
@@ -388,17 +388,18 @@
#define __NR_pkey_alloc (__NR_Linux + 364)
#define __NR_pkey_free (__NR_Linux + 365)
#define __NR_statx (__NR_Linux + 366)
+#define __NR_rseq (__NR_Linux + 367)
/*
* Offset of the last Linux o32 flavoured syscall
*/
-#define __NR_Linux_syscalls 366
+#define __NR_Linux_syscalls 367
#endif /* _MIPS_SIM == _MIPS_SIM_ABI32 */
#define __NR_O32_Linux 4000
-#define __NR_O32_Linux_syscalls 366
+#define __NR_O32_Linux_syscalls 367
#if _MIPS_SIM == _MIPS_SIM_ABI64
@@ -733,16 +734,17 @@
#define __NR_pkey_alloc (__NR_Linux + 324)
#define __NR_pkey_free (__NR_Linux + 325)
#define __NR_statx (__NR_Linux + 326)
+#define __NR_rseq (__NR_Linux + 327)
/*
* Offset of the last Linux 64-bit flavoured syscall
*/
-#define __NR_Linux_syscalls 326
+#define __NR_Linux_syscalls 327
#endif /* _MIPS_SIM == _MIPS_SIM_ABI64 */
#define __NR_64_Linux 5000
-#define __NR_64_Linux_syscalls 326
+#define __NR_64_Linux_syscalls 327
#if _MIPS_SIM == _MIPS_SIM_NABI32
@@ -1081,15 +1083,16 @@
#define __NR_pkey_alloc (__NR_Linux + 328)
#define __NR_pkey_free (__NR_Linux + 329)
#define __NR_statx (__NR_Linux + 330)
+#define __NR_rseq (__NR_Linux + 331)
/*
* Offset of the last N32 flavoured syscall
*/
-#define __NR_Linux_syscalls 330
+#define __NR_Linux_syscalls 331
#endif /* _MIPS_SIM == _MIPS_SIM_NABI32 */
#define __NR_N32_Linux 6000
-#define __NR_N32_Linux_syscalls 330
+#define __NR_N32_Linux_syscalls 331
#endif /* _UAPI_ASM_UNISTD_H */
diff --git a/arch/mips/kernel/scall32-o32.S b/arch/mips/kernel/scall32-o32.S
index a9a7d78803cd..842ff1612893 100644
--- a/arch/mips/kernel/scall32-o32.S
+++ b/arch/mips/kernel/scall32-o32.S
@@ -590,3 +590,4 @@ EXPORT(sys_call_table)
PTR sys_pkey_alloc
PTR sys_pkey_free /* 4365 */
PTR sys_statx
+ PTR sys_rseq
diff --git a/arch/mips/kernel/scall64-64.S b/arch/mips/kernel/scall64-64.S
index 65d5aeeb9bdb..558830d1e5ba 100644
--- a/arch/mips/kernel/scall64-64.S
+++ b/arch/mips/kernel/scall64-64.S
@@ -439,4 +439,5 @@ EXPORT(sys_call_table)
PTR sys_pkey_alloc
PTR sys_pkey_free /* 5325 */
PTR sys_statx
+ PTR sys_rseq
.size sys_call_table,.-sys_call_table
diff --git a/arch/mips/kernel/scall64-n32.S b/arch/mips/kernel/scall64-n32.S
index cbf190ef9e8a..293f0b0119f3 100644
--- a/arch/mips/kernel/scall64-n32.S
+++ b/arch/mips/kernel/scall64-n32.S
@@ -434,4 +434,5 @@ EXPORT(sysn32_call_table)
PTR sys_pkey_alloc
PTR sys_pkey_free
PTR sys_statx /* 6330 */
+ PTR sys_rseq
.size sysn32_call_table,.-sysn32_call_table
diff --git a/arch/mips/kernel/scall64-o32.S b/arch/mips/kernel/scall64-o32.S
index 9ebe3e2403b1..f13a08de8078 100644
--- a/arch/mips/kernel/scall64-o32.S
+++ b/arch/mips/kernel/scall64-o32.S
@@ -583,4 +583,5 @@ EXPORT(sys32_call_table)
PTR sys_pkey_alloc
PTR sys_pkey_free /* 4365 */
PTR sys_statx
+ PTR sys_rseq
.size sys32_call_table,.-sys32_call_table
--
2.17.1
On Thu, Jun 14, 2018 at 04:52:10PM -0700, Paul Burton wrote:
> +#define __RSEQ_ASM_DEFINE_TABLE(version, flags, start_ip, \
Nit: technically all these \'s are on 81st column...
> +#define __RSEQ_ASM_DEFINE_ABORT(table_label, label, teardown, \
> + abort_label, version, flags, \
> + start_ip, post_commit_offset, abort_ip) \
> + ".balign 32\n\t" \
ARM doesn't do this for DEFINE_ABORT. Is it intentional that we do for
MIPS?
Otherwise this whole series looks reasonable to me, so feel free to add
my rb on the whole series if you do apply youself:
Reviewed-by: James Hogan <[email protected]>
Thanks
James
----- On Jun 15, 2018, at 6:58 AM, James Hogan [email protected] wrote:
> On Thu, Jun 14, 2018 at 04:52:10PM -0700, Paul Burton wrote:
>> +#define __RSEQ_ASM_DEFINE_TABLE(version, flags, start_ip, \
>
> Nit: technically all these \'s are on 81st column...
>
>> +#define __RSEQ_ASM_DEFINE_ABORT(table_label, label, teardown, \
>> + abort_label, version, flags, \
>> + start_ip, post_commit_offset, abort_ip) \
>> + ".balign 32\n\t" \
>
> ARM doesn't do this for DEFINE_ABORT. Is it intentional that we do for
> MIPS?
Given that include/uapi/linux/rseq.h declares struct rseq_cs as
__attribute__((aligned(4 * sizeof(__u64)))), and considering this
comment:
/*
* struct rseq_cs is aligned on 4 * 8 bytes to ensure it is always
* contained within a single cache-line. It is usually declared as
* link-time constant data.
*/
The .balign 32 is the right thing to do here. I will add a .balign 32
to ARM selftests code as well.
Thanks,
Mathieu
>
> Thanks
> James
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
----- On Jun 14, 2018, at 7:52 PM, Paul Burton [email protected] wrote:
> Syscalls are not allowed inside restartable sequences, so add a call to
> rseq_syscall() at the very beginning of the system call exit path when
> CONFIG_DEBUG_RSEQ=y. This will help us to detect whether there is a
> syscall issued erroneously inside a restartable sequence.
>
> Signed-off-by: Paul Burton <[email protected]>
> Cc: James Hogan <[email protected]>
> Cc: Ralf Baechle <[email protected]>
> Cc: Mathieu Desnoyers <[email protected]>
> Cc: Peter Zijlstra <[email protected]>
> Cc: Paul E. McKenney <[email protected]>
> Cc: Boqun Feng <[email protected]>
> Cc: [email protected]
> Cc: [email protected]
> ---
>
> arch/mips/kernel/entry.S | 8 ++++++++
> 1 file changed, 8 insertions(+)
>
> diff --git a/arch/mips/kernel/entry.S b/arch/mips/kernel/entry.S
> index 38a302919e6b..d7de8adcfcc8 100644
> --- a/arch/mips/kernel/entry.S
> +++ b/arch/mips/kernel/entry.S
> @@ -79,6 +79,10 @@ FEXPORT(ret_from_fork)
> jal schedule_tail # a0 = struct task_struct *prev
>
> FEXPORT(syscall_exit)
> +#ifdef CONFIG_DEBUG_RSEQ
> + move a0, sp
> + jal rseq_syscall
> +#endif
> local_irq_disable # make sure need_resched and
> # signals dont change between
> # sampling and return
> @@ -141,6 +145,10 @@ work_notifysig: # deal with pending signals and
> j resume_userspace_check
>
> FEXPORT(syscall_exit_partial)
> +#ifdef CONFIG_DEBUG_RSEQ
> + move a0, sp
> + jal rseq_syscall
> +#endif
> local_irq_disable # make sure need_resched doesn't
> # change between and return
> LONG_L a2, TI_FLAGS($28) # current->work
Just to double-check: you did test with CONFIG_DEBUG_RSEQ=y, right ?
Are there any live registers that need to be saved before calling
rseq_syscall ?
Thanks,
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
On Fri, Jun 15, 2018 at 11:58:10AM +0100, James Hogan wrote:
> On Thu, Jun 14, 2018 at 04:52:10PM -0700, Paul Burton wrote:
> > +#define __RSEQ_ASM_DEFINE_TABLE(version, flags, start_ip, \
>
> Nit: technically all these \'s are on 81st column...
True... I'll replace the runs of tabs with a single space.
> > +#define __RSEQ_ASM_DEFINE_ABORT(table_label, label, teardown, \
> > + abort_label, version, flags, \
> > + start_ip, post_commit_offset, abort_ip) \
> > + ".balign 32\n\t" \
>
> ARM doesn't do this for DEFINE_ABORT. Is it intentional that we do for
> MIPS?
We need to align this structure at least in the MIPS64 case because the
.dword directive seems to lead to extra padding if we don't, and that
messes up the offsets of the fields.
For example here's an extract from basic_percpu_ops_test built for
MIPS64r6el without the .balign directive, starting from the
RSEQ_ASM_STORE_RSEQ_CS macro in rseq_cmpeqv_storev():
120001298: df848068 ld a0,-32664(gp)
12000129c: fc640008 sd a0,8(v1)
1200012a0: 8c640004 lw a0,4(v1)
1200012a4: 14820011 bne a0,v0,1200012ec <.L4^B1>
1200012a8: 00000000 nop
1200012ac: dca40000 ld a0,0(a1)
1200012b0: 14870015 bne a0,a3,120001308 <.L5>
1200012b4: 00000000 nop
1200012b8: fca60000 sd a2,0(a1)
1200012bc: 1000000d b 1200012f4 <.L5^B1>
1200012c0: 00000000 nop
1200012c4: 00000000 nop
1200012c8: 00000000 nop
1200012cc: 00000000 nop
1200012d0: 200012a0 bovc zero,zero,120005d54 <__FRAME_END__+0x3e28>
1200012d4: 00000001 0x1
1200012d8: 0000001c 0x1c
1200012dc: 00000000 nop
1200012e0: 200012ec bovc zero,zero,120005e94 <__FRAME_END__+0x3f68>
1200012e4: 00000001 0x1
1200012e8: 53053053 0x53053053
...
120012118: 200012c4 bovc zero,zero,120016c2c <_end+0x49bc>
12001211c: 00000001 0x1
And _gp, which the gp register contains the address of:
$ nm tools/testing/selftests/rseq/basic_percpu_ops_test | grep gp
000000012001a0b0 d _gp
The ld instruction @120001298 is what the "dla $4, 3f" expanded to, so
it's loading the address of the table which we're going to write to
rseq_cs. It loads from gp-32664, ie. 0x12001a0b0-32664, ie. 0x120012118,
so the table address loaded is 0x1200012c4.
If we take that as the start of the struct rseq_cs then we get:
1200012c4: __u32 version = 0x0
1200012c8: __u32 flags = 0x0
1200012cc: __u64 start_ip = 200012a000000000
Where start_ip is both not naturally aligned, so might be slow to access
or involve T&E, and more importantly doesn't have the intended value.
What happened is that gas inserted 4 bytes of padding at 1200012cc to
naturally align the .dword directive for start_ip, and that throws us
off.
Using the .balign directive avoids this, and I went with 32 bytes
because that's what struct rseq_cs is declared with in linux/rseq.h.
The ARM code has probably gotten away with it because it's 32 bit only,
so isn't emitting any 64 bit values (though if it did I don't know what
an ARM64 assembler would do with regards to alignment & padding anyway).
> Otherwise this whole series looks reasonable to me, so feel free to add
> my rb on the whole series if you do apply youself:
>
> Reviewed-by: James Hogan <[email protected]>
Thanks James :)
Paul
Hi Mathieu,
On Fri, Jun 15, 2018 at 01:41:25PM -0400, Mathieu Desnoyers wrote:
> > diff --git a/arch/mips/kernel/entry.S b/arch/mips/kernel/entry.S
> > index 38a302919e6b..d7de8adcfcc8 100644
> > --- a/arch/mips/kernel/entry.S
> > +++ b/arch/mips/kernel/entry.S
> > @@ -79,6 +79,10 @@ FEXPORT(ret_from_fork)
> > jal schedule_tail # a0 = struct task_struct *prev
> >
> > FEXPORT(syscall_exit)
> > +#ifdef CONFIG_DEBUG_RSEQ
> > + move a0, sp
> > + jal rseq_syscall
> > +#endif
> > local_irq_disable # make sure need_resched and
> > # signals dont change between
> > # sampling and return
> > @@ -141,6 +145,10 @@ work_notifysig: # deal with pending signals and
> > j resume_userspace_check
> >
> > FEXPORT(syscall_exit_partial)
> > +#ifdef CONFIG_DEBUG_RSEQ
> > + move a0, sp
> > + jal rseq_syscall
> > +#endif
> > local_irq_disable # make sure need_resched doesn't
> > # change between and return
> > LONG_L a2, TI_FLAGS($28) # current->work
>
> Just to double-check: you did test with CONFIG_DEBUG_RSEQ=y, right ?
Yes, I did. Although I only ran the selftests, which I don't believe
would actually trigger the SIGSEGV condition.
Side-note: maybe it'd be useful to have a test that does intentionally
perform a syscall within a restartable sequence & checks that it
actually receives a SIGSEGV?.
> Are there any live registers that need to be saved before calling
> rseq_syscall ?
No - we just need gp/$28 & sp/$29, and the calling convention means
rseq_syscall() should return with those unmodified. Everything else that
we or userland care about is about to be loaded from the stack anyway.
Thanks,
Paul
----- On Jun 14, 2018, at 7:52 PM, Paul Burton [email protected] wrote:
> This series implements MIPS support for restartable sequences, hooks up
> the rseq syscall & implements MIPS support in the rseq selftests.
>
> Applies atop Linus' master as of 2837461dbe6f ("Merge tag 'scsi-fixes'
> of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi").
>
For the entire series:
Reviewed-by: Mathieu Desnoyers <[email protected]>
Unless you ask otherwise, I will expect this patchset to be picked
up through the MIPS maintainer tree.
Thanks!
Mathieu
> Thanks,
> Paul
>
> Paul Burton (4):
> MIPS: Add support for restartable sequences
> MIPS: Add syscall detection for restartable sequences
> MIPS: Wire up the restartable sequences (rseq) syscall
> rseq/selftests: Implement MIPS support
>
> arch/mips/Kconfig | 1 +
> arch/mips/include/uapi/asm/unistd.h | 15 +-
> arch/mips/kernel/entry.S | 8 +
> arch/mips/kernel/scall32-o32.S | 1 +
> arch/mips/kernel/scall64-64.S | 1 +
> arch/mips/kernel/scall64-n32.S | 1 +
> arch/mips/kernel/scall64-o32.S | 1 +
> arch/mips/kernel/signal.c | 3 +
> tools/testing/selftests/rseq/param_test.c | 24 +
> tools/testing/selftests/rseq/rseq-mips.h | 725 ++++++++++++++++++++++
> tools/testing/selftests/rseq/rseq.h | 2 +
> 11 files changed, 776 insertions(+), 6 deletions(-)
> create mode 100644 tools/testing/selftests/rseq/rseq-mips.h
>
> --
> 2.17.1
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
----- On Jun 15, 2018, at 2:43 PM, Paul Burton [email protected] wrote:
> Hi Mathieu,
>
> On Fri, Jun 15, 2018 at 01:41:25PM -0400, Mathieu Desnoyers wrote:
>> > diff --git a/arch/mips/kernel/entry.S b/arch/mips/kernel/entry.S
>> > index 38a302919e6b..d7de8adcfcc8 100644
>> > --- a/arch/mips/kernel/entry.S
>> > +++ b/arch/mips/kernel/entry.S
>> > @@ -79,6 +79,10 @@ FEXPORT(ret_from_fork)
>> > jal schedule_tail # a0 = struct task_struct *prev
>> >
>> > FEXPORT(syscall_exit)
>> > +#ifdef CONFIG_DEBUG_RSEQ
>> > + move a0, sp
>> > + jal rseq_syscall
>> > +#endif
>> > local_irq_disable # make sure need_resched and
>> > # signals dont change between
>> > # sampling and return
>> > @@ -141,6 +145,10 @@ work_notifysig: # deal with pending signals and
>> > j resume_userspace_check
>> >
>> > FEXPORT(syscall_exit_partial)
>> > +#ifdef CONFIG_DEBUG_RSEQ
>> > + move a0, sp
>> > + jal rseq_syscall
>> > +#endif
>> > local_irq_disable # make sure need_resched doesn't
>> > # change between and return
>> > LONG_L a2, TI_FLAGS($28) # current->work
>>
>> Just to double-check: you did test with CONFIG_DEBUG_RSEQ=y, right ?
>
> Yes, I did. Although I only ran the selftests, which I don't believe
> would actually trigger the SIGSEGV condition.
Yeah, I typically hand-craft a critical section that generate a
system call in order to trigger this. It's hackish however, and
only triggers the SIGSEGV on kernels with CONFIG_DEBUG_RSEQ=y.
>
> Side-note: maybe it'd be useful to have a test that does intentionally
> perform a syscall within a restartable sequence & checks that it
> actually receives a SIGSEGV?.
We'd have to craft asm code for each architecture issuing a system
call in a rseq c.s. to test this. And we'd need to make this test
only runs on a kernel with CONFIG_DEBUG_RSEQ=y.
I'm not convinced yet it's worth the effort to cleanly integrate this
in selftests, but I'm very open to the idea.
>
>> Are there any live registers that need to be saved before calling
>> rseq_syscall ?
>
> No - we just need gp/$28 & sp/$29, and the calling convention means
> rseq_syscall() should return with those unmodified. Everything else that
> we or userland care about is about to be loaded from the stack anyway.
Sounds good!
Thanks,
Mathieu
>
> Thanks,
> Paul
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com