Received: by 2002:a05:6a11:4021:0:0:0:0 with SMTP id ky33csp56598pxb; Fri, 17 Sep 2021 18:40:34 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxKotmHw/UaiujjEyueXQIebNRq01kM7L9J/RjE3GQtApl7W7xliPIYlei1HOc/MLBd8mz3 X-Received: by 2002:a92:b301:: with SMTP id p1mr9650061ilh.10.1631929234073; Fri, 17 Sep 2021 18:40:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1631929234; cv=none; d=google.com; s=arc-20160816; b=RP418eCefqsJzYWwdeYjNN61IX5k25lxPvOhQ8pSyeD/4PiqxT2msCICuRVhR8nD82 fYkw5hU9oqSnaBomtIJxW3uvTWvxqw+hpChFarwD1TTUJzcfV1yUa4nXMmdU+HXwbXi7 qkH5o5+3zCg2oGje9ZnFDOuoC8Rgv4A5v1+w/+tXqtmnb5BhU1aCrPs6la5lqLZMTuXC 6U4Ui+HL4kBlzXO6iMh6sA5o24p3l00ZWps5O1Fc50ONqq1A+L8tya3O/jnrmDU9tVU+ FchlvDjHro4s5kt1DyVd7P3uWR85c0KD8DoGm6MfvHz3Igh1o0OgZElC2aqT9DPfvA8k 7MOg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=JLdxpR7e5qoomVVKT+7/RLfFQA2fUy9OidkDkwgIHmk=; b=0uzFWlVM12/1iLmjM3u9Y+f9tTWprwiU63/MgsegSvISVFZ7tx/taqDU9h6fVFQ3Oj Sh5Z2t6c1fV8aKvpsCNGcAjhv/c8aB1Zg/r6KIhIm2Fz6CRbyN/5L8NAooPOlsSekpb7 jZB2YxpsAGRni/VFfAqray7zf1XrSvtyWaN7PrltmY+IuxXMlV5yvYtU8sTqZ4lzHy3l +u9+3ezq4X+6Vh+MNfPUGzuvpTXfleV/MqRMfBY7earMLel7wc55Y0sdeKsBYpoICH/E 4ouM61K8Qqi/WsB+g6Lt5YprXzSZAYIHt5/XGz/wXv7g3kpQOUbnQg5quG0f1KP71JrL cslg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@posk.io header.s=google header.b=XHGPR+ps; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id a13si4748860ilv.48.2021.09.17.18.40.22; Fri, 17 Sep 2021 18:40:34 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@posk.io header.s=google header.b=XHGPR+ps; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245373AbhIQSE5 (ORCPT + 99 others); Fri, 17 Sep 2021 14:04:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58016 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245253AbhIQSEy (ORCPT ); Fri, 17 Sep 2021 14:04:54 -0400 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D13EC061764 for ; Fri, 17 Sep 2021 11:03:32 -0700 (PDT) Received: by mail-pj1-x102e.google.com with SMTP id i19so7475955pjv.4 for ; Fri, 17 Sep 2021 11:03:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=posk.io; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JLdxpR7e5qoomVVKT+7/RLfFQA2fUy9OidkDkwgIHmk=; b=XHGPR+psb3orXsp1XhzHUqgxgnw5bmQa8OGk/mz1eTAn/7DK8P2h4ys9T/F7Ik03TE cL2MLH2nvE1iyrTN1vIgk/dVvDYcw7ygCPRJqoI1WHYGbxurQEYzI8IpS43DyicIgGuY L9gaXKCXsRVsdISYq5SPrzaAfKNTDjH5C4Q+7oIx1c/p3lTXwpftsSRM2624cBZP1TfU zRW1+ov1/0whShXsRTDnpggm0f+VSNyq45+6TZbEnlJb/Ym5GLgVx+zFrygOCYFtTtD8 4NC+tf0yZkiCQw/qrQyT9jYUlrxWpSzYnI2p2S2t5Rj0Wvauq46EzEOFBywlzCX8Sp/f CMCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JLdxpR7e5qoomVVKT+7/RLfFQA2fUy9OidkDkwgIHmk=; b=K2c+bQ/3DxXO6tWJVIhNL4vOlehKFrKw5BAq5gGKlppaEo2Uncsm2aq6mcg0Isaf9q 53t/b44wEIU3dyky4pvuxzJ3XhvRfkJ24baWJf635kL/Mm0Hovjyl9dbXDZt8kPYnw3L J6Tn58c9aixM+BunKrq4CIwOnNqDdcJh+DB3sbXNAz+HXN719hhwkUgXjG/Fpa6ENC9o INdYMQ2antmU8CV12ccn5kG4wnQtTE9wbLnhN2i1yRx9tY9dz60wGCg6eWzLcjvT67sI aqXl9DHR+/58uMep7gMHMrrbS2mhuzz+5LgxRmPad1LDFsCqaStsuuTqQ2kMmgAOEoty 52Jg== X-Gm-Message-State: AOAM532FndMYkwizJTrnsh0ExA85WnqpV4OsSZJnDuY+EBx/vPXbgDrm /xbJaMBjjWEnCPbxbsm6so40NP/RyQoAUw== X-Received: by 2002:a17:90a:940d:: with SMTP id r13mr13916717pjo.63.1631901811628; Fri, 17 Sep 2021 11:03:31 -0700 (PDT) Received: from posk-g1.lan (23-118-52-46.lightspeed.sntcca.sbcglobal.net. [23.118.52.46]) by smtp.gmail.com with ESMTPSA id b24sm5781773pfi.205.2021.09.17.11.03.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Sep 2021 11:03:31 -0700 (PDT) From: Peter Oskolkov X-Google-Original-From: Peter Oskolkov To: Peter Zijlstra , Ingo Molnar , Thomas Gleixner , linux-kernel@vger.kernel.org, linux-api@vger.kernel.org Cc: Paul Turner , Ben Segall , Peter Oskolkov , Peter Oskolkov , Andrei Vagin , Jann Horn , Thierry Delisle Subject: [PATCH 2/5 v0.6] sched/umcg: RFC: add userspace atomic helpers Date: Fri, 17 Sep 2021 11:03:20 -0700 Message-Id: <20210917180323.278250-3-posk@google.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210917180323.278250-1-posk@google.com> References: <20210917180323.278250-1-posk@google.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Add helper functions to work atomically with userspace 32/64 bit values - there are some .*futex.* named helpers, but they are not exactly what is needed for UMCG; I haven't found what else I could use, so I rolled these. At the moment only X86_64 is supported. Note: the helpers should probably go into arch/ somewhere; I have them in kernel/sched/umcg_uaccess.h temporarily for convenience. Please let me know where I should put them. Changelog: v0.5->v0.6: - replaced mmap_read_lock with mmap_read_lock_killable in fix_pagefault(); - fix_pagefault now validates proper uaddr alignment; - renamed umcg.h to umcg_uaccess.h; v0.4->v0.5: - added xchg_user_** helpers; v0.3->v0.4: - added put_user_nosleep; - removed linked list/stack operations patch; v0.2->v0.3: - renamed and refactored the helpers a bit, as described above; - moved linked list/stack operations into a separate patch. Signed-off-by: Peter Oskolkov --- kernel/sched/umcg_uaccess.h | 344 ++++++++++++++++++++++++++++++++++++ 1 file changed, 344 insertions(+) create mode 100644 kernel/sched/umcg_uaccess.h diff --git a/kernel/sched/umcg_uaccess.h b/kernel/sched/umcg_uaccess.h new file mode 100644 index 000000000000..e4ead8d2fd62 --- /dev/null +++ b/kernel/sched/umcg_uaccess.h @@ -0,0 +1,344 @@ +/* SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note */ +#ifndef _KERNEL_SCHED_UMCG_UACCESS_H +#define _KERNEL_SCHED_UMCG_UACCESS_H + +#ifdef CONFIG_X86_64 + +#include + +#include +#include +#include + +/* TODO: move atomic operations below into arch/ headers */ +static inline int __try_cmpxchg_user_32(u32 *uval, u32 __user *uaddr, + u32 oldval, u32 newval) +{ + int ret = 0; + + asm volatile("\n" + "1:\t" LOCK_PREFIX "cmpxchgl %4, %2\n" + "2:\n" + "\t.section .fixup, \"ax\"\n" + "3:\tmov %3, %0\n" + "\tjmp 2b\n" + "\t.previous\n" + _ASM_EXTABLE_UA(1b, 3b) + : "+r" (ret), "=a" (oldval), "+m" (*uaddr) + : "i" (-EFAULT), "r" (newval), "1" (oldval) + : "memory" + ); + *uval = oldval; + return ret; +} + +static inline int __try_cmpxchg_user_64(u64 *uval, u64 __user *uaddr, + u64 oldval, u64 newval) +{ + int ret = 0; + + asm volatile("\n" + "1:\t" LOCK_PREFIX "cmpxchgq %4, %2\n" + "2:\n" + "\t.section .fixup, \"ax\"\n" + "3:\tmov %3, %0\n" + "\tjmp 2b\n" + "\t.previous\n" + _ASM_EXTABLE_UA(1b, 3b) + : "+r" (ret), "=a" (oldval), "+m" (*uaddr) + : "i" (-EFAULT), "r" (newval), "1" (oldval) + : "memory" + ); + *uval = oldval; + return ret; +} + +static inline int fix_pagefault(unsigned long uaddr, bool write_fault, int bytes) +{ + struct mm_struct *mm = current->mm; + int ret; + + /* Validate proper alignment. */ + if (uaddr % bytes) + return -EINVAL; + + if (mmap_read_lock_killable(mm)) + return -EINTR; + ret = fixup_user_fault(mm, uaddr, write_fault ? FAULT_FLAG_WRITE : 0, + NULL); + mmap_read_unlock(mm); + + return ret < 0 ? ret : 0; +} + +/** + * cmpxchg_32_user_nosleep - compare_exchange 32-bit values + * + * Return: + * 0 - OK + * -EFAULT: memory access error + * -EAGAIN: @expected did not match; consult @prev + */ +static inline int cmpxchg_user_32_nosleep(u32 __user *uaddr, u32 *old, u32 new) +{ + int ret = -EFAULT; + u32 __old = *old; + + if (unlikely(!access_ok(uaddr, sizeof(*uaddr)))) + return -EFAULT; + + pagefault_disable(); + + __uaccess_begin_nospec(); + ret = __try_cmpxchg_user_32(old, uaddr, __old, new); + user_access_end(); + + if (!ret) + ret = *old == __old ? 0 : -EAGAIN; + + pagefault_enable(); + return ret; +} + +/** + * cmpxchg_64_user_nosleep - compare_exchange 64-bit values + * + * Return: + * 0 - OK + * -EFAULT: memory access error + * -EAGAIN: @expected did not match; consult @prev + */ +static inline int cmpxchg_user_64_nosleep(u64 __user *uaddr, u64 *old, u64 new) +{ + int ret = -EFAULT; + u64 __old = *old; + + if (unlikely(!access_ok(uaddr, sizeof(*uaddr)))) + return -EFAULT; + + pagefault_disable(); + + __uaccess_begin_nospec(); + ret = __try_cmpxchg_user_64(old, uaddr, __old, new); + user_access_end(); + + if (!ret) + ret = *old == __old ? 0 : -EAGAIN; + + pagefault_enable(); + + return ret; +} + +/** + * cmpxchg_32_user - compare_exchange 32-bit values + * + * Return: + * 0 - OK + * -EFAULT: memory access error + * -EAGAIN: @expected did not match; consult @prev + */ +static inline int cmpxchg_user_32(u32 __user *uaddr, u32 *old, u32 new) +{ + int ret = -EFAULT; + u32 __old = *old; + + if (unlikely(!access_ok(uaddr, sizeof(*uaddr)))) + return -EFAULT; + + pagefault_disable(); + + while (true) { + __uaccess_begin_nospec(); + ret = __try_cmpxchg_user_32(old, uaddr, __old, new); + user_access_end(); + + if (!ret) { + ret = *old == __old ? 0 : -EAGAIN; + break; + } + + if (fix_pagefault((unsigned long)uaddr, true, sizeof(*uaddr)) < 0) + break; + } + + pagefault_enable(); + return ret; +} + +/** + * cmpxchg_64_user - compare_exchange 64-bit values + * + * Return: + * 0 - OK + * -EFAULT: memory access error + * -EAGAIN: @expected did not match; consult @prev + */ +static inline int cmpxchg_user_64(u64 __user *uaddr, u64 *old, u64 new) +{ + int ret = -EFAULT; + u64 __old = *old; + + if (unlikely(!access_ok(uaddr, sizeof(*uaddr)))) + return -EFAULT; + + pagefault_disable(); + + while (true) { + __uaccess_begin_nospec(); + ret = __try_cmpxchg_user_64(old, uaddr, __old, new); + user_access_end(); + + if (!ret) { + ret = *old == __old ? 0 : -EAGAIN; + break; + } + + if (fix_pagefault((unsigned long)uaddr, true, sizeof(*uaddr)) < 0) + break; + } + + pagefault_enable(); + + return ret; +} + +static inline int __try_xchg_user_32(u32 *oval, u32 __user *uaddr, u32 newval) +{ + u32 oldval = 0; + int ret = 0; + + asm volatile("\n" + "1:\txchgl %0, %2\n" + "2:\n" + "\t.section .fixup, \"ax\"\n" + "3:\tmov %3, %0\n" + "\tjmp 2b\n" + "\t.previous\n" + _ASM_EXTABLE_UA(1b, 3b) + : "=r" (oldval), "=r" (ret), "+m" (*uaddr) + : "i" (-EFAULT), "0" (newval), "1" (0) + ); + + if (ret) + return ret; + + *oval = oldval; + return 0; +} + +static inline int __try_xchg_user_64(u64 *oval, u64 __user *uaddr, u64 newval) +{ + u64 oldval = 0; + int ret = 0; + + asm volatile("\n" + "1:\txchgq %0, %2\n" + "2:\n" + "\t.section .fixup, \"ax\"\n" + "3:\tmov %3, %0\n" + "\tjmp 2b\n" + "\t.previous\n" + _ASM_EXTABLE_UA(1b, 3b) + : "=r" (oldval), "=r" (ret), "+m" (*uaddr) + : "i" (-EFAULT), "0" (newval), "1" (0) + ); + + if (ret) + return ret; + + *oval = oldval; + return 0; +} + +/** + * xchg_32_user - atomically exchange 64-bit values + * + * Return: + * 0 - OK + * -EFAULT: memory access error + */ +static inline int xchg_user_32(u32 __user *uaddr, u32 *val) +{ + int ret = -EFAULT; + + if (unlikely(!access_ok(uaddr, sizeof(*uaddr)))) + return -EFAULT; + + pagefault_disable(); + + while (true) { + + __uaccess_begin_nospec(); + ret = __try_xchg_user_32(val, uaddr, *val); + user_access_end(); + + if (!ret) + break; + + if (fix_pagefault((unsigned long)uaddr, true, sizeof(*uaddr)) < 0) + break; + } + + pagefault_enable(); + + return ret; +} + +/** + * xchg_64_user - atomically exchange 64-bit values + * + * Return: + * 0 - OK + * -EFAULT: memory access error + */ +static inline int xchg_user_64(u64 __user *uaddr, u64 *val) +{ + int ret = -EFAULT; + + if (unlikely(!access_ok(uaddr, sizeof(*uaddr)))) + return -EFAULT; + + pagefault_disable(); + + while (true) { + + __uaccess_begin_nospec(); + ret = __try_xchg_user_64(val, uaddr, *val); + user_access_end(); + + if (!ret) + break; + + if (fix_pagefault((unsigned long)uaddr, true, sizeof(*uaddr)) < 0) + break; + } + + pagefault_enable(); + + return ret; +} + +/** + * get_user_nosleep - get user value without sleeping. + * + * get_user() might sleep and therefore cannot be used in preempt-disabled + * regions. + */ +#define get_user_nosleep(out, uaddr) \ +({ \ + int ret = -EFAULT; \ + \ + if (access_ok((uaddr), sizeof(*(uaddr)))) { \ + pagefault_disable(); \ + \ + if (!__get_user((out), (uaddr))) \ + ret = 0; \ + \ + pagefault_enable(); \ + } \ + ret; \ +}) + +#endif /* CONFIG_X86_64 */ +#endif /* _KERNEL_SCHED_UMCG_UACCESS_H */ -- 2.25.1