Received: by 2002:ad5:4acb:0:0:0:0:0 with SMTP id n11csp3517998imw; Mon, 11 Jul 2022 10:06:51 -0700 (PDT) X-Google-Smtp-Source: AGRyM1tSa6UVNQPx6zIbP9X3xf1bv7qD9MArgKzPJfuieKB1BjmGc5l53Y42YzY8w9yByY61hYgf X-Received: by 2002:a17:906:5d08:b0:6ff:8ed:db63 with SMTP id g8-20020a1709065d0800b006ff08eddb63mr19789058ejt.408.1657559211622; Mon, 11 Jul 2022 10:06:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1657559211; cv=none; d=google.com; s=arc-20160816; b=hQKtntUbFmV9Ekvf+zu663gsJqq9CNnZepuQwm/dzhKdpOJCqMuQmxphaCJ5MLPn/w Ax2vMu8lmTV5le2z6GiD1JiZDI96kkgpum34RIR+eKhBVZ92Yk38CgDHua6Xph0ezGW8 5Jy9xWAfRYUO/kxZMQazB0hQDu0Rlbb2nr5a72XWAdlWGgP1mAtmkWArMtdVwBej6Two jnQxUYNkOYVBmoPV9kWxlxvIJUoUL+1gTEwwCs50IZgqmvfR2rZYDkeBDQZjN8/y2Il5 8k+f7bOBexKHpVv96jGjZjtduYU1KsnHbR2OAXkMecdrz025PjbfMg85uuyf5XIyOo63 yYVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:cc:to:subject :message-id:date:from:in-reply-to:references:mime-version :dkim-signature; bh=m736QbvuW1TAkCAGJYZ6QP23qiPikZXAhNz97EPWCnA=; b=kkx9FnQKC4UPwdB6u8Crr90eGY3kUQp441klPcb5qTPqCMzFmALBZfZ/7cLCN/gCYI JA8iotXMEy282qyip6F1vcoyX1vYvRxeWH9JxcE8GFzIzWzE4J9NSmKG/1oml+wV/g2y T6VCzoA0pYYnKUqwxO39AsbuQmgDVy/xQTlNvnLyFayeiMd3q0/djfnvSS/lLaqCBXXf 5pdOz8e4r9pa6YyKrpVYHLMpz4FYOktlbliJKqUtCxtugGXbU0O06Zalag2MjM2eWy9U Xnz6vNyqqFBfZbryAYo15LphIsceqAqznNN+Ste0a9RQgmpR3W0xpD1lG6IIfYIFdN7+ IOjA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=fNwvIw0Y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f24-20020aa7d858000000b0043a71f2ffc1si10013166eds.49.2022.07.11.10.06.18; Mon, 11 Jul 2022 10:06:51 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=fNwvIw0Y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229765AbiGKQuY (ORCPT + 99 others); Mon, 11 Jul 2022 12:50:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38108 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229621AbiGKQuW (ORCPT ); Mon, 11 Jul 2022 12:50:22 -0400 Received: from mail-yw1-x1132.google.com (mail-yw1-x1132.google.com [IPv6:2607:f8b0:4864:20::1132]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D9C332B96 for ; Mon, 11 Jul 2022 09:50:19 -0700 (PDT) Received: by mail-yw1-x1132.google.com with SMTP id 00721157ae682-2ef5380669cso55211027b3.9 for ; Mon, 11 Jul 2022 09:50:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=m736QbvuW1TAkCAGJYZ6QP23qiPikZXAhNz97EPWCnA=; b=fNwvIw0Y4vT9l4lfxZhl821T0VMScIgkn+Eu6UGBMesjGzbBo2YrZmzcvHS3475l00 J7dwrApc25c3SyMrFeckVnsinVaqxVSUX4GDZvDO10hHgBTiwFT3sgov5xQJBNaz6HSH twezozjpwvwh4W0tQ1ga5icjyEkcGX1yqIaX4WlMs3RFHzvwZ7Dd3VyMHqjSYFMiesH5 rJfmN7a4niO4xzyPSLCchOEWdMem9pjeJ4pet3cSQSzOi7S/FKq1Ntc+xuxfBSQT67HR X+W/7D0cGJ92I5qbF7uvme+gDBcvAmh9OxZorik/JAzisYKp08YeFTAt+A09ksk6SvPy nVKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=m736QbvuW1TAkCAGJYZ6QP23qiPikZXAhNz97EPWCnA=; b=Nv2P4Om1yHYa8Qo1PV6yUMvWW88qsk+peGTVqZh1phi4YZtVTlZVThjKrO71hDn9lh KgydyR7GbZOeHi4HVEE1aVTm8r7/Uf9nSAzG27h5Gfzuo/3Uo436op4m/D3VmWRgkTu8 XRc4d4aQkuEJrOtnGOmfC7S+mSKsbpEcBhMazn2ar3ZhHXBjx/nKjqJZNrBactfAqw0p jM6G9r7SGSzO8VF7NFCDG4+nhIhHd2ouXoX+B49R5047+kdn9TfipW7EoOS/w4bfWA1P LBU2BIo26mlQHCxY8C876rJXHAWnZVub0ecczKMPajUA/K6kgfY8By6STKm5lCNAbof7 uboQ== X-Gm-Message-State: AJIora8ORYloQnTyA/ZQSD/Ragj9OlhtclTDNNvDywVdx5R21gH5b1+5 2aEZcZY9FJ25bBAwyhRaBjXjONPdYjltUw8WFdiZUQ== X-Received: by 2002:a0d:cf07:0:b0:31d:17cb:ec11 with SMTP id r7-20020a0dcf07000000b0031d17cbec11mr20826537ywd.264.1657558217787; Mon, 11 Jul 2022 09:50:17 -0700 (PDT) MIME-Version: 1.0 References: <20220701142310.2188015-1-glider@google.com> <20220701142310.2188015-12-glider@google.com> In-Reply-To: <20220701142310.2188015-12-glider@google.com> From: Marco Elver Date: Mon, 11 Jul 2022 18:49:41 +0200 Message-ID: Subject: Re: [PATCH v4 11/45] kmsan: add KMSAN runtime core To: Alexander Potapenko Cc: Alexander Viro , Alexei Starovoitov , Andrew Morton , Andrey Konovalov , Andy Lutomirski , Arnd Bergmann , Borislav Petkov , Christoph Hellwig , Christoph Lameter , David Rientjes , Dmitry Vyukov , Eric Dumazet , Greg Kroah-Hartman , Herbert Xu , Ilya Leoshkevich , Ingo Molnar , Jens Axboe , Joonsoo Kim , Kees Cook , Mark Rutland , Matthew Wilcox , "Michael S. Tsirkin" , Pekka Enberg , Peter Zijlstra , Petr Mladek , Steven Rostedt , Thomas Gleixner , Vasily Gorbik , Vegard Nossum , Vlastimil Babka , kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-17.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL,USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, 1 Jul 2022 at 16:23, 'Alexander Potapenko' via kasan-dev wrote: > > For each memory location KernelMemorySanitizer maintains two types of > metadata: > 1. The so-called shadow of that location - =D0=B0 byte:byte mapping descr= ibing > whether or not individual bits of memory are initialized (shadow is 0) > or not (shadow is 1). > 2. The origins of that location - =D0=B0 4-byte:4-byte mapping containing > 4-byte IDs of the stack traces where uninitialized values were > created. > > Each struct page now contains pointers to two struct pages holding > KMSAN metadata (shadow and origins) for the original struct page. > Utility routines in mm/kmsan/core.c and mm/kmsan/shadow.c handle the > metadata creation, addressing, copying and checking. > mm/kmsan/report.c performs error reporting in the cases an uninitialized > value is used in a way that leads to undefined behavior. > > KMSAN compiler instrumentation is responsible for tracking the metadata > along with the kernel memory. mm/kmsan/instrumentation.c provides the > implementation for instrumentation hooks that are called from files > compiled with -fsanitize=3Dkernel-memory. > > To aid parameter passing (also done at instrumentation level), each > task_struct now contains a struct kmsan_task_state used to track the > metadata of function parameters and return values for that task. > > Finally, this patch provides CONFIG_KMSAN that enables KMSAN, and > declares CFLAGS_KMSAN, which are applied to files compiled with KMSAN. > The KMSAN_SANITIZE:=3Dn Makefile directive can be used to completely > disable KMSAN instrumentation for certain files. > > Similarly, KMSAN_ENABLE_CHECKS:=3Dn disables KMSAN checks and makes newly > created stack memory initialized. > > Users can also use functions from include/linux/kmsan-checks.h to mark > certain memory regions as uninitialized or initialized (this is called > "poisoning" and "unpoisoning") or check that a particular region is > initialized. > > Signed-off-by: Alexander Potapenko Acked-by: Marco Elver Overall looks clean and well organized now. > --- > v2: > -- as requested by Greg K-H, moved hooks for different subsystems to res= pective patches, > rewrote the patch description; > -- addressed comments by Dmitry Vyukov; > -- added a note about KMSAN being not intended for production use. > -- fix case of unaligned dst in kmsan_internal_memmove_metadata() > > v3: > -- print build IDs in reports where applicable > -- drop redundant filter_irq_stacks(), unpoison the local passed to __st= ack_depot_save() > -- remove a stray BUG() > > v4: (mostly fixes suggested by Marco Elver) > -- add missing SPDX headers > -- move CC_IS_CLANG && CLANG_VERSION under HAVE_KMSAN_COMPILER > -- replace occurrences of |var| with @var > -- reflow KMSAN_WARN_ON(), fix code comments > -- remove x86-specific code from shadow.c to improve portability > -- convert kmsan_report_lock to raw spinlock > -- add enter_runtime/exit_runtime around kmsan_internal_memmove_metadata= () > -- remove unnecessary include from kmsan.h (reported by ) > -- introduce CONFIG_KMSAN_CHECK_PARAM_RETVAL (on by default), which > maps to -fsanitize-memory-param-retval and makes KMSAN eagerly check > values passed as function parameters and returned from functions. > -- use real shadow in instrumented functions called from runtime > > Link: https://linux-review.googlesource.com/id/I9b71bfe3425466c97159f9de0= 062e5e8e4fec866 > --- > Makefile | 1 + > include/linux/kmsan-checks.h | 64 +++++ > include/linux/kmsan.h | 46 ++++ > include/linux/mm_types.h | 12 + > include/linux/sched.h | 5 + > lib/Kconfig.debug | 1 + > lib/Kconfig.kmsan | 50 ++++ > mm/Makefile | 1 + > mm/kmsan/Makefile | 23 ++ > mm/kmsan/core.c | 458 +++++++++++++++++++++++++++++++++++ > mm/kmsan/hooks.c | 66 +++++ > mm/kmsan/instrumentation.c | 271 +++++++++++++++++++++ > mm/kmsan/kmsan.h | 190 +++++++++++++++ > mm/kmsan/report.c | 211 ++++++++++++++++ > mm/kmsan/shadow.c | 147 +++++++++++ > scripts/Makefile.kmsan | 8 + > scripts/Makefile.lib | 9 + > 17 files changed, 1563 insertions(+) > create mode 100644 include/linux/kmsan-checks.h > create mode 100644 include/linux/kmsan.h > create mode 100644 lib/Kconfig.kmsan > create mode 100644 mm/kmsan/Makefile > create mode 100644 mm/kmsan/core.c > create mode 100644 mm/kmsan/hooks.c > create mode 100644 mm/kmsan/instrumentation.c > create mode 100644 mm/kmsan/kmsan.h > create mode 100644 mm/kmsan/report.c > create mode 100644 mm/kmsan/shadow.c > create mode 100644 scripts/Makefile.kmsan > > diff --git a/Makefile b/Makefile > index 8973b285ce6c7..7c93482f6df3d 100644 > --- a/Makefile > +++ b/Makefile > @@ -1014,6 +1014,7 @@ include-y :=3D scripts/Makefile.ext= rawarn > include-$(CONFIG_DEBUG_INFO) +=3D scripts/Makefile.debug > include-$(CONFIG_KASAN) +=3D scripts/Makefile.kasan > include-$(CONFIG_KCSAN) +=3D scripts/Makefile.kcsan > +include-$(CONFIG_KMSAN) +=3D scripts/Makefile.kmsan > include-$(CONFIG_UBSAN) +=3D scripts/Makefile.ubsan > include-$(CONFIG_KCOV) +=3D scripts/Makefile.kcov > include-$(CONFIG_RANDSTRUCT) +=3D scripts/Makefile.randstruct > diff --git a/include/linux/kmsan-checks.h b/include/linux/kmsan-checks.h > new file mode 100644 > index 0000000000000..a6522a0c28df9 > --- /dev/null > +++ b/include/linux/kmsan-checks.h > @@ -0,0 +1,64 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* > + * KMSAN checks to be used for one-off annotations in subsystems. > + * > + * Copyright (C) 2017-2022 Google LLC > + * Author: Alexander Potapenko > + * > + */ > + > +#ifndef _LINUX_KMSAN_CHECKS_H > +#define _LINUX_KMSAN_CHECKS_H > + > +#include > + > +#ifdef CONFIG_KMSAN > + > +/** > + * kmsan_poison_memory() - Mark the memory range as uninitialized. > + * @address: address to start with. > + * @size: size of buffer to poison. > + * @flags: GFP flags for allocations done by this function. > + * > + * Until other data is written to this range, KMSAN will treat it as > + * uninitialized. Error reports for this memory will reference the call = site of > + * kmsan_poison_memory() as origin. > + */ > +void kmsan_poison_memory(const void *address, size_t size, gfp_t flags); > + > +/** > + * kmsan_unpoison_memory() - Mark the memory range as initialized. > + * @address: address to start with. > + * @size: size of buffer to unpoison. > + * > + * Until other data is written to this range, KMSAN will treat it as > + * initialized. > + */ > +void kmsan_unpoison_memory(const void *address, size_t size); > + > +/** > + * kmsan_check_memory() - Check the memory range for being initialized. > + * @address: address to start with. > + * @size: size of buffer to check. > + * > + * If any piece of the given range is marked as uninitialized, KMSAN wil= l report > + * an error. > + */ > +void kmsan_check_memory(const void *address, size_t size); > + > +#else > + > +static inline void kmsan_poison_memory(const void *address, size_t size, > + gfp_t flags) > +{ > +} > +static inline void kmsan_unpoison_memory(const void *address, size_t siz= e) > +{ > +} > +static inline void kmsan_check_memory(const void *address, size_t size) > +{ > +} > + > +#endif > + > +#endif /* _LINUX_KMSAN_CHECKS_H */ > diff --git a/include/linux/kmsan.h b/include/linux/kmsan.h > new file mode 100644 > index 0000000000000..99e48c6b049d9 > --- /dev/null > +++ b/include/linux/kmsan.h > @@ -0,0 +1,46 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* > + * KMSAN API for subsystems. > + * > + * Copyright (C) 2017-2022 Google LLC > + * Author: Alexander Potapenko > + * > + */ > +#ifndef _LINUX_KMSAN_H > +#define _LINUX_KMSAN_H > + > +#include > +#include > +#include > +#include > + > +struct page; Either I'm not looking hard enough, but I can't see where this is used in this file. > +#ifdef CONFIG_KMSAN > + > +/* These constants are defined in the MSan LLVM instrumentation pass. */ > +#define KMSAN_RETVAL_SIZE 800 > +#define KMSAN_PARAM_SIZE 800 > + > +struct kmsan_context_state { > + char param_tls[KMSAN_PARAM_SIZE]; > + char retval_tls[KMSAN_RETVAL_SIZE]; > + char va_arg_tls[KMSAN_PARAM_SIZE]; > + char va_arg_origin_tls[KMSAN_PARAM_SIZE]; > + u64 va_arg_overflow_size_tls; > + char param_origin_tls[KMSAN_PARAM_SIZE]; > + depot_stack_handle_t retval_origin_tls; > +}; > + > +#undef KMSAN_PARAM_SIZE > +#undef KMSAN_RETVAL_SIZE > + > +struct kmsan_ctx { > + struct kmsan_context_state cstate; > + int kmsan_in_runtime; > + bool allow_reporting; > +}; > + > +#endif > + > +#endif /* _LINUX_KMSAN_H */ > diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h > index c29ab4c0cd5c6..3cc0ebdd9625f 100644 > --- a/include/linux/mm_types.h > +++ b/include/linux/mm_types.h > @@ -218,6 +218,18 @@ struct page { > not kmapped, ie. highmem) */ > #endif /* WANT_PAGE_VIRTUAL */ > > +#ifdef CONFIG_KMSAN > + /* > + * KMSAN metadata for this page: > + * - shadow page: every bit indicates whether the corresponding > + * bit of the original page is initialized (0) or not (1); > + * - origin page: every 4 bytes contain an id of the stack trace > + * where the uninitialized value was created. > + */ > + struct page *kmsan_shadow; > + struct page *kmsan_origin; > +#endif > + > #ifdef LAST_CPUPID_NOT_IN_PAGE_FLAGS > int _last_cpupid; > #endif > diff --git a/include/linux/sched.h b/include/linux/sched.h > index c46f3a63b758f..f9bb2c954e794 100644 > --- a/include/linux/sched.h > +++ b/include/linux/sched.h > @@ -14,6 +14,7 @@ > #include > #include > #include > +#include > #include > #include > #include > @@ -1353,6 +1354,10 @@ struct task_struct { > #endif > #endif > > +#ifdef CONFIG_KMSAN > + struct kmsan_ctx kmsan_ctx; > +#endif > + > #if IS_ENABLED(CONFIG_KUNIT) > struct kunit *kunit_test; > #endif > diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug > index 2e24db4bff192..59819e6fa5865 100644 > --- a/lib/Kconfig.debug > +++ b/lib/Kconfig.debug > @@ -963,6 +963,7 @@ config DEBUG_STACKOVERFLOW > > source "lib/Kconfig.kasan" > source "lib/Kconfig.kfence" > +source "lib/Kconfig.kmsan" > > endmenu # "Memory Debugging" > > diff --git a/lib/Kconfig.kmsan b/lib/Kconfig.kmsan > new file mode 100644 > index 0000000000000..8f768d4034e3c > --- /dev/null > +++ b/lib/Kconfig.kmsan > @@ -0,0 +1,50 @@ > +# SPDX-License-Identifier: GPL-2.0-only > +config HAVE_ARCH_KMSAN > + bool > + > +config HAVE_KMSAN_COMPILER > + # Clang versions <14.0.0 also support -fsanitize=3Dkernel-memory,= but not > + # all the features necessary to build the kernel with KMSAN. > + depends on CC_IS_CLANG && CLANG_VERSION >=3D 140000 > + def_bool $(cc-option,-fsanitize=3Dkernel-memory -mllvm -msan-disa= ble-checks=3D1) > + > +config HAVE_KMSAN_PARAM_RETVAL > + # Separate check for -fsanitize-memory-param-retval support. > + depends on CC_IS_CLANG && CLANG_VERSION >=3D 140000 > + def_bool $(cc-option,-fsanitize=3Dkernel-memory -fsanitize-memory= -param-retval) > + > + > +config KMSAN > + bool "KMSAN: detector of uninitialized values use" > + depends on HAVE_ARCH_KMSAN && HAVE_KMSAN_COMPILER > + depends on SLUB && DEBUG_KERNEL && !KASAN && !KCSAN > + select STACKDEPOT > + select STACKDEPOT_ALWAYS_INIT > + help > + KernelMemorySanitizer (KMSAN) is a dynamic detector of uses of > + uninitialized values in the kernel. It is based on compiler > + instrumentation provided by Clang and thus requires Clang to bu= ild. > + > + An important note is that KMSAN is not intended for production = use, > + because it drastically increases kernel memory footprint and sl= ows > + the whole system down. > + > + See for more details. > + > +if KMSAN > + > +config KMSAN_CHECK_PARAM_RETVAL > + bool "Check for uninitialized values passed to and returned from = functions" > + default HAVE_KMSAN_PARAM_RETVAL > + help > + If the compiler supports -fsanitize-memory-param-retval, KMSAN = will > + eagerly check every function parameter passed by value and ever= y > + function return value. > + > + Disabling KMSAN_CHECK_PARAM_RETVAL will result in tracking shad= ow for > + function parameters and return values across function borders. = This > + is a more relaxed mode, but it generates more instrumentation c= ode and > + may potentially report errors in corner cases when non-instrume= nted > + functions call instrumented ones. > + > +endif > diff --git a/mm/Makefile b/mm/Makefile > index 6f9ffa968a1a1..ff96830153221 100644 > --- a/mm/Makefile > +++ b/mm/Makefile > @@ -89,6 +89,7 @@ obj-$(CONFIG_SLAB) +=3D slab.o > obj-$(CONFIG_SLUB) +=3D slub.o > obj-$(CONFIG_KASAN) +=3D kasan/ > obj-$(CONFIG_KFENCE) +=3D kfence/ > +obj-$(CONFIG_KMSAN) +=3D kmsan/ > obj-$(CONFIG_FAILSLAB) +=3D failslab.o > obj-$(CONFIG_MEMTEST) +=3D memtest.o > obj-$(CONFIG_MIGRATION) +=3D migrate.o > diff --git a/mm/kmsan/Makefile b/mm/kmsan/Makefile > new file mode 100644 > index 0000000000000..550ad8625e4f9 > --- /dev/null > +++ b/mm/kmsan/Makefile > @@ -0,0 +1,23 @@ > +# SPDX-License-Identifier: GPL-2.0 > +# > +# Makefile for KernelMemorySanitizer (KMSAN). > +# > +# > +obj-y :=3D core.o instrumentation.o hooks.o report.o shadow.o > + > +KMSAN_SANITIZE :=3D n > +KCOV_INSTRUMENT :=3D n > +UBSAN_SANITIZE :=3D n > + > +# Disable instrumentation of KMSAN runtime with other tools. > +CC_FLAGS_KMSAN_RUNTIME :=3D -fno-stack-protector > +CC_FLAGS_KMSAN_RUNTIME +=3D $(call cc-option,-fno-conserve-stack) > +CC_FLAGS_KMSAN_RUNTIME +=3D -DDISABLE_BRANCH_PROFILING > + > +CFLAGS_REMOVE.o =3D $(CC_FLAGS_FTRACE) > + > +CFLAGS_core.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) > +CFLAGS_hooks.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) > +CFLAGS_instrumentation.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) > +CFLAGS_report.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) > +CFLAGS_shadow.o :=3D $(CC_FLAGS_KMSAN_RUNTIME) > diff --git a/mm/kmsan/core.c b/mm/kmsan/core.c > new file mode 100644 > index 0000000000000..16fb8880a9c6d > --- /dev/null > +++ b/mm/kmsan/core.c > @@ -0,0 +1,458 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * KMSAN runtime library. > + * > + * Copyright (C) 2017-2022 Google LLC > + * Author: Alexander Potapenko > + * > + */ > + > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > + > +#include "../slab.h" > +#include "kmsan.h" > + > +/* > + * Avoid creating too long origin chains, these are unlikely to particip= ate in > + * real reports. > + */ > +#define MAX_CHAIN_DEPTH 7 > +#define NUM_SKIPPED_TO_WARN 10000 > + > +bool kmsan_enabled __read_mostly; > + > +/* > + * Per-CPU KMSAN context to be used in interrupts, where current->kmsan = is > + * unavaliable. > + */ > +DEFINE_PER_CPU(struct kmsan_ctx, kmsan_percpu_ctx); > + > +void kmsan_internal_poison_memory(void *address, size_t size, gfp_t flag= s, > + unsigned int poison_flags) > +{ > + u32 extra_bits =3D > + kmsan_extra_bits(/*depth*/ 0, poison_flags & KMSAN_POISON= _FREE); > + bool checked =3D poison_flags & KMSAN_POISON_CHECK; > + depot_stack_handle_t handle; > + > + handle =3D kmsan_save_stack_with_flags(flags, extra_bits); > + kmsan_internal_set_shadow_origin(address, size, -1, handle, check= ed); > +} > + > +void kmsan_internal_unpoison_memory(void *address, size_t size, bool che= cked) > +{ > + kmsan_internal_set_shadow_origin(address, size, 0, 0, checked); > +} > + > +depot_stack_handle_t kmsan_save_stack_with_flags(gfp_t flags, > + unsigned int extra) > +{ > + unsigned long entries[KMSAN_STACK_DEPTH]; > + unsigned int nr_entries; > + > + nr_entries =3D stack_trace_save(entries, KMSAN_STACK_DEPTH, 0); > + > + /* Don't sleep (see might_sleep_if() in __alloc_pages_nodemask())= . */ > + flags &=3D ~__GFP_DIRECT_RECLAIM; > + > + return __stack_depot_save(entries, nr_entries, extra, flags, true= ); > +} > + > +/* Copy the metadata following the memmove() behavior. */ > +void kmsan_internal_memmove_metadata(void *dst, void *src, size_t n) > +{ > + depot_stack_handle_t old_origin =3D 0, new_origin =3D 0; > + int src_slots, dst_slots, i, iter, step, skip_bits; > + depot_stack_handle_t *origin_src, *origin_dst; > + void *shadow_src, *shadow_dst; > + u32 *align_shadow_src, shadow; > + bool backwards; > + > + shadow_dst =3D kmsan_get_metadata(dst, KMSAN_META_SHADOW); > + if (!shadow_dst) > + return; > + KMSAN_WARN_ON(!kmsan_metadata_is_contiguous(dst, n)); > + > + shadow_src =3D kmsan_get_metadata(src, KMSAN_META_SHADOW); > + if (!shadow_src) { > + /* > + * @src is untracked: zero out destination shadow, ignore= the > + * origins, we're done. > + */ > + __memset(shadow_dst, 0, n); > + return; > + } > + KMSAN_WARN_ON(!kmsan_metadata_is_contiguous(src, n)); > + > + __memmove(shadow_dst, shadow_src, n); > + > + origin_dst =3D kmsan_get_metadata(dst, KMSAN_META_ORIGIN); > + origin_src =3D kmsan_get_metadata(src, KMSAN_META_ORIGIN); > + KMSAN_WARN_ON(!origin_dst || !origin_src); > + src_slots =3D (ALIGN((u64)src + n, KMSAN_ORIGIN_SIZE) - > + ALIGN_DOWN((u64)src, KMSAN_ORIGIN_SIZE)) / > + KMSAN_ORIGIN_SIZE; > + dst_slots =3D (ALIGN((u64)dst + n, KMSAN_ORIGIN_SIZE) - > + ALIGN_DOWN((u64)dst, KMSAN_ORIGIN_SIZE)) / > + KMSAN_ORIGIN_SIZE; > + KMSAN_WARN_ON((src_slots < 1) || (dst_slots < 1)); > + KMSAN_WARN_ON((src_slots - dst_slots > 1) || > + (dst_slots - src_slots < -1)); > + > + backwards =3D dst > src; > + i =3D backwards ? min(src_slots, dst_slots) - 1 : 0; > + iter =3D backwards ? -1 : 1; > + > + align_shadow_src =3D > + (u32 *)ALIGN_DOWN((u64)shadow_src, KMSAN_ORIGIN_SIZE); > + for (step =3D 0; step < min(src_slots, dst_slots); step++, i +=3D= iter) { > + KMSAN_WARN_ON(i < 0); > + shadow =3D align_shadow_src[i]; > + if (i =3D=3D 0) { > + /* > + * If @src isn't aligned on KMSAN_ORIGIN_SIZE, do= n't > + * look at the first @src % KMSAN_ORIGIN_SIZE byt= es > + * of the first shadow slot. > + */ > + skip_bits =3D ((u64)src % KMSAN_ORIGIN_SIZE) * 8; > + shadow =3D (shadow >> skip_bits) << skip_bits; > + } > + if (i =3D=3D src_slots - 1) { > + /* > + * If @src + n isn't aligned on > + * KMSAN_ORIGIN_SIZE, don't look at the last > + * (@src + n) % KMSAN_ORIGIN_SIZE bytes of the > + * last shadow slot. > + */ > + skip_bits =3D (((u64)src + n) % KMSAN_ORIGIN_SIZE= ) * 8; > + shadow =3D (shadow << skip_bits) >> skip_bits; > + } > + /* > + * Overwrite the origin only if the corresponding > + * shadow is nonempty. > + */ > + if (origin_src[i] && (origin_src[i] !=3D old_origin) && s= hadow) { > + old_origin =3D origin_src[i]; > + new_origin =3D kmsan_internal_chain_origin(old_or= igin); > + /* > + * kmsan_internal_chain_origin() may return > + * NULL, but we don't want to lose the previous > + * origin value. > + */ > + if (!new_origin) > + new_origin =3D old_origin; > + } > + if (shadow) > + origin_dst[i] =3D new_origin; > + else > + origin_dst[i] =3D 0; > + } > + /* > + * If dst_slots is greater than src_slots (i.e. > + * dst_slots =3D=3D src_slots + 1), there is an extra origin slot= at the > + * beginning or end of the destination buffer, for which we take = the > + * origin from the previous slot. > + * This is only done if the part of the source shadow correspondi= ng to > + * slot is non-zero. > + * > + * E.g. if we copy 8 aligned bytes that are marked as uninitializ= ed > + * and have origins o111 and o222, to an unaligned buffer with of= fset 1, > + * these two origins are copied to three origin slots, so one of = then > + * needs to be duplicated, depending on the copy direction (@back= wards) > + * > + * src shadow: |uuuu|uuuu|....| > + * src origin: |o111|o222|....| > + * > + * backwards =3D 0: > + * dst shadow: |.uuu|uuuu|u...| > + * dst origin: |....|o111|o222| - fill the empty slot with o111 > + * backwards =3D 1: > + * dst shadow: |.uuu|uuuu|u...| > + * dst origin: |o111|o222|....| - fill the empty slot with o222 > + */ > + if (src_slots < dst_slots) { > + if (backwards) { > + shadow =3D align_shadow_src[src_slots - 1]; > + skip_bits =3D (((u64)dst + n) % KMSAN_ORIGIN_SIZE= ) * 8; > + shadow =3D (shadow << skip_bits) >> skip_bits; > + if (shadow) > + /* src_slots > 0, therefore dst_slots is = at least 2 */ > + origin_dst[dst_slots - 1] =3D origin_dst[= dst_slots - 2]; > + } else { > + shadow =3D align_shadow_src[0]; > + skip_bits =3D ((u64)dst % KMSAN_ORIGIN_SIZE) * 8; > + shadow =3D (shadow >> skip_bits) << skip_bits; > + if (shadow) > + origin_dst[0] =3D origin_dst[1]; > + } > + } > +} > + > +depot_stack_handle_t kmsan_internal_chain_origin(depot_stack_handle_t id= ) > +{ > + unsigned long entries[3]; > + u32 extra_bits; > + int depth; > + bool uaf; > + > + if (!id) > + return id; > + /* > + * Make sure we have enough spare bits in @id to hold the UAF bit= and > + * the chain depth. > + */ > + BUILD_BUG_ON((1 << STACK_DEPOT_EXTRA_BITS) <=3D (MAX_CHAIN_DEPTH = << 1)); > + > + extra_bits =3D stack_depot_get_extra_bits(id); > + depth =3D kmsan_depth_from_eb(extra_bits); > + uaf =3D kmsan_uaf_from_eb(extra_bits); > + > + if (depth >=3D MAX_CHAIN_DEPTH) { > + static atomic_long_t kmsan_skipped_origins; > + long skipped =3D atomic_long_inc_return(&kmsan_skipped_or= igins); > + > + if (skipped % NUM_SKIPPED_TO_WARN =3D=3D 0) { > + pr_warn("not chained %ld origins\n", skipped); > + dump_stack(); > + kmsan_print_origin(id); > + } > + return id; > + } > + depth++; > + extra_bits =3D kmsan_extra_bits(depth, uaf); > + > + entries[0] =3D KMSAN_CHAIN_MAGIC_ORIGIN; > + entries[1] =3D kmsan_save_stack_with_flags(GFP_ATOMIC, 0); > + entries[2] =3D id; > + /* > + * @entries is a local var in non-instrumented code, so KMSAN doe= s not > + * know it is initialized. Explicitly unpoison it to avoid false > + * positives when __stack_depot_save() passes it to instrumented = code. > + */ > + kmsan_internal_unpoison_memory(entries, sizeof(entries), false); > + return __stack_depot_save(entries, ARRAY_SIZE(entries), extra_bit= s, > + GFP_ATOMIC, true); > +} > + > +void kmsan_internal_set_shadow_origin(void *addr, size_t size, int b, > + u32 origin, bool checked) > +{ > + u64 address =3D (u64)addr; > + void *shadow_start; > + u32 *origin_start; > + size_t pad =3D 0; > + int i; > + > + KMSAN_WARN_ON(!kmsan_metadata_is_contiguous(addr, size)); > + shadow_start =3D kmsan_get_metadata(addr, KMSAN_META_SHADOW); > + if (!shadow_start) { > + /* > + * kmsan_metadata_is_contiguous() is true, so either all = shadow > + * and origin pages are NULL, or all are non-NULL. > + */ > + if (checked) { > + pr_err("%s: not memsetting %ld bytes starting at = %px, because the shadow is NULL\n", > + __func__, size, addr); > + KMSAN_WARN_ON(true); > + } > + return; > + } > + __memset(shadow_start, b, size); > + > + if (!IS_ALIGNED(address, KMSAN_ORIGIN_SIZE)) { > + pad =3D address % KMSAN_ORIGIN_SIZE; > + address -=3D pad; > + size +=3D pad; > + } > + size =3D ALIGN(size, KMSAN_ORIGIN_SIZE); > + origin_start =3D > + (u32 *)kmsan_get_metadata((void *)address, KMSAN_META_ORI= GIN); > + > + for (i =3D 0; i < size / KMSAN_ORIGIN_SIZE; i++) > + origin_start[i] =3D origin; > +} > + > +struct page *kmsan_vmalloc_to_page_or_null(void *vaddr) > +{ > + struct page *page; > + > + if (!kmsan_internal_is_vmalloc_addr(vaddr) && > + !kmsan_internal_is_module_addr(vaddr)) > + return NULL; > + page =3D vmalloc_to_page(vaddr); > + if (pfn_valid(page_to_pfn(page))) > + return page; > + else > + return NULL; > +} > + > +void kmsan_internal_check_memory(void *addr, size_t size, const void *us= er_addr, > + int reason) > +{ > + depot_stack_handle_t cur_origin =3D 0, new_origin =3D 0; > + unsigned long addr64 =3D (unsigned long)addr; > + depot_stack_handle_t *origin =3D NULL; > + unsigned char *shadow =3D NULL; > + int cur_off_start =3D -1; > + int i, chunk_size; > + size_t pos =3D 0; > + > + if (!size) > + return; > + KMSAN_WARN_ON(!kmsan_metadata_is_contiguous(addr, size)); > + while (pos < size) { > + chunk_size =3D min(size - pos, > + PAGE_SIZE - ((addr64 + pos) % PAGE_SIZE)= ); > + shadow =3D kmsan_get_metadata((void *)(addr64 + pos), > + KMSAN_META_SHADOW); > + if (!shadow) { > + /* > + * This page is untracked. If there were uninitia= lized > + * bytes before, report them. > + */ > + if (cur_origin) { > + kmsan_enter_runtime(); > + kmsan_report(cur_origin, addr, size, > + cur_off_start, pos - 1, user= _addr, > + reason); > + kmsan_leave_runtime(); > + } > + cur_origin =3D 0; > + cur_off_start =3D -1; > + pos +=3D chunk_size; > + continue; > + } > + for (i =3D 0; i < chunk_size; i++) { > + if (!shadow[i]) { > + /* > + * This byte is unpoisoned. If there were > + * poisoned bytes before, report them. > + */ > + if (cur_origin) { > + kmsan_enter_runtime(); > + kmsan_report(cur_origin, addr, si= ze, > + cur_off_start, pos += i - 1, > + user_addr, reason); > + kmsan_leave_runtime(); > + } > + cur_origin =3D 0; > + cur_off_start =3D -1; > + continue; > + } > + origin =3D kmsan_get_metadata((void *)(addr64 + p= os + i), > + KMSAN_META_ORIGIN); > + KMSAN_WARN_ON(!origin); > + new_origin =3D *origin; > + /* > + * Encountered new origin - report the previous > + * uninitialized range. > + */ > + if (cur_origin !=3D new_origin) { > + if (cur_origin) { > + kmsan_enter_runtime(); > + kmsan_report(cur_origin, addr, si= ze, > + cur_off_start, pos += i - 1, > + user_addr, reason); > + kmsan_leave_runtime(); > + } > + cur_origin =3D new_origin; > + cur_off_start =3D pos + i; > + } > + } > + pos +=3D chunk_size; > + } > + KMSAN_WARN_ON(pos !=3D size); > + if (cur_origin) { > + kmsan_enter_runtime(); > + kmsan_report(cur_origin, addr, size, cur_off_start, pos -= 1, > + user_addr, reason); > + kmsan_leave_runtime(); > + } > +} > + > +bool kmsan_metadata_is_contiguous(void *addr, size_t size) > +{ > + char *cur_shadow =3D NULL, *next_shadow =3D NULL, *cur_origin =3D= NULL, > + *next_origin =3D NULL; > + u64 cur_addr =3D (u64)addr, next_addr =3D cur_addr + PAGE_SIZE; > + depot_stack_handle_t *origin_p; > + bool all_untracked =3D false; > + > + if (!size) > + return true; > + > + /* The whole range belongs to the same page. */ > + if (ALIGN_DOWN(cur_addr + size - 1, PAGE_SIZE) =3D=3D > + ALIGN_DOWN(cur_addr, PAGE_SIZE)) > + return true; > + > + cur_shadow =3D kmsan_get_metadata((void *)cur_addr, /*is_origin*/= false); > + if (!cur_shadow) > + all_untracked =3D true; > + cur_origin =3D kmsan_get_metadata((void *)cur_addr, /*is_origin*/= true); > + if (all_untracked && cur_origin) > + goto report; > + > + for (; next_addr < (u64)addr + size; > + cur_addr =3D next_addr, cur_shadow =3D next_shadow, > + cur_origin =3D next_origin, next_addr +=3D PAGE_SIZE) { > + next_shadow =3D kmsan_get_metadata((void *)next_addr, fal= se); > + next_origin =3D kmsan_get_metadata((void *)next_addr, tru= e); > + if (all_untracked) { > + if (next_shadow || next_origin) > + goto report; > + if (!next_shadow && !next_origin) > + continue; > + } > + if (((u64)cur_shadow =3D=3D ((u64)next_shadow - PAGE_SIZE= )) && > + ((u64)cur_origin =3D=3D ((u64)next_origin - PAGE_SIZE= ))) > + continue; > + goto report; > + } > + return true; > + > +report: > + pr_err("%s: attempting to access two shadow page ranges.\n", __fu= nc__); > + pr_err("Access of size %ld at %px.\n", size, addr); > + pr_err("Addresses belonging to different ranges: %px and %px\n", > + (void *)cur_addr, (void *)next_addr); > + pr_err("page[0].shadow: %px, page[1].shadow: %px\n", cur_shadow, > + next_shadow); > + pr_err("page[0].origin: %px, page[1].origin: %px\n", cur_origin, > + next_origin); > + origin_p =3D kmsan_get_metadata(addr, KMSAN_META_ORIGIN); > + if (origin_p) { > + pr_err("Origin: %08x\n", *origin_p); > + kmsan_print_origin(*origin_p); > + } else { > + pr_err("Origin: unavailable\n"); > + } > + return false; > +} > + > +bool kmsan_internal_is_module_addr(void *vaddr) > +{ > + return ((u64)vaddr >=3D MODULES_VADDR) && ((u64)vaddr < MODULES_E= ND); > +} > + > +bool kmsan_internal_is_vmalloc_addr(void *addr) > +{ > + return ((u64)addr >=3D VMALLOC_START) && ((u64)addr < VMALLOC_END= ); > +} > diff --git a/mm/kmsan/hooks.c b/mm/kmsan/hooks.c > new file mode 100644 > index 0000000000000..4ac62fa67a02a > --- /dev/null > +++ b/mm/kmsan/hooks.c > @@ -0,0 +1,66 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * KMSAN hooks for kernel subsystems. > + * > + * These functions handle creation of KMSAN metadata for memory allocati= ons. > + * > + * Copyright (C) 2018-2022 Google LLC > + * Author: Alexander Potapenko > + * > + */ > + > +#include > +#include > +#include > +#include > +#include > +#include > + > +#include "../internal.h" > +#include "../slab.h" > +#include "kmsan.h" > + > +/* > + * Instrumented functions shouldn't be called under > + * kmsan_enter_runtime()/kmsan_leave_runtime(), because this will lead t= o > + * skipping effects of functions like memset() inside instrumented code. > + */ > + > +/* Functions from kmsan-checks.h follow. */ > +void kmsan_poison_memory(const void *address, size_t size, gfp_t flags) > +{ > + if (!kmsan_enabled || kmsan_in_runtime()) > + return; > + kmsan_enter_runtime(); > + /* The users may want to poison/unpoison random memory. */ > + kmsan_internal_poison_memory((void *)address, size, flags, > + KMSAN_POISON_NOCHECK); > + kmsan_leave_runtime(); > +} > +EXPORT_SYMBOL(kmsan_poison_memory); > + > +void kmsan_unpoison_memory(const void *address, size_t size) > +{ > + unsigned long ua_flags; > + > + if (!kmsan_enabled || kmsan_in_runtime()) > + return; > + > + ua_flags =3D user_access_save(); > + kmsan_enter_runtime(); > + /* The users may want to poison/unpoison random memory. */ > + kmsan_internal_unpoison_memory((void *)address, size, > + KMSAN_POISON_NOCHECK); > + kmsan_leave_runtime(); > + user_access_restore(ua_flags); > +} > +EXPORT_SYMBOL(kmsan_unpoison_memory); > + > +void kmsan_check_memory(const void *addr, size_t size) > +{ > + if (!kmsan_enabled) > + return; > + return kmsan_internal_check_memory((void *)addr, size, /*user_add= r*/ 0, > + REASON_ANY); > +} > +EXPORT_SYMBOL(kmsan_check_memory); > diff --git a/mm/kmsan/instrumentation.c b/mm/kmsan/instrumentation.c > new file mode 100644 > index 0000000000000..1b705162be8c2 > --- /dev/null > +++ b/mm/kmsan/instrumentation.c > @@ -0,0 +1,271 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * KMSAN compiler API. > + * > + * This file implements __msan_XXX hooks that Clang inserts into the cod= e > + * compiled with -fsanitize=3Dkernel-memory. > + * See Documentation/dev-tools/kmsan.rst for more information on how KMS= AN > + * instrumentation works. > + * > + * Copyright (C) 2017-2022 Google LLC > + * Author: Alexander Potapenko > + * > + */ > + > +#include "kmsan.h" > +#include > +#include > +#include > + > +static inline bool is_bad_asm_addr(void *addr, uintptr_t size, bool is_s= tore) > +{ > + if ((u64)addr < TASK_SIZE) > + return true; > + if (!kmsan_get_metadata(addr, KMSAN_META_SHADOW)) > + return true; > + return false; > +} > + > +static inline struct shadow_origin_ptr > +get_shadow_origin_ptr(void *addr, u64 size, bool store) > +{ > + unsigned long ua_flags =3D user_access_save(); > + struct shadow_origin_ptr ret; > + > + ret =3D kmsan_get_shadow_origin_ptr(addr, size, store); > + user_access_restore(ua_flags); > + return ret; > +} > + > +/* Get shadow and origin pointers for a memory load with non-standard si= ze. */ > +struct shadow_origin_ptr __msan_metadata_ptr_for_load_n(void *addr, > + uintptr_t size) > +{ > + return get_shadow_origin_ptr(addr, size, /*store*/ false); > +} > +EXPORT_SYMBOL(__msan_metadata_ptr_for_load_n); > + > +/* Get shadow and origin pointers for a memory store with non-standard s= ize. */ > +struct shadow_origin_ptr __msan_metadata_ptr_for_store_n(void *addr, > + uintptr_t size) > +{ > + return get_shadow_origin_ptr(addr, size, /*store*/ true); > +} > +EXPORT_SYMBOL(__msan_metadata_ptr_for_store_n); > + > +/* > + * Declare functions that obtain shadow/origin pointers for loads and st= ores > + * with fixed size. > + */ > +#define DECLARE_METADATA_PTR_GETTER(size) = \ > + struct shadow_origin_ptr __msan_metadata_ptr_for_load_##size( = \ > + void *addr) = \ > + { = \ > + return get_shadow_origin_ptr(addr, size, /*store*/ false)= ; \ > + } = \ > + EXPORT_SYMBOL(__msan_metadata_ptr_for_load_##size); = \ > + struct shadow_origin_ptr __msan_metadata_ptr_for_store_##size( = \ > + void *addr) = \ > + { = \ > + return get_shadow_origin_ptr(addr, size, /*store*/ true);= \ > + } = \ > + EXPORT_SYMBOL(__msan_metadata_ptr_for_store_##size) > + > +DECLARE_METADATA_PTR_GETTER(1); > +DECLARE_METADATA_PTR_GETTER(2); > +DECLARE_METADATA_PTR_GETTER(4); > +DECLARE_METADATA_PTR_GETTER(8); > + > +/* > + * Handle a memory store performed by inline assembly. KMSAN conservativ= ely > + * attempts to unpoison the outputs of asm() directives to prevent false > + * positives caused by missed stores. > + */ > +void __msan_instrument_asm_store(void *addr, uintptr_t size) > +{ > + unsigned long ua_flags; > + > + if (!kmsan_enabled || kmsan_in_runtime()) > + return; > + > + ua_flags =3D user_access_save(); > + /* > + * Most of the accesses are below 32 bytes. The two exceptions so= far > + * are clwb() (64 bytes) and FPU state (512 bytes). > + * It's unlikely that the assembly will touch more than 512 bytes= . > + */ > + if (size > 512) { > + WARN_ONCE(1, "assembly store size too big: %ld\n", size); > + size =3D 8; > + } > + if (is_bad_asm_addr(addr, size, /*is_store*/ true)) { > + user_access_restore(ua_flags); > + return; > + } > + kmsan_enter_runtime(); > + /* Unpoisoning the memory on best effort. */ > + kmsan_internal_unpoison_memory(addr, size, /*checked*/ false); > + kmsan_leave_runtime(); > + user_access_restore(ua_flags); > +} > +EXPORT_SYMBOL(__msan_instrument_asm_store); > + > +/* Handle llvm.memmove intrinsic. */ > +void *__msan_memmove(void *dst, const void *src, uintptr_t n) > +{ > + void *result; > + > + result =3D __memmove(dst, src, n); > + if (!n) > + /* Some people call memmove() with zero length. */ > + return result; > + if (!kmsan_enabled || kmsan_in_runtime()) > + return result; > + > + kmsan_enter_runtime(); > + kmsan_internal_memmove_metadata(dst, (void *)src, n); > + kmsan_leave_runtime(); > + > + return result; > +} > +EXPORT_SYMBOL(__msan_memmove); > + > +/* Handle llvm.memcpy intrinsic. */ > +void *__msan_memcpy(void *dst, const void *src, uintptr_t n) > +{ > + void *result; > + > + result =3D __memcpy(dst, src, n); > + if (!n) > + /* Some people call memcpy() with zero length. */ > + return result; > + > + if (!kmsan_enabled || kmsan_in_runtime()) > + return result; > + > + kmsan_enter_runtime(); > + /* Using memmove instead of memcpy doesn't affect correctness. */ > + kmsan_internal_memmove_metadata(dst, (void *)src, n); > + kmsan_leave_runtime(); > + > + return result; > +} > +EXPORT_SYMBOL(__msan_memcpy); > + > +/* Handle llvm.memset intrinsic. */ > +void *__msan_memset(void *dst, int c, uintptr_t n) > +{ > + void *result; > + > + result =3D __memset(dst, c, n); > + if (!kmsan_enabled || kmsan_in_runtime()) > + return result; > + > + kmsan_enter_runtime(); > + /* > + * Clang doesn't pass parameter metadata here, so it is impossibl= e to > + * use shadow of @c to set up the shadow for @dst. > + */ > + kmsan_internal_unpoison_memory(dst, n, /*checked*/ false); > + kmsan_leave_runtime(); > + > + return result; > +} > +EXPORT_SYMBOL(__msan_memset); > + > +/* > + * Create a new origin from an old one. This is done when storing an > + * uninitialized value to memory. When reporting an error, KMSAN unrolls= and > + * prints the whole chain of stores that preceded the use of this value. > + */ > +depot_stack_handle_t __msan_chain_origin(depot_stack_handle_t origin) > +{ > + depot_stack_handle_t ret =3D 0; > + unsigned long ua_flags; > + > + if (!kmsan_enabled || kmsan_in_runtime()) > + return ret; > + > + ua_flags =3D user_access_save(); > + > + /* Creating new origins may allocate memory. */ > + kmsan_enter_runtime(); > + ret =3D kmsan_internal_chain_origin(origin); > + kmsan_leave_runtime(); > + user_access_restore(ua_flags); > + return ret; > +} > +EXPORT_SYMBOL(__msan_chain_origin); > + > +/* Poison a local variable when entering a function. */ > +void __msan_poison_alloca(void *address, uintptr_t size, char *descr) > +{ > + depot_stack_handle_t handle; > + unsigned long entries[4]; > + unsigned long ua_flags; > + > + if (!kmsan_enabled || kmsan_in_runtime()) > + return; > + > + ua_flags =3D user_access_save(); > + entries[0] =3D KMSAN_ALLOCA_MAGIC_ORIGIN; > + entries[1] =3D (u64)descr; > + entries[2] =3D (u64)__builtin_return_address(0); > + /* > + * With frame pointers enabled, it is possible to quickly fetch t= he > + * second frame of the caller stack without calling the unwinder. > + * Without them, simply do not bother. > + */ > + if (IS_ENABLED(CONFIG_UNWINDER_FRAME_POINTER)) > + entries[3] =3D (u64)__builtin_return_address(1); > + else > + entries[3] =3D 0; > + > + /* stack_depot_save() may allocate memory. */ > + kmsan_enter_runtime(); > + handle =3D stack_depot_save(entries, ARRAY_SIZE(entries), GFP_ATO= MIC); > + kmsan_leave_runtime(); > + > + kmsan_internal_set_shadow_origin(address, size, -1, handle, > + /*checked*/ true); > + user_access_restore(ua_flags); > +} > +EXPORT_SYMBOL(__msan_poison_alloca); > + > +/* Unpoison a local variable. */ > +void __msan_unpoison_alloca(void *address, uintptr_t size) > +{ > + if (!kmsan_enabled || kmsan_in_runtime()) > + return; > + > + kmsan_enter_runtime(); > + kmsan_internal_unpoison_memory(address, size, /*checked*/ true); > + kmsan_leave_runtime(); > +} > +EXPORT_SYMBOL(__msan_unpoison_alloca); > + > +/* > + * Report that an uninitialized value with the given origin was used in = a way > + * that constituted undefined behavior. > + */ > +void __msan_warning(u32 origin) > +{ > + if (!kmsan_enabled || kmsan_in_runtime()) > + return; > + kmsan_enter_runtime(); > + kmsan_report(origin, /*address*/ 0, /*size*/ 0, > + /*off_first*/ 0, /*off_last*/ 0, /*user_addr*/ 0, > + REASON_ANY); > + kmsan_leave_runtime(); > +} > +EXPORT_SYMBOL(__msan_warning); > + > +/* > + * At the beginning of an instrumented function, obtain the pointer to > + * `struct kmsan_context_state` holding the metadata for function parame= ters. > + */ > +struct kmsan_context_state *__msan_get_context_state(void) > +{ > + return &kmsan_get_context()->cstate; > +} > +EXPORT_SYMBOL(__msan_get_context_state); > diff --git a/mm/kmsan/kmsan.h b/mm/kmsan/kmsan.h > new file mode 100644 > index 0000000000000..d3c400ca097ba > --- /dev/null > +++ b/mm/kmsan/kmsan.h > @@ -0,0 +1,190 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +/* > + * Functions used by the KMSAN runtime. > + * > + * Copyright (C) 2017-2022 Google LLC > + * Author: Alexander Potapenko > + * > + */ > + > +#ifndef __MM_KMSAN_KMSAN_H > +#define __MM_KMSAN_KMSAN_H > + > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > + > +#define KMSAN_ALLOCA_MAGIC_ORIGIN 0xabcd0100 > +#define KMSAN_CHAIN_MAGIC_ORIGIN 0xabcd0200 > + > +#define KMSAN_POISON_NOCHECK 0x0 > +#define KMSAN_POISON_CHECK 0x1 > +#define KMSAN_POISON_FREE 0x2 > + > +#define KMSAN_ORIGIN_SIZE 4 > + > +#define KMSAN_STACK_DEPTH 64 > + > +#define KMSAN_META_SHADOW (false) > +#define KMSAN_META_ORIGIN (true) > + > +extern bool kmsan_enabled; > +extern int panic_on_kmsan; > + > +/* > + * KMSAN performs a lot of consistency checks that are currently enabled= by > + * default. BUG_ON is normally discouraged in the kernel, unless used fo= r > + * debugging, but KMSAN itself is a debugging tool, so it makes little s= ense to > + * recover if something goes wrong. > + */ > +#define KMSAN_WARN_ON(cond) \ > + ({ \ > + const bool __cond =3D WARN_ON(cond); = \ > + if (unlikely(__cond)) { \ > + WRITE_ONCE(kmsan_enabled, false); \ > + if (panic_on_kmsan) { \ > + /* Can't call panic() here because */ \ > + /* of uaccess checks. */ \ > + BUG(); \ > + } \ > + } \ > + __cond; \ > + }) > + > +/* > + * A pair of metadata pointers to be returned by the instrumentation fun= ctions. > + */ > +struct shadow_origin_ptr { > + void *shadow, *origin; > +}; > + > +struct shadow_origin_ptr kmsan_get_shadow_origin_ptr(void *addr, u64 siz= e, > + bool store); > +void *kmsan_get_metadata(void *addr, bool is_origin); > + > +enum kmsan_bug_reason { > + REASON_ANY, > + REASON_COPY_TO_USER, > + REASON_SUBMIT_URB, > +}; > + > +void kmsan_print_origin(depot_stack_handle_t origin); > + > +/** > + * kmsan_report() - Report a use of uninitialized value. > + * @origin: Stack ID of the uninitialized value. > + * @address: Address at which the memory access happens. > + * @size: Memory access size. > + * @off_first: Offset (from @address) of the first byte to be reported. > + * @off_last: Offset (from @address) of the last byte to be reported. > + * @user_addr: When non-NULL, denotes the userspace address to which the= kernel > + * is leaking data. > + * @reason: Error type from enum kmsan_bug_reason. > + * > + * kmsan_report() prints an error message for a consequent group of byte= s > + * sharing the same origin. If an uninitialized value is used in a compa= rison, > + * this function is called once without specifying the addresses. When c= hecking > + * a memory range, KMSAN may call kmsan_report() multiple times with the= same > + * @address, @size, @user_addr and @reason, but different @off_first and > + * @off_last corresponding to different @origin values. > + */ > +void kmsan_report(depot_stack_handle_t origin, void *address, int size, > + int off_first, int off_last, const void *user_addr, > + enum kmsan_bug_reason reason); > + > +DECLARE_PER_CPU(struct kmsan_ctx, kmsan_percpu_ctx); > + > +static __always_inline struct kmsan_ctx *kmsan_get_context(void) > +{ > + return in_task() ? ¤t->kmsan_ctx : raw_cpu_ptr(&kmsan_percp= u_ctx); > +} > + > +/* > + * When a compiler hook or KMSAN runtime function is invoked, it may mak= e a > + * call to instrumented code and eventually call itself recursively. To = avoid > + * that, we guard the runtime entry regions with > + * kmsan_enter_runtime()/kmsan_leave_runtime() and exit the hook if > + * kmsan_in_runtime() is true. > + * > + * Non-runtime code may occasionally get executed in nested IRQs from th= e > + * runtime code (e.g. when called via smp_call_function_single()). Becau= se some > + * KMSAN routines may take locks (e.g. for memory allocation), we conser= vatively > + * bail out instead of calling them. To minimize the effect of this (pot= entially > + * missing initialization events) kmsan_in_runtime() is not checked in > + * non-blocking runtime functions. > + */ > +static __always_inline bool kmsan_in_runtime(void) > +{ > + if ((hardirq_count() >> HARDIRQ_SHIFT) > 1) > + return true; > + return kmsan_get_context()->kmsan_in_runtime; > +} > + > +static __always_inline void kmsan_enter_runtime(void) > +{ > + struct kmsan_ctx *ctx; > + > + ctx =3D kmsan_get_context(); > + KMSAN_WARN_ON(ctx->kmsan_in_runtime++); > +} > + > +static __always_inline void kmsan_leave_runtime(void) > +{ > + struct kmsan_ctx *ctx =3D kmsan_get_context(); > + > + KMSAN_WARN_ON(--ctx->kmsan_in_runtime); > +} > + > +depot_stack_handle_t kmsan_save_stack(void); > +depot_stack_handle_t kmsan_save_stack_with_flags(gfp_t flags, > + unsigned int extra_bits)= ; > + > +/* > + * Pack and unpack the origin chain depth and UAF flag to/from the extra= bits > + * provided by the stack depot. > + * The UAF flag is stored in the lowest bit, followed by the depth in th= e upper > + * bits. > + * set_dsh_extra_bits() is responsible for clamping the value. > + */ > +static __always_inline unsigned int kmsan_extra_bits(unsigned int depth, > + bool uaf) > +{ > + return (depth << 1) | uaf; > +} > + > +static __always_inline bool kmsan_uaf_from_eb(unsigned int extra_bits) > +{ > + return extra_bits & 1; > +} > + > +static __always_inline unsigned int kmsan_depth_from_eb(unsigned int ext= ra_bits) > +{ > + return extra_bits >> 1; > +} > + > +/* > + * kmsan_internal_ functions are supposed to be very simple and not requ= ire the > + * kmsan_in_runtime() checks. > + */ > +void kmsan_internal_memmove_metadata(void *dst, void *src, size_t n); > +void kmsan_internal_poison_memory(void *address, size_t size, gfp_t flag= s, > + unsigned int poison_flags); > +void kmsan_internal_unpoison_memory(void *address, size_t size, bool che= cked); > +void kmsan_internal_set_shadow_origin(void *address, size_t size, int b, > + u32 origin, bool checked); > +depot_stack_handle_t kmsan_internal_chain_origin(depot_stack_handle_t id= ); > + > +bool kmsan_metadata_is_contiguous(void *addr, size_t size); > +void kmsan_internal_check_memory(void *addr, size_t size, const void *us= er_addr, > + int reason); > +bool kmsan_internal_is_module_addr(void *vaddr); > +bool kmsan_internal_is_vmalloc_addr(void *addr); > + > +struct page *kmsan_vmalloc_to_page_or_null(void *vaddr); > + > +#endif /* __MM_KMSAN_KMSAN_H */ > diff --git a/mm/kmsan/report.c b/mm/kmsan/report.c > new file mode 100644 > index 0000000000000..c298edcf49ee5 > --- /dev/null > +++ b/mm/kmsan/report.c > @@ -0,0 +1,211 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * KMSAN error reporting routines. > + * > + * Copyright (C) 2019-2022 Google LLC > + * Author: Alexander Potapenko > + * > + */ > + > +#include > +#include > +#include > +#include > +#include > + > +#include "kmsan.h" > + > +static DEFINE_RAW_SPINLOCK(kmsan_report_lock); > +#define DESCR_SIZE 128 > +/* Protected by kmsan_report_lock */ > +static char report_local_descr[DESCR_SIZE]; > +int panic_on_kmsan __read_mostly; > + > +#ifdef MODULE_PARAM_PREFIX > +#undef MODULE_PARAM_PREFIX > +#endif > +#define MODULE_PARAM_PREFIX "kmsan." > +module_param_named(panic, panic_on_kmsan, int, 0); > + > +/* > + * Skip internal KMSAN frames. > + */ > +static int get_stack_skipnr(const unsigned long stack_entries[], > + int num_entries) > +{ > + int len, skip; > + char buf[64]; > + > + for (skip =3D 0; skip < num_entries; ++skip) { > + len =3D scnprintf(buf, sizeof(buf), "%ps", > + (void *)stack_entries[skip]); > + > + /* Never show __msan_* or kmsan_* functions. */ > + if ((strnstr(buf, "__msan_", len) =3D=3D buf) || > + (strnstr(buf, "kmsan_", len) =3D=3D buf)) > + continue; > + > + /* > + * No match for runtime functions -- @skip entries to ski= p to > + * get to first frame of interest. > + */ > + break; > + } > + > + return skip; > +} > + > +/* > + * Currently the descriptions of locals generated by Clang look as follo= ws: > + * ----local_name@function_name > + * We want to print only the name of the local, as other information in = that > + * description can be confusing. > + * The meaningful part of the description is copied to a global buffer t= o avoid > + * allocating memory. > + */ > +static char *pretty_descr(char *descr) > +{ > + int i, pos =3D 0, len =3D strlen(descr); > + > + for (i =3D 0; i < len; i++) { > + if (descr[i] =3D=3D '@') > + break; > + if (descr[i] =3D=3D '-') > + continue; > + report_local_descr[pos] =3D descr[i]; > + if (pos + 1 =3D=3D DESCR_SIZE) > + break; > + pos++; > + } > + report_local_descr[pos] =3D 0; > + return report_local_descr; > +} > + > +void kmsan_print_origin(depot_stack_handle_t origin) > +{ > + unsigned long *entries =3D NULL, *chained_entries =3D NULL; > + unsigned int nr_entries, chained_nr_entries, skipnr; > + void *pc1 =3D NULL, *pc2 =3D NULL; > + depot_stack_handle_t head; > + unsigned long magic; > + char *descr =3D NULL; > + > + if (!origin) > + return; > + > + while (true) { > + nr_entries =3D stack_depot_fetch(origin, &entries); > + magic =3D nr_entries ? entries[0] : 0; > + if ((nr_entries =3D=3D 4) && (magic =3D=3D KMSAN_ALLOCA_M= AGIC_ORIGIN)) { > + descr =3D (char *)entries[1]; > + pc1 =3D (void *)entries[2]; > + pc2 =3D (void *)entries[3]; > + pr_err("Local variable %s created at:\n", > + pretty_descr(descr)); > + if (pc1) > + pr_err(" %pSb\n", pc1); > + if (pc2) > + pr_err(" %pSb\n", pc2); > + break; > + } > + if ((nr_entries =3D=3D 3) && (magic =3D=3D KMSAN_CHAIN_MA= GIC_ORIGIN)) { > + head =3D entries[1]; > + origin =3D entries[2]; > + pr_err("Uninit was stored to memory at:\n"); > + chained_nr_entries =3D > + stack_depot_fetch(head, &chained_entries)= ; > + kmsan_internal_unpoison_memory( > + chained_entries, > + chained_nr_entries * sizeof(*chained_entr= ies), > + /*checked*/ false); > + skipnr =3D get_stack_skipnr(chained_entries, > + chained_nr_entries); > + stack_trace_print(chained_entries + skipnr, > + chained_nr_entries - skipnr, 0)= ; > + pr_err("\n"); > + continue; > + } > + pr_err("Uninit was created at:\n"); > + if (nr_entries) { > + skipnr =3D get_stack_skipnr(entries, nr_entries); > + stack_trace_print(entries + skipnr, nr_entries - = skipnr, > + 0); > + } else { > + pr_err("(stack is not available)\n"); > + } > + break; > + } > +} > + > +void kmsan_report(depot_stack_handle_t origin, void *address, int size, > + int off_first, int off_last, const void *user_addr, > + enum kmsan_bug_reason reason) > +{ > + unsigned long stack_entries[KMSAN_STACK_DEPTH]; > + int num_stack_entries, skipnr; > + char *bug_type =3D NULL; > + unsigned long ua_flags; > + bool is_uaf; > + > + if (!kmsan_enabled) > + return; > + if (!current->kmsan_ctx.allow_reporting) > + return; > + if (!origin) > + return; > + > + current->kmsan_ctx.allow_reporting =3D false; > + ua_flags =3D user_access_save(); > + raw_spin_lock(&kmsan_report_lock); > + pr_err("=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D\n"); > + is_uaf =3D kmsan_uaf_from_eb(stack_depot_get_extra_bits(origin)); > + switch (reason) { > + case REASON_ANY: > + bug_type =3D is_uaf ? "use-after-free" : "uninit-value"; > + break; > + case REASON_COPY_TO_USER: > + bug_type =3D is_uaf ? "kernel-infoleak-after-free" : > + "kernel-infoleak"; > + break; > + case REASON_SUBMIT_URB: > + bug_type =3D is_uaf ? "kernel-usb-infoleak-after-free" : > + "kernel-usb-infoleak"; > + break; > + } > + > + num_stack_entries =3D > + stack_trace_save(stack_entries, KMSAN_STACK_DEPTH, 1); > + skipnr =3D get_stack_skipnr(stack_entries, num_stack_entries); > + > + pr_err("BUG: KMSAN: %s in %pSb\n", > + bug_type, (void *)stack_entries[skipnr]); > + stack_trace_print(stack_entries + skipnr, num_stack_entries - ski= pnr, > + 0); > + pr_err("\n"); > + > + kmsan_print_origin(origin); > + > + if (size) { > + pr_err("\n"); > + if (off_first =3D=3D off_last) > + pr_err("Byte %d of %d is uninitialized\n", off_fi= rst, > + size); > + else > + pr_err("Bytes %d-%d of %d are uninitialized\n", > + off_first, off_last, size); > + } > + if (address) > + pr_err("Memory access of size %d starts at %px\n", size, > + address); > + if (user_addr && reason =3D=3D REASON_COPY_TO_USER) > + pr_err("Data copied to user address %px\n", user_addr); > + pr_err("\n"); > + dump_stack_print_info(KERN_ERR); > + pr_err("=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D\n"); > + add_taint(TAINT_BAD_PAGE, LOCKDEP_NOW_UNRELIABLE); > + raw_spin_unlock(&kmsan_report_lock); > + if (panic_on_kmsan) > + panic("kmsan.panic set ...\n"); > + user_access_restore(ua_flags); > + current->kmsan_ctx.allow_reporting =3D true; > +} > diff --git a/mm/kmsan/shadow.c b/mm/kmsan/shadow.c > new file mode 100644 > index 0000000000000..e5ad2972d7362 > --- /dev/null > +++ b/mm/kmsan/shadow.c > @@ -0,0 +1,147 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * KMSAN shadow implementation. > + * > + * Copyright (C) 2017-2022 Google LLC > + * Author: Alexander Potapenko > + * > + */ > + > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > + > +#include "../internal.h" > +#include "kmsan.h" > + > +#define shadow_page_for(page) ((page)->kmsan_shadow) > + > +#define origin_page_for(page) ((page)->kmsan_origin) > + > +static void *shadow_ptr_for(struct page *page) > +{ > + return page_address(shadow_page_for(page)); > +} > + > +static void *origin_ptr_for(struct page *page) > +{ > + return page_address(origin_page_for(page)); > +} > + > +static bool page_has_metadata(struct page *page) > +{ > + return shadow_page_for(page) && origin_page_for(page); > +} > + > +static void set_no_shadow_origin_page(struct page *page) > +{ > + shadow_page_for(page) =3D NULL; > + origin_page_for(page) =3D NULL; > +} > + > +/* > + * Dummy load and store pages to be used when the real metadata is unava= ilable. > + * There are separate pages for loads and stores, so that every load ret= urns a > + * zero, and every store doesn't affect other loads. > + */ > +static char dummy_load_page[PAGE_SIZE] __aligned(PAGE_SIZE); > +static char dummy_store_page[PAGE_SIZE] __aligned(PAGE_SIZE); > + > +static unsigned long vmalloc_meta(void *addr, bool is_origin) > +{ > + unsigned long addr64 =3D (unsigned long)addr, off; > + > + KMSAN_WARN_ON(is_origin && !IS_ALIGNED(addr64, KMSAN_ORIGIN_SIZE)= ); > + if (kmsan_internal_is_vmalloc_addr(addr)) { > + off =3D addr64 - VMALLOC_START; > + return off + (is_origin ? KMSAN_VMALLOC_ORIGIN_START : > + KMSAN_VMALLOC_SHADOW_STAR= T); > + } > + if (kmsan_internal_is_module_addr(addr)) { > + off =3D addr64 - MODULES_VADDR; > + return off + (is_origin ? KMSAN_MODULES_ORIGIN_START : > + KMSAN_MODULES_SHADOW_STAR= T); > + } > + return 0; > +} > + > +static struct page *virt_to_page_or_null(void *vaddr) > +{ > + if (kmsan_virt_addr_valid(vaddr)) > + return virt_to_page(vaddr); > + else > + return NULL; > +} > + > +struct shadow_origin_ptr kmsan_get_shadow_origin_ptr(void *address, u64 = size, > + bool store) > +{ > + struct shadow_origin_ptr ret; > + void *shadow; > + > + /* > + * Even if we redirect this memory access to the dummy page, it w= ill > + * go out of bounds. > + */ > + KMSAN_WARN_ON(size > PAGE_SIZE); > + > + if (!kmsan_enabled) > + goto return_dummy; > + > + KMSAN_WARN_ON(!kmsan_metadata_is_contiguous(address, size)); > + shadow =3D kmsan_get_metadata(address, KMSAN_META_SHADOW); > + if (!shadow) > + goto return_dummy; > + > + ret.shadow =3D shadow; > + ret.origin =3D kmsan_get_metadata(address, KMSAN_META_ORIGIN); > + return ret; > + > +return_dummy: > + if (store) { > + /* Ignore this store. */ > + ret.shadow =3D dummy_store_page; > + ret.origin =3D dummy_store_page; > + } else { > + /* This load will return zero. */ > + ret.shadow =3D dummy_load_page; > + ret.origin =3D dummy_load_page; > + } > + return ret; > +} > + > +/* > + * Obtain the shadow or origin pointer for the given address, or NULL if= there's > + * none. The caller must check the return value for being non-NULL if ne= eded. > + * The return value of this function should not depend on whether we're = in the > + * runtime or not. > + */ > +void *kmsan_get_metadata(void *address, bool is_origin) > +{ > + u64 addr =3D (u64)address, pad, off; > + struct page *page; > + > + if (is_origin && !IS_ALIGNED(addr, KMSAN_ORIGIN_SIZE)) { > + pad =3D addr % KMSAN_ORIGIN_SIZE; > + addr -=3D pad; > + } > + address =3D (void *)addr; > + if (kmsan_internal_is_vmalloc_addr(address) || > + kmsan_internal_is_module_addr(address)) > + return (void *)vmalloc_meta(address, is_origin); > + > + page =3D virt_to_page_or_null(address); > + if (!page) > + return NULL; > + if (!page_has_metadata(page)) > + return NULL; > + off =3D addr % PAGE_SIZE; > + > + return (is_origin ? origin_ptr_for(page) : shadow_ptr_for(page)) = + off; > +} > diff --git a/scripts/Makefile.kmsan b/scripts/Makefile.kmsan > new file mode 100644 > index 0000000000000..b5b0aa61322ec > --- /dev/null > +++ b/scripts/Makefile.kmsan > @@ -0,0 +1,8 @@ > +# SPDX-License-Identifier: GPL-2.0 > +kmsan-cflags :=3D -fsanitize=3Dkernel-memory > + > +ifdef CONFIG_KMSAN_CHECK_PARAM_RETVAL > +kmsan-cflags +=3D -fsanitize-memory-param-retval > +endif > + > +export CFLAGS_KMSAN :=3D $(kmsan-cflags) > diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib > index d1425778664b9..46ebf7cb081f6 100644 > --- a/scripts/Makefile.lib > +++ b/scripts/Makefile.lib > @@ -157,6 +157,15 @@ _c_flags +=3D $(if $(patsubst n%,, \ > endif > endif > > +ifeq ($(CONFIG_KMSAN),y) > +_c_flags +=3D $(if $(patsubst n%,, \ > + $(KMSAN_SANITIZE_$(basetarget).o)$(KMSAN_SANITIZE)y), \ > + $(CFLAGS_KMSAN)) > +_c_flags +=3D $(if $(patsubst n%,, \ > + $(KMSAN_ENABLE_CHECKS_$(basetarget).o)$(KMSAN_ENABLE_CHEC= KS)y), \ > + , -mllvm -msan-disable-checks=3D1) > +endif > + > ifeq ($(CONFIG_UBSAN),y) > _c_flags +=3D $(if $(patsubst n%,, \ > $(UBSAN_SANITIZE_$(basetarget).o)$(UBSAN_SANITIZE)$(CONFI= G_UBSAN_SANITIZE_ALL)), \ > -- > 2.37.0.rc0.161.g10f37bed90-goog > > -- > You received this message because you are subscribed to the Google Groups= "kasan-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an= email to kasan-dev+unsubscribe@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/msgi= d/kasan-dev/20220701142310.2188015-12-glider%40google.com.