From: Andrey Konovalov <[email protected]>
This series makes the tag-based KASAN modes use a ring buffer for storing
stack depot handles for alloc/free stack traces for slab objects instead
of per-object metadata. This ring buffer is referred to as the stack ring.
On each alloc/free of a slab object, the tagged address of the object and
the current stack trace are recorded in the stack ring.
On each bug report, if the accessed address belongs to a slab object, the
stack ring is scanned for matching entries. The newest entries are used to
print the alloc/free stack traces in the report: one entry for alloc and
one for free.
The advantages of this approach over storing stack trace handles in
per-object metadata with the tag-based KASAN modes:
- Allows to find relevant stack traces for use-after-free bugs without
using quarantine for freed memory. (Currently, if the object was
reallocated multiple times, the report contains the latest alloc/free
stack traces, not necessarily the ones relevant to the buggy allocation.)
- Allows to better identify and mark use-after-free bugs, effectively
making the CONFIG_KASAN_TAGS_IDENTIFY functionality always-on.
- Has fixed memory overhead.
The disadvantage:
- If the affected object was allocated/freed long before the bug happened
and the stack trace events were purged from the stack ring, the report
will have no stack traces.
Discussion
==========
The proposed implementation of the stack ring uses a single ring buffer for
the whole kernel. This might lead to contention due to atomic accesses to
the ring buffer index on multicore systems.
At this point, it is unknown whether the performance impact from this
contention would be significant compared to the slowdown introduced by
collecting stack traces due to the planned changes to the latter part,
see the section below.
For now, the proposed implementation is deemed to be good enough, but this
might need to be revisited once the stack collection becomes faster.
A considered alternative is to keep a separate ring buffer for each CPU
and then iterate over all of them when printing a bug report. This approach
requires somehow figuring out which of the stack rings has the freshest
stack traces for an object if multiple stack rings have them.
Further plans
=============
This series is a part of an effort to make KASAN stack trace collection
suitable for production. This requires stack trace collection to be fast
and memory-bounded.
The planned steps are:
1. Speed up stack trace collection (potentially, by using SCS;
patches on-hold until steps #2 and #3 are completed).
2. Keep stack trace handles in the stack ring (this series).
3. Add a memory-bounded mode to stack depot or provide an alternative
memory-bounded stack storage.
4. Potentially, implement stack trace collection sampling to minimize
the performance impact.
Thanks!
---
Changes v2->v3:
- Addressed Marco's comments, see the last 3 patches for list of changes.
Changes v1->v2:
- Rework synchronization in the stack ring implementation.
- Dynamically allocate stack ring based on the kasan.stack_ring_size
command-line parameter.
- Multiple less significant changes, see the notes in patches for details.
Andrey Konovalov (34):
kasan: check KASAN_NO_FREE_META in __kasan_metadata_size
kasan: rename kasan_set_*_info to kasan_save_*_info
kasan: move is_kmalloc check out of save_alloc_info
kasan: split save_alloc_info implementations
kasan: drop CONFIG_KASAN_TAGS_IDENTIFY
kasan: introduce kasan_print_aux_stacks
kasan: introduce kasan_get_alloc_track
kasan: introduce kasan_init_object_meta
kasan: clear metadata functions for tag-based modes
kasan: move kasan_get_*_meta to generic.c
kasan: introduce kasan_requires_meta
kasan: introduce kasan_init_cache_meta
kasan: drop CONFIG_KASAN_GENERIC check from kasan_init_cache_meta
kasan: only define kasan_metadata_size for Generic mode
kasan: only define kasan_never_merge for Generic mode
kasan: only define metadata offsets for Generic mode
kasan: only define metadata structs for Generic mode
kasan: only define kasan_cache_create for Generic mode
kasan: pass tagged pointers to kasan_save_alloc/free_info
kasan: move kasan_get_alloc/free_track definitions
kasan: cosmetic changes in report.c
kasan: use virt_addr_valid in kasan_addr_to_page/slab
kasan: use kasan_addr_to_slab in print_address_description
kasan: make kasan_addr_to_page static
kasan: simplify print_report
kasan: introduce complete_report_info
kasan: fill in cache and object in complete_report_info
kasan: rework function arguments in report.c
kasan: introduce kasan_complete_mode_report_info
kasan: implement stack ring for tag-based modes
kasan: support kasan.stacktrace for SW_TAGS
kasan: dynamically allocate stack ring entries
kasan: better identify bug types for tag-based modes
kasan: add another use-after-free test
Documentation/dev-tools/kasan.rst | 17 ++-
include/linux/kasan.h | 55 ++++------
include/linux/slab.h | 2 +-
lib/Kconfig.kasan | 8 --
lib/test_kasan.c | 24 ++++
mm/kasan/common.c | 175 +++---------------------------
mm/kasan/generic.c | 154 ++++++++++++++++++++++++--
mm/kasan/hw_tags.c | 39 +------
mm/kasan/kasan.h | 171 ++++++++++++++++++++---------
mm/kasan/report.c | 117 +++++++++-----------
mm/kasan/report_generic.c | 45 +++++++-
mm/kasan/report_tags.c | 123 ++++++++++++++++-----
mm/kasan/sw_tags.c | 5 +-
mm/kasan/tags.c | 141 +++++++++++++++++++-----
14 files changed, 642 insertions(+), 434 deletions(-)
--
2.25.1
From: Andrey Konovalov <[email protected]>
Add a kasan_init_cache_meta() helper that initializes metadata-related
cache parameters and use this helper in the common KASAN code.
Put the implementation of this new helper into generic.c, as only the
Generic mode uses per-object metadata.
Reviewed-by: Marco Elver <[email protected]>
Signed-off-by: Andrey Konovalov <[email protected]>
---
mm/kasan/common.c | 80 ++--------------------------------------------
mm/kasan/generic.c | 79 +++++++++++++++++++++++++++++++++++++++++++++
mm/kasan/kasan.h | 2 ++
3 files changed, 83 insertions(+), 78 deletions(-)
diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index d0300954d76b..b6a74fe5e740 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -118,28 +118,9 @@ void __kasan_poison_pages(struct page *page, unsigned int order, bool init)
KASAN_PAGE_FREE, init);
}
-/*
- * Adaptive redzone policy taken from the userspace AddressSanitizer runtime.
- * For larger allocations larger redzones are used.
- */
-static inline unsigned int optimal_redzone(unsigned int object_size)
-{
- return
- object_size <= 64 - 16 ? 16 :
- object_size <= 128 - 32 ? 32 :
- object_size <= 512 - 64 ? 64 :
- object_size <= 4096 - 128 ? 128 :
- object_size <= (1 << 14) - 256 ? 256 :
- object_size <= (1 << 15) - 512 ? 512 :
- object_size <= (1 << 16) - 1024 ? 1024 : 2048;
-}
-
void __kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
slab_flags_t *flags)
{
- unsigned int ok_size;
- unsigned int optimal_size;
-
/*
* SLAB_KASAN is used to mark caches as ones that are sanitized by
* KASAN. Currently this flag is used in two places:
@@ -149,65 +130,8 @@ void __kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
*/
*flags |= SLAB_KASAN;
- if (!kasan_requires_meta())
- return;
-
- ok_size = *size;
-
- /* Add alloc meta into redzone. */
- cache->kasan_info.alloc_meta_offset = *size;
- *size += sizeof(struct kasan_alloc_meta);
-
- /*
- * If alloc meta doesn't fit, don't add it.
- * This can only happen with SLAB, as it has KMALLOC_MAX_SIZE equal
- * to KMALLOC_MAX_CACHE_SIZE and doesn't fall back to page_alloc for
- * larger sizes.
- */
- if (*size > KMALLOC_MAX_SIZE) {
- cache->kasan_info.alloc_meta_offset = 0;
- *size = ok_size;
- /* Continue, since free meta might still fit. */
- }
-
- /* Only the generic mode uses free meta or flexible redzones. */
- if (!IS_ENABLED(CONFIG_KASAN_GENERIC)) {
- cache->kasan_info.free_meta_offset = KASAN_NO_FREE_META;
- return;
- }
-
- /*
- * Add free meta into redzone when it's not possible to store
- * it in the object. This is the case when:
- * 1. Object is SLAB_TYPESAFE_BY_RCU, which means that it can
- * be touched after it was freed, or
- * 2. Object has a constructor, which means it's expected to
- * retain its content until the next allocation, or
- * 3. Object is too small.
- * Otherwise cache->kasan_info.free_meta_offset = 0 is implied.
- */
- if ((cache->flags & SLAB_TYPESAFE_BY_RCU) || cache->ctor ||
- cache->object_size < sizeof(struct kasan_free_meta)) {
- ok_size = *size;
-
- cache->kasan_info.free_meta_offset = *size;
- *size += sizeof(struct kasan_free_meta);
-
- /* If free meta doesn't fit, don't add it. */
- if (*size > KMALLOC_MAX_SIZE) {
- cache->kasan_info.free_meta_offset = KASAN_NO_FREE_META;
- *size = ok_size;
- }
- }
-
- /* Calculate size with optimal redzone. */
- optimal_size = cache->object_size + optimal_redzone(cache->object_size);
- /* Limit it with KMALLOC_MAX_SIZE (relevant for SLAB only). */
- if (optimal_size > KMALLOC_MAX_SIZE)
- optimal_size = KMALLOC_MAX_SIZE;
- /* Use optimal size if the size with added metas is not large enough. */
- if (*size < optimal_size)
- *size = optimal_size;
+ if (kasan_requires_meta())
+ kasan_init_cache_meta(cache, size);
}
void __kasan_cache_create_kmalloc(struct kmem_cache *cache)
diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c
index fa654cb96a0d..73aea784040a 100644
--- a/mm/kasan/generic.c
+++ b/mm/kasan/generic.c
@@ -328,6 +328,85 @@ DEFINE_ASAN_SET_SHADOW(f3);
DEFINE_ASAN_SET_SHADOW(f5);
DEFINE_ASAN_SET_SHADOW(f8);
+/*
+ * Adaptive redzone policy taken from the userspace AddressSanitizer runtime.
+ * For larger allocations larger redzones are used.
+ */
+static inline unsigned int optimal_redzone(unsigned int object_size)
+{
+ return
+ object_size <= 64 - 16 ? 16 :
+ object_size <= 128 - 32 ? 32 :
+ object_size <= 512 - 64 ? 64 :
+ object_size <= 4096 - 128 ? 128 :
+ object_size <= (1 << 14) - 256 ? 256 :
+ object_size <= (1 << 15) - 512 ? 512 :
+ object_size <= (1 << 16) - 1024 ? 1024 : 2048;
+}
+
+void kasan_init_cache_meta(struct kmem_cache *cache, unsigned int *size)
+{
+ unsigned int ok_size;
+ unsigned int optimal_size;
+
+ ok_size = *size;
+
+ /* Add alloc meta into redzone. */
+ cache->kasan_info.alloc_meta_offset = *size;
+ *size += sizeof(struct kasan_alloc_meta);
+
+ /*
+ * If alloc meta doesn't fit, don't add it.
+ * This can only happen with SLAB, as it has KMALLOC_MAX_SIZE equal
+ * to KMALLOC_MAX_CACHE_SIZE and doesn't fall back to page_alloc for
+ * larger sizes.
+ */
+ if (*size > KMALLOC_MAX_SIZE) {
+ cache->kasan_info.alloc_meta_offset = 0;
+ *size = ok_size;
+ /* Continue, since free meta might still fit. */
+ }
+
+ /* Only the generic mode uses free meta or flexible redzones. */
+ if (!IS_ENABLED(CONFIG_KASAN_GENERIC)) {
+ cache->kasan_info.free_meta_offset = KASAN_NO_FREE_META;
+ return;
+ }
+
+ /*
+ * Add free meta into redzone when it's not possible to store
+ * it in the object. This is the case when:
+ * 1. Object is SLAB_TYPESAFE_BY_RCU, which means that it can
+ * be touched after it was freed, or
+ * 2. Object has a constructor, which means it's expected to
+ * retain its content until the next allocation, or
+ * 3. Object is too small.
+ * Otherwise cache->kasan_info.free_meta_offset = 0 is implied.
+ */
+ if ((cache->flags & SLAB_TYPESAFE_BY_RCU) || cache->ctor ||
+ cache->object_size < sizeof(struct kasan_free_meta)) {
+ ok_size = *size;
+
+ cache->kasan_info.free_meta_offset = *size;
+ *size += sizeof(struct kasan_free_meta);
+
+ /* If free meta doesn't fit, don't add it. */
+ if (*size > KMALLOC_MAX_SIZE) {
+ cache->kasan_info.free_meta_offset = KASAN_NO_FREE_META;
+ *size = ok_size;
+ }
+ }
+
+ /* Calculate size with optimal redzone. */
+ optimal_size = cache->object_size + optimal_redzone(cache->object_size);
+ /* Limit it with KMALLOC_MAX_SIZE (relevant for SLAB only). */
+ if (optimal_size > KMALLOC_MAX_SIZE)
+ optimal_size = KMALLOC_MAX_SIZE;
+ /* Use optimal size if the size with added metas is not large enough. */
+ if (*size < optimal_size)
+ *size = optimal_size;
+}
+
struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,
const void *object)
{
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index 1736abd661b6..6da35370ba37 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -297,12 +297,14 @@ struct page *kasan_addr_to_page(const void *addr);
struct slab *kasan_addr_to_slab(const void *addr);
#ifdef CONFIG_KASAN_GENERIC
+void kasan_init_cache_meta(struct kmem_cache *cache, unsigned int *size);
void kasan_init_object_meta(struct kmem_cache *cache, const void *object);
struct kasan_alloc_meta *kasan_get_alloc_meta(struct kmem_cache *cache,
const void *object);
struct kasan_free_meta *kasan_get_free_meta(struct kmem_cache *cache,
const void *object);
#else
+static inline void kasan_init_cache_meta(struct kmem_cache *cache, unsigned int *size) { }
static inline void kasan_init_object_meta(struct kmem_cache *cache, const void *object) { }
#endif
--
2.25.1
From: Andrey Konovalov <[email protected]>
As kasan_addr_to_page() is only used in report.c, rename it to
addr_to_page() and make it static.
Reviewed-by: Marco Elver <[email protected]>
Signed-off-by: Andrey Konovalov <[email protected]>
---
mm/kasan/kasan.h | 1 -
mm/kasan/report.c | 4 ++--
2 files changed, 2 insertions(+), 3 deletions(-)
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index cca49ab029f1..4fddfdb08abf 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -291,7 +291,6 @@ bool kasan_report(unsigned long addr, size_t size,
bool is_write, unsigned long ip);
void kasan_report_invalid_free(void *object, unsigned long ip, enum kasan_report_type type);
-struct page *kasan_addr_to_page(const void *addr);
struct slab *kasan_addr_to_slab(const void *addr);
#ifdef CONFIG_KASAN_GENERIC
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index cd31b3b89ca1..ac526c10ebff 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -206,7 +206,7 @@ static void print_track(struct kasan_track *track, const char *prefix)
pr_err("(stack is not available)\n");
}
-struct page *kasan_addr_to_page(const void *addr)
+static inline struct page *addr_to_page(const void *addr)
{
if (virt_addr_valid(addr))
return virt_to_head_page(addr);
@@ -289,7 +289,7 @@ static inline bool init_task_stack_addr(const void *addr)
static void print_address_description(void *addr, u8 tag)
{
- struct page *page = kasan_addr_to_page(addr);
+ struct page *page = addr_to_page(addr);
struct slab *slab = kasan_addr_to_slab(addr);
dump_stack_lvl(KERN_ERR);
--
2.25.1
From: Andrey Konovalov <[email protected]>
Instead of using a large static array, allocate the stack ring dynamically
via memblock_alloc().
The size of the stack ring is controlled by a new kasan.stack_ring_size
command-line parameter. When kasan.stack_ring_size is not provided, the
default value of 32 << 10 is used.
When the stack trace collection is disabled via kasan.stacktrace=off,
the stack ring is not allocated.
Signed-off-by: Andrey Konovalov <[email protected]>
---
Changes v2->v3:
- Move KASAN_STACK_RING_SIZE_DEFAULT definition to tags.c
- Improve comment for early_kasan_flag_stack_ring_size().
- WARN_ON and disable stack traces on failed memblock_alloc.
- Add kasan.stack_ring_size to documentation.
Changes v1->v2:
- This is a new patch.
---
Documentation/dev-tools/kasan.rst | 4 +++-
mm/kasan/kasan.h | 5 ++---
mm/kasan/report_tags.c | 4 ++--
mm/kasan/tags.c | 25 ++++++++++++++++++++++++-
4 files changed, 31 insertions(+), 7 deletions(-)
diff --git a/Documentation/dev-tools/kasan.rst b/Documentation/dev-tools/kasan.rst
index 7bd38c181018..5c93ab915049 100644
--- a/Documentation/dev-tools/kasan.rst
+++ b/Documentation/dev-tools/kasan.rst
@@ -112,10 +112,12 @@ parameter can be used to control panic and reporting behaviour:
if ``kasan_multi_shot`` is enabled.
Software and Hardware Tag-Based KASAN modes (see the section about various
-modes below) support disabling stack trace collection:
+modes below) support altering stack trace collection behavior:
- ``kasan.stacktrace=off`` or ``=on`` disables or enables alloc and free stack
traces collection (default: ``on``).
+- ``kasan.stack_ring_size=<number of entries>`` specifies the number of entries
+ in the stack ring (default: ``32768``).
Hardware Tag-Based KASAN mode is intended for use in production as a security
mitigation. Therefore, it supports additional boot parameters that allow
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index 447baf1a7a2e..abbcc1b0eec5 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -252,12 +252,11 @@ struct kasan_stack_ring_entry {
bool is_free;
};
-#define KASAN_STACK_RING_SIZE (32 << 10)
-
struct kasan_stack_ring {
rwlock_t lock;
+ size_t size;
atomic64_t pos;
- struct kasan_stack_ring_entry entries[KASAN_STACK_RING_SIZE];
+ struct kasan_stack_ring_entry *entries;
};
#endif /* CONFIG_KASAN_SW_TAGS || CONFIG_KASAN_HW_TAGS */
diff --git a/mm/kasan/report_tags.c b/mm/kasan/report_tags.c
index 1b78136542bb..57f7355377f1 100644
--- a/mm/kasan/report_tags.c
+++ b/mm/kasan/report_tags.c
@@ -56,11 +56,11 @@ void kasan_complete_mode_report_info(struct kasan_report_info *info)
* entries relevant to the buggy object can be overwritten.
*/
- for (u64 i = pos - 1; i != pos - 1 - KASAN_STACK_RING_SIZE; i--) {
+ for (u64 i = pos - 1; i != pos - 1 - stack_ring.size; i--) {
if (alloc_found && free_found)
break;
- entry = &stack_ring.entries[i % KASAN_STACK_RING_SIZE];
+ entry = &stack_ring.entries[i % stack_ring.size];
/* Paired with smp_store_release() in save_stack_info(). */
ptr = (void *)smp_load_acquire(&entry->ptr);
diff --git a/mm/kasan/tags.c b/mm/kasan/tags.c
index 0eb6cf6717db..9d867cae1b7b 100644
--- a/mm/kasan/tags.c
+++ b/mm/kasan/tags.c
@@ -10,6 +10,7 @@
#include <linux/init.h>
#include <linux/kasan.h>
#include <linux/kernel.h>
+#include <linux/memblock.h>
#include <linux/memory.h>
#include <linux/mm.h>
#include <linux/static_key.h>
@@ -19,6 +20,8 @@
#include "kasan.h"
#include "../slab.h"
+#define KASAN_STACK_RING_SIZE_DEFAULT (32 << 10)
+
enum kasan_arg_stacktrace {
KASAN_ARG_STACKTRACE_DEFAULT,
KASAN_ARG_STACKTRACE_OFF,
@@ -52,6 +55,16 @@ static int __init early_kasan_flag_stacktrace(char *arg)
}
early_param("kasan.stacktrace", early_kasan_flag_stacktrace);
+/* kasan.stack_ring_size=<number of entries> */
+static int __init early_kasan_flag_stack_ring_size(char *arg)
+{
+ if (!arg)
+ return -EINVAL;
+
+ return kstrtoul(arg, 0, &stack_ring.size);
+}
+early_param("kasan.stack_ring_size", early_kasan_flag_stack_ring_size);
+
void __init kasan_init_tags(void)
{
switch (kasan_arg_stacktrace) {
@@ -65,6 +78,16 @@ void __init kasan_init_tags(void)
static_branch_enable(&kasan_flag_stacktrace);
break;
}
+
+ if (kasan_stack_collection_enabled()) {
+ if (!stack_ring.size)
+ stack_ring.size = KASAN_STACK_RING_SIZE_DEFAULT;
+ stack_ring.entries = memblock_alloc(
+ sizeof(stack_ring.entries[0]) * stack_ring.size,
+ SMP_CACHE_BYTES);
+ if (WARN_ON(!stack_ring.entries))
+ static_branch_disable(&kasan_flag_stacktrace);
+ }
}
static void save_stack_info(struct kmem_cache *cache, void *object,
@@ -86,7 +109,7 @@ static void save_stack_info(struct kmem_cache *cache, void *object,
next:
pos = atomic64_fetch_add(1, &stack_ring.pos);
- entry = &stack_ring.entries[pos % KASAN_STACK_RING_SIZE];
+ entry = &stack_ring.entries[pos % stack_ring.size];
/* Detect stack ring entry slots that are being written to. */
old_ptr = READ_ONCE(entry->ptr);
--
2.25.1
From: Andrey Konovalov <[email protected]>
Do a few non-functional style fixes for the code in report.c.
Reviewed-by: Marco Elver <[email protected]>
Signed-off-by: Andrey Konovalov <[email protected]>
---
mm/kasan/report.c | 11 ++++-------
1 file changed, 4 insertions(+), 7 deletions(-)
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index 5d225d7d9c4c..83f420a28c0b 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -200,25 +200,22 @@ static void print_error_description(struct kasan_report_info *info)
static void print_track(struct kasan_track *track, const char *prefix)
{
pr_err("%s by task %u:\n", prefix, track->pid);
- if (track->stack) {
+ if (track->stack)
stack_depot_print(track->stack);
- } else {
+ else
pr_err("(stack is not available)\n");
- }
}
struct page *kasan_addr_to_page(const void *addr)
{
- if ((addr >= (void *)PAGE_OFFSET) &&
- (addr < high_memory))
+ if ((addr >= (void *)PAGE_OFFSET) && (addr < high_memory))
return virt_to_head_page(addr);
return NULL;
}
struct slab *kasan_addr_to_slab(const void *addr)
{
- if ((addr >= (void *)PAGE_OFFSET) &&
- (addr < high_memory))
+ if ((addr >= (void *)PAGE_OFFSET) && (addr < high_memory))
return virt_to_slab(addr);
return NULL;
}
--
2.25.1
From: Andrey Konovalov <[email protected]>
Use the kasan_addr_to_slab() helper in print_address_description()
instead of separately invoking PageSlab() and page_slab().
Reviewed-by: Marco Elver <[email protected]>
Signed-off-by: Andrey Konovalov <[email protected]>
---
mm/kasan/common.c | 7 +++++++
mm/kasan/report.c | 11 ++---------
2 files changed, 9 insertions(+), 9 deletions(-)
diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index f8e16a242197..50f4338b477f 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -30,6 +30,13 @@
#include "kasan.h"
#include "../slab.h"
+struct slab *kasan_addr_to_slab(const void *addr)
+{
+ if (virt_addr_valid(addr))
+ return virt_to_slab(addr);
+ return NULL;
+}
+
depot_stack_handle_t kasan_save_stack(gfp_t flags, bool can_alloc)
{
unsigned long entries[KASAN_STACK_DEPTH];
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index 570f9419b90c..cd31b3b89ca1 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -213,13 +213,6 @@ struct page *kasan_addr_to_page(const void *addr)
return NULL;
}
-struct slab *kasan_addr_to_slab(const void *addr)
-{
- if (virt_addr_valid(addr))
- return virt_to_slab(addr);
- return NULL;
-}
-
static void describe_object_addr(struct kmem_cache *cache, void *object,
const void *addr)
{
@@ -297,12 +290,12 @@ static inline bool init_task_stack_addr(const void *addr)
static void print_address_description(void *addr, u8 tag)
{
struct page *page = kasan_addr_to_page(addr);
+ struct slab *slab = kasan_addr_to_slab(addr);
dump_stack_lvl(KERN_ERR);
pr_err("\n");
- if (page && PageSlab(page)) {
- struct slab *slab = page_slab(page);
+ if (slab) {
struct kmem_cache *cache = slab->slab_cache;
void *object = nearest_obj(cache, slab, addr);
--
2.25.1
From: Andrey Konovalov <[email protected]>
Pass a pointer to kasan_report_info to describe_object() and
describe_object_stacks(), instead of passing the structure's fields.
The untagged pointer and the tag are still passed as separate arguments
to some of the functions to avoid duplicating the untagging logic.
This is preparatory change for the next patch.
Reviewed-by: Marco Elver <[email protected]>
Signed-off-by: Andrey Konovalov <[email protected]>
---
mm/kasan/report.c | 23 +++++++++++------------
1 file changed, 11 insertions(+), 12 deletions(-)
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index 763de8e68887..ec018f849992 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -213,8 +213,8 @@ static inline struct page *addr_to_page(const void *addr)
return NULL;
}
-static void describe_object_addr(struct kmem_cache *cache, void *object,
- const void *addr)
+static void describe_object_addr(const void *addr, struct kmem_cache *cache,
+ void *object)
{
unsigned long access_addr = (unsigned long)addr;
unsigned long object_addr = (unsigned long)object;
@@ -242,33 +242,32 @@ static void describe_object_addr(struct kmem_cache *cache, void *object,
(void *)(object_addr + cache->object_size));
}
-static void describe_object_stacks(struct kmem_cache *cache, void *object,
- const void *addr, u8 tag)
+static void describe_object_stacks(u8 tag, struct kasan_report_info *info)
{
struct kasan_track *alloc_track;
struct kasan_track *free_track;
- alloc_track = kasan_get_alloc_track(cache, object);
+ alloc_track = kasan_get_alloc_track(info->cache, info->object);
if (alloc_track) {
print_track(alloc_track, "Allocated");
pr_err("\n");
}
- free_track = kasan_get_free_track(cache, object, tag);
+ free_track = kasan_get_free_track(info->cache, info->object, tag);
if (free_track) {
print_track(free_track, "Freed");
pr_err("\n");
}
- kasan_print_aux_stacks(cache, object);
+ kasan_print_aux_stacks(info->cache, info->object);
}
-static void describe_object(struct kmem_cache *cache, void *object,
- const void *addr, u8 tag)
+static void describe_object(const void *addr, u8 tag,
+ struct kasan_report_info *info)
{
if (kasan_stack_collection_enabled())
- describe_object_stacks(cache, object, addr, tag);
- describe_object_addr(cache, object, addr);
+ describe_object_stacks(tag, info);
+ describe_object_addr(addr, info->cache, info->object);
}
static inline bool kernel_or_module_addr(const void *addr)
@@ -296,7 +295,7 @@ static void print_address_description(void *addr, u8 tag,
pr_err("\n");
if (info->cache && info->object) {
- describe_object(info->cache, info->object, addr, tag);
+ describe_object(addr, tag, info);
pr_err("\n");
}
--
2.25.1
From: Andrey Konovalov <[email protected]>
Instead of open-coding the validity checks for addr in
kasan_addr_to_page/slab(), use the virt_addr_valid() helper.
Reviewed-by: Marco Elver <[email protected]>
Signed-off-by: Andrey Konovalov <[email protected]>
---
Changes v1->v2:
- This is a new patch.
---
mm/kasan/report.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index 83f420a28c0b..570f9419b90c 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -208,14 +208,14 @@ static void print_track(struct kasan_track *track, const char *prefix)
struct page *kasan_addr_to_page(const void *addr)
{
- if ((addr >= (void *)PAGE_OFFSET) && (addr < high_memory))
+ if (virt_addr_valid(addr))
return virt_to_head_page(addr);
return NULL;
}
struct slab *kasan_addr_to_slab(const void *addr)
{
- if ((addr >= (void *)PAGE_OFFSET) && (addr < high_memory))
+ if (virt_addr_valid(addr))
return virt_to_slab(addr);
return NULL;
}
--
2.25.1
From: Andrey Konovalov <[email protected]>
Right now, kasan_cache_create() assigns SLAB_KASAN for all KASAN modes
and then sets up metadata-related cache parameters for the Generic mode.
SLAB_KASAN is used in two places:
1. In slab_ksize() to account for per-object metadata when
calculating the size of the accessible memory within the object.
2. In slab_common.c via kasan_never_merge() to prevent merging of
caches with per-object metadata.
Both cases are only relevant when per-object metadata is present, which
is only the case with the Generic mode.
Thus, assign SLAB_KASAN and define kasan_cache_create() only for the
Generic mode.
Also update the SLAB_KASAN-related comment.
Reviewed-by: Marco Elver <[email protected]>
Signed-off-by: Andrey Konovalov <[email protected]>
---
include/linux/kasan.h | 18 ++++++------------
include/linux/slab.h | 2 +-
mm/kasan/common.c | 16 ----------------
mm/kasan/generic.c | 17 ++++++++++++++++-
4 files changed, 23 insertions(+), 30 deletions(-)
diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index a212c2e3f32d..d811b3d7d2a1 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -128,15 +128,6 @@ static __always_inline void kasan_unpoison_pages(struct page *page,
__kasan_unpoison_pages(page, order, init);
}
-void __kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
- slab_flags_t *flags);
-static __always_inline void kasan_cache_create(struct kmem_cache *cache,
- unsigned int *size, slab_flags_t *flags)
-{
- if (kasan_enabled())
- __kasan_cache_create(cache, size, flags);
-}
-
void __kasan_cache_create_kmalloc(struct kmem_cache *cache);
static __always_inline void kasan_cache_create_kmalloc(struct kmem_cache *cache)
{
@@ -260,9 +251,6 @@ static inline void kasan_poison_pages(struct page *page, unsigned int order,
bool init) {}
static inline void kasan_unpoison_pages(struct page *page, unsigned int order,
bool init) {}
-static inline void kasan_cache_create(struct kmem_cache *cache,
- unsigned int *size,
- slab_flags_t *flags) {}
static inline void kasan_cache_create_kmalloc(struct kmem_cache *cache) {}
static inline void kasan_poison_slab(struct slab *slab) {}
static inline void kasan_unpoison_object_data(struct kmem_cache *cache,
@@ -316,6 +304,8 @@ static inline void kasan_unpoison_task_stack(struct task_struct *task) {}
size_t kasan_metadata_size(struct kmem_cache *cache);
slab_flags_t kasan_never_merge(void);
+void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
+ slab_flags_t *flags);
void kasan_cache_shrink(struct kmem_cache *cache);
void kasan_cache_shutdown(struct kmem_cache *cache);
@@ -334,6 +324,10 @@ static inline slab_flags_t kasan_never_merge(void)
{
return 0;
}
+/* And no cache-related metadata initialization is required. */
+static inline void kasan_cache_create(struct kmem_cache *cache,
+ unsigned int *size,
+ slab_flags_t *flags) {}
static inline void kasan_cache_shrink(struct kmem_cache *cache) {}
static inline void kasan_cache_shutdown(struct kmem_cache *cache) {}
diff --git a/include/linux/slab.h b/include/linux/slab.h
index 0fefdf528e0d..1c6b7362e82b 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -106,7 +106,7 @@
# define SLAB_ACCOUNT 0
#endif
-#ifdef CONFIG_KASAN
+#ifdef CONFIG_KASAN_GENERIC
#define SLAB_KASAN ((slab_flags_t __force)0x08000000U)
#else
#define SLAB_KASAN 0
diff --git a/mm/kasan/common.c b/mm/kasan/common.c
index c2690e938030..8efa63190951 100644
--- a/mm/kasan/common.c
+++ b/mm/kasan/common.c
@@ -110,22 +110,6 @@ void __kasan_poison_pages(struct page *page, unsigned int order, bool init)
KASAN_PAGE_FREE, init);
}
-void __kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
- slab_flags_t *flags)
-{
- /*
- * SLAB_KASAN is used to mark caches as ones that are sanitized by
- * KASAN. Currently this flag is used in two places:
- * 1. In slab_ksize() when calculating the size of the accessible
- * memory within the object.
- * 2. In slab_common.c to prevent merging of sanitized caches.
- */
- *flags |= SLAB_KASAN;
-
- if (kasan_requires_meta())
- kasan_init_cache_meta(cache, size);
-}
-
void __kasan_cache_create_kmalloc(struct kmem_cache *cache)
{
cache->kasan_info.is_kmalloc = true;
diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c
index 25333bf3c99f..f6bef347de87 100644
--- a/mm/kasan/generic.c
+++ b/mm/kasan/generic.c
@@ -352,11 +352,26 @@ static inline unsigned int optimal_redzone(unsigned int object_size)
object_size <= (1 << 16) - 1024 ? 1024 : 2048;
}
-void kasan_init_cache_meta(struct kmem_cache *cache, unsigned int *size)
+void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
+ slab_flags_t *flags)
{
unsigned int ok_size;
unsigned int optimal_size;
+ if (!kasan_requires_meta())
+ return;
+
+ /*
+ * SLAB_KASAN is used to mark caches that are sanitized by KASAN
+ * and that thus have per-object metadata.
+ * Currently this flag is used in two places:
+ * 1. In slab_ksize() to account for per-object metadata when
+ * calculating the size of the accessible memory within the object.
+ * 2. In slab_common.c via kasan_never_merge() to prevent merging of
+ * caches with per-object metadata.
+ */
+ *flags |= SLAB_KASAN;
+
ok_size = *size;
/* Add alloc meta into redzone. */
--
2.25.1
From: Andrey Konovalov <[email protected]>
Implement storing stack depot handles for alloc/free stack traces for
slab objects for the tag-based KASAN modes in a ring buffer.
This ring buffer is referred to as the stack ring.
On each alloc/free of a slab object, the tagged address of the object and
the current stack trace are recorded in the stack ring.
On each bug report, if the accessed address belongs to a slab object, the
stack ring is scanned for matching entries. The newest entries are used to
print the alloc/free stack traces in the report: one entry for alloc and
one for free.
The number of entries in the stack ring is fixed in this patch, but one of
the following patches adds a command-line argument to control it.
Signed-off-by: Andrey Konovalov <[email protected]>
---
Changes v2->v3:
- Drop redundant check for concurrent overwrites of stack ring entries.
Changes v1->v2:
- Only use the atomic type for pos, use READ/WRITE_ONCE() for the rest.
- Rename KASAN_STACK_RING_ENTRIES to KASAN_STACK_RING_SIZE.
- Rename object local variable in kasan_complete_mode_report_info() to
ptr to match the name in kasan_stack_ring_entry.
- Detect stack ring entry slots that are being written to.
- Use read-write lock to disallow reading half-written stack ring entries.
- Add a comment about the stack ring being best-effort.
---
mm/kasan/kasan.h | 21 +++++++++++++
mm/kasan/report_tags.c | 71 ++++++++++++++++++++++++++++++++++++++++++
mm/kasan/tags.c | 50 +++++++++++++++++++++++++++++
3 files changed, 142 insertions(+)
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index 7df107dc400a..cfff81139d67 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -2,6 +2,7 @@
#ifndef __MM_KASAN_KASAN_H
#define __MM_KASAN_KASAN_H
+#include <linux/atomic.h>
#include <linux/kasan.h>
#include <linux/kasan-tags.h>
#include <linux/kfence.h>
@@ -233,6 +234,26 @@ struct kasan_free_meta {
#endif /* CONFIG_KASAN_GENERIC */
+#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
+
+struct kasan_stack_ring_entry {
+ void *ptr;
+ size_t size;
+ u32 pid;
+ depot_stack_handle_t stack;
+ bool is_free;
+};
+
+#define KASAN_STACK_RING_SIZE (32 << 10)
+
+struct kasan_stack_ring {
+ rwlock_t lock;
+ atomic64_t pos;
+ struct kasan_stack_ring_entry entries[KASAN_STACK_RING_SIZE];
+};
+
+#endif /* CONFIG_KASAN_SW_TAGS || CONFIG_KASAN_HW_TAGS */
+
#if IS_ENABLED(CONFIG_KASAN_KUNIT_TEST)
/* Used in KUnit-compatible KASAN tests. */
struct kunit_kasan_status {
diff --git a/mm/kasan/report_tags.c b/mm/kasan/report_tags.c
index 5cbac2cdb177..1b78136542bb 100644
--- a/mm/kasan/report_tags.c
+++ b/mm/kasan/report_tags.c
@@ -4,8 +4,12 @@
* Copyright (c) 2020 Google, Inc.
*/
+#include <linux/atomic.h>
+
#include "kasan.h"
+extern struct kasan_stack_ring stack_ring;
+
static const char *get_bug_type(struct kasan_report_info *info)
{
/*
@@ -24,5 +28,72 @@ static const char *get_bug_type(struct kasan_report_info *info)
void kasan_complete_mode_report_info(struct kasan_report_info *info)
{
+ unsigned long flags;
+ u64 pos;
+ struct kasan_stack_ring_entry *entry;
+ void *ptr;
+ u32 pid;
+ depot_stack_handle_t stack;
+ bool is_free;
+ bool alloc_found = false, free_found = false;
+
info->bug_type = get_bug_type(info);
+
+ if (!info->cache || !info->object)
+ return;
+ }
+
+ write_lock_irqsave(&stack_ring.lock, flags);
+
+ pos = atomic64_read(&stack_ring.pos);
+
+ /*
+ * The loop below tries to find stack ring entries relevant to the
+ * buggy object. This is a best-effort process.
+ *
+ * First, another object with the same tag can be allocated in place of
+ * the buggy object. Also, since the number of entries is limited, the
+ * entries relevant to the buggy object can be overwritten.
+ */
+
+ for (u64 i = pos - 1; i != pos - 1 - KASAN_STACK_RING_SIZE; i--) {
+ if (alloc_found && free_found)
+ break;
+
+ entry = &stack_ring.entries[i % KASAN_STACK_RING_SIZE];
+
+ /* Paired with smp_store_release() in save_stack_info(). */
+ ptr = (void *)smp_load_acquire(&entry->ptr);
+
+ if (kasan_reset_tag(ptr) != info->object ||
+ get_tag(ptr) != get_tag(info->access_addr))
+ continue;
+
+ pid = READ_ONCE(entry->pid);
+ stack = READ_ONCE(entry->stack);
+ is_free = READ_ONCE(entry->is_free);
+
+ if (is_free) {
+ /*
+ * Second free of the same object.
+ * Give up on trying to find the alloc entry.
+ */
+ if (free_found)
+ break;
+
+ info->free_track.pid = pid;
+ info->free_track.stack = stack;
+ free_found = true;
+ } else {
+ /* Second alloc of the same object. Give up. */
+ if (alloc_found)
+ break;
+
+ info->alloc_track.pid = pid;
+ info->alloc_track.stack = stack;
+ alloc_found = true;
+ }
+ }
+
+ write_unlock_irqrestore(&stack_ring.lock, flags);
}
diff --git a/mm/kasan/tags.c b/mm/kasan/tags.c
index 39a0481e5228..07828021c1f5 100644
--- a/mm/kasan/tags.c
+++ b/mm/kasan/tags.c
@@ -6,6 +6,7 @@
* Copyright (c) 2020 Google, Inc.
*/
+#include <linux/atomic.h>
#include <linux/init.h>
#include <linux/kasan.h>
#include <linux/kernel.h>
@@ -16,11 +17,60 @@
#include <linux/types.h>
#include "kasan.h"
+#include "../slab.h"
+
+/* Non-zero, as initial pointer values are 0. */
+#define STACK_RING_BUSY_PTR ((void *)1)
+
+struct kasan_stack_ring stack_ring;
+
+static void save_stack_info(struct kmem_cache *cache, void *object,
+ gfp_t gfp_flags, bool is_free)
+{
+ unsigned long flags;
+ depot_stack_handle_t stack;
+ u64 pos;
+ struct kasan_stack_ring_entry *entry;
+ void *old_ptr;
+
+ stack = kasan_save_stack(gfp_flags, true);
+
+ /*
+ * Prevent save_stack_info() from modifying stack ring
+ * when kasan_complete_mode_report_info() is walking it.
+ */
+ read_lock_irqsave(&stack_ring.lock, flags);
+
+next:
+ pos = atomic64_fetch_add(1, &stack_ring.pos);
+ entry = &stack_ring.entries[pos % KASAN_STACK_RING_SIZE];
+
+ /* Detect stack ring entry slots that are being written to. */
+ old_ptr = READ_ONCE(entry->ptr);
+ if (old_ptr == STACK_RING_BUSY_PTR)
+ goto next; /* Busy slot. */
+ if (!try_cmpxchg(&entry->ptr, &old_ptr, STACK_RING_BUSY_PTR))
+ goto next; /* Busy slot. */
+
+ WRITE_ONCE(entry->size, cache->object_size);
+ WRITE_ONCE(entry->pid, current->pid);
+ WRITE_ONCE(entry->stack, stack);
+ WRITE_ONCE(entry->is_free, is_free);
+
+ /*
+ * Paired with smp_load_acquire() in kasan_complete_mode_report_info().
+ */
+ smp_store_release(&entry->ptr, (s64)object);
+
+ read_unlock_irqrestore(&stack_ring.lock, flags);
+}
void kasan_save_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags)
{
+ save_stack_info(cache, object, flags, false);
}
void kasan_save_free_info(struct kmem_cache *cache, void *object)
{
+ save_stack_info(cache, object, GFP_NOWAIT, true);
}
--
2.25.1
From: Andrey Konovalov <[email protected]>
Add a new use-after-free test that checks that KASAN detects use-after-free
when another object was allocated in the same slot.
This test is mainly relevant for the tag-based modes, which do not use
quarantine.
Once [1] is resolved, this test can be extended to check that the stack
traces in the report point to the proper kmalloc/kfree calls.
[1] https://bugzilla.kernel.org/show_bug.cgi?id=212203
Signed-off-by: Andrey Konovalov <[email protected]>
---
Changes v2->v3:
- This is a new patch.
---
lib/test_kasan.c | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
diff --git a/lib/test_kasan.c b/lib/test_kasan.c
index 58c1b01ccfe2..505f77ffad27 100644
--- a/lib/test_kasan.c
+++ b/lib/test_kasan.c
@@ -612,6 +612,29 @@ static void kmalloc_uaf2(struct kunit *test)
kfree(ptr2);
}
+/*
+ * Check that KASAN detects use-after-free when another object was allocated in
+ * the same slot. Relevant for the tag-based modes, which do not use quarantine.
+ */
+static void kmalloc_uaf3(struct kunit *test)
+{
+ char *ptr1, *ptr2;
+ size_t size = 100;
+
+ /* This test is specifically crafted for tag-based modes. */
+ KASAN_TEST_NEEDS_CONFIG_OFF(test, CONFIG_KASAN_GENERIC);
+
+ ptr1 = kmalloc(size, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr1);
+ kfree(ptr1);
+
+ ptr2 = kmalloc(size, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr2);
+ kfree(ptr2);
+
+ KUNIT_EXPECT_KASAN_FAIL(test, ((volatile char *)ptr1)[8]);
+}
+
static void kfree_via_page(struct kunit *test)
{
char *ptr;
@@ -1382,6 +1405,7 @@ static struct kunit_case kasan_kunit_test_cases[] = {
KUNIT_CASE(kmalloc_uaf),
KUNIT_CASE(kmalloc_uaf_memset),
KUNIT_CASE(kmalloc_uaf2),
+ KUNIT_CASE(kmalloc_uaf3),
KUNIT_CASE(kfree_via_page),
KUNIT_CASE(kfree_via_phys),
KUNIT_CASE(kmem_cache_oob),
--
2.25.1
From: Andrey Konovalov <[email protected]>
Add support for the kasan.stacktrace command-line argument for Software
Tag-Based KASAN.
The following patch adds a command-line argument for selecting the stack
ring size, and, as the stack ring is supported by both the Software and
the Hardware Tag-Based KASAN modes, it is natural that both of them have
support for kasan.stacktrace too.
Reviewed-by: Marco Elver <[email protected]>
Signed-off-by: Andrey Konovalov <[email protected]>
---
Changes v1->v2:
- This is a new patch.
---
Documentation/dev-tools/kasan.rst | 15 ++++++-----
mm/kasan/hw_tags.c | 39 +---------------------------
mm/kasan/kasan.h | 36 +++++++++++++++++---------
mm/kasan/sw_tags.c | 5 +++-
mm/kasan/tags.c | 43 +++++++++++++++++++++++++++++++
5 files changed, 81 insertions(+), 57 deletions(-)
diff --git a/Documentation/dev-tools/kasan.rst b/Documentation/dev-tools/kasan.rst
index 1772fd457fed..7bd38c181018 100644
--- a/Documentation/dev-tools/kasan.rst
+++ b/Documentation/dev-tools/kasan.rst
@@ -111,9 +111,15 @@ parameter can be used to control panic and reporting behaviour:
report or also panic the kernel (default: ``report``). The panic happens even
if ``kasan_multi_shot`` is enabled.
-Hardware Tag-Based KASAN mode (see the section about various modes below) is
-intended for use in production as a security mitigation. Therefore, it supports
-additional boot parameters that allow disabling KASAN or controlling features:
+Software and Hardware Tag-Based KASAN modes (see the section about various
+modes below) support disabling stack trace collection:
+
+- ``kasan.stacktrace=off`` or ``=on`` disables or enables alloc and free stack
+ traces collection (default: ``on``).
+
+Hardware Tag-Based KASAN mode is intended for use in production as a security
+mitigation. Therefore, it supports additional boot parameters that allow
+disabling KASAN altogether or controlling its features:
- ``kasan=off`` or ``=on`` controls whether KASAN is enabled (default: ``on``).
@@ -132,9 +138,6 @@ additional boot parameters that allow disabling KASAN or controlling features:
- ``kasan.vmalloc=off`` or ``=on`` disables or enables tagging of vmalloc
allocations (default: ``on``).
-- ``kasan.stacktrace=off`` or ``=on`` disables or enables alloc and free stack
- traces collection (default: ``on``).
-
Error reports
~~~~~~~~~~~~~
diff --git a/mm/kasan/hw_tags.c b/mm/kasan/hw_tags.c
index 9ad8eff71b28..b22c4f461cb0 100644
--- a/mm/kasan/hw_tags.c
+++ b/mm/kasan/hw_tags.c
@@ -38,16 +38,9 @@ enum kasan_arg_vmalloc {
KASAN_ARG_VMALLOC_ON,
};
-enum kasan_arg_stacktrace {
- KASAN_ARG_STACKTRACE_DEFAULT,
- KASAN_ARG_STACKTRACE_OFF,
- KASAN_ARG_STACKTRACE_ON,
-};
-
static enum kasan_arg kasan_arg __ro_after_init;
static enum kasan_arg_mode kasan_arg_mode __ro_after_init;
static enum kasan_arg_vmalloc kasan_arg_vmalloc __initdata;
-static enum kasan_arg_stacktrace kasan_arg_stacktrace __initdata;
/*
* Whether KASAN is enabled at all.
@@ -66,9 +59,6 @@ EXPORT_SYMBOL_GPL(kasan_mode);
/* Whether to enable vmalloc tagging. */
DEFINE_STATIC_KEY_TRUE(kasan_flag_vmalloc);
-/* Whether to collect alloc/free stack traces. */
-DEFINE_STATIC_KEY_TRUE(kasan_flag_stacktrace);
-
/* kasan=off/on */
static int __init early_kasan_flag(char *arg)
{
@@ -122,23 +112,6 @@ static int __init early_kasan_flag_vmalloc(char *arg)
}
early_param("kasan.vmalloc", early_kasan_flag_vmalloc);
-/* kasan.stacktrace=off/on */
-static int __init early_kasan_flag_stacktrace(char *arg)
-{
- if (!arg)
- return -EINVAL;
-
- if (!strcmp(arg, "off"))
- kasan_arg_stacktrace = KASAN_ARG_STACKTRACE_OFF;
- else if (!strcmp(arg, "on"))
- kasan_arg_stacktrace = KASAN_ARG_STACKTRACE_ON;
- else
- return -EINVAL;
-
- return 0;
-}
-early_param("kasan.stacktrace", early_kasan_flag_stacktrace);
-
static inline const char *kasan_mode_info(void)
{
if (kasan_mode == KASAN_MODE_ASYNC)
@@ -213,17 +186,7 @@ void __init kasan_init_hw_tags(void)
break;
}
- switch (kasan_arg_stacktrace) {
- case KASAN_ARG_STACKTRACE_DEFAULT:
- /* Default is specified by kasan_flag_stacktrace definition. */
- break;
- case KASAN_ARG_STACKTRACE_OFF:
- static_branch_disable(&kasan_flag_stacktrace);
- break;
- case KASAN_ARG_STACKTRACE_ON:
- static_branch_enable(&kasan_flag_stacktrace);
- break;
- }
+ kasan_init_tags();
/* KASAN is now initialized, enable it. */
static_branch_enable(&kasan_flag_enabled);
diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h
index cfff81139d67..447baf1a7a2e 100644
--- a/mm/kasan/kasan.h
+++ b/mm/kasan/kasan.h
@@ -8,13 +8,31 @@
#include <linux/kfence.h>
#include <linux/stackdepot.h>
-#ifdef CONFIG_KASAN_HW_TAGS
+#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
#include <linux/static_key.h>
+
+DECLARE_STATIC_KEY_TRUE(kasan_flag_stacktrace);
+
+static inline bool kasan_stack_collection_enabled(void)
+{
+ return static_branch_unlikely(&kasan_flag_stacktrace);
+}
+
+#else /* CONFIG_KASAN_SW_TAGS || CONFIG_KASAN_HW_TAGS */
+
+static inline bool kasan_stack_collection_enabled(void)
+{
+ return true;
+}
+
+#endif /* CONFIG_KASAN_SW_TAGS || CONFIG_KASAN_HW_TAGS */
+
+#ifdef CONFIG_KASAN_HW_TAGS
+
#include "../slab.h"
DECLARE_STATIC_KEY_TRUE(kasan_flag_vmalloc);
-DECLARE_STATIC_KEY_TRUE(kasan_flag_stacktrace);
enum kasan_mode {
KASAN_MODE_SYNC,
@@ -29,11 +47,6 @@ static inline bool kasan_vmalloc_enabled(void)
return static_branch_likely(&kasan_flag_vmalloc);
}
-static inline bool kasan_stack_collection_enabled(void)
-{
- return static_branch_unlikely(&kasan_flag_stacktrace);
-}
-
static inline bool kasan_async_fault_possible(void)
{
return kasan_mode == KASAN_MODE_ASYNC || kasan_mode == KASAN_MODE_ASYMM;
@@ -46,11 +59,6 @@ static inline bool kasan_sync_fault_possible(void)
#else /* CONFIG_KASAN_HW_TAGS */
-static inline bool kasan_stack_collection_enabled(void)
-{
- return true;
-}
-
static inline bool kasan_async_fault_possible(void)
{
return false;
@@ -410,6 +418,10 @@ static inline void kasan_enable_tagging(void) { }
#endif /* CONFIG_KASAN_HW_TAGS */
+#if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
+void __init kasan_init_tags(void);
+#endif /* CONFIG_KASAN_SW_TAGS || CONFIG_KASAN_HW_TAGS */
+
#if defined(CONFIG_KASAN_HW_TAGS) && IS_ENABLED(CONFIG_KASAN_KUNIT_TEST)
void kasan_force_async_fault(void);
diff --git a/mm/kasan/sw_tags.c b/mm/kasan/sw_tags.c
index 77f13f391b57..a3afaf2ad1b1 100644
--- a/mm/kasan/sw_tags.c
+++ b/mm/kasan/sw_tags.c
@@ -42,7 +42,10 @@ void __init kasan_init_sw_tags(void)
for_each_possible_cpu(cpu)
per_cpu(prng_state, cpu) = (u32)get_cycles();
- pr_info("KernelAddressSanitizer initialized (sw-tags)\n");
+ kasan_init_tags();
+
+ pr_info("KernelAddressSanitizer initialized (sw-tags, stacktrace=%s)\n",
+ kasan_stack_collection_enabled() ? "on" : "off");
}
/*
diff --git a/mm/kasan/tags.c b/mm/kasan/tags.c
index 07828021c1f5..0eb6cf6717db 100644
--- a/mm/kasan/tags.c
+++ b/mm/kasan/tags.c
@@ -19,11 +19,54 @@
#include "kasan.h"
#include "../slab.h"
+enum kasan_arg_stacktrace {
+ KASAN_ARG_STACKTRACE_DEFAULT,
+ KASAN_ARG_STACKTRACE_OFF,
+ KASAN_ARG_STACKTRACE_ON,
+};
+
+static enum kasan_arg_stacktrace kasan_arg_stacktrace __initdata;
+
+/* Whether to collect alloc/free stack traces. */
+DEFINE_STATIC_KEY_TRUE(kasan_flag_stacktrace);
+
/* Non-zero, as initial pointer values are 0. */
#define STACK_RING_BUSY_PTR ((void *)1)
struct kasan_stack_ring stack_ring;
+/* kasan.stacktrace=off/on */
+static int __init early_kasan_flag_stacktrace(char *arg)
+{
+ if (!arg)
+ return -EINVAL;
+
+ if (!strcmp(arg, "off"))
+ kasan_arg_stacktrace = KASAN_ARG_STACKTRACE_OFF;
+ else if (!strcmp(arg, "on"))
+ kasan_arg_stacktrace = KASAN_ARG_STACKTRACE_ON;
+ else
+ return -EINVAL;
+
+ return 0;
+}
+early_param("kasan.stacktrace", early_kasan_flag_stacktrace);
+
+void __init kasan_init_tags(void)
+{
+ switch (kasan_arg_stacktrace) {
+ case KASAN_ARG_STACKTRACE_DEFAULT:
+ /* Default is specified by kasan_flag_stacktrace definition. */
+ break;
+ case KASAN_ARG_STACKTRACE_OFF:
+ static_branch_disable(&kasan_flag_stacktrace);
+ break;
+ case KASAN_ARG_STACKTRACE_ON:
+ static_branch_enable(&kasan_flag_stacktrace);
+ break;
+ }
+}
+
static void save_stack_info(struct kmem_cache *cache, void *object,
gfp_t gfp_flags, bool is_free)
{
--
2.25.1
On Mon, Sep 5, 2022 at 11:05 PM <[email protected]> wrote:
>
> From: Andrey Konovalov <[email protected]>
>
> This series makes the tag-based KASAN modes use a ring buffer for storing
> stack depot handles for alloc/free stack traces for slab objects instead
> of per-object metadata. This ring buffer is referred to as the stack ring.
>
> On each alloc/free of a slab object, the tagged address of the object and
> the current stack trace are recorded in the stack ring.
>
> On each bug report, if the accessed address belongs to a slab object, the
> stack ring is scanned for matching entries. The newest entries are used to
> print the alloc/free stack traces in the report: one entry for alloc and
> one for free.
>
> The advantages of this approach over storing stack trace handles in
> per-object metadata with the tag-based KASAN modes:
>
> - Allows to find relevant stack traces for use-after-free bugs without
> using quarantine for freed memory. (Currently, if the object was
> reallocated multiple times, the report contains the latest alloc/free
> stack traces, not necessarily the ones relevant to the buggy allocation.)
> - Allows to better identify and mark use-after-free bugs, effectively
> making the CONFIG_KASAN_TAGS_IDENTIFY functionality always-on.
> - Has fixed memory overhead.
>
> The disadvantage:
>
> - If the affected object was allocated/freed long before the bug happened
> and the stack trace events were purged from the stack ring, the report
> will have no stack traces.
>
> Discussion
> ==========
>
> The proposed implementation of the stack ring uses a single ring buffer for
> the whole kernel. This might lead to contention due to atomic accesses to
> the ring buffer index on multicore systems.
>
> At this point, it is unknown whether the performance impact from this
> contention would be significant compared to the slowdown introduced by
> collecting stack traces due to the planned changes to the latter part,
> see the section below.
>
> For now, the proposed implementation is deemed to be good enough, but this
> might need to be revisited once the stack collection becomes faster.
>
> A considered alternative is to keep a separate ring buffer for each CPU
> and then iterate over all of them when printing a bug report. This approach
> requires somehow figuring out which of the stack rings has the freshest
> stack traces for an object if multiple stack rings have them.
>
> Further plans
> =============
>
> This series is a part of an effort to make KASAN stack trace collection
> suitable for production. This requires stack trace collection to be fast
> and memory-bounded.
>
> The planned steps are:
>
> 1. Speed up stack trace collection (potentially, by using SCS;
> patches on-hold until steps #2 and #3 are completed).
> 2. Keep stack trace handles in the stack ring (this series).
> 3. Add a memory-bounded mode to stack depot or provide an alternative
> memory-bounded stack storage.
> 4. Potentially, implement stack trace collection sampling to minimize
> the performance impact.
>
> Thanks!
Hi Andrew,
Could you consider picking up this series into mm?
Most of the patches have a Reviewed-by tag from Marco, and I've
addressed the last few comments he had in v3.
Thanks!
On Sun, 11 Sept 2022 at 13:50, Andrey Konovalov <[email protected]> wrote:
>
> On Mon, Sep 5, 2022 at 11:05 PM <[email protected]> wrote:
> >
> > From: Andrey Konovalov <[email protected]>
> >
> > This series makes the tag-based KASAN modes use a ring buffer for storing
> > stack depot handles for alloc/free stack traces for slab objects instead
> > of per-object metadata. This ring buffer is referred to as the stack ring.
> >
> > On each alloc/free of a slab object, the tagged address of the object and
> > the current stack trace are recorded in the stack ring.
> >
> > On each bug report, if the accessed address belongs to a slab object, the
> > stack ring is scanned for matching entries. The newest entries are used to
> > print the alloc/free stack traces in the report: one entry for alloc and
> > one for free.
> >
> > The advantages of this approach over storing stack trace handles in
> > per-object metadata with the tag-based KASAN modes:
> >
> > - Allows to find relevant stack traces for use-after-free bugs without
> > using quarantine for freed memory. (Currently, if the object was
> > reallocated multiple times, the report contains the latest alloc/free
> > stack traces, not necessarily the ones relevant to the buggy allocation.)
> > - Allows to better identify and mark use-after-free bugs, effectively
> > making the CONFIG_KASAN_TAGS_IDENTIFY functionality always-on.
> > - Has fixed memory overhead.
> >
> > The disadvantage:
> >
> > - If the affected object was allocated/freed long before the bug happened
> > and the stack trace events were purged from the stack ring, the report
> > will have no stack traces.
> >
> > Discussion
> > ==========
> >
> > The proposed implementation of the stack ring uses a single ring buffer for
> > the whole kernel. This might lead to contention due to atomic accesses to
> > the ring buffer index on multicore systems.
> >
> > At this point, it is unknown whether the performance impact from this
> > contention would be significant compared to the slowdown introduced by
> > collecting stack traces due to the planned changes to the latter part,
> > see the section below.
> >
> > For now, the proposed implementation is deemed to be good enough, but this
> > might need to be revisited once the stack collection becomes faster.
> >
> > A considered alternative is to keep a separate ring buffer for each CPU
> > and then iterate over all of them when printing a bug report. This approach
> > requires somehow figuring out which of the stack rings has the freshest
> > stack traces for an object if multiple stack rings have them.
> >
> > Further plans
> > =============
> >
> > This series is a part of an effort to make KASAN stack trace collection
> > suitable for production. This requires stack trace collection to be fast
> > and memory-bounded.
> >
> > The planned steps are:
> >
> > 1. Speed up stack trace collection (potentially, by using SCS;
> > patches on-hold until steps #2 and #3 are completed).
> > 2. Keep stack trace handles in the stack ring (this series).
> > 3. Add a memory-bounded mode to stack depot or provide an alternative
> > memory-bounded stack storage.
> > 4. Potentially, implement stack trace collection sampling to minimize
> > the performance impact.
> >
> > Thanks!
>
> Hi Andrew,
>
> Could you consider picking up this series into mm?
>
> Most of the patches have a Reviewed-by tag from Marco, and I've
> addressed the last few comments he had in v3.
>
> Thanks!
I see them in -next, so they've been picked up?
FWIW, my concerns have been addressed, so for patches that don't yet
have my Reviewed:
Acked-by: Marco Elver <[email protected]>
On Mon, 12 Sep 2022 11:39:07 +0200 Marco Elver <[email protected]> wrote:
>
> ...
>
> > Hi Andrew,
> >
> > Could you consider picking up this series into mm?
> >
> > Most of the patches have a Reviewed-by tag from Marco, and I've
> > addressed the last few comments he had in v3.
> >
> > Thanks!
>
> I see them in -next, so they've been picked up?
yup.
> FWIW, my concerns have been addressed, so for patches that don't yet
> have my Reviewed:
>
>
> Acked-by: Marco Elver <[email protected]>
Updated, thanks.
On Mon, Sep 12, 2022 at 2:06 PM Andrew Morton <[email protected]> wrote:
>
> On Mon, 12 Sep 2022 11:39:07 +0200 Marco Elver <[email protected]> wrote:
>
> >
> > ...
> >
> > > Hi Andrew,
> > >
> > > Could you consider picking up this series into mm?
> > >
> > > Most of the patches have a Reviewed-by tag from Marco, and I've
> > > addressed the last few comments he had in v3.
> > >
> > > Thanks!
> >
> > I see them in -next, so they've been picked up?
>
> yup.
>
> > FWIW, my concerns have been addressed, so for patches that don't yet
> > have my Reviewed:
> >
> >
> > Acked-by: Marco Elver <[email protected]>
>
> Updated, thanks.
Hit the following with the latest mm-unstable. Please take a look. Thanks.
BUG: rwlock bad magic on CPU#0, swapper/0, ffffffdc589d8218
CPU: 0 PID: 0 Comm: swapper Not tainted 6.0.0-rc3-lockdep+ #36
Call trace:
dump_backtrace+0xfc/0x14c
show_stack+0x24/0x58
dump_stack_lvl+0x7c/0xa0
dump_stack+0x18/0x44
rwlock_bug+0x88/0x8c
do_raw_read_unlock+0x7c/0x90
_raw_read_unlock_irqrestore+0x54/0xa0
save_stack_info+0x100/0x118
kasan_save_alloc_info+0x20/0x2c
__kasan_slab_alloc+0x90/0x94
early_kmem_cache_node_alloc+0x8c/0x1a8
__kmem_cache_create+0x1ac/0x338
create_boot_cache+0xac/0xec
kmem_cache_init+0x8c/0x174
mm_init+0x3c/0x78
start_kernel+0x188/0x49c
On Mon, Sep 19, 2022 at 10:08 AM Yu Zhao <[email protected]> wrote:
>
> Hit the following with the latest mm-unstable. Please take a look. Thanks.
>
> BUG: rwlock bad magic on CPU#0, swapper/0, ffffffdc589d8218
> CPU: 0 PID: 0 Comm: swapper Not tainted 6.0.0-rc3-lockdep+ #36
> Call trace:
> dump_backtrace+0xfc/0x14c
> show_stack+0x24/0x58
> dump_stack_lvl+0x7c/0xa0
> dump_stack+0x18/0x44
> rwlock_bug+0x88/0x8c
> do_raw_read_unlock+0x7c/0x90
> _raw_read_unlock_irqrestore+0x54/0xa0
> save_stack_info+0x100/0x118
> kasan_save_alloc_info+0x20/0x2c
> __kasan_slab_alloc+0x90/0x94
> early_kmem_cache_node_alloc+0x8c/0x1a8
> __kmem_cache_create+0x1ac/0x338
> create_boot_cache+0xac/0xec
> kmem_cache_init+0x8c/0x174
> mm_init+0x3c/0x78
> start_kernel+0x188/0x49c
Hi Yu,
Just mailed a fix.
Thank you for the report!