Received: by 2002:a05:6358:45e:b0:b5:b6eb:e1f9 with SMTP id 30csp4714829rwe; Tue, 30 Aug 2022 15:47:55 -0700 (PDT) X-Google-Smtp-Source: AA6agR4dqKG/FJmdZqGEELf2zFF1pF0aoT++iY4QwcXI5oP/EZ54S4B1MkaupSo4iXu3U8xR6yS0 X-Received: by 2002:a05:6402:5008:b0:440:941a:93c3 with SMTP id p8-20020a056402500800b00440941a93c3mr22289503eda.47.1661899674795; Tue, 30 Aug 2022 15:47:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1661899674; cv=none; d=google.com; s=arc-20160816; b=Z8qtxxSmE7er0IUlERhYqCy/VB98Xk4r9K4GOR+2Knz2pn5Yd3y7DBhE4Hl8nJoGYi VI04MIFXW8yTX7W49ZFt7mmUi/2qpHgKHF4CVxflHohrRq1c+6OxJAuxKKGv+55MUW19 vL7ne2bPuM6saKgSVefQI+IHmQE6EdNPEvdE8XvA74wwLSUriCTYr1lBLchpKUvfKAGR M2FOPsf0z1TlLcnzDcYQYRXS7Po0fWRoBPLB+E+TS6GyDLMzNgtD5E85w6PQytDROmYZ fNJZyIoxGX8MA2jfBXuo/GuG6M6suJdmkDHCO5AoW2Lv9RY12T1Zci3fnOi7vb9FBc4D tnNQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=w9gVcgQRdZitDZxjqrDoJq6Gs/0aEjUQjbVWDU06BvI=; b=r4NnfDBJOgMScg3ZCxpFhfR5pSTchGTePNXAQQaiR800uM7sP6uXpQfqoXs5IHD27c /79Gqwv6K4YgSc5Vv2bhevGzH1Cc5Y8MhDjFzHIxDqIGbO9dx1+CRGNSJArDSy5SxiVg p685teTHvO5GKz3Rml7172nyZixmlqDWdzu+YW6jpjoKhpWjLxV+C61EVk8BvNoK8gQ8 QjYPaYDw78N6KrTOH2WT6ChXRRZYAQAtnYwBBoQYGafwFMHJej79ptgSK8iSATWyp2+c EjEkvQLDcuIw24oRoqrgG/GOfnqokYDYvw8/y62d3zsJAQQImuM3PVqM8Edx3tZ/SED4 6XLg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=Q1PzL2bT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hb44-20020a170907162c00b0073fb5d40522si8994749ejc.142.2022.08.30.15.47.27; Tue, 30 Aug 2022 15:47:54 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=Q1PzL2bT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232115AbiH3VwK (ORCPT + 99 others); Tue, 30 Aug 2022 17:52:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58658 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232062AbiH3Vv0 (ORCPT ); Tue, 30 Aug 2022 17:51:26 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E968725C66 for ; Tue, 30 Aug 2022 14:50:04 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-33dbe61eed8so188495057b3.1 for ; Tue, 30 Aug 2022 14:50:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc; bh=w9gVcgQRdZitDZxjqrDoJq6Gs/0aEjUQjbVWDU06BvI=; b=Q1PzL2bThBJ4iiSWL0d7IheB5VfLNxWH1gqUohex8OE2QSiGR/fpg9gqy0yixW+Isb pxJFdeTMvRwdBwAcg7flVNtX1j9eJ9C9tS/tDg05AImXSgWHVN2GUlakKwYKuvgZhIEZ 53DiKTHKeM16TZHvnMmCYNdJyP0y2AnTT97Oi3Zv6WKFja6rmmzzL6Teq1ArHTNPI2TC NVjO3wFAVKlA4rp9aDdNwzT8IVNC+AXM7IGd6mywh1LPpaNIM2cA8K+l/DF+pItxSR+n jG8tcEuC2nhcd/7nioVG87S+Xo+Y8ZaGYAgx8HTdNWhJ4JbCh9BURQy523rvt+lhf5xu HCDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc; bh=w9gVcgQRdZitDZxjqrDoJq6Gs/0aEjUQjbVWDU06BvI=; b=rF7KHPc5dDejijQZHQRjezHRDOv9Lz5i04dy0FinFam23MS1qej+C/lyn4CmQBWz/C U8RJ2Z/mroDZqFdUzqILxbfpcBw/affTTYE855EIYwCjQmbYdgE5170ZBIa6AyIiuzS9 qHCSGiX813QxKAfgBtEJu38HbglGJIi1WQlwNF58ihMFo52bafVE2/Dl1L8jqiM+wB1p T4X5uybJ5agx9Mg3D/AIGJHuSrgAVj4ddATfXVlzjy/a5mUM/JPcsLfMEtAss9dOlnBo +bo2jmOFIPcQTiSyeYD1YuND/8UF0Q1Xqf1/1n+D9HR/jO5lhmKPSaayv1x31Oa1z8eK QLng== X-Gm-Message-State: ACgBeo0iRnum+tpy7lbsrIwgSAdWAXYocdFjOco9UjEfOeOzka1TIfLl zEXu+ChCsQ2Ak7N71nJn+7N/JQSvvaA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:a005:55b3:6c26:b3e4]) (user=surenb job=sendgmr) by 2002:a25:e6c6:0:b0:695:f4dc:8c4f with SMTP id d189-20020a25e6c6000000b00695f4dc8c4fmr13235724ybh.329.1661896203524; Tue, 30 Aug 2022 14:50:03 -0700 (PDT) Date: Tue, 30 Aug 2022 14:49:04 -0700 In-Reply-To: <20220830214919.53220-1-surenb@google.com> Mime-Version: 1.0 References: <20220830214919.53220-1-surenb@google.com> X-Mailer: git-send-email 2.37.2.672.g94769d06f0-goog Message-ID: <20220830214919.53220-16-surenb@google.com> Subject: [RFC PATCH 15/30] lib: introduce slab allocation tagging From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, changbin.du@intel.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, arnd@arndb.de, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-mm@kvack.org, iommu@lists.linux.dev, kasan-dev@googlegroups.com, io-uring@vger.kernel.org, linux-arch@vger.kernel.org, xen-devel@lists.xenproject.org, linux-bcache@vger.kernel.org, linux-modules@vger.kernel.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Introduce CONFIG_SLAB_ALLOC_TAGGING which provides helper functions to easily instrument slab allocators and adds a codetag_ref field into slabobj_ext to store a pointer to the allocation tag associated with the code that allocated the slab object. Signed-off-by: Suren Baghdasaryan Co-developed-by: Kent Overstreet Signed-off-by: Kent Overstreet --- include/linux/memcontrol.h | 5 +++++ include/linux/slab.h | 25 +++++++++++++++++++++++++ include/linux/slab_def.h | 2 +- include/linux/slub_def.h | 4 ++-- lib/Kconfig.debug | 11 +++++++++++ mm/slab_common.c | 33 +++++++++++++++++++++++++++++++++ 6 files changed, 77 insertions(+), 3 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 315399f77173..97c0153f0247 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -232,7 +232,12 @@ struct obj_cgroup { * if MEMCG_DATA_OBJEXTS is set. */ struct slabobj_ext { +#ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *objcg; +#endif +#ifdef CONFIG_SLAB_ALLOC_TAGGING + union codetag_ref ref; +#endif } __aligned(8); /* diff --git a/include/linux/slab.h b/include/linux/slab.h index 55ae3ea864a4..5a198aa02a08 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -438,6 +438,31 @@ static __always_inline unsigned int __kmalloc_index(size_t size, #define kmalloc_index(s) __kmalloc_index(s, true) #endif /* !CONFIG_SLOB */ +#ifdef CONFIG_SLAB_ALLOC_TAGGING + +#include + +union codetag_ref *get_slab_tag_ref(const void *objp); + +#define slab_tag_add(_old, _new) \ +do { \ + if (!ZERO_OR_NULL_PTR(_new) && _old != _new) \ + alloc_tag_add(get_slab_tag_ref(_new), __ksize(_new)); \ +} while (0) + +static inline void slab_tag_dec(const void *ptr) +{ + if (!ZERO_OR_NULL_PTR(ptr)) + alloc_tag_sub(get_slab_tag_ref(ptr), __ksize(ptr)); +} + +#else + +#define slab_tag_add(_old, _new) do {} while (0) +static inline void slab_tag_dec(const void *ptr) {} + +#endif + void *__kmalloc(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_size(1); void *kmem_cache_alloc(struct kmem_cache *s, gfp_t flags) __assume_slab_alignment __malloc; void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, diff --git a/include/linux/slab_def.h b/include/linux/slab_def.h index e24c9aff6fed..25feb5f7dc32 100644 --- a/include/linux/slab_def.h +++ b/include/linux/slab_def.h @@ -106,7 +106,7 @@ static inline void *nearest_obj(struct kmem_cache *cache, const struct slab *sla * reciprocal_divide(offset, cache->reciprocal_buffer_size) */ static inline unsigned int obj_to_index(const struct kmem_cache *cache, - const struct slab *slab, void *obj) + const struct slab *slab, const void *obj) { u32 offset = (obj - slab->s_mem); return reciprocal_divide(offset, cache->reciprocal_buffer_size); diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index f9c68a9dac04..940c146768d4 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -170,14 +170,14 @@ static inline void *nearest_obj(struct kmem_cache *cache, const struct slab *sla /* Determine object index from a given position */ static inline unsigned int __obj_to_index(const struct kmem_cache *cache, - void *addr, void *obj) + void *addr, const void *obj) { return reciprocal_divide(kasan_reset_tag(obj) - addr, cache->reciprocal_size); } static inline unsigned int obj_to_index(const struct kmem_cache *cache, - const struct slab *slab, void *obj) + const struct slab *slab, const void *obj) { if (is_kfence_address(obj)) return 0; diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 6686648843b3..08c97a978906 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -989,6 +989,17 @@ config PAGE_ALLOC_TAGGING initiated at that code location. The mechanism can be used to track memory leaks with a low performance impact. +config SLAB_ALLOC_TAGGING + bool "Enable slab allocation tagging" + default n + select ALLOC_TAGGING + select SLAB_OBJ_EXT + help + Instrument slab allocators to track allocation source code and + collect statistics on the number of allocations and their total size + initiated at that code location. The mechanism can be used to track + memory leaks with a low performance impact. + source "lib/Kconfig.kasan" source "lib/Kconfig.kfence" diff --git a/mm/slab_common.c b/mm/slab_common.c index 17996649cfe3..272eda62ecaa 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -202,6 +202,39 @@ struct kmem_cache *find_mergeable(unsigned int size, unsigned int align, return NULL; } +#ifdef CONFIG_SLAB_ALLOC_TAGGING + +union codetag_ref *get_slab_tag_ref(const void *objp) +{ + struct slabobj_ext *obj_exts; + union codetag_ref *res = NULL; + struct slab *slab; + unsigned int off; + + slab = virt_to_slab(objp); + /* + * We could be given a kmalloc_large() object, skip those. They use + * alloc_pages and can be tracked by page allocation tracking. + */ + if (!slab) + goto out; + + obj_exts = slab_obj_exts(slab); + if (!obj_exts) + goto out; + + if (!slab->slab_cache) + goto out; + + off = obj_to_index(slab->slab_cache, slab, objp); + res = &obj_exts[off].ref; +out: + return res; +} +EXPORT_SYMBOL(get_slab_tag_ref); + +#endif /* CONFIG_SLAB_ALLOC_TAGGING */ + static struct kmem_cache *create_cache(const char *name, unsigned int object_size, unsigned int align, slab_flags_t flags, unsigned int useroffset, -- 2.37.2.672.g94769d06f0-goog