Received-SPF: pass (google.com: domain of linux-kernel+bounces-65842-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <4f24986587b53be3f9ece187a3105774eb27c12f.camel@linux.intel.com>
Subject: Re: [PATCH v3 00/35] Memory allocation profiling
From: Tim Chen <tim.c.chen@linux.intel.com>
To: Suren Baghdasaryan <surenb@google.com>, akpm@linux-foundation.org
Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, 
 hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de,
 dave@stgolabs.net,  willy@infradead.org, liam.howlett@oracle.com,
 corbet@lwn.net, void@manifault.com,  peterz@infradead.org,
 juri.lelli@redhat.com, catalin.marinas@arm.com,  will@kernel.org,
 arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, 
 dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com,
 david@redhat.com,  axboe@kernel.dk, mcgrof@kernel.org,
 masahiroy@kernel.org, nathan@kernel.org,  dennis@kernel.org, tj@kernel.org,
 muchun.song@linux.dev, rppt@kernel.org,  paulmck@kernel.org,
 pasha.tatashin@soleen.com, yosryahmed@google.com,  yuzhao@google.com,
 dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, 
 keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, 
 gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, 
 vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, 
 bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, 
 penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com,
 glider@google.com,  elver@google.com, dvyukov@google.com,
 shakeelb@google.com,  songmuchun@bytedance.com, jbaron@akamai.com,
 rientjes@google.com,  minchan@google.com, kaleshsingh@google.com,
 kernel-team@android.com,  linux-doc@vger.kernel.org,
 linux-kernel@vger.kernel.org, iommu@lists.linux.dev, 
 linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org,
 linux-mm@kvack.org,  linux-modules@vger.kernel.org,
 kasan-dev@googlegroups.com,  cgroups@vger.kernel.org
Date: Wed, 14 Feb 2024 10:53:53 -0800
In-Reply-To: <20240212213922.783301-1-surenb@google.com>
References: <20240212213922.783301-1-surenb@google.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.44.4 (3.44.4-2.fc36) 
Precedence: bulk
MIME-Version: 1.0

On Mon, 2024-02-12 at 13:38 -0800, Suren Baghdasaryan wrote:
> Memory allocation, v3 and final:
>=20
> Overview:
> Low overhead [1] per-callsite memory allocation profiling. Not just for d=
ebug
> kernels, overhead low enough to be deployed in production.
>=20
> We're aiming to get this in the next merge window, for 6.9. The feedback
> we've gotten has been that even out of tree this patchset has already
> been useful, and there's a significant amount of other work gated on the
> code tagging functionality included in this patchset [2].
>=20
> Example output:
>   root@moria-kvm:~# sort -h /proc/allocinfo|tail
>    3.11MiB     2850 fs/ext4/super.c:1408 module:ext4 func:ext4_alloc_inod=
e
>    3.52MiB      225 kernel/fork.c:356 module:fork func:alloc_thread_stack=
_node
>    3.75MiB      960 mm/page_ext.c:270 module:page_ext func:alloc_page_ext
>    4.00MiB        2 mm/khugepaged.c:893 module:khugepaged func:hpage_coll=
apse_alloc_folio
>    10.5MiB      168 block/blk-mq.c:3421 module:blk_mq func:blk_mq_alloc_r=
qs
>    14.0MiB     3594 include/linux/gfp.h:295 module:filemap func:folio_all=
oc_noprof
>    26.8MiB     6856 include/linux/gfp.h:295 module:memory func:folio_allo=
c_noprof
>    64.5MiB    98315 fs/xfs/xfs_rmap_item.c:147 module:xfs func:xfs_rui_in=
it
>    98.7MiB    25264 include/linux/gfp.h:295 module:readahead func:folio_a=
lloc_noprof
>     125MiB     7357 mm/slub.c:2201 module:slub func:alloc_slab_page
>=20
> Since v2:
>  - tglx noticed a circular header dependency between sched.h and percpu.h=
;
>    a bunch of header cleanups were merged into 6.8 to ameliorate this [3]=
.
>=20
>  - a number of improvements, moving alloc_hooks() annotations to the
>    correct place for better tracking (mempool), and bugfixes.
>=20
>  - looked at alternate hooking methods.
>    There were suggestions on alternate methods (compiler attribute,
>    trampolines), but they wouldn't have made the patchset any cleaner
>    (we still need to have different function versions for accounting vs. =
no
>    accounting to control at which point in a call chain the accounting
>    happens), and they would have added a dependency on toolchain
>    support.
>=20
> Usage:
> kconfig options:
>  - CONFIG_MEM_ALLOC_PROFILING
>  - CONFIG_MEM_ALLOC_PROFILING_ENABLED_BY_DEFAULT
>  - CONFIG_MEM_ALLOC_PROFILING_DEBUG
>    adds warnings for allocations that weren't accounted because of a
>    missing annotation
>=20
> sysctl:
>   /proc/sys/vm/mem_profiling
>=20
> Runtime info:
>   /proc/allocinfo
>=20
> Notes:
>=20
> [1]: Overhead
> To measure the overhead we are comparing the following configurations:
> (1) Baseline with CONFIG_MEMCG_KMEM=3Dn
> (2) Disabled by default (CONFIG_MEM_ALLOC_PROFILING=3Dy &&
>     CONFIG_MEM_ALLOC_PROFILING_BY_DEFAULT=3Dn)
> (3) Enabled by default (CONFIG_MEM_ALLOC_PROFILING=3Dy &&
>     CONFIG_MEM_ALLOC_PROFILING_BY_DEFAULT=3Dy)
> (4) Enabled at runtime (CONFIG_MEM_ALLOC_PROFILING=3Dy &&
>     CONFIG_MEM_ALLOC_PROFILING_BY_DEFAULT=3Dn && /proc/sys/vm/mem_profili=
ng=3D1)
> (5) Baseline with CONFIG_MEMCG_KMEM=3Dy && allocating with __GFP_ACCOUNT
>=20

Thanks for the work on this patchset and it is quite useful.
A clarification question on the data:

I assume Config (2), (3) and (4) has CONFIG_MEMCG_KMEM=3Dn, right?
If so do you have similar data for config (2), (3) and (4) but with
CONFIG_MEMCG_KMEM=3Dy for comparison with (5)?

Tim

> Performance overhead:
> To evaluate performance we implemented an in-kernel test executing
> multiple get_free_page/free_page and kmalloc/kfree calls with allocation
> sizes growing from 8 to 240 bytes with CPU frequency set to max and CPU
> affinity set to a specific CPU to minimize the noise. Below are results
> from running the test on Ubuntu 22.04.2 LTS with 6.8.0-rc1 kernel on
> 56 core Intel Xeon:
>=20
>                         kmalloc                 pgalloc
> (1 baseline)            6.764s                  16.902s
> (2 default disabled)    6.793s (+0.43%)         17.007s (+0.62%)
> (3 default enabled)     7.197s (+6.40%)         23.666s (+40.02%)
> (4 runtime enabled)     7.405s (+9.48%)         23.901s (+41.41%)
> (5 memcg)               13.388s (+97.94%)       48.460s (+186.71%)
>=20