A 64 bit architecture can allow reducing the size of the kernel image by
selecting HAVE_DYNAMIC_DEBUG_RELATIVE_POINTERS, but it must provide
a proper DEFINE_DYNAMIC_DEBUG_METADATA macro for emitting the struct
_ddebug in assembly. However, since that does not involve any
instructions, this generic implementation should be usable by most if
not all.
It relies on
(1) standard assembly directives that should work on
all architectures
(2) the "i" constraint for an constant, and
(3) %cN emitting the constant operand N without punctuation
and of course the layout of _ddebug being what one expects.
Now, clang before 9.0 doesn't satisfy (3) for non-x86 targets.
Moreover, it turns out that gcc 4.8 in some corner cases emits (dead) code
which ends up causing the link to fail (emitting a reference to the
dynamic debug descriptor without emitting the assembly that defines
it).
Hence, make the config option available only for relatively new
compilers - I have not been able to reproduce the gcc problem with any
newer versions, and this implementation has been in -next for a whole
cycle without other problems reported.
I've succesfully built x86-64, arm64 and ppc64 defconfig +
CONFIG_DYNAMIC_DEBUG + patches to hook up this header, and boot-tested
the x86-64 one.
Signed-off-by: Rasmus Villemoes <[email protected]>
---
include/asm-generic/dynamic_debug.h | 116 ++++++++++++++++++++++++++++
include/linux/jump_label.h | 2 +
lib/Kconfig.debug | 5 +-
3 files changed, 122 insertions(+), 1 deletion(-)
create mode 100644 include/asm-generic/dynamic_debug.h
diff --git a/include/asm-generic/dynamic_debug.h b/include/asm-generic/dynamic_debug.h
new file mode 100644
index 000000000000..f1dd3019cd98
--- /dev/null
+++ b/include/asm-generic/dynamic_debug.h
@@ -0,0 +1,116 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_GENERIC_DYNAMIC_DEBUG_H
+#define _ASM_GENERIC_DYNAMIC_DEBUG_H
+
+#ifndef _DYNAMIC_DEBUG_H
+#error "don't include asm/dynamic_debug.h directly"
+#endif
+
+#include <linux/build_bug.h>
+#ifdef CONFIG_JUMP_LABEL
+#include <linux/jump_label.h>
+#endif
+
+/*
+ * We need to know the exact layout of struct _ddebug in order to
+ * initialize it in assembly. Check that all members are at expected
+ * offsets - if any of these fail, the arch cannot use this generic
+ * dynamic_debug.h. DYNAMIC_DEBUG_RELATIVE_POINTERS is pointless for
+ * !64BIT, so we expect the static_key to be at an 8-byte boundary
+ * since it contains stuff which is long-aligned.
+ */
+
+static_assert(BITS_PER_LONG == 64);
+static_assert(offsetof(struct _ddebug, modname_disp) == 0);
+static_assert(offsetof(struct _ddebug, function_disp) == 4);
+static_assert(offsetof(struct _ddebug, filename_disp) == 8);
+static_assert(offsetof(struct _ddebug, format_disp) == 12);
+static_assert(offsetof(struct _ddebug, flags_lineno) == 16);
+
+#ifdef CONFIG_JUMP_LABEL
+static_assert(offsetof(struct _ddebug, key) == 24);
+static_assert(offsetof(struct static_key, enabled) == 0);
+static_assert(offsetof(struct static_key, type) == 8);
+
+/* <4 bytes padding>, .enabled, <4 bytes padding>, .type */
+# ifdef DEBUG
+# define _DPRINTK_ASM_KEY_INIT \
+ "\t.int 0\n" "\t.int 1\n" "\t.int 0\n" "\t.quad "__stringify(__JUMP_TYPE_TRUE)"\n"
+# else
+# define _DPRINTK_ASM_KEY_INIT \
+ "\t.int 0\n" "\t.int 0\n" "\t.int 0\n" "\t.quad "__stringify(__JUMP_TYPE_FALSE)"\n"
+# endif
+#else /* !CONFIG_JUMP_LABEL */
+#define _DPRINTK_ASM_KEY_INIT ""
+#endif
+
+/*
+ * There's a bit of magic involved here.
+ *
+ * First, unlike the bug table entries, we need to define an object in
+ * assembly which we can reference from C code (for use by the
+ * DYNAMIC_DEBUG_BRANCH macro), but we don't want 'name' to have
+ * external linkage (that would require use of globally unique
+ * identifiers, which we can't guarantee). Fortunately, the "extern"
+ * keyword just tells the compiler that _somebody_ provides that
+ * symbol - usually that somebody is the linker, but in this case it's
+ * the assembler, and since we do not do .globl name, the symbol gets
+ * internal linkage.
+ *
+ * So far so good. The next problem is that there's no scope in
+ * assembly, so the identifier 'name' has to be unique within each
+ * translation unit - otherwise all uses of that identifier end up
+ * referring to the same struct _ddebug instance. pr_debug and friends
+ * do this by use of indirection and __UNIQUE_ID(), and new users of
+ * DEFINE_DYNAMIC_DEBUG_METADATA() should do something similar. We
+ * need to catch cases where this is not done at build time.
+ *
+ * With assembly-level .ifndef we can ensure that we only define a
+ * given identifier once, preventing "symbol 'foo' already defined"
+ * errors. But we still need to detect and fail on multiple uses of
+ * the same identifer. The simplest, and wrong, solution to that is to
+ * add an .else .error branch to the .ifndef. The problem is that just
+ * because the DEFINE_DYNAMIC_DEBUG_METADATA() macro is only expanded
+ * once with a given identifier, the compiler may emit the assembly
+ * code multiple times, e.g. if the macro appears in an inline
+ * function. Now, in a normal case like
+ *
+ * static inline get_next_id(void) { static int v; return ++v; }
+ *
+ * all inlined copies of get_next_id are _supposed_ to refer to the
+ * same object 'v'. So we do need to allow this chunk of assembly to
+ * appear multiple times with the same 'name', as long as they all
+ * came from the same DEFINE_DYNAMIC_DEBUG_METADATA() instance. To do
+ * that, we pass __COUNTER__ to the asm(), and set an assembler symbol
+ * name.ddebug.once to that value when we first define 'name'. When we
+ * meet a second attempt at defining 'name', we compare
+ * name.ddebug.once to %6 and error out if they are different.
+ */
+
+#define DEFINE_DYNAMIC_DEBUG_METADATA(name, fmt) \
+ extern struct _ddebug name; \
+ asm volatile(".ifndef " __stringify(name) "\n" \
+ ".pushsection __verbose,\"aw\"\n" \
+ ".type "__stringify(name)", STT_OBJECT\n" \
+ ".size "__stringify(name)", %c5\n" \
+ "1:\n" \
+ __stringify(name) ":\n" \
+ "\t.int %c0 - 1b /* _ddebug::modname_disp */\n" \
+ "\t.int %c1 - 1b /* _ddebug::function_disp */\n" \
+ "\t.int %c2 - 1b /* _ddebug::filename_disp */\n" \
+ "\t.int %c3 - 1b /* _ddebug::format_disp */\n" \
+ "\t.int %c4 /* _ddebug::flags_lineno */\n" \
+ _DPRINTK_ASM_KEY_INIT \
+ "\t.org 1b+%c5\n" \
+ ".popsection\n" \
+ ".set "__stringify(name)".ddebug.once, %c6\n" \
+ ".elseif "__stringify(name)".ddebug.once - %c6\n" \
+ ".line "__stringify(__LINE__) " - 1\n" \
+ ".error \"'"__stringify(name)"' used as _ddebug identifier more than once\"\n" \
+ ".endif\n" \
+ : : "i" (KBUILD_MODNAME), "i" (__func__), \
+ "i" (__FILE__), "i" (fmt), \
+ "i" (_DPRINTK_FLAGS_LINENO_INIT), \
+ "i" (sizeof(struct _ddebug)), "i" (__COUNTER__))
+
+#endif /* _ASM_GENERIC_DYNAMIC_DEBUG_H */
diff --git a/include/linux/jump_label.h b/include/linux/jump_label.h
index 3e113a1fa0f1..2b027d2ef3d0 100644
--- a/include/linux/jump_label.h
+++ b/include/linux/jump_label.h
@@ -190,6 +190,8 @@ struct module;
#ifdef CONFIG_JUMP_LABEL
+#define __JUMP_TYPE_FALSE 0
+#define __JUMP_TYPE_TRUE 1
#define JUMP_TYPE_FALSE 0UL
#define JUMP_TYPE_TRUE 1UL
#define JUMP_TYPE_LINKED 2UL
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index f3d2e234c15f..7f2bc0175b88 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -171,11 +171,14 @@ config DYNAMIC_DEBUG_RELATIVE_POINTERS
bool "Reduce size of dynamic debug metadata"
depends on DYNAMIC_DEBUG
depends on HAVE_DYNAMIC_DEBUG_RELATIVE_POINTERS
+ depends on (GCC_VERSION >= 50000 || CLANG_VERSION >= 90000)
help
If you say Y here, the static memory footprint of the kernel
image will be reduced somewhat (about 40K for a typical
distro kernel). There is no performance difference either
- way.
+ way. This relies on some compile-time magic, so if your
+ toolchain fails to build the kernel with this option, just
+ say N.
endmenu # "printk and dmesg options"
--
2.20.1
On Mon, Jun 17, 2019 at 3:20 PM Rasmus Villemoes
<[email protected]> wrote:
>
> A 64 bit architecture can allow reducing the size of the kernel image by
> selecting HAVE_DYNAMIC_DEBUG_RELATIVE_POINTERS, but it must provide
> a proper DEFINE_DYNAMIC_DEBUG_METADATA macro for emitting the struct
> _ddebug in assembly. However, since that does not involve any
> instructions, this generic implementation should be usable by most if
> not all.
>
> It relies on
>
> (1) standard assembly directives that should work on
> all architectures
> (2) the "i" constraint for an constant, and
> (3) %cN emitting the constant operand N without punctuation
>
> and of course the layout of _ddebug being what one expects.
>
> Now, clang before 9.0 doesn't satisfy (3) for non-x86 targets.
Thanks so much for resending with this case fixed, and sorry I did not
implement (3) sooner! I appreciate your patience.
Acked-by: Nick Desaulniers <[email protected]>
I'm happy to help test this series, do you have a tree I could pull
these from quickly? Anything I should test at runtime besides a boot
test?
--
Thanks,
~Nick Desaulniers
On 18/06/2019 00.35, Nick Desaulniers wrote:
> On Mon, Jun 17, 2019 at 3:20 PM Rasmus Villemoes
> <[email protected]> wrote:
>>
>> It relies on
>>
>> (1) standard assembly directives that should work on
>> all architectures
>> (2) the "i" constraint for an constant, and
>> (3) %cN emitting the constant operand N without punctuation
>>
>> and of course the layout of _ddebug being what one expects.
>>
>> Now, clang before 9.0 doesn't satisfy (3) for non-x86 targets.
>
> Thanks so much for resending with this case fixed, and sorry I did not
> implement (3) sooner! I appreciate your patience.
> Acked-by: Nick Desaulniers <[email protected]>
>
> I'm happy to help test this series, do you have a tree I could pull
> these from quickly?
I've pushed them to https://github.com/Villemoes/linux/tree/dyndebug_v6
. They rebase pretty cleanly to just about anything you might prefer
testing on. Enabling it for arm64 or ppc64 is a trivial two-liner
similar to the x86 patch (and similar to the previous patches for those
arches). Thanks for volunteering to test this :)
> Anything I should test at runtime besides a boot
> test?
Well, apart from booting, I've mostly just tested that the debugfs
control file is identical before and after enabling relative pointers,
and that enabling/disabling various pr_debug()s by writing to the
control file takes effect. I should only be changing the format for
storing the metadata in the kernel image, so I think that should be enough.
While this is still not merged, some new user of one of the string
members could creep in, but that should be caught at build time.
Rasmus
On Thu, Jun 20, 2019 at 1:46 PM Rasmus Villemoes
<[email protected]> wrote:
>
> On 18/06/2019 00.35, Nick Desaulniers wrote:
> > On Mon, Jun 17, 2019 at 3:20 PM Rasmus Villemoes
> > <[email protected]> wrote:
> >>
> >> It relies on
> >>
> >> (1) standard assembly directives that should work on
> >> all architectures
> >> (2) the "i" constraint for an constant, and
> >> (3) %cN emitting the constant operand N without punctuation
> >>
> >> and of course the layout of _ddebug being what one expects.
> >>
> >> Now, clang before 9.0 doesn't satisfy (3) for non-x86 targets.
> >
> > Thanks so much for resending with this case fixed, and sorry I did not
> > implement (3) sooner! I appreciate your patience.
> > Acked-by: Nick Desaulniers <[email protected]>
> >
> > I'm happy to help test this series, do you have a tree I could pull
> > these from quickly?
>
> I've pushed them to https://github.com/Villemoes/linux/tree/dyndebug_v6
> . They rebase pretty cleanly to just about anything you might prefer
> testing on. Enabling it for arm64 or ppc64 is a trivial two-liner
> similar to the x86 patch (and similar to the previous patches for those
> arches). Thanks for volunteering to test this :)
Compile tested x86_64 allyesconfig
boot tested x86_64 defconfig+CONFIG_DYNAMIC_DEBUG
(just curious why the Kconfig changes for arm64 or ppc64 aren't
included in this set?)
>
> > Anything I should test at runtime besides a boot
> > test?
>
> Well, apart from booting, I've mostly just tested that the debugfs
> control file is identical before and after enabling relative pointers,
mainline x86_64 defconfig+CONFIG_DYNAMIC_DEBUG
$ cat /dfs/dynamic_debug/control | wc -l
2488
mainline x86_64 defconfig+CONFIG_DYNAMIC_DEBUG+this patch series
$ cat /dfs/dynamic_debug/control | wc -l
2486
(seems like maybe 2 are missing? Let me try to collect a diff. Maybe
2 were removed in this series?)
> and that enabling/disabling various pr_debug()s by writing to the
> control file takes effect. I should only be changing the format for
Can you suggest one that's easy to test?
> storing the metadata in the kernel image, so I think that should be enough.
>
> While this is still not merged, some new user of one of the string
> members could creep in, but that should be caught at build time.
--
Thanks,
~Nick Desaulniers
On 24/06/2019 23.53, Nick Desaulniers wrote:
> On Thu, Jun 20, 2019 at 1:46 PM Rasmus Villemoes
> <[email protected]> wrote:
>>
>> I've pushed them to https://github.com/Villemoes/linux/tree/dyndebug_v6
>> . They rebase pretty cleanly to just about anything you might prefer
>> testing on. Enabling it for arm64 or ppc64 is a trivial two-liner
>> similar to the x86 patch (and similar to the previous patches for those
>> arches). Thanks for volunteering to test this :)
>
> Compile tested x86_64 allyesconfig
> boot tested x86_64 defconfig+CONFIG_DYNAMIC_DEBUG
>
> (just curious why the Kconfig changes for arm64 or ppc64 aren't
> included in this set?)
Partly because I can't boot test those and this has proven much more
delicate than I thought, partly because none of the maintainers for
those arches have weighed in. So I'd rather have the bare minimal land,
then send specific individual patches for arm64 and ppc64.
>>> Anything I should test at runtime besides a boot
>>> test?
>>
>> Well, apart from booting, I've mostly just tested that the debugfs
>> control file is identical before and after enabling relative pointers,
>
> mainline x86_64 defconfig+CONFIG_DYNAMIC_DEBUG
> $ cat /dfs/dynamic_debug/control | wc -l
> 2488
>
>
> mainline x86_64 defconfig+CONFIG_DYNAMIC_DEBUG+this patch series
> $ cat /dfs/dynamic_debug/control | wc -l
> 2486
>
> (seems like maybe 2 are missing? Let me try to collect a diff. Maybe
> 2 were removed in this series?)
Hm, no pr_debugs should have been added or removed. Perhaps you have a
slightly different set of modules loaded? Otherwise there's something
odd going on, and a diff would be really nice. It's possible that the
order of the lines are different, so you may have to sort them to get a
meaningful diff. (A diff is nice extra sanity check even if the line
count matches, of course).
>> and that enabling/disabling various pr_debug()s by writing to the
>> control file takes effect. I should only be changing the format for
>
> Can you suggest one that's easy to test?
The ones in lib/kobject.c are triggered fairly often I think.
Thanks,
Rasmus
On Mon, Jun 24, 2019 at 11:35 PM Rasmus Villemoes
<[email protected]> wrote:
>
> On 24/06/2019 23.53, Nick Desaulniers wrote:
> > On Thu, Jun 20, 2019 at 1:46 PM Rasmus Villemoes
> > <[email protected]> wrote:
> >> Well, apart from booting, I've mostly just tested that the debugfs
> >> control file is identical before and after enabling relative pointers,
> >
> > mainline x86_64 defconfig+CONFIG_DYNAMIC_DEBUG
> > $ cat /dfs/dynamic_debug/control | wc -l
> > 2488
> >
> >
> > mainline x86_64 defconfig+CONFIG_DYNAMIC_DEBUG+this patch series
> > $ cat /dfs/dynamic_debug/control | wc -l
> > 2486
> >
> > (seems like maybe 2 are missing? Let me try to collect a diff. Maybe
> > 2 were removed in this series?)
>
> Hm, no pr_debugs should have been added or removed. Perhaps you have a
> slightly different set of modules loaded? Otherwise there's something
> odd going on, and a diff would be really nice. It's possible that the
> order of the lines are different, so you may have to sort them to get a
> meaningful diff. (A diff is nice extra sanity check even if the line
> count matches, of course).
You can fetch my logs from the latest commit to this dummy branch:
https://github.com/ClangBuiltLinux/linux/commit/90096d926aaf94eb84584a4418fde7c8d42dddea
Looking at `meld wo_patches.txt w_patches.txt`, it looks like:
1. line numbers in some translation units are adjusted. maybe this is
intentional?
2. pci_pm_suspend_noirq seems to exist twice(?) before your patches, once after
3. xhci_urb_enqueue seems to exist three times before your patches, twice after
--
Thanks,
~Nick Desaulniers
On Tue, Jun 25, 2019 at 3:18 PM Nick Desaulniers
<[email protected]> wrote:
> 2. pci_pm_suspend_noirq seems to exist twice(?) before your patches, once after
> 3. xhci_urb_enqueue seems to exist three times before your patches, twice after
PEBKAC, I advanced my master branch without rebasing my local branch
with these changes. With the local branch rebased onto master, there
are no differences. 2888 in both cases, line numbers do not differ.
I wasn't able to trip any logs in kobject:
(initramfs) echo "file kobject.c +p" > /dfs/dynamic_debug/control
(initramfs) cat /dfs/dynamic_debug/control | grep kobject
lib/kobject.c:161 [kobject]fill_kobj_path =p "kobject: '%s' (%p): %s:
path = '%s'\012"
lib/kobject.c:668 [kobject]kobject_cleanup =p "kobject: '%s' (%p): %s,
parent %p\012"
lib/kobject.c:672 [kobject]kobject_cleanup =p "kobject: '%s' (%p):
does not have a release() func"
lib/kobject.c:677 [kobject]kobject_cleanup =p "kobject: '%s' (%p):
auto cleanup 'remove' event\01"
lib/kobject.c:684 [kobject]kobject_cleanup =p "kobject: '%s' (%p):
auto cleanup kobject_del\012"
lib/kobject.c:690 [kobject]kobject_cleanup =p "kobject: '%s' (%p):
calling ktype release\012"
lib/kobject.c:696 [kobject]kobject_cleanup =p "kobject: '%s': free name\012"
lib/kobject.c:744 [kobject]dynamic_kobj_release =p "kobject: (%p): %s\012"
lib/kobject.c:253 [kobject]kobject_add_internal =p "kobject: '%s'
(%p): %s: parent: '%s', set: '%"
lib/kobject.c:915 [kobject]kset_release =p "kobject: '%s' (%p): %s\012"
The prints should show up in dmesg right, assuming you do something to
trigger them? Can you provide more details for a test case that's
easy to trip? What's an easy case to reproduce from a limited
buildroot env (basic shell/toybox)?
--
Thanks,
~Nick Desaulniers
On 27/06/2019 01.16, Nick Desaulniers wrote:
> On Tue, Jun 25, 2019 at 3:18 PM Nick Desaulniers
> <[email protected]> wrote:
>
> The prints should show up in dmesg right, assuming you do something to
> trigger them? Can you provide more details for a test case that's
> easy to trip? What's an easy case to reproduce from a limited
> buildroot env (basic shell/toybox)?
>
Hm, I seemed to remember that those kobject events triggered all the
time. Oh well, try this one:
echo 'file ping.c +p' > control
ping localhost
dmesg | grep ping
Rasmus
On Wed, Jun 26, 2019 at 4:52 PM Rasmus Villemoes
<[email protected]> wrote:
>
> On 27/06/2019 01.16, Nick Desaulniers wrote:
> > On Tue, Jun 25, 2019 at 3:18 PM Nick Desaulniers
> > <[email protected]> wrote:
> >
> > The prints should show up in dmesg right, assuming you do something to
> > trigger them? Can you provide more details for a test case that's
> > easy to trip? What's an easy case to reproduce from a limited
> > buildroot env (basic shell/toybox)?
> >
>
> Hm, I seemed to remember that those kobject events triggered all the
> time. Oh well, try this one:
>
> echo 'file ping.c +p' > control
> ping localhost
> dmesg | grep ping
I don't have guest networking setup from QEMU to host, so there's no
network available to ping. :(
but:
(initramfs) echo 'file drivers/tty/*.c +p' > /dfs/dynamic_debug/control
(initramfs) grep tty /dfs/dynamic_debug/control
...
drivers/tty/serial/8250/8250_core.c:113 [8250]serial8250_interrupt =p
"%s(%d): start\012"
drivers/tty/serial/8250/8250_core.c:139 [8250]serial8250_interrupt =p
"%s(%d): end\012"
...
(initramfs) dmesg
...
[ 134.895846] serial8250_interrupt(4): start
[ 134.895967] serial8250_interrupt(4): end
[ 134.895970] serial8250_interrupt(4): start
[ 134.895981] serial8250_interrupt(4): end
[ 134.895998] serial8250_interrupt(4): start
[ 134.896053] serial8250_interrupt(4): end
I then verified that nothing new appears in dmesg related to these
traces after running:
(initramfs) echo 'file drivers/tty/*.c -p' > /dfs/dynamic_debug/control
so if that's good enough, then for the series:
Tested-by: Nick Desaulniers <[email protected]>
Thanks,
~Nick Desaulniers