The kernel can be configured to contain a lot of introspection or
debugging information built-in, such as ORC for unwinding stack traces,
BTF for type information, and of course kallsyms. Debuggers could use
this information to navigate a core dump or live system, but they need
to be able to find it.
This patch series adds the necessary symbols into vmcoreinfo, which
would allow a debugger to find and interpret the kallsyms table. Using
the kallsyms data, the debugger can then lookup any symbol, allowing it
to find ORC, BTF, or any other useful data.
This would allow a live kernel, or core dump, to be debugged without
any DWARF debuginfo. This is useful for many cases: the debuginfo may
not have been generated, or you may not want to deploy the large files
everywhere you need them.
I've demonstrated a proof of concept for this at LSF/MM+BPF during a
lighting talk. Using a work-in-progress branch of the drgn debugger, and
an extended set of BTF generated by a patched version of dwarves, I've
been able to open a core dump without any DWARF info and do basic tasks
such as enumerating slab caches, block devices, tasks, and doing
backtraces. I hope this series can be a first step toward a new
possibility of "DWARFless debugging".
Related discussion around the BTF side of this:
https://lore.kernel.org/bpf/[email protected]/T/#u
Some work-in-progress branches using this feature:
https://github.com/brenns10/dwarves/tree/remove_percpu_restriction_1
https://github.com/brenns10/drgn/tree/kallsyms_plus_btf
Stephen Brennan (2):
kallsyms: Move declarations to internal header
vmcoreinfo: Include kallsyms symbols
kernel/crash_core.c | 14 ++++++++++++++
kernel/kallsyms.c | 23 +----------------------
kernel/kallsyms_internal.h | 30 ++++++++++++++++++++++++++++++
3 files changed, 45 insertions(+), 22 deletions(-)
create mode 100644 kernel/kallsyms_internal.h
--
2.30.2
On 05/16/22 at 05:05pm, Stephen Brennan wrote:
> The kernel can be configured to contain a lot of introspection or
> debugging information built-in, such as ORC for unwinding stack traces,
> BTF for type information, and of course kallsyms. Debuggers could use
> this information to navigate a core dump or live system, but they need
> to be able to find it.
>
> This patch series adds the necessary symbols into vmcoreinfo, which
> would allow a debugger to find and interpret the kallsyms table. Using
> the kallsyms data, the debugger can then lookup any symbol, allowing it
> to find ORC, BTF, or any other useful data.
>
> This would allow a live kernel, or core dump, to be debugged without
> any DWARF debuginfo. This is useful for many cases: the debuginfo may
> not have been generated, or you may not want to deploy the large files
> everywhere you need them.
>
> I've demonstrated a proof of concept for this at LSF/MM+BPF during a
> lighting talk. Using a work-in-progress branch of the drgn debugger, and
> an extended set of BTF generated by a patched version of dwarves, I've
> been able to open a core dump without any DWARF info and do basic tasks
> such as enumerating slab caches, block devices, tasks, and doing
> backtraces. I hope this series can be a first step toward a new
> possibility of "DWARFless debugging".
Thanks. Seems no reason to reject, even though I haven't tried drgn.
And hope it has no security issue, e.g info leakage, at least I don't
see it has. So,
Acked-by: Baoquan He <[email protected]>
Baoquan He <[email protected]> writes:
> On 05/16/22 at 05:05pm, Stephen Brennan wrote:
>> The kernel can be configured to contain a lot of introspection or
>> debugging information built-in, such as ORC for unwinding stack traces,
>> BTF for type information, and of course kallsyms. Debuggers could use
>> this information to navigate a core dump or live system, but they need
>> to be able to find it.
>>
>> This patch series adds the necessary symbols into vmcoreinfo, which
>> would allow a debugger to find and interpret the kallsyms table. Using
>> the kallsyms data, the debugger can then lookup any symbol, allowing it
>> to find ORC, BTF, or any other useful data.
>>
>> This would allow a live kernel, or core dump, to be debugged without
>> any DWARF debuginfo. This is useful for many cases: the debuginfo may
>> not have been generated, or you may not want to deploy the large files
>> everywhere you need them.
>>
>> I've demonstrated a proof of concept for this at LSF/MM+BPF during a
>> lighting talk. Using a work-in-progress branch of the drgn debugger, and
>> an extended set of BTF generated by a patched version of dwarves, I've
>> been able to open a core dump without any DWARF info and do basic tasks
>> such as enumerating slab caches, block devices, tasks, and doing
>> backtraces. I hope this series can be a first step toward a new
>> possibility of "DWARFless debugging".
>
> Thanks. Seems no reason to reject, even though I haven't tried drgn.
> And hope it has no security issue, e.g info leakage, at least I don't
> see it has. So,
>
> Acked-by: Baoquan He <[email protected]>
Thanks Baoquan! I don't believe we have any worries regarding security,
since the kallsyms data itself is already available to root via
/proc/kallsyms, and core dumps should already be treated as highly
sensitive data by anybody handling them.
Do you know which tree this patch will go through?
Thanks,
Stephen
On 05/23/22 at 11:00am, Stephen Brennan wrote:
> Baoquan He <[email protected]> writes:
> > On 05/16/22 at 05:05pm, Stephen Brennan wrote:
> >> The kernel can be configured to contain a lot of introspection or
> >> debugging information built-in, such as ORC for unwinding stack traces,
> >> BTF for type information, and of course kallsyms. Debuggers could use
> >> this information to navigate a core dump or live system, but they need
> >> to be able to find it.
> >>
> >> This patch series adds the necessary symbols into vmcoreinfo, which
> >> would allow a debugger to find and interpret the kallsyms table. Using
> >> the kallsyms data, the debugger can then lookup any symbol, allowing it
> >> to find ORC, BTF, or any other useful data.
> >>
> >> This would allow a live kernel, or core dump, to be debugged without
> >> any DWARF debuginfo. This is useful for many cases: the debuginfo may
> >> not have been generated, or you may not want to deploy the large files
> >> everywhere you need them.
> >>
> >> I've demonstrated a proof of concept for this at LSF/MM+BPF during a
> >> lighting talk. Using a work-in-progress branch of the drgn debugger, and
> >> an extended set of BTF generated by a patched version of dwarves, I've
> >> been able to open a core dump without any DWARF info and do basic tasks
> >> such as enumerating slab caches, block devices, tasks, and doing
> >> backtraces. I hope this series can be a first step toward a new
> >> possibility of "DWARFless debugging".
> >
> > Thanks. Seems no reason to reject, even though I haven't tried drgn.
> > And hope it has no security issue, e.g info leakage, at least I don't
> > see it has. So,
> >
> > Acked-by: Baoquan He <[email protected]>
>
> Thanks Baoquan! I don't believe we have any worries regarding security,
> since the kallsyms data itself is already available to root via
> /proc/kallsyms, and core dumps should already be treated as highly
> sensitive data by anybody handling them.
>
> Do you know which tree this patch will go through?
I would like to ask Andrew to help check and pick this if no concern.
Thanks
Baoquan
Hi Andrew,
Baoquan He <[email protected]> writes:
> On 05/23/22 at 11:00am, Stephen Brennan wrote:
>> Do you know which tree this patch will go through?
>
> I would like to ask Andrew to help check and pick this if no concern.
Just wanted to check if you would be willing to take a look at this
series and possibly queue it up. I can resend the series directly to you
if you need.
Thanks,
Stephen
On Mon, 16 May 2022 17:05:06 -0700 Stephen Brennan <[email protected]> wrote:
> The kernel can be configured to contain a lot of introspection or
> debugging information built-in, such as ORC for unwinding stack traces,
> BTF for type information, and of course kallsyms. Debuggers could use
> this information to navigate a core dump or live system, but they need
> to be able to find it.
>
> This patch series adds the necessary symbols into vmcoreinfo, which
> would allow a debugger to find and interpret the kallsyms table. Using
> the kallsyms data, the debugger can then lookup any symbol, allowing it
> to find ORC, BTF, or any other useful data.
>
> This would allow a live kernel, or core dump, to be debugged without
> any DWARF debuginfo. This is useful for many cases: the debuginfo may
> not have been generated, or you may not want to deploy the large files
> everywhere you need them.
Am trying to understand the value of all of this. Can you explain
further why carrying the dwarf info is problematic? How problematic
are these large files?
> I've demonstrated a proof of concept for this at LSF/MM+BPF during a
> lighting talk. Using a work-in-progress branch of the drgn debugger, and
> an extended set of BTF generated by a patched version of dwarves, I've
> been able to open a core dump without any DWARF info and do basic tasks
> such as enumerating slab caches, block devices, tasks, and doing
> backtraces. I hope this series can be a first step toward a new
> possibility of "DWARFless debugging".
>
> Related discussion around the BTF side of this:
> https://lore.kernel.org/bpf/[email protected]/T/#u
>
> Some work-in-progress branches using this feature:
> https://github.com/brenns10/dwarves/tree/remove_percpu_restriction_1
> https://github.com/brenns10/drgn/tree/kallsyms_plus_btf
What's the story on using gdb with this?
Andrew Morton <[email protected]> writes:
> On Mon, 16 May 2022 17:05:06 -0700 Stephen Brennan <[email protected]> wrote:
>
>> The kernel can be configured to contain a lot of introspection or
>> debugging information built-in, such as ORC for unwinding stack traces,
>> BTF for type information, and of course kallsyms. Debuggers could use
>> this information to navigate a core dump or live system, but they need
>> to be able to find it.
>>
>> This patch series adds the necessary symbols into vmcoreinfo, which
>> would allow a debugger to find and interpret the kallsyms table. Using
>> the kallsyms data, the debugger can then lookup any symbol, allowing it
>> to find ORC, BTF, or any other useful data.
>>
>> This would allow a live kernel, or core dump, to be debugged without
>> any DWARF debuginfo. This is useful for many cases: the debuginfo may
>> not have been generated, or you may not want to deploy the large files
>> everywhere you need them.
>
> Am trying to understand the value of all of this. Can you explain
> further why carrying the dwarf info is problematic? How problematic
> are these large files?
The primary value I see in this is that it unlocks data that's already
present in core dumps. Most distribution kernels already have
CONFIG_KALLSYMS_ALL, but you can't reliably find the symbol table in a
core dump without a change like this. Most distribution kernels already
have a fair amount of type information via BTF already built-in (though
as I mention below, there's a bit more that would be useful). So this
change can unlock that data, and make core dumps self-describing: you
can open one up at any time without extra debuginfo, and start reading
symbols and data values out of memory.
As far as the DWARF, I do find it unwieldy, we're talking about file
sizes ranging from 500 MiB to a 1 GiB or maybe a bit more. This isn't
huge for on-disk storage. But for things like OS images, it's pretty
hefty just for the off-chance that you need debugging.
A bigger concern than the size of the files is that, we frequently deal
with customers who, due to some sort of requirements, have a policy
against installing debuginfo packages on production machine. When they
inevitably ask folks like me to help debug the kernel on their "no
debuginfo" machines, I'd like to be able to just get my job done using
the info already there, rather than battling them over debuginfo.
There's also some corner cases. I've definitely taken a core dump on my
personal machine, only to find that the debuginfo had been deleted due
to a kernel package upgrade, so the core dump was useless. And some
distributions build kernels with no debuginfo packages. This can suit
all of those possibilities without undue overhead.
>> I've demonstrated a proof of concept for this at LSF/MM+BPF during a
>> lighting talk. Using a work-in-progress branch of the drgn debugger, and
>> an extended set of BTF generated by a patched version of dwarves, I've
>> been able to open a core dump without any DWARF info and do basic tasks
>> such as enumerating slab caches, block devices, tasks, and doing
>> backtraces. I hope this series can be a first step toward a new
>> possibility of "DWARFless debugging".
>>
>> Related discussion around the BTF side of this:
>> https://lore.kernel.org/bpf/[email protected]/T/#u
>>
>> Some work-in-progress branches using this feature:
>> https://github.com/brenns10/dwarves/tree/remove_percpu_restriction_1
>> https://github.com/brenns10/drgn/tree/kallsyms_plus_btf
>
> What's the story on using gdb with this?
There is no story with GDB as of yet. I was already familiar with the
code of drgn when I started down this path, so that's what I used. Drgn
happens to have a very extensible type system which made it quite simple
to do. I'd love to see support for doing this with GDB, and might look
into the feasibility of it, but it's not on my roadmap right now.
That said, the drgn work is moving ahead nicely, much of this code has
moved from a "work-in-progress branch" to pull requests which are in
review. I have every intention of seeing those changes through.
Thanks for taking a look at this. I'm happy to answer more questions or
clarify anytihng you want.
Stephen
On Mon, 13 Jun 2022 14:59:44 -0700 Stephen Brennan <[email protected]> wrote:
> >> Related discussion around the BTF side of this:
> >> https://lore.kernel.org/bpf/[email protected]/T/#u
> >>
> >> Some work-in-progress branches using this feature:
> >> https://github.com/brenns10/dwarves/tree/remove_percpu_restriction_1
> >> https://github.com/brenns10/drgn/tree/kallsyms_plus_btf
> >
> > What's the story on using gdb with this?
>
> There is no story with GDB as of yet. I was already familiar with the
> code of drgn when I started down this path, so that's what I used. Drgn
> happens to have a very extensible type system which made it quite simple
> to do. I'd love to see support for doing this with GDB, and might look
> into the feasibility of it, but it's not on my roadmap right now.
Naive question - could some standalone tool take this kallsyms-based
info, combine it with a core image and create a minimally-dwarfified
file which any debugger can munch on?
Andrew Morton <[email protected]> writes:
> On Mon, 13 Jun 2022 14:59:44 -0700 Stephen Brennan <[email protected]> wrote:
>> >> Related discussion around the BTF side of this:
>> >> https://lore.kernel.org/bpf/[email protected]/T/#u
>> >>
>> >> Some work-in-progress branches using this feature:
>> >> https://github.com/brenns10/dwarves/tree/remove_percpu_restriction_1
>> >> https://github.com/brenns10/drgn/tree/kallsyms_plus_btf
>> >
>> > What's the story on using gdb with this?
>>
>> There is no story with GDB as of yet. I was already familiar with the
>> code of drgn when I started down this path, so that's what I used. Drgn
>> happens to have a very extensible type system which made it quite simple
>> to do. I'd love to see support for doing this with GDB, and might look
>> into the feasibility of it, but it's not on my roadmap right now.
>
> Naive question - could some standalone tool take this kallsyms-based
> info, combine it with a core image and create a minimally-dwarfified
> file which any debugger can munch on?
I'm not too familiar with the guts of DWARF, so honestly my guess may
not be any better than yours.
One thing that strikes me is that DWARF is typically included in an ELF
section, whereas x86_64 kernels are typically a bzImage, so I'm not
entirely sure how you'd get back an ELF file suitable to stick the DWARF
into. And from there, I really have no guess about the DWARF.
I will say that Compact Type Format (CTF) [1] is rather similar to BTF
in scope, and it already has support in the GNU Binutils. I'd imagine
that there's much more of a fighting chance of converting BTF to CTF and
getting GDB to use that instead.
[1] https://lwn.net/Articles/795384/
Stephen