This patch series allows the arch-specific kernel fault handlers to dump
in addition to the typical info (IP address, fault type, backtrace and so on)
the command line of the faulting process.
The motivation for this patch is that it's sometimes quite hard to find out and
annoying to not know which program *exactly* faulted when looking at the syslog.
Some examples from the syslog are:
On parisc:
do_page_fault() command='cc1' type=15 address=0x00000000 in libc-2.33.so[f6abb000+184000]
CPU: 1 PID: 13472 Comm: cc1 Tainted: G E 5.10.133+ #45
Hardware name: 9000/785/C8000
-> We see the "cc1" compiler crashed, but it would be useful to know which file was compiled.
With this patch series, the kernel now prints in addition:
cc1[13472] cmdline: /usr/lib/gcc/hppa-linux-gnu/12/cc1 -quiet @/tmp/ccRkFSfY -imultilib . -imultiarch hppa-linux-gnu -D USE_MINIINTERPRETER -D NO_REGS -D _HPUX_SOURCE -D NOSMP -D THREADED_RTS -include /build/ghc/ghc-9.0.2/includes/dist-install/build/ghcversion.h -iquote compiler/GHC/Iface -quiet -dumpdir /tmp/ghc13413_0/ -dumpbase ghc_5.hc -dumpbase-ext .hc -O -Wimplicit -fno-PIC -fwrapv -fno-builtin -fno-strict-aliasing -o /tmp/ghc13413_0/ghc_5.s
-> now we know that cc1 crashed while compiling some haskell code.
Another parisc example:
do_page_fault() command='ld.so.1' type=15 address=0x565921d8 in libc.so[f7339000+1bb000]
CPU: 1 PID: 1151 Comm: cc1 Tainted: G E 5.10.133+ #45
Hardware name: 9000/785/C8000
-> apparently here a program from the glibc testsuite segfaulted.
With this patch we now additionally get:
ld.so.1[1151] cmdline: /home/gnu/glibc/objdir/elf/ld.so.1 --library-path /home/gnu/glibc/objdir:/home/gnu/glibc/objdir/math:/home/gnu/
/home/gnu/glibc/objdir/malloc/tst-safe-linking-malloc-hugetlb1
-> it was the tst-safe-linking-malloc-hugetlb1 testcase which faulted.
An example of a typical x86 fault shows up as:
crash[2326]: segfault at 0 ip 0000561a7969c12e sp 00007ffe97a05630 error 6 in crash[561a7969c000+1000]
Code: 68 ff ff ff c6 05 19 2f 00 00 01 5d c3 0f 1f 80 00 00 00 00 c3 0f 1f 80 00 00 00 00 e9 7b ff ff ff 55 48 89 e5 b8 00 00 00 00 <c7> 00 01 00 00 00 b8 00 00 00 00 5d c3 0f 1f 44 00 00 41 57 4c 8d
-> with this patch we now will see the whole command line:
crash[2326] cmdline: ./crash test_write_to_page_0
The patches are relatively small, and reuses functions which are used
to create the output for the /proc/<pid>/cmdline files.
This is the version 1 of the patch series.
I'm interested if people find this useful too, and if so, I'm
happy for any feedback on those patches.
Thanks!
Helge
Helge Deller (3):
proc: Add get_task_cmdline_kernel() function
lib/dump_stack: Add dump_stack_print_cmdline() and wire up in
dump_stack_print_info()
x86/fault: Dump command line of faulting process to syslog
arch/x86/mm/fault.c | 2 ++
fs/proc/base.c | 68 +++++++++++++++++++++++++++--------------
include/linux/printk.h | 5 +++
include/linux/proc_fs.h | 5 +++
lib/dump_stack.c | 17 +++++++++++
5 files changed, 74 insertions(+), 23 deletions(-)
--
2.37.1
On Mon, Aug 01, 2022 at 05:20:13PM +0200, Helge Deller wrote:
> This patch series allows the arch-specific kernel fault handlers to dump
> in addition to the typical info (IP address, fault type, backtrace and so on)
> the command line of the faulting process.
>
> The motivation for this patch is that it's sometimes quite hard to find out and
> annoying to not know which program *exactly* faulted when looking at the syslog.
>
> Some examples from the syslog are:
>
> On parisc:
> do_page_fault() command='cc1' type=15 address=0x00000000 in libc-2.33.so[f6abb000+184000]
> CPU: 1 PID: 13472 Comm: cc1 Tainted: G E 5.10.133+ #45
> Hardware name: 9000/785/C8000
>
> -> We see the "cc1" compiler crashed, but it would be useful to know which file was compiled.
>
> With this patch series, the kernel now prints in addition:
> cc1[13472] cmdline: /usr/lib/gcc/hppa-linux-gnu/12/cc1 -quiet @/tmp/ccRkFSfY -imultilib . -imultiarch hppa-linux-gnu -D USE_MINIINTERPRETER -D NO_REGS -D _HPUX_SOURCE -D NOSMP -D THREADED_RTS -include /build/ghc/ghc-9.0.2/includes/dist-install/build/ghcversion.h -iquote compiler/GHC/Iface -quiet -dumpdir /tmp/ghc13413_0/ -dumpbase ghc_5.hc -dumpbase-ext .hc -O -Wimplicit -fno-PIC -fwrapv -fno-builtin -fno-strict-aliasing -o /tmp/ghc13413_0/ghc_5.s
>
> -> now we know that cc1 crashed while compiling some haskell code.
This does seem really useful for debugging.
However, it's also an information disclosure in various ways. The
arguments of a program are often more sensitive than the name, and logs
have a tendency to end up in various places, such as bug reports.
An example of how this can be an issue:
- You receive an email or other message with a sensitive link to follow
- You open the link, which launches `firefox https://...`
- You continue browsing from that window
- Firefox crashes (and recovers and restarts, so you don't think
anything of it)
- Later, you report a bug on a different piece of software, and the bug
reporting process includes a copy of the kernel log
I am *not* saying that we shouldn't do this; it seems quite helpful.
However, I think we need to arrange to treat this as sensitive
information, similar to kptr_restrict. (It would also be helpful if
there was a way to tell `dmesg` "please give me the redacted version of
the log", and bug reporting software used that by default.)
- Josh Triplett
On 8/1/22 18:57, Josh Triplett wrote:
> On Mon, Aug 01, 2022 at 05:20:13PM +0200, Helge Deller wrote:
>> This patch series allows the arch-specific kernel fault handlers to dump
>> in addition to the typical info (IP address, fault type, backtrace and so on)
>> the command line of the faulting process.
>>
>> The motivation for this patch is that it's sometimes quite hard to find out and
>> annoying to not know which program *exactly* faulted when looking at the syslog.
>>
>> Some examples from the syslog are:
>>
>> On parisc:
>> do_page_fault() command='cc1' type=15 address=0x00000000 in libc-2.33.so[f6abb000+184000]
>> CPU: 1 PID: 13472 Comm: cc1 Tainted: G E 5.10.133+ #45
>> Hardware name: 9000/785/C8000
>>
>> -> We see the "cc1" compiler crashed, but it would be useful to know which file was compiled.
>>
>> With this patch series, the kernel now prints in addition:
>> cc1[13472] cmdline: /usr/lib/gcc/hppa-linux-gnu/12/cc1 -quiet @/tmp/ccRkFSfY -imultilib . -imultiarch hppa-linux-gnu -D USE_MINIINTERPRETER -D NO_REGS -D _HPUX_SOURCE -D NOSMP -D THREADED_RTS -include /build/ghc/ghc-9.0.2/includes/dist-install/build/ghcversion.h -iquote compiler/GHC/Iface -quiet -dumpdir /tmp/ghc13413_0/ -dumpbase ghc_5.hc -dumpbase-ext .hc -O -Wimplicit -fno-PIC -fwrapv -fno-builtin -fno-strict-aliasing -o /tmp/ghc13413_0/ghc_5.s
>>
>> -> now we know that cc1 crashed while compiling some haskell code.
>
> This does seem really useful for debugging.
Yes.
> However, it's also an information disclosure in various ways. The
> arguments of a program are often more sensitive than the name, and logs
> have a tendency to end up in various places, such as bug reports.
>
> An example of how this can be an issue:
> - You receive an email or other message with a sensitive link to follow
> - You open the link, which launches `firefox https://...`
> - You continue browsing from that window
> - Firefox crashes (and recovers and restarts, so you don't think
> anything of it)
> - Later, you report a bug on a different piece of software, and the bug
> reporting process includes a copy of the kernel log
Yes, that's a possible way how such information can leak.
> I am *not* saying that we shouldn't do this; it seems quite helpful.
> However, I think we need to arrange to treat this as sensitive
> information, similar to kptr_restrict.
I wonder what the best solution could be.
A somewhat trivial solution is to combine it with the dmesg_restrict sysctl, e.g.:
* When ``dmesg_restrict`` is set to 0 there are no restrictions for users to read
dmesg. In this case my patch would limit the information (based on example above):
cc1[13472] cmdline: /usr/lib/gcc/hppa-linux-gnu/12/cc1 [note: other parameters hidden due to dmesg_restrict=0 sysctl]
So it would show the full argv[0] with a hint that people would need to change dmesg_restrict.
* When ``dmesg_restrict`` is set to 1, users must have ``CAP_SYSLOG`` to use dmesg(8)
and the patch could output all parameters:
cc1[13472] cmdline: /usr/lib/gcc/hppa-linux-gnu/12/cc1 -quiet @/tmp/ccRkFSfY -imultilib . -imultiarch hppa-linux-gnu ....
That would of course still leave few possible corner-cases where information
could leak, but since usually programs shouldn't crash and that
people usually shouldn't put sensitive information into the parameter
list directly, it's somewhat unlikely to happen.
Another different solution would be to add another sysctl.
Any other ideas?
Helge
On Tue, Aug 02, 2022 at 09:40:50PM +0200, Helge Deller wrote:
> On 8/1/22 18:57, Josh Triplett wrote:
> > On Mon, Aug 01, 2022 at 05:20:13PM +0200, Helge Deller wrote:
> >> This patch series allows the arch-specific kernel fault handlers to dump
> >> in addition to the typical info (IP address, fault type, backtrace and so on)
> >> the command line of the faulting process.
> >>
> >> The motivation for this patch is that it's sometimes quite hard to find out and
> >> annoying to not know which program *exactly* faulted when looking at the syslog.
> >>
> >> Some examples from the syslog are:
> >>
> >> On parisc:
> >> do_page_fault() command='cc1' type=15 address=0x00000000 in libc-2.33.so[f6abb000+184000]
> >> CPU: 1 PID: 13472 Comm: cc1 Tainted: G E 5.10.133+ #45
> >> Hardware name: 9000/785/C8000
> >>
> >> -> We see the "cc1" compiler crashed, but it would be useful to know which file was compiled.
> >>
> >> With this patch series, the kernel now prints in addition:
> >> cc1[13472] cmdline: /usr/lib/gcc/hppa-linux-gnu/12/cc1 -quiet @/tmp/ccRkFSfY -imultilib . -imultiarch hppa-linux-gnu -D USE_MINIINTERPRETER -D NO_REGS -D _HPUX_SOURCE -D NOSMP -D THREADED_RTS -include /build/ghc/ghc-9.0.2/includes/dist-install/build/ghcversion.h -iquote compiler/GHC/Iface -quiet -dumpdir /tmp/ghc13413_0/ -dumpbase ghc_5.hc -dumpbase-ext .hc -O -Wimplicit -fno-PIC -fwrapv -fno-builtin -fno-strict-aliasing -o /tmp/ghc13413_0/ghc_5.s
> >>
> >> -> now we know that cc1 crashed while compiling some haskell code.
> >
> > This does seem really useful for debugging.
>
> Yes.
>
> > However, it's also an information disclosure in various ways. The
> > arguments of a program are often more sensitive than the name, and logs
> > have a tendency to end up in various places, such as bug reports.
> >
> > An example of how this can be an issue:
> > - You receive an email or other message with a sensitive link to follow
> > - You open the link, which launches `firefox https://...`
> > - You continue browsing from that window
> > - Firefox crashes (and recovers and restarts, so you don't think
> > anything of it)
> > - Later, you report a bug on a different piece of software, and the bug
> > reporting process includes a copy of the kernel log
>
> Yes, that's a possible way how such information can leak.
>
> > I am *not* saying that we shouldn't do this; it seems quite helpful.
> > However, I think we need to arrange to treat this as sensitive
> > information, similar to kptr_restrict.
>
> I wonder what the best solution could be.
>
> A somewhat trivial solution is to combine it with the dmesg_restrict sysctl, e.g.:
>
> * When ``dmesg_restrict`` is set to 0 there are no restrictions for users to read
> dmesg. In this case my patch would limit the information (based on example above):
> cc1[13472] cmdline: /usr/lib/gcc/hppa-linux-gnu/12/cc1 [note: other parameters hidden due to dmesg_restrict=0 sysctl]
> So it would show the full argv[0] with a hint that people would need to change dmesg_restrict.
>
> * When ``dmesg_restrict`` is set to 1, users must have ``CAP_SYSLOG`` to use dmesg(8)
> and the patch could output all parameters:
> cc1[13472] cmdline: /usr/lib/gcc/hppa-linux-gnu/12/cc1 -quiet @/tmp/ccRkFSfY -imultilib . -imultiarch hppa-linux-gnu ....
>
> That would of course still leave few possible corner-cases where information
> could leak, but since usually programs shouldn't crash and that
> people usually shouldn't put sensitive information into the parameter
> list directly, it's somewhat unlikely to happen.
>
> Another different solution would be to add another sysctl.
>
> Any other ideas?
I don't think we should overload the meaning of dmesg_restrict. But
overloading kptr_restrict seems reasonable to me. (Including respecting
kptr_restrict==2 by not showing this at all.)
On 8/4/22 10:09, Josh Triplett wrote:
> On Tue, Aug 02, 2022 at 09:40:50PM +0200, Helge Deller wrote:
>> On 8/1/22 18:57, Josh Triplett wrote:
>>> On Mon, Aug 01, 2022 at 05:20:13PM +0200, Helge Deller wrote:
>>>> This patch series allows the arch-specific kernel fault handlers to dump
>>>> in addition to the typical info (IP address, fault type, backtrace and so on)
>>>> the command line of the faulting process.
>>>>
>>>> The motivation for this patch is that it's sometimes quite hard to find out and
>>>> annoying to not know which program *exactly* faulted when looking at the syslog.
>>>>
>>>> Some examples from the syslog are:
>>>>
>>>> On parisc:
>>>> do_page_fault() command='cc1' type=15 address=0x00000000 in libc-2.33.so[f6abb000+184000]
>>>> CPU: 1 PID: 13472 Comm: cc1 Tainted: G E 5.10.133+ #45
>>>> Hardware name: 9000/785/C8000
>>>>
>>>> -> We see the "cc1" compiler crashed, but it would be useful to know which file was compiled.
>>>>
>>>> With this patch series, the kernel now prints in addition:
>>>> cc1[13472] cmdline: /usr/lib/gcc/hppa-linux-gnu/12/cc1 -quiet @/tmp/ccRkFSfY -imultilib . -imultiarch hppa-linux-gnu -D USE_MINIINTERPRETER -D NO_REGS -D _HPUX_SOURCE -D NOSMP -D THREADED_RTS -include /build/ghc/ghc-9.0.2/includes/dist-install/build/ghcversion.h -iquote compiler/GHC/Iface -quiet -dumpdir /tmp/ghc13413_0/ -dumpbase ghc_5.hc -dumpbase-ext .hc -O -Wimplicit -fno-PIC -fwrapv -fno-builtin -fno-strict-aliasing -o /tmp/ghc13413_0/ghc_5.s
>>>>
>>>> -> now we know that cc1 crashed while compiling some haskell code.
>>>
>>> This does seem really useful for debugging.
>>
>> Yes.
>>
>>> However, it's also an information disclosure in various ways. The
>>> arguments of a program are often more sensitive than the name, and logs
>>> have a tendency to end up in various places, such as bug reports.
>>>
>>> An example of how this can be an issue:
>>> - You receive an email or other message with a sensitive link to follow
>>> - You open the link, which launches `firefox https://...`
>>> - You continue browsing from that window
>>> - Firefox crashes (and recovers and restarts, so you don't think
>>> anything of it)
>>> - Later, you report a bug on a different piece of software, and the bug
>>> reporting process includes a copy of the kernel log
>>
>> Yes, that's a possible way how such information can leak.
>>
>>> I am *not* saying that we shouldn't do this; it seems quite helpful.
>>> However, I think we need to arrange to treat this as sensitive
>>> information, similar to kptr_restrict.
>>
>> I wonder what the best solution could be.
>>
>> A somewhat trivial solution is to combine it with the dmesg_restrict sysctl, e.g.:
>>
>> * When ``dmesg_restrict`` is set to 0 there are no restrictions for users to read
>> dmesg. In this case my patch would limit the information (based on example above):
>> cc1[13472] cmdline: /usr/lib/gcc/hppa-linux-gnu/12/cc1 [note: other parameters hidden due to dmesg_restrict=0 sysctl]
>> So it would show the full argv[0] with a hint that people would need to change dmesg_restrict.
>>
>> * When ``dmesg_restrict`` is set to 1, users must have ``CAP_SYSLOG`` to use dmesg(8)
>> and the patch could output all parameters:
>> cc1[13472] cmdline: /usr/lib/gcc/hppa-linux-gnu/12/cc1 -quiet @/tmp/ccRkFSfY -imultilib . -imultiarch hppa-linux-gnu ....
>>
>> That would of course still leave few possible corner-cases where information
>> could leak, but since usually programs shouldn't crash and that
>> people usually shouldn't put sensitive information into the parameter
>> list directly, it's somewhat unlikely to happen.
>>
>> Another different solution would be to add another sysctl.
>>
>> Any other ideas?
>
> I don't think we should overload the meaning of dmesg_restrict. But
> overloading kptr_restrict seems reasonable to me. (Including respecting
> kptr_restrict==2 by not showing this at all.)
I'm fine with kptr_restrict, but I'm puzzled for which value of kptr_restrict
the command line should be shown then.
By looking at the meaning of kptr_restrict, I think the command line should be
hidden for values 0-2.
Do you suggest to add a new value "3" or am I missing something?
Helge
On Thu, Aug 04, 2022 at 10:39:52AM +0200, Helge Deller wrote:
> On 8/4/22 10:09, Josh Triplett wrote:
> > On Tue, Aug 02, 2022 at 09:40:50PM +0200, Helge Deller wrote:
> >> On 8/1/22 18:57, Josh Triplett wrote:
> >>> However, it's also an information disclosure in various ways. The
> >>> arguments of a program are often more sensitive than the name, and logs
> >>> have a tendency to end up in various places, such as bug reports.
> >>>
> >>> An example of how this can be an issue:
> >>> - You receive an email or other message with a sensitive link to follow
> >>> - You open the link, which launches `firefox https://...`
> >>> - You continue browsing from that window
> >>> - Firefox crashes (and recovers and restarts, so you don't think
> >>> anything of it)
> >>> - Later, you report a bug on a different piece of software, and the bug
> >>> reporting process includes a copy of the kernel log
> >>
> >> Yes, that's a possible way how such information can leak.
> >>
> >>> I am *not* saying that we shouldn't do this; it seems quite helpful.
> >>> However, I think we need to arrange to treat this as sensitive
> >>> information, similar to kptr_restrict.
[...]
> > I don't think we should overload the meaning of dmesg_restrict. But
> > overloading kptr_restrict seems reasonable to me. (Including respecting
> > kptr_restrict==2 by not showing this at all.)
>
> I'm fine with kptr_restrict, but I'm puzzled for which value of kptr_restrict
> the command line should be shown then.
> By looking at the meaning of kptr_restrict, I think the command line should be
> hidden for values 0-2.
> Do you suggest to add a new value "3" or am I missing something?
I'm suggesting treating it the same as a pointer value:
0: always show command line
1: show command line if read by privileged caller
2: never show command line
That could either use kptr_restrict or use a separate cmdline_restrict.