The stackleak_erase() function is called on the trampoline stack at the
end of syscall. This stack is not big enough for ftrace operations,
e.g. it can be overflowed if we enable kprobe_events for stackleak_erase().
Let's disable ftrace for stackleak.c to avoid such situations.
Reported-by: kernel test robot <[email protected]>
Signed-off-by: Alexander Popov <[email protected]>
Reviewed-by: Kees Cook <[email protected]>
---
kernel/Makefile | 1 +
1 file changed, 1 insertion(+)
diff --git a/kernel/Makefile b/kernel/Makefile
index 7343b3a..0906f6d 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -18,6 +18,7 @@ obj-$(CONFIG_MULTIUSER) += groups.o
ifdef CONFIG_FUNCTION_TRACER
# Do not trace internal ftrace files
CFLAGS_REMOVE_irq_work.o = $(CC_FLAGS_FTRACE)
+CFLAGS_REMOVE_stackleak.o = $(CC_FLAGS_FTRACE)
endif
# Prevents flicker of uninteresting __do_softirq()/__local_bh_disable_ip()
--
2.7.4
On Sun, 11 Nov 2018 01:05:30 +0300
Alexander Popov <[email protected]> wrote:
> The stackleak_erase() function is called on the trampoline stack at the
> end of syscall. This stack is not big enough for ftrace operations,
> e.g. it can be overflowed if we enable kprobe_events for stackleak_erase().
Is the issue with kprobes or with function tracing? Because this stops
function tracing which I only want disabled if function tracing itself
is an issue, not for other things that may use the function tracing
infrastructure.
-- Steve
On 11.11.2018 2:30, Steven Rostedt wrote:
> On Sun, 11 Nov 2018 01:05:30 +0300
> Alexander Popov <[email protected]> wrote:
>
>> The stackleak_erase() function is called on the trampoline stack at the
>> end of syscall. This stack is not big enough for ftrace operations,
>> e.g. it can be overflowed if we enable kprobe_events for stackleak_erase().
>
> Is the issue with kprobes or with function tracing? Because this stops
> function tracing which I only want disabled if function tracing itself
> is an issue, not for other things that may use the function tracing
> infrastructure.
Hello Steven,
I believe that stackleak erasing is not compatible with function tracing itself.
That's what the kernel testing robot has hit:
https://www.openwall.com/lists/kernel-hardening/2018/11/09/1
I used kprobe_events just to reproduce the problem:
https://www.openwall.com/lists/kernel-hardening/2018/11/09/4
Best regards,
Alexander
On Sun, 11 Nov 2018 13:19:45 +0300
Alexander Popov <[email protected]> wrote:
> On 11.11.2018 2:30, Steven Rostedt wrote:
> > On Sun, 11 Nov 2018 01:05:30 +0300
> > Alexander Popov <[email protected]> wrote:
> >
> >> The stackleak_erase() function is called on the trampoline stack at the
> >> end of syscall. This stack is not big enough for ftrace operations,
> >> e.g. it can be overflowed if we enable kprobe_events for stackleak_erase().
> >
> > Is the issue with kprobes or with function tracing? Because this stops
> > function tracing which I only want disabled if function tracing itself
> > is an issue, not for other things that may use the function tracing
> > infrastructure.
>
> Hello Steven,
>
> I believe that stackleak erasing is not compatible with function tracing itself.
> That's what the kernel testing robot has hit:
> https://www.openwall.com/lists/kernel-hardening/2018/11/09/1
>
> I used kprobe_events just to reproduce the problem:
> https://www.openwall.com/lists/kernel-hardening/2018/11/09/4
Have you tried adding a "notrace" to stackleak_erase()?
Not tracing the entire file is a bit of overkill. There's no reason
ftrace can't trace stack_erasing_sysctl() or perhaps even
stackleak_track_stack() as that may be very interesting to trace.
-- Steve
Hi Alexander and Steve,
On Sun, 11 Nov 2018 20:53:51 -0500
Steven Rostedt <[email protected]> wrote:
> On Sun, 11 Nov 2018 13:19:45 +0300
> Alexander Popov <[email protected]> wrote:
>
> > On 11.11.2018 2:30, Steven Rostedt wrote:
> > > On Sun, 11 Nov 2018 01:05:30 +0300
> > > Alexander Popov <[email protected]> wrote:
> > >
> > >> The stackleak_erase() function is called on the trampoline stack at the
> > >> end of syscall. This stack is not big enough for ftrace operations,
> > >> e.g. it can be overflowed if we enable kprobe_events for stackleak_erase().
> > >
> > > Is the issue with kprobes or with function tracing? Because this stops
> > > function tracing which I only want disabled if function tracing itself
> > > is an issue, not for other things that may use the function tracing
> > > infrastructure.
> >
> > Hello Steven,
> >
> > I believe that stackleak erasing is not compatible with function tracing itself.
> > That's what the kernel testing robot has hit:
> > https://www.openwall.com/lists/kernel-hardening/2018/11/09/1
> >
> > I used kprobe_events just to reproduce the problem:
> > https://www.openwall.com/lists/kernel-hardening/2018/11/09/4
>
> Have you tried adding a "notrace" to stackleak_erase()?
>
> Not tracing the entire file is a bit of overkill. There's no reason
> ftrace can't trace stack_erasing_sysctl() or perhaps even
> stackleak_track_stack() as that may be very interesting to trace.
I think it is not enough for stopping kprobes. If you want to stop the kprobes
(int3 version) on stackleak_erase(), you should use NOKPROBE_SYMBOL(stackleak_erase),
since kprobes can work without ftrace.
Thank you,
--
Masami Hiramatsu <[email protected]>
----- On Nov 11, 2018, at 9:50 PM, Masami Hiramatsu [email protected] wrote:
> Hi Alexander and Steve,
>
> On Sun, 11 Nov 2018 20:53:51 -0500
> Steven Rostedt <[email protected]> wrote:
>
>> On Sun, 11 Nov 2018 13:19:45 +0300
>> Alexander Popov <[email protected]> wrote:
>>
>> > On 11.11.2018 2:30, Steven Rostedt wrote:
>> > > On Sun, 11 Nov 2018 01:05:30 +0300
>> > > Alexander Popov <[email protected]> wrote:
>> > >
>> > >> The stackleak_erase() function is called on the trampoline stack at the
>> > >> end of syscall. This stack is not big enough for ftrace operations,
>> > >> e.g. it can be overflowed if we enable kprobe_events for stackleak_erase().
>> > >
>> > > Is the issue with kprobes or with function tracing? Because this stops
>> > > function tracing which I only want disabled if function tracing itself
>> > > is an issue, not for other things that may use the function tracing
>> > > infrastructure.
>> >
>> > Hello Steven,
>> >
>> > I believe that stackleak erasing is not compatible with function tracing itself.
>> > That's what the kernel testing robot has hit:
>> > https://www.openwall.com/lists/kernel-hardening/2018/11/09/1
>> >
>> > I used kprobe_events just to reproduce the problem:
>> > https://www.openwall.com/lists/kernel-hardening/2018/11/09/4
>>
>> Have you tried adding a "notrace" to stackleak_erase()?
>>
>> Not tracing the entire file is a bit of overkill. There's no reason
>> ftrace can't trace stack_erasing_sysctl() or perhaps even
>> stackleak_track_stack() as that may be very interesting to trace.
>
> I think it is not enough for stopping kprobes. If you want to stop the kprobes
> (int3 version) on stackleak_erase(), you should use
> NOKPROBE_SYMBOL(stackleak_erase),
> since kprobes can work without ftrace.
Just to clarify: AFAIU you guys are recommending to add _both_ a "notrace"
annotation to stackleak_erase() _and_ a NOKPROBE_SYMBOL(stackleak_erase),
so neither function tracing nor kprobes can hook on that function.
Thanks,
Mathieu
--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
On Mon, 12 Nov 2018 09:53:29 -0500 (EST)
Mathieu Desnoyers <[email protected]> wrote:
> Just to clarify: AFAIU you guys are recommending to add _both_ a "notrace"
> annotation to stackleak_erase() _and_ a NOKPROBE_SYMBOL(stackleak_erase),
> so neither function tracing nor kprobes can hook on that function.
Correct.
-- Steve
Hello Steven and Masami,
Thanks for your comments.
On 12.11.2018 5:50, Masami Hiramatsu wrote:
> Hi Alexander and Steve,
>
> On Sun, 11 Nov 2018 20:53:51 -0500
> Steven Rostedt <[email protected]> wrote:
>
>> On Sun, 11 Nov 2018 13:19:45 +0300
>> Alexander Popov <[email protected]> wrote:
>>
>>> On 11.11.2018 2:30, Steven Rostedt wrote:
>>>> On Sun, 11 Nov 2018 01:05:30 +0300
>>>> Alexander Popov <[email protected]> wrote:
>>>>
>>>>> The stackleak_erase() function is called on the trampoline stack at the
>>>>> end of syscall. This stack is not big enough for ftrace operations,
>>>>> e.g. it can be overflowed if we enable kprobe_events for stackleak_erase().
>>>>
>>>> Is the issue with kprobes or with function tracing? Because this stops
>>>> function tracing which I only want disabled if function tracing itself
>>>> is an issue, not for other things that may use the function tracing
>>>> infrastructure.
>>>
>>> Hello Steven,
>>>
>>> I believe that stackleak erasing is not compatible with function tracing itself.
>>> That's what the kernel testing robot has hit:
>>> https://www.openwall.com/lists/kernel-hardening/2018/11/09/1
>>>
>>> I used kprobe_events just to reproduce the problem:
>>> https://www.openwall.com/lists/kernel-hardening/2018/11/09/4
>>
>> Have you tried adding a "notrace" to stackleak_erase()?
>>
>> Not tracing the entire file is a bit of overkill. There's no reason
>> ftrace can't trace stack_erasing_sysctl() or perhaps even
>> stackleak_track_stack() as that may be very interesting to trace.
Yes, thank you. It's much better.
> I think it is not enough for stopping kprobes. If you want to stop the kprobes
> (int3 version) on stackleak_erase(), you should use NOKPROBE_SYMBOL(stackleak_erase),
> since kprobes can work without ftrace.
Thanks!
I learned how to use kprobes without ftrace and managed to reproduce the problem
as well (I modified kprobe_example.c and kretprobe_example.c). NOKPROBE_SYMBOL()
allowed to avoid it.
I'll send the patch soon.
By the way, are there any other tracing/instrumentation mechanisms that should
be disabled?
Best regards,
Alexander
On Mon, 12 Nov 2018 19:51:00 +0300
Alexander Popov <[email protected]> wrote:
> By the way, are there any other tracing/instrumentation mechanisms that should
> be disabled?
ftrace and kprobes are pretty much the only ones that currently do self
modification of code all over the kernel. Kprobes even more so than
ftrace.
-- Steve
On Mon, 12 Nov 2018 12:21:48 -0500
Steven Rostedt <[email protected]> wrote:
> On Mon, 12 Nov 2018 19:51:00 +0300
> Alexander Popov <[email protected]> wrote:
>
> > By the way, are there any other tracing/instrumentation mechanisms that should
> > be disabled?
>
> ftrace and kprobes are pretty much the only ones that currently do self
> modification of code all over the kernel. Kprobes even more so than
> ftrace.
Right, since kprobes uses int3 or sw breakpoint exception for hooking into
the code, it consumes stack much more.
Thank you,
--
Masami Hiramatsu <[email protected]>