by 平松雅巳 / HIRAMATU，MASAMI

[permalink] [raw]

Subject: Re: Re: perf, kprobes: fuzzer generates huge number of WARNings

On 2015/07/08 6:21, Alexei Starovoitov wrote:
> On Tue, Jul 07, 2015 at 05:08:51PM -0400, Vince Weaver wrote:
>> On Tue, 7 Jul 2015, Alexei Starovoitov wrote:
>>
>>> On Tue, Jul 07, 2015 at 12:00:12AM -0400, Vince Weaver wrote:
>>>>
>>>> Well the BPF hack is in the fuzzer, not the kernel. And it's not really a
>>>> hack, it just turned out to be a huge pain to figure out how to
>>>> manually create a valid BPF program in conjunction with a valid kprobe
>>>> event.
>>>
>>> You mean automatically generating valid bpf program? That's definitely hard.
>>> If you mean just few hardcoded programs then take them from samples or
>>> from test_bpf ?
>>
>> there's already code in trinity that in theory autogenerates bpf programs,
>> but for now I was just trying to hook up a short known valid one.
>>
>> it might not be possible to really test things though, as you need to be
>> root to create a kprobe and attach a BPF program, but my fuzzer when run
>> as root often does all kinds of other stuff that will crash a machine.
>> Is it ever planned to allow using bpf/kprobes without requiring full
>> CAP_ADMIN privledges?
>
> I suspect kprobes will forever be root only, whereas for bpf I'm thinking
> to introduce CAP_BPF, but before that we need to finish constant blinding
> and add address leak prevention. So not soon.

Currently I don't plan to do that. Actually systemtap allows that, but
with much bigger blacklist. I think we can make a tool which also
allows user to add new events on the limited functions (white-list).
But anyway, since these can expose kernel function addresses to users,
it is highly recommended to limit users by some capabilities.

>>>> I did have to sprinkle printks in the kprobe and bpf code to find out
>>>> where various EINVAL returns were coming from, so potentially this is just
>>>> a problem of printks happening where they shouldn't. I'll remove those
>>>> changes and try to reproduce this tomorrow.
>>>
>>> could you please elaborate on this further. Which EINVALs you talking about?
>>
>> When you are trying to create a kprobe and bpf file there's about 10
>> different ways to get EINVAL as a return value and no way of knowing which
>> one you are hitting. I added printks so I could know what issue was
>> causing the einval. (from memory, the problems I hit were not zeroing out
>> the attr structure, having a wrong instruction count, and a few others).

Hmm I must fix some parts of kprobes by changing retval or showing more
precise messages.

Thanks!

> I see. I guess anyone trying to use syscall directly will be facing such
> issues, but libbpf that is being developed to be used by perf and others
> should solve these problems.

--
Masami HIRAMATSU
Linux Technology Research Center, System Productivity Research Dept.
Center for Technology Innovation - Systems Engineering
Hitachi, Ltd., Research & Development Group
E-mail: [email protected]

2015-07-09 09:58:16

by 平松雅巳 / HIRAMATU，MASAMI

[permalink] [raw]

Subject: Re: Re: perf, kprobes: fuzzer generates huge number of WARNings

On 2015/07/08 0:11, Vince Weaver wrote:
> On Tue, 7 Jul 2015, Vince Weaver wrote:
>
>> On Tue, 7 Jul 2015, Masami Hiramatsu wrote:
>>
>>>> It's a Hawell machine running 4.2-rc1. I can post the .config if it's of
>>>> interest.
>>>
>>> Yes, I'm interested in.
>>
>> I've put it here:
>> http://web.eece.maine.edu/~vweaver/junk/haswell-config
>>
>>> OK, and also, if you have a chance, please run the ftracetest as below.
>>>
>>> $ cd tools/testing/selftest/ftrace/
>>> $ sudo ./ftracetest
>>>
>>> This will do a series of basic tests on ftrace and report it.
>>
>> It won't run:
>> sudo ./ftracetest
>> Error: No ftrace directory found
>>
>
> OK here are the results when I mounted debugfs in the right place.

Thanks, I've found do_fork does not exist anymore... that caused the
failure.
Unsupported results are just because of the lack of function profiler
from kconfig.

Thank you for reporting!

>
> sudo ./ftracetest
> === Ftrace unit tests ===
> [1] Basic trace file check [PASS]
> [2] Basic test for tracers [PASS]
> [3] Basic trace clock test [PASS]
> [4] Basic event tracing check [PASS]
> [5] event tracing - enable/disable with event level files [PASS]
> [6] event tracing - enable/disable with subsystem level files [PASS]
> [7] event tracing - enable/disable with top level files [PASS]
> [8] ftrace - function graph filters with stack tracer [UNSUPPORTED]
> execute:
> /home/vince/research/linux-kernel/linux-git/tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter-stack.tc
> + .
> /home/vince/research/linux-kernel/linux-git/tools/testing/selftests/ftrace/test.d/ftrace/fgraph-filter-stack.tc
> + grep -q function_graph available_tracers
> + [ ! -f set_ftrace_filter ]
> + disable_tracing
> + echo 0
> + clear_trace
> + echo
> + echo schedule
> + echo function_graph
> + [ ! -f stack_trace ]
> + echo Stack tracer not configured
> Stack tracer not configured
> + do_reset
> + reset_tracer
> + echo nop
> + [ -e /proc/sys/kernel/stack_tracer_enabled ]
> + enable_tracing
> + echo 1
> + clear_trace
> + echo
> + echo
> + exit_unsupported
> + kill -s 40 2725
> + exit 0
> [9] ftrace - function graph filters [PASS]
> [10] ftrace - function profiler with function tracing [UNSUPPORTED]
> execute:
> /home/vince/research/linux-kernel/linux-git/tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc
> + .
> /home/vince/research/linux-kernel/linux-git/tools/testing/selftests/ftrace/test.d/ftrace/func_profiler.tc
> + grep -q function_graph available_tracers
> + [ ! -f set_ftrace_filter ]
> + [ ! -f function_profile_enabled ]
> + echo function_profile_enabled not found, function profiling enabled?
> function_profile_enabled not found, function profiling enabled?
> + exit_unsupported
> + kill -s 40 2725
> + exit 0
> [11] Kprobe dynamic event - adding and removing [FAIL]
> execute:
> /home/vince/research/linux-kernel/linux-git/tools/testing/selftests/ftrace/test.d/kprobe/add_and_remove.tc
> + .
> /home/vince/research/linux-kernel/linux-git/tools/testing/selftests/ftrace/test.d/kprobe/add_and_remove.tc
> + [ -f kprobe_events ]
> + echo 0
> + echo
> + echo p:myevent do_fork
> sh: echo: I/O error
> [12] Kprobe dynamic event - busy event check [FAIL]
> execute:
> /home/vince/research/linux-kernel/linux-git/tools/testing/selftests/ftrace/test.d/kprobe/busy_check.tc
> + .
> /home/vince/research/linux-kernel/linux-git/tools/testing/selftests/ftrace/test.d/kprobe/busy_check.tc
> + [ -f kprobe_events ]
> + echo 0
> + echo
> + echo p:myevent do_fork
> sh: echo: I/O error
> [13] Kprobe dynamic event with arguments [FAIL]
> execute:
> /home/vince/research/linux-kernel/linux-git/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_args.tc
> + .
> /home/vince/research/linux-kernel/linux-git/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_args.tc
> + [ -f kprobe_events ]
> + echo 0
> + echo
> + echo p:testprobe do_fork $stack $stack0 +0($stack)
> sh: echo: I/O error
> [14] Kprobe dynamic event with function tracer [FAIL]
> execute:
> /home/vince/research/linux-kernel/linux-git/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_ftrace.tc
> + .
> /home/vince/research/linux-kernel/linux-git/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_ftrace.tc
> + [ -f kprobe_events ]
> + grep function available_tracers
> blk function_graph function nop
> + echo nop
> + echo do_fork
> sh: echo: I/O error
> [15] Kretprobe dynamic event with arguments [FAIL]
> execute:
> /home/vince/research/linux-kernel/linux-git/tools/testing/selftests/ftrace/test.d/kprobe/kretprobe_args.tc
> + .
> /home/vince/research/linux-kernel/linux-git/tools/testing/selftests/ftrace/test.d/kprobe/kretprobe_args.tc
> + [ -f kprobe_events ]
> + echo 0
> + echo
> + echo r:testprobe2 do_fork $retval
> sh: echo: I/O error
>
> # of passed: 8
> # of failed: 5
> # of unresolved: 0
> # of untested: 0
> # of unsupported: 2
> # of xfailed: 0
> # of undefined(test bug): 0
>
>

--
Masami HIRAMATSU
Linux Technology Research Center, System Productivity Research Dept.
Center for Technology Innovation - Systems Engineering
Hitachi, Ltd., Research & Development Group
E-mail: [email protected]