2018-04-10 14:52:05

by Vince Weaver

[permalink] [raw]
Subject: perf: fuzzer leads to trace_kprobe: Could not insert message flood

Author: Song Liu <[email protected]>
Date: Wed Dec 6 14:45:15 2017 -0800

When running the perf_fuzzer on a current git checkout my logs are flooded
with messages such as this:
[71487.869077] trace_kprobe: Could not insert probe at unknown+0: -22
[71488.174479] trace_kprobe: Could not insert probe at unknown+0: -22

Presumably this is due to the introduction of the perf_kprobe PMU in
commit e12f03d7031a977356e3d7b75a68c2185ff8d155
Author: Song Liu <[email protected]>
Date: Wed Dec 6 14:45:15 2017 -0800

Is there a way to get this error disabled, or else rate-limited?

Vince


2018-04-10 20:03:16

by Song Liu

[permalink] [raw]
Subject: Re: perf: fuzzer leads to trace_kprobe: Could not insert message flood



> On Apr 10, 2018, at 7:48 AM, Vince Weaver <[email protected]> wrote:
>
> Author: Song Liu <[email protected]>
> Date: Wed Dec 6 14:45:15 2017 -0800
>
> When running the perf_fuzzer on a current git checkout my logs are flooded
> with messages such as this:
> [71487.869077] trace_kprobe: Could not insert probe at unknown+0: -22
> [71488.174479] trace_kprobe: Could not insert probe at unknown+0: -22
>
> Presumably this is due to the introduction of the perf_kprobe PMU in
> commit e12f03d7031a977356e3d7b75a68c2185ff8d155
> Author: Song Liu <[email protected]>
> Date: Wed Dec 6 14:45:15 2017 -0800
>
> Is there a way to get this error disabled, or else rate-limited?
>
> Vince

Hi Vince,

Thanks for the report.

This is a new API that creates probe together with perf_event_open(). Based on
my limited understanding of perf_fuzzer, it doesn't understand this API, and uses
it in an abnormal way. I would recommend perf_fuzzer to understand this new
API and test it. For more information about using this API, please refer to the
man-page diff available at:

https://patchwork.kernel.org/patch/10097283/

Thanks again for the test and report. Please let me know if you have further
questions.

Best,
Song

2018-04-11 12:08:48

by Ingo Molnar

[permalink] [raw]
Subject: Re: perf: fuzzer leads to trace_kprobe: Could not insert message flood


* Song Liu <[email protected]> wrote:

>
>
> > On Apr 10, 2018, at 7:48 AM, Vince Weaver <[email protected]> wrote:
> >
> > Author: Song Liu <[email protected]>
> > Date: Wed Dec 6 14:45:15 2017 -0800
> >
> > When running the perf_fuzzer on a current git checkout my logs are flooded
> > with messages such as this:
> > [71487.869077] trace_kprobe: Could not insert probe at unknown+0: -22
> > [71488.174479] trace_kprobe: Could not insert probe at unknown+0: -22
> >
> > Presumably this is due to the introduction of the perf_kprobe PMU in
> > commit e12f03d7031a977356e3d7b75a68c2185ff8d155
> > Author: Song Liu <[email protected]>
> > Date: Wed Dec 6 14:45:15 2017 -0800
> >
> > Is there a way to get this error disabled, or else rate-limited?
> >
> > Vince
>
> Hi Vince,
>
> Thanks for the report.
>
> This is a new API that creates probe together with perf_event_open(). Based on
> my limited understanding of perf_fuzzer, it doesn't understand this API, and uses
> it in an abnormal way. [...]

Vince's point is valid: the kernel log should not be flooded with pointless
messages as a response to user-space ABI uses ...

Why is there a kernel log message at all, isn't an error returned?

> [...] I would recommend perf_fuzzer to understand this new API and test it.
> [...]

This bug needs to be fixed: a new API must not effectively DoS fuzzing efforts by
spamming the kernel log ...

Thanks,

Ingo

2018-04-11 18:11:58

by Song Liu

[permalink] [raw]
Subject: Re: perf: fuzzer leads to trace_kprobe: Could not insert message flood



> On Apr 11, 2018, at 5:04 AM, Ingo Molnar <[email protected]> wrote:
>
>
> * Song Liu <[email protected]> wrote:
>
>>
>>
>>> On Apr 10, 2018, at 7:48 AM, Vince Weaver <[email protected]> wrote:
>>>
>>> Author: Song Liu <[email protected]>
>>> Date: Wed Dec 6 14:45:15 2017 -0800
>>>
>>> When running the perf_fuzzer on a current git checkout my logs are flooded
>>> with messages such as this:
>>> [71487.869077] trace_kprobe: Could not insert probe at unknown+0: -22
>>> [71488.174479] trace_kprobe: Could not insert probe at unknown+0: -22
>>>
>>> Presumably this is due to the introduction of the perf_kprobe PMU in
>>> commit e12f03d7031a977356e3d7b75a68c2185ff8d155
>>> Author: Song Liu <[email protected]>
>>> Date: Wed Dec 6 14:45:15 2017 -0800
>>>
>>> Is there a way to get this error disabled, or else rate-limited?
>>>
>>> Vince
>>
>> Hi Vince,
>>
>> Thanks for the report.
>>
>> This is a new API that creates probe together with perf_event_open(). Based on
>> my limited understanding of perf_fuzzer, it doesn't understand this API, and uses
>> it in an abnormal way. [...]
>
> Vince's point is valid: the kernel log should not be flooded with pointless
> messages as a response to user-space ABI uses ...
>
> Why is there a kernel log message at all, isn't an error returned?
>
>> [...] I would recommend perf_fuzzer to understand this new API and test it.
>> [...]
>
> This bug needs to be fixed: a new API must not effectively DoS fuzzing efforts by
> spamming the kernel log ...

Yeah, the new API allows non-root user to trigger this message. We should only
allow root to create kprobe with perf_event_open().

On the other hand, do we need to fix this for root? In fact, a simple bash loop
can create something similar through the text interface (with root):

root@virt-test:~# for x in {0..5} ; do echo p:xx xx+$x >> /sys/kernel/debug/tracing/kprobe_events ; done
-bash: echo: write error: No such file or directory
-bash: echo: write error: No such file or directory
-bash: echo: write error: No such file or directory
-bash: echo: write error: No such file or directory
-bash: echo: write error: No such file or directory
-bash: echo: write error: No such file or directory
root@virt-test:~# dmesg | tail -n 5
[ 664.208374] trace_kprobe: Could not insert probe at xx+1: -2
[ 664.237882] trace_kprobe: Could not insert probe at xx+2: -2
[ 664.268067] trace_kprobe: Could not insert probe at xx+3: -2
[ 664.297395] trace_kprobe: Could not insert probe at xx+4: -2
[ 664.327614] trace_kprobe: Could not insert probe at xx+5: -2

This happens before the new API is introduced.

The following patch does capable(CAP_SYS_ADMIN) for perf_kprobe and
perf_uprobe at an earlier stage, so non-root user cannot trigger
this error message. Please let me know whether we need to fix this
for root.

Thanks,
Song



From c6708e9e3cd5ba7afb5a7f693b04abf64fec031e Mon Sep 17 00:00:00 2001
From: Song Liu <[email protected]>
Date: Wed, 11 Apr 2018 10:37:00 -0700
Subject: [PATCH] perf: need CAP_SYS_ADMIN to create k/uprobe with
perf_event_open()

Non-root user cannot create kprobe or uprobe through the text-based
interface (kprobe_events, uprobe_events). So they cannot create the
probes with perf_event_open(). To ensure this, we check
capable(CAP_SYS_ADMIN) at perf_[k,u]probe_event_init().

Fixes: e12f03d7031a ("perf/core: Implement the 'perf_kprobe' PMU")
Fixes: 33ea4b24277b ("perf/core: Implement the 'perf_uprobe' PMU")

Signed-off-by: Song Liu <[email protected]>
Reported-by: Vince Weaver <[email protected]>
Cc: Ingo Molnar <[email protected]>
---
kernel/events/core.c | 8 ++++++++
1 file changed, 8 insertions(+)

diff --git a/kernel/events/core.c b/kernel/events/core.c
index d7af828..2d5fe26 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -8400,6 +8400,10 @@ static int perf_kprobe_event_init(struct perf_event *event)

if (event->attr.type != perf_kprobe.type)
return -ENOENT;
+
+ if (!capable(CAP_SYS_ADMIN))
+ return -EACCES;
+
/*
* no branch sampling for probe events
*/
@@ -8437,6 +8441,10 @@ static int perf_uprobe_event_init(struct perf_event *event)

if (event->attr.type != perf_uprobe.type)
return -ENOENT;
+
+ if (!capable(CAP_SYS_ADMIN))
+ return -EACCES;
+
/*
* no branch sampling for probe events
*/
--
2.9.5





2018-04-12 07:56:31

by Ingo Molnar

[permalink] [raw]
Subject: Re: perf: fuzzer leads to trace_kprobe: Could not insert message flood


* Song Liu <[email protected]> wrote:

> > spamming the kernel log ...
>
> Yeah, the new API allows non-root user to trigger this message. We should only
> allow root to create kprobe with perf_event_open().
>
> On the other hand, do we need to fix this for root? In fact, a simple bash loop
> can create something similar through the text interface (with root):
>
> root@virt-test:~# for x in {0..5} ; do echo p:xx xx+$x >> /sys/kernel/debug/tracing/kprobe_events ; done
> -bash: echo: write error: No such file or directory
> -bash: echo: write error: No such file or directory
> -bash: echo: write error: No such file or directory
> -bash: echo: write error: No such file or directory
> -bash: echo: write error: No such file or directory
> -bash: echo: write error: No such file or directory
> root@virt-test:~# dmesg | tail -n 5
> [ 664.208374] trace_kprobe: Could not insert probe at xx+1: -2
> [ 664.237882] trace_kprobe: Could not insert probe at xx+2: -2
> [ 664.268067] trace_kprobe: Could not insert probe at xx+3: -2
> [ 664.297395] trace_kprobe: Could not insert probe at xx+4: -2
> [ 664.327614] trace_kprobe: Could not insert probe at xx+5: -2
>
> This happens before the new API is introduced.
>
> The following patch does capable(CAP_SYS_ADMIN) for perf_kprobe and
> perf_uprobe at an earlier stage, so non-root user cannot trigger
> this error message. Please let me know whether we need to fix this
> for root.

That's two bugs then, and yes, I think we should fix the log spamming: what's the
point? We already get an error code from the write.

I'll apply your CAP_SYS_ADMIN fix.

Thanks,

Ingo

2018-04-12 07:57:26

by Ingo Molnar

[permalink] [raw]
Subject: Re: perf: fuzzer leads to trace_kprobe: Could not insert message flood


* Song Liu <[email protected]> wrote:

> diff --git a/kernel/events/core.c b/kernel/events/core.c
> index d7af828..2d5fe26 100644
> --- a/kernel/events/core.c
> +++ b/kernel/events/core.c
> @@ -8400,6 +8400,10 @@ static int perf_kprobe_event_init(struct perf_event *event)
>
> if (event->attr.type != perf_kprobe.type)
> return -ENOENT;
> +
> + if (!capable(CAP_SYS_ADMIN))
> + return -EACCES;
> +
> /*
> * no branch sampling for probe events
> */
> @@ -8437,6 +8441,10 @@ static int perf_uprobe_event_init(struct perf_event *event)
>
> if (event->attr.type != perf_uprobe.type)
> return -ENOENT;
> +
> + if (!capable(CAP_SYS_ADMIN))
> + return -EACCES;

This is seriously whitespace damaged: all tabs are spaces ...

Thanks,

Ingo

Subject: [tip:perf/urgent] perf/core: Need CAP_SYS_ADMIN to create k/uprobe with perf_event_open()

Commit-ID: 32e6e967fb36bf77ed99221ae3ce1909f045d8f9
Gitweb: https://git.kernel.org/tip/32e6e967fb36bf77ed99221ae3ce1909f045d8f9
Author: Song Liu <[email protected]>
AuthorDate: Wed, 11 Apr 2018 18:02:37 +0000
Committer: Ingo Molnar <[email protected]>
CommitDate: Thu, 12 Apr 2018 09:55:50 +0200

perf/core: Need CAP_SYS_ADMIN to create k/uprobe with perf_event_open()

Non-root user cannot create kprobe or uprobe through the text-based
interface (kprobe_events, uprobe_events),so they should not be able
to create probes via perf_event_open() either.

Reported-by: Vince Weaver <[email protected]>
Signed-off-by: Song Liu <[email protected]>
Cc: Linus Torvalds <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Fixes: 33ea4b24277b ("perf/core: Implement the 'perf_uprobe' PMU")
Fixes: e12f03d7031a ("perf/core: Implement the 'perf_kprobe' PMU")
Link: http://lkml.kernel.org/r/[email protected]
Signed-off-by: Ingo Molnar <[email protected]>
---
kernel/events/core.c | 8 ++++++++
1 file changed, 8 insertions(+)

diff --git a/kernel/events/core.c b/kernel/events/core.c
index d7af82827373..2d5fe26551f8 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -8400,6 +8400,10 @@ static int perf_kprobe_event_init(struct perf_event *event)

if (event->attr.type != perf_kprobe.type)
return -ENOENT;
+
+ if (!capable(CAP_SYS_ADMIN))
+ return -EACCES;
+
/*
* no branch sampling for probe events
*/
@@ -8437,6 +8441,10 @@ static int perf_uprobe_event_init(struct perf_event *event)

if (event->attr.type != perf_uprobe.type)
return -ENOENT;
+
+ if (!capable(CAP_SYS_ADMIN))
+ return -EACCES;
+
/*
* no branch sampling for probe events
*/