This patch set has been posted multiple times (with and without
corresponding 'perf tool' patches), and doesn't receive further
comment. I think it should be okay to merge them into mainline.
There are many perf's improvement depend on it. However, Peter
is not responsive after I fixed some problems he pointed out.
Introduces 'write_backward' into perf_event_attr, allows kernel
writing the ring buffer from the end of it. This feature allows
extracting data from overwritable ring buffer.
Wang Nan (5):
perf core: Introduce new ioctl options to pause and resume ring buffer
perf core: Set event's default overflow_handler
perf core: Prepare writing into ring buffer from end
perf core: Add backward attribute to perf event
perf core: Reduce perf event output overhead by new overflow handler
include/linux/perf_event.h | 22 +++++++++++--
include/uapi/linux/perf_event.h | 4 ++-
kernel/events/core.c | 73 +++++++++++++++++++++++++++++++++++------
kernel/events/internal.h | 11 +++++++
kernel/events/ring_buffer.c | 63 +++++++++++++++++++++++++++++++----
5 files changed, 153 insertions(+), 20 deletions(-)
Cc: He Kuang <[email protected]>
Cc: Alexei Starovoitov <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: Brendan Gregg <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Jiri Olsa <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Namhyung Kim <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Zefan Li <[email protected]>
Cc: [email protected]
--
1.8.3.4
Set a default event->overflow_handler in perf_event_alloc() so don't
need checking event->overflow_handler in __perf_event_overflow().
Following commits can give a different default overflow_handler.
No extra performance introduced into hot path because in the original
code we still need reading this handler from memory. A conditional branch
is avoided so actually we remove some instructions.
Initial idea comes from Peter at [1].
[1] http://lkml.kernel.org/r/[email protected]
Signed-off-by: Wang Nan <[email protected]>
Cc: He Kuang <[email protected]>
Cc: Alexei Starovoitov <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: Brendan Gregg <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Jiri Olsa <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Namhyung Kim <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Zefan Li <[email protected]>
Cc: [email protected]
---
kernel/events/core.c | 14 ++++++++------
1 file changed, 8 insertions(+), 6 deletions(-)
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 088c7fe..ed94c91 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -6467,10 +6467,7 @@ static int __perf_event_overflow(struct perf_event *event,
irq_work_queue(&event->pending);
}
- if (event->overflow_handler)
- event->overflow_handler(event, data, regs);
- else
- perf_event_output(event, data, regs);
+ event->overflow_handler(event, data, regs);
if (*perf_event_fasync(event) && event->pending_kill) {
event->pending_wakeup = 1;
@@ -7963,8 +7960,13 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,
context = parent_event->overflow_handler_context;
}
- event->overflow_handler = overflow_handler;
- event->overflow_handler_context = context;
+ if (overflow_handler) {
+ event->overflow_handler = overflow_handler;
+ event->overflow_handler_context = context;
+ } else {
+ event->overflow_handler = perf_event_output;
+ event->overflow_handler_context = NULL;
+ }
perf_event__state_init(event);
--
1.8.3.4
By creating onward and backward specific overflow handlers and setting
them according to event's backward setting, normal sampling events
don't need checking backward setting of an event any more.
This is the last patch of backward writing patchset. After this patch,
there's no extra overhead introduced to the fast path of sampling
output.
Signed-off-by: Wang Nan <[email protected]>
Cc: He Kuang <[email protected]>
Cc: Alexei Starovoitov <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: Brendan Gregg <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Jiri Olsa <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Namhyung Kim <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Zefan Li <[email protected]>
Cc: [email protected]
---
include/linux/perf_event.h | 17 +++++++++++++++--
kernel/events/core.c | 41 ++++++++++++++++++++++++++++++++++++-----
kernel/events/ring_buffer.c | 12 ++++++++++++
3 files changed, 63 insertions(+), 7 deletions(-)
diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index 88e2b47..bbe913f 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -829,9 +829,15 @@ extern int perf_event_overflow(struct perf_event *event,
struct perf_sample_data *data,
struct pt_regs *regs);
+extern void perf_event_output_onward(struct perf_event *event,
+ struct perf_sample_data *data,
+ struct pt_regs *regs);
+extern void perf_event_output_backward(struct perf_event *event,
+ struct perf_sample_data *data,
+ struct pt_regs *regs);
extern void perf_event_output(struct perf_event *event,
- struct perf_sample_data *data,
- struct pt_regs *regs);
+ struct perf_sample_data *data,
+ struct pt_regs *regs);
extern void
perf_event_header__init_id(struct perf_event_header *header,
@@ -1038,6 +1044,13 @@ static inline bool is_write_backward(struct perf_event *event)
extern int perf_output_begin(struct perf_output_handle *handle,
struct perf_event *event, unsigned int size);
+extern int perf_output_begin_onward(struct perf_output_handle *handle,
+ struct perf_event *event,
+ unsigned int size);
+extern int perf_output_begin_backward(struct perf_output_handle *handle,
+ struct perf_event *event,
+ unsigned int size);
+
extern void perf_output_end(struct perf_output_handle *handle);
extern unsigned int perf_output_copy(struct perf_output_handle *handle,
const void *buf, unsigned int len);
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 513b556..969c43e 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -5606,9 +5606,13 @@ void perf_prepare_sample(struct perf_event_header *header,
}
}
-void perf_event_output(struct perf_event *event,
- struct perf_sample_data *data,
- struct pt_regs *regs)
+static void __always_inline
+__perf_event_output(struct perf_event *event,
+ struct perf_sample_data *data,
+ struct pt_regs *regs,
+ int (*output_begin)(struct perf_output_handle *,
+ struct perf_event *,
+ unsigned int))
{
struct perf_output_handle handle;
struct perf_event_header header;
@@ -5618,7 +5622,7 @@ void perf_event_output(struct perf_event *event,
perf_prepare_sample(&header, data, event, regs);
- if (perf_output_begin(&handle, event, header.size))
+ if (output_begin(&handle, event, header.size))
goto exit;
perf_output_sample(&handle, &header, data, event);
@@ -5629,6 +5633,30 @@ exit:
rcu_read_unlock();
}
+void
+perf_event_output_onward(struct perf_event *event,
+ struct perf_sample_data *data,
+ struct pt_regs *regs)
+{
+ __perf_event_output(event, data, regs, perf_output_begin_onward);
+}
+
+void
+perf_event_output_backward(struct perf_event *event,
+ struct perf_sample_data *data,
+ struct pt_regs *regs)
+{
+ __perf_event_output(event, data, regs, perf_output_begin_backward);
+}
+
+void
+perf_event_output(struct perf_event *event,
+ struct perf_sample_data *data,
+ struct pt_regs *regs)
+{
+ __perf_event_output(event, data, regs, perf_output_begin);
+}
+
/*
* read event_id
*/
@@ -7963,8 +7991,11 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,
if (overflow_handler) {
event->overflow_handler = overflow_handler;
event->overflow_handler_context = context;
+ } else if (is_write_backward(event)){
+ event->overflow_handler = perf_event_output_backward;
+ event->overflow_handler_context = NULL;
} else {
- event->overflow_handler = perf_event_output;
+ event->overflow_handler = perf_event_output_onward;
event->overflow_handler_context = NULL;
}
diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c
index 80b1fa7..7e30e012 100644
--- a/kernel/events/ring_buffer.c
+++ b/kernel/events/ring_buffer.c
@@ -230,6 +230,18 @@ out:
return -ENOSPC;
}
+int perf_output_begin_onward(struct perf_output_handle *handle,
+ struct perf_event *event, unsigned int size)
+{
+ return __perf_output_begin(handle, event, size, false);
+}
+
+int perf_output_begin_backward(struct perf_output_handle *handle,
+ struct perf_event *event, unsigned int size)
+{
+ return __perf_output_begin(handle, event, size, true);
+}
+
int perf_output_begin(struct perf_output_handle *handle,
struct perf_event *event, unsigned int size)
{
--
1.8.3.4
In perf_event_attr a new bit 'write_backward' is appended to indicate
this event should write ring buffer from its end to beginning.
In perf_output_begin(), prepare ring buffer according this bit.
This patch introduces small overhead into perf_output_begin():
an extra memory read and a conditional branch. Further patch can remove
this overhead by using custom output handler.
Signed-off-by: Wang Nan <[email protected]>
Cc: He Kuang <[email protected]>
Cc: Alexei Starovoitov <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: Brendan Gregg <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Jiri Olsa <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Namhyung Kim <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Zefan Li <[email protected]>
Cc: [email protected]
---
include/linux/perf_event.h | 5 +++++
include/uapi/linux/perf_event.h | 3 ++-
kernel/events/core.c | 7 +++++++
kernel/events/ring_buffer.c | 2 ++
4 files changed, 16 insertions(+), 1 deletion(-)
diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index a9d8cab..88e2b47 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -1031,6 +1031,11 @@ static inline bool has_aux(struct perf_event *event)
return event->pmu->setup_aux;
}
+static inline bool is_write_backward(struct perf_event *event)
+{
+ return !!event->attr.write_backward;
+}
+
extern int perf_output_begin(struct perf_output_handle *handle,
struct perf_event *event, unsigned int size);
extern void perf_output_end(struct perf_output_handle *handle);
diff --git a/include/uapi/linux/perf_event.h b/include/uapi/linux/perf_event.h
index a3c1903..43fc8d2 100644
--- a/include/uapi/linux/perf_event.h
+++ b/include/uapi/linux/perf_event.h
@@ -340,7 +340,8 @@ struct perf_event_attr {
comm_exec : 1, /* flag comm events that are due to an exec */
use_clockid : 1, /* use @clockid for time fields */
context_switch : 1, /* context switch data */
- __reserved_1 : 37;
+ write_backward : 1, /* Write ring buffer from end to beginning */
+ __reserved_1 : 36;
union {
__u32 wakeup_events; /* wakeup every n events */
diff --git a/kernel/events/core.c b/kernel/events/core.c
index ed94c91..513b556 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -8196,6 +8196,13 @@ perf_event_set_output(struct perf_event *event, struct perf_event *output_event)
goto out;
/*
+ * Either writing ring buffer from beginning or from end.
+ * Mixing is not allowed.
+ */
+ if (is_write_backward(output_event) != is_write_backward(event))
+ goto out;
+
+ /*
* If both events generate aux data, they must be on the same PMU
*/
if (has_aux(event) && has_aux(output_event) &&
diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c
index 37c11c6..80b1fa7 100644
--- a/kernel/events/ring_buffer.c
+++ b/kernel/events/ring_buffer.c
@@ -233,6 +233,8 @@ out:
int perf_output_begin(struct perf_output_handle *handle,
struct perf_event *event, unsigned int size)
{
+ if (unlikely(is_write_backward(event)))
+ return __perf_output_begin(handle, event, size, true);
return __perf_output_begin(handle, event, size, false);
}
--
1.8.3.4
Convert perf_output_begin to __perf_output_begin and make the later
function able to write records from the end of the ring buffer.
Following commits will utilize the 'backward' flag.
This patch doesn't introduce any extra performance overhead since we
use always_inline.
Signed-off-by: Wang Nan <[email protected]>
Cc: He Kuang <[email protected]>
Cc: Alexei Starovoitov <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: Brendan Gregg <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Jiri Olsa <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Namhyung Kim <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Zefan Li <[email protected]>
Cc: [email protected]
---
kernel/events/ring_buffer.c | 42 ++++++++++++++++++++++++++++++++++++------
1 file changed, 36 insertions(+), 6 deletions(-)
diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c
index 22e1a47..37c11c6 100644
--- a/kernel/events/ring_buffer.c
+++ b/kernel/events/ring_buffer.c
@@ -102,8 +102,21 @@ out:
preempt_enable();
}
-int perf_output_begin(struct perf_output_handle *handle,
- struct perf_event *event, unsigned int size)
+static bool __always_inline
+ring_buffer_has_space(unsigned long head, unsigned long tail,
+ unsigned long data_size, unsigned int size,
+ bool backward)
+{
+ if (!backward)
+ return CIRC_SPACE(head, tail, data_size) >= size;
+ else
+ return CIRC_SPACE(tail, head, data_size) >= size;
+}
+
+static int __always_inline
+__perf_output_begin(struct perf_output_handle *handle,
+ struct perf_event *event, unsigned int size,
+ bool backward)
{
struct ring_buffer *rb;
unsigned long tail, offset, head;
@@ -146,9 +159,12 @@ int perf_output_begin(struct perf_output_handle *handle,
do {
tail = READ_ONCE(rb->user_page->data_tail);
offset = head = local_read(&rb->head);
- if (!rb->overwrite &&
- unlikely(CIRC_SPACE(head, tail, perf_data_size(rb)) < size))
- goto fail;
+ if (!rb->overwrite) {
+ if (unlikely(!ring_buffer_has_space(head, tail,
+ perf_data_size(rb),
+ size, backward)))
+ goto fail;
+ }
/*
* The above forms a control dependency barrier separating the
@@ -162,9 +178,17 @@ int perf_output_begin(struct perf_output_handle *handle,
* See perf_output_put_handle().
*/
- head += size;
+ if (!backward)
+ head += size;
+ else
+ head -= size;
} while (local_cmpxchg(&rb->head, offset, head) != offset);
+ if (backward) {
+ offset = head;
+ head = (u64)(-head);
+ }
+
/*
* We rely on the implied barrier() by local_cmpxchg() to ensure
* none of the data stores below can be lifted up by the compiler.
@@ -206,6 +230,12 @@ out:
return -ENOSPC;
}
+int perf_output_begin(struct perf_output_handle *handle,
+ struct perf_event *event, unsigned int size)
+{
+ return __perf_output_begin(handle, event, size, false);
+}
+
unsigned int perf_output_copy(struct perf_output_handle *handle,
const void *buf, unsigned int len)
{
--
1.8.3.4
Add new ioctl() to pause/resume ring-buffer output.
In some situations we want to read from ring buffer only when we
ensure nothing can write to the ring buffer during reading. Without
this patch we have to turn off all events attached to this ring buffer
to achieve this.
This patch is for supporting overwrite ring buffer. Following
commits will introduce new methods support reading from overwrite ring
buffer. Before reading caller must ensure the ring buffer is frozen, or
the reading is unreliable.
Signed-off-by: Wang Nan <[email protected]>
Cc: He Kuang <[email protected]>
Cc: Alexei Starovoitov <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: Brendan Gregg <[email protected]>
Cc: Jiri Olsa <[email protected]>
Cc: Masami Hiramatsu <[email protected]>
Cc: Namhyung Kim <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Zefan Li <[email protected]>
Cc: [email protected]
---
include/uapi/linux/perf_event.h | 1 +
kernel/events/core.c | 13 +++++++++++++
kernel/events/internal.h | 11 +++++++++++
kernel/events/ring_buffer.c | 7 ++++++-
4 files changed, 31 insertions(+), 1 deletion(-)
diff --git a/include/uapi/linux/perf_event.h b/include/uapi/linux/perf_event.h
index 1afe962..a3c1903 100644
--- a/include/uapi/linux/perf_event.h
+++ b/include/uapi/linux/perf_event.h
@@ -401,6 +401,7 @@ struct perf_event_attr {
#define PERF_EVENT_IOC_SET_FILTER _IOW('$', 6, char *)
#define PERF_EVENT_IOC_ID _IOR('$', 7, __u64 *)
#define PERF_EVENT_IOC_SET_BPF _IOW('$', 8, __u32)
+#define PERF_EVENT_IOC_PAUSE_OUTPUT _IOW('$', 9, __u32)
enum perf_event_ioc_flags {
PERF_IOC_FLAG_GROUP = 1U << 0,
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 5dcc0bd..088c7fe 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -4306,6 +4306,19 @@ static long _perf_ioctl(struct perf_event *event, unsigned int cmd, unsigned lon
case PERF_EVENT_IOC_SET_BPF:
return perf_event_set_bpf_prog(event, arg);
+ case PERF_EVENT_IOC_PAUSE_OUTPUT: {
+ struct ring_buffer *rb;
+
+ rcu_read_lock();
+ rb = rcu_dereference(event->rb);
+ if (!event->rb) {
+ rcu_read_unlock();
+ return -EINVAL;
+ }
+ rb_toggle_paused(rb, !!arg);
+ rcu_read_unlock();
+ return 0;
+ }
default:
return -ENOTTY;
}
diff --git a/kernel/events/internal.h b/kernel/events/internal.h
index 2bbad9c..6a93d1b 100644
--- a/kernel/events/internal.h
+++ b/kernel/events/internal.h
@@ -18,6 +18,7 @@ struct ring_buffer {
#endif
int nr_pages; /* nr of data pages */
int overwrite; /* can overwrite itself */
+ int paused; /* can write into ring buffer */
atomic_t poll; /* POLL_ for wakeups */
@@ -65,6 +66,16 @@ static inline void rb_free_rcu(struct rcu_head *rcu_head)
rb_free(rb);
}
+static inline void
+rb_toggle_paused(struct ring_buffer *rb,
+ bool pause)
+{
+ if (!pause && rb->nr_pages)
+ rb->paused = 0;
+ else
+ rb->paused = 1;
+}
+
extern struct ring_buffer *
rb_alloc(int nr_pages, long watermark, int cpu, int flags);
extern void perf_event_wakeup(struct perf_event *event);
diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c
index 1faad2c..22e1a47 100644
--- a/kernel/events/ring_buffer.c
+++ b/kernel/events/ring_buffer.c
@@ -125,8 +125,11 @@ int perf_output_begin(struct perf_output_handle *handle,
if (unlikely(!rb))
goto out;
- if (unlikely(!rb->nr_pages))
+ if (unlikely(rb->paused)) {
+ if (rb->nr_pages)
+ local_inc(&rb->lost);
goto out;
+ }
handle->rb = rb;
handle->event = event;
@@ -244,6 +247,8 @@ ring_buffer_init(struct ring_buffer *rb, long watermark, int flags)
INIT_LIST_HEAD(&rb->event_list);
spin_lock_init(&rb->event_lock);
init_irq_work(&rb->irq_work, rb_irq_work);
+
+ rb->paused = rb->nr_pages ? 0 : 1;
}
static void ring_buffer_put_async(struct ring_buffer *rb)
--
1.8.3.4
On Mon, Mar 07, 2016 at 03:50:14AM +0000, Wang Nan wrote:
> This patch set has been posted multiple times (with and without
> corresponding 'perf tool' patches), and doesn't receive further
> comment. I think it should be okay to merge them into mainline.
> There are many perf's improvement depend on it. However, Peter
> is not responsive after I fixed some problems he pointed out.
>
> Introduces 'write_backward' into perf_event_attr, allows kernel
> writing the ring buffer from the end of it. This feature allows
> extracting data from overwritable ring buffer.
>
> Wang Nan (5):
> perf core: Introduce new ioctl options to pause and resume ring buffer
> perf core: Set event's default overflow_handler
> perf core: Prepare writing into ring buffer from end
> perf core: Add backward attribute to perf event
> perf core: Reduce perf event output overhead by new overflow handler
perf kernel features are currently on hold until I can manage to run a
fuzzer for more than a few minutes without my machine having a seizure.
* Peter Zijlstra <[email protected]> wrote:
> On Mon, Mar 07, 2016 at 03:50:14AM +0000, Wang Nan wrote:
> > This patch set has been posted multiple times (with and without
> > corresponding 'perf tool' patches), and doesn't receive further
> > comment. I think it should be okay to merge them into mainline.
> > There are many perf's improvement depend on it. However, Peter
> > is not responsive after I fixed some problems he pointed out.
> >
> > Introduces 'write_backward' into perf_event_attr, allows kernel
> > writing the ring buffer from the end of it. This feature allows
> > extracting data from overwritable ring buffer.
> >
> > Wang Nan (5):
> > perf core: Introduce new ioctl options to pause and resume ring buffer
> > perf core: Set event's default overflow_handler
> > perf core: Prepare writing into ring buffer from end
> > perf core: Add backward attribute to perf event
> > perf core: Reduce perf event output overhead by new overflow handler
>
> perf kernel features are currently on hold until I can manage to run a
> fuzzer for more than a few minutes without my machine having a seizure.
Btw., could you describe exactly what commands you are running, with what
configuration options (if that matters), so that people who'd like our feature
freeze to be lifted can help out?
Thanks,
Ingo
On Tue, Mar 08, 2016 at 02:49:01PM +0100, Ingo Molnar wrote:
>
> * Peter Zijlstra <[email protected]> wrote:
>
> > On Mon, Mar 07, 2016 at 03:50:14AM +0000, Wang Nan wrote:
> > > This patch set has been posted multiple times (with and without
> > > corresponding 'perf tool' patches), and doesn't receive further
> > > comment. I think it should be okay to merge them into mainline.
> > > There are many perf's improvement depend on it. However, Peter
> > > is not responsive after I fixed some problems he pointed out.
> > >
> > > Introduces 'write_backward' into perf_event_attr, allows kernel
> > > writing the ring buffer from the end of it. This feature allows
> > > extracting data from overwritable ring buffer.
> > >
> > > Wang Nan (5):
> > > perf core: Introduce new ioctl options to pause and resume ring buffer
> > > perf core: Set event's default overflow_handler
> > > perf core: Prepare writing into ring buffer from end
> > > perf core: Add backward attribute to perf event
> > > perf core: Reduce perf event output overhead by new overflow handler
> >
> > perf kernel features are currently on hold until I can manage to run a
> > fuzzer for more than a few minutes without my machine having a seizure.
>
> Btw., could you describe exactly what commands you are running, with what
> configuration options (if that matters), so that people who'd like our feature
> freeze to be lifted can help out?
Mostly syz-kaller, but also Vince's perf-fuzzer and your perf-stress
script, which I'm not sure is publicly available.
perf_fuzzer lives at:
https://github.com/deater/perf_event_tests.git
Here's a thread on syz-kaller:
lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com
If things have shifted again I'm sure Dmitry is willing to help.
I run the thing natively on actual real hardware, which ensure I get to
test the PMU drivers too.
# cat go-fuzz.sh
#!/bin/bash
echo 1 > /proc/sys/kernel/traceoff_on_warning
echo 1 > /debug/tracing/options/stacktrace
echo 1 > /debug/tracing/events/sched/enable
cd gopath/src/github.com/google/syzkaller/
./bin/syz-manager -config perf.cfg
# cat gopath/src/github.com/google/syzkaller/perf.cfg
{
"http": "localhost:50000",
"workdir": "/root/gopath/src/github.com/google/syzkaller/workdir",
"syzkaller": "/root/gopath/src/github.com/google/syzkaller",
"vmlinux": "-",
"type": "local",
"count": 1,
"procs": 160,
"cover": false,
"dropprivs": false,
"enable_syscalls": [
"getpid",
"gettid",
"perf_event_open",
"ioctl$PERF*",
"prctl$void",
"bpf$*",
"sched_yield"
]
}
* Peter Zijlstra <[email protected]> wrote:
> Here's a thread on syz-kaller:
>
> lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com
>
> If things have shifted again I'm sure Dmitry is willing to help.
So I tried to install 'go' but it's a _really_ unintuitive tool I have to say.
I installed golang-go on Ubuntu, which gave me a 'go' command:
triton:~> go version
go version go1.5.1 linux/amd64
that was the only step that worked. It's a dead end from that point on:
triton:~> go get github.com/google/syzkaller
package github.com/google/syzkaller: cannot download, $GOPATH not set. For more details see: go help gopath
ok, so lets set GOPATH to the distro src directory:
triton:~> export GOPATH=/usr/lib/go/src/
triton:~> go get github.com/google/syzkaller
package github.com/google/syzkaller: mkdir /usr/lib/go/src/src: permission denied
btw., this is what the directory contains:
triton:~> ls /usr/lib/go/src/
all.bash bootstrap.bash bytes compress debug flag html io make.bat nacltest.bash race.bash run.bash strconv testing unsafe
all.bat bufio clean.bash container encoding fmt image iostest.bash Make.dist net race.bat run.bat strings text
androidtest.bash buildall.bash clean.bat crypto errors go index log math os reflect runtime sync time
archive builtin cmd database expvar hash internal make.bash mime path regexp sort syscall unicode
so, according to the error message it wants a writable directory. Lets try it that
way:
triton:~> mkdir go
triton:~>
triton:~> export GOPATH=/home/mingo/go/
triton:~> go get github.com/google/syzkaller
can't load package: package github.com/google/syzkaller: no buildable Go source files in /home/mingo/go/src/github.com/google/syzkaller
looks like someone wants 'Go' to be used as little as possible! ;-)
Thanks,
Ingo
On Tue, Mar 8, 2016 at 4:29 PM, Ingo Molnar <[email protected]> wrote:
>
> * Peter Zijlstra <[email protected]> wrote:
>
>> Here's a thread on syz-kaller:
>>
>> lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com
>>
>> If things have shifted again I'm sure Dmitry is willing to help.
>
> So I tried to install 'go' but it's a _really_ unintuitive tool I have to say.
>
> I installed golang-go on Ubuntu, which gave me a 'go' command:
>
> triton:~> go version
> go version go1.5.1 linux/amd64
>
> that was the only step that worked. It's a dead end from that point on:
>
> triton:~> go get github.com/google/syzkaller
> package github.com/google/syzkaller: cannot download, $GOPATH not set. For more details see: go help gopath
>
> ok, so lets set GOPATH to the distro src directory:
>
> triton:~> export GOPATH=/usr/lib/go/src/
> triton:~> go get github.com/google/syzkaller
> package github.com/google/syzkaller: mkdir /usr/lib/go/src/src: permission denied
>
> btw., this is what the directory contains:
>
> triton:~> ls /usr/lib/go/src/
> all.bash bootstrap.bash bytes compress debug flag html io make.bat nacltest.bash race.bash run.bash strconv testing unsafe
> all.bat bufio clean.bash container encoding fmt image iostest.bash Make.dist net race.bat run.bat strings text
> androidtest.bash buildall.bash clean.bat crypto errors go index log math os reflect runtime sync time
> archive builtin cmd database expvar hash internal make.bash mime path regexp sort syscall unicode
>
> so, according to the error message it wants a writable directory. Lets try it that
> way:
>
> triton:~> mkdir go
> triton:~>
> triton:~> export GOPATH=/home/mingo/go/
> triton:~> go get github.com/google/syzkaller
> can't load package: package github.com/google/syzkaller: no buildable Go source files in /home/mingo/go/src/github.com/google/syzkaller
Yes, GOPATH needs to be set a writable dir.
You can ignore "can't load package" error. The goal of that step is
checkout syzkaller with all dependencies into correct dirs under
GOPATH. That's already done by now.
Or you can do (/... at the end):
$ go get github.com/google/syzkaller/...
That will checkout and build.
Either way you can continue with the make step.
> looks like someone wants 'Go' to be used as little as possible! ;-)
You probably did not observe recently a noobie trying to build a C
project with sufficiently-complicate-build-system and a bunch of
dependencies that needs to be on specific, unknown revisions, and that
still does not compile with you compiler and does not link with your
linker, and then of course target machine has a wrong glibc version :)
* Dmitry Vyukov <[email protected]> wrote:
> > so, according to the error message it wants a writable directory. Lets try it that
> > way:
> >
> > triton:~> mkdir go
> > triton:~>
> > triton:~> export GOPATH=/home/mingo/go/
> > triton:~> go get github.com/google/syzkaller
> > can't load package: package github.com/google/syzkaller: no buildable Go source files in /home/mingo/go/src/github.com/google/syzkaller
>
> Yes, GOPATH needs to be set a writable dir.
> You can ignore "can't load package" error. The goal of that step is
> checkout syzkaller with all dependencies into correct dirs under
> GOPATH. That's already done by now.
> Or you can do (/... at the end):
> $ go get github.com/google/syzkaller/...
> That will checkout and build.
> Either way you can continue with the make step.
Cool, the '/...' trick works.
> > looks like someone wants 'Go' to be used as little as possible! ;-)
>
> You probably did not observe recently a noobie trying to build a C
> project with sufficiently-complicate-build-system and a bunch of
> dependencies that needs to be on specific, unknown revisions, and that
> still does not compile with you compiler and does not link with your
> linker, and then of course target machine has a wrong glibc version :)
Nah, old Linux tools very much suck, we know that and we suffer from it.
But new tools should not suck! :-)
So, going from the description at:
http://lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com
I now have it built, and created $GOPATH/src/github.com/google/syzkaller/perf.cfg
with:
triton:~/go/src/github.com/google/syzkaller> cat perf.cfg
{
"http": "localhost:50000",
"workdir": "/home/mingo/go/src/github.com/google/syzkaller/workdir",
"syzkaller": "/home/mingo/go/src/github.com/google/syzkaller",
"vmlinux": "-",
"type": "local",
"count": 1,
"procs": 16,
"nocover": true,
"nodropprivs": true,
"enable_syscalls": [
"getpid",
"perf_event_open",
"ioctl$PERF*",
"prctl$void",
"bpf$*",
"sched_yield"
]
}
Then I tried to do:
$ bin/syz-manager -config perf.cfg
in ~/go/src/github.com/google/syzkaller, but that doesn't work because there's no
'bin' directory:
triton:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
bash: bin/syz-manager: No such file or directory
So that should really read something like:
cd ~/go
bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
next roadblock:
fomalhaut:~/go> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
2016/03/08 16:53:44 bad config syzkaller param: can't find bin/syz-fuzzer
so how do I proceed from here?
Thanks,
Ingo
On Tue, Mar 8, 2016 at 4:54 PM, Ingo Molnar <[email protected]> wrote:
>
> * Dmitry Vyukov <[email protected]> wrote:
>
>> > so, according to the error message it wants a writable directory. Lets try it that
>> > way:
>> >
>> > triton:~> mkdir go
>> > triton:~>
>> > triton:~> export GOPATH=/home/mingo/go/
>> > triton:~> go get github.com/google/syzkaller
>> > can't load package: package github.com/google/syzkaller: no buildable Go source files in /home/mingo/go/src/github.com/google/syzkaller
>>
>> Yes, GOPATH needs to be set a writable dir.
>> You can ignore "can't load package" error. The goal of that step is
>> checkout syzkaller with all dependencies into correct dirs under
>> GOPATH. That's already done by now.
>> Or you can do (/... at the end):
>> $ go get github.com/google/syzkaller/...
>> That will checkout and build.
>> Either way you can continue with the make step.
>
> Cool, the '/...' trick works.
>
>
>> > looks like someone wants 'Go' to be used as little as possible! ;-)
>>
>> You probably did not observe recently a noobie trying to build a C
>> project with sufficiently-complicate-build-system and a bunch of
>> dependencies that needs to be on specific, unknown revisions, and that
>> still does not compile with you compiler and does not link with your
>> linker, and then of course target machine has a wrong glibc version :)
>
> Nah, old Linux tools very much suck, we know that and we suffer from it.
>
> But new tools should not suck! :-)
>
> So, going from the description at:
>
> http://lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com
>
> I now have it built, and created $GOPATH/src/github.com/google/syzkaller/perf.cfg
> with:
>
>
> triton:~/go/src/github.com/google/syzkaller> cat perf.cfg
> {
> "http": "localhost:50000",
> "workdir": "/home/mingo/go/src/github.com/google/syzkaller/workdir",
> "syzkaller": "/home/mingo/go/src/github.com/google/syzkaller",
> "vmlinux": "-",
> "type": "local",
> "count": 1,
> "procs": 16,
> "nocover": true,
> "nodropprivs": true,
> "enable_syscalls": [
> "getpid",
> "perf_event_open",
> "ioctl$PERF*",
> "prctl$void",
> "bpf$*",
> "sched_yield"
> ]
> }
>
> Then I tried to do:
>
> $ bin/syz-manager -config perf.cfg
>
> in ~/go/src/github.com/google/syzkaller, but that doesn't work because there's no
> 'bin' directory:
>
> triton:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
> bash: bin/syz-manager: No such file or directory
>
> So that should really read something like:
>
> cd ~/go
> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
>
> next roadblock:
>
> fomalhaut:~/go> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> 2016/03/08 16:53:44 bad config syzkaller param: can't find bin/syz-fuzzer
>
> so how do I proceed from here?
You need to call make in syzkaller dir, it will create
syzkaller/bin/syz-manager. I.e.
$ cd /home/mingo/go/src/github.com/google/syzkaller
$ make
$ bin/syz-manager -config perf.cfg
* Dmitry Vyukov <[email protected]> wrote:
> On Tue, Mar 8, 2016 at 4:54 PM, Ingo Molnar <[email protected]> wrote:
> >
> > * Dmitry Vyukov <[email protected]> wrote:
> >
> >> > so, according to the error message it wants a writable directory. Lets try it that
> >> > way:
> >> >
> >> > triton:~> mkdir go
> >> > triton:~>
> >> > triton:~> export GOPATH=/home/mingo/go/
> >> > triton:~> go get github.com/google/syzkaller
> >> > can't load package: package github.com/google/syzkaller: no buildable Go source files in /home/mingo/go/src/github.com/google/syzkaller
> >>
> >> Yes, GOPATH needs to be set a writable dir.
> >> You can ignore "can't load package" error. The goal of that step is
> >> checkout syzkaller with all dependencies into correct dirs under
> >> GOPATH. That's already done by now.
> >> Or you can do (/... at the end):
> >> $ go get github.com/google/syzkaller/...
> >> That will checkout and build.
> >> Either way you can continue with the make step.
> >
> > Cool, the '/...' trick works.
> >
> >
> >> > looks like someone wants 'Go' to be used as little as possible! ;-)
> >>
> >> You probably did not observe recently a noobie trying to build a C
> >> project with sufficiently-complicate-build-system and a bunch of
> >> dependencies that needs to be on specific, unknown revisions, and that
> >> still does not compile with you compiler and does not link with your
> >> linker, and then of course target machine has a wrong glibc version :)
> >
> > Nah, old Linux tools very much suck, we know that and we suffer from it.
> >
> > But new tools should not suck! :-)
> >
> > So, going from the description at:
> >
> > http://lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com
> >
> > I now have it built, and created $GOPATH/src/github.com/google/syzkaller/perf.cfg
> > with:
> >
> >
> > triton:~/go/src/github.com/google/syzkaller> cat perf.cfg
> > {
> > "http": "localhost:50000",
> > "workdir": "/home/mingo/go/src/github.com/google/syzkaller/workdir",
> > "syzkaller": "/home/mingo/go/src/github.com/google/syzkaller",
> > "vmlinux": "-",
> > "type": "local",
> > "count": 1,
> > "procs": 16,
> > "nocover": true,
> > "nodropprivs": true,
> > "enable_syscalls": [
> > "getpid",
> > "perf_event_open",
> > "ioctl$PERF*",
> > "prctl$void",
> > "bpf$*",
> > "sched_yield"
> > ]
> > }
> >
> > Then I tried to do:
> >
> > $ bin/syz-manager -config perf.cfg
> >
> > in ~/go/src/github.com/google/syzkaller, but that doesn't work because there's no
> > 'bin' directory:
> >
> > triton:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
> > bash: bin/syz-manager: No such file or directory
> >
> > So that should really read something like:
> >
> > cd ~/go
> > bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> >
> > next roadblock:
> >
> > fomalhaut:~/go> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> > 2016/03/08 16:53:44 bad config syzkaller param: can't find bin/syz-fuzzer
> >
> > so how do I proceed from here?
>
>
> You need to call make in syzkaller dir, it will create
> syzkaller/bin/syz-manager. I.e.
>
> $ cd /home/mingo/go/src/github.com/google/syzkaller
> $ make
Ok, cool, this got me further.
> $ bin/syz-manager -config perf.cfg
I now get periodic output of:
fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
2016/03/08 17:24:07 failed to read config file: open src/github.com/google/syzkaller/perf.cfg: no such file or directory
fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
2016/03/08 17:24:19 loading corpus...
2016/03/08 17:24:19 loaded 0 programs
2016/03/08 17:24:19 serving http on http://localhost:50000
2016/03/08 17:24:19 serving rpc on tcp://127.0.0.1:37006
2016/03/08 17:24:34 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454274467286949
2016/03/08 17:24:34 local-0: lost connection: exit status 1
2016/03/08 17:24:34 local-0: saving crash 'lost connection' to crash-local-0-1457454274467603509
2016/03/08 17:24:49 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454289719432704
2016/03/08 17:24:49 local-0: lost connection: exit status 1
2016/03/08 17:24:49 local-0: saving crash 'lost connection' to crash-local-0-1457454289719774031
2016/03/08 17:25:04 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454304992856310
2016/03/08 17:25:04 local-0: lost connection: exit status 1
2016/03/08 17:25:04 local-0: saving crash 'lost connection' to crash-local-0-1457454304993224299
2016/03/08 17:25:20 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454320280216980
2016/03/08 17:25:20 local-0: lost connection: exit status 1
2016/03/08 17:25:20 local-0: saving crash 'lost connection' to crash-local-0-1457454320280581459
2016/03/08 17:25:35 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454335572633035
2016/03/08 17:25:35 local-0: lost connection: exit status 1
2016/03/08 17:25:35 local-0: saving crash 'lost connection' to crash-local-0-1457454335572967343
2016/03/08 17:25:50 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454350865099485
2016/03/08 17:25:50 local-0: lost connection: exit status 1
2016/03/08 17:25:50 local-0: saving crash 'lost connection' to crash-local-0-1457454350865429049
is CONFIG_KCOV=y a must-have feature? There's no KCOV support upstream that I can
see.
Thanks,
Ingo
On Tue, Mar 8, 2016 at 5:27 PM, Ingo Molnar <[email protected]> wrote:
>
> * Dmitry Vyukov <[email protected]> wrote:
>
>> On Tue, Mar 8, 2016 at 4:54 PM, Ingo Molnar <[email protected]> wrote:
>> >
>> > * Dmitry Vyukov <[email protected]> wrote:
>> >
>> >> > so, according to the error message it wants a writable directory. Lets try it that
>> >> > way:
>> >> >
>> >> > triton:~> mkdir go
>> >> > triton:~>
>> >> > triton:~> export GOPATH=/home/mingo/go/
>> >> > triton:~> go get github.com/google/syzkaller
>> >> > can't load package: package github.com/google/syzkaller: no buildable Go source files in /home/mingo/go/src/github.com/google/syzkaller
>> >>
>> >> Yes, GOPATH needs to be set a writable dir.
>> >> You can ignore "can't load package" error. The goal of that step is
>> >> checkout syzkaller with all dependencies into correct dirs under
>> >> GOPATH. That's already done by now.
>> >> Or you can do (/... at the end):
>> >> $ go get github.com/google/syzkaller/...
>> >> That will checkout and build.
>> >> Either way you can continue with the make step.
>> >
>> > Cool, the '/...' trick works.
>> >
>> >
>> >> > looks like someone wants 'Go' to be used as little as possible! ;-)
>> >>
>> >> You probably did not observe recently a noobie trying to build a C
>> >> project with sufficiently-complicate-build-system and a bunch of
>> >> dependencies that needs to be on specific, unknown revisions, and that
>> >> still does not compile with you compiler and does not link with your
>> >> linker, and then of course target machine has a wrong glibc version :)
>> >
>> > Nah, old Linux tools very much suck, we know that and we suffer from it.
>> >
>> > But new tools should not suck! :-)
>> >
>> > So, going from the description at:
>> >
>> > http://lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com
>> >
>> > I now have it built, and created $GOPATH/src/github.com/google/syzkaller/perf.cfg
>> > with:
>> >
>> >
>> > triton:~/go/src/github.com/google/syzkaller> cat perf.cfg
>> > {
>> > "http": "localhost:50000",
>> > "workdir": "/home/mingo/go/src/github.com/google/syzkaller/workdir",
>> > "syzkaller": "/home/mingo/go/src/github.com/google/syzkaller",
>> > "vmlinux": "-",
>> > "type": "local",
>> > "count": 1,
>> > "procs": 16,
>> > "nocover": true,
>> > "nodropprivs": true,
>> > "enable_syscalls": [
>> > "getpid",
>> > "perf_event_open",
>> > "ioctl$PERF*",
>> > "prctl$void",
>> > "bpf$*",
>> > "sched_yield"
>> > ]
>> > }
>> >
>> > Then I tried to do:
>> >
>> > $ bin/syz-manager -config perf.cfg
>> >
>> > in ~/go/src/github.com/google/syzkaller, but that doesn't work because there's no
>> > 'bin' directory:
>> >
>> > triton:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
>> > bash: bin/syz-manager: No such file or directory
>> >
>> > So that should really read something like:
>> >
>> > cd ~/go
>> > bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
>> >
>> > next roadblock:
>> >
>> > fomalhaut:~/go> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
>> > 2016/03/08 16:53:44 bad config syzkaller param: can't find bin/syz-fuzzer
>> >
>> > so how do I proceed from here?
>>
>>
>> You need to call make in syzkaller dir, it will create
>> syzkaller/bin/syz-manager. I.e.
>>
>> $ cd /home/mingo/go/src/github.com/google/syzkaller
>> $ make
>
> Ok, cool, this got me further.
>
>> $ bin/syz-manager -config perf.cfg
>
> I now get periodic output of:
>
> fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> 2016/03/08 17:24:07 failed to read config file: open src/github.com/google/syzkaller/perf.cfg: no such file or directory
> fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
> 2016/03/08 17:24:19 loading corpus...
> 2016/03/08 17:24:19 loaded 0 programs
> 2016/03/08 17:24:19 serving http on http://localhost:50000
> 2016/03/08 17:24:19 serving rpc on tcp://127.0.0.1:37006
> 2016/03/08 17:24:34 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454274467286949
> 2016/03/08 17:24:34 local-0: lost connection: exit status 1
> 2016/03/08 17:24:34 local-0: saving crash 'lost connection' to crash-local-0-1457454274467603509
> 2016/03/08 17:24:49 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454289719432704
> 2016/03/08 17:24:49 local-0: lost connection: exit status 1
> 2016/03/08 17:24:49 local-0: saving crash 'lost connection' to crash-local-0-1457454289719774031
> 2016/03/08 17:25:04 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454304992856310
> 2016/03/08 17:25:04 local-0: lost connection: exit status 1
> 2016/03/08 17:25:04 local-0: saving crash 'lost connection' to crash-local-0-1457454304993224299
> 2016/03/08 17:25:20 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454320280216980
> 2016/03/08 17:25:20 local-0: lost connection: exit status 1
> 2016/03/08 17:25:20 local-0: saving crash 'lost connection' to crash-local-0-1457454320280581459
> 2016/03/08 17:25:35 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454335572633035
> 2016/03/08 17:25:35 local-0: lost connection: exit status 1
> 2016/03/08 17:25:35 local-0: saving crash 'lost connection' to crash-local-0-1457454335572967343
> 2016/03/08 17:25:50 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454350865099485
> 2016/03/08 17:25:50 local-0: lost connection: exit status 1
> 2016/03/08 17:25:50 local-0: saving crash 'lost connection' to crash-local-0-1457454350865429049
>
> is CONFIG_KCOV=y a must-have feature? There's no KCOV support upstream that I can
> see.
Change:
"nocover": true,
to:
"cover": false,
in the config file.
KCOV will increase efficiency of the fuzzer, but it is not necessary.
As far as I understand Peter tested without KCOV.
On Tue, Mar 08, 2016 at 05:27:03PM +0100, Ingo Molnar wrote:
> > > triton:~/go/src/github.com/google/syzkaller> cat perf.cfg
> > > {
> > > "http": "localhost:50000",
> > > "workdir": "/home/mingo/go/src/github.com/google/syzkaller/workdir",
> > > "syzkaller": "/home/mingo/go/src/github.com/google/syzkaller",
> > > "vmlinux": "-",
> > > "type": "local",
> > > "count": 1,
> > > "procs": 16,
> > > "nocover": true,
this^^^
> I now get periodic output of:
>
> fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> 2016/03/08 17:24:07 failed to read config file: open src/github.com/google/syzkaller/perf.cfg: no such file or directory
> fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
> 2016/03/08 17:24:19 loading corpus...
> 2016/03/08 17:24:19 loaded 0 programs
> 2016/03/08 17:24:19 serving http on http://localhost:50000
> 2016/03/08 17:24:19 serving rpc on tcp://127.0.0.1:37006
> 2016/03/08 17:24:34 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454274467286949
> 2016/03/08 17:24:34 local-0: lost connection: exit status 1
> is CONFIG_KCOV=y a must-have feature? There's no KCOV support upstream that I can
> see.
Should have disabled that.
On Tue, Mar 08, 2016 at 05:29:50PM +0100, Dmitry Vyukov wrote:
> KCOV will increase efficiency of the fuzzer, but it is not necessary.
> As far as I understand Peter tested without KCOV.
Correct, I never quite got around to enabling it. Still on the TODO list
somewhere.
* Dmitry Vyukov <[email protected]> wrote:
> On Tue, Mar 8, 2016 at 5:27 PM, Ingo Molnar <[email protected]> wrote:
> >
> > * Dmitry Vyukov <[email protected]> wrote:
> >
> >> On Tue, Mar 8, 2016 at 4:54 PM, Ingo Molnar <[email protected]> wrote:
> >> >
> >> > * Dmitry Vyukov <[email protected]> wrote:
> >> >
> >> >> > so, according to the error message it wants a writable directory. Lets try it that
> >> >> > way:
> >> >> >
> >> >> > triton:~> mkdir go
> >> >> > triton:~>
> >> >> > triton:~> export GOPATH=/home/mingo/go/
> >> >> > triton:~> go get github.com/google/syzkaller
> >> >> > can't load package: package github.com/google/syzkaller: no buildable Go source files in /home/mingo/go/src/github.com/google/syzkaller
> >> >>
> >> >> Yes, GOPATH needs to be set a writable dir.
> >> >> You can ignore "can't load package" error. The goal of that step is
> >> >> checkout syzkaller with all dependencies into correct dirs under
> >> >> GOPATH. That's already done by now.
> >> >> Or you can do (/... at the end):
> >> >> $ go get github.com/google/syzkaller/...
> >> >> That will checkout and build.
> >> >> Either way you can continue with the make step.
> >> >
> >> > Cool, the '/...' trick works.
> >> >
> >> >
> >> >> > looks like someone wants 'Go' to be used as little as possible! ;-)
> >> >>
> >> >> You probably did not observe recently a noobie trying to build a C
> >> >> project with sufficiently-complicate-build-system and a bunch of
> >> >> dependencies that needs to be on specific, unknown revisions, and that
> >> >> still does not compile with you compiler and does not link with your
> >> >> linker, and then of course target machine has a wrong glibc version :)
> >> >
> >> > Nah, old Linux tools very much suck, we know that and we suffer from it.
> >> >
> >> > But new tools should not suck! :-)
> >> >
> >> > So, going from the description at:
> >> >
> >> > http://lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com
> >> >
> >> > I now have it built, and created $GOPATH/src/github.com/google/syzkaller/perf.cfg
> >> > with:
> >> >
> >> >
> >> > triton:~/go/src/github.com/google/syzkaller> cat perf.cfg
> >> > {
> >> > "http": "localhost:50000",
> >> > "workdir": "/home/mingo/go/src/github.com/google/syzkaller/workdir",
> >> > "syzkaller": "/home/mingo/go/src/github.com/google/syzkaller",
> >> > "vmlinux": "-",
> >> > "type": "local",
> >> > "count": 1,
> >> > "procs": 16,
> >> > "nocover": true,
> >> > "nodropprivs": true,
> >> > "enable_syscalls": [
> >> > "getpid",
> >> > "perf_event_open",
> >> > "ioctl$PERF*",
> >> > "prctl$void",
> >> > "bpf$*",
> >> > "sched_yield"
> >> > ]
> >> > }
> >> >
> >> > Then I tried to do:
> >> >
> >> > $ bin/syz-manager -config perf.cfg
> >> >
> >> > in ~/go/src/github.com/google/syzkaller, but that doesn't work because there's no
> >> > 'bin' directory:
> >> >
> >> > triton:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
> >> > bash: bin/syz-manager: No such file or directory
> >> >
> >> > So that should really read something like:
> >> >
> >> > cd ~/go
> >> > bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> >> >
> >> > next roadblock:
> >> >
> >> > fomalhaut:~/go> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> >> > 2016/03/08 16:53:44 bad config syzkaller param: can't find bin/syz-fuzzer
> >> >
> >> > so how do I proceed from here?
> >>
> >>
> >> You need to call make in syzkaller dir, it will create
> >> syzkaller/bin/syz-manager. I.e.
> >>
> >> $ cd /home/mingo/go/src/github.com/google/syzkaller
> >> $ make
> >
> > Ok, cool, this got me further.
> >
> >> $ bin/syz-manager -config perf.cfg
> >
> > I now get periodic output of:
> >
> > fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config src/github.com/google/syzkaller/perf.cfg
> > 2016/03/08 17:24:07 failed to read config file: open src/github.com/google/syzkaller/perf.cfg: no such file or directory
> > fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
> > 2016/03/08 17:24:19 loading corpus...
> > 2016/03/08 17:24:19 loaded 0 programs
> > 2016/03/08 17:24:19 serving http on http://localhost:50000
> > 2016/03/08 17:24:19 serving rpc on tcp://127.0.0.1:37006
> > 2016/03/08 17:24:34 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454274467286949
> > 2016/03/08 17:24:34 local-0: lost connection: exit status 1
> > 2016/03/08 17:24:34 local-0: saving crash 'lost connection' to crash-local-0-1457454274467603509
> > 2016/03/08 17:24:49 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454289719432704
> > 2016/03/08 17:24:49 local-0: lost connection: exit status 1
> > 2016/03/08 17:24:49 local-0: saving crash 'lost connection' to crash-local-0-1457454289719774031
> > 2016/03/08 17:25:04 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454304992856310
> > 2016/03/08 17:25:04 local-0: lost connection: exit status 1
> > 2016/03/08 17:25:04 local-0: saving crash 'lost connection' to crash-local-0-1457454304993224299
> > 2016/03/08 17:25:20 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454320280216980
> > 2016/03/08 17:25:20 local-0: lost connection: exit status 1
> > 2016/03/08 17:25:20 local-0: saving crash 'lost connection' to crash-local-0-1457454320280581459
> > 2016/03/08 17:25:35 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454335572633035
> > 2016/03/08 17:25:35 local-0: lost connection: exit status 1
> > 2016/03/08 17:25:35 local-0: saving crash 'lost connection' to crash-local-0-1457454335572967343
> > 2016/03/08 17:25:50 local-0: saving crash 'BUG: /sys/kernel/debug/kcov is missing (permission denied). Enable CONFIG_KCOV and mount debugfs.' to crash-local-0-1457454350865099485
> > 2016/03/08 17:25:50 local-0: lost connection: exit status 1
> > 2016/03/08 17:25:50 local-0: saving crash 'lost connection' to crash-local-0-1457454350865429049
> >
> > is CONFIG_KCOV=y a must-have feature? There's no KCOV support upstream that I can
> > see.
>
> Change:
> "nocover": true,
> to:
> "cover": false,
> in the config file.
>
> KCOV will increase efficiency of the fuzzer, but it is not necessary.
> As far as I understand Peter tested without KCOV.
Ok, so now I get:
fomalhaut:~/go/src/github.com/google/syzkaller> bin/syz-manager -config perf.cfg
2016/03/08 17:39:25 loading corpus...
2016/03/08 17:39:25 loaded 0 programs
2016/03/08 17:39:25 serving http on http://localhost:50000
2016/03/08 17:39:25 serving rpc on tcp://127.0.0.1:33809
| |-sshd(49552)---sshd(49555)---bash(49562)---syz-manager(49652)-+-syz-fuzzer(49661)-+-syz-executor(49935)
| | | |-syz-executor(49936)
| | | |-syz-executor(49937)
| | | |-syz-executor(49938)
| | | |-syz-executor(49939)
| | | |-syz-executor(49940)
| | | |-syz-executor(49941)
| | | |-syz-executor(49942)
| | | |-syz-executor(49943)
| | | |-syz-executor(49944)
| | | |-syz-executor(49945)
| | | |-syz-executor(49946)
| | | |-syz-executor(49947)
| | | |-syz-executor(49948)
| | | |-syz-executor(49949)
| | | |-syz-executor(49950)
| | | |-{syz-fuzzer}(49662)
| | | |-{syz-fuzzer}(49663)
| | | |-{syz-fuzzer}(49664)
| | | |-{syz-fuzzer}(49665)
| | | |-{syz-fuzzer}(49666)
| | | |-{syz-fuzzer}(49688)
| | | |-{syz-fuzzer}(49689)
| | | |-{syz-fuzzer}(49690)
| | | |-{syz-fuzzer}(49691)
| | | |-{syz-fuzzer}(49692)
| | | |-{syz-fuzzer}(49693)
| | | |-{syz-fuzzer}(49694)
| | | |-{syz-fuzzer}(49695)
| | | |-{syz-fuzzer}(49696)
| | | |-{syz-fuzzer}(49697)
| | | |-{syz-fuzzer}(49698)
| | | |-{syz-fuzzer}(49699)
| | | |-{syz-fuzzer}(49700)
| | | |-{syz-fuzzer}(49701)
| | | |-{syz-fuzzer}(49704)
| | | |-{syz-fuzzer}(49705)
| | | |-{syz-fuzzer}(49706)
| | | |-{syz-fuzzer}(49710)
| | | |-{syz-fuzzer}(49711)
| | | |-{syz-fuzzer}(49737)
| | | |-{syz-fuzzer}(49739)
| | | |-{syz-fuzzer}(49762)
| | | |-{syz-fuzzer}(49764)
| | | |-{syz-fuzzer}(49787)
| | | |-{syz-fuzzer}(49789)
| | | |-{syz-fuzzer}(49802)
| | | |-{syz-fuzzer}(49803)
| | | |-{syz-fuzzer}(49804)
| | | |-{syz-fuzzer}(49818)
| | | |-{syz-fuzzer}(49822)
| | | |-{syz-fuzzer}(49846)
| | | |-{syz-fuzzer}(49847)
| | | |-{syz-fuzzer}(49848)
| | | |-{syz-fuzzer}(49849)
| | | |-{syz-fuzzer}(49850)
| | | |-{syz-fuzzer}(49851)
| | | |-{syz-fuzzer}(49852)
| | | |-{syz-fuzzer}(49893)
| | | |-{syz-fuzzer}(49894)
| | | |-{syz-fuzzer}(49895)
| | | |-{syz-fuzzer}(49896)
| | | |-{syz-fuzzer}(49933)
| | | `-{syz-fuzzer}(49934)
| | |-{syz-manager}(49653)
| | |-{syz-manager}(49654)
| | |-{syz-manager}(49655)
| | |-{syz-manager}(49656)
| | |-{syz-manager}(49657)
| | |-{syz-manager}(49658)
| | |-{syz-manager}(49659)
| | |-{syz-manager}(49667)
| | |-{syz-manager}(49668)
| | |-{syz-manager}(49669)
| | |-{syz-manager}(49670)
| | |-{syz-manager}(49671)
| | |-{syz-manager}(49672)
| | |-{syz-manager}(49673)
| | |-{syz-manager}(49674)
| | |-{syz-manager}(49675)
| | |-{syz-manager}(49676)
| | |-{syz-manager}(49677)
| | |-{syz-manager}(49678)
| | |-{syz-manager}(49679)
| | |-{syz-manager}(49680)
| | |-{syz-manager}(49681)
| | |-{syz-manager}(49682)
| | |-{syz-manager}(49683)
| | |-{syz-manager}(49684)
| | |-{syz-manager}(49685)
| | |-{syz-manager}(49686)
| | |-{syz-manager}(49687)
| | |-{syz-manager}(49913)
| | `-{syz-manager}(49914)
I'm seeing some zombies:
49935 pts/2 Z+ 0:00 [syz-executor] <defunct>
49936 pts/2 Z+ 0:00 [syz-executor] <defunct>
49937 pts/2 Z+ 0:00 [syz-executor] <defunct>
49938 pts/2 Z+ 0:00 [syz-executor] <defunct>
49939 pts/2 Z+ 0:00 [syz-executor] <defunct>
49940 pts/2 Z+ 0:00 [syz-executor] <defunct>
49941 pts/2 Z+ 0:00 [syz-executor] <defunct>
49942 pts/2 Z+ 0:00 [syz-executor] <defunct>
49943 pts/2 Z+ 0:00 [syz-executor] <defunct>
49944 pts/2 Z+ 0:00 [syz-executor] <defunct>
49945 pts/2 Z+ 0:00 [syz-executor] <defunct>
49946 pts/2 Z+ 0:00 [syz-executor] <defunct>
49947 pts/2 Z+ 0:00 [syz-executor] <defunct>
49948 pts/2 Z+ 0:00 [syz-executor] <defunct>
49949 pts/2 Z+ 0:00 [syz-executor] <defunct>
49950 pts/2 Z+ 0:00 [syz-executor] <defun
does it mean it found a kernel bug already?
It only had a couple of seconds of runtime:
49652 mingo 20 0 1434276 52144 11344 S 0.0 0.0 0:00.54 syz-manager
49661 mingo 20 0 2196672 43948 10448 S 0.0 0.0 0:05.59 syz-fuzzer
Thanks,
Ingo
* Ingo Molnar <[email protected]> wrote:
> It only had a couple of seconds of runtime:
>
> 49652 mingo 20 0 1434276 52144 11344 S 0.0 0.0 0:00.54 syz-manager
> 49661 mingo 20 0 2196672 43948 10448 S 0.0 0.0 0:05.59 syz-fuzzer
Ah, so it appears to making some progress:
49652 mingo 20 0 1581740 47600 11344 S 0.0 0.0 0:00.58 syz-manager
49661 mingo 20 0 2204868 43720 10448 S 0.0 0.0 0:07.49 syz-fuzzer
49652 mingo 20 0 1598132 31512 11344 S 0.0 0.0 0:00.61 syz-manager
49661 mingo 20 0 2204868 44252 10448 S 0.0 0.0 0:09.09 syz-fuzzer
but only about +1 second runtime added every minute or so. Is that expected?
There's no progress mark anywhere suggesting that the tool thinks it is going
fine. You might want to emit periodic (once a minute or so) 'I am still OK!'
messages or so.
Thanks,
Ingo
On Tue, Mar 8, 2016 at 5:48 PM, Ingo Molnar <[email protected]> wrote:
>
> * Ingo Molnar <[email protected]> wrote:
>
>> It only had a couple of seconds of runtime:
>>
>> 49652 mingo 20 0 1434276 52144 11344 S 0.0 0.0 0:00.54 syz-manager
>> 49661 mingo 20 0 2196672 43948 10448 S 0.0 0.0 0:05.59 syz-fuzzer
>
> Ah, so it appears to making some progress:
>
> 49652 mingo 20 0 1581740 47600 11344 S 0.0 0.0 0:00.58 syz-manager
> 49661 mingo 20 0 2204868 43720 10448 S 0.0 0.0 0:07.49 syz-fuzzer
>
> 49652 mingo 20 0 1598132 31512 11344 S 0.0 0.0 0:00.61 syz-manager
> 49661 mingo 20 0 2204868 44252 10448 S 0.0 0.0 0:09.09 syz-fuzzer
>
> but only about +1 second runtime added every minute or so. Is that expected?
The main work is done by child syz-executor processes.
syz-manager/syz-fuzzer only guide the process.
You can set "procs" param in config to higher value to increase CPU
utilization. To get more bugs you want to saturate all CPUs to trigger
more unusual thread interleavings.
> There's no progress mark anywhere suggesting that the tool thinks it is going
> fine. You might want to emit periodic (once a minute or so) 'I am still OK!'
> messages or so.
Will do.
Regarding the zombie processes, it may or may not be OK. If they hang
for minutes and you can't kill them, then it is a kernel bug. If they
hang for minutes and you can kill them, then it is either kernel bug
of my bug. If they are recycled eventually, then it is OK. The first
thing I do in such cases is:
$ cat /proc/$PID/task/**/stack
If there is a second unfinished thread hanging on a kernel spinlock or
mutex, then it's definitely bad.
It also helps to enable CONFIG_RCU_STALL_COMMON=y,
CONFIG_DEBUG_ATOMIC_SLEEP=y, CONFIG_WQ_WATCHDOG=y and spinlock/mutex
debugging. These can detect various stalls.
* Dmitry Vyukov <[email protected]> wrote:
> On Tue, Mar 8, 2016 at 5:48 PM, Ingo Molnar <[email protected]> wrote:
> >
> > * Ingo Molnar <[email protected]> wrote:
> >
> >> It only had a couple of seconds of runtime:
> >>
> >> 49652 mingo 20 0 1434276 52144 11344 S 0.0 0.0 0:00.54 syz-manager
> >> 49661 mingo 20 0 2196672 43948 10448 S 0.0 0.0 0:05.59 syz-fuzzer
> >
> > Ah, so it appears to making some progress:
> >
> > 49652 mingo 20 0 1581740 47600 11344 S 0.0 0.0 0:00.58 syz-manager
> > 49661 mingo 20 0 2204868 43720 10448 S 0.0 0.0 0:07.49 syz-fuzzer
> >
> > 49652 mingo 20 0 1598132 31512 11344 S 0.0 0.0 0:00.61 syz-manager
> > 49661 mingo 20 0 2204868 44252 10448 S 0.0 0.0 0:09.09 syz-fuzzer
> >
> > but only about +1 second runtime added every minute or so. Is that expected?
>
> The main work is done by child syz-executor processes.
Hm, they don't seem to be doing anything:
fomalhaut:~> ps aux | grep syz-executor
mingo 41506 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
mingo 41509 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
mingo 41513 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
mingo 41515 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
mingo 41523 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
mingo 41601 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
mingo 41608 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
mingo 41662 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
mingo 41764 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
mingo 41966 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
mingo 42029 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
mingo 42084 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
mingo 42145 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
mingo 42149 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
mingo 42166 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
mingo 42175 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
mingo 57627 2.2 0.0 1860540 44884 pts/2 Sl+ 18:16 0:04 /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-fuzzer -executor /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-executor -name local-0 -manager 127.0.0.1:33809 -output=none -procs 16 -leak=false -cover=false -nobody=true -v 0
... because they are recycling:
fomalhaut:~> ps aux | grep syz-executor
mingo 57627 1.6 0.0 1942468 44624 pts/2 Sl+ 18:16 0:05 /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-fuzzer -executor /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-executor -name local-0 -manager 127.0.0.1:33809 -output=none -procs 16 -leak=false -cover=false -nobody=true -v 0
mingo 98448 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
mingo 98454 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
mingo 98468 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
mingo 98472 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
mingo 98476 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
mingo 98522 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
mingo 98525 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
mingo 98548 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
mingo 98568 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
mingo 98596 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
mingo 98618 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
mingo 98644 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
mingo 98695 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
mingo 98708 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
mingo 98737 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
mingo 98756 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
I'm not seeing anything happen in 'top' - only a mostly idle system.
> syz-manager/syz-fuzzer only guide the process.
> You can set "procs" param in config to higher value to increase CPU
> utilization. To get more bugs you want to saturate all CPUs to trigger
> more unusual thread interleavings.
So right now it doesn't seem to saturate 16 CPUs - not even close to it.
> If there is a second unfinished thread hanging on a kernel spinlock or
> mutex, then it's definitely bad.
> It also helps to enable CONFIG_RCU_STALL_COMMON=y,
> CONFIG_DEBUG_ATOMIC_SLEEP=y, CONFIG_WQ_WATCHDOG=y and spinlock/mutex
> debugging. These can detect various stalls.
I can just Ctrl-C the manager and it shuts down within a few seconds:
2016/03/08 17:39:25 serving rpc on tcp://127.0.0.1:33809
2016/03/08 17:51:45 local-0: lost connection: exit status 2
2016/03/08 17:51:45 local-0: saving crash 'lost connection' to crash-local-0-1457455905403390570
2016/03/08 18:04:04 local-0: lost connection: exit status 2
2016/03/08 18:04:04 local-0: saving crash 'lost connection' to crash-local-0-1457456644779165131
2016/03/08 18:16:24 local-0: lost connection: exit status 2
2016/03/08 18:16:24 local-0: saving crash 'lost connection' to crash-local-0-1457457384707190124
^C2016/03/08 18:22:53 shutting down...
with nothing hanging around:
fomalhaut:~/go/src/github.com/google/syzkaller> ps aux | grep -i syz
mingo 1374 0.0 0.0 118476 2376 pts/2 S+ 18:23 0:00 grep --color=auto -i syz
and with no kernel messages in dmesg - and with a fully functional system.
I'm running the 16-task load on a 120 CPU system - should I increase it to 120?
Does the code expect to saturate the system?
Thanks,
Ingo
On Tue, Mar 8, 2016 at 6:24 PM, Ingo Molnar <[email protected]> wrote:
>
> * Dmitry Vyukov <[email protected]> wrote:
>
>> On Tue, Mar 8, 2016 at 5:48 PM, Ingo Molnar <[email protected]> wrote:
>> >
>> > * Ingo Molnar <[email protected]> wrote:
>> >
>> >> It only had a couple of seconds of runtime:
>> >>
>> >> 49652 mingo 20 0 1434276 52144 11344 S 0.0 0.0 0:00.54 syz-manager
>> >> 49661 mingo 20 0 2196672 43948 10448 S 0.0 0.0 0:05.59 syz-fuzzer
>> >
>> > Ah, so it appears to making some progress:
>> >
>> > 49652 mingo 20 0 1581740 47600 11344 S 0.0 0.0 0:00.58 syz-manager
>> > 49661 mingo 20 0 2204868 43720 10448 S 0.0 0.0 0:07.49 syz-fuzzer
>> >
>> > 49652 mingo 20 0 1598132 31512 11344 S 0.0 0.0 0:00.61 syz-manager
>> > 49661 mingo 20 0 2204868 44252 10448 S 0.0 0.0 0:09.09 syz-fuzzer
>> >
>> > but only about +1 second runtime added every minute or so. Is that expected?
>>
>> The main work is done by child syz-executor processes.
>
> Hm, they don't seem to be doing anything:
>
> fomalhaut:~> ps aux | grep syz-executor
> mingo 41506 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
> mingo 41509 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
> mingo 41513 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
> mingo 41515 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
> mingo 41523 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
> mingo 41601 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
> mingo 41608 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
> mingo 41662 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
> mingo 41764 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
> mingo 41966 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
> mingo 42029 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
> mingo 42084 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
> mingo 42145 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
> mingo 42149 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
> mingo 42166 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
> mingo 42175 0.0 0.0 0 0 pts/2 Z+ 18:19 0:00 [syz-executor] <defunct>
> mingo 57627 2.2 0.0 1860540 44884 pts/2 Sl+ 18:16 0:04 /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-fuzzer -executor /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-executor -name local-0 -manager 127.0.0.1:33809 -output=none -procs 16 -leak=false -cover=false -nobody=true -v 0
>
> ... because they are recycling:
>
> fomalhaut:~> ps aux | grep syz-executor
> mingo 57627 1.6 0.0 1942468 44624 pts/2 Sl+ 18:16 0:05 /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-fuzzer -executor /home/mingo/go/src/github.com/google/syzkaller/workdir/instance-0/syz-executor -name local-0 -manager 127.0.0.1:33809 -output=none -procs 16 -leak=false -cover=false -nobody=true -v 0
> mingo 98448 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
> mingo 98454 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
> mingo 98468 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
> mingo 98472 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
> mingo 98476 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
> mingo 98522 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
> mingo 98525 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
> mingo 98548 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
> mingo 98568 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
> mingo 98596 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
> mingo 98618 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
> mingo 98644 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
> mingo 98695 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
> mingo 98708 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
> mingo 98737 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
> mingo 98756 0.0 0.0 0 0 pts/2 Z+ 18:21 0:00 [syz-executor] <defunct>
>
> I'm not seeing anything happen in 'top' - only a mostly idle system.
>
>> syz-manager/syz-fuzzer only guide the process.
>> You can set "procs" param in config to higher value to increase CPU
>> utilization. To get more bugs you want to saturate all CPUs to trigger
>> more unusual thread interleavings.
>
> So right now it doesn't seem to saturate 16 CPUs - not even close to it.
>
>> If there is a second unfinished thread hanging on a kernel spinlock or
>> mutex, then it's definitely bad.
>> It also helps to enable CONFIG_RCU_STALL_COMMON=y,
>> CONFIG_DEBUG_ATOMIC_SLEEP=y, CONFIG_WQ_WATCHDOG=y and spinlock/mutex
>> debugging. These can detect various stalls.
>
> I can just Ctrl-C the manager and it shuts down within a few seconds:
>
> 2016/03/08 17:39:25 serving rpc on tcp://127.0.0.1:33809
> 2016/03/08 17:51:45 local-0: lost connection: exit status 2
> 2016/03/08 17:51:45 local-0: saving crash 'lost connection' to crash-local-0-1457455905403390570
> 2016/03/08 18:04:04 local-0: lost connection: exit status 2
> 2016/03/08 18:04:04 local-0: saving crash 'lost connection' to crash-local-0-1457456644779165131
> 2016/03/08 18:16:24 local-0: lost connection: exit status 2
> 2016/03/08 18:16:24 local-0: saving crash 'lost connection' to crash-local-0-1457457384707190124
> ^C2016/03/08 18:22:53 shutting down...
>
> with nothing hanging around:
OK, this all looks good.
> fomalhaut:~/go/src/github.com/google/syzkaller> ps aux | grep -i syz
> mingo 1374 0.0 0.0 118476 2376 pts/2 S+ 18:23 0:00 grep --color=auto -i syz
>
> and with no kernel messages in dmesg - and with a fully functional system.
>
> I'm running the 16-task load on a 120 CPU system - should I increase it to 120?
> Does the code expect to saturate the system?
No, it does not expect to saturate the system. Set "procs" to 480, or
something like that.
* Dmitry Vyukov <[email protected]> wrote:
> > fomalhaut:~/go/src/github.com/google/syzkaller> ps aux | grep -i syz
> > mingo 1374 0.0 0.0 118476 2376 pts/2 S+ 18:23 0:00 grep --color=auto -i syz
> >
> > and with no kernel messages in dmesg - and with a fully functional system.
> >
> > I'm running the 16-task load on a 120 CPU system - should I increase it to 120?
> > Does the code expect to saturate the system?
>
> No, it does not expect to saturate the system. Set "procs" to 480, or
> something like that.
Does not seem to help much:
fomalhaut:~> vmstat 10
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 257465904 219940 4736092 0 0 0 102 16022 4396 0 1 99 0 0
2 0 0 257452144 220496 4755052 0 0 2 3649 14286 4627 0 1 99 0 0
2 0 0 257473408 221188 4770824 0 0 15 1898 17175 4474 0 1 99 0 0
Only around 1% system utilization. Should I go for 1,000 or more? :)
Peter, do you experience with running syz-kaller on larger CPU count Intel
systems?
Thanks,
Ingo
On Tue, Mar 8, 2016 at 6:37 PM, Ingo Molnar <[email protected]> wrote:
>
> * Dmitry Vyukov <[email protected]> wrote:
>
>> > fomalhaut:~/go/src/github.com/google/syzkaller> ps aux | grep -i syz
>> > mingo 1374 0.0 0.0 118476 2376 pts/2 S+ 18:23 0:00 grep --color=auto -i syz
>> >
>> > and with no kernel messages in dmesg - and with a fully functional system.
>> >
>> > I'm running the 16-task load on a 120 CPU system - should I increase it to 120?
>> > Does the code expect to saturate the system?
>>
>> No, it does not expect to saturate the system. Set "procs" to 480, or
>> something like that.
>
> Does not seem to help much:
>
> fomalhaut:~> vmstat 10
> procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
> r b swpd free buff cache si so bi bo in cs us sy id wa st
>
> 1 0 0 257465904 219940 4736092 0 0 0 102 16022 4396 0 1 99 0 0
> 2 0 0 257452144 220496 4755052 0 0 2 3649 14286 4627 0 1 99 0 0
> 2 0 0 257473408 221188 4770824 0 0 15 1898 17175 4474 0 1 99 0 0
>
> Only around 1% system utilization. Should I go for 1,000 or more? :)
>
> Peter, do you experience with running syz-kaller on larger CPU count Intel
> systems?
Try to set "dropprivs": false in config.
I've noticed that creation/destruction of namespaces is very slow and
globally serialized. So sometimes it takes tens of seconds for each
worker processes to startup. For perf-related syscalls it should be
"safe" to just run as root. And perf subsystem operation is also
unaffected by namespaces as far as I know, so it should not affect
behavior as well.
* Dmitry Vyukov <[email protected]> wrote:
> On Tue, Mar 8, 2016 at 6:37 PM, Ingo Molnar <[email protected]> wrote:
> >
> > * Dmitry Vyukov <[email protected]> wrote:
> >
> >> > fomalhaut:~/go/src/github.com/google/syzkaller> ps aux | grep -i syz
> >> > mingo 1374 0.0 0.0 118476 2376 pts/2 S+ 18:23 0:00 grep --color=auto -i syz
> >> >
> >> > and with no kernel messages in dmesg - and with a fully functional system.
> >> >
> >> > I'm running the 16-task load on a 120 CPU system - should I increase it to 120?
> >> > Does the code expect to saturate the system?
> >>
> >> No, it does not expect to saturate the system. Set "procs" to 480, or
> >> something like that.
> >
> > Does not seem to help much:
> >
> > fomalhaut:~> vmstat 10
> > procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
> > r b swpd free buff cache si so bi bo in cs us sy id wa st
> >
> > 1 0 0 257465904 219940 4736092 0 0 0 102 16022 4396 0 1 99 0 0
> > 2 0 0 257452144 220496 4755052 0 0 2 3649 14286 4627 0 1 99 0 0
> > 2 0 0 257473408 221188 4770824 0 0 15 1898 17175 4474 0 1 99 0 0
> >
> > Only around 1% system utilization. Should I go for 1,000 or more? :)
> >
> > Peter, do you experience with running syz-kaller on larger CPU count Intel
> > systems?
>
>
> Try to set "dropprivs": false in config.
Things got a lot more lively after that!
But most of the overhead seems to come from systemd trying to dump core or
something like that:
85872 mingo 20 0 34712 3016 2656 S 4.6 0.0 0:00.14 systemd-coredum
85440 mingo 20 0 34712 3028 2664 S 4.2 0.0 0:00.13 systemd-coredum
85751 mingo 20 0 34712 3076 2716 S 4.2 0.0 0:00.13 systemd-coredum
85840 mingo 20 0 34712 2988 2624 S 4.2 0.0 0:00.13 systemd-coredum
85861 mingo 20 0 34712 3080 2720 S 4.2 0.0 0:00.13 systemd-coredum
85954 mingo 20 0 34712 3028 2664 S 4.2 0.0 0:00.13 systemd-coredum
and I have:
fomalhaut:~/go/src/github.com/google/syzkaller> ulimit -c
0
weird ... Has any of you seen such behavior?
Thanks,
Ingo
On Tue, Mar 08, 2016 at 06:37:09PM +0100, Ingo Molnar wrote:
> Peter, do you experience with running syz-kaller on larger CPU count Intel
> systems?
I run it on a 20 core (40 cpu) system mostly, as that tends to not take
ages to come back up again once I wrecked it.
On Tue, Mar 8, 2016 at 6:48 PM, Ingo Molnar <[email protected]> wrote:
>
> * Dmitry Vyukov <[email protected]> wrote:
>
>> On Tue, Mar 8, 2016 at 6:37 PM, Ingo Molnar <[email protected]> wrote:
>> >
>> > * Dmitry Vyukov <[email protected]> wrote:
>> >
>> >> > fomalhaut:~/go/src/github.com/google/syzkaller> ps aux | grep -i syz
>> >> > mingo 1374 0.0 0.0 118476 2376 pts/2 S+ 18:23 0:00 grep --color=auto -i syz
>> >> >
>> >> > and with no kernel messages in dmesg - and with a fully functional system.
>> >> >
>> >> > I'm running the 16-task load on a 120 CPU system - should I increase it to 120?
>> >> > Does the code expect to saturate the system?
>> >>
>> >> No, it does not expect to saturate the system. Set "procs" to 480, or
>> >> something like that.
>> >
>> > Does not seem to help much:
>> >
>> > fomalhaut:~> vmstat 10
>> > procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
>> > r b swpd free buff cache si so bi bo in cs us sy id wa st
>> >
>> > 1 0 0 257465904 219940 4736092 0 0 0 102 16022 4396 0 1 99 0 0
>> > 2 0 0 257452144 220496 4755052 0 0 2 3649 14286 4627 0 1 99 0 0
>> > 2 0 0 257473408 221188 4770824 0 0 15 1898 17175 4474 0 1 99 0 0
>> >
>> > Only around 1% system utilization. Should I go for 1,000 or more? :)
>> >
>> > Peter, do you experience with running syz-kaller on larger CPU count Intel
>> > systems?
>>
>>
>> Try to set "dropprivs": false in config.
>
> Things got a lot more lively after that!
>
> But most of the overhead seems to come from systemd trying to dump core or
> something like that:
>
> 85872 mingo 20 0 34712 3016 2656 S 4.6 0.0 0:00.14 systemd-coredum
> 85440 mingo 20 0 34712 3028 2664 S 4.2 0.0 0:00.13 systemd-coredum
> 85751 mingo 20 0 34712 3076 2716 S 4.2 0.0 0:00.13 systemd-coredum
> 85840 mingo 20 0 34712 2988 2624 S 4.2 0.0 0:00.13 systemd-coredum
> 85861 mingo 20 0 34712 3080 2720 S 4.2 0.0 0:00.13 systemd-coredum
> 85954 mingo 20 0 34712 3028 2664 S 4.2 0.0 0:00.13 systemd-coredum
>
> and I have:
>
> fomalhaut:~/go/src/github.com/google/syzkaller> ulimit -c
> 0
>
> weird ... Has any of you seen such behavior?
I have not seen it.
Probably I need to directly disable core dumps within the syz-executor process.
On Tue, Mar 08, 2016 at 06:48:56PM +0100, Ingo Molnar wrote:
> weird ... Has any of you seen such behavior?
No systemd on any of my machines ;-)
* Ingo Molnar <[email protected]> wrote:
> Things got a lot more lively after that!
>
> But most of the overhead seems to come from systemd trying to dump core or
> something like that:
>
> 85872 mingo 20 0 34712 3016 2656 S 4.6 0.0 0:00.14 systemd-coredum
> 85440 mingo 20 0 34712 3028 2664 S 4.2 0.0 0:00.13 systemd-coredum
> 85751 mingo 20 0 34712 3076 2716 S 4.2 0.0 0:00.13 systemd-coredum
> 85840 mingo 20 0 34712 2988 2624 S 4.2 0.0 0:00.13 systemd-coredum
> 85861 mingo 20 0 34712 3080 2720 S 4.2 0.0 0:00.13 systemd-coredum
> 85954 mingo 20 0 34712 3028 2664 S 4.2 0.0 0:00.13 systemd-coredum
>
> and I have:
>
> fomalhaut:~/go/src/github.com/google/syzkaller> ulimit -c
> 0
>
> weird ... Has any of you seen such behavior?
So the workaround for that is to disable systemd trying to log every core dump to
the system journal (!), via:
echo > /proc/sys/kernel/core_pattern
Thanks,
Ingo
* Ingo Molnar <[email protected]> wrote:
>
> * Ingo Molnar <[email protected]> wrote:
>
> > Things got a lot more lively after that!
> >
> > But most of the overhead seems to come from systemd trying to dump core or
> > something like that:
> >
> > 85872 mingo 20 0 34712 3016 2656 S 4.6 0.0 0:00.14 systemd-coredum
> > 85440 mingo 20 0 34712 3028 2664 S 4.2 0.0 0:00.13 systemd-coredum
> > 85751 mingo 20 0 34712 3076 2716 S 4.2 0.0 0:00.13 systemd-coredum
> > 85840 mingo 20 0 34712 2988 2624 S 4.2 0.0 0:00.13 systemd-coredum
> > 85861 mingo 20 0 34712 3080 2720 S 4.2 0.0 0:00.13 systemd-coredum
> > 85954 mingo 20 0 34712 3028 2664 S 4.2 0.0 0:00.13 systemd-coredum
> >
> > and I have:
> >
> > fomalhaut:~/go/src/github.com/google/syzkaller> ulimit -c
> > 0
> >
> > weird ... Has any of you seen such behavior?
>
> So the workaround for that is to disable systemd trying to log every core dump to
> the system journal (!), via:
>
> echo > /proc/sys/kernel/core_pattern
So with that fixed, I finally started fuzzing for real.
With nproc set to 120 it seems to be chugging along at about 25% system
utilization:
Tasks: 1271 total, 1 running, 1270 sleeping, 0 stopped, 0 zombie
%Cpu(s): 4.5 us, 35.3 sy, 0.0 ni, 55.2 id, 0.5 wa, 4.5 hi, 0.0 si, 0.0 st
KiB Mem : 26401230+total, 25401017+free, 1624640 used, 8377496 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 26143329+avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
87593 mingo 20 0 7850436 162284 11672 S 2941 0.1 33:01.83 syz-fuzzer
923 root 20 0 84772 44344 43840 S 22.8 0.0 1:23.86 systemd-journal
1369 root 16 -4 114636 3256 2832 S 15.7 0.0 0:29.03 auditd
1379 root 12 -8 80236 1764 1432 S 8.3 0.0 0:15.79 audispd
878 root 20 0 0 0 0 S 6.9 0.0 0:16.55 jbd2/sda1-8
1381 root 16 -4 52216 3232 2892 S 3.8 0.0 0:07.08 sedispatch
Even that one is not ideal - obviously there's way too much systemd-journal
overhead, but I'm unable to turn the darn thing off ...
with nproc=480 it does not seem to be working very well - it quickly generates:
2016/03/08 18:59:38 local-0: lost connection: exit status 2
2016/03/08 18:59:38 local-0: saving crash 'lost connection' to crash-local-0-1457459978295423413
and then after some time starts to ramp up again. It's mostly idling around.
Thanks,
Ingo
* Ingo Molnar <[email protected]> wrote:
> With nproc set to 120 it seems to be chugging along at about 25% system
> utilization:
>
> Tasks: 1271 total, 1 running, 1270 sleeping, 0 stopped, 0 zombie
> %Cpu(s): 4.5 us, 35.3 sy, 0.0 ni, 55.2 id, 0.5 wa, 4.5 hi, 0.0 si, 0.0 st
> KiB Mem : 26401230+total, 25401017+free, 1624640 used, 8377496 buff/cache
> KiB Swap: 0 total, 0 free, 0 used. 26143329+avail Mem
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 87593 mingo 20 0 7850436 162284 11672 S 2941 0.1 33:01.83 syz-fuzzer
> 923 root 20 0 84772 44344 43840 S 22.8 0.0 1:23.86 systemd-journal
> 1369 root 16 -4 114636 3256 2832 S 15.7 0.0 0:29.03 auditd
> 1379 root 12 -8 80236 1764 1432 S 8.3 0.0 0:15.79 audispd
> 878 root 20 0 0 0 0 S 6.9 0.0 0:16.55 jbd2/sda1-8
> 1381 root 16 -4 52216 3232 2892 S 3.8 0.0 0:07.08 sedispatch
>
> Even that one is not ideal - obviously there's way too much systemd-journal
> overhead, but I'm unable to turn the darn thing off ...
The journal.conf man page suggests that putting 'Storage=none' into
/etc/systemd/journal.conf disables journalling - but that's not true.
It's apparently impossible to disable systemd logging via any normal means ...
So the workaround for all that is a brutal:
mv /var/log/journal /var/log/journal.dontuse
after that there does not seem to be systemd logging anymore.
... but there is tons of auditd logging now! ;-)
Fortunately that's easily stopped via:
service stop auditd
with that there's no log IO anymore during fuzzing (yay!).
Except that systemd journald rears its ugly head back:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
111464 mingo 20 0 7249024 143348 11476 S 2741 0.1 7:41.31 syz-fuzzer
122632 root 20 0 29200 2768 2524 S 9.5 0.0 0:28.74 systemd-journal
111463 root 20 0 165596 5772 3840 R 4.8 0.0 0:00.80 top
112078 mingo 20 0 20928 704 644 S 2.9 0.0 0:00.17 syz-executor
112516 mingo 20 0 20928 708 644 S 2.9 0.0 0:00.16 syz-executor
878 root 20 0 0 0 0 S 1.9 0.0 0:53.83 jbd2/sda1-8
111711 mingo 20 0 20928 704 644 S 1.9 0.0 0:00.16 syz-executor
111901 mingo 20 0 20928 704 644 S 1.9 0.0 0:00.15 syz-executor
8 root 20 0 0 0 0 S 1.0 0.0 1:00.55 rcu_sched
so it's eating about 10% of system overhead despite doing nothing (!).
The workaround for that systemd bug is a brutal:
[root@fomalhaut ~]# mv /usr/lib/systemd/systemd-journald /usr/lib/systemd/systemd-journald.dontuse
[root@fomalhaut ~]#
Thanks,
Ingo
* Ingo Molnar <[email protected]> wrote:
> so it's eating about 10% of system overhead despite doing nothing (!).
>
> The workaround for that systemd bug is a brutal:
>
> [root@fomalhaut ~]# mv /usr/lib/systemd/systemd-journald /usr/lib/systemd/systemd-journald.dontuse
> [root@fomalhaut ~]#
Except that if I do that, systemd stops working altogether - for example:
[root@fomalhaut ~]# systemctl disable audit
Failed to execute operation: Connection timed out
only reinstating the binary and re-starting journald fixes that.
systemd is the most passive-agressive utility I've ever seen.
So it's not possible to disable journald. Does anyone know any solution for that,
which does not involve reinstalling a whole distro?
Thanks,
Ingo
On Tue, Mar 08, 2016 at 02:57:59PM +0100, Peter Zijlstra wrote:
> On Tue, Mar 08, 2016 at 02:49:01PM +0100, Ingo Molnar wrote:
> >
> > * Peter Zijlstra <[email protected]> wrote:
> >
> > > On Mon, Mar 07, 2016 at 03:50:14AM +0000, Wang Nan wrote:
> > > > This patch set has been posted multiple times (with and without
> > > > corresponding 'perf tool' patches), and doesn't receive further
> > > > comment. I think it should be okay to merge them into mainline.
> > > > There are many perf's improvement depend on it. However, Peter
> > > > is not responsive after I fixed some problems he pointed out.
> > > >
> > > > Introduces 'write_backward' into perf_event_attr, allows kernel
> > > > writing the ring buffer from the end of it. This feature allows
> > > > extracting data from overwritable ring buffer.
> > > >
> > > > Wang Nan (5):
> > > > perf core: Introduce new ioctl options to pause and resume ring buffer
> > > > perf core: Set event's default overflow_handler
> > > > perf core: Prepare writing into ring buffer from end
> > > > perf core: Add backward attribute to perf event
> > > > perf core: Reduce perf event output overhead by new overflow handler
> > >
> > > perf kernel features are currently on hold until I can manage to run a
> > > fuzzer for more than a few minutes without my machine having a seizure.
> >
> > Btw., could you describe exactly what commands you are running, with what
> > configuration options (if that matters), so that people who'd like our feature
> > freeze to be lifted can help out?
>
> Mostly syz-kaller, but also Vince's perf-fuzzer and your perf-stress
> script, which I'm not sure is publicly available.
>
> perf_fuzzer lives at:
>
> https://github.com/deater/perf_event_tests.git
>
> Here's a thread on syz-kaller:
>
> lkml.kernel.org/r/CACT4Y+Ym0TZLkmRrM0ZGgLpu8kqS-YjoWTMrvaLz=tx2tnyO3w@mail.gmail.com
>
> If things have shifted again I'm sure Dmitry is willing to help.
>
> I run the thing natively on actual real hardware, which ensure I get to
> test the PMU drivers too.
>
> # cat go-fuzz.sh
> #!/bin/bash
>
> echo 1 > /proc/sys/kernel/traceoff_on_warning
> echo 1 > /debug/tracing/options/stacktrace
> echo 1 > /debug/tracing/events/sched/enable
are you running this under root?
jirka
> cd gopath/src/github.com/google/syzkaller/
> ./bin/syz-manager -config perf.cfg
>
> # cat gopath/src/github.com/google/syzkaller/perf.cfg
>
> {
> "http": "localhost:50000",
> "workdir": "/root/gopath/src/github.com/google/syzkaller/workdir",
> "syzkaller": "/root/gopath/src/github.com/google/syzkaller",
> "vmlinux": "-",
> "type": "local",
> "count": 1,
> "procs": 160,
> "cover": false,
> "dropprivs": false,
> "enable_syscalls": [
> "getpid",
> "gettid",
> "perf_event_open",
> "ioctl$PERF*",
> "prctl$void",
> "bpf$*",
> "sched_yield"
> ]
> }
On Tue, Mar 08, 2016 at 08:56:42PM +0100, Jiri Olsa wrote:
> > # cat go-fuzz.sh
> > #!/bin/bash
> >
> > echo 1 > /proc/sys/kernel/traceoff_on_warning
> > echo 1 > /debug/tracing/options/stacktrace
> > echo 1 > /debug/tracing/events/sched/enable
>
> are you running this under root?
Of course ;-) I'm not even sure these machines have a user account.
Also, what's the point of fuzzing if half your 'fun' options return
-EPERM.
On Tue, Mar 08, 2016 at 09:07:46PM +0100, Peter Zijlstra wrote:
> On Tue, Mar 08, 2016 at 08:56:42PM +0100, Jiri Olsa wrote:
> > > # cat go-fuzz.sh
> > > #!/bin/bash
> > >
> > > echo 1 > /proc/sys/kernel/traceoff_on_warning
> > > echo 1 > /debug/tracing/options/stacktrace
> > > echo 1 > /debug/tracing/events/sched/enable
> >
> > are you running this under root?
>
> Of course ;-) I'm not even sure these machines have a user account.
>
> Also, what's the point of fuzzing if half your 'fun' options return
> -EPERM.
IIRC the perf fuzzer (one from Vince) could screw your machine having
it run under root
jirka
On Tue, Mar 08, 2016 at 09:44:58PM +0100, Jiri Olsa wrote:
> On Tue, Mar 08, 2016 at 09:07:46PM +0100, Peter Zijlstra wrote:
> > On Tue, Mar 08, 2016 at 08:56:42PM +0100, Jiri Olsa wrote:
> > > > # cat go-fuzz.sh
> > > > #!/bin/bash
> > > >
> > > > echo 1 > /proc/sys/kernel/traceoff_on_warning
> > > > echo 1 > /debug/tracing/options/stacktrace
> > > > echo 1 > /debug/tracing/events/sched/enable
> > >
> > > are you running this under root?
> >
> > Of course ;-) I'm not even sure these machines have a user account.
> >
> > Also, what's the point of fuzzing if half your 'fun' options return
> > -EPERM.
>
> IIRC the perf fuzzer (one from Vince) could screw your machine having
> it run under root
Never noticed :-) Guess how I usually run it..
On Tue, Mar 08, 2016 at 04:54:23PM +0100, Ingo Molnar wrote:
> triton:~/go/src/github.com/google/syzkaller> cat perf.cfg
> {
> "http": "localhost:50000",
> "workdir": "/home/mingo/go/src/github.com/google/syzkaller/workdir",
> "syzkaller": "/home/mingo/go/src/github.com/google/syzkaller",
> "vmlinux": "-",
> "type": "local",
> "count": 1,
> "procs": 16,
> "nocover": true,
> "nodropprivs": true,
> "enable_syscalls": [
> "getpid",
> "perf_event_open",
Btw, is there a way to specify range of arguments to feed into
perf_event_open? Like, limit @attr_uptr to single or multiple event IDs
or so, for example?
--
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
On Wed, Mar 9, 2016 at 11:53 AM, Borislav Petkov <[email protected]> wrote:
> On Tue, Mar 08, 2016 at 04:54:23PM +0100, Ingo Molnar wrote:
>> triton:~/go/src/github.com/google/syzkaller> cat perf.cfg
>> {
>> "http": "localhost:50000",
>> "workdir": "/home/mingo/go/src/github.com/google/syzkaller/workdir",
>> "syzkaller": "/home/mingo/go/src/github.com/google/syzkaller",
>> "vmlinux": "-",
>> "type": "local",
>> "count": 1,
>> "procs": 16,
>> "nocover": true,
>> "nodropprivs": true,
>> "enable_syscalls": [
>> "getpid",
>> "perf_event_open",
>
> Btw, is there a way to specify range of arguments to feed into
> perf_event_open? Like, limit @attr_uptr to single or multiple event IDs
> or so, for example?
No, there is no _that_ level of granularity in the config.
If you really-really want that, then you can alter description of the
perf_event_open syscall in sys/perf.txt file:
https://github.com/google/syzkaller/blob/master/sys/perf.txt
to be more restrictive. For example, you can limit set of values
passed in a particular argument, or even set some args/fields to const
values.
Then you will need to do:
$ make generate LINUX=/path/to/fresh/linux/checkout
$ make
But I would suggest to not do that. That perf config already limits
set of syscalls to a very small set. The fuzzer should be able to
examine all interesting combinations of arguments for these syscalls
in a reasonable time (provided that you use CONFIG_KCOV). And in the
end you don't know where the bugs. They are usually where you don't
expect them. So I would suggest the opposite: describe and more
perf-related syscalls, describe arguments in greater detail, enable
other syscalls that can have effect on perf subsystem. And then just
run it for longer using more machines.