Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751238AbcCHSWp (ORCPT ); Tue, 8 Mar 2016 13:22:45 -0500 Received: from mail-wm0-f68.google.com ([74.125.82.68]:36369 "EHLO mail-wm0-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750935AbcCHSWk (ORCPT ); Tue, 8 Mar 2016 13:22:40 -0500 Date: Tue, 8 Mar 2016 19:22:35 +0100 From: Ingo Molnar To: Dmitry Vyukov Cc: Peter Zijlstra , Wang Nan , Ingo Molnar , LKML , He Kuang , Alexei Starovoitov , Arnaldo Carvalho de Melo , Brendan Gregg , Jiri Olsa , Masami Hiramatsu , Namhyung Kim , Zefan Li , pi3orama@163.com Subject: Re: [RESEND PATCH 0/5] perf core: Support overwrite ring buffer Message-ID: <20160308182235.GB4220@gmail.com> References: <20160308164438.GA24109@gmail.com> <20160308164859.GA27516@gmail.com> <20160308172425.GA3017@gmail.com> <20160308173709.GC3017@gmail.com> <20160308174856.GA28862@gmail.com> <20160308175753.GA3015@gmail.com> <20160308180238.GA4220@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160308180238.GA4220@gmail.com> User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4412 Lines: 64 * Ingo Molnar wrote: > With nproc set to 120 it seems to be chugging along at about 25% system > utilization: > > Tasks: 1271 total, 1 running, 1270 sleeping, 0 stopped, 0 zombie > %Cpu(s): 4.5 us, 35.3 sy, 0.0 ni, 55.2 id, 0.5 wa, 4.5 hi, 0.0 si, 0.0 st > KiB Mem : 26401230+total, 25401017+free, 1624640 used, 8377496 buff/cache > KiB Swap: 0 total, 0 free, 0 used. 26143329+avail Mem > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 87593 mingo 20 0 7850436 162284 11672 S 2941 0.1 33:01.83 syz-fuzzer > 923 root 20 0 84772 44344 43840 S 22.8 0.0 1:23.86 systemd-journal > 1369 root 16 -4 114636 3256 2832 S 15.7 0.0 0:29.03 auditd > 1379 root 12 -8 80236 1764 1432 S 8.3 0.0 0:15.79 audispd > 878 root 20 0 0 0 0 S 6.9 0.0 0:16.55 jbd2/sda1-8 > 1381 root 16 -4 52216 3232 2892 S 3.8 0.0 0:07.08 sedispatch > > Even that one is not ideal - obviously there's way too much systemd-journal > overhead, but I'm unable to turn the darn thing off ... The journal.conf man page suggests that putting 'Storage=none' into /etc/systemd/journal.conf disables journalling - but that's not true. It's apparently impossible to disable systemd logging via any normal means ... So the workaround for all that is a brutal: mv /var/log/journal /var/log/journal.dontuse after that there does not seem to be systemd logging anymore. ... but there is tons of auditd logging now! ;-) Fortunately that's easily stopped via: service stop auditd with that there's no log IO anymore during fuzzing (yay!). Except that systemd journald rears its ugly head back: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 111464 mingo 20 0 7249024 143348 11476 S 2741 0.1 7:41.31 syz-fuzzer 122632 root 20 0 29200 2768 2524 S 9.5 0.0 0:28.74 systemd-journal 111463 root 20 0 165596 5772 3840 R 4.8 0.0 0:00.80 top 112078 mingo 20 0 20928 704 644 S 2.9 0.0 0:00.17 syz-executor 112516 mingo 20 0 20928 708 644 S 2.9 0.0 0:00.16 syz-executor 878 root 20 0 0 0 0 S 1.9 0.0 0:53.83 jbd2/sda1-8 111711 mingo 20 0 20928 704 644 S 1.9 0.0 0:00.16 syz-executor 111901 mingo 20 0 20928 704 644 S 1.9 0.0 0:00.15 syz-executor 8 root 20 0 0 0 0 S 1.0 0.0 1:00.55 rcu_sched so it's eating about 10% of system overhead despite doing nothing (!). The workaround for that systemd bug is a brutal: [root@fomalhaut ~]# mv /usr/lib/systemd/systemd-journald /usr/lib/systemd/systemd-journald.dontuse [root@fomalhaut ~]# Thanks, Ingo