Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
MIME-Version: 1.0
References: <20220527110959.54559-1-zhangyuchen.lcr@bytedance.com>
In-Reply-To: <20220527110959.54559-1-zhangyuchen.lcr@bytedance.com>
From:   Alexei Starovoitov <alexei.starovoitov@gmail.com>
Date:   Fri, 27 May 2022 07:13:54 -0700
Message-ID: <CAADnVQL69J8MWhaNzNG=ANL_i8_QqABON+pWJDuqRTkFGPJYUQ@mail.gmail.com>
Subject: Re: [PATCH] procfs: add syscall statistics
To:     Zhang Yuchen <zhangyuchen.lcr@bytedance.com>
Cc:     Andrew Morton <akpm@linux-foundation.org>,
        David Hildenbrand <david@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@redhat.com>,
        Alexei Starovoitov <ast@kernel.org>,
        LKML <linux-kernel@vger.kernel.org>,
        Linux-Fsdevel <linux-fsdevel@vger.kernel.org>,
        "open list:DOCUMENTATION" <linux-doc@vger.kernel.org>,
        linux-arm-kernel <linux-arm-kernel@lists.infradead.org>,
        linux-s390 <linux-s390@vger.kernel.org>,
        Linux API <linux-api@vger.kernel.org>, fam.zheng@bytedance.com,
        bpf <bpf@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Fri, May 27, 2022 at 4:10 AM Zhang Yuchen
<zhangyuchen.lcr@bytedance.com> wrote:
>
> Add /proc/syscalls to display percpu syscall count.

I second Peter's nack.
We don't add debug features to the production kernel.

> We need a less resource-intensive way to count syscall per cpu
> for system problem location.
>
> There is a similar utility syscount in the BCC project, but syscount
> has a high performance cost.

There are two syscount tools in BCC:
tools/syscount
libbpf-tools/syscount

Which one has this 42% overhead?

The former tool is obsolete though.
It was written in the days when bpf had 1/10 of
the features it has today.
Both tools can be optimized.
They attach to raw_syscalls tracepoint.
tracepoints are not cheap.
In terms of overhead:
tracepoint > raw_tracepoint > fentry.
bpf can attach to all three.

Please profile libbpf-tools/syscount tool
with perf and unixbench, understand where overhead
comes from and then optimize the tool.