Received-SPF: pass (google.com: domain of linux-kernel+bounces-180621-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Precedence: bulk
MIME-Version: 1.0
References: <20240424024805.144759-1-howardchu95@gmail.com>
 <CAM9d7chOdrPyeGk=O+7Hxzdm5ziBXLES8PLbpNJvA7_DMrrGHA@mail.gmail.com>
 <Zil1ZKc7mibs6ONQ@x1> <CAP-5=fVYHjUk8OyidXbutBvZMPxf48LW7v-N3zvHBe5QME1vVQ@mail.gmail.com>
 <CAM9d7cggak7qZcX7tFZvJ69H3cwEnWvNOnBsQrkFQkQVf+bUjQ@mail.gmail.com>
In-Reply-To: <CAM9d7cggak7qZcX7tFZvJ69H3cwEnWvNOnBsQrkFQkQVf+bUjQ@mail.gmail.com>
From: Howard Chu <howardchu95@gmail.com>
Date: Thu, 16 May 2024 12:24:35 +0800
Message-ID: <CAH0uvohPg7LtSOLDNaPwnC5ePwjwg0NtKzLZ_oJcAz7zOwdwdw@mail.gmail.com>
Subject: Re: [PATCH v2 0/4] Dump off-cpu samples directly
To: Namhyung Kim <namhyung@kernel.org>
Cc: Ian Rogers <irogers@google.com>, Arnaldo Carvalho de Melo <acme@kernel.org>, peterz@infradead.org, 
	mingo@redhat.com, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, 
	jolsa@kernel.org, adrian.hunter@intel.com, kan.liang@linux.intel.com, 
	zegao2021@gmail.com, leo.yan@linux.dev, ravi.bangoria@amd.com, 
	linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org, 
	bpf@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hello,

Here is a little update on --off-cpu.

> > It would be nice to start landing this work so I'm wondering what the
> > minimal way to do that is. It seems putting behavior behind a flag is
> > a first step.

The flag to determine output threshold of off-cpu has been implemented.
If the accumulated off-cpu time exceeds this threshold, output the sample
directly; otherwise, save it later for off_cpu_write.

But adding an extra pass to handle off-cpu samples introduces performance
issues, here's the processing rate of --off-cpu sampling(with the
extra pass to extract raw
sample data) and without. The --off-cpu-threshold is in nanoseconds.

+-----------------------------------------------------+--------------------=
-------------------+----------------------+
| comm                                                | type
                       | process rate         |
+-----------------------------------------------------+--------------------=
-------------------+----------------------+
| -F 4999 -a                                          | regular
samples (w/o extra pass)      | 13128.675 samples/ms |
+-----------------------------------------------------+--------------------=
-------------------+----------------------+
| -F 1 -a --off-cpu --off-cpu-threshold 100           | offcpu samples
(extra pass)           |  2843.247 samples/ms |
+-----------------------------------------------------+--------------------=
-------------------+----------------------+
| -F 4999 -a --off-cpu --off-cpu-threshold 100        | offcpu &
regular samples (extra pass) |  3910.686 samples/ms |
+-----------------------------------------------------+--------------------=
-------------------+----------------------+
| -F 4999 -a --off-cpu --off-cpu-threshold 1000000000 | few offcpu &
regular (extra pass)     |  4661.229 samples/ms |
+-----------------------------------------------------+--------------------=
-------------------+----------------------+

It's not ideal. I will find a way to reduce overhead. For example
process them samples
at save time as Ian mentioned.

> > To turn the bpf-output samples into off-cpu events there is a pass
> > added to the saving. I wonder if that can be more generic, like a save
> > time perf inject.

And I will find a default value for such a threshold based on performance
and common use cases.

> Sounds good.  We might add an option to specify the threshold to
> determine whether to dump the data or to save it for later.  But ideally
> it should be able to find a good default.

These will be done before the GSoC kick-off on May 27.

Thanks,
Howard

On Thu, Apr 25, 2024 at 6:57=E2=80=AFAM Namhyung Kim <namhyung@kernel.org> =
wrote:
>
> On Wed, Apr 24, 2024 at 3:19=E2=80=AFPM Ian Rogers <irogers@google.com> w=
rote:
> >
> > On Wed, Apr 24, 2024 at 2:11=E2=80=AFPM Arnaldo Carvalho de Melo
> > <acme@kernel.org> wrote:
> > >
> > > On Wed, Apr 24, 2024 at 12:12:26PM -0700, Namhyung Kim wrote:
> > > > Hello,
> > > >
> > > > On Tue, Apr 23, 2024 at 7:46=E2=80=AFPM Howard Chu <howardchu95@gma=
il.com> wrote:
> > > > >
> > > > > As mentioned in: https://bugzilla.kernel.org/show_bug.cgi?id=3D20=
7323
> > > > >
> > > > > Currently, off-cpu samples are dumped when perf record is exiting=
 This
> > > > > results in off-cpu samples being after the regular samples. Also,=
 samples
> > > > > are stored in large BPF maps which contain all the stack traces a=
nd
> > > > > accumulated off-cpu time, but they are eventually going to fill u=
p after
> > > > > running for an extensive period. This patch fixes those problems =
by dumping
> > > > > samples directly into perf ring buffer, and dispatching those sam=
ples to the
> > > > > correct format.
> > > >
> > > > Thanks for working on this.
> > > >
> > > > But the problem of dumping all sched-switch events is that it can b=
e
> > > > too frequent on loaded machines.  Copying many events to the buffer
> > > > can result in losing other records.  As perf report doesn't care ab=
out
> > > > timing much, I decided to aggregate the result in a BPF map and dum=
p
> > > > them at the end of the profiling session.
> > >
> > > Should we try to adapt when there are too many context switches, i.e.
> > > the BPF program can notice that the interval from the last context
> > > switch is too small and then avoid adding samples, while if the inter=
val
> > > is a long one then indeed this is a problem where the workload is
> > > waiting for a long time for something and we want to know what is tha=
t,
> > > and in that case capturing callchains is both desirable and not costl=
y,
> > > no?
>
> Sounds interesting.  Yeah we could make it adaptive based on the
> off-cpu time at the moment.
>
> > >
> > > The tool could then at the end produce one of two outputs: the most
> > > common reasons for being off cpu, or some sort of counter stating tha=
t
> > > there are way too many context switches?
> > >
> > > And perhaps we should think about what is best to have as a default, =
not
> > > to present just plain old cycles, but point out that the workload is
> > > most of the time waiting for IO, etc, i.e. the default should give
> > > interesting clues instead of expecting that the tool user knows all t=
he
> > > possible knobs and try them in all sorts of combinations to then reac=
h
> > > some conclusion.
> > >
> > > The default should use stuff that isn't that costly, thus not getting=
 in
> > > the way of what is being observed, but at the same time look for comm=
on
> > > patterns, etc.
> > >
> > > - Arnaldo
> >
> > I really appreciate Howard doing this work!
> >
> > I wonder there are other cases where we want to synthesize events in
> > BPF, for example, we may have fast and slow memory on a system, we
> > could turn memory events on a system into either fast or slow ones in
> > BPF based on the memory accessed, so that fast/slow memory systems can
> > be simulated without access to hardware. This also feels like a perf
> > script type problem. Perhaps we can add something to the bpf-output
> > event so it can have multiple uses and not just off-cpu.
> >
> >
> > I worry about dropping short samples we can create a property that
> > off-cpu time + on-cpu time !=3D wall clock time. Perhaps such short
> > things can get pushed into Namhyung's "at the end" approach while
> > longer things get samples. Perhaps we only do that when the frequency
> > is too great.
>
> Sounds good.  We might add an option to specify the threshold to
> determine whether to dump the data or to save it for later.  But ideally
> it should be able to find a good default.
>
> >

>
> Agreed!
>
> Thanks,
> Namhyung