Received-SPF: pass (google.com: domain of linux-kernel+bounces-54611-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
From: Valentin Schneider <vschneid@redhat.com>
To: richard clark <richard.xnu.clark@gmail.com>
Cc: Mark Rutland <mark.rutland@arm.com>, Steven Rostedt
 <rostedt@goodmis.org>, nico@fluxnic.net, mhiramat@kernel.org,
 linux-arm-kernel@lists.infradead.org, linux-trace-kernel@vger.kernel.org,
 linux-kernel@vger.kernel.org
Subject: Re: Question about the ipi_raise filter usage and output
In-Reply-To: <CAJNi4rMiGcP4wdA=1dSOXwYXOKSCWnN8FYxBaFdaAXBqAU_ePQ@mail.gmail.com>
References: <ZcDwoce6Ok25K6Dm@FVFF77S0Q05N>
 <xhsmhmssehp6t.mognet@vschneid-thinkpadt14sgen2i.remote.csb>
 <CAJNi4rMiGcP4wdA=1dSOXwYXOKSCWnN8FYxBaFdaAXBqAU_ePQ@mail.gmail.com>
Date: Tue, 06 Feb 2024 10:39:28 +0100
Message-ID: <xhsmhjznigcdr.mognet@vschneid-thinkpadt14sgen2i.remote.csb>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 06/02/24 16:42, richard clark wrote:
> On Tue, Feb 6, 2024 at 12:05=E2=80=AFAM Valentin Schneider <vschneid@redh=
at.com> wrote:
>>
>> The CPUS{} thingie only works with an event field that is either declare=
d as a
>> cpumask (__cpumask) or a scalar. That's not the case for ipi_raise, the
>> target_cpus event field is saved as a "raw" bitmask.
>>
>> There /should/ have been a warning about the event filter though, but I
>> think it's not happening because I'm allowing more than just FILTER_CPUM=
ASK
>> in parse_pred() to make it work for scalars. I'll go poke around some mo=
re.
>>
>> Generally for this sort of IPI investigation I'd recommend using the new=
er
>> trace_ipi_send_cpu() and trace_ipi_send_cpumask() (for which CPUS{}
>> filtering works).
>> If it's only the function call interrupts you're interesting in, have a
>> look at trace_csd_queue_cpu().
>
> This should be supported by newer version kernels like v6.5, but I am
> using v6.1 and this trace event has not been supported yet... so ipi
> is more suitable for me. ipi_entry and ipi_exit is ok, but seems the
> filter doesn't support a specific cpu, maybe we need to add this?
>>

You should have access to the generic fields which include the CPU from
which the event happens. Any of "CPU", "cpu" or "common_cpu" would match
this.

So if you're on a recent enough kernel (v6.6 or above AFAICT), you should
be able to do something like so:

  trace-cmd record -e 'ipi_raise' -f 'CPU & CPUS{7-42}' ./foo.sh

If you just want to match a single CPU, or are on an older kernel, this
should work as well:

  trace-cmd record -e 'ipi_raise' -f 'CPU =3D=3D 42' ./foo.sh

For example on a QEMU x86 environment:

  # trace-cmd record -e 'call_function*' -f 'CPU & CPUS{3}' hackbench
  Running in process mode with 10 groups using 40 file descriptors each (=
=3D=3D 400 tasks)
  Each sender will pass 100 messages of 100 bytes
  Time: 0.396
  CPU0 data recorded at offset=3D0x738000
      0 bytes in size
  CPU1 data recorded at offset=3D0x738000
      0 bytes in size
  CPU2 data recorded at offset=3D0x738000
      0 bytes in size
  CPU3 data recorded at offset=3D0x738000
      4096 bytes in size

  # trace-cmd report
  CPU 0 is empty
  CPU 1 is empty
  CPU 2 is empty
  cpus=3D4
            <idle>-0     [003]    29.704387: call_function_single_entry: ve=
ctor=3D251
            <idle>-0     [003]    29.704388: call_function_single_exit: vec=
tor=3D251
            <idle>-0     [003]    29.705950: call_function_single_entry: ve=
ctor=3D251
            <idle>-0     [003]    29.705951: call_function_single_exit: vec=
tor=3D251
            <idle>-0     [003]    29.706462: call_function_single_entry: ve=
ctor=3D251
            <idle>-0     [003]    29.706463: call_function_single_exit: vec=
tor=3D251
         hackbench-962   [003]    29.706501: call_function_single_entry: ve=
ctor=3D251
         hackbench-962   [003]    29.706502: call_function_single_exit: vec=
tor=3D251
         hackbench-955   [003]    29.706521: call_function_single_entry: ve=
ctor=3D251
         hackbench-955   [003]    29.706522: call_function_single_exit: vec=
tor=3D251
            <idle>-0     [003]    30.101812: call_function_single_entry: ve=
ctor=3D251
            <idle>-0     [003]    30.101814: call_function_single_exit: vec=
tor=3D251
            <idle>-0     [003]    30.101897: call_function_single_entry: ve=
ctor=3D251
            <idle>-0     [003]    30.101898: call_function_single_exit: vec=
tor=3D251
            <idle>-0     [003]    30.101985: call_function_single_entry: ve=
ctor=3D251
            <idle>-0     [003]    30.101986: call_function_single_exit: vec=
tor=3D251
            <idle>-0     [003]    30.102072: call_function_single_entry: ve=
ctor=3D251
            <idle>-0     [003]    30.102072: call_function_single_exit: vec=
tor=3D251
            <idle>-0     [003]    30.102161: call_function_single_entry: ve=
ctor=3D251
            <idle>-0     [003]    30.102161: call_function_single_exit: vec=
tor=3D251
            <idle>-0     [003]    30.102250: call_function_single_entry: ve=
ctor=3D251
            <idle>-0     [003]    30.102251: call_function_single_exit: vec=
tor=3D251
            <idle>-0     [003]    30.102372: call_function_single_entry: ve=
ctor=3D251
            <idle>-0     [003]    30.102372: call_function_single_exit: vec=
tor=3D251


  CPU 0 is empty
  CPU 1 is empty
  CPU 2 is empty
  cpus=3D4
          <idle>-0     [003]  1067.718304: call_function_single_entry: vect=
or=3D251
          <idle>-0     [003]  1067.718309: call_function_single_exit: vecto=
r=3D251

and that behaves the same as

  trace-cmd record -e 'call_function*' -f 'CPU =3D=3D 3' hackbench