Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38;
Date:   Thu, 23 Nov 2023 14:45:30 +0000
Message-ID: <86edggzfxx.wl-maz@kernel.org>
From:   Marc Zyngier <maz@kernel.org>
To:     Mark Rutland <mark.rutland@arm.com>
Cc:     Hector Martin <marcan@marcan.st>,
        Arnaldo Carvalho de Melo <acme@redhat.com>,
        Ian Rogers <irogers@google.com>,
        James Clark <james.clark@arm.com>,
        linux-perf-users@vger.kernel.org,
        LKML <linux-kernel@vger.kernel.org>,
        Asahi Linux <asahi@lists.linux.dev>
Subject: Re: [REGRESSION] Perf (userspace) broken on big.LITTLE systems since v6.5
In-Reply-To: <ZV9gThJ52slPHqlV@FVFF77S0Q05N.cambridge.arm.com>
References: <08f1f185-e259-4014-9ca4-6411d5c1bc65@marcan.st>
        <86pm03z0kw.wl-maz@kernel.org>
        <86o7fnyvrq.wl-maz@kernel.org>
        <ZV9gThJ52slPHqlV@FVFF77S0Q05N.cambridge.arm.com>
User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue)
 FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/29.1
 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO)
MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue")
Content-Type: text/plain; charset=US-ASCII
Precedence: bulk

On Thu, 23 Nov 2023 14:23:10 +0000,
Mark Rutland <mark.rutland@arm.com> wrote:
> 
> On Tue, Nov 21, 2023 at 03:24:25PM +0000, Marc Zyngier wrote:
> > On Tue, 21 Nov 2023 13:40:31 +0000,
> > Marc Zyngier <maz@kernel.org> wrote:
> > > 
> > > [Adding key people on Cc]
> > > 
> > > On Tue, 21 Nov 2023 12:08:48 +0000,
> > > Hector Martin <marcan@marcan.st> wrote:
> > > > 
> > > > Perf broke on all Apple ARM64 systems (tested almost everything), and
> > > > according to maz also on Juno (so, probably all big.LITTLE) since v6.5.
> > > 
> > > I can confirm that at least on 6.7-rc2, perf is pretty busted on any
> > > asymmetric ARM platform. It isn't clear what criteria is used to pick
> > > the PMU, but nothing works anymore.
> > > 
> > > The saving grace in my case is that Debian still ships a 6.1 perftool
> > > package, but that's obviously not going to last.
> > > 
> > > I'm happy to test potential fixes.
> > 
> > At Mark's request, I've dumped a couple of perf (as of -rc2) runs with
> > -vvv.  And it is quite entertaining (this is taskset to an 'icestorm'
> > CPU):
> 
> Looking at this with fresh(er) eyes, I think there's a userspace bug here,
> regardless of whether one believes it's correct to convert a named-pmu event to
> a PERF_TYPE_HARDWARE event directed at that PMU.
> 
> It looks like the userspace tool is dropping the extended type ID after an
> initial probe, and requests events with plain PERF_TYPE_HARDWARE (without an
> extended type ID), which explains why we seem to get events from one PMU only.
> 
> More detail below...
> 
> Marc, if you have time, could you run the same commands (on the same kernel)
> with a perf tool build from v6.4?

Here you go:

<quote>
$ sudo taskset -c 0 ./perf stat -vvv -e apple_icestorm_pmu/cycles/ -e  apple_firestorm_pmu/cycles/ -e cycles ls >/dev/null
Using CPUID 0x00000000610f0280
Attempting to add event pmu 'apple_icestorm_pmu' with 'cycles,' that may result in non-fatal errors
After aliases, add event pmu 'apple_icestorm_pmu' with 'event,' that may result in non-fatal errors
Attempting to add event pmu 'apple_firestorm_pmu' with 'cycles,' that may result in non-fatal errors
After aliases, add event pmu 'apple_firestorm_pmu' with 'event,' that may result in non-fatal errors
Control descriptor is not initialized
------------------------------------------------------------
perf_event_attr:
  type                             10
  size                             136
  config                           0x2
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
  disabled                         1
  inherit                          1
  enable_on_exec                   1
  exclude_guest                    1
------------------------------------------------------------
sys_perf_event_open: pid 1624462  cpu -1  group_fd -1  flags 0x8 = 3
------------------------------------------------------------
perf_event_attr:
  type                             11
  size                             136
  config                           0x2
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
  disabled                         1
  inherit                          1
  enable_on_exec                   1
  exclude_guest                    1
------------------------------------------------------------
sys_perf_event_open: pid 1624462  cpu -1  group_fd -1  flags 0x8 = 4
------------------------------------------------------------
perf_event_attr:
  size                             136
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
  disabled                         1
  inherit                          1
  enable_on_exec                   1
  exclude_guest                    1
------------------------------------------------------------
sys_perf_event_open: pid 1624462  cpu -1  group_fd -1  flags 0x8 = 5
apple_icestorm_pmu/cycles/: -1: 1492180 724333 724333
apple_firestorm_pmu/cycles/: -1: 0 724333 0
cycles: -1: 0 724333 0
apple_icestorm_pmu/cycles/: 1492180 724333 724333
apple_firestorm_pmu/cycles/: 0 724333 0
cycles: 0 724333 0

 Performance counter stats for 'ls':

         1,492,180      apple_icestorm_pmu/cycles/                                            
     <not counted>      apple_firestorm_pmu/cycles/                                             (0.00%)
     <not counted>      cycles                                                                  (0.00%)

       0.000001917 seconds time elapsed

       0.000000000 seconds user
       0.000000000 seconds sys
</quote>

and on the other cluster:

<quote>
$ sudo taskset -c 2 ./perf stat -vvv -e apple_icestorm_pmu/cycles/ -e  apple_firestorm_pmu/cycles/ -e cycles ls >/dev/null
Using CPUID 0x00000000610f0280
Attempting to add event pmu 'apple_icestorm_pmu' with 'cycles,' that may result in non-fatal errors
After aliases, add event pmu 'apple_icestorm_pmu' with 'event,' that may result in non-fatal errors
Attempting to add event pmu 'apple_firestorm_pmu' with 'cycles,' that may result in non-fatal errors
After aliases, add event pmu 'apple_firestorm_pmu' with 'event,' that may result in non-fatal errors
Control descriptor is not initialized
------------------------------------------------------------
perf_event_attr:
  type                             10
  size                             136
  config                           0x2
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
  disabled                         1
  inherit                          1
  enable_on_exec                   1
  exclude_guest                    1
------------------------------------------------------------
sys_perf_event_open: pid 1624466  cpu -1  group_fd -1  flags 0x8 = 3
------------------------------------------------------------
perf_event_attr:
  type                             11
  size                             136
  config                           0x2
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
  disabled                         1
  inherit                          1
  enable_on_exec                   1
  exclude_guest                    1
------------------------------------------------------------
sys_perf_event_open: pid 1624466  cpu -1  group_fd -1  flags 0x8 = 4
------------------------------------------------------------
perf_event_attr:
  size                             136
  sample_type                      IDENTIFIER
  read_format                      TOTAL_TIME_ENABLED|TOTAL_TIME_RUNNING
  disabled                         1
  inherit                          1
  enable_on_exec                   1
  exclude_guest                    1
------------------------------------------------------------
sys_perf_event_open: pid 1624466  cpu -1  group_fd -1  flags 0x8 = 5
apple_icestorm_pmu/cycles/: -1: 0 593209 0
apple_firestorm_pmu/cycles/: -1: 1038247 593209 593209
cycles: -1: 1037870 593209 593209
apple_icestorm_pmu/cycles/: 0 593209 0
apple_firestorm_pmu/cycles/: 1038247 593209 593209
cycles: 1037870 593209 593209

 Performance counter stats for 'ls':

     <not counted>      apple_icestorm_pmu/cycles/                                              (0.00%)
         1,038,247      apple_firestorm_pmu/cycles/                                           
         1,037,870      cycles                                                                

       0.000001500 seconds time elapsed

       0.000000000 seconds user
       0.000000000 seconds sys
</quote>

For the record, this is on a 6.6-rc6 kernel, userspace perf as of v6.4.0.

	M.

-- 
Without deviation from the norm, progress is not possible.