Received-SPF: pass (google.com: domain of linux-kernel+bounces-160323-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <88b460e5-ef2f-4b2f-a173-deac34c99788@intel.com>
Date: Fri, 26 Apr 2024 18:42:42 +0300
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v1] perf tests: Run tests in parallel by default
To: James Clark <james.clark@arm.com>, Ian Rogers <irogers@google.com>
Cc: Peter Zijlstra <peterz@infradead.org>, Ingo Molnar <mingo@redhat.com>,
 Arnaldo Carvalho de Melo <acme@kernel.org>,
 Namhyung Kim <namhyung@kernel.org>, Mark Rutland <mark.rutland@arm.com>,
 Alexander Shishkin <alexander.shishkin@linux.intel.com>,
 Jiri Olsa <jolsa@kernel.org>, Kan Liang <kan.liang@linux.intel.com>,
 linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org
References: <20240301174711.2646944-1-irogers@google.com>
 <3e54fea7-2a37-4774-8a6a-85f75cc4a9ea@arm.com>
Content-Language: en-US
From: Adrian Hunter <adrian.hunter@intel.com>
Organization: Intel Finland Oy, Registered Address: PL 281, 00181 Helsinki,
 Business Identity Code: 0357606 - 4, Domiciled in Helsinki
In-Reply-To: <3e54fea7-2a37-4774-8a6a-85f75cc4a9ea@arm.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

On 26/04/24 18:06, James Clark wrote:
> 
> 
> On 01/03/2024 17:47, Ian Rogers wrote:
>> Switch from running tests sequentially to running in parallel by
>> default. Change the opt-in '-p' or '--parallel' flag to '-S' or
>> '--sequential'.
>>
>> On an 8 core tigerlake an address sanitizer run time changes from:
>> 326.54user 622.73system 6:59.91elapsed 226%CPU
>> to:
>> 973.02user 583.98system 3:01.17elapsed 859%CPU
>>
>> So over twice as fast, saving 4 minutes.
>>
> 
> Apologies for not replying earlier before this was applied. But IMO this
> isn't a good default. Tests that use things like exclusive PMUs
> (Coresight for example) can never pass when run in parallel.

Yes, that is an issue for Intel PT also.

> 
> For CI it's arguable whether you'd want to trade stability for speed.
> And for interactive sessions there was already the --parallel option
> which was easy to add and have it in your bash history.
> 
> Now we've changed the default, any CI will need to be updated to add
> --sequential if it wants all the tests to pass.

Same here.

>                                                 Maybe we could do some
> hack and gate it on interactive vs non interactive sessions, but that
> might be getting too clever. (Or a "don't run in parallel" flag on
> certain tests)

Perhaps more attention is needed for the tests that take so long.
For example, maybe they could do things in parallel.

Also -F option doesn't seem to work.

$ tools/perf/perf test -F
Couldn't find a vmlinux that matches the kernel running on this machine, skipping test
  1: vmlinux symtab matches kallsyms                                 : Skip
  2: Detect openat syscall event                                     : Ok
  3: Detect openat syscall event on all cpus                         : Ok
  4.1: Read samples using the mmap interface                         : Ok
  4.2: User space counter reading of instructions                    : Ok
  4.3: User space counter reading of cycles                          : Ok
  5: Test data source output                                         : Ok
WARNING: event 'numpmu' not valid (bits 16-17,20,22 of config '6530160' not supported by kernel)!
  6.1: Test event parsing                                            : Ok
  6.2: Parsing of all PMU events from sysfs                          : Ok
WARNING: event 'N/A' not valid (bits 0-1 of config2 '3' not supported by kernel)!
  6.3: Parsing of given PMU events from sysfs                        : Ok
  6.4: Parsing of aliased events from sysfs                          : Skip (no aliases in sysfs)
  6.5: Parsing of aliased events                                     : Ok
  6.6: Parsing of terms (event modifiers)                            : Ok
  7: Simple expression parser                                        : Ok
  8: PERF_RECORD_* events & perf_sample fields                       : Ok
  9: Parse perf pmu format                                           : Ok
 10.1: PMU event table sanity                                        : Ok
 10.2: PMU event map aliases                                         : Ok
failed: recursion detected for M1
failed: recursion detected for M2
failed: recursion detected for M3
Not grouping metric tma_memory_fence's events.
Try disabling the NMI watchdog to comply NO_NMI_WATCHDOG metric constraint:
    echo 0 > /proc/sys/kernel/nmi_watchdog
    perf stat ...
    echo 1 > /proc/sys/kernel/nmi_watchdog
Try disabling the NMI watchdog to comply NO_NMI_WATCHDOG metric constraint:
    echo 0 > /proc/sys/kernel/nmi_watchdog
    perf stat ...
    echo 1 > /proc/sys/kernel/nmi_watchdog
..
^C