Message-ID: <20b09a5b-99b3-75b8-3ec8-e978965497f2@linux.intel.com>
Date:   Wed, 1 Feb 2023 10:27:51 -0500
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.6.1
Subject: Re: [6.1.7][6.2-rc5] perf all metrics test: FAILED!
Content-Language: en-US
To:     Ian Rogers <irogers@google.com>, sedat.dilek@gmail.com
Cc:     "Xing, Zhengjun" <zhengjun.xing@intel.com>,
        Arnaldo Carvalho de Melo <acme@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Ingo Molnar <mingo@redhat.com>,
        Mark Rutland <mark.rutland@arm.com>,
        Alexander Shishkin <alexander.shishkin@linux.intel.com>,
        Jiri Olsa <jolsa@kernel.org>,
        Namhyung Kim <namhyung@kernel.org>,
        linux-perf-users@vger.kernel.org, linux-kernel@vger.kernel.org,
        Nick Desaulniers <ndesaulniers@google.com>,
        Nathan Chancellor <natechancellor@gmail.com>,
        llvm@lists.linux.dev, Ben Hutchings <benh@debian.org>,
        James Clark <james.clark@arm.com>,
        Stephane Eranian <eranian@google.com>
References: <CA+icZUU_ew7pzWJJZLbj1xsU6MQTPrj8tkFfDhNdTDRQfGUBMQ@mail.gmail.com>
 <CAP-5=fULKkEYXOVV5tXd8J0occGJwgV+BiJLkA=exW=bfgyEBw@mail.gmail.com>
 <CA+icZUUOZoLOFiBcYkccWPSusk9G_Rhf3DOZYWat-K+VfWFHQw@mail.gmail.com>
 <c6605d52-b9e1-e8d3-ed29-562146f64eea@arm.com>
 <CAP-5=fWcb8m9vkfqSC9H2Gqi2dBjbPuGb2F27Fq-ejmR25foQw@mail.gmail.com>
 <CA+icZUWKFCdhQ9XN0wLN6BF5ft3WaNrtXDb9D8ckNOPG9gUf+g@mail.gmail.com>
 <CAP-5=fUSyN5wCn5miG79cT30YruJjkAYMeR58XXgRBQw-6hHkQ@mail.gmail.com>
From:   "Liang, Kan" <kan.liang@linux.intel.com>
In-Reply-To: <CAP-5=fUSyN5wCn5miG79cT30YruJjkAYMeR58XXgRBQw-6hHkQ@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Precedence: bulk

Hi Ian,

On 2023-01-30 10:55 p.m., Ian Rogers wrote:
>>> There's a question about what we should do in the perf test about
>>> this? I have a few solutions:
>>>
>>> 1) try metric tests again with the --metric-no-group flag and don't
>>> fail the test if this succeeds. This allows kernel bugs to hide, so
>>> I'm not a huge fan.
>>>
>>> 2) add a new metric flag/constraint to say not to group, this way the
>>> metric will automatically apply the "--metric-no-group" flag. It is a
>>> bit of work to wire this up but this kind of failure is common enough
>>> in PMUs that it is probably worthwhile. We also need to add the flag
>>> to metrics and I'm not sure how to get a good list of the metrics that
>>> currently fail and require it. This is okay but error prone.
>>>
>>> 3) fix the kernel bug and let the perf test fail until an adequate
>>> kernel is installed. Probably the best option.
>>>
>> Hi Ian,
>>
>> I can confirm:
>>
>> $ echo 0 | sudo tee /proc/sys/kernel/kptr_restrict
>> /proc/sys/kernel/perf_event_paranoid
>> 0
>>
>> $ ~/bin/perf stat -M tma_l3_bound --metric-no-group -a sleep 1
>>
>> Performance counter stats for 'system wide':
>>
>>         2.058.892      MEM_LOAD_UOPS_RETIRED.LLC_HIT    #      1,5 %
>> tma_l3_bound             (99,30%)
>>       173.254.697      CYCLE_ACTIVITY.STALLS_L2_PENDING
>>                         (99,10%)
>>     2.396.130.501      CPU_CLK_UNHALTED.THREAD
>>                         (99,60%)
>>         1.110.486      MEM_LOAD_UOPS_MISC_RETIRED.LLC_MISS
>>                            (99,53%)
>>
>>       1,001989022 seconds time elapsed
>>
>> $ ~/bin/perf stat -M tma_dram_bound --metric-no-group -a sleep 1
>>
>> Performance counter stats for 'system wide':
>>
>>         1.729.208      MEM_LOAD_UOPS_RETIRED.LLC_HIT    #      1,2 %
>> tma_dram_bound           (99,50%)
>>        50.346.734      CYCLE_ACTIVITY.STALLS_L2_PENDING
>>                         (99,50%)
>>     2.354.963.862      CPU_CLK_UNHALTED.THREAD
>>                         (99,80%)
>>           306.500      MEM_LOAD_UOPS_MISC_RETIRED.LLC_MISS
>>                            (99,61%)
>>
>>       1,001981392 seconds time elapsed
>>
>> Thanks!
> Thanks, apparently it is an issue with SandyBridge/IvyBridge that some
> counters on one hyperthread will limit what can be on the other. I
> believe that's the comment related to EXCL access here:
> https://github.com/torvalds/linux/blob/master/arch/x86/events/intel/core.c#L124
> So you may have more success with the metric if you disable
> hyperthreading, but I imagine that's not a popular option.

Thanks for debugging the issue. Yes, it's caused by the HT workaround
for SNB/IVB/HSW.

The weak group check in the kernel is in validate_group(). It only does
a sanity check. It doesn't check all the workarounds and the current
status of counters (e.g., whether the fixed counter is occupied by NMI
watchdog.) It's possible that a false positive is returned to the perf
tool. I once tried to fix the NMI watchdog check in the kernel, but the
proposal was rejected. So the metric constraint is introduced.

For this issue, I think the above option2 should be a better and
practical choice. The issue is only observed on old machines, which
usually has a stable kernel running on it. I don't think the user wants
to update their kernel just to workaround an issue for several metrics.
But it should be much easier for them to update the perf tool.

We know that the below events are the problematic events.
/* MEM_UOPS_RETIRED.* */
/* MEM_LOAD_UOPS_RETIRED.* */
/* MEM_LOAD_UOPS_LLC_HIT_RETIRED.* */
/* MEM_LOAD_UOPS_LLC_MISS_RETIRED.* */
Can we update the convertor script and apply the "--metric-no-group"
flag or add a new constraint if the above events are detected in
SNB/IVB/HSW?

Thanks,
Kan