Received-SPF: pass (google.com: domain of linux-kernel+bounces-72128-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Precedence: bulk
MIME-Version: 1.0
References: <CAG48ez3RmV6SsVw9oyTXxQXHp3rqtKDk2qwJWo9TGvXCq7Xr-w@mail.gmail.com>
In-Reply-To: <CAG48ez3RmV6SsVw9oyTXxQXHp3rqtKDk2qwJWo9TGvXCq7Xr-w@mail.gmail.com>
From: Ian Rogers <irogers@google.com>
Date: Mon, 19 Feb 2024 15:20:00 -0800
Message-ID: <CAP-5=fXsFBwt9ARmQja0pKGL-_Vms_NDKeaH5CX=_om1aSvssw@mail.gmail.com>
Subject: Re: [BUG] perf/x86/intel: HitM false-positives on Ice Lake / Tiger
 Lake (I think?)
To: Jann Horn <jannh@google.com>
Cc: Jiri Olsa <jolsa@kernel.org>, Peter Zijlstra <peterz@infradead.org>, 
	Ingo Molnar <mingo@redhat.com>, Arnaldo Carvalho de Melo <acme@kernel.org>, Namhyung Kim <namhyung@kernel.org>, 
	Mark Rutland <mark.rutland@arm.com>, 
	Alexander Shishkin <alexander.shishkin@linux.intel.com>, 
	Adrian Hunter <adrian.hunter@intel.com>, Feng Tang <feng.tang@intel.com>, 
	Andi Kleen <ak@linux.intel.com>, "the arch/x86 maintainers" <x86@kernel.org>, 
	kernel list <linux-kernel@vger.kernel.org>, linux-perf-users@vger.kernel.org, 
	"Liang, Kan" <kan.liang@linux.intel.com>, Stephane Eranian <eranian@google.com>, 
	"Taylor, Perry" <perry.taylor@intel.com>, "Alt, Samantha" <samantha.alt@intel.com>, 
	"Biggers, Caleb" <caleb.biggers@intel.com>, "Wang, Weilin" <weilin.wang@intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Mon, Feb 19, 2024 at 5:01=E2=80=AFAM Jann Horn <jannh@google.com> wrote:
>
> Hi!
>
> From what I understand, "perf c2c" shows bogus HitM events on Ice Lake
> (and newer) because Intel added some feature where *clean* cachelines
> can get snoop-forwarded ("cross-core FWD"), and the PMU apparently
> treats this mostly the same as snoop-forwarding of modified cache
> lines (HitM)? On a Tiger Lake CPU, I can see addresses from the kernel
> rodata section in "perf c2c report".
>
> This is mentioned in the SDM, Volume 3B, section "20.9.7 Load Latency
> Facility", table "Table 20-101. Data Source Encoding for Memory
> Accesses (Ice Lake and Later Microarchitectures)", encoding 07H:
> "XCORE FWD. This request was satisfied by a sibling core where either
> a modified (cross-core HITM) or a non-modified (cross-core FWD)
> cache-line copy was found."
>
> I don't see anything about this in arch/x86/events/intel/ds.c - if I
> understand correctly, the kernel's PEBS data source decoding assumes
> that 0x07 means "L3 hit, snoop hitm" on these CPUs. I think this needs
> to be adjusted somehow - and maybe it just isn't possible to actually
> distinguish between HitM and cross-core FWD in PEBS events on these
> CPUs (without big-hammer chicken bit trickery)? Maybe someone from
> Intel can clarify?
>
> (The SDM describes that E-cores on the newer 12th Gen have more
> precise PEBS encodings that distinguish between "L3 HITM" and "L3
> HITF"; but I guess the P-cores there maybe still don't let you
> distinguish HITM/HITF?)
>
>
> I think https://perfmon-events.intel.com/tigerLake.html is also
> outdated, or at least it uses ambiguous grammar: The
> MEM_LOAD_L3_HIT_RETIRED.XSNP_FWD event (EventSel=3DD2H UMask=3D04H) is
> documented as "Counts retired load instructions where a cross-core
> snoop hit in another cores caches on this socket, the data was
> forwarded back to the requesting core as the data was modified
> (SNOOP_HITM) or the L3 did not have the data(SNOOP_HIT_WITH_FWD)" -
> from what I understand, a "cross-core FWD" should be a case where the
> L3 does have the data, unless L3 has become non-inclusive on Ice Lake?
>
> On a Tiger Lake CPU, I can see this event trigger for the
> sys_call_table, which is located in the rodata region and probably
> shouldn't be containing Modified cache lines:
>
> # grep -A1 -w sys_call_table /proc/kallsyms
> ffffffff82800280 D sys_call_table
> ffffffff82801100 d vdso_mapping
> # perf record -e mem_load_l3_hit_retired.xsnp_fwd:ppp --all-kernel -c 100=
 --data
> ^C[ perf record: Woken up 11 times to write data ]
> [ perf record: Captured and wrote 22.851 MB perf.data (43176 samples) ]
> # perf script -F event,ip,sym,addr | egrep --color 'ffffffff828002[89abcd=
ef]'
> mem_load_l3_hit_retired.xsnp_fwd:ppp: ffffffff82800280
> ffffffff82526275 do_syscall_64
> mem_load_l3_hit_retired.xsnp_fwd:ppp: ffffffff828002d8
> ffffffff82526275 do_syscall_64
> mem_load_l3_hit_retired.xsnp_fwd:ppp: ffffffff82800280
> ffffffff82526275 do_syscall_64
> mem_load_l3_hit_retired.xsnp_fwd:ppp: ffffffff828002b8
> ffffffff82526275 do_syscall_64
> mem_load_l3_hit_retired.xsnp_fwd:ppp: ffffffff828002b8
> ffffffff82526275 do_syscall_64
> mem_load_l3_hit_retired.xsnp_fwd:ppp: ffffffff828002b8
> ffffffff82526275 do_syscall_64
> mem_load_l3_hit_retired.xsnp_fwd:ppp: ffffffff82800280
> ffffffff82526275 do_syscall_64
> mem_load_l3_hit_retired.xsnp_fwd:ppp: ffffffff82800288
> ffffffff82526275 do_syscall_64
> mem_load_l3_hit_retired.xsnp_fwd:ppp: ffffffff828002b8
> ffffffff82526275 do_syscall_64
>
>
> (For what it's worth, there is a thread on LKML where "cross-core FWD"
> got mentioned: <https://lore.kernel.org/lkml/b4aaf1ed-124d-1339-3e99-a120=
f6cc4d28@linux.intel.com/>)

+others better qualified than me to respond.

Hi Jann,

I'm not overly familiar with the issue, but it appears a similar issue
has been reported for Broadwell Xeon here:
https://community.intel.com/t5/Software-Tuning-Performance/Broadwell-Xeon-p=
erf-c2c-showing-remote-HITM-but-remote-socket-is/td-p/1172120
I'm not sure that thread will be particularly useful, but having the
Intel people better qualified than me to answer is probably the better
service of this email.

Thanks,
Ian