Received-SPF: pass (google.com: domain of linux-kernel+bounces-73266-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Tue, 20 Feb 2024 12:42:24 -0300
From: Arnaldo Carvalho de Melo <acme@kernel.org>
To: Ian Rogers <irogers@google.com>
Cc: Jann Horn <jannh@google.com>, Joe Mario <jmario@redhat.com>,
	Jiri Olsa <jolsa@kernel.org>, Peter Zijlstra <peterz@infradead.org>,
	Ingo Molnar <mingo@redhat.com>, Namhyung Kim <namhyung@kernel.org>,
	Mark Rutland <mark.rutland@arm.com>,
	Alexander Shishkin <alexander.shishkin@linux.intel.com>,
	Adrian Hunter <adrian.hunter@intel.com>,
	Feng Tang <feng.tang@intel.com>, Andi Kleen <ak@linux.intel.com>,
	the arch/x86 maintainers <x86@kernel.org>,
	kernel list <linux-kernel@vger.kernel.org>,
	linux-perf-users@vger.kernel.org,
	"Liang, Kan" <kan.liang@linux.intel.com>,
	Stephane Eranian <eranian@google.com>,
	"Taylor, Perry" <perry.taylor@intel.com>,
	"Alt, Samantha" <samantha.alt@intel.com>,
	"Biggers, Caleb" <caleb.biggers@intel.com>,
	"Wang, Weilin" <weilin.wang@intel.com>
Subject: Re: [BUG] perf/x86/intel: HitM false-positives on Ice Lake / Tiger
 Lake (I think?)
Message-ID: <ZdTIYLoJOdyu62LU@x1>
References: <CAG48ez3RmV6SsVw9oyTXxQXHp3rqtKDk2qwJWo9TGvXCq7Xr-w@mail.gmail.com>
 <CAP-5=fXsFBwt9ARmQja0pKGL-_Vms_NDKeaH5CX=_om1aSvssw@mail.gmail.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <CAP-5=fXsFBwt9ARmQja0pKGL-_Vms_NDKeaH5CX=_om1aSvssw@mail.gmail.com>

Just adding Joe Mario to the CC list.

On Mon, Feb 19, 2024 at 03:20:00PM -0800, Ian Rogers wrote:
> On Mon, Feb 19, 2024 at 5:01 AM Jann Horn <jannh@google.com> wrote:
> >
> > Hi!
> >
> > From what I understand, "perf c2c" shows bogus HitM events on Ice Lake
> > (and newer) because Intel added some feature where *clean* cachelines
> > can get snoop-forwarded ("cross-core FWD"), and the PMU apparently
> > treats this mostly the same as snoop-forwarding of modified cache
> > lines (HitM)? On a Tiger Lake CPU, I can see addresses from the kernel
> > rodata section in "perf c2c report".
> >
> > This is mentioned in the SDM, Volume 3B, section "20.9.7 Load Latency
> > Facility", table "Table 20-101. Data Source Encoding for Memory
> > Accesses (Ice Lake and Later Microarchitectures)", encoding 07H:
> > "XCORE FWD. This request was satisfied by a sibling core where either
> > a modified (cross-core HITM) or a non-modified (cross-core FWD)
> > cache-line copy was found."
> >
> > I don't see anything about this in arch/x86/events/intel/ds.c - if I
> > understand correctly, the kernel's PEBS data source decoding assumes
> > that 0x07 means "L3 hit, snoop hitm" on these CPUs. I think this needs
> > to be adjusted somehow - and maybe it just isn't possible to actually
> > distinguish between HitM and cross-core FWD in PEBS events on these
> > CPUs (without big-hammer chicken bit trickery)? Maybe someone from
> > Intel can clarify?
> >
> > (The SDM describes that E-cores on the newer 12th Gen have more
> > precise PEBS encodings that distinguish between "L3 HITM" and "L3
> > HITF"; but I guess the P-cores there maybe still don't let you
> > distinguish HITM/HITF?)
> >
> >
> > I think https://perfmon-events.intel.com/tigerLake.html is also
> > outdated, or at least it uses ambiguous grammar: The
> > MEM_LOAD_L3_HIT_RETIRED.XSNP_FWD event (EventSel=D2H UMask=04H) is
> > documented as "Counts retired load instructions where a cross-core
> > snoop hit in another cores caches on this socket, the data was
> > forwarded back to the requesting core as the data was modified
> > (SNOOP_HITM) or the L3 did not have the data(SNOOP_HIT_WITH_FWD)" -
> > from what I understand, a "cross-core FWD" should be a case where the
> > L3 does have the data, unless L3 has become non-inclusive on Ice Lake?
> >
> > On a Tiger Lake CPU, I can see this event trigger for the
> > sys_call_table, which is located in the rodata region and probably
> > shouldn't be containing Modified cache lines:
> >
> > # grep -A1 -w sys_call_table /proc/kallsyms
> > ffffffff82800280 D sys_call_table
> > ffffffff82801100 d vdso_mapping
> > # perf record -e mem_load_l3_hit_retired.xsnp_fwd:ppp --all-kernel -c 100 --data
> > ^C[ perf record: Woken up 11 times to write data ]
> > [ perf record: Captured and wrote 22.851 MB perf.data (43176 samples) ]
> > # perf script -F event,ip,sym,addr | egrep --color 'ffffffff828002[89abcdef]'
> > mem_load_l3_hit_retired.xsnp_fwd:ppp: ffffffff82800280
> > ffffffff82526275 do_syscall_64
> > mem_load_l3_hit_retired.xsnp_fwd:ppp: ffffffff828002d8
> > ffffffff82526275 do_syscall_64
> > mem_load_l3_hit_retired.xsnp_fwd:ppp: ffffffff82800280
> > ffffffff82526275 do_syscall_64
> > mem_load_l3_hit_retired.xsnp_fwd:ppp: ffffffff828002b8
> > ffffffff82526275 do_syscall_64
> > mem_load_l3_hit_retired.xsnp_fwd:ppp: ffffffff828002b8
> > ffffffff82526275 do_syscall_64
> > mem_load_l3_hit_retired.xsnp_fwd:ppp: ffffffff828002b8
> > ffffffff82526275 do_syscall_64
> > mem_load_l3_hit_retired.xsnp_fwd:ppp: ffffffff82800280
> > ffffffff82526275 do_syscall_64
> > mem_load_l3_hit_retired.xsnp_fwd:ppp: ffffffff82800288
> > ffffffff82526275 do_syscall_64
> > mem_load_l3_hit_retired.xsnp_fwd:ppp: ffffffff828002b8
> > ffffffff82526275 do_syscall_64
> >
> >
> > (For what it's worth, there is a thread on LKML where "cross-core FWD"
> > got mentioned: <https://lore.kernel.org/lkml/b4aaf1ed-124d-1339-3e99-a120f6cc4d28@linux.intel.com/>)
> 
> +others better qualified than me to respond.
> 
> Hi Jann,
> 
> I'm not overly familiar with the issue, but it appears a similar issue
> has been reported for Broadwell Xeon here:
> https://community.intel.com/t5/Software-Tuning-Performance/Broadwell-Xeon-perf-c2c-showing-remote-HITM-but-remote-socket-is/td-p/1172120
> I'm not sure that thread will be particularly useful, but having the
> Intel people better qualified than me to answer is probably the better
> service of this email.
> 
> Thanks,
> Ian