Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2;
Date:   Tue, 12 Sep 2023 09:01:17 -0700
From:   Tony Luck <tony.luck@intel.com>
To:     Reinette Chatre <reinette.chatre@intel.com>
Cc:     Fenghua Yu <fenghua.yu@intel.com>,
        Peter Newman <peternewman@google.com>,
        Jonathan Corbet <corbet@lwn.net>,
        Shuah Khan <skhan@linuxfoundation.org>, x86@kernel.org,
        Shaopeng Tan <tan.shaopeng@fujitsu.com>,
        James Morse <james.morse@arm.com>,
        Jamie Iles <quic_jiles@quicinc.com>,
        Babu Moger <babu.moger@amd.com>,
        Randy Dunlap <rdunlap@infradead.org>,
        linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org,
        patches@lists.linux.dev
Subject: Re: [PATCH v5 0/8] Add support for Sub-NUMA cluster (SNC) systems
Message-ID: <ZQCLTS0XAs/H2min@agluck-desk3>
References: <20230722190740.326190-1-tony.luck@intel.com>
 <20230829234426.64421-1-tony.luck@intel.com>
 <45e81506-3c43-95ba-56a4-38e1bb2e42dc@intel.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <45e81506-3c43-95ba-56a4-38e1bb2e42dc@intel.com>
Precedence: bulk

On Mon, Sep 11, 2023 at 01:23:35PM -0700, Reinette Chatre wrote:
> Hi Tony,
> 
> On 8/29/2023 4:44 PM, Tony Luck wrote:
> > The Sub-NUMA cluster feature on some Intel processors partitions
> > the CPUs that share an L3 cache into two or more sets. This plays
> > havoc with the Resource Director Technology (RDT) monitoring features.
> > Prior to this patch Intel has advised that SNC and RDT are incompatible.
> > 
> > Some of these CPU support an MSR that can partition the RMID
> > counters in the same way. This allows for monitoring features
> > to be used (with the caveat that memory accesses between different
> > SNC NUMA nodes may still not be counted accuratlely.
> 
> Same typo as in V4.

Sorry. Will fix and re-post.

> > 
> > Note that this patch series improves resctrl reporting considerably
> > on systems with SNC enabled, but there will still be some anomalies
> > for processes accessing memory from other sub-NUMA nodes.
> 
> I have the same question as with V4 that was not answered in that email
> thread nor in this new version.
> https://lore.kernel.org/lkml/e350514e-76ed-14ea-3e74-c0852658182f@intel.com/

Non-SNC systems already have an issue when reporting memory bandwidth
for a task that Linux may migrate the task to a CPU on a different node
which means that logging for that task will also move to different files
in the mon_data/mon_L3_*/ for the new node.

With SNC enabled, migration between NUMA nodes on the same socket may happen
much more frequently because:
1) The CPUs on the other NUMA nodes in the socket are in the same Linux
   L3 cache domain. So Linux regard the migration as "cheap".
2) The ACPI SLIT table on SNC enabled systems may also report the
   latency for remote access to another NUMA node on the same socket
   as significantly lower than the latency for cross-socket access. On
   my test system the SLIT distance for same socket nodes is 0xC,
   compared to 0x15 for cross-socket distance. This will also lead
   to Linux being more likely to migrate a task to a CPU on another
   SNC NUMA node in the same socket.

To avoid migration issues, users may use sched_setaffinity(2) to bind
tasks to the subset of CPUs that share an SNC NUMA node.

I can write this up in a new cover letter.

> I stop my review of this series here.

Reinette

Should I repost the whole series as v6 with the new cover letter. The
only change to the patches so far is to the selftest reported by
Shaopeng Tan[1].

-Tony

[1] https://lore.kernel.org/all/TYAPR01MB633033C489AAC0E514CBC6688BEEA@TYAPR01MB6330.jpnprd01.prod.outlook.com/