Received-SPF: pass (google.com: domain of linux-kernel+bounces-85616-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
From: Tony Luck <tony.luck@intel.com>
To: Fenghua Yu <fenghua.yu@intel.com>,
	Reinette Chatre <reinette.chatre@intel.com>,
	Peter Newman <peternewman@google.com>,
	Jonathan Corbet <corbet@lwn.net>,
	Shuah Khan <skhan@linuxfoundation.org>,
	x86@kernel.org
Cc: Shaopeng Tan <tan.shaopeng@fujitsu.com>,
	James Morse <james.morse@arm.com>,
	Jamie Iles <quic_jiles@quicinc.com>,
	Babu Moger <babu.moger@amd.com>,
	Randy Dunlap <rdunlap@infradead.org>,
	Drew Fustini <dfustini@baylibre.com>,
	linux-kernel@vger.kernel.org,
	linux-doc@vger.kernel.org,
	patches@lists.linux.dev,
	Tony Luck <tony.luck@intel.com>
Subject: [PATCH v15 0/8] Add support for Sub-NUMA cluster (SNC) systems
Date: Wed, 28 Feb 2024 11:36:55 -0800
Message-ID: <20240228112935.8087-tony.luck@intel.com>
In-Reply-To: <20240228112215.8044-tony.luck@intel.com>
References: <20240228112215.8044-tony.luck@intel.com>
Precedence: bulk
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit

The Sub-NUMA cluster feature on some Intel processors partitions the CPUs
that share an L3 cache into two or more sets. This plays havoc with the
Resource Director Technology (RDT) monitoring features.  Prior to this
patch Intel has advised that SNC and RDT are incompatible.

Some of these CPU support an MSR that can partition the RMID counters in
the same way. This allows monitoring features to be used. With the caveat
that users must be aware that Linux may migrate tasks more frequently
between SNC nodes than between "regular" NUMA nodes, so reading counters
from all SNC nodes may be needed to get a complete picture of activity
for tasks.

Cache and memory bandwidth allocation features continue to operate at
the scope of the L3 cache.

Signed-off-by: Tony Luck <tony.luck@intel.com>

---
Changes since v14: https://lore.kernel.org/all/20240126223837.21835-1-tony.luck@intel.com/

1) Rebase to TIP x86/cache + my 2-patch cleanup

2) Dropped all Reviewed/Tested tags (enough changed in TIP that this
needs looking at, and testing, again).

3) Added ATOM_CRESTMONT_X (Sierra Forest) to list of SNC supporting CPUs.

4) Added a console "INFO" message when SNC detected:
	pr_info("Sub-NUMA cluster detected with %d nodes per L3 cache\n", ret);

Note that my alternate approach posted as v15-RFC
https://lore.kernel.org/all/20240130222034.37181-1-tony.luck@intel.com/
has been abandoned. I'd still like to explore splitting the L3
rdt_resource into separate control/monitor elements, but that
can wait for another series of patches.

Tony Luck (8):
  x86/resctrl: Prepare for new domain scope
  x86/resctrl: Prepare to split rdt_domain structure
  x86/resctrl: Prepare for different scope for control/monitor
    operations
  x86/resctrl: Split the rdt_domain and rdt_hw_domain structures
  x86/resctrl: Add node-scope to the options for feature scope
  x86/resctrl: Introduce snc_nodes_per_l3_cache
  x86/resctrl: Sub NUMA Cluster detection and enable
  x86/resctrl: Update documentation with Sub-NUMA cluster changes

 Documentation/arch/x86/resctrl.rst        |  25 +-
 include/linux/resctrl.h                   |  85 +++--
 arch/x86/include/asm/msr-index.h          |   1 +
 arch/x86/kernel/cpu/resctrl/internal.h    |  67 ++--
 arch/x86/kernel/cpu/resctrl/core.c        | 428 ++++++++++++++++++----
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c |  56 +--
 arch/x86/kernel/cpu/resctrl/monitor.c     |  70 ++--
 arch/x86/kernel/cpu/resctrl/pseudo_lock.c |  26 +-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c    | 156 ++++----
 9 files changed, 642 insertions(+), 272 deletions(-)

-- 
2.43.0