Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 18 May 2023 17:40:31 -0700
From:   Dan Williams <dan.j.williams@intel.com>
To:     Alison Schofield <alison.schofield@intel.com>,
        Dave Hansen <dave.hansen@intel.com>
CC:     "Rafael J. Wysocki" <rafael@kernel.org>,
        Len Brown <lenb@kernel.org>,
        "Dan Williams" <dan.j.williams@intel.com>,
        Thomas Gleixner <tglx@linutronix.de>,
        Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
        Dave Hansen <dave.hansen@linux.intel.com>,
        "H. Peter Anvin" <hpa@zytor.com>,
        "Andy Lutomirski" <luto@kernel.org>,
        Peter Zijlstra <peterz@infradead.org>,
        "Andrew Morton" <akpm@linux-foundation.org>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Dave Jiang <dave.jiang@intel.com>, <x86@kernel.org>,
        <linux-cxl@vger.kernel.org>, <linux-acpi@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 1/2] x86/numa: Introduce numa_fill_memblks()
Message-ID: <6466c57f7c2ee_682c1294d3@dwillia2-xfh.jf.intel.com.notmuch>
References: <cover.1684448934.git.alison.schofield@intel.com>
 <e365f4dfa7fa974118eb4e59aebc7cc423cf19a1.1684448934.git.alison.schofield@intel.com>
 <8dc725c8-613a-b51b-6cc1-80d2275ca130@intel.com>
 <ZGbCG4gRGow0VCmQ@aschofie-mobl2>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <ZGbCG4gRGow0VCmQ@aschofie-mobl2>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?7BE1gWuYdEmm4/VefZngEAstG9XkGJGDknscluq9k9LFPzdXI46LSXmyGxxe?=
 =?us-ascii?Q?tF5iNt8m1ZT4j/hE3+UygLcf7pDQHCf3E3Rpow4f9shhFXiIq4Az6HWgsOQt?=
 =?us-ascii?Q?oPPvM3/IB34bXnTz0IryDaa/pcnI0uES9jkRJ2ghwnZS5pRuAwL79P36gWDy?=
 =?us-ascii?Q?fhKTtTTxC8AbFzkRpC1XVybm3ViIvjZDVit0JY+WFuxz2YZ9L5UG1PNYOI/H?=
 =?us-ascii?Q?YxRsnUvyvTdLV6wIIQ+OqVdxSilZWMDshqWfttY99PzAdZGmMBqGTgqcC53m?=
 =?us-ascii?Q?KZhV6FUOj9B2Pr85wo0eVUUzfAbDEZFt9uSgM1/2j8zye0rIjcGF7C4op0vn?=
 =?us-ascii?Q?lBNHzujQzuLJBTq6Q4gqQc2wD5FQRIM0Y3QW/oLyPZMIun7jt5K4hp99i7sx?=
 =?us-ascii?Q?zZ+/Voa0a8TyhEoi4GWcWH8Sf1ORsmyJE0o1s5klDsfSbJePLuMinEostwb5?=
 =?us-ascii?Q?z2LJ2tXYFM3SkfxIL//FU+hsw0hjG1WKF+kKRKONoN0lB/bfcY8cutAHfU2r?=
 =?us-ascii?Q?VVdf5YXFZICv+HxsZD7S9nhbqFfT+9ejw1smild1/WOLAEqTOvsYk/8zSrGm?=
 =?us-ascii?Q?2EEBube+Xj1o0AW5j4N1fY6sSzWpHxnagY3p50l+TtkMHSBeXYRuTQFdMhnQ?=
 =?us-ascii?Q?5mhI9bRaQLhZHMGtFTY5rvP1JyD/L3JyqHn1JiMChoZlXLNhzyhmYDkMH1TU?=
 =?us-ascii?Q?oflnLolmUgbe/h6cxJZQCj/VDCB87YYcyyhIIwduqD0boyvTXdQWVmMkaBol?=
 =?us-ascii?Q?xkgF2psvWYPe43Uon8Y+zNRrzdSGmtYDVMmvdn05tZjOnig0J9Lv8e5epqGJ?=
 =?us-ascii?Q?lziAJndJnru4yveTzjUsG+8ni8BQVeW/wj4+eW67FsSaBo+fsZ8guwdKWyaL?=
 =?us-ascii?Q?9eS0m86TA1nKEBJv90IRzvgwwIlauexSdkgNR5q9swNuGpaTas/KHUwJq5mP?=
 =?us-ascii?Q?0AodANFdWpCLfhOMH8xqAXOr2VCU7k4KA7gJlnF8BwgPromT4gybbJD7jZXI?=
 =?us-ascii?Q?3nDgkk+RaB0Y3/m0q4NnZt9KNkL/EjPbJW+Td10XoSC7PZjpA3+DPR9kUcJC?=
 =?us-ascii?Q?3fGRMlIZ+BDkaYhfiWWHGBKkbfmhtIi8cGFqzjLHXQ4d7V7vYQH7Gz2sEWz5?=
 =?us-ascii?Q?LDWfnUFpnMADmmROZiwyTgMx9yaawgM/rvaNlLPeaXHqAZscbTssVjx89QmF?=
 =?us-ascii?Q?rA1Zb+AXSg98qfbDCT12ALnd54hTBIiGcYuj0uq9EdyD8WqHFPdK/0bFckTv?=
 =?us-ascii?Q?qmptkJDW3Q/B3OfWm+sQ8PJrYlX0tex5Oxr+pIXS55zjEassbWArETQr5nxn?=
 =?us-ascii?Q?1gQ6I9K6pqna/RfSFI7Drkh4UJ7JvIHc0tfp7rocIfrRHb1gyi5FchdUOFhP?=
 =?us-ascii?Q?p7Art1n99fRXwUHXEINJyX/afNnmHOmNpp6zyX9B41ZsZZvetC3kvRi6l/1b?=
 =?us-ascii?Q?/d3YTVHJ0NH/mf/EtE+ZGj3k+pA4rVILx6MSBCdXUCrXfT6QHQ/p3LhAWIdw?=
 =?us-ascii?Q?LXgCdteHjajstMU9WA4fkZKBubJX6zhphJ/0kCA2GloHJjIqEvAhwkFFCTgW?=
 =?us-ascii?Q?Y0iCbUCCZtcOjCt1lEpaYXizYZosACS/n5MzEL8+EI2r0qx8b81hEzqKp/3X?=
 =?us-ascii?Q?/g=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 0cc1e67c-8d99-47f4-0bad-08db5801a7fc
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB8107.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 May 2023 00:40:34.2528
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: pCKlUAvpfw2+64DdsZ3yW3zYdaIYuAVuqkkV0l9IbZvxQ7h+vKyyPM02YTdq1/VbBDf1cdFDtIvrNPxl0Zez5F0uG1CIjIXgIIB9wmFK4Ko=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR11MB8143
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Alison Schofield wrote:
> On Thu, May 18, 2023 at 05:08:16PM -0700, Dave Hansen wrote:
> > On 5/18/23 17:04, alison.schofield@intel.com wrote:
> > > The initial use case is the ACPI driver that needs to extend
> > > SRAT defined proximity domains to an entire CXL CFMWS Window[1].
> > 
> > Dumb question time: Why didn't the SRAT just cover this sucker in the
> > first place?  Are we fixing up a BIOS bug or is there a legitimate
> > reason that the SRAT didn't cover it up front?
> > 
> >
> 
> There is no requirement that the BIOS describe (in the SRAT) all the
> HPA assigned to a CFMWS Window. The HPA range may not actually map to
> any memory at boot time. It can be persistent capacity or may be there
> to enable hot-plug. IIUC BIOS can pick and choose and define volatile
> regions wherever it pleases.
> 
> So, no we're not fixing up a BIOS bug, nor doing a BIOS sanity check.
> 

Another way to think about it is that CXL is dynamic and SRAT is static.
ACPI hotplug assumes you're just onlining an address range that was
declared offline in SRAT. CXL hotplug allows various capacities and
performance types to be added. So, how many potential proximity domains
performance targets could exist in a given CXL window? It depends, and
because it depends BIOS goes hands off and lets the OS define that
policy.

The Linux policy for now is just keep it simple. Add a proximity domain
for every unmapped (no SRAT intersections) CXL Window, and put the onus
on the platform owner to assign devices of similar performance to a
given CXL window to keep the proximity domain proliferation to a
minimum.