Received-SPF: pass (google.com: domain of linux-kernel+bounces-64188-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Date: Tue, 13 Feb 2024 11:45:48 -0800
From: Dan Williams <dan.j.williams@intel.com>
To: Robert Richter <rrichter@amd.com>, Dan Williams <dan.j.williams@intel.com>
CC: Davidlohr Bueso <dave@stgolabs.net>, Jonathan Cameron
	<jonathan.cameron@huawei.com>, Dave Jiang <dave.jiang@intel.com>, "Alison
 Schofield" <alison.schofield@intel.com>, Vishal Verma
	<vishal.l.verma@intel.com>, Ira Weiny <ira.weiny@intel.com>,
	<linux-cxl@vger.kernel.org>, <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] cxl/pci: Fix disabling CXL memory for zero-based
 addressing
Message-ID: <65cbc6ec7a0f6_29b129456@dwillia2-mobl3.amr.corp.intel.com.notmuch>
References: <20240209193451.163564-1-rrichter@amd.com>
 <65c68969903b1_afa429460@dwillia2-xfh.jf.intel.com.notmuch>
 <Zctg4Lx_y_hbChUW@rric.localdomain>
 <65cbb787e73c_29b129432@dwillia2-mobl3.amr.corp.intel.com.notmuch>
 <Zcu-cVG9deqfwdiV@rric.localdomain>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <Zcu-cVG9deqfwdiV@rric.localdomain>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?KHAtliB6CX0188TJ6qaCy5HfdVl68i8sMv7DfPVNlu2ppX5Kvx29i4JoNmLe?=
 =?us-ascii?Q?m/ANiDWc3fWjGhIsTxl/kJQ5zoqdYGXEB82NtRHpyxWG2AgVJt3yCFBYZonU?=
 =?us-ascii?Q?zhfejerdNiUofIDdnEf3OdimcHlLzMjdkYkLUKJVamqbOc2g74SEpjuMbZ0b?=
 =?us-ascii?Q?V33qkzOLscnyFrZSZn3upPganeguiWPl4vVkW3ydZXmxgbs8Rkr8ZqxJq0xN?=
 =?us-ascii?Q?zZdcPiF3i19P5O4LvvTv5YUAfinHS3OAER4vYlGR3ZsKc9Xsh4EijK17+Tov?=
 =?us-ascii?Q?a7Mr9BY/c9Odyy5V5AJiDKcQ+XjLWyM33I+IuLfcKaFL89taxcGEO07Q8MGH?=
 =?us-ascii?Q?cSzyT3A54fEt5aLnREcdPndUqNLljqR9VFYYPJzL5Gq9soI8fmbMsARz/GEw?=
 =?us-ascii?Q?q90nBCDnqcGgLAMCp9SviVoJ4ga42HAwSnPKwW7+BjeRsW10d7NWXz1+e9yY?=
 =?us-ascii?Q?YB9FlODxnSRGyKj8ssyGdEtBmT/zjX1C9UHjZOWQKHiFGRrIJVi2AB/MpZCl?=
 =?us-ascii?Q?722943O9CyzMdTgBCHhiYOKQ1za06q3S9soIWjqz2suz3XKpy/G48XLd5W+y?=
 =?us-ascii?Q?lQDjIqzE79F+Y94NHyQQfKS4L7pfiC4KgpCrOYgCHa8CewuXSWS6PNw+Hwq9?=
 =?us-ascii?Q?O4D2Z20BXalSBWkk7x1L0DXMEEd6BDawyPqCFiIspgAFhuca0tJUcTjJ1j59?=
 =?us-ascii?Q?hGBC9wD+MyqlyLeJEM9x9g6gQsvKOn1vIGQjtSDtFHVeif5AYfDwqGon+uIw?=
 =?us-ascii?Q?MN/5H94qMydO5o+D2EfhMoZj12e8BpLmmuvmm4bACD7KQp1LuIh5BfHUEAQd?=
 =?us-ascii?Q?0ymuoaOekgFrQzNUACeFJ8Dk9ExFD/vpeKDIiG4cuZnEIpJiNBPlaEcTi+h9?=
 =?us-ascii?Q?CAb486+v7SByisLv7QKDVQRVinih4mH8UBJiuQXb7iteHA+ezn9aGDOV2wUJ?=
 =?us-ascii?Q?nXrOVeX2E/2gxlJSjgUe+pt7O3dTRax14a4RtQ44fbMlHXXIPJ0BGMGv0Jx3?=
 =?us-ascii?Q?hlwAlxWwm+4GnzVS+/xgGuJVc1h8dbjfK8Mh9zB+ryvZK0YnBm1LYltZvEbT?=
 =?us-ascii?Q?doXwGhNWFZYvXPFY4BmIalT76VXMlbwO7YI1zCbtsKhZ5g9x93XHQUGpDJyh?=
 =?us-ascii?Q?H4Ml//RQuo2VkESZ1sEWbGUON36SKuTYCBrovYe3c8aha0MQyov8yZta7hw1?=
 =?us-ascii?Q?Ur3r9B3qyAJ0CqIXpqaDfIxR4QC4/wLMi/vVLfFALAJQQDqE+WbW0vdS6Sa4?=
 =?us-ascii?Q?6uRRg6QZMK/cuCXOHVXXGqYR1YBUtBbcEbfU6nmJcFzsRlbbYwwDeZwDPXbb?=
 =?us-ascii?Q?+fowcVs/xF+S+3CzpP8Yz4m8qxIhEV14ylNVv5lQVvkpoBQphN7HtRKsVFs0?=
 =?us-ascii?Q?yRZGRCmTpXfyaxMlr5YxoWdz2G3wXR2C14FXg95XmHuUCzGCrPmY9CR55qza?=
 =?us-ascii?Q?EWrhCQcVJNRcapioSLRs35ux6qReBGwHF2uQjahlkhqfEjVnQOXD3UszkfQ8?=
 =?us-ascii?Q?uL32etZq17zLHj3gMT7p9ukLcNgBCzJzoxbIamI/IA9plDkDJjVi6Z8tKEy9?=
 =?us-ascii?Q?BTrCm1pN4EZaH54nmMJR4MjuNeNwhTW774ffsPe5YMV1wMqUdpibpnyympRP?=
 =?us-ascii?Q?eA=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: baaec6d5-3709-476a-c792-08dc2ccc61f2
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB8107.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Feb 2024 19:45:50.9769
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ukspEb/piYTF62dLCuw+dS47+U1Q46e8VS3QeoXwC9xk4lzSNNe1aXbtbougYpfY3uv0uqxLNQtbEmyPwNDJSewGKDTykjEfNXlKYYD+nYM=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR11MB8299
X-OriginatorOrg: intel.com

Robert Richter wrote:
> On 13.02.24 10:40:07, Dan Williams wrote:
> > Robert Richter wrote:
> > > Dan,
> > > 
> > > On 09.02.24 12:22:01, Dan Williams wrote:
> > > > Robert Richter wrote:
> > > 
> > > > > diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c
> > > > > index 569354a5536f..3a36a2f0c94f 100644
> > > > > --- a/drivers/cxl/core/pci.c
> > > > > +++ b/drivers/cxl/core/pci.c
> > > > > @@ -466,6 +466,18 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm,
> > > > >  	for (i = 0, allowed = 0; info->mem_enabled && i < info->ranges; i++) {
> > > > >  		struct device *cxld_dev;
> > > > >  
> > > > > +		/*
> > > > > +		 * Handle zero-based hardware addresses
> > > > > +		 */
> > > > > +		if (!info->dvsec_range[i].start &&
> > > > > +		    info->dvsec_range[i].end != CXL_RESOURCE_NONE &&
> > > > > +		    info->dvsec_range[i].end) {
> > > > > +			dev_dbg(dev, "Zero-based hardware range found [%#llx - %#llx]\n",
> > > > > +				info->dvsec_range[i].start, info->dvsec_range[i].end);
> > > > > +			allowed++;
> > > > > +			continue;
> > > > > +		}
> > > > > +
> > > > 
> > > > I am not comfortable with this. It should be checking a platform
> > > > specific quirk, or similar for the possibility of HPA != SPA. The
> > > > entirety of the Linux CXL subsystem is built on the assumption that HPA
> > > > == SPA, and if a platform wants to inject an offset between those Linux
> > > > needs some way to enumerate that it is running in that new world. Yes,
> > > > nothing in the CXL specification precludes HPA != SPA, but Linux has
> > > > long since shipped the opposite assumption.
> > > 
> > > this check prevents the memory from disabling an enabled decoder. So it
> > > just keeps everything as it comes out of firmware.
> > > 
> > > Can you explain the motivation why active memory is disabled?
> > 
> > It is a sanity check that Linux is operating in a CXL world that it
> > understands. The fundamental assumption is that the CFMWS correctly
> > conveys the CXL space, and that the HW decoder resources match platform
> > expectations match Linux resource management.
> 
> It would be sane to just not use CXL if assumptions on it are not
> valid and not to break system to boot.

I can get on board with that.

> 
> > 
> > > This may take system memory offline and could lead to a kernel hang.
> > 
> > Yes, that is not an unreasonable result when Linux fundamental
> > assumptions are violated.
> 
> BUG_ON(fw_table_broken)? If at all, it is not mandatory to have a
> CFMWS. Btw, the check is more strict and also checks memory
> attributes. It is very likely something can break.

Sure, I'll take a patch like this:

diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c
index 6c9c8d92f8f7..e4e5a917f1f4 100644
--- a/drivers/cxl/core/pci.c
+++ b/drivers/cxl/core/pci.c
@@ -477,10 +477,11 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm,
                allowed++;
        }
 
-       if (!allowed) {
-               cxl_set_mem_enable(cxlds, 0);
-               info->mem_enabled = 0;
-       }
+       WARN_TAINT(!allowed, TAINT_FIRMWARE_WORKAROUND,
+                  FW_BUG "%s: Range register decodes outside platform defined CXL ranges.",
+                  dev_name(dev));
+       if (!allowed)
+               return -ENXIO;
 
        /*
         * Per CXL 2.0 Section 8.1.3.8.3 and 8.1.3.8.4 DVSEC CXL Range 1 Base