Received-SPF: pass (google.com: domain of linux-kernel+bounces-64138-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Date: Tue, 13 Feb 2024 20:09:37 +0100
From: Robert Richter <rrichter@amd.com>
To: Dan Williams <dan.j.williams@intel.com>
Cc: Davidlohr Bueso <dave@stgolabs.net>,
	Jonathan Cameron <jonathan.cameron@huawei.com>,
	Dave Jiang <dave.jiang@intel.com>,
	Alison Schofield <alison.schofield@intel.com>,
	Vishal Verma <vishal.l.verma@intel.com>,
	Ira Weiny <ira.weiny@intel.com>, linux-cxl@vger.kernel.org,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH] cxl/pci: Fix disabling CXL memory for zero-based
 addressing
Message-ID: <Zcu-cVG9deqfwdiV@rric.localdomain>
References: <20240209193451.163564-1-rrichter@amd.com>
 <65c68969903b1_afa429460@dwillia2-xfh.jf.intel.com.notmuch>
 <Zctg4Lx_y_hbChUW@rric.localdomain>
 <65cbb787e73c_29b129432@dwillia2-mobl3.amr.corp.intel.com.notmuch>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <65cbb787e73c_29b129432@dwillia2-mobl3.amr.corp.intel.com.notmuch>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?EoWwK0vH+8l7gkSZ6Bi+qsy4r6qz764lwvm61VkPzGRvKj3MvNdx6REuk55U?=
 =?us-ascii?Q?z4RjuhMziNk72d7J+RFqVAAj4IBvtx8unkX66+GYT0EULr2jyyJbZ0VE42m/?=
 =?us-ascii?Q?AnHM4gevHgWmXDvi6HPUh6TOA2UwOQqT7B/1nGDMUUjhyMXjqXMdbw6y38uP?=
 =?us-ascii?Q?RHlDhy2j/Cmn082/WfQbI5lsQ+Fz486lDTZQfgV8pUgeEmigt24iWMO2tDum?=
 =?us-ascii?Q?xIz0/1mZXrU2PmZAezr9uUwp+okmLA5A1aZAD5FKHeBI8g/Jr80SUIFZ7BHy?=
 =?us-ascii?Q?lMn8BxUokYCo/vmj1wDFIvvVr5IfYm663xI8KcXGhEW4cm76ESecOrAPZZn8?=
 =?us-ascii?Q?JNF7HRsTM51BOfhEBCzUN1lS7zf4KE2WOc+PX37El2CMwatEnz5iHsOSnevK?=
 =?us-ascii?Q?HoD75U3RrcSNUH2eiI3nLyAJzF97s3wdTdB+ntcvcajlFvJWZFlxU14OcONk?=
 =?us-ascii?Q?3zpgKaP7NJXwj6rlMXSbyBdWxVEn1mHl8ucWpeKHx8p+Q6NjAPDwwftvvboH?=
 =?us-ascii?Q?sjse1L6sD1imGGZ7yMd/6PUh+ljG5wiCwNe9XIVK6whEBCyFGUXdh6/PN5E2?=
 =?us-ascii?Q?3gcG6EL3VSQGPus0zPju7HZR4EanVsYRsIC5Xnng8b34IPMX/becr05JO7d0?=
 =?us-ascii?Q?iCkxugAmbgRMXqPzMw+YH4Jn8LWo26zBXnQ9TR27FbShZFOa4g5UV2om5EjY?=
 =?us-ascii?Q?1qHNgGTbZDlSWrEZriK9Phghyszz7vDuV/7DUThKg65ZlIiL/2XORyvvuPKf?=
 =?us-ascii?Q?JSUi7OCN6K2Vf5O/bRR8YszNqOL1qYFmdIrNHDWR++sFjGzWDswFs1o1afrU?=
 =?us-ascii?Q?IVUwYSXRWa8uI5Kahj58aCscE2Bn9SEZ9M8tkO2iJedc683lR/FcfdzA58e/?=
 =?us-ascii?Q?7+wHIC4r+gmw+PUsLcpAvtRhgXLosohFlV652wZMa66QkWLcqep5XfNUnhM3?=
 =?us-ascii?Q?dqlOM6L+IfucWNnI9Ia55eBSbjC7zw+kpwqGB1M76i4JJgk0+8ke7xb9tKuM?=
 =?us-ascii?Q?4Ar/0IbAKl/Eaqz+pZwizTpjEANTsnLDjj0CE/JOeBkoGM9KJeegS4QyHUFT?=
 =?us-ascii?Q?mCI4J75E4JOyek7f/2yarSwD49v5S+1wISNhKdPgf4GQVLtjEkHUTz+i+JkE?=
 =?us-ascii?Q?dmXBYWOGNPqhDIWeqx9DmgiPjT6FPoj3mnTcEQ81UovXXd4E/mQ/QMQ0fOH7?=
 =?us-ascii?Q?NH/p5vY0d28/vNPxzQ79Cc7zO9dSRpy9eWwvSPxRdyvdoT9rnYDmpCNsO8Qq?=
 =?us-ascii?Q?B9pYq1wwXtWTP6St1XBFrulUSLnC3w55w+u2BPS18hc0FOcWCSYUBy0i9XvU?=
 =?us-ascii?Q?zQqAeqv+6QaOiWIeHFBOVxVR+bA38CjS74Cjc3ICjgXY+awwFTkWVGeYRHoK?=
 =?us-ascii?Q?tR/l29jeGBh52JY1mSrA/Hmu9SrJFiXi/RQCHQgoshXj4C7s3WA/IzHjdwpZ?=
 =?us-ascii?Q?kHarRVHWp8IlbvwpQdZHhQx341rGR4+tzDRwkO34rS+r4n3H7al7gFzbRzKd?=
 =?us-ascii?Q?TGAQLiEs1q0+FXiL7vRJtem/mLXs9AY4v+jeM63cRVfL3Ag2ROYNo08boqI7?=
 =?us-ascii?Q?kCWRt0zs+SYBNAm8jKIJIKHBqrNgC/L9Bz7e1ajk?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 15c75b9e-0e2b-4be4-ecaf-08dc2cc7559b
X-MS-Exchange-CrossTenant-AuthSource: CYYPR12MB8750.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Feb 2024 19:09:42.7231
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: DsEf+yvIBbAEdsHio9LtuJOTYwv8U7ap+ZPgYDT3IocoPdTUXcr7nirKNJo3dyfZ1sSFGPRAnO1hkepzPg5drg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8935

On 13.02.24 10:40:07, Dan Williams wrote:
> Robert Richter wrote:
> > Dan,
> > 
> > On 09.02.24 12:22:01, Dan Williams wrote:
> > > Robert Richter wrote:
> > 
> > > > diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c
> > > > index 569354a5536f..3a36a2f0c94f 100644
> > > > --- a/drivers/cxl/core/pci.c
> > > > +++ b/drivers/cxl/core/pci.c
> > > > @@ -466,6 +466,18 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm,
> > > >  	for (i = 0, allowed = 0; info->mem_enabled && i < info->ranges; i++) {
> > > >  		struct device *cxld_dev;
> > > >  
> > > > +		/*
> > > > +		 * Handle zero-based hardware addresses
> > > > +		 */
> > > > +		if (!info->dvsec_range[i].start &&
> > > > +		    info->dvsec_range[i].end != CXL_RESOURCE_NONE &&
> > > > +		    info->dvsec_range[i].end) {
> > > > +			dev_dbg(dev, "Zero-based hardware range found [%#llx - %#llx]\n",
> > > > +				info->dvsec_range[i].start, info->dvsec_range[i].end);
> > > > +			allowed++;
> > > > +			continue;
> > > > +		}
> > > > +
> > > 
> > > I am not comfortable with this. It should be checking a platform
> > > specific quirk, or similar for the possibility of HPA != SPA. The
> > > entirety of the Linux CXL subsystem is built on the assumption that HPA
> > > == SPA, and if a platform wants to inject an offset between those Linux
> > > needs some way to enumerate that it is running in that new world. Yes,
> > > nothing in the CXL specification precludes HPA != SPA, but Linux has
> > > long since shipped the opposite assumption.
> > 
> > this check prevents the memory from disabling an enabled decoder. So it
> > just keeps everything as it comes out of firmware.
> > 
> > Can you explain the motivation why active memory is disabled?
> 
> It is a sanity check that Linux is operating in a CXL world that it
> understands. The fundamental assumption is that the CFMWS correctly
> conveys the CXL space, and that the HW decoder resources match platform
> expectations match Linux resource management.

It would be sane to just not use CXL if assumptions on it are not
valid and not to break system to boot.

> 
> > This may take system memory offline and could lead to a kernel hang.
> 
> Yes, that is not an unreasonable result when Linux fundamental
> assumptions are violated.

BUG_ON(fw_table_broken)? If at all, it is not mandatory to have a
CFMWS. Btw, the check is more strict and also checks memory
attributes. It is very likely something can break.

> 
> > The same could happen if the CEDT is broken or just missing (it is not
> > mandatory for 1.1). Such systems just die when booting. So the check
> > to take memory offline should be changed in a way that it will be safe
> > to disable it.
> > 
> > A platform check would fix that only for certain systems.
> 
> I am not worried about platforms that accidentally break the CEDT, those
> mistakes are typically caught pre-production. Otherwise, if the platform
> is designed to break assumptions then Linux needs explicit enabling for
> the address translation sitting between the endpoint and the platform
> address map.

As said, if assumptions break, just let cxl init fail.

Decoders should just be disabled if the address mapping is invalid.
There is no point to tear down the system.

A platform check is fine to me, but in the end this looks like a
whitelist to let systems boot.

Thanks,

-Robert