Received-SPF: pass (google.com: domain of linux-kernel+bounces-64088-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Date: Tue, 13 Feb 2024 10:40:07 -0800
From: Dan Williams <dan.j.williams@intel.com>
To: Robert Richter <rrichter@amd.com>, Dan Williams <dan.j.williams@intel.com>
CC: Davidlohr Bueso <dave@stgolabs.net>, Jonathan Cameron
	<jonathan.cameron@huawei.com>, Dave Jiang <dave.jiang@intel.com>, "Alison
 Schofield" <alison.schofield@intel.com>, Vishal Verma
	<vishal.l.verma@intel.com>, Ira Weiny <ira.weiny@intel.com>,
	<linux-cxl@vger.kernel.org>, <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] cxl/pci: Fix disabling CXL memory for zero-based
 addressing
Message-ID: <65cbb787e73c_29b129432@dwillia2-mobl3.amr.corp.intel.com.notmuch>
References: <20240209193451.163564-1-rrichter@amd.com>
 <65c68969903b1_afa429460@dwillia2-xfh.jf.intel.com.notmuch>
 <Zctg4Lx_y_hbChUW@rric.localdomain>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <Zctg4Lx_y_hbChUW@rric.localdomain>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?/RXA7vIpKVKaJlHsLcKMQnwYlRf+v9UxAjoq7c5JHKgyi/oHmNtlDxnmtQNo?=
 =?us-ascii?Q?kYK8XXzJFPrsQEztDKu/HAPDrZJIGOox8utRj9AW9EdT5WfiiOzRiciPU441?=
 =?us-ascii?Q?C7u/pHFxLjFu0M41Suk2i9h+mBC1bBixHImp05aj9AtQXhuDmssfUQ/6dZ3/?=
 =?us-ascii?Q?qPF3KTLuwv1WMdaxZCVIzzZ1rpm8RmbRw/xVM1folbh4eKzVXLQSfxFiZpAi?=
 =?us-ascii?Q?+BuJTwZ/I9aRZSDFkwZZVrT92Feu5rG9hR2xz7sRUkfLWih004wdSQtpJ/Hi?=
 =?us-ascii?Q?Sq7JesSxl1KvfB2sGkaLSOIKU+CnCNZ7VA5sJLVVu1yXmLy/W00nJyjvqWQe?=
 =?us-ascii?Q?XBx5kxdh6WUVdUMNeCzFqKQvnOlGiGxg2o76/7Y+RVrZ0FAhZDrnMhFyC+xp?=
 =?us-ascii?Q?BICYTxRTae0659z3/6mZ7+UrbvZgqORwrySujMLuajKqLmfKVuaRcZmFdiOZ?=
 =?us-ascii?Q?2koDd2oJ/wy62lpE6N9guEerEvX6mhIKeN38vyNoOZzH9OxX+5R4xUZXOuhH?=
 =?us-ascii?Q?ViFBAG8EzpqZkBUf4vJgpZ3cRWKsTNFy6MZfD8BatHDrqOp2OjYVNO7rkpMB?=
 =?us-ascii?Q?0BHKq/Appdh2efRGtnxkwUMMBkzE8xIP+ICuqzMHdDu0dWhRQc6OpPkkjGXD?=
 =?us-ascii?Q?/5gGQ9xohf2DXaOAO2y4UG9VwJpaMQ5RBI738g+2CaDe/91hcxniCllOAju+?=
 =?us-ascii?Q?3LHQUvOILStLvVttKInPbfjqu6QjMwbpE/tbH2bpyvePhfLtqRJ3XgqbsALn?=
 =?us-ascii?Q?K3JIyPpym806e9560W2C7NXSuzirF3t3zdAKPlyk/xxdtoI5ITQXCunhQAnz?=
 =?us-ascii?Q?FVhV0kz7SGgnX/41MjHmQQ9KBDbPnAatA9gHw3KptLv21qg2dzvR9n9GWyQ3?=
 =?us-ascii?Q?Dct/NVo+dSUmgOZWnkm+COt5C6m+bsq3IQJAWKP+5iEVS3kJM5liZXzP8x0l?=
 =?us-ascii?Q?KDY97ANh9pt0AUYeE5HCvne9+HiUF4k1X/eT/m5SCfCAMR+KnT8ESLLvGw+F?=
 =?us-ascii?Q?EuZx6WJ/RR0KxHMS9ANbM5qRViS4qyAIrqosLG1DJng2Sj1IIY9a1jqAVUkk?=
 =?us-ascii?Q?lMoaLp6zvVhMITBXkNYh2cpZ291CxkBzael0kd0Suv/Qro91Utq/hh1d0f/Z?=
 =?us-ascii?Q?l1MOoHwTfE9LYdtOGxYr2m1t04r801+NNmxndg99J6/hhehyHdeEXn9hV2cw?=
 =?us-ascii?Q?UDlDTwHSYD5yUSbWws4dwwdzXXjdyPrZx3MWCyn41PH4qdKmwCNzS073tnnl?=
 =?us-ascii?Q?SNVWL6Xl6SfMwln2OTXo2SJvmQl6HEj3NSezgvTAMq0d7xiEznbTWsPcSGfK?=
 =?us-ascii?Q?oITfYY5RoVse19Szy5yoJS52kUs0hvVLi29hXdT/ywNfTaHRyWEYP0c/zfeK?=
 =?us-ascii?Q?jnp0Aksp0FK0TJlEXv852RATlItULlWbz9WRrHDmRIIlmCgb3EsAfw/VGwVY?=
 =?us-ascii?Q?HOc8O2703LNPxembTQNCslrRJEyVkz/9Wr8t6pX97zAPO2BDmiFJYUHUKmWP?=
 =?us-ascii?Q?5QNR77vOISoFg8pkfKRzIwVn/xEcVe4Kf/q33jDsDKEBrHAsRtIr1AeBWbqa?=
 =?us-ascii?Q?tUXA2EA36VCsY1T9oAmBomHRehafNdxuwTWeAKECQ2TbLG9TtUUZG7tBA/hB?=
 =?us-ascii?Q?jQ=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: a7875f43-e77b-4ec3-a32f-08dc2cc3349e
X-MS-Exchange-CrossTenant-AuthSource: PH8PR11MB8107.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Feb 2024 18:40:09.4239
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: FBA52wf0hB5Djpch9Ms5MlPkNITPDA2NEwmcIRbJMkfyWhJq1vVRrvnZTTJoRTQbuYa4WW4C8CEHA8VVpeNRxWwa/w447u/B3azsJ14Bqlk=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ2PR11MB8588
X-OriginatorOrg: intel.com

Robert Richter wrote:
> Dan,
> 
> On 09.02.24 12:22:01, Dan Williams wrote:
> > Robert Richter wrote:
> 
> > > diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c
> > > index 569354a5536f..3a36a2f0c94f 100644
> > > --- a/drivers/cxl/core/pci.c
> > > +++ b/drivers/cxl/core/pci.c
> > > @@ -466,6 +466,18 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm,
> > >  	for (i = 0, allowed = 0; info->mem_enabled && i < info->ranges; i++) {
> > >  		struct device *cxld_dev;
> > >  
> > > +		/*
> > > +		 * Handle zero-based hardware addresses
> > > +		 */
> > > +		if (!info->dvsec_range[i].start &&
> > > +		    info->dvsec_range[i].end != CXL_RESOURCE_NONE &&
> > > +		    info->dvsec_range[i].end) {
> > > +			dev_dbg(dev, "Zero-based hardware range found [%#llx - %#llx]\n",
> > > +				info->dvsec_range[i].start, info->dvsec_range[i].end);
> > > +			allowed++;
> > > +			continue;
> > > +		}
> > > +
> > 
> > I am not comfortable with this. It should be checking a platform
> > specific quirk, or similar for the possibility of HPA != SPA. The
> > entirety of the Linux CXL subsystem is built on the assumption that HPA
> > == SPA, and if a platform wants to inject an offset between those Linux
> > needs some way to enumerate that it is running in that new world. Yes,
> > nothing in the CXL specification precludes HPA != SPA, but Linux has
> > long since shipped the opposite assumption.
> 
> this check prevents the memory from disabling an enabled decoder. So it
> just keeps everything as it comes out of firmware.
> 
> Can you explain the motivation why active memory is disabled?

It is a sanity check that Linux is operating in a CXL world that it
understands. The fundamental assumption is that the CFMWS correctly
conveys the CXL space, and that the HW decoder resources match platform
expectations match Linux resource management.

> This may take system memory offline and could lead to a kernel hang.

Yes, that is not an unreasonable result when Linux fundamental
assumptions are violated.

> The same could happen if the CEDT is broken or just missing (it is not
> mandatory for 1.1). Such systems just die when booting. So the check
> to take memory offline should be changed in a way that it will be safe
> to disable it.
> 
> A platform check would fix that only for certain systems.

I am not worried about platforms that accidentally break the CEDT, those
mistakes are typically caught pre-production. Otherwise, if the platform
is designed to break assumptions then Linux needs explicit enabling for
the address translation sitting between the endpoint and the platform
address map.