Received-SPF: pass (google.com: domain of linux-kernel+bounces-70094-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Date: Sat, 17 Feb 2024 22:27:40 +0100
From: Robert Richter <rrichter@amd.com>
To: Dan Williams <dan.j.williams@intel.com>
Cc: Davidlohr Bueso <dave@stgolabs.net>,
	Jonathan Cameron <jonathan.cameron@huawei.com>,
	Dave Jiang <dave.jiang@intel.com>,
	Alison Schofield <alison.schofield@intel.com>,
	Vishal Verma <vishal.l.verma@intel.com>,
	Ira Weiny <ira.weiny@intel.com>, linux-cxl@vger.kernel.org,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH v2] cxl/pci: Fix disabling memory if DVSEC CXL Range does
 not match a CFMWS window
Message-ID: <ZdEkzBbm7aqcghAl@rric.localdomain>
References: <20240216160113.407141-1-rrichter@amd.com>
 <65d034a22e258_29b129484@dwillia2-mobl3.amr.corp.intel.com.notmuch>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <65d034a22e258_29b129484@dwillia2-mobl3.amr.corp.intel.com.notmuch>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?iHd6bzeExtYfKicYX9/81Aky0nvJyUbSiCl5UEy2o+bHEMgzXhev7wJsZKx4?=
 =?us-ascii?Q?YMaTZpw5sVz+aYmRQEYVU/8u8BNGyth/uqp7Z3nreAHPJuUZ24hwsR8dda90?=
 =?us-ascii?Q?pRZndKt47/fw8E45lVNvszXhslMN98KJoI9OK1xdiPSlUMS1lbeMFgZDFTay?=
 =?us-ascii?Q?M/7UIEqZn8cWeydb7AJ3q9v69YUm4VDSFg/t0WY/lZl0z/x90Pnnp36LfeBo?=
 =?us-ascii?Q?tE6IOud3jx2Mql3SXtUAzdJEDQfGAvoCJQOT1iblOTZS4i/56oQKqXSVNQVb?=
 =?us-ascii?Q?uNXGP2p6oAITxws5OZbdAH9V4A5pX9okzVWlKwvTAzPDYHcAflyX9S+H/49p?=
 =?us-ascii?Q?6O4qjQrFXdzVS9DpEdIdC+tYXGaJ1yMFbenLyUcPhbgMa/6BJFMsIn2Ffq17?=
 =?us-ascii?Q?ylhSFd2f9wTzw4oLE7qfsTmZsm7+3OBOelWwbfWTHWyYkMeOovCWSuBfvwLy?=
 =?us-ascii?Q?qMA7XhEwRjdSJk4BOAYQkZIQjEbUGaW/CxtH68NShRZBp5XMFc08D/REtrei?=
 =?us-ascii?Q?fW//tBzB/GbP9QXF24S+Do2kLgTF36/fOuF5NASh6vUs4bHlIZ693h7eR/iD?=
 =?us-ascii?Q?gqucEnjaoqei+UQYkaNL28LMD8N06kGqjuT8D11FhDYXD4iZejC1u9OyBibH?=
 =?us-ascii?Q?rq71HomtG60xyDFdO5/YD1IszL525KeKXsL3Q8p0XATwZ5UcKxPywC3/KW8k?=
 =?us-ascii?Q?VBW8QomqBBrF9XFstAFiy5cx04ImuUOtDfYzNEudrj55bz3EOOlzfnh0Eb7c?=
 =?us-ascii?Q?LzXSQZCbzjyjD9Rr1K39ukCzQ3wdhdLQAjMt2aNooL0AoVn8DwKqE/JBxrp2?=
 =?us-ascii?Q?RBVmEzokxcojEmwplH4eonoChAHjuGYybV80XAMH6TUZFzC/F8W/JoTXG5B9?=
 =?us-ascii?Q?ur0LFQNDj1xwk9ZbuGCPei8AOll4jXKgnORoxlpB2jZ68R9o/wjzd6rPrs6E?=
 =?us-ascii?Q?hu9wsHc+OngHS5QWixo+QQBbFIN1B5uf/rimrw6KxBNeoKZm0GfOsoO4btUG?=
 =?us-ascii?Q?K+QmynPX5XkXehp/hCudbFk1pvThs6ejD7wbkX711gjyS32YDQz/TmQTKU2q?=
 =?us-ascii?Q?hJG6Y4YL6IglJbKVy5FMnPmS5ailZfX6wj+QGycG13X0av+NHriTZoMBSa8b?=
 =?us-ascii?Q?xwqK8JLnEQMK6S0MCQELr3mr++9TR+PJfZpgAdZ96NCQsCLcxqxDpjiEjSs2?=
 =?us-ascii?Q?jLqtg4k7bSNhWbKoOI4IkC1PlW6p9AkHKD+WulPNZ7DE+2ZmGuY3OOgMb7iT?=
 =?us-ascii?Q?K/f7XLzqU3+bCG63POyIN6tvHqht+liv1n/FWkXTHnZX25pQ30vXMmMQzT0M?=
 =?us-ascii?Q?PM7yPnh/U5zeGp1COFAKPWuNmQAwI+VXq8jOAaYjLqw5uBhD1m5xSBwOSqq+?=
 =?us-ascii?Q?rr1EZ54Hh0MXGnFHMjiFAq7DxaOQcvP3VPZffCXGFz6+MK7mJKOGCelI1yvY?=
 =?us-ascii?Q?5Zg0zzLloqRDmDVDDmPZmRpfMnyKKwW1JXjrZM+/9Gr5c+0THnvpgbL72fdV?=
 =?us-ascii?Q?bMqgoXdpQJrZJoqjhQPJWNBkaX1AULfIXhiSrebMNfuPZW8FHVXWdMOfK0/b?=
 =?us-ascii?Q?tiezDv13QyK5oLpba9OTTClb86Y9GHRxHhpR7I8i?=
X-OriginatorOrg: amd.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 05da3396-9050-494a-7ce3-08dc2fff48a4
X-MS-Exchange-CrossTenant-AuthSource: CYYPR12MB8750.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Feb 2024 21:27:46.3023
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: LBn4HdZUKVFiMUoC9E/Frmn57unQoCkLKHwibXMoPsmZmpC9b5pxwRC2TnoKEDf8x+FMRQ4F1tozX8sZTkD5ug==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB5974

On 16.02.24 20:22:58, Dan Williams wrote:
> Robert Richter wrote:
> > The Linux CXL subsystem is built on the assumption that HPA == SPA.
> > That is, the host physical address (HPA) the HDM decoder registers are
> > programmed with are system physical addresses (SPA).
> > 
> > During HDM decoder setup, the DVSEC CXL range registers (cxl-3.1,
> > 8.1.3.8) are checked if the memory is enabled and the CXL range is in
> > a HPA window that is described in a CFMWS structure of the CXL host
> > bridge (cxl-3.1, 9.18.1.3).
> > 
> > Now, if the HPA is not an SPA, the CXL range does not match a CFMWS
> > window and the CXL memory range will be disabled then. The HDM decoder
> > stops working which causes system memory being disabled and further a
> > system hang during HDM decoder initialization, typically when a CXL
> > enabled kernel boots.
> > 
> > Prevent a system hang and do not disable the HDM decoder if the
> > decoder's CXL range is not found in a CFMWS window.
> > 
> > Note the change only fixes a hardware hang, but does not implement
> > HPA/SPA translation. Support for this can be added in a follow on
> > patch series.
> > 
> > Signed-off-by: Robert Richter <rrichter@amd.com>
> > ---
> >  drivers/cxl/core/pci.c | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c
> > index a0e7ed5ae25f..18616ca873e5 100644
> > --- a/drivers/cxl/core/pci.c
> > +++ b/drivers/cxl/core/pci.c
> > @@ -478,8 +478,8 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm,
> >  	}
> >  
> >  	if (!allowed) {
> > -		cxl_set_mem_enable(cxlds, 0);
> > -		info->mem_enabled = 0;
> > +		dev_err(dev, "Range register decodes outside platform defined CXL ranges.\n");
> > +		return -ENXIO;
> 
> While testing I found this needs the following fixup:
> 
> diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c
> index e24ffae8135f..e9e6c81ce034 100644
> --- a/drivers/cxl/core/pci.c
> +++ b/drivers/cxl/core/pci.c
> @@ -477,7 +477,7 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm,
>                 allowed++;
>         }
>  
> -       if (!allowed) {
> +       if (!allowed && info->mem_enabled) {
>                 dev_err(dev, "Range register decodes outside platform defined CXL ranges.\n");
>                 return -ENXIO;
>         }

The change looks correct to me, thanks for fixing.

-Robert

> 
> 
> ...i.e. Linux should only give up if it does not understand an active
> decode region.
> 
> Now this SPA/HPA mismatch will still cause problems later in region
> creation flow, but that's a separate issue.