Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 7 Sep 2022 22:43:27 -0700
From:   Dan Williams <dan.j.williams@intel.com>
To:     Robert Richter <rrichter@amd.com>,
        Alison Schofield <alison.schofield@intel.com>,
        Vishal Verma <vishal.l.verma@intel.com>,
        "Ira Weiny" <ira.weiny@intel.com>,
        Ben Widawsky <bwidawsk@kernel.org>,
        "Dan Williams" <dan.j.williams@intel.com>
CC:     <linux-cxl@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        "Rafael J. Wysocki" <rafael@kernel.org>,
        Len Brown <lenb@kernel.org>, Robert Richter <rrichter@amd.com>
Subject: RE: [PATCH 00/15] cxl: Add support for Restricted CXL hosts (RCD
 mode)
Message-ID: <631980ff90319_58016294cc@dwillia2-xfh.jf.intel.com.notmuch>
References: <20220831081603.3415-1-rrichter@amd.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20220831081603.3415-1-rrichter@amd.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?vO0vYoBKW70fPt74F5LmwPnOkGxlt7L4C/yGGnxZHqJLGlpqkHbs3Bwvfw9e?=
 =?us-ascii?Q?d/EQm9O3s0U+xsDPDSL6dGLsf4qxfJ+DenHhwh5yXQoJF13/+i7FyshIB3Cf?=
 =?us-ascii?Q?ZlPISwExrwI3W+DDMolbcB7hkXcJQgtSqhXS8QtT02fK3aQZWwVSGah9D8gj?=
 =?us-ascii?Q?H5KoRqt422B8y9Ptuj19TgDkoGuxOAhuCHKwWzSyCG2AsbNZ9OWhFxBKjGj3?=
 =?us-ascii?Q?fmXgihP3l2RBC6Bx7UX7B2jYwqnQzLqf2TagXlYcLVz8AJY4sxfg3/T46sFx?=
 =?us-ascii?Q?2cURbOJakgVXZEQ0zm5M11ma4PUsgB/fRE+QKbY8MyFXAesefHTVaBD+0p5a?=
 =?us-ascii?Q?iNVwK4eNigI5+7LGtVu5u6CkS8+6oWpMWPT4k3ThlR+ca1FPE9PLGHHdOStg?=
 =?us-ascii?Q?AzjmJwe+JUDDd32J0DfVJknL+kBrjK95s/2tf1VkMJKdXTjU++/ag2wPyb6t?=
 =?us-ascii?Q?Gu7wO/XjHsh9bfZSkW2R2hdSK1Mr+pABW+hqid9WNLEsSO+pQEWLHgrsgIm6?=
 =?us-ascii?Q?dXDjs+I34Gv/syzazqMyQ4OaqDbW5k9Io7f5rNjAmEAsksiAVuV7WMFf3MDY?=
 =?us-ascii?Q?XNn11OUHknwmZ5sn75bmHf5adPsrJ/G2EIQ7O4CX/q39j333C3BIX73Aowom?=
 =?us-ascii?Q?3BMVgC1fINLUKQUaBepcIZyybh6coyoqZ7NzJWy6r3kv0yqH6qSK1j0QCU3b?=
 =?us-ascii?Q?RnpC0GGoUynO3CUFaH1Q8JMSG4cm6Q+pK1Ga5LmSyywSvIdv44cg2/z0L0/s?=
 =?us-ascii?Q?WStVu6wHYqTmozKG/r7y48jlkIES2Ed5L/389NJEkqUgxI7VM3BH6Q+GF8Bv?=
 =?us-ascii?Q?uhj8j7xHogp67Tfe6VKRySl2Sckqu2qHG4vOFDVjQJDlIDW66mU1ldxsCLTs?=
 =?us-ascii?Q?DPc1NQBJllDh9wCGbR37jpXpo/MuUor52v56rtppAlUZLbMvefAiRxWL7IO/?=
 =?us-ascii?Q?+lDire8EdwpwIESoBwh1z1CdvMCcwtqPN7wgUpT4Ys3ws1hUfElL8LsUr2xl?=
 =?us-ascii?Q?OhXR8SbNVDrwCNl/ONfZsusH80P54VfO7eq1c8U8ul3bCyXYK7GjDxiEha3x?=
 =?us-ascii?Q?3idc+WKidSYxAU7F8cpPTYTerCBq4dWzfYTk4xa+7WjhE78ggJVmwStPXZRd?=
 =?us-ascii?Q?kCRmhPKQNG7jkVDobiCUmvq/43SjApgT/BARI4rfP6eN1qEs3C8ml3TWvEp0?=
 =?us-ascii?Q?5TD1142NkyyDdxhPTRRCfiUnxzbrCAzoylGFkCfk+vfKl++HvrfHpEO8FCIk?=
 =?us-ascii?Q?eUnkjo9fIzbQMQA4VuR6uKStiKMrp9Cx4OpSuPhZEI2Cb9tL+vAnz1xe//KK?=
 =?us-ascii?Q?2elvGf1YQWg39weSg58QFchNyAxb99yWmF7CuXUJJiBIaW3EHxr5Jv2GIn0u?=
 =?us-ascii?Q?ZwayvzPFKJifw6caQO9n1fMUkgEVx/Qn9Pt/wLK/Iu+0xBiXKgh2IsoznCzq?=
 =?us-ascii?Q?A/sDar2lm3dY2H+lCplvVBwuvltFSeoKuGEXoirTzj5VqCAHRvhuybi+JTpv?=
 =?us-ascii?Q?tFqiiOxI/dJ3CAEu7ZVXJpjTsP0mLMnE2SVPCP/edGeUl0OvuoQAUIIkPy5J?=
 =?us-ascii?Q?SoYHA6NRP4bsWus/baFHiDPX6MwAlvCs/m42puirVVLZ2Hup6V9eBDLwHEFm?=
 =?us-ascii?Q?wg=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 9c39cce7-de3d-46e5-c4cb-08da915d0f42
X-MS-Exchange-CrossTenant-AuthSource: MWHPR1101MB2126.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Sep 2022 05:43:30.2553
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 86lqlue8TzpteIHNmJFQLq4Aqc82FR7CYKmrFidJE509dHHOI9pobagpx2dJIl60xem1oxUmYdGjTOJBKnWtBteF2cDuPIjYKXsAmHe4xeU=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR11MB7032
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_PASS,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Apologies for the delay in getting to this I had hoped to be able to
finish up some other DAX work to focus on this, but time is getting
short so I will need to do both in parallel.

Robert Richter wrote:
> In Restricted CXL Device (RCD) mode (formerly referred to as CXL 1.1)
> the PCIe enumeration hierarchy is different from CXL VH Enumeration
> (formerly referred to as 2.0, for both modes see CXL spec 3.0: 9.11
> and 9.12, [1]). This series adds support for RCD mode. It implements
> the detection of Restricted CXL Hosts (RCHs) and its corresponding
> Restricted CXL Devices (RCDs). It does the necessary enumeration of
> ports and connects the endpoints. With all the plumbing an RCH/RCD
> pair is registered at the Linux CXL bus and becomes visible in sysfs
> in the same way as CXL VH hosts and devices do already. RCDs are
> brought up as CXL endpoints and bound to subsequent drivers such as
> cxl_mem.
> 
> For CXL VH the host driver (cxl_acpi) starts host bridge discovery
> once the ACPI0017 CXL root device is detected and then searches for
> ACPI0016 host bridges to enable CXL. In RCD mode an ACPI0017 device
> might not necessarily exist 

That's a broken BIOS as far as I can see. No ACPI0017 == no OS CXL
services and the CXL aspects of the device need to be 100% managed by
the BIOS. You can still run the cxl_pci driver in that case for mailbox
operation, but error handling must be firmware-first without ACPI0017.

> PCIe host bridge PNP0A08 ID, there aren't any CXL port or switches in
> the PCIe hierarchy visible. As such the RCD mode enumeration and host
> discovery is very different from CXL VH. See patch #5 for
> implementation details.
> 
> This implementation expects the host's downstream and upstream port
> RCRBs base address being reported by firmware using the optional CEDT
> CHBS entry of the host bridge (see CXL spec 3.0, 9.17.1.2).
> 
> RCD mode does not support hot-plug, so host discovery is at boot time
> only.
> 
> Patches #1 to #4 are prerequisites of the series with fixes needed and
> a rework of debug messages for port enumeration. Those are general
> patches and could be applied earlier and independently from the rest
> assuming there are no objections with them. Patches #5 to #15 contain
> the actual implementation of RCD mode support.
> 
> [1] https://www.computeexpresslink.org/spec-landing
> 
> Robert Richter (15):
>   cxl/core: Remove duplicate declaration of devm_cxl_iomap_block()
>   cxl/core: Check physical address before mapping it in
>     devm_cxl_iomap_block()
>   cxl: Unify debug messages when calling devm_cxl_add_port()
>   cxl: Unify debug messages when calling devm_cxl_add_dport()
>   cxl/acpi: Add probe function to detect restricted CXL hosts in RCD
>     mode
>   PCI/ACPI: Link host bridge to its ACPI fw node
>   cxl/acpi: Check RCH's PCIe Host Bridge ACPI ID
>   cxl/acpi: Check RCH's CXL DVSEC capabilities
>   cxl/acpi: Determine PCI host bridge's ACPI UID
>   cxl/acpi: Extract the RCH's RCRB base address from CEDT
>   cxl/acpi: Extract the host's component register base address from RCRB
>   cxl/acpi: Skip devm_cxl_port_enumerate_dports() when in RCD mode
>   cxl/acpi: Rework devm_cxl_enumerate_ports() to support RCD mode
>   cxl/acpi: Enumerate ports in RCD mode to enable RCHs and RCDs
>   cxl/acpi: Specify module load order dependency for the cxl_acpi module
> 
>  drivers/acpi/pci_root.c      |   1 +
>  drivers/cxl/acpi.c           | 311 ++++++++++++++++++++++++++++++++++-
>  drivers/cxl/core/pci.c       |  22 ++-
>  drivers/cxl/core/port.c      | 103 ++++++++----
>  drivers/cxl/core/regs.c      |   3 +
>  drivers/cxl/cxl.h            |   2 -
>  drivers/cxl/mem.c            |   1 +
>  tools/testing/cxl/test/cxl.c |   8 +-
>  8 files changed, 400 insertions(+), 51 deletions(-)
> 
> -- 
> 2.30.2
>