Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Fri, 16 Sep 2022 11:16:31 -0700
From:   Dan Williams <dan.j.williams@intel.com>
To:     Robert Richter <rrichter@amd.com>,
        Alison Schofield <alison.schofield@intel.com>,
        Vishal Verma <vishal.l.verma@intel.com>,
        "Ira Weiny" <ira.weiny@intel.com>,
        Ben Widawsky <bwidawsk@kernel.org>,
        "Dan Williams" <dan.j.williams@intel.com>
CC:     <linux-cxl@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        "Rafael J. Wysocki" <rafael@kernel.org>,
        Len Brown <lenb@kernel.org>, Robert Richter <rrichter@amd.com>
Subject: RE: [PATCH 00/15] cxl: Add support for Restricted CXL hosts (RCD
 mode)
Message-ID: <6324bd7f45899_2a6ded294f5@dwillia2-xfh.jf.intel.com.notmuch>
References: <20220831081603.3415-1-rrichter@amd.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20220831081603.3415-1-rrichter@amd.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?O1PZopm2Ssd//I+XMbvCfOgaf7F0JzeGPPLiks3Z1Hru5c1zMN5AzR0TuzPt?=
 =?us-ascii?Q?srl/KSfB/NIbU+xR+aDcmBjXmGJMXSNg/iUHJCumuC/hTLqBXeo2SJZsfpW3?=
 =?us-ascii?Q?NvcJl4jTlOOovzvAxzefVckJiM1xb3+rgMO1uPfOJlhhpLjLj/9fqu/cDx8w?=
 =?us-ascii?Q?FIAVmsvPtL5CVOZmScQdX9gM/g5R8Z0SwhkcEdV3JfgW9iXc1nkP8SZasrsQ?=
 =?us-ascii?Q?lWedGdwNVVqfBvCj+C85BR5ga7lahwBWE0zJcJ0gGY0dbqGxJrfGTsc2UyhG?=
 =?us-ascii?Q?uDycbKSqTpJ0g64qLwLm7iZs5pKIxptAIhzx3Ga4DnySs227DS33jXaXTOiS?=
 =?us-ascii?Q?+422IgiUxFk0ftEattukPxpIrLmx1dnhsa2I23HQGEmkDohnB/68czRHMNkC?=
 =?us-ascii?Q?aM1KidOk1ysPP+rFemuVV8YypDy6p4KZUHeRd67QhdSEnbgugBowETc2HxjT?=
 =?us-ascii?Q?9vIlG01nXH0IwgtR4ou1JLv8gPLIH5o8weT+plity2jT/ygDFrCcGHlMuMTw?=
 =?us-ascii?Q?ZMWP7Fd0MMcKYqF2uFRsDK1tQ6lWDJzoy8263d+fR7igVkq/pRVP6v/6YCDf?=
 =?us-ascii?Q?aKco58t+U1kB1iQ1+zkNU+3N2PqeH7NlK25zfL0qoz3P2j3u+A7LtdtUGuca?=
 =?us-ascii?Q?/LFcvWeuj3NBU2kS0yeunyoOQg5UGbQ7U++28nxFungaVMlLVobZIFD2FGEe?=
 =?us-ascii?Q?iM4g3o1Ci1RLC8bpL80G2kADGTWrr943xGO3Fz84dxtG0DgGDCG4OpVLcJVV?=
 =?us-ascii?Q?aBTdk4Yo2X/9hqBgZlrJj0JGa5xY+GMey0cqSfp0CqieYz1DuD1/DsKaypm6?=
 =?us-ascii?Q?D2U0nH5Bc6XkFo5l7ZEyHQxgkThu2E8SfMus2O+7IvnJUkjwnIoTY1Y/0SmW?=
 =?us-ascii?Q?rtbF8HV8U23yo8k1GnAgFf8LqR+tehuw4YzdLkWCMcho6rzai+UxAuDIftsI?=
 =?us-ascii?Q?/cQUBiQo7FBOH1xb9lJ4PnEQhBCrgccjvHXagmhcPhsBSFaqACR8y/NCe/XO?=
 =?us-ascii?Q?sHcI+sawi6d5/RMQKTHD/fl4V/fdTjNMKkdNRutos0fHag9wUHtA/66dV4/T?=
 =?us-ascii?Q?SlJMKT9rvw75qahCexDyvotaLC+Rs9D/VC+j/W6RT95YC4uWjDj5se8Emz4T?=
 =?us-ascii?Q?kWMfjSg86Jtjk59Vdbum5PiHeNwqpLCGN28EbQix+uKBVV6P/MQqKapTsE2k?=
 =?us-ascii?Q?7ET2LxABN/oUYCEyCO81pCqEDGfIrTYcZNDHjr/xf20b4ohMDYEc9DemD8v9?=
 =?us-ascii?Q?GKnYHSJ/8LXHoXSpYhEYy36aQ272LvCMSmCs1EofZNPYCrpBRGBjWNA5mCBU?=
 =?us-ascii?Q?m5YbJELQZTIZEkefIsee8ZxKYxXUdy+WFqLE3m8/is0bv39g8pra/DlbjlsX?=
 =?us-ascii?Q?Te62xDR4ITFSE/fJuEtq67yGPODTuPIF/rpz3cikc7xUVWvTY2e0NZX2JTjd?=
 =?us-ascii?Q?/hA2MAr+Kun5E7QuvjYXAvAfwppL6op6fxSw4g7UubQA9bHDo73xqN7pqnFN?=
 =?us-ascii?Q?sax4ULgpeP2YAB+e9cY2qxuTenCX0xRmhJkHWag1u71G1aWcd4gPikxIfbsL?=
 =?us-ascii?Q?NhAFudvIcgkt8LZNPvRqV3kYkGYRxx9VGZbaoAVZmzwigJ19quXf5VZM/142?=
 =?us-ascii?Q?2Q=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 3a93203b-8b22-4e98-28b0-08da980f9658
X-MS-Exchange-CrossTenant-AuthSource: MWHPR1101MB2126.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Sep 2022 18:16:34.2825
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: kc161FtIbDSXELBtvtPIX84r4LlGe+goAegObZpzYzUE740aGqx9RbF0IxqMQzHFKt0W4qM8j4bSlmr9QxoorKMAVR9nGljBnq0A57k5q58=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB6945
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Robert Richter wrote:
> In Restricted CXL Device (RCD) mode (formerly referred to as CXL 1.1)
> the PCIe enumeration hierarchy is different from CXL VH Enumeration
> (formerly referred to as 2.0, for both modes see CXL spec 3.0: 9.11
> and 9.12, [1]). This series adds support for RCD mode. It implements
> the detection of Restricted CXL Hosts (RCHs) and its corresponding
> Restricted CXL Devices (RCDs). It does the necessary enumeration of
> ports and connects the endpoints. With all the plumbing an RCH/RCD
> pair is registered at the Linux CXL bus and becomes visible in sysfs
> in the same way as CXL VH hosts and devices do already. RCDs are
> brought up as CXL endpoints and bound to subsequent drivers such as
> cxl_mem.
> 
> For CXL VH the host driver (cxl_acpi) starts host bridge discovery
> once the ACPI0017 CXL root device is detected and then searches for
> ACPI0016 host bridges to enable CXL. In RCD mode an ACPI0017 device
> might not necessarily exist and the host bridge can have a standard
> PCIe host bridge PNP0A08 ID, there aren't any CXL port or switches in
> the PCIe hierarchy visible. As such the RCD mode enumeration and host
> discovery is very different from CXL VH. See patch #5 for
> implementation details.
> 
> This implementation expects the host's downstream and upstream port
> RCRBs base address being reported by firmware using the optional CEDT
> CHBS entry of the host bridge (see CXL spec 3.0, 9.17.1.2).
> 
> RCD mode does not support hot-plug, so host discovery is at boot time
> only.
> 
> Patches #1 to #4 are prerequisites of the series with fixes needed and
> a rework of debug messages for port enumeration. Those are general
> patches and could be applied earlier and independently from the rest
> assuming there are no objections with them. Patches #5 to #15 contain
> the actual implementation of RCD mode support.

Hi Robert,

I did not see a response to some of my feedback but wanted to summarize
where I think the next version of this set needs to go:

1/ ACPI0017 is mandatory. If a BIOS does not provide ACPI0017 it is
explicitly opting the OS out of managing anything other than the CXL.io
side of memory expanders.

2/ Per table 8-22 in CXL 3.0 RCDs are not permitted to have HDM decoders
so that assumption in the driver needs to be reworked.

3/ It's not even clear that the Register Locator DVSEC has any role to
play in an RCD as every register the driver needs should be relative to
the RCRB. So the assumptions in the driver need to consider RCRB located
registers as a first class citizen.