Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 8 Sep 2022 14:02:07 -0700
From:   Dan Williams <dan.j.williams@intel.com>
To:     "Jonathan Zhang (Infra)" <jonzhang@fb.com>,
        Dan Williams <dan.j.williams@intel.com>
CC:     Robert Richter <rrichter@amd.com>,
        Alison Schofield <alison.schofield@intel.com>,
        Vishal Verma <vishal.l.verma@intel.com>,
        "Ira Weiny" <ira.weiny@intel.com>,
        Ben Widawsky <bwidawsk@kernel.org>,
        "linux-cxl@vger.kernel.org" <linux-cxl@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        "Rafael J. Wysocki" <rafael@kernel.org>,
        Len Brown <lenb@kernel.org>
Subject: Re: [PATCH 00/15] cxl: Add support for Restricted CXL hosts (RCD
 mode)
Message-ID: <631a584f46cc9_166f2946@dwillia2-xfh.jf.intel.com.notmuch>
References: <20220831081603.3415-1-rrichter@amd.com>
 <631980ff90319_58016294cc@dwillia2-xfh.jf.intel.com.notmuch>
 <17F02AD3-D68A-4E6A-A549-A47611B335E7@fb.com>
 <631a47b0326e_166f294a7@dwillia2-xfh.jf.intel.com.notmuch>
 <14D1BEC2-4921-46AD-B767-3A7E555F6E97@fb.com>
Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <14D1BEC2-4921-46AD-B767-3A7E555F6E97@fb.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZnQyQ1lqcmhDQW5MZm52c3RVbkJwRTlaMTc5Y3FlZVBQWUcycS9oRUFBRzNt?=
 =?utf-8?B?MXppbE0vSEo0RDdoZmh0azFyQTlaVG9DSmVLVE4wVEo2TEpWZmxqd3NyTVpz?=
 =?utf-8?B?ZTIyYW82eHRJSm1IWXdqS09wcm82Z1RWYkNZNzQ4VGVSdHVDSHdJczNhN2V4?=
 =?utf-8?B?OG5XT1ZrMFV2c1MzZFZkdnRsTS9YalpMUEpQK0FHZXppZTVDaEdBQnc4Zll3?=
 =?utf-8?B?bDh5Wm1SWjU5bXhObUNQdGc4ZDd1OGFDSmFSaDZQQ0xHTEtSNCtaYVFjaTZK?=
 =?utf-8?B?OW1waE8xRXZHRmp1QktLc0tlSlJaY2d5WEZkdEtXdFpoVWdYcDlBTFVMWUwr?=
 =?utf-8?B?WHY1VVFQOGNXR0J4TFNnczcyVjRVYUorYVkzVVJoWUxXT3k0cGpyQTNCK3RL?=
 =?utf-8?B?SXNkSHo3NCswK0V0MW9pa3h4SjFEREVRSVBtcE5nSWpuU3dLUUFZMXU3aXdT?=
 =?utf-8?B?UVA2OWtObWZEenN6QVVZUjBCSlFjSlVuTHRKK05GL2RINWF0RVZtdXZFZlFR?=
 =?utf-8?B?VmdOQlZtVzlsYXFWeUlEbU9nTFNWY2pydVdxZkJGT2dqeFQxR0hBYTIxcXZq?=
 =?utf-8?B?WWhHbE1RL3JqWlVMWm5NSkJJRnRabC92M1NHNjhZeG9mYUo5WXMveGlReGpk?=
 =?utf-8?B?aHNWLzVFbU9XV2g5OFRveXd0VTZySE5VZHZHazA4czFqOHNzNzZ2MnhLeFJD?=
 =?utf-8?B?TCttTU1vcVB2WGZHYUFQMHZoUnljYS9aL3NkbWdqZllBekNIV0FpMVNzNEFP?=
 =?utf-8?B?VmZIekROay9YekJnemNuRTl2YzR6NnlSOTFGRjF3RmdHWjJIU3VhVjNMWmZ3?=
 =?utf-8?B?Q0c1dk11dmR6QnFGbzNWRUVsMkVPMkpOOEZ5NTNDblYxckRWRHE0eWhVeUJC?=
 =?utf-8?B?Ky9YVHdDVE1UM3FIeGEwL2hFbjJQYW1XQUp1NzZWK2pqYUdVdDNYUlcwdzhr?=
 =?utf-8?B?QTdaUXVvM0dMZE5NcmlZU0krTnpueXA5cTJDVmUxZEQ1cU5xZVBRaytrQys3?=
 =?utf-8?B?NWFHZkNQeDBhRmRlUzc3Vlk1ZEQ2ekJHZ0xvQ3NWeXJFclR2eTc3YktzOUEr?=
 =?utf-8?B?MTVLQVJVU000eUNycmpWekZCVEYxK0syYVpKWUJ1R3BHUUtzMTRYYm1scHFu?=
 =?utf-8?B?SWtwbzBxQ0RPbkVGQzdrMm9uRldLbVlKdUFjT0JzdVQ5bVJhUHpjZWVia2Qr?=
 =?utf-8?B?aklTeDNSVmJvakN6SWRDcDBPakRCNVZuRmJzRVJUem9FQXJ0KzVhZHlWejlS?=
 =?utf-8?B?bElpZFkvQ2ptTVNLWS9XdUlkbUNHMUdCUVp4eXFvWWJLZnkwbFk5UlkrMjhv?=
 =?utf-8?B?RlFraFVQNDIrSGR1NEdTS2Zoc2dtTjR3bVZPRlFTQW9Vc2VmNFpHTWpLOWZY?=
 =?utf-8?B?OE5kRE5SMzJkcXlWdWJMWXFHZG1JVnAyQjNGTS9Bb1hzdjFSRDhYWjdRcGpC?=
 =?utf-8?B?K3pIdVprTCsvQ2JXNnlpMnVrc2FUaTZTcXFpdTNFMlpXd2xzN1R3RmlKekpL?=
 =?utf-8?B?M3RieGdyOVRkZHp2UUgwOFBHL1k5TFAzOCtTbThHYVhVa2d6NzU2T2p1K2U4?=
 =?utf-8?B?VVJRd01LcXBILzhhREcxVVo0eDRUYmlGSTdFcnZIamErYkppUEphSVErWi9C?=
 =?utf-8?B?Y3g2dmF0Ny9WN2NQT3B2THZxOVlKNEhHR0JyemxvSFZCYkNzUk9JRXBObk9B?=
 =?utf-8?B?aWI0YXM4bklRWjBOSEVYckIyUktlRFpYSGU4V2E2RWNqR045UEs4dzgvWkZ0?=
 =?utf-8?B?VGRCci9CamlNQUxWdC92U3VsQVRQanhyU2N2SGp2M25DSWZnMkZKUDdzSVVE?=
 =?utf-8?B?S1JFbENQN2RsTVkrRGVRK2g3RGprVU9hYWNzR2ZtdU0rcWNnMWVrU0hHR1ZI?=
 =?utf-8?B?TUlkU2E1bWE3ZlNWdTNTek92b3h4YzllMU1YM0o3c2h6NUF2cWlKcmZJaVo3?=
 =?utf-8?B?QUpzUGtGeDBnMjBKQUcvTGF6dVVacEpUSXdCY0ZiWHYrODNJOE94MHRrR2lN?=
 =?utf-8?B?cWQ4L21VV1lmMW1ZbHViN3ZHcklIL29EWVJZZy8zZ1ZrYTRXendCdnR5K0o4?=
 =?utf-8?B?bWg2WnBlV1VVTEFMazdDdVZXcWwwV0c5cWpteVpGMnJsVERBNEJxZVVkb2U0?=
 =?utf-8?B?QzZGdWl2bWZscjBWZHdVc0x2Sm81MTJDbDlpb1JCT1FLWWpmL1QyeUUvcEJX?=
 =?utf-8?B?TEE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: ad78b93a-e0fa-4fcb-b188-08da91dd64ef
X-MS-Exchange-CrossTenant-AuthSource: MWHPR1101MB2126.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Sep 2022 21:02:09.5295
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: L4ZMmmeLqTyUiI0LLFmm3TVMGs74GFLb4MXf2lP7WHiAiBclZ+ecIaBhEUHFWFIGQqW0M8361Uzxx+M8aPzW0O7OsX7FV/S9OQcgysLapUw=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR11MB4670
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,
        SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Jonathan Zhang (Infra) wrote:
> 
> 
> > On Sep 8, 2022, at 12:51 PM, Dan Williams <dan.j.williams@intel.com> wrote:
> > 
> > !-------------------------------------------------------------------|
> >  This Message Is From an External Sender
> > 
> > |-------------------------------------------------------------------!
> > 
> > Jonathan Zhang (Infra) wrote:
> >> 
> >> 
> >>> On Sep 7, 2022, at 10:43 PM, Dan Williams <dan.j.williams@intel.com> wrote:
> >>> 
> >>> Apologies for the delay in getting to this I had hoped to be able to
> >>> finish up some other DAX work to focus on this, but time is getting
> >>> short so I will need to do both in parallel.
> >>> 
> >>> Robert Richter wrote:
> >>>> In Restricted CXL Device (RCD) mode (formerly referred to as CXL 1.1)
> >>>> the PCIe enumeration hierarchy is different from CXL VH Enumeration
> >>>> (formerly referred to as 2.0, for both modes see CXL spec 3.0: 9.11
> >>>> and 9.12, [1]). This series adds support for RCD mode. It implements
> >>>> the detection of Restricted CXL Hosts (RCHs) and its corresponding
> >>>> Restricted CXL Devices (RCDs). It does the necessary enumeration of
> >>>> ports and connects the endpoints. With all the plumbing an RCH/RCD
> >>>> pair is registered at the Linux CXL bus and becomes visible in sysfs
> >>>> in the same way as CXL VH hosts and devices do already. RCDs are
> >>>> brought up as CXL endpoints and bound to subsequent drivers such as
> >>>> cxl_mem.
> >>>> 
> >>>> For CXL VH the host driver (cxl_acpi) starts host bridge discovery
> >>>> once the ACPI0017 CXL root device is detected and then searches for
> >>>> ACPI0016 host bridges to enable CXL. In RCD mode an ACPI0017 device
> >>>> might not necessarily exist 
> >>> 
> >>> That's a broken BIOS as far as I can see. No ACPI0017 == no OS CXL
> >>> services and the CXL aspects of the device need to be 100% managed by
> >>> the BIOS. You can still run the cxl_pci driver in that case for mailbox
> >>> operation, but error handling must be firmware-first without ACPI0017.
> >> Firmware-first or OS-first applies to CXL protocol error handling. For CXL 
> >> memory error handling, the device generates a DRAM error record, the OS
> >> parses such record and act accordingly. According to CXL spec (section
> >> 8.2.9.2.1.2 DRAM Event Record), DPA but not HPA is in such record. The OS
> >> needs to translate such DPA into HPA to act on. I am taking this as an example
> >> to show that OS CXL services is needed.
> >> Instead of using ACPI0016 to tell whether the system is under RCH mode,
> >> I suppose one way is to check “CXL version” field of CHBS structure in CEDT?
> > 
> > Unless the OS has negotiated CXL _OSC the BIOS owns the event retrieval
> > and translating it from DPA to HPA. I do want to add OS CXL services to
> > Linux, but only in the case when the BIOS is actively enabling OS native
> > address translation which includes populating ACPI0017, CFMWS, and
> > devices with the HDM decoder capability registers instead of DVSEC range
> > registers. Everything else is early-gen CXL that is 100% BIOS supported,
> > similar to DDR where a driver is not expected.
> 
> 
> It makes sense that the BIOS and OS need to negotiate CXL _OSC so that OS
> would take care of address translation. That being said, only DVSEC range 
> register (but not HDM decoder capability register) is available when the device is in
> RCRB mode (section 9.11.8 figure 9-7) attached to a RCH. This type of
> configuration needs to be supported with OS CXL service.
> 

So that figure does have the HDM capabilty pictured in the RCD upstream
port. However, Table 8-22 does seem to incidate that Type 3 D1 devices
are not permitted to have an HDM Decoder Capabilitiy Structure.

However that then leave me confused about figure 9-8 as that shows an
HDM decoder capability in the BAR and not the RCRB. Is that picture
wrong with respect what Table 8-22 indicates?