Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 8 Sep 2022 12:51:12 -0700
From:   Dan Williams <dan.j.williams@intel.com>
To:     "Jonathan Zhang (Infra)" <jonzhang@fb.com>,
        Dan Williams <dan.j.williams@intel.com>
CC:     Robert Richter <rrichter@amd.com>,
        Alison Schofield <alison.schofield@intel.com>,
        Vishal Verma <vishal.l.verma@intel.com>,
        "Ira Weiny" <ira.weiny@intel.com>,
        Ben Widawsky <bwidawsk@kernel.org>,
        "linux-cxl@vger.kernel.org" <linux-cxl@vger.kernel.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        "Rafael J. Wysocki" <rafael@kernel.org>,
        Len Brown <lenb@kernel.org>
Subject: Re: [PATCH 00/15] cxl: Add support for Restricted CXL hosts (RCD
 mode)
Message-ID: <631a47b0326e_166f294a7@dwillia2-xfh.jf.intel.com.notmuch>
References: <20220831081603.3415-1-rrichter@amd.com>
 <631980ff90319_58016294cc@dwillia2-xfh.jf.intel.com.notmuch>
 <17F02AD3-D68A-4E6A-A549-A47611B335E7@fb.com>
Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <17F02AD3-D68A-4E6A-A549-A47611B335E7@fb.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZDFkV0IyK2FmMmFLc1FnenIwenNqcDhlWlRUYkNpZUZJNituODdla21rVVBs?=
 =?utf-8?B?NXRqejZzemRaWnptalQ0TmFHRjFwWHplWU1qZVcxMWRZVkZFT0pEOEtXZjZn?=
 =?utf-8?B?OFpEbW4wVHV3S3RWOXQwdTBJT214ek01WisxQ3Npb1JZeityaHNOUmlVK0d3?=
 =?utf-8?B?VGdaSEFpUDFjZzJLTERyeG12QUcvSnc3UTdBNFlhQkRiUTJMVHJpVEtUTSs1?=
 =?utf-8?B?OUNvMytrb2xsRUd3MEJnejZ3TzdSNGxlZEF1dXFlNFhZQks4WGhOaHozRklP?=
 =?utf-8?B?dmZKbFROd3FaZUNCT25jSlBPNHA0WHJPZHYva0d6d0dUZGVTWkRFdlZpYlhh?=
 =?utf-8?B?UEtia3FqbDBJRkxoWk9hb2dLa0RacGYvc3BjVFdtbDBIYWxNUkxza3RYQ1ZP?=
 =?utf-8?B?TTQyVmNCaUxsUjg2TUJDU2hadHczT29DUzBWejFXQ2I1TFVrY0NYeVFjYkpP?=
 =?utf-8?B?UHBmZjdLZTBCTUhmNzJYbTFYUXdHTGhzakdPaWVTQlR5WFBYbnQ3SE1zV1FR?=
 =?utf-8?B?SXFSNWl4dnpISXFXeitydk51YmJvRHVTMXlPMlp3cXpvT0ZFa1d4azcvNDQr?=
 =?utf-8?B?a204NGlISXJGRUFrYmZ5WS9oMVFqSVZ0Y1FveEgvakU4Qy8zSVFxa1pPM1F0?=
 =?utf-8?B?SGUxZWR3WjhMSUx4M0dkckQ3WGRQNHN1UlF3QXV6a3B5MXdPWEdGNW1yVFA3?=
 =?utf-8?B?L2pOT0RiczA1V09tT0VWZ01VRGVvY2JWY1BCb3ZReS9EV3I4ekRoSXd6d0Ja?=
 =?utf-8?B?OEkzVmVCL2ZHYXJud1NsWmxpeGtiYUJjbjJKSktXRTNtY3BJZFZ4NnY2Y3pj?=
 =?utf-8?B?amN0a2RtRSsvcUpEZ0pQRTNBQXoxeFRRaVhqQVlwY0t0UTBXcXB0MUFyOUxS?=
 =?utf-8?B?S2RCbWVmZ2puK2FUVWdEaWtsV3ZvdisyNnNCT3ZIUVZOVm94Z2JCSnh4NXdw?=
 =?utf-8?B?RW5wTnpBL1krZHc5aitRT0lwaVVTdEpwMEh0bzV4WWs5NTRPN1FCUmlsWVRy?=
 =?utf-8?B?REdoaUZxdnlhNEVYZVZVdHJUSS90N2U4a1h0c3dPR080WEdCdjNSaWxiWmdX?=
 =?utf-8?B?cmNKZnlmWFlWbkNWN0g0NnZHT3psTGhVZFN0WmIzN2Jna2JhbXhMWjVHWWVP?=
 =?utf-8?B?aXdpdmhNY29PRkd5UThaWjBVYVloS0ZBS2tOdUVib01KS3NJa1hURDQ5QSs2?=
 =?utf-8?B?TG9laTgwMGFRMVVUQlpiNkhYSG1ScmRjNisrNHA5NmJEa09FaXNTMko0VFBa?=
 =?utf-8?B?RnFrQ3hxRnBHOEQ2U1VUSWd0TUJHTDh1TFIyam9LdVhCM0JRblJiMjc3WG9J?=
 =?utf-8?B?L0pzcmNUN2pVQWJ3NDZnM3E0dlY0cmI5eDkraTYvdE90SU1VNmc5azRWaG9w?=
 =?utf-8?B?enZoTHlLMnVWSE16eUFPc0t5TlRWRDBTRlFxdXo2UWNpbHp0clJtcnRjajZG?=
 =?utf-8?B?dldaQ2FkYmNNY0I5MnlsQmljd0Nhc0dxcEIveDVjSW0rSFhGck50dUhYOXpi?=
 =?utf-8?B?SnBBTzUwZGdPZDlic3cyMlZOaHAwcmFOdThHSjVISjl4TmUvdDFWRXZwc0E5?=
 =?utf-8?B?Q0xFSklqUkh5RFQyTmdqNmpPNkRJa2ZyTktDS3JBV1paTDhoQ2d1S1U5dW1w?=
 =?utf-8?B?NUhaRkJNbGFHajEvd2MrbXFrUlhBMGxFTlZWM1QwK1BBRDlnZHEyOUNSUHBW?=
 =?utf-8?B?RGFXU2RjQytIcXBuMzd2bFhyOGhlVm9lbURHMDM4TzJuN2Y3WjhlZVEzSkw1?=
 =?utf-8?B?VmZWRUR1Yklvek1XNFpnUUQrN3RwSE9TdFI0em9IalFMVEpHMkl1cTFBWCtv?=
 =?utf-8?B?OFQ0ZDNNZVpyQnhsSGxoTWN6d0FETUNxVm1rLzEwTEFCeXVkZUVsc0NYZHE0?=
 =?utf-8?B?MDI4dTJyZFJXcytRWWNBVnhUTDBYcVAzY2gyOGNIZFlWUlJodlk0OTE3bFYx?=
 =?utf-8?B?TWhHQjh0WnQyTGdaRWI3RzlmZWdhejhabzltenZKaEE3QjMvN2lNRjVQa3VK?=
 =?utf-8?B?RHFCSDJCRThodGxGQWllTHoraGhXeFhjRHZhSVhGMVBmZ09XS1lyUU4xUmhz?=
 =?utf-8?B?RHQ0WTJ0TlZyUytJUSt5dmJjSWl2MFR0RkEwVTllR01tcG9DRlRleFhwN0Za?=
 =?utf-8?B?LzJOd3E3OUF4bmFnb0JmcDY4ZWpBcEd1bFg3aHVra1FkN1pwYzAvenF3eHBL?=
 =?utf-8?B?TlE9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: fecc14e8-40aa-4c1d-ddfc-08da91d37d09
X-MS-Exchange-CrossTenant-AuthSource: MWHPR1101MB2126.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Sep 2022 19:51:14.9678
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: uDTHOk9h86q3BpT2QGtOCt7551GSJI7lGR9ZwCcUWxCORS34J91eMYbRCA1AAlzl/wtzw6of0PB3Dp38NGYa1Htla+Ncb5234MoovNHu35M=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR11MB3221
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Jonathan Zhang (Infra) wrote:
> 
> 
> > On Sep 7, 2022, at 10:43 PM, Dan Williams <dan.j.williams@intel.com> wrote:
> > 
> > Apologies for the delay in getting to this I had hoped to be able to
> > finish up some other DAX work to focus on this, but time is getting
> > short so I will need to do both in parallel.
> > 
> > Robert Richter wrote:
> >> In Restricted CXL Device (RCD) mode (formerly referred to as CXL 1.1)
> >> the PCIe enumeration hierarchy is different from CXL VH Enumeration
> >> (formerly referred to as 2.0, for both modes see CXL spec 3.0: 9.11
> >> and 9.12, [1]). This series adds support for RCD mode. It implements
> >> the detection of Restricted CXL Hosts (RCHs) and its corresponding
> >> Restricted CXL Devices (RCDs). It does the necessary enumeration of
> >> ports and connects the endpoints. With all the plumbing an RCH/RCD
> >> pair is registered at the Linux CXL bus and becomes visible in sysfs
> >> in the same way as CXL VH hosts and devices do already. RCDs are
> >> brought up as CXL endpoints and bound to subsequent drivers such as
> >> cxl_mem.
> >> 
> >> For CXL VH the host driver (cxl_acpi) starts host bridge discovery
> >> once the ACPI0017 CXL root device is detected and then searches for
> >> ACPI0016 host bridges to enable CXL. In RCD mode an ACPI0017 device
> >> might not necessarily exist 
> > 
> > That's a broken BIOS as far as I can see. No ACPI0017 == no OS CXL
> > services and the CXL aspects of the device need to be 100% managed by
> > the BIOS. You can still run the cxl_pci driver in that case for mailbox
> > operation, but error handling must be firmware-first without ACPI0017.
> Firmware-first or OS-first applies to CXL protocol error handling. For CXL 
> memory error handling, the device generates a DRAM error record, the OS
> parses such record and act accordingly. According to CXL spec (section
> 8.2.9.2.1.2 DRAM Event Record), DPA but not HPA is in such record. The OS
> needs to translate such DPA into HPA to act on. I am taking this as an example
> to show that OS CXL services is needed.
> Instead of using ACPI0016 to tell whether the system is under RCH mode,
> I suppose one way is to check “CXL version” field of CHBS structure in CEDT?

Unless the OS has negotiated CXL _OSC the BIOS owns the event retrieval
and translating it from DPA to HPA. I do want to add OS CXL services to
Linux, but only in the case when the BIOS is actively enabling OS native
address translation which includes populating ACPI0017, CFMWS, and
devices with the HDM decoder capability registers instead of DVSEC range
registers. Everything else is early-gen CXL that is 100% BIOS supported,
similar to DDR where a driver is not expected.