Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 16 Nov 2022 12:55:16 -0800
From:   Dan Williams <dan.j.williams@intel.com>
To:     Robert Richter <rrichter@amd.com>,
        Alison Schofield <alison.schofield@intel.com>,
        Vishal Verma <vishal.l.verma@intel.com>,
        "Ira Weiny" <ira.weiny@intel.com>,
        Ben Widawsky <bwidawsk@kernel.org>,
        "Dan Williams" <dan.j.williams@intel.com>
CC:     <linux-cxl@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        "Rafael J. Wysocki" <rafael@kernel.org>,
        Len Brown <lenb@kernel.org>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        "Davidlohr Bueso" <dave@stgolabs.net>,
        Dave Jiang <dave.jiang@intel.com>,
        Robert Richter <rrichter@amd.com>
Subject: RE: [PATCH v3 8/9] cxl/pci: Extend devm_cxl_port_enumerate_dports()
 to support restricted hosts (RCH)
Message-ID: <63754e3472a4f_12cdff2949@dwillia2-xfh.jf.intel.com.notmuch>
References: <20221109104059.766720-1-rrichter@amd.com>
 <20221109104059.766720-9-rrichter@amd.com>
Content-Type: text/plain; charset="utf-8"
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <20221109104059.766720-9-rrichter@amd.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?RE9WYUxpeEFWK1BqNjQyeklWeVNzc3cxbHVoVWpEcHN6VmNOam5GaUZNVnh5?=
 =?utf-8?B?bTBJQ1ZIdlpqMEJ0emtpV3RXbU5DUWk1bWVnYkgzL1crdHBRZ3ZiTUhkOEpu?=
 =?utf-8?B?cnFVNkpGelB0ZlVpVndrYk82S1B0N1dwNnd4OUY1WTlsVlRVQlh4NEdTZkZQ?=
 =?utf-8?B?RVZLcTFHQW9OTXAvTzFPUmIzazdudkcxejV3cWVaZkRTcTBvZmJ5SXN4eTRQ?=
 =?utf-8?B?Vktma1MvcUwyT0pEdlBEQmdCbVgwUEV3eU1lWk1ZTzZSRkFNWG1ESUhFRy9M?=
 =?utf-8?B?VEcveG5OL2VYOVpSQXpnU2dmUXlpMU10SFdKaVBOOG5kMWZ6bGo1L3N6WFhk?=
 =?utf-8?B?VktVczZDbERRSnNyZVVwTldrMjk4UXNFM1A4M2NQN0tYRDJBQ285bVhHOS9C?=
 =?utf-8?B?VE1ta2dKalYrWHdDVG5qNitMVEhpTjFEQkFkRE1iNU95VjQ0TStVcWx3clk0?=
 =?utf-8?B?czdqdGRBNHVudnZNVG5sckpFa0tjd0JQaHNITS9NdnY4RVZaSS9KNlRWQm5G?=
 =?utf-8?B?UHRNc0RhTmppMmxXQkpuV2NkL1ErM3YrNW51c3BrMzFTUkZhVjhWTUt6L2Qx?=
 =?utf-8?B?QWt0RTRycHhqbW9TdlhRRFh3Tm9FVUp4MWcyU2dEcWd6UnVPcDA2Rjl5b1pk?=
 =?utf-8?B?eGhoUERxQitRWDhSYXpEblllNGx3VExXdW0wVG9mSlRoL1cyNUU4NFRwMUxt?=
 =?utf-8?B?Ukc2dGdLc3g2WjBoNloxWk02Q0ovM0l1SWxCQzlKSkY2Y3JOSjBSWnRCclRI?=
 =?utf-8?B?T1hPTTIwQ0JIWGpoS0ozaXduVWpldTJtbDJFamhFUHNXZHBSbzcxc1B2MXov?=
 =?utf-8?B?Ykt5ZnFkaXo4aDYvT3JDcmMwcWVCdUhTTDYxYWoxL2VBSCs1dmJpYnY3TWNp?=
 =?utf-8?B?enYwT0o1MVJPT2tWeVJvc0NqejQ1SGFUUjJROHZ1ay9iTzdILzB4RE5raDJy?=
 =?utf-8?B?bE5RWmlMOEtFTjZ1UGJDck5VV0JwN24vQWJuWEk3TmwwbXZYTzg1NDR3TmNM?=
 =?utf-8?B?emJrYXRoM2tNSlkydERMRTlIVUw3Z01HYTd4UjltalBqbFFhYkdhQjc1SlBs?=
 =?utf-8?B?blRXYzc0SlMxRFdCdHNRMGdJZ1RkMlM4WGFTYWlkSU01bForKzZOU0Vxa21N?=
 =?utf-8?B?WXlJWUhic3NBK08xM29WOStsL0IwcVBOaVdmRmlrVTdjWi9VbjdNRW5VNWd4?=
 =?utf-8?B?UFBHWDM0cUt2NndIZWg5UzdyQTliaEEvMnJMQkF0dUlTTytER0JwRHlNYzkr?=
 =?utf-8?B?dGVNMFNGeXVDM2dsejc3NkZFWFBUeUI3VTNwUm5WZjJnaFVWeU1YSGxmUEpM?=
 =?utf-8?B?eVVrblE2OHQ0MUcrMWVuUE5Ed2VNcXJ4V1h0REN5d1BJUWFaYkNyM29EanMw?=
 =?utf-8?B?aGprbTc4dytOdzk1eTU0NzZTK0Y5akJnbXU0a1ZGR2hwTElUVHBLa1FWR2ZT?=
 =?utf-8?B?OXdjMVUrNzViS2VhSHl5Rm5XbDNuMWdCUFAxTjhUUHBiNU5DemtLaXdHc0JF?=
 =?utf-8?B?dWVqUnZiYXNiejFIVDNoYU84cXFzSG9sUGNzd2RsRjlxeG03OHRLSmNqYzVh?=
 =?utf-8?B?cUhmd3A2S0NMRnk1Umh5WmxWTVViVGZ3THlOYStrK1NIbk9CSDVaY0pzWnJv?=
 =?utf-8?B?SmNzekZyYmppTS9PVWpoU3ROcVZwVVNsQ2dQVDg0ck1wUXEyZE5NTk1ZbWVO?=
 =?utf-8?B?YXI5dHBqTzFIRUpXQVNKRDd6SVpKQk83ZWVxVXJDZ2NzaWFzQ2hmQlRJNldm?=
 =?utf-8?B?MkVOSy9FY2VVMDlVbFl5c2VkRnhPSVZuQlFFQzNIeG5QUitDSEZpeG11eHNn?=
 =?utf-8?B?dmdnTi9GZHIzVDE2NXVWVXRpOGdqUm80Tm82RS9Hb3lkejBlUHI2Y1p0K1J3?=
 =?utf-8?B?RnhCM3pWTjllRFB0WURiVTl1T3E1eExKenZRZk0wbWlGSjFtTDhxelJVN2l5?=
 =?utf-8?B?VjdNMHloYmhaQ0hxMkVibTNmZ1Nudzczd0RhbkZ1T1RNSzkyVzQ5VFJXQnhU?=
 =?utf-8?B?SXFyRTRuOGR1SkhZR0NCSlRpbFRRK3FmdEV4U1BueXM5YllFWm5laUlkWnRO?=
 =?utf-8?B?T25jeUh1OG0rVWtVZ2lvL1dYR01PUnE2Q2JBakhZMm5GaUgwSDZlYXlyY0o3?=
 =?utf-8?B?V2ZvS0RWY2FpQWcxd21LajgxVUZKZ1FhQmtKUEdRZ1ExS3BNSC9EbEpXQWdB?=
 =?utf-8?B?YWc9PQ==?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 833a7ee9-3bf2-49bb-1984-08dac814debd
X-MS-Exchange-CrossTenant-AuthSource: MWHPR1101MB2126.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Nov 2022 20:55:18.9588
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 67sLo5CYE2wlpOLMGnYgnul74yI9Fim6cX5+uOuhgOhDG/tZZaYjDxDE+8kTtFx0EAgSwfPDFxCx5xs6x8pCOq6fdC0DsQ4+uTZ664kXYM0=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR11MB5817
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Robert Richter wrote:
> The PCIe Software View of an RCH and RCD is different to VH mode. An
> RCD is paired with an RCH and shows up as RCiEP. Its downstream and
> upstream ports are hidden to the PCI hierarchy. This different PCI
> topology requires a different handling of RCHs.
> 
> Extend devm_cxl_port_enumerate_dports() to support restricted hosts
> (RCH). If an RCH is detected, register its port as dport to the
> device. An RCH is found if the host's dev 0 func 0 devices is an RCiEP
> with an existing PCIe DVSEC for CXL Devices (ID 0000h).

It is not clear to me what this extra dport represents. Here are the
Linux CXL objects I see in a VH vs an RCH topology:

               VH
          ┌──────────┐
          │ ACPI0017 │
          │  root0   │
          └─────┬────┘
                │
          ┌─────┴────┐
          │  dport0  │
    ┌─────┤ ACPI0016 ├─────┐
    │     │  port1   │     │
    │     └────┬─────┘     │
    │          │           │
 ┌──┴───┐   ┌──┴───┐   ┌───┴──┐
 │dport0│   │dport1│   │dport2│
 │ RP0  │   │ RP1  │   │ RP2  │
 └──────┘   └──┬───┘   └──────┘
               │
           ┌───┴─────┐
           │endpoint0│
           │  port2  │
           └─────────┘


              RCH
          ┌──────────┐
          │ ACPI0017 │
          │  root0   │
          └────┬─────┘
               │
           ┌───┴────┐
           │ dport0 │
           │ACPI0016│
           └───┬────┘
               │
          ┌────┴─────┐
          │endpoint0 │
          │  port1   │
          └──────────┘

So in the RCH case the only dport is the dport that root0 targets, and
then that dport is directly connected to the RCIEP endpoint-port.

In the VH case another level of dports are needed to route from root0 to
the fan out across the CXL root ports.