Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 20 Oct 2022 22:38:03 -0700
From:   Dan Williams <dan.j.williams@intel.com>
To:     Robert Richter <rrichter@amd.com>,
        Alison Schofield <alison.schofield@intel.com>,
        Vishal Verma <vishal.l.verma@intel.com>,
        "Ira Weiny" <ira.weiny@intel.com>,
        Ben Widawsky <bwidawsk@kernel.org>,
        "Dan Williams" <dan.j.williams@intel.com>
CC:     <linux-cxl@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        "Rafael J. Wysocki" <rafael@kernel.org>,
        Len Brown <lenb@kernel.org>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        "Davidlohr Bueso" <dave@stgolabs.net>,
        Robert Richter <rrichter@amd.com>
Subject: RE: [PATCH v2 07/12] cxl: Remove dev_is_cxl_root_child() check in
 devm_cxl_enumerate_ports()
Message-ID: <6352303b266c_24ac29423@dwillia2-mobl3.amr.corp.intel.com.notmuch>
References: <20221018132341.76259-1-rrichter@amd.com>
 <20221018132341.76259-8-rrichter@amd.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20221018132341.76259-8-rrichter@amd.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?pNyR4t0Yk6R7REVp5K4bb/Q5xB77tXoMu45b4N0xjtr1F3FEIqkNMOffoqcq?=
 =?us-ascii?Q?DfPut21yZnG+MyJMQkyvZ5XYa4syb9SHnFcUbIGWTLpjDXlA24azrqcLtvDM?=
 =?us-ascii?Q?oj4Qrd0/SnJCL+aNwevIkjG7AwWEZkykN8uEMeKlYt9T2uHkwltnTpx+Qwpk?=
 =?us-ascii?Q?5627G1iQNXAbN1tFQuFpdIHJTCELaXliz6mN71rCwzQD6yNYumEqhEEUxHa4?=
 =?us-ascii?Q?/kgcCndEqPHFEk49x/wIpGCwgHrE6tmscp78ZIRThbEyFv2qxoNo81BfBNZx?=
 =?us-ascii?Q?bGAMLK6abrIF/UHC1YoTrOEjHYUkoW2Y+Z2TldhlejYjH9UM8dzE4dAll6Hn?=
 =?us-ascii?Q?C18n6tjed6N3zcJMkefueE1PThAlGBJlvZ+MvrzTuapAnsq8KegFLFeZRU81?=
 =?us-ascii?Q?koanSL+uocKkykwcdAPu5n9FiqGO2SQsll58P/kr6EhJRODZiVbgjR+n9fFL?=
 =?us-ascii?Q?LG7lCiFQ+4rfvFSme/yEYIL9CeBJ6xfa37I8Fj6oiD82V7KygLBgUAnH823h?=
 =?us-ascii?Q?Kr5Gf7dqzX+ZZ86zzSyN+bKOQ6EFz3g5HX0rU8xzKoUDElqHJxVRQq5Eea5t?=
 =?us-ascii?Q?9M0N+INonxQDRdywNZ0xUENMDVWn7ZqLbqvOc4AL4P/OLIXUJdHGfB8A+uA5?=
 =?us-ascii?Q?nRyKtxgRRk4opORUqvx376WuRxHnz5UNbVVmWtRDYjbilO10UAL71y03PDM8?=
 =?us-ascii?Q?Iv2M7D7rBLzTeiKLF047ZZx0/B5BH8mZi6wA+pTqqzNG0d1lhr9uGs3lxTkk?=
 =?us-ascii?Q?VDTTEUavDV6+IdbuYLMxWITQDlyft9XYKaJR5gISmiKB3yW2/hDo7oIGuOSD?=
 =?us-ascii?Q?cLQCpxyS5agFNGXcvhawZl0wNgKboYQBGY5Ic1+eFxHHQnRPsvKT79ikK4Oq?=
 =?us-ascii?Q?e0wHHhzaSh/fsE0yxbKYeB0tnj3WOSyKThOhpRYnvwjVH/xette7UCopcej2?=
 =?us-ascii?Q?Lzj5AzqoohNyUmUaUl3b38OXXljAnTHAFUTC9OupUtT/TJf5N2rOWZwykPAH?=
 =?us-ascii?Q?QtUJGcF6BBuonLchtUJL/Kr33sjUxzxx8u6saFcRXFWuLCd3tQaelJI6DODe?=
 =?us-ascii?Q?n8OTF/KAeLHF+RZ0pIzdhajQTmc1QUI28LMZdp6WuNviZ12ormBlZynnSz7U?=
 =?us-ascii?Q?R3Q9VM16Q4VQuqhr4qljNDMGb9M2MD4SCZ2A8X5cv0Dyfq3fbZVNlLMA4Ofc?=
 =?us-ascii?Q?eI6C6S0rkHvevmH4moY1PFxFLgs9ZJ8ScrGgU4kob+5LmlwcLgKMD0GyIU8C?=
 =?us-ascii?Q?LIIAGZ43tzeh7lHyClDY9pR2LPXBpeUimX0N7JLG9WuXQTluooVa/+IrHstD?=
 =?us-ascii?Q?7AR1ImHa/o3YtmXHW2ZKyLEi8ooHVoJI4OT6HUyq6c0qUy+9uQ3rnqPEpoYv?=
 =?us-ascii?Q?G5S36MtnDu+endAIf1toDU2HdPj8nRWtbsxsSst8YanJQ7R6eBpSMYYr6h1I?=
 =?us-ascii?Q?yEJmhxzRXEQK1VG3tL1QWYvanM6p8c0V9F81RFNS7/MzpZPA0A2ZSjVAxPDW?=
 =?us-ascii?Q?/7bFhZotRZOX7ndImrl4FmYiZeUEqh6TMkz6Ai+ONQolhY61aSKWGzxmp71O?=
 =?us-ascii?Q?l0zBryC0nwpr3qVDf99VpdwQg5IrlnUtmFAhXgjl7emkW//98/RG5NJbSBMJ?=
 =?us-ascii?Q?2Q=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 82df6124-f3e6-47ed-f66f-08dab3266db1
X-MS-Exchange-CrossTenant-AuthSource: MWHPR1101MB2126.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Oct 2022 05:38:05.8886
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: +ZyiM2eIpq5AZUmjrdWY5RO1saPL0zLP8IIEVHWRlXEHCvGurkKpZt/ljmLaOtW3XIMpNIfnXv9oFFVn9YrXlPUy0gUEzf5tSrA0pOs9IMw=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL3PR11MB6362
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.6 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Robert Richter wrote:
> The dev_is_cxl_root_child() check adds complexity to the control flow
> of the iterator loop in devm_cxl_enumerate_ports(). This check is
> unnecessary and can safely be removed: If the port of a dport_dev is
> connected to the cxl root port, the grandparent of dport_dev will be
> null. This is checked early in the iterator loop and the function is
> left in this case.
> 
> Drop this check to ease the control flow in devm_cxl_enumerate_
> ports().
> 
> This change is a prerequisite of factoring out parts of the loop.

Ok, so this seems to be where we diverge on how an RCH topology maps
into the CXL subsystem object hierarchy. The main observation going
through this with Dave before this set came out is that
devm_cxl_enumerate_ports() is a nop and should be skipped in the RCH
case. devm_cxl_enumerate_ports() is only for discovering intermediate
ports between a host bridge and an endpoint.

In a CXL VH topology a direct attached endpoint is downstream of a
host-bridge's root port. In the CXL RCH topology the
Root-Complex-Integrated-Endpoint is a *peer* of a host-bridge's root
port. So one level of the hierarchy is removed and
devm_cxl_enumerate_ports() can be skipped.

The proposal is to have cxl_mem_probe() do something like:

       if (!cxlds->is_rcd) {
               rc = devm_cxl_enumerate_ports(cxlmd);
               if (rc)
                       return rc;
       }

The existing:

       parent_port = cxl_mem_find_port(cxlmd, &dport);

...should do the right thing as long as cxl_acpi registers the host
bridge as a dport with this change:

-       dport = devm_cxl_add_dport(root_port, match, uid, ...
+       dport = devm_cxl_add_dport(root_port, pci_root->bus->bridge, uid, ... 

That way the dport device is not the ACPI device but the 'struct
pci_dev' for the host-bridge.

With that scheme in place and some cxl-cli fixups from Vishal we are
seeing:

# cxl list -BEMPTu
{
  "bus":"root0",
  "provider":"ACPI.CXL",
  "nr_dports":1,
  "dports":[
    {
      "dport":"pci0000:38",
      "id":"0x31"
    }
  ],
  "endpoints:root0":[
    {
      "endpoint":"endpoint1",
      "host":"mem0",
      "depth":1,
      "memdev":{
        "memdev":"mem0",
        "pmem_size":0,
        "ram_size":"16.00 GiB (17.18 GB)",
        "serial":"0",
        "numa_node":0,
        "host":"0000:38:00.0"
      }
    }
  ]
}

Does that make sense? I think this patchset gets a lot simpler if it
does not try to make devm_cxl_enumerate_ports() understand the RCH
topology.