Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 25 Oct 2022 11:21:43 -0700
From:   Dan Williams <dan.j.williams@intel.com>
To:     Terry Bowman <Terry.Bowman@amd.com>,
        Dan Williams <dan.j.williams@intel.com>,
        <alison.schofield@intel.com>, <vishal.l.verma@intel.com>,
        <dave.jiang@intel.com>, <ira.weiny@intel.com>,
        <bwidawsk@kernel.org>
CC:     <linux-cxl@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        <bhelgaas@google.com>, <rafael@kernel.org>, <lenb@kernel.org>,
        <Jonathan.Cameron@huawei.com>, <dave@stgolabs.net>,
        <rrichter@amd.com>
Subject: Re: [PATCH 2/5] cxl/pci: Discover and cache pointer to RCD dport's
 PCIe AER capability
Message-ID: <6358293722b51_4da329420@dwillia2-xfh.jf.intel.com.notmuch>
References: <20221021185615.605233-1-terry.bowman@amd.com>
 <20221021185615.605233-3-terry.bowman@amd.com>
 <6354648d9b004_141929463@dwillia2-mobl3.amr.corp.intel.com.notmuch>
 <a65a2259-9835-bb12-5856-8e695b194191@amd.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <a65a2259-9835-bb12-5856-8e695b194191@amd.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?wiQYKzAZ8jyt5iu7TI9HQBiaiHjb3G0QmiCYpFJuSq3qqi7wcp7DevbBn4iB?=
 =?us-ascii?Q?g3k9EjobXON7Zbo2zRLJOdUBiHExkatAdtkN77KQjeKJna7zFHGQ8Cc0Zn4L?=
 =?us-ascii?Q?Zg3DJW5qGKkaKLDqbGe5m0AdloAlB72W+WjnIKDj4OmAqSO6FlnqWyPzwqFo?=
 =?us-ascii?Q?5lqQO9iPV6sfBB7jV3Wn4HgY+S30Z8sMC6QuPXOKrtTmH5jKmHVwVEuteptR?=
 =?us-ascii?Q?J7Eq8wJAbrPR332T9Q8hJCQFwwjC+7YlyWQjSFrcMnSn+fbydyPszA+Lh3rN?=
 =?us-ascii?Q?FF/8QGdvgOPs1eGo1UOKT08VIcTJJg7hD/05PGehhwzutqL6+/yB4FNP1jiT?=
 =?us-ascii?Q?VIgmhGsGUJJOTklU273qcpZXd3e9ezWZoeL0UI7H3himJzAhgXCfaYuLd2mS?=
 =?us-ascii?Q?WGAFPzqPF625/Y17/pW+w1svttrOiyKKbYpHb2Kh2tlMeCKyiTKBDSg65rgV?=
 =?us-ascii?Q?3Ag4T6p8+UaeE1PrQ9g0FN4T+4WDlWsycTF7g/0uEIewPUlm/2NDGN6TLUbF?=
 =?us-ascii?Q?zT79Rrrs/9m2zKzHaLhs3+19gyfW6nstCDZm9Mdgi8P1eibGsnOM0nJclYPN?=
 =?us-ascii?Q?IWQc1yc3oFsMzVFaLqZIKWGQLjqYU5Jl/n7GfeEhhRf846pZ7j24r79TjADZ?=
 =?us-ascii?Q?0asXbeCGboGGYF7fNHzOt8Il5cLL8jmfGRhwpdE1FDQZWGr2PbGaKXs2YosD?=
 =?us-ascii?Q?J6nbkYIU1rK81baoaFzoCJEsBjWm6NzoGNuCXw9lorwlul4I0ClR5bKFb/Gw?=
 =?us-ascii?Q?J+e/G9dQdB1quMiIIqESGe6WK72LOK0QbR21/wbyLBerqi90dpzldhV8ty3S?=
 =?us-ascii?Q?yoLfJPXhL3LnnA9RY+QGvvJdEy20M9JLZjagfZ7q3m32AVYUUTMSBY67y5zB?=
 =?us-ascii?Q?ygCdhxi6dGEOw5UOjgXL5a7rqVSbYBK1yJTFm/mBmt+1IgADUinLBIAhyI38?=
 =?us-ascii?Q?/EaWRZcQuxg7GkHJcd20QTRFgOYt6sxoOrJNTzC/E8RjJoFYreH4Nru9mi1u?=
 =?us-ascii?Q?7zhDd7QM+ZfBr1UV+lh8J5RKC1KmPVGHwO9B0JxwM8X0KgxAREx3PrSMryJ7?=
 =?us-ascii?Q?KX84cr5CdJayUoepXVngaVvFOA3WhKUlVHAWjzJzk/jHC7Duo+9mcpoygEsl?=
 =?us-ascii?Q?kNu2vtmQdvTSS3CUNDZsD/yg0lYEpg6pua1stlhz2Dj+gfTgo+eM+Fg9mGW1?=
 =?us-ascii?Q?YWfbEdztrGkrb6crDPXOszFf6xK6RZUM+Pm2IsWG2ZoyASjBea6QIMmz5LGB?=
 =?us-ascii?Q?fNjEMZpwOa9kH9kbnRMK/CRd9Oyrxq1xkHBp4KlYHch6bYpl/HgZrWNxmdli?=
 =?us-ascii?Q?pFpvfveoyafeejB93nIOvqOluNNvH+EyD/i5CCguTya3BkFf8itsMSuzdlwc?=
 =?us-ascii?Q?qiyE3iye1iIMgAZDmDTcIHimAHuGXeJdtGNm4yGsA62uoANNJa2T0LjTjwZB?=
 =?us-ascii?Q?R7xxN8rBNLWyQ/fewBWKg8Ty+BY+ZtNcJb/NGlq3d8I5TNmira0lwDR8lM6u?=
 =?us-ascii?Q?gfG/Yf5vGnTy/utzeckRCzOxXyZg2cgohRXhnkKThVq/SNI1/4CU0iK1uBnA?=
 =?us-ascii?Q?JkcsKY2neT1AYZRJ/Qxcn2agdL43JSCSaNDLtw0GYJMplCEmB/6BlBAA0CNP?=
 =?us-ascii?Q?lg=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: e41d16c5-208b-4149-5f38-08dab6b5c66e
X-MS-Exchange-CrossTenant-AuthSource: MWHPR1101MB2126.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Oct 2022 18:21:46.2835
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: vEnjPrLIU1pVV2qADRivK4eIPiJ1jZpZjsVNc/oZ8mflvXMkbzgVuLifz3IH/8h9AvoZyYVNpW71EIgACiUZWPcNsC7Nf8JDxYd0o59X5O8=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR11MB4990
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.9 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Terry Bowman wrote:
[..]
> > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> > index f680450f0b16..b42f4759743b 100644
> > --- a/drivers/cxl/cxl.h
> > +++ b/drivers/cxl/cxl.h
> > @@ -499,12 +499,19 @@ cxl_find_dport_by_dev(struct cxl_port *port, const struct device *dport_dev)
> >   * @port_id: unique hardware identifier for dport in decoder target list
> >   * @component_reg_phys: downstream port component registers
> >   * @port: reference to cxl_port that contains this downstream port
> > + * @is_rch: enable RCH vs VH enumeration (see CXL 3.0 9.11.8)
> >   */
> >  struct cxl_dport {
> >         struct device *dport;
> >         int port_id;
> >         resource_size_t component_reg_phys;
> >         struct cxl_port *port;
> > +       bool is_rch;
> > +};
> > +
> > +struct cxl_rch_dport {
> > +       struct cxl_dport dport;
> > +       resource_size_t rcrb_phys;
> >  };
> >  
> 
> The same is needed for uport as well, correct ?

Hmm, I don't think there are any 'struct cxl_port' instances that need
an RCH flag in the Linux CXL topology model.  That was the feedback /
realization that Dave and I came to while reviewing Robert's RCH series.
In the RCH case the ACPI0016 host-bridge houses a
root-complex-integrated endpoint that is a peer of the
downstream-PCI-root-ports in the bridge. The topology ends up looking
like this:

# cxl list -BEMPTu
{
  "bus":"root0",
  "provider":"ACPI.CXL",
  "nr_dports":1,
  "dports":[
    {
      "dport":"pci0000:38",
      "id":"0x31"
    }
  ],
  "endpoints:root0":[
    {
      "endpoint":"endpoint1",
      "host":"mem0",
      "depth":1,
      "memdev":{
        "memdev":"mem0",
        "pmem_size":0,
        "ram_size":"16.00 GiB (17.18 GB)",
        "serial":"0",
        "numa_node":0,
        "host":"0000:38:00.0"
      }
    }
  ]
}

Notice that the logical ACPI0017 (CXL root) device lists the ACPI0016
device (associated with pci0000:38) as a 'struct cxl_dport'. The
endpoint is then connected to that dport. There are no intervening
'struct cxl_port' instances between the root0 and endpoint1 like there
would be in the CXL VH case.

Now, the problem is that there is no viable object that can drive access
to the upstream port component registers. It may be the case that when
the cxl_port driver attaches to an endpoint port that the endpoint port
driver maps both the upstream and downstream hardware-port registers for
the purposes of conveying RAS information.

> >  /**
> > 
> > Then, when cxl_mem notices that the memdev is being produced by an
> > RCIEP, it can skip devm_cxl_enumerate_ports() and jump straight to
> > cxl_mem_find_port(). That will return this dport with the rcrb base
> > where cxl_mem can arrange the AER handling. Likely we will need some
> > notification mechanism to route Root Complex AER events to cxl_acpi,
> > cxl_pci, and/or cxl_mem to optionally add the CXL RAS data to the log.
> > 
> Isn't the notification mechanism through the AER interrupt processing? 
> I will have more related comments in patch 3/5.

In this case I was talking about notifying the cxl_mem driver and/or the
cxl_port driver that it needs to decorate an AER event with more
object-local information.