Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 21 Jun 2022 12:10:03 -0700
From:   Dan Williams <dan.j.williams@intel.com>
To:     Dan Williams <dan.j.williams@intel.com>, <ira.weiny@intel.com>,
        "Bjorn Helgaas" <bhelgaas@google.com>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>
CC:     Ira Weiny <ira.weiny@intel.com>,
        Alison Schofield <alison.schofield@intel.com>,
        Vishal Verma <vishal.l.verma@intel.com>,
        "Dave Jiang" <dave.jiang@intel.com>,
        Ben Widawsky <bwidawsk@kernel.org>,
        <linux-kernel@vger.kernel.org>, <linux-cxl@vger.kernel.org>,
        <linux-pci@vger.kernel.org>
Subject: RE: [PATCH V11 5/8] cxl/port: Read CDAT table
Message-ID: <62b2178bdcf5d_89207294ac@dwillia2-xfh.notmuch>
References: <20220610202259.3544623-1-ira.weiny@intel.com>
 <20220610202259.3544623-6-ira.weiny@intel.com>
 <62ad1fb69d742_8920729490@dwillia2-xfh.notmuch>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <62ad1fb69d742_8920729490@dwillia2-xfh.notmuch>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?jxmT3u2HoIdLJ5CVbHBNkcKWdAGqQ+NRhzItEtqJomn2Jez6KO5P2o7Zw5hT?=
 =?us-ascii?Q?+8BZe4IvT101mOXfMfDC1g7aQBFK650hZpT1Z///vT6biQxLZweSSfpoI3an?=
 =?us-ascii?Q?NY+jSGWXnn8RN5+ORwivO0WRVQP/kUFScJ6LKQOKU0EIyPzfHvF4dDT0DVNl?=
 =?us-ascii?Q?0rYC0Rz03VW9Mp3l6VES2xPD0ne4gOR594w5tvxpbn14ul9WnIm6c8btLEM6?=
 =?us-ascii?Q?XrwPv4iJnvsd/D7xo/cTSYUClyXTy6hYM2K9/asAjCSfJ52ereWmp4hliD1d?=
 =?us-ascii?Q?AU6RGLZ+bfUDQg8ivZuUoGePvP+E3N7w750bDBnCGSuortTYAFFsJZBa93pd?=
 =?us-ascii?Q?EB6glhlmjFB2HZKH/5z4DeT2Al80naK4k/95tJi2nBeLv8LlRMKGEJdypbiM?=
 =?us-ascii?Q?uLiz/i4tSA7e+y1GDElJXuCCdEaTPLhtXhHLYOcaCItz5OTOZbIwiFvhyDth?=
 =?us-ascii?Q?tdQKwLvsL1SjQbP9mlbGns/L8HCvO/Ubd23GoAWp+q0GtrgXEk9btRMYuiQX?=
 =?us-ascii?Q?X+jPKdEwLta5wa/klennFhQgrrPAh/ePTo77Y0YlToWesPrmP7Gu9o/W/AZx?=
 =?us-ascii?Q?vPwTPakh+F7HjMp0V+4WwWVza/AVbi7ujvm0loydwimtdfxnhwerTgZVytlk?=
 =?us-ascii?Q?Z9Ve+bLy/GarEYZyEp5rHm+jVrsZEDbYfmKkeyPdHvHyXTuArAzbOCBhztt0?=
 =?us-ascii?Q?2OYUcHcX0hiogHcNfvrBGp9csh9U9NsE3XEIW+wKv145EQ6hBNtK59Xyasmu?=
 =?us-ascii?Q?xkiaQwQbP4iMmH0wmNk4dD5nc4S6qrcYtke5VbiGuxeNYW1oGJxxOnsnJ7t5?=
 =?us-ascii?Q?zOJw4pB3bfkFkae1Z+M5B+oBGONJLAx5Z55NG3GeQ4ieCOSbVgXpEnGnUMSO?=
 =?us-ascii?Q?Pe5BR7nevy1cOL1wLloVVHXtNHN67iq8WIbZkZpjPgOZ7BjZQ9qZ023yLZgS?=
 =?us-ascii?Q?VQILfVL83hhSdHHo1YsZAn9v985vhdTnQQQ3/u56lhjjYfGY09bNT8rDm3aa?=
 =?us-ascii?Q?OS7pE1jK0WF0FGegi0nWyobVitbvPJZgY5He0ZNncLOG4q0e0pyD63UCqsNj?=
 =?us-ascii?Q?FV2cbjyQ5imhfWT1aMkUIUf//6XijEclZjtthpJT1t318+Hi0EtdJ7Wfjssw?=
 =?us-ascii?Q?4s2d/7KVkip9eozDDcX+031h5IudVUq/99azB0+zf0GxI+rH6glzWK7vO/4C?=
 =?us-ascii?Q?xj6TgQZVFIC6FrHp+7zT6bnGxO51Y0q8KNWZlaN1E83ars642kXJp8ykrbIE?=
 =?us-ascii?Q?PMQZKX9QzmYMqG2G9CcSPYYDPAJT63ZVLjPfA6H2+uFtn/LKAAWL+rE+Bx6H?=
 =?us-ascii?Q?SpzIqyamUIx5TLhHon7nNjMsgxh7tfzBYGlw7CQCaVR6o3wU4IMbkqH2RymW?=
 =?us-ascii?Q?4PL0K9vWq7TIE4XGKSvvALo8Ys4HlLepEs+VZyUuXOZKrR4TsFDDwkCInv3T?=
 =?us-ascii?Q?bDpBDBXv1hayWgYNwbNN98wD6xauN0fbL15FIwqytHtbknwssABKqAnGljhb?=
 =?us-ascii?Q?gt3j52VGF1+Wsz1v7VHJMwl05aid/swNT5Ykfk7+1VrMZKBgXJ05DcMkXzyO?=
 =?us-ascii?Q?QDlViO4VVCYCPhnmKpGdNhnZZ1J6aSw1cEEbd8KshUiQ8lmssurVjdpVltee?=
 =?us-ascii?Q?DFDDNaSJNXdVomEL4SBVsIhEhcLXtS2UZyCfwXJGJcd81SN99yRF4YNBa9xX?=
 =?us-ascii?Q?LfiV1OqBISOgcehVHdEdZF2ElKm00VP0Tiv+LZCMqFQV9VfQEMtCWednVKh6?=
 =?us-ascii?Q?jCw82NJs+CNaJd3dky7JRTPjJ9MXsS0=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: f8511487-cdb2-4c97-1cf9-08da53b9a74d
X-MS-Exchange-CrossTenant-AuthSource: MWHPR1101MB2126.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jun 2022 19:10:06.9750
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: AMub0rDcsM2SzFYbP+eGfY98IOHkHAGN0R2n8ND4hWtyVUARwj9C1AnBikbK2cqVlvVPLkTmb0yi3AX35x+jc0FeXY+SAwiXFEEzamIGHM4=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR1101MB2231
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Dan Williams wrote:
> ira.weiny@ wrote:
> > From: Ira Weiny <ira.weiny@intel.com>
[..]
> > diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c
> > index c4c99ff7b55e..84dc82f7dff0 100644
> > --- a/drivers/cxl/core/pci.c
> > +++ b/drivers/cxl/core/pci.c
> > @@ -4,10 +4,12 @@
> >  #include <linux/device.h>
> >  #include <linux/delay.h>
> >  #include <linux/pci.h>
> > +#include <linux/pci-doe.h>
> >  #include <cxlpci.h>
> >  #include <cxlmem.h>
> >  #include <cxl.h>
> >  #include "core.h"
> > +#include "cdat.h"
> >  
> >  /**
> >   * DOC: cxl core pci
> > @@ -458,3 +460,173 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm)
> >  	return 0;
> >  }
> >  EXPORT_SYMBOL_NS_GPL(cxl_hdm_decode_init, CXL);
> > +
> > +static struct pci_doe_mb *find_cdat_mb(struct device *uport)
> > +{
> > +	struct cxl_memdev *cxlmd;
> > +	struct cxl_dev_state *cxlds;
> > +	int i;
> > +
> > +	if (!is_cxl_memdev(uport))
> > +		return NULL;
> > +
> > +	cxlmd = to_cxl_memdev(uport);
> > +	cxlds = cxlmd->cxlds;
> 
> This feels stuck between 2 worlds. Either cxl_port_probe() needs to do
> the enumeration, or the attribute needs to move to be memdev relative.
> Given that CXL switches are going to also have CDAT data, then the
> former path needs to happen. Yes, cxl_pci still needs to do the vector
> allocation, but it does not need to do the PCI DOE probing.

It is really the interrupt setup that makes this an awkward fit all
around. The PCI core knows how to handle capabilities with interrupts,
but only for PCIe port services. DOE is both a PCIe port service *and*
and "endpoint service" like VPD (pci_vpd_init()). The more I think about
this the closer I get to the recommendation from Lukas which is that
DOE is more like pci_vpd_init() than pci_aer_init(), or a custom
enabling per driver.

If the DOE enumeration moves to a sub-function of
pci_init_capabilities() then the cxl_pci and/or cxl_port drivers just
look those up and use them. The DOE instances would remain in polled
mode unless and until a PCI driver added interrupt support late. In
other words, DOE can follow the VPD init model as long as interrupts are
not involved, and if interrupts are desired it requires late allocation
of IRQ vectors.