Received: by 2002:ad5:4acb:0:0:0:0:0 with SMTP id n11csp42573imw; Thu, 14 Jul 2022 20:07:19 -0700 (PDT) X-Google-Smtp-Source: AGRyM1vkEXWTD2pG0zppUcoUYq9t4O6CkC5EFfXKSmt0FE2VF3IOe/tM8W24sKpScSQbnO1oPZLO X-Received: by 2002:a17:907:9715:b0:72b:340b:9059 with SMTP id jg21-20020a170907971500b0072b340b9059mr11466738ejc.384.1657854439441; Thu, 14 Jul 2022 20:07:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1657854439; cv=none; d=google.com; s=arc-20160816; b=vQMC5SjssQ7EkdRRGqxZ9oIjw+wn4wp2VBhWKoFMRfn9KF+pI0VgvZFV1lpt76tGnV 5vpTu8D8/KECJfJ4s1PGYY7zz13GeACi1s2N8maBAdjxWciIxi1XWXzdOg7/FhNNAUp7 1OauaSuNAn/SkYQBa5FCvcD4xMFr5Pfg0CQLptlMqmuG2+A84T7u941j8tsuqKL0tILM BTEfCnze5Zh4yEfRjnEmCeFSqqqtm5bZPrb0E+sh03ZqT0l+m4JDLN60YZl693WBRGPs 6/HSc86I/pIJQo/0j1XsKaWOiVs0Eob/jaZYS/+Q2aFdO51LDEc5oPvHwy17KOzKBFKb TOrw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=/TQpcRBJ+CB5eWMH3TD/E2Z2c35wUY/joEMF2/e9Jkw=; b=vIISg7KU18xiB3c5S7e2Pzzt/sQl3LUXy/a/tFIu7Smh6qvdmZXEv2Z9L0i50qm2eF GO01p6ECgvuNpsY1e8e0ZDbkb7486pH7FtUZKaCF3SBVvwyMjEblmv6vl1qpVvs8R18I bipnZtsXaYO5GfBses+VFBcWW+EP/970h8TOnqDkJCQrPaTvbyF0pxR11sub8dQerfl3 9Jzbog28lpUjDVaz5nrWsFNlcvEjOkIuZXSTu4NGOTeozmXJSepBOr8LpkvsQ0OvGwgd H+z+4BD94xiPzYFsx6CwJ9hJKCXnuawAzfgpafpeietNY4blDfUdrqG8IfTa0xSDWDuT IZsw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=koO7tekl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id js1-20020a17090797c100b0072b9b3d42cbsi5525111ejc.525.2022.07.14.20.06.54; Thu, 14 Jul 2022 20:07:19 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=koO7tekl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241301AbiGODFE (ORCPT + 99 others); Thu, 14 Jul 2022 23:05:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33752 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241276AbiGODEy (ORCPT ); Thu, 14 Jul 2022 23:04:54 -0400 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BDC1576959; Thu, 14 Jul 2022 20:04:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657854289; x=1689390289; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bjcFNxZSbyIUMe7CoijU9+ghKqsvF1aEjrtLIzNGwpc=; b=koO7tekltZNAmHHl4DZD8i8HQVlO4gPqR44wCsxEBUVUBQTBlcbsmIQR zgEuN95ffpzE6b/wocNrrqHUWl7iNeqjinRoV00E/ZULUNEZ9ZbGnA+Km 6LwqVaMyA6pl60hdoPwadvno4+8PNxhz7BpQEArySXbw4Sv8pUnbNd1/q 8b3A8dLLSqjRZkgqu+jEXgCAJJ3FyRNQilEtF+upyT12Yd1O9AYpyVv3t OP9lcimvR0Y87a8H1eknwJYauzJayE/UQH18guBabXwcWlwbpzKMBUrJC zwggwIZsjistfVYZqjchQ6wnBb+bWsShlglILeGMuVJa513KsvTj0p5s8 w==; X-IronPort-AV: E=McAfee;i="6400,9594,10408"; a="284445263" X-IronPort-AV: E=Sophos;i="5.92,272,1650956400"; d="scan'208";a="284445263" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2022 20:04:49 -0700 X-IronPort-AV: E=Sophos;i="5.92,272,1650956400"; d="scan'208";a="600347925" Received: from mbordone-mobl.amr.corp.intel.com (HELO localhost) ([10.255.5.217]) by fmsmga007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jul 2022 20:04:46 -0700 From: ira.weiny@intel.com To: Dan Williams , Bjorn Helgaas , Jonathan Cameron Cc: Ira Weiny , Davidlohr Bueso , Lukas Wunner , Alison Schofield , Vishal Verma , Dave Jiang , Ben Widawsky , linux-kernel@vger.kernel.org, linux-cxl@vger.kernel.org, linux-pci@vger.kernel.org Subject: [PATCH V14 4/7] cxl/pci: Create PCI DOE mailbox's for memory devices Date: Thu, 14 Jul 2022 20:04:21 -0700 Message-Id: <20220715030424.462963-5-ira.weiny@intel.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20220715030424.462963-1-ira.weiny@intel.com> References: <20220715030424.462963-1-ira.weiny@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-5.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Ira Weiny DOE mailbox objects will be needed for various mailbox communications with each memory device. Iterate each DOE mailbox capability and create PCI DOE mailbox objects as found. It is not anticipated that this is the final resting place for the iteration of the DOE devices. The support of switch ports will drive this code into the PCIe side. In this imagined architecture the CXL port driver would then query into the PCI device for the DOE mailbox array. For now creating the mailboxes in the CXL port is good enough for the endpoints. Later PCIe ports will need to support this to support switch ports more generically. Cc: Dan Williams Cc: Davidlohr Bueso Cc: Lukas Wunner Signed-off-by: Ira Weiny --- Changes from V13: Dan: s/pci_err/dev_err s/pci_dbg/dev_dbg Move xa_init() into devm_cxl_pci_create_doe() Changes from V12: remove irq param from CXL Jonathan: remove xa local variable clarify MB creation as best effort But ensure pci_err() if they fail Check devm_add_action() return for failure Davidlohr and Jonathan: Return error ... Changes from V11: Drop review from: Ben Widawsky Remove irq code for now Adjust for pci_doe_get_int_msg_num() Adjust for pcim_doe_create_mb() (No longer need to handle the destroy.) Use xarray for DOE mailbox array Changes from V9: Bug fix: ensure DOE mailboxes are iterated before memdev add Ben Widawsky Set use_irq to false and just return on error. Don't return a value from devm_cxl_pci_create_doe() Skip allocating doe_mb array if there are no mailboxes Skip requesting irqs if none found. Ben/Jonathan Cameron s/num_irqs/max_irqs Changes from V8: Move PCI_DOE selection to CXL_BUS to support future patches which move queries into the port code. Remove Auxiliary device arch Squash the functionality of the auxiliary driver into this patch. Split out the irq handling a bit. Changes from V7: Minor code clean ups Rebased on cxl-pending Changes from V6: Move all the auxiliary device stuff to the CXL layer Changes from V5: Split the CXL specific stuff off from the PCI DOE create auxiliary device code. --- drivers/cxl/Kconfig | 1 + drivers/cxl/cxlmem.h | 3 +++ drivers/cxl/pci.c | 44 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 48 insertions(+) diff --git a/drivers/cxl/Kconfig b/drivers/cxl/Kconfig index f64e3984689f..7adaaf80b302 100644 --- a/drivers/cxl/Kconfig +++ b/drivers/cxl/Kconfig @@ -2,6 +2,7 @@ menuconfig CXL_BUS tristate "CXL (Compute Express Link) Devices Support" depends on PCI + select PCI_DOE help CXL is a bus that is electrically compatible with PCI Express, but layers three protocols on that signalling (CXL.io, CXL.cache, and diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h index c6d6f57856cc..bfa2eaf649a9 100644 --- a/drivers/cxl/cxlmem.h +++ b/drivers/cxl/cxlmem.h @@ -192,6 +192,7 @@ struct cxl_endpoint_dvsec_info { * @component_reg_phys: register base of component registers * @info: Cached DVSEC information about the device. * @serial: PCIe Device Serial Number + * @doe_mbs: PCI DOE mailbox array * @mbox_send: @dev specific transport for transmitting mailbox commands * * See section 8.2.9.5.2 Capacity Configuration and Label Storage for @@ -226,6 +227,8 @@ struct cxl_dev_state { resource_size_t component_reg_phys; u64 serial; + struct xarray doe_mbs; + int (*mbox_send)(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *cmd); }; diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index eeff9599acda..faeb5d9d7a7a 100644 --- a/drivers/cxl/pci.c +++ b/drivers/cxl/pci.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include "cxlmem.h" #include "cxlpci.h" @@ -386,6 +387,47 @@ static int cxl_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type, return rc; } +static void cxl_pci_destroy_doe(void *mbs) +{ + xa_destroy(mbs); +} + +static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds) +{ + struct device *dev = cxlds->dev; + struct pci_dev *pdev = to_pci_dev(dev); + u16 off = 0; + + xa_init(&cxlds->doe_mbs); + if (devm_add_action(&pdev->dev, cxl_pci_destroy_doe, &cxlds->doe_mbs)) { + dev_err(dev, "Failed to create XArray for DOE's\n"); + return; + } + + /* + * Mailbox creation is best effort. Higher layers must determine if + * the lack of a mailbox for their protocol is a device failure or not. + */ + pci_doe_for_each_off(pdev, off) { + struct pci_doe_mb *doe_mb; + + doe_mb = pcim_doe_create_mb(pdev, off); + if (IS_ERR(doe_mb)) { + dev_err(dev, "Failed to create MB object for MB @ %x\n", + off); + continue; + } + + if (xa_insert(&cxlds->doe_mbs, off, doe_mb, GFP_KERNEL)) { + dev_err(dev, "xa_insert failed to insert MB @ %x\n", + off); + continue; + } + + dev_dbg(dev, "Created DOE mailbox @%x\n", off); + } +} + static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) { struct cxl_register_map map; @@ -434,6 +476,8 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) cxlds->component_reg_phys = cxl_regmap_to_base(pdev, &map); + devm_cxl_pci_create_doe(cxlds); + rc = cxl_pci_setup_mailbox(cxlds); if (rc) return rc; -- 2.35.3