Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 1 Dec 2022 16:23:21 -0800
From:   Dan Williams <dan.j.williams@intel.com>
To:     <ira.weiny@intel.com>, Dan Williams <dan.j.williams@intel.com>
CC:     Davidlohr Bueso <dave@stgolabs.net>,
        Bjorn Helgaas <helgaas@kernel.org>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Ira Weiny <ira.weiny@intel.com>,
        Alison Schofield <alison.schofield@intel.com>,
        "Vishal Verma" <vishal.l.verma@intel.com>,
        Ben Widawsky <bwidawsk@kernel.org>,
        Steven Rostedt <rostedt@goodmis.org>,
        Dave Jiang <dave.jiang@intel.com>,
        <linux-kernel@vger.kernel.org>, <linux-cxl@vger.kernel.org>
Subject: RE: [PATCH V2 01/11] cxl/pci: Add generic MSI-X/MSI irq support
Message-ID: <63894579bf550_3cbe029458@dwillia2-xfh.jf.intel.com.notmuch>
References: <20221201002719.2596558-1-ira.weiny@intel.com>
 <20221201002719.2596558-2-ira.weiny@intel.com>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20221201002719.2596558-2-ira.weiny@intel.com>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?e5+qfybx51vk803eXqY2G21yCjWtm6xYYDb2+FESuwnEX9QW87bpUCiACX3e?=
 =?us-ascii?Q?5jBGcBiNAVqEpCM/h/knQwdoBckEeSdq/J42EM4lZ4nM8R2sluFuSW45hDIu?=
 =?us-ascii?Q?5KeMJ2Rdstn72BSMCfJbYgv54QTqLFSqkyeR+g+SZITmlEambVDLNtkh0+wu?=
 =?us-ascii?Q?NujQE9npwc43WIO88mDBWAb5UqyyfwsU59cLx2o/HF2NNeuxGpr7uD4tpqA6?=
 =?us-ascii?Q?dQciBfZ7psQpRxU+AL0gzTpyFs/t6i8RL35stdNtJskDKNCWY0Dp3qSJD5aD?=
 =?us-ascii?Q?rIburv4TY5SdOhESKikXoluEp4ah2N1BRtDrMJKUW/JxUuRZDNLOVmmrtZeW?=
 =?us-ascii?Q?wgUKcOIC/NrUuoTyoFSJJnGrUgDC5fjZdtn0m7SPgBTb28le4txMJ3ue1Enx?=
 =?us-ascii?Q?Ddce0lqUqJdbuydQiuP5n7SyB2tVk9N69+49FbcfNOb56+5aCMhi5dFc3QZo?=
 =?us-ascii?Q?L0/Smn7fSihWnqb3z8TmvwG1yljBORj/bkmGWQKPwwoEVlOZIOM56Cw5T2Cw?=
 =?us-ascii?Q?2t81OGhRWmPxOCNoiQ/46yTCSAoZ5F7v9DCn++rlZ3SfvUPyJcBhGLf7DWs9?=
 =?us-ascii?Q?pT0awEzhxxE5P/ernKwC5zMWfAXerMLDtwBxGGKhthlVlMdC9FOTDJxDmvJo?=
 =?us-ascii?Q?K7HQ2yUeqh7sDHi1sUzquc8eW+36eG1nwdrgKw7gTgwIr11uWs1T0+LhqdnB?=
 =?us-ascii?Q?GM7emi9qZ50P688xvkPkYgpLG12Z5h8y0ZwFAGag5lRtS0n3trbMljGYCSkt?=
 =?us-ascii?Q?4Cc2muZVb7RWGjCztoBhz97tSxwST+HHGVeAHV0A+lp8AU71DQP/E+BFBa0J?=
 =?us-ascii?Q?hzla/1CqfEA8t5vbEgg5BTsBKfjD7+xSe4t3ME2LK/OaUAwqcL4AawGoCgu6?=
 =?us-ascii?Q?WNF6+0bMZpWv31sUw4MsWTLIgm4f6XcxYzsQYgqt3TTo4b6TqCuprMUTN00P?=
 =?us-ascii?Q?82UnonZLN5uKxFHV/Xgp20hkoEJs30mXd7KDT5BHiCb+iH1j3OMZj2R50X19?=
 =?us-ascii?Q?yEh8ao3g36ayNcUWHhXqQy+3e6NPiuZC7BD/fPfsGUEddAqDMFgNjtUPJg0M?=
 =?us-ascii?Q?gN6jV425agGPJvH1cj2DdRpmWWDJGLH9aT5rjOtmuHOiX7ikc4ATPF2LD/QU?=
 =?us-ascii?Q?3qzWyy3t+rNB0Mk2+xXPXKmbHh7S6S0z2i2H+QEs79Rd1edN8/+TfQw94Hl9?=
 =?us-ascii?Q?x2GDinrPpPR2kEwXfUg9e8VGQgc42sVDbGPJ2Pc8KbpDR/rGWbPrk5KQjE4e?=
 =?us-ascii?Q?sEiYGksF9FKoBJKBMKn3u0GcC9tgX6oijtZBgiZlwYoWg4/41im3eA2S1MnB?=
 =?us-ascii?Q?yqtDp1DwCqjBAzzbqwB6b33Rt6WBqYbr9jQorSl3p26xNrwFI3pEg8RCPZD+?=
 =?us-ascii?Q?Zrs/QkrmFQP+0h7gu6uoCX+HZQ6Hj87sFZwHZySPFCG4bZPIPmKoxUAh86mC?=
 =?us-ascii?Q?WbJL3622pB/1F6eS6oV9658KRDeMzQ8AOqQBk714fuZBqtBu9xE1jExvo/+o?=
 =?us-ascii?Q?xW7/v3llGmgmbFXZEvFu+D6d1juxxfBjG4T1fFpnpsUzHZxtimLB6Km01IHI?=
 =?us-ascii?Q?1N3oTJtGlyNQ8rCDjAanIXd8RFWMGg2ZjQAFYRYi5Mmec9lSp1893k1RrhW8?=
 =?us-ascii?Q?xA=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 7ad93b26-b386-4d77-90d1-08dad3fb6fd6
X-MS-Exchange-CrossTenant-AuthSource: MWHPR1101MB2126.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Dec 2022 00:23:29.6368
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: oMuECQauUQxeuEoNaJPp65JWfzr4c54QsL2pCer1Rh18jtL2FtBywedMWUZ0A9tQk6dOAz1vuNPxeMTT+qX8COEDKuPJOZWDWZZ4nL/Lkqk=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR11MB5457
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

ira.weiny@ wrote:
> From: Davidlohr Bueso <dave@stgolabs.net>
> 
> Currently the only CXL features targeted for irq support require their
> message numbers to be within the first 16 entries.  The device may
> however support less than 16 entries depending on the support it
> provides.
> 
> Attempt to allocate these 16 irq vectors.  If the device supports less
> then the PCI infrastructure will allocate that number.

What happens if the device supports 16, but irq-core allocates less? I
believe the answer is with the first user, but this patch does not
include a user.

> Store the number of vectors actually allocated in the device state for
> later use by individual functions.

The patch does not do that.

I know this patch has gone through a lot of discussion, but this
mismatch shows it should really be squashed with the first user because
it does not stand on its own anymore.

> Upon successful allocation, users can plug in their respective isr at
> any point thereafter, for example, if the irq setup is not done in the
> PCI driver, such as the case of the CXL-PMU.
> 
> Cc: Bjorn Helgaas <helgaas@kernel.org>
> Cc: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> Co-developed-by: Ira Weiny <ira.weiny@intel.com>
> Signed-off-by: Ira Weiny <ira.weiny@intel.com>
> Signed-off-by: Davidlohr Bueso <dave@stgolabs.net>
> 
> ---
> Changes from V1:
> 	Jonathan
> 		pci_alloc_irq_vectors() cleans up the vectors automatically
> 		use msi_enabled rather than nr_irq_vecs
> 
> Changes from Ira
> 	Remove reviews
> 	Allocate up to a static 16 vectors.
> 	Change cover letter
> ---
>  drivers/cxl/cxlmem.h |  3 +++
>  drivers/cxl/cxlpci.h |  6 ++++++
>  drivers/cxl/pci.c    | 23 +++++++++++++++++++++++
>  3 files changed, 32 insertions(+)
> 
> diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
> index 88e3a8e54b6a..cd35f43fedd4 100644
> --- a/drivers/cxl/cxlmem.h
> +++ b/drivers/cxl/cxlmem.h
> @@ -211,6 +211,7 @@ struct cxl_endpoint_dvsec_info {
>   * @info: Cached DVSEC information about the device.
>   * @serial: PCIe Device Serial Number
>   * @doe_mbs: PCI DOE mailbox array
> + * @msi_enabled: MSI-X/MSI has been enabled
>   * @mbox_send: @dev specific transport for transmitting mailbox commands
>   *
>   * See section 8.2.9.5.2 Capacity Configuration and Label Storage for
> @@ -247,6 +248,8 @@ struct cxl_dev_state {
>  
>  	struct xarray doe_mbs;
>  
> +	bool msi_enabled;
> +

This goes unused in this patch and it also duplicates what the core
offers with pdev->{msi,msix}_enabled.

>  	int (*mbox_send)(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *cmd);
>  };
>  
> diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h
> index eec597dbe763..b7f4e2f417d3 100644
> --- a/drivers/cxl/cxlpci.h
> +++ b/drivers/cxl/cxlpci.h
> @@ -53,6 +53,12 @@
>  #define	    CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK			GENMASK(15, 8)
>  #define     CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK		GENMASK(31, 16)
>  
> +/*
> + * NOTE: Currently all the functions which are enabled for CXL require their
> + * vectors to be in the first 16.  Use this as the max.
> + */
> +#define CXL_PCI_REQUIRED_VECTORS 16
> +
>  /* Register Block Identifier (RBI) */
>  enum cxl_regloc_type {
>  	CXL_REGLOC_RBI_EMPTY = 0,
> diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
> index faeb5d9d7a7a..8f86f85d89c7 100644
> --- a/drivers/cxl/pci.c
> +++ b/drivers/cxl/pci.c
> @@ -428,6 +428,27 @@ static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds)
>  	}
>  }
>  
> +static void cxl_pci_alloc_irq_vectors(struct cxl_dev_state *cxlds)
> +{
> +	struct device *dev = cxlds->dev;
> +	struct pci_dev *pdev = to_pci_dev(dev);
> +	int nvecs;
> +
> +	/*
> +	 * NOTE: pci_alloc_irq_vectors() handles calling pci_free_irq_vectors()
> +	 * automatically despite not being called pcim_*.  See
> +	 * pci_setup_msi_context().
> +	 */
> +	nvecs = pci_alloc_irq_vectors(pdev, 1, CXL_PCI_REQUIRED_VECTORS,
> +				   PCI_IRQ_MSIX | PCI_IRQ_MSI);

clang-format would scooch that second line in for you.

Might also be worth a comment for the next person that goes looking for
why this isn't PCI_IRQ_ALL_TYPES.

From CXL 3.0 3.1.1 CXL.io Endpoint:
A Function on a CXL device must not generate INTx messages if that
Function participates in CXL.cache protocol or CXL.mem protocols.


> +	if (nvecs < 0) {
> +		dev_dbg(dev, "Failed to alloc irq vectors; use polling instead.\n");
> +		return;
> +	}
> +
> +	cxlds->msi_enabled = true;
> +}
> +
>  static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>  {
>  	struct cxl_register_map map;
> @@ -494,6 +515,8 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>  	if (rc)
>  		return rc;
>  
> +	cxl_pci_alloc_irq_vectors(cxlds);
> +
>  	cxlmd = devm_cxl_add_memdev(cxlds);
>  	if (IS_ERR(cxlmd))
>  		return PTR_ERR(cxlmd);
> -- 
> 2.37.2
>