Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 21 Jun 2022 11:24:56 -0700
From:   Ira Weiny <ira.weiny@intel.com>
To:     Davidlohr Bueso <dave@stgolabs.net>
CC:     Dan Williams <dan.j.williams@intel.com>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        "Ben Widawsky" <bwidawsk@kernel.org>,
        Alison Schofield <alison.schofield@intel.com>,
        Vishal Verma <vishal.l.verma@intel.com>,
        "Dave Jiang" <dave.jiang@intel.com>,
        <linux-kernel@vger.kernel.org>, <linux-cxl@vger.kernel.org>,
        <linux-pci@vger.kernel.org>, <a.manzanares@samsung.com>
Subject: Re: [PATCH v11 4/8] cxl/pci: Create PCI DOE mailbox's for memory
 devices
Message-ID: <YrIM+GnP+g0HbwaK@iweiny-desk3>
References: <20220610202259.3544623-1-ira.weiny@intel.com>
 <20220610202259.3544623-5-ira.weiny@intel.com>
 <20220617204046.qdkza6iemkfv2aze@offworld>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20220617204046.qdkza6iemkfv2aze@offworld>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?zfpNSwQDtWJ5CqAcIwVzT4ifyE9FnU7IiwLIGW2yBigSKOn7Hoprbef9U6uh?=
 =?us-ascii?Q?qVZw76w/fyZkYLGrtPWOLStblrWYpEbjevYgcYJZrJxQBJ1shmipv4kxLOg3?=
 =?us-ascii?Q?qB53FGrKW+7G3o9uK6StcKyYM7pGAGxv06j3p/AbA6Z+Fkw9+otmvlnskeLz?=
 =?us-ascii?Q?Jo/00YjpLzTwrCsHoQ5NHNJagU1GmI9cdgJwo/CSbIYXXs4G1x77k9ThwyFq?=
 =?us-ascii?Q?Ki0dzFC0ZSrOPttnwQdlK5e288UAmcuLlrr9DtSr+knyq9SmI/xs97WdYTXX?=
 =?us-ascii?Q?J02UDe4zWVaXQ0Fo/cMfOnfhGi6P9PZ4ABdWBov3zLeTDE0prqJ4OToC/A7s?=
 =?us-ascii?Q?UZlrnbr4CTz9UVCYg9PiCUOWfcZ47uzATXx7GZEXhfZl3pYTOsZ9E8EFJ8yF?=
 =?us-ascii?Q?1hX7Uaq0haZHahy6QImLF63NDL2njHWa+j44oLqgsKBgP/iLIANJLE4vS/Gc?=
 =?us-ascii?Q?gOhL8bfB6b10YhfjMhbwS96EA82oKFIrwcPemZcgPL7XeeIaYYN9aLEv/luU?=
 =?us-ascii?Q?L3GF4b7DA14fGHwcv/BIDYgu0k8AQMrTng/MsBejlZVsH43ckfIdImmLFWyJ?=
 =?us-ascii?Q?+1YaRlKsGhrZ+zs/Ue1MMRhFJKbWGtCckjkirC94SQjOqUuP9nk/BR2Cm/3H?=
 =?us-ascii?Q?7UpdTIMzC+OTaYl+zuDEcHHyY07ibXfOFFJeANNuW0GsenyfCpJe1jFhHllV?=
 =?us-ascii?Q?pfOE440jP66AitynG3ajL2y3MWohg5id36lR4uNVGKI9NYxrqufX6pUyeoeZ?=
 =?us-ascii?Q?WdxaDl7JWaby+nesLnIoUv+Ej4XoDo/HE27RjPsjTEefK2nE+StnnYQCpmtI?=
 =?us-ascii?Q?SAum+jk+ldwahb9WeHWO6KwdhgdPFi5uS1mFOlRSE+QPBCgKkfkh89GmcGDF?=
 =?us-ascii?Q?ejF1fQBWjGOmsvqk60TQY8Q1PPqnmJREJ7FJgJhNAXtJzMICoP9Kfkg5Wstw?=
 =?us-ascii?Q?WAmoe5/bRvpSxtRzjJEW4t31zkpYL/WnTZ2zyztl8ZLR2cQcmYoU4ypa/37a?=
 =?us-ascii?Q?4JxaVPnPxBjZDOktbHu2pVa7VLWC9LRjFWtCIpV+fFn9QFZWjLs8CR/MB33I?=
 =?us-ascii?Q?VMIogNPUXsBEwqcyAexXwqpW8k2lx4o1e+0KS7YCsxqC0yQoYLlefK2YjKxd?=
 =?us-ascii?Q?ApFJHnH/VZZtrMPsh2lBxXn4sI1Oz9FdOyIEiZaS8QfG60GvWXbKMZyzeMgy?=
 =?us-ascii?Q?M+N8rmZxRaEZG3EpZBDplyDpLcqSYcM2GSlq6nVlrFEqmi/YhlhrSIvYdo+V?=
 =?us-ascii?Q?0rx6T1kzydYq1l6hy+ua3GbZaf0R/Rs9wsk/AruD9E/lnVliHnORAQlUhwdQ?=
 =?us-ascii?Q?QoIFrwnRvk2+2lvVJmqBOGe38/WPv2rp+JzVr6CSojGQ9eCNW+VYV/HinrQQ?=
 =?us-ascii?Q?b3/eg8qYJiMNPBkBee50gRP9aNvYRlowTALy8X0+guzWr1hFpP1W2h7A40Pr?=
 =?us-ascii?Q?i5yzxnQz4TseQUI3XMsyNFO39YUbcJbKTzPHXC2wC3HFSwrOsvai67jKmvM8?=
 =?us-ascii?Q?4vZdNFy7D0c9syjpvkAyEEojmbc3ucKgwh24Hq2e2Oy8vlRnaFKB/F9CI2lv?=
 =?us-ascii?Q?siDseIRyLigggCV+G6RekJ90C+t5x1NT+l3AyfesMYuCrvkfCCU6FVwQV33C?=
 =?us-ascii?Q?rcQwK9OPrnwq+LyJ43EW80kKeKjU8uRXQOgJybPkHUus0PY9lHe+VmG2UkAw?=
 =?us-ascii?Q?+PRLyfdYLBNkpD4xtdHNdfndg7JK/0SobpERPcIxtklYZR4isyuVgpXtoOYF?=
 =?us-ascii?Q?LOW5tSVQnA=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 008df86b-24af-4e23-67b4-08da53b35a86
X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB6311.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jun 2022 18:25:01.0596
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: t3CJhRnTOJdye/PaD1+qq5lJ+OTx0lGhJ3Mz6S10239W9Hkuw6wRfh8S/BA6yND4q/IuWW+BNVaZL1qKR3r4jg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR11MB5690
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,
        SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no
        version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Jun 17, 2022 at 01:40:46PM -0700, Davidlohr Bueso wrote:
> On Fri, 10 Jun 2022, ira.weiny@intel.com wrote:
> > +++ b/drivers/cxl/cxlmem.h
> > @@ -191,6 +191,8 @@ struct cxl_endpoint_dvsec_info {
> >  * @component_reg_phys: register base of component registers
> >  * @info: Cached DVSEC information about the device.
> >  * @serial: PCIe Device Serial Number
> 
> Missing doc:
> 
> @doe_use_irq: Use interrupt vectors for DOEs over polling.
> 
> However introducing such flags is not pretty, and this is only used by
> devm_cxl_pci_create_doe(). Do we really need it? See below.

Yes Dan had the same feedback to get rid of the member.

[snip]

> > +
> > static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
> > {
> > 	struct cxl_register_map map;
> > @@ -434,6 +545,9 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
> > 
> > 	cxlds->component_reg_phys = cxl_regmap_to_base(pdev, &map);
> > 
> > +	cxl_alloc_irq_vectors(cxlds);
> > +	devm_cxl_pci_create_doe(cxlds);
> 
> Should cxl_alloc_irq_vectors() just be called directly from devm_cxl_pci_create_doe()
> instead?

I was anticipating having to merge cxl_alloc_irq_vectors() with other code
which is supporting irq's for other things later on.  So I kept the 2 separate.
This is also why the use irq flag was in the device state.  The irq vector call
could have specified to use other types of irq's.  But those can be passed
directly as Dan suggested.

> Also if devm_cxl_pci_create_doe() fails (say ENOMEM), why do we
> bother continuing the cxl_pci probing?

Because the DOE is only required for CDAT data which is optional at this point.

Thanks for the suggested diff below.  But I'm going to go with Dan's suggestion
to use a flag which is set and passed between the functions.

Thanks for the review!
Ira

> 
> Thanks,
> Davidlohr
> 
> ------
> diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
> index ce5b00f3ebcb..44098c785a8b 100644
> --- a/drivers/cxl/cxlmem.h
> +++ b/drivers/cxl/cxlmem.h
> @@ -230,7 +230,6 @@ struct cxl_dev_state {
> 	resource_size_t component_reg_phys;
> 	u64 serial;
> 
> -	bool doe_use_irq;
> 	struct pci_doe_mb **doe_mbs;
> 	int num_mbs;
> 
> diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
> index 72c7b535f5df..47c3741f7768 100644
> --- a/drivers/cxl/pci.c
> +++ b/drivers/cxl/pci.c
> @@ -403,7 +403,7 @@ static void cxl_doe_destroy_mb(void *ds)
> 	}
>  }
> 
> -static void cxl_alloc_irq_vectors(struct cxl_dev_state *cxlds)
> +static int cxl_alloc_irq_vectors(struct cxl_dev_state *cxlds)
>  {
> 	struct device *dev = cxlds->dev;
> 	struct pci_dev *pdev = to_pci_dev(dev);
> @@ -421,9 +421,7 @@ static void cxl_alloc_irq_vectors(struct cxl_dev_state *cxlds)
> 	}
> 
> 	if (!max_irqs)
> -		return;
> -
> -	cxlds->doe_use_irq = false;
> +		return -ENOMEM;
> 
> 	/*
> 	 * Allocate enough vectors for the DOE's
> @@ -435,14 +433,10 @@ static void cxl_alloc_irq_vectors(struct cxl_dev_state *cxlds)
> 		/* Some got allocated; clean them up */
> 		if (rc > 0)
> 			cxl_pci_free_irq_vectors(pdev);
> -		return;
> +		return -ENOMEM;
> 	}
> 
> -	rc = devm_add_action_or_reset(dev, cxl_pci_free_irq_vectors, pdev);
> -	if (rc)
> -		return;
> -
> -	cxlds->doe_use_irq = true;
> +	return devm_add_action_or_reset(dev, cxl_pci_free_irq_vectors, pdev);
>  }
> 
>  /**
> @@ -457,6 +451,10 @@ static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds)
> 	u16 off = 0;
> 	int num_mbs = 0;
> 	int rc;
> +	bool doe_use_irq = false;
> +
> +	if (cxl_alloc_irq_vectors(cxlds))
> +		doe_use_irq = true;
> 
> 	pci_doe_for_each_off(pdev, off)
> 		num_mbs++;
> @@ -475,7 +473,7 @@ static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds)
> 		struct pci_doe_mb *doe_mb;
> 		int irq = -1;
> 
> -		if (cxlds->doe_use_irq)
> +		if (doe_use_irq)
> 			irq = pci_doe_get_irq_num(pdev, off);
> 
> 		doe_mb = pci_doe_create_mb(pdev, off, irq);
> @@ -545,7 +543,6 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
> 
> 	cxlds->component_reg_phys = cxl_regmap_to_base(pdev, &map);
> 
> -	cxl_alloc_irq_vectors(cxlds);
> 	devm_cxl_pci_create_doe(cxlds);
> 
> 	rc = cxl_pci_setup_mailbox(cxlds);