Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Tue, 21 Jun 2022 11:29:35 -0700
From:   Ira Weiny <ira.weiny@intel.com>
To:     Dan Williams <dan.j.williams@intel.com>
CC:     Bjorn Helgaas <bhelgaas@google.com>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Ben Widawsky <bwidawsk@kernel.org>,
        "Alison Schofield" <alison.schofield@intel.com>,
        Vishal Verma <vishal.l.verma@intel.com>,
        Dave Jiang <dave.jiang@intel.com>,
        <linux-kernel@vger.kernel.org>, <linux-cxl@vger.kernel.org>,
        <linux-pci@vger.kernel.org>
Subject: Re: [PATCH V11 4/8] cxl/pci: Create PCI DOE mailbox's for memory
 devices
Message-ID: <YrIOD9ttvAabfyNu@iweiny-desk3>
References: <20220610202259.3544623-1-ira.weiny@intel.com>
 <20220610202259.3544623-5-ira.weiny@intel.com>
 <62ad11db18883_844b129415@dwillia2-xfh.notmuch>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <62ad11db18883_844b129415@dwillia2-xfh.notmuch>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?U4lY5H43qRojIjpcwvQAwJ2GMH13PklfDqWelqYnoTbikhWinjvWlDdXc2el?=
 =?us-ascii?Q?dWA+92IHLHQ+EGUcGxvMDiTI3fNqIeNw5hYizwL1VvOXYJeyHlRacwKdQRF8?=
 =?us-ascii?Q?ZQ+oRdN4oE3RDYybwDcWfAxMvCjiVFoew0aDSAJmbbgWPBG9R1VY4fHB5zUe?=
 =?us-ascii?Q?Av9gwtu7Jj+xBYJOZu5xEENcQQeIDMIeA+PzyJlWHiHUrBfNwuVxFRW3Cwyj?=
 =?us-ascii?Q?xzzGS/RnyXbup59pQ6EZAazdpzqwvd2Cjx76Y8IfWD+F7KLVLJt/kQ00ZaJ8?=
 =?us-ascii?Q?4j+aR/c88ys+U5I6Ct7/Qcml+Rbz5+nNSIQpc/UQuyxxvfXlMUj3sEleBIG3?=
 =?us-ascii?Q?N9FISXIey3JFvWoiMeTCtc9jV8DCR8cZF8Wn1JyWEtezS+fNt/aP3LH65B7+?=
 =?us-ascii?Q?aWEm6/N9MEadPohQqGdlcY2tfs/lteO7t1yy+jrnb9kaPEmJRQNbHHVH6EuL?=
 =?us-ascii?Q?kE0Xp7V9mcmetvlcabeZBQ6A2ty9SsSlCRryjvnRCsJ3LFCGZz0LOIPRibfY?=
 =?us-ascii?Q?23SlYvcHMgFTinb0TCJynQvUXD+UJUlyTVm1VCXnedUaYcleNY8XOklpfdTr?=
 =?us-ascii?Q?kf34AGAl9szSZZjVi/JDLw79FAABN5t/rmNt8E5Krqf6ZnvBz6dVu1GPXj0U?=
 =?us-ascii?Q?YDirNZg3ctAjZh//Mz2tq2iU2chGmHN03ENfVCp9KyfEyLlcZWi+rDJzJJ2T?=
 =?us-ascii?Q?n+nGe5+pgM3ZvUL2GcluuXBEkH/9lPM9C04PDFvGUrIoI7JWJIGqga4xSxAE?=
 =?us-ascii?Q?l1FqzrGtXLyZH0S0i7GwmCht1QwIXMUK04SBSArJQU5DUoN/n3NhFLnyhSBO?=
 =?us-ascii?Q?qk0R53c2kN0fnmZMwNtWBnmuhGiQd+bdPQ7J70/ePP7rgjF9JW8OWLQAsvEs?=
 =?us-ascii?Q?K9aVy5h/BrL2u4JeYGa8YuuuBEbPkN+kWq+/t7j3TtOwqJ3IyhQ0p0RR73R5?=
 =?us-ascii?Q?8RU12qYFAgMe7Pzh2ShKZebdx8Yakwo3GJYnDY/6xHCkS0Z/o3sF5SMZU2ix?=
 =?us-ascii?Q?ZWM4knTingTZfHVydFtUdd/sUbXiwL/niwC8SrnmJWjWGlXIAE3NWdR2hdnH?=
 =?us-ascii?Q?SmenFvlpcecJWbuxY+XCs0OSyeUe6ioQcKCtz9kjpLzOBw+EmP/iltMmQCbX?=
 =?us-ascii?Q?1X5Bwuqb04LgA3lmN+tdTrfe0UCvxZ1l/Cq7nznm/HWch5z14sGzPc6+FitT?=
 =?us-ascii?Q?gTcCTmxv/usi4TYNVxyPCXMuA3bPtXHYNiGAlofagZ+thtcgl881+bpJwErL?=
 =?us-ascii?Q?F6Vdzw+hX3mfChrgjRyBpp+s6cyx8py0cjCvH0fvTbBOlvvLtiePiryBmnv+?=
 =?us-ascii?Q?CMs6z0q8nkLYnIH/883tF7c0Vh5+F7mfQohYILalsZX2tViWiB6AsF82gVM+?=
 =?us-ascii?Q?vYaGI0+yzaVJbLeqEpoYP1fQmr7MjlTThMVNs4bCvevFTJ7H2vRU2Py+yZQw?=
 =?us-ascii?Q?aCeIE1gQS0GF7eAsKiHvEkkjMJUUZbJMl/j4Z/ZwTclioi2ORmOM6lj4uXyd?=
 =?us-ascii?Q?VEw3cip/FRxySCpfmPMQNYi9LOWR0zUEyaEpxa0hnnJaqSGJgneM4uoTq2pM?=
 =?us-ascii?Q?c44KA8lKNxGGkPN7acEVpvQ6zaq5QJsJTMyHKvgqhuttD++lozVVBaGBw/cz?=
 =?us-ascii?Q?//eayDtNQepNp3eZzvEAOzl81dVgJsL5FgGPdvcN6yc34mwjQRdGNBwBW/VX?=
 =?us-ascii?Q?0pWysFBkfWB/cwkXI+jOHbIkkF6csKdiglLOVPm9bauVygKz/Nd8mT+nLjTR?=
 =?us-ascii?Q?IwH032xA+A=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: a5d064b3-3313-4d5b-8da5-08da53b4004f
X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB6311.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jun 2022 18:29:39.2183
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 3IkpxflvHU1srJIDoIoLTCQIFBmfwX3AwldvzVrYtdK9D3f8SpdwtwDJJ5doxquYsya0pCxMVDr+8bc5pcHb4A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR11MB1468
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-5.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Fri, Jun 17, 2022 at 04:44:27PM -0700, Dan Williams wrote:
> ira.weiny@ wrote:

[snip]

> > diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
> > index 60d10ee1e7fc..4d2764b865ab 100644
> > --- a/drivers/cxl/cxlmem.h
> > +++ b/drivers/cxl/cxlmem.h
> > @@ -191,6 +191,8 @@ struct cxl_endpoint_dvsec_info {
> >   * @component_reg_phys: register base of component registers
> >   * @info: Cached DVSEC information about the device.
> >   * @serial: PCIe Device Serial Number
> > + * @doe_mbs: PCI DOE mailbox array
> > + * @num_mbs: Number of DOE mailboxes
> >   * @mbox_send: @dev specific transport for transmitting mailbox commands
> >   *
> >   * See section 8.2.9.5.2 Capacity Configuration and Label Storage for
> > @@ -224,6 +226,10 @@ struct cxl_dev_state {
> >  	resource_size_t component_reg_phys;
> >  	u64 serial;
> >  
> > +	bool doe_use_irq;
> 
> Don't pass temporary state through a long lived data structure. Just
> pass flag by reference between the functions that want to coordinate
> this.

Done.

[snip]

> > +
> > +static void cxl_alloc_irq_vectors(struct cxl_dev_state *cxlds)
> > +{
> > +	struct device *dev = cxlds->dev;
> > +	struct pci_dev *pdev = to_pci_dev(dev);
> > +	int max_irqs = 0;
> > +	int off = 0;
> > +	int rc;
> > +
> > +	/* Account for all the DOE vectors needed */
> > +	pci_doe_for_each_off(pdev, off) {
> > +		int irq = pci_doe_get_irq_num(pdev, off);
> > +
> > +		if (irq < 0)
> > +			continue;
> > +		max_irqs = max(max_irqs, irq + 1);
> 
> This seems to assume that different DOEs will get independent vectors.
> The driver needs to be prepared for DOE instances, Event notifications,
> and mailbox commands to share a single MSI vector in the worst case.
> Lets focus on polled mode DOE, or explicitly only support interrupt
> based operation when no vector sharing is detected.
> 

Ok I see now.  I was under the impression they had to be unique.

Do you think it is sufficient to check in this loop for duplicates and bail if
any are shared?

Ira