Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Thu, 1 Dec 2022 18:00:59 -0800
From:   Dan Williams <dan.j.williams@intel.com>
To:     Ira Weiny <ira.weiny@intel.com>,
        Dan Williams <dan.j.williams@intel.com>
CC:     Davidlohr Bueso <dave@stgolabs.net>,
        Bjorn Helgaas <helgaas@kernel.org>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Alison Schofield <alison.schofield@intel.com>,
        Vishal Verma <vishal.l.verma@intel.com>,
        "Ben Widawsky" <bwidawsk@kernel.org>,
        Steven Rostedt <rostedt@goodmis.org>,
        "Dave Jiang" <dave.jiang@intel.com>,
        <linux-kernel@vger.kernel.org>, <linux-cxl@vger.kernel.org>
Subject: Re: [PATCH V2 01/11] cxl/pci: Add generic MSI-X/MSI irq support
Message-ID: <63895c5b40770_3cbe029432@dwillia2-xfh.jf.intel.com.notmuch>
References: <20221201002719.2596558-1-ira.weiny@intel.com>
 <20221201002719.2596558-2-ira.weiny@intel.com>
 <63894579bf550_3cbe029458@dwillia2-xfh.jf.intel.com.notmuch>
 <Y4lIE7Fyctj6GagE@iweiny-desk3>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <Y4lIE7Fyctj6GagE@iweiny-desk3>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?L7I4rmgSVuzsgFSSCatg0TC5nvSPODyszXM5qWZaswuuxzly7pEPA/fw6jSQ?=
 =?us-ascii?Q?BLS1Quak0sdC+ASAw+i+bX90no77cdo/fWMfq8DZ1HRxhW+H5FpHGmThAfn1?=
 =?us-ascii?Q?0YA8nu95gJ8CPtlnbs10IpHcqnNwPNqHqH5+ImCV3l/izlbEPUZUH9uKPg6a?=
 =?us-ascii?Q?qHD0Da7nurnp1U421JFr7upWgvHvy30t5uSJ/88ZIh0GMUs2dcT5dUcaiSEf?=
 =?us-ascii?Q?SEUEnJg1B392yMZwCKDGZ1VQDQSwVBnBsbp2a86FL9KQnsoD7Uu8p/HOMjeK?=
 =?us-ascii?Q?h8b3R1HNFmtPwslsdviTB6AoYE6hk1HEdt5vgOYLOv4gFFR3pB2lAfrMdjon?=
 =?us-ascii?Q?wpnTmr4WrZzsMUCUTDkpfN2D6XQY3NZs8R7tLZu1fU6e3M9flFyJNEVRqtaZ?=
 =?us-ascii?Q?HHlrK9g8u91cTJZGcjClKp0hSa9wMzK2c5vzc+8y9pz/Y0TuNWgqlZWY2QU8?=
 =?us-ascii?Q?T2cF9uTE8Nb4UDHu9eUdd6Cir8J9KqDJkyizOsFdDzvGSuVHwA2CJiJrq49z?=
 =?us-ascii?Q?i4MF2bNMHvizXcPQZD6kVDqZilNTtL+Yhbv+3YZsLjWSOSDf8eJaKRE2mjK6?=
 =?us-ascii?Q?irmlHYWxnKE4fGTc26QtcZfOFLeXiRe5+ugAfO05X3xFSqkDsqMzwWNqVmRf?=
 =?us-ascii?Q?+8OEBBHoTXKAFNMmWqoIPEeT0iVtASqWY8K7eMP3y4j9E0sNWlAXaEsqNwnr?=
 =?us-ascii?Q?Mql0iQkX/Dqgnk+ETnQiAcRAx5JaLOh7swzRk71ww5qpVFeDRc81hKySRMSK?=
 =?us-ascii?Q?EvbG2tPhvdiMrWJGjwrFFMtdOtZuD7WzckHeuHnjP4mdGQeFjj8vtjqRkfBc?=
 =?us-ascii?Q?qpnwHn4zK1ZJceWxU2WMsPOUYsIXj+3na0FQgfWVJVOhURoauJKu6lnl0RoN?=
 =?us-ascii?Q?tiTaMMlPXvZGvGyA6ey1DcwDH3Vb3i2hGTe5M4U4a9VQnYZlLXutqEoDoS+c?=
 =?us-ascii?Q?w66X4NgkHrKCBMVRX9AFkMzp/2B3BQhguTzYkTXz+hoPECDKZ2AHmelGry2/?=
 =?us-ascii?Q?kT5xgm57lhiqj8qFtwBpD19bUOBXQS7wW7J7OyOamYefaXPA8/e5I/IzsWUz?=
 =?us-ascii?Q?94PHSXILgmLpTQJAlQRYj93RCXfcBrgNzOlCUsmt4CBvGMMCRmeazQkGBrXQ?=
 =?us-ascii?Q?NI/vjjH7eG93FxsAZxKQ7tkmNvQA875hgHKDubPOyGcDlexbnBox4G5Z76oh?=
 =?us-ascii?Q?n/iuo2/CpeTsS47mt0Hieb45PhFfBISU5ambVBhCiXN9/G/FZtHebMhVwt0k?=
 =?us-ascii?Q?Q4G4vrbPqU4TQiqjCEjyNYJLlLU+wmiBic0KI0GQQtUfPdbnz4rDdDnrhMdn?=
 =?us-ascii?Q?PN/JFd1FUbxEk5+k1U3OJl8/f4THQlctsdt3bwaiYQqdvNMQHQ53vfksdqiV?=
 =?us-ascii?Q?xEjqJ1DhKJXQrSJNTW2v8mZdQn2O0d0UnF2xDyu96gkSYtKyEz8f9qxuVXkH?=
 =?us-ascii?Q?18Cg+SvEFzwWR7pS4mfgEE74C/JTYtm27j2D37lbiAiAUpWL2l+gNZalTMfh?=
 =?us-ascii?Q?R/552hhoQ+mb3p1byJnyQNcTLngMlhHt6SgGiBLIQIYp12jLs5nPS910MAAf?=
 =?us-ascii?Q?jQDdFKakSn/ffD0i8F8Q0EEzKcGdNzycRTeol8wbo1vtBrfB/4916BlGQCnf?=
 =?us-ascii?Q?DQ=3D=3D?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 5ff5958d-baf0-45b8-083a-08dad4091001
X-MS-Exchange-CrossTenant-AuthSource: MWHPR1101MB2126.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Dec 2022 02:01:02.0268
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: aMX/wuQmPs9OaKCOEkYCR9L+QPWwDDMn2LmtEnueouFpY3VE7jkqnI9IQG18zkbqyj2T2OGnmZkKBl1cjavzCXJADDyBpTH4wGO8wtGwF1o=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR11MB6186
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Ira Weiny wrote:
> On Thu, Dec 01, 2022 at 04:23:21PM -0800, Dan Williams wrote:
> > ira.weiny@ wrote:
> > > From: Davidlohr Bueso <dave@stgolabs.net>
> > > 
> > > Currently the only CXL features targeted for irq support require their
> > > message numbers to be within the first 16 entries.  The device may
> > > however support less than 16 entries depending on the support it
> > > provides.
> > > 
> > > Attempt to allocate these 16 irq vectors.  If the device supports less
> > > then the PCI infrastructure will allocate that number.
> > 
> > What happens if the device supports 16, but irq-core allocates less? I
> > believe the answer is with the first user, but this patch does not
> > include a user.
> > 
> > > Store the number of vectors actually allocated in the device state for
> > > later use by individual functions.
> > 
> > The patch does not do that.
> 
> Sorry missed updating this message.
> 
> > 
> > I know this patch has gone through a lot of discussion, but this
> > mismatch shows it should really be squashed with the first user because
> > it does not stand on its own anymore.
> 
> It is separate because it was Davidlohr's to begin with.
> 
> I'll squash it back.
> 
> > 
> > > Upon successful allocation, users can plug in their respective isr at
> > > any point thereafter, for example, if the irq setup is not done in the
> > > PCI driver, such as the case of the CXL-PMU.
> > > 
> > > Cc: Bjorn Helgaas <helgaas@kernel.org>
> > > Cc: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> > > Co-developed-by: Ira Weiny <ira.weiny@intel.com>
> > > Signed-off-by: Ira Weiny <ira.weiny@intel.com>
> > > Signed-off-by: Davidlohr Bueso <dave@stgolabs.net>
> > > 
> > > ---
> > > Changes from V1:
> > > 	Jonathan
> > > 		pci_alloc_irq_vectors() cleans up the vectors automatically
> > > 		use msi_enabled rather than nr_irq_vecs
> > > 
> > > Changes from Ira
> > > 	Remove reviews
> > > 	Allocate up to a static 16 vectors.
> > > 	Change cover letter
> > > ---
> > >  drivers/cxl/cxlmem.h |  3 +++
> > >  drivers/cxl/cxlpci.h |  6 ++++++
> > >  drivers/cxl/pci.c    | 23 +++++++++++++++++++++++
> > >  3 files changed, 32 insertions(+)
> > > 
> > > diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
> > > index 88e3a8e54b6a..cd35f43fedd4 100644
> > > --- a/drivers/cxl/cxlmem.h
> > > +++ b/drivers/cxl/cxlmem.h
> > > @@ -211,6 +211,7 @@ struct cxl_endpoint_dvsec_info {
> > >   * @info: Cached DVSEC information about the device.
> > >   * @serial: PCIe Device Serial Number
> > >   * @doe_mbs: PCI DOE mailbox array
> > > + * @msi_enabled: MSI-X/MSI has been enabled
> > >   * @mbox_send: @dev specific transport for transmitting mailbox commands
> > >   *
> > >   * See section 8.2.9.5.2 Capacity Configuration and Label Storage for
> > > @@ -247,6 +248,8 @@ struct cxl_dev_state {
> > >  
> > >  	struct xarray doe_mbs;
> > >  
> > > +	bool msi_enabled;
> > > +
> > 
> > This goes unused in this patch and it also duplicates what the core
> > offers with pdev->{msi,msix}_enabled.
> 
> I tried to argue that with Jonathan and lost.  What I had in V1 was to store
> the number actually allocated.  Then if a function reports something higher
> later it can't be used.

A successful pci_alloc_irq_vectors() call assigns a vector number to all
interrupt sources on the device regardless of how many interrupt sources
there are. If the device has 32 interrupt sources and 16 irqs are returned
from pci_alloc_irq_vectors() then each interrupt source will be sharing
a vector with one or more other vectors. All PCI IRQ vectors are shared.

So I do not see the point of this msi_enabled flag cxl_dev_state. If
pci_alloc_irq_vectors() returns at least 1 then you are good to go.