Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 2 Nov 2022 16:42:45 -0700
From:   Ira Weiny <ira.weiny@intel.com>
To:     Davidlohr Bueso <dave@stgolabs.net>
CC:     Bjorn Helgaas <helgaas@kernel.org>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Dan Williams <dan.j.williams@intel.com>,
        <dave.jiang@intel.com>, <alison.schofield@intel.com>,
        <bwidawsk@kernel.org>, <vishal.l.verma@intel.com>,
        <a.manzanares@samsung.com>, <linux-cxl@vger.kernel.org>,
        <linux-kernel@vger.kernel.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        <linux-pci@vger.kernel.org>, Christoph Hellwig <hch@lst.de>
Subject: Re: [PATCH 1/2] cxl/pci: Add generic MSI-X/MSI irq support
Message-ID: <Y2MAdaCES+aUc2QH@iweiny-desk3>
References: <20221024133633.00000467@huawei.com>
 <20221025232535.GA579167@bhelgaas>
 <20221102171524.thsz2kwtirhxn7ee@offworld>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <20221102171524.thsz2kwtirhxn7ee@offworld>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?WWwF8HRdsnOFpGgPmdoV4Ijy+0IKS/ynNQuYhM4wxFmlEjRSHxCZBo61lN8M?=
 =?us-ascii?Q?FTJowY8W/25RCrmkJypOVsPvP52LU3IxgBNT0gCMMQDnGVYCzzS1vgg5ewUE?=
 =?us-ascii?Q?fj8NHCW5X5zhFZyccPALLEVdafqRB2u9yLSzhkk5+tVHuH1mbTTMJN2Ida9R?=
 =?us-ascii?Q?g3qW07Q1K01+6qoGEUeLg6qyrztf4wMYRO6MHSq/W40e4HKHYrmMGUen1Vgo?=
 =?us-ascii?Q?Sc7yxpM3sErhczAIQMFu8p6liyhVYPgsIkQ5JNpSqdu1HspTembk3rfaTNpZ?=
 =?us-ascii?Q?H/DxUtnyQZi+cCvVvBkS1j09DE3GKo+pQGObb4UXHIVWdEKtWxVihLYXxEfe?=
 =?us-ascii?Q?dX+/sBdaiFj1xCs0TAOoQOc53hzuMK3RCRoTqGevlJjG5wCZ0u6xqHpg5GEQ?=
 =?us-ascii?Q?VhZ8lVf0PUUzdiYaKH7CZJy7Pf+2iVNM8eNJL+fKLv0TRP4uU6Z6mr0a2R4e?=
 =?us-ascii?Q?IDiaSdEXXQQ0SyX5OJh00BC+KUCjvYTnl1XClV5RVZzSVT93E7WKOmGlvjQf?=
 =?us-ascii?Q?pI+V6uXVhHjYPIG0fiLo0ttSNJqeZJDH0XNnAHH8U7QI/Y3WcxW7w95CTYE9?=
 =?us-ascii?Q?4Yop9tm/m3tAOgGPohtBx9EJTP5SVUdO+QMUrOvF58/xdkhML8Zrltt2+Gm4?=
 =?us-ascii?Q?Tq83hySxC3FVoH3bdixjCmY36ZvCkDGTV0XQ7XxKstcJeMY+iDnook2ZvHUR?=
 =?us-ascii?Q?vcxoKxcuwUjllJ/WjB9wRilcu57A7pPsEzHwykiP13DoM4U4zIvgDjnzAAH0?=
 =?us-ascii?Q?+C4doiW8HjBiqdhsj857L0aN2LBXQ4TIFDiwJk9ylVmAtx0Ad7eNSRVAPX8D?=
 =?us-ascii?Q?0WSiDZxcQD4J3VyyHbK3JusZdulDQBfUfpCiXNovnmgURAk3xllMU+Tx67Sg?=
 =?us-ascii?Q?8/4ElDmVDqFkCIVoII1vABjsNEYV2EesdDbT3QeU+2hgmunYkUZ6NKzZE/IY?=
 =?us-ascii?Q?b5UBFcTK8USX4yPWfFEK4sZahVnmZPKXvdRopDkz2Y7y27lgAsw7JLzD/qFB?=
 =?us-ascii?Q?5zF7pDPMFLQ/riL2IRlCTINzvMQ/IUN/FqV6Tk/TSpD5SZtoQia9meC4UN/p?=
 =?us-ascii?Q?Qmb1sZVyrSAY1qu9LJ3Fbc6TT/m2jeLupXukcJSr0vnoyfAjmeNC055bzhoG?=
 =?us-ascii?Q?tgdgFWtuj41PQ5L/MuxMQbRc4FrcmE1MAeJ43Bc/1mHdTddjy5fKlnEuT+gT?=
 =?us-ascii?Q?W/CIn/L36lrSDHI48sS6RVMml7qrre1N222zWc5hH1IsApSKqe0WR+dyarE9?=
 =?us-ascii?Q?quLs8EuLujlJg+82MtYHd3qdYXo6mMi5QpL8Fbjurvl7pfb/d42iA4NipdGH?=
 =?us-ascii?Q?tuFUbvVHMu7+c+E8RL/tFqaxvv/6gaMnHcT85p/iWiUjFoG+w9+95CyCzOjm?=
 =?us-ascii?Q?Iv0u0lvez2yHNRreq7pxDMxIIvozKKN8EBer6NUeHIPWqtxvg2/rGUyGPneW?=
 =?us-ascii?Q?D6gVwk0d+b52Kjz0WXm0LSr+y0fdUPczPKUe6bvu5YZfKG8FYyiP0FSqxTsq?=
 =?us-ascii?Q?ucOL+94/k1hRmIxo4ZnjwPKYtbOZKuvFyLkUQ0A+bTDLgXLioz0GSWJYATOZ?=
 =?us-ascii?Q?1nKb+yaoMMZ8U8j8dZa0dJysGPGy1ePbbFmP82ow?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 768026b4-c5f0-413b-41ee-08dabd2bf48d
X-MS-Exchange-CrossTenant-AuthSource: SA1PR11MB6733.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Nov 2022 23:42:51.2362
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: NlpS3gZJmXWZTWa74T16aDPXVa+7ysCZpF11MMzwloBRF9e5wkSea2GachGd7DGLbO7RgHScGvjMilEx7nBadA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR11MB6749
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-5.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,
        RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Wed, Nov 02, 2022 at 10:15:24AM -0700, Davidlohr Bueso wrote:
> On Tue, 25 Oct 2022, Bjorn Helgaas wrote:
> 
> > > In short that calls:
> > > /* Allocate the maximum possible number of MSI/MSI-X vectors */
> > > nr_entries = pci_alloc_irq_vectors(dev, 1, PCIE_PORT_MAX_MSI_ENTRIES,
> > > 			PCI_IRQ_MSIX | PCI_IRQ_MSI);
> > > 
> > > /* See how many and which Interrupt Message Numbers we actually use */
> > > nvec = pcie_message_numbers(dev, mask, &pme, &aer, &dpc);
> > > 
> > > if (nvec != nr_entries) {
> > > 	pci_free_irq_vectors(dev);
> > > 
> > > 	nr_entries = pci_alloc_irq_vectors(dev, nvec, nvec,
> > > 			PCI_IRQ_MSIX | PCI_IRQ_MSI);
> > > }
> > > 
> > > My worry here is that the implicit assumption is that the vectors
> > > won't move if we reduce the overall number of vectors we are asking
> > > for...
> 
> This would also apply to what is currently in portdrv machinery, no?
> 
> > > 
> > > However, imagine the case that we have a feature the driver doesn't
> > > know about that was previously at a higher vector.  After reducing
> > > the vectors allocated the hardware might decide that feature needs
> > > its own vector whereas some others can be combined.  Hence we'd end
> > > up with a less than ideal packing for the features we actually
> > > support.
> > > 
> > > Could do something iterative to solve this if it actually matters
> > > (increase number of vectors until the layout matches what we get
> > > with max possible vectors).
> 
> Maybe do a bounded retry loop until we get stable value?
> 
> retry = 1;
> do {
> 	pci_alloc_irq_vectors(1, 32);
> 	nvecs = get_max_msgnum(); // max(pmu, events, mbox, isolation)
> 	pci_free_irq_vectors();
> 
> 	pci_alloc_irq_vectors(nvecs, nvecs);
> 	new_nvecs = get_max_msgnum();
> 
> 	if (likely(new_nvecs == nvecs))
> 		return 0;
> 
> 	pci_free_irq_vectors();
> }  while (retry--);
> 
> return -1; // no irq support
> 
> But yeah I'm not sure how much we actually care about this. But if so,
> it  also might be worth re-visiting the generic table thing, as if
> nothing else it can standalone co-exist and avoid allocating any irqs
> altogether if we know a-priori that there is no irq support.
> 
> > 
> > Is this cxl code allocating vectors for devices that might also be
> > claimed by portdrv?  I assume not because that sounds like a problem.
> > 
> > Ugh.  I always feel like the portdrv design must be sub-optimal
> > because this seems so hard to do cleanly.
> > 
> > pci_alloc_irq_vectors() has a lot of magic inside it and is great for
> > most drivers, but the PCIe service IRQs are definitely unusual and
> > maybe it's not the best fit for this situation.
> > 
> > If I understand correctly, Interrupt Message Numbers for all these
> > PCIe services (hotplug, AER, DPC, etc) are restricted to 0-31 for both
> > MSI and MSI-X, and the reason we don't just allocate 32 vectors all
> > the time is to avoid consuming too many IRQs.
> 
> Most CXL features that can have irqs will normally use only the first 16,
> with the exception of isolation (cxl 3.0), which per the spec is up to 32.

Dan, Dave, and I were discussing this and we agree.  For now the only things
people are working on are within the first 16 so why not just request 16 as the
max for now?

Ira

> 
> > The MSI case is ugly because the Interrupt Message Number can change
> > when we set Multiple Message Enable.  Maybe we can separate it out and
> > have a less than optimal solution for this case, like allocating one
> > or two vectors and polling if that's not enough.  I expect most
> > devices will support MSI-X.
> 
> Would only supporting MSI-X be so terrible?
> 
> Thanks,
> Davidlohr