Received-SPF: pass (google.com: domain of linux-kernel+bounces-136102-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
From: lyx634449800 <yuxue.liu@jaguarmicro.com>
To: mst@redhat.com,
	jasowang@redhat.com
Cc: angus.chen@jaguarmicro.com,
	virtualization@lists.linux.dev,
	xuanzhuo@linux.alibaba.com,
	yuxue.liu@jaguarmicro.com,
	linux-kernel@vger.kernel.org
Subject: [PATCH v3] vp_vdpa: fix the method of calculating vectors
Date: Tue,  9 Apr 2024 09:49:35 +0800
Message-Id: <20240409014935.944-1-yuxue.liu@jaguarmicro.com>
In-Reply-To: <20240408035346-mutt-send-email-mst@kernel.org>
References: <20240408035346-mutt-send-email-mst@kernel.org>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0:
	=?us-ascii?Q?TjrwEwRf8PR2NujH2pno/l4vXzSTep4i4uONJV31usTuNWQFfw6ikdmj7/eE?=
 =?us-ascii?Q?GHk8nYrfIwf9WtskYem5vo83ZqGC44ylUkxz4iClrPv1BtxWUlhPHnkCKhWk?=
 =?us-ascii?Q?3HrEWSDL8/7Bv+FWq1myAMzTwAGWz4452RA7KDKNMYSY1iE6uuph6TDU/uZs?=
 =?us-ascii?Q?xXWfO1YIxaCuxexARbya3MXc0aT5CyeHtkfljM57ffxDS7myXdANMyCVQ002?=
 =?us-ascii?Q?QcH9iDkiYRNDqrQTYyvQ7hxNCV4eQQyf1fMpJySh3R3Mb2oW8UjmOQ3RbPAL?=
 =?us-ascii?Q?t2Kj8saCoCWZy8Kp+KixFqNl6a/En94EUyGVmFmQDrecOD/wfdykcs63t5n/?=
 =?us-ascii?Q?2e35uF5XFWaXyMZqcgV+ybMFxOgwFyUzP+n9FndVyezAPChTjVZRGFGYe4NN?=
 =?us-ascii?Q?E4wU6SJFwxCa0ZwIWNkl1j8t/eXVv13mAVEeZeFN+9T8G2MGcFLP2Cp4sg0C?=
 =?us-ascii?Q?Yft+Hul9hUuNnS2zYvEKPmrIlw3oEz381/2v1dckQOhnaFSfrRvPqYyFBITi?=
 =?us-ascii?Q?RgSM/QokQosI9k2L9E9j8zCHY/hgzDlt7IzUk+gxJzFz6dpZuz7R8bmjp9+W?=
 =?us-ascii?Q?YGYtjSl7AC232wXlLPjRz1EIaQVX4RwPJaGQP2xxFIwi8VMmMieL6eoWanXq?=
 =?us-ascii?Q?gHPL1sZ9U/1RM0eCyr4HjJqMfbFyHud7U8LQ2UN525YY1NQrrH04/fHwXHcG?=
 =?us-ascii?Q?kNbqfiiqveAAAgOlAucbd+4na4b86CAVLYf5yIGuzaGf/zEip4OQiNN0MZuD?=
 =?us-ascii?Q?Ut1V9H4r9EdrJCpUxQVibXVqtU3FNRyQKCR94UjC/Y1DJ2yHPYEM2bimVi0n?=
 =?us-ascii?Q?7iOe2UWvlR0tugwbg/N/suGcjNoLWHa23J4b8c3zIup8mU6cYZKMia8cPAYL?=
 =?us-ascii?Q?agbJsyfUVwpsyjsqnXOzgFhuwRgPOHnqxPfh+lKmu0Mk+dgUCbctO+/+4en8?=
 =?us-ascii?Q?31A+btrtHVfFvy1KQLlPbFL4bOyhR2eqreWX5Yi/FjaA35mxyUOYANAbDKEg?=
 =?us-ascii?Q?zml0/9pAHwKfzwGTYE1UWoZX7i78ve5sGNyc4BwuLotkAyg7tWbNb3ZRffim?=
 =?us-ascii?Q?6Ja6JCevExgTcZf1X7sn59ru4h9puET7D8g7R+/30ddvIfSrCxFBXdgXLK4q?=
 =?us-ascii?Q?xN1heqN9/F9TjCRa6fLgkD36CkjjzQjI7QYTWGvPv7C1dsF/hWTIkf3b5Ud/?=
 =?us-ascii?Q?WSCT4xL+RUTI8GCBipUgOULnmKVcr8lW455uDJX9TZVeUbMz07f3MMvChVKF?=
 =?us-ascii?Q?OyYWb7HL6PwKCCQIvdeX5ITgZbGCvkS2W1ZhOTE54I47KGxNl6BbztOxZ/PI?=
 =?us-ascii?Q?kLp6iGffMF6cAM1Ljo0hu7ybhqCVzFnJ8uxAAlC4Xds/64wPjfxPlvgasz7D?=
 =?us-ascii?Q?aVkz2MeJ5Fiix/w45SkANNAuwI0kpMHaedXAFwG885Dwwt3XNBGedClBWl7A?=
 =?us-ascii?Q?jcIPN3FE3JVKtQd5nIpgL4pnYnprv3d5Ykx97V2fx95Tpuoy/Ku00CKO0ks8?=
 =?us-ascii?Q?O6ibRgmhi/I99azTSycN5F7WcPYwoZlx9ddKJ92runBgDXWGfzbXHL8a/HED?=
 =?us-ascii?Q?luePWpyLAUSNyncrDCBh+Yww/Vp02tx5coocR7Mt0NLgW9Bhy6D3G04leK/t?=
 =?us-ascii?Q?3A=3D=3D?=
X-OriginatorOrg: jaguarmicro.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0102e1ac-1eab-4cbd-6085-08dc583751a6
X-MS-Exchange-CrossTenant-AuthSource: SEYPR06MB6756.apcprd06.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Apr 2024 01:49:39.7978
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 1e45a5c2-d3e1-46b3-a0e6-c5ebf6d8ba7b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: jxfPuFJ2mGFi3D/kneuFVM/e4S8FSgNcNfxzQS3Labdhh9P4yxLMduntJLfWtuZLQaMwPm+a2AMhafbGW8FJiSOqGOoUYV5mR+UDfwErdYg=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: TY0PR06MB6839

When there is a ctlq and it doesn't require interrupt
callbacks,the original method of calculating vectors
wastes hardware msi or msix resources as well as system
IRQ resources.

When conducting performance testing using testpmd in the
guest os, it was found that the performance was lower compared
to directly using vfio-pci to passthrough the device

In scenarios where the virtio device in the guest os does
not utilize interrupts, the vdpa driver still configures
the hardware's msix vector. Therefore, the hardware still
sends interrupts to the host os. Because of this unnecessary
action by the hardware, hardware performance decreases, and
it also affects the performance of the host os.

Before modification:(interrupt mode)
 32:  0   0  0  0 PCI-MSI 32768-edge    vp-vdpa[0000:00:02.0]-0
 33:  0   0  0  0 PCI-MSI 32769-edge    vp-vdpa[0000:00:02.0]-1
 34:  0   0  0  0 PCI-MSI 32770-edge    vp-vdpa[0000:00:02.0]-2
 35:  0   0  0  0 PCI-MSI 32771-edge    vp-vdpa[0000:00:02.0]-config

After modification:(interrupt mode)
 32:  0  0  1  7   PCI-MSI 32768-edge  vp-vdpa[0000:00:02.0]-0
 33: 36  0  3  0   PCI-MSI 32769-edge  vp-vdpa[0000:00:02.0]-1
 34:  0  0  0  0   PCI-MSI 32770-edge  vp-vdpa[0000:00:02.0]-config

Before modification:(virtio pmd mode for guest os)
 32:  0   0  0  0 PCI-MSI 32768-edge    vp-vdpa[0000:00:02.0]-0
 33:  0   0  0  0 PCI-MSI 32769-edge    vp-vdpa[0000:00:02.0]-1
 34:  0   0  0  0 PCI-MSI 32770-edge    vp-vdpa[0000:00:02.0]-2
 35:  0   0  0  0 PCI-MSI 32771-edge    vp-vdpa[0000:00:02.0]-config

After modification:(virtio pmd mode for guest os)
 32: 0  0  0   0   PCI-MSI 32768-edge   vp-vdpa[0000:00:02.0]-config

To verify the use of the virtio PMD mode in the guest operating
system, the following patch needs to be applied to QEMU:
https://lore.kernel.org/all/20240408073311.2049-1-yuxue.liu@jaguarmicro.com

Signed-off-by: lyx634449800 <yuxue.liu@jaguarmicro.com>
---

V3: delete unused variables and add validation records
V2: fix when allocating IRQs, scan all queues

 drivers/vdpa/virtio_pci/vp_vdpa.c | 35 +++++++++++++++++++------------
 1 file changed, 22 insertions(+), 13 deletions(-)

diff --git a/drivers/vdpa/virtio_pci/vp_vdpa.c b/drivers/vdpa/virtio_pci/vp_vdpa.c
index df5f4a3bccb5..cd3aeb3b8f21 100644
--- a/drivers/vdpa/virtio_pci/vp_vdpa.c
+++ b/drivers/vdpa/virtio_pci/vp_vdpa.c
@@ -160,22 +160,31 @@ static int vp_vdpa_request_irq(struct vp_vdpa *vp_vdpa)
 	struct pci_dev *pdev = mdev->pci_dev;
 	int i, ret, irq;
 	int queues = vp_vdpa->queues;
-	int vectors = queues + 1;
+	int msix_vec, allocated_vectors = 0;
 
-	ret = pci_alloc_irq_vectors(pdev, vectors, vectors, PCI_IRQ_MSIX);
-	if (ret != vectors) {
+	for (i = 0; i < queues; i++) {
+		if (vp_vdpa->vring[i].cb.callback)
+			allocated_vectors++;
+	}
+	allocated_vectors = allocated_vectors + 1;
+
+	ret = pci_alloc_irq_vectors(pdev, allocated_vectors, allocated_vectors,
+								PCI_IRQ_MSIX);
+	if (ret != allocated_vectors) {
 		dev_err(&pdev->dev,
 			"vp_vdpa: fail to allocate irq vectors want %d but %d\n",
-			vectors, ret);
+			allocated_vectors, ret);
 		return ret;
 	}
-
-	vp_vdpa->vectors = vectors;
+	vp_vdpa->vectors = allocated_vectors;
 
 	for (i = 0; i < queues; i++) {
+		if (!vp_vdpa->vring[i].cb.callback)
+			continue;
+
 		snprintf(vp_vdpa->vring[i].msix_name, VP_VDPA_NAME_SIZE,
 			"vp-vdpa[%s]-%d\n", pci_name(pdev), i);
-		irq = pci_irq_vector(pdev, i);
+		irq = pci_irq_vector(pdev, msix_vec);
 		ret = devm_request_irq(&pdev->dev, irq,
 				       vp_vdpa_vq_handler,
 				       0, vp_vdpa->vring[i].msix_name,
@@ -185,23 +194,23 @@ static int vp_vdpa_request_irq(struct vp_vdpa *vp_vdpa)
 				"vp_vdpa: fail to request irq for vq %d\n", i);
 			goto err;
 		}
-		vp_modern_queue_vector(mdev, i, i);
+		vp_modern_queue_vector(mdev, i, msix_vec);
 		vp_vdpa->vring[i].irq = irq;
+		msix_vec++;
 	}
 
 	snprintf(vp_vdpa->msix_name, VP_VDPA_NAME_SIZE, "vp-vdpa[%s]-config\n",
-		 pci_name(pdev));
-	irq = pci_irq_vector(pdev, queues);
+			pci_name(pdev));
+	irq = pci_irq_vector(pdev, msix_vec);
 	ret = devm_request_irq(&pdev->dev, irq,	vp_vdpa_config_handler, 0,
 			       vp_vdpa->msix_name, vp_vdpa);
 	if (ret) {
 		dev_err(&pdev->dev,
-			"vp_vdpa: fail to request irq for vq %d\n", i);
+			"vp_vdpa: fail to request irq for config\n");
 			goto err;
 	}
-	vp_modern_config_vector(mdev, queues);
+	vp_modern_config_vector(mdev, msix_vec);
 	vp_vdpa->config_irq = irq;
-
 	return 0;
 err:
 	vp_vdpa_free_irq(vp_vdpa);
-- 
2.43.0