Received-SPF: pass (google.com: domain of linux-kernel+bounces-87555-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Thu, 29 Feb 2024 14:07:24 -0600
From: Dimitri Sivanich <sivanich@hpe.com>
To: Dimitri Sivanich <sivanich@hpe.com>, David Woodhouse <dwmw2@infradead.org>,
        Lu Baolu <baolu.lu@linux.intel.com>, Joerg Roedel <joro@8bytes.org>,
        Will Deacon <will@kernel.org>, Robin Murphy <robin.murphy@arm.com>,
        Thomas Gleixner <tglx@linutronix.de>, iommu@lists.linux.dev
Cc: linux-kernel@vger.kernel.org, Steve Wahl <steve.wahl@hpe.com>,
        Russ Anderson <russ.anderson@hpe.com>
Subject: [PATCH] Allocate DMAR fault interrupts locally
Message-ID: <ZeDj/LK56borSxO4@hpe.com>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

The Intel IOMMU code currently tries to allocate all DMAR fault interrupt
vectors on the boot cpu.  On large systems with high DMAR counts this
results in vector exhaustion, and most of the vectors are not initially
allocated socket local.

Instead, have a cpu on each node do the vector allocation for the DMARs on
that node.  The boot cpu still does the allocation for its node during its
boot sequence.

Signed-off-by: Dimitri Sivanich <sivanich@hpe.com>
---
 drivers/iommu/intel/dmar.c | 34 +++++++++++++++++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)

diff --git a/drivers/iommu/intel/dmar.c b/drivers/iommu/intel/dmar.c
index 23cb80d62a9a..41ef72ba7509 100644
--- a/drivers/iommu/intel/dmar.c
+++ b/drivers/iommu/intel/dmar.c
@@ -2108,8 +2108,12 @@ int __init enable_drhd_fault_handling(void)
 	 */
 	for_each_iommu(iommu, drhd) {
 		u32 fault_status;
-		int ret = dmar_set_interrupt(iommu);
+		int ret;
 
+		if (iommu->node != cpu_to_node(smp_processor_id()))
+			continue;
+
+		ret = dmar_set_interrupt(iommu);
 		if (ret) {
 			pr_err("DRHD %Lx: failed to enable fault, interrupt, ret %d\n",
 			       (unsigned long long)drhd->reg_base_addr, ret);
@@ -2192,6 +2196,34 @@ static int __init dmar_free_unused_resources(void)
 
 late_initcall(dmar_free_unused_resources);
 
+#ifdef CONFIG_X86_LOCAL_APIC
+static void __init irq_remap_enable_fault_handling_thr(struct work_struct *work)
+{
+	irq_remap_enable_fault_handling();
+}
+
+static int __init assign_dmar_vectors(void)
+{
+	struct work_struct irq_remap_work;
+	int nid;
+
+	INIT_WORK(&irq_remap_work, irq_remap_enable_fault_handling_thr);
+	cpus_read_lock();
+	for_each_online_node(nid) {
+		/* Boot cpu dmar vectors are assigned before the rest */
+		if (nid == cpu_to_node(get_boot_cpu_id()))
+			continue;
+		schedule_work_on(cpumask_first(cpumask_of_node(nid)),
+				 &irq_remap_work);
+		flush_work(&irq_remap_work);
+	}
+	cpus_read_unlock();
+	return 0;
+}
+
+arch_initcall(assign_dmar_vectors);
+#endif /* CONFIG_X86_LOCAL_APIC */
+
 /*
  * DMAR Hotplug Support
  * For more details, please refer to Intel(R) Virtualization Technology
-- 
2.35.3