Received-SPF: pass (google.com: domain of linux-kernel+bounces-49314-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223;
Date: Fri, 2 Feb 2024 14:55:33 +0800
From: Chen Yu <yu.c.chen@intel.com>
To: Thomas Gleixner <tglx@linutronix.de>, Ming Lei <ming.lei@redhat.com>, "Rei
 Yamamoto" <yamamoto.rei@jp.fujitsu.com>
CC: <linux-kernel@vger.kernel.org>, Sasha Neftin <sasha.neftin@intel.com>,
	"Lifshits, Vitaly" <vitaly.lifshits@intel.com>, "Ruinskiy, Dima"
	<dima.ruinskiy@intel.com>
Subject: Managed interrupt spread based on Numa locality
Message-ID: <ZbyR5RCFWDORmkBk@chenyu5-mobl2>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?pj9mq2X+aON1MVRIqzbL14vA3BcXeL7+zUZMQyKPpu3f7q7vOx/KyarkuUko?=
 =?us-ascii?Q?k/Muw1R5W6hwt0H1Ys4HAUaGiKmWLOnFLyqqZ+X+K/FQ2sL6w6Mhc24SA7SC?=
 =?us-ascii?Q?yzwAIhCSoqKQXAacZlxOtutnte57wgxSb0UMvROO0E6pp2Ji1lw4kFo/Esk3?=
 =?us-ascii?Q?lI+Ubf/MyNB4V8shC7wZoGzGVIqbjGLTvbiIaMO3592Pv4zdF4ZDNobTDhkz?=
 =?us-ascii?Q?1ZILbfXy1CrBYIxkRn3/Eb2cv4PZPnqn83sBhRpaO0FUF1CdNl8wyeEdkzNP?=
 =?us-ascii?Q?TXWbYEuJJhIVHDyAa6+Y9iAo5DJJLJPsRa0GTF57j7cneOe1brQXcbFMGHGa?=
 =?us-ascii?Q?GZRe2Rtpf0kLGEMkjPoaB3OX4wUrwefW43uwgi1/MBVQSSSV3gO5CYGov2V8?=
 =?us-ascii?Q?vjmXgsGjWPCkxXlo3Qa2CW/HjlslVQIRsI7dfkHoudIfyJEx37DKKdYWF2IB?=
 =?us-ascii?Q?IvLx33XGqGwq2uwN5bQTC2FICoo27fUZLUmF1+ICSEHoYFhweZA3zW/JEzvd?=
 =?us-ascii?Q?7aRsnB6YSwUyKIagPphFJHGcFzvd/bPzZ+RUh/21nMhcPtdtikkxhVzPZFCh?=
 =?us-ascii?Q?tM0i7jfnYf9SRgR/76Rl77vdA95ItqRwKltkAZrA9RG+Y08PMlt+CmE9ZX5J?=
 =?us-ascii?Q?0ODvbBtPJjzOADEm3CDHeG77eymw7FSNRZnKrwu/e2eI7ooWoFCDwT+6fYuP?=
 =?us-ascii?Q?8Q6u8FhpM6JmjwOs3rpnGAUM5sUNMLtL1RfjS1ZCul28eCmAABYjK/8uJR4o?=
 =?us-ascii?Q?mR4K0U4tXrrfoggLlZ5CxNBnJj9uCSJBAXVImt+1tXlmM7Pge0+X0IyX+5Zl?=
 =?us-ascii?Q?SQlhgdYkxgKKyi3BUaWT8t11rhzXBxl82TP4f3ZRj1ZAH7K2xJ8dOivZ/daS?=
 =?us-ascii?Q?6xsRRdDIJKYDUtzpj8eFZvyDHecZ99U+dg0JhBOczXAhZ8Hf+q5+XMDOkKhc?=
 =?us-ascii?Q?CYh8jwCZhyUnVMMBrJt9hzXldWhIcRyXfI8z+YFmnAkRXidY4F6HZneEOUr6?=
 =?us-ascii?Q?yf4V6wtEABg4YUZvacYa8MyqeAAHE2P1I/nYl45HRtQrIZw935hfXSrY5jWV?=
 =?us-ascii?Q?uUzIr13Jf4GPOOvpC6sHtGTCyOsRav5heCaE+QOG5p0Hg5iSa5B5CNyZTOsD?=
 =?us-ascii?Q?eCozamB0kvNZGCW0rz26+oaNPpiVsVxyMxGQETJ2WY25o3A0wpSQJNYm5pUx?=
 =?us-ascii?Q?6F4wih5ZLfz0k4jlZUnPwvYt05outQ8D1g8xFaG5a0iBFB6Mi8U+SaT6Jyf6?=
 =?us-ascii?Q?SlNRrCLmPlimhqJBJJApbOPac++pRi8vj/qHp1WLYqpXFhxpxITdtkIg+vlI?=
 =?us-ascii?Q?maLGoln5cbVnV27apm4QLyZ4BIejSEag2O701SMC4WsxkIia4IO7lQw3M7UX?=
 =?us-ascii?Q?jGVhAk4ll9ilHC3TG4LxcwvgqWzlEMIdqzekJNritURbGXq5dn3s0QARIqD4?=
 =?us-ascii?Q?prhobxREbBNYMvHdhdwp6ZdJUWE+jVACrorHJxSCAuUHpIgO99pF3YBbLtNJ?=
 =?us-ascii?Q?jSr4VUIDFlBH4F3rSLH8t/m74bpmNqqx6FcZi2SDS8AVREldmxyIl3miYkBy?=
 =?us-ascii?Q?87Ts9DmyW5QJASL7VpbIFKIhPkXUNV0tmRzxPpLz?=
X-MS-Exchange-CrossTenant-Network-Message-Id: c7838dc3-cea3-4c2d-8e44-08dc23bbfb5d
X-MS-Exchange-CrossTenant-AuthSource: DM4PR11MB6020.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Feb 2024 06:55:46.3866
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: sN+Y2ihbasSrYs2sa2wQmcKzzYAcXYu6VW5qGv+dDOinWzHYuBavMz0KZic10F6hQit44lodcY3ehbKweswKdw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR11MB7383
X-OriginatorOrg: intel.com

Dear experts,
 
Recently we are evaluating some multi-queue NIC device drivers,
to switch them from conventional interrupt to managed interrupt.

In this way, the managed interrupts do not have to be migrated
during CPU offline, when the last CPU of the cpumask is offline.

This can save the space of vectors and help the hibernation put
every nonboot CPUs offline. Otherwise, an error would occur:
[48175.409994] CPU 239 has 165 vectors, 36 available. Cannot disable CPU

However after switching to the managed interrupt, there is a question
about the interrupt spreading among numa nodes.

If the device d1 is attached to node n1, can d1's managed interrupts
be allocated on the CPUs of n1 first? In this way, we can let
the driver of d1 to allocate buffer on n1, and with the managed
interrupt of d1 delivered to CPUs on n1, the path of DMA->DRAM->CPU
->net_rx_action() would be Numa friendly.

Question:
Does it make sense to make the interrupt spreading aware of numa
locality, or is there existing mechanism to do this? The driver can provide
the preferred numa node in the struct irq_affinity->node, and passed it to
the managed interrupt spreading logic, then the interrupts are spread within
that node.

Thanks in advance.

diff --git a/lib/group_cpus.c b/lib/group_cpus.c
index aa3f6815bb12..836e9d374c19 100644
--- a/lib/group_cpus.c
+++ b/lib/group_cpus.c
@@ -344,7 +344,7 @@ static int __group_cpus_evenly(unsigned int startgrp, unsigned int numgrps,
  * We guarantee in the resulted grouping that all CPUs are covered, and
  * no same CPU is assigned to multiple groups
  */
-struct cpumask *group_cpus_evenly(unsigned int numgrps)
+struct cpumask *group_cpus_evenly(unsigned int numgrps, int node)
 {
 	unsigned int curgrp = 0, nr_present = 0, nr_others = 0;
 	cpumask_var_t *node_to_cpumask;
@@ -370,9 +370,14 @@ struct cpumask *group_cpus_evenly(unsigned int numgrps)
 	cpus_read_lock();
 	build_node_to_cpumask(node_to_cpumask);
 
+	if (node != NUMA_NO_NODE)
+		cpumask_and(npresmsk, cpu_present_mask, node_to_cpumask[node]);
+	else
+		cpumask_copy(npresmsk, cpu_present_mask);
+
 	/* grouping present CPUs first */
 	ret = __group_cpus_evenly(curgrp, numgrps, node_to_cpumask,
-				  cpu_present_mask, nmsk, masks);
+				  npresmsk, nmsk, masks);
 	if (ret < 0)
 		goto fail_build_affinity;
 	nr_present = ret;
-- 
2.25.1