Received-SPF: pass (google.com: domain of linux-kernel+bounces-150905-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
From: Dongli Zhang <dongli.zhang@oracle.com>
To: linux-kernel@vger.kernel.org
Cc: virtualization@lists.linux.dev, tglx@linutronix.de, joe.jin@oracle.com
Subject: [PATCH 1/1] genirq/cpuhotplug: retry with online CPUs on irq_do_set_affinity failure
Date: Thu, 18 Apr 2024 18:33:22 -0700
Message-Id: <20240419013322.58500-2-dongli.zhang@oracle.com>
In-Reply-To: <20240419013322.58500-1-dongli.zhang@oracle.com>
References: <20240419013322.58500-1-dongli.zhang@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?Y3HbnoAFWkbbbDei3NbftVj3OhDzSR//NwZxSrmK9ZrPqRtRiRPPiDv5hN9F?=
 =?us-ascii?Q?H8g2zrhcwgk7a7G792ougzKbiwzKeh1PCjS8NNDUPVu5jGgtkZyD4dhIBuBh?=
 =?us-ascii?Q?xAZP9otCUQW4dIlvz+AYWTrJBXE9PN9N1bJfvf4z/5qLQjkVEMzysr8tf8ri?=
 =?us-ascii?Q?FZO91noJCEq9Al0zMfkRPcjzDcAbboqkF2IEfPiUKO3mP6OP3lBi+t9CgO0S?=
 =?us-ascii?Q?cOwMsleemi9K5mRDmlKkd3EnqQ0gh95ltHX8UkDQgt858w1nAqOiMAeHULRa?=
 =?us-ascii?Q?H5Mb9Rhwcdzn935IOO3rQlo+3IhCINwbhaRM1TOyY3drHNJTWSwizSQJyjKp?=
 =?us-ascii?Q?NWM4BQWyj2ZgDxaawLaLCNjmab9sI8js3nOZAICwbct4SxMTmd+GVqbBbZus?=
 =?us-ascii?Q?3Crpqw9sAEV/ZoNGxc6R1jYwQvsuPj8PmKMmiZGeEVr+k+qAd2fmm9KVULfw?=
 =?us-ascii?Q?6ubbtueEv6pkysK4rqOFeCOuFJLwnOZrf6oXnEJ5od4PC3G6+r+1G5ecPuMK?=
 =?us-ascii?Q?nikOxl6kl2HxtNV/mv4FUSxRHSX6nEIRu5PgMclOaSm9TMzrx5Klp/M8t2mR?=
 =?us-ascii?Q?GU7OL3iXV+G0oWqUvIDt/BGiVe4o7+yEhwP/mRpgXsAC4Lr4yxVOGvvinU3f?=
 =?us-ascii?Q?WH41FAn4r9TfleFT98Wo9KMBBR6w4rlpzDjNxVOZiLJxzPn/ikaTYU1Ius2m?=
 =?us-ascii?Q?KP0TnOvoHdEJZm0scgvk7tF+jKfX18j6FXzQuZH0Y12ZwpOeTSks0BnuggC9?=
 =?us-ascii?Q?8XqCeSqv4P+HWg9zDE6R5r4sOAWmIl5QgYXRKjsptENtahYw/fpP727h1kOa?=
 =?us-ascii?Q?cOLkvKjmJHs8UFhbHstMhtsYF8f0rf32iQ1h3nVklDbRj1QM/f/jCUFzMvJX?=
 =?us-ascii?Q?WpvUdaNmbFnLkG8scPoCPMZz1PX21oNnDIc6Rcdx2jqTAel99l+aSFQ9fTxn?=
 =?us-ascii?Q?ZehLLdq+xO6PP9jv88+fXTVXhnaICN2xy/GXZx+YSF71nPqkxEk1cHlqWNre?=
 =?us-ascii?Q?2iotWucMSnfO2GxRMoXkwG2SqEooFbUKqvv6I+Opct0rpGnrGu+G03h29qqO?=
 =?us-ascii?Q?fLmlvuiiHzftzZknqGb11LO6TQKma6hzxDmJiY65DQDSNuKKEu9iZ2ood2cQ?=
 =?us-ascii?Q?QIAVKQaptVQfnO7TBlHlwe3ugecnCXbpybE97wbxglArsIC+OeVVXfIgOugd?=
 =?us-ascii?Q?NZG60rXzKKLQBrHvq6bnGFA0l+oVV+C51VmjqJCi7CEzJCaZafykmyt2HxRR?=
 =?us-ascii?Q?Vd29z/ycRRCXQwD/f2BBd5q3A7B8lRpmRkq8xVNCpcvvnJg0+LGRcNy8T2Ur?=
 =?us-ascii?Q?rDb7DCCVksfcUKbhiX4FLqiBNgBMwlsw96nBnbNJNtlS/xAVHhnyPIFNgAeJ?=
 =?us-ascii?Q?/ns34H1D2UmcjO7sIefdY8qJ/MSJ3R27ZvG4jBbBsPNeJAStboapcIwTi0Yd?=
 =?us-ascii?Q?htOhBA9KjY1Hp4dcwUSpFqhxZ9YJxXr3fk1cDA/kfmpSwtHi6K+CLcAmWla4?=
 =?us-ascii?Q?VmpUGA/5VfX8yyJqeNdAAw3VFH1QyUhuE4gc4KQS+j8QRKr0L76Fpo2CEkUN?=
 =?us-ascii?Q?O2q60caUOdrHjluMJ/g5ABwuFeE+d+Ouf1PQIGuI?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	k/QpTmCU/G521f5yHGu+ZUMM9PhcHfF7oykafMQ1r/tCkZMfPmWsWcvxj/5URw6K3MID6L36/sKuOrqSIwz3OqqgKzAjmkYgvVajb3MInFWpj9L+O1YORDXjNESoiL7QAFEPuWu0SCDPDEDSiWFHzSTU+H4G/Pusd94kswaFWl5/g3tZFdNlnFGcBEzzd93N77QWUXvrtmg/ZLd3wBYMnXguUhA6sF0nEDEtXCiJ+CmojqN+Mek1hZkKkwOQzGjf1pk3PXO+wi7FhzKDWo3OqLsp3PJNNyb865vwlGUfeYxX/JJRnnlKrGHmtdVRrzxRasSSOH0VlxJa7paIW/NbiUCsn/oxbSu/r2UFObmQJ9N1PDsXtK8v7heT7UHIiPaEsmkwQToSkIH/eSqbnICoYc7URf0zkcX8cvM4F/pfHleZFVSSBxSWe2btwdrKc3s8CZBSQc05UzwDyZgfF5JonkPDQKX6/vLm3pyXLWX2kW554KZAMsaRASsBZTUzbBDlAZON32xZcyIxH5YCG/ON9KgGEmhDQeDh90ll4Np4JYRz1I65VNpLB7BjRXugr9Z+CygxS8Dj2qfadh7O/dVH/HuDgDUsmEKgCs0j1xbxdFk=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 25c7f895-8b7f-4a7a-43eb-08dc60110378
X-MS-Exchange-CrossTenant-AuthSource: SA0PR10MB6425.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Apr 2024 01:35:36.9817
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: DQDNxMJUB9QACDoEdOkhHNcbqvYks6BzIUsk5/aPuT0krQ0RV1OVhDKRF1l6w4c+LAXszwL/b9OUYeaMLZzuvg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR10MB7285
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-04-18_22,2024-04-17_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 mlxlogscore=999
 bulkscore=0 adultscore=0 suspectscore=0 phishscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2404010000
 definitions=main-2404190010
X-Proofpoint-ORIG-GUID: HGu7v6CxpHZ9N4m8LjAdOz-7BkTp0y73
X-Proofpoint-GUID: HGu7v6CxpHZ9N4m8LjAdOz-7BkTp0y73

When a CPU is offline, its IRQs may migrate to other CPUs. For managed
IRQs, they are migrated, or shutdown (if all CPUs of the managed IRQ
affinity are offline). For regular IRQs, there will only be a migration.

The migrate_one_irq() first uses pending_mask or affinity_mask of the IRQ.

104         if (irq_fixup_move_pending(desc, true))
105                 affinity = irq_desc_get_pending_mask(desc);
106         else
107                 affinity = irq_data_get_affinity_mask(d);

The migrate_one_irq() may use all online CPUs, if all CPUs in
pending_mask/affinity_mask are already offline.

113         if (cpumask_any_and(affinity, cpu_online_mask) >= nr_cpu_ids) {
114                 /*
115                  * If the interrupt is managed, then shut it down and leave
116                  * the affinity untouched.
117                  */
118                 if (irqd_affinity_is_managed(d)) {
119                         irqd_set_managed_shutdown(d);
120                         irq_shutdown_and_deactivate(desc);
121                         return false;
122                 }
123                 affinity = cpu_online_mask;
124                 brokeaff = true;
125         }

However, there is a corner case. Although some CPUs in
pending_mask/affinity_mask are still online, they are lack of available
vectors. If the kernel continues calling irq_do_set_affinity() with those CPUs,
there will be -ENOSPC error.

This is not reasonable as other online CPUs still have many available vectors.

name:   VECTOR
 size:   0
 mapped: 529
 flags:  0x00000103
Online bitmaps:        7
Global available:    884
Global reserved:       6
Total allocated:     539
System: 36: 0-19,21,50,128,236,243-244,246-255
 | CPU | avl | man | mac | act | vectors
     0   147     0     0   55  32-49,51-87
     1   147     0     0   55  32-49,51-87
     2     0     0     0  202  32-49,51-127,129-235
     4   147     0     0   55  32-49,51-87
     5   147     0     0   55  32-49,51-87
     6   148     0     0   54  32-49,51-86
     7   148     0     0   54  32-49,51-86

This issue should not happen for managed IRQs because the vectors are already
reserved before CPU hotplug. For regular IRQs, do a re-try with all online
CPUs if the prior irq_do_set_affinity() is failed with -ENOSPC.

Cc: Joe Jin <joe.jin@oracle.com>
Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
---
 kernel/irq/cpuhotplug.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/kernel/irq/cpuhotplug.c b/kernel/irq/cpuhotplug.c
index 1ed2b1739363..d1666a6b73f4 100644
--- a/kernel/irq/cpuhotplug.c
+++ b/kernel/irq/cpuhotplug.c
@@ -130,6 +130,19 @@ static bool migrate_one_irq(struct irq_desc *desc)
 	 * CPU.
 	 */
 	err = irq_do_set_affinity(d, affinity, false);
+
+	if (err == -ENOSPC &&
+	    !irqd_affinity_is_managed(d) &&
+	    affinity != cpu_online_mask) {
+		affinity = cpu_online_mask;
+		brokeaff = true;
+
+		pr_debug("IRQ%u: set affinity failed for %*pbl, re-try with all online CPUs\n",
+			 d->irq, cpumask_pr_args(affinity));
+
+		err = irq_do_set_affinity(d, affinity, false);
+	}
+
 	if (err) {
 		pr_warn_ratelimited("IRQ%u: set affinity failed(%d).\n",
 				    d->irq, err);
-- 
2.34.1