Received-SPF: pass (google.com: domain of linux-kernel+bounces-154610-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
From: Dongli Zhang <dongli.zhang@oracle.com>
To: linux-kernel@vger.kernel.org
Cc: virtualization@lists.linux.dev, tglx@linutronix.de, joe.jin@oracle.com
Subject: [PATCH v2 1/1] genirq/cpuhotplug: retry with cpu_online_mask when irq_do_set_affinity return -ENOSPC
Date: Tue, 23 Apr 2024 00:34:13 -0700
Message-Id: <20240423073413.79625-1-dongli.zhang@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?OO5CWzPqn3YPhL+eRpraWd6LbMBfOYtKaLrWDO3KHAXXIrBYej07X3mlACyV?=
 =?us-ascii?Q?0ZDFkMgudcI1myC2cWkBpt+kMQ/wNSlHWLyrc2RSvBARsqlqbkjS6KwAV61x?=
 =?us-ascii?Q?kJvJNF8/u9dKrkTKqlwm5Hcit9GXnIn3ePEPAZivz2z9PbyQ3bYeV2k8E562?=
 =?us-ascii?Q?8njd9mqVgfITgPxUi4eYhHKi1bF3R9e+DCFTL/M8tv44+MuJSkEFWAS099TB?=
 =?us-ascii?Q?qSdfvdW5Enka1n/933J/mzWaDD9S/rr26hnUnlfiTI5gWF05DROWoqX951V+?=
 =?us-ascii?Q?jthgTe6jH8yGpRj8P2SDOtAEl9tpVBQMrZbxu30SOngOpfL2SQqCg8R6mhjX?=
 =?us-ascii?Q?3uSy2EaEbA/tIPlSrkgzdyXnYlVvzW7vL7JBjmALR1CcbEX0N1hqArNDrzXb?=
 =?us-ascii?Q?CagX/9B17CY8hgmXHSg7SCiTFJBLmV+ZeFM8w4NC15dPeLmxghcyYFsNDTJU?=
 =?us-ascii?Q?pNJPppMNUyFuD9X5w99yLm279UULiZcKBBuxhjq/sZYfKQBwIkJRQ12NRCkQ?=
 =?us-ascii?Q?RC4EdJgczEO5Z9CB3gqnLTYamie7NAJ6Di5qxkWbro47kul7p7RcNTCfC3Sa?=
 =?us-ascii?Q?8k1fAUo4pA6PLlkh68BnGnCOwn8aeJns3oCQjCpFz45atE0l2npR3gyxAtqF?=
 =?us-ascii?Q?MFUJuW4N9xxC4adDvZ5LhaTy0wMVrHSLQ3rFaPV/mKgUOz9q6Icm0amgh2ee?=
 =?us-ascii?Q?vB38l5yurhlemmglTWRt6clGYy8KYiEkjCrMYt/WDkZca+RyinQkti3MQ0DI?=
 =?us-ascii?Q?IPzx6Djac32aqj6UV/dL0CRqA2kLGFCv0NGHzq21re9jYEJGJaGKD4N0yEdT?=
 =?us-ascii?Q?L8HYNyX7eiNb7mOb82gUB6iuhtAb06bYY9L+bPS/7/WZAhrPN7XlYfiXH6MQ?=
 =?us-ascii?Q?f7NDyYKLXW8EymkJ2RaCyWxhYrep6hI+zi0iv6fwWQIHHO+5sJMQckxSkTOW?=
 =?us-ascii?Q?frCxdXL7j0fZSawe8ykCuv7/sMdvxkPzVBFBW4aQBX4+Le77GA0Lx5yThApL?=
 =?us-ascii?Q?QDcpuMugoRkzYJzezQBbN19efL8IC26W3rIZxM8IuhYBQbn3y8bOWQuFtldn?=
 =?us-ascii?Q?dPVuiUiVxQc+wOFYOdQZsv726qquBQa3NsVbNFrBmlHWLtI+LxUOWnJJ2Od1?=
 =?us-ascii?Q?8bjvlcg9crxGB4igOoKmH+/Dh0PES4lRkCOmpYteb3aiE/GPtLJAl28b8jcj?=
 =?us-ascii?Q?iRPrN4t7+fXIFyFRWRnKB10zVwev7imS2/ubF+aYIVH86yydwH0q+N8QF35H?=
 =?us-ascii?Q?VXEWMSkzNZ+N4Oz8SHIUmew9OAcqcUiGC8zVgeNiOxUEtGr1aFBySn4wS9Ly?=
 =?us-ascii?Q?fbdxrKJUP0/5NmVevKsLnlxtXKlw5Y2DpooEj8iAPssH3AKhcsWFHTDswrMq?=
 =?us-ascii?Q?LBjX4wKcdVSY2vRjukiQfusUu147kxx378x4KWydH6qkIlt5lIVVjSBuVXGh?=
 =?us-ascii?Q?sSJUhi+8fXSsrss3vLUmL38gwiyzvsls6TC/oVqV/MpuGZpW4RMqgxSGmMDe?=
 =?us-ascii?Q?3xW5JeA/DKKxnzyWMkt3r9cnx8YFOujXjZQCjgEDLOMqPeTcfgePLA2Rj8vO?=
 =?us-ascii?Q?i+MHNc0MD/juKO7iuy5SDqLO3BR31jtl4ume7+Uk?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	j6bm/aurQSfal1zDMu5iBReDQlJtrGBmU8SHRdCV9Z9rfuZtCihBrw9rFx/hLXYQvd8hQC2xfQcgm7D6ErHDihZbyC9A9hTnvdOqGcKql1IySpelfWIgVbQLG6+hVkIDCgyHh/1qHu4qlMgkOMFS9Xuj3oKq9IITQzCkQvJ5HkSP8fg0Lln2TJAByqZu4PDkVovGjzRfnpKotJlndntGK72uYD+NwKo1sR0c5ZVbTv2mqYbHtsI29i0+457Xi9yt/2DpcHZ4q8QvE+QlzW47y+/UZpYk+tJRHqahqYqmfPxCU5F432dcEeno0nBnFwG/Tw0GNsZ5Mlv/FgTRO0nH5XSztPkvEuRLCvm585IrI2HZC+CZ6CMMPDCu89QuRZ1qfDitEkK3poFnVTf0T9VGATN9BgNoBDtEmQHsbIAijiN2FfLuPwuHV5qkZx0pmO4c/TTxYtgHI5OOt0o6LugplahdY/Wg5YWkQmEepuNEW6Ncd8iv7GCYM+01oEGWcrs+y00prn25L50rtHuZ8BeZ4RG1l8mVofsK40ztZCwavXsHmnHh5DhO6zYaFjSc+AhWfuOjU3TC5dD1oPVsficBEPRXV/Jotgfw405YvUNhb/Y=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: e062ca41-9148-4f50-5fa4-08dc6368287d
X-MS-Exchange-CrossTenant-AuthSource: SA0PR10MB6425.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Apr 2024 07:36:58.7897
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: cKgj8fWxskbgUwZ4F0mwYJkjbT90ufm3jBDPI3Yf6ffhObkd0oYX7qOb24R3QnPKfQnw411Gy3dihvCC0h8Oyw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN4PR10MB5798
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-04-23_04,2024-04-22_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 spamscore=0
 bulkscore=0 suspectscore=0 mlxlogscore=999 adultscore=0 phishscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2404010000
 definitions=main-2404230021
X-Proofpoint-ORIG-GUID: mNZaVbk5gsvYW9pX0NW6ypriaSXny78Y
X-Proofpoint-GUID: mNZaVbk5gsvYW9pX0NW6ypriaSXny78Y

When a CPU goes offline, the interrupts pinned to that CPU are
re-configured.

Its managed interrupts undergo either migration to other CPUs or shutdown
if all CPUs listed in the affinity are offline. This patch doesn't affect
managed interrupts.

For regular interrupts, they are migrated to other selected online CPUs.
The target CPUs are chosen from either desc->pending_mask (suppose
CONFIG_GENERIC_PENDING_IRQ) or d->common->affinity (suppose CONFIG_SMP).
The cpu_online_mask is used as target CPUs only when CPUs in both
desc->pending_mask and d->common->affinity are offline.

However, there is a bad corner case, when desc->pending_mask or
d->common->affinity is selected as the target cpumask, but none of their
CPUs has any available vectors.

In this case the migration fails and the device interrupt becomes
stale. This is not any different from the case where the affinity
mask does not contain any online CPU, but there is no fallback
operation for this.

Instead of giving up, retry the migration attempt with the online CPU
mask if the interrupt is not managed, as managed interrupts cannot be
affected by this problem.

Cc: Joe Jin <joe.jin@oracle.com>
Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
[tglx: massage some changelog]
---
Changed since v1:
  - Re-work the commit message
  - Move pr_debug before setting affinity
  - Remove 'all' from pr_debug message

 kernel/irq/cpuhotplug.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/kernel/irq/cpuhotplug.c b/kernel/irq/cpuhotplug.c
index 1ed2b1739363..19babb914949 100644
--- a/kernel/irq/cpuhotplug.c
+++ b/kernel/irq/cpuhotplug.c
@@ -130,6 +130,17 @@ static bool migrate_one_irq(struct irq_desc *desc)
 	 * CPU.
 	 */
 	err = irq_do_set_affinity(d, affinity, false);
+
+	if (err == -ENOSPC && !irqd_affinity_is_managed(d) && affinity != cpu_online_mask) {
+		pr_debug("IRQ%u: set affinity failed for %*pbl, re-try with online CPUs\n",
+			 d->irq, cpumask_pr_args(affinity));
+
+		affinity = cpu_online_mask;
+		brokeaff = true;
+
+		err = irq_do_set_affinity(d, affinity, false);
+	}
+
 	if (err) {
 		pr_warn_ratelimited("IRQ%u: set affinity failed(%d).\n",
 				    d->irq, err);
-- 
2.34.1