Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Pingfan Liu <kernelfans@gmail.com>
To:     linux-kernel@vger.kernel.org
Cc:     Pingfan Liu <kernelfans@gmail.com>,
        Yury Norov <yury.norov@gmail.com>,
        Andy Shevchenko <andriy.shevchenko@linux.intel.com>,
        Rasmus Villemoes <linux@rasmusvillemoes.dk>,
        Thomas Gleixner <tglx@linutronix.de>,
        Steven Price <steven.price@arm.com>,
        Mark Rutland <mark.rutland@arm.com>,
        "Jason A. Donenfeld" <Jason@zx2c4.com>,
        Kuppuswamy Sathyanarayanan 
        <sathyanarayanan.kuppuswamy@linux.intel.com>
Subject: [RFC 07/10] lib/cpumask: Introduce cpumask_not_dying_but()
Date:   Mon, 22 Aug 2022 10:15:17 +0800
Message-Id: <20220822021520.6996-8-kernelfans@gmail.com>
In-Reply-To: <20220822021520.6996-1-kernelfans@gmail.com>
References: <20220822021520.6996-1-kernelfans@gmail.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk

During cpu hot-removing, the dying cpus are still in cpu_online_mask.
On the other hand, A subsystem will migrate its broker from the dying
cpu to a online cpu in its teardown cpuhp_step.

After enabling the teardown of cpus in parallel, cpu_online_mask can not
tell those dying from the real online.

Introducing a function cpumask_not_dying_but() to pick a real online
cpu.

Signed-off-by: Pingfan Liu <kernelfans@gmail.com>
Cc: Yury Norov <yury.norov@gmail.com>
Cc: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Cc: Rasmus Villemoes <linux@rasmusvillemoes.dk>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Steven Price <steven.price@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: "Jason A. Donenfeld" <Jason@zx2c4.com>
Cc: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
To: linux-kernel@vger.kernel.org
---
 include/linux/cpumask.h |  3 +++
 kernel/cpu.c            |  3 +++
 lib/cpumask.c           | 18 ++++++++++++++++++
 3 files changed, 24 insertions(+)

diff --git a/include/linux/cpumask.h b/include/linux/cpumask.h
index 0d435d0edbcb..d2033a239a07 100644
--- a/include/linux/cpumask.h
+++ b/include/linux/cpumask.h
@@ -317,6 +317,9 @@ unsigned int cpumask_any_but(const struct cpumask *mask, unsigned int cpu)
 	return i;
 }
 
+/* for parallel kexec reboot */
+int cpumask_not_dying_but(const struct cpumask *mask, unsigned int cpu);
+
 #define CPU_BITS_NONE						\
 {								\
 	[0 ... BITS_TO_LONGS(NR_CPUS)-1] = 0UL			\
diff --git a/kernel/cpu.c b/kernel/cpu.c
index 90debbe28e85..771e344f8ff9 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -1282,6 +1282,9 @@ static void cpus_down_no_rollback(struct cpumask *cpus)
 	struct cpuhp_cpu_state *st;
 	unsigned int cpu;
 
+	for_each_cpu(cpu, cpus)
+		set_cpu_dying(cpu, true);
+
 	/* launch ap work one by one, but not wait for completion */
 	for_each_cpu(cpu, cpus) {
 		st = per_cpu_ptr(&cpuhp_state, cpu);
diff --git a/lib/cpumask.c b/lib/cpumask.c
index 8baeb37e23d3..6474f07ed87a 100644
--- a/lib/cpumask.c
+++ b/lib/cpumask.c
@@ -7,6 +7,24 @@
 #include <linux/memblock.h>
 #include <linux/numa.h>
 
+/* Used in parallel kexec-reboot cpuhp callbacks */
+int cpumask_not_dying_but(const struct cpumask *mask,
+					   unsigned int cpu)
+{
+	unsigned int i;
+
+	if (CONFIG_SHUTDOWN_NONBOOT_CPUS) {
+		cpumask_check(cpu);
+		for_each_cpu(i, mask)
+			if (i != cpu && !cpumask_test_cpu(i, cpu_dying_mask))
+				break;
+		return i;
+	} else {
+		return cpumask_any_but(mask, cpu);
+	}
+}
+EXPORT_SYMBOL(cpumask_not_dying_but);
+
 /**
  * cpumask_next_wrap - helper to implement for_each_cpu_wrap
  * @n: the cpu prior to the place to search
-- 
2.31.1