When adding CPUs to an already big system (test show it seems to start with
more than 256 CPUs), the kernel is showing error messages when building the
FDT for the kexec kernel (kdump or kexec).
It's worth to mention that the kdump kernel is reloaded after a CPU add
operation.
The messages look like (property's name may vary):
10175.025675] Unable to add 32-64-bridge property: FDT_ERR_NOSPACE
This happens because the size of the FDT is computed based on the size of
the FDT the kernel received at boot time. There is additional space added
in kexec_extra_fdt_size_ppc64() for the added memory but nothing is done
for the added CPUs.
This patch adds this feature so adding new CPUs will increase the size of
the FDT for the kexec kernel.
To compute the additional size required, the number of CPU nodes of the
initial FDT (the one the kernel receive at boot time) are recorded. When a
kexec FDT is created, the number of CPU nodes in the current FDT is used to
compute the additional size.
The first patch of this series is creating a variable provided by the boot
code when parsing the initial FDT at boot time.
The second patch is computing the required additional space.
This has been tested on a PowerVM LPAR running with than 256 CPUs in shared
mode, adding 320 CPUs to this LPAR.
Changes in v2:
- Fix build issue, moving definition in prom.h
Laurent Dufour (2):
powerpc: export the CPU node count
powerpc: Take in account addition CPU node when building kexec FDT
arch/powerpc/include/asm/prom.h | 1 +
arch/powerpc/kernel/prom.c | 3 ++
arch/powerpc/kexec/file_load_64.c | 60 ++++++++++++++++++++++++++++++-
3 files changed, 63 insertions(+), 1 deletion(-)
--
2.38.1
At boot time, the FDT is parsed to compute the number of CPUs.
In addition count the number of CPU nodes and export it.
This is useful when building the FDT for a kexeced kernel since we need to
take in account the CPU node added since the boot time during CPU hotplug
operations.
Signed-off-by: Laurent Dufour <[email protected]>
---
arch/powerpc/include/asm/prom.h | 1 +
arch/powerpc/kernel/prom.c | 3 +++
2 files changed, 4 insertions(+)
diff --git a/arch/powerpc/include/asm/prom.h b/arch/powerpc/include/asm/prom.h
index 2e82820fbd64..c0107d8ddd8c 100644
--- a/arch/powerpc/include/asm/prom.h
+++ b/arch/powerpc/include/asm/prom.h
@@ -85,6 +85,7 @@ struct of_drc_info {
extern int of_read_drc_info_cell(struct property **prop,
const __be32 **curval, struct of_drc_info *data);
+extern unsigned int boot_cpu_node_count;
/*
* There are two methods for telling firmware what our capabilities are.
diff --git a/arch/powerpc/kernel/prom.c b/arch/powerpc/kernel/prom.c
index 1eed87d954ba..645f4450dfc3 100644
--- a/arch/powerpc/kernel/prom.c
+++ b/arch/powerpc/kernel/prom.c
@@ -72,6 +72,7 @@ int __initdata iommu_is_off;
int __initdata iommu_force_on;
unsigned long tce_alloc_start, tce_alloc_end;
u64 ppc64_rma_size;
+unsigned int boot_cpu_node_count __ro_after_init;
#endif
static phys_addr_t first_memblock_size;
static int __initdata boot_cpu_count;
@@ -335,6 +336,8 @@ static int __init early_init_dt_scan_cpus(unsigned long node,
if (type == NULL || strcmp(type, "cpu") != 0)
return 0;
+ boot_cpu_node_count++;
+
/* Get physical cpuid */
intserv = of_get_flat_dt_prop(node, "ibm,ppc-interrupt-server#s", &len);
if (!intserv)
--
2.38.1
On a system with a large number of CPUs, the creation of the FDT for a
kexec kernel may fail because the allocated FDT is not large enough.
When this happens, such a message is displayed on the console:
Unable to add ibm,processor-vadd-size property: FDT_ERR_NOSPACE
The property's name may change depending when the buffer overwrite is
detected.
Obviously the created FDT is missing information, and it is expected that
system dump or kexec kernel failed to run properly.
When the FDT is allocated, the size of the FDT the kernel received at boot
time is used and an extra size can be applied. Currently, only memory added
after boot time is taken in account, not the CPU nodes.
The extra size should take in account these additional CPU nodes and
compute the required extra space. To achieve that, the size of a CPU node,
including its subnode is computed once and multiplied by the number of
additional CPU nodes.
The assumption is that the size of the CPU node is _same_ for all the node,
the only variable part should be the name "PowerPC,POWERxx@##" where "##"
may vary a bit.
Signed-off-by: Laurent Dufour <[email protected]>
---
arch/powerpc/kexec/file_load_64.c | 60 ++++++++++++++++++++++++++++++-
1 file changed, 59 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/kexec/file_load_64.c b/arch/powerpc/kexec/file_load_64.c
index 349a781cea0b..6865cd7dc3ca 100644
--- a/arch/powerpc/kexec/file_load_64.c
+++ b/arch/powerpc/kexec/file_load_64.c
@@ -26,6 +26,7 @@
#include <asm/firmware.h>
#include <asm/kexec_ranges.h>
#include <asm/crashdump-ppc64.h>
+#include <asm/prom.h>
struct umem_info {
u64 *buf; /* data buffer for usable-memory property */
@@ -928,6 +929,46 @@ int setup_purgatory_ppc64(struct kimage *image, const void *slave_code,
return ret;
}
+/**
+ * get_cpu_node_size - Compute the size of a CPU node in the FDT.
+ * This should be done only once and the value is stored in
+ * a static variable.
+ * Returns the max size of a CPU node in the FDT.
+ */
+static unsigned int cpu_node_size(void)
+{
+ static unsigned int cpu_node_size;
+ struct device_node *dn;
+ struct property *pp;
+
+ /*
+ * Don't compute it twice, we are assuming that the per CPU node size
+ * doesn't change during the system's life.
+ */
+ if (cpu_node_size)
+ return cpu_node_size;
+
+ dn = of_find_node_by_type(NULL, "cpu");
+ if (!dn) {
+ /* Unlikely to happen */
+ WARN_ON_ONCE(1);
+ return 0;
+ }
+
+ /*
+ * We compute the sub node size for a CPU node, assuming it
+ * will be the same for all.
+ */
+ cpu_node_size += strlen(dn->name) + 5;
+ for_each_property_of_node(dn, pp) {
+ cpu_node_size += strlen(pp->name);
+ cpu_node_size += pp->length;
+ }
+
+ of_node_put(dn);
+ return cpu_node_size;
+}
+
/**
* kexec_extra_fdt_size_ppc64 - Return the estimated additional size needed to
* setup FDT for kexec/kdump kernel.
@@ -937,7 +978,10 @@ int setup_purgatory_ppc64(struct kimage *image, const void *slave_code,
*/
unsigned int kexec_extra_fdt_size_ppc64(struct kimage *image)
{
+ struct device_node *dn;
u64 usm_entries;
+ unsigned int cpu_nodes = 0;
+ unsigned int extra_size;
if (image->type != KEXEC_TYPE_CRASH)
return 0;
@@ -949,7 +993,21 @@ unsigned int kexec_extra_fdt_size_ppc64(struct kimage *image)
*/
usm_entries = ((memblock_end_of_DRAM() / drmem_lmb_size()) +
(2 * (resource_size(&crashk_res) / drmem_lmb_size())));
- return (unsigned int)(usm_entries * sizeof(u64));
+
+ extra_size = (unsigned int)(usm_entries * sizeof(u64));
+
+ /*
+ * Get the number of CPU nodes in the current DT. This allows to
+ * reserve places for CPU nodes added since the boot time.
+ */
+ for_each_node_by_type(dn, "cpu") {
+ cpu_nodes++;
+ }
+
+ if (cpu_nodes > boot_cpu_node_count)
+ extra_size += (cpu_nodes - boot_cpu_node_count) * cpu_node_size();
+
+ return extra_size;
}
/**
--
2.38.1
On Thu, 10 Nov 2022 19:06:17 +0100, Laurent Dufour wrote:
> When adding CPUs to an already big system (test show it seems to start with
> more than 256 CPUs), the kernel is showing error messages when building the
> FDT for the kexec kernel (kdump or kexec).
>
> It's worth to mention that the kdump kernel is reloaded after a CPU add
> operation.
>
> [...]
Applied to powerpc/next.
[1/2] powerpc: export the CPU node count
https://git.kernel.org/powerpc/c/e13d23a404f2e6dfaf8b1ef7d161a0836fce4fa5
[2/2] powerpc: Take in account addition CPU node when building kexec FDT
https://git.kernel.org/powerpc/c/340a4a9f8773e102cc5ef531665970a686dfa245
cheers