On a customer system running Xen a boot problem was observed due to
the kernel not respecting the memory size limit imposed by the Xen
hypervisor.
During analysis I found the same problem should be able to occur on
bare metal in case the memory would be limited via the "mem=" boot
parameter.
The system this problem has been observed on has tons of memory
added via PCI. So while in the E820 map the not to be used memory has
been wiped out the additional PCI memory is detected during ACPI scan
and it is added via __add_memory().
This small series tries to repair the issue by testing the imposed
memory limit during the memory hotplug process and refusing to add it
in case the limit is being violated.
I've chosen to refuse adding the complete memory chunk in case the
limit is reached instead of adding only some of the memory, as I
thought this would result in less problems (e.g. avoiding to add
only parts of a 128MB memory bar which might be difficult to remove
later).
Changes in V3:
- patch 2: remember possible max_mem_size value from kernel parameters
- patch 2: set allowed size to end of local E820 map's RAM
Changes in V2:
- patch 1: set initial allowed size to U64_MAX instead -1
- patch 2: set initial allowed size to end of E820 RAM
Juergen Gross (2):
x86: respect memory size limiting via mem= parameter
x86/xen: dont add memory above max allowed allocation
arch/x86/kernel/e820.c | 5 +++++
arch/x86/xen/setup.c | 13 +++++++++++++
drivers/xen/xen-balloon.c | 11 +++++++++++
include/linux/memory_hotplug.h | 2 ++
include/xen/xen.h | 4 ++++
mm/memory_hotplug.c | 6 ++++++
6 files changed, 41 insertions(+)
--
2.16.4
When limiting memory size via kernel parameter "mem=" this should be
respected even in case of memory made accessible via a PCI card.
Today this kind of memory won't be made usable in initial memory
setup as the memory won't be visible in E820 map, but it might be
added when adding PCI devices due to corresponding ACPI table entries.
Not respecting "mem=" can be corrected by adding a global max_mem_size
variable set by parse_memopt() which will result in rejecting adding
memory areas resulting in a memory size above the allowed limit.
Signed-off-by: Juergen Gross <[email protected]>
Acked-by: Ingo Molnar <[email protected]>
---
arch/x86/kernel/e820.c | 5 +++++
include/linux/memory_hotplug.h | 2 ++
mm/memory_hotplug.c | 6 ++++++
3 files changed, 13 insertions(+)
diff --git a/arch/x86/kernel/e820.c b/arch/x86/kernel/e820.c
index 50895c2f937d..e67513e2cbbb 100644
--- a/arch/x86/kernel/e820.c
+++ b/arch/x86/kernel/e820.c
@@ -14,6 +14,7 @@
#include <linux/acpi.h>
#include <linux/firmware-map.h>
#include <linux/sort.h>
+#include <linux/memory_hotplug.h>
#include <asm/e820/api.h>
#include <asm/setup.h>
@@ -881,6 +882,10 @@ static int __init parse_memopt(char *p)
e820__range_remove(mem_size, ULLONG_MAX - mem_size, E820_TYPE_RAM, 1);
+#ifdef CONFIG_MEMORY_HOTPLUG
+ max_mem_size = mem_size;
+#endif
+
return 0;
}
early_param("mem", parse_memopt);
diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index 368267c1b71b..cfd12078172a 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -100,6 +100,8 @@ extern void __online_page_free(struct page *page);
extern int try_online_node(int nid);
+extern u64 max_mem_size;
+
extern bool memhp_auto_online;
/* If movable_node boot option specified */
extern bool movable_node_enabled;
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 124e794867c5..519f9db063ff 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -96,10 +96,16 @@ void mem_hotplug_done(void)
cpus_read_unlock();
}
+u64 max_mem_size = U64_MAX;
+
/* add this memory to iomem resource */
static struct resource *register_memory_resource(u64 start, u64 size)
{
struct resource *res, *conflict;
+
+ if (start + size > max_mem_size)
+ return ERR_PTR(-E2BIG);
+
res = kzalloc(sizeof(struct resource), GFP_KERNEL);
if (!res)
return ERR_PTR(-ENOMEM);
--
2.16.4
Don't allow memory to be added above the allowed maximum allocation
limit set by Xen.
Trying to do so would result in cases like the following:
[ 584.559652] ------------[ cut here ]------------
[ 584.564897] WARNING: CPU: 2 PID: 1 at ../arch/x86/xen/multicalls.c:129 xen_alloc_pte+0x1c7/0x390()
[ 584.575151] Modules linked in:
[ 584.578643] Supported: Yes
[ 584.581750] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 4.4.120-92.70-default #1
[ 584.590000] Hardware name: Cisco Systems Inc UCSC-C460-M4/UCSC-C460-M4, BIOS C460M4.4.0.1b.0.0629181419 06/29/2018
[ 584.601862] 0000000000000000 ffffffff813175a0 0000000000000000 ffffffff8184777c
[ 584.610200] ffffffff8107f4e1 ffff880487eb7000 ffff8801862b79c0 ffff88048608d290
[ 584.618537] 0000000000487eb7 ffffea0000000201 ffffffff81009de7 ffffffff81068561
[ 584.626876] Call Trace:
[ 584.629699] [<ffffffff81019ad9>] dump_trace+0x59/0x340
[ 584.635645] [<ffffffff81019eaa>] show_stack_log_lvl+0xea/0x170
[ 584.642391] [<ffffffff8101ac51>] show_stack+0x21/0x40
[ 584.648238] [<ffffffff813175a0>] dump_stack+0x5c/0x7c
[ 584.654085] [<ffffffff8107f4e1>] warn_slowpath_common+0x81/0xb0
[ 584.660932] [<ffffffff81009de7>] xen_alloc_pte+0x1c7/0x390
[ 584.667289] [<ffffffff810647f0>] pmd_populate_kernel.constprop.6+0x40/0x80
[ 584.675241] [<ffffffff815ecfe8>] phys_pmd_init+0x210/0x255
[ 584.681587] [<ffffffff815ed207>] phys_pud_init+0x1da/0x247
[ 584.687931] [<ffffffff815edb3b>] kernel_physical_mapping_init+0xf5/0x1d4
[ 584.695682] [<ffffffff815e9bdd>] init_memory_mapping+0x18d/0x380
[ 584.702631] [<ffffffff81064699>] arch_add_memory+0x59/0xf0
Signed-off-by: Juergen Gross <[email protected]>
---
arch/x86/xen/setup.c | 13 +++++++++++++
drivers/xen/xen-balloon.c | 11 +++++++++++
include/xen/xen.h | 4 ++++
3 files changed, 28 insertions(+)
diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
index d5f303c0e656..0e770f5e5e8c 100644
--- a/arch/x86/xen/setup.c
+++ b/arch/x86/xen/setup.c
@@ -12,6 +12,7 @@
#include <linux/memblock.h>
#include <linux/cpuidle.h>
#include <linux/cpufreq.h>
+#include <linux/memory_hotplug.h>
#include <asm/elf.h>
#include <asm/vdso.h>
@@ -589,6 +590,14 @@ static void __init xen_align_and_add_e820_region(phys_addr_t start,
if (type == E820_TYPE_RAM) {
start = PAGE_ALIGN(start);
end &= ~((phys_addr_t)PAGE_SIZE - 1);
+#ifdef CONFIG_MEMORY_HOTPLUG
+ /*
+ * Don't allow adding memory not in E820 map while booting the
+ * system. Once the balloon driver is up it will remove that
+ * restriction again.
+ */
+ max_mem_size = end;
+#endif
}
e820__range_add(start, end - start, type);
@@ -748,6 +757,10 @@ char * __init xen_memory_setup(void)
memmap.nr_entries = ARRAY_SIZE(xen_e820_table.entries);
set_xen_guest_handle(memmap.buffer, xen_e820_table.entries);
+#ifdef CONFIG_MEMORY_HOTPLUG
+ xen_saved_max_mem_size = max_mem_size;
+#endif
+
op = xen_initial_domain() ?
XENMEM_machine_memory_map :
XENMEM_memory_map;
diff --git a/drivers/xen/xen-balloon.c b/drivers/xen/xen-balloon.c
index 2acbfe104e46..a67236b02452 100644
--- a/drivers/xen/xen-balloon.c
+++ b/drivers/xen/xen-balloon.c
@@ -37,6 +37,7 @@
#include <linux/mm_types.h>
#include <linux/init.h>
#include <linux/capability.h>
+#include <linux/memory_hotplug.h>
#include <xen/xen.h>
#include <xen/interface/xen.h>
@@ -50,6 +51,10 @@
#define BALLOON_CLASS_NAME "xen_memory"
+#ifdef CONFIG_MEMORY_HOTPLUG
+u64 xen_saved_max_mem_size = 0;
+#endif
+
static struct device balloon_dev;
static int register_balloon(struct device *dev);
@@ -63,6 +68,12 @@ static void watch_target(struct xenbus_watch *watch,
static bool watch_fired;
static long target_diff;
+#ifdef CONFIG_MEMORY_HOTPLUG
+ /* The balloon driver will take care of adding memory now. */
+ if (xen_saved_max_mem_size)
+ max_mem_size = xen_saved_max_mem_size;
+#endif
+
err = xenbus_scanf(XBT_NIL, "memory", "target", "%llu", &new_target);
if (err != 1) {
/* This is ok (for domain0 at least) - so just return */
diff --git a/include/xen/xen.h b/include/xen/xen.h
index 0e2156786ad2..d8f1ab43ab56 100644
--- a/include/xen/xen.h
+++ b/include/xen/xen.h
@@ -46,4 +46,8 @@ struct bio_vec;
bool xen_biovec_phys_mergeable(const struct bio_vec *vec1,
const struct bio_vec *vec2);
+#ifdef CONFIG_MEMORY_HOTPLUG
+extern u64 xen_saved_max_mem_size;
+#endif
+
#endif /* _XEN_XEN_H */
--
2.16.4
> On Feb 14, 2019, at 3:42 AM, Juergen Gross <[email protected]> wrote:
>
> When limiting memory size via kernel parameter "mem=" this should be
> respected even in case of memory made accessible via a PCI card.
>
> Today this kind of memory won't be made usable in initial memory
> setup as the memory won't be visible in E820 map, but it might be
> added when adding PCI devices due to corresponding ACPI table entries.
>
> Not respecting "mem=" can be corrected by adding a global max_mem_size
> variable set by parse_memopt() which will result in rejecting adding
> memory areas resulting in a memory size above the allowed limit.
>
> Signed-off-by: Juergen Gross <[email protected]>
> Acked-by: Ingo Molnar <[email protected]>
Reviewed-by: William Kucharski <[email protected]>
On 2/14/19 5:42 AM, Juergen Gross wrote:
> Don't allow memory to be added above the allowed maximum allocation
> limit set by Xen.
>
> Trying to do so would result in cases like the following:
>
> [ 584.559652] ------------[ cut here ]------------
> [ 584.564897] WARNING: CPU: 2 PID: 1 at ../arch/x86/xen/multicalls.c:129 xen_alloc_pte+0x1c7/0x390()
> [ 584.575151] Modules linked in:
> [ 584.578643] Supported: Yes
> [ 584.581750] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 4.4.120-92.70-default #1
> [ 584.590000] Hardware name: Cisco Systems Inc UCSC-C460-M4/UCSC-C460-M4, BIOS C460M4.4.0.1b.0.0629181419 06/29/2018
> [ 584.601862] 0000000000000000 ffffffff813175a0 0000000000000000 ffffffff8184777c
> [ 584.610200] ffffffff8107f4e1 ffff880487eb7000 ffff8801862b79c0 ffff88048608d290
> [ 584.618537] 0000000000487eb7 ffffea0000000201 ffffffff81009de7 ffffffff81068561
> [ 584.626876] Call Trace:
> [ 584.629699] [<ffffffff81019ad9>] dump_trace+0x59/0x340
> [ 584.635645] [<ffffffff81019eaa>] show_stack_log_lvl+0xea/0x170
> [ 584.642391] [<ffffffff8101ac51>] show_stack+0x21/0x40
> [ 584.648238] [<ffffffff813175a0>] dump_stack+0x5c/0x7c
> [ 584.654085] [<ffffffff8107f4e1>] warn_slowpath_common+0x81/0xb0
> [ 584.660932] [<ffffffff81009de7>] xen_alloc_pte+0x1c7/0x390
> [ 584.667289] [<ffffffff810647f0>] pmd_populate_kernel.constprop.6+0x40/0x80
> [ 584.675241] [<ffffffff815ecfe8>] phys_pmd_init+0x210/0x255
> [ 584.681587] [<ffffffff815ed207>] phys_pud_init+0x1da/0x247
> [ 584.687931] [<ffffffff815edb3b>] kernel_physical_mapping_init+0xf5/0x1d4
> [ 584.695682] [<ffffffff815e9bdd>] init_memory_mapping+0x18d/0x380
> [ 584.702631] [<ffffffff81064699>] arch_add_memory+0x59/0xf0
>
> Signed-off-by: Juergen Gross <[email protected]>
Reviewed-by: Boris Ostrovsky <[email protected]>