Extend memory probe interface to support an extra paramter nid,
the reserved memory can be added into this node if node exists.
Add a memory section(128M) to node 3(boots with mem=1024m)
echo 0x40000000,3 > memory/probe
And more we make it friendly, it is possible to add memory to do
echo 3g > memory/probe
echo 1024m,3 > memory/probe
It maintains backwards compatibility.
Another format suggested by Dave Hansen:
echo physical_address=0x40000000 numa_node=3 > memory/probe
it is more explicit to show meaning of the parameters.
CC: Dave Hansen <[email protected]>
Signed-off-by: Shaohui Zheng <[email protected]>
Signed-off-by: Haicheng Li <[email protected]>
Signed-off-by: Wu Fengguang <[email protected]>
---
Index: linux-hpe4/Documentation/ABI/testing/sysfs-devices-memory
===================================================================
--- linux-hpe4.orig/Documentation/ABI/testing/sysfs-devices-memory 2010-11-17 09:00:50.653461798 +0800
+++ linux-hpe4/Documentation/ABI/testing/sysfs-devices-memory 2010-11-17 09:01:10.262838849 +0800
@@ -60,6 +60,23 @@
Users: hotplug memory remove tools
http://www.ibm.com/developerworks/wikis/display/LinuxP/powerpc-utils
+What: /sys/devices/system/memory/probe
+Date: Nov 2010
+Contact: Linux Memory Management list <[email protected]>
+Description:
+ memory probe interface is for memory hotplug emulation. it is a software
+ interface to test memory hotplug. We provide the start address and numa
+ nodes id, it will add a memory section to the specified node.
+
+ Add a memory section(128M) to node 3(boots with mem=1024m)
+ echo 0x40000000,3 > memory/probe
+
+ A more friendly method
+ echo 3g > memory/probe
+ echo 1024m,3 > memory/probe
+
+ Another format suggested by Dave Hansen:
+ echo physical_address=0x40000000 numa_node=3 > memory/probe
What: /sys/devices/system/memoryX/nodeY
Date: October 2009
Index: linux-hpe4/arch/x86/Kconfig
===================================================================
--- linux-hpe4.orig/arch/x86/Kconfig 2010-11-17 09:00:50.673463029 +0800
+++ linux-hpe4/arch/x86/Kconfig 2010-11-17 09:01:10.282838829 +0800
@@ -1276,10 +1276,6 @@
def_bool y
depends on ARCH_SPARSEMEM_ENABLE
-config ARCH_MEMORY_PROBE
- def_bool X86_64
- depends on MEMORY_HOTPLUG
-
config ILLEGAL_POINTER_VALUE
hex
default 0 if X86_32
Index: linux-hpe4/drivers/base/memory.c
===================================================================
--- linux-hpe4.orig/drivers/base/memory.c 2010-11-17 09:00:50.673463029 +0800
+++ linux-hpe4/drivers/base/memory.c 2010-11-17 09:01:10.302838792 +0800
@@ -329,6 +329,9 @@
* will not need to do it from userspace. The fake hot-add code
* as well as ppc64 will do all of their discovery in userspace
* and will require this interface.
+ *
+ * Parameter format 1: physical_address,numa_node
+ * Parameter format 2: physical_address=0x40000000 numa_node=3
*/
#ifdef CONFIG_ARCH_MEMORY_PROBE
static ssize_t
@@ -336,13 +339,53 @@
const char *buf, size_t count)
{
u64 phys_addr;
- int nid;
+ int nid = 0;
int ret;
+ char *p = NULL, *q = NULL;
+ /* format: physical_address=0x40000000 numa_node=3 */
+ p = strchr(buf, '=');
+ if (p != NULL) {
+ *p = '\0';
+ q = strchr(buf, ' ');
+ if (q == NULL) {
+ if (strcmp(buf, "physical_address") != 0)
+ ret = -EPERM;
+ else
+ phys_addr = memparse(p+1, NULL);
+ } else {
+ *q++ = '\0';
+ p = strchr(q, '=');
+ if (strcmp(buf, "physical_address") == 0)
+ phys_addr = memparse(p+1, NULL);
+ if (strcmp(buf, "numa_node") == 0)
+ nid = simple_strtoul(p+1, NULL, 0);
+ if (strcmp(q, "physical_address") == 0)
+ phys_addr = memparse(p+1, NULL);
+ if (strcmp(q, "numa_node") == 0)
+ nid = simple_strtoul(p+1, NULL, 0);
+ }
+ } else { /* physical_address,numa_node */
+ p = strchr(buf, ',');
+ if (p != NULL && strlen(p+1) > 0) {
+ /* nid specified */
+ *p++ = '\0';
+ nid = simple_strtoul(p, NULL, 0);
+ phys_addr = memparse(buf, NULL);
+ } else {
+ phys_addr = memparse(buf, NULL);
+ nid = memory_add_physaddr_to_nid(phys_addr);
+ }
+ }
- phys_addr = simple_strtoull(buf, NULL, 0);
-
- nid = memory_add_physaddr_to_nid(phys_addr);
- ret = add_memory(nid, phys_addr, PAGES_PER_SECTION << PAGE_SHIFT);
+ if (nid < 0 || nid > nr_node_ids - 1) {
+ printk(KERN_ERR "Invalid node id %d(0<=nid<%d).\n", nid, nr_node_ids);
+ ret = -EPERM;
+ } else {
+ printk(KERN_INFO "Add a memory section to node: %d.\n", nid);
+ ret = add_memory(nid, phys_addr, PAGES_PER_SECTION << PAGE_SHIFT);
+ if (ret)
+ count = ret;
+ }
if (ret)
count = ret;
Index: linux-hpe4/mm/Kconfig
===================================================================
--- linux-hpe4.orig/mm/Kconfig 2010-11-17 09:01:10.212839478 +0800
+++ linux-hpe4/mm/Kconfig 2010-11-17 09:01:10.302838792 +0800
@@ -173,6 +173,17 @@
is for cpu hot-add/hot-remove to specified node in software method.
This is for debuging and testing purpose
+config ARCH_MEMORY_PROBE
+ def_bool y
+ bool "Memory hotplug emulation"
+ depends on NUMA_HOTPLUG_EMU
+ ---help---
+ Enable memory hotplug emulation. Reserve memory with grub parameter
+ "mem=N"(such as mem=1024M), where N is the initial memory size, the
+ rest physical memory will be removed from e820 table; the memory probe
+ interface is for memory hot-add to specified node in software method.
+ This is for debuging and testing purpose
+
#
# If we have space for more page flags then we can enable additional
# optimizations and functionality.
--
Thanks & Regards,
Shaohui
On Wed, 2010-11-17 at 10:08 +0800, [email protected] wrote:
> And more we make it friendly, it is possible to add memory to do
>
> echo 3g > memory/probe
> echo 1024m,3 > memory/probe
>
> It maintains backwards compatibility.
>
> Another format suggested by Dave Hansen:
>
> echo physical_address=0x40000000 numa_node=3 > memory/probe
>
> it is more explicit to show meaning of the parameters.
The other thing that Greg suggested was to use configfs. Looking back
on it, that makes a lot of sense. We can do better than these "probe"
files.
In your case, it might be useful to tell the kernel to be able to add
memory in a node and add the node all in one go. That'll probably be
closer to what the hardware will do, and will exercise different code
paths that the separate "add node", "then add memory" steps that you're
using here.
For the emulator, I also have to wonder if using debugfs is the right
was since its ABI is a bit more, well, _flexible_ over time. :)
> + depends on NUMA_HOTPLUG_EMU
> + ---help---
> + Enable memory hotplug emulation. Reserve memory with grub parameter
> + "mem=N"(such as mem=1024M), where N is the initial memory size, the
> + rest physical memory will be removed from e820 table; the memory probe
> + interface is for memory hot-add to specified node in software method.
> + This is for debuging and testing purpose
mem= actually sets the largest physical address that we're trying to
use. If you have a 256MB hole at 768MB, then mem=1G will only get you
768MB of memory. We probably get this wrong in a number of other places
in the documentation, but we might as well get it right here.
Maybe something like:
Enable emulation of hotplug of NUMA nodes. To use this, you
must also boot with the kernel command-line parameter
"mem=N"(such as mem=1024M), where N is the highest physical
address you would like to use at boot. The rest of physical
memory will be removed from firmware tables and may be then be
hotplugged with this feature. This is for debuging and testing
purposes.
Note that you can still examine the original, non-modified
firmware tables in: /sys/firmware/memmap
-- Dave
On Wed, 17 Nov 2010, Dave Hansen wrote:
> The other thing that Greg suggested was to use configfs. Looking back
> on it, that makes a lot of sense. We can do better than these "probe"
> files.
>
> In your case, it might be useful to tell the kernel to be able to add
> memory in a node and add the node all in one go. That'll probably be
> closer to what the hardware will do, and will exercise different code
> paths that the separate "add node", "then add memory" steps that you're
> using here.
>
That seems like a seperate issue of moving the memory hotplug interface
over to configfs and that seems like it will cause a lot of userspace
breakage. The memory hotplug interface can already add memory to a node
without using the ACPI notifier, so what does it have to do with this
patchset?
I think what this patchset really wants to do is map offline hot-added
memory to a different node id before it is onlined. It needs no
additional command-line interface or kconfig options, users just need to
physically hot-add memory at runtime or use mem= when booting to reserve
present memory from being used.
Then, export the amount of memory that is actually physically present in
the e820 but was truncated by mem= and allow users to hot-add the memory
via the probe interface. Add a writeable 'node' file to offlined memory
section directories and allow it to be changed prior to online.
> On Wed, 2010-11-17 at 13:18 -0800, David Rientjes wrote:
> On Wed, 17 Nov 2010, Dave Hansen wrote:
> > The other thing that Greg suggested was to use configfs. Looking back
> > on it, that makes a lot of sense. We can do better than these "probe"
> > files.
> >
> > In your case, it might be useful to tell the kernel to be able to add
> > memory in a node and add the node all in one go. That'll probably be
> > closer to what the hardware will do, and will exercise different code
> > paths that the separate "add node", "then add memory" steps that you're
> > using here.
>
> That seems like a seperate issue of moving the memory hotplug interface
> over to configfs and that seems like it will cause a lot of userspace
> breakage. The memory hotplug interface can already add memory to a node
> without using the ACPI notifier, so what does it have to do with this
> patchset?
I was actually just thinking of the node hotplug interface not using a
'probe' file. But, you make a good point. They _have_ to be tied
together, and doing one via configfs would mean that we probably have to
do the other that way. We wouldn't have to _remove_ the ...memory/probe
interface (breaking userspace), but we would add some redundancy.
> I think what this patchset really wants to do is map offline hot-added
> memory to a different node id before it is onlined. It needs no
> additional command-line interface or kconfig options, users just need to
> physically hot-add memory at runtime or use mem= when booting to reserve
> present memory from being used.
>
> Then, export the amount of memory that is actually physically present in
> the e820 but was truncated by mem=
I _think_ that's already effectively done in /sys/firmware/memmap.
> and allow users to hot-add the memory
> via the probe interface. Add a writeable 'node' file to offlined memory
> section directories and allow it to be changed prior to online.
That would work, in theory. But, in practice, we allocate the mem_map[]
at probe time. So, we've already effectively picked a node at probe.
That was done because the probe is equivalent to the hardware "add"
event. Once the hardware where in the address space the memory is, it
always also knows the node.
But, I guess it also wouldn't be horrible if we just hot-removed and
hot-added an offline section if someone did write to a node file like
you're suggesting. It might actually exercise some interesting code
paths.
-- Dave
On Wed, 17 Nov 2010, Dave Hansen wrote:
> > Then, export the amount of memory that is actually physically present in
> > the e820 but was truncated by mem=
>
> I _think_ that's already effectively done in /sys/firmware/memmap.
>
Ok.
It's a little complicated because we don't export each online node's
physical address range so you have to parse the dmesg to find what nodes
were allocated at boot and determine how much physically present memory
you have that's hidden but can be hotplugged using the probe files.
Adding Aaron Durbin <[email protected]> to the cc because he has a patch
that exports the physical address range of each node in their sysfs
directories.
> > and allow users to hot-add the memory
> > via the probe interface. Add a writeable 'node' file to offlined memory
> > section directories and allow it to be changed prior to online.
>
> That would work, in theory. But, in practice, we allocate the mem_map[]
> at probe time. So, we've already effectively picked a node at probe.
> That was done because the probe is equivalent to the hardware "add"
> event. Once the hardware where in the address space the memory is, it
> always also knows the node.
>
> But, I guess it also wouldn't be horrible if we just hot-removed and
> hot-added an offline section if someone did write to a node file like
> you're suggesting. It might actually exercise some interesting code
> paths.
>
Since the pages are offline you should be able to modify the memmap when
the 'node' file is written and use populate_memnodemap() since that file
is only writeable in an offline state.
On Wed, 2010-11-17 at 14:44 -0800, David Rientjes wrote:
> > That would work, in theory. But, in practice, we allocate the mem_map[]
> > at probe time. So, we've already effectively picked a node at probe.
> > That was done because the probe is equivalent to the hardware "add"
> > event. Once the hardware where in the address space the memory is, it
> > always also knows the node.
> >
> > But, I guess it also wouldn't be horrible if we just hot-removed and
> > hot-added an offline section if someone did write to a node file like
> > you're suggesting. It might actually exercise some interesting code
> > paths.
>
> Since the pages are offline you should be able to modify the memmap when
> the 'node' file is written and use populate_memnodemap() since that file
> is only writeable in an offline state.
It's not just the mem_map[], though. When a section is sitting
"offline", it's pretty much all ready to go, except that its pages
aren't in the allocators. But, all of the other mm structures have
already been modified to make room for the pages. Zones have been added
or modified, pgdats resized, 'struct page's initialized.
Changing the node implies changing _all_ of those, which requires
unrolling most of what happened when the "echo $foo > probe" operation
happened in the first place.
This is all _doable_, but it's not trivial.
-- Dave
On Wed, 17 Nov 2010, Dave Hansen wrote:
> It's not just the mem_map[], though. When a section is sitting
> "offline", it's pretty much all ready to go, except that its pages
> aren't in the allocators. But, all of the other mm structures have
> already been modified to make room for the pages. Zones have been added
> or modified, pgdats resized, 'struct page's initialized.
>
Ok, so let's create an interface that compliments the probe interface that
takes a quantity of memory to be hot-added from the amount of hidden RAM
only after we fake the nodes_add array for each section within that
quantity by calling update_nodes_add() and then looping through for each
section calling add_memory().
On Wed, Nov 17, 2010 at 10:50:07AM -0800, Dave Hansen wrote:
> On Wed, 2010-11-17 at 10:08 +0800, [email protected] wrote:
> > And more we make it friendly, it is possible to add memory to do
> >
> > echo 3g > memory/probe
> > echo 1024m,3 > memory/probe
> >
> > It maintains backwards compatibility.
> >
> > Another format suggested by Dave Hansen:
> >
> > echo physical_address=0x40000000 numa_node=3 > memory/probe
> >
> > it is more explicit to show meaning of the parameters.
>
> The other thing that Greg suggested was to use configfs. Looking back
> on it, that makes a lot of sense. We can do better than these "probe"
> files.
>
> In your case, it might be useful to tell the kernel to be able to add
> memory in a node and add the node all in one go. That'll probably be
> closer to what the hardware will do, and will exercise different code
> paths that the separate "add node", "then add memory" steps that you're
> using here.
>
> For the emulator, I also have to wonder if using debugfs is the right
> was since its ABI is a bit more, well, _flexible_ over time. :)
First, the emulator is just for test purpose, and I believe that only very
few people will use it, so we did not want to take so many modification.
the more you changed, the more bugs you may get. the memory/probe interface
is already enough to test memory hot-add.
Second, if we want to use configfs and debugfs for cpu/memory probe interface,
it should implemented in another series patch since it is not part of the emulator.
We have 8 patches in this patchset now, it is should be very long patch if
want to add all in.
>
> > + depends on NUMA_HOTPLUG_EMU
> > + ---help---
> > + Enable memory hotplug emulation. Reserve memory with grub parameter
> > + "mem=N"(such as mem=1024M), where N is the initial memory size, the
> > + rest physical memory will be removed from e820 table; the memory probe
> > + interface is for memory hot-add to specified node in software method.
> > + This is for debuging and testing purpose
>
> mem= actually sets the largest physical address that we're trying to
> use. If you have a 256MB hole at 768MB, then mem=1G will only get you
> 768MB of memory. We probably get this wrong in a number of other places
> in the documentation, but we might as well get it right here.
>
> Maybe something like:
>
> Enable emulation of hotplug of NUMA nodes. To use this, you
> must also boot with the kernel command-line parameter
> "mem=N"(such as mem=1024M), where N is the highest physical
> address you would like to use at boot. The rest of physical
> memory will be removed from firmware tables and may be then be
> hotplugged with this feature. This is for debuging and testing
> purposes.
>
> Note that you can still examine the original, non-modified
> firmware tables in: /sys/firmware/memmap
>
> -- Dave
I did not aware the memory hole here, good catching.
--
Thanks & Regards,
Shaohui
On Wed, Nov 17, 2010 at 01:18:50PM -0800, David Rientjes wrote:
> On Wed, 17 Nov 2010, Dave Hansen wrote:
>
> > The other thing that Greg suggested was to use configfs. Looking back
> > on it, that makes a lot of sense. We can do better than these "probe"
> > files.
> >
> > In your case, it might be useful to tell the kernel to be able to add
> > memory in a node and add the node all in one go. That'll probably be
> > closer to what the hardware will do, and will exercise different code
> > paths that the separate "add node", "then add memory" steps that you're
> > using here.
> >
>
> That seems like a seperate issue of moving the memory hotplug interface
> over to configfs and that seems like it will cause a lot of userspace
> breakage. The memory hotplug interface can already add memory to a node
> without using the ACPI notifier, so what does it have to do with this
> patchset?
Agree with you, I do not suggest to implement it in this patchset.
>
> I think what this patchset really wants to do is map offline hot-added
> memory to a different node id before it is onlined. It needs no
> additional command-line interface or kconfig options, users just need to
> physically hot-add memory at runtime or use mem= when booting to reserve
> present memory from being used.
I already send out the implementation in another email, you can help to do
a review.
>
> Then, export the amount of memory that is actually physically present in
> the e820 but was truncated by mem= and allow users to hot-add the memory
> via the probe interface. Add a writeable 'node' file to offlined memory
> section directories and allow it to be changed prior to online.
for memory offlining, it is a known diffcult thing, and it is not supported
well in current kernel, so I do not suggest to provide the offline interface
in the emulator, it just take more pains. We can consider to add it when
the memory offlining works well.
--
Thanks & Regards,
Shaohui
On Thu, Nov 18, 2010 at 12:48:50PM +0800, Shaohui Zheng wrote:
> On Wed, Nov 17, 2010 at 01:18:50PM -0800, David Rientjes wrote:
> > Then, export the amount of memory that is actually physically present in
> > the e820 but was truncated by mem= and allow users to hot-add the memory
> > via the probe interface. Add a writeable 'node' file to offlined memory
> > section directories and allow it to be changed prior to online.
>
> for memory offlining, it is a known diffcult thing, and it is not supported
> well in current kernel, so I do not suggest to provide the offline interface
> in the emulator, it just take more pains. We can consider to add it when
> the memory offlining works well.
>
This is all stuff that the memblock API can deal with, I'm not sure why
there seems to be an insistence on wedging all manner of unrelated bits
in to e820. Many platforms using memblock today already offline large
amounts of contiguous physical memory for use in drivers, if you were to
follow this scheme and simply layer a node creation shim on top of that
you would end up with something that is almost entirely generic.
On Wed, Nov 17, 2010 at 2:44 PM, David Rientjes <[email protected]> wrote:
> On Wed, 17 Nov 2010, Dave Hansen wrote:
>
>> > Then, export the amount of memory that is actually physically present in
>> > the e820 but was truncated by mem=
>>
>> I _think_ that's already effectively done in /sys/firmware/memmap.
>>
>
> Ok.
>
> It's a little complicated because we don't export each online node's
> physical address range so you have to parse the dmesg to find what nodes
> were allocated at boot and determine how much physically present memory
> you have that's hidden but can be hotplugged using the probe files.
>
> Adding Aaron Durbin <[email protected]> to the cc because he has a patch
> that exports the physical address range of each node in their sysfs
> directories.
Is this something that is needed upstream? I can post it if that is the case.
Sorry, I don't have a lot of context w.r.t. this thread.
>
>> > and allow users to hot-add the memory
>> > via the probe interface. ?Add a writeable 'node' file to offlined memory
>> > section directories and allow it to be changed prior to online.
>>
>> That would work, in theory. ?But, in practice, we allocate the mem_map[]
>> at probe time. ?So, we've already effectively picked a node at probe.
>> That was done because the probe is equivalent to the hardware "add"
>> event. ?Once the hardware where in the address space the memory is, it
>> always also knows the node.
>>
>> But, I guess it also wouldn't be horrible if we just hot-removed and
>> hot-added an offline section if someone did write to a node file like
>> you're suggesting. ?It might actually exercise some interesting code
>> paths.
>>
>
> Since the pages are offline you should be able to modify the memmap when
> the 'node' file is written and use populate_memnodemap() since that file
> is only writeable in an offline state.
>
On Thu, 18 Nov 2010, Paul Mundt wrote:
> This is all stuff that the memblock API can deal with, I'm not sure why
> there seems to be an insistence on wedging all manner of unrelated bits
> in to e820. Many platforms using memblock today already offline large
> amounts of contiguous physical memory for use in drivers, if you were to
> follow this scheme and simply layer a node creation shim on top of that
> you would end up with something that is almost entirely generic.
>
I don't see why this patchset needs to use the memblock API at all, it
should be built entirely on the generic mem-hotplug API. The only
extension needed is the remapping of removed memory to a new node id (done
on x86 with update_nodes_add()) prior to add_memory() for each arch that
supports onlining new nodes.
On Thu, 18 Nov 2010, Shaohui Zheng wrote:
> > Then, export the amount of memory that is actually physically present in
> > the e820 but was truncated by mem= and allow users to hot-add the memory
> > via the probe interface. Add a writeable 'node' file to offlined memory
> > section directories and allow it to be changed prior to online.
>
> for memory offlining, it is a known diffcult thing, and it is not supported
> well in current kernel, so I do not suggest to provide the offline interface
> in the emulator, it just take more pains. We can consider to add it when
> the memory offlining works well.
>
You're referring to the inability to remove memory sections for
CONFIG_SPARSEMEM_VMEMMAP? You should still able to test the offlining
with other memory models of emulated nodes by using the generic support
already implemented for CONFIG_MEMORY_HOTREMOVE; the short answer is that
it probably shouldn't matter at all since we already support node
hot-remove and the fact that they are emulated nodes isn't really of
interest.
On Fri, 2010-11-19 at 15:51 +0800, Shaohui Zheng wrote:
> the purpose of hotplug emulator is providing a possible solution for cpu/memory
> hotplug testing, the interface upgrading is not part of emulator. Let's forget
> configfs here.
If it's just for testing, you're right, we probably shouldn't go to the
trouble of making a new interface. At the same time, we shouldn't put
something in /sys or configfs that we're not committed to, long-term.
So, not to replace the memory probe file, but _only_ to drive the new
debug-only node hot-add, I think its appropriate place is debugfs.
-- Dave