I would like to start discussion about balancing the occupancy of
memory zones in a node in the system whose imabalance may be caused by
migration of pages to other zones during hotremove and then hotadding
same memory. In this case there is a lot of free memory in newly hotadd
memory which can be filled up by the previous migrated pages(as part of
offline/hotremove) thus may free up some pressure in other zones of the
node.
Say that in system has 2 zones(Normal and Movable), where Normal zone is
almost filled up by the pages of the movable zone as part of offline
operation and then we hot add a memory node as movable zone. At this
moment, Movable zone is almost empty and Normal zone is almost filled
up(by the migrated pages as part of previous offline/hot-remove
operation). At this point of time, allocation requests from Normal zone
may have to go through reclaim cycles thus cause some pressure. This
problem is quite common in the system where they aggressively offline
and online the memory blocks in the system where the former part do the
migration of pages to the lower zones and the later part don't reverse
them and as a result the offline operation may contribute to the
pressure in other zones of the system.
To overcome this problem, we can do the reverse of what offline
operation did, after onlining the memory block i.e. **try to reverse
migrate the pages from other zones which were migrated as part of the
offline operation**. This may freeup some pressure built in the other
zones because of offline operation. Since we are reverse migrating the
pages in the system, we can name this as "reverse migration feature" or
since we are actually balancing the occupancy of the zones in the system
by migrating the pages we can name it as "balancing the system zones
occupancy" or any name...
We have the proof-of-concept code tried on the Snapdragon systems with
the system configuration, single memory node of just 2 zones, 6GB normal
zone and 2GB movable zone. And this Movable zone is such that hot-added
once and there after offline/online based on the need.
We run the below unit test to evalutate this:
1) Completely fill up the already hot added movable zone with anon
pages
2) Offline this hot added movable zone. At this point there is no
pressure in the normal zone yet, but this migration of pages can
contribute to it in the future.
3) Now fill up the normal zone such that there is 300MB or less left in
the system.
4) And now the user onlined the movable zone memory blocks in the
system.
5) Run the tests of allocating 512MB of memory from the normal zone and
in doing so try allocating the higher order pages first and then
gradually fall back to lower orders. I took the help from ion system heap
memory allocation which try to allocate the memory in available orders:
9, 4 and 0.
6) Repeat the above steps for 10 iterations and below is the average of
the results.
We did try to collect the time it takes to complete the tests and the
distribution of anon pages(are the ones participated in the tests) in
the system node zones.
a) With out the reverse migration, it took an average of around 145msec
to complete the test.
b) With the reverse migration, it took an average of the 120msec to
complete the tests.
For distribution of the anon pages in the system we did try collect the
anon pages left in the individual zone before and after the test:
------------------------------------- |-------------------------------
Base | Reverse Migration
--------------------------------------|-------------------------------
Beforetest Aftertest | Beforetest Aftertest
Normal zone(Anon) |
Active 499825 45756 | 481441 203124
Inactive 46008 446687 | 51553 58602
Free 80350 224252 | 84877 **440586**
Movable zone(Anon) |
Active 2224 2626 | 2239 **484622**
Inactive 8 8 | 9 7663
--------------------------------------|-------------------------------
The above table shows that, On base case(left column), there exists a
lot of anon pages in the system which can be migrated back to Movable
zone(almost totally free), thus may freeup some space in the normal
zone. With the reverse migration(Right coloumn), we see that the anon
pages are evenly distributed in the system and lot of free memory
left in the normal zones caused by the reverse migration.
The code shows the PoC by assuming just 2 zones(normal and Movable) of a
single node in the system. The number of pages to be reverse migrated is
written on the sysfs interface from the userspace by monitoring the
memory pressure events in the system.
Charan Teja Reddy (1):
mm: proof-of-concept for balancing node zones occupancy
include/linux/migrate.h | 8 +-
include/linux/mm.h | 3 +
include/linux/mmzone.h | 2 +
kernel/sysctl.c | 11 ++
mm/compaction.c | 4 +-
mm/memory_hotplug.c | 265 ++++++++++++++++++++++++++++++++++++++++++++++++
6 files changed, 290 insertions(+), 3 deletions(-)
--
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a
member of the Code Aurora Forum, hosted by The Linux Foundation
On 18.02.21 18:24, Charan Teja Reddy wrote:
> I would like to start discussion about balancing the occupancy of
> memory zones in a node in the system whose imabalance may be caused by
> migration of pages to other zones during hotremove and then hotadding
> same memory. In this case there is a lot of free memory in newly hotadd
> memory which can be filled up by the previous migrated pages(as part of
> offline/hotremove) thus may free up some pressure in other zones of the
> node.
Why is this specific to memory hot(un)plug? I think the problem is more
generic:
Assume
1. Application 1 allocates a lot of memory and gets ZONE_MOVABLE.
2. Application 2 allocates a lot of memory and gets ZONE_NORMAL.
3. Application 1 quits.
Same problem, no?
--
Thanks,
David / dhildenb
On 2/18/21 6:24 PM, Charan Teja Reddy wrote:
> I would like to start discussion about balancing the occupancy of
> memory zones in a node in the system whose imabalance may be caused by
> migration of pages to other zones during hotremove and then hotadding
> same memory. In this case there is a lot of free memory in newly hotadd
> memory which can be filled up by the previous migrated pages(as part of
> offline/hotremove) thus may free up some pressure in other zones of the
> node.
Can you share the use case for doing this? If it's to replace a failed RAM, then
it's probably extremely rare, right.
> We have the proof-of-concept code tried on the Snapdragon systems with
> the system configuration, single memory node of just 2 zones, 6GB normal
> zone and 2GB movable zone. And this Movable zone is such that hot-added
> once and there after offline/online based on the need.
Hm, snapdragon... so is this some kind of power saving thing?
Anyway, shouln't auto NUMA balancing help here, and especially "Migrate Pages in
lieu of discard" (CC'd Dave) as a generic mechanism, so we wouldn't need to have
hotplug-specific actions?
Thanks David for the review comments!!
On 2/18/2021 11:46 PM, David Hildenbrand wrote:
>> I would like to start discussion about balancing the occupancy of
>> memory zones in a node in the system whose imabalance may be caused by
>> migration of pages to other zones during hotremove and then hotadding
>> same memory. In this case there is a lot of free memory in newly hotadd
>> memory which can be filled up by the previous migrated pages(as part of
>> offline/hotremove) thus may free up some pressure in other zones of the
>> node.
>
> Why is this specific to memory hot(un)plug? I think the problem is more
> generic:
>
> Assume
>
> 1. Application 1 allocates a lot of memory and gets ZONE_MOVABLE.
> 2. Application 2 allocates a lot of memory and gets ZONE_NORMAL.
> 3. Application 1 quits.
>
> Same problem, no?
Thanks for simplifying this problem. Yeah, this looks more generic
problem. But for these type of problems, user/system administrator has
clear view about the state of the system and thus may need to take some
decisions to maintain the the node zones balancing e.g. like this change
where migrate the eligible pages to other zones.
--
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora
Forum, a Linux Foundation Collaborative Project
Thanks Vlastimil for the review comments!!
On 2/19/2021 4:56 PM, Vlastimil Babka wrote:
> Can you share the use case for doing this? If it's to replace a failed RAM, then
> it's probably extremely rare, right.
>
>> We have the proof-of-concept code tried on the Snapdragon systems with
>> the system configuration, single memory node of just 2 zones, 6GB normal
>> zone and 2GB movable zone. And this Movable zone is such that hot-added
>> once and there after offline/online based on the need.
> Hm, snapdragon... so is this some kind of power saving thing?
>
You are correct. This is the power saving usecase which does the offline
and online of the memory blocks in the system by the user. This is not a
failed RAM.
> Anyway, shouln't auto NUMA balancing help here, and especially "Migrate Pages in
> lieu of discard" (CC'd Dave) as a generic mechanism,
On the Snapdragon systems we have got only single memory node with
Normal and movable zones. And my little understanding is that on most
embedded systems we will just have single memory node.
My limited understanding about this auto NUMA balancing is that there
should be min. 2 nodes for this balancing to trigger. Please correct if
I am wrong here. If I am correct then this approach is not suitable for
us. Moreover the idea I would like to convey in this RFC patch is about
__balancing the zones in a node but not across NUMA nodes__.
> so we wouldn't need to have hotplug-specific actions?
David has told a very simple view of this problem which is nothing todo
with the hotplug specific actions.
With just 2 zones(Normal and Movable) in a single node in the system,
1. Application 1 allocates a lot of memory and gets ZONE_MOVABLE.
2. Application 2 allocates a lot of memory and gets ZONE_NORMAL.
3. Application 1 quits.
Then after step3, we can expect a lot free memory in the Movable zone
but normal zone is under pressure. Applying the similar semantics of
Auto numa balancing("Migrate pages in lieu of swap/discard"), we could
migrate some eligible pages of Application 2 to Movable zone there by
can relieve some pressure in Normal zone.
--
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora
Forum, a Linux Foundation Collaborative Project