Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Sender: charante=codeaurora.org@mg.codeaurora.org
DMARC-Filter: OpenDMARC Filter v1.3.2 smtp.codeaurora.org 70FCBC433C6
From:   Charan Teja Reddy <charante@codeaurora.org>
To:     akpm@linux-foundation.org, rientjes@google.com, vbabka@suse.cz,
        mhocko@suse.com, david@redhat.com, mgorman@techsingularity.net,
        linux-mm@kvack.org
Cc:     vinmenon@codeaurora.org, sudaraja@codeaurora.org,
        linux-kernel@vger.kernel.org,
        Charan Teja Reddy <charante@codeaurora.org>
Subject: [PATCH RFC 0/1] mm: balancing the node zones occupancy 
Date:   Thu, 18 Feb 2021 22:54:31 +0530
Message-Id: <cover.1613661472.git.charante@codeaurora.org>
Precedence: bulk

I would like to start discussion about  balancing the occupancy of
memory zones in a node in the system whose imabalance may be caused by
migration of pages to other zones during hotremove and then hotadding
same memory. In this case there is a lot of free memory in newly hotadd
memory which can be filled up by the previous migrated pages(as part of
offline/hotremove) thus may free up some pressure in other zones of the
node.

Say that in system has 2 zones(Normal and Movable), where Normal zone is
almost filled up by the pages of the movable zone as part of offline
operation and then we hot add a memory node as movable zone. At this
moment, Movable zone is almost empty and Normal zone is almost filled
up(by the migrated pages as part of previous offline/hot-remove
operation). At this point of time, allocation requests from Normal zone
may have to go through reclaim cycles thus cause some pressure. This
problem is quite common in the system where they aggressively offline
and online the memory blocks in the system where the former part do the
migration of pages to the lower zones and the later part don't reverse
them and as a result the offline operation may contribute to the
pressure in other zones of the system.

To overcome this problem, we can do the reverse of what offline
operation did, after onlining the memory block i.e. **try to reverse
migrate the pages from other zones which were migrated as part of the
offline operation**. This may freeup some pressure built in the other
zones because of offline operation. Since we are reverse migrating the
pages in the system, we can name this as "reverse migration feature" or
since we are actually balancing the occupancy of the zones in the system
by migrating the pages we can name it as "balancing the system zones
occupancy" or any name...

We have the proof-of-concept code tried on the Snapdragon systems with
the system configuration, single memory node of just 2 zones, 6GB normal
zone and 2GB movable zone. And this Movable zone is such that hot-added
once and there after offline/online based on the need.

We run the below unit test to evalutate this:
1) Completely fill up the already hot added movable zone with anon
pages

2) Offline this hot added movable zone. At this point there is no
pressure in the normal zone yet, but this migration of pages can
contribute to it in the future.

3) Now fill up the normal zone such that there is 300MB or less left in
the system.

4) And now the user onlined the movable zone memory blocks in the
system.

5) Run the tests of allocating 512MB of memory from the normal zone and
in doing so try allocating the higher order pages first and then
gradually fall back to lower orders. I took the help from ion system heap
memory allocation which try to allocate the memory in available orders:
9, 4 and 0.

6) Repeat the above steps for 10 iterations and below is the average of
the results.

We did try to collect the time it takes to complete the tests and the
distribution of anon pages(are the ones participated in the tests) in
the system node zones.
a) With out the reverse migration, it took an average of around 145msec
to complete the test.
b) With the reverse migration, it took an average of the 120msec to
complete the tests.

For distribution of the anon pages in the system we did try collect the
anon pages left in the individual zone before and after the test:
------------------------------------- |-------------------------------
         Base			      |		Reverse Migration
--------------------------------------|-------------------------------
	   Beforetest Aftertest	      | Beforetest	 Aftertest
Normal zone(Anon)                     |
  Active  499825	45756         |  481441		  203124
  Inactive 46008	446687	      |	 51553	          58602
  Free     80350	224252	      |  84877		**440586**
Movable zone(Anon)                    |
  Active   2224		2626	      |  2239		**484622**
  Inactive    8		   8	      |  9		7663
--------------------------------------|-------------------------------

The above table shows that, On base case(left column), there exists a
lot of anon pages in the system which can be migrated back to Movable
zone(almost totally free), thus may freeup some space in the normal
zone. With the reverse migration(Right coloumn), we see that the anon
pages are evenly distributed in the system and lot of free memory
left in the normal zones caused by the reverse migration.

The code shows the PoC by assuming just 2 zones(normal and Movable) of a
single node in the system. The number of pages to be reverse migrated is
written on the sysfs interface from the userspace by monitoring the
memory pressure events in the system.

Charan Teja Reddy (1):
  mm: proof-of-concept for balancing node zones occupancy

 include/linux/migrate.h |   8 +-
 include/linux/mm.h      |   3 +
 include/linux/mmzone.h  |   2 +
 kernel/sysctl.c         |  11 ++
 mm/compaction.c         |   4 +-
 mm/memory_hotplug.c     | 265 ++++++++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 290 insertions(+), 3 deletions(-)

-- 
QUALCOMM INDIA, on behalf of Qualcomm Innovation Center, Inc. is a
member of the Code Aurora Forum, hosted by The Linux Foundation