Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: [PATCH v2 2/2] mm,thp,compaction,cma: allow THP migration for CMA
 allocations
To:     Rik van Riel <riel@surriel.com>, linux-kernel@vger.kernel.org
Cc:     kernel-team@fb.com, akpm@linux-foundation.org, linux-mm@kvack.org,
        mhocko@kernel.org, vbabka@suse.cz, mgorman@techsingularity.net,
        rientjes@google.com, aarcange@redhat.com, ziy@nvidia.com
References: <cover.1582321646.git.riel@surriel.com>
 <20200227213238.1298752-2-riel@surriel.com>
 <df83c62f-209f-b1fd-3a5c-c81c82cb2606@oracle.com>
 <7800e98e3688c124ac3672284b87d67321e1c29e.camel@surriel.com>
From:   Mike Kravetz <mike.kravetz@oracle.com>
Message-ID: <c1b450f2-59ac-c6e0-3a99-f51bdebbe9c9@oracle.com>
Date:   Thu, 27 Feb 2020 20:30:03 -0800
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101
 Thunderbird/68.4.1
MIME-Version: 1.0
In-Reply-To: <7800e98e3688c124ac3672284b87d67321e1c29e.camel@surriel.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On 2/27/20 5:21 PM, Rik van Riel wrote:
> On Thu, 2020-02-27 at 15:41 -0800, Mike Kravetz wrote:
>> On 2/27/20 1:32 PM, Rik van Riel wrote:
>>>
>>> +++ b/mm/page_alloc.c
>>> @@ -8253,14 +8253,19 @@ struct page *has_unmovable_pages(struct
>>> zone *zone, struct page *page,
>>>  
>>>  		/*
>>>  		 * Hugepages are not in LRU lists, but they're movable.
>>> +		 * THPs are on the LRU, but need to be counted as
>>> #small pages.
>>>  		 * We need not scan over tail pages because we don't
>>>  		 * handle each tail page individually in migration.
>>>  		 */
>>> -		if (PageHuge(page)) {
>>> +		if (PageHuge(page) || PageTransCompound(page)) {
>>>  			struct page *head = compound_head(page);
>>>  			unsigned int skip_pages;
>>>  
>>> -			if
>>> (!hugepage_migration_supported(page_hstate(head)))
>>> +			if (PageHuge(page) &&
>>> +			    !hugepage_migration_supported(page_hstate(h
>>> ead)))
>>> +				return page;
>>> +
>>> +			if (!PageLRU(head) && !__PageMovable(head))
>>
>> Pretty sure this is going to be true for hugetlb pages.  So, this
>> will change
>> behavior and make all hugetlb pages look unmovable.  Perhaps, only
>> check this
>> condition for THP pages?
> 
> Does that need to be the following, then?
> 
>      if (PageTransHuge(head) && !PageHuge(page) && !PageLRU(head) &&
> !__PageMovable(head))
>                  return page;

Yes, that is what I would suggest.  Results of a simple test on small VM.

Unmodified Kernel
-----------------
# cat /sys/devices/system/memory/memory*/removable | grep 1 | wc -l
50
# hugeadm --hard --pool-pages-min DEFAULT:4G
# cat /sys/devices/system/memory/memory*/removable | grep 1 | wc -l
50

V2 patches
----------
# cat /sys/devices/system/memory/memory*/removable | grep 1 | wc -l
50
# hugeadm --hard --pool-pages-min DEFAULT:4G
# cat /sys/devices/system/memory/memory*/removable | grep 1 | wc -l
14

V2 patches + above modification
-------------------------------
# cat /sys/devices/system/memory/memory*/removable | grep 1 | wc -l
50
# hugeadm --hard --pool-pages-min DEFAULT:4G
# cat /sys/devices/system/memory/memory*/removable | grep 1 | wc -l
50

> That's an easy one liner I would be happy to send in
> if everybody agrees that should fix things :)

Another option might be to make hugetlb pages more like other pages and
__SetPageMovable on movable hugetlb pages.  This could be used instead
of hugepage_migration_supported() calls.  That is only a quick thought
and would be beyond the scope of this patch.  If people think that is
worth doing, and I have not overlooked something,  I could look into it
separately.
-- 
Mike Kravetz