Received-SPF: softfail (google.com: domain of transitioning linux-kernel-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
Message-ID: <046c373a-f30b-091d-47a1-e28bfb7e9394@linux.ibm.com>
Date:   Fri, 3 Jun 2022 20:39:47 +0530
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
 Thunderbird/91.10.0
Subject: Re: [RFC PATCH v4 7/7] mm/demotion: Demote pages according to
 allocation fallback order
Content-Language: en-US
To:     Ying Huang <ying.huang@intel.com>, linux-mm@kvack.org,
        akpm@linux-foundation.org
Cc:     Greg Thelen <gthelen@google.com>, Yang Shi <shy828301@gmail.com>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Tim C Chen <tim.c.chen@intel.com>,
        Brice Goglin <brice.goglin@gmail.com>,
        Michal Hocko <mhocko@kernel.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Hesham Almatary <hesham.almatary@huawei.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Alistair Popple <apopple@nvidia.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Feng Tang <feng.tang@intel.com>,
        Jagdish Gediya <jvgediya@linux.ibm.com>,
        Baolin Wang <baolin.wang@linux.alibaba.com>,
        David Rientjes <rientjes@google.com>
References: <CAAPL-u-dFp7PwPH6DfbYdnY8xaGsHz3tRQ0CPGVkiqURvdN8=A@mail.gmail.com>
 <20220527122528.129445-1-aneesh.kumar@linux.ibm.com>
 <20220527122528.129445-8-aneesh.kumar@linux.ibm.com>
 <b102d5773bffd6391283773044f756e810c1f044.camel@intel.com>
From:   Aneesh Kumar K V <aneesh.kumar@linux.ibm.com>
In-Reply-To: <b102d5773bffd6391283773044f756e810c1f044.camel@intel.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
Precedence: bulk

On 6/2/22 1:05 PM, Ying Huang wrote:
> On Fri, 2022-05-27 at 17:55 +0530, Aneesh Kumar K.V wrote:
>> From: Jagdish Gediya <jvgediya@linux.ibm.com>
>>
>> currently, a higher tier node can only be demoted to selected
>> nodes on the next lower tier as defined by the demotion path,
>> not any other node from any lower tier.  This strict, hard-coded
>> demotion order does not work in all use cases (e.g. some use cases
>> may want to allow cross-socket demotion to another node in the same
>> demotion tier as a fallback when the preferred demotion node is out
>> of space). This demotion order is also inconsistent with the page
>> allocation fallback order when all the nodes in a higher tier are
>> out of space: The page allocation can fall back to any node from any
>> lower tier, whereas the demotion order doesn't allow that currently.
>>
>> This patch adds support to get all the allowed demotion targets mask
>> for node, also demote_page_list() function is modified to utilize this
>> allowed node mask by filling it in migration_target_control structure
>> before passing it to migrate_pages().
>

...

>>    * Take pages on @demote_list and attempt to demote them to
>>    * another node.  Pages which are not demoted are left on
>> @@ -1481,6 +1464,19 @@ static unsigned int demote_page_list(struct list_head *demote_pages,
>>   {
>>   	int target_nid = next_demotion_node(pgdat->node_id);
>>   	unsigned int nr_succeeded;
>> +	nodemask_t allowed_mask;
>> +
>> +	struct migration_target_control mtc = {
>> +		/*
>> +		 * Allocate from 'node', or fail quickly and quietly.
>> +		 * When this happens, 'page' will likely just be discarded
>> +		 * instead of migrated.
>> +		 */
>> +		.gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) | __GFP_NOWARN |
>> +			__GFP_NOMEMALLOC | GFP_NOWAIT,
>> +		.nid = target_nid,
>> +		.nmask = &allowed_mask
>> +	};
> 
> IMHO, we should try to allocate from preferred node firstly (which will
> kick kswapd of the preferred node if necessary).  If failed, we will
> fallback to all allowed node.
> 
> As we discussed as follows,
> 
> https://lore.kernel.org/lkml/69f2d063a15f8c4afb4688af7b7890f32af55391.camel@intel.com/
> 
> That is, something like below,
> 
> static struct page *alloc_demote_page(struct page *page, unsigned long node)
> {
> 	struct page *page;
> 	nodemask_t allowed_mask;
> 	struct migration_target_control mtc = {
> 		/*
> 		 * Allocate from 'node', or fail quickly and quietly.
> 		 * When this happens, 'page' will likely just be discarded
> 		 * instead of migrated.
> 		 */
> 		.gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) |
> 			    __GFP_THISNODE  | __GFP_NOWARN |
> 			    __GFP_NOMEMALLOC | GFP_NOWAIT,
> 		.nid = node
> 	};
> 
> 	page = alloc_migration_target(page, (unsigned long)&mtc);
> 	if (page)
> 		return page;
> 
> 	mtc.gfp_mask &= ~__GFP_THISNODE;
> 	mtc.nmask = &allowed_mask;
> 
> 	return alloc_migration_target(page, (unsigned long)&mtc);
> }

I skipped doing this in v5 because I was not sure this is really what we 
want. I guess we can do this as part of the change that is going to 
introduce the usage of memory policy for the allocation?

-aneesh