Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
References: <20220622082513.467538-1-aneesh.kumar@linux.ibm.com>
 <20220622082513.467538-10-aneesh.kumar@linux.ibm.com>
 <87czf06q1i.fsf@nvdebian.thelocal> <87czeuzrpk.fsf@linux.ibm.com>
User-agent: mu4e 1.6.9; emacs 27.1
From:   Alistair Popple <apopple@nvidia.com>
To:     "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>
Cc:     linux-mm@kvack.org, akpm@linux-foundation.org,
        Wei Xu <weixugc@google.com>, Huang Ying <ying.huang@intel.com>,
        Yang Shi <shy828301@gmail.com>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Tim C Chen <tim.c.chen@intel.com>,
        Michal Hocko <mhocko@kernel.org>,
        Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
        Hesham Almatary <hesham.almatary@huawei.com>,
        Dave Hansen <dave.hansen@intel.com>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Dan Williams <dan.j.williams@intel.com>,
        Jagdish Gediya <jvgediya@linux.ibm.com>
Subject: Re: [PATCH v7 09/12] mm/demotion: Demote pages according to
 allocation fallback order
Date:   Mon, 27 Jun 2022 14:18:26 +1000
In-reply-to: <87czeuzrpk.fsf@linux.ibm.com>
Message-ID: <87leti7n5h.fsf@nvdebian.thelocal>
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?2gFKVMF2/bkmAh+Zi8wE34GBpv7cVSopgvddGI8ty8HiyolP+qYUYbc8aiW9?=
 =?us-ascii?Q?XhcuRUEmD2jsgZCQ6qa3qTX/GyDA7SASrMuIbF5SPH29dMf4tIYbfNt+H6dH?=
 =?us-ascii?Q?8SJnrpF6xY3nG/YnkOgeqmNRNfQJSu7V/tDKkuePWKVbtUtpE3ye4DZwJSs1?=
 =?us-ascii?Q?idM20ioHNI/8g+SF8d/eaRoXbTOxkPhTXSZc6Nq+YgABTPlU7wg+H6O2ZyHy?=
 =?us-ascii?Q?2ILP4Dh8bglr5noJzxQ1wde1eqF/TfG17glsf3QVy6A5+v6LT4VSH2fZRJpz?=
 =?us-ascii?Q?7kHQPtmVqZXVKwEC6EQQZ+XTm4BoBaJD1LxXsdHS0ipcxVTMnvpASjMjvpmn?=
 =?us-ascii?Q?/q5HILkstxpgo/sXyt6M1EoCVQ6klB1gjNRHfFZm4igu9eNWAqoKgWuCBmWv?=
 =?us-ascii?Q?oogfAPDcj4ZtjxoZW+1TzOLOv22h8ncJ6vWLZ2nrSqG09QZFgHg3dHs7w91N?=
 =?us-ascii?Q?+uII1HVghB4fU3a7aMxSzXNh3jkAsjlNDwGIXh7P/7AF2t+1YYbakpwv/aWd?=
 =?us-ascii?Q?/iDSLqw2kiSZwt9kFVwQm+Rh8MOEfQtt+3AWWS0DTM06DyWLOd3mo2WS3De/?=
 =?us-ascii?Q?MuIwQDjQT28dNFO9aQPkINVVRDgjDmCwxH4wcRnZaT6O1cY8VmyCpZpNhiDC?=
 =?us-ascii?Q?TZs+LVD7tsPhiyYohcmWbfxTuur4oUzQeLYZ4YRNB+cyiYdMnrlmyaTaz1nv?=
 =?us-ascii?Q?kxoGDFu3dYqAuCQ8mIA/pSc8VXvqEqEo1nnpCylM2rL+nm6iJ6hQwr+QfTuu?=
 =?us-ascii?Q?83I/1oDcbcDhFEqkL7CfBwFSDbF4gJM+glF+M5voEyqg9ybxCkLr5CazxdBq?=
 =?us-ascii?Q?MAU30+8Zh0JeeOIE5uuRCRpEYhQ1hLRZgQR6UTtcd9OIn7qvupeFm9Wp7gTK?=
 =?us-ascii?Q?Vaq818n7zO67XEJJjJa8QzrBPJi8qAqIHdytdYYksQOxd/5zRdtAGEPP4EKV?=
 =?us-ascii?Q?fyIOsYcIB0rDv6W0NBIdy9TJRyb9b4t+eXevwUjBEeJ4iBQEel4rUliXO5ci?=
 =?us-ascii?Q?v+yfzqAxOgkgimkf/2EpgG3x7A0vIBx3EkauC1jLt9v/ReweW9vkOV1b6suW?=
 =?us-ascii?Q?6OxgLmmiMYtu3gB/2YKAhVU97CI+sPA7GU0fhxjrgc5MIT08D0TR0fE4VlGo?=
 =?us-ascii?Q?B6EwzJtkU1HxBml8E2xOa6iDXkRMXtEEHrX7sufoBXe/Uod2Q2oIegKub7u9?=
 =?us-ascii?Q?Yfx8PlagstIBUnbD3r1z+oMWLdT65oRV3eR//NgZffjh6MsXnqgAU1yH2ns0?=
 =?us-ascii?Q?3wF7gQ+n98mKryZ2POZ2kzoP4me1SIC5DD4//Au9EewDeCBRMRnBReTnWlR6?=
 =?us-ascii?Q?TqM5RBc+W/ynl7AKJwmlBfmkqsu4HbNALrzWsN+SzeBNcySPetBb3GZ13FEx?=
 =?us-ascii?Q?KgiB1OwsnaZr1w2EUlMCdIsJkb9GgCJzkqMaFVPQyxB0aLqgogY6JOaBcS3Z?=
 =?us-ascii?Q?WUXh2nWPJJsIFVhWn5mq+rM/dAaChb6yRwQ+906uzYBGyX+v2WlhzK6jVf/W?=
 =?us-ascii?Q?bCOjHJJPrKi4UTM0uDyAoJBkN6RdTwgtoOq0KjhvapwLvsEuRP/qgC6X0tn3?=
 =?us-ascii?Q?lV/5SOehRfXpLkB/4Py4bUfrL79t+rIHnelQfiE1?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 2a75ab4f-0436-432a-d555-08da57f45e23
X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Jun 2022 04:20:29.0952
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: VdTl9Cn3kOYSvqFMdqodwDDddrnpIae+p6rE8yGoRy+xsApq5FRjxSVf3C57tZwFKoOPAoO+wJ3ovggz9SJ6VA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4243
X-Spam-Status: No, score=-2.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO,
        RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


"Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com> writes:

> Alistair Popple <apopple@nvidia.com> writes:
>
>> "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com> writes:
>>
>>> From: Jagdish Gediya <jvgediya@linux.ibm.com>
>>
>> [...]
>>
>>> -static struct page *alloc_demote_page(struct page *page, unsigned long node)
>>> +static struct page *alloc_demote_page(struct page *page, unsigned long private)
>>>  {
>>> -	struct migration_target_control mtc = {
>>> -		/*
>>> -		 * Allocate from 'node', or fail quickly and quietly.
>>> -		 * When this happens, 'page' will likely just be discarded
>>> -		 * instead of migrated.
>>> -		 */
>>> -		.gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) |
>>> -			    __GFP_THISNODE  | __GFP_NOWARN |
>>> -			    __GFP_NOMEMALLOC | GFP_NOWAIT,
>>> -		.nid = node
>>> -	};
>>> +	struct page *target_page;
>>> +	nodemask_t *allowed_mask;
>>> +	struct migration_target_control *mtc;
>>> +
>>> +	mtc = (struct migration_target_control *)private;
>>> +
>>> +	allowed_mask = mtc->nmask;
>>> +	/*
>>> +	 * make sure we allocate from the target node first also trying to
>>> +	 * reclaim pages from the target node via kswapd if we are low on
>>> +	 * free memory on target node. If we don't do this and if we have low
>>> +	 * free memory on the target memtier, we would start allocating pages
>>> +	 * from higher memory tiers without even forcing a demotion of cold
>>> +	 * pages from the target memtier. This can result in the kernel placing
>>> +	 * hotpages in higher memory tiers.
>>> +	 */
>>> +	mtc->nmask = NULL;
>>> +	mtc->gfp_mask |= __GFP_THISNODE;
>>> +	target_page = alloc_migration_target(page, (unsigned long)&mtc);
>>
>> I finally managed to get a system setup to start testing some of this
>> out. However it quickly crashed due to the bad pointer in the above call
>> - you need mtc not &mtc here.
>
> I remember fixing that during earlier testing. I guess I missed to copy
> the change from test to my development. Thanks for testing this. I have
> now also tested the complete series with the above-suggested changes and did
> make sure we are indeed doing demotion by looking at
> /proc/vmstat:pgdemote_kswapd/pgdemote_direct

No worries. I'm still testing but the early results are looking really
promising for some of our use cases so thanks for picking up this work.

 - Alistair

>>
>>> +	if (target_page)
>>> +		return target_page;
>>> +
>>> +	mtc->gfp_mask &= ~__GFP_THISNODE;
>>> +	mtc->nmask = allowed_mask;
>>>
>>>  	return alloc_migration_target(page, (unsigned long)&mtc);
>>
>> And here.
>>
>
> I will fold this changes in and send a v8 after waiting for review
> feedback from others.
>
> -aneesh