Received: by 2002:ab2:710b:0:b0:1ef:a325:1205 with SMTP id z11csp645086lql; Mon, 11 Mar 2024 12:59:21 -0700 (PDT) X-Forwarded-Encrypted: i=3; AJvYcCVnQY1xfdkPx23dJUcKiNkbR6xsTKTXfCUHT5KxS21kSQuuuOu0nVpHmBugA3d0zHTRTwHSM14itAE5OLp2vnUIq2MfETRFKXs+fQwayg== X-Google-Smtp-Source: AGHT+IHz8+SWQ7HVY/dPO7QECt25a/BqWHXYRQg8PPyNxXp5eQtRLoApWdx1BCiijsOF4tzVhK9C X-Received: by 2002:a17:902:d486:b0:1dd:7d20:63ef with SMTP id c6-20020a170902d48600b001dd7d2063efmr8207043plg.11.1710187161166; Mon, 11 Mar 2024 12:59:21 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1710187161; cv=pass; d=google.com; s=arc-20160816; b=XGxMDfGWARWFvarGl7JLYoYKgPQiQbRE19RSQ3t3P0I+H/rjGk7va7LwIViUCxXCEs /LxVzGRPHNkBSquTl4a+aYsImp1TmRbrFOvxyRDtHwi/F4fe4WUy4jIFG40px+e/jSoI 4HhSel5l8YCWxia+AXF4ZFsS6Nl6NmmYEc4ZoGarUKeaZdgAke+14DBrXDQkH7K6rRA6 xIXy/HrEVSM09DNSSDtjRzDW8YKWXjZdG5VOpwDelzcrjVlsein1eA0C55jzxyWMH6Ce wTyvOXsj5EB0skRZyTnNvjrWTclc7kWm/Wv3b6nOtJkY5+/fhbqJJyRSRnwuFQ9zQhW6 y7Hw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:reply-to:message-id:date:subject :cc:to:from:feedback-id:dkim-signature:dkim-signature; bh=A/MBFnXXdSSgAn+JMHfZprV2OLIqaCRbl8i6tPL49MY=; fh=CUdSQdegif00KF0idljkPluhvc6qPMs3rIdiGygTk0g=; b=atTiVN44trHALlkusMovnq55K7Rg5Tay8jYgI3/dFL30cBADumN1oa1/VOOs1Fhgmu EZTZTR6jTjMEM678rZcqmrte/iLIXqTKmrBAjX/i9TpIe0mxBkPjMIBOFn28pmV9Nxrm HndKJNLumjDAqLEIWZhjwXHIDPdRPbuY5K55rGTcq8s6NFiKemi1nY2XVKucYfJxIEYR RFjfmUGox46uRwjXojKMtDhdfKrVaoTPvIPw3oIf9jqT39ww4Gg7HKAe1J3Nn7Mh7W2Q 7eL8xvBBY+LSwcJkXbAmmineKUg1pLUHM+TZModaeZ57xWhnw/m4ED21Nf+jGAEMreju L+Zg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@sent.com header.s=fm1 header.b=owxgzEgP; dkim=pass header.i=@messagingengine.com header.s=fm1 header.b="d0/uvwRh"; arc=pass (i=1 spf=pass spfdomain=sent.com dkim=pass dkdomain=sent.com dkim=pass dkdomain=messagingengine.com dmarc=pass fromdomain=sent.com); spf=pass (google.com: domain of linux-kernel+bounces-99495-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-99495-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sent.com Return-Path: Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id h11-20020a170902f7cb00b001db45912e65si5208865plw.501.2024.03.11.12.59.20 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 Mar 2024 12:59:21 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel+bounces-99495-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@sent.com header.s=fm1 header.b=owxgzEgP; dkim=pass header.i=@messagingengine.com header.s=fm1 header.b="d0/uvwRh"; arc=pass (i=1 spf=pass spfdomain=sent.com dkim=pass dkdomain=sent.com dkim=pass dkdomain=messagingengine.com dmarc=pass fromdomain=sent.com); spf=pass (google.com: domain of linux-kernel+bounces-99495-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-99495-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sent.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 1823DB21636 for ; Mon, 11 Mar 2024 19:59:12 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5C94C56751; Mon, 11 Mar 2024 19:58:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b="owxgzEgP"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="d0/uvwRh" Received: from fhigh3-smtp.messagingengine.com (fhigh3-smtp.messagingengine.com [103.168.172.154]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AD3A755E75 for ; Mon, 11 Mar 2024 19:58:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.154 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710187134; cv=none; b=YQOVNlm7J/plDuL/7bWJx2LlxcRq3Z3ZwMWnWVbaZMxLcNFIB+G42J/+u1nGfNWAmJ8Qt/cOInGODOhgQnOzlp8jQiGfyUo/zA0a3ONsATHOwjVdsGnc6xhOe5vHtt75bwIdGLQPcN8YM7xy1P4AuwKVVFu3AzV5tHENZQds6RE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710187134; c=relaxed/simple; bh=HNBUTAaMykG6ISWX6c10ERvaCy/S6UH8VZlm6dphoJE=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:Content-Type; b=Qkv1wPvvDsbMQMFmKGzzndHPnXIFYUokacPUhKa6NTm3ignj7Nev3NpsJV3oJEBRNVXLO5tSJZ1cNCAdPq9z5j9YnF3vdd8E4bJeMbaUUpHYYZ32dLuLHN/Xdn+THsxQJI+RnFXWejrH58f5xs2a9DBNLbtGIX7RCeaQLxpzgIA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=sent.com; spf=pass smtp.mailfrom=sent.com; dkim=pass (2048-bit key) header.d=sent.com header.i=@sent.com header.b=owxgzEgP; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=d0/uvwRh; arc=none smtp.client-ip=103.168.172.154 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=sent.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sent.com Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailfhigh.nyi.internal (Postfix) with ESMTP id A6EB411400E2; Mon, 11 Mar 2024 15:58:51 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Mon, 11 Mar 2024 15:58:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:message-id:mime-version:reply-to :reply-to:subject:subject:to:to; s=fm1; t=1710187131; x= 1710273531; bh=A/MBFnXXdSSgAn+JMHfZprV2OLIqaCRbl8i6tPL49MY=; b=o wxgzEgPviaIEemket13I0jAVdGZHZxIDDtw/tQDKpuGKZMhiLYHwwZJoO/1l2coz Oa/A478hyUdh4SiG31PGTNntuctQuzT85RXwY76sBHqWiG+JRk/UPIEoSioNFEY0 RyQ5LaQsyv9Cv8ebPHUOzQVerfANAJXbdcxKD6OQzmKym0StFPr1PJWw7XulUCiC d3mt4qrQiSyJyo7M2oKw0bVn5M7bAg49bd2UUMjhv23tFMGrnfO16NNTf6/NxZqg cwvEgqR4kNxgq/Qm0jrGZAfxyaogKkz4azmhzddxaSFj1TJfYnOxA2sAecq4cNmg kPABiC1PILMJluCdOoFrw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:message-id:mime-version:reply-to:reply-to :subject:subject:to:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1710187131; x=1710273531; bh=A /MBFnXXdSSgAn+JMHfZprV2OLIqaCRbl8i6tPL49MY=; b=d0/uvwRhXJgD9iqcn WBcsjvIJvpUjudTVTN5GbDN9kf4Nqa7A+v05yqcaDHtbr+zoq93ruVn/WjvzYCjW 3jahNJJ63N5q9rVWzuGscUNM6U8KCR4RWKzmJtChIxlV/gsxdCLTkJz1Tz5UspTy RfHRmgQ9uJjvqowS+wAJsWpm/waTr9L4EBZRCycIz0QxtiDa+sYHCo4szuW4GA9p PoW2RpQ1MsWKLBWoCXT4uX99IKUCT2aeT4VxbLnqIhgtMMaMvTjYvECPl5mvLigm RVbLtyHLMTIqjhcmFx5m6bCKUVDUD0ArJub6eQiYCsIYwNeebPRr4ePIwAH71mqf H3R9w== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrjedugddufedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofhrgggtgfesthekredtredtjeenucfhrhhomhepkghiucgj rghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepteduve ehteehheeiteeihfejveejledtgfdvieeuiedutefftdevtdfhteevtdffnecuffhomhgr ihhnpehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmh epmhgrihhlfhhrohhmpeiiihdrhigrnhesshgvnhhtrdgtohhm X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 11 Mar 2024 15:58:50 -0400 (EDT) From: Zi Yan To: linux-mm@kvack.org Cc: Zi Yan , Andrew Morton , "Matthew Wilcox (Oracle)" , Yang Shi , Huang Ying , "Kirill A . Shutemov" , Ryan Roberts , linux-kernel@vger.kernel.org Subject: [PATCH v2] mm/migrate: put dest folio on deferred split list if source was there. Date: Mon, 11 Mar 2024 15:58:48 -0400 Message-ID: <20240311195848.135067-1-zi.yan@sent.com> X-Mailer: git-send-email 2.43.0 Reply-To: Zi Yan Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit From: Zi Yan Commit 616b8371539a6 ("mm: thp: enable thp migration in generic path") did not check if a THP is on deferred split list before migration, thus, the destination THP is never put on deferred split list even if the source THP might be. The opportunity of reclaiming free pages in a partially mapped THP during deferred list scanning is lost, but no other harmful consequence is present[1]. Checking source folio deferred split list status before page unmapped and add destination folio to the list if source was after migration. [1]: https://lore.kernel.org/linux-mm/03CE3A00-917C-48CC-8E1C-6A98713C817C@nvidia.com/ From v1: 1. Used dst to get correct deferred split list after migration (per Ryan Roberts). Fixes: 616b8371539a ("mm: thp: enable thp migration in generic path") Signed-off-by: Zi Yan --- mm/huge_memory.c | 22 ---------------------- mm/internal.h | 23 +++++++++++++++++++++++ mm/migrate.c | 26 +++++++++++++++++++++++++- 3 files changed, 48 insertions(+), 23 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9859aa4f7553..c6d4d0cdf4b3 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -766,28 +766,6 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) return pmd; } -#ifdef CONFIG_MEMCG -static inline -struct deferred_split *get_deferred_split_queue(struct folio *folio) -{ - struct mem_cgroup *memcg = folio_memcg(folio); - struct pglist_data *pgdat = NODE_DATA(folio_nid(folio)); - - if (memcg) - return &memcg->deferred_split_queue; - else - return &pgdat->deferred_split_queue; -} -#else -static inline -struct deferred_split *get_deferred_split_queue(struct folio *folio) -{ - struct pglist_data *pgdat = NODE_DATA(folio_nid(folio)); - - return &pgdat->deferred_split_queue; -} -#endif - void folio_prep_large_rmappable(struct folio *folio) { if (!folio || !folio_test_large(folio)) diff --git a/mm/internal.h b/mm/internal.h index d1c69119b24f..8fa36e84463a 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1107,6 +1107,29 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, unsigned int flags); +#ifdef CONFIG_MEMCG +static inline +struct deferred_split *get_deferred_split_queue(struct folio *folio) +{ + struct mem_cgroup *memcg = folio_memcg(folio); + struct pglist_data *pgdat = NODE_DATA(folio_nid(folio)); + + if (memcg) + return &memcg->deferred_split_queue; + else + return &pgdat->deferred_split_queue; +} +#else +static inline +struct deferred_split *get_deferred_split_queue(struct folio *folio) +{ + struct pglist_data *pgdat = NODE_DATA(folio_nid(folio)); + + return &pgdat->deferred_split_queue; +} +#endif + + /* * mm/mmap.c */ diff --git a/mm/migrate.c b/mm/migrate.c index 73a052a382f1..591e65658535 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -1037,7 +1038,10 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, enum { PAGE_WAS_MAPPED = BIT(0), PAGE_WAS_MLOCKED = BIT(1), - PAGE_OLD_STATES = PAGE_WAS_MAPPED | PAGE_WAS_MLOCKED, + PAGE_WAS_ON_DEFERRED_LIST = BIT(2), + PAGE_OLD_STATES = PAGE_WAS_MAPPED | + PAGE_WAS_MLOCKED | + PAGE_WAS_ON_DEFERRED_LIST, }; static void __migrate_folio_record(struct folio *dst, @@ -1168,6 +1172,17 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, folio_lock(src); } locked = true; + if (folio_test_large_rmappable(src) && + !list_empty(&src->_deferred_list)) { + struct deferred_split *ds_queue = get_deferred_split_queue(src); + + spin_lock(&ds_queue->split_queue_lock); + ds_queue->split_queue_len--; + list_del_init(&src->_deferred_list); + spin_unlock(&ds_queue->split_queue_lock); + old_page_state |= PAGE_WAS_ON_DEFERRED_LIST; + } + if (folio_test_mlocked(src)) old_page_state |= PAGE_WAS_MLOCKED; @@ -1307,6 +1322,15 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, if (old_page_state & PAGE_WAS_MAPPED) remove_migration_ptes(src, dst, false); + if (old_page_state & PAGE_WAS_ON_DEFERRED_LIST) { + struct deferred_split *ds_queue = get_deferred_split_queue(dst); + + spin_lock(&ds_queue->split_queue_lock); + ds_queue->split_queue_len++; + list_add(&dst->_deferred_list, &ds_queue->split_queue); + spin_unlock(&ds_queue->split_queue_lock); + } + out_unlock_both: folio_unlock(dst); set_page_owner_migrate_reason(&dst->page, reason); -- 2.43.0