Received: by 2002:a05:7412:251c:b0:e2:908c:2ebd with SMTP id w28csp941196rda; Sun, 22 Oct 2023 17:57:11 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEkENjx8rZyvZWbABHTVkOqfD1xXXueWEvNbm4MjDU2XsSCUvRUm4j0lyT+p8mn7E5EA84U X-Received: by 2002:a05:6a00:b47:b0:68f:f650:3035 with SMTP id p7-20020a056a000b4700b0068ff6503035mr6764774pfo.12.1698022631146; Sun, 22 Oct 2023 17:57:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1698022631; cv=none; d=google.com; s=arc-20160816; b=YQeYb5rFn23LA4s90xNZjcgFHSmDtdwtiG76Ne0OafmKJljXYasmQXPZsQHyfw6g3k qZLxm1bDX3nzlSwfC1kO1JNKWGGq8m1f8FfCP0dkJJlwqXLdCTdxjOYBNXz2BFoYUsyb gQy128CZQGsj0JYLJCUyGxxfwXo5VYkIWHSGY/oFuw9yZEIe0hy7CfY3PmBCTb8j7jbq SbLi0gQsaPEJRFMLfWafTtrzXLctpWU375cu8+sTzfiw8MH/6CUP+GzYsGPHG+xsCLKU xKLZJmpehLoXFhOYYF6nDaXQTQQlgJbrjm3/aWzPQn1P4pAX/vmRDbda+FqERMarK1Qa f4Wg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:user-agent:message-id:date :references:in-reply-to:subject:cc:to:from:dkim-signature; bh=hGtTM2evG7LAmBs24FIv+w85m5VGRFDz5JECdSlJyQM=; fh=qvTINQSJI9J643GnVBOiE9pTDdBzpaAkp5jnBQF9bU0=; b=SnXT1/VoW6RWMaVp820khdS1N2Jd6KNobCnxu8CtJg5kbMwxEvdajyBa7fmyVVLyUX nMFXK+XL6MzNBo3Xjcb0Kmq10sxosTOfZExNO1bXh88hWukQEUKyzbLvBDTBoMxwPrT5 ybjMAn3CpHi6SvZUYgpCASAIHC74oWnzTUugT5YRS0/vY8QeWGL+6TZg+xynRZrJH/R2 cyWLfbMD0PLBmrdDH7nMeDjNzKYAQReEJT/Viz08uZX1iD9JOS+LELNkUmb4pgPOAeu3 wngnPYoJYk+Baxl72fo5u4rc5aCcahSzVBFVtAh5Kr2xSen/A+qBU018AiiOyDB5DX2M SmUA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=iN4J7NDz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id r18-20020a63ce52000000b005b896f26b15si2598894pgi.268.2023.10.22.17.57.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 22 Oct 2023 17:57:11 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=iN4J7NDz; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 7B0FE80787F3; Sun, 22 Oct 2023 17:57:07 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232913AbjJWA4z (ORCPT + 99 others); Sun, 22 Oct 2023 20:56:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48324 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229574AbjJWA4z (ORCPT ); Sun, 22 Oct 2023 20:56:55 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0E1CEB7 for ; Sun, 22 Oct 2023 17:56:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1698022613; x=1729558613; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version; bh=hS8mjo8Zr5a5TG8vn4Z2xY74poRaGE5x1DCwvuN39kk=; b=iN4J7NDz0pDMVlLKe3UWQb8Sky4y+brhrn99A/HJGIs8iJ9NBvqYqPXd jDBkL5tK5aBeqViC2e+UX2UxlEP3/048of6VUAtUfRAhtQBZcFupchB/m ni8loRuX4qzCMa+tOXgXFAK+mWP2QkpxnzgAeLEvKkI6iGllm9KhJeE6I iKpgUNq9pLKHHhi44qmSGbMwcKA/Twl5clfbNsoo1Ga5eQzE2i6bmZfPX /sHkuSaGBu8iJgM+iGTpwbrvkYj0T01cQ4c0giEQ2ChtHYu+gwhYDiEM4 zCwG+TOV6kHghsVCZ33I9mftvjUdH8X6G4FkdqSr354VRZqGh0wrUl8ei Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10871"; a="371811453" X-IronPort-AV: E=Sophos;i="6.03,244,1694761200"; d="scan'208";a="371811453" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Oct 2023 17:56:52 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10871"; a="823806134" X-IronPort-AV: E=Sophos;i="6.03,244,1694761200"; d="scan'208";a="823806134" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Oct 2023 17:56:50 -0700 From: "Huang, Ying" To: Baolin Wang Cc: , , , , , , , Subject: Re: [PATCH v3] mm: migrate: record the mlocked page status to remove unnecessary lru drain In-Reply-To: <06e9153a7a4850352ec36602df3a3a844de45698.1697859741.git.baolin.wang@linux.alibaba.com> (Baolin Wang's message of "Sat, 21 Oct 2023 12:33:22 +0800") References: <06e9153a7a4850352ec36602df3a3a844de45698.1697859741.git.baolin.wang@linux.alibaba.com> Date: Mon, 23 Oct 2023 08:54:47 +0800 Message-ID: <877cnef99k.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Sun, 22 Oct 2023 17:57:07 -0700 (PDT) Baolin Wang writes: > When doing compaction, I found the lru_add_drain() is an obvious hotspot > when migrating pages. The distribution of this hotspot is as follows: > - 18.75% compact_zone > - 17.39% migrate_pages > - 13.79% migrate_pages_batch > - 11.66% migrate_folio_move > - 7.02% lru_add_drain > + 7.02% lru_add_drain_cpu > + 3.00% move_to_new_folio > 1.23% rmap_walk > + 1.92% migrate_folio_unmap > + 3.20% migrate_pages_sync > + 0.90% isolate_migratepages > > The lru_add_drain() was added by commit c3096e6782b7 ("mm/migrate: > __unmap_and_move() push good newpage to LRU") to drain the newpage to LRU > immediately, to help to build up the correct newpage->mlock_count in > remove_migration_ptes() for mlocked pages. However, if there are no mlocked > pages are migrating, then we can avoid this lru drain operation, especailly > for the heavy concurrent scenarios. > > So we can record the source pages' mlocked status in migrate_folio_unmap(), > and only drain the lru list when the mlocked status is set in migrate_folio_move(). > In addition, the page was already isolated from lru when migrating, so checking > the mlocked status is stable by folio_test_mlocked() in migrate_folio_unmap(). > > After this patch, I can see the hotpot of the lru_add_drain() is gone: > - 9.41% migrate_pages_batch > - 6.15% migrate_folio_move > - 3.64% move_to_new_folio > + 1.80% migrate_folio_extra > + 1.70% buffer_migrate_folio > + 1.41% rmap_walk > + 0.62% folio_add_lru > + 3.07% migrate_folio_unmap > > Meanwhile, the compaction latency shows some improvements when running > thpscale: > base patched > Amean fault-both-1 1131.22 ( 0.00%) 1112.55 * 1.65%* > Amean fault-both-3 2489.75 ( 0.00%) 2324.15 * 6.65%* > Amean fault-both-5 3257.37 ( 0.00%) 3183.18 * 2.28%* > Amean fault-both-7 4257.99 ( 0.00%) 4079.04 * 4.20%* > Amean fault-both-12 6614.02 ( 0.00%) 6075.60 * 8.14%* > Amean fault-both-18 10607.78 ( 0.00%) 8978.86 * 15.36%* > Amean fault-both-24 14911.65 ( 0.00%) 11619.55 * 22.08%* > Amean fault-both-30 14954.67 ( 0.00%) 14925.66 * 0.19%* > Amean fault-both-32 16654.87 ( 0.00%) 15580.31 * 6.45%* > > Signed-off-by: Baolin Wang LGTM, Thanks! Reviewed-by: "Huang, Ying" > --- > Changes from v2: > - Use BIT() to define macros. > - Simplify handing page_was_mapped and page_was_mlocked variables with > introducing old_page_state variable. > > Changes from v1: > - Use separate flags in __migrate_folio_record() to avoid to pack flags > in each call site per Ying. > --- > mm/migrate.c | 48 +++++++++++++++++++++++++++++------------------- > 1 file changed, 29 insertions(+), 19 deletions(-) > > diff --git a/mm/migrate.c b/mm/migrate.c > index 125194f5af0f..35a88334bb3c 100644 > --- a/mm/migrate.c > +++ b/mm/migrate.c > @@ -1027,22 +1027,28 @@ union migration_ptr { > struct anon_vma *anon_vma; > struct address_space *mapping; > }; > + > +enum { > + PAGE_WAS_MAPPED = BIT(0), > + PAGE_WAS_MLOCKED = BIT(1), > +}; > + > static void __migrate_folio_record(struct folio *dst, > - unsigned long page_was_mapped, > + unsigned long old_page_state, > struct anon_vma *anon_vma) > { > union migration_ptr ptr = { .anon_vma = anon_vma }; > dst->mapping = ptr.mapping; > - dst->private = (void *)page_was_mapped; > + dst->private = (void *)old_page_state; > } > > static void __migrate_folio_extract(struct folio *dst, > - int *page_was_mappedp, > + int *old_page_state, > struct anon_vma **anon_vmap) > { > union migration_ptr ptr = { .mapping = dst->mapping }; > *anon_vmap = ptr.anon_vma; > - *page_was_mappedp = (unsigned long)dst->private; > + *old_page_state = (unsigned long)dst->private; > dst->mapping = NULL; > dst->private = NULL; > } > @@ -1103,7 +1109,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, > { > struct folio *dst; > int rc = -EAGAIN; > - int page_was_mapped = 0; > + int old_page_state = 0; > struct anon_vma *anon_vma = NULL; > bool is_lru = !__folio_test_movable(src); > bool locked = false; > @@ -1157,6 +1163,8 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, > folio_lock(src); > } > locked = true; > + if (folio_test_mlocked(src)) > + old_page_state |= PAGE_WAS_MLOCKED; > > if (folio_test_writeback(src)) { > /* > @@ -1206,7 +1214,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, > dst_locked = true; > > if (unlikely(!is_lru)) { > - __migrate_folio_record(dst, page_was_mapped, anon_vma); > + __migrate_folio_record(dst, old_page_state, anon_vma); > return MIGRATEPAGE_UNMAP; > } > > @@ -1232,11 +1240,11 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, > VM_BUG_ON_FOLIO(folio_test_anon(src) && > !folio_test_ksm(src) && !anon_vma, src); > try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0); > - page_was_mapped = 1; > + old_page_state |= PAGE_WAS_MAPPED; > } > > if (!folio_mapped(src)) { > - __migrate_folio_record(dst, page_was_mapped, anon_vma); > + __migrate_folio_record(dst, old_page_state, anon_vma); > return MIGRATEPAGE_UNMAP; > } > > @@ -1248,7 +1256,8 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, > if (rc == -EAGAIN) > ret = NULL; > > - migrate_folio_undo_src(src, page_was_mapped, anon_vma, locked, ret); > + migrate_folio_undo_src(src, old_page_state & PAGE_WAS_MAPPED, > + anon_vma, locked, ret); > migrate_folio_undo_dst(dst, dst_locked, put_new_folio, private); > > return rc; > @@ -1261,12 +1270,12 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, > struct list_head *ret) > { > int rc; > - int page_was_mapped = 0; > + int old_page_state = 0; > struct anon_vma *anon_vma = NULL; > bool is_lru = !__folio_test_movable(src); > struct list_head *prev; > > - __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); > + __migrate_folio_extract(dst, &old_page_state, &anon_vma); > prev = dst->lru.prev; > list_del(&dst->lru); > > @@ -1287,10 +1296,10 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, > * isolated from the unevictable LRU: but this case is the easiest. > */ > folio_add_lru(dst); > - if (page_was_mapped) > + if (old_page_state & PAGE_WAS_MLOCKED) > lru_add_drain(); > > - if (page_was_mapped) > + if (old_page_state & PAGE_WAS_MAPPED) > remove_migration_ptes(src, dst, false); > > out_unlock_both: > @@ -1322,11 +1331,12 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, > */ > if (rc == -EAGAIN) { > list_add(&dst->lru, prev); > - __migrate_folio_record(dst, page_was_mapped, anon_vma); > + __migrate_folio_record(dst, old_page_state, anon_vma); > return rc; > } > > - migrate_folio_undo_src(src, page_was_mapped, anon_vma, true, ret); > + migrate_folio_undo_src(src, old_page_state & PAGE_WAS_MAPPED, > + anon_vma, true, ret); > migrate_folio_undo_dst(dst, true, put_new_folio, private); > > return rc; > @@ -1799,12 +1809,12 @@ static int migrate_pages_batch(struct list_head *from, > dst = list_first_entry(&dst_folios, struct folio, lru); > dst2 = list_next_entry(dst, lru); > list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { > - int page_was_mapped = 0; > + int old_page_state = 0; > struct anon_vma *anon_vma = NULL; > > - __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); > - migrate_folio_undo_src(folio, page_was_mapped, anon_vma, > - true, ret_folios); > + __migrate_folio_extract(dst, &old_page_state, &anon_vma); > + migrate_folio_undo_src(folio, old_page_state & PAGE_WAS_MAPPED, > + anon_vma, true, ret_folios); > list_del(&dst->lru); > migrate_folio_undo_dst(dst, true, put_new_folio, private); > dst = dst2;