Received: by 2002:ad5:474a:0:0:0:0:0 with SMTP id i10csp3412812imu; Fri, 18 Jan 2019 09:56:06 -0800 (PST) X-Google-Smtp-Source: ALg8bN4WHyBvC5Vsjnp54gak1WjyBnQd3dGL+JmjbvwanMq9Q/TTmrOcYgbw9Hdy9yvnibehO5Dp X-Received: by 2002:a17:902:2c03:: with SMTP id m3mr20808468plb.125.1547834166790; Fri, 18 Jan 2019 09:56:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1547834166; cv=none; d=google.com; s=arc-20160816; b=sRIf3iExKX4qbNY7KidDZXYx2vqkKIKzWTTMMdryU4E9w2yeOR9JVT+jvr0lmIzwWf Hc4tDt55ERNTRHHCB5ohlBvEta8Cx+owiZglcCk7BZDR4UgUixUWWBb5kejJt0fJHj7I Bf4uLyafdFfRahnHsK45GhLMQhnP1OZERmLqFbw0vOK+Ls16628kSpo68NvQWhi3NXNa KuyQGf2C5KuPmBchH2+qdpjx8neFVA7RCwOUOh9Oyo06e+ugvX4SIiSHzLg8SqdNwZ9w /oAYDaHkHjhdrO0BJLONJTg0MgJaezYpVyv8/F+Ph3IN2+4axGMrMrdrND9MAIk4Py+v PnNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from; bh=q4XbqljV4csmVEUb4HH/NDfBhps7WKZmhbq0+dowWqs=; b=tyWirM/kB7Rh7Yeka5sYorXhDHXETStcyupboflPx78tNxoSDjuoQ0Za98wa/hMSf9 5dXs/btk5EImv/MQvL1/GoaeDiPaRBFfBcmu46jqjAODiiTs+AeQJM3sOZXfy9WbDSB2 R8yjv9i4DFKQUCbrbPDemIq7mJWMkYih7pJ4L1e7zBducfN3j0A5LyJkq4hs61Yykket vfvpZ7u/bpHTv9QPxciJYYXNplF3G395OjFwS9fFGNCUGDFPhXFT5XyYKj4JJxHdHxsb K5bdu8dnczoKSgTdeBoIFIsvSEyE5iucVldcazqyhaVBWsKw+W188DCxf5XEWT9pwD3B +M5w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id j20si4803814pgb.520.2019.01.18.09.55.47; Fri, 18 Jan 2019 09:56:06 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728903AbfARRxw (ORCPT + 99 others); Fri, 18 Jan 2019 12:53:52 -0500 Received: from outbound-smtp12.blacknight.com ([46.22.139.17]:35359 "EHLO outbound-smtp12.blacknight.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728613AbfARRxw (ORCPT ); Fri, 18 Jan 2019 12:53:52 -0500 Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp12.blacknight.com (Postfix) with ESMTPS id 2A6011C35BC for ; Fri, 18 Jan 2019 17:53:49 +0000 (GMT) Received: (qmail 4431 invoked from network); 18 Jan 2019 17:53:49 -0000 Received: from unknown (HELO stampy.163woodhaven.lan) (mgorman@techsingularity.net@[37.228.229.96]) by 81.17.254.9 with ESMTPA; 18 Jan 2019 17:53:49 -0000 From: Mel Gorman To: Andrew Morton Cc: David Rientjes , Andrea Arcangeli , Vlastimil Babka , Linux List Kernel Mailing , Linux-MM , Mel Gorman Subject: [PATCH 12/22] mm, compaction: Avoid rescanning the same pageblock multiple times Date: Fri, 18 Jan 2019 17:51:26 +0000 Message-Id: <20190118175136.31341-13-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20190118175136.31341-1-mgorman@techsingularity.net> References: <20190118175136.31341-1-mgorman@techsingularity.net> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Pageblocks are marked for skip when no pages are isolated after a scan. However, it's possible to hit corner cases where the migration scanner gets stuck near the boundary between the source and target scanner. Due to pages being migrated in blocks of COMPACT_CLUSTER_MAX, pages that are migrated can be reallocated before the pageblock is complete. The pageblock is not necessarily skipped so it can be rescanned multiple times. Similarly, a pageblock with some dirty/writeback pages may fail to migrate and be rescanned until writeback completes which is wasteful. This patch tracks if a pageblock is being rescanned. If so, then the entire pageblock will be migrated as one operation. This narrows the race window during which pages can be reallocated during migration. Secondly, if there are pages that cannot be isolated then the pageblock will still be fully scanned and marked for skipping. On the second rescan, the pageblock skip is set and the migration scanner makes progress. 5.0.0-rc1 5.0.0-rc1 findfree-v3r16 norescan-v3r16 Amean fault-both-1 0.00 ( 0.00%) 0.00 * 0.00%* Amean fault-both-3 3200.68 ( 0.00%) 3002.07 ( 6.21%) Amean fault-both-5 4847.75 ( 0.00%) 4684.47 ( 3.37%) Amean fault-both-7 6658.92 ( 0.00%) 6815.54 ( -2.35%) Amean fault-both-12 11077.62 ( 0.00%) 10864.02 ( 1.93%) Amean fault-both-18 12403.97 ( 0.00%) 12247.52 ( 1.26%) Amean fault-both-24 15607.10 ( 0.00%) 15683.99 ( -0.49%) Amean fault-both-30 18752.27 ( 0.00%) 18620.02 ( 0.71%) Amean fault-both-32 21207.54 ( 0.00%) 19250.28 * 9.23%* 5.0.0-rc1 5.0.0-rc1 findfree-v3r16 norescan-v3r16 Percentage huge-3 96.86 ( 0.00%) 95.00 ( -1.91%) Percentage huge-5 93.72 ( 0.00%) 94.22 ( 0.53%) Percentage huge-7 94.31 ( 0.00%) 92.35 ( -2.08%) Percentage huge-12 92.66 ( 0.00%) 91.90 ( -0.82%) Percentage huge-18 91.51 ( 0.00%) 89.58 ( -2.11%) Percentage huge-24 90.50 ( 0.00%) 90.03 ( -0.52%) Percentage huge-30 91.57 ( 0.00%) 89.14 ( -2.65%) Percentage huge-32 91.00 ( 0.00%) 90.58 ( -0.46%) Negligible difference but this was likely a case when the specific corner case was not hit. A previous run of the same patch based on an earlier iteration of the series showed large differences where migration rates could be halved when the corner case was hit. The specific corner case where migration scan rates go through the roof was due to a dirty/writeback pageblock located at the boundary of the migration/free scanner did not happen in this case. When it does happen, the scan rates multipled by massive margins. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka --- mm/compaction.c | 32 ++++++++++++++++++++++++++------ mm/internal.h | 1 + 2 files changed, 27 insertions(+), 6 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 19fea4a7b3f4..a31fea7b3f96 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -949,8 +949,11 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, cc->nr_migratepages++; nr_isolated++; - /* Avoid isolating too much */ - if (cc->nr_migratepages == COMPACT_CLUSTER_MAX) { + /* + * Avoid isolating too much unless this block is being + * rescanned (e.g. dirty/writeback pages, parallel allocation). + */ + if (cc->nr_migratepages == COMPACT_CLUSTER_MAX && !cc->rescan) { ++low_pfn; break; } @@ -997,11 +1000,14 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, spin_unlock_irqrestore(zone_lru_lock(zone), flags); /* - * Updated the cached scanner pfn if the pageblock was scanned - * without isolating a page. The pageblock may not be marked - * skipped already if there were no LRU pages in the block. + * Updated the cached scanner pfn once the pageblock has been scanned + * Pages will either be migrated in which case there is no point + * scanning in the near future or migration failed in which case the + * failure reason may persist. The block is marked for skipping if + * there were no pages isolated in the block or if the block is + * rescanned twice in a row. */ - if (low_pfn == end_pfn && !nr_isolated) { + if (low_pfn == end_pfn && (!nr_isolated || cc->rescan)) { if (valid_page && !skip_updated) set_pageblock_skip(valid_page); update_cached_migrate(cc, low_pfn); @@ -2033,6 +2039,20 @@ static enum compact_result compact_zone(struct compact_control *cc) int err; unsigned long start_pfn = cc->migrate_pfn; + /* + * Avoid multiple rescans which can happen if a page cannot be + * isolated (dirty/writeback in async mode) or if the migrated + * pages are being allocated before the pageblock is cleared. + * The first rescan will capture the entire pageblock for + * migration. If it fails, it'll be marked skip and scanning + * will proceed as normal. + */ + cc->rescan = false; + if (pageblock_start_pfn(last_migrated_pfn) == + pageblock_start_pfn(start_pfn)) { + cc->rescan = true; + } + switch (isolate_migratepages(cc->zone, cc)) { case ISOLATE_ABORT: ret = COMPACT_CONTENDED; diff --git a/mm/internal.h b/mm/internal.h index 983cb975545f..d5b999e5eb5f 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -205,6 +205,7 @@ struct compact_control { bool direct_compaction; /* False from kcompactd or /proc/... */ bool whole_zone; /* Whole zone should/has been scanned */ bool contended; /* Signal lock or sched contention */ + bool rescan; /* Rescanning the same pageblock */ }; unsigned long -- 2.16.4