Received: by 2002:a05:6a10:5bc5:0:0:0:0 with SMTP id os5csp3598970pxb; Wed, 13 Oct 2021 09:07:05 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyDBus64fcGZo/jP2JnAA0Uol7wceAXHTOx/E/WRznfBEPN+zpyIRuKmj/Va3R+UHzleUeg X-Received: by 2002:a17:90a:d3c3:: with SMTP id d3mr171129pjw.140.1634141225740; Wed, 13 Oct 2021 09:07:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1634141225; cv=none; d=google.com; s=arc-20160816; b=eg8ERa+j1/arhPtEfIAvHbg2uncUB1pzWba9OmqoXbLZusLbR6HFVZa7SbWI+vYoq+ qmYlvb/8zLMvrHHjki9g/ZlE9dhq5sUwY81wPm3J28cn2BPv5wlpz7TqW9gjrTplC7DP Nzg9P/s9AvALCcSgfumCxJ2C7V3KUzA7HLGpRhB/+2EtRoPQcLBU4hQc1y4Qro9lUyOw Y40JheWkLf7jQneHmmqpitogeYi7xE6fshqzSJ9D+qEC9P5szJLDuQXYGiegE0Mrv2Sx 0bPI4mgTmESlLZ2xu9v0PGD9zpJo8gqS3xCTWBhW+GJ/zkfTFQKiyGmBvOorVqtOfzcG efTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=4aTU3EupNAF3fXFzgSlutYMVrbUDuqlP4zujHMtjctE=; b=I36LVAoLocKgW0vpnt9nH/q0XTdjjxen+wbqrRpCsBB7rhNbEajZQo51JFIiuXbfa/ /5d9I/R7KEJUdvkYrfWkVILRo8mxUVryVm7CqmVJ0CM66xtO8TFcIp1IrOPQFil/dxLo rEcoctGkrIp2Mk7w9NUtYIqw8G1Zvq9cQmFyvuR6Vwr+fFA6UoduXa/SIWNH0piNjL1s ipJGHezNbhrwiej3jX8O4sqRBtrvDfI2TicTBG8UX2A3gNnZUfXzceMo/aWGx1z5cTHd csE2LOPblXc+727gEESIYUM0PRHlVx5bDlndMaZ/QE77/c4UixZ5I7nIQL3Wr7KvWcGL x95A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="h/tWd++e"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id e4si15397909pgm.562.2021.10.13.09.06.30; Wed, 13 Oct 2021 09:07:05 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b="h/tWd++e"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235623AbhJMQDD (ORCPT + 99 others); Wed, 13 Oct 2021 12:03:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43438 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235922AbhJMQC5 (ORCPT ); Wed, 13 Oct 2021 12:02:57 -0400 Received: from mail-qk1-x735.google.com (mail-qk1-x735.google.com [IPv6:2607:f8b0:4864:20::735]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E6C42C061749; Wed, 13 Oct 2021 09:00:53 -0700 (PDT) Received: by mail-qk1-x735.google.com with SMTP id t63so2686795qkf.1; Wed, 13 Oct 2021 09:00:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4aTU3EupNAF3fXFzgSlutYMVrbUDuqlP4zujHMtjctE=; b=h/tWd++epVlpSaGdmTnilIlcyoE4elRhjQ5cQBfgBNNAybjIUgMF7G+BxL4Gz2wyxE 0QHkDhRjnB65xE1/FxwYiKeBBUyNpqjrhXNEeuAwQK2YgOAWwD3GWnJblUt3hHwlAm0A q/xJLwP2grqJWENkukwpicdLRAGXk7bor0Pnf2JQWAoRefnvQfbBtxXm+L7qz+VeaeTD sm8GfA57hO9VmllHYZ9HS59wzeMYZWIGl2pK6SKPAFWFqgFhJuRXtz+ysgQSpv/uCGni 48pLrcUZKicCLOdT/4Tj8qMNu1U/g5KGJHUb4946n0GOQXrrTaNoBbvbKxTe05aqbQKq 1Abg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4aTU3EupNAF3fXFzgSlutYMVrbUDuqlP4zujHMtjctE=; b=tdd0fghaM89iNL/HGMwdebYitWcS6kBrHwvrMo6Jk5IC76Qs8a04Nf6hsmPIvBfCF6 Cc4F9weHD0sD2p5SmbAdUREeUoEvH29QS81jQciTTZEywJ7+1rVsxHUPEbyAGXx857YU taJwxN1xTopSP8Ao+cP12BLS5DdWsQH+T5GeqQspJhxvsn86Ljc+RiYjLBeZCFQEArEU jYLMBZ5Sjm7XZAxnN7rvTkxTi3//NSIY7WK10wFtIzNLhaBbtUcT7g+ezu/li2iE6ciX 5FbUCBL01gDsaGYnHl4UdtBpeiKe5yBBJ16mIcw51TxQYw/Ft0iw5ItYDHPElrfkh8w1 CkTA== X-Gm-Message-State: AOAM5317rGmYDkYkaZbiGcDwosdBAsJxGcsZ9dnYNpX8G2nXp+Da40YO le3svqCT51tI8uoYrkJmytoQj2eqZxT3 X-Received: by 2002:a37:8883:: with SMTP id k125mr100968qkd.144.1634140844944; Wed, 13 Oct 2021 09:00:44 -0700 (PDT) Received: from moria.home.lan (c-73-219-103-14.hsd1.vt.comcast.net. [73.219.103.14]) by smtp.gmail.com with ESMTPSA id w17sm20161qts.53.2021.10.13.09.00.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Oct 2021 09:00:44 -0700 (PDT) From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, linux-raid@vger.kernel.org, linux-block@vger.kernel.org, axboe@kernel.dk Cc: Kent Overstreet , alexander.h.duyck@linux.intel.com Subject: [PATCH 3/5] mm/page_reporting: Improve control flow Date: Wed, 13 Oct 2021 12:00:32 -0400 Message-Id: <20211013160034.3472923-4-kent.overstreet@gmail.com> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20211013160034.3472923-1-kent.overstreet@gmail.com> References: <20211013160034.3472923-1-kent.overstreet@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This splits out page_reporting_get_pages() from page_reporting_cycle(), which is a considerable simplification and lets us delete some duplicated code. We're cleaning up code that touches page freelists as prep work for possibly converting them to radix trees, but this is a worthy cleanup on its own. Signed-off-by: Kent Overstreet --- mm/page_reporting.c | 154 ++++++++++++++++---------------------------- 1 file changed, 54 insertions(+), 100 deletions(-) diff --git a/mm/page_reporting.c b/mm/page_reporting.c index c4362b4b0c..ab2be13d8e 100644 --- a/mm/page_reporting.c +++ b/mm/page_reporting.c @@ -71,10 +71,8 @@ void __page_reporting_notify(void) static void page_reporting_drain(struct page_reporting_dev_info *prdev, - struct scatterlist *sgl, unsigned int nents, bool reported) + struct scatterlist *sg, bool reported) { - struct scatterlist *sg = sgl; - /* * Drain the now reported pages back into their respective * free lists/areas. We assume at least one page is populated. @@ -100,9 +98,45 @@ page_reporting_drain(struct page_reporting_dev_info *prdev, if (PageBuddy(page) && buddy_order(page) == order) __SetPageReported(page); } while ((sg = sg_next(sg))); +} + +static int +page_reporting_get_pages(struct page_reporting_dev_info *prdev, struct zone *zone, + unsigned int order, unsigned int mt, + struct scatterlist *sgl) +{ + struct page_free_list *list = &zone->free_area[order].free[mt]; + unsigned int page_len = PAGE_SIZE << order; + struct page *page, *next; + unsigned nr_got = 0; + + spin_lock_irq(&zone->lock); + + /* loop through free list adding unreported pages to sg list */ + list_for_each_entry_safe(page, next, &list->list, lru) { + /* We are going to skip over the reported pages. */ + if (PageReported(page)) + continue; + + /* Attempt to pull page from list and place in scatterlist */ + if (!__isolate_free_page(page, order)) { + next = page; + break; + } + + sg_set_page(&sgl[nr_got++], page, page_len, 0); + if (nr_got == PAGE_REPORTING_CAPACITY) + break; + } + + /* Rotate any leftover pages to the head of the freelist */ + if (!list_entry_is_head(next, &list->list, lru) && + !list_is_first(&next->lru, &list->list)) + list_rotate_to_front(&next->lru, &list->list); + + spin_unlock_irq(&zone->lock); - /* reinitialize scatterlist now that it is empty */ - sg_init_table(sgl, nents); + return nr_got; } /* @@ -113,23 +147,13 @@ page_reporting_drain(struct page_reporting_dev_info *prdev, static int page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone, unsigned int order, unsigned int mt, - struct scatterlist *sgl, unsigned int *offset) + struct scatterlist *sgl) { struct page_free_list *list = &zone->free_area[order].free[mt]; - unsigned int page_len = PAGE_SIZE << order; - struct page *page, *next; + unsigned nr_pages; long budget; int err = 0; - /* - * Perform early check, if free area is empty there is - * nothing to process so we can skip this free_list. - */ - if (list_empty(&list->list)) - return err; - - spin_lock_irq(&zone->lock); - /* * Limit how many calls we will be making to the page reporting * device for this list. By doing this we avoid processing any @@ -146,80 +170,25 @@ page_reporting_cycle(struct page_reporting_dev_info *prdev, struct zone *zone, */ budget = DIV_ROUND_UP(list->nr, PAGE_REPORTING_CAPACITY * 16); - /* loop through free list adding unreported pages to sg list */ - list_for_each_entry_safe(page, next, &list->list, lru) { - /* We are going to skip over the reported pages. */ - if (PageReported(page)) - continue; + while (budget > 0 && !err) { + sg_init_table(sgl, PAGE_REPORTING_CAPACITY); - /* - * If we fully consumed our budget then update our - * state to indicate that we are requesting additional - * processing and exit this list. - */ - if (budget < 0) { - atomic_set(&prdev->state, PAGE_REPORTING_REQUESTED); - next = page; + nr_pages = page_reporting_get_pages(prdev, zone, order, mt, sgl); + if (!nr_pages) break; - } - - /* Attempt to pull page from list and place in scatterlist */ - if (*offset) { - if (!__isolate_free_page(page, order)) { - next = page; - break; - } - - /* Add page to scatter list */ - --(*offset); - sg_set_page(&sgl[*offset], page, page_len, 0); - - continue; - } - - /* - * Make the first non-reported page in the free list - * the new head of the free list before we release the - * zone lock. - */ - if (!list_is_first(&page->lru, &list->list)) - list_rotate_to_front(&page->lru, &list->list); - - /* release lock before waiting on report processing */ - spin_unlock_irq(&zone->lock); - - /* begin processing pages in local list */ - err = prdev->report(prdev, sgl, PAGE_REPORTING_CAPACITY); - /* reset offset since the full list was reported */ - *offset = PAGE_REPORTING_CAPACITY; + sg_mark_end(sgl + nr_pages); - /* update budget to reflect call to report function */ - budget--; + budget -= nr_pages; + err = prdev->report(prdev, sgl, nr_pages); - /* reacquire zone lock and resume processing */ spin_lock_irq(&zone->lock); - - /* flush reported pages from the sg list */ - page_reporting_drain(prdev, sgl, PAGE_REPORTING_CAPACITY, !err); - - /* - * Reset next to first entry, the old next isn't valid - * since we dropped the lock to report the pages - */ - next = list_first_entry(&list->list, struct page, lru); - - /* exit on error */ - if (err) - break; + page_reporting_drain(prdev, sgl, !err); + spin_unlock_irq(&zone->lock); } - /* Rotate any leftover pages to the head of the freelist */ - if (!list_entry_is_head(next, &list->list, lru) && - !list_is_first(&next->lru, &list->list)) - list_rotate_to_front(&next->lru, &list->list); - - spin_unlock_irq(&zone->lock); + if (budget <= 0 && list->nr) + atomic_set(&prdev->state, PAGE_REPORTING_REQUESTED); return err; } @@ -228,7 +197,7 @@ static int page_reporting_process_zone(struct page_reporting_dev_info *prdev, struct scatterlist *sgl, struct zone *zone) { - unsigned int order, mt, leftover, offset = PAGE_REPORTING_CAPACITY; + unsigned int order, mt; unsigned long watermark; int err = 0; @@ -250,25 +219,12 @@ page_reporting_process_zone(struct page_reporting_dev_info *prdev, if (is_migrate_isolate(mt)) continue; - err = page_reporting_cycle(prdev, zone, order, mt, - sgl, &offset); + err = page_reporting_cycle(prdev, zone, order, mt, sgl); if (err) return err; } } - /* report the leftover pages before going idle */ - leftover = PAGE_REPORTING_CAPACITY - offset; - if (leftover) { - sgl = &sgl[offset]; - err = prdev->report(prdev, sgl, leftover); - - /* flush any remaining pages out from the last report */ - spin_lock_irq(&zone->lock); - page_reporting_drain(prdev, sgl, leftover, !err); - spin_unlock_irq(&zone->lock); - } - return err; } @@ -294,8 +250,6 @@ static void page_reporting_process(struct work_struct *work) if (!sgl) goto err_out; - sg_init_table(sgl, PAGE_REPORTING_CAPACITY); - for_each_zone(zone) { err = page_reporting_process_zone(prdev, sgl, zone); if (err) -- 2.33.0