Received: by 2002:a05:6a10:9848:0:0:0:0 with SMTP id x8csp1590288pxf; Fri, 9 Apr 2021 12:13:45 -0700 (PDT) X-Google-Smtp-Source: ABdhPJw1ctkB4T6A9RyWXtC1vNiaVyfNaa5CVnN2pmGsH2dKP/jl0/ZJcR1QxbLg8CZHpvoYoePh X-Received: by 2002:a17:902:7886:b029:e7:1052:adad with SMTP id q6-20020a1709027886b02900e71052adadmr14196078pll.75.1617995625471; Fri, 09 Apr 2021 12:13:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1617995625; cv=none; d=google.com; s=arc-20160816; b=tQZB2qQK3wGSL38kc6F3DTkoMMnYsV2qhn6nx7pZVFUFD57WjSlKHd0uvxQeo5k5Dc 141RQyQGFPY6OK1E9ziz+2L8D8H2M4jruG29f0THw3hfIUUosdofn6yNXs3k3squHj59 AICBLqhF3i7uAms3Hp4pan9W6SgYzywY/7zpphWA0rmfjVhh0Gs+s+8IZUuGJkKBk01z A+y2Yj37o0HXgeLRD6RamjfjKnXsSiT1y/Ob693Zp4J6sE/sVLlqSsUqJeD7vrV9aKZm o86gQntXg6RpPDUWpLPW3emhDB+iT6dy7Zy0G8dYipL0R3hzUBoldJgP/dL38DGYxC3S Bt1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=8llFfVFQeSOldbPAX6YFfl8k4K4jSiDJr4B2tQQ4Duk=; b=s9tm9uCx31wMsqp6YPO1tEZkS3Vj8I8Tn7ytAUb3xUQDb9J+7OeRCjTzFJE78/m5/c tcFXaN2U9MsIUyaKSpjASrVkZgJCwlzTwQkNrfmb3ZmY0/cpcJ1B6DDnsFdcihT2irF5 8UqXcPK2NdoXtf+hbG8NXNRsZxUvbJKZBVm1oHZxYJHdYpITTxk/TgKYy9v7CKkUoLTZ Fx1tJgrgPjeg9Q8fyPnljuRiFcMEQgY/wMxpMqD1a63qPt05VcIJpaE/l0S7bbtf5/g/ v4LBoI7pJBs2FmMq/2fjGM38G71BQVrmWoiNca1JMUTEFyzcZFONfY/hjEL5eP8OvkYa UnAA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=Mm+Et4Ee; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id w62si3625683pfd.306.2021.04.09.12.13.32; Fri, 09 Apr 2021 12:13:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=Mm+Et4Ee; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234942AbhDITLz (ORCPT + 99 others); Fri, 9 Apr 2021 15:11:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34470 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234692AbhDITLz (ORCPT ); Fri, 9 Apr 2021 15:11:55 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3755BC061762; Fri, 9 Apr 2021 12:11:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=8llFfVFQeSOldbPAX6YFfl8k4K4jSiDJr4B2tQQ4Duk=; b=Mm+Et4EebEbBqrpAk/68zJO/AU +jsST+RQ5XF3tPFRK8bm/KBi/pXXs+cniAhOsHqo4iF2+9Gr3Bvx/qqhTMzDHIWGiA4bJajafOzXT SyIV7W27P5ClX2287QvNrPhS1x5W6P1piVHrquHGVO6+9yXX0j8jO+VS13j5hXsNJjtTUNNXv16LB z3COttthLNL6f7fgol5IWLu4Qg6Y+LixIek/QDE6omUferlnZxmj7oG7NhWVKAh8qQDS5Q3hOIT+S sIm0ywXz1oYrDLvDHIoZ4YvLoBDYOSGUdL1z9B38FmaD/kSUCKt2tZFn/oXuqE+kXEJ3KgeDvLyxj 3vEcThdA==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lUwWW-000og0-Fx; Fri, 09 Apr 2021 19:10:49 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-cachefs@redhat.com, linux-afs@lists.infradead.org, Christoph Hellwig , Jeff Layton Subject: [PATCH v7 26/28] mm/filemap: Convert wait_on_page_bit to wait_on_folio_bit Date: Fri, 9 Apr 2021 19:51:03 +0100 Message-Id: <20210409185105.188284-27-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210409185105.188284-1-willy@infradead.org> References: <20210409185105.188284-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org We must always wait on the folio, otherwise we won't be woken up. This commit shrinks the kernel by 691 bytes, mostly due to moving the page waitqueue lookup into wait_on_folio_bit_common(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton --- include/linux/pagemap.h | 10 +++--- mm/filemap.c | 67 ++++++++++++++++++++--------------------- mm/page-writeback.c | 4 +-- 3 files changed, 39 insertions(+), 42 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index d50fc5adbee1..5bccccff48eb 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -790,11 +790,11 @@ static inline int lock_page_or_retry(struct page *page, struct mm_struct *mm, } /* - * This is exported only for wait_on_page_locked/wait_on_page_writeback, etc., + * This is exported only for wait_on_folio_locked/wait_on_folio_writeback, etc., * and should not be used directly. */ -extern void wait_on_page_bit(struct page *page, int bit_nr); -extern int wait_on_page_bit_killable(struct page *page, int bit_nr); +extern void wait_on_folio_bit(struct folio *folio, int bit_nr); +extern int wait_on_folio_bit_killable(struct folio *folio, int bit_nr); /* * Wait for a folio to be unlocked. @@ -806,14 +806,14 @@ extern int wait_on_page_bit_killable(struct page *page, int bit_nr); static inline void wait_on_folio_locked(struct folio *folio) { if (FolioLocked(folio)) - wait_on_page_bit(&folio->page, PG_locked); + wait_on_folio_bit(folio, PG_locked); } static inline int wait_on_folio_locked_killable(struct folio *folio) { if (!FolioLocked(folio)) return 0; - return wait_on_page_bit_killable(&folio->page, PG_locked); + return wait_on_folio_bit_killable(folio, PG_locked); } static inline void wait_on_page_locked(struct page *page) diff --git a/mm/filemap.c b/mm/filemap.c index cdb8250af510..8f07e21a8f29 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1102,7 +1102,7 @@ static int wake_page_function(wait_queue_entry_t *wait, unsigned mode, int sync, * * So update the flags atomically, and wake up the waiter * afterwards to avoid any races. This store-release pairs - * with the load-acquire in wait_on_page_bit_common(). + * with the load-acquire in wait_on_folio_bit_common(). */ smp_store_release(&wait->flags, flags | WQ_FLAG_WOKEN); wake_up_state(wait->private, mode); @@ -1183,7 +1183,7 @@ static void wake_up_folio(struct folio *folio, int bit) } /* - * A choice of three behaviors for wait_on_page_bit_common(): + * A choice of three behaviors for wait_on_folio_bit_common(): */ enum behavior { EXCLUSIVE, /* Hold ref to page and take the bit when woken, like @@ -1217,9 +1217,10 @@ static inline bool trylock_page_bit_common(struct page *page, int bit_nr, /* How many times do we accept lock stealing from under a waiter? */ int sysctl_page_lock_unfairness = 5; -static inline int wait_on_page_bit_common(wait_queue_head_t *q, - struct page *page, int bit_nr, int state, enum behavior behavior) +static inline int wait_on_folio_bit_common(struct folio *folio, int bit_nr, + int state, enum behavior behavior) { + wait_queue_head_t *q = page_waitqueue(&folio->page); int unfairness = sysctl_page_lock_unfairness; struct wait_page_queue wait_page; wait_queue_entry_t *wait = &wait_page.wait; @@ -1228,8 +1229,8 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, unsigned long pflags; if (bit_nr == PG_locked && - !PageUptodate(page) && PageWorkingset(page)) { - if (!PageSwapBacked(page)) { + !FolioUptodate(folio) && FolioWorkingset(folio)) { + if (!FolioSwapBacked(folio)) { delayacct_thrashing_start(); delayacct = true; } @@ -1239,7 +1240,7 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, init_wait(wait); wait->func = wake_page_function; - wait_page.page = page; + wait_page.page = &folio->page; wait_page.bit_nr = bit_nr; repeat: @@ -1254,7 +1255,7 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, * Do one last check whether we can get the * page bit synchronously. * - * Do the SetPageWaiters() marking before that + * Do the SetFolioWaiters() marking before that * to let any waker we _just_ missed know they * need to wake us up (otherwise they'll never * even go to the slow case that looks at the @@ -1265,8 +1266,8 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, * lock to avoid races. */ spin_lock_irq(&q->lock); - SetPageWaiters(page); - if (!trylock_page_bit_common(page, bit_nr, wait)) + SetFolioWaiters(folio); + if (!trylock_page_bit_common(&folio->page, bit_nr, wait)) __add_wait_queue_entry_tail(q, wait); spin_unlock_irq(&q->lock); @@ -1276,10 +1277,10 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, * see whether the page bit testing has already * been done by the wake function. * - * We can drop our reference to the page. + * We can drop our reference to the folio. */ if (behavior == DROP) - put_page(page); + put_folio(folio); /* * Note that until the "finish_wait()", or until @@ -1316,7 +1317,7 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, * * And if that fails, we'll have to retry this all. */ - if (unlikely(test_and_set_bit(bit_nr, &page->flags))) + if (unlikely(test_and_set_bit(bit_nr, folio_flags(folio, 0)))) goto repeat; wait->flags |= WQ_FLAG_DONE; @@ -1325,7 +1326,7 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, /* * If a signal happened, this 'finish_wait()' may remove the last - * waiter from the wait-queues, but the PageWaiters bit will remain + * waiter from the wait-queues, but the FolioWaiters bit will remain * set. That's ok. The next wakeup will take care of it, and trying * to do it here would be difficult and prone to races. */ @@ -1356,19 +1357,17 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, return wait->flags & WQ_FLAG_WOKEN ? 0 : -EINTR; } -void wait_on_page_bit(struct page *page, int bit_nr) +void wait_on_folio_bit(struct folio *folio, int bit_nr) { - wait_queue_head_t *q = page_waitqueue(page); - wait_on_page_bit_common(q, page, bit_nr, TASK_UNINTERRUPTIBLE, SHARED); + wait_on_folio_bit_common(folio, bit_nr, TASK_UNINTERRUPTIBLE, SHARED); } -EXPORT_SYMBOL(wait_on_page_bit); +EXPORT_SYMBOL(wait_on_folio_bit); -int wait_on_page_bit_killable(struct page *page, int bit_nr) +int wait_on_folio_bit_killable(struct folio *folio, int bit_nr) { - wait_queue_head_t *q = page_waitqueue(page); - return wait_on_page_bit_common(q, page, bit_nr, TASK_KILLABLE, SHARED); + return wait_on_folio_bit_common(folio, bit_nr, TASK_KILLABLE, SHARED); } -EXPORT_SYMBOL(wait_on_page_bit_killable); +EXPORT_SYMBOL(wait_on_folio_bit_killable); /** * put_and_wait_on_page_locked - Drop a reference and wait for it to be unlocked @@ -1385,11 +1384,8 @@ EXPORT_SYMBOL(wait_on_page_bit_killable); */ int put_and_wait_on_page_locked(struct page *page, int state) { - wait_queue_head_t *q; - - page = compound_head(page); - q = page_waitqueue(page); - return wait_on_page_bit_common(q, page, PG_locked, state, DROP); + return wait_on_folio_bit_common(page_folio(page), PG_locked, state, + DROP); } /** @@ -1481,8 +1477,10 @@ EXPORT_SYMBOL(end_page_private_2); */ void wait_on_page_private_2(struct page *page) { - while (PagePrivate2(page)) - wait_on_page_bit(page, PG_private_2); + struct folio *folio = page_folio(page); + + while (FolioPrivate2(folio)) + wait_on_folio_bit(folio, PG_private_2); } EXPORT_SYMBOL(wait_on_page_private_2); @@ -1499,10 +1497,11 @@ EXPORT_SYMBOL(wait_on_page_private_2); */ int wait_on_page_private_2_killable(struct page *page) { + struct folio *folio = page_folio(page); int ret = 0; - while (PagePrivate2(page)) { - ret = wait_on_page_bit_killable(page, PG_private_2); + while (FolioPrivate2(folio)) { + ret = wait_on_folio_bit_killable(folio, PG_private_2); if (ret < 0) break; } @@ -1579,16 +1578,14 @@ EXPORT_SYMBOL_GPL(page_endio); */ void __lock_folio(struct folio *folio) { - wait_queue_head_t *q = page_waitqueue(&folio->page); - wait_on_page_bit_common(q, &folio->page, PG_locked, TASK_UNINTERRUPTIBLE, + wait_on_folio_bit_common(folio, PG_locked, TASK_UNINTERRUPTIBLE, EXCLUSIVE); } EXPORT_SYMBOL(__lock_folio); int __lock_folio_killable(struct folio *folio) { - wait_queue_head_t *q = page_waitqueue(&folio->page); - return wait_on_page_bit_common(q, &folio->page, PG_locked, TASK_KILLABLE, + return wait_on_folio_bit_common(folio, PG_locked, TASK_KILLABLE, EXCLUSIVE); } EXPORT_SYMBOL_GPL(__lock_folio_killable); diff --git a/mm/page-writeback.c b/mm/page-writeback.c index 9d55ceec05c0..7aed4feabdd2 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -2834,7 +2834,7 @@ void wait_on_folio_writeback(struct folio *folio) { while (FolioWriteback(folio)) { trace_wait_on_page_writeback(&folio->page, folio_mapping(folio)); - wait_on_page_bit(&folio->page, PG_writeback); + wait_on_folio_bit(folio, PG_writeback); } } EXPORT_SYMBOL_GPL(wait_on_folio_writeback); @@ -2856,7 +2856,7 @@ int wait_on_folio_writeback_killable(struct folio *folio) { while (FolioWriteback(folio)) { trace_wait_on_page_writeback(&folio->page, folio_mapping(folio)); - if (wait_on_page_bit_killable(&folio->page, PG_writeback)) + if (wait_on_folio_bit_killable(folio, PG_writeback)) return -EINTR; } -- 2.30.2