Received: by 2002:ac0:a594:0:0:0:0:0 with SMTP id m20-v6csp2036002imm; Thu, 24 May 2018 04:54:58 -0700 (PDT) X-Google-Smtp-Source: AB8JxZpFQj/SCglyFLpGO4P3I/G9J/ytPWjCLK4PM7kg/B28WAgLCbLsm9sf6NyBd/OK2OO4PWKd X-Received: by 2002:a62:9696:: with SMTP id s22-v6mr6938629pfk.191.1527162898396; Thu, 24 May 2018 04:54:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1527162898; cv=none; d=google.com; s=arc-20160816; b=bCL8yBjgMfwN4F7fYI7EtYskIkX256RaiKXhYrcKRZNmbhULoOnKAyYnqXTbQctvru UKFQZqdGrLKLxXRfzCuf80yWaczZkrEnK4cL2sgX08hlY5J+dDSsMLjG+drheYXF0Eun 5epJI2XP2MWLdaPW3Pbo7UbmVskAc8GTDTekZBwpVkTTulTX2B4dZmhr5OTpssrP87nD P018Jg3eZNaSAP4sWJeo4EdIL2e6iV2PhL5FfjTCOABvd993FPtNMFYtgr4FauEgus+p gK47PCWbwmzYUv5AF5v7BSK1hdf8YZtyvTvUb2gQRq89qQjz+Q++JfSSVvQNkhIBDZUU aqqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:mime-version:user-agent:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature :arc-authentication-results; bh=TqskCnkVG+W9zviwrWHgCFdupV/FcOoifbN+nA5qrHw=; b=FJorr+NTdjmsYNaQpdhItGzYcy9sQPMYSMl3s0RjdEuoa9A0xAa5xBncWIMd+1ZifV pD61FUvdBVJn+lCNISV/2ti9acy92xpklWr0uC7RhyT6b8QNkBPaRYwR9MejMXUJoDxK ncgGos1aaWWj4j80LwzrTPow2A0ULf3UBxV3Heq6/VwEMuzNyWPiS79TmhOU+vm+sn3l 5ZdEcSK9m3cZpR1SemQBKmvJ7loHyNqtSYgKd9KLVAjrCesDiPo31rpMTa2X8Z1EsoM/ LaOwWa8Dy7bw5rI4wDqHjqAidfkuozFqpFihXkLfsj5qPANK5FVsv0VjtEAu3kOg7g3o A8MQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=zbKy/vae; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id o11-v6si10608852pgp.331.2018.05.24.04.54.43; Thu, 24 May 2018 04:54:58 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=zbKy/vae; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1033194AbeEXLyD (ORCPT + 99 others); Thu, 24 May 2018 07:54:03 -0400 Received: from mail.kernel.org ([198.145.29.99]:57482 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S966895AbeEXJqS (ORCPT ); Thu, 24 May 2018 05:46:18 -0400 Received: from localhost (LFbn-1-12247-202.w90-92.abo.wanadoo.fr [90.92.61.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 980BC20892; Thu, 24 May 2018 09:46:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1527155178; bh=yUT1BFWm0a7fKsHzLISfnSUpoK3f3k5tG7+eFbgG5Uc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=zbKy/vaeMfX65p+Sx88A+65rGbz4SfghegfX57duwDIJJtINjKajADPbMRv0vX9g3 s11QFgTwQjsEL8E8+eZcElRPdBv/jT4ZHvcjDybjKV9DMZvePd3qyB/zJF/O1vz6Qi uGc6hHZGFBfrfVhRFeNvPfRiddsfJypA7vDeV9kw= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Mel Gorman , Jan Kara , Hugh Dickins , Andrew Morton , Linus Torvalds , Mel Gorman Subject: [PATCH 4.4 50/92] mm: filemap: avoid unnecessary calls to lock_page when waiting for IO to complete during a read Date: Thu, 24 May 2018 11:38:27 +0200 Message-Id: <20180524093204.290399449@linuxfoundation.org> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180524093159.286472249@linuxfoundation.org> References: <20180524093159.286472249@linuxfoundation.org> User-Agent: quilt/0.65 X-stable: review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org 4.4-stable review patch. If anyone has any objections, please let me know. ------------------ From: Mel Gorman commit ebded02788b5d7c7600f8cff26ae07896d568649 upstream. In the generic read paths the kernel looks up a page in the page cache and if it's up to date, it is used. If not, the page lock is acquired to wait for IO to complete and then check the page. If multiple processes are waiting on IO, they all serialise against the lock and duplicate the checks. This is unnecessary. The page lock in itself does not give any guarantees to the callers about the page state as it can be immediately truncated or reclaimed after the page is unlocked. It's sufficient to wait_on_page_locked and then continue if the page is up to date on wakeup. It is possible that a truncated but up-to-date page is returned but the reference taken during read prevents it disappearing underneath the caller and the data is still valid if PageUptodate. The overall impact is small as even if processes serialise on the lock, the lock section is tiny once the IO is complete. Profiles indicated that unlock_page and friends are generally a tiny portion of a read-intensive workload. An artificial test was created that had instances of dd access a cache-cold file on an ext4 filesystem and measure how long the read took. paralleldd 4.4.0 4.4.0 vanilla avoidlock Amean Elapsd-1 5.28 ( 0.00%) 5.15 ( 2.50%) Amean Elapsd-4 5.29 ( 0.00%) 5.17 ( 2.12%) Amean Elapsd-7 5.28 ( 0.00%) 5.18 ( 1.78%) Amean Elapsd-12 5.20 ( 0.00%) 5.33 ( -2.50%) Amean Elapsd-21 5.14 ( 0.00%) 5.21 ( -1.41%) Amean Elapsd-30 5.30 ( 0.00%) 5.12 ( 3.38%) Amean Elapsd-48 5.78 ( 0.00%) 5.42 ( 6.21%) Amean Elapsd-79 6.78 ( 0.00%) 6.62 ( 2.46%) Amean Elapsd-110 9.09 ( 0.00%) 8.99 ( 1.15%) Amean Elapsd-128 10.60 ( 0.00%) 10.43 ( 1.66%) The impact is small but intuitively, it makes sense to avoid unnecessary calls to lock_page. Signed-off-by: Mel Gorman Reviewed-by: Jan Kara Cc: Hugh Dickins Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds Signed-off-by: Mel Gorman Signed-off-by: Greg Kroah-Hartman --- mm/filemap.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1581,6 +1581,15 @@ find_page: index, last_index - index); } if (!PageUptodate(page)) { + /* + * See comment in do_read_cache_page on why + * wait_on_page_locked is used to avoid unnecessarily + * serialisations and why it's safe. + */ + wait_on_page_locked_killable(page); + if (PageUptodate(page)) + goto page_ok; + if (inode->i_blkbits == PAGE_CACHE_SHIFT || !mapping->a_ops->is_partially_uptodate) goto page_not_up_to_date; @@ -2253,12 +2262,52 @@ filler: if (PageUptodate(page)) goto out; + /* + * Page is not up to date and may be locked due one of the following + * case a: Page is being filled and the page lock is held + * case b: Read/write error clearing the page uptodate status + * case c: Truncation in progress (page locked) + * case d: Reclaim in progress + * + * Case a, the page will be up to date when the page is unlocked. + * There is no need to serialise on the page lock here as the page + * is pinned so the lock gives no additional protection. Even if the + * the page is truncated, the data is still valid if PageUptodate as + * it's a race vs truncate race. + * Case b, the page will not be up to date + * Case c, the page may be truncated but in itself, the data may still + * be valid after IO completes as it's a read vs truncate race. The + * operation must restart if the page is not uptodate on unlock but + * otherwise serialising on page lock to stabilise the mapping gives + * no additional guarantees to the caller as the page lock is + * released before return. + * Case d, similar to truncation. If reclaim holds the page lock, it + * will be a race with remove_mapping that determines if the mapping + * is valid on unlock but otherwise the data is valid and there is + * no need to serialise with page lock. + * + * As the page lock gives no additional guarantee, we optimistically + * wait on the page to be unlocked and check if it's up to date and + * use the page if it is. Otherwise, the page lock is required to + * distinguish between the different cases. The motivation is that we + * avoid spurious serialisations and wakeups when multiple processes + * wait on the same page for IO to complete. + */ + wait_on_page_locked(page); + if (PageUptodate(page)) + goto out; + + /* Distinguish between all the cases under the safety of the lock */ lock_page(page); + + /* Case c or d, restart the operation */ if (!page->mapping) { unlock_page(page); page_cache_release(page); goto repeat; } + + /* Someone else locked and filled the page in a very small window */ if (PageUptodate(page)) { unlock_page(page); goto out;