Received: by 2002:a25:4158:0:0:0:0:0 with SMTP id o85csp285914yba; Mon, 20 May 2019 08:41:35 -0700 (PDT) X-Google-Smtp-Source: APXvYqwjeX5bsgSBj/PiReyvbho0ecIqHyIvEg4FlZ2gkyn1Ezg/lmWiKrIOBrt5ZpBip74w9sKP X-Received: by 2002:a63:170a:: with SMTP id x10mr74779517pgl.355.1558366895292; Mon, 20 May 2019 08:41:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558366895; cv=none; d=google.com; s=arc-20160816; b=mExBXjLokpK7NMlmP58f1C3caZgTrjk46IeQGfqdArQwxSl6aO7HyrK7tffbCjX29b 4sYl7KFW7FP13Mfr5Qd+bxdlT2iiRf6eyg8PVGMVgSfX9GTfZVTCIpZzOB76Z08vAYE+ VG85Lzu5cC3sWygIEQ2xcxCHBF78v1GnSw/YcNNV9a1zeatXPRJjF2AyjUM1ighVxfjJ Lrre3LTCVrTy5AeNa6owq6ue86fvLFFXqBxZZNP5xWqHl8n5shVj+JSHso4WJ3IErjen FcUf1QphwI45JiA4Rjz1kA5oPzmtHcyrSWCCEbCdINhm6luYm/tY1pXaZA5HMoGdjcZI q0bQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=59Qsyy52Etp6wTkFwKHyibof4h4K4kqP2U7JvPjQQO8=; b=kAxfu6vNRPL6d5+ENYXPgRiq55kUsvBy3GX7+4354J1dGkO+MWXh3bTmsXe7jI9iVG OhoHtJlXCrScvX7s8VP77FmgitfckWOn2351po8jvp33D1GaGm13hI3TOsXwRp2fvJcg MsO+M2KH1/Eb0qrLIWPE697c4tbc8+PbSWtomjqdAMws0H2FOIeQqaaSFPV4wISGQrFl cSzXNtuFXd5nHe1K79t6HMB701SisyahAkix3gJs95c3CV0MdB8/G4Ui6aUiV1KfMgJF CP81zI/ZsIn2NFGKCSD7/+nehIATtYB3B98cqsTwkgMRowd8ZrkogkzktzrRRs4EhX2g QQsg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=xp2ucKI8; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id v3si18265574pgi.171.2019.05.20.08.41.19; Mon, 20 May 2019 08:41:35 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=xp2ucKI8; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388972AbfETMZD (ORCPT + 99 others); Mon, 20 May 2019 08:25:03 -0400 Received: from mail.kernel.org ([198.145.29.99]:40030 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388944AbfETMZC (ORCPT ); Mon, 20 May 2019 08:25:02 -0400 Received: from localhost (83-86-89-107.cable.dynamic.v4.ziggo.nl [83.86.89.107]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 4D58020815; Mon, 20 May 2019 12:25:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1558355101; bh=0pBuoUQ/MsGpZKKsbJ6wezWXwWTvffx32ikV0Olb63c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=xp2ucKI8RI8CseTIFuWRYTdwkeQgtEkud/Y50X7idZKFTPLV5T3FRJaerixvEBCuL gmuAINJFkYiw0bXtquqYjGOOGHlNZGoKxkVK+tbhowWp4SGJRudtegZgD8q1N9UlQl jfHxVcOaHHljcGPio+0+1x6eeaFn7Fn8BjD+w8QU= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Eric Dumazet , Al Viro , Matthew Wilcox Subject: [PATCH 4.19 098/105] iov_iter: optimize page_copy_sane() Date: Mon, 20 May 2019 14:14:44 +0200 Message-Id: <20190520115253.993660256@linuxfoundation.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190520115247.060821231@linuxfoundation.org> References: <20190520115247.060821231@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Eric Dumazet commit 6daef95b8c914866a46247232a048447fff97279 upstream. Avoid cache line miss dereferencing struct page if we can. page_copy_sane() mostly deals with order-0 pages. Extra cache line miss is visible on TCP recvmsg() calls dealing with GRO packets (typically 45 page frags are attached to one skb). Bringing the 45 struct pages into cpu cache while copying the data is not free, since the freeing of the skb (and associated page frags put_page()) can happen after cache lines have been evicted. Signed-off-by: Eric Dumazet Cc: Al Viro Signed-off-by: Al Viro Cc: Matthew Wilcox Signed-off-by: Greg Kroah-Hartman --- lib/iov_iter.c | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -817,8 +817,21 @@ EXPORT_SYMBOL(_copy_from_iter_full_nocac static inline bool page_copy_sane(struct page *page, size_t offset, size_t n) { - struct page *head = compound_head(page); - size_t v = n + offset + page_address(page) - page_address(head); + struct page *head; + size_t v = n + offset; + + /* + * The general case needs to access the page order in order + * to compute the page size. + * However, we mostly deal with order-0 pages and thus can + * avoid a possible cache line miss for requests that fit all + * page orders. + */ + if (n <= v && v <= PAGE_SIZE) + return true; + + head = compound_head(page); + v += (page - head) << PAGE_SHIFT; if (likely(n <= v && v <= (PAGE_SIZE << compound_order(head)))) return true;