Received: by 2002:a05:6a10:a0d1:0:0:0:0 with SMTP id j17csp2479211pxa; Mon, 17 Aug 2020 10:34:50 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwSsNFGf6Im2qNJlYcJuUTlL7AGga4JYOKeq50AVFiFEQmAAUCHkMf3Lm6FiOACkf4Qc7ku X-Received: by 2002:a17:906:3c59:: with SMTP id i25mr15645452ejg.202.1597685689897; Mon, 17 Aug 2020 10:34:49 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1597685689; cv=none; d=google.com; s=arc-20160816; b=cnGZcbZx0+gRY2lwUGtxFs+411iaMNuVkej78B66Orf4nfIL94kJOWMJ0u8gI3J8+T NB+tDzdS8kFm84l+0araKbquLF/QF/lwBykQ0BQSYsj0MBrHfXIGD1JSe6iFgWwVmQi0 U9IXU5ekJCGyIZ24W+r2T3J8h9fo7W842tOe+leiTrzsxi87X/y51InizLAQDfcTZmJF bWDjqJPGz5JM8rx0x88nh/tDquZSFgfQCAZFaqmyPZ0EuYtUP1ab0dtaF4OX67SzXA6P ezQNDAqpSIuORdYsjMAlUQhvEKNXKl10TpMGvasvvoG1BijD8CK1YQDqbWaIkeS43wj4 D40A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=q1k0oCkzNlyf9haEyDCHvVVWRKDoM3DdBmBg1DVuiZg=; b=aifrfrEazXbB/ha7K165wrWT7cst0FxWAHXbjrAOdLQfij4CVLPB/YKaCEfgeiQvZ3 RpqCPtCoEhQu7JF7gc0JY/wHXwIRrJ/co06epXWWyfEtVeRJFG4+Gj4a6HznpfUlWAfb wn8YaPyiobUBjOF/70+QnPaOvg2gL8+vvoj8ioJTNqDvJSUEZz611k4iTL65VkXhR+jp RMQwU7uksOKr0XlwSGBAyk8ZrxeBls+uvh7B5k68zztOCruImC948k4pLt3wKmngEOjt NJdy5RrpJ15qT5aAuvc/MSWbbfaWssz0jVokR/lDAU/8pOefpjs/TrHAVVxOAi0x3zSI PCSw== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@gmail.com header.s=20161025 header.b=T9XVfvTA; spf=pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-nfs-owner@vger.kernel.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id h12si12540692ejx.85.2020.08.17.10.34.26; Mon, 17 Aug 2020 10:34:49 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=fail header.i=@gmail.com header.s=20161025 header.b=T9XVfvTA; spf=pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-nfs-owner@vger.kernel.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389102AbgHQRdg (ORCPT + 99 others); Mon, 17 Aug 2020 13:33:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46714 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389098AbgHQQxQ (ORCPT ); Mon, 17 Aug 2020 12:53:16 -0400 Received: from mail-io1-xd41.google.com (mail-io1-xd41.google.com [IPv6:2607:f8b0:4864:20::d41]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E1FCCC061342 for ; Mon, 17 Aug 2020 09:53:12 -0700 (PDT) Received: by mail-io1-xd41.google.com with SMTP id t15so18363005iob.3 for ; Mon, 17 Aug 2020 09:53:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=q1k0oCkzNlyf9haEyDCHvVVWRKDoM3DdBmBg1DVuiZg=; b=T9XVfvTA6HnJ0LzXfrGDlT41wQYUuwhfEse01FGWztajMrBtNBtbAf06qA6PJeOhbB AFzri0LJSj1lFx+gz0PZrp2JDSUQqn5NFm9ZfMv8/1Am4mj1Kruv9PTODwQAAxsb/p2t CihfLAlx0ccwHoF0ntYwwOEZ7bMf154i6NeANVvFQb5hUc5kpTpZJjJfuyQd2W63cQgO DUhj3H6kT+nA+Dc2MxYiiBAWqTn0XElDPIsyeLjbHTnjr+OhW6f5Zq6r+DIuSXfIAl3Q LnnoG9mCnyvbxbH4hIqW2soTPrX4qFXaa7KCsxXeahnx+G81ZHo0wP9ZzMnetzVKwBmL W/Cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :mime-version:content-transfer-encoding; bh=q1k0oCkzNlyf9haEyDCHvVVWRKDoM3DdBmBg1DVuiZg=; b=W8RVCwkMt87tSYtBLNN0xyKiFJkrRbvZzET5VfrvZ0/kR85CWQUTu1ng+GyNSCX5ra Q8i+q6yN38J8cw2wsWJRUzs7rhLen/pa/F3M0oYUKDdVGkh8TBmdwAv5VJb75cjMXau1 2ocsymKMb2JK/aFdl4KDmvoAK4yLebAFYtEtUR18zRebS/EFBENMS7N1euqFj7lZUOD8 1az5dFviDtjJJKPxPGEpkRKS8hAToPxgK60/byeOl96Lu9hYuixraZMKLBTJAgYnBUDR 2ELr1VbhiHHmdLqC5g9U/kHVCKEfyMqTBJCiapZlGITRgq07y5FQrg82rXWnME9aLobF D3hg== X-Gm-Message-State: AOAM532/RpVrKMhaJGx9ReUjNJFkJJkDojb2P/QbgOXdcKv3b13xNi+X mg44kwa0R7ccnX8FmRw1+jI= X-Received: by 2002:a05:6638:12d4:: with SMTP id v20mr14654371jas.108.1597683191985; Mon, 17 Aug 2020 09:53:11 -0700 (PDT) Received: from gouda.nowheycreamery.com (c-68-32-74-190.hsd1.mi.comcast.net. [68.32.74.190]) by smtp.gmail.com with ESMTPSA id s3sm9410039iol.49.2020.08.17.09.53.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 17 Aug 2020 09:53:11 -0700 (PDT) From: schumaker.anna@gmail.com X-Google-Original-From: Anna.Schumaker@Netapp.com To: bfields@redhat.com, chuck.lever@oracle.com, linux-nfs@vger.kernel.org Cc: Anna.Schumaker@Netapp.com Subject: [PATCH v4 0/5] NFSD: Add support for the v4.2 READ_PLUS operation Date: Mon, 17 Aug 2020 12:53:05 -0400 Message-Id: <20200817165310.354092-1-Anna.Schumaker@Netapp.com> X-Mailer: git-send-email 2.28.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org From: Anna Schumaker These patches add server support for the READ_PLUS operation, which breaks read requests into several "data" and "hole" segments when replying to the client. - Changes since v3: - Combine first two patches related to xdr_reserve_space_vec() - Remove unnecessary call to svc_encode_read_payload() Here are the results of some performance tests I ran on some lab machines. I tested by reading various 2G files from a few different underlying filesystems and across several NFS versions. I used the `vmtouch` utility to make sure files were only cached when we wanted them to be. In addition to 100% data and 100% hole cases, I also tested with files that alternate between data and hole segments. These files have either 4K, 8K, 16K, or 32K segment sizes and start with either data or hole segments. So the file mixed-4d has a 4K segment size beginning with a data segment, but mixed-32h has 32K segments beginning with a hole. The units are in seconds, with the first number for each NFS version being the uncached read time and the second number is for when the file is cached on the server. I added some extra data collection (client cpu percentage and sys time), but the extra data means I couldn't figure out a way to break this down into a concise table. I cut out v3 and v4.0 performance numbers to get the size down, but I kept v4.1 for comparison because it uses the same code that v4.2 without read plus uses. Read Plus Results (ext4): data :... v4.1 ... Uncached ... 20.540 s, 105 MB/s, 0.65 s kern, 3% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu :... v4.2 ... Uncached ... 20.605 s, 104 MB/s, 0.65 s kern, 3% cpu :....... Cached ..... 18.253 s, 118 MB/s, 0.67 s kern, 3% cpu hole :... v4.1 ... Uncached ... 18.255 s, 118 MB/s, 0.72 s kern, 3% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.72 s kern, 3% cpu :... v4.2 ... Uncached ... 0.847 s, 2.5 GB/s, 0.73 s kern, 86% cpu :....... Cached ..... 0.845 s, 2.5 GB/s, 0.72 s kern, 85% cpu mixed-4d :... v4.1 ... Uncached ... 54.691 s, 39 MB/s, 0.75 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu :... v4.2 ... Uncached ... 51.587 s, 42 MB/s, 0.75 s kern, 1% cpu :....... Cached ..... 9.215 s, 233 MB/s, 0.67 s kern, 7% cpu mixed-8d :... v4.1 ... Uncached ... 37.072 s, 58 MB/s, 0.67 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu :... v4.2 ... Uncached ... 33.259 s, 65 MB/s, 0.68 s kern, 2% cpu :....... Cached ..... 9.172 s, 234 MB/s, 0.67 s kern, 7% cpu mixed-16d :... v4.1 ... Uncached ... 27.138 s, 79 MB/s, 0.73 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu :... v4.2 ... Uncached ... 23.042 s, 93 MB/s, 0.73 s kern, 3% cpu :....... Cached ..... 9.150 s, 235 MB/s, 0.66 s kern, 7% cpu mixed-32d :... v4.1 ... Uncached ... 25.326 s, 85 MB/s, 0.68 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu :... v4.2 ... Uncached ... 21.125 s, 102 MB/s, 0.69 s kern, 3% cpu :....... Cached ..... 9.140 s, 235 MB/s, 0.67 s kern, 7% cpu mixed-4h :... v4.1 ... Uncached ... 58.317 s, 37 MB/s, 0.75 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu :... v4.2 ... Uncached ... 51.878 s, 41 MB/s, 0.74 s kern, 1% cpu :....... Cached ..... 9.215 s, 233 MB/s, 0.68 s kern, 7% cpu mixed-8h :... v4.1 ... Uncached ... 36.855 s, 58 MB/s, 0.68 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.72 s kern, 3% cpu :... v4.2 ... Uncached ... 29.457 s, 73 MB/s, 0.68 s kern, 2% cpu :....... Cached ..... 9.172 s, 234 MB/s, 0.67 s kern, 7% cpu mixed-16h :... v4.1 ... Uncached ... 26.460 s, 81 MB/s, 0.74 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu :... v4.2 ... Uncached ... 19.587 s, 110 MB/s, 0.74 s kern, 3% cpu :....... Cached ..... 9.150 s, 235 MB/s, 0.67 s kern, 7% cpu mixed-32h :... v4.1 ... Uncached ... 25.495 s, 84 MB/s, 0.69 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.65 s kern, 3% cpu :... v4.2 ... Uncached ... 17.634 s, 122 MB/s, 0.69 s kern, 3% cpu :....... Cached ..... 9.140 s, 235 MB/s, 0.68 s kern, 7% cpu Read Plus Results (xfs): data :... v4.1 ... Uncached ... 20.230 s, 106 MB/s, 0.65 s kern, 3% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.68 s kern, 3% cpu :... v4.2 ... Uncached ... 20.724 s, 104 MB/s, 0.65 s kern, 3% cpu :....... Cached ..... 18.253 s, 118 MB/s, 0.67 s kern, 3% cpu hole :... v4.1 ... Uncached ... 18.255 s, 118 MB/s, 0.68 s kern, 3% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.69 s kern, 3% cpu :... v4.2 ... Uncached ... 0.904 s, 2.4 GB/s, 0.72 s kern, 79% cpu :....... Cached ..... 0.908 s, 2.4 GB/s, 0.73 s kern, 80% cpu mixed-4d :... v4.1 ... Uncached ... 57.553 s, 37 MB/s, 0.77 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu :... v4.2 ... Uncached ... 37.162 s, 58 MB/s, 0.73 s kern, 1% cpu :....... Cached ..... 9.215 s, 233 MB/s, 0.67 s kern, 7% cpu mixed-8d :... v4.1 ... Uncached ... 36.754 s, 58 MB/s, 0.69 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.68 s kern, 3% cpu :... v4.2 ... Uncached ... 24.454 s, 88 MB/s, 0.69 s kern, 2% cpu :....... Cached ..... 9.172 s, 234 MB/s, 0.66 s kern, 7% cpu mixed-16d :... v4.1 ... Uncached ... 27.156 s, 79 MB/s, 0.73 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu :... v4.2 ... Uncached ... 22.934 s, 94 MB/s, 0.72 s kern, 3% cpu :....... Cached ..... 9.150 s, 235 MB/s, 0.68 s kern, 7% cpu mixed-32d :... v4.1 ... Uncached ... 27.849 s, 77 MB/s, 0.68 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.72 s kern, 3% cpu :... v4.2 ... Uncached ... 23.670 s, 91 MB/s, 0.67 s kern, 2% cpu :....... Cached ..... 9.139 s, 235 MB/s, 0.64 s kern, 7% cpu mixed-4h :... v4.1 ... Uncached ... 57.639 s, 37 MB/s, 0.72 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.69 s kern, 3% cpu :... v4.2 ... Uncached ... 35.503 s, 61 MB/s, 0.72 s kern, 2% cpu :....... Cached ..... 9.215 s, 233 MB/s, 0.66 s kern, 7% cpu mixed-8h :... v4.1 ... Uncached ... 37.044 s, 58 MB/s, 0.71 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.68 s kern, 3% cpu :... v4.2 ... Uncached ... 23.779 s, 90 MB/s, 0.69 s kern, 2% cpu :....... Cached ..... 9.172 s, 234 MB/s, 0.65 s kern, 7% cpu mixed-16h :... v4.1 ... Uncached ... 27.167 s, 79 MB/s, 0.73 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.67 s kern, 3% cpu :... v4.2 ... Uncached ... 19.088 s, 113 MB/s, 0.75 s kern, 3% cpu :....... Cached ..... 9.159 s, 234 MB/s, 0.66 s kern, 7% cpu mixed-32h :... v4.1 ... Uncached ... 27.592 s, 78 MB/s, 0.71 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.68 s kern, 3% cpu :... v4.2 ... Uncached ... 19.682 s, 109 MB/s, 0.67 s kern, 3% cpu :....... Cached ..... 9.140 s, 235 MB/s, 0.67 s kern, 7% cpu Read Plus Results (btrfs): data :... v4.1 ... Uncached ... 21.317 s, 101 MB/s, 0.63 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.67 s kern, 3% cpu :... v4.2 ... Uncached ... 28.665 s, 75 MB/s, 0.65 s kern, 2% cpu :....... Cached ..... 18.253 s, 118 MB/s, 0.66 s kern, 3% cpu hole :... v4.1 ... Uncached ... 18.256 s, 118 MB/s, 0.70 s kern, 3% cpu : :....... Cached ..... 18.254 s, 118 MB/s, 0.73 s kern, 4% cpu :... v4.2 ... Uncached ... 0.851 s, 2.5 GB/s, 0.72 s kern, 84% cpu :....... Cached ..... 0.847 s, 2.5 GB/s, 0.73 s kern, 86% cpu mixed-4d :... v4.1 ... Uncached ... 56.857 s, 38 MB/s, 0.76 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.72 s kern, 3% cpu :... v4.2 ... Uncached ... 54.455 s, 39 MB/s, 0.73 s kern, 1% cpu :....... Cached ..... 9.215 s, 233 MB/s, 0.68 s kern, 7% cpu mixed-8d :... v4.1 ... Uncached ... 36.641 s, 59 MB/s, 0.68 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu :... v4.2 ... Uncached ... 33.205 s, 65 MB/s, 0.67 s kern, 2% cpu :....... Cached ..... 9.172 s, 234 MB/s, 0.65 s kern, 7% cpu mixed-16d :... v4.1 ... Uncached ... 28.653 s, 75 MB/s, 0.72 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu :... v4.2 ... Uncached ... 25.748 s, 83 MB/s, 0.71 s kern, 2% cpu :....... Cached ..... 9.150 s, 235 MB/s, 0.64 s kern, 7% cpu mixed-32d :... v4.1 ... Uncached ... 28.886 s, 74 MB/s, 0.67 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu :... v4.2 ... Uncached ... 24.724 s, 87 MB/s, 0.74 s kern, 2% cpu :....... Cached ..... 9.140 s, 235 MB/s, 0.63 s kern, 6% cpu mixed-4h :... v4.1 ... Uncached ... 52.181 s, 41 MB/s, 0.73 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.66 s kern, 3% cpu :... v4.2 ... Uncached ... 150.341 s, 14 MB/s, 0.72 s kern, 0% cpu :....... Cached ..... 9.216 s, 233 MB/s, 0.63 s kern, 6% cpu mixed-8h :... v4.1 ... Uncached ... 36.945 s, 58 MB/s, 0.68 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.65 s kern, 3% cpu :... v4.2 ... Uncached ... 79.781 s, 27 MB/s, 0.68 s kern, 0% cpu :....... Cached ..... 9.172 s, 234 MB/s, 0.66 s kern, 7% cpu mixed-16h :... v4.1 ... Uncached ... 28.651 s, 75 MB/s, 0.73 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.66 s kern, 3% cpu :... v4.2 ... Uncached ... 47.428 s, 45 MB/s, 0.71 s kern, 1% cpu :....... Cached ..... 9.150 s, 235 MB/s, 0.67 s kern, 7% cpu mixed-32h :... v4.1 ... Uncached ... 28.618 s, 75 MB/s, 0.69 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu :... v4.2 ... Uncached ... 38.813 s, 55 MB/s, 0.67 s kern, 1% cpu :....... Cached ..... 9.140 s, 235 MB/s, 0.61 s kern, 6% cpu Thoughts? Anna Anna Schumaker (5): SUNRPC/NFSD: Implement xdr_reserve_space_vec() NFSD: Add READ_PLUS data support NFSD: Add READ_PLUS hole segment encoding NFSD: Return both a hole and a data segment NFSD: Encode a full READ_PLUS reply fs/nfsd/nfs4proc.c | 17 ++++ fs/nfsd/nfs4xdr.c | 167 +++++++++++++++++++++++++++++++------ include/linux/sunrpc/xdr.h | 2 + net/sunrpc/xdr.c | 45 ++++++++++ 4 files changed, 204 insertions(+), 27 deletions(-) -- 2.28.0