Received: by 2002:a05:6a10:22f:0:0:0:0 with SMTP id 15csp3176832pxk; Mon, 28 Sep 2020 10:09:45 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz+1ES6P/G7YKtzCAOQF/Lafx+DdBEyvdk5EkqQt/8sPjYYEjkkTfKrEdqqTNvi2iWQJhxx X-Received: by 2002:a17:906:c411:: with SMTP id u17mr2547645ejz.319.1601312984864; Mon, 28 Sep 2020 10:09:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1601312984; cv=none; d=google.com; s=arc-20160816; b=qHm1bVaviW+3T2ST7ptdpSFrP797vjQO2rcYDU1enx/N/JjD70GzbFuH3gcxfeHXg+ YQ0YsasRWYOY9J8ZT32mxAPdtE4uwsXlJZOvBnf3HHWUnb7C7GmqQEpRTYvNv4gAzKmY lDxUcccH29hUwZ1wqvOdlMqWzlpQIOAlnHPFp1TWJb7n8MpU5UwAyMVYB+BVzpX5oO8N zr4O37yCQeWazhxE5VoMw6MaGpjFxhic6ugALtIX1bEisAaDs3EjZaw8LUlvynUE7I3j MhjTa+DmQflZcn4qiCWO/F2e82pyXLRLJPSfu6mDAG1LjO+3CSC0kKYJ2+jxjKZFxBfO /k9g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=YcQ8rra3NPnXWY406yj7ihQd6xlsIOxNSWD2vGesaj0=; b=XoHtT9nqXOc7DtSyaILhlp52MdvPpo5ERb9HUui3r39Q0bdijBBb1ymovEHLlynGBE oXku/FPMr44ti86p9Ql0MCAcdd3AfdKGTQWpfAy3MK1xosfHkDUwh9MgziXRJ6C9ygvG 2Pt526qfN5mtdGzz4YdjPIHrH+dGlN41eUj5UhWM8NYoDdF1n+9+U++EEVJnKvLRsPSy aPUYu/Jty4qNXBe+V6cyDZ6eKjyocYCnH8B6sbj08Vra/4DEZ8MrnBCRSU10UQYBSFB/ YPJr97kaRcs0LTLiCS7prhwKNvbGGeK1mSYJPSs7c6PkbOpSQD5RY6wLfC7360sbeErr uc4A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=HzhAY1+A; spf=pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-nfs-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id f23si1043450edm.413.2020.09.28.10.09.10; Mon, 28 Sep 2020 10:09:44 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=HzhAY1+A; spf=pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-nfs-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726348AbgI1RJF (ORCPT + 99 others); Mon, 28 Sep 2020 13:09:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36558 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726328AbgI1RJF (ORCPT ); Mon, 28 Sep 2020 13:09:05 -0400 Received: from mail-qk1-x742.google.com (mail-qk1-x742.google.com [IPv6:2607:f8b0:4864:20::742]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DB484C061755 for ; Mon, 28 Sep 2020 10:09:04 -0700 (PDT) Received: by mail-qk1-x742.google.com with SMTP id v123so1636796qkd.9 for ; Mon, 28 Sep 2020 10:09:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=YcQ8rra3NPnXWY406yj7ihQd6xlsIOxNSWD2vGesaj0=; b=HzhAY1+AD13++DGgXeh8M3cPq8KOFlZun8yyKfy94ImqNdhTzVnA/1OKCxF9Jk/XU3 ZYKxuS+knURAnAX3f9FPYdvRgtN6+Dge3lHOs9hJwsPiU7Shpu9F9uM9+6MTNyTmzixy U0al6GzHeG39RA+AeCllWdP3GR10wiAN/pQugXXou9Oy8HpjQn5byjyr4eOHNw/6RQZ3 TbJcIVztyHHrXRTXVO2gtxFivXPnM8Luxb3I62DhkX5Ohb1iVTepDwKkk7z1iYirWyWm zlr1Rb8Tw5SW964ImPKnPqmAsJoG8VTCUzUH3AlJjRDNFw+1/Go9gDcQ/VCvEA0vwLsb hiqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :mime-version:content-transfer-encoding; bh=YcQ8rra3NPnXWY406yj7ihQd6xlsIOxNSWD2vGesaj0=; b=WSoLmqB/fFBbo8X3JZSs8zUGRkBQbUYwqyqGeZHh4bzIF2FHNj4zrIX85G8bW/j7Nr xKZ0QnPUtlVgcH+1lWmWVdxf3+HuaAvjbL+e4rlGcgyhCSpUi2sNaxaT3kuML0UvoLZQ b9Y3QZtmuQb+EzxCfadlxF+Oiss2kxAKGAnhW6ln8+kA3v9tOetycLh4j/kzcWdPlgdX YSXXLObgO1+LWOZvyfrs8YNmpYSLGmN6nP2acZooDiWm9RgYo+2mLveyy7ZzqmdBNZJU duD0TnZGafEQi+ESlKyUuQZ5N4o4mfewNEZDg58oeZN/EJguqlseaIyoLC4HsTh/ou61 /XJA== X-Gm-Message-State: AOAM530pjCwIqYkpc+F7yfuXAIPOPNmF5Nakpe0FsXoJTn06ranIPfwI Dx7yKSkIRoX0fXgRSxbUlEk= X-Received: by 2002:a05:620a:148:: with SMTP id e8mr470849qkn.186.1601312943816; Mon, 28 Sep 2020 10:09:03 -0700 (PDT) Received: from gouda.nowheycreamery.com (c-68-32-74-190.hsd1.mi.comcast.net. [68.32.74.190]) by smtp.gmail.com with ESMTPSA id k20sm2011631qtb.34.2020.09.28.10.09.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Sep 2020 10:09:03 -0700 (PDT) Sender: Anna Schumaker From: schumaker.anna@gmail.com X-Google-Original-From: Anna.Schumaker@Netapp.com To: bfields@redhat.com, chuck.lever@oracle.com, linux-nfs@vger.kernel.org Cc: Anna.Schumaker@Netapp.com Subject: [PATCH v6 0/5] NFSD: Add support for the v4.2 READ_PLUS operation Date: Mon, 28 Sep 2020 13:08:56 -0400 Message-Id: <20200928170901.707554-1-Anna.Schumaker@Netapp.com> X-Mailer: git-send-email 2.28.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org From: Anna Schumaker These patches add server support for the READ_PLUS operation, which breaks read requests into several "data" and "hole" segments when replying to the client. - Changes since v5: - Set the right buffer size through svc_reserve() - Fix up nfsd4_read_plus_rsize() to make sure we have enough buffer - Limit maxcount to the amount of buffer space when encoding data - Bail out of unexpected hole values by encoding a data segment - Rebase to v5.9-rc7 Here are the results of some performance tests I ran on some lab machines. I tested by reading various 2G files from a few different underlying filesystems and across several NFS versions. I used the `vmtouch` utility to make sure files were only cached when we wanted them to be. In addition to 100% data and 100% hole cases, I also tested with files that alternate between data and hole segments. These files have either 4K, 8K, 16K, or 32K segment sizes and start with either data or hole segments. So the file mixed-4d has a 4K segment size beginning with a data segment, but mixed-32h has 32K segments beginning with a hole. The units are in seconds, with the first number for each NFS version being the uncached read time and the second number is for when the file is cached on the server. I added some extra data collection (client cpu percentage and sys time), but the extra data means I couldn't figure out a way to break this down into a concise table. I cut out v3 and v4.0 performance numbers to get the size down, but I kept v4.1 for comparison because it uses the same code that v4.2 without read plus uses. Read Plus Results (ext4): data :... v4.1 ... Uncached ... 20.540 s, 105 MB/s, 0.65 s kern, 3% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu :... v4.2 ... Uncached ... 20.605 s, 104 MB/s, 0.65 s kern, 3% cpu :....... Cached ..... 18.253 s, 118 MB/s, 0.67 s kern, 3% cpu hole :... v4.1 ... Uncached ... 18.255 s, 118 MB/s, 0.72 s kern, 3% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.72 s kern, 3% cpu :... v4.2 ... Uncached ... 0.847 s, 2.5 GB/s, 0.73 s kern, 86% cpu :....... Cached ..... 0.845 s, 2.5 GB/s, 0.72 s kern, 85% cpu mixed-4d :... v4.1 ... Uncached ... 54.691 s, 39 MB/s, 0.75 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu :... v4.2 ... Uncached ... 51.587 s, 42 MB/s, 0.75 s kern, 1% cpu :....... Cached ..... 9.215 s, 233 MB/s, 0.67 s kern, 7% cpu mixed-8d :... v4.1 ... Uncached ... 37.072 s, 58 MB/s, 0.67 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu :... v4.2 ... Uncached ... 33.259 s, 65 MB/s, 0.68 s kern, 2% cpu :....... Cached ..... 9.172 s, 234 MB/s, 0.67 s kern, 7% cpu mixed-16d :... v4.1 ... Uncached ... 27.138 s, 79 MB/s, 0.73 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu :... v4.2 ... Uncached ... 23.042 s, 93 MB/s, 0.73 s kern, 3% cpu :....... Cached ..... 9.150 s, 235 MB/s, 0.66 s kern, 7% cpu mixed-32d :... v4.1 ... Uncached ... 25.326 s, 85 MB/s, 0.68 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu :... v4.2 ... Uncached ... 21.125 s, 102 MB/s, 0.69 s kern, 3% cpu :....... Cached ..... 9.140 s, 235 MB/s, 0.67 s kern, 7% cpu mixed-4h :... v4.1 ... Uncached ... 58.317 s, 37 MB/s, 0.75 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu :... v4.2 ... Uncached ... 51.878 s, 41 MB/s, 0.74 s kern, 1% cpu :....... Cached ..... 9.215 s, 233 MB/s, 0.68 s kern, 7% cpu mixed-8h :... v4.1 ... Uncached ... 36.855 s, 58 MB/s, 0.68 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.72 s kern, 3% cpu :... v4.2 ... Uncached ... 29.457 s, 73 MB/s, 0.68 s kern, 2% cpu :....... Cached ..... 9.172 s, 234 MB/s, 0.67 s kern, 7% cpu mixed-16h :... v4.1 ... Uncached ... 26.460 s, 81 MB/s, 0.74 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu :... v4.2 ... Uncached ... 19.587 s, 110 MB/s, 0.74 s kern, 3% cpu :....... Cached ..... 9.150 s, 235 MB/s, 0.67 s kern, 7% cpu mixed-32h :... v4.1 ... Uncached ... 25.495 s, 84 MB/s, 0.69 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.65 s kern, 3% cpu :... v4.2 ... Uncached ... 17.634 s, 122 MB/s, 0.69 s kern, 3% cpu :....... Cached ..... 9.140 s, 235 MB/s, 0.68 s kern, 7% cpu Read Plus Results (xfs): data :... v4.1 ... Uncached ... 20.230 s, 106 MB/s, 0.65 s kern, 3% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.68 s kern, 3% cpu :... v4.2 ... Uncached ... 20.724 s, 104 MB/s, 0.65 s kern, 3% cpu :....... Cached ..... 18.253 s, 118 MB/s, 0.67 s kern, 3% cpu hole :... v4.1 ... Uncached ... 18.255 s, 118 MB/s, 0.68 s kern, 3% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.69 s kern, 3% cpu :... v4.2 ... Uncached ... 0.904 s, 2.4 GB/s, 0.72 s kern, 79% cpu :....... Cached ..... 0.908 s, 2.4 GB/s, 0.73 s kern, 80% cpu mixed-4d :... v4.1 ... Uncached ... 57.553 s, 37 MB/s, 0.77 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu :... v4.2 ... Uncached ... 37.162 s, 58 MB/s, 0.73 s kern, 1% cpu :....... Cached ..... 9.215 s, 233 MB/s, 0.67 s kern, 7% cpu mixed-8d :... v4.1 ... Uncached ... 36.754 s, 58 MB/s, 0.69 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.68 s kern, 3% cpu :... v4.2 ... Uncached ... 24.454 s, 88 MB/s, 0.69 s kern, 2% cpu :....... Cached ..... 9.172 s, 234 MB/s, 0.66 s kern, 7% cpu mixed-16d :... v4.1 ... Uncached ... 27.156 s, 79 MB/s, 0.73 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu :... v4.2 ... Uncached ... 22.934 s, 94 MB/s, 0.72 s kern, 3% cpu :....... Cached ..... 9.150 s, 235 MB/s, 0.68 s kern, 7% cpu mixed-32d :... v4.1 ... Uncached ... 27.849 s, 77 MB/s, 0.68 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.72 s kern, 3% cpu :... v4.2 ... Uncached ... 23.670 s, 91 MB/s, 0.67 s kern, 2% cpu :....... Cached ..... 9.139 s, 235 MB/s, 0.64 s kern, 7% cpu mixed-4h :... v4.1 ... Uncached ... 57.639 s, 37 MB/s, 0.72 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.69 s kern, 3% cpu :... v4.2 ... Uncached ... 35.503 s, 61 MB/s, 0.72 s kern, 2% cpu :....... Cached ..... 9.215 s, 233 MB/s, 0.66 s kern, 7% cpu mixed-8h :... v4.1 ... Uncached ... 37.044 s, 58 MB/s, 0.71 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.68 s kern, 3% cpu :... v4.2 ... Uncached ... 23.779 s, 90 MB/s, 0.69 s kern, 2% cpu :....... Cached ..... 9.172 s, 234 MB/s, 0.65 s kern, 7% cpu mixed-16h :... v4.1 ... Uncached ... 27.167 s, 79 MB/s, 0.73 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.67 s kern, 3% cpu :... v4.2 ... Uncached ... 19.088 s, 113 MB/s, 0.75 s kern, 3% cpu :....... Cached ..... 9.159 s, 234 MB/s, 0.66 s kern, 7% cpu mixed-32h :... v4.1 ... Uncached ... 27.592 s, 78 MB/s, 0.71 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.68 s kern, 3% cpu :... v4.2 ... Uncached ... 19.682 s, 109 MB/s, 0.67 s kern, 3% cpu :....... Cached ..... 9.140 s, 235 MB/s, 0.67 s kern, 7% cpu Read Plus Results (btrfs): data :... v4.1 ... Uncached ... 21.317 s, 101 MB/s, 0.63 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.67 s kern, 3% cpu :... v4.2 ... Uncached ... 28.665 s, 75 MB/s, 0.65 s kern, 2% cpu :....... Cached ..... 18.253 s, 118 MB/s, 0.66 s kern, 3% cpu hole :... v4.1 ... Uncached ... 18.256 s, 118 MB/s, 0.70 s kern, 3% cpu : :....... Cached ..... 18.254 s, 118 MB/s, 0.73 s kern, 4% cpu :... v4.2 ... Uncached ... 0.851 s, 2.5 GB/s, 0.72 s kern, 84% cpu :....... Cached ..... 0.847 s, 2.5 GB/s, 0.73 s kern, 86% cpu mixed-4d :... v4.1 ... Uncached ... 56.857 s, 38 MB/s, 0.76 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.72 s kern, 3% cpu :... v4.2 ... Uncached ... 54.455 s, 39 MB/s, 0.73 s kern, 1% cpu :....... Cached ..... 9.215 s, 233 MB/s, 0.68 s kern, 7% cpu mixed-8d :... v4.1 ... Uncached ... 36.641 s, 59 MB/s, 0.68 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu :... v4.2 ... Uncached ... 33.205 s, 65 MB/s, 0.67 s kern, 2% cpu :....... Cached ..... 9.172 s, 234 MB/s, 0.65 s kern, 7% cpu mixed-16d :... v4.1 ... Uncached ... 28.653 s, 75 MB/s, 0.72 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu :... v4.2 ... Uncached ... 25.748 s, 83 MB/s, 0.71 s kern, 2% cpu :....... Cached ..... 9.150 s, 235 MB/s, 0.64 s kern, 7% cpu mixed-32d :... v4.1 ... Uncached ... 28.886 s, 74 MB/s, 0.67 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.71 s kern, 3% cpu :... v4.2 ... Uncached ... 24.724 s, 87 MB/s, 0.74 s kern, 2% cpu :....... Cached ..... 9.140 s, 235 MB/s, 0.63 s kern, 6% cpu mixed-4h :... v4.1 ... Uncached ... 52.181 s, 41 MB/s, 0.73 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.66 s kern, 3% cpu :... v4.2 ... Uncached ... 150.341 s, 14 MB/s, 0.72 s kern, 0% cpu :....... Cached ..... 9.216 s, 233 MB/s, 0.63 s kern, 6% cpu mixed-8h :... v4.1 ... Uncached ... 36.945 s, 58 MB/s, 0.68 s kern, 1% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.65 s kern, 3% cpu :... v4.2 ... Uncached ... 79.781 s, 27 MB/s, 0.68 s kern, 0% cpu :....... Cached ..... 9.172 s, 234 MB/s, 0.66 s kern, 7% cpu mixed-16h :... v4.1 ... Uncached ... 28.651 s, 75 MB/s, 0.73 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.66 s kern, 3% cpu :... v4.2 ... Uncached ... 47.428 s, 45 MB/s, 0.71 s kern, 1% cpu :....... Cached ..... 9.150 s, 235 MB/s, 0.67 s kern, 7% cpu mixed-32h :... v4.1 ... Uncached ... 28.618 s, 75 MB/s, 0.69 s kern, 2% cpu : :....... Cached ..... 18.252 s, 118 MB/s, 0.70 s kern, 3% cpu :... v4.2 ... Uncached ... 38.813 s, 55 MB/s, 0.67 s kern, 1% cpu :....... Cached ..... 9.140 s, 235 MB/s, 0.61 s kern, 6% cpu Thoughts? Anna Anna Schumaker (5): SUNRPC/NFSD: Implement xdr_reserve_space_vec() NFSD: Add READ_PLUS data support NFSD: Add READ_PLUS hole segment encoding NFSD: Return both a hole and a data segment NFSD: Encode a full READ_PLUS reply fs/nfsd/nfs4proc.c | 21 +++++ fs/nfsd/nfs4xdr.c | 177 +++++++++++++++++++++++++++++++------ include/linux/sunrpc/xdr.h | 2 + net/sunrpc/xdr.c | 45 ++++++++++ 4 files changed, 217 insertions(+), 28 deletions(-) -- 2.28.0