Received-SPF: pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Content-Type: text/plain;
        charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\))
Subject: Re: [PATCH RFC 00/12] for-5.3 NFS/RDMA patches for review
From:   Chuck Lever <chuck.lever@oracle.com>
In-Reply-To: <20190529064055.GA10492@infradead.org>
Date:   Wed, 29 May 2019 10:35:09 -0400
Cc:     linux-rdma@vger.kernel.org,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <9557E3ED-6C62-4C7D-8854-2DFC39ED690E@oracle.com>
References: <20190528181018.19012.61210.stgit@manet.1015granger.net>
 <20190529064055.GA10492@infradead.org>
To:     Christoph Hellwig <hch@infradead.org>
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk


> On May 29, 2019, at 2:40 AM, Christoph Hellwig <hch@infradead.org> =
wrote:
>=20
> On Tue, May 28, 2019 at 02:20:50PM -0400, Chuck Lever wrote:
>> This is a series of fixes and architectural changes that should
>> improve robustness and result in better scalability of NFS/RDMA.
>> I'm sure one or two of these could be broken down a little more,
>> comments welcome.
>=20
> Just curious, do you have any performance numbers.

To watch for performance regressions and improvements, I regularly run =
several
variations of iozone, fio 70/30 mix, and multi-threaded software builds. =
I did
not note any change in throughput after applying these patches.

I'm unsure how to measure context switch rate precisely during these =
tests.

This is typical for fio 8KB random 70/30 mix on FDR Infiniband on a NUMA =
client.
Not impressive compared to NVMe, I know, but much better than NFS/TCP. =
On a
single socket client, the IOPS numbers more than double.

Jobs: 12 (f=3D12): [m(12)][100.0%][r=3D897MiB/s,w=3D386MiB/s][r=3D115k,w=3D=
49.5k IOPS][eta 00m:00s]
8k7030test: (groupid=3D0, jobs=3D12): err=3D 0: pid=3D2107: Fri May 24 =
15:22:38 2019
   read: IOPS=3D115k, BW=3D897MiB/s (941MB/s)(8603MiB/9588msec)
    slat (usec): min=3D2, max=3D6203, avg=3D 7.02, stdev=3D27.49
    clat (usec): min=3D33, max=3D13553, avg=3D1131.12, stdev=3D536.34
     lat (usec): min=3D47, max=3D13557, avg=3D1138.37, stdev=3D537.11
    clat percentiles (usec):
     |  1.00th=3D[  338],  5.00th=3D[  515], 10.00th=3D[  619], =
20.00th=3D[  750],
     | 30.00th=3D[  857], 40.00th=3D[  955], 50.00th=3D[ 1057], =
60.00th=3D[ 1156],
     | 70.00th=3D[ 1270], 80.00th=3D[ 1434], 90.00th=3D[ 1696], =
95.00th=3D[ 1926],
     | 99.00th=3D[ 2966], 99.50th=3D[ 3785], 99.90th=3D[ 5866], =
99.95th=3D[ 6652],
     | 99.99th=3D[ 8586]
   bw (  KiB/s): min=3D64624, max=3D82800, per=3D8.34%, avg=3D76631.87, =
stdev=3D2877.97, samples=3D227
   iops        : min=3D 8078, max=3D10350, avg=3D9578.91, stdev=3D359.76, =
samples=3D227
  write: IOPS=3D49.2k, BW=3D384MiB/s (403MB/s)(3685MiB/9588msec)
    slat (usec): min=3D3, max=3D7226, avg=3D 7.54, stdev=3D29.53
    clat (usec): min=3D64, max=3D14828, avg=3D1210.36, stdev=3D584.82
     lat (usec): min=3D78, max=3D14834, avg=3D1218.11, stdev=3D585.77
    clat percentiles (usec):
     |  1.00th=3D[  359],  5.00th=3D[  545], 10.00th=3D[  652], =
20.00th=3D[  791],
     | 30.00th=3D[  906], 40.00th=3D[ 1004], 50.00th=3D[ 1106], =
60.00th=3D[ 1221],
     | 70.00th=3D[ 1369], 80.00th=3D[ 1549], 90.00th=3D[ 1844], =
95.00th=3D[ 2147],
     | 99.00th=3D[ 3163], 99.50th=3D[ 4015], 99.90th=3D[ 6194], =
99.95th=3D[ 7308],
     | 99.99th=3D[ 9372]
   bw (  KiB/s): min=3D27520, max=3D36128, per=3D8.34%, avg=3D32816.45, =
stdev=3D1323.08, samples=3D227
   iops        : min=3D 3440, max=3D 4516, avg=3D4101.97, stdev=3D165.38, =
samples=3D227
  lat (usec)   : 50=3D0.01%, 100=3D0.01%, 250=3D0.31%, 500=3D3.91%, =
750=3D14.66%
  lat (usec)   : 1000=3D24.41%
  lat (msec)   : 2=3D51.69%, 4=3D4.57%, 10=3D0.44%, 20=3D0.01%
  cpu          : usr=3D3.24%, sys=3D8.11%, ctx=3D786935, majf=3D0, =
minf=3D117
  IO depths    : 1=3D0.1%, 2=3D0.1%, 4=3D0.1%, 8=3D0.1%, 16=3D100.0%, =
32=3D0.0%, >=3D64=3D0.0%
     submit    : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.0%, 32=3D0.0%, =
64=3D0.0%, >=3D64=3D0.0%
     complete  : 0=3D0.0%, 4=3D100.0%, 8=3D0.0%, 16=3D0.1%, 32=3D0.0%, =
64=3D0.0%, >=3D64=3D0.0%
     issued rwt: total=3D1101195,471669,0, short=3D0,0,0, dropped=3D0,0,0
     latency   : target=3D0, window=3D0, percentile=3D100.00%, depth=3D16

Run status group 0 (all jobs):
   READ: bw=3D897MiB/s (941MB/s), 897MiB/s-897MiB/s (941MB/s-941MB/s), =
io=3D8603MiB (9021MB), run=3D9588-9588msec
  WRITE: bw=3D384MiB/s (403MB/s), 384MiB/s-384MiB/s (403MB/s-403MB/s), =
io=3D3685MiB (3864MB), run=3D9588-9588msec


--
Chuck Lever