Content-Type: text/plain;
        charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\))
Subject: Re: [PATCH v3 26/44] SUNRPC: Improve latency for interactive tasks
From:   Chuck Lever <chuck.lever@oracle.com>
In-Reply-To: <4e46f31cc59315533e477cb667f903731598f7f1.camel@hammerspace.com>
Date:   Mon, 31 Dec 2018 13:44:27 -0500
Cc:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <3BD5C006-55D6-4D1B-9F3D-48FA65CDE09D@oracle.com>
References: <20180917130335.112832-1-trond.myklebust@hammerspace.com>
 <20180917130335.112832-2-trond.myklebust@hammerspace.com>
 <20180917130335.112832-3-trond.myklebust@hammerspace.com>
 <20180917130335.112832-4-trond.myklebust@hammerspace.com>
 <20180917130335.112832-5-trond.myklebust@hammerspace.com>
 <20180917130335.112832-6-trond.myklebust@hammerspace.com>
 <20180917130335.112832-7-trond.myklebust@hammerspace.com>
 <20180917130335.112832-8-trond.myklebust@hammerspace.com>
 <20180917130335.112832-9-trond.myklebust@hammerspace.com>
 <20180917130335.112832-10-trond.myklebust@hammerspace.com>
 <20180917130335.112832-11-trond.myklebust@hammerspace.com>
 <20180917130335.112832-12-trond.myklebust@hammerspace.com>
 <20180917130335.112832-13-trond.myklebust@hammerspace.com>
 <20180917130335.112832-14-trond.myklebust@hammerspace.com>
 <20180917130335.112832-15-trond.myklebust@hammerspace.com>
 <20180917130335.112832-16-trond.myklebust@hammerspace.com>
 <20180917130335.112832-17-trond.myklebust@! hammerspace.com>
 <20180917130335.112832-18-trond.myklebust@hammerspace.com>
 <20180917130335.112832-19-trond.myklebust@hammerspace.com>
 <20180917130335.112832-20-trond.myklebust@hammerspace.com>
 <20180917130335.112832-21-trond.myklebust@hammerspace.com>
 <20180917130335.112832-22-trond.myklebust@hammerspace.com>
 <20180917130335.112832-23-trond.myklebust@hammerspace.com>
 <20180917130335.112832-24-trond.myklebust@hammerspace.com>
 <20180917130335.112832-25-trond.myklebust@hammerspace.com>
 <20180917130335.112832-26-trond.myklebust@hammerspace.com>
 <20180917130335.112832-27-trond.myklebust@hammerspace.com>
 <4D3465FB-041C-4BB1-AB75-03511FA5AAF1@oracle.com>
 <BC34B04D-4AD5-46FB-8DC6-310BF3FBEBD3@hammerspace.com>
 <4FB643C8-4790-42B9-AF38-622E10F6A1B2@oracle.com>
 <4e46f31cc59315533e477cb667f903731598f7f1.camel@hammerspace.com>
To:     Trond Myklebust <trondmy@hammerspace.com>
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk


> On Dec 31, 2018, at 1:09 PM, Trond Myklebust <trondmy@hammerspace.com> =
wrote:
>=20
> On Thu, 2018-12-27 at 17:34 -0500, Chuck Lever wrote:
>>> On Dec 27, 2018, at 5:14 PM, Trond Myklebust <
>>> trondmy@hammerspace.com> wrote:
>>>=20
>>>=20
>>>=20
>>>> On Dec 27, 2018, at 20:21, Chuck Lever <chuck.lever@oracle.com>
>>>> wrote:
>>>>=20
>>>> Hi Trond-
>>>>=20
>>>> I've chased down a couple of remaining regressions with the v4.20
>>>> NFS client,
>>>> and they seem to be rooted in this commit.
>>>>=20
>>>> When using sec=3Dkrb5, krb5i, or krb5p I found that multi-threaded
>>>> workloads
>>>> trigger a lot of server-side disconnects. This is with TCP and
>>>> RDMA transports.
>>>> An instrumented server shows that the client is under-running the
>>>> GSS sequence
>>>> number window. I monitored the order in which GSS sequence
>>>> numbers appear on
>>>> the wire, and after this commit, the sequence numbers are wildly
>>>> misordered.
>>>> If I revert the hunk in xprt_request_enqueue_transmit, the
>>>> problem goes away.
>>>>=20
>>>> I also found that reverting that hunk results in a 3-4%
>>>> improvement in fio
>>>> IOPS rates, as well as improvement in average and maximum latency
>>>> as reported
>>>> by fio.
>>>>=20
>>>=20
>>> Hmm=E2=80=A6 Provided the sequence numbers still lie within the =
window,
>>> then why would the order matter?
>>=20
>> The misordering is so bad that one request is delayed long enough to
>> fall outside the window. The new =E2=80=9Cneed re-encode=E2=80=9D =
logic does not
>> trigger.
>>=20
>=20
> That's weird. I can't see anything wrong with need re-encode at this
> point.

I don't think there is anything wrong with it, it looks like it's
not called in this case.


> Do the window sizes agree on the client and the server?

Yes, both are 128. I also tried with 64 on the client side and 128
on the server side. That reduces the frequency of disconnects, but
does not eliminate them.

I'm not clear what problem the logic in xprt_request_enqueue_transmit
is trying to address. It seems to me that the initial, simple
implementation of this function is entirely adequate..?


--
Chuck Lever