Received-SPF: pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
From:   Kevin Vasko <kvasko@gmail.com>
Mime-Version: 1.0 (1.0)
Subject: Re: NFSv4 client locks up on larger writes with Kerberos enabled
Date:   Mon, 30 Sep 2019 09:51:23 -0500
Message-Id: <3A4773DD-A4E4-47BD-957A-E24E4604E390@gmail.com>
References: <20190926195557.GC2849@fieldses.org>
Cc:     Chuck Lever <chuck.lever@oracle.com>,
        Trond Myklebust <trond.myklebust@hammerspace.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
In-Reply-To: <20190926195557.GC2849@fieldses.org>
To:     Bruce Fields <bfields@fieldses.org>
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk

What does the GSS window control? I mentioned it in another thread somewhere=
 but I found out that clients with a slower connection seem to not exhibit t=
his issue.

All of these clients are local to the NFS server (in same room, which is in K=
orea). I=E2=80=99ve got clients in the USA and they don=E2=80=99t seem to ex=
hibit this lockup behavior. I haven=E2=80=99t done extensive testing but we c=
an get 3-4MB/s across the ocean and as of yet I haven=E2=80=99t see a client=
 from the USA lock up like the ones local. It obviously takes a lot longer b=
ut haven=E2=80=99t seen it lock up on transferring several 5GB+ files.=20

Could the GSS window not be overflowing with the slower connection and we wo=
uldn=E2=80=99t see the issue?=20

-Kevin

> On Sep 26, 2019, at 2:55 PM, Bruce Fields <bfields@fieldses.org> wrote:
>=20
> =EF=BB=BFOn Thu, Sep 26, 2019 at 08:55:17AM -0700, Chuck Lever wrote:
>>>> On Sep 25, 2019, at 1:07 PM, Bruce Fields <bfields@fieldses.org> wrote:=

>>> In that case--I seem to remember there's a way to configure the size of
>>> the client's slot table, maybe lowering that (decreasing the number of
>>> rpc's allowed to be outstanding at a time) would work around the
>>> problem.
>>=20
>>> Should the client be doing something different to avoid or recover from
>>> overflows of the gss window?
>>=20
>> The client attempts to meter the request stream so that it stays
>> within the bounds of the GSS sequence number window. The stream
>> of requests is typically unordered coming out of the transmit
>> queue.
>>=20
>> There is some new code (since maybe v5.0?) that handles the
>> metering: gss_xmit_need_reencode().
>=20
> I guess I was thinking he could write a small number (say 2 digits) into
> /sys/module/sunrpc/parameters/tcp_max_slot_table_entries (before
> mounting, I guess?) and see if the problem's reproducable.
>=20
> If not, that's a little more evidence that it's the gss sequence window.
>=20
> (And might be an adequate workaround for now.)
>=20
> --b.