Received-SPF: pass (google.com: best guess record for domain of linux-nfs-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
From:   Kevin Vasko <kvasko@gmail.com>
Mime-Version: 1.0 (1.0)
Date:   Thu, 26 Sep 2019 11:25:42 -0500
Subject: Re: NFSv4 client locks up on larger writes with Kerberos enabled
Message-Id: <3260B7EA-2B89-4EA2-BCFE-833C63EF0441@gmail.com>
Cc:     Chuck Lever <chuck.lever@oracle.com>,
        Trond Myklebust <trond.myklebust@hammerspace.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
To:     Bruce Fields <bfields@fieldses.org>
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk

Bruce,

Yes. It=E2=80=99s a Dell EMC Unity 300 storage appliance.=20

=46rom what I have gathered, under the hood it is running SLES12SP1

3.12.74-60.64.66.1.NEOKERNEL_SLES12SP1 #2 SMP Fri May 17 06;41:36 EDT 2019 x=
86_64

Unfortunately I only have limited access to the appliance through a =E2=80=9C=
service=E2=80=9D account. I do not have root privileges and I can only run c=
ertain commands under the service account.=20

I=E2=80=99ve got a ticket open with them and it=E2=80=99s been escalated to t=
heir =E2=80=9Cengineering=E2=80=9D department but they haven=E2=80=99t provi=
ded much insight as of yet.

If you give me some idea on what might help I=E2=80=99ll try to get more inf=
o if I can.


-Kevin

> On Sep 26, 2019, at 11:06 AM, Bruce Fields <bfields@fieldses.org> wrote:
>=20
> =EF=BB=BFOn Thu, Sep 26, 2019 at 08:55:17AM -0700, Chuck Lever wrote:
>>>> On Sep 25, 2019, at 1:07 PM, Bruce Fields <bfields@fieldses.org> wrote:=

>>>=20
>>> On Wed, Sep 25, 2019 at 11:49:14AM -0700, Chuck Lever wrote:
>>>> Sounds like the NFS server is dropping the connection. With
>>>> GSS enabled, that's usually a sign that the GSS window has
>>>> overflowed.
>>>=20
>>> Would that show up in the rpc statistics on the client somehow?
>>=20
>> More likely on the server. The client just sees a disconnect
>> without any explanation attached.
>=20
> So watching a count of disconnects might at least tell us something?
>=20
>> gss_verify_header is where the checking is done on the server.
>> Disappointingly, I see some dprintk's in there, but no static
>> trace events.
>=20
> Kevin, was this a Linux server?
>=20
> --b.
>=20
>>> In that case--I seem to remember there's a way to configure the size of
>>> the client's slot table, maybe lowering that (decreasing the number of
>>> rpc's allowed to be outstanding at a time) would work around the
>>> problem.
>>=20
>>> Should the client be doing something different to avoid or recover from
>>> overflows of the gss window?
>>=20
>> The client attempts to meter the request stream so that it stays
>> within the bounds of the GSS sequence number window. The stream
>> of requests is typically unordered coming out of the transmit
>> queue.
>>=20
>> There is some new code (since maybe v5.0?) that handles the
>> metering: gss_xmit_need_reencode().