Hi Trond-
I'm seeing some interesting client hangs that arise from a well-
timed server crash or network partition.
The easiest to see is gss_destroy() on an Kerberized NFSv4 mount.
NFSv4 asserts the RPC_TASK_NO_RETRANS_TIMEOUT flag (hereafter I'll
refer to it as NORTO) when creating a new rpc_clnt. The initial
rpc_ping() for that rpc_clnt is done before the logic that sets
cl_noretranstimeo, thus that ping works as expected (SOFT |
SOFTCONN) and can time out properly if the server isn't
responsive.
However, once that ping succeeds, cl_noretranstimeo is asserted,
and all subsequent RPC requests on that rpc_clnt are with NORTO
semantics.
When it comes time to destroy the GSS context for that rpc_clnt,
the NULL procedure with the GSS decorations is sent with SOFT |
SOFTCONN | NORTO. If the server isn't responding at that point,
the client continues to retransmit the GSS context destruction
request forever, and the xprt and possibly the nfs_client are
pinned.
The problem also arises for lease management operations such as
singleton SEQUENCE or RENEW requests. These are also done with
SOFT, as I recall they need to time out properly. But with
NORTO + SOFT, they will be retried until a connection loss that
might never come.
I've thought of some ways to modify the cl_noretranstimeo logic
such that it can be disabled for particular RPC tasks, though
none is really striking me as exceptionally clever:
- Add a field to struct rpc_procinfo that contains a mask of
RPC_TASK flags to clear for each procedure.
- Add logic to rpc_task_set_client() that clears NORTO in
some special cases.
- Reverse the meaning of NORTO (e.g., make it
RPC_TASK_RETRANS_TIMEOUT) so that it can be set by a caller
for particular RPC tasks if the rpc_clnt-default behavior
is NORTO.
Any thoughts?
--
Chuck Lever
On Mon, 2021-07-12 at 17:07 +0000, Chuck Lever III wrote:
> Hi Trond-
>
> I'm seeing some interesting client hangs that arise from a well-
> timed server crash or network partition.
>
> The easiest to see is gss_destroy() on an Kerberized NFSv4 mount.
>
> NFSv4 asserts the RPC_TASK_NO_RETRANS_TIMEOUT flag (hereafter I'll
> refer to it as NORTO) when creating a new rpc_clnt. The initial
> rpc_ping() for that rpc_clnt is done before the logic that sets
> cl_noretranstimeo, thus that ping works as expected (SOFT |
> SOFTCONN) and can time out properly if the server isn't
> responsive.
>
> However, once that ping succeeds, cl_noretranstimeo is asserted,
> and all subsequent RPC requests on that rpc_clnt are with NORTO
> semantics.
>
> When it comes time to destroy the GSS context for that rpc_clnt,
> the NULL procedure with the GSS decorations is sent with SOFT |
> SOFTCONN | NORTO. If the server isn't responding at that point,
> the client continues to retransmit the GSS context destruction
> request forever, and the xprt and possibly the nfs_client are
> pinned.
>
> The problem also arises for lease management operations such as
> singleton SEQUENCE or RENEW requests. These are also done with
> SOFT, as I recall they need to time out properly. But with
> NORTO + SOFT, they will be retried until a connection loss that
> might never come.
>
> I've thought of some ways to modify the cl_noretranstimeo logic
> such that it can be disabled for particular RPC tasks, though
> none is really striking me as exceptionally clever:
>
> - Add a field to struct rpc_procinfo that contains a mask of
> RPC_TASK flags to clear for each procedure.
> - Add logic to rpc_task_set_client() that clears NORTO in
> some special cases.
> - Reverse the meaning of NORTO (e.g., make it
> RPC_TASK_RETRANS_TIMEOUT) so that it can be set by a caller
> for particular RPC tasks if the rpc_clnt-default behavior
> is NORTO.
>
> Any thoughts?
>
Why would the connection not break when the server goes down? Aren't
the TCP_USER_TIMEOUT or the TCP_KEEPALIVE kicking in as they should?
Is this an RDMA problem?
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
> On Jul 12, 2021, at 1:36 PM, Trond Myklebust <[email protected]> wrote:
>
> On Mon, 2021-07-12 at 17:07 +0000, Chuck Lever III wrote:
>> Hi Trond-
>>
>> I'm seeing some interesting client hangs that arise from a well-
>> timed server crash or network partition.
>>
>> The easiest to see is gss_destroy() on an Kerberized NFSv4 mount.
>>
>> NFSv4 asserts the RPC_TASK_NO_RETRANS_TIMEOUT flag (hereafter I'll
>> refer to it as NORTO) when creating a new rpc_clnt. The initial
>> rpc_ping() for that rpc_clnt is done before the logic that sets
>> cl_noretranstimeo, thus that ping works as expected (SOFT |
>> SOFTCONN) and can time out properly if the server isn't
>> responsive.
>>
>> However, once that ping succeeds, cl_noretranstimeo is asserted,
>> and all subsequent RPC requests on that rpc_clnt are with NORTO
>> semantics.
>>
>> When it comes time to destroy the GSS context for that rpc_clnt,
>> the NULL procedure with the GSS decorations is sent with SOFT |
>> SOFTCONN | NORTO. If the server isn't responding at that point,
>> the client continues to retransmit the GSS context destruction
>> request forever, and the xprt and possibly the nfs_client are
>> pinned.
>>
>> The problem also arises for lease management operations such as
>> singleton SEQUENCE or RENEW requests. These are also done with
>> SOFT, as I recall they need to time out properly. But with
>> NORTO + SOFT, they will be retried until a connection loss that
>> might never come.
>>
>> I've thought of some ways to modify the cl_noretranstimeo logic
>> such that it can be disabled for particular RPC tasks, though
>> none is really striking me as exceptionally clever:
>>
>> - Add a field to struct rpc_procinfo that contains a mask of
>> RPC_TASK flags to clear for each procedure.
>> - Add logic to rpc_task_set_client() that clears NORTO in
>> some special cases.
>> - Reverse the meaning of NORTO (e.g., make it
>> RPC_TASK_RETRANS_TIMEOUT) so that it can be set by a caller
>> for particular RPC tasks if the rpc_clnt-default behavior
>> is NORTO.
>>
>> Any thoughts?
>>
>
> Why would the connection not break when the server goes down?
The server can't actively RST or FIN the connection if a network
partition occurs; and some servers might crash while their kernel
is still alive to respond to keep-alive.
> Aren't
> the TCP_USER_TIMEOUT or the TCP_KEEPALIVE kicking in as they should?
I don't see them kicking in, but I let the test run only for about
12 minutes.
--
Chuck Lever
On Mon, 2021-07-12 at 17:48 +0000, Chuck Lever III wrote:
>
>
> > On Jul 12, 2021, at 1:36 PM, Trond Myklebust
> > <[email protected]> wrote:
> >
> > On Mon, 2021-07-12 at 17:07 +0000, Chuck Lever III wrote:
> > > Hi Trond-
> > >
> > > I'm seeing some interesting client hangs that arise from a well-
> > > timed server crash or network partition.
> > >
> > > The easiest to see is gss_destroy() on an Kerberized NFSv4 mount.
> > >
> > > NFSv4 asserts the RPC_TASK_NO_RETRANS_TIMEOUT flag (hereafter
> > > I'll
> > > refer to it as NORTO) when creating a new rpc_clnt. The initial
> > > rpc_ping() for that rpc_clnt is done before the logic that sets
> > > cl_noretranstimeo, thus that ping works as expected (SOFT |
> > > SOFTCONN) and can time out properly if the server isn't
> > > responsive.
> > >
> > > However, once that ping succeeds, cl_noretranstimeo is asserted,
> > > and all subsequent RPC requests on that rpc_clnt are with NORTO
> > > semantics.
> > >
> > > When it comes time to destroy the GSS context for that rpc_clnt,
> > > the NULL procedure with the GSS decorations is sent with SOFT |
> > > SOFTCONN | NORTO. If the server isn't responding at that point,
> > > the client continues to retransmit the GSS context destruction
> > > request forever, and the xprt and possibly the nfs_client are
> > > pinned.
> > >
> > > The problem also arises for lease management operations such as
> > > singleton SEQUENCE or RENEW requests. These are also done with
> > > SOFT, as I recall they need to time out properly. But with
> > > NORTO + SOFT, they will be retried until a connection loss that
> > > might never come.
> > >
> > > I've thought of some ways to modify the cl_noretranstimeo logic
> > > such that it can be disabled for particular RPC tasks, though
> > > none is really striking me as exceptionally clever:
> > >
> > > - Add a field to struct rpc_procinfo that contains a mask of
> > > RPC_TASK flags to clear for each procedure.
> > > - Add logic to rpc_task_set_client() that clears NORTO in
> > > some special cases.
> > > - Reverse the meaning of NORTO (e.g., make it
> > > RPC_TASK_RETRANS_TIMEOUT) so that it can be set by a caller
> > > for particular RPC tasks if the rpc_clnt-default behavior
> > > is NORTO.
> > >
> > > Any thoughts?
> > >
> >
> > Why would the connection not break when the server goes down?
>
> The server can't actively RST or FIN the connection if a network
> partition occurs; and some servers might crash while their kernel
> is still alive to respond to keep-alive.
>
>
> > Aren't
> > the TCP_USER_TIMEOUT or the TCP_KEEPALIVE kicking in as they
> > should?
>
> I don't see them kicking in, but I let the test run only for about
> 12 minutes.
>
TCP_USER_TIMEOUT should kick in any time when the server is failing to
read the socket contents, and should close the connection.
A more likely scenario is that the server is actually reading the
socket, but is just dropping the requests on the floor. I agree that
needs to be handled correctly.
One way to do that could be to add a flag that says "don't apply any
other default task flags" for special cases like this one?
--
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]