MIME-Version: 1.0
References: <20190220145650.21566-1-olga.kornievskaia@gmail.com>
 <1550837576.6456.3.camel@redhat.com> <a124c78ca5a5eefce34e8a84e92d87516345ee0c.camel@hammerspace.com>
 <CAN-5tyE5dm07V_8Qqse5qrLQVzrYXELM1EY2ySpCedBUBKYsLQ@mail.gmail.com>
 <05e439b3c419f3ac173feb770d3d2ae1d7500a2d.camel@hammerspace.com>
 <CAN-5tyFu4g40XJ7JzTSbidaEA30NeLQXFWD4t9pkRufesV-vJw@mail.gmail.com> <132998d93b8eb3d85c63a4176536b7b1403e0f88.camel@hammerspace.com>
In-Reply-To: <132998d93b8eb3d85c63a4176536b7b1403e0f88.camel@hammerspace.com>
From:   Olga Kornievskaia <olga.kornievskaia@gmail.com>
Date:   Fri, 22 Feb 2019 12:02:42 -0500
Message-ID: <CAN-5tyE3iWR1655w-ixRpo3v-_C2+t44F8W=ou7oByHGnn8+cg@mail.gmail.com>
Subject: Re: [PATCH 1/1] SUNRPC: fix handling of half-closed connection
To:     Trond Myklebust <trondmy@hammerspace.com>
Cc:     "anna.schumaker@netapp.com" <anna.schumaker@netapp.com>,
        "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>,
        "dwysocha@redhat.com" <dwysocha@redhat.com>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk

On Fri, Feb 22, 2019 at 10:50 AM Trond Myklebust
<trondmy@hammerspace.com> wrote:
>
> On Fri, 2019-02-22 at 10:11 -0500, Olga Kornievskaia wrote:
> > On Fri, Feb 22, 2019 at 10:06 AM Trond Myklebust
> > <trondmy@hammerspace.com> wrote:
> > > On Fri, 2019-02-22 at 09:46 -0500, Olga Kornievskaia wrote:
> > > > On Fri, Feb 22, 2019 at 8:45 AM Trond Myklebust <
> > > > trondmy@hammerspace.com> wrote:
> > > > > On Fri, 2019-02-22 at 07:12 -0500, Dave Wysochanski wrote:
> > > > > > Hi Olga,
> > > > > >
> > > > > > Do you have a reproducer for this?  A number of months ago I
> > > > > > did
> > > > > > a
> > > > > > significant amount of testing with half-closed connections,
> > > > > > after
> > > > > > we
> > > > > > had reports of connections stuck in FIN_WAIT2 in some older
> > > > > > kernels.
> > > > > > What I found was with kernels that had the tcp keepalives
> > > > > > (commit
> > > > > > 7f260e8575bf53b93b77978c1e39f8e67612759c), I could only
> > > > > > reproduce
> > > > > > a
> > > > > > hang of a few minutes, after which time the tcp keepalive
> > > > > > code
> > > > > > would
> > > > > > reset the connection.
> > > > > >
> > > > > > That said it was a while ago and something subtle may have
> > > > > > changed.
> > > > > > Also I'm not not sure if your header implies an indefinite
> > > > > > hang
> > > > > > or
> > > > > > just
> > > > > > a few minutes.
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > >
> > > > > > On Wed, 2019-02-20 at 09:56 -0500, Olga Kornievskaia wrote:
> > > > > > > From: Olga Kornievskaia <kolga@netapp.com>
> > > > > > >
> > > > > > > When server replies with an ACK to client's FIN/ACK, client
> > > > > > > ends
> > > > > > > up stuck in a TCP_FIN_WAIT2 state and client's mount hangs.
> > > > > > > Instead, make sure to close and reset client's socket and
> > > > > > > transport
> > > > > > > when transitioned into that state.
> > > >
> > > > Hi Trond,
> > > >
> > > > > So, please do note that we do not want to ignore the FIN_WAIT2
> > > > > state
> > > >
> > > > But we do ignore the FIN_WAIT2 state.
> > >
> > > We do not. We wait for the server to send a FIN, which is precisely
> > > the
> > > reason for which FIN_WAIT2 exists.
> > >
> > > > > because it implies that the server has not closed the socket on
> > > > > its
> > > > > side.
> > > >
> > > > That's correct.
> > > >
> > > > > That again means that we cannot re-establish a connection using
> > > > > the same source IP+port to the server, which is problematic for
> > > > > protocols such as NFSv3 which rely on standard duplicate reply
> > > > > cache
> > > > > for correct replay semantics.
> > > >
> > > > that's exactly what's happening that a client is unable to
> > > > establish
> > > > a
> > > > new connection to the server. With the patch, the client does an
> > > > RST
> > > > and it re-uses the port and all is well for NFSv3.
> > >
> > > RST is not guaranteed to be delivered to the recipient. That's why
> > > the
> > > TCP protocol defines FIN: it is a guaranteed to be delivered
> > > because it
> > > is ACKed.
> > >
> > > > > This is why we don't just set the TCP_LINGER2 socket option and
> > > > > call
> > > > > sock_release(). The choice to try to wait it out is deliberate
> > > > > because
> > > > > the alternative is that we end up with busy-waiting re-
> > > > > connection
> > > > > attempts.
> > > >
> > > > Why would it busy-wait? In my testing, RST happens and new
> > > > connection
> > > > is established?
> > >
> > > Only if the server has dropped the connection without notifying the
> > > client.
> >
> > Yes the server dropped the connection without notifying the client
> > (or
> > perhaps something in the middle did it as an attack). Again, I raise
> > this concern for the sake of dealing with this as an attack. I have
> > no
> > intentions of catering to broken servers. If this is not a possible
> > attack, then we don't have to deal with it.
>
> A man in the middle might be able to intercept the FIN from the server
> and ACK it, causing the connection to be closed on that server.
> However, as Dave pointed out, why wouldn't the keepalive mechanism then
> eventually kick in and close the socket on the client as well?

The mechanism is already kicked in and got stuck in FIN_WAIT2. NFS
connection was idle, so TCP layer was sending keep-alives. Then it
sent a FIN/ACK to which the server replied with just an ACK. Kernel
notified NFS that we are in FIN_WAIT2 and I believe it is NFS
responsibility to act accordingly. Kernel then keeps sending
"keep-alives" forever.  Because of this code:

        case TCP_FIN_WAIT1:
        case TCP_FIN_WAIT2:
                /* RFC 793 says to queue data in these states,
                 * RFC 1122 says we MUST send a reset.
                 * BSD 4.4 also does reset.
                 */
                if (sk->sk_shutdown & RCV_SHUTDOWN) {
                        if (TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(skb)->seq &&
                            after(TCP_SKB_CB(skb)->end_seq - th->fin,
tp->rcv_nxt)) {
                                NET_INC_STATS(sock_net(sk),
LINUX_MIB_TCPABORTONDATA);
                                tcp_reset(sk);   << this is never triggered
                                return 1;
                        }
                }

In our case TCP_SKB_CB(skb)->end_seq always equals
TCP_SKB_CB(skb)->seq. (No i don't know the meaning of end_seq and seq
:-/).

> If the FIN is not ACKed, then the server is supposed to keep
> retransmitting it. Until that ACK is received, it cannot close the
> socket without violating the TCP protocol.

Something in the middle can keep intercepting the the FIN/ACK from the
server and keep sending an ACK back?

>
> --
> Trond Myklebust
> Linux NFS client maintainer, Hammerspace
> trond.myklebust@hammerspace.com
>
>