Received-SPF: pass (google.com: domain of linux-nfs+bounces-3502-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Precedence: bulk
MIME-Version: 1.0
References: <CAPKjjnoeLkkvSZwpWw_bTCOCTfWYgjCyvjpoa95E4yroQR5zyA@mail.gmail.com>
 <CAPKjjnrez+hip+VBVrLT_g6Uzxd5DGSCoCSXQnRLx-qXT09yQA@mail.gmail.com>
In-Reply-To: <CAPKjjnrez+hip+VBVrLT_g6Uzxd5DGSCoCSXQnRLx-qXT09yQA@mail.gmail.com>
From: Zhitao Li <zhitao.li@smartx.com>
Date: Fri, 31 May 2024 15:28:49 +0800
Message-ID: <CAPKjjnqiapiFkUNcpBm=bUT9OrQOwSFcw02N087at3JShp5EXw@mail.gmail.com>
Subject: Re: Question: How to customize retransmission timeout of
 unacknowledged NFS v3 TCP packet?
To: Trond Myklebust <trond.myklebust@hammerspace.com>, Anna Schumaker <anna@kernel.org>
Cc: linux-nfs@vger.kernel.org, Ping Huang <huangping@smartx.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

This problem is duplicated with
https://lore.kernel.org/linux-nfs/YQBPR01MB10724B629B69F7969AC6BDF9586C89@Y=
QBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM/

According to the discussion, the patch is submitted to fix the timeout
used in xprt_socket, which is still on the way. On the other hand,
"tcp_retries2" doesn't work in control the transmission timeout of an
unacknowledged packet. Is there any workaround to change the
transmission timeout?

Best regards,
Zhitao Li

On Wed, May 29, 2024 at 6:18=E2=80=AFPM Zhitao Li <zhitao.li@smartx.com> wr=
ote:
>
> Essentially, we need a mechanism to quickly reconnect with new
> nfs-server nodes for failover.
> I also tried to adjust mount options like "timeo" to 10s and "retrans"
> to 1,  and found that they don't work, either.  It seems that the NFS
> v3 client always tries to reconnect after some request hangs for 3
> minutes no matter what "timeo" and "retrans" is.
>
> On Wed, May 29, 2024 at 6:10=E2=80=AFPM Zhitao Li <zhitao.li@smartx.com> =
wrote:
> >
> > Hi, dear community,
> >
> > In our NFS environment, NFS client mounts remote NFS export with its
> > VIP. The VIP can be assigned to another server node for failover.
> > However, the NFS client sends the unacknowledged packet 50s+ after the
> > VIP is ready on the new node, which is because of the exponential
> > backoff retransmission algorithm.  I tried to set this parameter
> > "tcp_retries2" smaller so that the NFS client can reconnect with the
> > new node more quickly, but this parameter didn't take effect. From
> > tcpdump entries as follows,
> >   1. At "2024-05-29 11:47:00",  ARP is updated.
> >   2. At "2024-05-29 11:47:52" ,  the NFS client retried to send the pac=
ket.
> >   3. Then the connection is reset and a new connection starts.
> >
> > I guess the parameter just takes effect for applications and doesn't
> > take effect for kernel modules like the NFS client. Could anyone give
> > some advice to customize  retransmission timeout of unacknowledged NFS
> > v3 TCP packet?
> >
> >
> > OS: Linux kernel v6.7.0
> > NFS mount options:
> > vers=3D3,nolock,proto=3Dtcp,rsize=3D1048576,wsize=3D1048576,hard,timeo=
=3D600,retrans=3D2,noresvport
> >
> > tcp_retries2:
> > [root@vm-play zhitaoli]# sysctl -w net.ipv4.tcp_retries2=3D5
> > net.ipv4.tcp_retries2 =3D 5
> > [root@vm-play zhitaoli]# cat /proc/sys/net/ipv4/tcp_retries2
> > 5
> >
> > tcpdump entries:
> >
> > 2024-05-29 11:46:02.331891 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> > ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356,
> > options [nop,nop,TS val 1973659245 ecr 28456
> > 58566], length 124: NFS request xid 1954624602 120 access fh
> > Unknown/43000001180100000000000000DE40020000000000F43900000000000000000=
0
> > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|N=
FS_ACCESS_DELETE
> >
> > 2024-05-29 11:46:02.542836 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> > ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356,
> > options [nop,nop,TS val 1973659456 ecr 28456
> > 58566], length 124: NFS request xid 1954624602 120 access fh
> > Unknown/43000001180100000000000000DE40020000000000F43900000000000000000=
0
> > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|N=
FS_ACCESS_DELETE
> >
> > 2024-05-29 11:46:02.751013 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> > ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356,
> > options [nop,nop,TS val 1973659664 ecr 28456
> > 58566], length 124: NFS request xid 1954624602 120 access fh
> > Unknown/43000001180100000000000000DE40020000000000F43900000000000000000=
0
> > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|N=
FS_ACCESS_DELETE
> >
> > 2024-05-29 11:46:03.166958 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> > ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356,
> > options [nop,nop,TS val 1973660080 ecr 28456
> > 58566], length 124: NFS request xid 1954624602 120 access fh
> > Unknown/43000001180100000000000000DE40020000000000F43900000000000000000=
0
> > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|N=
FS_ACCESS_DELETE
> >
> > 2024-05-29 11:46:04.046882 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> > ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356,
> > options [nop,nop,TS val 1973660960 ecr 28456
> > 58566], length 124: NFS request xid 1954624602 120 access fh
> > Unknown/43000001180100000000000000DE40020000000000F43900000000000000000=
0
> > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|N=
FS_ACCESS_DELETE
> >
> > 2024-05-29 11:46:05.710910 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> > ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356,
> > options [nop,nop,TS val 1973662624 ecr 28456
> > 58566], length 124: NFS request xid 1954624602 120 access fh
> > Unknown/43000001180100000000000000DE40020000000000F43900000000000000000=
0
> > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|N=
FS_ACCESS_DELETE
> >
> > 2024-05-29 11:46:09.039310 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> > ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356,
> > options [nop,nop,TS val 1973665952 ecr 28456
> > 58566], length 124: NFS request xid 1954624602 120 access fh
> > Unknown/43000001180100000000000000DE40020000000000F43900000000000000000=
0
> > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|N=
FS_ACCESS_DELETE
> >
> > 2024-05-29 11:46:16.017889 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> > ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356,
> > options [nop,nop,TS val 1973672930 ecr 28456
> > 58566], length 124: NFS request xid 1954624602 120 access fh
> > Unknown/43000001180100000000000000DE40020000000000F43900000000000000000=
0
> > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|N=
FS_ACCESS_DELETE
> >
> > 2024-05-29 11:46:29.326891 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> > ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356,
> > options [nop,nop,TS val 1973686240 ecr 28456
> > 58566], length 124: NFS request xid 1954624602 120 access fh
> > Unknown/43000001180100000000000000DE40020000000000F43900000000000000000=
0
> > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|N=
FS_ACCESS_DELETE
> >
> > 2024-05-29 11:46:55.950915 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> > ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356,
> > options [nop,nop,TS val 1973712864 ecr 28456
> > 58566], length 124: NFS request xid 1954624602 120 access fh
> > Unknown/43000001180100000000000000DE40020000000000F43900000000000000000=
0
> > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|N=
FS_ACCESS_DELETE
> >
> > 2024-05-29 11:47:00.379844 52:54:00:13:1f:34 > Broadcast, ethertype
> > ARP (0x0806), length 60: Reply 10.125.1.85 is-at 52:54:00:13:1f:34,
> > length 46
> >
> > 2024-05-29 11:47:52.271192 52:54:00:1d:a4:24 > 52:54:00:13:1f:34,
> > ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 2356,
> > options [nop,nop,TS val 1973769184 ecr 28456
> > 58566], length 124: NFS request xid 1954624602 120 access fh
> > Unknown/43000001180100000000000000DE40020000000000F43900000000000000000=
0
> > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTEND|N=
FS_ACCESS_DELETE
> >
> > 2024-05-29 11:47:52.272041 52:54:00:13:1f:34 > 52:54:00:1d:a4:24,
> > ethertype IPv4 (0x0800), length 54: 10.125.1.85.nfs >
> > 10.125.1.214.58428: Flags [R], seq 1148562527, win 0, length 0
> >
> > 2024-05-29 11:47:52.272909 52:54:00:1d:a4:24 > 52:54:00:13:1f:34,
> > ethertype IPv4 (0x0800), length 74: 10.125.1.214.58428 >
> > 10.125.1.85.nfs: Flags [S], seq 1734997801, win 32120, options [mss
> > 1460,sackOK,TS val 1973769186 ecr 0,nop,wscale 7], length 0
> >
> > 2024-05-29 11:47:52.273503 52:54:00:13:1f:34 > 52:54:00:1d:a4:24,
> > ethertype IPv4 (0x0800), length 74: 10.125.1.85.nfs >
> > 10.125.1.214.58428: Flags [S.], seq 1078843840, ack 1734997802, win
> > 28960, options [mss 1460,sackOK,TS val 2235915769 ecr
> > 1973769186,nop,wscale 7], length 0
> >
> >
> > Best regards,
> > Zhitao Li