Received-SPF: pass (google.com: domain of linux-nfs+bounces-3538-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3;
Precedence: bulk
MIME-Version: 1.0
References: <CAPKjjnoeLkkvSZwpWw_bTCOCTfWYgjCyvjpoa95E4yroQR5zyA@mail.gmail.com>
 <CAPKjjnrez+hip+VBVrLT_g6Uzxd5DGSCoCSXQnRLx-qXT09yQA@mail.gmail.com>
 <CAPKjjnqiapiFkUNcpBm=bUT9OrQOwSFcw02N087at3JShp5EXw@mail.gmail.com> <CAPKjjno1LxnNd+6ttNsmjSp1HFZvsRKoh+6Z-FCrkrakHzNbew@mail.gmail.com>
In-Reply-To: <CAPKjjno1LxnNd+6ttNsmjSp1HFZvsRKoh+6Z-FCrkrakHzNbew@mail.gmail.com>
From: Zhitao Li <zhitao.li@smartx.com>
Date: Tue, 4 Jun 2024 12:47:32 +0800
Message-ID: <CAPKjjnpik3QZOH07xgzEZnJqKsxK_fCwRAz32QwJFpgcvGz1rg@mail.gmail.com>
Subject: Re: Question: How to customize retransmission timeout of
 unacknowledged NFS v3 TCP packet?
To: linux-nfs@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

I think I have found the reason why "tcp_retries2" doesn't work to
reduce the retransmission timeout for NFS client packets.

Now the xprt timeout of SUNRPC is hard-coded as
"xs_tcp_default_timeout". At the same time, the socket USER_TIMEOUT is
also configured based on this default variable, which is 3 minutes.
When a packet is retransmitted because of timeout, exponential backoff
is used. "tcp_retries2" is exported to change the default behaviour,
which retries 15 times, 200ms ~ 120s during each retry.

However, if USER_TIMEOUT is configured on the socket, "tcp_retries2"
is ignored for this socket. The packet is retried until USER_TIMEOUT
reaches.

The details are in
https://elixir.bootlin.com/linux/latest/source/net/ipv4/tcp_timer.c#L284
if someone is interested.

Up to now, I think there is no way to change the retransmission
timeout of NFS client packets.

If there is something misunderstood, please let me know.

Regards,
Zhitao Li.

On Tue, Jun 4, 2024 at 10:38=E2=80=AFAM Zhitao Li <zhitao.li@smartx.com> wr=
ote:
>
> Could anyone give some help about this issue? I've spent some days on
> this issue, both "tcp_retries2" and mount options like "timeo" and
> "retrans" do not work to give up retransmission earlier.
>
> Regards,
> Zhitao Li.
>
> On Fri, May 31, 2024 at 3:28=E2=80=AFPM Zhitao Li <zhitao.li@smartx.com> =
wrote:
> >
> > This problem is duplicated with
> > https://lore.kernel.org/linux-nfs/YQBPR01MB10724B629B69F7969AC6BDF9586C=
89@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM/
> >
> > According to the discussion, the patch is submitted to fix the timeout
> > used in xprt_socket, which is still on the way. On the other hand,
> > "tcp_retries2" doesn't work in control the transmission timeout of an
> > unacknowledged packet. Is there any workaround to change the
> > transmission timeout?
> >
> > Best regards,
> > Zhitao Li
> >
> > On Wed, May 29, 2024 at 6:18=E2=80=AFPM Zhitao Li <zhitao.li@smartx.com=
> wrote:
> > >
> > > Essentially, we need a mechanism to quickly reconnect with new
> > > nfs-server nodes for failover.
> > > I also tried to adjust mount options like "timeo" to 10s and "retrans=
"
> > > to 1,  and found that they don't work, either.  It seems that the NFS
> > > v3 client always tries to reconnect after some request hangs for 3
> > > minutes no matter what "timeo" and "retrans" is.
> > >
> > > On Wed, May 29, 2024 at 6:10=E2=80=AFPM Zhitao Li <zhitao.li@smartx.c=
om> wrote:
> > > >
> > > > Hi, dear community,
> > > >
> > > > In our NFS environment, NFS client mounts remote NFS export with it=
s
> > > > VIP. The VIP can be assigned to another server node for failover.
> > > > However, the NFS client sends the unacknowledged packet 50s+ after =
the
> > > > VIP is ready on the new node, which is because of the exponential
> > > > backoff retransmission algorithm.  I tried to set this parameter
> > > > "tcp_retries2" smaller so that the NFS client can reconnect with th=
e
> > > > new node more quickly, but this parameter didn't take effect. From
> > > > tcpdump entries as follows,
> > > >   1. At "2024-05-29 11:47:00",  ARP is updated.
> > > >   2. At "2024-05-29 11:47:52" ,  the NFS client retried to send the=
 packet.
> > > >   3. Then the connection is reset and a new connection starts.
> > > >
> > > > I guess the parameter just takes effect for applications and doesn'=
t
> > > > take effect for kernel modules like the NFS client. Could anyone gi=
ve
> > > > some advice to customize  retransmission timeout of unacknowledged =
NFS
> > > > v3 TCP packet?
> > > >
> > > >
> > > > OS: Linux kernel v6.7.0
> > > > NFS mount options:
> > > > vers=3D3,nolock,proto=3Dtcp,rsize=3D1048576,wsize=3D1048576,hard,ti=
meo=3D600,retrans=3D2,noresvport
> > > >
> > > > tcp_retries2:
> > > > [root@vm-play zhitaoli]# sysctl -w net.ipv4.tcp_retries2=3D5
> > > > net.ipv4.tcp_retries2 =3D 5
> > > > [root@vm-play zhitaoli]# cat /proc/sys/net/ipv4/tcp_retries2
> > > > 5
> > > >
> > > > tcpdump entries:
> > > >
> > > > 2024-05-29 11:46:02.331891 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> > > > ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> > > > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 235=
6,
> > > > options [nop,nop,TS val 1973659245 ecr 28456
> > > > 58566], length 124: NFS request xid 1954624602 120 access fh
> > > > Unknown/43000001180100000000000000DE40020000000000F4390000000000000=
00000
> > > > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTE=
ND|NFS_ACCESS_DELETE
> > > >
> > > > 2024-05-29 11:46:02.542836 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> > > > ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> > > > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 235=
6,
> > > > options [nop,nop,TS val 1973659456 ecr 28456
> > > > 58566], length 124: NFS request xid 1954624602 120 access fh
> > > > Unknown/43000001180100000000000000DE40020000000000F4390000000000000=
00000
> > > > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTE=
ND|NFS_ACCESS_DELETE
> > > >
> > > > 2024-05-29 11:46:02.751013 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> > > > ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> > > > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 235=
6,
> > > > options [nop,nop,TS val 1973659664 ecr 28456
> > > > 58566], length 124: NFS request xid 1954624602 120 access fh
> > > > Unknown/43000001180100000000000000DE40020000000000F4390000000000000=
00000
> > > > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTE=
ND|NFS_ACCESS_DELETE
> > > >
> > > > 2024-05-29 11:46:03.166958 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> > > > ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> > > > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 235=
6,
> > > > options [nop,nop,TS val 1973660080 ecr 28456
> > > > 58566], length 124: NFS request xid 1954624602 120 access fh
> > > > Unknown/43000001180100000000000000DE40020000000000F4390000000000000=
00000
> > > > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTE=
ND|NFS_ACCESS_DELETE
> > > >
> > > > 2024-05-29 11:46:04.046882 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> > > > ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> > > > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 235=
6,
> > > > options [nop,nop,TS val 1973660960 ecr 28456
> > > > 58566], length 124: NFS request xid 1954624602 120 access fh
> > > > Unknown/43000001180100000000000000DE40020000000000F4390000000000000=
00000
> > > > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTE=
ND|NFS_ACCESS_DELETE
> > > >
> > > > 2024-05-29 11:46:05.710910 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> > > > ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> > > > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 235=
6,
> > > > options [nop,nop,TS val 1973662624 ecr 28456
> > > > 58566], length 124: NFS request xid 1954624602 120 access fh
> > > > Unknown/43000001180100000000000000DE40020000000000F4390000000000000=
00000
> > > > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTE=
ND|NFS_ACCESS_DELETE
> > > >
> > > > 2024-05-29 11:46:09.039310 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> > > > ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> > > > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 235=
6,
> > > > options [nop,nop,TS val 1973665952 ecr 28456
> > > > 58566], length 124: NFS request xid 1954624602 120 access fh
> > > > Unknown/43000001180100000000000000DE40020000000000F4390000000000000=
00000
> > > > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTE=
ND|NFS_ACCESS_DELETE
> > > >
> > > > 2024-05-29 11:46:16.017889 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> > > > ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> > > > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 235=
6,
> > > > options [nop,nop,TS val 1973672930 ecr 28456
> > > > 58566], length 124: NFS request xid 1954624602 120 access fh
> > > > Unknown/43000001180100000000000000DE40020000000000F4390000000000000=
00000
> > > > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTE=
ND|NFS_ACCESS_DELETE
> > > >
> > > > 2024-05-29 11:46:29.326891 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> > > > ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> > > > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 235=
6,
> > > > options [nop,nop,TS val 1973686240 ecr 28456
> > > > 58566], length 124: NFS request xid 1954624602 120 access fh
> > > > Unknown/43000001180100000000000000DE40020000000000F4390000000000000=
00000
> > > > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTE=
ND|NFS_ACCESS_DELETE
> > > >
> > > > 2024-05-29 11:46:55.950915 52:54:00:1d:a4:24 > 52:54:00:a0:93:93,
> > > > ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> > > > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 235=
6,
> > > > options [nop,nop,TS val 1973712864 ecr 28456
> > > > 58566], length 124: NFS request xid 1954624602 120 access fh
> > > > Unknown/43000001180100000000000000DE40020000000000F4390000000000000=
00000
> > > > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTE=
ND|NFS_ACCESS_DELETE
> > > >
> > > > 2024-05-29 11:47:00.379844 52:54:00:13:1f:34 > Broadcast, ethertype
> > > > ARP (0x0806), length 60: Reply 10.125.1.85 is-at 52:54:00:13:1f:34,
> > > > length 46
> > > >
> > > > 2024-05-29 11:47:52.271192 52:54:00:1d:a4:24 > 52:54:00:13:1f:34,
> > > > ethertype IPv4 (0x0800), length 190: 10.125.1.214.58428 >
> > > > 10.125.1.85.nfs: Flags [P.], seq 129897:130021, ack 171633, win 235=
6,
> > > > options [nop,nop,TS val 1973769184 ecr 28456
> > > > 58566], length 124: NFS request xid 1954624602 120 access fh
> > > > Unknown/43000001180100000000000000DE40020000000000F4390000000000000=
00000
> > > > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ACCESS_EXTE=
ND|NFS_ACCESS_DELETE
> > > >
> > > > 2024-05-29 11:47:52.272041 52:54:00:13:1f:34 > 52:54:00:1d:a4:24,
> > > > ethertype IPv4 (0x0800), length 54: 10.125.1.85.nfs >
> > > > 10.125.1.214.58428: Flags [R], seq 1148562527, win 0, length 0
> > > >
> > > > 2024-05-29 11:47:52.272909 52:54:00:1d:a4:24 > 52:54:00:13:1f:34,
> > > > ethertype IPv4 (0x0800), length 74: 10.125.1.214.58428 >
> > > > 10.125.1.85.nfs: Flags [S], seq 1734997801, win 32120, options [mss
> > > > 1460,sackOK,TS val 1973769186 ecr 0,nop,wscale 7], length 0
> > > >
> > > > 2024-05-29 11:47:52.273503 52:54:00:13:1f:34 > 52:54:00:1d:a4:24,
> > > > ethertype IPv4 (0x0800), length 74: 10.125.1.85.nfs >
> > > > 10.125.1.214.58428: Flags [S.], seq 1078843840, ack 1734997802, win
> > > > 28960, options [mss 1460,sackOK,TS val 2235915769 ecr
> > > > 1973769186,nop,wscale 7], length 0
> > > >
> > > >
> > > > Best regards,
> > > > Zhitao Li