From:   Andrew Klaassen <andrew.klaassen@boatrocker.com>
To:     "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Subject: RE: Trying to reduce NFSv4 timeouts to a few seconds on an
 established connection
Thread-Topic: Trying to reduce NFSv4 timeouts to a few seconds on an
 established connection
Thread-Index: AdkvR33g3mzLGSb7R/abGwkCKolj9QCUyz5A
Date:   Thu, 26 Jan 2023 15:31:35 +0000
Message-ID: <YQBPR01MB10724AEE306F99C844101EED086CF9@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
References: <YQBPR01MB10724B629B69F7969AC6BDF9586C89@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
In-Reply-To: <YQBPR01MB10724B629B69F7969AC6BDF9586C89@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?LxgLJJ3bGXT9s+GP06p1cxPZEiSjQZqVAsWqzhmpU/TN7C41i3ATgRBGQzvw?=
 =?us-ascii?Q?lXb4sL8AIWAiATow+VsieO6+wy8tkixdPdecpt0xuejo1n8e3xjUwSLqB+hA?=
 =?us-ascii?Q?GSu/T5x654HYV2Cjxig/wkvuWOdcZbMiAKe6auURqah8w8DUo/Qbj9ObmETB?=
 =?us-ascii?Q?DieOETTSmiIxLNNuREYlH+CzAUBKkxPt4axNnl8OIffu997y+0FPl2gq+ryI?=
 =?us-ascii?Q?GkV5UXgmyyx+GvcVPK93yw4aJUCXqXf9MTsCuW4K52NYoh3iZbsSJqF3hdmM?=
 =?us-ascii?Q?qOMg9G+QR1v/80Pv1SedbTOAMeKaZ9T8AKwWA2ifC3YkdI8aLQ8VZBL9X++g?=
 =?us-ascii?Q?R2g6xDDpyJDnPrYvZAHC1VXI21NuSUV0nPUnQpSy+sakcNoebojZjJ2Mf/kA?=
 =?us-ascii?Q?68DLo9wOcviIkWBV61l9xwg3dXsCQ2h55xmsnjENbUmXgRL8HTcAWmLDfo9P?=
 =?us-ascii?Q?eH4uRMtnvHbPsJX0ukAsczVtf9KPIn7frvWct8HX6s/r2WY0CPYVQ5ixGBsq?=
 =?us-ascii?Q?wiZOS2l32u2iJjBwngsur3oThAUzgZHL8a6cnMgHr/XQVmOsnPXIH5zS/02j?=
 =?us-ascii?Q?Pwh5+9nr3BcOQwF7lHwuEGKz7/pwolo7lcG8FtROjUeZG08xeCaLx/FVu3Nf?=
 =?us-ascii?Q?de5flYZkA2mMygLPguLZhI0js2XS2VxJAeRtzosTqcTKJJllSw7lDfNnOFOm?=
 =?us-ascii?Q?OXYEySzKo1BZjja9ETMny/M45vU6XCQvRa2QIsUjBmnMkoe0ENaLeNSySotF?=
 =?us-ascii?Q?5uIsdS5gI/BYNSV4npIWku+5LNfycA/VFHLg6pnpoSCTInX8KCu4zt+FuFwt?=
 =?us-ascii?Q?746kdsjKnvVv+MsTgat1Injg3rTVV8jL/bTWeszZntjL7h+ynoXNgZHfVz/q?=
 =?us-ascii?Q?LR6GOheJhtlgJcAMTjZ68M6bYtkEiMX9s5RTAINQg/Gcj7I1H2pESCtVu1Wj?=
 =?us-ascii?Q?O8/MUmwohnz1g2QK267EvtTQl7zVFQxzq8tFKXUvxCT6FRr6BuolQjef0SM5?=
 =?us-ascii?Q?LzTpvjFtX4r4pHxbS4KJy4NWSvmyBhahC0wcMToy13ITeoe+Vg5R1vXZVfN3?=
 =?us-ascii?Q?aEtwPWErAhSsxTagxIfIUMgQUdLep2O8vRa6M3FWM/2N4PP0dbsXZvsVnS18?=
 =?us-ascii?Q?VcMxbCs8Noo54EH9d7O9EdPRKdeGvT74M6OiREjwqpA+ZedRw/yRsf/2zeN5?=
 =?us-ascii?Q?U02ew7jkGtcwJDWRXCwVCnmGe6e7rzdaFkINsrFmtjPlhureX21w6XhNgVTh?=
 =?us-ascii?Q?VTYPiInWdFNKIeh7gvz5NWKIFbjc9Q37r92ImE12fJwI6mIYZNC/bMgv1tvt?=
 =?us-ascii?Q?sEr8dF4EfgA6ll+qL2RZKf9loKR9Q34rMrjOvGF0PqA3YhfWN+az/nT72ND2?=
 =?us-ascii?Q?DB8oW3AG/+WTfCd2N7LqMk2ku8t/BgsB8zxfb/StQMox+AWzUdnl/o0L1vs7?=
 =?us-ascii?Q?D8Omda7Lv5DZ//lgQYpCvK5bH/MG4qIqqGrnMS+Wa0VFl1ifoj9jNM5si+Xy?=
 =?us-ascii?Q?//Z+bu+MigEY7n+VBrSMvtKQz9GagA2oa8NkSX1LzqsEc0r7argsU+ZsqVqI?=
 =?us-ascii?Q?HfXvx8hTN1p6iy62IPAfRjguo/okOWWBFUpbfnn5b5SRGU+ig8FftLg/UpJW?=
 =?us-ascii?Q?HQ=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: boatrocker.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM
X-MS-Exchange-CrossTenant-Network-Message-Id: dc050d30-e1df-416c-f914-08daffb268d0
X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Jan 2023 15:31:35.3821
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: fd92a966-cd05-4664-965e-b69e7529781a
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: JO3MNtoX7s+hUPRKyUYnalCUG/NdgZv+mVpa/2bkitwXA/X4eZ9Nyxfh18smCcXFCOStTSCWwlH2+h6nUfKkS6i7efjQQFyBJLKo0LbAuSs=
X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR01MB6495
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

> From: Andrew Klaassen <andrew.klaassen@boatrocker.com>
> Sent: Monday, January 23, 2023 11:31 AM
>=20
> Hello,
>=20
> There's a specific NFSv4 mount on a specific machine which we'd like to
> timeout and return an error after a few seconds if the server goes away.
>=20
> I've confirmed the following on two different kernels, 4.18.0-
> 348.12.2.el8_5.x86_64 and 6.1.7-200.fc37.x86_64.
>=20
> I've been able to get both autofs and the mount command to cooperate, so
> that the mount attempt fails after an arbitrary number of seconds.  This
> mount command, for example, will fail after 6 seconds, as expected based =
on
> the timeo=3D20,retrans=3D2,retry=3D0 options:
>=20
> $ time sudo mount -t nfs4 -o
> rw,relatime,sync,vers=3D4.2,rsize=3D131072,wsize=3D131072,namlen=3D255,ac=
regmin
> =3D0,acregmax=3D0,acdirmin=3D0,acdirmax=3D0,soft,noac,proto=3Dtcp,timeo=
=3D20,retran
> s=3D2,retry=3D0,sec=3Dsys thor04:/mnt/thorfs04  /mnt/thor04
> mount.nfs4: Connection timed out
>=20
> real    0m6.084s
> user    0m0.007s
> sys     0m0.015s
>=20
> However, if the share is already mounted and the server goes away, the
> timeout is always 2 minutes plus the time I expect based on timeo and
> retrans.  In this case, 2 minutes and 6 seconds:
>=20
> $ time ls /mnt/thor04
> ls: cannot access '/mnt/thor04': Connection timed out
>=20
> real    2m6.025s
> user    0m0.003s
> sys     0m0.000s
>=20
> Watching the outgoing packets in the second case, the pattern is always t=
he
> same:
>  - 0.2 seconds between the first two, then doubling each time until the t=
wo
> minute mark is exceeded (so the last NFS packet, which is always the 11th
> packet, is sent around 1:45 after the first).
>  - Then some generic packets that start exactly-ish on the two minute mar=
k, 1
> second between the first two, then doubling each time.  (By this time the
> NFS command has given up.)
>=20
> 11:10:21.898305 IP 10.30.13.2.916 > 10.31.3.13.2049: Flags [P.], seq
> 14452:14652, ack 18561, win 501, options [nop,nop,TS val 834889483 ecr
> 1589769203], length 200: NFS request xid 3614904256 196 getattr fh 0,2/53
> 11:10:22.105189 IP 10.30.13.2.916 > 10.31.3.13.2049: Flags [P.], seq
> 14452:14652, ack 18561, win 501, options [nop,nop,TS val 834889690 ecr
> 1589769203], length 200: NFS request xid 3614904256 196 getattr fh 0,2/53
> 11:10:22.313290 IP 10.30.13.2.916 > 10.31.3.13.2049: Flags [P.], seq
> 14452:14652, ack 18561, win 501, options [nop,nop,TS val 834889898 ecr
> 1589769203], length 200: NFS request xid 3614904256 196 getattr fh 0,2/53
> 11:10:22.721269 IP 10.30.13.2.916 > 10.31.3.13.2049: Flags [P.], seq
> 14452:14652, ack 18561, win 501, options [nop,nop,TS val 834890306 ecr
> 1589769203], length 200: NFS request xid 3614904256 196 getattr fh 0,2/53
> 11:10:23.569192 IP 10.30.13.2.916 > 10.31.3.13.2049: Flags [P.], seq
> 14452:14652, ack 18561, win 501, options [nop,nop,TS val 834891154 ecr
> 1589769203], length 200: NFS request xid 3614904256 196 getattr fh 0,2/53
> 11:10:25.233212 IP 10.30.13.2.916 > 10.31.3.13.2049: Flags [P.], seq
> 14452:14652, ack 18561, win 501, options [nop,nop,TS val 834892818 ecr
> 1589769203], length 200: NFS request xid 3614904256 196 getattr fh 0,2/53
> 11:10:28.497282 IP 10.30.13.2.916 > 10.31.3.13.2049: Flags [P.], seq
> 14452:14652, ack 18561, win 501, options [nop,nop,TS val 834896082 ecr
> 1589769203], length 200: NFS request xid 3614904256 196 getattr fh 0,2/53
> 11:10:35.025219 IP 10.30.13.2.916 > 10.31.3.13.2049: Flags [P.], seq
> 14452:14652, ack 18561, win 501, options [nop,nop,TS val 834902610 ecr
> 1589769203], length 200: NFS request xid 3614904256 196 getattr fh 0,2/53
> 11:10:48.337201 IP 10.30.13.2.916 > 10.31.3.13.2049: Flags [P.], seq
> 14452:14652, ack 18561, win 501, options [nop,nop,TS val 834915922 ecr
> 1589769203], length 200: NFS request xid 3614904256 196 getattr fh 0,2/53
> 11:11:14.449303 IP 10.30.13.2.916 > 10.31.3.13.2049: Flags [P.], seq
> 14452:14652, ack 18561, win 501, options [nop,nop,TS val 834942034 ecr
> 1589769203], length 200: NFS request xid 3614904256 196 getattr fh 0,2/53
> 11:12:08.721251 IP 10.30.13.2.916 > 10.31.3.13.2049: Flags [P.], seq
> 14452:14652, ack 18561, win 501, options [nop,nop,TS val 834996306 ecr
> 1589769203], length 200: NFS request xid 3614904256 196 getattr fh 0,2/53
> 11:12:22.545394 IP 10.30.13.2.942 > 10.31.3.13.2049: Flags [S], seq 13752=
56951,
> win 64240, options [mss 1460,sackOK,TS val 835010130 ecr 0,nop,wscale 7],
> length 0
> 11:12:23.570199 IP 10.30.13.2.942 > 10.31.3.13.2049: Flags [S], seq 13752=
56951,
> win 64240, options [mss 1460,sackOK,TS val 835011155 ecr 0,nop,wscale 7],
> length 0
> 11:12:25.617284 IP 10.30.13.2.942 > 10.31.3.13.2049: Flags [S], seq 13752=
56951,
> win 64240, options [mss 1460,sackOK,TS val 835013202 ecr 0,nop,wscale 7],
> length 0
> 11:12:29.649219 IP 10.30.13.2.942 > 10.31.3.13.2049: Flags [S], seq 13752=
56951,
> win 64240, options [mss 1460,sackOK,TS val 835017234 ecr 0,nop,wscale 7],
> length 0
> 11:12:37.905274 IP 10.30.13.2.942 > 10.31.3.13.2049: Flags [S], seq 13752=
56951,
> win 64240, options [mss 1460,sackOK,TS val 835025490 ecr 0,nop,wscale 7],
> length 0
> 11:12:54.289212 IP 10.30.13.2.942 > 10.31.3.13.2049: Flags [S], seq 13752=
56951,
> win 64240, options [mss 1460,sackOK,TS val 835041874 ecr 0,nop,wscale 7],
> length 0
> 11:13:26.545304 IP 10.30.13.2.942 > 10.31.3.13.2049: Flags [S], seq 13752=
56951,
> win 64240, options [mss 1460,sackOK,TS val 835074130 ecr 0,nop,wscale 7],
> length 0
>=20
> I tried changing tcp_retries2 as suggested in another thread from this li=
st:
>=20
> # echo 3 > /proc/sys/net/ipv4/tcp_retries2
>=20
> ...but it made no difference on either kernel.  The 2 minute timeout also
> doesn't seem to match with what I'd calculate from the initial value of
> tcp_retries2, which should give a much higher timeout.
>=20
> The only clue I've been able to find is in the retry=3Dn entry in the NFS
> manpage:
>=20
> " For TCP the default is 3 minutes, but system TCP connection timeouts wi=
ll
> sometimes limit the timeout of each retransmission to around 2 minutes."
>=20
> What I'm not able to make sense of:
>  - The retry option says that it applies to mount operations, not read/wr=
ite
> operations.  However, in this case I'm seeing the 2 minute delay on
> read/write operations but *not* mount operations.
>  - A couple of hours of searching didn't lead me to any kernel settings t=
hat
> would result in a 2 minute timeout.
>=20
> Does anyone have any clues about a) what's happening and b) how to get
> our desired behaviour of being able to control both mount and read/write
> timeouts down to a few seconds?
>=20
> Thanks.

I thought that changing TCP_RTO_MAX in include/net/tcp.h from 120 to someth=
ing smaller and recompiling the kernel would change the 2 minute timeout, b=
ut it had no effect.  I'm going to keep poking through the kernel code to s=
ee if there's a knob I can turn to change the 2 minute timeout, so that I c=
an at least understand where it's coming from.

Any hints as to where I should be looking?

Andrew