From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Andrew Klaassen <andrew.klaassen@boatrocker.com>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: Trying to reduce NFSv4 timeouts to a few seconds on an
 established connection
Thread-Topic: Trying to reduce NFSv4 timeouts to a few seconds on an
 established connection
Thread-Index: AdkvR33g3mzLGSb7R/abGwkCKolj9QAASdUA
Date:   Mon, 23 Jan 2023 16:35:04 +0000
Message-ID: <A00E49F5-9BD7-498F-95A9-FA2D5572694F@oracle.com>
References: <YQBPR01MB10724B629B69F7969AC6BDF9586C89@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
In-Reply-To: <YQBPR01MB10724B629B69F7969AC6BDF9586C89@YQBPR01MB10724.CANPRD01.PROD.OUTLOOK.COM>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?CpVAzVxUSLAcbI/rZ4XUWvgmgvvsFQn4vumir8jvlKlOEZXjY2VQPENwC50j?=
 =?us-ascii?Q?3O1uz2PXZo1K2+Lx+TMOxKVYb8GQhTVxEbjq2pnhkBvs0nYzsB4WXk+n6lEg?=
 =?us-ascii?Q?8QVGd1sZrlC2/CzooeG669adBT+UlUV1OONCU79q+Fs1HxCCoUOB5Re5Q3ZE?=
 =?us-ascii?Q?3Ry3sIjTf8kwRLr4JtqUw7nzHLMilPfKRmlv427epvcMe9VVrwgZ4mKurDks?=
 =?us-ascii?Q?JajjmAILy0CivffqKsP+THYiU5XPNfkgLSIkSOLK/M0B1M7FIOumbjZnyhJk?=
 =?us-ascii?Q?Riw+2diJFx4yZGXRCKW+zkRRLRxZmlYRI8osaY8WTX5DzrLiQ4REi3AwKf0E?=
 =?us-ascii?Q?ibawhpz9xmpPXDX73gFOccAGUvhf//wYKF3xd494zxDToNqTkV8IiCfpf92a?=
 =?us-ascii?Q?S3R/PDFza7Jp2jFngxh8MYInLl5Zle80wajvPI1JKNhuAgz/xx9+qYkkTWXH?=
 =?us-ascii?Q?LhoS2oW6/mjavVbPHAZgr4RdyhrlI/y0It423bs2gG9L7jdyQcOw3ZS8FL2k?=
 =?us-ascii?Q?GTh2wLgeCQrLXSWfNXTZmdO10sjz3athcd8MT/+2Omx/efA3qcHI7Tj+RkqX?=
 =?us-ascii?Q?jxdz51Md3RAZQHI5pgWBP6qH1/WkG8ujjCX2NB4HQnd8lgrKAi8IUefJHzKJ?=
 =?us-ascii?Q?tf8d0b9D05UFGFMaKOkQ2jw22xItxrrImXONWpd+1NAT630oSPUATNS7PuJo?=
 =?us-ascii?Q?HLYUtO4cPtlyeO6fyqSy6jzeE+ukaVbB+Yu8hA1S2qL1u/Pe0mnNoTB29Rt0?=
 =?us-ascii?Q?bZJnhiJriJdb5vU5tnTZ789ONpqEI6gQbDkG72yVBH9qMQailIiQ+funNjlG?=
 =?us-ascii?Q?KRBeVYTrhptRdkcbTb2y1VjWX+WD00sEg/sXn3flsM+PpN61k1PjQdIjga9M?=
 =?us-ascii?Q?mHXFcVvetUmMhrhwbPyod/vOOY/ZCBL5ID6IcVZPbpQ1+vcxkViZk7qJXLp5?=
 =?us-ascii?Q?VPZWGyE6pbCYwfYsfcYQwpr4OlL9gzSUHDVFScdgUC5knC30kLNBx8+XhkY4?=
 =?us-ascii?Q?Sxek5E0LkkqlAhJpe9UjlzcTIQH1hWCkXVlDd5i/jllZGa7XBylpFL9UjaqH?=
 =?us-ascii?Q?xpMhDwKTzTmbnGVpm4BbIEGv+rvBqaWZlrEDjF1sLwInRImTNZYYm3AZK9Ey?=
 =?us-ascii?Q?BeGjpR3Pqu0CuQgJNqAtzcPWRIB2an5qOOKrsK0A+ivekAL9QeLMf3KJqISz?=
 =?us-ascii?Q?/TKx+TeZZGAVo3v0YU6DXNMDyGGpC9v73txsg1ISWDWAVbWY2U+c4/Uf3e5w?=
 =?us-ascii?Q?lshDf8LhBskdjtwUFSr7ft0C0yLXDX4Fodk5twiWvoo3hGVEUxvAMJWZGSu+?=
 =?us-ascii?Q?Lsfp9a4O2UyApDv1noYJVwKY/TEE9wXXQcBUFcLNVTIJJJIlRtyw/zTomopg?=
 =?us-ascii?Q?txQir205H2ElRUizVPhsh4jtlD7sZzS8Cw+yzCtI3pza7infhrJ8fXZjltAN?=
 =?us-ascii?Q?6ND6X/C4UIno4LM0s+dTRas7ThCFXSM4VWqNk4/pNA2RURV5+KJ/+zryhPr1?=
 =?us-ascii?Q?PX9OvwVRQmQUP/Hm2nb8LhCZjuwxz+mV8+TiETrgGMyqhIsbxgJakBpiVbfF?=
 =?us-ascii?Q?YzfuxsKSKBI6OhhexUoHHdNAC1+2qjy4gvojwRwT/1ZPTzhceQT8uza+2kov?=
 =?us-ascii?Q?Tw=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <9A93EEB2F29A964482E26630C9776F9B@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 7btfe++QSqMIUJdAKaa+xiSte4l2CH1Gy7DJ65c8rqBRobnH6PgUrTaMxiCKtxvL8hzqXogIBXZNWvDYU+XQwBT1EQbUvyt3lyhM/0xRZgpEsejel6bL7Ih4eu9x4QdLWDIoOLcz37SJFhpGX0zyouWfI19wC9Fsx+WRH72nExv6XINPX9rojFCq/lr4A0Iw0CaqGoUdOXe8LQV+YQPEZjP1uc0G7kT1rHqr7oNuKRSvu+IkJv885jYKYC8vrN3q0bw48LrYbWR2WKWJ6UJRe2Ozd0HqxfqPFh5nf820AhMns1DpmsigKV86mnKrKo+pjI7vHtymOLduU67U8rKYAw/xF950zuGHjG90Ou/2SlNGGusj0RfTVhmsYjARGWx9Ikye3ze4FDLt2JDJjlJyZf8gWiBcCnK9zvYSbHNKSTy1IEly65BolapMAFVkwwCYGTgG5qqS7i6ikm5G5I/GDKBIAJmIm8UTt87mW4HBMffDw6f3eW2f6Ye8k7sNhGTTWjtVp5lN8nHi5ryH0lXzbRysYSwpEiHcRuCxtrOE5JOHTlNSeXzyhAEL2DBd7MGU1QDoc2vP2/QXVWwdGIuquAgc5kMxhewIAXqQg1yGaKsXXGtUmOtUDhvHd2vm9v7pzlrK/YcigKeRsvrvkDhq25/RlNRdqLirY1PdWJqruGN6OWAvVAXeoJR4O6OEp0H4ZPvC3l+duKi5ml0l8RboNndvBnHGLTX2lQOPMc4WgIoRwmVMtwKtwTNMAdqc42LiZxSTyt6X/9GAVegt7RJes2mt2XUvUXRZdSbXWpTNgNI=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 1e9643ef-b890-4c3a-039d-08dafd5fc83b
X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Jan 2023 16:35:04.8752
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: OaQPiW1BOEaE/ltnB9HmZ7RrNMcBwe+ioIIHZ7VZiFsnVq0fASJrLNVC9Kwe+PXtxyHtJSFRVsmyamhq8hyP4w==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR10MB6171
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.122.1
 definitions=2023-01-23_11,2023-01-23_01,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 phishscore=0
 mlxlogscore=999 adultscore=0 mlxscore=0 suspectscore=0 spamscore=0
 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2212070000 definitions=main-2301230159
X-Proofpoint-GUID: KlrteOlHADpAv7hQn4gtIa6ANiKQsIGq
X-Proofpoint-ORIG-GUID: KlrteOlHADpAv7hQn4gtIa6ANiKQsIGq
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jan 23, 2023, at 11:31 AM, Andrew Klaassen <andrew.klaassen@boatrocker=
.com> wrote:
>=20
> Hello,
>=20
> There's a specific NFSv4 mount on a specific machine which we'd like to t=
imeout and return an error after a few seconds if the server goes away.
>=20
> I've confirmed the following on two different kernels, 4.18.0-348.12.2.el=
8_5.x86_64 and 6.1.7-200.fc37.x86_64.
>=20
> I've been able to get both autofs and the mount command to cooperate, so =
that the mount attempt fails after an arbitrary number of seconds.  This mo=
unt command, for example, will fail after 6 seconds, as expected based on t=
he timeo=3D20,retrans=3D2,retry=3D0 options:
>=20
> $ time sudo mount -t nfs4 -o rw,relatime,sync,vers=3D4.2,rsize=3D131072,w=
size=3D131072,namlen=3D255,acregmin=3D0,acregmax=3D0,acdirmin=3D0,acdirmax=
=3D0,soft,noac,proto=3Dtcp,timeo=3D20,retrans=3D2,retry=3D0,sec=3Dsys thor0=
4:/mnt/thorfs04  /mnt/thor04
> mount.nfs4: Connection timed out
>=20
> real    0m6.084s
> user    0m0.007s
> sys     0m0.015s
>=20
> However, if the share is already mounted and the server goes away, the ti=
meout is always 2 minutes plus the time I expect based on timeo and retrans=
.  In this case, 2 minutes and 6 seconds:
>=20
> $ time ls /mnt/thor04
> ls: cannot access '/mnt/thor04': Connection timed out
>=20
> real    2m6.025s
> user    0m0.003s
> sys     0m0.000s
>=20
> Watching the outgoing packets in the second case, the pattern is always t=
he same:
> - 0.2 seconds between the first two, then doubling each time until the tw=
o minute mark is exceeded (so the last NFS packet, which is always the 11th=
 packet, is sent around 1:45 after the first).
> - Then some generic packets that start exactly-ish on the two minute mark=
, 1 second between the first two, then doubling each time.  (By this time t=
he NFS command has given up.)
>=20
> 11:10:21.898305 IP 10.30.13.2.916 > 10.31.3.13.2049: Flags [P.], seq 1445=
2:14652, ack 18561, win 501, options [nop,nop,TS val 834889483 ecr 15897692=
03], length 200: NFS request xid 3614904256 196 getattr fh 0,2/53
> 11:10:22.105189 IP 10.30.13.2.916 > 10.31.3.13.2049: Flags [P.], seq 1445=
2:14652, ack 18561, win 501, options [nop,nop,TS val 834889690 ecr 15897692=
03], length 200: NFS request xid 3614904256 196 getattr fh 0,2/53
> 11:10:22.313290 IP 10.30.13.2.916 > 10.31.3.13.2049: Flags [P.], seq 1445=
2:14652, ack 18561, win 501, options [nop,nop,TS val 834889898 ecr 15897692=
03], length 200: NFS request xid 3614904256 196 getattr fh 0,2/53
> 11:10:22.721269 IP 10.30.13.2.916 > 10.31.3.13.2049: Flags [P.], seq 1445=
2:14652, ack 18561, win 501, options [nop,nop,TS val 834890306 ecr 15897692=
03], length 200: NFS request xid 3614904256 196 getattr fh 0,2/53
> 11:10:23.569192 IP 10.30.13.2.916 > 10.31.3.13.2049: Flags [P.], seq 1445=
2:14652, ack 18561, win 501, options [nop,nop,TS val 834891154 ecr 15897692=
03], length 200: NFS request xid 3614904256 196 getattr fh 0,2/53
> 11:10:25.233212 IP 10.30.13.2.916 > 10.31.3.13.2049: Flags [P.], seq 1445=
2:14652, ack 18561, win 501, options [nop,nop,TS val 834892818 ecr 15897692=
03], length 200: NFS request xid 3614904256 196 getattr fh 0,2/53
> 11:10:28.497282 IP 10.30.13.2.916 > 10.31.3.13.2049: Flags [P.], seq 1445=
2:14652, ack 18561, win 501, options [nop,nop,TS val 834896082 ecr 15897692=
03], length 200: NFS request xid 3614904256 196 getattr fh 0,2/53
> 11:10:35.025219 IP 10.30.13.2.916 > 10.31.3.13.2049: Flags [P.], seq 1445=
2:14652, ack 18561, win 501, options [nop,nop,TS val 834902610 ecr 15897692=
03], length 200: NFS request xid 3614904256 196 getattr fh 0,2/53
> 11:10:48.337201 IP 10.30.13.2.916 > 10.31.3.13.2049: Flags [P.], seq 1445=
2:14652, ack 18561, win 501, options [nop,nop,TS val 834915922 ecr 15897692=
03], length 200: NFS request xid 3614904256 196 getattr fh 0,2/53
> 11:11:14.449303 IP 10.30.13.2.916 > 10.31.3.13.2049: Flags [P.], seq 1445=
2:14652, ack 18561, win 501, options [nop,nop,TS val 834942034 ecr 15897692=
03], length 200: NFS request xid 3614904256 196 getattr fh 0,2/53
> 11:12:08.721251 IP 10.30.13.2.916 > 10.31.3.13.2049: Flags [P.], seq 1445=
2:14652, ack 18561, win 501, options [nop,nop,TS val 834996306 ecr 15897692=
03], length 200: NFS request xid 3614904256 196 getattr fh 0,2/53
> 11:12:22.545394 IP 10.30.13.2.942 > 10.31.3.13.2049: Flags [S], seq 13752=
56951, win 64240, options [mss 1460,sackOK,TS val 835010130 ecr 0,nop,wscal=
e 7], length 0
> 11:12:23.570199 IP 10.30.13.2.942 > 10.31.3.13.2049: Flags [S], seq 13752=
56951, win 64240, options [mss 1460,sackOK,TS val 835011155 ecr 0,nop,wscal=
e 7], length 0
> 11:12:25.617284 IP 10.30.13.2.942 > 10.31.3.13.2049: Flags [S], seq 13752=
56951, win 64240, options [mss 1460,sackOK,TS val 835013202 ecr 0,nop,wscal=
e 7], length 0
> 11:12:29.649219 IP 10.30.13.2.942 > 10.31.3.13.2049: Flags [S], seq 13752=
56951, win 64240, options [mss 1460,sackOK,TS val 835017234 ecr 0,nop,wscal=
e 7], length 0
> 11:12:37.905274 IP 10.30.13.2.942 > 10.31.3.13.2049: Flags [S], seq 13752=
56951, win 64240, options [mss 1460,sackOK,TS val 835025490 ecr 0,nop,wscal=
e 7], length 0
> 11:12:54.289212 IP 10.30.13.2.942 > 10.31.3.13.2049: Flags [S], seq 13752=
56951, win 64240, options [mss 1460,sackOK,TS val 835041874 ecr 0,nop,wscal=
e 7], length 0
> 11:13:26.545304 IP 10.30.13.2.942 > 10.31.3.13.2049: Flags [S], seq 13752=
56951, win 64240, options [mss 1460,sackOK,TS val 835074130 ecr 0,nop,wscal=
e 7], length 0
>=20
> I tried changing tcp_retries2 as suggested in another thread from this li=
st:
>=20
> # echo 3 > /proc/sys/net/ipv4/tcp_retries2
>=20
> ...but it made no difference on either kernel.  The 2 minute timeout also=
 doesn't seem to match with what I'd calculate from the initial value of tc=
p_retries2, which should give a much higher timeout.
>=20
> The only clue I've been able to find is in the retry=3Dn entry in the NFS=
 manpage:
>=20
> " For TCP the default is 3 minutes, but system TCP connection timeouts wi=
ll sometimes limit the timeout of each retransmission to around 2 minutes."
>=20
> What I'm not able to make sense of:
> - The retry option says that it applies to mount operations, not read/wri=
te operations.  However, in this case I'm seeing the 2 minute delay on read=
/write operations but *not* mount operations.
> - A couple of hours of searching didn't lead me to any kernel settings th=
at would result in a 2 minute timeout.
>=20
> Does anyone have any clues about a) what's happening and b) how to get ou=
r desired behaviour of being able to control both mount and read/write time=
outs down to a few seconds?

If the server is already mounted on that client at another mount point,
then the client will share the transport amongst mounts of the same server.

The first mount's options take precedent, and subsequent mounts re-use
that mount's transport and the mount options that control it.


--
Chuck Lever