Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Trond Myklebust <trondmy@hammerspace.com>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: SOFT + NO_RETRANS_TIMEOUT semantics
Thread-Topic: SOFT + NO_RETRANS_TIMEOUT semantics
Thread-Index: AQHXd0BYAPlUDv7myUCZIzGi/bl7LQ==
Date:   Mon, 12 Jul 2021 17:07:07 +0000
Message-ID: <981B8D74-2193-498C-8C4F-190E263FD8F6@oracle.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?CWg3CeNQLTMe5zetxuJllGP96iXDyCGPlmoAe8oUiksUy0JVdN7JrFBZ4cIN?=
 =?us-ascii?Q?GULelJtslb/M2dfIyh1hPtIVF5MjFWrPMIWZytOiNKpe9/LDbfZ5aviNARsK?=
 =?us-ascii?Q?JHmJeXxXl/GftQYYxDROf5HcD4K+A/jWg73S3oPSB5T0sNh/CZHye+WLjFps?=
 =?us-ascii?Q?pW2ynBNi4hIuFY4GzlJBgf/BTKm4wejsA854kTBCNnV7Qq1b+oNSpTd2ym7v?=
 =?us-ascii?Q?IiqKCqWQtAW3BMok5hUOyunX/7F9GYvx/Y5y+b4/8ioL+/MgapYeaM2ePXVO?=
 =?us-ascii?Q?FO8DWdub4fTTt+8NxGXgFeroN3geOfhQy2qwmziTXT1qEMUwgEyYzcATWoxw?=
 =?us-ascii?Q?mo8PhFi2wNK+MGQFwSDNt8VVmdWsYOJCP82SZTh3XNeZu34QaVlyxdff6GC5?=
 =?us-ascii?Q?PrVDCOCuzDxhCL/jx6wkp2LQupnkoofsxSp5TTh/nYMlbfLTCc+KvovyQPko?=
 =?us-ascii?Q?oC6mOH2cgknZz8NSnJ8wCwJwMIYTmAUJueOGlqnwCxlx3xDv1igN2aEc/eKe?=
 =?us-ascii?Q?UJpJFaIANEcPODTX8bmPGwUexg1uwGGtuuvtz1UBEU0B7SwJ4h7dQCwN870B?=
 =?us-ascii?Q?6CkQZDvwMYfgqRaTX3hDJqR5qzdmrbJFADxTsqBZCsvLEvnV7xxmBMonJ3kc?=
 =?us-ascii?Q?wsO+kpbRNuXFUXYYa/xGDoT2bj9C6as1HvxnfZDaLhbOyy5q48tK9VhIGRg5?=
 =?us-ascii?Q?bsYiCY9mpQ6dZVntKG5E3+XtOWI258W+zeZI+H88r/q3PDUEVzC/cGQoqou/?=
 =?us-ascii?Q?gjeVe1r2tWGdY1AjusWb0Jl8xfOjuiAQNWDNCJXCndgFamlfKgxTItwPmIsN?=
 =?us-ascii?Q?Oszegkou82eS6IjzTSTiZJZtSZFh2Pz/OepA6BGv2mX6JNAtPqCvipdqBbSK?=
 =?us-ascii?Q?hptyEJdq99Mh70XPD6yITTJs5enhb90sdrL5m1UIJiIT0IzGcVo9jUbFe2Ga?=
 =?us-ascii?Q?VJQ7ddRhvQKrkDQzX5zzjxEPSLwF+hhjN8i3mPYTNsxA/6iUiPfuYJCxc0y6?=
 =?us-ascii?Q?Jh/LXtnPcSub5yQ6qaiK5yHN62X4Efr4mhdBMpy4w6BmyvKkj36MsL3B/9/E?=
 =?us-ascii?Q?4nVPOAdpZZfJp8PxPmsrhhyEMx4bZYY4Hmu30tNqWXm7eK6wqOvdRN8Q0xEj?=
 =?us-ascii?Q?DFOd4jNhFnlRN8PPgR4BF/Hu5FO1ZRDK4CbfWe3EZ14qx2cSgxVinTMJpk8R?=
 =?us-ascii?Q?OsVQvDmYhhVykY+C2K4H3YdDmTEGkD33zp9CbzPgC/sXciPOmXH3gXXRM2Pr?=
 =?us-ascii?Q?6YrtFZJ1VlaLd5RBkLpUpy47msNTT6vqEybX/svJQmpkA8Wqp77qHqnFvJGk?=
 =?us-ascii?Q?TdQh8vQmRYeWt0HdWBcGacT9?=
x-ms-exchange-transport-forked: True
Content-Type: text/plain; charset="us-ascii"
Content-ID: <C3E65C69248A9A4DB20ACEFABDDE6B12@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4688.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 34931bf2-921f-43a5-548f-08d945577ae9
X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Jul 2021 17:07:07.5113
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: WCtvgOozK/+wv6d0efXlHi67GsP2PkW58UI+sRBrtFBjVXwLUSsa3+6lGNXGBrow5zmJGvkZ/fMlnFM3Mc82Uw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB3860
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10043 signatures=668682
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0 malwarescore=0
 spamscore=0 suspectscore=0 adultscore=0 mlxlogscore=999 mlxscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000
 definitions=main-2107120126
X-Proofpoint-ORIG-GUID: 3uxKkT3pyXB00nzRECc40YfwarjOgPGR
X-Proofpoint-GUID: 3uxKkT3pyXB00nzRECc40YfwarjOgPGR
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

Hi Trond-

I'm seeing some interesting client hangs that arise from a well-
timed server crash or network partition.

The easiest to see is gss_destroy() on an Kerberized NFSv4 mount.

NFSv4 asserts the RPC_TASK_NO_RETRANS_TIMEOUT flag (hereafter I'll
refer to it as NORTO) when creating a new rpc_clnt. The initial
rpc_ping() for that rpc_clnt is done before the logic that sets
cl_noretranstimeo, thus that ping works as expected (SOFT |
SOFTCONN) and can time out properly if the server isn't
responsive.

However, once that ping succeeds, cl_noretranstimeo is asserted,
and all subsequent RPC requests on that rpc_clnt are with NORTO
semantics.

When it comes time to destroy the GSS context for that rpc_clnt,
the NULL procedure with the GSS decorations is sent with SOFT |
SOFTCONN | NORTO. If the server isn't responding at that point,
the client continues to retransmit the GSS context destruction
request forever, and the xprt and possibly the nfs_client are
pinned.

The problem also arises for lease management operations such as
singleton SEQUENCE or RENEW requests. These are also done with
SOFT, as I recall they need to time out properly. But with
NORTO + SOFT, they will be retried until a connection loss that
might never come.

I've thought of some ways to modify the cl_noretranstimeo logic
such that it can be disabled for particular RPC tasks, though
none is really striking me as exceptionally clever:

 - Add a field to struct rpc_procinfo that contains a mask of
   RPC_TASK flags to clear for each procedure.
 - Add logic to rpc_task_set_client() that clears NORTO in
   some special cases.
 - Reverse the meaning of NORTO (e.g., make it
   RPC_TASK_RETRANS_TIMEOUT) so that it can be set by a caller
   for particular RPC tasks if the rpc_clnt-default behavior
   is NORTO.

Any thoughts?

--
Chuck Lever