Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Trond Myklebust <trondmy@hammerspace.com>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: SOFT + NO_RETRANS_TIMEOUT semantics
Thread-Topic: SOFT + NO_RETRANS_TIMEOUT semantics
Thread-Index: AQHXd0BYvyFlrd8CPk+sd3U4H8DdPKs/mk0AgAADYIA=
Date:   Mon, 12 Jul 2021 17:48:16 +0000
Message-ID: <FCDB802E-52FE-442E-A6D3-D623ED170EF6@oracle.com>
References: <981B8D74-2193-498C-8C4F-190E263FD8F6@oracle.com>
 <c902bfca197cac1c1328e833f768908ae518b829.camel@hammerspace.com>
In-Reply-To: <c902bfca197cac1c1328e833f768908ae518b829.camel@hammerspace.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?UkEHQcybV6rGmfgUflUvvQR+4h9uGpUWZBexhLjY9ahYFGGhr48raqXCF/8i?=
 =?us-ascii?Q?syeQ/U/QvUFD9AUaRMxgyW7oVBeLUWwmisaXymJs0U0sFGbvdgrWc6SAhdRx?=
 =?us-ascii?Q?HT6U0ISyzqE7FR80TmtEToHGyELPfuD24pk4Tlv3UJ8AfuN1ncVEtzes0AeO?=
 =?us-ascii?Q?1+33SJ/zpJsjVF2fXP1muVaxjyh32c4eV1kLveciwW5eIyMJVp05bVm50O5j?=
 =?us-ascii?Q?iFJQEKmjTCo277uFNXBy4JZNeZI6YHiETXwE/W4eR0t3dVjKI2ZIsURAdA/J?=
 =?us-ascii?Q?KjJUGBgz+YJMXNbHIof16PXP35UkJ69T4tAApHQgZQ5mtRiEw8rIGnKDT3rH?=
 =?us-ascii?Q?e1Ok2tWYOkxXby3wl4NT0WPoTWDUqBsGN7C20WJwsoYzSvHgxy99qGBfaBBe?=
 =?us-ascii?Q?KHMedkj0oX2ZzHKRvqjtoRM6lVHp1fSV7Rd2H2zfpI/JEoCVBnPTbvn+PuR3?=
 =?us-ascii?Q?errd/hjypY+/9AdG/do3WFkmerf/joj5aQDx4EaRdtUCOMIwYHzq4eW2n5ez?=
 =?us-ascii?Q?sfTOsGfvNSb+nBtymqw/N+f/6O8aOIcrOPoxhPI1H2/m2tQTOV6+lUnTlhhn?=
 =?us-ascii?Q?raasaW5cXx67FVR+pDBWk0PS7M5GWMxVQXwAdwbxvb4TTqJsT2Uf44+IlULY?=
 =?us-ascii?Q?co5P1oMn7VWXUnBoXKN0TI+s2YtTq5Tw/MFw0DWmqv1vhBxZze6KP4naVt5+?=
 =?us-ascii?Q?X/35gz9+2YwRFoJrJtOJWxTF8HJtND45VeSgiTff7K48RcMypbCCPNa/LDGm?=
 =?us-ascii?Q?jbZas8dz7h4iMIBTTF0VWzLXGuAWuuUShHKpeMIoyVgFIGJh5xxvzZzQI4wM?=
 =?us-ascii?Q?d99foO6XQ3srdSwa2qGZs/fDiFAcXKwt7C1H3l1ldakYN+a9Ed3EfPv757+1?=
 =?us-ascii?Q?RAh1RGXmhEeghePyC/mVSqMrr6rlJIGzwbGAbTF41DqPHVMGCnfO4tZDqDQD?=
 =?us-ascii?Q?e+ohWIerIFkxeCmjqjZJ1MJqvnx4ENRR06xlyjQGUoVs9mvqHDwTXY9JnaYm?=
 =?us-ascii?Q?EcO0Xm/XIWRb/b2gW1uiGzBCNw58bQ7RVJ7DupwLR4K0oo062fTVIHaCtJxY?=
 =?us-ascii?Q?/C1tbYH63cA+8PBW7mqiZG5++Q3X5rek1AiH5SZFg/XR9iolgqEAX5DBsnkc?=
 =?us-ascii?Q?blrCWRkG9oiaxeZXNycNRkJ0A1D3CYyWb7cAlssL8rCyYHuffxzbGphG66Hk?=
 =?us-ascii?Q?8aKd9Rnxxf1+eeOJy7p8K8ZDmfJy6dt2SyKQ8DfzLB4hXbBl6fiuSippFbwy?=
 =?us-ascii?Q?6uQ6t8Dn66rq+9Lr6ZYSlQveuyMIwisoQ2vKoEsk5s/+A/l3fyYCigk/5Y8H?=
 =?us-ascii?Q?O3+szAwd4ENvWzY1KLk04oEn?=
x-ms-exchange-transport-forked: True
Content-Type: text/plain; charset="us-ascii"
Content-ID: <98616F962FBB794DB6E8D1E4248EFE60@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4688.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 4336fcc9-a461-4761-1773-08d9455d3a74
X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Jul 2021 17:48:16.3721
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: i3pisz2YY8VEPRA7SKIr7NMl/D8YQBVLLGqsidwgZnjyIZD1fSRAob7uru1mtldD1eEBSpBFwvNEEkYP1Y2HXg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3048
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10043 signatures=668682
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999
 adultscore=0 malwarescore=0 bulkscore=0 mlxscore=0 suspectscore=0
 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2104190000 definitions=main-2107120128
X-Proofpoint-ORIG-GUID: FEP93w4VYHHoBnjJeIx6v_T9YmjOmYL-
X-Proofpoint-GUID: FEP93w4VYHHoBnjJeIx6v_T9YmjOmYL-
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jul 12, 2021, at 1:36 PM, Trond Myklebust <trondmy@hammerspace.com> wr=
ote:
>=20
> On Mon, 2021-07-12 at 17:07 +0000, Chuck Lever III wrote:
>> Hi Trond-
>>=20
>> I'm seeing some interesting client hangs that arise from a well-
>> timed server crash or network partition.
>>=20
>> The easiest to see is gss_destroy() on an Kerberized NFSv4 mount.
>>=20
>> NFSv4 asserts the RPC_TASK_NO_RETRANS_TIMEOUT flag (hereafter I'll
>> refer to it as NORTO) when creating a new rpc_clnt. The initial
>> rpc_ping() for that rpc_clnt is done before the logic that sets
>> cl_noretranstimeo, thus that ping works as expected (SOFT |
>> SOFTCONN) and can time out properly if the server isn't
>> responsive.
>>=20
>> However, once that ping succeeds, cl_noretranstimeo is asserted,
>> and all subsequent RPC requests on that rpc_clnt are with NORTO
>> semantics.
>>=20
>> When it comes time to destroy the GSS context for that rpc_clnt,
>> the NULL procedure with the GSS decorations is sent with SOFT |
>> SOFTCONN | NORTO. If the server isn't responding at that point,
>> the client continues to retransmit the GSS context destruction
>> request forever, and the xprt and possibly the nfs_client are
>> pinned.
>>=20
>> The problem also arises for lease management operations such as
>> singleton SEQUENCE or RENEW requests. These are also done with
>> SOFT, as I recall they need to time out properly. But with
>> NORTO + SOFT, they will be retried until a connection loss that
>> might never come.
>>=20
>> I've thought of some ways to modify the cl_noretranstimeo logic
>> such that it can be disabled for particular RPC tasks, though
>> none is really striking me as exceptionally clever:
>>=20
>>  - Add a field to struct rpc_procinfo that contains a mask of
>>    RPC_TASK flags to clear for each procedure.
>>  - Add logic to rpc_task_set_client() that clears NORTO in
>>    some special cases.
>>  - Reverse the meaning of NORTO (e.g., make it
>>    RPC_TASK_RETRANS_TIMEOUT) so that it can be set by a caller
>>    for particular RPC tasks if the rpc_clnt-default behavior
>>    is NORTO.
>>=20
>> Any thoughts?
>>=20
>=20
> Why would the connection not break when the server goes down?

The server can't actively RST or FIN the connection if a network
partition occurs; and some servers might crash while their kernel
is still alive to respond to keep-alive.


> Aren't
> the TCP_USER_TIMEOUT or the TCP_KEEPALIVE kicking in as they should?

I don't see them kicking in, but I let the test run only for about
12 minutes.=20


--
Chuck Lever