Received-SPF: pass (google.com: domain of linux-nfs+bounces-928-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Date: Thu, 4 Jan 2024 09:16:31 -0500
From: Chuck Lever <chuck.lever@oracle.com>
To: Benjamin Coddington <bcodding@redhat.com>
Cc: Jeff Layton <jlayton@kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: hangs during fstests testing with TLS
Message-ID: <ZZa9v3FQcwoZ7PCE@tissot.1015granger.net>
References: <117352d5dc94d8f31bc6770e4bbb93a357982a93.camel@kernel.org>
 <ABBA5E37-2E13-4603-A79C-F9B9B8488AE3@redhat.com>
 <D1091C94-9F23-47E9-A9CF-31CCEFE5EF8A@oracle.com>
 <AB421B16-F1FD-414C-A9BE-652D868F03A9@redhat.com>
 <DC58D77B-FC07-4B96-88A8-67F9ECDFD0CE@oracle.com>
 <8C3DFB5D-B967-4D59-BFC5-7B25315DB9AB@redhat.com>
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <8C3DFB5D-B967-4D59-BFC5-7B25315DB9AB@redhat.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?Sm1rbGFYSi8vZjcxclNvTng1b0sxeWtCUDBSMDhYZVBGbmlCd1doYTcwaDNC?=
 =?utf-8?B?UG90K2NVdmhBcVRrSk9DbWdzVWxscmlYck5sSjcyN3FqTXpWVDZUakNVYzRZ?=
 =?utf-8?B?RjNxTjFHTFJVK0xCZ3RvNllTQVM0bG43Z2VLVEdBVVRJSEZlaDU4amhhTEhN?=
 =?utf-8?B?UGtaazRLbFdqUm1GdkJmanl0NmxsVVRMUlBTblZzRitURjNHM2pRa1RFenU0?=
 =?utf-8?B?THZqZ2VuNmpFdlNSVExWbUxNSGRleEFVaDhaUjNycTRxaGJaWEx3RUVUR0N4?=
 =?utf-8?B?U2E0bkRIRmM2NWU1RHgrNGo5Wm9pQzRwTXVVM0ZTaUk5WkJVSGcyRnJZKzRa?=
 =?utf-8?B?dmNGb3E0ejFtb0Y0UWM2aVdiK1kwbDRnZ3ErOUFNcFR3VWErS3VuSEZCS1Vk?=
 =?utf-8?B?eUpGbzh6SG4rMDFlbWVDa25QOThoWS9XZG0rSmZsbTgyRFNhS01jekc0VjMy?=
 =?utf-8?B?WnpwcmQvMTdkSTYrUmxaeHRwVTdIaUVqSUkzM2llM1NpNi9FeHd6RzkxYkt5?=
 =?utf-8?B?K2J3NldiRldQZnpiQkFzNXV3M0FGcVBYS3dHV2V2Znc3d1BWNnpYbkR2OU16?=
 =?utf-8?B?OHFBWUR3VzVBYVR0MEZkY2syRkpmS3hPd2VFbjdPZGJxWHl6NmQrcjNIRUZD?=
 =?utf-8?B?cGxxbVQzMkxYUmoxV0tjbkRPbzIxY0swYmdEMkZQV0FiWS9ab0RvWTlCTWhm?=
 =?utf-8?B?U0RSVUJ2ZnlibndWMERsc1JvRGJiTmFBdHNBTUo3OWozbWZ2K2c2ZEVQRlNj?=
 =?utf-8?B?OE52TWpxRGtnaWFLZjBad2MrRXZ3ckJHOS9McHp1eGdTMTI2MnFNZ1BXMkZS?=
 =?utf-8?B?b2VCS24zWjRkbnQrRmtiYlNmRXMzOUZBd0l5MCtJQ3B4SkhualF0TVNFMWww?=
 =?utf-8?B?aEp6NW9KcGtvbjVNTkR4Y2VVZVNzeVREZ25RbVhVaWlSclJyMVlQUWRnRVJ0?=
 =?utf-8?B?M3UzcHpkOHpzOFBiVW9SOG5qVnFvcTJZQVlLTkVkVTJ6TkV0dG9lMjdvOFNz?=
 =?utf-8?B?N3hJNE1iNDdkOXo1dWxyd29nTjZDSTBoVHVyaldFQmRqaHVjRUpqNEowR1Z3?=
 =?utf-8?B?ampZeC9Ibm83V3FCZjBQcVYyUE9ZUGd6Zk1FdmN3OG1MYk0zaXdSVldVN0tL?=
 =?utf-8?B?ZVg0YzhISDgwM3BRYVhNY2N3ejhrcjE1WUsxNWQvVVpCcXlRU0VCYWNqTnNP?=
 =?utf-8?B?Z3BETTdhRk1lMDFNWSsrYlFYWHNtdUhvMVQ5UXpoMjJwNVpBMDhCNEVGbldx?=
 =?utf-8?B?MzEzRnNUaVhzRnFjY21ZMDJSNW1WMVVFeW1RN0pBL2Z0cE5WSHhIWWZmMlBG?=
 =?utf-8?B?NVVBandtbXBwV3RSb09RZWNwdnZzZis4djZCazl0WXY5WU1uL3R3aGxKL1pk?=
 =?utf-8?B?T05iYVR0MUFES2g0d2JPN3I3NXRCbWE1cjRobTJZRlB4dkc3NE5Va1lSaDNZ?=
 =?utf-8?B?bjVCZlhDZ0htQlA4SVpsVzVxajZFUjI1MFN5S2hSUVloZXpTa0JIcE5qOHc1?=
 =?utf-8?B?VjlRcWtCTHJQcmRreE5aNXlveDJ2ZzlIdDNNUS9PSjhjTFRHeDdnQW5vcktS?=
 =?utf-8?B?VXA5U2VZS2tyOHFUTE85TGtPbHc2emVNVXRIK2JONE80YTNtVkVjMy9HRk1x?=
 =?utf-8?B?VzdjSllsQlNrU1BxSTVaNFNwdE8wUjZpdkF2cFJmS0xyMUxaQ0FaSmxlUnI3?=
 =?utf-8?B?UWQrR1lCbnU3bno2dUdxSEZ1Ui9MV1NBOXdhUzg0blBlcjYzcHp2aHpsYUZM?=
 =?utf-8?B?N1FzeGV5ZmpFU3o5V1FJamhVMXpaWE1sVjVJSU83WU9ETmhmUmVjdGxUU2dU?=
 =?utf-8?B?Y2NsZHI4N0VtZ1hoNDlKV3k1Wm1FNUduYXV1V2xpUFhNR3UrdHNwK21YRW9E?=
 =?utf-8?B?bC9wQzE0bitNNXNWTzRWazBockl2T1NYVlljRXdINGlOSUwwblBqK0xiSmdt?=
 =?utf-8?B?NWZEY0JudXEvMlJtSXNKOGZCMUU2ajUvTzg4RFBoRlJoS2lYMlVFSTZ4cVVR?=
 =?utf-8?B?eTBXbzV5TVAxamV3K05VQVdPSGFPRjJDdTl1UHkvMGlQb0dMN096ZUR2YlJm?=
 =?utf-8?B?ZFJvaTZ2ZFFuRTlBZkszZkcwSVZDbkZHekVwYjE4RDZ5WHMreGtuU0xSQU84?=
 =?utf-8?B?NHBPZVBvd0JKRlhLdExnOUJsaFpEeE1FNW9XK3NhZnJ3NkpWdEFsS2sraXJt?=
 =?utf-8?B?V2c9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	CeWUyaamE+g/9g7hKq49VCq7G2PS/PhsHCjMqeywggiAb4gRnqyVKx0AIrOGs73H4hrNiMwpupdLd1qttxYAFSlaFje/BoiVl9cDjlkB8yoJt756OlP55P8k2K5Fgf29oPVJdy80U92zYQnrvMGc94W8jn038+MjsmhcjWcuYymt2V3S9d8hNMWlVeRZ/tmnj3x+69ckjItvQ7tw1BmC8TEu7BJwHKOfyHwYuCTNe4sZZhj6Q6+hT5suNPJbniI6uP1dvJG4LPbG/ZYGFIbl3KeS+jZpGfPchPBajPhYK/VgxiXCocWZLd+gWwOO2aeptcm6mTiX/tV3R5bBkhRVcZBau4obOv3NgsKV1K3cILJYjy8QaLDPbkeLINlCovyTeEuXhXJaCEB1OXDAqKJaQVkFnYhplRGgrjeijPU0NNQbOp5Xf2C0Pw6E96n3l1yWMSady0qefAJCz2AvzEeogXCjuESs9ba7jdnHMi8GpWH2vtXV4mfEnKV4ivvzaeypgWTmqhdgbcpRbyzUWvYrHiN2EHAQvZHDUPubk11fODAAdh88Ba4cyxqVGHIC9PSKi26WL5JJg2LVBAYdYVFkyMsdIV/nEfxiMVXiRbnX0Ps=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6d04932d-e93c-414c-a98b-08dc0d2fc123
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Jan 2024 14:16:33.6324
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: gYsrFT4fF727Y3LajW+CimVkXxByZhFXfqpU6HsPiDziyU6K8IaY19c39uYZfClmf5m/7PCnU53PhvNjDKEX+A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR10MB5682
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-01-04_08,2024-01-03_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0
 malwarescore=0 bulkscore=0 adultscore=0 mlxlogscore=999 mlxscore=0
 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311290000 definitions=main-2401040112
X-Proofpoint-GUID: lSGrYBok1oDT8pwktsq7fe3dIbRBAFiI
X-Proofpoint-ORIG-GUID: lSGrYBok1oDT8pwktsq7fe3dIbRBAFiI

On Thu, Jan 04, 2024 at 07:22:18AM -0500, Benjamin Coddington wrote:
> On 3 Jan 2024, at 16:46, Chuck Lever III wrote:
> 
> >> On Jan 3, 2024, at 3:16 PM, Benjamin Coddington <bcodding@redhat.com> wrote:
> >>
> >> On 3 Jan 2024, at 14:12, Chuck Lever III wrote:
> >>
> >>>> On Jan 3, 2024, at 1:47 PM, Benjamin Coddington <bcodding@redhat.com> wrote:
> >>>>
> >>>> This looks like it started out as the problem I've been sending patches to
> >>>> fix on 6.7, latest here:
> >>>> https://lore.kernel.org/linux-nfs/e28038fba1243f00b0dd66b7c5296a1e181645ea.1702496910.git.bcodding@redhat.com/
> >>>>
> >>>> .. however whenever I encounter the issue, the client reconnects the
> >>>> transport again - so I think there might be an additional problem here.
> >>>
> >>> I'm looking at the same problem as you, Ben. It doesn't seem to be
> >>> similar to what Jeff reports.
> >>>
> >>> But I'm wondering if gerry-rigging the timeouts is the right answer
> >>> for backchannel replies. The problem, fundamentally, is that when a
> >>> forechannel RPC task holds the transport lock, the backchannel's reply
> >>> transmit path thinks that means the transport connection is down and
> >>> triggers a transport disconnect.
> >>
> >> Why shouldn't backchannel replies have normal timeout values?
> >
> > RPC Replies are "send and forget". The server forechannel sends
> > its Replies without a timeout. There is no such thing as a
> > retransmitted RPC Reply (though a reliable transport might
> > retransmit portions of it, the RPC server itself is not aware of
> > that).
> >
> > And I don't see anything in the client's backchannel path that
> > makes me think there's a different protocol-level requirement
> > in the backchannel.
> 
> Its not strictly a protocol thing, the timeouts are used to decide what to
> do with a req or flag the transport state even if the request doesn't make
> it to the wire.  That's why the zero timeout values for this req improperly
> resets the transport.

I guess I'm harping on this a bit because forechannel v. backchannel
is already confusing enough. The use of timeouts for RPC Replies is
just heaping on to that confusion.

If we're going to keep an explicit timeout when sending the Reply,
it should have a little documentation. I suggest adding this to
xprt_init_bc_request() before the new call to
xprt_init_majortimeo():

	/*
	 * Backchannel Replies are sent with !RPC_TASK_SOFT and
	 * RPC_TASK_NO_RETRANS_TIMEOUT. The major timeout setting
	 * affects only how long each Reply waits to be sent when
	 * a transport connection cannot be established.
	 */
	xprt_init_majortimeo(task, req, ...

HTH

-- 
Chuck Lever