Received-SPF: pass (google.com: domain of linux-nfs+bounces-3189-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Date: Mon, 6 May 2024 20:27:52 -0400
From: Chuck Lever <chuck.lever@oracle.com>
To: Dai Ngo <dai.ngo@oracle.com>
Cc: cel@kernel.org, linux-nfs@vger.kernel.org
Subject: Re: [RFC PATCH] NFSD: Force all NFSv4.2 COPY requests to be
 synchronous
Message-ID: <Zjl1iNUJVBNMqlOd@tissot.1015granger.net>
References: <20240506210408.4760-1-cel@kernel.org>
 <35d2a207-7671-4f8a-be2b-3da03fedee50@oracle.com>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <35d2a207-7671-4f8a-be2b-3da03fedee50@oracle.com>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?ZBNoefatQU2NP87bWYI/l8LsOSzgXy2IeBpkI4MS5Xs6GxKE2ymnhBiH3G4c?=
 =?us-ascii?Q?v1DKsulXrlQcvivfuAKtSDVRYEy7eBg3L0Eyl4sbI5VbNTiMIc0JDkoa/Gfl?=
 =?us-ascii?Q?80MLJhkMXWfhOfxkSxFGxN/Utzr/+QVDvJbeDvQ0owfu21OfRDn+fRwGrTL7?=
 =?us-ascii?Q?LEdsIMVZmVxSnIx41Q++uYqgTZcjXihIXFVHEeoEiR1+sYsd6c2FwDL7iLjk?=
 =?us-ascii?Q?pFs8811mgnSvCzRcnijHZWaMEB+qGVFSZkop1282GDpO68BnSeI9Sj7KNtak?=
 =?us-ascii?Q?k5bpiR5An+qWDmnVpuG8N6kJGB6TXx32MFLJ/A6av30Rby297dpjn6raneji?=
 =?us-ascii?Q?2Qb1wEbOLGQWdfTyWNK/jLXAjJgTuS/Zh8Orec5/W9ks8B+3VgDXtwVFA5If?=
 =?us-ascii?Q?aTpiqSfpBjX2tcFnBHLUgAGNsG1hnxcW3TI5Oc9RcJJj2HlTUKeDt0ll99YF?=
 =?us-ascii?Q?bPSMtAxBsqpOKQJ4/gk7oplD4+wsu4CjpaMoPPPs+AVx6Id7bJI7EMzpWXKc?=
 =?us-ascii?Q?THMhwLgJYDWgyy3DpMnR07Bz75IyV/9JbUStOTY6B/cy2NevVNeNzzF8Kxv7?=
 =?us-ascii?Q?urMOMJrQL8XOepyeEtoZ7FyAtYGIvp4Dj5teQv3Mh/ClNZeiWUuGfV/qeT/c?=
 =?us-ascii?Q?//ngh5gyXIouInD1N0Q5cPeK3HBn9ZeG/QAYdn5A1qHavqoKiDciDxpPuJp8?=
 =?us-ascii?Q?u/V/VIL9Ss2ftAEb35IIRHB/qFf5GpYVSeug8H/dZ7JUdMy6sc/Cc4fTcVu4?=
 =?us-ascii?Q?H7Kh9ivtjb/qo3zT/0A6B0Jm+LKMa3+T/F1iokIFt3FAjtOW9OkEmrMR27SR?=
 =?us-ascii?Q?ZFiA4/l0FWf5h7/H5Raf7lKVD/tBGjC7t1OGCvUBtwRhM8HkFkfY/K7bl2Fu?=
 =?us-ascii?Q?pOiZoKh7PKITHsaW1I/NekOejPzdu6D4ImORjXd4KRiiB3UcaxNYZB7prfhU?=
 =?us-ascii?Q?ba9dAEhEYzSWBqhG+lRAVuYjwHA4XfqnvTYJYJJpVDgneEM4thGU5I2Rmvcy?=
 =?us-ascii?Q?QrhIVgjtAZM090bKoqoHA37K4HIJPHugOpMn1tqUgGK6jxrm1RBDM975Xq9f?=
 =?us-ascii?Q?MUe+OGH1DxLlOGA0fvQsmgvwS8OOxy/sJJxY/WI3Vr2zjftP3kZz1rOXAEkn?=
 =?us-ascii?Q?HaTfuUPShMBICI9N0Ozcse8ZIq5sV/LQH1H0vKAycXCl5JJFSkYFeI9Ho+9f?=
 =?us-ascii?Q?UGLLJAJoyfvdM2HDDXBmKJspbuKmgq4qpXcrQBFA6XFef+WvBAaMXuDK1OW5?=
 =?us-ascii?Q?tRiprbGACFgfkaV6Qy9c8FOyzNbTOoFl1sYeyzu2BIdZfhuZNc23ZuvOEbG5?=
 =?us-ascii?Q?wVdY8EQfCZoaMQo5dweE9XI80wK1Zh4dK0iBofaOqR4a0zAuzVEbp7koFEmd?=
 =?us-ascii?Q?Lfg8clSSjjnl76fRcElUhX2sYHrvCBLbW6W6ooC4zjvYt1aFEBx2va4zDOTV?=
 =?us-ascii?Q?HmpWCOgfyhEt8Bu7j+Vf+i2cBybgza0tapl+9k/ZIlfQ2leH8n90sBFDLJOl?=
 =?us-ascii?Q?gu2ajWrtP5gwfSknIC9Cqn86onRIUdjjOOZmhfc3ngP17gWPawrt82e2P6t+?=
 =?us-ascii?Q?IBHP4jl2HqKfDJfb6u2wYVBNhBLbLYdYy8914crMWHrOM4rj9vL7OhYSQE/x?=
 =?us-ascii?Q?Ng=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	11oRtOAyUlORqbxuP9oU1lMRtrv3q6XCA4XtrYTOp8r2imxV/tQt+V7OwpmCvofJamKFgs6jaflD9Dm5jlSGTIDuQ+rr1ZYY3O2aNwtKN/Rq1/cr5Bh2wlit+lr/enQCBTC9J5VwzIRg4REZp+RLqqtUYVub5znuWI6AiK4cQ91Fn5Kt+isZo3gp2JVg0qQq7NTQBJWEKSL2eE6/bsiqFfDL+ErDd0r0bF1NuaQ/0YC/Crrm161c1BEnP2FDgGkvA4swQ9gj2pXyolnPwSHkpOATfUchc/3Ygr/ntiJpsV1d7eEWCrbtZDHAe/7FyRZSSThmnS1bSV51YR5HwZZhGa2TrUn6Wd2ZBYbIaC9AXg2Qh/b4/qnaP2SFXoGJEP51ywvfe3GKIEMWLxdf8sTnmnxiIUPYO8c5r37V7KF493HR7bSSbPt/V+c3a4SzCXm+yUb4KDsepRie5LvX/2A6pOOdYsq+3YyHscdsEkORS7NFV/q4kI++Y6Ktf8Wxw88ZNx2NbqAm+D9KGvb103sPB4pmSFAqiyz3CIRDVYMDBrSKhLZySsQgHANE3z1MFgRPh23qcvUro+1F+pRfio3J8UGHAov8RrEtzmqXVxRNUyA=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 0d4c7e15-23cb-4bf1-72f2-08dc6e2c89c6
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 May 2024 00:27:54.9497
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: EtgT3TiFXjh+FEOK4Wy4yFFg5Ndaebj7kVV4xaUvwY/uBjPK7aYCQZw61F+FNnFxUqgZOWPyu8ulLMYMpj2bEw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR10MB6103
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.650,FMLib:17.11.176.26
 definitions=2024-05-06_17,2024-05-06_02,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0
 mlxlogscore=999 bulkscore=0 mlxscore=0 adultscore=0 malwarescore=0
 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2404010000 definitions=main-2405070001
X-Proofpoint-GUID: 63CgsoFNI7pN6RNxjSHLoORArSjHMZ1Q
X-Proofpoint-ORIG-GUID: 63CgsoFNI7pN6RNxjSHLoORArSjHMZ1Q

On Mon, May 06, 2024 at 04:37:15PM -0700, Dai Ngo wrote:
> 
> On 5/6/24 2:04 PM, cel@kernel.org wrote:
> > From: Chuck Lever <chuck.lever@oracle.com>
> > 
> > We've discovered that delivering a CB_OFFLOAD operation can be
> > unreliable in some pretty unremarkable situations,
> 
> Since the fore and back channel use the same connection so I assume
> this is not a connection related problem.

This is totally a connection-related problem. The underlying issue
is that NFSD does not retransmit backchannel requests when the
connection is lost, and the Linux NFS client does not implement
OFFLOAD_STATUS. Neither side right now recovers from connection
loss while a COPY operation is pending.


> Sounds like this is a bug that we should find and fix if possible
> instead of work around it.

I've been looking for a fix for the past several months. The last
fix I put in, you asked me to revert. So, I would prefer to fix
the root cause of this issue, but right now the best we can do is
create a surgical patch that can be backported to LTS kernels, and
keep working on a longer term fix.

It's either temporarily force all COPY operations to become
synchronous, or temporarily drop support for COPY in NFSD. Actually
the latter sounds safer.


> Do you know any scenarios where the CB_OFFLOAD operation is
> unreliable?

Any scenario where the connection is dropped (say, because the
server wants the client to retransmit forechannel requests, or
because of a GSS sequence number window under-run, or because of a
network partition, etc) can potentially result in the loss of a
backchannel operation.

I can reproduce this issue 100% of the time with an NFSv4.2 mount
from a 6.8.7-200.fc39.x86_64 NFS client, using the git regression
suite.


> > and the Linux
> > NFS client does not yet support sending an OFFLOAD_STATUS
> > operation to probe whether an asynchronous COPY operation has
> > finished. On Linux NFS clients, COPY can hang until manually
> > interrupted.
> > 
> > I've tried a couple of remedies, but so far the side-effects are
> > worse than the disease. For now, force COPY operations to be
> > synchronous so that the use of CB_OFFLOAD is avoided entirely.
> > 
> > I have some patches that add an OFFLOAD_STATUS implementation to the
> > Linux NFS client, but that is not likely to fix older clients.
> > 
> > Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
> > ---
> >   fs/nfsd/nfs4proc.c | 7 +++++++
> >   1 file changed, 7 insertions(+)
> > 
> > diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
> > index ea3cc3e870a7..12722c709cc6 100644
> > --- a/fs/nfsd/nfs4proc.c
> > +++ b/fs/nfsd/nfs4proc.c
> > @@ -1807,6 +1807,13 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
> >   	__be32 status;
> >   	struct nfsd4_copy *async_copy = NULL;
> > +	/*
> > +	 * Currently, async COPY is not reliable. Force all COPY
> > +	 * requests to be synchronous to avoid client application
> > +	 * hangs waiting for completion.
> > +	 */
> > +	nfsd4_copy_set_sync(copy, true);
> > +
> >   	copy->cp_clp = cstate->clp;
> >   	if (nfsd4_ssc_is_inter(copy)) {
> >   		trace_nfsd_copy_inter(copy);
> > 
> > base-commit: 939cb14d51a150e3c12ef7a8ce0ba04ce6131bd2

-- 
Chuck Lever