Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Dai Ngo <dai.ngo@oracle.com>
CC:     Trond Myklebust <trondmy@hammerspace.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: NFSv4.0 Linux client fails to return delegation
Thread-Topic: NFSv4.0 Linux client fails to return delegation
Thread-Index: AQHZos/yj80TeHCL8EaHc5sJIN1lPa+SgHAAgAFRTACAABzmAA==
Date:   Tue, 20 Jun 2023 17:10:46 +0000
Message-ID: <9B7E8AD9-C418-4E22-8EEE-B101F1D2F662@oracle.com>
References: <f9651599-846e-3331-9353-8a8264de1a27@oracle.com>
 <d5efe21b5f6a4bc7edd0f8ed441f63aa76b7e41a.camel@hammerspace.com>
 <5a43e57a-eb00-6f76-87a8-1a80c584b9f8@oracle.com>
In-Reply-To: <5a43e57a-eb00-6f76-87a8-1a80c584b9f8@oracle.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?mqjRw/g+jH6Zha3DCj1m/b3/5Ns9EM5Bxd1vhxS+88fvU5yBgiiFMdNLS28C?=
 =?us-ascii?Q?hAnhkVP1MTCHopg9G1qe1eNItCLZPoUtlOcAZm4jDF4lqc9rF7YmZBSbpr5b?=
 =?us-ascii?Q?tfcwHTkFMccZzf9Q5hESEqMVggl+QVFLqVUQuKREBSeLdlKbNPnGiazov9D6?=
 =?us-ascii?Q?jIHrHDf+/9871Tig1zl+3BXm/sv7R+CuQI2iRpdeK3e69i3d7wooDpcuPq8G?=
 =?us-ascii?Q?M5AHrM+rUDv1JRE4Ihz2gO3NRwX075+0rYP8fb7vrV9VGRB2zU9uigEpOaQ6?=
 =?us-ascii?Q?sibNeoWu+/qsYKjTvu30Hk7orUcJd/MD4sKetAoR6dnzRG4C0bwHj8MjyR+n?=
 =?us-ascii?Q?+l+CNutxvERfFqVh2fRjyksaqiOEBtSq8MJTXvhxwT5Cs/y6as4kyInQol+s?=
 =?us-ascii?Q?EF1LDuEyzuCm+n3po/l1S39U+kvTHmoFTKrqWt1sC55UeCPxn1SeyLlzobdS?=
 =?us-ascii?Q?fKM0b3cRxpeCjmggGWSi4xs3UOLIemKe9Kh21LrvWbVqXxvICq1+Z2KvVrl0?=
 =?us-ascii?Q?Fy6Nc4KduosSYniOczVFHt6W28YmY1VRRjcs4tNnbAaKXWTtrsVj6U4xatVj?=
 =?us-ascii?Q?ZiYfo3McuG2iaxkDqDKc+GbE+4u2T8aA/xNjqfKNBzPv2N2iSBvOoND4o0in?=
 =?us-ascii?Q?JDSkN+qGc+VWlUfH3wc0sP3X0i4p6lHNW2ffsKgHppPrwGXZB3MJh2VCAgXK?=
 =?us-ascii?Q?iOMabzPRZqd+PRse/LN1sXFnwPVlQl44KYkwAkWVik0imI+u8RX9IzPkfN4p?=
 =?us-ascii?Q?g3v7R5UOSjZAtfbgYZ6bQsVUdLknMSnvl2Ji7+lJsZcSx5BjbKAZD2Skq/7G?=
 =?us-ascii?Q?713qkp6S8EDlFAAg9OqG1rxHqZCkWhdPGnm1OSrAPmI8Tx/3xJQZRxfM/Ok5?=
 =?us-ascii?Q?xbeGlRrYwZA3Qjyu7hIE5WCw09WObKTnp2oP1C3eviWjE/HlRM59nI1ig2TR?=
 =?us-ascii?Q?M6hSmskfAyJAnhyUOLjK5BTMEZM9Ak3MXOlGRqKspv3Q9M+AuTTcgeQ/xxYA?=
 =?us-ascii?Q?y/ALn++zniFNp2AyWXS0oA5oBZl7a/kAtcsxIe67h3kWhi4xTQbSMRevCA15?=
 =?us-ascii?Q?OpZ53y8EKC/yx+Hmku7OZ09Hj6z4CckPMH1mMSRrRObG5f1Qxg2711Z83wLG?=
 =?us-ascii?Q?phzgl+GRF4XsL3PYa8sVn2nzPOWWVtpRg+vuAFoLFqixVanUHL+ZZbekvMGP?=
 =?us-ascii?Q?6doOHa3q+m/AhMqPSVDVFAX1Sn4eVQ2ogpJRJxGImms4hz+cYHsPMExYKAzo?=
 =?us-ascii?Q?m0tpJv0eIFjfmEyR/vMAoWVfYbNzavruTPyqIM0DSVKKcwJ7bdlN2t6gh2GS?=
 =?us-ascii?Q?ZiR2VmH6CxI+jbW4qIlygTQZvQ0+kcow6O8mhZFLda0ilQGO4c5ovmcv8V8Q?=
 =?us-ascii?Q?k+Ic1nA+R8V3TbYGtNmAr/ba0YdvOTw+AkGN7Ij6Z8zqm4GWflZgap2vjVoD?=
 =?us-ascii?Q?HBn2nfiD4kMaRM57sE+gt9AEhHfpmRJXt/9uWmMtABtuUk+wRk+WCPxFGnMd?=
 =?us-ascii?Q?on21M/+FZtDHPJrCvo7M/uQ4bzginWBQW/XFGbc/nhxdVAGy9qAIDITxfsVv?=
 =?us-ascii?Q?aqR/5yJxCgDojE8DFj6OJlwvu8nYTY8hpJLpGDlBDuBBYc4UFnm2HC8ILKA3?=
 =?us-ascii?Q?Ow=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <B8F251CD34FA4F42821867EB2B04DB18@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: FWXE6Xh0Jir9pE+hn4t9Fn9wWdSSQCYluWixtigTYnFxqDCbdJ8tx7aOGv6uRqp4VEGGlMUppSOWGynKd1bQVO43QaNtWY7DnnWPSpyS7M2RTEM2nOJMLOjgxiiLSMC4TmeAMhaJI3rGVS8HNWz9NsQwktaPZ6bleutJ7RukhGkU5+0hfNuvoySxomsu9f9kLcF6CciiEiCc8WGS6nhh05/d/+92S63eCq+wasH+TQu/biiCEnGKF6h42TcT/Hhch3rGCdyxzrFvaRnvSaBITYkYqbvqsTHaW9G8emHMVjf/fiJWzfrj7iZw8soPJFHbp2rdqSF1+YuH+vO9XHYYUepSen5OeXut8r/zw268ugW8lGL13KRwAHetb27SHB0ndjHG3IY00EOSqqmrDcny39CNiQ/OhbAaLE1nDJyFW3S0i0gbKrJ0b1ZQJeUBfMVcCwXvoGxgYuUxzZctxPZvjA4dsHwFGOv2AiqgjI3nIVp+lZYual9mIE8UOPSTD8DUFmtoNMvG+bmwXwqdtcDNzguPa4ICiKy+0PRgaJUJbcyzrHdFIR7J2udqYRUS6my5zDAOprtavgMdSwbrto+LAFHsRT1dwDkwsVuEq6UxLeo5PZ2yHtSVMGpTMSgmFEojHcAR23goyxaKmn0iZqYkWzC2E3WNsTGy6h4YsTAGur8uCoD3obrEmMNOjrOVIHRkPquXdDnhD7TRe0isLQrQQMZulYVRjMgjQcsnXybhhROwQ3FbgGZpN7fI68XSO/UAKgchWaTuOk/3KlOUKop1SP/tBF2kw/qpD5Gt24LcqRl0BgEtAFvKDjhZ/2befsP8
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 39ff9476-91d5-44b4-d99e-08db71b14a05
X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Jun 2023 17:10:46.8088
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: shWgFUGws17Z7H+4mK28eZKrfpgzBhvM6e1aUzTd/v2jxIAiGS0SAhlMKn2OyV7SzU6h3N159xFM1yXgol1Nng==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLAPR10MB4817
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.591,FMLib:17.11.176.26
 definitions=2023-06-20_12,2023-06-16_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 adultscore=0
 spamscore=0 phishscore=0 suspectscore=0 malwarescore=0 bulkscore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2305260000 definitions=main-2306200155
X-Proofpoint-GUID: FIkI6ZFFcg1LYt03rfc2lKNOpBYF5-kf
X-Proofpoint-ORIG-GUID: FIkI6ZFFcg1LYt03rfc2lKNOpBYF5-kf
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jun 20, 2023, at 11:27 AM, dai.ngo@oracle.com wrote:
>=20
> On 6/19/23 12:19 PM, Trond Myklebust wrote:
>> Hi Dai,
>>=20
>> On Mon, 2023-06-19 at 10:02 -0700, dai.ngo@oracle.com wrote:
>>> Hi Trond,
>>>=20
>>> I'm testing the NFS server with write delegation support and the
>>> Linux client
>>> using NFSv4.0 and run into a situation that needs your advise.
>>>=20
>>> In this scenario, the NFS server grants the write delegation to the
>>> client.
>>> Later when the client returns delegation it sends the compound PUTFH,
>>> GETATTR
>>> and DELERETURN.
>>>=20
>>> When the NFS server services the GETATTR, it detects that there is a
>>> write
>>> delegation on this file but it can not detect that this GETATTR
>>> request was
>>> sent from the same client that owns the write delegation (due to the
>>> nature
>>> of NFSv4.0 compound). As the result, the server sends CB_RECALL to
>>> recall
>>> the delegation and replies NFS4ERR_DELAY to the GETATTR request.
>>>=20
>>> When the client receives the NFS4ERR_DELAY it retries with the same
>>> compound
>>> PUTFH, GETATTR, DELERETURN and server again replies the
>>> NFS4ERR_DELAY. This
>>> process repeats until the recall times out and the delegation is
>>> revoked by
>>> the server.
>>>=20
>>> I noticed that the current order of GETATTR and DELEGRETURN was done
>>> by
>>> commit e144cbcc251f. Then later on, commit 8ac2b42238f5 was added to
>>> drop
>>> the GETATTR if the request was rejected with EACCES.
>>>=20
>>> Do you have any advise on where, on server or client, this issue
>>> should
>>> be addressed?
>> This wants to be addressed in the server. The client has a very good
>> reason for wanting to retrieve the attributes before returning the
>> delegation here: it needs to update the change attribute while it is
>> still holding the delegation in order to ensure close-to-open cache
>> consistency.
>>=20
>> Since you do have a stateid in the DELEGRETURN, it should be possible
>> to determine that this is indeed the client that holds the delegation.
>=20
> Thank you Trond. I'll wait for Chuck to decide what to do next.

I don't have a technical objection to Trond's proposal. But until
the WG documents the interoperability requirements, I don't believe
it's something NFSD can sensibly support.

So, until the WG weighs in, NFSD won't offer write delegations to
NFSv4.0 clients. Let's be sure to document why in code comments.


--
Chuck Lever