Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Dai Ngo <dai.ngo@oracle.com>
CC:     Jeff Layton <jlayton@kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        "linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>
Subject: Re: [PATCH v5 2/3] NFSD: handle GETATTR conflict with write
 delegation
Thread-Topic: [PATCH v5 2/3] NFSD: handle GETATTR conflict with write
 delegation
Thread-Index: AQHZjQiHYb+9V1yj3kqcohFaOFOV+a9piOSAgAAtP4CAAYpwAA==
Date:   Thu, 25 May 2023 17:21:30 +0000
Message-ID: <7A5BD9CE-5592-41C4-8C48-D7E489C59C37@oracle.com>
References: <1684799560-31663-1-git-send-email-dai.ngo@oracle.com>
 <1684799560-31663-3-git-send-email-dai.ngo@oracle.com>
 <33ffcd5fd5d794fb642bbabf93f34a61d2f0d4e9.camel@kernel.org>
 <58191055-96c8-d693-bbec-2081b1bc5a75@oracle.com>
In-Reply-To: <58191055-96c8-d693-bbec-2081b1bc5a75@oracle.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?VKi0LIdfW4CBrYIcBawd9dxu/mKrQlPeW8MyVcF5/cIPsaVzxVfiXtDn2VJU?=
 =?us-ascii?Q?37+bFnaLx0Y4dsYB0Qv7UIHX4yZAMcPhk9G9312QJGWKjn9a2aamCdjeZ+h2?=
 =?us-ascii?Q?bZjlQPaeemUM+vOo9T2xwbJkh0NB5seq2JtI1arzFmxyzbiMaF/Cqi4CG972?=
 =?us-ascii?Q?7G4Sre8JnJ8PfpzLCeRU4u1gYhd/8nQw09FLx+IvLSUiE4Z8ucilpGbSGUlt?=
 =?us-ascii?Q?9SVWvk4X/mj8kLLv0uMJ9iuUqVFrHNUJqPEOD9M0xp7umcT0ampDyKNqA0Am?=
 =?us-ascii?Q?VFJAHQklcm9VIJERTdWWfmW/752swpfsDO8YildmCYXDH9hh3SzaGw0QWMQx?=
 =?us-ascii?Q?1ojaUn0cwodCRU7shvwZmPVZjIBCEAfFGOOlfvzleT5Y8q5oQfzVTg0ZIxDu?=
 =?us-ascii?Q?sjRhp/MDpzdshuFoOwmCCnNakei514tGkkWsmv5CW2E4CW4PmUGDJBdU7NOY?=
 =?us-ascii?Q?ftFDYTpd/F1/X5LqPeycjbptdOOIvX5HjSF5dlZ7K5sXcKJgpTA9JRaToIA0?=
 =?us-ascii?Q?KVaYdTzPkTZ+FSz2F9KK/N4getF+XuQwqQzn1eGggPlEJYUgO6KmymDGC9o2?=
 =?us-ascii?Q?eJNsJfLfQCO+94EwSrx4A4M75OLZ61X0mWtKQ15WmQMVLQ8qYztflaDBdu31?=
 =?us-ascii?Q?abk72fF+2IJ+iIoVPX4r1keTSmHouVd9TK1RmXg8NESr7CRiBgdCWG32sE1h?=
 =?us-ascii?Q?rkUZwe2C+m0dPtKmIX4vm8VQCc2cvpBVzh5zaB7HAYNMCoxg/57Jfvuzdpin?=
 =?us-ascii?Q?OBnEgXIVJSigTS5+1hqkh3Vf6LtML8wWu4LmxEJ6ymIZByv8oDoiZFe0goDY?=
 =?us-ascii?Q?JCiom23RtNlqJM3qxCfgMDGbDcsscjJIEnVCpULUSnF3WgF8JXj3Zf0advyO?=
 =?us-ascii?Q?nl62SKR3onAr13qKSexS65Fsx8RrhzKkHA5fNRh8bYqK6Esbjini16krzb48?=
 =?us-ascii?Q?pA75Kl/a4YzZY7h1AWKGFH61wsR7woJ2CGpNVNDe0OUXXTKjWDen1SMgT1i8?=
 =?us-ascii?Q?vFxBRn8Li9yKrUwXX9AlNUJ2fUPmftZTk0OhBlUM9nMD1qmZ+YZYzfdSjFyG?=
 =?us-ascii?Q?DpvRXPi6s5SfMyCXR73C7G4iRa54RqNdlar/q7zXNLRfrcjGG9kF1hk4NyqA?=
 =?us-ascii?Q?ZpiYJgGmcgqxNcc2Vw8Ro0DLur0/FvdCs2MOdNhQn9IkF/p7i5pTOZuPtaSa?=
 =?us-ascii?Q?YH5uMmckXmDKO6bbyBK6JZiEqbhksJJKqZtKlQ6ulPO+1r492U35L2h9jbsM?=
 =?us-ascii?Q?UKz4VTa8HYdFc69HKs8KiPuRGIXiqDDOUiQ18ZYa1c/x7Wkfa6piNOfnfun7?=
 =?us-ascii?Q?vYGeyrFYqX4y0o5GLJj3dv+tlgz6ZfUCDnsb0spyru0bxY1Wt//2sZ4oJjoF?=
 =?us-ascii?Q?F66WyRszhRVi4ZUrIsCyb+tkFchPlnhiitRLzNnSj8Btcwm+2ebFxvIdAvO4?=
 =?us-ascii?Q?yDeUS0wwti5AKBHgI8WrmSAEbbJKbFg9QIjbUXVmeHLL5GQuMpXp24BD4r0v?=
 =?us-ascii?Q?IBBmrQI0TNA+4XAnb2cW9TrB6mYbM8LNjTNYJygIR5oAVDfwlJKtONwFtXtL?=
 =?us-ascii?Q?JUiJnGhWHZccmKzaGtJf4tUXW8YA6bssvX/jdxG9hvuGEXy3HjGwVwzCypwD?=
 =?us-ascii?Q?/Q=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <6403EFD8735F68419097687DD06F143F@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: FLX+voeJDLMYuJjA1usSPZOTWQY+D5p2uOi8DfEkcSjJkQEiH94hvnvhVnIRQBP+S1IhJBOaCOYBmFN/wwh5/2j2ez8MtTbqCd1ovI/BvLn8/L7rtvzDUW470l8cSuZrdKoLi7l4+Plo0xxGfnDGsifYyo26frnUBGRw1toqjIaK2buq3ntKEZNNXLWI/UTmnW/IMwhGb5QeWvH92E6hlWvwpS5+2otoSQFBk6kk7dOIlhTT+uycXPaKlvdEMpgsBz0xltaxZM+nzserGLx4cqgvaeEPHoXb5BXcMXFLgPtmIKTwljzDjejw+0LwYg15unTN+8Rs+fqsE49Pe1RtuA8piM+ZmTt/3H9xFMGKAJFDbU8lRiTeN1g8Bx1AUTjShkcJQdOPkQuIK8kw9/5OJySmCVszowDPiIW+8YDxQfqYxoNhURhUqJoIAeOR+zvh816DDkdd2D7gNWnneQNVQyN6f5THlQCxn+GNVWUSschQajq+PKZG981Ie54+orHS/UFNXURCv6PMb2T4lNs2LtezXfCbWiKNMD7wS+YgqHrycdkk9m9wMMLKp2bMsCg7bZRWdNipge50wbR5uyODjsuYk7lz/qr7uAvo0tFzsXyJAbAVs9WmBrofwdOfQE9ByaM4FslbSnWAd8bdHZC/f+qenIc1+7yPC0fozynpWmxdpvPwW8LPDgO6032BsRP6DVq/yvjpNHqJ69SK1cqqg+KbM6aKvPjOZTLxzBgIXGmFzBpQ4hQANDnzMNYz/IaONCBbIrXi+3p8WAe2zsQfmUuZ3KaPBggF72pXzTAQHW8=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 98564b20-637d-49ad-9bb0-08db5d447b03
X-MS-Exchange-CrossTenant-originalarrivaltime: 25 May 2023 17:21:30.5680
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: +9hnrEMCd/3c2bGxt8+EQDQXu9pqvKLZjwAKLtlxnuC71Rpfd+jJtJ+kMcsKkzgVHNxejaZTJ/lVibyTvH2G/Q==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB7132
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.573,FMLib:17.11.176.26
 definitions=2023-05-25_10,2023-05-25_03,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 bulkscore=0
 mlxlogscore=999 mlxscore=0 adultscore=0 spamscore=0 suspectscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2304280000
 definitions=main-2305250144
X-Proofpoint-GUID: WDvQ4LyF7U-m_B14TDH92o6_DwECJ3y8
X-Proofpoint-ORIG-GUID: WDvQ4LyF7U-m_B14TDH92o6_DwECJ3y8
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE,
        T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On May 24, 2023, at 1:49 PM, Dai Ngo <dai.ngo@oracle.com> wrote:
>=20
>=20
> On 5/24/23 8:07 AM, Jeff Layton wrote:
>> On Mon, 2023-05-22 at 16:52 -0700, Dai Ngo wrote:
>>> If the GETATTR request on a file that has write delegation in effect
>>> and the request attributes include the change info and size attribute
>>> then the write delegation is recalled and NFS4ERR_DELAY is returned
>>> for the GETATTR.
>>>=20
>>> Signed-off-by: Dai Ngo <dai.ngo@oracle.com>
>>> ---
>>>  fs/nfsd/nfs4state.c | 37 +++++++++++++++++++++++++++++++++++++
>>>  fs/nfsd/nfs4xdr.c   |  5 +++++
>>>  fs/nfsd/state.h     |  3 +++
>>>  3 files changed, 45 insertions(+)
>>>=20
>>> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
>>> index b90b74a5e66e..ea9cd781db5f 100644
>>> --- a/fs/nfsd/nfs4state.c
>>> +++ b/fs/nfsd/nfs4state.c
>>> @@ -8353,3 +8353,40 @@ nfsd4_get_writestateid(struct nfsd4_compound_sta=
te *cstate,
>>>  {
>>>   get_stateid(cstate, &u->write.wr_stateid);
>>>  }
>>> +
>>> +__be32
>>> +nfs4_handle_wrdeleg_conflict(struct svc_rqst *rqstp, struct inode *ino=
de)
>>> +{
>>> + struct file_lock_context *ctx;
>>> + struct file_lock *fl;
>>> + struct nfs4_delegation *dp;
>>> +
>>> + ctx =3D locks_inode_context(inode);
>>> + if (!ctx)
>>> + return 0;
>>> + spin_lock(&ctx->flc_lock);
>>> + list_for_each_entry(fl, &ctx->flc_lease, fl_list) {
>>> + if (fl->fl_flags =3D=3D FL_LAYOUT ||
>>> + fl->fl_lmops !=3D &nfsd_lease_mng_ops)
>>> + continue;
>>> + if (fl->fl_type =3D=3D F_WRLCK) {
>>> + dp =3D fl->fl_owner;
>>> + /*
>>> +  * increment the sc_count to prevent the delegation to
>>> +  * be freed while sending the CB_RECALL. The refcount is
>>> +  * decremented by nfs4_put_stid in nfsd4_cb_recall_release
>>> +  * after the request was sent.
>>> +  */
>>> + if (dp->dl_recall.cb_clp =3D=3D *(rqstp->rq_lease_breaker) ||
>>> + !refcount_inc_not_zero(&dp->dl_stid.sc_count)) {
>> I still don't get why you're incrementing the refcount of this stateid.
>> At this point, you know that this stateid is owned by a different client
>> altogether,  and breaking its lease doesn't require a reference to the
>> stateid.
>=20
> You're right, the intention was to make sure the delegation does not go
> away when the recall is being sent. However, this was already done in
> nfsd_break_one_deleg where the sc_count is incremented. Incrementing the
> sc_count refcount would be needed here if we do the CB_GETATTR. I'll remo=
ve
> this in next version.
>=20
> But should we drop the this patch altogether? since there is no value in
> recall the write delegation when there is an GETATTR from another client
> as I mentioned in the previous email.

a. Neither Solaris nor OnTAP do a CB_RECALL or a CB_GETATTR in this case

b. RFC 8881 Section 18.7.4 states that a server MUST NOT proceed with a
   GETATTR response unless it has done one of those callbacks

So we have two examples of non-compliant server implementations that
don't seem to have consequences for not following that MUST.

It doesn't seem unreasonable to recall in this scenario, but it's
unknown what kind of performance impact that will have.

Can you send an updated version of this patch, rebased on nfsd-next
(which now has the other two applied), plus a patch to add a counter
for this type of conflict?


--
Chuck Lever