Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Dai Ngo <dai.ngo@oracle.com>
CC:     "jlayton@kernel.org" <jlayton@kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: [PATCH v3 1/2] NFSD: add support for sending CB_RECALL_ANY
Thread-Topic: [PATCH v3 1/2] NFSD: add support for sending CB_RECALL_ANY
Thread-Index: AQHY6nNLIPitKTs0XEalLwD47leXtq4j2ZQAgANHfwCAAA+UAA==
Date:   Sun, 30 Oct 2022 17:12:25 +0000
Message-ID: <91927B86-76D7-4C5C-B8C5-8AF30BF52547@oracle.com>
References: <1666923369-21235-1-git-send-email-dai.ngo@oracle.com>
 <1666923369-21235-2-git-send-email-dai.ngo@oracle.com>
 <D080D405-5567-4581-A347-417E859A568D@oracle.com>
 <a29c1888-d260-1108-5365-8385192b4367@oracle.com>
In-Reply-To: <a29c1888-d260-1108-5365-8385192b4367@oracle.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?dur64Pj9P+7pnKQx0KyuI/gaW2hromjGIxfmFTdTjZHnexFtGvbf463zBcgP?=
 =?us-ascii?Q?b4jLnnzVNYDg6p05WdvQtz4ugsXIJRSRzUXgVHT3hw6hOdDcQchByKqWXK+R?=
 =?us-ascii?Q?BUb768mFf4sUnkwSBKA5lmzIw1ggSAWQgiL8WL4+0NELxxZaLBZ2JZYLJApM?=
 =?us-ascii?Q?Aaac8O97UMmCOOZ//G2R5ElbmImSTAHTgLzgLorWuirR2VE5xqcHYq6+E1pp?=
 =?us-ascii?Q?ykDIsLjIIO3ZxYQQAHKhjTdvXHctvZcrKZdJo9ZF4Q+l846toLZwF46f32ug?=
 =?us-ascii?Q?75BwSRnXtsNoSz1N2/E4r+kIRkpG8b0X0xHA8N0GNk5WiI8bjKwyecfOckCa?=
 =?us-ascii?Q?CX//gm7EdAOpTIJp0Sso4qhuCh0Fwfk49pXk9tmTygNZVPYzn7RGpP9UoWZq?=
 =?us-ascii?Q?4gruJAJ39QE02fOlkbqTWuk4QglgGOT37B8idhepUI5X+9yPrb5n6N4sho6K?=
 =?us-ascii?Q?uh7vgcV+1SAsTa7ddHBswM5HPj70Sb5OEpKM2rYeNTIy6tPIQxm3noUkgXoA?=
 =?us-ascii?Q?BkEQ9CnJzKkUHTi5QMgpDFSajTf892kUEme0QQKPxmgsrimHKqwGD8lHT3EI?=
 =?us-ascii?Q?hvXwTY3YMJsiLaYbmb66Pf247VB9HCOpCJovMGsXZNX1ZoJOOfmLUS9olt9b?=
 =?us-ascii?Q?+t8kwb5K2wQf8iX6kC50+P1vUaas2TRfpzH6Sum01/qgUZaUR9aBzbqWQYGT?=
 =?us-ascii?Q?ECzAX4mKhk96sM3sQLZVs3mOFkncq5LvgFbDTt1Ppmgt9+yRmPulIQfU0OfB?=
 =?us-ascii?Q?hJ5xnssCfiOrTS3lhdSuuCpF0Crt2Y9OkK72crc7W9nj2Q454siYz/EOReOQ?=
 =?us-ascii?Q?dR5bItrJyDGaZAlEzC8Q9YutHFxZicbG8NNdCokLLXdQiasrdPF+77fgk5gY?=
 =?us-ascii?Q?RMips43/o+vJr2X+s6oQYZ6bYMId1co5v6yDIt+mnN8mn1ocrqp+fsEwNg40?=
 =?us-ascii?Q?DZaYMrSAyTk3csPIQOcHTT9LwIuRnEE5w8P0U8ZTEhRkfVspVxYHRH6qFvNm?=
 =?us-ascii?Q?ELoIqNcExy4My+k2KhxfK8Gdrj13OBJN9QZ3bawmlxqNwSG0IMOYhuscoint?=
 =?us-ascii?Q?d9R8+No2N2uuefeK0yOdGqIV99fm9inl2TIB2hbtKEFiPErKwSMVwA1NZO7S?=
 =?us-ascii?Q?GNmPhyJx52CHiH+/B/1WRg3DU6lGee7PO1X99b3bnsrDc4M3n3A5hfoy2d2M?=
 =?us-ascii?Q?ZvBDvXwHYT2MujMeE3P3VU/S7QGHWmJ6HF/mysLxFVaZmvXM0TC/OwtkKd/2?=
 =?us-ascii?Q?+H9dyuJu5IsaUtjdKWUeCU8yAJDjk0uwsgRMPVOFNlikYFRrfPWK6YDHb8Mk?=
 =?us-ascii?Q?nD6WrsroyzNAD07Xy+7H7yH4s7C6a1CWObQtTJ1tbgd6iMaOuIeaMDJJKWot?=
 =?us-ascii?Q?AjCE1Sk1fugjDlthDxWkOlDk5SzG3nT56V1x4azR5HCN+wvugwnIWXBiaOHc?=
 =?us-ascii?Q?xXkcpRrcSvJDgHdsYL/SKqfo1ITFHAJeAIAvsUxnML1vf5Aa6rqsbdsUvGfl?=
 =?us-ascii?Q?YNuXaNUWMbdV4QRRObwAHzfyTLdRWPzediLv2a7Lkp16vpL1fhXio1WsPTyJ?=
 =?us-ascii?Q?OeZTugKi+x9HHw+8/GPZZMqzDy2hH0z67QfeMZtFv2t5a2m9pwNGBIG0kHWM?=
 =?us-ascii?Q?Ag=3D=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <02BD50F82D5DBB4A8E67C30FD75CD80E@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f57635e0-3561-412c-af50-08daba99eac4
X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Oct 2022 17:12:25.7555
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: lMno+KQWbQ3kdRTQI/9kAdrOv8H1H29gfAJqv/tLxGgk8BnNZS+oyR7/Oz+2bZNeFke3X2djoVnHh9bk8XEMrg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA2PR10MB4524
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1
 definitions=2022-10-30_10,2022-10-27_01,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 mlxscore=0
 suspectscore=0 malwarescore=0 spamscore=0 adultscore=0 phishscore=0
 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2210170000 definitions=main-2210300114
X-Proofpoint-ORIG-GUID: q81foS69dzzwoPTnadL41Id8vWr33D3C
X-Proofpoint-GUID: q81foS69dzzwoPTnadL41Id8vWr33D3C
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Oct 30, 2022, at 12:16 PM, Dai Ngo <dai.ngo@oracle.com> wrote:
>=20
> On 10/28/22 7:11 AM, Chuck Lever III wrote:
>>=20
>>> On Oct 27, 2022, at 10:16 PM, Dai Ngo <dai.ngo@oracle.com> wrote:
>>>=20
>>> There is only one nfsd4_callback, cl_recall_any, added for each
>>> nfs4_client.
>> Is only one needed at a time, or might NFSD need to send more
>> than one concurrently? So far it looks like the only reason to
>> limit it to one at a time is the way the cb_recall_any arguments
>> are passed to the XDR layer.
>=20
> We only need to send one CB_RECALL_ANY for each client that hold
> delegations so just one nfsd4_callback needed for this purpose.
> Do you see a need to support multiple nfsd4_callback's per client
> for cl_recall_any?

CB_RECALL_ANY has a bitmap argument so that it can be used for
recalling many different types of state. I don't pretend to
understand all of it, but it seems like eventually NFSD might
want to send more than one of these at a time, and not just in
low memory scenarios. I suggest a way below of doing this
without adding complexity.


>>> Access to it must be serialized. For now it's done
>>> by the cl_recall_any_busy flag since it's used only by the
>>> delegation shrinker. If there is another consumer of CB_RECALL_ANY
>>> then a spinlock must be used.
>> The usual arrangement is to add the XDR infrastructure for a new
>> operation in one patch, and then add consumers in subsequent
>> patches. Can you move the hunks that change fs/nfsd/nfs4state.c
>> to 2/2 and update the above description accordingly?
>=20
> fix in v4.
>=20
>>=20
>> In a separate patch you should add a trace_nfsd_cb_recall_any and
>> a trace_nfsd_cb_recall_any_done tracepoint. There are already nice
>> examples in fs/nfsd/trace.h for the other callback operations.
>=20
> fix in v4.
>=20
>>=20
>>=20
>> A little more below.
>>=20
>>=20
>>> Signed-off-by: Dai Ngo <dai.ngo@oracle.com>
>>> ---
>>> fs/nfsd/nfs4callback.c | 64 +++++++++++++++++++++++++++++++++++++++++++=
+++++++
>>> fs/nfsd/nfs4state.c    | 32 +++++++++++++++++++++++++
>>> fs/nfsd/state.h        |  8 +++++++
>>> fs/nfsd/xdr4cb.h       |  6 +++++
>>> 4 files changed, 110 insertions(+)
>>>=20
>>> diff --git a/fs/nfsd/nfs4callback.c b/fs/nfsd/nfs4callback.c
>>> index f0e69edf5f0f..03587e1397f4 100644
>>> --- a/fs/nfsd/nfs4callback.c
>>> +++ b/fs/nfsd/nfs4callback.c
>>> @@ -329,6 +329,29 @@ static void encode_cb_recall4args(struct xdr_strea=
m *xdr,
>>> }
>>>=20
>>> /*
>>> + * CB_RECALLANY4args
>>> + *
>>> + *	struct CB_RECALLANY4args {
>>> + *		uint32_t	craa_objects_to_keep;
>>> + *		bitmap4		craa_type_mask;
>>> + *	};
>>> + */
>>> +static void
>>> +encode_cb_recallany4args(struct xdr_stream *xdr,
>>> +			struct nfs4_cb_compound_hdr *hdr, uint32_t bmval)
>>> +{
>>> +	__be32 *p;
>>> +
>>> +	encode_nfs_cb_opnum4(xdr, OP_CB_RECALL_ANY);
>>> +	p =3D xdr_reserve_space(xdr, 4);
>>> +	*p++ =3D xdr_zero;	/* craa_objects_to_keep */
>> Let's use xdr_stream_encode_u32() here.
>=20
> Yes, we can use xdr_stream_encode_u32() here. However xdr_stream_encode_u=
32
> can return error and currently all the xdr encoding functions,
> nfs4_xdr_enc_xxxx, are defined to return void so rpcauth_wrap_req_encode
> just ignores encode errors always return 0. Note that gss_wrap_req_integ
> and gss_wrap_req_priv actually return encoding errors.

The convention it seems is to ignore errors in this code and assume
that the upper layer has set up the send buffer properly. Anything
else would be a local software bug, so it is probably a safe
assumption.


>>  Would it be reasonable
>> for the upper layer to provide this value, or will NFSD always
>> want it to be zero?
>=20
> Ideally the XDR encode routine should not make any decision regarding
> how many objects the client can keep, it's up to the consumer of the
> CB_RECALL_ANY to decide. However in this case, NFSD always want to set
> this to 0

"in this case" I assume means in the low-memory scenario. I don't
know enough to say if other scenarios might want to choose a
different value.


> so instead of passing this in as another argument to the
> encoder, I just did the short cut here. Do you want to pass this in
> as an argument?

I don't think it would be difficult to parametrize this, so give it
a shot. Suggestion way below.


>>> +	p =3D xdr_reserve_space(xdr, 8);
>> Let's use xdr_stream_encode_uint32_array() here. encode_cb_recallany4arg=
s's
>> caller should pass a u32 * and a length, not just a simple u32.
>=20
> fix in v4.
>=20
>>=20
>>=20
>>> +	*p++ =3D cpu_to_be32(1);
>>> +	*p++ =3D cpu_to_be32(bmval);
>>> +	hdr->nops++;
>>> +}
>>> +
>>> +/*
>>>  * CB_SEQUENCE4args
>>>  *
>>>  *	struct CB_SEQUENCE4args {
>>> @@ -482,6 +505,24 @@ static void nfs4_xdr_enc_cb_recall(struct rpc_rqst=
 *req, struct xdr_stream *xdr,
>>> 	encode_cb_nops(&hdr);
>>> }
>>>=20
>>> +/*
>>> + * 20.6. Operation 8: CB_RECALL_ANY - Keep Any N Recallable Objects
>>> + */
>>> +static void
>>> +nfs4_xdr_enc_cb_recall_any(struct rpc_rqst *req,
>>> +		struct xdr_stream *xdr, const void *data)
>>> +{
>>> +	const struct nfsd4_callback *cb =3D data;
>>> +	struct nfs4_cb_compound_hdr hdr =3D {
>>> +		.ident =3D cb->cb_clp->cl_cb_ident,
>>> +		.minorversion =3D cb->cb_clp->cl_minorversion,
>>> +	};
>>> +
>>> +	encode_cb_compound4args(xdr, &hdr);
>>> +	encode_cb_sequence4args(xdr, cb, &hdr);
>>> +	encode_cb_recallany4args(xdr, &hdr, cb->cb_clp->cl_recall_any_bm);
>>> +	encode_cb_nops(&hdr);
>>> +}
>>>=20
>>> /*
>>>  * NFSv4.0 and NFSv4.1 XDR decode functions
>>> @@ -520,6 +561,28 @@ static int nfs4_xdr_dec_cb_recall(struct rpc_rqst =
*rqstp,
>>> 	return decode_cb_op_status(xdr, OP_CB_RECALL, &cb->cb_status);
>>> }
>>>=20
>>> +/*
>>> + * 20.6. Operation 8: CB_RECALL_ANY - Keep Any N Recallable Objects
>>> + */
>>> +static int
>>> +nfs4_xdr_dec_cb_recall_any(struct rpc_rqst *rqstp,
>>> +				  struct xdr_stream *xdr,
>>> +				  void *data)
>>> +{
>>> +	struct nfsd4_callback *cb =3D data;
>>> +	struct nfs4_cb_compound_hdr hdr;
>>> +	int status;
>>> +
>>> +	status =3D decode_cb_compound4res(xdr, &hdr);
>>> +	if (unlikely(status))
>>> +		return status;
>>> +	status =3D decode_cb_sequence4res(xdr, cb);
>>> +	if (unlikely(status || cb->cb_seq_status))
>>> +		return status;
>>> +	status =3D  decode_cb_op_status(xdr, OP_CB_RECALL_ANY, &cb->cb_status=
);
>>> +	return status;
>>> +}
>>> +
>>> #ifdef CONFIG_NFSD_PNFS
>>> /*
>>>  * CB_LAYOUTRECALL4args
>>> @@ -783,6 +846,7 @@ static const struct rpc_procinfo nfs4_cb_procedures=
[] =3D {
>>> #endif
>>> 	PROC(CB_NOTIFY_LOCK,	COMPOUND,	cb_notify_lock,	cb_notify_lock),
>>> 	PROC(CB_OFFLOAD,	COMPOUND,	cb_offload,	cb_offload),
>>> +	PROC(CB_RECALL_ANY,	COMPOUND,	cb_recall_any,	cb_recall_any),
>>> };
>>>=20
>>> static unsigned int nfs4_cb_counts[ARRAY_SIZE(nfs4_cb_procedures)];
>>> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
>>> index 4e718500a00c..68d049973ce3 100644
>>> --- a/fs/nfsd/nfs4state.c
>>> +++ b/fs/nfsd/nfs4state.c
>>> @@ -2854,6 +2854,36 @@ static const struct tree_descr client_files[] =
=3D {
>>> 	[3] =3D {""},
>>> };
>>>=20
>>> +static int
>>> +nfsd4_cb_recall_any_done(struct nfsd4_callback *cb,
>>> +			struct rpc_task *task)
>>> +{
>>> +	switch (task->tk_status) {
>>> +	case -NFS4ERR_DELAY:
>>> +		rpc_delay(task, 2 * HZ);
>>> +		return 0;
>>> +	default:
>>> +		return 1;
>>> +	}
>>> +}
>>> +
>>> +static void
>>> +nfsd4_cb_recall_any_release(struct nfsd4_callback *cb)
>>> +{
>>> +	struct nfs4_client *clp =3D cb->cb_clp;
>>> +	struct nfsd_net *nn =3D net_generic(clp->net, nfsd_net_id);
>>> +
>>> +	spin_lock(&nn->client_lock);
>>> +	clp->cl_recall_any_busy =3D false;
>>> +	put_client_renew_locked(clp);
>>> +	spin_unlock(&nn->client_lock);
>>> +}
>>> +
>>> +static const struct nfsd4_callback_ops nfsd4_cb_recall_any_ops =3D {
>>> +	.done		=3D nfsd4_cb_recall_any_done,
>>> +	.release	=3D nfsd4_cb_recall_any_release,
>>> +};
>>> +
>>> static struct nfs4_client *create_client(struct xdr_netobj name,
>>> 		struct svc_rqst *rqstp, nfs4_verifier *verf)
>>> {
>>> @@ -2891,6 +2921,8 @@ static struct nfs4_client *create_client(struct x=
dr_netobj name,
>>> 		free_client(clp);
>>> 		return NULL;
>>> 	}
>>> +	nfsd4_init_cb(&clp->cl_recall_any, clp, &nfsd4_cb_recall_any_ops,
>>> +			NFSPROC4_CLNT_CB_RECALL_ANY);
>>> 	return clp;
>>> }
>>>=20
>>> diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
>>> index e2daef3cc003..49ca06169642 100644
>>> --- a/fs/nfsd/state.h
>>> +++ b/fs/nfsd/state.h
>>> @@ -411,6 +411,10 @@ struct nfs4_client {
>>>=20
>>> 	unsigned int		cl_state;
>>> 	atomic_t		cl_delegs_in_recall;
>>> +
>>> +	bool			cl_recall_any_busy;
>> Rather than adding a boolean field, you could add a bit to
>> cl_flags.
>=20
> fix in v4.
>=20
>>=20
>> I'm not convinced you need to add the argument fields here...
>> I think kmalloc'ing the arguments and then freeing them in
>> nfsd4_cb_recall_any_release() would be sufficient.
>=20
> Since cb_recall_any is sent when we're running low on system memory,
> I'm trying not to use kmalloc'ing to avoid potential deadlock or adding
> more stress to the system.

Fair. Well, again, there are potentially other use cases that
aren't concerned about memory pressure.

Construct something like this (in fs/nfsd/xdr4.h):

struct nfsd4_cb_recall_any {
	struct nfsd4_callback	ra_cb;
	u32			ra_keep;
	u32			ra_bmval[1];
};

Then either add it to another long-term structure like nfs4_client
or the upper layer can kmalloc it on demand. That way the XDR code
supports both ways and does not have to care which the upper layer
wants to use.


> Thanks!
> -Dai
>=20
>>=20
>>=20
>>> +	uint32_t		cl_recall_any_bm;
>>> +	struct nfsd4_callback	cl_recall_any;
>>> };
>>>=20
>>> /* struct nfs4_client_reset
>>> @@ -639,8 +643,12 @@ enum nfsd4_cb_op {
>>> 	NFSPROC4_CLNT_CB_OFFLOAD,
>>> 	NFSPROC4_CLNT_CB_SEQUENCE,
>>> 	NFSPROC4_CLNT_CB_NOTIFY_LOCK,
>>> +	NFSPROC4_CLNT_CB_RECALL_ANY,
>>> };
>>>=20
>>> +#define RCA4_TYPE_MASK_RDATA_DLG	0
>>> +#define RCA4_TYPE_MASK_WDATA_DLG	1
>>> +
>>> /* Returns true iff a is later than b: */
>>> static inline bool nfsd4_stateid_generation_after(stateid_t *a, stateid=
_t *b)
>>> {
>>> diff --git a/fs/nfsd/xdr4cb.h b/fs/nfsd/xdr4cb.h
>>> index 547cf07cf4e0..0d39af1b00a0 100644
>>> --- a/fs/nfsd/xdr4cb.h
>>> +++ b/fs/nfsd/xdr4cb.h
>>> @@ -48,3 +48,9 @@
>>> #define NFS4_dec_cb_offload_sz		(cb_compound_dec_hdr_sz  +      \
>>> 					cb_sequence_dec_sz +            \
>>> 					op_dec_sz)
>>> +#define NFS4_enc_cb_recall_any_sz	(cb_compound_enc_hdr_sz +       \
>>> +					cb_sequence_enc_sz +            \
>>> +					1 + 1 + 1)
>>> +#define NFS4_dec_cb_recall_any_sz	(cb_compound_dec_hdr_sz  +      \
>>> +					cb_sequence_dec_sz +            \
>>> +					op_dec_sz)
>>> --=20
>>> 2.9.5
>>>=20
>> --
>> Chuck Lever

--
Chuck Lever