Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <3755605a-40ce-546f-b287-aa1cee10cd86@oracle.com>
Date:   Wed, 27 Apr 2022 20:44:25 -0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.8.1
Subject: Re: [PATCH RFC v22 1/7] NFSD: add courteous server support for thread
 with only delegation
Content-Language: en-US
To:     "J. Bruce Fields" <bfields@fieldses.org>
Cc:     chuck.lever@oracle.com, jlayton@redhat.com,
        viro@zeniv.linux.org.uk, linux-nfs@vger.kernel.org,
        linux-fsdevel@vger.kernel.org
References: <1651049573-29552-1-git-send-email-dai.ngo@oracle.com>
 <1651049573-29552-2-git-send-email-dai.ngo@oracle.com>
 <20220427215614.GH13471@fieldses.org>
 <24607d8d-9a69-b139-ce1a-c0f70814de05@oracle.com>
 <20220428015234.GJ13471@fieldses.org>
From:   dai.ngo@oracle.com
In-Reply-To: <20220428015234.GJ13471@fieldses.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?d2hSa0tPQjRsVlA1RmJKOVB5Yy92a1BQOVlacWY5YzEzQndYQnpVRWMxcTVo?=
 =?utf-8?B?WE5SZE5mWDBNSGZyZXV2RXMwOE9sVWFTelhXOEwvb2xGODI1dWFaN1BObktt?=
 =?utf-8?B?UEY1NHVBdDhiRVdJd21yZWVnSnhSbUhYWjJUQndsNUV4ZEliaGg5VzMzbkt5?=
 =?utf-8?B?dndPRTB0RlF1THVyd2VHQnhRU1lLZnI5THZ4YjBkQUoveVYyTU9pU0YyR2lS?=
 =?utf-8?B?cWNKdXFJYTJkdW5Ucjlwd1R6K1Q4K1ZndUYwbVIya0lsNXNsa3FzaEJCUFNv?=
 =?utf-8?B?TDEzemtEL25KbFJpOE1CNy9SZnZHYUU5OEx1QjNvU25DOWdNUFdLVGJGQ1A0?=
 =?utf-8?B?OStnZmt6d0dUVHRBZ3BMMjh6YjdFZTg0aFl0UzNqbE8rOWRIK3U4czhPSjF2?=
 =?utf-8?B?SmlMVHlhbW4ycUlRWlUzanJ0RWNUVWJwNlJOL1hQb0M1dENmQ25qMThOck9o?=
 =?utf-8?B?b29YRXlMcXYrMXhveDl3em81U01TWE9FM3F5QWNMUS9aWm1BaDdUbnltVi9v?=
 =?utf-8?B?bW5ERzNFTzBLOWlqeE5scGdKd0FKekYwdWZLSUIwZVFTL0Z5djRjd2trblBX?=
 =?utf-8?B?Y0dnajhCRGJDMkNXcU1oaHFMMkl2Z0x2ZFlaam01STgvanl6WDF4THY0amF4?=
 =?utf-8?B?eVVwMnFDNk9iZ3FlZEd5cEk1MHVyOTNwb25haFJzOHprbms5VHNUaDk1Y0Ew?=
 =?utf-8?B?dHhDY0pxQlNydEJwdkxsYXducU9iSVN3L2lwT3lDQ0U1RE52eEFCN1JHYXJH?=
 =?utf-8?B?b3pvMncvZFZJTjZBQUpldlJvN0ZiZHFkcmh3SjVJb29sckNvTGRwWTdnK09X?=
 =?utf-8?B?MS84UEx2ZmtmZ2xNKytyVUlzUDBIanljNlNELysxNEU2WmVLSnJBWHRYUmFs?=
 =?utf-8?B?MUkwaEFXYk11bGNxYi9kWGJjL3BrV29lQWF3U1BaT2ZWN0h4Vm52SWo0YXZa?=
 =?utf-8?B?K0ZxZ25pTVUvZVJtckxnenE1ZGpiN2c1bi9JTnAyOHl6UHQ2ZHFiZ3p6SDlF?=
 =?utf-8?B?TEJsZjNsdmVzdjVTUzdkT3ZGUlE0UUpRTlUxYW4vS0RhSXZIczhXREpyaDkw?=
 =?utf-8?B?LzdJT0lzSmgxR1VUR3Rqc2ZSY24zM0hPM0hWdytZR0l0bUl4T3RjSDRQYlI1?=
 =?utf-8?B?SG9rMTY5Rld0YzFtQUdBQlZ3aGZPYU1rd2hHZjlQWDhhS2p6eVVNMTZSYkFX?=
 =?utf-8?B?SGlZVDNCd283RGZOS2V4STRzM2w4UVR1MmdGcG9ZNTgyMGMrMWRGYzNhMFhv?=
 =?utf-8?B?Qy8zeDhLSkduelpYVkhEbGY0YXNJZzBlOFBRR2hJc09VYUx6K1J6ZlplMDlS?=
 =?utf-8?B?QXBuL1ZIM3drQlNvS1hYYmg0bW95V2JvV1V4Si9yYlpuUVBmOUxFNEpuTUNJ?=
 =?utf-8?B?SlBSa3NiTDd2ZERDTHFGeDhyT0FLbVJtTGxCU0JaRVQrSE80NzlsUW04VTJm?=
 =?utf-8?B?dDB6eXN3Mll5WUgxQUZTR0ZvaEQ3bjBSbDNHVXNsQjFIUXMrREVicEFJVnIz?=
 =?utf-8?B?cDdyT01oelFGSmo1d0VncHVMWjI3MGlaK01DQ0svVU5wYTJXTzNjbzJyQlB3?=
 =?utf-8?B?QmdXcHBtOWttTklTaUJkbjdwQ1p3b1ZLUW5Yc2prOGtKWDlXR1RIZDBMNVc3?=
 =?utf-8?B?U096Y2dhVk5XUlplNm55d1o3NXV4ZitMc25JT0xHNnI5S25FcktMaDJ3Sm1z?=
 =?utf-8?B?TGRIRkkzdGxGV2cydXZuRU9ERVZ5T2doWWxQaUJEejNUbGlDUVdoeFBJTXlX?=
 =?utf-8?B?YmVpN25oeU1jazMyV0x6RVdxcTlFMno2NjlaUmtYOTRhZStVWTlLWmxFanVo?=
 =?utf-8?B?OURITFMyaUg2MlJmNnhRSTFNaCtuZ1VqYzZpOS95S1JTSXVLT2dqV2p4K1BW?=
 =?utf-8?B?UjBPZzQ2VHpiQm9zKzRRU2g1aU10UnZsMFFMWG40cjVCcWFESk9hN1AvcXdK?=
 =?utf-8?B?OEwyLytWVnN4SUZobTd5OW9tTDl0b0ptRGlGamxqMTVBQkhhNHBGOE1DWlpN?=
 =?utf-8?B?NW4zQTlIRjVSSTdIWEsvWWI3MFh5OXo1Z05HeUZMUVdHMnc2WnJuSVpLbjRU?=
 =?utf-8?B?VXNOZGFBa0lPTjNTcjR3ejdyRi9hbUQwQ0E2d1NxdWhpN2FFZFFndG4rV0V0?=
 =?utf-8?B?dUVjNWl5d1JZcVlVRFV6SDdadnYrK2RHYmtTbksweXZ2WW52T2I1SnkwVmNk?=
 =?utf-8?B?d3NlM2gwdG92Tm5wcGtQdVhId0xRYzZJSnlOWjF4c3hVY2R6ZkZGakZrQXBQ?=
 =?utf-8?B?WGJMeGkycC9JSlhqdDQ1eVRDVkNPQXpvekhhK2JMSmxramhpK3MvQ3ozbzZN?=
 =?utf-8?B?RjdoMDFpczhxSmo0QmlNUmpleFZSM01jb3ArclhDakhyUmNwRXJiZz09?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 713c45d1-750d-433d-1f6c-08da28c964b6
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4257.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Apr 2022 03:44:27.3134
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: hS+1d6QWBzwW5kH3YaMeZ8XBf6/LCxa3niWoNHl3Yl9FfZl1MtpxztHauxKvzM7FBHlELflZyGl3DzXOtVkoTg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR10MB4256
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.858
 definitions=2022-04-27_04:2022-04-27,2022-04-27 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999 mlxscore=0
 suspectscore=0 malwarescore=0 spamscore=0 adultscore=0 bulkscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000
 definitions=main-2204280020
X-Proofpoint-GUID: tCvUwmSEfnEpDfHxnKLFajo-5_xYJPd_
X-Proofpoint-ORIG-GUID: tCvUwmSEfnEpDfHxnKLFajo-5_xYJPd_
X-Spam-Status: No, score=-4.7 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


On 4/27/22 6:52 PM, J. Bruce Fields wrote:
> On Wed, Apr 27, 2022 at 03:52:37PM -0700, dai.ngo@oracle.com wrote:
>> On 4/27/22 2:56 PM, J. Bruce Fields wrote:
>>> On Wed, Apr 27, 2022 at 01:52:47AM -0700, Dai Ngo wrote:
>>>> This patch provides courteous server support for delegation only.
>>>> Only expired client with delegation but no conflict and no open
>>>> or lock state is allowed to be in COURTESY state.
>>>>
>>>> Delegation conflict with COURTESY/EXPIRABLE client is resolved by
>>>> setting it to EXPIRABLE, queue work for the laundromat and return
>>>> delay to the caller. Conflict is resolved when the laudromat runs
>>>> and expires the EXIRABLE client while the NFS client retries the
>>>> OPEN request. Local thread request that gets conflict is doing the
>>>> retry in _break_lease.
>>>>
>>>> Client in COURTESY or EXPIRABLE state is allowed to reconnect and
>>>> continues to have access to its state. Access to the nfs4_client by
>>>> the reconnecting thread and the laundromat is serialized via the
>>>> client_lock.
>>>>
>>>> Signed-off-by: Dai Ngo <dai.ngo@oracle.com>
>>>> ---
>>>>   fs/nfsd/nfs4state.c | 86 +++++++++++++++++++++++++++++++++++++++++++++--------
>>>>   fs/nfsd/nfsd.h      |  1 +
>>>>   fs/nfsd/state.h     | 32 ++++++++++++++++++++
>>>>   3 files changed, 106 insertions(+), 13 deletions(-)
>>>>
>>>> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
>>>> index 234e852fcdfa..216bd77a8764 100644
>>>> --- a/fs/nfsd/nfs4state.c
>>>> +++ b/fs/nfsd/nfs4state.c
>>>> @@ -125,6 +125,8 @@ static void free_session(struct nfsd4_session *);
>>>>   static const struct nfsd4_callback_ops nfsd4_cb_recall_ops;
>>>>   static const struct nfsd4_callback_ops nfsd4_cb_notify_lock_ops;
>>>> +static struct workqueue_struct *laundry_wq;
>>>> +
>>>>   static bool is_session_dead(struct nfsd4_session *ses)
>>>>   {
>>>>   	return ses->se_flags & NFS4_SESSION_DEAD;
>>>> @@ -152,6 +154,7 @@ static __be32 get_client_locked(struct nfs4_client *clp)
>>>>   	if (is_client_expired(clp))
>>>>   		return nfserr_expired;
>>>>   	atomic_inc(&clp->cl_rpc_users);
>>>> +	clp->cl_state = NFSD4_ACTIVE;
>>>>   	return nfs_ok;
>>>>   }
>>>> @@ -172,6 +175,7 @@ renew_client_locked(struct nfs4_client *clp)
>>>>   	list_move_tail(&clp->cl_lru, &nn->client_lru);
>>>>   	clp->cl_time = ktime_get_boottime_seconds();
>>>> +	clp->cl_state = NFSD4_ACTIVE;
>>>>   }
>>>>   static void put_client_renew_locked(struct nfs4_client *clp)
>>>> @@ -2004,6 +2008,7 @@ static struct nfs4_client *alloc_client(struct xdr_netobj name)
>>>>   	idr_init(&clp->cl_stateids);
>>>>   	atomic_set(&clp->cl_rpc_users, 0);
>>>>   	clp->cl_cb_state = NFSD4_CB_UNKNOWN;
>>>> +	clp->cl_state = NFSD4_ACTIVE;
>>>>   	INIT_LIST_HEAD(&clp->cl_idhash);
>>>>   	INIT_LIST_HEAD(&clp->cl_openowners);
>>>>   	INIT_LIST_HEAD(&clp->cl_delegations);
>>>> @@ -4694,9 +4699,16 @@ nfsd_break_deleg_cb(struct file_lock *fl)
>>>>   	bool ret = false;
>>>>   	struct nfs4_delegation *dp = (struct nfs4_delegation *)fl->fl_owner;
>>>>   	struct nfs4_file *fp = dp->dl_stid.sc_file;
>>>> +	struct nfs4_client *clp = dp->dl_stid.sc_client;
>>>> +	struct nfsd_net *nn;
>>>>   	trace_nfsd_cb_recall(&dp->dl_stid);
>>>> +	if (!try_to_expire_client(clp)) {
>>>> +		nn = net_generic(clp->net, nfsd_net_id);
>>>> +		mod_delayed_work(laundry_wq, &nn->laundromat_work, 0);
>>>> +	}
>>>> +
>>>>   	/*
>>>>   	 * We don't want the locks code to timeout the lease for us;
>>>>   	 * we'll remove it ourself if a delegation isn't returned
>>>> @@ -5605,6 +5617,65 @@ static void nfsd4_ssc_expire_umount(struct nfsd_net *nn)
>>>>   }
>>>>   #endif
>>>> +/*
>>>> + * place holder for now, no check for lock blockers yet
>>>> + */
>>>> +static bool
>>>> +nfs4_anylock_blockers(struct nfs4_client *clp)
>>>> +{
>>>> +	/*
>>>> +	 * don't want to check for delegation conflict here since
>>>> +	 * we need the state_lock for it. The laundromat willexpire
>>>> +	 * COURTESY later when checking for delegation recall timeout.
>>>> +	 */
>>>> +	return false;
>>>> +}
>>>> +
>>>> +static bool client_has_state_tmp(struct nfs4_client *clp)
>>>> +{
>>>> +	if (!list_empty(&clp->cl_delegations) &&
>>>> +			!client_has_openowners(clp) &&
>>>> +			list_empty(&clp->async_copies))
>>>> +		return true;
>>>> +	return false;
>>>> +}
>>>> +
>>>> +static void
>>>> +nfs4_get_client_reaplist(struct nfsd_net *nn, struct list_head *reaplist,
>>>> +				struct laundry_time *lt)
>>>> +{
>>>> +	struct list_head *pos, *next;
>>>> +	struct nfs4_client *clp;
>>>> +	bool cour;
>>>> +
>>>> +	INIT_LIST_HEAD(reaplist);
>>>> +	spin_lock(&nn->client_lock);
>>>> +	list_for_each_safe(pos, next, &nn->client_lru) {
>>>> +		clp = list_entry(pos, struct nfs4_client, cl_lru);
>>>> +		if (clp->cl_state == NFSD4_EXPIRABLE)
>>>> +			goto exp_client;
>>>> +		if (!state_expired(lt, clp->cl_time))
>>>> +			break;
>>>> +		if (!client_has_state_tmp(clp))
>>>> +			goto exp_client;
>>>> +		cour = (clp->cl_state == NFSD4_COURTESY);
>>>> +		if (cour && ktime_get_boottime_seconds() >=
>>>> +				(clp->cl_time + NFSD_COURTESY_CLIENT_TIMEOUT)) {
>>>> +			goto exp_client;
>>>> +		}
>>>> +		if (nfs4_anylock_blockers(clp)) {
>>>> +exp_client:
>>>> +			if (mark_client_expired_locked(clp))
>>>> +				continue;
>>>> +			list_add(&clp->cl_lru, reaplist);
>>>> +			continue;
>>>> +		}
>>>> +		if (!cour)
>>>> +			cmpxchg(&clp->cl_state, NFSD4_ACTIVE, NFSD4_COURTESY);
>>> I just noticed there's a small race here: a lock conflict (for example)
>>> could intervene between checking nfs4_anylock_blockers and setting
>>> COURTESY.
>> If there is lock conflict intervenes before setting COURTESY then that
>> lock request is denied since the client is ACTIVE. Does NFSv4, NLM
>> client retry the lock request? if it does then on next retry the
>> COURTESY client will be expired.
> I'm thinking of a local request for a blocking lock.  Yes, the request
> will be denied, but then the process will block on the lock forever
> (well, for 24 hours anyway).

got it.

>
>>> I think what you want to do is set COURTESY first--right after you check
>>> state_expired()--instead of doing it at the end.
>> Yes, I can make this change. I think this still has a tiny window
>> where a lock conflict comes in after state_expired and before
>> COURTESY is set?
> No, I think it's OK.  A lock that comes before COURTESY is set will be
> caught by nfs4_anylock_blockers().

That's right.

-Dai

>
> --b.
>
>> -Dai
>>
>>> --b.
>>>
>>>> +	}
>>>> +	spin_unlock(&nn->client_lock);
>>>> +}
>>>> +
>>>>   static time64_t
>>>>   nfs4_laundromat(struct nfsd_net *nn)
>>>>   {
>>>> @@ -5627,7 +5698,6 @@ nfs4_laundromat(struct nfsd_net *nn)
>>>>   		goto out;
>>>>   	}
>>>>   	nfsd4_end_grace(nn);
>>>> -	INIT_LIST_HEAD(&reaplist);
>>>>   	spin_lock(&nn->s2s_cp_lock);
>>>>   	idr_for_each_entry(&nn->s2s_cp_stateids, cps_t, i) {
>>>> @@ -5637,17 +5707,7 @@ nfs4_laundromat(struct nfsd_net *nn)
>>>>   			_free_cpntf_state_locked(nn, cps);
>>>>   	}
>>>>   	spin_unlock(&nn->s2s_cp_lock);
>>>> -
>>>> -	spin_lock(&nn->client_lock);
>>>> -	list_for_each_safe(pos, next, &nn->client_lru) {
>>>> -		clp = list_entry(pos, struct nfs4_client, cl_lru);
>>>> -		if (!state_expired(&lt, clp->cl_time))
>>>> -			break;
>>>> -		if (mark_client_expired_locked(clp))
>>>> -			continue;
>>>> -		list_add(&clp->cl_lru, &reaplist);
>>>> -	}
>>>> -	spin_unlock(&nn->client_lock);
>>>> +	nfs4_get_client_reaplist(nn, &reaplist, &lt);
>>>>   	list_for_each_safe(pos, next, &reaplist) {
>>>>   		clp = list_entry(pos, struct nfs4_client, cl_lru);
>>>>   		trace_nfsd_clid_purged(&clp->cl_clientid);
>>>> @@ -5657,6 +5717,7 @@ nfs4_laundromat(struct nfsd_net *nn)
>>>>   	spin_lock(&state_lock);
>>>>   	list_for_each_safe(pos, next, &nn->del_recall_lru) {
>>>>   		dp = list_entry (pos, struct nfs4_delegation, dl_recall_lru);
>>>> +		try_to_expire_client(dp->dl_stid.sc_client);
>>>>   		if (!state_expired(&lt, dp->dl_time))
>>>>   			break;
>>>>   		WARN_ON(!unhash_delegation_locked(dp));
>>>> @@ -5722,7 +5783,6 @@ nfs4_laundromat(struct nfsd_net *nn)
>>>>   	return max_t(time64_t, lt.new_timeo, NFSD_LAUNDROMAT_MINTIMEOUT);
>>>>   }
>>>> -static struct workqueue_struct *laundry_wq;
>>>>   static void laundromat_main(struct work_struct *);
>>>>   static void
>>>> diff --git a/fs/nfsd/nfsd.h b/fs/nfsd/nfsd.h
>>>> index 4fc1fd639527..23996c6ca75e 100644
>>>> --- a/fs/nfsd/nfsd.h
>>>> +++ b/fs/nfsd/nfsd.h
>>>> @@ -336,6 +336,7 @@ void		nfsd_lockd_shutdown(void);
>>>>   #define COMPOUND_ERR_SLACK_SPACE	16     /* OP_SETATTR */
>>>>   #define NFSD_LAUNDROMAT_MINTIMEOUT      1   /* seconds */
>>>> +#define	NFSD_COURTESY_CLIENT_TIMEOUT	(24 * 60 * 60)	/* seconds */
>>>>   /*
>>>>    * The following attributes are currently not supported by the NFSv4 server:
>>>> diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h
>>>> index 95457cfd37fc..6130376c438b 100644
>>>> --- a/fs/nfsd/state.h
>>>> +++ b/fs/nfsd/state.h
>>>> @@ -283,6 +283,28 @@ struct nfsd4_sessionid {
>>>>   #define HEXDIR_LEN     33 /* hex version of 16 byte md5 of cl_name plus '\0' */
>>>>   /*
>>>> + *       State                Meaning                  Where set
>>>> + * --------------------------------------------------------------------------
>>>> + * | NFSD4_ACTIVE      | Confirmed, active    | Default                     |
>>>> + * |------------------- ----------------------------------------------------|
>>>> + * | NFSD4_COURTESY    | Courtesy state.      | nfs4_get_client_reaplist    |
>>>> + * |                   | Lease/lock/share     |                             |
>>>> + * |                   | reservation conflict |                             |
>>>> + * |                   | can cause Courtesy   |                             |
>>>> + * |                   | client to be expired |                             |
>>>> + * |------------------------------------------------------------------------|
>>>> + * | NFSD4_EXPIRABLE   | Courtesy client to be| nfs4_laundromat             |
>>>> + * |                   | expired by Laundromat| try_to_expire_client        |
>>>> + * |                   | due to conflict      |                             |
>>>> + * |------------------------------------------------------------------------|
>>>> + */
>>>> +enum {
>>>> +	NFSD4_ACTIVE = 0,
>>>> +	NFSD4_COURTESY,
>>>> +	NFSD4_EXPIRABLE,
>>>> +};
>>>> +
>>>> +/*
>>>>    * struct nfs4_client - one per client.  Clientids live here.
>>>>    *
>>>>    * The initial object created by an NFS client using SETCLIENTID (for NFSv4.0)
>>>> @@ -385,6 +407,8 @@ struct nfs4_client {
>>>>   	struct list_head	async_copies;	/* list of async copies */
>>>>   	spinlock_t		async_lock;	/* lock for async copies */
>>>>   	atomic_t		cl_cb_inflight;	/* Outstanding callbacks */
>>>> +
>>>> +	unsigned int		cl_state;
>>>>   };
>>>>   /* struct nfs4_client_reset
>>>> @@ -702,4 +726,12 @@ extern void nfsd4_client_record_remove(struct nfs4_client *clp);
>>>>   extern int nfsd4_client_record_check(struct nfs4_client *clp);
>>>>   extern void nfsd4_record_grace_done(struct nfsd_net *nn);
>>>> +static inline bool try_to_expire_client(struct nfs4_client *clp)
>>>> +{
>>>> +	bool ret;
>>>> +
>>>> +	ret = NFSD4_ACTIVE ==
>>>> +		cmpxchg(&clp->cl_state, NFSD4_COURTESY, NFSD4_EXPIRABLE);
>>>> +	return ret;
>>>> +}
>>>>   #endif   /* NFSD4_STATE_H */
>>>> -- 
>>>> 2.9.5