Received-SPF: pass (google.com: domain of linux-nfs+bounces-778-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Date: Fri, 22 Dec 2023 10:01:48 -0500
From: Chuck Lever <chuck.lever@oracle.com>
To: NeilBrown <neilb@suse.de>
Cc: Jeff Layton <jlayton@kernel.org>, linux-nfs@vger.kernel.org,
        Olga Kornievskaia <kolga@netapp.com>, Dai Ngo <Dai.Ngo@oracle.com>,
        Tom Talpey <tom@talpey.com>
Subject: Re: [PATCH 11/11] nfsd: allow layout state to be admin-revoked.
Message-ID: <ZYWk3KBmZs28QQKY@tissot.1015granger.net>
References: <20231124002925.1816-1-neilb@suse.de>
 <20231124002925.1816-12-neilb@suse.de>
 <ZWS1B2qGiu2SqARc@tissot.1015granger.net>
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <ZWS1B2qGiu2SqARc@tissot.1015granger.net>
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?us-ascii?Q?WXDqy/JZmDRejRblFwkgzAZITkUH3hEkOojpegybohCSHf8/Ufl95sjbQzTX?=
 =?us-ascii?Q?KelFV7FiwXyv/UNq3SZzHHc7e+kUZWBAPnrzyNGkG8DEflcVuEMETm7LG97v?=
 =?us-ascii?Q?EMUHrZrVTvPz3v6cDkg3Nfy5kuXeuKkJaRF0drm1yShZzTiK5zMlcHFXfz67?=
 =?us-ascii?Q?RYUk9mXuaJ1dqNsN1oqnz7P2pW6A7PFiNNxeBdFE8leBULFUNL8Mc6ExZ/md?=
 =?us-ascii?Q?fKJaxnYdj7VgQ3GzcPI02GuDs6/9dSYIAc6lsjgcRIbTQ+fgItLDUL3J6y2r?=
 =?us-ascii?Q?ysLtMPfigytjGggTV3L0gqwV5/ITNYw7YRSuFU85WJiBKsa+FPdTmvK11b3H?=
 =?us-ascii?Q?uj4IOycrJanapQhZyRm6/6pkTqyR/8qDnZQ/xGkcLOVTMij0hQmqbXum6sdZ?=
 =?us-ascii?Q?V1dkqp2g72v88tbBgiKv/u/vXbyFyC7zpLsNJFR55/bBfjQBAQ7Xc+iyLQ4G?=
 =?us-ascii?Q?rmW6+Y+hO7gqhhnb0rtwlP0eMDhWq4GeH8BumYFvqQREfOJGR7mKXVW2cXYk?=
 =?us-ascii?Q?pObyKgIIuYcSj9jED2pZt0nhYNa0pg+CMyz/NCpfBGsBoIPfZc518qxfapdo?=
 =?us-ascii?Q?uPtPG2zsTEDXLoTL4+ORByhWWowk9nIta1e9x9WackSlmxBZIlmd0gkwmuWX?=
 =?us-ascii?Q?QDtYw0IuRc2XM0DO5w04cP5agoj80MNjC0G4tSSCccI56M9dGiUGFjfMR4NX?=
 =?us-ascii?Q?ZFJ12rI9L88aJarYiGo2E6SR6qWHa9W4Og6PztCBlqu1Arj0/D7Xz1IOZvVr?=
 =?us-ascii?Q?eERek3AmzPfyuKB4CmoxsQumr4ZUDy0jE6J1v0MYYFkv8cOvXe2cYeN35HN6?=
 =?us-ascii?Q?3mX95QtEwg42cHo6NZGIIDeyXdfbhymHI2AVxsTAt8ZTgTyxD9lzRgCI33Gr?=
 =?us-ascii?Q?WMd/XE9kcundFCYwD1tXcV9ujDShHKon9v2dg5eBtAUbuUXmzjYW5PLCK1yN?=
 =?us-ascii?Q?KV7h2nmrl8mZaaR0eP53a3a3/wdHx2yF6dtvEGq7lHpjGHpCadkwmJUOCIzP?=
 =?us-ascii?Q?vpfVUxabr6EJ8Rk6FbztdGi4dOptjoqu01jt0MSwHcu9b1f010IGa8zKWRRq?=
 =?us-ascii?Q?G5s2ayNUXtQUb3QEQF3vQZ/We22cVR11rQUfie6TUPZqoz7MqsfNvv3Pkw4n?=
 =?us-ascii?Q?iCrzMNZR+2vY//1mxCDU4yj+ckn94ulShw9XffSqnT+97CjEnoNIeOVtH6Xq?=
 =?us-ascii?Q?Qq3BgkmezCggkzomeGKslFWzXf8AiDq9EYZeSsmOQKdGqWcShucrKAk5WNuq?=
 =?us-ascii?Q?89Ve5+xVxBMtaHsRkfykFb53zjDM77Es+r2gXBdd1cSOa2NIgvlwI4AeOyJ8?=
 =?us-ascii?Q?xTx5pzUF/Bxuytu7vob23GKn4i6doKohZDL9iWuwuh32FKl7Fw1R0uWU7Xr+?=
 =?us-ascii?Q?u2QHjne2OCuUbpBGbXf743fGsHJRgv5XqlwDDDQL2eHJlxnRgf2MR6qBPShg?=
 =?us-ascii?Q?8+Ua774D0GnO9r30ow26ibJRJ/3rv19dy9s6gPCvi/1mc8kEJ7ktjBMNZoG1?=
 =?us-ascii?Q?ljJxOiauGsZdRt6pYtzAeK2ZKCPhWK4ExhaO+NA9IRRyCwzZgVe8cBu4iBaT?=
 =?us-ascii?Q?u+ExiGrzZ+g1bMPUSyIO1fO3dWPTLWRotIWTpp9xO4dLOT4PBVWx+Ur/dgMH?=
 =?us-ascii?Q?5A=3D=3D?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	0j39tYvlM1cwWHW7lGZkjf4C5sj8k9nBYZPt2AsjdHWYgQb8EinFScSnOoCNgjablbef0/Rw4Inw0C2DbxIcq7BfAgTu50iIgeKj4m4HEYPIWInot/J7qoOLRJ6er3WmHDmrOKMQ1e3N3WBeV2XjUpUkaHac3dOjUP1rsTW2GOmzs12FN93lQwLOuoFVD+7am2WRpcbFqp/zvzgkGcOpY3LdiQxtbAa4BRQSFz/s/MdyG4sTjNx8sKbhXVwiBqg9off5JBH4EgGPANo6C2oLXffVmWP5lM9mououl2Ot2dOQ/flZb0mlsL8gj+yXtiWpuor04COkpv/0BUGUJgLtUhv0ArvFxU4H9YAq3/dT/oBnUkY5U97eGzAqZ4TvU95MeVZ0Hmvp2uEP1MV/eXAgrzDCHVQdwtqAEOuAfkVwiD4mP7EyM6Z3hs8DssYRiOJYSQVZ98amfzW59tOEZWtY3fzwRslRuInX7yO/n3AVYUOy7y6vI8XU0m0vRSyVs5XzqxFqVP697zDb844YIH5cBHWQYA2+QNGHo2NtVlb3HwWi3AGq5leRvX6iJrcQ+JQfRi/EnijrU+d94e8C6VCDS3gNvcAUYAbpqQbkrOE8UMQ=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 53fbafd8-8c64-4229-cbbd-08dc02feed1e
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Dec 2023 15:01:50.4074
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: cLQhJQkdSRZu+o/I7P0jhVybYEFSCBrcW/RW02IAAil3e0q/nJmU+4RWmWvHBF8aMdcbA9VdkWyOX/3UTVjT1A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB7108
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2023-12-22_08,2023-12-21_02,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 adultscore=0
 malwarescore=0 phishscore=0 mlxscore=0 bulkscore=0 spamscore=0
 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311290000 definitions=main-2312220110
X-Proofpoint-ORIG-GUID: qY_k_R5E_IbSCa746cl1Zzh39gC45odB
X-Proofpoint-GUID: qY_k_R5E_IbSCa746cl1Zzh39gC45odB

On Mon, Nov 27, 2023 at 10:25:59AM -0500, Chuck Lever wrote:
> On Fri, Nov 24, 2023 at 11:28:46AM +1100, NeilBrown wrote:
> > When there is layout state on a filesystem that is being "unlocked" that
> > is now revoked, which involves closing the nfsd_file and releasing the
> > vfs lease.
> > 
> > To avoid races, all users of ->ls_file - after the layout state has been
> > successfully created - now need to take a counted reference under
> > rcu_read_lock().  To support this, ->fence_client and
> > nfsd4_cb_layout_fail() now take a second argument being the nfsd_file.
> > 
> > Signed-off-by: NeilBrown <neilb@suse.de>
> 
> Hi Neil, would you Cc: Christoph Hellwig <hch@lst.de> and Tom Haynes
> <loghyr@gmail.com> to this patch next time you post this series?

Re-visiting. Did you send out a v5 of this series and I missed it?


> > ---
> >  fs/nfsd/blocklayout.c |  4 ++--
> >  fs/nfsd/nfs4layouts.c | 38 +++++++++++++++++++++++++++-----------
> >  fs/nfsd/nfs4state.c   | 28 +++++++++++++++++++++-------
> >  fs/nfsd/pnfs.h        |  7 ++++++-
> >  4 files changed, 56 insertions(+), 21 deletions(-)
> > 
> > diff --git a/fs/nfsd/blocklayout.c b/fs/nfsd/blocklayout.c
> > index 46fd74d91ea9..3c040c81c77d 100644
> > --- a/fs/nfsd/blocklayout.c
> > +++ b/fs/nfsd/blocklayout.c
> > @@ -328,10 +328,10 @@ nfsd4_scsi_proc_layoutcommit(struct inode *inode,
> >  }
> >  
> >  static void
> > -nfsd4_scsi_fence_client(struct nfs4_layout_stateid *ls)
> > +nfsd4_scsi_fence_client(struct nfs4_layout_stateid *ls, struct nfsd_file *file)
> >  {
> >  	struct nfs4_client *clp = ls->ls_stid.sc_client;
> > -	struct block_device *bdev = ls->ls_file->nf_file->f_path.mnt->mnt_sb->s_bdev;
> > +	struct block_device *bdev = file->nf_file->f_path.mnt->mnt_sb->s_bdev;
> >  
> >  	bdev->bd_disk->fops->pr_ops->pr_preempt(bdev, NFSD_MDS_PR_KEY,
> >  			nfsd4_scsi_pr_key(clp), 0, true);
> > diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
> > index 77656126ad2a..dbc52413ce57 100644
> > --- a/fs/nfsd/nfs4layouts.c
> > +++ b/fs/nfsd/nfs4layouts.c
> > @@ -152,6 +152,18 @@ void nfsd4_setup_layout_type(struct svc_export *exp)
> >  #endif
> >  }
> >  
> > +void nfsd4_close_layout(struct nfs4_layout_stateid *ls)
> > +{
> > +	struct nfsd_file *fl = xchg(&ls->ls_file, NULL);
> > +
> > +	if (fl) {
> > +		if (!nfsd4_layout_ops[ls->ls_layout_type]->disable_recalls)
> > +			vfs_setlease(fl->nf_file, F_UNLCK, NULL,
> > +				     (void **)&ls);
> > +		nfsd_file_put(fl);
> > +	}
> > +}
> > +
> >  static void
> >  nfsd4_free_layout_stateid(struct nfs4_stid *stid)
> >  {
> > @@ -169,9 +181,7 @@ nfsd4_free_layout_stateid(struct nfs4_stid *stid)
> >  	list_del_init(&ls->ls_perfile);
> >  	spin_unlock(&fp->fi_lock);
> >  
> > -	if (!nfsd4_layout_ops[ls->ls_layout_type]->disable_recalls)
> > -		vfs_setlease(ls->ls_file->nf_file, F_UNLCK, NULL, (void **)&ls);
> > -	nfsd_file_put(ls->ls_file);
> > +	nfsd4_close_layout(ls);
> >  
> >  	if (ls->ls_recalled)
> >  		atomic_dec(&ls->ls_stid.sc_file->fi_lo_recalls);
> > @@ -605,7 +615,7 @@ nfsd4_return_all_file_layouts(struct nfs4_client *clp, struct nfs4_file *fp)
> >  }
> >  
> >  static void
> > -nfsd4_cb_layout_fail(struct nfs4_layout_stateid *ls)
> > +nfsd4_cb_layout_fail(struct nfs4_layout_stateid *ls, struct nfsd_file *file)
> >  {
> >  	struct nfs4_client *clp = ls->ls_stid.sc_client;
> >  	char addr_str[INET6_ADDRSTRLEN];
> > @@ -627,7 +637,7 @@ nfsd4_cb_layout_fail(struct nfs4_layout_stateid *ls)
> >  
> >  	argv[0] = (char *)nfsd_recall_failed;
> >  	argv[1] = addr_str;
> > -	argv[2] = ls->ls_file->nf_file->f_path.mnt->mnt_sb->s_id;
> > +	argv[2] = file->nf_file->f_path.mnt->mnt_sb->s_id;
> >  	argv[3] = NULL;
> >  
> >  	error = call_usermodehelper(nfsd_recall_failed, argv, envp,
> > @@ -657,6 +667,7 @@ nfsd4_cb_layout_done(struct nfsd4_callback *cb, struct rpc_task *task)
> >  	struct nfsd_net *nn;
> >  	ktime_t now, cutoff;
> >  	const struct nfsd4_layout_ops *ops;
> > +	struct nfsd_file *fl;
> >  
> >  	trace_nfsd_cb_layout_done(&ls->ls_stid.sc_stateid, task);
> >  	switch (task->tk_status) {
> > @@ -688,12 +699,17 @@ nfsd4_cb_layout_done(struct nfsd4_callback *cb, struct rpc_task *task)
> >  		 * Unknown error or non-responding client, we'll need to fence.
> >  		 */
> >  		trace_nfsd_layout_recall_fail(&ls->ls_stid.sc_stateid);
> > -
> > -		ops = nfsd4_layout_ops[ls->ls_layout_type];
> > -		if (ops->fence_client)
> > -			ops->fence_client(ls);
> > -		else
> > -			nfsd4_cb_layout_fail(ls);
> > +		rcu_read_lock();
> > +		fl = nfsd_file_get(ls->ls_file);
> > +		rcu_read_unlock();
> > +		if (fl) {
> > +			ops = nfsd4_layout_ops[ls->ls_layout_type];
> > +			if (ops->fence_client)
> > +				ops->fence_client(ls, fl);
> > +			else
> > +				nfsd4_cb_layout_fail(ls, fl);
> > +			nfsd_file_put(fl);
> > +		}
> >  		return 1;
> >  	case -NFS4ERR_NOMATCHING_LAYOUT:
> >  		trace_nfsd_layout_recall_done(&ls->ls_stid.sc_stateid);
> > diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
> > index 3d85c88ec4d7..d82ca209eb96 100644
> > --- a/fs/nfsd/nfs4state.c
> > +++ b/fs/nfsd/nfs4state.c
> > @@ -1712,7 +1712,8 @@ void nfsd4_revoke_states(struct net *net, struct super_block *sb)
> >  	unsigned int idhashval;
> >  	unsigned int sc_types;
> >  
> > -	sc_types = NFS4_OPEN_STID | NFS4_LOCK_STID | NFS4_DELEG_STID;
> > +	sc_types = (NFS4_OPEN_STID | NFS4_LOCK_STID |
> > +		    NFS4_DELEG_STID | NFS4_LAYOUT_STID);
> >  
> >  	spin_lock(&nn->client_lock);
> >  	for (idhashval = 0; idhashval < CLIENT_HASH_MASK; idhashval++) {
> > @@ -1725,6 +1726,7 @@ void nfsd4_revoke_states(struct net *net, struct super_block *sb)
> >  			if (stid) {
> >  				struct nfs4_ol_stateid *stp;
> >  				struct nfs4_delegation *dp;
> > +				struct nfs4_layout_stateid *ls;
> >  
> >  				spin_unlock(&nn->client_lock);
> >  				switch (stid->sc_type) {
> > @@ -1780,6 +1782,10 @@ void nfsd4_revoke_states(struct net *net, struct super_block *sb)
> >  					if (dp)
> >  						revoke_delegation(dp);
> >  					break;
> > +				case NFS4_LAYOUT_STID:
> > +					ls = layoutstateid(stid);
> > +					nfsd4_close_layout(ls);
> > +					break;
> >  				}
> >  				nfs4_put_stid(stid);
> >  				spin_lock(&nn->client_lock);
> > @@ -2859,17 +2865,25 @@ static int nfs4_show_layout(struct seq_file *s, struct nfs4_stid *st)
> >  	struct nfsd_file *file;
> >  
> >  	ls = container_of(st, struct nfs4_layout_stateid, ls_stid);
> > -	file = ls->ls_file;
> > +	rcu_read_lock();
> > +	file = nfsd_file_get(ls->ls_file);
> > +	rcu_read_unlock();
> >  
> > -	seq_printf(s, "- ");
> > +	seq_puts(s, "- ");
> >  	nfs4_show_stateid(s, &st->sc_stateid);
> > -	seq_printf(s, ": { type: layout, ");
> > +	seq_puts(s, ": { type: layout");
> >  
> >  	/* XXX: What else would be useful? */
> >  
> > -	nfs4_show_superblock(s, file);
> > -	seq_printf(s, ", ");
> > -	nfs4_show_fname(s, file);
> > +	if (file) {
> > +		seq_puts(s, ", ");
> > +		nfs4_show_superblock(s, file);
> > +		seq_puts(s, ", ");
> > +		nfs4_show_fname(s, file);
> > +		nfsd_file_put(file);
> > +	}
> > +	if (st->sc_status & NFS4_STID_ADMIN_REVOKED)
> > +		seq_puts(s, ", admin-revoked");
> >  	seq_printf(s, " }\n");
> >  
> >  	return 0;
> > diff --git a/fs/nfsd/pnfs.h b/fs/nfsd/pnfs.h
> > index de1e0dfed06a..f2777577865e 100644
> > --- a/fs/nfsd/pnfs.h
> > +++ b/fs/nfsd/pnfs.h
> > @@ -37,7 +37,8 @@ struct nfsd4_layout_ops {
> >  	__be32 (*proc_layoutcommit)(struct inode *inode,
> >  			struct nfsd4_layoutcommit *lcp);
> >  
> > -	void (*fence_client)(struct nfs4_layout_stateid *ls);
> > +	void (*fence_client)(struct nfs4_layout_stateid *ls,
> > +			     struct nfsd_file *file);
> >  };
> >  
> >  extern const struct nfsd4_layout_ops *nfsd4_layout_ops[];
> > @@ -72,6 +73,7 @@ void nfsd4_setup_layout_type(struct svc_export *exp);
> >  void nfsd4_return_all_client_layouts(struct nfs4_client *);
> >  void nfsd4_return_all_file_layouts(struct nfs4_client *clp,
> >  		struct nfs4_file *fp);
> > +void nfsd4_close_layout(struct nfs4_layout_stateid *ls);
> >  int nfsd4_init_pnfs(void);
> >  void nfsd4_exit_pnfs(void);
> >  #else
> > @@ -89,6 +91,9 @@ static inline void nfsd4_return_all_file_layouts(struct nfs4_client *clp,
> >  		struct nfs4_file *fp)
> >  {
> >  }
> > +static inline void nfsd4_close_layout(struct nfs4_layout_stateid *ls)
> > +{
> > +}
> >  static inline void nfsd4_exit_pnfs(void)
> >  {
> >  }
> > -- 
> > 2.42.1
> > 
> 
> -- 
> Chuck Lever
> 

-- 
Chuck Lever