Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <902ec81f-139f-2b15-42ad-c721dcbe8a75@oracle.com>
Date:   Fri, 29 Apr 2022 10:20:37 -0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.8.1
Subject: Re: [PATCH RFC v23 2/7] NFSD: add support for share reservation
 conflict to courteous server
Content-Language: en-US
To:     "J. Bruce Fields" <bfields@fieldses.org>
Cc:     chuck.lever@oracle.com, jlayton@redhat.com,
        viro@zeniv.linux.org.uk, linux-nfs@vger.kernel.org,
        linux-fsdevel@vger.kernel.org
References: <1651129595-6904-1-git-send-email-dai.ngo@oracle.com>
 <1651129595-6904-3-git-send-email-dai.ngo@oracle.com>
 <20220429143619.GB7107@fieldses.org>
From:   dai.ngo@oracle.com
In-Reply-To: <20220429143619.GB7107@fieldses.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MGZtN2h4L2s1ZVZ3Yk54SlA4V2JHS0drM3NIWU5pS3RkdEY5RmFhaU9BbVhW?=
 =?utf-8?B?MERObHVEd3JHZFBjT1NncGY0QkU1ZDJSVlcxUHNQazFnQ2d0ME9PMGNZbWxS?=
 =?utf-8?B?SnR1dTBQK2xsNlQyUE0xMXFRV0pJVk9yZThYeXlVcEVGTGRDTXR2Nk5tSHdo?=
 =?utf-8?B?b29yY1NDeE1wTEZrRSs5SEpQYzltYnNmd00rNlBYKzZ0N1VQU1luU3pkOXhI?=
 =?utf-8?B?SHRtQmdzQTBHYnZIOXFObnl1OE9vSFp6TVZuTHNISFRVVS9Fam5WeUZNYXhX?=
 =?utf-8?B?MEZrTFlKZkpLNGl3eUdWYWRDSW93M2pUbmpHNGZxRlg4UWRyQlIvaVJVQ2Q4?=
 =?utf-8?B?WDBWczN3TXQrS3U2REVzTXpwRXA2VXZoWlhTVTB4anloN1Q2S0FhVTQrY0l1?=
 =?utf-8?B?K0xhVDRXbU9aenZRcFllZUxzT0hPek5oc1NYeEpZdjFPNytpcUk5ZG5RNzhQ?=
 =?utf-8?B?eXNFNitHbGU1V3paUzY5RUZTbTFTZGhjQUszWDZLTkR6dTQ5ZjVBc21qSUJz?=
 =?utf-8?B?RVJ3WENWRWtHNXk4dTRqR3dSNmF3bXBWT1BadXBKR2tGaVNjaytDb1UrVHg0?=
 =?utf-8?B?RVVlVE8xMHRESVhYLzdaNjF4YUFNT09SSTgxWkNjWDVVRGdGOGxjanN2Si94?=
 =?utf-8?B?R1FUYmlYNG9EYjEzMDNoUUs4ZVd2NDVrRWZ0ZlVhbDhCc3hlTjlmVGhydVlN?=
 =?utf-8?B?aS9BdUpVVUFaU1gwb092R1Qwd1lCUzkrRDFKcUduTWdUT3RlNDg5MkNMcmJJ?=
 =?utf-8?B?U2FaRU52Nlc2YnJWalM5cUF5VUZHeVE4cytETlJHREIvNkVzeDA2aVByQXZB?=
 =?utf-8?B?UzJrWmJKVmwvWncxWFo1M0hFTVdOVjM4L05iR3BHaVdaVlV2TFBvR1dMY0Nr?=
 =?utf-8?B?OFAvd0UxR1NPZDVFcEdSVm41aGxmbEVmb3FtSm56TGY2NHdiS1F5QmZVMjZJ?=
 =?utf-8?B?WTg0L3hQb24ydnl0VjF3S1pUR01iVEs0aWtGdFpyQXpNNnR3NEd1RU1WN0oy?=
 =?utf-8?B?OVJFYjQydHI2NzFTMURSSWYvbVZsR3lQMGwva1Rzek9NcVhjVWw2bm1ZaVNU?=
 =?utf-8?B?RE96bStLZ09RN2VvZGcrRXpSdEpwTC84Y0pBWUhSUXh4TzlYRUZZQzlHV2hn?=
 =?utf-8?B?S2hrRE80cnRaTXB6dnZMd2djMnkrUU4wUzVVSGovcmJtZmNsU2J0SkF3REln?=
 =?utf-8?B?dUp0SUNlVmRzSkt0V0RYK1cvRW5nUGpJd1NYcTBKYit1K25DcDdTcGlzcUVL?=
 =?utf-8?B?Ynp2b0IrYzR4S0hKaWgrOWtFei85WmdJQ3dzLyswbzZHY2VuUm9SdnVjKzEr?=
 =?utf-8?B?N0lHSTN0RTBDbmJQTEdLUEg2TVg1VWhzL2FFUEdONy9CcnR4bDdMSGR3aVNV?=
 =?utf-8?B?eW5ONGgrUVgzOHIwMUkwRVBWVVVodnU4TktubklKREw3UGtqK2tUUHk0dW9z?=
 =?utf-8?B?OE41U2lIVFRZYVdhYmNLWWk0UVZJT2JCVDF0YUF5L2hmbmxsbjdHSWRsL0sr?=
 =?utf-8?B?L3IvOVM2RGVVYjJkenV0SEVpSEgzY3E1Tm45b1lING96WXRzMDlzWTNKUHEv?=
 =?utf-8?B?OFFYTDRxQk1rUm5Zc1VtWWRyV3ZBdWw3MlVmRDVCb01RTUlyUzR6N3ZsZ0VF?=
 =?utf-8?B?cnFaMWR5YmcxKzZXU3VSc2RFU3FMbGc5eHl4YzU3cnpvc3VMc04vKzZ1RnZE?=
 =?utf-8?B?azZoVU42RGNET2tQSGVJQ281WkQ0bGpUOGZiZTE2WFdmSFpRT0NHd2VuUkZG?=
 =?utf-8?B?QlloWjlaS1BLdy9hQklTY3hNMDVZNkRMTXViTkk4WEhwYXppcUhnZnROWTJK?=
 =?utf-8?B?NitkdTN1ektkM3hOSDdoUDNaN216QUJPQm1CRUkyRTErdFFFeTAwVk9pYnVx?=
 =?utf-8?B?YzNqR3JBaVBqZWord2RWZUxZTk16UjJOL2RMayttYnIvNERsdzFDVVV5aDdT?=
 =?utf-8?B?Z1hEb2Y2cS9KQ2kzWG1HUWVsMGZxbW9MenJWTGp6WXl2REgrMnN6bG8wVWY5?=
 =?utf-8?B?ZWw3VklGVGYvcHhGeWVFYWc3SCthRXdKUUltMkdMYm1hWTh2OTljOGM2WEtq?=
 =?utf-8?B?Y2tEQzRIUnk0QkdydkptaTZpdWtzRCtTTVBkNjRSZUFyVWhRTTVSd3c2WDkz?=
 =?utf-8?B?U3p0Y3RsV1FPOEp5UjQ0bWR5QmRuVERaeGRLbFZaT200Vm55TEpGdGRoV25z?=
 =?utf-8?B?Z3ZEV2hHRS9IUXNBRUdDMnZJMkZYNk5PVW5aeFUwVW1tcHg5SE1Bb3FjUEhl?=
 =?utf-8?B?b2c5cENHME1kVlNEd2RSQ2lWdkIwVm9FUjdxMTJPTkxxZTZ5THBKUzJ5WjhJ?=
 =?utf-8?B?UjRYOFRETC95UTFNaUlqOHhsM0FEMlpKczdyS1U4MkJzNUNuamU0QT09?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 44ea59ef-d386-4f0f-bad8-08da2a04965f
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4257.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Apr 2022 17:20:41.9312
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ANFoZPfUdxG09OSLHFYoIrDunrCUcgWlYBXRBwoUQHGZnlkxEkhaCkoKFmyNnzASzml6rHnlDhJ3NF/SLcrBJw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB4019
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.858
 definitions=2022-04-29_06:2022-04-28,2022-04-29 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 adultscore=0
 mlxscore=0 bulkscore=0 suspectscore=0 malwarescore=0 mlxlogscore=999
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000
 definitions=main-2204290087
X-Proofpoint-ORIG-GUID: nFsh-nebDDHbETgI0bK3ccH_0P4RP9it
X-Proofpoint-GUID: nFsh-nebDDHbETgI0bK3ccH_0P4RP9it
X-Spam-Status: No, score=-5.7 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


On 4/29/22 7:36 AM, J. Bruce Fields wrote:
> On Thu, Apr 28, 2022 at 12:06:30AM -0700, Dai Ngo wrote:
>> This patch allows expired client with open state to be in COURTESY
>> state. Share/access conflict with COURTESY client is resolved by
>> setting COURTESY client to EXPIRABLE state, schedule laundromat
>> to run and returning nfserr_jukebox to the request client.
>>
>> Signed-off-by: Dai Ngo <dai.ngo@oracle.com>
>> ---
>>   fs/nfsd/nfs4state.c | 112 ++++++++++++++++++++++++++++++++++++++++++++++++----
>>   1 file changed, 104 insertions(+), 8 deletions(-)
>>
>> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
>> index b84ba19c856b..d2cb820de0ab 100644
>> --- a/fs/nfsd/nfs4state.c
>> +++ b/fs/nfsd/nfs4state.c
>> @@ -694,6 +694,57 @@ static unsigned int file_hashval(struct svc_fh *fh)
>>   
>>   static struct hlist_head file_hashtbl[FILE_HASH_SIZE];
>>   
>> +/*
>> + * Check if courtesy clients have conflicting access and resolve it if possible
>> + *
>> + * access:  is op_share_access if share_access is true.
>> + *	    Check if access mode, op_share_access, would conflict with
>> + *	    the current deny mode of the file 'fp'.
>> + * access:  is op_share_deny if share_access is false.
>> + *	    Check if the deny mode, op_share_deny, would conflict with
>> + *	    current access of the file 'fp'.
>> + * stp:     skip checking this entry.
>> + * new_stp: normal open, not open upgrade.
>> + *
>> + * Function returns:
>> + *	false - access/deny mode conflict with normal client.
>> + *	true  - no conflict or conflict with courtesy client(s) is resolved.
>> + */
>> +static bool
>> +nfs4_resolve_deny_conflicts_locked(struct nfs4_file *fp, bool new_stp,
>> +		struct nfs4_ol_stateid *stp, u32 access, bool share_access)
>> +{
>> +	struct nfs4_ol_stateid *st;
>> +	bool conflict = true;
>> +	unsigned char bmap;
>> +	struct nfsd_net *nn;
>> +	struct nfs4_client *clp;
>> +
>> +	lockdep_assert_held(&fp->fi_lock);
>> +	list_for_each_entry(st, &fp->fi_stateids, st_perfile) {
>> +		/* ignore lock stateid */
>> +		if (st->st_openstp)
>> +			continue;
>> +		if (st == stp && new_stp)
>> +			continue;
>> +		/* check file access against deny mode or vice versa */
>> +		bmap = share_access ? st->st_deny_bmap : st->st_access_bmap;
>> +		if (!(access & bmap_to_share_mode(bmap)))
>> +			continue;
>> +		clp = st->st_stid.sc_client;
>> +		if (try_to_expire_client(clp))
>> +			continue;
>> +		conflict = false;
>> +		break;
>> +	}
>> +	if (conflict) {
>> +		clp = stp->st_stid.sc_client;
>> +		nn = net_generic(clp->net, nfsd_net_id);
>> +		mod_delayed_work(laundry_wq, &nn->laundromat_work, 0);
>> +	}
>> +	return conflict;
> "conflict" is a confusing name for that variable.  Maybe "resolvable"?

fix in v24.

-Dai

>
> --b.
>
>> +}
>> +
>>   static void
>>   __nfs4_file_get_access(struct nfs4_file *fp, u32 access)
>>   {
>> @@ -4959,7 +5010,7 @@ nfsd4_truncate(struct svc_rqst *rqstp, struct svc_fh *fh,
>>   
>>   static __be32 nfs4_get_vfs_file(struct svc_rqst *rqstp, struct nfs4_file *fp,
>>   		struct svc_fh *cur_fh, struct nfs4_ol_stateid *stp,
>> -		struct nfsd4_open *open)
>> +		struct nfsd4_open *open, bool new_stp)
>>   {
>>   	struct nfsd_file *nf = NULL;
>>   	__be32 status;
>> @@ -4975,6 +5026,13 @@ static __be32 nfs4_get_vfs_file(struct svc_rqst *rqstp, struct nfs4_file *fp,
>>   	 */
>>   	status = nfs4_file_check_deny(fp, open->op_share_deny);
>>   	if (status != nfs_ok) {
>> +		if (status != nfserr_share_denied) {
>> +			spin_unlock(&fp->fi_lock);
>> +			goto out;
>> +		}
>> +		if (nfs4_resolve_deny_conflicts_locked(fp, new_stp,
>> +				stp, open->op_share_deny, false))
>> +			status = nfserr_jukebox;
>>   		spin_unlock(&fp->fi_lock);
>>   		goto out;
>>   	}
>> @@ -4982,6 +5040,13 @@ static __be32 nfs4_get_vfs_file(struct svc_rqst *rqstp, struct nfs4_file *fp,
>>   	/* set access to the file */
>>   	status = nfs4_file_get_access(fp, open->op_share_access);
>>   	if (status != nfs_ok) {
>> +		if (status != nfserr_share_denied) {
>> +			spin_unlock(&fp->fi_lock);
>> +			goto out;
>> +		}
>> +		if (nfs4_resolve_deny_conflicts_locked(fp, new_stp,
>> +				stp, open->op_share_access, true))
>> +			status = nfserr_jukebox;
>>   		spin_unlock(&fp->fi_lock);
>>   		goto out;
>>   	}
>> @@ -5028,21 +5093,29 @@ static __be32 nfs4_get_vfs_file(struct svc_rqst *rqstp, struct nfs4_file *fp,
>>   }
>>   
>>   static __be32
>> -nfs4_upgrade_open(struct svc_rqst *rqstp, struct nfs4_file *fp, struct svc_fh *cur_fh, struct nfs4_ol_stateid *stp, struct nfsd4_open *open)
>> +nfs4_upgrade_open(struct svc_rqst *rqstp, struct nfs4_file *fp,
>> +		struct svc_fh *cur_fh, struct nfs4_ol_stateid *stp,
>> +		struct nfsd4_open *open)
>>   {
>>   	__be32 status;
>>   	unsigned char old_deny_bmap = stp->st_deny_bmap;
>>   
>>   	if (!test_access(open->op_share_access, stp))
>> -		return nfs4_get_vfs_file(rqstp, fp, cur_fh, stp, open);
>> +		return nfs4_get_vfs_file(rqstp, fp, cur_fh, stp, open, false);
>>   
>>   	/* test and set deny mode */
>>   	spin_lock(&fp->fi_lock);
>>   	status = nfs4_file_check_deny(fp, open->op_share_deny);
>>   	if (status == nfs_ok) {
>> -		set_deny(open->op_share_deny, stp);
>> -		fp->fi_share_deny |=
>> +		if (status != nfserr_share_denied) {
>> +			set_deny(open->op_share_deny, stp);
>> +			fp->fi_share_deny |=
>>   				(open->op_share_deny & NFS4_SHARE_DENY_BOTH);
>> +		} else {
>> +			if (nfs4_resolve_deny_conflicts_locked(fp, false,
>> +					stp, open->op_share_deny, false))
>> +				status = nfserr_jukebox;
>> +		}
>>   	}
>>   	spin_unlock(&fp->fi_lock);
>>   
>> @@ -5383,7 +5456,7 @@ nfsd4_process_open2(struct svc_rqst *rqstp, struct svc_fh *current_fh, struct nf
>>   			goto out;
>>   		}
>>   	} else {
>> -		status = nfs4_get_vfs_file(rqstp, fp, current_fh, stp, open);
>> +		status = nfs4_get_vfs_file(rqstp, fp, current_fh, stp, open, true);
>>   		if (status) {
>>   			stp->st_stid.sc_type = NFS4_CLOSED_STID;
>>   			release_open_stateid(stp);
>> @@ -5617,12 +5690,35 @@ static void nfsd4_ssc_expire_umount(struct nfsd_net *nn)
>>   }
>>   #endif
>>   
>> +static bool
>> +nfs4_has_any_locks(struct nfs4_client *clp)
>> +{
>> +	int i;
>> +	struct nfs4_stateowner *so;
>> +
>> +	spin_lock(&clp->cl_lock);
>> +	for (i = 0; i < OWNER_HASH_SIZE; i++) {
>> +		list_for_each_entry(so, &clp->cl_ownerstr_hashtbl[i],
>> +				so_strhash) {
>> +			if (so->so_is_open_owner)
>> +				continue;
>> +			spin_unlock(&clp->cl_lock);
>> +			return true;
>> +		}
>> +	}
>> +	spin_unlock(&clp->cl_lock);
>> +	return false;
>> +}
>> +
>>   /*
>>    * place holder for now, no check for lock blockers yet
>>    */
>>   static bool
>>   nfs4_anylock_blockers(struct nfs4_client *clp)
>>   {
>> +	/* not allow locks yet */
>> +	if (nfs4_has_any_locks(clp))
>> +		return true;
>>   	/*
>>   	 * don't want to check for delegation conflict here since
>>   	 * we need the state_lock for it. The laundromat willexpire
>> @@ -5633,8 +5729,8 @@ nfs4_anylock_blockers(struct nfs4_client *clp)
>>   
>>   static bool client_has_state_tmp(struct nfs4_client *clp)
>>   {
>> -	if (!list_empty(&clp->cl_delegations) &&
>> -			!client_has_openowners(clp) &&
>> +	if (((!list_empty(&clp->cl_delegations)) ||
>> +			client_has_openowners(clp)) &&
>>   			list_empty(&clp->async_copies))
>>   		return true;
>>   	return false;
>> -- 
>> 2.9.5