Message-ID: <d0673e79-fa19-3452-0fef-f2c06683037c@oracle.com>
Date:   Fri, 17 Feb 2023 10:22:31 -0800
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.7.2
Subject: Re: [RFC PATCH 1/1] SUNRPC: increase max timeout for rebind to handle
 NFS server restart
Content-Language: en-US
From:   dai.ngo@oracle.com
To:     trondmy@hammerspace.com
Cc:     linux-nfs@vger.kernel.org
References: <1676016656-26195-1-git-send-email-dai.ngo@oracle.com>
In-Reply-To: <1676016656-26195-1-git-send-email-dai.ngo@oracle.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?aGpTSTRuR1BHRmdSTFViUUtKbERSVlEvdDdneXdhVktYMnRtajRuZk9CQlRU?=
 =?utf-8?B?cEUrOEdvN21TczU5QndsaDQ5U2NxbHRwK3NDdzBnY3pIbjdsczIrM1lTdkZx?=
 =?utf-8?B?MkV0SjkxejZKNXFheWEzbHd6UFdtQnNEZEpJcis5RDdta0d0TnBwR0oycDVG?=
 =?utf-8?B?cGkwWFc3UzdvT1RMRm1DcDlxMWxwb29raVhNb3l4YWc1amVUY3prL3lMWVNu?=
 =?utf-8?B?bktPemlZdHV3TGl5alp6UG8zQXVjY0FDVnFOOERxRWszcUQxNDRMdGxHMTE3?=
 =?utf-8?B?ckhaSURoZ2VOUjYxQWdlT0x6UWg0d0ZuK01DQmJYMXg3SnJTTDBjY1lVMTBx?=
 =?utf-8?B?bGxrVUlKMFpnZHAxdDVJQ09pOWR1SVljT1IrTVdqMkJkS3dTWmd2cnpvL2ZH?=
 =?utf-8?B?VXhabXJxR29FQ1Arakl4NzJJMkEwWEI1OENXL244NWNKZXBTM0NobkJDck9B?=
 =?utf-8?B?RjRpa0tDTjRyTWN0TlZqMGIwQnZ3Slp4UWMzdzRTTktsZFF4d1NUbzVwQkU3?=
 =?utf-8?B?eHh4cCtDYytvZnM1ZjY3MXRRYWFFZEVMa2JIaWF0RW5KekVOci9wYW9KOUd0?=
 =?utf-8?B?YWYxUFlEVHVZaW4vcWNFVWhIRUcreEo2NVVXVldoaThTQ0dMTC9XOFRCL3dT?=
 =?utf-8?B?RHhsUGEzakJEMUJ0RHltZWkrOWM4N1hBTGpzZ05Hcnl1QXdrYzdJQ0hhcGxR?=
 =?utf-8?B?b2xTeGRBT01rdGQybGdNaGszWVJWbGJPTk9CS1lRSFkyNHdWM2VFemcxQ2dv?=
 =?utf-8?B?ME1PWVJZaGt1SWRHalhvRkY3T3ZNTDFFOEYwUkxkNjc3djlHTDFwQjk2RE9Y?=
 =?utf-8?B?QXpWUml4OENlcUQvb3B6RUFlaHN4L2xTSnp2T0RzS0IwT2VFMk9VQUMzL0l5?=
 =?utf-8?B?cUpnczQyaGVXSllnSWxRN3FPM2QxWGpCd1pQTUdWZ3JLN3JGOWcrMmhxdWh1?=
 =?utf-8?B?ck12SWlaWDZBVVJTRDRaQkE3VmNBTW95d0lNQkFLeVovd0tYZjlmMS8wTVU3?=
 =?utf-8?B?VzFiOGlESjZjai81c1RndzNrSFZjYkR0bk52OEJYc0FBc2Q3TE96WlJkR3Za?=
 =?utf-8?B?aVd0QnNiazhzU284WnF5UWphTE42Rkp0VVhseHdpZUF4VGp6eFdZUDZncTF2?=
 =?utf-8?B?OFRmcmRIYjBOSVIyVVZybTIyblJFMkx1dFQ3cS92RTRNSWdBSUF5Z00wUTRM?=
 =?utf-8?B?S0JOSGFQRjFrbEFuNFUxSmhPU21iTHE3RE93aU8xd3VFcFMvTUFhZjZCd1NM?=
 =?utf-8?B?eDkwRG1ScWx4TE5wQ0x5RUhib3FnNkhGVGUvSmI1bkJIdVo4WVBoeFhFalIx?=
 =?utf-8?B?bytZYldUbVVPZysvc3JkVTZpYXJJNXdKVzROSzR2UkZXK3dVSURCRzBqYXFV?=
 =?utf-8?B?VWdZaDViMFFBUE1YVlQzazh1S0oySldVQng1NllFUFVHSnJIRGk5NmFFNmpW?=
 =?utf-8?B?NWtnSGVaNk1TRzNDaTlyRzF0aEZXMjhYMWlZdWdQN1E5YU8xVGtGZ0dxOE9u?=
 =?utf-8?B?NGMvMXFZWkYrdVVCQzR3WWpTUElxOGxESXQyNGd0L0xJdXV2L2pDT2NYVWV0?=
 =?utf-8?B?Ylh4b0FucG8wSlI5Y2hoTDVSN3YyVEErc2kwd3p0aUo4U2dISnp6Y3JNa3R4?=
 =?utf-8?B?UzRhQVovWTNnN0dlc1BGRmo2dGNIZVhkWUdkUlAvQUo4VHR5M3oxN1A1TW93?=
 =?utf-8?B?OUU4T0JrMWFGR2tNRzE1ZFdwUUxGaDVhMVN2Wm1YRVlQclgveXMvdmNHdUFq?=
 =?utf-8?B?UllncEx6MjloVEFieHZnUjhTeTRkQW45WmlGenREK3NnSGlielBiRlZFb3hz?=
 =?utf-8?B?WDdUQ1RsY0hOR0NIRWFjck0xYTcycGZVTGVuUVRMRWNrMU92dnJmb3hBWTNv?=
 =?utf-8?B?Z0h6eWhVVW5IWXlSa0ozUm1za3A3WEpIZzFqM0x0RWJzVXNPTDZQeXFiaC9M?=
 =?utf-8?B?Rjh4QjBYNXgzTjU2MnIxWmdIeGVLWVFiMDBiWkc1cDdPbUcyUVpIVGkxWHpX?=
 =?utf-8?B?bklIZHZwTnJtdTA3RHVvb3l4OEo0cTV4VHVOZmg4VDJNUlNmZmMrYVNLdjhU?=
 =?utf-8?B?c2JhSHRVVUpEeG1ZQkUraG91NjJpVXJBNGt1ajJGSjVuMzVMVzdCNzdUVTdM?=
 =?utf-8?Q?e7lvlrfRi+TZC/9Tx3IwqKHCQ?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: AWYHwWZlU4n23qUVoOeJm0i3oVGxM2ZDM6nK6PWMYV7uWw3jBfFIWbLpkHHKgwNRLnjeBV0jAWTL7yowd/sgQADxWRamVsLS9FMIlzorLJB0v7Nln6FjL8e4PArycbmtAmyr3u2EeQVVe8G9AOg/S0BNxBCvCVGYWQUFfOXEPUkhOg2NQQmlxLnV/5v/6fH/Yv9PDTBF9A29AVgje6GTN93Q1ISEoECWc7MueBzxJpE37Kpfy/saPcezhpcXNjJTjq81NAxfH2j70gi59nTbo2eyrNwNUJnpf5P81YxT/0FVmI96rsLd9NYUA7mdfZugk3JO+UgoMPP4yex9/VWNbtTzB2YUfo+f4gcA9xv/Mgs3cYqu6Y6QqfEAhmzK0ZkUc4xwyZhynwQzAjTboGKdHronNVMvR4HyBpn7y4hH4nkw0Kw2o5audXpe5/Pqca6wz60RwZ2TqZPnvTm3sn7bCDtzE9aalCeV0s/jtkzBO7fi5+/Pl7chWcJAoj2LW+qaHyB62oveBKkdCLOMcjHEHIeSp24o5f2XLzO/m4arZzKk2CKrJU7jfflEqziADrsY/wkhJGmYzMnKhVwbFV/h/BNxBX1/RKNOpQq4bHlgBDypvV7b1AWC75oeEfRrmg2xDbZYbc6eDMxB/QOu+NBF0a2WvEGyoS6Wj+V7fuUGem8l0ne5ncTSVUBsxDZjudp9FFoBtoLYG4vih7fx3zo2eMnEkKHYztcDQVag8jyIGcwMY9Exh5tEdSlvUpzP/PGNYqJb9dGP2iVqERoAesmlI7byVUZoWS+b6/F+sUCXhwci9GmZUhy7n8wDwiaxdO8E
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 85b1c053-135b-42b4-e544-08db1113f05d
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4257.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Feb 2023 18:22:33.9824
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ZTb8p3NwVtRVNaTke2MT9fPpmfEnCeVJByflvhTMkWiSaMjj4VOR0wwNpBFVxQAlHYNc6ebpDCl1kg+aCUhWlw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA3PR10MB7093
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22
 definitions=2023-02-17_12,2023-02-17_01,2023-02-09_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 suspectscore=0
 phishscore=0 spamscore=0 bulkscore=0 mlxlogscore=999 malwarescore=0
 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2212070000 definitions=main-2302170162
X-Proofpoint-GUID: loIsPAKvvytPVl426t5TMXfCRoJsCVJf
X-Proofpoint-ORIG-GUID: loIsPAKvvytPVl426t5TMXfCRoJsCVJf
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

Hi Trond,

Could you please let me know your opinion on this patch?

Thanks,
-Dai

On 2/10/23 12:10 AM, Dai Ngo wrote:
> Occasionally NLM lock and unlock request fail with EIO and ENOLCK
> respectively. This usually happens when the NFS server is restarted
> while NLM lock test is running.
>
> Currently there is a 9 seconds limit for retrying the bind operation.
> If the server is under load the port mapper might take more than 9
> seconds to become ready after the NFS server restarted.
>
> This patch increases the timeout for rebind from 9 to 30 seconds
> allowing a bit more time for the port mapper to become ready.
>
> Signed-off-by: Dai Ngo <dai.ngo@oracle.com>
> ---
>   include/linux/sunrpc/clnt.h  | 3 +++
>   include/linux/sunrpc/sched.h | 4 ++--
>   net/sunrpc/clnt.c            | 2 +-
>   net/sunrpc/sched.c           | 3 ++-
>   4 files changed, 8 insertions(+), 4 deletions(-)
>
> diff --git a/include/linux/sunrpc/clnt.h b/include/linux/sunrpc/clnt.h
> index 770ef2cb5775..7f2dee56c121 100644
> --- a/include/linux/sunrpc/clnt.h
> +++ b/include/linux/sunrpc/clnt.h
> @@ -162,6 +162,9 @@ struct rpc_add_xprt_test {
>   #define RPC_CLNT_CREATE_REUSEPORT	(1UL << 11)
>   #define RPC_CLNT_CREATE_CONNECTED	(1UL << 12)
>   
> +#define	RPC_CLNT_REBIND_DELAY		3
> +#define	RPC_CLNT_REBIND_MAX_TIMEOUT	30
> +
>   struct rpc_clnt *rpc_create(struct rpc_create_args *args);
>   struct rpc_clnt	*rpc_bind_new_program(struct rpc_clnt *,
>   				const struct rpc_program *, u32);
> diff --git a/include/linux/sunrpc/sched.h b/include/linux/sunrpc/sched.h
> index b8ca3ecaf8d7..e9dc142f10bb 100644
> --- a/include/linux/sunrpc/sched.h
> +++ b/include/linux/sunrpc/sched.h
> @@ -90,8 +90,8 @@ struct rpc_task {
>   #endif
>   	unsigned char		tk_priority : 2,/* Task priority */
>   				tk_garb_retry : 2,
> -				tk_cred_retry : 2,
> -				tk_rebind_retry : 2;
> +				tk_cred_retry : 2;
> +	unsigned char		tk_rebind_retry;
>   };
>   
>   typedef void			(*rpc_action)(struct rpc_task *);
> diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
> index 0b0b9f1eed46..6c89a1fa40bf 100644
> --- a/net/sunrpc/clnt.c
> +++ b/net/sunrpc/clnt.c
> @@ -2053,7 +2053,7 @@ call_bind_status(struct rpc_task *task)
>   		if (task->tk_rebind_retry == 0)
>   			break;
>   		task->tk_rebind_retry--;
> -		rpc_delay(task, 3*HZ);
> +		rpc_delay(task, RPC_CLNT_REBIND_DELAY * HZ);
>   		goto retry_timeout;
>   	case -ENOBUFS:
>   		rpc_delay(task, HZ >> 2);
> diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c
> index be587a308e05..5c18a35752aa 100644
> --- a/net/sunrpc/sched.c
> +++ b/net/sunrpc/sched.c
> @@ -817,7 +817,8 @@ rpc_init_task_statistics(struct rpc_task *task)
>   	/* Initialize retry counters */
>   	task->tk_garb_retry = 2;
>   	task->tk_cred_retry = 2;
> -	task->tk_rebind_retry = 2;
> +	task->tk_rebind_retry = RPC_CLNT_REBIND_MAX_TIMEOUT /
> +					RPC_CLNT_REBIND_DELAY;
>   
>   	/* starting timestamp */
>   	task->tk_start = ktime_get();