Message-ID: <3165b445-f4da-3dfd-7036-712b605865be@oracle.com>
Date:   Wed, 22 Feb 2023 21:40:33 -0800
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.8.0
Subject: Re: [RFC PATCH 1/1] SUNRPC: increase max timeout for rebind to handle
 NFS server restart
Content-Language: en-US
From:   dai.ngo@oracle.com
To:     Anna Schumaker <anna@kernel.org>
Cc:     linux-nfs@vger.kernel.org,
        Trond Myklebust <trondmy@hammerspace.com>
References: <1676016656-26195-1-git-send-email-dai.ngo@oracle.com>
 <d0673e79-fa19-3452-0fef-f2c06683037c@oracle.com>
In-Reply-To: <d0673e79-fa19-3452-0fef-f2c06683037c@oracle.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ek5hWUN0M1BOZ1lyVWYxbkE0VVNOS0VyTVpZY2JPTG5TeUpSNE1YRkJjMW9l?=
 =?utf-8?B?L1NKSjdqTStrMGRtUEFWcVVlN0gvREh4ZkUrT08rNEc4dERodUFFVWx1SFd6?=
 =?utf-8?B?QWgzK2lQRk81REViR2JTMWsrZm9qWk1mZ1AzMkhQcUs1cjlyRTlINTVvbzRv?=
 =?utf-8?B?aWJIcVF0SUo3cE11U0pIc2p3c2huNndXNU9vWEVaQTloTlhlcU5EOWZzR1M5?=
 =?utf-8?B?c3RzbXh6WDJnZ2pXOEVsaG5ON2VDRkdDdnZ2NzhLOVlYZ2I5TGdQM1NMem5x?=
 =?utf-8?B?Z0FaWS9yM2R2ZTZiZEdxSVM1c1BFODFKU1crUzhlcnA2RnJURUFud1RhTUJH?=
 =?utf-8?B?VFRmYWJWUDMwUWp6NXFyR3lDaXJLWE5lbDl3anJxQ3pGOWtmV0tPUFZXTTdz?=
 =?utf-8?B?MTQzWVZUaFRncXZVNUE1T1hwZU1kdHVSaE5uZnNmTDlRMUNXSHdCazJNOWs0?=
 =?utf-8?B?bUNnSDRsZjkzeGZ1MjM5YTlOcXdVZlQvMDFxZHh4d2c1T04xRzlLblFvZ2Np?=
 =?utf-8?B?Zm91Y09YZDVUT1Y5VlBEc0hBTjZhUWJscVRxZHFSd2p2blV5QzZhbEJJb3VC?=
 =?utf-8?B?MUJnZGZPOVh3THpzSzF6aVVaSi9xTDNEbFdJU0VTMStQT2RBbEZxSTNYemhy?=
 =?utf-8?B?WlgrZG1VdUExRjFDTDFMRlJaSUQrcEJJMDdvRmVZMVI1YUt0R0NNV0l2RDU2?=
 =?utf-8?B?SEgwcmtrSVZUN1JJUVVGNmw1TUJWcytBb2NtcnlWalhMeEhMOWd6TlU2dkhZ?=
 =?utf-8?B?MTZ1d2JueE0reU9WSHVhQkhhS0RtUVlHQXliZWNyQzN1MzNKQTJBQ1NwRzBX?=
 =?utf-8?B?bE5QUXl0d21PNU1RNC9RRy9IOFBhN0RndUttUFhodlJQMFZBV3JsNFJGUkc2?=
 =?utf-8?B?eHJpdkQ1UXhoaXZTSUQ5c2FCNFNRT0NPMlFYQ3Y4MkR5WHJpWXlMR1RSRzI3?=
 =?utf-8?B?cm1OUHpkblg3cE5OL01EQ3FjMytqck0vYWM1RDVldkFFQkp4YWNLTHlJWDVo?=
 =?utf-8?B?RlBhNVRHSkVQOFE0Vlloa3UyZmlIUldSYVp6cDdBQ0JpQnhHTTQxZjlmZUtD?=
 =?utf-8?B?YTFKTnFFWmUwdk16b0c0Vm5ITXBKUG5BcXJtYmhKNzlVUnBaN2ovK0JhRVph?=
 =?utf-8?B?ZzRnak9aS0I3Wmx5bGVzUkdCMmlaaTNvRVhSRHE3QVI0aDNaTUMzbGtmaHdv?=
 =?utf-8?B?ckJmUkJzMW9uU0xvSzlCUEtIQkxteXNCanptTU5TZVBzWUpFaTQxK2FOL3ln?=
 =?utf-8?B?NE4xYWpUYUNkRVAyZVJOUjN0S09kbGlnOW1HRUhNZyszZklEWXVuekZla2Rm?=
 =?utf-8?B?ZUprb2pjcHBLSVdSTllJS080MW94OTQ3dm90ajBVbTNhNVVNaVd1TWFUK0l1?=
 =?utf-8?B?SURmaEU0RTQzOHQ0NkFhYVVBSzFuMkZsNXdwYkczeWw4azd2RlRPNHNxTUt5?=
 =?utf-8?B?aDU5ZU9zRlBzWEwvb25YOUo5aXExT0lqNml6YTh1ZmhkaHNldWtQMWViY0ww?=
 =?utf-8?B?MUFPMm1XR2V1WnJuZFhSSXFyY3BjNHp1dWxKZk9XUVMyV2YrTVA0alFPaHVT?=
 =?utf-8?B?aEx5RG53ZHl2K2pZZUNWRmJLaHBPOFArakIxTUprOVl3cTRrTzRVZ2hVOElJ?=
 =?utf-8?B?cWJjbHJsdHgwTUI3aml2bVg1SXQxWkRKUGlqR0ZrZjU4U3BaQWpxajVTaSsv?=
 =?utf-8?B?QkFFVWVtL043QnY2eXJZWWhqSk9vOHFvWWZnb2RKMUVqNHJhVmxQU0xrSWxa?=
 =?utf-8?B?bDI3NmFhTHJ3Z3ZmVTg4c2ZTSUpORnNYSmFCMlVNTDhHeDBGc2l3Z0VhbUlE?=
 =?utf-8?B?UStEV2xuNjQ1eUMrRkNYdGIyQW1UcWRyeENnRERydHgwcUhVVGZWd3FreDJX?=
 =?utf-8?B?QWR5djdlcnhZK0xubGZZVlArL0pwZ0NsWTV0NitHR2JjdTBGeVh6cEI0c09p?=
 =?utf-8?B?bUJROEpRUW9BdExZT29UWDloeVJRaXJkMWNqa0JEaUFYRks2Mi9aYmgxZ2Jw?=
 =?utf-8?B?RlZBNnFjSVRhcm9jN0RWM05FdUVwbHdJelVIUVRidnkwTHZ2VCtqMDJhYUh3?=
 =?utf-8?B?WjdmblV4ZkFTTE1TMDRNcVlkM04xT2FOemV6WGt1Vk5pVTk2eC9DSkJVOUp2?=
 =?utf-8?Q?JhamBrg2SSnHv3pbxUwln2yn2?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: RhGlR/jx2XL3IjAc6g6SidFhpX/OZrzEugqUg9IQpQl5faLAAimtnEtlIWYRzwzNTra5nn/mZlENmBsoBDYGssR/YNIJP7pyxiAEDasQ9bcVH4nNrhWZtdCUFkgM1k8dTqDTMhQfkust8XSUBlKdw+5f4mqVV+oUHYnL9kON6K5ZR1ATW3Jgu9G97PFFaqxZvFZUuV535qauDgnRWbSvTwkzVoGM7+ZZrEAyZVbWaTIUlWzPBZ5uzN9o7ZufncnYwZyyadblbQStUIGFdm4Py66185I5XiQdXEjk2cbMDoSAtnDLN+tSHCx5Ytf3reWzc8bLXvOFiXx1TOeN1yYzGTx3paEeV77S+Ozur9ZZ57I4RaitCYDrmCZ4CNDCUIAGIAUKta8cGkNW8zU8+7r+RcSwHIrCu+8rnyn671vnZhKpFCwh+prcXZ6frUQtQxWYzIUuHjJVga5klaCbN8M9CoEDHdun9wm7ka7S2olC63rOv2W0SLwlVq6zb/qmZUe44aUmiJYD+tomwa7YVHBUAdCY9qGiIarN0kbkoesNPQJRHXuK8x7oTVh4MAW/E5Wi2vzIlivmeJN6qV/k7nLABJg3p2ytnvH4Pz6Hrn7R3hdij811VJLYZZ5saz7qFOCdnaNjz6Jlr/MWoTeMsbe9pUc0NxH9kFpQyOFSwf4LXHDXvzoKUp6GVIDEmoUYAuamF+U8oUsi/yYkX2W8QyCM2qTxVJn9c5orJgXoTID8/g3QqBvViBfUiaS7TRuRSv+zyN+tEC8/kKCe2T5tSNRfhl5mjJSNKMMorLodQrkvU+DlFltoBrZyDGBhHqydc0EMvIhH+tUWjAP/OLL2Cbp2Ww==
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 82ef91af-21f9-4c8a-64b0-08db15607cf9
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4257.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Feb 2023 05:40:36.3788
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: x9XbgaoG6eAFGcbkPr8GXL8QBhQjc3njL6tfvpeF+TkipqUUmY3X8IlSNQP8MiAZr48bB9h3l+S4bUwYYTnEGA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR10MB7359
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.219,Aquarius:18.0.930,Hydra:6.0.562,FMLib:17.11.170.22
 definitions=2023-02-23_02,2023-02-22_02,2023-02-09_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 phishscore=0
 mlxlogscore=999 bulkscore=0 suspectscore=0 mlxscore=0 spamscore=0
 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2212070000 definitions=main-2302230049
X-Proofpoint-ORIG-GUID: 9c2pD85GIphfhhjSkXy5DrbgUAqDM5et
X-Proofpoint-GUID: 9c2pD85GIphfhhjSkXy5DrbgUAqDM5et
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

Hi Anna,

Just a reminder that this patch is still waiting for a review.

Thanks,
-Dai

On 2/17/23 10:22 AM, dai.ngo@oracle.com wrote:
> Hi Trond,
>
> Could you please let me know your opinion on this patch?
>
> Thanks,
> -Dai
>
> On 2/10/23 12:10 AM, Dai Ngo wrote:
>> Occasionally NLM lock and unlock request fail with EIO and ENOLCK
>> respectively. This usually happens when the NFS server is restarted
>> while NLM lock test is running.
>>
>> Currently there is a 9 seconds limit for retrying the bind operation.
>> If the server is under load the port mapper might take more than 9
>> seconds to become ready after the NFS server restarted.
>>
>> This patch increases the timeout for rebind from 9 to 30 seconds
>> allowing a bit more time for the port mapper to become ready.
>>
>> Signed-off-by: Dai Ngo <dai.ngo@oracle.com>
>> ---
>>   include/linux/sunrpc/clnt.h  | 3 +++
>>   include/linux/sunrpc/sched.h | 4 ++--
>>   net/sunrpc/clnt.c            | 2 +-
>>   net/sunrpc/sched.c           | 3 ++-
>>   4 files changed, 8 insertions(+), 4 deletions(-)
>>
>> diff --git a/include/linux/sunrpc/clnt.h b/include/linux/sunrpc/clnt.h
>> index 770ef2cb5775..7f2dee56c121 100644
>> --- a/include/linux/sunrpc/clnt.h
>> +++ b/include/linux/sunrpc/clnt.h
>> @@ -162,6 +162,9 @@ struct rpc_add_xprt_test {
>>   #define RPC_CLNT_CREATE_REUSEPORT    (1UL << 11)
>>   #define RPC_CLNT_CREATE_CONNECTED    (1UL << 12)
>>   +#define    RPC_CLNT_REBIND_DELAY        3
>> +#define    RPC_CLNT_REBIND_MAX_TIMEOUT    30
>> +
>>   struct rpc_clnt *rpc_create(struct rpc_create_args *args);
>>   struct rpc_clnt    *rpc_bind_new_program(struct rpc_clnt *,
>>                   const struct rpc_program *, u32);
>> diff --git a/include/linux/sunrpc/sched.h b/include/linux/sunrpc/sched.h
>> index b8ca3ecaf8d7..e9dc142f10bb 100644
>> --- a/include/linux/sunrpc/sched.h
>> +++ b/include/linux/sunrpc/sched.h
>> @@ -90,8 +90,8 @@ struct rpc_task {
>>   #endif
>>       unsigned char        tk_priority : 2,/* Task priority */
>>                   tk_garb_retry : 2,
>> -                tk_cred_retry : 2,
>> -                tk_rebind_retry : 2;
>> +                tk_cred_retry : 2;
>> +    unsigned char        tk_rebind_retry;
>>   };
>>     typedef void            (*rpc_action)(struct rpc_task *);
>> diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
>> index 0b0b9f1eed46..6c89a1fa40bf 100644
>> --- a/net/sunrpc/clnt.c
>> +++ b/net/sunrpc/clnt.c
>> @@ -2053,7 +2053,7 @@ call_bind_status(struct rpc_task *task)
>>           if (task->tk_rebind_retry == 0)
>>               break;
>>           task->tk_rebind_retry--;
>> -        rpc_delay(task, 3*HZ);
>> +        rpc_delay(task, RPC_CLNT_REBIND_DELAY * HZ);
>>           goto retry_timeout;
>>       case -ENOBUFS:
>>           rpc_delay(task, HZ >> 2);
>> diff --git a/net/sunrpc/sched.c b/net/sunrpc/sched.c
>> index be587a308e05..5c18a35752aa 100644
>> --- a/net/sunrpc/sched.c
>> +++ b/net/sunrpc/sched.c
>> @@ -817,7 +817,8 @@ rpc_init_task_statistics(struct rpc_task *task)
>>       /* Initialize retry counters */
>>       task->tk_garb_retry = 2;
>>       task->tk_cred_retry = 2;
>> -    task->tk_rebind_retry = 2;
>> +    task->tk_rebind_retry = RPC_CLNT_REBIND_MAX_TIMEOUT /
>> +                    RPC_CLNT_REBIND_DELAY;
>>         /* starting timestamp */
>>       task->tk_start = ktime_get();