Received-SPF: pass (google.com: domain of linux-nfs+bounces-2005-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Message-ID: <7a7f678a-eda7-4360-aeb1-67129af2b1f5@oracle.com>
Date: Sat, 17 Feb 2024 09:42:57 -0800
User-Agent: Mozilla Thunderbird
Subject: Re: PATCH [v3 0/2] NFSD: use CB_GETATTR to handle GETATTR conflict
 with write delegation
Content-Language: en-US
To: Chuck Lever <chuck.lever@oracle.com>,
        Dan Shelton <dan.f.shelton@gmail.com>
Cc: linux-nfs@vger.kernel.org
References: <1708034722-15329-1-git-send-email-dai.ngo@oracle.com>
 <CAAvCNcDDb4L2tcbBCcufFg=TLeFSrui4MHFuEeNUA+1VZyXLxQ@mail.gmail.com>
 <ZdAXI1oAJx0atlch@manet.1015granger.net>
From: dai.ngo@oracle.com
In-Reply-To: <ZdAXI1oAJx0atlch@manet.1015granger.net>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?SldEZlozQmJMQlhyckVNa29sOTlST0hraFFzWHJuQmgvK3hXTy8xVUxCS1VH?=
 =?utf-8?B?amxsSzFUTWc0QmJmdXhtQ0tFMEZBMERmQ0NhZ3pTejgxNmQrRSt1ZTRrZGNv?=
 =?utf-8?B?Rm8xT1ZaelNub0lTRWFZM09sbVpJczJYMEwxdGw2bXZZOEZTTVArMlBxZkZo?=
 =?utf-8?B?WGhuVmdmcWN5aEdOTHorRTFXcjMrdTI5MExVeDZyYkF5eWh2L0VFaFhrbk0y?=
 =?utf-8?B?UHlDTGpPTjlVZHdiYjBTRkNYLzRjMGpPR3JQOU5XODhTRERKTTNpT2RpRkNx?=
 =?utf-8?B?OUFta1NOT0JPZmdyZERiK1BaVWdaL2tqVm01aTZleWZJc3h6c3N0THplSEJN?=
 =?utf-8?B?YTIyRzFmazNaV2FnN0h1SUwvS2pVQW10ZGJKc1Rxdkh6eUQ5RldSSVM4R25v?=
 =?utf-8?B?VGJyQnluWFMzUTNWNGFxc256ck5FMEFlYVY5bDIxQzdZWk51MTh2SDVWMStI?=
 =?utf-8?B?a01yYTZ3T29jTkVPcGFkQjFIMnBCaXlDTWFXeUppRVQrYW1rR0loUTc1em5h?=
 =?utf-8?B?VDlTL1pjVmpxbUJ6QS9sUStucUlUTXFEdXBaMHFZNDJWZTZKZ0gveGJpVUph?=
 =?utf-8?B?OGp6eEVTVXdOb1BScXIzenVYeitnUGhvMHlRZlF6bk5YZzdrQmZ6c0pqem0w?=
 =?utf-8?B?b1Iya0ZxMW45NlpQRlBpK1QwSE8wWTUrejNhY3J0ZWdZNjZSby95VndyOGR5?=
 =?utf-8?B?S0hpOGJYdjYrVXRaTExmOUg4cFh6b2lSVW1LeHFoSDRyczFrRjQxV2hBQlEv?=
 =?utf-8?B?SytaS2V3akxuTDIxeU1pQjRoM1ZUNTNEKzlhT1hDNUM0My9kRXQ2QTBKN2VT?=
 =?utf-8?B?T2lCR1lldWFIWHFtY1I5SmNjL2gxOXoxNGNYWFlYeCtaSXdMOXlPbmZGTXhR?=
 =?utf-8?B?c3ArZm13bFE2MDdCTHgrOGF0RE9sSE8vdGZNRE9HN0F0Wkc1ak52NXpVOFhQ?=
 =?utf-8?B?bUtSbkplVkZ3bGkxZHdXUVpnaEJxT1pVVVVWL2UyYVBzOFlLMGNxMnVhWmI4?=
 =?utf-8?B?S3RMSmVxZ1hWTU9RMDlVZHhUUWtNbWw4OGxZbkZsOGFWSEU2eC83Q3c1VHNW?=
 =?utf-8?B?UGwySHlhYnAyTjRUMzRRam1vL2o2Sm1ITmlWVnFNVTloOFc1QTA1TUpDeGRh?=
 =?utf-8?B?TmR6YzBCWHEyN2MzR0JuL2I4emdhenp3a3k5di8vckNnbGRjSEF4ak9NQUEz?=
 =?utf-8?B?b2xkbzlZV1ZxNlNiVllBSnpNaXNucmdQV0ZlTnVQM0c5VUkvRVlvVmNQZmN5?=
 =?utf-8?B?MEN3bEtOREJEaDFpS0V1b2xsakhQNGFFNGJkYVgwTW0yZjB6ZEpoRzJEZUsw?=
 =?utf-8?B?UzhvenhnS0tvbTZ2YTdkdFNOaEJJVVlkYi9pUWtxVVg5eDcyVlorei96MjRl?=
 =?utf-8?B?dWx0NzN5dmJRdzk5K21nRTdkQXJsRjJIbnYxTkpiMHNrYjNDalFiMW5yZVJH?=
 =?utf-8?B?QTUxaGJxU2VvY3JJZ0kveTRpYmdmZnc4WTczZ0dHNjZPUiswTFJVZ1JVYXI2?=
 =?utf-8?B?bEplRm9ieHgvOG9hd0xCTEN2Q01iQkhwZko1SVh0UzE4RVJiNnVIaHJ3M3Iw?=
 =?utf-8?B?amlYZXh1SlVENStvM3ExSDJBbEZDMlVRNitOb0xEdFFpYjNsNng1SERNT085?=
 =?utf-8?B?Rm1VOUxDKzhVR0huanpxMy9mQUhoWVlFMEpPLytTVjcrWStqUWlScVJKTExM?=
 =?utf-8?B?cy92NnBJV0s0bmJHNnBmNGxRRXlyZ21nWGpXVVNudnViOVhaSWtsWUhITlN4?=
 =?utf-8?B?SVd3THhlN1FiZjFWVUp5NGFqb1J5N2lvVmxtUzlub3FZMlRYVXI5R0ZhS05Z?=
 =?utf-8?B?QXVkTWVxMWNsUDJ5NWMyVEdQRDV6VDRlTTVLKys0Ni80NDZpWnBmU3pYQzRQ?=
 =?utf-8?B?bEtlTDIvNExTS3Z5QkxsS25RclJZYkdCZ2pUc01sdDMzbUpCUmVRRVNSQ0Fk?=
 =?utf-8?B?bW11UWpRRzFncVdhMWtINk01R2tTY09zVG16WW1BSzUvdjZuVFVmVWNQUEg5?=
 =?utf-8?B?NXFObmhxTW4rQWh4dFNhaEk5eFlWK04vU0dZQXVpYXlkYUZoWUpHeW9mZitC?=
 =?utf-8?B?cGsxNkZ1NzVuSmMzeW03Wkd3YzluREd3UlBOUWRWWEM1bTU0akNXVFN5cXhv?=
 =?utf-8?Q?YuiesBdLHqbxmyFu4k7BJSWII?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	aSBtW5PrYa49GOtcLWRs1vpl700ggNPTh8FbjxEFSDD+6RI037rAITU/9I2mhofWYN1eQhCaTPl7w75HuK4v76shPaNBcXppkdW8o5Xz8sgX5t27A5lTSdIvTrlLsECy2ielP/B9h2ZMQEYBWxOjxSN5KLt71uOcwPvBFdc57LZPHH6wyV/ZntusT/loAM+Ozq7j3oFYxpxJ8Z5hvi1e8RAFbFRPFMk8wjRAqpXOcvSN55wIewncL5za6z4GN1OXScuRdb2wtqp6FvJmLuthb2RmhCiH0FNMq/bcXaldppkVdakCke1IGtG+Ad1JbGqX+Y95+0oz+bRmBU0vyRiA7goEvqmxtcSFmkTKH594ovwNJ284oKwE2YomtSWhPyy+9ICj1Ag+6hEO/tiFviD/yvCT1j16rmp5//F1cJHEA1tdXjKCcOh2MbAvCIqTQo4Kq2FvXMtUcT1Y/FMgH/a3QDEgRfSnrpxsGOGPn/v68t0Mz1GjBvcLCv46/ZneSKv3xBMqSxnlqvO2RurJzEamF20mvsJrcBLnmtUkBoRVzq4TwM7QTw2QjtAM3BZFDTarf/DSX3A+IG5ZOw7t2i/hqbQOMJXZeXvrMSGeKBwA4+8=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: c60a9d5c-8c26-4ec9-872f-08dc2fdfe18d
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4257.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Feb 2024 17:42:59.1191
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: GPis27oc672YKUKao1sW1xhTlbQiRr7cUs0+sdaSeRgv6ci0RROURN2jGK7SkF3743Z9ZD3IKJZKs9thCUeabA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW4PR10MB6608
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-02-17_16,2024-02-16_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 phishscore=0
 suspectscore=0 mlxscore=0 bulkscore=0 spamscore=0 mlxlogscore=999
 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311290000 definitions=main-2402170145
X-Proofpoint-GUID: arbY2o5jGrDZqhqG0oI8JQwxD199PAEJ
X-Proofpoint-ORIG-GUID: arbY2o5jGrDZqhqG0oI8JQwxD199PAEJ


On 2/16/24 6:17 PM, Chuck Lever wrote:
> On Sat, Feb 17, 2024 at 12:34:45AM +0100, Dan Shelton wrote:
>> On Thu, 15 Feb 2024 at 23:11, Dai Ngo <dai.ngo@oracle.com> wrote:
>>> Currently GETATTR conflict with a write delegation is handled by
>>> recalling the delegation before replying to the GETATTR.
>>>
>>> This patch series add supports for CB_GETATTR callback to get the latest
>>> change_info and size information of the file from the client that holds
>>> the delegation to reply to the GETATTR from the second client.
>>>
>>> NOTE: this patch series is mostly the same as the previous patches which
>>> were backed out when un unrelated problem of NFSD server hang on reboot
>>> was reported.
>>>
>>> The only difference is the wait_on_bit() in nfsd4_deleg_getattr_conflict was
>>> replaced with wait_on_bit_timeout() with 30ms timeout to avoid a potential
>>> DOS attack by exhausting NFSD kernel threads with GETATTR conflicts.
>> I have a concern about this static and very tiny timeout.
>> What will happen if the ICMPv6 latency is well over 30ms, like 660ms
>> (average 250mbit/s satellite latency)?
> CB_GETATTR is an optimization for write delegation. Without
> CB_GETATTR, or if the client does not respond within 30ms, the
> server recalls the delegation. We expect no impact on clients
> that connect on a high bandwidth-latency product link.
>
> To lengthen that timeout would require the implementation of a
> mechanism for NFSD to defer requests without tying up an NFSD
> thread. So for the moment, the proposed CB_GETATTR implementation
> will help fast local clients but should not negatively impact
> remote clients, and we cannot in good faith provide a tunable
> to extend that timeout beyond a few dozen milliseconds.
>
>
>> Would that not ruin delegations?
> As stated above, it should not impact write delegation, and Dai can
> correct me if I'm wrong, but I believe CB_GETATTR is not used if the
> server has offered a read delegation.

Yes, CB_GETATTR is only used for write delegation.

>   NFSD implemented only read
> delegation until very recently.
>
> IIRC, there is instrumentation in v6.6 or v6.7's NFSD to measure
> how often a CB_GETATTR might have been beneficial. I can provide
> more detail when I'm in front of my desktop computer.

# cat  /proc/net/rpc/nfsd |grep wdele

-Dai

>
>