Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
Message-ID: <a4b46792-49f0-b2eb-77f9-547b96e61224@oracle.com>
Date:   Wed, 12 Jan 2022 10:56:30 -0800
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.4.1
Subject: Re: [PATCH RFC v9 2/2] nfsd: Initial implementation of NFSv4
 Courteous Server
Content-Language: en-US
To:     Bruce Fields <bfields@fieldses.org>,
        Chuck Lever III <chuck.lever@oracle.com>
Cc:     Jeff Layton <jlayton@redhat.com>,
        Al Viro <viro@zeniv.linux.org.uk>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
        "linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>
References: <1641840653-23059-1-git-send-email-dai.ngo@oracle.com>
 <1641840653-23059-3-git-send-email-dai.ngo@oracle.com>
 <E70D7FE7-59BA-4200-BF31-B346956C9880@oracle.com>
 <f5ba01ea-33e8-03e9-2cee-8ff7814abceb@oracle.com>
 <D2CF771C-FF71-4356-8567-33427EAC0DA9@oracle.com>
 <20220112185327.GA10518@fieldses.org>
From:   dai.ngo@oracle.com
In-Reply-To: <20220112185327.GA10518@fieldses.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OEI2WUpqdjJBMXp3WWZyU3dJQnF0V3h2QTVGT2NGbUV4elZYTXhXL0RLMHpS?=
 =?utf-8?B?eGhWYVNXUlpad2JIUWx4cFEySHFxWGVLRUtFV0ViZXA3VjNWZk5hcUlHTTg1?=
 =?utf-8?B?MVJhS0h2R2ROZVJzQjdlRXV5UWxDN3drQmRFV3dBN1hwMERqUnBzcXB0dVh4?=
 =?utf-8?B?a2hqNzNSYUgyQmdqNEp6dkhVa0dSOG42MStHanJONEVuYWJkbUtYR0phSXRZ?=
 =?utf-8?B?cXdxQ3M3Z0dwcnAwd0h2NC9uanlJVzRTOEM5b2VacWZlUlFoYUVndUtYOU5H?=
 =?utf-8?B?OC9PMHAzYXVwOVRLUXNQVHQ0WjZMTUtjN3RqbUVHZkFxVG9jaVZHemRxRExj?=
 =?utf-8?B?NzQ1YXZqc2xpS0VSR2NCNDRDOHIwNEVPSEdNMUt0bTd4Rjl4bHg2YlYrdm9y?=
 =?utf-8?B?Y0ZydGxZS3lEMENuYUpKWE1VVDdReDB3Tlo1cjh3V050K3BFYkxMYlU0Zk1D?=
 =?utf-8?B?UVVnNEN1ZTFla1g1L2NMTWJlNWlJbkR6M2pQVzdqUGJSODJTamg3Mk5BWmNT?=
 =?utf-8?B?RTRBZXplUjFMU3M3QTV2M0ZLS01iV09ZcTNSR0pMTnFOaWpQUUlXSElwbTNH?=
 =?utf-8?B?UUFIclRSamhlOE52c3lFT0hIUWQ1VU1adDg5NnUxZ295V2k2VWU5eHVhaG1y?=
 =?utf-8?B?MW5iUkZvbUVkMHphSmhzSnlhSUh2Y3hycXBvNWNzbmRIVk0xZVRxVk1ZUmVq?=
 =?utf-8?B?by83WXR0RTRrWVZHQ2JscWZ0KzJxb3Jid2pEZXoxYmFrV05wcTdSTnlwUHlR?=
 =?utf-8?B?dklnSWtkbXIza2RGL3ZFTGZNQXZZb25tM1FXUVM1OWRRUW93MEJjU0FPSlRy?=
 =?utf-8?B?eVNsdVJvQjcyTmlpYmdyRklkTjNYZnQyNFRNVno5Q3FrdjRtWWNJUXFJUUgv?=
 =?utf-8?B?ZUVEamtNTzRTWmI4L1d0VWNuWVhLMVBrYkU4dTRzV1NvN2dyLzVLb0RENFZ5?=
 =?utf-8?B?cGtHdnhEL243azE3VHpHeWJvdVg3aFl2RUlzNUl3Q1hGcW8xY1VuaTNBQ1JJ?=
 =?utf-8?B?L3lGb2s3YVA1aGdoWW0vamd6cnMrT1dKb2JCQnZoRml2TS9heHdMZmV1OFhW?=
 =?utf-8?B?bm1KZW1vZHZFNDFtQUUvak8rcEIza2cvYWZybHBwS1JIS3FscVh0NGNQUXk2?=
 =?utf-8?B?cmRsN0psc25OY0Rna3JQQlkvaVUxRkFHOHI4c3ZOUXNobm92Zy8xL0RINTU2?=
 =?utf-8?B?OVJNSzNwSnYxQThBRnd3R3pkeHROcDRScmFYTThXRzEyZXUxSTRTZWlPc1hZ?=
 =?utf-8?B?Z2QxbHJEUHN0SEl6dkpZWkhGOGJMRnZSL1E4VFlSZVpBTC9JQjVSZnFaVVM3?=
 =?utf-8?B?SXpxTFhwUUdiQ0tJaVV4VkNMZ3U4MFkvcVhZZzB0ZmFxOHlZQ2V0WlAwR09W?=
 =?utf-8?B?bzBQK1pJbzB5QlYxMjhtNWdBZXY0ZnhGZmk1cmVwZGNIR2dIUjl3WllvbEhX?=
 =?utf-8?B?b1M2N2R2ZjZ3Q0hwR3hBc1AyanNJaEhDeVQyb0NSVWpkaVlQczI0ZkQ3bjdF?=
 =?utf-8?B?NDU5eHZXVVRVT2tRZ3Fkd0JBMmdyRTY3Wnl1bmJJRk1rWGRqSlRQcHBxZ1Vj?=
 =?utf-8?B?QjZheGs4ZDFHUnhpeHRqTzVGZ21CcFcxWWFuTHVRenUzcUp1OTRjSHBUNmlN?=
 =?utf-8?B?ZUFTbTN4Z3FyTzBsS3hjUU9DcTRlV1o3U201QTBZRHBNUzhMRmIwRjBsK25U?=
 =?utf-8?B?cDMzVDYvVHdhRkRzVDdZQ0NuMjhRbHRiYUZaeGpaeUtoZ2s0bU1EMythTXlB?=
 =?utf-8?B?NklJSlQxSVR4YWJzWHBvZVQxWkprYXQrWHlyM0lUYjBZVzdaVkdzeUpQUGx0?=
 =?utf-8?B?clJqdlFHQTVEMERwek9YSHJCLzBVOWVya0ZwdVA0OVpMN29RUXJ3aVFxSkRr?=
 =?utf-8?B?QWRZc1FWOExnbDZHSWVOcnRCM29iNmg5a3VMSE5abVBhQ3BDYXlFZHJDN2Jx?=
 =?utf-8?B?cGg5djFoYkI1dU9wem9yMEE3OGlCVUNESzRzaitGVVlCSWZzZWhQQ0ZleU1P?=
 =?utf-8?B?THoxc2RPMTcxZkF5OHBadkd2dTdYV3VGeTlKaW82Mk5wMy9ud1BEZW9NUnZk?=
 =?utf-8?B?MHFtWjlBYzVmYnJmK2dFdy82d3BBVTB2SjdWeDNTT0J3c2lieUk0RVpRaWk3?=
 =?utf-8?B?Ni85c3JTY0dsZjErUUduZ1lqK3hobW5iOUZzdU5kZGs1TkFMN292NW5ETnps?=
 =?utf-8?Q?EoybMC+ELil3a6kFNC5+KTM=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 298c6df5-faad-42eb-5407-08d9d5fd3f61
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4257.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jan 2022 18:56:31.8452
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: o16uYXrAV/35QB3gjH/WYRS8Te0WfzyCCWb0Zvln4mBwZ9xbpTPCNnDfropxaxzhTgkT27PpBJo5Y4jqGrnkfA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR10MB1726
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10225 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 suspectscore=0 spamscore=0
 mlxlogscore=999 adultscore=0 phishscore=0 malwarescore=0 bulkscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000
 definitions=main-2201120112
X-Proofpoint-GUID: CudW8VPXThiaLSg0LsSTweGj6bnzUIr5
X-Proofpoint-ORIG-GUID: CudW8VPXThiaLSg0LsSTweGj6bnzUIr5
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


On 1/12/22 10:53 AM, Bruce Fields wrote:
> On Tue, Jan 11, 2022 at 03:49:19PM +0000, Chuck Lever III wrote:
>>> On Jan 10, 2022, at 8:03 PM, Dai Ngo <dai.ngo@oracle.com> wrote:
>>> I think this is something you and Bruce have been discussing
>>> on whether when we should remove and add the client record from
>>> the database when the client transits from active to COURTESY
>>> and vice versa. With this patch we now expire the courtesy clients
>>> asynchronously in the background so the overhead/delay from
>>> removing the record from the database does not have any impact
>>> on resolving conflicts.
>> As I recall, our idea was to record the client as expired when
>> it transitions from active to COURTEOUS so that if the server
>> happens to reboot, it doesn't allow a courteous client to
>> reclaim locks the server may have already given to another
>> active client.
>>
>> So I think the server needs to do an nfsdtrack upcall when
>> transitioning from active -> COURTEOUS to prevent that edge
>> case. That would happen only in the laundromat, right?
>>
>> So when a COURTEOUS client comes back to the server, the server
>> will need to persistently record the transition from COURTEOUS
>> to active.
> Yep.  The bad case would be:
>
> 	- client A is marked DESTROY_COURTESY, client B is given A's
> 	  lock.
> 	- server goes down before laundromat thread removes the
> 	  DESTROY_COURTESY client.
> 	- client A's network comes back up.
> 	- server comes back up and starts grace period.
>
> At this point, both A and B believe they have the lock.  Also both still
> have nfsdcltrack records, so the server can't tell which is in the
> right.
>
> We can't start granting A's locks to B until we've recorded in stable
> storage that A has expired.
>
> What we'd like to do:
>
> 	- When a client transitions from active to courteous, it needs
> 	  to do nfsdcltrack upcall to expire it.
> 	- We mark client as COURTESY only after that upcall has
> 	  returned.
> 	- When the client comes back, we do an nfsdcltrack upcall to
> 	  mark it as active again.  We don't remove the COURTESY mark
> 	  until that's returned.

Got it Bruce and Chuck, I will add this in v10.

Thanks,
-Dai