Message-ID: <da7394e0-26f6-b243-ce9a-d669e51c1a5e@oracle.com>
Date:   Mon, 29 Nov 2021 10:32:10 -0800
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.3.2
Subject: Re: [PATCH RFC v5 0/2] nfsd: Initial implementation of NFSv4
 Courteous Server
Content-Language: en-US
To:     "J. Bruce Fields" <bfields@fieldses.org>
Cc:     chuck.lever@oracle.com, linux-nfs@vger.kernel.org,
        linux-fsdevel@vger.kernel.org
References: <20210929005641.60861-1-dai.ngo@oracle.com>
 <20211001205327.GN959@fieldses.org>
 <a6c9ba13-43d7-4ea9-e05d-f454c2c9f4c2@oracle.com>
 <33c8ea5a-4187-a9fa-d507-a2dcec06416c@oracle.com>
 <20211117141433.GB24762@fieldses.org>
 <400143c8-c12a-6224-1b36-3e19f20a7ee4@oracle.com>
 <908ded64-6412-66d3-6ad5-429700610660@oracle.com>
 <20211118003454.GA29787@fieldses.org>
 <bef516d0-19cf-3f30-00cd-8359daeff6ab@oracle.com>
 <b7e3aee5-9496-7ede-ca88-34287876e2f4@oracle.com>
 <20211129173058.GD24258@fieldses.org>
From:   dai.ngo@oracle.com
In-Reply-To: <20211129173058.GD24258@fieldses.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?WEJGNHpPRFJ3V0pqVHMrTHpBeVFLM091eUlMcHk5YjJlZ0FVdGRUL0pHYlJq?=
 =?utf-8?B?eWt6YUVqM282V1AwK3AvcXZ5NlRpNUw1aXNVT0F5ckl1NUxVQ3QzSXo1eXNr?=
 =?utf-8?B?WHlYTVZBaVVUZUd5RmExWTIrNUJRYnE3UWE2TUt1NkxCS2QxQ20yQzVXSnBR?=
 =?utf-8?B?eGVNOGxBUzg2aGtjQlRDWDE0S3BjQ0ExVmw3aXNHV3FvM092a05Kd09SU3By?=
 =?utf-8?B?R2dPOVpKWTRjdWFFSGNyYjRTMWhtWlBaQklyWDI3bXBVN2lNeURRZ2tNYXNh?=
 =?utf-8?B?RDU2WlR4bC9OWWxWQjVLa3d5ZnpWdW9DcDZRNnFlS1plVHlVQzJMNWs3QXZJ?=
 =?utf-8?B?K3E0cjF2UkVPNnhpWUNwS0xJSlY2ck5YTGlnUTJlMkdTZDQwZ2JZTGFMS3Bm?=
 =?utf-8?B?amcwcVhIWDhSTXdCUzdPNkxEZTZwK3ZSMUYyem5YWVNBRGVyaGlwSml5UER1?=
 =?utf-8?B?SXFXeTBHSkI3M2k2MkFrU3pqY0hURWVOUEkvTGFzTzZEZWhDc0hLc2RoQlZP?=
 =?utf-8?B?eUw5Z1FsNzU0V21HaTBoaU5UdVJKSUdqT05yUE5US090MXR4bXBzcWdROE9y?=
 =?utf-8?B?d0k2cHoxTWVXSUthUWZWWlVYeXllWUZqK2d2TTdhNnFZVG5xQzg4d2Y2V093?=
 =?utf-8?B?TFRrTThlTHg3bjJySFhlaE52QkJMSGIwUE15V2ZiZG9ma0JhN2ZabDNDeFlO?=
 =?utf-8?B?eU4yNTN4cEUzSVpUN0JRWHJyUUhiU2lBUUE4TWRUbHpzcmp3d0M4TDZ3UGVM?=
 =?utf-8?B?MWM1bDJYVjk4VDU4YkJCTVFGOXE5eVlLbU9SNEc4T1dlYTljUUp1dkhuK0F0?=
 =?utf-8?B?MEtKMm1LOHQvQWl4UzFKdFZVOHFEaHpPOXJHYXE0QmYyd3Z2dmdNQ3N1WFRy?=
 =?utf-8?B?TjhGb0pleWNlRXpRS25Od3JZU3A5djQ3WXc5Yzdsb0N1OGh5MW8zUlpjWlFE?=
 =?utf-8?B?RDdYalNNVnhjdlJaUFF4dG90VnVNbmpHS001djlPaUlsVlFmU00yK09xR2pC?=
 =?utf-8?B?aHFtdEVkVVBCa09ZZDA2MTRwT0tOWTdjeklKVlBZc0NUN3dvTU9lQTREc3c4?=
 =?utf-8?B?eG00b3JpYUUvai9PZk1ZWlpRZnJ3L2tHWWdnM0xGdXc4R3R1WkZrMm9lcnRk?=
 =?utf-8?B?UHEvOUc3UFhLRy9zWnhPUFUzcEx3ZWtwbjFBZ2JaTVdCYUU3NGdBQmR3dDVJ?=
 =?utf-8?B?M1IwVmNuRFVvZm96bmFaZEEvenpIWVJUWTkzbCtwdis3NHJ1L0RWVVJWaU03?=
 =?utf-8?B?UzFOb0czN1I0Z2JjQjZacXM0eHBJZFhOTmYzVHU0enYxbmtid0R3NitRWmJr?=
 =?utf-8?B?NWpWVCtIaSt1WFc5bHJXKyt2YWxSZEF6WVBCek42OFlWb1ZzdE9IOUQvRUFq?=
 =?utf-8?B?SWorRWVtRjNSS0hwdElMVEpJU2wwQUVHYS81NG9iSmZIQ0Z0MTNSWm80eWt5?=
 =?utf-8?B?dHNNZ2RKdGM1UlhwdVdlL1cvamkzR21HL1RWMkpzM0RLZVpJNldTVFdVSkpH?=
 =?utf-8?B?NThTNy93cHFwV1pBRE5IdGw3WHJoQW5QeVcwRDNoS3RnaGxnbFRKQzdIZHF3?=
 =?utf-8?B?WXRLcE1qd0k5WUpzMVlzQTBMRkVzMFd6ZXIvUTZnbGV2MkR5bVpHM01idmZR?=
 =?utf-8?B?eEJoM2FXVXBFRnNyUTBMcWwrZWY1ZVVVQVFSUHQraW94WVZQbVhiMHB2Qzhz?=
 =?utf-8?B?M3VHbmhRZ2lUT1hrUTdLTTBZRTdiQlFvTjQydTkwczF5M1lGZUZqYW8rQ0Zr?=
 =?utf-8?B?aVNhYndSd1BOdnJaZXZSZnB1aFZ3enJJNjUwQW45cEhQbGJGbFQ1MzErMXNP?=
 =?utf-8?B?VFJ2SkU0cGpvd1VwS0FlOGVNd0NUbWkvdVdvOE5TTDRUdmJGT0lSaDF4TWxM?=
 =?utf-8?B?RnRXTWUvKzNzK0VFRXY3QzM2dWdpaWhpdElscndDMUR4aFlrUk1WTlRuUm02?=
 =?utf-8?B?WGxocFRQQldtR1gxUjBVeXBBdTFkM2pHWDU2NzU4a0RsL3oveEJmNlhaMjNP?=
 =?utf-8?B?L1ZWdVUxYUhDd3pYVmFJMHhlSkc3UENxZ1N4UUltam45ZG1iRXJvcXI5OVZG?=
 =?utf-8?B?VzFGMHpOTzArU0xFdHhGczRxbUxzbkh2UklzWC9PTHJBbTZWWGJoeWNpTjFv?=
 =?utf-8?B?ellORU82eklLb0FBRnZpOENISFl1dlB6a0ZnSDZZNGEyeGJ0TklUa21RbU5i?=
 =?utf-8?Q?Cs+krxOk/w0/EvepC8a6glM=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6509019d-778c-422b-933b-08d9b3668eff
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4257.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Nov 2021 18:32:11.8319
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ayIwNuMChh42zZ4ph1nlezLPPdV8bajmocmkAkYrngBb/hYMGgSMdwByzRxEw1hoRDb8JnD1NL17rpzG98BKLw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4495
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10183 signatures=668683
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 malwarescore=0 mlxscore=0
 suspectscore=0 mlxlogscore=999 spamscore=0 phishscore=0 bulkscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000
 definitions=main-2111290085
X-Proofpoint-GUID: IVl7HW7FAbu42NVimV9-Jc9Zi8TjaBGV
X-Proofpoint-ORIG-GUID: IVl7HW7FAbu42NVimV9-Jc9Zi8TjaBGV
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


On 11/29/21 9:30 AM, J. Bruce Fields wrote:
> On Mon, Nov 29, 2021 at 09:13:16AM -0800, dai.ngo@oracle.com wrote:
>> Hi Bruce,
>>
>> On 11/21/21 7:04 PM, dai.ngo@oracle.com wrote:
>>> On 11/17/21 4:34 PM, J. Bruce Fields wrote:
>>>> On Wed, Nov 17, 2021 at 01:46:02PM -0800, dai.ngo@oracle.com wrote:
>>>>> On 11/17/21 9:59 AM, dai.ngo@oracle.com wrote:
>>>>>> On 11/17/21 6:14 AM, J. Bruce Fields wrote:
>>>>>>> On Tue, Nov 16, 2021 at 03:06:32PM -0800, dai.ngo@oracle.com wrote:
>>>>>>>> Just a reminder that this patch is still waiting for your review.
>>>>>>> Yeah, I was procrastinating and hoping yo'ud figure out the pynfs
>>>>>>> failure for me....
>>>>>> Last time I ran 4.0 OPEN18 test by itself and it passed. I will run
>>>>>> all OPEN tests together with 5.15-rc7 to see if the problem you've
>>>>>> seen still there.
>>>>> I ran all tests in nfsv4.1 and nfsv4.0 with courteous and non-courteous
>>>>> 5.15-rc7 server.
>>>>>
>>>>> Nfs4.1 results are the same for both courteous and
>>>>> non-courteous server:
>>>>>> Of those: 0 Skipped, 0 Failed, 0 Warned, 169 Passed
>>>>> Results of nfs4.0 with non-courteous server:
>>>>>> Of those: 8 Skipped, 1 Failed, 0 Warned, 577 Passed
>>>>> test failed: LOCK24
>>>>>
>>>>> Results of nfs4.0 with courteous server:
>>>>>> Of those: 8 Skipped, 3 Failed, 0 Warned, 575 Passed
>>>>> tests failed: LOCK24, OPEN18, OPEN30
>>>>>
>>>>> OPEN18 and OPEN30 test pass if each is run by itself.
>>>> Could well be a bug in the tests, I don't know.
>>> The reason OPEN18 failed was because the test timed out waiting for
>>> the reply of an OPEN call. The RPC connection used for the test was
>>> configured with 15 secs timeout. Note that OPEN18 only fails when
>>> the tests were run with 'all' option, this test passes if it's run
>>> by itself.
>>>
>>> With courteous server, by the time OPEN18 runs, there are about 1026
>>> courtesy 4.0 clients on the server and all of these clients have opened
>>> the same file X with WRITE access. These clients were created by the
>>> previous tests. After each test completed, since 4.0 does not have
>>> session, the client states are not cleaned up immediately on the
>>> server and are allowed to become courtesy clients.
>>>
>>> When OPEN18 runs (about 20 minutes after the 1st test started), it
>>> sends OPEN of file X with OPEN4_SHARE_DENY_WRITE which causes the
>>> server to check for conflicts with courtesy clients. The loop that
>>> checks 1026 courtesy clients for share/access conflict took less
>>> than 1 sec. But it took about 55 secs, on my VM, for the server
>>> to expire all 1026 courtesy clients.
>>>
>>> I modified pynfs to configure the 4.0 RPC connection with 60 seconds
>>> timeout and OPEN18 now consistently passed. The 4.0 test results are
>>> now the same for courteous and non-courteous server:
>>>
>>> 8 Skipped, 1 Failed, 0 Warned, 577 Passed
>>>
>>> Note that 4.1 tests do not suffer this timeout problem because the
>>> 4.1 clients and sessions are destroyed after each test completes.
>> Do you want me to send the patch to increase the timeout for pynfs?
>> or is there any other things you think we should do?
> I don't know.
>
> 55 seconds to clean up 1026 clients is about 50ms per client, which is
> pretty slow.  I wonder why.  I guess it's probably updating the stable
> storage information.  Is /var/lib/nfs/ on your server backed by a hard
> drive or an SSD or something else?

My server is a virtualbox VM that has 1 CPU, 4GB RAM and 64GB of hard
disk. I think a production system that supports this many clients should
have faster CPUs, faster storage.

>
> I wonder if that's an argument for limiting the number of courtesy
> clients.

I think we might want to treat 4.0 clients a bit different from 4.1
clients. With 4.0, every client will become a courtesy client after
the client is done with the export and unmounts it. Since there is
no destroy session/client with 4.0, the courteous server allows the
client to be around and becomes a courtesy client. So after awhile,
even with normal usage, there will be lots 4.0 courtesy clients
hanging around and these clients won't be destroyed until 24hrs
later, or until they cause conflicts with other clients.

We can reduce the courtesy_client_expiry time for 4.0 clients from
24hrs to 15/20 mins, enough for most network partition to heal?,
or limit the number of 4.0 courtesy clients. Or don't support 4.0
clients at all which is my preference since I think in general users
should skip 4.0 and use 4.1 instead.

-Dai