Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <9d814666-6e95-e331-62a7-ec36fe1ca062@cornelisnetworks.com>
Date:   Thu, 28 Apr 2022 11:42:39 -0400
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.7.0
Subject: Re: NFS regression between 5.17 and 5.18
Content-Language: en-US
To:     Chuck Lever III <chuck.lever@oracle.com>
Cc:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
References: <979544aa-a7b1-ab22-678f-5ac19f03e17a@cornelisnetworks.com>
 <8E8485F8-F56F-4A93-85AC-44BD8436DF6A@oracle.com>
From:   Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com>
In-Reply-To: <8E8485F8-F56F-4A93-85AC-44BD8436DF6A@oracle.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?VHBvSDdRY1I5TTM0OXpQVHAyR3FheGRIM2hlelh0Q2VEMHpiSDRwRzlHaHdY?=
 =?utf-8?B?QmZZR1oza2crR1dSSEpFZS9xZFZ0ZWI4U2NkbXZqcllGRUNlQkg2aU5xRzRh?=
 =?utf-8?B?S3NORTlOc1BQUHEvREhUNHZDUmdQZzl0QUtiWFVIUFp4NHFtRmloSjVoMGdS?=
 =?utf-8?B?VTdieVhHSzQydm5mckhtS0wwUEVpekhYcXhUZnBkUGZ1b2NhUGlnM2NUR0kv?=
 =?utf-8?B?VGhwbGtoS0lSY250N05FMXBTU1AyVzF3REswMGVDYjh0bmRDUys4OGg3SU85?=
 =?utf-8?B?dTgwZUIrK0VsM2Rncnpud3BvNC9xWWhIQmRRTUxFMXAxV09LRTIvTHkwaVJY?=
 =?utf-8?B?UUYzbDJKNWtQRWFFTmVLQnh3V2Y4MGFuOVZDQXVYQ3ZuNzFoL25ZblZYVFNP?=
 =?utf-8?B?bVdlZFJiUkFZSnk5VlF2UmNsNlVFeTdhUkxZQ2FOdTI3dmhkY043elJlOFdo?=
 =?utf-8?B?VXhvM2V6RzdEd0ZIa3RieDNSbDdUaDRkV3BMYzZmMjc4MlV5bEZTOGdZbStB?=
 =?utf-8?B?a2dibUNuOGFCaWhiUm5FL0w3dDVhRG9HajU4NDZPU1Y0eGgwMG13Tk03dDFx?=
 =?utf-8?B?aDBVZ1lKV2dreUNscHpDUnh2MVRwSmxPdzR6YzVmSCtkcEVZVE9YMHo4a0t2?=
 =?utf-8?B?MTdPUjVmU3ljbWZFbWdxek9VM3J3aGU4THcwL0hKLy9aN3Vadi9iV0ZHeDhs?=
 =?utf-8?B?NnB0elNWQy9lWmRIejdoOHJPZldZaXI3VkM5UXNJZ3hTaTlXT01zODNBN1hM?=
 =?utf-8?B?Vng1Q09SWEhDSEpDMXYxUXYvMCs5Q1JycVNBVW5QWnJkejFLRkR4Rkkzckx0?=
 =?utf-8?B?WHdyM09HS045UEk1MkdBRE1TcVlmdzcrSXRPejZZakhHS0psb0dLMmRoTlAr?=
 =?utf-8?B?eXBSeVo1YnY5TWZmZU9XSytVSVI4dnI4dWcvb3plbFRPS3hDaDBKcGFSR0sy?=
 =?utf-8?B?bHdrUE92c0YwOGNlakJoaHlSR1lkNnpCRDFlWWxQbjdrcm5tUm9YUU5DY0VX?=
 =?utf-8?B?SGNKcXdnUEh5ajQzT05HUmJFN2dsWi9Wd2cvRWpvMU9kSytJZFFrNkdYUWFM?=
 =?utf-8?B?VEpEYlNET2JTTFkyK1ZMMGxUb0hYV004bFBFeVBQeVZnMnpqUGQ4a21KOEtE?=
 =?utf-8?B?dlF6SW9odmVBVjhIb3Z1ZW56R05WRG5NZXZ0U0d0RmJ6MnE4OUxWSHNTVit1?=
 =?utf-8?B?dkovU041VzBWZ2k4UzhJcUlXZUgyUFdsclRFT2F2YTZNSTV4b0NoSFRrVHhP?=
 =?utf-8?B?TFUvSWJtR0QveEVCN24vRE9pb0Yvc1pHRGI3TGRJSDdXVU5yaXNIYk92Lyt3?=
 =?utf-8?B?WmRCNk8zbmoxcDluQ2F5MVpPREtlcGJPYklIdjErakpOM29BMDg4MllQbHBa?=
 =?utf-8?B?dWc5U08rSkJQdTJpdVdEYzhYT201L1ljclcyRHA1M2kxMnczT2cvN29vaTcr?=
 =?utf-8?B?UFBNbmM2MEVDNG8yRjRqTXA4cXZ3V2l1b1RQTEtqbUMwOWc5MjhuMXNNUmpj?=
 =?utf-8?B?N2pIUm84UDNaMEpvcC9pSmQwYk81dnAvNzY5SnBFVkEvRDduVEFobG44ZHJZ?=
 =?utf-8?B?a3NncnJ6anNrRlZMT1diQk5nWjYrSHZoaFByWTZEdGF0dE5HK1Ixb040WW5q?=
 =?utf-8?B?YjJjL3krYTFKMnJOK0tKaFNZNDJxenlHaStjYURuc2VQbTN3RGk4Q0hyRTZT?=
 =?utf-8?B?SkJTLzh1TUpFeURBbnpGazhqOXMxOFdKVU82NWxJUDI4Tmd2LzNOejdtcWMy?=
 =?utf-8?B?d3ZWZzcxZzVuS2JmL2xSVGR6TzN1ZExEV1hlSTY5cml1d2FVS1Q3a3BsMFVY?=
 =?utf-8?B?djBnSTE2N0N6UDNGb1BrQ1pNYVBzZlZvWklYZmI4SEF5dzNFa0ZKOVExUjVD?=
 =?utf-8?B?Ykkwd2tMSkFQeEptNnJ4TCtsTGNHNTBmWGZkUXhyVkpJckpFTGJiWWFKdzhW?=
 =?utf-8?B?TG1JY1BZT2FOUDc2Sy9HV0FOTmJ1R0VLOEExMDZQR0dHcHFzL1pYQ3Y4QUZO?=
 =?utf-8?B?MWZ4ejVtcVg5bnRhQk00aTRZL0NmNlpmdlBxVlpaT2NKODR2Y3lhbC9pTnBG?=
 =?utf-8?B?MEpnR1QyQ0Jyb292VmZUenpSTHROT05DTk5zOEVNaitBMWFOYStiK1dXd0tZ?=
 =?utf-8?B?eUwxTjl3WG9NdlZaVDhRVHk2MkUwUlN5cTg1aHFkREMyRUQ4RExmeVJPUUdJ?=
 =?utf-8?B?Q0ZNYVR1QXc1U0c2UUQwUm9nNm9mQ3dyNTU1b2dPY0VZTFFMSnZXazBGZU9w?=
 =?utf-8?B?ZjdKQUtESzZQOXJmY3hrNENucWsyRnBML3JyNFJ6NVNkK0RHd0VOcTZweDZ4?=
 =?utf-8?B?UVBMbVNaK1Ryb3A1RE5VTGRoM2RXZDA5N2RYa3ZOdG4yMGtENFBuczNpMXZH?=
 =?utf-8?Q?KyhER+V75jSzbwmsNClZvGZlDxlx/AVyZGkGG?=
X-OriginatorOrg: cornelisnetworks.com
X-MS-Exchange-CrossTenant-Network-Message-Id: bffe5d60-7819-4f40-7e8d-08da292dbc15
X-MS-Exchange-CrossTenant-AuthSource: PH0PR01MB6439.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Apr 2022 15:42:43.4795
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4dbdb7da-74ee-4b45-8747-ef5ce5ebe68a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: Id1C4Hsj6dVw9BtyBmw8Qd7CevVNzfJ6iGurvsElTcu466Xg38iJcjx0w3YOmJOFR8+U1/AEYsgSbGKNAN/B5t1IqxLJ59I8rtnv/lcyCQ+FBeI+nq1xGaswTLiqgXy2
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR01MB5530
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

On 4/28/22 10:57 AM, Chuck Lever III wrote:
>> On Apr 28, 2022, at 9:05 AM, Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com> wrote:
>>
>> Hi NFS folks,
>>
>> I've noticed a pretty nasty regression in our NFS capability between 5.17 and
>> 5.18-rc1. I've tried to bisect but not having any luck. The problem I'm seeing
>> is it takes 3 minutes to copy a file from NFS to the local disk. When it should
>> take less than half a second, which it did up through 5.17.
>>
>> It doesn't seem to be network related, but can't rule that out completely.
>>
>> I tried to bisect but the problem can be intermittent. Some runs I'll see a
>> problem in 3 out of 100 cycles, sometimes 0 out of 100. Sometimes I'll see it
>> 100 out of 100.
> 
> It's not clear from your problem report whether the problem appears
> when it's the server running v5.18-rc or the client.

That's because I don't know which it is. I'll do a quick test and find out. I
was testing the same kernel across both nodes.

-Denny