Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <04edca2f-d54f-4c52-9877-978bf48208fb@cornelisnetworks.com>
Date:   Thu, 28 Apr 2022 15:47:08 -0400
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.7.0
Subject: Re: NFS regression between 5.17 and 5.18
Content-Language: en-US
From:   Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com>
To:     Chuck Lever III <chuck.lever@oracle.com>
Cc:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
References: <979544aa-a7b1-ab22-678f-5ac19f03e17a@cornelisnetworks.com>
 <8E8485F8-F56F-4A93-85AC-44BD8436DF6A@oracle.com>
 <9d814666-6e95-e331-62a7-ec36fe1ca062@cornelisnetworks.com>
In-Reply-To: <9d814666-6e95-e331-62a7-ec36fe1ca062@cornelisnetworks.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?djc4K1ZjL3UxOEFTU05OT24ycGdrMkJmRXVyUzBGZW5TVGZ5WlF0TVFoWWI2?=
 =?utf-8?B?Zm1weUVGYTNQR0RKSmlac2wvWTRxakhMTUYvNDU3bWxOd2lwcUExRktkYWVh?=
 =?utf-8?B?d3RUVUl5c243N1hQK0RCdDVJYkUrT1JDcmV5RHk5MHJmbldlRVIrQTBwSVE5?=
 =?utf-8?B?aXM1R0ZXNm42aUZUOW8wZGtYcHloNytQU1lrZGd6NERtVjJjSGdweHByVnlM?=
 =?utf-8?B?WFZlOWFQMW1yU1N4VzZXaGNZRkZuNWt2U3l2RnlBVXZHY3hESTlkbmtWc0tP?=
 =?utf-8?B?akxIb1M2Q3ByZm10bzB1czBRTkxaWnRsd2kxZjJiRzhxTWVrSDl2aW1wdWpN?=
 =?utf-8?B?Nnp6UXNJWUdUUU00WlRWemZLRzhWQ003b2F1YlptMEg0Yng3aERzNkc5ZmdN?=
 =?utf-8?B?WlRtRTN0T0hqT1duaDNmenVPblVSWWkxeGFDV0NHYkZ4MDhHcnc5Q0d4ZVVr?=
 =?utf-8?B?bURyVVhlbEtEa3hHdUJ1VzJRWE05QmZXQkVoM3ZiSTNhWnJ3bkZwSzFIOWhh?=
 =?utf-8?B?ZHhiWHBkek1Na1lWdDVuNkpwK2I0WmFHMzZOeGh6aFdxUUZucUF3c3hvaDFm?=
 =?utf-8?B?Y3NhclVBOWdNZVNtOTg3SGNVeHFja0E4S2Z2SXVjRFhJU3N3Vm1VWkUrNXd5?=
 =?utf-8?B?Rm50d0xJWUpGOFlLSkpRcmxaSDhHNFFhWE9pV2ljWGJvV2QvcEZCRjdUYVlL?=
 =?utf-8?B?TldCdENTUFFSc21DWVVjTGdCZkEwMlVKTGNtSFd3amo3ZGdwSGVDbG82VS9k?=
 =?utf-8?B?MWx3TDFjVFk0ZWxTN3VmREp1eE15OG14NlhKcTNuNXRoajlkNzBvOEJhL1FU?=
 =?utf-8?B?elBYWVNmNEhmOTl4V2xOREdoMmVnUjB6YUpVUmpLOVNLdWpqNVRHR2w4M1hU?=
 =?utf-8?B?ekJlbjMzNEFqRnhBWGk4Y3hPVWcwaDRiRXA5TjVpUnlUdkFINjJoOFdrKzZk?=
 =?utf-8?B?ZE9zUVdjbUpueHZKc3YvbCtoUkdJRWluSEZSd3lwYlFQc2lwUjEvTGRpUEZI?=
 =?utf-8?B?elVuSVcrVlR1cEZ1dSswWHhIZXl1Y21JV2xwTDk1UDFDKzlDQlZFeDk2Q0VQ?=
 =?utf-8?B?bEpPWlFLYnpqd0tzTWtGb3I1S0tmVnE3bGR2eWIySUtzdUZzSVYzY2ljak0y?=
 =?utf-8?B?L0Q1V3pzYkYzS290ZHJvWXN2ekpZSThoNE1oUGY5NDdYUk9TSFNvd0pVSzR4?=
 =?utf-8?B?T1JDY1U2OXdpeE1LaTJ2clBOZVZnMit5SnlCbmoxZmZGV0I3S0lVZkVlVHZv?=
 =?utf-8?B?dUd0T2p5U3pOWUN5SExMOWZockx1eXFDc3kvbkIxNzlYdldWS3h0LzdReGFZ?=
 =?utf-8?B?NGhLbmdXS1JYSDVuUXdNOXlyN05uMm1aZUFoSnRzVkIvbzdSeTJ6cHRuYmF5?=
 =?utf-8?B?a0M0UU5UQXgwK2IwNTFNM3VGallhWGlqeXRlaEQ0YXRxdWt2NWF1Q2JwV1BJ?=
 =?utf-8?B?ZitwblR1a096cFBpQ3NMRVhrTTVMemdzVmdzdHZyNUdlMU1UWHZQaVh6aEE2?=
 =?utf-8?B?TjhHL2dMemRmSzdWemFQa09uOHhvQ3FQbkFrdlI1Y09RWG8yeXpoUnFHQVFB?=
 =?utf-8?B?U1BEcUNxbTlicHZ1bHJIZTdJUEVkWXY4S2YzK3BjdGNsUkh4S3ZxNkwwb1Yw?=
 =?utf-8?B?M2NQWjQ3elRZakpZNC9MOUZDTXc4eU9JTTF4d3Njb01zSnd3YWFNV0g2NWcy?=
 =?utf-8?B?aXJFbXVLb3MrY1RYRk9vN3I3L1dxSWJZZXFBdGU0emJXYm5VY2dsVkF2cGNJ?=
 =?utf-8?B?UGtzR0dSK3ZhUXYveXV3Nk9ZWk5ZVzRZbENPbisvcmNEMGhBZUxENHFtdG1y?=
 =?utf-8?B?VFltd1lmbVZqemdwU281ZUVwU1RXWFRWUjA0T09IMzl2eEZ4Y0Jtei9pcUlR?=
 =?utf-8?B?NmljM3BzeW43YzZHSS9lMUY3a3lPRlpsc1h0U2dNOWJJM3ptTFZlTUt1TGhU?=
 =?utf-8?B?Q0tKbkJNTWNKRjUwMkduZlR0TzZkL2VweVdQb1ZBT3V0YzloWFI0NHJhYzdl?=
 =?utf-8?B?VGZWd0FoUURyazZheXBsL2tiaVl2RXdyOC8rOUh0YllVeFBPSkE2YXpoVmRl?=
 =?utf-8?B?MEsySmFkOVhnclVhWnhBNVc5Vk1NZHdCOGx2WGRQaGcrR3ZNOEhzdHMxWnE0?=
 =?utf-8?B?MUZBQldUeWVURGtpWUxRQ2ZBYnRjMWlZLzBJL3l4RHA2dnJvdmR1cmZrZkF6?=
 =?utf-8?B?L3o5ejRtSHhYaHBTRlEzTHpKbU1KbUdkdWdTbDcrT2hwMkl3QWtQZEplRDF3?=
 =?utf-8?B?dEdteEQ4eHkwVnhFTmpuNHYxeWhlSGxPMCtBS0FSZnB2Zi9hbXZnTFJEWFJl?=
 =?utf-8?B?djJvdkcyblZzanMyWXZicmw4bWd4MGJkYmpwUWMwZHFLL20rY1VlNitPYlUx?=
 =?utf-8?Q?MsrTexAxa3NgrVkSjnKY4/S/IlzOrgWg6/dTK?=
X-OriginatorOrg: cornelisnetworks.com
X-MS-Exchange-CrossTenant-Network-Message-Id: bbb6b095-8dde-4c1e-10d8-08da294fe2de
X-MS-Exchange-CrossTenant-AuthSource: PH0PR01MB6439.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Apr 2022 19:47:11.4673
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4dbdb7da-74ee-4b45-8747-ef5ce5ebe68a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: DoOzgcmCHH4GfBResv2UjS2G8ZnY6cqYd20/Iuy+I89aohbXGPwaslVJZU5SYR+WNwKB+S4ElT1ueUWkHz2areJVQG0qnwavtcNvclSh47A0Pg9BmKMDXq5bUzKuKxzL
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR01MB4642
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

On 4/28/22 11:42 AM, Dennis Dalessandro wrote:
> On 4/28/22 10:57 AM, Chuck Lever III wrote:
>>> On Apr 28, 2022, at 9:05 AM, Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com> wrote:
>>>
>>> Hi NFS folks,
>>>
>>> I've noticed a pretty nasty regression in our NFS capability between 5.17 and
>>> 5.18-rc1. I've tried to bisect but not having any luck. The problem I'm seeing
>>> is it takes 3 minutes to copy a file from NFS to the local disk. When it should
>>> take less than half a second, which it did up through 5.17.
>>>
>>> It doesn't seem to be network related, but can't rule that out completely.
>>>
>>> I tried to bisect but the problem can be intermittent. Some runs I'll see a
>>> problem in 3 out of 100 cycles, sometimes 0 out of 100. Sometimes I'll see it
>>> 100 out of 100.
>>
>> It's not clear from your problem report whether the problem appears
>> when it's the server running v5.18-rc or the client.
> 
> That's because I don't know which it is. I'll do a quick test and find out. I
> was testing the same kernel across both nodes.

Looks like it is the client.

server  client  result
------  ------  ------
5.17    5.17    Pass
5.17    5.18    Fail
5.18    5.18    Fail
5.18    5.17    Pass

Is there a patch for the client issue you mentioned that I could try?

-Denny