Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <bb2c7dec-dc34-6a14-044d-b6487c9e1018@cornelisnetworks.com>
Date:   Fri, 29 Apr 2022 08:54:50 -0400
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.7.0
Subject: Re: NFS regression between 5.17 and 5.18
Content-Language: en-US
To:     Trond Myklebust <trondmy@hammerspace.com>,
        "chuck.lever@oracle.com" <chuck.lever@oracle.com>
Cc:     "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
References: <979544aa-a7b1-ab22-678f-5ac19f03e17a@cornelisnetworks.com>
 <8E8485F8-F56F-4A93-85AC-44BD8436DF6A@oracle.com>
 <9d814666-6e95-e331-62a7-ec36fe1ca062@cornelisnetworks.com>
 <04edca2f-d54f-4c52-9877-978bf48208fb@cornelisnetworks.com>
 <ca84dc10f073284c9219808bb521201f246cf558.camel@hammerspace.com>
From:   Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com>
In-Reply-To: <ca84dc10f073284c9219808bb521201f246cf558.camel@hammerspace.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?cWdWUDJIdnUrblNGNzMxeU9jZ3d4Skx0dFZvMzllb09xclgvYjNVbVVxSEI1?=
 =?utf-8?B?K1ZRblB0ekRqQ2Z1RThoeFhIV24rakZyY3U1dVhhVHBhaVoyakdyOThSREMy?=
 =?utf-8?B?ZjBkQVBLSVpNd1dJT1N0THdieGZxMVBIZFdKeFlCT2lPRHlnSXdJT05UbEc2?=
 =?utf-8?B?SmxRRWpBQ1FuZlljd3hra3dlUktDcGFlT3VoMlh1d1BWaXlhbmw1ZTZ2cEtn?=
 =?utf-8?B?Q1lXR01LNHpoMUsxeTdsZ1d2T2Z2elRqdTNpS0U1a21kdlBMNU0rU2hVa1hF?=
 =?utf-8?B?dVhtVVYzVmZxVlpIejFqL1NtdGJWZGkwV3I5UFJEVDVqaUxBeEdkWE44M1hU?=
 =?utf-8?B?TXZ2SHZZNFZRTk5Wd2pYSXZycEg5d3NZWWpKSitPREw4ZHRZNEoxSWVnRFcr?=
 =?utf-8?B?TDJhLzV3d2N5WWMwaFlzQzRiQldMNnQrdEpCSGo4SkRRTzFRV0MxY2p3NFFj?=
 =?utf-8?B?amVaMmNBR3N2ckhPYWdqSGY5bk5jZXFoUVI4VDF6R3BWQ0g0WEVSWEZzbVZt?=
 =?utf-8?B?NnkyWTJYbE1pRmhrNGMyY3ZTKzJQbkdqRkNZaXcxcFUyTHlaVGxuY1RwRmo0?=
 =?utf-8?B?Nm1zZ3IvMUNKcU9oT1pDa0tBTk85Tnd4K0ZlWjNqaVRjTmxZWWNyNVZkZ2hu?=
 =?utf-8?B?eVNsZXNBVHJNTFB3Um5kZDBhdlQvUFFIZTdpTkc3UXhMNmczMnB0SjJQam5M?=
 =?utf-8?B?aXVuZGVRdjRsQmJ1Y3NRMndEbk0zcmJVSjFhQ2RtNDg1MFhJSitQUWs4bllW?=
 =?utf-8?B?ekxSVnhjV3pDamJ1T2xQK2hRaTdqL1F3bE4zRUdLdWNmNnJSTk9IRDU4ZmNx?=
 =?utf-8?B?MHZqbnRPRnJQMXhvYUwxVWU4NTRvcDd1MTJTMC9qcGEzY0tRVFFNd0FHYWo1?=
 =?utf-8?B?c0lJV1pXc04zb2VTMVMvdWVyaDBPN29Ya2l6ck5ZV0VGQ1QwWkVtYTVwakQv?=
 =?utf-8?B?REptV25ObHczT1NwaFg3UXVabFNzemVTMjBuRi84UzJlcklJcUlsMGNjZlJY?=
 =?utf-8?B?YkVLZDJjT0M3U0VYWlpKL3IyaHlVMzNnengvNjVhYmpOVnE4d01jMUFQenpX?=
 =?utf-8?B?ZU01eldmT0dYNGEzYUdDamtEQjZHNjNRb3pLakw1bU5OWGg1UmwzaGlHT0RE?=
 =?utf-8?B?S1VuUzNMcnVxbGJFTXY5VU9IaVdUUk93UTRjNXFwU3laY2kzOTZ6U3lFcnp0?=
 =?utf-8?B?V21ZMjNLNjZ5eFhhcGNLSGMzOElYRWFLbFE3NG5adkV6ODdvdHY5MnhOK3VV?=
 =?utf-8?B?bGQ2THFxUVpscTBmN0RTbXNLc3BlTWVGSUJoVC9WNmpNV2QxcjUyQ2hLMTZE?=
 =?utf-8?B?MVdGMVB2am81VmU5K3hxSzNWR2lncTZkQmNzT01aeUk0NkYwZXJYYmJRdmhL?=
 =?utf-8?B?L3Bvb2pYMXlXdWVhbHdRWURwQUJlUUpqdUVWaDhJQ1BQcHBHcUxMYmh1N0l1?=
 =?utf-8?B?ZnZRQ01uTzl2Y3BoUFp2blZweVd6K2x4WHdkV3dSUjdEMlQ2TG5zOSs1QVI1?=
 =?utf-8?B?RWpwY3JQdloyTlRYWU4rTUdtVVoxV2l6ZUw0aFBMTTZoUFZTTjBJTWZRZS9D?=
 =?utf-8?B?YkFYWlV1YkNLU0J4ZC9kMi85Rk0yQUhBM1F1K1RjU09Jb010a3EvR2ZxeDZE?=
 =?utf-8?B?OFZUU0MreW9VMXFwbkNONWJNUENzbFZGUEVqdjAzMVNXSDZlN2hYTEdsZ2Z0?=
 =?utf-8?B?eUxPdS9aOWdnSXZLUThianpkT255UlozSFZFbWFmSm5hL2hLWGE5QnlPSUtx?=
 =?utf-8?B?VVJnWXh2b3RLZ1VjTFBrSGZsZ3YyOFJZWUlWRnE1Yk9VTC9JOXpUVVBLVFlW?=
 =?utf-8?B?dzk0SzRFNXNqVGFLWnNlWURnNEZSQmxiL0hnNC85QkhQNi9mdzVnQ1p0WGlM?=
 =?utf-8?B?dWJWMGkwRGFSZ0YvekYwN2VuQ25XeU1IYVMwVGl2ZzBYdy9EVVVDZWdLUkY2?=
 =?utf-8?B?R2NEZWFVVlpyQ3BOUjc4RUJlZWRlK3BRVk8rVkc3S1RiekpxTGRZYUsvRUdo?=
 =?utf-8?B?NjNwcFRwc2NQVWl0bXZDMW5HcU1kUWpITG42Y3A4a3FWNHNRRUlkanhPM3BE?=
 =?utf-8?B?RVVvTGhZUGZFcWVRVGNJa3BrQnRkZ2NlWmNUUDJmTkZRdEl3MWt3SlBOR1JP?=
 =?utf-8?B?aGxkd0dBK2xiaXVZWW5uVVptQUluZWVOYm90KzFlTGxkd01NOUtXbnk4YmFP?=
 =?utf-8?B?SHpRZHhEa3U4TFNsaFVGN0R1NUVGaE00OFN5S01ZMysvUFIvSlJjVS8rNzl6?=
 =?utf-8?B?Q3g4WVZHQlFpQ005OGk3SVJWMDU5ZnNHeVdZQjRYSFdMYzVPeTZGb2kzMS9F?=
 =?utf-8?B?TmcyK1Rkcyt6cy9kYzZhcGZXSW92eXRDNXhxbHZiRXJoa05FU2pJYTJCV3Jk?=
 =?utf-8?Q?jmNuISET2TQLxzg6xSi6nUok59mjKnYcUBJnG?=
X-OriginatorOrg: cornelisnetworks.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b8c8461a-0a97-4862-bdeb-08da29df749c
X-MS-Exchange-CrossTenant-AuthSource: PH0PR01MB6439.prod.exchangelabs.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Apr 2022 12:54:54.1282
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4dbdb7da-74ee-4b45-8747-ef5ce5ebe68a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: E1xnrb8hGfUJBy+yWcbWWQTIZFsZVtpLe+MpcspIjs//SzIjeWL2R8nCId/e8g9ZZzHxqn77N+T/xp1ME0A/XxVMUmBt08j8jGkY1GziD2/bN7Byij5StedSyNWSZ5GY
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR01MB4068
X-Spam-Status: No, score=-5.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_NONE,
        RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

On 4/28/22 3:56 PM, Trond Myklebust wrote:
> On Thu, 2022-04-28 at 15:47 -0400, Dennis Dalessandro wrote:
>> On 4/28/22 11:42 AM, Dennis Dalessandro wrote:
>>> On 4/28/22 10:57 AM, Chuck Lever III wrote:
>>>>> On Apr 28, 2022, at 9:05 AM, Dennis Dalessandro
>>>>> <dennis.dalessandro@cornelisnetworks.com> wrote:
>>>>>
>>>>> Hi NFS folks,
>>>>>
>>>>> I've noticed a pretty nasty regression in our NFS capability
>>>>> between 5.17 and
>>>>> 5.18-rc1. I've tried to bisect but not having any luck. The
>>>>> problem I'm seeing
>>>>> is it takes 3 minutes to copy a file from NFS to the local
>>>>> disk. When it should
>>>>> take less than half a second, which it did up through 5.17.
>>>>>
>>>>> It doesn't seem to be network related, but can't rule that out
>>>>> completely.
>>>>>
>>>>> I tried to bisect but the problem can be intermittent. Some
>>>>> runs I'll see a
>>>>> problem in 3 out of 100 cycles, sometimes 0 out of 100.
>>>>> Sometimes I'll see it
>>>>> 100 out of 100.
>>>>
>>>> It's not clear from your problem report whether the problem
>>>> appears
>>>> when it's the server running v5.18-rc or the client.
>>>
>>> That's because I don't know which it is. I'll do a quick test and
>>> find out. I
>>> was testing the same kernel across both nodes.
>>
>> Looks like it is the client.
>>
>> server  client  result
>> ------  ------  ------
>> 5.17    5.17    Pass
>> 5.17    5.18    Fail
>> 5.18    5.18    Fail
>> 5.18    5.17    Pass
>>
>> Is there a patch for the client issue you mentioned that I could try?
>>
>> -Denny
> 
> Try this one

Thanks for the patch. Unfortunately it doesn't seem to solve the issue, still
see intermittent hangs. I applied it on top of -rc4:

copy /mnt/nfs_test/run_nfs_test.junk to /dev/shm/run_nfs_test.tmp...

real    2m6.072s
user    0m0.002s
sys     0m0.263s
Done

While it was hung I checked the mem usage on the machine:

# free -h
              total        used        free      shared  buff/cache   available
Mem:           62Gi       871Mi        61Gi       342Mi       889Mi        61Gi
Swap:         4.0Gi          0B       4.0Gi

Doesn't appear to be under memory pressure.

-Denny