Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com>
CC:     Trond Myklebust <trondmy@hammerspace.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: NFS regression between 5.17 and 5.18
Thread-Topic: NFS regression between 5.17 and 5.18
Thread-Index: AQHYWwCixuCyjj49XkSuf+bm0GIRX60Faq4AgAAMfoCAAERPAIAAApwAgAEchwCAAAwEAA==
Date:   Fri, 29 Apr 2022 13:37:51 +0000
Message-ID: <A04B2E88-9F29-4CF7-8ACB-1308100F1478@oracle.com>
References: <979544aa-a7b1-ab22-678f-5ac19f03e17a@cornelisnetworks.com>
 <8E8485F8-F56F-4A93-85AC-44BD8436DF6A@oracle.com>
 <9d814666-6e95-e331-62a7-ec36fe1ca062@cornelisnetworks.com>
 <04edca2f-d54f-4c52-9877-978bf48208fb@cornelisnetworks.com>
 <ca84dc10f073284c9219808bb521201f246cf558.camel@hammerspace.com>
 <bb2c7dec-dc34-6a14-044d-b6487c9e1018@cornelisnetworks.com>
In-Reply-To: <bb2c7dec-dc34-6a14-044d-b6487c9e1018@cornelisnetworks.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?vhvihnOCcg0zDYHb+Novb9tAyf+nUujz28gXRR4wBKsyBOybep59r1FUlrwx?=
 =?us-ascii?Q?qZb3mYP9uJx4WIb4V2zp9t/ai8OPjmX71+UjrffMEMpi4JIGQ7cgiV0VcEoG?=
 =?us-ascii?Q?4siIzAV1NnVhMo++Bbm9XFlgZguuRButQel/55wv7QbG+mCQEhU+HaA/flAt?=
 =?us-ascii?Q?c97HZTsU6fmOp8XLDvpBO3Egz6PQ0+dN5UUIiZoN5u2v3O7k9HAT5zeibXRb?=
 =?us-ascii?Q?En7EkfPVUFsuV5panYIvQym1CylfMOZ2qlvhogLkzHKetUDH6SpZMYWTm5WQ?=
 =?us-ascii?Q?XvOPPXTuz8sWnfhlC1P6HDwNmW+wra1Z92Wk8uhPVqIHT2tql4oCKL/Ygrgh?=
 =?us-ascii?Q?NUp/ZhvypNSDJQxJmL1b9zU5KisGA1uV0w+GjzjeW4hg5B9NNUxJmQrV0Ume?=
 =?us-ascii?Q?7t4v73RVBHj4z4mof1yAk9jw0N6Bw7/SnOCu03w18GKu0TJD6wtu+DMKuqbG?=
 =?us-ascii?Q?TT2hcyHNbWdtHJZB07U11v6ZhxoWttSUU+dyU4J69Qm9c6/KWB0OlJ/KbN2b?=
 =?us-ascii?Q?RHCGpLxz/6M9Rh2TiQyD3qGYyZk0Fl4FqX4tz0nF2j1+emn/1PbtuwHDbfL/?=
 =?us-ascii?Q?FEgBRrlGzZ9raZf6KxaDVgWI96MWX1mLNh9pA2yhU/38oQztGaWkNLNmnIE+?=
 =?us-ascii?Q?mRhuCOrDS6qEhsGui8PpM5A+UXgZDtCvvAHD3wdsdHnK0dm5Y6gzyHsQFLGW?=
 =?us-ascii?Q?XIOiDCjn9gv2gbgIsjpTXzYWyDtT5eUV1Fkcf5WzLyQUs1uU8XjmDvzsuo4R?=
 =?us-ascii?Q?hyN4SJyrhNDH62QWmqtx2/2st2y55r2GAUbVQt+OATzqbIKZubi5HAYB9J9d?=
 =?us-ascii?Q?ycVPzdfaffgFwzWtKCdhJue02pBisiwE1r085qLbgQ50+zOjNJhJKu1piaD6?=
 =?us-ascii?Q?xr13WJW4GTs3BKk4gOAIuioKyzmo4lMnBGCE2GzGatFFe82iRUgQvnDKxdnN?=
 =?us-ascii?Q?1ueCeTlM7AvExtbwcHaSDXeVF6cJkJRgtNQ1qHU2Vp3RiMUETndHbD5jo+2H?=
 =?us-ascii?Q?z5/8soJ9On/EoRTsQsoEZyoJRiR3flaDMCUsqGOtCll2lv6xUHTpDY2M5/kt?=
 =?us-ascii?Q?ksBIrpF/rxUj/rIdLV6lpEoC3VK4ZrTckHb0sqb05TH7C5R1+yfyQAnhnqEz?=
 =?us-ascii?Q?S0WMiMNcgv51uSABby1n/8KgzacmB8IcTXPH7fSVW1hJcQN03dw7KB42Md93?=
 =?us-ascii?Q?VA2VcQwL5Pt7XWs+HTxlDPI0gItUq4bHJ3ErsokZCh1BIki0kbcF0ctyQPwG?=
 =?us-ascii?Q?J6QxojVDxT6y7y+eIovBdWc1Ht5h/nnkGvkK0w/bVoxUr279XdNVoUyBJFse?=
 =?us-ascii?Q?EeQWcMeydIqtcRNYKwAzDNRrdPR+D1fS5AKPOV56c3gpXrftSH6s97Zh2wRx?=
 =?us-ascii?Q?W+3NUYAbD6tmxugyzzrx0ZS/DNCgj9/MXgdn9mOWM64rOklpIO7gxvY8Jx8W?=
 =?us-ascii?Q?jOge0q+arboTBPP5d4xojnF0AcLfYUEpCVCGlEr58M/EdgKAMAHG8zZqxlf6?=
 =?us-ascii?Q?5J4Nfs+D6WXPHUVUFaSU94ldiSJ8e4Cjsy6fmTLtXgaDny9zxppscMXTId+l?=
 =?us-ascii?Q?6L6wBusu09lclgcP7R/zBsww8d2e+/KkmJ6uOSctZUG8B7T0R3DV0rdzYmZS?=
 =?us-ascii?Q?qpqA6Oaw8H7k7WER6wSpgvQmBUegbgRZHQPQzusXPbwPkm5YsSsBYh2oMV/I?=
 =?us-ascii?Q?gq5KtI+WZIM02YJ2G+p+eGxU6/1VkyOUUjHG6losLXJ1+zevY3OOAbyiSLFw?=
 =?us-ascii?Q?BdnO+kHgny9+7UcsuKlcmdPYb1Cx0Vc=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <05652FFD3F225A4393CCF0CDF7498D9C@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 25136ec6-5507-4fb1-7cf3-08da29e57543
X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Apr 2022 13:37:51.7797
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: FvIfOVZogatPojL/KKZpYWBHgUk7p9oJzPQSeomjzVMMNo8xkbmGtmvq4G0uyq6Tx4M6tn/46b9KOaJrDNg1Tw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR10MB1975
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.858
 definitions=2022-04-29_04:2022-04-28,2022-04-29 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxlogscore=684
 malwarescore=0 mlxscore=0 phishscore=0 bulkscore=0 adultscore=0
 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2202240000 definitions=main-2204290078
X-Proofpoint-ORIG-GUID: 2bpJz43kP1sFFfzQOumwAUef0Wqw0eI_
X-Proofpoint-GUID: 2bpJz43kP1sFFfzQOumwAUef0Wqw0eI_
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,SPF_HELO_NONE,
        SPF_NONE autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Apr 29, 2022, at 8:54 AM, Dennis Dalessandro <dennis.dalessandro@corne=
lisnetworks.com> wrote:
>=20
> On 4/28/22 3:56 PM, Trond Myklebust wrote:
>> On Thu, 2022-04-28 at 15:47 -0400, Dennis Dalessandro wrote:
>>> On 4/28/22 11:42 AM, Dennis Dalessandro wrote:
>>>> On 4/28/22 10:57 AM, Chuck Lever III wrote:
>>>>>> On Apr 28, 2022, at 9:05 AM, Dennis Dalessandro
>>>>>> <dennis.dalessandro@cornelisnetworks.com> wrote:
>>>>>>=20
>>>>>> Hi NFS folks,
>>>>>>=20
>>>>>> I've noticed a pretty nasty regression in our NFS capability
>>>>>> between 5.17 and
>>>>>> 5.18-rc1. I've tried to bisect but not having any luck. The
>>>>>> problem I'm seeing
>>>>>> is it takes 3 minutes to copy a file from NFS to the local
>>>>>> disk. When it should
>>>>>> take less than half a second, which it did up through 5.17.
>>>>>>=20
>>>>>> It doesn't seem to be network related, but can't rule that out
>>>>>> completely.
>>>>>>=20
>>>>>> I tried to bisect but the problem can be intermittent. Some
>>>>>> runs I'll see a
>>>>>> problem in 3 out of 100 cycles, sometimes 0 out of 100.
>>>>>> Sometimes I'll see it
>>>>>> 100 out of 100.
>>>>>=20
>>>>> It's not clear from your problem report whether the problem
>>>>> appears
>>>>> when it's the server running v5.18-rc or the client.
>>>>=20
>>>> That's because I don't know which it is. I'll do a quick test and
>>>> find out. I
>>>> was testing the same kernel across both nodes.
>>>=20
>>> Looks like it is the client.
>>>=20
>>> server  client  result
>>> ------  ------  ------
>>> 5.17    5.17    Pass
>>> 5.17    5.18    Fail
>>> 5.18    5.18    Fail
>>> 5.18    5.17    Pass
>>>=20
>>> Is there a patch for the client issue you mentioned that I could try?
>>>=20
>>> -Denny
>>=20
>> Try this one
>=20
> Thanks for the patch. Unfortunately it doesn't seem to solve the issue, s=
till
> see intermittent hangs. I applied it on top of -rc4:
>=20
> copy /mnt/nfs_test/run_nfs_test.junk to /dev/shm/run_nfs_test.tmp...
>=20
> real    2m6.072s
> user    0m0.002s
> sys     0m0.263s
> Done
>=20
> While it was hung I checked the mem usage on the machine:
>=20
> # free -h
>              total        used        free      shared  buff/cache   avai=
lable
> Mem:           62Gi       871Mi        61Gi       342Mi       889Mi      =
  61Gi
> Swap:         4.0Gi          0B       4.0Gi
>=20
> Doesn't appear to be under memory pressure.

Hi, since you know now that it is the client, perhaps a bisect
would be more successful?


--
Chuck Lever