Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Dennis Dalessandro <dennis.dalessandro@cornelisnetworks.com>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: NFS regression between 5.17 and 5.18
Thread-Topic: NFS regression between 5.17 and 5.18
Thread-Index: AQHYWwCixuCyjj49XkSuf+bm0GIRX60Faq4A
Date:   Thu, 28 Apr 2022 14:57:56 +0000
Message-ID: <8E8485F8-F56F-4A93-85AC-44BD8436DF6A@oracle.com>
References: <979544aa-a7b1-ab22-678f-5ac19f03e17a@cornelisnetworks.com>
In-Reply-To: <979544aa-a7b1-ab22-678f-5ac19f03e17a@cornelisnetworks.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?2A9P3q5zFgM4Mxve/mwmeRDT+KhW0ETts3foQs+Zjk3gPvdrBihuHQzezfsE?=
 =?us-ascii?Q?PvUnSqaQD4fw4SEzVvl2hOrjHj4mMUjI2koXx4fP4YHzFHabXNgZ1VDizzVg?=
 =?us-ascii?Q?dCDTp0JBNkY8v0Gtdse3F5oZfWe75Kq/DHa4+X19hQq9hmZFsxkz7Oe0jaRN?=
 =?us-ascii?Q?YEPwKToJiigXYWqQ/fML1LdZQ2QE7SlmT1WgQvtFLL3RJ8tAfSi9hQ1mvTL7?=
 =?us-ascii?Q?N7hngZS9mTVW+q4iR2r8SYN9+1DflSz06XWjJTv0HMWm/R/7VBgni0cc1FMf?=
 =?us-ascii?Q?0Mlq09N7yHetui67VlLTEa3DK1rP22SnlwtMP901yPNbtnDvscW/A0G8QPrZ?=
 =?us-ascii?Q?8HsE6bV6WtrzKv/wxgX+f7DQ0imqkrlvWOPPij8RGS9rrrq+1jiItid9sgo4?=
 =?us-ascii?Q?qbZ2YCWUcomOlydWqgCUA8sEDwNJsIR5YYvIEszaSDwQ4IwY5WuznqPYPjq+?=
 =?us-ascii?Q?Mc/9qkwXpFykIlPPCj8ir2tliPvjyO74hLYSpbNBat+ExlYuKpO0DFlgGkGN?=
 =?us-ascii?Q?X9oaq0oMjVmtWGwQO0GGSlG5pj4WMMopbs/O9QKCDzTGixpwahBECMfwr/ir?=
 =?us-ascii?Q?MhQY+Tu0o1iTYZgu4/Y0tG4vzjOU1QkeINVMg0qJoQU8fGXArXik4RjSIbWI?=
 =?us-ascii?Q?meb3cDdKpbiPUqcq3kpno4XbjHCStpJYvf5rqHTVM4hWUolcVlEEvUiskVwh?=
 =?us-ascii?Q?3X2SEQdAAkhSanqHxg14sLpM98+9YpIt5D0uHAXcthZdgjGKzuu4tKxYJ2Oi?=
 =?us-ascii?Q?BvadsQjWmU1UsaJighpVTyRJYvOkMNGsGW+y3FdoZP5fW0hMsgqqqP9Tq9K3?=
 =?us-ascii?Q?P8pw8+Ref2w2b9E58GWteCbROSR09E+4e1aZYqJwDWoYQrXrbM5RVPzbIqdS?=
 =?us-ascii?Q?gSfbyrTHDb6lewQHFHOge2C28ES85qO9OnRpp4Z2pWFDYLoGVBDytt1znh/F?=
 =?us-ascii?Q?C5CG4VcXPmOs4MlB+ood8e62d6UHWE8stfsZY249lzBtuIF0/3FD8xtD0xdv?=
 =?us-ascii?Q?q2laCuH7sCKFFnlln5se2A/VrI8oySYsgEKrUq2cW4AdsvyNAn1tXkqTcuww?=
 =?us-ascii?Q?6Zzx49UCiMol/nzvgvspadD6DTqHfPZCOPc0wNzJgnfJ1TYVh9ggl5jMKqQm?=
 =?us-ascii?Q?78usfjIQV9htGgOl4MNGnSUTq2q7y0hKc+NkOC+kGPJ7chrG5/O8iYs11o4m?=
 =?us-ascii?Q?DwHiUshssHebvGKzwKPybt8RJ4UHrvYzmPodzn0qk6WTHN/Y84L60s4R0Vgx?=
 =?us-ascii?Q?HpuRVnKA5yq+Mfy6YbIx0JfwRyiHUqLyRPLrpphrOJ3JziN2SQOsEBh5Y2ZZ?=
 =?us-ascii?Q?LpfTPN6jdkDi3dSrcf480d/dvVBCdnsa8L3oEszIkHt5JdihZd0UCHmGaW1s?=
 =?us-ascii?Q?G0rh0JoMT6e/uAYIuxuhNsqxPfTjh/RTd/Y2u8qJR0fJ66arEL3Jn/RCtHvD?=
 =?us-ascii?Q?gjm4hXTGAtMv2lfbs51cH5E/kHjhmYMijraLumjghRmtGym2rvKUNESVqZXj?=
 =?us-ascii?Q?Ds7wTN/jkcS2vj655vC7hNPQyooGprVrHS2vOEn+qrC7sVoCiU0DifPpXDNR?=
 =?us-ascii?Q?kVJ3yam8XqGTjPLzbwxTIZcjSycjDds8QdubV9ukxnOUHXveBnZa6webG1+G?=
 =?us-ascii?Q?4tvj+YR4XWIBLC8+u0AwOnp5Mv9XTIU6ETesSYiASNDfKYPNsrU6Ey20pB+P?=
 =?us-ascii?Q?Gz3FD8v0H5enHqkSNafeMFM9d5zKc0VUcoXgtxautD4prp8MwTh/osDisJ/w?=
 =?us-ascii?Q?iKX6zz7aE2HzFNfWyQZhNCnOB9+yg90=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <2F09E4DCCF02C04BA60AC5FC64419915@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5128.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a04e504f-378a-4d28-171c-08da29277ad7
X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Apr 2022 14:57:56.7101
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: NgrSKD7d5CvSUFsbZOri5b9tJgQAJI79f9czORpReVKtiCBRLv7Zlg+gfFitdZQWXVdw1C5tEjwxNJCL0IfxdA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR10MB3316
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486,18.0.858
 definitions=2022-04-28_02:2022-04-28,2022-04-28 signatures=0
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 spamscore=0 adultscore=0
 mlxscore=0 bulkscore=0 suspectscore=0 malwarescore=0 mlxlogscore=826
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000
 definitions=main-2204280091
X-Proofpoint-ORIG-GUID: BG30O4lUbyEMCHnRucp8PzNqLKLAxABA
X-Proofpoint-GUID: BG30O4lUbyEMCHnRucp8PzNqLKLAxABA
X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,RCVD_IN_DNSWL_LOW,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Apr 28, 2022, at 9:05 AM, Dennis Dalessandro <dennis.dalessandro@corne=
lisnetworks.com> wrote:
>=20
> Hi NFS folks,
>=20
> I've noticed a pretty nasty regression in our NFS capability between 5.17=
 and
> 5.18-rc1. I've tried to bisect but not having any luck. The problem I'm s=
eeing
> is it takes 3 minutes to copy a file from NFS to the local disk. When it =
should
> take less than half a second, which it did up through 5.17.
>=20
> It doesn't seem to be network related, but can't rule that out completely=
.
>=20
> I tried to bisect but the problem can be intermittent. Some runs I'll see=
 a
> problem in 3 out of 100 cycles, sometimes 0 out of 100. Sometimes I'll se=
e it
> 100 out of 100.

It's not clear from your problem report whether the problem appears
when it's the server running v5.18-rc or the client.

It looks vaguely like a recent client issue where it encounters a
memory shortage while preparing an RPC and a delay is needed.


> The latest attempt I have is at 5.18-rc4 and here's what I see when I 1) =
create
> file with dd, 2) copy local disk to NFS mount, 3) copy NFS to local disk.
>=20
> Test 2
> Creating /dev/shm/run_nfs_test.junk...
> 262144+0 records in
> 262144+0 records out
> 268435456 bytes (268 MB, 256 MiB) copied, 1.29037 s, 208 MB/s
> Done
> copy /dev/shm/run_nfs_test.junk to /mnt/nfs_test/run_nfs_test.junk...
>=20
> real    0m0.502s
> user    0m0.001s
> sys     0m0.250s
> Done
> copy /mnt/nfs_test/run_nfs_test.junk to /dev/shm/run_nfs_test.tmp...
>=20
> real    4m11.835s
> user    0m0.001s
> sys     0m0.277s
> Done
> Comparing files...
> Done
> Remove /dev/shm/run_nfs_test.tmp...
>=20
> real    0m0.031s
> user    0m0.000s
> sys     0m0.031s
> Done
> Remove /dev/shm/run_nfs_test.junk...
>=20
> real    0m0.031s
> user    0m0.001s
> sys     0m0.030s
> Done
> Remove /mnt/nfs_test/run_nfs_test.junk...
>=20
> real    0m0.024s
> user    0m0.001s
> sys     0m0.022s
> Done
>=20
> Create the server with a 4G RAM disk:
> mount -t tmpfs -o size=3D4096M tmpfs /mnt/nfs_test
>=20
> exportfs -o fsid=3D0,rw,async,insecure,no_root_squash
> 192.168.254.0/255.255.255.0:/mnt/nfs_test
>=20
> Mount on the client side with:
> mount -o rdma,port=3D20049 192.168.254.1:/mnt/nfs_test /mnt/nfs_test
>=20
> Any advice on tracking this down further would be greatly appreciated.
>=20
> Thanks
>=20
> -Denny
>=20

--
Chuck Lever