Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Mike Javorski <mike.javorski@gmail.com>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: NFS server regression in kernel 5.13 (tested w/ 5.13.9)
Thread-Topic: NFS server regression in kernel 5.13 (tested w/ 5.13.9)
Thread-Index: AQHXjKYRxKTFwD0+C02+ZMVduTdsB6tqNWCA
Date:   Sun, 8 Aug 2021 22:47:18 +0000
Message-ID: <3349D119-2F35-4A58-8061-A2659E8C6BB9@oracle.com>
References: <CAOv1SKCmdtchm5Z2NU80o49tkrHpAkPFaHKj4-vLDN5bZNCz-Q@mail.gmail.com>
In-Reply-To: <CAOv1SKCmdtchm5Z2NU80o49tkrHpAkPFaHKj4-vLDN5bZNCz-Q@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?6oglu2mXjMp/esOThB0gJvNxLcmpgx+k0W640W/4IwT0Mw0WTQivPt+Z3GpC?=
 =?us-ascii?Q?Dm9CHRHSSkugceikKmdJnZ3Jv7MwBdUjcRy77Nq6yZmC5CjcUtgmzFUYp+MT?=
 =?us-ascii?Q?c1KWJSnf3BhDSyfumPjLEz4yhSmHWgzd88o901l1Yrit+z5zW0yHmCCZRebc?=
 =?us-ascii?Q?ghgM408TgJLC+uVuHNlBgELApsKmvB3DZwFeY7E7n05fnqkIANQueLEUFAiw?=
 =?us-ascii?Q?SoNg1kKyBLpRZjSnTHEY/5tRoyayRICkQTS1S5uXJVB2kkUMPG+9DEevqezi?=
 =?us-ascii?Q?mjVMh+fNXYwL9ebc8Pk+8QxmSomvbZ2e3m+jHAcKUParZtUXIaTv2rT6tXMm?=
 =?us-ascii?Q?CWltklmcOzbQtrxCGkUY/+xC2W4tu08YfdSW8P7soHVgVRn9nmFDuqUqF9yo?=
 =?us-ascii?Q?9pgUliBUZiyrIXUy0cIUzkgWK4f4YLfPeR5Y+POgewsTvWntFHyt7nwXrhQW?=
 =?us-ascii?Q?ycQdDtFywj/VpCw9VpPwbu4fk3jxQQvUlWJ1v3Nx0qPA7lQtpEsgWmVVLtDp?=
 =?us-ascii?Q?wXARuHmm4vkOJeNofa+DEWC/6ULP2dNldCf3JUaCzOdt7Gqwc0SSKbbqmwag?=
 =?us-ascii?Q?hOVaTfYjOkigIUUgtgg6VfzcLjJ/8kCXOr24a4YJTik5gWDgLKoP6+PhRegC?=
 =?us-ascii?Q?CIUKjfrMVCbnqcmDFtBwSWm4YqfXIYpFqXLsZhDCeMVzcjFSxPtz4H1txoxi?=
 =?us-ascii?Q?pzPfIBkTHS7ULWz/4N3Q+IVNm0mq/lfGBrWL6stQKkiyWS7VCGrPXKcfXEMA?=
 =?us-ascii?Q?L9Vg9kgb2x/RdqsqUG5umx5j5XlwnjME6OcxFSl1UzJAOGABie4bKKbnDgVP?=
 =?us-ascii?Q?WOO4642agUMDGgx6+T+lQR8WLIMPc9t/pBBJQ8MGPH2WUxA4HicVyFPoidA1?=
 =?us-ascii?Q?G4VYjdvy56rK0KOo2N7LpUzbdmuERa8tNwRZPAqt87o/4j5WknIE+Om/GPXo?=
 =?us-ascii?Q?1gcSLzqIjqz62i/tJKhkh6jsYHxCtqakD1hBn30WiHLXOQnHZA+8U9RlGfmw?=
 =?us-ascii?Q?qtHxXG5zYsdTykZ/ghK/t6sCbJPX3JBSS1qwH8hhnRlB8L1aVFNkDPiVH10B?=
 =?us-ascii?Q?RGwJZGf98Qt5cvutKmmTiX2qfRZ41waaXY+PYe3LNFcXKaA3Y9QjQFxTUCNr?=
 =?us-ascii?Q?Y5WaFHXltXqQO8CDKBsZUxLUL/mO9h05w748220icr7GD8FZrLiq0RMo3MUh?=
 =?us-ascii?Q?N9m2qJyF9S0nCRwwFEXzfCB7U3cEqfXQiAwKD41UZkyl5SrbE1yv2YzT2n+Q?=
 =?us-ascii?Q?P9abDqrHpB7pU0vx2Gc7sHVAwE8+/y2+DzRroUvZtLA9AHp9K11u+iJt7DVp?=
 =?us-ascii?Q?JYwzGxGQ5b3kiWtebFhRxS6HzCyJWRrbZaFaeaoQ1BxKWw=3D=3D?=
x-ms-exchange-transport-forked: True
Content-Type: text/plain; charset="us-ascii"
Content-ID: <A82F161E405D754991F9A117E39FA440@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4688.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9e48a468-ebe6-4a4b-cc26-08d95abe7a27
X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Aug 2021 22:47:18.8305
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: nA5ODeW2hh6zi8xY+H4fPmTSXjrX6oY2M4zu9C0vS/FAAR+gDC4mFLez6RFtt3wufzqgjtHZIgV5jriaWXD31g==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB4387
X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10070 signatures=668682
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0 mlxscore=0
 phishscore=0 spamscore=0 mlxlogscore=999 malwarescore=0 adultscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000
 definitions=main-2108080144
X-Proofpoint-GUID: hNz17CSvgLH1jhPObkMTuWYn8PLbQ6RV
X-Proofpoint-ORIG-GUID: hNz17CSvgLH1jhPObkMTuWYn8PLbQ6RV
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Aug 8, 2021, at 6:37 PM, Mike Javorski <mike.javorski@gmail.com> wrote=
:
>=20
> I have been experiencing nfs file access hangs with multiple release
> versions of the 5.13.x linux kernel. In each case, all file transfers
> freeze for 5-10 seconds and then resume. This seems worse when reading
> through many files sequentially.
>=20
> My server:
> - Archlinux w/ a distribution provided kernel package
> - filesystems exported with "rw,sync,no_subtree_check,insecure" options
>=20
> Client:
> - Archlinux w/ latest distribution provided kernel (5.13.9-arch1-1 at wri=
ting)
> - nfs mounted via /net autofs with "soft,nodev,nosuid" options
> (ver=3D4.2 is indicated in mount)
>=20
> I have tried the 5.13.x kernel several times since the first arch
> release (most recently with 5.13.9-arch1-1), all with similar results.
> Each time, I am forced to downgrade the linux package to a 5.12.x
> kernel (5.12.15-arch1 as of writing) to clear up the transfer issues
> and stabilize performance. No other changes are made between tests. I
> have confirmed the freezing behavior using both ext4 and btrfs
> filesystems exported from this server.
>=20
> At this point I would appreciate some guidance in what to provide in
> order to diagnose and resolve this issue. I don't have a lot of kernel
> debugging experience, so instruction would be helpful.

Hi Mike-

Thanks for the report.

Since you are using a distribution kernel and don't have much
kernel debugging experience, we typically ask you to work with
your distributor first. linux-nfs@ is a developer's mailing
list, we're not really prepared to provide user support.

When you report the problem to Arch, you might want to have
a description of your workload (especially if you can
reproduce the problem often with a particular application
or user activity), and maybe be prepared to capture a network
trace of the failure using tcpdump.


--
Chuck Lever