Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Chuck Lever III <chuck.lever@oracle.com>
To:     Trond Myklebust <trondmy@hammerspace.com>
CC:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: recent intermittent fsx-related failures
Thread-Topic: recent intermittent fsx-related failures
Thread-Index: AQHXf/8fslPfHYBEJkGXNGWvoY4o9atRAYKAgFtTqAA=
Date:   Sun, 19 Sep 2021 23:03:38 +0000
Message-ID: <680A4FB2-B90D-47E1-A390-36B3081B1464@oracle.com>
References: <67E1CF9F-61C5-4BB9-97FE-61B598EFC382@oracle.com>
 <2e8bce7bc15b02bbd1dcf740f2d993d6e3d58367.camel@hammerspace.com>
In-Reply-To: <2e8bce7bc15b02bbd1dcf740f2d993d6e3d58367.camel@hammerspace.com>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?32bUGT7+tzChJ4lgIFq6uch7ozTdmh5SJuc4o6jQ002AaOGfL8/2KTbOafT0?=
 =?us-ascii?Q?pFFhjO6BoaE0T57Ll/HGClmKrix5MGXrvxOy6Mrz1Jzb/Qly0832rU5hyXqn?=
 =?us-ascii?Q?Uo6S7jNI9VnuMEcI0RIIt6ay+3AgF/61KmK+2pw7qXKy6kwak0p2cU2l+s7T?=
 =?us-ascii?Q?f6CAJnL99oHE7rzqtx8E48MykMMcx2Co2Q8VoGTM5pgRzRao/qO3K4Tf6i1O?=
 =?us-ascii?Q?OKVuQKH2tDla54MummWqwhnHsSV6cfOMSVu/lAEav/lAytzsYZFQUE3LZ4Dv?=
 =?us-ascii?Q?uJiCtRC6mEJ3aHzGbAYIizMYqSfmqeNUAMQ2Vx0haU0jbdaubq2evYKfhOub?=
 =?us-ascii?Q?B+yE3G4DQLcM71qq0nlNQ47Mtkj3UU/A6Rzt9VKRITUguHRKg92oi/NfRkC5?=
 =?us-ascii?Q?vL8QKFWeUakDfucEbrXaqdsnaJBF7JMb58ehxGIxzLQfGQvbbbE9u5zRthfa?=
 =?us-ascii?Q?Zj5JrqSDYEK9QRCU7jaE8ETTMzRp0rdN1FVs2CxP3Es8RSiqR4kjieJWAE3H?=
 =?us-ascii?Q?8zOQNNChIUrAYUjFEnIzRrE87UUUenlEjw/JFoNtTNgoAyPvg+XHV0AsPhHp?=
 =?us-ascii?Q?W9sIylMsD1lcBVEDqyCeo0xstc2v0RWmNhwsS2upDtEVE0EdPP3W7Silgmua?=
 =?us-ascii?Q?XEhl9JfoykzyYSzRCMQV7o7VQWiUXRE6ew105CAOhDicGJT430ziTOTNmFN4?=
 =?us-ascii?Q?Ns8ZfAsF26PdAmff92qTMe1041ws/MIV9Qem+eQa9eeBr4w5Dw/VrXXoPxpO?=
 =?us-ascii?Q?DFzfSjPkZEZapjjtcdSLqtzbaqTXTJoA9ksIGIlMA45vcl0VVfjvlOrHvzdH?=
 =?us-ascii?Q?aYxKtZBNXykwQTs5qETVy+RgQsk840jf2enNmZKbcS2hnRRKO8etHqTGEvb6?=
 =?us-ascii?Q?l+YhnEnq/K9tlyk3/6Y/ZO7lAZLEpT7PnOdGkjrSvpfqMtt/WjrszlUhNcFD?=
 =?us-ascii?Q?7TwjcVhV4fYTccxUxjmnGPGq6TZhRr4+h50APPWsoTLJw2XWmwr01h2f0kY2?=
 =?us-ascii?Q?mVomTQhkhMMiw6VQeFNHKb4crYK+0fBa0KEsJSgQZ4V+06TuESL49sDp0ZpP?=
 =?us-ascii?Q?tSz9aOKkjVjfZ2qzYz/0ZT6y2KKFaei4P82tC4PoPxcWVhQweujXsAYU/6sq?=
 =?us-ascii?Q?wwdcp0EVY3h+GxDWMrxGJ/xAhauuQd+GfIm4rR43aeFMaI4/Qex7vfcg8cyY?=
 =?us-ascii?Q?OI2P8eWiluMJpfdvANYAsbikyTq/3h7EQC3L6QP8r3roXmwDkKjpclo0akvr?=
 =?us-ascii?Q?q3hjlfdYmydkeedhccIFDh/i/U298MVCq05pkSAQt/vLatVSH1XiXOquoSyz?=
 =?us-ascii?Q?PtVJOi6b5XrBFf8KIrDlaCcg?=
x-ms-exchange-transport-forked: True
Content-Type: text/plain; charset="us-ascii"
Content-ID: <DFD517F75683A2438BEA9046EF770B8E@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4688.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: a7024fab-ba51-4389-a5ee-08d97bc1b7a2
X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Sep 2021 23:03:38.8733
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: dfO4aYadvpIWq5xDM3LSqiSawVSsTLiHWgM9mp7ctMt1M4H/NYB/XHJmg2g/mwr8z/GWyzHS5vTfH3BO1ejaYA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4655
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10112 signatures=668682
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 adultscore=0
 bulkscore=0 mlxscore=0 suspectscore=0 spamscore=0 phishscore=0
 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2109030001 definitions=main-2109190171
X-Proofpoint-GUID: PAgpqhAOoU2a-vqiruu5FVhL6YQhb2mI
X-Proofpoint-ORIG-GUID: PAgpqhAOoU2a-vqiruu5FVhL6YQhb2mI
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


> On Jul 23, 2021, at 4:24 PM, Trond Myklebust <trondmy@hammerspace.com> wr=
ote:
>=20
> On Fri, 2021-07-23 at 20:12 +0000, Chuck Lever III wrote:
>> Hi-
>>=20
>> I noticed recently that generic/075, generic/112, and generic/127
>> were
>> failing intermittently on NFSv3 mounts. All three of these tests are
>> based on fsx.
>>=20
>> "git bisect" landed on this commit:
>>=20
>> 7b24dacf0840 ("NFS: Another inode revalidation improvement")
>>=20
>> After reverting 7b24dacf0840 on v5.14-rc1, I can no longer reproduce
>> the test failures.
>>=20
>>=20
>=20
> So you are seeing file metadata updates that end up not changing the
> ctime?

As far as I can tell, a WRITE and two SETATTRs are happening in
sequence to the same file during the same jiffy. The WRITE does
not report pre/post attributes, but the SETATTRs do. The reported
pre- and post- mtime and ctime are all the same value for both
SETATTRs, I believe due to timestamp_truncate().

My theory is that persistent-storage-backed filesystems seem to
go slow enough that it doesn't become a significant problem. But
with tmpfs, this can happen often enough that the client gets
confused. And I can make the problem unreproducable if I enable
enough debugging paraphernalia on the server to slow it down.

I'm not exactly sure how the client becomes confused by this
behavior, but fsx reports a stale size value, or it can hit a
bus error. I'm seeing at least four of the fsx-based xfs tests
fail intermittently.


--
Chuck Lever