Received-SPF: pass (google.com: domain of linux-nfs+bounces-2048-linux.lists.archive=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99;
Date: Thu, 22 Feb 2024 15:54:26 +0100
From: Jacek Tomaka <Jacek.Tomaka@poczta.fm>
Subject: NFS data corruption on congested network
To: trond.myklebust@hammerspace.com, anna.schumaker@netapp.com, neilb@suse.de
Cc: linux-nfs@vger.kernel.org
Message-Id: <ujvntmhlfharduyanjob@tgqn>
Precedence: bulk
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

Hello,
I ran into an issue where the NFS file ends up being corrupted on disk. We started noticing it on certain, quite old hardware after upgrading OS from Centos 6 to Rocky 9.2. We do see it on Rocky 9.3 but not on 9.1.

After some investigation we have reasons to believe that the change was introduced by the following commit: 
https://github.com/torvalds/linux/commit/6df25e58532be7a4cd6fb15bcd85805947402d91

We write a number of files on a single thread. Each file is up to 4GB. Before closing we call fdatasync. Sometimes the file ends up being corrupted. The corruptions is in a form of a number ( more than 3k pages in one case) of zero filled pages.
When this happens the file cannot be deleted from the client machine which created the file, even when the process which wrote the file completed successfully.

The machines have about 128GB of memory, i think and probably network that leaves to be desired.

My reproducer is currently tied up to our internal software, but i suspect setting the write_congested flag randomly should allow to reproduce the issue.

Regards.
Jacek Tomaka