Received-SPF: pass (google.com: domain of linux-nfs+bounces-2073-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
MIME-Version: 1.0
From: "NeilBrown" <neilb@suse.de>
To: "Jacek Tomaka" <Jacek.Tomaka@poczta.fm>
Cc: trond.myklebust@hammerspace.com, anna.schumaker@netapp.com,
 linux-nfs@vger.kernel.org
Subject: Re: NFS data corruption on congested network
In-reply-to: <170890214013.24797.3257981274610636720@noble.neil.brown.name>
References: <ujvntmhlfharduyanjob@tgqn>,
 <170890214013.24797.3257981274610636720@noble.neil.brown.name>
Date: Mon, 26 Feb 2024 10:19:08 +1100
Message-id: <170890314859.24797.16728369357798855399@noble.neil.brown.name>

On Mon, 26 Feb 2024, NeilBrown wrote:
> On Fri, 23 Feb 2024, Jacek Tomaka wrote:
> > Hello,
> > I ran into an issue where the NFS file ends up being corrupted on disk. W=
e started noticing it on certain, quite old hardware after upgrading OS from =
Centos 6 to Rocky 9.2. We do see it on Rocky 9.3 but not on 9.1.
> >=20
> > After some investigation we have reasons to believe that the change was i=
ntroduced by the following commit:=20
> > https://github.com/torvalds/linux/commit/6df25e58532be7a4cd6fb15bcd858059=
47402d91
>=20
> Thanks for the report.
> Can you try a change to your kernel?
>=20
> diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> index bb79d3a886ae..08a787147bd2 100644
> --- a/fs/nfs/write.c
> +++ b/fs/nfs/write.c
> @@ -668,8 +668,10 @@ static int nfs_writepage_locked(struct folio *folio,
>  	int err;
> =20
>  	if (wbc->sync_mode =3D=3D WB_SYNC_NONE &&
> -	    NFS_SERVER(inode)->write_congested)
> +	    NFS_SERVER(inode)->write_congested) {
> +		folio_redirty_for_writepage(wbc, folio);
>  		return AOP_WRITEPAGE_ACTIVATE;
> +	}
> =20
>  	nfs_inc_stats(inode, NFSIOS_VFSWRITEPAGE);
>  	nfs_pageio_init_write(&pgio, inode, 0, false,

Actually this is only needed before linux 6.8 as only nfs_writepage()
can call nfs_writepage_locked() with sync_mode of WB_SYNC_NONE.
So v5.18 through v6.7 might need fixing.

NeilBrown


>=20
>=20
> though if your kernel is older than 6.3, that will be
>          redirty_for_writepage(wbc, page);
>=20
> Thanks,
> NeilBrown
>=20
>=20
> >=20
> > We write a number of files on a single thread. Each file is up to 4GB. Be=
fore closing we call fdatasync. Sometimes the file ends up being corrupted. T=
he corruptions is in a form of a number ( more than 3k pages in one case) of =
zero filled pages.
> > When this happens the file cannot be deleted from the client machine whic=
h created the file, even when the process which wrote the file completed succ=
essfully.
> >=20
> > The machines have about 128GB of memory, i think and probably network tha=
t leaves to be desired.
> >=20
> > My reproducer is currently tied up to our internal software, but i suspec=
t setting the write_congested flag randomly should allow to reproduce the iss=
ue.
> >=20
> > Regards.
> > Jacek Tomaka
> >=20
>=20
>=20
>=20