Received-SPF: pass (google.com: domain of linux-nfs+bounces-2097-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.80.249 as permitted sender) client-ip=147.75.80.249;
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
MIME-Version: 1.0
From: "NeilBrown" <neilb@suse.de>
To: "Jacek Tomaka" <Jacek.Tomaka@poczta.fm>
Cc: "trond.myklebust@hammerspace.com" <trond.myklebust@hammerspace.com>,
 "anna.schumaker@netapp.com" <anna.schumaker@netapp.com>,
 "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Subject: Re: NFS data corruption on congested network
In-reply-to: <flfkkydzpicimncinmba@mlpw>
References: <ujvntmhlfharduyanjob@tgqn>,
 <170890314859.24797.16728369357798855399@noble.neil.brown.name>,
 <flfkkydzpicimncinmba@mlpw>
Date: Wed, 28 Feb 2024 09:59:56 +1100
Message-id: <170907479656.24797.5574165235102920031@noble.neil.brown.name>

On Mon, 26 Feb 2024, Jacek Tomaka wrote:
> Hi NeilBrown,=20
>=20
> > though if your kernel is older than 6.3, that will be
> >          redirty_for_writepage(wbc, page);
>=20
> Things are looking good. I have ran it on 15 machines for good couple of ho=
urs and i do not see the problem. Usually i would see it after 1-3 iterations=
 but now they are reaching 20 iterations without the problem.
>=20
> Thank you for the fix.

Thanks for testing!  I'll get the fix submitted.

NeilBrown


> Regards.
> Jacek Tomaka
>=20
> Temat: Re: NFS data corruption on congested network
> Data: 2024-02-26 0:19
> Nadawca: "NeilBrown" &lt;neilb@suse.de>
> Adresat: "Jacek Tomaka" &lt;Jacek.Tomaka@poczta.fm>;=20
> DW: trond.myklebust@hammerspace.com; anna.schumaker@netapp.com; linux-nfs@v=
ger.kernel.org;=20
>=20
> >=20
> >> On Mon, 26 Feb 2024, NeilBrown wrote:
> >> On Fri, 23 Feb 2024, Jacek Tomaka wrote:
> >>> Hello,
> >>> I ran into an issue where the NFS file ends up being corrupted on
> disk. We started noticing it on certain, quite old hardware after upgrading
> OS from Centos 6 to Rocky 9.2. We do see it on Rocky 9.3 but not on 9.1.
> >>>=20
> >>> After some investigation we have reasons to believe that the
> change was introduced by the following commit:=20
> >>>
> https://github.com/torvalds/linux/commit/6df25e58532be7a4cd6fb15bcd85805947=
402d91
> >>=20
> >> Thanks for the report.
> >> Can you try a change to your kernel?
> >>=20
> >> diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> >> index bb79d3a886ae..08a787147bd2 100644
> >> --- a/fs/nfs/write.c
> >> +++ b/fs/nfs/write.c
> >> @@ -668,8 +668,10 @@ static int nfs_writepage_locked(struct folio
> *folio,
> >>  	int err;
> >> =20
> >>  	if (wbc->sync_mode =3D=3D WB_SYNC_NONE &amp;&amp;
> >> -	    NFS_SERVER(inode)->write_congested)
> >> +	    NFS_SERVER(inode)->write_congested) {
> >> +		folio_redirty_for_writepage(wbc, folio);
> >>  		return AOP_WRITEPAGE_ACTIVATE;
> >> +	}
> >> =20
> >>  	nfs_inc_stats(inode, NFSIOS_VFSWRITEPAGE);
> >>  	nfs_pageio_init_write(&amp;pgio, inode, 0, false,
> >=20
> > Actually this is only needed before linux 6.8 as only nfs_writepage()
> > can call nfs_writepage_locked() with sync_mode of WB_SYNC_NONE.
> > So v5.18 through v6.7 might need fixing.
> >=20
> > NeilBrown
> >=20
> >=20
> >>=20
> >>=20
> >> though if your kernel is older than 6.3, that will be
> >>          redirty_for_writepage(wbc, page);
> >>=20
> >> Thanks,
> >> NeilBrown
> >>=20
> >>=20
> >>>=20
> >>> We write a number of files on a single thread. Each file is up to
> 4GB. Before closing we call fdatasync. Sometimes the file ends up being
> corrupted. The corruptions is in a form of a number ( more than 3k pages in
> one case) of zero filled pages.
> >>> When this happens the file cannot be deleted from the client
> machine which created the file, even when the process which wrote the file
> completed successfully.
> >>>=20
> >>> The machines have about 128GB of memory, i think and probably
> network that leaves to be desired.
> >>>=20
> >>> My reproducer is currently tied up to our internal software, but i
> suspect setting the write_congested flag randomly should allow to reproduce
> the issue.
> >>>=20
> >>> Regards.
> >>> Jacek Tomaka
> >>>=20
> >>=20
> >>=20
> >>=20
> >=20
> >=20
> >=20
>=20