Received-SPF: pass (google.com: domain of linux-nfs+bounces-2432-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Feedback-ID: i197146af:Fastmail
Date: Thu, 21 Mar 2024 16:14:56 +0100
From: Patrick Steinhardt <ps@pks.im>
To: Jeff King <peff@peff.net>
Cc: Chuck Lever <chuck.lever@oracle.com>, git@vger.kernel.org,
	linux-nfs@vger.kernel.org
Subject: Re: t0032 fails on NFS mounts
Message-ID: <ZfxO8ApCDUbBHJyc@tanuki>
References: <ZfBwZTL9zqDsac5m@manet.1015granger.net>
 <20240313072052.GC125150@coredump.intra.peff.net>
Precedence: bulk
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="I/xz9JawEjrSgC+S"
Content-Disposition: inline
In-Reply-To: <20240313072052.GC125150@coredump.intra.peff.net>


--I/xz9JawEjrSgC+S
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Wed, Mar 13, 2024 at 03:20:52AM -0400, Jeff King wrote:
> +cc Patrick for reftable
>=20
> On Tue, Mar 12, 2024 at 11:10:29AM -0400, Chuck Lever wrote:
>=20
> > Unit test t0032 fails when run on an NFS mount:
> >=20
> > [vagrant@cel t]$ ./t0032-reftable-unittest.sh=20
> > not ok 1 - unittests
> > #=09
> > #		TMPDIR=3D$(pwd) && export TMPDIR &&
> > #		test-tool reftable
> > #=09
> > # failed 1 among 1 test(s)
> > 1..1
>=20
> The output for this test script is particularly unhelpful because it's
> not using our test harness at all, but just running a bunch of internal
> tests using a single program.
>=20
> Running with "-v" should give more details about what's failing.
>=20
> I set up a basic loopback server like:
>=20
>   mkdir /mnt/{server,client}
>   exportfs -o rw,sync 127.0.0.1:/mnt/server
>   mount -t nfs 127.0.0.1:/mnt/server /mnt/client
>=20
> and then ran:
>=20
>   ./t0032-reftable-unittest.sh --root=3D/mnt/client -v
>=20
> Looks like it fails at:
>=20
>   running test_reftable_stack_compaction_concurrent_clean
>   reftable/stack_test.c: 1063: failed assertion count_dir_entries(dir) =
=3D=3D 2
>   Aborted
>=20
> > v2.43.2 seems to work OK.
>=20
> For me, too. Bisecting shows the problem appearing in 4f36b8597c
> (reftable/stack: fix race in up-to-date check, 2024-01-18).

I think this is actually benign. I set a breakpoint in the respective
test right before double-checking our conditions, and curiously I got
back the following list of files:

    ./stack_test-1027.QJBpnd
    ./stack_test-1027.QJBpnd/0x000000000001-0x000000000003-dad7ac80.ref
    ./stack_test-1027.QJBpnd/.nfs000000000001729f00001e11
    ./stack_test-1027.QJBpnd/tables.list

Notice the ".nfs*" thing? This is a temporary file managed by the NFS
client that maintains delete-on-close behaviour because we have unlinked
the file while it was still open [1]. But of course we count that file
when executing `count_dir_entries()`, and thus we arrive at an
unexpected number of files.

I will send a patch to fix the test.

> PS That test seems to run ~20x slower on NFS versus directly on ext4.
>    I'd expect a little overhead, but that's quite a bit.

I'm not all that surprised here given that the reftable library is quite
prone to stat(3P)ing the "tables.list" file, and potentially re-reading
it. I kind of suspect that this is what's going on. An alternative
explanation might be that mmap'ing over NFS is really slow.

Anyway, I will have a deeper look at this and see where we spend all the
time.

Patrick

[1]: https://nfs.sourceforge.net/#faq_d2

--I/xz9JawEjrSgC+S
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEF9hrgiFbCdvenl/rVbJhu7ckPpQFAmX8Tu8ACgkQVbJhu7ck
PpQEKA//dsf+RIUY7SkHxNbzDNgL1mrIxEi2KQx7BykCpQXhB2/SAirKdYwqexsF
xKN7k69cXPoufCg2W8zxpAZSLWPdXKf1/zcI90fjYuQ4B6G+5j1jLNKm0gVBgJN/
yM4hoZmz8D9ecfLAspwcbEObJqYOMwHSaus6tYWKDNIAjpflhmYb48RK7wp8awCm
7AfSDPBZQRlRsJubWwu5rMGPg1Ea+i/HNZpTsODVpXIPpvKZthAozqPqy3hCb4oh
eOWTNZfQk+4yENS9czYWjaT2SO2RVmheEXZx8QR2igP9QOzApp3O6pSGutbUhqWI
dJmVLOYRIWiunZW8ew/6P/nRYb4qMykCblEegBP1jc0axWGMrcUxPznHbHlAElMN
SxpMwtOnLWAEjuLHEJIdlLJUo9u5haKDCsrDipVZ96tQCuDBaCR8PypDyIvgfgiL
EBBkmh4vulUrNkzC044fF+sSu8YYEjNpVWP0Z31RkH1DL+O/BJ41sa9lziAODCAr
0S/GVMJYf+cYq67QytWq10Btlb0ZJgwpA64UfrTn7kTnOiEqolWe5hF01v+njGzQ
2Jw3dEGFoN94SCUHXiEXaH9GYyMYAmMieZcri/FyFZ6SlPMGe05DS4pruBJGSxfX
l5RwZicNpV3IsFAowcZ6lFlJee06WneoGOnCAPMVQbQ8fd7275E=
=Acwc
-----END PGP SIGNATURE-----

--I/xz9JawEjrSgC+S--