Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <CAAmbk-f_U8CPcTQM866L572uUHdK4p5iWKnUQs4r8fkW=6RW9g@mail.gmail.com>
 <CALF+zO=1oASFeb5WOezeZm_fbCuw=L8AL-n-mbCt8A==ZMAy3Q@mail.gmail.com> <CAAmbk-cgTR+FxLY2C=upuPBwNaBYff=GHedXyQiFf=Hr5i3G0A@mail.gmail.com>
In-Reply-To: <CAAmbk-cgTR+FxLY2C=upuPBwNaBYff=GHedXyQiFf=Hr5i3G0A@mail.gmail.com>
From:   David Wysochanski <dwysocha@redhat.com>
Date:   Fri, 19 May 2023 07:53:41 -0400
Message-ID: <CALF+zO=vjPjs8QevssUmGHA_bZaTaF3HcqQa+OQgxegqB5yzmw@mail.gmail.com>
Subject: Re: [Linux-cachefs] [BUG] fscache writing but not reading
To:     Chris Chilvers <chilversc@gmail.com>
Cc:     linux-nfs@vger.kernel.org, linux-cachefs@redhat.com,
        brennandoyle@google.com, Benjamin Maynard <benmaynard@google.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Thu, May 18, 2023 at 4:21=E2=80=AFPM Chris Chilvers <chilversc@gmail.com=
> wrote:
>
> On Tue, 16 May 2023 at 20:28, David Wysochanski <dwysocha@redhat.com> wro=
te:
> >
> > On Tue, May 16, 2023 at 11:42=E2=80=AFAM Chris Chilvers <chilversc@gmai=
l.com> wrote:
> > >
> > > While testing the fscache performance fixes [1] that were merged into=
 6.4-rc1
> > > it appears that the caching no longer works. The client will write to=
 the cache
> > > but never reads.
> > >
> > Thanks for the report.
> >
> > If you reboot do you see reads from the cache?
>
> On the first read after a reboot it uses the cache, but subsequent
> reads do not use the cache.
>
> > You can check if the cache is being read from by looking in
> > /proc/fs/fscache/stats
> > at the "IO" line:
> > # grep IO /proc/fs/fscache/stats
> > IO     : rd=3D80030 wr=3D0
>
> Running the tests 4 times (twice before reboot, and twice after) give
> the following metrics:
> FS-Cache I/O (delta)
> Run       rd        wr
>  1         0    39,250
>  2       130    38,894
>  3    39,000         0
>  4        72    38,991
>
> > Can you share:
> > 1. NFS server you're using (is it localhost or something else)
> > 2. NFS version
>
> The NFS server and client are separate VMs on the same network.
> Server NFS version: Ubuntu 22.04 jammy, kernel 5.15.0-1021-gcp
> Client NFS version: Ubuntu 22.04 jammy, kernel 6.4.0-060400rc1-generic
> (https://kernel.ubuntu.com/~kernel-ppa/mainline/v6.4-rc1/)
> Client nfs-utils: 2.6.3-rc6
> Client cachefilesd: 0.10.10-0.2ubuntu1
>
> > In addition to checking the above for the reads from the cache, you can=
 also
> > see whether NFS reads are going over the wire pretty easily with a simi=
lar
> > technique.
> >
> > Copy /proc/self/mounstats to a file before your test, then make a secon=
d copy
> > after the test, then run mountstats as follows:
> > mountstats -S /tmp/mountstats.1 -f /tmp/mountstats.2
>
> app read    =3D applications read bytes via read(2)
> client read =3D client read bytes via NFS READ
>
> Run           app read        client read
>  1     322,122,547,200    322,122,547,200
>  2     322,122,547,200    321,048,805,376
>  3     322,122,547,200                  0
>  4     322,122,547,200    321,593,053,184
>
> I've put the full data in a GitHub gist, along with a graph collected
> from a metrics agent:
> https://gist.github.com/chilversc/54eb76155ad37b66cb85186e7449beaa
> https://gist.githubusercontent.com/chilversc/54eb76155ad37b66cb85186e7449=
beaa/raw/09828c596d0cfc44bc0eb67f40e4033db202326e/metrics.png
>

Thanks Chris for all this info.  I see you're using NFSv3 so I'll
focus on that, and review all this info for clues.
I also have been working on some updated test cases and see some very
unusual behavior like you're reporting.

I also confirmed that adding the two patches for "Issue #1"  onto
6.4-rc1 resolve _most_ of the caching issues.
However, even after those patches, in some limited instances, there
are still NFS reads over the wire when there should only be reads from
the cache.
There may be multiple bugs here.