Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
References: <CAAmbk-f_U8CPcTQM866L572uUHdK4p5iWKnUQs4r8fkW=6RW9g@mail.gmail.com>
 <CALF+zO=1oASFeb5WOezeZm_fbCuw=L8AL-n-mbCt8A==ZMAy3Q@mail.gmail.com>
In-Reply-To: <CALF+zO=1oASFeb5WOezeZm_fbCuw=L8AL-n-mbCt8A==ZMAy3Q@mail.gmail.com>
From:   Chris Chilvers <chilversc@gmail.com>
Date:   Thu, 18 May 2023 21:20:46 +0100
Message-ID: <CAAmbk-cgTR+FxLY2C=upuPBwNaBYff=GHedXyQiFf=Hr5i3G0A@mail.gmail.com>
Subject: Re: [Linux-cachefs] [BUG] fscache writing but not reading
To:     David Wysochanski <dwysocha@redhat.com>
Cc:     linux-nfs@vger.kernel.org, linux-cachefs@redhat.com,
        brennandoyle@google.com, Benjamin Maynard <benmaynard@google.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Precedence: bulk

On Tue, 16 May 2023 at 20:28, David Wysochanski <dwysocha@redhat.com> wrote=
:
>
> On Tue, May 16, 2023 at 11:42=E2=80=AFAM Chris Chilvers <chilversc@gmail.=
com> wrote:
> >
> > While testing the fscache performance fixes [1] that were merged into 6=
.4-rc1
> > it appears that the caching no longer works. The client will write to t=
he cache
> > but never reads.
> >
> Thanks for the report.
>
> If you reboot do you see reads from the cache?

On the first read after a reboot it uses the cache, but subsequent
reads do not use the cache.

> You can check if the cache is being read from by looking in
> /proc/fs/fscache/stats
> at the "IO" line:
> # grep IO /proc/fs/fscache/stats
> IO     : rd=3D80030 wr=3D0

Running the tests 4 times (twice before reboot, and twice after) give
the following metrics:
FS-Cache I/O (delta)
Run       rd        wr
 1         0    39,250
 2       130    38,894
 3    39,000         0
 4        72    38,991

> Can you share:
> 1. NFS server you're using (is it localhost or something else)
> 2. NFS version

The NFS server and client are separate VMs on the same network.
Server NFS version: Ubuntu 22.04 jammy, kernel 5.15.0-1021-gcp
Client NFS version: Ubuntu 22.04 jammy, kernel 6.4.0-060400rc1-generic
(https://kernel.ubuntu.com/~kernel-ppa/mainline/v6.4-rc1/)
Client nfs-utils: 2.6.3-rc6
Client cachefilesd: 0.10.10-0.2ubuntu1

> In addition to checking the above for the reads from the cache, you can a=
lso
> see whether NFS reads are going over the wire pretty easily with a simila=
r
> technique.
>
> Copy /proc/self/mounstats to a file before your test, then make a second =
copy
> after the test, then run mountstats as follows:
> mountstats -S /tmp/mountstats.1 -f /tmp/mountstats.2

app read    =3D applications read bytes via read(2)
client read =3D client read bytes via NFS READ

Run           app read        client read
 1     322,122,547,200    322,122,547,200
 2     322,122,547,200    321,048,805,376
 3     322,122,547,200                  0
 4     322,122,547,200    321,593,053,184

I've put the full data in a GitHub gist, along with a graph collected
from a metrics agent:
https://gist.github.com/chilversc/54eb76155ad37b66cb85186e7449beaa
https://gist.githubusercontent.com/chilversc/54eb76155ad37b66cb85186e7449be=
aa/raw/09828c596d0cfc44bc0eb67f40e4033db202326e/metrics.png