While testing the fscache performance fixes [1] that were merged into 6.4-rc1
it appears that the caching no longer works. The client will write to the cache
but never reads.
I suspect this is related to known issue #1. However, I tested the client
with rsize less than, equal to, and greater than readahead, and in all cases
I see the issue.
If I apply both the patches [2], [3] from the known issues to 6.4-rc1 then the
cache works as expected. I suspect only patch [2] is required but have not
tested patch [2] without [3].
Testing
=======
For the test I was just using dd to read 300 x 1gb files from an NFS
share to fill the cache, then repeating the read.
In the first test run, /var/cache/fscache steadily filled until reaching
300 GB. The read I/O was less than 1 MB/s, and the write speed was fairly
constant 270 MB/s.
In the second run, /var/cache/fscache remained at 300 GB, so no new data was
being written. However, the read I/O remained at less than 1 MB/s and the write
rate at 270 MB/s.
/var/cache/fscache
| 1st run | 2nd run
disk usage | 0 -> 300 GB | 300 GB
read speed | < 1 MB/s | < 1 MB/s
write speed | 270 MB/s | 270 MB/s
This seems to imply that the already cached data was being read from the source
server and re-written to the cache.
Known Issues
============
1. Unit test setting rsize < readahead does not properly read from
fscache but re-reads data from the NFS server
* This will be fixed with another dhowells patch [2]:
"[PATCH v6 2/2] mm, netfs, fscache: Stop read optimisation when
folio removed from pagecache"
2. "Cache volume key already in use" after xfstest runs involving
multiple mounts
* Simple reproducer requires just two mounts as follows:
mount -overs=4.1,fsc,nosharecache -o
context=system_u:object_r:root_t:s0 nfs-server:/exp1 /mnt1
mount -overs=4.1,fsc,nosharecache -o
context=system_u:object_r:root_t:s0 nfs-server:/exp2 /mnt2
* This should be fixed with dhowells patch [3]:
"[PATCH v5] vfs, security: Fix automount superblock LSM init
problem, preventing NFS sb sharing"
References
==========
[1] https://lore.kernel.org/linux-nfs/[email protected]/
[2] https://lore.kernel.org/linux-nfs/[email protected]/T/#mf3807fa68fb6d495b87dde0d76b5237833a0cc81
[3] https://lore.kernel.org/linux-kernel/[email protected]/
On Tue, May 16, 2023 at 11:42 AM Chris Chilvers <[email protected]> wrote:
>
> While testing the fscache performance fixes [1] that were merged into 6.4-rc1
> it appears that the caching no longer works. The client will write to the cache
> but never reads.
>
Thanks for the report.
If you reboot do you see reads from the cache?
You can check if the cache is being read from by looking in
/proc/fs/fscache/stats
at the "IO" line:
# grep IO /proc/fs/fscache/stats
IO : rd=80030 wr=0
You might consider saving that file before your test, then another copy after,
and doing a diff:
# diff -u /tmp/fscachestats.1 /tmp/fscachestats.2
--- /tmp/fscachestats.1 2023-05-16 14:48:43.126158403 -0400
+++ /tmp/fscachestats.2 2023-05-16 14:54:05.421402702 -0400
@@ -1,14 +1,14 @@
FS-Cache statistics
-Cookies: n=0 v=1 vcol=0 voom=0
-Acquire: n=0 ok=0 oom=0
-LRU : n=0 exp=0 rmv=0 drp=0 at=0
+Cookies: n=5 v=1 vcol=0 voom=0
+Acquire: n=5 ok=5 oom=0
+LRU : n=0 exp=5 rmv=0 drp=0 at=0
Invals : n=0
-Updates: n=0 rsz=0 rsn=0
+Updates: n=5 rsz=0 rsn=0
Relinqs: n=0 rtr=0 drop=0
NoSpace: nwr=0 ncr=0 cull=6
-IO : rd=0 wr=0
-RdHelp : RA=0 RP=0 WB=0 WBZ=0 rr=0 sr=0
+IO : rd=40015 wr=0
+RdHelp : RA=40015 RP=0 WB=0 WBZ=0 rr=0 sr=0
RdHelp : ZR=0 sh=0 sk=0
RdHelp : DL=0 ds=0 df=0 di=0
-RdHelp : RD=0 rs=0 rf=0
+RdHelp : RD=40015 rs=40015 rf=0
RdHelp : WR=0 ws=0 wf=0
> I suspect this is related to known issue #1. However, I tested the client
> with rsize less than, equal to, and greater than readahead, and in all cases
> I see the issue.
>
> If I apply both the patches [2], [3] from the known issues to 6.4-rc1 then the
> cache works as expected. I suspect only patch [2] is required but have not
> tested patch [2] without [3].
>
Agree it's likely only the patches from issue #1 are needed.
Let me ping dhowells and willy on that thread for issue #1 as it looks
stalled.
> Testing
> =======
> For the test I was just using dd to read 300 x 1gb files from an NFS
> share to fill the cache, then repeating the read.
>
Can you share:
1. NFS server you're using (is it localhost or something else)
2. NFS version
> In the first test run, /var/cache/fscache steadily filled until reaching
> 300 GB. The read I/O was less than 1 MB/s, and the write speed was fairly
> constant 270 MB/s.
>
> In the second run, /var/cache/fscache remained at 300 GB, so no new data was
> being written. However, the read I/O remained at less than 1 MB/s and the write
> rate at 270 MB/s.
>
> /var/cache/fscache
> | 1st run | 2nd run
> disk usage | 0 -> 300 GB | 300 GB
> read speed | < 1 MB/s | < 1 MB/s
> write speed | 270 MB/s | 270 MB/s
>
> This seems to imply that the already cached data was being read from the source
> server and re-written to the cache.
>
In addition to checking the above for the reads from the cache, you can also
see whether NFS reads are going over the wire pretty easily with a similar
technique.
Copy /proc/self/mounstats to a file before your test, then make a second copy
after the test, then run mountstats as follows:
mountstats -S /tmp/mountstats.1 -f /tmp/mountstats.2
> Known Issues
> ============
> 1. Unit test setting rsize < readahead does not properly read from
> fscache but re-reads data from the NFS server
> * This will be fixed with another dhowells patch [2]:
> "[PATCH v6 2/2] mm, netfs, fscache: Stop read optimisation when
> folio removed from pagecache"
>
> 2. "Cache volume key already in use" after xfstest runs involving
> multiple mounts
> * Simple reproducer requires just two mounts as follows:
> mount -overs=4.1,fsc,nosharecache -o
> context=system_u:object_r:root_t:s0 nfs-server:/exp1 /mnt1
> mount -overs=4.1,fsc,nosharecache -o
> context=system_u:object_r:root_t:s0 nfs-server:/exp2 /mnt2
> * This should be fixed with dhowells patch [3]:
> "[PATCH v5] vfs, security: Fix automount superblock LSM init
> problem, preventing NFS sb sharing"
>
> References
> ==========
>
> [1] https://lore.kernel.org/linux-nfs/[email protected]/
> [2] https://lore.kernel.org/linux-nfs/[email protected]/T/#mf3807fa68fb6d495b87dde0d76b5237833a0cc81
> [3] https://lore.kernel.org/linux-kernel/[email protected]/
>
> --
> Linux-cachefs mailing list
> [email protected]
> https://listman.redhat.com/mailman/listinfo/linux-cachefs
>
On Tue, 16 May 2023 at 20:28, David Wysochanski <[email protected]> wrote:
>
> On Tue, May 16, 2023 at 11:42 AM Chris Chilvers <[email protected]> wrote:
> >
> > While testing the fscache performance fixes [1] that were merged into 6.4-rc1
> > it appears that the caching no longer works. The client will write to the cache
> > but never reads.
> >
> Thanks for the report.
>
> If you reboot do you see reads from the cache?
On the first read after a reboot it uses the cache, but subsequent
reads do not use the cache.
> You can check if the cache is being read from by looking in
> /proc/fs/fscache/stats
> at the "IO" line:
> # grep IO /proc/fs/fscache/stats
> IO : rd=80030 wr=0
Running the tests 4 times (twice before reboot, and twice after) give
the following metrics:
FS-Cache I/O (delta)
Run rd wr
1 0 39,250
2 130 38,894
3 39,000 0
4 72 38,991
> Can you share:
> 1. NFS server you're using (is it localhost or something else)
> 2. NFS version
The NFS server and client are separate VMs on the same network.
Server NFS version: Ubuntu 22.04 jammy, kernel 5.15.0-1021-gcp
Client NFS version: Ubuntu 22.04 jammy, kernel 6.4.0-060400rc1-generic
(https://kernel.ubuntu.com/~kernel-ppa/mainline/v6.4-rc1/)
Client nfs-utils: 2.6.3-rc6
Client cachefilesd: 0.10.10-0.2ubuntu1
> In addition to checking the above for the reads from the cache, you can also
> see whether NFS reads are going over the wire pretty easily with a similar
> technique.
>
> Copy /proc/self/mounstats to a file before your test, then make a second copy
> after the test, then run mountstats as follows:
> mountstats -S /tmp/mountstats.1 -f /tmp/mountstats.2
app read = applications read bytes via read(2)
client read = client read bytes via NFS READ
Run app read client read
1 322,122,547,200 322,122,547,200
2 322,122,547,200 321,048,805,376
3 322,122,547,200 0
4 322,122,547,200 321,593,053,184
I've put the full data in a GitHub gist, along with a graph collected
from a metrics agent:
https://gist.github.com/chilversc/54eb76155ad37b66cb85186e7449beaa
https://gist.githubusercontent.com/chilversc/54eb76155ad37b66cb85186e7449beaa/raw/09828c596d0cfc44bc0eb67f40e4033db202326e/metrics.png
Chris Chilvers <[email protected]> wrote:
> While testing the fscache performance fixes [1] that were merged into 6.4-rc1
> it appears that the caching no longer works. The client will write to the cache
> but never reads.
Can you try reading from afs? You would need to enable CONFIG_AFS_FS in your
kernel if it's not already set.
Install kafs-client and do:
systemctl enable afs.mount
md5sum /afs/openafs.org/software/openafs/1.9.1/openafs-1.9.1-doc.tar.bz2
cat /proc/fs/fscache/stats
umount /afs/openafs.org
md5sum /afs/openafs.org/software/openafs/1.9.1/openafs-1.9.1-doc.tar.bz2
cat /proc/fs/fscache/stats
David
On Thu, May 18, 2023 at 4:21 PM Chris Chilvers <[email protected]> wrote:
>
> On Tue, 16 May 2023 at 20:28, David Wysochanski <[email protected]> wrote:
> >
> > On Tue, May 16, 2023 at 11:42 AM Chris Chilvers <[email protected]> wrote:
> > >
> > > While testing the fscache performance fixes [1] that were merged into 6.4-rc1
> > > it appears that the caching no longer works. The client will write to the cache
> > > but never reads.
> > >
> > Thanks for the report.
> >
> > If you reboot do you see reads from the cache?
>
> On the first read after a reboot it uses the cache, but subsequent
> reads do not use the cache.
>
> > You can check if the cache is being read from by looking in
> > /proc/fs/fscache/stats
> > at the "IO" line:
> > # grep IO /proc/fs/fscache/stats
> > IO : rd=80030 wr=0
>
> Running the tests 4 times (twice before reboot, and twice after) give
> the following metrics:
> FS-Cache I/O (delta)
> Run rd wr
> 1 0 39,250
> 2 130 38,894
> 3 39,000 0
> 4 72 38,991
>
> > Can you share:
> > 1. NFS server you're using (is it localhost or something else)
> > 2. NFS version
>
> The NFS server and client are separate VMs on the same network.
> Server NFS version: Ubuntu 22.04 jammy, kernel 5.15.0-1021-gcp
> Client NFS version: Ubuntu 22.04 jammy, kernel 6.4.0-060400rc1-generic
> (https://kernel.ubuntu.com/~kernel-ppa/mainline/v6.4-rc1/)
> Client nfs-utils: 2.6.3-rc6
> Client cachefilesd: 0.10.10-0.2ubuntu1
>
> > In addition to checking the above for the reads from the cache, you can also
> > see whether NFS reads are going over the wire pretty easily with a similar
> > technique.
> >
> > Copy /proc/self/mounstats to a file before your test, then make a second copy
> > after the test, then run mountstats as follows:
> > mountstats -S /tmp/mountstats.1 -f /tmp/mountstats.2
>
> app read = applications read bytes via read(2)
> client read = client read bytes via NFS READ
>
> Run app read client read
> 1 322,122,547,200 322,122,547,200
> 2 322,122,547,200 321,048,805,376
> 3 322,122,547,200 0
> 4 322,122,547,200 321,593,053,184
>
> I've put the full data in a GitHub gist, along with a graph collected
> from a metrics agent:
> https://gist.github.com/chilversc/54eb76155ad37b66cb85186e7449beaa
> https://gist.githubusercontent.com/chilversc/54eb76155ad37b66cb85186e7449beaa/raw/09828c596d0cfc44bc0eb67f40e4033db202326e/metrics.png
>
Thanks Chris for all this info. I see you're using NFSv3 so I'll
focus on that, and review all this info for clues.
I also have been working on some updated test cases and see some very
unusual behavior like you're reporting.
I also confirmed that adding the two patches for "Issue #1" onto
6.4-rc1 resolve _most_ of the caching issues.
However, even after those patches, in some limited instances, there
are still NFS reads over the wire when there should only be reads from
the cache.
There may be multiple bugs here.
On Fri, 19 May 2023 at 10:40, David Howells <[email protected]> wrote:
>
> Chris Chilvers <[email protected]> wrote:
>
> > While testing the fscache performance fixes [1] that were merged into 6.4-rc1
> > it appears that the caching no longer works. The client will write to the cache
> > but never reads.
>
> Can you try reading from afs? You would need to enable CONFIG_AFS_FS in your
> kernel if it's not already set.
>
> Install kafs-client and do:
>
> systemctl enable afs.mount
> md5sum /afs/openafs.org/software/openafs/1.9.1/openafs-1.9.1-doc.tar.bz2
> cat /proc/fs/fscache/stats
FS-Cache statistics
Cookies: n=13 v=7 vcol=0 voom=0
Acquire: n=13 ok=13 oom=0
LRU : n=1 exp=0 rmv=0 drp=0 at=2833
Invals : n=0
Updates: n=0 rsz=0 rsn=0
Relinqs: n=0 rtr=0 drop=0
NoSpace: nwr=0 ncr=0 cull=0
IO : rd=0 wr=18
RdHelp : RA=18 RP=0 WB=0 WBZ=0 rr=0 sr=0
RdHelp : ZR=1 sh=0 sk=0
RdHelp : DL=18 ds=18 df=0 di=0
RdHelp : RD=0 rs=0 rf=0
RdHelp : WR=18 ws=18 wf=0
This was on an instance that was only just created, so the cache was
initially unused (all the counters were 0).
> umount /afs/openafs.org
> md5sum /afs/openafs.org/software/openafs/1.9.1/openafs-1.9.1-doc.tar.bz2
> cat /proc/fs/fscache/stats
FS-Cache statistics
Cookies: n=13 v=7 vcol=0 voom=0
Acquire: n=26 ok=26 oom=0
LRU : n=1 exp=1 rmv=0 drp=0 at=467
Invals : n=0
Updates: n=0 rsz=0 rsn=0
Relinqs: n=13 rtr=0 drop=13
NoSpace: nwr=0 ncr=0 cull=0
IO : rd=18 wr=18
RdHelp : RA=36 RP=0 WB=0 WBZ=0 rr=0 sr=0
RdHelp : ZR=1 sh=0 sk=0
RdHelp : DL=18 ds=18 df=0 di=0
RdHelp : RD=18 rs=18 rf=0
RdHelp : WR=18 ws=18 wf=0
Looks like the cache is working fine with AFS. The second md5sum
seemed a lot quicker than the first.
On Tue, May 23, 2023 at 7:18 AM Chris Chilvers <[email protected]> wrote:
>
> On Fri, 19 May 2023 at 10:40, David Howells <[email protected]> wrote:
> >
> > Chris Chilvers <[email protected]> wrote:
> >
> > > While testing the fscache performance fixes [1] that were merged into 6.4-rc1
> > > it appears that the caching no longer works. The client will write to the cache
> > > but never reads.
> >
> > Can you try reading from afs? You would need to enable CONFIG_AFS_FS in your
> > kernel if it's not already set.
> >
> > Install kafs-client and do:
> >
> > systemctl enable afs.mount
> > md5sum /afs/openafs.org/software/openafs/1.9.1/openafs-1.9.1-doc.tar.bz2
> > cat /proc/fs/fscache/stats
>
> FS-Cache statistics
> Cookies: n=13 v=7 vcol=0 voom=0
> Acquire: n=13 ok=13 oom=0
> LRU : n=1 exp=0 rmv=0 drp=0 at=2833
> Invals : n=0
> Updates: n=0 rsz=0 rsn=0
> Relinqs: n=0 rtr=0 drop=0
> NoSpace: nwr=0 ncr=0 cull=0
> IO : rd=0 wr=18
> RdHelp : RA=18 RP=0 WB=0 WBZ=0 rr=0 sr=0
> RdHelp : ZR=1 sh=0 sk=0
> RdHelp : DL=18 ds=18 df=0 di=0
> RdHelp : RD=0 rs=0 rf=0
> RdHelp : WR=18 ws=18 wf=0
>
> This was on an instance that was only just created, so the cache was
> initially unused (all the counters were 0).
>
> > umount /afs/openafs.org
> > md5sum /afs/openafs.org/software/openafs/1.9.1/openafs-1.9.1-doc.tar.bz2
> > cat /proc/fs/fscache/stats
>
> FS-Cache statistics
> Cookies: n=13 v=7 vcol=0 voom=0
> Acquire: n=26 ok=26 oom=0
> LRU : n=1 exp=1 rmv=0 drp=0 at=467
> Invals : n=0
> Updates: n=0 rsz=0 rsn=0
> Relinqs: n=13 rtr=0 drop=13
> NoSpace: nwr=0 ncr=0 cull=0
> IO : rd=18 wr=18
> RdHelp : RA=36 RP=0 WB=0 WBZ=0 rr=0 sr=0
> RdHelp : ZR=1 sh=0 sk=0
> RdHelp : DL=18 ds=18 df=0 di=0
> RdHelp : RD=18 rs=18 rf=0
> RdHelp : WR=18 ws=18 wf=0
>
> Looks like the cache is working fine with AFS. The second md5sum
> seemed a lot quicker than the first.
>
> --
> Linux-cachefs mailing list
> [email protected]
> https://listman.redhat.com/mailman/listinfo/linux-cachefs
>
I don't think this test demonstrates the failure Chris is seeing, or
correct me if I'm wrong Chris.
Chris, does this same test fail for you with NFS on 6.4-rc1? The same
test passes for me with NFSv3.
1st time through: writes to the cache seen, no reads: IO : rd=0 wr=16
2nd time through: reads from the cache seen, no writes: IO : rd=16 wr=16
# mount -o vers=3,fsc 127.0.0.1:/export/ /mnt/nfs
# cat /proc/fs/fscache/stats
FS-Cache statistics
Cookies: n=0 v=1 vcol=0 voom=0
Acquire: n=0 ok=0 oom=0
LRU : n=0 exp=0 rmv=0 drp=0 at=0
Invals : n=0
Updates: n=0 rsz=0 rsn=0
Relinqs: n=0 rtr=0 drop=0
NoSpace: nwr=0 ncr=0 cull=0
IO : rd=0 wr=0
RdHelp : RA=0 RP=0 WB=0 WBZ=0 rr=0 sr=0
RdHelp : ZR=0 sh=0 sk=0
RdHelp : DL=0 ds=0 df=0 di=0
RdHelp : RD=0 rs=0 rf=0
RdHelp : WR=0 ws=0 wf=0
# md5sum /mnt/nfs/openafs-1.9.1-doc.tar.bz2
ca25530ec01afc3b71bc081bc93b87cf /mnt/nfs/openafs-1.9.1-doc.tar.bz2
# cat /proc/fs/fscache/stats
FS-Cache statistics
Cookies: n=1 v=1 vcol=0 voom=0
Acquire: n=1 ok=1 oom=0
LRU : n=1 exp=0 rmv=0 drp=0 at=6444
Invals : n=0
Updates: n=1 rsz=0 rsn=0
Relinqs: n=0 rtr=0 drop=0
NoSpace: nwr=0 ncr=0 cull=0
IO : rd=0 wr=16
RdHelp : RA=16 RP=0 WB=0 WBZ=0 rr=0 sr=0
RdHelp : ZR=1 sh=0 sk=0
RdHelp : DL=16 ds=16 df=0 di=0
RdHelp : RD=0 rs=0 rf=0
RdHelp : WR=16 ws=16 wf=0
# umount /mnt/nfs
# mount -o vers=3,fsc 127.0.0.1:/export/ /mnt/nfs
# md5sum /mnt/nfs/openafs-1.9.1-doc.tar.bz2
ca25530ec01afc3b71bc081bc93b87cf /mnt/nfs/openafs-1.9.1-doc.tar.bz2
# cat /proc/fs/fscache/stats
FS-Cache statistics
Cookies: n=1 v=1 vcol=0 voom=0
Acquire: n=2 ok=2 oom=0
LRU : n=1 exp=1 rmv=0 drp=0 at=6710
Invals : n=0
Updates: n=2 rsz=0 rsn=0
Relinqs: n=1 rtr=0 drop=1
NoSpace: nwr=0 ncr=0 cull=0
IO : rd=16 wr=16
RdHelp : RA=32 RP=0 WB=0 WBZ=0 rr=0 sr=0
RdHelp : ZR=1 sh=0 sk=0
RdHelp : DL=16 ds=16 df=0 di=0
RdHelp : RD=16 rs=16 rf=0
RdHelp : WR=16 ws=16 wf=0
# uname -r
6.4.0-rc1
On Fri, May 19, 2023 at 7:53 AM David Wysochanski <[email protected]> wrote:
>
> On Thu, May 18, 2023 at 4:21 PM Chris Chilvers <[email protected]> wrote:
> >
> > On Tue, 16 May 2023 at 20:28, David Wysochanski <[email protected]> wrote:
> > >
> > > On Tue, May 16, 2023 at 11:42 AM Chris Chilvers <[email protected]> wrote:
> > > >
> > > > While testing the fscache performance fixes [1] that were merged into 6.4-rc1
> > > > it appears that the caching no longer works. The client will write to the cache
> > > > but never reads.
> > > >
> > > Thanks for the report.
> > >
> > > If you reboot do you see reads from the cache?
> >
> > On the first read after a reboot it uses the cache, but subsequent
> > reads do not use the cache.
> >
> > > You can check if the cache is being read from by looking in
> > > /proc/fs/fscache/stats
> > > at the "IO" line:
> > > # grep IO /proc/fs/fscache/stats
> > > IO : rd=80030 wr=0
> >
> > Running the tests 4 times (twice before reboot, and twice after) give
> > the following metrics:
> > FS-Cache I/O (delta)
> > Run rd wr
> > 1 0 39,250
> > 2 130 38,894
> > 3 39,000 0
> > 4 72 38,991
> >
> > > Can you share:
> > > 1. NFS server you're using (is it localhost or something else)
> > > 2. NFS version
> >
> > The NFS server and client are separate VMs on the same network.
> > Server NFS version: Ubuntu 22.04 jammy, kernel 5.15.0-1021-gcp
> > Client NFS version: Ubuntu 22.04 jammy, kernel 6.4.0-060400rc1-generic
> > (https://kernel.ubuntu.com/~kernel-ppa/mainline/v6.4-rc1/)
> > Client nfs-utils: 2.6.3-rc6
> > Client cachefilesd: 0.10.10-0.2ubuntu1
> >
> > > In addition to checking the above for the reads from the cache, you can also
> > > see whether NFS reads are going over the wire pretty easily with a similar
> > > technique.
> > >
> > > Copy /proc/self/mounstats to a file before your test, then make a second copy
> > > after the test, then run mountstats as follows:
> > > mountstats -S /tmp/mountstats.1 -f /tmp/mountstats.2
> >
> > app read = applications read bytes via read(2)
> > client read = client read bytes via NFS READ
> >
> > Run app read client read
> > 1 322,122,547,200 322,122,547,200
> > 2 322,122,547,200 321,048,805,376
> > 3 322,122,547,200 0
> > 4 322,122,547,200 321,593,053,184
> >
> > I've put the full data in a GitHub gist, along with a graph collected
> > from a metrics agent:
> > https://gist.github.com/chilversc/54eb76155ad37b66cb85186e7449beaa
> > https://gist.githubusercontent.com/chilversc/54eb76155ad37b66cb85186e7449beaa/raw/09828c596d0cfc44bc0eb67f40e4033db202326e/metrics.png
> >
>
> Thanks Chris for all this info. I see you're using NFSv3 so I'll
> focus on that, and review all this info for clues.
> I also have been working on some updated test cases and see some very
> unusual behavior like you're reporting.
>
> I also confirmed that adding the two patches for "Issue #1" onto
> 6.4-rc1 resolve _most_ of the caching issues.
> However, even after those patches, in some limited instances, there
> are still NFS reads over the wire when there should only be reads from
> the cache.
> There may be multiple bugs here.
I actually misspoke regarding "multiple bugs", as I forgot to add a
small NFS hunk (see below) needed to dhowells 2nd patch (v6 of mm,
netfs, fscache: Stop read optimisation when folio removed from
pagecache).
After the below small hunk was added on top of dhowells 2nd patch, all
my tests pass.
I've also enhanced my existing tests to check NFS READs, fscache
READs, and fscache WRITEs as expected.
And I added an additional test to create files the size of RAM, read
them multiple times, and check for the ops are as expected.
So I'm confident if we add dhowells 2 patches, plus the below hunk for
NFS, these caching issues will be resolved.
diff --git a/fs/nfs/fscache.c b/fs/nfs/fscache.c
index 8c35d88a84b1..d4a20748b14f 100644
--- a/fs/nfs/fscache.c
+++ b/fs/nfs/fscache.c
@@ -180,6 +180,10 @@ void nfs_fscache_init_inode(struct inode *inode)
&auxdata, /* aux_data */
sizeof(auxdata),
i_size_read(inode));
+
+ if (netfs_inode(inode)->cache)
+ mapping_set_release_always(inode->i_mapping);
+
}
/*
