Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
MIME-Version: 1.0
From:   Chris Chilvers <chilversc@gmail.com>
Date:   Tue, 16 May 2023 16:42:39 +0100
Message-ID: <CAAmbk-f_U8CPcTQM866L572uUHdK4p5iWKnUQs4r8fkW=6RW9g@mail.gmail.com>
Subject: [BUG] fscache writing but not reading
To:     linux-nfs@vger.kernel.org, linux-cachefs@redhat.com
Cc:     Benjamin Maynard <benmaynard@google.com>, brennandoyle@google.com
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

While testing the fscache performance fixes [1] that were merged into 6.4-rc1
it appears that the caching no longer works. The client will write to the cache
but never reads.

I suspect this is related to known issue #1. However, I tested the client
with rsize less than, equal to, and greater than readahead, and in all cases
I see the issue.

If I apply both the patches [2], [3] from the known issues to 6.4-rc1 then the
cache works as expected. I suspect only patch [2] is required but have not
tested patch [2] without [3].

Testing
=======
For the test I was just using dd to read 300 x 1gb files from an NFS
share to fill the cache, then repeating the read.

In the first test run, /var/cache/fscache steadily filled until reaching
300 GB. The read I/O was less than 1 MB/s, and the write speed was fairly
constant 270 MB/s.

In the second run, /var/cache/fscache remained at 300 GB, so no new data was
being written. However, the read I/O remained at less than 1 MB/s and the write
rate at 270 MB/s.

    /var/cache/fscache
                | 1st run     | 2nd run
    disk usage  | 0 -> 300 GB | 300 GB
    read speed  | < 1 MB/s    | < 1 MB/s
    write speed | 270 MB/s    | 270 MB/s

This seems to imply that the already cached data was being read from the source
server and re-written to the cache.

Known Issues
============
1. Unit test setting rsize < readahead does not properly read from
fscache but re-reads data from the NFS server
* This will be fixed with another dhowells patch [2]:
  "[PATCH v6 2/2] mm, netfs, fscache: Stop read optimisation when
folio removed from pagecache"

2. "Cache volume key already in use" after xfstest runs involving
multiple mounts
* Simple reproducer requires just two mounts as follows:
 mount -overs=4.1,fsc,nosharecache -o
context=system_u:object_r:root_t:s0  nfs-server:/exp1 /mnt1
 mount -overs=4.1,fsc,nosharecache -o
context=system_u:object_r:root_t:s0  nfs-server:/exp2 /mnt2
* This should be fixed with dhowells patch [3]:
  "[PATCH v5] vfs, security: Fix automount superblock LSM init
problem, preventing NFS sb sharing"

References
==========

[1] https://lore.kernel.org/linux-nfs/20230220134308.1193219-1-dwysocha@redhat.com/
[2] https://lore.kernel.org/linux-nfs/20230216150701.3654894-1-dhowells@redhat.com/T/#mf3807fa68fb6d495b87dde0d76b5237833a0cc81
[3] https://lore.kernel.org/linux-kernel/217595.1662033775@warthog.procyon.org.uk/