Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
From:   Benjamin Coddington <bcodding@redhat.com>
To:     linux-nfs@vger.kernel.org
Subject: [PATCH v1 00/10] NFS client readdir per-page validation
Date:   Wed, 20 Jan 2021 11:59:44 -0500
Message-Id: <cover.1611160120.git.bcodding@redhat.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk

Due to the constraint that the NFS readdir page cache must contain every
entry in cookie order from zero up to the entry of interest, the time or
operations required to complete a directory listing increase exponentially
with the size of the directory if the client is unable to keep the pagecache
stable.  The pagecache can be invalidated by a changing directory, or by
memory pressure on the client.  This can cause some trouble for the NFS
client reading large directories over slow connections.

We have a hueristic that allows eventual completion, but it only works as
long as there are no other readers simultaneously filling the pagecache.

I think we can resolve this problem by implementing per-page validation.  By
storing the directory's change version on the page, and checking for changes
to the directory on every READDIR, we can validate pages against each
reader's version of entry aligment.  Rather than attempting to assemble the
entire directory in a consistent manner in the pagecache, we can just
retrieve the section we're interested in emitting.

This set is a first pass at implementing this idea.  Please help me pound it
into acceptable shape or point out problems!  Thanks for any feedback.

Here's a small program that does a great job of demonstraing the client's
current readdir pagecache performance problem by dropping the directory's
pagecache at an interval while trying to emit every entry:

#define _GNU_SOURCE
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <sched.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <signal.h>

#define BUF_SIZE 1024
#define EVICT_INTERVAL 5

int evict_pagecache(int fd) {
                return posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED);
}

int main(int argc, char **argv)
{
        int dir_fd;
        pid_t pid;
        cpu_set_t *cpusetp = CPU_ALLOC(2);
        off_t off;

        char buf[BUF_SIZE];

        if (argc < 2) { printf("%s <dir>\n", argv[0]); return 1; }

        dir_fd = open(argv[1], O_RDONLY|O_DIRECTORY|O_CLOEXEC);
        if (dir_fd < 0) { printf("cannot open dir\n"); return 1; }

        pid = fork();
        if (pid == 0) {
                CPU_SET(1, cpusetp);
                sched_setaffinity(0, sizeof(cpu_set_t), cpusetp);
                do {
                        evict_pagecache(dir_fd);
                        off = lseek(dir_fd, 0, SEEK_CUR);
                        printf("currently at %llu\n", off);
                        usleep(EVICT_INTERVAL * 1000000);
                } while (1);
        } else {
                CPU_SET(0, cpusetp);
                sched_setaffinity(0, sizeof(cpu_set_t), cpusetp);
                while (syscall(SYS_getdents, dir_fd, buf, BUF_SIZE)) {}
                kill(pid, SIGINT);
        }

        close(dir_fd);
        return 0;
}


Benjamin Coddington (10):
  NFS: save the directory's change attribute on pagecache pages
  NFSv4: Send GETATTR with READDIR
  NFS: Add a struct to track readdir pagecache location
  NFS: Keep the readdir pagecache cursor updated
  NFS: readdir per-page cache validation
  NFS: stash the readdir pagecache cursor on the open directory context
  NFS: Support headless readdir pagecache pages
  NFS: Reset pagecache cursor on llseek
  NFS: Remove nfs_readdir_dont_search_cache()
  NFS: Revalidate the directory pagecache on every nfs_readdir()

 fs/nfs/dir.c              | 210 +++++++++++++++++++++++++++-----------
 fs/nfs/nfs42proc.c        |   2 +-
 fs/nfs/nfs4proc.c         |  27 +++--
 fs/nfs/nfs4xdr.c          |   6 ++
 include/linux/nfs_fs.h    |   8 +-
 include/linux/nfs_fs_sb.h |   5 +
 include/linux/nfs_xdr.h   |   2 +
 7 files changed, 188 insertions(+), 72 deletions(-)

-- 
2.25.4