Received-SPF: softfail (google.com: domain of transitioning linux-nfs-owner@vger.kernel.org does not designate 23.128.96.19 as permitted sender) client-ip=23.128.96.19;
MIME-Version: 1.0
References: <20220227231227.9038-1-trondmy@kernel.org> <20220227231227.9038-2-trondmy@kernel.org>
 <20220227231227.9038-3-trondmy@kernel.org> <20220227231227.9038-4-trondmy@kernel.org>
 <20220227231227.9038-5-trondmy@kernel.org> <20220227231227.9038-6-trondmy@kernel.org>
 <20220227231227.9038-7-trondmy@kernel.org> <20220227231227.9038-8-trondmy@kernel.org>
 <20220227231227.9038-9-trondmy@kernel.org> <20220227231227.9038-10-trondmy@kernel.org>
 <20220227231227.9038-11-trondmy@kernel.org> <20220227231227.9038-12-trondmy@kernel.org>
 <20220227231227.9038-13-trondmy@kernel.org> <20220227231227.9038-14-trondmy@kernel.org>
 <20220227231227.9038-15-trondmy@kernel.org> <A7BBBBF2-768E-487C-A890-7E5AF1D40027@redhat.com>
 <88cb6a4d7a074fd4c4c6b59076df766c7de54105.1646922313.git.bcodding@redhat.com>
In-Reply-To: <88cb6a4d7a074fd4c4c6b59076df766c7de54105.1646922313.git.bcodding@redhat.com>
From:   Olga Kornievskaia <aglo@umich.edu>
Date:   Wed, 16 Mar 2022 18:25:10 -0400
Message-ID: <CAN-5tyFsJy-D9cEfk8zCH8-0KErWteVW8Zb7BMqjL7==2qtT_A@mail.gmail.com>
Subject: Re: [PATCH] NFS: Trigger "ls -l" readdir heuristic sooner
To:     Benjamin Coddington <bcodding@redhat.com>
Cc:     trondmy@kernel.org, linux-nfs <linux-nfs@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Precedence: bulk

On Fri, Mar 11, 2022 at 9:40 AM Benjamin Coddington <bcodding@redhat.com> wrote:
>
> .. Something like this does the trick in my testing, but yes will have an
> impact on regular workloads:
>
> 8<------------------------------------------------------------------------
>
> Since commit 1a34c8c9a49e ("NFS: Support larger readdir buffers") has
> updated dtsize and recent improvements to the READDIRPLUS helper heuristic,
> the heuristic may not trigger until many dentries are emitted to userspace,
> which may cause many thousands of GETATTR calls for "ls -l" when the
> directory's pagecache has already been populated.  This typically manifests
> as a much slower total runtime for a _second_ invocation of "ls -l" within
> the directory attribute cache timeouts.
>
> Fix this by emitting only 17 entries for any first pass through the NFS
> directory's ->iterate_shared(), which will allow userpace to prime the
> counters for the heuristic.

Here's for what it's worth. An experiment between linux to linux where
the linux server had a "small" directory structure of 57274
directories, 5727390 files in total where each directory had ~100
files each.
With this patch:

date; time tree vol1 > tree.out && date; time tree vol1 > tree.out
Wed Mar 16 12:21:30 EDT 2022

real    11m7.923s
user    0m20.507s
sys     0m39.683s
Wed Mar 16 12:32:38 EDT 2022

real    40m1.751s
user    0m23.477s
sys     0m45.663s

Without the patch:
date; time tree vol1 > tree.out && date; time tree vol1 > tree.out
Wed Mar 16 13:49:12 EDT 2022

real    10m52.909s
user    0m21.342s
sys     0m39.198s
Wed Mar 16 14:00:05 EDT 2022

real    222m56.990s
user    0m30.392s
sys     2m25.202s


>
> Signed-off-by: Benjamin Coddington <bcodding@redhat.com>
> ---
>  fs/nfs/dir.c | 9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/fs/nfs/dir.c b/fs/nfs/dir.c
> index 7e12102b29e7..dc5fc9ba2c49 100644
> --- a/fs/nfs/dir.c
> +++ b/fs/nfs/dir.c
> @@ -1060,6 +1060,8 @@ static int readdir_search_pagecache(struct nfs_readdir_descriptor *desc)
>         return res;
>  }
>
> +#define NFS_READDIR_CACHE_MISS_THRESHOLD (16UL)
> +
>  /*
>   * Once we've found the start of the dirent within a page: fill 'er up...
>   */
> @@ -1069,6 +1071,7 @@ static void nfs_do_filldir(struct nfs_readdir_descriptor *desc,
>         struct file     *file = desc->file;
>         struct nfs_cache_array *array;
>         unsigned int i;
> +       bool first_emit = !desc->dir_cookie;
>
>         array = kmap(desc->page);
>         for (i = desc->cache_entry_index; i < array->size; i++) {
> @@ -1092,6 +1095,10 @@ static void nfs_do_filldir(struct nfs_readdir_descriptor *desc,
>                         desc->ctx->pos = desc->dir_cookie;
>                 else
>                         desc->ctx->pos++;
> +               if (first_emit && i > NFS_READDIR_CACHE_MISS_THRESHOLD + 1) {
> +                       desc->eob = true;
> +                       break;
> +               }
>         }
>         if (array->page_is_eof)
>                 desc->eof = !desc->eob;
> @@ -1173,8 +1180,6 @@ static int uncached_readdir(struct nfs_readdir_descriptor *desc)
>         return status;
>  }
>
> -#define NFS_READDIR_CACHE_MISS_THRESHOLD (16UL)
> -
>  static bool nfs_readdir_handle_cache_misses(struct inode *inode,
>                                             struct nfs_readdir_descriptor *desc,
>                                             unsigned int cache_misses,
> --
> 2.31.1
>