Received-SPF: pass (google.com: domain of linux-nfs+bounces-1071-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1;
Precedence: bulk
MIME-Version: 1.0
References: <CAAvCNcDtTNDRvUVjUy4BE7eBCgmkb6hfkq3P0jaGDC=OXg0=6g@mail.gmail.com>
 <89fba598b0b93cf97bf208e106001f74eadd1829.camel@kernel.org>
 <CAAvCNcBSW7ZH4LpSLLo+Bmh2kQ=w5sFrLJ-PKDDSBKwkS5fd5g@mail.gmail.com> <BFA98D60-DBA4-4CB1-AC9C-D546B369855F@oracle.com>
In-Reply-To: <BFA98D60-DBA4-4CB1-AC9C-D546B369855F@oracle.com>
From: Rick Macklem <rick.macklem@gmail.com>
Date: Fri, 12 Jan 2024 18:40:08 -0800
Message-ID: <CAM5tNy614H9T4QwWBd1DGJKhMHO-U0yrMrwzgvx33MaWqFdSPA@mail.gmail.com>
Subject: Re: Increasing NFSD_MAX_OPS_PER_COMPOUND to 96
To: Chuck Lever III <chuck.lever@oracle.com>
Cc: Dan Shelton <dan.f.shelton@gmail.com>, Jeff Layton <jlayton@kernel.org>, 
	Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Fri, Jan 12, 2024 at 5:53=E2=80=AFPM Chuck Lever III <chuck.lever@oracle=
.com> wrote:
>
>
>
> > On Jan 12, 2024, at 8:47=E2=80=AFPM, Dan Shelton <dan.f.shelton@gmail.c=
om> wrote:
> >
> > On Sat, 13 Jan 2024 at 02:32, Jeff Layton <jlayton@kernel.org> wrote:
> >>
> >> On Sat, 2024-01-13 at 01:19 +0100, Dan Shelton wrote:
> >>> Hello!
> >>>
> >>> We've been experiencing significant nfsd performance problems with a
> >>> customer who has a deeply nested filesystem hierarchy, lots of
> >>> subdirs, some of them 60-80 dirs deep (!!), which leads to an
> >>> exponentially slowdown with nfsd accesses.
> >>>
> >>> Some of the issues have been addressed by implementing a better
> >>> directory walker via multiple dir fds and openat() (instead of just
> >>> cwd+open()), but the nfsd side still was a pretty dramatic issue,
> >>> until we bumped #define NFSD_MAX_OPS_PER_COMPOUND in
> >>> linux-6.7/fs/nfsd/nfsd.h from 50 to 96. After that the nfsd side
> >>> behaved MUCH more performant.
> >>>
> >>
> >> I guess your clients are trying to do a long pathwalk in a single
> >> COMPOUND?
> >
> > Likely.
>
> That's known bad client behavior, btw. It won't scale
> in the number of path components.
>
>
> >> Is this the windows client?
> >
> > No, clients are Solaris 11, Linux and freeBSD
>
> Solaris 11 is known to send COMPOUNDs that are too large
> during mount, but the rest of the time these three client
> implementations are not known to send large COMPOUNDs.
Actually the FreeBSD client is the same as Solaris, in that it does the
entire mount path in one compound. If you were to attempt a mount
with more than 48 components, it would exceed 50 ops in the compound.
I don't think it can exceed 50 ops any other way.

rick

>
>
> >> At first glance, I don't see any real downside to increasing that valu=
e.
> >> Maybe we can bump it to 100 or so? What would probably be best is to
> >> propose a patch so we can discuss the change formally.
> >
> > OK. How does this work?
>
> Let's back up a minute.
>
> I'd like to see raw packet captures with the current
> MAX_OPS setting and the new larger one. Something is not
> adding up.
>
>
> --
> Chuck Lever
>
>