Received-SPF: pass (google.com: domain of linux-nfs+bounces-2386-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Precedence: bulk
MIME-Version: 1.0
References: <CAAvCNcDtTNDRvUVjUy4BE7eBCgmkb6hfkq3P0jaGDC=OXg0=6g@mail.gmail.com>
 <CAKAoaQmmEv+HRjmBMrSMGZn9RQr8C=2W4yeX4vNnohXFJPCV5A@mail.gmail.com>
 <65a29ca8.6b0a0220.ad415.d6d8.GMR@mx.google.com> <CAKAoaQkZ+b7NfrVi=gu1vCJBvv10=k85bG_kZV9G3jE45OOquw@mail.gmail.com>
 <0cd8fbfc707f86784dc7d88653b05cd355f89aad.camel@kernel.org>
 <24ACA376-5239-4941-BE53-70BF5E5E4683@oracle.com> <CAKAoaQny6G=JcKpJTYeLmNBEMgNkkc--T0Uvs1YbEX+JUD-PoA@mail.gmail.com>
 <CANH4o6NcMbcNKxARcqhthXWkKk6_r31iKGjnS-RhFBB_AJFaJg@mail.gmail.com>
 <470318C6-3252-445F-94F3-DDB7727F84C7@oracle.com> <CAKAoaQ=6nGHD0uA+9EaQQPWBk8dvq0XVUPPgPAhbh=XPk+ecSg@mail.gmail.com>
 <76D8D54D-AE71-4D4C-AD61-2D2232FB1ECB@oracle.com>
In-Reply-To: <76D8D54D-AE71-4D4C-AD61-2D2232FB1ECB@oracle.com>
From: Cedric Blancher <cedric.blancher@gmail.com>
Date: Tue, 19 Mar 2024 07:25:00 +0100
Message-ID: <CALXu0UeFBmX0y3dJH-nbdm7eQeOCMFNp7GDzthstLW0iFd6cUw@mail.gmail.com>
Subject: Re: |ca_maxoperations| - tuneable ? / was: Re: RFE: Linux nfsd's
 |ca_maxoperations| should be at *least* |64| ... / was: Re: kernel.org list
 issues... / was: Fwd: Turn NFSD_MAX_* into tuneables ? / was: Re: Increasing
 NFSD_MAX_OPS_PER_COMPOUND to 96
To: Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Sat, 16 Mar 2024 at 17:35, Chuck Lever III <chuck.lever@oracle.com> wrot=
e:
>
>
>
> > On Mar 16, 2024, at 7:55=E2=80=AFAM, Roland Mainz <roland.mainz@nrubsig=
.org> wrote:
> >
> > On Thu, Jan 18, 2024 at 3:52=E2=80=AFPM Chuck Lever III <chuck.lever@or=
acle.com> wrote:
> >>> On Jan 18, 2024, at 4:44=E2=80=AFAM, Martin Wege <martin.l.wege@gmail=
.com> wrote:
> >>> On Thu, Jan 18, 2024 at 2:57=E2=80=AFAM Roland Mainz <roland.mainz@nr=
ubsig.org> wrote:
> >>>> On Sat, Jan 13, 2024 at 5:10=E2=80=AFPM Chuck Lever III <chuck.lever=
@oracle.com> wrote:
> >>>>>> On Jan 13, 2024, at 10:09=E2=80=AFAM, Jeff Layton <jlayton@kernel.=
org> wrote:
> >>>>>> On Sat, 2024-01-13 at 15:47 +0100, Roland Mainz wrote:
> >>>>>>> On Sat, Jan 13, 2024 at 1:19=E2=80=AFAM Dan Shelton <dan.f.shelto=
n@gmail.com> wrote:
> > [snip]
> >>>> That assumes that no process does random access into deep subdirs. I=
n
> >>>> that case the performance is absolutely terrible, unless you devote
> >>>> lots of memory to a giant cache (which is not feasible due to cache
> >>>> expiration limits, unless someone (please!) finally implements
> >>>> directory delegations).
> >>
> >> Do you mean not feasible for your client? Lookup caches
> >> have been part of operating systems for decades. Solaris,
> >> FreeBSD, and Linux all have one. Does the Windows kernel
> >> have one that mfs-nfs41-client can use?
> >
> > The ms-nfs41-client has its own cache.
> > Technically Windows has another, but that is in the kernel and
> > difficult to connect to the NFS client daemon without performance
> > issues.
> >
> > [snip]
> >> Sending a full path in a single COMPOUND is one way to
> >> handle path resolution, but it has so many limitations
> >> that it's really not the mechanism of choice.
>
> Yes, COMPOUND was added to NFSv4 as a possible
> way to manage network latency, but in hindsight
> I think the NFS community now recognizes that
> there are more effective strategies to deal with
> network latency than creating more and more
> complicated COMPOUND operations. Client-side
> caching, for instance, is a much better choice.

I have a severe hiccup now after reading THAT comment. Every
generation of IT engineers makes the same damn mistakes, and it takes
them ~10 years to realise their mistakes.

So here is the comment - before my first coffee - from someone with a
grey beard, who is old enough to deal with Mintel, the first UNIX and
the first RFS, NFS, AFS, DFS:
Mistake 1: Caching will solve it all. DFS (the follow up to AFS) tried
that to an absurd extent, and failed badly, too complex, too buggy and
too cpu and memory intensive. Granted the bugs were fixed over time,
but by then the reputation was ruined.
Mistake 2: Caching is always possible. Mounting /var/mail with
actimeo=3D0 is the classical example, HPC another popular one.
Mistake 3: The cache memory is unlimited. We had that one with
Solaris's builtin name cache, and then ZFS. Memory is limited, and
just making the caches 2x, 8x, 32x times bigger doesn't give you any
benefits, because cache expiration/timeout. Of course you can try to
keep the cache "hot", or try delegations, or move data ownership to
another server closer to the client. See DFS above. Did not work.
Google also "law of diminishing returns"
Mistake 4: The network has unlimited bandwidth, so we can keep the
local cache updated/hot, or abuse it otherwise. Unlike our dreams in
the 1990 that we will have 100GB/s Infiniband networks in our laptops
by 2020, the real word laptop in 2024 maxes out at 1000baseT, and most
rural offices still have 100baseT
Mistake 5: The main memory is unlimited. That ignores the fact that
SUN promised us that NFSv4 will not require more memory than NFSv3.
NFSv4 still has to serve the embedded/IoT use case, either for data,
or for diskless boot from NFS(v4). Those machines cannot waste 512MB
on your dream cache with their 8MB main memory, which is also not
going to work because of "Mistake 3". The law of diminishing returns
sends you your greetings.

So complex COMPOUND operations are not that bad, but they are also not
the perfect solution for everything. Likewise, giant client-side
caches are not the perfect solution for everything, neither are they
feasible in all scenarios. Oh delete the "all" and replace with
"most".

Ced
--=20
Cedric Blancher <cedric.blancher@gmail.com>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur