Received-SPF: pass (google.com: domain of linux-nfs+bounces-1998-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Precedence: bulk
MIME-Version: 1.0
References: <PH0PR14MB5493F59229B802B871407F9CAA442@PH0PR14MB5493.namprd14.prod.outlook.com>
 <1AF9A62E-55BE-4A08-95D6-26784218C940@oracle.com> <PH0PR14MB5493D4DF2877FDD93BB49AF4AA442@PH0PR14MB5493.namprd14.prod.outlook.com>
 <A14BC53D-18C7-43CB-B64E-3B215EB12D04@oracle.com>
In-Reply-To: <A14BC53D-18C7-43CB-B64E-3B215EB12D04@oracle.com>
From: Dan Shelton <dan.f.shelton@gmail.com>
Date: Sat, 17 Feb 2024 00:31:34 +0100
Message-ID: <CAAvCNcCo1phpMqLsVz_GtrKrVb4Pgv_UfTXtokvqpSLFjVJKoA@mail.gmail.com>
Subject: Re: apparent scaling problem with delegations
To: Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Thu, 8 Feb 2024 at 23:06, Chuck Lever III <chuck.lever@oracle.com> wrote=
:
>
>
>
> > On Feb 8, 2024, at 4:45=E2=80=AFPM, Charles Hedrick <hedrick@rutgers.ed=
u> wrote:
> >
> >> From: Chuck Lever III <chuck.lever@oracle.com>
> >>> On Feb 8, 2024, at 3:26=E2=80=AFPM, Charles Hedrick <hedrick@rutgers.=
edu> wrote:
> >>>
> >>> We just turned delegations on for two big NFS servers. One characteri=
stic
> >>> of our site is that we have lots of small files and lots of files ope=
n.
> >>>
> >>> On one server, CPU in system state went to 30%, and NFS performance g=
round
> >>> to a halt. When I disabled delegations it came back. The other server=
 was
> >>> showing high CPU on nfsd, but not enough to disable the server, so I =
looked
> >>> around. The server where delegations are still on is spending most of=
 its time
> >>> in nfsd_file_lru_cb. That's not the case with the server where we've =
disabled
> >>> delegations. Here's a typical perf top
> >>>
> >>> Overhead  Shared Object                                 Symbol
> >>>    44.87%  [kernel]                                      [k] __list_l=
ru_walk_one
> >>>    13.18%  [kernel]                                      [k] native_q=
ueued_spin_lock_slowpath.part.0
> >>>     7.24%  [kernel]                                      [k] nfsd_fil=
e_lru_cb
> >>>     2.61%  [kernel]                                      [k] sha1_tra=
nsform
> >>>     0.99%  [kernel]                                      [k] __crypto=
_alg_lookup
> >>>     0.95%  [kernel]                                      [k] _raw_spi=
n_lock
> >>>     0.89%  [kernel]                                      [k] memcpy_e=
rms
> >>>     0.77%  [kernel]                                      [k] mutex_lo=
ck
> >>>     0.65%  [kernel]                                      [k] svc_tcp_=
recvfrom
> >>>
> >>> I looked at the code. I'm not clear whether there's a problem with GC=
'ing the
> >>> entries, or it's just being called too often (maybe a table is too sm=
all?)
> >>>
> >>> When I disabled delegations, it immediately stopped spending all that=
 time
> >>> in nfsd_file_lru_cb. The number of delegations starting going down sl=
owly.
> >>> I suspect our system needs a lot more delegations than the maximum ta=
ble
> >>> size, and it's thrashing. The sizes were about 40,000 and
> >>> 60,000 on the two machines.  Systems are 384 G and 768 G, respectivel=
y.
> >>> The maximum number of delegations is smaller than I would have expect=
ed
> >>> based on comments in the code.
> >
> >> When reporting such problems, please include the kernel version
> >> on your NFS servers. Some late 5.x kernels have known problems
> >> with the NFSD file cache.
> >
> > My apologies.Ubuntu 5.15.0-91-generic , which is 5.15.131.
>
> That kernel is likely to have file cache issues with symptoms
> very much as you described above. The issues are thought to
> be addressed by kernel release v6.2.

Is there a way to turn the file cache off for nfsd?

Dan
--=20
Dan Shelton - Cluster Specialist Win/Lin/Bsd