Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Subject: [PATCH RFC 0/8] RPC service thread scheduler optimizations
From:   Chuck Lever <cel@kernel.org>
To:     linux-nfs@vger.kernel.org
Cc:     Chuck Lever <chuck.lever@oracle.com>, lorenzo@kernel.org,
        neilb@suse.de, jlayton@redhat.com, david@fromorbit.com
Date:   Thu, 29 Jun 2023 14:42:23 -0400
Message-ID: <168806401782.1034990.9686296943273298604.stgit@morisot.1015granger.net>
User-Agent: StGit/1.5
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
Precedence: bulk

Hi -

Walking a linked list to find an idle thread is not CPU cache-
friendly, and in fact I've noted palpable per-request latency
impacts as the number of nfsd threads on the server increases.

After discussing some possible improvements with Jeff at LSF/MM,
I've been experimenting with the following series. I've measured an
order of magnitude latency improvement in the thread lookup time,
and have managed to keep the whole thing lockless.

The only thing I don't like is that allocating the idle bitmaps in
advance means we've got an /a priori/ cap on the number of NFSD
threads that can be created. I'd love to find a way to enable
the pool idle bitmaps to expand dynamically. Suggestions welcome.

Cc: Lorenzo - I'm not sure whether this work would be applied first
or whether your rpc_status patch would be. If yours goes first, I
can handle converting nfsd_rpc_status() to use the pool's thread
xarray.

---

Chuck Lever (8):
      SUNRPC: Deduplicate thread wake-up code
      SUNRPC: Report when no service thread is available.
      SUNRPC: Split the svc_xprt_dequeue tracepoint
      SUNRPC: Clean up svc_set_num_threads
      SUNRPC: Replace dprintk() call site in __svc_create()
      SUNRPC: Replace sp_threads_all with an xarray
      SUNRPC: Convert RQ_BUSY into a per-pool bitmap
      SUNRPC: Don't disable BH's when taking sp_lock


 fs/nfsd/nfssvc.c              |   3 +-
 include/linux/sunrpc/svc.h    |  17 ++--
 include/trace/events/sunrpc.h | 160 ++++++++++++++++++++++++++++----
 net/sunrpc/svc.c              | 169 ++++++++++++++++++++++------------
 net/sunrpc/svc_xprt.c         | 103 +++++++++++----------
 5 files changed, 316 insertions(+), 136 deletions(-)

--
Chuck Lever