Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Wed, 16 Feb 2022 14:22:15 -0500
To:     Patrick Goetz <pgoetz@math.utexas.edu>
Cc:     "linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Subject: Re: How are client requests load balanced across multiple nfsd
 processes?
Message-ID: <20220216192215.GB29074@fieldses.org>
References: <19e14932-ed88-60a1-844a-0e17deee269d@math.utexas.edu>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <19e14932-ed88-60a1-844a-0e17deee269d@math.utexas.edu>
User-Agent: Mutt/1.5.21 (2010-09-15)
From:   bfields@fieldses.org (J. Bruce Fields)
Precedence: bulk

On Tue, Feb 15, 2022 at 04:13:25PM -0600, Patrick Goetz wrote:
> When I set
> 
>   RPCNFSDCOUNT=16
> 
> what I thought this did was spin up an nfsd thread master with 15
> threads and the thread master would pass out client requests to the
> threads, but looking at /proc/$PID/status -> TGID clearly indicates
> these are all entirely separate processes. (I wasn't sure if ps
> showed threads as separate processes; apparently it doesn't.)

They're all kernel tasks, which makes the distinction between "thread"
and "process" a little vague.

> So the question is how do different client requests get farmed out
> to different nfsd daemons for service? Who's actually listening on
> port 2049?

There's no user process that calls "listen"; knfsd's normal rpc handling
is all in-kernel.  Incoming rpc's may be handed to any of those 16 tasks
for processing.  A single task just runs a loop where it receives an
rpc, handles it, and sends a response back.

> This was all prompted by some vendor trying to sell me an EC
> (Erasure Coding) n+m system who commented "NFS isn't multi-threaded,
> NFS can only communicate with one server, for a shared/mounted
> filesystem, so it will always be limited to the speed of that NFS
> Server. POSIX/Multi-threaded means the filesystem is parallel and
> can be reading/writing to multiple nodes at once in a storage
> cluster/setup. The opposite of NFS."

That explanation is a little muddled.  NFS clients and servers both
typically have lots of parallelism.  Whether it's sufficient for your
purposes depends on exactly what you need.

But, yes, they're mostly correct to say that, in the absence of pNFS,
"NFS can only communicate with one server, for a shared/mounted
filesystem, so it will always be limited to the speed of that NFS
Server".

> I think pNFS addresses this, but then how does one implement pNFS?

So, right, pNFS can let you perform IO to multiple servers
simultaneously, if that's what you need.

The Linux NFS client has support for pNFS, but the kernel server
doesn't, so you'd need to look elsewhere for a pNFS server.

Whether any of this is useful to you depends on exactly what problem
you're trying to solve.

--b.