MIME-Version: 1.0
References: <CAK3fRr9EL-YOp3yUr5tMe5zuiERxHjkLDhAaWkxxEHjtfSwisQ@mail.gmail.com>
 <20190221041820.GA4625@fieldses.org>
In-Reply-To: <20190221041820.GA4625@fieldses.org>
From:   James Pearson <jcpearson@gmail.com>
Date:   Thu, 21 Feb 2019 12:35:46 +0000
Message-ID: <CAK3fRr-y3YjX-8hMtutvpy=MQCyqH8YmRBwxyGJ4HmJQwsLyGw@mail.gmail.com>
Subject: Re: nfsd thread limit and UDP ?
To:     "J. Bruce Fields" <bfields@fieldses.org>
Cc:     linux-nfs@vger.kernel.org
Content-Type: text/plain; charset="UTF-8"
Sender: linux-nfs-owner@vger.kernel.org
Precedence: bulk

On Thu, 21 Feb 2019 at 04:18, J. Bruce Fields <bfields@fieldses.org> wrote:
>
> On Wed, Feb 20, 2019 at 11:28:53AM +0000, James Pearson wrote:
> > On a very busy NFSv3 server (running CentOS 6), we recently upped the
> > nfsd thread count to 1024 - but this caused client mount requests over
> > UDP to fail.
> >
> > We configure all our clients to use TCP for NFS mounts, but the
> > automounter (automountd) on MacOS (up to version MacOS 10.12) seeds a
> > 'null call' to the NFS server over UDP before attempting the mount -
> > but the server appears to ignore any UDP requests - and the automount
> > fails
>
> By the way, you might also just turn off UDP.  (Start run rpc.nfsd with
> the -U option.)  Hopefully MacOS can handle that case.

We tried that - but when we restarted nfs, some existing mounts hung
(not sure why, as we should be just using TCP everywhere) ... although
when tested on a test server, the MacOS automounter worked fine

I tried your patch - it doesn't apply 'as is' on a CentOS 6 kernel -
but with a bit of manual hacking, I can get it to fit

However, the net/sunrpc/svcsock.c in these kernels has an extra call
to svc_sock_setbufsize() :

        /* Initialize the socket */
        if (sock->type == SOCK_DGRAM)
                svc_udp_init(svsk, serv);
        else {
                /* initialise setting must have enough space to
                 * receive and respond to one request.
                 */
                svc_sock_setbufsize(svsk->sk_sock, 4 * serv->sv_max_mesg,
                                        4 * serv->sv_max_mesg);
                svc_tcp_init(svsk, serv);
        }

I tried replacing that svc_sock_setbufsize() with:

                svc_sock_setbufsize(svsk, 4);

but that just caused the whole machine to lock up shortly after
sunrpc.ko was loaded ...

However, things seem to work fine if I call a copy of the original
svc_sock_setbufsize() at that point in the code with the original args
...

i.e. mounts over UDP (and MacOS automounts) now work with nfsd threads
over 1017 (I tried 2048 ... and it worked)

Incidentally, I came across an old thread on this list that appears to
be related to this issue (well, it mentions a 1020 thread limit and
buffer size wraps in svc_sock_setbufsize() ???) :

 https://www.spinics.net/lists/linux-nfs/msg34927.html

... but I'm not sure what the result of that was (nor if it is
actually related to the issue here) ?

Thanks

James Pearson