2008-07-16 19:07:00

by J. Bruce Fields

[permalink] [raw]
Subject: Re: Massive NFS problems on large cluster with large number of mounts

On Wed, Jul 16, 2008 at 11:49:51AM +0200, Carsten Aulbert wrote:
> Hi Trond et al.
>
> I'm following up on this discussion because we hit another problem:
>
> Trond Myklebust wrote:
>
> >
> > Alternatively, just change the values of /proc/sys/sunrpc/min_resvport
> > and /proc/sys/sunrpc/max_resvport to whatever range of ports you
> > actually want to use.
>
> This works like a charm, however, if you set these values before
> restarting the nfs-kernel-server then you are in deep trouble, since
> when nfsd wants to start it needs to register with the portmapper, right?
>
> But what happens if this requests comes from a high^Wunpriviliged port?
> Right:
> Jul 16 11:46:43 d23 portmap[8216]: connect from 127.0.0.1 to set(nfs):
> request from unprivileged port
> Jul 16 11:46:43 d23 nfsd[8214]: nfssvc: writting fds to kernel failed:
> errno 13 (Permission denied)
> Jul 16 11:46:44 d23 kernel: [ 8437.726223] NFSD: Using
> /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
> Jul 16 11:46:44 d23 kernel: [ 8437.800607] NFSD: starting 90-second
> grace period
> Jul 16 11:46:44 d23 kernel: [ 8437.842891] nfsd: last server has exited
> Jul 16 11:46:44 d23 kernel: [ 8437.879940] nfsd: unexporting all filesystems
> Jul 16 11:46:44 d23 nfsd[8214]: nfssvc: Address already in use
>
>
> Changing /proc/sys/sunrpc/max_resvport to 1023 again resolves this
> issue, however defeats the purpose for the initial problem. I still need
> to look into the code for hte portmapper, but is it easily possible that
> the portmapper would accept nfsd requests from "insecure" ports also?
> Since e are (mostly) in a controlled environment that should not pose a
> problem.
>
> Anyone with an idea?

The immediate problem seems like a kernel bug to me--it seems to me that
the calls to local daemons should be ignoring {min_,max}_resvport. (Or
is there some way the daemons can still know that those calls come from
the local kernel?)

--b.


2008-07-17 05:53:31

by Carsten Aulbert

[permalink] [raw]
Subject: Re: Massive NFS problems on large cluster with large number of mounts

Hi all,

J. Bruce Fields wrote:

>> Changing /proc/sys/sunrpc/max_resvport to 1023 again resolves this
>> issue, however defeats the purpose for the initial problem. I still need
>> to look into the code for hte portmapper, but is it easily possible that
>> the portmapper would accept nfsd requests from "insecure" ports also?
>> Since e are (mostly) in a controlled environment that should not pose a
>> problem.
>>
>> Anyone with an idea?
>
> The immediate problem seems like a kernel bug to me--it seems to me that
> the calls to local daemons should be ignoring {min_,max}_resvport. (Or
> is there some way the daemons can still know that those calls come from
> the local kernel?)

I just found this in the Makefile for the portmapper:

# To disable tcp-wrapper style access control, comment out the following
# macro definitions. Access control can also be turned off by providing
# no access control tables. The local system, since it runs the portmap
# daemon, is always treated as an authorized host.

HOSTS_ACCESS= -DHOSTS_ACCESS
#WRAP_LIB = $(WRAP_DIR)/libwrap.a
WRAP_LIB = -lwrap

# Comment out if your RPC library does not allocate privileged ports for
# requests from processes with root privilege, or the new portmap will
# always reject requests to register/unregister services on privileged
# ports. You can find out by running "rpcinfo -p"; if all mountd and NIS
# daemons use a port >= 1024 you should probably disable the next line.

CHECK_PORT = -DCHECK_PORT

I'll try to head down the road of not checking for the ports anymore -
on exposed ports I could block the listening daemons from the outside
world by iptables. Not nice, but probably a solution (and yet another
custom package for us).

Anyone who knows a good reason not to walk this route?

Cheers

Carsten

2008-07-17 14:47:27

by Chuck Lever III

[permalink] [raw]
Subject: Re: Massive NFS problems on large cluster with large number of mounts

On Wed, Jul 16, 2008 at 3:06 PM, J. Bruce Fields <[email protected]> wrote:
> On Wed, Jul 16, 2008 at 11:49:51AM +0200, Carsten Aulbert wrote:
>> Hi Trond et al.
>>
>> I'm following up on this discussion because we hit another problem:
>>
>> Trond Myklebust wrote:
>>
>> >
>> > Alternatively, just change the values of /proc/sys/sunrpc/min_resvport
>> > and /proc/sys/sunrpc/max_resvport to whatever range of ports you
>> > actually want to use.
>>
>> This works like a charm, however, if you set these values before
>> restarting the nfs-kernel-server then you are in deep trouble, since
>> when nfsd wants to start it needs to register with the portmapper, right?
>>
>> But what happens if this requests comes from a high^Wunpriviliged port?
>> Right:
>> Jul 16 11:46:43 d23 portmap[8216]: connect from 127.0.0.1 to set(nfs):
>> request from unprivileged port
>> Jul 16 11:46:43 d23 nfsd[8214]: nfssvc: writting fds to kernel failed:
>> errno 13 (Permission denied)
>> Jul 16 11:46:44 d23 kernel: [ 8437.726223] NFSD: Using
>> /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
>> Jul 16 11:46:44 d23 kernel: [ 8437.800607] NFSD: starting 90-second
>> grace period
>> Jul 16 11:46:44 d23 kernel: [ 8437.842891] nfsd: last server has exited
>> Jul 16 11:46:44 d23 kernel: [ 8437.879940] nfsd: unexporting all filesystems
>> Jul 16 11:46:44 d23 nfsd[8214]: nfssvc: Address already in use
>>
>>
>> Changing /proc/sys/sunrpc/max_resvport to 1023 again resolves this
>> issue, however defeats the purpose for the initial problem. I still need
>> to look into the code for hte portmapper, but is it easily possible that
>> the portmapper would accept nfsd requests from "insecure" ports also?
>> Since e are (mostly) in a controlled environment that should not pose a
>> problem.
>>
>> Anyone with an idea?
>
> The immediate problem seems like a kernel bug to me--it seems to me that
> the calls to local daemons should be ignoring {min_,max}_resvport. (Or
> is there some way the daemons can still know that those calls come from
> the local kernel?)

I tend to agree. The rpcbind client (at least) does specifically
require a privileged port, so a large min/max port range would be out
of the question for those rpc_clients.

--
Chuck Lever