Good evening!
How can I get showmount -e to use a non-2049 TCP port number to show
mounts on a NFSv4 server?
/sbin/showmount -e localhost@30000
clnt_create: RPC: Unknown host
Ced
--
Cedric Blancher <[email protected]>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur
Good evening!
Generic issue in all of nfs-utils: None of the utils properly support
NFSv4 with non.standard (TCP/2049) port numbers.
mount supports it for mounting, but does not show it for listing mounts
/proc/mounts does not show the port number either
showmount -e does not support a port number
autofs does not support non-2049 port numbers
nfsd referrals do not support setting non-2049 port numbers
...
Could you please make a concentrated effort and allow non-2049 port
numbers for NFSv4 mounts, in all of the lifecycle of a NFSv4 mount?
From nfsd, nfsd referrals, client mount/umount, autofs
mount/umount+LDAP spec
Ced
---------- Forwarded message ---------
From: Cedric Blancher <[email protected]>
Date: Sun, 7 Jan 2024 at 22:32
Subject: showmount -e with custom port number?
To: Linux NFS Mailing List <[email protected]>
Good evening!
How can I get showmount -e to use a non-2049 TCP port number to show
mounts on a NFSv4 server?
/sbin/showmount -e localhost@30000
clnt_create: RPC: Unknown host
Ced
--
Cedric Blancher <[email protected]>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur
--
Cedric Blancher <[email protected]>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur
On 8/1/24 06:33, Cedric Blancher wrote:
> Good evening!
>
> Generic issue in all of nfs-utils: None of the utils properly support
> NFSv4 with non.standard (TCP/2049) port numbers.
>
> mount supports it for mounting, but does not show it for listing mounts
> /proc/mounts does not show the port number either
I'll leave this to the nfs folks but, looking at the nfs kernel code,
neither of the above appear to be accurate. But then you haven't provided
information about what kernel and utility versions your report is based on
so I can't check.
> showmount -e does not support a port number
Now this does look to be a problem, even with the current code, and the
mountd port can't be specified at all.
> autofs does not support non-2049 port numbers
What is it you expect to be able to do with autofs that you can't do?
It's interesting that you sound like you assume mountd "must" not be
used for NFSv4 and it's not the first time I've heard that.
I thought NFSv4 "must" be able to be used without mountd but there was
nothing that says mountd "must" not be used to provide port information
to clients if they ask, such as on a local (probably isolated) network.
So, based on that assumption, the fact that it (the kernel => mount(8),
proc mount listings) doesn't print the value of the port option if the
default NFS server port is being used seems sensible enough.
I'm not aware of any protocol in NFS itself for communicating the list
of exports. And if your not on an isolated network would being able to
list the exports be a wise idea?
So I'm not sure what your expecting to be done ... other than some
improvements to showmount(8).
> nfsd referrals do not support setting non-2049 port numbers
> ...
Another interesting comment, I don't actually know about NFS referrals
these days. I'd be interested in hearing about them.
Ian
>
> Could you please make a concentrated effort and allow non-2049 port
> numbers for NFSv4 mounts, in all of the lifecycle of a NFSv4 mount?
> From nfsd, nfsd referrals, client mount/umount, autofs
> mount/umount+LDAP spec
>
> Ced
>
> ---------- Forwarded message ---------
> From: Cedric Blancher <[email protected]>
> Date: Sun, 7 Jan 2024 at 22:32
> Subject: showmount -e with custom port number?
> To: Linux NFS Mailing List <[email protected]>
>
>
> Good evening!
>
> How can I get showmount -e to use a non-2049 TCP port number to show
> mounts on a NFSv4 server?
>
> /sbin/showmount -e localhost@30000
> clnt_create: RPC: Unknown host
>
> Ced
> --
> Cedric Blancher <[email protected]>
> [https://plus.google.com/u/0/+CedricBlancher/]
> Institute Pasteur
>
>
On Sun, Jan 07, 2024 at 11:33:31PM +0100, Cedric Blancher wrote:
> Could you please make a concentrated effort and allow non-2049 port
> numbers for NFSv4 mounts, in all of the lifecycle of a NFSv4 mount?
> From nfsd, nfsd referrals, client mount/umount, autofs
> mount/umount+LDAP spec
One reason we have not pursued stack-wide NFSv4 support for
alternate ports is that they are not firewall-friendly. A major
design point of NFSv4 (and NFSv4.1, with its backchannel) is that
it is supposed to be more firewall-friendly than NFSv3, its
auxiliary protocols, and its requirement to deploy rpcbind.
Also, these days it is relatively easy in Linux to deploy multiple
NFS services on a single physical host by using containers (or just
separate network namespaces). So instead of alternate ports, each
NFS service is on port 2049, but it has its own IP address. That
kind of deployment is supposed to be fully supported with NFSD today.
Commercial NFS server implementations also typically make it easy
to add distinct NFSv4 services at unique IP addresses but all on
port 2049.
--
Chuck Lever
On Sun, 2024-01-07 at 23:33 +0100, Cedric Blancher wrote:
> Good evening!
>
> Generic issue in all of nfs-utils: None of the utils properly support
> NFSv4 with non.standard (TCP/2049) port numbers.
>
> mount supports it for mounting, but does not show it for listing mounts
> /proc/mounts does not show the port number either
mount just gets the mount info from /proc/mounts, which absolutely does
show a port= option when you use a different port than 2049.
> showmount -e does not support a port number
showmount talks directly to mountd so you wouldn't usually point it at
the same port nfs is listening on. As with most sunrpc based tools, it
queries rpcbind in order to determine what port to talk to. If you're
going to use stuff like showmount, then you need to ensure that rpcbind
is reachable and that the daemons register with it properly.
> autofs does not support non-2049 port numbers
> nfsd referrals do not support setting non-2049 port numbers
The same goes for autofs. It just calls /bin/mount. I think (but am not
sure) that even with NFSv4, when the client can't contact the server on
port 2049, it'll try to use rpcbind to determine the port.
Alternately, if you don't want to deal with rpcbind for mounting, you
could also specify a hardcoded "port=" mount option in your autofs maps.
> ...
>
> Could you please make a concentrated effort and allow non-2049 port
> numbers for NFSv4 mounts, in all of the lifecycle of a NFSv4 mount?
> From nfsd, nfsd referrals, client mount/umount, autofs
> mount/umount+LDAP spec
>
> Ced
>
> ---------- Forwarded message ---------
> From: Cedric Blancher <[email protected]>
> Date: Sun, 7 Jan 2024 at 22:32
> Subject: showmount -e with custom port number?
> To: Linux NFS Mailing List <[email protected]>
>
>
> Good evening!
>
> How can I get showmount -e to use a non-2049 TCP port number to show
> mounts on a NFSv4 server?
>
> /sbin/showmount -e localhost@30000
> clnt_create: RPC: Unknown host
--
Jeff Layton <[email protected]>
On 8/1/24 22:52, Jeff Layton wrote:
> On Sun, 2024-01-07 at 23:33 +0100, Cedric Blancher wrote:
>> Good evening!
>>
>> Generic issue in all of nfs-utils: None of the utils properly support
>> NFSv4 with non.standard (TCP/2049) port numbers.
>>
>> mount supports it for mounting, but does not show it for listing mounts
>> /proc/mounts does not show the port number either
> mount just gets the mount info from /proc/mounts, which absolutely does
> show a port= option when you use a different port than 2049.
>
>> showmount -e does not support a port number
> showmount talks directly to mountd so you wouldn't usually point it at
> the same port nfs is listening on. As with most sunrpc based tools, it
> queries rpcbind in order to determine what port to talk to. If you're
> going to use stuff like showmount, then you need to ensure that rpcbind
> is reachable and that the daemons register with it properly.
>
>> autofs does not support non-2049 port numbers
>> nfsd referrals do not support setting non-2049 port numbers
> The same goes for autofs. It just calls /bin/mount. I think (but am not
> sure) that even with NFSv4, when the client can't contact the server on
> port 2049, it'll try to use rpcbind to determine the port.
That is exactly what autofs does, and always has done.
The other thing to be aware of is if you want NFSv4 only you need to tell
autofs that by using the option fstype=nfs4 so that it doesn't try to fall
back to earlier NFS versions and also tries to avoid contacting rpcbind.
Ian
>
> Alternately, if you don't want to deal with rpcbind for mounting, you
> could also specify a hardcoded "port=" mount option in your autofs maps.
>
>
>> ...
>>
>> Could you please make a concentrated effort and allow non-2049 port
>> numbers for NFSv4 mounts, in all of the lifecycle of a NFSv4 mount?
>> From nfsd, nfsd referrals, client mount/umount, autofs
>> mount/umount+LDAP spec
>>
>> Ced
>>
>> ---------- Forwarded message ---------
>> From: Cedric Blancher <[email protected]>
>> Date: Sun, 7 Jan 2024 at 22:32
>> Subject: showmount -e with custom port number?
>> To: Linux NFS Mailing List <[email protected]>
>>
>>
>> Good evening!
>>
>> How can I get showmount -e to use a non-2049 TCP port number to show
>> mounts on a NFSv4 server?
>>
>> /sbin/showmount -e localhost@30000
>> clnt_create: RPC: Unknown host
On Mon, 8 Jan 2024 at 15:39, Chuck Lever <[email protected]> wrote:
>
> On Sun, Jan 07, 2024 at 11:33:31PM +0100, Cedric Blancher wrote:
> > Could you please make a concentrated effort and allow non-2049 port
> > numbers for NFSv4 mounts, in all of the lifecycle of a NFSv4 mount?
> > From nfsd, nfsd referrals, client mount/umount, autofs
> > mount/umount+LDAP spec
>
> One reason we have not pursued stack-wide NFSv4 support for
> alternate ports is that they are not firewall-friendly. A major
> design point of NFSv4 (and NFSv4.1, with its backchannel) is that
> it is supposed to be more firewall-friendly than NFSv3, its
> auxiliary protocols, and its requirement to deploy rpcbind.
Who came up with THAT argument?!
NFSv4 was designed around the concept of ONE TCP port for everything
(fs, mount, lock daemons, ...), so multiple servers per IPv4 address
can become a reality, but not specifically 2049 - with the clear
assumption that using port 2049 might not be feasible for all
organisations.
If you look at Solaris BUGSTER (remember, we were a big SUN customer
in the 1990/2000, so we had lots of bugs open for this mess), you'll
find lots of reasons why one single port for NFS is not feasible in
all scenarios.
Just some examples, but certainly not limited to:
- Fine-grained HSM, all on one host
- Fine-grained project/resource management, i.e. one nfs server per
project, all on one host
- Competing teams
- Hostile IT department (e.g. port 2049 blocked out of FEAR - not
reason, no further discussion/negotiation possible)
- NFSv4 tunneled via ssh
- NAT, e.g. private IPv4 address range inside, only one IPv4 address outside
- IPv4 address shortage
- Software test deployments in parallel to the production systems, on
the same machine
- ...
In any of these scenarios you'll end up with NFSv4 certainly not using
TCP port 2049.
>
> Also, these days it is relatively easy in Linux to deploy multiple
> NFS services on a single physical host by using containers (or just
> separate network namespaces).
That would assume people want to waste resources on the container
madness. OK, I better stop here, before I start counting how many MB
are wasted on the container/VM cr*p.
> So instead of alternate ports, each
> NFS service is on port 2049, but it has its own IP address.
Not feasible with NAT, or IPv4 address shortage.
> That
> kind of deployment is supposed to be fully supported with NFSD today.
>
> Commercial NFS server implementations also typically make it easy
> to add distinct NFSv4 services at unique IP addresses but all on
> port 2049.
Unfortunately this is just wishful thinking. Please check how NIH or
CERN do their setups - you'll get a nasty surprise in terms of NFSv4
port==2049.
Just the reality, from our side:
Active, per-project nfs servers: 14030. Only 1717 (less than 10%!!)
use TCP port 2049, all others do their duty in the port range of
10000-16999.
Ced
--
Cedric Blancher <[email protected]>
[https://plus.google.com/u/0/+CedricBlancher/]
Institute Pasteur
> On Jan 10, 2024, at 1:06 AM, Cedric Blancher <[email protected]> wrote:
>
> On Mon, 8 Jan 2024 at 15:39, Chuck Lever <[email protected]> wrote:
>>
>> On Sun, Jan 07, 2024 at 11:33:31PM +0100, Cedric Blancher wrote:
>>> Could you please make a concentrated effort and allow non-2049 port
>>> numbers for NFSv4 mounts, in all of the lifecycle of a NFSv4 mount?
>>> From nfsd, nfsd referrals, client mount/umount, autofs
>>> mount/umount+LDAP spec
>>
>> One reason we have not pursued stack-wide NFSv4 support for
>> alternate ports is that they are not firewall-friendly. A major
>> design point of NFSv4 (and NFSv4.1, with its backchannel) is that
>> it is supposed to be more firewall-friendly than NFSv3, its
>> auxiliary protocols, and its requirement to deploy rpcbind.
>
> Who came up with THAT argument?!
The IETF's NFSv4 standards group.
NLM, NSM, MNT, NFSACL, and NFS were made a single protocol with
NFSv4.0. One reason for this design was the necessity to add
OPEN/CLOSE operations to make file locking more reliable. But
another was to reduce the number of open ports needed, and to
eliminate the need to run rpcbind on NFS servers.
With NFSv4.1, callback operation was also combined into the
main protocol by making servers do callbacks on the same
transport connection as forechannel operation. This is precisely
because callbacks could not transition NAT boxes and firewalls
in a convenient fashion.
You can find a generic mention of it here:
https://www.rfc-editor.org/rfc/rfc8881#name-nfsv4-goals
Though that is not as detailed a discussion as I might have
hoped. The overall goal mentioned here is good operation on
the open internet and the ability to transition firewalls
easily.
> NFSv4 was designed around the concept of ONE TCP port for everything
> (fs, mount, lock daemons, ...), so multiple servers per IPv4 address
> can become a reality, but not specifically 2049 - with the clear
> assumption that using port 2049 might not be feasible for all
> organisations.
I've never heard that, and I've been active in the NFS standards
group for many years, and have worked closely with the authors
of these standards and the folks who created the implementation
of NFSv4 on several platforms.
So if that truly was a goal of the single-port design, it has
not been mentioned recently, and there are certain protocol gaps
(like no explicit port field in the fs_locations attribute)
that make me doubt your assertion that alternate ports was a
primary protocol design goal.
But this is only a sidebar.
> If you look at Solaris BUGSTER (remember, we were a big SUN customer
> in the 1990/2000, so we had lots of bugs open for this mess), you'll
> find lots of reasons why one single port for NFS is not feasible in
> all scenarios.
> Just some examples, but certainly not limited to:
> - Fine-grained HSM, all on one host
> - Fine-grained project/resource management, i.e. one nfs server per
> project, all on one host
> - Competing teams
> - Hostile IT department (e.g. port 2049 blocked out of FEAR - not
> reason, no further discussion/negotiation possible)
> - NFSv4 tunneled via ssh
> - NAT, e.g. private IPv4 address range inside, only one IPv4 address outside
> - IPv4 address shortage
> - Software test deployments in parallel to the production systems, on
> the same machine
> - ...
>
> In any of these scenarios you'll end up with NFSv4 certainly not using
> TCP port 2049.
In most of these cases, the use of alternate ports has been
superceded in the past 20 years. I'm not saying there are no
legitimate uses for alternate ports, only that they seem to
be increasingly a corner case, and that's the way the Linux
NFS community has prioritized them. (And incidentally, they
do mostly work today in Linux. There are just a few missing
spots).
Look, you are really not hearing me, clearly. No-one has said
no we won't. We have said: this is harder than you think, and
here is why. We have said: if you want to do this sooner, here
is what we, as the stewards of this code, will accept. We do
have a plan for this, after all. Just not enough time to do it.
This is open source. If you have an itch, you can scratch it
yourself. You do not demand that others implement what you
want, for free.
Where do you think the resources for developing new features
comes from?
Please stop arguing with strawmen and write some code.
--
Chuck Lever
On Wed, Jan 10, 2024 at 4:11 PM Chuck Lever III <[email protected]> wrote:
>
>
>
> > On Jan 10, 2024, at 1:06 AM, Cedric Blancher <[email protected]> wrote:
> >
> > On Mon, 8 Jan 2024 at 15:39, Chuck Lever <[email protected]> wrote:
> > If you look at Solaris BUGSTER (remember, we were a big SUN customer
> > in the 1990/2000, so we had lots of bugs open for this mess), you'll
> > find lots of reasons why one single port for NFS is not feasible in
> > all scenarios.
>
> > Just some examples, but certainly not limited to:
> > - Fine-grained HSM, all on one host
> > - Fine-grained project/resource management, i.e. one nfs server per
> > project, all on one host
> > - Competing teams
> > - Hostile IT department (e.g. port 2049 blocked out of FEAR - not
> > reason, no further discussion/negotiation possible)
> > - NFSv4 tunneled via ssh
> > - NAT, e.g. private IPv4 address range inside, only one IPv4 address outside
> > - IPv4 address shortage
> > - Software test deployments in parallel to the production systems, on
> > the same machine
> > - ...
> >
> > In any of these scenarios you'll end up with NFSv4 certainly not using
> > TCP port 2049.
>
> In most of these cases, the use of alternate ports has been
> superceded in the past 20 years.
From a viewpoint of university hosting, HPC environments and pretty
much everything else I've seen, that statement is FAR from reality.
This even gets worse in Germany, Europe and Asia (not US of course,
you're hogging public IPv4 addresses), where we have IPv4 address
shortage, lots of NAT, and only a small amount of IPv6 (except Asia).
In all these scenarios you have NFSv4 connections all over the port
numbers, and not only 2049.
Also, reality is, storage virtualisation for NFSv4 on the outgoing
side is typically done on the port level, and not IP address level,
e.g. many servers behind NAT, and NAT then translates the accesses to
the NFSv4 server into a single IPv4 address with different ports
(because of address shortage). And because of convenience, the NFSv4
servers start with the same port number as used by NAT on the
outside...
Short: Non-2049 port number are the not a "corner case"
Thanks,
Martin
> On Jan 11, 2024, at 10:06 AM, Martin Wege <[email protected]> wrote:
>
> On Wed, Jan 10, 2024 at 4:11 PM Chuck Lever III <[email protected]> wrote:
>>
>>
>> In most of these cases, the use of alternate ports has been
>> superceded in the past 20 years.
>
> From a viewpoint of university hosting, HPC environments and pretty
> much everything else I've seen, that statement is FAR from reality.
> This even gets worse in Germany, Europe and Asia (not US of course,
> you're hogging public IPv4 addresses), where we have IPv4 address
> shortage, lots of NAT, and only a small amount of IPv6 (except Asia).
> In all these scenarios you have NFSv4 connections all over the port
> numbers, and not only 2049.
>
> Also, reality is, storage virtualisation for NFSv4 on the outgoing
> side is typically done on the port level, and not IP address level,
> e.g. many servers behind NAT, and NAT then translates the accesses to
> the NFSv4 server into a single IPv4 address with different ports
> (because of address shortage). And because of convenience, the NFSv4
> servers start with the same port number as used by NAT on the
> outside...
>
> Short: Non-2049 port number are the not a "corner case"
Well that's very nice, but this is the first I've heard of these
requests and use cases. Telling me my lived experience is "far
from reality" is not a good way to get your feature into our code
base. And it's really not on point, unless your only point is to
tell me how wrong I am.
I will repeat, yet again:
a. Linux NFS /does/ support alternate ports, but there are bugs
b. We /do/ have a plan to support alternate ports in NFSv4
referrals
c. We currently don't have the resources to focus on the parts
of that that will be a heavy lift. But we will get to it
eventually.
d. To get your needs prioritized, send patches.
You guys clearly do not know what else is on our plate, nor do
you understand how few there are in the Linux NFS community. We
think that good security, interoperability, and performance are
the top priority, and dealing with bugs and regressions is up
there as well.
If this feature is important to the HPC community, they should
fund a software consultant to implement what you need and work
with us to merge it. That is the way open source works. That
kind of technology transfer has worked for many many years.
Since that has not happened in the past two decades, I think I
am correct to assume that alternate ports are, in reality,
working well enough for that community already.
If you're simply going to demand that we implement your favorite
feature without offering user stories or use cases or patches or
any other context or help, then we will treat you like trolls.
Because honestly, guys, you are acting like trolls.
--
Chuck Lever