With the commit of the statd patches over the weekend, we're now
positioned to be able to ship IPv6-enabled nfs-utils in distros. There
is a potential snag though...
Consider this situation:
Admin has a Linux server set up. Server has both IPv4 and IPv6 addrs.
Both addresses are in DNS.
Without an IPv6-enabled nfs-utils, he mounts via IPv4 and all works
fine. Now with an IPv6 enabled nfs-utils, mount.nfs prefers the IPv6
addr and the mount fails (or hangs for a long time and then fails, if
it's using NFSv4)...
While I don't really like it, I think we may need to consider making
mount.nfs prefer IPv4 addrs when it can resolve a hostname to both v4
and v6. Otherwise, we run the risk of breaking an awful lot of working
setups...
Thoughts?
--
Jeff Layton <[email protected]>
On Mon, 18 Jan 2010 16:28:39 -0500
Chuck Lever <[email protected]> wrote:
>
> On Jan 18, 2010, at 2:47 PM, Jeff Layton wrote:
>
> > With the commit of the statd patches over the weekend, we're now
> > positioned to be able to ship IPv6-enabled nfs-utils in distros. There
> > is a potential snag though...
> >
> > Consider this situation:
> >
> > Admin has a Linux server set up. Server has both IPv4 and IPv6 addrs.
> > Both addresses are in DNS.
> >
> > Without an IPv6-enabled nfs-utils, he mounts via IPv4 and all works
> > fine. Now with an IPv6 enabled nfs-utils, mount.nfs prefers the IPv6
> > addr and the mount fails (or hangs for a long time and then fails, if
> > it's using NFSv4)...
>
> Why should it fail?
>
Because Linux NFS servers that work over IPv6 pretty much don't exist
yet.
> > While I don't really like it, I think we may need to consider making
> > mount.nfs prefer IPv4 addrs when it can resolve a hostname to both v4
> > and v6. Otherwise, we run the risk of breaking an awful lot of working
> > setups...
>
> Isn't that what "proto=udp" vs. "proto=udp6" is for?
>
Yes. But that assumes that the admin will know to use that. As it
stands now, they'll have to do a bit of investigation to figure out why
the mount failed and then figure out how to fix it.
Obviously, that's less ideal than a solution that leaves setups working
that are working today. I don't think we ought to put the onus on users
to add extra mount options to stay working.
--
Jeff Layton <[email protected]>
On Mon, 18 Jan 2010 16:23:58 -0500
Trond Myklebust <[email protected]> wrote:
> On Mon, 2010-01-18 at 15:46 -0500, Jeff Layton wrote:
> > On Mon, 18 Jan 2010 15:27:30 -0500
> > Trond Myklebust <[email protected]> wrote:
> >
> > > On Mon, 2010-01-18 at 15:20 -0500, Jeff Layton wrote:
> > > > On Mon, 18 Jan 2010 14:47:46 -0500
> > > > Jeff Layton <[email protected]> wrote:
> > > >
> > > > > With the commit of the statd patches over the weekend, we're now
> > > > > positioned to be able to ship IPv6-enabled nfs-utils in distros. There
> > > > > is a potential snag though...
> > > > >
> > > > > Consider this situation:
> > > > >
> > > > > Admin has a Linux server set up. Server has both IPv4 and IPv6 addrs.
> > > > > Both addresses are in DNS.
> > > > >
> > > > > Without an IPv6-enabled nfs-utils, he mounts via IPv4 and all works
> > > > > fine. Now with an IPv6 enabled nfs-utils, mount.nfs prefers the IPv6
> > > > > addr and the mount fails (or hangs for a long time and then fails, if
> > > > > it's using NFSv4)...
> > > > >
> > > > > While I don't really like it, I think we may need to consider making
> > > > > mount.nfs prefer IPv4 addrs when it can resolve a hostname to both v4
> > > > > and v6. Otherwise, we run the risk of breaking an awful lot of working
> > > > > setups...
> > > > >
> > > >
> > > > Apologies for the self reply and non-descript initial title.
> > > >
> > > > For discussion purposes, here's a patch to do what I'm thinking of. It
> > > > seems to work and do the right thing. You can still force a mount over
> > > > ipv6 by using the right proto/mountproto options.
> > >
> > > How do other applications deal with this problem? Surely, not every
> > > application out there is having to filter away IPv6 addresses?
> > >
> > > I thought that glibc was by default set up to prefer IPv4 addresses, and
> > > allows you to further configure that using /etc/gai.conf. Why is this
> > > failing?
> > >
> > > Cheers
> > > Trond
> > >
> >
> > At least on my test box (which has no gai.conf set up), IPv6 addrs seem
> > to come first in the list returned by getaddrinfo, and I think that's
> > how it'll turn out for most people. There's a bit of discussion on this
> > in Ulrich's writeup here:
> >
> > http://people.redhat.com/drepper/linux-rfc3484.html
> >
> > (see "The BIG Problem")
> >
> > Though that's sort of orthogonal to the issue. I'm not worried about
> > unreachable IPv6 addresses. The problem is that there is a large
> > installed base of servers that do not support serving NFS over IPv6.
> > Many of those may have IPv6 connectivity set up for other reasons. We
> > don't want clients to start failing to mount them when they get an
> > ipv6-enabled nfs-utils.
>
> We already have set AI_ADDRCONFIG, so unreachable addresses should not
> be the problem here.
>
> We do have a 'family' parameter to allow the caller to specify what kind
> of address to request. So my first point would be that your sorting only
> makes sense if the requested family is AF_UNSPEC.
>
True. In the event that it's AF_INET, the sorting will stop at the first
entry. In the event it's AF_INET6, it'll walk the whole list and end up
using the first entry. We could optimize the sorting out in the non
AF_UNSPEC case. I don't think it creates a lot of overhead though.
> Secondly, in the future, I can quite see that some servers may start to
> prefer IPv6. They may even decide to turn off or to block the NFS IPv4
> port. What happens then if the mount program is enforcing a preference
> for IPv4?
>
That's a good point. I tend to think that problem will be less
widespread than this issue, but it's something to consider. Those
people can use proto=tcp6 or equivalent to force IPv6. I think it's
reasonable to impose the use of an extra mount option for people in the
situation you describe. They will be changing their configuration in
such a way that they should expect to have to deal with it.
My main interest here is to avoid a bunch of angry admins who had a
setup that worked before, upgraded nfs-utils to one that supports IPv6
and then ended up with a broken environment. That situation is a bit
different in that those people aren't expecting to be surprised. If we
can achieve that in another way, I'm definitely open to suggestions.
--
Jeff Layton <[email protected]>
On Mon, 2010-01-18 at 15:20 -0500, Jeff Layton wrote:
> On Mon, 18 Jan 2010 14:47:46 -0500
> Jeff Layton <[email protected]> wrote:
>
> > With the commit of the statd patches over the weekend, we're now
> > positioned to be able to ship IPv6-enabled nfs-utils in distros. There
> > is a potential snag though...
> >
> > Consider this situation:
> >
> > Admin has a Linux server set up. Server has both IPv4 and IPv6 addrs.
> > Both addresses are in DNS.
> >
> > Without an IPv6-enabled nfs-utils, he mounts via IPv4 and all works
> > fine. Now with an IPv6 enabled nfs-utils, mount.nfs prefers the IPv6
> > addr and the mount fails (or hangs for a long time and then fails, if
> > it's using NFSv4)...
> >
> > While I don't really like it, I think we may need to consider making
> > mount.nfs prefer IPv4 addrs when it can resolve a hostname to both v4
> > and v6. Otherwise, we run the risk of breaking an awful lot of working
> > setups...
> >
>
> Apologies for the self reply and non-descript initial title.
>
> For discussion purposes, here's a patch to do what I'm thinking of. It
> seems to work and do the right thing. You can still force a mount over
> ipv6 by using the right proto/mountproto options.
How do other applications deal with this problem? Surely, not every
application out there is having to filter away IPv6 addresses?
I thought that glibc was by default set up to prefer IPv4 addresses, and
allows you to further configure that using /etc/gai.conf. Why is this
failing?
Cheers
Trond
On Mon, 18 Jan 2010 18:28:02 -0500
Chuck Lever <[email protected]> wrote:
>
> On Jan 18, 2010, at 5:33 PM, Jeff Layton wrote:
>
> > On Mon, 18 Jan 2010 16:28:39 -0500
> > Chuck Lever <[email protected]> wrote:
> >
> >>
> >> On Jan 18, 2010, at 2:47 PM, Jeff Layton wrote:
> >>
> >>> With the commit of the statd patches over the weekend, we're now
> >>> positioned to be able to ship IPv6-enabled nfs-utils in distros.
> >>> There
> >>> is a potential snag though...
> >>>
> >>> Consider this situation:
> >>>
> >>> Admin has a Linux server set up. Server has both IPv4 and IPv6
> >>> addrs.
> >>> Both addresses are in DNS.
> >>>
> >>> Without an IPv6-enabled nfs-utils, he mounts via IPv4 and all works
> >>> fine. Now with an IPv6 enabled nfs-utils, mount.nfs prefers the IPv6
> >>> addr and the mount fails (or hangs for a long time and then fails,
> >>> if
> >>> it's using NFSv4)...
> >>
> >> Why should it fail?
> >
> > Because Linux NFS servers that work over IPv6 pretty much don't exist
> > yet.
>
> More specifically, mount version and transport negotation is not
> working in this case. Otherwise, specifying "proto=" would be an
> adequate solution.
>
> >>> While I don't really like it, I think we may need to consider making
> >>> mount.nfs prefer IPv4 addrs when it can resolve a hostname to both
> >>> v4
> >>> and v6. Otherwise, we run the risk of breaking an awful lot of
> >>> working
> >>> setups...
> >>
> >> Isn't that what "proto=udp" vs. "proto=udp6" is for?
> >
> > Yes. But that assumes that the admin will know to use that. As it
> > stands now, they'll have to do a bit of investigation to figure out
> > why
> > the mount failed and then figure out how to fix it.
> >
> > Obviously, that's less ideal than a solution that leaves setups
> > working
> > that are working today. I don't think we ought to put the onus on
> > users
> > to add extra mount options to stay working.
>
> Notice that the server already tells you what it can and can't
> support. If the server doesn't support rpcbind v3 or v4, you can't
> use IPv6. If it does, it can report to the client that "udp6" and
> "tcp6" are not registered. Really, then, what we want is to teach
> mount's version and transport negotiation to negotiate the protocol
> family as well as the transport and NFS version.
>
Maybe I'm not understanding the terminology correctly. Here's sort of
the "problem" as I see it:
mount.nfs decides the address of the server very early on. Once it
decides that address, it doesn't ever go back and try another. In the
absence of a proto= option, you get whatever address family happens to
be first in the list.
So, even if you can do a rpcbind query to determine what the server
supports (which isn't a given with NFSv4), by the time you find out
that it doesn't support NFS over IPv6 it's too late to do anything but
fail.
This problem is compounded by the fact that we now attempt NFSv4 by
default. In that case we can't really rely on being able to talk to
rpcbind and determine this quickly. It also takes a while for the
kernel's mount attempt to time out, so it's not really reasonable to
try multiple addresses in succession.
> But for now, it seems like the only case we are worried about here is
> when the "proto/mountproto" options aren't specified on a text-based
> mount. Always choosing an IPv4 address in this case is likely a good
> compromise. nfs_validate_options() might be a better spot to sort all
> this out than nfs_lookup(), though.
>
> It should also be pretty easy to parametrize this behavior in the
> config file.
>
Yes, the absence of a "proto=" option is the main worry.
We could put this in nfs_validate_options. It'll potentially mean 2
getaddrinfo calls for an IPv6 server, but it's doable if you think
that'll be cleaner.
By parameterize, you mean adding something like a "afpref=" option?
Something like:
afpref=inet|inet6
If so, that should be doable too. I'm reticent to add new options if
we can help it, but am willing to go along with that if you and others
think it's the right solution.
--
Jeff Layton <[email protected]>
On Mon, 18 Jan 2010 14:47:46 -0500
Jeff Layton <[email protected]> wrote:
> With the commit of the statd patches over the weekend, we're now
> positioned to be able to ship IPv6-enabled nfs-utils in distros. There
> is a potential snag though...
>
> Consider this situation:
>
> Admin has a Linux server set up. Server has both IPv4 and IPv6 addrs.
> Both addresses are in DNS.
>
> Without an IPv6-enabled nfs-utils, he mounts via IPv4 and all works
> fine. Now with an IPv6 enabled nfs-utils, mount.nfs prefers the IPv6
> addr and the mount fails (or hangs for a long time and then fails, if
> it's using NFSv4)...
>
> While I don't really like it, I think we may need to consider making
> mount.nfs prefer IPv4 addrs when it can resolve a hostname to both v4
> and v6. Otherwise, we run the risk of breaking an awful lot of working
> setups...
>
Apologies for the self reply and non-descript initial title.
For discussion purposes, here's a patch to do what I'm thinking of. It
seems to work and do the right thing. You can still force a mount over
ipv6 by using the right proto/mountproto options.
Comments welcome...
---------------------[snip]----------------------
mount.nfs: prefer IPv4 addresses over IPv6
We're poised to enable IPv6 in nfs-utils in distros. There is a potential
problem however. glibc seems to prefer IPv6 addrs.
If someone has a working IPv4 server today that has an IPv6 address,
then clients may start trying to mount over that address. If the server
doesn't support NFS serving over IPv6 (and virtually no linux servers
currently do), then the mount will start failing.
Avoid this problem by making the mount.nfs code prefer IPv4 addresses
when they are available.
Signed-off-by: Jeff Layton <[email protected]>
---
utils/mount/network.c | 27 ++++++++++++++++++++++-----
1 files changed, 22 insertions(+), 5 deletions(-)
diff --git a/utils/mount/network.c b/utils/mount/network.c
index 92bba2d..6723f8f 100644
--- a/utils/mount/network.c
+++ b/utils/mount/network.c
@@ -203,7 +203,7 @@ static const unsigned int *nfs_default_proto()
int nfs_lookup(const char *hostname, const sa_family_t family,
struct sockaddr *sap, socklen_t *salen)
{
- struct addrinfo *gai_results;
+ struct addrinfo *gai_results, *gai_pref;
struct addrinfo gai_hint = {
#ifdef HAVE_DECL_AI_ADDRCONFIG
.ai_flags = AI_ADDRCONFIG,
@@ -229,12 +229,29 @@ int nfs_lookup(const char *hostname, const sa_family_t family,
return ret;
}
- switch (gai_results->ai_addr->sa_family) {
+ /*
+ * It will probably be quite a while before we have enough IPv6
+ * capable servers to be able to prefer using IPv6. For now, we
+ * only use IPv6 when there is no IPv4 address available in the
+ * results.
+ */
+ gai_pref = gai_results;
+ while (gai_pref) {
+ if (gai_pref->ai_addr->sa_family == AF_INET)
+ break;
+ gai_pref = gai_pref->ai_next;
+ }
+
+ /* no IPv4 addr found, just use first on the list */
+ if (gai_pref == NULL)
+ gai_pref = gai_results;
+
+ switch (gai_pref->ai_addr->sa_family) {
case AF_INET:
case AF_INET6:
- if (len >= gai_results->ai_addrlen) {
- *salen = gai_results->ai_addrlen;
- memcpy(sap, gai_results->ai_addr, *salen);
+ if (len >= gai_pref->ai_addrlen) {
+ *salen = gai_pref->ai_addrlen;
+ memcpy(sap, gai_pref->ai_addr, *salen);
ret = 1;
}
break;
--
1.6.5.2
On Jan 18, 2010, at 5:33 PM, Jeff Layton wrote:
> On Mon, 18 Jan 2010 16:28:39 -0500
> Chuck Lever <[email protected]> wrote:
>
>>
>> On Jan 18, 2010, at 2:47 PM, Jeff Layton wrote:
>>
>>> With the commit of the statd patches over the weekend, we're now
>>> positioned to be able to ship IPv6-enabled nfs-utils in distros.
>>> There
>>> is a potential snag though...
>>>
>>> Consider this situation:
>>>
>>> Admin has a Linux server set up. Server has both IPv4 and IPv6
>>> addrs.
>>> Both addresses are in DNS.
>>>
>>> Without an IPv6-enabled nfs-utils, he mounts via IPv4 and all works
>>> fine. Now with an IPv6 enabled nfs-utils, mount.nfs prefers the IPv6
>>> addr and the mount fails (or hangs for a long time and then fails,
>>> if
>>> it's using NFSv4)...
>>
>> Why should it fail?
>
> Because Linux NFS servers that work over IPv6 pretty much don't exist
> yet.
More specifically, mount version and transport negotation is not
working in this case. Otherwise, specifying "proto=" would be an
adequate solution.
>>> While I don't really like it, I think we may need to consider making
>>> mount.nfs prefer IPv4 addrs when it can resolve a hostname to both
>>> v4
>>> and v6. Otherwise, we run the risk of breaking an awful lot of
>>> working
>>> setups...
>>
>> Isn't that what "proto=udp" vs. "proto=udp6" is for?
>
> Yes. But that assumes that the admin will know to use that. As it
> stands now, they'll have to do a bit of investigation to figure out
> why
> the mount failed and then figure out how to fix it.
>
> Obviously, that's less ideal than a solution that leaves setups
> working
> that are working today. I don't think we ought to put the onus on
> users
> to add extra mount options to stay working.
Notice that the server already tells you what it can and can't
support. If the server doesn't support rpcbind v3 or v4, you can't
use IPv6. If it does, it can report to the client that "udp6" and
"tcp6" are not registered. Really, then, what we want is to teach
mount's version and transport negotiation to negotiate the protocol
family as well as the transport and NFS version.
But for now, it seems like the only case we are worried about here is
when the "proto/mountproto" options aren't specified on a text-based
mount. Always choosing an IPv4 address in this case is likely a good
compromise. nfs_validate_options() might be a better spot to sort all
this out than nfs_lookup(), though.
It should also be pretty easy to parametrize this behavior in the
config file.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
On Mon, 2010-01-18 at 15:46 -0500, Jeff Layton wrote:
> On Mon, 18 Jan 2010 15:27:30 -0500
> Trond Myklebust <[email protected]> wrote:
>
> > On Mon, 2010-01-18 at 15:20 -0500, Jeff Layton wrote:
> > > On Mon, 18 Jan 2010 14:47:46 -0500
> > > Jeff Layton <[email protected]> wrote:
> > >
> > > > With the commit of the statd patches over the weekend, we're now
> > > > positioned to be able to ship IPv6-enabled nfs-utils in distros. There
> > > > is a potential snag though...
> > > >
> > > > Consider this situation:
> > > >
> > > > Admin has a Linux server set up. Server has both IPv4 and IPv6 addrs.
> > > > Both addresses are in DNS.
> > > >
> > > > Without an IPv6-enabled nfs-utils, he mounts via IPv4 and all works
> > > > fine. Now with an IPv6 enabled nfs-utils, mount.nfs prefers the IPv6
> > > > addr and the mount fails (or hangs for a long time and then fails, if
> > > > it's using NFSv4)...
> > > >
> > > > While I don't really like it, I think we may need to consider making
> > > > mount.nfs prefer IPv4 addrs when it can resolve a hostname to both v4
> > > > and v6. Otherwise, we run the risk of breaking an awful lot of working
> > > > setups...
> > > >
> > >
> > > Apologies for the self reply and non-descript initial title.
> > >
> > > For discussion purposes, here's a patch to do what I'm thinking of. It
> > > seems to work and do the right thing. You can still force a mount over
> > > ipv6 by using the right proto/mountproto options.
> >
> > How do other applications deal with this problem? Surely, not every
> > application out there is having to filter away IPv6 addresses?
> >
> > I thought that glibc was by default set up to prefer IPv4 addresses, and
> > allows you to further configure that using /etc/gai.conf. Why is this
> > failing?
> >
> > Cheers
> > Trond
> >
>
> At least on my test box (which has no gai.conf set up), IPv6 addrs seem
> to come first in the list returned by getaddrinfo, and I think that's
> how it'll turn out for most people. There's a bit of discussion on this
> in Ulrich's writeup here:
>
> http://people.redhat.com/drepper/linux-rfc3484.html
>
> (see "The BIG Problem")
>
> Though that's sort of orthogonal to the issue. I'm not worried about
> unreachable IPv6 addresses. The problem is that there is a large
> installed base of servers that do not support serving NFS over IPv6.
> Many of those may have IPv6 connectivity set up for other reasons. We
> don't want clients to start failing to mount them when they get an
> ipv6-enabled nfs-utils.
We already have set AI_ADDRCONFIG, so unreachable addresses should not
be the problem here.
We do have a 'family' parameter to allow the caller to specify what kind
of address to request. So my first point would be that your sorting only
makes sense if the requested family is AF_UNSPEC.
Secondly, in the future, I can quite see that some servers may start to
prefer IPv6. They may even decide to turn off or to block the NFS IPv4
port. What happens then if the mount program is enforcing a preference
for IPv4?
Trond
On Mon, 18 Jan 2010 15:27:30 -0500
Trond Myklebust <[email protected]> wrote:
> On Mon, 2010-01-18 at 15:20 -0500, Jeff Layton wrote:
> > On Mon, 18 Jan 2010 14:47:46 -0500
> > Jeff Layton <[email protected]> wrote:
> >
> > > With the commit of the statd patches over the weekend, we're now
> > > positioned to be able to ship IPv6-enabled nfs-utils in distros. There
> > > is a potential snag though...
> > >
> > > Consider this situation:
> > >
> > > Admin has a Linux server set up. Server has both IPv4 and IPv6 addrs.
> > > Both addresses are in DNS.
> > >
> > > Without an IPv6-enabled nfs-utils, he mounts via IPv4 and all works
> > > fine. Now with an IPv6 enabled nfs-utils, mount.nfs prefers the IPv6
> > > addr and the mount fails (or hangs for a long time and then fails, if
> > > it's using NFSv4)...
> > >
> > > While I don't really like it, I think we may need to consider making
> > > mount.nfs prefer IPv4 addrs when it can resolve a hostname to both v4
> > > and v6. Otherwise, we run the risk of breaking an awful lot of working
> > > setups...
> > >
> >
> > Apologies for the self reply and non-descript initial title.
> >
> > For discussion purposes, here's a patch to do what I'm thinking of. It
> > seems to work and do the right thing. You can still force a mount over
> > ipv6 by using the right proto/mountproto options.
>
> How do other applications deal with this problem? Surely, not every
> application out there is having to filter away IPv6 addresses?
>
> I thought that glibc was by default set up to prefer IPv4 addresses, and
> allows you to further configure that using /etc/gai.conf. Why is this
> failing?
>
> Cheers
> Trond
>
At least on my test box (which has no gai.conf set up), IPv6 addrs seem
to come first in the list returned by getaddrinfo, and I think that's
how it'll turn out for most people. There's a bit of discussion on this
in Ulrich's writeup here:
http://people.redhat.com/drepper/linux-rfc3484.html
(see "The BIG Problem")
Though that's sort of orthogonal to the issue. I'm not worried about
unreachable IPv6 addresses. The problem is that there is a large
installed base of servers that do not support serving NFS over IPv6.
Many of those may have IPv6 connectivity set up for other reasons. We
don't want clients to start failing to mount them when they get an
ipv6-enabled nfs-utils.
--
Jeff Layton <[email protected]>
On Jan 18, 2010, at 2:47 PM, Jeff Layton wrote:
> With the commit of the statd patches over the weekend, we're now
> positioned to be able to ship IPv6-enabled nfs-utils in distros. There
> is a potential snag though...
>
> Consider this situation:
>
> Admin has a Linux server set up. Server has both IPv4 and IPv6 addrs.
> Both addresses are in DNS.
>
> Without an IPv6-enabled nfs-utils, he mounts via IPv4 and all works
> fine. Now with an IPv6 enabled nfs-utils, mount.nfs prefers the IPv6
> addr and the mount fails (or hangs for a long time and then fails, if
> it's using NFSv4)...
Why should it fail?
> While I don't really like it, I think we may need to consider making
> mount.nfs prefer IPv4 addrs when it can resolve a hostname to both v4
> and v6. Otherwise, we run the risk of breaking an awful lot of working
> setups...
Isn't that what "proto=udp" vs. "proto=udp6" is for?
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com