Hi,
I've gotten a few bug reports lately that if UDP is blocked at the server,
NFS mounts (even with -o tcp) fail. git bisect shows that the culprit is
the change "Get rid of the "-i" option for mount.nfs[4] and always use the
text-based mount(2) system call for kernel version 2.6.23 and later." -- in
other words, the bug is specific to the text interface. Does anyone know why
this would be the case?
/* Steinar */
--
Homepage: http://www.sesse.net/
Hi Steinar-
On Apr 23, 2008, at 5:49 PM, Steinar H. Gunderson wrote:
> I've gotten a few bug reports lately that if UDP is blocked at the
> server,
> NFS mounts (even with -o tcp) fail. git bisect shows that the
> culprit is
> the change "Get rid of the "-i" option for mount.nfs[4] and always
> use the
> text-based mount(2) system call for kernel version 2.6.23 and
> later." -- in
> other words, the bug is specific to the text interface. Does anyone
> know why
> this would be the case?
I specifically tested that when developing the text-based mount
interface, so if it doesn't work, it's a bug or a missed corner case.
Start with "mount.nfs -v" to see if the mount command tells you
anything interesting. Then do a raw network trace to see what's
failing during the mount version/protocol negotiation.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
On Thu, Apr 24, 2008 at 10:12:58AM -0400, Chuck Lever wrote:
> Start with "mount.nfs -v" to see if the mount command tells you anything
> interesting. Then do a raw network trace to see what's failing during
> the mount version/protocol negotiation.
There's nothing in -v except that it adds an addr= field. As for the
packet dump, that's the most interesting part -- for my part, the server
doesn't have iptables, so I'm blocking on the client. In that case, exactly
zero packets enter the write before mount gives up...
The other submitters have iptables blocking on the server, though, and it
doesn't work for them either (one is using NFSv3, the other NFSv4). Is this
really working for you?
/* Steinar */
--
Homepage: http://www.sesse.net/
On Apr 25, 2008, at 4:01 AM, Steinar H. Gunderson wrote:
> On Thu, Apr 24, 2008 at 10:12:58AM -0400, Chuck Lever wrote:
>> Start with "mount.nfs -v" to see if the mount command tells you
>> anything
>> interesting. Then do a raw network trace to see what's failing
>> during
>> the mount version/protocol negotiation.
>
> There's nothing in -v except that it adds an addr= field. As for the
> packet dump, that's the most interesting part -- for my part, the
> server
> doesn't have iptables, so I'm blocking on the client. In that case,
> exactly zero packets enter the write before mount gives up...
The kernel may be returning an incorrect return code to the mount
command.
Set the NFSDBG_MOUNT flag before you try your mount, then look in /var/
log/messages. That will show you the kernel part of the mount
process. And/or use strace with the mount command.
> The other submitters have iptables blocking on the server, though,
> and it
> doesn't work for them either (one is using NFSv3, the other NFSv4).
> Is this really working for you?
I can't say until you post a complete description of a specific test
case.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
On Fri, Apr 25, 2008 at 10:19:54AM -0400, Chuck Lever wrote:
>> The other submitters have iptables blocking on the server, though, and it
>> doesn't work for them either (one is using NFSv3, the other NFSv4). Is
>> this really working for you?
> I can't say until you post a complete description of a specific test
> case.
On the client:
iptables -A OUTPUT -d 10.0.0.10 -p udp -j DROP
mount -t nfs 10.0.0.10:/foo /bar
Substitute 10.0.0.10 with the server, of course.
/* Steinar */
--
Homepage: http://www.sesse.net/
On Apr 25, 2008, at 10:27 AM, Steinar H. Gunderson wrote:
> On Fri, Apr 25, 2008 at 10:19:54AM -0400, Chuck Lever wrote:
>>> The other submitters have iptables blocking on the server, though,
>>> and it
>>> doesn't work for them either (one is using NFSv3, the other
>>> NFSv4). Is
>>> this really working for you?
>> I can't say until you post a complete description of a specific test
>> case.
>
> On the client:
>
> iptables -A OUTPUT -d 10.0.0.10 -p udp -j DROP
> mount -t nfs 10.0.0.10:/foo /bar
>
> Substitute 10.0.0.10 with the server, of course.
I just tried this specific use case.
Because of the local packet filtering on the client, the kernel's RPC
client is getting -EPERM when trying to send the initial rpcbind
request.
As far as I can see, nothing in the RPC client knows how to deal
specifically with that error code, so it punts the request, and the
mount fails.
When I originally tested mount protocol/version negotiation, I used
only server-side filtering.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
On Fri, Apr 25, 2008 at 06:13:03PM -0400, Chuck Lever wrote:
> I just tried this specific use case.
>
> Because of the local packet filtering on the client, the kernel's RPC
> client is getting -EPERM when trying to send the initial rpcbind
> request.
>
> As far as I can see, nothing in the RPC client knows how to deal
> specifically with that error code, so it punts the request, and the
> mount fails.
>
> When I originally tested mount protocol/version negotiation, I used only
> server-side filtering.
I checked the original bug report, and indeed it seems that the blocking was
on the client there as well. I'll ask the user to check what happens when the
filtering is done server-side instead.
There's also http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=476128, but I
believe that's a different bug.
/* Steinar */
--
Homepage: http://www.sesse.net/
On Sat, Apr 26, 2008 at 01:55:46AM +0200, Steinar H. Gunderson wrote:
> I checked the original bug report, and indeed it seems that the blocking was
> on the client there as well. I'll ask the user to check what happens when the
> filtering is done server-side instead.
Hm, no, there's also reports of blocking on a firewall in the middle; see
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=473171
/* Steinar */
--
Homepage: http://www.sesse.net/
Steinar H. Gunderson wrote:
> Hi,
>
> I've gotten a few bug reports lately that if UDP is blocked at the server,
> NFS mounts (even with -o tcp) fail. git bisect shows that the culprit is
> the change "Get rid of the "-i" option for mount.nfs[4] and always use the
> text-based mount(2) system call for kernel version 2.6.23 and later." -- in
> other words, the bug is specific to the text interface. Does anyone know why
> this would be the case?
This also fails for me. My setup is through an ssh tunnel
ssh -L2049:srv:2049 -L2050:srv:2050 -L2051:srv:2051ser@gateway
and I follow that with
strace mount localhost:/export /mountpoint \
-o nfsvers=3,port=2049,mountport=2050,...
This hangs in this kernel call
mount("localhost:/export", "/mountpoint", "nfs",
MS_NOSUID|MS_NOATIME,
"tcp,intr,fg,soft,nfsvers=3,port=2049,mountport=2050,nolock,addr=127.0.0.1"
This hack makes it work for me again:
diff -ru nfs-utils-1.1.2/utils/mount/mount.c
nfs-utils-1.1.2-changed/utils/mount/mount.c
--- nfs-utils-1.1.2/utils/mount/mount.c 2008-03-14 15:46:29 +0000
+++ nfs-utils-hack/utils/mount/mount.c 2008-04-26 15:49:55 +0100
@@ -191,7 +191,8 @@
nfs_mount_data_version = NFS_MOUNT_VERSION;
else
if (kernel_version > MAKE_VERSION(2, 6, 22))
- string++;
+ string=0;
}
static void print_one(char *spec, char *node, char *type, char *opts)
Frank
Frank A. Kingswood wrote:
> ssh -L2049:srv:2049 -L2050:srv:2050 -L2051:srv:2051ser@gateway
sorry, duff mailer and message wrapping got that. it should say
ssh -L2049:srv:2049 -L2050:srv:2050 -L2051:srv:2051 user@gateway
Steinar H. Gunderson wrote:
> I've gotten a few bug reports lately that if UDP is blocked at the server,
> NFS mounts (even with -o tcp) fail. git bisect shows that the culprit is
> the change "Get rid of the "-i" option for mount.nfs[4] and always use the
> text-based mount(2) system call for kernel version 2.6.23 and later." -- in
> other words, the bug is specific to the text interface. Does anyone know why
> this would be the case?
The NFS client code (nfs-utils-1.1.2) tests for
if (kernel_version > MAKE_VERSION(2, 6, 22))
yet the kernel patch for NFS did not go un until 2.6.25-rc2
> From: Linus Torvalds <torvalds@...>
> Subject: Linux 2.6.25-rc2
> Date: Feb 15, 5:23 pm 2008
>
> Ok,
> this kernel is a winner.
>
> [...]
>
> Chuck Lever (1):
> NFS: Allow text-based mounts via compat_sys_mount
>
So the test should at least be > MAKE_VERSION(2, 6, 24).
Frank
On Apr 26, 2008, at 11:38 AM, Frank A. Kingswood wrote:
> Steinar H. Gunderson wrote:
>> I've gotten a few bug reports lately that if UDP is blocked at the
>> server,
>> NFS mounts (even with -o tcp) fail. git bisect shows that the
>> culprit is
>> the change "Get rid of the "-i" option for mount.nfs[4] and always
>> use the
>> text-based mount(2) system call for kernel version 2.6.23 and
>> later." -- in
>> other words, the bug is specific to the text interface. Does anyone
>> know why
>> this would be the case?
>
> The NFS client code (nfs-utils-1.1.2) tests for
> if (kernel_version > MAKE_VERSION(2, 6, 22))
>
> yet the kernel patch for NFS did not go un until 2.6.25-rc2
>
>> From: Linus Torvalds <torvalds@...>
>> Subject: Linux 2.6.25-rc2
>> Date: Feb 15, 5:23 pm 2008
>> Ok,
>> this kernel is a winner.
>> [...]
>> Chuck Lever (1):
>> NFS: Allow text-based mounts via compat_sys_mount
>
> So the test should at least be > MAKE_VERSION(2, 6, 24).
Frank, the patch you refer to fixes a bug with the compat_sys_mount
interface, which is separate from the sys_mount interface, and used
only on a handful of hardware platforms (SPARC?).
The original NFS text-based mount support was added in 2.6.23.
I don't have a problem with adjusting MAKE_VERSION in the master nfs-
utils repo as you describe, since that bug does break text-based
mounts on those few hardware platforms that use compat_sys_mount
instead of sys_mount. It might affect some early adopters of NFS over
RDMA or FS cache, however.
On distributions that don't support NFS over RDMA or the FS cache
facility, it should be harmless for now to make the mount command use
the legacy mount API even on newer kernels. Simply adjust the
MAKE_VERSION() macro as needed, or disable it entirely as you did with
your "string = 0" hack.
New features will be added only to the text-based mount interface,
however, so eventually everyone will need to use the text-based mount
interface on new kernels.
Because we have little documented history and only a handful of use
cases and unit tests for the mount command, it's important for
everyone to test the new API and report problems here so we can
address them. NFS mount is complex and has many subtle and hidden
historical behaviors.
Even if 70-80% of the common use cases are working properly, there are
going to be outliers that will experience issues until we can get the
corner cases fixed.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
On Apr 25, 2008, at 7:55 PM, Steinar H. Gunderson wrote:
> On Fri, Apr 25, 2008 at 06:13:03PM -0400, Chuck Lever wrote:
>> I just tried this specific use case.
>>
>> Because of the local packet filtering on the client, the kernel's RPC
>> client is getting -EPERM when trying to send the initial rpcbind
>> request.
>>
>> As far as I can see, nothing in the RPC client knows how to deal
>> specifically with that error code, so it punts the request, and the
>> mount fails.
>>
>> When I originally tested mount protocol/version negotiation, I used
>> only
>> server-side filtering.
>
> I checked the original bug report, and indeed it seems that the
> blocking was
> on the client there as well. I'll ask the user to check what happens
> when the filtering is done server-side instead.
>
> There's also http://bugs.debian.org/cgi-bin/bugreport.cgi?
> bug=476128, but I believe that's a different bug.
FYI: Steve Dickson (nfs-utils maintainer) is already looking at
removing EACCES from the list of temporary errors, so yes, we're aware
of that issue.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
On Apr 25, 2008, at 7:57 PM, Steinar H. Gunderson wrote:
> On Sat, Apr 26, 2008 at 01:55:46AM +0200, Steinar H. Gunderson wrote:
>> I checked the original bug report, and indeed it seems that the
>> blocking was
>> on the client there as well. I'll ask the user to check what
>> happens when the
>> filtering is done server-side instead.
>
> Hm, no, there's also reports of blocking on a firewall in the
> middle; see
>
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=473171
The user is correct that the mount process for NFSv4 should not
attempt to use the portmapper unless the port=0 option is specified.
I will look into the problem.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Chuck Lever wrote:
>>
>> The NFS client code (nfs-utils-1.1.2) tests for
>> if (kernel_version > MAKE_VERSION(2, 6, 22))
>>
>> yet the kernel patch for NFS did not go un until 2.6.25-rc2
>> So the test should at least be > MAKE_VERSION(2, 6, 24).
> The original NFS text-based mount support was added in 2.6.23.
OK.
> New features will be added only to the text-based mount interface,
> however, so eventually everyone will need to use the text-based mount
> interface on new kernels.
That sounds like a good idea.
> Because we have little documented history and only a handful of use
> cases and unit tests for the mount command, it's important for
> everyone to test the new API and report problems here so we can
> address them. NFS mount is complex and has many subtle and hidden
> historical behaviors.
My system fails with 2.6.23 and 2.6.25 kernels. As I described in my
original mail, I only have TCP access to the server, and only through an
ssh tunnel.
Would you like me to try and produce some debug traces, or do you think
you can find the problem with the information you have already? I've
seen quite a few reports of this now.
Frank
--
------------------------------------------------------------------------
Frank A. Kingswood frank-YzJE6wl5KmBjuoFY8sQqqY5Lr7LD32fptUK59QYPAWc@public.gmane.org
Cambridge, United Kingdom +44-7943-217 571
On Apr 28, 2008, at 2:24 PM, Frank A. Kingswood wrote:
> Chuck Lever wrote:
>>>
>>> The NFS client code (nfs-utils-1.1.2) tests for
>>> if (kernel_version > MAKE_VERSION(2, 6, 22))
>>>
>>> yet the kernel patch for NFS did not go un until 2.6.25-rc2
>>> So the test should at least be > MAKE_VERSION(2, 6, 24).
>> The original NFS text-based mount support was added in 2.6.23.
> OK.
>> New features will be added only to the text-based mount interface,
>> however, so eventually everyone will need to use the text-based
>> mount interface on new kernels.
> That sounds like a good idea.
>> Because we have little documented history and only a handful of use
>> cases and unit tests for the mount command, it's important for
>> everyone to test the new API and report problems here so we can
>> address them. NFS mount is complex and has many subtle and hidden
>> historical behaviors.
> My system fails with 2.6.23 and 2.6.25 kernels. As I described in my
> original mail, I only have TCP access to the server, and only
> through an ssh tunnel.
>
> Would you like me to try and produce some debug traces, or do you
> think you can find the problem with the information you have
> already? I've seen quite a few reports of this now.
Thanks for the offer. I tested against a server configuration that
dropped UDP packets, but I never tested through an ssh tunnel.
Network traces might be a little hard with a tunnel, but give it a
shot. The best information comes from a full-packet raw trace. Use
the "-s0" option on tcpdump.
You can also trace NFS mount processing with:
# sysctl -w sunrpc.nfs_debug = 128
Or trace all RPC activity (including rpcbind and RPC network transport
errors) with:
# sysctl -w sunrpc.rpc_debug = 32767
You can set both of these at the same time. Trace output will appear
in /var/log/messages. To disable tracing when you're done, set these
sysctls to zero.
You can post these to me privately.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
On Apr 28, 2008, at 11:28 AM, Chuck Lever wrote:
> On Apr 25, 2008, at 7:57 PM, Steinar H. Gunderson wrote:
>> On Sat, Apr 26, 2008 at 01:55:46AM +0200, Steinar H. Gunderson wrote:
>>> I checked the original bug report, and indeed it seems that the
>>> blocking was
>>> on the client there as well. I'll ask the user to check what
>>> happens when the
>>> filtering is done server-side instead.
>>
>> Hm, no, there's also reports of blocking on a firewall in the
>> middle; see
>>
>> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=473171
>
> The user is correct that the mount process for NFSv4 should not
> attempt to use the portmapper unless the port=0 option is
> specified. I will look into the problem.
This issue may be addressed as a side-effect of:
commit f22d6d79fe227245363a8849ea8c85fe6c6598c3
Author: Chuck Lever <[email protected]>
Date: Fri Mar 14 14:10:22 2008 -0400
NFS: Save the value of the "port=" mount option
During a remount based on the mount options displayed in
/proc/mounts, we want to preserve the original behavior of
the mount request. Let's save the original setting of the
"port=" mount option in the mount's nfs_server structure.
Please give 2.6.25 final a try to see if it prevents the noted
extraneous rpcbind activity during an NFSv4 mount.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
> Chuck Lever wrote:
> Thanks for the offer. I tested against a server configuration that
> dropped UDP packets, but I never tested through an ssh tunnel.
In the ssh tunnel case, what can be seen in network captures for
localhost on the client is that mounting requests are sent over UDP, and
since the ports are those specified for the ssh tunnel, they lead to
nothing on localhost UDP, and an ICMP dest unreach (port) is sent back.
This is made twice before mount.nfs exits with an internal error, 2
packets each time.
I have the same problem than Frank, using NFS through a ssh tunnel. It
was working with 2.6.20 and I've just upgraded to 2.6.25 and it does not
works anymore.
Vincent
vincent wrote:
> I have the same problem than Frank, using NFS through a ssh tunnel. It
> was working with 2.6.20 and I've just upgraded to 2.6.25 and it does not
> works anymore.
Chuck sent me a workaround. The kernel understands an option that is not
present in the userland mount code, mountproto. Set that along with the
other options, so -o tcp,mountproto=tcp,... and it will force the mount
call to go over tcp too.
Alternatively, you can grab the source for mount.nfs and disable
text-based mounts by setting string=0, or by changing the kernel version
it tests for to >2.6.25.
I think a fix for the kernel is in the works, so that should appear by
2.6.26 and possibly for a 2.6.25.y too.
Frank
Chuck Lever wrote:
> On Apr 25, 2008, at 7:55 PM, Steinar H. Gunderson wrote:
>> On Fri, Apr 25, 2008 at 06:13:03PM -0400, Chuck Lever wrote:
>>> I just tried this specific use case.
>>>
>>> Because of the local packet filtering on the client, the kernel's RPC
>>> client is getting -EPERM when trying to send the initial rpcbind
>>> request.
>>>
>>> As far as I can see, nothing in the RPC client knows how to deal
>>> specifically with that error code, so it punts the request, and the
>>> mount fails.
>>>
>>> When I originally tested mount protocol/version negotiation, I used only
>>> server-side filtering.
>>
>> I checked the original bug report, and indeed it seems that the
>> blocking was
>> on the client there as well. I'll ask the user to check what happens
>> when the filtering is done server-side instead.
>>
>> There's also http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=476128,
>> but I believe that's a different bug.
>
>
> FYI: Steve Dickson (nfs-utils maintainer) is already looking at removing
> EACCES from the list of temporary errors, so yes, we're aware of that
> issue.
Actually it has been removed already...
steved.
On May 4, 2008, at 6:39 PM, vincent wrote:
>> Chuck Lever wrote:
>> Thanks for the offer. I tested against a server configuration that
>> dropped UDP packets, but I never tested through an ssh tunnel.
>
> In the ssh tunnel case, what can be seen in network captures for
> localhost on the client is that mounting requests are sent over UDP,
> and
> since the ports are those specified for the ssh tunnel, they lead to
> nothing on localhost UDP, and an ICMP dest unreach (port) is sent
> back.
> This is made twice before mount.nfs exits with an internal error, 2
> packets each time.
>
> I have the same problem than Frank, using NFS through a ssh tunnel. It
> was working with 2.6.20 and I've just upgraded to 2.6.25 and it does
> not
> works anymore.
After 2.6.23, mount options are passed to the kernel as a C string
instead of binary data structure. The mount command formerly
contacted the server via the mountd protocol, and passed mount
information to the kernel. Now the kernel has its own mountd client
and it uses that instead.
The new kernel mount option parser behaves a little differently than
the user-space mount command with regard to the "proto=" option. If
you specify "proto=tcp" the legacy mount command contacts the server's
mountd via TCP. However, the kernel continues to use the default
setting (UDP) for the mount client's transport protocol.
This is a bug, and I have a kernel patch that makes the kernel mount
option parser behave like the legacy mount command.
Until this bug is addressed, you can use "mountproto=" to specify the
transport protocol explicitly for the kernel's mount client. See
nfs(5) for details.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com