2008-08-25 07:45:28

by Jari Aalto

[permalink] [raw]
Subject: Linux NFS-HOWTO - improvement to Troubleshooting permission denied


Thank you for the NFS howto (last modified 2006). Please add further
diagnostics to section:

=> "I do not have permission to access files on the mounted volume"
http://nfs.sourceforge.net/nfs-howto/ar01s07.html#pemission_issues

The error condition "Permission denied" also happens in certain NFS
(client/server) combinations and the workaround is to add additional
mount parameter at client:

sec=sys

Please add this to the "#pemission_issues" (tag the word seems to have a
small typo).

See reported incidents:

http://thread.gmane.org/gmane.linux.debian.user/333162
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=492970

Jari

----
CC: team http://sourceforge.net/projects/nfs

You're all in CC, as I did not know who is responsible for updating the
NFS-HOWTO at sourceforge.


2008-08-25 15:41:09

by Chuck Lever III

[permalink] [raw]
Subject: Re: Linux NFS-HOWTO - improvement to Troubleshooting permission denied

Hi Jari-

On Mon, Aug 25, 2008 at 3:36 AM, Jari Aalto <[email protected]> wrote:
>
> Thank you for the NFS howto (last modified 2006). Please add further
> diagnostics to section:
>
> => "I do not have permission to access files on the mounted volume"
> http://nfs.sourceforge.net/nfs-howto/ar01s07.html#pemission_issues
>
> The error condition "Permission denied" also happens in certain NFS
> (client/server) combinations and the workaround is to add additional
> mount parameter at client:
>
> sec=sys
>
> Please add this to the "#pemission_issues" (tag the word seems to have a
> small typo).

This is a known bug in nfs-utils 1.1.3 when used with older client
kernels that do not support text-based mounts.

Seems like an esoteric (ie not widespread) problem, but I could easily
be wrong about that.

--
Chuck Lever

2008-08-25 15:55:08

by Bruce Fields

[permalink] [raw]
Subject: Re: Linux NFS-HOWTO - improvement to Troubleshooting permission denied

On Mon, Aug 25, 2008 at 11:41:02AM -0400, Chuck Lever wrote:
> Hi Jari-
>
> On Mon, Aug 25, 2008 at 3:36 AM, Jari Aalto <[email protected]> wrote:
> >
> > Thank you for the NFS howto (last modified 2006). Please add further
> > diagnostics to section:
> >
> > => "I do not have permission to access files on the mounted volume"
> > http://nfs.sourceforge.net/nfs-howto/ar01s07.html#pemission_issues
> >
> > The error condition "Permission denied" also happens in certain NFS
> > (client/server) combinations and the workaround is to add additional
> > mount parameter at client:
> >
> > sec=sys
> >
> > Please add this to the "#pemission_issues" (tag the word seems to have a
> > small typo).
>
> This is a known bug in nfs-utils 1.1.3 when used with older client
> kernels that do not support text-based mounts.
>
> Seems like an esoteric (ie not widespread) problem, but I could easily
> be wrong about that.

People should be able to upgrade nfs-utils without upgrading their
kernel!

The referenced debian bug report:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=492970

claims the first bad commit is a server-side commit that itself seems to
claim to work around a problem? ("For mountd, it does not advertise
AUTH_NULL anymore. This is necessary to avoid backward compatibility
issue. If AUTH_NULL appears in the list, either the new or the old
client will choose that over AUTH_SYS.")

Did we ever find the actual root cause?

--b.

2008-08-25 16:58:46

by Chuck Lever III

[permalink] [raw]
Subject: Re: Linux NFS-HOWTO - improvement to Troubleshooting permission denied

Some CC's corrected.

On Aug 25, 2008, at Aug 25, 2008, 11:55 AM, J. Bruce Fields wrote:
> On Mon, Aug 25, 2008 at 11:41:02AM -0400, Chuck Lever wrote:
>> Hi Jari-
>>
>> On Mon, Aug 25, 2008 at 3:36 AM, Jari Aalto <[email protected]>
>> wrote:
>>>
>>> Thank you for the NFS howto (last modified 2006). Please add further
>>> diagnostics to section:
>>>
>>> => "I do not have permission to access files on the mounted
>>> volume"
>>> http://nfs.sourceforge.net/nfs-howto/ar01s07.html#pemission_issues
>>>
>>> The error condition "Permission denied" also happens in certain NFS
>>> (client/server) combinations and the workaround is to add additional
>>> mount parameter at client:
>>>
>>> sec=sys
>>>
>>> Please add this to the "#pemission_issues" (tag the word seems to
>>> have a
>>> small typo).
>>
>> This is a known bug in nfs-utils 1.1.3 when used with older client
>> kernels that do not support text-based mounts.
>>
>> Seems like an esoteric (ie not widespread) problem, but I could
>> easily
>> be wrong about that.
>
> People should be able to upgrade nfs-utils without upgrading their
> kernel!

Of course. But Bugs Happen. I'm not implying that nfs-utils 1.1.3
requires a kernel upgrade, but merely describing the conditions where
the issue appears.

In fact, you may recall a recent and rather protracted e-mail
conversation where we determined that the text-based NFS mount
implementation is not working correctly in this case either.

> The referenced debian bug report:
>
> http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=492970
>
> claims the first bad commit is a server-side commit that itself
> seems to
> claim to work around a problem? ("For mountd, it does not advertise
> AUTH_NULL anymore. This is necessary to avoid backward compatibility
> issue. If AUTH_NULL appears in the list, either the new or the old
> client will choose that over AUTH_SYS.")
>
> Did we ever find the actual root cause?

The bad commit does not live up to the comments it introduced, which
claim that the legacy mount command will skip AUTH_NULL as a default
if it receives it in the auth flavor list from mountd. I haven't
verified this, but otherwise, yes, we have an RCA.

I copied the patch author of record, [email protected], on the recent
discussion, but we haven't heard any response. I think we will need
to fix this ourselves. IMO the proper fix is to eliminate the
possibility that the client will ever use AUTH_NULL unless "sec=null"
is explicitly specified on the mount command line, then to get nfs-
utils 1.1.4 posted with this fix.

If I may also observe: the bad commit attempts to change both server-
side and client-side behavior in one go. Hindsight suggests the
changes to nfsmount.c and mountd.c should have occurred in two
separate commits so that automated bisection and surgical reversion
are possible.

This reminds us again that we have an exponentially expanding test
matrix for nfs-utils, especially considering the backwards-
compatibility requirements. I would love to see significant test
resources committed in this area.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

2008-08-25 20:50:20

by Jari Aalto

[permalink] [raw]
Subject: Re: Linux NFS-HOWTO - improvement to Troubleshooting permission denied

"Chuck Lever" <[email protected]> writes:
> On Mon, Aug 25, 2008 at 3:36 AM, Jari Aalto <[email protected]> wrote:
>>
>> Thank you for the NFS howto (last modified 2006). Please add further
>> diagnostics to section:
>>
>> => "I do not have permission to access files on the mounted volume"
>> http://nfs.sourceforge.net/nfs-howto/ar01s07.html#pemission_issues
>>
>> The error condition "Permission denied" also happens in certain NFS
>> (client/server) combinations and the workaround is to add additional
>> mount parameter at client:
>>
>> sec=sys
>>
>> Please add this to the "#pemission_issues" (tag the word seems to have a
>> small typo).
>
> This is a known bug in nfs-utils 1.1.3 when used with older client
> kernels that do not support text-based mounts.
>
> Seems like an esoteric (ie not widespread) problem, but I could easily
> be wrong about that.

Please add the tip, although the bug may be fixed in later NFS releases.
There are great deal of hosts running old versions whose admins scratch
they head and seek for guidance from the HOWTO.

Mentioning "sec=sys" would have saved me countless of hours tracing what
was wrong in my setup[1]. Without to Bob <grantura <at> spamcop.net> and
his knowledge about Debian bug, I would never had resolved the issue in
decent time.

Jari

[1]
"NFS and no_root_squash - cannot write as root - permission denied"
http://thread.gmane.org/gmane.linux.debian.user/333162

http://bugs.debian.org/492970