Hi,
I have a sudden problem when my server reboot and mount to a NFS
failed, so I neeed to mount
by hand. (But it was okay before...)
Error:
mount: trying xxxx prog 100003 vers 3 prot udp port 2049
mount: trying xxxx prog 100005 vers 3 prot udp port 891
mount: xxxx:/data0/tmp failed, reason given by server: Permission
denied
Many pages in the web said it is related to the hostname and ip
mapping, which made permission issues,
but in my NFS server, I see the permission is ok:
(/var/log/message)
authenticated mount request from xxxx for yyyy...
Any idea for this funny problem?
Thanks.
Hello,
On Tue, Sep 30, 2008 at 1:26 AM, J. Bruce Fields <[email protected]> wrote:
> It looks like nfsd is supposed to be mounted on load of the nfsd module,
> by a line in /etc/modprobe.d/modprobe.conf.
>
> (Maybe you built a new kernel with nfsd built-in instead of built as a
> module?)
>
> But I thought nfs-utils was supposed to fall back on old behavior when
> the nfsd filesystem wasn't found.
Are there any related docs I should read?
As mentioned before, the server was working before the reboot, really
don't understand why it behave in this way...Need to find out the
reason.
THanks!
Just because it's "authenticated" doesn't mean you have "permission".
What are the contents of your server's export file, and the umode of
the target /data0/tmp directory? And btw, what types/versions are
your client and server?
Tom.
At 07:32 AM 9/26/2008, howard chen wrote:
>Hi,
>
>I have a sudden problem when my server reboot and mount to a NFS
>failed, so I neeed to mount
>by hand. (But it was okay before...)
>
>Error:
>
>mount: trying xxxx prog 100003 vers 3 prot udp port 2049
>mount: trying xxxx prog 100005 vers 3 prot udp port 891
>mount: xxxx:/data0/tmp failed, reason given by server: Permission
>denied
>
>Many pages in the web said it is related to the hostname and ip
>mapping, which made permission issues,
>but in my NFS server, I see the permission is ok:
>
>(/var/log/message)
>
>authenticated mount request from xxxx for yyyy...
>
>Any idea for this funny problem?
>
Thanks.
On Fri, Sep 26, 2008 at 7:50 PM, Talpey, Thomas
<[email protected]> wrote:
> Just because it's "authenticated" doesn't mean you have "permission".
> What are the contents of your server's export file, and the umode of
> the target /data0/tmp directory? And btw, what types/versions are
> your client and server?
>
Thanks. Consider below as the full picture:
NFS Server (10.10.10.1)
=========================
/data0/tmp 10.10.10.2(rw,sync)
NFS Client (10.10.10.2)
=======================
# Add in /etc/rc.local
mount -t nfs -o
async,noatime,noexec,nosuid,hard,intr,udp,retry=3,rsize=32768,wsize=32768
10.10.10.1:/data0/tmp /home/www/tmp
ls /data0/
drwxrwxr-x 261 root web 12288 Sep 26 22:58 tmp
Thanks.
At 11:05 AM 9/26/2008, howard chen wrote:
>Thanks.
>
>
>On Fri, Sep 26, 2008 at 7:50 PM, Talpey, Thomas
><[email protected]> wrote:
>> Just because it's "authenticated" doesn't mean you have "permission".
>> What are the contents of your server's export file, and the umode of
>> the target /data0/tmp directory? And btw, what types/versions are
>> your client and server?
>>
>
>Thanks. Consider below as the full picture:
You didn't indicate what the client and server were, btw.
Do you have multiple interfaces on the client? If the client routing
originates from a different address than 10.10.10.2, then the server
will deny it because you have specified a numerical address. You might
try exporting to "*" to see if this changes anything. Apart from that
possibility, the configuration looks plausible.
This isn't the source of the permissions error, but why are you doing
a UDP mount, and with only three retries? Generally, TCP will perform
better, and more robustly. Also, the "noatime" option is a no-op for
the NFS client (servers are in charge of maintaining atime).
Tom.
>
>NFS Server (10.10.10.1)
>=========================
>/data0/tmp 10.10.10.2(rw,sync)
>
>NFS Client (10.10.10.2)
>=======================
># Add in /etc/rc.local
>mount -t nfs -o
>async,noatime,noexec,nosuid,hard,intr,udp,retry=3,rsize=32768,wsize=32768
>10.10.10.1:/data0/tmp /home/www/tmp
>
>ls /data0/
>
>drwxrwxr-x 261 root web 12288 Sep 26 22:58 tmp
>
>
>
>Thanks.
Hi,
On Fri, Sep 26, 2008 at 11:25 PM, Talpey, Thomas
<[email protected]> wrote:
> At 11:05 AM 9/26/2008, howard chen wrote:
> You didn't indicate what the client and server were, btw.
client is 3, as I see when mount with verbose option: ... mount:
trying xxxx prog 100003 vers 3 prot udp port 2049
server should be also 3, default by CentOS 4.4, 64bit
>
> Do you have multiple interfaces on the client? If the client routing
> originates from a different address than 10.10.10.2, then the server
> will deny it because you have specified a numerical address.
Yes, both servers have interface for public IP, but as I can see the
10.10.10.2 in /var/log/message of NFS server, so I think routing is
ok.
> This isn't the source of the permissions error, but why are you doing
> a UDP mount, and with only three retries? Generally, TCP will perform
> better, and more robustly. Also, the "noatime" option is a no-op for
> the NFS client (servers are in charge of maintaining atime).
I agree TCP is more robust, but isn't UDP will have a better performance?
Thanks.
On Fri, 2008-09-26 at 23:57 +0800, howard chen wrote:
> Hi,
>
> On Fri, Sep 26, 2008 at 11:25 PM, Talpey, Thomas
> <[email protected]> wrote:
> > At 11:05 AM 9/26/2008, howard chen wrote:
> > You didn't indicate what the client and server were, btw.
>
> client is 3, as I see when mount with verbose option: ... mount:
> trying xxxx prog 100003 vers 3 prot udp port 2049
>
> server should be also 3, default by CentOS 4.4, 64bit
What does
'rpcinfo -p 10.10.10.1'
give you? Also,
'showmount -e 10.10.10.1'
Finally, what kind of filesystem are you exporting on /data0/tmp?
> >
> > Do you have multiple interfaces on the client? If the client routing
> > originates from a different address than 10.10.10.2, then the server
> > will deny it because you have specified a numerical address.
>
> Yes, both servers have interface for public IP, but as I can see the
> 10.10.10.2 in /var/log/message of NFS server, so I think routing is
> ok.
>
>
>
> > This isn't the source of the permissions error, but why are you doing
> > a UDP mount, and with only three retries? Generally, TCP will perform
> > better, and more robustly. Also, the "noatime" option is a no-op for
> > the NFS client (servers are in charge of maintaining atime).
>
> I agree TCP is more robust, but isn't UDP will have a better performance?
That depends. In my experience, the difference in performance on an
unloaded network, then UDP will outperform TCP by ~10%. However, if you
have a heavily loaded network with lots of dropped packets, then TCP
will usually give much better performance than UDP.
Trond
Hi,
Thanks all for your input first.
On Sat, Sep 27, 2008 at 12:06 AM, Trond Myklebust
<[email protected]> wrote:
> What does
>
> 'rpcinfo -p 10.10.10.1'
program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100021 1 udp 32773 nlockmgr
100021 3 udp 32773 nlockmgr
100021 4 udp 32773 nlockmgr
100021 1 tcp 32779 nlockmgr
100021 3 tcp 32779 nlockmgr
100021 4 tcp 32779 nlockmgr
100024 1 udp 883 status
100024 1 tcp 886 status
100011 1 udp 821 rquotad
100011 2 udp 821 rquotad
100011 1 tcp 824 rquotad
100011 2 tcp 824 rquotad
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100005 1 udp 891 mountd
100005 1 tcp 894 mountd
100005 2 udp 891 mountd
100005 2 tcp 894 mountd
100005 3 udp 891 mountd
100005 3 tcp 894 mountd
> give you? Also,
>
> 'showmount -e 10.10.10.1'
>
Export list for 10.10.10.1
/data0/tmp 10.10.10.2
> That depends. In my experience, the difference in performance on an
> unloaded network, then UDP will outperform TCP by ~10%. However, if you
> have a heavily loaded network with lots of dropped packets, then TCP
> will usually give much better performance than UDP.
Good to know! I will definitely have a test, any parameters are also
recommend together with TCP so I can do a fair benchmark?
Thank you again.
Have you tried my suggestion of replacing "10.10.10.2" with "*" in
your server's exports, as a test? It will tell us whether you have
an IP aliasing issue.
As for TCP, no special mount parameters are required. The timers
and algorithms are all adaptive and preconfiguring them is often
counterproductive. You still haven't shown "uname -a" for your
client, but if it's anything recent, you don't even need to specify
tcp, it's the default.
Tom.
At 12:13 PM 9/26/2008, howard chen wrote:
>Hi,
>
>Thanks all for your input first.
>
>On Sat, Sep 27, 2008 at 12:06 AM, Trond Myklebust
><[email protected]> wrote:
>
>> What does
>>
>> 'rpcinfo -p 10.10.10.1'
>
> program vers proto port
> 100000 2 tcp 111 portmapper
> 100000 2 udp 111 portmapper
> 100021 1 udp 32773 nlockmgr
> 100021 3 udp 32773 nlockmgr
> 100021 4 udp 32773 nlockmgr
> 100021 1 tcp 32779 nlockmgr
> 100021 3 tcp 32779 nlockmgr
> 100021 4 tcp 32779 nlockmgr
> 100024 1 udp 883 status
> 100024 1 tcp 886 status
> 100011 1 udp 821 rquotad
> 100011 2 udp 821 rquotad
> 100011 1 tcp 824 rquotad
> 100011 2 tcp 824 rquotad
> 100003 2 udp 2049 nfs
> 100003 3 udp 2049 nfs
> 100003 4 udp 2049 nfs
> 100003 2 tcp 2049 nfs
> 100003 3 tcp 2049 nfs
> 100003 4 tcp 2049 nfs
> 100005 1 udp 891 mountd
> 100005 1 tcp 894 mountd
> 100005 2 udp 891 mountd
> 100005 2 tcp 894 mountd
> 100005 3 udp 891 mountd
> 100005 3 tcp 894 mountd
>
>
>> give you? Also,
>>
>> 'showmount -e 10.10.10.1'
>>
>
>
>Export list for 10.10.10.1
>/data0/tmp 10.10.10.2
>
>
>
>
>> That depends. In my experience, the difference in performance on an
>> unloaded network, then UDP will outperform TCP by ~10%. However, if you
>> have a heavily loaded network with lots of dropped packets, then TCP
>> will usually give much better performance than UDP.
>
>Good to know! I will definitely have a test, any parameters are also
>recommend together with TCP so I can do a fair benchmark?
>
>
>Thank you again.
Hello,
On Sat, Sep 27, 2008 at 12:23 AM, Talpey, Thomas
<[email protected]> wrote:
> Have you tried my suggestion of replacing "10.10.10.2" with "*" in
> your server's exports, as a test? It will tell us whether you have
> an IP aliasing issue.
>
Yes, tested, and same error... Also tried "no_root_squash"...
Really have no idea what is going on...
Thanks.
Hello all,
On Mon, Sep 29, 2008 at 12:21 PM, howard chen <[email protected]> wrote:
>
> Yes, tested, and same error... Also tried "no_root_squash"...
>
> Really have no idea what is going on...
>
Problem solved by adding the following line in fstab, according to
this site: http://www.linuxforums.org/forum/linux-networking/44779-nfs-permission-denied-error.html
nfsd /proc/fs/nfsd nfsd auto,defaults 0 0
This is really funny as I never have this option in my other NFS servers....
Anyone know the reason? Redhat's bug?
Thanks
On Mon, Sep 29, 2008 at 12:53:16PM +0800, howard chen wrote:
> Hello all,
>
> On Mon, Sep 29, 2008 at 12:21 PM, howard chen <[email protected]> wrote:
> >
> > Yes, tested, and same error... Also tried "no_root_squash"...
> >
> > Really have no idea what is going on...
> >
>
> Problem solved by adding the following line in fstab, according to
> this site: http://www.linuxforums.org/forum/linux-networking/44779-nfs-permission-denied-error.html
Huh. Surely mountd or knfsd could have given a more helpful error
message, at least....
> nfsd /proc/fs/nfsd nfsd auto,defaults 0 0
>
>
> This is really funny as I never have this option in my other NFS servers....
>
> Anyone know the reason? Redhat's bug?
It looks like nfsd is supposed to be mounted on load of the nfsd module,
by a line in /etc/modprobe.d/modprobe.conf.
(Maybe you built a new kernel with nfsd built-in instead of built as a
module?)
But I thought nfs-utils was supposed to fall back on old behavior when
the nfsd filesystem wasn't found.
--b.
On Sep. 29, 2008, 20:26 +0300, "J. Bruce Fields" <[email protected]> wrote:
> On Mon, Sep 29, 2008 at 12:53:16PM +0800, howard chen wrote:
>> Hello all,
>>
>> On Mon, Sep 29, 2008 at 12:21 PM, howard chen <[email protected]> wrote:
>>> Yes, tested, and same error... Also tried "no_root_squash"...
>>>
>>> Really have no idea what is going on...
>>>
>> Problem solved by adding the following line in fstab, according to
>> this site: http://www.linuxforums.org/forum/linux-networking/44779-nfs-permission-denied-error.html
>
> Huh. Surely mountd or knfsd could have given a more helpful error
> message, at least....
I agree.
I've just hit this problem again today (running Fedora 9).
(and I think http://bugzilla.redhat.com/show_bug.cgi?id=218994 might
be the same thing BTW)
rpc.mountd just prints
authenticated mount request from 192.168.0.140:669 for /usr0/export
even when invoked with --debug all.
However it denies access. Here's tshark -V output:
Remote Procedure Call, Type:Call XID:0xee9360da
Fragment header: Last fragment, 124 bytes
1... .... .... .... .... .... .... .... = Last Fragment: Yes
.000 0000 0000 0000 0000 0000 0111 1100 = Fragment Length: 124
XID: 0xee9360da (4002635994)
Message Type: Call (0)
RPC Version: 2
Program: MOUNT (100005)
Program Version: 3
Procedure: MNT (1)
Credentials
Flavor: AUTH_UNIX (1)
Length: 68
Stamp: 0x004235f6
Machine Name: pangw.bhalevy.com
length: 17
contents: pangw.bhalevy.com
fill bytes: opaque data
UID: 0
GID: 0
Auxiliary GIDs
GID: 0
GID: 1
GID: 2
GID: 3
GID: 4
GID: 6
GID: 10
Verifier
Flavor: AUTH_NULL (0)
Length: 0
Mount Service
[Program Version: 3]
[V3 Procedure: MNT (1)]
Path: /usr0/export
length: 12
contents: /usr0/export
...
Remote Procedure Call, Type:Reply XID:0xee9360da
Fragment header: Last fragment, 28 bytes
1... .... .... .... .... .... .... .... = Last Fragment: Yes
.000 0000 0000 0000 0000 0000 0001 1100 = Fragment Length: 28
XID: 0xee9360da (4002635994)
Message Type: Reply (1)
[Program: MOUNT (100005)]
[Program Version: 3]
[Procedure: MNT (1)]
Reply State: accepted (0)
[This is a reply to a request in frame 3474]
[Time from request: 0.000868000 seconds]
Verifier
Flavor: AUTH_NULL (0)
Length: 0
Accept State: RPC executed successfully (0)
Mount Service
[Program Version: 3]
[V3 Procedure: MNT (1)]
Status: ERR_ACCESS (13)
Running strace rpc.mountd -F --debug all shows:
...
open("/proc/fs/nfsd/filehandle", O_RDWR) = -1 ENOENT (No such file or directory)
open("/proc/fs/nfs/filehandle", O_RDWR) = -1 ENOENT (No such file or directory)
>
>> nfsd /proc/fs/nfsd nfsd auto,defaults 0 0
doping this manually using
mount -t nfsd nfsd /proc/fs/nfsd
does the same trick
>>
>>
>> This is really funny as I never have this option in my other NFS servers....
>>
>> Anyone know the reason? Redhat's bug?
>
> It looks like nfsd is supposed to be mounted on load of the nfsd module,
> by a line in /etc/modprobe.d/modprobe.conf.
Right.
# grep nfsd /etc/modprobe.d/modprobe.conf.dist
install nfsd /sbin/modprobe --first-time --ignore-install nfsd && { /bin/mount -t nfsd nfsd /proc/fs/nfsd > /dev/null 2>&1 || :; }
remove nfsd { /bin/umount /proc/fs/nfsd > /dev/null 2>&1 || :; } ; /sbin/modprobe -r --first-time --ignore-remove nfsd
>
> (Maybe you built a new kernel with nfsd built-in instead of built as a
> module?)
>
> But I thought nfs-utils was supposed to fall back on old behavior when
> the nfsd filesystem wasn't found.
Doesn't seem so.
Steve: any chance this bug is fixed anywhere?
Benny
>
> --b.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Mar 03, 2009 at 12:04:17PM +0200, Benny Halevy wrote:
> On Sep. 29, 2008, 20:26 +0300, "J. Bruce Fields" <[email protected]> wrote:
> >
> > (Maybe you built a new kernel with nfsd built-in instead of built as a
> > module?)
> >
> > But I thought nfs-utils was supposed to fall back on old behavior when
> > the nfsd filesystem wasn't found.
>
> Doesn't seem so.
Right. The old behavior depends on mountd telling the server about each
client as it mounts, and that isn't adequate in some cases: for example,
since nfsv4 clients never talk directly to mountd at all (they first
call nfsd directly), they need to be able to call up to mountd to ask
about each client as it arrives, and the old interface doesn't provide
a mechanism to do that.
But in this case I don't see any reason the old interface shouldn't be
good enough.
> Steve: any chance this bug is fixed anywhere?
It shouldn't be hard to fix.
If someone has the time to work on it but doesn't know the code, just
testing with varied versions of nfs-utils and the kernel would help
determine whether the problem is in mountd or in the kernel, and when it
was introduced.
--b.