Hi all,
I have discovered the following problem in Linux NFS: When you mount
many filesystems from the same server, mountd sometimes refuses the
request with "illegal port" message in syslog. To reproduce this, I did
the following:
On the NFS server:
# for i in `seq 1 2048`; do mkdir /tmp/nfs/$i; echo "$i" >/tmp/nfs/$i/index-$i.txt; echo "/tmp/nfs/$i *.fi.muni.cz(ro)"; done >/etc/exports
# exportfs -a
On the NFS client;
# for i in `seq 1 2048`; do mkdir /tmp/nfs/$i; done
# for i in `seq 1 100`; do echo $i; mount -t nfs -r nfs-server:/tmp/nfs/$i /tmp/nfs/$i || break; done
It fails after random number of iterations (seen 6 to 57) with
mount: nfs-server:/tmp/nfs/6 failed, reason given by server: Permission denied
And in NFS server the following is added to syslog:
Oct 27 14:28:48 nfs-server rpc.mountd: refused mount request from nfs-client.fi.muni.cz for /tmp/nfs/6 (/tmp/nfs/6): illegal port 54071
The situation seems a bit better when you add "sleep 5" to the mount loop.
Sleep 2 does not help, though.
-Yenya
--
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839 Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/ Czech Linux Homepage: http://www.linux.cz/ |
> Whatever the Java applications and desktop dances may lead to, Unix will <
> still be pushing the packets around for a quite a while. --Rob Pike <
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Wed, Oct 27, 2004 at 02:42:28PM +0200, Jan Kasprzak wrote:
> I have discovered the following problem in Linux NFS: When you mount
> many filesystems from the same server, mountd sometimes refuses the
> request with "illegal port" message in syslog. To reproduce this, I did
> the following:
Just add the "insecure" export option to the server's export.
--Bruce Fields
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Jan Kasprzak wrote:
: I have discovered the following problem in Linux NFS: When you mount
: many filesystems from the same server, mountd sometimes refuses the
: request with "illegal port" message in syslog.
Well, the mountd is probably OK, and it is the client who runs
out of privileged ports. netstat -ta on the client looks like this:
tcp 0 0 nfs-client.fi.muni.cz:880 nfs-server.fi.muni.c:sunrpc TIME_WAIT
tcp 0 0 nfs-client.fi.muni.cz:883 nfs-server.fi.muni.c:sunrpc TIME_WAIT
tcp 0 0 nfs-client.fi.muni.cz:884 nfs-server.fi.muni.c:sunrpc TIME_WAIT
tcp 0 0 nfs-client.fi.muni.cz:887 nfs-server.fi.muni.c:sunrpc TIME_WAIT
tcp 0 0 nfs-client.fi.muni.cz:888 nfs-server.fi.muni.c:sunrpc TIME_WAIT
tcp 0 0 nfs-client.fi.muni.cz:891 nfs-server.fi.muni.c:sunrpc TIME_WAIT
[...]
The solution might be to query portmap over UDP from the NFS client
instead of TCP. Or to query portmap from an unprivileged port (why we
send query to portmap from privileged port anyway?).
I have even tried echo 2 > /proc/sys/net/ipv4/tcp_fin_timeout,
but this did not solve the problem - connections to server's portmap
remain for 60 seconds in FIN_WAIT.
-Yenya
--
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839 Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/ Czech Linux Homepage: http://www.linux.cz/ |
> Whatever the Java applications and desktop dances may lead to, Unix will <
> still be pushing the packets around for a quite a while. --Rob Pike <
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
J. Bruce Fields wrote:
: On Wed, Oct 27, 2004 at 02:42:28PM +0200, Jan Kasprzak wrote:
: > I have discovered the following problem in Linux NFS: When you mount
: > many filesystems from the same server, mountd sometimes refuses the
: > request with "illegal port" message in syslog. To reproduce this, I did
: > the following:
:
: Just add the "insecure" export option to the server's export.
:
Of course, but I don't want to do this. The problem is that client
is querying portmap over TCP and from privileged port. By changing this
we can use twice as much mounts in the TIME_WAIT period. We may also
force mount to query mountd over UDP, but I don't know how to do this.
-o udp is probably for nfsd traffic, isn't it?
-Yenya
--
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839 Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/ Czech Linux Homepage: http://www.linux.cz/ |
> Whatever the Java applications and desktop dances may lead to, Unix will <
> still be pushing the packets around for a quite a while. --Rob Pike <
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Wed, Oct 27, 2004 at 06:23:22PM +0200, Jan Kasprzak wrote:
> J. Bruce Fields wrote:
> : On Wed, Oct 27, 2004 at 02:42:28PM +0200, Jan Kasprzak wrote:
> : > I have discovered the following problem in Linux NFS: When you mount
> : > many filesystems from the same server, mountd sometimes refuses the
> : > request with "illegal port" message in syslog. To reproduce this, I did
> : > the following:
> :
> : Just add the "insecure" export option to the server's export.
> :
> Of course, but I don't want to do this.
Why not?
> The problem is that client is querying portmap over TCP and from
> privileged port. By changing this we can use twice as much mounts in
> the TIME_WAIT period.
OK. It seems like kind of an obscure thing to care about....
> We may also force mount to query mountd over UDP, but I don't know how
> to do this. -o udp is probably for nfsd traffic, isn't it?
That's what I'd assume, but I don't know.--b.
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
J. Bruce Fields wrote:
: > :
: > : Just add the "insecure" export option to the server's export.
: > :
: > Of course, but I don't want to do this.
:
: Why not?
:
NFS client is a host where ordinary (=evil :-) users have accounts.
-Y.
--
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839 Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/ Czech Linux Homepage: http://www.linux.cz/ |
> Whatever the Java applications and desktop dances may lead to, Unix will <
> still be pushing the packets around for a quite a while. --Rob Pike <
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Wed, 27 Oct 2004, Jan Kasprzak wrote:
> Of course, but I don't want to do this. The problem is that client
> is querying portmap over TCP and from privileged port. By changing this
> we can use twice as much mounts in the TIME_WAIT period. We may also
> force mount to query mountd over UDP, but I don't know how to do this.
> -o udp is probably for nfsd traffic, isn't it?
As far as I know, this has never been documented. This is somewhat related
to the Fedora bug 128100: Even if you specify -o tcp, port= and
mountport=, the mount command tries to "autodetect" if it should use TCP
or UDP for the MOUNT protocol by contacting the portmapper (which fails in
many cases, if you use SSH tunnels, for example).
Most people nowadays have realized that it was a mistake to separate the
NFS and the MOUNT protocol. It's much more convenient to run them on the
same port, using the same transport. With the current mountd/knfsd
implementation it might be a little hard to use the same port number, but
with unfsd, for example, it's easy to use the same port. Using the same
transport type, though, is easy.
So, my opinion is that you should either use TCP for both the NFS and
MOUNT protocol, or UDP for both.
Perhaps this "illegal port" problem can be solved by making the mount
command use SO_REUSEADDR?
--
Peter ?strand Chief Developer
Cendio http://www.thinlinc.com
Teknikringen 3 http://www.cendio.se
583 30 Link?ping Phone: +46-13-21 46 00
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
-----BEGIN PGP SIGNED MESSAGE-----
>>>>> "Jan" == Jan Kasprzak <[email protected]> writes:
Jan> I have discovered the following problem in Linux NFS: When you
What version are you using?
On client? on server?
Jan> And in NFS server the following is added to syslog:
Jan> Oct 27 14:28:48 nfs-server rpc.mountd: refused mount request
Jan> from nfs-client.fi.muni.cz for /tmp/nfs/6 (/tmp/nfs/6): illegal
Jan> port 54071
Does your server expect to see reserved ports only (<1024).
Can you do a tcpdump of the mounts?
To do this, on the server do:
rpcinfo -p localhost
note the port number of the mountd service. Let's say it is X.
tcpdump -s 1600 -i eth0 -n -p -w somefile.pcap udp port X or tcp port X
tcpdump -r somefile.pcap -T rpc -n
- --
] "Elmo went to the wrong fundraiser" - The Simpson | firewalls [
] Michael Richardson, Xelerance Corporation, Ottawa, ON |net architect[
] [email protected] http://www.sandelman.ottawa.on.ca/mcr/ |device driver[
] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)
Comment: Finger me for keys
iQCVAwUBQYANXoqHRg3pndX9AQGncAP/bj6qf9FcHKdu0I37JGPzisrdeiI9VBV7
wYjwONqRjfyhub+STaM5Upid8uwpzSw3rs1ixZJeXX5PklSmDz5XW0Gik/oauwWL
OfPik8NEUsZNjSnS6sjaLCRRUC7PKAB3wTap6IMq/cskLjBTACJ6Gzj2vNYu9sSD
96BPplHUrcs=
=liY1
-----END PGP SIGNATURE-----
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Michael Richardson wrote:
: What version are you using?
: On client? on server?
:
Kernel 2.6.9 on both hosts (server is x86-64, client i386),
client nfs-utils-1.0.6-22 on Fedora Core 2, server nfs-utils-1.0.6-34
on Fedora Core 3 test 3.
: Jan> And in NFS server the following is added to syslog:
:
: Jan> Oct 27 14:28:48 nfs-server rpc.mountd: refused mount request
: Jan> from nfs-client.fi.muni.cz for /tmp/nfs/6 (/tmp/nfs/6): illegal
: Jan> port 54071
:
: Does your server expect to see reserved ports only (<1024).
Yes (there is no "insecure" option in /etc/exports).
: Can you do a tcpdump of the mounts?
:
Yes - I did. The failing mount request uses unprivileged source port
unlike the previous ones.
: tcpdump -s 1600 -i eth0 -n -p -w somefile.pcap udp port X or tcp port X
:
:
: tcpdump -r somefile.pcap -T rpc -n
I will do this on Monday - now I am not nowhere near these servers.
But as I said, the problem is that the client runs out of privileged
ports, because both portmap request and mountd request are done over TCP.
-Yenya
--
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839 Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/ Czech Linux Homepage: http://www.linux.cz/ |
> Whatever the Java applications and desktop dances may lead to, Unix will <
> still be pushing the packets around for a quite a while. --Rob Pike <
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
-----BEGIN PGP SIGNED MESSAGE-----
>>>>> "Jan" == Jan Kasprzak <[email protected]> writes:
Jan> : Does your server expect to see reserved ports only (<1024).
Jan> Yes (there is no "insecure" option in /etc/exports).
Yes, the rest of the thread arrived.
Clearly, you are running out of reserved ports.
- --
] "Elmo went to the wrong fundraiser" - The Simpson | firewalls [
] Michael Richardson, Xelerance Corporation, Ottawa, ON |net architect[
] [email protected] http://www.sandelman.ottawa.on.ca/mcr/ |device driver[
] panic("Just another Debian GNU/Linux using, kernel hacking, security guy"); [
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)
Comment: Finger me for keys
iQCVAwUBQYAe1IqHRg3pndX9AQHbNwP9FAC+myGfExU3Cjl4phvRW48fpOUkOYci
80blUbA/Br+qOQvX+MMX52rSN0z7laQRV00LK7i1QXkk/uVUuwfIrxPVuZYIFm0S
OSXHXA8jp6bH9ypcBivd9B4t1FvyLFb199VsyoMkav0JeJ1tornCQBKDK0TECfm5
yX6BwrP579k=
=FsWx
-----END PGP SIGNATURE-----
-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs