2004-11-22 19:43:01

by Dwight Marzolf

[permalink] [raw]
Subject: udp ports not being released

We are running RedHat 8, kernel 2.4.20-20.8smp and autofs 4.1.2. We
were having a variety of automount problems until a kernel patch was
recommended. We added this patch, recompiled our kernel, and
distributed the patched kernel to all of our workstations. About 90% of
our problems disappeared. But one problem seems to remain.

When /net is used to mount directories, most of the time the udp port
used for that nfs mount operation is not released once the /net mount
times out. Depending on the level of activity on a machine, over the
period of several days, all the available reserved ports are used up.
At that point the user on the workstation cannot cd to any mount points
in our automount maps except the ones already mounted. In the
/var/log/messages the following error occurs for attempted mounts:

RPC: Can't bind to reserved port

In my experiments I found that doing a "umount -l /net" releases all the
udp ports immediately. Our workaround has been to kill all the autofs
processes, do a umount -l on all mount points in /etc/mtab, and them
restart the autofs process.

My question is (1) is there any fix for this problem or some switch I
need to turn on/off in order for these udp ports to be released when the
mount point times out and (2) is there some command that I can use to
release these ports rather than going through the multiple step process
I described above to clean up a machine?

Dwight Marzolf




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs


2004-11-22 19:56:32

by Mike Waychison

[permalink] [raw]
Subject: Re: udp ports not being released

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dwight Marzolf wrote:
> We are running RedHat 8, kernel 2.4.20-20.8smp and autofs 4.1.2. We
> were having a variety of automount problems until a kernel patch was
> recommended. We added this patch, recompiled our kernel, and
> distributed the patched kernel to all of our workstations. About 90% of
> our problems disappeared. But one problem seems to remain.
>
> When /net is used to mount directories, most of the time the udp port
> used for that nfs mount operation is not released once the /net mount
> times out. Depending on the level of activity on a machine, over the
> period of several days, all the available reserved ports are used up.
> At that point the user on the workstation cannot cd to any mount points
> in our automount maps except the ones already mounted. In the
> /var/log/messages the following error occurs for attempted mounts:

IIRC, automount 4.1.2 had a fd leak, that was fixed in 4.1.3

Ian?

>
> RPC: Can't bind to reserved port
>
> In my experiments I found that doing a "umount -l /net" releases all the
> udp ports immediately. Our workaround has been to kill all the autofs
> processes, do a umount -l on all mount points in /etc/mtab, and them
> restart the autofs process.
>
> My question is (1) is there any fix for this problem or some switch I
> need to turn on/off in order for these udp ports to be released when the
> mount point times out and (2) is there some command that I can use to
> release these ports rather than going through the multiple step process
> I described above to clean up a machine?
>
> Dwight Marzolf
>
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> NFS maillist - [email protected]
> https://lists.sourceforge.net/lists/listinfo/nfs


- --
Mike Waychison
Sun Microsystems, Inc.
1 (650) 352-5299 voice
1 (416) 202-8336 voice

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NOTICE: The opinions expressed in this email are held by me,
and may not represent the views of Sun Microsystems, Inc.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBokRTdQs4kOxk3/MRAmNFAJ9vYo0GjHbM2RlquSgAVjbJ32lnlQCeOZkc
8hRRC7gWmZwspf9ySrnkZE4=
=H26H
-----END PGP SIGNATURE-----


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2004-11-23 01:33:26

by Ian Kent

[permalink] [raw]
Subject: Re: udp ports not being released

On Mon, 22 Nov 2004, Mike Waychison wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Dwight Marzolf wrote:
> > We are running RedHat 8, kernel 2.4.20-20.8smp and autofs 4.1.2. We
> > were having a variety of automount problems until a kernel patch was
> > recommended. We added this patch, recompiled our kernel, and
> > distributed the patched kernel to all of our workstations. About 90% of
> > our problems disappeared. But one problem seems to remain.
> >
> > When /net is used to mount directories, most of the time the udp port
> > used for that nfs mount operation is not released once the /net mount
> > times out. Depending on the level of activity on a machine, over the
> > period of several days, all the available reserved ports are used up.
> > At that point the user on the workstation cannot cd to any mount points
> > in our automount maps except the ones already mounted. In the
> > /var/log/messages the following error occurs for attempted mounts:
>
> IIRC, automount 4.1.2 had a fd leak, that was fixed in 4.1.3

Yes and also in 4.1.3. There is a patch on kernel.org but it has been
superceeded and not updated. I'll update it soon as I can (called
autofs-4.1.3-sock-leak-fix.patch).

I'm not sure about the socket leaks anyway. We fix them because it's good
practice but it appears that most of this happens in sub-processes, which
once finished close all their open fds. This certainly seems to work
correctly from what I've tested.

For Redhat Jeff Moyers' rpms track the important fixes. Dwight, you should
try using one of them since you have problems.

>
> Ian?
>
> >
> > RPC: Can't bind to reserved port
> >
> > In my experiments I found that doing a "umount -l /net" releases all the
> > udp ports immediately. Our workaround has been to kill all the autofs
> > processes, do a umount -l on all mount points in /etc/mtab, and them
> > restart the autofs process.
> >
> > My question is (1) is there any fix for this problem or some switch I
> > need to turn on/off in order for these udp ports to be released when the
> > mount point times out and (2) is there some command that I can use to
> > release these ports rather than going through the multiple step process
> > I described above to clean up a machine?

See above.

There is of course the possiblity this is a new problem so let us know
about your progress if any.

Ian



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs