2003-06-17 19:46:08

by Matt Schillinger

[permalink] [raw]
Subject: exportfs operations hang

Hello,

I am having reproducible problems with exportfs.

When I try to run exportfs or exportfs -u, it simply stalls and never
drops back to commandline.. if i CTRL+C, i see that (in the case of an
exportfs) the mountpoint is still not shared. In the case of an exportfs
-u, If i CTRL+C it, the mountpoint IS still shared. I do not know what
is causing the problem.. the only workaround that I have found is to
restart nfsd.. Is there something in rmtab/xtab/etab that could cause an
exportfs hangup?

I am using 64 threads, and am having no thread utilization issues
according to /proc/net/rpc/nfsd

Has anyone else had this problem?

Thanks for your help
--
Matt Schillinger
System Administrator
FlightSafety International
[email protected]
314-551-8403




-------------------------------------------------------
This SF.Net email is sponsored by: INetU
Attention Web Developers & Consultants: Become An INetU Hosting Partner.
Refer Dedicated Servers. We Manage Them. You Get 10% Monthly Commission!
INetU Dedicated Managed Hosting http://www.inetu.net/partner/index.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs


2003-06-17 22:08:26

by Steve Dickson

[permalink] [raw]
Subject: Re: exportfs operations hang

Here does strace say its hanging?

SteveD.

Matt Schillinger wrote:

>Hello,
>
>I am having reproducible problems with exportfs.
>
>When I try to run exportfs or exportfs -u, it simply stalls and never
>drops back to commandline.. if i CTRL+C, i see that (in the case of an
>exportfs) the mountpoint is still not shared. In the case of an exportfs
>-u, If i CTRL+C it, the mountpoint IS still shared. I do not know what
>is causing the problem.. the only workaround that I have found is to
>restart nfsd.. Is there something in rmtab/xtab/etab that could cause an
>exportfs hangup?
>
>I am using 64 threads, and am having no thread utilization issues
>according to /proc/net/rpc/nfsd
>
>Has anyone else had this problem?
>
>Thanks for your help
>
>



-------------------------------------------------------
This SF.Net email is sponsored by: INetU
Attention Web Developers & Consultants: Become An INetU Hosting Partner.
Refer Dedicated Servers. We Manage Them. You Get 10% Monthly Commission!
INetU Dedicated Managed Hosting http://www.inetu.net/partner/index.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2003-06-18 16:47:34

by Matt Schillinger

[permalink] [raw]
Subject: Re: exportfs operations hang

On Tue, 2003-06-17 at 17:08, Steve Dickson wrote:
> Here does strace say its hanging?
>
> SteveD.

Duh! No I haven't.. I will do that when issues occur again.. However, I
may or may not have found the problem..

A few new things that I've found.

rpc.mountd is not displaying any syslog entries when hangs occur on
clients.. the client gets 'Permission Denied' when trying to mount (via
automount). Other clients are accessing the mounts just fine..

I found that there is a -o or --descriptor option for file descriptors,
which the default operation is 256 file descriptors. This got me
thinking about how many file descriptors that I should have.. I couldn't
find anything about tuning file descriptors for rpc.mountd. Could
someone give me a guideline or idea?

I went ahead and bumped the file descriptors up to 512 for good measure.
( this required a kill of the current rpc.mountd and restart of it.)
After the restart, the clients that were getting 'Permission Denied',
were able to access the mounts. I do not know yet if the restart fixed
the problem, or the increased file descriptors, or both.

I would really like to know what the recommended file descriptor number
should be.

Default is 256, but the default nfsd threads is 8.
I am running 64 threads, with an Input Queue of 2097152

Should rpc.mountd file descriptors move up in relation to number of
threads?

Thanks for your help,

Matt Schillinger
[email protected]
>
> Matt Schillinger wrote:
>
> >Hello,
> >
> >I am having reproducible problems with exportfs.
> >
> >When I try to run exportfs or exportfs -u, it simply stalls and never
> >drops back to commandline.. if i CTRL+C, i see that (in the case of an
> >exportfs) the mountpoint is still not shared. In the case of an exportfs
> >-u, If i CTRL+C it, the mountpoint IS still shared. I do not know what
> >is causing the problem.. the only workaround that I have found is to
> >restart nfsd.. Is there something in rmtab/xtab/etab that could cause an
> >exportfs hangup?
> >
> >I am using 64 threads, and am having no thread utilization issues
> >according to /proc/net/rpc/nfsd
> >
> >Has anyone else had this problem?
> >
> >Thanks for your help
> >
> >
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: INetU
> Attention Web Developers & Consultants: Become An INetU Hosting Partner.
> Refer Dedicated Servers. We Manage Them. You Get 10% Monthly Commission!
> INetU Dedicated Managed Hosting http://www.inetu.net/partner/index.php
> _______________________________________________
> NFS maillist - [email protected]
> https://lists.sourceforge.net/lists/listinfo/nfs




-------------------------------------------------------
This SF.Net email is sponsored by: INetU
Attention Web Developers & Consultants: Become An INetU Hosting Partner.
Refer Dedicated Servers. We Manage Them. You Get 10% Monthly Commission!
INetU Dedicated Managed Hosting http://www.inetu.net/partner/index.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2003-07-03 14:56:50

by Matt Schillinger

[permalink] [raw]
Subject: Re: exportfs operations hang

I finally got an opportunity to reproduce the result of exportfs..
attached is the strace output..


I previously had a problem that an incorrectly set broadcast address was
causing rpc.mountd to 'stop functioning' when alot of new machines mount
in a short amount of time.. Anyway, the only thing that showed up in
the log, was a heartbeat message (string2msg) that was actually noting a
problem on the system.. anyway, i noted the same log message from when i
did the exportfs.

I did email the heartbeat mailinglist, but they did not have any
definitive answer as to what the message meant to me, but said that it
was not a heartbeat error which i believe.

As a possible avenue of troubleshooting, when i do exportfs, i do it
like this::

exportfs -o rw,async @vss_grp:/export/vss/db/mv22

I did some examination of my netgroups file, and found a few things..

1. there are host entries in netgroup that do not exist in host (because
the hosts are no longer online, and when they cleaned out hosts, they
didn't clean netgroup.

2. there are a couple cases of inherited groups (the vss_grp calls 4
other groups, and some of those groups are also calling subgroups). I
believe that there is no more than 2 levels of groups.. I was unsure if
this would have an effect..

------

Action that I immediately took was to clear out the old hosts, and
verify that all hosts existed in /etc/hosts or in dns (if the netgroup
called the FQDN.

Thanks for your help,
Matt
[email protected]


On Tue, 2003-06-17 at 17:08, Steve Dickson wrote:
> Here does strace say its hanging?
>
> SteveD.
>
> Matt Schillinger wrote:
>
> >Hello,
> >
> >I am having reproducible problems with exportfs.
> >
> >When I try to run exportfs or exportfs -u, it simply stalls and never
> >drops back to commandline.. if i CTRL+C, i see that (in the case of an
> >exportfs) the mountpoint is still not shared. In the case of an exportfs
> >-u, If i CTRL+C it, the mountpoint IS still shared. I do not know what
> >is causing the problem.. the only workaround that I have found is to
> >restart nfsd.. Is there something in rmtab/xtab/etab that could cause an
> >exportfs hangup?
> >
> >I am using 64 threads, and am having no thread utilization issues
> >according to /proc/net/rpc/nfsd
> >
> >Has anyone else had this problem?
> >
> >Thanks for your help
> >
> >
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by: INetU
> Attention Web Developers & Consultants: Become An INetU Hosting Partner.
> Refer Dedicated Servers. We Manage Them. You Get 10% Monthly Commission!
> INetU Dedicated Managed Hosting http://www.inetu.net/partner/index.php
> _______________________________________________
> NFS maillist - [email protected]
> https://lists.sourceforge.net/lists/listinfo/nfs
--
Matt Schillinger
System Administrator
FlightSafety International
[email protected]
314-551-8403


Attachments:
exportfs.outstrace (146.71 kB)