I ran into some problem yesterday where ls on a directory containing
several unexported mountpoints was hanging. Watching writes to the
/proc/net/rpc/nfsd.export/channel file, what I saw was:
write("gss/krb5i /path/a expiry\n") -> -ENOENT
write("gss/krb5i /path/a expiry\ngss/krb5i /path/b\n" -> -ENOENT
write("gss/krb5i /path/a expiry\ngss/krb5i /path/b\ngss/krb5i/path/b\n) ->ENOENT
etc. You get the idea. The cache code that handles the write throws
away all but the first line, so the extra lines are ignored. The kernel
keeps doing upcalls to ask about /path/b, but never sees the results
because mountd keeps passing down the /path/a downcall again as the
first line each time.
I don't see how nfs-utils itself could be producing multiple lines in
the upcall handler, so I assume what's happening is that the stdio code
is taking the error returns as a sign that it should leave the data in
its buffer and just try again next time another fwrite adds more. Does
that make sense? I don't really understand how stdio is supposed to
handle errors.
The -ENOENTs are spurious in this case, actually; I'll submit a patch
to eliminate them. But could we also just ditch the use of stdio in all
the nfs-utils cache code? The current problem aside it seems likely to
complicate any error handling (which the current nfs-utils code doesn't
even attempt).
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Wed, Jul 25, 2007 at 06:37:23PM -0400, J. Bruce Fields wrote:
> I assume what's happening is that the stdio code is taking the error
> returns as a sign that it should leave the data in its buffer and just
> try again next time another fwrite adds more. Does that make sense?
> I don't really understand how stdio is supposed to handle errors.
So I tried a test program that just does
while (1) {
fprintf(f, "%d\n", i++)
fflush(f);
}
and pointed it at one of those proc files, which will always return an
error on such a write. On my fedora rawhide system, each write only
writes the new data. On my debian sid system, each write writes all the
data accumulated so far. Huh.
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Wednesday July 25, [email protected] wrote:
> On Wed, Jul 25, 2007 at 06:37:23PM -0400, J. Bruce Fields wrote:
> > I assume what's happening is that the stdio code is taking the error
> > returns as a sign that it should leave the data in its buffer and just
> > try again next time another fwrite adds more. Does that make sense?
> > I don't really understand how stdio is supposed to handle errors.
>
> So I tried a test program that just does
>
> while (1) {
> fprintf(f, "%d\n", i++)
> fflush(f);
> }
>
> and pointed it at one of those proc files, which will always return an
> error on such a write. On my fedora rawhide system, each write only
> writes the new data. On my debian sid system, each write writes all the
> data accumulated so far. Huh.
:-)
While I can see the value it avoiding the stdio code, I can also see
some in hanging on to it, so I would first like to see if we can make
the current code work with just minor modifications.
Could you check how calling 'clearerr' interacts with this code on the
two different systems?
Thanks,
NeilBrown
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Thu, Jul 26, 2007 at 12:59:09PM +1000, Neil Brown wrote:
> While I can see the value it avoiding the stdio code, I can also see
> some in hanging on to it, so I would first like to see if we can make
> the current code work with just minor modifications.
>
> Could you check how calling 'clearerr' interacts with this code on the
> two different systems?
I tried inserting a clearerr into the loop:
fprintf();
fflush();
clearerr();
No change in behavior.
I see the advantage to doing just one step at a time--so if we can
figure out a minimal fix for now, fine. But surely we've got to replace
the stdio stuff at some point. We need precise control over the writes,
so the buffering can only get in the way.
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Wednesday July 25, [email protected] wrote:
> On Thu, Jul 26, 2007 at 12:59:09PM +1000, Neil Brown wrote:
> > While I can see the value it avoiding the stdio code, I can also see
> > some in hanging on to it, so I would first like to see if we can make
> > the current code work with just minor modifications.
> >
> > Could you check how calling 'clearerr' interacts with this code on the
> > two different systems?
>
> I tried inserting a clearerr into the loop:
>
> fprintf();
> fflush();
> clearerr();
>
> No change in behavior.
Hmmm.... (reads man page)...
#include <stdio_ext.h>
...
__fpurge(f)
seems to do the trick.
>
> I see the advantage to doing just one step at a time--so if we can
> figure out a minimal fix for now, fine. But surely we've got to replace
> the stdio stuff at some point. We need precise control over the writes,
> so the buffering can only get in the way.
But we want to build a line up from multiple words, and so we need
buffering for that.
If we toss stdio, we will need to allocate a buffer, keep track of the
end point, and append to the buffer in various places. It feels like
reinventing the wheel.
NeilBrown
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Could all this be related to a problem that we've been seeing while
working on v4/4.1 at netapp, where exportfs doesn't show any output even
when there are exported file systems? Ricardo recently observed that if
you redirect the output of 'exportfs' then you actually do get the
correct output (e.g., exportfs > afile, or exportfs | more). This lead
to some experiments with strace that show exportfs behaving differently
when its output is going to a tty. We've seen this behavior across a
variety of kernels (2.6.18-2.6.20) and a variety of nfs-utils
(prepackaged with FC6, RHEL4/5, and built from a local copy of 1.0.12).
N.B. no output from the first exportfs, but correct output when piped to
'more':
[root@teal ~]# exportfs -a
[root@teal ~]# exportfs
[root@teal ~]# exportfs | more
/ <world>
Now looking at straces with and without redirect:
[root@teal ~]# strace exportfs 2> exportfs.noredirect
[root@teal ~]# strace exportfs | more 2> exportfs.redirect
In the 'no redirection' case, there's an ioctl that fails, and the
output never appears in the xterm window (assuming that's related to the
ioctl failure the returns ENOTTY). N.B., we see the 'write' system call
in the strace, but don't see any output from that call. From
exportfs.noredirect:
ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbf8aabc8) = -1 ENOTTY
(Inappropriate ioctl for device)
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7fba000
write(1, "/ \t<world>\n", 23) = 23
exit_group(0) = ?
In the case of redirection, we don't do the ioctl, and you can see the
'write' of the exportfs output in the strace as well as the result of
the 'write'. From exportfs.redirect:
fstat64(1, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0 mmap2(NULL, 4096,
PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f8e000
write(1, "/ \t<world>\n", 23) = 23
exit_group(0) = ?
/ <world>
Process 8408 detached
-----Original Message-----
From: J. Bruce Fields [mailto:[email protected]]
Sent: Wednesday, July 25, 2007 8:18 PM
To: Neil Brown
Cc: [email protected]
Subject: Re: [NFS] stdio and nfs-utils
On Thu, Jul 26, 2007 at 12:59:09PM +1000, Neil Brown wrote:
> While I can see the value it avoiding the stdio code, I can also see
> some in hanging on to it, so I would first like to see if we can make
> the current code work with just minor modifications.
>
> Could you check how calling 'clearerr' interacts with this code on the
> two different systems?
I tried inserting a clearerr into the loop:
fprintf();
fflush();
clearerr();
No change in behavior.
I see the advantage to doing just one step at a time--so if we can
figure out a minimal fix for now, fine. But surely we've got to replace
the stdio stuff at some point. We need precise control over the writes,
so the buffering can only get in the way.
--b.
------------------------------------------------------------------------
-
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Thu, Jul 26, 2007 at 11:43:10AM -0700, Muntz, Daniel wrote:
> Could all this be related to a problem that we've been seeing while
> working on v4/4.1 at netapp, where exportfs doesn't show any output even
> when there are exported file systems? Ricardo recently observed that if
> you redirect the output of 'exportfs' then you actually do get the
> correct output (e.g., exportfs > afile, or exportfs | more).
I wouldn't expect that to be related to the problem with i/o to these
proc files.
But, yes, I've seen the same problem with exportfs output. (Shame on me
for not reporting it at at the time.... And thanks for looking into it!)
But I don't know what the problem is.
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Thu, 26 Jul 2007 11:43:10 -0700
"Muntz, Daniel" <[email protected]> wrote:
> Could all this be related to a problem that we've been seeing while
> working on v4/4.1 at netapp, where exportfs doesn't show any output even
> when there are exported file systems? Ricardo recently observed that if
> you redirect the output of 'exportfs' then you actually do get the
> correct output (e.g., exportfs > afile, or exportfs | more). This lead
> to some experiments with strace that show exportfs behaving differently
> when its output is going to a tty. We've seen this behavior across a
> variety of kernels (2.6.18-2.6.20) and a variety of nfs-utils
> (prepackaged with FC6, RHEL4/5, and built from a local copy of 1.0.12).
>
I saw a very similar problem with RHEL5 prior to its release. It turned
out to be a selinux policy issue. I believe the latest selinux-policy
packages in Fedora and RHEL have fixed it if it's the same problem.
--
Jeff Layton <[email protected]>
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Thu, Jul 26, 2007 at 03:48:06PM -0400, Jeff Layton wrote:
> I saw a very similar problem with RHEL5 prior to its release. It turned
> out to be a selinux policy issue. I believe the latest selinux-policy
> packages in Fedora and RHEL have fixed it if it's the same problem.
That makes sense. I don't remember for sure where I saw this, but it
may well have been on my Fedora (rawhide) desktop, and I haven't seen it
recently.
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
From: J. Bruce Fields <[email protected]>
On a recent Debian/Sid machine, I saw libc retrying stdio writes that
returned write errors. The result is that if an export downcall returns
an error (which it can in normal operation, since it currently
(incorrectly) returns -ENOENT on any negative downcall), then subsequent
downcalls will write multiple lines (including the original line that
received the error).
The result is that the server fails to respond to any rpc call that
refers to an unexported mount point (such as a readdir of a directory
containing such a mountpoint), so client commands hang.
I don't know whether this libc behavior is correct or expected, but it
seems safest to add the __fpurge() (suggested by Neil) to ensure data is
thrown away.
Signed-off-by: "J. Bruce Fields" <[email protected]>
---
support/nfs/cacheio.c | 12 ++++++++++++
1 files changed, 12 insertions(+), 0 deletions(-)
On Thu, Jul 26, 2007 at 02:10:15PM +1000, Neil Brown wrote:
> __fpurge(f)
>
> seems to do the trick.
Yep, thanks!
> But we want to build a line up from multiple words, and so we need
> buffering for that. If we toss stdio, we will need to allocate a
> buffer, keep track of the end point, and append to the buffer in
> various places. It feels like reinventing the wheel.
Reinventing a rather simple part of the wheel, I'd say. Oh well,
another day.
--b.
diff --git a/support/nfs/cacheio.c b/support/nfs/cacheio.c
index a76915b..9d271cd 100644
--- a/support/nfs/cacheio.c
+++ b/support/nfs/cacheio.c
@@ -17,6 +17,7 @@
#include <nfslib.h>
#include <stdio.h>
+#include <stdio_ext.h>
#include <ctype.h>
#include <unistd.h>
#include <sys/types.h>
@@ -111,7 +112,18 @@ void qword_printint(FILE *f, int num)
int qword_eol(FILE *f)
{
+ int err;
+
fprintf(f,"\n");
+ err = fflush(f);
+ /*
+ * We must send one line (and one line only) in a single write
+ * call. In case of a write error, libc may accumulate the
+ * unwritten data and try to write it again later, resulting in a
+ * multi-line write. So we must explicitly ask it to throw away
+ * any such cached data:
+ */
+ __fpurge(f);
return fflush(f);
}
--
1.5.3.rc2
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
From: J. Bruce Fields <[email protected]>
A succesful downcall with a negative result (which indicates that the
given filesystem is not exported to the given user) should not return an
error.
Currently mountd is depending on stdio to write these downcalls. With
some versions of libc this appears to cause subsequent writes to attempt
to write all accumulated data (for which writes previously failed) along
with any new data. This can prevent the kernel from seeing responses to
later downcalls. Symptoms will be that nfsd fails to respond to certain
requests.
Signed-off-by: "J. Bruce Fields" <[email protected]>
---
fs/nfsd/export.c | 5 +++--
1 files changed, 3 insertions(+), 2 deletions(-)
And here's the kernel fix.--b.
diff --git a/fs/nfsd/export.c b/fs/nfsd/export.c
index 2d295dd..cba899a 100644
--- a/fs/nfsd/export.c
+++ b/fs/nfsd/export.c
@@ -564,9 +564,10 @@ static int svc_export_parse(struct cache_detail *cd, char *mesg, int mlen)
/* flags */
err = get_int(&mesg, &an_int);
- if (err == -ENOENT)
+ if (err == -ENOENT) {
+ err = 0;
set_bit(CACHE_NEGATIVE, &exp.h.flags);
- else {
+ } else {
if (err || an_int < 0) goto out;
exp.ex_flags= an_int;
--
1.5.3.rc2
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Any idea what one would need to change to verify if it's the same
problem? Our base installs are almost all FC6 and RHEL 4/5 (begging the
question of whether it was fixed in RHEL 5 prior to release). Someone
here can probably test on an FC7 system, but it would be really handy to
know how to fix the many existing installations, assuming it's the same
problem.
Thanks,
Dan
-----Original Message-----
From: Jeff Layton [mailto:[email protected]]
Sent: Thursday, July 26, 2007 12:48 PM
To: Muntz, Daniel
Cc: J. Bruce Fields; Neil Brown; [email protected]
Subject: Re: [NFS] Exportfs not showing exported file systems (was RE:
stdio and nfs-utils)
On Thu, 26 Jul 2007 11:43:10 -0700
"Muntz, Daniel" <[email protected]> wrote:
> Could all this be related to a problem that we've been seeing while
> working on v4/4.1 at netapp, where exportfs doesn't show any output
> even when there are exported file systems? Ricardo recently observed
> that if you redirect the output of 'exportfs' then you actually do get
> the correct output (e.g., exportfs > afile, or exportfs | more). This
> lead to some experiments with strace that show exportfs behaving
> differently when its output is going to a tty. We've seen this
> behavior across a variety of kernels (2.6.18-2.6.20) and a variety of
> nfs-utils (prepackaged with FC6, RHEL4/5, and built from a local copy
of 1.0.12).
>
I saw a very similar problem with RHEL5 prior to its release. It turned
out to be a selinux policy issue. I believe the latest selinux-policy
packages in Fedora and RHEL have fixed it if it's the same problem.
--
Jeff Layton <[email protected]>
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Thu, 26 Jul 2007 13:42:08 -0700
"Muntz, Daniel" <[email protected]> wrote:
> Any idea what one would need to change to verify if it's the same
> problem? Our base installs are almost all FC6 and RHEL 4/5 (begging the
> question of whether it was fixed in RHEL 5 prior to release). Someone
> here can probably test on an FC7 system, but it would be really handy to
> know how to fix the many existing installations, assuming it's the same
> problem.
>
> Thanks,
> Dan
>
While it's discouraged for us to suggest, you could do what I did and
temporarily put selinux in permissive mode: ;-)
# setenforce 0
...and then see if exportfs still has the issue. To turn it back to
enforcing do:
# setenforce 1
I've not been able to reproduce the problem since it was reportedly
fixed. The BZ I opened on it is here:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=221181
...if you find that selinux is at fault then you may want to reopen it.
It might also be worthwhile to do the same with Bruce's original
problem and see if it behaves any differently.
--
Jeff Layton <[email protected]>
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Thu, Jul 26, 2007 at 05:12:09PM -0400, Jeff Layton wrote:
> ...if you find that selinux is at fault then you may want to reopen it.
> It might also be worthwhile to do the same with Bruce's original
> problem and see if it behaves any differently.
The system where I was seeing the problem was a debian system without
selinux.
--b.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Yep, disabling selinux fixed it on my FC6 machine.
-Dan
> -----Original Message-----
> From: Jeff Layton [mailto:[email protected]]
> Sent: Thursday, July 26, 2007 2:12 PM
> To: Muntz, Daniel
> Cc: J. Bruce Fields; Neil Brown; [email protected]
> Subject: Re: [NFS] Exportfs not showing exported file systems
> (was RE: stdio and nfs-utils)
>
> On Thu, 26 Jul 2007 13:42:08 -0700
> "Muntz, Daniel" <[email protected]> wrote:
>
> > Any idea what one would need to change to verify if it's the same
> > problem? Our base installs are almost all FC6 and RHEL 4/5
> (begging the
> > question of whether it was fixed in RHEL 5 prior to
> release). Someone
> > here can probably test on an FC7 system, but it would be
> really handy to
> > know how to fix the many existing installations, assuming
> it's the same
> > problem.
> >
> > Thanks,
> > Dan
> >
>
> While it's discouraged for us to suggest, you could do what I did and
> temporarily put selinux in permissive mode: ;-)
>
> # setenforce 0
>
> ...and then see if exportfs still has the issue. To turn it back to
> enforcing do:
>
> # setenforce 1
>
> I've not been able to reproduce the problem since it was reportedly
> fixed. The BZ I opened on it is here:
>
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=221181
>
> ...if you find that selinux is at fault then you may want to
> reopen it.
> It might also be worthwhile to do the same with Bruce's original
> problem and see if it behaves any differently.
>
> --
> Jeff Layton <[email protected]>
>
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Muntz, Daniel wrote:
> Yep, disabling selinux fixed it on my FC6 machine.
A better idea might be to simply 'yum update selinux-policy'
steved.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems? Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs