Hi all,
I'm running red hat 7.2 with the 2.4.9-31 kernel.
Its an ext3 file system, exported for nfsv3 udp and nfsv2 clients.
nothing fancy at all.
I have quotas set for my users. If a user hits their hard quota and the
process continues to attempt to write to a file in their quota'd space
- the process load on the server shoots through the roof.
I can reliably recreate this by setting a hard quota, having the system
hit the hard quota and continue to attempt to write to that file - the
server becomes terribly unhappy.
I never saw this problem on older 2.2.X servers and I'm not sure why I'm
seeing it now.
Any places I should be looking first?
thanks
-sv
--
GPG Public Key: http://www.phy.duke.edu/~skvidal/skvidal.gpg
On April 9, [email protected] wrote:
> Hi all,
> I'm running red hat 7.2 with the 2.4.9-31 kernel.
> Its an ext3 file system, exported for nfsv3 udp and nfsv2 clients.
>
> nothing fancy at all.
>
> I have quotas set for my users. If a user hits their hard quota and the
> process continues to attempt to write to a file in their quota'd space
> - the process load on the server shoots through the roof.
>
> I can reliably recreate this by setting a hard quota, having the system
> hit the hard quota and continue to attempt to write to that file - the
> server becomes terribly unhappy.
>
> I never saw this problem on older 2.2.X servers and I'm not sure why I'm
> seeing it now.
>
> Any places I should be looking first?
>
tcpdump?
The server should return nfserr_dquot (69 == 0x45) and the client
should return this to the application.
But it sounds like the client is repeating the request for some
reason.
You could also
echo 32 > /proc/sys/sunrpc/nfsd_debug
and watch for
"nfsd: write complete err=%d"
messages in the kernel log. These will tell you whether nfsd thinks
it got an error for the write.
NeilBrown
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Tue, 2002-04-09 at 00:45, Neil Brown wrote:
> On April 9, [email protected] wrote:
> > Hi all,
> > I'm running red hat 7.2 with the 2.4.9-31 kernel.
> > Its an ext3 file system, exported for nfsv3 udp and nfsv2 clients.
> >
> > nothing fancy at all.
> >
> > I have quotas set for my users. If a user hits their hard quota and the
> > process continues to attempt to write to a file in their quota'd space
> > - the process load on the server shoots through the roof.
> >
> > I can reliably recreate this by setting a hard quota, having the system
> > hit the hard quota and continue to attempt to write to that file - the
> > server becomes terribly unhappy.
> >
> > I never saw this problem on older 2.2.X servers and I'm not sure why I'm
> > seeing it now.
> >
> > Any places I should be looking first?
> >
>
> tcpdump?
> The server should return nfserr_dquot (69 == 0x45) and the client
> should return this to the application.
> But it sounds like the client is repeating the request for some
> reason.
>
> You could also
> echo 32 > /proc/sys/sunrpc/nfsd_debug
> and watch for
> "nfsd: write complete err=%d"
> messages in the kernel log. These will tell you whether nfsd thinks
> it got an error for the write.
>
It appears to return -122
However, the program will exit correctly if it catches that problem. But
the system does not respond well if there is a continued attempt to
write a lot of data when the quota is filled.
That doesn't seem to make sense.
-sv
Neil Brown wrote:
>
> You could also
> echo 32 > /proc/sys/sunrpc/nfsd_debug
> and watch for
> "nfsd: write complete err=%d"
> messages in the kernel log. These will tell you whether nfsd thinks
> it got an error for the write.
Can you elaborate on what the various values one could write to
/proc/sys/sunrpc/nfsd_debug and what you would get with different values in
terms of logging? This sounds like something that could be useful to many
others and included in the HOWTO or FAQ as well. I've checked the kernel docs
and they basically say to 'use the source,' but since we have some of the
core NFS client/server developers here perhaps someone can offer a few hints
for people debugging NFS configurations.
Also, is there any known or performance or stability hit to running with
NFS debug on?
thanks,
andrew
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Tuesday April 9, [email protected] wrote:
> Neil Brown wrote:
>
> >
> > You could also
> > echo 32 > /proc/sys/sunrpc/nfsd_debug
> > and watch for
> > "nfsd: write complete err=%d"
> > messages in the kernel log. These will tell you whether nfsd thinks
> > it got an error for the write.
>
> Can you elaborate on what the various values one could write to
> /proc/sys/sunrpc/nfsd_debug and what you would get with different values in
> terms of logging? This sounds like something that could be useful to many
> others and included in the HOWTO or FAQ as well. I've checked the kernel docs
> and they basically say to 'use the source,' but since we have some of the
> core NFS client/server developers here perhaps someone can offer a few hints
> for people debugging NFS configurations.
"use the source" is really the right answer :-)
If you look in the source, it is fairly easy to see what to put in
/proc/sys/sunrpc/nfsd_debug, and you can see what the results means.
If you don't look at the source, then the debugging output probably
isn't very helpful.
However:
If you look in
include/linux/nfsd/debug.h
you will see:
/*
* knfsd debug flags
*/
#define NFSDDBG_SOCK 0x0001
#define NFSDDBG_FH 0x0002
#define NFSDDBG_EXPORT 0x0004
#define NFSDDBG_SVC 0x0008
#define NFSDDBG_PROC 0x0010
#define NFSDDBG_FILEOP 0x0020
#define NFSDDBG_AUTH 0x0040
#define NFSDDBG_REPCACHE 0x0080
#define NFSDDBG_XDR 0x0100
#define NFSDDBG_LOCKD 0x0200
So if you
echo 1 > /proc/sys/sunrpc/nfsd_debug
it will turn on "SOCK" debugging and turn the others off.
If you
echo 1023 > /proc/sys/sunrpc/nfsd_debug
it will turn on all debugging.
Each file in fs/nfsd/*.c has something like
#define NFSDDBG_FACILITY NFSDDBG_FILEOP
so all dprintk calls in that file will use that facility and
will only be printed if the right number has been written to
..../nfsd_debug
>
> Also, is there any known or performance or stability hit to running with
> NFS debug on?
It will generate lots and lots (and lots) of messages which will
definately hurt performance, just scrolling the console if nothing
else.
I only turn it on for a few seconds at a time, or when I am
generating a very small amount of carefully controlled traffic.
NeilBrown
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs