2005-06-04 11:41:18

by Christoph Bartoschek

[permalink] [raw]
Subject: Unknown error 525 from getdents from a connection to an AIX nfs server

Hi,

I have the problem that rm does not work reliable over the nfs connection to
our AIX nfs server. If one deletes a whole directoy recursively one gets an
"Unknown error 525". All files but the directory itself are deleted.

The error can be reproduced as follows:

bartosch@feiler:/fs/data/bartosch/temp> mkdir testdir
bartosch@feiler:/fs/data/bartosch/temp> touch testdir/a
bartosch@feiler:/fs/data/bartosch/temp> rm -rf testdir
rm: reading directory `testdir': Unknown error 525
bartosch@feiler:/fs/data/bartosch/temp> rm -rf testdir
bartosch@feiler:/fs/data/bartosch/temp>

strace says that the error message comes from getdents.

This is the output of strace on rm showing the important parts:

..
lstat(".", {st_mode=S_IFDIR|S_ISGID|0755, st_size=288, ...}) = 0
open(".", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|S_ISGID|0755, st_size=288, ...}) = 0
fcntl(4, F_SETFD, FD_CLOEXEC) = 0
getdents64(4, /* 3 entries */, 512) = 72
unlink("a") = 0
getdents64(4, 0x50e1c8, 512) = -1 ETOOSMALL (Unknown error 525)
close(4) = 0
open("/usr/share/locale/en_US/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT
(No such file or directory)
open("/usr/share/locale/en/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT
(No such file or directory)
write(2, "rm: ", 4rm: ) = 4
write(2, "reading directory `testdir\'", 27reading directory `testdir') = 27
...
write(2, ": Unknown error 525", 19: Unknown error 525) = 19
...

You can see that getdents returns ETOOSMALL which is the error.

The client machine feiler is has an AMD opteron 64bit CPU runnning Linux
kernel:

Linux feiler 2.4.21-286-smp #1 SMP Sun Apr 3 05:18:39 UTC 2005 x86_64 x86_64
x86_64 GNU/Linux

We get the same error from a machine with a newer kernel:

Linux maximator 2.6.4-52-smp #1 SMP Wed Apr 7 01:58:54 UTC 2004 x86_64 x86_64
x86_64 GNU/Linux

Or one with a very new one:

Linux castell.vlsi.uni-bonn.de 2.6.11-APIC-ID #1 SMP Tue May 24 11:31:43 CEST
2005 x86_64 x86_64 x86_64 GNU/Linux

Even on 32bit machines this error occurs

Linux inzadi-en0 2.4.21-231-smp #1 SMP Mon Jun 28 15:04:14 UTC 2004 i686
unknown

The Server is an AIX 4.3.3 machine: AIX cook 3 4 0045B0AA4C00


How can I get rid of this problem? If you need more information I will
provide it for you.

Greets,
Christoph Bartoschek


-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games. How far can you shotput
a projector? How fast can you ride your desk chair down the office luge track?
If you want to score the big prize, get to know the little guy.
Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs


2005-06-04 15:19:31

by Trond Myklebust

[permalink] [raw]
Subject: Re: Unknown error 525 from getdents from a connection to an AIX nfs server

lau den 04.06.2005 Klokka 13:41 (+0200) skreiv Christoph Bartoschek:
> Hi,
>
> I have the problem that rm does not work reliable over the nfs connection to
> our AIX nfs server. If one deletes a whole directoy recursively one gets an
> "Unknown error 525". All files but the directory itself are deleted.
>
> The error can be reproduced as follows:
>
> bartosch@feiler:/fs/data/bartosch/temp> mkdir testdir
> bartosch@feiler:/fs/data/bartosch/temp> touch testdir/a
> bartosch@feiler:/fs/data/bartosch/temp> rm -rf testdir
> rm: reading directory `testdir': Unknown error 525
> bartosch@feiler:/fs/data/bartosch/temp> rm -rf testdir
> bartosch@feiler:/fs/data/bartosch/temp>
>
> strace says that the error message comes from getdents.

Sounds like a server bug. 525 is the translations of NFSERR_TOOSMALL,
which means "buffer is too small".
i.e. the server is claiming that it suddenly is incapable of fitting the
result of the readdir operation into the same buffer that apparently had
no problems fitting the 3 entry case.

Cheers,
Trond



-------------------------------------------------------
This SF.Net email is sponsored by: NEC IT Guy Games. How far can you shotput
a projector? How fast can you ride your desk chair down the office luge track?
If you want to score the big prize, get to know the little guy.
Play to win an NEC 61" plasma display: http://www.necitguy.com/?r=20
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2005-06-21 19:50:18

by Christoph Bartoschek

[permalink] [raw]
Subject: Re: Unknown error 525 from getdents from a connection to an AIX nfs server

Hi,

we have updated our NFS Server to AIX 5.2. Now the problems occur only on the
clients with 64bit processors. Clients on 32bit machines run fine. (However,
maybe I made some mistakes when I checked the 32bit machines for the last
post).

Are there any issues known for the 64bit Version of the NFS Client?

Christoph Bartoschek


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2005-06-21 20:05:05

by Trond Myklebust

[permalink] [raw]
Subject: Re: Re: Unknown error 525 from getdents from a connection to an AIX nfs server

ty den 21.06.2005 Klokka 14:33 (+0200) skreiv Christoph Bartoschek:
> Hi,
>
> we have updated our NFS Server to AIX 5.2. Now the problems occur only on the
> clients with 64bit processors. Clients on 32bit machines run fine. (However,
> maybe I made some mistakes when I checked the 32bit machines for the last
> post).

Remind me please. Is this error appearing on the wire? i.e. do you see a
readdir request coming across and an error being returned when you use
tcpdump?

Cheers,
Trond



-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

2005-06-22 09:24:35

by Christoph Bartoschek

[permalink] [raw]
Subject: Re: Re: Unknown error 525 from getdents from a connection to an AIX nfs server

Am Dienstag 21 Juni 2005 22:04 schrieb Trond Myklebust:
> ty den 21.06.2005 Klokka 14:33 (+0200) skreiv Christoph Bartoschek:
> > Hi,
> >
> > we have updated our NFS Server to AIX 5.2. Now the problems occur only on
> > the clients with 64bit processors. Clients on 32bit machines run fine.
> > (However, maybe I made some mistakes when I checked the 32bit machines
> > for the last post).
>
> Remind me please. Is this error appearing on the wire? i.e. do you see a
> readdir request coming across and an error being returned when you use
> tcpdump?


Hi,

I do not know how to read NFS traffic. Therefore I made an ethereal dump
during the rm -rf a in the following sequence:

mkdir a
touch a/b
rm -rf a

I hope you find some useful information. If you need other information, give
me short instruction on how to obtain it.

Christoph Bartoschek


Attachments:
(No filename) (852.00 B)
remove-nfs.log (9.50 kB)
Download all attachments

2005-06-22 10:11:18

by Trond Myklebust

[permalink] [raw]
Subject: Re: Re: Unknown error 525 from getdents from a connection to an AIX nfs server

on den 22.06.2005 Klokka 11:24 (+0200) skreiv Christoph Bartoschek:

> I do not know how to read NFS traffic. Therefore I made an ethereal dump
> during the rm -rf a in the following sequence:
>
> mkdir a
> touch a/b
> rm -rf a
>
> I hope you find some useful information. If you need other information, give
> me short instruction on how to obtain it.

That appears to be a server bug.

The client is performing a READDIRPLUS operation using a cookie that
used to be valid but now refers to an entry that was deleted. Instead of
returning the correct error of NFS3ERR_BADCOOKIE, the server is
returning NFS3ERR_TOOSMALL.

Cheers,
Trond



-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs