Sorry, I'm sending it here again, since my first mail, in which I CCed LKML
and Andi Kleen was refused by the listserver.
Hi,
I'm just looking into a very strange problem. Some of our systems have
athlon64 CPUs. Due to our diskless nfs environment we currently still prefer
a 32bit userspace environment, but would like to be able to use a 64-bit
chroot environment.
Well, currently there seems to be a stat64() NFS problem when a x86_64 kernel
is booted and stat64() comes from a 32bit libc.
Here's just an example:
hitchcock:/home/bernd/src/tests# ./test_stat64 /mnt/test/yp
stat() works fine.
hitchcock:/home/bernd/src/tests# ./test_stat32 /mnt/test/yp
stat for /mnt/test/yp failed
The test program looks rather simple:
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>
#include <string.h>
#include <stdlib.h>
int main(int argc, char **argv)
{
char *dir;
struct stat buf;
dir = argv[1];
if (stat (dir, &buf) == -1)
fprintf(stderr, "stat for %s failed \n", dir);
else
fprintf(stderr, "stat() works fine.\n");
return (0);
}
Here are the strace outputs:
=====================
32bit:
------
hitchcock:/home/bernd/src/tests# strace32 ./test_stat32 /mnt/test/yp
execve("./test_stat32", ["./test_stat32", "/mnt/test/yp"], [/* 39 vars */]) =
0
uname({sys="Linux", node="hitchcock", ...}) = 0
brk(0) = 0x80ad000
brk(0x80ce000) = 0x80ce000
stat64("/mnt/test/yp", {st_mode=S_IFDIR|0755, st_size=2704, ...}) = 0
write(2, "stat for /mnt/test/yp failed \n", 30stat for /mnt/test/yp failed
) = 30
exit_group(0) = ?
64bit:
-------
hitchcock:/home/bernd/src/tests# strace ./test_stat64 /mnt/test/yp
execve("./test_stat64", ["./test_stat64", "/mnt/test/yp"], [/* 39 vars */]) =
0
uname({sys="Linux", node="hitchcock", ...}) = 0
brk(0) = 0x572000
brk(0x593000) = 0x593000
stat("/mnt/test/yp", {st_mode=S_IFDIR|0755, st_size=2704, ...}) = 0
write(2, "stat() works fine.\n", 19stat() works fine.
) = 19
_exit(0) = ?
Anyone having an idea whats going on? The ethereal capture also looks pretty
normal. The kernel of this system is 2.6.9, but it also happens on another
system with 2.6.11-rc5.
As usual we are using unfs3 for /etc and /var, but for me that looks like a
client problem. I'm now also pretty sure that its not a problem for a local
mount point (so a local disk).
Thanks in advance,
Bernd
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Given that strace shows that both syscalls (stat64() and stat())
succeed, I expect the "problem" is probably just glibc setting an
EOVERFLOW error in the 32-bit case. That's what it is supposed to do if
a 64 bit value overflows the 32-bit buffers.
Have you tried looking at errno?
Cheers,
Trond
m=E5 den 28.02.2005 Klokka 22:08 (+0100) skreiv Bernd Schubert:
> Sorry, I'm sending it here again, since my first mail, in which I CCed LK=
ML=20
> and Andi Kleen was refused by the listserver.
>=20
>=20
> Hi,
>=20
> I'm just looking into a very strange problem. Some of our systems have=20
> athlon64 CPUs. Due to our diskless nfs environment we currently still pre=
fer=20
> a 32bit userspace environment, but would like to be able to use a 64-bit=20
> chroot environment.
>=20
> Well, currently there seems to be a stat64() NFS problem when a x86_64 k=
ernel=20
> is booted and stat64() comes from a 32bit libc.
>=20
> Here's just an example:
>=20
> hitchcock:/home/bernd/src/tests# ./test_stat64 /mnt/test/yp
> stat() works fine.
>=20
>=20
> hitchcock:/home/bernd/src/tests# ./test_stat32 /mnt/test/yp
> stat for /mnt/test/yp failed=20
>=20
>=20
> The test program looks rather simple:
>=20
> #include <sys/types.h>
> #include <sys/stat.h>
> #include <unistd.h>
> #include <stdio.h>
> #include <errno.h>
> #include <string.h>
> #include <stdlib.h>
>=20
>=20
> int main(int argc, char **argv)
> {
> char *dir;
> struct stat buf;
>=20
> dir =3D argv[1];
>=20
> if (stat (dir, &buf) =3D=3D -1)
> fprintf(stderr, "stat for %s failed \n", dir);
> else
> fprintf(stderr, "stat() works fine.\n");
> return (0);
> }
>=20
>=20
> Here are the strace outputs:
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>=20
> 32bit:
> ------
> hitchcock:/home/bernd/src/tests# strace32 ./test_stat32 /mnt/test/yp
> execve("./test_stat32", ["./test_stat32", "/mnt/test/yp"], [/* 39 vars */=
]) =3D=20
> 0
> uname({sys=3D"Linux", node=3D"hitchcock", ...}) =3D 0
> brk(0) =3D 0x80ad000
> brk(0x80ce000) =3D 0x80ce000
> stat64("/mnt/test/yp", {st_mode=3DS_IFDIR|0755, st_size=3D2704, ...}) =3D=
0
> write(2, "stat for /mnt/test/yp failed \n", 30stat for /mnt/test/yp faile=
d=20
> ) =3D 30
> exit_group(0) =3D ?
>=20
> 64bit:
> -------
> hitchcock:/home/bernd/src/tests# strace ./test_stat64 /mnt/test/yp
> execve("./test_stat64", ["./test_stat64", "/mnt/test/yp"], [/* 39 vars */=
]) =3D=20
> 0
> uname({sys=3D"Linux", node=3D"hitchcock", ...}) =3D 0
> brk(0) =3D 0x572000
> brk(0x593000) =3D 0x593000
> stat("/mnt/test/yp", {st_mode=3DS_IFDIR|0755, st_size=3D2704, ...}) =3D 0
> write(2, "stat() works fine.\n", 19stat() works fine.
> ) =3D 19
> _exit(0) =3D ?
>=20
>=20
>=20
> Anyone having an idea whats going on? The ethereal capture also looks pre=
tty=20
> normal. The kernel of this system is 2.6.9, but it also happens on anothe=
r=20
> system with 2.6.11-rc5.
> As usual we are using unfs3 for /etc and /var, but for me that looks like=
a=20
> client problem. I'm now also pretty sure that its not a problem for a loc=
al=20
> mount point (so a local disk).
>=20
>=20
> Thanks in advance,
> Bernd
>=20
>=20
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=3D6595&alloc_id=3D14396&op=3Dclick
> _______________________________________________
> NFS maillist - [email protected]
> https://lists.sourceforge.net/lists/listinfo/nfs
--=20
Trond Myklebust <[email protected]>
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Tue, Mar 01, 2005 at 10:07:01PM +0100, Bernd Schubert wrote:
> Hello Andi,
>
> sorry, due to some mail sending/refusing problems, I had to resend to the
> nfs-list, which prevented the answers there to be posted to the other CCs.
>
> > It is most likely some kind of user space problem. I would change
> > it to int err = stat(dir, &buf);
> > and then go through it with gdb and see what value err gets assigned.
> >
> > I cannot see any kernel problem.
>
> The err value will become -1 here.
strace didn't say so, and normally it doesn't lie about things like this.
> > bernd@hitchcock tests>./test_stat32 /mnt/test/yp
> > stat for /mnt/test/yp failed
> > ernno: 75 (Value too large for defined data type)
errno is undefined unless a system call returned -1 before or
you set it to 0 before.
> > But why does stat64() on a 64-bit kernel tries to fill in larger data than
A 64bit kernel has no stat64(). All stats are 64bit.
> > on a 32-bit kernel and larger data also only for nfs-mount points? Hmm, I
> > will tomorrow compare the tcp-packges sent by the server.
>
> So I still think thats a kernel bug.
Your data so far doesn't support this assertion.
-Andi
Bernd Schubert <[email protected]> writes:
>> It is most likely some kind of user space problem. I would change
>> it to int err =3D stat(dir, &buf);
>> and then go through it with gdb and see what value err gets assigned=
=2E
>>
>> I cannot see any kernel problem.
>
> The err value will become -1 here.
That's because there are some values in the stat64 buffer delivered by =
the
kernel which cannot be packed into the stat buffer that you pass to sta=
t.
Use stat64 or _FILE_OFFSET_BITS=3D64.
> Trond Myklebust already suggested to look at the results of errno:
>
> On Tuesday 01 March 2005 00:43, Bernd Schubert wrote:
>> On Monday 28 February 2005 23:26, you wrote:
>> > Given that strace shows that both syscalls (stat64() and stat())
>> > succeed,
The trace does not say anything about the user-level stat().
>> bernd@hitchcock tests>./test_stat32 /mnt/test/yp
>> stat for /mnt/test/yp failed
>> ernno: 75 (Value too large for defined data type)
>>
>> But why does stat64() on a 64-bit kernel tries to fill in larger dat=
a than
>> on a 32-bit kernel and larger data also only for nfs-mount points? H=
mm, I
>> will tomorrow compare the tcp-packges sent by the server.
>
> So I still think thats a kernel bug.
This has nothing to do with the kernel.
Andreas.
--=20
Andreas Schwab, SuSE Labs, [email protected]
SuSE Linux Products GmbH, Maxfeldstra=DFe 5, 90409 N=FCrnberg, Germany
Key fingerprint =3D 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."
On Tue, Mar 01, 2005 at 11:10:38PM +0100, Andreas Schwab wrote:
> That's because there are some values in the stat64 buffer delivered by the
> kernel which cannot be packed into the stat buffer that you pass to stat.
> Use stat64 or _FILE_OFFSET_BITS=64.
If that had been the case strace would have reported EOVERFLOW
or E2BIG. But it returned 0 according to the log that was posted.
-Andi
> strace didn't say so, and normally it doesn't lie about things like t=
his.
Well, I show you the updated source code and strace output and if you s=
till=20
don't believe me, ask me for a login to our system ;)
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>
#include <string.h>
#include <stdlib.h>
int main(int argc, char **argv)
{
char *dir;
struct stat *buf;
int err;
dir =3D argv[1];
buf =3D malloc(sizeof(struct stat));
errno =3D 0;
err =3D stat(dir, buf);
if ( err ) {
fprintf(stderr, "err =3D %i\n", err);
fprintf(stderr, "stat for %s failed \n", dir);
fprintf(stderr, "ernno: %i (%s)\n", errno, strerror(err=
no));
} else
fprintf(stderr, "stat() works fine.\n");
return (0);
}
>
> > > bernd@hitchcock tests>./test_stat32 /mnt/test/yp
> > > stat for /mnt/test/yp failed
> > > ernno: 75 (Value too large for defined data type)
>
> errno is undefined unless a system call returned -1 before or
> you set it to 0 before.
See above.
>
> > > But why does stat64() on a 64-bit kernel tries to fill in larger =
data
> > > than
>
> A 64bit kernel has no stat64(). All stats are 64bit.
bernd@hitchcock tests>strace32 ./test_stat32 /mnt/test/yp
execve("./test_stat32", ["./test_stat32", "/mnt/test/yp"], [/* 43 vars =
*/]) =3D=20
0
uname({sys=3D"Linux", node=3D"hitchcock", ...}) =3D 0
brk(0) =3D 0x80ad000
brk(0x80ce000) =3D 0x80ce000
stat64("/mnt/test/yp", {st_mode=3DS_IFDIR|0755, st_size=3D2704, ...}) =3D=
0
write(2, "err =3D -1\n", 9err =3D -1
) =3D 9
write(2, "stat for /mnt/test/yp failed \n", 30stat for /mnt/test/yp fai=
led
) =3D 30
write(2, "ernno: 75 (Value too large for d"..., 50ernno: 75 (Value too =
large=20
for defined data type)
) =3D 50
exit_group(0) =3D ?
You certainly know much better than me, but I think strace shows that i=
ts=20
calling stat64.
>
> > > on a 32-bit kernel and larger data also only for nfs-mount points=
? Hmm,
> > > I will tomorrow compare the tcp-packges sent by the server.
> >
> > So I still think thats a kernel bug.
>
> Your data so far doesn't support this assertion.
I have to admit that knfsd-mount moints are not affected, but on the ot=
her=20
hand, I really cant't see anything in the ethereal captures. If someone=
=20
should be interested, I have uploaded them:
http://www.pci.uni-heidelberg.de/tc/usr/bernd/downloads/nfs-stat/
Cheers,
Bernd
--=20
Bernd Schubert
Physikalisch Chemisches Institut / Theoretische Chemie
Universit=E4t Heidelberg
INF 229
69120 Heidelberg
e-mail: [email protected]
> stat64("/mnt/test/yp", {st_mode=S_IFDIR|0755, st_size=2704, ...}) = 0
It returns 0. No error. Someone else in user space must be adding the EOVERFLOW.
glibc code does quite a lot of strange things with stat, perhaps
it comes from there.
> write(2, "err = -1\n", 9err = -1
> ) = 9
> write(2, "stat for /mnt/test/yp failed \n", 30stat for /mnt/test/yp failed
> ) = 30
> write(2, "ernno: 75 (Value too large for d"..., 50ernno: 75 (Value too large
> for defined data type)
> ) = 50
> exit_group(0) = ?
-Andi
On Tuesday 01 March 2005 23:10, Andreas Schwab wrote:
> Bernd Schubert <[email protected]> writes:
> >> It is most likely some kind of user space problem. I would change
> >> it to int err = stat(dir, &buf);
> >> and then go through it with gdb and see what value err gets assigned.
> >>
> >> I cannot see any kernel problem.
> >
> > The err value will become -1 here.
>
> That's because there are some values in the stat64 buffer delivered by the
> kernel which cannot be packed into the stat buffer that you pass to stat.
> Use stat64 or _FILE_OFFSET_BITS=64.
Hmm, after compiling with -D_FILE_OFFSET_BITS=64 it works fine. But why does
it work without this option on a 32bit kernel, but not on a 64bit kernel?
32bit kernel, 32bit binary: always works
64bit kernel, 64bit binary: always works
64bit kernel, 32bit binary:
- always works on knfsd mount points
- always works with -D_FILE_OFFSET_BITS=64
- only works on unfs3 mount points with _FILE_OFFSET_BITS=64
Do I really have to write a bug report for every single debian package that
access /etc and /var to make the maintainers recompile it with
-D_FILE_OFFSET_BITS=64?
Btw, whats about Suse, are there all packages compiled with this option? ;)
Cheers,
(a completely confused) Bernd
Andi Kleen <[email protected]> writes:
> On Tue, Mar 01, 2005 at 11:10:38PM +0100, Andreas Schwab wrote:
>> That's because there are some values in the stat64 buffer delivered =
by the
>> kernel which cannot be packed into the stat buffer that you pass to =
stat.
>> Use stat64 or _FILE_OFFSET_BITS=3D64.
>
> If that had been the case strace would have reported EOVERFLOW
> or E2BIG.
No, the values are ok for stat64.
Andreas.
--=20
Andreas Schwab, SuSE Labs, [email protected]
SuSE Linux Products GmbH, Maxfeldstra=DFe 5, 90409 N=FCrnberg, Germany
Key fingerprint =3D 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."
Bernd Schubert <[email protected]> writes:
> Hmm, after compiling with -D_FILE_OFFSET_BITS=3D64 it works fine. But=
why does=20
> it work without this option on a 32bit kernel, but not on a 64bit ker=
nel?
Most likely the inode number (which is the only non-filesize related it=
em
that is different between struct stat and struct stat64) overflows ino_=
t.
Andreas.
--=20
Andreas Schwab, SuSE Labs, [email protected]
SuSE Linux Products GmbH, Maxfeldstra=DFe 5, 90409 N=FCrnberg, Germany
Key fingerprint =3D 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."
Bernd Schubert <[email protected]> writes:
> Hmm, after compiling with -D_FILE_OFFSET_BITS=3D64 it works fine. But=
why does=20
> it work without this option on a 32bit kernel, but not on a 64bit ker=
nel?
See nfs_fileid_to_ino_t for why the inode number is different between
32bit and 64bit kernels.
Andreas.
--=20
Andreas Schwab, SuSE Labs, [email protected]
SuSE Linux Products GmbH, Maxfeldstra=DFe 5, 90409 N=FCrnberg, Germany
Key fingerprint =3D 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5
"And now for something completely different."
On Monday 28 February 2005 23:26, you wrote:
> Given that strace shows that both syscalls (stat64() and stat())
> succeed, I expect the "problem" is probably just glibc setting an
> EOVERFLOW error in the 32-bit case. That's what it is supposed to do if
> a 64 bit value overflows the 32-bit buffers.
Right, thanks.
>
> Have you tried looking at errno?
>
bernd@hitchcock tests>./test_stat32 /mnt/test/yp
stat for /mnt/test/yp failed
ernno: 75 (Value too large for defined data type)
But why does stat64() on a 64-bit kernel tries to fill in larger data than on
a 32-bit kernel and larger data also only for nfs-mount points? Hmm, I will
tomorrow compare the tcp-packges sent by the server.
Many thanks,
Bernd
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Monday 28 February 2005 23:26, you wrote:
> Given that strace shows that both syscalls (stat64() and stat())
> succeed, I expect the "problem" is probably just glibc setting an
> EOVERFLOW error in the 32-bit case. That's what it is supposed to do if
> a 64 bit value overflows the 32-bit buffers.
Right, thanks.
>
> Have you tried looking at errno?
>
bernd@hitchcock tests>./test_stat32 /mnt/test/yp
stat for /mnt/test/yp failed
ernno: 75 (Value too large for defined data type)
But why does stat64() on a 64-bit kernel tries to fill in larger data than on
a 32-bit kernel and larger data also only for nfs-mount points? Hmm, I will
tomorrow compare the tcp-packges sent by the server.
Many thanks,
Bernd
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Sounds like a good time to try out Xen or similar:
http://dcs.nac.uci.edu/~strombrg/multios.html
On Mon, 2005-02-28 at 22:08 +0100, Bernd Schubert wrote:
> Sorry, I'm sending it here again, since my first mail, in which I CCed LKML
> and Andi Kleen was refused by the listserver.
>
>
> Hi,
>
> I'm just looking into a very strange problem. Some of our systems have
> athlon64 CPUs. Due to our diskless nfs environment we currently still prefer
> a 32bit userspace environment, but would like to be able to use a 64-bit
> chroot environment.
>
> Well, currently there seems to be a stat64() NFS problem when a x86_64 kernel
> is booted and stat64() comes from a 32bit libc.
>
> Here's just an example:
>
> hitchcock:/home/bernd/src/tests# ./test_stat64 /mnt/test/yp
> stat() works fine.
>
>
> hitchcock:/home/bernd/src/tests# ./test_stat32 /mnt/test/yp
> stat for /mnt/test/yp failed
>
>
> The test program looks rather simple:
>
> #include <sys/types.h>
> #include <sys/stat.h>
> #include <unistd.h>
> #include <stdio.h>
> #include <errno.h>
> #include <string.h>
> #include <stdlib.h>
>
>
> int main(int argc, char **argv)
> {
> char *dir;
> struct stat buf;
>
> dir = argv[1];
>
> if (stat (dir, &buf) == -1)
> fprintf(stderr, "stat for %s failed \n", dir);
> else
> fprintf(stderr, "stat() works fine.\n");
> return (0);
> }
>
>
> Here are the strace outputs:
> =====================
>
> 32bit:
> ------
> hitchcock:/home/bernd/src/tests# strace32 ./test_stat32 /mnt/test/yp
> execve("./test_stat32", ["./test_stat32", "/mnt/test/yp"], [/* 39 vars */]) =
> 0
> uname({sys="Linux", node="hitchcock", ...}) = 0
> brk(0) = 0x80ad000
> brk(0x80ce000) = 0x80ce000
> stat64("/mnt/test/yp", {st_mode=S_IFDIR|0755, st_size=2704, ...}) = 0
> write(2, "stat for /mnt/test/yp failed \n", 30stat for /mnt/test/yp failed
> ) = 30
> exit_group(0) = ?
>
> 64bit:
> -------
> hitchcock:/home/bernd/src/tests# strace ./test_stat64 /mnt/test/yp
> execve("./test_stat64", ["./test_stat64", "/mnt/test/yp"], [/* 39 vars */]) =
> 0
> uname({sys="Linux", node="hitchcock", ...}) = 0
> brk(0) = 0x572000
> brk(0x593000) = 0x593000
> stat("/mnt/test/yp", {st_mode=S_IFDIR|0755, st_size=2704, ...}) = 0
> write(2, "stat() works fine.\n", 19stat() works fine.
> ) = 19
> _exit(0) = ?
>
>
>
> Anyone having an idea whats going on? The ethereal capture also looks pretty
> normal. The kernel of this system is 2.6.9, but it also happens on another
> system with 2.6.11-rc5.
> As usual we are using unfs3 for /etc and /var, but for me that looks like a
> client problem. I'm now also pretty sure that its not a problem for a local
> mount point (so a local disk).
>
>
> Thanks in advance,
> Bernd
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> NFS maillist - [email protected]
> https://lists.sourceforge.net/lists/listinfo/nfs
>