Hi all,
I've got a "New and Exciting" <tm> bug using 2.4.18 with Trond's client
patches (sans the rpc_tweaks dif), but I'm not sure it's a client side
problem.
This is an intermitent problem: When attempting to read a file (any file)
on an NFS mounted volume, there appear to be "holes" in the file filled with
ASCII NULL characters. Reading the file on the local machines shows no such
holes, and other NFS clients don't see the holes, either. A simple
umount/remount solves the problem.
Oh, I should add that these are autofs mounted volumes.
We thought these might be network driver problems, but we're seeing it on
natsemi, acenic and eepro100 drivers, and I doubt they could all be flaky.
Any ideas?
Thanks,
Dan
--
Dan Yocum
Sloan Digital Sky Survey, Fermilab 630.840.6509
[email protected], http://www.sdss.org
SDSS. Mapping the Universe.
_______________________________________________________________
Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: [email protected]
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
>>>>> " " == Dan Yocum <[email protected]> writes:
> Hi all, I've got a "New and Exciting" <tm> bug using 2.4.18
> with Trond's client patches (sans the rpc_tweaks dif), but I'm
> not sure it's a client side problem.
> This is an intermitent problem: When attempting to read a file
> (any file) on an NFS mounted volume, there appear to be "holes"
> in the file filled with ASCII NULL characters. Reading the
> file on the local machines shows no such holes, and other NFS
> clients don't see the holes, either. A simple umount/remount
> solves the problem.
Any details?
Are these for instance ordinary files? Are you perhaps engaging in
some multiple NFS client read/writes to the same file? ... Is it tied
to the client patches, or does it occur with stock 2.4.18 fresh off
the l-k presses?
Cheers,
Trond
_______________________________________________________________
Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: [email protected]
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Trond,
Nothing special in /var/log/messages on either the client or server that
would help in debugging the problem, unfortunately.
I'm seeing the problem on these 2 client systems:
vanilla 2.4.18 + linux-2.4.18-NFS_ALL.dif - linux-2.4.18-rpc_tweaks.dif +
xfs-1.1-2.4.18-all.patch.
("-", above, meaning 'patch -R')
And:
kernel-2.4.9-13SGI_XFS_1.0.2, which is a Red Hat 2.4.9 kernel with XFS
v1.0.2 patches (built by SGI folks). I can get you the appropriate patches
from their patch set, if you want.
The server system is the 1st kernel described above.
The files are FITS files (text header tables, followed by binary data). The
data was written once to disk and is read by many clients.
If I could reproduce the problem at will, I'd try a generic kernel, but it's
intermittent at best, and we need the patches to access the IRIX servers.
:-(
When it happens again, I'll do some more debugging on the problem.
Thanks,
Dan
Trond Myklebust wrote:
>
> >>>>> " " == Dan Yocum <[email protected]> writes:
>
> > Hi all, I've got a "New and Exciting" <tm> bug using 2.4.18
> > with Trond's client patches (sans the rpc_tweaks dif), but I'm
> > not sure it's a client side problem.
>
> > This is an intermitent problem: When attempting to read a file
> > (any file) on an NFS mounted volume, there appear to be "holes"
> > in the file filled with ASCII NULL characters. Reading the
> > file on the local machines shows no such holes, and other NFS
> > clients don't see the holes, either. A simple umount/remount
> > solves the problem.
>
> Any details?
>
> Are these for instance ordinary files? Are you perhaps engaging in
> some multiple NFS client read/writes to the same file? ... Is it tied
> to the client patches, or does it occur with stock 2.4.18 fresh off
> the l-k presses?
>
> Cheers,
> Trond
--
Dan Yocum
Sloan Digital Sky Survey, Fermilab 630.840.6509
[email protected], http://www.sdss.org
SDSS. Mapping the Universe.
_______________________________________________________________
Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: [email protected]
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Monday May 13, [email protected] wrote:
> Hi all,
>
> I've got a "New and Exciting" <tm> bug using 2.4.18 with Trond's client
> patches (sans the rpc_tweaks dif), but I'm not sure it's a client side
> problem.
>
> This is an intermitent problem: When attempting to read a file (any file)
> on an NFS mounted volume, there appear to be "holes" in the file filled with
> ASCII NULL characters. Reading the file on the local machines shows no such
> holes, and other NFS clients don't see the holes, either. A simple
> umount/remount solves the problem.
I remembers long ago some referred to NFS as
Nulls Frequently Substituted
because they were experiencing exactly this sort of thing, and it
wasn't on Linux at all.
You wouldn't be using 'soft' mounts would you?
NeilBrown
_______________________________________________________________
Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: [email protected]
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
M=E5ndag 13. mai 2002 18:54 skreiv Dan Yocum:
> Trond,
>
> Nothing special in /var/log/messages on either the client or server tha=
t
> would help in debugging the problem, unfortunately.
>
> I'm seeing the problem on these 2 client systems:
>
> vanilla 2.4.18 + linux-2.4.18-NFS_ALL.dif - linux-2.4.18-rpc_tweaks.dif=
+
> xfs-1.1-2.4.18-all.patch.
>
> ("-", above, meaning 'patch -R')
>
> And:
>
> kernel-2.4.9-13SGI_XFS_1.0.2, which is a Red Hat 2.4.9 kernel with XFS
> v1.0.2 patches (built by SGI folks). I can get you the appropriate pat=
ches
> from their patch set, if you want.
>
> The server system is the 1st kernel described above.
Are you able to reproduce it on a non-XFS server?
Cheers,
Trond
_______________________________________________________________
Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: [email protected]
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Trond Myklebust wrote:
>
> Are you able to reproduce it on a non-XFS server?
Heh. I don't have any non-XFS machines. But, I'll post the problem to the
XFS lists - maybe someone there has seen it.
Cheers,
Dan
--
Dan Yocum
Sloan Digital Sky Survey, Fermilab 630.840.6509
[email protected], http://www.sdss.org
SDSS. Mapping the Universe.
_______________________________________________________________
Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: [email protected]
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Neil Brown wrote:
>
> On Monday May 13, [email protected] wrote:
> > Hi all,
> >
> > I've got a "New and Exciting" <tm> bug using 2.4.18 with Trond's client
> > patches (sans the rpc_tweaks dif), but I'm not sure it's a client side
> > problem.
> >
> > This is an intermitent problem: When attempting to read a file (any file)
> > on an NFS mounted volume, there appear to be "holes" in the file filled with
> > ASCII NULL characters. Reading the file on the local machines shows no such
> > holes, and other NFS clients don't see the holes, either. A simple
> > umount/remount solves the problem.
>
> I remembers long ago some referred to NFS as
> Nulls Frequently Substituted
> because they were experiencing exactly this sort of thing, and it
> wasn't on Linux at all.
Ha! :-)
I did find a similar occurence while googling
(http://www.geocrawler.com/archives/3/4/1998/11/0/59164/), reported back in
'98 on LinuxPPC, that produced similar results, but since this happened 3
times on the same day to 3 different machines, I thought I'd report it.
>
> You wouldn't be using 'soft' mounts would you?
yup, automounted with these options:
-rw,grpid,soft,intr,quota,nodev,nosuid,timeo=10,retrans=3
I suppose they could be hard.... I'll think about that.
Cheers,
Dan
--
Dan Yocum
Sloan Digital Sky Survey, Fermilab 630.840.6509
[email protected], http://www.sdss.org
SDSS. Mapping the Universe.
_______________________________________________________________
Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: [email protected]
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
On Tue, 14 May 2002 09:41:39 -0500, Dan Yocum <[email protected]> wrote:
>> > This is an intermitent problem: When attempting to read a file (any file)
>> > on an NFS mounted volume, there appear to be "holes" in the file filled with
>> > ASCII NULL characters. Reading the file on the local machines shows no such
>> > holes, and other NFS clients don't see the holes, either. A simple
>> > umount/remount solves the problem.
>>
> Neil Brown wrote:
>> You wouldn't be using 'soft' mounts would you?
>
> yup, automounted with these options:
>
> -rw,grpid,soft,intr,quota,nodev,nosuid,timeo=10,retrans=3
In that case let me ask you this: is your application checking the return
code of the read()? or are you using mmap() instead of read()?
If read() returns zero-filled buffers without signalling an errors, it's
a problem in the NFS client. That is, assumming it's not some bad interaction
between XFS and the NFS server -- so try to clear up that possibility
by mounting with the hard option.
If you're using mmap(), then I don't even know how the system should tell
you about the problem... I suppose it would be sending a SIGSEGV or SIGBUS
signal, since it's ultimately a failing page fault, but I haven't tested it.
> I suppose they could be hard.... I'll think about that.
Yes, they _should_ be hard if you care about your data integrity.
But that shouldn't stop us from fixing bugs in the 'soft' model, if they
exist... which is why I'm asking you all these details. :-)
Ion
--
It is better to keep your mouth shut and be thought a fool,
than to open it and remove all doubt.
_______________________________________________________________
Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: [email protected]
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Ion,
The apps are apache and plain old 'less,' so I bet they just use read().
And the problem hasn't cropped up again (yet) and hadn't cropped up in ~1
year with older kernels (specifically 2.4.5). I just upgraded to 2.4.18
last Thursday when it did start.
I'll change the options to hard and see if that makes any changes.
Thanks,
Dan
Ion Badulescu wrote:
>
> On Tue, 14 May 2002 09:41:39 -0500, Dan Yocum <[email protected]> wrote:
> >> > This is an intermitent problem: When attempting to read a file (any file)
> >> > on an NFS mounted volume, there appear to be "holes" in the file filled with
> >> > ASCII NULL characters. Reading the file on the local machines shows no such
> >> > holes, and other NFS clients don't see the holes, either. A simple
> >> > umount/remount solves the problem.
> >>
> > Neil Brown wrote:
> >> You wouldn't be using 'soft' mounts would you?
> >
> > yup, automounted with these options:
> >
> > -rw,grpid,soft,intr,quota,nodev,nosuid,timeo=10,retrans=3
>
> In that case let me ask you this: is your application checking the return
> code of the read()? or are you using mmap() instead of read()?
>
> If read() returns zero-filled buffers without signalling an errors, it's
> a problem in the NFS client. That is, assumming it's not some bad interaction
> between XFS and the NFS server -- so try to clear up that possibility
> by mounting with the hard option.
>
> If you're using mmap(), then I don't even know how the system should tell
> you about the problem... I suppose it would be sending a SIGSEGV or SIGBUS
> signal, since it's ultimately a failing page fault, but I haven't tested it.
>
> > I suppose they could be hard.... I'll think about that.
>
> Yes, they _should_ be hard if you care about your data integrity.
>
> But that shouldn't stop us from fixing bugs in the 'soft' model, if they
> exist... which is why I'm asking you all these details. :-)
>
> Ion
>
> --
> It is better to keep your mouth shut and be thought a fool,
> than to open it and remove all doubt.
--
Dan Yocum
Sloan Digital Sky Survey, Fermilab 630.840.6509
[email protected], http://www.sdss.org
SDSS. Mapping the Universe.
_______________________________________________________________
Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: [email protected]
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs