LinuxLists.cc - nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

[permalink] [raw]

Subject: Re: nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

ty den 02.11.2004 Klokka 16:02 (+0400) skreiv Brad Campbell:

> /raid 192.168.2.81(rw,async,no_root_squash)
> /raid 192.168.3.80(rw,async,no_root_squash)
> /raid0 192.168.2.81(rw,async,no_root_squash)
> /raid0/tmp 192.168.2.81(rw,async,no_root_squash)
> /raid2 192.168.2.81(rw,async,no_root_squash)
> /raid2 192.168.3.80(rw,async,no_root_squash)
> /nfsroot 192.168.2.81(rw,async,no_root_squash)

You should only have 1 line per directory.

You are not allowed to export a directory and its child (unless the
child is on a different partition - which does not appear to be the case
here).

http://nfs.sourceforge.net/nfs-howto/server.html#CONFIG

Cheers,
Trond

--
Trond Myklebust <[email protected]>

2004-11-02 19:51:05

[permalink] [raw]

Subject: Re: nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

Trond Myklebust wrote:
> ty den 02.11.2004 Klokka 16:02 (+0400) skreiv Brad Campbell:
>
>
>>/raid 192.168.2.81(rw,async,no_root_squash)
>>/raid 192.168.3.80(rw,async,no_root_squash)
>>/raid0 192.168.2.81(rw,async,no_root_squash)
>>/raid0/tmp 192.168.2.81(rw,async,no_root_squash)
>>/raid2 192.168.2.81(rw,async,no_root_squash)
>>/raid2 192.168.3.80(rw,async,no_root_squash)
>>/nfsroot 192.168.2.81(rw,async,no_root_squash)
>
>
> You should only have 1 line per directory.
>
> You are not allowed to export a directory and its child (unless the
> child is on a different partition - which does not appear to be the case
> here).
>
> http://nfs.sourceforge.net/nfs-howto/server.html#CONFIG

I'm also seeing stale filehandle problems here in recent kernels.

Setup: x86 or x86-64, TCP, NFSv4 compiled in to both server and client,
but not specified in mount options.

This is readily reproducible with rsync -- I just boot to an earlier
version of the kernel on the NFS client, and the stale filehandle
problems go away.

JEff

2004-11-02 20:11:20

by Jakob Oestergaard

[permalink] [raw]

Subject: Re: nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

On Tue, Nov 02, 2004 at 02:49:53PM -0500, Jeff Garzik wrote:
> Trond Myklebust wrote:
...
> > http://nfs.sourceforge.net/nfs-howto/server.html#CONFIG
>
>
> I'm also seeing stale filehandle problems here in recent kernels.
>
> Setup: x86 or x86-64, TCP, NFSv4 compiled in to both server and client,
> but not specified in mount options.
>
> This is readily reproducible with rsync -- I just boot to an earlier
> version of the kernel on the NFS client, and the stale filehandle
> problems go away.

Hi Jeff,

Does running an 'ls' on the server in the exported directory that is
stale on the client resolve the problem (temporarily)?

(working with really weird stale handle theories here I admit...)

--

/ jakob

2004-11-02 20:43:03

[permalink] [raw]

Subject: Re: nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

Jeff Garzik wrote:

>
> I'm also seeing stale filehandle problems here in recent kernels.
>
> Setup: x86 or x86-64, TCP, NFSv4 compiled in to both server and
> client, but not specified in mount options.
>
> This is readily reproducible with rsync -- I just boot to an earlier
> version of the kernel on the NFS client, and the stale filehandle
> problems go away.
>
> JEff
>
>
I am seeing severe nfs problems between kernels again, 2.4 - > 2.6 ->
2.4 V3 and V4 problems connecting and filesize mismatches between
actual/reported. Might point to where the bug is.

Jeff

2004-11-02 21:39:28

[permalink] [raw]

Subject: Re: nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

ty den 02.11.2004 Klokka 14:05 (-0700) skreiv Jeff V. Merkey:
> >
> I am seeing severe nfs problems between kernels again, 2.4 - > 2.6 ->
> 2.4 V3 and V4 problems connecting and filesize mismatches between
> actual/reported. Might point to where the bug is.

Nope. I'm not seeing that at all (besides, that is entirely unrelated to
ESTALE errors).

Mind telling us how to reproduce the problem?

Cheers,
Trond

--
Trond Myklebust <[email protected]>

2004-11-02 21:40:20

[permalink] [raw]

Subject: Re: nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

ty den 02.11.2004 Klokka 14:49 (-0500) skreiv Jeff Garzik:

> This is readily reproducible with rsync -- I just boot to an earlier
> version of the kernel on the NFS client, and the stale filehandle
> problems go away.

Huh? The client cannot generate stale filehandle errors: only the server
does that.
Have you got a binary tcpdump that shows the problem?

Cheers
Trond

--
Trond Myklebust <[email protected]>

2004-11-02 21:52:17

by J. Bruce Fields

[permalink] [raw]

Subject: Re: nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

On Tue, Nov 02, 2004 at 02:49:53PM -0500, Jeff Garzik wrote:
> >ty den 02.11.2004 Klokka 16:02 (+0400) skreiv Brad Campbell:
> >
> >
> >>/raid 192.168.2.81(rw,async,no_root_squash)
> >>/raid 192.168.3.80(rw,async,no_root_squash)
> >>/raid0 192.168.2.81(rw,async,no_root_squash)
> >>/raid0/tmp 192.168.2.81(rw,async,no_root_squash)
> >>/raid2 192.168.2.81(rw,async,no_root_squash)
> >>/raid2 192.168.3.80(rw,async,no_root_squash)
> >>/nfsroot 192.168.2.81(rw,async,no_root_squash)
....
> I'm also seeing stale filehandle problems here in recent kernels.
>
> Setup: x86 or x86-64, TCP, NFSv4 compiled in to both server and client,
> but not specified in mount options.
>
> This is readily reproducible with rsync -- I just boot to an earlier
> version of the kernel on the NFS client, and the stale filehandle
> problems go away.

Are any of the people seeing these problems able to reproduce them with
the no_subtree_check export option set?

--b.

2004-11-02 23:57:59

[permalink] [raw]

Subject: Re: nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

[email protected] wrote:

>>Nope. I'm not seeing that at all (besides, that is entirely unrelated to
>>ESTALE errors).
>>
>>Mind telling us how to reproduce the problem?
>>
>>Cheers,
>> Trond
>>
>>--
>>Trond Myklebust <[email protected]>
>>
>>
>
>Connect 2.4.18 and 2.6.9 with NFS 3 enabled. I am seeing problems
>connecting and file size mismatches. I also see errors with zero
>length files (host side) that get opened and populated with data
>and the remote side is unable to read them -- keeps seeing
>them as zero length.
>
>I can setup this back up tommorrow morning in the lab and provide
>you all sorts of good debug and trace info. The problem seems to
>happen if the local side from the FS has open handles and starts
>writing blocks to a zero length file, and the remote side doesn't
>seem to see the data right away.
>
>On typical FS read behavior, even if the file has a zero length, this
>seems to be ignored and read requests continue until read returns
>0 bytes read for file size. On the remote side, if the file size
>is not reflected, this is not the behavior, and probably should
>be. I can see someone truncating or changing a file size and NFS
>gets into a weird state.
>
>Mismatches between configured v2/v3/v4 NFS configs between kernels
>also have some issues. Please provide me with a sample /etc/exports
>config you wish me to use exporting a directory you think would be
>helpful and I'll test and provide pcap traces of the traffic between
>the nodes.
>
>Jeff
>
>
>
>
>

2004-11-03 01:15:51

[permalink] [raw]

Subject: Re: nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

ty den 02.11.2004 Klokka 17:19 (-0700) skreiv Jeff V. Merkey:

> >Connect 2.4.18 and 2.6.9 with NFS 3 enabled. I am seeing problems
> >connecting and file size mismatches. I also see errors with zero
> >length files (host side) that get opened and populated with data
> >and the remote side is unable to read them -- keeps seeing
> >them as zero length.

That's entirely expected. NFS has always been forced to use a polling
model for attribute cache consistency. "man 5 nfs" and read all about
the "actimeo" mount options that control this behaviour.

Cheers,
Trond

--
Trond Myklebust <[email protected]>

2004-11-03 15:10:21

by Jakob Oestergaard

[permalink] [raw]

Subject: Re: nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

On Tue, Nov 02, 2004 at 08:37:04AM -0800, Trond Myklebust wrote:
> ty den 02.11.2004 Klokka 16:02 (+0400) skreiv Brad Campbell:
>
> > /raid 192.168.2.81(rw,async,no_root_squash)
> > /raid 192.168.3.80(rw,async,no_root_squash)
> > /raid0 192.168.2.81(rw,async,no_root_squash)
> > /raid0/tmp 192.168.2.81(rw,async,no_root_squash)
> > /raid2 192.168.2.81(rw,async,no_root_squash)
> > /raid2 192.168.3.80(rw,async,no_root_squash)
> > /nfsroot 192.168.2.81(rw,async,no_root_squash)
>
> You should only have 1 line per directory.

But exportfs won't complain about this.

Additionally, it seems to be allowed to use NIS netgroups in the exports
file, except, well, it "sort of doesn't quite entirely always work".
The best part of course being the errors you get: None on the server -
but mounts might misbehave randomly at some point in the future on some
client machines.

I just edited an exports file (according to your recommendations -
because my server occationally gave stale file handles on some client
machines), and found that the netmask 255.255.254.0 doesn't work either
- you need to specify two nets with 255.255.255.0 instead. My point
being, that even though I administer a network that I would describe as
"small", even I run into major problems with the lack of proper parsing
and error reporting of /etc/exports.

Generally, it seems exportfs would accept virtually any input, export it
one way or another to the kernel, with unpredictable results and
spurious errors at random times in the future.

What I would like was for exportfs to either say "No! Fix your file
stupid" or "Good! This setup will work reliably for all eternity then".

Is anyone considering fixing this? And; is the problem mainly in
exportfs, in the kernel, or both? (relevant for me to know if I want to
go fix it myself) Thanks!

--

/ jakob

2004-11-03 16:02:31

by J. Bruce Fields

[permalink] [raw]

Subject: Re: nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

On Wed, Nov 03, 2004 at 04:10:11PM +0100, Jakob Oestergaard wrote:
> What I would like was for exportfs to either say "No! Fix your file
> stupid" or "Good! This setup will work reliably for all eternity then".
>
> Is anyone considering fixing this? And; is the problem mainly in
> exportfs, in the kernel, or both? (relevant for me to know if I want to
> go fix it myself)

With recent kernels the kernel generally only queries userland for
export information on demand. So if you want an error returned at
exportfs time, you need to modify exportfs.--b.

2004-11-03 16:07:29

[permalink] [raw]

Subject: Re: nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

on den 03.11.2004 Klokka 16:10 (+0100) skreiv Jakob Oestergaard:
> On Tue, Nov 02, 2004 at 08:37:04AM -0800, Trond Myklebust wrote:
> > ty den 02.11.2004 Klokka 16:02 (+0400) skreiv Brad Campbell:
> >
> > > /raid 192.168.2.81(rw,async,no_root_squash)
> > > /raid 192.168.3.80(rw,async,no_root_squash)
> > > /raid0 192.168.2.81(rw,async,no_root_squash)
> > > /raid0/tmp 192.168.2.81(rw,async,no_root_squash)
> > > /raid2 192.168.2.81(rw,async,no_root_squash)
> > > /raid2 192.168.3.80(rw,async,no_root_squash)
> > > /nfsroot 192.168.2.81(rw,async,no_root_squash)
> >
> > You should only have 1 line per directory.
>
> But exportfs won't complain about this.
>
> Additionally, it seems to be allowed to use NIS netgroups in the exports
> file, except, well, it "sort of doesn't quite entirely always work".
> The best part of course being the errors you get: None on the server -
> but mounts might misbehave randomly at some point in the future on some
> client machines.
>
> I just edited an exports file (according to your recommendations -
> because my server occationally gave stale file handles on some client
> machines), and found that the netmask 255.255.254.0 doesn't work either
> - you need to specify two nets with 255.255.255.0 instead. My point
> being, that even though I administer a network that I would describe as
> "small", even I run into major problems with the lack of proper parsing
> and error reporting of /etc/exports.
>
> Generally, it seems exportfs would accept virtually any input, export it
> one way or another to the kernel, with unpredictable results and
> spurious errors at random times in the future.
>
> What I would like was for exportfs to either say "No! Fix your file
> stupid" or "Good! This setup will work reliably for all eternity then".
>
> Is anyone considering fixing this? And; is the problem mainly in
> exportfs, in the kernel, or both? (relevant for me to know if I want to
> go fix it myself) Thanks!

I agree entirely.

The above two questions should really be directed to Neil (so I've Cced
him). My guess would be that he would welcome any help here.

Although the limitations on what you are allowed to export are
kernel-based, I believe you only need to add the checks to the nfs-utils
package's parser for /etc/exports.

Cheers,
Trond

--
Trond Myklebust <[email protected]>

2004-11-03 16:32:37

[permalink] [raw]

Subject: Re: nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

Trond Myklebust wrote:

>ty den 02.11.2004 Klokka 17:19 (-0700) skreiv Jeff V. Merkey:
>
>
>
>>>Connect 2.4.18 and 2.6.9 with NFS 3 enabled. I am seeing problems
>>>connecting and file size mismatches. I also see errors with zero
>>>length files (host side) that get opened and populated with data
>>>and the remote side is unable to read them -- keeps seeing
>>>them as zero length.
>>>
>>>
>
>That's entirely expected. NFS has always been forced to use a polling
>model for attribute cache consistency. "man 5 nfs" and read all about
>the "actimeo" mount options that control this behaviour.
>
>Cheers,
> Trond
>
>
>
Trond,

Thanks for the update. I noticed from another post on this thread that
the problems with
/etc/exports are being addressed. This was the other problem I was
seeing but it appears
to be getting fixed.

Jeff

2004-11-03 21:22:36

[permalink] [raw]

Subject: Re: nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

Jeff V. Merkey wrote:

> Trond Myklebust wrote:
>
>>
>>>> Connect 2.4.18 and 2.6.9 with NFS 3 enabled. I am seeing problems
>>>> connecting and file size mismatches. I also see errors with zero
>>>> length files (host side) that get opened and populated with data
>>>> and the remote side is unable to read them -- keeps seeing them as
>>>> zero length.
>>>
>>
>> That's entirely expected. NFS has always been forced to use a polling
>> model for attribute cache consistency. "man 5 nfs" and read all about
>> the "actimeo" mount options that control this behaviour.
>>
>> Cheers,
>> Trond
>>
>>
>>
> Trond,
>
> Thanks for the update. I noticed from another post on this thread
> that the problems with
> /etc/exports are being addressed. This was the other problem I was
> seeing but it appears
> to be getting fixed.
>
> Jeff
>
>
>
Trond,

While you are running down this problem, I've located another odd
behavior you
may (or may not) chose to address. In my dsfs file system, I use sector
runs (large ones)
that can sometimes return odd sizes during running down a file for
reading.

i.e. I can return 4096,4096,4096,512,1024,4096,4096 block sizes on size
returns
from vfs_read() for 512 or 4096 block requests when the file is read
sequentially.
Under NTFS and other OS architectures, unless the final read attempt
returns 0 size
denoting end of file, this seems to be allowed. I noticed that all of
the linux code with
the exception of NFS also handles this situation quite nicely. NFS does
not. I have
noted that if NFS receives a read size smaller than the requested block,
it always assumes
end of file and terminates the next read. I hace modified dsfs to
always return
block sizes in a uniform manner so NFS will work properly, even though
the rest
of the Linux apps work just fine without assuming the end of file has
been reached.

I personally think this is a broken behavior, but perhaps it's in line
with some NFS
spec somewhere. I have coded around it, but thout I would mention it to
you.

Jeff

2004-11-03 23:39:37

[permalink] [raw]

Subject: Re: nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

on den 03.11.2004 Klokka 14:38 (-0700) skreiv Jeff V. Merkey:
> I personally think this is a broken behavior, but perhaps it's in line
> with some NFS
> spec somewhere. I have coded around it, but thout I would mention it to
> you.

This should already be fixed in 2.6.x kernels.

Cheers,
Trond

--
Trond Myklebust <[email protected]>

2004-11-03 23:55:55

[permalink] [raw]

Subject: Re: nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

Trond Myklebust wrote:

>on den 03.11.2004 Klokka 14:38 (-0700) skreiv Jeff V. Merkey:
>
>
>>I personally think this is a broken behavior, but perhaps it's in line
>>with some NFS
>>spec somewhere. I have coded around it, but thout I would mention it to
>>you.
>>
>>
>
>This should already be fixed in 2.6.x kernels.
>
>Cheers,
> Trond
>
>
>
I'll test it and let you know. I saw this on a 2.4.18 kernel in a 2.6.X
NFS network test setup. I have not verified
it on 2.6.X. Thanks for the heads up, sounds like you fixed it.

Jeff

2004-11-05 02:48:27

[permalink] [raw]

Subject: Re: nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

Jakob Oestergaard wrote:
> On Tue, Nov 02, 2004 at 02:49:53PM -0500, Jeff Garzik wrote:
>
>>Trond Myklebust wrote:
>
> ...
>
>>> http://nfs.sourceforge.net/nfs-howto/server.html#CONFIG
>>
>>
>>I'm also seeing stale filehandle problems here in recent kernels.
>>
>>Setup: x86 or x86-64, TCP, NFSv4 compiled in to both server and client,
>>but not specified in mount options.
>>
>>This is readily reproducible with rsync -- I just boot to an earlier
>>version of the kernel on the NFS client, and the stale filehandle
>>problems go away.
>
>
> Hi Jeff,
>
> Does running an 'ls' on the server in the exported directory that is
> stale on the client resolve the problem (temporarily)?

Yes.

Jeff

2004-11-05 02:57:40

[permalink] [raw]

Subject: Re: nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

Trond Myklebust wrote:
> ty den 02.11.2004 Klokka 14:49 (-0500) skreiv Jeff Garzik:
>
>
>>This is readily reproducible with rsync -- I just boot to an earlier
>>version of the kernel on the NFS client, and the stale filehandle
>>problems go away.
>
>
> Huh? The client cannot generate stale filehandle errors: only the server
> does that.

Not saying that the client is _generating_ the stale filehandle errors,
only saying that they appear to go away when I boot the _client_ into
older 2.6.9 kernels.

> Have you got a binary tcpdump that shows the problem?

I'll create one if I get time and can reliably reproduce it (rsync
_sometimes_ shows the behavior, but not always).

Jeff

2004-11-05 02:59:36

[permalink] [raw]

Subject: Re: nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

to den 04.11.2004 Klokka 21:47 (-0500) skreiv Jeff Garzik:
> >
> > Does running an 'ls' on the server in the exported directory that is
> > stale on the client resolve the problem (temporarily)?
>
> Yes.

This still looks very much like a server issue to me. Could someone who
is seeing the bug try to capture an instance of the ESTALE error going
across the wire, and then do a fresh lookup of the same file from an
"ls" call. I'd like to check how the stale filehandle differs from the
freshly looked up one...

Please use "tcpdump -s 9000 -w /tmp/binary.pcap port 2049 and host
my.servers.name" for the actual capture.

Cheers,
Trond

--
Trond Myklebust <[email protected]>

2004-11-05 03:08:14

[permalink] [raw]

Subject: Re: nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

Trond Myklebust wrote:
> to den 04.11.2004 Klokka 21:47 (-0500) skreiv Jeff Garzik:
>
>>>Does running an 'ls' on the server in the exported directory that is
>>>stale on the client resolve the problem (temporarily)?
>>
>>Yes.
>
>
> This still looks very much like a server issue to me. Could someone who
> is seeing the bug try to capture an instance of the ESTALE error going
> across the wire, and then do a fresh lookup of the same file from an
> "ls" call. I'd like to check how the stale filehandle differs from the
> freshly looked up one...
>
> Please use "tcpdump -s 9000 -w /tmp/binary.pcap port 2049 and host
> my.servers.name" for the actual capture.

Will do.

FWIW my server is running 2.6.9-final. Client mount options in fstab
are "defaults,tcp" and server options are (rw,no_root_squash,async).

Jeff

2004-11-05 03:10:28

[permalink] [raw]

Subject: Re: nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

to den 04.11.2004 Klokka 21:57 (-0500) skreiv Jeff Garzik:

> Not saying that the client is _generating_ the stale filehandle errors,
> only saying that they appear to go away when I boot the _client_ into
> older 2.6.9 kernels.

That would point to some pretty nasty memory corruption issues on the
client then (affecting the cached filehandle in the inode itself).

So... I can't see that any NFS client changes have been pushed to Linus
after the release of 2.6.9-rc2. Is the latter afflicted with the ESTALE
problem?

Cheers,
Trond

--
Trond Myklebust <[email protected]>

2004-11-05 07:27:47