Hi,
Since upgrading from 2.6.39-ish to 3.1-ish, and on 3.2.5, we are seeing a
lot of occurrences of Stale NFS file handle errors when accessing a mount
whose NFSv3 source is a subdirectory of another mount point. For example,
in this case:
# mount | grep /shared
10.10.1.1:/storage/vg1/shared on /shared type nfs (rw,hard,intr,tcp,timeo=300,retrans=2,vers=3,addr=10.10.1.1)
10.10.1.1:/storage/vg1/shared/fp on /usr/local/fp type nfs (rw,hard,intr,tcp,timeo=300,retrans=2,vers=3,addr=10.10.1.1)
When the issue occurs, the /shared mount point is fine as is /shared/fp,
but "df" or "ls" or anything on /usr/local/fp will ESTALE. This somehow
corrected itself while I was trying to gather information this time, but
usually the d_ino returned by getdents() on the parent directory shows a
different inode number than for /shared/fp.
When this happens, I am unable to umount -f or umount -l /usr/local/fp
(ESTALE), but I can actually umount /shared; umount /usr/local/fp; and
mount -a, which seems to "fix" it.
is this acting similar to a bind mount internally now and revalidation or
something is breaking in this case? This is happening fairly often, so I
will try to collect more info again next time.
Simon-
On Wed, Feb 29, 2012 at 01:11:31AM +0000, Myklebust, Trond wrote:
> On Tue, 2012-02-28 at 17:06 -0800, Simon Kirby wrote:
> > Hi,
> >
> > Since upgrading from 2.6.39-ish to 3.1-ish, and on 3.2.5, we are seeing a
> > lot of occurrences of Stale NFS file handle errors when accessing a mount
> > whose NFSv3 source is a subdirectory of another mount point. For example,
> > in this case:
> >
> > # mount | grep /shared
> > 10.10.1.1:/storage/vg1/shared on /shared type nfs (rw,hard,intr,tcp,timeo=300,retrans=2,vers=3,addr=10.10.1.1)
> > 10.10.1.1:/storage/vg1/shared/fp on /usr/local/fp type nfs (rw,hard,intr,tcp,timeo=300,retrans=2,vers=3,addr=10.10.1.1)
> >
> > When the issue occurs, the /shared mount point is fine as is /shared/fp,
> > but "df" or "ls" or anything on /usr/local/fp will ESTALE. This somehow
> > corrected itself while I was trying to gather information this time, but
> > usually the d_ino returned by getdents() on the parent directory shows a
> > different inode number than for /shared/fp.
> >
> > When this happens, I am unable to umount -f or umount -l /usr/local/fp
> > (ESTALE), but I can actually umount /shared; umount /usr/local/fp; and
> > mount -a, which seems to "fix" it.
> >
> > is this acting similar to a bind mount internally now and revalidation or
> > something is breaking in this case? This is happening fairly often, so I
> > will try to collect more info again next time.
>
> ESTALE is a server side error, not a client side error. What server are
> you using here, and what do the export options look like?
An older 2.6.33 host running DRBD HA knfsd bits. We had problems with the
XFS inode reclaim changes causing crashes on newer kernels (actually on
2.6.33, too, but not so much on this node with only locally-attached
disks), so these kernels haven't been upgraded for some time. It's likely
time to try again. The export is:
/storage/vg1 /10.10.1.0/24(rw,sync,no_root_squash,no_subtree_check,fsid=1)
I just found it weird to see the ESTALE when accessing /usr/local/fp,
while /shared/fp works fine at the same time, even though they're the
same path on the server.
The plan was to upgrade this pair first anyway, so we can do that if
the problem is likely coming from the server side.
Simon-
On Wed, Feb 29, 2012 at 11:59:16AM -0800, Simon Kirby wrote:
> On Wed, Feb 29, 2012 at 01:11:31AM +0000, Myklebust, Trond wrote:
>
> > On Tue, 2012-02-28 at 17:06 -0800, Simon Kirby wrote:
> > > Hi,
> > >
> > > Since upgrading from 2.6.39-ish to 3.1-ish, and on 3.2.5, we are seeing a
> > > lot of occurrences of Stale NFS file handle errors when accessing a mount
> > > whose NFSv3 source is a subdirectory of another mount point. For example,
> > > in this case:
> > >
> > > # mount | grep /shared
> > > 10.10.1.1:/storage/vg1/shared on /shared type nfs (rw,hard,intr,tcp,timeo=300,retrans=2,vers=3,addr=10.10.1.1)
> > > 10.10.1.1:/storage/vg1/shared/fp on /usr/local/fp type nfs (rw,hard,intr,tcp,timeo=300,retrans=2,vers=3,addr=10.10.1.1)
> > >
> > > When the issue occurs, the /shared mount point is fine as is /shared/fp,
> > > but "df" or "ls" or anything on /usr/local/fp will ESTALE. This somehow
> > > corrected itself while I was trying to gather information this time, but
> > > usually the d_ino returned by getdents() on the parent directory shows a
> > > different inode number than for /shared/fp.
> > >
> > > When this happens, I am unable to umount -f or umount -l /usr/local/fp
> > > (ESTALE), but I can actually umount /shared; umount /usr/local/fp; and
> > > mount -a, which seems to "fix" it.
> > >
> > > is this acting similar to a bind mount internally now and revalidation or
> > > something is breaking in this case? This is happening fairly often, so I
> > > will try to collect more info again next time.
> >
> > ESTALE is a server side error, not a client side error. What server are
> > you using here, and what do the export options look like?
>
> An older 2.6.33 host running DRBD HA knfsd bits. We had problems with the
> XFS inode reclaim changes causing crashes on newer kernels (actually on
> 2.6.33, too, but not so much on this node with only locally-attached
> disks), so these kernels haven't been upgraded for some time. It's likely
> time to try again. The export is:
>
> /storage/vg1 /10.10.1.0/24(rw,sync,no_root_squash,no_subtree_check,fsid=1)
>
> I just found it weird to see the ESTALE when accessing /usr/local/fp,
> while /shared/fp works fine at the same time, even though they're the
> same path on the server.
Is /storage/vg1/shared/fp on that server something that can ever be
removed? (Say to be replaced by something else?)
For normal directories that's not a problem, the client's used to
dealing with the fact that directories may come and go.
For a directory that you've told the client to *mount*, that's dirty
trick--it's really expecting that directory to be there as long as it's
mounted....
--b.
>
> The plan was to upgrade this pair first anyway, so we can do that if
> the problem is likely coming from the server side.
>
> Simon-
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email protected]
> More majordomo info at http://vger.kernel.org/majordomo-info.html
T24gVHVlLCAyMDEyLTAyLTI4IGF0IDE3OjA2IC0wODAwLCBTaW1vbiBLaXJieSB3cm90ZToNCj4g
SGksDQo+IA0KPiBTaW5jZSB1cGdyYWRpbmcgZnJvbSAyLjYuMzktaXNoIHRvIDMuMS1pc2gsIGFu
ZCBvbiAzLjIuNSwgd2UgYXJlIHNlZWluZyBhDQo+IGxvdCBvZiBvY2N1cnJlbmNlcyBvZiBTdGFs
ZSBORlMgZmlsZSBoYW5kbGUgZXJyb3JzIHdoZW4gYWNjZXNzaW5nIGEgbW91bnQNCj4gd2hvc2Ug
TkZTdjMgc291cmNlIGlzIGEgc3ViZGlyZWN0b3J5IG9mIGFub3RoZXIgbW91bnQgcG9pbnQuIEZv
ciBleGFtcGxlLA0KPiBpbiB0aGlzIGNhc2U6DQo+IA0KPiAjIG1vdW50IHwgZ3JlcCAvc2hhcmVk
DQo+IDEwLjEwLjEuMTovc3RvcmFnZS92ZzEvc2hhcmVkIG9uIC9zaGFyZWQgdHlwZSBuZnMgKHJ3
LGhhcmQsaW50cix0Y3AsdGltZW89MzAwLHJldHJhbnM9Mix2ZXJzPTMsYWRkcj0xMC4xMC4xLjEp
DQo+IDEwLjEwLjEuMTovc3RvcmFnZS92ZzEvc2hhcmVkL2ZwIG9uIC91c3IvbG9jYWwvZnAgdHlw
ZSBuZnMgKHJ3LGhhcmQsaW50cix0Y3AsdGltZW89MzAwLHJldHJhbnM9Mix2ZXJzPTMsYWRkcj0x
MC4xMC4xLjEpDQo+IA0KPiBXaGVuIHRoZSBpc3N1ZSBvY2N1cnMsIHRoZSAvc2hhcmVkIG1vdW50
IHBvaW50IGlzIGZpbmUgYXMgaXMgL3NoYXJlZC9mcCwNCj4gYnV0ICJkZiIgb3IgImxzIiBvciBh
bnl0aGluZyBvbiAvdXNyL2xvY2FsL2ZwIHdpbGwgRVNUQUxFLiBUaGlzIHNvbWVob3cNCj4gY29y
cmVjdGVkIGl0c2VsZiB3aGlsZSBJIHdhcyB0cnlpbmcgdG8gZ2F0aGVyIGluZm9ybWF0aW9uIHRo
aXMgdGltZSwgYnV0DQo+IHVzdWFsbHkgdGhlIGRfaW5vIHJldHVybmVkIGJ5IGdldGRlbnRzKCkg
b24gdGhlIHBhcmVudCBkaXJlY3Rvcnkgc2hvd3MgYQ0KPiBkaWZmZXJlbnQgaW5vZGUgbnVtYmVy
IHRoYW4gZm9yIC9zaGFyZWQvZnAuDQo+IA0KPiBXaGVuIHRoaXMgaGFwcGVucywgSSBhbSB1bmFi
bGUgdG8gdW1vdW50IC1mIG9yIHVtb3VudCAtbCAvdXNyL2xvY2FsL2ZwDQo+IChFU1RBTEUpLCBi
dXQgSSBjYW4gYWN0dWFsbHkgdW1vdW50IC9zaGFyZWQ7IHVtb3VudCAvdXNyL2xvY2FsL2ZwOyBh
bmQNCj4gbW91bnQgLWEsIHdoaWNoIHNlZW1zIHRvICJmaXgiIGl0Lg0KPiANCj4gaXMgdGhpcyBh
Y3Rpbmcgc2ltaWxhciB0byBhIGJpbmQgbW91bnQgaW50ZXJuYWxseSBub3cgYW5kIHJldmFsaWRh
dGlvbiBvcg0KPiBzb21ldGhpbmcgaXMgYnJlYWtpbmcgaW4gdGhpcyBjYXNlPyBUaGlzIGlzIGhh
cHBlbmluZyBmYWlybHkgb2Z0ZW4sIHNvIEkNCj4gd2lsbCB0cnkgdG8gY29sbGVjdCBtb3JlIGlu
Zm8gYWdhaW4gbmV4dCB0aW1lLg0KDQpFU1RBTEUgaXMgYSBzZXJ2ZXIgc2lkZSBlcnJvciwgbm90
IGEgY2xpZW50IHNpZGUgZXJyb3IuIFdoYXQgc2VydmVyIGFyZQ0KeW91IHVzaW5nIGhlcmUsIGFu
ZCB3aGF0IGRvIHRoZSBleHBvcnQgb3B0aW9ucyBsb29rIGxpa2U/DQoNCi0tIA0KVHJvbmQgTXlr
bGVidXN0DQpMaW51eCBORlMgY2xpZW50IG1haW50YWluZXINCg0KTmV0QXBwDQpUcm9uZC5NeWts
ZWJ1c3RAbmV0YXBwLmNvbQ0Kd3d3Lm5ldGFwcC5jb20NCg0K
On Wed, Feb 29, 2012 at 03:14:01PM -0500, J. Bruce Fields wrote:
> On Wed, Feb 29, 2012 at 11:59:16AM -0800, Simon Kirby wrote:
> > On Wed, Feb 29, 2012 at 01:11:31AM +0000, Myklebust, Trond wrote:
> >
> > > On Tue, 2012-02-28 at 17:06 -0800, Simon Kirby wrote:
> > > > Hi,
> > > >
> > > > Since upgrading from 2.6.39-ish to 3.1-ish, and on 3.2.5, we are seeing a
> > > > lot of occurrences of Stale NFS file handle errors when accessing a mount
> > > > whose NFSv3 source is a subdirectory of another mount point. For example,
> > > > in this case:
> > > >
> > > > # mount | grep /shared
> > > > 10.10.1.1:/storage/vg1/shared on /shared type nfs (rw,hard,intr,tcp,timeo=300,retrans=2,vers=3,addr=10.10.1.1)
> > > > 10.10.1.1:/storage/vg1/shared/fp on /usr/local/fp type nfs (rw,hard,intr,tcp,timeo=300,retrans=2,vers=3,addr=10.10.1.1)
> > > >
> > > > When the issue occurs, the /shared mount point is fine as is /shared/fp,
> > > > but "df" or "ls" or anything on /usr/local/fp will ESTALE. This somehow
> > > > corrected itself while I was trying to gather information this time, but
> > > > usually the d_ino returned by getdents() on the parent directory shows a
> > > > different inode number than for /shared/fp.
> > > >
> > > > When this happens, I am unable to umount -f or umount -l /usr/local/fp
> > > > (ESTALE), but I can actually umount /shared; umount /usr/local/fp; and
> > > > mount -a, which seems to "fix" it.
> > > >
> > > > is this acting similar to a bind mount internally now and revalidation or
> > > > something is breaking in this case? This is happening fairly often, so I
> > > > will try to collect more info again next time.
> > >
> > > ESTALE is a server side error, not a client side error. What server are
> > > you using here, and what do the export options look like?
> >
> > An older 2.6.33 host running DRBD HA knfsd bits. We had problems with the
> > XFS inode reclaim changes causing crashes on newer kernels (actually on
> > 2.6.33, too, but not so much on this node with only locally-attached
> > disks), so these kernels haven't been upgraded for some time. It's likely
> > time to try again. The export is:
> >
> > /storage/vg1 /10.10.1.0/24(rw,sync,no_root_squash,no_subtree_check,fsid=1)
> >
> > I just found it weird to see the ESTALE when accessing /usr/local/fp,
> > while /shared/fp works fine at the same time, even though they're the
> > same path on the server.
>
> Is /storage/vg1/shared/fp on that server something that can ever be
> removed? (Say to be replaced by something else?)
>
> For normal directories that's not a problem, the client's used to
> dealing with the fact that directories may come and go.
>
> For a directory that you've told the client to *mount*, that's dirty
> trick--it's really expecting that directory to be there as long as it's
> mounted....
Sure, but nope, the directory hasn't moved or been deleted since it was
created. It only grows.
Simon-