Hi,
Folks testing linux nfs + Ganesha with bonnie++ have noticed the issue described here (but I didn't see traffic on this list):
http://bugs.centos.org/view.php?id=5496
https://bugzilla.redhat.com/show_bug.cgi?id=789452
Ie, an application which unlinks on an open (nfs) directory stream, and continues to read and unlink progressively, will generally not see all the entries originally in the stream. Reopening the stream ives the "correct" result. So, apparently older clients were more forgiving, as mentioned in the links. Is the current behavior one the client is intending?
Thanks,
Matt
--
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI 48104
http://linuxbox.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
On Mon, 26 Mar 2012 11:17:18 -0700
Boaz Harrosh <[email protected]> wrote:
> On 03/24/2012 10:12 AM, Myklebust, Trond wrote:
>
> > On Sat, 2012-03-24 at 12:53 -0400, Matt W. Benjamin wrote:
> >> Hi,
> >>
> >> I don't think anything is. Or, people originally reported the behavior against knfsd.
> >>
> >> Matt
> >
> > There is a known issue with ext2/3/4 generating non-unique readdir
> > cookies. It rarely hits you when you are creating small directories, but
> > it frequently hits you with larger ones. A fix is underway that should
> > significantly reduce the frequency of cookie collisions.
> >
> > Recent NFS clients will actually detect the presence of those cookie
> > loops, and log them in the kernel syslog. That would therefore be the
> > first thing that I'd check if confronted with this kind of problem.
> >
> > Cheers
> > Trond
> >
>
>
> Trond please look on the bug report links below. It's not the "cookie collisions" case.
>
> It's the new (post RHEL 6.0 Kernel) NFS need for opendir after an unlink.
> Now the POSIX man page *does* say that applications must re-opendir after
> unlink, but there are some applications who did not read the manual, and since
> it works with local filesystems and old nfs, (What Kernel RHEL 6.0 is based on?)
> they never noticed the bug and never fixed it.
>
The RHEL6 kernel is 2.6.32 based, but we have (as always) done some
fairly extensive backports from upstream.
One of the things that was backported was the Bryan's work to clean up
readdir and to eliminate the readdirplus limit. With that change, you
get fewer entries per READDIRPLUS call, so you make more READDIRPLUS
calls to the server in order to traverse an entire directory.
Since you're unlinking as you go, then I suspect that you're just more
likely to see this problem crop up, but I suspect Trond is correct and
this is preexisting bug.
One way to confirm this might be to have them mount with -o nordirplus.
Does that make the problem go away?
--
Jeff Layton <[email protected]>
Hi,
Thanks for providing all this reasoning and additional background.
I guess I've always constructed an imaginary intention of the protocol feature that's different from the actual one. Yet the feature nevertheless seemed useful.
There seem to be two parts to my construction:
1. negatively, it never occurred to me that a client implementation would assume that it could protect an application from invalidation, unless it was able to perform, and cache a complete traversal--this would be one with a consistent verifier for all component readdirs
2. at the same time, it seems a conforming server implementation can be made which would often support a client in doing this, presuming it was able to use the cookie verifier to select a prior version of the directory; not only do some filesystems now appear able to do this, this technique is used by the dcache nfs server, though iiuc, dcache doesn't make any promises that it will have a specific component cached at every version, etc
Matt
----- "Trond Myklebust" <[email protected]> wrote:
> On Fri, 2012-03-30 at 11:24 -0400, Peter Staubach wrote:
> > Yes, hmmm, ummm...
> >
> > A long time, I thought that the readdir cookie verifier sounded like
> a good idea. After all, if directories can be compacted, it would be
> nice to be able to tell when the cookie has become invalid, right?
> So, I added it to the NFSv3 protocol.
> >
> > It was only afterwards, in attempting to implement it, that the real
> truth was discovered. This is that the interfaces are not sufficient
> to be able to convey this information sufficiently and to be able to
> recover from the situation. Currently, recovery require action on the
> part of the application.
> >
> > In retrospect, I wouldn't add the cookie verifier if I was doing it
> again.
> >
> > Thanks for the support, though. :-)
> >
> > I do disagree with Trond though, in that applications must be coded
> to be able to handle directories where cookies can become invalidated.
> That's why rm and such have to be coded to keep track of their
> location in the directory by more than just d_off.
>
> The problem with that is that there is no standard for how
> applications
> should proceed to do this, and so there are no guidelines for how the
> client should behave.
>
> Worse, if you've looked at the glibc readdir library code, you'll
> have
> noticed that it _depends_ on being able to use a strictly
> POSIX-conforming version of telldir()/seekdir(). If it can't rewind to
> a
> previously saved offset, then it can end up skipping entries.
>
> So no, I can't rely on applications either...
>
> > ps
> >
> >
> > -----Original Message-----
> > From: [email protected]
> [mailto:[email protected]] On Behalf Of Matt W.
> Benjamin
> > Sent: Monday, March 26, 2012 2:55 PM
> > To: Boaz Harrosh
> > Cc: linux-nfs; Ganesha NFS List; Trond Myklebust
> > Subject: Re: unlink within an open directory stream
> >
> > Hi,
> >
> > Boaz: we do not any longer send a readdir index. We do send a
> cookieverf.
> >
> > Fist of all, I haven't established that the issue we're actually
> observing is caused by the Linux client sending old cookies to
> readdir(something). However, if it is, it's in no way better to try
> to make cookies "more persistent." Nor should the Linux client be
> expecting it. The assumption is simply flawed. The protocol
> introduced cookie verifier (a LONG time ago) for a reason.
> >
> > Matt
> > ----- "Boaz Harrosh" <[email protected]> wrote:
> >
> > > On 03/26/2012 11:25 AM, Myklebust, Trond wrote:
> > >
> > > > On Mon, 2012-03-26 at 11:17 -0700, Boaz Harrosh wrote:
> > > >> On 03/24/2012 10:12 AM, Myklebust, Trond wrote:
> > > >>
> > > >>
> > > >> It's the new (post RHEL 6.0 Kernel) NFS need for opendir after
> an
> > > unlink.
> > > >
> > > > What?
> > > >
> > >
> > >
> > > The links below report on regression in RHEL 6.2 which have a new
> NFS
> > > implementation where 6.0 used to be fine.
> > > (Or it's what I understood from the reporters)
> > >
> > > <snip>
> > >
> > > >
> > >
> > > > No.
> > > >
> > > > If the server supports permanent readdir cookies, then there is
> no
> > > need
> > > > for opendir after unlink.
> > > >
> > > > If the server does not have permanent readdir cookies, then our
> > > client
> > > > has never supported it anyway (nor will it ever do so). That
> whole
> > > > 'cookieverf' READDIR bullshit has never provided a workable
> model
> > > for a
> > > > POSIX client...
> > >
> > >
> > > Thanks Trond for the explanation. Forgive my slowness. So you are
>
> > > saying that with knfsd it's actually filesystem dependent and
> maybe
> > > the reporters did not compare apples-to-apples and the difference
> is
> > > in the behind file system.
> > >
> > > Matt I'm sure Trond is right with regard to Ganesha, because we
> send a
> > > readdir index and a cookieverf, which will change after unlink.
> Since
> > > the application does not call opendir again the client will send
> the
> > > old cookies, which is now a different file or might get
> out-of-bounds
> > > for Ganesha.
> > >
> > > Let me think about it a bit. I'm sure we can send a Better 64bit
> > > cookie, which will be more persistent, even across unlinks.
> > >
> > > >
> > > >
> > >
> > >
> > > Thanks
> > > Boaz
> >
> > --
> > Matt Benjamin
> > The Linux Box
> > 206 South Fifth Ave. Suite 150
> > Ann Arbor, MI 48104
> >
> > http://linuxbox.com
> >
> > tel. 734-761-4689
> > fax. 734-769-8938
> > cel. 734-216-5309
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs"
> in the body of a message to [email protected] More majordomo
> info at http://vger.kernel.org/majordomo-info.html
>
> --
> Trond Myklebust
> Linux NFS client maintainer
>
> NetApp
> [email protected]
> http://www.netapp.com
--
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI 48104
http://linuxbox.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
On 03/24/2012 10:12 AM, Myklebust, Trond wrote:
> On Sat, 2012-03-24 at 12:53 -0400, Matt W. Benjamin wrote:
>> Hi,
>>
>> I don't think anything is. Or, people originally reported the behavior against knfsd.
>>
>> Matt
>
> There is a known issue with ext2/3/4 generating non-unique readdir
> cookies. It rarely hits you when you are creating small directories, but
> it frequently hits you with larger ones. A fix is underway that should
> significantly reduce the frequency of cookie collisions.
>
> Recent NFS clients will actually detect the presence of those cookie
> loops, and log them in the kernel syslog. That would therefore be the
> first thing that I'd check if confronted with this kind of problem.
>
> Cheers
> Trond
>
Trond please look on the bug report links below. It's not the "cookie collisions" case.
It's the new (post RHEL 6.0 Kernel) NFS need for opendir after an unlink.
Now the POSIX man page *does* say that applications must re-opendir after
unlink, but there are some applications who did not read the manual, and since
it works with local filesystems and old nfs, (What Kernel RHEL 6.0 is based on?)
they never noticed the bug and never fixed it.
Could we easily support the broken application by being bug compatible to
old NFS versions?
.i.e Don't require re-opendir after unlink of a file.
There are more examples in the bug reports below but basically bonnie++
does the following:
DIR *d = opendir(".");
dirent *file_ent;
while((file_ent = readdir(d)) != NULL) {
unlink( file_ent->d_name))
}
closedir(d);
where it actually needs to do:
DIR *d = opendir(".");
dirent *file_ent;
while((file_ent = readdir(d)) != NULL) {
unlink( file_ent->d_name))
closedir(d);
d = opendir(".");
}
closedir(d);
But again case one used to work with old NFS. And it looks like
it is not Server dependent. We saw this both with Ganesha as well
as knfsd
<snip>
>> http://bugs.centos.org/view.php?id=5496
>> https://bugzilla.redhat.com/show_bug.cgi?id=789452
Thanks
Boaz
Hi,
I don't think anything is. Or, people originally reported the behavior against knfsd.
Matt
----- "Trond Myklebust" <[email protected]> wrote:
> On Sat, 2012-03-24 at 12:34 -0400, Matt W. Benjamin wrote:
> > Hi,
> >
> > Folks testing linux nfs + Ganesha with bonnie++ have noticed the
> issue described here (but I didn't see traffic on this list):
> >
> > http://bugs.centos.org/view.php?id=5496
> > https://bugzilla.redhat.com/show_bug.cgi?id=789452
> >
> > Ie, an application which unlinks on an open (nfs) directory stream,
> and continues to read and unlink progressively, will generally not see
> all the entries originally in the stream. Reopening the stream ives
> the "correct" result. So, apparently older clients were more
> forgiving, as mentioned in the links. Is the current behavior one the
> client is intending?
> >
> > Thanks,
> >
> > Matt
>
> What is so particular about the ganesha readdir implementation?
>
> --
> Trond Myklebust
> Linux NFS client maintainer
>
> NetApp
> [email protected]
> http://www.netapp.com
--
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI 48104
http://linuxbox.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
T24gTW9uLCAyMDEyLTAzLTI2IGF0IDE0OjU1IC0wNDAwLCBNYXR0IFcuIEJlbmphbWluIHdyb3Rl
Og0KPiBIaSwNCj4gDQo+IEJvYXo6ICB3ZSBkbyBub3QgYW55IGxvbmdlciBzZW5kIGEgcmVhZGRp
ciBpbmRleC4gIFdlIGRvIHNlbmQgYSBjb29raWV2ZXJmLg0KDQpVaGguLi4gVGhhdCB3b3VsZCBi
ZSBzbyBicm9rZW4sIHRoYXQgSSBjYW4ndCBpbWFnaW5lIHRoYXQgaXMgdHJ1ZS4uLg0KDQo+IEZp
c3Qgb2YgYWxsLCBJIGhhdmVuJ3QgZXN0YWJsaXNoZWQgdGhhdCB0aGUgaXNzdWUgd2UncmUgYWN0
dWFsbHkgb2JzZXJ2aW5nIGlzIGNhdXNlZA0KPiBieSB0aGUgTGludXggY2xpZW50IHNlbmRpbmcg
b2xkIGNvb2tpZXMgdG8gcmVhZGRpcihzb21ldGhpbmcpLiAgSG93ZXZlciwgaWYgaXQgaXMsDQo+
IGl0J3MgaW4gbm8gd2F5IGJldHRlciB0byB0cnkgdG8gbWFrZSBjb29raWVzICJtb3JlIHBlcnNp
c3RlbnQuIiAgTm9yIHNob3VsZCB0aGUgTGludXgNCj4gY2xpZW50IGJlIGV4cGVjdGluZyBpdC4g
IFRoZSBhc3N1bXB0aW9uIGlzIHNpbXBseSBmbGF3ZWQuICBUaGUgcHJvdG9jb2wgaW50cm9kdWNl
ZA0KPiBjb29raWUgdmVyaWZpZXIgKGEgTE9ORyB0aW1lIGFnbykgZm9yIGEgcmVhc29uLg0KDQpC
dWxsc2hpdC4uLiBUaGUgY29va2llIHZlcmlmaWVyIGlzIHVuaW1wbGVtZW50YWJsZS4gRmVlbCBm
cmVlIHRvIHRyeSBvcg0KdG8gc2hvdyBtZSBhbiBpbXBsZW1lbnRhdGlvbiB0aGF0IHdvcmtzLiBI
b3dldmVyIEkgcmVmdXNlIHRvIHdhc3RlIGFueQ0KbW9yZSBvZiBteSB0aW1lIHRyeWluZyB0byBt
YWtlIHRoYXQgY3JhcCB3b3JrLCBoYXZpbmcgYWxyZWFkeSB3YXN0ZWQNCnNldmVyYWwgbW9udGhz
IG9mIG15IGxpZmUgZG9pbmcganVzdCB0aGF0Lg0KDQotLSANClRyb25kIE15a2xlYnVzdA0KTGlu
dXggTkZTIGNsaWVudCBtYWludGFpbmVyDQoNCk5ldEFwcA0KVHJvbmQuTXlrbGVidXN0QG5ldGFw
cC5jb20NCnd3dy5uZXRhcHAuY29tDQoNCg==
T24gTW9uLCAyMDEyLTAzLTI2IGF0IDExOjE3IC0wNzAwLCBCb2F6IEhhcnJvc2ggd3JvdGU6DQo+
IE9uIDAzLzI0LzIwMTIgMTA6MTIgQU0sIE15a2xlYnVzdCwgVHJvbmQgd3JvdGU6DQo+IA0KPiA+
IE9uIFNhdCwgMjAxMi0wMy0yNCBhdCAxMjo1MyAtMDQwMCwgTWF0dCBXLiBCZW5qYW1pbiB3cm90
ZToNCj4gPj4gSGksDQo+ID4+DQo+ID4+IEkgZG9uJ3QgdGhpbmsgYW55dGhpbmcgaXMuICBPciwg
cGVvcGxlIG9yaWdpbmFsbHkgcmVwb3J0ZWQgdGhlIGJlaGF2aW9yIGFnYWluc3Qga25mc2QuDQo+
ID4+DQo+ID4+IE1hdHQNCj4gPiANCj4gPiBUaGVyZSBpcyBhIGtub3duIGlzc3VlIHdpdGggZXh0
Mi8zLzQgZ2VuZXJhdGluZyBub24tdW5pcXVlIHJlYWRkaXINCj4gPiBjb29raWVzLiBJdCByYXJl
bHkgaGl0cyB5b3Ugd2hlbiB5b3UgYXJlIGNyZWF0aW5nIHNtYWxsIGRpcmVjdG9yaWVzLCBidXQN
Cj4gPiBpdCBmcmVxdWVudGx5IGhpdHMgeW91IHdpdGggbGFyZ2VyIG9uZXMuIEEgZml4IGlzIHVu
ZGVyd2F5IHRoYXQgc2hvdWxkDQo+ID4gc2lnbmlmaWNhbnRseSByZWR1Y2UgdGhlIGZyZXF1ZW5j
eSBvZiBjb29raWUgY29sbGlzaW9ucy4NCj4gPiANCj4gPiBSZWNlbnQgTkZTIGNsaWVudHMgd2ls
bCBhY3R1YWxseSBkZXRlY3QgdGhlIHByZXNlbmNlIG9mIHRob3NlIGNvb2tpZQ0KPiA+IGxvb3Bz
LCBhbmQgbG9nIHRoZW0gaW4gdGhlIGtlcm5lbCBzeXNsb2cuIFRoYXQgd291bGQgdGhlcmVmb3Jl
IGJlIHRoZQ0KPiA+IGZpcnN0IHRoaW5nIHRoYXQgSSdkIGNoZWNrIGlmIGNvbmZyb250ZWQgd2l0
aCB0aGlzIGtpbmQgb2YgcHJvYmxlbS4NCj4gPiANCj4gPiBDaGVlcnMNCj4gPiAgIFRyb25kDQo+
ID4gDQo+IA0KPiANCj4gVHJvbmQgcGxlYXNlIGxvb2sgb24gdGhlIGJ1ZyByZXBvcnQgbGlua3Mg
YmVsb3cuIEl0J3Mgbm90IHRoZSAiY29va2llIGNvbGxpc2lvbnMiIGNhc2UuDQo+IA0KPiBJdCdz
IHRoZSBuZXcgKHBvc3QgUkhFTCA2LjAgS2VybmVsKSBORlMgbmVlZCBmb3Igb3BlbmRpciBhZnRl
ciBhbiB1bmxpbmsuDQoNCldoYXQ/DQoNCj4gTm93IHRoZSBQT1NJWCBtYW4gcGFnZSAqZG9lcyog
c2F5IHRoYXQgYXBwbGljYXRpb25zIG11c3QgcmUtb3BlbmRpciBhZnRlcg0KPiB1bmxpbmssIGJ1
dCB0aGVyZSBhcmUgc29tZSBhcHBsaWNhdGlvbnMgd2hvIGRpZCBub3QgcmVhZCB0aGUgbWFudWFs
LCBhbmQgc2luY2UNCj4gaXQgd29ya3Mgd2l0aCBsb2NhbCBmaWxlc3lzdGVtcyBhbmQgb2xkIG5m
cywgKFdoYXQgS2VybmVsIFJIRUwgNi4wIGlzIGJhc2VkIG9uPykNCj4gdGhleSBuZXZlciBub3Rp
Y2VkIHRoZSBidWcgYW5kIG5ldmVyIGZpeGVkIGl0Lg0KPiANCj4gQ291bGQgd2UgZWFzaWx5IHN1
cHBvcnQgdGhlIGJyb2tlbiBhcHBsaWNhdGlvbiBieSBiZWluZyBidWcgY29tcGF0aWJsZSB0bw0K
PiBvbGQgTkZTIHZlcnNpb25zPw0KPiAuaS5lIERvbid0IHJlcXVpcmUgcmUtb3BlbmRpciBhZnRl
ciB1bmxpbmsgb2YgYSBmaWxlLg0KPiANCj4gVGhlcmUgYXJlIG1vcmUgZXhhbXBsZXMgaW4gdGhl
IGJ1ZyByZXBvcnRzIGJlbG93IGJ1dCBiYXNpY2FsbHkgYm9ubmllKysNCj4gZG9lcyB0aGUgZm9s
bG93aW5nOg0KPiAJRElSICpkID0gb3BlbmRpcigiLiIpOw0KPiAJZGlyZW50ICpmaWxlX2VudDsN
Cj4gCXdoaWxlKChmaWxlX2VudCA9IHJlYWRkaXIoZCkpICE9IE5VTEwpIHsNCj4gCQl1bmxpbmso
IGZpbGVfZW50LT5kX25hbWUpKQ0KPiAJfQ0KPiAJY2xvc2VkaXIoZCk7DQo+IA0KPiB3aGVyZSBp
dCBhY3R1YWxseSBuZWVkcyB0byBkbzoNCj4gDQo+IAlESVIgKmQgPSBvcGVuZGlyKCIuIik7DQo+
IAlkaXJlbnQgKmZpbGVfZW50Ow0KPiAJd2hpbGUoKGZpbGVfZW50ID0gcmVhZGRpcihkKSkgIT0g
TlVMTCkgew0KPiAJCXVubGluayggZmlsZV9lbnQtPmRfbmFtZSkpDQo+IA0KPiAJCWNsb3NlZGly
KGQpOw0KPiAJCWQgPSBvcGVuZGlyKCIuIik7DQo+IAl9DQo+IAljbG9zZWRpcihkKTsNCj4gDQo+
IEJ1dCBhZ2FpbiBjYXNlIG9uZSB1c2VkIHRvIHdvcmsgd2l0aCBvbGQgTkZTLiBBbmQgaXQgbG9v
a3MgbGlrZQ0KPiBpdCBpcyBub3QgU2VydmVyIGRlcGVuZGVudC4gV2Ugc2F3IHRoaXMgYm90aCB3
aXRoIEdhbmVzaGEgYXMgd2VsbA0KPiBhcyBrbmZzZA0KDQpOby4NCg0KSWYgdGhlIHNlcnZlciBz
dXBwb3J0cyBwZXJtYW5lbnQgcmVhZGRpciBjb29raWVzLCB0aGVuIHRoZXJlIGlzIG5vIG5lZWQN
CmZvciBvcGVuZGlyIGFmdGVyIHVubGluay4NCg0KSWYgdGhlIHNlcnZlciBkb2VzIG5vdCBoYXZl
IHBlcm1hbmVudCByZWFkZGlyIGNvb2tpZXMsIHRoZW4gb3VyIGNsaWVudA0KaGFzIG5ldmVyIHN1
cHBvcnRlZCBpdCBhbnl3YXkgKG5vciB3aWxsIGl0IGV2ZXIgZG8gc28pLiBUaGF0IHdob2xlDQon
Y29va2lldmVyZicgUkVBRERJUiBidWxsc2hpdCBoYXMgbmV2ZXIgcHJvdmlkZWQgYSB3b3JrYWJs
ZSBtb2RlbCBmb3IgYQ0KUE9TSVggY2xpZW50Li4uDQoNCg0KLS0gDQpUcm9uZCBNeWtsZWJ1c3QN
CkxpbnV4IE5GUyBjbGllbnQgbWFpbnRhaW5lcg0KDQpOZXRBcHANClRyb25kLk15a2xlYnVzdEBu
ZXRhcHAuY29tDQp3d3cubmV0YXBwLmNvbQ0KDQo=
Hi,
Thanks. I'm pretty sure that's not happening (haven't seen any such message from Linux 3.3.0-rc7, for example), but I'll retest.
Matt
----- "Trond Myklebust" <[email protected]> wrote:
--
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI 48104
http://linuxbox.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
T24gU2F0LCAyMDEyLTAzLTI0IGF0IDEyOjM0IC0wNDAwLCBNYXR0IFcuIEJlbmphbWluIHdyb3Rl
Og0KPiBIaSwNCj4gDQo+IEZvbGtzIHRlc3RpbmcgbGludXggbmZzICsgR2FuZXNoYSB3aXRoIGJv
bm5pZSsrIGhhdmUgbm90aWNlZCB0aGUgaXNzdWUgZGVzY3JpYmVkIGhlcmUgKGJ1dCBJIGRpZG4n
dCBzZWUgdHJhZmZpYyBvbiB0aGlzIGxpc3QpOg0KPiANCj4gaHR0cDovL2J1Z3MuY2VudG9zLm9y
Zy92aWV3LnBocD9pZD01NDk2DQo+IGh0dHBzOi8vYnVnemlsbGEucmVkaGF0LmNvbS9zaG93X2J1
Zy5jZ2k/aWQ9Nzg5NDUyDQo+IA0KPiBJZSwgYW4gYXBwbGljYXRpb24gd2hpY2ggdW5saW5rcyBv
biBhbiBvcGVuIChuZnMpIGRpcmVjdG9yeSBzdHJlYW0sIGFuZCBjb250aW51ZXMgdG8gcmVhZCBh
bmQgdW5saW5rIHByb2dyZXNzaXZlbHksIHdpbGwgZ2VuZXJhbGx5IG5vdCBzZWUgYWxsIHRoZSBl
bnRyaWVzIG9yaWdpbmFsbHkgaW4gdGhlIHN0cmVhbS4gIFJlb3BlbmluZyB0aGUgc3RyZWFtIGl2
ZXMgdGhlICJjb3JyZWN0IiByZXN1bHQuICBTbywgYXBwYXJlbnRseSBvbGRlciBjbGllbnRzIHdl
cmUgbW9yZSBmb3JnaXZpbmcsIGFzIG1lbnRpb25lZCBpbiB0aGUgbGlua3MuICBJcyB0aGUgY3Vy
cmVudCBiZWhhdmlvciBvbmUgdGhlIGNsaWVudCBpcyBpbnRlbmRpbmc/DQo+IA0KPiBUaGFua3Ms
DQo+IA0KPiBNYXR0DQoNCldoYXQgaXMgc28gcGFydGljdWxhciBhYm91dCB0aGUgZ2FuZXNoYSBy
ZWFkZGlyIGltcGxlbWVudGF0aW9uPw0KDQotLSANClRyb25kIE15a2xlYnVzdA0KTGludXggTkZT
IGNsaWVudCBtYWludGFpbmVyDQoNCk5ldEFwcA0KVHJvbmQuTXlrbGVidXN0QG5ldGFwcC5jb20N
Cnd3dy5uZXRhcHAuY29tDQoNCg==
On 03/26/2012 11:25 AM, Myklebust, Trond wrote:
> On Mon, 2012-03-26 at 11:17 -0700, Boaz Harrosh wrote:
>> On 03/24/2012 10:12 AM, Myklebust, Trond wrote:
>>
>>
>> It's the new (post RHEL 6.0 Kernel) NFS need for opendir after an unlink.
>
> What?
>
The links below report on regression in RHEL 6.2 which have a new NFS
implementation where 6.0 used to be fine.
(Or it's what I understood from the reporters)
<snip>
>
> No.
>
> If the server supports permanent readdir cookies, then there is no need
> for opendir after unlink.
>
> If the server does not have permanent readdir cookies, then our client
> has never supported it anyway (nor will it ever do so). That whole
> 'cookieverf' READDIR bullshit has never provided a workable model for a
> POSIX client...
Thanks Trond for the explanation. Forgive my slowness. So you are
saying that with knfsd it's actually filesystem dependent and maybe
the reporters did not compare apples-to-apples and the difference is
in the behind file system.
Matt I'm sure Trond is right with regard to Ganesha, because we
send a readdir index and a cookieverf, which will change after
unlink. Since the application does not call opendir again the
client will send the old cookies, which is now a different file
or might get out-of-bounds for Ganesha.
Let me think about it a bit. I'm sure we can send a Better
64bit cookie, which will be more persistent, even across unlinks.
>
>
Thanks
Boaz
Hi,
----- "Trond Myklebust" <[email protected]> wrote:
> On Mon, 2012-03-26 at 14:55 -0400, Matt W. Benjamin wrote:
> > Hi,
> >
> > Boaz: we do not any longer send a readdir index. We do send a
> cookieverf.
>
> Uhh... That would be so broken, that I can't imagine that is true...
I think you're misreading my statement. My intended meaning had to do with Ganesha's generator
for the cookies themselves. It's not relevant.
>
> > Fist of all, I haven't established that the issue we're actually
> observing is caused
> > by the Linux client sending old cookies to readdir(something).
> However, if it is,
> > it's in no way better to try to make cookies "more persistent." Nor
> should the Linux
> > client be expecting it. The assumption is simply flawed. The
> protocol introduced
> > cookie verifier (a LONG time ago) for a reason.
>
> Bullshit... The cookie verifier is unimplementable. Feel free to try
> or
> to show me an implementation that works. However I refuse to waste
> any
> more of my time trying to make that crap work, having already wasted
> several months of my life doing just that.
There must be something I'm missing here, but ok.
>
> --
> Trond Myklebust
> Linux NFS client maintainer
>
> NetApp
> [email protected]
> http://www.netapp.com
--
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI 48104
http://linuxbox.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
T24gU2F0LCAyMDEyLTAzLTI0IGF0IDEyOjUzIC0wNDAwLCBNYXR0IFcuIEJlbmphbWluIHdyb3Rl
Og0KPiBIaSwNCj4gDQo+IEkgZG9uJ3QgdGhpbmsgYW55dGhpbmcgaXMuICBPciwgcGVvcGxlIG9y
aWdpbmFsbHkgcmVwb3J0ZWQgdGhlIGJlaGF2aW9yIGFnYWluc3Qga25mc2QuDQo+IA0KPiBNYXR0
DQoNClRoZXJlIGlzIGEga25vd24gaXNzdWUgd2l0aCBleHQyLzMvNCBnZW5lcmF0aW5nIG5vbi11
bmlxdWUgcmVhZGRpcg0KY29va2llcy4gSXQgcmFyZWx5IGhpdHMgeW91IHdoZW4geW91IGFyZSBj
cmVhdGluZyBzbWFsbCBkaXJlY3RvcmllcywgYnV0DQppdCBmcmVxdWVudGx5IGhpdHMgeW91IHdp
dGggbGFyZ2VyIG9uZXMuIEEgZml4IGlzIHVuZGVyd2F5IHRoYXQgc2hvdWxkDQpzaWduaWZpY2Fu
dGx5IHJlZHVjZSB0aGUgZnJlcXVlbmN5IG9mIGNvb2tpZSBjb2xsaXNpb25zLg0KDQpSZWNlbnQg
TkZTIGNsaWVudHMgd2lsbCBhY3R1YWxseSBkZXRlY3QgdGhlIHByZXNlbmNlIG9mIHRob3NlIGNv
b2tpZQ0KbG9vcHMsIGFuZCBsb2cgdGhlbSBpbiB0aGUga2VybmVsIHN5c2xvZy4gVGhhdCB3b3Vs
ZCB0aGVyZWZvcmUgYmUgdGhlDQpmaXJzdCB0aGluZyB0aGF0IEknZCBjaGVjayBpZiBjb25mcm9u
dGVkIHdpdGggdGhpcyBraW5kIG9mIHByb2JsZW0uDQoNCkNoZWVycw0KICBUcm9uZA0KDQo+IC0t
LS0tICJUcm9uZCBNeWtsZWJ1c3QiIDxUcm9uZC5NeWtsZWJ1c3RAbmV0YXBwLmNvbT4gd3JvdGU6
DQo+IA0KPiA+IE9uIFNhdCwgMjAxMi0wMy0yNCBhdCAxMjozNCAtMDQwMCwgTWF0dCBXLiBCZW5q
YW1pbiB3cm90ZToNCj4gPiA+IEhpLA0KPiA+ID4gDQo+ID4gPiBGb2xrcyB0ZXN0aW5nIGxpbnV4
IG5mcyArIEdhbmVzaGEgd2l0aCBib25uaWUrKyBoYXZlIG5vdGljZWQgdGhlDQo+ID4gaXNzdWUg
ZGVzY3JpYmVkIGhlcmUgKGJ1dCBJIGRpZG4ndCBzZWUgdHJhZmZpYyBvbiB0aGlzIGxpc3QpOg0K
PiA+ID4gDQo+ID4gPiBodHRwOi8vYnVncy5jZW50b3Mub3JnL3ZpZXcucGhwP2lkPTU0OTYNCj4g
PiA+IGh0dHBzOi8vYnVnemlsbGEucmVkaGF0LmNvbS9zaG93X2J1Zy5jZ2k/aWQ9Nzg5NDUyDQo+
ID4gPiANCj4gPiA+IEllLCBhbiBhcHBsaWNhdGlvbiB3aGljaCB1bmxpbmtzIG9uIGFuIG9wZW4g
KG5mcykgZGlyZWN0b3J5IHN0cmVhbSwNCj4gPiBhbmQgY29udGludWVzIHRvIHJlYWQgYW5kIHVu
bGluayBwcm9ncmVzc2l2ZWx5LCB3aWxsIGdlbmVyYWxseSBub3Qgc2VlDQo+ID4gYWxsIHRoZSBl
bnRyaWVzIG9yaWdpbmFsbHkgaW4gdGhlIHN0cmVhbS4gIFJlb3BlbmluZyB0aGUgc3RyZWFtIGl2
ZXMNCj4gPiB0aGUgImNvcnJlY3QiIHJlc3VsdC4gIFNvLCBhcHBhcmVudGx5IG9sZGVyIGNsaWVu
dHMgd2VyZSBtb3JlDQo+ID4gZm9yZ2l2aW5nLCBhcyBtZW50aW9uZWQgaW4gdGhlIGxpbmtzLiAg
SXMgdGhlIGN1cnJlbnQgYmVoYXZpb3Igb25lIHRoZQ0KPiA+IGNsaWVudCBpcyBpbnRlbmRpbmc/
DQo+ID4gPiANCj4gPiA+IFRoYW5rcywNCj4gPiA+IA0KPiA+ID4gTWF0dA0KPiA+IA0KPiA+IFdo
YXQgaXMgc28gcGFydGljdWxhciBhYm91dCB0aGUgZ2FuZXNoYSByZWFkZGlyIGltcGxlbWVudGF0
aW9uPw0KPiA+IA0KPiA+IC0tIA0KPiA+IFRyb25kIE15a2xlYnVzdA0KPiA+IExpbnV4IE5GUyBj
bGllbnQgbWFpbnRhaW5lcg0KPiA+IA0KPiA+IE5ldEFwcA0KPiA+IFRyb25kLk15a2xlYnVzdEBu
ZXRhcHAuY29tDQo+ID4gd3d3Lm5ldGFwcC5jb20NCj4gDQoNCi0tIA0KVHJvbmQgTXlrbGVidXN0
DQpMaW51eCBORlMgY2xpZW50IG1haW50YWluZXINCg0KTmV0QXBwDQpUcm9uZC5NeWtsZWJ1c3RA
bmV0YXBwLmNvbQ0Kd3d3Lm5ldGFwcC5jb20NCg0K
On 03/26/2012 11:55 AM, Matt W. Benjamin wrote:
> Hi,
>
> Boaz: we do not any longer send a readdir index. We do send a cookieverf.
>
> Fist of all, I haven't established that the issue we're actually observing is caused
> by the Linux client sending old cookies to readdir(something). However, if it is,
> it's in no way better to try to make cookies "more persistent." Nor should the Linux
> client be expecting it. The assumption is simply flawed. The protocol introduced
> cookie verifier (a LONG time ago) for a reason.
>
> Matt
Please don't top-post in a Linux-mailing lists. Us Linux guys are not used to it
and get confused.
Regarding the cookie-verifier it looks like the Linux client never supported
that. So even if it will start to, from here on, we are in trouble with old
clients.
Since we are in the business of being bug backward compatible then fixing the
client will not help. Because you must agree the easiest is to fix bonnie++
but that's a privilege we do not have.
I think we should imitate the behavior of knfsd+exfs3 I'm sure it's easy for
Ganesha to fix it. Lets talk about it in this week call. I have ideas that
will not change current code, just a small enhancement.
Thanks
Boaz
> ----- "Boaz Harrosh" <[email protected]> wrote:
>
>> On 03/26/2012 11:25 AM, Myklebust, Trond wrote:
>>
>>> On Mon, 2012-03-26 at 11:17 -0700, Boaz Harrosh wrote:
>>>> On 03/24/2012 10:12 AM, Myklebust, Trond wrote:
>>>>
>>>>
>>>> It's the new (post RHEL 6.0 Kernel) NFS need for opendir after an
>> unlink.
>>>
>>> What?
>>>
>>
>>
>> The links below report on regression in RHEL 6.2 which have a new NFS
>> implementation where 6.0 used to be fine.
>> (Or it's what I understood from the reporters)
>>
>> <snip>
>>
>>>
>>
>>> No.
>>>
>>> If the server supports permanent readdir cookies, then there is no
>> need
>>> for opendir after unlink.
>>>
>>> If the server does not have permanent readdir cookies, then our
>> client
>>> has never supported it anyway (nor will it ever do so). That whole
>>> 'cookieverf' READDIR bullshit has never provided a workable model
>> for a
>>> POSIX client...
>>
>>
>> Thanks Trond for the explanation. Forgive my slowness. So you are
>> saying that with knfsd it's actually filesystem dependent and maybe
>> the reporters did not compare apples-to-apples and the difference is
>> in the behind file system.
>>
>> Matt I'm sure Trond is right with regard to Ganesha, because we
>> send a readdir index and a cookieverf, which will change after
>> unlink. Since the application does not call opendir again the
>> client will send the old cookies, which is now a different file
>> or might get out-of-bounds for Ganesha.
>>
>> Let me think about it a bit. I'm sure we can send a Better
>> 64bit cookie, which will be more persistent, even across unlinks.
>>
>>>
>>>
>>
>>
>> Thanks
>> Boaz
>
UHJlY2lzZWx5LCBUcm9uZCEgIDotKQ0KDQpUaGF0J3Mgd2h5IHRoZSBjb29raWUgdmVyaWZpZXIg
dHVybmVkIG91dCB0byBiZSBhIHVudGVuYWJsZSBpZGVhLg0KDQpUaGUgYXBwbGljYXRpb24gaXRz
ZWxmLCBub3QganVzdCB0aGUgYXBwbGljYXRpb24gbGV2ZWwsIG11c3QgcGFydGljaXBhdGUgaW4g
dGhlIHJlY292ZXJ5LiAgSWYgdGhlIHJtIGNvbW1hbmQgd2FudHMgdG8gZW5zdXJlIHRoYXQgaXQg
cmVtb3ZlcyBldmVyeSBlbnRyeSBpbiBhIGRpcmVjdG9yeSBiZWZvcmUgYXR0ZW1wdGluZyB0byBy
ZW1vdmUgdGhlIGRpcmVjdG9yeSBpdHNlbGYsIHRoZW4gaXQgbXVzdCBhcnJhbmdlIHRvIGRvIHNv
IGFuZCBjYW4ndCByZWFsbHkgYXNzdW1lIHRoYXQgZGlyZWN0b3J5IGNvb2tpZXMgY2FuIG5ldmVy
IGJlY29tZSBpbnZhbGlkLiAgUGVyaGFwcyBpdCBjb3VsZCBvbiBhIHB1cmVseSBQT1NJWCBzeXN0
ZW0sIEkgZHVubm8sIGJ1dCBJIGFtIG5vdCBzdXJlIHRoYXQgYSBwdXJlIFBPU0lYIHN5c3RlbSBl
eGlzdHMgYW55d2hlcmUuDQoNCgkJcHMNCg0KDQotLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0K
RnJvbTogTXlrbGVidXN0LCBUcm9uZCBbbWFpbHRvOlRyb25kLk15a2xlYnVzdEBuZXRhcHAuY29t
XSANClNlbnQ6IEZyaWRheSwgTWFyY2ggMzAsIDIwMTIgMTE6NDcgQU0NClRvOiBQZXRlciBTdGF1
YmFjaA0KQ2M6IE1hdHQgVy4gQmVuamFtaW47IEJvYXogSGFycm9zaDsgbGludXgtbmZzOyBHYW5l
c2hhIE5GUyBMaXN0DQpTdWJqZWN0OiBSRTogdW5saW5rIHdpdGhpbiBhbiBvcGVuIGRpcmVjdG9y
eSBzdHJlYW0NCg0KT24gRnJpLCAyMDEyLTAzLTMwIGF0IDExOjI0IC0wNDAwLCBQZXRlciBTdGF1
YmFjaCB3cm90ZToNCj4gWWVzLCBobW1tLCB1bW1tLi4uDQo+IA0KPiBBIGxvbmcgdGltZSwgSSB0
aG91Z2h0IHRoYXQgdGhlIHJlYWRkaXIgY29va2llIHZlcmlmaWVyIHNvdW5kZWQgbGlrZSBhIGdv
b2QgaWRlYS4gIEFmdGVyIGFsbCwgaWYgZGlyZWN0b3JpZXMgY2FuIGJlIGNvbXBhY3RlZCwgaXQg
d291bGQgYmUgbmljZSB0byBiZSBhYmxlIHRvIHRlbGwgd2hlbiB0aGUgY29va2llIGhhcyBiZWNv
bWUgaW52YWxpZCwgcmlnaHQ/ICBTbywgSSBhZGRlZCBpdCB0byB0aGUgTkZTdjMgcHJvdG9jb2wu
DQo+IA0KPiBJdCB3YXMgb25seSBhZnRlcndhcmRzLCBpbiBhdHRlbXB0aW5nIHRvIGltcGxlbWVu
dCBpdCwgdGhhdCB0aGUgcmVhbCB0cnV0aCB3YXMgZGlzY292ZXJlZC4gIFRoaXMgaXMgdGhhdCB0
aGUgaW50ZXJmYWNlcyBhcmUgbm90IHN1ZmZpY2llbnQgdG8gYmUgYWJsZSB0byBjb252ZXkgdGhp
cyBpbmZvcm1hdGlvbiBzdWZmaWNpZW50bHkgYW5kIHRvIGJlIGFibGUgdG8gcmVjb3ZlciBmcm9t
IHRoZSBzaXR1YXRpb24uICBDdXJyZW50bHksIHJlY292ZXJ5IHJlcXVpcmUgYWN0aW9uIG9uIHRo
ZSBwYXJ0IG9mIHRoZSBhcHBsaWNhdGlvbi4NCj4gDQo+IEluIHJldHJvc3BlY3QsIEkgd291bGRu
J3QgYWRkIHRoZSBjb29raWUgdmVyaWZpZXIgaWYgSSB3YXMgZG9pbmcgaXQgYWdhaW4uDQo+IA0K
PiBUaGFua3MgZm9yIHRoZSBzdXBwb3J0LCB0aG91Z2guICA6LSkNCj4gDQo+IEkgZG8gZGlzYWdy
ZWUgd2l0aCBUcm9uZCB0aG91Z2gsIGluIHRoYXQgYXBwbGljYXRpb25zIG11c3QgYmUgY29kZWQg
dG8gYmUgYWJsZSB0byBoYW5kbGUgZGlyZWN0b3JpZXMgd2hlcmUgY29va2llcyBjYW4gYmVjb21l
IGludmFsaWRhdGVkLiAgVGhhdCdzIHdoeSBybSBhbmQgc3VjaCBoYXZlIHRvIGJlIGNvZGVkIHRv
IGtlZXAgdHJhY2sgb2YgdGhlaXIgbG9jYXRpb24gaW4gdGhlIGRpcmVjdG9yeSBieSBtb3JlIHRo
YW4ganVzdCBkX29mZi4NCg0KDQpUaGUgcHJvYmxlbSB3aXRoIHRoYXQgaXMgdGhhdCB0aGVyZSBp
cyBubyBzdGFuZGFyZCBmb3IgaG93IGFwcGxpY2F0aW9ucyBzaG91bGQgcHJvY2VlZCB0byBkbyB0
aGlzLCBhbmQgc28gdGhlcmUgYXJlIG5vIGd1aWRlbGluZXMgZm9yIGhvdyB0aGUgY2xpZW50IHNo
b3VsZCBiZWhhdmUuDQoNCldvcnNlLCBpZiB5b3UndmUgbG9va2VkIGF0IHRoZSBnbGliYyByZWFk
ZGlyIGxpYnJhcnkgY29kZSwgeW91J2xsIGhhdmUgbm90aWNlZCB0aGF0IGl0IF9kZXBlbmRzXyBv
biBiZWluZyBhYmxlIHRvIHVzZSBhIHN0cmljdGx5IFBPU0lYLWNvbmZvcm1pbmcgdmVyc2lvbiBv
ZiB0ZWxsZGlyKCkvc2Vla2RpcigpLiBJZiBpdCBjYW4ndCByZXdpbmQgdG8gYSBwcmV2aW91c2x5
IHNhdmVkIG9mZnNldCwgdGhlbiBpdCBjYW4gZW5kIHVwIHNraXBwaW5nIGVudHJpZXMuDQoNClNv
IG5vLCBJIGNhbid0IHJlbHkgb24gYXBwbGljYXRpb25zIGVpdGhlci4uLg0KDQo+IAkJcHMNCj4g
DQo+IA0KPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBsaW51eC1uZnMtb3du
ZXJAdmdlci5rZXJuZWwub3JnIA0KPiBbbWFpbHRvOmxpbnV4LW5mcy1vd25lckB2Z2VyLmtlcm5l
bC5vcmddIE9uIEJlaGFsZiBPZiBNYXR0IFcuIEJlbmphbWluDQo+IFNlbnQ6IE1vbmRheSwgTWFy
Y2ggMjYsIDIwMTIgMjo1NSBQTQ0KPiBUbzogQm9heiBIYXJyb3NoDQo+IENjOiBsaW51eC1uZnM7
IEdhbmVzaGEgTkZTIExpc3Q7IFRyb25kIE15a2xlYnVzdA0KPiBTdWJqZWN0OiBSZTogdW5saW5r
IHdpdGhpbiBhbiBvcGVuIGRpcmVjdG9yeSBzdHJlYW0NCj4gDQo+IEhpLA0KPiANCj4gQm9hejog
IHdlIGRvIG5vdCBhbnkgbG9uZ2VyIHNlbmQgYSByZWFkZGlyIGluZGV4LiAgV2UgZG8gc2VuZCBh
IGNvb2tpZXZlcmYuDQo+IA0KPiBGaXN0IG9mIGFsbCwgSSBoYXZlbid0IGVzdGFibGlzaGVkIHRo
YXQgdGhlIGlzc3VlIHdlJ3JlIGFjdHVhbGx5IG9ic2VydmluZyBpcyBjYXVzZWQgYnkgdGhlIExp
bnV4IGNsaWVudCBzZW5kaW5nIG9sZCBjb29raWVzIHRvIHJlYWRkaXIoc29tZXRoaW5nKS4gIEhv
d2V2ZXIsIGlmIGl0IGlzLCBpdCdzIGluIG5vIHdheSBiZXR0ZXIgdG8gdHJ5IHRvIG1ha2UgY29v
a2llcyAibW9yZSBwZXJzaXN0ZW50LiIgIE5vciBzaG91bGQgdGhlIExpbnV4IGNsaWVudCBiZSBl
eHBlY3RpbmcgaXQuICBUaGUgYXNzdW1wdGlvbiBpcyBzaW1wbHkgZmxhd2VkLiAgVGhlIHByb3Rv
Y29sIGludHJvZHVjZWQgY29va2llIHZlcmlmaWVyIChhIExPTkcgdGltZSBhZ28pIGZvciBhIHJl
YXNvbi4NCj4gDQo+IE1hdHQNCj4gLS0tLS0gIkJvYXogSGFycm9zaCIgPGJoYXJyb3NoQHBhbmFz
YXMuY29tPiB3cm90ZToNCj4gDQo+ID4gT24gMDMvMjYvMjAxMiAxMToyNSBBTSwgTXlrbGVidXN0
LCBUcm9uZCB3cm90ZToNCj4gPiANCj4gPiA+IE9uIE1vbiwgMjAxMi0wMy0yNiBhdCAxMToxNyAt
MDcwMCwgQm9heiBIYXJyb3NoIHdyb3RlOg0KPiA+ID4+IE9uIDAzLzI0LzIwMTIgMTA6MTIgQU0s
IE15a2xlYnVzdCwgVHJvbmQgd3JvdGU6DQo+ID4gPj4NCj4gPiA+Pg0KPiA+ID4+IEl0J3MgdGhl
IG5ldyAocG9zdCBSSEVMIDYuMCBLZXJuZWwpIE5GUyBuZWVkIGZvciBvcGVuZGlyIGFmdGVyIGFu
DQo+ID4gdW5saW5rLg0KPiA+ID4gDQo+ID4gPiBXaGF0Pw0KPiA+ID4gDQo+ID4gDQo+ID4gDQo+
ID4gVGhlIGxpbmtzIGJlbG93IHJlcG9ydCBvbiByZWdyZXNzaW9uIGluIFJIRUwgNi4yIHdoaWNo
IGhhdmUgYSBuZXcgDQo+ID4gTkZTIGltcGxlbWVudGF0aW9uIHdoZXJlICA2LjAgdXNlZCB0byBi
ZSBmaW5lLg0KPiA+IChPciBpdCdzIHdoYXQgSSB1bmRlcnN0b29kIGZyb20gdGhlIHJlcG9ydGVy
cykNCj4gPiANCj4gPiA8c25pcD4NCj4gPiANCj4gPiA+IA0KPiA+IA0KPiA+ID4gTm8uDQo+ID4g
PiANCj4gPiA+IElmIHRoZSBzZXJ2ZXIgc3VwcG9ydHMgcGVybWFuZW50IHJlYWRkaXIgY29va2ll
cywgdGhlbiB0aGVyZSBpcyBubw0KPiA+IG5lZWQNCj4gPiA+IGZvciBvcGVuZGlyIGFmdGVyIHVu
bGluay4NCj4gPiA+IA0KPiA+ID4gSWYgdGhlIHNlcnZlciBkb2VzIG5vdCBoYXZlIHBlcm1hbmVu
dCByZWFkZGlyIGNvb2tpZXMsIHRoZW4gb3VyDQo+ID4gY2xpZW50DQo+ID4gPiBoYXMgbmV2ZXIg
c3VwcG9ydGVkIGl0IGFueXdheSAobm9yIHdpbGwgaXQgZXZlciBkbyBzbykuIFRoYXQgd2hvbGUg
DQo+ID4gPiAnY29va2lldmVyZicgUkVBRERJUiBidWxsc2hpdCBoYXMgbmV2ZXIgcHJvdmlkZWQg
YSB3b3JrYWJsZSBtb2RlbA0KPiA+IGZvciBhDQo+ID4gPiBQT1NJWCBjbGllbnQuLi4NCj4gPiAN
Cj4gPiANCj4gPiBUaGFua3MgVHJvbmQgZm9yIHRoZSBleHBsYW5hdGlvbi4gRm9yZ2l2ZSBteSBz
bG93bmVzcy4gU28geW91IGFyZSANCj4gPiBzYXlpbmcgdGhhdCB3aXRoIGtuZnNkIGl0J3MgYWN0
dWFsbHkgZmlsZXN5c3RlbSBkZXBlbmRlbnQgYW5kIG1heWJlIA0KPiA+IHRoZSByZXBvcnRlcnMg
ZGlkIG5vdCBjb21wYXJlIGFwcGxlcy10by1hcHBsZXMgYW5kIHRoZSBkaWZmZXJlbmNlIGlzIA0K
PiA+IGluIHRoZSBiZWhpbmQgZmlsZSBzeXN0ZW0uDQo+ID4gDQo+ID4gTWF0dCBJJ20gc3VyZSBU
cm9uZCBpcyByaWdodCB3aXRoIHJlZ2FyZCB0byBHYW5lc2hhLCBiZWNhdXNlIHdlIHNlbmQgDQo+
ID4gYSByZWFkZGlyIGluZGV4IGFuZCBhIGNvb2tpZXZlcmYsIHdoaWNoIHdpbGwgY2hhbmdlIGFm
dGVyIHVubGluay4gDQo+ID4gU2luY2UgdGhlIGFwcGxpY2F0aW9uIGRvZXMgbm90IGNhbGwgb3Bl
bmRpciBhZ2FpbiB0aGUgY2xpZW50IHdpbGwgDQo+ID4gc2VuZCB0aGUgb2xkIGNvb2tpZXMsIHdo
aWNoIGlzIG5vdyBhIGRpZmZlcmVudCBmaWxlIG9yIG1pZ2h0IGdldCANCj4gPiBvdXQtb2YtYm91
bmRzIGZvciBHYW5lc2hhLg0KPiA+IA0KPiA+IExldCBtZSB0aGluayBhYm91dCBpdCBhIGJpdC4g
SSdtIHN1cmUgd2UgY2FuIHNlbmQgYSBCZXR0ZXIgNjRiaXQgDQo+ID4gY29va2llLCB3aGljaCB3
aWxsIGJlIG1vcmUgcGVyc2lzdGVudCwgZXZlbiBhY3Jvc3MgdW5saW5rcy4NCj4gPiANCj4gPiA+
IA0KPiA+ID4gDQo+ID4gDQo+ID4gDQo+ID4gVGhhbmtzDQo+ID4gQm9heg0KPiANCj4gLS0NCj4g
TWF0dCBCZW5qYW1pbg0KPiBUaGUgTGludXggQm94DQo+IDIwNiBTb3V0aCBGaWZ0aCBBdmUuIFN1
aXRlIDE1MA0KPiBBbm4gQXJib3IsIE1JICA0ODEwNA0KPiANCj4gaHR0cDovL2xpbnV4Ym94LmNv
bQ0KPiANCj4gdGVsLiA3MzQtNzYxLTQ2ODkNCj4gZmF4LiA3MzQtNzY5LTg5MzgNCj4gY2VsLiA3
MzQtMjE2LTUzMDkNCj4gLS0NCj4gVG8gdW5zdWJzY3JpYmUgZnJvbSB0aGlzIGxpc3Q6IHNlbmQg
dGhlIGxpbmUgInVuc3Vic2NyaWJlIGxpbnV4LW5mcyIgDQo+IGluIHRoZSBib2R5IG9mIGEgbWVz
c2FnZSB0byBtYWpvcmRvbW9Admdlci5rZXJuZWwub3JnIE1vcmUgbWFqb3Jkb21vIA0KPiBpbmZv
IGF0ICBodHRwOi8vdmdlci5rZXJuZWwub3JnL21ham9yZG9tby1pbmZvLmh0bWwNCg0KLS0NClRy
b25kIE15a2xlYnVzdA0KTGludXggTkZTIGNsaWVudCBtYWludGFpbmVyDQoNCk5ldEFwcA0KVHJv
bmQuTXlrbGVidXN0QG5ldGFwcC5jb20NCnd3dy5uZXRhcHAuY29tDQoNCg==
Hi,
Boaz: we do not any longer send a readdir index. We do send a cookieverf.
Fist of all, I haven't established that the issue we're actually observing is caused
by the Linux client sending old cookies to readdir(something). However, if it is,
it's in no way better to try to make cookies "more persistent." Nor should the Linux
client be expecting it. The assumption is simply flawed. The protocol introduced
cookie verifier (a LONG time ago) for a reason.
Matt
----- "Boaz Harrosh" <[email protected]> wrote:
> On 03/26/2012 11:25 AM, Myklebust, Trond wrote:
>
> > On Mon, 2012-03-26 at 11:17 -0700, Boaz Harrosh wrote:
> >> On 03/24/2012 10:12 AM, Myklebust, Trond wrote:
> >>
> >>
> >> It's the new (post RHEL 6.0 Kernel) NFS need for opendir after an
> unlink.
> >
> > What?
> >
>
>
> The links below report on regression in RHEL 6.2 which have a new NFS
> implementation where 6.0 used to be fine.
> (Or it's what I understood from the reporters)
>
> <snip>
>
> >
>
> > No.
> >
> > If the server supports permanent readdir cookies, then there is no
> need
> > for opendir after unlink.
> >
> > If the server does not have permanent readdir cookies, then our
> client
> > has never supported it anyway (nor will it ever do so). That whole
> > 'cookieverf' READDIR bullshit has never provided a workable model
> for a
> > POSIX client...
>
>
> Thanks Trond for the explanation. Forgive my slowness. So you are
> saying that with knfsd it's actually filesystem dependent and maybe
> the reporters did not compare apples-to-apples and the difference is
> in the behind file system.
>
> Matt I'm sure Trond is right with regard to Ganesha, because we
> send a readdir index and a cookieverf, which will change after
> unlink. Since the application does not call opendir again the
> client will send the old cookies, which is now a different file
> or might get out-of-bounds for Ganesha.
>
> Let me think about it a bit. I'm sure we can send a Better
> 64bit cookie, which will be more persistent, even across unlinks.
>
> >
> >
>
>
> Thanks
> Boaz
--
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI 48104
http://linuxbox.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
T24gRnJpLCAyMDEyLTAzLTMwIGF0IDExOjI0IC0wNDAwLCBQZXRlciBTdGF1YmFjaCB3cm90ZToN
Cj4gWWVzLCBobW1tLCB1bW1tLi4uDQo+IA0KPiBBIGxvbmcgdGltZSwgSSB0aG91Z2h0IHRoYXQg
dGhlIHJlYWRkaXIgY29va2llIHZlcmlmaWVyIHNvdW5kZWQgbGlrZSBhIGdvb2QgaWRlYS4gIEFm
dGVyIGFsbCwgaWYgZGlyZWN0b3JpZXMgY2FuIGJlIGNvbXBhY3RlZCwgaXQgd291bGQgYmUgbmlj
ZSB0byBiZSBhYmxlIHRvIHRlbGwgd2hlbiB0aGUgY29va2llIGhhcyBiZWNvbWUgaW52YWxpZCwg
cmlnaHQ/ICBTbywgSSBhZGRlZCBpdCB0byB0aGUgTkZTdjMgcHJvdG9jb2wuDQo+IA0KPiBJdCB3
YXMgb25seSBhZnRlcndhcmRzLCBpbiBhdHRlbXB0aW5nIHRvIGltcGxlbWVudCBpdCwgdGhhdCB0
aGUgcmVhbCB0cnV0aCB3YXMgZGlzY292ZXJlZC4gIFRoaXMgaXMgdGhhdCB0aGUgaW50ZXJmYWNl
cyBhcmUgbm90IHN1ZmZpY2llbnQgdG8gYmUgYWJsZSB0byBjb252ZXkgdGhpcyBpbmZvcm1hdGlv
biBzdWZmaWNpZW50bHkgYW5kIHRvIGJlIGFibGUgdG8gcmVjb3ZlciBmcm9tIHRoZSBzaXR1YXRp
b24uICBDdXJyZW50bHksIHJlY292ZXJ5IHJlcXVpcmUgYWN0aW9uIG9uIHRoZSBwYXJ0IG9mIHRo
ZSBhcHBsaWNhdGlvbi4NCj4gDQo+IEluIHJldHJvc3BlY3QsIEkgd291bGRuJ3QgYWRkIHRoZSBj
b29raWUgdmVyaWZpZXIgaWYgSSB3YXMgZG9pbmcgaXQgYWdhaW4uDQo+IA0KPiBUaGFua3MgZm9y
IHRoZSBzdXBwb3J0LCB0aG91Z2guICA6LSkNCj4gDQo+IEkgZG8gZGlzYWdyZWUgd2l0aCBUcm9u
ZCB0aG91Z2gsIGluIHRoYXQgYXBwbGljYXRpb25zIG11c3QgYmUgY29kZWQgdG8gYmUgYWJsZSB0
byBoYW5kbGUgZGlyZWN0b3JpZXMgd2hlcmUgY29va2llcyBjYW4gYmVjb21lIGludmFsaWRhdGVk
LiAgVGhhdCdzIHdoeSBybSBhbmQgc3VjaCBoYXZlIHRvIGJlIGNvZGVkIHRvIGtlZXAgdHJhY2sg
b2YgdGhlaXIgbG9jYXRpb24gaW4gdGhlIGRpcmVjdG9yeSBieSBtb3JlIHRoYW4ganVzdCBkX29m
Zi4NCg0KVGhlIHByb2JsZW0gd2l0aCB0aGF0IGlzIHRoYXQgdGhlcmUgaXMgbm8gc3RhbmRhcmQg
Zm9yIGhvdyBhcHBsaWNhdGlvbnMNCnNob3VsZCBwcm9jZWVkIHRvIGRvIHRoaXMsIGFuZCBzbyB0
aGVyZSBhcmUgbm8gZ3VpZGVsaW5lcyBmb3IgaG93IHRoZQ0KY2xpZW50IHNob3VsZCBiZWhhdmUu
DQoNCldvcnNlLCBpZiB5b3UndmUgbG9va2VkIGF0IHRoZSBnbGliYyByZWFkZGlyIGxpYnJhcnkg
Y29kZSwgeW91J2xsIGhhdmUNCm5vdGljZWQgdGhhdCBpdCBfZGVwZW5kc18gb24gYmVpbmcgYWJs
ZSB0byB1c2UgYSBzdHJpY3RseQ0KUE9TSVgtY29uZm9ybWluZyB2ZXJzaW9uIG9mIHRlbGxkaXIo
KS9zZWVrZGlyKCkuIElmIGl0IGNhbid0IHJld2luZCB0byBhDQpwcmV2aW91c2x5IHNhdmVkIG9m
ZnNldCwgdGhlbiBpdCBjYW4gZW5kIHVwIHNraXBwaW5nIGVudHJpZXMuDQoNClNvIG5vLCBJIGNh
bid0IHJlbHkgb24gYXBwbGljYXRpb25zIGVpdGhlci4uLg0KDQo+IAkJcHMNCj4gDQo+IA0KPiAt
LS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBsaW51eC1uZnMtb3duZXJAdmdlci5r
ZXJuZWwub3JnIFttYWlsdG86bGludXgtbmZzLW93bmVyQHZnZXIua2VybmVsLm9yZ10gT24gQmVo
YWxmIE9mIE1hdHQgVy4gQmVuamFtaW4NCj4gU2VudDogTW9uZGF5LCBNYXJjaCAyNiwgMjAxMiAy
OjU1IFBNDQo+IFRvOiBCb2F6IEhhcnJvc2gNCj4gQ2M6IGxpbnV4LW5mczsgR2FuZXNoYSBORlMg
TGlzdDsgVHJvbmQgTXlrbGVidXN0DQo+IFN1YmplY3Q6IFJlOiB1bmxpbmsgd2l0aGluIGFuIG9w
ZW4gZGlyZWN0b3J5IHN0cmVhbQ0KPiANCj4gSGksDQo+IA0KPiBCb2F6OiAgd2UgZG8gbm90IGFu
eSBsb25nZXIgc2VuZCBhIHJlYWRkaXIgaW5kZXguICBXZSBkbyBzZW5kIGEgY29va2lldmVyZi4N
Cj4gDQo+IEZpc3Qgb2YgYWxsLCBJIGhhdmVuJ3QgZXN0YWJsaXNoZWQgdGhhdCB0aGUgaXNzdWUg
d2UncmUgYWN0dWFsbHkgb2JzZXJ2aW5nIGlzIGNhdXNlZCBieSB0aGUgTGludXggY2xpZW50IHNl
bmRpbmcgb2xkIGNvb2tpZXMgdG8gcmVhZGRpcihzb21ldGhpbmcpLiAgSG93ZXZlciwgaWYgaXQg
aXMsIGl0J3MgaW4gbm8gd2F5IGJldHRlciB0byB0cnkgdG8gbWFrZSBjb29raWVzICJtb3JlIHBl
cnNpc3RlbnQuIiAgTm9yIHNob3VsZCB0aGUgTGludXggY2xpZW50IGJlIGV4cGVjdGluZyBpdC4g
IFRoZSBhc3N1bXB0aW9uIGlzIHNpbXBseSBmbGF3ZWQuICBUaGUgcHJvdG9jb2wgaW50cm9kdWNl
ZCBjb29raWUgdmVyaWZpZXIgKGEgTE9ORyB0aW1lIGFnbykgZm9yIGEgcmVhc29uLg0KPiANCj4g
TWF0dA0KPiAtLS0tLSAiQm9heiBIYXJyb3NoIiA8YmhhcnJvc2hAcGFuYXNhcy5jb20+IHdyb3Rl
Og0KPiANCj4gPiBPbiAwMy8yNi8yMDEyIDExOjI1IEFNLCBNeWtsZWJ1c3QsIFRyb25kIHdyb3Rl
Og0KPiA+IA0KPiA+ID4gT24gTW9uLCAyMDEyLTAzLTI2IGF0IDExOjE3IC0wNzAwLCBCb2F6IEhh
cnJvc2ggd3JvdGU6DQo+ID4gPj4gT24gMDMvMjQvMjAxMiAxMDoxMiBBTSwgTXlrbGVidXN0LCBU
cm9uZCB3cm90ZToNCj4gPiA+Pg0KPiA+ID4+DQo+ID4gPj4gSXQncyB0aGUgbmV3IChwb3N0IFJI
RUwgNi4wIEtlcm5lbCkgTkZTIG5lZWQgZm9yIG9wZW5kaXIgYWZ0ZXIgYW4NCj4gPiB1bmxpbmsu
DQo+ID4gPiANCj4gPiA+IFdoYXQ/DQo+ID4gPiANCj4gPiANCj4gPiANCj4gPiBUaGUgbGlua3Mg
YmVsb3cgcmVwb3J0IG9uIHJlZ3Jlc3Npb24gaW4gUkhFTCA2LjIgd2hpY2ggaGF2ZSBhIG5ldyBO
RlMgDQo+ID4gaW1wbGVtZW50YXRpb24gd2hlcmUgIDYuMCB1c2VkIHRvIGJlIGZpbmUuDQo+ID4g
KE9yIGl0J3Mgd2hhdCBJIHVuZGVyc3Rvb2QgZnJvbSB0aGUgcmVwb3J0ZXJzKQ0KPiA+IA0KPiA+
IDxzbmlwPg0KPiA+IA0KPiA+ID4gDQo+ID4gDQo+ID4gPiBOby4NCj4gPiA+IA0KPiA+ID4gSWYg
dGhlIHNlcnZlciBzdXBwb3J0cyBwZXJtYW5lbnQgcmVhZGRpciBjb29raWVzLCB0aGVuIHRoZXJl
IGlzIG5vDQo+ID4gbmVlZA0KPiA+ID4gZm9yIG9wZW5kaXIgYWZ0ZXIgdW5saW5rLg0KPiA+ID4g
DQo+ID4gPiBJZiB0aGUgc2VydmVyIGRvZXMgbm90IGhhdmUgcGVybWFuZW50IHJlYWRkaXIgY29v
a2llcywgdGhlbiBvdXINCj4gPiBjbGllbnQNCj4gPiA+IGhhcyBuZXZlciBzdXBwb3J0ZWQgaXQg
YW55d2F5IChub3Igd2lsbCBpdCBldmVyIGRvIHNvKS4gVGhhdCB3aG9sZSANCj4gPiA+ICdjb29r
aWV2ZXJmJyBSRUFERElSIGJ1bGxzaGl0IGhhcyBuZXZlciBwcm92aWRlZCBhIHdvcmthYmxlIG1v
ZGVsDQo+ID4gZm9yIGENCj4gPiA+IFBPU0lYIGNsaWVudC4uLg0KPiA+IA0KPiA+IA0KPiA+IFRo
YW5rcyBUcm9uZCBmb3IgdGhlIGV4cGxhbmF0aW9uLiBGb3JnaXZlIG15IHNsb3duZXNzLiBTbyB5
b3UgYXJlIA0KPiA+IHNheWluZyB0aGF0IHdpdGgga25mc2QgaXQncyBhY3R1YWxseSBmaWxlc3lz
dGVtIGRlcGVuZGVudCBhbmQgbWF5YmUgDQo+ID4gdGhlIHJlcG9ydGVycyBkaWQgbm90IGNvbXBh
cmUgYXBwbGVzLXRvLWFwcGxlcyBhbmQgdGhlIGRpZmZlcmVuY2UgaXMgDQo+ID4gaW4gdGhlIGJl
aGluZCBmaWxlIHN5c3RlbS4NCj4gPiANCj4gPiBNYXR0IEknbSBzdXJlIFRyb25kIGlzIHJpZ2h0
IHdpdGggcmVnYXJkIHRvIEdhbmVzaGEsIGJlY2F1c2Ugd2Ugc2VuZCBhIA0KPiA+IHJlYWRkaXIg
aW5kZXggYW5kIGEgY29va2lldmVyZiwgd2hpY2ggd2lsbCBjaGFuZ2UgYWZ0ZXIgdW5saW5rLiBT
aW5jZSANCj4gPiB0aGUgYXBwbGljYXRpb24gZG9lcyBub3QgY2FsbCBvcGVuZGlyIGFnYWluIHRo
ZSBjbGllbnQgd2lsbCBzZW5kIHRoZSANCj4gPiBvbGQgY29va2llcywgd2hpY2ggaXMgbm93IGEg
ZGlmZmVyZW50IGZpbGUgb3IgbWlnaHQgZ2V0IG91dC1vZi1ib3VuZHMgDQo+ID4gZm9yIEdhbmVz
aGEuDQo+ID4gDQo+ID4gTGV0IG1lIHRoaW5rIGFib3V0IGl0IGEgYml0LiBJJ20gc3VyZSB3ZSBj
YW4gc2VuZCBhIEJldHRlciA2NGJpdCANCj4gPiBjb29raWUsIHdoaWNoIHdpbGwgYmUgbW9yZSBw
ZXJzaXN0ZW50LCBldmVuIGFjcm9zcyB1bmxpbmtzLg0KPiA+IA0KPiA+ID4gDQo+ID4gPiANCj4g
PiANCj4gPiANCj4gPiBUaGFua3MNCj4gPiBCb2F6DQo+IA0KPiAtLQ0KPiBNYXR0IEJlbmphbWlu
DQo+IFRoZSBMaW51eCBCb3gNCj4gMjA2IFNvdXRoIEZpZnRoIEF2ZS4gU3VpdGUgMTUwDQo+IEFu
biBBcmJvciwgTUkgIDQ4MTA0DQo+IA0KPiBodHRwOi8vbGludXhib3guY29tDQo+IA0KPiB0ZWwu
IDczNC03NjEtNDY4OQ0KPiBmYXguIDczNC03NjktODkzOA0KPiBjZWwuIDczNC0yMTYtNTMwOQ0K
PiAtLQ0KPiBUbyB1bnN1YnNjcmliZSBmcm9tIHRoaXMgbGlzdDogc2VuZCB0aGUgbGluZSAidW5z
dWJzY3JpYmUgbGludXgtbmZzIiBpbiB0aGUgYm9keSBvZiBhIG1lc3NhZ2UgdG8gbWFqb3Jkb21v
QHZnZXIua2VybmVsLm9yZyBNb3JlIG1ham9yZG9tbyBpbmZvIGF0ICBodHRwOi8vdmdlci5rZXJu
ZWwub3JnL21ham9yZG9tby1pbmZvLmh0bWwNCg0KLS0gDQpUcm9uZCBNeWtsZWJ1c3QNCkxpbnV4
IE5GUyBjbGllbnQgbWFpbnRhaW5lcg0KDQpOZXRBcHANClRyb25kLk15a2xlYnVzdEBuZXRhcHAu
Y29tDQp3d3cubmV0YXBwLmNvbQ0KDQo=
WWVzLCBobW1tLCB1bW1tLi4uDQoNCkEgbG9uZyB0aW1lLCBJIHRob3VnaHQgdGhhdCB0aGUgcmVh
ZGRpciBjb29raWUgdmVyaWZpZXIgc291bmRlZCBsaWtlIGEgZ29vZCBpZGVhLiAgQWZ0ZXIgYWxs
LCBpZiBkaXJlY3RvcmllcyBjYW4gYmUgY29tcGFjdGVkLCBpdCB3b3VsZCBiZSBuaWNlIHRvIGJl
IGFibGUgdG8gdGVsbCB3aGVuIHRoZSBjb29raWUgaGFzIGJlY29tZSBpbnZhbGlkLCByaWdodD8g
IFNvLCBJIGFkZGVkIGl0IHRvIHRoZSBORlN2MyBwcm90b2NvbC4NCg0KSXQgd2FzIG9ubHkgYWZ0
ZXJ3YXJkcywgaW4gYXR0ZW1wdGluZyB0byBpbXBsZW1lbnQgaXQsIHRoYXQgdGhlIHJlYWwgdHJ1
dGggd2FzIGRpc2NvdmVyZWQuICBUaGlzIGlzIHRoYXQgdGhlIGludGVyZmFjZXMgYXJlIG5vdCBz
dWZmaWNpZW50IHRvIGJlIGFibGUgdG8gY29udmV5IHRoaXMgaW5mb3JtYXRpb24gc3VmZmljaWVu
dGx5IGFuZCB0byBiZSBhYmxlIHRvIHJlY292ZXIgZnJvbSB0aGUgc2l0dWF0aW9uLiAgQ3VycmVu
dGx5LCByZWNvdmVyeSByZXF1aXJlIGFjdGlvbiBvbiB0aGUgcGFydCBvZiB0aGUgYXBwbGljYXRp
b24uDQoNCkluIHJldHJvc3BlY3QsIEkgd291bGRuJ3QgYWRkIHRoZSBjb29raWUgdmVyaWZpZXIg
aWYgSSB3YXMgZG9pbmcgaXQgYWdhaW4uDQoNClRoYW5rcyBmb3IgdGhlIHN1cHBvcnQsIHRob3Vn
aC4gIDotKQ0KDQpJIGRvIGRpc2FncmVlIHdpdGggVHJvbmQgdGhvdWdoLCBpbiB0aGF0IGFwcGxp
Y2F0aW9ucyBtdXN0IGJlIGNvZGVkIHRvIGJlIGFibGUgdG8gaGFuZGxlIGRpcmVjdG9yaWVzIHdo
ZXJlIGNvb2tpZXMgY2FuIGJlY29tZSBpbnZhbGlkYXRlZC4gIFRoYXQncyB3aHkgcm0gYW5kIHN1
Y2ggaGF2ZSB0byBiZSBjb2RlZCB0byBrZWVwIHRyYWNrIG9mIHRoZWlyIGxvY2F0aW9uIGluIHRo
ZSBkaXJlY3RvcnkgYnkgbW9yZSB0aGFuIGp1c3QgZF9vZmYuDQoNCgkJcHMNCg0KDQotLS0tLU9y
aWdpbmFsIE1lc3NhZ2UtLS0tLQ0KRnJvbTogbGludXgtbmZzLW93bmVyQHZnZXIua2VybmVsLm9y
ZyBbbWFpbHRvOmxpbnV4LW5mcy1vd25lckB2Z2VyLmtlcm5lbC5vcmddIE9uIEJlaGFsZiBPZiBN
YXR0IFcuIEJlbmphbWluDQpTZW50OiBNb25kYXksIE1hcmNoIDI2LCAyMDEyIDI6NTUgUE0NClRv
OiBCb2F6IEhhcnJvc2gNCkNjOiBsaW51eC1uZnM7IEdhbmVzaGEgTkZTIExpc3Q7IFRyb25kIE15
a2xlYnVzdA0KU3ViamVjdDogUmU6IHVubGluayB3aXRoaW4gYW4gb3BlbiBkaXJlY3Rvcnkgc3Ry
ZWFtDQoNCkhpLA0KDQpCb2F6OiAgd2UgZG8gbm90IGFueSBsb25nZXIgc2VuZCBhIHJlYWRkaXIg
aW5kZXguICBXZSBkbyBzZW5kIGEgY29va2lldmVyZi4NCg0KRmlzdCBvZiBhbGwsIEkgaGF2ZW4n
dCBlc3RhYmxpc2hlZCB0aGF0IHRoZSBpc3N1ZSB3ZSdyZSBhY3R1YWxseSBvYnNlcnZpbmcgaXMg
Y2F1c2VkIGJ5IHRoZSBMaW51eCBjbGllbnQgc2VuZGluZyBvbGQgY29va2llcyB0byByZWFkZGly
KHNvbWV0aGluZykuICBIb3dldmVyLCBpZiBpdCBpcywgaXQncyBpbiBubyB3YXkgYmV0dGVyIHRv
IHRyeSB0byBtYWtlIGNvb2tpZXMgIm1vcmUgcGVyc2lzdGVudC4iICBOb3Igc2hvdWxkIHRoZSBM
aW51eCBjbGllbnQgYmUgZXhwZWN0aW5nIGl0LiAgVGhlIGFzc3VtcHRpb24gaXMgc2ltcGx5IGZs
YXdlZC4gIFRoZSBwcm90b2NvbCBpbnRyb2R1Y2VkIGNvb2tpZSB2ZXJpZmllciAoYSBMT05HIHRp
bWUgYWdvKSBmb3IgYSByZWFzb24uDQoNCk1hdHQNCi0tLS0tICJCb2F6IEhhcnJvc2giIDxiaGFy
cm9zaEBwYW5hc2FzLmNvbT4gd3JvdGU6DQoNCj4gT24gMDMvMjYvMjAxMiAxMToyNSBBTSwgTXlr
bGVidXN0LCBUcm9uZCB3cm90ZToNCj4gDQo+ID4gT24gTW9uLCAyMDEyLTAzLTI2IGF0IDExOjE3
IC0wNzAwLCBCb2F6IEhhcnJvc2ggd3JvdGU6DQo+ID4+IE9uIDAzLzI0LzIwMTIgMTA6MTIgQU0s
IE15a2xlYnVzdCwgVHJvbmQgd3JvdGU6DQo+ID4+DQo+ID4+DQo+ID4+IEl0J3MgdGhlIG5ldyAo
cG9zdCBSSEVMIDYuMCBLZXJuZWwpIE5GUyBuZWVkIGZvciBvcGVuZGlyIGFmdGVyIGFuDQo+IHVu
bGluay4NCj4gPiANCj4gPiBXaGF0Pw0KPiA+IA0KPiANCj4gDQo+IFRoZSBsaW5rcyBiZWxvdyBy
ZXBvcnQgb24gcmVncmVzc2lvbiBpbiBSSEVMIDYuMiB3aGljaCBoYXZlIGEgbmV3IE5GUyANCj4g
aW1wbGVtZW50YXRpb24gd2hlcmUgIDYuMCB1c2VkIHRvIGJlIGZpbmUuDQo+IChPciBpdCdzIHdo
YXQgSSB1bmRlcnN0b29kIGZyb20gdGhlIHJlcG9ydGVycykNCj4gDQo+IDxzbmlwPg0KPiANCj4g
PiANCj4gDQo+ID4gTm8uDQo+ID4gDQo+ID4gSWYgdGhlIHNlcnZlciBzdXBwb3J0cyBwZXJtYW5l
bnQgcmVhZGRpciBjb29raWVzLCB0aGVuIHRoZXJlIGlzIG5vDQo+IG5lZWQNCj4gPiBmb3Igb3Bl
bmRpciBhZnRlciB1bmxpbmsuDQo+ID4gDQo+ID4gSWYgdGhlIHNlcnZlciBkb2VzIG5vdCBoYXZl
IHBlcm1hbmVudCByZWFkZGlyIGNvb2tpZXMsIHRoZW4gb3VyDQo+IGNsaWVudA0KPiA+IGhhcyBu
ZXZlciBzdXBwb3J0ZWQgaXQgYW55d2F5IChub3Igd2lsbCBpdCBldmVyIGRvIHNvKS4gVGhhdCB3
aG9sZSANCj4gPiAnY29va2lldmVyZicgUkVBRERJUiBidWxsc2hpdCBoYXMgbmV2ZXIgcHJvdmlk
ZWQgYSB3b3JrYWJsZSBtb2RlbA0KPiBmb3IgYQ0KPiA+IFBPU0lYIGNsaWVudC4uLg0KPiANCj4g
DQo+IFRoYW5rcyBUcm9uZCBmb3IgdGhlIGV4cGxhbmF0aW9uLiBGb3JnaXZlIG15IHNsb3duZXNz
LiBTbyB5b3UgYXJlIA0KPiBzYXlpbmcgdGhhdCB3aXRoIGtuZnNkIGl0J3MgYWN0dWFsbHkgZmls
ZXN5c3RlbSBkZXBlbmRlbnQgYW5kIG1heWJlIA0KPiB0aGUgcmVwb3J0ZXJzIGRpZCBub3QgY29t
cGFyZSBhcHBsZXMtdG8tYXBwbGVzIGFuZCB0aGUgZGlmZmVyZW5jZSBpcyANCj4gaW4gdGhlIGJl
aGluZCBmaWxlIHN5c3RlbS4NCj4gDQo+IE1hdHQgSSdtIHN1cmUgVHJvbmQgaXMgcmlnaHQgd2l0
aCByZWdhcmQgdG8gR2FuZXNoYSwgYmVjYXVzZSB3ZSBzZW5kIGEgDQo+IHJlYWRkaXIgaW5kZXgg
YW5kIGEgY29va2lldmVyZiwgd2hpY2ggd2lsbCBjaGFuZ2UgYWZ0ZXIgdW5saW5rLiBTaW5jZSAN
Cj4gdGhlIGFwcGxpY2F0aW9uIGRvZXMgbm90IGNhbGwgb3BlbmRpciBhZ2FpbiB0aGUgY2xpZW50
IHdpbGwgc2VuZCB0aGUgDQo+IG9sZCBjb29raWVzLCB3aGljaCBpcyBub3cgYSBkaWZmZXJlbnQg
ZmlsZSBvciBtaWdodCBnZXQgb3V0LW9mLWJvdW5kcyANCj4gZm9yIEdhbmVzaGEuDQo+IA0KPiBM
ZXQgbWUgdGhpbmsgYWJvdXQgaXQgYSBiaXQuIEknbSBzdXJlIHdlIGNhbiBzZW5kIGEgQmV0dGVy
IDY0Yml0IA0KPiBjb29raWUsIHdoaWNoIHdpbGwgYmUgbW9yZSBwZXJzaXN0ZW50LCBldmVuIGFj
cm9zcyB1bmxpbmtzLg0KPiANCj4gPiANCj4gPiANCj4gDQo+IA0KPiBUaGFua3MNCj4gQm9heg0K
DQotLQ0KTWF0dCBCZW5qYW1pbg0KVGhlIExpbnV4IEJveA0KMjA2IFNvdXRoIEZpZnRoIEF2ZS4g
U3VpdGUgMTUwDQpBbm4gQXJib3IsIE1JICA0ODEwNA0KDQpodHRwOi8vbGludXhib3guY29tDQoN
CnRlbC4gNzM0LTc2MS00Njg5DQpmYXguIDczNC03NjktODkzOA0KY2VsLiA3MzQtMjE2LTUzMDkN
Ci0tDQpUbyB1bnN1YnNjcmliZSBmcm9tIHRoaXMgbGlzdDogc2VuZCB0aGUgbGluZSAidW5zdWJz
Y3JpYmUgbGludXgtbmZzIiBpbiB0aGUgYm9keSBvZiBhIG1lc3NhZ2UgdG8gbWFqb3Jkb21vQHZn
ZXIua2VybmVsLm9yZyBNb3JlIG1ham9yZG9tbyBpbmZvIGF0ICBodHRwOi8vdmdlci5rZXJuZWwu
b3JnL21ham9yZG9tby1pbmZvLmh0bWwNCg==
On Mon, 26 Mar 2012 11:17:18 -0700
Boaz Harrosh <[email protected]> wrote:
> On 03/24/2012 10:12 AM, Myklebust, Trond wrote:
>
> > On Sat, 2012-03-24 at 12:53 -0400, Matt W. Benjamin wrote:
> >> Hi,
> >>
> >> I don't think anything is. Or, people originally reported the behavior against knfsd.
> >>
> >> Matt
> >
> > There is a known issue with ext2/3/4 generating non-unique readdir
> > cookies. It rarely hits you when you are creating small directories, but
> > it frequently hits you with larger ones. A fix is underway that should
> > significantly reduce the frequency of cookie collisions.
> >
> > Recent NFS clients will actually detect the presence of those cookie
> > loops, and log them in the kernel syslog. That would therefore be the
> > first thing that I'd check if confronted with this kind of problem.
> >
> > Cheers
> > Trond
> >
>
>
> Trond please look on the bug report links below. It's not the "cookie collisions" case.
>
> It's the new (post RHEL 6.0 Kernel) NFS need for opendir after an unlink.
> Now the POSIX man page *does* say that applications must re-opendir after
> unlink, but there are some applications who did not read the manual, and since
> it works with local filesystems and old nfs, (What Kernel RHEL 6.0 is based on?)
> they never noticed the bug and never fixed it.
>
^^^^^
Can you tell me which manpage says this? I'd like to be able to point
application developers at it if possible...
> Could we easily support the broken application by being bug compatible to
> old NFS versions?
> .i.e Don't require re-opendir after unlink of a file.
>
> There are more examples in the bug reports below but basically bonnie++
> does the following:
> DIR *d = opendir(".");
> dirent *file_ent;
> while((file_ent = readdir(d)) != NULL) {
> unlink( file_ent->d_name))
> }
> closedir(d);
>
> where it actually needs to do:
>
> DIR *d = opendir(".");
> dirent *file_ent;
> while((file_ent = readdir(d)) != NULL) {
> unlink( file_ent->d_name))
>
> closedir(d);
> d = opendir(".");
> }
> closedir(d);
>
> But again case one used to work with old NFS. And it looks like
> it is not Server dependent. We saw this both with Ganesha as well
> as knfsd
>
> <snip>
>
Again, my suspicion is that the change that triggered this is the
switch to use READDIRPLUS on larger directories. Before that, we'd use
READDIR on larger ones and wouldn't need to make as many RPCs to fetch
directory contents. More continuation READDIRPLUS calls means that you
have more opportunity to hit problems with cookies.
What might be an interesting test is to see whether this is still
reproducible on newer clients when you mount with '-o nordirplus'.
Cheers,
--
Jeff Layton <[email protected]>
On 2012-04-04 18:35, Jeff Layton wrote:
> On Mon, 26 Mar 2012 11:17:18 -0700
> Boaz Harrosh <[email protected]> wrote:
>
>> On 03/24/2012 10:12 AM, Myklebust, Trond wrote:
>>
>>> On Sat, 2012-03-24 at 12:53 -0400, Matt W. Benjamin wrote:
>>>> Hi,
>>>>
>>>> I don't think anything is. Or, people originally reported the behavior against knfsd.
>>>>
>>>> Matt
>>>
>>> There is a known issue with ext2/3/4 generating non-unique readdir
>>> cookies. It rarely hits you when you are creating small directories, but
>>> it frequently hits you with larger ones. A fix is underway that should
>>> significantly reduce the frequency of cookie collisions.
>>>
>>> Recent NFS clients will actually detect the presence of those cookie
>>> loops, and log them in the kernel syslog. That would therefore be the
>>> first thing that I'd check if confronted with this kind of problem.
>>>
>>> Cheers
>>> Trond
>>>
>>
>>
>> Trond please look on the bug report links below. It's not the "cookie collisions" case.
>>
>> It's the new (post RHEL 6.0 Kernel) NFS need for opendir after an unlink.
>> Now the POSIX man page *does* say that applications must re-opendir after
>> unlink, but there are some applications who did not read the manual, and since
>> it works with local filesystems and old nfs, (What Kernel RHEL 6.0 is based on?)
>> they never noticed the bug and never fixed it.
>>
>
> ^^^^^
> Can you tell me which manpage says this? I'd like to be able to point
> application developers at it if possible...
>
Hmm, http://pubs.opengroup.org/onlinepubs/9699919799/functions/readdir.html says:
files may be removed from a directory or added to a directory asynchronously to the operation of readdir().
...
If a file is removed from or added to the directory after the most recent call to opendir() or rewinddir(), whether a subsequent call to readdir() returns an entry for that file is unspecified.
...
If a file is removed from or added to the directory after the most recent call to opendir() or rewinddir(), whether a subsequent call to readdir_r() returns an entry for that file is unspecified.
Benny
>> Could we easily support the broken application by being bug compatible to
>> old NFS versions?
>> .i.e Don't require re-opendir after unlink of a file.
>>
>> There are more examples in the bug reports below but basically bonnie++
>> does the following:
>> DIR *d = opendir(".");
>> dirent *file_ent;
>> while((file_ent = readdir(d)) != NULL) {
>> unlink( file_ent->d_name))
>> }
>> closedir(d);
>>
>> where it actually needs to do:
>>
>> DIR *d = opendir(".");
>> dirent *file_ent;
>> while((file_ent = readdir(d)) != NULL) {
>> unlink( file_ent->d_name))
>>
>> closedir(d);
>> d = opendir(".");
>> }
>> closedir(d);
>>
>> But again case one used to work with old NFS. And it looks like
>> it is not Server dependent. We saw this both with Ganesha as well
>> as knfsd
>>
>> <snip>
>>
>
> Again, my suspicion is that the change that triggered this is the
> switch to use READDIRPLUS on larger directories. Before that, we'd use
> READDIR on larger ones and wouldn't need to make as many RPCs to fetch
> directory contents. More continuation READDIRPLUS calls means that you
> have more opportunity to hit problems with cookies.
>
> What might be an interesting test is to see whether this is still
> reproducible on newer clients when you mount with '-o nordirplus'.
>
> Cheers,