Hi Trond,
I appreciate your point of view. I appreciate Rick's comments. I hope some other comments will trickle in. Do I think you and Rick are saying the same thing? Not really. Do I think you're making sense when you imply I want to change the standard? No, I don't follow that. BIND_CONN_TO_SESSION is in the published standard. Can I show a dedicated back channel improves performance? Not at present. That's a sensible question. I think if the protocol in general is doing what it is intended to do, it should be possible for some workloads, at some point. (If callbacks are, somehow generally detrimental to performance, as you state, I think maybe we have some more work to do.) (I note only one more thing--I did raise this topic on list 12 months ago. Only Bruce commented, at the time.)
Thanks again,
Matt
--
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI 48104
http://linuxbox.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
> I have _always_ argued that callbacks should be the exception and not the rule,
Sounds right.
> due to the fact that in pretty much all cases the server will have to send
> out a NFS4ERR_DELAY result to whatever RPC call it is that triggered a callback.
I may disagree depending on what you mean by "petty much" always.
V4.1 does have callbacks that are specified so as to useful asynchronously, and
if people take advantage of them, they won't have this NFS4ERR_DELAY problem.
Examples are CB_NOTIFY, CB_RECALL_ANY, CB_PUSH_DELEG. Over time, future versions may
expand this list and make callbacks less exceptional, although not the rule.
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Myklebust, Trond
Sent: Thursday, October 06, 2011 11:39 PM
To: Matt W. Benjamin
Cc: linux-nfs; nfs-ganesha-devel; nfsv4
Subject: Re: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION
> -----Original Message-----
> From: Matt W. Benjamin [mailto:[email protected]]
> Sent: Thursday, October 06, 2011 10:55 PM
> To: Myklebust, Trond
> Cc: linux-nfs; nfs-ganesha-devel; nfsv4
> Subject: Re: [nfsv4] back channel flags, CREATE_SESSION,
> BIND_CONN_TO_SESSION
>
> Hi Trond,
>
> I appreciate your point of view. I appreciate Rick's comments. I hope some
> other comments will trickle in. Do I think you and Rick are saying the same
> thing? Not really. Do I think you're making sense when you imply I want to
> change the standard? No, I don't follow that. BIND_CONN_TO_SESSION is in
> the published standard. Can I show a dedicated back channel improves
I never accused you of trying to change the standard. What I said was that you need to convince me that there is a good reason to accept two implementations of the same feature in the client.
> performance? Not at present. That's a sensible question. I think if the
> protocol in general is doing what it is intended to do, it should be possible for
> some workloads, at some point. (If callbacks are, somehow generally
> detrimental to performance, as you state, I think maybe we have some more
> work to do.) (I note only one more thing--I did raise this topic on list 12
> months ago. Only Bruce commented, at the time.)
Fair enough, but 12 months ago, rfc5661 was in last call status, and we were all trying to move on to final implementations.
I have _always_ argued that callbacks should be the exception and not the rule, due to the fact that in pretty much all cases the server will have to send out a NFS4ERR_DELAY result to whatever RPC call it is that triggered a callback. Nothing new there in NFSv4.1. I'll add that nor has anybody successfully argued that there is any change in NFSv4.1 compared to our existing NFSv4 implementations. NFSv4 delegation callbacks are (measurably) extremely slow, and so I'd expect the exact same behavior for NFSv4.1 layout recalls etc. Nothing has changed in NFSv4 callback technology to warrant an expectation of performance improvements.
IOW: Nobody (including you) has ever convinced me that we need to consider callbacks as a scalability issue. I'd require some pretty firm evidence before I could accept that as a fact. So far, all I have seen is more or less eloquent prose. No hard numbers.
Trond
_______________________________________________
nfsv4 mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/nfsv4
I meant CB_RECALL_ANY. The server can use this to make sure that it is
Likely to have an extra recallable object available in hand. If he indicates
the desire early, than the interaction is asynchronous, and so the first
request requesting the layout/delegation get that one and you start another
asynchronous operation to have another given back.
In some cases, not having space does not man DELAY but rather not getting the
delegation you want, with that being provided to the client by CB_RECALL_OBJ_AVAIL.
This sort of interaction suggests to me that requests and callbacks may wind up
being more connected than we have thought before and that we might address this in
some v4.x for x >= 3.
-----Original Message-----
From: Trond Myklebust [mailto:[email protected]]
Sent: Tuesday, October 18, 2011 6:39 PM
To: Noveck, David
Cc: [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION
On Tue, 2011-10-18 at 17:28 -0400, [email protected] wrote:
> > I have _always_ argued that callbacks should be the exception and not the rule,
>
> Sounds right.
>
> > due to the fact that in pretty much all cases the server will have to send
> > out a NFS4ERR_DELAY result to whatever RPC call it is that triggered a callback.
>
> I may disagree depending on what you mean by "petty much" always.
>
> V4.1 does have callbacks that are specified so as to useful asynchronously, and
> if people take advantage of them, they won't have this NFS4ERR_DELAY problem.
>
> Examples are CB_NOTIFY, CB_RECALL_ANY, CB_PUSH_DELEG. Over time, future versions may
> expand this list and make callbacks less exceptional, although not the rule.
I'm not sure why you included CB_RECALL_ANY in the above list. Did you
mean CB_RECALLABLE_OBJ_AVAIL?
The main consumer of callbacks in NFSv4.1 is expected by some to be
layouts, which do not really have an asynchronous mode to notify you
when a particular layout has been made available.
Ditto for the NFSv4.0 number 1 consumer: delegations. The new
CB_PUSH_DELEG operation notifies you that you may now request a
delegation, but it doesn't help when you are just trying to OPEN a file,
and the server has to recall a delegation from someone else.
So I still think that NFS4ERR_DELAY will still be needed in order to
slow things down, unless we're careful about getting into these
situations where recalls are needed.
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On Behalf Of Myklebust, Trond
> Sent: Thursday, October 06, 2011 11:39 PM
> To: Matt W. Benjamin
> Cc: linux-nfs; nfs-ganesha-devel; nfsv4
> Subject: Re: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION
>
> > -----Original Message-----
> > From: Matt W. Benjamin [mailto:[email protected]]
> > Sent: Thursday, October 06, 2011 10:55 PM
> > To: Myklebust, Trond
> > Cc: linux-nfs; nfs-ganesha-devel; nfsv4
> > Subject: Re: [nfsv4] back channel flags, CREATE_SESSION,
> > BIND_CONN_TO_SESSION
> >
> > Hi Trond,
> >
> > I appreciate your point of view. I appreciate Rick's comments. I hope some
> > other comments will trickle in. Do I think you and Rick are saying the same
> > thing? Not really. Do I think you're making sense when you imply I want to
> > change the standard? No, I don't follow that. BIND_CONN_TO_SESSION is in
> > the published standard. Can I show a dedicated back channel improves
>
> I never accused you of trying to change the standard. What I said was that you need to convince me that there is a good reason to accept two implementations of the same feature in the client.
>
> > performance? Not at present. That's a sensible question. I think if the
> > protocol in general is doing what it is intended to do, it should be possible for
> > some workloads, at some point. (If callbacks are, somehow generally
> > detrimental to performance, as you state, I think maybe we have some more
> > work to do.) (I note only one more thing--I did raise this topic on list 12
> > months ago. Only Bruce commented, at the time.)
>
> Fair enough, but 12 months ago, rfc5661 was in last call status, and we were all trying to move on to final implementations.
>
> I have _always_ argued that callbacks should be the exception and not the rule, due to the fact that in pretty much all cases the server will have to send out a NFS4ERR_DELAY result to whatever RPC call it is that triggered a callback. Nothing new there in NFSv4.1. I'll add that nor has anybody successfully argued that there is any change in NFSv4.1 compared to our existing NFSv4 implementations. NFSv4 delegation callbacks are (measurably) extremely slow, and so I'd expect the exact same behavior for NFSv4.1 layout recalls etc. Nothing has changed in NFSv4 callback technology to warrant an expectation of performance improvements.
>
> IOW: Nobody (including you) has ever convinced me that we need to consider callbacks as a scalability issue. I'd require some pretty firm evidence before I could accept that as a fact. So far, all I have seen is more or less eloquent prose. No hard numbers.
>
> Trond
> _______________________________________________
> nfsv4 mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/nfsv4
>
> _______________________________________________
> nfsv4 mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/nfsv4
--
Trond Myklebust
Linux NFS client maintainer
NetApp
[email protected]
http://www.netapp.com
PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBNYXR0IFcuIEJlbmphbWluIFtt
YWlsdG86bWF0dEBsaW51eGJveC5jb21dDQo+IFNlbnQ6IFRodXJzZGF5LCBPY3RvYmVyIDA2LCAy
MDExIDEwOjU1IFBNDQo+IFRvOiBNeWtsZWJ1c3QsIFRyb25kDQo+IENjOiBsaW51eC1uZnM7IG5m
cy1nYW5lc2hhLWRldmVsOyBuZnN2NA0KPiBTdWJqZWN0OiBSZTogW25mc3Y0XSBiYWNrIGNoYW5u
ZWwgZmxhZ3MsIENSRUFURV9TRVNTSU9OLA0KPiBCSU5EX0NPTk5fVE9fU0VTU0lPTg0KPiANCj4g
SGkgVHJvbmQsDQo+IA0KPiBJIGFwcHJlY2lhdGUgeW91ciBwb2ludCBvZiB2aWV3LiAgSSBhcHBy
ZWNpYXRlIFJpY2sncyBjb21tZW50cy4gIEkgaG9wZSBzb21lDQo+IG90aGVyIGNvbW1lbnRzIHdp
bGwgdHJpY2tsZSBpbi4gIERvIEkgdGhpbmsgeW91IGFuZCBSaWNrIGFyZSBzYXlpbmcgdGhlIHNh
bWUNCj4gdGhpbmc/ICBOb3QgcmVhbGx5LiAgRG8gSSB0aGluayB5b3UncmUgbWFraW5nIHNlbnNl
IHdoZW4geW91IGltcGx5IEkgd2FudCB0bw0KPiBjaGFuZ2UgdGhlIHN0YW5kYXJkPyAgTm8sIEkg
ZG9uJ3QgZm9sbG93IHRoYXQuICBCSU5EX0NPTk5fVE9fU0VTU0lPTiBpcyBpbg0KPiB0aGUgcHVi
bGlzaGVkIHN0YW5kYXJkLiAgQ2FuIEkgc2hvdyBhIGRlZGljYXRlZCBiYWNrIGNoYW5uZWwgaW1w
cm92ZXMNCg0KSSBuZXZlciBhY2N1c2VkIHlvdSBvZiB0cnlpbmcgdG8gY2hhbmdlIHRoZSBzdGFu
ZGFyZC4gV2hhdCBJIHNhaWQgd2FzIHRoYXQgeW91IG5lZWQgdG8gY29udmluY2UgbWUgdGhhdCB0
aGVyZSBpcyBhIGdvb2QgcmVhc29uIHRvIGFjY2VwdCB0d28gaW1wbGVtZW50YXRpb25zIG9mIHRo
ZSBzYW1lIGZlYXR1cmUgaW4gdGhlIGNsaWVudC4NCg0KPiBwZXJmb3JtYW5jZT8gIE5vdCBhdCBw
cmVzZW50LiAgVGhhdCdzIGEgc2Vuc2libGUgcXVlc3Rpb24uICBJIHRoaW5rIGlmIHRoZQ0KPiBw
cm90b2NvbCBpbiBnZW5lcmFsIGlzIGRvaW5nIHdoYXQgaXQgaXMgaW50ZW5kZWQgdG8gZG8sIGl0
IHNob3VsZCBiZSBwb3NzaWJsZSBmb3INCj4gc29tZSB3b3JrbG9hZHMsIGF0IHNvbWUgcG9pbnQu
ICAoSWYgY2FsbGJhY2tzIGFyZSwgc29tZWhvdyBnZW5lcmFsbHkNCj4gZGV0cmltZW50YWwgdG8g
cGVyZm9ybWFuY2UsIGFzIHlvdSBzdGF0ZSwgSSB0aGluayBtYXliZSB3ZSBoYXZlIHNvbWUgbW9y
ZQ0KPiB3b3JrIHRvIGRvLikgIChJIG5vdGUgb25seSBvbmUgbW9yZSB0aGluZy0tSSBkaWQgcmFp
c2UgdGhpcyB0b3BpYyBvbiBsaXN0IDEyDQo+IG1vbnRocyBhZ28uICBPbmx5IEJydWNlIGNvbW1l
bnRlZCwgYXQgdGhlIHRpbWUuKQ0KDQpGYWlyIGVub3VnaCwgYnV0IDEyIG1vbnRocyBhZ28sIHJm
YzU2NjEgd2FzIGluIGxhc3QgY2FsbCBzdGF0dXMsIGFuZCB3ZSB3ZXJlIGFsbCB0cnlpbmcgdG8g
bW92ZSBvbiB0byBmaW5hbCBpbXBsZW1lbnRhdGlvbnMuDQoNCkkgaGF2ZSBfYWx3YXlzXyBhcmd1
ZWQgdGhhdCBjYWxsYmFja3Mgc2hvdWxkIGJlIHRoZSBleGNlcHRpb24gYW5kIG5vdCB0aGUgcnVs
ZSwgZHVlIHRvIHRoZSBmYWN0IHRoYXQgaW4gcHJldHR5IG11Y2ggYWxsIGNhc2VzIHRoZSBzZXJ2
ZXIgd2lsbCBoYXZlIHRvIHNlbmQgb3V0IGEgTkZTNEVSUl9ERUxBWSByZXN1bHQgdG8gd2hhdGV2
ZXIgUlBDIGNhbGwgaXQgaXMgdGhhdCB0cmlnZ2VyZWQgYSBjYWxsYmFjay4gTm90aGluZyBuZXcg
dGhlcmUgaW4gTkZTdjQuMS4gSSdsbCBhZGQgdGhhdCBub3IgaGFzIGFueWJvZHkgc3VjY2Vzc2Z1
bGx5IGFyZ3VlZCB0aGF0IHRoZXJlIGlzIGFueSBjaGFuZ2UgaW4gTkZTdjQuMSBjb21wYXJlZCB0
byBvdXIgZXhpc3RpbmcgTkZTdjQgaW1wbGVtZW50YXRpb25zLiBORlN2NCBkZWxlZ2F0aW9uIGNh
bGxiYWNrcyBhcmUgKG1lYXN1cmFibHkpIGV4dHJlbWVseSBzbG93LCBhbmQgc28gSSdkIGV4cGVj
dCB0aGUgZXhhY3Qgc2FtZSBiZWhhdmlvciBmb3IgTkZTdjQuMSBsYXlvdXQgcmVjYWxscyBldGMu
IE5vdGhpbmcgaGFzIGNoYW5nZWQgaW4gTkZTdjQgY2FsbGJhY2sgdGVjaG5vbG9neSB0byB3YXJy
YW50IGFuIGV4cGVjdGF0aW9uIG9mIHBlcmZvcm1hbmNlIGltcHJvdmVtZW50cy4NCg0KSU9XOiBO
b2JvZHkgKGluY2x1ZGluZyB5b3UpIGhhcyBldmVyIGNvbnZpbmNlZCBtZSB0aGF0IHdlIG5lZWQg
dG8gY29uc2lkZXIgY2FsbGJhY2tzIGFzIGEgc2NhbGFiaWxpdHkgaXNzdWUuIEknZCByZXF1aXJl
IHNvbWUgcHJldHR5IGZpcm0gZXZpZGVuY2UgYmVmb3JlIEkgY291bGQgYWNjZXB0IHRoYXQgYXMg
YSBmYWN0LiBTbyBmYXIsIGFsbCBJIGhhdmUgc2VlbiBpcyBtb3JlIG9yIGxlc3MgZWxvcXVlbnQg
cHJvc2UuIE5vIGhhcmQgbnVtYmVycy4NCg0KVHJvbmQNCg==
On Tue, 2011-10-18 at 17:28 -0400, [email protected] wrote:
> > I have _always_ argued that callbacks should be the exception and not the rule,
>
> Sounds right.
>
> > due to the fact that in pretty much all cases the server will have to send
> > out a NFS4ERR_DELAY result to whatever RPC call it is that triggered a callback.
>
> I may disagree depending on what you mean by "petty much" always.
>
> V4.1 does have callbacks that are specified so as to useful asynchronously, and
> if people take advantage of them, they won't have this NFS4ERR_DELAY problem.
>
> Examples are CB_NOTIFY, CB_RECALL_ANY, CB_PUSH_DELEG. Over time, future versions may
> expand this list and make callbacks less exceptional, although not the rule.
I'm not sure why you included CB_RECALL_ANY in the above list. Did you
mean CB_RECALLABLE_OBJ_AVAIL?
The main consumer of callbacks in NFSv4.1 is expected by some to be
layouts, which do not really have an asynchronous mode to notify you
when a particular layout has been made available.
Ditto for the NFSv4.0 number 1 consumer: delegations. The new
CB_PUSH_DELEG operation notifies you that you may now request a
delegation, but it doesn't help when you are just trying to OPEN a file,
and the server has to recall a delegation from someone else.
So I still think that NFS4ERR_DELAY will still be needed in order to
slow things down, unless we're careful about getting into these
situations where recalls are needed.
> -----Original Message-----
> From: [email protected] [mailto:[email protected]] On Behalf Of Myklebust, Trond
> Sent: Thursday, October 06, 2011 11:39 PM
> To: Matt W. Benjamin
> Cc: linux-nfs; nfs-ganesha-devel; nfsv4
> Subject: Re: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION
>
> > -----Original Message-----
> > From: Matt W. Benjamin [mailto:[email protected]]
> > Sent: Thursday, October 06, 2011 10:55 PM
> > To: Myklebust, Trond
> > Cc: linux-nfs; nfs-ganesha-devel; nfsv4
> > Subject: Re: [nfsv4] back channel flags, CREATE_SESSION,
> > BIND_CONN_TO_SESSION
> >
> > Hi Trond,
> >
> > I appreciate your point of view. I appreciate Rick's comments. I hope some
> > other comments will trickle in. Do I think you and Rick are saying the same
> > thing? Not really. Do I think you're making sense when you imply I want to
> > change the standard? No, I don't follow that. BIND_CONN_TO_SESSION is in
> > the published standard. Can I show a dedicated back channel improves
>
> I never accused you of trying to change the standard. What I said was that you need to convince me that there is a good reason to accept two implementations of the same feature in the client.
>
> > performance? Not at present. That's a sensible question. I think if the
> > protocol in general is doing what it is intended to do, it should be possible for
> > some workloads, at some point. (If callbacks are, somehow generally
> > detrimental to performance, as you state, I think maybe we have some more
> > work to do.) (I note only one more thing--I did raise this topic on list 12
> > months ago. Only Bruce commented, at the time.)
>
> Fair enough, but 12 months ago, rfc5661 was in last call status, and we were all trying to move on to final implementations.
>
> I have _always_ argued that callbacks should be the exception and not the rule, due to the fact that in pretty much all cases the server will have to send out a NFS4ERR_DELAY result to whatever RPC call it is that triggered a callback. Nothing new there in NFSv4.1. I'll add that nor has anybody successfully argued that there is any change in NFSv4.1 compared to our existing NFSv4 implementations. NFSv4 delegation callbacks are (measurably) extremely slow, and so I'd expect the exact same behavior for NFSv4.1 layout recalls etc. Nothing has changed in NFSv4 callback technology to warrant an expectation of performance improvements.
>
> IOW: Nobody (including you) has ever convinced me that we need to consider callbacks as a scalability issue. I'd require some pretty firm evidence before I could accept that as a fact. So far, all I have seen is more or less eloquent prose. No hard numbers.
>
> Trond
> _______________________________________________
> nfsv4 mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/nfsv4
>
> _______________________________________________
> nfsv4 mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/nfsv4
--
Trond Myklebust
Linux NFS client maintainer
NetApp
[email protected]
http://www.netapp.com