Hi Trond,
The rationale for making it possible to arrive at a dedicated back channel for a session is, I think, mainly that it can yield greater throughput, a benefit which grows as workload increases. Conversely, channel negotiation appears to not have many downsides.
My reasoning would be mainly that the dedicated configuration appears to minimise the throughput potentially lost to flow control or head-of-line blocking, in particular when the the underlying transport is TCP. A slightly different way to look at it is, that it enables a simpler RPC dispatcher at both endpoints. (I doubt the latter is more than a short-term consideration. Still, having an ability to work around a class of transport/interop problems transparently, which the fallback strategy could permit, might be viewed as a win.)
If one is persuaded of the potential utility of a dedicated session back channel, it still might not appear clearly better to have the client initially propose a mixed channel and assume the server will object if it cares (as opposed to making a dedicated fore channel its initial proposal). I think the fallback strategy is slightly better. It offers both client and server input into the transport setup, and it has the benefit of being closest to what 41 servers currently do and the Linux client currently does.
As you correctly state, the client could proceed without a back channel in the presence of a "disagreeing" server and still be conformant with RFC 5661 (if perhaps not in spirit), but actually, I'm not certain that the Linux client actually does proceed as specified in this case, as presently implemented (3.1.0-rc8)--I think it may incorrectly assume that a mixed channel has been agreed on.
Back channel negotiation does what it does, seemingly, at very little cost in development (a few hundred lines, mostly structure formulas). Maintenance costs are said to be proportional to code size (or so I have read, e.g., Page-Jones). I'm not making light of this point, obviously everything has a cost.
Matt
----- "Trond Myklebust" <[email protected]> wrote:
> On Wed, 2011-10-05 at 23:28 -0400, Trond Myklebust wrote:
> > On Wed, 2011-10-05 at 19:21 -0400, Matt W. Benjamin wrote:
> > > Currently, the Linux and I believe also the CITI Windows client
> always propose channels in both directions. The Linux mainline Linux
> client doesn't know how to BIND_CONN_TO_SESSION, so trivially it won't
> negotiate any back channel if a server didn't agree to both directions
> today, either. I've experimentally implemented a "fallback" model in
> a Linux client and (partly in a) Ganesha server. I'd appreciate any
> feedback on the idea.
> >
> > Yep. As I said, why should we bother adding support for servers
> that
> > don't? I can function perfectly well without pNFS support or
> delegation
> > support in such a case. Performance will suck, but why do I care?
>
> To put it in more basic terms: what you are proposing will add
> development costs to the client and and an extra code burden to
> maintain
> long term. So what is in it for me?
>
> --
> Trond Myklebust
> Linux NFS client maintainer
>
> NetApp
> [email protected]
> http://www.netapp.com
--
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI 48104
http://linuxbox.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
Hi Trond,
> For whom? We have already implemented a callback model that is working
> fine for us. I have yet to see any evidence of the callback
> scalability breakdown scenario that you claim as a motivation. What I
> have (frequently) seen is scalability issues due to clients and
> servers running out of free TCP ports.
For whom? Well, I work primarily on servers. I have worked experimentally on the Linux v41 client. I am pretty sure I wasn't speaking of a "callback scalability breakdown." I was discussing a tradeoff of ports for reduced overhead of flow control, lock contention, etc. I wasn't trying to start an inflated theory discussion about these, I don't feel fully at ease doing so. Still, I don't intuit that potential for port exhaustion trumps other considerations, it is one consideration among several. I also mentioned some potential benefit for server implementations, such as the one we have been collaborating on. I'd like to maximize its potential for success.
----- "Trond Myklebust" <[email protected]> wrote:
>
> My objection is to the lack of a consensus on a single system for
> implementing features. I'm tired of being told that the spec allows
> you to do the same thing in 10 different ways, with an expectation
> that we should implement all 10 ways.
> If we can't find a single good way of implementing a feature, then my
> preference is to drop support for that feature altogether (or to
> choose one implementation and to stick with it). My expectation for a
> standard is that it should aim to _reduce_ the number of different
> implementations so that we can concentrate our development and testing
> efforts. (BTW: pNFS is definitely not beyond criticism in this
> respect.)
You think the standard should influence, so as to reduce, the number of competing implementations? Maybe you meant 'implementation choices for <x> feature.'
My expectation from NFSv4 is that there is some room for innovation and experimentation. I don't think we can avoid this if we wish to compete effectively with ad hoc storage protocols.
>
> IOW: I can see valid reasons for why we should test the case where the
> server refuses a mixed fore+back channel, but I don't see that as a
> reason to implement a second back channel model. That requires us to
> add code + tests (and perform regular regression tests) for that back
> channel mode, as well as dealing with the case where that model of
> operation too is rejected by the server.
I find I need to keep reminding my self what "assume" does. But I assumed that the 5661 language for negotiating a back channel was in the draft (standard) because it added value to someone, and perhaps was implemented in some environment. If so, perhaps such a person will chime in with an opinion.
>
> Trond
Thanks,
Matt
--
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI 48104
http://linuxbox.com
tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBNYXR0IFcuIEJlbmphbWluIFtt
YWlsdG86bWF0dEBsaW51eGJveC5jb21dDQo+IFNlbnQ6IFRodXJzZGF5LCBPY3RvYmVyIDA2LCAy
MDExIDExOjExIEFNDQo+IFRvOiBNeWtsZWJ1c3QsIFRyb25kDQo+IENjOiBuZnN2NDsgbGludXgt
bmZzOyBuZnMtZ2FuZXNoYS1kZXZlbA0KPiBTdWJqZWN0OiBSZTogW25mc3Y0XSBiYWNrIGNoYW5u
ZWwgZmxhZ3MsIENSRUFURV9TRVNTSU9OLA0KPiBCSU5EX0NPTk5fVE9fU0VTU0lPTg0KPiANCj4g
SGkgVHJvbmQsDQo+IA0KPiBUaGUgcmF0aW9uYWxlIGZvciBtYWtpbmcgaXQgcG9zc2libGUgdG8g
YXJyaXZlIGF0IGEgZGVkaWNhdGVkIGJhY2sgY2hhbm5lbCBmb3IgYQ0KPiBzZXNzaW9uIGlzLCBJ
IHRoaW5rLCBtYWlubHkgdGhhdCBpdCBjYW4geWllbGQgZ3JlYXRlciB0aHJvdWdocHV0LCBhIGJl
bmVmaXQgd2hpY2gNCj4gZ3Jvd3MgYXMgd29ya2xvYWQgaW5jcmVhc2VzLiAgQ29udmVyc2VseSwg
Y2hhbm5lbCBuZWdvdGlhdGlvbiBhcHBlYXJzIHRvDQo+IG5vdCBoYXZlIG1hbnkgZG93bnNpZGVz
Lg0KDQpXaHkgZG8gd2UgbmVlZCBzdWNoIGEgaHVnZSB2b2x1bWUgb2YgY2FsbGJhY2tzPyBUaGV5
IHNob3VsZCBiZSB0aGUgZXhjZXB0aW9uIHJhdGhlciB0aGFuIHRoZSBydWxlLCBzaW5jZSB0aGVp
ciBlZmZlY3QgaXMgYWx3YXlzIHRvIHNsb3cgZG93biBwZXJmb3JtYW5jZS4gVGhyb3VnaHB1dCBp
cyBpbiBhZGRpdGlvbiBsaW1pdGVkIGJ5IHRoZSBudW1iZXIgb2Ygc2Vzc2lvbiBzbG90cyAoY3Vy
cmVudGx5IDEpLg0KDQo+IE15IHJlYXNvbmluZyB3b3VsZCBiZSBtYWlubHkgdGhhdCB0aGUgZGVk
aWNhdGVkIGNvbmZpZ3VyYXRpb24gYXBwZWFycyB0bw0KPiBtaW5pbWlzZSB0aGUgdGhyb3VnaHB1
dCBwb3RlbnRpYWxseSBsb3N0IHRvIGZsb3cgY29udHJvbCBvciBoZWFkLW9mLWxpbmUNCj4gYmxv
Y2tpbmcsIGluIHBhcnRpY3VsYXIgd2hlbiB0aGUgdGhlIHVuZGVybHlpbmcgdHJhbnNwb3J0IGlz
IFRDUC4gIEEgc2xpZ2h0bHkNCj4gZGlmZmVyZW50IHdheSB0byBsb29rIGF0IGl0IGlzLCB0aGF0
IGl0IGVuYWJsZXMgYSBzaW1wbGVyIFJQQyBkaXNwYXRjaGVyIGF0IGJvdGgNCj4gZW5kcG9pbnRz
LiAgKEkgZG91YnQgdGhlIGxhdHRlciBpcyBtb3JlIHRoYW4gYSBzaG9ydC10ZXJtIGNvbnNpZGVy
YXRpb24uICBTdGlsbCwNCj4gaGF2aW5nIGFuIGFiaWxpdHkgdG8gd29yayBhcm91bmQgYSBjbGFz
cyBvZiB0cmFuc3BvcnQvaW50ZXJvcCBwcm9ibGVtcw0KPiB0cmFuc3BhcmVudGx5LCB3aGljaCB0
aGUgZmFsbGJhY2sgc3RyYXRlZ3kgY291bGQgcGVybWl0LCBtaWdodCBiZSB2aWV3ZWQgYXMgYQ0K
PiB3aW4uKQ0KDQpGb3Igd2hvbT8gV2UgaGF2ZSBhbHJlYWR5IGltcGxlbWVudGVkIGEgY2FsbGJh
Y2sgbW9kZWwgdGhhdCBpcyB3b3JraW5nIGZpbmUgZm9yIHVzLiBJIGhhdmUgeWV0IHRvIHNlZSBh
bnkgZXZpZGVuY2Ugb2YgdGhlIGNhbGxiYWNrIHNjYWxhYmlsaXR5IGJyZWFrZG93biBzY2VuYXJp
byB0aGF0IHlvdSBjbGFpbSBhcyBhIG1vdGl2YXRpb24uIFdoYXQgSSBoYXZlIChmcmVxdWVudGx5
KSBzZWVuIGlzIHNjYWxhYmlsaXR5IGlzc3VlcyBkdWUgdG8gY2xpZW50cyBhbmQgc2VydmVycyBy
dW5uaW5nIG91dCBvZiBmcmVlIFRDUCBwb3J0cy4NCg0KPiBJZiBvbmUgaXMgcGVyc3VhZGVkIG9m
IHRoZSBwb3RlbnRpYWwgdXRpbGl0eSBvZiBhIGRlZGljYXRlZCBzZXNzaW9uIGJhY2sNCj4gY2hh
bm5lbCwgaXQgc3RpbGwgbWlnaHQgbm90IGFwcGVhciBjbGVhcmx5IGJldHRlciB0byBoYXZlIHRo
ZSBjbGllbnQgaW5pdGlhbGx5DQo+IHByb3Bvc2UgYSBtaXhlZCBjaGFubmVsIGFuZCBhc3N1bWUg
dGhlIHNlcnZlciB3aWxsIG9iamVjdCBpZiBpdCBjYXJlcyAoYXMNCj4gb3Bwb3NlZCB0byBtYWtp
bmcgYSBkZWRpY2F0ZWQgZm9yZSBjaGFubmVsIGl0cyBpbml0aWFsIHByb3Bvc2FsKS4gIEkgdGhp
bmsgdGhlDQo+IGZhbGxiYWNrIHN0cmF0ZWd5IGlzIHNsaWdodGx5IGJldHRlci4gIEl0IG9mZmVy
cyBib3RoIGNsaWVudCBhbmQgc2VydmVyIGlucHV0IGludG8NCj4gdGhlIHRyYW5zcG9ydCBzZXR1
cCwgYW5kIGl0IGhhcyB0aGUgYmVuZWZpdCBvZiBiZWluZyBjbG9zZXN0IHRvIHdoYXQgNDEgc2Vy
dmVycw0KPiBjdXJyZW50bHkgZG8gYW5kIHRoZSBMaW51eCBjbGllbnQgY3VycmVudGx5IGRvZXMu
DQo+DQo+IEFzIHlvdSBjb3JyZWN0bHkgc3RhdGUsIHRoZSBjbGllbnQgY291bGQgcHJvY2VlZCB3
aXRob3V0IGEgYmFjayBjaGFubmVsIGluIHRoZQ0KPiBwcmVzZW5jZSBvZiBhICJkaXNhZ3JlZWlu
ZyIgc2VydmVyIGFuZCBzdGlsbCBiZSBjb25mb3JtYW50IHdpdGggUkZDIDU2NjEgKGlmDQo+IHBl
cmhhcHMgbm90IGluIHNwaXJpdCksIGJ1dCBhY3R1YWxseSwgSSdtIG5vdCBjZXJ0YWluIHRoYXQg
dGhlIExpbnV4IGNsaWVudCBhY3R1YWxseQ0KPiBkb2VzIHByb2NlZWQgYXMgc3BlY2lmaWVkIGlu
IHRoaXMgY2FzZSwgYXMgcHJlc2VudGx5IGltcGxlbWVudGVkICgzLjEuMC1yYzgpLS0NCj4gSSB0
aGluayBpdCBtYXkgaW5jb3JyZWN0bHkgYXNzdW1lIHRoYXQgYSBtaXhlZCBjaGFubmVsIGhhcyBi
ZWVuIGFncmVlZCBvbi4NCg0KSSBhZ3JlZSB0aGF0IHdlIHNob3VsZCBmaXggdGhhdCBidWcuIEkg
ZGlzYWdyZWUgdGhhdCB3ZSBuZWVkIGEgc2Vjb25kIGNhbGxiYWNrIG1vZGVsIHRvIGRvIHNvLiBK
dXN0IHR1cm4gb2ZmIHBORlMgKGFuZCBwb3NzaWJseSBXQU5UcyBpZiB3ZSBldmVyIGltcGxlbWVu
dCB0aGVtKSwgYW5kIHdlIHNob3VsZCBiZSBmaW5lLg0KDQo+IEJhY2sgY2hhbm5lbCBuZWdvdGlh
dGlvbiBkb2VzIHdoYXQgaXQgZG9lcywgc2VlbWluZ2x5LCBhdCB2ZXJ5IGxpdHRsZSBjb3N0IGlu
DQo+IGRldmVsb3BtZW50IChhIGZldyBodW5kcmVkIGxpbmVzLCBtb3N0bHkgc3RydWN0dXJlIGZv
cm11bGFzKS4NCj4gTWFpbnRlbmFuY2UgY29zdHMgYXJlIHNhaWQgdG8gYmUgcHJvcG9ydGlvbmFs
IHRvIGNvZGUgc2l6ZSAob3Igc28gSSBoYXZlIHJlYWQsDQo+IGUuZy4sIFBhZ2UtSm9uZXMpLiAg
SSdtIG5vdCBtYWtpbmcgbGlnaHQgb2YgdGhpcyBwb2ludCwgb2J2aW91c2x5IGV2ZXJ5dGhpbmcg
aGFzDQo+IGEgY29zdC4NCg0KTXkgb2JqZWN0aW9uIGlzIHRvIHRoZSBsYWNrIG9mIGEgY29uc2Vu
c3VzIG9uIGEgc2luZ2xlIHN5c3RlbSBmb3IgaW1wbGVtZW50aW5nIGZlYXR1cmVzLiBJJ20gdGly
ZWQgb2YgYmVpbmcgdG9sZCB0aGF0IHRoZSBzcGVjIGFsbG93cyB5b3UgdG8gZG8gdGhlIHNhbWUg
dGhpbmcgaW4gMTAgZGlmZmVyZW50IHdheXMsIHdpdGggYW4gZXhwZWN0YXRpb24gdGhhdCB3ZSBz
aG91bGQgaW1wbGVtZW50IGFsbCAxMCB3YXlzLg0KSWYgd2UgY2FuJ3QgZmluZCBhIHNpbmdsZSBn
b29kIHdheSBvZiBpbXBsZW1lbnRpbmcgYSBmZWF0dXJlLCB0aGVuIG15IHByZWZlcmVuY2UgaXMg
dG8gZHJvcCBzdXBwb3J0IGZvciB0aGF0IGZlYXR1cmUgYWx0b2dldGhlciAob3IgdG8gY2hvb3Nl
IG9uZSBpbXBsZW1lbnRhdGlvbiBhbmQgdG8gc3RpY2sgd2l0aCBpdCkuIE15IGV4cGVjdGF0aW9u
IGZvciBhIHN0YW5kYXJkIGlzIHRoYXQgaXQgc2hvdWxkIGFpbSB0byBfcmVkdWNlXyB0aGUgbnVt
YmVyIG9mIGRpZmZlcmVudCBpbXBsZW1lbnRhdGlvbnMgc28gdGhhdCB3ZSBjYW4gY29uY2VudHJh
dGUgb3VyIGRldmVsb3BtZW50IGFuZCB0ZXN0aW5nIGVmZm9ydHMuIChCVFc6IHBORlMgaXMgZGVm
aW5pdGVseSBub3QgYmV5b25kIGNyaXRpY2lzbSBpbiB0aGlzIHJlc3BlY3QuKQ0KDQpJT1c6IEkg
Y2FuIHNlZSB2YWxpZCByZWFzb25zIGZvciB3aHkgd2Ugc2hvdWxkIHRlc3QgdGhlIGNhc2Ugd2hl
cmUgdGhlIHNlcnZlciByZWZ1c2VzIGEgbWl4ZWQgZm9yZStiYWNrIGNoYW5uZWwsIGJ1dCBJIGRv
bid0IHNlZSB0aGF0IGFzIGEgcmVhc29uIHRvIGltcGxlbWVudCBhIHNlY29uZCBiYWNrIGNoYW5u
ZWwgbW9kZWwuIFRoYXQgcmVxdWlyZXMgdXMgdG8gYWRkIGNvZGUgKyB0ZXN0cyAoYW5kIHBlcmZv
cm0gcmVndWxhciByZWdyZXNzaW9uIHRlc3RzKSBmb3IgdGhhdCBiYWNrIGNoYW5uZWwgbW9kZSwg
YXMgd2VsbCBhcyBkZWFsaW5nIHdpdGggdGhlIGNhc2Ugd2hlcmUgdGhhdCBtb2RlbCBvZiBvcGVy
YXRpb24gdG9vIGlzIHJlamVjdGVkIGJ5IHRoZSBzZXJ2ZXIuDQoNClRyb25kDQo=
On Thu, 2011-10-06 at 16:12 -0400, Matt W. Benjamin wrote:
> Hi Trond,
>
> > For whom? We have already implemented a callback model that is working
> > fine for us. I have yet to see any evidence of the callback
> > scalability breakdown scenario that you claim as a motivation. What I
> > have (frequently) seen is scalability issues due to clients and
> > servers running out of free TCP ports.
>
> For whom? Well, I work primarily on servers. I have worked experimentally on the Linux v41 client. I am pretty sure I wasn't speaking of a "callback scalability breakdown." I was discussing a tradeoff of ports for reduced overhead of flow control, lock contention, etc. I wasn't trying to start an inflated theory discussion about these, I don't feel fully at ease doing so. Still, I don't intuit that potential for port exhaustion trumps other considerations, it is one consideration among several. I also mentioned some potential benefit for server implementations, such as the one we have been collaborating on. I'd like to maximize its potential for success.
>
> ----- "Trond Myklebust" <[email protected]> wrote:
> >
> > My objection is to the lack of a consensus on a single system for
> > implementing features. I'm tired of being told that the spec allows
> > you to do the same thing in 10 different ways, with an expectation
> > that we should implement all 10 ways.
> > If we can't find a single good way of implementing a feature, then my
> > preference is to drop support for that feature altogether (or to
> > choose one implementation and to stick with it). My expectation for a
> > standard is that it should aim to _reduce_ the number of different
> > implementations so that we can concentrate our development and testing
> > efforts. (BTW: pNFS is definitely not beyond criticism in this
> > respect.)
>
> You think the standard should influence, so as to reduce, the number of competing implementations? Maybe you meant 'implementation choices for <x> feature.'
I meant to say exactly what I said above: that a standard should reduce
the number of _different_ ways to implement the exact same feature. Find
something that works, and then add it to the standard. Do not waste my
time by adding experimental features and then coming back 9 months after
the standard has been published and stating that 'I just discovered that
feature A is better implemented using method X rather than method Y'.
That's not a standard: it's an experiment in progress.
> My expectation from NFSv4 is that there is some room for innovation and experimentation. I don't think we can avoid this if we wish to compete effectively with ad hoc storage protocols.
I'm not trying to compete with ad-hoc storage protocols: they can always
find a niche market to peddle their ideas. Once somebody starts talking
about a standard, then the main concept is that there is room for
cooperation in order to achieve mutual benefit. I see no benefit here.
> > IOW: I can see valid reasons for why we should test the case where the
> > server refuses a mixed fore+back channel, but I don't see that as a
> > reason to implement a second back channel model. That requires us to
> > add code + tests (and perform regular regression tests) for that back
> > channel mode, as well as dealing with the case where that model of
> > operation too is rejected by the server.
>
> I find I need to keep reminding my self what "assume" does. But I assumed that the 5661 language for negotiating a back channel was in the draft (standard) because it added value to someone, and perhaps was implemented in some environment. If so, perhaps such a person will chime in with an opinion.
Then you need to communicate that value to me. I never signed an
agreement to implement all of RFC5661.
--
Trond Myklebust
Linux NFS client maintainer
NetApp
[email protected]
http://www.netapp.com