On 06/11/2012 07:01 PM, Andy Adamson wrote:
> I'm coding file layout data server recovery for the Linux NFS client,
> and came across an issue with LAYOUTRETURN that
> could use some comment from the list.
>
> The error case I'm handling is an RPC layer dis-connection error
> during heavy WRITE i/o to a file layout data server. Our response is
> to internally mark the deviceid as invalid which prevents all pNFS
> calls using the deviceid - e.g. no new I/O using any layout that uses
> the invalid deviceid, and to redirect all I/O to the MDS (any queued
> RPC request that has not been sent is redirected to the MDS).
>
> Plus - and here is where the clarification is needed - we immediately
> send a LAYOUTRETURN for any layout with in-flight requests to the
> dis-connected data server. By in-flight I mean transmitted WRT the
> RPC layer. The purpose of this LAYOUTRETURN is to notify the file
> layout MDS to fence the DS for the specified LAYOUTs, as the WRITEs
> will also be sent to the MDS.
>
I do not disagree with this completely. The point here is very fine
grained and should be specified explicitly. I would like to see text
as of something like.
There are 3 types of in-flght RPC/IO
1. Client has sent RPC header + all of associated data and is waiting
for DS WRITE/READ_DONE reply.
(For me this case can be, client may return LAYOUTRETURN as your
suggestion)
2. Client has sent the RPC header but has got stuck sending the rest
of the RPC message. Then received a network disconnect. This is the
most common part. Putting aside the RPC that got the error for a second.
The most important is what to do with parallel RPC/IO which are in this
state. Are parallel RPCs allowed to continue sending network packets
after the LAYOUTRETURN was sent?
The specific RPC that got stuck is not interesting because it's kind of
1.5, We are not going to send any bytes on that channel. The interesting
is these other DSs which are still streaming
3. Client has some internal RPC queue which do to some client parallelism
will start sending RPC header + data after the LAYOUTRETURN was sent
What my point was that with the code you submitted we are clearly violating
2. and even 3. Because I do not see anything avoiding this.
And if the STD allows you 2 and 3. Then that's a big change to the concept.
Not like you let it seem.
> I contend that sending the LAYOUTRETURN in this error case does not
> violate the two sections of RFC 5661 below, as the client has stopped
> sending any I/O requests using the returned layout.
>
I would not mind if this was true. That is if the LAYOUTRETURN was
a very clear barrier where our client would "magically" completely
plug the network interface and will not continue to send a single
byte on the wire to *any* DS involved with the layout. That's fine.
That is only allow sate 1 and 1.5 RPCs above. Some/all bytes where
presented on the wire, until the LAYOUTRETURN, from which point all
RPCs are hard aborted and not a single byte is sent.
> Others contend that since the in-flight RPCs reference the returned
> layout, the client is still 'using' the layout with these in-flight
> requests, and can not call LAYOUTRETURN until all in-flight RPCs
> return, with or without an error.
>
With our client code I don't see how the guaranty of 2 and 3 above
will happen without actually implementing this here.
So in principal I agree with your principle, I only do not agree
with your practice. In your new code you are violating 2 and 3
which are not to be allowed.
And again, please explain why do you want it. What is wrong with the
case we all agree with? ie: "Client can not call LAYOUTRETURN until
all in-flight RPCs return, with or without an error"
Thanks
Boaz
>
> Section 18.44.3 - the description section of the LAYOUTRETURN operation:
>
> After this call,
> the client MUST NOT use the returned layout(s) and the associated
> storage protocol to access the file data.
>
> Section 13.6 Operations Sent to NFSv4.1 Data Servers
>
> As described in Section 12.5.1, a client
> MUST NOT send an I/O to a data server for which it does not hold a
> valid layout; the data server MUST reject such an I/O.
>
>
> -->Andy
On 06/11/2012 09:59 PM, Myklebust, Trond wrote:
> On Mon, 2012-06-11 at 21:40 +0300, Boaz Harrosh wrote:
>
>> And again, please explain why do you want it. What is wrong with the
>> case we all agree with? ie: "Client can not call LAYOUTRETURN until
>> all in-flight RPCs return, with or without an error"
>
> Who "agreed" to this? This would mean that if the DS goes down, we can't
> ever send LAYOUTRETURN which is patently wrong.
>
"DS goes down" is under the above "RPC return an error" the error condition
of an RPC is well defined.
>From what of my words did you understand that I said
"we can't ever send a LAYOUTRETURN"
If my English is wrongly worded. Which is perfectly possible. Please correct
me so I can learn. Did you honestly think that's what I meant?
I meant we all agree, that this case is covered by RFC. That is - no one would
accuse a client who does that, as violating the RFC.
And again my question. The motivation?
Thanks
Boaz
This is why the objects layout communicates errors upon layout return.
When you suffer write timeouts (or any errors) with a DS, then the client promptly returns the layout with an indication of what went wrong. This should be the intent of whatever words are in the RFC.
--
Brent
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Myklebust, Trond
Sent: Monday, June 11, 2012 11:59 AM
To: Harrosh, Boaz
Cc: NFS list; Adamson, Andy; Andy Adamson; NFSv4
Subject: Re: [nfsv4] RFC 5661 LAYOUTRETURN clarification.
On Mon, 2012-06-11 at 21:40 +0300, Boaz Harrosh wrote:
> And again, please explain why do you want it. What is wrong with the
> case we all agree with? ie: "Client can not call LAYOUTRETURN until
> all in-flight RPCs return, with or without an error"
Who "agreed" to this? This would mean that if the DS goes down, we can't
ever send LAYOUTRETURN which is patently wrong.
--
Trond Myklebust
Linux NFS client maintainer
NetApp
[email protected]
http://www.netapp.com
_______________________________________________
nfsv4 mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/nfsv4
QlRXOiANCg0KSSBmb3Jnb3QgdG8gYWRkIHRoYXQgdGhlIGZlbmNpbmcgaXNzdWVzIGFyZSBhbHNv
IHRoZSByZWFzb24gd2h5IHRoZQ0KTGludXggY2xpZW50IGlzIHVubGlrZWx5IHRvIGNvbXBseSBh
bnkgdGltZSBzb29uIHdpdGggUkZDNTY2MSBTZWN0aW9uDQoxMy45LjEuJ3MgcmVxdWVzdCB0aGF0
IHdlIHByZWZlciB1c2Ugb2YgdGhlIE9QRU4gc3RhdGVpZCBvdmVyIHRoZSBMT0NLDQpzdGF0ZWlk
IHdoZW4gdGFsa2luZyB0byB0aGUgRFMuDQpJZiB0aGUgc2VydmVyIHJldm9rZXMgdGhlIGxvY2ss
IG9yIGlmIHRoZSBjbGllbnQgY2FsbHMgTE9DS1UsIGFsbCBXUklURXMNCnRoYXQgd2VyZSBtYWRl
IHVuZGVyIHRoYXQgbG9jayBuZWVkIHRvIGJlIGZlbmNlZCBvZmYuIFVubGVzcyBtYW5kYXRvcnkN
CmxvY2tpbmcgaXMgaW4gZWZmZWN0LCB0aGF0IHdvbid0IGhhcHBlbiBpZiB0aGUgV1JJVEUgb3Bz
IHdlcmUgc2VudCB1c2luZw0KdGhlIE9QRU4gc3RhdGVpZC4NCg0KVGhpcyBpcyBhbHNvIHdoeSBJ
IGJlbGlldmUgd2Ugc2hvdWxkIHJldmlzaXQgdGhlIHJ1bGUgdGhhdCB0aGUgY2xpZW50DQpzaG91
bGQgb25seSBzZW5kIHN0YXRlaWRzIHdpdGggYSB6ZXJvIHNlcWlkIHRvIHRoZSBEUy4NCg0KQ2hl
ZXJzDQogIFRyb25kDQoNCk9uIE1vbiwgMjAxMi0wNi0xMSBhdCAxNTo0MyAtMDQwMCwgVHJvbmQg
TXlrbGVidXN0IHdyb3RlOg0KPiBUaGUgX29ubHlfIHJlYXNvbiB3aHkgYSBwTkZTIGZpbGVzIGNs
aWVudCB3b3VsZCBldmVyIHdhbnQgdG8gc2VuZCBhDQo+IExBWU9VVFJFVFVSTiBpcyBpbiBvcmRl
ciB0byBoYXZlIHRoZSBNRFMgdGFrZSBhY3Rpb24gdG8gZmVuY2Ugb2ZmIGFueQ0KPiBvdXRzdGFu
ZGluZyB3cml0ZXMgdG8gdGhlIERTLg0KPiANCj4gVGhlIF9vbmx5XyBjYXNlIHdoZXJlIHRoYXQg
aXMgYWN0dWFsbHkgYW4gaW1wb3J0YW50IGlzc3VlIGlzIHdoZW4NCj4gc29tZXRoaW5nIGhhcHBl
bnMgdG8gdGhlIERTIHdoaWNoIGZvcmNlcyB0aGUgY2xpZW50IHRvIGZhbGwgYmFjayB0bw0KPiB3
cml0aW5nIHRocm91Z2ggdGhlIE1EUy4NCj4gDQo+IF9BTExfIG90aGVyIGNhc2VzIGFyZSB0cml2
aWFsbHkgY292ZXJlZCBieSB0aGUgZXhpc3RpbmcgTkZTdjQgc3RhdGUNCj4gbW9kZWwgaW4gdGhh
dCB3aGVuIHRoZSBjbGllbnQgdW5sb2NrcyBhbmQvb3IgY2xvc2VzIHRoZSBmaWxlLCB0aGVuIHRo
ZQ0KPiBsb2NrL29wZW4gc3RhdGVpZHMgdGhhdCBhcmUgdXNlZCBpbiB0aGUgUkVBRCBhbmQgV1JJ
VEUgb3BlcmF0aW9ucyB3aWxsDQo+IGJlIHVwZGF0ZWQsIGFuZCB3aWxsIGNhdXNlIHRob3NlIG9w
ZXJhdGlvbnMgdG8gYmUgcmVqZWN0ZWQgd2l0aCBhDQo+IEJBRF9TVEFURUlEIGVycm9yLiBUaGlz
IGZlbmNpbmcgbW9kZWwgaXMgaXJyZXNwZWN0aXZlIG9mIHdoZXRoZXIgb3Igbm90DQo+IGEgbGF5
b3V0IGlzIGhlbGQsIGFuZCBpcyBpcnJlc3BlY3RpdmUgb2Ygd2hldGhlciB0aGUgUkVBRC9XUklU
RSB3YXMgc2VudA0KPiB0byB0aGUgTURTIG9yIHRoZSBEUy4NCj4gDQo+IA0KPiBJT1c6IGlmIHBO
RlMgZmlsZXMgc2VydmVycyBkb24ndCB3YW50IHRvIGRvIHRoaXMga2luZCBvZiBmZW5jaW5nLCB0
aGVuIEkNCj4gc3VnZ2VzdCB3ZSBmaWxlIGFuIGVycmF0YSB0aGF0IGxhYmVscyB0aGUgTEFZT1VU
UkVUVVJOIG9wZXJhdGlvbiBhcw0KPiBtYW5kYXRvcnkgdG8gbm90IGltcGxlbWVudCBmb3IgdGhv
c2Ugc2VydmVycy4NCj4gDQo+IE9uIE1vbiwgMjAxMi0wNi0xMSBhdCAxNTowMiAtMDQwMCwgZGF2
aWQubm92ZWNrQGVtYy5jb20gd3JvdGU6DQo+ID4gPiBBbmQgYWdhaW4sIHBsZWFzZSBleHBsYWlu
IHdoeSBkbyB5b3Ugd2FudCBpdC4gV2hhdCBpcyB3cm9uZyB3aXRoIHRoZQ0KPiA+ID4gY2FzZSB3
ZSBhbGwgYWdyZWUgd2l0aD8gaWU6ICJDbGllbnQgY2FuIG5vdCBjYWxsIExBWU9VVFJFVFVSTiB1
bnRpbA0KPiA+ID4gYWxsIGluLWZsaWdodCBSUENzIHJldHVybiwgd2l0aCBvciB3aXRob3V0IGFu
IGVycm9yIg0KPiA+IA0KPiA+IEl0J3MgYSByZWNpcGUgZm9yIGRhdGEgY29ycnVwdGlvbi4gIElm
LCBhcyBBbmR5IGV4cGxhaW5lZCwgaGUgc3RhcnRzIGRvaW5nDQo+ID4gSU8ncyAobGV0J3Mgc3Vw
cG9zZSBXUklURXMpIHRvIHRoZSBNRFMgYW55IGxpbmdlcmluZyBXUklURXMgdG8gdGhlIERTDQo+
ID4gc2luY2UgdGhleSByZWZsZWN0IGFuIGVhcmxpZXIgc3RhdGUgb2YgYWZmYWlycyBjYW4gY2F1
c2UgZGF0YSBjb3JydXB0aW9uLg0KPiA+IA0KPiA+IFRoZXJlIGFyZSB0aHJlZSB3YXlzIHRvIHBy
ZXZlbnQgdGhvc2UgbGluZ2VyaW5nIERTIHdyaXRlcyBmcm9tIGNvcnJ1cHRpbmcgDQo+ID4gZGF0
YToNCj4gPiANCj4gPiAxKSBEb2luZyBhIExBWU9VVFJFVFVSTg0KPiA+IDIpIHdhaXRpbmcgdW50
aWwgdGhlIElPJ3MgcmV0dXJuLg0KPiA+IDMpICJtYWdpY2FsbHkgcGx1Z2dpbmcgdGhlIG5ldHdv
cmsgaW50ZXJmYWNlIi4NCj4gPiANCj4gPiANCj4gPiBTaW5jZSB0aGVyZSBpcyBubyB3YXkgdG8g
ZG8gMyksIHNheWluZyB0aGF0IHlvdSBvbmx5IGNhbiBkbyAxKSB1bnRpbCBhZnRlcg0KPiA+IDIp
IGlzIGRvbmUgaXMgZXNzZW50aWFsbHkgZ29pbmcgdG8gbWVhbjoNCj4gPiANCj4gPiBhKSB0aGF0
IGl0IG1heSB0YWtlIGEgdmVyeSBsb25nIHRpbWU6DQo+ID4gYikgdGhhdCB5b3Ugd2lsbCBvbmx5
IGRvIGl0LCB3aGVuIGl0IGlzIG5vIGxvbmdlciB1c2VmdWwuDQo+ID4gDQo+ID4gSWYgeW91IGRv
IDEpIGFzYXAsIHRoZW4gdGhlIGxpbmdlcmluZyBEUyB3cml0ZSBwcm9ibGVtIGlzIGdvbmUgc29v
bmVyLA0KPiA+IGFuZCB0aGF0J3MgYSBnb29kIHRoaW5nLiANCj4gPiANCj4gPiAtLS0tLU9yaWdp
bmFsIE1lc3NhZ2UtLS0tLQ0KPiA+IEZyb206IG5mc3Y0LWJvdW5jZXNAaWV0Zi5vcmcgW21haWx0
bzpuZnN2NC1ib3VuY2VzQGlldGYub3JnXSBPbiBCZWhhbGYgT2YgQm9heiBIYXJyb3NoDQo+ID4g
U2VudDogTW9uZGF5LCBKdW5lIDExLCAyMDEyIDI6NDEgUE0NCj4gPiBUbzogQW5keSBBZGFtc29u
DQo+ID4gQ2M6IEFuZHkgQWRhbXNvbjsgTkZTIGxpc3Q7IFRyb25kIE15a2xlYnVzdDsgTkZTdjQN
Cj4gPiBTdWJqZWN0OiBSZTogW25mc3Y0XSBSRkMgNTY2MSBMQVlPVVRSRVRVUk4gY2xhcmlmaWNh
dGlvbi4NCj4gPiANCj4gPiBPbiAwNi8xMS8yMDEyIDA3OjAxIFBNLCBBbmR5IEFkYW1zb24gd3Jv
dGU6DQo+ID4gDQo+ID4gPiBJJ20gY29kaW5nIGZpbGUgbGF5b3V0IGRhdGEgc2VydmVyIHJlY292
ZXJ5IGZvciB0aGUgTGludXggTkZTIGNsaWVudCwNCj4gPiA+IGFuZCBjYW1lIGFjcm9zcyBhbiBp
c3N1ZSB3aXRoIExBWU9VVFJFVFVSTiB0aGF0DQo+ID4gPiBjb3VsZCB1c2Ugc29tZSBjb21tZW50
IGZyb20gdGhlIGxpc3QuDQo+ID4gPiANCj4gPiA+IFRoZSBlcnJvciBjYXNlIEknbSBoYW5kbGlu
ZyBpcyBhbiBSUEMgbGF5ZXIgZGlzLWNvbm5lY3Rpb24gZXJyb3INCj4gPiA+IGR1cmluZyBoZWF2
eSBXUklURSBpL28gdG8gYSBmaWxlIGxheW91dCBkYXRhIHNlcnZlci4gT3VyIHJlc3BvbnNlIGlz
DQo+ID4gPiB0byBpbnRlcm5hbGx5IG1hcmsgdGhlIGRldmljZWlkIGFzIGludmFsaWQgd2hpY2gg
cHJldmVudHMgYWxsIHBORlMNCj4gPiA+IGNhbGxzIHVzaW5nIHRoZSBkZXZpY2VpZCAtIGUuZy4g
bm8gbmV3IEkvTyB1c2luZyBhbnkgbGF5b3V0IHRoYXQgdXNlcw0KPiA+ID4gdGhlIGludmFsaWQg
ZGV2aWNlaWQsIGFuZCB0byByZWRpcmVjdCBhbGwgSS9PIHRvIHRoZSBNRFMgKGFueSBxdWV1ZWQN
Cj4gPiA+IFJQQyByZXF1ZXN0IHRoYXQgaGFzIG5vdCBiZWVuIHNlbnQgaXMgcmVkaXJlY3RlZCB0
byB0aGUgTURTKS4NCj4gPiA+IA0KPiA+ID4gUGx1cyAtIGFuZCBoZXJlIGlzIHdoZXJlIHRoZSBj
bGFyaWZpY2F0aW9uIGlzIG5lZWRlZCAtIHdlIGltbWVkaWF0ZWx5DQo+ID4gPiBzZW5kIGEgTEFZ
T1VUUkVUVVJOIGZvciBhbnkgbGF5b3V0IHdpdGggaW4tZmxpZ2h0IHJlcXVlc3RzIHRvIHRoZQ0K
PiA+ID4gZGlzLWNvbm5lY3RlZCBkYXRhIHNlcnZlci4gIEJ5IGluLWZsaWdodCBJIG1lYW4gdHJh
bnNtaXR0ZWQgV1JUIHRoZQ0KPiA+ID4gUlBDIGxheWVyLiAgVGhlIHB1cnBvc2Ugb2YgdGhpcyBM
QVlPVVRSRVRVUk4gaXMgdG8gbm90aWZ5IHRoZSBmaWxlDQo+ID4gPiBsYXlvdXQgTURTIHRvIGZl
bmNlIHRoZSBEUyBmb3IgdGhlIHNwZWNpZmllZCBMQVlPVVRzLCBhcyB0aGUgV1JJVEVzDQo+ID4g
PiB3aWxsIGFsc28gYmUgc2VudCB0byB0aGUgTURTLg0KPiA+ID4gDQo+ID4gDQo+ID4gDQo+ID4g
SSBkbyBub3QgZGlzYWdyZWUgd2l0aCB0aGlzIGNvbXBsZXRlbHkuIFRoZSBwb2ludCBoZXJlIGlz
IHZlcnkgZmluZQ0KPiA+IGdyYWluZWQgYW5kIHNob3VsZCBiZSBzcGVjaWZpZWQgZXhwbGljaXRs
eS4gSSB3b3VsZCBsaWtlIHRvIHNlZSB0ZXh0DQo+ID4gYXMgb2Ygc29tZXRoaW5nIGxpa2UuDQo+
ID4gDQo+ID4gVGhlcmUgYXJlIDMgdHlwZXMgb2YgaW4tZmxnaHQgUlBDL0lPDQo+ID4gMS4gQ2xp
ZW50IGhhcyBzZW50IFJQQyBoZWFkZXIgKyBhbGwgb2YgYXNzb2NpYXRlZCBkYXRhIGFuZCBpcyB3
YWl0aW5nDQo+ID4gICAgZm9yIERTIFdSSVRFL1JFQURfRE9ORSByZXBseS4NCj4gPiANCj4gPiAg
ICAoRm9yIG1lIHRoaXMgY2FzZSBjYW4gYmUsIGNsaWVudCBtYXkgcmV0dXJuIExBWU9VVFJFVFVS
TiBhcyB5b3VyDQo+ID4gICAgIHN1Z2dlc3Rpb24pDQo+ID4gDQo+ID4gMi4gQ2xpZW50IGhhcyBz
ZW50IHRoZSBSUEMgaGVhZGVyIGJ1dCBoYXMgZ290IHN0dWNrIHNlbmRpbmcgdGhlIHJlc3QNCj4g
PiAgICBvZiB0aGUgUlBDIG1lc3NhZ2UuIFRoZW4gcmVjZWl2ZWQgYSBuZXR3b3JrIGRpc2Nvbm5l
Y3QuIFRoaXMgaXMgdGhlDQo+ID4gICAgbW9zdCBjb21tb24gcGFydC4gUHV0dGluZyBhc2lkZSB0
aGUgUlBDIHRoYXQgZ290IHRoZSBlcnJvciBmb3IgYSBzZWNvbmQuDQo+ID4gICAgVGhlIG1vc3Qg
aW1wb3J0YW50IGlzIHdoYXQgdG8gZG8gd2l0aCBwYXJhbGxlbCBSUEMvSU8gd2hpY2ggYXJlIGlu
IHRoaXMNCj4gPiAgICBzdGF0ZS4gQXJlIHBhcmFsbGVsIFJQQ3MgYWxsb3dlZCB0byBjb250aW51
ZSBzZW5kaW5nIG5ldHdvcmsgcGFja2V0cw0KPiA+ICAgIGFmdGVyIHRoZSBMQVlPVVRSRVRVUk4g
d2FzIHNlbnQ/DQo+ID4gDQo+ID4gICAgVGhlIHNwZWNpZmljIFJQQyB0aGF0IGdvdCBzdHVjayBp
cyBub3QgaW50ZXJlc3RpbmcgYmVjYXVzZSBpdCdzIGtpbmQgb2YNCj4gPiAgICAxLjUsIFdlIGFy
ZSBub3QgZ29pbmcgdG8gc2VuZCBhbnkgYnl0ZXMgb24gdGhhdCBjaGFubmVsLiBUaGUgaW50ZXJl
c3RpbmcNCj4gPiAgICBpcyB0aGVzZSBvdGhlciBEU3Mgd2hpY2ggYXJlIHN0aWxsIHN0cmVhbWlu
Zw0KPiA+IA0KPiA+IDMuIENsaWVudCBoYXMgc29tZSBpbnRlcm5hbCBSUEMgcXVldWUgd2hpY2gg
ZG8gdG8gc29tZSBjbGllbnQgcGFyYWxsZWxpc20NCj4gPiAgICB3aWxsIHN0YXJ0IHNlbmRpbmcg
UlBDIGhlYWRlciArIGRhdGEgYWZ0ZXIgdGhlIExBWU9VVFJFVFVSTiB3YXMgc2VudA0KPiA+ICAg
IA0KPiA+IFdoYXQgbXkgcG9pbnQgd2FzIHRoYXQgd2l0aCB0aGUgY29kZSB5b3Ugc3VibWl0dGVk
IHdlIGFyZSBjbGVhcmx5IHZpb2xhdGluZw0KPiA+IDIuIGFuZCBldmVuIDMuIEJlY2F1c2UgSSBk
byBub3Qgc2VlIGFueXRoaW5nIGF2b2lkaW5nIHRoaXMuDQo+ID4gDQo+ID4gQW5kIGlmIHRoZSBT
VEQgYWxsb3dzIHlvdSAyIGFuZCAzLiBUaGVuIHRoYXQncyBhIGJpZyBjaGFuZ2UgdG8gdGhlIGNv
bmNlcHQuDQo+ID4gTm90IGxpa2UgeW91IGxldCBpdCBzZWVtLg0KPiA+IA0KPiA+ID4gSSBjb250
ZW5kIHRoYXQgc2VuZGluZyB0aGUgTEFZT1VUUkVUVVJOIGluIHRoaXMgZXJyb3IgY2FzZSBkb2Vz
IG5vdA0KPiA+ID4gdmlvbGF0ZSB0aGUgdHdvIHNlY3Rpb25zIG9mIFJGQyA1NjYxIGJlbG93LCBh
cyB0aGUgY2xpZW50IGhhcyBzdG9wcGVkDQo+ID4gPiBzZW5kaW5nIGFueSBJL08gcmVxdWVzdHMg
dXNpbmcgdGhlIHJldHVybmVkIGxheW91dC4NCj4gPiA+IA0KPiA+IA0KPiA+IA0KPiA+IEkgd291
bGQgbm90IG1pbmQgaWYgdGhpcyB3YXMgdHJ1ZS4gVGhhdCBpcyBpZiB0aGUgTEFZT1VUUkVUVVJO
IHdhcw0KPiA+IGEgdmVyeSBjbGVhciBiYXJyaWVyIHdoZXJlIG91ciBjbGllbnQgd291bGQgIm1h
Z2ljYWxseSIgY29tcGxldGVseQ0KPiA+IHBsdWcgdGhlIG5ldHdvcmsgaW50ZXJmYWNlIGFuZCB3
aWxsIG5vdCBjb250aW51ZSB0byBzZW5kIGEgc2luZ2xlDQo+ID4gYnl0ZSBvbiB0aGUgd2lyZSB0
byAqYW55KiBEUyBpbnZvbHZlZCB3aXRoIHRoZSBsYXlvdXQuIFRoYXQncyBmaW5lLg0KPiA+IA0K
PiA+IFRoYXQgaXMgb25seSBhbGxvdyBzYXRlIDEgYW5kIDEuNSBSUENzIGFib3ZlLiBTb21lL2Fs
bCBieXRlcyB3aGVyZQ0KPiA+IHByZXNlbnRlZCBvbiB0aGUgd2lyZSwgdW50aWwgdGhlIExBWU9V
VFJFVFVSTiwgZnJvbSB3aGljaCBwb2ludCBhbGwNCj4gPiBSUENzIGFyZSBoYXJkIGFib3J0ZWQg
YW5kIG5vdCBhIHNpbmdsZSBieXRlIGlzIHNlbnQuDQo+ID4gDQo+ID4gDQo+ID4gPiBPdGhlcnMg
Y29udGVuZCB0aGF0IHNpbmNlIHRoZSBpbi1mbGlnaHQgUlBDcyByZWZlcmVuY2UgdGhlIHJldHVy
bmVkDQo+ID4gPiBsYXlvdXQsIHRoZSBjbGllbnQgaXMgc3RpbGwgJ3VzaW5nJyB0aGUgbGF5b3V0
IHdpdGggdGhlc2UgaW4tZmxpZ2h0DQo+ID4gPiByZXF1ZXN0cywgYW5kIGNhbiBub3QgY2FsbCBM
QVlPVVRSRVRVUk4gdW50aWwgYWxsIGluLWZsaWdodCBSUENzDQo+ID4gPiByZXR1cm4sIHdpdGgg
b3Igd2l0aG91dCBhbiBlcnJvci4NCj4gPiA+IA0KPiA+IA0KPiA+IA0KPiA+IFdpdGggb3VyIGNs
aWVudCBjb2RlIEkgZG9uJ3Qgc2VlIGhvdyB0aGUgZ3VhcmFudHkgb2YgMiBhbmQgMyBhYm92ZQ0K
PiA+IHdpbGwgaGFwcGVuIHdpdGhvdXQgYWN0dWFsbHkgaW1wbGVtZW50aW5nIHRoaXMgaGVyZS4N
Cj4gPiANCj4gPiBTbyBpbiBwcmluY2lwYWwgSSBhZ3JlZSB3aXRoIHlvdXIgcHJpbmNpcGxlLCBJ
IG9ubHkgZG8gbm90IGFncmVlDQo+ID4gd2l0aCB5b3VyIHByYWN0aWNlLiBJbiB5b3VyIG5ldyBj
b2RlIHlvdSBhcmUgdmlvbGF0aW5nIDIgYW5kIDMNCj4gPiB3aGljaCBhcmUgbm90IHRvIGJlIGFs
bG93ZWQuDQo+ID4gDQo+ID4gQW5kIGFnYWluLCBwbGVhc2UgZXhwbGFpbiB3aHkgZG8geW91IHdh
bnQgaXQuIFdoYXQgaXMgd3Jvbmcgd2l0aCB0aGUNCj4gPiBjYXNlIHdlIGFsbCBhZ3JlZSB3aXRo
PyBpZTogIkNsaWVudCBjYW4gbm90IGNhbGwgTEFZT1VUUkVUVVJOIHVudGlsDQo+ID4gYWxsIGlu
LWZsaWdodCBSUENzIHJldHVybiwgd2l0aCBvciB3aXRob3V0IGFuIGVycm9yIg0KPiA+IA0KPiA+
IFRoYW5rcw0KPiA+IEJvYXoNCj4gPiANCj4gPiA+IA0KPiA+ID4gU2VjdGlvbiAxOC40NC4zIC0g
dGhlIGRlc2NyaXB0aW9uIHNlY3Rpb24gb2YgdGhlIExBWU9VVFJFVFVSTiBvcGVyYXRpb246DQo+
ID4gPiANCj4gPiA+ICAgIEFmdGVyIHRoaXMgY2FsbCwNCj4gPiA+ICAgIHRoZSBjbGllbnQgTVVT
VCBOT1QgdXNlIHRoZSByZXR1cm5lZCBsYXlvdXQocykgYW5kIHRoZSBhc3NvY2lhdGVkDQo+ID4g
PiAgICBzdG9yYWdlIHByb3RvY29sIHRvIGFjY2VzcyB0aGUgZmlsZSBkYXRhLg0KPiA+ID4gDQo+
ID4gPiBTZWN0aW9uIDEzLjYgT3BlcmF0aW9ucyBTZW50IHRvIE5GU3Y0LjEgRGF0YSBTZXJ2ZXJz
DQo+ID4gPiANCj4gPiA+ICAgQXMgZGVzY3JpYmVkIGluIFNlY3Rpb24gMTIuNS4xLCBhIGNsaWVu
dA0KPiA+ID4gICBNVVNUIE5PVCBzZW5kIGFuIEkvTyB0byBhIGRhdGEgc2VydmVyIGZvciB3aGlj
aCBpdCBkb2VzIG5vdCBob2xkIGENCj4gPiA+ICAgdmFsaWQgbGF5b3V0OyB0aGUgZGF0YSBzZXJ2
ZXIgTVVTVCByZWplY3Qgc3VjaCBhbiBJL08uDQo+ID4gPiANCj4gPiA+IA0KPiA+ID4gLS0+QW5k
eQ0KPiA+IA0KPiA+IA0KPiA+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fDQo+ID4gbmZzdjQgbWFpbGluZyBsaXN0DQo+ID4gbmZzdjRAaWV0Zi5vcmcNCj4g
PiBodHRwczovL3d3dy5pZXRmLm9yZy9tYWlsbWFuL2xpc3RpbmZvL25mc3Y0DQo+ID4gDQo+IA0K
DQotLSANClRyb25kIE15a2xlYnVzdA0KTGludXggTkZTIGNsaWVudCBtYWludGFpbmVyDQoNCk5l
dEFwcA0KVHJvbmQuTXlrbGVidXN0QG5ldGFwcC5jb20NCnd3dy5uZXRhcHAuY29tDQoNCg==
DQpUaGUgX29ubHlfIHJlYXNvbiB3aHkgYSBwTkZTIGZpbGVzIGNsaWVudCB3b3VsZCBldmVyIHdh
bnQgdG8gc2VuZCBhDQpMQVlPVVRSRVRVUk4gaXMgaW4gb3JkZXIgdG8gaGF2ZSB0aGUgTURTIHRh
a2UgYWN0aW9uIHRvIGZlbmNlIG9mZiBhbnkNCm91dHN0YW5kaW5nIHdyaXRlcyB0byB0aGUgRFMu
DQoNClRoZSBfb25seV8gY2FzZSB3aGVyZSB0aGF0IGlzIGFjdHVhbGx5IGFuIGltcG9ydGFudCBp
c3N1ZSBpcyB3aGVuDQpzb21ldGhpbmcgaGFwcGVucyB0byB0aGUgRFMgd2hpY2ggZm9yY2VzIHRo
ZSBjbGllbnQgdG8gZmFsbCBiYWNrIHRvDQp3cml0aW5nIHRocm91Z2ggdGhlIE1EUy4NCg0KX0FM
TF8gb3RoZXIgY2FzZXMgYXJlIHRyaXZpYWxseSBjb3ZlcmVkIGJ5IHRoZSBleGlzdGluZyBORlN2
NCBzdGF0ZQ0KbW9kZWwgaW4gdGhhdCB3aGVuIHRoZSBjbGllbnQgdW5sb2NrcyBhbmQvb3IgY2xv
c2VzIHRoZSBmaWxlLCB0aGVuIHRoZQ0KbG9jay9vcGVuIHN0YXRlaWRzIHRoYXQgYXJlIHVzZWQg
aW4gdGhlIFJFQUQgYW5kIFdSSVRFIG9wZXJhdGlvbnMgd2lsbA0KYmUgdXBkYXRlZCwgYW5kIHdp
bGwgY2F1c2UgdGhvc2Ugb3BlcmF0aW9ucyB0byBiZSByZWplY3RlZCB3aXRoIGENCkJBRF9TVEFU
RUlEIGVycm9yLiBUaGlzIGZlbmNpbmcgbW9kZWwgaXMgaXJyZXNwZWN0aXZlIG9mIHdoZXRoZXIg
b3Igbm90DQphIGxheW91dCBpcyBoZWxkLCBhbmQgaXMgaXJyZXNwZWN0aXZlIG9mIHdoZXRoZXIg
dGhlIFJFQUQvV1JJVEUgd2FzIHNlbnQNCnRvIHRoZSBNRFMgb3IgdGhlIERTLg0KDQoNCklPVzog
aWYgcE5GUyBmaWxlcyBzZXJ2ZXJzIGRvbid0IHdhbnQgdG8gZG8gdGhpcyBraW5kIG9mIGZlbmNp
bmcsIHRoZW4gSQ0Kc3VnZ2VzdCB3ZSBmaWxlIGFuIGVycmF0YSB0aGF0IGxhYmVscyB0aGUgTEFZ
T1VUUkVUVVJOIG9wZXJhdGlvbiBhcw0KbWFuZGF0b3J5IHRvIG5vdCBpbXBsZW1lbnQgZm9yIHRo
b3NlIHNlcnZlcnMuDQoNCk9uIE1vbiwgMjAxMi0wNi0xMSBhdCAxNTowMiAtMDQwMCwgZGF2aWQu
bm92ZWNrQGVtYy5jb20gd3JvdGU6DQo+ID4gQW5kIGFnYWluLCBwbGVhc2UgZXhwbGFpbiB3aHkg
ZG8geW91IHdhbnQgaXQuIFdoYXQgaXMgd3Jvbmcgd2l0aCB0aGUNCj4gPiBjYXNlIHdlIGFsbCBh
Z3JlZSB3aXRoPyBpZTogIkNsaWVudCBjYW4gbm90IGNhbGwgTEFZT1VUUkVUVVJOIHVudGlsDQo+
ID4gYWxsIGluLWZsaWdodCBSUENzIHJldHVybiwgd2l0aCBvciB3aXRob3V0IGFuIGVycm9yIg0K
PiANCj4gSXQncyBhIHJlY2lwZSBmb3IgZGF0YSBjb3JydXB0aW9uLiAgSWYsIGFzIEFuZHkgZXhw
bGFpbmVkLCBoZSBzdGFydHMgZG9pbmcNCj4gSU8ncyAobGV0J3Mgc3VwcG9zZSBXUklURXMpIHRv
IHRoZSBNRFMgYW55IGxpbmdlcmluZyBXUklURXMgdG8gdGhlIERTDQo+IHNpbmNlIHRoZXkgcmVm
bGVjdCBhbiBlYXJsaWVyIHN0YXRlIG9mIGFmZmFpcnMgY2FuIGNhdXNlIGRhdGEgY29ycnVwdGlv
bi4NCj4gDQo+IFRoZXJlIGFyZSB0aHJlZSB3YXlzIHRvIHByZXZlbnQgdGhvc2UgbGluZ2VyaW5n
IERTIHdyaXRlcyBmcm9tIGNvcnJ1cHRpbmcgDQo+IGRhdGE6DQo+IA0KPiAxKSBEb2luZyBhIExB
WU9VVFJFVFVSTg0KPiAyKSB3YWl0aW5nIHVudGlsIHRoZSBJTydzIHJldHVybi4NCj4gMykgIm1h
Z2ljYWxseSBwbHVnZ2luZyB0aGUgbmV0d29yayBpbnRlcmZhY2UiLg0KPiANCj4gDQo+IFNpbmNl
IHRoZXJlIGlzIG5vIHdheSB0byBkbyAzKSwgc2F5aW5nIHRoYXQgeW91IG9ubHkgY2FuIGRvIDEp
IHVudGlsIGFmdGVyDQo+IDIpIGlzIGRvbmUgaXMgZXNzZW50aWFsbHkgZ29pbmcgdG8gbWVhbjoN
Cj4gDQo+IGEpIHRoYXQgaXQgbWF5IHRha2UgYSB2ZXJ5IGxvbmcgdGltZToNCj4gYikgdGhhdCB5
b3Ugd2lsbCBvbmx5IGRvIGl0LCB3aGVuIGl0IGlzIG5vIGxvbmdlciB1c2VmdWwuDQo+IA0KPiBJ
ZiB5b3UgZG8gMSkgYXNhcCwgdGhlbiB0aGUgbGluZ2VyaW5nIERTIHdyaXRlIHByb2JsZW0gaXMg
Z29uZSBzb29uZXIsDQo+IGFuZCB0aGF0J3MgYSBnb29kIHRoaW5nLiANCj4gDQo+IC0tLS0tT3Jp
Z2luYWwgTWVzc2FnZS0tLS0tDQo+IEZyb206IG5mc3Y0LWJvdW5jZXNAaWV0Zi5vcmcgW21haWx0
bzpuZnN2NC1ib3VuY2VzQGlldGYub3JnXSBPbiBCZWhhbGYgT2YgQm9heiBIYXJyb3NoDQo+IFNl
bnQ6IE1vbmRheSwgSnVuZSAxMSwgMjAxMiAyOjQxIFBNDQo+IFRvOiBBbmR5IEFkYW1zb24NCj4g
Q2M6IEFuZHkgQWRhbXNvbjsgTkZTIGxpc3Q7IFRyb25kIE15a2xlYnVzdDsgTkZTdjQNCj4gU3Vi
amVjdDogUmU6IFtuZnN2NF0gUkZDIDU2NjEgTEFZT1VUUkVUVVJOIGNsYXJpZmljYXRpb24uDQo+
IA0KPiBPbiAwNi8xMS8yMDEyIDA3OjAxIFBNLCBBbmR5IEFkYW1zb24gd3JvdGU6DQo+IA0KPiA+
IEknbSBjb2RpbmcgZmlsZSBsYXlvdXQgZGF0YSBzZXJ2ZXIgcmVjb3ZlcnkgZm9yIHRoZSBMaW51
eCBORlMgY2xpZW50LA0KPiA+IGFuZCBjYW1lIGFjcm9zcyBhbiBpc3N1ZSB3aXRoIExBWU9VVFJF
VFVSTiB0aGF0DQo+ID4gY291bGQgdXNlIHNvbWUgY29tbWVudCBmcm9tIHRoZSBsaXN0Lg0KPiA+
IA0KPiA+IFRoZSBlcnJvciBjYXNlIEknbSBoYW5kbGluZyBpcyBhbiBSUEMgbGF5ZXIgZGlzLWNv
bm5lY3Rpb24gZXJyb3INCj4gPiBkdXJpbmcgaGVhdnkgV1JJVEUgaS9vIHRvIGEgZmlsZSBsYXlv
dXQgZGF0YSBzZXJ2ZXIuIE91ciByZXNwb25zZSBpcw0KPiA+IHRvIGludGVybmFsbHkgbWFyayB0
aGUgZGV2aWNlaWQgYXMgaW52YWxpZCB3aGljaCBwcmV2ZW50cyBhbGwgcE5GUw0KPiA+IGNhbGxz
IHVzaW5nIHRoZSBkZXZpY2VpZCAtIGUuZy4gbm8gbmV3IEkvTyB1c2luZyBhbnkgbGF5b3V0IHRo
YXQgdXNlcw0KPiA+IHRoZSBpbnZhbGlkIGRldmljZWlkLCBhbmQgdG8gcmVkaXJlY3QgYWxsIEkv
TyB0byB0aGUgTURTIChhbnkgcXVldWVkDQo+ID4gUlBDIHJlcXVlc3QgdGhhdCBoYXMgbm90IGJl
ZW4gc2VudCBpcyByZWRpcmVjdGVkIHRvIHRoZSBNRFMpLg0KPiA+IA0KPiA+IFBsdXMgLSBhbmQg
aGVyZSBpcyB3aGVyZSB0aGUgY2xhcmlmaWNhdGlvbiBpcyBuZWVkZWQgLSB3ZSBpbW1lZGlhdGVs
eQ0KPiA+IHNlbmQgYSBMQVlPVVRSRVRVUk4gZm9yIGFueSBsYXlvdXQgd2l0aCBpbi1mbGlnaHQg
cmVxdWVzdHMgdG8gdGhlDQo+ID4gZGlzLWNvbm5lY3RlZCBkYXRhIHNlcnZlci4gIEJ5IGluLWZs
aWdodCBJIG1lYW4gdHJhbnNtaXR0ZWQgV1JUIHRoZQ0KPiA+IFJQQyBsYXllci4gIFRoZSBwdXJw
b3NlIG9mIHRoaXMgTEFZT1VUUkVUVVJOIGlzIHRvIG5vdGlmeSB0aGUgZmlsZQ0KPiA+IGxheW91
dCBNRFMgdG8gZmVuY2UgdGhlIERTIGZvciB0aGUgc3BlY2lmaWVkIExBWU9VVHMsIGFzIHRoZSBX
UklURXMNCj4gPiB3aWxsIGFsc28gYmUgc2VudCB0byB0aGUgTURTLg0KPiA+IA0KPiANCj4gDQo+
IEkgZG8gbm90IGRpc2FncmVlIHdpdGggdGhpcyBjb21wbGV0ZWx5LiBUaGUgcG9pbnQgaGVyZSBp
cyB2ZXJ5IGZpbmUNCj4gZ3JhaW5lZCBhbmQgc2hvdWxkIGJlIHNwZWNpZmllZCBleHBsaWNpdGx5
LiBJIHdvdWxkIGxpa2UgdG8gc2VlIHRleHQNCj4gYXMgb2Ygc29tZXRoaW5nIGxpa2UuDQo+IA0K
PiBUaGVyZSBhcmUgMyB0eXBlcyBvZiBpbi1mbGdodCBSUEMvSU8NCj4gMS4gQ2xpZW50IGhhcyBz
ZW50IFJQQyBoZWFkZXIgKyBhbGwgb2YgYXNzb2NpYXRlZCBkYXRhIGFuZCBpcyB3YWl0aW5nDQo+
ICAgIGZvciBEUyBXUklURS9SRUFEX0RPTkUgcmVwbHkuDQo+IA0KPiAgICAoRm9yIG1lIHRoaXMg
Y2FzZSBjYW4gYmUsIGNsaWVudCBtYXkgcmV0dXJuIExBWU9VVFJFVFVSTiBhcyB5b3VyDQo+ICAg
ICBzdWdnZXN0aW9uKQ0KPiANCj4gMi4gQ2xpZW50IGhhcyBzZW50IHRoZSBSUEMgaGVhZGVyIGJ1
dCBoYXMgZ290IHN0dWNrIHNlbmRpbmcgdGhlIHJlc3QNCj4gICAgb2YgdGhlIFJQQyBtZXNzYWdl
LiBUaGVuIHJlY2VpdmVkIGEgbmV0d29yayBkaXNjb25uZWN0LiBUaGlzIGlzIHRoZQ0KPiAgICBt
b3N0IGNvbW1vbiBwYXJ0LiBQdXR0aW5nIGFzaWRlIHRoZSBSUEMgdGhhdCBnb3QgdGhlIGVycm9y
IGZvciBhIHNlY29uZC4NCj4gICAgVGhlIG1vc3QgaW1wb3J0YW50IGlzIHdoYXQgdG8gZG8gd2l0
aCBwYXJhbGxlbCBSUEMvSU8gd2hpY2ggYXJlIGluIHRoaXMNCj4gICAgc3RhdGUuIEFyZSBwYXJh
bGxlbCBSUENzIGFsbG93ZWQgdG8gY29udGludWUgc2VuZGluZyBuZXR3b3JrIHBhY2tldHMNCj4g
ICAgYWZ0ZXIgdGhlIExBWU9VVFJFVFVSTiB3YXMgc2VudD8NCj4gDQo+ICAgIFRoZSBzcGVjaWZp
YyBSUEMgdGhhdCBnb3Qgc3R1Y2sgaXMgbm90IGludGVyZXN0aW5nIGJlY2F1c2UgaXQncyBraW5k
IG9mDQo+ICAgIDEuNSwgV2UgYXJlIG5vdCBnb2luZyB0byBzZW5kIGFueSBieXRlcyBvbiB0aGF0
IGNoYW5uZWwuIFRoZSBpbnRlcmVzdGluZw0KPiAgICBpcyB0aGVzZSBvdGhlciBEU3Mgd2hpY2gg
YXJlIHN0aWxsIHN0cmVhbWluZw0KPiANCj4gMy4gQ2xpZW50IGhhcyBzb21lIGludGVybmFsIFJQ
QyBxdWV1ZSB3aGljaCBkbyB0byBzb21lIGNsaWVudCBwYXJhbGxlbGlzbQ0KPiAgICB3aWxsIHN0
YXJ0IHNlbmRpbmcgUlBDIGhlYWRlciArIGRhdGEgYWZ0ZXIgdGhlIExBWU9VVFJFVFVSTiB3YXMg
c2VudA0KPiAgICANCj4gV2hhdCBteSBwb2ludCB3YXMgdGhhdCB3aXRoIHRoZSBjb2RlIHlvdSBz
dWJtaXR0ZWQgd2UgYXJlIGNsZWFybHkgdmlvbGF0aW5nDQo+IDIuIGFuZCBldmVuIDMuIEJlY2F1
c2UgSSBkbyBub3Qgc2VlIGFueXRoaW5nIGF2b2lkaW5nIHRoaXMuDQo+IA0KPiBBbmQgaWYgdGhl
IFNURCBhbGxvd3MgeW91IDIgYW5kIDMuIFRoZW4gdGhhdCdzIGEgYmlnIGNoYW5nZSB0byB0aGUg
Y29uY2VwdC4NCj4gTm90IGxpa2UgeW91IGxldCBpdCBzZWVtLg0KPiANCj4gPiBJIGNvbnRlbmQg
dGhhdCBzZW5kaW5nIHRoZSBMQVlPVVRSRVRVUk4gaW4gdGhpcyBlcnJvciBjYXNlIGRvZXMgbm90
DQo+ID4gdmlvbGF0ZSB0aGUgdHdvIHNlY3Rpb25zIG9mIFJGQyA1NjYxIGJlbG93LCBhcyB0aGUg
Y2xpZW50IGhhcyBzdG9wcGVkDQo+ID4gc2VuZGluZyBhbnkgSS9PIHJlcXVlc3RzIHVzaW5nIHRo
ZSByZXR1cm5lZCBsYXlvdXQuDQo+ID4gDQo+IA0KPiANCj4gSSB3b3VsZCBub3QgbWluZCBpZiB0
aGlzIHdhcyB0cnVlLiBUaGF0IGlzIGlmIHRoZSBMQVlPVVRSRVRVUk4gd2FzDQo+IGEgdmVyeSBj
bGVhciBiYXJyaWVyIHdoZXJlIG91ciBjbGllbnQgd291bGQgIm1hZ2ljYWxseSIgY29tcGxldGVs
eQ0KPiBwbHVnIHRoZSBuZXR3b3JrIGludGVyZmFjZSBhbmQgd2lsbCBub3QgY29udGludWUgdG8g
c2VuZCBhIHNpbmdsZQ0KPiBieXRlIG9uIHRoZSB3aXJlIHRvICphbnkqIERTIGludm9sdmVkIHdp
dGggdGhlIGxheW91dC4gVGhhdCdzIGZpbmUuDQo+IA0KPiBUaGF0IGlzIG9ubHkgYWxsb3cgc2F0
ZSAxIGFuZCAxLjUgUlBDcyBhYm92ZS4gU29tZS9hbGwgYnl0ZXMgd2hlcmUNCj4gcHJlc2VudGVk
IG9uIHRoZSB3aXJlLCB1bnRpbCB0aGUgTEFZT1VUUkVUVVJOLCBmcm9tIHdoaWNoIHBvaW50IGFs
bA0KPiBSUENzIGFyZSBoYXJkIGFib3J0ZWQgYW5kIG5vdCBhIHNpbmdsZSBieXRlIGlzIHNlbnQu
DQo+IA0KPiANCj4gPiBPdGhlcnMgY29udGVuZCB0aGF0IHNpbmNlIHRoZSBpbi1mbGlnaHQgUlBD
cyByZWZlcmVuY2UgdGhlIHJldHVybmVkDQo+ID4gbGF5b3V0LCB0aGUgY2xpZW50IGlzIHN0aWxs
ICd1c2luZycgdGhlIGxheW91dCB3aXRoIHRoZXNlIGluLWZsaWdodA0KPiA+IHJlcXVlc3RzLCBh
bmQgY2FuIG5vdCBjYWxsIExBWU9VVFJFVFVSTiB1bnRpbCBhbGwgaW4tZmxpZ2h0IFJQQ3MNCj4g
PiByZXR1cm4sIHdpdGggb3Igd2l0aG91dCBhbiBlcnJvci4NCj4gPiANCj4gDQo+IA0KPiBXaXRo
IG91ciBjbGllbnQgY29kZSBJIGRvbid0IHNlZSBob3cgdGhlIGd1YXJhbnR5IG9mIDIgYW5kIDMg
YWJvdmUNCj4gd2lsbCBoYXBwZW4gd2l0aG91dCBhY3R1YWxseSBpbXBsZW1lbnRpbmcgdGhpcyBo
ZXJlLg0KPiANCj4gU28gaW4gcHJpbmNpcGFsIEkgYWdyZWUgd2l0aCB5b3VyIHByaW5jaXBsZSwg
SSBvbmx5IGRvIG5vdCBhZ3JlZQ0KPiB3aXRoIHlvdXIgcHJhY3RpY2UuIEluIHlvdXIgbmV3IGNv
ZGUgeW91IGFyZSB2aW9sYXRpbmcgMiBhbmQgMw0KPiB3aGljaCBhcmUgbm90IHRvIGJlIGFsbG93
ZWQuDQo+IA0KPiBBbmQgYWdhaW4sIHBsZWFzZSBleHBsYWluIHdoeSBkbyB5b3Ugd2FudCBpdC4g
V2hhdCBpcyB3cm9uZyB3aXRoIHRoZQ0KPiBjYXNlIHdlIGFsbCBhZ3JlZSB3aXRoPyBpZTogIkNs
aWVudCBjYW4gbm90IGNhbGwgTEFZT1VUUkVUVVJOIHVudGlsDQo+IGFsbCBpbi1mbGlnaHQgUlBD
cyByZXR1cm4sIHdpdGggb3Igd2l0aG91dCBhbiBlcnJvciINCj4gDQo+IFRoYW5rcw0KPiBCb2F6
DQo+IA0KPiA+IA0KPiA+IFNlY3Rpb24gMTguNDQuMyAtIHRoZSBkZXNjcmlwdGlvbiBzZWN0aW9u
IG9mIHRoZSBMQVlPVVRSRVRVUk4gb3BlcmF0aW9uOg0KPiA+IA0KPiA+ICAgIEFmdGVyIHRoaXMg
Y2FsbCwNCj4gPiAgICB0aGUgY2xpZW50IE1VU1QgTk9UIHVzZSB0aGUgcmV0dXJuZWQgbGF5b3V0
KHMpIGFuZCB0aGUgYXNzb2NpYXRlZA0KPiA+ICAgIHN0b3JhZ2UgcHJvdG9jb2wgdG8gYWNjZXNz
IHRoZSBmaWxlIGRhdGEuDQo+ID4gDQo+ID4gU2VjdGlvbiAxMy42IE9wZXJhdGlvbnMgU2VudCB0
byBORlN2NC4xIERhdGEgU2VydmVycw0KPiA+IA0KPiA+ICAgQXMgZGVzY3JpYmVkIGluIFNlY3Rp
b24gMTIuNS4xLCBhIGNsaWVudA0KPiA+ICAgTVVTVCBOT1Qgc2VuZCBhbiBJL08gdG8gYSBkYXRh
IHNlcnZlciBmb3Igd2hpY2ggaXQgZG9lcyBub3QgaG9sZCBhDQo+ID4gICB2YWxpZCBsYXlvdXQ7
IHRoZSBkYXRhIHNlcnZlciBNVVNUIHJlamVjdCBzdWNoIGFuIEkvTy4NCj4gPiANCj4gPiANCj4g
PiAtLT5BbmR5DQo+IA0KPiANCj4gX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX18NCj4gbmZzdjQgbWFpbGluZyBsaXN0DQo+IG5mc3Y0QGlldGYub3JnDQo+IGh0
dHBzOi8vd3d3LmlldGYub3JnL21haWxtYW4vbGlzdGluZm8vbmZzdjQNCj4gDQoNCi0tIA0KVHJv
bmQgTXlrbGVidXN0DQpMaW51eCBORlMgY2xpZW50IG1haW50YWluZXINCg0KTmV0QXBwDQpUcm9u
ZC5NeWtsZWJ1c3RAbmV0YXBwLmNvbQ0Kd3d3Lm5ldGFwcC5jb20NCg0K
T24gVHVlLCAyMDEyLTA2LTEyIGF0IDAxOjE4ICswMzAwLCBCb2F6IEhhcnJvc2ggd3JvdGU6DQo+
IE9uIDA2LzExLzIwMTIgMDk6NTkgUE0sIE15a2xlYnVzdCwgVHJvbmQgd3JvdGU6DQo+IA0KPiA+
IE9uIE1vbiwgMjAxMi0wNi0xMSBhdCAyMTo0MCArMDMwMCwgQm9heiBIYXJyb3NoIHdyb3RlOg0K
PiA+IA0KPiA+PiBBbmQgYWdhaW4sIHBsZWFzZSBleHBsYWluIHdoeSBkbyB5b3Ugd2FudCBpdC4g
V2hhdCBpcyB3cm9uZyB3aXRoIHRoZQ0KPiA+PiBjYXNlIHdlIGFsbCBhZ3JlZSB3aXRoPyBpZTog
IkNsaWVudCBjYW4gbm90IGNhbGwgTEFZT1VUUkVUVVJOIHVudGlsDQo+ID4+IGFsbCBpbi1mbGln
aHQgUlBDcyByZXR1cm4sIHdpdGggb3Igd2l0aG91dCBhbiBlcnJvciINCj4gPiANCj4gPiBXaG8g
ImFncmVlZCIgdG8gdGhpcz8gVGhpcyB3b3VsZCBtZWFuIHRoYXQgaWYgdGhlIERTIGdvZXMgZG93
biwgd2UgY2FuJ3QNCj4gPiBldmVyIHNlbmQgTEFZT1VUUkVUVVJOIHdoaWNoIGlzIHBhdGVudGx5
IHdyb25nLg0KPiA+IA0KPiANCj4gDQo+ICJEUyBnb2VzIGRvd24iIGlzIHVuZGVyIHRoZSBhYm92
ZSAiUlBDIHJldHVybiBhbiBlcnJvciIgdGhlIGVycm9yIGNvbmRpdGlvbg0KPiBvZiBhbiBSUEMg
aXMgd2VsbCBkZWZpbmVkLg0KDQo/Pz8/IE5vdyB5b3UgaGF2ZSBtZSBleHRyZW1lbHkgY29uZnVz
ZWQuIEhvdyBkb2VzIHRoZSBjbGllbnQgZGlzdGluZ3Vpc2gNCmJldHdlZW4gRVRJTUVET1VULWJl
Y2F1c2UtRFMtd2VudC1kb3duIGFuZA0KRVRJTUVET1VULWJ1dC1pbi1mbGlnaHQtUlBDcy13aWxs
LWV2ZW50dWFsbHktc3VjY2VlZC1zby1wbGVhc2UtaG9sZC10aGF0LUxBWU9VVFJFVFVSTj8NCg0K
PiA+RnJvbSB3aGF0IG9mIG15IHdvcmRzIGRpZCB5b3UgdW5kZXJzdGFuZCB0aGF0IEkgc2FpZA0K
PiAJIndlIGNhbid0IGV2ZXIgc2VuZCBhIExBWU9VVFJFVFVSTiINCj4gDQo+IElmIG15IEVuZ2xp
c2ggaXMgd3JvbmdseSB3b3JkZWQuIFdoaWNoIGlzIHBlcmZlY3RseSBwb3NzaWJsZS4gUGxlYXNl
IGNvcnJlY3QNCj4gbWUgc28gSSBjYW4gbGVhcm4uIERpZCB5b3UgaG9uZXN0bHkgdGhpbmsgdGhh
dCdzIHdoYXQgSSBtZWFudD8gDQo+IA0KPiBJIG1lYW50IHdlIGFsbCBhZ3JlZSwgdGhhdCB0aGlz
IGNhc2UgaXMgY292ZXJlZCBieSBSRkMuIFRoYXQgaXMgIC0gbm8gb25lIHdvdWxkDQo+IGFjY3Vz
ZSBhIGNsaWVudCB3aG8gZG9lcyB0aGF0LCBhcyB2aW9sYXRpbmcgdGhlIFJGQy4NCj4gDQo+IEFu
ZCBhZ2FpbiBteSBxdWVzdGlvbi4gVGhlIG1vdGl2YXRpb24/DQoNCk1ha2luZyBmYWxsYmFjay10
by1NRFMgd29yayB3aXRob3V0IGNvcnJ1cHRpbmcgeW91ciBkYXRhLiBQbGVhc2UgcmVhZCB1cA0K
b24gbXVsdGlwYXRoLTEwMSBhbmQgdGhlIG5lZWQgZm9yIGZlbmNpbmcuLi4NCg0KLS0gDQpUcm9u
ZCBNeWtsZWJ1c3QNCkxpbnV4IE5GUyBjbGllbnQgbWFpbnRhaW5lcg0KDQpOZXRBcHANClRyb25k
Lk15a2xlYnVzdEBuZXRhcHAuY29tDQp3d3cubmV0YXBwLmNvbQ0KDQo=
T24gTW9uLCAyMDEyLTA2LTExIGF0IDIxOjQwICswMzAwLCBCb2F6IEhhcnJvc2ggd3JvdGU6DQoN
Cj4gQW5kIGFnYWluLCBwbGVhc2UgZXhwbGFpbiB3aHkgZG8geW91IHdhbnQgaXQuIFdoYXQgaXMg
d3Jvbmcgd2l0aCB0aGUNCj4gY2FzZSB3ZSBhbGwgYWdyZWUgd2l0aD8gaWU6ICJDbGllbnQgY2Fu
IG5vdCBjYWxsIExBWU9VVFJFVFVSTiB1bnRpbA0KPiBhbGwgaW4tZmxpZ2h0IFJQQ3MgcmV0dXJu
LCB3aXRoIG9yIHdpdGhvdXQgYW4gZXJyb3IiDQoNCldobyAiYWdyZWVkIiB0byB0aGlzPyBUaGlz
IHdvdWxkIG1lYW4gdGhhdCBpZiB0aGUgRFMgZ29lcyBkb3duLCB3ZSBjYW4ndA0KZXZlciBzZW5k
IExBWU9VVFJFVFVSTiB3aGljaCBpcyBwYXRlbnRseSB3cm9uZy4NCg0KLS0gDQpUcm9uZCBNeWts
ZWJ1c3QNCkxpbnV4IE5GUyBjbGllbnQgbWFpbnRhaW5lcg0KDQpOZXRBcHANClRyb25kLk15a2xl
YnVzdEBuZXRhcHAuY29tDQp3d3cubmV0YXBwLmNvbQ0KDQo=
> And again, please explain why do you want it. What is wrong with the
> case we all agree with? ie: "Client can not call LAYOUTRETURN until
> all in-flight RPCs return, with or without an error"
It's a recipe for data corruption. If, as Andy explained, he starts doing
IO's (let's suppose WRITEs) to the MDS any lingering WRITEs to the DS
since they reflect an earlier state of affairs can cause data corruption.
There are three ways to prevent those lingering DS writes from corrupting
data:
1) Doing a LAYOUTRETURN
2) waiting until the IO's return.
3) "magically plugging the network interface".
Since there is no way to do 3), saying that you only can do 1) until after
2) is done is essentially going to mean:
a) that it may take a very long time:
b) that you will only do it, when it is no longer useful.
If you do 1) asap, then the lingering DS write problem is gone sooner,
and that's a good thing.
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Boaz Harrosh
Sent: Monday, June 11, 2012 2:41 PM
To: Andy Adamson
Cc: Andy Adamson; NFS list; Trond Myklebust; NFSv4
Subject: Re: [nfsv4] RFC 5661 LAYOUTRETURN clarification.
On 06/11/2012 07:01 PM, Andy Adamson wrote:
> I'm coding file layout data server recovery for the Linux NFS client,
> and came across an issue with LAYOUTRETURN that
> could use some comment from the list.
>
> The error case I'm handling is an RPC layer dis-connection error
> during heavy WRITE i/o to a file layout data server. Our response is
> to internally mark the deviceid as invalid which prevents all pNFS
> calls using the deviceid - e.g. no new I/O using any layout that uses
> the invalid deviceid, and to redirect all I/O to the MDS (any queued
> RPC request that has not been sent is redirected to the MDS).
>
> Plus - and here is where the clarification is needed - we immediately
> send a LAYOUTRETURN for any layout with in-flight requests to the
> dis-connected data server. By in-flight I mean transmitted WRT the
> RPC layer. The purpose of this LAYOUTRETURN is to notify the file
> layout MDS to fence the DS for the specified LAYOUTs, as the WRITEs
> will also be sent to the MDS.
>
I do not disagree with this completely. The point here is very fine
grained and should be specified explicitly. I would like to see text
as of something like.
There are 3 types of in-flght RPC/IO
1. Client has sent RPC header + all of associated data and is waiting
for DS WRITE/READ_DONE reply.
(For me this case can be, client may return LAYOUTRETURN as your
suggestion)
2. Client has sent the RPC header but has got stuck sending the rest
of the RPC message. Then received a network disconnect. This is the
most common part. Putting aside the RPC that got the error for a second.
The most important is what to do with parallel RPC/IO which are in this
state. Are parallel RPCs allowed to continue sending network packets
after the LAYOUTRETURN was sent?
The specific RPC that got stuck is not interesting because it's kind of
1.5, We are not going to send any bytes on that channel. The interesting
is these other DSs which are still streaming
3. Client has some internal RPC queue which do to some client parallelism
will start sending RPC header + data after the LAYOUTRETURN was sent
What my point was that with the code you submitted we are clearly violating
2. and even 3. Because I do not see anything avoiding this.
And if the STD allows you 2 and 3. Then that's a big change to the concept.
Not like you let it seem.
> I contend that sending the LAYOUTRETURN in this error case does not
> violate the two sections of RFC 5661 below, as the client has stopped
> sending any I/O requests using the returned layout.
>
I would not mind if this was true. That is if the LAYOUTRETURN was
a very clear barrier where our client would "magically" completely
plug the network interface and will not continue to send a single
byte on the wire to *any* DS involved with the layout. That's fine.
That is only allow sate 1 and 1.5 RPCs above. Some/all bytes where
presented on the wire, until the LAYOUTRETURN, from which point all
RPCs are hard aborted and not a single byte is sent.
> Others contend that since the in-flight RPCs reference the returned
> layout, the client is still 'using' the layout with these in-flight
> requests, and can not call LAYOUTRETURN until all in-flight RPCs
> return, with or without an error.
>
With our client code I don't see how the guaranty of 2 and 3 above
will happen without actually implementing this here.
So in principal I agree with your principle, I only do not agree
with your practice. In your new code you are violating 2 and 3
which are not to be allowed.
And again, please explain why do you want it. What is wrong with the
case we all agree with? ie: "Client can not call LAYOUTRETURN until
all in-flight RPCs return, with or without an error"
Thanks
Boaz
>
> Section 18.44.3 - the description section of the LAYOUTRETURN operation:
>
> After this call,
> the client MUST NOT use the returned layout(s) and the associated
> storage protocol to access the file data.
>
> Section 13.6 Operations Sent to NFSv4.1 Data Servers
>
> As described in Section 12.5.1, a client
> MUST NOT send an I/O to a data server for which it does not hold a
> valid layout; the data server MUST reject such an I/O.
>
>
> -->Andy
_______________________________________________
nfsv4 mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/nfsv4