FYI, I found this in my notes. It gives a flavor of the
kind of lock contention we were discussing last week.
My client is a 12-core dual socket system. Networking
is 56Gbps IB, NFS/RDMA. I'm running some multi-threaded
iozone tests. Here I was mixing buffered and direct I/O
tests, so I can't say for certain if one is worse than
the other.
The output below is the first entry in /proc/lock_stat.
"sb->s_type->i_lock_key" looked like it was related to
the superblock, but looking at the actual code in the
functions named below, this is probably the inode lock.
The max lock holdtime is 82 milliseconds, if I read this
correctly. It's not clear how often that happens, but
that looks pathological.
lock_stat version 0.4
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
class name con-bounces contentions waittime-min waittime-max waittime-total waittime-avg acq-bounces acquisitions holdtime-min holdtime-max holdtime-total holdtime-avg
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
&sb->s_type->i_lock_key#2: 23185746 23275952 0.10 81771.03 27980172.31 1.20 81467297 238209945 0.09 81842.12 80513710.68 0.34
-------------------------
&sb->s_type->i_lock_key#2 1453378 [<ffffffffa06c6e49>] nfs_flush_incompatible+0x75/0x1ad [nfs]
&sb->s_type->i_lock_key#2 4777000 [<ffffffffa06c4ed6>] nfs_lock_and_join_requests+0x68/0x2a3 [nfs]
&sb->s_type->i_lock_key#2 157315 [<ffffffffa06c72f5>] nfs_updatepage+0x374/0x912 [nfs]
&sb->s_type->i_lock_key#2 2085447 [<ffffffffa06c74d7>] nfs_updatepage+0x556/0x912 [nfs]
-------------------------
&sb->s_type->i_lock_key#2 132 [<ffffffff8123cc2c>] writeback_sb_inodes+0xfb/0x50c
&sb->s_type->i_lock_key#2 7858928 [<ffffffffa06c4ed6>] nfs_lock_and_join_requests+0x68/0x2a3 [nfs]
&sb->s_type->i_lock_key#2 1225835 [<ffffffffa06c6e49>] nfs_flush_incompatible+0x75/0x1ad [nfs]
&sb->s_type->i_lock_key#2 111821 [<ffffffffa06c72f5>] nfs_updatepage+0x374/0x912 [nfs]
--
Chuck Lever
T24gTW9uLCAyMDE3LTA3LTE3IGF0IDExOjIyICswMjAwLCBDaHVjayBMZXZlciB3cm90ZToNCj4g
RllJLCBJIGZvdW5kIHRoaXMgaW4gbXkgbm90ZXMuIEl0IGdpdmVzIGEgZmxhdm9yIG9mIHRoZQ0K
PiBraW5kIG9mIGxvY2sgY29udGVudGlvbiB3ZSB3ZXJlIGRpc2N1c3NpbmcgbGFzdCB3ZWVrLg0K
PiANCj4gTXkgY2xpZW50IGlzIGEgMTItY29yZSBkdWFsIHNvY2tldCBzeXN0ZW0uIE5ldHdvcmtp
bmcNCj4gaXMgNTZHYnBzIElCLCBORlMvUkRNQS4gSSdtIHJ1bm5pbmcgc29tZSBtdWx0aS10aHJl
YWRlZA0KPiBpb3pvbmUgdGVzdHMuIEhlcmUgSSB3YXMgbWl4aW5nIGJ1ZmZlcmVkIGFuZCBkaXJl
Y3QgSS9PDQo+IHRlc3RzLCBzbyBJIGNhbid0IHNheSBmb3IgY2VydGFpbiBpZiBvbmUgaXMgd29y
c2UgdGhhbg0KPiB0aGUgb3RoZXIuDQo+IA0KPiBUaGUgb3V0cHV0IGJlbG93IGlzIHRoZSBmaXJz
dCBlbnRyeSBpbiAvcHJvYy9sb2NrX3N0YXQuDQo+IA0KPiAic2ItPnNfdHlwZS0+aV9sb2NrX2tl
eSIgbG9va2VkIGxpa2UgaXQgd2FzIHJlbGF0ZWQgdG8NCj4gdGhlIHN1cGVyYmxvY2ssIGJ1dCBs
b29raW5nIGF0IHRoZSBhY3R1YWwgY29kZSBpbiB0aGUNCj4gZnVuY3Rpb25zIG5hbWVkIGJlbG93
LCB0aGlzIGlzIHByb2JhYmx5IHRoZSBpbm9kZSBsb2NrLg0KPiANCj4gVGhlIG1heCBsb2NrIGhv
bGR0aW1lIGlzIDgyIG1pbGxpc2Vjb25kcywgaWYgSSByZWFkIHRoaXMNCj4gY29ycmVjdGx5LiBJ
dCdzIG5vdCBjbGVhciBob3cgb2Z0ZW4gdGhhdCBoYXBwZW5zLCBidXQNCj4gdGhhdCBsb29rcyBw
YXRob2xvZ2ljYWwuDQo+IA0KPiANCj4gbG9ja19zdGF0IHZlcnNpb24gMC40DQo+IC0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0NCj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLQ0KPiAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+IC0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGNsYXNzIG5hbWUgICAgY29uLQ0KPiBib3Vu
Y2VzICAgIGNvbnRlbnRpb25zICAgd2FpdHRpbWUtbWluICAgd2FpdHRpbWUtbWF4IHdhaXR0aW1l
LQ0KPiB0b3RhbCAgIHdhaXR0aW1lLWF2ZyAgICBhY3EtYm91bmNlcyAgIGFjcXVpc2l0aW9ucyAg
IGhvbGR0aW1lLQ0KPiBtaW4gICBob2xkdGltZS1tYXggaG9sZHRpbWUtdG90YWwgICBob2xkdGlt
ZS1hdmcNCj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+IC0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4gLS0tLS0t
LS0tLS0tLS0tLS0tLS0NCj4gDQo+ICAgICAgICAgICAgICAgICZzYi0+c190eXBlLQ0KPiA+aV9s
b2NrX2tleSMyOiAgICAgIDIzMTg1NzQ2ICAgICAgIDIzMjc1OTUyICAgICAgICAgICAwLjEwICAg
ICAgIDgxNzcNCj4gMS4wMyAgICAyNzk4MDE3Mi4zMSAgICAgICAgICAgMS4yMCAgICAgICA4MTQ2
NzI5NyAgICAgIDIzODIwOTk0NSAgICAgDQo+ICAgICAgIDAuMDkgICAgICAgODE4NDIuMTIgICAg
ODA1MTM3MTAuNjggICAgICAgICAgIDAuMzQNCj4gICAgICAgICAgICAgICAgLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLQ0KPiAgICAgICAgICAgICAgICAmc2ItPnNfdHlwZS0NCj4gPmlfbG9ja19r
ZXkjMiAgICAgICAgMTQ1MzM3OCAgICAgICAgICBbPGZmZmZmZmZmYTA2YzZlNDk+XQ0KPiBuZnNf
Zmx1c2hfaW5jb21wYXRpYmxlKzB4NzUvMHgxYWQgW25mc10NCj4gICAgICAgICAgICAgICAgJnNi
LT5zX3R5cGUtDQo+ID5pX2xvY2tfa2V5IzIgICAgICAgIDQ3NzcwMDAgICAgICAgICAgWzxmZmZm
ZmZmZmEwNmM0ZWQ2Pl0NCj4gbmZzX2xvY2tfYW5kX2pvaW5fcmVxdWVzdHMrMHg2OC8weDJhMyBb
bmZzXQ0KPiAgICAgICAgICAgICAgICAmc2ItPnNfdHlwZS0NCj4gPmlfbG9ja19rZXkjMiAgICAg
ICAgIDE1NzMxNSAgICAgICAgICBbPGZmZmZmZmZmYTA2YzcyZjU+XQ0KPiBuZnNfdXBkYXRlcGFn
ZSsweDM3NC8weDkxMiBbbmZzXQ0KPiAgICAgICAgICAgICAgICAmc2ItPnNfdHlwZS0NCj4gPmlf
bG9ja19rZXkjMiAgICAgICAgMjA4NTQ0NyAgICAgICAgICBbPGZmZmZmZmZmYTA2Yzc0ZDc+XQ0K
PiBuZnNfdXBkYXRlcGFnZSsweDU1Ni8weDkxMiBbbmZzXQ0KPiAgICAgICAgICAgICAgICAtLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+ICAgICAgICAgICAgICAgICZzYi0+c190eXBlLQ0KPiA+
aV9sb2NrX2tleSMyICAgICAgICAgICAgMTMyICAgICAgICAgIFs8ZmZmZmZmZmY4MTIzY2MyYz5d
DQo+IHdyaXRlYmFja19zYl9pbm9kZXMrMHhmYi8weDUwYw0KPiAgICAgICAgICAgICAgICAmc2It
PnNfdHlwZS0NCj4gPmlfbG9ja19rZXkjMiAgICAgICAgNzg1ODkyOCAgICAgICAgICBbPGZmZmZm
ZmZmYTA2YzRlZDY+XQ0KPiBuZnNfbG9ja19hbmRfam9pbl9yZXF1ZXN0cysweDY4LzB4MmEzIFtu
ZnNdDQo+ICAgICAgICAgICAgICAgICZzYi0+c190eXBlLQ0KPiA+aV9sb2NrX2tleSMyICAgICAg
ICAxMjI1ODM1ICAgICAgICAgIFs8ZmZmZmZmZmZhMDZjNmU0OT5dDQo+IG5mc19mbHVzaF9pbmNv
bXBhdGlibGUrMHg3NS8weDFhZCBbbmZzXQ0KPiAgICAgICAgICAgICAgICAmc2ItPnNfdHlwZS0N
Cj4gPmlfbG9ja19rZXkjMiAgICAgICAgIDExMTgyMSAgICAgICAgICBbPGZmZmZmZmZmYTA2Yzcy
ZjU+XQ0KPiBuZnNfdXBkYXRlcGFnZSsweDM3NC8weDkxMiBbbmZzXQ0KPiANCj4gDQpJbnRlcmVz
dGluZy4uLiBUaGVyZSBhcmUgZGVmaW5pdGVseSBzb21lIG9wdGltaXNhdGlvbnMgd2UgY2FuIGRv
IHRoZXJlDQppbiBvcmRlciB0byBnZXQgcmlkIG9mIHNvbWUgb2YgdGhpcyBjb250ZW50aW9uIChm
b3IgaW5zdGFuY2Ugc2ltcGxlDQp0aGluZ3MgbGlrZSBkb2luZyBhIGxvY2tsZXNzIGNoZWNrIGZv
cg0KUGFnZVByaXZhdGUocGFnZSl8fFBhZ2VTd2FwQ2FjaGUocGFnZSkgYmVmb3JlIHdlIGdyYWIg
dGhlIGxvY2sgaW4NCm5mc19wYWdlX2ZpbmRfaGVhZF9yZXF1ZXN0KCkuDQoNClRoYW5rcyBDaHVj
ayEgSSdsbCB0YWtlIGEgZGVlcGVyIGxvb2sgYXQgdGhpcyB0b2RheS4NCg0KQ2hlZXJzDQogIFRy
b25kDQoNCi0tIA0KVHJvbmQgTXlrbGVidXN0DQpMaW51eCBORlMgY2xpZW50IG1haW50YWluZXIs
IFByaW1hcnlEYXRhDQp0cm9uZC5teWtsZWJ1c3RAcHJpbWFyeWRhdGEuY29tDQo=
T24gTW9uLCAyMDE3LTA3LTE3IGF0IDEzOjI2ICswMDAwLCBUcm9uZCBNeWtsZWJ1c3Qgd3JvdGU6
DQo+IE9uIE1vbiwgMjAxNy0wNy0xNyBhdCAxMToyMiArMDIwMCwgQ2h1Y2sgTGV2ZXIgd3JvdGU6
DQo+ID4gRllJLCBJIGZvdW5kIHRoaXMgaW4gbXkgbm90ZXMuIEl0IGdpdmVzIGEgZmxhdm9yIG9m
IHRoZQ0KPiA+IGtpbmQgb2YgbG9jayBjb250ZW50aW9uIHdlIHdlcmUgZGlzY3Vzc2luZyBsYXN0
IHdlZWsuDQo+ID4gDQo+ID4gTXkgY2xpZW50IGlzIGEgMTItY29yZSBkdWFsIHNvY2tldCBzeXN0
ZW0uIE5ldHdvcmtpbmcNCj4gPiBpcyA1NkdicHMgSUIsIE5GUy9SRE1BLiBJJ20gcnVubmluZyBz
b21lIG11bHRpLXRocmVhZGVkDQo+ID4gaW96b25lIHRlc3RzLiBIZXJlIEkgd2FzIG1peGluZyBi
dWZmZXJlZCBhbmQgZGlyZWN0IEkvTw0KPiA+IHRlc3RzLCBzbyBJIGNhbid0IHNheSBmb3IgY2Vy
dGFpbiBpZiBvbmUgaXMgd29yc2UgdGhhbg0KPiA+IHRoZSBvdGhlci4NCj4gPiANCj4gPiBUaGUg
b3V0cHV0IGJlbG93IGlzIHRoZSBmaXJzdCBlbnRyeSBpbiAvcHJvYy9sb2NrX3N0YXQuDQo+ID4g
DQo+ID4gInNiLT5zX3R5cGUtPmlfbG9ja19rZXkiIGxvb2tlZCBsaWtlIGl0IHdhcyByZWxhdGVk
IHRvDQo+ID4gdGhlIHN1cGVyYmxvY2ssIGJ1dCBsb29raW5nIGF0IHRoZSBhY3R1YWwgY29kZSBp
biB0aGUNCj4gPiBmdW5jdGlvbnMgbmFtZWQgYmVsb3csIHRoaXMgaXMgcHJvYmFibHkgdGhlIGlu
b2RlIGxvY2suDQo+ID4gDQo+ID4gVGhlIG1heCBsb2NrIGhvbGR0aW1lIGlzIDgyIG1pbGxpc2Vj
b25kcywgaWYgSSByZWFkIHRoaXMNCj4gPiBjb3JyZWN0bHkuIEl0J3Mgbm90IGNsZWFyIGhvdyBv
ZnRlbiB0aGF0IGhhcHBlbnMsIGJ1dA0KPiA+IHRoYXQgbG9va3MgcGF0aG9sb2dpY2FsLg0KPiA+
IA0KPiA+IA0KPiA+IGxvY2tfc3RhdCB2ZXJzaW9uIDAuNA0KPiA+IC0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4gPiAt
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tDQo+ID4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiA+IC0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+ID4g
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgY2xhc3MgbmFtZSAgICBjb24tDQo+ID4gYm91
bmNlcyAgICBjb250ZW50aW9ucyAgIHdhaXR0aW1lLW1pbiAgIHdhaXR0aW1lLW1heCB3YWl0dGlt
ZS0NCj4gPiB0b3RhbCAgIHdhaXR0aW1lLWF2ZyAgICBhY3EtYm91bmNlcyAgIGFjcXVpc2l0aW9u
cyAgIGhvbGR0aW1lLQ0KPiA+IG1pbiAgIGhvbGR0aW1lLW1heCBob2xkdGltZS10b3RhbCAgIGhv
bGR0aW1lLWF2Zw0KPiA+IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4gPiAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+ID4gLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LQ0KPiA+IC0tLS0tLS0tLS0tLS0tLS0tLS0tDQo+ID4gDQo+ID4gICAgICAgICAgICAgICAgJnNi
LT5zX3R5cGUtDQo+ID4gPiBpX2xvY2tfa2V5IzI6ICAgICAgMjMxODU3NDYgICAgICAgMjMyNzU5
NTIgICAgICAgICAgIDAuMTAgICAgICAgOA0KPiA+ID4gMTc3DQo+ID4gDQo+ID4gMS4wMyAgICAy
Nzk4MDE3Mi4zMSAgICAgICAgICAgMS4yMCAgICAgICA4MTQ2NzI5NyAgICAgIDIzODIwOTk0NSAg
IA0KPiA+ICAgDQo+ID4gICAgICAgMC4wOSAgICAgICA4MTg0Mi4xMiAgICA4MDUxMzcxMC42OCAg
ICAgICAgICAgMC4zNA0KPiA+ICAgICAgICAgICAgICAgIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0NCj4gPiAgICAgICAgICAgICAgICAmc2ItPnNfdHlwZS0NCj4gPiA+IGlfbG9ja19rZXkjMiAg
ICAgICAgMTQ1MzM3OCAgICAgICAgICBbPGZmZmZmZmZmYTA2YzZlNDk+XQ0KPiA+IA0KPiA+IG5m
c19mbHVzaF9pbmNvbXBhdGlibGUrMHg3NS8weDFhZCBbbmZzXQ0KPiA+ICAgICAgICAgICAgICAg
ICZzYi0+c190eXBlLQ0KPiA+ID4gaV9sb2NrX2tleSMyICAgICAgICA0Nzc3MDAwICAgICAgICAg
IFs8ZmZmZmZmZmZhMDZjNGVkNj5dDQo+ID4gDQo+ID4gbmZzX2xvY2tfYW5kX2pvaW5fcmVxdWVz
dHMrMHg2OC8weDJhMyBbbmZzXQ0KPiA+ICAgICAgICAgICAgICAgICZzYi0+c190eXBlLQ0KPiA+
ID4gaV9sb2NrX2tleSMyICAgICAgICAgMTU3MzE1ICAgICAgICAgIFs8ZmZmZmZmZmZhMDZjNzJm
NT5dDQo+ID4gDQo+ID4gbmZzX3VwZGF0ZXBhZ2UrMHgzNzQvMHg5MTIgW25mc10NCj4gPiAgICAg
ICAgICAgICAgICAmc2ItPnNfdHlwZS0NCj4gPiA+IGlfbG9ja19rZXkjMiAgICAgICAgMjA4NTQ0
NyAgICAgICAgICBbPGZmZmZmZmZmYTA2Yzc0ZDc+XQ0KPiA+IA0KPiA+IG5mc191cGRhdGVwYWdl
KzB4NTU2LzB4OTEyIFtuZnNdDQo+ID4gICAgICAgICAgICAgICAgLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLQ0KPiA+ICAgICAgICAgICAgICAgICZzYi0+c190eXBlLQ0KPiA+ID4gaV9sb2NrX2tl
eSMyICAgICAgICAgICAgMTMyICAgICAgICAgIFs8ZmZmZmZmZmY4MTIzY2MyYz5dDQo+ID4gDQo+
ID4gd3JpdGViYWNrX3NiX2lub2RlcysweGZiLzB4NTBjDQo+ID4gICAgICAgICAgICAgICAgJnNi
LT5zX3R5cGUtDQo+ID4gPiBpX2xvY2tfa2V5IzIgICAgICAgIDc4NTg5MjggICAgICAgICAgWzxm
ZmZmZmZmZmEwNmM0ZWQ2Pl0NCj4gPiANCj4gPiBuZnNfbG9ja19hbmRfam9pbl9yZXF1ZXN0cysw
eDY4LzB4MmEzIFtuZnNdDQo+ID4gICAgICAgICAgICAgICAgJnNiLT5zX3R5cGUtDQo+ID4gPiBp
X2xvY2tfa2V5IzIgICAgICAgIDEyMjU4MzUgICAgICAgICAgWzxmZmZmZmZmZmEwNmM2ZTQ5Pl0N
Cj4gPiANCj4gPiBuZnNfZmx1c2hfaW5jb21wYXRpYmxlKzB4NzUvMHgxYWQgW25mc10NCj4gPiAg
ICAgICAgICAgICAgICAmc2ItPnNfdHlwZS0NCj4gPiA+IGlfbG9ja19rZXkjMiAgICAgICAgIDEx
MTgyMSAgICAgICAgICBbPGZmZmZmZmZmYTA2YzcyZjU+XQ0KPiA+IA0KPiA+IG5mc191cGRhdGVw
YWdlKzB4Mzc0LzB4OTEyIFtuZnNdDQo+ID4gDQo+ID4gDQo+IA0KPiBJbnRlcmVzdGluZy4uLiBU
aGVyZSBhcmUgZGVmaW5pdGVseSBzb21lIG9wdGltaXNhdGlvbnMgd2UgY2FuIGRvDQo+IHRoZXJl
DQo+IGluIG9yZGVyIHRvIGdldCByaWQgb2Ygc29tZSBvZiB0aGlzIGNvbnRlbnRpb24gKGZvciBp
bnN0YW5jZSBzaW1wbGUNCj4gdGhpbmdzIGxpa2UgZG9pbmcgYSBsb2NrbGVzcyBjaGVjayBmb3IN
Cj4gUGFnZVByaXZhdGUocGFnZSl8fFBhZ2VTd2FwQ2FjaGUocGFnZSkgYmVmb3JlIHdlIGdyYWIg
dGhlIGxvY2sgaW4NCj4gbmZzX3BhZ2VfZmluZF9oZWFkX3JlcXVlc3QoKS4NCj4gDQo+IFRoYW5r
cyBDaHVjayEgSSdsbCB0YWtlIGEgZGVlcGVyIGxvb2sgYXQgdGhpcyB0b2RheS4NCj4gDQoNClNv
LCB0aGUgb3RoZXIgdGhpbmcgdG8gbm90ZSBpcyB0aGF0IHRoZSBpbm9kZS0+aV9sb2NrIHByb3Rl
Y3Rpb24gaW4NCm5mc19sb2NrX2FuZF9qb2luX3JlcXVlc3RzKCkgaXMgcHJldHR5IG11Y2ggd29y
dGhsZXNzIG9uY2Ugd2UndmUgZm91bmQNCnRoZSBwYWdlIGhlYWQuIEFmdGVyIHRoYXQsIHdlIGNh
biBhbmQgc2hvdWxkIHJlbHkgb24gdGhlIHBhZ2UgYW5kIHBhZ2UNCmdyb3VwIGxvY2tzLiBJJ20g
cnVubmluZyBhIHRlc3Qgb24gYSBzZXQgb2YgcGF0Y2hlcyB0aGF0IGNsZWFuIHVwIHRoZQ0KbG9j
a2luZyB0aGVyZSBhbmQgc2hvdWxkIGhvcGVmdWxseSBnZXQgcmlkIG9mIHRoZSBjb250ZW50aW9u
Lg0KDQotLSANClRyb25kIE15a2xlYnVzdA0KTGludXggTkZTIGNsaWVudCBtYWludGFpbmVyLCBQ
cmltYXJ5RGF0YQ0KdHJvbmQubXlrbGVidXN0QHByaW1hcnlkYXRhLmNvbQ0K