----- Original Message -----
> From: andros@xxxxxxxxxx
> To: "trond myklebust" <trond.myklebust@xxxxxxxxxxxxxxx>
> Cc: linux-nfs@xxxxxxxxxxxxxxx, "Andy Adamson" <andros@xxxxxxxxxx>
> Sent: Friday, February 5, 2016 4:08:32 PM
> Subject: [PATCH RFC Version 1 0/6] Request for Comment: NFS4.1 Session Trunking
>
> From: Andy Adamson <andros@xxxxxxxxxx>
>
> This early patch set implements NFSv4.1 session trunking utilizing Tronds
> multipath feature. This works against an upstream linux server with
> two NICs as the EXCHANGE_ID info is the same for both NICs.
>
> Unmodified code will create a new nfs_client structure for the mount as
> nfs_match_client fails due to the different full server address.
> Tunking is detected by comparing the eir_server_owner.major/minor/scope
> and the eir_server_impl_id to an existing mount. When trunking is detected,
> the 2nd mount point uses the existing (not new) nfs_client for the new
> mount point.
>
> When trunking is detected, this patch set adds an rpc_xprt for the new server
> address to the exisiting mount, calls BIND_CONN_TO_SESSION, and destroys
> the new clientid (no second mount).
>
> We then get round robin use of the two NICs off one mount point all
> using a single session.
>
>
> Is using mount a good way to setup NFSv4.1 session trunking?
>
> The first two patches fix issues with the multipath code.
>
> SUNRPC: allow rpc_xprt_switch_add_xprt to add xprts on the same net
> This patch adds a flag to allow multipathing within the origial net.
> I don't understand why this was not allowed, so I expect a different
> solution for this issue.
>
>
> Comments please.
>
> Thanks
>
> -->Andy
>
> Andy Adamson (6):
> SUNRPC fix rpc_clnt_add_xprt setup return
> SUNRPC fix rpc_clnt_xprt_iter_init infinite loop
> SUNRPC: allow rpc_xprt_switch_add_xprt to add xprts on the same net
> NFS session trunking support: add trunk xprt to nfs4.1 mount
> NFS add callback_ops to nfs4_proc_bind_conn_to_session_callback
> NFS schedule bind_conn_to_session for new NFSv4.1 transport
>
> fs/nfs/client.c | 2 ++
> fs/nfs/nfs4client.c | 29 +++++++++++++++++++++++++++--
> fs/nfs/nfs4proc.c | 10 ++++++++++
> include/linux/nfs_fs_sb.h | 1 +
> include/linux/sunrpc/clnt.h | 1 +
> include/linux/sunrpc/xprtmultipath.h | 6 +++++-
> net/sunrpc/clnt.c | 20 +++++++++++++-------
> net/sunrpc/xprtmultipath.c | 35
> +++++++++++++++++++++++++++++++++--
> 8 files changed, 92 insertions(+), 12 deletions(-)
>
> --
> 1.8.3.1
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
Tried and approved!
Here is the steps to make it works for those who wants to try it. :
Test configuration :
Running Server Kernel Version : 3.2.0
Running Client Kernel Version : 3.2.0
Patch deployment Client Kernel Version : 4.5.0 rc4+
Debian 7.9
GNS3 1.4.1
Wireshark 1.12.3
VirtualBox 5.0.14
Client and Server are Virtualbox VM's conected each other with a GNS3 Switch
NO AUTHENTIFICATION - this is only for test purpose
+----------+ +----------+
| | 192.168.1.2 /24 +----------+ 192.168.1.3 /24 | |
| +-----------------+ +-----------------+ |
| Client | | Switch | | Server |
| +-----------------+ +-----------------+ |
| | 192.168.2.20/24 +----------+ 192.168.2.30 /24| |
+----------+ +----------+
Steps :
-------------------------------------------------------------------
Server NFS4.1 configuration :
Make sure you have an internet connection.
Commands assuming you are root:
apt-get update && apt-get upgrade
apt-get install nfs-kernel-server
mkdir home/testnfs
chmod 777 home/testnfs
nano etc/exports
#Add these line in the "exports" file to set the "testnfs" folder available
/home/testnfs 192.168.1.2(rw,sync)
/home/testnfs 192.168.2.20(rw,sync)
#end
#Enable NFSv4.1
etc/init.d/nfs-kernel-server stop
nano proc/fs/nfsd/versions
#set +2 +3 +4 -4.1 to +2 +3 +4 +4.1
etc/init.d/nfs-kernel-server start
#server ready
#end server configuration
-------------------------------------------------------------------
Client NFS4.1 configuration :
Commands assuming you are root:
#------------Prepare new Kernel with new patches----------
apt-get install libncurses5-dev gcc make git exuberant-ctags bc libssl-dev
mkdir kernels
cd kernels
git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
#it takes a while...
cd linux-stable
cp /boot/config-`uname -r`* .config
#the -j4 stand for 4 jobs running at the same time, ideal for
multi-core processor.
make -j4
#It takes a while
#--------------------Install the patches------------------
apt-get install patch
cd /
mkdir patch
cd patch
mkdir rpcmultipathpatch
mkdir sessiontrunkingpatch
#Get the Trond Myklebust's patch :
http://www.spinics.net/lists/linux-nfs/msg56365.html
nano rpcmultipathpatch/patch1.patch
.
.
.
nano rpcmultipathpatch/patch13.patch
#Get the Andy Adamson's patches :
http://www.spinics.net/lists/linux-nfs/msg56437.html
nano sessiontrunkingpatch/patch1.patch
.
.
.
nano sessiontrunkingpatch/patch6.patch
cd /
#Apply rpc multipath patch
patch -p1 < rpcmultipathpatch/patch1.patch
patch -p1 < rpcmultipathpatch/patch2.patch
patch -p1 < rpcmultipathpatch/patch3.patch
patch -p1 < rpcmultipathpatch/patch4.patch
patch -p1 < rpcmultipathpatch/patch5.patch
patch -p1 < rpcmultipathpatch/patch6.patch
patch -p1 < rpcmultipathpatch/patch7.patch
patch -p1 < rpcmultipathpatch/patch8.patch
patch -p1 < rpcmultipathpatch/patch9.patch
patch -p1 < rpcmultipathpatch/patch10.patch
patch -p1 < rpcmultipathpatch/patch11.patch
patch -p1 < rpcmultipathpatch/patch12.patch
patch -p1 < rpcmultipathpatch/patch13.patch
#Apply client session trunking patches
patch -p1 < sessiontrunkingpatch/patch1.patch
patch -p1 < sessiontrunkingpatch/patch2.patch
patch -p1 < sessiontrunkingpatch/patch3.patch
patch -p1 < sessiontrunkingpatch/patch4.patch
patch -p1 < sessiontrunkingpatch/patch5.patch
patch -p1 < sessiontrunkingpatch/patch6.patch
make -j4
#I got a compilation error in the "net/sunrpc/xprtmultipath.c" line 220
#at WRITE_ONCE(&xpi->xpi_cursor,NULL);
#error: lvalue required as unary ‘&’ operand
#I remove the "&"
make -j4 M=net/sunrpc
make modules_install install
#reboot and boot on the new kernel (with GRUB)
#--------------------Mount--------------------
mkdir testmount
mount -tnfs4 -ominorversion=1 192.168.1.3:/home/testnfs testmount
mount -tnfs4 -ominorversion=1 192.168.2.30:/home/testnfs testmount
#end client configuration
The round-robin implementation distribute correctly the operations
between the two interfaces and the BIND_CONN_TO_SESSION seems to work.
If you cut a cable, the client continue to send nfs packets on it, so
you loose one operation on two.
Thanks to http://kernelnewbies.org/.
Thanks to Trond Myklebust and Andy Adamson for the patches
DQo+IE9uIEZlYiAxNywgMjAxNiwgYXQgNTozMSBBTSwgTWFydGluIEhvdXJ5IDxtYXJ0aW5ob3Vy
eUBnbWFpbC5jb20+IHdyb3RlOg0KPiANCj4gLS0tLS0gT3JpZ2luYWwgTWVzc2FnZSAtLS0tLQ0K
PiANCj4+IEZyb206IGFuZHJvc0B4eHh4eHh4eHh4DQo+PiBUbzogInRyb25kIG15a2xlYnVzdCIg
PHRyb25kLm15a2xlYnVzdEB4eHh4eHh4eHh4eHh4eHg+DQo+PiBDYzogbGludXgtbmZzQHh4eHh4
eHh4eHh4eHh4eCwgIkFuZHkgQWRhbXNvbiIgPGFuZHJvc0B4eHh4eHh4eHh4Pg0KPj4gU2VudDog
RnJpZGF5LCBGZWJydWFyeSA1LCAyMDE2IDQ6MDg6MzIgUE0NCj4+IFN1YmplY3Q6IFtQQVRDSCBS
RkMgVmVyc2lvbiAxIDAvNl0gUmVxdWVzdCBmb3IgQ29tbWVudDogTkZTNC4xIFNlc3Npb24gVHJ1
bmtpbmcNCj4+IA0KPj4gRnJvbTogQW5keSBBZGFtc29uIDxhbmRyb3NAeHh4eHh4eHh4eD4NCj4+
IA0KPj4gVGhpcyBlYXJseSBwYXRjaCBzZXQgaW1wbGVtZW50cyBORlN2NC4xIHNlc3Npb24gdHJ1
bmtpbmcgdXRpbGl6aW5nIFRyb25kcw0KPj4gbXVsdGlwYXRoIGZlYXR1cmUuICBUaGlzIHdvcmtz
IGFnYWluc3QgYW4gdXBzdHJlYW0gbGludXggc2VydmVyIHdpdGgNCj4+IHR3byBOSUNzIGFzIHRo
ZSBFWENIQU5HRV9JRCBpbmZvIGlzIHRoZSBzYW1lIGZvciBib3RoIE5JQ3MuDQo+PiANCj4+IFVu
bW9kaWZpZWQgY29kZSB3aWxsIGNyZWF0ZSBhIG5ldyBuZnNfY2xpZW50IHN0cnVjdHVyZSBmb3Ig
dGhlIG1vdW50IGFzDQo+PiBuZnNfbWF0Y2hfY2xpZW50IGZhaWxzIGR1ZSB0byB0aGUgZGlmZmVy
ZW50IGZ1bGwgc2VydmVyIGFkZHJlc3MuDQo+PiBUdW5raW5nIGlzIGRldGVjdGVkIGJ5IGNvbXBh
cmluZyB0aGUgZWlyX3NlcnZlcl9vd25lci5tYWpvci9taW5vci9zY29wZQ0KPj4gYW5kIHRoZSBl
aXJfc2VydmVyX2ltcGxfaWQgdG8gYW4gZXhpc3RpbmcgbW91bnQuIFdoZW4gdHJ1bmtpbmcgaXMg
ZGV0ZWN0ZWQsDQo+PiB0aGUgMm5kIG1vdW50IHBvaW50IHVzZXMgdGhlIGV4aXN0aW5nICAobm90
IG5ldykgbmZzX2NsaWVudCBmb3IgdGhlIG5ldw0KPj4gbW91bnQgcG9pbnQuDQo+PiANCj4+IFdo
ZW4gdHJ1bmtpbmcgaXMgZGV0ZWN0ZWQsIHRoaXMgcGF0Y2ggc2V0IGFkZHMgYW4gcnBjX3hwcnQg
Zm9yIHRoZSBuZXcgc2VydmVyDQo+PiBhZGRyZXNzIHRvIHRoZSBleGlzaXRpbmcgbW91bnQsIGNh
bGxzIEJJTkRfQ09OTl9UT19TRVNTSU9OLCBhbmQgZGVzdHJveXMNCj4+IHRoZSBuZXcgY2xpZW50
aWQgKG5vIHNlY29uZCBtb3VudCkuDQo+PiANCj4+IFdlIHRoZW4gZ2V0IHJvdW5kIHJvYmluIHVz
ZSBvZiB0aGUgdHdvIE5JQ3Mgb2ZmIG9uZSBtb3VudCBwb2ludCBhbGwNCj4+IHVzaW5nIGEgc2lu
Z2xlIHNlc3Npb24uDQo+PiANCj4+IA0KPj4gSXMgdXNpbmcgbW91bnQgYSBnb29kIHdheSB0byBz
ZXR1cCBORlN2NC4xIHNlc3Npb24gdHJ1bmtpbmc/DQo+PiANCj4+IFRoZSBmaXJzdCB0d28gcGF0
Y2hlcyBmaXggaXNzdWVzIHdpdGggdGhlIG11bHRpcGF0aCBjb2RlLg0KPj4gDQo+PiBTVU5SUEM6
IGFsbG93IHJwY194cHJ0X3N3aXRjaF9hZGRfeHBydCB0byBhZGQgeHBydHMgb24gdGhlIHNhbWUg
bmV0DQo+PiAgICBUaGlzIHBhdGNoIGFkZHMgYSBmbGFnIHRvIGFsbG93IG11bHRpcGF0aGluZyB3
aXRoaW4gdGhlIG9yaWdpYWwgbmV0Lg0KPj4gICAgSSBkb24ndCB1bmRlcnN0YW5kIHdoeSB0aGlz
IHdhcyBub3QgYWxsb3dlZCwgc28gSSBleHBlY3QgYSBkaWZmZXJlbnQNCj4+ICAgIHNvbHV0aW9u
IGZvciB0aGlzIGlzc3VlLg0KPj4gDQo+PiANCj4+IENvbW1lbnRzIHBsZWFzZS4NCj4+IA0KPj4g
VGhhbmtzDQo+PiANCj4+IC0tPkFuZHkNCj4+IA0KPj4gQW5keSBBZGFtc29uICg2KToNCj4+ICBT
VU5SUEMgZml4IHJwY19jbG50X2FkZF94cHJ0IHNldHVwIHJldHVybg0KPj4gIFNVTlJQQyBmaXgg
cnBjX2NsbnRfeHBydF9pdGVyX2luaXQgaW5maW5pdGUgbG9vcA0KPj4gIFNVTlJQQzogYWxsb3cg
cnBjX3hwcnRfc3dpdGNoX2FkZF94cHJ0IHRvIGFkZCB4cHJ0cyBvbiB0aGUgc2FtZSBuZXQNCj4+
ICBORlMgc2Vzc2lvbiB0cnVua2luZyBzdXBwb3J0OiBhZGQgdHJ1bmsgeHBydCB0byBuZnM0LjEg
bW91bnQNCj4+ICBORlMgYWRkIGNhbGxiYWNrX29wcyB0byBuZnM0X3Byb2NfYmluZF9jb25uX3Rv
X3Nlc3Npb25fY2FsbGJhY2sNCj4+ICBORlMgc2NoZWR1bGUgYmluZF9jb25uX3RvX3Nlc3Npb24g
Zm9yIG5ldyBORlN2NC4xIHRyYW5zcG9ydA0KPj4gDQo+PiBmcy9uZnMvY2xpZW50LmMgICAgICAg
ICAgICAgICAgICAgICAgfCAgMiArKw0KPj4gZnMvbmZzL25mczRjbGllbnQuYyAgICAgICAgICAg
ICAgICAgIHwgMjkgKysrKysrKysrKysrKysrKysrKysrKysrKysrLS0NCj4+IGZzL25mcy9uZnM0
cHJvYy5jICAgICAgICAgICAgICAgICAgICB8IDEwICsrKysrKysrKysNCj4+IGluY2x1ZGUvbGlu
dXgvbmZzX2ZzX3NiLmggICAgICAgICAgICB8ICAxICsNCj4+IGluY2x1ZGUvbGludXgvc3VucnBj
L2NsbnQuaCAgICAgICAgICB8ICAxICsNCj4+IGluY2x1ZGUvbGludXgvc3VucnBjL3hwcnRtdWx0
aXBhdGguaCB8ICA2ICsrKysrLQ0KPj4gbmV0L3N1bnJwYy9jbG50LmMgICAgICAgICAgICAgICAg
ICAgIHwgMjAgKysrKysrKysrKysrKy0tLS0tLS0NCj4+IG5ldC9zdW5ycGMveHBydG11bHRpcGF0
aC5jICAgICAgICAgICB8IDM1DQo+PiArKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKyst
LQ0KPj4gOCBmaWxlcyBjaGFuZ2VkLCA5MiBpbnNlcnRpb25zKCspLCAxMiBkZWxldGlvbnMoLSkN
Cj4+IA0KPj4gLS0NCj4+IDEuOC4zLjENCj4+IA0KPj4gLS0NCj4+IFRvIHVuc3Vic2NyaWJlIGZy
b20gdGhpcyBsaXN0OiBzZW5kIHRoZSBsaW5lICJ1bnN1YnNjcmliZSBsaW51eC1uZnMiIGluDQo+
PiB0aGUgYm9keSBvZiBhIG1lc3NhZ2UgdG8gbWFqb3Jkb21vQHh4eHh4eHh4eHh4eHh4eA0KPj4g
TW9yZSBtYWpvcmRvbW8gaW5mbyBhdCAgaHR0cDovL3ZnZXIua2VybmVsLm9yZy9tYWpvcmRvbW8t
aW5mby5odG1sDQo+PiANCj4gDQo+IFRyaWVkIGFuZCBhcHByb3ZlZCENCg0KVGhhbmtzIGZvciB0
ZXN0aW5nLiBBcyBUcm9uZCBwb2ludGVkIG91dCwgdGhlIGNvcnJlY3Qgd2F5IHRvIGluZGljYXRl
IG11bHRpcGxlIGhvc3RuYW1lcyBpcyBvbiB0aGUgbW91bnQgY29tbWFuZCBsaW5lDQoNCm1vdW50
IC1vIG1pbm9ydmVyc2lvbj0xIGhvc3QxLGhvc3QyLOKApixob3N0bjovPGV4cG9ydD4gIC88bW50
ZGlyPg0KDQpJIGhhdmUgYSBuZXcgcGF0Y2ggc2V0IHRoYXQgaXMgd29ya2luZyBhbmQgdGhhdCBJ
IGFtIHRlc3RpbmcgYW5kIGNsZWFuaW5nIHVwIHRoYXQgYWRkcyBtdWx0aXBsZSBob3N0bmFtZSBz
dXBwb3J0IHRvIG1vdW50Lm5mcyBmb3IgTkZTdjQuMSwgYW5kIHRoYXQga2lja3Mgb2ZmIGFuIEVY
Q0hBTkdFX0lEIGZvciBlYWNoIG11bHRpLWhvc3QgYWRkcmVzcy4gSWYgdGhlIEVYQ0hBTkdFX0lE
IHN1Y2NlZWRzLCBhIEJJTkRfQ09OTl9UT19TRVNTSU9OIGlzIGFsc28gY2FsbGVkLg0KDQrigJQ+
QW5keQ0KDQoNCj4gDQo+IEhlcmUgaXMgdGhlIHN0ZXBzIHRvIG1ha2UgaXQgd29ya3MgZm9yIHRo
b3NlIHdobyB3YW50cyB0byB0cnkgaXQuIDoNCj4gDQo+IFRlc3QgY29uZmlndXJhdGlvbiA6DQo+
IA0KPiBSdW5uaW5nIFNlcnZlciBLZXJuZWwgVmVyc2lvbiA6IDMuMi4wDQo+IFJ1bm5pbmcgQ2xp
ZW50IEtlcm5lbCBWZXJzaW9uIDogMy4yLjANCj4gUGF0Y2ggZGVwbG95bWVudCBDbGllbnQgS2Vy
bmVsIFZlcnNpb24gOiA0LjUuMCByYzQrDQo+IERlYmlhbiA3LjkNCj4gR05TMyAxLjQuMQ0KPiBX
aXJlc2hhcmsgMS4xMi4zDQo+IFZpcnR1YWxCb3ggNS4wLjE0DQo+IENsaWVudCBhbmQgU2VydmVy
IGFyZSBWaXJ0dWFsYm94IFZNJ3MgY29uZWN0ZWQgZWFjaCBvdGhlciB3aXRoIGEgR05TMyBTd2l0
Y2gNCj4gDQo+IE5PIEFVVEhFTlRJRklDQVRJT04gLSB0aGlzIGlzIG9ubHkgZm9yIHRlc3QgcHVy
cG9zZQ0KPiANCj4gKy0tLS0tLS0tLS0rICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICstLS0tLS0tLS0tKw0KPiB8ICAgICAgICAgIHwgMTkyLjE2OC4xLjIgLzI0
ICstLS0tLS0tLS0tKyAxOTIuMTY4LjEuMyAvMjQgfCAgICAgICAgICB8DQo+IHwgICAgICAgICAg
Ky0tLS0tLS0tLS0tLS0tLS0tKyAgICAgICAgICArLS0tLS0tLS0tLS0tLS0tLS0rICAgICAgICAg
IHwNCj4gfCAgQ2xpZW50ICB8ICAgICAgICAgICAgICAgICB8ICBTd2l0Y2ggIHwgICAgICAgICAg
ICAgICAgIHwgIFNlcnZlciAgfA0KPiB8ICAgICAgICAgICstLS0tLS0tLS0tLS0tLS0tLSsgICAg
ICAgICAgKy0tLS0tLS0tLS0tLS0tLS0tKyAgICAgICAgICB8DQo+IHwgICAgICAgICAgfCAxOTIu
MTY4LjIuMjAvMjQgKy0tLS0tLS0tLS0rIDE5Mi4xNjguMi4zMCAvMjR8ICAgICAgICAgIHwNCj4g
Ky0tLS0tLS0tLS0rICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICstLS0tLS0tLS0tKw0KPiANCj4gDQo+IFN0ZXBzIDoNCj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiBTZXJ2ZXIg
TkZTNC4xIGNvbmZpZ3VyYXRpb24gOg0KPiANCj4gTWFrZSBzdXJlIHlvdSBoYXZlIGFuIGludGVy
bmV0IGNvbm5lY3Rpb24uDQo+IA0KPiBDb21tYW5kcyBhc3N1bWluZyB5b3UgYXJlIHJvb3Q6DQo+
IA0KPiBhcHQtZ2V0IHVwZGF0ZSAmJiBhcHQtZ2V0IHVwZ3JhZGUNCj4gYXB0LWdldCBpbnN0YWxs
IG5mcy1rZXJuZWwtc2VydmVyDQo+IG1rZGlyIGhvbWUvdGVzdG5mcw0KPiBjaG1vZCA3NzcgaG9t
ZS90ZXN0bmZzDQo+IG5hbm8gZXRjL2V4cG9ydHMNCj4gI0FkZCB0aGVzZSBsaW5lIGluIHRoZSAi
ZXhwb3J0cyIgZmlsZSB0byBzZXQgdGhlICJ0ZXN0bmZzIiBmb2xkZXIgYXZhaWxhYmxlDQo+ICAg
IC9ob21lL3Rlc3RuZnMgICAgMTkyLjE2OC4xLjIocncsc3luYykNCj4gICAgL2hvbWUvdGVzdG5m
cyAgICAxOTIuMTY4LjIuMjAocncsc3luYykNCj4gI2VuZA0KPiANCj4gI0VuYWJsZSBORlN2NC4x
DQo+IGV0Yy9pbml0LmQvbmZzLWtlcm5lbC1zZXJ2ZXIgc3RvcA0KPiBuYW5vIHByb2MvZnMvbmZz
ZC92ZXJzaW9ucw0KPiAjc2V0ICsyICszICs0IC00LjEgdG8gKzIgKzMgKzQgKzQuMQ0KPiBldGMv
aW5pdC5kL25mcy1rZXJuZWwtc2VydmVyIHN0YXJ0DQo+IA0KPiAjc2VydmVyIHJlYWR5DQo+ICNl
bmQgc2VydmVyIGNvbmZpZ3VyYXRpb24NCj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0KPiBDbGllbnQgTkZTNC4xIGNv
bmZpZ3VyYXRpb24gOg0KPiANCj4gQ29tbWFuZHMgYXNzdW1pbmcgeW91IGFyZSByb290Og0KPiAj
LS0tLS0tLS0tLS0tUHJlcGFyZSBuZXcgS2VybmVsIHdpdGggbmV3IHBhdGNoZXMtLS0tLS0tLS0t
DQo+IGFwdC1nZXQgaW5zdGFsbCBsaWJuY3Vyc2VzNS1kZXYgZ2NjIG1ha2UgZ2l0IGV4dWJlcmFu
dC1jdGFncyBiYyBsaWJzc2wtZGV2DQo+IG1rZGlyIGtlcm5lbHMNCj4gY2Qga2VybmVscw0KPiBn
aXQgY2xvbmUgZ2l0Oi8vZ2l0Lmtlcm5lbC5vcmcvcHViL3NjbS9saW51eC9rZXJuZWwvZ2l0L3N0
YWJsZS9saW51eC1zdGFibGUuZ2l0DQo+ICNpdCB0YWtlcyBhIHdoaWxlLi4uDQo+IGNkIGxpbnV4
LXN0YWJsZQ0KPiBjcCAvYm9vdC9jb25maWctYHVuYW1lIC1yYCogLmNvbmZpZw0KPiAjdGhlIC1q
NCBzdGFuZCBmb3IgNCBqb2JzIHJ1bm5pbmcgYXQgdGhlIHNhbWUgdGltZSwgaWRlYWwgZm9yDQo+
IG11bHRpLWNvcmUgcHJvY2Vzc29yLg0KPiBtYWtlIC1qNA0KPiAjSXQgdGFrZXMgYSB3aGlsZQ0K
PiANCj4gIy0tLS0tLS0tLS0tLS0tLS0tLS0tSW5zdGFsbCB0aGUgcGF0Y2hlcy0tLS0tLS0tLS0t
LS0tLS0tLQ0KPiBhcHQtZ2V0IGluc3RhbGwgcGF0Y2gNCj4gY2QgLw0KPiBta2RpciBwYXRjaA0K
PiBjZCBwYXRjaA0KPiBta2RpciBycGNtdWx0aXBhdGhwYXRjaA0KPiBta2RpciBzZXNzaW9udHJ1
bmtpbmdwYXRjaA0KPiAjR2V0IHRoZSBUcm9uZCBNeWtsZWJ1c3QncyBwYXRjaCA6DQo+IGh0dHA6
Ly93d3cuc3Bpbmljcy5uZXQvbGlzdHMvbGludXgtbmZzL21zZzU2MzY1Lmh0bWwNCj4gbmFubyBy
cGNtdWx0aXBhdGhwYXRjaC9wYXRjaDEucGF0Y2gNCj4gLg0KPiAuDQo+IC4NCj4gbmFubyBycGNt
dWx0aXBhdGhwYXRjaC9wYXRjaDEzLnBhdGNoDQo+ICNHZXQgdGhlIEFuZHkgQWRhbXNvbidzIHBh
dGNoZXMgOg0KPiBodHRwOi8vd3d3LnNwaW5pY3MubmV0L2xpc3RzL2xpbnV4LW5mcy9tc2c1NjQz
Ny5odG1sDQo+IG5hbm8gc2Vzc2lvbnRydW5raW5ncGF0Y2gvcGF0Y2gxLnBhdGNoDQo+IC4NCj4g
Lg0KPiAuDQo+IG5hbm8gc2Vzc2lvbnRydW5raW5ncGF0Y2gvcGF0Y2g2LnBhdGNoDQo+IA0KPiAN
Cj4gY2QgLw0KPiANCj4gI0FwcGx5IHJwYyBtdWx0aXBhdGggcGF0Y2gNCj4gcGF0Y2ggLXAxIDwg
cnBjbXVsdGlwYXRocGF0Y2gvcGF0Y2gxLnBhdGNoDQo+IHBhdGNoIC1wMSA8IHJwY211bHRpcGF0
aHBhdGNoL3BhdGNoMi5wYXRjaA0KPiBwYXRjaCAtcDEgPCBycGNtdWx0aXBhdGhwYXRjaC9wYXRj
aDMucGF0Y2gNCj4gcGF0Y2ggLXAxIDwgcnBjbXVsdGlwYXRocGF0Y2gvcGF0Y2g0LnBhdGNoDQo+
IHBhdGNoIC1wMSA8IHJwY211bHRpcGF0aHBhdGNoL3BhdGNoNS5wYXRjaA0KPiBwYXRjaCAtcDEg
PCBycGNtdWx0aXBhdGhwYXRjaC9wYXRjaDYucGF0Y2gNCj4gcGF0Y2ggLXAxIDwgcnBjbXVsdGlw
YXRocGF0Y2gvcGF0Y2g3LnBhdGNoDQo+IHBhdGNoIC1wMSA8IHJwY211bHRpcGF0aHBhdGNoL3Bh
dGNoOC5wYXRjaA0KPiBwYXRjaCAtcDEgPCBycGNtdWx0aXBhdGhwYXRjaC9wYXRjaDkucGF0Y2gN
Cj4gcGF0Y2ggLXAxIDwgcnBjbXVsdGlwYXRocGF0Y2gvcGF0Y2gxMC5wYXRjaA0KPiBwYXRjaCAt
cDEgPCBycGNtdWx0aXBhdGhwYXRjaC9wYXRjaDExLnBhdGNoDQo+IHBhdGNoIC1wMSA8IHJwY211
bHRpcGF0aHBhdGNoL3BhdGNoMTIucGF0Y2gNCj4gcGF0Y2ggLXAxIDwgcnBjbXVsdGlwYXRocGF0
Y2gvcGF0Y2gxMy5wYXRjaA0KPiANCj4gI0FwcGx5IGNsaWVudCBzZXNzaW9uIHRydW5raW5nIHBh
dGNoZXMNCj4gcGF0Y2ggLXAxIDwgc2Vzc2lvbnRydW5raW5ncGF0Y2gvcGF0Y2gxLnBhdGNoDQo+
IHBhdGNoIC1wMSA8IHNlc3Npb250cnVua2luZ3BhdGNoL3BhdGNoMi5wYXRjaA0KPiBwYXRjaCAt
cDEgPCBzZXNzaW9udHJ1bmtpbmdwYXRjaC9wYXRjaDMucGF0Y2gNCj4gcGF0Y2ggLXAxIDwgc2Vz
c2lvbnRydW5raW5ncGF0Y2gvcGF0Y2g0LnBhdGNoDQo+IHBhdGNoIC1wMSA8IHNlc3Npb250cnVu
a2luZ3BhdGNoL3BhdGNoNS5wYXRjaA0KPiBwYXRjaCAtcDEgPCBzZXNzaW9udHJ1bmtpbmdwYXRj
aC9wYXRjaDYucGF0Y2gNCj4gDQo+IG1ha2UgLWo0DQo+IA0KPiAjSSBnb3QgYSBjb21waWxhdGlv
biBlcnJvciBpbiB0aGUgIm5ldC9zdW5ycGMveHBydG11bHRpcGF0aC5jIiBsaW5lIDIyMA0KPiAj
YXQgV1JJVEVfT05DRSgmeHBpLT54cGlfY3Vyc29yLE5VTEwpOw0KPiAjZXJyb3I6IGx2YWx1ZSBy
ZXF1aXJlZCBhcyB1bmFyeSDigJgm4oCZIG9wZXJhbmQNCj4gI0kgcmVtb3ZlIHRoZSAiJiINCj4g
DQo+IG1ha2UgLWo0IE09bmV0L3N1bnJwYw0KPiANCj4gbWFrZSBtb2R1bGVzX2luc3RhbGwgaW5z
dGFsbA0KPiAjcmVib290IGFuZCBib290IG9uIHRoZSBuZXcga2VybmVsICh3aXRoIEdSVUIpDQo+
IA0KPiANCj4gIy0tLS0tLS0tLS0tLS0tLS0tLS0tTW91bnQtLS0tLS0tLS0tLS0tLS0tLS0tLQ0K
PiBta2RpciB0ZXN0bW91bnQNCj4gbW91bnQgLXRuZnM0IC1vbWlub3J2ZXJzaW9uPTEgMTkyLjE2
OC4xLjM6L2hvbWUvdGVzdG5mcyB0ZXN0bW91bnQNCj4gbW91bnQgLXRuZnM0IC1vbWlub3J2ZXJz
aW9uPTEgMTkyLjE2OC4yLjMwOi9ob21lL3Rlc3RuZnMgdGVzdG1vdW50DQo+IA0KPiAjZW5kIGNs
aWVudCBjb25maWd1cmF0aW9uDQo+IA0KPiBUaGUgcm91bmQtcm9iaW4gaW1wbGVtZW50YXRpb24g
ZGlzdHJpYnV0ZSBjb3JyZWN0bHkgdGhlIG9wZXJhdGlvbnMNCj4gYmV0d2VlbiB0aGUgdHdvIGlu
dGVyZmFjZXMgYW5kIHRoZSBCSU5EX0NPTk5fVE9fU0VTU0lPTiBzZWVtcyB0byB3b3JrLg0KPiAN
Cj4gDQo+IElmIHlvdSBjdXQgYSBjYWJsZSwgdGhlIGNsaWVudCBjb250aW51ZSB0byBzZW5kIG5m
cyBwYWNrZXRzIG9uIGl0LCBzbw0KPiB5b3UgbG9vc2Ugb25lIG9wZXJhdGlvbiBvbiB0d28uDQo+
IA0KPiANCj4gDQo+IA0KPiANCj4gVGhhbmtzIHRvIGh0dHA6Ly9rZXJuZWxuZXdiaWVzLm9yZy8u
DQo+IFRoYW5rcyB0byBUcm9uZCBNeWtsZWJ1c3QgYW5kIEFuZHkgQWRhbXNvbiBmb3IgdGhlIHBh
dGNoZXMNCg0K
On Wed, Feb 17, 2016 at 02:50:44PM +0000, Adamson, Andy wrote:
>
> > On Feb 17, 2016, at 5:31 AM, Martin Houry <[email protected]> wrote:
> >
> > ----- Original Message -----
> >
> >> From: andros@xxxxxxxxxx
> >> To: "trond myklebust" <trond.myklebust@xxxxxxxxxxxxxxx>
> >> Cc: linux-nfs@xxxxxxxxxxxxxxx, "Andy Adamson" <andros@xxxxxxxxxx>
> >> Sent: Friday, February 5, 2016 4:08:32 PM
> >> Subject: [PATCH RFC Version 1 0/6] Request for Comment: NFS4.1 Session Trunking
> >>
> >> From: Andy Adamson <andros@xxxxxxxxxx>
> >>
> >> This early patch set implements NFSv4.1 session trunking utilizing Tronds
> >> multipath feature. This works against an upstream linux server with
> >> two NICs as the EXCHANGE_ID info is the same for both NICs.
> >>
> >> Unmodified code will create a new nfs_client structure for the mount as
> >> nfs_match_client fails due to the different full server address.
> >> Tunking is detected by comparing the eir_server_owner.major/minor/scope
> >> and the eir_server_impl_id to an existing mount. When trunking is detected,
> >> the 2nd mount point uses the existing (not new) nfs_client for the new
> >> mount point.
> >>
> >> When trunking is detected, this patch set adds an rpc_xprt for the new server
> >> address to the exisiting mount, calls BIND_CONN_TO_SESSION, and destroys
> >> the new clientid (no second mount).
> >>
> >> We then get round robin use of the two NICs off one mount point all
> >> using a single session.
> >>
> >>
> >> Is using mount a good way to setup NFSv4.1 session trunking?
> >>
> >> The first two patches fix issues with the multipath code.
> >>
> >> SUNRPC: allow rpc_xprt_switch_add_xprt to add xprts on the same net
> >> This patch adds a flag to allow multipathing within the origial net.
> >> I don't understand why this was not allowed, so I expect a different
> >> solution for this issue.
> >>
> >>
> >> Comments please.
> >>
> >> Thanks
> >>
> >> -->Andy
> >>
> >> Andy Adamson (6):
> >> SUNRPC fix rpc_clnt_add_xprt setup return
> >> SUNRPC fix rpc_clnt_xprt_iter_init infinite loop
> >> SUNRPC: allow rpc_xprt_switch_add_xprt to add xprts on the same net
> >> NFS session trunking support: add trunk xprt to nfs4.1 mount
> >> NFS add callback_ops to nfs4_proc_bind_conn_to_session_callback
> >> NFS schedule bind_conn_to_session for new NFSv4.1 transport
> >>
> >> fs/nfs/client.c | 2 ++
> >> fs/nfs/nfs4client.c | 29 +++++++++++++++++++++++++++--
> >> fs/nfs/nfs4proc.c | 10 ++++++++++
> >> include/linux/nfs_fs_sb.h | 1 +
> >> include/linux/sunrpc/clnt.h | 1 +
> >> include/linux/sunrpc/xprtmultipath.h | 6 +++++-
> >> net/sunrpc/clnt.c | 20 +++++++++++++-------
> >> net/sunrpc/xprtmultipath.c | 35
> >> +++++++++++++++++++++++++++++++++--
> >> 8 files changed, 92 insertions(+), 12 deletions(-)
> >>
> >> --
> >> 1.8.3.1
> >>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> >> the body of a message to majordomo@xxxxxxxxxxxxxxx
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>
> >
> > Tried and approved!
>
> Thanks for testing. As Trond pointed out, the correct way to indicate multiple hostnames is on the mount command line
>
> mount -o minorversion=1 host1,host2,…,hostn:/<export> /<mntdir>
>
> I have a new patch set that is working and that I am testing and cleaning up that adds multiple hostname support to mount.nfs for NFSv4.1, and that kicks off an EXCHANGE_ID for each multi-host address. If the EXCHANGE_ID succeeds, a BIND_CONN_TO_SESSION is also called.
I'm surprised there's BIND_CONN_TO_SESSION in the case Martin's
testing. I didn't think it was necessary unless you were using
kerberos. Maybe I'm forgetting some subtlety....
--b.
>
> —>Andy
>
>
> >
> > Here is the steps to make it works for those who wants to try it. :
> >
> > Test configuration :
> >
> > Running Server Kernel Version : 3.2.0
> > Running Client Kernel Version : 3.2.0
> > Patch deployment Client Kernel Version : 4.5.0 rc4+
> > Debian 7.9
> > GNS3 1.4.1
> > Wireshark 1.12.3
> > VirtualBox 5.0.14
> > Client and Server are Virtualbox VM's conected each other with a GNS3 Switch
> >
> > NO AUTHENTIFICATION - this is only for test purpose
> >
> > +----------+ +----------+
> > | | 192.168.1.2 /24 +----------+ 192.168.1.3 /24 | |
> > | +-----------------+ +-----------------+ |
> > | Client | | Switch | | Server |
> > | +-----------------+ +-----------------+ |
> > | | 192.168.2.20/24 +----------+ 192.168.2.30 /24| |
> > +----------+ +----------+
> >
> >
> > Steps :
> > -------------------------------------------------------------------
> > Server NFS4.1 configuration :
> >
> > Make sure you have an internet connection.
> >
> > Commands assuming you are root:
> >
> > apt-get update && apt-get upgrade
> > apt-get install nfs-kernel-server
> > mkdir home/testnfs
> > chmod 777 home/testnfs
> > nano etc/exports
> > #Add these line in the "exports" file to set the "testnfs" folder available
> > /home/testnfs 192.168.1.2(rw,sync)
> > /home/testnfs 192.168.2.20(rw,sync)
> > #end
> >
> > #Enable NFSv4.1
> > etc/init.d/nfs-kernel-server stop
> > nano proc/fs/nfsd/versions
> > #set +2 +3 +4 -4.1 to +2 +3 +4 +4.1
> > etc/init.d/nfs-kernel-server start
> >
> > #server ready
> > #end server configuration
> > -------------------------------------------------------------------
> > Client NFS4.1 configuration :
> >
> > Commands assuming you are root:
> > #------------Prepare new Kernel with new patches----------
> > apt-get install libncurses5-dev gcc make git exuberant-ctags bc libssl-dev
> > mkdir kernels
> > cd kernels
> > git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
> > #it takes a while...
> > cd linux-stable
> > cp /boot/config-`uname -r`* .config
> > #the -j4 stand for 4 jobs running at the same time, ideal for
> > multi-core processor.
> > make -j4
> > #It takes a while
> >
> > #--------------------Install the patches------------------
> > apt-get install patch
> > cd /
> > mkdir patch
> > cd patch
> > mkdir rpcmultipathpatch
> > mkdir sessiontrunkingpatch
> > #Get the Trond Myklebust's patch :
> > http://www.spinics.net/lists/linux-nfs/msg56365.html
> > nano rpcmultipathpatch/patch1.patch
> > .
> > .
> > .
> > nano rpcmultipathpatch/patch13.patch
> > #Get the Andy Adamson's patches :
> > http://www.spinics.net/lists/linux-nfs/msg56437.html
> > nano sessiontrunkingpatch/patch1.patch
> > .
> > .
> > .
> > nano sessiontrunkingpatch/patch6.patch
> >
> >
> > cd /
> >
> > #Apply rpc multipath patch
> > patch -p1 < rpcmultipathpatch/patch1.patch
> > patch -p1 < rpcmultipathpatch/patch2.patch
> > patch -p1 < rpcmultipathpatch/patch3.patch
> > patch -p1 < rpcmultipathpatch/patch4.patch
> > patch -p1 < rpcmultipathpatch/patch5.patch
> > patch -p1 < rpcmultipathpatch/patch6.patch
> > patch -p1 < rpcmultipathpatch/patch7.patch
> > patch -p1 < rpcmultipathpatch/patch8.patch
> > patch -p1 < rpcmultipathpatch/patch9.patch
> > patch -p1 < rpcmultipathpatch/patch10.patch
> > patch -p1 < rpcmultipathpatch/patch11.patch
> > patch -p1 < rpcmultipathpatch/patch12.patch
> > patch -p1 < rpcmultipathpatch/patch13.patch
> >
> > #Apply client session trunking patches
> > patch -p1 < sessiontrunkingpatch/patch1.patch
> > patch -p1 < sessiontrunkingpatch/patch2.patch
> > patch -p1 < sessiontrunkingpatch/patch3.patch
> > patch -p1 < sessiontrunkingpatch/patch4.patch
> > patch -p1 < sessiontrunkingpatch/patch5.patch
> > patch -p1 < sessiontrunkingpatch/patch6.patch
> >
> > make -j4
> >
> > #I got a compilation error in the "net/sunrpc/xprtmultipath.c" line 220
> > #at WRITE_ONCE(&xpi->xpi_cursor,NULL);
> > #error: lvalue required as unary ‘&’ operand
> > #I remove the "&"
> >
> > make -j4 M=net/sunrpc
> >
> > make modules_install install
> > #reboot and boot on the new kernel (with GRUB)
> >
> >
> > #--------------------Mount--------------------
> > mkdir testmount
> > mount -tnfs4 -ominorversion=1 192.168.1.3:/home/testnfs testmount
> > mount -tnfs4 -ominorversion=1 192.168.2.30:/home/testnfs testmount
> >
> > #end client configuration
> >
> > The round-robin implementation distribute correctly the operations
> > between the two interfaces and the BIND_CONN_TO_SESSION seems to work.
> >
> >
> > If you cut a cable, the client continue to send nfs packets on it, so
> > you loose one operation on two.
> >
> >
> >
> >
> >
> > Thanks to http://kernelnewbies.org/.
> > Thanks to Trond Myklebust and Andy Adamson for the patches
>
> N?????r??y????b?X??ǧv?^?){.n?+????{???"??^n?r???z???h?????&?? ?G???h?(?階?ݢj"???m??????z?ޖ???f???h???~?m
DQo+IE9uIEZlYiAxNywgMjAxNiwgYXQgMTE6MzQgQU0sIEouIEJydWNlIEZpZWxkcyA8YmZpZWxk
c0BmaWVsZHNlcy5vcmc+IHdyb3RlOg0KPiANCj4gT24gV2VkLCBGZWIgMTcsIDIwMTYgYXQgMDI6
NTA6NDRQTSArMDAwMCwgQWRhbXNvbiwgQW5keSB3cm90ZToNCj4+IA0KPj4+IE9uIEZlYiAxNywg
MjAxNiwgYXQgNTozMSBBTSwgTWFydGluIEhvdXJ5IDxtYXJ0aW5ob3VyeUBnbWFpbC5jb20+IHdy
b3RlOg0KPj4+IA0KPj4+IC0tLS0tIE9yaWdpbmFsIE1lc3NhZ2UgLS0tLS0NCj4+PiANCj4+Pj4g
RnJvbTogYW5kcm9zQHh4eHh4eHh4eHgNCj4+Pj4gVG86ICJ0cm9uZCBteWtsZWJ1c3QiIDx0cm9u
ZC5teWtsZWJ1c3RAeHh4eHh4eHh4eHh4eHh4Pg0KPj4+PiBDYzogbGludXgtbmZzQHh4eHh4eHh4
eHh4eHh4eCwgIkFuZHkgQWRhbXNvbiIgPGFuZHJvc0B4eHh4eHh4eHh4Pg0KPj4+PiBTZW50OiBG
cmlkYXksIEZlYnJ1YXJ5IDUsIDIwMTYgNDowODozMiBQTQ0KPj4+PiBTdWJqZWN0OiBbUEFUQ0gg
UkZDIFZlcnNpb24gMSAwLzZdIFJlcXVlc3QgZm9yIENvbW1lbnQ6IE5GUzQuMSBTZXNzaW9uIFRy
dW5raW5nDQo+Pj4+IA0KPj4+PiBGcm9tOiBBbmR5IEFkYW1zb24gPGFuZHJvc0B4eHh4eHh4eHh4
Pg0KPj4+PiANCj4+Pj4gVGhpcyBlYXJseSBwYXRjaCBzZXQgaW1wbGVtZW50cyBORlN2NC4xIHNl
c3Npb24gdHJ1bmtpbmcgdXRpbGl6aW5nIFRyb25kcw0KPj4+PiBtdWx0aXBhdGggZmVhdHVyZS4g
IFRoaXMgd29ya3MgYWdhaW5zdCBhbiB1cHN0cmVhbSBsaW51eCBzZXJ2ZXIgd2l0aA0KPj4+PiB0
d28gTklDcyBhcyB0aGUgRVhDSEFOR0VfSUQgaW5mbyBpcyB0aGUgc2FtZSBmb3IgYm90aCBOSUNz
Lg0KPj4+PiANCj4+Pj4gVW5tb2RpZmllZCBjb2RlIHdpbGwgY3JlYXRlIGEgbmV3IG5mc19jbGll
bnQgc3RydWN0dXJlIGZvciB0aGUgbW91bnQgYXMNCj4+Pj4gbmZzX21hdGNoX2NsaWVudCBmYWls
cyBkdWUgdG8gdGhlIGRpZmZlcmVudCBmdWxsIHNlcnZlciBhZGRyZXNzLg0KPj4+PiBUdW5raW5n
IGlzIGRldGVjdGVkIGJ5IGNvbXBhcmluZyB0aGUgZWlyX3NlcnZlcl9vd25lci5tYWpvci9taW5v
ci9zY29wZQ0KPj4+PiBhbmQgdGhlIGVpcl9zZXJ2ZXJfaW1wbF9pZCB0byBhbiBleGlzdGluZyBt
b3VudC4gV2hlbiB0cnVua2luZyBpcyBkZXRlY3RlZCwNCj4+Pj4gdGhlIDJuZCBtb3VudCBwb2lu
dCB1c2VzIHRoZSBleGlzdGluZyAgKG5vdCBuZXcpIG5mc19jbGllbnQgZm9yIHRoZSBuZXcNCj4+
Pj4gbW91bnQgcG9pbnQuDQo+Pj4+IA0KPj4+PiBXaGVuIHRydW5raW5nIGlzIGRldGVjdGVkLCB0
aGlzIHBhdGNoIHNldCBhZGRzIGFuIHJwY194cHJ0IGZvciB0aGUgbmV3IHNlcnZlcg0KPj4+PiBh
ZGRyZXNzIHRvIHRoZSBleGlzaXRpbmcgbW91bnQsIGNhbGxzIEJJTkRfQ09OTl9UT19TRVNTSU9O
LCBhbmQgZGVzdHJveXMNCj4+Pj4gdGhlIG5ldyBjbGllbnRpZCAobm8gc2Vjb25kIG1vdW50KS4N
Cj4+Pj4gDQo+Pj4+IFdlIHRoZW4gZ2V0IHJvdW5kIHJvYmluIHVzZSBvZiB0aGUgdHdvIE5JQ3Mg
b2ZmIG9uZSBtb3VudCBwb2ludCBhbGwNCj4+Pj4gdXNpbmcgYSBzaW5nbGUgc2Vzc2lvbi4NCj4+
Pj4gDQo+Pj4+IA0KPj4+PiBJcyB1c2luZyBtb3VudCBhIGdvb2Qgd2F5IHRvIHNldHVwIE5GU3Y0
LjEgc2Vzc2lvbiB0cnVua2luZz8NCj4+Pj4gDQo+Pj4+IFRoZSBmaXJzdCB0d28gcGF0Y2hlcyBm
aXggaXNzdWVzIHdpdGggdGhlIG11bHRpcGF0aCBjb2RlLg0KPj4+PiANCj4+Pj4gU1VOUlBDOiBh
bGxvdyBycGNfeHBydF9zd2l0Y2hfYWRkX3hwcnQgdG8gYWRkIHhwcnRzIG9uIHRoZSBzYW1lIG5l
dA0KPj4+PiAgIFRoaXMgcGF0Y2ggYWRkcyBhIGZsYWcgdG8gYWxsb3cgbXVsdGlwYXRoaW5nIHdp
dGhpbiB0aGUgb3JpZ2lhbCBuZXQuDQo+Pj4+ICAgSSBkb24ndCB1bmRlcnN0YW5kIHdoeSB0aGlz
IHdhcyBub3QgYWxsb3dlZCwgc28gSSBleHBlY3QgYSBkaWZmZXJlbnQNCj4+Pj4gICBzb2x1dGlv
biBmb3IgdGhpcyBpc3N1ZS4NCj4+Pj4gDQo+Pj4+IA0KPj4+PiBDb21tZW50cyBwbGVhc2UuDQo+
Pj4+IA0KPj4+PiBUaGFua3MNCj4+Pj4gDQo+Pj4+IC0tPkFuZHkNCj4+Pj4gDQo+Pj4+IEFuZHkg
QWRhbXNvbiAoNik6DQo+Pj4+IFNVTlJQQyBmaXggcnBjX2NsbnRfYWRkX3hwcnQgc2V0dXAgcmV0
dXJuDQo+Pj4+IFNVTlJQQyBmaXggcnBjX2NsbnRfeHBydF9pdGVyX2luaXQgaW5maW5pdGUgbG9v
cA0KPj4+PiBTVU5SUEM6IGFsbG93IHJwY194cHJ0X3N3aXRjaF9hZGRfeHBydCB0byBhZGQgeHBy
dHMgb24gdGhlIHNhbWUgbmV0DQo+Pj4+IE5GUyBzZXNzaW9uIHRydW5raW5nIHN1cHBvcnQ6IGFk
ZCB0cnVuayB4cHJ0IHRvIG5mczQuMSBtb3VudA0KPj4+PiBORlMgYWRkIGNhbGxiYWNrX29wcyB0
byBuZnM0X3Byb2NfYmluZF9jb25uX3RvX3Nlc3Npb25fY2FsbGJhY2sNCj4+Pj4gTkZTIHNjaGVk
dWxlIGJpbmRfY29ubl90b19zZXNzaW9uIGZvciBuZXcgTkZTdjQuMSB0cmFuc3BvcnQNCj4+Pj4g
DQo+Pj4+IGZzL25mcy9jbGllbnQuYyAgICAgICAgICAgICAgICAgICAgICB8ICAyICsrDQo+Pj4+
IGZzL25mcy9uZnM0Y2xpZW50LmMgICAgICAgICAgICAgICAgICB8IDI5ICsrKysrKysrKysrKysr
KysrKysrKysrKysrKy0tDQo+Pj4+IGZzL25mcy9uZnM0cHJvYy5jICAgICAgICAgICAgICAgICAg
ICB8IDEwICsrKysrKysrKysNCj4+Pj4gaW5jbHVkZS9saW51eC9uZnNfZnNfc2IuaCAgICAgICAg
ICAgIHwgIDEgKw0KPj4+PiBpbmNsdWRlL2xpbnV4L3N1bnJwYy9jbG50LmggICAgICAgICAgfCAg
MSArDQo+Pj4+IGluY2x1ZGUvbGludXgvc3VucnBjL3hwcnRtdWx0aXBhdGguaCB8ICA2ICsrKysr
LQ0KPj4+PiBuZXQvc3VucnBjL2NsbnQuYyAgICAgICAgICAgICAgICAgICAgfCAyMCArKysrKysr
KysrKysrLS0tLS0tLQ0KPj4+PiBuZXQvc3VucnBjL3hwcnRtdWx0aXBhdGguYyAgICAgICAgICAg
fCAzNQ0KPj4+PiArKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKystLQ0KPj4+PiA4IGZp
bGVzIGNoYW5nZWQsIDkyIGluc2VydGlvbnMoKyksIDEyIGRlbGV0aW9ucygtKQ0KPj4+PiANCj4+
Pj4gLS0NCj4+Pj4gMS44LjMuMQ0KPj4+PiANCj4+Pj4gLS0NCj4+Pj4gVG8gdW5zdWJzY3JpYmUg
ZnJvbSB0aGlzIGxpc3Q6IHNlbmQgdGhlIGxpbmUgInVuc3Vic2NyaWJlIGxpbnV4LW5mcyIgaW4N
Cj4+Pj4gdGhlIGJvZHkgb2YgYSBtZXNzYWdlIHRvIG1ham9yZG9tb0B4eHh4eHh4eHh4eHh4eHgN
Cj4+Pj4gTW9yZSBtYWpvcmRvbW8gaW5mbyBhdCAgaHR0cDovL3ZnZXIua2VybmVsLm9yZy9tYWpv
cmRvbW8taW5mby5odG1sDQo+Pj4+IA0KPj4+IA0KPj4+IFRyaWVkIGFuZCBhcHByb3ZlZCENCj4+
IA0KPj4gVGhhbmtzIGZvciB0ZXN0aW5nLiBBcyBUcm9uZCBwb2ludGVkIG91dCwgdGhlIGNvcnJl
Y3Qgd2F5IHRvIGluZGljYXRlIG11bHRpcGxlIGhvc3RuYW1lcyBpcyBvbiB0aGUgbW91bnQgY29t
bWFuZCBsaW5lDQo+PiANCj4+IG1vdW50IC1vIG1pbm9ydmVyc2lvbj0xIGhvc3QxLGhvc3QyLOKA
pixob3N0bjovPGV4cG9ydD4gIC88bW50ZGlyPg0KPj4gDQo+PiBJIGhhdmUgYSBuZXcgcGF0Y2gg
c2V0IHRoYXQgaXMgd29ya2luZyBhbmQgdGhhdCBJIGFtIHRlc3RpbmcgYW5kIGNsZWFuaW5nIHVw
IHRoYXQgYWRkcyBtdWx0aXBsZSBob3N0bmFtZSBzdXBwb3J0IHRvIG1vdW50Lm5mcyBmb3IgTkZT
djQuMSwgYW5kIHRoYXQga2lja3Mgb2ZmIGFuIEVYQ0hBTkdFX0lEIGZvciBlYWNoIG11bHRpLWhv
c3QgYWRkcmVzcy4gSWYgdGhlIEVYQ0hBTkdFX0lEIHN1Y2NlZWRzLCBhIEJJTkRfQ09OTl9UT19T
RVNTSU9OIGlzIGFsc28gY2FsbGVkLg0KPiANCj4gSSdtIHN1cnByaXNlZCB0aGVyZSdzICBCSU5E
X0NPTk5fVE9fU0VTU0lPTiBpbiB0aGUgY2FzZSBNYXJ0aW4ncw0KPiB0ZXN0aW5nLiAgSSBkaWRu
J3QgdGhpbmsgaXQgd2FzIG5lY2Vzc2FyeSB1bmxlc3MgeW91IHdlcmUgdXNpbmcNCj4ga2VyYmVy
b3MuICBNYXliZSBJJ20gZm9yZ2V0dGluZyBzb21lIHN1YnRsZXR54oCmLg0KDQpUaGUgZmlyc3Qg
aG9zdCAoaG9zdDEgYWJvdmUpIGlzIHRoZSBtb3VudCBob3N0LiBFYWNoIGFkZGl0aW9uYWwgaG9z
dCBpcyBhIG5ldyBjb25uZWN0aW9uLiBBbiBFWENIQU5HRV9JRCBpcyBzZW50IG9uIHRoZSBuZXcg
Y29ubmVjdGlvbiwgYW5kIHRoZSByZXN1bHRhbnQgc2VydmVyc2NvcGUsIHNlcnZlciBvd25lciwg
Y2xpZW50aWQsIGV0YyBmcm9tIHRoZSBuZXcgY29ubmVjdGlvbiBFWENIQU5HRV9JRCByZXBseSBp
cyBjb21wYXJlZCB0byB0aGUgbW91bnQgZXhjaGFuZ2VfaWQgYXJncyBzdG9yZWQgaW4gdGhlIG5m
c19jbGllbnQgc3RydWN0LiBJZiBpdCBwYXNzZXMgdGhlIHNlc3Npb24gdHJ1bmtpbmcgcmVxdWly
ZW1lbnRzLCB0aGVuIHRoZSBjb25uZWN0aW9uIG5lZWRzIHRvIGJlIGJvdW5kIHRvIHRoZSBzZXNz
aW9uIHRvIGVuYWJsZSBzZXNzaW9uIHRydW5raW5nLg0KDQrigJQ+QW5keQ0KDQpSRkM1NjYxIDIu
MTAuNQ0K4oCm4oCmDQogU2Vzc2lvbiBUcnVua2luZy4gIElmIHRoZSBlaWFfY2xpZW50b3duZXIg
YXJndW1lbnQgaXMgdGhlIHNhbWUgaW4gdHdvDQogICAgICBkaWZmZXJlbnQgRVhDSEFOR0VfSUQg
cmVxdWVzdHMsIGFuZCB0aGUgZWlyX2NsaWVudGlkLA0KICAgICAgZWlyX3NlcnZlcl9vd25lci5z
b19tYWpvcl9pZCwgZWlyX3NlcnZlcl9vd25lci5zb19taW5vcl9pZCwgYW5kDQogICAgICBlaXJf
c2VydmVyX3Njb3BlIHJlc3VsdHMgbWF0Y2ggaW4gYm90aCBFWENIQU5HRV9JRCByZXN1bHRzLCB0
aGVuDQogICAgICB0aGUgY2xpZW50IGlzIHBlcm1pdHRlZCB0byBwZXJmb3JtIHNlc3Npb24gdHJ1
bmtpbmcuICBJZiB0aGUNCiAgICAgIGNsaWVudCBoYXMgbm8gc2Vzc2lvbiBtYXBwaW5nIHRvIHRo
ZSB0dXBsZSBvZiBlaXJfY2xpZW50aWQsDQogICAgICBlaXJfc2VydmVyX293bmVyLnNvX21ham9y
X2lkLCBlaXJfc2VydmVyX3Njb3BlLCBhbmQNCiAgICAgIGVpcl9zZXJ2ZXJfb3duZXIuc29fbWlu
b3JfaWQsIHRoZW4gaXQgY3JlYXRlcyB0aGUgc2Vzc2lvbiB2aWEgYQ0KICAgICAgQ1JFQVRFX1NF
U1NJT04gb3BlcmF0aW9uIG92ZXIgb25lIG9mIHRoZSBjb25uZWN0aW9ucywgd2hpY2gNCiAgICAg
IGFzc29jaWF0ZXMgdGhlIGNvbm5lY3Rpb24gdG8gdGhlIHNlc3Npb24uICBJZiB0aGVyZSBpcyBh
IHNlc3Npb24NCiAgICAgIGZvciB0aGUgdHVwbGUsIHRoZSBjbGllbnQgY2FuIHNlbmQgQklORF9D
T05OX1RPX1NFU1NJT04gdG8NCiAgICAgIGFzc29jaWF0ZSB0aGUgY29ubmVjdGlvbiB0byB0aGUg
c2Vzc2lvbi4NCg0KICAgICAgT2YgY291cnNlLCBpZiB0aGUgY2xpZW50IGRvZXMgbm90IGRlc2ly
ZSB0byB1c2Ugc2Vzc2lvbiB0cnVua2luZywNCiAgICAgIGl0IGlzIG5vdCByZXF1aXJlZCB0byBk
byBzby4gIEl0IGNhbiBpbnZva2UgQ1JFQVRFX1NFU1NJT04gb24gdGhlDQogICAgICBjb25uZWN0
aW9uLiAgVGhpcyB3aWxsIHJlc3VsdCBpbiBjbGllbnQgSUQgdHJ1bmtpbmcgYXMgZGVzY3JpYmVk
DQogICAgICBiZWxvdy4gIEl0IGNhbiBhbHNvIGRlY2lkZSB0byBkcm9wIHRoZSBjb25uZWN0aW9u
IGlmIGl0IGRvZXMgbm90DQogICAgICBjaG9vc2UgdG8gdXNlIHRydW5raW5nLg0KPiANCj4gLS1i
Lg0KPiANCj4+IA0KPj4g4oCUPkFuZHkNCj4+IA0KPj4gDQo+Pj4gDQo+Pj4gSGVyZSBpcyB0aGUg
c3RlcHMgdG8gbWFrZSBpdCB3b3JrcyBmb3IgdGhvc2Ugd2hvIHdhbnRzIHRvIHRyeSBpdC4gOg0K
Pj4+IA0KPj4+IFRlc3QgY29uZmlndXJhdGlvbiA6DQo+Pj4gDQo+Pj4gUnVubmluZyBTZXJ2ZXIg
S2VybmVsIFZlcnNpb24gOiAzLjIuMA0KPj4+IFJ1bm5pbmcgQ2xpZW50IEtlcm5lbCBWZXJzaW9u
IDogMy4yLjANCj4+PiBQYXRjaCBkZXBsb3ltZW50IENsaWVudCBLZXJuZWwgVmVyc2lvbiA6IDQu
NS4wIHJjNCsNCj4+PiBEZWJpYW4gNy45DQo+Pj4gR05TMyAxLjQuMQ0KPj4+IFdpcmVzaGFyayAx
LjEyLjMNCj4+PiBWaXJ0dWFsQm94IDUuMC4xNA0KPj4+IENsaWVudCBhbmQgU2VydmVyIGFyZSBW
aXJ0dWFsYm94IFZNJ3MgY29uZWN0ZWQgZWFjaCBvdGhlciB3aXRoIGEgR05TMyBTd2l0Y2gNCj4+
PiANCj4+PiBOTyBBVVRIRU5USUZJQ0FUSU9OIC0gdGhpcyBpcyBvbmx5IGZvciB0ZXN0IHB1cnBv
c2UNCj4+PiANCj4+PiArLS0tLS0tLS0tLSsgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgKy0tLS0tLS0tLS0rDQo+Pj4gfCAgICAgICAgICB8IDE5Mi4xNjguMS4y
IC8yNCArLS0tLS0tLS0tLSsgMTkyLjE2OC4xLjMgLzI0IHwgICAgICAgICAgfA0KPj4+IHwgICAg
ICAgICAgKy0tLS0tLS0tLS0tLS0tLS0tKyAgICAgICAgICArLS0tLS0tLS0tLS0tLS0tLS0rICAg
ICAgICAgIHwNCj4+PiB8ICBDbGllbnQgIHwgICAgICAgICAgICAgICAgIHwgIFN3aXRjaCAgfCAg
ICAgICAgICAgICAgICAgfCAgU2VydmVyICB8DQo+Pj4gfCAgICAgICAgICArLS0tLS0tLS0tLS0t
LS0tLS0rICAgICAgICAgICstLS0tLS0tLS0tLS0tLS0tLSsgICAgICAgICAgfA0KPj4+IHwgICAg
ICAgICAgfCAxOTIuMTY4LjIuMjAvMjQgKy0tLS0tLS0tLS0rIDE5Mi4xNjguMi4zMCAvMjR8ICAg
ICAgICAgIHwNCj4+PiArLS0tLS0tLS0tLSsgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgKy0tLS0tLS0tLS0rDQo+Pj4gDQo+Pj4gDQo+Pj4gU3RlcHMgOg0KPj4+
IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0NCj4+PiBTZXJ2ZXIgTkZTNC4xIGNvbmZpZ3VyYXRpb24gOg0KPj4+IA0KPj4+
IE1ha2Ugc3VyZSB5b3UgaGF2ZSBhbiBpbnRlcm5ldCBjb25uZWN0aW9uLg0KPj4+IA0KPj4+IENv
bW1hbmRzIGFzc3VtaW5nIHlvdSBhcmUgcm9vdDoNCj4+PiANCj4+PiBhcHQtZ2V0IHVwZGF0ZSAm
JiBhcHQtZ2V0IHVwZ3JhZGUNCj4+PiBhcHQtZ2V0IGluc3RhbGwgbmZzLWtlcm5lbC1zZXJ2ZXIN
Cj4+PiBta2RpciBob21lL3Rlc3RuZnMNCj4+PiBjaG1vZCA3NzcgaG9tZS90ZXN0bmZzDQo+Pj4g
bmFubyBldGMvZXhwb3J0cw0KPj4+ICNBZGQgdGhlc2UgbGluZSBpbiB0aGUgImV4cG9ydHMiIGZp
bGUgdG8gc2V0IHRoZSAidGVzdG5mcyIgZm9sZGVyIGF2YWlsYWJsZQ0KPj4+ICAgL2hvbWUvdGVz
dG5mcyAgICAxOTIuMTY4LjEuMihydyxzeW5jKQ0KPj4+ICAgL2hvbWUvdGVzdG5mcyAgICAxOTIu
MTY4LjIuMjAocncsc3luYykNCj4+PiAjZW5kDQo+Pj4gDQo+Pj4gI0VuYWJsZSBORlN2NC4xDQo+
Pj4gZXRjL2luaXQuZC9uZnMta2VybmVsLXNlcnZlciBzdG9wDQo+Pj4gbmFubyBwcm9jL2ZzL25m
c2QvdmVyc2lvbnMNCj4+PiAjc2V0ICsyICszICs0IC00LjEgdG8gKzIgKzMgKzQgKzQuMQ0KPj4+
IGV0Yy9pbml0LmQvbmZzLWtlcm5lbC1zZXJ2ZXIgc3RhcnQNCj4+PiANCj4+PiAjc2VydmVyIHJl
YWR5DQo+Pj4gI2VuZCBzZXJ2ZXIgY29uZmlndXJhdGlvbg0KPj4+IC0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4+PiBD
bGllbnQgTkZTNC4xIGNvbmZpZ3VyYXRpb24gOg0KPj4+IA0KPj4+IENvbW1hbmRzIGFzc3VtaW5n
IHlvdSBhcmUgcm9vdDoNCj4+PiAjLS0tLS0tLS0tLS0tUHJlcGFyZSBuZXcgS2VybmVsIHdpdGgg
bmV3IHBhdGNoZXMtLS0tLS0tLS0tDQo+Pj4gYXB0LWdldCBpbnN0YWxsIGxpYm5jdXJzZXM1LWRl
diBnY2MgbWFrZSBnaXQgZXh1YmVyYW50LWN0YWdzIGJjIGxpYnNzbC1kZXYNCj4+PiBta2RpciBr
ZXJuZWxzDQo+Pj4gY2Qga2VybmVscw0KPj4+IGdpdCBjbG9uZSBnaXQ6Ly9naXQua2VybmVsLm9y
Zy9wdWIvc2NtL2xpbnV4L2tlcm5lbC9naXQvc3RhYmxlL2xpbnV4LXN0YWJsZS5naXQNCj4+PiAj
aXQgdGFrZXMgYSB3aGlsZS4uLg0KPj4+IGNkIGxpbnV4LXN0YWJsZQ0KPj4+IGNwIC9ib290L2Nv
bmZpZy1gdW5hbWUgLXJgKiAuY29uZmlnDQo+Pj4gI3RoZSAtajQgc3RhbmQgZm9yIDQgam9icyBy
dW5uaW5nIGF0IHRoZSBzYW1lIHRpbWUsIGlkZWFsIGZvcg0KPj4+IG11bHRpLWNvcmUgcHJvY2Vz
c29yLg0KPj4+IG1ha2UgLWo0DQo+Pj4gI0l0IHRha2VzIGEgd2hpbGUNCj4+PiANCj4+PiAjLS0t
LS0tLS0tLS0tLS0tLS0tLS1JbnN0YWxsIHRoZSBwYXRjaGVzLS0tLS0tLS0tLS0tLS0tLS0tDQo+
Pj4gYXB0LWdldCBpbnN0YWxsIHBhdGNoDQo+Pj4gY2QgLw0KPj4+IG1rZGlyIHBhdGNoDQo+Pj4g
Y2QgcGF0Y2gNCj4+PiBta2RpciBycGNtdWx0aXBhdGhwYXRjaA0KPj4+IG1rZGlyIHNlc3Npb250
cnVua2luZ3BhdGNoDQo+Pj4gI0dldCB0aGUgVHJvbmQgTXlrbGVidXN0J3MgcGF0Y2ggOg0KPj4+
IGh0dHA6Ly93d3cuc3Bpbmljcy5uZXQvbGlzdHMvbGludXgtbmZzL21zZzU2MzY1Lmh0bWwNCj4+
PiBuYW5vIHJwY211bHRpcGF0aHBhdGNoL3BhdGNoMS5wYXRjaA0KPj4+IC4NCj4+PiAuDQo+Pj4g
Lg0KPj4+IG5hbm8gcnBjbXVsdGlwYXRocGF0Y2gvcGF0Y2gxMy5wYXRjaA0KPj4+ICNHZXQgdGhl
IEFuZHkgQWRhbXNvbidzIHBhdGNoZXMgOg0KPj4+IGh0dHA6Ly93d3cuc3Bpbmljcy5uZXQvbGlz
dHMvbGludXgtbmZzL21zZzU2NDM3Lmh0bWwNCj4+PiBuYW5vIHNlc3Npb250cnVua2luZ3BhdGNo
L3BhdGNoMS5wYXRjaA0KPj4+IC4NCj4+PiAuDQo+Pj4gLg0KPj4+IG5hbm8gc2Vzc2lvbnRydW5r
aW5ncGF0Y2gvcGF0Y2g2LnBhdGNoDQo+Pj4gDQo+Pj4gDQo+Pj4gY2QgLw0KPj4+IA0KPj4+ICNB
cHBseSBycGMgbXVsdGlwYXRoIHBhdGNoDQo+Pj4gcGF0Y2ggLXAxIDwgcnBjbXVsdGlwYXRocGF0
Y2gvcGF0Y2gxLnBhdGNoDQo+Pj4gcGF0Y2ggLXAxIDwgcnBjbXVsdGlwYXRocGF0Y2gvcGF0Y2gy
LnBhdGNoDQo+Pj4gcGF0Y2ggLXAxIDwgcnBjbXVsdGlwYXRocGF0Y2gvcGF0Y2gzLnBhdGNoDQo+
Pj4gcGF0Y2ggLXAxIDwgcnBjbXVsdGlwYXRocGF0Y2gvcGF0Y2g0LnBhdGNoDQo+Pj4gcGF0Y2gg
LXAxIDwgcnBjbXVsdGlwYXRocGF0Y2gvcGF0Y2g1LnBhdGNoDQo+Pj4gcGF0Y2ggLXAxIDwgcnBj
bXVsdGlwYXRocGF0Y2gvcGF0Y2g2LnBhdGNoDQo+Pj4gcGF0Y2ggLXAxIDwgcnBjbXVsdGlwYXRo
cGF0Y2gvcGF0Y2g3LnBhdGNoDQo+Pj4gcGF0Y2ggLXAxIDwgcnBjbXVsdGlwYXRocGF0Y2gvcGF0
Y2g4LnBhdGNoDQo+Pj4gcGF0Y2ggLXAxIDwgcnBjbXVsdGlwYXRocGF0Y2gvcGF0Y2g5LnBhdGNo
DQo+Pj4gcGF0Y2ggLXAxIDwgcnBjbXVsdGlwYXRocGF0Y2gvcGF0Y2gxMC5wYXRjaA0KPj4+IHBh
dGNoIC1wMSA8IHJwY211bHRpcGF0aHBhdGNoL3BhdGNoMTEucGF0Y2gNCj4+PiBwYXRjaCAtcDEg
PCBycGNtdWx0aXBhdGhwYXRjaC9wYXRjaDEyLnBhdGNoDQo+Pj4gcGF0Y2ggLXAxIDwgcnBjbXVs
dGlwYXRocGF0Y2gvcGF0Y2gxMy5wYXRjaA0KPj4+IA0KPj4+ICNBcHBseSBjbGllbnQgc2Vzc2lv
biB0cnVua2luZyBwYXRjaGVzDQo+Pj4gcGF0Y2ggLXAxIDwgc2Vzc2lvbnRydW5raW5ncGF0Y2gv
cGF0Y2gxLnBhdGNoDQo+Pj4gcGF0Y2ggLXAxIDwgc2Vzc2lvbnRydW5raW5ncGF0Y2gvcGF0Y2gy
LnBhdGNoDQo+Pj4gcGF0Y2ggLXAxIDwgc2Vzc2lvbnRydW5raW5ncGF0Y2gvcGF0Y2gzLnBhdGNo
DQo+Pj4gcGF0Y2ggLXAxIDwgc2Vzc2lvbnRydW5raW5ncGF0Y2gvcGF0Y2g0LnBhdGNoDQo+Pj4g
cGF0Y2ggLXAxIDwgc2Vzc2lvbnRydW5raW5ncGF0Y2gvcGF0Y2g1LnBhdGNoDQo+Pj4gcGF0Y2gg
LXAxIDwgc2Vzc2lvbnRydW5raW5ncGF0Y2gvcGF0Y2g2LnBhdGNoDQo+Pj4gDQo+Pj4gbWFrZSAt
ajQNCj4+PiANCj4+PiAjSSBnb3QgYSBjb21waWxhdGlvbiBlcnJvciBpbiB0aGUgIm5ldC9zdW5y
cGMveHBydG11bHRpcGF0aC5jIiBsaW5lIDIyMA0KPj4+ICNhdCBXUklURV9PTkNFKCZ4cGktPnhw
aV9jdXJzb3IsTlVMTCk7DQo+Pj4gI2Vycm9yOiBsdmFsdWUgcmVxdWlyZWQgYXMgdW5hcnkg4oCY
JuKAmSBvcGVyYW5kDQo+Pj4gI0kgcmVtb3ZlIHRoZSAiJiINCj4+PiANCj4+PiBtYWtlIC1qNCBN
PW5ldC9zdW5ycGMNCj4+PiANCj4+PiBtYWtlIG1vZHVsZXNfaW5zdGFsbCBpbnN0YWxsDQo+Pj4g
I3JlYm9vdCBhbmQgYm9vdCBvbiB0aGUgbmV3IGtlcm5lbCAod2l0aCBHUlVCKQ0KPj4+IA0KPj4+
IA0KPj4+ICMtLS0tLS0tLS0tLS0tLS0tLS0tLU1vdW50LS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4+
PiBta2RpciB0ZXN0bW91bnQNCj4+PiBtb3VudCAtdG5mczQgLW9taW5vcnZlcnNpb249MSAxOTIu
MTY4LjEuMzovaG9tZS90ZXN0bmZzIHRlc3Rtb3VudA0KPj4+IG1vdW50IC10bmZzNCAtb21pbm9y
dmVyc2lvbj0xIDE5Mi4xNjguMi4zMDovaG9tZS90ZXN0bmZzIHRlc3Rtb3VudA0KPj4+IA0KPj4+
ICNlbmQgY2xpZW50IGNvbmZpZ3VyYXRpb24NCj4+PiANCj4+PiBUaGUgcm91bmQtcm9iaW4gaW1w
bGVtZW50YXRpb24gZGlzdHJpYnV0ZSBjb3JyZWN0bHkgdGhlIG9wZXJhdGlvbnMNCj4+PiBiZXR3
ZWVuIHRoZSB0d28gaW50ZXJmYWNlcyBhbmQgdGhlIEJJTkRfQ09OTl9UT19TRVNTSU9OIHNlZW1z
IHRvIHdvcmsuDQo+Pj4gDQo+Pj4gDQo+Pj4gSWYgeW91IGN1dCBhIGNhYmxlLCB0aGUgY2xpZW50
IGNvbnRpbnVlIHRvIHNlbmQgbmZzIHBhY2tldHMgb24gaXQsIHNvDQo+Pj4geW91IGxvb3NlIG9u
ZSBvcGVyYXRpb24gb24gdHdvLg0KPj4+IA0KPj4+IA0KPj4+IA0KPj4+IA0KPj4+IA0KPj4+IFRo
YW5rcyB0byBodHRwOi8va2VybmVsbmV3Ymllcy5vcmcvLg0KPj4+IFRoYW5rcyB0byBUcm9uZCBN
eWtsZWJ1c3QgYW5kIEFuZHkgQWRhbXNvbiBmb3IgdGhlIHBhdGNoZXMNCj4+IA0KPj4gTj8/Pz8/
cj8/eT8/Pz9iP1g/P8endj9ePyneunsubj8rPz8/P3s/Pz8iPz9ebj9yPz8/ej8aPz9oPz8/Pz8m
Pz8eP0c/Pz9oPwMoP+majj/domoiPz8aPxttPz8/Pz8/ej/elj8/P2Y/Pz9oPz8/fj9tDQoNCg==
On Wed, Feb 17, 2016 at 05:49:39PM +0000, Adamson, Andy wrote:
> On Feb 17, 2016, at 11:34 AM, J. Bruce Fields <[email protected]> wrote:
> > I'm surprised there's BIND_CONN_TO_SESSION in the case Martin's
> > testing. I didn't think it was necessary unless you were using
> > kerberos. Maybe I'm forgetting some subtlety….
>
> The first host (host1 above) is the mount host. Each additional host is a new connection. An EXCHANGE_ID is sent on the new connection, and the resultant serverscope, server owner, clientid, etc from the new connection EXCHANGE_ID reply is compared to the mount exchange_id args stored in the nfs_client struct. If it passes the session trunking requirements, then the connection needs to be bound to the session to enable session trunking.
That all makes sense to me, but:
"If, when the client ID was created, the client opted for
SP4_NONE state protection, the client is not required to use
BIND_CONN_TO_SESSION to associate the connection with the
session, unless the client wishes to associate the connection
with the backchannel. When SP4_NONE protection is used, simply
sending a COMPOUND request with a SEQUENCE operation is
sufficient to associate the connection with the session
specified in SEQUENCE."
Well, but maybe that's academic, there's probably no harm in sending the
BIND_CONN_TO_SESSION.
--b.
>
> —>Andy
>
> RFC5661 2.10.5
> ……
> Session Trunking. If the eia_clientowner argument is the same in two
> different EXCHANGE_ID requests, and the eir_clientid,
> eir_server_owner.so_major_id, eir_server_owner.so_minor_id, and
> eir_server_scope results match in both EXCHANGE_ID results, then
> the client is permitted to perform session trunking. If the
> client has no session mapping to the tuple of eir_clientid,
> eir_server_owner.so_major_id, eir_server_scope, and
> eir_server_owner.so_minor_id, then it creates the session via a
> CREATE_SESSION operation over one of the connections, which
> associates the connection to the session. If there is a session
> for the tuple, the client can send BIND_CONN_TO_SESSION to
> associate the connection to the session.
>
> Of course, if the client does not desire to use session trunking,
> it is not required to do so. It can invoke CREATE_SESSION on the
> connection. This will result in client ID trunking as described
> below. It can also decide to drop the connection if it does not
> choose to use trunking.
> >
> > --b.
> >
> >>
> >> —>Andy
> >>
> >>
> >>>
> >>> Here is the steps to make it works for those who wants to try it. :
> >>>
> >>> Test configuration :
> >>>
> >>> Running Server Kernel Version : 3.2.0
> >>> Running Client Kernel Version : 3.2.0
> >>> Patch deployment Client Kernel Version : 4.5.0 rc4+
> >>> Debian 7.9
> >>> GNS3 1.4.1
> >>> Wireshark 1.12.3
> >>> VirtualBox 5.0.14
> >>> Client and Server are Virtualbox VM's conected each other with a GNS3 Switch
> >>>
> >>> NO AUTHENTIFICATION - this is only for test purpose
> >>>
> >>> +----------+ +----------+
> >>> | | 192.168.1.2 /24 +----------+ 192.168.1.3 /24 | |
> >>> | +-----------------+ +-----------------+ |
> >>> | Client | | Switch | | Server |
> >>> | +-----------------+ +-----------------+ |
> >>> | | 192.168.2.20/24 +----------+ 192.168.2.30 /24| |
> >>> +----------+ +----------+
> >>>
> >>>
> >>> Steps :
> >>> -------------------------------------------------------------------
> >>> Server NFS4.1 configuration :
> >>>
> >>> Make sure you have an internet connection.
> >>>
> >>> Commands assuming you are root:
> >>>
> >>> apt-get update && apt-get upgrade
> >>> apt-get install nfs-kernel-server
> >>> mkdir home/testnfs
> >>> chmod 777 home/testnfs
> >>> nano etc/exports
> >>> #Add these line in the "exports" file to set the "testnfs" folder available
> >>> /home/testnfs 192.168.1.2(rw,sync)
> >>> /home/testnfs 192.168.2.20(rw,sync)
> >>> #end
> >>>
> >>> #Enable NFSv4.1
> >>> etc/init.d/nfs-kernel-server stop
> >>> nano proc/fs/nfsd/versions
> >>> #set +2 +3 +4 -4.1 to +2 +3 +4 +4.1
> >>> etc/init.d/nfs-kernel-server start
> >>>
> >>> #server ready
> >>> #end server configuration
> >>> -------------------------------------------------------------------
> >>> Client NFS4.1 configuration :
> >>>
> >>> Commands assuming you are root:
> >>> #------------Prepare new Kernel with new patches----------
> >>> apt-get install libncurses5-dev gcc make git exuberant-ctags bc libssl-dev
> >>> mkdir kernels
> >>> cd kernels
> >>> git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
> >>> #it takes a while...
> >>> cd linux-stable
> >>> cp /boot/config-`uname -r`* .config
> >>> #the -j4 stand for 4 jobs running at the same time, ideal for
> >>> multi-core processor.
> >>> make -j4
> >>> #It takes a while
> >>>
> >>> #--------------------Install the patches------------------
> >>> apt-get install patch
> >>> cd /
> >>> mkdir patch
> >>> cd patch
> >>> mkdir rpcmultipathpatch
> >>> mkdir sessiontrunkingpatch
> >>> #Get the Trond Myklebust's patch :
> >>> http://www.spinics.net/lists/linux-nfs/msg56365.html
> >>> nano rpcmultipathpatch/patch1.patch
> >>> .
> >>> .
> >>> .
> >>> nano rpcmultipathpatch/patch13.patch
> >>> #Get the Andy Adamson's patches :
> >>> http://www.spinics.net/lists/linux-nfs/msg56437.html
> >>> nano sessiontrunkingpatch/patch1.patch
> >>> .
> >>> .
> >>> .
> >>> nano sessiontrunkingpatch/patch6.patch
> >>>
> >>>
> >>> cd /
> >>>
> >>> #Apply rpc multipath patch
> >>> patch -p1 < rpcmultipathpatch/patch1.patch
> >>> patch -p1 < rpcmultipathpatch/patch2.patch
> >>> patch -p1 < rpcmultipathpatch/patch3.patch
> >>> patch -p1 < rpcmultipathpatch/patch4.patch
> >>> patch -p1 < rpcmultipathpatch/patch5.patch
> >>> patch -p1 < rpcmultipathpatch/patch6.patch
> >>> patch -p1 < rpcmultipathpatch/patch7.patch
> >>> patch -p1 < rpcmultipathpatch/patch8.patch
> >>> patch -p1 < rpcmultipathpatch/patch9.patch
> >>> patch -p1 < rpcmultipathpatch/patch10.patch
> >>> patch -p1 < rpcmultipathpatch/patch11.patch
> >>> patch -p1 < rpcmultipathpatch/patch12.patch
> >>> patch -p1 < rpcmultipathpatch/patch13.patch
> >>>
> >>> #Apply client session trunking patches
> >>> patch -p1 < sessiontrunkingpatch/patch1.patch
> >>> patch -p1 < sessiontrunkingpatch/patch2.patch
> >>> patch -p1 < sessiontrunkingpatch/patch3.patch
> >>> patch -p1 < sessiontrunkingpatch/patch4.patch
> >>> patch -p1 < sessiontrunkingpatch/patch5.patch
> >>> patch -p1 < sessiontrunkingpatch/patch6.patch
> >>>
> >>> make -j4
> >>>
> >>> #I got a compilation error in the "net/sunrpc/xprtmultipath.c" line 220
> >>> #at WRITE_ONCE(&xpi->xpi_cursor,NULL);
> >>> #error: lvalue required as unary ‘&’ operand
> >>> #I remove the "&"
> >>>
> >>> make -j4 M=net/sunrpc
> >>>
> >>> make modules_install install
> >>> #reboot and boot on the new kernel (with GRUB)
> >>>
> >>>
> >>> #--------------------Mount--------------------
> >>> mkdir testmount
> >>> mount -tnfs4 -ominorversion=1 192.168.1.3:/home/testnfs testmount
> >>> mount -tnfs4 -ominorversion=1 192.168.2.30:/home/testnfs testmount
> >>>
> >>> #end client configuration
> >>>
> >>> The round-robin implementation distribute correctly the operations
> >>> between the two interfaces and the BIND_CONN_TO_SESSION seems to work.
> >>>
> >>>
> >>> If you cut a cable, the client continue to send nfs packets on it, so
> >>> you loose one operation on two.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Thanks to http://kernelnewbies.org/.
> >>> Thanks to Trond Myklebust and Andy Adamson for the patches
> >>
> >> N?????r??y????b?X??ǧv?^?){.n?+????{???"??^n?r???z???h?????&?? ?G???h?(?階?ݢj"???m??????z?ޖ???f???h???~?m
>
> On Feb 17, 2016, at 9:50 AM, Adamson, Andy <[email protected]> wrote:
>
>
>> On Feb 17, 2016, at 5:31 AM, Martin Houry <[email protected]> wrote:
>>
>> ----- Original Message -----
>>
>>> From: andros@xxxxxxxxxx
>>> To: "trond myklebust" <trond.myklebust@xxxxxxxxxxxxxxx>
>>> Cc: linux-nfs@xxxxxxxxxxxxxxx, "Andy Adamson" <andros@xxxxxxxxxx>
>>> Sent: Friday, February 5, 2016 4:08:32 PM
>>> Subject: [PATCH RFC Version 1 0/6] Request for Comment: NFS4.1 Session Trunking
>>>
>>> From: Andy Adamson <andros@xxxxxxxxxx>
>>>
>>> This early patch set implements NFSv4.1 session trunking utilizing Tronds
>>> multipath feature. This works against an upstream linux server with
>>> two NICs as the EXCHANGE_ID info is the same for both NICs.
>>>
>>> Unmodified code will create a new nfs_client structure for the mount as
>>> nfs_match_client fails due to the different full server address.
>>> Tunking is detected by comparing the eir_server_owner.major/minor/scope
>>> and the eir_server_impl_id to an existing mount. When trunking is detected,
>>> the 2nd mount point uses the existing (not new) nfs_client for the new
>>> mount point.
>>>
>>> When trunking is detected, this patch set adds an rpc_xprt for the new server
>>> address to the exisiting mount, calls BIND_CONN_TO_SESSION, and destroys
>>> the new clientid (no second mount).
>>>
>>> We then get round robin use of the two NICs off one mount point all
>>> using a single session.
>>>
>>>
>>> Is using mount a good way to setup NFSv4.1 session trunking?
>>>
>>> The first two patches fix issues with the multipath code.
>>>
>>> SUNRPC: allow rpc_xprt_switch_add_xprt to add xprts on the same net
>>> This patch adds a flag to allow multipathing within the origial net.
>>> I don't understand why this was not allowed, so I expect a different
>>> solution for this issue.
>>>
>>>
>>> Comments please.
>>>
>>> Thanks
>>>
>>> -->Andy
>>>
>>> Andy Adamson (6):
>>> SUNRPC fix rpc_clnt_add_xprt setup return
>>> SUNRPC fix rpc_clnt_xprt_iter_init infinite loop
>>> SUNRPC: allow rpc_xprt_switch_add_xprt to add xprts on the same net
>>> NFS session trunking support: add trunk xprt to nfs4.1 mount
>>> NFS add callback_ops to nfs4_proc_bind_conn_to_session_callback
>>> NFS schedule bind_conn_to_session for new NFSv4.1 transport
>>>
>>> fs/nfs/client.c | 2 ++
>>> fs/nfs/nfs4client.c | 29 +++++++++++++++++++++++++++--
>>> fs/nfs/nfs4proc.c | 10 ++++++++++
>>> include/linux/nfs_fs_sb.h | 1 +
>>> include/linux/sunrpc/clnt.h | 1 +
>>> include/linux/sunrpc/xprtmultipath.h | 6 +++++-
>>> net/sunrpc/clnt.c | 20 +++++++++++++-------
>>> net/sunrpc/xprtmultipath.c | 35
>>> +++++++++++++++++++++++++++++++++--
>>> 8 files changed, 92 insertions(+), 12 deletions(-)
>>>
>>> --
>>> 1.8.3.1
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>> Tried and approved!
>
> Thanks for testing. As Trond pointed out, the correct way to indicate multiple hostnames is on the mount command line
>
> mount -o minorversion=1 host1,host2,…,hostn:/<export> /<mntdir>
It might be more natural for NFSv4.x to use a referral or a pNFS layout instead.
Do you think that's a viable approach?
> I have a new patch set that is working and that I am testing and cleaning up that adds multiple hostname support to mount.nfs for NFSv4.1, and that kicks off an EXCHANGE_ID for each multi-host address. If the EXCHANGE_ID succeeds, a BIND_CONN_TO_SESSION is also called.
>
> —>Andy
>
>
>>
>> Here is the steps to make it works for those who wants to try it. :
>>
>> Test configuration :
>>
>> Running Server Kernel Version : 3.2.0
>> Running Client Kernel Version : 3.2.0
>> Patch deployment Client Kernel Version : 4.5.0 rc4+
>> Debian 7.9
>> GNS3 1.4.1
>> Wireshark 1.12.3
>> VirtualBox 5.0.14
>> Client and Server are Virtualbox VM's conected each other with a GNS3 Switch
>>
>> NO AUTHENTIFICATION - this is only for test purpose
>>
>> +----------+ +----------+
>> | | 192.168.1.2 /24 +----------+ 192.168.1.3 /24 | |
>> | +-----------------+ +-----------------+ |
>> | Client | | Switch | | Server |
>> | +-----------------+ +-----------------+ |
>> | | 192.168.2.20/24 +----------+ 192.168.2.30 /24| |
>> +----------+ +----------+
>>
>>
>> Steps :
>> -------------------------------------------------------------------
>> Server NFS4.1 configuration :
>>
>> Make sure you have an internet connection.
>>
>> Commands assuming you are root:
>>
>> apt-get update && apt-get upgrade
>> apt-get install nfs-kernel-server
>> mkdir home/testnfs
>> chmod 777 home/testnfs
>> nano etc/exports
>> #Add these line in the "exports" file to set the "testnfs" folder available
>> /home/testnfs 192.168.1.2(rw,sync)
>> /home/testnfs 192.168.2.20(rw,sync)
>> #end
>>
>> #Enable NFSv4.1
>> etc/init.d/nfs-kernel-server stop
>> nano proc/fs/nfsd/versions
>> #set +2 +3 +4 -4.1 to +2 +3 +4 +4.1
>> etc/init.d/nfs-kernel-server start
>>
>> #server ready
>> #end server configuration
>> -------------------------------------------------------------------
>> Client NFS4.1 configuration :
>>
>> Commands assuming you are root:
>> #------------Prepare new Kernel with new patches----------
>> apt-get install libncurses5-dev gcc make git exuberant-ctags bc libssl-dev
>> mkdir kernels
>> cd kernels
>> git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
>> #it takes a while...
>> cd linux-stable
>> cp /boot/config-`uname -r`* .config
>> #the -j4 stand for 4 jobs running at the same time, ideal for
>> multi-core processor.
>> make -j4
>> #It takes a while
>>
>> #--------------------Install the patches------------------
>> apt-get install patch
>> cd /
>> mkdir patch
>> cd patch
>> mkdir rpcmultipathpatch
>> mkdir sessiontrunkingpatch
>> #Get the Trond Myklebust's patch :
>> http://www.spinics.net/lists/linux-nfs/msg56365.html
>> nano rpcmultipathpatch/patch1.patch
>> .
>> .
>> .
>> nano rpcmultipathpatch/patch13.patch
>> #Get the Andy Adamson's patches :
>> http://www.spinics.net/lists/linux-nfs/msg56437.html
>> nano sessiontrunkingpatch/patch1.patch
>> .
>> .
>> .
>> nano sessiontrunkingpatch/patch6.patch
>>
>>
>> cd /
>>
>> #Apply rpc multipath patch
>> patch -p1 < rpcmultipathpatch/patch1.patch
>> patch -p1 < rpcmultipathpatch/patch2.patch
>> patch -p1 < rpcmultipathpatch/patch3.patch
>> patch -p1 < rpcmultipathpatch/patch4.patch
>> patch -p1 < rpcmultipathpatch/patch5.patch
>> patch -p1 < rpcmultipathpatch/patch6.patch
>> patch -p1 < rpcmultipathpatch/patch7.patch
>> patch -p1 < rpcmultipathpatch/patch8.patch
>> patch -p1 < rpcmultipathpatch/patch9.patch
>> patch -p1 < rpcmultipathpatch/patch10.patch
>> patch -p1 < rpcmultipathpatch/patch11.patch
>> patch -p1 < rpcmultipathpatch/patch12.patch
>> patch -p1 < rpcmultipathpatch/patch13.patch
>>
>> #Apply client session trunking patches
>> patch -p1 < sessiontrunkingpatch/patch1.patch
>> patch -p1 < sessiontrunkingpatch/patch2.patch
>> patch -p1 < sessiontrunkingpatch/patch3.patch
>> patch -p1 < sessiontrunkingpatch/patch4.patch
>> patch -p1 < sessiontrunkingpatch/patch5.patch
>> patch -p1 < sessiontrunkingpatch/patch6.patch
>>
>> make -j4
>>
>> #I got a compilation error in the "net/sunrpc/xprtmultipath.c" line 220
>> #at WRITE_ONCE(&xpi->xpi_cursor,NULL);
>> #error: lvalue required as unary ‘&’ operand
>> #I remove the "&"
>>
>> make -j4 M=net/sunrpc
>>
>> make modules_install install
>> #reboot and boot on the new kernel (with GRUB)
>>
>>
>> #--------------------Mount--------------------
>> mkdir testmount
>> mount -tnfs4 -ominorversion=1 192.168.1.3:/home/testnfs testmount
>> mount -tnfs4 -ominorversion=1 192.168.2.30:/home/testnfs testmount
>>
>> #end client configuration
>>
>> The round-robin implementation distribute correctly the operations
>> between the two interfaces and the BIND_CONN_TO_SESSION seems to work.
>>
>>
>> If you cut a cable, the client continue to send nfs packets on it, so
>> you loose one operation on two.
>>
>>
>>
>>
>>
>> Thanks to http://kernelnewbies.org/.
>> Thanks to Trond Myklebust and Andy Adamson for the patches
>
> N‹§²æìr¸›yúèšØb²X¬¶Ç§vØ^–)Þº{.nÇ+‰·¥Š{±û"žØ^n‡r¡ö¦zËëh™¨èÚ&¢ø ®G«éh®(階ŠÝ¢j"ú¶m§ÿïêäz¹Þ–Šàþf£¢·hšˆ§~ˆmš
DQo+IE9uIEZlYiAxNywgMjAxNiwgYXQgMTo0NCBQTSwgSi4gQnJ1Y2UgRmllbGRzIDxiZmllbGRz
QGZpZWxkc2VzLm9yZz4gd3JvdGU6DQo+IA0KPiBPbiBXZWQsIEZlYiAxNywgMjAxNiBhdCAwNTo0
OTozOVBNICswMDAwLCBBZGFtc29uLCBBbmR5IHdyb3RlOg0KPj4gT24gRmViIDE3LCAyMDE2LCBh
dCAxMTozNCBBTSwgSi4gQnJ1Y2UgRmllbGRzIDxiZmllbGRzQGZpZWxkc2VzLm9yZz4gd3JvdGU6
DQo+Pj4gSSdtIHN1cnByaXNlZCB0aGVyZSdzICBCSU5EX0NPTk5fVE9fU0VTU0lPTiBpbiB0aGUg
Y2FzZSBNYXJ0aW4ncw0KPj4+IHRlc3RpbmcuICBJIGRpZG4ndCB0aGluayBpdCB3YXMgbmVjZXNz
YXJ5IHVubGVzcyB5b3Ugd2VyZSB1c2luZw0KPj4+IGtlcmJlcm9zLiAgTWF5YmUgSSdtIGZvcmdl
dHRpbmcgc29tZSBzdWJ0bGV0eeKApi4NCj4+IA0KPj4gVGhlIGZpcnN0IGhvc3QgKGhvc3QxIGFi
b3ZlKSBpcyB0aGUgbW91bnQgaG9zdC4gRWFjaCBhZGRpdGlvbmFsIGhvc3QgaXMgYSBuZXcgY29u
bmVjdGlvbi4gQW4gRVhDSEFOR0VfSUQgaXMgc2VudCBvbiB0aGUgbmV3IGNvbm5lY3Rpb24sIGFu
ZCB0aGUgcmVzdWx0YW50IHNlcnZlcnNjb3BlLCBzZXJ2ZXIgb3duZXIsIGNsaWVudGlkLCBldGMg
ZnJvbSB0aGUgbmV3IGNvbm5lY3Rpb24gRVhDSEFOR0VfSUQgcmVwbHkgaXMgY29tcGFyZWQgdG8g
dGhlIG1vdW50IGV4Y2hhbmdlX2lkIGFyZ3Mgc3RvcmVkIGluIHRoZSBuZnNfY2xpZW50IHN0cnVj
dC4gSWYgaXQgcGFzc2VzIHRoZSBzZXNzaW9uIHRydW5raW5nIHJlcXVpcmVtZW50cywgdGhlbiB0
aGUgY29ubmVjdGlvbiBuZWVkcyB0byBiZSBib3VuZCB0byB0aGUgc2Vzc2lvbiB0byBlbmFibGUg
c2Vzc2lvbiB0cnVua2luZy4NCj4gDQo+IFRoYXQgYWxsIG1ha2VzIHNlbnNlIHRvIG1lLCBidXQ6
DQo+IA0KPiAJIklmLCB3aGVuIHRoZSBjbGllbnQgSUQgd2FzIGNyZWF0ZWQsIHRoZSBjbGllbnQg
b3B0ZWQgZm9yDQo+IAlTUDRfTk9ORSBzdGF0ZSBwcm90ZWN0aW9uLCB0aGUgY2xpZW50IGlzIG5v
dCByZXF1aXJlZCB0byB1c2UNCj4gCUJJTkRfQ09OTl9UT19TRVNTSU9OIHRvIGFzc29jaWF0ZSB0
aGUgY29ubmVjdGlvbiB3aXRoIHRoZQ0KPiAJc2Vzc2lvbiwgdW5sZXNzIHRoZSBjbGllbnQgd2lz
aGVzIHRvIGFzc29jaWF0ZSB0aGUgY29ubmVjdGlvbg0KPiAJd2l0aCB0aGUgYmFja2NoYW5uZWwu
ICBXaGVuIFNQNF9OT05FIHByb3RlY3Rpb24gaXMgdXNlZCwgc2ltcGx5DQo+IAlzZW5kaW5nIGEg
Q09NUE9VTkQgcmVxdWVzdCB3aXRoIGEgU0VRVUVOQ0Ugb3BlcmF0aW9uIGlzDQo+IAlzdWZmaWNp
ZW50IHRvIGFzc29jaWF0ZSB0aGUgY29ubmVjdGlvbiB3aXRoIHRoZSBzZXNzaW9uDQo+IAlzcGVj
aWZpZWQgaW4gU0VRVUVOQ0UuIg0KPiANCj4gV2VsbCwgYnV0IG1heWJlIHRoYXQncyBhY2FkZW1p
YywgdGhlcmUncyBwcm9iYWJseSBubyBoYXJtIGluIHNlbmRpbmcgdGhlDQo+IEJJTkRfQ09OTl9U
T19TRVNTSU9OLg0KDQpUaGF0IGFsc28gbWFrZXMgc2Vuc2UgdG8gbWUuIFRoZSBCSU5EX0NPTk5f
VE9fU0VTU0lPTiBkb2VzIG1ha2UgaXQgcGVyZmVjdGx5IGNsZWFyIDpfDQoNCuKAlD5BbmR5DQo+
IA0KPiAtLWIuDQo+IA0KPj4gDQo+PiDigJQ+QW5keQ0KPj4gDQo+PiBSRkM1NjYxIDIuMTAuNQ0K
Pj4g4oCm4oCmDQo+PiBTZXNzaW9uIFRydW5raW5nLiAgSWYgdGhlIGVpYV9jbGllbnRvd25lciBh
cmd1bWVudCBpcyB0aGUgc2FtZSBpbiB0d28NCj4+ICAgICAgZGlmZmVyZW50IEVYQ0hBTkdFX0lE
IHJlcXVlc3RzLCBhbmQgdGhlIGVpcl9jbGllbnRpZCwNCj4+ICAgICAgZWlyX3NlcnZlcl9vd25l
ci5zb19tYWpvcl9pZCwgZWlyX3NlcnZlcl9vd25lci5zb19taW5vcl9pZCwgYW5kDQo+PiAgICAg
IGVpcl9zZXJ2ZXJfc2NvcGUgcmVzdWx0cyBtYXRjaCBpbiBib3RoIEVYQ0hBTkdFX0lEIHJlc3Vs
dHMsIHRoZW4NCj4+ICAgICAgdGhlIGNsaWVudCBpcyBwZXJtaXR0ZWQgdG8gcGVyZm9ybSBzZXNz
aW9uIHRydW5raW5nLiAgSWYgdGhlDQo+PiAgICAgIGNsaWVudCBoYXMgbm8gc2Vzc2lvbiBtYXBw
aW5nIHRvIHRoZSB0dXBsZSBvZiBlaXJfY2xpZW50aWQsDQo+PiAgICAgIGVpcl9zZXJ2ZXJfb3du
ZXIuc29fbWFqb3JfaWQsIGVpcl9zZXJ2ZXJfc2NvcGUsIGFuZA0KPj4gICAgICBlaXJfc2VydmVy
X293bmVyLnNvX21pbm9yX2lkLCB0aGVuIGl0IGNyZWF0ZXMgdGhlIHNlc3Npb24gdmlhIGENCj4+
ICAgICAgQ1JFQVRFX1NFU1NJT04gb3BlcmF0aW9uIG92ZXIgb25lIG9mIHRoZSBjb25uZWN0aW9u
cywgd2hpY2gNCj4+ICAgICAgYXNzb2NpYXRlcyB0aGUgY29ubmVjdGlvbiB0byB0aGUgc2Vzc2lv
bi4gIElmIHRoZXJlIGlzIGEgc2Vzc2lvbg0KPj4gICAgICBmb3IgdGhlIHR1cGxlLCB0aGUgY2xp
ZW50IGNhbiBzZW5kIEJJTkRfQ09OTl9UT19TRVNTSU9OIHRvDQo+PiAgICAgIGFzc29jaWF0ZSB0
aGUgY29ubmVjdGlvbiB0byB0aGUgc2Vzc2lvbi4NCj4+IA0KPj4gICAgICBPZiBjb3Vyc2UsIGlm
IHRoZSBjbGllbnQgZG9lcyBub3QgZGVzaXJlIHRvIHVzZSBzZXNzaW9uIHRydW5raW5nLA0KPj4g
ICAgICBpdCBpcyBub3QgcmVxdWlyZWQgdG8gZG8gc28uICBJdCBjYW4gaW52b2tlIENSRUFURV9T
RVNTSU9OIG9uIHRoZQ0KPj4gICAgICBjb25uZWN0aW9uLiAgVGhpcyB3aWxsIHJlc3VsdCBpbiBj
bGllbnQgSUQgdHJ1bmtpbmcgYXMgZGVzY3JpYmVkDQo+PiAgICAgIGJlbG93LiAgSXQgY2FuIGFs
c28gZGVjaWRlIHRvIGRyb3AgdGhlIGNvbm5lY3Rpb24gaWYgaXQgZG9lcyBub3QNCj4+ICAgICAg
Y2hvb3NlIHRvIHVzZSB0cnVua2luZy4NCj4+PiANCj4+PiAtLWIuDQo+Pj4gDQo+Pj4+IA0KPj4+
PiDigJQ+QW5keQ0KPj4+PiANCj4+Pj4gDQo+Pj4+PiANCj4+Pj4+IEhlcmUgaXMgdGhlIHN0ZXBz
IHRvIG1ha2UgaXQgd29ya3MgZm9yIHRob3NlIHdobyB3YW50cyB0byB0cnkgaXQuIDoNCj4+Pj4+
IA0KPj4+Pj4gVGVzdCBjb25maWd1cmF0aW9uIDoNCj4+Pj4+IA0KPj4+Pj4gUnVubmluZyBTZXJ2
ZXIgS2VybmVsIFZlcnNpb24gOiAzLjIuMA0KPj4+Pj4gUnVubmluZyBDbGllbnQgS2VybmVsIFZl
cnNpb24gOiAzLjIuMA0KPj4+Pj4gUGF0Y2ggZGVwbG95bWVudCBDbGllbnQgS2VybmVsIFZlcnNp
b24gOiA0LjUuMCByYzQrDQo+Pj4+PiBEZWJpYW4gNy45DQo+Pj4+PiBHTlMzIDEuNC4xDQo+Pj4+
PiBXaXJlc2hhcmsgMS4xMi4zDQo+Pj4+PiBWaXJ0dWFsQm94IDUuMC4xNA0KPj4+Pj4gQ2xpZW50
IGFuZCBTZXJ2ZXIgYXJlIFZpcnR1YWxib3ggVk0ncyBjb25lY3RlZCBlYWNoIG90aGVyIHdpdGgg
YSBHTlMzIFN3aXRjaA0KPj4+Pj4gDQo+Pj4+PiBOTyBBVVRIRU5USUZJQ0FUSU9OIC0gdGhpcyBp
cyBvbmx5IGZvciB0ZXN0IHB1cnBvc2UNCj4+Pj4+IA0KPj4+Pj4gKy0tLS0tLS0tLS0rICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICstLS0tLS0tLS0tKw0KPj4+
Pj4gfCAgICAgICAgICB8IDE5Mi4xNjguMS4yIC8yNCArLS0tLS0tLS0tLSsgMTkyLjE2OC4xLjMg
LzI0IHwgICAgICAgICAgfA0KPj4+Pj4gfCAgICAgICAgICArLS0tLS0tLS0tLS0tLS0tLS0rICAg
ICAgICAgICstLS0tLS0tLS0tLS0tLS0tLSsgICAgICAgICAgfA0KPj4+Pj4gfCAgQ2xpZW50ICB8
ICAgICAgICAgICAgICAgICB8ICBTd2l0Y2ggIHwgICAgICAgICAgICAgICAgIHwgIFNlcnZlciAg
fA0KPj4+Pj4gfCAgICAgICAgICArLS0tLS0tLS0tLS0tLS0tLS0rICAgICAgICAgICstLS0tLS0t
LS0tLS0tLS0tLSsgICAgICAgICAgfA0KPj4+Pj4gfCAgICAgICAgICB8IDE5Mi4xNjguMi4yMC8y
NCArLS0tLS0tLS0tLSsgMTkyLjE2OC4yLjMwIC8yNHwgICAgICAgICAgfA0KPj4+Pj4gKy0tLS0t
LS0tLS0rICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICstLS0t
LS0tLS0tKw0KPj4+Pj4gDQo+Pj4+PiANCj4+Pj4+IFN0ZXBzIDoNCj4+Pj4+IC0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0N
Cj4+Pj4+IFNlcnZlciBORlM0LjEgY29uZmlndXJhdGlvbiA6DQo+Pj4+PiANCj4+Pj4+IE1ha2Ug
c3VyZSB5b3UgaGF2ZSBhbiBpbnRlcm5ldCBjb25uZWN0aW9uLg0KPj4+Pj4gDQo+Pj4+PiBDb21t
YW5kcyBhc3N1bWluZyB5b3UgYXJlIHJvb3Q6DQo+Pj4+PiANCj4+Pj4+IGFwdC1nZXQgdXBkYXRl
ICYmIGFwdC1nZXQgdXBncmFkZQ0KPj4+Pj4gYXB0LWdldCBpbnN0YWxsIG5mcy1rZXJuZWwtc2Vy
dmVyDQo+Pj4+PiBta2RpciBob21lL3Rlc3RuZnMNCj4+Pj4+IGNobW9kIDc3NyBob21lL3Rlc3Ru
ZnMNCj4+Pj4+IG5hbm8gZXRjL2V4cG9ydHMNCj4+Pj4+ICNBZGQgdGhlc2UgbGluZSBpbiB0aGUg
ImV4cG9ydHMiIGZpbGUgdG8gc2V0IHRoZSAidGVzdG5mcyIgZm9sZGVyIGF2YWlsYWJsZQ0KPj4+
Pj4gIC9ob21lL3Rlc3RuZnMgICAgMTkyLjE2OC4xLjIocncsc3luYykNCj4+Pj4+ICAvaG9tZS90
ZXN0bmZzICAgIDE5Mi4xNjguMi4yMChydyxzeW5jKQ0KPj4+Pj4gI2VuZA0KPj4+Pj4gDQo+Pj4+
PiAjRW5hYmxlIE5GU3Y0LjENCj4+Pj4+IGV0Yy9pbml0LmQvbmZzLWtlcm5lbC1zZXJ2ZXIgc3Rv
cA0KPj4+Pj4gbmFubyBwcm9jL2ZzL25mc2QvdmVyc2lvbnMNCj4+Pj4+ICNzZXQgKzIgKzMgKzQg
LTQuMSB0byArMiArMyArNCArNC4xDQo+Pj4+PiBldGMvaW5pdC5kL25mcy1rZXJuZWwtc2VydmVy
IHN0YXJ0DQo+Pj4+PiANCj4+Pj4+ICNzZXJ2ZXIgcmVhZHkNCj4+Pj4+ICNlbmQgc2VydmVyIGNv
bmZpZ3VyYXRpb24NCj4+Pj4+IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCj4+Pj4+IENsaWVudCBORlM0LjEgY29uZmln
dXJhdGlvbiA6DQo+Pj4+PiANCj4+Pj4+IENvbW1hbmRzIGFzc3VtaW5nIHlvdSBhcmUgcm9vdDoN
Cj4+Pj4+ICMtLS0tLS0tLS0tLS1QcmVwYXJlIG5ldyBLZXJuZWwgd2l0aCBuZXcgcGF0Y2hlcy0t
LS0tLS0tLS0NCj4+Pj4+IGFwdC1nZXQgaW5zdGFsbCBsaWJuY3Vyc2VzNS1kZXYgZ2NjIG1ha2Ug
Z2l0IGV4dWJlcmFudC1jdGFncyBiYyBsaWJzc2wtZGV2DQo+Pj4+PiBta2RpciBrZXJuZWxzDQo+
Pj4+PiBjZCBrZXJuZWxzDQo+Pj4+PiBnaXQgY2xvbmUgZ2l0Oi8vZ2l0Lmtlcm5lbC5vcmcvcHVi
L3NjbS9saW51eC9rZXJuZWwvZ2l0L3N0YWJsZS9saW51eC1zdGFibGUuZ2l0DQo+Pj4+PiAjaXQg
dGFrZXMgYSB3aGlsZS4uLg0KPj4+Pj4gY2QgbGludXgtc3RhYmxlDQo+Pj4+PiBjcCAvYm9vdC9j
b25maWctYHVuYW1lIC1yYCogLmNvbmZpZw0KPj4+Pj4gI3RoZSAtajQgc3RhbmQgZm9yIDQgam9i
cyBydW5uaW5nIGF0IHRoZSBzYW1lIHRpbWUsIGlkZWFsIGZvcg0KPj4+Pj4gbXVsdGktY29yZSBw
cm9jZXNzb3IuDQo+Pj4+PiBtYWtlIC1qNA0KPj4+Pj4gI0l0IHRha2VzIGEgd2hpbGUNCj4+Pj4+
IA0KPj4+Pj4gIy0tLS0tLS0tLS0tLS0tLS0tLS0tSW5zdGFsbCB0aGUgcGF0Y2hlcy0tLS0tLS0t
LS0tLS0tLS0tLQ0KPj4+Pj4gYXB0LWdldCBpbnN0YWxsIHBhdGNoDQo+Pj4+PiBjZCAvDQo+Pj4+
PiBta2RpciBwYXRjaA0KPj4+Pj4gY2QgcGF0Y2gNCj4+Pj4+IG1rZGlyIHJwY211bHRpcGF0aHBh
dGNoDQo+Pj4+PiBta2RpciBzZXNzaW9udHJ1bmtpbmdwYXRjaA0KPj4+Pj4gI0dldCB0aGUgVHJv
bmQgTXlrbGVidXN0J3MgcGF0Y2ggOg0KPj4+Pj4gaHR0cDovL3d3dy5zcGluaWNzLm5ldC9saXN0
cy9saW51eC1uZnMvbXNnNTYzNjUuaHRtbA0KPj4+Pj4gbmFubyBycGNtdWx0aXBhdGhwYXRjaC9w
YXRjaDEucGF0Y2gNCj4+Pj4+IC4NCj4+Pj4+IC4NCj4+Pj4+IC4NCj4+Pj4+IG5hbm8gcnBjbXVs
dGlwYXRocGF0Y2gvcGF0Y2gxMy5wYXRjaA0KPj4+Pj4gI0dldCB0aGUgQW5keSBBZGFtc29uJ3Mg
cGF0Y2hlcyA6DQo+Pj4+PiBodHRwOi8vd3d3LnNwaW5pY3MubmV0L2xpc3RzL2xpbnV4LW5mcy9t
c2c1NjQzNy5odG1sDQo+Pj4+PiBuYW5vIHNlc3Npb250cnVua2luZ3BhdGNoL3BhdGNoMS5wYXRj
aA0KPj4+Pj4gLg0KPj4+Pj4gLg0KPj4+Pj4gLg0KPj4+Pj4gbmFubyBzZXNzaW9udHJ1bmtpbmdw
YXRjaC9wYXRjaDYucGF0Y2gNCj4+Pj4+IA0KPj4+Pj4gDQo+Pj4+PiBjZCAvDQo+Pj4+PiANCj4+
Pj4+ICNBcHBseSBycGMgbXVsdGlwYXRoIHBhdGNoDQo+Pj4+PiBwYXRjaCAtcDEgPCBycGNtdWx0
aXBhdGhwYXRjaC9wYXRjaDEucGF0Y2gNCj4+Pj4+IHBhdGNoIC1wMSA8IHJwY211bHRpcGF0aHBh
dGNoL3BhdGNoMi5wYXRjaA0KPj4+Pj4gcGF0Y2ggLXAxIDwgcnBjbXVsdGlwYXRocGF0Y2gvcGF0
Y2gzLnBhdGNoDQo+Pj4+PiBwYXRjaCAtcDEgPCBycGNtdWx0aXBhdGhwYXRjaC9wYXRjaDQucGF0
Y2gNCj4+Pj4+IHBhdGNoIC1wMSA8IHJwY211bHRpcGF0aHBhdGNoL3BhdGNoNS5wYXRjaA0KPj4+
Pj4gcGF0Y2ggLXAxIDwgcnBjbXVsdGlwYXRocGF0Y2gvcGF0Y2g2LnBhdGNoDQo+Pj4+PiBwYXRj
aCAtcDEgPCBycGNtdWx0aXBhdGhwYXRjaC9wYXRjaDcucGF0Y2gNCj4+Pj4+IHBhdGNoIC1wMSA8
IHJwY211bHRpcGF0aHBhdGNoL3BhdGNoOC5wYXRjaA0KPj4+Pj4gcGF0Y2ggLXAxIDwgcnBjbXVs
dGlwYXRocGF0Y2gvcGF0Y2g5LnBhdGNoDQo+Pj4+PiBwYXRjaCAtcDEgPCBycGNtdWx0aXBhdGhw
YXRjaC9wYXRjaDEwLnBhdGNoDQo+Pj4+PiBwYXRjaCAtcDEgPCBycGNtdWx0aXBhdGhwYXRjaC9w
YXRjaDExLnBhdGNoDQo+Pj4+PiBwYXRjaCAtcDEgPCBycGNtdWx0aXBhdGhwYXRjaC9wYXRjaDEy
LnBhdGNoDQo+Pj4+PiBwYXRjaCAtcDEgPCBycGNtdWx0aXBhdGhwYXRjaC9wYXRjaDEzLnBhdGNo
DQo+Pj4+PiANCj4+Pj4+ICNBcHBseSBjbGllbnQgc2Vzc2lvbiB0cnVua2luZyBwYXRjaGVzDQo+
Pj4+PiBwYXRjaCAtcDEgPCBzZXNzaW9udHJ1bmtpbmdwYXRjaC9wYXRjaDEucGF0Y2gNCj4+Pj4+
IHBhdGNoIC1wMSA8IHNlc3Npb250cnVua2luZ3BhdGNoL3BhdGNoMi5wYXRjaA0KPj4+Pj4gcGF0
Y2ggLXAxIDwgc2Vzc2lvbnRydW5raW5ncGF0Y2gvcGF0Y2gzLnBhdGNoDQo+Pj4+PiBwYXRjaCAt
cDEgPCBzZXNzaW9udHJ1bmtpbmdwYXRjaC9wYXRjaDQucGF0Y2gNCj4+Pj4+IHBhdGNoIC1wMSA8
IHNlc3Npb250cnVua2luZ3BhdGNoL3BhdGNoNS5wYXRjaA0KPj4+Pj4gcGF0Y2ggLXAxIDwgc2Vz
c2lvbnRydW5raW5ncGF0Y2gvcGF0Y2g2LnBhdGNoDQo+Pj4+PiANCj4+Pj4+IG1ha2UgLWo0DQo+
Pj4+PiANCj4+Pj4+ICNJIGdvdCBhIGNvbXBpbGF0aW9uIGVycm9yIGluIHRoZSAibmV0L3N1bnJw
Yy94cHJ0bXVsdGlwYXRoLmMiIGxpbmUgMjIwDQo+Pj4+PiAjYXQgV1JJVEVfT05DRSgmeHBpLT54
cGlfY3Vyc29yLE5VTEwpOw0KPj4+Pj4gI2Vycm9yOiBsdmFsdWUgcmVxdWlyZWQgYXMgdW5hcnkg
4oCYJuKAmSBvcGVyYW5kDQo+Pj4+PiAjSSByZW1vdmUgdGhlICImIg0KPj4+Pj4gDQo+Pj4+PiBt
YWtlIC1qNCBNPW5ldC9zdW5ycGMNCj4+Pj4+IA0KPj4+Pj4gbWFrZSBtb2R1bGVzX2luc3RhbGwg
aW5zdGFsbA0KPj4+Pj4gI3JlYm9vdCBhbmQgYm9vdCBvbiB0aGUgbmV3IGtlcm5lbCAod2l0aCBH
UlVCKQ0KPj4+Pj4gDQo+Pj4+PiANCj4+Pj4+ICMtLS0tLS0tLS0tLS0tLS0tLS0tLU1vdW50LS0t
LS0tLS0tLS0tLS0tLS0tLS0NCj4+Pj4+IG1rZGlyIHRlc3Rtb3VudA0KPj4+Pj4gbW91bnQgLXRu
ZnM0IC1vbWlub3J2ZXJzaW9uPTEgMTkyLjE2OC4xLjM6L2hvbWUvdGVzdG5mcyB0ZXN0bW91bnQN
Cj4+Pj4+IG1vdW50IC10bmZzNCAtb21pbm9ydmVyc2lvbj0xIDE5Mi4xNjguMi4zMDovaG9tZS90
ZXN0bmZzIHRlc3Rtb3VudA0KPj4+Pj4gDQo+Pj4+PiAjZW5kIGNsaWVudCBjb25maWd1cmF0aW9u
DQo+Pj4+PiANCj4+Pj4+IFRoZSByb3VuZC1yb2JpbiBpbXBsZW1lbnRhdGlvbiBkaXN0cmlidXRl
IGNvcnJlY3RseSB0aGUgb3BlcmF0aW9ucw0KPj4+Pj4gYmV0d2VlbiB0aGUgdHdvIGludGVyZmFj
ZXMgYW5kIHRoZSBCSU5EX0NPTk5fVE9fU0VTU0lPTiBzZWVtcyB0byB3b3JrLg0KPj4+Pj4gDQo+
Pj4+PiANCj4+Pj4+IElmIHlvdSBjdXQgYSBjYWJsZSwgdGhlIGNsaWVudCBjb250aW51ZSB0byBz
ZW5kIG5mcyBwYWNrZXRzIG9uIGl0LCBzbw0KPj4+Pj4geW91IGxvb3NlIG9uZSBvcGVyYXRpb24g
b24gdHdvLg0KPj4+Pj4gDQo+Pj4+PiANCj4+Pj4+IA0KPj4+Pj4gDQo+Pj4+PiANCj4+Pj4+IFRo
YW5rcyB0byBodHRwOi8va2VybmVsbmV3Ymllcy5vcmcvLg0KPj4+Pj4gVGhhbmtzIHRvIFRyb25k
IE15a2xlYnVzdCBhbmQgQW5keSBBZGFtc29uIGZvciB0aGUgcGF0Y2hlcw0KPj4+PiANCj4+Pj4g
Tj8/Pz8/cj8/eT8/Pz9iP1g/P8endj9ePyneunsubj8rPz8/P3s/Pz8iPz9ebj9yPz8/ej8aPz9o
Pz8/Pz8mPz8eP0c/Pz9oPwMoP+majj/domoiPz8aPxttPz8/Pz8/ej/elj8/P2Y/Pz9oPz8/fj9t
DQo+PiANCg0K
On Wed, Feb 17, 2016 at 02:06:35PM -0500, Chuck Lever wrote:
>
> > On Feb 17, 2016, at 9:50 AM, Adamson, Andy
> > <[email protected]> wrote: Thanks for testing. As Trond
> > pointed out, the correct way to indicate multiple hostnames is on
> > the mount command line
> >
> > mount -o minorversion=1 host1,host2,…,hostn:/<export> /<mntdir>
>
> It might be more natural for NFSv4.x to use a referral or a pNFS
> layout instead. Do you think that's a viable approach?
Seems like an easy application for fs_locations{_info?}. It'd need
server support too, and I think you probably want this manual method as
well for now.
I wonder if the server can easily generate a list of IP addresses to
advertise automatically? In which case the whole thing would be
zero-configuration, which would be nice.
pNFS sounds like overkill.
--b.
DQo+IE9uIEZlYiAxNywgMjAxNiwgYXQgMzo1OSBQTSwgSi4gQnJ1Y2UgRmllbGRzIDxiZmllbGRz
QGZpZWxkc2VzLm9yZz4gd3JvdGU6DQo+IA0KPiBPbiBXZWQsIEZlYiAxNywgMjAxNiBhdCAwMjow
NjozNVBNIC0wNTAwLCBDaHVjayBMZXZlciB3cm90ZToNCj4+IA0KPj4+IE9uIEZlYiAxNywgMjAx
NiwgYXQgOTo1MCBBTSwgQWRhbXNvbiwgQW5keQ0KPj4+IDxXaWxsaWFtLkFkYW1zb25AbmV0YXBw
LmNvbT4gd3JvdGU6IFRoYW5rcyBmb3IgdGVzdGluZy4gQXMgVHJvbmQNCj4+PiBwb2ludGVkIG91
dCwgdGhlIGNvcnJlY3Qgd2F5IHRvIGluZGljYXRlIG11bHRpcGxlIGhvc3RuYW1lcyBpcyBvbg0K
Pj4+IHRoZSBtb3VudCBjb21tYW5kIGxpbmUNCj4+PiANCj4+PiBtb3VudCAtbyBtaW5vcnZlcnNp
b249MSBob3N0MSxob3N0MizDouKCrMKmLGhvc3RuOi88ZXhwb3J0PiAgLzxtbnRkaXI+DQo+PiAN
Cj4+IEl0IG1pZ2h0IGJlIG1vcmUgbmF0dXJhbCBmb3IgTkZTdjQueCB0byB1c2UgYSByZWZlcnJh
bCBvciBhIHBORlMNCj4+IGxheW91dCBpbnN0ZWFkLiAgRG8geW91IHRoaW5rIHRoYXQncyBhIHZp
YWJsZSBhcHByb2FjaD8NCj4gDQo+IFNlZW1zIGxpa2UgYW4gZWFzeSBhcHBsaWNhdGlvbiBmb3Ig
ZnNfbG9jYXRpb25ze19pbmZvP30uICBJdCdkIG5lZWQNCj4gc2VydmVyIHN1cHBvcnQgdG9vLCBh
bmQgSSB0aGluayB5b3UgcHJvYmFibHkgd2FudCB0aGlzIG1hbnVhbCBtZXRob2QgYXMNCj4gd2Vs
bCBmb3Igbm93Lg0KDQpJIGFncmVlIHRoYXQgdGhlIG1hbnVhbCBtZXRob2QgaXMgdXNlZnVsIGFz
IGl0IGFsbG93cyB0aGUgY2xpZW50IGFkbWluIHRvIGRlY2lkZSBpZiB0aGUgbW91bnQgcmVxdWly
ZXMgc2Vzc2lvbiB0cnVua2luZy4NClRoZSBmc19sb2NhdGlvbnMgd291bGQgbmVlZCB0byBiZSBy
ZXF1ZXN0ZWQgYnkgdGhlIGNsaWVudC4gSSBndWVzcyB3ZSByZXFlc3QgdGhlbSBhdCBldmVyeSBt
b3VudOKApi4NCg0KPiANCj4gSSB3b25kZXIgaWYgdGhlIHNlcnZlciBjYW4gZWFzaWx5IGdlbmVy
YXRlIGEgbGlzdCBvZiBJUCBhZGRyZXNzZXMgdG8NCj4gYWR2ZXJ0aXNlIGF1dG9tYXRpY2FsbHk/
ICANCg0KWWVzIC0gZnNfbG9jYXRpb25zIQ0KDQrigJQ+QW5keQ0KDQo+IEluIHdoaWNoIGNhc2Ug
dGhlIHdob2xlIHRoaW5nIHdvdWxkIGJlDQo+IHplcm8tY29uZmlndXJhdGlvbiwgd2hpY2ggd291
bGQgYmUgbmljZS4NCj4gDQo+IHBORlMgc291bmRzIGxpa2Ugb3ZlcmtpbGwuDQo+IA0KPiAtLWIu
DQoNCg==
> On Feb 17, 2016, at 5:35 PM, Adamson, Andy <[email protected]> wrote:
>
>
>> On Feb 17, 2016, at 3:59 PM, J. Bruce Fields <[email protected]> wrote:
>>
>> On Wed, Feb 17, 2016 at 02:06:35PM -0500, Chuck Lever wrote:
>>>
>>>> On Feb 17, 2016, at 9:50 AM, Adamson, Andy
>>>> <[email protected]> wrote: Thanks for testing. As Trond
>>>> pointed out, the correct way to indicate multiple hostnames is on
>>>> the mount command line
>>>>
>>>> mount -o minorversion=1 host1,host2,ÃĒâŽÂĶ,hostn:/<export> /<mntdir>
>>>
>>> It might be more natural for NFSv4.x to use a referral or a pNFS
>>> layout instead. Do you think that's a viable approach?
>>
>> Seems like an easy application for fs_locations{_info?}. It'd need
>> server support too, and I think you probably want this manual method as
>> well for now.
>
> I agree that the manual method is useful as it allows the client admin to decide if the mount requires session trunking.
Seems like the server admin is in a better position to know
the locations of replicated data. The server can advertise
the most up-to-date location information. More scalable
than telling every client admin how to set this up.
Adding this CLI to mount means it will be around a long time,
so we'd better be sure we want to support it for that long.
(Yes, I know, who is this "we").
> The fs_locations would need to be requested by the client. I guess we reqest them at every mountâĶ.
Yep, and fetch them again every so often. There's no real
cache coherency protocol for this information. (That's
where a pNFS layout might be more valuable).
The Linux server should be able to advertise replicas using
the replicas= export option.
>> I wonder if the server can easily generate a list of IP addresses to
>> advertise automatically?
>
> Yes - fs_locations!
>
> â>Andy
>
>> In which case the whole thing would be
>> zero-configuration, which would be nice.
>>
>> pNFS sounds like overkill.
>>
>> --b.
>
> N§ēæėrļyúčØbēXŽķĮ§vØ^)Þš{.nĮ+·Ĩ{ąû"Ø^nrĄöĶzËëhĻčÚ&Ēø ŪGŦéhŪ(éÝĒj"úķm§ĸïęäzđÞāþfĢĒ·h§~m
--
Chuck Lever
On Wed, Feb 17, 2016 at 5:52 PM, Chuck Lever <[email protected]> wrote:
>
>> On Feb 17, 2016, at 5:35 PM, Adamson, Andy <[email protected]> wrote:
>> The fs_locations would need to be requested by the client. I guess we reqest them at every mountâ€Ķ.
>
> Yep, and fetch them again every so often. There's no real
> cache coherency protocol for this information. (That's
> where a pNFS layout might be more valuable).
If your goal is to do session trunking, you only really need to check
the fs_locations attribute on the root file system. (so
GETROOTFH+GETATTR(fs_locations)). That's the natural place for a
server to advertise its full set of IP addresses, and the session
trunking protocol itself will allow you to winnow out any that might
belong to a replica server.
You might want to refresh that info whenever the connection goes away
on one or more addresses without a reboot so you can detect when NICs
are going away.
Otherwise, polling every couple of hours or so for new NICs shouldn't
be too burdensome...
Cheers,
Trond
Ok, if we have multi host addresses can we specify the client
interfaces adresses?
In other words is it possible for the client to choose the interface
which'll be used to send the data?
An another point is what is the best way to delete an existing
connection of an existing session?
Martin
On Wed, Feb 17, 2016 at 05:52:03PM -0500, Chuck Lever wrote:
>
> > On Feb 17, 2016, at 5:35 PM, Adamson, Andy <[email protected]> wrote:
> >
> >
> >> On Feb 17, 2016, at 3:59 PM, J. Bruce Fields <[email protected]> wrote:
> >>
> >> On Wed, Feb 17, 2016 at 02:06:35PM -0500, Chuck Lever wrote:
> >>>
> >>>> On Feb 17, 2016, at 9:50 AM, Adamson, Andy
> >>>> <[email protected]> wrote: Thanks for testing. As Trond
> >>>> pointed out, the correct way to indicate multiple hostnames is on
> >>>> the mount command line
> >>>>
> >>>> mount -o minorversion=1 host1,host2,ÃĒâŽÂĶ,hostn:/<export> /<mntdir>
> >>>
> >>> It might be more natural for NFSv4.x to use a referral or a pNFS
> >>> layout instead. Do you think that's a viable approach?
> >>
> >> Seems like an easy application for fs_locations{_info?}. It'd need
> >> server support too, and I think you probably want this manual method as
> >> well for now.
> >
> > I agree that the manual method is useful as it allows the client admin to decide if the mount requires session trunking.
>
> Seems like the server admin is in a better position to know
> the locations of replicated data. The server can advertise
> the most up-to-date location information. More scalable
> than telling every client admin how to set this up.
>
> Adding this CLI to mount means it will be around a long time,
> so we'd better be sure we want to support it for that long.
> (Yes, I know, who is this "we").
>
>
> > The fs_locations would need to be requested by the client. I guess we reqest them at every mountâĶ.
>
> Yep, and fetch them again every so often. There's no real
> cache coherency protocol for this information. (That's
> where a pNFS layout might be more valuable).
>
> The Linux server should be able to advertise replicas using
> the replicas= export option.
Right, but for the simple case of a server with multiple interfaces but
no other replicas it might be nice if the server could generate the
fs_locations info on its own (an explicit replicas= could still override
that).
But I don't know if it's possible to reliably enumerate the right set of
IP addresses. Some interfaces may not be on the right network, etc.
Or interfaces might have different performance characteristics, in which
case round-robin wouldn't be a good idea. So I guess this shouldn't be
automatic unless we want to put in way more smarts.
--b.
On Wed, Feb 17, 2016 at 06:55:43PM -0500, Trond Myklebust wrote:
> On Wed, Feb 17, 2016 at 5:52 PM, Chuck Lever <[email protected]> wrote:
> >
> >> On Feb 17, 2016, at 5:35 PM, Adamson, Andy <[email protected]> wrote:
> >> The fs_locations would need to be requested by the client. I guess we reqest them at every mountâ€Ķ.
> >
> > Yep, and fetch them again every so often. There's no real
> > cache coherency protocol for this information. (That's
> > where a pNFS layout might be more valuable).
>
> If your goal is to do session trunking, you only really need to check
> the fs_locations attribute on the root file system. (so
> GETROOTFH+GETATTR(fs_locations)). That's the natural place for a
> server to advertise its full set of IP addresses, and the session
> trunking protocol itself will allow you to winnow out any that might
> belong to a replica server.
I worry that round-robin could behave really badly if the client's path
to the two IP addresses have different performance characteristics. But
a server should probably still be allowed to advertise those as replicas
(e.g. maybe a slower interface is usable as a fallback?).
So maybe we should be careful about making this automatic. Unless the
load-balancing is a little smarter than pure round robin. Or unless we
can get some more fine-grained information (maybe someone could use
fs_location_info's preference information for this?).
--b.
>
> You might want to refresh that info whenever the connection goes away
> on one or more addresses without a reboot so you can detect when NICs
> are going away.
>
> Otherwise, polling every couple of hours or so for new NICs shouldn't
> be too burdensome...
>
> Cheers,
> Trond
Can we let the choice to the client if the session trunking is manual
or automatic?
Manual : with the load-balancing algorithm of his choice
Automatic : with a smarter algorithm just as M. Fields says, like an
adaptative load-balancing algo. for example.
Martin
2016-02-18 15:14 GMT+01:00 J. Bruce Fields <[email protected]>:
> On Wed, Feb 17, 2016 at 06:55:43PM -0500, Trond Myklebust wrote:
>> On Wed, Feb 17, 2016 at 5:52 PM, Chuck Lever <[email protected]> wrote:
>> >
>> >> On Feb 17, 2016, at 5:35 PM, Adamson, Andy <[email protected]> wrote:
>> >> The fs_locations would need to be requested by the client. I guess we reqest them at every mountâ€Ķ.
>> >
>> > Yep, and fetch them again every so often. There's no real
>> > cache coherency protocol for this information. (That's
>> > where a pNFS layout might be more valuable).
>>
>> If your goal is to do session trunking, you only really need to check
>> the fs_locations attribute on the root file system. (so
>> GETROOTFH+GETATTR(fs_locations)). That's the natural place for a
>> server to advertise its full set of IP addresses, and the session
>> trunking protocol itself will allow you to winnow out any that might
>> belong to a replica server.
>
> I worry that round-robin could behave really badly if the client's path
> to the two IP addresses have different performance characteristics. But
> a server should probably still be allowed to advertise those as replicas
> (e.g. maybe a slower interface is usable as a fallback?).
>
> So maybe we should be careful about making this automatic. Unless the
> load-balancing is a little smarter than pure round robin. Or unless we
> can get some more fine-grained information (maybe someone could use
> fs_location_info's preference information for this?).
>
> --b.
>
>>
>> You might want to refresh that info whenever the connection goes away
>> on one or more addresses without a reboot so you can detect when NICs
>> are going away.
>>
>> Otherwise, polling every couple of hours or so for new NICs shouldn't
>> be too burdensome...
>>
>> Cheers,
>> Trond
On Thu, Feb 18, 2016 at 9:14 AM, J. Bruce Fields <[email protected]> wrote:
>
> On Wed, Feb 17, 2016 at 06:55:43PM -0500, Trond Myklebust wrote:
> > On Wed, Feb 17, 2016 at 5:52 PM, Chuck Lever <[email protected]> wrote:
> > >
> > >> On Feb 17, 2016, at 5:35 PM, Adamson, Andy <[email protected]> wrote:
> > >> The fs_locations would need to be requested by the client. I guess we reqest them at every mountâ€Ķ.
> > >
> > > Yep, and fetch them again every so often. There's no real
> > > cache coherency protocol for this information. (That's
> > > where a pNFS layout might be more valuable).
> >
> > If your goal is to do session trunking, you only really need to check
> > the fs_locations attribute on the root file system. (so
> > GETROOTFH+GETATTR(fs_locations)). That's the natural place for a
> > server to advertise its full set of IP addresses, and the session
> > trunking protocol itself will allow you to winnow out any that might
> > belong to a replica server.
>
> I worry that round-robin could behave really badly if the client's path
> to the two IP addresses have different performance characteristics. But
> a server should probably still be allowed to advertise those as replicas
> (e.g. maybe a slower interface is usable as a fallback?).
>
> So maybe we should be careful about making this automatic. Unless the
> load-balancing is a little smarter than pure round robin. Or unless we
> can get some more fine-grained information (maybe someone could use
> fs_location_info's preference information for this?).
The multipath policy is pluggable. If you need something more clever
than round robin, then feel free to play. However do note that for
pNFS multipathing, both the files and flexfiles specs are clear that
you should not mix slow and fast transports. I imagine you probably
want to do the same for fs_locations.
As for fs_locations_info, please see FSLI4BX_(READ|WRITE)(RANK|ORDER).
DQo+IE9uIEZlYiAxOCwgMjAxNiwgYXQgMTozMiBQTSwgVHJvbmQgTXlrbGVidXN0IDx0cm9uZC5t
eWtsZWJ1c3RAcHJpbWFyeWRhdGEuY29tPiB3cm90ZToNCj4gDQo+IE9uIFRodSwgRmViIDE4LCAy
MDE2IGF0IDk6MTQgQU0sIEouIEJydWNlIEZpZWxkcyA8YmZpZWxkc0BmaWVsZHNlcy5vcmc+IHdy
b3RlOg0KPj4gDQo+PiBPbiBXZWQsIEZlYiAxNywgMjAxNiBhdCAwNjo1NTo0M1BNIC0wNTAwLCBU
cm9uZCBNeWtsZWJ1c3Qgd3JvdGU6DQo+Pj4gT24gV2VkLCBGZWIgMTcsIDIwMTYgYXQgNTo1MiBQ
TSwgQ2h1Y2sgTGV2ZXIgPGNodWNrLmxldmVyQG9yYWNsZS5jb20+IHdyb3RlOg0KPj4+PiANCj4+
Pj4+IE9uIEZlYiAxNywgMjAxNiwgYXQgNTozNSBQTSwgQWRhbXNvbiwgQW5keSA8V2lsbGlhbS5B
ZGFtc29uQG5ldGFwcC5jb20+IHdyb3RlOg0KPj4+Pj4gVGhlIGZzX2xvY2F0aW9ucyB3b3VsZCBu
ZWVkIHRvIGJlIHJlcXVlc3RlZCBieSB0aGUgY2xpZW50LiBJIGd1ZXNzIHdlIHJlcWVzdCB0aGVt
IGF0IGV2ZXJ5IG1vdW50w6LigqzEti4NCj4+Pj4gDQo+Pj4+IFllcCwgYW5kIGZldGNoIHRoZW0g
YWdhaW4gZXZlcnkgc28gb2Z0ZW4uIFRoZXJlJ3Mgbm8gcmVhbA0KPj4+PiBjYWNoZSBjb2hlcmVu
Y3kgcHJvdG9jb2wgZm9yIHRoaXMgaW5mb3JtYXRpb24uIChUaGF0J3MNCj4+Pj4gd2hlcmUgYSBw
TkZTIGxheW91dCBtaWdodCBiZSBtb3JlIHZhbHVhYmxlKS4NCj4+PiANCj4+PiBJZiB5b3VyIGdv
YWwgaXMgdG8gZG8gc2Vzc2lvbiB0cnVua2luZywgeW91IG9ubHkgcmVhbGx5IG5lZWQgdG8gY2hl
Y2sNCj4+PiB0aGUgZnNfbG9jYXRpb25zIGF0dHJpYnV0ZSBvbiB0aGUgcm9vdCBmaWxlIHN5c3Rl
bS4gKHNvDQo+Pj4gR0VUUk9PVEZIK0dFVEFUVFIoZnNfbG9jYXRpb25zKSkuIFRoYXQncyB0aGUg
bmF0dXJhbCBwbGFjZSBmb3IgYQ0KPj4+IHNlcnZlciB0byBhZHZlcnRpc2UgaXRzIGZ1bGwgc2V0
IG9mIElQIGFkZHJlc3NlcywgYW5kIHRoZSBzZXNzaW9uDQo+Pj4gdHJ1bmtpbmcgcHJvdG9jb2wg
aXRzZWxmIHdpbGwgYWxsb3cgeW91IHRvIHdpbm5vdyBvdXQgYW55IHRoYXQgbWlnaHQNCj4+PiBi
ZWxvbmcgdG8gYSByZXBsaWNhIHNlcnZlci4NCj4+IA0KPj4gSSB3b3JyeSB0aGF0IHJvdW5kLXJv
YmluIGNvdWxkIGJlaGF2ZSByZWFsbHkgYmFkbHkgaWYgdGhlIGNsaWVudCdzIHBhdGgNCj4+IHRv
IHRoZSB0d28gSVAgYWRkcmVzc2VzIGhhdmUgZGlmZmVyZW50IHBlcmZvcm1hbmNlIGNoYXJhY3Rl
cmlzdGljcy4gIEJ1dA0KPj4gYSBzZXJ2ZXIgc2hvdWxkIHByb2JhYmx5IHN0aWxsIGJlIGFsbG93
ZWQgdG8gYWR2ZXJ0aXNlIHRob3NlIGFzIHJlcGxpY2FzDQo+PiAoZS5nLiAgbWF5YmUgYSBzbG93
ZXIgaW50ZXJmYWNlIGlzIHVzYWJsZSBhcyBhIGZhbGxiYWNrPykuDQo+PiANCj4+IFNvIG1heWJl
IHdlIHNob3VsZCBiZSBjYXJlZnVsIGFib3V0IG1ha2luZyB0aGlzIGF1dG9tYXRpYy4gIFVubGVz
cyB0aGUNCj4+IGxvYWQtYmFsYW5jaW5nIGlzIGEgbGl0dGxlIHNtYXJ0ZXIgdGhhbiBwdXJlIHJv
dW5kIHJvYmluLiAgT3IgdW5sZXNzIHdlDQo+PiBjYW4gZ2V0IHNvbWUgbW9yZSBmaW5lLWdyYWlu
ZWQgaW5mb3JtYXRpb24gKG1heWJlIHNvbWVvbmUgY291bGQgdXNlDQo+PiBmc19sb2NhdGlvbl9p
bmZvJ3MgcHJlZmVyZW5jZSBpbmZvcm1hdGlvbiBmb3IgdGhpcz8pLg0KPiANCj4gVGhlIG11bHRp
cGF0aCBwb2xpY3kgaXMgcGx1Z2dhYmxlLiBJZiB5b3UgbmVlZCBzb21ldGhpbmcgbW9yZSBjbGV2
ZXINCj4gdGhhbiByb3VuZCByb2JpbiwgdGhlbiBmZWVsIGZyZWUgdG8gcGxheS4gSG93ZXZlciBk
byBub3RlIHRoYXQgZm9yDQo+IHBORlMgbXVsdGlwYXRoaW5nLCBib3RoIHRoZSBmaWxlcyBhbmQg
ZmxleGZpbGVzIHNwZWNzIGFyZSBjbGVhciB0aGF0DQo+IHlvdSBzaG91bGQgbm90IG1peCBzbG93
IGFuZCBmYXN0IHRyYW5zcG9ydHMuIEkgaW1hZ2luZSB5b3UgcHJvYmFibHkNCj4gd2FudCB0byBk
byB0aGUgc2FtZSBmb3IgZnNfbG9jYXRpb25zLg0KPiANCj4gQXMgZm9yIGZzX2xvY2F0aW9uc19p
bmZvLCBwbGVhc2Ugc2VlIEZTTEk0QlhfKFJFQUR8V1JJVEUpKFJBTkt8T1JERVIpLg0KDQpPSy4g
SeKAmW0gdGVzdGluZyBzZXNzaW9uIHRydW5raW5nIHVzaW5nIG5ldyBtdWx0aXBsZSBob3N0bmFt
ZSBtb3VudCBvcHRpb25zLiBJ4oCZbGwgc3VibWl0IGFub3RoZXIgUkZDIHBhdGNoc2V0Lg0KVGhl
biwgY2F2ZWF0IHBhdGNoc2V0IHJlc3BvbnNlLCBJ4oCZbGwgc3dpdGNoIGZyb20gdGhlIG11bHRp
cGxlIGhvc3RuYW1lIG1vdW50IG9wdGlvbnMgdG8gZnNfbG9jYXRpb25zX2luZm8NCg0K4oCUPkFu
ZHkNCg0K
On Thu, Feb 18, 2016 at 07:41:19PM +0000, Adamson, Andy wrote:
>
> > On Feb 18, 2016, at 1:32 PM, Trond Myklebust <[email protected]> wrote:
> >
> > On Thu, Feb 18, 2016 at 9:14 AM, J. Bruce Fields <[email protected]> wrote:
> >>
> >> On Wed, Feb 17, 2016 at 06:55:43PM -0500, Trond Myklebust wrote:
> >>> On Wed, Feb 17, 2016 at 5:52 PM, Chuck Lever <[email protected]> wrote:
> >>>>
> >>>>> On Feb 17, 2016, at 5:35 PM, Adamson, Andy <[email protected]> wrote:
> >>>>> The fs_locations would need to be requested by the client. I guess we reqest them at every mountâ€Ķ.
> >>>>
> >>>> Yep, and fetch them again every so often. There's no real
> >>>> cache coherency protocol for this information. (That's
> >>>> where a pNFS layout might be more valuable).
> >>>
> >>> If your goal is to do session trunking, you only really need to check
> >>> the fs_locations attribute on the root file system. (so
> >>> GETROOTFH+GETATTR(fs_locations)). That's the natural place for a
> >>> server to advertise its full set of IP addresses, and the session
> >>> trunking protocol itself will allow you to winnow out any that might
> >>> belong to a replica server.
> >>
> >> I worry that round-robin could behave really badly if the client's path
> >> to the two IP addresses have different performance characteristics. But
> >> a server should probably still be allowed to advertise those as replicas
> >> (e.g. maybe a slower interface is usable as a fallback?).
> >>
> >> So maybe we should be careful about making this automatic. Unless the
> >> load-balancing is a little smarter than pure round robin. Or unless we
> >> can get some more fine-grained information (maybe someone could use
> >> fs_location_info's preference information for this?).
> >
> > The multipath policy is pluggable. If you need something more clever
> > than round robin, then feel free to play. However do note that for
> > pNFS multipathing, both the files and flexfiles specs are clear that
> > you should not mix slow and fast transports. I imagine you probably
> > want to do the same for fs_locations.
> >
> > As for fs_locations_info, please see FSLI4BX_(READ|WRITE)(RANK|ORDER).
>
> OK. I’m testing session trunking using new multiple hostname mount options. I’ll submit another RFC patchset.
> Then, caveat patchset response, I’ll switch from the multiple hostname mount options to fs_locations_info
You mean you want to remove support for the commandline list of
hostnames at that point?
I'd rather keep support for listing them on the commandline. I think
the fs_locations_info is a little more complicated than I did at first
look. (Among other things, it requires server support, and some thought
about how exactly to interpret that fs_locations_info preference
information.)
--b.
> On Feb 18, 2016, at 3:39 PM, J. Bruce Fields <[email protected]> wrote:
>
> On Thu, Feb 18, 2016 at 07:41:19PM +0000, Adamson, Andy wrote:
>>
>>> On Feb 18, 2016, at 1:32 PM, Trond Myklebust <[email protected]> wrote:
>>>
>>> On Thu, Feb 18, 2016 at 9:14 AM, J. Bruce Fields <[email protected]> wrote:
>>>>
>>>> On Wed, Feb 17, 2016 at 06:55:43PM -0500, Trond Myklebust wrote:
>>>>> On Wed, Feb 17, 2016 at 5:52 PM, Chuck Lever <[email protected]> wrote:
>>>>>>
>>>>>>> On Feb 17, 2016, at 5:35 PM, Adamson, Andy <[email protected]> wrote:
>>>>>>> The fs_locations would need to be requested by the client. I guess we reqest them at every mountâ€Ķ.
>>>>>>
>>>>>> Yep, and fetch them again every so often. There's no real
>>>>>> cache coherency protocol for this information. (That's
>>>>>> where a pNFS layout might be more valuable).
>>>>>
>>>>> If your goal is to do session trunking, you only really need to check
>>>>> the fs_locations attribute on the root file system. (so
>>>>> GETROOTFH+GETATTR(fs_locations)). That's the natural place for a
>>>>> server to advertise its full set of IP addresses, and the session
>>>>> trunking protocol itself will allow you to winnow out any that might
>>>>> belong to a replica server.
>>>>
>>>> I worry that round-robin could behave really badly if the client's path
>>>> to the two IP addresses have different performance characteristics. But
>>>> a server should probably still be allowed to advertise those as replicas
>>>> (e.g. maybe a slower interface is usable as a fallback?).
>>>>
>>>> So maybe we should be careful about making this automatic. Unless the
>>>> load-balancing is a little smarter than pure round robin. Or unless we
>>>> can get some more fine-grained information (maybe someone could use
>>>> fs_location_info's preference information for this?).
>>>
>>> The multipath policy is pluggable. If you need something more clever
>>> than round robin, then feel free to play. However do note that for
>>> pNFS multipathing, both the files and flexfiles specs are clear that
>>> you should not mix slow and fast transports. I imagine you probably
>>> want to do the same for fs_locations.
Right, and barring the ability to mix TCP and RDMA transports,
I think there's no reliable way either the client or the
server can tell that any particular path has compromised
performance. It's going to have to be up to administrators
to make sure this is configured correctly, at least for now.
>>> As for fs_locations_info, please see FSLI4BX_(READ|WRITE)(RANK|ORDER).
>>
>> OK. I’m testing session trunking using new multiple hostname mount options. I’ll submit another RFC patchset.
>> Then, caveat patchset response, I’ll switch from the multiple hostname mount options to fs_locations_info
>
> You mean you want to remove support for the commandline list of
> hostnames at that point?
>
> I'd rather keep support for listing them on the commandline. I think
> the fs_locations_info is a little more complicated than I did at first
> look. (Among other things, it requires server support, and some thought
> about how exactly to interpret that fs_locations_info preference
> information.)
True; there's a reason I never got to implementing
fs_locations_info on the Linux server for FedFS. There
are sticky problems around the mountd upcall that is
used to communicate this information to the kernel,
for example.
However, I don't agree that this is a good reason to
go with multiple hostnames on the mount command line.
I like Andy's plan to keep this CLI change out of the
long term upstream code, but continue to use it for
testing.
--
Chuck Lever
On Thu, Feb 18, 2016 at 04:29:46PM -0500, Chuck Lever wrote:
> True; there's a reason I never got to implementing
> fs_locations_info on the Linux server for FedFS. There
> are sticky problems around the mountd upcall that is
> used to communicate this information to the kernel,
> for example.
>
> However, I don't agree that this is a good reason to
> go with multiple hostnames on the mount command line.
> I like Andy's plan to keep this CLI change out of the
> long term upstream code, but continue to use it for
> testing.
Even if we get an fs_locations_info-based solution to the point where
it's the best default, don't you think we'll still want some kind of
manual override on the client?
The decision here has to do with the path between client and server,
and the server may not always be in the best position to make that
decision.
--b.
> On Feb 19, 2016, at 10:01 AM, J. Bruce Fields <[email protected]> wrote:
>
> On Thu, Feb 18, 2016 at 04:29:46PM -0500, Chuck Lever wrote:
>> True; there's a reason I never got to implementing
>> fs_locations_info on the Linux server for FedFS. There
>> are sticky problems around the mountd upcall that is
>> used to communicate this information to the kernel,
>> for example.
>>
>> However, I don't agree that this is a good reason to
>> go with multiple hostnames on the mount command line.
>> I like Andy's plan to keep this CLI change out of the
>> long term upstream code, but continue to use it for
>> testing.
>
> Even if we get an fs_locations_info-based solution to the point where
> it's the best default, don't you think we'll still want some kind of
> manual override on the client?
Perhaps! But that doesn't mean multiple hostnames in
particular is the right CLI.
Suppose a "notrunk" mount option was added. The client
would ignore the server's fs_locations_info and use
only the hostname provided on the command line.
Another idea is to provide a blacklist as a mount
option. "notrunk=IPv4,IPv4,[IPv6],[IPv6]"
> The decision here has to do with the path between client and server,
> and the server may not always be in the best position to make that
> decision.
The server is not making any decision about the path.
It simply provides a list of its i/f's.
The client may choose to use one or more of the IP
addresses on the server's list. The client is not
required to use all the server's IP addresses, and
it does have the ability to determine that some
addresses are not reachable (eg IPv6 addresses,
private networks, and so on), though that may take
a timeout or two.
But back to the client CLI:
With multiple hostnames, client admins can specify
arbitrary hostnames on the command line; hostnames
for instance that correspond to some other server.
And, network conditions can change at anytime, making
the original set of trunks lop-sided, or some trunks
may become unreachable. What if the server reboots
with new i/f's or with one or more removed? The
client would likely have to remount in these cases
to adapt to network configuration changes.
Note that multiple hostnames could be nailed into
/etc/fstab on potentially hundreds of clients. When
server or network configuration changes, there would
have to be a manual change on all these clients.
--
Chuck Lever