Received-SPF: pass (google.com: domain of linux-nfs+bounces-3352-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Precedence: bulk
MIME-Version: 1.0
Date: Thu, 23 May 2024 16:35:18 +0200
From: Richard Kojedzinszky <richard+debian+bugreport@kojedz.in>
To: linux-nfs@vger.kernel.org, 1071501@bugs.debian.org
Subject: Re: Linux NFS client hangs in nfs4_lookup_revalidate
In-Reply-To: <73e081764d06746be27c5f0d2f181938@kojedz.in>
References: <0473c552b6fd8e96ef2ffbf0435a7552@kojedz.in>
 <73e081764d06746be27c5f0d2f181938@kojedz.in>
Message-ID: <162d12087ba8374a57e2263d7ea762b5@kojedz.in>
Content-Type: multipart/mixed;
 boundary="=_5fadaa6cfb7f5c70b7eda0bab46df208"

--=_5fadaa6cfb7f5c70b7eda0bab46df208
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8;
 format=flowed

Dear devs,

I am attaching a stripped down version of the little program which 
triggers the bug very quickly, in a few minutes in my test lab. It 
turned out that a single NFS mountpoint is enough. Just start the 
program giving it the NFS mount as first argument. It will chdir there, 
and do file operations, which will trigger a lockup in a few minutes.

Please take a look at it.

Thanks in advance,
Richard

2024-05-23 14:12 időpontban Richard Kojedzinszky ezt írta:
> Dear devs,
> 
> Now bisecting turned out that 3c59366c207e4c6c6569524af606baf017a55c61 
> is the bad commit for me. Strangely it only affects my dovecot process 
> accessing data over NFS.
> 
> Can you please confirm that this may be a bad commit?
> 
> My earlier attached programs may be used to demonstrate/trigger the 
> issue. It even could be stripped down to minimal operations to trigger 
> the bug.
> 
> Thanks in advance,
> Richard
> 
> 
> 2024-05-23 09:10 időpontban Richard Kojedzinszky ezt írta:
>> Dear NFS developers,
>> 
>> I am running multiple PODs on a Kubernetes node, they all mount 
>> different NFS shares from the same nfs server. I started to notice 
>> hangups in my dovecot process after I switched to Debian's kernel from 
>> upstream 5.15. You can find Debian bugreport at 
>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1071501.
>> 
>> So, effectively I am running dovecot in Kubernetes, and dovecot's data 
>> directory is accessed over NFS. Eventually one dovecot process stucks 
>> in nfs4_lookup_revalidate(). From that point, that process cannot be 
>> killed, howewer, other processes can access NFS as normal. Also, 
>> another dovecot process running on the very same node accessing the 
>> same NFS share works too.
>> 
>> Now, I am still in the process of bisecting, howewer, I cannot 
>> reliably trigger the bug. Originally it took a few days after I've 
>> noticed a hanging process. Now I am trying to mimic file operations 
>> what dovecot does in a faster way. Now it seems that it triggers the 
>> bug in a few hours, howewer, during bisects, I can still make 
>> mistakes.
>> 
>> I've scheduled many of my applications which use NFS shares to the 
>> same node, to have more NFS load on that node.
>> 
>> I am attaching my simple app which triggers the bug in a few hours, at 
>> least in my lab. I have two dedicated NFS shares for this test case, 
>> and I am running 3 instances of the applications for both shares. 
>> Also, I am running other production applications on the same node 
>> which also use NFS, howewer, I dont experience lockups with them. They 
>> are librenms, prometheus, and a docker private registry. This way I 
>> dont know if running the attached app only is enough to trigger the 
>> bug.
>> 
>> Once I have a suspectible commit based on my bisecting process, I will 
>> report it here.
>> 
>> My NFS server is a TrueNAS, based on FreeBSD 13.3.
>> 
>> Thanks in advance,
>> Richard

--=_5fadaa6cfb7f5c70b7eda0bab46df208
Content-Transfer-Encoding: base64
Content-Type: application/x-tar;
 name=ds.tar
Content-Disposition: attachment;
 filename=ds.tar;
 size=10240

ZG92ZWNvdC1zaW11bGF0b3IvAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADAwMDA3NTUAMDAwMTc1
MQAwMDAxNzUxADAwMDAwMDAwMDAwADE0NjIzNjUxNDQ0ADAxNDExNQAgNQAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB1c3RhciAgAGtyaWNoeQAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAa3JpY2h5AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABk
b3ZlY290LXNpbXVsYXRvci9tYWluLmdvAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMDAwMDY0NAAwMDAxNzUx
ADAwMDE3NTEAMDAwMDAwMDI1NDUAMTQ2MjM2NTE0NDQAMDE1Mzc2ACAwAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHVzdGFyICAAa3JpY2h5AAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAABrcmljaHkAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHBh
Y2thZ2UgbWFpbgoKaW1wb3J0ICgKCSJjb250ZXh0IgoJImxvZyIKCSJuZXQvaHR0cCIKCSJvcyIK
CSJvcy9zaWduYWwiCgkic3luYyIKCSJzeXNjYWxsIgoKCSJnaXRodWIuY29tL3Byb21ldGhldXMv
Y2xpZW50X2dvbGFuZy9wcm9tZXRoZXVzIgoJImdpdGh1Yi5jb20vcHJvbWV0aGV1cy9jbGllbnRf
Z29sYW5nL3Byb21ldGhldXMvcHJvbWh0dHAiCikKCmNvbnN0ICgKCWxvY2tGaWxlICAgID0gIm1h
aWwubG9jayIKCWxpc3RGaWxlICAgID0gIm1haWwubGlzdCIKCWxpc3RGaWxlTmV3ID0gIm1haWwu
bGlzdC5uZXciCikKCnZhciAoCgltZXRyaWNzID0gcHJvbWV0aGV1cy5OZXdIaXN0b2dyYW1WZWMo
cHJvbWV0aGV1cy5IaXN0b2dyYW1PcHRzewoJCVN1YnN5c3RlbTogImRvdmVjb3RzaW0iLAoJCU5h
bWU6ICAgICAgIm9wZXJhdGlvbnMiLAoJCUhlbHA6ICAgICAgIlNpbXVsYXRlZCBvcGVhdGlvbnMi
LAoJfSwgW11zdHJpbmd7InBhdGgiLCAiaW5zdCIsICJvcCJ9KQoKCXBtZXRyaWNzIHByb21ldGhl
dXMuT2JzZXJ2ZXJWZWMKKQoKZnVuYyBtYWluKCkgewoJcHJvbWV0aGV1cy5NdXN0UmVnaXN0ZXIo
bWV0cmljcykKCglwYXRoIDo9IG9zLkFyZ3NbMV0KCWlmIGVyciA6PSBvcy5DaGRpcihwYXRoKTsg
ZXJyICE9IG5pbCB7CgkJbG9nLkZhdGFsKGVycikKCX0KCglwbWV0cmljcyA9IG1ldHJpY3MuTXVz
dEN1cnJ5V2l0aChwcm9tZXRoZXVzLkxhYmVsc3sicGF0aCI6IHBhdGh9KQoKCWN0eCwgY2FuY2Vs
IDo9IGNvbnRleHQuV2l0aENhbmNlbChjb250ZXh0LkJhY2tncm91bmQoKSkKCWRlZmVyIGNhbmNl
bCgpCgoJZ28gZnVuYygpIHsKCQlzaWdjaGFuIDo9IG1ha2UoY2hhbiBvcy5TaWduYWwsIDEpCgkJ
c2lnbmFsLk5vdGlmeShzaWdjaGFuLCBzeXNjYWxsLlNJR0lOVCwgc3lzY2FsbC5TSUdURVJNKQoK
CQk8LXNpZ2NoYW4KCgkJbG9nLlByaW50KCJFeGl0aW5nIikKCgkJY2FuY2VsKCkKCX0oKQoKCXdn
IDo9ICZzeW5jLldhaXRHcm91cHt9CgoJbG9nLlByaW50KCJTdGFydGluZyByZWFkZXJzIGFuZCB3
cml0ZXJzIikKCglmb3IgaSA6PSAwOyBpIDwgNTsgaSsrIHsKCQl3Zy5BZGQoMSkKCQlnbyBmdW5j
KGluc3RhbmNlIGludCkgewoJCQlkZWZlciB3Zy5Eb25lKCkKCgkJCXcgOj0gJnJlYWRlcnt9CgoJ
CQl3LnJ1bihjdHgsIGluc3RhbmNlKQoJCX0oaSkKCX0KCglmb3IgaSA6PSAwOyBpIDwgNTsgaSsr
IHsKCQl3Zy5BZGQoMSkKCQlnbyBmdW5jKGluc3RhbmNlIGludCkgewoJCQlkZWZlciB3Zy5Eb25l
KCkKCgkJCXcgOj0gJndyaXRlcnt9CgoJCQl3LnJ1bihjdHgsIGluc3RhbmNlKQoJCX0oaSkKCX0K
CglnbyBodHRwLkxpc3RlbkFuZFNlcnZlKCI6OTA5MCIsIHByb21odHRwLkhhbmRsZXIoKSkKCgl3
Zy5XYWl0KCkKfQoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGRvdmVj
b3Qtc2ltdWxhdG9yL3JlYWRlci5nbwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwMDAwNjQ0ADAwMDE3NTEAMDAw
MTc1MQAwMDAwMDAwMzExMQAxNDYyMzY1MTQzMQAwMTU2NzYAIDAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAdXN0YXIgIABrcmljaHkAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAGtyaWNoeQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAcGFja2Fn
ZSBtYWluCgppbXBvcnQgKAoJImJ1ZmlvIgoJImNvbnRleHQiCgkiZm10IgoJImxvZyIKCSJtYXRo
L3JhbmQiCgkib3MiCgkidGltZSIKKQoKdHlwZSByZWFkZXIgc3RydWN0IHsKCWwgICAgICAgICAg
ICpsb2NrZmlsZQoJbGFzdE1vZFRpbWUgdGltZS5UaW1lCn0KCmZ1bmMgKHIgKnJlYWRlcikgcnVu
KGN0eCBjb250ZXh0LkNvbnRleHQsIGluc3RhbmNlIGludCkgewoJbSA6PSBwbWV0cmljcy5XaXRo
TGFiZWxWYWx1ZXMoZm10LlNwcmludGYoIiVkIiwgaW5zdGFuY2UpLCAicmVhZCIpCgoJci5sID0g
JmxvY2tmaWxle2ZpbGU6IGxvY2tGaWxlfQoKCXdkIDo9IG1ha2UoY2hhbiBib29sKQoKCWdvIGZ1
bmMoKSB7CgkJdGltZXIgOj0gdGltZS5OZXdUaW1lcih0aW1lLlNlY29uZCkKCQlkZWZlciB0aW1l
ci5TdG9wKCkKCgkJZm9yIHsKCQkJc2VsZWN0IHsKCQkJY2FzZSA8LWN0eC5Eb25lKCk6CgkJCQly
ZXR1cm4KCQkJY2FzZSA8LXRpbWVyLkM6CgkJCQlsb2cuUHJpbnRmKCJyZWFkZXIoJWQpIGhhcyBi
ZWVuIGJsb2NrZWQgZm9yIGEgc2Vjb25kIiwgaW5zdGFuY2UpCgkJCQlyZXR1cm4KCQkJY2FzZSA8
LXdkOgoJCQkJaWYgIXRpbWVyLlN0b3AoKSB7CgkJCQkJPC10aW1lci5DCgkJCQl9CgkJCQl0aW1l
ci5SZXNldCh0aW1lLlNlY29uZCkKCQkJfQoJCX0KCX0oKQoKCWZvciB7CgkJc2VsZWN0IHsKCQlj
YXNlIDwtY3R4LkRvbmUoKToKCQkJcmV0dXJuCgkJZGVmYXVsdDoKCQl9CgoJCXN0YXJ0IDo9IHRp
bWUuTm93KCkKCQlpZiBlcnIgOj0gci5kb19yZWFkKCk7IGVyciA9PSBuaWwgewoJCQltLk9ic2Vy
dmUodGltZS5TaW5jZShzdGFydCkuU2Vjb25kcygpKQoJCX0KCgkJd2QgPC0gdHJ1ZQoJfQp9Cgpm
dW5jIChyICpyZWFkZXIpIGRvX3JlYWQoKSBlcnJvciB7CglzdGF0LCBlcnIgOj0gb3MuU3RhdChs
aXN0RmlsZSkKCWlmIGVyciAhPSBuaWwgewoJCXJldHVybiBlcnIKCX0KCgltb2RUaW1lIDo9IHN0
YXQuTW9kVGltZSgpCglpZiAhbW9kVGltZS5BZnRlcihyLmxhc3RNb2RUaW1lKSB7CgkJcmV0dXJu
IG5pbAoJfQoKCWlmIGVyciA9IHIubC5Mb2NrKCk7IGVyciAhPSBuaWwgewoJCXJldHVybiBlcnIK
CX0KCWRlZmVyIHIubC5VbmxvY2soKQoKCXJmaCwgZXJyIDo9IG9zLk9wZW4obGlzdEZpbGUpCglp
ZiBlcnIgIT0gbmlsIHsKCQlyZXR1cm4gZXJyCgl9CglkZWZlciByZmguQ2xvc2UoKQoKCXdmaCwg
ZXJyIDo9IG9zLk9wZW5GaWxlKGxpc3RGaWxlTmV3LCBvcy5PX0NSRUFURXxvcy5PX1dST05MWSwg
MG82MDApCglpZiBlcnIgIT0gbmlsIHsKCQlyZXR1cm4gZXJyCgl9CglkZWZlciB3ZmguQ2xvc2Uo
KQoKCXJyIDo9IGJ1ZmlvLk5ld1JlYWRlcihyZmgpCgl3ciA6PSBidWZpby5OZXdXcml0ZXIod2Zo
KQoKCWZvciB7CgkJbGluZSwgXyA6PSByci5SZWFkU3RyaW5nKCdcbicpCgkJaWYgbGluZSA9PSAi
IiB7CgkJCWJyZWFrCgkJfQoKCQlpZiByYW5kLkludG4oMTApIDwgNCB7IC8vIGtlZXAgZmlsZXMg
YXQgNDAlIGNoYW5jZQoJCQl3ci5Xcml0ZVN0cmluZyhsaW5lKQoJCX0KCX0KCgl3ci5GbHVzaCgp
Cgl3ZmguQ2xvc2UoKQoKCW9zLlJlbmFtZShsaXN0RmlsZU5ldywgbGlzdEZpbGUpCgoJcmV0dXJu
IG5pbAp9CgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABkb3ZlY290LXNp
bXVsYXRvci93cml0ZXIuZ28AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMDAwMDY0NAAwMDAxNzUxADAwMDE3NTEA
MDAwMDAwMDIyMjQAMTQ2MjM2NTE0MzEAMDE1NzU0ACAwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAHVzdGFyICAAa3JpY2h5AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AABrcmljaHkAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHBhY2thZ2UgbWFp
bgoKaW1wb3J0ICgKCSJjb250ZXh0IgoJImZtdCIKCSJsb2ciCgkibWF0aC9yYW5kIgoJIm9zIgoJ
InRpbWUiCikKCnR5cGUgd3JpdGVyIHN0cnVjdCB7CglsICpsb2NrZmlsZQp9CgpmdW5jICh3ICp3
cml0ZXIpIHJ1bihjdHggY29udGV4dC5Db250ZXh0LCBpbnN0YW5jZSBpbnQpIHsKCW0gOj0gcG1l
dHJpY3MuV2l0aExhYmVsVmFsdWVzKGZtdC5TcHJpbnRmKCIlZCIsIGluc3RhbmNlKSwgIndyaXRl
IikKCgl3LmwgPSAmbG9ja2ZpbGV7ZmlsZTogbG9ja0ZpbGV9CgoJd2QgOj0gbWFrZShjaGFuIGJv
b2wpCgoJZ28gZnVuYygpIHsKCQl0aW1lciA6PSB0aW1lLk5ld1RpbWVyKHRpbWUuU2Vjb25kKQoJ
CWRlZmVyIHRpbWVyLlN0b3AoKQoKCQlmb3IgewoJCQlzZWxlY3QgewoJCQljYXNlIDwtY3R4LkRv
bmUoKToKCQkJCXJldHVybgoJCQljYXNlIDwtdGltZXIuQzoKCQkJCWxvZy5QcmludGYoIndyaXRl
ciglZCkgaGFzIGJlZW4gYmxvY2tlZCBmb3IgYSBzZWNvbmQiLCBpbnN0YW5jZSkKCQkJCXJldHVy
bgoJCQljYXNlIDwtd2Q6CgkJCQlpZiAhdGltZXIuU3RvcCgpIHsKCQkJCQk8LXRpbWVyLkMKCQkJ
CX0KCQkJCXRpbWVyLlJlc2V0KHRpbWUuU2Vjb25kKQoJCQl9CgkJfQoJfSgpCgoJZm9yIHsKCQlz
ZWxlY3QgewoJCWNhc2UgPC1jdHguRG9uZSgpOgoJCQlyZXR1cm4KCQlkZWZhdWx0OgoJCX0KCgkJ
c3RhcnQgOj0gdGltZS5Ob3coKQoJCWlmIGVyciA6PSB3LmRvX3dyaXRlKCk7IGVyciA9PSBuaWwg
ewoJCQltLk9ic2VydmUodGltZS5TaW5jZShzdGFydCkuU2Vjb25kcygpKQoJCX0KCgkJd2QgPC0g
dHJ1ZQoJfQp9CgpmdW5jICh3ICp3cml0ZXIpIGRvX3dyaXRlKCkgZXJyb3IgewoJaWYgZXJyIDo9
IHcubC5Mb2NrKCk7IGVyciAhPSBuaWwgewoJCXJldHVybiBlcnIKCX0KCWRlZmVyIHcubC5Vbmxv
Y2soKQoKCWZoLCBlcnIgOj0gb3MuT3BlbkZpbGUobGlzdEZpbGUsIG9zLk9fQ1JFQVRFfG9zLk9f
V1JPTkxZfG9zLk9fQVBQRU5ELCAwbzYwMCkKCWlmIGVyciAhPSBuaWwgewoJCXJldHVybiBlcnIK
CX0KCWRlZmVyIGZoLkNsb3NlKCkKCglsaW5lIDo9IGZtdC5TcHJpbnRmKCJmaWxlLSVkLiVkXG4i
LCB0aW1lLk5vdygpLk5hbm9zZWNvbmQoKSwgcmFuZC5JbnQoKSkKCWZoLldyaXRlKFtdYnl0ZShs
aW5lKSkKCglyZXR1cm4gbmlsCn0KAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGRvdmVjb3Qtc2ltdWxh
dG9yL2xvY2tmaWxlLmdvAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwMDAwNjQ0ADAwMDE3NTEAMDAwMTc1MQAwMDAw
MDAwMTEzNAAxNDYyMzY1MTQzMQAwMTYyMjcAIDAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAdXN0YXIgIABrcmljaHkAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGty
aWNoeQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAcGFja2FnZSBtYWluCgpp
bXBvcnQgKAoJIm1hdGgvcmFuZCIKCSJvcyIKCSJ0aW1lIgopCgp0eXBlIGxvY2tmaWxlIHN0cnVj
dCB7CglmaWxlIHN0cmluZwoJZmggICAqb3MuRmlsZQp9CgpmdW5jIChsICpsb2NrZmlsZSkgTG9j
aygpIChlcnIgZXJyb3IpIHsKCWkgOj0gMAoKCWZvciB7CgkJaWYgXywgZXJyID0gb3MuU3RhdChs
LmZpbGUpOyBlcnIgIT0gbmlsICYmIG9zLklzTm90RXhpc3QoZXJyKSB7CgkJCWwuZmgsIGVyciA9
IG9zLk9wZW5GaWxlKGwuZmlsZSwgb3MuT19DUkVBVEV8b3MuT19SRFdSfG9zLk9fRVhDTCwgMG82
NDQpCgkJCWlmIGVyciA9PSBuaWwgewoJCQkJcmV0dXJuCgkJCX0KCQl9IGVsc2UgewoJCQllcnIg
PSBvcy5FcnJFeGlzdAoJCX0KCgkJaSsrCgkJaWYgaSA9PSAxNSB7CgkJCXJldHVybgoJCX0KCgkJ
dGltZS5TbGVlcCh0aW1lLk1pbGxpc2Vjb25kICogdGltZS5EdXJhdGlvbigoMSArIHJhbmQuSW50
bigxMCkpKSkKCX0KfQoKZnVuYyAobCAqbG9ja2ZpbGUpIFVubG9jaygpIChlcnIgZXJyb3IpIHsK
CWlmIGwuZmggPT0gbmlsIHsKCQlyZXR1cm4KCX0KCglsLmZoLkNsb3NlKCkKCglvcy5SZW1vdmUo
bC5maWxlKQoKCXJldHVybgp9CgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==
--=_5fadaa6cfb7f5c70b7eda0bab46df208--